2022-04-03T19:31:19.417 INFO:root:teuthology version: 1.1.0-d877b023 2022-04-03T19:31:19.419 DEBUG:teuthology.run:Teuthology command: teuthology --name vshankar-2022-04-03_15:22:48-fs-wip-vshankar-testing-20220403-170344-testing-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} --verbose --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2022-04-03_15:22:48-fs-wip-vshankar-testing-20220403-170344-testing-default-smithi/6775570 -- /home/teuthworker/archive/vshankar-2022-04-03_15:22:48-fs-wip-vshankar-testing-20220403-170344-testing-default-smithi/6775570/orig.config.yaml 2022-04-03T19:31:19.476 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-04-03T19:31:19.546 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2022-04-03_15:22:48-fs-wip-vshankar-testing-20220403-170344-testing-default-smithi/6775570 branch: wip-vshankar-testing-20220403-170344 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: vshankar@redhat.com first_in_suite: false job_id: '6775570' kernel: client: branch: testing flavor: default kdb: true sha1: 26d8230c8f919f5e2d8da795fb3fbca12f8ebd85 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2022-04-03_15:22:48-fs-wip-vshankar-testing-20220403-170344-testing-default-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-vshankar-testing-20220403-170344 ceph: cephfs: mds_dir_prefetch: true session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: false mds op complaint time: 180 mds verify scatter: true mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - Replacing daemon mds log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: cfb8f943163b374162da0d7b0240f267dd46e4e1 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: cfb8f943163b374162da0d7b0240f267dd46e4e1 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20220403-170344 sha1: cfb8f943163b374162da0d7b0240f267dd46e4e1 owner: scheduled_vshankar@teuthology priority: 100 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 sha1: cfb8f943163b374162da0d7b0240f267dd46e4e1 sleep_before_teardown: 0 suite: fs suite_branch: wip-vshankar-testing-20220403-170344 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_cfb8f943163b374162da0d7b0240f267dd46e4e1/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: cfb8f943163b374162da0d7b0240f267dd46e4e1 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi077.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDC2KrHT+jt6QF/Qu3rNGoRnyFtqyrpecqHQBy+Ff7ubrJyqbuqVmcMVGk11XWW+H0KmzVLagrcZkVHJ3L0MLwB+aVdpulCHKEIHeVzMrLh7cV3DQ3RlQdL5uYg6yh138FzItXXVH/vKYeJwMQR5eQ+tuaivstmzCvmekQ2NtiNKJP3zD27if/5mXWVJtUT+ppHTEz9ZmtB6LRewmKTrqZXnsX0wm1iGNdJnoCuRw6nXwmtu2TV4rxyOqol/5MCG/sK2nEbKUD4KPS0K6WJZ8lSiWui2V+Ul8xdbGBOiXXkbpHMccLFaBa9QxEb+ywpf6vRBH+P7JLYFbMjvz3knoGkCyyxxFftdCLEibqxtXqhUmDzTCx6FkzDxKjtPzDC6sxU9VajeubSPC2sIJhoa2bPghkzQpCVUN6j0f0q6372woza/biE1NXtHdooSHlTWq0LAyhLRmvSIpw0gkJREQ6DurPfYgAawH/V4Z6ZMIN+EU2CIMQxFlyNiLPBqMQEE/8= smithi186.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPXYqvDQAuAe1jDXYrnh9kUFrXHgN4RYdGIEPXNa2Wijxrc7Fuj/rNhJHK+7mjqgVENXYgLaIrKZL4GPCMjiF+kiJO/W3+zmzpQqU9MEEQHykotUYOmlDV6IgtHxF1UeU8kl7pG3b0JBtImYwCJAvlj2Sja7CYBygFSxq2EpUFhIIWuIw6iEO+YCFXn9O7meZ5B3rKu8cY9pICRJauSZiI+nIvNSaEWcvZEqVqUQnUakkLRBGvnt7PbMtgk08tzO0Nrtt+XLbJACBqxTy3M/G+QnM/+MAXrijFsqmSxOaEkoxxlCBjpw/J9Ig57dkpcMdE+7LR3rCDmFs5a8SMAJGuZetberuhGr+CScKT689t/6D9NAHFngiyU7HnGOeyrn7Dkkj8qV5HTz4L5jVn3Nsl7Wo3xmkZgCzESK8lRE8HhJD3MWktTFPvXKRsK75UGv0BrGk7Lxwq4xgVgEoaopsTSN8IURj5WXAiOnL68QR4xCWh0BK/bTNW8vZdKumm16c= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - kclient: null - mds_thrash: null - workunit: clients: all: - fs/snaps teuthology_branch: master teuthology_sha1: d877b023166308f282187444de8e8e725ffb27d2 timestamp: 2022-04-03_15:22:48 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-04-03T19:31:19.546 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_cfb8f943163b374162da0d7b0240f267dd46e4e1/qa; will attempt to use it 2022-04-03T19:31:19.547 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_cfb8f943163b374162da0d7b0240f267dd46e4e1/qa/tasks 2022-04-03T19:31:19.547 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-04-03T19:31:19.549 INFO:teuthology.task.internal:Checking packages... 2022-04-03T19:31:19.570 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'cfb8f943163b374162da0d7b0240f267dd46e4e1' 2022-04-03T19:31:19.570 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-04-03T19:31:19.571 INFO:teuthology.packaging:ref: None 2022-04-03T19:31:19.571 INFO:teuthology.packaging:tag: None 2022-04-03T19:31:19.571 INFO:teuthology.packaging:branch: wip-vshankar-testing-20220403-170344 2022-04-03T19:31:19.572 INFO:teuthology.packaging:sha1: cfb8f943163b374162da0d7b0240f267dd46e4e1 2022-04-03T19:31:19.572 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-vshankar-testing-20220403-170344 2022-04-03T19:31:19.725 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-11442.gcfb8f943 2022-04-03T19:31:19.725 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-04-03T19:31:19.736 INFO:teuthology.task.internal:no buildpackages task found 2022-04-03T19:31:19.736 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-04-03T19:31:19.776 INFO:teuthology.task.internal:Saving configuration 2022-04-03T19:31:19.796 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-04-03T19:31:19.809 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-04-03T19:31:19.833 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2022-04-03_15:22:48-fs-wip-vshankar-testing-20220403-170344-testing-default-smithi/6775570', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-04-03 19:26:22.018538', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBsUsiqquVli76D17WWzYZS/K175onWUWB8CwCqQWw1KhZ2GO7eezWkJv0jdJZt6jTru/URJ7cc/O6mCj3j3MpQ='} 2022-04-03T19:31:19.854 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2022-04-03_15:22:48-fs-wip-vshankar-testing-20220403-170344-testing-default-smithi/6775570', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-04-03 19:26:22.017205', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGceCe5W0XK/VAgiMzigUjIEtGSt6ZZ3XkSEIUVQG0WaL3yFIp2OrhoHO9qA1FN/IKe7uBs3m4euahJqNPj9OSg='} 2022-04-03T19:31:19.855 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-04-03T19:31:19.865 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2022-04-03T19:31:19.865 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2022-04-03T19:31:19.865 INFO:teuthology.run_tasks:Running task console_log... 2022-04-03T19:31:19.993 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f29fa50bc80>, signals=[15]) 2022-04-03T19:31:19.994 INFO:teuthology.run_tasks:Running task internal.connect... 2022-04-03T19:31:20.005 INFO:teuthology.task.internal:Opening connections... 2022-04-03T19:31:20.005 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2022-04-03T19:31:20.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-03T19:31:20.080 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2022-04-03T19:31:20.081 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-03T19:31:20.206 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-04-03T19:31:20.217 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2022-04-03T19:31:20.235 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2022-04-03T19:31:20.235 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2022-04-03T19:31:20.292 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="Red Hat Enterprise Linux" 2022-04-03T19:31:20.293 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="8.4 (Ootpa)" 2022-04-03T19:31:20.293 INFO:teuthology.orchestra.run.smithi077.stdout:ID="rhel" 2022-04-03T19:31:20.293 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="fedora" 2022-04-03T19:31:20.293 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="8.4" 2022-04-03T19:31:20.294 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el8" 2022-04-03T19:31:20.294 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-04-03T19:31:20.294 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2022-04-03T19:31:20.294 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-04-03T19:31:20.295 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://www.redhat.com/" 2022-04-03T19:31:20.295 INFO:teuthology.orchestra.run.smithi077.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-04-03T19:31:20.295 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-04-03T19:31:20.295 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:31:20.296 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-04-03T19:31:20.296 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-04-03T19:31:20.296 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-04-03T19:31:20.296 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-04-03T19:31:20.297 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2022-04-03T19:31:20.318 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2022-04-03T19:31:20.340 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2022-04-03T19:31:20.342 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2022-04-03T19:31:20.403 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="Red Hat Enterprise Linux" 2022-04-03T19:31:20.404 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="8.4 (Ootpa)" 2022-04-03T19:31:20.405 INFO:teuthology.orchestra.run.smithi186.stdout:ID="rhel" 2022-04-03T19:31:20.405 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="fedora" 2022-04-03T19:31:20.406 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="8.4" 2022-04-03T19:31:20.406 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el8" 2022-04-03T19:31:20.407 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-04-03T19:31:20.408 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2022-04-03T19:31:20.408 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-04-03T19:31:20.409 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://www.redhat.com/" 2022-04-03T19:31:20.409 INFO:teuthology.orchestra.run.smithi186.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-04-03T19:31:20.410 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-04-03T19:31:20.410 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:31:20.410 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-04-03T19:31:20.411 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-04-03T19:31:20.411 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-04-03T19:31:20.411 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-04-03T19:31:20.412 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2022-04-03T19:31:20.456 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-04-03T19:31:20.470 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-04-03T19:31:20.630 INFO:teuthology.task.internal:Checking for old test directory... 2022-04-03T19:31:20.630 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2022-04-03T19:31:20.634 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2022-04-03T19:31:20.655 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-04-03T19:31:20.769 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-04-03T19:31:20.769 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2022-04-03T19:31:20.772 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2022-04-03T19:31:20.804 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-04-03T19:31:20.952 INFO:teuthology.run_tasks:Running task kernel... 2022-04-03T19:31:20.978 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '26d8230c8f919f5e2d8da795fb3fbca12f8ebd85'} 2022-04-03T19:31:20.979 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2022-04-03T19:31:20.979 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2022-04-03T19:31:20.980 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2022-04-03T19:31:20.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-03T19:31:20.998 DEBUG:teuthology.orchestra.run.smithi077:> sudo lsb_release -is 2022-04-03T19:31:21.106 INFO:teuthology.orchestra.run.smithi077.stdout:RedHatEnterprise 2022-04-03T19:31:21.108 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-04-03T19:31:21.108 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2022-04-03T19:31:21.364 DEBUG:teuthology.task.kernel:sha1 for client.0 is 26d8230c8f919f5e2d8da795fb3fbca12f8ebd85 2022-04-03T19:31:21.365 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "26d8230c8f919f5e2d8da795fb3fbca12f8ebd85"... 2022-04-03T19:31:21.365 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2022-04-03T19:31:21.382 INFO:teuthology.orchestra.run.smithi077.stdout:4.18.0-348.20.1.el8_5.x86_64 2022-04-03T19:31:21.383 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-348.20.1.el8_5.x86_64 vs 26d8230c8f919f5e2d8da795fb3fbca12f8ebd85 2022-04-03T19:31:21.383 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2022-04-03T19:31:21.384 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum upgrade -y linux-firmware 2022-04-03T19:31:23.056 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2022-04-03T19:31:23.058 INFO:teuthology.orchestra.run.smithi077.stdout:Unable to read consumer identity 2022-04-03T19:31:23.176 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:31:23.176 INFO:teuthology.orchestra.run.smithi077.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2022-04-03T19:31:23.177 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:31:27.102 INFO:teuthology.orchestra.run.smithi077.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2022-04-03T19:31:27.856 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 16 MB/s | 11 MB 00:00 2022-04-03T19:31:30.174 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2022-04-03T19:31:30.975 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2022-04-03T19:31:30.976 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2022-04-03T19:31:30.976 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2022-04-03T19:31:31.087 INFO:teuthology.task.kernel:Downloading kernel 26d8230c8f919f5e2d8da795fb3fbca12f8ebd85 on client.0... 2022-04-03T19:31:31.087 DEBUG:teuthology.orchestra.run.smithi077:> sudo lsb_release -is 2022-04-03T19:31:31.124 INFO:teuthology.orchestra.run.smithi077.stdout:RedHatEnterprise 2022-04-03T19:31:31.125 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-04-03T19:31:31.126 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=26d8230c8f919f5e2d8da795fb3fbca12f8ebd85 2022-04-03T19:31:31.291 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/26d8230c8f919f5e2d8da795fb3fbca12f8ebd85/centos/8/flavors/default/x86_64/ 2022-04-03T19:31:31.291 DEBUG:teuthology.orchestra.run.smithi077:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-5.17.0_rc6_ceph_g26d8230c8f91-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/26d8230c8f919f5e2d8da795fb3fbca12f8ebd85/centos/8/flavors/default/x86_64/ --input-file=- 2022-04-03T19:31:31.294 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi077.front.sepia.ceph.com to complete... 2022-04-03T19:31:48.350 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03 19:31:48 URL:https://3.chacra.ceph.com/r/kernel/testing/26d8230c8f919f5e2d8da795fb3fbca12f8ebd85/centos/8/flavors/default/x86_64/kernel-5.17.0_rc6_ceph_g26d8230c8f91-1.x86_64.rpm [510009892/510009892] -> "/tmp/kernel.x86_64.rpm" [1] 2022-04-03T19:31:48.351 INFO:teuthology.orchestra.run.smithi077.stderr:FINISHED --2022-04-03 19:31:48-- 2022-04-03T19:31:48.351 INFO:teuthology.orchestra.run.smithi077.stderr:Total wall clock time: 17s 2022-04-03T19:31:48.351 INFO:teuthology.orchestra.run.smithi077.stderr:Downloaded: 1 files, 486M in 17s (28.9 MB/s) 2022-04-03T19:31:48.352 INFO:teuthology.task.kernel:Installing kernel 26d8230c8f919f5e2d8da795fb3fbca12f8ebd85 on client.0... 2022-04-03T19:31:48.353 DEBUG:teuthology.orchestra.run.smithi077:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2022-04-03T19:31:50.534 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying... ######################################## 2022-04-03T19:31:50.925 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing... ######################################## 2022-04-03T19:31:51.045 INFO:teuthology.orchestra.run.smithi077.stdout:Updating / installing... 2022-04-03T19:32:29.070 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-5.17.0_rc6_ceph_g26d8230c8f91-1######################################## 2022-04-03T19:32:52.567 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi077.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2022-04-03T19:32:52.567 DEBUG:teuthology.orchestra.run.smithi077:> rpm -qlp /tmp/kernel.x86_64.rpm 2022-04-03T19:32:52.634 INFO:teuthology.orchestra.run.smithi077.stdout:/boot/System.map-5.17.0-rc6-ceph-g26d8230c8f91 2022-04-03T19:32:52.635 INFO:teuthology.orchestra.run.smithi077.stdout:/boot/config-5.17.0-rc6-ceph-g26d8230c8f91 2022-04-03T19:32:52.635 INFO:teuthology.orchestra.run.smithi077.stdout:/boot/vmlinux-5.17.0-rc6-ceph-g26d8230c8f91.bz2 2022-04-03T19:32:52.635 INFO:teuthology.orchestra.run.smithi077.stdout:/boot/vmlinuz-5.17.0-rc6-ceph-g26d8230c8f91 2022-04-03T19:32:52.636 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91 2022-04-03T19:32:52.636 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel 2022-04-03T19:32:52.636 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch 2022-04-03T19:32:52.636 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86 2022-04-03T19:32:52.637 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto 2022-04-03T19:32:52.637 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto/blowfish-x86_64.ko 2022-04-03T19:32:52.637 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2022-04-03T19:32:52.637 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2022-04-03T19:32:52.638 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto/camellia-x86_64.ko 2022-04-03T19:32:52.638 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2022-04-03T19:32:52.638 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2022-04-03T19:32:52.638 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto/crc32-pclmul.ko 2022-04-03T19:32:52.638 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto/crc32c-intel.ko 2022-04-03T19:32:52.639 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2022-04-03T19:32:52.639 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2022-04-03T19:32:52.639 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2022-04-03T19:32:52.639 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2022-04-03T19:32:52.640 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto/serpent-avx2.ko 2022-04-03T19:32:52.640 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2022-04-03T19:32:52.640 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto/sha1-ssse3.ko 2022-04-03T19:32:52.640 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto/sha256-ssse3.ko 2022-04-03T19:32:52.641 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto/sha512-ssse3.ko 2022-04-03T19:32:52.641 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2022-04-03T19:32:52.641 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2022-04-03T19:32:52.641 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/crypto/twofish-x86_64.ko 2022-04-03T19:32:52.642 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/kernel 2022-04-03T19:32:52.642 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/kernel/cpu 2022-04-03T19:32:52.642 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/kernel/cpu/mce 2022-04-03T19:32:52.642 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2022-04-03T19:32:52.643 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/kvm 2022-04-03T19:32:52.643 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/kvm/kvm-amd.ko 2022-04-03T19:32:52.643 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/kvm/kvm-intel.ko 2022-04-03T19:32:52.643 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/arch/x86/kvm/kvm.ko 2022-04-03T19:32:52.644 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto 2022-04-03T19:32:52.644 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/ansi_cprng.ko 2022-04-03T19:32:52.644 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/anubis.ko 2022-04-03T19:32:52.644 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/arc4.ko 2022-04-03T19:32:52.645 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/async_tx 2022-04-03T19:32:52.645 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/async_tx/async_memcpy.ko 2022-04-03T19:32:52.645 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/async_tx/async_pq.ko 2022-04-03T19:32:52.645 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/async_tx/async_raid6_recov.ko 2022-04-03T19:32:52.645 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/async_tx/async_tx.ko 2022-04-03T19:32:52.646 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/async_tx/async_xor.ko 2022-04-03T19:32:52.646 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/async_tx/raid6test.ko 2022-04-03T19:32:52.646 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/authenc.ko 2022-04-03T19:32:52.647 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/authencesn.ko 2022-04-03T19:32:52.647 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/blake2b_generic.ko 2022-04-03T19:32:52.647 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/blowfish_common.ko 2022-04-03T19:32:52.647 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/blowfish_generic.ko 2022-04-03T19:32:52.647 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/camellia_generic.ko 2022-04-03T19:32:52.648 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/cast5_generic.ko 2022-04-03T19:32:52.648 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/cast6_generic.ko 2022-04-03T19:32:52.648 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/cast_common.ko 2022-04-03T19:32:52.648 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/ccm.ko 2022-04-03T19:32:52.649 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/cmac.ko 2022-04-03T19:32:52.649 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/crc32_generic.ko 2022-04-03T19:32:52.649 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/crypto_engine.ko 2022-04-03T19:32:52.649 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/crypto_user.ko 2022-04-03T19:32:52.650 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/cts.ko 2022-04-03T19:32:52.650 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/des_generic.ko 2022-04-03T19:32:52.650 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/dh_generic.ko 2022-04-03T19:32:52.650 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/echainiv.ko 2022-04-03T19:32:52.651 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/essiv.ko 2022-04-03T19:32:52.651 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/fcrypt.ko 2022-04-03T19:32:52.651 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/khazad.ko 2022-04-03T19:32:52.651 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/lz4.ko 2022-04-03T19:32:52.652 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/lz4hc.ko 2022-04-03T19:32:52.652 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/md4.ko 2022-04-03T19:32:52.652 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/michael_mic.ko 2022-04-03T19:32:52.652 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/pcbc.ko 2022-04-03T19:32:52.652 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/pcrypt.ko 2022-04-03T19:32:52.653 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/rmd160.ko 2022-04-03T19:32:52.653 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/seed.ko 2022-04-03T19:32:52.653 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/serpent_generic.ko 2022-04-03T19:32:52.653 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/tcrypt.ko 2022-04-03T19:32:52.654 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/tea.ko 2022-04-03T19:32:52.654 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/twofish_common.ko 2022-04-03T19:32:52.654 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/twofish_generic.ko 2022-04-03T19:32:52.654 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/vmac.ko 2022-04-03T19:32:52.654 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/wp512.ko 2022-04-03T19:32:52.655 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/xcbc.ko 2022-04-03T19:32:52.655 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/xor.ko 2022-04-03T19:32:52.655 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/crypto/xxhash_generic.ko 2022-04-03T19:32:52.655 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers 2022-04-03T19:32:52.656 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/acpi 2022-04-03T19:32:52.656 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/acpi/acpi_ipmi.ko 2022-04-03T19:32:52.656 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/acpi/acpi_pad.ko 2022-04-03T19:32:52.656 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/acpi/custom_method.ko 2022-04-03T19:32:52.657 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/acpi/ec_sys.ko 2022-04-03T19:32:52.657 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/acpi/platform_profile.ko 2022-04-03T19:32:52.657 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/acpi/sbs.ko 2022-04-03T19:32:52.657 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/acpi/sbshc.ko 2022-04-03T19:32:52.658 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/acpi/video.ko 2022-04-03T19:32:52.658 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata 2022-04-03T19:32:52.658 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/acard-ahci.ko 2022-04-03T19:32:52.658 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/ahci_platform.ko 2022-04-03T19:32:52.658 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/ata_generic.ko 2022-04-03T19:32:52.659 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/libahci_platform.ko 2022-04-03T19:32:52.659 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_acpi.ko 2022-04-03T19:32:52.659 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_ali.ko 2022-04-03T19:32:52.659 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_amd.ko 2022-04-03T19:32:52.660 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_artop.ko 2022-04-03T19:32:52.660 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_atiixp.ko 2022-04-03T19:32:52.660 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_atp867x.ko 2022-04-03T19:32:52.660 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_cmd640.ko 2022-04-03T19:32:52.661 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_cmd64x.ko 2022-04-03T19:32:52.661 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_cypress.ko 2022-04-03T19:32:52.661 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_efar.ko 2022-04-03T19:32:52.661 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_hpt366.ko 2022-04-03T19:32:52.662 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_hpt37x.ko 2022-04-03T19:32:52.662 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_hpt3x2n.ko 2022-04-03T19:32:52.662 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_hpt3x3.ko 2022-04-03T19:32:52.662 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_it8213.ko 2022-04-03T19:32:52.662 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_it821x.ko 2022-04-03T19:32:52.663 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_jmicron.ko 2022-04-03T19:32:52.663 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_marvell.ko 2022-04-03T19:32:52.663 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_mpiix.ko 2022-04-03T19:32:52.663 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_netcell.ko 2022-04-03T19:32:52.664 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_ninja32.ko 2022-04-03T19:32:52.664 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_ns87410.ko 2022-04-03T19:32:52.664 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_ns87415.ko 2022-04-03T19:32:52.664 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_oldpiix.ko 2022-04-03T19:32:52.665 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_opti.ko 2022-04-03T19:32:52.665 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_optidma.ko 2022-04-03T19:32:52.665 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_pcmcia.ko 2022-04-03T19:32:52.665 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_pdc2027x.ko 2022-04-03T19:32:52.665 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_pdc202xx_old.ko 2022-04-03T19:32:52.666 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_piccolo.ko 2022-04-03T19:32:52.666 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_rdc.ko 2022-04-03T19:32:52.666 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_sch.ko 2022-04-03T19:32:52.666 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_serverworks.ko 2022-04-03T19:32:52.667 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_sil680.ko 2022-04-03T19:32:52.667 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_sis.ko 2022-04-03T19:32:52.667 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_sl82c105.ko 2022-04-03T19:32:52.667 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_triflex.ko 2022-04-03T19:32:52.668 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pata_via.ko 2022-04-03T19:32:52.668 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/pdc_adma.ko 2022-04-03T19:32:52.668 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/sata_inic162x.ko 2022-04-03T19:32:52.668 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/sata_mv.ko 2022-04-03T19:32:52.669 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/sata_nv.ko 2022-04-03T19:32:52.669 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/sata_promise.ko 2022-04-03T19:32:52.669 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/sata_qstor.ko 2022-04-03T19:32:52.669 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/sata_sil.ko 2022-04-03T19:32:52.670 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/sata_sil24.ko 2022-04-03T19:32:52.670 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/sata_sis.ko 2022-04-03T19:32:52.670 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/sata_svw.ko 2022-04-03T19:32:52.670 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/sata_sx4.ko 2022-04-03T19:32:52.670 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/sata_uli.ko 2022-04-03T19:32:52.671 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/sata_via.ko 2022-04-03T19:32:52.671 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ata/sata_vsc.ko 2022-04-03T19:32:52.671 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/auxdisplay 2022-04-03T19:32:52.672 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/auxdisplay/ks0108.ko 2022-04-03T19:32:52.672 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/base 2022-04-03T19:32:52.672 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/base/regmap 2022-04-03T19:32:52.672 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/base/regmap/regmap-i2c.ko 2022-04-03T19:32:52.672 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/bcma 2022-04-03T19:32:52.673 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/bcma/bcma.ko 2022-04-03T19:32:52.673 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block 2022-04-03T19:32:52.673 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/aoe 2022-04-03T19:32:52.673 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/aoe/aoe.ko 2022-04-03T19:32:52.674 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/brd.ko 2022-04-03T19:32:52.674 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/drbd 2022-04-03T19:32:52.674 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/drbd/drbd.ko 2022-04-03T19:32:52.674 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/floppy.ko 2022-04-03T19:32:52.675 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/loop.ko 2022-04-03T19:32:52.675 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/mtip32xx 2022-04-03T19:32:52.675 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/mtip32xx/mtip32xx.ko 2022-04-03T19:32:52.675 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/nbd.ko 2022-04-03T19:32:52.676 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/null_blk 2022-04-03T19:32:52.676 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/null_blk/null_blk.ko 2022-04-03T19:32:52.676 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/pktcdvd.ko 2022-04-03T19:32:52.677 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/rbd.ko 2022-04-03T19:32:52.677 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/sx8.ko 2022-04-03T19:32:52.677 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/virtio_blk.ko 2022-04-03T19:32:52.677 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/xen-blkback 2022-04-03T19:32:52.678 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/xen-blkback/xen-blkback.ko 2022-04-03T19:32:52.678 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/xen-blkfront.ko 2022-04-03T19:32:52.678 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/zram 2022-04-03T19:32:52.678 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/block/zram/zram.ko 2022-04-03T19:32:52.679 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char 2022-04-03T19:32:52.679 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/hangcheck-timer.ko 2022-04-03T19:32:52.679 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/hw_random 2022-04-03T19:32:52.679 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/hw_random/amd-rng.ko 2022-04-03T19:32:52.680 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/hw_random/intel-rng.ko 2022-04-03T19:32:52.680 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/hw_random/timeriomem-rng.ko 2022-04-03T19:32:52.680 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/hw_random/via-rng.ko 2022-04-03T19:32:52.680 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/hw_random/virtio-rng.ko 2022-04-03T19:32:52.681 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/ipmi 2022-04-03T19:32:52.681 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/ipmi/ipmi_devintf.ko 2022-04-03T19:32:52.681 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2022-04-03T19:32:52.681 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2022-04-03T19:32:52.682 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/ipmi/ipmi_si.ko 2022-04-03T19:32:52.682 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/ipmi/ipmi_ssif.ko 2022-04-03T19:32:52.682 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2022-04-03T19:32:52.682 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/lp.ko 2022-04-03T19:32:52.683 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/mwave 2022-04-03T19:32:52.683 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/mwave/mwave.ko 2022-04-03T19:32:52.683 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/pcmcia 2022-04-03T19:32:52.683 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/pcmcia/cm4000_cs.ko 2022-04-03T19:32:52.684 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/pcmcia/cm4040_cs.ko 2022-04-03T19:32:52.684 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/ppdev.ko 2022-04-03T19:32:52.684 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/tlclk.ko 2022-04-03T19:32:52.684 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/tpm 2022-04-03T19:32:52.684 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/tpm/tpm.ko 2022-04-03T19:32:52.685 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/tpm/tpm_atmel.ko 2022-04-03T19:32:52.685 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/tpm/tpm_infineon.ko 2022-04-03T19:32:52.685 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/tpm/tpm_nsc.ko 2022-04-03T19:32:52.685 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/tpm/tpm_tis.ko 2022-04-03T19:32:52.686 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/tpm/tpm_tis_core.ko 2022-04-03T19:32:52.686 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/uv_mmtimer.ko 2022-04-03T19:32:52.686 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/char/virtio_console.ko 2022-04-03T19:32:52.686 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/cpufreq 2022-04-03T19:32:52.687 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/cpufreq/acpi-cpufreq.ko 2022-04-03T19:32:52.687 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2022-04-03T19:32:52.687 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/cpufreq/p4-clockmod.ko 2022-04-03T19:32:52.687 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/cpufreq/pcc-cpufreq.ko 2022-04-03T19:32:52.688 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/cpufreq/powernow-k8.ko 2022-04-03T19:32:52.688 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/cpufreq/speedstep-lib.ko 2022-04-03T19:32:52.688 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/crypto 2022-04-03T19:32:52.688 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/crypto/ccp 2022-04-03T19:32:52.689 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/crypto/ccp/ccp-crypto.ko 2022-04-03T19:32:52.689 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/crypto/ccp/ccp.ko 2022-04-03T19:32:52.689 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/crypto/padlock-aes.ko 2022-04-03T19:32:52.689 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/crypto/padlock-sha.ko 2022-04-03T19:32:52.690 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/crypto/qat 2022-04-03T19:32:52.690 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/crypto/qat/qat_common 2022-04-03T19:32:52.690 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/crypto/qat/qat_common/intel_qat.ko 2022-04-03T19:32:52.690 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/crypto/qat/qat_dh895xcc 2022-04-03T19:32:52.691 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko 2022-04-03T19:32:52.691 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/crypto/virtio 2022-04-03T19:32:52.691 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/crypto/virtio/virtio_crypto.ko 2022-04-03T19:32:52.691 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/dca 2022-04-03T19:32:52.691 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/dca/dca.ko 2022-04-03T19:32:52.692 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/devfreq 2022-04-03T19:32:52.692 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/devfreq/governor_simpleondemand.ko 2022-04-03T19:32:52.692 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/dma 2022-04-03T19:32:52.692 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/dma/dw 2022-04-03T19:32:52.693 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/dma/dw/dw_dmac.ko 2022-04-03T19:32:52.693 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/dma/ioat 2022-04-03T19:32:52.693 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/dma/ioat/ioatdma.ko 2022-04-03T19:32:52.693 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/edac 2022-04-03T19:32:52.694 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/edac/amd64_edac.ko 2022-04-03T19:32:52.694 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/edac/e752x_edac.ko 2022-04-03T19:32:52.694 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/edac/edac_mce_amd.ko 2022-04-03T19:32:52.694 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/edac/i3000_edac.ko 2022-04-03T19:32:52.695 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/edac/i3200_edac.ko 2022-04-03T19:32:52.695 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/edac/i5000_edac.ko 2022-04-03T19:32:52.695 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/edac/i5100_edac.ko 2022-04-03T19:32:52.695 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/edac/i5400_edac.ko 2022-04-03T19:32:52.695 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/edac/i7300_edac.ko 2022-04-03T19:32:52.696 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/edac/i7core_edac.ko 2022-04-03T19:32:52.696 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/edac/i82975x_edac.ko 2022-04-03T19:32:52.696 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/edac/ie31200_edac.ko 2022-04-03T19:32:52.696 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/edac/sb_edac.ko 2022-04-03T19:32:52.697 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/edac/x38_edac.ko 2022-04-03T19:32:52.697 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/firmware 2022-04-03T19:32:52.697 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/firmware/edd.ko 2022-04-03T19:32:52.697 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/firmware/iscsi_ibft.ko 2022-04-03T19:32:52.698 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/gpio 2022-04-03T19:32:52.698 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/gpio/gpio-ich.ko 2022-04-03T19:32:52.698 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/gpio/gpio-viperboard.ko 2022-04-03T19:32:52.698 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid 2022-04-03T19:32:52.698 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-appleir.ko 2022-04-03T19:32:52.699 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-aureal.ko 2022-04-03T19:32:52.699 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-axff.ko 2022-04-03T19:32:52.699 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-dr.ko 2022-04-03T19:32:52.699 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-elecom.ko 2022-04-03T19:32:52.700 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-elo.ko 2022-04-03T19:32:52.700 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-emsff.ko 2022-04-03T19:32:52.700 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-gaff.ko 2022-04-03T19:32:52.700 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-gt683r.ko 2022-04-03T19:32:52.701 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-gyration.ko 2022-04-03T19:32:52.701 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-holtek-kbd.ko 2022-04-03T19:32:52.701 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-holtek-mouse.ko 2022-04-03T19:32:52.701 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-holtekff.ko 2022-04-03T19:32:52.702 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-hyperv.ko 2022-04-03T19:32:52.702 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-icade.ko 2022-04-03T19:32:52.702 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-keytouch.ko 2022-04-03T19:32:52.702 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-kye.ko 2022-04-03T19:32:52.703 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-lcpower.ko 2022-04-03T19:32:52.703 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-led.ko 2022-04-03T19:32:52.703 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-lenovo.ko 2022-04-03T19:32:52.703 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-logitech-dj.ko 2022-04-03T19:32:52.704 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-logitech-hidpp.ko 2022-04-03T19:32:52.704 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-multitouch.ko 2022-04-03T19:32:52.704 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-ortek.ko 2022-04-03T19:32:52.704 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-penmount.ko 2022-04-03T19:32:52.704 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-petalynx.ko 2022-04-03T19:32:52.705 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-picolcd.ko 2022-04-03T19:32:52.705 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-pl.ko 2022-04-03T19:32:52.705 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-plantronics.ko 2022-04-03T19:32:52.705 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-primax.ko 2022-04-03T19:32:52.706 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-rmi.ko 2022-04-03T19:32:52.706 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-roccat-arvo.ko 2022-04-03T19:32:52.706 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-roccat-common.ko 2022-04-03T19:32:52.706 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-roccat-isku.ko 2022-04-03T19:32:52.707 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-roccat-kone.ko 2022-04-03T19:32:52.707 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-roccat-koneplus.ko 2022-04-03T19:32:52.707 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-roccat-konepure.ko 2022-04-03T19:32:52.707 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-roccat-kovaplus.ko 2022-04-03T19:32:52.708 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-roccat-lua.ko 2022-04-03T19:32:52.708 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-roccat-pyra.ko 2022-04-03T19:32:52.708 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-roccat-ryos.ko 2022-04-03T19:32:52.708 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-roccat-savu.ko 2022-04-03T19:32:52.708 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-roccat.ko 2022-04-03T19:32:52.709 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-saitek.ko 2022-04-03T19:32:52.709 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-samsung.ko 2022-04-03T19:32:52.709 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-sensor-hub.ko 2022-04-03T19:32:52.709 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-sjoy.ko 2022-04-03T19:32:52.710 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-sony.ko 2022-04-03T19:32:52.710 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-speedlink.ko 2022-04-03T19:32:52.710 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-steelseries.ko 2022-04-03T19:32:52.710 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-sunplus.ko 2022-04-03T19:32:52.711 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-thrustmaster.ko 2022-04-03T19:32:52.711 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-tivo.ko 2022-04-03T19:32:52.711 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-tmff.ko 2022-04-03T19:32:52.711 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-topseed.ko 2022-04-03T19:32:52.712 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-twinhan.ko 2022-04-03T19:32:52.712 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-uclogic.ko 2022-04-03T19:32:52.712 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-waltop.ko 2022-04-03T19:32:52.712 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-wiimote.ko 2022-04-03T19:32:52.713 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-xinmo.ko 2022-04-03T19:32:52.713 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-zpff.ko 2022-04-03T19:32:52.713 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/hid-zydacron.ko 2022-04-03T19:32:52.713 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/uhid.ko 2022-04-03T19:32:52.713 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hid/wacom.ko 2022-04-03T19:32:52.714 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hv 2022-04-03T19:32:52.714 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hv/hv_balloon.ko 2022-04-03T19:32:52.714 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hv/hv_utils.ko 2022-04-03T19:32:52.714 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hv/hv_vmbus.ko 2022-04-03T19:32:52.715 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon 2022-04-03T19:32:52.715 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/abituguru.ko 2022-04-03T19:32:52.715 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/abituguru3.ko 2022-04-03T19:32:52.715 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/acpi_power_meter.ko 2022-04-03T19:32:52.716 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/ad7414.ko 2022-04-03T19:32:52.716 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/ad7418.ko 2022-04-03T19:32:52.716 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/adc128d818.ko 2022-04-03T19:32:52.716 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/adm1021.ko 2022-04-03T19:32:52.716 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/adm1025.ko 2022-04-03T19:32:52.717 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/adm1026.ko 2022-04-03T19:32:52.717 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/adm1029.ko 2022-04-03T19:32:52.717 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/adm1031.ko 2022-04-03T19:32:52.717 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/adm9240.ko 2022-04-03T19:32:52.718 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/ads7828.ko 2022-04-03T19:32:52.718 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/adt7410.ko 2022-04-03T19:32:52.718 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/adt7411.ko 2022-04-03T19:32:52.718 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/adt7462.ko 2022-04-03T19:32:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/adt7470.ko 2022-04-03T19:32:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/adt7475.ko 2022-04-03T19:32:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/adt7x10.ko 2022-04-03T19:32:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/amc6821.ko 2022-04-03T19:32:52.719 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/applesmc.ko 2022-04-03T19:32:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/asb100.ko 2022-04-03T19:32:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/asc7621.ko 2022-04-03T19:32:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/asus_atk0110.ko 2022-04-03T19:32:52.720 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/atxp1.ko 2022-04-03T19:32:52.721 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/coretemp.ko 2022-04-03T19:32:52.721 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/dell-smm-hwmon.ko 2022-04-03T19:32:52.721 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/dme1737.ko 2022-04-03T19:32:52.721 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/ds1621.ko 2022-04-03T19:32:52.721 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/ds620.ko 2022-04-03T19:32:52.722 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/emc1403.ko 2022-04-03T19:32:52.722 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/emc6w201.ko 2022-04-03T19:32:52.722 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/f71805f.ko 2022-04-03T19:32:52.722 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/f71882fg.ko 2022-04-03T19:32:52.723 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/f75375s.ko 2022-04-03T19:32:52.723 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/fam15h_power.ko 2022-04-03T19:32:52.723 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/fschmd.ko 2022-04-03T19:32:52.723 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/g760a.ko 2022-04-03T19:32:52.724 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/g762.ko 2022-04-03T19:32:52.724 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/gl518sm.ko 2022-04-03T19:32:52.724 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/gl520sm.ko 2022-04-03T19:32:52.724 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/hwmon-vid.ko 2022-04-03T19:32:52.725 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/i5500_temp.ko 2022-04-03T19:32:52.725 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/i5k_amb.ko 2022-04-03T19:32:52.725 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/ibmaem.ko 2022-04-03T19:32:52.725 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/ibmpex.ko 2022-04-03T19:32:52.725 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/ina209.ko 2022-04-03T19:32:52.726 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/ina2xx.ko 2022-04-03T19:32:52.726 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/it87.ko 2022-04-03T19:32:52.726 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/k10temp.ko 2022-04-03T19:32:52.726 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/k8temp.ko 2022-04-03T19:32:52.727 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/lineage-pem.ko 2022-04-03T19:32:52.727 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/lm63.ko 2022-04-03T19:32:52.727 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/lm73.ko 2022-04-03T19:32:52.727 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/lm75.ko 2022-04-03T19:32:52.728 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/lm77.ko 2022-04-03T19:32:52.728 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/lm78.ko 2022-04-03T19:32:52.728 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/lm80.ko 2022-04-03T19:32:52.728 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/lm83.ko 2022-04-03T19:32:52.729 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/lm85.ko 2022-04-03T19:32:52.729 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/lm87.ko 2022-04-03T19:32:52.729 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/lm90.ko 2022-04-03T19:32:52.729 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/lm92.ko 2022-04-03T19:32:52.730 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/lm93.ko 2022-04-03T19:32:52.730 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/lm95234.ko 2022-04-03T19:32:52.730 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/lm95241.ko 2022-04-03T19:32:52.730 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/lm95245.ko 2022-04-03T19:32:52.730 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/ltc2945.ko 2022-04-03T19:32:52.731 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/ltc4151.ko 2022-04-03T19:32:52.731 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/ltc4215.ko 2022-04-03T19:32:52.731 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/ltc4222.ko 2022-04-03T19:32:52.731 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/ltc4245.ko 2022-04-03T19:32:52.731 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/ltc4260.ko 2022-04-03T19:32:52.732 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/ltc4261.ko 2022-04-03T19:32:52.732 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/max16065.ko 2022-04-03T19:32:52.732 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/max1619.ko 2022-04-03T19:32:52.732 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/max1668.ko 2022-04-03T19:32:52.733 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/max197.ko 2022-04-03T19:32:52.733 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/max6639.ko 2022-04-03T19:32:52.733 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/max6642.ko 2022-04-03T19:32:52.733 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/max6650.ko 2022-04-03T19:32:52.734 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/max6697.ko 2022-04-03T19:32:52.734 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/mcp3021.ko 2022-04-03T19:32:52.734 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/nct6683.ko 2022-04-03T19:32:52.734 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/nct6775.ko 2022-04-03T19:32:52.735 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/nct7802.ko 2022-04-03T19:32:52.735 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/pc87360.ko 2022-04-03T19:32:52.735 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/pc87427.ko 2022-04-03T19:32:52.735 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/pcf8591.ko 2022-04-03T19:32:52.736 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/pmbus 2022-04-03T19:32:52.736 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/pmbus/adm1275.ko 2022-04-03T19:32:52.736 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/pmbus/lm25066.ko 2022-04-03T19:32:52.736 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/pmbus/ltc2978.ko 2022-04-03T19:32:52.736 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/pmbus/max16064.ko 2022-04-03T19:32:52.737 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/pmbus/max34440.ko 2022-04-03T19:32:52.737 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/pmbus/max8688.ko 2022-04-03T19:32:52.737 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/pmbus/pmbus.ko 2022-04-03T19:32:52.737 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2022-04-03T19:32:52.738 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/pmbus/tps40422.ko 2022-04-03T19:32:52.738 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/pmbus/ucd9000.ko 2022-04-03T19:32:52.738 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/pmbus/ucd9200.ko 2022-04-03T19:32:52.738 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/pmbus/zl6100.ko 2022-04-03T19:32:52.739 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/powr1220.ko 2022-04-03T19:32:52.739 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/sch5627.ko 2022-04-03T19:32:52.739 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/sch5636.ko 2022-04-03T19:32:52.739 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/sch56xx-common.ko 2022-04-03T19:32:52.739 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/sht15.ko 2022-04-03T19:32:52.740 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/sht21.ko 2022-04-03T19:32:52.740 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/shtc1.ko 2022-04-03T19:32:52.740 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/sis5595.ko 2022-04-03T19:32:52.740 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/smsc47b397.ko 2022-04-03T19:32:52.741 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/smsc47m1.ko 2022-04-03T19:32:52.741 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/smsc47m192.ko 2022-04-03T19:32:52.741 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/thmc50.ko 2022-04-03T19:32:52.741 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/tmp102.ko 2022-04-03T19:32:52.742 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/tmp103.ko 2022-04-03T19:32:52.742 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/tmp401.ko 2022-04-03T19:32:52.742 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/tmp421.ko 2022-04-03T19:32:52.742 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/via-cputemp.ko 2022-04-03T19:32:52.742 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/via686a.ko 2022-04-03T19:32:52.743 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/vt1211.ko 2022-04-03T19:32:52.743 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/vt8231.ko 2022-04-03T19:32:52.743 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/w83627ehf.ko 2022-04-03T19:32:52.743 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/w83627hf.ko 2022-04-03T19:32:52.744 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/w83781d.ko 2022-04-03T19:32:52.744 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/w83791d.ko 2022-04-03T19:32:52.744 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/w83792d.ko 2022-04-03T19:32:52.744 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/w83793.ko 2022-04-03T19:32:52.745 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/w83795.ko 2022-04-03T19:32:52.745 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/w83l785ts.ko 2022-04-03T19:32:52.745 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/hwmon/w83l786ng.ko 2022-04-03T19:32:52.745 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c 2022-04-03T19:32:52.745 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/algos 2022-04-03T19:32:52.746 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2022-04-03T19:32:52.746 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2022-04-03T19:32:52.746 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses 2022-04-03T19:32:52.746 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2022-04-03T19:32:52.747 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-amd756.ko 2022-04-03T19:32:52.747 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-amd8111.ko 2022-04-03T19:32:52.747 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-designware-core.ko 2022-04-03T19:32:52.747 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2022-04-03T19:32:52.748 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2022-04-03T19:32:52.748 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2022-04-03T19:32:52.748 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-i801.ko 2022-04-03T19:32:52.748 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-isch.ko 2022-04-03T19:32:52.749 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-ismt.ko 2022-04-03T19:32:52.749 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2022-04-03T19:32:52.749 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-nforce2.ko 2022-04-03T19:32:52.749 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-parport.ko 2022-04-03T19:32:52.749 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2022-04-03T19:32:52.750 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-piix4.ko 2022-04-03T19:32:52.750 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-scmi.ko 2022-04-03T19:32:52.750 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-simtec.ko 2022-04-03T19:32:52.750 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-sis96x.ko 2022-04-03T19:32:52.751 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2022-04-03T19:32:52.751 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-via.ko 2022-04-03T19:32:52.751 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-viapro.ko 2022-04-03T19:32:52.751 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/busses/i2c-viperboard.ko 2022-04-03T19:32:52.752 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/i2c-dev.ko 2022-04-03T19:32:52.752 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/i2c-mux.ko 2022-04-03T19:32:52.752 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/i2c-slave-eeprom.ko 2022-04-03T19:32:52.752 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/i2c-smbus.ko 2022-04-03T19:32:52.752 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/i2c/i2c-stub.ko 2022-04-03T19:32:52.753 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband 2022-04-03T19:32:52.753 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/core 2022-04-03T19:32:52.753 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/core/ib_cm.ko 2022-04-03T19:32:52.753 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/core/ib_core.ko 2022-04-03T19:32:52.754 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/core/ib_umad.ko 2022-04-03T19:32:52.754 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/core/ib_uverbs.ko 2022-04-03T19:32:52.754 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/core/iw_cm.ko 2022-04-03T19:32:52.754 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/core/rdma_cm.ko 2022-04-03T19:32:52.755 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/core/rdma_ucm.ko 2022-04-03T19:32:52.755 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/hw 2022-04-03T19:32:52.755 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/hw/cxgb4 2022-04-03T19:32:52.755 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2022-04-03T19:32:52.756 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/hw/mlx4 2022-04-03T19:32:52.756 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2022-04-03T19:32:52.756 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/hw/mlx5 2022-04-03T19:32:52.756 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2022-04-03T19:32:52.756 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/hw/mthca 2022-04-03T19:32:52.757 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2022-04-03T19:32:52.757 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/ulp 2022-04-03T19:32:52.757 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/ulp/ipoib 2022-04-03T19:32:52.757 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2022-04-03T19:32:52.758 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/ulp/iser 2022-04-03T19:32:52.758 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2022-04-03T19:32:52.758 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/ulp/isert 2022-04-03T19:32:52.758 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2022-04-03T19:32:52.759 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/ulp/srp 2022-04-03T19:32:52.759 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2022-04-03T19:32:52.759 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/ulp/srpt 2022-04-03T19:32:52.759 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2022-04-03T19:32:52.760 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input 2022-04-03T19:32:52.760 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/gameport 2022-04-03T19:32:52.760 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/gameport/emu10k1-gp.ko 2022-04-03T19:32:52.760 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/gameport/fm801-gp.ko 2022-04-03T19:32:52.760 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/gameport/gameport.ko 2022-04-03T19:32:52.761 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/gameport/lightning.ko 2022-04-03T19:32:52.761 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/gameport/ns558.ko 2022-04-03T19:32:52.761 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/joydev.ko 2022-04-03T19:32:52.761 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/keyboard 2022-04-03T19:32:52.762 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/keyboard/gpio_keys.ko 2022-04-03T19:32:52.762 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/mouse 2022-04-03T19:32:52.762 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/mouse/appletouch.ko 2022-04-03T19:32:52.762 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/mouse/bcm5974.ko 2022-04-03T19:32:52.763 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/mouse/cyapatp.ko 2022-04-03T19:32:52.763 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/mouse/elan_i2c.ko 2022-04-03T19:32:52.763 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/mouse/sermouse.ko 2022-04-03T19:32:52.764 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/mouse/synaptics_i2c.ko 2022-04-03T19:32:52.764 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/mouse/synaptics_usb.ko 2022-04-03T19:32:52.764 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/mouse/vsxxxaa.ko 2022-04-03T19:32:52.764 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/rmi4 2022-04-03T19:32:52.765 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/rmi4/rmi_core.ko 2022-04-03T19:32:52.765 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/serio 2022-04-03T19:32:52.765 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/serio/altera_ps2.ko 2022-04-03T19:32:52.765 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/serio/arc_ps2.ko 2022-04-03T19:32:52.766 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/serio/hyperv-keyboard.ko 2022-04-03T19:32:52.766 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/serio/serio_raw.ko 2022-04-03T19:32:52.766 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/input/sparse-keymap.ko 2022-04-03T19:32:52.766 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/iommu 2022-04-03T19:32:52.767 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/iommu/amd 2022-04-03T19:32:52.767 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/iommu/amd/iommu_v2.ko 2022-04-03T19:32:52.767 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/leds 2022-04-03T19:32:52.767 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/leds/leds-blinkm.ko 2022-04-03T19:32:52.768 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/leds/leds-clevo-mail.ko 2022-04-03T19:32:52.768 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/leds/leds-lm3530.ko 2022-04-03T19:32:52.768 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/leds/leds-lp3944.ko 2022-04-03T19:32:52.768 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/leds/leds-ss4200.ko 2022-04-03T19:32:52.768 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/leds/trigger 2022-04-03T19:32:52.769 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/leds/trigger/ledtrig-audio.ko 2022-04-03T19:32:52.769 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2022-04-03T19:32:52.769 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/leds/trigger/ledtrig-camera.ko 2022-04-03T19:32:52.769 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2022-04-03T19:32:52.770 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2022-04-03T19:32:52.770 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2022-04-03T19:32:52.770 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2022-04-03T19:32:52.770 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/leds/trigger/ledtrig-timer.ko 2022-04-03T19:32:52.771 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/leds/trigger/ledtrig-transient.ko 2022-04-03T19:32:52.771 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md 2022-04-03T19:32:52.771 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/bcache 2022-04-03T19:32:52.771 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/bcache/bcache.ko 2022-04-03T19:32:52.772 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/dm-bio-prison.ko 2022-04-03T19:32:52.772 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/dm-cache-smq.ko 2022-04-03T19:32:52.772 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/dm-cache.ko 2022-04-03T19:32:52.773 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/dm-crypt.ko 2022-04-03T19:32:52.773 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/dm-delay.ko 2022-04-03T19:32:52.773 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/dm-flakey.ko 2022-04-03T19:32:52.773 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/dm-log-userspace.ko 2022-04-03T19:32:52.774 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/dm-multipath.ko 2022-04-03T19:32:52.774 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/dm-queue-length.ko 2022-04-03T19:32:52.774 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/dm-raid.ko 2022-04-03T19:32:52.774 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/dm-round-robin.ko 2022-04-03T19:32:52.775 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/dm-service-time.ko 2022-04-03T19:32:52.775 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/dm-switch.ko 2022-04-03T19:32:52.775 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/dm-thin-pool.ko 2022-04-03T19:32:52.775 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/dm-verity.ko 2022-04-03T19:32:52.775 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/faulty.ko 2022-04-03T19:32:52.776 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/linear.ko 2022-04-03T19:32:52.776 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/multipath.ko 2022-04-03T19:32:52.776 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/persistent-data 2022-04-03T19:32:52.776 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2022-04-03T19:32:52.777 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/raid0.ko 2022-04-03T19:32:52.777 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/raid1.ko 2022-04-03T19:32:52.777 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/raid10.ko 2022-04-03T19:32:52.777 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/md/raid456.ko 2022-04-03T19:32:52.778 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/message 2022-04-03T19:32:52.778 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/message/fusion 2022-04-03T19:32:52.778 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/message/fusion/mptbase.ko 2022-04-03T19:32:52.778 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/message/fusion/mptctl.ko 2022-04-03T19:32:52.778 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/message/fusion/mptfc.ko 2022-04-03T19:32:52.779 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/message/fusion/mptlan.ko 2022-04-03T19:32:52.779 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/message/fusion/mptsas.ko 2022-04-03T19:32:52.779 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/message/fusion/mptscsih.ko 2022-04-03T19:32:52.779 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/message/fusion/mptspi.ko 2022-04-03T19:32:52.780 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/mfd 2022-04-03T19:32:52.780 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/mfd/lpc_ich.ko 2022-04-03T19:32:52.780 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/mfd/lpc_sch.ko 2022-04-03T19:32:52.780 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/mfd/mfd-core.ko 2022-04-03T19:32:52.781 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/mfd/sm501.ko 2022-04-03T19:32:52.781 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/mfd/viperboard.ko 2022-04-03T19:32:52.781 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/mfd/vx855.ko 2022-04-03T19:32:52.781 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/mfd/wl1273-core.ko 2022-04-03T19:32:52.782 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc 2022-04-03T19:32:52.782 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/altera-stapl 2022-04-03T19:32:52.782 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2022-04-03T19:32:52.782 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/apds9802als.ko 2022-04-03T19:32:52.783 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/apds990x.ko 2022-04-03T19:32:52.783 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/bh1770glc.ko 2022-04-03T19:32:52.783 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/cb710 2022-04-03T19:32:52.783 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/cb710/cb710.ko 2022-04-03T19:32:52.783 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/eeprom 2022-04-03T19:32:52.784 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/eeprom/at24.ko 2022-04-03T19:32:52.784 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/eeprom/eeprom.ko 2022-04-03T19:32:52.784 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2022-04-03T19:32:52.784 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/eeprom/max6875.ko 2022-04-03T19:32:52.785 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/enclosure.ko 2022-04-03T19:32:52.785 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/hpilo.ko 2022-04-03T19:32:52.785 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/isl29003.ko 2022-04-03T19:32:52.785 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/isl29020.ko 2022-04-03T19:32:52.786 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/lis3lv02d 2022-04-03T19:32:52.786 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2022-04-03T19:32:52.786 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2022-04-03T19:32:52.786 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/mei 2022-04-03T19:32:52.787 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/mei/mei-me.ko 2022-04-03T19:32:52.787 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/mei/mei-txe.ko 2022-04-03T19:32:52.787 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/mei/mei.ko 2022-04-03T19:32:52.787 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/sgi-gru 2022-04-03T19:32:52.787 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/sgi-gru/gru.ko 2022-04-03T19:32:52.788 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/sgi-xp 2022-04-03T19:32:52.788 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/sgi-xp/xp.ko 2022-04-03T19:32:52.788 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/sgi-xp/xpc.ko 2022-04-03T19:32:52.788 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/sgi-xp/xpnet.ko 2022-04-03T19:32:52.789 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/tifm_7xx1.ko 2022-04-03T19:32:52.789 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/tifm_core.ko 2022-04-03T19:32:52.789 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/tsl2550.ko 2022-04-03T19:32:52.789 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/vmw_balloon.ko 2022-04-03T19:32:52.790 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/vmw_vmci 2022-04-03T19:32:52.790 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2022-04-03T19:32:52.790 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net 2022-04-03T19:32:52.790 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/bonding 2022-04-03T19:32:52.791 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/bonding/bonding.ko 2022-04-03T19:32:52.791 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/dsa 2022-04-03T19:32:52.791 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/dsa/b53 2022-04-03T19:32:52.791 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/dsa/b53/b53_common.ko 2022-04-03T19:32:52.792 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/dsa/bcm-sf2.ko 2022-04-03T19:32:52.792 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/dsa/mv88e6060.ko 2022-04-03T19:32:52.792 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/dsa/mv88e6xxx 2022-04-03T19:32:52.792 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2022-04-03T19:32:52.792 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/dummy.ko 2022-04-03T19:32:52.793 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/eql.ko 2022-04-03T19:32:52.793 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet 2022-04-03T19:32:52.793 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/3com 2022-04-03T19:32:52.793 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2022-04-03T19:32:52.794 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2022-04-03T19:32:52.794 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/3com/3c59x.ko 2022-04-03T19:32:52.794 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/3com/typhoon.ko 2022-04-03T19:32:52.794 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/8390 2022-04-03T19:32:52.795 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/8390/8390.ko 2022-04-03T19:32:52.795 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2022-04-03T19:32:52.795 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2022-04-03T19:32:52.795 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2022-04-03T19:32:52.795 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/adaptec 2022-04-03T19:32:52.796 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/adaptec/starfire.ko 2022-04-03T19:32:52.796 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/agere 2022-04-03T19:32:52.796 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/agere/et131x.ko 2022-04-03T19:32:52.796 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/alteon 2022-04-03T19:32:52.797 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/alteon/acenic.ko 2022-04-03T19:32:52.797 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/altera 2022-04-03T19:32:52.797 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/altera/altera_tse.ko 2022-04-03T19:32:52.797 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/amd 2022-04-03T19:32:52.798 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/amd/amd8111e.ko 2022-04-03T19:32:52.798 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2022-04-03T19:32:52.798 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/amd/pcnet32.ko 2022-04-03T19:32:52.798 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/atheros 2022-04-03T19:32:52.799 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/atheros/alx 2022-04-03T19:32:52.799 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2022-04-03T19:32:52.799 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/atheros/atl1c 2022-04-03T19:32:52.799 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2022-04-03T19:32:52.799 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/atheros/atl1e 2022-04-03T19:32:52.800 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2022-04-03T19:32:52.800 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/atheros/atlx 2022-04-03T19:32:52.800 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2022-04-03T19:32:52.800 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2022-04-03T19:32:52.801 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/broadcom 2022-04-03T19:32:52.801 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/broadcom/b44.ko 2022-04-03T19:32:52.801 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2022-04-03T19:32:52.801 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/broadcom/bnx2x 2022-04-03T19:32:52.802 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2022-04-03T19:32:52.802 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/broadcom/cnic.ko 2022-04-03T19:32:52.802 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/broadcom/genet 2022-04-03T19:32:52.802 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2022-04-03T19:32:52.802 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/broadcom/tg3.ko 2022-04-03T19:32:52.803 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/brocade 2022-04-03T19:32:52.803 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/brocade/bna 2022-04-03T19:32:52.803 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2022-04-03T19:32:52.803 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/chelsio 2022-04-03T19:32:52.804 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/chelsio/cxgb 2022-04-03T19:32:52.804 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2022-04-03T19:32:52.804 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/chelsio/cxgb3 2022-04-03T19:32:52.804 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2022-04-03T19:32:52.805 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/chelsio/cxgb4 2022-04-03T19:32:52.805 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2022-04-03T19:32:52.805 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2022-04-03T19:32:52.805 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2022-04-03T19:32:52.806 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/chelsio/libcxgb 2022-04-03T19:32:52.806 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2022-04-03T19:32:52.806 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/cisco 2022-04-03T19:32:52.806 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/cisco/enic 2022-04-03T19:32:52.807 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2022-04-03T19:32:52.807 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/dec 2022-04-03T19:32:52.807 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/dec/tulip 2022-04-03T19:32:52.807 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2022-04-03T19:32:52.807 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/dec/tulip/de4x5.ko 2022-04-03T19:32:52.808 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2022-04-03T19:32:52.808 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2022-04-03T19:32:52.808 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2022-04-03T19:32:52.808 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2022-04-03T19:32:52.809 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2022-04-03T19:32:52.809 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/dlink 2022-04-03T19:32:52.809 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/dlink/dl2k.ko 2022-04-03T19:32:52.809 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/dlink/sundance.ko 2022-04-03T19:32:52.810 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/dnet.ko 2022-04-03T19:32:52.810 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/emulex 2022-04-03T19:32:52.810 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/emulex/benet 2022-04-03T19:32:52.810 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2022-04-03T19:32:52.810 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/ethoc.ko 2022-04-03T19:32:52.811 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/fealnx.ko 2022-04-03T19:32:52.811 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel 2022-04-03T19:32:52.811 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/e100.ko 2022-04-03T19:32:52.811 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/e1000 2022-04-03T19:32:52.812 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2022-04-03T19:32:52.812 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/e1000e 2022-04-03T19:32:52.812 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2022-04-03T19:32:52.812 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/fm10k 2022-04-03T19:32:52.813 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2022-04-03T19:32:52.813 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/i40e 2022-04-03T19:32:52.813 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2022-04-03T19:32:52.813 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/iavf 2022-04-03T19:32:52.814 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2022-04-03T19:32:52.814 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/igb 2022-04-03T19:32:52.814 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/igb/igb.ko 2022-04-03T19:32:52.814 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/igbvf 2022-04-03T19:32:52.814 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2022-04-03T19:32:52.815 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/ixgb 2022-04-03T19:32:52.815 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/ixgb/ixgb.ko 2022-04-03T19:32:52.815 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/ixgbe 2022-04-03T19:32:52.815 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2022-04-03T19:32:52.816 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/ixgbevf 2022-04-03T19:32:52.816 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2022-04-03T19:32:52.816 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/jme.ko 2022-04-03T19:32:52.816 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/marvell 2022-04-03T19:32:52.817 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2022-04-03T19:32:52.817 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/marvell/skge.ko 2022-04-03T19:32:52.817 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/marvell/sky2.ko 2022-04-03T19:32:52.817 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/mellanox 2022-04-03T19:32:52.817 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/mellanox/mlx4 2022-04-03T19:32:52.818 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2022-04-03T19:32:52.818 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2022-04-03T19:32:52.818 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/mellanox/mlx5 2022-04-03T19:32:52.818 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/mellanox/mlx5/core 2022-04-03T19:32:52.819 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2022-04-03T19:32:52.819 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/micrel 2022-04-03T19:32:52.819 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2022-04-03T19:32:52.819 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/myricom 2022-04-03T19:32:52.820 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/myricom/myri10ge 2022-04-03T19:32:52.820 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2022-04-03T19:32:52.820 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/natsemi 2022-04-03T19:32:52.820 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2022-04-03T19:32:52.820 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2022-04-03T19:32:52.821 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/neterion 2022-04-03T19:32:52.821 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/neterion/s2io.ko 2022-04-03T19:32:52.821 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/neterion/vxge 2022-04-03T19:32:52.821 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/neterion/vxge/vxge.ko 2022-04-03T19:32:52.822 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/nvidia 2022-04-03T19:32:52.822 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2022-04-03T19:32:52.822 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/packetengines 2022-04-03T19:32:52.822 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2022-04-03T19:32:52.823 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2022-04-03T19:32:52.823 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/qlogic 2022-04-03T19:32:52.823 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/qlogic/netxen 2022-04-03T19:32:52.823 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2022-04-03T19:32:52.823 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2022-04-03T19:32:52.824 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/qlogic/qlcnic 2022-04-03T19:32:52.824 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2022-04-03T19:32:52.824 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/rdc 2022-04-03T19:32:52.824 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/rdc/r6040.ko 2022-04-03T19:32:52.825 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/realtek 2022-04-03T19:32:52.825 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/realtek/8139cp.ko 2022-04-03T19:32:52.825 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/realtek/8139too.ko 2022-04-03T19:32:52.825 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/realtek/atp.ko 2022-04-03T19:32:52.826 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/realtek/r8169.ko 2022-04-03T19:32:52.826 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/rocker 2022-04-03T19:32:52.826 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/rocker/rocker.ko 2022-04-03T19:32:52.826 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/sfc 2022-04-03T19:32:52.826 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/sfc/sfc.ko 2022-04-03T19:32:52.827 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/silan 2022-04-03T19:32:52.827 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/silan/sc92031.ko 2022-04-03T19:32:52.827 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/sis 2022-04-03T19:32:52.827 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/sis/sis190.ko 2022-04-03T19:32:52.828 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/sis/sis900.ko 2022-04-03T19:32:52.828 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/smsc 2022-04-03T19:32:52.828 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/smsc/epic100.ko 2022-04-03T19:32:52.828 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2022-04-03T19:32:52.829 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2022-04-03T19:32:52.829 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2022-04-03T19:32:52.829 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/stmicro 2022-04-03T19:32:52.829 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/stmicro/stmmac 2022-04-03T19:32:52.829 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2022-04-03T19:32:52.830 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2022-04-03T19:32:52.830 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/sun 2022-04-03T19:32:52.830 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/sun/cassini.ko 2022-04-03T19:32:52.831 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/sun/niu.ko 2022-04-03T19:32:52.831 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/sun/sungem.ko 2022-04-03T19:32:52.831 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/sun/sunhme.ko 2022-04-03T19:32:52.831 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/tehuti 2022-04-03T19:32:52.832 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2022-04-03T19:32:52.832 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/ti 2022-04-03T19:32:52.832 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/ti/tlan.ko 2022-04-03T19:32:52.832 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/via 2022-04-03T19:32:52.832 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/via/via-rhine.ko 2022-04-03T19:32:52.833 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/via/via-velocity.ko 2022-04-03T19:32:52.833 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/wiznet 2022-04-03T19:32:52.833 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/wiznet/w5100.ko 2022-04-03T19:32:52.833 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/wiznet/w5300.ko 2022-04-03T19:32:52.833 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/xircom 2022-04-03T19:32:52.834 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2022-04-03T19:32:52.834 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/geneve.ko 2022-04-03T19:32:52.834 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/hyperv 2022-04-03T19:32:52.834 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/hyperv/hv_netvsc.ko 2022-04-03T19:32:52.834 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ieee802154 2022-04-03T19:32:52.835 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ieee802154/fakelb.ko 2022-04-03T19:32:52.835 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ifb.ko 2022-04-03T19:32:52.835 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ipvlan 2022-04-03T19:32:52.835 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ipvlan/ipvlan.ko 2022-04-03T19:32:52.836 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/macvlan.ko 2022-04-03T19:32:52.836 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/macvtap.ko 2022-04-03T19:32:52.836 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/mdio 2022-04-03T19:32:52.836 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/mdio.ko 2022-04-03T19:32:52.836 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2022-04-03T19:32:52.837 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/mdio/mdio-bitbang.ko 2022-04-03T19:32:52.837 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/mii.ko 2022-04-03T19:32:52.837 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/net_failover.ko 2022-04-03T19:32:52.837 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/netconsole.ko 2022-04-03T19:32:52.837 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/nlmon.ko 2022-04-03T19:32:52.838 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/pcs 2022-04-03T19:32:52.838 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/pcs/pcs_xpcs.ko 2022-04-03T19:32:52.838 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy 2022-04-03T19:32:52.838 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy/amd.ko 2022-04-03T19:32:52.838 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy/bcm-phy-lib.ko 2022-04-03T19:32:52.839 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy/bcm7xxx.ko 2022-04-03T19:32:52.839 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy/bcm87xx.ko 2022-04-03T19:32:52.839 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy/broadcom.ko 2022-04-03T19:32:52.839 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy/cicada.ko 2022-04-03T19:32:52.839 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy/davicom.ko 2022-04-03T19:32:52.840 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy/dp83640.ko 2022-04-03T19:32:52.840 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy/et1011c.ko 2022-04-03T19:32:52.840 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy/icplus.ko 2022-04-03T19:32:52.840 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy/lxt.ko 2022-04-03T19:32:52.841 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy/marvell.ko 2022-04-03T19:32:52.841 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy/micrel.ko 2022-04-03T19:32:52.841 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy/national.ko 2022-04-03T19:32:52.841 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy/phylink.ko 2022-04-03T19:32:52.841 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy/qsemi.ko 2022-04-03T19:32:52.842 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy/realtek.ko 2022-04-03T19:32:52.842 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy/smsc.ko 2022-04-03T19:32:52.842 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy/ste10Xp.ko 2022-04-03T19:32:52.842 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/phy/vitesse.ko 2022-04-03T19:32:52.842 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ppp 2022-04-03T19:32:52.843 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ppp/bsd_comp.ko 2022-04-03T19:32:52.843 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ppp/ppp_async.ko 2022-04-03T19:32:52.843 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ppp/ppp_deflate.ko 2022-04-03T19:32:52.843 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ppp/ppp_generic.ko 2022-04-03T19:32:52.843 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ppp/ppp_mppe.ko 2022-04-03T19:32:52.844 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ppp/ppp_synctty.ko 2022-04-03T19:32:52.844 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ppp/pppoe.ko 2022-04-03T19:32:52.844 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ppp/pppox.ko 2022-04-03T19:32:52.844 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/ppp/pptp.ko 2022-04-03T19:32:52.844 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/slip 2022-04-03T19:32:52.845 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/slip/slhc.ko 2022-04-03T19:32:52.845 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/slip/slip.ko 2022-04-03T19:32:52.845 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/sungem_phy.ko 2022-04-03T19:32:52.845 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/tap.ko 2022-04-03T19:32:52.846 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/team 2022-04-03T19:32:52.846 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/team/team.ko 2022-04-03T19:32:52.846 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/team/team_mode_activebackup.ko 2022-04-03T19:32:52.846 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/team/team_mode_broadcast.ko 2022-04-03T19:32:52.846 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/team/team_mode_loadbalance.ko 2022-04-03T19:32:52.847 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/team/team_mode_random.ko 2022-04-03T19:32:52.847 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/team/team_mode_roundrobin.ko 2022-04-03T19:32:52.847 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/tun.ko 2022-04-03T19:32:52.847 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/veth.ko 2022-04-03T19:32:52.847 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/virtio_net.ko 2022-04-03T19:32:52.848 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/vmxnet3 2022-04-03T19:32:52.848 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/vmxnet3/vmxnet3.ko 2022-04-03T19:32:52.848 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/vxlan.ko 2022-04-03T19:32:52.848 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/xen-netback 2022-04-03T19:32:52.848 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/xen-netback/xen-netback.ko 2022-04-03T19:32:52.849 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/net/xen-netfront.ko 2022-04-03T19:32:52.849 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ntb 2022-04-03T19:32:52.849 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ntb/ntb.ko 2022-04-03T19:32:52.849 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/nvme 2022-04-03T19:32:52.849 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/nvme/host 2022-04-03T19:32:52.850 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/nvme/host/nvme-core.ko 2022-04-03T19:32:52.850 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/nvme/host/nvme.ko 2022-04-03T19:32:52.850 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/parport 2022-04-03T19:32:52.850 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/parport/parport.ko 2022-04-03T19:32:52.851 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/parport/parport_cs.ko 2022-04-03T19:32:52.851 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/parport/parport_pc.ko 2022-04-03T19:32:52.851 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/parport/parport_serial.ko 2022-04-03T19:32:52.851 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/pci 2022-04-03T19:32:52.851 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/pci/hotplug 2022-04-03T19:32:52.852 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2022-04-03T19:32:52.852 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/pci/pcie 2022-04-03T19:32:52.852 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/pci/pcie/aer_inject.ko 2022-04-03T19:32:52.852 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/pci/xen-pcifront.ko 2022-04-03T19:32:52.852 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/pcmcia 2022-04-03T19:32:52.853 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/pcmcia/i82092.ko 2022-04-03T19:32:52.853 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/pcmcia/pd6729.ko 2022-04-03T19:32:52.853 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/pcmcia/yenta_socket.ko 2022-04-03T19:32:52.853 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/pinctrl 2022-04-03T19:32:52.854 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/pinctrl/intel 2022-04-03T19:32:52.854 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2022-04-03T19:32:52.854 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform 2022-04-03T19:32:52.854 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86 2022-04-03T19:32:52.854 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/acer-wmi.ko 2022-04-03T19:32:52.855 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/acerhdf.ko 2022-04-03T19:32:52.855 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/apple-gmux.ko 2022-04-03T19:32:52.855 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/asus-laptop.ko 2022-04-03T19:32:52.855 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/asus-nb-wmi.ko 2022-04-03T19:32:52.855 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/asus-wmi.ko 2022-04-03T19:32:52.856 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/classmate-laptop.ko 2022-04-03T19:32:52.856 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/eeepc-laptop.ko 2022-04-03T19:32:52.856 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/eeepc-wmi.ko 2022-04-03T19:32:52.856 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/fujitsu-laptop.ko 2022-04-03T19:32:52.856 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/fujitsu-tablet.ko 2022-04-03T19:32:52.857 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/hdaps.ko 2022-04-03T19:32:52.857 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/hp-wmi.ko 2022-04-03T19:32:52.857 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/hp_accel.ko 2022-04-03T19:32:52.857 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/intel 2022-04-03T19:32:52.857 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/intel/intel-rst.ko 2022-04-03T19:32:52.858 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/intel_ips.ko 2022-04-03T19:32:52.858 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/msi-wmi.ko 2022-04-03T19:32:52.858 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/mxm-wmi.ko 2022-04-03T19:32:52.858 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/panasonic-laptop.ko 2022-04-03T19:32:52.859 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/samsung-laptop.ko 2022-04-03T19:32:52.859 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/samsung-q10.ko 2022-04-03T19:32:52.859 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/topstar-laptop.ko 2022-04-03T19:32:52.859 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2022-04-03T19:32:52.859 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/toshiba_haps.ko 2022-04-03T19:32:52.859 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/wmi-bmof.ko 2022-04-03T19:32:52.860 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/platform/x86/wmi.ko 2022-04-03T19:32:52.860 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/powercap 2022-04-03T19:32:52.860 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/powercap/intel_rapl_common.ko 2022-04-03T19:32:52.860 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/powercap/intel_rapl_msr.ko 2022-04-03T19:32:52.860 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/pps 2022-04-03T19:32:52.861 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/pps/clients 2022-04-03T19:32:52.861 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/pps/clients/pps-gpio.ko 2022-04-03T19:32:52.861 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/pps/clients/pps-ldisc.ko 2022-04-03T19:32:52.861 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/pps/clients/pps_parport.ko 2022-04-03T19:32:52.861 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/pps/pps_core.ko 2022-04-03T19:32:52.862 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ptp 2022-04-03T19:32:52.862 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ptp/ptp.ko 2022-04-03T19:32:52.862 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ptp/ptp_kvm.ko 2022-04-03T19:32:52.862 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc 2022-04-03T19:32:52.863 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-bq32k.ko 2022-04-03T19:32:52.863 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-bq4802.ko 2022-04-03T19:32:52.863 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-ds1286.ko 2022-04-03T19:32:52.863 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-ds1307.ko 2022-04-03T19:32:52.863 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-ds1374.ko 2022-04-03T19:32:52.864 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-ds1511.ko 2022-04-03T19:32:52.864 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-ds1553.ko 2022-04-03T19:32:52.864 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-ds1672.ko 2022-04-03T19:32:52.864 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-ds1742.ko 2022-04-03T19:32:52.864 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-ds2404.ko 2022-04-03T19:32:52.865 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-ds3232.ko 2022-04-03T19:32:52.865 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-em3027.ko 2022-04-03T19:32:52.865 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-fm3130.ko 2022-04-03T19:32:52.865 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-isl12022.ko 2022-04-03T19:32:52.865 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-isl1208.ko 2022-04-03T19:32:52.866 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-m41t80.ko 2022-04-03T19:32:52.866 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-m48t35.ko 2022-04-03T19:32:52.866 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-m48t59.ko 2022-04-03T19:32:52.866 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-max6900.ko 2022-04-03T19:32:52.866 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-msm6242.ko 2022-04-03T19:32:52.867 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-pcf2127.ko 2022-04-03T19:32:52.867 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-pcf85063.ko 2022-04-03T19:32:52.867 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-pcf8523.ko 2022-04-03T19:32:52.867 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-pcf8563.ko 2022-04-03T19:32:52.867 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-pcf8583.ko 2022-04-03T19:32:52.868 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-rp5c01.ko 2022-04-03T19:32:52.868 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-rs5c372.ko 2022-04-03T19:32:52.868 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-rv3029c2.ko 2022-04-03T19:32:52.868 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-rx8025.ko 2022-04-03T19:32:52.868 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-rx8581.ko 2022-04-03T19:32:52.869 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-stk17ta8.ko 2022-04-03T19:32:52.869 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-v3020.ko 2022-04-03T19:32:52.869 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/rtc/rtc-x1205.ko 2022-04-03T19:32:52.869 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi 2022-04-03T19:32:52.870 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/3w-9xxx.ko 2022-04-03T19:32:52.870 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/3w-sas.ko 2022-04-03T19:32:52.870 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/3w-xxxx.ko 2022-04-03T19:32:52.870 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/BusLogic.ko 2022-04-03T19:32:52.870 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/a100u2w.ko 2022-04-03T19:32:52.871 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/aacraid 2022-04-03T19:32:52.871 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/aacraid/aacraid.ko 2022-04-03T19:32:52.871 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/advansys.ko 2022-04-03T19:32:52.871 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/aic7xxx 2022-04-03T19:32:52.871 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2022-04-03T19:32:52.872 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2022-04-03T19:32:52.872 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/am53c974.ko 2022-04-03T19:32:52.872 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/arcmsr 2022-04-03T19:32:52.872 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/arcmsr/arcmsr.ko 2022-04-03T19:32:52.872 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/atp870u.ko 2022-04-03T19:32:52.873 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/be2iscsi 2022-04-03T19:32:52.873 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2022-04-03T19:32:52.873 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/bfa 2022-04-03T19:32:52.873 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/bfa/bfa.ko 2022-04-03T19:32:52.873 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/bnx2fc 2022-04-03T19:32:52.874 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2022-04-03T19:32:52.874 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/bnx2i 2022-04-03T19:32:52.874 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/bnx2i/bnx2i.ko 2022-04-03T19:32:52.874 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/ch.ko 2022-04-03T19:32:52.875 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/csiostor 2022-04-03T19:32:52.875 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/csiostor/csiostor.ko 2022-04-03T19:32:52.875 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/cxgbi 2022-04-03T19:32:52.875 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/cxgbi/cxgb3i 2022-04-03T19:32:52.875 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2022-04-03T19:32:52.876 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/cxgbi/cxgb4i 2022-04-03T19:32:52.876 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2022-04-03T19:32:52.876 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2022-04-03T19:32:52.876 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/dc395x.ko 2022-04-03T19:32:52.876 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/device_handler 2022-04-03T19:32:52.877 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2022-04-03T19:32:52.877 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2022-04-03T19:32:52.877 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2022-04-03T19:32:52.877 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2022-04-03T19:32:52.877 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/dmx3191d.ko 2022-04-03T19:32:52.878 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/esas2r 2022-04-03T19:32:52.878 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/esas2r/esas2r.ko 2022-04-03T19:32:52.878 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/esp_scsi.ko 2022-04-03T19:32:52.878 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/fcoe 2022-04-03T19:32:52.879 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/fcoe/fcoe.ko 2022-04-03T19:32:52.879 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/fcoe/libfcoe.ko 2022-04-03T19:32:52.879 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/fnic 2022-04-03T19:32:52.879 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/fnic/fnic.ko 2022-04-03T19:32:52.879 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/hpsa.ko 2022-04-03T19:32:52.880 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/hptiop.ko 2022-04-03T19:32:52.880 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/hv_storvsc.ko 2022-04-03T19:32:52.880 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/initio.ko 2022-04-03T19:32:52.880 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/ipr.ko 2022-04-03T19:32:52.880 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/ips.ko 2022-04-03T19:32:52.881 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/isci 2022-04-03T19:32:52.881 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/isci/isci.ko 2022-04-03T19:32:52.881 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2022-04-03T19:32:52.881 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/iscsi_tcp.ko 2022-04-03T19:32:52.882 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/libfc 2022-04-03T19:32:52.882 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/libfc/libfc.ko 2022-04-03T19:32:52.882 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/libiscsi.ko 2022-04-03T19:32:52.882 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/libiscsi_tcp.ko 2022-04-03T19:32:52.882 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/libsas 2022-04-03T19:32:52.883 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/libsas/libsas.ko 2022-04-03T19:32:52.883 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/lpfc 2022-04-03T19:32:52.883 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/lpfc/lpfc.ko 2022-04-03T19:32:52.883 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/megaraid 2022-04-03T19:32:52.884 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/megaraid.ko 2022-04-03T19:32:52.884 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2022-04-03T19:32:52.884 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2022-04-03T19:32:52.884 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2022-04-03T19:32:52.884 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/mpt3sas 2022-04-03T19:32:52.885 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2022-04-03T19:32:52.885 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/mvsas 2022-04-03T19:32:52.885 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/mvsas/mvsas.ko 2022-04-03T19:32:52.885 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/mvumi.ko 2022-04-03T19:32:52.885 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/pm8001 2022-04-03T19:32:52.886 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/pm8001/pm80xx.ko 2022-04-03T19:32:52.886 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/pmcraid.ko 2022-04-03T19:32:52.886 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/qla1280.ko 2022-04-03T19:32:52.886 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/qla2xxx 2022-04-03T19:32:52.887 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2022-04-03T19:32:52.887 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2022-04-03T19:32:52.887 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/qla4xxx 2022-04-03T19:32:52.887 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2022-04-03T19:32:52.887 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/raid_class.ko 2022-04-03T19:32:52.888 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/scsi_debug.ko 2022-04-03T19:32:52.888 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/scsi_transport_fc.ko 2022-04-03T19:32:52.888 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/scsi_transport_iscsi.ko 2022-04-03T19:32:52.888 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/scsi_transport_sas.ko 2022-04-03T19:32:52.888 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/scsi_transport_spi.ko 2022-04-03T19:32:52.889 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/scsi_transport_srp.ko 2022-04-03T19:32:52.889 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/ses.ko 2022-04-03T19:32:52.889 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/st.ko 2022-04-03T19:32:52.889 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/stex.ko 2022-04-03T19:32:52.889 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/sym53c8xx_2 2022-04-03T19:32:52.890 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2022-04-03T19:32:52.890 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/ufs 2022-04-03T19:32:52.890 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/ufs/ufshcd-core.ko 2022-04-03T19:32:52.890 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/ufs/ufshcd-pci.ko 2022-04-03T19:32:52.891 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/virtio_scsi.ko 2022-04-03T19:32:52.891 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/vmw_pvscsi.ko 2022-04-03T19:32:52.891 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/scsi/wd719x.ko 2022-04-03T19:32:52.891 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ssb 2022-04-03T19:32:52.891 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/ssb/ssb.ko 2022-04-03T19:32:52.892 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/target 2022-04-03T19:32:52.892 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/target/iscsi 2022-04-03T19:32:52.892 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2022-04-03T19:32:52.892 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/target/loopback 2022-04-03T19:32:52.892 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/target/loopback/tcm_loop.ko 2022-04-03T19:32:52.893 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/target/target_core_file.ko 2022-04-03T19:32:52.893 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/target/target_core_iblock.ko 2022-04-03T19:32:52.893 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/target/target_core_mod.ko 2022-04-03T19:32:52.893 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/target/target_core_pscsi.ko 2022-04-03T19:32:52.893 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/target/target_core_user.ko 2022-04-03T19:32:52.894 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/target/tcm_fc 2022-04-03T19:32:52.894 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/target/tcm_fc/tcm_fc.ko 2022-04-03T19:32:52.894 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/thermal 2022-04-03T19:32:52.894 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/thermal/intel 2022-04-03T19:32:52.895 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/thermal/intel/int340x_thermal 2022-04-03T19:32:52.895 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2022-04-03T19:32:52.895 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2022-04-03T19:32:52.895 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2022-04-03T19:32:52.895 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2022-04-03T19:32:52.896 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2022-04-03T19:32:52.896 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2022-04-03T19:32:52.896 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2022-04-03T19:32:52.896 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2022-04-03T19:32:52.896 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2022-04-03T19:32:52.897 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2022-04-03T19:32:52.897 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2022-04-03T19:32:52.897 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2022-04-03T19:32:52.897 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2022-04-03T19:32:52.897 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2022-04-03T19:32:52.898 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2022-04-03T19:32:52.898 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/tty 2022-04-03T19:32:52.898 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/tty/ipwireless 2022-04-03T19:32:52.898 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/tty/ipwireless/ipwireless.ko 2022-04-03T19:32:52.899 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/tty/n_gsm.ko 2022-04-03T19:32:52.899 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/tty/n_hdlc.ko 2022-04-03T19:32:52.899 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/tty/nozomi.ko 2022-04-03T19:32:52.899 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/tty/serial 2022-04-03T19:32:52.899 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/tty/serial/8250 2022-04-03T19:32:52.900 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/tty/serial/8250/serial_cs.ko 2022-04-03T19:32:52.900 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/tty/serial/arc_uart.ko 2022-04-03T19:32:52.900 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/tty/serial/jsm 2022-04-03T19:32:52.900 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/tty/serial/jsm/jsm.ko 2022-04-03T19:32:52.900 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/tty/synclink_gt.ko 2022-04-03T19:32:52.901 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/uio 2022-04-03T19:32:52.901 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/uio/uio.ko 2022-04-03T19:32:52.901 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/uio/uio_aec.ko 2022-04-03T19:32:52.901 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/uio/uio_cif.ko 2022-04-03T19:32:52.901 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/uio/uio_pci_generic.ko 2022-04-03T19:32:52.902 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/uio/uio_sercos3.ko 2022-04-03T19:32:52.902 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb 2022-04-03T19:32:52.902 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/class 2022-04-03T19:32:52.902 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/class/cdc-acm.ko 2022-04-03T19:32:52.902 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/class/cdc-wdm.ko 2022-04-03T19:32:52.903 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/class/usblp.ko 2022-04-03T19:32:52.903 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/class/usbtmc.ko 2022-04-03T19:32:52.903 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/host 2022-04-03T19:32:52.903 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/host/sl811-hcd.ko 2022-04-03T19:32:52.904 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/host/u132-hcd.ko 2022-04-03T19:32:52.904 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/image 2022-04-03T19:32:52.904 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/image/mdc800.ko 2022-04-03T19:32:52.904 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/image/microtek.ko 2022-04-03T19:32:52.904 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/misc 2022-04-03T19:32:52.905 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/misc/adutux.ko 2022-04-03T19:32:52.905 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/misc/appledisplay.ko 2022-04-03T19:32:52.905 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/misc/emi26.ko 2022-04-03T19:32:52.905 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/misc/emi62.ko 2022-04-03T19:32:52.905 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/misc/ezusb.ko 2022-04-03T19:32:52.906 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/misc/ftdi-elan.ko 2022-04-03T19:32:52.906 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/misc/idmouse.ko 2022-04-03T19:32:52.906 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/misc/iowarrior.ko 2022-04-03T19:32:52.906 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/misc/isight_firmware.ko 2022-04-03T19:32:52.906 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/misc/ldusb.ko 2022-04-03T19:32:52.907 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/misc/legousbtower.ko 2022-04-03T19:32:52.907 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/misc/sisusbvga 2022-04-03T19:32:52.907 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2022-04-03T19:32:52.907 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/misc/trancevibrator.ko 2022-04-03T19:32:52.907 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/misc/usb3503.ko 2022-04-03T19:32:52.908 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/misc/usblcd.ko 2022-04-03T19:32:52.908 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/misc/usbsevseg.ko 2022-04-03T19:32:52.908 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/misc/uss720.ko 2022-04-03T19:32:52.908 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/misc/yurex.ko 2022-04-03T19:32:52.908 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/phy 2022-04-03T19:32:52.909 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/phy/phy-generic.ko 2022-04-03T19:32:52.909 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial 2022-04-03T19:32:52.909 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/aircable.ko 2022-04-03T19:32:52.909 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/ark3116.ko 2022-04-03T19:32:52.910 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/belkin_sa.ko 2022-04-03T19:32:52.910 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/ch341.ko 2022-04-03T19:32:52.910 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/cp210x.ko 2022-04-03T19:32:52.910 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/cyberjack.ko 2022-04-03T19:32:52.910 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/cypress_m8.ko 2022-04-03T19:32:52.911 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/digi_acceleport.ko 2022-04-03T19:32:52.911 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/empeg.ko 2022-04-03T19:32:52.911 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/ftdi_sio.ko 2022-04-03T19:32:52.911 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/garmin_gps.ko 2022-04-03T19:32:52.911 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/io_edgeport.ko 2022-04-03T19:32:52.912 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/io_ti.ko 2022-04-03T19:32:52.912 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/ipaq.ko 2022-04-03T19:32:52.912 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/ipw.ko 2022-04-03T19:32:52.912 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/ir-usb.ko 2022-04-03T19:32:52.912 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/iuu_phoenix.ko 2022-04-03T19:32:52.913 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/keyspan.ko 2022-04-03T19:32:52.913 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/keyspan_pda.ko 2022-04-03T19:32:52.913 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/kl5kusb105.ko 2022-04-03T19:32:52.913 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/kobil_sct.ko 2022-04-03T19:32:52.914 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/mct_u232.ko 2022-04-03T19:32:52.914 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/mos7720.ko 2022-04-03T19:32:52.914 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/mos7840.ko 2022-04-03T19:32:52.914 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/navman.ko 2022-04-03T19:32:52.914 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/omninet.ko 2022-04-03T19:32:52.915 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/opticon.ko 2022-04-03T19:32:52.915 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/option.ko 2022-04-03T19:32:52.915 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/oti6858.ko 2022-04-03T19:32:52.915 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/pl2303.ko 2022-04-03T19:32:52.915 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/qcaux.ko 2022-04-03T19:32:52.916 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/qcserial.ko 2022-04-03T19:32:52.916 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/quatech2.ko 2022-04-03T19:32:52.916 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/safe_serial.ko 2022-04-03T19:32:52.916 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/sierra.ko 2022-04-03T19:32:52.916 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/spcp8x5.ko 2022-04-03T19:32:52.917 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/ssu100.ko 2022-04-03T19:32:52.917 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/symbolserial.ko 2022-04-03T19:32:52.917 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2022-04-03T19:32:52.917 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/usb-serial-simple.ko 2022-04-03T19:32:52.917 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/usb_debug.ko 2022-04-03T19:32:52.918 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/usb_wwan.ko 2022-04-03T19:32:52.918 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/visor.ko 2022-04-03T19:32:52.918 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/whiteheat.ko 2022-04-03T19:32:52.918 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/serial/xsens_mt.ko 2022-04-03T19:32:52.918 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/storage 2022-04-03T19:32:52.919 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/storage/uas.ko 2022-04-03T19:32:52.919 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/storage/ums-alauda.ko 2022-04-03T19:32:52.919 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/storage/ums-cypress.ko 2022-04-03T19:32:52.919 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/storage/ums-datafab.ko 2022-04-03T19:32:52.919 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/storage/ums-eneub6250.ko 2022-04-03T19:32:52.920 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/storage/ums-freecom.ko 2022-04-03T19:32:52.920 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/storage/ums-isd200.ko 2022-04-03T19:32:52.920 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/storage/ums-jumpshot.ko 2022-04-03T19:32:52.920 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/storage/ums-karma.ko 2022-04-03T19:32:52.920 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/storage/ums-onetouch.ko 2022-04-03T19:32:52.921 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/storage/ums-realtek.ko 2022-04-03T19:32:52.921 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/storage/ums-sddr09.ko 2022-04-03T19:32:52.921 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/storage/ums-sddr55.ko 2022-04-03T19:32:52.921 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/storage/ums-usbat.ko 2022-04-03T19:32:52.921 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/storage/usb-storage.ko 2022-04-03T19:32:52.922 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/usbip 2022-04-03T19:32:52.922 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/usbip/usbip-core.ko 2022-04-03T19:32:52.922 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/usbip/usbip-host.ko 2022-04-03T19:32:52.922 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/usb/usbip/vhci-hcd.ko 2022-04-03T19:32:52.923 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/vfio 2022-04-03T19:32:52.923 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/vfio/pci 2022-04-03T19:32:52.923 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/vfio/pci/vfio-pci-core.ko 2022-04-03T19:32:52.923 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/vfio/pci/vfio-pci.ko 2022-04-03T19:32:52.923 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/vfio/vfio.ko 2022-04-03T19:32:52.924 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/vfio/vfio_iommu_type1.ko 2022-04-03T19:32:52.924 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/vfio/vfio_virqfd.ko 2022-04-03T19:32:52.924 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/vhost 2022-04-03T19:32:52.924 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/vhost/vhost.ko 2022-04-03T19:32:52.924 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/vhost/vhost_iotlb.ko 2022-04-03T19:32:52.925 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/vhost/vhost_net.ko 2022-04-03T19:32:52.925 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/vhost/vhost_scsi.ko 2022-04-03T19:32:52.925 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/video 2022-04-03T19:32:52.925 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/video/backlight 2022-04-03T19:32:52.925 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/video/backlight/apple_bl.ko 2022-04-03T19:32:52.926 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/video/backlight/lcd.ko 2022-04-03T19:32:52.926 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/video/backlight/platform_lcd.ko 2022-04-03T19:32:52.926 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/virtio 2022-04-03T19:32:52.926 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/virtio/virtio.ko 2022-04-03T19:32:52.927 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/virtio/virtio_balloon.ko 2022-04-03T19:32:52.927 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/virtio/virtio_mmio.ko 2022-04-03T19:32:52.927 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/virtio/virtio_pci.ko 2022-04-03T19:32:52.927 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2022-04-03T19:32:52.927 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2022-04-03T19:32:52.928 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/virtio/virtio_ring.ko 2022-04-03T19:32:52.928 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/w1 2022-04-03T19:32:52.928 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/w1/masters 2022-04-03T19:32:52.928 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/w1/masters/ds1wm.ko 2022-04-03T19:32:52.928 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/w1/masters/ds2482.ko 2022-04-03T19:32:52.929 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/w1/masters/ds2490.ko 2022-04-03T19:32:52.929 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/w1/slaves 2022-04-03T19:32:52.929 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/w1/slaves/w1_ds2406.ko 2022-04-03T19:32:52.929 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/w1/slaves/w1_ds2408.ko 2022-04-03T19:32:52.929 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/w1/slaves/w1_ds2413.ko 2022-04-03T19:32:52.930 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/w1/slaves/w1_ds2423.ko 2022-04-03T19:32:52.930 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/w1/slaves/w1_ds2431.ko 2022-04-03T19:32:52.930 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/w1/slaves/w1_ds2433.ko 2022-04-03T19:32:52.930 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/w1/slaves/w1_ds2780.ko 2022-04-03T19:32:52.931 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/w1/slaves/w1_ds2781.ko 2022-04-03T19:32:52.931 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/w1/slaves/w1_ds28e04.ko 2022-04-03T19:32:52.931 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/w1/slaves/w1_smem.ko 2022-04-03T19:32:52.931 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/w1/slaves/w1_therm.ko 2022-04-03T19:32:52.931 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/w1/wire.ko 2022-04-03T19:32:52.932 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog 2022-04-03T19:32:52.932 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/alim1535_wdt.ko 2022-04-03T19:32:52.932 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/alim7101_wdt.ko 2022-04-03T19:32:52.932 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/f71808e_wdt.ko 2022-04-03T19:32:52.933 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/hpwdt.ko 2022-04-03T19:32:52.933 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/i6300esb.ko 2022-04-03T19:32:52.933 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/iTCO_vendor_support.ko 2022-04-03T19:32:52.933 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/iTCO_wdt.ko 2022-04-03T19:32:52.933 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/ib700wdt.ko 2022-04-03T19:32:52.934 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/ibmasr.ko 2022-04-03T19:32:52.934 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/ie6xx_wdt.ko 2022-04-03T19:32:52.934 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/it8712f_wdt.ko 2022-04-03T19:32:52.934 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/it87_wdt.ko 2022-04-03T19:32:52.934 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/machzwd.ko 2022-04-03T19:32:52.935 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/nv_tco.ko 2022-04-03T19:32:52.935 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/pcwd_pci.ko 2022-04-03T19:32:52.935 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/pcwd_usb.ko 2022-04-03T19:32:52.935 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2022-04-03T19:32:52.935 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/sch311x_wdt.ko 2022-04-03T19:32:52.936 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/softdog.ko 2022-04-03T19:32:52.936 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/sp5100_tco.ko 2022-04-03T19:32:52.936 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/via_wdt.ko 2022-04-03T19:32:52.936 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/w83627hf_wdt.ko 2022-04-03T19:32:52.936 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/w83877f_wdt.ko 2022-04-03T19:32:52.937 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/w83977f_wdt.ko 2022-04-03T19:32:52.937 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/wdt_pci.ko 2022-04-03T19:32:52.937 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/watchdog/xen_wdt.ko 2022-04-03T19:32:52.937 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/xen 2022-04-03T19:32:52.937 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/xen/xen-acpi-processor.ko 2022-04-03T19:32:52.938 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/xen/xen-evtchn.ko 2022-04-03T19:32:52.938 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/xen/xen-gntalloc.ko 2022-04-03T19:32:52.938 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/xen/xen-gntdev.ko 2022-04-03T19:32:52.938 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/xen/xen-pciback 2022-04-03T19:32:52.938 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2022-04-03T19:32:52.939 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/xen/xen-privcmd.ko 2022-04-03T19:32:52.939 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/xen/xenfs 2022-04-03T19:32:52.939 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/drivers/xen/xenfs/xenfs.ko 2022-04-03T19:32:52.939 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs 2022-04-03T19:32:52.939 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/9p 2022-04-03T19:32:52.940 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/9p/9p.ko 2022-04-03T19:32:52.940 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/affs 2022-04-03T19:32:52.940 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/affs/affs.ko 2022-04-03T19:32:52.940 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/befs 2022-04-03T19:32:52.941 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/befs/befs.ko 2022-04-03T19:32:52.941 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/binfmt_misc.ko 2022-04-03T19:32:52.941 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/btrfs 2022-04-03T19:32:52.941 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/btrfs/btrfs.ko 2022-04-03T19:32:52.941 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/cachefiles 2022-04-03T19:32:52.942 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/cachefiles/cachefiles.ko 2022-04-03T19:32:52.942 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/ceph 2022-04-03T19:32:52.942 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/ceph/ceph.ko 2022-04-03T19:32:52.942 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/cifs 2022-04-03T19:32:52.942 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/cifs/cifs.ko 2022-04-03T19:32:52.943 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/coda 2022-04-03T19:32:52.943 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/coda/coda.ko 2022-04-03T19:32:52.943 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/cramfs 2022-04-03T19:32:52.943 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/cramfs/cramfs.ko 2022-04-03T19:32:52.943 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/dlm 2022-04-03T19:32:52.944 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/dlm/dlm.ko 2022-04-03T19:32:52.944 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/ecryptfs 2022-04-03T19:32:52.944 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/ecryptfs/ecryptfs.ko 2022-04-03T19:32:52.944 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/f2fs 2022-04-03T19:32:52.944 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/f2fs/f2fs.ko 2022-04-03T19:32:52.945 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/fat 2022-04-03T19:32:52.945 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/fat/fat.ko 2022-04-03T19:32:52.945 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/fat/msdos.ko 2022-04-03T19:32:52.945 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/fat/vfat.ko 2022-04-03T19:32:52.945 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/fscache 2022-04-03T19:32:52.946 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/fscache/fscache.ko 2022-04-03T19:32:52.946 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/fuse 2022-04-03T19:32:52.946 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/fuse/cuse.ko 2022-04-03T19:32:52.946 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/fuse/fuse.ko 2022-04-03T19:32:52.947 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/gfs2 2022-04-03T19:32:52.947 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/gfs2/gfs2.ko 2022-04-03T19:32:52.947 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/hfs 2022-04-03T19:32:52.947 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/hfs/hfs.ko 2022-04-03T19:32:52.947 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/hfsplus 2022-04-03T19:32:52.948 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/hfsplus/hfsplus.ko 2022-04-03T19:32:52.948 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/isofs 2022-04-03T19:32:52.948 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/isofs/isofs.ko 2022-04-03T19:32:52.948 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/jfs 2022-04-03T19:32:52.948 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/jfs/jfs.ko 2022-04-03T19:32:52.949 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/lockd 2022-04-03T19:32:52.949 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/lockd/lockd.ko 2022-04-03T19:32:52.949 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/minix 2022-04-03T19:32:52.949 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/minix/minix.ko 2022-04-03T19:32:52.949 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/netfs 2022-04-03T19:32:52.950 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/netfs/netfs.ko 2022-04-03T19:32:52.950 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nfs 2022-04-03T19:32:52.950 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nfs/blocklayout 2022-04-03T19:32:52.951 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2022-04-03T19:32:52.951 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nfs/filelayout 2022-04-03T19:32:52.951 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2022-04-03T19:32:52.951 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nfs/flexfilelayout 2022-04-03T19:32:52.951 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2022-04-03T19:32:52.952 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nfs/nfs.ko 2022-04-03T19:32:52.952 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nfs/nfsv3.ko 2022-04-03T19:32:52.952 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nfs/nfsv4.ko 2022-04-03T19:32:52.952 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nfs_common 2022-04-03T19:32:52.953 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nfs_common/grace.ko 2022-04-03T19:32:52.953 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nfs_common/nfs_acl.ko 2022-04-03T19:32:52.953 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nfsd 2022-04-03T19:32:52.953 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nfsd/nfsd.ko 2022-04-03T19:32:52.953 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nilfs2 2022-04-03T19:32:52.954 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nilfs2/nilfs2.ko 2022-04-03T19:32:52.954 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls 2022-04-03T19:32:52.954 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/mac-celtic.ko 2022-04-03T19:32:52.954 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/mac-centeuro.ko 2022-04-03T19:32:52.955 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/mac-croatian.ko 2022-04-03T19:32:52.955 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/mac-cyrillic.ko 2022-04-03T19:32:52.955 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/mac-gaelic.ko 2022-04-03T19:32:52.955 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/mac-greek.ko 2022-04-03T19:32:52.955 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/mac-iceland.ko 2022-04-03T19:32:52.956 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/mac-inuit.ko 2022-04-03T19:32:52.956 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/mac-roman.ko 2022-04-03T19:32:52.956 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/mac-romanian.ko 2022-04-03T19:32:52.956 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/mac-turkish.ko 2022-04-03T19:32:52.957 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp1250.ko 2022-04-03T19:32:52.957 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp1251.ko 2022-04-03T19:32:52.957 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp1255.ko 2022-04-03T19:32:52.957 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp737.ko 2022-04-03T19:32:52.957 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp775.ko 2022-04-03T19:32:52.958 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp850.ko 2022-04-03T19:32:52.958 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp852.ko 2022-04-03T19:32:52.958 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp855.ko 2022-04-03T19:32:52.958 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp857.ko 2022-04-03T19:32:52.959 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp860.ko 2022-04-03T19:32:52.959 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp861.ko 2022-04-03T19:32:52.959 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp862.ko 2022-04-03T19:32:52.959 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp863.ko 2022-04-03T19:32:52.960 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp864.ko 2022-04-03T19:32:52.960 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp865.ko 2022-04-03T19:32:52.960 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp866.ko 2022-04-03T19:32:52.960 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp869.ko 2022-04-03T19:32:52.960 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp874.ko 2022-04-03T19:32:52.961 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp932.ko 2022-04-03T19:32:52.961 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp936.ko 2022-04-03T19:32:52.961 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp949.ko 2022-04-03T19:32:52.961 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_cp950.ko 2022-04-03T19:32:52.962 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_euc-jp.ko 2022-04-03T19:32:52.962 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_iso8859-1.ko 2022-04-03T19:32:52.962 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_iso8859-13.ko 2022-04-03T19:32:52.962 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_iso8859-14.ko 2022-04-03T19:32:52.962 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_iso8859-15.ko 2022-04-03T19:32:52.963 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_iso8859-2.ko 2022-04-03T19:32:52.963 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_iso8859-3.ko 2022-04-03T19:32:52.963 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_iso8859-4.ko 2022-04-03T19:32:52.963 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_iso8859-5.ko 2022-04-03T19:32:52.964 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_iso8859-6.ko 2022-04-03T19:32:52.964 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_iso8859-7.ko 2022-04-03T19:32:52.964 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_iso8859-9.ko 2022-04-03T19:32:52.964 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_koi8-r.ko 2022-04-03T19:32:52.964 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_koi8-ru.ko 2022-04-03T19:32:52.965 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_koi8-u.ko 2022-04-03T19:32:52.965 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/nls/nls_utf8.ko 2022-04-03T19:32:52.965 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/ocfs2 2022-04-03T19:32:52.965 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/ocfs2/cluster 2022-04-03T19:32:52.966 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2022-04-03T19:32:52.966 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/ocfs2/dlm 2022-04-03T19:32:52.966 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2022-04-03T19:32:52.966 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/ocfs2/dlmfs 2022-04-03T19:32:52.966 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2022-04-03T19:32:52.967 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/ocfs2/ocfs2.ko 2022-04-03T19:32:52.967 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2022-04-03T19:32:52.967 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/ocfs2/ocfs2_stack_user.ko 2022-04-03T19:32:52.967 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/ocfs2/ocfs2_stackglue.ko 2022-04-03T19:32:52.968 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/overlayfs 2022-04-03T19:32:52.968 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/overlayfs/overlay.ko 2022-04-03T19:32:52.968 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/pstore 2022-04-03T19:32:52.968 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/pstore/ramoops.ko 2022-04-03T19:32:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/reiserfs 2022-04-03T19:32:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/reiserfs/reiserfs.ko 2022-04-03T19:32:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/romfs 2022-04-03T19:32:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/romfs/romfs.ko 2022-04-03T19:32:52.970 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/smbfs_common 2022-04-03T19:32:52.970 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/smbfs_common/cifs_arc4.ko 2022-04-03T19:32:52.970 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/smbfs_common/cifs_md4.ko 2022-04-03T19:32:52.970 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/squashfs 2022-04-03T19:32:52.970 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/squashfs/squashfs.ko 2022-04-03T19:32:52.971 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/sysv 2022-04-03T19:32:52.971 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/sysv/sysv.ko 2022-04-03T19:32:52.971 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/udf 2022-04-03T19:32:52.971 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/udf/udf.ko 2022-04-03T19:32:52.972 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/ufs 2022-04-03T19:32:52.972 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/ufs/ufs.ko 2022-04-03T19:32:52.972 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/xfs 2022-04-03T19:32:52.972 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/fs/xfs/xfs.ko 2022-04-03T19:32:52.973 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/kernel 2022-04-03T19:32:52.973 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/kernel/rcu 2022-04-03T19:32:52.973 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/kernel/rcu/rcutorture.ko 2022-04-03T19:32:52.973 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/kernel/torture.ko 2022-04-03T19:32:52.974 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/kernel/trace 2022-04-03T19:32:52.974 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/kernel/trace/ring_buffer_benchmark.ko 2022-04-03T19:32:52.974 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib 2022-04-03T19:32:52.974 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/asn1_encoder.ko 2022-04-03T19:32:52.974 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/crc-itu-t.ko 2022-04-03T19:32:52.975 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/crc64.ko 2022-04-03T19:32:52.975 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/crc8.ko 2022-04-03T19:32:52.975 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/crypto 2022-04-03T19:32:52.975 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/crypto/libarc4.ko 2022-04-03T19:32:52.976 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/crypto/libdes.ko 2022-04-03T19:32:52.976 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/libcrc32c.ko 2022-04-03T19:32:52.976 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/lru_cache.ko 2022-04-03T19:32:52.976 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/lz4 2022-04-03T19:32:52.976 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/lz4/lz4_compress.ko 2022-04-03T19:32:52.977 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/lz4/lz4hc_compress.ko 2022-04-03T19:32:52.977 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/math 2022-04-03T19:32:52.977 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/math/cordic.ko 2022-04-03T19:32:52.977 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/raid6 2022-04-03T19:32:52.978 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/raid6/raid6_pq.ko 2022-04-03T19:32:52.978 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/reed_solomon 2022-04-03T19:32:52.978 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/reed_solomon/reed_solomon.ko 2022-04-03T19:32:52.978 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/ts_bm.ko 2022-04-03T19:32:52.978 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/ts_fsm.ko 2022-04-03T19:32:52.979 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/ts_kmp.ko 2022-04-03T19:32:52.979 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/zstd 2022-04-03T19:32:52.979 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/lib/zstd/zstd_compress.ko 2022-04-03T19:32:52.979 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/mm 2022-04-03T19:32:52.980 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/mm/hwpoison-inject.ko 2022-04-03T19:32:52.980 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net 2022-04-03T19:32:52.980 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/6lowpan 2022-04-03T19:32:52.980 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/6lowpan/6lowpan.ko 2022-04-03T19:32:52.980 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/6lowpan/nhc_dest.ko 2022-04-03T19:32:52.981 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/6lowpan/nhc_fragment.ko 2022-04-03T19:32:52.981 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/6lowpan/nhc_hop.ko 2022-04-03T19:32:52.981 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/6lowpan/nhc_ipv6.ko 2022-04-03T19:32:52.981 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/6lowpan/nhc_mobility.ko 2022-04-03T19:32:52.982 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/6lowpan/nhc_routing.ko 2022-04-03T19:32:52.982 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/6lowpan/nhc_udp.ko 2022-04-03T19:32:52.982 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/802 2022-04-03T19:32:52.982 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/802/garp.ko 2022-04-03T19:32:52.982 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/802/mrp.ko 2022-04-03T19:32:52.983 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/802/p8022.ko 2022-04-03T19:32:52.983 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/802/psnap.ko 2022-04-03T19:32:52.983 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/802/stp.ko 2022-04-03T19:32:52.983 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/8021q 2022-04-03T19:32:52.984 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/8021q/8021q.ko 2022-04-03T19:32:52.984 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/9p 2022-04-03T19:32:52.984 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/9p/9pnet.ko 2022-04-03T19:32:52.984 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/9p/9pnet_fd.ko 2022-04-03T19:32:52.984 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/9p/9pnet_rdma.ko 2022-04-03T19:32:52.985 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/9p/9pnet_virtio.ko 2022-04-03T19:32:52.985 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/batman-adv 2022-04-03T19:32:52.985 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/batman-adv/batman-adv.ko 2022-04-03T19:32:52.985 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge 2022-04-03T19:32:52.986 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/br_netfilter.ko 2022-04-03T19:32:52.986 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/bridge.ko 2022-04-03T19:32:52.986 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter 2022-04-03T19:32:52.986 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebt_802_3.ko 2022-04-03T19:32:52.986 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebt_among.ko 2022-04-03T19:32:52.987 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebt_arp.ko 2022-04-03T19:32:52.987 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebt_arpreply.ko 2022-04-03T19:32:52.987 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebt_dnat.ko 2022-04-03T19:32:52.987 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebt_ip.ko 2022-04-03T19:32:52.988 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebt_ip6.ko 2022-04-03T19:32:52.988 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebt_limit.ko 2022-04-03T19:32:52.988 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebt_log.ko 2022-04-03T19:32:52.988 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebt_mark.ko 2022-04-03T19:32:52.988 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebt_mark_m.ko 2022-04-03T19:32:52.989 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebt_nflog.ko 2022-04-03T19:32:52.989 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebt_pkttype.ko 2022-04-03T19:32:52.989 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebt_redirect.ko 2022-04-03T19:32:52.989 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebt_snat.ko 2022-04-03T19:32:52.990 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebt_stp.ko 2022-04-03T19:32:52.990 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebt_vlan.ko 2022-04-03T19:32:52.990 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebtable_broute.ko 2022-04-03T19:32:52.990 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebtable_filter.ko 2022-04-03T19:32:52.990 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebtable_nat.ko 2022-04-03T19:32:52.991 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/ebtables.ko 2022-04-03T19:32:52.991 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2022-04-03T19:32:52.991 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2022-04-03T19:32:52.991 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2022-04-03T19:32:52.992 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ceph 2022-04-03T19:32:52.992 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ceph/libceph.ko 2022-04-03T19:32:52.992 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/core 2022-04-03T19:32:52.992 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/core/failover.ko 2022-04-03T19:32:52.993 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/core/pktgen.ko 2022-04-03T19:32:52.993 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/dccp 2022-04-03T19:32:52.993 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/dccp/dccp.ko 2022-04-03T19:32:52.993 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/dccp/dccp_diag.ko 2022-04-03T19:32:52.993 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/dccp/dccp_ipv4.ko 2022-04-03T19:32:52.994 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/dccp/dccp_ipv6.ko 2022-04-03T19:32:52.994 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/dns_resolver 2022-04-03T19:32:52.994 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/dns_resolver/dns_resolver.ko 2022-04-03T19:32:52.994 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/dsa 2022-04-03T19:32:52.994 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/dsa/dsa_core.ko 2022-04-03T19:32:52.995 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/dsa/tag_brcm.ko 2022-04-03T19:32:52.995 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/dsa/tag_dsa.ko 2022-04-03T19:32:52.995 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/dsa/tag_trailer.ko 2022-04-03T19:32:52.995 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ieee802154 2022-04-03T19:32:52.996 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ieee802154/6lowpan 2022-04-03T19:32:52.996 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2022-04-03T19:32:52.996 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ieee802154/ieee802154.ko 2022-04-03T19:32:52.996 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ieee802154/ieee802154_socket.ko 2022-04-03T19:32:52.996 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4 2022-04-03T19:32:52.997 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/ah4.ko 2022-04-03T19:32:52.997 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/esp4.ko 2022-04-03T19:32:52.997 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/fou.ko 2022-04-03T19:32:52.997 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/gre.ko 2022-04-03T19:32:52.998 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/inet_diag.ko 2022-04-03T19:32:52.998 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/ip_gre.ko 2022-04-03T19:32:52.998 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/ip_tunnel.ko 2022-04-03T19:32:52.998 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/ip_vti.ko 2022-04-03T19:32:52.998 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/ipcomp.ko 2022-04-03T19:32:52.999 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/ipip.ko 2022-04-03T19:32:52.999 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter 2022-04-03T19:32:52.999 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/arp_tables.ko 2022-04-03T19:32:52.999 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/arpt_mangle.ko 2022-04-03T19:32:53.000 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/arptable_filter.ko 2022-04-03T19:32:53.000 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/ipt_CLUSTERIP.ko 2022-04-03T19:32:53.000 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/ipt_ECN.ko 2022-04-03T19:32:53.000 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2022-04-03T19:32:53.000 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/ipt_ah.ko 2022-04-03T19:32:53.001 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2022-04-03T19:32:53.001 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/iptable_mangle.ko 2022-04-03T19:32:53.001 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/iptable_nat.ko 2022-04-03T19:32:53.001 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/iptable_raw.ko 2022-04-03T19:32:53.001 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/iptable_security.ko 2022-04-03T19:32:53.002 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2022-04-03T19:32:53.002 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2022-04-03T19:32:53.002 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2022-04-03T19:32:53.002 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2022-04-03T19:32:53.003 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2022-04-03T19:32:53.003 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2022-04-03T19:32:53.003 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2022-04-03T19:32:53.003 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2022-04-03T19:32:53.003 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2022-04-03T19:32:53.004 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2022-04-03T19:32:53.004 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/tcp_bic.ko 2022-04-03T19:32:53.004 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/tcp_dctcp.ko 2022-04-03T19:32:53.004 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/tcp_diag.ko 2022-04-03T19:32:53.005 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/tcp_highspeed.ko 2022-04-03T19:32:53.005 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/tcp_htcp.ko 2022-04-03T19:32:53.005 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/tcp_hybla.ko 2022-04-03T19:32:53.005 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/tcp_illinois.ko 2022-04-03T19:32:53.005 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/tcp_lp.ko 2022-04-03T19:32:53.006 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/tcp_scalable.ko 2022-04-03T19:32:53.006 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/tcp_vegas.ko 2022-04-03T19:32:53.006 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/tcp_veno.ko 2022-04-03T19:32:53.006 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/tcp_westwood.ko 2022-04-03T19:32:53.006 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/tcp_yeah.ko 2022-04-03T19:32:53.007 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/tunnel4.ko 2022-04-03T19:32:53.007 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/udp_diag.ko 2022-04-03T19:32:53.007 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/udp_tunnel.ko 2022-04-03T19:32:53.007 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv4/xfrm4_tunnel.ko 2022-04-03T19:32:53.007 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6 2022-04-03T19:32:53.008 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/ah6.ko 2022-04-03T19:32:53.008 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/esp6.ko 2022-04-03T19:32:53.008 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/fou6.ko 2022-04-03T19:32:53.008 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/ip6_tunnel.ko 2022-04-03T19:32:53.008 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/ip6_udp_tunnel.ko 2022-04-03T19:32:53.009 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/ip6_vti.ko 2022-04-03T19:32:53.009 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/ipcomp6.ko 2022-04-03T19:32:53.009 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter 2022-04-03T19:32:53.009 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/ip6_tables.ko 2022-04-03T19:32:53.009 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2022-04-03T19:32:53.010 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2022-04-03T19:32:53.010 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/ip6t_ah.ko 2022-04-03T19:32:53.010 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2022-04-03T19:32:53.010 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/ip6t_frag.ko 2022-04-03T19:32:53.011 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2022-04-03T19:32:53.011 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2022-04-03T19:32:53.011 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/ip6t_mh.ko 2022-04-03T19:32:53.011 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2022-04-03T19:32:53.011 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/ip6t_rt.ko 2022-04-03T19:32:53.012 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/ip6table_filter.ko 2022-04-03T19:32:53.012 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2022-04-03T19:32:53.012 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/ip6table_nat.ko 2022-04-03T19:32:53.012 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/ip6table_raw.ko 2022-04-03T19:32:53.012 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/ip6table_security.ko 2022-04-03T19:32:53.013 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2022-04-03T19:32:53.013 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2022-04-03T19:32:53.013 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2022-04-03T19:32:53.013 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2022-04-03T19:32:53.013 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2022-04-03T19:32:53.014 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2022-04-03T19:32:53.014 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2022-04-03T19:32:53.014 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2022-04-03T19:32:53.014 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/sit.ko 2022-04-03T19:32:53.015 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/tunnel6.ko 2022-04-03T19:32:53.015 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/ipv6/xfrm6_tunnel.ko 2022-04-03T19:32:53.015 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/key 2022-04-03T19:32:53.015 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/key/af_key.ko 2022-04-03T19:32:53.015 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/l2tp 2022-04-03T19:32:53.016 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/l2tp/l2tp_core.ko 2022-04-03T19:32:53.016 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/l2tp/l2tp_debugfs.ko 2022-04-03T19:32:53.016 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/l2tp/l2tp_eth.ko 2022-04-03T19:32:53.016 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/l2tp/l2tp_ip.ko 2022-04-03T19:32:53.016 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/l2tp/l2tp_ip6.ko 2022-04-03T19:32:53.017 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/l2tp/l2tp_netlink.ko 2022-04-03T19:32:53.017 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/l2tp/l2tp_ppp.ko 2022-04-03T19:32:53.017 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/llc 2022-04-03T19:32:53.017 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/llc/llc.ko 2022-04-03T19:32:53.017 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/mac802154 2022-04-03T19:32:53.018 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/mac802154/mac802154.ko 2022-04-03T19:32:53.018 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/mpls 2022-04-03T19:32:53.018 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/mpls/mpls_gso.ko 2022-04-03T19:32:53.018 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter 2022-04-03T19:32:53.018 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipset 2022-04-03T19:32:53.019 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipset/ip_set.ko 2022-04-03T19:32:53.019 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2022-04-03T19:32:53.019 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2022-04-03T19:32:53.019 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2022-04-03T19:32:53.020 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2022-04-03T19:32:53.020 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2022-04-03T19:32:53.020 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2022-04-03T19:32:53.020 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2022-04-03T19:32:53.020 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2022-04-03T19:32:53.021 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2022-04-03T19:32:53.021 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2022-04-03T19:32:53.021 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2022-04-03T19:32:53.021 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2022-04-03T19:32:53.021 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2022-04-03T19:32:53.022 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2022-04-03T19:32:53.022 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipset/ip_set_list_set.ko 2022-04-03T19:32:53.022 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipvs 2022-04-03T19:32:53.022 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipvs/ip_vs.ko 2022-04-03T19:32:53.022 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2022-04-03T19:32:53.023 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2022-04-03T19:32:53.023 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2022-04-03T19:32:53.023 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2022-04-03T19:32:53.023 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2022-04-03T19:32:53.023 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2022-04-03T19:32:53.024 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2022-04-03T19:32:53.024 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2022-04-03T19:32:53.024 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2022-04-03T19:32:53.024 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2022-04-03T19:32:53.024 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2022-04-03T19:32:53.025 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2022-04-03T19:32:53.025 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2022-04-03T19:32:53.025 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_conncount.ko 2022-04-03T19:32:53.025 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_conntrack.ko 2022-04-03T19:32:53.025 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_conntrack_amanda.ko 2022-04-03T19:32:53.026 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_conntrack_broadcast.ko 2022-04-03T19:32:53.026 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_conntrack_ftp.ko 2022-04-03T19:32:53.026 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_conntrack_h323.ko 2022-04-03T19:32:53.026 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_conntrack_irc.ko 2022-04-03T19:32:53.027 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2022-04-03T19:32:53.027 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_conntrack_netlink.ko 2022-04-03T19:32:53.027 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_conntrack_pptp.ko 2022-04-03T19:32:53.027 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_conntrack_sane.ko 2022-04-03T19:32:53.027 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_conntrack_sip.ko 2022-04-03T19:32:53.028 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_conntrack_snmp.ko 2022-04-03T19:32:53.028 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_conntrack_tftp.ko 2022-04-03T19:32:53.028 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_dup_netdev.ko 2022-04-03T19:32:53.028 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_flow_table.ko 2022-04-03T19:32:53.028 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_flow_table_inet.ko 2022-04-03T19:32:53.029 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_log_syslog.ko 2022-04-03T19:32:53.029 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_nat.ko 2022-04-03T19:32:53.029 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_nat_amanda.ko 2022-04-03T19:32:53.029 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_nat_ftp.ko 2022-04-03T19:32:53.029 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_nat_irc.ko 2022-04-03T19:32:53.030 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_nat_sip.ko 2022-04-03T19:32:53.030 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_nat_tftp.ko 2022-04-03T19:32:53.030 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_synproxy_core.ko 2022-04-03T19:32:53.030 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nf_tables.ko 2022-04-03T19:32:53.031 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nfnetlink.ko 2022-04-03T19:32:53.031 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nfnetlink_acct.ko 2022-04-03T19:32:53.031 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nfnetlink_log.ko 2022-04-03T19:32:53.031 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nfnetlink_osf.ko 2022-04-03T19:32:53.031 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nfnetlink_queue.ko 2022-04-03T19:32:53.032 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nft_chain_nat.ko 2022-04-03T19:32:53.032 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nft_compat.ko 2022-04-03T19:32:53.032 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nft_ct.ko 2022-04-03T19:32:53.032 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nft_dup_netdev.ko 2022-04-03T19:32:53.032 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nft_fib.ko 2022-04-03T19:32:53.033 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nft_fwd_netdev.ko 2022-04-03T19:32:53.033 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nft_hash.ko 2022-04-03T19:32:53.033 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nft_limit.ko 2022-04-03T19:32:53.033 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nft_log.ko 2022-04-03T19:32:53.033 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nft_masq.ko 2022-04-03T19:32:53.034 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nft_nat.ko 2022-04-03T19:32:53.034 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nft_numgen.ko 2022-04-03T19:32:53.034 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nft_objref.ko 2022-04-03T19:32:53.034 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nft_queue.ko 2022-04-03T19:32:53.034 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nft_quota.ko 2022-04-03T19:32:53.035 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nft_redir.ko 2022-04-03T19:32:53.035 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nft_reject.ko 2022-04-03T19:32:53.035 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nft_reject_inet.ko 2022-04-03T19:32:53.035 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nft_socket.ko 2022-04-03T19:32:53.035 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/nft_tunnel.ko 2022-04-03T19:32:53.036 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_AUDIT.ko 2022-04-03T19:32:53.036 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_CHECKSUM.ko 2022-04-03T19:32:53.036 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_CLASSIFY.ko 2022-04-03T19:32:53.036 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_CONNSECMARK.ko 2022-04-03T19:32:53.037 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_CT.ko 2022-04-03T19:32:53.037 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_DSCP.ko 2022-04-03T19:32:53.037 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_HL.ko 2022-04-03T19:32:53.037 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_HMARK.ko 2022-04-03T19:32:53.037 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_IDLETIMER.ko 2022-04-03T19:32:53.038 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_LED.ko 2022-04-03T19:32:53.038 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_LOG.ko 2022-04-03T19:32:53.038 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_MASQUERADE.ko 2022-04-03T19:32:53.038 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_NETMAP.ko 2022-04-03T19:32:53.038 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_NFLOG.ko 2022-04-03T19:32:53.039 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_NFQUEUE.ko 2022-04-03T19:32:53.039 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_RATEEST.ko 2022-04-03T19:32:53.039 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_REDIRECT.ko 2022-04-03T19:32:53.039 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_SECMARK.ko 2022-04-03T19:32:53.040 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_TCPMSS.ko 2022-04-03T19:32:53.040 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2022-04-03T19:32:53.040 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_TEE.ko 2022-04-03T19:32:53.040 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_TPROXY.ko 2022-04-03T19:32:53.040 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_TRACE.ko 2022-04-03T19:32:53.041 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_addrtype.ko 2022-04-03T19:32:53.041 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_bpf.ko 2022-04-03T19:32:53.041 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_cgroup.ko 2022-04-03T19:32:53.041 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_cluster.ko 2022-04-03T19:32:53.041 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_comment.ko 2022-04-03T19:32:53.042 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_connbytes.ko 2022-04-03T19:32:53.042 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_connlabel.ko 2022-04-03T19:32:53.042 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_connlimit.ko 2022-04-03T19:32:53.042 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_connmark.ko 2022-04-03T19:32:53.043 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_conntrack.ko 2022-04-03T19:32:53.043 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_cpu.ko 2022-04-03T19:32:53.043 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_dccp.ko 2022-04-03T19:32:53.043 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_devgroup.ko 2022-04-03T19:32:53.043 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_dscp.ko 2022-04-03T19:32:53.044 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_ecn.ko 2022-04-03T19:32:53.044 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_esp.ko 2022-04-03T19:32:53.044 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_hashlimit.ko 2022-04-03T19:32:53.044 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_helper.ko 2022-04-03T19:32:53.044 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_hl.ko 2022-04-03T19:32:53.045 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_ipcomp.ko 2022-04-03T19:32:53.045 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_iprange.ko 2022-04-03T19:32:53.045 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_ipvs.ko 2022-04-03T19:32:53.045 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_l2tp.ko 2022-04-03T19:32:53.045 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_length.ko 2022-04-03T19:32:53.046 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_limit.ko 2022-04-03T19:32:53.046 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_mac.ko 2022-04-03T19:32:53.046 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_mark.ko 2022-04-03T19:32:53.046 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_multiport.ko 2022-04-03T19:32:53.046 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_nat.ko 2022-04-03T19:32:53.047 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_nfacct.ko 2022-04-03T19:32:53.047 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_osf.ko 2022-04-03T19:32:53.047 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_owner.ko 2022-04-03T19:32:53.047 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_physdev.ko 2022-04-03T19:32:53.047 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_pkttype.ko 2022-04-03T19:32:53.048 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_policy.ko 2022-04-03T19:32:53.048 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_quota.ko 2022-04-03T19:32:53.048 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_rateest.ko 2022-04-03T19:32:53.048 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_realm.ko 2022-04-03T19:32:53.049 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_recent.ko 2022-04-03T19:32:53.049 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_sctp.ko 2022-04-03T19:32:53.049 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_set.ko 2022-04-03T19:32:53.049 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_socket.ko 2022-04-03T19:32:53.049 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_state.ko 2022-04-03T19:32:53.049 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_statistic.ko 2022-04-03T19:32:53.050 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_string.ko 2022-04-03T19:32:53.050 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_tcpmss.ko 2022-04-03T19:32:53.050 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_time.ko 2022-04-03T19:32:53.050 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netfilter/xt_u32.ko 2022-04-03T19:32:53.051 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netlink 2022-04-03T19:32:53.051 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/netlink/netlink_diag.ko 2022-04-03T19:32:53.051 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/nsh 2022-04-03T19:32:53.051 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/nsh/nsh.ko 2022-04-03T19:32:53.051 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/openvswitch 2022-04-03T19:32:53.052 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/openvswitch/openvswitch.ko 2022-04-03T19:32:53.052 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/openvswitch/vport-geneve.ko 2022-04-03T19:32:53.052 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/openvswitch/vport-gre.ko 2022-04-03T19:32:53.052 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/openvswitch/vport-vxlan.ko 2022-04-03T19:32:53.052 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/packet 2022-04-03T19:32:53.053 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/packet/af_packet_diag.ko 2022-04-03T19:32:53.053 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/rds 2022-04-03T19:32:53.053 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/rds/rds.ko 2022-04-03T19:32:53.053 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/rds/rds_rdma.ko 2022-04-03T19:32:53.053 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/rds/rds_tcp.ko 2022-04-03T19:32:53.054 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched 2022-04-03T19:32:53.054 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/act_csum.ko 2022-04-03T19:32:53.054 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/act_gact.ko 2022-04-03T19:32:53.054 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/act_ipt.ko 2022-04-03T19:32:53.054 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/act_mirred.ko 2022-04-03T19:32:53.055 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/act_nat.ko 2022-04-03T19:32:53.055 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/act_pedit.ko 2022-04-03T19:32:53.055 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/act_police.ko 2022-04-03T19:32:53.055 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/act_simple.ko 2022-04-03T19:32:53.055 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/act_skbedit.ko 2022-04-03T19:32:53.056 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/act_vlan.ko 2022-04-03T19:32:53.056 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/cls_basic.ko 2022-04-03T19:32:53.056 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/cls_bpf.ko 2022-04-03T19:32:53.056 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/cls_flow.ko 2022-04-03T19:32:53.056 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/cls_fw.ko 2022-04-03T19:32:53.057 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/cls_route.ko 2022-04-03T19:32:53.057 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/cls_rsvp.ko 2022-04-03T19:32:53.057 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/cls_rsvp6.ko 2022-04-03T19:32:53.057 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/cls_tcindex.ko 2022-04-03T19:32:53.057 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/cls_u32.ko 2022-04-03T19:32:53.058 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/em_cmp.ko 2022-04-03T19:32:53.058 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/em_ipset.ko 2022-04-03T19:32:53.058 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/em_meta.ko 2022-04-03T19:32:53.058 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/em_nbyte.ko 2022-04-03T19:32:53.058 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/em_text.ko 2022-04-03T19:32:53.059 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/em_u32.ko 2022-04-03T19:32:53.059 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_cbq.ko 2022-04-03T19:32:53.059 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_choke.ko 2022-04-03T19:32:53.059 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_codel.ko 2022-04-03T19:32:53.059 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_drr.ko 2022-04-03T19:32:53.060 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_dsmark.ko 2022-04-03T19:32:53.060 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_fq.ko 2022-04-03T19:32:53.060 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_gred.ko 2022-04-03T19:32:53.060 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_hfsc.ko 2022-04-03T19:32:53.060 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_hhf.ko 2022-04-03T19:32:53.061 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_htb.ko 2022-04-03T19:32:53.061 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_ingress.ko 2022-04-03T19:32:53.061 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_mqprio.ko 2022-04-03T19:32:53.061 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_multiq.ko 2022-04-03T19:32:53.062 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_netem.ko 2022-04-03T19:32:53.062 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_pie.ko 2022-04-03T19:32:53.062 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_plug.ko 2022-04-03T19:32:53.062 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_prio.ko 2022-04-03T19:32:53.062 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_qfq.ko 2022-04-03T19:32:53.063 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_red.ko 2022-04-03T19:32:53.063 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_sfb.ko 2022-04-03T19:32:53.063 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_sfq.ko 2022-04-03T19:32:53.063 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_tbf.ko 2022-04-03T19:32:53.063 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sched/sch_teql.ko 2022-04-03T19:32:53.064 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sctp 2022-04-03T19:32:53.064 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sctp/sctp.ko 2022-04-03T19:32:53.064 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sctp/sctp_diag.ko 2022-04-03T19:32:53.064 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sunrpc 2022-04-03T19:32:53.064 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sunrpc/auth_gss 2022-04-03T19:32:53.065 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2022-04-03T19:32:53.065 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2022-04-03T19:32:53.065 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sunrpc/sunrpc.ko 2022-04-03T19:32:53.065 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sunrpc/xprtrdma 2022-04-03T19:32:53.066 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2022-04-03T19:32:53.066 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/tipc 2022-04-03T19:32:53.066 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/tipc/diag.ko 2022-04-03T19:32:53.066 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/tipc/tipc.ko 2022-04-03T19:32:53.066 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/unix 2022-04-03T19:32:53.067 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/unix/unix_diag.ko 2022-04-03T19:32:53.067 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/vmw_vsock 2022-04-03T19:32:53.067 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2022-04-03T19:32:53.067 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2022-04-03T19:32:53.067 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/vmw_vsock/vsock.ko 2022-04-03T19:32:53.068 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/vmw_vsock/vsock_diag.ko 2022-04-03T19:32:53.068 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/vmw_vsock/vsock_loopback.ko 2022-04-03T19:32:53.068 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/xfrm 2022-04-03T19:32:53.068 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/net/xfrm/xfrm_ipcomp.ko 2022-04-03T19:32:53.068 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/security 2022-04-03T19:32:53.069 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/security/keys 2022-04-03T19:32:53.069 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/security/keys/encrypted-keys 2022-04-03T19:32:53.069 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2022-04-03T19:32:53.069 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/security/keys/trusted-keys 2022-04-03T19:32:53.070 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/security/keys/trusted-keys/trusted.ko 2022-04-03T19:32:53.070 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/virt 2022-04-03T19:32:53.070 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/virt/lib 2022-04-03T19:32:53.070 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/kernel/virt/lib/irqbypass.ko 2022-04-03T19:32:53.070 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/modules.alias 2022-04-03T19:32:53.071 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/modules.alias.bin 2022-04-03T19:32:53.071 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/modules.builtin 2022-04-03T19:32:53.071 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/modules.builtin.bin 2022-04-03T19:32:53.071 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/modules.builtin.modinfo 2022-04-03T19:32:53.071 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/modules.dep 2022-04-03T19:32:53.072 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/modules.dep.bin 2022-04-03T19:32:53.072 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/modules.devname 2022-04-03T19:32:53.072 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/modules.order 2022-04-03T19:32:53.072 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/modules.softdep 2022-04-03T19:32:53.072 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/modules.symbols 2022-04-03T19:32:53.073 INFO:teuthology.orchestra.run.smithi077.stdout:/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/modules.symbols.bin 2022-04-03T19:32:53.073 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/lib/.build-id 2022-04-03T19:32:53.073 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/lib/.build-id/1c 2022-04-03T19:32:53.073 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/lib/.build-id/1c/d444cc390db5679c4430ab1d870001eba1d44d 2022-04-03T19:32:53.074 DEBUG:teuthology.task.kernel:get_image_version: 5.17.0-rc6-ceph-g26d8230c8f91 2022-04-03T19:32:53.075 DEBUG:teuthology.orchestra.run.smithi077:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2022-04-03T19:32:53.099 INFO:teuthology.orchestra.run.smithi077.stdout:postinstall scriptlet (using /bin/sh): 2022-04-03T19:32:53.099 INFO:teuthology.orchestra.run.smithi077.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-5.17.0-rc6-ceph-g26d8230c8f91 -a -r /boot/System.map-5.17.0-rc6-ceph-g26d8230c8f91 ]; then 2022-04-03T19:32:53.099 INFO:teuthology.orchestra.run.smithi077.stdout:cp /boot/vmlinuz-5.17.0-rc6-ceph-g26d8230c8f91 /boot/.vmlinuz-5.17.0-rc6-ceph-g26d8230c8f91-rpm 2022-04-03T19:32:53.099 INFO:teuthology.orchestra.run.smithi077.stdout:cp /boot/System.map-5.17.0-rc6-ceph-g26d8230c8f91 /boot/.System.map-5.17.0-rc6-ceph-g26d8230c8f91-rpm 2022-04-03T19:32:53.100 INFO:teuthology.orchestra.run.smithi077.stdout:rm -f /boot/vmlinuz-5.17.0-rc6-ceph-g26d8230c8f91 /boot/System.map-5.17.0-rc6-ceph-g26d8230c8f91 2022-04-03T19:32:53.100 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/installkernel 5.17.0-rc6-ceph-g26d8230c8f91 /boot/.vmlinuz-5.17.0-rc6-ceph-g26d8230c8f91-rpm /boot/.System.map-5.17.0-rc6-ceph-g26d8230c8f91-rpm 2022-04-03T19:32:53.100 INFO:teuthology.orchestra.run.smithi077.stdout:rm -f /boot/.vmlinuz-5.17.0-rc6-ceph-g26d8230c8f91-rpm /boot/.System.map-5.17.0-rc6-ceph-g26d8230c8f91-rpm 2022-04-03T19:32:53.100 INFO:teuthology.orchestra.run.smithi077.stdout:fi 2022-04-03T19:32:53.100 INFO:teuthology.orchestra.run.smithi077.stdout:preuninstall scriptlet (using /bin/sh): 2022-04-03T19:32:53.101 INFO:teuthology.orchestra.run.smithi077.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2022-04-03T19:32:53.101 INFO:teuthology.orchestra.run.smithi077.stdout:new-kernel-pkg --remove 5.17.0-rc6-ceph-g26d8230c8f91 --rminitrd --initrdfile=/boot/initramfs-5.17.0-rc6-ceph-g26d8230c8f91.img 2022-04-03T19:32:53.101 INFO:teuthology.orchestra.run.smithi077.stdout:elif [ -x /usr/bin/kernel-install ]; then 2022-04-03T19:32:53.101 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-install remove 5.17.0-rc6-ceph-g26d8230c8f91 2022-04-03T19:32:53.101 INFO:teuthology.orchestra.run.smithi077.stdout:fi 2022-04-03T19:32:53.102 INFO:teuthology.orchestra.run.smithi077.stdout:postuninstall scriptlet (using /bin/sh): 2022-04-03T19:32:53.102 INFO:teuthology.orchestra.run.smithi077.stdout:if [ -x /sbin/update-bootloader ]; then 2022-04-03T19:32:53.102 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/update-bootloader --remove 5.17.0-rc6-ceph-g26d8230c8f91 2022-04-03T19:32:53.102 INFO:teuthology.orchestra.run.smithi077.stdout:fi 2022-04-03T19:32:53.103 DEBUG:teuthology.orchestra.run.smithi077:> sudo rpm -qi grub2-tools 2022-04-03T19:32:53.177 INFO:teuthology.orchestra.run.smithi077.stdout:Name : grub2-tools 2022-04-03T19:32:53.177 INFO:teuthology.orchestra.run.smithi077.stdout:Epoch : 1 2022-04-03T19:32:53.177 INFO:teuthology.orchestra.run.smithi077.stdout:Version : 2.02 2022-04-03T19:32:53.178 INFO:teuthology.orchestra.run.smithi077.stdout:Release : 99.el8 2022-04-03T19:32:53.178 INFO:teuthology.orchestra.run.smithi077.stdout:Architecture: x86_64 2022-04-03T19:32:53.178 INFO:teuthology.orchestra.run.smithi077.stdout:Install Date: Sun 03 Apr 2022 12:11:59 PM UTC 2022-04-03T19:32:53.178 INFO:teuthology.orchestra.run.smithi077.stdout:Group : System Environment/Base 2022-04-03T19:32:53.179 INFO:teuthology.orchestra.run.smithi077.stdout:Size : 9264713 2022-04-03T19:32:53.179 INFO:teuthology.orchestra.run.smithi077.stdout:License : GPLv3+ 2022-04-03T19:32:53.179 INFO:teuthology.orchestra.run.smithi077.stdout:Signature : RSA/SHA256, Tue 02 Mar 2021 08:11:27 PM UTC, Key ID 199e2f91fd431d51 2022-04-03T19:32:53.179 INFO:teuthology.orchestra.run.smithi077.stdout:Source RPM : grub2-2.02-99.el8.src.rpm 2022-04-03T19:32:53.179 INFO:teuthology.orchestra.run.smithi077.stdout:Build Date : Thu 25 Feb 2021 04:59:31 PM UTC 2022-04-03T19:32:53.180 INFO:teuthology.orchestra.run.smithi077.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2022-04-03T19:32:53.180 INFO:teuthology.orchestra.run.smithi077.stdout:Relocations : (not relocatable) 2022-04-03T19:32:53.180 INFO:teuthology.orchestra.run.smithi077.stdout:Packager : Red Hat, Inc. 2022-04-03T19:32:53.180 INFO:teuthology.orchestra.run.smithi077.stdout:Vendor : Red Hat, Inc. 2022-04-03T19:32:53.181 INFO:teuthology.orchestra.run.smithi077.stdout:URL : http://www.gnu.org/software/grub/ 2022-04-03T19:32:53.181 INFO:teuthology.orchestra.run.smithi077.stdout:Summary : Support tools for GRUB. 2022-04-03T19:32:53.181 INFO:teuthology.orchestra.run.smithi077.stdout:Description : 2022-04-03T19:32:53.181 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:32:53.181 INFO:teuthology.orchestra.run.smithi077.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2022-04-03T19:32:53.182 INFO:teuthology.orchestra.run.smithi077.stdout:customizable bootloader with modular architecture. It supports a rich 2022-04-03T19:32:53.182 INFO:teuthology.orchestra.run.smithi077.stdout:variety of kernel formats, file systems, computer architectures and 2022-04-03T19:32:53.182 INFO:teuthology.orchestra.run.smithi077.stdout:hardware devices. 2022-04-03T19:32:53.182 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:32:53.183 INFO:teuthology.orchestra.run.smithi077.stdout:This subpackage provides tools for support of all platforms. 2022-04-03T19:32:53.183 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2022-04-03T19:32:53.184 INFO:teuthology.task.kernel:Updating grub on smithi077 to boot 5.17.0-rc6-ceph-g26d8230c8f91 2022-04-03T19:32:53.184 DEBUG:teuthology.orchestra.run.smithi077:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2022-04-03T19:32:56.826 INFO:teuthology.orchestra.run.smithi077.stderr:Generating grub configuration file ... 2022-04-03T19:32:58.966 INFO:teuthology.orchestra.run.smithi077.stderr:done 2022-04-03T19:32:58.968 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2022-04-03T19:32:58.983 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.K5XUINuigv 2022-04-03T19:32:58.983 DEBUG:teuthology.orchestra.run.smithi077:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.K5XUINuigv 2022-04-03T19:32:59.066 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.K5XUINuigv 2022-04-03T19:32:59.244 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.K5XUINuigv is 6KB 2022-04-03T19:32:59.293 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.K5XUINuigv 2022-04-03T19:32:59.307 DEBUG:teuthology.orchestra.run.smithi077:> sudo /bin/ls /boot/loader/entries || true 2022-04-03T19:32:59.375 INFO:teuthology.orchestra.run.smithi077.stdout:545941180e054f2c980bf1100aaa9165-0-rescue.conf 2022-04-03T19:32:59.375 INFO:teuthology.orchestra.run.smithi077.stdout:545941180e054f2c980bf1100aaa9165-4.18.0-305.el8.x86_64.conf 2022-04-03T19:32:59.375 INFO:teuthology.orchestra.run.smithi077.stdout:545941180e054f2c980bf1100aaa9165-4.18.0-348.20.1.el8_5.x86_64.conf 2022-04-03T19:32:59.376 INFO:teuthology.orchestra.run.smithi077.stdout:545941180e054f2c980bf1100aaa9165-5.17.0-rc6-ceph-g26d8230c8f91.conf 2022-04-03T19:32:59.377 DEBUG:teuthology.orchestra.run.smithi077:> sudo grub2-set-default 545941180e054f2c980bf1100aaa9165-5.17.0-rc6-ceph-g26d8230c8f91 2022-04-03T19:32:59.552 DEBUG:teuthology.orchestra.run.smithi077:> sudo shutdown -r now 2022-04-03T19:33:29.555 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2022-04-03T19:33:29.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2022-04-03T19:33:29.557 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-03T19:33:48.107 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2022-04-03T19:34:18.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2022-04-03T19:34:18.110 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-03T19:34:36.495 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2022-04-03T19:35:06.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2022-04-03T19:35:06.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-03T19:35:06.904 DEBUG:teuthology.orchestra.run.smithi077:> true 2022-04-03T19:35:07.264 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2022-04-03T19:35:07.265 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.17.0-rc6-ceph-g26d8230c8f91"... 2022-04-03T19:35:07.265 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2022-04-03T19:35:07.309 INFO:teuthology.orchestra.run.smithi077.stdout:5.17.0-rc6-ceph-g26d8230c8f91 2022-04-03T19:35:07.310 DEBUG:teuthology.task.kernel:current kernel version is 5.17.0-rc6-ceph-g26d8230c8f91 vs 5.17.0-rc6-ceph-g26d8230c8f91 2022-04-03T19:35:07.310 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-04-03T19:35:07.310 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2022-04-03T19:35:08.312 DEBUG:teuthology.parallel:result is None 2022-04-03T19:35:08.312 INFO:teuthology.run_tasks:Running task internal.base... 2022-04-03T19:35:08.324 INFO:teuthology.task.internal:Creating test directory... 2022-04-03T19:35:08.325 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-04-03T19:35:08.330 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-04-03T19:35:08.373 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-04-03T19:35:08.383 INFO:teuthology.run_tasks:Running task internal.archive... 2022-04-03T19:35:08.471 INFO:teuthology.task.internal:Creating archive directory... 2022-04-03T19:35:08.472 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-04-03T19:35:08.477 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-04-03T19:35:08.534 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-04-03T19:35:08.571 INFO:teuthology.task.internal:Enabling coredump saving... 2022-04-03T19:35:08.572 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2022-04-03T19:35:08.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-03T19:35:08.589 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-04-03T19:35:08.593 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-04-03T19:35:08.663 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-03T19:35:08.684 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-03T19:35:08.755 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-03T19:35:08.791 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-04-03T19:35:08.796 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-04-03T19:35:08.807 INFO:teuthology.task.internal:Configuring sudo... 2022-04-03T19:35:08.807 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-04-03T19:35:08.812 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-04-03T19:35:08.879 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-04-03T19:35:08.889 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-04-03T19:35:08.889 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-04-03T19:35:08.894 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-04-03T19:35:08.933 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-04-03T19:35:09.047 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-04-03T19:35:09.129 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-04-03T19:35:09.201 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-04-03T19:35:09.286 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:35:09.287 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-04-03T19:35:09.358 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-04-03T19:35:09.399 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-04-03T19:35:09.452 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-04-03T19:35:09.533 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-04-03T19:35:09.576 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:35:09.577 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-04-03T19:35:09.643 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2022-04-03T19:35:09.648 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2022-04-03T19:35:09.745 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-04-03T19:35:09.771 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-04-03T19:35:10.233 INFO:teuthology.run_tasks:Running task internal.timer... 2022-04-03T19:35:10.243 INFO:teuthology.task.internal:Starting timer... 2022-04-03T19:35:10.243 INFO:teuthology.run_tasks:Running task pcp... 2022-04-03T19:35:10.282 INFO:teuthology.run_tasks:Running task selinux... 2022-04-03T19:35:10.380 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2022-04-03T19:35:10.381 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2022-04-03T19:35:10.522 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: [ OK ] 2022-04-03T19:35:10.527 DEBUG:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2022-04-03T19:35:10.585 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: [ OK ] 2022-04-03T19:35:10.588 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-04-03T19:35:10.588 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2022-04-03T19:35:10.648 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2022-04-03T19:35:10.649 DEBUG:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2022-04-03T19:35:10.683 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2022-04-03T19:35:10.684 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi077.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi186.front.sepia.ceph.com': 'permissive'} 2022-04-03T19:35:10.684 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2022-04-03T19:35:10.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-03T19:35:10.756 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2022-04-03T19:35:10.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-03T19:35:10.781 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-04-03T19:35:10.782 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2022-04-03T19:35:10.852 DEBUG:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2022-04-03T19:35:10.886 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-04-03T19:35:10.896 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2022-04-03T19:35:10.897 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2022-04-03T19:35:10.915 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-04-03T19:35:10.916 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi077.front.sepia.ceph.com,smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2022-04-03T19:40:34.655 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi077.front.sepia.ceph.com'), Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2022-04-03T19:40:34.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2022-04-03T19:40:34.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-03T19:40:34.754 DEBUG:teuthology.orchestra.run.smithi077:> true 2022-04-03T19:40:34.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2022-04-03T19:40:34.858 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-04-03T19:40:34.926 DEBUG:teuthology.orchestra.run.smithi186:> true 2022-04-03T19:40:34.994 INFO:teuthology.run_tasks:Running task clock... 2022-04-03T19:40:35.008 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-04-03T19:40:35.009 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-04-03T19:40:35.011 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-04-03T19:40:35.015 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-04-03T19:40:35.016 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-04-03T19:40:35.063 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-04-03T19:40:35.078 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-04-03T19:40:35.086 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-04-03T19:40:35.117 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: ntpd: command not found 2022-04-03T19:40:35.123 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-04-03T19:40:35.127 INFO:teuthology.orchestra.run.smithi186.stdout:506 Cannot talk to daemon 2022-04-03T19:40:35.141 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-04-03T19:40:35.155 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-04-03T19:40:35.189 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2022-04-03T19:40:35.217 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2022-04-03T19:40:35.223 INFO:teuthology.orchestra.run.smithi186.stderr:bash: ntpq: command not found 2022-04-03T19:40:35.226 INFO:teuthology.orchestra.run.smithi186.stdout:210 Number of sources = 4 2022-04-03T19:40:35.226 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-04-03T19:40:35.227 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2022-04-03T19:40:35.227 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-04-03T19:40:35.227 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-04-03T19:40:35.227 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-04-03T19:40:35.227 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-04-03T19:40:35.254 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-04-03T19:40:35.292 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-04-03T19:40:35.374 INFO:teuthology.orchestra.run.smithi077.stderr:bash: ntpq: command not found 2022-04-03T19:40:35.382 INFO:teuthology.orchestra.run.smithi077.stdout:210 Number of sources = 4 2022-04-03T19:40:35.383 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-04-03T19:40:35.383 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2022-04-03T19:40:35.383 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-04-03T19:40:35.383 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-04-03T19:40:35.383 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-04-03T19:40:35.384 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-04-03T19:40:35.384 INFO:teuthology.run_tasks:Running task install... 2022-04-03T19:40:35.396 DEBUG:teuthology.task.install:project ceph 2022-04-03T19:40:35.397 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'cfb8f943163b374162da0d7b0240f267dd46e4e1'}} 2022-04-03T19:40:35.397 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': 'cfb8f943163b374162da0d7b0240f267dd46e4e1'} 2022-04-03T19:40:35.398 INFO:teuthology.task.install:Using flavor: default 2022-04-03T19:40:35.408 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-04-03T19:40:35.408 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2022-04-03T19:40:35.409 DEBUG:teuthology.orchestra.run.smithi077:> sudo lsb_release -is 2022-04-03T19:40:35.531 INFO:teuthology.orchestra.run.smithi077.stdout:RedHatEnterprise 2022-04-03T19:40:35.535 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-04-03T19:40:35.536 DEBUG:teuthology.orchestra.run.smithi186:> sudo lsb_release -is 2022-04-03T19:40:35.536 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cfb8f943163b374162da0d7b0240f267dd46e4e1', 'tag': None, 'wait_for_package': False} 2022-04-03T19:40:35.538 DEBUG:teuthology.orchestra.run.smithi077:> sudo lsb_release -is 2022-04-03T19:40:35.578 INFO:teuthology.orchestra.run.smithi186.stdout:RedHatEnterprise 2022-04-03T19:40:35.579 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-04-03T19:40:35.580 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cfb8f943163b374162da0d7b0240f267dd46e4e1', 'tag': None, 'wait_for_package': False} 2022-04-03T19:40:35.580 DEBUG:teuthology.orchestra.run.smithi186:> sudo lsb_release -is 2022-04-03T19:40:35.657 INFO:teuthology.orchestra.run.smithi186.stdout:RedHatEnterprise 2022-04-03T19:40:35.658 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-04-03T19:40:35.659 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=cfb8f943163b374162da0d7b0240f267dd46e4e1 2022-04-03T19:40:35.664 INFO:teuthology.orchestra.run.smithi077.stdout:RedHatEnterprise 2022-04-03T19:40:35.664 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-04-03T19:40:35.665 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=cfb8f943163b374162da0d7b0240f267dd46e4e1 2022-04-03T19:40:35.818 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220403-170344/cfb8f943163b374162da0d7b0240f267dd46e4e1/centos/8/flavors/default/ 2022-04-03T19:40:35.818 INFO:teuthology.task.install.rpm:Package version is 17.0.0-11442.gcfb8f943 2022-04-03T19:40:35.823 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220403-170344/cfb8f943163b374162da0d7b0240f267dd46e4e1/centos/8/flavors/default/ 2022-04-03T19:40:35.823 INFO:teuthology.task.install.rpm:Package version is 17.0.0-11442.gcfb8f943 2022-04-03T19:40:35.940 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220403-170344/cfb8f943163b374162da0d7b0240f267dd46e4e1/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220403-170344/cfb8f943163b374162da0d7b0240f267dd46e4e1/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220403-170344/cfb8f943163b374162da0d7b0240f267dd46e4e1/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-04-03T19:40:35.940 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:40:35.941 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-04-03T19:40:35.962 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220403-170344/cfb8f943163b374162da0d7b0240f267dd46e4e1/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220403-170344/cfb8f943163b374162da0d7b0240f267dd46e4e1/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing-20220403-170344/cfb8f943163b374162da0d7b0240f267dd46e4e1/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-04-03T19:40:35.963 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:40:35.963 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-04-03T19:40:35.964 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2022-04-03T19:40:35.965 DEBUG:teuthology.orchestra.run.smithi186:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cfb8f943163b374162da0d7b0240f267dd46e4e1/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-04-03T19:40:36.030 DEBUG:teuthology.orchestra.run.smithi186:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-04-03T19:40:36.035 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2022-04-03T19:40:36.036 DEBUG:teuthology.orchestra.run.smithi077:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cfb8f943163b374162da0d7b0240f267dd46e4e1/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-04-03T19:40:36.104 DEBUG:teuthology.orchestra.run.smithi186:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-04-03T19:40:36.147 DEBUG:teuthology.orchestra.run.smithi077:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-04-03T19:40:36.168 INFO:teuthology.orchestra.run.smithi186.stdout:check_obsoletes = 1 2022-04-03T19:40:36.169 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum clean all 2022-04-03T19:40:36.258 DEBUG:teuthology.orchestra.run.smithi077:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-04-03T19:40:36.327 INFO:teuthology.orchestra.run.smithi077.stdout:check_obsoletes = 1 2022-04-03T19:40:36.332 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum clean all 2022-04-03T19:40:36.525 INFO:teuthology.orchestra.run.smithi186.stdout:Updating Subscription Management repositories. 2022-04-03T19:40:36.776 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2022-04-03T19:40:36.836 INFO:teuthology.orchestra.run.smithi186.stdout:54 files removed 2022-04-03T19:40:36.881 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2022-04-03T19:40:37.194 INFO:teuthology.orchestra.run.smithi077.stdout:54 files removed 2022-04-03T19:40:37.195 INFO:teuthology.orchestra.run.smithi186.stdout:Updating Subscription Management repositories. 2022-04-03T19:40:37.255 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2022-04-03T19:40:37.696 INFO:teuthology.orchestra.run.smithi077.stdout:Updating Subscription Management repositories. 2022-04-03T19:40:37.891 INFO:teuthology.orchestra.run.smithi186.stdout:ceph packages for x86_64 260 kB/s | 62 kB 00:00 2022-04-03T19:40:38.122 INFO:teuthology.orchestra.run.smithi186.stdout:ceph noarch packages 46 kB/s | 9.1 kB 00:00 2022-04-03T19:40:38.345 INFO:teuthology.orchestra.run.smithi186.stdout:ceph source packages 7.1 kB/s | 1.4 kB 00:00 2022-04-03T19:40:38.431 INFO:teuthology.orchestra.run.smithi077.stdout:ceph packages for x86_64 258 kB/s | 62 kB 00:00 2022-04-03T19:40:38.666 INFO:teuthology.orchestra.run.smithi077.stdout:ceph noarch packages 46 kB/s | 9.1 kB 00:00 2022-04-03T19:40:38.893 INFO:teuthology.orchestra.run.smithi077.stdout:ceph source packages 7.1 kB/s | 1.4 kB 00:00 2022-04-03T19:40:40.552 INFO:teuthology.orchestra.run.smithi077.stdout:Copr repo for python3-asyncssh owned by ceph 2.1 kB/s | 3.5 kB 00:01 2022-04-03T19:40:40.563 INFO:teuthology.orchestra.run.smithi186.stdout:Copr repo for python3-asyncssh owned by ceph 1.6 kB/s | 3.5 kB 00:02 2022-04-03T19:40:41.328 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 15 MB/s | 11 MB 00:00 2022-04-03T19:40:41.337 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 15 MB/s | 11 MB 00:00 2022-04-03T19:40:43.643 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 3.9 MB/s | 24 kB 00:00 2022-04-03T19:40:43.856 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 3.5 MB/s | 24 kB 00:00 2022-04-03T19:40:43.970 INFO:teuthology.orchestra.run.smithi186.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 126 MB/s | 33 MB 00:00 2022-04-03T19:40:44.365 INFO:teuthology.orchestra.run.smithi077.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 86 MB/s | 38 MB 00:00 2022-04-03T19:40:49.120 INFO:teuthology.orchestra.run.smithi186.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 133 MB/s | 38 MB 00:00 2022-04-03T19:40:50.259 INFO:teuthology.orchestra.run.smithi077.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 80 MB/s | 33 MB 00:00 2022-04-03T19:40:57.421 INFO:teuthology.orchestra.run.smithi186.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-04-03T19:40:57.422 INFO:teuthology.orchestra.run.smithi186.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-04-03T19:40:57.433 INFO:teuthology.orchestra.run.smithi186.stdout:Package NetworkManager-1:1.30.0-7.el8.x86_64 is already installed. 2022-04-03T19:40:57.434 INFO:teuthology.orchestra.run.smithi186.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2022-04-03T19:40:57.435 INFO:teuthology.orchestra.run.smithi186.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2022-04-03T19:40:57.435 INFO:teuthology.orchestra.run.smithi186.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-04-03T19:40:57.437 INFO:teuthology.orchestra.run.smithi186.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-04-03T19:40:57.440 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2022-04-03T19:40:57.440 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2022-04-03T19:40:57.904 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2022-04-03T19:40:57.916 INFO:teuthology.orchestra.run.smithi186.stdout:====================================================================================================================== 2022-04-03T19:40:57.917 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2022-04-03T19:40:57.917 INFO:teuthology.orchestra.run.smithi186.stdout:====================================================================================================================== 2022-04-03T19:40:57.917 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2022-04-03T19:40:57.917 INFO:teuthology.orchestra.run.smithi186.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-04-03T19:40:57.918 INFO:teuthology.orchestra.run.smithi186.stdout: ceph x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 6.5 k 2022-04-03T19:40:57.918 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 5.3 M 2022-04-03T19:40:57.918 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 880 k 2022-04-03T19:40:57.918 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 169 k 2022-04-03T19:40:57.919 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 1.5 M 2022-04-03T19:40:57.919 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm noarch 2:17.0.0-11442.gcfb8f943.el8 ceph-noarch 105 k 2022-04-03T19:40:57.919 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard noarch 2:17.0.0-11442.gcfb8f943.el8 ceph-noarch 1.6 M 2022-04-03T19:40:57.919 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-11442.gcfb8f943.el8 ceph-noarch 7.4 M 2022-04-03T19:40:57.920 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook noarch 2:17.0.0-11442.gcfb8f943.el8 ceph-noarch 47 k 2022-04-03T19:40:57.920 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 16 M 2022-04-03T19:40:57.920 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 52 M 2022-04-03T19:40:57.920 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume noarch 2:17.0.0-11442.gcfb8f943.el8 ceph-noarch 257 k 2022-04-03T19:40:57.921 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm noarch 2:17.0.0-11442.gcfb8f943.el8 ceph-noarch 71 k 2022-04-03T19:40:57.921 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 273 k 2022-04-03T19:40:57.921 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top noarch 2:17.0.0-11442.gcfb8f943.el8 ceph-noarch 14 k 2022-04-03T19:40:57.921 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2022-04-03T19:40:57.922 INFO:teuthology.orchestra.run.smithi186.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-04-03T19:40:57.922 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-04-03T19:40:57.922 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 25 k 2022-04-03T19:40:57.922 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2 x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 795 k 2022-04-03T19:40:57.923 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 125 k 2022-04-03T19:40:57.923 INFO:teuthology.orchestra.run.smithi186.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-04-03T19:40:57.923 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2022-04-03T19:40:57.923 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-04-03T19:40:57.923 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 179 k 2022-04-03T19:40:57.924 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 367 k 2022-04-03T19:40:57.924 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 345 k 2022-04-03T19:40:57.925 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 109 k 2022-04-03T19:40:57.925 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 94 k 2022-04-03T19:40:57.925 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 3.8 M 2022-04-03T19:40:57.925 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 175 k 2022-04-03T19:40:57.925 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2022-04-03T19:40:57.926 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrading: 2022-04-03T19:40:57.926 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 2.6 M 2022-04-03T19:40:57.926 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-libnm x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 1.8 M 2022-04-03T19:40:57.926 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-team x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 146 k 2022-04-03T19:40:57.927 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-tui x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 329 k 2022-04-03T19:40:57.927 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 3.4 M 2022-04-03T19:40:57.927 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1 x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 3.9 M 2022-04-03T19:40:57.927 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2022-04-03T19:40:57.927 INFO:teuthology.orchestra.run.smithi186.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-04-03T19:40:57.928 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 23 M 2022-04-03T19:40:57.928 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards noarch 2:17.0.0-11442.gcfb8f943.el8 ceph-noarch 23 k 2022-04-03T19:40:57.928 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 2.3 M 2022-04-03T19:40:57.928 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core noarch 2:17.0.0-11442.gcfb8f943.el8 ceph-noarch 227 k 2022-04-03T19:40:57.929 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 3.9 M 2022-04-03T19:40:57.929 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 17 M 2022-04-03T19:40:57.929 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts noarch 2:17.0.0-11442.gcfb8f943.el8 ceph-noarch 15 k 2022-04-03T19:40:57.929 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 24 k 2022-04-03T19:40:57.929 INFO:teuthology.orchestra.run.smithi186.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-04-03T19:40:57.930 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-04-03T19:40:57.930 INFO:teuthology.orchestra.run.smithi186.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2022-04-03T19:40:57.930 INFO:teuthology.orchestra.run.smithi186.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-04-03T19:40:57.930 INFO:teuthology.orchestra.run.smithi186.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2022-04-03T19:40:57.931 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 170 k 2022-04-03T19:40:57.931 INFO:teuthology.orchestra.run.smithi186.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2022-04-03T19:40:57.931 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2022-04-03T19:40:57.931 INFO:teuthology.orchestra.run.smithi186.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-04-03T19:40:57.932 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2022-04-03T19:40:57.932 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-04-03T19:40:57.932 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1 x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 484 k 2022-04-03T19:40:57.932 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2022-04-03T19:40:57.932 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2 x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 5.3 M 2022-04-03T19:40:57.933 INFO:teuthology.orchestra.run.smithi186.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-04-03T19:40:57.933 INFO:teuthology.orchestra.run.smithi186.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2022-04-03T19:40:57.933 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-04-03T19:40:57.933 INFO:teuthology.orchestra.run.smithi186.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-04-03T19:40:57.934 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-04-03T19:40:57.934 INFO:teuthology.orchestra.run.smithi186.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-04-03T19:40:57.934 INFO:teuthology.orchestra.run.smithi186.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2022-04-03T19:40:57.934 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-04-03T19:40:57.934 INFO:teuthology.orchestra.run.smithi186.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-04-03T19:40:57.935 INFO:teuthology.orchestra.run.smithi186.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-04-03T19:40:57.935 INFO:teuthology.orchestra.run.smithi186.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-04-03T19:40:57.935 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-04-03T19:40:57.936 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-04-03T19:40:57.936 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-04-03T19:40:57.936 INFO:teuthology.orchestra.run.smithi186.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-04-03T19:40:57.936 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-04-03T19:40:57.937 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 45 k 2022-04-03T19:40:57.937 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 105 k 2022-04-03T19:40:57.937 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-04-03T19:40:57.937 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-04-03T19:40:57.937 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-04-03T19:40:57.938 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-04-03T19:40:57.938 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography x86_64 3.2.1-4.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-04-03T19:40:57.938 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-04-03T19:40:57.938 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-04-03T19:40:57.939 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-04-03T19:40:57.939 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2022-04-03T19:40:57.939 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-04-03T19:40:57.939 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-04-03T19:40:57.939 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-04-03T19:40:57.940 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-04-03T19:40:57.940 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-04-03T19:40:57.940 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-04-03T19:40:57.940 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-04-03T19:40:57.941 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-04-03T19:40:57.941 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-04-03T19:40:57.941 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-04-03T19:40:57.941 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-04-03T19:40:57.942 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-04-03T19:40:57.942 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-04-03T19:40:57.942 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-04-03T19:40:57.942 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-04-03T19:40:57.943 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-04-03T19:40:57.943 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-04-03T19:40:57.943 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-04-03T19:40:57.943 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-04-03T19:40:57.943 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-04-03T19:40:57.944 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-04-03T19:40:57.944 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-04-03T19:40:57.944 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-04-03T19:40:57.945 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-04-03T19:40:57.945 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2022-04-03T19:40:57.945 INFO:teuthology.orchestra.run.smithi186.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-04-03T19:40:57.945 INFO:teuthology.orchestra.run.smithi186.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-04-03T19:40:57.946 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-04-03T19:40:57.946 INFO:teuthology.orchestra.run.smithi186.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-04-03T19:40:57.946 INFO:teuthology.orchestra.run.smithi186.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-04-03T19:40:57.946 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-04-03T19:40:57.946 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-04-03T19:40:57.947 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-04-03T19:40:57.947 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-04-03T19:40:57.947 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-04-03T19:40:57.947 INFO:teuthology.orchestra.run.smithi186.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2022-04-03T19:40:57.948 INFO:teuthology.orchestra.run.smithi186.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-04-03T19:40:57.948 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-04-03T19:40:57.948 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2022-04-03T19:40:57.948 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:40:57.948 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2022-04-03T19:40:57.949 INFO:teuthology.orchestra.run.smithi186.stdout:====================================================================================================================== 2022-04-03T19:40:57.949 INFO:teuthology.orchestra.run.smithi186.stdout:Install 122 Packages 2022-04-03T19:40:57.949 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrade 6 Packages 2022-04-03T19:40:57.949 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:40:57.950 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 207 M 2022-04-03T19:40:57.950 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2022-04-03T19:40:58.029 INFO:teuthology.orchestra.run.smithi186.stdout:(1/128): ceph-17.0.0-11442.gcfb8f943.el8.x86_64 66 kB/s | 6.5 kB 00:00 2022-04-03T19:40:58.182 INFO:teuthology.orchestra.run.smithi186.stdout:(2/128): ceph-fuse-17.0.0-11442.gcfb8f943.el8.x 5.6 MB/s | 880 kB 00:00 2022-04-03T19:40:58.229 INFO:teuthology.orchestra.run.smithi186.stdout:(3/128): ceph-immutable-object-cache-17.0.0-114 3.5 MB/s | 169 kB 00:00 2022-04-03T19:40:58.499 INFO:teuthology.orchestra.run.smithi186.stdout:(4/128): ceph-mds-17.0.0-11442.gcfb8f943.el8.x8 8.5 MB/s | 2.3 MB 00:00 2022-04-03T19:40:58.591 INFO:teuthology.orchestra.run.smithi186.stdout:(5/128): ceph-base-17.0.0-11442.gcfb8f943.el8.x 8.0 MB/s | 5.3 MB 00:00 2022-04-03T19:40:58.630 INFO:teuthology.orchestra.run.smithi077.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-04-03T19:40:58.631 INFO:teuthology.orchestra.run.smithi077.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-04-03T19:40:58.642 INFO:teuthology.orchestra.run.smithi077.stdout:Package NetworkManager-1:1.30.0-7.el8.x86_64 is already installed. 2022-04-03T19:40:58.643 INFO:teuthology.orchestra.run.smithi077.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2022-04-03T19:40:58.644 INFO:teuthology.orchestra.run.smithi077.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2022-04-03T19:40:58.645 INFO:teuthology.orchestra.run.smithi077.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-04-03T19:40:58.646 INFO:teuthology.orchestra.run.smithi077.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-04-03T19:40:58.649 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2022-04-03T19:40:58.650 INFO:teuthology.orchestra.run.smithi077.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2022-04-03T19:40:58.692 INFO:teuthology.orchestra.run.smithi186.stdout:(6/128): ceph-mgr-17.0.0-11442.gcfb8f943.el8.x8 8.0 MB/s | 1.5 MB 00:00 2022-04-03T19:40:58.971 INFO:teuthology.orchestra.run.smithi186.stdout:(7/128): ceph-mon-17.0.0-11442.gcfb8f943.el8.x8 10 MB/s | 3.9 MB 00:00 2022-04-03T19:40:59.137 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2022-04-03T19:40:59.149 INFO:teuthology.orchestra.run.smithi077.stdout:====================================================================================================================== 2022-04-03T19:40:59.149 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2022-04-03T19:40:59.150 INFO:teuthology.orchestra.run.smithi077.stdout:====================================================================================================================== 2022-04-03T19:40:59.150 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2022-04-03T19:40:59.150 INFO:teuthology.orchestra.run.smithi077.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-04-03T19:40:59.150 INFO:teuthology.orchestra.run.smithi077.stdout: ceph x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 6.5 k 2022-04-03T19:40:59.151 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 5.3 M 2022-04-03T19:40:59.151 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 880 k 2022-04-03T19:40:59.151 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 169 k 2022-04-03T19:40:59.151 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 1.5 M 2022-04-03T19:40:59.152 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm noarch 2:17.0.0-11442.gcfb8f943.el8 ceph-noarch 105 k 2022-04-03T19:40:59.152 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard noarch 2:17.0.0-11442.gcfb8f943.el8 ceph-noarch 1.6 M 2022-04-03T19:40:59.152 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-11442.gcfb8f943.el8 ceph-noarch 7.4 M 2022-04-03T19:40:59.153 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook noarch 2:17.0.0-11442.gcfb8f943.el8 ceph-noarch 47 k 2022-04-03T19:40:59.153 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 16 M 2022-04-03T19:40:59.153 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 52 M 2022-04-03T19:40:59.154 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume noarch 2:17.0.0-11442.gcfb8f943.el8 ceph-noarch 257 k 2022-04-03T19:40:59.154 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm noarch 2:17.0.0-11442.gcfb8f943.el8 ceph-noarch 71 k 2022-04-03T19:40:59.154 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 273 k 2022-04-03T19:40:59.154 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top noarch 2:17.0.0-11442.gcfb8f943.el8 ceph-noarch 14 k 2022-04-03T19:40:59.155 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2022-04-03T19:40:59.155 INFO:teuthology.orchestra.run.smithi077.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-04-03T19:40:59.155 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-04-03T19:40:59.155 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 25 k 2022-04-03T19:40:59.156 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2 x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 795 k 2022-04-03T19:40:59.156 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 125 k 2022-04-03T19:40:59.156 INFO:teuthology.orchestra.run.smithi077.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-04-03T19:40:59.156 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2022-04-03T19:40:59.156 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-04-03T19:40:59.157 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 179 k 2022-04-03T19:40:59.157 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 367 k 2022-04-03T19:40:59.157 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 345 k 2022-04-03T19:40:59.157 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 109 k 2022-04-03T19:40:59.158 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 94 k 2022-04-03T19:40:59.158 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 3.8 M 2022-04-03T19:40:59.158 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 175 k 2022-04-03T19:40:59.158 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2022-04-03T19:40:59.159 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2022-04-03T19:40:59.159 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 2.6 M 2022-04-03T19:40:59.159 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-libnm x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 1.8 M 2022-04-03T19:40:59.159 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-team x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 146 k 2022-04-03T19:40:59.160 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-tui x86_64 1:1.30.0-13.el8_4 rhel-8-for-x86_64-baseos-rpms 329 k 2022-04-03T19:40:59.160 INFO:teuthology.orchestra.run.smithi077.stdout: librados2 x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 3.4 M 2022-04-03T19:40:59.160 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1 x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 3.9 M 2022-04-03T19:40:59.160 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2022-04-03T19:40:59.161 INFO:teuthology.orchestra.run.smithi077.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-04-03T19:40:59.161 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 23 M 2022-04-03T19:40:59.161 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards noarch 2:17.0.0-11442.gcfb8f943.el8 ceph-noarch 23 k 2022-04-03T19:40:59.161 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 2.3 M 2022-04-03T19:40:59.161 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core noarch 2:17.0.0-11442.gcfb8f943.el8 ceph-noarch 227 k 2022-04-03T19:40:59.162 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 3.9 M 2022-04-03T19:40:59.162 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 17 M 2022-04-03T19:40:59.162 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts noarch 2:17.0.0-11442.gcfb8f943.el8 ceph-noarch 15 k 2022-04-03T19:40:59.162 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 24 k 2022-04-03T19:40:59.163 INFO:teuthology.orchestra.run.smithi077.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-04-03T19:40:59.163 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-04-03T19:40:59.163 INFO:teuthology.orchestra.run.smithi077.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2022-04-03T19:40:59.164 INFO:teuthology.orchestra.run.smithi077.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-04-03T19:40:59.164 INFO:teuthology.orchestra.run.smithi077.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2022-04-03T19:40:59.164 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 170 k 2022-04-03T19:40:59.164 INFO:teuthology.orchestra.run.smithi077.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2022-04-03T19:40:59.165 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2022-04-03T19:40:59.165 INFO:teuthology.orchestra.run.smithi077.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-04-03T19:40:59.165 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2022-04-03T19:40:59.166 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-04-03T19:40:59.166 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1 x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 484 k 2022-04-03T19:40:59.166 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2022-04-03T19:40:59.167 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2 x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 5.3 M 2022-04-03T19:40:59.167 INFO:teuthology.orchestra.run.smithi077.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-04-03T19:40:59.167 INFO:teuthology.orchestra.run.smithi077.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2022-04-03T19:40:59.167 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-04-03T19:40:59.167 INFO:teuthology.orchestra.run.smithi077.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-04-03T19:40:59.168 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-04-03T19:40:59.168 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-04-03T19:40:59.168 INFO:teuthology.orchestra.run.smithi077.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2022-04-03T19:40:59.168 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-04-03T19:40:59.169 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-04-03T19:40:59.169 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-04-03T19:40:59.169 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-04-03T19:40:59.169 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-04-03T19:40:59.169 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-04-03T19:40:59.170 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-04-03T19:40:59.170 INFO:teuthology.orchestra.run.smithi077.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-04-03T19:40:59.170 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-04-03T19:40:59.170 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 45 k 2022-04-03T19:40:59.171 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common x86_64 2:17.0.0-11442.gcfb8f943.el8 ceph 105 k 2022-04-03T19:40:59.171 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-04-03T19:40:59.171 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-04-03T19:40:59.171 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-04-03T19:40:59.172 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-04-03T19:40:59.172 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography x86_64 3.2.1-4.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-04-03T19:40:59.172 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-04-03T19:40:59.172 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-04-03T19:40:59.172 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-04-03T19:40:59.173 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2022-04-03T19:40:59.173 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-04-03T19:40:59.173 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-04-03T19:40:59.173 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-04-03T19:40:59.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-04-03T19:40:59.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-04-03T19:40:59.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-04-03T19:40:59.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-04-03T19:40:59.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-04-03T19:40:59.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-04-03T19:40:59.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-04-03T19:40:59.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-04-03T19:40:59.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-04-03T19:40:59.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-04-03T19:40:59.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-04-03T19:40:59.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-04-03T19:40:59.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-04-03T19:40:59.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-04-03T19:40:59.177 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-04-03T19:40:59.177 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-04-03T19:40:59.177 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-04-03T19:40:59.177 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-04-03T19:40:59.178 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-04-03T19:40:59.178 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-04-03T19:40:59.178 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-04-03T19:40:59.178 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2022-04-03T19:40:59.179 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-04-03T19:40:59.179 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-04-03T19:40:59.179 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-04-03T19:40:59.179 INFO:teuthology.orchestra.run.smithi077.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-04-03T19:40:59.180 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-04-03T19:40:59.180 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-04-03T19:40:59.180 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-04-03T19:40:59.180 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-04-03T19:40:59.180 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-04-03T19:40:59.181 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-04-03T19:40:59.181 INFO:teuthology.orchestra.run.smithi077.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2022-04-03T19:40:59.181 INFO:teuthology.orchestra.run.smithi077.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-04-03T19:40:59.181 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2022-04-03T19:40:59.182 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2022-04-03T19:40:59.182 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:40:59.182 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2022-04-03T19:40:59.182 INFO:teuthology.orchestra.run.smithi077.stdout:====================================================================================================================== 2022-04-03T19:40:59.182 INFO:teuthology.orchestra.run.smithi077.stdout:Install 122 Packages 2022-04-03T19:40:59.183 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 6 Packages 2022-04-03T19:40:59.183 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:40:59.184 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 207 M 2022-04-03T19:40:59.185 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2022-04-03T19:40:59.444 INFO:teuthology.orchestra.run.smithi077.stdout:(1/128): ceph-17.0.0-11442.gcfb8f943.el8.x86_64 24 kB/s | 6.5 kB 00:00 2022-04-03T19:41:00.089 INFO:teuthology.orchestra.run.smithi077.stdout:(2/128): ceph-fuse-17.0.0-11442.gcfb8f943.el8.x 1.3 MB/s | 880 kB 00:00 2022-04-03T19:41:00.216 INFO:teuthology.orchestra.run.smithi077.stdout:(3/128): ceph-immutable-object-cache-17.0.0-114 1.3 MB/s | 169 kB 00:00 2022-04-03T19:41:00.623 INFO:teuthology.orchestra.run.smithi186.stdout:(8/128): ceph-common-17.0.0-11442.gcfb8f943.el8 8.6 MB/s | 23 MB 00:02 2022-04-03T19:41:00.674 INFO:teuthology.orchestra.run.smithi186.stdout:(9/128): ceph-selinux-17.0.0-11442.gcfb8f943.el 476 kB/s | 24 kB 00:00 2022-04-03T19:41:00.940 INFO:teuthology.orchestra.run.smithi186.stdout:(10/128): ceph-osd-17.0.0-11442.gcfb8f943.el8.x 7.5 MB/s | 17 MB 00:02 2022-04-03T19:41:00.964 INFO:teuthology.orchestra.run.smithi077.stdout:(4/128): ceph-mds-17.0.0-11442.gcfb8f943.el8.x8 3.1 MB/s | 2.3 MB 00:00 2022-04-03T19:41:01.055 INFO:teuthology.orchestra.run.smithi186.stdout:(11/128): ceph-radosgw-17.0.0-11442.gcfb8f943.e 7.8 MB/s | 16 MB 00:02 2022-04-03T19:41:01.058 INFO:teuthology.orchestra.run.smithi186.stdout:(12/128): cephfs-mirror-17.0.0-11442.gcfb8f943. 2.3 MB/s | 273 kB 00:00 2022-04-03T19:41:01.089 INFO:teuthology.orchestra.run.smithi186.stdout:(13/128): libcephfs-devel-17.0.0-11442.gcfb8f94 748 kB/s | 25 kB 00:00 2022-04-03T19:41:01.147 INFO:teuthology.orchestra.run.smithi186.stdout:(14/128): libcephsqlite-17.0.0-11442.gcfb8f943. 2.8 MB/s | 170 kB 00:00 2022-04-03T19:41:01.159 INFO:teuthology.orchestra.run.smithi186.stdout:(15/128): libcephfs2-17.0.0-11442.gcfb8f943.el8 7.7 MB/s | 795 kB 00:00 2022-04-03T19:41:01.197 INFO:teuthology.orchestra.run.smithi186.stdout:(16/128): librados-devel-17.0.0-11442.gcfb8f943 2.5 MB/s | 125 kB 00:00 2022-04-03T19:41:01.218 INFO:teuthology.orchestra.run.smithi186.stdout:(17/128): libradosstriper1-17.0.0-11442.gcfb8f9 8.0 MB/s | 484 kB 00:00 2022-04-03T19:41:01.274 INFO:teuthology.orchestra.run.smithi186.stdout:(18/128): python3-ceph-argparse-17.0.0-11442.gc 802 kB/s | 45 kB 00:00 2022-04-03T19:41:01.333 INFO:teuthology.orchestra.run.smithi186.stdout:(19/128): python3-ceph-common-17.0.0-11442.gcfb 1.8 MB/s | 105 kB 00:00 2022-04-03T19:41:01.392 INFO:teuthology.orchestra.run.smithi186.stdout:(20/128): python3-cephfs-17.0.0-11442.gcfb8f943 3.0 MB/s | 179 kB 00:00 2022-04-03T19:41:01.402 INFO:teuthology.orchestra.run.smithi077.stdout:(5/128): ceph-base-17.0.0-11442.gcfb8f943.el8.x 2.4 MB/s | 5.3 MB 00:02 2022-04-03T19:41:01.409 INFO:teuthology.orchestra.run.smithi077.stdout:(6/128): ceph-mgr-17.0.0-11442.gcfb8f943.el8.x8 3.5 MB/s | 1.5 MB 00:00 2022-04-03T19:41:01.447 INFO:teuthology.orchestra.run.smithi186.stdout:(21/128): python3-rados-17.0.0-11442.gcfb8f943. 6.5 MB/s | 367 kB 00:00 2022-04-03T19:41:01.515 INFO:teuthology.orchestra.run.smithi186.stdout:(22/128): python3-rbd-17.0.0-11442.gcfb8f943.el 5.0 MB/s | 345 kB 00:00 2022-04-03T19:41:01.575 INFO:teuthology.orchestra.run.smithi186.stdout:(23/128): python3-rgw-17.0.0-11442.gcfb8f943.el 1.8 MB/s | 109 kB 00:00 2022-04-03T19:41:01.634 INFO:teuthology.orchestra.run.smithi186.stdout:(24/128): rbd-fuse-17.0.0-11442.gcfb8f943.el8.x 1.6 MB/s | 94 kB 00:00 2022-04-03T19:41:01.898 INFO:teuthology.orchestra.run.smithi186.stdout:(25/128): librgw2-17.0.0-11442.gcfb8f943.el8.x8 7.6 MB/s | 5.3 MB 00:00 2022-04-03T19:41:01.961 INFO:teuthology.orchestra.run.smithi186.stdout:(26/128): rbd-nbd-17.0.0-11442.gcfb8f943.el8.x8 2.7 MB/s | 175 kB 00:00 2022-04-03T19:41:02.017 INFO:teuthology.orchestra.run.smithi186.stdout:(27/128): ceph-grafana-dashboards-17.0.0-11442. 406 kB/s | 23 kB 00:00 2022-04-03T19:41:02.076 INFO:teuthology.orchestra.run.smithi186.stdout:(28/128): ceph-mgr-cephadm-17.0.0-11442.gcfb8f9 1.7 MB/s | 105 kB 00:00 2022-04-03T19:41:02.155 INFO:teuthology.orchestra.run.smithi186.stdout:(29/128): rbd-mirror-17.0.0-11442.gcfb8f943.el8 7.2 MB/s | 3.8 MB 00:00 2022-04-03T19:41:02.290 INFO:teuthology.orchestra.run.smithi186.stdout:(30/128): ceph-mgr-dashboard-17.0.0-11442.gcfb8 7.6 MB/s | 1.6 MB 00:00 2022-04-03T19:41:02.355 INFO:teuthology.orchestra.run.smithi186.stdout:(31/128): ceph-mgr-modules-core-17.0.0-11442.gc 3.4 MB/s | 227 kB 00:00 2022-04-03T19:41:02.413 INFO:teuthology.orchestra.run.smithi186.stdout:(32/128): ceph-mgr-rook-17.0.0-11442.gcfb8f943. 814 kB/s | 47 kB 00:00 2022-04-03T19:41:02.470 INFO:teuthology.orchestra.run.smithi186.stdout:(33/128): ceph-prometheus-alerts-17.0.0-11442.g 260 kB/s | 15 kB 00:00 2022-04-03T19:41:02.536 INFO:teuthology.orchestra.run.smithi186.stdout:(34/128): ceph-volume-17.0.0-11442.gcfb8f943.el 3.8 MB/s | 257 kB 00:00 2022-04-03T19:41:02.596 INFO:teuthology.orchestra.run.smithi186.stdout:(35/128): cephadm-17.0.0-11442.gcfb8f943.el8.no 1.2 MB/s | 71 kB 00:00 2022-04-03T19:41:02.653 INFO:teuthology.orchestra.run.smithi186.stdout:(36/128): cephfs-top-17.0.0-11442.gcfb8f943.el8 253 kB/s | 14 kB 00:00 2022-04-03T19:41:02.831 INFO:teuthology.orchestra.run.smithi077.stdout:(7/128): ceph-mon-17.0.0-11442.gcfb8f943.el8.x8 2.7 MB/s | 3.9 MB 00:01 2022-04-03T19:41:02.917 INFO:teuthology.orchestra.run.smithi186.stdout:(37/128): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2022-04-03T19:41:03.031 INFO:teuthology.orchestra.run.smithi186.stdout:(38/128): liboath-2.6.2-3.el8.x86_64.rpm 517 kB/s | 59 kB 00:00 2022-04-03T19:41:03.066 INFO:teuthology.orchestra.run.smithi186.stdout:(39/128): ceph-mgr-diskprediction-local-17.0.0- 8.1 MB/s | 7.4 MB 00:00 2022-04-03T19:41:03.271 INFO:teuthology.orchestra.run.smithi186.stdout:(40/128): libunwind-1.3.1-3.el8.x86_64.rpm 313 kB/s | 75 kB 00:00 2022-04-03T19:41:03.299 INFO:teuthology.orchestra.run.smithi186.stdout:(41/128): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.5 MB/s | 44 kB 00:00 2022-04-03T19:41:03.355 INFO:teuthology.orchestra.run.smithi186.stdout:(42/128): python3-beautifulsoup4-4.6.3-2.el8.1. 3.3 MB/s | 185 kB 00:00 2022-04-03T19:41:03.412 INFO:teuthology.orchestra.run.smithi186.stdout:(43/128): python3-cachetools-3.1.1-4.el8.noarch 582 kB/s | 33 kB 00:00 2022-04-03T19:41:03.440 INFO:teuthology.orchestra.run.smithi186.stdout:(44/128): python3-certifi-2018.10.15-7.el8.noar 588 kB/s | 16 kB 00:00 2022-04-03T19:41:03.470 INFO:teuthology.orchestra.run.smithi186.stdout:(45/128): python3-cheroot-8.5.2-1.el8.noarch.rp 5.7 MB/s | 173 kB 00:00 2022-04-03T19:41:03.526 INFO:teuthology.orchestra.run.smithi186.stdout:(46/128): python3-cherrypy-18.4.0-1.el8.noarch. 6.6 MB/s | 384 kB 00:00 2022-04-03T19:41:03.555 INFO:teuthology.orchestra.run.smithi186.stdout:(47/128): python3-asyncssh-2.7.0-2.el8.noarch.r 974 kB/s | 476 kB 00:00 2022-04-03T19:41:03.556 INFO:teuthology.orchestra.run.smithi186.stdout:(48/128): python3-google-auth-1.1.1-10.el8.noar 2.7 MB/s | 81 kB 00:00 2022-04-03T19:41:03.584 INFO:teuthology.orchestra.run.smithi186.stdout:(49/128): python3-jaraco-6.2-6.el8.noarch.rpm 391 kB/s | 11 kB 00:00 2022-04-03T19:41:03.584 INFO:teuthology.orchestra.run.smithi186.stdout:(50/128): python3-jaraco-functools-2.0-4.el8.no 629 kB/s | 18 kB 00:00 2022-04-03T19:41:03.613 INFO:teuthology.orchestra.run.smithi186.stdout:(51/128): python3-logutils-0.3.5-11.el8.noarch. 1.7 MB/s | 49 kB 00:00 2022-04-03T19:41:03.642 INFO:teuthology.orchestra.run.smithi186.stdout:(52/128): python3-more-itertools-7.2.0-3.el8.no 2.0 MB/s | 59 kB 00:00 2022-04-03T19:41:03.689 INFO:teuthology.orchestra.run.smithi186.stdout:(53/128): python3-natsort-7.1.1-2.el8.noarch.rp 1.3 MB/s | 60 kB 00:00 2022-04-03T19:41:03.720 INFO:teuthology.orchestra.run.smithi186.stdout:(54/128): python3-pecan-1.3.2-9.el8.noarch.rpm 8.8 MB/s | 283 kB 00:00 2022-04-03T19:41:03.748 INFO:teuthology.orchestra.run.smithi186.stdout:(55/128): python3-portend-2.6-1.el8.noarch.rpm 548 kB/s | 16 kB 00:00 2022-04-03T19:41:03.787 INFO:teuthology.orchestra.run.smithi186.stdout:(56/128): python3-repoze-lru-0.7-6.el8.noarch.r 871 kB/s | 34 kB 00:00 2022-04-03T19:41:03.793 INFO:teuthology.orchestra.run.smithi186.stdout:(57/128): python3-kubernetes-11.0.0-6.el8.noarc 5.6 MB/s | 1.2 MB 00:00 2022-04-03T19:41:03.841 INFO:teuthology.orchestra.run.smithi186.stdout:(58/128): python3-rsa-4.8-1.el8.noarch.rpm 1.3 MB/s | 61 kB 00:00 2022-04-03T19:41:03.864 INFO:teuthology.orchestra.run.smithi186.stdout:(59/128): python3-routes-2.4.1-12.el8.noarch.rp 2.5 MB/s | 196 kB 00:00 2022-04-03T19:41:03.868 INFO:teuthology.orchestra.run.smithi186.stdout:(60/128): python3-simplegeneric-0.8.1-17.el8.no 678 kB/s | 19 kB 00:00 2022-04-03T19:41:03.893 INFO:teuthology.orchestra.run.smithi186.stdout:(61/128): python3-singledispatch-3.4.0.3-18.el8 842 kB/s | 24 kB 00:00 2022-04-03T19:41:03.896 INFO:teuthology.orchestra.run.smithi186.stdout:(62/128): python3-tempora-1.14.1-5.el8.noarch.r 1.0 MB/s | 29 kB 00:00 2022-04-03T19:41:03.922 INFO:teuthology.orchestra.run.smithi186.stdout:(63/128): python3-trustme-0.6.0-4.el8.noarch.rp 961 kB/s | 27 kB 00:00 2022-04-03T19:41:03.927 INFO:teuthology.orchestra.run.smithi186.stdout:(64/128): python3-waitress-1.2.1-2.el8.1.noarch 7.8 MB/s | 241 kB 00:00 2022-04-03T19:41:03.953 INFO:teuthology.orchestra.run.smithi186.stdout:(65/128): python3-webob-1.8.5-1.el8.1.noarch.rp 7.9 MB/s | 251 kB 00:00 2022-04-03T19:41:03.966 INFO:teuthology.orchestra.run.smithi186.stdout:(66/128): python3-websocket-client-0.56.0-5.el8 1.5 MB/s | 61 kB 00:00 2022-04-03T19:41:03.983 INFO:teuthology.orchestra.run.smithi186.stdout:(67/128): python3-webtest-2.0.33-1.el8.noarch.r 2.8 MB/s | 85 kB 00:00 2022-04-03T19:41:03.994 INFO:teuthology.orchestra.run.smithi186.stdout:(68/128): python3-zc-lockfile-2.0-2.el8.noarch. 829 kB/s | 23 kB 00:00 2022-04-03T19:41:04.049 INFO:teuthology.orchestra.run.smithi186.stdout:(69/128): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.2 MB/s | 69 kB 00:00 2022-04-03T19:41:04.080 INFO:teuthology.orchestra.run.smithi186.stdout:(70/128): python3-mako-1.0.6-13.el8.noarch.rpm 4.9 MB/s | 157 kB 00:00 2022-04-03T19:41:04.102 INFO:teuthology.orchestra.run.smithi186.stdout:(71/128): flex-2.6.1-9.el8.x86_64.rpm 15 MB/s | 320 kB 00:00 2022-04-03T19:41:04.122 INFO:teuthology.orchestra.run.smithi186.stdout:(72/128): jq-1.5-12.el8.x86_64.rpm 8.1 MB/s | 161 kB 00:00 2022-04-03T19:41:04.143 INFO:teuthology.orchestra.run.smithi186.stdout:(73/128): python3-jsonpatch-1.21-2.el8.noarch.r 1.2 MB/s | 27 kB 00:00 2022-04-03T19:41:04.165 INFO:teuthology.orchestra.run.smithi186.stdout:(74/128): librdkafka-0.11.4-1.el8.x86_64.rpm 16 MB/s | 353 kB 00:00 2022-04-03T19:41:04.188 INFO:teuthology.orchestra.run.smithi186.stdout:(75/128): python3-jinja2-2.10.1-2.el8_0.noarch. 23 MB/s | 538 kB 00:00 2022-04-03T19:41:04.213 INFO:teuthology.orchestra.run.smithi186.stdout:(76/128): automake-1.16.1-7.el8.noarch.rpm 28 MB/s | 713 kB 00:00 2022-04-03T19:41:04.240 INFO:teuthology.orchestra.run.smithi186.stdout:(77/128): thrift-0.13.0-2.el8.x86_64.rpm 6.8 MB/s | 1.7 MB 00:00 2022-04-03T19:41:04.250 INFO:teuthology.orchestra.run.smithi186.stdout:(78/128): libtool-2.4.6-25.el8.x86_64.rpm 19 MB/s | 709 kB 00:00 2022-04-03T19:41:04.278 INFO:teuthology.orchestra.run.smithi186.stdout:(79/128): python3-pyasn1-modules-0.3.7-6.el8.no 3.8 MB/s | 110 kB 00:00 2022-04-03T19:41:04.306 INFO:teuthology.orchestra.run.smithi186.stdout:(80/128): python3-numpy-1.14.3-9.el8.x86_64.rpm 57 MB/s | 3.7 MB 00:00 2022-04-03T19:41:04.306 INFO:teuthology.orchestra.run.smithi186.stdout:(81/128): python3-jsonpointer-1.10-11.el8.noarc 716 kB/s | 20 kB 00:00 2022-04-03T19:41:04.334 INFO:teuthology.orchestra.run.smithi186.stdout:(82/128): python3-pytz-2017.2-9.el8.noarch.rpm 1.9 MB/s | 54 kB 00:00 2022-04-03T19:41:04.336 INFO:teuthology.orchestra.run.smithi186.stdout:(83/128): oniguruma-6.8.2-2.el8.x86_64.rpm 6.4 MB/s | 187 kB 00:00 2022-04-03T19:41:04.358 INFO:teuthology.orchestra.run.smithi186.stdout:(84/128): python3-pyasn1-0.3.7-6.el8.noarch.rpm 5.3 MB/s | 126 kB 00:00 2022-04-03T19:41:04.379 INFO:teuthology.orchestra.run.smithi186.stdout:(85/128): python3-pyOpenSSL-19.0.0-1.el8.noarch 4.8 MB/s | 103 kB 00:00 2022-04-03T19:41:04.403 INFO:teuthology.orchestra.run.smithi186.stdout:(86/128): lttng-ust-2.8.1-11.el8.x86_64.rpm 11 MB/s | 259 kB 00:00 2022-04-03T19:41:04.461 INFO:teuthology.orchestra.run.smithi186.stdout:(87/128): openblas-threads-0.3.12-1.el8.x86_64. 82 MB/s | 4.7 MB 00:00 2022-04-03T19:41:04.570 INFO:teuthology.orchestra.run.smithi186.stdout:(88/128): python3-scipy-1.0.0-20.module+el8.1.0 62 MB/s | 15 MB 00:00 2022-04-03T19:41:04.598 INFO:teuthology.orchestra.run.smithi186.stdout:(89/128): python3-werkzeug-0.12.2-4.el8.noarch. 16 MB/s | 457 kB 00:00 2022-04-03T19:41:04.629 INFO:teuthology.orchestra.run.smithi186.stdout:(90/128): python3-babel-2.5.1-5.el8.noarch.rpm 29 MB/s | 4.8 MB 00:00 2022-04-03T19:41:04.630 INFO:teuthology.orchestra.run.smithi186.stdout:(91/128): python3-markupsafe-0.23-19.el8.x86_64 1.2 MB/s | 39 kB 00:00 2022-04-03T19:41:04.654 INFO:teuthology.orchestra.run.smithi186.stdout:(92/128): python3-prettytable-0.7.2-14.el8.noar 1.7 MB/s | 44 kB 00:00 2022-04-03T19:41:04.662 INFO:teuthology.orchestra.run.smithi186.stdout:(93/128): socat-1.7.3.3-2.el8.x86_64.rpm 9.1 MB/s | 302 kB 00:00 2022-04-03T19:41:04.708 INFO:teuthology.orchestra.run.smithi186.stdout:(94/128): openblas-0.3.12-1.el8.x86_64.rpm 84 MB/s | 4.6 MB 00:00 2022-04-03T19:41:04.711 INFO:teuthology.orchestra.run.smithi186.stdout:(95/128): bison-3.0.4-10.el8.x86_64.rpm 14 MB/s | 688 kB 00:00 2022-04-03T19:41:04.734 INFO:teuthology.orchestra.run.smithi186.stdout:(96/128): python3-numpy-f2py-1.14.3-9.el8.x86_6 8.7 MB/s | 225 kB 00:00 2022-04-03T19:41:04.740 INFO:teuthology.orchestra.run.smithi186.stdout:(97/128): python3-jwt-1.6.1-2.el8.noarch.rpm 1.7 MB/s | 43 kB 00:00 2022-04-03T19:41:04.753 INFO:teuthology.orchestra.run.smithi186.stdout:(98/128): libaio-devel-0.3.112-1.el8.x86_64.rpm 995 kB/s | 19 kB 00:00 2022-04-03T19:41:04.764 INFO:teuthology.orchestra.run.smithi186.stdout:(99/128): python3-requests-oauthlib-1.0.0-1.el8 1.8 MB/s | 43 kB 00:00 2022-04-03T19:41:04.782 INFO:teuthology.orchestra.run.smithi186.stdout:(100/128): libquadmath-8.4.1-1.el8.x86_64.rpm 6.6 MB/s | 169 kB 00:00 2022-04-03T19:41:04.790 INFO:teuthology.orchestra.run.smithi186.stdout:(101/128): python3-cffi-1.11.5-5.el8.x86_64.rpm 8.8 MB/s | 238 kB 00:00 2022-04-03T19:41:04.811 INFO:teuthology.orchestra.run.smithi186.stdout:(102/128): libgfortran-8.4.1-1.el8.x86_64.rpm 22 MB/s | 639 kB 00:00 2022-04-03T19:41:04.818 INFO:teuthology.orchestra.run.smithi186.stdout:(103/128): elfutils-libelf-devel-0.182-3.el8.x8 2.1 MB/s | 59 kB 00:00 2022-04-03T19:41:04.836 INFO:teuthology.orchestra.run.smithi186.stdout:(104/128): libselinux-devel-2.9-5.el8.x86_64.rp 8.0 MB/s | 200 kB 00:00 2022-04-03T19:41:04.843 INFO:teuthology.orchestra.run.smithi186.stdout:(105/128): zlib-devel-1.2.11-17.el8.x86_64.rpm 2.3 MB/s | 58 kB 00:00 2022-04-03T19:41:04.861 INFO:teuthology.orchestra.run.smithi186.stdout:(106/128): pcre2-utf16-10.32-2.el8.x86_64.rpm 8.8 MB/s | 229 kB 00:00 2022-04-03T19:41:04.868 INFO:teuthology.orchestra.run.smithi186.stdout:(107/128): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.8 MB/s | 47 kB 00:00 2022-04-03T19:41:04.882 INFO:teuthology.orchestra.run.smithi186.stdout:(108/128): libverto-devel-0.3.0-5.el8.x86_64.rp 857 kB/s | 18 kB 00:00 2022-04-03T19:41:04.894 INFO:teuthology.orchestra.run.smithi186.stdout:(109/128): libsepol-devel-2.9-2.el8.x86_64.rpm 3.3 MB/s | 86 kB 00:00 2022-04-03T19:41:04.910 INFO:teuthology.orchestra.run.smithi186.stdout:(110/128): libuuid-devel-2.32.1-27.el8.x86_64.r 4.0 MB/s | 97 kB 00:00 2022-04-03T19:41:04.919 INFO:teuthology.orchestra.run.smithi186.stdout:(111/128): python3-oauthlib-2.1.0-1.el8.noarch. 6.1 MB/s | 155 kB 00:00 2022-04-03T19:41:04.944 INFO:teuthology.orchestra.run.smithi186.stdout:(112/128): fuse-2.9.7-12.el8.x86_64.rpm 3.3 MB/s | 83 kB 00:00 2022-04-03T19:41:04.951 INFO:teuthology.orchestra.run.smithi186.stdout:(113/128): openssl-devel-1.1.1g-15.el8_3.x86_64 57 MB/s | 2.3 MB 00:00 2022-04-03T19:41:04.974 INFO:teuthology.orchestra.run.smithi186.stdout:(114/128): python3-pycparser-2.14-14.el8.noarch 3.6 MB/s | 109 kB 00:00 2022-04-03T19:41:04.989 INFO:teuthology.orchestra.run.smithi186.stdout:(115/128): pcre2-devel-10.32-2.el8.x86_64.rpm 18 MB/s | 605 kB 00:00 2022-04-03T19:41:04.999 INFO:teuthology.orchestra.run.smithi186.stdout:(116/128): python3-cryptography-3.2.1-4.el8.x86 22 MB/s | 559 kB 00:00 2022-04-03T19:41:05.011 INFO:teuthology.orchestra.run.smithi186.stdout:(117/128): xfsprogs-devel-5.0.0-8.el8.x86_64.rp 3.1 MB/s | 70 kB 00:00 2022-04-03T19:41:05.031 INFO:teuthology.orchestra.run.smithi186.stdout:(118/128): pcre2-utf32-10.32-2.el8.x86_64.rpm 8.1 MB/s | 220 kB 00:00 2022-04-03T19:41:05.038 INFO:teuthology.orchestra.run.smithi186.stdout:(119/128): keyutils-libs-devel-1.5.10-6.el8.x86 1.8 MB/s | 48 kB 00:00 2022-04-03T19:41:05.056 INFO:teuthology.orchestra.run.smithi186.stdout:(120/128): libcom_err-devel-1.45.6-1.el8.x86_64 1.4 MB/s | 38 kB 00:00 2022-04-03T19:41:05.066 INFO:teuthology.orchestra.run.smithi186.stdout:(121/128): krb5-devel-1.18.2-8.3.el8_4.x86_64.r 20 MB/s | 559 kB 00:00 2022-04-03T19:41:05.443 INFO:teuthology.orchestra.run.smithi077.stdout:(8/128): ceph-osd-17.0.0-11442.gcfb8f943.el8.x8 4.2 MB/s | 17 MB 00:04 2022-04-03T19:41:05.467 INFO:teuthology.orchestra.run.smithi077.stdout:(9/128): ceph-selinux-17.0.0-11442.gcfb8f943.el 1.0 MB/s | 24 kB 00:00 2022-04-03T19:41:05.507 INFO:teuthology.orchestra.run.smithi186.stdout:(122/128): ceph-test-17.0.0-11442.gcfb8f943.el8 11 MB/s | 52 MB 00:04 2022-04-03T19:41:05.529 INFO:teuthology.orchestra.run.smithi186.stdout:(123/128): NetworkManager-team-1.30.0-13.el8_4. 6.7 MB/s | 146 kB 00:00 2022-04-03T19:41:05.562 INFO:teuthology.orchestra.run.smithi186.stdout:(124/128): NetworkManager-libnm-1.30.0-13.el8_4 55 MB/s | 1.8 MB 00:00 2022-04-03T19:41:05.585 INFO:teuthology.orchestra.run.smithi186.stdout:(125/128): NetworkManager-tui-1.30.0-13.el8_4.x 14 MB/s | 329 kB 00:00 2022-04-03T19:41:05.619 INFO:teuthology.orchestra.run.smithi077.stdout:(10/128): ceph-common-17.0.0-11442.gcfb8f943.el 3.6 MB/s | 23 MB 00:06 2022-04-03T19:41:05.620 INFO:teuthology.orchestra.run.smithi186.stdout:(126/128): NetworkManager-1.30.0-13.el8_4.x86_6 73 MB/s | 2.6 MB 00:00 2022-04-03T19:41:05.699 INFO:teuthology.orchestra.run.smithi186.stdout:(127/128): librbd1-17.0.0-11442.gcfb8f943.el8.x 6.1 MB/s | 3.9 MB 00:00 2022-04-03T19:41:05.712 INFO:teuthology.orchestra.run.smithi077.stdout:(11/128): cephfs-mirror-17.0.0-11442.gcfb8f943. 2.9 MB/s | 273 kB 00:00 2022-04-03T19:41:05.744 INFO:teuthology.orchestra.run.smithi077.stdout:(12/128): libcephfs-devel-17.0.0-11442.gcfb8f94 782 kB/s | 25 kB 00:00 2022-04-03T19:41:05.745 INFO:teuthology.orchestra.run.smithi186.stdout:(128/128): librados2-17.0.0-11442.gcfb8f943.el8 5.0 MB/s | 3.4 MB 00:00 2022-04-03T19:41:05.748 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2022-04-03T19:41:05.749 INFO:teuthology.orchestra.run.smithi186.stdout:Total 27 MB/s | 207 MB 00:07 2022-04-03T19:41:05.805 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2022-04-03T19:41:05.838 INFO:teuthology.orchestra.run.smithi077.stdout:(13/128): libcephfs2-17.0.0-11442.gcfb8f943.el8 8.3 MB/s | 795 kB 00:00 2022-04-03T19:41:05.864 INFO:teuthology.orchestra.run.smithi077.stdout:(14/128): libcephsqlite-17.0.0-11442.gcfb8f943. 6.5 MB/s | 170 kB 00:00 2022-04-03T19:41:05.889 INFO:teuthology.orchestra.run.smithi077.stdout:(15/128): librados-devel-17.0.0-11442.gcfb8f943 5.0 MB/s | 125 kB 00:00 2022-04-03T19:41:05.931 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2022-04-03T19:41:05.932 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2022-04-03T19:41:05.988 INFO:teuthology.orchestra.run.smithi077.stdout:(16/128): ceph-radosgw-17.0.0-11442.gcfb8f943.e 5.1 MB/s | 16 MB 00:03 2022-04-03T19:41:05.998 INFO:teuthology.orchestra.run.smithi077.stdout:(17/128): libradosstriper1-17.0.0-11442.gcfb8f9 4.3 MB/s | 484 kB 00:00 2022-04-03T19:41:06.023 INFO:teuthology.orchestra.run.smithi077.stdout:(18/128): python3-ceph-argparse-17.0.0-11442.gc 1.8 MB/s | 45 kB 00:00 2022-04-03T19:41:06.048 INFO:teuthology.orchestra.run.smithi077.stdout:(19/128): python3-ceph-common-17.0.0-11442.gcfb 4.2 MB/s | 105 kB 00:00 2022-04-03T19:41:06.074 INFO:teuthology.orchestra.run.smithi077.stdout:(20/128): python3-cephfs-17.0.0-11442.gcfb8f943 6.8 MB/s | 179 kB 00:00 2022-04-03T19:41:06.122 INFO:teuthology.orchestra.run.smithi077.stdout:(21/128): python3-rados-17.0.0-11442.gcfb8f943. 7.5 MB/s | 367 kB 00:00 2022-04-03T19:41:06.170 INFO:teuthology.orchestra.run.smithi077.stdout:(22/128): python3-rbd-17.0.0-11442.gcfb8f943.el 7.0 MB/s | 345 kB 00:00 2022-04-03T19:41:06.195 INFO:teuthology.orchestra.run.smithi077.stdout:(23/128): python3-rgw-17.0.0-11442.gcfb8f943.el 4.3 MB/s | 109 kB 00:00 2022-04-03T19:41:06.220 INFO:teuthology.orchestra.run.smithi077.stdout:(24/128): rbd-fuse-17.0.0-11442.gcfb8f943.el8.x 3.8 MB/s | 94 kB 00:00 2022-04-03T19:41:06.558 INFO:teuthology.orchestra.run.smithi077.stdout:(25/128): librgw2-17.0.0-11442.gcfb8f943.el8.x8 9.4 MB/s | 5.3 MB 00:00 2022-04-03T19:41:06.594 INFO:teuthology.orchestra.run.smithi077.stdout:(26/128): rbd-nbd-17.0.0-11442.gcfb8f943.el8.x8 4.9 MB/s | 175 kB 00:00 2022-04-03T19:41:06.617 INFO:teuthology.orchestra.run.smithi077.stdout:(27/128): ceph-grafana-dashboards-17.0.0-11442. 985 kB/s | 23 kB 00:00 2022-04-03T19:41:06.642 INFO:teuthology.orchestra.run.smithi077.stdout:(28/128): rbd-mirror-17.0.0-11442.gcfb8f943.el8 8.9 MB/s | 3.8 MB 00:00 2022-04-03T19:41:06.644 INFO:teuthology.orchestra.run.smithi077.stdout:(29/128): ceph-mgr-cephadm-17.0.0-11442.gcfb8f9 3.8 MB/s | 105 kB 00:00 2022-04-03T19:41:06.817 INFO:teuthology.orchestra.run.smithi077.stdout:(30/128): ceph-mgr-dashboard-17.0.0-11442.gcfb8 9.3 MB/s | 1.6 MB 00:00 2022-04-03T19:41:06.847 INFO:teuthology.orchestra.run.smithi077.stdout:(31/128): ceph-mgr-modules-core-17.0.0-11442.gc 7.6 MB/s | 227 kB 00:00 2022-04-03T19:41:06.871 INFO:teuthology.orchestra.run.smithi077.stdout:(32/128): ceph-mgr-rook-17.0.0-11442.gcfb8f943. 2.0 MB/s | 47 kB 00:00 2022-04-03T19:41:06.897 INFO:teuthology.orchestra.run.smithi077.stdout:(33/128): ceph-prometheus-alerts-17.0.0-11442.g 562 kB/s | 15 kB 00:00 2022-04-03T19:41:06.944 INFO:teuthology.orchestra.run.smithi077.stdout:(34/128): ceph-volume-17.0.0-11442.gcfb8f943.el 5.4 MB/s | 257 kB 00:00 2022-04-03T19:41:06.968 INFO:teuthology.orchestra.run.smithi077.stdout:(35/128): cephadm-17.0.0-11442.gcfb8f943.el8.no 2.9 MB/s | 71 kB 00:00 2022-04-03T19:41:06.992 INFO:teuthology.orchestra.run.smithi077.stdout:(36/128): cephfs-top-17.0.0-11442.gcfb8f943.el8 611 kB/s | 14 kB 00:00 2022-04-03T19:41:07.159 INFO:teuthology.orchestra.run.smithi077.stdout:(37/128): gperftools-libs-2.7-9.el8.x86_64.rpm 1.8 MB/s | 306 kB 00:00 2022-04-03T19:41:07.188 INFO:teuthology.orchestra.run.smithi077.stdout:(38/128): liboath-2.6.2-3.el8.x86_64.rpm 2.0 MB/s | 59 kB 00:00 2022-04-03T19:41:07.216 INFO:teuthology.orchestra.run.smithi077.stdout:(39/128): libunwind-1.3.1-3.el8.x86_64.rpm 2.6 MB/s | 75 kB 00:00 2022-04-03T19:41:07.275 INFO:teuthology.orchestra.run.smithi077.stdout:(40/128): python3-asyncssh-2.7.0-2.el8.noarch.r 8.1 MB/s | 476 kB 00:00 2022-04-03T19:41:07.304 INFO:teuthology.orchestra.run.smithi077.stdout:(41/128): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.5 MB/s | 44 kB 00:00 2022-04-03T19:41:07.334 INFO:teuthology.orchestra.run.smithi077.stdout:(42/128): python3-beautifulsoup4-4.6.3-2.el8.1. 6.0 MB/s | 185 kB 00:00 2022-04-03T19:41:07.362 INFO:teuthology.orchestra.run.smithi077.stdout:(43/128): ceph-mgr-diskprediction-local-17.0.0- 10 MB/s | 7.4 MB 00:00 2022-04-03T19:41:07.363 INFO:teuthology.orchestra.run.smithi077.stdout:(44/128): python3-cachetools-3.1.1-4.el8.noarch 1.1 MB/s | 33 kB 00:00 2022-04-03T19:41:07.394 INFO:teuthology.orchestra.run.smithi077.stdout:(45/128): python3-cheroot-8.5.2-1.el8.noarch.rp 5.7 MB/s | 173 kB 00:00 2022-04-03T19:41:07.448 INFO:teuthology.orchestra.run.smithi077.stdout:(46/128): python3-certifi-2018.10.15-7.el8.noar 192 kB/s | 16 kB 00:00 2022-04-03T19:41:07.451 INFO:teuthology.orchestra.run.smithi077.stdout:(47/128): python3-cherrypy-18.4.0-1.el8.noarch. 6.6 MB/s | 384 kB 00:00 2022-04-03T19:41:07.479 INFO:teuthology.orchestra.run.smithi077.stdout:(48/128): python3-jaraco-6.2-6.el8.noarch.rpm 398 kB/s | 11 kB 00:00 2022-04-03T19:41:07.507 INFO:teuthology.orchestra.run.smithi077.stdout:(49/128): python3-jaraco-functools-2.0-4.el8.no 626 kB/s | 18 kB 00:00 2022-04-03T19:41:07.533 INFO:teuthology.orchestra.run.smithi077.stdout:(50/128): python3-google-auth-1.1.1-10.el8.noar 956 kB/s | 81 kB 00:00 2022-04-03T19:41:07.562 INFO:teuthology.orchestra.run.smithi077.stdout:(51/128): python3-logutils-0.3.5-11.el8.noarch. 1.6 MB/s | 49 kB 00:00 2022-04-03T19:41:07.592 INFO:teuthology.orchestra.run.smithi077.stdout:(52/128): python3-more-itertools-7.2.0-3.el8.no 1.9 MB/s | 59 kB 00:00 2022-04-03T19:41:07.624 INFO:teuthology.orchestra.run.smithi077.stdout:(53/128): python3-kubernetes-11.0.0-6.el8.noarc 10 MB/s | 1.2 MB 00:00 2022-04-03T19:41:07.625 INFO:teuthology.orchestra.run.smithi077.stdout:(54/128): python3-natsort-7.1.1-2.el8.noarch.rp 1.8 MB/s | 60 kB 00:00 2022-04-03T19:41:07.655 INFO:teuthology.orchestra.run.smithi077.stdout:(55/128): python3-pecan-1.3.2-9.el8.noarch.rpm 8.8 MB/s | 283 kB 00:00 2022-04-03T19:41:07.656 INFO:teuthology.orchestra.run.smithi077.stdout:(56/128): python3-portend-2.6-1.el8.noarch.rpm 499 kB/s | 16 kB 00:00 2022-04-03T19:41:07.683 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2022-04-03T19:41:07.684 INFO:teuthology.orchestra.run.smithi077.stdout:(57/128): python3-repoze-lru-0.7-6.el8.noarch.r 1.2 MB/s | 34 kB 00:00 2022-04-03T19:41:07.687 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2022-04-03T19:41:07.713 INFO:teuthology.orchestra.run.smithi077.stdout:(58/128): python3-rsa-4.8-1.el8.noarch.rpm 2.1 MB/s | 61 kB 00:00 2022-04-03T19:41:07.715 INFO:teuthology.orchestra.run.smithi077.stdout:(59/128): python3-routes-2.4.1-12.el8.noarch.rp 3.3 MB/s | 196 kB 00:00 2022-04-03T19:41:07.741 INFO:teuthology.orchestra.run.smithi077.stdout:(60/128): python3-simplegeneric-0.8.1-17.el8.no 673 kB/s | 19 kB 00:00 2022-04-03T19:41:07.743 INFO:teuthology.orchestra.run.smithi077.stdout:(61/128): python3-singledispatch-3.4.0.3-18.el8 837 kB/s | 24 kB 00:00 2022-04-03T19:41:07.770 INFO:teuthology.orchestra.run.smithi077.stdout:(62/128): python3-tempora-1.14.1-5.el8.noarch.r 1.0 MB/s | 29 kB 00:00 2022-04-03T19:41:07.773 INFO:teuthology.orchestra.run.smithi077.stdout:(63/128): python3-trustme-0.6.0-4.el8.noarch.rp 957 kB/s | 27 kB 00:00 2022-04-03T19:41:07.802 INFO:teuthology.orchestra.run.smithi077.stdout:(64/128): python3-waitress-1.2.1-2.el8.1.noarch 7.5 MB/s | 241 kB 00:00 2022-04-03T19:41:07.804 INFO:teuthology.orchestra.run.smithi077.stdout:(65/128): python3-webob-1.8.5-1.el8.1.noarch.rp 7.8 MB/s | 251 kB 00:00 2022-04-03T19:41:07.830 INFO:teuthology.orchestra.run.smithi077.stdout:(66/128): python3-websocket-client-0.56.0-5.el8 2.1 MB/s | 61 kB 00:00 2022-04-03T19:41:07.835 INFO:teuthology.orchestra.run.smithi077.stdout:(67/128): python3-webtest-2.0.33-1.el8.noarch.r 2.7 MB/s | 85 kB 00:00 2022-04-03T19:41:07.859 INFO:teuthology.orchestra.run.smithi077.stdout:(68/128): python3-zc-lockfile-2.0-2.el8.noarch. 799 kB/s | 23 kB 00:00 2022-04-03T19:41:07.888 INFO:teuthology.orchestra.run.smithi077.stdout:(69/128): xmlstarlet-1.6.1-11.el8.x86_64.rpm 2.4 MB/s | 69 kB 00:00 2022-04-03T19:41:07.920 INFO:teuthology.orchestra.run.smithi077.stdout:(70/128): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2022-04-03T19:41:07.940 INFO:teuthology.orchestra.run.smithi077.stdout:(71/128): libaio-devel-0.3.112-1.el8.x86_64.rpm 927 kB/s | 19 kB 00:00 2022-04-03T19:41:07.962 INFO:teuthology.orchestra.run.smithi077.stdout:(72/128): python3-requests-oauthlib-1.0.0-1.el8 2.0 MB/s | 43 kB 00:00 2022-04-03T19:41:07.985 INFO:teuthology.orchestra.run.smithi077.stdout:(73/128): libquadmath-8.4.1-1.el8.x86_64.rpm 7.2 MB/s | 169 kB 00:00 2022-04-03T19:41:08.009 INFO:teuthology.orchestra.run.smithi077.stdout:(74/128): python3-cffi-1.11.5-5.el8.x86_64.rpm 10 MB/s | 238 kB 00:00 2022-04-03T19:41:08.022 INFO:teuthology.orchestra.run.smithi077.stdout:(75/128): thrift-0.13.0-2.el8.x86_64.rpm 9.3 MB/s | 1.7 MB 00:00 2022-04-03T19:41:08.051 INFO:teuthology.orchestra.run.smithi077.stdout:(76/128): libgfortran-8.4.1-1.el8.x86_64.rpm 15 MB/s | 639 kB 00:00 2022-04-03T19:41:08.059 INFO:teuthology.orchestra.run.smithi077.stdout:(77/128): elfutils-libelf-devel-0.182-3.el8.x86 1.6 MB/s | 59 kB 00:00 2022-04-03T19:41:08.078 INFO:teuthology.orchestra.run.smithi077.stdout:(78/128): libselinux-devel-2.9-5.el8.x86_64.rpm 7.3 MB/s | 200 kB 00:00 2022-04-03T19:41:08.086 INFO:teuthology.orchestra.run.smithi077.stdout:(79/128): zlib-devel-1.2.11-17.el8.x86_64.rpm 2.1 MB/s | 58 kB 00:00 2022-04-03T19:41:08.106 INFO:teuthology.orchestra.run.smithi077.stdout:(80/128): pcre2-utf16-10.32-2.el8.x86_64.rpm 8.3 MB/s | 229 kB 00:00 2022-04-03T19:41:08.113 INFO:teuthology.orchestra.run.smithi077.stdout:(81/128): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.7 MB/s | 47 kB 00:00 2022-04-03T19:41:08.130 INFO:teuthology.orchestra.run.smithi077.stdout:(82/128): libverto-devel-0.3.0-5.el8.x86_64.rpm 735 kB/s | 18 kB 00:00 2022-04-03T19:41:08.139 INFO:teuthology.orchestra.run.smithi077.stdout:(83/128): libsepol-devel-2.9-2.el8.x86_64.rpm 3.3 MB/s | 86 kB 00:00 2022-04-03T19:41:08.157 INFO:teuthology.orchestra.run.smithi077.stdout:(84/128): libuuid-devel-2.32.1-27.el8.x86_64.rp 3.6 MB/s | 97 kB 00:00 2022-04-03T19:41:08.166 INFO:teuthology.orchestra.run.smithi077.stdout:(85/128): python3-oauthlib-2.1.0-1.el8.noarch.r 5.6 MB/s | 155 kB 00:00 2022-04-03T19:41:08.193 INFO:teuthology.orchestra.run.smithi077.stdout:(86/128): fuse-2.9.7-12.el8.x86_64.rpm 3.0 MB/s | 83 kB 00:00 2022-04-03T19:41:08.220 INFO:teuthology.orchestra.run.smithi077.stdout:(87/128): openssl-devel-1.1.1g-15.el8_3.x86_64. 37 MB/s | 2.3 MB 00:00 2022-04-03T19:41:08.222 INFO:teuthology.orchestra.run.smithi077.stdout:(88/128): python3-pycparser-2.14-14.el8.noarch. 3.8 MB/s | 109 kB 00:00 2022-04-03T19:41:08.252 INFO:teuthology.orchestra.run.smithi077.stdout:(89/128): pcre2-devel-10.32-2.el8.x86_64.rpm 19 MB/s | 605 kB 00:00 2022-04-03T19:41:08.269 INFO:teuthology.orchestra.run.smithi077.stdout:(90/128): python3-cryptography-3.2.1-4.el8.x86_ 13 MB/s | 559 kB 00:00 2022-04-03T19:41:08.277 INFO:teuthology.orchestra.run.smithi077.stdout:(91/128): xfsprogs-devel-5.0.0-8.el8.x86_64.rpm 2.8 MB/s | 70 kB 00:00 2022-04-03T19:41:08.291 INFO:teuthology.orchestra.run.smithi077.stdout:(92/128): pcre2-utf32-10.32-2.el8.x86_64.rpm 9.9 MB/s | 220 kB 00:00 2022-04-03T19:41:08.303 INFO:teuthology.orchestra.run.smithi077.stdout:(93/128): keyutils-libs-devel-1.5.10-6.el8.x86_ 1.8 MB/s | 48 kB 00:00 2022-04-03T19:41:08.316 INFO:teuthology.orchestra.run.smithi077.stdout:(94/128): libcom_err-devel-1.45.6-1.el8.x86_64. 1.8 MB/s | 38 kB 00:00 2022-04-03T19:41:08.339 INFO:teuthology.orchestra.run.smithi077.stdout:(95/128): krb5-devel-1.18.2-8.3.el8_4.x86_64.rp 16 MB/s | 559 kB 00:00 2022-04-03T19:41:08.346 INFO:teuthology.orchestra.run.smithi077.stdout:(96/128): python3-mako-1.0.6-13.el8.noarch.rpm 6.0 MB/s | 157 kB 00:00 2022-04-03T19:41:08.367 INFO:teuthology.orchestra.run.smithi077.stdout:(97/128): flex-2.6.1-9.el8.x86_64.rpm 11 MB/s | 320 kB 00:00 2022-04-03T19:41:08.376 INFO:teuthology.orchestra.run.smithi077.stdout:(98/128): jq-1.5-12.el8.x86_64.rpm 5.3 MB/s | 161 kB 00:00 2022-04-03T19:41:08.388 INFO:teuthology.orchestra.run.smithi077.stdout:(99/128): python3-jsonpatch-1.21-2.el8.noarch.r 1.3 MB/s | 27 kB 00:00 2022-04-03T19:41:08.409 INFO:teuthology.orchestra.run.smithi077.stdout:(100/128): librdkafka-0.11.4-1.el8.x86_64.rpm 12 MB/s | 353 kB 00:00 2022-04-03T19:41:08.426 INFO:teuthology.orchestra.run.smithi077.stdout:(101/128): python3-jinja2-2.10.1-2.el8_0.noarch 14 MB/s | 538 kB 00:00 2022-04-03T19:41:08.441 INFO:teuthology.orchestra.run.smithi077.stdout:(102/128): automake-1.16.1-7.el8.noarch.rpm 22 MB/s | 713 kB 00:00 2022-04-03T19:41:08.465 INFO:teuthology.orchestra.run.smithi077.stdout:(103/128): libtool-2.4.6-25.el8.x86_64.rpm 18 MB/s | 709 kB 00:00 2022-04-03T19:41:08.488 INFO:teuthology.orchestra.run.smithi077.stdout:(104/128): python3-pyasn1-modules-0.3.7-6.el8.n 4.8 MB/s | 110 kB 00:00 2022-04-03T19:41:08.508 INFO:teuthology.orchestra.run.smithi077.stdout:(105/128): python3-jsonpointer-1.10-11.el8.noar 987 kB/s | 20 kB 00:00 2022-04-03T19:41:08.540 INFO:teuthology.orchestra.run.smithi077.stdout:(106/128): python3-numpy-1.14.3-9.el8.x86_64.rp 39 MB/s | 3.7 MB 00:00 2022-04-03T19:41:08.542 INFO:teuthology.orchestra.run.smithi077.stdout:(107/128): python3-pytz-2017.2-9.el8.noarch.rpm 1.6 MB/s | 54 kB 00:00 2022-04-03T19:41:08.568 INFO:teuthology.orchestra.run.smithi077.stdout:(108/128): oniguruma-6.8.2-2.el8.x86_64.rpm 6.9 MB/s | 187 kB 00:00 2022-04-03T19:41:08.576 INFO:teuthology.orchestra.run.smithi077.stdout:(109/128): python3-pyasn1-0.3.7-6.el8.noarch.rp 3.6 MB/s | 126 kB 00:00 2022-04-03T19:41:08.602 INFO:teuthology.orchestra.run.smithi077.stdout:(110/128): python3-pyOpenSSL-19.0.0-1.el8.noarc 4.6 MB/s | 103 kB 00:00 2022-04-03T19:41:08.627 INFO:teuthology.orchestra.run.smithi077.stdout:(111/128): lttng-ust-2.8.1-11.el8.x86_64.rpm 11 MB/s | 259 kB 00:00 2022-04-03T19:41:08.731 INFO:teuthology.orchestra.run.smithi077.stdout:(112/128): openblas-threads-0.3.12-1.el8.x86_64 46 MB/s | 4.7 MB 00:00 2022-04-03T19:41:08.870 INFO:teuthology.orchestra.run.smithi077.stdout:(113/128): python3-scipy-1.0.0-20.module+el8.1. 48 MB/s | 15 MB 00:00 2022-04-03T19:41:08.888 INFO:teuthology.orchestra.run.smithi077.stdout:(114/128): python3-babel-2.5.1-5.el8.noarch.rpm 31 MB/s | 4.8 MB 00:00 2022-04-03T19:41:08.918 INFO:teuthology.orchestra.run.smithi077.stdout:(115/128): python3-werkzeug-0.12.2-4.el8.noarch 9.3 MB/s | 457 kB 00:00 2022-04-03T19:41:08.919 INFO:teuthology.orchestra.run.smithi077.stdout:(116/128): python3-markupsafe-0.23-19.el8.x86_6 1.3 MB/s | 39 kB 00:00 2022-04-03T19:41:08.942 INFO:teuthology.orchestra.run.smithi077.stdout:(117/128): python3-prettytable-0.7.2-14.el8.noa 1.8 MB/s | 44 kB 00:00 2022-04-03T19:41:08.958 INFO:teuthology.orchestra.run.smithi077.stdout:(118/128): socat-1.7.3.3-2.el8.x86_64.rpm 7.5 MB/s | 302 kB 00:00 2022-04-03T19:41:08.991 INFO:teuthology.orchestra.run.smithi077.stdout:(119/128): bison-3.0.4-10.el8.x86_64.rpm 21 MB/s | 688 kB 00:00 2022-04-03T19:41:09.017 INFO:teuthology.orchestra.run.smithi077.stdout:(120/128): python3-numpy-f2py-1.14.3-9.el8.x86_ 8.7 MB/s | 225 kB 00:00 2022-04-03T19:41:09.048 INFO:teuthology.orchestra.run.smithi077.stdout:(121/128): openblas-0.3.12-1.el8.x86_64.rpm 44 MB/s | 4.6 MB 00:00 2022-04-03T19:41:09.496 INFO:teuthology.orchestra.run.smithi077.stdout:(122/128): librados2-17.0.0-11442.gcfb8f943.el8 7.2 MB/s | 3.4 MB 00:00 2022-04-03T19:41:09.521 INFO:teuthology.orchestra.run.smithi077.stdout:(123/128): NetworkManager-team-1.30.0-13.el8_4. 5.7 MB/s | 146 kB 00:00 2022-04-03T19:41:09.567 INFO:teuthology.orchestra.run.smithi077.stdout:(124/128): NetworkManager-libnm-1.30.0-13.el8_4 39 MB/s | 1.8 MB 00:00 2022-04-03T19:41:09.583 INFO:teuthology.orchestra.run.smithi077.stdout:(125/128): librbd1-17.0.0-11442.gcfb8f943.el8.x 7.2 MB/s | 3.9 MB 00:00 2022-04-03T19:41:09.612 INFO:teuthology.orchestra.run.smithi077.stdout:(126/128): NetworkManager-tui-1.30.0-13.el8_4.x 7.3 MB/s | 329 kB 00:00 2022-04-03T19:41:09.643 INFO:teuthology.orchestra.run.smithi077.stdout:(127/128): NetworkManager-1.30.0-13.el8_4.x86_6 43 MB/s | 2.6 MB 00:00 2022-04-03T19:41:10.357 INFO:teuthology.orchestra.run.smithi077.stdout:(128/128): ceph-test-17.0.0-11442.gcfb8f943.el8 11 MB/s | 52 MB 00:04 2022-04-03T19:41:10.360 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2022-04-03T19:41:10.361 INFO:teuthology.orchestra.run.smithi077.stdout:Total 19 MB/s | 207 MB 00:11 2022-04-03T19:41:10.460 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2022-04-03T19:41:10.637 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2022-04-03T19:41:10.638 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2022-04-03T19:41:12.388 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2022-04-03T19:41:12.430 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: liboath-2.6.2-3.el8.x86_64 1/1 2022-04-03T19:41:12.486 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2022-04-03T19:41:12.489 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2022-04-03T19:41:12.564 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/134 2022-04-03T19:41:12.809 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 2/134 2022-04-03T19:41:12.992 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/134 2022-04-03T19:41:13.374 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/134 2022-04-03T19:41:13.611 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librados2-2:17.0.0-11442.gcfb8f943.el8.x86_64 4/134 2022-04-03T19:41:14.033 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:17.0.0-11442.gcfb8f943.el8.x86_64 4/134 2022-04-03T19:41:14.234 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librbd1-2:17.0.0-11442.gcfb8f943.el8.x86_64 5/134 2022-04-03T19:41:14.349 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:17.0.0-11442.gcfb8f943.el8.x86_64 5/134 2022-04-03T19:41:14.505 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs2-2:17.0.0-11442.gcfb8f943.el8.x86_64 6/134 2022-04-03T19:41:14.592 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs2-2:17.0.0-11442.gcfb8f943.el8.x86_64 6/134 2022-04-03T19:41:14.841 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rados-2:17.0.0-11442.gcfb8f943.el8.x86_6 7/134 2022-04-03T19:41:14.984 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/134 2022-04-03T19:41:15.211 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/134 2022-04-03T19:41:15.926 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/134 2022-04-03T19:41:16.125 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librgw2-2:17.0.0-11442.gcfb8f943.el8.x86_64 10/134 2022-04-03T19:41:16.221 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librgw2-2:17.0.0-11442.gcfb8f943.el8.x86_64 10/134 2022-04-03T19:41:16.421 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2022-04-03T19:41:16.607 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2022-04-03T19:41:16.742 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephsqlite-2:17.0.0-11442.gcfb8f943.el8.x86_6 13/134 2022-04-03T19:41:16.850 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephsqlite-2:17.0.0-11442.gcfb8f943.el8.x86_6 13/134 2022-04-03T19:41:17.016 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libradosstriper1-2:17.0.0-11442.gcfb8f943.el8.x8 14/134 2022-04-03T19:41:17.367 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2022-04-03T19:41:17.421 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libradosstriper1-2:17.0.0-11442.gcfb8f943.el8.x8 14/134 2022-04-03T19:41:17.444 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: liboath-2.6.2-3.el8.x86_64 1/1 2022-04-03T19:41:17.700 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 15/134 2022-04-03T19:41:17.728 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/134 2022-04-03T19:41:17.752 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 15/134 2022-04-03T19:41:17.978 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/134 2022-04-03T19:41:18.051 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.30.0-13.el8_4.x86_64 16/134 2022-04-03T19:41:18.334 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-1:1.30.0-13.el8_4.x86_64 16/134 2022-04-03T19:41:18.461 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/134 2022-04-03T19:41:18.673 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.30.0-13.el8_4.x86_64 16/134 2022-04-03T19:41:18.767 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librados2-2:17.0.0-11442.gcfb8f943.el8.x86_64 3/134 2022-04-03T19:41:18.818 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 17/134 2022-04-03T19:41:19.008 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 18/134 2022-04-03T19:41:19.146 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 18/134 2022-04-03T19:41:19.291 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 19/134 2022-04-03T19:41:19.303 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:17.0.0-11442.gcfb8f943.el8.x86_64 3/134 2022-04-03T19:41:19.511 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librbd1-2:17.0.0-11442.gcfb8f943.el8.x86_64 4/134 2022-04-03T19:41:19.725 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:17.0.0-11442.gcfb8f943.el8.x86_64 4/134 2022-04-03T19:41:19.841 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 19/134 2022-04-03T19:41:19.978 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs2-2:17.0.0-11442.gcfb8f943.el8.x86_64 5/134 2022-04-03T19:41:20.000 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 20/134 2022-04-03T19:41:20.122 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs2-2:17.0.0-11442.gcfb8f943.el8.x86_64 5/134 2022-04-03T19:41:20.390 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rados-2:17.0.0-11442.gcfb8f943.el8.x86_6 6/134 2022-04-03T19:41:20.516 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 20/134 2022-04-03T19:41:20.561 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 7/134 2022-04-03T19:41:20.693 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 7/134 2022-04-03T19:41:20.725 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-0.3.12-1.el8.x86_64 21/134 2022-04-03T19:41:21.072 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/134 2022-04-03T19:41:21.467 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 21/134 2022-04-03T19:41:21.746 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 22/134 2022-04-03T19:41:21.853 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/134 2022-04-03T19:41:21.982 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2022-04-03T19:41:22.052 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librgw2-2:17.0.0-11442.gcfb8f943.el8.x86_64 10/134 2022-04-03T19:41:22.207 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librgw2-2:17.0.0-11442.gcfb8f943.el8.x86_64 10/134 2022-04-03T19:41:22.207 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/134 2022-04-03T19:41:22.435 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 25/134 2022-04-03T19:41:22.485 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2022-04-03T19:41:22.638 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2022-04-03T19:41:22.679 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 26/134 2022-04-03T19:41:22.835 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephsqlite-2:17.0.0-11442.gcfb8f943.el8.x86_6 13/134 2022-04-03T19:41:22.939 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-common-2:17.0.0-11442.gcfb8f943.el8 27/134 2022-04-03T19:41:22.986 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephsqlite-2:17.0.0-11442.gcfb8f943.el8.x86_6 13/134 2022-04-03T19:41:23.093 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-argparse-2:17.0.0-11442.gcfb8f943.e 28/134 2022-04-03T19:41:23.169 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libradosstriper1-2:17.0.0-11442.gcfb8f943.el8.x8 14/134 2022-04-03T19:41:23.939 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libradosstriper1-2:17.0.0-11442.gcfb8f943.el8.x8 14/134 2022-04-03T19:41:24.027 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cephfs-2:17.0.0-11442.gcfb8f943.el8.x86_ 29/134 2022-04-03T19:41:24.311 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 15/134 2022-04-03T19:41:24.356 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 30/134 2022-04-03T19:41:24.406 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 15/134 2022-04-03T19:41:24.643 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 31/134 2022-04-03T19:41:24.787 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.30.0-13.el8_4.x86_64 16/134 2022-04-03T19:41:24.918 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 32/134 2022-04-03T19:41:25.069 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-1:1.30.0-13.el8_4.x86_64 16/134 2022-04-03T19:41:25.107 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 33/134 2022-04-03T19:41:25.545 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.30.0-13.el8_4.x86_64 16/134 2022-04-03T19:41:25.825 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2022-04-03T19:41:26.030 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/134 2022-04-03T19:41:26.263 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 19/134 2022-04-03T19:41:26.419 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 20/134 2022-04-03T19:41:26.610 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 21/134 2022-04-03T19:41:26.791 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 21/134 2022-04-03T19:41:27.010 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 22/134 2022-04-03T19:41:27.157 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 33/134 2022-04-03T19:41:27.468 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 34/134 2022-04-03T19:41:27.675 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 22/134 2022-04-03T19:41:27.697 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rsa-4.8-1.el8.noarch 35/134 2022-04-03T19:41:27.827 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 23/134 2022-04-03T19:41:27.871 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 36/134 2022-04-03T19:41:28.481 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 23/134 2022-04-03T19:41:28.663 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rgw-2:17.0.0-11442.gcfb8f943.el8.x86_64 37/134 2022-04-03T19:41:28.835 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-0.3.12-1.el8.x86_64 24/134 2022-04-03T19:41:28.837 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rbd-2:17.0.0-11442.gcfb8f943.el8.x86_64 38/134 2022-04-03T19:41:28.985 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados-devel-2:17.0.0-11442.gcfb8f943.el8.x86_ 39/134 2022-04-03T19:41:29.153 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 40/134 2022-04-03T19:41:29.336 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 41/134 2022-04-03T19:41:29.511 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 42/134 2022-04-03T19:41:29.719 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2022-04-03T19:41:29.812 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 24/134 2022-04-03T19:41:30.109 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2022-04-03T19:41:30.180 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 25/134 2022-04-03T19:41:30.403 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cryptography-3.2.1-4.el8.x86_64 45/134 2022-04-03T19:41:30.470 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 26/134 2022-04-03T19:41:30.618 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2022-04-03T19:41:30.688 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-common-2:17.0.0-11442.gcfb8f943.el8 27/134 2022-04-03T19:41:30.865 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-argparse-2:17.0.0-11442.gcfb8f943.e 28/134 2022-04-03T19:41:30.884 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2022-04-03T19:41:31.133 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cephfs-2:17.0.0-11442.gcfb8f943.el8.x86_ 29/134 2022-04-03T19:41:31.172 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2022-04-03T19:41:31.378 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 30/134 2022-04-03T19:41:31.473 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2022-04-03T19:41:31.706 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2022-04-03T19:41:31.866 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2022-04-03T19:41:32.087 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2022-04-03T19:41:32.248 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : fuse-2.9.7-12.el8.x86_64 53/134 2022-04-03T19:41:32.433 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 54/134 2022-04-03T19:41:32.657 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 55/134 2022-04-03T19:41:32.802 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 56/134 2022-04-03T19:41:33.107 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 57/134 2022-04-03T19:41:33.431 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 58/134 2022-04-03T19:41:33.722 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 59/134 2022-04-03T19:41:33.908 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 30/134 2022-04-03T19:41:34.003 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 60/134 2022-04-03T19:41:34.165 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 61/134 2022-04-03T19:41:34.318 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2022-04-03T19:41:34.507 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 63/134 2022-04-03T19:41:34.590 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 63/134 2022-04-03T19:41:34.788 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : jq-1.5-12.el8.x86_64 64/134 2022-04-03T19:41:35.005 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 65/134 2022-04-03T19:41:35.191 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 31/134 2022-04-03T19:41:35.280 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 66/134 2022-04-03T19:41:35.484 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : automake-1.16.1-7.el8.noarch 67/134 2022-04-03T19:41:35.568 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 32/134 2022-04-03T19:41:35.656 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 68/134 2022-04-03T19:41:35.845 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2022-04-03T19:41:35.894 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 33/134 2022-04-03T19:41:36.095 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 70/134 2022-04-03T19:41:36.137 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 34/134 2022-04-03T19:41:36.321 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 71/134 2022-04-03T19:41:36.419 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rsa-4.8-1.el8.noarch 35/134 2022-04-03T19:41:36.460 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 72/134 2022-04-03T19:41:36.600 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 73/134 2022-04-03T19:41:36.623 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 36/134 2022-04-03T19:41:36.836 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rgw-2:17.0.0-11442.gcfb8f943.el8.x86_64 37/134 2022-04-03T19:41:36.860 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 74/134 2022-04-03T19:41:37.016 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rbd-2:17.0.0-11442.gcfb8f943.el8.x86_64 38/134 2022-04-03T19:41:37.056 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 75/134 2022-04-03T19:41:37.235 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librados-devel-2:17.0.0-11442.gcfb8f943.el8.x86_ 39/134 2022-04-03T19:41:37.236 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 76/134 2022-04-03T19:41:37.434 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 77/134 2022-04-03T19:41:37.461 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 40/134 2022-04-03T19:41:37.571 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 78/134 2022-04-03T19:41:37.693 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2022-04-03T19:41:37.787 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 79/134 2022-04-03T19:41:37.859 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 42/134 2022-04-03T19:41:37.989 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 42/134 2022-04-03T19:41:38.011 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 80/134 2022-04-03T19:41:38.169 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : jq-1.5-12.el8.x86_64 43/134 2022-04-03T19:41:38.170 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 81/134 2022-04-03T19:41:38.378 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 44/134 2022-04-03T19:41:38.424 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-portend-2.6-1.el8.noarch 82/134 2022-04-03T19:41:38.630 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 83/134 2022-04-03T19:41:38.709 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 45/134 2022-04-03T19:41:38.798 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 84/134 2022-04-03T19:41:38.929 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : automake-1.16.1-7.el8.noarch 46/134 2022-04-03T19:41:38.993 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 85/134 2022-04-03T19:41:39.115 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 47/134 2022-04-03T19:41:39.348 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 48/134 2022-04-03T19:41:39.552 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 49/134 2022-04-03T19:41:39.828 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 50/134 2022-04-03T19:41:40.159 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 86/134 2022-04-03T19:41:40.332 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 51/134 2022-04-03T19:41:40.462 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 87/134 2022-04-03T19:41:40.619 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cryptography-3.2.1-4.el8.x86_64 52/134 2022-04-03T19:41:40.690 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 88/134 2022-04-03T19:41:40.877 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 53/134 2022-04-03T19:41:41.026 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 89/134 2022-04-03T19:41:41.175 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 54/134 2022-04-03T19:41:41.270 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 90/134 2022-04-03T19:41:41.449 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 55/134 2022-04-03T19:41:41.472 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 91/134 2022-04-03T19:41:41.680 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 92/134 2022-04-03T19:41:41.838 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 56/134 2022-04-03T19:41:42.078 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 57/134 2022-04-03T19:41:42.258 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 58/134 2022-04-03T19:41:42.505 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 59/134 2022-04-03T19:41:42.670 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : fuse-2.9.7-12.el8.x86_64 60/134 2022-04-03T19:41:42.893 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 61/134 2022-04-03T19:41:43.103 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 62/134 2022-04-03T19:41:43.249 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 63/134 2022-04-03T19:41:43.607 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 64/134 2022-04-03T19:41:44.011 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 65/134 2022-04-03T19:41:44.304 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 66/134 2022-04-03T19:41:44.589 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 67/134 2022-04-03T19:41:44.745 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 68/134 2022-04-03T19:41:44.997 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2022-04-03T19:41:45.293 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 70/134 2022-04-03T19:41:45.492 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 71/134 2022-04-03T19:41:45.664 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 72/134 2022-04-03T19:41:45.832 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 73/134 2022-04-03T19:41:46.071 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 74/134 2022-04-03T19:41:46.214 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:17.0.0-11442.gcfb8f943.el8.x86_64 93/134 2022-04-03T19:41:46.296 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 75/134 2022-04-03T19:41:46.413 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-common-2:17.0.0-11442.gcfb8f943.el8.x86_64 93/134 2022-04-03T19:41:46.512 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 76/134 2022-04-03T19:41:46.721 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 77/134 2022-04-03T19:41:46.896 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 78/134 2022-04-03T19:41:47.017 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:17.0.0-11442.gcfb8f943.el8.x86_64 93/134 2022-04-03T19:41:47.172 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 79/134 2022-04-03T19:41:47.289 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-base-2:17.0.0-11442.gcfb8f943.el8.x86_64 94/134 2022-04-03T19:41:47.382 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-base-2:17.0.0-11442.gcfb8f943.el8.x86_64 94/134 2022-04-03T19:41:47.426 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 80/134 2022-04-03T19:41:47.504 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-selinux-2:17.0.0-11442.gcfb8f943.el8.x86_64 95/134 2022-04-03T19:41:47.641 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 81/134 2022-04-03T19:41:47.909 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-portend-2.6-1.el8.noarch 82/134 2022-04-03T19:41:48.152 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 83/134 2022-04-03T19:41:48.355 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 84/134 2022-04-03T19:41:48.576 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 85/134 2022-04-03T19:41:50.115 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 86/134 2022-04-03T19:41:50.460 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 87/134 2022-04-03T19:41:50.759 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 88/134 2022-04-03T19:41:51.164 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 89/134 2022-04-03T19:41:51.462 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 90/134 2022-04-03T19:41:51.679 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 91/134 2022-04-03T19:41:51.899 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 92/134 2022-04-03T19:41:58.850 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:17.0.0-11442.gcfb8f943.el8.x86_64 93/134 2022-04-03T19:41:59.107 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-common-2:17.0.0-11442.gcfb8f943.el8.x86_64 93/134 2022-04-03T19:41:59.866 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:17.0.0-11442.gcfb8f943.el8.x86_64 93/134 2022-04-03T19:42:00.704 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-base-2:17.0.0-11442.gcfb8f943.el8.x86_64 94/134 2022-04-03T19:42:00.874 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-base-2:17.0.0-11442.gcfb8f943.el8.x86_64 94/134 2022-04-03T19:42:00.997 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-selinux-2:17.0.0-11442.gcfb8f943.el8.x86_64 95/134 2022-04-03T19:42:04.977 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-selinux-2:17.0.0-11442.gcfb8f943.el8.x86_64 95/134 2022-04-03T19:42:04.977 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /sys 2022-04-03T19:42:04.978 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /proc 2022-04-03T19:42:04.978 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /dev 2022-04-03T19:42:04.978 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /run 2022-04-03T19:42:04.978 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /mnt 2022-04-03T19:42:04.979 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /var/tmp 2022-04-03T19:42:04.979 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /home 2022-04-03T19:42:04.979 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /tmp 2022-04-03T19:42:04.980 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /dev 2022-04-03T19:42:04.980 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:42:05.351 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-volume-2:17.0.0-11442.gcfb8f943.el8.noarch 96/134 2022-04-03T19:42:06.866 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-volume-2:17.0.0-11442.gcfb8f943.el8.noarch 96/134 2022-04-03T19:42:07.078 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-osd-2:17.0.0-11442.gcfb8f943.el8.x86_64 97/134 2022-04-03T19:42:07.309 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-osd-2:17.0.0-11442.gcfb8f943.el8.x86_64 97/134 2022-04-03T19:42:07.493 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mds-2:17.0.0-11442.gcfb8f943.el8.x86_64 98/134 2022-04-03T19:42:07.878 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mds-2:17.0.0-11442.gcfb8f943.el8.x86_64 98/134 2022-04-03T19:42:08.044 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mon-2:17.0.0-11442.gcfb8f943.el8.x86_64 99/134 2022-04-03T19:42:08.062 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mon-2:17.0.0-11442.gcfb8f943.el8.x86_64 99/134 2022-04-03T19:42:10.369 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:17.0.0-11442.gcfb8f943.el8.noarch 100/134 2022-04-03T19:42:10.589 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephadm-2:17.0.0-11442.gcfb8f943.el8.noarch 100/134 2022-04-03T19:42:10.731 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-11442.gcfb8f943. 101/134 2022-04-03T19:42:10.887 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-11442.gcfb8f943 102/134 2022-04-03T19:42:11.117 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-11442.gcfb8f943.el8.no 103/134 2022-04-03T19:42:11.320 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-11442.gcfb8f943.el8.no 103/134 2022-04-03T19:42:11.560 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-11442.gcfb8f943.el8. 104/134 2022-04-03T19:42:12.529 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-11442.gcfb8f943.el8. 104/134 2022-04-03T19:42:12.717 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-11442.gcf 105/134 2022-04-03T19:42:12.823 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-11442.gcf 105/134 2022-04-03T19:42:13.173 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-11442.gcfb8f943.e 106/134 2022-04-03T19:42:13.361 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-2:17.0.0-11442.gcfb8f943.el8.x86_64 107/134 2022-04-03T19:42:13.399 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-2:17.0.0-11442.gcfb8f943.el8.x86_64 107/134 2022-04-03T19:42:13.567 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-rook-2:17.0.0-11442.gcfb8f943.el8.noarc 108/134 2022-04-03T19:42:13.578 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-11442.gcfb8f943.el8.noarc 108/134 2022-04-03T19:42:15.100 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-2:17.0.0-11442.gcfb8f943.el8.x86_64 109/134 2022-04-03T19:42:15.294 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-radosgw-2:17.0.0-11442.gcfb8f943.el8.x86_64 110/134 2022-04-03T19:42:15.390 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-11442.gcfb8f943.el8.x86_64 110/134 2022-04-03T19:42:15.567 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-11442.gcfb8 111/134 2022-04-03T19:42:15.613 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-11442.gcfb8 111/134 2022-04-03T19:42:15.767 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-mirror-2:17.0.0-11442.gcfb8f943.el8.x86_6 112/134 2022-04-03T19:42:16.152 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephfs-mirror-2:17.0.0-11442.gcfb8f943.el8.x86_6 112/134 2022-04-03T19:42:16.334 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-mirror-2:17.0.0-11442.gcfb8f943.el8.x86_64 113/134 2022-04-03T19:42:21.072 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-selinux-2:17.0.0-11442.gcfb8f943.el8.x86_64 95/134 2022-04-03T19:42:21.072 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /sys 2022-04-03T19:42:21.072 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /proc 2022-04-03T19:42:21.073 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /dev 2022-04-03T19:42:21.073 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /run 2022-04-03T19:42:21.073 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /mnt 2022-04-03T19:42:21.073 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /var/tmp 2022-04-03T19:42:21.074 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /home 2022-04-03T19:42:21.074 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /tmp 2022-04-03T19:42:21.074 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /dev 2022-04-03T19:42:21.074 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:42:21.282 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-mirror-2:17.0.0-11442.gcfb8f943.el8.x86_64 113/134 2022-04-03T19:42:21.479 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-volume-2:17.0.0-11442.gcfb8f943.el8.noarch 96/134 2022-04-03T19:42:21.641 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-test-2:17.0.0-11442.gcfb8f943.el8.x86_64 114/134 2022-04-03T19:42:21.807 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libtool-2.4.6-25.el8.x86_64 115/134 2022-04-03T19:42:22.524 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 115/134 2022-04-03T19:42:22.794 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 116/134 2022-04-03T19:42:23.098 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 117/134 2022-04-03T19:42:23.232 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-volume-2:17.0.0-11442.gcfb8f943.el8.noarch 96/134 2022-04-03T19:42:23.294 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-fuse-2:17.0.0-11442.gcfb8f943.el8.x86_64 118/134 2022-04-03T19:42:23.428 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs-devel-2:17.0.0-11442.gcfb8f943.el8.x86 119/134 2022-04-03T19:42:23.467 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-osd-2:17.0.0-11442.gcfb8f943.el8.x86_64 97/134 2022-04-03T19:42:23.568 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 120/134 2022-04-03T19:42:23.754 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-team-1:1.30.0-13.el8_4.x86_64 121/134 2022-04-03T19:42:23.781 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-osd-2:17.0.0-11442.gcfb8f943.el8.x86_64 97/134 2022-04-03T19:42:23.913 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 122/134 2022-04-03T19:42:23.930 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mds-2:17.0.0-11442.gcfb8f943.el8.x86_64 98/134 2022-04-03T19:42:24.049 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-top-2:17.0.0-11442.gcfb8f943.el8.noarch 123/134 2022-04-03T19:42:24.257 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-fuse-2:17.0.0-11442.gcfb8f943.el8.x86_64 124/134 2022-04-03T19:42:24.417 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-nbd-2:17.0.0-11442.gcfb8f943.el8.x86_64 125/134 2022-04-03T19:42:24.421 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mds-2:17.0.0-11442.gcfb8f943.el8.x86_64 98/134 2022-04-03T19:42:24.579 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mon-2:17.0.0-11442.gcfb8f943.el8.x86_64 99/134 2022-04-03T19:42:24.632 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 126/134 2022-04-03T19:42:24.641 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mon-2:17.0.0-11442.gcfb8f943.el8.x86_64 99/134 2022-04-03T19:42:24.840 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : bison-3.0.4-10.el8.x86_64 127/134 2022-04-03T19:42:24.895 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 127/134 2022-04-03T19:42:25.090 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2022-04-03T19:42:25.099 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2022-04-03T19:42:25.217 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-tui-1:1.30.0-7.el8.x86_64 129/134 2022-04-03T19:42:25.217 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 130/134 2022-04-03T19:42:25.460 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 130/134 2022-04-03T19:42:25.567 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-team-1:1.30.0-7.el8.x86_64 131/134 2022-04-03T19:42:25.582 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.30.0-7.el8.x86_64 132/134 2022-04-03T19:42:25.583 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-1:1.30.0-7.el8.x86_64 132/134 2022-04-03T19:42:26.044 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.30.0-7.el8.x86_64 132/134 2022-04-03T19:42:26.044 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-libnm-1:1.30.0-7.el8.x86_64 133/134 2022-04-03T19:42:26.285 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-libnm-1:1.30.0-7.el8.x86_64 133/134 2022-04-03T19:42:26.285 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 134/134 2022-04-03T19:42:27.140 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 134/134 2022-04-03T19:42:27.140 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-2:17.0.0-11442.gcfb8f943.el8.x86_64 1/134 2022-04-03T19:42:27.141 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-base-2:17.0.0-11442.gcfb8f943.el8.x86_64 2/134 2022-04-03T19:42:27.141 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-common-2:17.0.0-11442.gcfb8f943.el8.x86_64 3/134 2022-04-03T19:42:27.141 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-fuse-2:17.0.0-11442.gcfb8f943.el8.x86_64 4/134 2022-04-03T19:42:27.141 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-11442.gcfb8 5/134 2022-04-03T19:42:27.142 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mds-2:17.0.0-11442.gcfb8f943.el8.x86_64 6/134 2022-04-03T19:42:27.142 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-2:17.0.0-11442.gcfb8f943.el8.x86_64 7/134 2022-04-03T19:42:27.142 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mon-2:17.0.0-11442.gcfb8f943.el8.x86_64 8/134 2022-04-03T19:42:27.142 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-osd-2:17.0.0-11442.gcfb8f943.el8.x86_64 9/134 2022-04-03T19:42:27.143 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-radosgw-2:17.0.0-11442.gcfb8f943.el8.x86_64 10/134 2022-04-03T19:42:27.143 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-selinux-2:17.0.0-11442.gcfb8f943.el8.x86_64 11/134 2022-04-03T19:42:27.143 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-test-2:17.0.0-11442.gcfb8f943.el8.x86_64 12/134 2022-04-03T19:42:27.143 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-mirror-2:17.0.0-11442.gcfb8f943.el8.x86_6 13/134 2022-04-03T19:42:27.143 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs-devel-2:17.0.0-11442.gcfb8f943.el8.x86 14/134 2022-04-03T19:42:27.144 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs2-2:17.0.0-11442.gcfb8f943.el8.x86_64 15/134 2022-04-03T19:42:27.144 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephsqlite-2:17.0.0-11442.gcfb8f943.el8.x86_6 16/134 2022-04-03T19:42:27.144 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados-devel-2:17.0.0-11442.gcfb8f943.el8.x86_ 17/134 2022-04-03T19:42:27.144 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libradosstriper1-2:17.0.0-11442.gcfb8f943.el8.x8 18/134 2022-04-03T19:42:27.145 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librgw2-2:17.0.0-11442.gcfb8f943.el8.x86_64 19/134 2022-04-03T19:42:27.145 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-argparse-2:17.0.0-11442.gcfb8f943.e 20/134 2022-04-03T19:42:27.145 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-common-2:17.0.0-11442.gcfb8f943.el8 21/134 2022-04-03T19:42:27.145 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cephfs-2:17.0.0-11442.gcfb8f943.el8.x86_ 22/134 2022-04-03T19:42:27.145 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rados-2:17.0.0-11442.gcfb8f943.el8.x86_6 23/134 2022-04-03T19:42:27.146 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rbd-2:17.0.0-11442.gcfb8f943.el8.x86_64 24/134 2022-04-03T19:42:27.146 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rgw-2:17.0.0-11442.gcfb8f943.el8.x86_64 25/134 2022-04-03T19:42:27.146 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-fuse-2:17.0.0-11442.gcfb8f943.el8.x86_64 26/134 2022-04-03T19:42:27.146 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-mirror-2:17.0.0-11442.gcfb8f943.el8.x86_64 27/134 2022-04-03T19:42:27.158 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-nbd-2:17.0.0-11442.gcfb8f943.el8.x86_64 28/134 2022-04-03T19:42:27.158 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-11442.gcfb8f943 29/134 2022-04-03T19:42:27.159 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-11442.gcfb8f943.el8.no 30/134 2022-04-03T19:42:27.159 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-11442.gcfb8f943.el8. 31/134 2022-04-03T19:42:27.159 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-11442.gcf 32/134 2022-04-03T19:42:27.159 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-11442.gcfb8f943.e 33/134 2022-04-03T19:42:27.160 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-rook-2:17.0.0-11442.gcfb8f943.el8.noarc 34/134 2022-04-03T19:42:27.160 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-11442.gcfb8f943. 35/134 2022-04-03T19:42:27.160 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-volume-2:17.0.0-11442.gcfb8f943.el8.noarch 36/134 2022-04-03T19:42:27.160 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephadm-2:17.0.0-11442.gcfb8f943.el8.noarch 37/134 2022-04-03T19:42:27.160 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-top-2:17.0.0-11442.gcfb8f943.el8.noarch 38/134 2022-04-03T19:42:27.161 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2022-04-03T19:42:27.161 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2022-04-03T19:42:27.161 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2022-04-03T19:42:27.161 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2022-04-03T19:42:27.162 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2022-04-03T19:42:27.162 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2022-04-03T19:42:27.162 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2022-04-03T19:42:27.162 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2022-04-03T19:42:27.163 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2022-04-03T19:42:27.163 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2022-04-03T19:42:27.163 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2022-04-03T19:42:27.163 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2022-04-03T19:42:27.163 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2022-04-03T19:42:27.164 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2022-04-03T19:42:27.164 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2022-04-03T19:42:27.164 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2022-04-03T19:42:27.165 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2022-04-03T19:42:27.165 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2022-04-03T19:42:27.165 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2022-04-03T19:42:27.165 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2022-04-03T19:42:27.166 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2022-04-03T19:42:27.166 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2022-04-03T19:42:27.166 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2022-04-03T19:42:27.166 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2022-04-03T19:42:27.166 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2022-04-03T19:42:27.167 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2022-04-03T19:42:27.167 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/134 2022-04-03T19:42:27.167 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2022-04-03T19:42:27.167 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2022-04-03T19:42:27.168 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2022-04-03T19:42:27.168 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2022-04-03T19:42:27.168 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2022-04-03T19:42:27.168 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 71/134 2022-04-03T19:42:27.169 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 72/134 2022-04-03T19:42:27.169 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flex-2.6.1-9.el8.x86_64 73/134 2022-04-03T19:42:27.169 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : jq-1.5-12.el8.x86_64 74/134 2022-04-03T19:42:27.169 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 75/134 2022-04-03T19:42:27.169 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 76/134 2022-04-03T19:42:27.170 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 77/134 2022-04-03T19:42:27.170 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : automake-1.16.1-7.el8.noarch 78/134 2022-04-03T19:42:27.170 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 79/134 2022-04-03T19:42:27.170 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 80/134 2022-04-03T19:42:27.171 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 81/134 2022-04-03T19:42:27.171 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 82/134 2022-04-03T19:42:27.171 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 83/134 2022-04-03T19:42:27.171 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 84/134 2022-04-03T19:42:27.172 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 85/134 2022-04-03T19:42:27.172 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 86/134 2022-04-03T19:42:27.172 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 87/134 2022-04-03T19:42:27.172 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 88/134 2022-04-03T19:42:27.173 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 89/134 2022-04-03T19:42:27.173 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 90/134 2022-04-03T19:42:27.173 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 91/134 2022-04-03T19:42:27.173 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 92/134 2022-04-03T19:42:27.173 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 93/134 2022-04-03T19:42:27.174 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 94/134 2022-04-03T19:42:27.174 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 95/134 2022-04-03T19:42:27.174 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : bison-3.0.4-10.el8.x86_64 96/134 2022-04-03T19:42:27.174 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 97/134 2022-04-03T19:42:27.175 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 98/134 2022-04-03T19:42:27.175 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 99/134 2022-04-03T19:42:27.175 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 100/134 2022-04-03T19:42:27.175 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 101/134 2022-04-03T19:42:27.176 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 102/134 2022-04-03T19:42:27.176 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 103/134 2022-04-03T19:42:27.176 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 104/134 2022-04-03T19:42:27.176 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 105/134 2022-04-03T19:42:27.177 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 106/134 2022-04-03T19:42:27.177 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 107/134 2022-04-03T19:42:27.177 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 108/134 2022-04-03T19:42:27.177 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 109/134 2022-04-03T19:42:27.177 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 110/134 2022-04-03T19:42:27.178 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 111/134 2022-04-03T19:42:27.178 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 112/134 2022-04-03T19:42:27.178 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 113/134 2022-04-03T19:42:27.178 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 114/134 2022-04-03T19:42:27.179 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 115/134 2022-04-03T19:42:27.179 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 116/134 2022-04-03T19:42:27.179 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cryptography-3.2.1-4.el8.x86_64 117/134 2022-04-03T19:42:27.179 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 118/134 2022-04-03T19:42:27.179 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 119/134 2022-04-03T19:42:27.180 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 120/134 2022-04-03T19:42:27.180 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 121/134 2022-04-03T19:42:27.180 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 122/134 2022-04-03T19:42:27.180 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:17.0.0-11442.gcfb8f943.el8.x86_64 123/134 2022-04-03T19:42:27.181 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 124/134 2022-04-03T19:42:27.181 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:17.0.0-11442.gcfb8f943.el8.x86_64 125/134 2022-04-03T19:42:27.181 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 126/134 2022-04-03T19:42:27.181 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-team-1:1.30.0-13.el8_4.x86_64 127/134 2022-04-03T19:42:27.182 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-team-1:1.30.0-7.el8.x86_64 128/134 2022-04-03T19:42:27.182 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 129/134 2022-04-03T19:42:27.182 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-libnm-1:1.30.0-7.el8.x86_64 130/134 2022-04-03T19:42:27.182 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 131/134 2022-04-03T19:42:27.183 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-tui-1:1.30.0-7.el8.x86_64 132/134 2022-04-03T19:42:27.183 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-1:1.30.0-13.el8_4.x86_64 133/134 2022-04-03T19:42:28.918 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:17.0.0-11442.gcfb8f943.el8.noarch 100/134 2022-04-03T19:42:29.112 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephadm-2:17.0.0-11442.gcfb8f943.el8.noarch 100/134 2022-04-03T19:42:29.312 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-11442.gcfb8f943. 101/134 2022-04-03T19:42:29.541 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-11442.gcfb8f943 102/134 2022-04-03T19:42:29.762 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-11442.gcfb8f943.el8.no 103/134 2022-04-03T19:42:30.046 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-11442.gcfb8f943.el8.no 103/134 2022-04-03T19:42:30.266 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-11442.gcfb8f943.el8. 104/134 2022-04-03T19:42:31.472 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-11442.gcfb8f943.el8. 104/134 2022-04-03T19:42:31.661 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-11442.gcf 105/134 2022-04-03T19:42:31.825 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-11442.gcf 105/134 2022-04-03T19:42:32.208 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-11442.gcfb8f943.e 106/134 2022-04-03T19:42:32.366 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-2:17.0.0-11442.gcfb8f943.el8.x86_64 107/134 2022-04-03T19:42:32.459 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-2:17.0.0-11442.gcfb8f943.el8.x86_64 107/134 2022-04-03T19:42:32.619 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-rook-2:17.0.0-11442.gcfb8f943.el8.noarc 108/134 2022-04-03T19:42:32.666 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-11442.gcfb8f943.el8.noarc 108/134 2022-04-03T19:42:34.414 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-2:17.0.0-11442.gcfb8f943.el8.x86_64 109/134 2022-04-03T19:42:34.616 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-radosgw-2:17.0.0-11442.gcfb8f943.el8.x86_64 110/134 2022-04-03T19:42:34.773 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-11442.gcfb8f943.el8.x86_64 110/134 2022-04-03T19:42:34.960 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-11442.gcfb8 111/134 2022-04-03T19:42:35.060 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-11442.gcfb8 111/134 2022-04-03T19:42:35.269 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-mirror-2:17.0.0-11442.gcfb8f943.el8.x86_6 112/134 2022-04-03T19:42:35.764 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephfs-mirror-2:17.0.0-11442.gcfb8f943.el8.x86_6 112/134 2022-04-03T19:42:35.969 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-mirror-2:17.0.0-11442.gcfb8f943.el8.x86_64 113/134 2022-04-03T19:42:41.726 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-mirror-2:17.0.0-11442.gcfb8f943.el8.x86_64 113/134 2022-04-03T19:42:42.170 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-1:1.30.0-7.el8.x86_64 134/134 2022-04-03T19:42:42.170 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:06 ago on Sun 03 Apr 2022 07:42:35 PM UTC. 2022-04-03T19:42:43.193 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-test-2:17.0.0-11442.gcfb8f943.el8.x86_64 114/134 2022-04-03T19:42:43.483 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 115/134 2022-04-03T19:42:43.803 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 116/134 2022-04-03T19:42:44.050 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-fuse-2:17.0.0-11442.gcfb8f943.el8.x86_64 117/134 2022-04-03T19:42:44.217 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libtool-2.4.6-25.el8.x86_64 118/134 2022-04-03T19:42:44.264 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 118/134 2022-04-03T19:42:44.440 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs-devel-2:17.0.0-11442.gcfb8f943.el8.x86 119/134 2022-04-03T19:42:44.587 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 120/134 2022-04-03T19:42:44.808 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-team-1:1.30.0-13.el8_4.x86_64 121/134 2022-04-03T19:42:44.965 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 122/134 2022-04-03T19:42:45.186 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephfs-top-2:17.0.0-11442.gcfb8f943.el8.noarch 123/134 2022-04-03T19:42:45.386 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-fuse-2:17.0.0-11442.gcfb8f943.el8.x86_64 124/134 2022-04-03T19:42:45.585 INFO:teuthology.orchestra.run.smithi186.stdout:Installed products updated. 2022-04-03T19:42:45.707 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:42:45.708 INFO:teuthology.orchestra.run.smithi186.stdout:Upgraded: 2022-04-03T19:42:45.708 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-1:1.30.0-13.el8_4.x86_64 2022-04-03T19:42:45.708 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 2022-04-03T19:42:45.709 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-team-1:1.30.0-13.el8_4.x86_64 2022-04-03T19:42:45.709 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 2022-04-03T19:42:45.709 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.709 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.710 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2022-04-03T19:42:45.710 INFO:teuthology.orchestra.run.smithi186.stdout: automake-1.16.1-7.el8.noarch 2022-04-03T19:42:45.710 INFO:teuthology.orchestra.run.smithi186.stdout: bison-3.0.4-10.el8.x86_64 2022-04-03T19:42:45.711 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.711 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.711 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.712 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.712 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards-2:17.0.0-11442.gcfb8f943.el8.noarch 2022-04-03T19:42:45.712 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.712 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.713 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.713 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm-2:17.0.0-11442.gcfb8f943.el8.noarch 2022-04-03T19:42:45.713 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard-2:17.0.0-11442.gcfb8f943.el8.noarch 2022-04-03T19:42:45.713 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local-2:17.0.0-11442.gcfb8f943.el8.noarch 2022-04-03T19:42:45.714 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core-2:17.0.0-11442.gcfb8f943.el8.noarch 2022-04-03T19:42:45.714 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook-2:17.0.0-11442.gcfb8f943.el8.noarch 2022-04-03T19:42:45.714 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.715 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.715 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts-2:17.0.0-11442.gcfb8f943.el8.noarch 2022-04-03T19:42:45.715 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.715 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.716 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.716 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume-2:17.0.0-11442.gcfb8f943.el8.noarch 2022-04-03T19:42:45.716 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm-2:17.0.0-11442.gcfb8f943.el8.noarch 2022-04-03T19:42:45.716 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.717 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top-2:17.0.0-11442.gcfb8f943.el8.noarch 2022-04-03T19:42:45.717 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2022-04-03T19:42:45.717 INFO:teuthology.orchestra.run.smithi186.stdout: flex-2.6.1-9.el8.x86_64 2022-04-03T19:42:45.718 INFO:teuthology.orchestra.run.smithi186.stdout: fuse-2.9.7-12.el8.x86_64 2022-04-03T19:42:45.718 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-04-03T19:42:45.718 INFO:teuthology.orchestra.run.smithi186.stdout: jq-1.5-12.el8.x86_64 2022-04-03T19:42:45.718 INFO:teuthology.orchestra.run.smithi186.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2022-04-03T19:42:45.719 INFO:teuthology.orchestra.run.smithi186.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2022-04-03T19:42:45.719 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-04-03T19:42:45.719 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.719 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.720 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.720 INFO:teuthology.orchestra.run.smithi186.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2022-04-03T19:42:45.720 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran-8.4.1-1.el8.x86_64 2022-04-03T19:42:45.720 INFO:teuthology.orchestra.run.smithi186.stdout: liboath-2.6.2-3.el8.x86_64 2022-04-03T19:42:45.721 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath-8.4.1-1.el8.x86_64 2022-04-03T19:42:45.721 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-04-03T19:42:45.721 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.722 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.722 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka-0.11.4-1.el8.x86_64 2022-04-03T19:42:45.722 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.723 INFO:teuthology.orchestra.run.smithi186.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-04-03T19:42:45.723 INFO:teuthology.orchestra.run.smithi186.stdout: libsepol-devel-2.9-2.el8.x86_64 2022-04-03T19:42:45.723 INFO:teuthology.orchestra.run.smithi186.stdout: libtool-2.4.6-25.el8.x86_64 2022-04-03T19:42:45.724 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind-1.3.1-3.el8.x86_64 2022-04-03T19:42:45.724 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2022-04-03T19:42:45.725 INFO:teuthology.orchestra.run.smithi186.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-04-03T19:42:45.725 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-04-03T19:42:45.726 INFO:teuthology.orchestra.run.smithi186.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-04-03T19:42:45.726 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-0.3.12-1.el8.x86_64 2022-04-03T19:42:45.727 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-threads-0.3.12-1.el8.x86_64 2022-04-03T19:42:45.727 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2022-04-03T19:42:45.728 INFO:teuthology.orchestra.run.smithi186.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-04-03T19:42:45.728 INFO:teuthology.orchestra.run.smithi186.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-04-03T19:42:45.729 INFO:teuthology.orchestra.run.smithi186.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-04-03T19:42:45.729 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-04-03T19:42:45.730 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel-2.5.1-5.el8.noarch 2022-04-03T19:42:45.730 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-04-03T19:42:45.730 INFO:teuthology.orchestra.run.smithi186.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-04-03T19:42:45.731 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-04-03T19:42:45.731 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.732 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.732 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.733 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-04-03T19:42:45.733 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-04-03T19:42:45.734 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-04-03T19:42:45.734 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-04-03T19:42:45.734 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-3.2.1-4.el8.x86_64 2022-04-03T19:42:45.735 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-04-03T19:42:45.735 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-6.2-6.el8.noarch 2022-04-03T19:42:45.736 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-04-03T19:42:45.736 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2022-04-03T19:42:45.737 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-04-03T19:42:45.737 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-04-03T19:42:45.738 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-04-03T19:42:45.738 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-04-03T19:42:45.738 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-04-03T19:42:45.739 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako-1.0.6-13.el8.noarch 2022-04-03T19:42:45.739 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-04-03T19:42:45.740 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-04-03T19:42:45.740 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-04-03T19:42:45.741 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2022-04-03T19:42:45.741 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2022-04-03T19:42:45.742 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-04-03T19:42:45.742 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-04-03T19:42:45.743 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend-2.6-1.el8.noarch 2022-04-03T19:42:45.743 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-04-03T19:42:45.744 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-04-03T19:42:45.744 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-04-03T19:42:45.744 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-04-03T19:42:45.745 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser-2.14-14.el8.noarch 2022-04-03T19:42:45.745 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz-2017.2-9.el8.noarch 2022-04-03T19:42:45.746 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.746 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.747 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-04-03T19:42:45.748 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-04-03T19:42:45.748 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.748 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes-2.4.1-12.el8.noarch 2022-04-03T19:42:45.749 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa-4.8-1.el8.noarch 2022-04-03T19:42:45.749 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2022-04-03T19:42:45.750 INFO:teuthology.orchestra.run.smithi186.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-04-03T19:42:45.750 INFO:teuthology.orchestra.run.smithi186.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-04-03T19:42:45.751 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-04-03T19:42:45.751 INFO:teuthology.orchestra.run.smithi186.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-04-03T19:42:45.751 INFO:teuthology.orchestra.run.smithi186.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-04-03T19:42:45.752 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-04-03T19:42:45.752 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-04-03T19:42:45.753 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-04-03T19:42:45.753 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-04-03T19:42:45.754 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-04-03T19:42:45.754 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.755 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.755 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:42:45.756 INFO:teuthology.orchestra.run.smithi186.stdout: socat-1.7.3.3-2.el8.x86_64 2022-04-03T19:42:45.756 INFO:teuthology.orchestra.run.smithi186.stdout: thrift-0.13.0-2.el8.x86_64 2022-04-03T19:42:45.757 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2022-04-03T19:42:45.757 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-04-03T19:42:45.759 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-nbd-2:17.0.0-11442.gcfb8f943.el8.x86_64 125/134 2022-04-03T19:42:45.759 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel-1.2.11-17.el8.x86_64 2022-04-03T19:42:45.760 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:42:45.760 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2022-04-03T19:42:45.862 DEBUG:teuthology.parallel:result is None 2022-04-03T19:42:45.951 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : bison-3.0.4-10.el8.x86_64 126/134 2022-04-03T19:42:46.049 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 126/134 2022-04-03T19:42:46.258 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flex-2.6.1-9.el8.x86_64 127/134 2022-04-03T19:42:46.301 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 127/134 2022-04-03T19:42:46.436 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 128/134 2022-04-03T19:42:46.587 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-tui-1:1.30.0-7.el8.x86_64 129/134 2022-04-03T19:42:46.587 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 130/134 2022-04-03T19:42:46.871 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 130/134 2022-04-03T19:42:47.011 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-team-1:1.30.0-7.el8.x86_64 131/134 2022-04-03T19:42:47.049 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.30.0-7.el8.x86_64 132/134 2022-04-03T19:42:47.050 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-1:1.30.0-7.el8.x86_64 132/134 2022-04-03T19:42:47.614 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-1:1.30.0-7.el8.x86_64 132/134 2022-04-03T19:42:47.614 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : NetworkManager-libnm-1:1.30.0-7.el8.x86_64 133/134 2022-04-03T19:42:47.949 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: NetworkManager-libnm-1:1.30.0-7.el8.x86_64 133/134 2022-04-03T19:42:47.950 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 134/134 2022-04-03T19:42:48.628 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 134/134 2022-04-03T19:42:48.628 INFO:teuthology.orchestra.run.smithi077.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2022-04-03T19:42:48.629 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:42:49.495 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-2:17.0.0-11442.gcfb8f943.el8.x86_64 1/134 2022-04-03T19:42:49.495 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-base-2:17.0.0-11442.gcfb8f943.el8.x86_64 2/134 2022-04-03T19:42:49.495 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-common-2:17.0.0-11442.gcfb8f943.el8.x86_64 3/134 2022-04-03T19:42:49.496 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-fuse-2:17.0.0-11442.gcfb8f943.el8.x86_64 4/134 2022-04-03T19:42:49.496 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-11442.gcfb8 5/134 2022-04-03T19:42:49.496 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mds-2:17.0.0-11442.gcfb8f943.el8.x86_64 6/134 2022-04-03T19:42:49.496 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-2:17.0.0-11442.gcfb8f943.el8.x86_64 7/134 2022-04-03T19:42:49.497 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mon-2:17.0.0-11442.gcfb8f943.el8.x86_64 8/134 2022-04-03T19:42:49.497 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-osd-2:17.0.0-11442.gcfb8f943.el8.x86_64 9/134 2022-04-03T19:42:49.497 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-radosgw-2:17.0.0-11442.gcfb8f943.el8.x86_64 10/134 2022-04-03T19:42:49.497 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-selinux-2:17.0.0-11442.gcfb8f943.el8.x86_64 11/134 2022-04-03T19:42:49.497 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-test-2:17.0.0-11442.gcfb8f943.el8.x86_64 12/134 2022-04-03T19:42:49.498 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-mirror-2:17.0.0-11442.gcfb8f943.el8.x86_6 13/134 2022-04-03T19:42:49.498 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs-devel-2:17.0.0-11442.gcfb8f943.el8.x86 14/134 2022-04-03T19:42:49.498 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs2-2:17.0.0-11442.gcfb8f943.el8.x86_64 15/134 2022-04-03T19:42:49.498 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephsqlite-2:17.0.0-11442.gcfb8f943.el8.x86_6 16/134 2022-04-03T19:42:49.499 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados-devel-2:17.0.0-11442.gcfb8f943.el8.x86_ 17/134 2022-04-03T19:42:49.499 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libradosstriper1-2:17.0.0-11442.gcfb8f943.el8.x8 18/134 2022-04-03T19:42:49.499 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librgw2-2:17.0.0-11442.gcfb8f943.el8.x86_64 19/134 2022-04-03T19:42:49.499 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-argparse-2:17.0.0-11442.gcfb8f943.e 20/134 2022-04-03T19:42:49.500 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-common-2:17.0.0-11442.gcfb8f943.el8 21/134 2022-04-03T19:42:49.500 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cephfs-2:17.0.0-11442.gcfb8f943.el8.x86_ 22/134 2022-04-03T19:42:49.500 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rados-2:17.0.0-11442.gcfb8f943.el8.x86_6 23/134 2022-04-03T19:42:49.500 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rbd-2:17.0.0-11442.gcfb8f943.el8.x86_64 24/134 2022-04-03T19:42:49.501 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rgw-2:17.0.0-11442.gcfb8f943.el8.x86_64 25/134 2022-04-03T19:42:49.501 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-fuse-2:17.0.0-11442.gcfb8f943.el8.x86_64 26/134 2022-04-03T19:42:49.501 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-mirror-2:17.0.0-11442.gcfb8f943.el8.x86_64 27/134 2022-04-03T19:42:49.501 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-nbd-2:17.0.0-11442.gcfb8f943.el8.x86_64 28/134 2022-04-03T19:42:49.501 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-11442.gcfb8f943 29/134 2022-04-03T19:42:49.502 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-11442.gcfb8f943.el8.no 30/134 2022-04-03T19:42:49.502 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-11442.gcfb8f943.el8. 31/134 2022-04-03T19:42:49.502 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-11442.gcf 32/134 2022-04-03T19:42:49.502 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-11442.gcfb8f943.e 33/134 2022-04-03T19:42:49.503 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-rook-2:17.0.0-11442.gcfb8f943.el8.noarc 34/134 2022-04-03T19:42:49.503 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-11442.gcfb8f943. 35/134 2022-04-03T19:42:49.503 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-volume-2:17.0.0-11442.gcfb8f943.el8.noarch 36/134 2022-04-03T19:42:49.503 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephadm-2:17.0.0-11442.gcfb8f943.el8.noarch 37/134 2022-04-03T19:42:49.504 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephfs-top-2:17.0.0-11442.gcfb8f943.el8.noarch 38/134 2022-04-03T19:42:49.504 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2022-04-03T19:42:49.504 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2022-04-03T19:42:49.504 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2022-04-03T19:42:49.504 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2022-04-03T19:42:49.505 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2022-04-03T19:42:49.505 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2022-04-03T19:42:49.505 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2022-04-03T19:42:49.505 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2022-04-03T19:42:49.506 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2022-04-03T19:42:49.506 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2022-04-03T19:42:49.506 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2022-04-03T19:42:49.506 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2022-04-03T19:42:49.507 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2022-04-03T19:42:49.507 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2022-04-03T19:42:49.507 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2022-04-03T19:42:49.507 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2022-04-03T19:42:49.508 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2022-04-03T19:42:49.508 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2022-04-03T19:42:49.508 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2022-04-03T19:42:49.508 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2022-04-03T19:42:49.509 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2022-04-03T19:42:49.509 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2022-04-03T19:42:49.509 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2022-04-03T19:42:49.509 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2022-04-03T19:42:49.510 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2022-04-03T19:42:49.510 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2022-04-03T19:42:49.510 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/134 2022-04-03T19:42:49.510 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2022-04-03T19:42:49.511 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2022-04-03T19:42:49.511 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2022-04-03T19:42:49.511 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2022-04-03T19:42:49.511 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2022-04-03T19:42:49.511 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 71/134 2022-04-03T19:42:49.512 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 72/134 2022-04-03T19:42:49.512 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 73/134 2022-04-03T19:42:49.512 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 74/134 2022-04-03T19:42:49.512 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 75/134 2022-04-03T19:42:49.513 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 76/134 2022-04-03T19:42:49.513 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 77/134 2022-04-03T19:42:49.513 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 78/134 2022-04-03T19:42:49.513 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 79/134 2022-04-03T19:42:49.514 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 80/134 2022-04-03T19:42:49.514 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 81/134 2022-04-03T19:42:49.514 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 82/134 2022-04-03T19:42:49.514 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 83/134 2022-04-03T19:42:49.514 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 84/134 2022-04-03T19:42:49.515 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 85/134 2022-04-03T19:42:49.515 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 86/134 2022-04-03T19:42:49.515 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 87/134 2022-04-03T19:42:49.515 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 88/134 2022-04-03T19:42:49.515 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 89/134 2022-04-03T19:42:49.516 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 90/134 2022-04-03T19:42:49.516 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cryptography-3.2.1-4.el8.x86_64 91/134 2022-04-03T19:42:49.516 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 92/134 2022-04-03T19:42:49.516 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 93/134 2022-04-03T19:42:49.517 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 94/134 2022-04-03T19:42:49.517 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 95/134 2022-04-03T19:42:49.517 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 96/134 2022-04-03T19:42:49.517 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 97/134 2022-04-03T19:42:49.517 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flex-2.6.1-9.el8.x86_64 98/134 2022-04-03T19:42:49.518 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : jq-1.5-12.el8.x86_64 99/134 2022-04-03T19:42:49.518 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 100/134 2022-04-03T19:42:49.518 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 101/134 2022-04-03T19:42:49.518 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 102/134 2022-04-03T19:42:49.519 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : automake-1.16.1-7.el8.noarch 103/134 2022-04-03T19:42:49.519 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 104/134 2022-04-03T19:42:49.519 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 105/134 2022-04-03T19:42:49.519 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 106/134 2022-04-03T19:42:49.519 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 107/134 2022-04-03T19:42:49.520 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 108/134 2022-04-03T19:42:49.520 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 109/134 2022-04-03T19:42:49.520 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 110/134 2022-04-03T19:42:49.520 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 111/134 2022-04-03T19:42:49.521 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 112/134 2022-04-03T19:42:49.521 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 113/134 2022-04-03T19:42:49.521 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 114/134 2022-04-03T19:42:49.521 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 115/134 2022-04-03T19:42:49.521 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 116/134 2022-04-03T19:42:49.522 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 117/134 2022-04-03T19:42:49.522 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 118/134 2022-04-03T19:42:49.522 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 119/134 2022-04-03T19:42:49.522 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 120/134 2022-04-03T19:42:49.523 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : bison-3.0.4-10.el8.x86_64 121/134 2022-04-03T19:42:49.523 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 122/134 2022-04-03T19:42:49.523 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:17.0.0-11442.gcfb8f943.el8.x86_64 123/134 2022-04-03T19:42:49.523 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 124/134 2022-04-03T19:42:49.523 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:17.0.0-11442.gcfb8f943.el8.x86_64 125/134 2022-04-03T19:42:49.524 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 126/134 2022-04-03T19:42:49.524 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-team-1:1.30.0-13.el8_4.x86_64 127/134 2022-04-03T19:42:49.524 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-team-1:1.30.0-7.el8.x86_64 128/134 2022-04-03T19:42:49.524 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 129/134 2022-04-03T19:42:49.524 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-libnm-1:1.30.0-7.el8.x86_64 130/134 2022-04-03T19:42:49.525 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 131/134 2022-04-03T19:42:49.525 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-tui-1:1.30.0-7.el8.x86_64 132/134 2022-04-03T19:42:49.525 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-1:1.30.0-13.el8_4.x86_64 133/134 2022-04-03T19:43:07.017 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : NetworkManager-1:1.30.0-7.el8.x86_64 134/134 2022-04-03T19:43:07.018 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:07 ago on Sun 03 Apr 2022 07:43:00 PM UTC. 2022-04-03T19:43:10.667 INFO:teuthology.orchestra.run.smithi077.stdout:Installed products updated. 2022-04-03T19:43:10.794 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:10.794 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2022-04-03T19:43:10.794 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-1:1.30.0-13.el8_4.x86_64 2022-04-03T19:43:10.795 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-libnm-1:1.30.0-13.el8_4.x86_64 2022-04-03T19:43:10.795 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-team-1:1.30.0-13.el8_4.x86_64 2022-04-03T19:43:10.795 INFO:teuthology.orchestra.run.smithi077.stdout: NetworkManager-tui-1:1.30.0-13.el8_4.x86_64 2022-04-03T19:43:10.795 INFO:teuthology.orchestra.run.smithi077.stdout: librados2-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.796 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.796 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2022-04-03T19:43:10.796 INFO:teuthology.orchestra.run.smithi077.stdout: automake-1.16.1-7.el8.noarch 2022-04-03T19:43:10.797 INFO:teuthology.orchestra.run.smithi077.stdout: bison-3.0.4-10.el8.x86_64 2022-04-03T19:43:10.797 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.797 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.797 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.798 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.798 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards-2:17.0.0-11442.gcfb8f943.el8.noarch 2022-04-03T19:43:10.798 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.798 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.799 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.799 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm-2:17.0.0-11442.gcfb8f943.el8.noarch 2022-04-03T19:43:10.799 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard-2:17.0.0-11442.gcfb8f943.el8.noarch 2022-04-03T19:43:10.800 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local-2:17.0.0-11442.gcfb8f943.el8.noarch 2022-04-03T19:43:10.801 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core-2:17.0.0-11442.gcfb8f943.el8.noarch 2022-04-03T19:43:10.801 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook-2:17.0.0-11442.gcfb8f943.el8.noarch 2022-04-03T19:43:10.801 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.802 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.802 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts-2:17.0.0-11442.gcfb8f943.el8.noarch 2022-04-03T19:43:10.803 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.803 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.804 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.804 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-volume-2:17.0.0-11442.gcfb8f943.el8.noarch 2022-04-03T19:43:10.804 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm-2:17.0.0-11442.gcfb8f943.el8.noarch 2022-04-03T19:43:10.805 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-mirror-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.805 INFO:teuthology.orchestra.run.smithi077.stdout: cephfs-top-2:17.0.0-11442.gcfb8f943.el8.noarch 2022-04-03T19:43:10.805 INFO:teuthology.orchestra.run.smithi077.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2022-04-03T19:43:10.805 INFO:teuthology.orchestra.run.smithi077.stdout: flex-2.6.1-9.el8.x86_64 2022-04-03T19:43:10.806 INFO:teuthology.orchestra.run.smithi077.stdout: fuse-2.9.7-12.el8.x86_64 2022-04-03T19:43:10.806 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-04-03T19:43:10.806 INFO:teuthology.orchestra.run.smithi077.stdout: jq-1.5-12.el8.x86_64 2022-04-03T19:43:10.806 INFO:teuthology.orchestra.run.smithi077.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2022-04-03T19:43:10.807 INFO:teuthology.orchestra.run.smithi077.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2022-04-03T19:43:10.807 INFO:teuthology.orchestra.run.smithi077.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-04-03T19:43:10.807 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.807 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.808 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.808 INFO:teuthology.orchestra.run.smithi077.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2022-04-03T19:43:10.808 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran-8.4.1-1.el8.x86_64 2022-04-03T19:43:10.808 INFO:teuthology.orchestra.run.smithi077.stdout: liboath-2.6.2-3.el8.x86_64 2022-04-03T19:43:10.808 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath-8.4.1-1.el8.x86_64 2022-04-03T19:43:10.809 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-04-03T19:43:10.809 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.809 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.809 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka-0.11.4-1.el8.x86_64 2022-04-03T19:43:10.810 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.810 INFO:teuthology.orchestra.run.smithi077.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-04-03T19:43:10.810 INFO:teuthology.orchestra.run.smithi077.stdout: libsepol-devel-2.9-2.el8.x86_64 2022-04-03T19:43:10.810 INFO:teuthology.orchestra.run.smithi077.stdout: libtool-2.4.6-25.el8.x86_64 2022-04-03T19:43:10.810 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind-1.3.1-3.el8.x86_64 2022-04-03T19:43:10.811 INFO:teuthology.orchestra.run.smithi077.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2022-04-03T19:43:10.811 INFO:teuthology.orchestra.run.smithi077.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-04-03T19:43:10.811 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-04-03T19:43:10.811 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-04-03T19:43:10.812 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-0.3.12-1.el8.x86_64 2022-04-03T19:43:10.812 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-threads-0.3.12-1.el8.x86_64 2022-04-03T19:43:10.812 INFO:teuthology.orchestra.run.smithi077.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2022-04-03T19:43:10.812 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-04-03T19:43:10.812 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-04-03T19:43:10.813 INFO:teuthology.orchestra.run.smithi077.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-04-03T19:43:10.813 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-04-03T19:43:10.813 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel-2.5.1-5.el8.noarch 2022-04-03T19:43:10.813 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-04-03T19:43:10.814 INFO:teuthology.orchestra.run.smithi077.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-04-03T19:43:10.814 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-04-03T19:43:10.814 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.814 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.814 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.815 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-04-03T19:43:10.815 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-04-03T19:43:10.815 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-04-03T19:43:10.815 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-04-03T19:43:10.816 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography-3.2.1-4.el8.x86_64 2022-04-03T19:43:10.816 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-04-03T19:43:10.816 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-6.2-6.el8.noarch 2022-04-03T19:43:10.816 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-04-03T19:43:10.816 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2022-04-03T19:43:10.817 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-04-03T19:43:10.817 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-04-03T19:43:10.817 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-04-03T19:43:10.817 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-04-03T19:43:10.818 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-04-03T19:43:10.818 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako-1.0.6-13.el8.noarch 2022-04-03T19:43:10.818 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-04-03T19:43:10.818 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-04-03T19:43:10.818 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-04-03T19:43:10.819 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2022-04-03T19:43:10.819 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2022-04-03T19:43:10.819 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-04-03T19:43:10.819 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-04-03T19:43:10.820 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend-2.6-1.el8.noarch 2022-04-03T19:43:10.820 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-04-03T19:43:10.820 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-04-03T19:43:10.820 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-04-03T19:43:10.820 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-04-03T19:43:10.821 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser-2.14-14.el8.noarch 2022-04-03T19:43:10.821 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz-2017.2-9.el8.noarch 2022-04-03T19:43:10.821 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.821 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.822 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-04-03T19:43:10.822 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-04-03T19:43:10.822 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.822 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes-2.4.1-12.el8.noarch 2022-04-03T19:43:10.823 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa-4.8-1.el8.noarch 2022-04-03T19:43:10.823 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2022-04-03T19:43:10.823 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-04-03T19:43:10.823 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-04-03T19:43:10.823 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-04-03T19:43:10.824 INFO:teuthology.orchestra.run.smithi077.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-04-03T19:43:10.824 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-04-03T19:43:10.824 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-04-03T19:43:10.824 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-04-03T19:43:10.825 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-04-03T19:43:10.825 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-04-03T19:43:10.825 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-04-03T19:43:10.825 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.825 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.826 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd-2:17.0.0-11442.gcfb8f943.el8.x86_64 2022-04-03T19:43:10.826 INFO:teuthology.orchestra.run.smithi077.stdout: socat-1.7.3.3-2.el8.x86_64 2022-04-03T19:43:10.826 INFO:teuthology.orchestra.run.smithi077.stdout: thrift-0.13.0-2.el8.x86_64 2022-04-03T19:43:10.826 INFO:teuthology.orchestra.run.smithi077.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2022-04-03T19:43:10.827 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2022-04-03T19:43:10.827 INFO:teuthology.orchestra.run.smithi077.stdout: zlib-devel-1.2.11-17.el8.x86_64 2022-04-03T19:43:10.828 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:10.828 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2022-04-03T19:43:12.360 DEBUG:teuthology.parallel:result is None 2022-04-03T19:43:12.361 DEBUG:teuthology.orchestra.run.smithi077:> sudo lsb_release -is 2022-04-03T19:43:12.504 INFO:teuthology.orchestra.run.smithi077.stdout:RedHatEnterprise 2022-04-03T19:43:12.508 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-04-03T19:43:12.508 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=cfb8f943163b374162da0d7b0240f267dd46e4e1 2022-04-03T19:43:12.664 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-04-03T19:43:12.718 INFO:teuthology.orchestra.run.smithi077.stdout:17.0.0-11442.gcfb8f943.el8 2022-04-03T19:43:12.719 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-11442.gcfb8f943.el8 2022-04-03T19:43:12.719 INFO:teuthology.task.install:The correct ceph version 17.0.0-11442.gcfb8f943 is installed. 2022-04-03T19:43:12.720 DEBUG:teuthology.orchestra.run.smithi186:> sudo lsb_release -is 2022-04-03T19:43:12.780 INFO:teuthology.orchestra.run.smithi186.stdout:RedHatEnterprise 2022-04-03T19:43:12.781 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-04-03T19:43:12.782 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=cfb8f943163b374162da0d7b0240f267dd46e4e1 2022-04-03T19:43:12.964 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-04-03T19:43:12.986 INFO:teuthology.orchestra.run.smithi186.stdout:17.0.0-11442.gcfb8f943.el8 2022-04-03T19:43:12.986 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-11442.gcfb8f943.el8 2022-04-03T19:43:12.987 INFO:teuthology.task.install:The correct ceph version 17.0.0-11442.gcfb8f943 is installed. 2022-04-03T19:43:12.987 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-04-03T19:43:12.988 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:12.988 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-04-03T19:43:13.058 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:43:13.059 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-04-03T19:43:13.086 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-04-03T19:43:13.087 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:13.087 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2022-04-03T19:43:13.168 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-04-03T19:43:13.237 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:43:13.238 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2022-04-03T19:43:13.265 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-04-03T19:43:13.332 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-04-03T19:43:13.332 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:13.333 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2022-04-03T19:43:13.402 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-04-03T19:43:13.471 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:43:13.472 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2022-04-03T19:43:13.498 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-04-03T19:43:13.563 INFO:teuthology.run_tasks:Running task ceph... 2022-04-03T19:43:13.624 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2022-04-03T19:43:13.625 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /var/log/ceph 2022-04-03T19:43:13.629 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /var/log/ceph 2022-04-03T19:43:13.694 INFO:tasks.ceph:Disabling ceph logrotate... 2022-04-03T19:43:13.694 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/logrotate.d/ceph 2022-04-03T19:43:13.699 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/logrotate.d/ceph 2022-04-03T19:43:13.763 INFO:tasks.ceph:Creating extra log directories... 2022-04-03T19:43:13.764 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-04-03T19:43:13.768 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-04-03T19:43:13.864 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2022-04-03T19:43:13.865 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2022-04-03T19:43:13.865 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2022-04-03T19:43:13.865 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:13.866 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-04-03T19:43:13.909 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-04-03T19:43:14.118 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2022-04-03T19:43:14.188 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:43:14.188 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-04-03T19:43:14.203 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-04-03T19:43:14.313 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2022-04-03T19:43:14.344 INFO:tasks.ceph:Creating ceph cluster ceph... 2022-04-03T19:43:14.345 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-04-03T19:43:14.345 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2022-04-03_15:22:48-fs-wip-vshankar-testing-20220403-170344-testing-default-smithi/6775570', 'branch': 'wip-vshankar-testing-20220403-170344', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '6775570', 'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '26d8230c8f919f5e2d8da795fb3fbca12f8ebd85'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'vshankar-2022-04-03_15:22:48-fs-wip-vshankar-testing-20220403-170344-testing-default-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20220403-170344'}, 'ceph': {'cephfs': {'mds_dir_prefetch': True, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'cfb8f943163b374162da0d7b0240f267dd46e4e1'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'cfb8f943163b374162da0d7b0240f267dd46e4e1'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20220403-170344', 'sha1': 'cfb8f943163b374162da0d7b0240f267dd46e4e1'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 100, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'sha1': 'cfb8f943163b374162da0d7b0240f267dd46e4e1', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20220403-170344', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_cfb8f943163b374162da0d7b0240f267dd46e4e1/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': 'cfb8f943163b374162da0d7b0240f267dd46e4e1', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi077.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDC2KrHT+jt6QF/Qu3rNGoRnyFtqyrpecqHQBy+Ff7ubrJyqbuqVmcMVGk11XWW+H0KmzVLagrcZkVHJ3L0MLwB+aVdpulCHKEIHeVzMrLh7cV3DQ3RlQdL5uYg6yh138FzItXXVH/vKYeJwMQR5eQ+tuaivstmzCvmekQ2NtiNKJP3zD27if/5mXWVJtUT+ppHTEz9ZmtB6LRewmKTrqZXnsX0wm1iGNdJnoCuRw6nXwmtu2TV4rxyOqol/5MCG/sK2nEbKUD4KPS0K6WJZ8lSiWui2V+Ul8xdbGBOiXXkbpHMccLFaBa9QxEb+ywpf6vRBH+P7JLYFbMjvz3knoGkCyyxxFftdCLEibqxtXqhUmDzTCx6FkzDxKjtPzDC6sxU9VajeubSPC2sIJhoa2bPghkzQpCVUN6j0f0q6372woza/biE1NXtHdooSHlTWq0LAyhLRmvSIpw0gkJREQ6DurPfYgAawH/V4Z6ZMIN+EU2CIMQxFlyNiLPBqMQEE/8=', 'smithi186.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPXYqvDQAuAe1jDXYrnh9kUFrXHgN4RYdGIEPXNa2Wijxrc7Fuj/rNhJHK+7mjqgVENXYgLaIrKZL4GPCMjiF+kiJO/W3+zmzpQqU9MEEQHykotUYOmlDV6IgtHxF1UeU8kl7pG3b0JBtImYwCJAvlj2Sja7CYBygFSxq2EpUFhIIWuIw6iEO+YCFXn9O7meZ5B3rKu8cY9pICRJauSZiI+nIvNSaEWcvZEqVqUQnUakkLRBGvnt7PbMtgk08tzO0Nrtt+XLbJACBqxTy3M/G+QnM/+MAXrijFsqmSxOaEkoxxlCBjpw/J9Ig57dkpcMdE+7LR3rCDmFs5a8SMAJGuZetberuhGr+CScKT689t/6D9NAHFngiyU7HnGOeyrn7Dkkj8qV5HTz4L5jVn3Nsl7Wo3xmkZgCzESK8lRE8HhJD3MWktTFPvXKRsK75UGv0BrGk7Lxwq4xgVgEoaopsTSN8IURj5WXAiOnL68QR4xCWh0BK/bTNW8vZdKumm16c='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '26d8230c8f919f5e2d8da795fb3fbca12f8ebd85'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': 'cfb8f943163b374162da0d7b0240f267dd46e4e1'}}, {'ceph': None}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}}}], 'teuthology_branch': 'master', 'teuthology_sha1': 'd877b023166308f282187444de8e8e725ffb27d2', 'timestamp': '2022-04-03_15:22:48', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.22301'} 2022-04-03T19:43:14.346 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-04-03T19:43:14.351 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-04-03T19:43:14.393 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/run/ceph 2022-04-03T19:43:14.398 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/run/ceph 2022-04-03T19:43:14.463 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:14.464 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2022-04-03T19:43:14.508 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-04-03T19:43:14.508 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-04-03T19:43:14.552 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-04-03T19:43:14.553 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-03T19:43:14.553 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 450 Links: 1 2022-04-03T19:43:14.553 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-03T19:43:14.553 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2022-04-03T19:43:14.554 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2022-04-03 19:42:48.790652774 +0000 2022-04-03T19:43:14.554 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2022-04-03 19:40:04.154092210 +0000 2022-04-03T19:43:14.554 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2022-04-03 19:40:04.154092210 +0000 2022-04-03T19:43:14.554 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2022-04-03T19:43:14.579 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2022-04-03T19:43:14.579 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2022-04-03T19:43:14.579 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000992681 s, 516 kB/s 2022-04-03T19:43:14.588 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-04-03T19:43:14.673 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-04-03T19:43:14.673 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-03T19:43:14.674 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 456 Links: 1 2022-04-03T19:43:14.674 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-03T19:43:14.675 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2022-04-03T19:43:14.675 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2022-04-03 19:42:48.790652774 +0000 2022-04-03T19:43:14.676 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2022-04-03 19:40:04.704080444 +0000 2022-04-03T19:43:14.676 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2022-04-03 19:40:04.704080444 +0000 2022-04-03T19:43:14.676 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2022-04-03T19:43:14.699 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2022-04-03T19:43:14.699 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2022-04-03T19:43:14.700 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.00089631 s, 571 kB/s 2022-04-03T19:43:14.708 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-04-03T19:43:14.753 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-04-03T19:43:14.753 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-03T19:43:14.753 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 462 Links: 1 2022-04-03T19:43:14.754 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-03T19:43:14.755 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2022-04-03T19:43:14.755 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2022-04-03 19:42:48.790652774 +0000 2022-04-03T19:43:14.755 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2022-04-03 19:40:05.221069384 +0000 2022-04-03T19:43:14.755 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2022-04-03 19:40:05.221069384 +0000 2022-04-03T19:43:14.756 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2022-04-03T19:43:14.779 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2022-04-03T19:43:14.780 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2022-04-03T19:43:14.780 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000636926 s, 804 kB/s 2022-04-03T19:43:14.789 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-04-03T19:43:14.874 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-04-03T19:43:14.874 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-03T19:43:14.874 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 468 Links: 1 2022-04-03T19:43:14.874 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-03T19:43:14.875 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2022-04-03T19:43:14.875 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2022-04-03 19:42:48.790652774 +0000 2022-04-03T19:43:14.875 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2022-04-03 19:40:05.722058667 +0000 2022-04-03T19:43:14.875 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2022-04-03 19:40:05.722058667 +0000 2022-04-03T19:43:14.875 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2022-04-03T19:43:14.900 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2022-04-03T19:43:14.900 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2022-04-03T19:43:14.900 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000671223 s, 763 kB/s 2022-04-03T19:43:14.909 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-04-03T19:43:14.909 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:43:14.910 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2022-04-03T19:43:14.926 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-04-03T19:43:14.926 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-04-03T19:43:14.982 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-04-03T19:43:14.982 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-03T19:43:14.982 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 6h/6d Inode: 60720 Links: 1 2022-04-03T19:43:14.983 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-03T19:43:14.983 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2022-04-03T19:43:14.983 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2022-04-03 19:42:26.748790181 +0000 2022-04-03T19:43:14.983 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2022-04-03 19:39:37.390306530 +0000 2022-04-03T19:43:14.984 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2022-04-03 19:39:37.390306530 +0000 2022-04-03T19:43:14.984 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2022-04-03T19:43:14.993 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2022-04-03T19:43:14.993 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2022-04-03T19:43:14.993 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000237548 s, 2.2 MB/s 2022-04-03T19:43:14.996 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-04-03T19:43:15.051 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-04-03T19:43:15.051 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-03T19:43:15.051 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 6h/6d Inode: 57218 Links: 1 2022-04-03T19:43:15.051 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-03T19:43:15.052 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2022-04-03T19:43:15.052 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2022-04-03 19:42:26.748790181 +0000 2022-04-03T19:43:15.052 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2022-04-03 19:39:37.739300231 +0000 2022-04-03T19:43:15.052 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2022-04-03 19:39:37.739300231 +0000 2022-04-03T19:43:15.053 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2022-04-03T19:43:15.061 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2022-04-03T19:43:15.061 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2022-04-03T19:43:15.061 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000238484 s, 2.1 MB/s 2022-04-03T19:43:15.064 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-04-03T19:43:15.118 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-04-03T19:43:15.119 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-03T19:43:15.119 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 6h/6d Inode: 60064 Links: 1 2022-04-03T19:43:15.119 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-03T19:43:15.120 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2022-04-03T19:43:15.120 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2022-04-03 19:42:26.748790181 +0000 2022-04-03T19:43:15.120 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2022-04-03 19:39:38.081294060 +0000 2022-04-03T19:43:15.120 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2022-04-03 19:39:38.081294060 +0000 2022-04-03T19:43:15.120 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2022-04-03T19:43:15.128 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2022-04-03T19:43:15.128 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2022-04-03T19:43:15.129 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000230643 s, 2.2 MB/s 2022-04-03T19:43:15.131 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-04-03T19:43:15.186 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-04-03T19:43:15.186 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-04-03T19:43:15.187 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 6h/6d Inode: 59122 Links: 1 2022-04-03T19:43:15.187 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-03T19:43:15.187 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2022-04-03T19:43:15.187 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2022-04-03 19:42:26.748790181 +0000 2022-04-03T19:43:15.188 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2022-04-03 19:39:38.432287727 +0000 2022-04-03T19:43:15.188 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2022-04-03 19:39:38.432287727 +0000 2022-04-03T19:43:15.188 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2022-04-03T19:43:15.196 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2022-04-03T19:43:15.196 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2022-04-03T19:43:15.196 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000186261 s, 2.7 MB/s 2022-04-03T19:43:15.199 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-04-03T19:43:15.199 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2022-04-03T19:43:15.199 INFO:tasks.ceph:Generating config... 2022-04-03T19:43:15.201 INFO:tasks.ceph:[client] client mount timeout = 600 2022-04-03T19:43:15.201 INFO:tasks.ceph:[client] debug client = 20 2022-04-03T19:43:15.201 INFO:tasks.ceph:[client] debug ms = 1 2022-04-03T19:43:15.202 INFO:tasks.ceph:[client] rados mon op timeout = 900 2022-04-03T19:43:15.202 INFO:tasks.ceph:[client] rados osd op timeout = 900 2022-04-03T19:43:15.202 INFO:tasks.ceph:[global] ms die on skipped message = False 2022-04-03T19:43:15.203 INFO:tasks.ceph:[mds] debug mds = 20 2022-04-03T19:43:15.203 INFO:tasks.ceph:[mds] debug ms = 1 2022-04-03T19:43:15.203 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2022-04-03T19:43:15.204 INFO:tasks.ceph:[mds] mds bal merge size = 5 2022-04-03T19:43:15.204 INFO:tasks.ceph:[mds] mds bal split bits = 3 2022-04-03T19:43:15.204 INFO:tasks.ceph:[mds] mds bal split size = 100 2022-04-03T19:43:15.205 INFO:tasks.ceph:[mds] mds debug frag = True 2022-04-03T19:43:15.205 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2022-04-03T19:43:15.205 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = False 2022-04-03T19:43:15.205 INFO:tasks.ceph:[mds] mds op complaint time = 180 2022-04-03T19:43:15.206 INFO:tasks.ceph:[mds] mds verify scatter = True 2022-04-03T19:43:15.206 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2022-04-03T19:43:15.206 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2022-04-03T19:43:15.207 INFO:tasks.ceph:[mds] ms inject delay max = 1 2022-04-03T19:43:15.207 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2022-04-03T19:43:15.207 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2022-04-03T19:43:15.207 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2022-04-03T19:43:15.208 INFO:tasks.ceph:[mds] osd op complaint time = 180 2022-04-03T19:43:15.208 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2022-04-03T19:43:15.208 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2022-04-03T19:43:15.208 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-04-03T19:43:15.209 INFO:tasks.ceph:[mgr] debug ms = 1 2022-04-03T19:43:15.209 INFO:tasks.ceph:[mon] debug mon = 20 2022-04-03T19:43:15.209 INFO:tasks.ceph:[mon] debug ms = 1 2022-04-03T19:43:15.210 INFO:tasks.ceph:[mon] debug paxos = 20 2022-04-03T19:43:15.210 INFO:tasks.ceph:[mon] mon op complaint time = 120 2022-04-03T19:43:15.210 INFO:tasks.ceph:[osd] bdev async discard = True 2022-04-03T19:43:15.211 INFO:tasks.ceph:[osd] bdev enable discard = True 2022-04-03T19:43:15.211 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2022-04-03T19:43:15.211 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2022-04-03T19:43:15.212 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2022-04-03T19:43:15.212 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2022-04-03T19:43:15.212 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2022-04-03T19:43:15.213 INFO:tasks.ceph:[osd] debug ms = 1 2022-04-03T19:43:15.213 INFO:tasks.ceph:[osd] debug osd = 20 2022-04-03T19:43:15.213 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2022-04-03T19:43:15.214 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2022-04-03T19:43:15.214 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2022-04-03T19:43:15.214 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2022-04-03T19:43:15.215 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2022-04-03T19:43:15.215 INFO:tasks.ceph:[osd] ms inject delay max = 1 2022-04-03T19:43:15.215 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2022-04-03T19:43:15.215 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2022-04-03T19:43:15.216 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2022-04-03T19:43:15.216 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2022-04-03T19:43:15.216 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2022-04-03T19:43:15.217 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2022-04-03T19:43:15.217 INFO:tasks.ceph:[osd] osd op complaint time = 180 2022-04-03T19:43:15.217 INFO:tasks.ceph:Setting up mon.a... 2022-04-03T19:43:15.218 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2022-04-03T19:43:15.330 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/ceph.keyring 2022-04-03T19:43:15.337 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2022-04-03T19:43:15.439 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-04-03T19:43:15.508 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.77'), ('mon.b', '172.21.15.186'), ('mon.c', '[v2:172.21.15.186:3301,v1:172.21.15.186:6790]')] 2022-04-03T19:43:15.509 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.77,172.21.15.186,[v2:172.21.15.186:3301,v1:172.21.15.186:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2022-04-03T19:43:15.510 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:15.511 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2022-04-03T19:43:15.554 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.77 --add b 172.21.15.186 --addv c '[v2:172.21.15.186:3301,v1:172.21.15.186:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2022-04-03T19:43:15.637 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2022-04-03T19:43:15.638 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: generated fsid 89c3a3f4-599d-4c4a-8f5a-7180eafb7292 2022-04-03T19:43:15.638 INFO:teuthology.orchestra.run.smithi077.stdout:setting min_mon_release = octopus 2022-04-03T19:43:15.638 INFO:teuthology.orchestra.run.smithi077.stdout:epoch 0 2022-04-03T19:43:15.638 INFO:teuthology.orchestra.run.smithi077.stdout:fsid 89c3a3f4-599d-4c4a-8f5a-7180eafb7292 2022-04-03T19:43:15.639 INFO:teuthology.orchestra.run.smithi077.stdout:last_changed 2022-04-03T19:43:15.637879+0000 2022-04-03T19:43:15.639 INFO:teuthology.orchestra.run.smithi077.stdout:created 2022-04-03T19:43:15.637879+0000 2022-04-03T19:43:15.639 INFO:teuthology.orchestra.run.smithi077.stdout:min_mon_release 15 (octopus) 2022-04-03T19:43:15.639 INFO:teuthology.orchestra.run.smithi077.stdout:election_strategy: 1 2022-04-03T19:43:15.639 INFO:teuthology.orchestra.run.smithi077.stdout:0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2022-04-03T19:43:15.640 INFO:teuthology.orchestra.run.smithi077.stdout:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2022-04-03T19:43:15.640 INFO:teuthology.orchestra.run.smithi077.stdout:2: [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] mon.c 2022-04-03T19:43:15.640 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2022-04-03T19:43:15.642 DEBUG:teuthology.orchestra.run.smithi077:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2022-04-03T19:43:15.685 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 89c3a3f4-599d-4c4a-8f5a-7180eafb7292... 2022-04-03T19:43:15.687 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-04-03T19:43:15.692 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-04-03T19:43:15.736 INFO:teuthology.orchestra.run.smithi186.stdout:[global] 2022-04-03T19:43:15.737 INFO:teuthology.orchestra.run.smithi186.stdout: chdir = "" 2022-04-03T19:43:15.737 INFO:teuthology.orchestra.run.smithi186.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-04-03T19:43:15.737 INFO:teuthology.orchestra.run.smithi186.stdout: auth supported = cephx 2022-04-03T19:43:15.738 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.738 INFO:teuthology.orchestra.run.smithi186.stdout: filestore xattr use omap = true 2022-04-03T19:43:15.738 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.738 INFO:teuthology.orchestra.run.smithi186.stdout: mon clock drift allowed = 1.000 2022-04-03T19:43:15.738 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.739 INFO:teuthology.orchestra.run.smithi186.stdout: osd crush chooseleaf type = 0 2022-04-03T19:43:15.739 INFO:teuthology.orchestra.run.smithi186.stdout: auth debug = true 2022-04-03T19:43:15.739 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.739 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on old message = true 2022-04-03T19:43:15.739 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on bug = true 2022-04-03T19:43:15.740 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.740 INFO:teuthology.orchestra.run.smithi186.stdout: mon max pg per osd = 10000 # >= luminous 2022-04-03T19:43:15.740 INFO:teuthology.orchestra.run.smithi186.stdout: mon pg warn max object skew = 0 2022-04-03T19:43:15.740 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.741 INFO:teuthology.orchestra.run.smithi186.stdout: # disable pg_autoscaler by default for new pools 2022-04-03T19:43:15.741 INFO:teuthology.orchestra.run.smithi186.stdout: osd_pool_default_pg_autoscale_mode = off 2022-04-03T19:43:15.741 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.742 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default size = 2 2022-04-03T19:43:15.742 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.742 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow primary affinity = true 2022-04-03T19:43:15.743 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow pg remap = true 2022-04-03T19:43:15.743 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on legacy crush tunables = false 2022-04-03T19:43:15.743 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on crush straw calc version zero = false 2022-04-03T19:43:15.743 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on no sortbitwise = false 2022-04-03T19:43:15.743 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on osd down out interval zero = false 2022-04-03T19:43:15.744 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on too few osds = false 2022-04-03T19:43:15.744 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-04-03T19:43:15.744 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_no_redundancy = false 2022-04-03T19:43:15.744 INFO:teuthology.orchestra.run.smithi186.stdout: mon_allow_pool_size_one = true 2022-04-03T19:43:15.744 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.745 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-04-03T19:43:15.745 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.745 INFO:teuthology.orchestra.run.smithi186.stdout: osd default data pool replay window = 5 2022-04-03T19:43:15.745 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.745 INFO:teuthology.orchestra.run.smithi186.stdout: mon allow pool delete = true 2022-04-03T19:43:15.746 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.746 INFO:teuthology.orchestra.run.smithi186.stdout: mon cluster log file level = debug 2022-04-03T19:43:15.746 INFO:teuthology.orchestra.run.smithi186.stdout: debug asserts on shutdown = true 2022-04-03T19:43:15.746 INFO:teuthology.orchestra.run.smithi186.stdout: mon health detail to clog = false 2022-04-03T19:43:15.747 INFO:teuthology.orchestra.run.smithi186.stdout: mon host = "172.21.15.77,172.21.15.186,[v2:172.21.15.186:3301,v1:172.21.15.186:6790]" 2022-04-03T19:43:15.747 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on skipped message = False 2022-04-03T19:43:15.747 INFO:teuthology.orchestra.run.smithi186.stdout: fsid = 89c3a3f4-599d-4c4a-8f5a-7180eafb7292 2022-04-03T19:43:15.747 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.747 INFO:teuthology.orchestra.run.smithi186.stdout:[osd] 2022-04-03T19:43:15.748 INFO:teuthology.orchestra.run.smithi186.stdout: osd journal size = 100 2022-04-03T19:43:15.748 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.748 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub load threshold = 5.0 2022-04-03T19:43:15.748 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub max interval = 600 2022-04-03T19:43:15.748 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.749 INFO:teuthology.orchestra.run.smithi186.stdout: osd recover clone overlap = true 2022-04-03T19:43:15.749 INFO:teuthology.orchestra.run.smithi186.stdout: osd recovery max chunk = 1048576 2022-04-03T19:43:15.749 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.749 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug shutdown = true 2022-04-03T19:43:15.749 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug op order = true 2022-04-03T19:43:15.750 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug verify stray on activate = true 2022-04-03T19:43:15.750 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.750 INFO:teuthology.orchestra.run.smithi186.stdout: osd open classes on start = true 2022-04-03T19:43:15.750 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug pg log writeout = true 2022-04-03T19:43:15.750 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.751 INFO:teuthology.orchestra.run.smithi186.stdout: osd deep scrub update digest min age = 30 2022-04-03T19:43:15.751 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.751 INFO:teuthology.orchestra.run.smithi186.stdout: osd map max advance = 10 2022-04-03T19:43:15.751 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.752 INFO:teuthology.orchestra.run.smithi186.stdout: journal zero on create = true 2022-04-03T19:43:15.752 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.752 INFO:teuthology.orchestra.run.smithi186.stdout: filestore ondisk finisher threads = 3 2022-04-03T19:43:15.752 INFO:teuthology.orchestra.run.smithi186.stdout: filestore apply finisher threads = 3 2022-04-03T19:43:15.752 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.753 INFO:teuthology.orchestra.run.smithi186.stdout: bdev debug aio = true 2022-04-03T19:43:15.753 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug misdirected ops = true 2022-04-03T19:43:15.753 INFO:teuthology.orchestra.run.smithi186.stdout: bdev async discard = True 2022-04-03T19:43:15.753 INFO:teuthology.orchestra.run.smithi186.stdout: bdev enable discard = True 2022-04-03T19:43:15.754 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore allocator = bitmap 2022-04-03T19:43:15.754 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore block size = 96636764160 2022-04-03T19:43:15.754 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore fsck on mount = True 2022-04-03T19:43:15.754 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluefs = 1/20 2022-04-03T19:43:15.755 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluestore = 1/20 2022-04-03T19:43:15.755 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2022-04-03T19:43:15.755 INFO:teuthology.orchestra.run.smithi186.stdout: debug osd = 20 2022-04-03T19:43:15.755 INFO:teuthology.orchestra.run.smithi186.stdout: debug rocksdb = 4/10 2022-04-03T19:43:15.756 INFO:teuthology.orchestra.run.smithi186.stdout: mon client directed command retry = 5 2022-04-03T19:43:15.756 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd backfillfull_ratio = 0.85 2022-04-03T19:43:15.756 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd full ratio = 0.9 2022-04-03T19:43:15.756 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd nearfull ratio = 0.8 2022-04-03T19:43:15.756 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject delay max = 1 2022-04-03T19:43:15.757 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject delay probability = 0.005 2022-04-03T19:43:15.757 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject delay type = client mds 2022-04-03T19:43:15.757 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject socket failures = 2500 2022-04-03T19:43:15.757 INFO:teuthology.orchestra.run.smithi186.stdout: osd failsafe full ratio = 0.95 2022-04-03T19:43:15.757 INFO:teuthology.orchestra.run.smithi186.stdout: osd max markdown count = 1000 2022-04-03T19:43:15.758 INFO:teuthology.orchestra.run.smithi186.stdout: osd objectstore = bluestore 2022-04-03T19:43:15.758 INFO:teuthology.orchestra.run.smithi186.stdout: osd op complaint time = 180 2022-04-03T19:43:15.758 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.758 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr] 2022-04-03T19:43:15.758 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2022-04-03T19:43:15.759 INFO:teuthology.orchestra.run.smithi186.stdout: debug mgr = 20 2022-04-03T19:43:15.759 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2022-04-03T19:43:15.759 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2022-04-03T19:43:15.759 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2022-04-03T19:43:15.759 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2022-04-03T19:43:15.760 INFO:teuthology.orchestra.run.smithi186.stdout: mgr/telemetry/nag = false 2022-04-03T19:43:15.760 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.760 INFO:teuthology.orchestra.run.smithi186.stdout:[mon] 2022-04-03T19:43:15.760 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2022-04-03T19:43:15.760 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2022-04-03T19:43:15.761 INFO:teuthology.orchestra.run.smithi186.stdout: debug paxos = 20 2022-04-03T19:43:15.761 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2022-04-03T19:43:15.761 INFO:teuthology.orchestra.run.smithi186.stdout: mon data avail warn = 5 2022-04-03T19:43:15.761 INFO:teuthology.orchestra.run.smithi186.stdout: mon mgr mkfs grace = 240 2022-04-03T19:43:15.761 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2022-04-03T19:43:15.762 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd reporter subtree level = osd 2022-04-03T19:43:15.762 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd prime pg temp = true 2022-04-03T19:43:15.762 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2022-04-03T19:43:15.762 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.762 INFO:teuthology.orchestra.run.smithi186.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-04-03T19:43:15.763 INFO:teuthology.orchestra.run.smithi186.stdout: auth mon ticket ttl = 660 # 11m 2022-04-03T19:43:15.763 INFO:teuthology.orchestra.run.smithi186.stdout: auth service ticket ttl = 240 # 4m 2022-04-03T19:43:15.763 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.763 INFO:teuthology.orchestra.run.smithi186.stdout: # don't complain about insecure global_id in the test suite 2022-04-03T19:43:15.764 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-04-03T19:43:15.764 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-04-03T19:43:15.764 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.764 INFO:teuthology.orchestra.run.smithi186.stdout: # 1m isn't quite enough 2022-04-03T19:43:15.764 INFO:teuthology.orchestra.run.smithi186.stdout: mon_down_mkfs_grace = 2m 2022-04-03T19:43:15.765 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.765 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_filestore_osds = false 2022-04-03T19:43:15.765 INFO:teuthology.orchestra.run.smithi186.stdout: mon op complaint time = 120 2022-04-03T19:43:15.765 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-04-03T19:43:15.765 INFO:teuthology.orchestra.run.smithi186.stdout:[client] 2022-04-03T19:43:15.766 INFO:teuthology.orchestra.run.smithi186.stdout: rgw cache enabled = true 2022-04-03T19:43:15.766 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable ops log = true 2022-04-03T19:43:15.766 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable usage log = true 2022-04-03T19:43:15.766 INFO:teuthology.orchestra.run.smithi186.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-04-03T19:43:15.766 INFO:teuthology.orchestra.run.smithi186.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-04-03T19:43:15.767 INFO:teuthology.orchestra.run.smithi186.stdout: client mount timeout = 600 2022-04-03T19:43:15.767 INFO:teuthology.orchestra.run.smithi186.stdout: debug client = 20 2022-04-03T19:43:15.767 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2022-04-03T19:43:15.767 INFO:teuthology.orchestra.run.smithi186.stdout: rados mon op timeout = 900 2022-04-03T19:43:15.767 INFO:teuthology.orchestra.run.smithi186.stdout: rados osd op timeout = 900 2022-04-03T19:43:15.768 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.a] 2022-04-03T19:43:15.768 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.b] 2022-04-03T19:43:15.768 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.c] 2022-04-03T19:43:15.768 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.a] 2022-04-03T19:43:15.768 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.c] 2022-04-03T19:43:15.769 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.e] 2022-04-03T19:43:15.769 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.b] 2022-04-03T19:43:15.769 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.d] 2022-04-03T19:43:15.769 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.f] 2022-04-03T19:43:15.769 INFO:teuthology.orchestra.run.smithi186.stdout:[mds] 2022-04-03T19:43:15.770 INFO:teuthology.orchestra.run.smithi186.stdout: debug mds = 20 2022-04-03T19:43:15.770 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2022-04-03T19:43:15.770 INFO:teuthology.orchestra.run.smithi186.stdout: mds bal fragment size max = 10000 2022-04-03T19:43:15.770 INFO:teuthology.orchestra.run.smithi186.stdout: mds bal merge size = 5 2022-04-03T19:43:15.770 INFO:teuthology.orchestra.run.smithi186.stdout: mds bal split bits = 3 2022-04-03T19:43:15.771 INFO:teuthology.orchestra.run.smithi186.stdout: mds bal split size = 100 2022-04-03T19:43:15.771 INFO:teuthology.orchestra.run.smithi186.stdout: mds debug frag = True 2022-04-03T19:43:15.771 INFO:teuthology.orchestra.run.smithi186.stdout: mds debug scatterstat = True 2022-04-03T19:43:15.771 INFO:teuthology.orchestra.run.smithi186.stdout: mds oft prefetch dirfrags = False 2022-04-03T19:43:15.771 INFO:teuthology.orchestra.run.smithi186.stdout: mds op complaint time = 180 2022-04-03T19:43:15.772 INFO:teuthology.orchestra.run.smithi186.stdout: mds verify scatter = True 2022-04-03T19:43:15.772 INFO:teuthology.orchestra.run.smithi186.stdout: mds_sleep_rank_change = 5000000.0 2022-04-03T19:43:15.772 INFO:teuthology.orchestra.run.smithi186.stdout: mon client directed command retry = 5 2022-04-03T19:43:15.772 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject delay max = 1 2022-04-03T19:43:15.772 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject delay probability = 0.005 2022-04-03T19:43:15.773 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject delay type = client mds osd 2022-04-03T19:43:15.773 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject socket failures = 2500 2022-04-03T19:43:15.773 INFO:teuthology.orchestra.run.smithi186.stdout: osd op complaint time = 180 2022-04-03T19:43:15.773 INFO:teuthology.orchestra.run.smithi186.stdout: rados mon op timeout = 900 2022-04-03T19:43:15.773 INFO:teuthology.orchestra.run.smithi186.stdout: rados osd op timeout = 900 2022-04-03T19:43:15.809 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2022-04-03T19:43:15.810 INFO:teuthology.orchestra.run.smithi077.stdout: chdir = "" 2022-04-03T19:43:15.810 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-04-03T19:43:15.810 INFO:teuthology.orchestra.run.smithi077.stdout: auth supported = cephx 2022-04-03T19:43:15.810 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.811 INFO:teuthology.orchestra.run.smithi077.stdout: filestore xattr use omap = true 2022-04-03T19:43:15.811 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.811 INFO:teuthology.orchestra.run.smithi077.stdout: mon clock drift allowed = 1.000 2022-04-03T19:43:15.812 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.812 INFO:teuthology.orchestra.run.smithi077.stdout: osd crush chooseleaf type = 0 2022-04-03T19:43:15.812 INFO:teuthology.orchestra.run.smithi077.stdout: auth debug = true 2022-04-03T19:43:15.812 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.813 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on old message = true 2022-04-03T19:43:15.813 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on bug = true 2022-04-03T19:43:15.813 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.813 INFO:teuthology.orchestra.run.smithi077.stdout: mon max pg per osd = 10000 # >= luminous 2022-04-03T19:43:15.814 INFO:teuthology.orchestra.run.smithi077.stdout: mon pg warn max object skew = 0 2022-04-03T19:43:15.814 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.815 INFO:teuthology.orchestra.run.smithi077.stdout: # disable pg_autoscaler by default for new pools 2022-04-03T19:43:15.815 INFO:teuthology.orchestra.run.smithi077.stdout: osd_pool_default_pg_autoscale_mode = off 2022-04-03T19:43:15.815 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.815 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default size = 2 2022-04-03T19:43:15.816 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.816 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow primary affinity = true 2022-04-03T19:43:15.816 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow pg remap = true 2022-04-03T19:43:15.816 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on legacy crush tunables = false 2022-04-03T19:43:15.816 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on crush straw calc version zero = false 2022-04-03T19:43:15.817 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on no sortbitwise = false 2022-04-03T19:43:15.817 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on osd down out interval zero = false 2022-04-03T19:43:15.817 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on too few osds = false 2022-04-03T19:43:15.817 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-04-03T19:43:15.818 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_no_redundancy = false 2022-04-03T19:43:15.818 INFO:teuthology.orchestra.run.smithi077.stdout: mon_allow_pool_size_one = true 2022-04-03T19:43:15.818 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.818 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2022-04-03T19:43:15.819 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.819 INFO:teuthology.orchestra.run.smithi077.stdout: osd default data pool replay window = 5 2022-04-03T19:43:15.819 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.820 INFO:teuthology.orchestra.run.smithi077.stdout: mon allow pool delete = true 2022-04-03T19:43:15.820 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.820 INFO:teuthology.orchestra.run.smithi077.stdout: mon cluster log file level = debug 2022-04-03T19:43:15.820 INFO:teuthology.orchestra.run.smithi077.stdout: debug asserts on shutdown = true 2022-04-03T19:43:15.821 INFO:teuthology.orchestra.run.smithi077.stdout: mon health detail to clog = false 2022-04-03T19:43:15.821 INFO:teuthology.orchestra.run.smithi077.stdout: mon host = "172.21.15.77,172.21.15.186,[v2:172.21.15.186:3301,v1:172.21.15.186:6790]" 2022-04-03T19:43:15.821 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on skipped message = False 2022-04-03T19:43:15.821 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = 89c3a3f4-599d-4c4a-8f5a-7180eafb7292 2022-04-03T19:43:15.822 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.822 INFO:teuthology.orchestra.run.smithi077.stdout:[osd] 2022-04-03T19:43:15.822 INFO:teuthology.orchestra.run.smithi077.stdout: osd journal size = 100 2022-04-03T19:43:15.822 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.823 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub load threshold = 5.0 2022-04-03T19:43:15.823 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub max interval = 600 2022-04-03T19:43:15.823 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.823 INFO:teuthology.orchestra.run.smithi077.stdout: osd recover clone overlap = true 2022-04-03T19:43:15.823 INFO:teuthology.orchestra.run.smithi077.stdout: osd recovery max chunk = 1048576 2022-04-03T19:43:15.824 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.824 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug shutdown = true 2022-04-03T19:43:15.824 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug op order = true 2022-04-03T19:43:15.825 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug verify stray on activate = true 2022-04-03T19:43:15.825 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.825 INFO:teuthology.orchestra.run.smithi077.stdout: osd open classes on start = true 2022-04-03T19:43:15.826 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug pg log writeout = true 2022-04-03T19:43:15.826 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.826 INFO:teuthology.orchestra.run.smithi077.stdout: osd deep scrub update digest min age = 30 2022-04-03T19:43:15.826 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.827 INFO:teuthology.orchestra.run.smithi077.stdout: osd map max advance = 10 2022-04-03T19:43:15.827 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.827 INFO:teuthology.orchestra.run.smithi077.stdout: journal zero on create = true 2022-04-03T19:43:15.827 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.827 INFO:teuthology.orchestra.run.smithi077.stdout: filestore ondisk finisher threads = 3 2022-04-03T19:43:15.828 INFO:teuthology.orchestra.run.smithi077.stdout: filestore apply finisher threads = 3 2022-04-03T19:43:15.828 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.828 INFO:teuthology.orchestra.run.smithi077.stdout: bdev debug aio = true 2022-04-03T19:43:15.828 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug misdirected ops = true 2022-04-03T19:43:15.829 INFO:teuthology.orchestra.run.smithi077.stdout: bdev async discard = True 2022-04-03T19:43:15.829 INFO:teuthology.orchestra.run.smithi077.stdout: bdev enable discard = True 2022-04-03T19:43:15.829 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore allocator = bitmap 2022-04-03T19:43:15.829 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore block size = 96636764160 2022-04-03T19:43:15.829 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore fsck on mount = True 2022-04-03T19:43:15.830 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluefs = 1/20 2022-04-03T19:43:15.830 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluestore = 1/20 2022-04-03T19:43:15.830 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2022-04-03T19:43:15.830 INFO:teuthology.orchestra.run.smithi077.stdout: debug osd = 20 2022-04-03T19:43:15.831 INFO:teuthology.orchestra.run.smithi077.stdout: debug rocksdb = 4/10 2022-04-03T19:43:15.831 INFO:teuthology.orchestra.run.smithi077.stdout: mon client directed command retry = 5 2022-04-03T19:43:15.831 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd backfillfull_ratio = 0.85 2022-04-03T19:43:15.831 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd full ratio = 0.9 2022-04-03T19:43:15.831 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd nearfull ratio = 0.8 2022-04-03T19:43:15.832 INFO:teuthology.orchestra.run.smithi077.stdout: ms inject delay max = 1 2022-04-03T19:43:15.832 INFO:teuthology.orchestra.run.smithi077.stdout: ms inject delay probability = 0.005 2022-04-03T19:43:15.832 INFO:teuthology.orchestra.run.smithi077.stdout: ms inject delay type = client mds 2022-04-03T19:43:15.832 INFO:teuthology.orchestra.run.smithi077.stdout: ms inject socket failures = 2500 2022-04-03T19:43:15.832 INFO:teuthology.orchestra.run.smithi077.stdout: osd failsafe full ratio = 0.95 2022-04-03T19:43:15.833 INFO:teuthology.orchestra.run.smithi077.stdout: osd max markdown count = 1000 2022-04-03T19:43:15.833 INFO:teuthology.orchestra.run.smithi077.stdout: osd objectstore = bluestore 2022-04-03T19:43:15.833 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2022-04-03T19:43:15.833 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.834 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr] 2022-04-03T19:43:15.834 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2022-04-03T19:43:15.834 INFO:teuthology.orchestra.run.smithi077.stdout: debug mgr = 20 2022-04-03T19:43:15.834 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2022-04-03T19:43:15.834 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2022-04-03T19:43:15.835 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2022-04-03T19:43:15.835 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2022-04-03T19:43:15.835 INFO:teuthology.orchestra.run.smithi077.stdout: mgr/telemetry/nag = false 2022-04-03T19:43:15.836 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.836 INFO:teuthology.orchestra.run.smithi077.stdout:[mon] 2022-04-03T19:43:15.836 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2022-04-03T19:43:15.836 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2022-04-03T19:43:15.837 INFO:teuthology.orchestra.run.smithi077.stdout: debug paxos = 20 2022-04-03T19:43:15.837 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2022-04-03T19:43:15.837 INFO:teuthology.orchestra.run.smithi077.stdout: mon data avail warn = 5 2022-04-03T19:43:15.837 INFO:teuthology.orchestra.run.smithi077.stdout: mon mgr mkfs grace = 240 2022-04-03T19:43:15.838 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2022-04-03T19:43:15.838 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd reporter subtree level = osd 2022-04-03T19:43:15.838 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd prime pg temp = true 2022-04-03T19:43:15.838 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2022-04-03T19:43:15.838 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.839 INFO:teuthology.orchestra.run.smithi077.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-04-03T19:43:15.839 INFO:teuthology.orchestra.run.smithi077.stdout: auth mon ticket ttl = 660 # 11m 2022-04-03T19:43:15.839 INFO:teuthology.orchestra.run.smithi077.stdout: auth service ticket ttl = 240 # 4m 2022-04-03T19:43:15.839 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.839 INFO:teuthology.orchestra.run.smithi077.stdout: # don't complain about insecure global_id in the test suite 2022-04-03T19:43:15.840 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-04-03T19:43:15.840 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-04-03T19:43:15.840 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.840 INFO:teuthology.orchestra.run.smithi077.stdout: # 1m isn't quite enough 2022-04-03T19:43:15.840 INFO:teuthology.orchestra.run.smithi077.stdout: mon_down_mkfs_grace = 2m 2022-04-03T19:43:15.841 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.841 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_filestore_osds = false 2022-04-03T19:43:15.841 INFO:teuthology.orchestra.run.smithi077.stdout: mon op complaint time = 120 2022-04-03T19:43:15.841 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:15.842 INFO:teuthology.orchestra.run.smithi077.stdout:[client] 2022-04-03T19:43:15.842 INFO:teuthology.orchestra.run.smithi077.stdout: rgw cache enabled = true 2022-04-03T19:43:15.842 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable ops log = true 2022-04-03T19:43:15.842 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable usage log = true 2022-04-03T19:43:15.843 INFO:teuthology.orchestra.run.smithi077.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-04-03T19:43:15.843 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-04-03T19:43:15.843 INFO:teuthology.orchestra.run.smithi077.stdout: client mount timeout = 600 2022-04-03T19:43:15.843 INFO:teuthology.orchestra.run.smithi077.stdout: debug client = 20 2022-04-03T19:43:15.843 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2022-04-03T19:43:15.844 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2022-04-03T19:43:15.844 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2022-04-03T19:43:15.844 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.a] 2022-04-03T19:43:15.844 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.b] 2022-04-03T19:43:15.845 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.c] 2022-04-03T19:43:15.845 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.a] 2022-04-03T19:43:15.845 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.c] 2022-04-03T19:43:15.845 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.e] 2022-04-03T19:43:15.845 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.b] 2022-04-03T19:43:15.846 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.d] 2022-04-03T19:43:15.846 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.f] 2022-04-03T19:43:15.846 INFO:teuthology.orchestra.run.smithi077.stdout:[mds] 2022-04-03T19:43:15.847 INFO:teuthology.orchestra.run.smithi077.stdout: debug mds = 20 2022-04-03T19:43:15.847 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2022-04-03T19:43:15.847 INFO:teuthology.orchestra.run.smithi077.stdout: mds bal fragment size max = 10000 2022-04-03T19:43:15.847 INFO:teuthology.orchestra.run.smithi077.stdout: mds bal merge size = 5 2022-04-03T19:43:15.847 INFO:teuthology.orchestra.run.smithi077.stdout: mds bal split bits = 3 2022-04-03T19:43:15.848 INFO:teuthology.orchestra.run.smithi077.stdout: mds bal split size = 100 2022-04-03T19:43:15.848 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug frag = True 2022-04-03T19:43:15.848 INFO:teuthology.orchestra.run.smithi077.stdout: mds debug scatterstat = True 2022-04-03T19:43:15.848 INFO:teuthology.orchestra.run.smithi077.stdout: mds oft prefetch dirfrags = False 2022-04-03T19:43:15.849 INFO:teuthology.orchestra.run.smithi077.stdout: mds op complaint time = 180 2022-04-03T19:43:15.849 INFO:teuthology.orchestra.run.smithi077.stdout: mds verify scatter = True 2022-04-03T19:43:15.849 INFO:teuthology.orchestra.run.smithi077.stdout: mds_sleep_rank_change = 5000000.0 2022-04-03T19:43:15.849 INFO:teuthology.orchestra.run.smithi077.stdout: mon client directed command retry = 5 2022-04-03T19:43:15.849 INFO:teuthology.orchestra.run.smithi077.stdout: ms inject delay max = 1 2022-04-03T19:43:15.850 INFO:teuthology.orchestra.run.smithi077.stdout: ms inject delay probability = 0.005 2022-04-03T19:43:15.850 INFO:teuthology.orchestra.run.smithi077.stdout: ms inject delay type = client mds osd 2022-04-03T19:43:15.850 INFO:teuthology.orchestra.run.smithi077.stdout: ms inject socket failures = 2500 2022-04-03T19:43:15.850 INFO:teuthology.orchestra.run.smithi077.stdout: osd op complaint time = 180 2022-04-03T19:43:15.851 INFO:teuthology.orchestra.run.smithi077.stdout: rados mon op timeout = 900 2022-04-03T19:43:15.851 INFO:teuthology.orchestra.run.smithi077.stdout: rados osd op timeout = 900 2022-04-03T19:43:15.852 INFO:tasks.ceph:Creating admin key on mon.a... 2022-04-03T19:43:15.852 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2022-04-03T19:43:15.965 INFO:tasks.ceph:Copying monmap to all nodes... 2022-04-03T19:43:15.966 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:15.966 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2022-04-03T19:43:16.010 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:16.011 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2022-04-03T19:43:16.094 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi077.front.sepia.ceph.com 2022-04-03T19:43:16.095 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:16.096 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.keyring 2022-04-03T19:43:16.096 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-04-03T19:43:16.231 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:16.232 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-04-03T19:43:16.314 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi186.front.sepia.ceph.com 2022-04-03T19:43:16.314 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:43:16.315 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.keyring 2022-04-03T19:43:16.315 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-04-03T19:43:16.354 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:43:16.355 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-04-03T19:43:16.410 INFO:tasks.ceph:Setting up mon nodes... 2022-04-03T19:43:16.411 INFO:tasks.ceph:Setting up mgr nodes... 2022-04-03T19:43:16.411 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2022-04-03T19:43:16.535 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2022-04-03T19:43:16.543 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2022-04-03T19:43:16.608 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2022-04-03T19:43:16.610 INFO:tasks.ceph:Setting up mds nodes... 2022-04-03T19:43:16.611 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2022-04-03T19:43:16.733 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2022-04-03T19:43:16.740 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2022-04-03T19:43:16.810 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2022-04-03T19:43:16.933 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2022-04-03T19:43:16.941 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2022-04-03T19:43:17.011 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2022-04-03T19:43:17.137 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2022-04-03T19:43:17.144 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2022-04-03T19:43:17.215 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2022-04-03T19:43:17.271 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2022-04-03T19:43:17.273 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2022-04-03T19:43:17.299 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2022-04-03T19:43:17.390 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2022-04-03T19:43:17.392 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2022-04-03T19:43:17.417 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2022-04-03T19:43:17.504 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2022-04-03T19:43:17.507 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2022-04-03T19:43:17.530 INFO:tasks.ceph_client:Setting up client nodes... 2022-04-03T19:43:17.531 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2022-04-03T19:43:17.626 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/ceph.client.0.keyring 2022-04-03T19:43:17.661 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-04-03T19:43:17.662 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2022-04-03T19:43:17.662 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2022-04-03T19:43:17.732 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-04-03T19:43:17.733 INFO:tasks.ceph:role: osd.0 2022-04-03T19:43:17.733 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com 2022-04-03T19:43:17.733 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-04-03T19:43:17.809 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-04-03T19:43:17.809 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2022-04-03T19:43:17.810 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-04-03T19:43:17.810 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 2022-04-03T19:43:17.810 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-04-03T19:43:17.810 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2022-04-03T19:43:17.811 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-04-03T19:43:17.811 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-04-03T19:43:17.811 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-04-03T19:43:17.811 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-04-03T19:43:18.253 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2022-04-03T19:43:18.259 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2022-04-03T19:43:18.260 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2022-04-03T19:43:18.453 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2022-04-03T19:43:18.533 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2022-04-03T19:43:18.604 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-04-03T19:43:18.604 INFO:tasks.ceph:role: osd.1 2022-04-03T19:43:18.604 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi077.front.sepia.ceph.com 2022-04-03T19:43:18.605 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-04-03T19:43:18.680 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-04-03T19:43:18.681 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2022-04-03T19:43:18.681 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-04-03T19:43:18.681 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 2022-04-03T19:43:18.682 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-04-03T19:43:18.682 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2022-04-03T19:43:18.682 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-04-03T19:43:18.682 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-04-03T19:43:18.682 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-04-03T19:43:18.683 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-04-03T19:43:19.099 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2022-04-03T19:43:19.105 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2022-04-03T19:43:19.105 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2022-04-03T19:43:19.253 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2022-04-03T19:43:19.331 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2022-04-03T19:43:19.402 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-04-03T19:43:19.402 INFO:tasks.ceph:role: osd.2 2022-04-03T19:43:19.402 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi077.front.sepia.ceph.com 2022-04-03T19:43:19.403 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-04-03T19:43:19.480 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-04-03T19:43:19.480 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2022-04-03T19:43:19.480 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-04-03T19:43:19.480 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 2022-04-03T19:43:19.481 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-04-03T19:43:19.481 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2022-04-03T19:43:19.481 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-04-03T19:43:19.482 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-04-03T19:43:19.482 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-04-03T19:43:19.482 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-04-03T19:43:19.924 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2022-04-03T19:43:19.931 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2022-04-03T19:43:19.931 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2022-04-03T19:43:20.111 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2022-04-03T19:43:20.190 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2022-04-03T19:43:20.260 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-04-03T19:43:20.261 INFO:tasks.ceph:role: osd.3 2022-04-03T19:43:20.261 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi077.front.sepia.ceph.com 2022-04-03T19:43:20.262 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2022-04-03T19:43:20.337 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2022-04-03T19:43:20.338 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2022-04-03T19:43:20.338 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-04-03T19:43:20.338 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 2022-04-03T19:43:20.339 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-04-03T19:43:20.339 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2022-04-03T19:43:20.339 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-04-03T19:43:20.339 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-04-03T19:43:20.340 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-04-03T19:43:20.340 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-04-03T19:43:20.776 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2022-04-03T19:43:20.782 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2022-04-03T19:43:20.782 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2022-04-03T19:43:20.898 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2022-04-03T19:43:20.976 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-04-03T19:43:21.104 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:21.102+0000 7f657c0a83c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2022-04-03T19:43:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:21.103+0000 7f657c0a83c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2022-04-03T19:43:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:21.103+0000 7f657c0a83c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2022-04-03T19:43:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:21.103+0000 7f657c0a83c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2022-04-03T19:43:21.106 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:21.103+0000 7f657c0a83c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2022-04-03T19:43:21.107 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:21.105+0000 7f657c0a83c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2022-04-03T19:43:24.521 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2022-04-03T19:43:24.594 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-04-03T19:43:24.699 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:24.697+0000 7fb2b55fb3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2022-04-03T19:43:24.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:24.698+0000 7fb2b55fb3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2022-04-03T19:43:24.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:24.698+0000 7fb2b55fb3c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2022-04-03T19:43:24.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:24.698+0000 7fb2b55fb3c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2022-04-03T19:43:24.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:24.698+0000 7fb2b55fb3c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2022-04-03T19:43:24.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:24.699+0000 7fb2b55fb3c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2022-04-03T19:43:28.121 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2022-04-03T19:43:28.194 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-04-03T19:43:28.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:28.298+0000 7f16e3fd33c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2022-04-03T19:43:28.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:28.298+0000 7f16e3fd33c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2022-04-03T19:43:28.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:28.298+0000 7f16e3fd33c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2022-04-03T19:43:28.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:28.298+0000 7f16e3fd33c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2022-04-03T19:43:28.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:28.298+0000 7f16e3fd33c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2022-04-03T19:43:28.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:28.300+0000 7f16e3fd33c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2022-04-03T19:43:31.715 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2022-04-03T19:43:31.788 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-04-03T19:43:31.892 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:31.892+0000 7f6f9627b3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2022-04-03T19:43:31.893 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:31.892+0000 7f6f9627b3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2022-04-03T19:43:31.894 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:31.893+0000 7f6f9627b3c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2022-04-03T19:43:31.894 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:31.893+0000 7f6f9627b3c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2022-04-03T19:43:31.894 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:31.893+0000 7f6f9627b3c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2022-04-03T19:43:31.895 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:31.894+0000 7f6f9627b3c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2022-04-03T19:43:35.312 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2022-04-03T19:43:35.383 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2022-04-03T19:43:35.413 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-04-03T19:43:35.414 INFO:tasks.ceph:role: osd.4 2022-04-03T19:43:35.414 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi186.front.sepia.ceph.com 2022-04-03T19:43:35.414 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-04-03T19:43:35.488 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-04-03T19:43:35.489 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2022-04-03T19:43:35.489 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-04-03T19:43:35.489 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 2022-04-03T19:43:35.490 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-04-03T19:43:35.490 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2022-04-03T19:43:35.490 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-04-03T19:43:35.490 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-04-03T19:43:35.491 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-04-03T19:43:35.491 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-04-03T19:43:35.849 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2022-04-03T19:43:35.851 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2022-04-03T19:43:35.851 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2022-04-03T19:43:35.987 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2022-04-03T19:43:36.020 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2022-04-03T19:43:36.088 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-04-03T19:43:36.088 INFO:tasks.ceph:role: osd.5 2022-04-03T19:43:36.089 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi186.front.sepia.ceph.com 2022-04-03T19:43:36.089 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-04-03T19:43:36.161 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-04-03T19:43:36.161 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2022-04-03T19:43:36.162 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-04-03T19:43:36.162 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 2022-04-03T19:43:36.162 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-04-03T19:43:36.162 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2022-04-03T19:43:36.163 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-04-03T19:43:36.163 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-04-03T19:43:36.163 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-04-03T19:43:36.163 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-04-03T19:43:36.515 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2022-04-03T19:43:36.517 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2022-04-03T19:43:36.518 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2022-04-03T19:43:36.610 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2022-04-03T19:43:36.642 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2022-04-03T19:43:36.707 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-04-03T19:43:36.708 INFO:tasks.ceph:role: osd.6 2022-04-03T19:43:36.708 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi186.front.sepia.ceph.com 2022-04-03T19:43:36.708 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-04-03T19:43:36.780 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-04-03T19:43:36.780 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2022-04-03T19:43:36.781 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-04-03T19:43:36.781 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 2022-04-03T19:43:36.781 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-04-03T19:43:36.781 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2022-04-03T19:43:36.782 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-04-03T19:43:36.782 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-04-03T19:43:36.782 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-04-03T19:43:36.782 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-04-03T19:43:37.155 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2022-04-03T19:43:37.157 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2022-04-03T19:43:37.157 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2022-04-03T19:43:37.278 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2022-04-03T19:43:37.310 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2022-04-03T19:43:37.376 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-04-03T19:43:37.376 INFO:tasks.ceph:role: osd.7 2022-04-03T19:43:37.376 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi186.front.sepia.ceph.com 2022-04-03T19:43:37.377 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2022-04-03T19:43:37.447 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2022-04-03T19:43:37.447 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2022-04-03T19:43:37.448 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-04-03T19:43:37.448 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 2022-04-03T19:43:37.448 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-04-03T19:43:37.448 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2022-04-03T19:43:37.449 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-04-03T19:43:37.449 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-04-03T19:43:37.449 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-04-03T19:43:37.449 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-04-03T19:43:37.816 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2022-04-03T19:43:37.818 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2022-04-03T19:43:37.818 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2022-04-03T19:43:37.908 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2022-04-03T19:43:37.939 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-04-03T19:43:38.021 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:38.020+0000 7fa1dda223c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2022-04-03T19:43:38.023 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:38.022+0000 7fa1dda223c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2022-04-03T19:43:38.024 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:38.022+0000 7fa1dda223c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2022-04-03T19:43:38.024 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:38.022+0000 7fa1dda223c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2022-04-03T19:43:38.024 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:38.022+0000 7fa1dda223c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2022-04-03T19:43:38.030 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:38.029+0000 7fa1dda223c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2022-04-03T19:43:40.883 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2022-04-03T19:43:40.951 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-04-03T19:43:41.038 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:41.036+0000 7f29bb9863c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2022-04-03T19:43:41.038 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:41.036+0000 7f29bb9863c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2022-04-03T19:43:41.038 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:41.036+0000 7f29bb9863c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2022-04-03T19:43:41.039 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:41.036+0000 7f29bb9863c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2022-04-03T19:43:41.039 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:41.036+0000 7f29bb9863c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2022-04-03T19:43:41.039 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:41.037+0000 7f29bb9863c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2022-04-03T19:43:43.890 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2022-04-03T19:43:43.958 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-04-03T19:43:44.041 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:44.040+0000 7fc6820783c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2022-04-03T19:43:44.042 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:44.040+0000 7fc6820783c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2022-04-03T19:43:44.042 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:44.040+0000 7fc6820783c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2022-04-03T19:43:44.042 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:44.040+0000 7fc6820783c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2022-04-03T19:43:44.042 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:44.040+0000 7fc6820783c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2022-04-03T19:43:44.043 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:44.040+0000 7fc6820783c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2022-04-03T19:43:44.348 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:43:44.354 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:43:46.898 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2022-04-03T19:43:46.924 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-04-03T19:43:47.007 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:47.006+0000 7f51327b73c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2022-04-03T19:43:47.007 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:47.006+0000 7f51327b73c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2022-04-03T19:43:47.008 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:47.006+0000 7f51327b73c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2022-04-03T19:43:47.008 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:47.006+0000 7f51327b73c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2022-04-03T19:43:47.008 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:47.006+0000 7f51327b73c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2022-04-03T19:43:47.008 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:47.006+0000 7f51327b73c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2022-04-03T19:43:49.861 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2022-04-03T19:43:49.927 INFO:tasks.ceph:Reading keys from all nodes... 2022-04-03T19:43:49.928 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:49.928 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2022-04-03T19:43:50.005 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:50.006 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2022-04-03T19:43:50.115 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:50.116 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2022-04-03T19:43:50.224 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:50.225 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2022-04-03T19:43:50.333 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:50.334 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2022-04-03T19:43:50.403 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:50.403 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2022-04-03T19:43:50.512 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:50.512 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2022-04-03T19:43:50.581 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:50.581 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2022-04-03T19:43:50.690 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:43:50.691 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2022-04-03T19:43:50.719 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:43:50.720 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2022-04-03T19:43:50.786 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:43:50.787 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2022-04-03T19:43:50.853 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:43:50.853 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2022-04-03T19:43:50.918 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:43:50.919 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2022-04-03T19:43:50.983 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:43:50.984 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2022-04-03T19:43:51.048 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:43:51.048 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2022-04-03T19:43:51.113 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:43:51.114 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2022-04-03T19:43:51.178 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:51.178 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2022-04-03T19:43:51.222 INFO:tasks.ceph:Adding keys to all mons... 2022-04-03T19:43:51.223 DEBUG:teuthology.orchestra.run.smithi077:> sudo tee -a /etc/ceph/ceph.keyring 2022-04-03T19:43:51.268 DEBUG:teuthology.orchestra.run.smithi186:> sudo tee -a /etc/ceph/ceph.keyring 2022-04-03T19:43:51.291 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr.y] 2022-04-03T19:43:51.291 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDU+EliN9buHxAABTijShsNGBY/V1sH86wgVw== 2022-04-03T19:43:51.291 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.a] 2022-04-03T19:43:51.291 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDU+EliqOO7KxAAKfoCcBjhS6iG2awjKn3UUg== 2022-04-03T19:43:51.292 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.c] 2022-04-03T19:43:51.292 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDU+Elilt2yNxAAigkrMdMt8oP8rFGsqAAFSA== 2022-04-03T19:43:51.292 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.e] 2022-04-03T19:43:51.292 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDV+EliSyo5CBAAMxEPT646+XNiiFbYAeBbRg== 2022-04-03T19:43:51.293 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.0] 2022-04-03T19:43:51.293 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDZ+Eli5Sw+BhAANasXdj4HmhUd78c363KVgA== 2022-04-03T19:43:51.293 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.1] 2022-04-03T19:43:51.293 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDc+Eli9dCvKRAAtmki/7LGbj0HPiNjW8YYiw== 2022-04-03T19:43:51.294 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.2] 2022-04-03T19:43:51.294 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDg+ElihqjhERAA7g8lrz5oF4CI9g0l8nVvsQ== 2022-04-03T19:43:51.294 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.3] 2022-04-03T19:43:51.294 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDj+ElikvA9NRAAT076SXVBv4OvvzHP4mAJlg== 2022-04-03T19:43:51.294 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr.x] 2022-04-03T19:43:51.295 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDU+EliMwJIJBAAJBnIfHJP8C298YGpdTdDKg== 2022-04-03T19:43:51.295 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.b] 2022-04-03T19:43:51.295 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDV+EliEY0wEBAAWPfDX0lbFNgTGinfjivmSA== 2022-04-03T19:43:51.295 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.d] 2022-04-03T19:43:51.295 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDV+EliqIJNFxAANUIKyrRmXnI2Pu+txQ8qgg== 2022-04-03T19:43:51.296 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.f] 2022-04-03T19:43:51.296 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDV+EliytMgHhAAmWOLh2FkOFOH1ht1fEsz8g== 2022-04-03T19:43:51.296 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.4] 2022-04-03T19:43:51.296 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDq+Eli8VdTARAAy5FMLI18PVZG5OihXIk7pA== 2022-04-03T19:43:51.296 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.5] 2022-04-03T19:43:51.297 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDt+EliDWtBAhAAtJl+pg0sym4yAKsjJbVF1A== 2022-04-03T19:43:51.297 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.6] 2022-04-03T19:43:51.297 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDw+Eliugh6AhAASw+5xnfIIh1rwD0/zWZDgA== 2022-04-03T19:43:51.297 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.7] 2022-04-03T19:43:51.298 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDz+EliWmdyABAAcsMQON2mdJhsaNtPbNd3Ow== 2022-04-03T19:43:51.298 INFO:teuthology.orchestra.run.smithi186.stdout:[client.0] 2022-04-03T19:43:51.298 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDV+ElixxpfJRAA++nlv2p4KQ1IRqHCyDSYFw== 2022-04-03T19:43:51.328 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr.y] 2022-04-03T19:43:51.328 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDU+EliN9buHxAABTijShsNGBY/V1sH86wgVw== 2022-04-03T19:43:51.329 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.a] 2022-04-03T19:43:51.329 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDU+EliqOO7KxAAKfoCcBjhS6iG2awjKn3UUg== 2022-04-03T19:43:51.329 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.c] 2022-04-03T19:43:51.329 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDU+Elilt2yNxAAigkrMdMt8oP8rFGsqAAFSA== 2022-04-03T19:43:51.329 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.e] 2022-04-03T19:43:51.330 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDV+EliSyo5CBAAMxEPT646+XNiiFbYAeBbRg== 2022-04-03T19:43:51.330 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.0] 2022-04-03T19:43:51.330 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDZ+Eli5Sw+BhAANasXdj4HmhUd78c363KVgA== 2022-04-03T19:43:51.330 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.1] 2022-04-03T19:43:51.331 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDc+Eli9dCvKRAAtmki/7LGbj0HPiNjW8YYiw== 2022-04-03T19:43:51.331 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.2] 2022-04-03T19:43:51.331 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDg+ElihqjhERAA7g8lrz5oF4CI9g0l8nVvsQ== 2022-04-03T19:43:51.331 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.3] 2022-04-03T19:43:51.332 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDj+ElikvA9NRAAT076SXVBv4OvvzHP4mAJlg== 2022-04-03T19:43:51.332 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr.x] 2022-04-03T19:43:51.332 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDU+EliMwJIJBAAJBnIfHJP8C298YGpdTdDKg== 2022-04-03T19:43:51.333 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.b] 2022-04-03T19:43:51.333 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDV+EliEY0wEBAAWPfDX0lbFNgTGinfjivmSA== 2022-04-03T19:43:51.333 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.d] 2022-04-03T19:43:51.333 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDV+EliqIJNFxAANUIKyrRmXnI2Pu+txQ8qgg== 2022-04-03T19:43:51.333 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.f] 2022-04-03T19:43:51.334 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDV+EliytMgHhAAmWOLh2FkOFOH1ht1fEsz8g== 2022-04-03T19:43:51.334 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.4] 2022-04-03T19:43:51.334 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDq+Eli8VdTARAAy5FMLI18PVZG5OihXIk7pA== 2022-04-03T19:43:51.334 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.5] 2022-04-03T19:43:51.335 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDt+EliDWtBAhAAtJl+pg0sym4yAKsjJbVF1A== 2022-04-03T19:43:51.335 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.6] 2022-04-03T19:43:51.335 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDw+Eliugh6AhAASw+5xnfIIh1rwD0/zWZDgA== 2022-04-03T19:43:51.335 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.7] 2022-04-03T19:43:51.336 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDz+EliWmdyABAAcsMQON2mdJhsaNtPbNd3Ow== 2022-04-03T19:43:51.336 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2022-04-03T19:43:51.336 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDV+ElixxpfJRAA++nlv2p4KQ1IRqHCyDSYFw== 2022-04-03T19:43:51.337 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-04-03T19:43:51.342 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-04-03T19:43:51.446 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-04-03T19:43:51.451 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-04-03T19:43:51.557 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-04-03T19:43:51.562 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-04-03T19:43:51.670 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-04-03T19:43:51.675 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-04-03T19:43:51.782 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-03T19:43:51.787 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-03T19:43:51.891 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-03T19:43:51.896 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-03T19:43:52.000 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-03T19:43:52.005 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-03T19:43:52.108 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-03T19:43:52.113 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-03T19:43:52.217 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-04-03T19:43:52.221 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-04-03T19:43:52.325 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-04-03T19:43:52.330 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-04-03T19:43:52.436 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-04-03T19:43:52.441 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-04-03T19:43:52.547 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-04-03T19:43:52.552 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-04-03T19:43:52.659 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-03T19:43:52.663 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-03T19:43:52.768 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-03T19:43:52.773 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-03T19:43:52.877 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-03T19:43:52.881 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-03T19:43:52.985 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-03T19:43:52.990 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2022-04-03T19:43:53.094 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-04-03T19:43:53.099 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-04-03T19:43:53.206 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-04-03T19:43:53.206 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2022-04-03T19:43:53.276 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-04-03T19:43:53.402 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2022-04-03T19:43:53.473 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2022-04-03T19:43:53.499 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-04-03T19:43:53.591 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2022-04-03T19:43:53.615 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2022-04-03T19:43:53.679 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-04-03T19:43:53.770 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2022-04-03T19:43:53.793 DEBUG:teuthology.orchestra.run.smithi077:> rm -- /home/ubuntu/cephtest/ceph.monmap 2022-04-03T19:43:53.798 DEBUG:teuthology.orchestra.run.smithi186:> rm -- /home/ubuntu/cephtest/ceph.monmap 2022-04-03T19:43:53.847 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2022-04-03T19:43:53.848 INFO:tasks.ceph.mon.a:Restarting daemon 2022-04-03T19:43:53.848 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2022-04-03T19:43:53.882 INFO:tasks.ceph.mon.a:Started 2022-04-03T19:43:53.882 INFO:tasks.ceph.mon.b:Restarting daemon 2022-04-03T19:43:53.882 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2022-04-03T19:43:53.890 INFO:tasks.ceph.mon.b:Started 2022-04-03T19:43:53.890 INFO:tasks.ceph.mon.c:Restarting daemon 2022-04-03T19:43:53.890 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2022-04-03T19:43:53.892 INFO:tasks.ceph.mon.c:Started 2022-04-03T19:43:53.893 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2022-04-03T19:43:53.893 INFO:tasks.ceph.mgr.y:Restarting daemon 2022-04-03T19:43:53.894 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2022-04-03T19:43:53.898 INFO:tasks.ceph.mgr.y:Started 2022-04-03T19:43:53.898 INFO:tasks.ceph.mgr.x:Restarting daemon 2022-04-03T19:43:53.899 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2022-04-03T19:43:53.901 INFO:tasks.ceph.mgr.x:Started 2022-04-03T19:43:53.901 INFO:tasks.ceph:Setting crush tunables to default 2022-04-03T19:43:53.902 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd crush tunables default 2022-04-03T19:43:54.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.097+0000 7f0f0f1d0700 1 Processor -- start 2022-04-03T19:43:54.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.097+0000 7f0f0f1d0700 1 -- start start 2022-04-03T19:43:54.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.098+0000 7f0f0f1d0700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0f080c5680 0x7f0f080c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:54.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.098+0000 7f0f0f1d0700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f080c7e80 con 0x7f0f080c52b0 2022-04-03T19:43:54.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.098+0000 7f0f0f1d0700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f080c6ea0 con 0x7f0f08158e10 2022-04-03T19:43:54.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.098+0000 7f0f0f1d0700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f080c4260 con 0x7f0f080c5680 2022-04-03T19:43:54.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.099+0000 7f0f0e9cf700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f0f08158e10 0x7f0f0814ed50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44066/0 (socket says 172.21.15.77:44066) 2022-04-03T19:43:54.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.099+0000 7f0f0e9cf700 1 -- 172.21.15.77:0/2719786216 learned_addr learned my addr 172.21.15.77:0/2719786216 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:43:54.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.099+0000 7f0f0d9cd700 1 --2- 172.21.15.77:0/2719786216 >> v2:172.21.15.186:3300/0 conn(0x7f0f080c5680 0x7f0f080c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:54.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.099+0000 7f0f0d9cd700 1 -- 172.21.15.77:0/2719786216 >> v2:172.21.15.186:3300/0 conn(0x7f0f080c5680 msgr2=0x7f0f080c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2022-04-03T19:43:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.099+0000 7f0f0d9cd700 1 -- 172.21.15.77:0/2719786216 >> v2:172.21.15.186:3300/0 conn(0x7f0f080c5680 msgr2=0x7f0f080c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-04-03T19:43:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.099+0000 7f0f0d9cd700 1 --2- 172.21.15.77:0/2719786216 >> v2:172.21.15.186:3300/0 conn(0x7f0f080c5680 0x7f0f080c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-04-03T19:43:54.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.099+0000 7f0f0d9cd700 1 --2- 172.21.15.77:0/2719786216 >> v2:172.21.15.186:3300/0 conn(0x7f0f080c5680 0x7f0f080c5f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-04-03T19:43:54.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.123+0000 7f0f0d1cc700 1 -- 172.21.15.77:0/2719786216 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715520338 0 0) 0x7f0f080c7e80 con 0x7f0f080c52b0 2022-04-03T19:43:54.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.123+0000 7f0f0d1cc700 1 -- 172.21.15.77:0/2719786216 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ef00033f0 con 0x7f0f080c52b0 2022-04-03T19:43:54.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.123+0000 7f0f0d1cc700 1 -- 172.21.15.77:0/2719786216 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 393185296 0 0) 0x7f0ef00033f0 con 0x7f0f080c52b0 2022-04-03T19:43:54.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.123+0000 7f0f0d1cc700 1 -- 172.21.15.77:0/2719786216 >> v2:172.21.15.186:3300/0 conn(0x7f0f080c5680 msgr2=0x7f0f080c5f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-04-03T19:43:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.123+0000 7f0f0d1cc700 1 --2- 172.21.15.77:0/2719786216 >> v2:172.21.15.186:3300/0 conn(0x7f0f080c5680 0x7f0f080c5f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.123+0000 7f0f0d1cc700 1 -- 172.21.15.77:0/2719786216 >> v1:172.21.15.186:6789/0 conn(0x7f0f08158e10 legacy=0x7f0f0814ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.123+0000 7f0f0d1cc700 1 -- 172.21.15.77:0/2719786216 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f080c6600 con 0x7f0f080c52b0 2022-04-03T19:43:54.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.123+0000 7f0f0d1cc700 1 -- 172.21.15.77:0/2719786216 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4219302955 0 0) 0x7f0f04001470 con 0x7f0f080c52b0 2022-04-03T19:43:54.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.123+0000 7f0f0d1cc700 1 -- 172.21.15.77:0/2719786216 >> v1:172.21.15.77:6789/0 conn(0x7f0f080c52b0 legacy=0x7f0f080c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:54.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.123+0000 7f0f0d1cc700 1 --2- 172.21.15.77:0/2719786216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ef00038e0 0x7f0ef0003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:54.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.123+0000 7f0f0d1cc700 1 --2- 172.21.15.77:0/2719786216 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ef0004690 0x7f0ef0004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:54.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.123+0000 7f0f0d1cc700 1 --2- 172.21.15.77:0/2719786216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ef0005160 0x7f0ef000b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:54.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.123+0000 7f0f0d1cc700 1 -- 172.21.15.77:0/2719786216 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f080c4260 con 0x7f0ef00038e0 2022-04-03T19:43:54.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.123+0000 7f0f0d1cc700 1 -- 172.21.15.77:0/2719786216 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef000c010 con 0x7f0ef0005160 2022-04-03T19:43:54.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.123+0000 7f0f0d1cc700 1 -- 172.21.15.77:0/2719786216 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef000c1d0 con 0x7f0ef0004690 2022-04-03T19:43:54.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.124+0000 7f0f0e9cf700 1 --2- 172.21.15.77:0/2719786216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ef0005160 0x7f0ef000b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:54.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.124+0000 7f0f0d9cd700 1 --2- 172.21.15.77:0/2719786216 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ef0004690 0x7f0ef0004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:54.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.124+0000 7f0f0e1ce700 1 --2- 172.21.15.77:0/2719786216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ef00038e0 0x7f0ef0003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:54.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.124+0000 7f0f0d9cd700 1 -- 172.21.15.77:0/2719786216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ef0005160 msgr2=0x7f0ef000b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:43:54.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.124+0000 7f0f0d9cd700 1 --2- 172.21.15.77:0/2719786216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ef0005160 0x7f0ef000b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:54.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.124+0000 7f0f0d9cd700 1 -- 172.21.15.77:0/2719786216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ef00038e0 msgr2=0x7f0ef0003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:43:54.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.124+0000 7f0f0d9cd700 1 --2- 172.21.15.77:0/2719786216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ef00038e0 0x7f0ef0003cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:54.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.124+0000 7f0f0d9cd700 1 -- 172.21.15.77:0/2719786216 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef00033f0 con 0x7f0ef0004690 2022-04-03T19:43:54.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.124+0000 7f0f0e9cf700 1 -- 172.21.15.77:0/2719786216 reap_dead start 2022-04-03T19:43:54.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.124+0000 7f0f0d9cd700 1 --2- 172.21.15.77:0/2719786216 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ef0004690 0x7f0ef0004b00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8006e50 tx=0x7f0ef8008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fba207fc26f058bd server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:54.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.125+0000 7f0f0d1cc700 1 -- 172.21.15.77:0/2719786216 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8017070 con 0x7f0ef0004690 2022-04-03T19:43:54.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.125+0000 7f0f0d1cc700 1 -- 172.21.15.77:0/2719786216 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef8004020 con 0x7f0ef0004690 2022-04-03T19:43:54.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.125+0000 7f0f0d1cc700 1 -- 172.21.15.77:0/2719786216 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef80043c0 con 0x7f0ef0004690 2022-04-03T19:43:54.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.126+0000 7f0f0f1d0700 1 -- 172.21.15.77:0/2719786216 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ef0004690 msgr2=0x7f0ef0004b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:54.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.126+0000 7f0f0f1d0700 1 --2- 172.21.15.77:0/2719786216 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ef0004690 0x7f0ef0004b00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8006e50 tx=0x7f0ef8008040 comp rx=0 tx=0).stop 2022-04-03T19:43:54.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.126+0000 7f0f0f1d0700 1 -- 172.21.15.77:0/2719786216 shutdown_connections 2022-04-03T19:43:54.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.126+0000 7f0f0f1d0700 1 --2- 172.21.15.77:0/2719786216 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ef0004690 0x7f0ef0004b00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:54.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.126+0000 7f0f0f1d0700 1 -- 172.21.15.77:0/2719786216 >> 172.21.15.77:0/2719786216 conn(0x7f0f080626d0 msgr2=0x7f0f08062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:54.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.126+0000 7f0f0f1d0700 1 -- 172.21.15.77:0/2719786216 shutdown_connections 2022-04-03T19:43:54.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.126+0000 7f0f0f1d0700 1 -- 172.21.15.77:0/2719786216 wait complete. 2022-04-03T19:43:54.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.128+0000 7f0f0f1d0700 1 Processor -- start 2022-04-03T19:43:54.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.128+0000 7f0f0f1d0700 1 -- start start 2022-04-03T19:43:54.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.129+0000 7f0f0f1d0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ef0004690 0x7f0f08165c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:54.137 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.129+0000 7f0f0f1d0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f080bf100 0x7f0f080c2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:54.137 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.129+0000 7f0f0f1d0700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f080bf570 0x7f0f081549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:54.137 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.129+0000 7f0f0f1d0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f080bfad0 con 0x7f0ef0004690 2022-04-03T19:43:54.137 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.129+0000 7f0f0f1d0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f080bfd40 con 0x7f0f080bf100 2022-04-03T19:43:54.137 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.129+0000 7f0f0f1d0700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f080c0020 con 0x7f0f080bf570 2022-04-03T19:43:54.138 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.129+0000 7f0f0d9cd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f080bf100 0x7f0f080c2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:54.138 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.129+0000 7f0f0d9cd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f080bf100 0x7f0f080c2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37572/0 (socket says 172.21.15.77:37572) 2022-04-03T19:43:54.138 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.129+0000 7f0f0d9cd700 1 -- 172.21.15.77:0/1617168970 learned_addr learned my addr 172.21.15.77:0/1617168970 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:43:54.138 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.129+0000 7f0f0e9cf700 1 --2- 172.21.15.77:0/1617168970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f080bf570 0x7f0f081549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:54.139 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.129+0000 7f0f0e1ce700 1 --2- 172.21.15.77:0/1617168970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ef0004690 0x7f0f08165c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:54.139 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.130+0000 7f0f0d9cd700 1 -- 172.21.15.77:0/1617168970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f080bf570 msgr2=0x7f0f081549d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:54.139 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.130+0000 7f0f0d9cd700 1 --2- 172.21.15.77:0/1617168970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f080bf570 0x7f0f081549d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:54.139 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.130+0000 7f0f0d9cd700 1 -- 172.21.15.77:0/1617168970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ef0004690 msgr2=0x7f0f08165c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:54.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.130+0000 7f0f0d9cd700 1 --2- 172.21.15.77:0/1617168970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ef0004690 0x7f0f08165c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:54.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.130+0000 7f0f0d9cd700 1 -- 172.21.15.77:0/1617168970 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f080c0300 con 0x7f0f080bf100 2022-04-03T19:43:54.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.130+0000 7f0f0e1ce700 1 --2- 172.21.15.77:0/1617168970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ef0004690 0x7f0f08165c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:43:54.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.130+0000 7f0f0d9cd700 1 --2- 172.21.15.77:0/1617168970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f080bf100 0x7f0f080c2000 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800bf90 tx=0x7f0ef8019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:54.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.130+0000 7f0ef67fc700 1 -- 172.21.15.77:0/1617168970 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8013480 con 0x7f0f080bf100 2022-04-03T19:43:54.141 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.130+0000 7f0ef67fc700 1 -- 172.21.15.77:0/1617168970 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef80135e0 con 0x7f0f080bf100 2022-04-03T19:43:54.141 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.130+0000 7f0ef67fc700 1 -- 172.21.15.77:0/1617168970 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef80139a0 con 0x7f0f080bf100 2022-04-03T19:43:54.141 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.131+0000 7f0f0f1d0700 1 -- 172.21.15.77:0/1617168970 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f080c05e0 con 0x7f0f080bf100 2022-04-03T19:43:54.141 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.131+0000 7f0f0f1d0700 1 -- 172.21.15.77:0/1617168970 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f080c2540 con 0x7f0f080bf100 2022-04-03T19:43:54.142 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.131+0000 7f0ef67fc700 1 -- 172.21.15.77:0/1617168970 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f0ef801f440 con 0x7f0f080bf100 2022-04-03T19:43:54.142 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.132+0000 7f0f0f1d0700 1 -- 172.21.15.77:0/1617168970 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f080bfd40 con 0x7f0f080bf100 2022-04-03T19:43:54.142 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.136+0000 7f0ef67fc700 1 -- 172.21.15.77:0/1617168970 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0ef8013d90 con 0x7f0f080bf100 2022-04-03T19:43:54.143 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.138+0000 7f0ef67fc700 1 -- 172.21.15.77:0/1617168970 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70738 (secure 0 0 0) 0x7f0ef802c020 con 0x7f0f080bf100 2022-04-03T19:43:54.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.217+0000 7f0f0f1d0700 1 -- 172.21.15.77:0/1617168970 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f0f080c07d0 con 0x7f0f080bf100 2022-04-03T19:43:54.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.225+0000 7f0ef67fc700 1 -- 172.21.15.77:0/1617168970 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f0ef801f900 con 0x7f0f080bf100 2022-04-03T19:43:54.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.226+0000 7f0f0f1d0700 1 -- 172.21.15.77:0/1617168970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f080bf100 msgr2=0x7f0f080c2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:54.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.226+0000 7f0f0f1d0700 1 --2- 172.21.15.77:0/1617168970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f080bf100 0x7f0f080c2000 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800bf90 tx=0x7f0ef8019000 comp rx=0 tx=0).stop 2022-04-03T19:43:54.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.226+0000 7f0f0f1d0700 1 -- 172.21.15.77:0/1617168970 shutdown_connections 2022-04-03T19:43:54.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.226+0000 7f0f0f1d0700 1 --2- 172.21.15.77:0/1617168970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f080bf570 0x7f0f081549d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:54.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.226+0000 7f0f0f1d0700 1 --2- 172.21.15.77:0/1617168970 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ef0004690 0x7f0f08165c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:54.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.226+0000 7f0f0f1d0700 1 --2- 172.21.15.77:0/1617168970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f080bf100 0x7f0f080c2000 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:54.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.226+0000 7f0f0f1d0700 1 -- 172.21.15.77:0/1617168970 >> 172.21.15.77:0/1617168970 conn(0x7f0f080626d0 msgr2=0x7f0f080c85a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:54.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.227+0000 7f0f0f1d0700 1 -- 172.21.15.77:0/1617168970 shutdown_connections 2022-04-03T19:43:54.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:54.227+0000 7f0f0f1d0700 1 -- 172.21.15.77:0/1617168970 wait complete. 2022-04-03T19:43:54.230 INFO:teuthology.orchestra.run.smithi077.stderr:adjusted tunables profile to default 2022-04-03T19:43:54.251 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2022-04-03T19:43:54.252 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:54.252 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2022-04-03T19:43:54.322 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:54.323 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2022-04-03T19:43:54.432 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:54.433 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2022-04-03T19:43:54.503 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:43:54.504 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2022-04-03T19:43:54.614 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:43:54.614 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2022-04-03T19:43:54.638 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:43:54.639 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2022-04-03T19:43:54.674 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:54.675+0000 7f207b38e000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-04-03T19:43:54.698 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:43:54.698 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2022-04-03T19:43:54.762 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2022-04-03T19:43:54.763 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2022-04-03T19:43:54.775 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:43:54.775+0000 7f493b032000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-04-03T19:43:54.817 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:54.817+0000 7f207b38e000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-04-03T19:43:54.826 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new f35f5195-3729-4d78-85d1-0539f259283f 0 2022-04-03T19:43:54.961 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:43:54.960+0000 7f493b032000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-04-03T19:43:54.982 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.982+0000 7fe34dae3700 1 Processor -- start 2022-04-03T19:43:54.982 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.982+0000 7fe34dae3700 1 -- start start 2022-04-03T19:43:54.985 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.982+0000 7fe34dae3700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe34806bbe0 0x7fe3480c2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:54.985 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.982+0000 7fe34dae3700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe34806af90 0x7fe3480c2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:54.985 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.982+0000 7fe34dae3700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe34806a3f0 con 0x7fe3480c5700 2022-04-03T19:43:54.986 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.982+0000 7fe34dae3700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe348155150 con 0x7fe34806bbe0 2022-04-03T19:43:54.986 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.982+0000 7fe34dae3700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe348155340 con 0x7fe34806af90 2022-04-03T19:43:54.986 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.983+0000 7fe347fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe34806af90 0x7fe3480c2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:54.986 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.983+0000 7fe347fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe34806af90 0x7fe3480c2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:49592/0 (socket says 172.21.15.186:49592) 2022-04-03T19:43:54.986 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.983+0000 7fe3477fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe34806bbe0 0x7fe3480c2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:54.987 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.983+0000 7fe347fff700 1 -- 172.21.15.186:0/3723609800 learned_addr learned my addr 172.21.15.186:0/3723609800 (peer_addr_for_me v2:172.21.15.186:0/0) 2022-04-03T19:43:54.987 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.983+0000 7fe347fff700 1 -- 172.21.15.186:0/3723609800 >> v2:172.21.15.186:3300/0 conn(0x7fe34806bbe0 msgr2=0x7fe3480c2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:43:54.987 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.983+0000 7fe347fff700 1 --2- 172.21.15.186:0/3723609800 >> v2:172.21.15.186:3300/0 conn(0x7fe34806bbe0 0x7fe3480c2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:54.987 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.983+0000 7fe347fff700 1 -- 172.21.15.186:0/3723609800 >> v1:172.21.15.77:6789/0 conn(0x7fe3480c5700 legacy=0x7fe34806a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:43:54.988 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.983+0000 7fe347fff700 1 -- 172.21.15.186:0/3723609800 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe348155620 con 0x7fe34806af90 2022-04-03T19:43:54.988 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.983+0000 7fe3477fe700 1 --2- 172.21.15.186:0/3723609800 >> v2:172.21.15.186:3300/0 conn(0x7fe34806bbe0 0x7fe3480c2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:43:54.988 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.983+0000 7fe347fff700 1 --2- 172.21.15.186:0/3723609800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe34806af90 0x7fe3480c2590 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe340009f20 tx=0x7fe340009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f49a600db3e39b0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:54.988 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.983+0000 7fe346ffd700 1 -- 172.21.15.186:0/3723609800 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34000e070 con 0x7fe34806af90 2022-04-03T19:43:54.988 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.983+0000 7fe346ffd700 1 -- 172.21.15.186:0/3723609800 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe34000b990 con 0x7fe34806af90 2022-04-03T19:43:54.989 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.983+0000 7fe346ffd700 1 -- 172.21.15.186:0/3723609800 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3400092e0 con 0x7fe34806af90 2022-04-03T19:43:54.989 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.984+0000 7fe34dae3700 1 -- 172.21.15.186:0/3723609800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe34806af90 msgr2=0x7fe3480c2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:54.989 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.984+0000 7fe34dae3700 1 --2- 172.21.15.186:0/3723609800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe34806af90 0x7fe3480c2590 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe340009f20 tx=0x7fe340009840 comp rx=0 tx=0).stop 2022-04-03T19:43:54.989 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.984+0000 7fe34dae3700 1 -- 172.21.15.186:0/3723609800 shutdown_connections 2022-04-03T19:43:54.989 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.984+0000 7fe34dae3700 1 --2- 172.21.15.186:0/3723609800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe34806af90 0x7fe3480c2590 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:54.990 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.984+0000 7fe34dae3700 1 --2- 172.21.15.186:0/3723609800 >> v2:172.21.15.186:3300/0 conn(0x7fe34806bbe0 0x7fe3480c2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:54.990 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.984+0000 7fe34dae3700 1 -- 172.21.15.186:0/3723609800 >> 172.21.15.186:0/3723609800 conn(0x7fe3480c9020 msgr2=0x7fe3480c4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:54.990 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.984+0000 7fe34dae3700 1 -- 172.21.15.186:0/3723609800 shutdown_connections 2022-04-03T19:43:54.990 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.984+0000 7fe34dae3700 1 -- 172.21.15.186:0/3723609800 wait complete. 2022-04-03T19:43:54.991 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.985+0000 7fe34dae3700 1 Processor -- start 2022-04-03T19:43:54.991 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.985+0000 7fe34dae3700 1 -- start start 2022-04-03T19:43:54.993 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe34dae3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe34806af90 0x7fe3480c1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:54.993 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe34dae3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe34806bbe0 0x7fe34814fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:54.994 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe34dae3700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3480c5700 0x7fe348150050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:54.994 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe34dae3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3481658b0 con 0x7fe34806bbe0 2022-04-03T19:43:54.994 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe34dae3700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3481655d0 con 0x7fe34806af90 2022-04-03T19:43:54.994 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe34dae3700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe348156440 con 0x7fe3480c5700 2022-04-03T19:43:54.995 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe3477fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe34806af90 0x7fe3480c1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:54.995 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe347fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3480c5700 0x7fe348150050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:54.995 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe3477fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe34806af90 0x7fe3480c1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40080/0 (socket says 172.21.15.186:40080) 2022-04-03T19:43:54.995 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe3477fe700 1 -- 172.21.15.186:0/3463334203 learned_addr learned my addr 172.21.15.186:0/3463334203 (peer_addr_for_me v2:172.21.15.186:0/0) 2022-04-03T19:43:54.996 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe33effd700 1 --2- 172.21.15.186:0/3463334203 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe34806bbe0 0x7fe34814fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:54.996 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe3477fe700 1 -- 172.21.15.186:0/3463334203 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3480c5700 msgr2=0x7fe348150050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:54.996 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe3477fe700 1 --2- 172.21.15.186:0/3463334203 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3480c5700 0x7fe348150050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:54.996 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe3477fe700 1 -- 172.21.15.186:0/3463334203 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe34806bbe0 msgr2=0x7fe34814fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:54.996 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe3477fe700 1 --2- 172.21.15.186:0/3463334203 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe34806bbe0 0x7fe34814fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:54.997 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe3477fe700 1 -- 172.21.15.186:0/3463334203 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3380092a0 con 0x7fe34806af90 2022-04-03T19:43:54.997 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe3477fe700 1 --2- 172.21.15.186:0/3463334203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe34806af90 0x7fe3480c1fe0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe33800ea20 tx=0x7fe338009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:54.997 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe33ffff700 1 -- 172.21.15.186:0/3463334203 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33800c850 con 0x7fe34806af90 2022-04-03T19:43:54.997 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe33ffff700 1 -- 172.21.15.186:0/3463334203 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33800c9b0 con 0x7fe34806af90 2022-04-03T19:43:54.998 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe34dae3700 1 -- 172.21.15.186:0/3463334203 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe340009ae0 con 0x7fe34806af90 2022-04-03T19:43:54.998 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe33ffff700 1 -- 172.21.15.186:0/3463334203 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33800cd30 con 0x7fe34806af90 2022-04-03T19:43:54.998 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.986+0000 7fe34dae3700 1 -- 172.21.15.186:0/3463334203 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe348156a00 con 0x7fe34806af90 2022-04-03T19:43:54.998 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.987+0000 7fe33ffff700 1 -- 172.21.15.186:0/3463334203 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7fe338019030 con 0x7fe34806af90 2022-04-03T19:43:54.999 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.987+0000 7fe33ffff700 1 -- 172.21.15.186:0/3463334203 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe338006440 con 0x7fe34806af90 2022-04-03T19:43:54.999 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.987+0000 7fe34dae3700 1 -- 172.21.15.186:0/3463334203 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe34804b890 con 0x7fe34806af90 2022-04-03T19:43:54.999 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:54.989+0000 7fe33ffff700 1 -- 172.21.15.186:0/3463334203 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70738 (secure 0 0 0) 0x7fe338021060 con 0x7fe34806af90 2022-04-03T19:43:54.999 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:54.990+0000 7f207b38e000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-04-03T19:43:55.070 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.070+0000 7fe34dae3700 1 -- 172.21.15.186:0/3463334203 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f35f5195-3729-4d78-85d1-0539f259283f", "id": 0} v 0) v1 -- 0x7fe348070b00 con 0x7fe34806af90 2022-04-03T19:43:55.076 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.076+0000 7fe33ffff700 1 -- 172.21.15.186:0/3463334203 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f35f5195-3729-4d78-85d1-0539f259283f", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe338017070 con 0x7fe34806af90 2022-04-03T19:43:55.077 INFO:teuthology.orchestra.run.smithi186.stdout:0 2022-04-03T19:43:55.078 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.077+0000 7fe34dae3700 1 -- 172.21.15.186:0/3463334203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe34806af90 msgr2=0x7fe3480c1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:55.078 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.077+0000 7fe34dae3700 1 --2- 172.21.15.186:0/3463334203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe34806af90 0x7fe3480c1fe0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe33800ea20 tx=0x7fe338009060 comp rx=0 tx=0).stop 2022-04-03T19:43:55.078 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.077+0000 7fe34dae3700 1 -- 172.21.15.186:0/3463334203 shutdown_connections 2022-04-03T19:43:55.079 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.077+0000 7fe34dae3700 1 --2- 172.21.15.186:0/3463334203 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3480c5700 0x7fe348150050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.079 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.077+0000 7fe34dae3700 1 --2- 172.21.15.186:0/3463334203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe34806af90 0x7fe3480c1fe0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.079 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.077+0000 7fe34dae3700 1 --2- 172.21.15.186:0/3463334203 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe34806bbe0 0x7fe34814fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.079 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.077+0000 7fe34dae3700 1 -- 172.21.15.186:0/3463334203 >> 172.21.15.186:0/3463334203 conn(0x7fe3480c9020 msgr2=0x7fe34806bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:55.080 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.077+0000 7fe34dae3700 1 -- 172.21.15.186:0/3463334203 shutdown_connections 2022-04-03T19:43:55.080 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.077+0000 7fe34dae3700 1 -- 172.21.15.186:0/3463334203 wait complete. 2022-04-03T19:43:55.089 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new ab1c89dc-57ce-4cc4-b8c7-9b0c19939601 1 2022-04-03T19:43:55.125 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:55.125+0000 7f207b38e000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-04-03T19:43:55.185 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:43:55.185+0000 7f493b032000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-04-03T19:43:55.192 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:55.193+0000 7f207b38e000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-04-03T19:43:55.206 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.206+0000 7f202c105700 1 Processor -- start 2022-04-03T19:43:55.206 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.206+0000 7f202c105700 1 -- start start 2022-04-03T19:43:55.209 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.207+0000 7f202c105700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20240c5260 0x7f20240c2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.210 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.207+0000 7f202c105700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f20240c89e0 0x7f202406c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.210 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.207+0000 7f202c105700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20240c7d00 con 0x7f20240c71e0 2022-04-03T19:43:55.210 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.207+0000 7f202c105700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20240c7740 con 0x7f20240c89e0 2022-04-03T19:43:55.210 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.207+0000 7f202c105700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20240c7a20 con 0x7f20240c5260 2022-04-03T19:43:55.211 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.207+0000 7f2029ea1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20240c5260 0x7f20240c2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.211 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.207+0000 7f202a6a2700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f20240c89e0 0x7f202406c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.211 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.207+0000 7f202a6a2700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f20240c89e0 0x7f202406c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40090/0 (socket says 172.21.15.186:40090) 2022-04-03T19:43:55.211 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.207+0000 7f202a6a2700 1 -- 172.21.15.186:0/2970119151 learned_addr learned my addr 172.21.15.186:0/2970119151 (peer_addr_for_me v2:172.21.15.186:0/0) 2022-04-03T19:43:55.212 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.207+0000 7f2029ea1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20240c5260 0x7f20240c2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:49600/0 (socket says 172.21.15.186:49600) 2022-04-03T19:43:55.212 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.208+0000 7f2029ea1700 1 -- 172.21.15.186:0/2970119151 >> v2:172.21.15.186:3300/0 conn(0x7f20240c89e0 msgr2=0x7f202406c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:43:55.212 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.208+0000 7f2029ea1700 1 --2- 172.21.15.186:0/2970119151 >> v2:172.21.15.186:3300/0 conn(0x7f20240c89e0 0x7f202406c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.212 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.208+0000 7f2029ea1700 1 -- 172.21.15.186:0/2970119151 >> v1:172.21.15.77:6789/0 conn(0x7f20240c71e0 legacy=0x7f20240c3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:43:55.212 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.208+0000 7f2029ea1700 1 -- 172.21.15.186:0/2970119151 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f202406bb50 con 0x7f20240c5260 2022-04-03T19:43:55.213 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.208+0000 7f202a6a2700 1 --2- 172.21.15.186:0/2970119151 >> v2:172.21.15.186:3300/0 conn(0x7f20240c89e0 0x7f202406c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:43:55.213 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.208+0000 7f2029ea1700 1 --2- 172.21.15.186:0/2970119151 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20240c5260 0x7f20240c2b50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2020009760 tx=0x7f2020009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=876097d98fa287c7 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:55.213 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.208+0000 7f2028e9f700 1 -- 172.21.15.186:0/2970119151 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2020004030 con 0x7f20240c5260 2022-04-03T19:43:55.213 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.208+0000 7f2028e9f700 1 -- 172.21.15.186:0/2970119151 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2020004190 con 0x7f20240c5260 2022-04-03T19:43:55.213 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.208+0000 7f2028e9f700 1 -- 172.21.15.186:0/2970119151 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2020011630 con 0x7f20240c5260 2022-04-03T19:43:55.214 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.208+0000 7f202c105700 1 -- 172.21.15.186:0/2970119151 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20240c5260 msgr2=0x7f20240c2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:55.214 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.208+0000 7f202c105700 1 --2- 172.21.15.186:0/2970119151 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20240c5260 0x7f20240c2b50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2020009760 tx=0x7f2020009060 comp rx=0 tx=0).stop 2022-04-03T19:43:55.214 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.209+0000 7f202c105700 1 -- 172.21.15.186:0/2970119151 shutdown_connections 2022-04-03T19:43:55.214 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.209+0000 7f202c105700 1 --2- 172.21.15.186:0/2970119151 >> v2:172.21.15.186:3300/0 conn(0x7f20240c89e0 0x7f202406c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.215 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.209+0000 7f202c105700 1 --2- 172.21.15.186:0/2970119151 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20240c5260 0x7f20240c2b50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.215 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.209+0000 7f202c105700 1 -- 172.21.15.186:0/2970119151 >> 172.21.15.186:0/2970119151 conn(0x7f20240cc2c0 msgr2=0x7f20240c82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:55.215 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.209+0000 7f202c105700 1 -- 172.21.15.186:0/2970119151 shutdown_connections 2022-04-03T19:43:55.215 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.209+0000 7f202c105700 1 -- 172.21.15.186:0/2970119151 wait complete. 2022-04-03T19:43:55.215 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.210+0000 7f202c105700 1 Processor -- start 2022-04-03T19:43:55.216 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.210+0000 7f202c105700 1 -- start start 2022-04-03T19:43:55.218 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.210+0000 7f202c105700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20240c5260 0x7f2024168460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.219 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.210+0000 7f202c105700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20240c71e0 0x7f20240c10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.219 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.210+0000 7f202c105700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20240c89e0 0x7f20240c15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.219 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.210+0000 7f202c105700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202414fc30 con 0x7f20240c71e0 2022-04-03T19:43:55.219 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.210+0000 7f202c105700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20240c0c10 con 0x7f20240c89e0 2022-04-03T19:43:55.220 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.210+0000 7f202c105700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2024168ec0 con 0x7f20240c5260 2022-04-03T19:43:55.220 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.211+0000 7f2029ea1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20240c5260 0x7f2024168460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.220 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.211+0000 7f202a6a2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20240c89e0 0x7f20240c15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.220 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.211+0000 7f2029ea1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20240c5260 0x7f2024168460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:49606/0 (socket says 172.21.15.186:49606) 2022-04-03T19:43:55.220 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.211+0000 7f2029ea1700 1 -- 172.21.15.186:0/3206064933 learned_addr learned my addr 172.21.15.186:0/3206064933 (peer_addr_for_me v2:172.21.15.186:0/0) 2022-04-03T19:43:55.221 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.211+0000 7f20296a0700 1 --2- 172.21.15.186:0/3206064933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20240c71e0 0x7f20240c10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.221 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.211+0000 7f2029ea1700 1 -- 172.21.15.186:0/3206064933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20240c89e0 msgr2=0x7f20240c15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:55.221 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.211+0000 7f2029ea1700 1 --2- 172.21.15.186:0/3206064933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20240c89e0 0x7f20240c15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.221 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.211+0000 7f2029ea1700 1 -- 172.21.15.186:0/3206064933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20240c71e0 msgr2=0x7f20240c10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:55.222 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.211+0000 7f2029ea1700 1 --2- 172.21.15.186:0/3206064933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20240c71e0 0x7f20240c10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.222 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.211+0000 7f2029ea1700 1 -- 172.21.15.186:0/3206064933 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201c009ae0 con 0x7f20240c5260 2022-04-03T19:43:55.222 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.211+0000 7f202a6a2700 1 --2- 172.21.15.186:0/3206064933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20240c89e0 0x7f20240c15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:43:55.222 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.211+0000 7f2029ea1700 1 --2- 172.21.15.186:0/3206064933 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20240c5260 0x7f2024168460 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f20240c61a0 tx=0x7f2020016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:55.223 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.211+0000 7f2015ffb700 1 -- 172.21.15.186:0/3206064933 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2020003560 con 0x7f20240c5260 2022-04-03T19:43:55.223 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.211+0000 7f2015ffb700 1 -- 172.21.15.186:0/3206064933 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2020020070 con 0x7f20240c5260 2022-04-03T19:43:55.223 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.211+0000 7f202c105700 1 -- 172.21.15.186:0/3206064933 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20200092a0 con 0x7f20240c5260 2022-04-03T19:43:55.223 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.211+0000 7f2015ffb700 1 -- 172.21.15.186:0/3206064933 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2020003930 con 0x7f20240c5260 2022-04-03T19:43:55.223 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.211+0000 7f202c105700 1 -- 172.21.15.186:0/3206064933 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2024150120 con 0x7f20240c5260 2022-04-03T19:43:55.224 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.211+0000 7f2015ffb700 1 -- 172.21.15.186:0/3206064933 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f202002b020 con 0x7f20240c5260 2022-04-03T19:43:55.224 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.212+0000 7f2015ffb700 1 -- 172.21.15.186:0/3206064933 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f202001bd60 con 0x7f20240c5260 2022-04-03T19:43:55.224 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.212+0000 7f202c105700 1 -- 172.21.15.186:0/3206064933 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20240c0ef0 con 0x7f20240c5260 2022-04-03T19:43:55.224 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.215+0000 7f2015ffb700 1 -- 172.21.15.186:0/3206064933 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70738 (secure 0 0 0) 0x7f202001b450 con 0x7f20240c5260 2022-04-03T19:43:55.295 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.295+0000 7f202c105700 1 -- 172.21.15.186:0/3206064933 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ab1c89dc-57ce-4cc4-b8c7-9b0c19939601", "id": 1} v 0) v1 -- 0x7f20240c0ef0 con 0x7f20240c5260 2022-04-03T19:43:55.302 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.303+0000 7f2015ffb700 1 -- 172.21.15.186:0/3206064933 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ab1c89dc-57ce-4cc4-b8c7-9b0c19939601", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f202001c450 con 0x7f20240c5260 2022-04-03T19:43:55.303 INFO:teuthology.orchestra.run.smithi186.stdout:1 2022-04-03T19:43:55.304 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.304+0000 7f202c105700 1 -- 172.21.15.186:0/3206064933 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20240c5260 msgr2=0x7f2024168460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:55.304 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.304+0000 7f202c105700 1 --2- 172.21.15.186:0/3206064933 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20240c5260 0x7f2024168460 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f20240c61a0 tx=0x7f2020016000 comp rx=0 tx=0).stop 2022-04-03T19:43:55.305 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.304+0000 7f202c105700 1 -- 172.21.15.186:0/3206064933 shutdown_connections 2022-04-03T19:43:55.305 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.304+0000 7f202c105700 1 --2- 172.21.15.186:0/3206064933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20240c89e0 0x7f20240c15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.305 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.304+0000 7f202c105700 1 --2- 172.21.15.186:0/3206064933 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20240c5260 0x7f2024168460 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.305 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.304+0000 7f202c105700 1 --2- 172.21.15.186:0/3206064933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20240c71e0 0x7f20240c10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.305 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.304+0000 7f202c105700 1 -- 172.21.15.186:0/3206064933 >> 172.21.15.186:0/3206064933 conn(0x7f20240cc2c0 msgr2=0x7f202415b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:55.306 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.304+0000 7f202c105700 1 -- 172.21.15.186:0/3206064933 shutdown_connections 2022-04-03T19:43:55.306 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.304+0000 7f202c105700 1 -- 172.21.15.186:0/3206064933 wait complete. 2022-04-03T19:43:55.315 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 351e218b-7749-4ee9-b44a-0a5b52fb9be1 2 2022-04-03T19:43:55.362 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:43:55.362+0000 7f493b032000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-04-03T19:43:55.408 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:55.408+0000 7f207b38e000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-04-03T19:43:55.455 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:43:55.455+0000 7f493b032000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-04-03T19:43:55.481 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.482+0000 7f7ada419700 1 Processor -- start 2022-04-03T19:43:55.482 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.482+0000 7f7ada419700 1 -- start start 2022-04-03T19:43:55.486 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.482+0000 7f7ada419700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ad4067cb0 0x7f7ad40c5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.486 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.482+0000 7f7ada419700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7ad4068080 0x7f7ad4068ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.486 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.482+0000 7f7ada419700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7ad4069400 0x7f7ad40c08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.487 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.482+0000 7f7ada419700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ad40c1a40 con 0x7f7ad4068080 2022-04-03T19:43:55.487 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.482+0000 7f7ada419700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ad40c32b0 con 0x7f7ad4069400 2022-04-03T19:43:55.487 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.482+0000 7f7ada419700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad40c4b30 con 0x7f7ad4067cb0 2022-04-03T19:43:55.487 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.482+0000 7f7ad89b6700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7ad4069400 0x7f7ad40c08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.488 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.482+0000 7f7ad3fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ad4067cb0 0x7f7ad40c5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.488 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.482+0000 7f7ad89b6700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7ad4069400 0x7f7ad40c08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40102/0 (socket says 172.21.15.186:40102) 2022-04-03T19:43:55.488 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.482+0000 7f7ad89b6700 1 -- 172.21.15.186:0/391853000 learned_addr learned my addr 172.21.15.186:0/391853000 (peer_addr_for_me v2:172.21.15.186:0/0) 2022-04-03T19:43:55.488 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.483+0000 7f7ad37fe700 1 --2- 172.21.15.186:0/391853000 >> v2:172.21.15.77:3300/0 conn(0x7f7ad4068080 0x7f7ad4068ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.488 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.483+0000 7f7ad89b6700 1 -- 172.21.15.186:0/391853000 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ad4067cb0 msgr2=0x7f7ad40c5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:43:55.489 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.483+0000 7f7ad89b6700 1 --2- 172.21.15.186:0/391853000 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ad4067cb0 0x7f7ad40c5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.489 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.483+0000 7f7ad89b6700 1 -- 172.21.15.186:0/391853000 >> v2:172.21.15.77:3300/0 conn(0x7f7ad4068080 msgr2=0x7f7ad4068ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:43:55.489 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.483+0000 7f7ad89b6700 1 --2- 172.21.15.186:0/391853000 >> v2:172.21.15.77:3300/0 conn(0x7f7ad4068080 0x7f7ad4068ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.490 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.483+0000 7f7ad89b6700 1 -- 172.21.15.186:0/391853000 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad40c46a0 con 0x7f7ad4069400 2022-04-03T19:43:55.490 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.483+0000 7f7ad3fff700 1 --2- 172.21.15.186:0/391853000 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ad4067cb0 0x7f7ad40c5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:43:55.490 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.483+0000 7f7ad89b6700 1 --2- 172.21.15.186:0/391853000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ad4069400 0x7f7ad40c08c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7ad40c9640 tx=0x7f7acc009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c0441238690bf9f server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:55.490 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.483+0000 7f7ad2ffd700 1 -- 172.21.15.186:0/391853000 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7acc00e070 con 0x7f7ad4069400 2022-04-03T19:43:55.490 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.483+0000 7f7ad2ffd700 1 -- 172.21.15.186:0/391853000 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7acc00ba20 con 0x7f7ad4069400 2022-04-03T19:43:55.491 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.483+0000 7f7ad2ffd700 1 -- 172.21.15.186:0/391853000 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7acc0092e0 con 0x7f7ad4069400 2022-04-03T19:43:55.491 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.484+0000 7f7ada419700 1 -- 172.21.15.186:0/391853000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ad4069400 msgr2=0x7f7ad40c08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:55.491 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.484+0000 7f7ada419700 1 --2- 172.21.15.186:0/391853000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ad4069400 0x7f7ad40c08c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7ad40c9640 tx=0x7f7acc009840 comp rx=0 tx=0).stop 2022-04-03T19:43:55.491 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.484+0000 7f7ada419700 1 -- 172.21.15.186:0/391853000 shutdown_connections 2022-04-03T19:43:55.491 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.484+0000 7f7ada419700 1 --2- 172.21.15.186:0/391853000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ad4069400 0x7f7ad40c08c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.492 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.484+0000 7f7ada419700 1 --2- 172.21.15.186:0/391853000 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ad4067cb0 0x7f7ad40c5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.492 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.484+0000 7f7ada419700 1 --2- 172.21.15.186:0/391853000 >> v2:172.21.15.77:3300/0 conn(0x7f7ad4068080 0x7f7ad4068ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.492 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.484+0000 7f7ada419700 1 -- 172.21.15.186:0/391853000 >> 172.21.15.186:0/391853000 conn(0x7f7ad40626d0 msgr2=0x7f7ad4062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:55.492 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.484+0000 7f7ada419700 1 -- 172.21.15.186:0/391853000 shutdown_connections 2022-04-03T19:43:55.493 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.484+0000 7f7ada419700 1 -- 172.21.15.186:0/391853000 wait complete. 2022-04-03T19:43:55.493 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.485+0000 7f7ada419700 1 Processor -- start 2022-04-03T19:43:55.493 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.485+0000 7f7ada419700 1 -- start start 2022-04-03T19:43:55.493 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7ada419700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ad4067cb0 0x7f7ad4163690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.493 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7ada419700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ad4068080 0x7f7ad4151590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.494 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7ada419700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ad4069400 0x7f7ad4158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.494 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7ada419700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad4156000 con 0x7f7ad4069400 2022-04-03T19:43:55.494 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7ada419700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad414e930 con 0x7f7ad4067cb0 2022-04-03T19:43:55.494 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7ada419700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad414ee20 con 0x7f7ad4068080 2022-04-03T19:43:55.495 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7ad3fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ad4067cb0 0x7f7ad4163690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.495 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7ad37fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ad4068080 0x7f7ad4151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.495 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7ad3fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ad4067cb0 0x7f7ad4163690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40104/0 (socket says 172.21.15.186:40104) 2022-04-03T19:43:55.495 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7ad3fff700 1 -- 172.21.15.186:0/1940049191 learned_addr learned my addr 172.21.15.186:0/1940049191 (peer_addr_for_me v2:172.21.15.186:0/0) 2022-04-03T19:43:55.496 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f207b38e000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-04-03T19:43:55.497 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7ad89b6700 1 --2- 172.21.15.186:0/1940049191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ad4069400 0x7f7ad4158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.497 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7ad3fff700 1 -- 172.21.15.186:0/1940049191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ad4068080 msgr2=0x7f7ad4151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:55.498 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7ad3fff700 1 --2- 172.21.15.186:0/1940049191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ad4068080 0x7f7ad4151590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.498 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7ad3fff700 1 -- 172.21.15.186:0/1940049191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ad4069400 msgr2=0x7f7ad4158720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:55.498 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7ad3fff700 1 --2- 172.21.15.186:0/1940049191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ad4069400 0x7f7ad4158720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.498 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7ad3fff700 1 -- 172.21.15.186:0/1940049191 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac8009250 con 0x7f7ad4067cb0 2022-04-03T19:43:55.499 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7ad3fff700 1 --2- 172.21.15.186:0/1940049191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ad4067cb0 0x7f7ad4163690 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7ac800ea50 tx=0x7f7ac8006f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:55.499 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7abbfff700 1 -- 172.21.15.186:0/1940049191 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac8009600 con 0x7f7ad4067cb0 2022-04-03T19:43:55.499 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7abbfff700 1 -- 172.21.15.186:0/1940049191 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac8005880 con 0x7f7ad4067cb0 2022-04-03T19:43:55.499 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7abbfff700 1 -- 172.21.15.186:0/1940049191 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac8005c20 con 0x7f7ad4067cb0 2022-04-03T19:43:55.499 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.486+0000 7f7ada419700 1 -- 172.21.15.186:0/1940049191 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7acc009ae0 con 0x7f7ad4067cb0 2022-04-03T19:43:55.500 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.487+0000 7f7ada419700 1 -- 172.21.15.186:0/1940049191 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad40bf700 con 0x7f7ad4067cb0 2022-04-03T19:43:55.500 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.487+0000 7f7abbfff700 1 -- 172.21.15.186:0/1940049191 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f7ac8019070 con 0x7f7ad4067cb0 2022-04-03T19:43:55.500 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.487+0000 7f7abbfff700 1 -- 172.21.15.186:0/1940049191 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7f7ac8005d80 con 0x7f7ad4067cb0 2022-04-03T19:43:55.500 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.488+0000 7f7ada419700 1 -- 172.21.15.186:0/1940049191 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad404b890 con 0x7f7ad4067cb0 2022-04-03T19:43:55.501 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.490+0000 7f7abbfff700 1 -- 172.21.15.186:0/1940049191 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70738 (secure 0 0 0) 0x7f7ac803a470 con 0x7f7ad4067cb0 2022-04-03T19:43:55.575 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:55.575+0000 7f207b38e000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-04-03T19:43:55.584 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.584+0000 7f7ada419700 1 -- 172.21.15.186:0/1940049191 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "351e218b-7749-4ee9-b44a-0a5b52fb9be1", "id": 2} v 0) v1 -- 0x7f7ad4068630 con 0x7f7ad4067cb0 2022-04-03T19:43:55.591 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.592+0000 7f7abbfff700 1 -- 172.21.15.186:0/1940049191 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "351e218b-7749-4ee9-b44a-0a5b52fb9be1", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7ac8029030 con 0x7f7ad4067cb0 2022-04-03T19:43:55.592 INFO:teuthology.orchestra.run.smithi186.stdout:2 2022-04-03T19:43:55.593 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.593+0000 7f7ada419700 1 -- 172.21.15.186:0/1940049191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ad4067cb0 msgr2=0x7f7ad4163690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:55.593 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.593+0000 7f7ada419700 1 --2- 172.21.15.186:0/1940049191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ad4067cb0 0x7f7ad4163690 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7ac800ea50 tx=0x7f7ac8006f40 comp rx=0 tx=0).stop 2022-04-03T19:43:55.594 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.593+0000 7f7ada419700 1 -- 172.21.15.186:0/1940049191 shutdown_connections 2022-04-03T19:43:55.594 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.593+0000 7f7ada419700 1 --2- 172.21.15.186:0/1940049191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ad4069400 0x7f7ad4158720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.594 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.593+0000 7f7ada419700 1 --2- 172.21.15.186:0/1940049191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ad4067cb0 0x7f7ad4163690 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.594 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.593+0000 7f7ada419700 1 --2- 172.21.15.186:0/1940049191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ad4068080 0x7f7ad4151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.595 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.593+0000 7f7ada419700 1 -- 172.21.15.186:0/1940049191 >> 172.21.15.186:0/1940049191 conn(0x7f7ad40626d0 msgr2=0x7f7ad4153cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:55.595 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.593+0000 7f7ada419700 1 -- 172.21.15.186:0/1940049191 shutdown_connections 2022-04-03T19:43:55.595 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.593+0000 7f7ada419700 1 -- 172.21.15.186:0/1940049191 wait complete. 2022-04-03T19:43:55.605 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 1ba6ec8a-7cbf-4c12-8e21-c3097a263ae8 3 2022-04-03T19:43:55.717 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:43:55.717+0000 7f493b032000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-04-03T19:43:55.721 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.721+0000 7f546a991700 1 Processor -- start 2022-04-03T19:43:55.721 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.721+0000 7f546a991700 1 -- start start 2022-04-03T19:43:55.724 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.722+0000 7f546a991700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54640c7bb0 0x7f5464155100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.724 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.722+0000 7f546a991700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f54640c1d30 0x7f54641590b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.724 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.722+0000 7f546a991700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54641595f0 con 0x7f54640c1960 2022-04-03T19:43:55.724 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.722+0000 7f546a991700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f546406be30 con 0x7f54640c1d30 2022-04-03T19:43:55.725 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.722+0000 7f546a991700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5464069c80 con 0x7f54640c7bb0 2022-04-03T19:43:55.725 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.722+0000 7f546998f700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f54640c1960 0x7f546406c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:41916/0 (socket says 172.21.15.186:41916) 2022-04-03T19:43:55.725 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.722+0000 7f546998f700 1 -- 172.21.15.186:0/1149519611 learned_addr learned my addr 172.21.15.186:0/1149519611 (peer_addr_for_me v1:172.21.15.186:0/0) 2022-04-03T19:43:55.725 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.722+0000 7f546918e700 1 --2- 172.21.15.186:0/1149519611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54640c7bb0 0x7f5464155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.726 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.723+0000 7f546a190700 1 --2- 172.21.15.186:0/1149519611 >> v2:172.21.15.77:3300/0 conn(0x7f54640c1d30 0x7f54641590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.726 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.723+0000 7f546918e700 1 -- 172.21.15.186:0/1149519611 >> v2:172.21.15.77:3300/0 conn(0x7f54640c1d30 msgr2=0x7f54641590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:43:55.726 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.723+0000 7f546918e700 1 --2- 172.21.15.186:0/1149519611 >> v2:172.21.15.77:3300/0 conn(0x7f54640c1d30 0x7f54641590b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.726 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.723+0000 7f546918e700 1 -- 172.21.15.186:0/1149519611 >> v1:172.21.15.186:6789/0 conn(0x7f54640c1960 legacy=0x7f546406c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:55.726 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.723+0000 7f546918e700 1 -- 172.21.15.186:0/1149519611 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5464156010 con 0x7f54640c7bb0 2022-04-03T19:43:55.727 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.723+0000 7f546a190700 1 --2- 172.21.15.186:0/1149519611 >> v2:172.21.15.77:3300/0 conn(0x7f54640c1d30 0x7f54641590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-04-03T19:43:55.727 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.723+0000 7f546918e700 1 --2- 172.21.15.186:0/1149519611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54640c7bb0 0x7f5464155100 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5460009760 tx=0x7f5460009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=242625f07871b166 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:55.727 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.723+0000 7f546898d700 1 -- 172.21.15.186:0/1149519611 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5460011630 con 0x7f54640c7bb0 2022-04-03T19:43:55.727 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.723+0000 7f546898d700 1 -- 172.21.15.186:0/1149519611 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5460011790 con 0x7f54640c7bb0 2022-04-03T19:43:55.727 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.723+0000 7f546898d700 1 -- 172.21.15.186:0/1149519611 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5460011b30 con 0x7f54640c7bb0 2022-04-03T19:43:55.728 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.724+0000 7f546a991700 1 -- 172.21.15.186:0/1149519611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54640c7bb0 msgr2=0x7f5464155100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:55.728 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.724+0000 7f546a991700 1 --2- 172.21.15.186:0/1149519611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54640c7bb0 0x7f5464155100 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5460009760 tx=0x7f5460009060 comp rx=0 tx=0).stop 2022-04-03T19:43:55.728 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.724+0000 7f546a991700 1 -- 172.21.15.186:0/1149519611 shutdown_connections 2022-04-03T19:43:55.728 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.724+0000 7f546a991700 1 --2- 172.21.15.186:0/1149519611 >> v2:172.21.15.77:3300/0 conn(0x7f54640c1d30 0x7f54641590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.728 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.724+0000 7f546a991700 1 --2- 172.21.15.186:0/1149519611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54640c7bb0 0x7f5464155100 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.729 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.724+0000 7f546a991700 1 -- 172.21.15.186:0/1149519611 >> 172.21.15.186:0/1149519611 conn(0x7f5464067720 msgr2=0x7f5464067b20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:55.729 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.724+0000 7f546a991700 1 -- 172.21.15.186:0/1149519611 shutdown_connections 2022-04-03T19:43:55.729 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.724+0000 7f546a991700 1 -- 172.21.15.186:0/1149519611 wait complete. 2022-04-03T19:43:55.732 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.725+0000 7f546a991700 1 Processor -- start 2022-04-03T19:43:55.732 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.725+0000 7f546a991700 1 -- start start 2022-04-03T19:43:55.732 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.725+0000 7f546a991700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54640c1960 0x7f5464157460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.732 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.725+0000 7f546a991700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54640c1d30 0x7f5464153810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.733 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.725+0000 7f546a991700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54640c7bb0 0x7f5464153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.733 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.725+0000 7f546a991700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5464153380 con 0x7f54640c1d30 2022-04-03T19:43:55.733 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.725+0000 7f546a991700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5464152d70 con 0x7f54640c7bb0 2022-04-03T19:43:55.733 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.725+0000 7f546a991700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5464166490 con 0x7f54640c1960 2022-04-03T19:43:55.733 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.725+0000 7f546a190700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54640c7bb0 0x7f5464153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.734 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.726+0000 7f546998f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54640c1960 0x7f5464157460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.734 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.726+0000 7f546a190700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54640c7bb0 0x7f5464153d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40120/0 (socket says 172.21.15.186:40120) 2022-04-03T19:43:55.734 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.726+0000 7f546a190700 1 -- 172.21.15.186:0/247437637 learned_addr learned my addr 172.21.15.186:0/247437637 (peer_addr_for_me v2:172.21.15.186:0/0) 2022-04-03T19:43:55.734 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.726+0000 7f546918e700 1 --2- 172.21.15.186:0/247437637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54640c1d30 0x7f5464153810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.734 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.726+0000 7f546998f700 1 -- 172.21.15.186:0/247437637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54640c7bb0 msgr2=0x7f5464153d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:55.735 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.726+0000 7f546998f700 1 --2- 172.21.15.186:0/247437637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54640c7bb0 0x7f5464153d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.735 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.726+0000 7f546998f700 1 -- 172.21.15.186:0/247437637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54640c1d30 msgr2=0x7f5464153810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:55.735 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.726+0000 7f546998f700 1 --2- 172.21.15.186:0/247437637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54640c1d30 0x7f5464153810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.735 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.726+0000 7f546998f700 1 -- 172.21.15.186:0/247437637 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54600092a0 con 0x7f54640c1960 2022-04-03T19:43:55.736 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.726+0000 7f546a190700 1 --2- 172.21.15.186:0/247437637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54640c7bb0 0x7f5464153d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:43:55.736 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.726+0000 7f546998f700 1 --2- 172.21.15.186:0/247437637 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54640c1960 0x7f5464157460 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f545400c410 tx=0x7f5454007d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:55.736 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.726+0000 7f5451ffb700 1 -- 172.21.15.186:0/247437637 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545400ce70 con 0x7f54640c1960 2022-04-03T19:43:55.736 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.726+0000 7f5451ffb700 1 -- 172.21.15.186:0/247437637 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54540072e0 con 0x7f54640c1960 2022-04-03T19:43:55.736 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.726+0000 7f546a991700 1 -- 172.21.15.186:0/247437637 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5464154eb0 con 0x7f54640c1960 2022-04-03T19:43:55.737 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.726+0000 7f5451ffb700 1 -- 172.21.15.186:0/247437637 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5454005780 con 0x7f54640c1960 2022-04-03T19:43:55.737 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.726+0000 7f546a991700 1 -- 172.21.15.186:0/247437637 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5464153010 con 0x7f54640c1960 2022-04-03T19:43:55.737 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.727+0000 7f5451ffb700 1 -- 172.21.15.186:0/247437637 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f5454019030 con 0x7f54640c1960 2022-04-03T19:43:55.737 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.727+0000 7f5451ffb700 1 -- 172.21.15.186:0/247437637 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f54540128a0 con 0x7f54640c1960 2022-04-03T19:43:55.737 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.727+0000 7f546a991700 1 -- 172.21.15.186:0/247437637 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5464068eb0 con 0x7f54640c1960 2022-04-03T19:43:55.738 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.729+0000 7f5451ffb700 1 -- 172.21.15.186:0/247437637 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70738 (secure 0 0 0) 0x7f5454017070 con 0x7f54640c1960 2022-04-03T19:43:55.810 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.811+0000 7f546a991700 1 -- 172.21.15.186:0/247437637 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1ba6ec8a-7cbf-4c12-8e21-c3097a263ae8", "id": 3} v 0) v1 -- 0x7f5464155770 con 0x7f54640c1960 2022-04-03T19:43:55.818 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.818+0000 7f5451ffb700 1 -- 172.21.15.186:0/247437637 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1ba6ec8a-7cbf-4c12-8e21-c3097a263ae8", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f54641595f0 con 0x7f54640c1960 2022-04-03T19:43:55.818 INFO:teuthology.orchestra.run.smithi186.stdout:3 2022-04-03T19:43:55.820 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.819+0000 7f546a991700 1 -- 172.21.15.186:0/247437637 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54640c1960 msgr2=0x7f5464157460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:55.820 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.819+0000 7f546a991700 1 --2- 172.21.15.186:0/247437637 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54640c1960 0x7f5464157460 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f545400c410 tx=0x7f5454007d30 comp rx=0 tx=0).stop 2022-04-03T19:43:55.820 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.819+0000 7f546a991700 1 -- 172.21.15.186:0/247437637 shutdown_connections 2022-04-03T19:43:55.820 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.819+0000 7f546a991700 1 --2- 172.21.15.186:0/247437637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54640c7bb0 0x7f5464153d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.821 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.819+0000 7f546a991700 1 --2- 172.21.15.186:0/247437637 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54640c1960 0x7f5464157460 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.821 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.819+0000 7f546a991700 1 --2- 172.21.15.186:0/247437637 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f54640c1d30 0x7f5464153810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.821 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.819+0000 7f546a991700 1 -- 172.21.15.186:0/247437637 >> 172.21.15.186:0/247437637 conn(0x7f5464067720 msgr2=0x7f54640c00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:55.821 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.819+0000 7f546a991700 1 -- 172.21.15.186:0/247437637 shutdown_connections 2022-04-03T19:43:55.821 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.819+0000 7f546a991700 1 -- 172.21.15.186:0/247437637 wait complete. 2022-04-03T19:43:55.825 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:43:55.825+0000 7f493b032000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-04-03T19:43:55.831 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 3faf4c09-846c-4133-856b-7f53c535a67d 4 2022-04-03T19:43:55.934 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:43:55.934+0000 7f493b032000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-04-03T19:43:55.940 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:55.940+0000 7f207b38e000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-04-03T19:43:55.945 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.945+0000 7f62ff59e700 1 Processor -- start 2022-04-03T19:43:55.945 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.945+0000 7f62ff59e700 1 -- start start 2022-04-03T19:43:55.948 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.946+0000 7f62ff59e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f63000c1960 0x7f6300155100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.948 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.946+0000 7f62ff59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63000c1d30 0x7f63001590b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.949 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.946+0000 7f62ff59e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63001595f0 con 0x7f63000c7bb0 2022-04-03T19:43:55.949 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.946+0000 7f62ff59e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f630006be30 con 0x7f63000c1960 2022-04-03T19:43:55.949 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.946+0000 7f62ff59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6300069c80 con 0x7f63000c1d30 2022-04-03T19:43:55.949 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.946+0000 7f62fed9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63000c1d30 0x7f63001590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.950 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.946+0000 7f62fe59c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f63000c1960 0x7f6300155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.950 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.946+0000 7f62fed9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63000c1d30 0x7f63001590b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:49640/0 (socket says 172.21.15.186:49640) 2022-04-03T19:43:55.950 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.946+0000 7f62fe59c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f63000c1960 0x7f6300155100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40122/0 (socket says 172.21.15.186:40122) 2022-04-03T19:43:55.950 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.946+0000 7f62fed9d700 1 -- 172.21.15.186:0/3963271786 learned_addr learned my addr 172.21.15.186:0/3963271786 (peer_addr_for_me v2:172.21.15.186:0/0) 2022-04-03T19:43:55.951 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.946+0000 7f62fe59c700 1 -- 172.21.15.186:0/3963271786 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63000c1d30 msgr2=0x7f63001590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:43:55.951 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.946+0000 7f62fe59c700 1 --2- 172.21.15.186:0/3963271786 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63000c1d30 0x7f63001590b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.951 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.946+0000 7f62fe59c700 1 -- 172.21.15.186:0/3963271786 >> v1:172.21.15.77:6789/0 conn(0x7f63000c7bb0 legacy=0x7f630006c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:43:55.951 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.946+0000 7f62fe59c700 1 -- 172.21.15.186:0/3963271786 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6300156010 con 0x7f63000c1960 2022-04-03T19:43:55.951 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.946+0000 7f62fe59c700 1 --2- 172.21.15.186:0/3963271786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63000c1960 0x7f6300155100 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009760 tx=0x7f62f4009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=754200662b4a51a7 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:55.952 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.947+0000 7f62fd59a700 1 -- 172.21.15.186:0/3963271786 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4004030 con 0x7f63000c1960 2022-04-03T19:43:55.952 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.947+0000 7f62fd59a700 1 -- 172.21.15.186:0/3963271786 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f4004190 con 0x7f63000c1960 2022-04-03T19:43:55.952 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.947+0000 7f62fd59a700 1 -- 172.21.15.186:0/3963271786 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4011630 con 0x7f63000c1960 2022-04-03T19:43:55.952 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.947+0000 7f62ff59e700 1 -- 172.21.15.186:0/3963271786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63000c1960 msgr2=0x7f6300155100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:55.952 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.947+0000 7f62ff59e700 1 --2- 172.21.15.186:0/3963271786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63000c1960 0x7f6300155100 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009760 tx=0x7f62f4009060 comp rx=0 tx=0).stop 2022-04-03T19:43:55.953 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.947+0000 7f62ff59e700 1 -- 172.21.15.186:0/3963271786 shutdown_connections 2022-04-03T19:43:55.953 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.947+0000 7f62ff59e700 1 --2- 172.21.15.186:0/3963271786 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63000c1d30 0x7f63001590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.953 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.947+0000 7f62ff59e700 1 --2- 172.21.15.186:0/3963271786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63000c1960 0x7f6300155100 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.953 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.947+0000 7f62ff59e700 1 -- 172.21.15.186:0/3963271786 >> 172.21.15.186:0/3963271786 conn(0x7f6300067720 msgr2=0x7f6300067b20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:55.954 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.947+0000 7f62ff59e700 1 -- 172.21.15.186:0/3963271786 shutdown_connections 2022-04-03T19:43:55.954 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.947+0000 7f62ff59e700 1 -- 172.21.15.186:0/3963271786 wait complete. 2022-04-03T19:43:55.954 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.948+0000 7f62ff59e700 1 Processor -- start 2022-04-03T19:43:55.954 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.948+0000 7f62ff59e700 1 -- start start 2022-04-03T19:43:55.957 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.949+0000 7f62ff59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63000c1960 0x7f6300166120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.957 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.949+0000 7f62ff59e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63000c1d30 0x7f6300152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.958 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.949+0000 7f62ff59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63000c7bb0 0x7f63000aac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:55.958 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.949+0000 7f62ff59e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63000a98a0 con 0x7f63000c7bb0 2022-04-03T19:43:55.958 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.949+0000 7f62ff59e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63000a9b80 con 0x7f63000c1d30 2022-04-03T19:43:55.958 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.949+0000 7f62ff59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63000a9e60 con 0x7f63000c1960 2022-04-03T19:43:55.959 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.949+0000 7f62fe59c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63000c1960 0x7f6300166120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.959 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.949+0000 7f62fdd9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63000c1d30 0x7f6300152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.959 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.949+0000 7f62fe59c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63000c1960 0x7f6300166120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:49642/0 (socket says 172.21.15.186:49642) 2022-04-03T19:43:55.959 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.949+0000 7f62fe59c700 1 -- 172.21.15.186:0/3259402425 learned_addr learned my addr 172.21.15.186:0/3259402425 (peer_addr_for_me v2:172.21.15.186:0/0) 2022-04-03T19:43:55.959 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.949+0000 7f62fed9d700 1 --2- 172.21.15.186:0/3259402425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63000c7bb0 0x7f63000aac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:55.960 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.949+0000 7f62fe59c700 1 -- 172.21.15.186:0/3259402425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63000c1d30 msgr2=0x7f6300152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:55.960 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.949+0000 7f62fe59c700 1 --2- 172.21.15.186:0/3259402425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63000c1d30 0x7f6300152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.960 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.949+0000 7f62fe59c700 1 -- 172.21.15.186:0/3259402425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63000c7bb0 msgr2=0x7f63000aac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:55.960 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.949+0000 7f62fe59c700 1 --2- 172.21.15.186:0/3259402425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63000c7bb0 0x7f63000aac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:55.961 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.949+0000 7f62fe59c700 1 -- 172.21.15.186:0/3259402425 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f40092a0 con 0x7f63000c1960 2022-04-03T19:43:55.961 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.949+0000 7f62fed9d700 1 --2- 172.21.15.186:0/3259402425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63000c7bb0 0x7f63000aac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:43:55.961 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.949+0000 7f62fdd9b700 1 --2- 172.21.15.186:0/3259402425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63000c1d30 0x7f6300152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:43:55.961 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.950+0000 7f62fe59c700 1 --2- 172.21.15.186:0/3259402425 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63000c1960 0x7f6300166120 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009e00 tx=0x7f62f4016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:55.961 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.950+0000 7f62ee7fc700 1 -- 172.21.15.186:0/3259402425 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4003770 con 0x7f63000c1960 2022-04-03T19:43:55.962 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.950+0000 7f62ee7fc700 1 -- 172.21.15.186:0/3259402425 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f40038f0 con 0x7f63000c1960 2022-04-03T19:43:55.962 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.950+0000 7f62ff59e700 1 -- 172.21.15.186:0/3259402425 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63000aa140 con 0x7f63000c1960 2022-04-03T19:43:55.962 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.950+0000 7f62ee7fc700 1 -- 172.21.15.186:0/3259402425 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4003cb0 con 0x7f63000c1960 2022-04-03T19:43:55.962 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.950+0000 7f62ff59e700 1 -- 172.21.15.186:0/3259402425 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63000aa700 con 0x7f63000c1960 2022-04-03T19:43:55.963 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.951+0000 7f62ee7fc700 1 -- 172.21.15.186:0/3259402425 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f62f4017020 con 0x7f63000c1960 2022-04-03T19:43:55.963 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.951+0000 7f62ee7fc700 1 -- 172.21.15.186:0/3259402425 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f62f4003e10 con 0x7f63000c1960 2022-04-03T19:43:55.963 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.951+0000 7f62ff59e700 1 -- 172.21.15.186:0/3259402425 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6300068eb0 con 0x7f63000c1960 2022-04-03T19:43:55.963 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:55.954+0000 7f62ee7fc700 1 -- 172.21.15.186:0/3259402425 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70738 (secure 0 0 0) 0x7f62f4033060 con 0x7f63000c1960 2022-04-03T19:43:56.010 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:56.010+0000 7f207b38e000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-04-03T19:43:56.035 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.036+0000 7f62ff59e700 1 -- 172.21.15.186:0/3259402425 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3faf4c09-846c-4133-856b-7f53c535a67d", "id": 4} v 0) v1 -- 0x7f6300155770 con 0x7f63000c1960 2022-04-03T19:43:56.043 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.043+0000 7f62ee7fc700 1 -- 172.21.15.186:0/3259402425 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3faf4c09-846c-4133-856b-7f53c535a67d", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f63001595f0 con 0x7f63000c1960 2022-04-03T19:43:56.043 INFO:teuthology.orchestra.run.smithi186.stdout:4 2022-04-03T19:43:56.044 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.044+0000 7f62ff59e700 1 -- 172.21.15.186:0/3259402425 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63000c1960 msgr2=0x7f6300166120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:56.045 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.044+0000 7f62ff59e700 1 --2- 172.21.15.186:0/3259402425 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63000c1960 0x7f6300166120 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009e00 tx=0x7f62f4016000 comp rx=0 tx=0).stop 2022-04-03T19:43:56.045 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.044+0000 7f62ff59e700 1 -- 172.21.15.186:0/3259402425 shutdown_connections 2022-04-03T19:43:56.045 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.044+0000 7f62ff59e700 1 --2- 172.21.15.186:0/3259402425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63000c7bb0 0x7f63000aac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.045 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.044+0000 7f62ff59e700 1 --2- 172.21.15.186:0/3259402425 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63000c1960 0x7f6300166120 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.046 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.044+0000 7f62ff59e700 1 --2- 172.21.15.186:0/3259402425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63000c1d30 0x7f6300152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.046 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.044+0000 7f62ff59e700 1 -- 172.21.15.186:0/3259402425 >> 172.21.15.186:0/3259402425 conn(0x7f6300067720 msgr2=0x7f630006cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:56.046 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.044+0000 7f62ff59e700 1 -- 172.21.15.186:0/3259402425 shutdown_connections 2022-04-03T19:43:56.046 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.044+0000 7f62ff59e700 1 -- 172.21.15.186:0/3259402425 wait complete. 2022-04-03T19:43:56.056 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new ffe41d00-d0aa-42f4-82c1-5cced13c1bc3 5 2022-04-03T19:43:56.174 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.174+0000 7f7503b38700 1 Processor -- start 2022-04-03T19:43:56.175 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.175+0000 7f7503b38700 1 -- start start 2022-04-03T19:43:56.178 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.175+0000 7f7503b38700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74fc0c5260 0x7f74fc0c2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:56.178 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.175+0000 7f7503b38700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f74fc0c89e0 0x7f74fc06c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:56.178 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.175+0000 7f7503b38700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74fc0c7d00 con 0x7f74fc0c71e0 2022-04-03T19:43:56.179 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.175+0000 7f7503b38700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74fc0c7740 con 0x7f74fc0c89e0 2022-04-03T19:43:56.179 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.176+0000 7f7503b38700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc0c7a20 con 0x7f74fc0c5260 2022-04-03T19:43:56.179 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.176+0000 7f75018d4700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74fc0c5260 0x7f74fc0c2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:56.179 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.176+0000 7f75020d5700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f74fc0c89e0 0x7f74fc06c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:56.180 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.176+0000 7f75020d5700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f74fc0c89e0 0x7f74fc06c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40138/0 (socket says 172.21.15.186:40138) 2022-04-03T19:43:56.180 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.176+0000 7f75018d4700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74fc0c5260 0x7f74fc0c2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:49648/0 (socket says 172.21.15.186:49648) 2022-04-03T19:43:56.180 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.176+0000 7f75020d5700 1 -- 172.21.15.186:0/3978556921 learned_addr learned my addr 172.21.15.186:0/3978556921 (peer_addr_for_me v2:172.21.15.186:0/0) 2022-04-03T19:43:56.180 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.176+0000 7f75018d4700 1 -- 172.21.15.186:0/3978556921 >> v2:172.21.15.186:3300/0 conn(0x7f74fc0c89e0 msgr2=0x7f74fc06c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:43:56.181 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.176+0000 7f75018d4700 1 --2- 172.21.15.186:0/3978556921 >> v2:172.21.15.186:3300/0 conn(0x7f74fc0c89e0 0x7f74fc06c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.181 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.176+0000 7f75018d4700 1 -- 172.21.15.186:0/3978556921 >> v1:172.21.15.77:6789/0 conn(0x7f74fc0c71e0 legacy=0x7f74fc0c3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:43:56.181 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.176+0000 7f75018d4700 1 -- 172.21.15.186:0/3978556921 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74fc06bb50 con 0x7f74fc0c5260 2022-04-03T19:43:56.181 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.176+0000 7f75020d5700 1 --2- 172.21.15.186:0/3978556921 >> v2:172.21.15.186:3300/0 conn(0x7f74fc0c89e0 0x7f74fc06c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:43:56.181 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.176+0000 7f75018d4700 1 --2- 172.21.15.186:0/3978556921 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74fc0c5260 0x7f74fc0c2b50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009760 tx=0x7f74f0009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ceb3c457bd7be60 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:56.182 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.176+0000 7f75008d2700 1 -- 172.21.15.186:0/3978556921 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f0004030 con 0x7f74fc0c5260 2022-04-03T19:43:56.182 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.176+0000 7f75008d2700 1 -- 172.21.15.186:0/3978556921 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f0004190 con 0x7f74fc0c5260 2022-04-03T19:43:56.182 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.176+0000 7f75008d2700 1 -- 172.21.15.186:0/3978556921 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f0011630 con 0x7f74fc0c5260 2022-04-03T19:43:56.182 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.177+0000 7f7503b38700 1 -- 172.21.15.186:0/3978556921 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74fc0c5260 msgr2=0x7f74fc0c2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:56.183 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.177+0000 7f7503b38700 1 --2- 172.21.15.186:0/3978556921 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74fc0c5260 0x7f74fc0c2b50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009760 tx=0x7f74f0009060 comp rx=0 tx=0).stop 2022-04-03T19:43:56.183 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.177+0000 7f7503b38700 1 -- 172.21.15.186:0/3978556921 shutdown_connections 2022-04-03T19:43:56.183 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.177+0000 7f7503b38700 1 --2- 172.21.15.186:0/3978556921 >> v2:172.21.15.186:3300/0 conn(0x7f74fc0c89e0 0x7f74fc06c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.183 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.177+0000 7f7503b38700 1 --2- 172.21.15.186:0/3978556921 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74fc0c5260 0x7f74fc0c2b50 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.183 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.177+0000 7f7503b38700 1 -- 172.21.15.186:0/3978556921 >> 172.21.15.186:0/3978556921 conn(0x7f74fc0cc2c0 msgr2=0x7f74fc0c82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:56.184 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.177+0000 7f7503b38700 1 -- 172.21.15.186:0/3978556921 shutdown_connections 2022-04-03T19:43:56.184 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.177+0000 7f7503b38700 1 -- 172.21.15.186:0/3978556921 wait complete. 2022-04-03T19:43:56.184 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.178+0000 7f7503b38700 1 Processor -- start 2022-04-03T19:43:56.184 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.178+0000 7f7503b38700 1 -- start start 2022-04-03T19:43:56.187 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f7503b38700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74fc0c5260 0x7f74fc168460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:56.187 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f7503b38700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74fc0c71e0 0x7f74fc0c10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:56.187 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f7503b38700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74fc0c89e0 0x7f74fc0c15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:56.188 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f7503b38700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc14fc30 con 0x7f74fc0c89e0 2022-04-03T19:43:56.188 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f7503b38700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc0c0c10 con 0x7f74fc0c5260 2022-04-03T19:43:56.188 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f7503b38700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc168ec0 con 0x7f74fc0c71e0 2022-04-03T19:43:56.188 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f75018d4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74fc0c5260 0x7f74fc168460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:56.189 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f75018d4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74fc0c5260 0x7f74fc168460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40140/0 (socket says 172.21.15.186:40140) 2022-04-03T19:43:56.189 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f75018d4700 1 -- 172.21.15.186:0/2305966230 learned_addr learned my addr 172.21.15.186:0/2305966230 (peer_addr_for_me v2:172.21.15.186:0/0) 2022-04-03T19:43:56.189 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f75010d3700 1 --2- 172.21.15.186:0/2305966230 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74fc0c71e0 0x7f74fc0c10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:56.189 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f75020d5700 1 --2- 172.21.15.186:0/2305966230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74fc0c89e0 0x7f74fc0c15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:56.189 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f75018d4700 1 -- 172.21.15.186:0/2305966230 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74fc0c71e0 msgr2=0x7f74fc0c10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:56.190 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f75018d4700 1 --2- 172.21.15.186:0/2305966230 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74fc0c71e0 0x7f74fc0c10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.190 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f75018d4700 1 -- 172.21.15.186:0/2305966230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74fc0c89e0 msgr2=0x7f74fc0c15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:56.190 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f75018d4700 1 --2- 172.21.15.186:0/2305966230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74fc0c89e0 0x7f74fc0c15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.190 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f75018d4700 1 -- 172.21.15.186:0/2305966230 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f4009ae0 con 0x7f74fc0c5260 2022-04-03T19:43:56.191 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f75020d5700 1 --2- 172.21.15.186:0/2305966230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74fc0c89e0 0x7f74fc0c15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-04-03T19:43:56.191 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f75018d4700 1 --2- 172.21.15.186:0/2305966230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74fc0c5260 0x7f74fc168460 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f74fc0c61a0 tx=0x7f74f0016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:56.191 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f74edffb700 1 -- 172.21.15.186:0/2305966230 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f0003560 con 0x7f74fc0c5260 2022-04-03T19:43:56.191 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f74edffb700 1 -- 172.21.15.186:0/2305966230 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f0020070 con 0x7f74fc0c5260 2022-04-03T19:43:56.191 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f7503b38700 1 -- 172.21.15.186:0/2305966230 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f00092a0 con 0x7f74fc0c5260 2022-04-03T19:43:56.192 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.179+0000 7f74edffb700 1 -- 172.21.15.186:0/2305966230 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f0003930 con 0x7f74fc0c5260 2022-04-03T19:43:56.192 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.180+0000 7f7503b38700 1 -- 172.21.15.186:0/2305966230 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74fc150120 con 0x7f74fc0c5260 2022-04-03T19:43:56.192 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.180+0000 7f74edffb700 1 -- 172.21.15.186:0/2305966230 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f74f002b020 con 0x7f74fc0c5260 2022-04-03T19:43:56.192 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.180+0000 7f74edffb700 1 -- 172.21.15.186:0/2305966230 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7f74f001c450 con 0x7f74fc0c5260 2022-04-03T19:43:56.193 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.181+0000 7f7503b38700 1 -- 172.21.15.186:0/2305966230 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74fc0c0ef0 con 0x7f74fc0c5260 2022-04-03T19:43:56.193 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.183+0000 7f74edffb700 1 -- 172.21.15.186:0/2305966230 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70738 (secure 0 0 0) 0x7f74f001b450 con 0x7f74fc0c5260 2022-04-03T19:43:56.276 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.277+0000 7f7503b38700 1 -- 172.21.15.186:0/2305966230 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ffe41d00-d0aa-42f4-82c1-5cced13c1bc3", "id": 5} v 0) v1 -- 0x7f74fc0c0ef0 con 0x7f74fc0c5260 2022-04-03T19:43:56.284 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.284+0000 7f74edffb700 1 -- 172.21.15.186:0/2305966230 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ffe41d00-d0aa-42f4-82c1-5cced13c1bc3", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f74f001b950 con 0x7f74fc0c5260 2022-04-03T19:43:56.284 INFO:teuthology.orchestra.run.smithi186.stdout:5 2022-04-03T19:43:56.286 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.285+0000 7f7503b38700 1 -- 172.21.15.186:0/2305966230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74fc0c5260 msgr2=0x7f74fc168460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:56.286 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.285+0000 7f7503b38700 1 --2- 172.21.15.186:0/2305966230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74fc0c5260 0x7f74fc168460 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f74fc0c61a0 tx=0x7f74f0016000 comp rx=0 tx=0).stop 2022-04-03T19:43:56.286 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.285+0000 7f7503b38700 1 -- 172.21.15.186:0/2305966230 shutdown_connections 2022-04-03T19:43:56.287 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.285+0000 7f7503b38700 1 --2- 172.21.15.186:0/2305966230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74fc0c89e0 0x7f74fc0c15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.287 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.285+0000 7f7503b38700 1 --2- 172.21.15.186:0/2305966230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74fc0c5260 0x7f74fc168460 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.287 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.285+0000 7f7503b38700 1 --2- 172.21.15.186:0/2305966230 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74fc0c71e0 0x7f74fc0c10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.287 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.285+0000 7f7503b38700 1 -- 172.21.15.186:0/2305966230 >> 172.21.15.186:0/2305966230 conn(0x7f74fc0cc2c0 msgr2=0x7f74fc15b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:56.287 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.286+0000 7f7503b38700 1 -- 172.21.15.186:0/2305966230 shutdown_connections 2022-04-03T19:43:56.288 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.286+0000 7f7503b38700 1 -- 172.21.15.186:0/2305966230 wait complete. 2022-04-03T19:43:56.297 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new b0344ee8-deca-4e9b-934c-5866d90d0073 6 2022-04-03T19:43:56.402 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:43:56.402+0000 7f493b032000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-04-03T19:43:56.411 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.412+0000 7f6573861700 1 Processor -- start 2022-04-03T19:43:56.412 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.412+0000 7f6573861700 1 -- start start 2022-04-03T19:43:56.414 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.413+0000 7f6573861700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f656c0c1210 0x7f656c0c3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:56.414 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.413+0000 7f6573861700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656c0c2de0 con 0x7f656c06b330 2022-04-03T19:43:56.414 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.413+0000 7f6573861700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656c0c3960 con 0x7f656c0bf8a0 2022-04-03T19:43:56.415 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.413+0000 7f6573861700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f656c156880 con 0x7f656c0c1210 2022-04-03T19:43:56.415 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.413+0000 7f65715fd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f656c0c1210 0x7f656c0c3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:56.415 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.413+0000 7f6570dfc700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f656c0bf8a0 0x7f656c155a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:41954/0 (socket says 172.21.15.186:41954) 2022-04-03T19:43:56.415 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.413+0000 7f6570dfc700 1 -- 172.21.15.186:0/768917227 learned_addr learned my addr 172.21.15.186:0/768917227 (peer_addr_for_me v1:172.21.15.186:0/0) 2022-04-03T19:43:56.416 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.413+0000 7f655ffff700 1 -- 172.21.15.186:0/768917227 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407702319 0 0) 0x7f656c0c3960 con 0x7f656c0bf8a0 2022-04-03T19:43:56.416 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.413+0000 7f65715fd700 1 -- 172.21.15.186:0/768917227 >> v1:172.21.15.186:6789/0 conn(0x7f656c0bf8a0 legacy=0x7f656c155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:56.416 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.413+0000 7f65715fd700 1 -- 172.21.15.186:0/768917227 >> v1:172.21.15.77:6789/0 conn(0x7f656c06b330 legacy=0x7f656c156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:43:56.416 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.413+0000 7f65715fd700 1 -- 172.21.15.186:0/768917227 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656c0c5750 con 0x7f656c0c1210 2022-04-03T19:43:56.416 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.413+0000 7f65715fd700 1 --2- 172.21.15.186:0/768917227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f656c0c1210 0x7f656c0c3df0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f656000ba30 tx=0x7f656000b300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e30fa7eca5784ae8 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:56.417 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.413+0000 7f655ffff700 1 -- 172.21.15.186:0/768917227 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6560004030 con 0x7f656c0c1210 2022-04-03T19:43:56.417 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.413+0000 7f655ffff700 1 -- 172.21.15.186:0/768917227 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6560004190 con 0x7f656c0c1210 2022-04-03T19:43:56.417 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.413+0000 7f655ffff700 1 -- 172.21.15.186:0/768917227 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656000da30 con 0x7f656c0c1210 2022-04-03T19:43:56.417 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.414+0000 7f6573861700 1 -- 172.21.15.186:0/768917227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f656c0c1210 msgr2=0x7f656c0c3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:56.417 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.414+0000 7f6573861700 1 --2- 172.21.15.186:0/768917227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f656c0c1210 0x7f656c0c3df0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f656000ba30 tx=0x7f656000b300 comp rx=0 tx=0).stop 2022-04-03T19:43:56.418 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.414+0000 7f6573861700 1 -- 172.21.15.186:0/768917227 shutdown_connections 2022-04-03T19:43:56.418 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.414+0000 7f6573861700 1 --2- 172.21.15.186:0/768917227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f656c0c1210 0x7f656c0c3df0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.418 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.414+0000 7f6573861700 1 -- 172.21.15.186:0/768917227 >> 172.21.15.186:0/768917227 conn(0x7f656c0c8fb0 msgr2=0x7f656c0c4fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:56.418 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.414+0000 7f6573861700 1 -- 172.21.15.186:0/768917227 shutdown_connections 2022-04-03T19:43:56.419 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.414+0000 7f6573861700 1 -- 172.21.15.186:0/768917227 wait complete. 2022-04-03T19:43:56.421 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.415+0000 7f6573861700 1 Processor -- start 2022-04-03T19:43:56.421 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.415+0000 7f6573861700 1 -- start start 2022-04-03T19:43:56.422 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.416+0000 7f6573861700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f656c06b330 0x7f656c155780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:56.422 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.416+0000 7f6573861700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f656c0bf8a0 0x7f656c164050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:56.422 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.416+0000 7f6573861700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f656c0c1210 0x7f656c164590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:56.422 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.416+0000 7f6573861700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c151730 con 0x7f656c0c1210 2022-04-03T19:43:56.423 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.416+0000 7f6573861700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c152650 con 0x7f656c0bf8a0 2022-04-03T19:43:56.423 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.416+0000 7f6573861700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f656c1528f0 con 0x7f656c06b330 2022-04-03T19:43:56.423 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.416+0000 7f65715fd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f656c06b330 0x7f656c155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:56.423 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.416+0000 7f65715fd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f656c06b330 0x7f656c155780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:49666/0 (socket says 172.21.15.186:49666) 2022-04-03T19:43:56.423 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.416+0000 7f6570dfc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f656c0bf8a0 0x7f656c164050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:56.424 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.416+0000 7f65715fd700 1 -- 172.21.15.186:0/1057836028 learned_addr learned my addr 172.21.15.186:0/1057836028 (peer_addr_for_me v2:172.21.15.186:0/0) 2022-04-03T19:43:56.424 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.416+0000 7f6571dfe700 1 --2- 172.21.15.186:0/1057836028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f656c0c1210 0x7f656c164590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:56.424 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.416+0000 7f65715fd700 1 -- 172.21.15.186:0/1057836028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f656c0bf8a0 msgr2=0x7f656c164050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:56.424 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.416+0000 7f65715fd700 1 --2- 172.21.15.186:0/1057836028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f656c0bf8a0 0x7f656c164050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.424 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.416+0000 7f65715fd700 1 -- 172.21.15.186:0/1057836028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f656c0c1210 msgr2=0x7f656c164590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:56.425 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.416+0000 7f65715fd700 1 --2- 172.21.15.186:0/1057836028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f656c0c1210 0x7f656c164590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.425 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.416+0000 7f65715fd700 1 -- 172.21.15.186:0/1057836028 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656000b540 con 0x7f656c06b330 2022-04-03T19:43:56.425 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.416+0000 7f6570dfc700 1 --2- 172.21.15.186:0/1057836028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f656c0bf8a0 0x7f656c164050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:43:56.425 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.417+0000 7f65715fd700 1 --2- 172.21.15.186:0/1057836028 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f656c06b330 0x7f656c155780 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6560004000 tx=0x7f656001a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:56.425 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.417+0000 7f655d7fa700 1 -- 172.21.15.186:0/1057836028 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6560004300 con 0x7f656c06b330 2022-04-03T19:43:56.426 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.417+0000 7f655d7fa700 1 -- 172.21.15.186:0/1057836028 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6560004460 con 0x7f656c06b330 2022-04-03T19:43:56.426 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.417+0000 7f6573861700 1 -- 172.21.15.186:0/1057836028 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656c151b80 con 0x7f656c06b330 2022-04-03T19:43:56.426 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.417+0000 7f655d7fa700 1 -- 172.21.15.186:0/1057836028 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65600106c0 con 0x7f656c06b330 2022-04-03T19:43:56.426 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.417+0000 7f6573861700 1 -- 172.21.15.186:0/1057836028 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656c150a10 con 0x7f656c06b330 2022-04-03T19:43:56.426 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.417+0000 7f655d7fa700 1 -- 172.21.15.186:0/1057836028 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f6560010a30 con 0x7f656c06b330 2022-04-03T19:43:56.427 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.417+0000 7f655d7fa700 1 -- 172.21.15.186:0/1057836028 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f6560014070 con 0x7f656c06b330 2022-04-03T19:43:56.427 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.418+0000 7f6573861700 1 -- 172.21.15.186:0/1057836028 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656c04b890 con 0x7f656c06b330 2022-04-03T19:43:56.427 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.420+0000 7f655d7fa700 1 -- 172.21.15.186:0/1057836028 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70738 (secure 0 0 0) 0x7f6560029070 con 0x7f656c06b330 2022-04-03T19:43:56.500 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.500+0000 7f6573861700 1 -- 172.21.15.186:0/1057836028 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b0344ee8-deca-4e9b-934c-5866d90d0073", "id": 6} v 0) v1 -- 0x7f656c070ac0 con 0x7f656c06b330 2022-04-03T19:43:56.502 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:43:56.502+0000 7f493b032000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-04-03T19:43:56.506 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.507+0000 7f655d7fa700 1 -- 172.21.15.186:0/1057836028 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b0344ee8-deca-4e9b-934c-5866d90d0073", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6560018aa0 con 0x7f656c06b330 2022-04-03T19:43:56.508 INFO:teuthology.orchestra.run.smithi186.stdout:6 2022-04-03T19:43:56.509 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.508+0000 7f6573861700 1 -- 172.21.15.186:0/1057836028 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f656c06b330 msgr2=0x7f656c155780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:56.509 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.508+0000 7f6573861700 1 --2- 172.21.15.186:0/1057836028 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f656c06b330 0x7f656c155780 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6560004000 tx=0x7f656001a000 comp rx=0 tx=0).stop 2022-04-03T19:43:56.509 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.508+0000 7f6573861700 1 -- 172.21.15.186:0/1057836028 shutdown_connections 2022-04-03T19:43:56.510 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.508+0000 7f6573861700 1 --2- 172.21.15.186:0/1057836028 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f656c0c1210 0x7f656c164590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.510 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.508+0000 7f6573861700 1 --2- 172.21.15.186:0/1057836028 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f656c06b330 0x7f656c155780 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.510 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.508+0000 7f6573861700 1 --2- 172.21.15.186:0/1057836028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f656c0bf8a0 0x7f656c164050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.510 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.508+0000 7f6573861700 1 -- 172.21.15.186:0/1057836028 >> 172.21.15.186:0/1057836028 conn(0x7f656c0c8fb0 msgr2=0x7f656c0687a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:56.511 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.508+0000 7f6573861700 1 -- 172.21.15.186:0/1057836028 shutdown_connections 2022-04-03T19:43:56.511 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.508+0000 7f6573861700 1 -- 172.21.15.186:0/1057836028 wait complete. 2022-04-03T19:43:56.519 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 0eb0e5d3-3dad-4f49-9b50-3b24a3924bdb 7 2022-04-03T19:43:56.528 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:56.528+0000 7f207b38e000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-04-03T19:43:56.638 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.638+0000 7fd0e2316700 1 Processor -- start 2022-04-03T19:43:56.638 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.638+0000 7fd0e2316700 1 -- start start 2022-04-03T19:43:56.641 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.639+0000 7fd0e2316700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0dc0c5260 0x7fd0dc0c2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:56.641 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.639+0000 7fd0e2316700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0dc0c7d00 con 0x7fd0dc0c71e0 2022-04-03T19:43:56.642 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.639+0000 7fd0e2316700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0dc0c7740 con 0x7fd0dc0c89e0 2022-04-03T19:43:56.642 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.639+0000 7fd0e2316700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc0c7a20 con 0x7fd0dc0c5260 2022-04-03T19:43:56.642 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.639+0000 7fd0e08b3700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fd0dc0c89e0 0x7fd0dc15fb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:41968/0 (socket says 172.21.15.186:41968) 2022-04-03T19:43:56.642 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.639+0000 7fd0e08b3700 1 -- 172.21.15.186:0/2525891733 learned_addr learned my addr 172.21.15.186:0/2525891733 (peer_addr_for_me v1:172.21.15.186:0/0) 2022-04-03T19:43:56.643 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.639+0000 7fd0dbfff700 1 --2- 172.21.15.186:0/2525891733 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0dc0c5260 0x7fd0dc0c2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:56.643 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.639+0000 7fd0daffd700 1 -- 172.21.15.186:0/2525891733 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172784513 0 0) 0x7fd0dc0c7740 con 0x7fd0dc0c89e0 2022-04-03T19:43:56.643 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.639+0000 7fd0daffd700 1 -- 172.21.15.186:0/2525891733 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0c80033f0 con 0x7fd0dc0c89e0 2022-04-03T19:43:56.643 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.639+0000 7fd0dbfff700 1 -- 172.21.15.186:0/2525891733 >> v1:172.21.15.186:6789/0 conn(0x7fd0dc0c89e0 legacy=0x7fd0dc15fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:56.644 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.639+0000 7fd0dbfff700 1 -- 172.21.15.186:0/2525891733 >> v1:172.21.15.77:6789/0 conn(0x7fd0dc0c71e0 legacy=0x7fd0dc0c3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:56.644 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.639+0000 7fd0dbfff700 1 -- 172.21.15.186:0/2525891733 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc06bb50 con 0x7fd0dc0c5260 2022-04-03T19:43:56.644 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.639+0000 7fd0dbfff700 1 --2- 172.21.15.186:0/2525891733 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0dc0c5260 0x7fd0dc0c2b50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0009790 tx=0x7fd0d0009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce2606fd0c94e014 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:56.644 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.639+0000 7fd0daffd700 1 -- 172.21.15.186:0/2525891733 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0011630 con 0x7fd0dc0c5260 2022-04-03T19:43:56.644 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.639+0000 7fd0daffd700 1 -- 172.21.15.186:0/2525891733 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d0011790 con 0x7fd0dc0c5260 2022-04-03T19:43:56.645 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.639+0000 7fd0daffd700 1 -- 172.21.15.186:0/2525891733 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0011b30 con 0x7fd0dc0c5260 2022-04-03T19:43:56.645 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.640+0000 7fd0e2316700 1 -- 172.21.15.186:0/2525891733 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0dc0c5260 msgr2=0x7fd0dc0c2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:56.645 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.640+0000 7fd0e2316700 1 --2- 172.21.15.186:0/2525891733 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0dc0c5260 0x7fd0dc0c2b50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0009790 tx=0x7fd0d0009060 comp rx=0 tx=0).stop 2022-04-03T19:43:56.645 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.640+0000 7fd0e2316700 1 -- 172.21.15.186:0/2525891733 shutdown_connections 2022-04-03T19:43:56.646 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.640+0000 7fd0e2316700 1 --2- 172.21.15.186:0/2525891733 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0dc0c5260 0x7fd0dc0c2b50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.646 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.640+0000 7fd0e2316700 1 -- 172.21.15.186:0/2525891733 >> 172.21.15.186:0/2525891733 conn(0x7fd0dc0cc2c0 msgr2=0x7fd0dc0c82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:56.646 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.640+0000 7fd0e2316700 1 -- 172.21.15.186:0/2525891733 shutdown_connections 2022-04-03T19:43:56.646 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.640+0000 7fd0e2316700 1 -- 172.21.15.186:0/2525891733 wait complete. 2022-04-03T19:43:56.647 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.641+0000 7fd0e2316700 1 Processor -- start 2022-04-03T19:43:56.647 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.641+0000 7fd0e2316700 1 -- start start 2022-04-03T19:43:56.649 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.642+0000 7fd0e2316700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0dc0c5260 0x7fd0dc0c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:56.650 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.642+0000 7fd0e2316700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc0c71e0 0x7fd0dc0c3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:56.650 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.642+0000 7fd0e2316700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0dc0c89e0 0x7fd0dc168500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:56.650 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.642+0000 7fd0e2316700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc165eb0 con 0x7fd0dc0c71e0 2022-04-03T19:43:56.651 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.642+0000 7fd0e2316700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc0cafe0 con 0x7fd0dc0c89e0 2022-04-03T19:43:56.651 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.642+0000 7fd0e2316700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc0c24d0 con 0x7fd0dc0c5260 2022-04-03T19:43:56.651 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.642+0000 7fd0dbfff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0dc0c5260 0x7fd0dc0c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:56.651 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.642+0000 7fd0e08b3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0dc0c89e0 0x7fd0dc168500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:56.652 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.642+0000 7fd0e08b3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0dc0c89e0 0x7fd0dc168500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40168/0 (socket says 172.21.15.186:40168) 2022-04-03T19:43:56.652 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.642+0000 7fd0dbfff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0dc0c5260 0x7fd0dc0c1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:49678/0 (socket says 172.21.15.186:49678) 2022-04-03T19:43:56.652 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.642+0000 7fd0e08b3700 1 -- 172.21.15.186:0/1023103906 learned_addr learned my addr 172.21.15.186:0/1023103906 (peer_addr_for_me v2:172.21.15.186:0/0) 2022-04-03T19:43:56.652 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.642+0000 7fd0db7fe700 1 --2- 172.21.15.186:0/1023103906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc0c71e0 0x7fd0dc0c3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:56.653 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.642+0000 7fd0e08b3700 1 -- 172.21.15.186:0/1023103906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0dc0c5260 msgr2=0x7fd0dc0c1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:56.653 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.642+0000 7fd0e08b3700 1 --2- 172.21.15.186:0/1023103906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0dc0c5260 0x7fd0dc0c1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.653 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.642+0000 7fd0e08b3700 1 -- 172.21.15.186:0/1023103906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc0c71e0 msgr2=0x7fd0dc0c3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:56.653 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.642+0000 7fd0e08b3700 1 --2- 172.21.15.186:0/1023103906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc0c71e0 0x7fd0dc0c3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.653 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.642+0000 7fd0e08b3700 1 -- 172.21.15.186:0/1023103906 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d00092a0 con 0x7fd0dc0c89e0 2022-04-03T19:43:56.654 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.642+0000 7fd0db7fe700 1 --2- 172.21.15.186:0/1023103906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc0c71e0 0x7fd0dc0c3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:43:56.654 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.643+0000 7fd0e08b3700 1 --2- 172.21.15.186:0/1023103906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0dc0c89e0 0x7fd0dc168500 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4001460 tx=0x7fd0d40015b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:56.654 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.643+0000 7fd0c3fff700 1 -- 172.21.15.186:0/1023103906 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d4005d30 con 0x7fd0dc0c89e0 2022-04-03T19:43:56.654 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.643+0000 7fd0c3fff700 1 -- 172.21.15.186:0/1023103906 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d4005e90 con 0x7fd0dc0c89e0 2022-04-03T19:43:56.655 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.643+0000 7fd0e2316700 1 -- 172.21.15.186:0/1023103906 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0dc0cad00 con 0x7fd0dc0c89e0 2022-04-03T19:43:56.655 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.643+0000 7fd0c3fff700 1 -- 172.21.15.186:0/1023103906 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d4007300 con 0x7fd0dc0c89e0 2022-04-03T19:43:56.655 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.643+0000 7fd0e2316700 1 -- 172.21.15.186:0/1023103906 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0dc1653d0 con 0x7fd0dc0c89e0 2022-04-03T19:43:56.655 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.643+0000 7fd0c3fff700 1 -- 172.21.15.186:0/1023103906 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7fd0d4010070 con 0x7fd0dc0c89e0 2022-04-03T19:43:56.656 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.643+0000 7fd0c3fff700 1 -- 172.21.15.186:0/1023103906 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7fd0d401bd40 con 0x7fd0dc0c89e0 2022-04-03T19:43:56.656 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.644+0000 7fd0e2316700 1 -- 172.21.15.186:0/1023103906 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0dc0c4e20 con 0x7fd0dc0c89e0 2022-04-03T19:43:56.656 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.646+0000 7fd0c3fff700 1 -- 172.21.15.186:0/1023103906 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70738 (secure 0 0 0) 0x7fd0d4007760 con 0x7fd0dc0c89e0 2022-04-03T19:43:56.697 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:56.697+0000 7f207b38e000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-04-03T19:43:56.728 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.728+0000 7fd0e2316700 1 -- 172.21.15.186:0/1023103906 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0eb0e5d3-3dad-4f49-9b50-3b24a3924bdb", "id": 7} v 0) v1 -- 0x7fd0dc088f60 con 0x7fd0dc0c89e0 2022-04-03T19:43:56.736 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.736+0000 7fd0c3fff700 1 -- 172.21.15.186:0/1023103906 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0eb0e5d3-3dad-4f49-9b50-3b24a3924bdb", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd0d4007960 con 0x7fd0dc0c89e0 2022-04-03T19:43:56.736 INFO:teuthology.orchestra.run.smithi186.stdout:7 2022-04-03T19:43:56.738 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.737+0000 7fd0e2316700 1 -- 172.21.15.186:0/1023103906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0dc0c89e0 msgr2=0x7fd0dc168500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:56.738 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.737+0000 7fd0e2316700 1 --2- 172.21.15.186:0/1023103906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0dc0c89e0 0x7fd0dc168500 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4001460 tx=0x7fd0d40015b0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.739 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.737+0000 7fd0e2316700 1 -- 172.21.15.186:0/1023103906 shutdown_connections 2022-04-03T19:43:56.739 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.737+0000 7fd0e2316700 1 --2- 172.21.15.186:0/1023103906 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0dc0c89e0 0x7fd0dc168500 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.739 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.737+0000 7fd0e2316700 1 --2- 172.21.15.186:0/1023103906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0dc0c5260 0x7fd0dc0c1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.739 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.737+0000 7fd0e2316700 1 --2- 172.21.15.186:0/1023103906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0dc0c71e0 0x7fd0dc0c3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:56.739 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.737+0000 7fd0e2316700 1 -- 172.21.15.186:0/1023103906 >> 172.21.15.186:0/1023103906 conn(0x7fd0dc0cc2c0 msgr2=0x7fd0dc156fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:56.740 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.738+0000 7fd0e2316700 1 -- 172.21.15.186:0/1023103906 shutdown_connections 2022-04-03T19:43:56.740 INFO:teuthology.orchestra.run.smithi186.stderr:2022-04-03T19:43:56.738+0000 7fd0e2316700 1 -- 172.21.15.186:0/1023103906 wait complete. 2022-04-03T19:43:56.749 INFO:tasks.ceph.osd.0:Restarting daemon 2022-04-03T19:43:56.749 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2022-04-03T19:43:56.754 INFO:tasks.ceph.osd.0:Started 2022-04-03T19:43:56.754 INFO:tasks.ceph.osd.1:Restarting daemon 2022-04-03T19:43:56.755 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2022-04-03T19:43:56.759 INFO:tasks.ceph.osd.1:Started 2022-04-03T19:43:56.759 INFO:tasks.ceph.osd.2:Restarting daemon 2022-04-03T19:43:56.760 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2022-04-03T19:43:56.764 INFO:tasks.ceph.osd.2:Started 2022-04-03T19:43:56.764 INFO:tasks.ceph.osd.3:Restarting daemon 2022-04-03T19:43:56.765 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2022-04-03T19:43:56.771 INFO:tasks.ceph.osd.3:Started 2022-04-03T19:43:56.771 INFO:tasks.ceph.osd.4:Restarting daemon 2022-04-03T19:43:56.771 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2022-04-03T19:43:56.791 INFO:tasks.ceph.osd.4:Started 2022-04-03T19:43:56.791 INFO:tasks.ceph.osd.5:Restarting daemon 2022-04-03T19:43:56.791 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2022-04-03T19:43:56.794 INFO:tasks.ceph.osd.5:Started 2022-04-03T19:43:56.794 INFO:tasks.ceph.osd.6:Restarting daemon 2022-04-03T19:43:56.794 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2022-04-03T19:43:56.796 INFO:tasks.ceph.osd.6:Started 2022-04-03T19:43:56.797 INFO:tasks.ceph.osd.7:Restarting daemon 2022-04-03T19:43:56.797 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2022-04-03T19:43:56.800 INFO:tasks.ceph.osd.7:Started 2022-04-03T19:43:56.800 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-04-03T19:43:57.063 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.063+0000 7f2caa622700 1 Processor -- start 2022-04-03T19:43:57.064 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.063+0000 7f2caa622700 1 -- start start 2022-04-03T19:43:57.065 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.064+0000 7f2caa622700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2ca406d3f0 0x7f2ca406d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:57.066 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.064+0000 7f2caa622700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ca40c5930 0x7f2ca4155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:57.066 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.064+0000 7f2caa622700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca40c2f80 con 0x7f2ca4156120 2022-04-03T19:43:57.066 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.064+0000 7f2caa622700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ca40c3260 con 0x7f2ca406d3f0 2022-04-03T19:43:57.066 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.064+0000 7f2caa622700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca40c3540 con 0x7f2ca40c5930 2022-04-03T19:43:57.067 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.065+0000 7f2ca8e1f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ca40c5930 0x7f2ca4155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:57.067 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.065+0000 7f2ca9620700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2ca406d3f0 0x7f2ca406d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:57.067 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.065+0000 7f2ca9e21700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2ca4156120 0x7f2ca406c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38258/0 (socket says 172.21.15.77:38258) 2022-04-03T19:43:57.067 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.065+0000 7f2ca9e21700 1 -- 172.21.15.77:0/3318552463 learned_addr learned my addr 172.21.15.77:0/3318552463 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:43:57.068 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.065+0000 7f2ca8e1f700 1 -- 172.21.15.77:0/3318552463 >> v2:172.21.15.186:3300/0 conn(0x7f2ca406d3f0 msgr2=0x7f2ca406d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:43:57.068 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.065+0000 7f2ca8e1f700 1 --2- 172.21.15.77:0/3318552463 >> v2:172.21.15.186:3300/0 conn(0x7f2ca406d3f0 0x7f2ca406d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:57.068 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.065+0000 7f2ca8e1f700 1 -- 172.21.15.77:0/3318552463 >> v1:172.21.15.77:6789/0 conn(0x7f2ca4156120 legacy=0x7f2ca406c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:57.068 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.065+0000 7f2ca8e1f700 1 -- 172.21.15.77:0/3318552463 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca40c3820 con 0x7f2ca40c5930 2022-04-03T19:43:57.072 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.065+0000 7f2ca8e1f700 1 --2- 172.21.15.77:0/3318552463 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ca40c5930 0x7f2ca4155be0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2c94009790 tx=0x7f2c94009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3805bc1da0ec7770 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:57.072 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.066+0000 7f2c93fff700 1 -- 172.21.15.77:0/3318552463 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c94011630 con 0x7f2ca40c5930 2022-04-03T19:43:57.072 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.066+0000 7f2c93fff700 1 -- 172.21.15.77:0/3318552463 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c94011790 con 0x7f2ca40c5930 2022-04-03T19:43:57.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.066+0000 7f2c93fff700 1 -- 172.21.15.77:0/3318552463 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c94011b50 con 0x7f2ca40c5930 2022-04-03T19:43:57.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.066+0000 7f2caa622700 1 -- 172.21.15.77:0/3318552463 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ca40c5930 msgr2=0x7f2ca4155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:57.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.066+0000 7f2caa622700 1 --2- 172.21.15.77:0/3318552463 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ca40c5930 0x7f2ca4155be0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2c94009790 tx=0x7f2c94009060 comp rx=0 tx=0).stop 2022-04-03T19:43:57.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.066+0000 7f2caa622700 1 -- 172.21.15.77:0/3318552463 shutdown_connections 2022-04-03T19:43:57.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.066+0000 7f2caa622700 1 --2- 172.21.15.77:0/3318552463 >> v2:172.21.15.186:3300/0 conn(0x7f2ca406d3f0 0x7f2ca406d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:57.074 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.066+0000 7f2caa622700 1 --2- 172.21.15.77:0/3318552463 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ca40c5930 0x7f2ca4155be0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:57.074 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.066+0000 7f2caa622700 1 -- 172.21.15.77:0/3318552463 >> 172.21.15.77:0/3318552463 conn(0x7f2ca40cb720 msgr2=0x7f2ca40cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:57.074 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.066+0000 7f2caa622700 1 -- 172.21.15.77:0/3318552463 shutdown_connections 2022-04-03T19:43:57.074 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.066+0000 7f2caa622700 1 -- 172.21.15.77:0/3318552463 wait complete. 2022-04-03T19:43:57.075 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.068+0000 7f2caa622700 1 Processor -- start 2022-04-03T19:43:57.075 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.068+0000 7f2caa622700 1 -- start start 2022-04-03T19:43:57.075 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.069+0000 7f2caa622700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ca406d3f0 0x7f2ca414de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:57.075 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.069+0000 7f2caa622700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ca40c5930 0x7f2ca4151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:57.075 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.069+0000 7f2caa622700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ca4156120 0x7f2ca4152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:57.076 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.069+0000 7f2caa622700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca4151250 con 0x7f2ca4156120 2022-04-03T19:43:57.076 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.069+0000 7f2caa622700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca4169be0 con 0x7f2ca406d3f0 2022-04-03T19:43:57.076 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.069+0000 7f2caa622700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca41514c0 con 0x7f2ca40c5930 2022-04-03T19:43:57.076 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.069+0000 7f2ca9620700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ca406d3f0 0x7f2ca414de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:57.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.069+0000 7f2ca9620700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ca406d3f0 0x7f2ca414de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37590/0 (socket says 172.21.15.77:37590) 2022-04-03T19:43:57.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.069+0000 7f2ca9620700 1 -- 172.21.15.77:0/2011227804 learned_addr learned my addr 172.21.15.77:0/2011227804 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:43:57.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.069+0000 7f2ca8e1f700 1 --2- 172.21.15.77:0/2011227804 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ca40c5930 0x7f2ca4151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:57.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.069+0000 7f2ca9e21700 1 --2- 172.21.15.77:0/2011227804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ca4156120 0x7f2ca4152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.070+0000 7f2ca9620700 1 -- 172.21.15.77:0/2011227804 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ca40c5930 msgr2=0x7f2ca4151cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.070+0000 7f2ca9620700 1 --2- 172.21.15.77:0/2011227804 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ca40c5930 0x7f2ca4151cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.070+0000 7f2ca9620700 1 -- 172.21.15.77:0/2011227804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ca4156120 msgr2=0x7f2ca4152200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.070+0000 7f2ca9620700 1 --2- 172.21.15.77:0/2011227804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ca4156120 0x7f2ca4152200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:57.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.070+0000 7f2ca9620700 1 -- 172.21.15.77:0/2011227804 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c940092a0 con 0x7f2ca406d3f0 2022-04-03T19:43:57.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.070+0000 7f2ca8e1f700 1 --2- 172.21.15.77:0/2011227804 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ca40c5930 0x7f2ca4151cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:43:57.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.070+0000 7f2ca9620700 1 --2- 172.21.15.77:0/2011227804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ca406d3f0 0x7f2ca414de80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2ca000e930 tx=0x7f2ca0009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:57.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.070+0000 7f2c917fa700 1 -- 172.21.15.77:0/2011227804 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca0013680 con 0x7f2ca406d3f0 2022-04-03T19:43:57.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.070+0000 7f2c917fa700 1 -- 172.21.15.77:0/2011227804 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca00137e0 con 0x7f2ca406d3f0 2022-04-03T19:43:57.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.070+0000 7f2caa622700 1 -- 172.21.15.77:0/2011227804 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca41517a0 con 0x7f2ca406d3f0 2022-04-03T19:43:57.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.070+0000 7f2c917fa700 1 -- 172.21.15.77:0/2011227804 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca0013b60 con 0x7f2ca406d3f0 2022-04-03T19:43:57.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.070+0000 7f2caa622700 1 -- 172.21.15.77:0/2011227804 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca4169e80 con 0x7f2ca406d3f0 2022-04-03T19:43:57.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.071+0000 7f2c917fa700 1 -- 172.21.15.77:0/2011227804 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f2ca0019030 con 0x7f2ca406d3f0 2022-04-03T19:43:57.081 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.071+0000 7f2c917fa700 1 -- 172.21.15.77:0/2011227804 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f2ca000c3e0 con 0x7f2ca406d3f0 2022-04-03T19:43:57.081 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.071+0000 7f2caa622700 1 -- 172.21.15.77:0/2011227804 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca4151990 con 0x7f2ca406d3f0 2022-04-03T19:43:57.081 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.074+0000 7f2c917fa700 1 -- 172.21.15.77:0/2011227804 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70738 (secure 0 0 0) 0x7f2ca0021060 con 0x7f2ca406d3f0 2022-04-03T19:43:57.148 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.148+0000 7f2caa622700 1 -- 172.21.15.77:0/2011227804 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ca404b050 con 0x7f2ca406d3f0 2022-04-03T19:43:57.149 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.148+0000 7f2c917fa700 1 -- 172.21.15.77:0/2011227804 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+6992 (secure 0 0 0) 0x7f2ca0017070 con 0x7f2ca406d3f0 2022-04-03T19:43:57.149 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:43:57.149 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":10,"fsid":"89c3a3f4-599d-4c4a-8f5a-7180eafb7292","created":"2022-04-03T19:43:54.101559+0000","modified":"2022-04-03T19:43:56.729760+0000","last_up_change":"0.000000","last_in_change":"2022-04-03T19:43:56.729760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"f35f5195-3729-4d78-85d1-0539f259283f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ab1c89dc-57ce-4cc4-b8c7-9b0c19939601","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"351e218b-7749-4ee9-b44a-0a5b52fb9be1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ba6ec8a-7cbf-4c12-8e21-c3097a263ae8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3faf4c09-846c-4133-856b-7f53c535a67d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ffe41d00-d0aa-42f4-82c1-5cced13c1bc3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b0344ee8-deca-4e9b-934c-5866d90d0073","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0eb0e5d3-3dad-4f49-9b50-3b24a3924bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-03T19:43:57.150 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.149+0000 7f2caa622700 1 -- 172.21.15.77:0/2011227804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ca406d3f0 msgr2=0x7f2ca414de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:57.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.149+0000 7f2caa622700 1 --2- 172.21.15.77:0/2011227804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ca406d3f0 0x7f2ca414de80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2ca000e930 tx=0x7f2ca0009060 comp rx=0 tx=0).stop 2022-04-03T19:43:57.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.149+0000 7f2caa622700 1 -- 172.21.15.77:0/2011227804 shutdown_connections 2022-04-03T19:43:57.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.149+0000 7f2caa622700 1 --2- 172.21.15.77:0/2011227804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ca4156120 0x7f2ca4152200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:57.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.149+0000 7f2caa622700 1 --2- 172.21.15.77:0/2011227804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ca406d3f0 0x7f2ca414de80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:57.152 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.149+0000 7f2caa622700 1 --2- 172.21.15.77:0/2011227804 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ca40c5930 0x7f2ca4151cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:57.152 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.149+0000 7f2caa622700 1 -- 172.21.15.77:0/2011227804 >> 172.21.15.77:0/2011227804 conn(0x7f2ca40cb720 msgr2=0x7f2ca40c5dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:57.152 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.150+0000 7f2caa622700 1 -- 172.21.15.77:0/2011227804 shutdown_connections 2022-04-03T19:43:57.152 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:43:57.150+0000 7f2caa622700 1 -- 172.21.15.77:0/2011227804 wait complete. 2022-04-03T19:43:57.173 INFO:tasks.ceph.ceph_manager.ceph:[] 2022-04-03T19:43:57.174 INFO:tasks.ceph:Waiting for OSDs to come up 2022-04-03T19:43:57.195 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:57.195+0000 7f207b38e000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-04-03T19:43:57.230 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:43:57.230+0000 7f493b032000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-04-03T19:43:57.390 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:57.391+0000 7f207b38e000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-04-03T19:43:57.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T19:43:57.404+0000 7fd3c0b533c0 -1 Falling back to public interface 2022-04-03T19:43:57.405 INFO:tasks.ceph.osd.5.smithi186.stderr:2022-04-03T19:43:57.405+0000 7f772ef543c0 -1 Falling back to public interface 2022-04-03T19:43:57.413 INFO:tasks.ceph.osd.7.smithi186.stderr:2022-04-03T19:43:57.413+0000 7fc1cb69f3c0 -1 Falling back to public interface 2022-04-03T19:43:57.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T19:43:57.414+0000 7f88dfd093c0 -1 Falling back to public interface 2022-04-03T19:43:57.440 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:43:57.440+0000 7f493b032000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-04-03T19:43:57.464 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:57.464+0000 7f207b38e000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-04-03T19:43:57.479 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T19:43:57.479+0000 7f8814b433c0 -1 Falling back to public interface 2022-04-03T19:43:57.480 INFO:tasks.ceph.osd.1.smithi077.stderr:2022-04-03T19:43:57.480+0000 7f421fb763c0 -1 Falling back to public interface 2022-04-03T19:43:57.481 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T19:43:57.481+0000 7f880ef6d700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6805/0: (98) Address already in use 2022-04-03T19:43:57.481 INFO:tasks.ceph.osd.3.smithi077.stderr:2022-04-03T19:43:57.481+0000 7fc4cf0773c0 -1 Falling back to public interface 2022-04-03T19:43:57.490 INFO:tasks.ceph.osd.0.smithi077.stderr:2022-04-03T19:43:57.490+0000 7f28749013c0 -1 Falling back to public interface 2022-04-03T19:43:57.557 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:57.557+0000 7f207b38e000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-04-03T19:43:57.634 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:57.634+0000 7f207b38e000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-04-03T19:43:57.778 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:57.778+0000 7f207b38e000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-04-03T19:43:57.871 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:57.871+0000 7f207b38e000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-04-03T19:43:57.979 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-04-03T19:43:58.088 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:58.088+0000 7f207b38e000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-04-03T19:43:58.115 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:43:58.115+0000 7f493b032000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-04-03T19:43:58.158 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.157+0000 7effde624700 1 Processor -- start 2022-04-03T19:43:58.158 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.157+0000 7effde624700 1 -- start start 2022-04-03T19:43:58.160 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.158+0000 7effde624700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7effd80c5280 0x7effd8069630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:58.160 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.158+0000 7effde624700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effd81586f0 0x7effd80c7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:58.160 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.158+0000 7effde624700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd8155a00 con 0x7effd80c5650 2022-04-03T19:43:58.161 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.158+0000 7effde624700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effd81595d0 con 0x7effd80c5280 2022-04-03T19:43:58.161 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.158+0000 7effde624700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd81551a0 con 0x7effd81586f0 2022-04-03T19:43:58.161 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.159+0000 7effdd622700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7effd80c5280 0x7effd8069630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:58.161 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.159+0000 7effdde23700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effd81586f0 0x7effd80c7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:58.162 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.159+0000 7effdce21700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7effd80c5650 0x7effd80c6a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44070/0 (socket says 172.21.15.77:44070) 2022-04-03T19:43:58.162 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.159+0000 7effdce21700 1 -- 172.21.15.77:0/2171545034 learned_addr learned my addr 172.21.15.77:0/2171545034 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:43:58.162 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.159+0000 7effdde23700 1 -- 172.21.15.77:0/2171545034 >> v2:172.21.15.186:3300/0 conn(0x7effd80c5280 msgr2=0x7effd8069630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:43:58.162 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.159+0000 7effdde23700 1 --2- 172.21.15.77:0/2171545034 >> v2:172.21.15.186:3300/0 conn(0x7effd80c5280 0x7effd8069630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:58.163 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.159+0000 7effdde23700 1 -- 172.21.15.77:0/2171545034 >> v1:172.21.15.186:6789/0 conn(0x7effd80c5650 legacy=0x7effd80c6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:58.163 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.159+0000 7effdde23700 1 -- 172.21.15.77:0/2171545034 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd8159010 con 0x7effd81586f0 2022-04-03T19:43:58.163 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.159+0000 7effc7fff700 1 -- 172.21.15.77:0/2171545034 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550656435 0 0) 0x7effd8155a00 con 0x7effd80c5650 2022-04-03T19:43:58.163 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.159+0000 7effdde23700 1 --2- 172.21.15.77:0/2171545034 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effd81586f0 0x7effd80c7640 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7effd0009e60 tx=0x7effd0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c7efd89d87398d5 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:58.163 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.160+0000 7effc7fff700 1 -- 172.21.15.77:0/2171545034 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd000e070 con 0x7effd81586f0 2022-04-03T19:43:58.164 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.160+0000 7effc7fff700 1 -- 172.21.15.77:0/2171545034 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd000bbf0 con 0x7effd81586f0 2022-04-03T19:43:58.164 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.160+0000 7effc7fff700 1 -- 172.21.15.77:0/2171545034 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd0012430 con 0x7effd81586f0 2022-04-03T19:43:58.167 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.160+0000 7effde624700 1 -- 172.21.15.77:0/2171545034 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effd81586f0 msgr2=0x7effd80c7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:58.168 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.160+0000 7effde624700 1 --2- 172.21.15.77:0/2171545034 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effd81586f0 0x7effd80c7640 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7effd0009e60 tx=0x7effd0009780 comp rx=0 tx=0).stop 2022-04-03T19:43:58.168 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.160+0000 7effde624700 1 -- 172.21.15.77:0/2171545034 shutdown_connections 2022-04-03T19:43:58.168 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.160+0000 7effde624700 1 --2- 172.21.15.77:0/2171545034 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effd81586f0 0x7effd80c7640 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:58.169 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.160+0000 7effde624700 1 --2- 172.21.15.77:0/2171545034 >> v2:172.21.15.186:3300/0 conn(0x7effd80c5280 0x7effd8069630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:58.169 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.160+0000 7effde624700 1 -- 172.21.15.77:0/2171545034 >> 172.21.15.77:0/2171545034 conn(0x7effd80cb700 msgr2=0x7effd80cbb00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:58.169 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.160+0000 7effde624700 1 -- 172.21.15.77:0/2171545034 shutdown_connections 2022-04-03T19:43:58.169 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.160+0000 7effde624700 1 -- 172.21.15.77:0/2171545034 wait complete. 2022-04-03T19:43:58.170 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.162+0000 7effde624700 1 Processor -- start 2022-04-03T19:43:58.170 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.162+0000 7effde624700 1 -- start start 2022-04-03T19:43:58.170 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.163+0000 7effde624700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effd80c5280 0x7effd80c3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:58.170 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.163+0000 7effde624700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effd80c5650 0x7effd80bf640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:58.171 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.163+0000 7effde624700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effd81586f0 0x7effd8151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:43:58.171 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.163+0000 7effde624700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd8158400 con 0x7effd81586f0 2022-04-03T19:43:58.171 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.163+0000 7effde624700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd80c2df0 con 0x7effd80c5280 2022-04-03T19:43:58.171 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.163+0000 7effde624700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd8165bf0 con 0x7effd80c5650 2022-04-03T19:43:58.172 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.163+0000 7effdd622700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effd80c5280 0x7effd80c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:58.172 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.163+0000 7effdce21700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effd80c5650 0x7effd80bf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:58.172 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.163+0000 7effdd622700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effd80c5280 0x7effd80c3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37594/0 (socket says 172.21.15.77:37594) 2022-04-03T19:43:58.172 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.163+0000 7effdd622700 1 -- 172.21.15.77:0/3265337286 learned_addr learned my addr 172.21.15.77:0/3265337286 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:43:58.172 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.163+0000 7effdde23700 1 --2- 172.21.15.77:0/3265337286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effd81586f0 0x7effd8151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:43:58.173 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.164+0000 7effdd622700 1 -- 172.21.15.77:0/3265337286 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effd80c5650 msgr2=0x7effd80bf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:58.173 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.164+0000 7effdd622700 1 --2- 172.21.15.77:0/3265337286 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effd80c5650 0x7effd80bf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:58.173 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.164+0000 7effdd622700 1 -- 172.21.15.77:0/3265337286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effd81586f0 msgr2=0x7effd8151b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:58.173 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.164+0000 7effdd622700 1 --2- 172.21.15.77:0/3265337286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effd81586f0 0x7effd8151b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:58.174 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.164+0000 7effdd622700 1 -- 172.21.15.77:0/3265337286 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd0009a20 con 0x7effd80c5280 2022-04-03T19:43:58.174 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.164+0000 7effdd622700 1 --2- 172.21.15.77:0/3265337286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effd80c5280 0x7effd80c3280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7effd400e900 tx=0x7effd4009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:43:58.174 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.164+0000 7effc57fa700 1 -- 172.21.15.77:0/3265337286 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4013680 con 0x7effd80c5280 2022-04-03T19:43:58.174 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.164+0000 7effc57fa700 1 -- 172.21.15.77:0/3265337286 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd40137e0 con 0x7effd80c5280 2022-04-03T19:43:58.175 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.164+0000 7effde624700 1 -- 172.21.15.77:0/3265337286 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd80c21e0 con 0x7effd80c5280 2022-04-03T19:43:58.175 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.164+0000 7effc57fa700 1 -- 172.21.15.77:0/3265337286 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4013b60 con 0x7effd80c5280 2022-04-03T19:43:58.175 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.164+0000 7effde624700 1 -- 172.21.15.77:0/3265337286 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd80c27a0 con 0x7effd80c5280 2022-04-03T19:43:58.175 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.165+0000 7effc57fa700 1 -- 172.21.15.77:0/3265337286 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7effd4019030 con 0x7effd80c5280 2022-04-03T19:43:58.176 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.165+0000 7effc57fa700 1 -- 172.21.15.77:0/3265337286 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7effd4006750 con 0x7effd80c5280 2022-04-03T19:43:58.176 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.165+0000 7effde624700 1 -- 172.21.15.77:0/3265337286 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd80c30d0 con 0x7effd80c5280 2022-04-03T19:43:58.176 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.169+0000 7effc57fa700 1 -- 172.21.15.77:0/3265337286 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70738 (secure 0 0 0) 0x7effd4021060 con 0x7effd80c5280 2022-04-03T19:43:58.242 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.242+0000 7effde624700 1 -- 172.21.15.77:0/3265337286 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effd8068920 con 0x7effd80c5280 2022-04-03T19:43:58.243 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.243+0000 7effc57fa700 1 -- 172.21.15.77:0/3265337286 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+6992 (secure 0 0 0) 0x7effd4029020 con 0x7effd80c5280 2022-04-03T19:43:58.243 INFO:teuthology.misc.health.smithi077.stdout: 2022-04-03T19:43:58.244 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":10,"fsid":"89c3a3f4-599d-4c4a-8f5a-7180eafb7292","created":"2022-04-03T19:43:54.101559+0000","modified":"2022-04-03T19:43:56.729760+0000","last_up_change":"0.000000","last_in_change":"2022-04-03T19:43:56.729760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"f35f5195-3729-4d78-85d1-0539f259283f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ab1c89dc-57ce-4cc4-b8c7-9b0c19939601","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"351e218b-7749-4ee9-b44a-0a5b52fb9be1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ba6ec8a-7cbf-4c12-8e21-c3097a263ae8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3faf4c09-846c-4133-856b-7f53c535a67d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ffe41d00-d0aa-42f4-82c1-5cced13c1bc3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b0344ee8-deca-4e9b-934c-5866d90d0073","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0eb0e5d3-3dad-4f49-9b50-3b24a3924bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-03T19:43:58.245 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.244+0000 7effde624700 1 -- 172.21.15.77:0/3265337286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effd80c5280 msgr2=0x7effd80c3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:43:58.245 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.244+0000 7effde624700 1 --2- 172.21.15.77:0/3265337286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effd80c5280 0x7effd80c3280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7effd400e900 tx=0x7effd4009060 comp rx=0 tx=0).stop 2022-04-03T19:43:58.245 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.244+0000 7effde624700 1 -- 172.21.15.77:0/3265337286 shutdown_connections 2022-04-03T19:43:58.246 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.244+0000 7effde624700 1 --2- 172.21.15.77:0/3265337286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7effd81586f0 0x7effd8151b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:58.246 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.244+0000 7effde624700 1 --2- 172.21.15.77:0/3265337286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effd80c5280 0x7effd80c3280 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:58.246 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.244+0000 7effde624700 1 --2- 172.21.15.77:0/3265337286 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effd80c5650 0x7effd80bf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:43:58.246 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.244+0000 7effde624700 1 -- 172.21.15.77:0/3265337286 >> 172.21.15.77:0/3265337286 conn(0x7effd80cb700 msgr2=0x7effd8154060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:43:58.247 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.244+0000 7effde624700 1 -- 172.21.15.77:0/3265337286 shutdown_connections 2022-04-03T19:43:58.247 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:43:58.244+0000 7effde624700 1 -- 172.21.15.77:0/3265337286 wait complete. 2022-04-03T19:43:58.264 DEBUG:teuthology.misc:0 of 8 OSDs are up 2022-04-03T19:43:58.382 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:43:58.382+0000 7f493b032000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-04-03T19:43:58.496 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:43:58.496+0000 7f493b032000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-04-03T19:43:58.632 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:43:58.631+0000 7f493b032000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-04-03T19:43:58.760 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:43:58.760+0000 7f493b032000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-04-03T19:43:58.999 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:43:58.999+0000 7f493b032000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-04-03T19:43:59.115 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:59.115+0000 7f207b38e000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-04-03T19:43:59.151 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:43:59.150+0000 7f493b032000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-04-03T19:43:59.250 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:59.250+0000 7f207b38e000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-04-03T19:43:59.382 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:59.382+0000 7f207b38e000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-04-03T19:43:59.434 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:43:59.434+0000 7f493b032000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-04-03T19:43:59.460 INFO:tasks.ceph.mgr.x.smithi186.stderr:2022-04-03T19:43:59.461+0000 7f207b38e000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-04-03T19:44:00.892 INFO:tasks.ceph.osd.5.smithi186.stderr:2022-04-03T19:44:00.892+0000 7f772ef543c0 -1 osd.5 0 log_to_monitors true 2022-04-03T19:44:00.929 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T19:44:00.929+0000 7fd3c0b533c0 -1 osd.6 0 log_to_monitors true 2022-04-03T19:44:00.936 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T19:44:00.936+0000 7f88dfd093c0 -1 osd.4 0 log_to_monitors true 2022-04-03T19:44:00.948 INFO:tasks.ceph.osd.7.smithi186.stderr:2022-04-03T19:44:00.948+0000 7fc1cb69f3c0 -1 osd.7 0 log_to_monitors true 2022-04-03T19:44:01.094 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:44:01.094+0000 7f493b032000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-04-03T19:44:01.157 INFO:tasks.ceph.osd.0.smithi077.stderr:2022-04-03T19:44:01.157+0000 7f28749013c0 -1 osd.0 0 log_to_monitors true 2022-04-03T19:44:01.344 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:44:01.343+0000 7f493b032000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-04-03T19:44:01.379 INFO:tasks.ceph.osd.3.smithi077.stderr:2022-04-03T19:44:01.379+0000 7fc4cf0773c0 -1 osd.3 0 log_to_monitors true 2022-04-03T19:44:01.401 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T19:44:01.401+0000 7f8814b433c0 -1 osd.2 0 log_to_monitors true 2022-04-03T19:44:01.445 INFO:tasks.ceph.osd.1.smithi077.stderr:2022-04-03T19:44:01.445+0000 7f421fb763c0 -1 osd.1 0 log_to_monitors true 2022-04-03T19:44:01.568 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:44:01.568+0000 7f493b032000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-04-03T19:44:01.704 INFO:tasks.ceph.mgr.y.smithi077.stderr:2022-04-03T19:44:01.704+0000 7f493b032000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-04-03T19:44:04.575 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T19:44:04.575+0000 7fd3b7776700 -1 osd.6 0 waiting for initial osdmap 2022-04-03T19:44:04.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T19:44:04.579+0000 7fd3b0108700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-03T19:44:04.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T19:44:04.587+0000 7f88d692c700 -1 osd.4 0 waiting for initial osdmap 2022-04-03T19:44:04.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T19:44:04.591+0000 7f88cf2be700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-03T19:44:04.601 INFO:tasks.ceph.osd.7.smithi186.stderr:2022-04-03T19:44:04.601+0000 7fc1c22c2700 -1 osd.7 0 waiting for initial osdmap 2022-04-03T19:44:04.605 INFO:tasks.ceph.osd.7.smithi186.stderr:2022-04-03T19:44:04.605+0000 7fc1bac54700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-03T19:44:04.610 INFO:tasks.ceph.osd.5.smithi186.stderr:2022-04-03T19:44:04.611+0000 7f7725b77700 -1 osd.5 0 waiting for initial osdmap 2022-04-03T19:44:04.614 INFO:tasks.ceph.osd.5.smithi186.stderr:2022-04-03T19:44:04.614+0000 7f771e509700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-03T19:44:04.928 INFO:tasks.ceph.osd.0.smithi077.stderr:2022-04-03T19:44:04.928+0000 7f286b524700 -1 osd.0 0 waiting for initial osdmap 2022-04-03T19:44:04.932 INFO:tasks.ceph.osd.3.smithi077.stderr:2022-04-03T19:44:04.931+0000 7fc4c5c9a700 -1 osd.3 0 waiting for initial osdmap 2022-04-03T19:44:04.935 INFO:tasks.ceph.osd.0.smithi077.stderr:2022-04-03T19:44:04.935+0000 7f2863eb6700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-03T19:44:04.938 INFO:tasks.ceph.osd.3.smithi077.stderr:2022-04-03T19:44:04.938+0000 7fc4be62c700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-03T19:44:04.983 INFO:tasks.ceph.osd.1.smithi077.stderr:2022-04-03T19:44:04.983+0000 7f4216799700 -1 osd.1 0 waiting for initial osdmap 2022-04-03T19:44:04.990 INFO:tasks.ceph.osd.1.smithi077.stderr:2022-04-03T19:44:04.990+0000 7f420f12b700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-03T19:44:05.003 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T19:44:05.003+0000 7f880b766700 -1 osd.2 0 waiting for initial osdmap 2022-04-03T19:44:05.008 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T19:44:05.008+0000 7f88040f8700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-04-03T19:44:05.071 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-04-03T19:44:05.242 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.241+0000 7f0160800700 1 Processor -- start 2022-04-03T19:44:05.242 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.241+0000 7f0160800700 1 -- start start 2022-04-03T19:44:05.245 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.242+0000 7f0160800700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f015c0c5280 0x7f015c069630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:05.245 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.242+0000 7f0160800700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f015c1586f0 0x7f015c0c7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:05.245 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.242+0000 7f0160800700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015c155a00 con 0x7f015c0c5650 2022-04-03T19:44:05.246 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.242+0000 7f0160800700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f015c1595d0 con 0x7f015c0c5280 2022-04-03T19:44:05.246 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.242+0000 7f0160800700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c1551a0 con 0x7f015c1586f0 2022-04-03T19:44:05.246 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.243+0000 7f015ad9d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f015c0c5280 0x7f015c069630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:05.246 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.243+0000 7f015b59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f015c1586f0 0x7f015c0c7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:05.247 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.243+0000 7f015ad9d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f015c0c5280 0x7f015c069630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37604/0 (socket says 172.21.15.77:37604) 2022-04-03T19:44:05.247 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.243+0000 7f015a59c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f015c0c5650 0x7f015c0c6a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38260/0 (socket says 172.21.15.77:38260) 2022-04-03T19:44:05.247 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.243+0000 7f015ad9d700 1 -- 172.21.15.77:0/1547018760 learned_addr learned my addr 172.21.15.77:0/1547018760 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:05.248 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.243+0000 7f015ad9d700 1 -- 172.21.15.77:0/1547018760 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f015c1586f0 msgr2=0x7f015c0c7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:05.248 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.243+0000 7f015ad9d700 1 --2- 172.21.15.77:0/1547018760 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f015c1586f0 0x7f015c0c7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:05.248 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.243+0000 7f0159d9b700 1 -- 172.21.15.77:0/1547018760 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882861054 0 0) 0x7f015c155a00 con 0x7f015c0c5650 2022-04-03T19:44:05.249 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.243+0000 7f015ad9d700 1 -- 172.21.15.77:0/1547018760 >> v1:172.21.15.77:6789/0 conn(0x7f015c0c5650 legacy=0x7f015c0c6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:05.249 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.243+0000 7f015ad9d700 1 -- 172.21.15.77:0/1547018760 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c159010 con 0x7f015c0c5280 2022-04-03T19:44:05.249 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.243+0000 7f015b59e700 1 --2- 172.21.15.77:0/1547018760 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f015c1586f0 0x7f015c0c7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:05.249 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.244+0000 7f015ad9d700 1 --2- 172.21.15.77:0/1547018760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f015c0c5280 0x7f015c069630 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f014c006fd0 tx=0x7f014c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=292995e52dba30b7 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:05.250 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.244+0000 7f0159d9b700 1 -- 172.21.15.77:0/1547018760 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c004030 con 0x7f015c0c5280 2022-04-03T19:44:05.250 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.244+0000 7f0159d9b700 1 -- 172.21.15.77:0/1547018760 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014c0041d0 con 0x7f015c0c5280 2022-04-03T19:44:05.250 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.244+0000 7f0159d9b700 1 -- 172.21.15.77:0/1547018760 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c011630 con 0x7f015c0c5280 2022-04-03T19:44:05.250 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.244+0000 7f0160800700 1 -- 172.21.15.77:0/1547018760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f015c0c5280 msgr2=0x7f015c069630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:05.250 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.244+0000 7f0160800700 1 --2- 172.21.15.77:0/1547018760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f015c0c5280 0x7f015c069630 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f014c006fd0 tx=0x7f014c0090f0 comp rx=0 tx=0).stop 2022-04-03T19:44:05.251 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.244+0000 7f0160800700 1 -- 172.21.15.77:0/1547018760 shutdown_connections 2022-04-03T19:44:05.251 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.244+0000 7f0160800700 1 --2- 172.21.15.77:0/1547018760 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f015c1586f0 0x7f015c0c7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:05.251 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.244+0000 7f0160800700 1 --2- 172.21.15.77:0/1547018760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f015c0c5280 0x7f015c069630 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:05.252 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.244+0000 7f0160800700 1 -- 172.21.15.77:0/1547018760 >> 172.21.15.77:0/1547018760 conn(0x7f015c0cb700 msgr2=0x7f015c0cbb00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:05.252 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.244+0000 7f0160800700 1 -- 172.21.15.77:0/1547018760 shutdown_connections 2022-04-03T19:44:05.253 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.244+0000 7f0160800700 1 -- 172.21.15.77:0/1547018760 wait complete. 2022-04-03T19:44:05.259 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.246+0000 7f0160800700 1 Processor -- start 2022-04-03T19:44:05.259 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.246+0000 7f0160800700 1 -- start start 2022-04-03T19:44:05.260 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.247+0000 7f0160800700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f015c0c5280 0x7f015c1506c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:05.260 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.247+0000 7f0160800700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f015c0c5650 0x7f015c151710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:05.261 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.247+0000 7f0160800700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f015c1586f0 0x7f015c1651e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:05.261 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.247+0000 7f0160800700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c164db0 con 0x7f015c0c5650 2022-04-03T19:44:05.261 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.247+0000 7f0160800700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c152470 con 0x7f015c0c5280 2022-04-03T19:44:05.262 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.247+0000 7f0160800700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c0bf430 con 0x7f015c1586f0 2022-04-03T19:44:05.262 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.247+0000 7f015a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f015c0c5650 0x7f015c151710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:05.263 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.247+0000 7f015ad9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f015c0c5280 0x7f015c1506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:05.263 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.247+0000 7f015ad9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f015c0c5280 0x7f015c1506c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37606/0 (socket says 172.21.15.77:37606) 2022-04-03T19:44:05.264 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.248+0000 7f015a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f015c0c5650 0x7f015c151710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42426/0 (socket says 172.21.15.77:42426) 2022-04-03T19:44:05.264 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.248+0000 7f015ad9d700 1 -- 172.21.15.77:0/2942834420 learned_addr learned my addr 172.21.15.77:0/2942834420 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:05.264 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.248+0000 7f015b59e700 1 --2- 172.21.15.77:0/2942834420 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f015c1586f0 0x7f015c1651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:05.265 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.248+0000 7f015ad9d700 1 -- 172.21.15.77:0/2942834420 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f015c1586f0 msgr2=0x7f015c1651e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:05.265 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.248+0000 7f015ad9d700 1 --2- 172.21.15.77:0/2942834420 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f015c1586f0 0x7f015c1651e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:05.265 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.248+0000 7f015ad9d700 1 -- 172.21.15.77:0/2942834420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f015c0c5650 msgr2=0x7f015c151710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:05.266 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.248+0000 7f015ad9d700 1 --2- 172.21.15.77:0/2942834420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f015c0c5650 0x7f015c151710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:05.266 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.248+0000 7f015ad9d700 1 -- 172.21.15.77:0/2942834420 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0154009a20 con 0x7f015c0c5280 2022-04-03T19:44:05.266 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.248+0000 7f015ad9d700 1 --2- 172.21.15.77:0/2942834420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f015c0c5280 0x7f015c1506c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f014c008000 tx=0x7f014c016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:05.267 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.248+0000 7f0142ffd700 1 -- 172.21.15.77:0/2942834420 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c003770 con 0x7f015c0c5280 2022-04-03T19:44:05.267 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.249+0000 7f0142ffd700 1 -- 172.21.15.77:0/2942834420 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014c0038d0 con 0x7f015c0c5280 2022-04-03T19:44:05.267 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.249+0000 7f0160800700 1 -- 172.21.15.77:0/2942834420 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f014c009370 con 0x7f015c0c5280 2022-04-03T19:44:05.267 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.249+0000 7f0142ffd700 1 -- 172.21.15.77:0/2942834420 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c003c90 con 0x7f015c0c5280 2022-04-03T19:44:05.268 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.249+0000 7f0160800700 1 -- 172.21.15.77:0/2942834420 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f015c0bf990 con 0x7f015c0c5280 2022-04-03T19:44:05.268 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.250+0000 7f0160800700 1 -- 172.21.15.77:0/2942834420 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015c088e40 con 0x7f015c0c5280 2022-04-03T19:44:05.268 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.250+0000 7f0142ffd700 1 -- 172.21.15.77:0/2942834420 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) v1 ==== 81882+0+0 (secure 0 0 0) 0x7f014c02b030 con 0x7f015c0c5280 2022-04-03T19:44:05.268 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.251+0000 7f0142ffd700 1 --2- 172.21.15.77:0/2942834420 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f013c063d40 0x7f013c0661f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:05.268 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.251+0000 7f0142ffd700 1 -- 172.21.15.77:0/2942834420 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2227+0+0 (secure 0 0 0) 0x7f014c0811a0 con 0x7f015c0c5280 2022-04-03T19:44:05.269 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.251+0000 7f015a59c700 1 --2- 172.21.15.77:0/2942834420 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f013c063d40 0x7f013c0661f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:05.269 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.252+0000 7f015a59c700 1 --2- 172.21.15.77:0/2942834420 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f013c063d40 0x7f013c0661f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f015c0beec0 tx=0x7f0150007bb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:05.269 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.256+0000 7f0142ffd700 1 -- 172.21.15.77:0/2942834420 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f014c050c90 con 0x7f015c0c5280 2022-04-03T19:44:05.450 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.450+0000 7f0160800700 1 -- 172.21.15.77:0/2942834420 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f015c165030 con 0x7f015c0c5280 2022-04-03T19:44:05.451 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.451+0000 7f0142ffd700 1 -- 172.21.15.77:0/2942834420 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+6992 (secure 0 0 0) 0x7f014c029030 con 0x7f015c0c5280 2022-04-03T19:44:05.452 INFO:teuthology.misc.health.smithi077.stdout: 2022-04-03T19:44:05.452 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":12,"fsid":"89c3a3f4-599d-4c4a-8f5a-7180eafb7292","created":"2022-04-03T19:43:54.101559+0000","modified":"2022-04-03T19:44:02.492748+0000","last_up_change":"0.000000","last_in_change":"2022-04-03T19:43:56.729760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"f35f5195-3729-4d78-85d1-0539f259283f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ab1c89dc-57ce-4cc4-b8c7-9b0c19939601","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"351e218b-7749-4ee9-b44a-0a5b52fb9be1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1ba6ec8a-7cbf-4c12-8e21-c3097a263ae8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3faf4c09-846c-4133-856b-7f53c535a67d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ffe41d00-d0aa-42f4-82c1-5cced13c1bc3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b0344ee8-deca-4e9b-934c-5866d90d0073","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0eb0e5d3-3dad-4f49-9b50-3b24a3924bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-03T19:44:05.455 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.453+0000 7f0160800700 1 -- 172.21.15.77:0/2942834420 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f013c063d40 msgr2=0x7f013c0661f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:05.455 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.453+0000 7f0160800700 1 --2- 172.21.15.77:0/2942834420 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f013c063d40 0x7f013c0661f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f015c0beec0 tx=0x7f0150007bb0 comp rx=0 tx=0).stop 2022-04-03T19:44:05.455 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.453+0000 7f0160800700 1 -- 172.21.15.77:0/2942834420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f015c0c5280 msgr2=0x7f015c1506c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:05.456 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.453+0000 7f0160800700 1 --2- 172.21.15.77:0/2942834420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f015c0c5280 0x7f015c1506c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f014c008000 tx=0x7f014c016000 comp rx=0 tx=0).stop 2022-04-03T19:44:05.456 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.453+0000 7f0160800700 1 -- 172.21.15.77:0/2942834420 shutdown_connections 2022-04-03T19:44:05.457 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.453+0000 7f0160800700 1 --2- 172.21.15.77:0/2942834420 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f013c063d40 0x7f013c0661f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:05.457 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.453+0000 7f0160800700 1 --2- 172.21.15.77:0/2942834420 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f015c1586f0 0x7f015c1651e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:05.458 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.453+0000 7f0160800700 1 --2- 172.21.15.77:0/2942834420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f015c0c5280 0x7f015c1506c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:05.458 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.453+0000 7f0160800700 1 --2- 172.21.15.77:0/2942834420 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f015c0c5650 0x7f015c151710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:05.458 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.453+0000 7f0160800700 1 -- 172.21.15.77:0/2942834420 >> 172.21.15.77:0/2942834420 conn(0x7f015c0cb700 msgr2=0x7f015c154310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:05.459 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.454+0000 7f0160800700 1 -- 172.21.15.77:0/2942834420 shutdown_connections 2022-04-03T19:44:05.459 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:05.454+0000 7f0160800700 1 -- 172.21.15.77:0/2942834420 wait complete. 2022-04-03T19:44:05.474 DEBUG:teuthology.misc:0 of 8 OSDs are up 2022-04-03T19:44:12.281 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-04-03T19:44:12.459 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.457+0000 7f8fc901a700 1 Processor -- start 2022-04-03T19:44:12.459 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.458+0000 7f8fc901a700 1 -- start start 2022-04-03T19:44:12.461 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.459+0000 7f8fc901a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8fc4067ea0 0x7f8fc40c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:12.461 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.459+0000 7f8fc901a700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8fc40674f0 0x7f8fc40c0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:12.462 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.459+0000 7f8fc901a700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc40bfba0 con 0x7f8fc4067120 2022-04-03T19:44:12.462 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.459+0000 7f8fc901a700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fc40c2700 con 0x7f8fc4067ea0 2022-04-03T19:44:12.462 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.459+0000 7f8fc901a700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fc40c29e0 con 0x7f8fc40674f0 2022-04-03T19:44:12.462 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.459+0000 7f8fc37fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8fc4067ea0 0x7f8fc40c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:12.463 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.459+0000 7f8fc37fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8fc4067ea0 0x7f8fc40c0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42428/0 (socket says 172.21.15.77:42428) 2022-04-03T19:44:12.463 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.459+0000 7f8fc37fe700 1 -- 172.21.15.77:0/1791000214 learned_addr learned my addr 172.21.15.77:0/1791000214 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:12.463 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.459+0000 7f8fc8819700 1 --2- 172.21.15.77:0/1791000214 >> v2:172.21.15.186:3300/0 conn(0x7f8fc40674f0 0x7f8fc40c0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:12.464 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.460+0000 7f8fc37fe700 1 -- 172.21.15.77:0/1791000214 >> v2:172.21.15.186:3300/0 conn(0x7f8fc40674f0 msgr2=0x7f8fc40c0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:12.464 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.460+0000 7f8fc37fe700 1 --2- 172.21.15.77:0/1791000214 >> v2:172.21.15.186:3300/0 conn(0x7f8fc40674f0 0x7f8fc40c0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:12.464 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.460+0000 7f8fc37fe700 1 -- 172.21.15.77:0/1791000214 >> v1:172.21.15.186:6789/0 conn(0x7f8fc4067120 legacy=0x7f8fc40c1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:12.464 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.460+0000 7f8fc37fe700 1 -- 172.21.15.77:0/1791000214 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc40c2cc0 con 0x7f8fc4067ea0 2022-04-03T19:44:12.465 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.460+0000 7f8fc37fe700 1 --2- 172.21.15.77:0/1791000214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fc4067ea0 0x7f8fc40c0330 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8fc40c1870 tx=0x7f8fb80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51230337dec5f003 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:12.471 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.460+0000 7f8fc2ffd700 1 -- 172.21.15.77:0/1791000214 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8004030 con 0x7f8fc4067ea0 2022-04-03T19:44:12.471 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.460+0000 7f8fc2ffd700 1 -- 172.21.15.77:0/1791000214 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb8004190 con 0x7f8fc4067ea0 2022-04-03T19:44:12.471 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.460+0000 7f8fc2ffd700 1 -- 172.21.15.77:0/1791000214 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8011630 con 0x7f8fc4067ea0 2022-04-03T19:44:12.471 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.461+0000 7f8fc901a700 1 -- 172.21.15.77:0/1791000214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fc4067ea0 msgr2=0x7f8fc40c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:12.472 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.461+0000 7f8fc901a700 1 --2- 172.21.15.77:0/1791000214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fc4067ea0 0x7f8fc40c0330 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8fc40c1870 tx=0x7f8fb80090f0 comp rx=0 tx=0).stop 2022-04-03T19:44:12.472 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.461+0000 7f8fc901a700 1 -- 172.21.15.77:0/1791000214 shutdown_connections 2022-04-03T19:44:12.472 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.461+0000 7f8fc901a700 1 --2- 172.21.15.77:0/1791000214 >> v2:172.21.15.186:3300/0 conn(0x7f8fc40674f0 0x7f8fc40c0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:12.472 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.461+0000 7f8fc901a700 1 --2- 172.21.15.77:0/1791000214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fc4067ea0 0x7f8fc40c0330 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:12.472 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.461+0000 7f8fc901a700 1 -- 172.21.15.77:0/1791000214 >> 172.21.15.77:0/1791000214 conn(0x7f8fc40c72c0 msgr2=0x7f8fc40c76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:12.473 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.461+0000 7f8fc901a700 1 -- 172.21.15.77:0/1791000214 shutdown_connections 2022-04-03T19:44:12.473 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.461+0000 7f8fc901a700 1 -- 172.21.15.77:0/1791000214 wait complete. 2022-04-03T19:44:12.473 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.462+0000 7f8fc901a700 1 Processor -- start 2022-04-03T19:44:12.473 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.462+0000 7f8fc901a700 1 -- start start 2022-04-03T19:44:12.474 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.463+0000 7f8fc901a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fc4067120 0x7f8fc413e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:12.474 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.463+0000 7f8fc901a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc40674f0 0x7f8fc414b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:12.474 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.463+0000 7f8fc901a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fc4067ea0 0x7f8fc414d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:12.475 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.463+0000 7f8fc901a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc40b3ef0 con 0x7f8fc4067120 2022-04-03T19:44:12.475 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.463+0000 7f8fc901a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc40b6060 con 0x7f8fc40674f0 2022-04-03T19:44:12.475 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.463+0000 7f8fc901a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc40b6340 con 0x7f8fc4067ea0 2022-04-03T19:44:12.476 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.464+0000 7f8fc3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fc4067120 0x7f8fc413e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:12.476 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.464+0000 7f8fc37fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc40674f0 0x7f8fc414b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:12.476 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.464+0000 7f8fc8819700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fc4067ea0 0x7f8fc414d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:12.476 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.464+0000 7f8fc37fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc40674f0 0x7f8fc414b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37610/0 (socket says 172.21.15.77:37610) 2022-04-03T19:44:12.477 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.464+0000 7f8fc37fe700 1 -- 172.21.15.77:0/44625703 learned_addr learned my addr 172.21.15.77:0/44625703 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:12.477 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.464+0000 7f8fc3fff700 1 -- 172.21.15.77:0/44625703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fc4067ea0 msgr2=0x7f8fc414d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:12.477 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.464+0000 7f8fc3fff700 1 --2- 172.21.15.77:0/44625703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fc4067ea0 0x7f8fc414d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:12.477 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.464+0000 7f8fc3fff700 1 -- 172.21.15.77:0/44625703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc40674f0 msgr2=0x7f8fc414b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:12.477 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.464+0000 7f8fc3fff700 1 --2- 172.21.15.77:0/44625703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc40674f0 0x7f8fc414b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:12.478 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.464+0000 7f8fc3fff700 1 -- 172.21.15.77:0/44625703 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb8009370 con 0x7f8fc4067120 2022-04-03T19:44:12.478 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.464+0000 7f8fc3fff700 1 --2- 172.21.15.77:0/44625703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fc4067120 0x7f8fc413e780 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400c440 tx=0x7f8fb4007d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:12.478 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.465+0000 7f8fabfff700 1 -- 172.21.15.77:0/44625703 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb40050d0 con 0x7f8fc4067120 2022-04-03T19:44:12.478 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.465+0000 7f8fabfff700 1 -- 172.21.15.77:0/44625703 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb4005230 con 0x7f8fc4067120 2022-04-03T19:44:12.478 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.465+0000 7f8fc901a700 1 -- 172.21.15.77:0/44625703 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc40b41b0 con 0x7f8fc4067120 2022-04-03T19:44:12.479 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.465+0000 7f8fabfff700 1 -- 172.21.15.77:0/44625703 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4003710 con 0x7f8fc4067120 2022-04-03T19:44:12.479 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.465+0000 7f8fc901a700 1 -- 172.21.15.77:0/44625703 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fc40b5d60 con 0x7f8fc4067120 2022-04-03T19:44:12.479 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.466+0000 7f8fc901a700 1 -- 172.21.15.77:0/44625703 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fc40b4520 con 0x7f8fc4067120 2022-04-03T19:44:12.479 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.467+0000 7f8fabfff700 1 -- 172.21.15.77:0/44625703 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f8fb4019030 con 0x7f8fc4067120 2022-04-03T19:44:12.480 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.467+0000 7f8fabfff700 1 --2- 172.21.15.77:0/44625703 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8fa4063ce0 0x7f8fa4066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:12.480 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.467+0000 7f8fabfff700 1 -- 172.21.15.77:0/44625703 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f8fb407f480 con 0x7f8fc4067120 2022-04-03T19:44:12.480 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.468+0000 7f8fc37fe700 1 --2- 172.21.15.77:0/44625703 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8fa4063ce0 0x7f8fa4066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:12.480 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.468+0000 7f8fc37fe700 1 --2- 172.21.15.77:0/44625703 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8fa4063ce0 0x7f8fa4066190 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8fc41467f0 tx=0x7f8fb8016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:12.481 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.472+0000 7f8fabfff700 1 -- 172.21.15.77:0/44625703 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f8fb404e4f0 con 0x7f8fc4067120 2022-04-03T19:44:12.668 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.666+0000 7f8fc901a700 1 -- 172.21.15.77:0/44625703 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fc4156300 con 0x7f8fc4067120 2022-04-03T19:44:12.668 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.667+0000 7f8fabfff700 1 -- 172.21.15.77:0/44625703 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12106 (secure 0 0 0) 0x7f8fb4017020 con 0x7f8fc4067120 2022-04-03T19:44:12.669 INFO:teuthology.misc.health.smithi077.stdout: 2022-04-03T19:44:12.670 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":17,"fsid":"89c3a3f4-599d-4c4a-8f5a-7180eafb7292","created":"2022-04-03T19:43:54.101559+0000","modified":"2022-04-03T19:44:09.718681+0000","last_up_change":"2022-04-03T19:44:05.589174+0000","last_in_change":"2022-04-03T19:43:56.729760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-03T19:44:07.567638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f35f5195-3729-4d78-85d1-0539f259283f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6823","nonce":20252},{"type":"v1","addr":"172.21.15.77:6825","nonce":20252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":20252},{"type":"v1","addr":"172.21.15.77:6827","nonce":20252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":20252},{"type":"v1","addr":"172.21.15.77:6831","nonce":20252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":20252},{"type":"v1","addr":"172.21.15.77:6829","nonce":20252}]},"public_addr":"172.21.15.77:6825/20252","cluster_addr":"172.21.15.77:6827/20252","heartbeat_back_addr":"172.21.15.77:6831/20252","heartbeat_front_addr":"172.21.15.77:6829/20252","state":["exists","up"]},{"osd":1,"uuid":"ab1c89dc-57ce-4cc4-b8c7-9b0c19939601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":20249},{"type":"v1","addr":"172.21.15.77:6805","nonce":20249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":20249},{"type":"v1","addr":"172.21.15.77:6809","nonce":20249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":20249},{"type":"v1","addr":"172.21.15.77:6819","nonce":20249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":20249},{"type":"v1","addr":"172.21.15.77:6815","nonce":20249}]},"public_addr":"172.21.15.77:6805/20249","cluster_addr":"172.21.15.77:6809/20249","heartbeat_back_addr":"172.21.15.77:6819/20249","heartbeat_front_addr":"172.21.15.77:6815/20249","state":["exists","up"]},{"osd":2,"uuid":"351e218b-7749-4ee9-b44a-0a5b52fb9be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":20248},{"type":"v1","addr":"172.21.15.77:6801","nonce":20248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":20248},{"type":"v1","addr":"172.21.15.77:6803","nonce":20248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":20248},{"type":"v1","addr":"172.21.15.77:6813","nonce":20248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":20248},{"type":"v1","addr":"172.21.15.77:6808","nonce":20248}]},"public_addr":"172.21.15.77:6801/20248","cluster_addr":"172.21.15.77:6803/20248","heartbeat_back_addr":"172.21.15.77:6813/20248","heartbeat_front_addr":"172.21.15.77:6808/20248","state":["exists","up"]},{"osd":3,"uuid":"1ba6ec8a-7cbf-4c12-8e21-c3097a263ae8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":20250},{"type":"v1","addr":"172.21.15.77:6814","nonce":20250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":20250},{"type":"v1","addr":"172.21.15.77:6818","nonce":20250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":20250},{"type":"v1","addr":"172.21.15.77:6824","nonce":20250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":20250},{"type":"v1","addr":"172.21.15.77:6821","nonce":20250}]},"public_addr":"172.21.15.77:6814/20250","cluster_addr":"172.21.15.77:6818/20250","heartbeat_back_addr":"172.21.15.77:6824/20250","heartbeat_front_addr":"172.21.15.77:6821/20250","state":["exists","up"]},{"osd":4,"uuid":"3faf4c09-846c-4133-856b-7f53c535a67d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":25551},{"type":"v1","addr":"172.21.15.186:6822","nonce":25551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":25551},{"type":"v1","addr":"172.21.15.186:6826","nonce":25551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":25551},{"type":"v1","addr":"172.21.15.186:6831","nonce":25551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":25551},{"type":"v1","addr":"172.21.15.186:6829","nonce":25551}]},"public_addr":"172.21.15.186:6822/25551","cluster_addr":"172.21.15.186:6826/25551","heartbeat_back_addr":"172.21.15.186:6831/25551","heartbeat_front_addr":"172.21.15.186:6829/25551","state":["exists","up"]},{"osd":5,"uuid":"ffe41d00-d0aa-42f4-82c1-5cced13c1bc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":25541},{"type":"v1","addr":"172.21.15.186:6809","nonce":25541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":25541},{"type":"v1","addr":"172.21.15.186:6811","nonce":25541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":25541},{"type":"v1","addr":"172.21.15.186:6815","nonce":25541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":25541},{"type":"v1","addr":"172.21.15.186:6813","nonce":25541}]},"public_addr":"172.21.15.186:6809/25541","cluster_addr":"172.21.15.186:6811/25541","heartbeat_back_addr":"172.21.15.186:6815/25541","heartbeat_front_addr":"172.21.15.186:6813/25541","state":["exists","up"]},{"osd":6,"uuid":"b0344ee8-deca-4e9b-934c-5866d90d0073","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":25542},{"type":"v1","addr":"172.21.15.186:6801","nonce":25542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":25542},{"type":"v1","addr":"172.21.15.186:6803","nonce":25542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":25542},{"type":"v1","addr":"172.21.15.186:6807","nonce":25542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":25542},{"type":"v1","addr":"172.21.15.186:6805","nonce":25542}]},"public_addr":"172.21.15.186:6801/25542","cluster_addr":"172.21.15.186:6803/25542","heartbeat_back_addr":"172.21.15.186:6807/25542","heartbeat_front_addr":"172.21.15.186:6805/25542","state":["exists","up"]},{"osd":7,"uuid":"0eb0e5d3-3dad-4f49-9b50-3b24a3924bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":25572},{"type":"v1","addr":"172.21.15.186:6817","nonce":25572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":25572},{"type":"v1","addr":"172.21.15.186:6819","nonce":25572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6825","nonce":25572},{"type":"v1","addr":"172.21.15.186:6827","nonce":25572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":25572},{"type":"v1","addr":"172.21.15.186:6823","nonce":25572}]},"public_addr":"172.21.15.186:6817/25572","cluster_addr":"172.21.15.186:6819/25572","heartbeat_back_addr":"172.21.15.186:6827/25572","heartbeat_front_addr":"172.21.15.186:6823/25572","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.143602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.490366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.361292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.343544+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.905429+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969298+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.949524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-03T19:44:12.672 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.669+0000 7f8fc901a700 1 -- 172.21.15.77:0/44625703 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8fa4063ce0 msgr2=0x7f8fa4066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:12.672 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.669+0000 7f8fc901a700 1 --2- 172.21.15.77:0/44625703 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8fa4063ce0 0x7f8fa4066190 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8fc41467f0 tx=0x7f8fb8016000 comp rx=0 tx=0).stop 2022-04-03T19:44:12.673 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.669+0000 7f8fc901a700 1 -- 172.21.15.77:0/44625703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fc4067120 msgr2=0x7f8fc413e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:12.673 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.669+0000 7f8fc901a700 1 --2- 172.21.15.77:0/44625703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fc4067120 0x7f8fc413e780 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400c440 tx=0x7f8fb4007d40 comp rx=0 tx=0).stop 2022-04-03T19:44:12.673 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.670+0000 7f8fc901a700 1 -- 172.21.15.77:0/44625703 shutdown_connections 2022-04-03T19:44:12.673 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.670+0000 7f8fc901a700 1 --2- 172.21.15.77:0/44625703 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8fa4063ce0 0x7f8fa4066190 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:12.673 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.670+0000 7f8fc901a700 1 --2- 172.21.15.77:0/44625703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fc4067ea0 0x7f8fc414d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:12.674 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.670+0000 7f8fc901a700 1 --2- 172.21.15.77:0/44625703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fc4067120 0x7f8fc413e780 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:12.674 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.670+0000 7f8fc901a700 1 --2- 172.21.15.77:0/44625703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc40674f0 0x7f8fc414b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:12.674 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.670+0000 7f8fc901a700 1 -- 172.21.15.77:0/44625703 >> 172.21.15.77:0/44625703 conn(0x7f8fc40c72c0 msgr2=0x7f8fc406d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:12.674 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.670+0000 7f8fc901a700 1 -- 172.21.15.77:0/44625703 shutdown_connections 2022-04-03T19:44:12.674 INFO:teuthology.misc.health.smithi077.stderr:2022-04-03T19:44:12.670+0000 7f8fc901a700 1 -- 172.21.15.77:0/44625703 wait complete. 2022-04-03T19:44:12.691 DEBUG:teuthology.misc:8 of 8 OSDs are up 2022-04-03T19:44:12.692 INFO:tasks.ceph:Creating RBD pool 2022-04-03T19:44:12.692 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd pool create rbd 8 2022-04-03T19:44:12.879 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.878+0000 7f3593fff700 1 Processor -- start 2022-04-03T19:44:12.880 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.878+0000 7f3593fff700 1 -- start start 2022-04-03T19:44:12.883 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.879+0000 7f3593fff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3594158d20 0x7f35941590f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:12.883 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.879+0000 7f3593fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359414f890 con 0x7f359406ba20 2022-04-03T19:44:12.883 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.879+0000 7f3593fff700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35940c7a60 con 0x7f35940c8790 2022-04-03T19:44:12.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.879+0000 7f3593fff700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35940c6a30 con 0x7f3594158d20 2022-04-03T19:44:12.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.880+0000 7f3592ffd700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3594158d20 0x7f35941590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:12.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.880+0000 7f35927fc700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f35940c8790 0x7f35940c2af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44074/0 (socket says 172.21.15.77:44074) 2022-04-03T19:44:12.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.880+0000 7f35937fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f359406ba20 0x7f35940c5f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38262/0 (socket says 172.21.15.77:38262) 2022-04-03T19:44:12.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.880+0000 7f35927fc700 1 -- 172.21.15.77:0/3120238866 learned_addr learned my addr 172.21.15.77:0/3120238866 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:12.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.880+0000 7f3591ffb700 1 -- 172.21.15.77:0/3120238866 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337773732 0 0) 0x7f35940c7a60 con 0x7f35940c8790 2022-04-03T19:44:12.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.880+0000 7f3591ffb700 1 -- 172.21.15.77:0/3120238866 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35800033f0 con 0x7f35940c8790 2022-04-03T19:44:12.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.880+0000 7f3591ffb700 1 -- 172.21.15.77:0/3120238866 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3738140964 0 0) 0x7f359414f890 con 0x7f359406ba20 2022-04-03T19:44:12.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.880+0000 7f3591ffb700 1 -- 172.21.15.77:0/3120238866 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35940c7a60 con 0x7f359406ba20 2022-04-03T19:44:12.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.880+0000 7f3592ffd700 1 -- 172.21.15.77:0/3120238866 >> v1:172.21.15.186:6789/0 conn(0x7f35940c8790 legacy=0x7f35940c2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:12.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.880+0000 7f3592ffd700 1 -- 172.21.15.77:0/3120238866 >> v1:172.21.15.77:6789/0 conn(0x7f359406ba20 legacy=0x7f35940c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:12.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.880+0000 7f3592ffd700 1 -- 172.21.15.77:0/3120238866 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35940c6d10 con 0x7f3594158d20 2022-04-03T19:44:12.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.881+0000 7f3592ffd700 1 --2- 172.21.15.77:0/3120238866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3594158d20 0x7f35941590f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3584009760 tx=0x7f3584009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c4d86b45a57ac5c server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:12.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.881+0000 7f3591ffb700 1 -- 172.21.15.77:0/3120238866 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35840105b0 con 0x7f3594158d20 2022-04-03T19:44:12.887 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.881+0000 7f3591ffb700 1 -- 172.21.15.77:0/3120238866 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3584010710 con 0x7f3594158d20 2022-04-03T19:44:12.887 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.881+0000 7f3591ffb700 1 -- 172.21.15.77:0/3120238866 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3584010a90 con 0x7f3594158d20 2022-04-03T19:44:12.887 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.881+0000 7f3593fff700 1 -- 172.21.15.77:0/3120238866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3594158d20 msgr2=0x7f35941590f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:12.887 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.881+0000 7f3593fff700 1 --2- 172.21.15.77:0/3120238866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3594158d20 0x7f35941590f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3584009760 tx=0x7f3584009060 comp rx=0 tx=0).stop 2022-04-03T19:44:12.887 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.881+0000 7f3593fff700 1 -- 172.21.15.77:0/3120238866 shutdown_connections 2022-04-03T19:44:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.881+0000 7f3593fff700 1 --2- 172.21.15.77:0/3120238866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3594158d20 0x7f35941590f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.881+0000 7f3593fff700 1 -- 172.21.15.77:0/3120238866 >> 172.21.15.77:0/3120238866 conn(0x7f35940626d0 msgr2=0x7f3594062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.882+0000 7f3593fff700 1 -- 172.21.15.77:0/3120238866 shutdown_connections 2022-04-03T19:44:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.882+0000 7f3593fff700 1 -- 172.21.15.77:0/3120238866 wait complete. 2022-04-03T19:44:12.892 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.883+0000 7f3593fff700 1 Processor -- start 2022-04-03T19:44:12.892 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.883+0000 7f3593fff700 1 -- start start 2022-04-03T19:44:12.892 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.884+0000 7f3593fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f359406ba20 0x7f359414eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:12.892 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.884+0000 7f3593fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35940c8790 0x7f3594168df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:12.893 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.884+0000 7f3593fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3594158d20 0x7f3594169330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:12.893 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.884+0000 7f3593fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3594168960 con 0x7f3594158d20 2022-04-03T19:44:12.893 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.884+0000 7f3593fff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35940a8220 con 0x7f359406ba20 2022-04-03T19:44:12.893 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.884+0000 7f3593fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3594165e80 con 0x7f35940c8790 2022-04-03T19:44:12.893 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.884+0000 7f3592ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f359406ba20 0x7f359414eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:12.894 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.884+0000 7f35937fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3594158d20 0x7f3594169330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:12.894 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.885+0000 7f3592ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f359406ba20 0x7f359414eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37614/0 (socket says 172.21.15.77:37614) 2022-04-03T19:44:12.894 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.885+0000 7f3592ffd700 1 -- 172.21.15.77:0/291549210 learned_addr learned my addr 172.21.15.77:0/291549210 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:12.894 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.885+0000 7f35927fc700 1 --2- 172.21.15.77:0/291549210 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35940c8790 0x7f3594168df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:12.895 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.885+0000 7f3592ffd700 1 -- 172.21.15.77:0/291549210 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35940c8790 msgr2=0x7f3594168df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:12.895 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.885+0000 7f3592ffd700 1 --2- 172.21.15.77:0/291549210 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35940c8790 0x7f3594168df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:12.895 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.885+0000 7f3592ffd700 1 -- 172.21.15.77:0/291549210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3594158d20 msgr2=0x7f3594169330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:12.895 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.885+0000 7f3592ffd700 1 --2- 172.21.15.77:0/291549210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3594158d20 0x7f3594169330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:12.895 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.885+0000 7f3592ffd700 1 -- 172.21.15.77:0/291549210 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35840092a0 con 0x7f359406ba20 2022-04-03T19:44:12.896 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.885+0000 7f3592ffd700 1 --2- 172.21.15.77:0/291549210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f359406ba20 0x7f359414eb80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3594089470 tx=0x7f3584016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:12.896 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.886+0000 7f357affd700 1 -- 172.21.15.77:0/291549210 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3584010880 con 0x7f359406ba20 2022-04-03T19:44:12.896 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.886+0000 7f357affd700 1 -- 172.21.15.77:0/291549210 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3584010a00 con 0x7f359406ba20 2022-04-03T19:44:12.896 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.886+0000 7f357affd700 1 -- 172.21.15.77:0/291549210 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3584025640 con 0x7f359406ba20 2022-04-03T19:44:12.897 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.886+0000 7f3593fff700 1 -- 172.21.15.77:0/291549210 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3594166140 con 0x7f359406ba20 2022-04-03T19:44:12.897 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.886+0000 7f3593fff700 1 -- 172.21.15.77:0/291549210 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3594153f60 con 0x7f359406ba20 2022-04-03T19:44:12.897 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.887+0000 7f3593fff700 1 -- 172.21.15.77:0/291549210 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3594168c40 con 0x7f359406ba20 2022-04-03T19:44:12.897 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.888+0000 7f357affd700 1 -- 172.21.15.77:0/291549210 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f358400f070 con 0x7f359406ba20 2022-04-03T19:44:12.898 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.888+0000 7f357affd700 1 --2- 172.21.15.77:0/291549210 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3574063ce0 0x7f3574066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:12.898 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.888+0000 7f357affd700 1 -- 172.21.15.77:0/291549210 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f3584081950 con 0x7f359406ba20 2022-04-03T19:44:12.898 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.889+0000 7f35927fc700 1 --2- 172.21.15.77:0/291549210 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3574063ce0 0x7f3574066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:12.898 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.889+0000 7f35927fc700 1 --2- 172.21.15.77:0/291549210 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3574063ce0 0x7f3574066190 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3594152560 tx=0x7f3588009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:12.899 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:12.893+0000 7f357affd700 1 -- 172.21.15.77:0/291549210 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f35840508f0 con 0x7f359406ba20 2022-04-03T19:44:13.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.092+0000 7f3593fff700 1 -- 172.21.15.77:0/291549210 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f35940c7cb0 con 0x7f359406ba20 2022-04-03T19:44:13.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.737+0000 7f357affd700 1 -- 172.21.15.77:0/291549210 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v18) v1 ==== 109+0+0 (secure 0 0 0) 0x7f3584053f10 con 0x7f359406ba20 2022-04-03T19:44:13.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.740+0000 7f3593fff700 1 -- 172.21.15.77:0/291549210 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3574063ce0 msgr2=0x7f3574066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:13.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.740+0000 7f3593fff700 1 --2- 172.21.15.77:0/291549210 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3574063ce0 0x7f3574066190 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3594152560 tx=0x7f3588009040 comp rx=0 tx=0).stop 2022-04-03T19:44:13.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.740+0000 7f3593fff700 1 -- 172.21.15.77:0/291549210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f359406ba20 msgr2=0x7f359414eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:13.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.740+0000 7f3593fff700 1 --2- 172.21.15.77:0/291549210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f359406ba20 0x7f359414eb80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3594089470 tx=0x7f3584016000 comp rx=0 tx=0).stop 2022-04-03T19:44:13.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.741+0000 7f3593fff700 1 -- 172.21.15.77:0/291549210 shutdown_connections 2022-04-03T19:44:13.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.741+0000 7f3593fff700 1 --2- 172.21.15.77:0/291549210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3594158d20 0x7f3594169330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:13.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.741+0000 7f3593fff700 1 --2- 172.21.15.77:0/291549210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f359406ba20 0x7f359414eb80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:13.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.741+0000 7f3593fff700 1 --2- 172.21.15.77:0/291549210 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3574063ce0 0x7f3574066190 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:13.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.741+0000 7f3593fff700 1 --2- 172.21.15.77:0/291549210 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35940c8790 0x7f3594168df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:13.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.741+0000 7f3593fff700 1 -- 172.21.15.77:0/291549210 >> 172.21.15.77:0/291549210 conn(0x7f35940626d0 msgr2=0x7f35940cac80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:13.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.741+0000 7f3593fff700 1 -- 172.21.15.77:0/291549210 shutdown_connections 2022-04-03T19:44:13.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.741+0000 7f3593fff700 1 -- 172.21.15.77:0/291549210 wait complete. 2022-04-03T19:44:13.745 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'rbd' created 2022-04-03T19:44:13.768 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2022-04-03T19:44:13.963 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.961+0000 7f690ae29700 1 Processor -- start 2022-04-03T19:44:13.963 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.961+0000 7f690ae29700 1 -- start start 2022-04-03T19:44:13.966 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.963+0000 7f690ae29700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6904067930 0x7f69041568f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:13.966 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.963+0000 7f690ae29700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6904156330 con 0x7f69040bf810 2022-04-03T19:44:13.967 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.963+0000 7f690ae29700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69040c4860 con 0x7f69040bf440 2022-04-03T19:44:13.967 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.963+0000 7f690ae29700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69040c94e0 con 0x7f6904067930 2022-04-03T19:44:13.967 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.963+0000 7f6909e27700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f69040bf440 0x7f69040bd540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44076/0 (socket says 172.21.15.77:44076) 2022-04-03T19:44:13.967 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.963+0000 7f6909e27700 1 -- 172.21.15.77:0/2485307739 learned_addr learned my addr 172.21.15.77:0/2485307739 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:13.968 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.963+0000 7f690a628700 1 --2- 172.21.15.77:0/2485307739 >> v2:172.21.15.186:3300/0 conn(0x7f6904067930 0x7f69041568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:13.968 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.963+0000 7f6908e25700 1 -- 172.21.15.77:0/2485307739 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080204667 0 0) 0x7f69040c4860 con 0x7f69040bf440 2022-04-03T19:44:13.968 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.963+0000 7f6908e25700 1 -- 172.21.15.77:0/2485307739 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68ec0033f0 con 0x7f69040bf440 2022-04-03T19:44:13.968 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.963+0000 7f690a628700 1 -- 172.21.15.77:0/2485307739 >> v1:172.21.15.186:6789/0 conn(0x7f69040bf440 legacy=0x7f69040bd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:13.969 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.963+0000 7f690a628700 1 -- 172.21.15.77:0/2485307739 >> v1:172.21.15.77:6789/0 conn(0x7f69040bf810 legacy=0x7f69040c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:13.969 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.964+0000 7f690a628700 1 -- 172.21.15.77:0/2485307739 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69040680c0 con 0x7f6904067930 2022-04-03T19:44:13.969 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.964+0000 7f690a628700 1 --2- 172.21.15.77:0/2485307739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6904067930 0x7f69041568f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f68fc009e60 tx=0x7f68fc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f23e173dcbde920 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:13.969 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.964+0000 7f6908e25700 1 -- 172.21.15.77:0/2485307739 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc00e070 con 0x7f6904067930 2022-04-03T19:44:13.969 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.964+0000 7f6908e25700 1 -- 172.21.15.77:0/2485307739 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68fc00bbf0 con 0x7f6904067930 2022-04-03T19:44:13.970 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.964+0000 7f6908e25700 1 -- 172.21.15.77:0/2485307739 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc012430 con 0x7f6904067930 2022-04-03T19:44:13.970 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.965+0000 7f690ae29700 1 -- 172.21.15.77:0/2485307739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6904067930 msgr2=0x7f69041568f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:13.970 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.965+0000 7f690ae29700 1 --2- 172.21.15.77:0/2485307739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6904067930 0x7f69041568f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f68fc009e60 tx=0x7f68fc009780 comp rx=0 tx=0).stop 2022-04-03T19:44:13.970 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.965+0000 7f690ae29700 1 -- 172.21.15.77:0/2485307739 shutdown_connections 2022-04-03T19:44:13.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.965+0000 7f690ae29700 1 --2- 172.21.15.77:0/2485307739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6904067930 0x7f69041568f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:13.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.965+0000 7f690ae29700 1 -- 172.21.15.77:0/2485307739 >> 172.21.15.77:0/2485307739 conn(0x7f690406ba20 msgr2=0x7f6904068ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:13.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.965+0000 7f690ae29700 1 -- 172.21.15.77:0/2485307739 shutdown_connections 2022-04-03T19:44:13.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.965+0000 7f690ae29700 1 -- 172.21.15.77:0/2485307739 wait complete. 2022-04-03T19:44:13.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.966+0000 7f690ae29700 1 Processor -- start 2022-04-03T19:44:13.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.966+0000 7f690ae29700 1 -- start start 2022-04-03T19:44:13.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.967+0000 7f690ae29700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6904067930 0x7f6904158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:13.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.967+0000 7f690ae29700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69040bf440 0x7f690414db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:13.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.967+0000 7f690ae29700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69040bf810 0x7f690414e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:13.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.967+0000 7f690ae29700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690414fbc0 con 0x7f69040bf810 2022-04-03T19:44:13.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.967+0000 7f690ae29700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690414d700 con 0x7f69040bf440 2022-04-03T19:44:13.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.967+0000 7f690ae29700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69041589c0 con 0x7f6904067930 2022-04-03T19:44:13.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.968+0000 7f690a628700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69040bf810 0x7f690414e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:13.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.968+0000 7f6909e27700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6904067930 0x7f6904158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:13.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.968+0000 7f6909626700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69040bf440 0x7f690414db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:13.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.968+0000 7f6909e27700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6904067930 0x7f6904158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59736/0 (socket says 172.21.15.77:59736) 2022-04-03T19:44:13.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.968+0000 7f6909e27700 1 -- 172.21.15.77:0/1318745199 learned_addr learned my addr 172.21.15.77:0/1318745199 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:13.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.968+0000 7f690a628700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69040bf810 0x7f690414e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42434/0 (socket says 172.21.15.77:42434) 2022-04-03T19:44:13.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.968+0000 7f6909e27700 1 -- 172.21.15.77:0/1318745199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69040bf440 msgr2=0x7f690414db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:13.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.968+0000 7f6909e27700 1 --2- 172.21.15.77:0/1318745199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69040bf440 0x7f690414db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:13.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.968+0000 7f6909e27700 1 -- 172.21.15.77:0/1318745199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69040bf810 msgr2=0x7f690414e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:13.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.968+0000 7f6909e27700 1 --2- 172.21.15.77:0/1318745199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69040bf810 0x7f690414e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:13.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.968+0000 7f6909e27700 1 -- 172.21.15.77:0/1318745199 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68fc009a20 con 0x7f6904067930 2022-04-03T19:44:13.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.968+0000 7f6909e27700 1 --2- 172.21.15.77:0/1318745199 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6904067930 0x7f6904158480 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f690000c4c0 tx=0x7f6900007c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:13.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.969+0000 7f68f1ffb700 1 -- 172.21.15.77:0/1318745199 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690000ce30 con 0x7f6904067930 2022-04-03T19:44:13.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.969+0000 7f68f1ffb700 1 -- 172.21.15.77:0/1318745199 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69000072e0 con 0x7f6904067930 2022-04-03T19:44:13.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.969+0000 7f690ae29700 1 -- 172.21.15.77:0/1318745199 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f690414fda0 con 0x7f6904067930 2022-04-03T19:44:13.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.969+0000 7f68f1ffb700 1 -- 172.21.15.77:0/1318745199 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6900005780 con 0x7f6904067930 2022-04-03T19:44:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.969+0000 7f690ae29700 1 -- 172.21.15.77:0/1318745199 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69041537d0 con 0x7f6904067930 2022-04-03T19:44:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.970+0000 7f690ae29700 1 -- 172.21.15.77:0/1318745199 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f690404b890 con 0x7f6904067930 2022-04-03T19:44:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.971+0000 7f68f1ffb700 1 -- 172.21.15.77:0/1318745199 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f6900019030 con 0x7f6904067930 2022-04-03T19:44:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.971+0000 7f68f1ffb700 1 --2- 172.21.15.77:0/1318745199 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f68e4063ce0 0x7f68e4066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:13.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.972+0000 7f68f1ffb700 1 -- 172.21.15.77:0/1318745199 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5195+0+0 (secure 0 0 0) 0x7f690007fb30 con 0x7f6904067930 2022-04-03T19:44:13.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.972+0000 7f6909626700 1 --2- 172.21.15.77:0/1318745199 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f68e4063ce0 0x7f68e4066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:13.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.972+0000 7f6909626700 1 --2- 172.21.15.77:0/1318745199 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f68e4063ce0 0x7f68e4066190 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f68f400e4a0 tx=0x7f68f4007dd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:13.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:13.975+0000 7f68f1ffb700 1 -- 172.21.15.77:0/1318745199 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f690004ea90 con 0x7f6904067930 2022-04-03T19:44:14.181 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:14.180+0000 7f690ae29700 1 -- 172.21.15.77:0/1318745199 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f69040c84c0 con 0x7f6904067930 2022-04-03T19:44:14.467 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:44:14.473 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:44:14.748 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:14.747+0000 7f68f1ffb700 1 -- 172.21.15.77:0/1318745199 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v19) v1 ==== 173+0+0 (secure 0 0 0) 0x7f6900017070 con 0x7f6904067930 2022-04-03T19:44:14.753 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:14.751+0000 7f690ae29700 1 -- 172.21.15.77:0/1318745199 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f68e4063ce0 msgr2=0x7f68e4066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:14.753 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:14.751+0000 7f690ae29700 1 --2- 172.21.15.77:0/1318745199 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f68e4063ce0 0x7f68e4066190 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f68f400e4a0 tx=0x7f68f4007dd0 comp rx=0 tx=0).stop 2022-04-03T19:44:14.753 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:14.751+0000 7f690ae29700 1 -- 172.21.15.77:0/1318745199 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6904067930 msgr2=0x7f6904158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:14.754 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:14.751+0000 7f690ae29700 1 --2- 172.21.15.77:0/1318745199 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6904067930 0x7f6904158480 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f690000c4c0 tx=0x7f6900007c70 comp rx=0 tx=0).stop 2022-04-03T19:44:14.754 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:14.751+0000 7f690ae29700 1 -- 172.21.15.77:0/1318745199 shutdown_connections 2022-04-03T19:44:14.754 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:14.751+0000 7f690ae29700 1 --2- 172.21.15.77:0/1318745199 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69040bf810 0x7f690414e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:14.754 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:14.751+0000 7f690ae29700 1 --2- 172.21.15.77:0/1318745199 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f68e4063ce0 0x7f68e4066190 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:14.755 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:14.751+0000 7f690ae29700 1 --2- 172.21.15.77:0/1318745199 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6904067930 0x7f6904158480 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:14.755 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:14.751+0000 7f690ae29700 1 --2- 172.21.15.77:0/1318745199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69040bf440 0x7f690414db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:14.755 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:14.751+0000 7f690ae29700 1 -- 172.21.15.77:0/1318745199 >> 172.21.15.77:0/1318745199 conn(0x7f690406ba20 msgr2=0x7f69040692e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:14.755 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:14.751+0000 7f690ae29700 1 -- 172.21.15.77:0/1318745199 shutdown_connections 2022-04-03T19:44:14.756 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:14.752+0000 7f690ae29700 1 -- 172.21.15.77:0/1318745199 wait complete. 2022-04-03T19:44:14.757 INFO:teuthology.orchestra.run.smithi077.stderr:enabled application 'rbd' on pool 'rbd' 2022-04-03T19:44:14.779 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2022-04-03T19:44:14.780 INFO:tasks.ceph.mds.a:Restarting daemon 2022-04-03T19:44:14.780 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-04-03T19:44:14.785 INFO:tasks.ceph.mds.a:Started 2022-04-03T19:44:14.786 INFO:tasks.ceph.mds.c:Restarting daemon 2022-04-03T19:44:14.786 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-04-03T19:44:14.790 INFO:tasks.ceph.mds.c:Started 2022-04-03T19:44:14.791 INFO:tasks.ceph.mds.e:Restarting daemon 2022-04-03T19:44:14.791 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2022-04-03T19:44:14.795 INFO:tasks.ceph.mds.e:Started 2022-04-03T19:44:14.795 INFO:tasks.ceph.mds.b:Restarting daemon 2022-04-03T19:44:14.796 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-04-03T19:44:14.798 INFO:tasks.ceph.mds.b:Started 2022-04-03T19:44:14.799 INFO:tasks.ceph.mds.d:Restarting daemon 2022-04-03T19:44:14.799 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2022-04-03T19:44:14.801 INFO:tasks.ceph.mds.d:Started 2022-04-03T19:44:14.802 INFO:tasks.ceph.mds.f:Restarting daemon 2022-04-03T19:44:14.802 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2022-04-03T19:44:14.804 INFO:tasks.ceph.mds.f:Started 2022-04-03T19:44:14.805 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2022-04-03T19:44:14.805 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-04-03T19:44:14.878 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2022-04-03T19:44:14.879 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2022-04-03T19:44:14.881 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-03T19:44:14.946 INFO:tasks.ceph.mds.a.smithi077.stdout:starting mds.a at 2022-04-03T19:44:14.946 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2022-04-03T19:44:14.959 INFO:tasks.ceph.mds.e.smithi077.stdout:starting mds.e at 2022-04-03T19:44:15.028 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.026+0000 7f3b75312700 1 Processor -- start 2022-04-03T19:44:15.028 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.026+0000 7f3b75312700 1 -- start start 2022-04-03T19:44:15.029 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.027+0000 7f3b75312700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3b7006d3f0 0x7f3b7006d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:15.029 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.028+0000 7f3b75312700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b701562f0 con 0x7f3b700c2a90 2022-04-03T19:44:15.030 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.028+0000 7f3b75312700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b700c2f80 con 0x7f3b700c5930 2022-04-03T19:44:15.030 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.028+0000 7f3b75312700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b700c3260 con 0x7f3b7006d3f0 2022-04-03T19:44:15.030 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.028+0000 7f3b6ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3b7006d3f0 0x7f3b7006d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:15.030 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.028+0000 7f3b6ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3b7006d3f0 0x7f3b7006d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42450/0 (socket says 172.21.15.77:42450) 2022-04-03T19:44:15.030 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.028+0000 7f3b6ffff700 1 -- 172.21.15.77:0/2455301401 learned_addr learned my addr 172.21.15.77:0/2455301401 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:15.031 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.028+0000 7f3b74b11700 1 --1- 172.21.15.77:0/2455301401 >> v1:172.21.15.77:6789/0 conn(0x7f3b700c2a90 0x7f3b7006c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38274/0 (socket says 172.21.15.77:38274) 2022-04-03T19:44:15.032 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.028+0000 7f3b6ffff700 1 -- 172.21.15.77:0/2455301401 >> v1:172.21.15.186:6789/0 conn(0x7f3b700c5930 legacy=0x7f3b70155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:15.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.028+0000 7f3b6ffff700 1 -- 172.21.15.77:0/2455301401 >> v1:172.21.15.77:6789/0 conn(0x7f3b700c2a90 legacy=0x7f3b7006c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:15.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.028+0000 7f3b6ffff700 1 -- 172.21.15.77:0/2455301401 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b700c3540 con 0x7f3b7006d3f0 2022-04-03T19:44:15.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.029+0000 7f3b6ffff700 1 --2- 172.21.15.77:0/2455301401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7006d3f0 0x7f3b7006d7c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3b60009830 tx=0x7f3b600090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f92bc5216c9bdec server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:15.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.029+0000 7f3b6effd700 1 -- 172.21.15.77:0/2455301401 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b60004030 con 0x7f3b7006d3f0 2022-04-03T19:44:15.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.029+0000 7f3b6effd700 1 -- 172.21.15.77:0/2455301401 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b60004190 con 0x7f3b7006d3f0 2022-04-03T19:44:15.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.029+0000 7f3b6effd700 1 -- 172.21.15.77:0/2455301401 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b60011630 con 0x7f3b7006d3f0 2022-04-03T19:44:15.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.029+0000 7f3b75312700 1 -- 172.21.15.77:0/2455301401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7006d3f0 msgr2=0x7f3b7006d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:15.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.029+0000 7f3b75312700 1 --2- 172.21.15.77:0/2455301401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7006d3f0 0x7f3b7006d7c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3b60009830 tx=0x7f3b600090f0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.030+0000 7f3b75312700 1 -- 172.21.15.77:0/2455301401 shutdown_connections 2022-04-03T19:44:15.035 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.030+0000 7f3b75312700 1 --2- 172.21.15.77:0/2455301401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b7006d3f0 0x7f3b7006d7c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.035 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.030+0000 7f3b75312700 1 -- 172.21.15.77:0/2455301401 >> 172.21.15.77:0/2455301401 conn(0x7f3b700cb720 msgr2=0x7f3b700cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:15.035 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.030+0000 7f3b75312700 1 -- 172.21.15.77:0/2455301401 shutdown_connections 2022-04-03T19:44:15.035 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.030+0000 7f3b75312700 1 -- 172.21.15.77:0/2455301401 wait complete. 2022-04-03T19:44:15.036 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.031+0000 7f3b75312700 1 Processor -- start 2022-04-03T19:44:15.036 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.031+0000 7f3b75312700 1 -- start start 2022-04-03T19:44:15.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.032+0000 7f3b75312700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b7006d3f0 0x7f3b701695b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:15.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.032+0000 7f3b75312700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b700c2a90 0x7f3b700c5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:15.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.032+0000 7f3b75312700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b700c5930 0x7f3b700c07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:15.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.032+0000 7f3b75312700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b701663e0 con 0x7f3b700c2a90 2022-04-03T19:44:15.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.032+0000 7f3b75312700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b70169af0 con 0x7f3b700c5930 2022-04-03T19:44:15.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.032+0000 7f3b75312700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b701657a0 con 0x7f3b7006d3f0 2022-04-03T19:44:15.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.033+0000 7f3b6f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b700c2a90 0x7f3b700c5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:15.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.033+0000 7f3b6ffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b7006d3f0 0x7f3b701695b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:15.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.033+0000 7f3b74b11700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b700c5930 0x7f3b700c07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:15.041 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.033+0000 7f3b6ffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b7006d3f0 0x7f3b701695b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59744/0 (socket says 172.21.15.77:59744) 2022-04-03T19:44:15.041 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.033+0000 7f3b6ffff700 1 -- 172.21.15.77:0/1834899470 learned_addr learned my addr 172.21.15.77:0/1834899470 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:15.041 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.033+0000 7f3b6ffff700 1 -- 172.21.15.77:0/1834899470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b700c5930 msgr2=0x7f3b700c07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:15.041 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.033+0000 7f3b6ffff700 1 --2- 172.21.15.77:0/1834899470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b700c5930 0x7f3b700c07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.041 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.033+0000 7f3b6ffff700 1 -- 172.21.15.77:0/1834899470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b700c2a90 msgr2=0x7f3b700c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:15.042 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.033+0000 7f3b6ffff700 1 --2- 172.21.15.77:0/1834899470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b700c2a90 0x7f3b700c5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.042 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.033+0000 7f3b6ffff700 1 -- 172.21.15.77:0/1834899470 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b60009370 con 0x7f3b7006d3f0 2022-04-03T19:44:15.042 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.033+0000 7f3b6ffff700 1 --2- 172.21.15.77:0/1834899470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b7006d3f0 0x7f3b701695b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3b60004000 tx=0x7f3b60016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:15.042 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.033+0000 7f3b57fff700 1 -- 172.21.15.77:0/1834899470 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b60003840 con 0x7f3b7006d3f0 2022-04-03T19:44:15.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.034+0000 7f3b57fff700 1 -- 172.21.15.77:0/1834899470 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b600039c0 con 0x7f3b7006d3f0 2022-04-03T19:44:15.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.034+0000 7f3b75312700 1 -- 172.21.15.77:0/1834899470 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b70165a40 con 0x7f3b7006d3f0 2022-04-03T19:44:15.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.034+0000 7f3b57fff700 1 -- 172.21.15.77:0/1834899470 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b60003d80 con 0x7f3b7006d3f0 2022-04-03T19:44:15.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.034+0000 7f3b75312700 1 -- 172.21.15.77:0/1834899470 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b70166000 con 0x7f3b7006d3f0 2022-04-03T19:44:15.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.035+0000 7f3b75312700 1 -- 172.21.15.77:0/1834899470 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b7004b050 con 0x7f3b7006d3f0 2022-04-03T19:44:15.044 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.035+0000 7f3b57fff700 1 -- 172.21.15.77:0/1834899470 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f3b6002b030 con 0x7f3b7006d3f0 2022-04-03T19:44:15.044 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.036+0000 7f3b57fff700 1 --2- 172.21.15.77:0/1834899470 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3b50063d10 0x7f3b500661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:15.044 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.036+0000 7f3b57fff700 1 -- 172.21.15.77:0/1834899470 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f3b60081c50 con 0x7f3b7006d3f0 2022-04-03T19:44:15.044 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.036+0000 7f3b6f7fe700 1 --2- 172.21.15.77:0/1834899470 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3b50063d10 0x7f3b500661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:15.045 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.037+0000 7f3b6f7fe700 1 --2- 172.21.15.77:0/1834899470 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3b50063d10 0x7f3b500661c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3b701598d0 tx=0x7f3b64009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:15.045 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.040+0000 7f3b57fff700 1 -- 172.21.15.77:0/1834899470 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f3b60050c40 con 0x7f3b7006d3f0 2022-04-03T19:44:15.237 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.235+0000 7f3b75312700 1 -- 172.21.15.77:0/1834899470 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b7006e1e0 con 0x7f3b7006d3f0 2022-04-03T19:44:15.238 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.236+0000 7f3b57fff700 1 -- 172.21.15.77:0/1834899470 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+13639 (secure 0 0 0) 0x7f3b60054260 con 0x7f3b7006d3f0 2022-04-03T19:44:15.238 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:15.239 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":19,"fsid":"89c3a3f4-599d-4c4a-8f5a-7180eafb7292","created":"2022-04-03T19:43:54.101559+0000","modified":"2022-04-03T19:44:14.735099+0000","last_up_change":"2022-04-03T19:44:05.589174+0000","last_in_change":"2022-04-03T19:43:56.729760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-03T19:44:07.567638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-04-03T19:44:13.149481+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f35f5195-3729-4d78-85d1-0539f259283f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6823","nonce":20252},{"type":"v1","addr":"172.21.15.77:6825","nonce":20252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":20252},{"type":"v1","addr":"172.21.15.77:6827","nonce":20252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":20252},{"type":"v1","addr":"172.21.15.77:6831","nonce":20252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":20252},{"type":"v1","addr":"172.21.15.77:6829","nonce":20252}]},"public_addr":"172.21.15.77:6825/20252","cluster_addr":"172.21.15.77:6827/20252","heartbeat_back_addr":"172.21.15.77:6831/20252","heartbeat_front_addr":"172.21.15.77:6829/20252","state":["exists","up"]},{"osd":1,"uuid":"ab1c89dc-57ce-4cc4-b8c7-9b0c19939601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":20249},{"type":"v1","addr":"172.21.15.77:6805","nonce":20249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":20249},{"type":"v1","addr":"172.21.15.77:6809","nonce":20249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":20249},{"type":"v1","addr":"172.21.15.77:6819","nonce":20249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":20249},{"type":"v1","addr":"172.21.15.77:6815","nonce":20249}]},"public_addr":"172.21.15.77:6805/20249","cluster_addr":"172.21.15.77:6809/20249","heartbeat_back_addr":"172.21.15.77:6819/20249","heartbeat_front_addr":"172.21.15.77:6815/20249","state":["exists","up"]},{"osd":2,"uuid":"351e218b-7749-4ee9-b44a-0a5b52fb9be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":20248},{"type":"v1","addr":"172.21.15.77:6801","nonce":20248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":20248},{"type":"v1","addr":"172.21.15.77:6803","nonce":20248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":20248},{"type":"v1","addr":"172.21.15.77:6813","nonce":20248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":20248},{"type":"v1","addr":"172.21.15.77:6808","nonce":20248}]},"public_addr":"172.21.15.77:6801/20248","cluster_addr":"172.21.15.77:6803/20248","heartbeat_back_addr":"172.21.15.77:6813/20248","heartbeat_front_addr":"172.21.15.77:6808/20248","state":["exists","up"]},{"osd":3,"uuid":"1ba6ec8a-7cbf-4c12-8e21-c3097a263ae8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":20250},{"type":"v1","addr":"172.21.15.77:6814","nonce":20250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":20250},{"type":"v1","addr":"172.21.15.77:6818","nonce":20250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":20250},{"type":"v1","addr":"172.21.15.77:6824","nonce":20250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":20250},{"type":"v1","addr":"172.21.15.77:6821","nonce":20250}]},"public_addr":"172.21.15.77:6814/20250","cluster_addr":"172.21.15.77:6818/20250","heartbeat_back_addr":"172.21.15.77:6824/20250","heartbeat_front_addr":"172.21.15.77:6821/20250","state":["exists","up"]},{"osd":4,"uuid":"3faf4c09-846c-4133-856b-7f53c535a67d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":25551},{"type":"v1","addr":"172.21.15.186:6822","nonce":25551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":25551},{"type":"v1","addr":"172.21.15.186:6826","nonce":25551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":25551},{"type":"v1","addr":"172.21.15.186:6831","nonce":25551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":25551},{"type":"v1","addr":"172.21.15.186:6829","nonce":25551}]},"public_addr":"172.21.15.186:6822/25551","cluster_addr":"172.21.15.186:6826/25551","heartbeat_back_addr":"172.21.15.186:6831/25551","heartbeat_front_addr":"172.21.15.186:6829/25551","state":["exists","up"]},{"osd":5,"uuid":"ffe41d00-d0aa-42f4-82c1-5cced13c1bc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":25541},{"type":"v1","addr":"172.21.15.186:6809","nonce":25541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":25541},{"type":"v1","addr":"172.21.15.186:6811","nonce":25541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":25541},{"type":"v1","addr":"172.21.15.186:6815","nonce":25541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":25541},{"type":"v1","addr":"172.21.15.186:6813","nonce":25541}]},"public_addr":"172.21.15.186:6809/25541","cluster_addr":"172.21.15.186:6811/25541","heartbeat_back_addr":"172.21.15.186:6815/25541","heartbeat_front_addr":"172.21.15.186:6813/25541","state":["exists","up"]},{"osd":6,"uuid":"b0344ee8-deca-4e9b-934c-5866d90d0073","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":25542},{"type":"v1","addr":"172.21.15.186:6801","nonce":25542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":25542},{"type":"v1","addr":"172.21.15.186:6803","nonce":25542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":25542},{"type":"v1","addr":"172.21.15.186:6807","nonce":25542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":25542},{"type":"v1","addr":"172.21.15.186:6805","nonce":25542}]},"public_addr":"172.21.15.186:6801/25542","cluster_addr":"172.21.15.186:6803/25542","heartbeat_back_addr":"172.21.15.186:6807/25542","heartbeat_front_addr":"172.21.15.186:6805/25542","state":["exists","up"]},{"osd":7,"uuid":"0eb0e5d3-3dad-4f49-9b50-3b24a3924bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":25572},{"type":"v1","addr":"172.21.15.186:6817","nonce":25572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":25572},{"type":"v1","addr":"172.21.15.186:6819","nonce":25572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6825","nonce":25572},{"type":"v1","addr":"172.21.15.186:6827","nonce":25572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":25572},{"type":"v1","addr":"172.21.15.186:6823","nonce":25572}]},"public_addr":"172.21.15.186:6817/25572","cluster_addr":"172.21.15.186:6819/25572","heartbeat_back_addr":"172.21.15.186:6827/25572","heartbeat_front_addr":"172.21.15.186:6823/25572","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.143602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.490366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.361292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.343544+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.905429+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969298+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.949524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-03T19:44:15.241 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.239+0000 7f3b75312700 1 -- 172.21.15.77:0/1834899470 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3b50063d10 msgr2=0x7f3b500661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:15.241 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.239+0000 7f3b75312700 1 --2- 172.21.15.77:0/1834899470 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3b50063d10 0x7f3b500661c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3b701598d0 tx=0x7f3b64009040 comp rx=0 tx=0).stop 2022-04-03T19:44:15.241 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.239+0000 7f3b75312700 1 -- 172.21.15.77:0/1834899470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b7006d3f0 msgr2=0x7f3b701695b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:15.241 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.239+0000 7f3b75312700 1 --2- 172.21.15.77:0/1834899470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b7006d3f0 0x7f3b701695b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3b60004000 tx=0x7f3b60016000 comp rx=0 tx=0).stop 2022-04-03T19:44:15.242 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.239+0000 7f3b75312700 1 -- 172.21.15.77:0/1834899470 shutdown_connections 2022-04-03T19:44:15.242 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.239+0000 7f3b75312700 1 --2- 172.21.15.77:0/1834899470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b700c5930 0x7f3b700c07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.242 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.239+0000 7f3b75312700 1 --2- 172.21.15.77:0/1834899470 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3b50063d10 0x7f3b500661c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.242 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.239+0000 7f3b75312700 1 --2- 172.21.15.77:0/1834899470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b7006d3f0 0x7f3b701695b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.243 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.239+0000 7f3b75312700 1 --2- 172.21.15.77:0/1834899470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b700c2a90 0x7f3b700c5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.243 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.239+0000 7f3b75312700 1 -- 172.21.15.77:0/1834899470 >> 172.21.15.77:0/1834899470 conn(0x7f3b700cb720 msgr2=0x7f3b701531f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:15.243 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.239+0000 7f3b75312700 1 -- 172.21.15.77:0/1834899470 shutdown_connections 2022-04-03T19:44:15.243 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.239+0000 7f3b75312700 1 -- 172.21.15.77:0/1834899470 wait complete. 2022-04-03T19:44:15.264 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-04-03T19:44:07.567638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-04-03T19:44:13.149481+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-04-03T19:44:15.265 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-04-03T19:44:15.465 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.464+0000 7fd84e3fc700 1 Processor -- start 2022-04-03T19:44:15.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.464+0000 7fd84e3fc700 1 -- start start 2022-04-03T19:44:15.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.465+0000 7fd84e3fc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8480c1960 0x7fd848156010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:15.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.465+0000 7fd84e3fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd84806a7a0 0x7fd84806ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:15.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.465+0000 7fd84e3fc700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd848062dd0 con 0x7fd8480c2470 2022-04-03T19:44:15.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.465+0000 7fd84e3fc700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd84814fb90 con 0x7fd84806a7a0 2022-04-03T19:44:15.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.465+0000 7fd84e3fc700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd848156550 con 0x7fd8480c1960 2022-04-03T19:44:15.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.465+0000 7fd84dbfb700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd84806a7a0 0x7fd84806ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:15.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.465+0000 7fd84d3fa700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8480c1960 0x7fd848156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:15.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.465+0000 7fd84dbfb700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd84806a7a0 0x7fd84806ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42454/0 (socket says 172.21.15.77:42454) 2022-04-03T19:44:15.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.465+0000 7fd84dbfb700 1 -- 172.21.15.77:0/3236161695 learned_addr learned my addr 172.21.15.77:0/3236161695 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:15.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.465+0000 7fd84cbf9700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd8480c2470 0x7fd8480c45d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38276/0 (socket says 172.21.15.77:38276) 2022-04-03T19:44:15.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.465+0000 7fd84d3fa700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8480c1960 0x7fd848156010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59746/0 (socket says 172.21.15.77:59746) 2022-04-03T19:44:15.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.466+0000 7fd84dbfb700 1 -- 172.21.15.77:0/3236161695 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8480c1960 msgr2=0x7fd848156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:15.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.466+0000 7fd84dbfb700 1 --2- 172.21.15.77:0/3236161695 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8480c1960 0x7fd848156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.466+0000 7fd84dbfb700 1 -- 172.21.15.77:0/3236161695 >> v1:172.21.15.77:6789/0 conn(0x7fd8480c2470 legacy=0x7fd8480c45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:15.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.466+0000 7fd84dbfb700 1 -- 172.21.15.77:0/3236161695 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84806a310 con 0x7fd84806a7a0 2022-04-03T19:44:15.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.466+0000 7fd84d3fa700 1 --2- 172.21.15.77:0/3236161695 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8480c1960 0x7fd848156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:15.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.466+0000 7fd84dbfb700 1 --2- 172.21.15.77:0/3236161695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd84806a7a0 0x7fd84806ac40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd840009f20 tx=0x7fd840009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f32b7c178b28aae server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:15.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.466+0000 7fd837fff700 1 -- 172.21.15.77:0/3236161695 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84000e070 con 0x7fd84806a7a0 2022-04-03T19:44:15.472 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.466+0000 7fd837fff700 1 -- 172.21.15.77:0/3236161695 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd84000ba60 con 0x7fd84806a7a0 2022-04-03T19:44:15.472 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.466+0000 7fd837fff700 1 -- 172.21.15.77:0/3236161695 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8400092e0 con 0x7fd84806a7a0 2022-04-03T19:44:15.475 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.467+0000 7fd84e3fc700 1 -- 172.21.15.77:0/3236161695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd84806a7a0 msgr2=0x7fd84806ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:15.475 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.467+0000 7fd84e3fc700 1 --2- 172.21.15.77:0/3236161695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd84806a7a0 0x7fd84806ac40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd840009f20 tx=0x7fd840009840 comp rx=0 tx=0).stop 2022-04-03T19:44:15.475 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.467+0000 7fd84e3fc700 1 -- 172.21.15.77:0/3236161695 shutdown_connections 2022-04-03T19:44:15.475 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.467+0000 7fd84e3fc700 1 --2- 172.21.15.77:0/3236161695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd84806a7a0 0x7fd84806ac40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.476 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.467+0000 7fd84e3fc700 1 --2- 172.21.15.77:0/3236161695 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8480c1960 0x7fd848156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.476 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.467+0000 7fd84e3fc700 1 -- 172.21.15.77:0/3236161695 >> 172.21.15.77:0/3236161695 conn(0x7fd848067750 msgr2=0x7fd848067b50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:15.476 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.467+0000 7fd84e3fc700 1 -- 172.21.15.77:0/3236161695 shutdown_connections 2022-04-03T19:44:15.476 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.467+0000 7fd84e3fc700 1 -- 172.21.15.77:0/3236161695 wait complete. 2022-04-03T19:44:15.476 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.468+0000 7fd84e3fc700 1 Processor -- start 2022-04-03T19:44:15.477 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.469+0000 7fd84e3fc700 1 -- start start 2022-04-03T19:44:15.477 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.469+0000 7fd84e3fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd84806a7a0 0x7fd8480c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:15.477 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.469+0000 7fd84e3fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8480c1960 0x7fd848152460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:15.477 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.469+0000 7fd84e3fc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8480c2470 0x7fd8480c7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:15.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.469+0000 7fd84e3fc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8480c7490 con 0x7fd8480c1960 2022-04-03T19:44:15.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.469+0000 7fd84e3fc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd848152dd0 con 0x7fd84806a7a0 2022-04-03T19:44:15.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.469+0000 7fd84e3fc700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8481518e0 con 0x7fd8480c2470 2022-04-03T19:44:15.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.470+0000 7fd84cbf9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8480c1960 0x7fd848152460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:15.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.470+0000 7fd84cbf9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8480c1960 0x7fd848152460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42456/0 (socket says 172.21.15.77:42456) 2022-04-03T19:44:15.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.470+0000 7fd84cbf9700 1 -- 172.21.15.77:0/436537390 learned_addr learned my addr 172.21.15.77:0/436537390 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:15.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.470+0000 7fd84d3fa700 1 --2- 172.21.15.77:0/436537390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd84806a7a0 0x7fd8480c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:15.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.470+0000 7fd84dbfb700 1 --2- 172.21.15.77:0/436537390 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8480c2470 0x7fd8480c7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:15.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.470+0000 7fd84cbf9700 1 -- 172.21.15.77:0/436537390 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8480c2470 msgr2=0x7fd8480c7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:15.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.470+0000 7fd84cbf9700 1 --2- 172.21.15.77:0/436537390 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8480c2470 0x7fd8480c7920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.470+0000 7fd84cbf9700 1 -- 172.21.15.77:0/436537390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd84806a7a0 msgr2=0x7fd8480c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:15.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.470+0000 7fd84cbf9700 1 --2- 172.21.15.77:0/436537390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd84806a7a0 0x7fd8480c57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.470+0000 7fd84cbf9700 1 -- 172.21.15.77:0/436537390 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8440092a0 con 0x7fd8480c1960 2022-04-03T19:44:15.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.470+0000 7fd84cbf9700 1 --2- 172.21.15.77:0/436537390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8480c1960 0x7fd848152460 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd83800c470 tx=0x7fd838007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:15.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.471+0000 7fd8357fa700 1 -- 172.21.15.77:0/436537390 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8380070d0 con 0x7fd8480c1960 2022-04-03T19:44:15.481 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.471+0000 7fd8357fa700 1 -- 172.21.15.77:0/436537390 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd838007230 con 0x7fd8480c1960 2022-04-03T19:44:15.481 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.471+0000 7fd8357fa700 1 -- 172.21.15.77:0/436537390 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8380075d0 con 0x7fd8480c1960 2022-04-03T19:44:15.481 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.471+0000 7fd84e3fc700 1 -- 172.21.15.77:0/436537390 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd840009ae0 con 0x7fd8480c1960 2022-04-03T19:44:15.481 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.471+0000 7fd84e3fc700 1 -- 172.21.15.77:0/436537390 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd84814ffa0 con 0x7fd8480c1960 2022-04-03T19:44:15.481 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.472+0000 7fd84e3fc700 1 -- 172.21.15.77:0/436537390 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd848070690 con 0x7fd8480c1960 2022-04-03T19:44:15.482 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.473+0000 7fd8357fa700 1 -- 172.21.15.77:0/436537390 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fd838010070 con 0x7fd8480c1960 2022-04-03T19:44:15.482 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.474+0000 7fd8357fa700 1 --2- 172.21.15.77:0/436537390 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd828063ce0 0x7fd828066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:15.482 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.474+0000 7fd8357fa700 1 -- 172.21.15.77:0/436537390 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fd83807f540 con 0x7fd8480c1960 2022-04-03T19:44:15.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.474+0000 7fd84d3fa700 1 --2- 172.21.15.77:0/436537390 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd828063ce0 0x7fd828066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:15.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.475+0000 7fd84d3fa700 1 --2- 172.21.15.77:0/436537390 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd828063ce0 0x7fd828066190 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd844009dc0 tx=0x7fd844009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:15.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.479+0000 7fd8357fa700 1 -- 172.21.15.77:0/436537390 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fd838048ae0 con 0x7fd8480c1960 2022-04-03T19:44:15.674 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.672+0000 7fd84e3fc700 1 -- 172.21.15.77:0/436537390 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd84806b2a0 con 0x7fd8480c1960 2022-04-03T19:44:15.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.673+0000 7fd8357fa700 1 -- 172.21.15.77:0/436537390 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd83804e320 con 0x7fd8480c1960 2022-04-03T19:44:15.675 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2022-04-03T19:44:15.677 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.676+0000 7fd84e3fc700 1 -- 172.21.15.77:0/436537390 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd828063ce0 msgr2=0x7fd828066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:15.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.676+0000 7fd84e3fc700 1 --2- 172.21.15.77:0/436537390 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd828063ce0 0x7fd828066190 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd844009dc0 tx=0x7fd844009060 comp rx=0 tx=0).stop 2022-04-03T19:44:15.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.676+0000 7fd84e3fc700 1 -- 172.21.15.77:0/436537390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8480c1960 msgr2=0x7fd848152460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:15.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.676+0000 7fd84e3fc700 1 --2- 172.21.15.77:0/436537390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8480c1960 0x7fd848152460 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd83800c470 tx=0x7fd838007dc0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.676+0000 7fd84e3fc700 1 -- 172.21.15.77:0/436537390 shutdown_connections 2022-04-03T19:44:15.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.676+0000 7fd84e3fc700 1 --2- 172.21.15.77:0/436537390 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd828063ce0 0x7fd828066190 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.676+0000 7fd84e3fc700 1 --2- 172.21.15.77:0/436537390 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8480c2470 0x7fd8480c7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.676+0000 7fd84e3fc700 1 --2- 172.21.15.77:0/436537390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd84806a7a0 0x7fd8480c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.676+0000 7fd84e3fc700 1 --2- 172.21.15.77:0/436537390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8480c1960 0x7fd848152460 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.680 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.676+0000 7fd84e3fc700 1 -- 172.21.15.77:0/436537390 >> 172.21.15.77:0/436537390 conn(0x7fd848067750 msgr2=0x7fd84806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:15.680 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.676+0000 7fd84e3fc700 1 -- 172.21.15.77:0/436537390 shutdown_connections 2022-04-03T19:44:15.680 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.676+0000 7fd84e3fc700 1 -- 172.21.15.77:0/436537390 wait complete. 2022-04-03T19:44:15.701 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-04-03T19:44:15.907 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.905+0000 7f6bd3fff700 1 Processor -- start 2022-04-03T19:44:15.907 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.905+0000 7f6bd3fff700 1 -- start start 2022-04-03T19:44:15.909 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.907+0000 7f6bd3fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6bd40c5680 0x7f6bd40c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:15.910 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.907+0000 7f6bd3fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bd4158e10 0x7f6bd414ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:15.910 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.907+0000 7f6bd3fff700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd40c7e80 con 0x7f6bd40c52b0 2022-04-03T19:44:15.910 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.907+0000 7f6bd3fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd40c4260 con 0x7f6bd40c5680 2022-04-03T19:44:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.907+0000 7f6bd3fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd40c8a20 con 0x7f6bd4158e10 2022-04-03T19:44:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.907+0000 7f6bd37fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bd4158e10 0x7f6bd414ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.907+0000 7f6bd2ffd700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f6bd40c52b0 0x7f6bd40c8310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44082/0 (socket says 172.21.15.77:44082) 2022-04-03T19:44:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.907+0000 7f6bd2ffd700 1 -- 172.21.15.77:0/1117886561 learned_addr learned my addr 172.21.15.77:0/1117886561 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:15.912 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.908+0000 7f6bd27fc700 1 --2- 172.21.15.77:0/1117886561 >> v2:172.21.15.77:3300/0 conn(0x7f6bd40c5680 0x7f6bd40c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:15.912 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.908+0000 7f6bd1ffb700 1 -- 172.21.15.77:0/1117886561 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036677361 0 0) 0x7f6bd40c7e80 con 0x7f6bd40c52b0 2022-04-03T19:44:15.912 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.908+0000 7f6bd1ffb700 1 -- 172.21.15.77:0/1117886561 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bc00033f0 con 0x7f6bd40c52b0 2022-04-03T19:44:15.912 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.908+0000 7f6bd37fe700 1 -- 172.21.15.77:0/1117886561 >> v2:172.21.15.77:3300/0 conn(0x7f6bd40c5680 msgr2=0x7f6bd40c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:15.913 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.908+0000 7f6bd37fe700 1 --2- 172.21.15.77:0/1117886561 >> v2:172.21.15.77:3300/0 conn(0x7f6bd40c5680 0x7f6bd40c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.913 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.908+0000 7f6bd37fe700 1 -- 172.21.15.77:0/1117886561 >> v1:172.21.15.186:6789/0 conn(0x7f6bd40c52b0 legacy=0x7f6bd40c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:15.913 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.908+0000 7f6bd37fe700 1 -- 172.21.15.77:0/1117886561 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd40c6ea0 con 0x7f6bd4158e10 2022-04-03T19:44:15.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.909+0000 7f6bd37fe700 1 --2- 172.21.15.77:0/1117886561 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bd4158e10 0x7f6bd414ed50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc009e60 tx=0x7f6bcc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad570d70a70d3303 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:15.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.909+0000 7f6bd1ffb700 1 -- 172.21.15.77:0/1117886561 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc00e070 con 0x7f6bd4158e10 2022-04-03T19:44:15.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.909+0000 7f6bd1ffb700 1 -- 172.21.15.77:0/1117886561 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bcc00bb40 con 0x7f6bd4158e10 2022-04-03T19:44:15.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.909+0000 7f6bd1ffb700 1 -- 172.21.15.77:0/1117886561 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc012430 con 0x7f6bd4158e10 2022-04-03T19:44:15.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.910+0000 7f6bd3fff700 1 -- 172.21.15.77:0/1117886561 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bd4158e10 msgr2=0x7f6bd414ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:15.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.910+0000 7f6bd3fff700 1 --2- 172.21.15.77:0/1117886561 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bd4158e10 0x7f6bd414ed50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc009e60 tx=0x7f6bcc009780 comp rx=0 tx=0).stop 2022-04-03T19:44:15.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.911+0000 7f6bd3fff700 1 -- 172.21.15.77:0/1117886561 shutdown_connections 2022-04-03T19:44:15.920 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.911+0000 7f6bd3fff700 1 --2- 172.21.15.77:0/1117886561 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bd4158e10 0x7f6bd414ed50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.920 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.911+0000 7f6bd3fff700 1 --2- 172.21.15.77:0/1117886561 >> v2:172.21.15.77:3300/0 conn(0x7f6bd40c5680 0x7f6bd40c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.920 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.911+0000 7f6bd3fff700 1 -- 172.21.15.77:0/1117886561 >> 172.21.15.77:0/1117886561 conn(0x7f6bd40626d0 msgr2=0x7f6bd4062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:15.920 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.911+0000 7f6bd3fff700 1 -- 172.21.15.77:0/1117886561 shutdown_connections 2022-04-03T19:44:15.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.911+0000 7f6bd3fff700 1 -- 172.21.15.77:0/1117886561 wait complete. 2022-04-03T19:44:15.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.913+0000 7f6bd3fff700 1 Processor -- start 2022-04-03T19:44:15.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.913+0000 7f6bd3fff700 1 -- start start 2022-04-03T19:44:15.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.914+0000 7f6bd3fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bd40c52b0 0x7f6bd4148540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:15.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.914+0000 7f6bd3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bd40c5680 0x7f6bd4154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:15.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.914+0000 7f6bd3fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bd4158e10 0x7f6bd4154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:15.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.914+0000 7f6bd3fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd41491f0 con 0x7f6bd40c5680 2022-04-03T19:44:15.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.914+0000 7f6bd3fff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd4149480 con 0x7f6bd4158e10 2022-04-03T19:44:15.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.914+0000 7f6bd3fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd4149760 con 0x7f6bd40c52b0 2022-04-03T19:44:15.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.914+0000 7f6bd2ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bd40c52b0 0x7f6bd4148540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:15.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.914+0000 7f6bd37fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bd4158e10 0x7f6bd4154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:15.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.914+0000 7f6bd2ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bd40c52b0 0x7f6bd4148540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59752/0 (socket says 172.21.15.77:59752) 2022-04-03T19:44:15.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.914+0000 7f6bd2ffd700 1 -- 172.21.15.77:0/3415985114 learned_addr learned my addr 172.21.15.77:0/3415985114 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:15.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.914+0000 7f6bd27fc700 1 --2- 172.21.15.77:0/3415985114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bd40c5680 0x7f6bd4154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:15.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.915+0000 7f6bd2ffd700 1 -- 172.21.15.77:0/3415985114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bd4158e10 msgr2=0x7f6bd4154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:15.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.915+0000 7f6bd2ffd700 1 --2- 172.21.15.77:0/3415985114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bd4158e10 0x7f6bd4154980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.915+0000 7f6bd2ffd700 1 -- 172.21.15.77:0/3415985114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bd40c5680 msgr2=0x7f6bd4154440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:15.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.915+0000 7f6bd2ffd700 1 --2- 172.21.15.77:0/3415985114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bd40c5680 0x7f6bd4154440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:15.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.915+0000 7f6bd2ffd700 1 -- 172.21.15.77:0/3415985114 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc009a20 con 0x7f6bd40c52b0 2022-04-03T19:44:15.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.915+0000 7f6bd37fe700 1 --2- 172.21.15.77:0/3415985114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bd4158e10 0x7f6bd4154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:15.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.915+0000 7f6bd2ffd700 1 --2- 172.21.15.77:0/3415985114 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bd40c52b0 0x7f6bd4148540 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6bc400c490 tx=0x7f6bc4007c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:15.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.915+0000 7f6bbaffd700 1 -- 172.21.15.77:0/3415985114 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc400ce00 con 0x7f6bd40c52b0 2022-04-03T19:44:15.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.915+0000 7f6bbaffd700 1 -- 172.21.15.77:0/3415985114 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc4007360 con 0x7f6bd40c52b0 2022-04-03T19:44:15.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.915+0000 7f6bd3fff700 1 -- 172.21.15.77:0/3415985114 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd4149a40 con 0x7f6bd40c52b0 2022-04-03T19:44:15.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.915+0000 7f6bbaffd700 1 -- 172.21.15.77:0/3415985114 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc40046b0 con 0x7f6bd40c52b0 2022-04-03T19:44:15.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.915+0000 7f6bd3fff700 1 -- 172.21.15.77:0/3415985114 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd414a000 con 0x7f6bd40c52b0 2022-04-03T19:44:15.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.916+0000 7f6bd3fff700 1 -- 172.21.15.77:0/3415985114 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd406be60 con 0x7f6bd40c52b0 2022-04-03T19:44:15.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.917+0000 7f6bbaffd700 1 -- 172.21.15.77:0/3415985114 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f6bc4010030 con 0x7f6bd40c52b0 2022-04-03T19:44:15.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.918+0000 7f6bbaffd700 1 --2- 172.21.15.77:0/3415985114 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6bb4063ce0 0x7f6bb4066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:15.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.918+0000 7f6bbaffd700 1 -- 172.21.15.77:0/3415985114 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f6bc407ff70 con 0x7f6bd40c52b0 2022-04-03T19:44:15.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.918+0000 7f6bd27fc700 1 --2- 172.21.15.77:0/3415985114 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6bb4063ce0 0x7f6bb4066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:15.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.919+0000 7f6bd27fc700 1 --2- 172.21.15.77:0/3415985114 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6bb4063ce0 0x7f6bb4066190 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40c3ad0 tx=0x7f6bc8009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:15.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:15.922+0000 7f6bbaffd700 1 -- 172.21.15.77:0/3415985114 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f6bc404ef20 con 0x7f6bd40c52b0 2022-04-03T19:44:16.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.117+0000 7f6bd3fff700 1 -- 172.21.15.77:0/3415985114 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f6bd404b050 con 0x7f6bd40c52b0 2022-04-03T19:44:16.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.119+0000 7f6bbaffd700 1 -- 172.21.15.77:0/3415985114 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6bc4052540 con 0x7f6bd40c52b0 2022-04-03T19:44:16.120 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2022-04-03T19:44:16.122 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.120+0000 7f6bd3fff700 1 -- 172.21.15.77:0/3415985114 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6bb4063ce0 msgr2=0x7f6bb4066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:16.123 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.120+0000 7f6bd3fff700 1 --2- 172.21.15.77:0/3415985114 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6bb4063ce0 0x7f6bb4066190 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40c3ad0 tx=0x7f6bc8009060 comp rx=0 tx=0).stop 2022-04-03T19:44:16.123 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.120+0000 7f6bd3fff700 1 -- 172.21.15.77:0/3415985114 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bd40c52b0 msgr2=0x7f6bd4148540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:16.123 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.120+0000 7f6bd3fff700 1 --2- 172.21.15.77:0/3415985114 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bd40c52b0 0x7f6bd4148540 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6bc400c490 tx=0x7f6bc4007c60 comp rx=0 tx=0).stop 2022-04-03T19:44:16.123 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.120+0000 7f6bd3fff700 1 -- 172.21.15.77:0/3415985114 shutdown_connections 2022-04-03T19:44:16.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.120+0000 7f6bd3fff700 1 --2- 172.21.15.77:0/3415985114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bd4158e10 0x7f6bd4154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:16.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.120+0000 7f6bd3fff700 1 --2- 172.21.15.77:0/3415985114 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6bb4063ce0 0x7f6bb4066190 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:16.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.120+0000 7f6bd3fff700 1 --2- 172.21.15.77:0/3415985114 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bd40c52b0 0x7f6bd4148540 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:16.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.120+0000 7f6bd3fff700 1 --2- 172.21.15.77:0/3415985114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bd40c5680 0x7f6bd4154440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:16.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.120+0000 7f6bd3fff700 1 -- 172.21.15.77:0/3415985114 >> 172.21.15.77:0/3415985114 conn(0x7f6bd40626d0 msgr2=0x7f6bd414bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:16.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.120+0000 7f6bd3fff700 1 -- 172.21.15.77:0/3415985114 shutdown_connections 2022-04-03T19:44:16.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.120+0000 7f6bd3fff700 1 -- 172.21.15.77:0/3415985114 wait complete. 2022-04-03T19:44:16.146 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:44:16.345 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.343+0000 7fcf5b4f4700 1 Processor -- start 2022-04-03T19:44:16.345 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.343+0000 7fcf5b4f4700 1 -- start start 2022-04-03T19:44:16.347 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.344+0000 7fcf5b4f4700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf540c5930 0x7fcf54155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:16.348 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.346+0000 7fcf5b4f4700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcf54156120 0x7fcf5406c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:16.348 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.346+0000 7fcf5b4f4700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf540c2f80 con 0x7fcf5406d3f0 2022-04-03T19:44:16.348 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.346+0000 7fcf5b4f4700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf540c3260 con 0x7fcf54156120 2022-04-03T19:44:16.348 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.346+0000 7fcf5b4f4700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf540c3540 con 0x7fcf540c5930 2022-04-03T19:44:16.348 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.346+0000 7fcf5a4f2700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fcf5406d3f0 0x7fcf5406d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38278/0 (socket says 172.21.15.77:38278) 2022-04-03T19:44:16.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.346+0000 7fcf5a4f2700 1 -- 172.21.15.77:0/738860556 learned_addr learned my addr 172.21.15.77:0/738860556 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:16.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.346+0000 7fcf5acf3700 1 --2- 172.21.15.77:0/738860556 >> v2:172.21.15.186:3300/0 conn(0x7fcf54156120 0x7fcf5406c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:16.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.346+0000 7fcf59cf1700 1 --2- 172.21.15.77:0/738860556 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf540c5930 0x7fcf54155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:16.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.346+0000 7fcf594f0700 1 -- 172.21.15.77:0/738860556 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058615997 0 0) 0x7fcf540c2f80 con 0x7fcf5406d3f0 2022-04-03T19:44:16.350 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.346+0000 7fcf594f0700 1 -- 172.21.15.77:0/738860556 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf3c0033f0 con 0x7fcf5406d3f0 2022-04-03T19:44:16.350 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.346+0000 7fcf5acf3700 1 -- 172.21.15.77:0/738860556 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf540c5930 msgr2=0x7fcf54155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:16.350 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.346+0000 7fcf5acf3700 1 --2- 172.21.15.77:0/738860556 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf540c5930 0x7fcf54155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:16.350 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.346+0000 7fcf5acf3700 1 -- 172.21.15.77:0/738860556 >> v1:172.21.15.77:6789/0 conn(0x7fcf5406d3f0 legacy=0x7fcf5406d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:16.351 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.346+0000 7fcf5acf3700 1 -- 172.21.15.77:0/738860556 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf540c3820 con 0x7fcf54156120 2022-04-03T19:44:16.351 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.346+0000 7fcf5acf3700 1 --2- 172.21.15.77:0/738860556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf54156120 0x7fcf5406c040 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c009e90 tx=0x7fcf4c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2d092824b469a5e server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:16.351 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.346+0000 7fcf594f0700 1 -- 172.21.15.77:0/738860556 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c00e070 con 0x7fcf54156120 2022-04-03T19:44:16.351 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.346+0000 7fcf594f0700 1 -- 172.21.15.77:0/738860556 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf4c00bbd0 con 0x7fcf54156120 2022-04-03T19:44:16.351 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.346+0000 7fcf594f0700 1 -- 172.21.15.77:0/738860556 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c012430 con 0x7fcf54156120 2022-04-03T19:44:16.354 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.347+0000 7fcf5b4f4700 1 -- 172.21.15.77:0/738860556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf54156120 msgr2=0x7fcf5406c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:16.355 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.347+0000 7fcf5b4f4700 1 --2- 172.21.15.77:0/738860556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf54156120 0x7fcf5406c040 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c009e90 tx=0x7fcf4c009780 comp rx=0 tx=0).stop 2022-04-03T19:44:16.355 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.347+0000 7fcf5b4f4700 1 -- 172.21.15.77:0/738860556 shutdown_connections 2022-04-03T19:44:16.355 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.347+0000 7fcf5b4f4700 1 --2- 172.21.15.77:0/738860556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf54156120 0x7fcf5406c040 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:16.355 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.347+0000 7fcf5b4f4700 1 --2- 172.21.15.77:0/738860556 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf540c5930 0x7fcf54155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:16.355 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.347+0000 7fcf5b4f4700 1 -- 172.21.15.77:0/738860556 >> 172.21.15.77:0/738860556 conn(0x7fcf540cb720 msgr2=0x7fcf540cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:16.356 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.347+0000 7fcf5b4f4700 1 -- 172.21.15.77:0/738860556 shutdown_connections 2022-04-03T19:44:16.356 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.347+0000 7fcf5b4f4700 1 -- 172.21.15.77:0/738860556 wait complete. 2022-04-03T19:44:16.356 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.348+0000 7fcf5b4f4700 1 Processor -- start 2022-04-03T19:44:16.356 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.349+0000 7fcf5b4f4700 1 -- start start 2022-04-03T19:44:16.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.349+0000 7fcf5b4f4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf5406d3f0 0x7fcf54165bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:16.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.350+0000 7fcf5b4f4700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf540c5930 0x7fcf54156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:16.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.350+0000 7fcf5b4f4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf54156120 0x7fcf540ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:16.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.350+0000 7fcf5b4f4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf54166100 con 0x7fcf54156120 2022-04-03T19:44:16.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.350+0000 7fcf5b4f4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf54150f60 con 0x7fcf5406d3f0 2022-04-03T19:44:16.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.350+0000 7fcf5b4f4700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf541506c0 con 0x7fcf540c5930 2022-04-03T19:44:16.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.350+0000 7fcf5a4f2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf5406d3f0 0x7fcf54165bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:16.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.350+0000 7fcf5acf3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf54156120 0x7fcf540ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:16.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.350+0000 7fcf5a4f2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf5406d3f0 0x7fcf54165bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37642/0 (socket says 172.21.15.77:37642) 2022-04-03T19:44:16.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.350+0000 7fcf5a4f2700 1 -- 172.21.15.77:0/842886775 learned_addr learned my addr 172.21.15.77:0/842886775 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:16.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.350+0000 7fcf59cf1700 1 --2- 172.21.15.77:0/842886775 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf540c5930 0x7fcf54156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:16.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.350+0000 7fcf5a4f2700 1 -- 172.21.15.77:0/842886775 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf540c5930 msgr2=0x7fcf54156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:16.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.350+0000 7fcf5a4f2700 1 --2- 172.21.15.77:0/842886775 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf540c5930 0x7fcf54156540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:16.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.350+0000 7fcf5a4f2700 1 -- 172.21.15.77:0/842886775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf54156120 msgr2=0x7fcf540ca6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:16.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.350+0000 7fcf5a4f2700 1 --2- 172.21.15.77:0/842886775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf54156120 0x7fcf540ca6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:16.360 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.350+0000 7fcf5a4f2700 1 -- 172.21.15.77:0/842886775 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf4c009a20 con 0x7fcf5406d3f0 2022-04-03T19:44:16.360 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.351+0000 7fcf5a4f2700 1 --2- 172.21.15.77:0/842886775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf5406d3f0 0x7fcf54165bc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcf5000c490 tx=0x7fcf50007c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:16.360 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.351+0000 7fcf427fc700 1 -- 172.21.15.77:0/842886775 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5000ce40 con 0x7fcf5406d3f0 2022-04-03T19:44:16.360 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.351+0000 7fcf427fc700 1 -- 172.21.15.77:0/842886775 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf500072e0 con 0x7fcf5406d3f0 2022-04-03T19:44:16.361 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.351+0000 7fcf5b4f4700 1 -- 172.21.15.77:0/842886775 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf54150060 con 0x7fcf5406d3f0 2022-04-03T19:44:16.361 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.351+0000 7fcf427fc700 1 -- 172.21.15.77:0/842886775 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf50005780 con 0x7fcf5406d3f0 2022-04-03T19:44:16.361 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.351+0000 7fcf5b4f4700 1 -- 172.21.15.77:0/842886775 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf54150910 con 0x7fcf5406d3f0 2022-04-03T19:44:16.361 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.352+0000 7fcf5b4f4700 1 -- 172.21.15.77:0/842886775 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf5406ba20 con 0x7fcf5406d3f0 2022-04-03T19:44:16.361 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.353+0000 7fcf427fc700 1 -- 172.21.15.77:0/842886775 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fcf50019030 con 0x7fcf5406d3f0 2022-04-03T19:44:16.362 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.354+0000 7fcf427fc700 1 --2- 172.21.15.77:0/842886775 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcf34063ce0 0x7fcf34066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:16.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.354+0000 7fcf427fc700 1 -- 172.21.15.77:0/842886775 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fcf5007f830 con 0x7fcf5406d3f0 2022-04-03T19:44:16.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.354+0000 7fcf59cf1700 1 --2- 172.21.15.77:0/842886775 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcf34063ce0 0x7fcf34066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:16.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.355+0000 7fcf59cf1700 1 --2- 172.21.15.77:0/842886775 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcf34063ce0 0x7fcf34066190 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcf540cb6b0 tx=0x7fcf44009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:16.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.358+0000 7fcf427fc700 1 -- 172.21.15.77:0/842886775 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fcf5004e670 con 0x7fcf5406d3f0 2022-04-03T19:44:16.592 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.591+0000 7fcf5b4f4700 1 -- 172.21.15.77:0/842886775 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf5404b050 con 0x7fcf5406d3f0 2022-04-03T19:44:16.593 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.592+0000 7fcf427fc700 1 -- 172.21.15.77:0/842886775 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5039 (secure 0 0 0) 0x7fcf50017070 con 0x7fcf5406d3f0 2022-04-03T19:44:16.594 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:16.594 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4303,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4327,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3665969042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3665969042},{"type":"v1","addr":"172.21.15.77:6837","nonce":3665969042}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2022-04-03T19:44:16.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.594+0000 7fcf5b4f4700 1 -- 172.21.15.77:0/842886775 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcf34063ce0 msgr2=0x7fcf34066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:16.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.594+0000 7fcf5b4f4700 1 --2- 172.21.15.77:0/842886775 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcf34063ce0 0x7fcf34066190 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcf540cb6b0 tx=0x7fcf44009060 comp rx=0 tx=0).stop 2022-04-03T19:44:16.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.594+0000 7fcf5b4f4700 1 -- 172.21.15.77:0/842886775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf5406d3f0 msgr2=0x7fcf54165bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:16.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.594+0000 7fcf5b4f4700 1 --2- 172.21.15.77:0/842886775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf5406d3f0 0x7fcf54165bc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcf5000c490 tx=0x7fcf50007c70 comp rx=0 tx=0).stop 2022-04-03T19:44:16.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.595+0000 7fcf5b4f4700 1 -- 172.21.15.77:0/842886775 shutdown_connections 2022-04-03T19:44:16.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.595+0000 7fcf5b4f4700 1 --2- 172.21.15.77:0/842886775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf54156120 0x7fcf540ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:16.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.595+0000 7fcf5b4f4700 1 --2- 172.21.15.77:0/842886775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf5406d3f0 0x7fcf54165bc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:16.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.595+0000 7fcf5b4f4700 1 --2- 172.21.15.77:0/842886775 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcf34063ce0 0x7fcf34066190 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:16.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.595+0000 7fcf5b4f4700 1 --2- 172.21.15.77:0/842886775 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf540c5930 0x7fcf54156540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:16.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.595+0000 7fcf5b4f4700 1 -- 172.21.15.77:0/842886775 >> 172.21.15.77:0/842886775 conn(0x7fcf540cb720 msgr2=0x7fcf5414dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:16.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.595+0000 7fcf5b4f4700 1 -- 172.21.15.77:0/842886775 shutdown_connections 2022-04-03T19:44:16.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.595+0000 7fcf5b4f4700 1 -- 172.21.15.77:0/842886775 wait complete. 2022-04-03T19:44:16.599 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 3 2022-04-03T19:44:16.620 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-04-03T19:44:16.819 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.817+0000 7f354b59e700 1 Processor -- start 2022-04-03T19:44:16.819 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.817+0000 7f354b59e700 1 -- start start 2022-04-03T19:44:16.821 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.819+0000 7f354b59e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f354c0c5930 0x7f354c155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:16.821 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.819+0000 7f354b59e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f354c156120 0x7f354c06c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:16.822 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.819+0000 7f354b59e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354c0c2f80 con 0x7f354c06d3f0 2022-04-03T19:44:16.822 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.819+0000 7f354b59e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f354c0c3260 con 0x7f354c156120 2022-04-03T19:44:16.822 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.819+0000 7f354b59e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f354c0c3540 con 0x7f354c0c5930 2022-04-03T19:44:16.823 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.819+0000 7f354a59c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f354c06d3f0 0x7f354c06d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38280/0 (socket says 172.21.15.77:38280) 2022-04-03T19:44:16.823 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.819+0000 7f354a59c700 1 -- 172.21.15.77:0/3204016838 learned_addr learned my addr 172.21.15.77:0/3204016838 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:16.823 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.819+0000 7f354ad9d700 1 --2- 172.21.15.77:0/3204016838 >> v2:172.21.15.77:3300/0 conn(0x7f354c156120 0x7f354c06c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:16.823 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.819+0000 7f3549d9b700 1 --2- 172.21.15.77:0/3204016838 >> v2:172.21.15.186:3300/0 conn(0x7f354c0c5930 0x7f354c155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:16.823 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.819+0000 7f354ad9d700 1 -- 172.21.15.77:0/3204016838 >> v2:172.21.15.186:3300/0 conn(0x7f354c0c5930 msgr2=0x7f354c155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:16.824 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.819+0000 7f354ad9d700 1 --2- 172.21.15.77:0/3204016838 >> v2:172.21.15.186:3300/0 conn(0x7f354c0c5930 0x7f354c155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:16.824 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.819+0000 7f354ad9d700 1 -- 172.21.15.77:0/3204016838 >> v1:172.21.15.77:6789/0 conn(0x7f354c06d3f0 legacy=0x7f354c06d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:16.824 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.819+0000 7f354959a700 1 -- 172.21.15.77:0/3204016838 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629242860 0 0) 0x7f354c0c2f80 con 0x7f354c06d3f0 2022-04-03T19:44:16.824 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.820+0000 7f354ad9d700 1 -- 172.21.15.77:0/3204016838 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354c0c3820 con 0x7f354c156120 2022-04-03T19:44:16.825 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.820+0000 7f354ad9d700 1 --2- 172.21.15.77:0/3204016838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f354c156120 0x7f354c06c040 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3544009e60 tx=0x7f3544009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96edc225cb15f758 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:16.825 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.820+0000 7f354959a700 1 -- 172.21.15.77:0/3204016838 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354400e070 con 0x7f354c156120 2022-04-03T19:44:16.825 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.820+0000 7f354959a700 1 -- 172.21.15.77:0/3204016838 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354400bbb0 con 0x7f354c156120 2022-04-03T19:44:16.825 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.820+0000 7f354959a700 1 -- 172.21.15.77:0/3204016838 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3544012430 con 0x7f354c156120 2022-04-03T19:44:16.829 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.821+0000 7f354b59e700 1 -- 172.21.15.77:0/3204016838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f354c156120 msgr2=0x7f354c06c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:16.829 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.821+0000 7f354b59e700 1 --2- 172.21.15.77:0/3204016838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f354c156120 0x7f354c06c040 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3544009e60 tx=0x7f3544009780 comp rx=0 tx=0).stop 2022-04-03T19:44:16.830 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.821+0000 7f354b59e700 1 -- 172.21.15.77:0/3204016838 shutdown_connections 2022-04-03T19:44:16.830 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.821+0000 7f354b59e700 1 --2- 172.21.15.77:0/3204016838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f354c156120 0x7f354c06c040 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:16.830 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.821+0000 7f354b59e700 1 --2- 172.21.15.77:0/3204016838 >> v2:172.21.15.186:3300/0 conn(0x7f354c0c5930 0x7f354c155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:16.830 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.821+0000 7f354b59e700 1 -- 172.21.15.77:0/3204016838 >> 172.21.15.77:0/3204016838 conn(0x7f354c0cb720 msgr2=0x7f354c0cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:16.830 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.821+0000 7f354b59e700 1 -- 172.21.15.77:0/3204016838 shutdown_connections 2022-04-03T19:44:16.831 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.821+0000 7f354b59e700 1 -- 172.21.15.77:0/3204016838 wait complete. 2022-04-03T19:44:16.831 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.822+0000 7f354b59e700 1 Processor -- start 2022-04-03T19:44:16.831 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.823+0000 7f354b59e700 1 -- start start 2022-04-03T19:44:16.831 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.823+0000 7f354b59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f354c06d3f0 0x7f354c169390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:16.832 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.823+0000 7f354b59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f354c0c5930 0x7f354c14fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:16.832 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.823+0000 7f354b59e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f354c156120 0x7f354c0bfb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:16.832 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.823+0000 7f354b59e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354c14f2b0 con 0x7f354c06d3f0 2022-04-03T19:44:16.832 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.823+0000 7f354b59e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354c1509c0 con 0x7f354c156120 2022-04-03T19:44:16.832 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.823+0000 7f354b59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f354c0bf6f0 con 0x7f354c0c5930 2022-04-03T19:44:16.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.824+0000 7f354a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f354c06d3f0 0x7f354c169390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:16.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.824+0000 7f354a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f354c06d3f0 0x7f354c169390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42466/0 (socket says 172.21.15.77:42466) 2022-04-03T19:44:16.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.824+0000 7f354a59c700 1 -- 172.21.15.77:0/1372775396 learned_addr learned my addr 172.21.15.77:0/1372775396 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:16.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.824+0000 7f3549d9b700 1 --2- 172.21.15.77:0/1372775396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f354c0c5930 0x7f354c14fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:16.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.824+0000 7f354ad9d700 1 --2- 172.21.15.77:0/1372775396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f354c156120 0x7f354c0bfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:16.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.824+0000 7f354a59c700 1 -- 172.21.15.77:0/1372775396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f354c0c5930 msgr2=0x7f354c14fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:16.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.824+0000 7f354a59c700 1 --2- 172.21.15.77:0/1372775396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f354c0c5930 0x7f354c14fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:16.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.824+0000 7f354a59c700 1 -- 172.21.15.77:0/1372775396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f354c156120 msgr2=0x7f354c0bfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:16.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.824+0000 7f354a59c700 1 --2- 172.21.15.77:0/1372775396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f354c156120 0x7f354c0bfb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:16.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.824+0000 7f354a59c700 1 -- 172.21.15.77:0/1372775396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3544009a20 con 0x7f354c06d3f0 2022-04-03T19:44:16.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.825+0000 7f354a59c700 1 --2- 172.21.15.77:0/1372775396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f354c06d3f0 0x7f354c169390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f353c00c510 tx=0x7f353c007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:16.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.825+0000 7f353a7fc700 1 -- 172.21.15.77:0/1372775396 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c00ce80 con 0x7f354c06d3f0 2022-04-03T19:44:16.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.825+0000 7f353a7fc700 1 -- 172.21.15.77:0/1372775396 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353c007360 con 0x7f354c06d3f0 2022-04-03T19:44:16.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.825+0000 7f354b59e700 1 -- 172.21.15.77:0/1372775396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f354c14e750 con 0x7f354c06d3f0 2022-04-03T19:44:16.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.825+0000 7f353a7fc700 1 -- 172.21.15.77:0/1372775396 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c0046b0 con 0x7f354c06d3f0 2022-04-03T19:44:16.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.825+0000 7f354b59e700 1 -- 172.21.15.77:0/1372775396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f354c14ecf0 con 0x7f354c06d3f0 2022-04-03T19:44:16.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.826+0000 7f354b59e700 1 -- 172.21.15.77:0/1372775396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f354c0bf4c0 con 0x7f354c06d3f0 2022-04-03T19:44:16.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.827+0000 7f353a7fc700 1 -- 172.21.15.77:0/1372775396 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f353c010030 con 0x7f354c06d3f0 2022-04-03T19:44:16.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.828+0000 7f353a7fc700 1 --2- 172.21.15.77:0/1372775396 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3528063ce0 0x7f3528066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:16.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.828+0000 7f353a7fc700 1 -- 172.21.15.77:0/1372775396 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f353c0800c0 con 0x7f354c06d3f0 2022-04-03T19:44:16.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.828+0000 7f3549d9b700 1 --2- 172.21.15.77:0/1372775396 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3528063ce0 0x7f3528066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:16.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.829+0000 7f3549d9b700 1 --2- 172.21.15.77:0/1372775396 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3528063ce0 0x7f3528066190 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f354c152f90 tx=0x7f3540009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:16.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:16.834+0000 7f353a7fc700 1 -- 172.21.15.77:0/1372775396 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f354c0bf4c0 con 0x7f354c06d3f0 2022-04-03T19:44:17.028 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.027+0000 7f354b59e700 1 -- 172.21.15.77:0/1372775396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f354c0bf4c0 con 0x7f354c06d3f0 2022-04-03T19:44:17.029 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.028+0000 7f353a7fc700 1 -- 172.21.15.77:0/1372775396 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13626 (secure 0 0 0) 0x7f354c0bf4c0 con 0x7f354c06d3f0 2022-04-03T19:44:17.029 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:17.030 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":20,"fsid":"89c3a3f4-599d-4c4a-8f5a-7180eafb7292","created":"2022-04-03T19:43:54.101559+0000","modified":"2022-04-03T19:44:15.742748+0000","last_up_change":"2022-04-03T19:44:05.589174+0000","last_in_change":"2022-04-03T19:43:56.729760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-03T19:44:07.567638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-04-03T19:44:13.149481+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f35f5195-3729-4d78-85d1-0539f259283f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6823","nonce":20252},{"type":"v1","addr":"172.21.15.77:6825","nonce":20252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":20252},{"type":"v1","addr":"172.21.15.77:6827","nonce":20252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":20252},{"type":"v1","addr":"172.21.15.77:6831","nonce":20252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":20252},{"type":"v1","addr":"172.21.15.77:6829","nonce":20252}]},"public_addr":"172.21.15.77:6825/20252","cluster_addr":"172.21.15.77:6827/20252","heartbeat_back_addr":"172.21.15.77:6831/20252","heartbeat_front_addr":"172.21.15.77:6829/20252","state":["exists","up"]},{"osd":1,"uuid":"ab1c89dc-57ce-4cc4-b8c7-9b0c19939601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":20249},{"type":"v1","addr":"172.21.15.77:6805","nonce":20249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":20249},{"type":"v1","addr":"172.21.15.77:6809","nonce":20249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":20249},{"type":"v1","addr":"172.21.15.77:6819","nonce":20249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":20249},{"type":"v1","addr":"172.21.15.77:6815","nonce":20249}]},"public_addr":"172.21.15.77:6805/20249","cluster_addr":"172.21.15.77:6809/20249","heartbeat_back_addr":"172.21.15.77:6819/20249","heartbeat_front_addr":"172.21.15.77:6815/20249","state":["exists","up"]},{"osd":2,"uuid":"351e218b-7749-4ee9-b44a-0a5b52fb9be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":20248},{"type":"v1","addr":"172.21.15.77:6801","nonce":20248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":20248},{"type":"v1","addr":"172.21.15.77:6803","nonce":20248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":20248},{"type":"v1","addr":"172.21.15.77:6813","nonce":20248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":20248},{"type":"v1","addr":"172.21.15.77:6808","nonce":20248}]},"public_addr":"172.21.15.77:6801/20248","cluster_addr":"172.21.15.77:6803/20248","heartbeat_back_addr":"172.21.15.77:6813/20248","heartbeat_front_addr":"172.21.15.77:6808/20248","state":["exists","up"]},{"osd":3,"uuid":"1ba6ec8a-7cbf-4c12-8e21-c3097a263ae8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":20250},{"type":"v1","addr":"172.21.15.77:6814","nonce":20250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":20250},{"type":"v1","addr":"172.21.15.77:6818","nonce":20250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":20250},{"type":"v1","addr":"172.21.15.77:6824","nonce":20250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":20250},{"type":"v1","addr":"172.21.15.77:6821","nonce":20250}]},"public_addr":"172.21.15.77:6814/20250","cluster_addr":"172.21.15.77:6818/20250","heartbeat_back_addr":"172.21.15.77:6824/20250","heartbeat_front_addr":"172.21.15.77:6821/20250","state":["exists","up"]},{"osd":4,"uuid":"3faf4c09-846c-4133-856b-7f53c535a67d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":25551},{"type":"v1","addr":"172.21.15.186:6822","nonce":25551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":25551},{"type":"v1","addr":"172.21.15.186:6826","nonce":25551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":25551},{"type":"v1","addr":"172.21.15.186:6831","nonce":25551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":25551},{"type":"v1","addr":"172.21.15.186:6829","nonce":25551}]},"public_addr":"172.21.15.186:6822/25551","cluster_addr":"172.21.15.186:6826/25551","heartbeat_back_addr":"172.21.15.186:6831/25551","heartbeat_front_addr":"172.21.15.186:6829/25551","state":["exists","up"]},{"osd":5,"uuid":"ffe41d00-d0aa-42f4-82c1-5cced13c1bc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":25541},{"type":"v1","addr":"172.21.15.186:6809","nonce":25541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":25541},{"type":"v1","addr":"172.21.15.186:6811","nonce":25541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":25541},{"type":"v1","addr":"172.21.15.186:6815","nonce":25541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":25541},{"type":"v1","addr":"172.21.15.186:6813","nonce":25541}]},"public_addr":"172.21.15.186:6809/25541","cluster_addr":"172.21.15.186:6811/25541","heartbeat_back_addr":"172.21.15.186:6815/25541","heartbeat_front_addr":"172.21.15.186:6813/25541","state":["exists","up"]},{"osd":6,"uuid":"b0344ee8-deca-4e9b-934c-5866d90d0073","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":25542},{"type":"v1","addr":"172.21.15.186:6801","nonce":25542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":25542},{"type":"v1","addr":"172.21.15.186:6803","nonce":25542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":25542},{"type":"v1","addr":"172.21.15.186:6807","nonce":25542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":25542},{"type":"v1","addr":"172.21.15.186:6805","nonce":25542}]},"public_addr":"172.21.15.186:6801/25542","cluster_addr":"172.21.15.186:6803/25542","heartbeat_back_addr":"172.21.15.186:6807/25542","heartbeat_front_addr":"172.21.15.186:6805/25542","state":["exists","up"]},{"osd":7,"uuid":"0eb0e5d3-3dad-4f49-9b50-3b24a3924bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":25572},{"type":"v1","addr":"172.21.15.186:6817","nonce":25572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":25572},{"type":"v1","addr":"172.21.15.186:6819","nonce":25572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6825","nonce":25572},{"type":"v1","addr":"172.21.15.186:6827","nonce":25572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":25572},{"type":"v1","addr":"172.21.15.186:6823","nonce":25572}]},"public_addr":"172.21.15.186:6817/25572","cluster_addr":"172.21.15.186:6819/25572","heartbeat_back_addr":"172.21.15.186:6827/25572","heartbeat_front_addr":"172.21.15.186:6823/25572","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.143602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.490366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.361292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.343544+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.905429+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969298+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.949524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-03T19:44:17.032 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.030+0000 7f354b59e700 1 -- 172.21.15.77:0/1372775396 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3528063ce0 msgr2=0x7f3528066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:17.032 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.030+0000 7f354b59e700 1 --2- 172.21.15.77:0/1372775396 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3528063ce0 0x7f3528066190 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f354c152f90 tx=0x7f3540009060 comp rx=0 tx=0).stop 2022-04-03T19:44:17.032 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.030+0000 7f354b59e700 1 -- 172.21.15.77:0/1372775396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f354c06d3f0 msgr2=0x7f354c169390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:17.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.030+0000 7f354b59e700 1 --2- 172.21.15.77:0/1372775396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f354c06d3f0 0x7f354c169390 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f353c00c510 tx=0x7f353c007c70 comp rx=0 tx=0).stop 2022-04-03T19:44:17.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.030+0000 7f354b59e700 1 -- 172.21.15.77:0/1372775396 shutdown_connections 2022-04-03T19:44:17.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.030+0000 7f354b59e700 1 --2- 172.21.15.77:0/1372775396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f354c156120 0x7f354c0bfb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.030+0000 7f354b59e700 1 --2- 172.21.15.77:0/1372775396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f354c06d3f0 0x7f354c169390 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.030+0000 7f354b59e700 1 --2- 172.21.15.77:0/1372775396 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3528063ce0 0x7f3528066190 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.030+0000 7f354b59e700 1 --2- 172.21.15.77:0/1372775396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f354c0c5930 0x7f354c14fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.030+0000 7f354b59e700 1 -- 172.21.15.77:0/1372775396 >> 172.21.15.77:0/1372775396 conn(0x7f354c0cb720 msgr2=0x7f354c14dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:17.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.030+0000 7f354b59e700 1 -- 172.21.15.77:0/1372775396 shutdown_connections 2022-04-03T19:44:17.035 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.030+0000 7f354b59e700 1 -- 172.21.15.77:0/1372775396 wait complete. 2022-04-03T19:44:17.055 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-04-03T19:44:07.567638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-04-03T19:44:13.149481+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-04-03T19:44:17.056 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-04-03T19:44:17.256 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.255+0000 7f86b33ad700 1 Processor -- start 2022-04-03T19:44:17.257 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.255+0000 7f86b33ad700 1 -- start start 2022-04-03T19:44:17.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.256+0000 7f86b33ad700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f86ac0c52b0 0x7f86ac0c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:17.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.256+0000 7f86b33ad700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f86ac158e10 0x7f86ac14ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:17.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.256+0000 7f86b33ad700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86ac0c7e80 con 0x7f86ac0c5680 2022-04-03T19:44:17.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.256+0000 7f86b33ad700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86ac0c4260 con 0x7f86ac0c52b0 2022-04-03T19:44:17.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.256+0000 7f86b33ad700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86ac0c8a20 con 0x7f86ac158e10 2022-04-03T19:44:17.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.256+0000 7f86b23ab700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f86ac0c52b0 0x7f86ac0c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:17.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.256+0000 7f86b1baa700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f86ac0c5680 0x7f86ac0c8310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44084/0 (socket says 172.21.15.77:44084) 2022-04-03T19:44:17.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.256+0000 7f86b1baa700 1 -- 172.21.15.77:0/662701719 learned_addr learned my addr 172.21.15.77:0/662701719 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:17.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.256+0000 7f86b2bac700 1 --2- 172.21.15.77:0/662701719 >> v2:172.21.15.186:3300/0 conn(0x7f86ac158e10 0x7f86ac14ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:17.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.257+0000 7f86b13a9700 1 -- 172.21.15.77:0/662701719 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271783465 0 0) 0x7f86ac0c7e80 con 0x7f86ac0c5680 2022-04-03T19:44:17.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.257+0000 7f86b13a9700 1 -- 172.21.15.77:0/662701719 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86940033f0 con 0x7f86ac0c5680 2022-04-03T19:44:17.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.257+0000 7f86b2bac700 1 -- 172.21.15.77:0/662701719 >> v2:172.21.15.77:3300/0 conn(0x7f86ac0c52b0 msgr2=0x7f86ac0c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:17.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.257+0000 7f86b2bac700 1 --2- 172.21.15.77:0/662701719 >> v2:172.21.15.77:3300/0 conn(0x7f86ac0c52b0 0x7f86ac0c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.257+0000 7f86b2bac700 1 -- 172.21.15.77:0/662701719 >> v1:172.21.15.186:6789/0 conn(0x7f86ac0c5680 legacy=0x7f86ac0c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:17.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.257+0000 7f86b2bac700 1 -- 172.21.15.77:0/662701719 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ac0c6ea0 con 0x7f86ac158e10 2022-04-03T19:44:17.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.257+0000 7f86b2bac700 1 --2- 172.21.15.77:0/662701719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86ac158e10 0x7f86ac14ed50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f86ac0bf380 tx=0x7f86a4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de24fd07242f6f06 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:17.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.257+0000 7f86b13a9700 1 -- 172.21.15.77:0/662701719 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a400e070 con 0x7f86ac158e10 2022-04-03T19:44:17.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.259+0000 7f86b13a9700 1 -- 172.21.15.77:0/662701719 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a400bb40 con 0x7f86ac158e10 2022-04-03T19:44:17.263 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.259+0000 7f86b13a9700 1 -- 172.21.15.77:0/662701719 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a4012430 con 0x7f86ac158e10 2022-04-03T19:44:17.265 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.259+0000 7f86b33ad700 1 -- 172.21.15.77:0/662701719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86ac158e10 msgr2=0x7f86ac14ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:17.266 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.259+0000 7f86b33ad700 1 --2- 172.21.15.77:0/662701719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86ac158e10 0x7f86ac14ed50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f86ac0bf380 tx=0x7f86a4009780 comp rx=0 tx=0).stop 2022-04-03T19:44:17.266 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.259+0000 7f86b33ad700 1 -- 172.21.15.77:0/662701719 shutdown_connections 2022-04-03T19:44:17.266 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.259+0000 7f86b33ad700 1 --2- 172.21.15.77:0/662701719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86ac158e10 0x7f86ac14ed50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.266 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.259+0000 7f86b33ad700 1 --2- 172.21.15.77:0/662701719 >> v2:172.21.15.77:3300/0 conn(0x7f86ac0c52b0 0x7f86ac0c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.267 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.259+0000 7f86b33ad700 1 -- 172.21.15.77:0/662701719 >> 172.21.15.77:0/662701719 conn(0x7f86ac0626d0 msgr2=0x7f86ac062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:17.267 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.259+0000 7f86b33ad700 1 -- 172.21.15.77:0/662701719 shutdown_connections 2022-04-03T19:44:17.267 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.259+0000 7f86b33ad700 1 -- 172.21.15.77:0/662701719 wait complete. 2022-04-03T19:44:17.267 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.261+0000 7f86b33ad700 1 Processor -- start 2022-04-03T19:44:17.267 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.261+0000 7f86b33ad700 1 -- start start 2022-04-03T19:44:17.268 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.262+0000 7f86b33ad700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86ac0c52b0 0x7f86ac0cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:17.268 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.262+0000 7f86b33ad700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86ac0c5680 0x7f86ac157650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:17.268 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.262+0000 7f86b33ad700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86ac158e10 0x7f86ac14d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:17.268 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.262+0000 7f86b33ad700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac154e40 con 0x7f86ac0c5680 2022-04-03T19:44:17.269 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.262+0000 7f86b33ad700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac1496e0 con 0x7f86ac0c52b0 2022-04-03T19:44:17.269 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.262+0000 7f86b33ad700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac155050 con 0x7f86ac158e10 2022-04-03T19:44:17.269 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.262+0000 7f86b1baa700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86ac0c5680 0x7f86ac157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:17.269 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.262+0000 7f86b1baa700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86ac0c5680 0x7f86ac157650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42470/0 (socket says 172.21.15.77:42470) 2022-04-03T19:44:17.269 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.262+0000 7f86b23ab700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86ac0c52b0 0x7f86ac0cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:17.270 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.262+0000 7f86b1baa700 1 -- 172.21.15.77:0/174650951 learned_addr learned my addr 172.21.15.77:0/174650951 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:17.270 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.262+0000 7f86b2bac700 1 --2- 172.21.15.77:0/174650951 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86ac158e10 0x7f86ac14d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:17.270 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.262+0000 7f86b1baa700 1 -- 172.21.15.77:0/174650951 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86ac158e10 msgr2=0x7f86ac14d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:17.270 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.262+0000 7f86b1baa700 1 --2- 172.21.15.77:0/174650951 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86ac158e10 0x7f86ac14d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.270 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.263+0000 7f86b1baa700 1 -- 172.21.15.77:0/174650951 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86ac0c52b0 msgr2=0x7f86ac0cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:17.271 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.263+0000 7f86b1baa700 1 --2- 172.21.15.77:0/174650951 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86ac0c52b0 0x7f86ac0cad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.271 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.263+0000 7f86b1baa700 1 -- 172.21.15.77:0/174650951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a4009a20 con 0x7f86ac0c5680 2022-04-03T19:44:17.271 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.263+0000 7f86b1baa700 1 --2- 172.21.15.77:0/174650951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86ac0c5680 0x7f86ac157650 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f869c00c490 tx=0x7f869c007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:17.271 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.263+0000 7f869a7fc700 1 -- 172.21.15.77:0/174650951 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c00ce40 con 0x7f86ac0c5680 2022-04-03T19:44:17.272 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.263+0000 7f869a7fc700 1 -- 172.21.15.77:0/174650951 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869c0072e0 con 0x7f86ac0c5680 2022-04-03T19:44:17.272 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.263+0000 7f86b33ad700 1 -- 172.21.15.77:0/174650951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86ac155330 con 0x7f86ac0c5680 2022-04-03T19:44:17.272 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.263+0000 7f869a7fc700 1 -- 172.21.15.77:0/174650951 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c005780 con 0x7f86ac0c5680 2022-04-03T19:44:17.272 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.263+0000 7f86b33ad700 1 -- 172.21.15.77:0/174650951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86ac152b80 con 0x7f86ac0c5680 2022-04-03T19:44:17.272 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.264+0000 7f86b33ad700 1 -- 172.21.15.77:0/174650951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86ac155850 con 0x7f86ac0c5680 2022-04-03T19:44:17.273 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.265+0000 7f869a7fc700 1 -- 172.21.15.77:0/174650951 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f869c019030 con 0x7f86ac0c5680 2022-04-03T19:44:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.265+0000 7f869a7fc700 1 --2- 172.21.15.77:0/174650951 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f868c063ce0 0x7f868c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.265+0000 7f869a7fc700 1 -- 172.21.15.77:0/174650951 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f869c07f780 con 0x7f86ac0c5680 2022-04-03T19:44:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.265+0000 7f86b23ab700 1 --2- 172.21.15.77:0/174650951 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f868c063ce0 0x7f868c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:17.274 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.267+0000 7f86b23ab700 1 --2- 172.21.15.77:0/174650951 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f868c063ce0 0x7f868c066190 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f86a8009c30 tx=0x7f86a8009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:17.275 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.269+0000 7f869a7fc700 1 -- 172.21.15.77:0/174650951 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f86ac155850 con 0x7f86ac0c5680 2022-04-03T19:44:17.464 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.464+0000 7f86b33ad700 1 -- 172.21.15.77:0/174650951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f86ac155850 con 0x7f86ac0c5680 2022-04-03T19:44:17.465 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.465+0000 7f869a7fc700 1 -- 172.21.15.77:0/174650951 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f86ac155850 con 0x7f86ac0c5680 2022-04-03T19:44:17.465 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2022-04-03T19:44:17.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.467+0000 7f86b33ad700 1 -- 172.21.15.77:0/174650951 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f868c063ce0 msgr2=0x7f868c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:17.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.467+0000 7f86b33ad700 1 --2- 172.21.15.77:0/174650951 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f868c063ce0 0x7f868c066190 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f86a8009c30 tx=0x7f86a8009060 comp rx=0 tx=0).stop 2022-04-03T19:44:17.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.467+0000 7f86b33ad700 1 -- 172.21.15.77:0/174650951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86ac0c5680 msgr2=0x7f86ac157650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:17.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.467+0000 7f86b33ad700 1 --2- 172.21.15.77:0/174650951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86ac0c5680 0x7f86ac157650 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f869c00c490 tx=0x7f869c007c70 comp rx=0 tx=0).stop 2022-04-03T19:44:17.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.467+0000 7f86b33ad700 1 -- 172.21.15.77:0/174650951 shutdown_connections 2022-04-03T19:44:17.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.467+0000 7f86b33ad700 1 --2- 172.21.15.77:0/174650951 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f868c063ce0 0x7f868c066190 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.467+0000 7f86b33ad700 1 --2- 172.21.15.77:0/174650951 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86ac158e10 0x7f86ac14d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.467+0000 7f86b33ad700 1 --2- 172.21.15.77:0/174650951 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86ac0c52b0 0x7f86ac0cad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.467+0000 7f86b33ad700 1 --2- 172.21.15.77:0/174650951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86ac0c5680 0x7f86ac157650 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.467+0000 7f86b33ad700 1 -- 172.21.15.77:0/174650951 >> 172.21.15.77:0/174650951 conn(0x7f86ac0626d0 msgr2=0x7f86ac147040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:17.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.467+0000 7f86b33ad700 1 -- 172.21.15.77:0/174650951 shutdown_connections 2022-04-03T19:44:17.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.467+0000 7f86b33ad700 1 -- 172.21.15.77:0/174650951 wait complete. 2022-04-03T19:44:17.491 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-04-03T19:44:17.690 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.690+0000 7fc0127a5700 1 Processor -- start 2022-04-03T19:44:17.691 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.690+0000 7fc0127a5700 1 -- start start 2022-04-03T19:44:17.692 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.691+0000 7fc0127a5700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc00c14ef20 0x7fc00c1559d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:17.692 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.691+0000 7fc0127a5700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc00c0c5170 0x7fc00c0c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:17.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.691+0000 7fc0127a5700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc00c06df20 con 0x7fc00c06bdb0 2022-04-03T19:44:17.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.691+0000 7fc0127a5700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc00c155f10 con 0x7fc00c14ef20 2022-04-03T19:44:17.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.691+0000 7fc0127a5700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc00c0c6690 con 0x7fc00c0c5170 2022-04-03T19:44:17.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.691+0000 7fc0117a3700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc00c14ef20 0x7fc00c1559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:17.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.691+0000 7fc010fa2700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc00c06bdb0 0x7fc00c0c5f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38282/0 (socket says 172.21.15.77:38282) 2022-04-03T19:44:17.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.691+0000 7fc010fa2700 1 -- 172.21.15.77:0/1877381200 learned_addr learned my addr 172.21.15.77:0/1877381200 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:17.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.692+0000 7fc011fa4700 1 --2- 172.21.15.77:0/1877381200 >> v2:172.21.15.186:3300/0 conn(0x7fc00c0c5170 0x7fc00c0c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:17.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.692+0000 7fc0117a3700 1 -- 172.21.15.77:0/1877381200 >> v2:172.21.15.186:3300/0 conn(0x7fc00c0c5170 msgr2=0x7fc00c0c55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:17.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.692+0000 7fc0117a3700 1 --2- 172.21.15.77:0/1877381200 >> v2:172.21.15.186:3300/0 conn(0x7fc00c0c5170 0x7fc00c0c55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.692+0000 7fc0117a3700 1 -- 172.21.15.77:0/1877381200 >> v1:172.21.15.77:6789/0 conn(0x7fc00c06bdb0 legacy=0x7fc00c0c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:17.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.692+0000 7fc0117a3700 1 -- 172.21.15.77:0/1877381200 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00c06cd90 con 0x7fc00c14ef20 2022-04-03T19:44:17.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.692+0000 7fc0117a3700 1 --2- 172.21.15.77:0/1877381200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc00c14ef20 0x7fc00c1559d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc00c06c230 tx=0x7fc008009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9940a41fe0ae39fe server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:17.699 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.693+0000 7fbffbfff700 1 -- 172.21.15.77:0/1877381200 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc008011630 con 0x7fc00c14ef20 2022-04-03T19:44:17.699 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.693+0000 7fbffbfff700 1 -- 172.21.15.77:0/1877381200 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc008011790 con 0x7fc00c14ef20 2022-04-03T19:44:17.699 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.693+0000 7fbffbfff700 1 -- 172.21.15.77:0/1877381200 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc008011b30 con 0x7fc00c14ef20 2022-04-03T19:44:17.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.693+0000 7fc0127a5700 1 -- 172.21.15.77:0/1877381200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc00c14ef20 msgr2=0x7fc00c1559d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:17.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.693+0000 7fc0127a5700 1 --2- 172.21.15.77:0/1877381200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc00c14ef20 0x7fc00c1559d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc00c06c230 tx=0x7fc008009060 comp rx=0 tx=0).stop 2022-04-03T19:44:17.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.693+0000 7fc0127a5700 1 -- 172.21.15.77:0/1877381200 shutdown_connections 2022-04-03T19:44:17.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.693+0000 7fc0127a5700 1 --2- 172.21.15.77:0/1877381200 >> v2:172.21.15.186:3300/0 conn(0x7fc00c0c5170 0x7fc00c0c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.693+0000 7fc0127a5700 1 --2- 172.21.15.77:0/1877381200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc00c14ef20 0x7fc00c1559d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.693+0000 7fc0127a5700 1 -- 172.21.15.77:0/1877381200 >> 172.21.15.77:0/1877381200 conn(0x7fc00c0cb6d0 msgr2=0x7fc00c0cbad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:17.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.693+0000 7fc0127a5700 1 -- 172.21.15.77:0/1877381200 shutdown_connections 2022-04-03T19:44:17.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.693+0000 7fc0127a5700 1 -- 172.21.15.77:0/1877381200 wait complete. 2022-04-03T19:44:17.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.695+0000 7fc0127a5700 1 Processor -- start 2022-04-03T19:44:17.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.695+0000 7fc0127a5700 1 -- start start 2022-04-03T19:44:17.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.696+0000 7fc0127a5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc00c06bdb0 0x7fc00c1684c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:17.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.696+0000 7fc0127a5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc00c0c5170 0x7fc00c1659d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:17.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.696+0000 7fc0127a5700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc00c14ef20 0x7fc00c158840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:17.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.696+0000 7fc0127a5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c0c1a90 con 0x7fc00c06bdb0 2022-04-03T19:44:17.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.696+0000 7fc0127a5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c0c1080 con 0x7fc00c0c5170 2022-04-03T19:44:17.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.696+0000 7fc0127a5700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c164f50 con 0x7fc00c14ef20 2022-04-03T19:44:17.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.696+0000 7fc0117a3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc00c06bdb0 0x7fc00c1684c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:17.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.696+0000 7fc010fa2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc00c0c5170 0x7fc00c1659d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:17.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.696+0000 7fc011fa4700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc00c14ef20 0x7fc00c158840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:17.704 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.696+0000 7fc010fa2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc00c0c5170 0x7fc00c1659d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37654/0 (socket says 172.21.15.77:37654) 2022-04-03T19:44:17.704 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.696+0000 7fc010fa2700 1 -- 172.21.15.77:0/4290001181 learned_addr learned my addr 172.21.15.77:0/4290001181 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:17.704 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.697+0000 7fc011fa4700 1 -- 172.21.15.77:0/4290001181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc00c0c5170 msgr2=0x7fc00c1659d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:17.704 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.697+0000 7fc011fa4700 1 --2- 172.21.15.77:0/4290001181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc00c0c5170 0x7fc00c1659d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.705 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.697+0000 7fc011fa4700 1 -- 172.21.15.77:0/4290001181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc00c06bdb0 msgr2=0x7fc00c1684c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:17.705 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.697+0000 7fc011fa4700 1 --2- 172.21.15.77:0/4290001181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc00c06bdb0 0x7fc00c1684c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.705 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.697+0000 7fc011fa4700 1 -- 172.21.15.77:0/4290001181 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0080092a0 con 0x7fc00c14ef20 2022-04-03T19:44:17.705 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.697+0000 7fc011fa4700 1 --2- 172.21.15.77:0/4290001181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc00c14ef20 0x7fc00c158840 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc004009e50 tx=0x7fc00400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:17.705 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.697+0000 7fbff97fa700 1 -- 172.21.15.77:0/4290001181 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00400caa0 con 0x7fc00c14ef20 2022-04-03T19:44:17.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.697+0000 7fbff97fa700 1 -- 172.21.15.77:0/4290001181 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00400f040 con 0x7fc00c14ef20 2022-04-03T19:44:17.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.697+0000 7fc0127a5700 1 -- 172.21.15.77:0/4290001181 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc00c1503b0 con 0x7fc00c14ef20 2022-04-03T19:44:17.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.697+0000 7fbff97fa700 1 -- 172.21.15.77:0/4290001181 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00400ce70 con 0x7fc00c14ef20 2022-04-03T19:44:17.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.698+0000 7fc0127a5700 1 -- 172.21.15.77:0/4290001181 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc00c1520f0 con 0x7fc00c14ef20 2022-04-03T19:44:17.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.698+0000 7fc0127a5700 1 -- 172.21.15.77:0/4290001181 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc00c088dd0 con 0x7fc00c14ef20 2022-04-03T19:44:17.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.699+0000 7fbff97fa700 1 -- 172.21.15.77:0/4290001181 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fc004014690 con 0x7fc00c14ef20 2022-04-03T19:44:17.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.700+0000 7fbff97fa700 1 --2- 172.21.15.77:0/4290001181 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbfec063ce0 0x7fbfec066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:17.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.700+0000 7fbff97fa700 1 -- 172.21.15.77:0/4290001181 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fc004081350 con 0x7fc00c14ef20 2022-04-03T19:44:17.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.700+0000 7fc0117a3700 1 --2- 172.21.15.77:0/4290001181 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbfec063ce0 0x7fbfec066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:17.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.701+0000 7fc0117a3700 1 --2- 172.21.15.77:0/4290001181 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbfec063ce0 0x7fbfec066190 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc008011fd0 tx=0x7fc008017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:17.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.705+0000 7fbff97fa700 1 -- 172.21.15.77:0/4290001181 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fc004050280 con 0x7fc00c14ef20 2022-04-03T19:44:17.896 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.896+0000 7fc0127a5700 1 -- 172.21.15.77:0/4290001181 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc00c06c940 con 0x7fc00c14ef20 2022-04-03T19:44:17.897 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.897+0000 7fbff97fa700 1 -- 172.21.15.77:0/4290001181 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc004018020 con 0x7fc00c14ef20 2022-04-03T19:44:17.897 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2022-04-03T19:44:17.900 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.899+0000 7fc0127a5700 1 -- 172.21.15.77:0/4290001181 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbfec063ce0 msgr2=0x7fbfec066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:17.900 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.899+0000 7fc0127a5700 1 --2- 172.21.15.77:0/4290001181 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbfec063ce0 0x7fbfec066190 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc008011fd0 tx=0x7fc008017000 comp rx=0 tx=0).stop 2022-04-03T19:44:17.900 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.899+0000 7fc0127a5700 1 -- 172.21.15.77:0/4290001181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc00c14ef20 msgr2=0x7fc00c158840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:17.900 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.899+0000 7fc0127a5700 1 --2- 172.21.15.77:0/4290001181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc00c14ef20 0x7fc00c158840 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc004009e50 tx=0x7fc00400b040 comp rx=0 tx=0).stop 2022-04-03T19:44:17.901 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.899+0000 7fc0127a5700 1 -- 172.21.15.77:0/4290001181 shutdown_connections 2022-04-03T19:44:17.901 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.899+0000 7fc0127a5700 1 --2- 172.21.15.77:0/4290001181 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbfec063ce0 0x7fbfec066190 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.901 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.899+0000 7fc0127a5700 1 --2- 172.21.15.77:0/4290001181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc00c14ef20 0x7fc00c158840 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.901 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.899+0000 7fc0127a5700 1 --2- 172.21.15.77:0/4290001181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc00c06bdb0 0x7fc00c1684c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.901 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.899+0000 7fc0127a5700 1 --2- 172.21.15.77:0/4290001181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc00c0c5170 0x7fc00c1659d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:17.902 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.899+0000 7fc0127a5700 1 -- 172.21.15.77:0/4290001181 >> 172.21.15.77:0/4290001181 conn(0x7fc00c0cb6d0 msgr2=0x7fc00c0c5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:17.902 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.899+0000 7fc0127a5700 1 -- 172.21.15.77:0/4290001181 shutdown_connections 2022-04-03T19:44:17.902 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:17.899+0000 7fc0127a5700 1 -- 172.21.15.77:0/4290001181 wait complete. 2022-04-03T19:44:17.924 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-04-03T19:44:18.121 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.120+0000 7f72a3fff700 1 Processor -- start 2022-04-03T19:44:18.121 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.120+0000 7f72a3fff700 1 -- start start 2022-04-03T19:44:18.123 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.121+0000 7f72a3fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72a406bbe0 0x7f72a41584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:18.123 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.121+0000 7f72a3fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72a406bfb0 0x7f72a406d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:18.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.121+0000 7f72a3fff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f72a406c500 0x7f72a414eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:18.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.121+0000 7f72a3fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a4159450 con 0x7f72a406bfb0 2022-04-03T19:44:18.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.121+0000 7f72a3fff700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a406d260 con 0x7f72a406c500 2022-04-03T19:44:18.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.121+0000 7f72a3fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a40c6330 con 0x7f72a406bbe0 2022-04-03T19:44:18.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.122+0000 7f72a27fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72a406bfb0 0x7f72a406d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:18.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.122+0000 7f72a37fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f72a406c500 0x7f72a414eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:18.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.122+0000 7f72a2ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72a406bbe0 0x7f72a41584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:18.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.122+0000 7f72a27fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72a406bfb0 0x7f72a406d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42476/0 (socket says 172.21.15.77:42476) 2022-04-03T19:44:18.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.122+0000 7f72a27fc700 1 -- 172.21.15.77:0/815697205 learned_addr learned my addr 172.21.15.77:0/815697205 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:18.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.122+0000 7f72a27fc700 1 -- 172.21.15.77:0/815697205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72a406bbe0 msgr2=0x7f72a41584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:18.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.122+0000 7f72a27fc700 1 --2- 172.21.15.77:0/815697205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72a406bbe0 0x7f72a41584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:18.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.122+0000 7f72a27fc700 1 -- 172.21.15.77:0/815697205 >> v2:172.21.15.186:3300/0 conn(0x7f72a406c500 msgr2=0x7f72a414eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:18.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.122+0000 7f72a27fc700 1 --2- 172.21.15.77:0/815697205 >> v2:172.21.15.186:3300/0 conn(0x7f72a406c500 0x7f72a414eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:18.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.122+0000 7f72a27fc700 1 -- 172.21.15.77:0/815697205 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a4158bb0 con 0x7f72a406bfb0 2022-04-03T19:44:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.123+0000 7f72a27fc700 1 --2- 172.21.15.77:0/815697205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72a406bfb0 0x7f72a406d6f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f72a40c4fe0 tx=0x7f72980090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=444a62a2cdfaf71e server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.123+0000 7f72a1ffb700 1 -- 172.21.15.77:0/815697205 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7298004030 con 0x7f72a406bfb0 2022-04-03T19:44:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.123+0000 7f72a1ffb700 1 -- 172.21.15.77:0/815697205 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7298004190 con 0x7f72a406bfb0 2022-04-03T19:44:18.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.123+0000 7f72a1ffb700 1 -- 172.21.15.77:0/815697205 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7298011630 con 0x7f72a406bfb0 2022-04-03T19:44:18.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.123+0000 7f72a3fff700 1 -- 172.21.15.77:0/815697205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72a406bfb0 msgr2=0x7f72a406d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:18.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.123+0000 7f72a3fff700 1 --2- 172.21.15.77:0/815697205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72a406bfb0 0x7f72a406d6f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f72a40c4fe0 tx=0x7f72980090f0 comp rx=0 tx=0).stop 2022-04-03T19:44:18.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.123+0000 7f72a3fff700 1 -- 172.21.15.77:0/815697205 shutdown_connections 2022-04-03T19:44:18.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.123+0000 7f72a3fff700 1 --2- 172.21.15.77:0/815697205 >> v2:172.21.15.186:3300/0 conn(0x7f72a406c500 0x7f72a414eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:18.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.123+0000 7f72a3fff700 1 --2- 172.21.15.77:0/815697205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72a406bbe0 0x7f72a41584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:18.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.123+0000 7f72a3fff700 1 --2- 172.21.15.77:0/815697205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72a406bfb0 0x7f72a406d6f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:18.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.123+0000 7f72a3fff700 1 -- 172.21.15.77:0/815697205 >> 172.21.15.77:0/815697205 conn(0x7f72a40cb6b0 msgr2=0x7f72a40cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:18.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.123+0000 7f72a3fff700 1 -- 172.21.15.77:0/815697205 shutdown_connections 2022-04-03T19:44:18.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.124+0000 7f72a3fff700 1 -- 172.21.15.77:0/815697205 wait complete. 2022-04-03T19:44:18.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.125+0000 7f72a3fff700 1 Processor -- start 2022-04-03T19:44:18.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.125+0000 7f72a3fff700 1 -- start start 2022-04-03T19:44:18.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.126+0000 7f72a3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72a406bbe0 0x7f72a4168ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:18.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.126+0000 7f72a3fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72a406bfb0 0x7f72a4152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:18.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.126+0000 7f72a3fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72a406c500 0x7f72a4151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:18.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.126+0000 7f72a3fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a4152d90 con 0x7f72a406bbe0 2022-04-03T19:44:18.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.126+0000 7f72a3fff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a40c3350 con 0x7f72a406bfb0 2022-04-03T19:44:18.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.126+0000 7f72a3fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a40c2740 con 0x7f72a406c500 2022-04-03T19:44:18.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.126+0000 7f72a2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72a406bbe0 0x7f72a4168ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:18.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.126+0000 7f72a27fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72a406bfb0 0x7f72a4152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:18.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.126+0000 7f72a37fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72a406c500 0x7f72a4151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:18.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.126+0000 7f72a2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72a406bbe0 0x7f72a4168ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42478/0 (socket says 172.21.15.77:42478) 2022-04-03T19:44:18.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.126+0000 7f72a2ffd700 1 -- 172.21.15.77:0/1807236011 learned_addr learned my addr 172.21.15.77:0/1807236011 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:18.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.127+0000 7f72a37fe700 1 -- 172.21.15.77:0/1807236011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72a406bfb0 msgr2=0x7f72a4152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:18.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.127+0000 7f72a37fe700 1 --2- 172.21.15.77:0/1807236011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72a406bfb0 0x7f72a4152850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:18.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.127+0000 7f72a37fe700 1 -- 172.21.15.77:0/1807236011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72a406bbe0 msgr2=0x7f72a4168ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:18.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.127+0000 7f72a37fe700 1 --2- 172.21.15.77:0/1807236011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72a406bbe0 0x7f72a4168ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:18.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.127+0000 7f72a37fe700 1 -- 172.21.15.77:0/1807236011 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7298009370 con 0x7f72a406c500 2022-04-03T19:44:18.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.127+0000 7f72a2ffd700 1 --2- 172.21.15.77:0/1807236011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72a406bbe0 0x7f72a4168ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:18.137 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.127+0000 7f72a37fe700 1 --2- 172.21.15.77:0/1807236011 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72a406c500 0x7f72a4151b00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f729c00ecf0 tx=0x7f729c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:18.137 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.127+0000 7f728affd700 1 -- 172.21.15.77:0/1807236011 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729c00c980 con 0x7f72a406c500 2022-04-03T19:44:18.137 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.127+0000 7f728affd700 1 -- 172.21.15.77:0/1807236011 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f729c00f040 con 0x7f72a406c500 2022-04-03T19:44:18.137 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.127+0000 7f72a3fff700 1 -- 172.21.15.77:0/1807236011 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a40c2a20 con 0x7f72a406c500 2022-04-03T19:44:18.138 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.127+0000 7f728affd700 1 -- 172.21.15.77:0/1807236011 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729c00cd50 con 0x7f72a406c500 2022-04-03T19:44:18.138 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.127+0000 7f72a3fff700 1 -- 172.21.15.77:0/1807236011 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a40c2fe0 con 0x7f72a406c500 2022-04-03T19:44:18.138 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.128+0000 7f72a3fff700 1 -- 172.21.15.77:0/1807236011 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a40c3500 con 0x7f72a406c500 2022-04-03T19:44:18.138 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.129+0000 7f728affd700 1 -- 172.21.15.77:0/1807236011 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f729c014690 con 0x7f72a406c500 2022-04-03T19:44:18.138 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.130+0000 7f728affd700 1 --2- 172.21.15.77:0/1807236011 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7284063d60 0x7f7284066210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:18.139 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.130+0000 7f728affd700 1 -- 172.21.15.77:0/1807236011 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f729c081410 con 0x7f72a406c500 2022-04-03T19:44:18.139 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.130+0000 7f72a2ffd700 1 --2- 172.21.15.77:0/1807236011 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7284063d60 0x7f7284066210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:18.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.131+0000 7f72a2ffd700 1 --2- 172.21.15.77:0/1807236011 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7284063d60 0x7f7284066210 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7294009fd0 tx=0x7f7294009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:18.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.134+0000 7f728affd700 1 -- 172.21.15.77:0/1807236011 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f72a40c3500 con 0x7f72a406c500 2022-04-03T19:44:18.330 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.330+0000 7f72a3fff700 1 -- 172.21.15.77:0/1807236011 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f72a40c3500 con 0x7f72a406c500 2022-04-03T19:44:18.331 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.331+0000 7f728affd700 1 -- 172.21.15.77:0/1807236011 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v20) v1 ==== 84+0+126 (secure 0 0 0) 0x7f72a40c3500 con 0x7f72a406c500 2022-04-03T19:44:18.332 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:18.332 INFO:teuthology.orchestra.run.smithi077.stdout:[ 2022-04-03T19:44:18.332 INFO:teuthology.orchestra.run.smithi077.stdout: { 2022-04-03T19:44:18.332 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 1, 2022-04-03T19:44:18.333 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": ".mgr" 2022-04-03T19:44:18.333 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2022-04-03T19:44:18.333 INFO:teuthology.orchestra.run.smithi077.stdout: { 2022-04-03T19:44:18.333 INFO:teuthology.orchestra.run.smithi077.stdout: "poolnum": 2, 2022-04-03T19:44:18.333 INFO:teuthology.orchestra.run.smithi077.stdout: "poolname": "rbd" 2022-04-03T19:44:18.334 INFO:teuthology.orchestra.run.smithi077.stdout: } 2022-04-03T19:44:18.334 INFO:teuthology.orchestra.run.smithi077.stdout:] 2022-04-03T19:44:18.335 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.333+0000 7f72a3fff700 1 -- 172.21.15.77:0/1807236011 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7284063d60 msgr2=0x7f7284066210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:18.335 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.333+0000 7f72a3fff700 1 --2- 172.21.15.77:0/1807236011 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7284063d60 0x7f7284066210 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7294009fd0 tx=0x7f7294009060 comp rx=0 tx=0).stop 2022-04-03T19:44:18.336 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.334+0000 7f72a3fff700 1 -- 172.21.15.77:0/1807236011 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72a406c500 msgr2=0x7f72a4151b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:18.336 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.334+0000 7f72a3fff700 1 --2- 172.21.15.77:0/1807236011 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72a406c500 0x7f72a4151b00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f729c00ecf0 tx=0x7f729c00b040 comp rx=0 tx=0).stop 2022-04-03T19:44:18.336 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.334+0000 7f72a3fff700 1 -- 172.21.15.77:0/1807236011 shutdown_connections 2022-04-03T19:44:18.336 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.334+0000 7f72a3fff700 1 --2- 172.21.15.77:0/1807236011 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7284063d60 0x7f7284066210 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:18.336 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.334+0000 7f72a3fff700 1 --2- 172.21.15.77:0/1807236011 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72a406c500 0x7f72a4151b00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:18.337 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.334+0000 7f72a3fff700 1 --2- 172.21.15.77:0/1807236011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72a406bbe0 0x7f72a4168ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:18.337 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.334+0000 7f72a3fff700 1 --2- 172.21.15.77:0/1807236011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72a406bfb0 0x7f72a4152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:18.337 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.334+0000 7f72a3fff700 1 -- 172.21.15.77:0/1807236011 >> 172.21.15.77:0/1807236011 conn(0x7f72a40cb6b0 msgr2=0x7f72a40c7350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:18.337 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.334+0000 7f72a3fff700 1 -- 172.21.15.77:0/1807236011 shutdown_connections 2022-04-03T19:44:18.337 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.334+0000 7f72a3fff700 1 -- 172.21.15.77:0/1807236011 wait complete. 2022-04-03T19:44:18.358 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-04-03T19:44:18.359 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2022-04-03T19:44:18.560 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.559+0000 7fe6db4e6700 1 Processor -- start 2022-04-03T19:44:18.560 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.559+0000 7fe6db4e6700 1 -- start start 2022-04-03T19:44:18.563 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.560+0000 7fe6db4e6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6d40c3420 0x7fe6d40c2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:18.563 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.560+0000 7fe6db4e6700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe6d40c37f0 0x7fe6d4155b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:18.564 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.560+0000 7fe6db4e6700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d406c8d0 con 0x7fe6d406c430 2022-04-03T19:44:18.564 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.561+0000 7fe6db4e6700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d41556e0 con 0x7fe6d40c37f0 2022-04-03T19:44:18.564 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.561+0000 7fe6db4e6700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d4155040 con 0x7fe6d40c3420 2022-04-03T19:44:18.564 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.561+0000 7fe6d9ce3700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe6d40c37f0 0x7fe6d4155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:18.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.561+0000 7fe6d9ce3700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe6d40c37f0 0x7fe6d4155b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42482/0 (socket says 172.21.15.77:42482) 2022-04-03T19:44:18.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.561+0000 7fe6d9ce3700 1 -- 172.21.15.77:0/2357671913 learned_addr learned my addr 172.21.15.77:0/2357671913 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:18.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.561+0000 7fe6dace5700 1 --1- 172.21.15.77:0/2357671913 >> v1:172.21.15.77:6789/0 conn(0x7fe6d406c430 0x7fe6d40c4ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38284/0 (socket says 172.21.15.77:38284) 2022-04-03T19:44:18.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.561+0000 7fe6da4e4700 1 --2- 172.21.15.77:0/2357671913 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6d40c3420 0x7fe6d40c2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:18.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.561+0000 7fe6d9ce3700 1 -- 172.21.15.77:0/2357671913 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6d40c3420 msgr2=0x7fe6d40c2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:18.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.561+0000 7fe6d9ce3700 1 --2- 172.21.15.77:0/2357671913 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6d40c3420 0x7fe6d40c2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:18.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.561+0000 7fe6d9ce3700 1 -- 172.21.15.77:0/2357671913 >> v1:172.21.15.77:6789/0 conn(0x7fe6d406c430 legacy=0x7fe6d40c4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:18.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.561+0000 7fe6d9ce3700 1 -- 172.21.15.77:0/2357671913 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d40c6270 con 0x7fe6d40c37f0 2022-04-03T19:44:18.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.561+0000 7fe6d94e2700 1 -- 172.21.15.77:0/2357671913 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867319836 0 0) 0x7fe6d406c8d0 con 0x7fe6d406c430 2022-04-03T19:44:18.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.562+0000 7fe6d9ce3700 1 --2- 172.21.15.77:0/2357671913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d40c37f0 0x7fe6d4155b00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4006fd0 tx=0x7fe6c40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c759dde49facdc server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:18.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.562+0000 7fe6d94e2700 1 -- 172.21.15.77:0/2357671913 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4004030 con 0x7fe6d40c37f0 2022-04-03T19:44:18.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.562+0000 7fe6d94e2700 1 -- 172.21.15.77:0/2357671913 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c40041b0 con 0x7fe6d40c37f0 2022-04-03T19:44:18.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.562+0000 7fe6d94e2700 1 -- 172.21.15.77:0/2357671913 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4011630 con 0x7fe6d40c37f0 2022-04-03T19:44:18.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.562+0000 7fe6db4e6700 1 -- 172.21.15.77:0/2357671913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d40c37f0 msgr2=0x7fe6d4155b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:18.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.562+0000 7fe6db4e6700 1 --2- 172.21.15.77:0/2357671913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d40c37f0 0x7fe6d4155b00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4006fd0 tx=0x7fe6c40090f0 comp rx=0 tx=0).stop 2022-04-03T19:44:18.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.563+0000 7fe6db4e6700 1 -- 172.21.15.77:0/2357671913 shutdown_connections 2022-04-03T19:44:18.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.563+0000 7fe6db4e6700 1 --2- 172.21.15.77:0/2357671913 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6d40c3420 0x7fe6d40c2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:18.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.563+0000 7fe6db4e6700 1 --2- 172.21.15.77:0/2357671913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d40c37f0 0x7fe6d4155b00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:18.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.563+0000 7fe6db4e6700 1 -- 172.21.15.77:0/2357671913 >> 172.21.15.77:0/2357671913 conn(0x7fe6d40cb740 msgr2=0x7fe6d40cbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:18.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.563+0000 7fe6db4e6700 1 -- 172.21.15.77:0/2357671913 shutdown_connections 2022-04-03T19:44:18.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.563+0000 7fe6db4e6700 1 -- 172.21.15.77:0/2357671913 wait complete. 2022-04-03T19:44:18.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.564+0000 7fe6db4e6700 1 Processor -- start 2022-04-03T19:44:18.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.564+0000 7fe6db4e6700 1 -- start start 2022-04-03T19:44:18.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.565+0000 7fe6db4e6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d406c430 0x7fe6d4156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:18.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.565+0000 7fe6db4e6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6d40c3420 0x7fe6d414cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:18.573 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.565+0000 7fe6db4e6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6d40c37f0 0x7fe6d41680a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:18.573 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.565+0000 7fe6db4e6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d415af00 con 0x7fe6d406c430 2022-04-03T19:44:18.573 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.565+0000 7fe6db4e6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d40c1300 con 0x7fe6d40c3420 2022-04-03T19:44:18.573 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.565+0000 7fe6db4e6700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d40c15c0 con 0x7fe6d40c37f0 2022-04-03T19:44:18.573 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.565+0000 7fe6da4e4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d406c430 0x7fe6d4156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:18.574 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.566+0000 7fe6da4e4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d406c430 0x7fe6d4156210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42484/0 (socket says 172.21.15.77:42484) 2022-04-03T19:44:18.574 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.566+0000 7fe6da4e4700 1 -- 172.21.15.77:0/254402761 learned_addr learned my addr 172.21.15.77:0/254402761 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:18.574 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.566+0000 7fe6dace5700 1 --2- 172.21.15.77:0/254402761 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6d40c37f0 0x7fe6d41680a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:18.574 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.566+0000 7fe6d9ce3700 1 --2- 172.21.15.77:0/254402761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6d40c3420 0x7fe6d414cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:18.574 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.566+0000 7fe6da4e4700 1 -- 172.21.15.77:0/254402761 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6d40c37f0 msgr2=0x7fe6d41680a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:18.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.566+0000 7fe6da4e4700 1 --2- 172.21.15.77:0/254402761 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6d40c37f0 0x7fe6d41680a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:18.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.566+0000 7fe6da4e4700 1 -- 172.21.15.77:0/254402761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6d40c3420 msgr2=0x7fe6d414cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:18.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.566+0000 7fe6da4e4700 1 --2- 172.21.15.77:0/254402761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6d40c3420 0x7fe6d414cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:18.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.566+0000 7fe6da4e4700 1 -- 172.21.15.77:0/254402761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c4009370 con 0x7fe6d406c430 2022-04-03T19:44:18.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.566+0000 7fe6da4e4700 1 --2- 172.21.15.77:0/254402761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d406c430 0x7fe6d4156210 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000e900 tx=0x7fe6d0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:18.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.566+0000 7fe6c27fc700 1 -- 172.21.15.77:0/254402761 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0013680 con 0x7fe6d406c430 2022-04-03T19:44:18.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.567+0000 7fe6c27fc700 1 -- 172.21.15.77:0/254402761 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d00137e0 con 0x7fe6d406c430 2022-04-03T19:44:18.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.567+0000 7fe6db4e6700 1 -- 172.21.15.77:0/254402761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d40bfa10 con 0x7fe6d406c430 2022-04-03T19:44:18.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.567+0000 7fe6c27fc700 1 -- 172.21.15.77:0/254402761 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0013b60 con 0x7fe6d406c430 2022-04-03T19:44:18.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.567+0000 7fe6db4e6700 1 -- 172.21.15.77:0/254402761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d414fa90 con 0x7fe6d406c430 2022-04-03T19:44:18.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.568+0000 7fe6db4e6700 1 -- 172.21.15.77:0/254402761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d415b1e0 con 0x7fe6d406c430 2022-04-03T19:44:18.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.569+0000 7fe6c27fc700 1 -- 172.21.15.77:0/254402761 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fe6d0019030 con 0x7fe6d406c430 2022-04-03T19:44:18.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.569+0000 7fe6c27fc700 1 --2- 172.21.15.77:0/254402761 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe6b4063ce0 0x7fe6b4066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:18.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.569+0000 7fe6c27fc700 1 -- 172.21.15.77:0/254402761 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fe6d0080280 con 0x7fe6d406c430 2022-04-03T19:44:18.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.570+0000 7fe6d9ce3700 1 --2- 172.21.15.77:0/254402761 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe6b4063ce0 0x7fe6b4066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:18.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.570+0000 7fe6d9ce3700 1 --2- 172.21.15.77:0/254402761 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe6b4063ce0 0x7fe6b4066190 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4009340 tx=0x7fe6c4016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:18.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.574+0000 7fe6c27fc700 1 -- 172.21.15.77:0/254402761 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fe6d004f1b0 con 0x7fe6d406c430 2022-04-03T19:44:18.769 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:18.769+0000 7fe6db4e6700 1 -- 172.21.15.77:0/254402761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe6d40bf870 con 0x7fe6d406c430 2022-04-03T19:44:19.764 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:19.764+0000 7fe6c27fc700 1 -- 172.21.15.77:0/254402761 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v21) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe6d40bf870 con 0x7fe6d406c430 2022-04-03T19:44:19.774 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:19.773+0000 7fe6db4e6700 1 -- 172.21.15.77:0/254402761 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe6b4063ce0 msgr2=0x7fe6b4066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:19.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:19.773+0000 7fe6db4e6700 1 --2- 172.21.15.77:0/254402761 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe6b4063ce0 0x7fe6b4066190 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4009340 tx=0x7fe6c4016000 comp rx=0 tx=0).stop 2022-04-03T19:44:19.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:19.774+0000 7fe6db4e6700 1 -- 172.21.15.77:0/254402761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d406c430 msgr2=0x7fe6d4156210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:19.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:19.774+0000 7fe6db4e6700 1 --2- 172.21.15.77:0/254402761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d406c430 0x7fe6d4156210 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000e900 tx=0x7fe6d0009060 comp rx=0 tx=0).stop 2022-04-03T19:44:19.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:19.774+0000 7fe6db4e6700 1 -- 172.21.15.77:0/254402761 shutdown_connections 2022-04-03T19:44:19.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:19.774+0000 7fe6db4e6700 1 --2- 172.21.15.77:0/254402761 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe6b4063ce0 0x7fe6b4066190 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:19.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:19.774+0000 7fe6db4e6700 1 --2- 172.21.15.77:0/254402761 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6d40c37f0 0x7fe6d41680a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:19.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:19.774+0000 7fe6db4e6700 1 --2- 172.21.15.77:0/254402761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe6d406c430 0x7fe6d4156210 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:19.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:19.774+0000 7fe6db4e6700 1 --2- 172.21.15.77:0/254402761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6d40c3420 0x7fe6d414cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:19.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:19.774+0000 7fe6db4e6700 1 -- 172.21.15.77:0/254402761 >> 172.21.15.77:0/254402761 conn(0x7fe6d40cb740 msgr2=0x7fe6d40ca060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:19.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:19.774+0000 7fe6db4e6700 1 -- 172.21.15.77:0/254402761 shutdown_connections 2022-04-03T19:44:19.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:19.774+0000 7fe6db4e6700 1 -- 172.21.15.77:0/254402761 wait complete. 2022-04-03T19:44:19.777 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_metadata' created 2022-04-03T19:44:19.801 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2022-04-03T19:44:20.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.008+0000 7f7a77328700 1 Processor -- start 2022-04-03T19:44:20.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.009+0000 7f7a77328700 1 -- start start 2022-04-03T19:44:20.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.010+0000 7f7a77328700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7a700c5450 0x7f7a70159070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:20.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.010+0000 7f7a77328700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a700c5820 0x7f7a700c5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:20.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.010+0000 7f7a77328700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a70158be0 con 0x7f7a7006bb20 2022-04-03T19:44:20.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.010+0000 7f7a77328700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a701557b0 con 0x7f7a700c5450 2022-04-03T19:44:20.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.010+0000 7f7a77328700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a70155a90 con 0x7f7a700c5820 2022-04-03T19:44:20.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.010+0000 7f7a76326700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7a700c5450 0x7f7a70159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:20.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.010+0000 7f7a76b27700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a700c5820 0x7f7a700c5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:20.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.010+0000 7f7a75b25700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f7a7006bb20 0x7f7a70154f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44086/0 (socket says 172.21.15.77:44086) 2022-04-03T19:44:20.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.010+0000 7f7a75b25700 1 -- 172.21.15.77:0/774996961 learned_addr learned my addr 172.21.15.77:0/774996961 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:20.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.010+0000 7f7a75324700 1 -- 172.21.15.77:0/774996961 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901263169 0 0) 0x7f7a70158be0 con 0x7f7a7006bb20 2022-04-03T19:44:20.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.010+0000 7f7a75324700 1 -- 172.21.15.77:0/774996961 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a580033f0 con 0x7f7a7006bb20 2022-04-03T19:44:20.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.011+0000 7f7a76326700 1 -- 172.21.15.77:0/774996961 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a700c5820 msgr2=0x7f7a700c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:20.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.011+0000 7f7a76326700 1 --2- 172.21.15.77:0/774996961 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a700c5820 0x7f7a700c5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:20.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.011+0000 7f7a76326700 1 -- 172.21.15.77:0/774996961 >> v1:172.21.15.186:6789/0 conn(0x7f7a7006bb20 legacy=0x7f7a70154f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:20.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.011+0000 7f7a76326700 1 -- 172.21.15.77:0/774996961 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a70155d70 con 0x7f7a700c5450 2022-04-03T19:44:20.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.011+0000 7f7a76b27700 1 --2- 172.21.15.77:0/774996961 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a700c5820 0x7f7a700c5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:20.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.011+0000 7f7a76326700 1 --2- 172.21.15.77:0/774996961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a700c5450 0x7f7a70159070 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c009760 tx=0x7f7a6c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78d3e711dc80c563 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:20.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.011+0000 7f7a75324700 1 -- 172.21.15.77:0/774996961 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c004030 con 0x7f7a700c5450 2022-04-03T19:44:20.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.011+0000 7f7a75324700 1 -- 172.21.15.77:0/774996961 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6c004190 con 0x7f7a700c5450 2022-04-03T19:44:20.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.011+0000 7f7a75324700 1 -- 172.21.15.77:0/774996961 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c011630 con 0x7f7a700c5450 2022-04-03T19:44:20.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.012+0000 7f7a77328700 1 -- 172.21.15.77:0/774996961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a700c5450 msgr2=0x7f7a70159070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:20.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.012+0000 7f7a77328700 1 --2- 172.21.15.77:0/774996961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a700c5450 0x7f7a70159070 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c009760 tx=0x7f7a6c009060 comp rx=0 tx=0).stop 2022-04-03T19:44:20.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.012+0000 7f7a77328700 1 -- 172.21.15.77:0/774996961 shutdown_connections 2022-04-03T19:44:20.021 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.012+0000 7f7a77328700 1 --2- 172.21.15.77:0/774996961 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a700c5820 0x7f7a700c5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:20.021 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.012+0000 7f7a77328700 1 --2- 172.21.15.77:0/774996961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a700c5450 0x7f7a70159070 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:20.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.012+0000 7f7a77328700 1 -- 172.21.15.77:0/774996961 >> 172.21.15.77:0/774996961 conn(0x7f7a700626d0 msgr2=0x7f7a70062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:20.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.012+0000 7f7a77328700 1 -- 172.21.15.77:0/774996961 shutdown_connections 2022-04-03T19:44:20.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.012+0000 7f7a77328700 1 -- 172.21.15.77:0/774996961 wait complete. 2022-04-03T19:44:20.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.013+0000 7f7a77328700 1 Processor -- start 2022-04-03T19:44:20.023 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.014+0000 7f7a77328700 1 -- start start 2022-04-03T19:44:20.023 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.014+0000 7f7a77328700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a700c5450 0x7f7a70154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:20.023 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.014+0000 7f7a77328700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a700c5820 0x7f7a70153f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:20.023 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.014+0000 7f7a77328700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a701650b0 0x7f7a70165520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:20.024 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.015+0000 7f7a77328700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a70165a60 con 0x7f7a701650b0 2022-04-03T19:44:20.024 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.015+0000 7f7a77328700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a701553e0 con 0x7f7a700c5450 2022-04-03T19:44:20.024 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.015+0000 7f7a77328700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a70164bb0 con 0x7f7a700c5820 2022-04-03T19:44:20.024 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.015+0000 7f7a76b27700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a701650b0 0x7f7a70165520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:20.024 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.015+0000 7f7a76326700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a700c5450 0x7f7a70154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:20.025 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.015+0000 7f7a75b25700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a700c5820 0x7f7a70153f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:20.025 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.015+0000 7f7a76b27700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a701650b0 0x7f7a70165520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42486/0 (socket says 172.21.15.77:42486) 2022-04-03T19:44:20.025 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.015+0000 7f7a76b27700 1 -- 172.21.15.77:0/3126855005 learned_addr learned my addr 172.21.15.77:0/3126855005 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:20.025 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.015+0000 7f7a76b27700 1 -- 172.21.15.77:0/3126855005 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a700c5820 msgr2=0x7f7a70153f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:20.025 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.015+0000 7f7a76b27700 1 --2- 172.21.15.77:0/3126855005 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a700c5820 0x7f7a70153f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:20.026 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.015+0000 7f7a76b27700 1 -- 172.21.15.77:0/3126855005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a700c5450 msgr2=0x7f7a70154ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:20.026 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.015+0000 7f7a76b27700 1 --2- 172.21.15.77:0/3126855005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a700c5450 0x7f7a70154ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:20.026 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.015+0000 7f7a76b27700 1 -- 172.21.15.77:0/3126855005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a68009a20 con 0x7f7a701650b0 2022-04-03T19:44:20.026 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.015+0000 7f7a76326700 1 --2- 172.21.15.77:0/3126855005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a700c5450 0x7f7a70154ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:20.027 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.016+0000 7f7a76b27700 1 --2- 172.21.15.77:0/3126855005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a701650b0 0x7f7a70165520 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7a68009f50 tx=0x7f7a6800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:20.027 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.016+0000 7f7a5e7fc700 1 -- 172.21.15.77:0/3126855005 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6800caf0 con 0x7f7a701650b0 2022-04-03T19:44:20.027 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.016+0000 7f7a5e7fc700 1 -- 172.21.15.77:0/3126855005 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6800f040 con 0x7f7a701650b0 2022-04-03T19:44:20.027 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.016+0000 7f7a5e7fc700 1 -- 172.21.15.77:0/3126855005 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6800ce80 con 0x7f7a701650b0 2022-04-03T19:44:20.028 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.016+0000 7f7a77328700 1 -- 172.21.15.77:0/3126855005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6c0092a0 con 0x7f7a701650b0 2022-04-03T19:44:20.028 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.016+0000 7f7a77328700 1 -- 172.21.15.77:0/3126855005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a70151c60 con 0x7f7a701650b0 2022-04-03T19:44:20.028 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.017+0000 7f7a77328700 1 -- 172.21.15.77:0/3126855005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a70164d70 con 0x7f7a701650b0 2022-04-03T19:44:20.028 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.018+0000 7f7a5e7fc700 1 -- 172.21.15.77:0/3126855005 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f7a68014690 con 0x7f7a701650b0 2022-04-03T19:44:20.028 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.019+0000 7f7a5e7fc700 1 --2- 172.21.15.77:0/3126855005 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7a50063ce0 0x7f7a50066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:20.029 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.019+0000 7f7a5e7fc700 1 -- 172.21.15.77:0/3126855005 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5576+0+0 (secure 0 0 0) 0x7f7a680815c0 con 0x7f7a701650b0 2022-04-03T19:44:20.029 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.019+0000 7f7a76326700 1 --2- 172.21.15.77:0/3126855005 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7a50063ce0 0x7f7a50066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:20.030 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.020+0000 7f7a76326700 1 --2- 172.21.15.77:0/3126855005 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7a50063ce0 0x7f7a50066190 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c009fd0 tx=0x7f7a6c016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:20.030 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.023+0000 7f7a5e7fc700 1 -- 172.21.15.77:0/3126855005 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f7a68050320 con 0x7f7a701650b0 2022-04-03T19:44:20.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.220+0000 7f7a77328700 1 -- 172.21.15.77:0/3126855005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7a70158590 con 0x7f7a701650b0 2022-04-03T19:44:20.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.775+0000 7f7a5e7fc700 1 -- 172.21.15.77:0/3126855005 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v22) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7a68053940 con 0x7f7a701650b0 2022-04-03T19:44:20.790 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.786+0000 7f7a77328700 1 -- 172.21.15.77:0/3126855005 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7a50063ce0 msgr2=0x7f7a50066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:20.791 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.786+0000 7f7a77328700 1 --2- 172.21.15.77:0/3126855005 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7a50063ce0 0x7f7a50066190 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c009fd0 tx=0x7f7a6c016000 comp rx=0 tx=0).stop 2022-04-03T19:44:20.791 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.786+0000 7f7a77328700 1 -- 172.21.15.77:0/3126855005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a701650b0 msgr2=0x7f7a70165520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:20.791 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.786+0000 7f7a77328700 1 --2- 172.21.15.77:0/3126855005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a701650b0 0x7f7a70165520 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7a68009f50 tx=0x7f7a6800b040 comp rx=0 tx=0).stop 2022-04-03T19:44:20.791 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.788+0000 7f7a77328700 1 -- 172.21.15.77:0/3126855005 shutdown_connections 2022-04-03T19:44:20.792 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.788+0000 7f7a77328700 1 --2- 172.21.15.77:0/3126855005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a701650b0 0x7f7a70165520 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:20.792 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.788+0000 7f7a77328700 1 --2- 172.21.15.77:0/3126855005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a700c5450 0x7f7a70154ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:20.792 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.788+0000 7f7a77328700 1 --2- 172.21.15.77:0/3126855005 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7a50063ce0 0x7f7a50066190 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:20.792 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.788+0000 7f7a77328700 1 --2- 172.21.15.77:0/3126855005 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a700c5820 0x7f7a70153f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:20.793 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.788+0000 7f7a77328700 1 -- 172.21.15.77:0/3126855005 >> 172.21.15.77:0/3126855005 conn(0x7f7a700626d0 msgr2=0x7f7a700c95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:20.793 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.789+0000 7f7a77328700 1 -- 172.21.15.77:0/3126855005 shutdown_connections 2022-04-03T19:44:20.793 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:20.789+0000 7f7a77328700 1 -- 172.21.15.77:0/3126855005 wait complete. 2022-04-03T19:44:20.795 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'cephfs_data' created 2022-04-03T19:44:20.819 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2022-04-03T19:44:21.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.019+0000 7f98f886c700 1 Processor -- start 2022-04-03T19:44:21.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.019+0000 7f98f886c700 1 -- start start 2022-04-03T19:44:21.021 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.020+0000 7f98f886c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f98f40c3c60 0x7f98f4066da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:21.021 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.020+0000 7f98f886c700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f4069f60 con 0x7f98f414f250 2022-04-03T19:44:21.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.020+0000 7f98f886c700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f4155680 con 0x7f98f414f620 2022-04-03T19:44:21.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.020+0000 7f98f886c700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98f4152e40 con 0x7f98f40c3c60 2022-04-03T19:44:21.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.021+0000 7f98f37fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f98f414f250 0x7f98f4155b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38286/0 (socket says 172.21.15.77:38286) 2022-04-03T19:44:21.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.021+0000 7f98f37fe700 1 -- 172.21.15.77:0/1149606986 learned_addr learned my addr 172.21.15.77:0/1149606986 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:21.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.021+0000 7f98f2ffd700 1 --2- 172.21.15.77:0/1149606986 >> v2:172.21.15.186:3300/0 conn(0x7f98f40c3c60 0x7f98f4066da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:21.025 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.021+0000 7f98f27fc700 1 -- 172.21.15.77:0/1149606986 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062988143 0 0) 0x7f98f4069f60 con 0x7f98f414f250 2022-04-03T19:44:21.025 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.021+0000 7f98f27fc700 1 -- 172.21.15.77:0/1149606986 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98e00033f0 con 0x7f98f414f250 2022-04-03T19:44:21.025 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.021+0000 7f98f27fc700 1 -- 172.21.15.77:0/1149606986 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821118308 0 0) 0x7f98f4155680 con 0x7f98f414f620 2022-04-03T19:44:21.025 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.021+0000 7f98f27fc700 1 -- 172.21.15.77:0/1149606986 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98f4069f60 con 0x7f98f414f620 2022-04-03T19:44:21.026 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.021+0000 7f98f2ffd700 1 -- 172.21.15.77:0/1149606986 >> v1:172.21.15.186:6789/0 conn(0x7f98f414f620 legacy=0x7f98f415c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:21.026 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.021+0000 7f98f2ffd700 1 -- 172.21.15.77:0/1149606986 >> v1:172.21.15.77:6789/0 conn(0x7f98f414f250 legacy=0x7f98f4155b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:21.026 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.021+0000 7f98f2ffd700 1 -- 172.21.15.77:0/1149606986 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f41522c0 con 0x7f98f40c3c60 2022-04-03T19:44:21.026 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.022+0000 7f98f2ffd700 1 --2- 172.21.15.77:0/1149606986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98f40c3c60 0x7f98f4066da0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f98e8009760 tx=0x7f98e8009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=479f534bbbaf0663 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:21.027 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.022+0000 7f98f27fc700 1 -- 172.21.15.77:0/1149606986 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e80105b0 con 0x7f98f40c3c60 2022-04-03T19:44:21.027 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.022+0000 7f98f27fc700 1 -- 172.21.15.77:0/1149606986 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e8010710 con 0x7f98f40c3c60 2022-04-03T19:44:21.027 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.022+0000 7f98f27fc700 1 -- 172.21.15.77:0/1149606986 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e8010a90 con 0x7f98f40c3c60 2022-04-03T19:44:21.027 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.022+0000 7f98f886c700 1 -- 172.21.15.77:0/1149606986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98f40c3c60 msgr2=0x7f98f4066da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:21.027 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.022+0000 7f98f886c700 1 --2- 172.21.15.77:0/1149606986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98f40c3c60 0x7f98f4066da0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f98e8009760 tx=0x7f98e8009060 comp rx=0 tx=0).stop 2022-04-03T19:44:21.028 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.023+0000 7f98f886c700 1 -- 172.21.15.77:0/1149606986 shutdown_connections 2022-04-03T19:44:21.028 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.023+0000 7f98f886c700 1 --2- 172.21.15.77:0/1149606986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98f40c3c60 0x7f98f4066da0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:21.028 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.023+0000 7f98f886c700 1 -- 172.21.15.77:0/1149606986 >> 172.21.15.77:0/1149606986 conn(0x7f98f40626d0 msgr2=0x7f98f4062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:21.028 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.023+0000 7f98f886c700 1 -- 172.21.15.77:0/1149606986 shutdown_connections 2022-04-03T19:44:21.028 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.023+0000 7f98f886c700 1 -- 172.21.15.77:0/1149606986 wait complete. 2022-04-03T19:44:21.029 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.024+0000 7f98f886c700 1 Processor -- start 2022-04-03T19:44:21.029 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.024+0000 7f98f886c700 1 -- start start 2022-04-03T19:44:21.031 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.025+0000 7f98f886c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98f40c3c60 0x7f98f40c0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:21.032 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.025+0000 7f98f886c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98f414f250 0x7f98f4165e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:21.032 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.025+0000 7f98f886c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98f414f620 0x7f98f4166360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:21.032 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.025+0000 7f98f886c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f4165a00 con 0x7f98f414f620 2022-04-03T19:44:21.032 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.025+0000 7f98f886c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f40bc2e0 con 0x7f98f414f250 2022-04-03T19:44:21.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.025+0000 7f98f886c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f40c5bc0 con 0x7f98f40c3c60 2022-04-03T19:44:21.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.025+0000 7f98f2ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98f414f250 0x7f98f4165e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:21.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.025+0000 7f98f3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98f414f620 0x7f98f4166360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:21.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.025+0000 7f98f37fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98f40c3c60 0x7f98f40c0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:21.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.025+0000 7f98f2ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98f414f250 0x7f98f4165e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37670/0 (socket says 172.21.15.77:37670) 2022-04-03T19:44:21.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.026+0000 7f98f2ffd700 1 -- 172.21.15.77:0/3217226502 learned_addr learned my addr 172.21.15.77:0/3217226502 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:21.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.026+0000 7f98f2ffd700 1 -- 172.21.15.77:0/3217226502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98f40c3c60 msgr2=0x7f98f40c0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:21.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.026+0000 7f98f2ffd700 1 --2- 172.21.15.77:0/3217226502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98f40c3c60 0x7f98f40c0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:21.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.026+0000 7f98f2ffd700 1 -- 172.21.15.77:0/3217226502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98f414f620 msgr2=0x7f98f4166360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:21.035 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.026+0000 7f98f2ffd700 1 --2- 172.21.15.77:0/3217226502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98f414f620 0x7f98f4166360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:21.035 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.026+0000 7f98f2ffd700 1 -- 172.21.15.77:0/3217226502 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e80092a0 con 0x7f98f414f250 2022-04-03T19:44:21.035 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.026+0000 7f98f2ffd700 1 --2- 172.21.15.77:0/3217226502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98f414f250 0x7f98f4165e20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f98e8009d50 tx=0x7f98e8016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:21.035 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.027+0000 7f98db7fe700 1 -- 172.21.15.77:0/3217226502 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e8000bc0 con 0x7f98f414f250 2022-04-03T19:44:21.035 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.027+0000 7f98db7fe700 1 -- 172.21.15.77:0/3217226502 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e8020070 con 0x7f98f414f250 2022-04-03T19:44:21.036 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.027+0000 7f98f886c700 1 -- 172.21.15.77:0/3217226502 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f40bb250 con 0x7f98f414f250 2022-04-03T19:44:21.036 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.027+0000 7f98db7fe700 1 -- 172.21.15.77:0/3217226502 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e8025430 con 0x7f98f414f250 2022-04-03T19:44:21.036 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.027+0000 7f98f886c700 1 -- 172.21.15.77:0/3217226502 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98f40bb810 con 0x7f98f414f250 2022-04-03T19:44:21.036 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.028+0000 7f98f886c700 1 -- 172.21.15.77:0/3217226502 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98f4165b40 con 0x7f98f414f250 2022-04-03T19:44:21.036 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.029+0000 7f98db7fe700 1 -- 172.21.15.77:0/3217226502 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f98e802b020 con 0x7f98f414f250 2022-04-03T19:44:21.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.029+0000 7f98db7fe700 1 --2- 172.21.15.77:0/3217226502 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f98d4063d10 0x7f98d40661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:21.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.030+0000 7f98db7fe700 1 -- 172.21.15.77:0/3217226502 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5958+0+0 (secure 0 0 0) 0x7f98e80821b0 con 0x7f98f414f250 2022-04-03T19:44:21.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.030+0000 7f98f37fe700 1 --2- 172.21.15.77:0/3217226502 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f98d4063d10 0x7f98d40661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:21.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.030+0000 7f98f37fe700 1 --2- 172.21.15.77:0/3217226502 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f98d4063d10 0x7f98d40661c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f98e4007fd0 tx=0x7f98e4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:21.038 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.033+0000 7f98db7fe700 1 -- 172.21.15.77:0/3217226502 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f98e8050e70 con 0x7f98f414f250 2022-04-03T19:44:21.273 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.273+0000 7f98f886c700 1 -- 172.21.15.77:0/3217226502 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f98f4165b40 con 0x7f98f414f250 2022-04-03T19:44:21.284 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.283+0000 7f98db7fe700 1 -- 172.21.15.77:0/3217226502 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f98f4165b40 con 0x7f98f414f250 2022-04-03T19:44:21.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.292+0000 7f98f886c700 1 -- 172.21.15.77:0/3217226502 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f98d4063d10 msgr2=0x7f98d40661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:21.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.292+0000 7f98f886c700 1 --2- 172.21.15.77:0/3217226502 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f98d4063d10 0x7f98d40661c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f98e4007fd0 tx=0x7f98e4009040 comp rx=0 tx=0).stop 2022-04-03T19:44:21.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.292+0000 7f98f886c700 1 -- 172.21.15.77:0/3217226502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98f414f250 msgr2=0x7f98f4165e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:21.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.292+0000 7f98f886c700 1 --2- 172.21.15.77:0/3217226502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98f414f250 0x7f98f4165e20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f98e8009d50 tx=0x7f98e8016000 comp rx=0 tx=0).stop 2022-04-03T19:44:21.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.294+0000 7f98f886c700 1 -- 172.21.15.77:0/3217226502 shutdown_connections 2022-04-03T19:44:21.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.294+0000 7f98f886c700 1 --2- 172.21.15.77:0/3217226502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98f414f620 0x7f98f4166360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:21.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.294+0000 7f98f886c700 1 --2- 172.21.15.77:0/3217226502 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f98d4063d10 0x7f98d40661c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:21.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.294+0000 7f98f886c700 1 --2- 172.21.15.77:0/3217226502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98f40c3c60 0x7f98f40c0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:21.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.294+0000 7f98f886c700 1 --2- 172.21.15.77:0/3217226502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98f414f250 0x7f98f4165e20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:21.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.295+0000 7f98f886c700 1 -- 172.21.15.77:0/3217226502 >> 172.21.15.77:0/3217226502 conn(0x7f98f40626d0 msgr2=0x7f98f40c2330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:21.306 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.306+0000 7f98f886c700 1 -- 172.21.15.77:0/3217226502 shutdown_connections 2022-04-03T19:44:21.308 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.307+0000 7f98f886c700 1 -- 172.21.15.77:0/3217226502 wait complete. 2022-04-03T19:44:21.311 INFO:teuthology.orchestra.run.smithi077.stderr:new fs with metadata pool 3 and data pool 4 2022-04-03T19:44:21.334 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-04-03T19:44:21.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.531+0000 7f5d30c58700 1 Processor -- start 2022-04-03T19:44:21.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.531+0000 7f5d30c58700 1 -- start start 2022-04-03T19:44:21.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.532+0000 7f5d30c58700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5d2c0c1170 0x7f5d2c1527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:21.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.533+0000 7f5d30c58700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2c0c3710 con 0x7f5d2c155370 2022-04-03T19:44:21.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.533+0000 7f5d30c58700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2c152330 con 0x7f5d2c0c3b10 2022-04-03T19:44:21.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.533+0000 7f5d30c58700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d2c0c1540 con 0x7f5d2c0c1170 2022-04-03T19:44:21.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.533+0000 7f5d2affd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5d2c155370 0x7f5d2c069110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38288/0 (socket says 172.21.15.77:38288) 2022-04-03T19:44:21.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.533+0000 7f5d2affd700 1 -- 172.21.15.77:0/174040521 learned_addr learned my addr 172.21.15.77:0/174040521 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:21.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.533+0000 7f5d2b7fe700 1 --2- 172.21.15.77:0/174040521 >> v2:172.21.15.186:3300/0 conn(0x7f5d2c0c1170 0x7f5d2c1527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:21.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.533+0000 7f5d2a7fc700 1 -- 172.21.15.77:0/174040521 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126248997 0 0) 0x7f5d2c152330 con 0x7f5d2c0c3b10 2022-04-03T19:44:21.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.533+0000 7f5d2a7fc700 1 -- 172.21.15.77:0/174040521 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d180033f0 con 0x7f5d2c0c3b10 2022-04-03T19:44:21.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.533+0000 7f5d2b7fe700 1 -- 172.21.15.77:0/174040521 >> v1:172.21.15.186:6789/0 conn(0x7f5d2c0c3b10 legacy=0x7f5d2c15c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:21.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.534+0000 7f5d2a7fc700 1 -- 172.21.15.77:0/174040521 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192855234 0 0) 0x7f5d2c0c3710 con 0x7f5d2c155370 2022-04-03T19:44:21.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.534+0000 7f5d2b7fe700 1 -- 172.21.15.77:0/174040521 >> v1:172.21.15.77:6789/0 conn(0x7f5d2c155370 legacy=0x7f5d2c069110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:21.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.534+0000 7f5d2b7fe700 1 -- 172.21.15.77:0/174040521 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2c0c4270 con 0x7f5d2c0c1170 2022-04-03T19:44:21.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.534+0000 7f5d2b7fe700 1 --2- 172.21.15.77:0/174040521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2c0c1170 0x7f5d2c1527c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009760 tx=0x7f5d1c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=361d687a30f79db8 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:21.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.534+0000 7f5d2a7fc700 1 -- 172.21.15.77:0/174040521 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c011630 con 0x7f5d2c0c1170 2022-04-03T19:44:21.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.534+0000 7f5d2a7fc700 1 -- 172.21.15.77:0/174040521 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1c011790 con 0x7f5d2c0c1170 2022-04-03T19:44:21.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.534+0000 7f5d2a7fc700 1 -- 172.21.15.77:0/174040521 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c011b10 con 0x7f5d2c0c1170 2022-04-03T19:44:21.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.535+0000 7f5d30c58700 1 -- 172.21.15.77:0/174040521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2c0c1170 msgr2=0x7f5d2c1527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:21.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.535+0000 7f5d30c58700 1 --2- 172.21.15.77:0/174040521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2c0c1170 0x7f5d2c1527c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009760 tx=0x7f5d1c009060 comp rx=0 tx=0).stop 2022-04-03T19:44:21.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.535+0000 7f5d30c58700 1 -- 172.21.15.77:0/174040521 shutdown_connections 2022-04-03T19:44:21.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.535+0000 7f5d30c58700 1 --2- 172.21.15.77:0/174040521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2c0c1170 0x7f5d2c1527c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:21.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.535+0000 7f5d30c58700 1 -- 172.21.15.77:0/174040521 >> 172.21.15.77:0/174040521 conn(0x7f5d2c066da0 msgr2=0x7f5d2c0671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:21.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.535+0000 7f5d30c58700 1 -- 172.21.15.77:0/174040521 shutdown_connections 2022-04-03T19:44:21.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.535+0000 7f5d30c58700 1 -- 172.21.15.77:0/174040521 wait complete. 2022-04-03T19:44:21.542 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.536+0000 7f5d30c58700 1 Processor -- start 2022-04-03T19:44:21.543 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.537+0000 7f5d30c58700 1 -- start start 2022-04-03T19:44:21.543 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.537+0000 7f5d30c58700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2c0c1170 0x7f5d2c14aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:21.543 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.537+0000 7f5d30c58700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d2c0c3b10 0x7f5d2c14b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:21.543 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.537+0000 7f5d30c58700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d2c155370 0x7f5d2c145b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:21.544 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.538+0000 7f5d30c58700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c0bd660 con 0x7f5d2c0c3b10 2022-04-03T19:44:21.544 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.538+0000 7f5d30c58700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c168f20 con 0x7f5d2c0c1170 2022-04-03T19:44:21.544 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.538+0000 7f5d30c58700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c0c5450 con 0x7f5d2c155370 2022-04-03T19:44:21.544 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.538+0000 7f5d2affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d2c0c3b10 0x7f5d2c14b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:21.544 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.538+0000 7f5d2affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d2c0c3b10 0x7f5d2c14b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42490/0 (socket says 172.21.15.77:42490) 2022-04-03T19:44:21.545 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.538+0000 7f5d2affd700 1 -- 172.21.15.77:0/262695796 learned_addr learned my addr 172.21.15.77:0/262695796 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:21.545 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.538+0000 7f5d2b7fe700 1 --2- 172.21.15.77:0/262695796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2c0c1170 0x7f5d2c14aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:21.545 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.538+0000 7f5d2bfff700 1 --2- 172.21.15.77:0/262695796 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d2c155370 0x7f5d2c145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:21.545 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.538+0000 7f5d2affd700 1 -- 172.21.15.77:0/262695796 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d2c155370 msgr2=0x7f5d2c145b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:21.545 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.538+0000 7f5d2affd700 1 --2- 172.21.15.77:0/262695796 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d2c155370 0x7f5d2c145b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.538+0000 7f5d2affd700 1 -- 172.21.15.77:0/262695796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2c0c1170 msgr2=0x7f5d2c14aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.538+0000 7f5d2affd700 1 --2- 172.21.15.77:0/262695796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2c0c1170 0x7f5d2c14aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.538+0000 7f5d2affd700 1 -- 172.21.15.77:0/262695796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d1c0092a0 con 0x7f5d2c0c3b10 2022-04-03T19:44:21.546 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.538+0000 7f5d2b7fe700 1 --2- 172.21.15.77:0/262695796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2c0c1170 0x7f5d2c14aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.538+0000 7f5d2affd700 1 --2- 172.21.15.77:0/262695796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d2c0c3b10 0x7f5d2c14b000 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5d2000c440 tx=0x7f5d20007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.539+0000 7f5d137fe700 1 -- 172.21.15.77:0/262695796 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d20007360 con 0x7f5d2c0c3b10 2022-04-03T19:44:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.539+0000 7f5d137fe700 1 -- 172.21.15.77:0/262695796 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d200074c0 con 0x7f5d2c0c3b10 2022-04-03T19:44:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.539+0000 7f5d137fe700 1 -- 172.21.15.77:0/262695796 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d20004900 con 0x7f5d2c0c3b10 2022-04-03T19:44:21.547 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.539+0000 7f5d30c58700 1 -- 172.21.15.77:0/262695796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d2c164dc0 con 0x7f5d2c0c3b10 2022-04-03T19:44:21.548 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.539+0000 7f5d30c58700 1 -- 172.21.15.77:0/262695796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d2c168350 con 0x7f5d2c0c3b10 2022-04-03T19:44:21.548 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.540+0000 7f5d30c58700 1 -- 172.21.15.77:0/262695796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d2c169060 con 0x7f5d2c0c3b10 2022-04-03T19:44:21.548 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.541+0000 7f5d137fe700 1 -- 172.21.15.77:0/262695796 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f5d20010030 con 0x7f5d2c0c3b10 2022-04-03T19:44:21.548 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.542+0000 7f5d137fe700 1 --2- 172.21.15.77:0/262695796 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5d0c063ce0 0x7f5d0c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:21.548 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.542+0000 7f5d137fe700 1 -- 172.21.15.77:0/262695796 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f5d20080280 con 0x7f5d2c0c3b10 2022-04-03T19:44:21.549 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.542+0000 7f5d2b7fe700 1 --2- 172.21.15.77:0/262695796 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5d0c063ce0 0x7f5d0c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:21.549 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.543+0000 7f5d2b7fe700 1 --2- 172.21.15.77:0/262695796 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5d0c063ce0 0x7f5d0c066190 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009fd0 tx=0x7f5d1c017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:21.550 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.546+0000 7f5d137fe700 1 -- 172.21.15.77:0/262695796 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f5d2004eee0 con 0x7f5d2c0c3b10 2022-04-03T19:44:21.740 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.740+0000 7f5d30c58700 1 -- 172.21.15.77:0/262695796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d2c04b050 con 0x7f5d2c0c3b10 2022-04-03T19:44:21.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.741+0000 7f5d137fe700 1 -- 172.21.15.77:0/262695796 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16858 (secure 0 0 0) 0x7f5d20052500 con 0x7f5d2c0c3b10 2022-04-03T19:44:21.741 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:21.742 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"fsid":"89c3a3f4-599d-4c4a-8f5a-7180eafb7292","created":"2022-04-03T19:43:54.101559+0000","modified":"2022-04-03T19:44:21.275377+0000","last_up_change":"2022-04-03T19:44:05.589174+0000","last_in_change":"2022-04-03T19:43:56.729760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-03T19:44:07.567638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-04-03T19:44:13.149481+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-04-03T19:44:18.823926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-04-03T19:44:20.275479+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f35f5195-3729-4d78-85d1-0539f259283f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6823","nonce":20252},{"type":"v1","addr":"172.21.15.77:6825","nonce":20252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":20252},{"type":"v1","addr":"172.21.15.77:6827","nonce":20252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":20252},{"type":"v1","addr":"172.21.15.77:6831","nonce":20252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":20252},{"type":"v1","addr":"172.21.15.77:6829","nonce":20252}]},"public_addr":"172.21.15.77:6825/20252","cluster_addr":"172.21.15.77:6827/20252","heartbeat_back_addr":"172.21.15.77:6831/20252","heartbeat_front_addr":"172.21.15.77:6829/20252","state":["exists","up"]},{"osd":1,"uuid":"ab1c89dc-57ce-4cc4-b8c7-9b0c19939601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":20249},{"type":"v1","addr":"172.21.15.77:6805","nonce":20249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":20249},{"type":"v1","addr":"172.21.15.77:6809","nonce":20249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":20249},{"type":"v1","addr":"172.21.15.77:6819","nonce":20249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":20249},{"type":"v1","addr":"172.21.15.77:6815","nonce":20249}]},"public_addr":"172.21.15.77:6805/20249","cluster_addr":"172.21.15.77:6809/20249","heartbeat_back_addr":"172.21.15.77:6819/20249","heartbeat_front_addr":"172.21.15.77:6815/20249","state":["exists","up"]},{"osd":2,"uuid":"351e218b-7749-4ee9-b44a-0a5b52fb9be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":20248},{"type":"v1","addr":"172.21.15.77:6801","nonce":20248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":20248},{"type":"v1","addr":"172.21.15.77:6803","nonce":20248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":20248},{"type":"v1","addr":"172.21.15.77:6813","nonce":20248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":20248},{"type":"v1","addr":"172.21.15.77:6808","nonce":20248}]},"public_addr":"172.21.15.77:6801/20248","cluster_addr":"172.21.15.77:6803/20248","heartbeat_back_addr":"172.21.15.77:6813/20248","heartbeat_front_addr":"172.21.15.77:6808/20248","state":["exists","up"]},{"osd":3,"uuid":"1ba6ec8a-7cbf-4c12-8e21-c3097a263ae8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":20250},{"type":"v1","addr":"172.21.15.77:6814","nonce":20250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":20250},{"type":"v1","addr":"172.21.15.77:6818","nonce":20250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":20250},{"type":"v1","addr":"172.21.15.77:6824","nonce":20250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":20250},{"type":"v1","addr":"172.21.15.77:6821","nonce":20250}]},"public_addr":"172.21.15.77:6814/20250","cluster_addr":"172.21.15.77:6818/20250","heartbeat_back_addr":"172.21.15.77:6824/20250","heartbeat_front_addr":"172.21.15.77:6821/20250","state":["exists","up"]},{"osd":4,"uuid":"3faf4c09-846c-4133-856b-7f53c535a67d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":25551},{"type":"v1","addr":"172.21.15.186:6822","nonce":25551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":25551},{"type":"v1","addr":"172.21.15.186:6826","nonce":25551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":25551},{"type":"v1","addr":"172.21.15.186:6831","nonce":25551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":25551},{"type":"v1","addr":"172.21.15.186:6829","nonce":25551}]},"public_addr":"172.21.15.186:6822/25551","cluster_addr":"172.21.15.186:6826/25551","heartbeat_back_addr":"172.21.15.186:6831/25551","heartbeat_front_addr":"172.21.15.186:6829/25551","state":["exists","up"]},{"osd":5,"uuid":"ffe41d00-d0aa-42f4-82c1-5cced13c1bc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":25541},{"type":"v1","addr":"172.21.15.186:6809","nonce":25541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":25541},{"type":"v1","addr":"172.21.15.186:6811","nonce":25541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":25541},{"type":"v1","addr":"172.21.15.186:6815","nonce":25541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":25541},{"type":"v1","addr":"172.21.15.186:6813","nonce":25541}]},"public_addr":"172.21.15.186:6809/25541","cluster_addr":"172.21.15.186:6811/25541","heartbeat_back_addr":"172.21.15.186:6815/25541","heartbeat_front_addr":"172.21.15.186:6813/25541","state":["exists","up"]},{"osd":6,"uuid":"b0344ee8-deca-4e9b-934c-5866d90d0073","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":25542},{"type":"v1","addr":"172.21.15.186:6801","nonce":25542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":25542},{"type":"v1","addr":"172.21.15.186:6803","nonce":25542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":25542},{"type":"v1","addr":"172.21.15.186:6807","nonce":25542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":25542},{"type":"v1","addr":"172.21.15.186:6805","nonce":25542}]},"public_addr":"172.21.15.186:6801/25542","cluster_addr":"172.21.15.186:6803/25542","heartbeat_back_addr":"172.21.15.186:6807/25542","heartbeat_front_addr":"172.21.15.186:6805/25542","state":["exists","up"]},{"osd":7,"uuid":"0eb0e5d3-3dad-4f49-9b50-3b24a3924bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":25572},{"type":"v1","addr":"172.21.15.186:6817","nonce":25572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":25572},{"type":"v1","addr":"172.21.15.186:6819","nonce":25572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6825","nonce":25572},{"type":"v1","addr":"172.21.15.186:6827","nonce":25572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":25572},{"type":"v1","addr":"172.21.15.186:6823","nonce":25572}]},"public_addr":"172.21.15.186:6817/25572","cluster_addr":"172.21.15.186:6819/25572","heartbeat_back_addr":"172.21.15.186:6827/25572","heartbeat_front_addr":"172.21.15.186:6823/25572","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.143602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.490366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.361292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.343544+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.905429+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969298+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.949524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-03T19:44:21.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.743+0000 7f5d30c58700 1 -- 172.21.15.77:0/262695796 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5d0c063ce0 msgr2=0x7f5d0c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:21.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.743+0000 7f5d30c58700 1 --2- 172.21.15.77:0/262695796 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5d0c063ce0 0x7f5d0c066190 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009fd0 tx=0x7f5d1c017000 comp rx=0 tx=0).stop 2022-04-03T19:44:21.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.743+0000 7f5d30c58700 1 -- 172.21.15.77:0/262695796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d2c0c3b10 msgr2=0x7f5d2c14b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:21.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.743+0000 7f5d30c58700 1 --2- 172.21.15.77:0/262695796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d2c0c3b10 0x7f5d2c14b000 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5d2000c440 tx=0x7f5d20007d90 comp rx=0 tx=0).stop 2022-04-03T19:44:21.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.743+0000 7f5d30c58700 1 -- 172.21.15.77:0/262695796 shutdown_connections 2022-04-03T19:44:21.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.743+0000 7f5d30c58700 1 --2- 172.21.15.77:0/262695796 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5d0c063ce0 0x7f5d0c066190 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:21.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.743+0000 7f5d30c58700 1 --2- 172.21.15.77:0/262695796 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d2c155370 0x7f5d2c145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:21.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.743+0000 7f5d30c58700 1 --2- 172.21.15.77:0/262695796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d2c0c1170 0x7f5d2c14aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:21.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.743+0000 7f5d30c58700 1 --2- 172.21.15.77:0/262695796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5d2c0c3b10 0x7f5d2c14b000 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:21.747 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.743+0000 7f5d30c58700 1 -- 172.21.15.77:0/262695796 >> 172.21.15.77:0/262695796 conn(0x7f5d2c066da0 msgr2=0x7f5d2c155780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:21.747 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.743+0000 7f5d30c58700 1 -- 172.21.15.77:0/262695796 shutdown_connections 2022-04-03T19:44:21.747 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.743+0000 7f5d30c58700 1 -- 172.21.15.77:0/262695796 wait complete. 2022-04-03T19:44:21.768 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-04-03T19:44:21.969 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.969+0000 7f4391968700 1 Processor -- start 2022-04-03T19:44:21.969 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.969+0000 7f4391968700 1 -- start start 2022-04-03T19:44:21.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.970+0000 7f4391968700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f438c06bbe0 0x7f438c1584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:21.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.970+0000 7f4391968700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f438c06c500 0x7f438c14eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:21.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.970+0000 7f4391968700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f438c06d260 con 0x7f438c06bfb0 2022-04-03T19:44:21.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.970+0000 7f4391968700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f438c159450 con 0x7f438c06bbe0 2022-04-03T19:44:21.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.970+0000 7f4391968700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438c0c6330 con 0x7f438c06c500 2022-04-03T19:44:21.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.970+0000 7f4390966700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f438c06bbe0 0x7f438c1584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:21.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.970+0000 7f438bfff700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f438c06bfb0 0x7f438c06d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44092/0 (socket says 172.21.15.77:44092) 2022-04-03T19:44:21.973 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.970+0000 7f438bfff700 1 -- 172.21.15.77:0/3660335102 learned_addr learned my addr 172.21.15.77:0/3660335102 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:21.973 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.970+0000 7f4391167700 1 --2- 172.21.15.77:0/3660335102 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f438c06c500 0x7f438c14eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:21.973 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.971+0000 7f4390966700 1 -- 172.21.15.77:0/3660335102 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f438c06c500 msgr2=0x7f438c14eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:21.973 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.971+0000 7f4390966700 1 --2- 172.21.15.77:0/3660335102 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f438c06c500 0x7f438c14eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:21.973 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.971+0000 7f4390966700 1 -- 172.21.15.77:0/3660335102 >> v1:172.21.15.186:6789/0 conn(0x7f438c06bfb0 legacy=0x7f438c06d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:21.974 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.971+0000 7f4390966700 1 -- 172.21.15.77:0/3660335102 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438c158bb0 con 0x7f438c06bbe0 2022-04-03T19:44:21.974 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.971+0000 7f4390966700 1 --2- 172.21.15.77:0/3660335102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f438c06bbe0 0x7f438c1584b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4380009760 tx=0x7f4380009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf1469574dd19eb2 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:21.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.971+0000 7f438b7fe700 1 -- 172.21.15.77:0/3660335102 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4380011630 con 0x7f438c06bbe0 2022-04-03T19:44:21.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.971+0000 7f438b7fe700 1 -- 172.21.15.77:0/3660335102 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4380011790 con 0x7f438c06bbe0 2022-04-03T19:44:21.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.971+0000 7f438b7fe700 1 -- 172.21.15.77:0/3660335102 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4380011b50 con 0x7f438c06bbe0 2022-04-03T19:44:21.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.972+0000 7f4391968700 1 -- 172.21.15.77:0/3660335102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f438c06bbe0 msgr2=0x7f438c1584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:21.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.972+0000 7f4391968700 1 --2- 172.21.15.77:0/3660335102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f438c06bbe0 0x7f438c1584b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4380009760 tx=0x7f4380009060 comp rx=0 tx=0).stop 2022-04-03T19:44:21.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.972+0000 7f4391968700 1 -- 172.21.15.77:0/3660335102 shutdown_connections 2022-04-03T19:44:21.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.972+0000 7f4391968700 1 --2- 172.21.15.77:0/3660335102 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f438c06c500 0x7f438c14eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:21.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.972+0000 7f4391968700 1 --2- 172.21.15.77:0/3660335102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f438c06bbe0 0x7f438c1584b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:21.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.972+0000 7f4391968700 1 -- 172.21.15.77:0/3660335102 >> 172.21.15.77:0/3660335102 conn(0x7f438c0cb6b0 msgr2=0x7f438c0cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:21.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.972+0000 7f4391968700 1 -- 172.21.15.77:0/3660335102 shutdown_connections 2022-04-03T19:44:21.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.972+0000 7f4391968700 1 -- 172.21.15.77:0/3660335102 wait complete. 2022-04-03T19:44:21.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.973+0000 7f4391968700 1 Processor -- start 2022-04-03T19:44:21.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.973+0000 7f4391968700 1 -- start start 2022-04-03T19:44:21.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.974+0000 7f4391968700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f438c06bbe0 0x7f438c166630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:21.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.974+0000 7f4391968700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f438c06bfb0 0x7f438c152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:21.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.974+0000 7f4391968700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f438c06c500 0x7f438c151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:21.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.974+0000 7f4391968700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c158150 con 0x7f438c06bbe0 2022-04-03T19:44:21.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.974+0000 7f4391968700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c151780 con 0x7f438c06c500 2022-04-03T19:44:21.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.974+0000 7f4391968700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438c150890 con 0x7f438c06bfb0 2022-04-03T19:44:21.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.975+0000 7f4390966700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f438c06bbe0 0x7f438c166630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:21.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.975+0000 7f4390966700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f438c06bbe0 0x7f438c166630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42492/0 (socket says 172.21.15.77:42492) 2022-04-03T19:44:21.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.975+0000 7f4390966700 1 -- 172.21.15.77:0/3803311321 learned_addr learned my addr 172.21.15.77:0/3803311321 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:21.983 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.975+0000 7f438bfff700 1 --2- 172.21.15.77:0/3803311321 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f438c06bfb0 0x7f438c152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:21.983 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.975+0000 7f4391167700 1 --2- 172.21.15.77:0/3803311321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f438c06c500 0x7f438c151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:21.983 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.975+0000 7f4390966700 1 -- 172.21.15.77:0/3803311321 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f438c06bfb0 msgr2=0x7f438c152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:21.983 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.975+0000 7f4390966700 1 --2- 172.21.15.77:0/3803311321 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f438c06bfb0 0x7f438c152950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:21.984 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.975+0000 7f4390966700 1 -- 172.21.15.77:0/3803311321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f438c06c500 msgr2=0x7f438c151c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:21.984 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.975+0000 7f4390966700 1 --2- 172.21.15.77:0/3803311321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f438c06c500 0x7f438c151c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:21.984 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.975+0000 7f4390966700 1 -- 172.21.15.77:0/3803311321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43800092a0 con 0x7f438c06bbe0 2022-04-03T19:44:21.984 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.975+0000 7f4391167700 1 --2- 172.21.15.77:0/3803311321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f438c06c500 0x7f438c151c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:21.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.975+0000 7f4390966700 1 --2- 172.21.15.77:0/3803311321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f438c06bbe0 0x7f438c166630 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f43800091e0 tx=0x7f4380017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:21.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.976+0000 7f4388ff9700 1 -- 172.21.15.77:0/3803311321 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4380000bc0 con 0x7f438c06bbe0 2022-04-03T19:44:21.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.976+0000 7f4388ff9700 1 -- 172.21.15.77:0/3803311321 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4380011900 con 0x7f438c06bbe0 2022-04-03T19:44:21.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.976+0000 7f4388ff9700 1 -- 172.21.15.77:0/3803311321 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4380000d20 con 0x7f438c06bbe0 2022-04-03T19:44:21.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.976+0000 7f4391968700 1 -- 172.21.15.77:0/3803311321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f438c150b70 con 0x7f438c06bbe0 2022-04-03T19:44:21.986 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.976+0000 7f4391968700 1 -- 172.21.15.77:0/3803311321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438c151130 con 0x7f438c06bbe0 2022-04-03T19:44:21.986 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.977+0000 7f4391968700 1 -- 172.21.15.77:0/3803311321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f438c04b050 con 0x7f438c06bbe0 2022-04-03T19:44:21.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.978+0000 7f4388ff9700 1 -- 172.21.15.77:0/3803311321 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f438002c030 con 0x7f438c06bbe0 2022-04-03T19:44:21.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.978+0000 7f4388ff9700 1 --2- 172.21.15.77:0/3803311321 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4374063ce0 0x7f4374066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:21.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.978+0000 7f4388ff9700 1 -- 172.21.15.77:0/3803311321 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f4380081bf0 con 0x7f438c06bbe0 2022-04-03T19:44:21.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.979+0000 7f438bfff700 1 --2- 172.21.15.77:0/3803311321 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4374063ce0 0x7f4374066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:21.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.979+0000 7f438bfff700 1 --2- 172.21.15.77:0/3803311321 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4374063ce0 0x7f4374066190 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f438c0b1810 tx=0x7f4378009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:21.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:21.983+0000 7f4388ff9700 1 -- 172.21.15.77:0/3803311321 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f43800506e0 con 0x7f438c06bbe0 2022-04-03T19:44:22.180 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.180+0000 7f4391968700 1 -- 172.21.15.77:0/3803311321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f438c0c7ec0 con 0x7f438c06bbe0 2022-04-03T19:44:22.181 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.181+0000 7f4388ff9700 1 -- 172.21.15.77:0/3803311321 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16858 (secure 0 0 0) 0x7f438002a020 con 0x7f438c06bbe0 2022-04-03T19:44:22.181 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:22.182 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"fsid":"89c3a3f4-599d-4c4a-8f5a-7180eafb7292","created":"2022-04-03T19:43:54.101559+0000","modified":"2022-04-03T19:44:21.275377+0000","last_up_change":"2022-04-03T19:44:05.589174+0000","last_in_change":"2022-04-03T19:43:56.729760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-03T19:44:07.567638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-04-03T19:44:13.149481+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-04-03T19:44:18.823926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-04-03T19:44:20.275479+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f35f5195-3729-4d78-85d1-0539f259283f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6823","nonce":20252},{"type":"v1","addr":"172.21.15.77:6825","nonce":20252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":20252},{"type":"v1","addr":"172.21.15.77:6827","nonce":20252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":20252},{"type":"v1","addr":"172.21.15.77:6831","nonce":20252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":20252},{"type":"v1","addr":"172.21.15.77:6829","nonce":20252}]},"public_addr":"172.21.15.77:6825/20252","cluster_addr":"172.21.15.77:6827/20252","heartbeat_back_addr":"172.21.15.77:6831/20252","heartbeat_front_addr":"172.21.15.77:6829/20252","state":["exists","up"]},{"osd":1,"uuid":"ab1c89dc-57ce-4cc4-b8c7-9b0c19939601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":20249},{"type":"v1","addr":"172.21.15.77:6805","nonce":20249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":20249},{"type":"v1","addr":"172.21.15.77:6809","nonce":20249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":20249},{"type":"v1","addr":"172.21.15.77:6819","nonce":20249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":20249},{"type":"v1","addr":"172.21.15.77:6815","nonce":20249}]},"public_addr":"172.21.15.77:6805/20249","cluster_addr":"172.21.15.77:6809/20249","heartbeat_back_addr":"172.21.15.77:6819/20249","heartbeat_front_addr":"172.21.15.77:6815/20249","state":["exists","up"]},{"osd":2,"uuid":"351e218b-7749-4ee9-b44a-0a5b52fb9be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":20248},{"type":"v1","addr":"172.21.15.77:6801","nonce":20248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":20248},{"type":"v1","addr":"172.21.15.77:6803","nonce":20248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":20248},{"type":"v1","addr":"172.21.15.77:6813","nonce":20248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":20248},{"type":"v1","addr":"172.21.15.77:6808","nonce":20248}]},"public_addr":"172.21.15.77:6801/20248","cluster_addr":"172.21.15.77:6803/20248","heartbeat_back_addr":"172.21.15.77:6813/20248","heartbeat_front_addr":"172.21.15.77:6808/20248","state":["exists","up"]},{"osd":3,"uuid":"1ba6ec8a-7cbf-4c12-8e21-c3097a263ae8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":20250},{"type":"v1","addr":"172.21.15.77:6814","nonce":20250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":20250},{"type":"v1","addr":"172.21.15.77:6818","nonce":20250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":20250},{"type":"v1","addr":"172.21.15.77:6824","nonce":20250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":20250},{"type":"v1","addr":"172.21.15.77:6821","nonce":20250}]},"public_addr":"172.21.15.77:6814/20250","cluster_addr":"172.21.15.77:6818/20250","heartbeat_back_addr":"172.21.15.77:6824/20250","heartbeat_front_addr":"172.21.15.77:6821/20250","state":["exists","up"]},{"osd":4,"uuid":"3faf4c09-846c-4133-856b-7f53c535a67d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":25551},{"type":"v1","addr":"172.21.15.186:6822","nonce":25551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":25551},{"type":"v1","addr":"172.21.15.186:6826","nonce":25551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":25551},{"type":"v1","addr":"172.21.15.186:6831","nonce":25551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":25551},{"type":"v1","addr":"172.21.15.186:6829","nonce":25551}]},"public_addr":"172.21.15.186:6822/25551","cluster_addr":"172.21.15.186:6826/25551","heartbeat_back_addr":"172.21.15.186:6831/25551","heartbeat_front_addr":"172.21.15.186:6829/25551","state":["exists","up"]},{"osd":5,"uuid":"ffe41d00-d0aa-42f4-82c1-5cced13c1bc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":25541},{"type":"v1","addr":"172.21.15.186:6809","nonce":25541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":25541},{"type":"v1","addr":"172.21.15.186:6811","nonce":25541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":25541},{"type":"v1","addr":"172.21.15.186:6815","nonce":25541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":25541},{"type":"v1","addr":"172.21.15.186:6813","nonce":25541}]},"public_addr":"172.21.15.186:6809/25541","cluster_addr":"172.21.15.186:6811/25541","heartbeat_back_addr":"172.21.15.186:6815/25541","heartbeat_front_addr":"172.21.15.186:6813/25541","state":["exists","up"]},{"osd":6,"uuid":"b0344ee8-deca-4e9b-934c-5866d90d0073","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":25542},{"type":"v1","addr":"172.21.15.186:6801","nonce":25542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":25542},{"type":"v1","addr":"172.21.15.186:6803","nonce":25542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":25542},{"type":"v1","addr":"172.21.15.186:6807","nonce":25542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":25542},{"type":"v1","addr":"172.21.15.186:6805","nonce":25542}]},"public_addr":"172.21.15.186:6801/25542","cluster_addr":"172.21.15.186:6803/25542","heartbeat_back_addr":"172.21.15.186:6807/25542","heartbeat_front_addr":"172.21.15.186:6805/25542","state":["exists","up"]},{"osd":7,"uuid":"0eb0e5d3-3dad-4f49-9b50-3b24a3924bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":25572},{"type":"v1","addr":"172.21.15.186:6817","nonce":25572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":25572},{"type":"v1","addr":"172.21.15.186:6819","nonce":25572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6825","nonce":25572},{"type":"v1","addr":"172.21.15.186:6827","nonce":25572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":25572},{"type":"v1","addr":"172.21.15.186:6823","nonce":25572}]},"public_addr":"172.21.15.186:6817/25572","cluster_addr":"172.21.15.186:6819/25572","heartbeat_back_addr":"172.21.15.186:6827/25572","heartbeat_front_addr":"172.21.15.186:6823/25572","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.143602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.490366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.361292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.343544+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.905429+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969298+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.949524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-03T19:44:22.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.183+0000 7f4391968700 1 -- 172.21.15.77:0/3803311321 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4374063ce0 msgr2=0x7f4374066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:22.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.183+0000 7f4391968700 1 --2- 172.21.15.77:0/3803311321 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4374063ce0 0x7f4374066190 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f438c0b1810 tx=0x7f4378009040 comp rx=0 tx=0).stop 2022-04-03T19:44:22.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.183+0000 7f4391968700 1 -- 172.21.15.77:0/3803311321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f438c06bbe0 msgr2=0x7f438c166630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:22.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.183+0000 7f4391968700 1 --2- 172.21.15.77:0/3803311321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f438c06bbe0 0x7f438c166630 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f43800091e0 tx=0x7f4380017000 comp rx=0 tx=0).stop 2022-04-03T19:44:22.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.183+0000 7f4391968700 1 -- 172.21.15.77:0/3803311321 shutdown_connections 2022-04-03T19:44:22.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.183+0000 7f4391968700 1 --2- 172.21.15.77:0/3803311321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f438c06c500 0x7f438c151c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:22.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.183+0000 7f4391968700 1 --2- 172.21.15.77:0/3803311321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f438c06bbe0 0x7f438c166630 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:22.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.183+0000 7f4391968700 1 --2- 172.21.15.77:0/3803311321 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4374063ce0 0x7f4374066190 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:22.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.183+0000 7f4391968700 1 --2- 172.21.15.77:0/3803311321 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f438c06bfb0 0x7f438c152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:22.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.183+0000 7f4391968700 1 -- 172.21.15.77:0/3803311321 >> 172.21.15.77:0/3803311321 conn(0x7f438c0cb6b0 msgr2=0x7f438c0cba80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:22.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.183+0000 7f4391968700 1 -- 172.21.15.77:0/3803311321 shutdown_connections 2022-04-03T19:44:22.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.183+0000 7f4391968700 1 -- 172.21.15.77:0/3803311321 wait complete. 2022-04-03T19:44:22.208 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-04-03T19:44:22.411 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.411+0000 7fa68f59e700 1 Processor -- start 2022-04-03T19:44:22.412 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.411+0000 7fa68f59e700 1 -- start start 2022-04-03T19:44:22.412 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.412+0000 7fa68f59e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa6900c5680 0x7fa6900c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:22.413 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.412+0000 7fa68f59e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6900c7e80 con 0x7fa690158e10 2022-04-03T19:44:22.413 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.412+0000 7fa68f59e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6900c6ea0 con 0x7fa6900c52b0 2022-04-03T19:44:22.413 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.412+0000 7fa68f59e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6900c4260 con 0x7fa6900c5680 2022-04-03T19:44:22.415 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.412+0000 7fa68ed9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa690158e10 0x7fa69014ed50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38290/0 (socket says 172.21.15.77:38290) 2022-04-03T19:44:22.415 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.412+0000 7fa68ed9d700 1 -- 172.21.15.77:0/107304840 learned_addr learned my addr 172.21.15.77:0/107304840 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:22.415 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.412+0000 7fa687fff700 1 --2- 172.21.15.77:0/107304840 >> v2:172.21.15.77:3300/0 conn(0x7fa6900c5680 0x7fa6900c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:22.416 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.413+0000 7fa68dd9b700 1 -- 172.21.15.77:0/107304840 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594278878 0 0) 0x7fa6900c7e80 con 0x7fa690158e10 2022-04-03T19:44:22.416 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.413+0000 7fa68dd9b700 1 -- 172.21.15.77:0/107304840 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6700033f0 con 0x7fa690158e10 2022-04-03T19:44:22.416 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.413+0000 7fa687fff700 1 -- 172.21.15.77:0/107304840 >> v1:172.21.15.186:6789/0 conn(0x7fa6900c52b0 legacy=0x7fa6900c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:22.416 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.413+0000 7fa68dd9b700 1 -- 172.21.15.77:0/107304840 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106067416 0 0) 0x7fa6900c6ea0 con 0x7fa6900c52b0 2022-04-03T19:44:22.416 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.413+0000 7fa687fff700 1 -- 172.21.15.77:0/107304840 >> v1:172.21.15.77:6789/0 conn(0x7fa690158e10 legacy=0x7fa69014ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:22.417 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.413+0000 7fa687fff700 1 -- 172.21.15.77:0/107304840 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6900c6600 con 0x7fa6900c5680 2022-04-03T19:44:22.417 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.413+0000 7fa687fff700 1 --2- 172.21.15.77:0/107304840 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6900c5680 0x7fa6900c5f70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa678009760 tx=0x7fa678009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed4a7289b025c12 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:22.417 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.414+0000 7fa68dd9b700 1 -- 172.21.15.77:0/107304840 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa678011630 con 0x7fa6900c5680 2022-04-03T19:44:22.417 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.414+0000 7fa68dd9b700 1 -- 172.21.15.77:0/107304840 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa678011790 con 0x7fa6900c5680 2022-04-03T19:44:22.418 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.414+0000 7fa68dd9b700 1 -- 172.21.15.77:0/107304840 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa678011b30 con 0x7fa6900c5680 2022-04-03T19:44:22.418 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.414+0000 7fa68f59e700 1 -- 172.21.15.77:0/107304840 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6900c5680 msgr2=0x7fa6900c5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:22.418 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.414+0000 7fa68f59e700 1 --2- 172.21.15.77:0/107304840 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6900c5680 0x7fa6900c5f70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa678009760 tx=0x7fa678009060 comp rx=0 tx=0).stop 2022-04-03T19:44:22.418 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.414+0000 7fa68f59e700 1 -- 172.21.15.77:0/107304840 shutdown_connections 2022-04-03T19:44:22.418 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.414+0000 7fa68f59e700 1 --2- 172.21.15.77:0/107304840 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6900c5680 0x7fa6900c5f70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:22.419 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.414+0000 7fa68f59e700 1 -- 172.21.15.77:0/107304840 >> 172.21.15.77:0/107304840 conn(0x7fa6900626d0 msgr2=0x7fa690062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:22.419 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.414+0000 7fa68f59e700 1 -- 172.21.15.77:0/107304840 shutdown_connections 2022-04-03T19:44:22.420 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.414+0000 7fa68f59e700 1 -- 172.21.15.77:0/107304840 wait complete. 2022-04-03T19:44:22.425 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.416+0000 7fa68f59e700 1 Processor -- start 2022-04-03T19:44:22.425 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.416+0000 7fa68f59e700 1 -- start start 2022-04-03T19:44:22.426 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.417+0000 7fa68f59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6900c52b0 0x7fa690151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:22.426 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.417+0000 7fa68f59e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6900c5680 0x7fa69014ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:22.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.417+0000 7fa68f59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa690158e10 0x7fa69014bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:22.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.417+0000 7fa68f59e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6900b1360 con 0x7fa6900c52b0 2022-04-03T19:44:22.428 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.417+0000 7fa68f59e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69014fc10 con 0x7fa6900c5680 2022-04-03T19:44:22.428 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.417+0000 7fa68f59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa690151370 con 0x7fa690158e10 2022-04-03T19:44:22.428 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.417+0000 7fa68e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6900c52b0 0x7fa690151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:22.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.417+0000 7fa68e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6900c52b0 0x7fa690151800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42496/0 (socket says 172.21.15.77:42496) 2022-04-03T19:44:22.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.417+0000 7fa68e59c700 1 -- 172.21.15.77:0/3533111539 learned_addr learned my addr 172.21.15.77:0/3533111539 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:22.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.417+0000 7fa687fff700 1 --2- 172.21.15.77:0/3533111539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6900c5680 0x7fa69014ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:22.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.417+0000 7fa68ed9d700 1 --2- 172.21.15.77:0/3533111539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa690158e10 0x7fa69014bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:22.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.418+0000 7fa68e59c700 1 -- 172.21.15.77:0/3533111539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa690158e10 msgr2=0x7fa69014bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:22.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.418+0000 7fa68e59c700 1 --2- 172.21.15.77:0/3533111539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa690158e10 0x7fa69014bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:22.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.418+0000 7fa68e59c700 1 -- 172.21.15.77:0/3533111539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6900c5680 msgr2=0x7fa69014ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:22.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.418+0000 7fa68e59c700 1 --2- 172.21.15.77:0/3533111539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6900c5680 0x7fa69014ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:22.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.418+0000 7fa68e59c700 1 -- 172.21.15.77:0/3533111539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6780092a0 con 0x7fa6900c52b0 2022-04-03T19:44:22.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.418+0000 7fa68e59c700 1 --2- 172.21.15.77:0/3533111539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6900c52b0 0x7fa690151800 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa68000c4f0 tx=0x7fa680007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:22.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.418+0000 7fa6867fc700 1 -- 172.21.15.77:0/3533111539 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68000ce60 con 0x7fa6900c52b0 2022-04-03T19:44:22.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.418+0000 7fa6867fc700 1 -- 172.21.15.77:0/3533111539 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6800052e0 con 0x7fa6900c52b0 2022-04-03T19:44:22.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.418+0000 7fa68f59e700 1 -- 172.21.15.77:0/3533111539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6900cb600 con 0x7fa6900c52b0 2022-04-03T19:44:22.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.418+0000 7fa6867fc700 1 -- 172.21.15.77:0/3533111539 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6800036f0 con 0x7fa6900c52b0 2022-04-03T19:44:22.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.418+0000 7fa68f59e700 1 -- 172.21.15.77:0/3533111539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6900b1590 con 0x7fa6900c52b0 2022-04-03T19:44:22.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.419+0000 7fa68f59e700 1 -- 172.21.15.77:0/3533111539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa690151650 con 0x7fa6900c52b0 2022-04-03T19:44:22.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.420+0000 7fa6867fc700 1 -- 172.21.15.77:0/3533111539 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fa680019030 con 0x7fa6900c52b0 2022-04-03T19:44:22.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.421+0000 7fa6867fc700 1 --2- 172.21.15.77:0/3533111539 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa674063d10 0x7fa6740661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:22.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.421+0000 7fa6867fc700 1 -- 172.21.15.77:0/3533111539 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fa68007fab0 con 0x7fa6900c52b0 2022-04-03T19:44:22.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.421+0000 7fa687fff700 1 --2- 172.21.15.77:0/3533111539 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa674063d10 0x7fa6740661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:22.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.422+0000 7fa687fff700 1 --2- 172.21.15.77:0/3533111539 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa674063d10 0x7fa6740661c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa678009fd0 tx=0x7fa678017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:22.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.425+0000 7fa6867fc700 1 -- 172.21.15.77:0/3533111539 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fa68004e690 con 0x7fa6900c52b0 2022-04-03T19:44:22.659 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:22.659+0000 7fa68f59e700 1 -- 172.21.15.77:0/3533111539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa6900c9b40 con 0x7fa6900c52b0 2022-04-03T19:44:23.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.285+0000 7fa6867fc700 1 -- 172.21.15.77:0/3533111539 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa680017070 con 0x7fa6900c52b0 2022-04-03T19:44:23.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.288+0000 7fa68f59e700 1 -- 172.21.15.77:0/3533111539 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa674063d10 msgr2=0x7fa6740661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:23.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.288+0000 7fa68f59e700 1 --2- 172.21.15.77:0/3533111539 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa674063d10 0x7fa6740661c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa678009fd0 tx=0x7fa678017000 comp rx=0 tx=0).stop 2022-04-03T19:44:23.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.288+0000 7fa68f59e700 1 -- 172.21.15.77:0/3533111539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6900c52b0 msgr2=0x7fa690151800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:23.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.288+0000 7fa68f59e700 1 --2- 172.21.15.77:0/3533111539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6900c52b0 0x7fa690151800 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa68000c4f0 tx=0x7fa680007c70 comp rx=0 tx=0).stop 2022-04-03T19:44:23.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.288+0000 7fa68f59e700 1 -- 172.21.15.77:0/3533111539 shutdown_connections 2022-04-03T19:44:23.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.288+0000 7fa68f59e700 1 --2- 172.21.15.77:0/3533111539 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa674063d10 0x7fa6740661c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:23.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.288+0000 7fa68f59e700 1 --2- 172.21.15.77:0/3533111539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa690158e10 0x7fa69014bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:23.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.288+0000 7fa68f59e700 1 --2- 172.21.15.77:0/3533111539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6900c52b0 0x7fa690151800 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:23.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.288+0000 7fa68f59e700 1 --2- 172.21.15.77:0/3533111539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6900c5680 0x7fa69014ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:23.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.288+0000 7fa68f59e700 1 -- 172.21.15.77:0/3533111539 >> 172.21.15.77:0/3533111539 conn(0x7fa6900626d0 msgr2=0x7fa69014e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:23.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.288+0000 7fa68f59e700 1 -- 172.21.15.77:0/3533111539 shutdown_connections 2022-04-03T19:44:23.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.288+0000 7fa68f59e700 1 -- 172.21.15.77:0/3533111539 wait complete. 2022-04-03T19:44:23.313 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2022-04-03T19:44:23.512 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.512+0000 7fcdfe260700 1 Processor -- start 2022-04-03T19:44:23.512 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.512+0000 7fcdfe260700 1 -- start start 2022-04-03T19:44:23.514 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.513+0000 7fcdfe260700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf80c52b0 0x7fcdf80c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:23.514 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.513+0000 7fcdfe260700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcdf8158e10 0x7fcdf814ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:23.514 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.513+0000 7fcdfe260700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf80c7e80 con 0x7fcdf80c5680 2022-04-03T19:44:23.514 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.513+0000 7fcdfe260700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdf80c4260 con 0x7fcdf8158e10 2022-04-03T19:44:23.515 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.513+0000 7fcdfe260700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf80c8a20 con 0x7fcdf80c52b0 2022-04-03T19:44:23.515 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.513+0000 7fcdfca5d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fcdf80c5680 0x7fcdf80c8310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38292/0 (socket says 172.21.15.77:38292) 2022-04-03T19:44:23.515 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.513+0000 7fcdfca5d700 1 -- 172.21.15.77:0/3818851555 learned_addr learned my addr 172.21.15.77:0/3818851555 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:23.515 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.514+0000 7fcdfd25e700 1 --2- 172.21.15.77:0/3818851555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf80c52b0 0x7fcdf80c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:23.515 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.514+0000 7fcdfda5f700 1 --2- 172.21.15.77:0/3818851555 >> v2:172.21.15.186:3300/0 conn(0x7fcdf8158e10 0x7fcdf814ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:23.517 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.514+0000 7fcde7fff700 1 -- 172.21.15.77:0/3818851555 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792537485 0 0) 0x7fcdf80c7e80 con 0x7fcdf80c5680 2022-04-03T19:44:23.518 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.514+0000 7fcde7fff700 1 -- 172.21.15.77:0/3818851555 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcde00033f0 con 0x7fcdf80c5680 2022-04-03T19:44:23.518 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.514+0000 7fcdfd25e700 1 -- 172.21.15.77:0/3818851555 >> v2:172.21.15.186:3300/0 conn(0x7fcdf8158e10 msgr2=0x7fcdf814ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:23.518 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.514+0000 7fcdfd25e700 1 --2- 172.21.15.77:0/3818851555 >> v2:172.21.15.186:3300/0 conn(0x7fcdf8158e10 0x7fcdf814ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:23.518 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.514+0000 7fcdfd25e700 1 -- 172.21.15.77:0/3818851555 >> v1:172.21.15.77:6789/0 conn(0x7fcdf80c5680 legacy=0x7fcdf80c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:23.519 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.514+0000 7fcdfd25e700 1 -- 172.21.15.77:0/3818851555 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf80c6ea0 con 0x7fcdf80c52b0 2022-04-03T19:44:23.519 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.514+0000 7fcdfda5f700 1 --2- 172.21.15.77:0/3818851555 >> v2:172.21.15.186:3300/0 conn(0x7fcdf8158e10 0x7fcdf814ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:23.519 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.514+0000 7fcdfd25e700 1 --2- 172.21.15.77:0/3818851555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf80c52b0 0x7fcdf80c5f70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4009760 tx=0x7fcdf4009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d0e6696a4aa00a4 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:23.519 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.515+0000 7fcde7fff700 1 -- 172.21.15.77:0/3818851555 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4011630 con 0x7fcdf80c52b0 2022-04-03T19:44:23.519 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.515+0000 7fcde7fff700 1 -- 172.21.15.77:0/3818851555 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf4011790 con 0x7fcdf80c52b0 2022-04-03T19:44:23.520 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.515+0000 7fcde7fff700 1 -- 172.21.15.77:0/3818851555 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4011b30 con 0x7fcdf80c52b0 2022-04-03T19:44:23.520 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.515+0000 7fcdfe260700 1 -- 172.21.15.77:0/3818851555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf80c52b0 msgr2=0x7fcdf80c5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:23.520 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.515+0000 7fcdfe260700 1 --2- 172.21.15.77:0/3818851555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf80c52b0 0x7fcdf80c5f70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4009760 tx=0x7fcdf4009060 comp rx=0 tx=0).stop 2022-04-03T19:44:23.520 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.515+0000 7fcdfe260700 1 -- 172.21.15.77:0/3818851555 shutdown_connections 2022-04-03T19:44:23.520 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.515+0000 7fcdfe260700 1 --2- 172.21.15.77:0/3818851555 >> v2:172.21.15.186:3300/0 conn(0x7fcdf8158e10 0x7fcdf814ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:23.521 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.515+0000 7fcdfe260700 1 --2- 172.21.15.77:0/3818851555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf80c52b0 0x7fcdf80c5f70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:23.521 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.515+0000 7fcdfe260700 1 -- 172.21.15.77:0/3818851555 >> 172.21.15.77:0/3818851555 conn(0x7fcdf80626d0 msgr2=0x7fcdf8062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:23.521 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.515+0000 7fcdfe260700 1 -- 172.21.15.77:0/3818851555 shutdown_connections 2022-04-03T19:44:23.521 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.515+0000 7fcdfe260700 1 -- 172.21.15.77:0/3818851555 wait complete. 2022-04-03T19:44:23.521 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.517+0000 7fcdfe260700 1 Processor -- start 2022-04-03T19:44:23.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.517+0000 7fcdfe260700 1 -- start start 2022-04-03T19:44:23.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.518+0000 7fcdfe260700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdf80c52b0 0x7fcdf81554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:23.525 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.518+0000 7fcdfe260700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf80c5680 0x7fcdf814b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:23.525 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.518+0000 7fcdfe260700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf8158e10 0x7fcdf81478f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:23.525 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.518+0000 7fcdfe260700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf81683e0 con 0x7fcdf80c52b0 2022-04-03T19:44:23.525 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.518+0000 7fcdfe260700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf8169240 con 0x7fcdf8158e10 2022-04-03T19:44:23.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.518+0000 7fcdfe260700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf814c1e0 con 0x7fcdf80c5680 2022-04-03T19:44:23.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.518+0000 7fcdfd25e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdf80c52b0 0x7fcdf81554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:23.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.518+0000 7fcdfd25e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdf80c52b0 0x7fcdf81554e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42498/0 (socket says 172.21.15.77:42498) 2022-04-03T19:44:23.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.518+0000 7fcdfca5d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf80c5680 0x7fcdf814b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:23.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.518+0000 7fcdfd25e700 1 -- 172.21.15.77:0/781603919 learned_addr learned my addr 172.21.15.77:0/781603919 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:23.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.518+0000 7fcdfda5f700 1 --2- 172.21.15.77:0/781603919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf8158e10 0x7fcdf81478f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:23.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.519+0000 7fcdfd25e700 1 -- 172.21.15.77:0/781603919 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf80c5680 msgr2=0x7fcdf814b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:23.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.519+0000 7fcdfd25e700 1 --2- 172.21.15.77:0/781603919 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf80c5680 0x7fcdf814b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:23.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.519+0000 7fcdfd25e700 1 -- 172.21.15.77:0/781603919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf8158e10 msgr2=0x7fcdf81478f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:23.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.519+0000 7fcdfd25e700 1 --2- 172.21.15.77:0/781603919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf8158e10 0x7fcdf81478f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:23.528 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.519+0000 7fcdfd25e700 1 -- 172.21.15.77:0/781603919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf0009a20 con 0x7fcdf80c52b0 2022-04-03T19:44:23.528 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.519+0000 7fcdfd25e700 1 --2- 172.21.15.77:0/781603919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdf80c52b0 0x7fcdf81554e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4009d50 tx=0x7fcdf4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:23.528 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.519+0000 7fcde57fa700 1 -- 172.21.15.77:0/781603919 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4003770 con 0x7fcdf80c52b0 2022-04-03T19:44:23.528 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.519+0000 7fcde57fa700 1 -- 172.21.15.77:0/781603919 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf40038d0 con 0x7fcdf80c52b0 2022-04-03T19:44:23.528 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.519+0000 7fcde57fa700 1 -- 172.21.15.77:0/781603919 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4003c90 con 0x7fcdf80c52b0 2022-04-03T19:44:23.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.520+0000 7fcdfe260700 1 -- 172.21.15.77:0/781603919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf40092a0 con 0x7fcdf80c52b0 2022-04-03T19:44:23.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.520+0000 7fcdfe260700 1 -- 172.21.15.77:0/781603919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf8168910 con 0x7fcdf80c52b0 2022-04-03T19:44:23.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.521+0000 7fcdfe260700 1 -- 172.21.15.77:0/781603919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf8088e10 con 0x7fcdf80c52b0 2022-04-03T19:44:23.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.521+0000 7fcde57fa700 1 -- 172.21.15.77:0/781603919 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fcdf400f050 con 0x7fcdf80c52b0 2022-04-03T19:44:23.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.523+0000 7fcde57fa700 1 --2- 172.21.15.77:0/781603919 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcdd8063d10 0x7fcdd80661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:23.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.523+0000 7fcde57fa700 1 -- 172.21.15.77:0/781603919 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fcdf4081cd0 con 0x7fcdf80c52b0 2022-04-03T19:44:23.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.523+0000 7fcdfca5d700 1 --2- 172.21.15.77:0/781603919 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcdd8063d10 0x7fcdd80661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:23.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.524+0000 7fcdfca5d700 1 --2- 172.21.15.77:0/781603919 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcdd8063d10 0x7fcdd80661c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcdf80b1660 tx=0x7fcde8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:23.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.527+0000 7fcde57fa700 1 -- 172.21.15.77:0/781603919 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fcdf4050970 con 0x7fcdf80c52b0 2022-04-03T19:44:23.767 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:23.767+0000 7fcdfe260700 1 -- 172.21.15.77:0/781603919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fcdf804b050 con 0x7fcdf80c52b0 2022-04-03T19:44:24.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.290+0000 7fcde57fa700 1 -- 172.21.15.77:0/781603919 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcdf4015070 con 0x7fcdf80c52b0 2022-04-03T19:44:24.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.293+0000 7fcdfe260700 1 -- 172.21.15.77:0/781603919 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcdd8063d10 msgr2=0x7fcdd80661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:24.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.293+0000 7fcdfe260700 1 --2- 172.21.15.77:0/781603919 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcdd8063d10 0x7fcdd80661c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcdf80b1660 tx=0x7fcde8009040 comp rx=0 tx=0).stop 2022-04-03T19:44:24.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.293+0000 7fcdfe260700 1 -- 172.21.15.77:0/781603919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdf80c52b0 msgr2=0x7fcdf81554e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:24.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.293+0000 7fcdfe260700 1 --2- 172.21.15.77:0/781603919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdf80c52b0 0x7fcdf81554e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4009d50 tx=0x7fcdf4017000 comp rx=0 tx=0).stop 2022-04-03T19:44:24.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.293+0000 7fcdfe260700 1 -- 172.21.15.77:0/781603919 shutdown_connections 2022-04-03T19:44:24.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.293+0000 7fcdfe260700 1 --2- 172.21.15.77:0/781603919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf8158e10 0x7fcdf81478f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:24.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.293+0000 7fcdfe260700 1 --2- 172.21.15.77:0/781603919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcdf80c52b0 0x7fcdf81554e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:24.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.293+0000 7fcdfe260700 1 --2- 172.21.15.77:0/781603919 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcdd8063d10 0x7fcdd80661c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:24.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.293+0000 7fcdfe260700 1 --2- 172.21.15.77:0/781603919 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf80c5680 0x7fcdf814b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:24.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.293+0000 7fcdfe260700 1 -- 172.21.15.77:0/781603919 >> 172.21.15.77:0/781603919 conn(0x7fcdf80626d0 msgr2=0x7fcdf8062aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:24.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.293+0000 7fcdfe260700 1 -- 172.21.15.77:0/781603919 shutdown_connections 2022-04-03T19:44:24.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.293+0000 7fcdfe260700 1 -- 172.21.15.77:0/781603919 wait complete. 2022-04-03T19:44:24.319 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2022-04-03T19:44:24.518 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.518+0000 7f6409b9d700 1 Processor -- start 2022-04-03T19:44:24.519 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.518+0000 7f6409b9d700 1 -- start start 2022-04-03T19:44:24.520 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.519+0000 7f6409b9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64040c52b0 0x7f64040c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:24.521 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.519+0000 7f6409b9d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6404158e10 0x7f640414ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:24.521 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.519+0000 7f6409b9d700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64040c7e80 con 0x7f64040c5680 2022-04-03T19:44:24.521 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.519+0000 7f6409b9d700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64040c4260 con 0x7f6404158e10 2022-04-03T19:44:24.521 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.519+0000 7f6409b9d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64040c8a20 con 0x7f64040c52b0 2022-04-03T19:44:24.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.519+0000 7f640939c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6404158e10 0x7f640414ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:24.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.519+0000 7f6408b9b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64040c52b0 0x7f64040c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:24.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.520+0000 7f6403fff700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f64040c5680 0x7f64040c8310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44096/0 (socket says 172.21.15.77:44096) 2022-04-03T19:44:24.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.520+0000 7f6403fff700 1 -- 172.21.15.77:0/4041055198 learned_addr learned my addr 172.21.15.77:0/4041055198 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:24.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.520+0000 7f64037fe700 1 -- 172.21.15.77:0/4041055198 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808625658 0 0) 0x7f64040c7e80 con 0x7f64040c5680 2022-04-03T19:44:24.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.520+0000 7f64037fe700 1 -- 172.21.15.77:0/4041055198 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63e80033f0 con 0x7f64040c5680 2022-04-03T19:44:24.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.520+0000 7f6408b9b700 1 -- 172.21.15.77:0/4041055198 >> v2:172.21.15.186:3300/0 conn(0x7f6404158e10 msgr2=0x7f640414ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:24.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.520+0000 7f6408b9b700 1 --2- 172.21.15.77:0/4041055198 >> v2:172.21.15.186:3300/0 conn(0x7f6404158e10 0x7f640414ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:24.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.520+0000 7f6408b9b700 1 -- 172.21.15.77:0/4041055198 >> v1:172.21.15.186:6789/0 conn(0x7f64040c5680 legacy=0x7f64040c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:24.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.520+0000 7f6408b9b700 1 -- 172.21.15.77:0/4041055198 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64040c6ea0 con 0x7f64040c52b0 2022-04-03T19:44:24.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.520+0000 7f6408b9b700 1 --2- 172.21.15.77:0/4041055198 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64040c52b0 0x7f64040c5f70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009760 tx=0x7f63f8009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef884aee7bc17c47 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:24.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.521+0000 7f64037fe700 1 -- 172.21.15.77:0/4041055198 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f8011630 con 0x7f64040c52b0 2022-04-03T19:44:24.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.521+0000 7f64037fe700 1 -- 172.21.15.77:0/4041055198 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f8011790 con 0x7f64040c52b0 2022-04-03T19:44:24.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.521+0000 7f64037fe700 1 -- 172.21.15.77:0/4041055198 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f8011b30 con 0x7f64040c52b0 2022-04-03T19:44:24.528 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.521+0000 7f6409b9d700 1 -- 172.21.15.77:0/4041055198 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64040c52b0 msgr2=0x7f64040c5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:24.528 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.521+0000 7f6409b9d700 1 --2- 172.21.15.77:0/4041055198 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64040c52b0 0x7f64040c5f70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009760 tx=0x7f63f8009060 comp rx=0 tx=0).stop 2022-04-03T19:44:24.528 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.521+0000 7f6409b9d700 1 -- 172.21.15.77:0/4041055198 shutdown_connections 2022-04-03T19:44:24.528 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.521+0000 7f6409b9d700 1 --2- 172.21.15.77:0/4041055198 >> v2:172.21.15.186:3300/0 conn(0x7f6404158e10 0x7f640414ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:24.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.521+0000 7f6409b9d700 1 --2- 172.21.15.77:0/4041055198 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f64040c52b0 0x7f64040c5f70 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:24.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.521+0000 7f6409b9d700 1 -- 172.21.15.77:0/4041055198 >> 172.21.15.77:0/4041055198 conn(0x7f64040626d0 msgr2=0x7f6404062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:24.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.521+0000 7f6409b9d700 1 -- 172.21.15.77:0/4041055198 shutdown_connections 2022-04-03T19:44:24.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.521+0000 7f6409b9d700 1 -- 172.21.15.77:0/4041055198 wait complete. 2022-04-03T19:44:24.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.523+0000 7f6409b9d700 1 Processor -- start 2022-04-03T19:44:24.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.523+0000 7f6409b9d700 1 -- start start 2022-04-03T19:44:24.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.524+0000 7f6409b9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64040c52b0 0x7f6404157850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:24.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.524+0000 7f6409b9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64040c5680 0x7f6404168400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:24.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.524+0000 7f6409b9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6404158e10 0x7f6404168940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:24.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.524+0000 7f6409b9d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6404167f70 con 0x7f64040c5680 2022-04-03T19:44:24.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.524+0000 7f6409b9d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6404164990 con 0x7f64040c52b0 2022-04-03T19:44:24.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.524+0000 7f6409b9d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6404164c30 con 0x7f6404158e10 2022-04-03T19:44:24.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.524+0000 7f6403fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64040c5680 0x7f6404168400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:24.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.524+0000 7f6408b9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64040c52b0 0x7f6404157850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:24.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.524+0000 7f6403fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64040c5680 0x7f6404168400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42500/0 (socket says 172.21.15.77:42500) 2022-04-03T19:44:24.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.524+0000 7f6403fff700 1 -- 172.21.15.77:0/2594389795 learned_addr learned my addr 172.21.15.77:0/2594389795 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:24.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.524+0000 7f640939c700 1 --2- 172.21.15.77:0/2594389795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6404158e10 0x7f6404168940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:24.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.525+0000 7f6408b9b700 1 -- 172.21.15.77:0/2594389795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6404158e10 msgr2=0x7f6404168940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:24.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.525+0000 7f6408b9b700 1 --2- 172.21.15.77:0/2594389795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6404158e10 0x7f6404168940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:24.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.525+0000 7f6408b9b700 1 -- 172.21.15.77:0/2594389795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64040c5680 msgr2=0x7f6404168400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:24.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.525+0000 7f6408b9b700 1 --2- 172.21.15.77:0/2594389795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64040c5680 0x7f6404168400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:24.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.525+0000 7f6408b9b700 1 -- 172.21.15.77:0/2594389795 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f80092a0 con 0x7f64040c52b0 2022-04-03T19:44:24.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.525+0000 7f640939c700 1 --2- 172.21.15.77:0/2594389795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6404158e10 0x7f6404168940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:24.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.525+0000 7f6408b9b700 1 --2- 172.21.15.77:0/2594389795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64040c52b0 0x7f6404157850 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f63f8016000 tx=0x7f63f8017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:24.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.525+0000 7f6400ff9700 1 -- 172.21.15.77:0/2594389795 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f80036f0 con 0x7f64040c52b0 2022-04-03T19:44:24.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.525+0000 7f6400ff9700 1 -- 172.21.15.77:0/2594389795 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f8003850 con 0x7f64040c52b0 2022-04-03T19:44:24.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.525+0000 7f6409b9d700 1 -- 172.21.15.77:0/2594389795 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6404164ef0 con 0x7f64040c52b0 2022-04-03T19:44:24.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.525+0000 7f6400ff9700 1 -- 172.21.15.77:0/2594389795 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f8003c10 con 0x7f64040c52b0 2022-04-03T19:44:24.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.526+0000 7f6409b9d700 1 -- 172.21.15.77:0/2594389795 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64041654b0 con 0x7f64040c52b0 2022-04-03T19:44:24.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.527+0000 7f6409b9d700 1 -- 172.21.15.77:0/2594389795 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6404168250 con 0x7f64040c52b0 2022-04-03T19:44:24.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.527+0000 7f6400ff9700 1 -- 172.21.15.77:0/2594389795 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f63f800f050 con 0x7f64040c52b0 2022-04-03T19:44:24.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.528+0000 7f6400ff9700 1 --2- 172.21.15.77:0/2594389795 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f63ec063ce0 0x7f63ec066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:24.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.528+0000 7f6400ff9700 1 -- 172.21.15.77:0/2594389795 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f63f8081c30 con 0x7f64040c52b0 2022-04-03T19:44:24.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.528+0000 7f6403fff700 1 --2- 172.21.15.77:0/2594389795 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f63ec063ce0 0x7f63ec066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:24.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.529+0000 7f6403fff700 1 --2- 172.21.15.77:0/2594389795 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f63ec063ce0 0x7f63ec066190 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6404168f50 tx=0x7f63f0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:24.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.533+0000 7f6400ff9700 1 -- 172.21.15.77:0/2594389795 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f63f8050850 con 0x7f64040c52b0 2022-04-03T19:44:24.771 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:24.771+0000 7f6409b9d700 1 -- 172.21.15.77:0/2594389795 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f640404b050 con 0x7f64040c52b0 2022-04-03T19:44:25.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.292+0000 7f6400ff9700 1 -- 172.21.15.77:0/2594389795 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v7) v1 ==== 115+0+0 (secure 0 0 0) 0x7f63f8015020 con 0x7f64040c52b0 2022-04-03T19:44:25.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.295+0000 7f6409b9d700 1 -- 172.21.15.77:0/2594389795 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f63ec063ce0 msgr2=0x7f63ec066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:25.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.295+0000 7f6409b9d700 1 --2- 172.21.15.77:0/2594389795 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f63ec063ce0 0x7f63ec066190 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6404168f50 tx=0x7f63f0009040 comp rx=0 tx=0).stop 2022-04-03T19:44:25.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.295+0000 7f6409b9d700 1 -- 172.21.15.77:0/2594389795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64040c52b0 msgr2=0x7f6404157850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:25.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.295+0000 7f6409b9d700 1 --2- 172.21.15.77:0/2594389795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64040c52b0 0x7f6404157850 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f63f8016000 tx=0x7f63f8017000 comp rx=0 tx=0).stop 2022-04-03T19:44:25.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.296+0000 7f6409b9d700 1 -- 172.21.15.77:0/2594389795 shutdown_connections 2022-04-03T19:44:25.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.296+0000 7f6409b9d700 1 --2- 172.21.15.77:0/2594389795 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f63ec063ce0 0x7f63ec066190 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:25.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.296+0000 7f6409b9d700 1 --2- 172.21.15.77:0/2594389795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6404158e10 0x7f6404168940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:25.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.296+0000 7f6409b9d700 1 --2- 172.21.15.77:0/2594389795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64040c52b0 0x7f6404157850 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:25.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.296+0000 7f6409b9d700 1 --2- 172.21.15.77:0/2594389795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64040c5680 0x7f6404168400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:25.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.296+0000 7f6409b9d700 1 -- 172.21.15.77:0/2594389795 >> 172.21.15.77:0/2594389795 conn(0x7f64040626d0 msgr2=0x7f6404062aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:25.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.296+0000 7f6409b9d700 1 -- 172.21.15.77:0/2594389795 shutdown_connections 2022-04-03T19:44:25.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.296+0000 7f6409b9d700 1 -- 172.21.15.77:0/2594389795 wait complete. 2022-04-03T19:44:25.321 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:44:25.518 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.517+0000 7f48dc966700 1 Processor -- start 2022-04-03T19:44:25.518 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.517+0000 7f48dc966700 1 -- start start 2022-04-03T19:44:25.521 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.518+0000 7f48dc966700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d806d3f0 0x7f48d806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:25.521 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.518+0000 7f48dc966700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f48d80c5930 0x7f48d8155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:25.521 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.518+0000 7f48dc966700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48d80c2f80 con 0x7f48d8156120 2022-04-03T19:44:25.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.518+0000 7f48dc966700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d80c3260 con 0x7f48d80c5930 2022-04-03T19:44:25.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.518+0000 7f48dc966700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d80c3540 con 0x7f48d806d3f0 2022-04-03T19:44:25.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.519+0000 7f48d659c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f48d80c5930 0x7f48d8155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:25.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.519+0000 7f48d6d9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d806d3f0 0x7f48d806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:25.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.519+0000 7f48d759e700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f48d8156120 0x7f48d806c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44098/0 (socket says 172.21.15.77:44098) 2022-04-03T19:44:25.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.519+0000 7f48d6d9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d806d3f0 0x7f48d806d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59798/0 (socket says 172.21.15.77:59798) 2022-04-03T19:44:25.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.519+0000 7f48d759e700 1 -- 172.21.15.77:0/375207673 learned_addr learned my addr 172.21.15.77:0/375207673 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:25.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.519+0000 7f48d659c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f48d80c5930 0x7f48d8155be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42502/0 (socket says 172.21.15.77:42502) 2022-04-03T19:44:25.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.519+0000 7f48d659c700 1 -- 172.21.15.77:0/375207673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d806d3f0 msgr2=0x7f48d806d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:25.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.519+0000 7f48d659c700 1 --2- 172.21.15.77:0/375207673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d806d3f0 0x7f48d806d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:25.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.519+0000 7f48d659c700 1 -- 172.21.15.77:0/375207673 >> v1:172.21.15.186:6789/0 conn(0x7f48d8156120 legacy=0x7f48d806c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:25.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.519+0000 7f48d659c700 1 -- 172.21.15.77:0/375207673 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d80c3820 con 0x7f48d80c5930 2022-04-03T19:44:25.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.519+0000 7f48d659c700 1 --2- 172.21.15.77:0/375207673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48d80c5930 0x7f48d8155be0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f48cc009790 tx=0x7f48cc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=623f6df6db9c99c6 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:25.525 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.520+0000 7f48d5d9b700 1 -- 172.21.15.77:0/375207673 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48cc011630 con 0x7f48d80c5930 2022-04-03T19:44:25.525 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.520+0000 7f48d5d9b700 1 -- 172.21.15.77:0/375207673 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48cc011790 con 0x7f48d80c5930 2022-04-03T19:44:25.525 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.520+0000 7f48d5d9b700 1 -- 172.21.15.77:0/375207673 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48cc011b50 con 0x7f48d80c5930 2022-04-03T19:44:25.525 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.520+0000 7f48dc966700 1 -- 172.21.15.77:0/375207673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48d80c5930 msgr2=0x7f48d8155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:25.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.520+0000 7f48dc966700 1 --2- 172.21.15.77:0/375207673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48d80c5930 0x7f48d8155be0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f48cc009790 tx=0x7f48cc009060 comp rx=0 tx=0).stop 2022-04-03T19:44:25.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.520+0000 7f48dc966700 1 -- 172.21.15.77:0/375207673 shutdown_connections 2022-04-03T19:44:25.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.520+0000 7f48dc966700 1 --2- 172.21.15.77:0/375207673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d806d3f0 0x7f48d806d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:25.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.520+0000 7f48dc966700 1 --2- 172.21.15.77:0/375207673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48d80c5930 0x7f48d8155be0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:25.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.520+0000 7f48dc966700 1 -- 172.21.15.77:0/375207673 >> 172.21.15.77:0/375207673 conn(0x7f48d80cb720 msgr2=0x7f48d80cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:25.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.520+0000 7f48dc966700 1 -- 172.21.15.77:0/375207673 shutdown_connections 2022-04-03T19:44:25.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.520+0000 7f48dc966700 1 -- 172.21.15.77:0/375207673 wait complete. 2022-04-03T19:44:25.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.522+0000 7f48dc966700 1 Processor -- start 2022-04-03T19:44:25.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.522+0000 7f48dc966700 1 -- start start 2022-04-03T19:44:25.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.523+0000 7f48dc966700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48d806d3f0 0x7f48d8165fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:25.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.523+0000 7f48dc966700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d80c5930 0x7f48d80c1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:25.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.523+0000 7f48dc966700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48d8156120 0x7f48d81535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:25.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.523+0000 7f48dc966700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d81565b0 con 0x7f48d8156120 2022-04-03T19:44:25.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.523+0000 7f48dc966700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d80c2320 con 0x7f48d806d3f0 2022-04-03T19:44:25.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.523+0000 7f48dc966700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d8168be0 con 0x7f48d80c5930 2022-04-03T19:44:25.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.523+0000 7f48d659c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d80c5930 0x7f48d80c1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:25.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.523+0000 7f48d759e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48d8156120 0x7f48d81535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:25.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.523+0000 7f48d6d9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48d806d3f0 0x7f48d8165fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:25.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.523+0000 7f48d659c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d80c5930 0x7f48d80c1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59800/0 (socket says 172.21.15.77:59800) 2022-04-03T19:44:25.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.523+0000 7f48d659c700 1 -- 172.21.15.77:0/2021463363 learned_addr learned my addr 172.21.15.77:0/2021463363 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:25.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.524+0000 7f48d759e700 1 -- 172.21.15.77:0/2021463363 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d80c5930 msgr2=0x7f48d80c1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:25.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.524+0000 7f48d759e700 1 --2- 172.21.15.77:0/2021463363 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d80c5930 0x7f48d80c1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:25.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.524+0000 7f48d759e700 1 -- 172.21.15.77:0/2021463363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48d806d3f0 msgr2=0x7f48d8165fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:25.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.524+0000 7f48d759e700 1 --2- 172.21.15.77:0/2021463363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48d806d3f0 0x7f48d8165fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:25.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.524+0000 7f48d759e700 1 -- 172.21.15.77:0/2021463363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48cc0092a0 con 0x7f48d8156120 2022-04-03T19:44:25.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.524+0000 7f48d659c700 1 --2- 172.21.15.77:0/2021463363 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d80c5930 0x7f48d80c1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:25.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.524+0000 7f48d759e700 1 --2- 172.21.15.77:0/2021463363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48d8156120 0x7f48d81535f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f48d0005ad0 tx=0x7f48d0001490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:25.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.524+0000 7f48beffd700 1 -- 172.21.15.77:0/2021463363 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d00070d0 con 0x7f48d8156120 2022-04-03T19:44:25.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.524+0000 7f48beffd700 1 -- 172.21.15.77:0/2021463363 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d0007230 con 0x7f48d8156120 2022-04-03T19:44:25.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.524+0000 7f48dc966700 1 -- 172.21.15.77:0/2021463363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d8168ec0 con 0x7f48d8156120 2022-04-03T19:44:25.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.524+0000 7f48beffd700 1 -- 172.21.15.77:0/2021463363 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d00075d0 con 0x7f48d8156120 2022-04-03T19:44:25.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.525+0000 7f48dc966700 1 -- 172.21.15.77:0/2021463363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d8169480 con 0x7f48d8156120 2022-04-03T19:44:25.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.526+0000 7f48dc966700 1 -- 172.21.15.77:0/2021463363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d8155360 con 0x7f48d8156120 2022-04-03T19:44:25.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.526+0000 7f48beffd700 1 -- 172.21.15.77:0/2021463363 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f48d0019030 con 0x7f48d8156120 2022-04-03T19:44:25.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.527+0000 7f48beffd700 1 --2- 172.21.15.77:0/2021463363 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f48b8063d10 0x7f48b80661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:25.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.527+0000 7f48beffd700 1 -- 172.21.15.77:0/2021463363 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f48d007fd20 con 0x7f48d8156120 2022-04-03T19:44:25.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.527+0000 7f48d6d9d700 1 --2- 172.21.15.77:0/2021463363 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f48b8063d10 0x7f48b80661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:25.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.528+0000 7f48d6d9d700 1 --2- 172.21.15.77:0/2021463363 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f48b8063d10 0x7f48b80661c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f48c8009c30 tx=0x7f48c8009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:25.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.531+0000 7f48beffd700 1 -- 172.21.15.77:0/2021463363 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f48d004e900 con 0x7f48d8156120 2022-04-03T19:44:25.764 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.764+0000 7f48dc966700 1 -- 172.21.15.77:0/2021463363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48d804b050 con 0x7f48d8156120 2022-04-03T19:44:25.765 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.765+0000 7f48beffd700 1 -- 172.21.15.77:0/2021463363 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+6109 (secure 0 0 0) 0x7f48d0017070 con 0x7f48d8156120 2022-04-03T19:44:25.766 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:25.766 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4303,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:44:25.289011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4327":{"gid":4327,"name":"e","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.77:6837/3665969042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3665969042},{"type":"v1","addr":"172.21.15.77:6837","nonce":3665969042}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:44:25.767 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.767+0000 7f48dc966700 1 -- 172.21.15.77:0/2021463363 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f48b8063d10 msgr2=0x7f48b80661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:25.768 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.767+0000 7f48dc966700 1 --2- 172.21.15.77:0/2021463363 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f48b8063d10 0x7f48b80661c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f48c8009c30 tx=0x7f48c8009060 comp rx=0 tx=0).stop 2022-04-03T19:44:25.768 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.767+0000 7f48dc966700 1 -- 172.21.15.77:0/2021463363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48d8156120 msgr2=0x7f48d81535f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:25.768 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.767+0000 7f48dc966700 1 --2- 172.21.15.77:0/2021463363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48d8156120 0x7f48d81535f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f48d0005ad0 tx=0x7f48d0001490 comp rx=0 tx=0).stop 2022-04-03T19:44:25.769 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.767+0000 7f48dc966700 1 -- 172.21.15.77:0/2021463363 shutdown_connections 2022-04-03T19:44:25.769 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.767+0000 7f48dc966700 1 --2- 172.21.15.77:0/2021463363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48d8156120 0x7f48d81535f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:25.769 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.767+0000 7f48dc966700 1 --2- 172.21.15.77:0/2021463363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48d806d3f0 0x7f48d8165fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:25.769 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.767+0000 7f48dc966700 1 --2- 172.21.15.77:0/2021463363 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f48b8063d10 0x7f48b80661c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:25.770 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.767+0000 7f48dc966700 1 --2- 172.21.15.77:0/2021463363 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48d80c5930 0x7f48d80c1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:25.770 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.767+0000 7f48dc966700 1 -- 172.21.15.77:0/2021463363 >> 172.21.15.77:0/2021463363 conn(0x7f48d80cb720 msgr2=0x7f48d80c8070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:25.770 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.767+0000 7f48dc966700 1 -- 172.21.15.77:0/2021463363 shutdown_connections 2022-04-03T19:44:25.770 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.767+0000 7f48dc966700 1 -- 172.21.15.77:0/2021463363 wait complete. 2022-04-03T19:44:25.771 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 7 2022-04-03T19:44:25.791 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-04-03T19:44:25.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.989+0000 7ff89da64700 1 Processor -- start 2022-04-03T19:44:25.990 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.989+0000 7ff89da64700 1 -- start start 2022-04-03T19:44:25.992 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.990+0000 7ff89da64700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff89806d3f0 0x7ff89806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:25.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.990+0000 7ff89da64700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff8980c5930 0x7ff898155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:25.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.990+0000 7ff89da64700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8980c2f80 con 0x7ff898156120 2022-04-03T19:44:25.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.990+0000 7ff89da64700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8980c3260 con 0x7ff8980c5930 2022-04-03T19:44:25.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.990+0000 7ff89da64700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8980c3540 con 0x7ff89806d3f0 2022-04-03T19:44:25.994 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.990+0000 7ff89ca62700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff89806d3f0 0x7ff89806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:25.994 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.990+0000 7ff89d263700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ff898156120 0x7ff89806c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44100/0 (socket says 172.21.15.77:44100) 2022-04-03T19:44:25.994 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.990+0000 7ff897fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff8980c5930 0x7ff898155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:25.994 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.990+0000 7ff89d263700 1 -- 172.21.15.77:0/1884663728 learned_addr learned my addr 172.21.15.77:0/1884663728 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:25.995 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.991+0000 7ff89ca62700 1 -- 172.21.15.77:0/1884663728 >> v2:172.21.15.77:3300/0 conn(0x7ff8980c5930 msgr2=0x7ff898155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:25.995 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.991+0000 7ff89ca62700 1 --2- 172.21.15.77:0/1884663728 >> v2:172.21.15.77:3300/0 conn(0x7ff8980c5930 0x7ff898155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:25.995 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.991+0000 7ff89ca62700 1 -- 172.21.15.77:0/1884663728 >> v1:172.21.15.186:6789/0 conn(0x7ff898156120 legacy=0x7ff89806c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:25.995 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.991+0000 7ff89ca62700 1 -- 172.21.15.77:0/1884663728 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8980c3820 con 0x7ff89806d3f0 2022-04-03T19:44:25.996 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.991+0000 7ff89ca62700 1 --2- 172.21.15.77:0/1884663728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff89806d3f0 0x7ff89806d7c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff88c009760 tx=0x7ff88c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d00aee93bb41d706 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:25.996 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.991+0000 7ff8977fe700 1 -- 172.21.15.77:0/1884663728 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c004030 con 0x7ff89806d3f0 2022-04-03T19:44:25.996 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.991+0000 7ff8977fe700 1 -- 172.21.15.77:0/1884663728 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88c004190 con 0x7ff89806d3f0 2022-04-03T19:44:25.996 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.991+0000 7ff8977fe700 1 -- 172.21.15.77:0/1884663728 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c0105b0 con 0x7ff89806d3f0 2022-04-03T19:44:25.996 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.992+0000 7ff89da64700 1 -- 172.21.15.77:0/1884663728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff89806d3f0 msgr2=0x7ff89806d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:25.997 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.992+0000 7ff89da64700 1 --2- 172.21.15.77:0/1884663728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff89806d3f0 0x7ff89806d7c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff88c009760 tx=0x7ff88c009060 comp rx=0 tx=0).stop 2022-04-03T19:44:25.997 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.992+0000 7ff89da64700 1 -- 172.21.15.77:0/1884663728 shutdown_connections 2022-04-03T19:44:25.997 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.992+0000 7ff89da64700 1 --2- 172.21.15.77:0/1884663728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff89806d3f0 0x7ff89806d7c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:25.997 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.992+0000 7ff89da64700 1 --2- 172.21.15.77:0/1884663728 >> v2:172.21.15.77:3300/0 conn(0x7ff8980c5930 0x7ff898155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:25.997 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.992+0000 7ff89da64700 1 -- 172.21.15.77:0/1884663728 >> 172.21.15.77:0/1884663728 conn(0x7ff8980cb720 msgr2=0x7ff8980cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:25.998 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.992+0000 7ff89da64700 1 -- 172.21.15.77:0/1884663728 shutdown_connections 2022-04-03T19:44:25.998 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.992+0000 7ff89da64700 1 -- 172.21.15.77:0/1884663728 wait complete. 2022-04-03T19:44:26.001 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.994+0000 7ff89da64700 1 Processor -- start 2022-04-03T19:44:26.001 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.994+0000 7ff89da64700 1 -- start start 2022-04-03T19:44:26.001 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.995+0000 7ff89da64700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff89806d3f0 0x7ff898153b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:26.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.995+0000 7ff89da64700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8980c5930 0x7ff8980c1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:26.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.995+0000 7ff89da64700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff898156120 0x7ff8981658e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:26.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.995+0000 7ff89da64700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff898165450 con 0x7ff8980c5930 2022-04-03T19:44:26.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.995+0000 7ff89da64700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff898150bf0 con 0x7ff89806d3f0 2022-04-03T19:44:26.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.995+0000 7ff89da64700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff898150ed0 con 0x7ff898156120 2022-04-03T19:44:26.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.995+0000 7ff897fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8980c5930 0x7ff8980c1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:26.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.995+0000 7ff89ca62700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff89806d3f0 0x7ff898153b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:26.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.995+0000 7ff897fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8980c5930 0x7ff8980c1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42508/0 (socket says 172.21.15.77:42508) 2022-04-03T19:44:26.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.995+0000 7ff897fff700 1 -- 172.21.15.77:0/4109256830 learned_addr learned my addr 172.21.15.77:0/4109256830 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:26.004 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.995+0000 7ff89d263700 1 --2- 172.21.15.77:0/4109256830 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff898156120 0x7ff8981658e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:26.004 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.995+0000 7ff897fff700 1 -- 172.21.15.77:0/4109256830 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff898156120 msgr2=0x7ff8981658e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:26.004 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.995+0000 7ff897fff700 1 --2- 172.21.15.77:0/4109256830 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff898156120 0x7ff8981658e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:26.004 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.995+0000 7ff897fff700 1 -- 172.21.15.77:0/4109256830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff89806d3f0 msgr2=0x7ff898153b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:26.004 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.995+0000 7ff897fff700 1 --2- 172.21.15.77:0/4109256830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff89806d3f0 0x7ff898153b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:26.005 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.995+0000 7ff897fff700 1 -- 172.21.15.77:0/4109256830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88c0092a0 con 0x7ff8980c5930 2022-04-03T19:44:26.005 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.995+0000 7ff89d263700 1 --2- 172.21.15.77:0/4109256830 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff898156120 0x7ff8981658e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:44:26.005 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.996+0000 7ff897fff700 1 --2- 172.21.15.77:0/4109256830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8980c5930 0x7ff8980c1d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff88400e930 tx=0x7ff884009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:26.005 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.996+0000 7ff894ff9700 1 -- 172.21.15.77:0/4109256830 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff884013680 con 0x7ff8980c5930 2022-04-03T19:44:26.006 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.996+0000 7ff894ff9700 1 -- 172.21.15.77:0/4109256830 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8840137e0 con 0x7ff8980c5930 2022-04-03T19:44:26.006 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.996+0000 7ff89da64700 1 -- 172.21.15.77:0/4109256830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8981511b0 con 0x7ff8980c5930 2022-04-03T19:44:26.006 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.996+0000 7ff894ff9700 1 -- 172.21.15.77:0/4109256830 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff884013b60 con 0x7ff8980c5930 2022-04-03T19:44:26.006 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.996+0000 7ff89da64700 1 -- 172.21.15.77:0/4109256830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff898151770 con 0x7ff8980c5930 2022-04-03T19:44:26.006 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.997+0000 7ff89da64700 1 -- 172.21.15.77:0/4109256830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff89804b050 con 0x7ff8980c5930 2022-04-03T19:44:26.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.998+0000 7ff894ff9700 1 -- 172.21.15.77:0/4109256830 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7ff884019030 con 0x7ff8980c5930 2022-04-03T19:44:26.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.999+0000 7ff894ff9700 1 --2- 172.21.15.77:0/4109256830 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff880063ce0 0x7ff880066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:26.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.999+0000 7ff894ff9700 1 -- 172.21.15.77:0/4109256830 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7ff8840808c0 con 0x7ff8980c5930 2022-04-03T19:44:26.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:25.999+0000 7ff89ca62700 1 --2- 172.21.15.77:0/4109256830 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff880063ce0 0x7ff880066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:26.008 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.000+0000 7ff89ca62700 1 --2- 172.21.15.77:0/4109256830 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff880063ce0 0x7ff880066190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff88c009760 tx=0x7ff88c012000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:26.008 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.003+0000 7ff894ff9700 1 -- 172.21.15.77:0/4109256830 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7ff88404f440 con 0x7ff8980c5930 2022-04-03T19:44:26.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.196+0000 7ff89da64700 1 -- 172.21.15.77:0/4109256830 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff89806d010 con 0x7ff8980c5930 2022-04-03T19:44:26.197 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.197+0000 7ff894ff9700 1 -- 172.21.15.77:0/4109256830 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16845 (secure 0 0 0) 0x7ff884021090 con 0x7ff8980c5930 2022-04-03T19:44:26.198 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:26.198 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"89c3a3f4-599d-4c4a-8f5a-7180eafb7292","created":"2022-04-03T19:43:54.101559+0000","modified":"2022-04-03T19:44:22.278254+0000","last_up_change":"2022-04-03T19:44:05.589174+0000","last_in_change":"2022-04-03T19:43:56.729760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-03T19:44:07.567638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-04-03T19:44:13.149481+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-04-03T19:44:18.823926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-04-03T19:44:20.275479+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f35f5195-3729-4d78-85d1-0539f259283f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6823","nonce":20252},{"type":"v1","addr":"172.21.15.77:6825","nonce":20252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":20252},{"type":"v1","addr":"172.21.15.77:6827","nonce":20252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":20252},{"type":"v1","addr":"172.21.15.77:6831","nonce":20252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":20252},{"type":"v1","addr":"172.21.15.77:6829","nonce":20252}]},"public_addr":"172.21.15.77:6825/20252","cluster_addr":"172.21.15.77:6827/20252","heartbeat_back_addr":"172.21.15.77:6831/20252","heartbeat_front_addr":"172.21.15.77:6829/20252","state":["exists","up"]},{"osd":1,"uuid":"ab1c89dc-57ce-4cc4-b8c7-9b0c19939601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":20249},{"type":"v1","addr":"172.21.15.77:6805","nonce":20249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":20249},{"type":"v1","addr":"172.21.15.77:6809","nonce":20249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":20249},{"type":"v1","addr":"172.21.15.77:6819","nonce":20249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":20249},{"type":"v1","addr":"172.21.15.77:6815","nonce":20249}]},"public_addr":"172.21.15.77:6805/20249","cluster_addr":"172.21.15.77:6809/20249","heartbeat_back_addr":"172.21.15.77:6819/20249","heartbeat_front_addr":"172.21.15.77:6815/20249","state":["exists","up"]},{"osd":2,"uuid":"351e218b-7749-4ee9-b44a-0a5b52fb9be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":20248},{"type":"v1","addr":"172.21.15.77:6801","nonce":20248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":20248},{"type":"v1","addr":"172.21.15.77:6803","nonce":20248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":20248},{"type":"v1","addr":"172.21.15.77:6813","nonce":20248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":20248},{"type":"v1","addr":"172.21.15.77:6808","nonce":20248}]},"public_addr":"172.21.15.77:6801/20248","cluster_addr":"172.21.15.77:6803/20248","heartbeat_back_addr":"172.21.15.77:6813/20248","heartbeat_front_addr":"172.21.15.77:6808/20248","state":["exists","up"]},{"osd":3,"uuid":"1ba6ec8a-7cbf-4c12-8e21-c3097a263ae8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":20250},{"type":"v1","addr":"172.21.15.77:6814","nonce":20250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":20250},{"type":"v1","addr":"172.21.15.77:6818","nonce":20250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":20250},{"type":"v1","addr":"172.21.15.77:6824","nonce":20250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":20250},{"type":"v1","addr":"172.21.15.77:6821","nonce":20250}]},"public_addr":"172.21.15.77:6814/20250","cluster_addr":"172.21.15.77:6818/20250","heartbeat_back_addr":"172.21.15.77:6824/20250","heartbeat_front_addr":"172.21.15.77:6821/20250","state":["exists","up"]},{"osd":4,"uuid":"3faf4c09-846c-4133-856b-7f53c535a67d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":25551},{"type":"v1","addr":"172.21.15.186:6822","nonce":25551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":25551},{"type":"v1","addr":"172.21.15.186:6826","nonce":25551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":25551},{"type":"v1","addr":"172.21.15.186:6831","nonce":25551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":25551},{"type":"v1","addr":"172.21.15.186:6829","nonce":25551}]},"public_addr":"172.21.15.186:6822/25551","cluster_addr":"172.21.15.186:6826/25551","heartbeat_back_addr":"172.21.15.186:6831/25551","heartbeat_front_addr":"172.21.15.186:6829/25551","state":["exists","up"]},{"osd":5,"uuid":"ffe41d00-d0aa-42f4-82c1-5cced13c1bc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":25541},{"type":"v1","addr":"172.21.15.186:6809","nonce":25541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":25541},{"type":"v1","addr":"172.21.15.186:6811","nonce":25541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":25541},{"type":"v1","addr":"172.21.15.186:6815","nonce":25541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":25541},{"type":"v1","addr":"172.21.15.186:6813","nonce":25541}]},"public_addr":"172.21.15.186:6809/25541","cluster_addr":"172.21.15.186:6811/25541","heartbeat_back_addr":"172.21.15.186:6815/25541","heartbeat_front_addr":"172.21.15.186:6813/25541","state":["exists","up"]},{"osd":6,"uuid":"b0344ee8-deca-4e9b-934c-5866d90d0073","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":25542},{"type":"v1","addr":"172.21.15.186:6801","nonce":25542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":25542},{"type":"v1","addr":"172.21.15.186:6803","nonce":25542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":25542},{"type":"v1","addr":"172.21.15.186:6807","nonce":25542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":25542},{"type":"v1","addr":"172.21.15.186:6805","nonce":25542}]},"public_addr":"172.21.15.186:6801/25542","cluster_addr":"172.21.15.186:6803/25542","heartbeat_back_addr":"172.21.15.186:6807/25542","heartbeat_front_addr":"172.21.15.186:6805/25542","state":["exists","up"]},{"osd":7,"uuid":"0eb0e5d3-3dad-4f49-9b50-3b24a3924bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":25572},{"type":"v1","addr":"172.21.15.186:6817","nonce":25572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":25572},{"type":"v1","addr":"172.21.15.186:6819","nonce":25572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6825","nonce":25572},{"type":"v1","addr":"172.21.15.186:6827","nonce":25572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":25572},{"type":"v1","addr":"172.21.15.186:6823","nonce":25572}]},"public_addr":"172.21.15.186:6817/25572","cluster_addr":"172.21.15.186:6819/25572","heartbeat_back_addr":"172.21.15.186:6827/25572","heartbeat_front_addr":"172.21.15.186:6823/25572","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.143602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.490366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.361292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.343544+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.905429+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969298+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.949524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-03T19:44:26.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.199+0000 7ff89da64700 1 -- 172.21.15.77:0/4109256830 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff880063ce0 msgr2=0x7ff880066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:26.201 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.199+0000 7ff89da64700 1 --2- 172.21.15.77:0/4109256830 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff880063ce0 0x7ff880066190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff88c009760 tx=0x7ff88c012000 comp rx=0 tx=0).stop 2022-04-03T19:44:26.201 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.200+0000 7ff89da64700 1 -- 172.21.15.77:0/4109256830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8980c5930 msgr2=0x7ff8980c1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:26.201 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.200+0000 7ff89da64700 1 --2- 172.21.15.77:0/4109256830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8980c5930 0x7ff8980c1d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff88400e930 tx=0x7ff884009060 comp rx=0 tx=0).stop 2022-04-03T19:44:26.202 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.200+0000 7ff89da64700 1 -- 172.21.15.77:0/4109256830 shutdown_connections 2022-04-03T19:44:26.202 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.200+0000 7ff89da64700 1 --2- 172.21.15.77:0/4109256830 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff880063ce0 0x7ff880066190 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:26.202 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.200+0000 7ff89da64700 1 --2- 172.21.15.77:0/4109256830 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff898156120 0x7ff8981658e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:26.202 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.200+0000 7ff89da64700 1 --2- 172.21.15.77:0/4109256830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff89806d3f0 0x7ff898153b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:26.203 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.200+0000 7ff89da64700 1 --2- 172.21.15.77:0/4109256830 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8980c5930 0x7ff8980c1d80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:26.203 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.200+0000 7ff89da64700 1 -- 172.21.15.77:0/4109256830 >> 172.21.15.77:0/4109256830 conn(0x7ff8980cb720 msgr2=0x7ff898154940 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:26.203 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.200+0000 7ff89da64700 1 -- 172.21.15.77:0/4109256830 shutdown_connections 2022-04-03T19:44:26.203 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.200+0000 7ff89da64700 1 -- 172.21.15.77:0/4109256830 wait complete. 2022-04-03T19:44:26.224 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2022-04-03T19:44:26.225 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-04-03T19:44:26.426 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.426+0000 7f2ed9137700 1 Processor -- start 2022-04-03T19:44:26.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.426+0000 7f2ed9137700 1 -- start start 2022-04-03T19:44:26.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.427+0000 7f2ed9137700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ed40c2a90 0x7f2ed406c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:26.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.427+0000 7f2ed9137700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed41562f0 con 0x7f2ed406d3f0 2022-04-03T19:44:26.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.427+0000 7f2ed9137700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed40c2f80 con 0x7f2ed40c5930 2022-04-03T19:44:26.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.427+0000 7f2ed9137700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed40c3260 con 0x7f2ed40c2a90 2022-04-03T19:44:26.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.427+0000 7f2ed3fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2ed406d3f0 0x7f2ed406d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38294/0 (socket says 172.21.15.77:38294) 2022-04-03T19:44:26.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.427+0000 7f2ed3fff700 1 -- 172.21.15.77:0/4009755814 learned_addr learned my addr 172.21.15.77:0/4009755814 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:26.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.428+0000 7f2ed8936700 1 --2- 172.21.15.77:0/4009755814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ed40c2a90 0x7f2ed406c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:26.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.428+0000 7f2ed2ffd700 1 -- 172.21.15.77:0/4009755814 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168328755 0 0) 0x7f2ed40c2f80 con 0x7f2ed40c5930 2022-04-03T19:44:26.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.428+0000 7f2ed2ffd700 1 -- 172.21.15.77:0/4009755814 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ec00033f0 con 0x7f2ed40c5930 2022-04-03T19:44:26.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.428+0000 7f2ed2ffd700 1 -- 172.21.15.77:0/4009755814 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502616387 0 0) 0x7f2ed41562f0 con 0x7f2ed406d3f0 2022-04-03T19:44:26.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.428+0000 7f2ed2ffd700 1 -- 172.21.15.77:0/4009755814 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ed40c2f80 con 0x7f2ed406d3f0 2022-04-03T19:44:26.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.428+0000 7f2ed8936700 1 -- 172.21.15.77:0/4009755814 >> v1:172.21.15.186:6789/0 conn(0x7f2ed40c5930 legacy=0x7f2ed4155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:26.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.428+0000 7f2ed8936700 1 -- 172.21.15.77:0/4009755814 >> v1:172.21.15.77:6789/0 conn(0x7f2ed406d3f0 legacy=0x7f2ed406d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:26.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.428+0000 7f2ed8936700 1 -- 172.21.15.77:0/4009755814 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed40c3540 con 0x7f2ed40c2a90 2022-04-03T19:44:26.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.428+0000 7f2ed8936700 1 --2- 172.21.15.77:0/4009755814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ed40c2a90 0x7f2ed406c040 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009ec0 tx=0x7f2ecc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a665c6f92e3fa6d9 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:26.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.429+0000 7f2ed2ffd700 1 -- 172.21.15.77:0/4009755814 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc00e070 con 0x7f2ed40c2a90 2022-04-03T19:44:26.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.429+0000 7f2ed2ffd700 1 -- 172.21.15.77:0/4009755814 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ecc00bbd0 con 0x7f2ed40c2a90 2022-04-03T19:44:26.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.429+0000 7f2ed2ffd700 1 -- 172.21.15.77:0/4009755814 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc012430 con 0x7f2ed40c2a90 2022-04-03T19:44:26.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.429+0000 7f2ed9137700 1 -- 172.21.15.77:0/4009755814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ed40c2a90 msgr2=0x7f2ed406c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:26.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.429+0000 7f2ed9137700 1 --2- 172.21.15.77:0/4009755814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ed40c2a90 0x7f2ed406c040 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009ec0 tx=0x7f2ecc009780 comp rx=0 tx=0).stop 2022-04-03T19:44:26.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.429+0000 7f2ed9137700 1 -- 172.21.15.77:0/4009755814 shutdown_connections 2022-04-03T19:44:26.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.429+0000 7f2ed9137700 1 --2- 172.21.15.77:0/4009755814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ed40c2a90 0x7f2ed406c040 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:26.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.429+0000 7f2ed9137700 1 -- 172.21.15.77:0/4009755814 >> 172.21.15.77:0/4009755814 conn(0x7f2ed40cb720 msgr2=0x7f2ed40cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:26.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.429+0000 7f2ed9137700 1 -- 172.21.15.77:0/4009755814 shutdown_connections 2022-04-03T19:44:26.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.429+0000 7f2ed9137700 1 -- 172.21.15.77:0/4009755814 wait complete. 2022-04-03T19:44:26.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.431+0000 7f2ed9137700 1 Processor -- start 2022-04-03T19:44:26.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.431+0000 7f2ed9137700 1 -- start start 2022-04-03T19:44:26.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.432+0000 7f2ed9137700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed406d3f0 0x7f2ed414fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:26.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.432+0000 7f2ed9137700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed40c2a90 0x7f2ed4159fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:26.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.432+0000 7f2ed9137700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ed40c5930 0x7f2ed415a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:26.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.432+0000 7f2ed9137700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed41526f0 con 0x7f2ed40c2a90 2022-04-03T19:44:26.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.432+0000 7f2ed9137700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed4159b30 con 0x7f2ed406d3f0 2022-04-03T19:44:26.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.432+0000 7f2ed9137700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed41652c0 con 0x7f2ed40c5930 2022-04-03T19:44:26.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.432+0000 7f2ed37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed40c2a90 0x7f2ed4159fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:26.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.432+0000 7f2ed37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed40c2a90 0x7f2ed4159fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42510/0 (socket says 172.21.15.77:42510) 2022-04-03T19:44:26.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.432+0000 7f2ed37fe700 1 -- 172.21.15.77:0/3588858721 learned_addr learned my addr 172.21.15.77:0/3588858721 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:26.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.432+0000 7f2ed8936700 1 --2- 172.21.15.77:0/3588858721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ed40c5930 0x7f2ed415a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:26.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.432+0000 7f2ed3fff700 1 --2- 172.21.15.77:0/3588858721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed406d3f0 0x7f2ed414fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:26.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.433+0000 7f2ed37fe700 1 -- 172.21.15.77:0/3588858721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ed40c5930 msgr2=0x7f2ed415a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:26.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.433+0000 7f2ed37fe700 1 --2- 172.21.15.77:0/3588858721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ed40c5930 0x7f2ed415a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:26.441 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.433+0000 7f2ed37fe700 1 -- 172.21.15.77:0/3588858721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed406d3f0 msgr2=0x7f2ed414fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:26.441 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.433+0000 7f2ed37fe700 1 --2- 172.21.15.77:0/3588858721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed406d3f0 0x7f2ed414fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:26.441 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.433+0000 7f2ed37fe700 1 -- 172.21.15.77:0/3588858721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ecc009a20 con 0x7f2ed40c2a90 2022-04-03T19:44:26.441 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.433+0000 7f2ed37fe700 1 --2- 172.21.15.77:0/3588858721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed40c2a90 0x7f2ed4159fc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2ec800c440 tx=0x7f2ec8007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:26.442 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.433+0000 7f2ebbfff700 1 -- 172.21.15.77:0/3588858721 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec800ce80 con 0x7f2ed40c2a90 2022-04-03T19:44:26.442 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.433+0000 7f2ed9137700 1 -- 172.21.15.77:0/3588858721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed4165560 con 0x7f2ed40c2a90 2022-04-03T19:44:26.442 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.433+0000 7f2ebbfff700 1 -- 172.21.15.77:0/3588858721 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec80050d0 con 0x7f2ed40c2a90 2022-04-03T19:44:26.442 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.433+0000 7f2ebbfff700 1 -- 172.21.15.77:0/3588858721 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec8011590 con 0x7f2ed40c2a90 2022-04-03T19:44:26.442 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.433+0000 7f2ed9137700 1 -- 172.21.15.77:0/3588858721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed4165b20 con 0x7f2ed40c2a90 2022-04-03T19:44:26.443 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.434+0000 7f2ed9137700 1 -- 172.21.15.77:0/3588858721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed4159ca0 con 0x7f2ed40c2a90 2022-04-03T19:44:26.443 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.435+0000 7f2ebbfff700 1 -- 172.21.15.77:0/3588858721 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f2ec8019030 con 0x7f2ed40c2a90 2022-04-03T19:44:26.444 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.436+0000 7f2ebbfff700 1 --2- 172.21.15.77:0/3588858721 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2eb4063ce0 0x7f2eb4066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:26.444 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.437+0000 7f2ebbfff700 1 -- 172.21.15.77:0/3588858721 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f2ec807fa80 con 0x7f2ed40c2a90 2022-04-03T19:44:26.444 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.437+0000 7f2ed3fff700 1 --2- 172.21.15.77:0/3588858721 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2eb4063ce0 0x7f2eb4066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:26.444 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.437+0000 7f2ed3fff700 1 --2- 172.21.15.77:0/3588858721 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2eb4063ce0 0x7f2eb4066190 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4005750 tx=0x7f2ec4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:26.445 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.441+0000 7f2ebbfff700 1 -- 172.21.15.77:0/3588858721 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f2ec804e600 con 0x7f2ed40c2a90 2022-04-03T19:44:26.628 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.627+0000 7f2ed9137700 1 -- 172.21.15.77:0/3588858721 --> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2ed404b050 con 0x7f2eb4063ce0 2022-04-03T19:44:26.637 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.637+0000 7f2ebbfff700 1 -- 172.21.15.77:0/3588858721 <== mgr.4106 v2:172.21.15.186:6832/24948 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284640 (secure 0 0 0) 0x7f2ed404b050 con 0x7f2eb4063ce0 2022-04-03T19:44:26.642 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:26.645 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.641+0000 7f2ed9137700 1 -- 172.21.15.77:0/3588858721 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2eb4063ce0 msgr2=0x7f2eb4066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:26.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.641+0000 7f2ed9137700 1 --2- 172.21.15.77:0/3588858721 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2eb4063ce0 0x7f2eb4066190 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4005750 tx=0x7f2ec4009040 comp rx=0 tx=0).stop 2022-04-03T19:44:26.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.641+0000 7f2ed9137700 1 -- 172.21.15.77:0/3588858721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed40c2a90 msgr2=0x7f2ed4159fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:26.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.641+0000 7f2ed9137700 1 --2- 172.21.15.77:0/3588858721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed40c2a90 0x7f2ed4159fc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2ec800c440 tx=0x7f2ec8007c90 comp rx=0 tx=0).stop 2022-04-03T19:44:26.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.641+0000 7f2ed9137700 1 -- 172.21.15.77:0/3588858721 shutdown_connections 2022-04-03T19:44:26.647 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.641+0000 7f2ed9137700 1 --2- 172.21.15.77:0/3588858721 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2eb4063ce0 0x7f2eb4066190 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:26.647 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.641+0000 7f2ed9137700 1 --2- 172.21.15.77:0/3588858721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ed40c5930 0x7f2ed415a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:26.647 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.641+0000 7f2ed9137700 1 --2- 172.21.15.77:0/3588858721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed406d3f0 0x7f2ed414fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:26.647 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.641+0000 7f2ed9137700 1 --2- 172.21.15.77:0/3588858721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ed40c2a90 0x7f2ed4159fc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:26.647 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.641+0000 7f2ed9137700 1 -- 172.21.15.77:0/3588858721 >> 172.21.15.77:0/3588858721 conn(0x7f2ed40cb720 msgr2=0x7f2ed414ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:26.648 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.641+0000 7f2ed9137700 1 -- 172.21.15.77:0/3588858721 shutdown_connections 2022-04-03T19:44:26.648 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.641+0000 7f2ed9137700 1 -- 172.21.15.77:0/3588858721 wait complete. 2022-04-03T19:44:26.648 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2022-04-03T19:44:26.666 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2022-04-03T19:44:25.489206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":64708,"kb_used_data":2820,"kb_used_omap":0,"kb_used_meta":61824,"kb_avail":754910012,"statfs":{"total":773094113280,"available":773027852288,"internally_reserved":0,"allocated":2887680,"data_stored":1693492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63307776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.724527"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300867+0000","last_change":"2022-04-03T19:44:21.300867+0000","last_active":"2022-04-03T19:44:21.300867+0000","last_peered":"2022-04-03T19:44:21.300867+0000","last_clean":"2022-04-03T19:44:21.300867+0000","last_became_active":"2022-04-03T19:44:21.300569+0000","last_became_peered":"2022-04-03T19:44:21.300569+0000","last_unstale":"2022-04-03T19:44:21.300867+0000","last_undegraded":"2022-04-03T19:44:21.300867+0000","last_fullsized":"2022-04-03T19:44:21.300867+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:10:32.827314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302410+0000","last_change":"2022-04-03T19:44:21.302410+0000","last_active":"2022-04-03T19:44:21.302410+0000","last_peered":"2022-04-03T19:44:21.302410+0000","last_clean":"2022-04-03T19:44:21.302410+0000","last_became_active":"2022-04-03T19:44:21.302142+0000","last_became_peered":"2022-04-03T19:44:21.302142+0000","last_unstale":"2022-04-03T19:44:21.302410+0000","last_undegraded":"2022-04-03T19:44:21.302410+0000","last_fullsized":"2022-04-03T19:44:21.302410+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:01:46.327996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293080+0000","last_change":"2022-04-03T19:44:21.293080+0000","last_active":"2022-04-03T19:44:21.293080+0000","last_peered":"2022-04-03T19:44:21.293080+0000","last_clean":"2022-04-03T19:44:21.293080+0000","last_became_active":"2022-04-03T19:44:21.292762+0000","last_became_peered":"2022-04-03T19:44:21.292762+0000","last_unstale":"2022-04-03T19:44:21.293080+0000","last_undegraded":"2022-04-03T19:44:21.293080+0000","last_fullsized":"2022-04-03T19:44:21.293080+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:28:36.726593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.294448+0000","last_change":"2022-04-03T19:44:21.294448+0000","last_active":"2022-04-03T19:44:21.294448+0000","last_peered":"2022-04-03T19:44:21.294448+0000","last_clean":"2022-04-03T19:44:21.294448+0000","last_became_active":"2022-04-03T19:44:21.294083+0000","last_became_peered":"2022-04-03T19:44:21.294083+0000","last_unstale":"2022-04-03T19:44:21.294448+0000","last_undegraded":"2022-04-03T19:44:21.294448+0000","last_fullsized":"2022-04-03T19:44:21.294448+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:57:02.374826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299551+0000","last_change":"2022-04-03T19:44:21.299551+0000","last_active":"2022-04-03T19:44:21.299551+0000","last_peered":"2022-04-03T19:44:21.299551+0000","last_clean":"2022-04-03T19:44:21.299551+0000","last_became_active":"2022-04-03T19:44:21.299123+0000","last_became_peered":"2022-04-03T19:44:21.299123+0000","last_unstale":"2022-04-03T19:44:21.299551+0000","last_undegraded":"2022-04-03T19:44:21.299551+0000","last_fullsized":"2022-04-03T19:44:21.299551+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:08:24.911647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293129+0000","last_change":"2022-04-03T19:44:21.293129+0000","last_active":"2022-04-03T19:44:21.293129+0000","last_peered":"2022-04-03T19:44:21.293129+0000","last_clean":"2022-04-03T19:44:21.293129+0000","last_became_active":"2022-04-03T19:44:21.292843+0000","last_became_peered":"2022-04-03T19:44:21.292843+0000","last_unstale":"2022-04-03T19:44:21.293129+0000","last_undegraded":"2022-04-03T19:44:21.293129+0000","last_fullsized":"2022-04-03T19:44:21.293129+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:38:32.432609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299112+0000","last_change":"2022-04-03T19:44:21.299112+0000","last_active":"2022-04-03T19:44:21.299112+0000","last_peered":"2022-04-03T19:44:21.299112+0000","last_clean":"2022-04-03T19:44:21.299112+0000","last_became_active":"2022-04-03T19:44:21.298803+0000","last_became_peered":"2022-04-03T19:44:21.298803+0000","last_unstale":"2022-04-03T19:44:21.299112+0000","last_undegraded":"2022-04-03T19:44:21.299112+0000","last_fullsized":"2022-04-03T19:44:21.299112+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:29:43.577843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293638+0000","last_change":"2022-04-03T19:44:21.293638+0000","last_active":"2022-04-03T19:44:21.293638+0000","last_peered":"2022-04-03T19:44:21.293638+0000","last_clean":"2022-04-03T19:44:21.293638+0000","last_became_active":"2022-04-03T19:44:21.293359+0000","last_became_peered":"2022-04-03T19:44:21.293359+0000","last_unstale":"2022-04-03T19:44:21.293638+0000","last_undegraded":"2022-04-03T19:44:21.293638+0000","last_fullsized":"2022-04-03T19:44:21.293638+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:28:50.113792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299916+0000","last_change":"2022-04-03T19:44:21.299916+0000","last_active":"2022-04-03T19:44:21.299916+0000","last_peered":"2022-04-03T19:44:21.299916+0000","last_clean":"2022-04-03T19:44:21.299916+0000","last_became_active":"2022-04-03T19:44:21.299578+0000","last_became_peered":"2022-04-03T19:44:21.299578+0000","last_unstale":"2022-04-03T19:44:21.299916+0000","last_undegraded":"2022-04-03T19:44:21.299916+0000","last_fullsized":"2022-04-03T19:44:21.299916+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:38:40.281496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.294003+0000","last_change":"2022-04-03T19:44:21.294003+0000","last_active":"2022-04-03T19:44:21.294003+0000","last_peered":"2022-04-03T19:44:21.294003+0000","last_clean":"2022-04-03T19:44:21.294003+0000","last_became_active":"2022-04-03T19:44:21.293640+0000","last_became_peered":"2022-04-03T19:44:21.293640+0000","last_unstale":"2022-04-03T19:44:21.294003+0000","last_undegraded":"2022-04-03T19:44:21.294003+0000","last_fullsized":"2022-04-03T19:44:21.294003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:40:13.436484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.301158+0000","last_change":"2022-04-03T19:44:21.301158+0000","last_active":"2022-04-03T19:44:21.301158+0000","last_peered":"2022-04-03T19:44:21.301158+0000","last_clean":"2022-04-03T19:44:21.301158+0000","last_became_active":"2022-04-03T19:44:21.300823+0000","last_became_peered":"2022-04-03T19:44:21.300823+0000","last_unstale":"2022-04-03T19:44:21.301158+0000","last_undegraded":"2022-04-03T19:44:21.301158+0000","last_fullsized":"2022-04-03T19:44:21.301158+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:09:28.365828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.295301+0000","last_change":"2022-04-03T19:44:21.295301+0000","last_active":"2022-04-03T19:44:21.295301+0000","last_peered":"2022-04-03T19:44:21.295301+0000","last_clean":"2022-04-03T19:44:21.295301+0000","last_became_active":"2022-04-03T19:44:21.295032+0000","last_became_peered":"2022-04-03T19:44:21.295032+0000","last_unstale":"2022-04-03T19:44:21.295301+0000","last_undegraded":"2022-04-03T19:44:21.295301+0000","last_fullsized":"2022-04-03T19:44:21.295301+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:55:55.637194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.296250+0000","last_change":"2022-04-03T19:44:21.296250+0000","last_active":"2022-04-03T19:44:21.296250+0000","last_peered":"2022-04-03T19:44:21.296250+0000","last_clean":"2022-04-03T19:44:21.296250+0000","last_became_active":"2022-04-03T19:44:21.295967+0000","last_became_peered":"2022-04-03T19:44:21.295967+0000","last_unstale":"2022-04-03T19:44:21.296250+0000","last_undegraded":"2022-04-03T19:44:21.296250+0000","last_fullsized":"2022-04-03T19:44:21.296250+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:43:34.301454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.297229+0000","last_change":"2022-04-03T19:44:21.297229+0000","last_active":"2022-04-03T19:44:21.297229+0000","last_peered":"2022-04-03T19:44:21.297229+0000","last_clean":"2022-04-03T19:44:21.297229+0000","last_became_active":"2022-04-03T19:44:21.296993+0000","last_became_peered":"2022-04-03T19:44:21.296993+0000","last_unstale":"2022-04-03T19:44:21.297229+0000","last_undegraded":"2022-04-03T19:44:21.297229+0000","last_fullsized":"2022-04-03T19:44:21.297229+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:27:17.280580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302427+0000","last_change":"2022-04-03T19:44:21.302427+0000","last_active":"2022-04-03T19:44:21.302427+0000","last_peered":"2022-04-03T19:44:21.302427+0000","last_clean":"2022-04-03T19:44:21.302427+0000","last_became_active":"2022-04-03T19:44:21.302157+0000","last_became_peered":"2022-04-03T19:44:21.302157+0000","last_unstale":"2022-04-03T19:44:21.302427+0000","last_undegraded":"2022-04-03T19:44:21.302427+0000","last_fullsized":"2022-04-03T19:44:21.302427+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:57:23.292801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293096+0000","last_change":"2022-04-03T19:44:21.293096+0000","last_active":"2022-04-03T19:44:21.293096+0000","last_peered":"2022-04-03T19:44:21.293096+0000","last_clean":"2022-04-03T19:44:21.293096+0000","last_became_active":"2022-04-03T19:44:21.292785+0000","last_became_peered":"2022-04-03T19:44:21.292785+0000","last_unstale":"2022-04-03T19:44:21.293096+0000","last_undegraded":"2022-04-03T19:44:21.293096+0000","last_fullsized":"2022-04-03T19:44:21.293096+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:27:06.759308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300957+0000","last_change":"2022-04-03T19:44:21.300957+0000","last_active":"2022-04-03T19:44:21.300957+0000","last_peered":"2022-04-03T19:44:21.300957+0000","last_clean":"2022-04-03T19:44:21.300957+0000","last_became_active":"2022-04-03T19:44:21.300342+0000","last_became_peered":"2022-04-03T19:44:21.300342+0000","last_unstale":"2022-04-03T19:44:21.300957+0000","last_undegraded":"2022-04-03T19:44:21.300957+0000","last_fullsized":"2022-04-03T19:44:21.300957+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:43:53.194230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.295996+0000","last_change":"2022-04-03T19:44:21.295996+0000","last_active":"2022-04-03T19:44:21.295996+0000","last_peered":"2022-04-03T19:44:21.295996+0000","last_clean":"2022-04-03T19:44:21.295996+0000","last_became_active":"2022-04-03T19:44:21.295766+0000","last_became_peered":"2022-04-03T19:44:21.295766+0000","last_unstale":"2022-04-03T19:44:21.295996+0000","last_undegraded":"2022-04-03T19:44:21.295996+0000","last_fullsized":"2022-04-03T19:44:21.295996+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:21:31.628478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299840+0000","last_change":"2022-04-03T19:44:21.299840+0000","last_active":"2022-04-03T19:44:21.299840+0000","last_peered":"2022-04-03T19:44:21.299840+0000","last_clean":"2022-04-03T19:44:21.299840+0000","last_became_active":"2022-04-03T19:44:21.299487+0000","last_became_peered":"2022-04-03T19:44:21.299487+0000","last_unstale":"2022-04-03T19:44:21.299840+0000","last_undegraded":"2022-04-03T19:44:21.299840+0000","last_fullsized":"2022-04-03T19:44:21.299840+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:24:39.583019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.291748+0000","last_change":"2022-04-03T19:44:21.291748+0000","last_active":"2022-04-03T19:44:21.291748+0000","last_peered":"2022-04-03T19:44:21.291748+0000","last_clean":"2022-04-03T19:44:21.291748+0000","last_became_active":"2022-04-03T19:44:21.290939+0000","last_became_peered":"2022-04-03T19:44:21.290939+0000","last_unstale":"2022-04-03T19:44:21.291748+0000","last_undegraded":"2022-04-03T19:44:21.291748+0000","last_fullsized":"2022-04-03T19:44:21.291748+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:28:45.710805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299442+0000","last_change":"2022-04-03T19:44:21.299442+0000","last_active":"2022-04-03T19:44:21.299442+0000","last_peered":"2022-04-03T19:44:21.299442+0000","last_clean":"2022-04-03T19:44:21.299442+0000","last_became_active":"2022-04-03T19:44:21.299166+0000","last_became_peered":"2022-04-03T19:44:21.299166+0000","last_unstale":"2022-04-03T19:44:21.299442+0000","last_undegraded":"2022-04-03T19:44:21.299442+0000","last_fullsized":"2022-04-03T19:44:21.299442+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:31:30.905517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.307007+0000","last_change":"2022-04-03T19:44:21.307007+0000","last_active":"2022-04-03T19:44:21.307007+0000","last_peered":"2022-04-03T19:44:21.307007+0000","last_clean":"2022-04-03T19:44:21.307007+0000","last_became_active":"2022-04-03T19:44:21.306650+0000","last_became_peered":"2022-04-03T19:44:21.306650+0000","last_unstale":"2022-04-03T19:44:21.307007+0000","last_undegraded":"2022-04-03T19:44:21.307007+0000","last_fullsized":"2022-04-03T19:44:21.307007+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:03:26.043624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293566+0000","last_change":"2022-04-03T19:44:21.293566+0000","last_active":"2022-04-03T19:44:21.293566+0000","last_peered":"2022-04-03T19:44:21.293566+0000","last_clean":"2022-04-03T19:44:21.293566+0000","last_became_active":"2022-04-03T19:44:21.293263+0000","last_became_peered":"2022-04-03T19:44:21.293263+0000","last_unstale":"2022-04-03T19:44:21.293566+0000","last_undegraded":"2022-04-03T19:44:21.293566+0000","last_fullsized":"2022-04-03T19:44:21.293566+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:39:52.523944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300329+0000","last_change":"2022-04-03T19:44:21.300329+0000","last_active":"2022-04-03T19:44:21.300329+0000","last_peered":"2022-04-03T19:44:21.300329+0000","last_clean":"2022-04-03T19:44:21.300329+0000","last_became_active":"2022-04-03T19:44:21.300000+0000","last_became_peered":"2022-04-03T19:44:21.300000+0000","last_unstale":"2022-04-03T19:44:21.300329+0000","last_undegraded":"2022-04-03T19:44:21.300329+0000","last_fullsized":"2022-04-03T19:44:21.300329+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:15:10.839571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302506+0000","last_change":"2022-04-03T19:44:21.302506+0000","last_active":"2022-04-03T19:44:21.302506+0000","last_peered":"2022-04-03T19:44:21.302506+0000","last_clean":"2022-04-03T19:44:21.302506+0000","last_became_active":"2022-04-03T19:44:21.302235+0000","last_became_peered":"2022-04-03T19:44:21.302235+0000","last_unstale":"2022-04-03T19:44:21.302506+0000","last_undegraded":"2022-04-03T19:44:21.302506+0000","last_fullsized":"2022-04-03T19:44:21.302506+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:25:25.801754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.306670+0000","last_change":"2022-04-03T19:44:21.306670+0000","last_active":"2022-04-03T19:44:21.306670+0000","last_peered":"2022-04-03T19:44:21.306670+0000","last_clean":"2022-04-03T19:44:21.306670+0000","last_became_active":"2022-04-03T19:44:21.305722+0000","last_became_peered":"2022-04-03T19:44:21.305722+0000","last_unstale":"2022-04-03T19:44:21.306670+0000","last_undegraded":"2022-04-03T19:44:21.306670+0000","last_fullsized":"2022-04-03T19:44:21.306670+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:44:09.515207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300024+0000","last_change":"2022-04-03T19:44:21.300024+0000","last_active":"2022-04-03T19:44:21.300024+0000","last_peered":"2022-04-03T19:44:21.300024+0000","last_clean":"2022-04-03T19:44:21.300024+0000","last_became_active":"2022-04-03T19:44:21.299753+0000","last_became_peered":"2022-04-03T19:44:21.299753+0000","last_unstale":"2022-04-03T19:44:21.300024+0000","last_undegraded":"2022-04-03T19:44:21.300024+0000","last_fullsized":"2022-04-03T19:44:21.300024+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:15:31.808623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.786136+0000","last_change":"2022-04-03T19:44:20.783898+0000","last_active":"2022-04-03T19:44:20.786136+0000","last_peered":"2022-04-03T19:44:20.786136+0000","last_clean":"2022-04-03T19:44:20.786136+0000","last_became_active":"2022-04-03T19:44:20.783612+0000","last_became_peered":"2022-04-03T19:44:20.783612+0000","last_unstale":"2022-04-03T19:44:20.786136+0000","last_undegraded":"2022-04-03T19:44:20.786136+0000","last_fullsized":"2022-04-03T19:44:20.786136+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:13:32.863927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.306904+0000","last_change":"2022-04-03T19:44:21.306904+0000","last_active":"2022-04-03T19:44:21.306904+0000","last_peered":"2022-04-03T19:44:21.306904+0000","last_clean":"2022-04-03T19:44:21.306904+0000","last_became_active":"2022-04-03T19:44:21.306616+0000","last_became_peered":"2022-04-03T19:44:21.306616+0000","last_unstale":"2022-04-03T19:44:21.306904+0000","last_undegraded":"2022-04-03T19:44:21.306904+0000","last_fullsized":"2022-04-03T19:44:21.306904+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:49:35.783912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.798367+0000","last_change":"2022-04-03T19:44:20.788125+0000","last_active":"2022-04-03T19:44:20.798367+0000","last_peered":"2022-04-03T19:44:20.798367+0000","last_clean":"2022-04-03T19:44:20.798367+0000","last_became_active":"2022-04-03T19:44:20.787792+0000","last_became_peered":"2022-04-03T19:44:20.787792+0000","last_unstale":"2022-04-03T19:44:20.798367+0000","last_undegraded":"2022-04-03T19:44:20.798367+0000","last_fullsized":"2022-04-03T19:44:20.798367+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:11:51.126774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302904+0000","last_change":"2022-04-03T19:44:21.302904+0000","last_active":"2022-04-03T19:44:21.302904+0000","last_peered":"2022-04-03T19:44:21.302904+0000","last_clean":"2022-04-03T19:44:21.302904+0000","last_became_active":"2022-04-03T19:44:21.301564+0000","last_became_peered":"2022-04-03T19:44:21.301564+0000","last_unstale":"2022-04-03T19:44:21.302904+0000","last_undegraded":"2022-04-03T19:44:21.302904+0000","last_fullsized":"2022-04-03T19:44:21.302904+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:25:51.611789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.793930+0000","last_change":"2022-04-03T19:44:20.792529+0000","last_active":"2022-04-03T19:44:20.793930+0000","last_peered":"2022-04-03T19:44:20.793930+0000","last_clean":"2022-04-03T19:44:20.793930+0000","last_became_active":"2022-04-03T19:44:20.792014+0000","last_became_peered":"2022-04-03T19:44:20.792014+0000","last_unstale":"2022-04-03T19:44:20.793930+0000","last_undegraded":"2022-04-03T19:44:20.793930+0000","last_fullsized":"2022-04-03T19:44:20.793930+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:01:38.319693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299302+0000","last_change":"2022-04-03T19:44:21.299302+0000","last_active":"2022-04-03T19:44:21.299302+0000","last_peered":"2022-04-03T19:44:21.299302+0000","last_clean":"2022-04-03T19:44:21.299302+0000","last_became_active":"2022-04-03T19:44:21.299007+0000","last_became_peered":"2022-04-03T19:44:21.299007+0000","last_unstale":"2022-04-03T19:44:21.299302+0000","last_undegraded":"2022-04-03T19:44:21.299302+0000","last_fullsized":"2022-04-03T19:44:21.299302+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:59:48.767335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.794765+0000","last_change":"2022-04-03T19:44:20.790672+0000","last_active":"2022-04-03T19:44:20.794765+0000","last_peered":"2022-04-03T19:44:20.794765+0000","last_clean":"2022-04-03T19:44:20.794765+0000","last_became_active":"2022-04-03T19:44:20.788526+0000","last_became_peered":"2022-04-03T19:44:20.788526+0000","last_unstale":"2022-04-03T19:44:20.794765+0000","last_undegraded":"2022-04-03T19:44:20.794765+0000","last_fullsized":"2022-04-03T19:44:20.794765+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:14:50.058917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.306472+0000","last_change":"2022-04-03T19:44:21.306472+0000","last_active":"2022-04-03T19:44:21.306472+0000","last_peered":"2022-04-03T19:44:21.306472+0000","last_clean":"2022-04-03T19:44:21.306472+0000","last_became_active":"2022-04-03T19:44:21.306071+0000","last_became_peered":"2022-04-03T19:44:21.306071+0000","last_unstale":"2022-04-03T19:44:21.306472+0000","last_undegraded":"2022-04-03T19:44:21.306472+0000","last_fullsized":"2022-04-03T19:44:21.306472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:43:22.221014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.788557+0000","last_change":"2022-04-03T19:44:20.785638+0000","last_active":"2022-04-03T19:44:20.788557+0000","last_peered":"2022-04-03T19:44:20.788557+0000","last_clean":"2022-04-03T19:44:20.788557+0000","last_became_active":"2022-04-03T19:44:20.784963+0000","last_became_peered":"2022-04-03T19:44:20.784963+0000","last_unstale":"2022-04-03T19:44:20.788557+0000","last_undegraded":"2022-04-03T19:44:20.788557+0000","last_fullsized":"2022-04-03T19:44:20.788557+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:39:26.933298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299583+0000","last_change":"2022-04-03T19:44:21.299583+0000","last_active":"2022-04-03T19:44:21.299583+0000","last_peered":"2022-04-03T19:44:21.299583+0000","last_clean":"2022-04-03T19:44:21.299583+0000","last_became_active":"2022-04-03T19:44:21.299301+0000","last_became_peered":"2022-04-03T19:44:21.299301+0000","last_unstale":"2022-04-03T19:44:21.299583+0000","last_undegraded":"2022-04-03T19:44:21.299583+0000","last_fullsized":"2022-04-03T19:44:21.299583+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:37:49.567741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.799084+0000","last_change":"2022-04-03T19:44:20.795890+0000","last_active":"2022-04-03T19:44:20.799084+0000","last_peered":"2022-04-03T19:44:20.799084+0000","last_clean":"2022-04-03T19:44:20.799084+0000","last_became_active":"2022-04-03T19:44:20.795611+0000","last_became_peered":"2022-04-03T19:44:20.795611+0000","last_unstale":"2022-04-03T19:44:20.799084+0000","last_undegraded":"2022-04-03T19:44:20.799084+0000","last_fullsized":"2022-04-03T19:44:20.799084+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:08:33.216934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.304724+0000","last_change":"2022-04-03T19:44:21.304724+0000","last_active":"2022-04-03T19:44:21.304724+0000","last_peered":"2022-04-03T19:44:21.304724+0000","last_clean":"2022-04-03T19:44:21.304724+0000","last_became_active":"2022-04-03T19:44:21.302708+0000","last_became_peered":"2022-04-03T19:44:21.302708+0000","last_unstale":"2022-04-03T19:44:21.304724+0000","last_undegraded":"2022-04-03T19:44:21.304724+0000","last_fullsized":"2022-04-03T19:44:21.304724+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:04:54.970100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.790488+0000","last_change":"2022-04-03T19:44:20.785659+0000","last_active":"2022-04-03T19:44:20.790488+0000","last_peered":"2022-04-03T19:44:20.790488+0000","last_clean":"2022-04-03T19:44:20.790488+0000","last_became_active":"2022-04-03T19:44:20.785308+0000","last_became_peered":"2022-04-03T19:44:20.785308+0000","last_unstale":"2022-04-03T19:44:20.790488+0000","last_undegraded":"2022-04-03T19:44:20.790488+0000","last_fullsized":"2022-04-03T19:44:20.790488+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:52:08.268532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.291033+0000","last_change":"2022-04-03T19:44:21.291033+0000","last_active":"2022-04-03T19:44:21.291033+0000","last_peered":"2022-04-03T19:44:21.291033+0000","last_clean":"2022-04-03T19:44:21.291033+0000","last_became_active":"2022-04-03T19:44:21.290713+0000","last_became_peered":"2022-04-03T19:44:21.290713+0000","last_unstale":"2022-04-03T19:44:21.291033+0000","last_undegraded":"2022-04-03T19:44:21.291033+0000","last_fullsized":"2022-04-03T19:44:21.291033+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:19:41.873591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.793588+0000","last_change":"2022-04-03T19:44:20.787511+0000","last_active":"2022-04-03T19:44:20.793588+0000","last_peered":"2022-04-03T19:44:20.793588+0000","last_clean":"2022-04-03T19:44:20.793588+0000","last_became_active":"2022-04-03T19:44:20.787189+0000","last_became_peered":"2022-04-03T19:44:20.787189+0000","last_unstale":"2022-04-03T19:44:20.793588+0000","last_undegraded":"2022-04-03T19:44:20.793588+0000","last_fullsized":"2022-04-03T19:44:20.793588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:11:56.297263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.307846+0000","last_change":"2022-04-03T19:44:21.307846+0000","last_active":"2022-04-03T19:44:21.307846+0000","last_peered":"2022-04-03T19:44:21.307846+0000","last_clean":"2022-04-03T19:44:21.307846+0000","last_became_active":"2022-04-03T19:44:21.307513+0000","last_became_peered":"2022-04-03T19:44:21.307513+0000","last_unstale":"2022-04-03T19:44:21.307846+0000","last_undegraded":"2022-04-03T19:44:21.307846+0000","last_fullsized":"2022-04-03T19:44:21.307846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:58:09.091062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.794878+0000","last_change":"2022-04-03T19:44:20.790212+0000","last_active":"2022-04-03T19:44:20.794878+0000","last_peered":"2022-04-03T19:44:20.794878+0000","last_clean":"2022-04-03T19:44:20.794878+0000","last_became_active":"2022-04-03T19:44:20.789863+0000","last_became_peered":"2022-04-03T19:44:20.789863+0000","last_unstale":"2022-04-03T19:44:20.794878+0000","last_undegraded":"2022-04-03T19:44:20.794878+0000","last_fullsized":"2022-04-03T19:44:20.794878+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:48:13.489309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299953+0000","last_change":"2022-04-03T19:44:21.299953+0000","last_active":"2022-04-03T19:44:21.299953+0000","last_peered":"2022-04-03T19:44:21.299953+0000","last_clean":"2022-04-03T19:44:21.299953+0000","last_became_active":"2022-04-03T19:44:21.299667+0000","last_became_peered":"2022-04-03T19:44:21.299667+0000","last_unstale":"2022-04-03T19:44:21.299953+0000","last_undegraded":"2022-04-03T19:44:21.299953+0000","last_fullsized":"2022-04-03T19:44:21.299953+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:51:16.125160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.796427+0000","last_change":"2022-04-03T19:44:20.791054+0000","last_active":"2022-04-03T19:44:20.796427+0000","last_peered":"2022-04-03T19:44:20.796427+0000","last_clean":"2022-04-03T19:44:20.796427+0000","last_became_active":"2022-04-03T19:44:20.790664+0000","last_became_peered":"2022-04-03T19:44:20.790664+0000","last_unstale":"2022-04-03T19:44:20.796427+0000","last_undegraded":"2022-04-03T19:44:20.796427+0000","last_fullsized":"2022-04-03T19:44:20.796427+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:07:27.501728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.295028+0000","last_change":"2022-04-03T19:44:21.295028+0000","last_active":"2022-04-03T19:44:21.295028+0000","last_peered":"2022-04-03T19:44:21.295028+0000","last_clean":"2022-04-03T19:44:21.295028+0000","last_became_active":"2022-04-03T19:44:21.294723+0000","last_became_peered":"2022-04-03T19:44:21.294723+0000","last_unstale":"2022-04-03T19:44:21.295028+0000","last_undegraded":"2022-04-03T19:44:21.295028+0000","last_fullsized":"2022-04-03T19:44:21.295028+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:32:45.455842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.797096+0000","last_change":"2022-04-03T19:44:20.792133+0000","last_active":"2022-04-03T19:44:20.797096+0000","last_peered":"2022-04-03T19:44:20.797096+0000","last_clean":"2022-04-03T19:44:20.797096+0000","last_became_active":"2022-04-03T19:44:20.791829+0000","last_became_peered":"2022-04-03T19:44:20.791829+0000","last_unstale":"2022-04-03T19:44:20.797096+0000","last_undegraded":"2022-04-03T19:44:20.797096+0000","last_fullsized":"2022-04-03T19:44:20.797096+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:01:21.966734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300267+0000","last_change":"2022-04-03T19:44:21.300267+0000","last_active":"2022-04-03T19:44:21.300267+0000","last_peered":"2022-04-03T19:44:21.300267+0000","last_clean":"2022-04-03T19:44:21.300267+0000","last_became_active":"2022-04-03T19:44:21.299948+0000","last_became_peered":"2022-04-03T19:44:21.299948+0000","last_unstale":"2022-04-03T19:44:21.300267+0000","last_undegraded":"2022-04-03T19:44:21.300267+0000","last_fullsized":"2022-04-03T19:44:21.300267+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:06:54.481379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.796291+0000","last_change":"2022-04-03T19:44:20.789793+0000","last_active":"2022-04-03T19:44:20.796291+0000","last_peered":"2022-04-03T19:44:20.796291+0000","last_clean":"2022-04-03T19:44:20.796291+0000","last_became_active":"2022-04-03T19:44:20.789371+0000","last_became_peered":"2022-04-03T19:44:20.789371+0000","last_unstale":"2022-04-03T19:44:20.796291+0000","last_undegraded":"2022-04-03T19:44:20.796291+0000","last_fullsized":"2022-04-03T19:44:20.796291+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:15:52.060059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.298477+0000","last_change":"2022-04-03T19:44:21.298477+0000","last_active":"2022-04-03T19:44:21.298477+0000","last_peered":"2022-04-03T19:44:21.298477+0000","last_clean":"2022-04-03T19:44:21.298477+0000","last_became_active":"2022-04-03T19:44:21.298165+0000","last_became_peered":"2022-04-03T19:44:21.298165+0000","last_unstale":"2022-04-03T19:44:21.298477+0000","last_undegraded":"2022-04-03T19:44:21.298477+0000","last_fullsized":"2022-04-03T19:44:21.298477+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:23:11.042589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.793792+0000","last_change":"2022-04-03T19:44:20.787949+0000","last_active":"2022-04-03T19:44:20.793792+0000","last_peered":"2022-04-03T19:44:20.793792+0000","last_clean":"2022-04-03T19:44:20.793792+0000","last_became_active":"2022-04-03T19:44:20.787592+0000","last_became_peered":"2022-04-03T19:44:20.787592+0000","last_unstale":"2022-04-03T19:44:20.793792+0000","last_undegraded":"2022-04-03T19:44:20.793792+0000","last_fullsized":"2022-04-03T19:44:20.793792+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:50:09.411246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.301200+0000","last_change":"2022-04-03T19:44:21.301200+0000","last_active":"2022-04-03T19:44:21.301200+0000","last_peered":"2022-04-03T19:44:21.301200+0000","last_clean":"2022-04-03T19:44:21.301200+0000","last_became_active":"2022-04-03T19:44:21.300878+0000","last_became_peered":"2022-04-03T19:44:21.300878+0000","last_unstale":"2022-04-03T19:44:21.301200+0000","last_undegraded":"2022-04-03T19:44:21.301200+0000","last_fullsized":"2022-04-03T19:44:21.301200+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:54:36.089411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.784650+0000","last_change":"2022-04-03T19:44:20.783711+0000","last_active":"2022-04-03T19:44:20.784650+0000","last_peered":"2022-04-03T19:44:20.784650+0000","last_clean":"2022-04-03T19:44:20.784650+0000","last_became_active":"2022-04-03T19:44:20.783423+0000","last_became_peered":"2022-04-03T19:44:20.783423+0000","last_unstale":"2022-04-03T19:44:20.784650+0000","last_undegraded":"2022-04-03T19:44:20.784650+0000","last_fullsized":"2022-04-03T19:44:20.784650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:22:37.665229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300815+0000","last_change":"2022-04-03T19:44:21.300815+0000","last_active":"2022-04-03T19:44:21.300815+0000","last_peered":"2022-04-03T19:44:21.300815+0000","last_clean":"2022-04-03T19:44:21.300815+0000","last_became_active":"2022-04-03T19:44:21.300545+0000","last_became_peered":"2022-04-03T19:44:21.300545+0000","last_unstale":"2022-04-03T19:44:21.300815+0000","last_undegraded":"2022-04-03T19:44:21.300815+0000","last_fullsized":"2022-04-03T19:44:21.300815+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:33:13.941191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.788584+0000","last_change":"2022-04-03T19:44:20.785480+0000","last_active":"2022-04-03T19:44:20.788584+0000","last_peered":"2022-04-03T19:44:20.788584+0000","last_clean":"2022-04-03T19:44:20.788584+0000","last_became_active":"2022-04-03T19:44:20.785178+0000","last_became_peered":"2022-04-03T19:44:20.785178+0000","last_unstale":"2022-04-03T19:44:20.788584+0000","last_undegraded":"2022-04-03T19:44:20.788584+0000","last_fullsized":"2022-04-03T19:44:20.788584+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:15:37.181452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.296509+0000","last_change":"2022-04-03T19:44:21.296509+0000","last_active":"2022-04-03T19:44:21.296509+0000","last_peered":"2022-04-03T19:44:21.296509+0000","last_clean":"2022-04-03T19:44:21.296509+0000","last_became_active":"2022-04-03T19:44:21.296240+0000","last_became_peered":"2022-04-03T19:44:21.296240+0000","last_unstale":"2022-04-03T19:44:21.296509+0000","last_undegraded":"2022-04-03T19:44:21.296509+0000","last_fullsized":"2022-04-03T19:44:21.296509+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:50:45.955195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.793983+0000","last_change":"2022-04-03T19:44:20.792550+0000","last_active":"2022-04-03T19:44:20.793983+0000","last_peered":"2022-04-03T19:44:20.793983+0000","last_clean":"2022-04-03T19:44:20.793983+0000","last_became_active":"2022-04-03T19:44:20.792191+0000","last_became_peered":"2022-04-03T19:44:20.792191+0000","last_unstale":"2022-04-03T19:44:20.793983+0000","last_undegraded":"2022-04-03T19:44:20.793983+0000","last_fullsized":"2022-04-03T19:44:20.793983+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:51:30.297728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.294898+0000","last_change":"2022-04-03T19:44:21.294898+0000","last_active":"2022-04-03T19:44:21.294898+0000","last_peered":"2022-04-03T19:44:21.294898+0000","last_clean":"2022-04-03T19:44:21.294898+0000","last_became_active":"2022-04-03T19:44:21.294635+0000","last_became_peered":"2022-04-03T19:44:21.294635+0000","last_unstale":"2022-04-03T19:44:21.294898+0000","last_undegraded":"2022-04-03T19:44:21.294898+0000","last_fullsized":"2022-04-03T19:44:21.294898+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:42:52.629552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.785610+0000","last_change":"2022-04-03T19:44:20.783711+0000","last_active":"2022-04-03T19:44:20.785610+0000","last_peered":"2022-04-03T19:44:20.785610+0000","last_clean":"2022-04-03T19:44:20.785610+0000","last_became_active":"2022-04-03T19:44:20.783426+0000","last_became_peered":"2022-04-03T19:44:20.783426+0000","last_unstale":"2022-04-03T19:44:20.785610+0000","last_undegraded":"2022-04-03T19:44:20.785610+0000","last_fullsized":"2022-04-03T19:44:20.785610+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:25:26.219558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.296081+0000","last_change":"2022-04-03T19:44:21.296081+0000","last_active":"2022-04-03T19:44:21.296081+0000","last_peered":"2022-04-03T19:44:21.296081+0000","last_clean":"2022-04-03T19:44:21.296081+0000","last_became_active":"2022-04-03T19:44:21.295784+0000","last_became_peered":"2022-04-03T19:44:21.295784+0000","last_unstale":"2022-04-03T19:44:21.296081+0000","last_undegraded":"2022-04-03T19:44:21.296081+0000","last_fullsized":"2022-04-03T19:44:21.296081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:23:40.568168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.791873+0000","last_change":"2022-04-03T19:44:20.789868+0000","last_active":"2022-04-03T19:44:20.791873+0000","last_peered":"2022-04-03T19:44:20.791873+0000","last_clean":"2022-04-03T19:44:20.791873+0000","last_became_active":"2022-04-03T19:44:20.789563+0000","last_became_peered":"2022-04-03T19:44:20.789563+0000","last_unstale":"2022-04-03T19:44:20.791873+0000","last_undegraded":"2022-04-03T19:44:20.791873+0000","last_fullsized":"2022-04-03T19:44:20.791873+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:04:22.814932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.291212+0000","last_change":"2022-04-03T19:44:21.291212+0000","last_active":"2022-04-03T19:44:21.291212+0000","last_peered":"2022-04-03T19:44:21.291212+0000","last_clean":"2022-04-03T19:44:21.291212+0000","last_became_active":"2022-04-03T19:44:21.290888+0000","last_became_peered":"2022-04-03T19:44:21.290888+0000","last_unstale":"2022-04-03T19:44:21.291212+0000","last_undegraded":"2022-04-03T19:44:21.291212+0000","last_fullsized":"2022-04-03T19:44:21.291212+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:29:34.850679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.793246+0000","last_change":"2022-04-03T19:44:20.789319+0000","last_active":"2022-04-03T19:44:20.793246+0000","last_peered":"2022-04-03T19:44:20.793246+0000","last_clean":"2022-04-03T19:44:20.793246+0000","last_became_active":"2022-04-03T19:44:20.788840+0000","last_became_peered":"2022-04-03T19:44:20.788840+0000","last_unstale":"2022-04-03T19:44:20.793246+0000","last_undegraded":"2022-04-03T19:44:20.793246+0000","last_fullsized":"2022-04-03T19:44:20.793246+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:50:21.536926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.311886+0000","last_change":"2022-04-03T19:44:21.311886+0000","last_active":"2022-04-03T19:44:21.311886+0000","last_peered":"2022-04-03T19:44:21.311886+0000","last_clean":"2022-04-03T19:44:21.311886+0000","last_became_active":"2022-04-03T19:44:21.311521+0000","last_became_peered":"2022-04-03T19:44:21.311521+0000","last_unstale":"2022-04-03T19:44:21.311886+0000","last_undegraded":"2022-04-03T19:44:21.311886+0000","last_fullsized":"2022-04-03T19:44:21.311886+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:25:47.761166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.780927+0000","last_change":"2022-04-03T19:44:14.754674+0000","last_active":"2022-04-03T19:44:20.780927+0000","last_peered":"2022-04-03T19:44:20.780927+0000","last_clean":"2022-04-03T19:44:20.780927+0000","last_became_active":"2022-04-03T19:44:14.754210+0000","last_became_peered":"2022-04-03T19:44:14.754210+0000","last_unstale":"2022-04-03T19:44:20.780927+0000","last_undegraded":"2022-04-03T19:44:20.780927+0000","last_fullsized":"2022-04-03T19:44:20.780927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:23:14.624494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.788936+0000","last_change":"2022-04-03T19:44:20.785400+0000","last_active":"2022-04-03T19:44:20.788936+0000","last_peered":"2022-04-03T19:44:20.788936+0000","last_clean":"2022-04-03T19:44:20.788936+0000","last_became_active":"2022-04-03T19:44:20.785125+0000","last_became_peered":"2022-04-03T19:44:20.785125+0000","last_unstale":"2022-04-03T19:44:20.788936+0000","last_undegraded":"2022-04-03T19:44:20.788936+0000","last_fullsized":"2022-04-03T19:44:20.788936+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:24:02.690646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305321+0000","last_change":"2022-04-03T19:44:21.305321+0000","last_active":"2022-04-03T19:44:21.305321+0000","last_peered":"2022-04-03T19:44:21.305321+0000","last_clean":"2022-04-03T19:44:21.305321+0000","last_became_active":"2022-04-03T19:44:21.304017+0000","last_became_peered":"2022-04-03T19:44:21.304017+0000","last_unstale":"2022-04-03T19:44:21.305321+0000","last_undegraded":"2022-04-03T19:44:21.305321+0000","last_fullsized":"2022-04-03T19:44:21.305321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:59:12.619058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.779686+0000","last_change":"2022-04-03T19:44:14.755580+0000","last_active":"2022-04-03T19:44:20.779686+0000","last_peered":"2022-04-03T19:44:20.779686+0000","last_clean":"2022-04-03T19:44:20.779686+0000","last_became_active":"2022-04-03T19:44:14.755222+0000","last_became_peered":"2022-04-03T19:44:14.755222+0000","last_unstale":"2022-04-03T19:44:20.779686+0000","last_undegraded":"2022-04-03T19:44:20.779686+0000","last_fullsized":"2022-04-03T19:44:20.779686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:08:18.314326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.796640+0000","last_change":"2022-04-03T19:44:20.791540+0000","last_active":"2022-04-03T19:44:20.796640+0000","last_peered":"2022-04-03T19:44:20.796640+0000","last_clean":"2022-04-03T19:44:20.796640+0000","last_became_active":"2022-04-03T19:44:20.791192+0000","last_became_peered":"2022-04-03T19:44:20.791192+0000","last_unstale":"2022-04-03T19:44:20.796640+0000","last_undegraded":"2022-04-03T19:44:20.796640+0000","last_fullsized":"2022-04-03T19:44:20.796640+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:58:59.674606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305930+0000","last_change":"2022-04-03T19:44:21.305930+0000","last_active":"2022-04-03T19:44:21.305930+0000","last_peered":"2022-04-03T19:44:21.305930+0000","last_clean":"2022-04-03T19:44:21.305930+0000","last_became_active":"2022-04-03T19:44:21.305659+0000","last_became_peered":"2022-04-03T19:44:21.305659+0000","last_unstale":"2022-04-03T19:44:21.305930+0000","last_undegraded":"2022-04-03T19:44:21.305930+0000","last_fullsized":"2022-04-03T19:44:21.305930+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:14:34.958569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.773041+0000","last_change":"2022-04-03T19:44:14.754598+0000","last_active":"2022-04-03T19:44:20.773041+0000","last_peered":"2022-04-03T19:44:20.773041+0000","last_clean":"2022-04-03T19:44:20.773041+0000","last_became_active":"2022-04-03T19:44:14.754287+0000","last_became_peered":"2022-04-03T19:44:14.754287+0000","last_unstale":"2022-04-03T19:44:20.773041+0000","last_undegraded":"2022-04-03T19:44:20.773041+0000","last_fullsized":"2022-04-03T19:44:20.773041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:32:16.232689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.798752+0000","last_change":"2022-04-03T19:44:20.789567+0000","last_active":"2022-04-03T19:44:20.798752+0000","last_peered":"2022-04-03T19:44:20.798752+0000","last_clean":"2022-04-03T19:44:20.798752+0000","last_became_active":"2022-04-03T19:44:20.788664+0000","last_became_peered":"2022-04-03T19:44:20.788664+0000","last_unstale":"2022-04-03T19:44:20.798752+0000","last_undegraded":"2022-04-03T19:44:20.798752+0000","last_fullsized":"2022-04-03T19:44:20.798752+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:15:37.098494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303079+0000","last_change":"2022-04-03T19:44:21.303079+0000","last_active":"2022-04-03T19:44:21.303079+0000","last_peered":"2022-04-03T19:44:21.303079+0000","last_clean":"2022-04-03T19:44:21.303079+0000","last_became_active":"2022-04-03T19:44:21.302014+0000","last_became_peered":"2022-04-03T19:44:21.302014+0000","last_unstale":"2022-04-03T19:44:21.303079+0000","last_undegraded":"2022-04-03T19:44:21.303079+0000","last_fullsized":"2022-04-03T19:44:21.303079+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:15:58.860701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.779613+0000","last_change":"2022-04-03T19:44:14.754385+0000","last_active":"2022-04-03T19:44:20.779613+0000","last_peered":"2022-04-03T19:44:20.779613+0000","last_clean":"2022-04-03T19:44:20.779613+0000","last_became_active":"2022-04-03T19:44:14.754003+0000","last_became_peered":"2022-04-03T19:44:14.754003+0000","last_unstale":"2022-04-03T19:44:20.779613+0000","last_undegraded":"2022-04-03T19:44:20.779613+0000","last_fullsized":"2022-04-03T19:44:20.779613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:05:22.716933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.778008+0000","last_change":"2022-04-03T19:44:08.726758+0000","last_active":"2022-04-03T19:44:20.778008+0000","last_peered":"2022-04-03T19:44:20.778008+0000","last_clean":"2022-04-03T19:44:20.778008+0000","last_became_active":"2022-04-03T19:44:08.725694+0000","last_became_peered":"2022-04-03T19:44:08.725694+0000","last_unstale":"2022-04-03T19:44:20.778008+0000","last_undegraded":"2022-04-03T19:44:20.778008+0000","last_fullsized":"2022-04-03T19:44:20.778008+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:07.708129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:07.708129+0000","last_clean_scrub_stamp":"2022-04-03T19:44:07.708129+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:27:43.354267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.786014+0000","last_change":"2022-04-03T19:44:20.781585+0000","last_active":"2022-04-03T19:44:20.786014+0000","last_peered":"2022-04-03T19:44:20.786014+0000","last_clean":"2022-04-03T19:44:20.786014+0000","last_became_active":"2022-04-03T19:44:20.781274+0000","last_became_peered":"2022-04-03T19:44:20.781274+0000","last_unstale":"2022-04-03T19:44:20.786014+0000","last_undegraded":"2022-04-03T19:44:20.786014+0000","last_fullsized":"2022-04-03T19:44:20.786014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:49:54.368001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.296620+0000","last_change":"2022-04-03T19:44:21.296620+0000","last_active":"2022-04-03T19:44:21.296620+0000","last_peered":"2022-04-03T19:44:21.296620+0000","last_clean":"2022-04-03T19:44:21.296620+0000","last_became_active":"2022-04-03T19:44:21.296314+0000","last_became_peered":"2022-04-03T19:44:21.296314+0000","last_unstale":"2022-04-03T19:44:21.296620+0000","last_undegraded":"2022-04-03T19:44:21.296620+0000","last_fullsized":"2022-04-03T19:44:21.296620+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:09:18.338683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.778178+0000","last_change":"2022-04-03T19:44:14.754664+0000","last_active":"2022-04-03T19:44:20.778178+0000","last_peered":"2022-04-03T19:44:20.778178+0000","last_clean":"2022-04-03T19:44:20.778178+0000","last_became_active":"2022-04-03T19:44:14.754328+0000","last_became_peered":"2022-04-03T19:44:14.754328+0000","last_unstale":"2022-04-03T19:44:20.778178+0000","last_undegraded":"2022-04-03T19:44:20.778178+0000","last_fullsized":"2022-04-03T19:44:20.778178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:39:58.022353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.793867+0000","last_change":"2022-04-03T19:44:20.787271+0000","last_active":"2022-04-03T19:44:20.793867+0000","last_peered":"2022-04-03T19:44:20.793867+0000","last_clean":"2022-04-03T19:44:20.793867+0000","last_became_active":"2022-04-03T19:44:20.786957+0000","last_became_peered":"2022-04-03T19:44:20.786957+0000","last_unstale":"2022-04-03T19:44:20.793867+0000","last_undegraded":"2022-04-03T19:44:20.793867+0000","last_fullsized":"2022-04-03T19:44:20.793867+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:21:45.992744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.306857+0000","last_change":"2022-04-03T19:44:21.306857+0000","last_active":"2022-04-03T19:44:21.306857+0000","last_peered":"2022-04-03T19:44:21.306857+0000","last_clean":"2022-04-03T19:44:21.306857+0000","last_became_active":"2022-04-03T19:44:21.306521+0000","last_became_peered":"2022-04-03T19:44:21.306521+0000","last_unstale":"2022-04-03T19:44:21.306857+0000","last_undegraded":"2022-04-03T19:44:21.306857+0000","last_fullsized":"2022-04-03T19:44:21.306857+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:50:03.912834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.779281+0000","last_change":"2022-04-03T19:44:14.754657+0000","last_active":"2022-04-03T19:44:20.779281+0000","last_peered":"2022-04-03T19:44:20.779281+0000","last_clean":"2022-04-03T19:44:20.779281+0000","last_became_active":"2022-04-03T19:44:14.754310+0000","last_became_peered":"2022-04-03T19:44:14.754310+0000","last_unstale":"2022-04-03T19:44:20.779281+0000","last_undegraded":"2022-04-03T19:44:20.779281+0000","last_fullsized":"2022-04-03T19:44:20.779281+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:42:05.279424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.798841+0000","last_change":"2022-04-03T19:44:20.794454+0000","last_active":"2022-04-03T19:44:20.798841+0000","last_peered":"2022-04-03T19:44:20.798841+0000","last_clean":"2022-04-03T19:44:20.798841+0000","last_became_active":"2022-04-03T19:44:20.789468+0000","last_became_peered":"2022-04-03T19:44:20.789468+0000","last_unstale":"2022-04-03T19:44:20.798841+0000","last_undegraded":"2022-04-03T19:44:20.798841+0000","last_fullsized":"2022-04-03T19:44:20.798841+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:02:49.184838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303669+0000","last_change":"2022-04-03T19:44:21.303669+0000","last_active":"2022-04-03T19:44:21.303669+0000","last_peered":"2022-04-03T19:44:21.303669+0000","last_clean":"2022-04-03T19:44:21.303669+0000","last_became_active":"2022-04-03T19:44:21.303381+0000","last_became_peered":"2022-04-03T19:44:21.303381+0000","last_unstale":"2022-04-03T19:44:21.303669+0000","last_undegraded":"2022-04-03T19:44:21.303669+0000","last_fullsized":"2022-04-03T19:44:21.303669+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:43:22.320070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.794015+0000","last_change":"2022-04-03T19:44:20.791594+0000","last_active":"2022-04-03T19:44:20.794015+0000","last_peered":"2022-04-03T19:44:20.794015+0000","last_clean":"2022-04-03T19:44:20.794015+0000","last_became_active":"2022-04-03T19:44:20.789693+0000","last_became_peered":"2022-04-03T19:44:20.789693+0000","last_unstale":"2022-04-03T19:44:20.794015+0000","last_undegraded":"2022-04-03T19:44:20.794015+0000","last_fullsized":"2022-04-03T19:44:20.794015+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:18:21.832066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.780669+0000","last_change":"2022-04-03T19:44:14.755096+0000","last_active":"2022-04-03T19:44:20.780669+0000","last_peered":"2022-04-03T19:44:20.780669+0000","last_clean":"2022-04-03T19:44:20.780669+0000","last_became_active":"2022-04-03T19:44:14.754799+0000","last_became_peered":"2022-04-03T19:44:14.754799+0000","last_unstale":"2022-04-03T19:44:20.780669+0000","last_undegraded":"2022-04-03T19:44:20.780669+0000","last_fullsized":"2022-04-03T19:44:20.780669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:04:17.179114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299649+0000","last_change":"2022-04-03T19:44:21.299649+0000","last_active":"2022-04-03T19:44:21.299649+0000","last_peered":"2022-04-03T19:44:21.299649+0000","last_clean":"2022-04-03T19:44:21.299649+0000","last_became_active":"2022-04-03T19:44:21.299399+0000","last_became_peered":"2022-04-03T19:44:21.299399+0000","last_unstale":"2022-04-03T19:44:21.299649+0000","last_undegraded":"2022-04-03T19:44:21.299649+0000","last_fullsized":"2022-04-03T19:44:21.299649+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:13:06.313729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.772869+0000","last_change":"2022-04-03T19:44:14.754659+0000","last_active":"2022-04-03T19:44:20.772869+0000","last_peered":"2022-04-03T19:44:20.772869+0000","last_clean":"2022-04-03T19:44:20.772869+0000","last_became_active":"2022-04-03T19:44:14.754386+0000","last_became_peered":"2022-04-03T19:44:14.754386+0000","last_unstale":"2022-04-03T19:44:20.772869+0000","last_undegraded":"2022-04-03T19:44:20.772869+0000","last_fullsized":"2022-04-03T19:44:20.772869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:33:33.013536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.791124+0000","last_change":"2022-04-03T19:44:20.784840+0000","last_active":"2022-04-03T19:44:20.791124+0000","last_peered":"2022-04-03T19:44:20.791124+0000","last_clean":"2022-04-03T19:44:20.791124+0000","last_became_active":"2022-04-03T19:44:20.784521+0000","last_became_peered":"2022-04-03T19:44:20.784521+0000","last_unstale":"2022-04-03T19:44:20.791124+0000","last_undegraded":"2022-04-03T19:44:20.791124+0000","last_fullsized":"2022-04-03T19:44:20.791124+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:15:38.240804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.290769+0000","last_change":"2022-04-03T19:44:21.290769+0000","last_active":"2022-04-03T19:44:21.290769+0000","last_peered":"2022-04-03T19:44:21.290769+0000","last_clean":"2022-04-03T19:44:21.290769+0000","last_became_active":"2022-04-03T19:44:21.290394+0000","last_became_peered":"2022-04-03T19:44:21.290394+0000","last_unstale":"2022-04-03T19:44:21.290769+0000","last_undegraded":"2022-04-03T19:44:21.290769+0000","last_fullsized":"2022-04-03T19:44:21.290769+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:57:39.528871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.794710+0000","last_change":"2022-04-03T19:44:20.789662+0000","last_active":"2022-04-03T19:44:20.794710+0000","last_peered":"2022-04-03T19:44:20.794710+0000","last_clean":"2022-04-03T19:44:20.794710+0000","last_became_active":"2022-04-03T19:44:20.787868+0000","last_became_peered":"2022-04-03T19:44:20.787868+0000","last_unstale":"2022-04-03T19:44:20.794710+0000","last_undegraded":"2022-04-03T19:44:20.794710+0000","last_fullsized":"2022-04-03T19:44:20.794710+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:13:16.092180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.304539+0000","last_change":"2022-04-03T19:44:21.304539+0000","last_active":"2022-04-03T19:44:21.304539+0000","last_peered":"2022-04-03T19:44:21.304539+0000","last_clean":"2022-04-03T19:44:21.304539+0000","last_became_active":"2022-04-03T19:44:21.304008+0000","last_became_peered":"2022-04-03T19:44:21.304008+0000","last_unstale":"2022-04-03T19:44:21.304539+0000","last_undegraded":"2022-04-03T19:44:21.304539+0000","last_fullsized":"2022-04-03T19:44:21.304539+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:45:01.346479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.790383+0000","last_change":"2022-04-03T19:44:20.786813+0000","last_active":"2022-04-03T19:44:20.790383+0000","last_peered":"2022-04-03T19:44:20.790383+0000","last_clean":"2022-04-03T19:44:20.790383+0000","last_became_active":"2022-04-03T19:44:20.786548+0000","last_became_peered":"2022-04-03T19:44:20.786548+0000","last_unstale":"2022-04-03T19:44:20.790383+0000","last_undegraded":"2022-04-03T19:44:20.790383+0000","last_fullsized":"2022-04-03T19:44:20.790383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:36:40.611720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299513+0000","last_change":"2022-04-03T19:44:21.299513+0000","last_active":"2022-04-03T19:44:21.299513+0000","last_peered":"2022-04-03T19:44:21.299513+0000","last_clean":"2022-04-03T19:44:21.299513+0000","last_became_active":"2022-04-03T19:44:21.299270+0000","last_became_peered":"2022-04-03T19:44:21.299270+0000","last_unstale":"2022-04-03T19:44:21.299513+0000","last_undegraded":"2022-04-03T19:44:21.299513+0000","last_fullsized":"2022-04-03T19:44:21.299513+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:54:16.302815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.790374+0000","last_change":"2022-04-03T19:44:20.786750+0000","last_active":"2022-04-03T19:44:20.790374+0000","last_peered":"2022-04-03T19:44:20.790374+0000","last_clean":"2022-04-03T19:44:20.790374+0000","last_became_active":"2022-04-03T19:44:20.786490+0000","last_became_peered":"2022-04-03T19:44:20.786490+0000","last_unstale":"2022-04-03T19:44:20.790374+0000","last_undegraded":"2022-04-03T19:44:20.790374+0000","last_fullsized":"2022-04-03T19:44:20.790374+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:30:31.980806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.294849+0000","last_change":"2022-04-03T19:44:21.294849+0000","last_active":"2022-04-03T19:44:21.294849+0000","last_peered":"2022-04-03T19:44:21.294849+0000","last_clean":"2022-04-03T19:44:21.294849+0000","last_became_active":"2022-04-03T19:44:21.294586+0000","last_became_peered":"2022-04-03T19:44:21.294586+0000","last_unstale":"2022-04-03T19:44:21.294849+0000","last_undegraded":"2022-04-03T19:44:21.294849+0000","last_fullsized":"2022-04-03T19:44:21.294849+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:24:11.050193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.786040+0000","last_change":"2022-04-03T19:44:20.781619+0000","last_active":"2022-04-03T19:44:20.786040+0000","last_peered":"2022-04-03T19:44:20.786040+0000","last_clean":"2022-04-03T19:44:20.786040+0000","last_became_active":"2022-04-03T19:44:20.781288+0000","last_became_peered":"2022-04-03T19:44:20.781288+0000","last_unstale":"2022-04-03T19:44:20.786040+0000","last_undegraded":"2022-04-03T19:44:20.786040+0000","last_fullsized":"2022-04-03T19:44:20.786040+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:05:56.179441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.296332+0000","last_change":"2022-04-03T19:44:21.296332+0000","last_active":"2022-04-03T19:44:21.296332+0000","last_peered":"2022-04-03T19:44:21.296332+0000","last_clean":"2022-04-03T19:44:21.296332+0000","last_became_active":"2022-04-03T19:44:21.296026+0000","last_became_peered":"2022-04-03T19:44:21.296026+0000","last_unstale":"2022-04-03T19:44:21.296332+0000","last_undegraded":"2022-04-03T19:44:21.296332+0000","last_fullsized":"2022-04-03T19:44:21.296332+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:43:33.822187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.794613+0000","last_change":"2022-04-03T19:44:20.789448+0000","last_active":"2022-04-03T19:44:20.794613+0000","last_peered":"2022-04-03T19:44:20.794613+0000","last_clean":"2022-04-03T19:44:20.794613+0000","last_became_active":"2022-04-03T19:44:20.789131+0000","last_became_peered":"2022-04-03T19:44:20.789131+0000","last_unstale":"2022-04-03T19:44:20.794613+0000","last_undegraded":"2022-04-03T19:44:20.794613+0000","last_fullsized":"2022-04-03T19:44:20.794613+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:46:20.830676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.291681+0000","last_change":"2022-04-03T19:44:21.291681+0000","last_active":"2022-04-03T19:44:21.291681+0000","last_peered":"2022-04-03T19:44:21.291681+0000","last_clean":"2022-04-03T19:44:21.291681+0000","last_became_active":"2022-04-03T19:44:21.290817+0000","last_became_peered":"2022-04-03T19:44:21.290817+0000","last_unstale":"2022-04-03T19:44:21.291681+0000","last_undegraded":"2022-04-03T19:44:21.291681+0000","last_fullsized":"2022-04-03T19:44:21.291681+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:37:30.833818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.295604+0000","last_change":"2022-04-03T19:44:21.295604+0000","last_active":"2022-04-03T19:44:21.295604+0000","last_peered":"2022-04-03T19:44:21.295604+0000","last_clean":"2022-04-03T19:44:21.295604+0000","last_became_active":"2022-04-03T19:44:21.295297+0000","last_became_peered":"2022-04-03T19:44:21.295297+0000","last_unstale":"2022-04-03T19:44:21.295604+0000","last_undegraded":"2022-04-03T19:44:21.295604+0000","last_fullsized":"2022-04-03T19:44:21.295604+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:26:34.341290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.306547+0000","last_change":"2022-04-03T19:44:21.306547+0000","last_active":"2022-04-03T19:44:21.306547+0000","last_peered":"2022-04-03T19:44:21.306547+0000","last_clean":"2022-04-03T19:44:21.306547+0000","last_became_active":"2022-04-03T19:44:21.306179+0000","last_became_peered":"2022-04-03T19:44:21.306179+0000","last_unstale":"2022-04-03T19:44:21.306547+0000","last_undegraded":"2022-04-03T19:44:21.306547+0000","last_fullsized":"2022-04-03T19:44:21.306547+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:34:58.854671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.291341+0000","last_change":"2022-04-03T19:44:21.291341+0000","last_active":"2022-04-03T19:44:21.291341+0000","last_peered":"2022-04-03T19:44:21.291341+0000","last_clean":"2022-04-03T19:44:21.291341+0000","last_became_active":"2022-04-03T19:44:21.290943+0000","last_became_peered":"2022-04-03T19:44:21.290943+0000","last_unstale":"2022-04-03T19:44:21.291341+0000","last_undegraded":"2022-04-03T19:44:21.291341+0000","last_fullsized":"2022-04-03T19:44:21.291341+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:50:56.877454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.307815+0000","last_change":"2022-04-03T19:44:21.307815+0000","last_active":"2022-04-03T19:44:21.307815+0000","last_peered":"2022-04-03T19:44:21.307815+0000","last_clean":"2022-04-03T19:44:21.307815+0000","last_became_active":"2022-04-03T19:44:21.307459+0000","last_became_peered":"2022-04-03T19:44:21.307459+0000","last_unstale":"2022-04-03T19:44:21.307815+0000","last_undegraded":"2022-04-03T19:44:21.307815+0000","last_fullsized":"2022-04-03T19:44:21.307815+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:16:52.723409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303095+0000","last_change":"2022-04-03T19:44:21.303095+0000","last_active":"2022-04-03T19:44:21.303095+0000","last_peered":"2022-04-03T19:44:21.303095+0000","last_clean":"2022-04-03T19:44:21.303095+0000","last_became_active":"2022-04-03T19:44:21.302363+0000","last_became_peered":"2022-04-03T19:44:21.302363+0000","last_unstale":"2022-04-03T19:44:21.303095+0000","last_undegraded":"2022-04-03T19:44:21.303095+0000","last_fullsized":"2022-04-03T19:44:21.303095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:42:45.617661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8880,"kb_used_data":744,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94362960,"statfs":{"total":96636764160,"available":96627671040,"internally_reserved":0,"allocated":761856,"data_stored":609145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0589999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]}]},{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8880,"kb_used_data":744,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94362960,"statfs":{"total":96636764160,"available":96627671040,"internally_reserved":0,"allocated":761856,"data_stored":609145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7860,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":7616,"kb_avail":94363980,"statfs":{"total":96636764160,"available":96628715520,"internally_reserved":0,"allocated":241664,"data_stored":87333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7798784},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7772,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":94364068,"statfs":{"total":96636764160,"available":96628805632,"internally_reserved":0,"allocated":217088,"data_stored":73609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7772,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":94364068,"statfs":{"total":96636764160,"available":96628805632,"internally_reserved":0,"allocated":217088,"data_stored":73609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7848,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":7616,"kb_avail":94363992,"statfs":{"total":96636764160,"available":96628727808,"internally_reserved":0,"allocated":229376,"data_stored":80217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7798784},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7848,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":7616,"kb_avail":94363992,"statfs":{"total":96636764160,"available":96628727808,"internally_reserved":0,"allocated":229376,"data_stored":80217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7798784},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7848,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":7616,"kb_avail":94363992,"statfs":{"total":96636764160,"available":96628727808,"internally_reserved":0,"allocated":229376,"data_stored":80217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7798784},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-03T19:44:26.676 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-04-03T19:44:26.904 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.904+0000 7fb0ebfff700 1 Processor -- start 2022-04-03T19:44:26.905 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.904+0000 7fb0ebfff700 1 -- start start 2022-04-03T19:44:26.906 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.905+0000 7fb0ebfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb0ec1584b0 0x7fb0ec14eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:26.906 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.905+0000 7fb0ebfff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec06d260 con 0x7fb0ec06bfb0 2022-04-03T19:44:26.906 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.905+0000 7fb0ebfff700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec0c6330 con 0x7fb0ec06bbe0 2022-04-03T19:44:26.906 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.905+0000 7fb0ebfff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0ec159450 con 0x7fb0ec1584b0 2022-04-03T19:44:26.907 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.905+0000 7fb0ea7fc700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb0ec06bfb0 0x7fb0ec06c500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38296/0 (socket says 172.21.15.77:38296) 2022-04-03T19:44:26.907 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.905+0000 7fb0ea7fc700 1 -- 172.21.15.77:0/2062961855 learned_addr learned my addr 172.21.15.77:0/2062961855 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:26.907 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.905+0000 7fb0eb7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb0ec1584b0 0x7fb0ec14eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:26.909 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.906+0000 7fb0eb7fe700 1 -- 172.21.15.77:0/2062961855 >> v1:172.21.15.186:6789/0 conn(0x7fb0ec06bbe0 legacy=0x7fb0ec06d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:26.909 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.906+0000 7fb0eb7fe700 1 -- 172.21.15.77:0/2062961855 >> v1:172.21.15.77:6789/0 conn(0x7fb0ec06bfb0 legacy=0x7fb0ec06c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:26.909 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.906+0000 7fb0eb7fe700 1 -- 172.21.15.77:0/2062961855 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ec158910 con 0x7fb0ec1584b0 2022-04-03T19:44:26.910 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.906+0000 7fb0e9ffb700 1 -- 172.21.15.77:0/2062961855 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272990178 0 0) 0x7fb0ec06d260 con 0x7fb0ec06bfb0 2022-04-03T19:44:26.910 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.906+0000 7fb0eb7fe700 1 --2- 172.21.15.77:0/2062961855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0ec1584b0 0x7fb0ec14eea0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009e60 tx=0x7fb0e4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53bd8420af662f63 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:26.910 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.906+0000 7fb0e9ffb700 1 -- 172.21.15.77:0/2062961855 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e400e070 con 0x7fb0ec1584b0 2022-04-03T19:44:26.910 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.906+0000 7fb0e9ffb700 1 -- 172.21.15.77:0/2062961855 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e400bc30 con 0x7fb0ec1584b0 2022-04-03T19:44:26.910 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.907+0000 7fb0e9ffb700 1 -- 172.21.15.77:0/2062961855 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4012430 con 0x7fb0ec1584b0 2022-04-03T19:44:26.911 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.907+0000 7fb0ebfff700 1 -- 172.21.15.77:0/2062961855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0ec1584b0 msgr2=0x7fb0ec14eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:26.911 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.907+0000 7fb0ebfff700 1 --2- 172.21.15.77:0/2062961855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0ec1584b0 0x7fb0ec14eea0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009e60 tx=0x7fb0e4009780 comp rx=0 tx=0).stop 2022-04-03T19:44:26.911 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.907+0000 7fb0ebfff700 1 -- 172.21.15.77:0/2062961855 shutdown_connections 2022-04-03T19:44:26.911 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.907+0000 7fb0ebfff700 1 --2- 172.21.15.77:0/2062961855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0ec1584b0 0x7fb0ec14eea0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:26.911 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.907+0000 7fb0ebfff700 1 -- 172.21.15.77:0/2062961855 >> 172.21.15.77:0/2062961855 conn(0x7fb0ec0cb6b0 msgr2=0x7fb0ec0cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:26.912 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.907+0000 7fb0ebfff700 1 -- 172.21.15.77:0/2062961855 shutdown_connections 2022-04-03T19:44:26.912 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.907+0000 7fb0ebfff700 1 -- 172.21.15.77:0/2062961855 wait complete. 2022-04-03T19:44:26.912 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.909+0000 7fb0ebfff700 1 Processor -- start 2022-04-03T19:44:26.912 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.909+0000 7fb0ebfff700 1 -- start start 2022-04-03T19:44:26.915 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.910+0000 7fb0ebfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0ec06bbe0 0x7fb0ec14fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:26.915 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.910+0000 7fb0ebfff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0ec06bfb0 0x7fb0ec149fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:26.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.910+0000 7fb0ebfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0ec1584b0 0x7fb0ec14ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:26.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.910+0000 7fb0ebfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec14a520 con 0x7fb0ec1584b0 2022-04-03T19:44:26.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.910+0000 7fb0ebfff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec155d90 con 0x7fb0ec06bbe0 2022-04-03T19:44:26.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.910+0000 7fb0ebfff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec1651c0 con 0x7fb0ec06bfb0 2022-04-03T19:44:26.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.910+0000 7fb0eb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0ec1584b0 0x7fb0ec14ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:26.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.910+0000 7fb0eb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0ec1584b0 0x7fb0ec14ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42514/0 (socket says 172.21.15.77:42514) 2022-04-03T19:44:26.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.910+0000 7fb0eb7fe700 1 -- 172.21.15.77:0/3413868686 learned_addr learned my addr 172.21.15.77:0/3413868686 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:26.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.910+0000 7fb0eaffd700 1 --2- 172.21.15.77:0/3413868686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0ec06bbe0 0x7fb0ec14fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:26.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.910+0000 7fb0ea7fc700 1 --2- 172.21.15.77:0/3413868686 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0ec06bfb0 0x7fb0ec149fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:26.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.910+0000 7fb0eb7fe700 1 -- 172.21.15.77:0/3413868686 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0ec06bfb0 msgr2=0x7fb0ec149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:26.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.910+0000 7fb0eb7fe700 1 --2- 172.21.15.77:0/3413868686 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0ec06bfb0 0x7fb0ec149fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:26.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.910+0000 7fb0eb7fe700 1 -- 172.21.15.77:0/3413868686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0ec06bbe0 msgr2=0x7fb0ec14fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:26.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.910+0000 7fb0eb7fe700 1 --2- 172.21.15.77:0/3413868686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0ec06bbe0 0x7fb0ec14fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:26.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.910+0000 7fb0eb7fe700 1 -- 172.21.15.77:0/3413868686 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e4009a20 con 0x7fb0ec1584b0 2022-04-03T19:44:26.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.910+0000 7fb0eaffd700 1 --2- 172.21.15.77:0/3413868686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0ec06bbe0 0x7fb0ec14fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:44:26.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.911+0000 7fb0eb7fe700 1 --2- 172.21.15.77:0/3413868686 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0ec1584b0 0x7fb0ec14ee30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec0632c0 tx=0x7fb0e4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:26.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.911+0000 7fb0d2ffd700 1 -- 172.21.15.77:0/3413868686 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e401b630 con 0x7fb0ec1584b0 2022-04-03T19:44:26.920 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.911+0000 7fb0d2ffd700 1 -- 172.21.15.77:0/3413868686 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e401b7d0 con 0x7fb0ec1584b0 2022-04-03T19:44:26.920 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.911+0000 7fb0d2ffd700 1 -- 172.21.15.77:0/3413868686 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e401bb90 con 0x7fb0ec1584b0 2022-04-03T19:44:26.920 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.911+0000 7fb0ebfff700 1 -- 172.21.15.77:0/3413868686 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ec15a8a0 con 0x7fb0ec1584b0 2022-04-03T19:44:26.920 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.911+0000 7fb0ebfff700 1 -- 172.21.15.77:0/3413868686 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ec155ab0 con 0x7fb0ec1584b0 2022-04-03T19:44:26.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.912+0000 7fb0ebfff700 1 -- 172.21.15.77:0/3413868686 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0ec165300 con 0x7fb0ec1584b0 2022-04-03T19:44:26.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.913+0000 7fb0d2ffd700 1 -- 172.21.15.77:0/3413868686 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fb0e402c030 con 0x7fb0ec1584b0 2022-04-03T19:44:26.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.914+0000 7fb0d2ffd700 1 --2- 172.21.15.77:0/3413868686 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb0cc063d10 0x7fb0cc0661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:26.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.914+0000 7fb0d2ffd700 1 -- 172.21.15.77:0/3413868686 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fb0e4082700 con 0x7fb0ec1584b0 2022-04-03T19:44:26.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.914+0000 7fb0eaffd700 1 --2- 172.21.15.77:0/3413868686 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb0cc063d10 0x7fb0cc0661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:26.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.915+0000 7fb0eaffd700 1 --2- 172.21.15.77:0/3413868686 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb0cc063d10 0x7fb0cc0661c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc001730 tx=0x7fb0dc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:26.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:26.918+0000 7fb0d2ffd700 1 -- 172.21.15.77:0/3413868686 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fb0e4051330 con 0x7fb0ec1584b0 2022-04-03T19:44:27.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.111+0000 7fb0ebfff700 1 -- 172.21.15.77:0/3413868686 --> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb0ec04b050 con 0x7fb0cc063d10 2022-04-03T19:44:27.120 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.120+0000 7fb0d2ffd700 1 -- 172.21.15.77:0/3413868686 <== mgr.4106 v2:172.21.15.186:6832/24948 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284640 (secure 0 0 0) 0x7fb0ec04b050 con 0x7fb0cc063d10 2022-04-03T19:44:27.124 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:27.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.124+0000 7fb0ebfff700 1 -- 172.21.15.77:0/3413868686 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb0cc063d10 msgr2=0x7fb0cc0661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:27.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.124+0000 7fb0ebfff700 1 --2- 172.21.15.77:0/3413868686 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb0cc063d10 0x7fb0cc0661c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc001730 tx=0x7fb0dc009040 comp rx=0 tx=0).stop 2022-04-03T19:44:27.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.124+0000 7fb0ebfff700 1 -- 172.21.15.77:0/3413868686 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0ec1584b0 msgr2=0x7fb0ec14ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:27.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.124+0000 7fb0ebfff700 1 --2- 172.21.15.77:0/3413868686 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0ec1584b0 0x7fb0ec14ee30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec0632c0 tx=0x7fb0e4019000 comp rx=0 tx=0).stop 2022-04-03T19:44:27.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.124+0000 7fb0ebfff700 1 -- 172.21.15.77:0/3413868686 shutdown_connections 2022-04-03T19:44:27.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.124+0000 7fb0ebfff700 1 --2- 172.21.15.77:0/3413868686 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0ec1584b0 0x7fb0ec14ee30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:27.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.124+0000 7fb0ebfff700 1 --2- 172.21.15.77:0/3413868686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0ec06bbe0 0x7fb0ec14fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:27.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.124+0000 7fb0ebfff700 1 --2- 172.21.15.77:0/3413868686 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb0cc063d10 0x7fb0cc0661c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:27.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.124+0000 7fb0ebfff700 1 --2- 172.21.15.77:0/3413868686 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0ec06bfb0 0x7fb0ec149fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:27.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.124+0000 7fb0ebfff700 1 -- 172.21.15.77:0/3413868686 >> 172.21.15.77:0/3413868686 conn(0x7fb0ec0cb6b0 msgr2=0x7fb0ec1521b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:27.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.124+0000 7fb0ebfff700 1 -- 172.21.15.77:0/3413868686 shutdown_connections 2022-04-03T19:44:27.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.124+0000 7fb0ebfff700 1 -- 172.21.15.77:0/3413868686 wait complete. 2022-04-03T19:44:27.129 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2022-04-03T19:44:27.149 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2022-04-03T19:44:25.489206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":64708,"kb_used_data":2820,"kb_used_omap":0,"kb_used_meta":61824,"kb_avail":754910012,"statfs":{"total":773094113280,"available":773027852288,"internally_reserved":0,"allocated":2887680,"data_stored":1693492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63307776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.724527"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300867+0000","last_change":"2022-04-03T19:44:21.300867+0000","last_active":"2022-04-03T19:44:21.300867+0000","last_peered":"2022-04-03T19:44:21.300867+0000","last_clean":"2022-04-03T19:44:21.300867+0000","last_became_active":"2022-04-03T19:44:21.300569+0000","last_became_peered":"2022-04-03T19:44:21.300569+0000","last_unstale":"2022-04-03T19:44:21.300867+0000","last_undegraded":"2022-04-03T19:44:21.300867+0000","last_fullsized":"2022-04-03T19:44:21.300867+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:10:32.827314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302410+0000","last_change":"2022-04-03T19:44:21.302410+0000","last_active":"2022-04-03T19:44:21.302410+0000","last_peered":"2022-04-03T19:44:21.302410+0000","last_clean":"2022-04-03T19:44:21.302410+0000","last_became_active":"2022-04-03T19:44:21.302142+0000","last_became_peered":"2022-04-03T19:44:21.302142+0000","last_unstale":"2022-04-03T19:44:21.302410+0000","last_undegraded":"2022-04-03T19:44:21.302410+0000","last_fullsized":"2022-04-03T19:44:21.302410+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:01:46.327996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293080+0000","last_change":"2022-04-03T19:44:21.293080+0000","last_active":"2022-04-03T19:44:21.293080+0000","last_peered":"2022-04-03T19:44:21.293080+0000","last_clean":"2022-04-03T19:44:21.293080+0000","last_became_active":"2022-04-03T19:44:21.292762+0000","last_became_peered":"2022-04-03T19:44:21.292762+0000","last_unstale":"2022-04-03T19:44:21.293080+0000","last_undegraded":"2022-04-03T19:44:21.293080+0000","last_fullsized":"2022-04-03T19:44:21.293080+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:28:36.726593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.294448+0000","last_change":"2022-04-03T19:44:21.294448+0000","last_active":"2022-04-03T19:44:21.294448+0000","last_peered":"2022-04-03T19:44:21.294448+0000","last_clean":"2022-04-03T19:44:21.294448+0000","last_became_active":"2022-04-03T19:44:21.294083+0000","last_became_peered":"2022-04-03T19:44:21.294083+0000","last_unstale":"2022-04-03T19:44:21.294448+0000","last_undegraded":"2022-04-03T19:44:21.294448+0000","last_fullsized":"2022-04-03T19:44:21.294448+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:57:02.374826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299551+0000","last_change":"2022-04-03T19:44:21.299551+0000","last_active":"2022-04-03T19:44:21.299551+0000","last_peered":"2022-04-03T19:44:21.299551+0000","last_clean":"2022-04-03T19:44:21.299551+0000","last_became_active":"2022-04-03T19:44:21.299123+0000","last_became_peered":"2022-04-03T19:44:21.299123+0000","last_unstale":"2022-04-03T19:44:21.299551+0000","last_undegraded":"2022-04-03T19:44:21.299551+0000","last_fullsized":"2022-04-03T19:44:21.299551+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:08:24.911647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293129+0000","last_change":"2022-04-03T19:44:21.293129+0000","last_active":"2022-04-03T19:44:21.293129+0000","last_peered":"2022-04-03T19:44:21.293129+0000","last_clean":"2022-04-03T19:44:21.293129+0000","last_became_active":"2022-04-03T19:44:21.292843+0000","last_became_peered":"2022-04-03T19:44:21.292843+0000","last_unstale":"2022-04-03T19:44:21.293129+0000","last_undegraded":"2022-04-03T19:44:21.293129+0000","last_fullsized":"2022-04-03T19:44:21.293129+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:38:32.432609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299112+0000","last_change":"2022-04-03T19:44:21.299112+0000","last_active":"2022-04-03T19:44:21.299112+0000","last_peered":"2022-04-03T19:44:21.299112+0000","last_clean":"2022-04-03T19:44:21.299112+0000","last_became_active":"2022-04-03T19:44:21.298803+0000","last_became_peered":"2022-04-03T19:44:21.298803+0000","last_unstale":"2022-04-03T19:44:21.299112+0000","last_undegraded":"2022-04-03T19:44:21.299112+0000","last_fullsized":"2022-04-03T19:44:21.299112+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:29:43.577843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293638+0000","last_change":"2022-04-03T19:44:21.293638+0000","last_active":"2022-04-03T19:44:21.293638+0000","last_peered":"2022-04-03T19:44:21.293638+0000","last_clean":"2022-04-03T19:44:21.293638+0000","last_became_active":"2022-04-03T19:44:21.293359+0000","last_became_peered":"2022-04-03T19:44:21.293359+0000","last_unstale":"2022-04-03T19:44:21.293638+0000","last_undegraded":"2022-04-03T19:44:21.293638+0000","last_fullsized":"2022-04-03T19:44:21.293638+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:28:50.113792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299916+0000","last_change":"2022-04-03T19:44:21.299916+0000","last_active":"2022-04-03T19:44:21.299916+0000","last_peered":"2022-04-03T19:44:21.299916+0000","last_clean":"2022-04-03T19:44:21.299916+0000","last_became_active":"2022-04-03T19:44:21.299578+0000","last_became_peered":"2022-04-03T19:44:21.299578+0000","last_unstale":"2022-04-03T19:44:21.299916+0000","last_undegraded":"2022-04-03T19:44:21.299916+0000","last_fullsized":"2022-04-03T19:44:21.299916+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:38:40.281496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.294003+0000","last_change":"2022-04-03T19:44:21.294003+0000","last_active":"2022-04-03T19:44:21.294003+0000","last_peered":"2022-04-03T19:44:21.294003+0000","last_clean":"2022-04-03T19:44:21.294003+0000","last_became_active":"2022-04-03T19:44:21.293640+0000","last_became_peered":"2022-04-03T19:44:21.293640+0000","last_unstale":"2022-04-03T19:44:21.294003+0000","last_undegraded":"2022-04-03T19:44:21.294003+0000","last_fullsized":"2022-04-03T19:44:21.294003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:40:13.436484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.301158+0000","last_change":"2022-04-03T19:44:21.301158+0000","last_active":"2022-04-03T19:44:21.301158+0000","last_peered":"2022-04-03T19:44:21.301158+0000","last_clean":"2022-04-03T19:44:21.301158+0000","last_became_active":"2022-04-03T19:44:21.300823+0000","last_became_peered":"2022-04-03T19:44:21.300823+0000","last_unstale":"2022-04-03T19:44:21.301158+0000","last_undegraded":"2022-04-03T19:44:21.301158+0000","last_fullsized":"2022-04-03T19:44:21.301158+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:09:28.365828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.295301+0000","last_change":"2022-04-03T19:44:21.295301+0000","last_active":"2022-04-03T19:44:21.295301+0000","last_peered":"2022-04-03T19:44:21.295301+0000","last_clean":"2022-04-03T19:44:21.295301+0000","last_became_active":"2022-04-03T19:44:21.295032+0000","last_became_peered":"2022-04-03T19:44:21.295032+0000","last_unstale":"2022-04-03T19:44:21.295301+0000","last_undegraded":"2022-04-03T19:44:21.295301+0000","last_fullsized":"2022-04-03T19:44:21.295301+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:55:55.637194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.296250+0000","last_change":"2022-04-03T19:44:21.296250+0000","last_active":"2022-04-03T19:44:21.296250+0000","last_peered":"2022-04-03T19:44:21.296250+0000","last_clean":"2022-04-03T19:44:21.296250+0000","last_became_active":"2022-04-03T19:44:21.295967+0000","last_became_peered":"2022-04-03T19:44:21.295967+0000","last_unstale":"2022-04-03T19:44:21.296250+0000","last_undegraded":"2022-04-03T19:44:21.296250+0000","last_fullsized":"2022-04-03T19:44:21.296250+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:43:34.301454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.297229+0000","last_change":"2022-04-03T19:44:21.297229+0000","last_active":"2022-04-03T19:44:21.297229+0000","last_peered":"2022-04-03T19:44:21.297229+0000","last_clean":"2022-04-03T19:44:21.297229+0000","last_became_active":"2022-04-03T19:44:21.296993+0000","last_became_peered":"2022-04-03T19:44:21.296993+0000","last_unstale":"2022-04-03T19:44:21.297229+0000","last_undegraded":"2022-04-03T19:44:21.297229+0000","last_fullsized":"2022-04-03T19:44:21.297229+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:27:17.280580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302427+0000","last_change":"2022-04-03T19:44:21.302427+0000","last_active":"2022-04-03T19:44:21.302427+0000","last_peered":"2022-04-03T19:44:21.302427+0000","last_clean":"2022-04-03T19:44:21.302427+0000","last_became_active":"2022-04-03T19:44:21.302157+0000","last_became_peered":"2022-04-03T19:44:21.302157+0000","last_unstale":"2022-04-03T19:44:21.302427+0000","last_undegraded":"2022-04-03T19:44:21.302427+0000","last_fullsized":"2022-04-03T19:44:21.302427+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:57:23.292801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293096+0000","last_change":"2022-04-03T19:44:21.293096+0000","last_active":"2022-04-03T19:44:21.293096+0000","last_peered":"2022-04-03T19:44:21.293096+0000","last_clean":"2022-04-03T19:44:21.293096+0000","last_became_active":"2022-04-03T19:44:21.292785+0000","last_became_peered":"2022-04-03T19:44:21.292785+0000","last_unstale":"2022-04-03T19:44:21.293096+0000","last_undegraded":"2022-04-03T19:44:21.293096+0000","last_fullsized":"2022-04-03T19:44:21.293096+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:27:06.759308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300957+0000","last_change":"2022-04-03T19:44:21.300957+0000","last_active":"2022-04-03T19:44:21.300957+0000","last_peered":"2022-04-03T19:44:21.300957+0000","last_clean":"2022-04-03T19:44:21.300957+0000","last_became_active":"2022-04-03T19:44:21.300342+0000","last_became_peered":"2022-04-03T19:44:21.300342+0000","last_unstale":"2022-04-03T19:44:21.300957+0000","last_undegraded":"2022-04-03T19:44:21.300957+0000","last_fullsized":"2022-04-03T19:44:21.300957+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:43:53.194230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.295996+0000","last_change":"2022-04-03T19:44:21.295996+0000","last_active":"2022-04-03T19:44:21.295996+0000","last_peered":"2022-04-03T19:44:21.295996+0000","last_clean":"2022-04-03T19:44:21.295996+0000","last_became_active":"2022-04-03T19:44:21.295766+0000","last_became_peered":"2022-04-03T19:44:21.295766+0000","last_unstale":"2022-04-03T19:44:21.295996+0000","last_undegraded":"2022-04-03T19:44:21.295996+0000","last_fullsized":"2022-04-03T19:44:21.295996+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:21:31.628478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299840+0000","last_change":"2022-04-03T19:44:21.299840+0000","last_active":"2022-04-03T19:44:21.299840+0000","last_peered":"2022-04-03T19:44:21.299840+0000","last_clean":"2022-04-03T19:44:21.299840+0000","last_became_active":"2022-04-03T19:44:21.299487+0000","last_became_peered":"2022-04-03T19:44:21.299487+0000","last_unstale":"2022-04-03T19:44:21.299840+0000","last_undegraded":"2022-04-03T19:44:21.299840+0000","last_fullsized":"2022-04-03T19:44:21.299840+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:24:39.583019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.291748+0000","last_change":"2022-04-03T19:44:21.291748+0000","last_active":"2022-04-03T19:44:21.291748+0000","last_peered":"2022-04-03T19:44:21.291748+0000","last_clean":"2022-04-03T19:44:21.291748+0000","last_became_active":"2022-04-03T19:44:21.290939+0000","last_became_peered":"2022-04-03T19:44:21.290939+0000","last_unstale":"2022-04-03T19:44:21.291748+0000","last_undegraded":"2022-04-03T19:44:21.291748+0000","last_fullsized":"2022-04-03T19:44:21.291748+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:28:45.710805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299442+0000","last_change":"2022-04-03T19:44:21.299442+0000","last_active":"2022-04-03T19:44:21.299442+0000","last_peered":"2022-04-03T19:44:21.299442+0000","last_clean":"2022-04-03T19:44:21.299442+0000","last_became_active":"2022-04-03T19:44:21.299166+0000","last_became_peered":"2022-04-03T19:44:21.299166+0000","last_unstale":"2022-04-03T19:44:21.299442+0000","last_undegraded":"2022-04-03T19:44:21.299442+0000","last_fullsized":"2022-04-03T19:44:21.299442+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:31:30.905517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.307007+0000","last_change":"2022-04-03T19:44:21.307007+0000","last_active":"2022-04-03T19:44:21.307007+0000","last_peered":"2022-04-03T19:44:21.307007+0000","last_clean":"2022-04-03T19:44:21.307007+0000","last_became_active":"2022-04-03T19:44:21.306650+0000","last_became_peered":"2022-04-03T19:44:21.306650+0000","last_unstale":"2022-04-03T19:44:21.307007+0000","last_undegraded":"2022-04-03T19:44:21.307007+0000","last_fullsized":"2022-04-03T19:44:21.307007+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:03:26.043624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293566+0000","last_change":"2022-04-03T19:44:21.293566+0000","last_active":"2022-04-03T19:44:21.293566+0000","last_peered":"2022-04-03T19:44:21.293566+0000","last_clean":"2022-04-03T19:44:21.293566+0000","last_became_active":"2022-04-03T19:44:21.293263+0000","last_became_peered":"2022-04-03T19:44:21.293263+0000","last_unstale":"2022-04-03T19:44:21.293566+0000","last_undegraded":"2022-04-03T19:44:21.293566+0000","last_fullsized":"2022-04-03T19:44:21.293566+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:39:52.523944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300329+0000","last_change":"2022-04-03T19:44:21.300329+0000","last_active":"2022-04-03T19:44:21.300329+0000","last_peered":"2022-04-03T19:44:21.300329+0000","last_clean":"2022-04-03T19:44:21.300329+0000","last_became_active":"2022-04-03T19:44:21.300000+0000","last_became_peered":"2022-04-03T19:44:21.300000+0000","last_unstale":"2022-04-03T19:44:21.300329+0000","last_undegraded":"2022-04-03T19:44:21.300329+0000","last_fullsized":"2022-04-03T19:44:21.300329+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:15:10.839571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302506+0000","last_change":"2022-04-03T19:44:21.302506+0000","last_active":"2022-04-03T19:44:21.302506+0000","last_peered":"2022-04-03T19:44:21.302506+0000","last_clean":"2022-04-03T19:44:21.302506+0000","last_became_active":"2022-04-03T19:44:21.302235+0000","last_became_peered":"2022-04-03T19:44:21.302235+0000","last_unstale":"2022-04-03T19:44:21.302506+0000","last_undegraded":"2022-04-03T19:44:21.302506+0000","last_fullsized":"2022-04-03T19:44:21.302506+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:25:25.801754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.306670+0000","last_change":"2022-04-03T19:44:21.306670+0000","last_active":"2022-04-03T19:44:21.306670+0000","last_peered":"2022-04-03T19:44:21.306670+0000","last_clean":"2022-04-03T19:44:21.306670+0000","last_became_active":"2022-04-03T19:44:21.305722+0000","last_became_peered":"2022-04-03T19:44:21.305722+0000","last_unstale":"2022-04-03T19:44:21.306670+0000","last_undegraded":"2022-04-03T19:44:21.306670+0000","last_fullsized":"2022-04-03T19:44:21.306670+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:44:09.515207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300024+0000","last_change":"2022-04-03T19:44:21.300024+0000","last_active":"2022-04-03T19:44:21.300024+0000","last_peered":"2022-04-03T19:44:21.300024+0000","last_clean":"2022-04-03T19:44:21.300024+0000","last_became_active":"2022-04-03T19:44:21.299753+0000","last_became_peered":"2022-04-03T19:44:21.299753+0000","last_unstale":"2022-04-03T19:44:21.300024+0000","last_undegraded":"2022-04-03T19:44:21.300024+0000","last_fullsized":"2022-04-03T19:44:21.300024+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:15:31.808623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.786136+0000","last_change":"2022-04-03T19:44:20.783898+0000","last_active":"2022-04-03T19:44:20.786136+0000","last_peered":"2022-04-03T19:44:20.786136+0000","last_clean":"2022-04-03T19:44:20.786136+0000","last_became_active":"2022-04-03T19:44:20.783612+0000","last_became_peered":"2022-04-03T19:44:20.783612+0000","last_unstale":"2022-04-03T19:44:20.786136+0000","last_undegraded":"2022-04-03T19:44:20.786136+0000","last_fullsized":"2022-04-03T19:44:20.786136+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:13:32.863927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.306904+0000","last_change":"2022-04-03T19:44:21.306904+0000","last_active":"2022-04-03T19:44:21.306904+0000","last_peered":"2022-04-03T19:44:21.306904+0000","last_clean":"2022-04-03T19:44:21.306904+0000","last_became_active":"2022-04-03T19:44:21.306616+0000","last_became_peered":"2022-04-03T19:44:21.306616+0000","last_unstale":"2022-04-03T19:44:21.306904+0000","last_undegraded":"2022-04-03T19:44:21.306904+0000","last_fullsized":"2022-04-03T19:44:21.306904+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:49:35.783912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.798367+0000","last_change":"2022-04-03T19:44:20.788125+0000","last_active":"2022-04-03T19:44:20.798367+0000","last_peered":"2022-04-03T19:44:20.798367+0000","last_clean":"2022-04-03T19:44:20.798367+0000","last_became_active":"2022-04-03T19:44:20.787792+0000","last_became_peered":"2022-04-03T19:44:20.787792+0000","last_unstale":"2022-04-03T19:44:20.798367+0000","last_undegraded":"2022-04-03T19:44:20.798367+0000","last_fullsized":"2022-04-03T19:44:20.798367+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:11:51.126774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302904+0000","last_change":"2022-04-03T19:44:21.302904+0000","last_active":"2022-04-03T19:44:21.302904+0000","last_peered":"2022-04-03T19:44:21.302904+0000","last_clean":"2022-04-03T19:44:21.302904+0000","last_became_active":"2022-04-03T19:44:21.301564+0000","last_became_peered":"2022-04-03T19:44:21.301564+0000","last_unstale":"2022-04-03T19:44:21.302904+0000","last_undegraded":"2022-04-03T19:44:21.302904+0000","last_fullsized":"2022-04-03T19:44:21.302904+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:25:51.611789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.793930+0000","last_change":"2022-04-03T19:44:20.792529+0000","last_active":"2022-04-03T19:44:20.793930+0000","last_peered":"2022-04-03T19:44:20.793930+0000","last_clean":"2022-04-03T19:44:20.793930+0000","last_became_active":"2022-04-03T19:44:20.792014+0000","last_became_peered":"2022-04-03T19:44:20.792014+0000","last_unstale":"2022-04-03T19:44:20.793930+0000","last_undegraded":"2022-04-03T19:44:20.793930+0000","last_fullsized":"2022-04-03T19:44:20.793930+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:01:38.319693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299302+0000","last_change":"2022-04-03T19:44:21.299302+0000","last_active":"2022-04-03T19:44:21.299302+0000","last_peered":"2022-04-03T19:44:21.299302+0000","last_clean":"2022-04-03T19:44:21.299302+0000","last_became_active":"2022-04-03T19:44:21.299007+0000","last_became_peered":"2022-04-03T19:44:21.299007+0000","last_unstale":"2022-04-03T19:44:21.299302+0000","last_undegraded":"2022-04-03T19:44:21.299302+0000","last_fullsized":"2022-04-03T19:44:21.299302+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:59:48.767335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.794765+0000","last_change":"2022-04-03T19:44:20.790672+0000","last_active":"2022-04-03T19:44:20.794765+0000","last_peered":"2022-04-03T19:44:20.794765+0000","last_clean":"2022-04-03T19:44:20.794765+0000","last_became_active":"2022-04-03T19:44:20.788526+0000","last_became_peered":"2022-04-03T19:44:20.788526+0000","last_unstale":"2022-04-03T19:44:20.794765+0000","last_undegraded":"2022-04-03T19:44:20.794765+0000","last_fullsized":"2022-04-03T19:44:20.794765+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:14:50.058917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.306472+0000","last_change":"2022-04-03T19:44:21.306472+0000","last_active":"2022-04-03T19:44:21.306472+0000","last_peered":"2022-04-03T19:44:21.306472+0000","last_clean":"2022-04-03T19:44:21.306472+0000","last_became_active":"2022-04-03T19:44:21.306071+0000","last_became_peered":"2022-04-03T19:44:21.306071+0000","last_unstale":"2022-04-03T19:44:21.306472+0000","last_undegraded":"2022-04-03T19:44:21.306472+0000","last_fullsized":"2022-04-03T19:44:21.306472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:43:22.221014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.788557+0000","last_change":"2022-04-03T19:44:20.785638+0000","last_active":"2022-04-03T19:44:20.788557+0000","last_peered":"2022-04-03T19:44:20.788557+0000","last_clean":"2022-04-03T19:44:20.788557+0000","last_became_active":"2022-04-03T19:44:20.784963+0000","last_became_peered":"2022-04-03T19:44:20.784963+0000","last_unstale":"2022-04-03T19:44:20.788557+0000","last_undegraded":"2022-04-03T19:44:20.788557+0000","last_fullsized":"2022-04-03T19:44:20.788557+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:39:26.933298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299583+0000","last_change":"2022-04-03T19:44:21.299583+0000","last_active":"2022-04-03T19:44:21.299583+0000","last_peered":"2022-04-03T19:44:21.299583+0000","last_clean":"2022-04-03T19:44:21.299583+0000","last_became_active":"2022-04-03T19:44:21.299301+0000","last_became_peered":"2022-04-03T19:44:21.299301+0000","last_unstale":"2022-04-03T19:44:21.299583+0000","last_undegraded":"2022-04-03T19:44:21.299583+0000","last_fullsized":"2022-04-03T19:44:21.299583+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:37:49.567741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.799084+0000","last_change":"2022-04-03T19:44:20.795890+0000","last_active":"2022-04-03T19:44:20.799084+0000","last_peered":"2022-04-03T19:44:20.799084+0000","last_clean":"2022-04-03T19:44:20.799084+0000","last_became_active":"2022-04-03T19:44:20.795611+0000","last_became_peered":"2022-04-03T19:44:20.795611+0000","last_unstale":"2022-04-03T19:44:20.799084+0000","last_undegraded":"2022-04-03T19:44:20.799084+0000","last_fullsized":"2022-04-03T19:44:20.799084+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:08:33.216934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.304724+0000","last_change":"2022-04-03T19:44:21.304724+0000","last_active":"2022-04-03T19:44:21.304724+0000","last_peered":"2022-04-03T19:44:21.304724+0000","last_clean":"2022-04-03T19:44:21.304724+0000","last_became_active":"2022-04-03T19:44:21.302708+0000","last_became_peered":"2022-04-03T19:44:21.302708+0000","last_unstale":"2022-04-03T19:44:21.304724+0000","last_undegraded":"2022-04-03T19:44:21.304724+0000","last_fullsized":"2022-04-03T19:44:21.304724+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:04:54.970100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.790488+0000","last_change":"2022-04-03T19:44:20.785659+0000","last_active":"2022-04-03T19:44:20.790488+0000","last_peered":"2022-04-03T19:44:20.790488+0000","last_clean":"2022-04-03T19:44:20.790488+0000","last_became_active":"2022-04-03T19:44:20.785308+0000","last_became_peered":"2022-04-03T19:44:20.785308+0000","last_unstale":"2022-04-03T19:44:20.790488+0000","last_undegraded":"2022-04-03T19:44:20.790488+0000","last_fullsized":"2022-04-03T19:44:20.790488+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:52:08.268532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.291033+0000","last_change":"2022-04-03T19:44:21.291033+0000","last_active":"2022-04-03T19:44:21.291033+0000","last_peered":"2022-04-03T19:44:21.291033+0000","last_clean":"2022-04-03T19:44:21.291033+0000","last_became_active":"2022-04-03T19:44:21.290713+0000","last_became_peered":"2022-04-03T19:44:21.290713+0000","last_unstale":"2022-04-03T19:44:21.291033+0000","last_undegraded":"2022-04-03T19:44:21.291033+0000","last_fullsized":"2022-04-03T19:44:21.291033+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:19:41.873591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.793588+0000","last_change":"2022-04-03T19:44:20.787511+0000","last_active":"2022-04-03T19:44:20.793588+0000","last_peered":"2022-04-03T19:44:20.793588+0000","last_clean":"2022-04-03T19:44:20.793588+0000","last_became_active":"2022-04-03T19:44:20.787189+0000","last_became_peered":"2022-04-03T19:44:20.787189+0000","last_unstale":"2022-04-03T19:44:20.793588+0000","last_undegraded":"2022-04-03T19:44:20.793588+0000","last_fullsized":"2022-04-03T19:44:20.793588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:11:56.297263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.307846+0000","last_change":"2022-04-03T19:44:21.307846+0000","last_active":"2022-04-03T19:44:21.307846+0000","last_peered":"2022-04-03T19:44:21.307846+0000","last_clean":"2022-04-03T19:44:21.307846+0000","last_became_active":"2022-04-03T19:44:21.307513+0000","last_became_peered":"2022-04-03T19:44:21.307513+0000","last_unstale":"2022-04-03T19:44:21.307846+0000","last_undegraded":"2022-04-03T19:44:21.307846+0000","last_fullsized":"2022-04-03T19:44:21.307846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:58:09.091062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.794878+0000","last_change":"2022-04-03T19:44:20.790212+0000","last_active":"2022-04-03T19:44:20.794878+0000","last_peered":"2022-04-03T19:44:20.794878+0000","last_clean":"2022-04-03T19:44:20.794878+0000","last_became_active":"2022-04-03T19:44:20.789863+0000","last_became_peered":"2022-04-03T19:44:20.789863+0000","last_unstale":"2022-04-03T19:44:20.794878+0000","last_undegraded":"2022-04-03T19:44:20.794878+0000","last_fullsized":"2022-04-03T19:44:20.794878+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:48:13.489309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299953+0000","last_change":"2022-04-03T19:44:21.299953+0000","last_active":"2022-04-03T19:44:21.299953+0000","last_peered":"2022-04-03T19:44:21.299953+0000","last_clean":"2022-04-03T19:44:21.299953+0000","last_became_active":"2022-04-03T19:44:21.299667+0000","last_became_peered":"2022-04-03T19:44:21.299667+0000","last_unstale":"2022-04-03T19:44:21.299953+0000","last_undegraded":"2022-04-03T19:44:21.299953+0000","last_fullsized":"2022-04-03T19:44:21.299953+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:51:16.125160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.796427+0000","last_change":"2022-04-03T19:44:20.791054+0000","last_active":"2022-04-03T19:44:20.796427+0000","last_peered":"2022-04-03T19:44:20.796427+0000","last_clean":"2022-04-03T19:44:20.796427+0000","last_became_active":"2022-04-03T19:44:20.790664+0000","last_became_peered":"2022-04-03T19:44:20.790664+0000","last_unstale":"2022-04-03T19:44:20.796427+0000","last_undegraded":"2022-04-03T19:44:20.796427+0000","last_fullsized":"2022-04-03T19:44:20.796427+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:07:27.501728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.295028+0000","last_change":"2022-04-03T19:44:21.295028+0000","last_active":"2022-04-03T19:44:21.295028+0000","last_peered":"2022-04-03T19:44:21.295028+0000","last_clean":"2022-04-03T19:44:21.295028+0000","last_became_active":"2022-04-03T19:44:21.294723+0000","last_became_peered":"2022-04-03T19:44:21.294723+0000","last_unstale":"2022-04-03T19:44:21.295028+0000","last_undegraded":"2022-04-03T19:44:21.295028+0000","last_fullsized":"2022-04-03T19:44:21.295028+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:32:45.455842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.797096+0000","last_change":"2022-04-03T19:44:20.792133+0000","last_active":"2022-04-03T19:44:20.797096+0000","last_peered":"2022-04-03T19:44:20.797096+0000","last_clean":"2022-04-03T19:44:20.797096+0000","last_became_active":"2022-04-03T19:44:20.791829+0000","last_became_peered":"2022-04-03T19:44:20.791829+0000","last_unstale":"2022-04-03T19:44:20.797096+0000","last_undegraded":"2022-04-03T19:44:20.797096+0000","last_fullsized":"2022-04-03T19:44:20.797096+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:01:21.966734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300267+0000","last_change":"2022-04-03T19:44:21.300267+0000","last_active":"2022-04-03T19:44:21.300267+0000","last_peered":"2022-04-03T19:44:21.300267+0000","last_clean":"2022-04-03T19:44:21.300267+0000","last_became_active":"2022-04-03T19:44:21.299948+0000","last_became_peered":"2022-04-03T19:44:21.299948+0000","last_unstale":"2022-04-03T19:44:21.300267+0000","last_undegraded":"2022-04-03T19:44:21.300267+0000","last_fullsized":"2022-04-03T19:44:21.300267+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:06:54.481379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.796291+0000","last_change":"2022-04-03T19:44:20.789793+0000","last_active":"2022-04-03T19:44:20.796291+0000","last_peered":"2022-04-03T19:44:20.796291+0000","last_clean":"2022-04-03T19:44:20.796291+0000","last_became_active":"2022-04-03T19:44:20.789371+0000","last_became_peered":"2022-04-03T19:44:20.789371+0000","last_unstale":"2022-04-03T19:44:20.796291+0000","last_undegraded":"2022-04-03T19:44:20.796291+0000","last_fullsized":"2022-04-03T19:44:20.796291+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:15:52.060059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.298477+0000","last_change":"2022-04-03T19:44:21.298477+0000","last_active":"2022-04-03T19:44:21.298477+0000","last_peered":"2022-04-03T19:44:21.298477+0000","last_clean":"2022-04-03T19:44:21.298477+0000","last_became_active":"2022-04-03T19:44:21.298165+0000","last_became_peered":"2022-04-03T19:44:21.298165+0000","last_unstale":"2022-04-03T19:44:21.298477+0000","last_undegraded":"2022-04-03T19:44:21.298477+0000","last_fullsized":"2022-04-03T19:44:21.298477+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:23:11.042589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.793792+0000","last_change":"2022-04-03T19:44:20.787949+0000","last_active":"2022-04-03T19:44:20.793792+0000","last_peered":"2022-04-03T19:44:20.793792+0000","last_clean":"2022-04-03T19:44:20.793792+0000","last_became_active":"2022-04-03T19:44:20.787592+0000","last_became_peered":"2022-04-03T19:44:20.787592+0000","last_unstale":"2022-04-03T19:44:20.793792+0000","last_undegraded":"2022-04-03T19:44:20.793792+0000","last_fullsized":"2022-04-03T19:44:20.793792+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:50:09.411246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.301200+0000","last_change":"2022-04-03T19:44:21.301200+0000","last_active":"2022-04-03T19:44:21.301200+0000","last_peered":"2022-04-03T19:44:21.301200+0000","last_clean":"2022-04-03T19:44:21.301200+0000","last_became_active":"2022-04-03T19:44:21.300878+0000","last_became_peered":"2022-04-03T19:44:21.300878+0000","last_unstale":"2022-04-03T19:44:21.301200+0000","last_undegraded":"2022-04-03T19:44:21.301200+0000","last_fullsized":"2022-04-03T19:44:21.301200+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:54:36.089411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.784650+0000","last_change":"2022-04-03T19:44:20.783711+0000","last_active":"2022-04-03T19:44:20.784650+0000","last_peered":"2022-04-03T19:44:20.784650+0000","last_clean":"2022-04-03T19:44:20.784650+0000","last_became_active":"2022-04-03T19:44:20.783423+0000","last_became_peered":"2022-04-03T19:44:20.783423+0000","last_unstale":"2022-04-03T19:44:20.784650+0000","last_undegraded":"2022-04-03T19:44:20.784650+0000","last_fullsized":"2022-04-03T19:44:20.784650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:22:37.665229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300815+0000","last_change":"2022-04-03T19:44:21.300815+0000","last_active":"2022-04-03T19:44:21.300815+0000","last_peered":"2022-04-03T19:44:21.300815+0000","last_clean":"2022-04-03T19:44:21.300815+0000","last_became_active":"2022-04-03T19:44:21.300545+0000","last_became_peered":"2022-04-03T19:44:21.300545+0000","last_unstale":"2022-04-03T19:44:21.300815+0000","last_undegraded":"2022-04-03T19:44:21.300815+0000","last_fullsized":"2022-04-03T19:44:21.300815+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:33:13.941191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.788584+0000","last_change":"2022-04-03T19:44:20.785480+0000","last_active":"2022-04-03T19:44:20.788584+0000","last_peered":"2022-04-03T19:44:20.788584+0000","last_clean":"2022-04-03T19:44:20.788584+0000","last_became_active":"2022-04-03T19:44:20.785178+0000","last_became_peered":"2022-04-03T19:44:20.785178+0000","last_unstale":"2022-04-03T19:44:20.788584+0000","last_undegraded":"2022-04-03T19:44:20.788584+0000","last_fullsized":"2022-04-03T19:44:20.788584+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:15:37.181452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.296509+0000","last_change":"2022-04-03T19:44:21.296509+0000","last_active":"2022-04-03T19:44:21.296509+0000","last_peered":"2022-04-03T19:44:21.296509+0000","last_clean":"2022-04-03T19:44:21.296509+0000","last_became_active":"2022-04-03T19:44:21.296240+0000","last_became_peered":"2022-04-03T19:44:21.296240+0000","last_unstale":"2022-04-03T19:44:21.296509+0000","last_undegraded":"2022-04-03T19:44:21.296509+0000","last_fullsized":"2022-04-03T19:44:21.296509+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:50:45.955195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.793983+0000","last_change":"2022-04-03T19:44:20.792550+0000","last_active":"2022-04-03T19:44:20.793983+0000","last_peered":"2022-04-03T19:44:20.793983+0000","last_clean":"2022-04-03T19:44:20.793983+0000","last_became_active":"2022-04-03T19:44:20.792191+0000","last_became_peered":"2022-04-03T19:44:20.792191+0000","last_unstale":"2022-04-03T19:44:20.793983+0000","last_undegraded":"2022-04-03T19:44:20.793983+0000","last_fullsized":"2022-04-03T19:44:20.793983+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:51:30.297728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.294898+0000","last_change":"2022-04-03T19:44:21.294898+0000","last_active":"2022-04-03T19:44:21.294898+0000","last_peered":"2022-04-03T19:44:21.294898+0000","last_clean":"2022-04-03T19:44:21.294898+0000","last_became_active":"2022-04-03T19:44:21.294635+0000","last_became_peered":"2022-04-03T19:44:21.294635+0000","last_unstale":"2022-04-03T19:44:21.294898+0000","last_undegraded":"2022-04-03T19:44:21.294898+0000","last_fullsized":"2022-04-03T19:44:21.294898+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:42:52.629552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.785610+0000","last_change":"2022-04-03T19:44:20.783711+0000","last_active":"2022-04-03T19:44:20.785610+0000","last_peered":"2022-04-03T19:44:20.785610+0000","last_clean":"2022-04-03T19:44:20.785610+0000","last_became_active":"2022-04-03T19:44:20.783426+0000","last_became_peered":"2022-04-03T19:44:20.783426+0000","last_unstale":"2022-04-03T19:44:20.785610+0000","last_undegraded":"2022-04-03T19:44:20.785610+0000","last_fullsized":"2022-04-03T19:44:20.785610+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:25:26.219558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.296081+0000","last_change":"2022-04-03T19:44:21.296081+0000","last_active":"2022-04-03T19:44:21.296081+0000","last_peered":"2022-04-03T19:44:21.296081+0000","last_clean":"2022-04-03T19:44:21.296081+0000","last_became_active":"2022-04-03T19:44:21.295784+0000","last_became_peered":"2022-04-03T19:44:21.295784+0000","last_unstale":"2022-04-03T19:44:21.296081+0000","last_undegraded":"2022-04-03T19:44:21.296081+0000","last_fullsized":"2022-04-03T19:44:21.296081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:23:40.568168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.791873+0000","last_change":"2022-04-03T19:44:20.789868+0000","last_active":"2022-04-03T19:44:20.791873+0000","last_peered":"2022-04-03T19:44:20.791873+0000","last_clean":"2022-04-03T19:44:20.791873+0000","last_became_active":"2022-04-03T19:44:20.789563+0000","last_became_peered":"2022-04-03T19:44:20.789563+0000","last_unstale":"2022-04-03T19:44:20.791873+0000","last_undegraded":"2022-04-03T19:44:20.791873+0000","last_fullsized":"2022-04-03T19:44:20.791873+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:04:22.814932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.291212+0000","last_change":"2022-04-03T19:44:21.291212+0000","last_active":"2022-04-03T19:44:21.291212+0000","last_peered":"2022-04-03T19:44:21.291212+0000","last_clean":"2022-04-03T19:44:21.291212+0000","last_became_active":"2022-04-03T19:44:21.290888+0000","last_became_peered":"2022-04-03T19:44:21.290888+0000","last_unstale":"2022-04-03T19:44:21.291212+0000","last_undegraded":"2022-04-03T19:44:21.291212+0000","last_fullsized":"2022-04-03T19:44:21.291212+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:29:34.850679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.793246+0000","last_change":"2022-04-03T19:44:20.789319+0000","last_active":"2022-04-03T19:44:20.793246+0000","last_peered":"2022-04-03T19:44:20.793246+0000","last_clean":"2022-04-03T19:44:20.793246+0000","last_became_active":"2022-04-03T19:44:20.788840+0000","last_became_peered":"2022-04-03T19:44:20.788840+0000","last_unstale":"2022-04-03T19:44:20.793246+0000","last_undegraded":"2022-04-03T19:44:20.793246+0000","last_fullsized":"2022-04-03T19:44:20.793246+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:50:21.536926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.311886+0000","last_change":"2022-04-03T19:44:21.311886+0000","last_active":"2022-04-03T19:44:21.311886+0000","last_peered":"2022-04-03T19:44:21.311886+0000","last_clean":"2022-04-03T19:44:21.311886+0000","last_became_active":"2022-04-03T19:44:21.311521+0000","last_became_peered":"2022-04-03T19:44:21.311521+0000","last_unstale":"2022-04-03T19:44:21.311886+0000","last_undegraded":"2022-04-03T19:44:21.311886+0000","last_fullsized":"2022-04-03T19:44:21.311886+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:25:47.761166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.780927+0000","last_change":"2022-04-03T19:44:14.754674+0000","last_active":"2022-04-03T19:44:20.780927+0000","last_peered":"2022-04-03T19:44:20.780927+0000","last_clean":"2022-04-03T19:44:20.780927+0000","last_became_active":"2022-04-03T19:44:14.754210+0000","last_became_peered":"2022-04-03T19:44:14.754210+0000","last_unstale":"2022-04-03T19:44:20.780927+0000","last_undegraded":"2022-04-03T19:44:20.780927+0000","last_fullsized":"2022-04-03T19:44:20.780927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:23:14.624494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.788936+0000","last_change":"2022-04-03T19:44:20.785400+0000","last_active":"2022-04-03T19:44:20.788936+0000","last_peered":"2022-04-03T19:44:20.788936+0000","last_clean":"2022-04-03T19:44:20.788936+0000","last_became_active":"2022-04-03T19:44:20.785125+0000","last_became_peered":"2022-04-03T19:44:20.785125+0000","last_unstale":"2022-04-03T19:44:20.788936+0000","last_undegraded":"2022-04-03T19:44:20.788936+0000","last_fullsized":"2022-04-03T19:44:20.788936+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:24:02.690646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305321+0000","last_change":"2022-04-03T19:44:21.305321+0000","last_active":"2022-04-03T19:44:21.305321+0000","last_peered":"2022-04-03T19:44:21.305321+0000","last_clean":"2022-04-03T19:44:21.305321+0000","last_became_active":"2022-04-03T19:44:21.304017+0000","last_became_peered":"2022-04-03T19:44:21.304017+0000","last_unstale":"2022-04-03T19:44:21.305321+0000","last_undegraded":"2022-04-03T19:44:21.305321+0000","last_fullsized":"2022-04-03T19:44:21.305321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:59:12.619058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.779686+0000","last_change":"2022-04-03T19:44:14.755580+0000","last_active":"2022-04-03T19:44:20.779686+0000","last_peered":"2022-04-03T19:44:20.779686+0000","last_clean":"2022-04-03T19:44:20.779686+0000","last_became_active":"2022-04-03T19:44:14.755222+0000","last_became_peered":"2022-04-03T19:44:14.755222+0000","last_unstale":"2022-04-03T19:44:20.779686+0000","last_undegraded":"2022-04-03T19:44:20.779686+0000","last_fullsized":"2022-04-03T19:44:20.779686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:08:18.314326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.796640+0000","last_change":"2022-04-03T19:44:20.791540+0000","last_active":"2022-04-03T19:44:20.796640+0000","last_peered":"2022-04-03T19:44:20.796640+0000","last_clean":"2022-04-03T19:44:20.796640+0000","last_became_active":"2022-04-03T19:44:20.791192+0000","last_became_peered":"2022-04-03T19:44:20.791192+0000","last_unstale":"2022-04-03T19:44:20.796640+0000","last_undegraded":"2022-04-03T19:44:20.796640+0000","last_fullsized":"2022-04-03T19:44:20.796640+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:58:59.674606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305930+0000","last_change":"2022-04-03T19:44:21.305930+0000","last_active":"2022-04-03T19:44:21.305930+0000","last_peered":"2022-04-03T19:44:21.305930+0000","last_clean":"2022-04-03T19:44:21.305930+0000","last_became_active":"2022-04-03T19:44:21.305659+0000","last_became_peered":"2022-04-03T19:44:21.305659+0000","last_unstale":"2022-04-03T19:44:21.305930+0000","last_undegraded":"2022-04-03T19:44:21.305930+0000","last_fullsized":"2022-04-03T19:44:21.305930+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:14:34.958569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.773041+0000","last_change":"2022-04-03T19:44:14.754598+0000","last_active":"2022-04-03T19:44:20.773041+0000","last_peered":"2022-04-03T19:44:20.773041+0000","last_clean":"2022-04-03T19:44:20.773041+0000","last_became_active":"2022-04-03T19:44:14.754287+0000","last_became_peered":"2022-04-03T19:44:14.754287+0000","last_unstale":"2022-04-03T19:44:20.773041+0000","last_undegraded":"2022-04-03T19:44:20.773041+0000","last_fullsized":"2022-04-03T19:44:20.773041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:32:16.232689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.798752+0000","last_change":"2022-04-03T19:44:20.789567+0000","last_active":"2022-04-03T19:44:20.798752+0000","last_peered":"2022-04-03T19:44:20.798752+0000","last_clean":"2022-04-03T19:44:20.798752+0000","last_became_active":"2022-04-03T19:44:20.788664+0000","last_became_peered":"2022-04-03T19:44:20.788664+0000","last_unstale":"2022-04-03T19:44:20.798752+0000","last_undegraded":"2022-04-03T19:44:20.798752+0000","last_fullsized":"2022-04-03T19:44:20.798752+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:15:37.098494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303079+0000","last_change":"2022-04-03T19:44:21.303079+0000","last_active":"2022-04-03T19:44:21.303079+0000","last_peered":"2022-04-03T19:44:21.303079+0000","last_clean":"2022-04-03T19:44:21.303079+0000","last_became_active":"2022-04-03T19:44:21.302014+0000","last_became_peered":"2022-04-03T19:44:21.302014+0000","last_unstale":"2022-04-03T19:44:21.303079+0000","last_undegraded":"2022-04-03T19:44:21.303079+0000","last_fullsized":"2022-04-03T19:44:21.303079+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:15:58.860701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.779613+0000","last_change":"2022-04-03T19:44:14.754385+0000","last_active":"2022-04-03T19:44:20.779613+0000","last_peered":"2022-04-03T19:44:20.779613+0000","last_clean":"2022-04-03T19:44:20.779613+0000","last_became_active":"2022-04-03T19:44:14.754003+0000","last_became_peered":"2022-04-03T19:44:14.754003+0000","last_unstale":"2022-04-03T19:44:20.779613+0000","last_undegraded":"2022-04-03T19:44:20.779613+0000","last_fullsized":"2022-04-03T19:44:20.779613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:05:22.716933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.778008+0000","last_change":"2022-04-03T19:44:08.726758+0000","last_active":"2022-04-03T19:44:20.778008+0000","last_peered":"2022-04-03T19:44:20.778008+0000","last_clean":"2022-04-03T19:44:20.778008+0000","last_became_active":"2022-04-03T19:44:08.725694+0000","last_became_peered":"2022-04-03T19:44:08.725694+0000","last_unstale":"2022-04-03T19:44:20.778008+0000","last_undegraded":"2022-04-03T19:44:20.778008+0000","last_fullsized":"2022-04-03T19:44:20.778008+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:07.708129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:07.708129+0000","last_clean_scrub_stamp":"2022-04-03T19:44:07.708129+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:27:43.354267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.786014+0000","last_change":"2022-04-03T19:44:20.781585+0000","last_active":"2022-04-03T19:44:20.786014+0000","last_peered":"2022-04-03T19:44:20.786014+0000","last_clean":"2022-04-03T19:44:20.786014+0000","last_became_active":"2022-04-03T19:44:20.781274+0000","last_became_peered":"2022-04-03T19:44:20.781274+0000","last_unstale":"2022-04-03T19:44:20.786014+0000","last_undegraded":"2022-04-03T19:44:20.786014+0000","last_fullsized":"2022-04-03T19:44:20.786014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:49:54.368001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.296620+0000","last_change":"2022-04-03T19:44:21.296620+0000","last_active":"2022-04-03T19:44:21.296620+0000","last_peered":"2022-04-03T19:44:21.296620+0000","last_clean":"2022-04-03T19:44:21.296620+0000","last_became_active":"2022-04-03T19:44:21.296314+0000","last_became_peered":"2022-04-03T19:44:21.296314+0000","last_unstale":"2022-04-03T19:44:21.296620+0000","last_undegraded":"2022-04-03T19:44:21.296620+0000","last_fullsized":"2022-04-03T19:44:21.296620+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:09:18.338683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.778178+0000","last_change":"2022-04-03T19:44:14.754664+0000","last_active":"2022-04-03T19:44:20.778178+0000","last_peered":"2022-04-03T19:44:20.778178+0000","last_clean":"2022-04-03T19:44:20.778178+0000","last_became_active":"2022-04-03T19:44:14.754328+0000","last_became_peered":"2022-04-03T19:44:14.754328+0000","last_unstale":"2022-04-03T19:44:20.778178+0000","last_undegraded":"2022-04-03T19:44:20.778178+0000","last_fullsized":"2022-04-03T19:44:20.778178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:39:58.022353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.793867+0000","last_change":"2022-04-03T19:44:20.787271+0000","last_active":"2022-04-03T19:44:20.793867+0000","last_peered":"2022-04-03T19:44:20.793867+0000","last_clean":"2022-04-03T19:44:20.793867+0000","last_became_active":"2022-04-03T19:44:20.786957+0000","last_became_peered":"2022-04-03T19:44:20.786957+0000","last_unstale":"2022-04-03T19:44:20.793867+0000","last_undegraded":"2022-04-03T19:44:20.793867+0000","last_fullsized":"2022-04-03T19:44:20.793867+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:21:45.992744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.306857+0000","last_change":"2022-04-03T19:44:21.306857+0000","last_active":"2022-04-03T19:44:21.306857+0000","last_peered":"2022-04-03T19:44:21.306857+0000","last_clean":"2022-04-03T19:44:21.306857+0000","last_became_active":"2022-04-03T19:44:21.306521+0000","last_became_peered":"2022-04-03T19:44:21.306521+0000","last_unstale":"2022-04-03T19:44:21.306857+0000","last_undegraded":"2022-04-03T19:44:21.306857+0000","last_fullsized":"2022-04-03T19:44:21.306857+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:50:03.912834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.779281+0000","last_change":"2022-04-03T19:44:14.754657+0000","last_active":"2022-04-03T19:44:20.779281+0000","last_peered":"2022-04-03T19:44:20.779281+0000","last_clean":"2022-04-03T19:44:20.779281+0000","last_became_active":"2022-04-03T19:44:14.754310+0000","last_became_peered":"2022-04-03T19:44:14.754310+0000","last_unstale":"2022-04-03T19:44:20.779281+0000","last_undegraded":"2022-04-03T19:44:20.779281+0000","last_fullsized":"2022-04-03T19:44:20.779281+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:42:05.279424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.798841+0000","last_change":"2022-04-03T19:44:20.794454+0000","last_active":"2022-04-03T19:44:20.798841+0000","last_peered":"2022-04-03T19:44:20.798841+0000","last_clean":"2022-04-03T19:44:20.798841+0000","last_became_active":"2022-04-03T19:44:20.789468+0000","last_became_peered":"2022-04-03T19:44:20.789468+0000","last_unstale":"2022-04-03T19:44:20.798841+0000","last_undegraded":"2022-04-03T19:44:20.798841+0000","last_fullsized":"2022-04-03T19:44:20.798841+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:02:49.184838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303669+0000","last_change":"2022-04-03T19:44:21.303669+0000","last_active":"2022-04-03T19:44:21.303669+0000","last_peered":"2022-04-03T19:44:21.303669+0000","last_clean":"2022-04-03T19:44:21.303669+0000","last_became_active":"2022-04-03T19:44:21.303381+0000","last_became_peered":"2022-04-03T19:44:21.303381+0000","last_unstale":"2022-04-03T19:44:21.303669+0000","last_undegraded":"2022-04-03T19:44:21.303669+0000","last_fullsized":"2022-04-03T19:44:21.303669+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:43:22.320070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.794015+0000","last_change":"2022-04-03T19:44:20.791594+0000","last_active":"2022-04-03T19:44:20.794015+0000","last_peered":"2022-04-03T19:44:20.794015+0000","last_clean":"2022-04-03T19:44:20.794015+0000","last_became_active":"2022-04-03T19:44:20.789693+0000","last_became_peered":"2022-04-03T19:44:20.789693+0000","last_unstale":"2022-04-03T19:44:20.794015+0000","last_undegraded":"2022-04-03T19:44:20.794015+0000","last_fullsized":"2022-04-03T19:44:20.794015+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:18:21.832066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.780669+0000","last_change":"2022-04-03T19:44:14.755096+0000","last_active":"2022-04-03T19:44:20.780669+0000","last_peered":"2022-04-03T19:44:20.780669+0000","last_clean":"2022-04-03T19:44:20.780669+0000","last_became_active":"2022-04-03T19:44:14.754799+0000","last_became_peered":"2022-04-03T19:44:14.754799+0000","last_unstale":"2022-04-03T19:44:20.780669+0000","last_undegraded":"2022-04-03T19:44:20.780669+0000","last_fullsized":"2022-04-03T19:44:20.780669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:04:17.179114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299649+0000","last_change":"2022-04-03T19:44:21.299649+0000","last_active":"2022-04-03T19:44:21.299649+0000","last_peered":"2022-04-03T19:44:21.299649+0000","last_clean":"2022-04-03T19:44:21.299649+0000","last_became_active":"2022-04-03T19:44:21.299399+0000","last_became_peered":"2022-04-03T19:44:21.299399+0000","last_unstale":"2022-04-03T19:44:21.299649+0000","last_undegraded":"2022-04-03T19:44:21.299649+0000","last_fullsized":"2022-04-03T19:44:21.299649+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:13:06.313729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.772869+0000","last_change":"2022-04-03T19:44:14.754659+0000","last_active":"2022-04-03T19:44:20.772869+0000","last_peered":"2022-04-03T19:44:20.772869+0000","last_clean":"2022-04-03T19:44:20.772869+0000","last_became_active":"2022-04-03T19:44:14.754386+0000","last_became_peered":"2022-04-03T19:44:14.754386+0000","last_unstale":"2022-04-03T19:44:20.772869+0000","last_undegraded":"2022-04-03T19:44:20.772869+0000","last_fullsized":"2022-04-03T19:44:20.772869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:33:33.013536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.791124+0000","last_change":"2022-04-03T19:44:20.784840+0000","last_active":"2022-04-03T19:44:20.791124+0000","last_peered":"2022-04-03T19:44:20.791124+0000","last_clean":"2022-04-03T19:44:20.791124+0000","last_became_active":"2022-04-03T19:44:20.784521+0000","last_became_peered":"2022-04-03T19:44:20.784521+0000","last_unstale":"2022-04-03T19:44:20.791124+0000","last_undegraded":"2022-04-03T19:44:20.791124+0000","last_fullsized":"2022-04-03T19:44:20.791124+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:15:38.240804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.290769+0000","last_change":"2022-04-03T19:44:21.290769+0000","last_active":"2022-04-03T19:44:21.290769+0000","last_peered":"2022-04-03T19:44:21.290769+0000","last_clean":"2022-04-03T19:44:21.290769+0000","last_became_active":"2022-04-03T19:44:21.290394+0000","last_became_peered":"2022-04-03T19:44:21.290394+0000","last_unstale":"2022-04-03T19:44:21.290769+0000","last_undegraded":"2022-04-03T19:44:21.290769+0000","last_fullsized":"2022-04-03T19:44:21.290769+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:57:39.528871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.794710+0000","last_change":"2022-04-03T19:44:20.789662+0000","last_active":"2022-04-03T19:44:20.794710+0000","last_peered":"2022-04-03T19:44:20.794710+0000","last_clean":"2022-04-03T19:44:20.794710+0000","last_became_active":"2022-04-03T19:44:20.787868+0000","last_became_peered":"2022-04-03T19:44:20.787868+0000","last_unstale":"2022-04-03T19:44:20.794710+0000","last_undegraded":"2022-04-03T19:44:20.794710+0000","last_fullsized":"2022-04-03T19:44:20.794710+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:13:16.092180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.304539+0000","last_change":"2022-04-03T19:44:21.304539+0000","last_active":"2022-04-03T19:44:21.304539+0000","last_peered":"2022-04-03T19:44:21.304539+0000","last_clean":"2022-04-03T19:44:21.304539+0000","last_became_active":"2022-04-03T19:44:21.304008+0000","last_became_peered":"2022-04-03T19:44:21.304008+0000","last_unstale":"2022-04-03T19:44:21.304539+0000","last_undegraded":"2022-04-03T19:44:21.304539+0000","last_fullsized":"2022-04-03T19:44:21.304539+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:45:01.346479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.790383+0000","last_change":"2022-04-03T19:44:20.786813+0000","last_active":"2022-04-03T19:44:20.790383+0000","last_peered":"2022-04-03T19:44:20.790383+0000","last_clean":"2022-04-03T19:44:20.790383+0000","last_became_active":"2022-04-03T19:44:20.786548+0000","last_became_peered":"2022-04-03T19:44:20.786548+0000","last_unstale":"2022-04-03T19:44:20.790383+0000","last_undegraded":"2022-04-03T19:44:20.790383+0000","last_fullsized":"2022-04-03T19:44:20.790383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:36:40.611720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299513+0000","last_change":"2022-04-03T19:44:21.299513+0000","last_active":"2022-04-03T19:44:21.299513+0000","last_peered":"2022-04-03T19:44:21.299513+0000","last_clean":"2022-04-03T19:44:21.299513+0000","last_became_active":"2022-04-03T19:44:21.299270+0000","last_became_peered":"2022-04-03T19:44:21.299270+0000","last_unstale":"2022-04-03T19:44:21.299513+0000","last_undegraded":"2022-04-03T19:44:21.299513+0000","last_fullsized":"2022-04-03T19:44:21.299513+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:54:16.302815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.790374+0000","last_change":"2022-04-03T19:44:20.786750+0000","last_active":"2022-04-03T19:44:20.790374+0000","last_peered":"2022-04-03T19:44:20.790374+0000","last_clean":"2022-04-03T19:44:20.790374+0000","last_became_active":"2022-04-03T19:44:20.786490+0000","last_became_peered":"2022-04-03T19:44:20.786490+0000","last_unstale":"2022-04-03T19:44:20.790374+0000","last_undegraded":"2022-04-03T19:44:20.790374+0000","last_fullsized":"2022-04-03T19:44:20.790374+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:30:31.980806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.294849+0000","last_change":"2022-04-03T19:44:21.294849+0000","last_active":"2022-04-03T19:44:21.294849+0000","last_peered":"2022-04-03T19:44:21.294849+0000","last_clean":"2022-04-03T19:44:21.294849+0000","last_became_active":"2022-04-03T19:44:21.294586+0000","last_became_peered":"2022-04-03T19:44:21.294586+0000","last_unstale":"2022-04-03T19:44:21.294849+0000","last_undegraded":"2022-04-03T19:44:21.294849+0000","last_fullsized":"2022-04-03T19:44:21.294849+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:24:11.050193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.786040+0000","last_change":"2022-04-03T19:44:20.781619+0000","last_active":"2022-04-03T19:44:20.786040+0000","last_peered":"2022-04-03T19:44:20.786040+0000","last_clean":"2022-04-03T19:44:20.786040+0000","last_became_active":"2022-04-03T19:44:20.781288+0000","last_became_peered":"2022-04-03T19:44:20.781288+0000","last_unstale":"2022-04-03T19:44:20.786040+0000","last_undegraded":"2022-04-03T19:44:20.786040+0000","last_fullsized":"2022-04-03T19:44:20.786040+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:05:56.179441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.296332+0000","last_change":"2022-04-03T19:44:21.296332+0000","last_active":"2022-04-03T19:44:21.296332+0000","last_peered":"2022-04-03T19:44:21.296332+0000","last_clean":"2022-04-03T19:44:21.296332+0000","last_became_active":"2022-04-03T19:44:21.296026+0000","last_became_peered":"2022-04-03T19:44:21.296026+0000","last_unstale":"2022-04-03T19:44:21.296332+0000","last_undegraded":"2022-04-03T19:44:21.296332+0000","last_fullsized":"2022-04-03T19:44:21.296332+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:43:33.822187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-04-03T19:44:20.794613+0000","last_change":"2022-04-03T19:44:20.789448+0000","last_active":"2022-04-03T19:44:20.794613+0000","last_peered":"2022-04-03T19:44:20.794613+0000","last_clean":"2022-04-03T19:44:20.794613+0000","last_became_active":"2022-04-03T19:44:20.789131+0000","last_became_peered":"2022-04-03T19:44:20.789131+0000","last_unstale":"2022-04-03T19:44:20.794613+0000","last_undegraded":"2022-04-03T19:44:20.794613+0000","last_fullsized":"2022-04-03T19:44:20.794613+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:46:20.830676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.291681+0000","last_change":"2022-04-03T19:44:21.291681+0000","last_active":"2022-04-03T19:44:21.291681+0000","last_peered":"2022-04-03T19:44:21.291681+0000","last_clean":"2022-04-03T19:44:21.291681+0000","last_became_active":"2022-04-03T19:44:21.290817+0000","last_became_peered":"2022-04-03T19:44:21.290817+0000","last_unstale":"2022-04-03T19:44:21.291681+0000","last_undegraded":"2022-04-03T19:44:21.291681+0000","last_fullsized":"2022-04-03T19:44:21.291681+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:37:30.833818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.295604+0000","last_change":"2022-04-03T19:44:21.295604+0000","last_active":"2022-04-03T19:44:21.295604+0000","last_peered":"2022-04-03T19:44:21.295604+0000","last_clean":"2022-04-03T19:44:21.295604+0000","last_became_active":"2022-04-03T19:44:21.295297+0000","last_became_peered":"2022-04-03T19:44:21.295297+0000","last_unstale":"2022-04-03T19:44:21.295604+0000","last_undegraded":"2022-04-03T19:44:21.295604+0000","last_fullsized":"2022-04-03T19:44:21.295604+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:26:34.341290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.306547+0000","last_change":"2022-04-03T19:44:21.306547+0000","last_active":"2022-04-03T19:44:21.306547+0000","last_peered":"2022-04-03T19:44:21.306547+0000","last_clean":"2022-04-03T19:44:21.306547+0000","last_became_active":"2022-04-03T19:44:21.306179+0000","last_became_peered":"2022-04-03T19:44:21.306179+0000","last_unstale":"2022-04-03T19:44:21.306547+0000","last_undegraded":"2022-04-03T19:44:21.306547+0000","last_fullsized":"2022-04-03T19:44:21.306547+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:34:58.854671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.291341+0000","last_change":"2022-04-03T19:44:21.291341+0000","last_active":"2022-04-03T19:44:21.291341+0000","last_peered":"2022-04-03T19:44:21.291341+0000","last_clean":"2022-04-03T19:44:21.291341+0000","last_became_active":"2022-04-03T19:44:21.290943+0000","last_became_peered":"2022-04-03T19:44:21.290943+0000","last_unstale":"2022-04-03T19:44:21.291341+0000","last_undegraded":"2022-04-03T19:44:21.291341+0000","last_fullsized":"2022-04-03T19:44:21.291341+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:50:56.877454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.307815+0000","last_change":"2022-04-03T19:44:21.307815+0000","last_active":"2022-04-03T19:44:21.307815+0000","last_peered":"2022-04-03T19:44:21.307815+0000","last_clean":"2022-04-03T19:44:21.307815+0000","last_became_active":"2022-04-03T19:44:21.307459+0000","last_became_peered":"2022-04-03T19:44:21.307459+0000","last_unstale":"2022-04-03T19:44:21.307815+0000","last_undegraded":"2022-04-03T19:44:21.307815+0000","last_fullsized":"2022-04-03T19:44:21.307815+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:16:52.723409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":8,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303095+0000","last_change":"2022-04-03T19:44:21.303095+0000","last_active":"2022-04-03T19:44:21.303095+0000","last_peered":"2022-04-03T19:44:21.303095+0000","last_clean":"2022-04-03T19:44:21.303095+0000","last_became_active":"2022-04-03T19:44:21.302363+0000","last_became_peered":"2022-04-03T19:44:21.302363+0000","last_unstale":"2022-04-03T19:44:21.303095+0000","last_undegraded":"2022-04-03T19:44:21.303095+0000","last_fullsized":"2022-04-03T19:44:21.303095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:42:45.617661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8880,"kb_used_data":744,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94362960,"statfs":{"total":96636764160,"available":96627671040,"internally_reserved":0,"allocated":761856,"data_stored":609145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0589999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]}]},{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8880,"kb_used_data":744,"kb_used_omap":0,"kb_used_meta":8128,"kb_avail":94362960,"statfs":{"total":96636764160,"available":96627671040,"internally_reserved":0,"allocated":761856,"data_stored":609145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8323072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7860,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":7616,"kb_avail":94363980,"statfs":{"total":96636764160,"available":96628715520,"internally_reserved":0,"allocated":241664,"data_stored":87333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7798784},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7772,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":94364068,"statfs":{"total":96636764160,"available":96628805632,"internally_reserved":0,"allocated":217088,"data_stored":73609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7772,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":94364068,"statfs":{"total":96636764160,"available":96628805632,"internally_reserved":0,"allocated":217088,"data_stored":73609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7848,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":7616,"kb_avail":94363992,"statfs":{"total":96636764160,"available":96628727808,"internally_reserved":0,"allocated":229376,"data_stored":80217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7798784},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7848,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":7616,"kb_avail":94363992,"statfs":{"total":96636764160,"available":96628727808,"internally_reserved":0,"allocated":229376,"data_stored":80217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7798784},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7848,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":7616,"kb_avail":94363992,"statfs":{"total":96636764160,"available":96628727808,"internally_reserved":0,"allocated":229376,"data_stored":80217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7798784},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-03T19:44:27.156 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2022-04-03T19:44:27.157 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-04-03T19:44:27.157 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2022-04-03T19:44:27.158 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2022-04-03T19:44:27.158 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-04-03T19:44:27.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.359+0000 7f6fd3fff700 1 Processor -- start 2022-04-03T19:44:27.360 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.359+0000 7f6fd3fff700 1 -- start start 2022-04-03T19:44:27.362 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.360+0000 7f6fd3fff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6fd406bfb0 0x7f6fd41584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:27.362 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.360+0000 7f6fd3fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6fd406c500 0x7f6fd414eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:27.362 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.360+0000 7f6fd3fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd406d260 con 0x7f6fd406bbe0 2022-04-03T19:44:27.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.360+0000 7f6fd3fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd4159450 con 0x7f6fd406c500 2022-04-03T19:44:27.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.360+0000 7f6fd3fff700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd40c6330 con 0x7f6fd406bfb0 2022-04-03T19:44:27.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.360+0000 7f6fd2ffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6fd406bbe0 0x7f6fd406d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38298/0 (socket says 172.21.15.77:38298) 2022-04-03T19:44:27.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.360+0000 7f6fd37fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6fd406c500 0x7f6fd414eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:27.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.360+0000 7f6fd2ffd700 1 -- 172.21.15.77:0/1638074650 learned_addr learned my addr 172.21.15.77:0/1638074650 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:27.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.360+0000 7f6fd27fc700 1 --2- 172.21.15.77:0/1638074650 >> v2:172.21.15.186:3300/0 conn(0x7f6fd406bfb0 0x7f6fd41584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:27.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.361+0000 7f6fd1ffb700 1 -- 172.21.15.77:0/1638074650 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002266009 0 0) 0x7f6fd406d260 con 0x7f6fd406bbe0 2022-04-03T19:44:27.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.361+0000 7f6fd1ffb700 1 -- 172.21.15.77:0/1638074650 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fc00033f0 con 0x7f6fd406bbe0 2022-04-03T19:44:27.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.361+0000 7f6fd37fe700 1 -- 172.21.15.77:0/1638074650 >> v2:172.21.15.186:3300/0 conn(0x7f6fd406bfb0 msgr2=0x7f6fd41584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:27.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.361+0000 7f6fd37fe700 1 --2- 172.21.15.77:0/1638074650 >> v2:172.21.15.186:3300/0 conn(0x7f6fd406bfb0 0x7f6fd41584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:27.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.361+0000 7f6fd37fe700 1 -- 172.21.15.77:0/1638074650 >> v1:172.21.15.77:6789/0 conn(0x7f6fd406bbe0 legacy=0x7f6fd406d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:27.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.361+0000 7f6fd37fe700 1 -- 172.21.15.77:0/1638074650 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd4158bb0 con 0x7f6fd406c500 2022-04-03T19:44:27.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.361+0000 7f6fd37fe700 1 --2- 172.21.15.77:0/1638074650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd406c500 0x7f6fd414eea0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6fd40898c0 tx=0x7f6fcc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c08e4a57792f5826 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:27.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.362+0000 7f6fd1ffb700 1 -- 172.21.15.77:0/1638074650 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc00e070 con 0x7f6fd406c500 2022-04-03T19:44:27.366 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.362+0000 7f6fd1ffb700 1 -- 172.21.15.77:0/1638074650 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fcc00bab0 con 0x7f6fd406c500 2022-04-03T19:44:27.366 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.362+0000 7f6fd1ffb700 1 -- 172.21.15.77:0/1638074650 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc00be70 con 0x7f6fd406c500 2022-04-03T19:44:27.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.362+0000 7f6fd3fff700 1 -- 172.21.15.77:0/1638074650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd406c500 msgr2=0x7f6fd414eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:27.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.362+0000 7f6fd3fff700 1 --2- 172.21.15.77:0/1638074650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd406c500 0x7f6fd414eea0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6fd40898c0 tx=0x7f6fcc009780 comp rx=0 tx=0).stop 2022-04-03T19:44:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.362+0000 7f6fd3fff700 1 -- 172.21.15.77:0/1638074650 shutdown_connections 2022-04-03T19:44:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.362+0000 7f6fd3fff700 1 --2- 172.21.15.77:0/1638074650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd406c500 0x7f6fd414eea0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.362+0000 7f6fd3fff700 1 --2- 172.21.15.77:0/1638074650 >> v2:172.21.15.186:3300/0 conn(0x7f6fd406bfb0 0x7f6fd41584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:27.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.362+0000 7f6fd3fff700 1 -- 172.21.15.77:0/1638074650 >> 172.21.15.77:0/1638074650 conn(0x7f6fd40cb6b0 msgr2=0x7f6fd40cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.362+0000 7f6fd3fff700 1 -- 172.21.15.77:0/1638074650 shutdown_connections 2022-04-03T19:44:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.362+0000 7f6fd3fff700 1 -- 172.21.15.77:0/1638074650 wait complete. 2022-04-03T19:44:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.364+0000 7f6fd3fff700 1 Processor -- start 2022-04-03T19:44:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.364+0000 7f6fd3fff700 1 -- start start 2022-04-03T19:44:27.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.365+0000 7f6fd3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd406bbe0 0x7f6fd414a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.365+0000 7f6fd3fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fd406bfb0 0x7f6fd4150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.365+0000 7f6fd3fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fd406c500 0x7f6fd4151400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.365+0000 7f6fd3fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd4155ba0 con 0x7f6fd406bbe0 2022-04-03T19:44:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.365+0000 7f6fd3fff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd4150430 con 0x7f6fd406c500 2022-04-03T19:44:27.372 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.365+0000 7f6fd3fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd4154f70 con 0x7f6fd406bfb0 2022-04-03T19:44:27.373 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.365+0000 7f6fd2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd406bbe0 0x7f6fd414a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:27.373 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.365+0000 7f6fd2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd406bbe0 0x7f6fd414a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42518/0 (socket says 172.21.15.77:42518) 2022-04-03T19:44:27.373 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.365+0000 7f6fd2ffd700 1 -- 172.21.15.77:0/2441072722 learned_addr learned my addr 172.21.15.77:0/2441072722 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:27.373 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.365+0000 7f6fd27fc700 1 --2- 172.21.15.77:0/2441072722 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fd406bfb0 0x7f6fd4150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:27.374 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.365+0000 7f6fd37fe700 1 --2- 172.21.15.77:0/2441072722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fd406c500 0x7f6fd4151400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:27.374 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.366+0000 7f6fd27fc700 1 -- 172.21.15.77:0/2441072722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fd406c500 msgr2=0x7f6fd4151400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:27.374 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.366+0000 7f6fd27fc700 1 --2- 172.21.15.77:0/2441072722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fd406c500 0x7f6fd4151400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:27.374 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.366+0000 7f6fd27fc700 1 -- 172.21.15.77:0/2441072722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd406bbe0 msgr2=0x7f6fd414a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:27.374 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.366+0000 7f6fd27fc700 1 --2- 172.21.15.77:0/2441072722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd406bbe0 0x7f6fd414a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:27.375 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.366+0000 7f6fd27fc700 1 -- 172.21.15.77:0/2441072722 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fcc009a20 con 0x7f6fd406bfb0 2022-04-03T19:44:27.375 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.366+0000 7f6fd37fe700 1 --2- 172.21.15.77:0/2441072722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fd406c500 0x7f6fd4151400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:27.375 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.366+0000 7f6fd27fc700 1 --2- 172.21.15.77:0/2441072722 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fd406bfb0 0x7f6fd4150ec0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6fc800e900 tx=0x7f6fc8009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:27.375 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.366+0000 7f6fbaffd700 1 -- 172.21.15.77:0/2441072722 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8013680 con 0x7f6fd406bfb0 2022-04-03T19:44:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.366+0000 7f6fbaffd700 1 -- 172.21.15.77:0/2441072722 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc80137e0 con 0x7f6fd406bfb0 2022-04-03T19:44:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.366+0000 7f6fd3fff700 1 -- 172.21.15.77:0/2441072722 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd4155250 con 0x7f6fd406bfb0 2022-04-03T19:44:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.366+0000 7f6fbaffd700 1 -- 172.21.15.77:0/2441072722 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8013b80 con 0x7f6fd406bfb0 2022-04-03T19:44:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.367+0000 7f6fd3fff700 1 -- 172.21.15.77:0/2441072722 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd4155830 con 0x7f6fd406bfb0 2022-04-03T19:44:27.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.367+0000 7f6fd3fff700 1 -- 172.21.15.77:0/2441072722 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd4155d50 con 0x7f6fd406bfb0 2022-04-03T19:44:27.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.368+0000 7f6fbaffd700 1 -- 172.21.15.77:0/2441072722 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f6fc8019030 con 0x7f6fd406bfb0 2022-04-03T19:44:27.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.369+0000 7f6fbaffd700 1 --2- 172.21.15.77:0/2441072722 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6fb4063d10 0x7f6fb40661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:27.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.369+0000 7f6fbaffd700 1 -- 172.21.15.77:0/2441072722 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f6fc8080740 con 0x7f6fd406bfb0 2022-04-03T19:44:27.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.370+0000 7f6fd2ffd700 1 --2- 172.21.15.77:0/2441072722 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6fb4063d10 0x7f6fb40661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:27.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.370+0000 7f6fd2ffd700 1 --2- 172.21.15.77:0/2441072722 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6fb4063d10 0x7f6fb40661c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4007fd0 tx=0x7f6fc4007c70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:27.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.374+0000 7f6fbaffd700 1 -- 172.21.15.77:0/2441072722 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f6fd4155d50 con 0x7f6fd406bfb0 2022-04-03T19:44:27.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.614+0000 7f6fd3fff700 1 -- 172.21.15.77:0/2441072722 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f6fd4155d10 con 0x7f6fd406bfb0 2022-04-03T19:44:27.618 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.617+0000 7f6fbaffd700 1 -- 172.21.15.77:0/2441072722 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v6) v1 ==== 74+0+158619 (secure 0 0 0) 0x7f6fc8017090 con 0x7f6fd406bfb0 2022-04-03T19:44:27.620 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:27.622 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.621+0000 7f6fd3fff700 1 -- 172.21.15.77:0/2441072722 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6fb4063d10 msgr2=0x7f6fb40661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:27.622 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.621+0000 7f6fd3fff700 1 --2- 172.21.15.77:0/2441072722 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6fb4063d10 0x7f6fb40661c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4007fd0 tx=0x7f6fc4007c70 comp rx=0 tx=0).stop 2022-04-03T19:44:27.623 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.621+0000 7f6fd3fff700 1 -- 172.21.15.77:0/2441072722 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fd406bfb0 msgr2=0x7f6fd4150ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:27.623 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.621+0000 7f6fd3fff700 1 --2- 172.21.15.77:0/2441072722 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fd406bfb0 0x7f6fd4150ec0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6fc800e900 tx=0x7f6fc8009060 comp rx=0 tx=0).stop 2022-04-03T19:44:27.623 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.621+0000 7f6fd3fff700 1 -- 172.21.15.77:0/2441072722 shutdown_connections 2022-04-03T19:44:27.623 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.621+0000 7f6fd3fff700 1 --2- 172.21.15.77:0/2441072722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fd406c500 0x7f6fd4151400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:27.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.621+0000 7f6fd3fff700 1 --2- 172.21.15.77:0/2441072722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fd406bbe0 0x7f6fd414a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:27.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.621+0000 7f6fd3fff700 1 --2- 172.21.15.77:0/2441072722 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6fb4063d10 0x7f6fb40661c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:27.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.621+0000 7f6fd3fff700 1 --2- 172.21.15.77:0/2441072722 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fd406bfb0 0x7f6fd4150ec0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:27.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.621+0000 7f6fd3fff700 1 -- 172.21.15.77:0/2441072722 >> 172.21.15.77:0/2441072722 conn(0x7f6fd40cb6b0 msgr2=0x7f6fd40c6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:27.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.621+0000 7f6fd3fff700 1 -- 172.21.15.77:0/2441072722 shutdown_connections 2022-04-03T19:44:27.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.621+0000 7f6fd3fff700 1 -- 172.21.15.77:0/2441072722 wait complete. 2022-04-03T19:44:27.646 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":6,"active_gid":4106,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":24948},{"type":"v1","addr":"172.21.15.186:6833","nonce":24948}]},"active_addr":"172.21.15.186:6833/24948","active_change":"2022-04-03T19:43:59.475406+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":4105,"name":"y","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":4144410376}]},{"addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":3968099647}]},{"addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":1218378044}]},{"addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":1937705315}]}]}} 2022-04-03T19:44:27.650 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2022-04-03T19:44:27.650 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2022-04-03T19:44:27.651 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-04-03T19:44:27.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.847+0000 7fb474cad700 1 Processor -- start 2022-04-03T19:44:27.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.847+0000 7fb474cad700 1 -- start start 2022-04-03T19:44:27.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.848+0000 7fb474cad700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb4700c3640 0x7fb470069600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:27.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.848+0000 7fb474cad700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4700c7af0 0x7fb470069b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:27.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.848+0000 7fb474cad700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4700c4b90 con 0x7fb4700c3270 2022-04-03T19:44:27.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.848+0000 7fb474cad700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4700c3d10 con 0x7fb4700c3640 2022-04-03T19:44:27.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.848+0000 7fb474cad700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4700c3f80 con 0x7fb4700c7af0 2022-04-03T19:44:27.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.849+0000 7fb46effd700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb4700c3640 0x7fb470069600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:27.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.849+0000 7fb46ffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4700c7af0 0x7fb470069b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:27.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.849+0000 7fb46f7fe700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fb4700c3270 0x7fb4700c2500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44108/0 (socket says 172.21.15.77:44108) 2022-04-03T19:44:27.851 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.849+0000 7fb46f7fe700 1 -- 172.21.15.77:0/2865003493 learned_addr learned my addr 172.21.15.77:0/2865003493 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:27.853 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.849+0000 7fb46effd700 1 -- 172.21.15.77:0/2865003493 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4700c7af0 msgr2=0x7fb470069b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:27.853 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.849+0000 7fb46effd700 1 --2- 172.21.15.77:0/2865003493 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4700c7af0 0x7fb470069b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:27.853 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.849+0000 7fb46effd700 1 -- 172.21.15.77:0/2865003493 >> v1:172.21.15.186:6789/0 conn(0x7fb4700c3270 legacy=0x7fb4700c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:27.853 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.849+0000 7fb46effd700 1 -- 172.21.15.77:0/2865003493 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4700c4260 con 0x7fb4700c3640 2022-04-03T19:44:27.853 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.849+0000 7fb46e7fc700 1 -- 172.21.15.77:0/2865003493 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465137370 0 0) 0x7fb4700c4b90 con 0x7fb4700c3270 2022-04-03T19:44:27.854 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.849+0000 7fb46effd700 1 --2- 172.21.15.77:0/2865003493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4700c3640 0x7fb470069600 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb46400a800 tx=0x7fb4640080e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1db8c1ba22b45aa server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:27.854 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.850+0000 7fb46e7fc700 1 -- 172.21.15.77:0/2865003493 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb464010480 con 0x7fb4700c3640 2022-04-03T19:44:27.854 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.850+0000 7fb46e7fc700 1 -- 172.21.15.77:0/2865003493 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4640105e0 con 0x7fb4700c3640 2022-04-03T19:44:27.854 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.850+0000 7fb46e7fc700 1 -- 172.21.15.77:0/2865003493 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4640109a0 con 0x7fb4700c3640 2022-04-03T19:44:27.855 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.850+0000 7fb474cad700 1 -- 172.21.15.77:0/2865003493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4700c3640 msgr2=0x7fb470069600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:27.855 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.850+0000 7fb474cad700 1 --2- 172.21.15.77:0/2865003493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4700c3640 0x7fb470069600 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb46400a800 tx=0x7fb4640080e0 comp rx=0 tx=0).stop 2022-04-03T19:44:27.855 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.850+0000 7fb474cad700 1 -- 172.21.15.77:0/2865003493 shutdown_connections 2022-04-03T19:44:27.855 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.850+0000 7fb474cad700 1 --2- 172.21.15.77:0/2865003493 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4700c7af0 0x7fb470069b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:27.855 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.850+0000 7fb474cad700 1 --2- 172.21.15.77:0/2865003493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4700c3640 0x7fb470069600 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:27.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.850+0000 7fb474cad700 1 -- 172.21.15.77:0/2865003493 >> 172.21.15.77:0/2865003493 conn(0x7fb470066da0 msgr2=0x7fb4700671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:27.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.850+0000 7fb474cad700 1 -- 172.21.15.77:0/2865003493 shutdown_connections 2022-04-03T19:44:27.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.850+0000 7fb474cad700 1 -- 172.21.15.77:0/2865003493 wait complete. 2022-04-03T19:44:27.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.852+0000 7fb474cad700 1 Processor -- start 2022-04-03T19:44:27.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.852+0000 7fb474cad700 1 -- start start 2022-04-03T19:44:27.859 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.853+0000 7fb474cad700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4700c3270 0x7fb4701690a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:27.859 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.853+0000 7fb474cad700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4700c3640 0x7fb4700bb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:27.860 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.853+0000 7fb474cad700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4700c7af0 0x7fb470151760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:27.860 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.853+0000 7fb474cad700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4701584a0 con 0x7fb4700c3270 2022-04-03T19:44:27.860 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.853+0000 7fb474cad700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4701695e0 con 0x7fb4700c3640 2022-04-03T19:44:27.860 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.853+0000 7fb474cad700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4700bcd50 con 0x7fb4700c7af0 2022-04-03T19:44:27.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.853+0000 7fb46f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4700c3270 0x7fb4701690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:27.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.853+0000 7fb46ffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4700c7af0 0x7fb470151760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:27.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.853+0000 7fb46f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4700c3270 0x7fb4701690a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42520/0 (socket says 172.21.15.77:42520) 2022-04-03T19:44:27.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.853+0000 7fb46f7fe700 1 -- 172.21.15.77:0/1466911237 learned_addr learned my addr 172.21.15.77:0/1466911237 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:27.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.853+0000 7fb46ffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4700c7af0 0x7fb470151760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59818/0 (socket says 172.21.15.77:59818) 2022-04-03T19:44:27.862 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.853+0000 7fb46effd700 1 --2- 172.21.15.77:0/1466911237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4700c3640 0x7fb4700bb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:27.862 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.854+0000 7fb46f7fe700 1 -- 172.21.15.77:0/1466911237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4700c7af0 msgr2=0x7fb470151760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:27.862 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.854+0000 7fb46f7fe700 1 --2- 172.21.15.77:0/1466911237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4700c7af0 0x7fb470151760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:27.862 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.854+0000 7fb46f7fe700 1 -- 172.21.15.77:0/1466911237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4700c3640 msgr2=0x7fb4700bb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:27.862 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.854+0000 7fb46f7fe700 1 --2- 172.21.15.77:0/1466911237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4700c3640 0x7fb4700bb820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:27.863 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.854+0000 7fb46f7fe700 1 -- 172.21.15.77:0/1466911237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46400a340 con 0x7fb4700c3270 2022-04-03T19:44:27.863 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.854+0000 7fb46f7fe700 1 --2- 172.21.15.77:0/1466911237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4700c3270 0x7fb4701690a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb46000c440 tx=0x7fb460007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:27.863 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.854+0000 7fb4577fe700 1 -- 172.21.15.77:0/1466911237 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb460007360 con 0x7fb4700c3270 2022-04-03T19:44:27.863 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.855+0000 7fb4577fe700 1 -- 172.21.15.77:0/1466911237 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4600074c0 con 0x7fb4700c3270 2022-04-03T19:44:27.864 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.855+0000 7fb474cad700 1 -- 172.21.15.77:0/1466911237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4700bbd60 con 0x7fb4700c3270 2022-04-03T19:44:27.864 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.855+0000 7fb4577fe700 1 -- 172.21.15.77:0/1466911237 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb460004900 con 0x7fb4700c3270 2022-04-03T19:44:27.864 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.855+0000 7fb474cad700 1 -- 172.21.15.77:0/1466911237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4700c1250 con 0x7fb4700c3270 2022-04-03T19:44:27.864 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.856+0000 7fb474cad700 1 -- 172.21.15.77:0/1466911237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4700c4b70 con 0x7fb4700c3270 2022-04-03T19:44:27.864 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.856+0000 7fb4577fe700 1 -- 172.21.15.77:0/1466911237 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fb460010030 con 0x7fb4700c3270 2022-04-03T19:44:27.865 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.857+0000 7fb4577fe700 1 --2- 172.21.15.77:0/1466911237 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb450063ce0 0x7fb450066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:27.865 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.857+0000 7fb4577fe700 1 -- 172.21.15.77:0/1466911237 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fb460080250 con 0x7fb4700c3270 2022-04-03T19:44:27.865 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.857+0000 7fb46effd700 1 --2- 172.21.15.77:0/1466911237 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb450063ce0 0x7fb450066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:27.865 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.858+0000 7fb46effd700 1 --2- 172.21.15.77:0/1466911237 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb450063ce0 0x7fb450066190 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb46400afd0 tx=0x7fb464019000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:27.866 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:27.861+0000 7fb4577fe700 1 -- 172.21.15.77:0/1466911237 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fb46004eeb0 con 0x7fb4700c3270 2022-04-03T19:44:28.057 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.056+0000 7fb474cad700 1 -- 172.21.15.77:0/1466911237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb47004b050 con 0x7fb4700c3270 2022-04-03T19:44:28.058 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.057+0000 7fb4577fe700 1 -- 172.21.15.77:0/1466911237 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16845 (secure 0 0 0) 0x7fb4600524d0 con 0x7fb4700c3270 2022-04-03T19:44:28.058 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:28.058 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"89c3a3f4-599d-4c4a-8f5a-7180eafb7292","created":"2022-04-03T19:43:54.101559+0000","modified":"2022-04-03T19:44:22.278254+0000","last_up_change":"2022-04-03T19:44:05.589174+0000","last_in_change":"2022-04-03T19:43:56.729760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-03T19:44:07.567638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-04-03T19:44:13.149481+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-04-03T19:44:18.823926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-04-03T19:44:20.275479+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f35f5195-3729-4d78-85d1-0539f259283f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6823","nonce":20252},{"type":"v1","addr":"172.21.15.77:6825","nonce":20252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":20252},{"type":"v1","addr":"172.21.15.77:6827","nonce":20252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":20252},{"type":"v1","addr":"172.21.15.77:6831","nonce":20252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":20252},{"type":"v1","addr":"172.21.15.77:6829","nonce":20252}]},"public_addr":"172.21.15.77:6825/20252","cluster_addr":"172.21.15.77:6827/20252","heartbeat_back_addr":"172.21.15.77:6831/20252","heartbeat_front_addr":"172.21.15.77:6829/20252","state":["exists","up"]},{"osd":1,"uuid":"ab1c89dc-57ce-4cc4-b8c7-9b0c19939601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":20249},{"type":"v1","addr":"172.21.15.77:6805","nonce":20249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":20249},{"type":"v1","addr":"172.21.15.77:6809","nonce":20249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":20249},{"type":"v1","addr":"172.21.15.77:6819","nonce":20249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":20249},{"type":"v1","addr":"172.21.15.77:6815","nonce":20249}]},"public_addr":"172.21.15.77:6805/20249","cluster_addr":"172.21.15.77:6809/20249","heartbeat_back_addr":"172.21.15.77:6819/20249","heartbeat_front_addr":"172.21.15.77:6815/20249","state":["exists","up"]},{"osd":2,"uuid":"351e218b-7749-4ee9-b44a-0a5b52fb9be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":20248},{"type":"v1","addr":"172.21.15.77:6801","nonce":20248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":20248},{"type":"v1","addr":"172.21.15.77:6803","nonce":20248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":20248},{"type":"v1","addr":"172.21.15.77:6813","nonce":20248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":20248},{"type":"v1","addr":"172.21.15.77:6808","nonce":20248}]},"public_addr":"172.21.15.77:6801/20248","cluster_addr":"172.21.15.77:6803/20248","heartbeat_back_addr":"172.21.15.77:6813/20248","heartbeat_front_addr":"172.21.15.77:6808/20248","state":["exists","up"]},{"osd":3,"uuid":"1ba6ec8a-7cbf-4c12-8e21-c3097a263ae8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":20250},{"type":"v1","addr":"172.21.15.77:6814","nonce":20250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":20250},{"type":"v1","addr":"172.21.15.77:6818","nonce":20250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":20250},{"type":"v1","addr":"172.21.15.77:6824","nonce":20250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":20250},{"type":"v1","addr":"172.21.15.77:6821","nonce":20250}]},"public_addr":"172.21.15.77:6814/20250","cluster_addr":"172.21.15.77:6818/20250","heartbeat_back_addr":"172.21.15.77:6824/20250","heartbeat_front_addr":"172.21.15.77:6821/20250","state":["exists","up"]},{"osd":4,"uuid":"3faf4c09-846c-4133-856b-7f53c535a67d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":25551},{"type":"v1","addr":"172.21.15.186:6822","nonce":25551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":25551},{"type":"v1","addr":"172.21.15.186:6826","nonce":25551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":25551},{"type":"v1","addr":"172.21.15.186:6831","nonce":25551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":25551},{"type":"v1","addr":"172.21.15.186:6829","nonce":25551}]},"public_addr":"172.21.15.186:6822/25551","cluster_addr":"172.21.15.186:6826/25551","heartbeat_back_addr":"172.21.15.186:6831/25551","heartbeat_front_addr":"172.21.15.186:6829/25551","state":["exists","up"]},{"osd":5,"uuid":"ffe41d00-d0aa-42f4-82c1-5cced13c1bc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":25541},{"type":"v1","addr":"172.21.15.186:6809","nonce":25541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":25541},{"type":"v1","addr":"172.21.15.186:6811","nonce":25541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":25541},{"type":"v1","addr":"172.21.15.186:6815","nonce":25541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":25541},{"type":"v1","addr":"172.21.15.186:6813","nonce":25541}]},"public_addr":"172.21.15.186:6809/25541","cluster_addr":"172.21.15.186:6811/25541","heartbeat_back_addr":"172.21.15.186:6815/25541","heartbeat_front_addr":"172.21.15.186:6813/25541","state":["exists","up"]},{"osd":6,"uuid":"b0344ee8-deca-4e9b-934c-5866d90d0073","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":25542},{"type":"v1","addr":"172.21.15.186:6801","nonce":25542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":25542},{"type":"v1","addr":"172.21.15.186:6803","nonce":25542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":25542},{"type":"v1","addr":"172.21.15.186:6807","nonce":25542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":25542},{"type":"v1","addr":"172.21.15.186:6805","nonce":25542}]},"public_addr":"172.21.15.186:6801/25542","cluster_addr":"172.21.15.186:6803/25542","heartbeat_back_addr":"172.21.15.186:6807/25542","heartbeat_front_addr":"172.21.15.186:6805/25542","state":["exists","up"]},{"osd":7,"uuid":"0eb0e5d3-3dad-4f49-9b50-3b24a3924bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":25572},{"type":"v1","addr":"172.21.15.186:6817","nonce":25572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":25572},{"type":"v1","addr":"172.21.15.186:6819","nonce":25572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6825","nonce":25572},{"type":"v1","addr":"172.21.15.186:6827","nonce":25572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":25572},{"type":"v1","addr":"172.21.15.186:6823","nonce":25572}]},"public_addr":"172.21.15.186:6817/25572","cluster_addr":"172.21.15.186:6819/25572","heartbeat_back_addr":"172.21.15.186:6827/25572","heartbeat_front_addr":"172.21.15.186:6823/25572","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.143602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.490366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.361292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.343544+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.905429+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969298+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.949524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-03T19:44:28.060 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.060+0000 7fb474cad700 1 -- 172.21.15.77:0/1466911237 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb450063ce0 msgr2=0x7fb450066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.061 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.060+0000 7fb474cad700 1 --2- 172.21.15.77:0/1466911237 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb450063ce0 0x7fb450066190 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb46400afd0 tx=0x7fb464019000 comp rx=0 tx=0).stop 2022-04-03T19:44:28.061 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.060+0000 7fb474cad700 1 -- 172.21.15.77:0/1466911237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4700c3270 msgr2=0x7fb4701690a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.061 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.060+0000 7fb474cad700 1 --2- 172.21.15.77:0/1466911237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4700c3270 0x7fb4701690a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb46000c440 tx=0x7fb460007d90 comp rx=0 tx=0).stop 2022-04-03T19:44:28.061 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.060+0000 7fb474cad700 1 -- 172.21.15.77:0/1466911237 shutdown_connections 2022-04-03T19:44:28.062 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.060+0000 7fb474cad700 1 --2- 172.21.15.77:0/1466911237 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb450063ce0 0x7fb450066190 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.062 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.060+0000 7fb474cad700 1 --2- 172.21.15.77:0/1466911237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4700c7af0 0x7fb470151760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.062 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.060+0000 7fb474cad700 1 --2- 172.21.15.77:0/1466911237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4700c3270 0x7fb4701690a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.062 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.060+0000 7fb474cad700 1 --2- 172.21.15.77:0/1466911237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4700c3640 0x7fb4700bb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.062 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.060+0000 7fb474cad700 1 -- 172.21.15.77:0/1466911237 >> 172.21.15.77:0/1466911237 conn(0x7fb470066da0 msgr2=0x7fb4700c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:28.063 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.060+0000 7fb474cad700 1 -- 172.21.15.77:0/1466911237 shutdown_connections 2022-04-03T19:44:28.063 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.060+0000 7fb474cad700 1 -- 172.21.15.77:0/1466911237 wait complete. 2022-04-03T19:44:28.084 INFO:tasks.ceph.ceph_manager.ceph:all up! 2022-04-03T19:44:28.085 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-04-03T19:44:28.282 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.282+0000 7fec9359e700 1 Processor -- start 2022-04-03T19:44:28.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.282+0000 7fec9359e700 1 -- start start 2022-04-03T19:44:28.284 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.283+0000 7fec9359e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fec9406d3f0 0x7fec9406d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.283+0000 7fec9359e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fec940c5930 0x7fec94155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.283+0000 7fec9359e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec940c2f80 con 0x7fec94156120 2022-04-03T19:44:28.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.283+0000 7fec9359e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec940c3260 con 0x7fec9406d3f0 2022-04-03T19:44:28.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.283+0000 7fec9359e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec940c3540 con 0x7fec940c5930 2022-04-03T19:44:28.286 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.283+0000 7fec9259c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fec9406d3f0 0x7fec9406d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.286 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.283+0000 7fec91d9b700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fec940c5930 0x7fec94155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.286 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.283+0000 7fec92d9d700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fec94156120 0x7fec9406c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44110/0 (socket says 172.21.15.77:44110) 2022-04-03T19:44:28.286 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.283+0000 7fec91d9b700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fec940c5930 0x7fec94155be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37710/0 (socket says 172.21.15.77:37710) 2022-04-03T19:44:28.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.283+0000 7fec92d9d700 1 -- 172.21.15.77:0/2526504426 learned_addr learned my addr 172.21.15.77:0/2526504426 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:28.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.283+0000 7fec9259c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fec9406d3f0 0x7fec9406d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42522/0 (socket says 172.21.15.77:42522) 2022-04-03T19:44:28.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.284+0000 7fec91d9b700 1 -- 172.21.15.77:0/2526504426 >> v2:172.21.15.77:3300/0 conn(0x7fec9406d3f0 msgr2=0x7fec9406d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:28.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.284+0000 7fec91d9b700 1 --2- 172.21.15.77:0/2526504426 >> v2:172.21.15.77:3300/0 conn(0x7fec9406d3f0 0x7fec9406d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.284+0000 7fec91d9b700 1 -- 172.21.15.77:0/2526504426 >> v1:172.21.15.186:6789/0 conn(0x7fec94156120 legacy=0x7fec9406c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.284+0000 7fec9159a700 1 -- 172.21.15.77:0/2526504426 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332218924 0 0) 0x7fec940c2f80 con 0x7fec94156120 2022-04-03T19:44:28.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.284+0000 7fec91d9b700 1 -- 172.21.15.77:0/2526504426 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec940c3820 con 0x7fec940c5930 2022-04-03T19:44:28.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.284+0000 7fec91d9b700 1 --2- 172.21.15.77:0/2526504426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec940c5930 0x7fec94155be0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fec88009760 tx=0x7fec88009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d4530b79df005ee server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:28.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.284+0000 7fec9159a700 1 -- 172.21.15.77:0/2526504426 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec88011630 con 0x7fec940c5930 2022-04-03T19:44:28.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.284+0000 7fec9159a700 1 -- 172.21.15.77:0/2526504426 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec88011790 con 0x7fec940c5930 2022-04-03T19:44:28.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.284+0000 7fec9159a700 1 -- 172.21.15.77:0/2526504426 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec88011b30 con 0x7fec940c5930 2022-04-03T19:44:28.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.285+0000 7fec9359e700 1 -- 172.21.15.77:0/2526504426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec940c5930 msgr2=0x7fec94155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.285+0000 7fec9359e700 1 --2- 172.21.15.77:0/2526504426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec940c5930 0x7fec94155be0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fec88009760 tx=0x7fec88009060 comp rx=0 tx=0).stop 2022-04-03T19:44:28.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.285+0000 7fec9359e700 1 -- 172.21.15.77:0/2526504426 shutdown_connections 2022-04-03T19:44:28.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.285+0000 7fec9359e700 1 --2- 172.21.15.77:0/2526504426 >> v2:172.21.15.77:3300/0 conn(0x7fec9406d3f0 0x7fec9406d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.285+0000 7fec9359e700 1 --2- 172.21.15.77:0/2526504426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec940c5930 0x7fec94155be0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.285+0000 7fec9359e700 1 -- 172.21.15.77:0/2526504426 >> 172.21.15.77:0/2526504426 conn(0x7fec940cb720 msgr2=0x7fec940cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:28.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.285+0000 7fec9359e700 1 -- 172.21.15.77:0/2526504426 shutdown_connections 2022-04-03T19:44:28.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.285+0000 7fec9359e700 1 -- 172.21.15.77:0/2526504426 wait complete. 2022-04-03T19:44:28.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.287+0000 7fec9359e700 1 Processor -- start 2022-04-03T19:44:28.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.287+0000 7fec9359e700 1 -- start start 2022-04-03T19:44:28.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.288+0000 7fec9359e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec9406d3f0 0x7fec94165b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.288+0000 7fec9359e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec940c5930 0x7fec9414cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.288+0000 7fec9359e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec94156120 0x7fec940c0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.288+0000 7fec9359e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec94168690 con 0x7fec940c5930 2022-04-03T19:44:28.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.288+0000 7fec9359e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec94169410 con 0x7fec9406d3f0 2022-04-03T19:44:28.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.288+0000 7fec9359e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec941529b0 con 0x7fec94156120 2022-04-03T19:44:28.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.288+0000 7fec91d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec940c5930 0x7fec9414cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.288+0000 7fec9259c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec9406d3f0 0x7fec94165b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.288+0000 7fec92d9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec94156120 0x7fec940c0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.288+0000 7fec9259c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec9406d3f0 0x7fec94165b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37712/0 (socket says 172.21.15.77:37712) 2022-04-03T19:44:28.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.288+0000 7fec9259c700 1 -- 172.21.15.77:0/3271152194 learned_addr learned my addr 172.21.15.77:0/3271152194 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:28.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.289+0000 7fec9259c700 1 -- 172.21.15.77:0/3271152194 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec94156120 msgr2=0x7fec940c0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.289+0000 7fec9259c700 1 --2- 172.21.15.77:0/3271152194 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec94156120 0x7fec940c0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.289+0000 7fec9259c700 1 -- 172.21.15.77:0/3271152194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec940c5930 msgr2=0x7fec9414cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.289+0000 7fec9259c700 1 --2- 172.21.15.77:0/3271152194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec940c5930 0x7fec9414cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.289+0000 7fec9259c700 1 -- 172.21.15.77:0/3271152194 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec880092a0 con 0x7fec9406d3f0 2022-04-03T19:44:28.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.289+0000 7fec91d9b700 1 --2- 172.21.15.77:0/3271152194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec940c5930 0x7fec9414cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:28.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.289+0000 7fec9259c700 1 --2- 172.21.15.77:0/3271152194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec9406d3f0 0x7fec94165b20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fec8400e900 tx=0x7fec84009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:28.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.289+0000 7fec827fc700 1 -- 172.21.15.77:0/3271152194 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec84013680 con 0x7fec9406d3f0 2022-04-03T19:44:28.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.289+0000 7fec827fc700 1 -- 172.21.15.77:0/3271152194 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec840137e0 con 0x7fec9406d3f0 2022-04-03T19:44:28.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.289+0000 7fec9359e700 1 -- 172.21.15.77:0/3271152194 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec94168930 con 0x7fec9406d3f0 2022-04-03T19:44:28.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.289+0000 7fec827fc700 1 -- 172.21.15.77:0/3271152194 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec84013b80 con 0x7fec9406d3f0 2022-04-03T19:44:28.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.289+0000 7fec9359e700 1 -- 172.21.15.77:0/3271152194 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec94168ef0 con 0x7fec9406d3f0 2022-04-03T19:44:28.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.290+0000 7fec9359e700 1 -- 172.21.15.77:0/3271152194 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec941690e0 con 0x7fec9406d3f0 2022-04-03T19:44:28.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.291+0000 7fec827fc700 1 -- 172.21.15.77:0/3271152194 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fec84019030 con 0x7fec9406d3f0 2022-04-03T19:44:28.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.292+0000 7fec827fc700 1 --2- 172.21.15.77:0/3271152194 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fec70063d10 0x7fec700661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.292+0000 7fec827fc700 1 -- 172.21.15.77:0/3271152194 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fec84080660 con 0x7fec9406d3f0 2022-04-03T19:44:28.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.292+0000 7fec91d9b700 1 --2- 172.21.15.77:0/3271152194 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fec70063d10 0x7fec700661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.293+0000 7fec91d9b700 1 --2- 172.21.15.77:0/3271152194 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fec70063d10 0x7fec700661c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fec940c0d90 tx=0x7fec88017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:28.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.297+0000 7fec827fc700 1 -- 172.21.15.77:0/3271152194 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fec8404f1e0 con 0x7fec9406d3f0 2022-04-03T19:44:28.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.495+0000 7fec9359e700 1 -- 172.21.15.77:0/3271152194 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec9404b050 con 0x7fec9406d3f0 2022-04-03T19:44:28.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.496+0000 7fec827fc700 1 -- 172.21.15.77:0/3271152194 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16845 (secure 0 0 0) 0x7fec84052800 con 0x7fec9406d3f0 2022-04-03T19:44:28.497 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:28.497 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"89c3a3f4-599d-4c4a-8f5a-7180eafb7292","created":"2022-04-03T19:43:54.101559+0000","modified":"2022-04-03T19:44:22.278254+0000","last_up_change":"2022-04-03T19:44:05.589174+0000","last_in_change":"2022-04-03T19:43:56.729760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-03T19:44:07.567638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-04-03T19:44:13.149481+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-04-03T19:44:18.823926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-04-03T19:44:20.275479+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f35f5195-3729-4d78-85d1-0539f259283f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6823","nonce":20252},{"type":"v1","addr":"172.21.15.77:6825","nonce":20252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":20252},{"type":"v1","addr":"172.21.15.77:6827","nonce":20252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":20252},{"type":"v1","addr":"172.21.15.77:6831","nonce":20252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":20252},{"type":"v1","addr":"172.21.15.77:6829","nonce":20252}]},"public_addr":"172.21.15.77:6825/20252","cluster_addr":"172.21.15.77:6827/20252","heartbeat_back_addr":"172.21.15.77:6831/20252","heartbeat_front_addr":"172.21.15.77:6829/20252","state":["exists","up"]},{"osd":1,"uuid":"ab1c89dc-57ce-4cc4-b8c7-9b0c19939601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":20249},{"type":"v1","addr":"172.21.15.77:6805","nonce":20249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":20249},{"type":"v1","addr":"172.21.15.77:6809","nonce":20249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":20249},{"type":"v1","addr":"172.21.15.77:6819","nonce":20249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":20249},{"type":"v1","addr":"172.21.15.77:6815","nonce":20249}]},"public_addr":"172.21.15.77:6805/20249","cluster_addr":"172.21.15.77:6809/20249","heartbeat_back_addr":"172.21.15.77:6819/20249","heartbeat_front_addr":"172.21.15.77:6815/20249","state":["exists","up"]},{"osd":2,"uuid":"351e218b-7749-4ee9-b44a-0a5b52fb9be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":20248},{"type":"v1","addr":"172.21.15.77:6801","nonce":20248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":20248},{"type":"v1","addr":"172.21.15.77:6803","nonce":20248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":20248},{"type":"v1","addr":"172.21.15.77:6813","nonce":20248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":20248},{"type":"v1","addr":"172.21.15.77:6808","nonce":20248}]},"public_addr":"172.21.15.77:6801/20248","cluster_addr":"172.21.15.77:6803/20248","heartbeat_back_addr":"172.21.15.77:6813/20248","heartbeat_front_addr":"172.21.15.77:6808/20248","state":["exists","up"]},{"osd":3,"uuid":"1ba6ec8a-7cbf-4c12-8e21-c3097a263ae8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":20250},{"type":"v1","addr":"172.21.15.77:6814","nonce":20250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":20250},{"type":"v1","addr":"172.21.15.77:6818","nonce":20250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":20250},{"type":"v1","addr":"172.21.15.77:6824","nonce":20250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":20250},{"type":"v1","addr":"172.21.15.77:6821","nonce":20250}]},"public_addr":"172.21.15.77:6814/20250","cluster_addr":"172.21.15.77:6818/20250","heartbeat_back_addr":"172.21.15.77:6824/20250","heartbeat_front_addr":"172.21.15.77:6821/20250","state":["exists","up"]},{"osd":4,"uuid":"3faf4c09-846c-4133-856b-7f53c535a67d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":25551},{"type":"v1","addr":"172.21.15.186:6822","nonce":25551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":25551},{"type":"v1","addr":"172.21.15.186:6826","nonce":25551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":25551},{"type":"v1","addr":"172.21.15.186:6831","nonce":25551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":25551},{"type":"v1","addr":"172.21.15.186:6829","nonce":25551}]},"public_addr":"172.21.15.186:6822/25551","cluster_addr":"172.21.15.186:6826/25551","heartbeat_back_addr":"172.21.15.186:6831/25551","heartbeat_front_addr":"172.21.15.186:6829/25551","state":["exists","up"]},{"osd":5,"uuid":"ffe41d00-d0aa-42f4-82c1-5cced13c1bc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":25541},{"type":"v1","addr":"172.21.15.186:6809","nonce":25541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":25541},{"type":"v1","addr":"172.21.15.186:6811","nonce":25541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":25541},{"type":"v1","addr":"172.21.15.186:6815","nonce":25541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":25541},{"type":"v1","addr":"172.21.15.186:6813","nonce":25541}]},"public_addr":"172.21.15.186:6809/25541","cluster_addr":"172.21.15.186:6811/25541","heartbeat_back_addr":"172.21.15.186:6815/25541","heartbeat_front_addr":"172.21.15.186:6813/25541","state":["exists","up"]},{"osd":6,"uuid":"b0344ee8-deca-4e9b-934c-5866d90d0073","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":25542},{"type":"v1","addr":"172.21.15.186:6801","nonce":25542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":25542},{"type":"v1","addr":"172.21.15.186:6803","nonce":25542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":25542},{"type":"v1","addr":"172.21.15.186:6807","nonce":25542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":25542},{"type":"v1","addr":"172.21.15.186:6805","nonce":25542}]},"public_addr":"172.21.15.186:6801/25542","cluster_addr":"172.21.15.186:6803/25542","heartbeat_back_addr":"172.21.15.186:6807/25542","heartbeat_front_addr":"172.21.15.186:6805/25542","state":["exists","up"]},{"osd":7,"uuid":"0eb0e5d3-3dad-4f49-9b50-3b24a3924bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":25572},{"type":"v1","addr":"172.21.15.186:6817","nonce":25572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":25572},{"type":"v1","addr":"172.21.15.186:6819","nonce":25572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6825","nonce":25572},{"type":"v1","addr":"172.21.15.186:6827","nonce":25572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":25572},{"type":"v1","addr":"172.21.15.186:6823","nonce":25572}]},"public_addr":"172.21.15.186:6817/25572","cluster_addr":"172.21.15.186:6819/25572","heartbeat_back_addr":"172.21.15.186:6827/25572","heartbeat_front_addr":"172.21.15.186:6823/25572","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.143602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.490366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.361292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.343544+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.905429+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969298+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.949524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-03T19:44:28.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.498+0000 7fec9359e700 1 -- 172.21.15.77:0/3271152194 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fec70063d10 msgr2=0x7fec700661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.498+0000 7fec9359e700 1 --2- 172.21.15.77:0/3271152194 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fec70063d10 0x7fec700661c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fec940c0d90 tx=0x7fec88017000 comp rx=0 tx=0).stop 2022-04-03T19:44:28.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.498+0000 7fec9359e700 1 -- 172.21.15.77:0/3271152194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec9406d3f0 msgr2=0x7fec94165b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.498+0000 7fec9359e700 1 --2- 172.21.15.77:0/3271152194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec9406d3f0 0x7fec94165b20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fec8400e900 tx=0x7fec84009060 comp rx=0 tx=0).stop 2022-04-03T19:44:28.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.498+0000 7fec9359e700 1 -- 172.21.15.77:0/3271152194 shutdown_connections 2022-04-03T19:44:28.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.498+0000 7fec9359e700 1 --2- 172.21.15.77:0/3271152194 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fec70063d10 0x7fec700661c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.498+0000 7fec9359e700 1 --2- 172.21.15.77:0/3271152194 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec94156120 0x7fec940c0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.498+0000 7fec9359e700 1 --2- 172.21.15.77:0/3271152194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec9406d3f0 0x7fec94165b20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.498+0000 7fec9359e700 1 --2- 172.21.15.77:0/3271152194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec940c5930 0x7fec9414cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.498+0000 7fec9359e700 1 -- 172.21.15.77:0/3271152194 >> 172.21.15.77:0/3271152194 conn(0x7fec940cb720 msgr2=0x7fec940c8140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:28.502 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.498+0000 7fec9359e700 1 -- 172.21.15.77:0/3271152194 shutdown_connections 2022-04-03T19:44:28.502 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.499+0000 7fec9359e700 1 -- 172.21.15.77:0/3271152194 wait complete. 2022-04-03T19:44:28.524 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2022-04-03T19:44:28.524 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2022-04-03T19:44:28.525 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2022-04-03T19:44:28.525 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2022-04-03T19:44:28.526 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2022-04-03T19:44:28.527 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2022-04-03T19:44:28.527 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2022-04-03T19:44:28.528 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2022-04-03T19:44:28.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.885+0000 7f280d646700 1 Processor -- start 2022-04-03T19:44:28.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.885+0000 7f1aa759e700 1 Processor -- start 2022-04-03T19:44:28.888 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.886+0000 7f1aa759e700 1 -- start start 2022-04-03T19:44:28.889 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.886+0000 7f280d646700 1 -- start start 2022-04-03T19:44:28.889 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.888+0000 7f280d646700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f28080c6920 0x7f2808067bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.889 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.888+0000 7f280d646700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f28080692e0 0x7f2808068130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.889 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.888+0000 7f280d646700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f280815dce0 con 0x7f2808068f10 2022-04-03T19:44:28.890 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.888+0000 7f280d646700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2808067760 con 0x7f28080c6920 2022-04-03T19:44:28.890 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.888+0000 7f280d646700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f280815d160 con 0x7f28080692e0 2022-04-03T19:44:28.890 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.888+0000 7f1aa759e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1aa000e2e0 0x7f1aa0039df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.888+0000 7f1aa759e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1aa0037cb0 0x7f1aa00d77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.888+0000 7f1aa759e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa0007170 con 0x7f1aa003a330 2022-04-03T19:44:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.888+0000 7f1aa759e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa00062a0 con 0x7f1aa000e2e0 2022-04-03T19:44:28.891 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.888+0000 7f1aa759e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa0006560 con 0x7f1aa0037cb0 2022-04-03T19:44:28.892 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.889+0000 7f1aa5d9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1aa003a330 0x7f1aa00375a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38300/0 (socket says 172.21.15.77:38300) 2022-04-03T19:44:28.892 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.889+0000 7f1aa5d9b700 1 -- 172.21.15.77:0/162035528 learned_addr learned my addr 172.21.15.77:0/162035528 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:28.892 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.889+0000 7f1aa659c700 1 --2- 172.21.15.77:0/162035528 >> v2:172.21.15.186:3300/0 conn(0x7f1aa000e2e0 0x7f1aa0039df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.892 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.889+0000 7f1aa6d9d700 1 --2- 172.21.15.77:0/162035528 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1aa0037cb0 0x7f1aa00d77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.893 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.888+0000 7f23b7585700 1 Processor -- start 2022-04-03T19:44:28.893 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.888+0000 7f28077fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f28080692e0 0x7f2808068130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.894 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.888+0000 7f28077fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f28080692e0 0x7f2808068130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37714/0 (socket says 172.21.15.77:37714) 2022-04-03T19:44:28.894 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.888+0000 7f28077fe700 1 -- 172.21.15.77:0/3694177308 learned_addr learned my addr 172.21.15.77:0/3694177308 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:28.894 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.889+0000 7f28077fe700 1 -- 172.21.15.77:0/3694177308 >> v2:172.21.15.77:3300/0 conn(0x7f28080c6920 msgr2=0x7f2808067bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:28.894 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.889+0000 7f28077fe700 1 --2- 172.21.15.77:0/3694177308 >> v2:172.21.15.77:3300/0 conn(0x7f28080c6920 0x7f2808067bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.895 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.889+0000 7f28077fe700 1 -- 172.21.15.77:0/3694177308 >> v1:172.21.15.77:6789/0 conn(0x7f2808068f10 legacy=0x7f28080c3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:28.895 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.889+0000 7f28077fe700 1 -- 172.21.15.77:0/3694177308 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f280815d440 con 0x7f28080692e0 2022-04-03T19:44:28.895 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.890+0000 7f28077fe700 1 --2- 172.21.15.77:0/3694177308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28080692e0 0x7f2808068130 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f28080af690 tx=0x7f2800009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0583abe7f93283b server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:28.896 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.890+0000 7f1aa659c700 1 -- 172.21.15.77:0/162035528 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1aa0037cb0 msgr2=0x7f1aa00d77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:28.896 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.890+0000 7f1aa659c700 1 --2- 172.21.15.77:0/162035528 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1aa0037cb0 0x7f1aa00d77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.897 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.890+0000 7f1aa659c700 1 -- 172.21.15.77:0/162035528 >> v1:172.21.15.77:6789/0 conn(0x7f1aa003a330 legacy=0x7f1aa00375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.897 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.891+0000 7f2805ffb700 1 -- 172.21.15.77:0/3694177308 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280000e070 con 0x7f28080692e0 2022-04-03T19:44:28.898 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.891+0000 7ffa40c63700 1 Processor -- start 2022-04-03T19:44:28.898 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.889+0000 7f23b7585700 1 -- start start 2022-04-03T19:44:28.899 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.891+0000 7f23b7585700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23a8049df0 0x7f23a804b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.899 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.891+0000 7f23b7585700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a80dc250 con 0x7f23a80d5c70 2022-04-03T19:44:28.899 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.891+0000 7f23b7585700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a800e890 con 0x7f23a80d58a0 2022-04-03T19:44:28.899 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.891+0000 7f23b7585700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a80db730 con 0x7f23a8049df0 2022-04-03T19:44:28.900 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.891+0000 7f2805ffb700 1 -- 172.21.15.77:0/3694177308 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280000ba10 con 0x7f28080692e0 2022-04-03T19:44:28.901 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.890+0000 7f1aa659c700 1 -- 172.21.15.77:0/162035528 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa0006840 con 0x7f1aa000e2e0 2022-04-03T19:44:28.901 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.891+0000 7f1aa659c700 1 --2- 172.21.15.77:0/162035528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1aa000e2e0 0x7f1aa0039df0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009760 tx=0x7f1a9c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=848a9b943db00847 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:28.901 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.892+0000 7f23b4b20700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f23a80d5c70 0x7f23a800c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38304/0 (socket says 172.21.15.77:38304) 2022-04-03T19:44:28.902 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.892+0000 7f23b4b20700 1 -- 172.21.15.77:0/2577442575 learned_addr learned my addr 172.21.15.77:0/2577442575 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:28.902 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.892+0000 7f1aa559a700 1 -- 172.21.15.77:0/162035528 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c011630 con 0x7f1aa000e2e0 2022-04-03T19:44:28.903 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.892+0000 7f280d646700 1 -- 172.21.15.77:0/3694177308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28080692e0 msgr2=0x7f2808068130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.903 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.892+0000 7f280d646700 1 --2- 172.21.15.77:0/3694177308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28080692e0 0x7f2808068130 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f28080af690 tx=0x7f2800009840 comp rx=0 tx=0).stop 2022-04-03T19:44:28.903 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.891+0000 7ffa40c63700 1 -- start start 2022-04-03T19:44:28.904 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.892+0000 7f1aa559a700 1 -- 172.21.15.77:0/162035528 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9c011790 con 0x7f1aa000e2e0 2022-04-03T19:44:28.904 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.892+0000 7f1aa559a700 1 -- 172.21.15.77:0/162035528 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c011b50 con 0x7f1aa000e2e0 2022-04-03T19:44:28.905 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.893+0000 7f239ffff700 1 -- 172.21.15.77:0/2577442575 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294396056 0 0) 0x7f23a80dc250 con 0x7f23a80d5c70 2022-04-03T19:44:28.905 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.893+0000 7f239ffff700 1 -- 172.21.15.77:0/2577442575 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23980033f0 con 0x7f23a80d5c70 2022-04-03T19:44:28.906 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.893+0000 7f23b5b22700 1 --2- 172.21.15.77:0/2577442575 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23a8049df0 0x7f23a804b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.906 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.893+0000 7ffa40c63700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffa3c06d3f0 0x7ffa3c06d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.906 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.893+0000 7ffa40c63700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ffa3c0c5930 0x7ffa3c155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.907 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.893+0000 7ffa40c63700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa3c0c2f80 con 0x7ffa3c156120 2022-04-03T19:44:28.907 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.893+0000 7ffa40c63700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa3c0c3260 con 0x7ffa3c06d3f0 2022-04-03T19:44:28.907 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.893+0000 7ffa40c63700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa3c0c3540 con 0x7ffa3c0c5930 2022-04-03T19:44:28.908 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.894+0000 7f280d646700 1 -- 172.21.15.77:0/3694177308 shutdown_connections 2022-04-03T19:44:28.908 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.894+0000 7f280d646700 1 --2- 172.21.15.77:0/3694177308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28080692e0 0x7f2808068130 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.908 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.894+0000 7f280d646700 1 --2- 172.21.15.77:0/3694177308 >> v2:172.21.15.77:3300/0 conn(0x7f28080c6920 0x7f2808067bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.908 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.894+0000 7f280d646700 1 -- 172.21.15.77:0/3694177308 >> 172.21.15.77:0/3694177308 conn(0x7f28080c9fd0 msgr2=0x7f28080c5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:28.909 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.894+0000 7ffa3b7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffa3c06d3f0 0x7ffa3c06d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.909 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.894+0000 7ffa3b7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ffa3c06d3f0 0x7ffa3c06d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42530/0 (socket says 172.21.15.77:42530) 2022-04-03T19:44:28.909 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.894+0000 7ffa3bfff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ffa3c156120 0x7ffa3c06c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38306/0 (socket says 172.21.15.77:38306) 2022-04-03T19:44:28.909 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.894+0000 7ffa3bfff700 1 -- 172.21.15.77:0/3537117982 learned_addr learned my addr 172.21.15.77:0/3537117982 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:28.910 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.895+0000 7f23b5b22700 1 -- 172.21.15.77:0/2577442575 >> v1:172.21.15.186:6789/0 conn(0x7f23a80d58a0 legacy=0x7f23a800de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.911 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.895+0000 7ffa3b7fe700 1 -- 172.21.15.77:0/3537117982 >> v2:172.21.15.186:3300/0 conn(0x7ffa3c0c5930 msgr2=0x7ffa3c155be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-04-03T19:44:28.911 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.895+0000 7f23b5b22700 1 -- 172.21.15.77:0/2577442575 >> v1:172.21.15.77:6789/0 conn(0x7f23a80d5c70 legacy=0x7f23a800c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.911 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.895+0000 7f23b5b22700 1 -- 172.21.15.77:0/2577442575 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a80db920 con 0x7f23a8049df0 2022-04-03T19:44:28.912 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.895+0000 7f1aa759e700 1 -- 172.21.15.77:0/162035528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1aa000e2e0 msgr2=0x7f1aa0039df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.912 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.895+0000 7f1aa759e700 1 --2- 172.21.15.77:0/162035528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1aa000e2e0 0x7f1aa0039df0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009760 tx=0x7f1a9c009060 comp rx=0 tx=0).stop 2022-04-03T19:44:28.913 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.895+0000 7ffa3b7fe700 1 --2- 172.21.15.77:0/3537117982 >> v2:172.21.15.186:3300/0 conn(0x7ffa3c0c5930 0x7ffa3c155be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.913 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.896+0000 7f23b5b22700 1 --2- 172.21.15.77:0/2577442575 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23a8049df0 0x7f23a804b800 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f23a0009710 tx=0x7f23a0009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efdfd685617e1b47 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:28.914 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.896+0000 7ffa3b7fe700 1 -- 172.21.15.77:0/3537117982 >> v1:172.21.15.77:6789/0 conn(0x7ffa3c156120 legacy=0x7ffa3c06c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.914 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.896+0000 7ffa3b7fe700 1 -- 172.21.15.77:0/3537117982 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa3c0c3820 con 0x7ffa3c06d3f0 2022-04-03T19:44:28.915 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.896+0000 7ffa3b7fe700 1 --2- 172.21.15.77:0/3537117982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa3c06d3f0 0x7ffa3c06d7c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009760 tx=0x7ffa24009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20c9f526d2d4358f server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:28.915 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.896+0000 7f239ffff700 1 -- 172.21.15.77:0/2577442575 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a0004030 con 0x7f23a8049df0 2022-04-03T19:44:28.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.897+0000 7f239ffff700 1 -- 172.21.15.77:0/2577442575 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a0004190 con 0x7f23a8049df0 2022-04-03T19:44:28.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.897+0000 7f239ffff700 1 -- 172.21.15.77:0/2577442575 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a00105b0 con 0x7f23a8049df0 2022-04-03T19:44:28.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.897+0000 7ffa3a7fc700 1 -- 172.21.15.77:0/3537117982 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24011630 con 0x7ffa3c06d3f0 2022-04-03T19:44:28.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.897+0000 7ffa3a7fc700 1 -- 172.21.15.77:0/3537117982 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa24011790 con 0x7ffa3c06d3f0 2022-04-03T19:44:28.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.897+0000 7ffa3a7fc700 1 -- 172.21.15.77:0/3537117982 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24011b30 con 0x7ffa3c06d3f0 2022-04-03T19:44:28.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.898+0000 7f23b7585700 1 -- 172.21.15.77:0/2577442575 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23a8049df0 msgr2=0x7f23a804b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.898+0000 7f23b7585700 1 --2- 172.21.15.77:0/2577442575 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23a8049df0 0x7f23a804b800 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f23a0009710 tx=0x7f23a0009060 comp rx=0 tx=0).stop 2022-04-03T19:44:28.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.898+0000 7f1aa759e700 1 -- 172.21.15.77:0/162035528 shutdown_connections 2022-04-03T19:44:28.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.898+0000 7f1aa759e700 1 --2- 172.21.15.77:0/162035528 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1aa0037cb0 0x7f1aa00d77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.898+0000 7f1aa759e700 1 --2- 172.21.15.77:0/162035528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1aa000e2e0 0x7f1aa0039df0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.898+0000 7f23b7585700 1 -- 172.21.15.77:0/2577442575 shutdown_connections 2022-04-03T19:44:28.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.898+0000 7f23b7585700 1 --2- 172.21.15.77:0/2577442575 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23a8049df0 0x7f23a804b800 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.920 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.898+0000 7f23b7585700 1 -- 172.21.15.77:0/2577442575 >> 172.21.15.77:0/2577442575 conn(0x7f23a8051fe0 msgr2=0x7f23a80523e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:28.920 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.898+0000 7f1aa759e700 1 -- 172.21.15.77:0/162035528 >> 172.21.15.77:0/162035528 conn(0x7f1aa0054290 msgr2=0x7f1aa0054690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:28.920 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.898+0000 7ffa40c63700 1 -- 172.21.15.77:0/3537117982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa3c06d3f0 msgr2=0x7ffa3c06d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.898+0000 7ffa40c63700 1 --2- 172.21.15.77:0/3537117982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa3c06d3f0 0x7ffa3c06d7c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009760 tx=0x7ffa24009060 comp rx=0 tx=0).stop 2022-04-03T19:44:28.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.898+0000 7f1aa759e700 1 -- 172.21.15.77:0/162035528 shutdown_connections 2022-04-03T19:44:28.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.898+0000 7f23b7585700 1 -- 172.21.15.77:0/2577442575 shutdown_connections 2022-04-03T19:44:28.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.898+0000 7f280d646700 1 -- 172.21.15.77:0/3694177308 shutdown_connections 2022-04-03T19:44:28.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.898+0000 7f23b7585700 1 -- 172.21.15.77:0/2577442575 wait complete. 2022-04-03T19:44:28.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.899+0000 7f280d646700 1 -- 172.21.15.77:0/3694177308 wait complete. 2022-04-03T19:44:28.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.899+0000 7ffa40c63700 1 -- 172.21.15.77:0/3537117982 shutdown_connections 2022-04-03T19:44:28.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.899+0000 7ffa40c63700 1 --2- 172.21.15.77:0/3537117982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa3c06d3f0 0x7ffa3c06d7c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.899+0000 7ffa40c63700 1 --2- 172.21.15.77:0/3537117982 >> v2:172.21.15.186:3300/0 conn(0x7ffa3c0c5930 0x7ffa3c155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.899+0000 7ffa40c63700 1 -- 172.21.15.77:0/3537117982 >> 172.21.15.77:0/3537117982 conn(0x7ffa3c0cb720 msgr2=0x7ffa3c0cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:28.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.899+0000 7f1aa759e700 1 -- 172.21.15.77:0/162035528 wait complete. 2022-04-03T19:44:28.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.901+0000 7f1aa759e700 1 Processor -- start 2022-04-03T19:44:28.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.901+0000 7f280d646700 1 Processor -- start 2022-04-03T19:44:28.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.900+0000 7ffa40c63700 1 -- 172.21.15.77:0/3537117982 shutdown_connections 2022-04-03T19:44:28.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.900+0000 7ffa40c63700 1 -- 172.21.15.77:0/3537117982 wait complete. 2022-04-03T19:44:28.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.901+0000 7f23b7585700 1 Processor -- start 2022-04-03T19:44:28.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.901+0000 7f1aa759e700 1 -- start start 2022-04-03T19:44:28.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.903+0000 7f1aa759e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa000e2e0 0x7f1aa00e35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.903+0000 7f1aa759e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1aa0037cb0 0x7f1aa00dd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.903+0000 7f1aa759e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1aa00dacd0 0x7f1aa00ddc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.903+0000 7f23b7585700 1 -- start start 2022-04-03T19:44:28.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.904+0000 7ffa40c63700 1 Processor -- start 2022-04-03T19:44:28.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.903+0000 7f1aa5d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1aa0037cb0 0x7f1aa00dd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.901+0000 7f280d646700 1 -- start start 2022-04-03T19:44:28.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.904+0000 7ffa40c63700 1 -- start start 2022-04-03T19:44:28.931 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.903+0000 7f280d646700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2808068f10 0x7f2808172710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.931 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.903+0000 7f280d646700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28080692e0 0x7f2808154700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.931 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.903+0000 7f280d646700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28080c6920 0x7f28080ae560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.932 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.904+0000 7f28067fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28080692e0 0x7f2808154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.932 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.904+0000 7f28067fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28080692e0 0x7f2808154700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42532/0 (socket says 172.21.15.77:42532) 2022-04-03T19:44:28.933 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.903+0000 7f1aa5d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1aa0037cb0 0x7f1aa00dd6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37718/0 (socket says 172.21.15.77:37718) 2022-04-03T19:44:28.933 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.903+0000 7f1aa5d9b700 1 -- 172.21.15.77:0/1258161292 learned_addr learned my addr 172.21.15.77:0/1258161292 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:28.933 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.904+0000 7f1aa6d9d700 1 --2- 172.21.15.77:0/1258161292 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1aa00dacd0 0x7f1aa00ddc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.934 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.905+0000 7f23b7585700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23a8049df0 0x7f23a80084c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.934 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.905+0000 7f23b7585700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23a80d58a0 0x7f23a804f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.905+0000 7ffa40c63700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa3c06d3f0 0x7ffa3c1579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.905+0000 7f23b7585700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23a80d5c70 0x7f23a80d7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.906+0000 7ffa40c63700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffa3c0c5930 0x7ffa3c157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.905+0000 7f1aa759e700 1 -- 172.21.15.77:0/1258161292 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa0009f20 con 0x7f1aa000e2e0 2022-04-03T19:44:28.937 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.906+0000 7ffa3b7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa3c06d3f0 0x7ffa3c1579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.938 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.905+0000 7f23b7585700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a80d6140 con 0x7f23a80d58a0 2022-04-03T19:44:28.938 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.905+0000 7f23b7585700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a8009790 con 0x7f23a80d5c70 2022-04-03T19:44:28.938 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.905+0000 7f23b7585700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a8009a70 con 0x7f23a8049df0 2022-04-03T19:44:28.939 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.906+0000 7ffa3b7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa3c06d3f0 0x7ffa3c1579e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37724/0 (socket says 172.21.15.77:37724) 2022-04-03T19:44:28.939 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.905+0000 7f23b5321700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23a8049df0 0x7f23a80084c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.939 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.905+0000 7f23b5321700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23a8049df0 0x7f23a80084c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59830/0 (socket says 172.21.15.77:59830) 2022-04-03T19:44:28.939 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.905+0000 7f23b5321700 1 -- 172.21.15.77:0/2217993779 learned_addr learned my addr 172.21.15.77:0/2217993779 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:28.940 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.906+0000 7ffa3b7fe700 1 -- 172.21.15.77:0/596656967 learned_addr learned my addr 172.21.15.77:0/596656967 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:28.940 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.906+0000 7f23b5b22700 1 --2- 172.21.15.77:0/2217993779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23a80d5c70 0x7f23a80d7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.941 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.906+0000 7f23b4b20700 1 --2- 172.21.15.77:0/2217993779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23a80d58a0 0x7f23a804f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.941 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.906+0000 7ffa3affd700 1 --2- 172.21.15.77:0/596656967 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffa3c0c5930 0x7ffa3c157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.942 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.906+0000 7f23b5321700 1 -- 172.21.15.77:0/2217993779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23a80d5c70 msgr2=0x7f23a80d7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.942 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.906+0000 7ffa40c63700 1 --2- 172.21.15.77:0/596656967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa3c156120 0x7ffa3c14a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.943 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.906+0000 7f23b5321700 1 --2- 172.21.15.77:0/2217993779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23a80d5c70 0x7f23a80d7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.944 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.907+0000 7ffa40c63700 1 -- 172.21.15.77:0/596656967 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3c1530c0 con 0x7ffa3c156120 2022-04-03T19:44:28.944 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.907+0000 7ffa40c63700 1 -- 172.21.15.77:0/596656967 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3c0c1f10 con 0x7ffa3c06d3f0 2022-04-03T19:44:28.944 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.907+0000 7ffa40c63700 1 -- 172.21.15.77:0/596656967 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3c0c1020 con 0x7ffa3c0c5930 2022-04-03T19:44:28.944 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.907+0000 7ffa3bfff700 1 --2- 172.21.15.77:0/596656967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa3c156120 0x7ffa3c14a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.945 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.905+0000 7f1aa759e700 1 -- 172.21.15.77:0/1258161292 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa00db210 con 0x7f1aa0037cb0 2022-04-03T19:44:28.945 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.905+0000 7f1aa759e700 1 -- 172.21.15.77:0/1258161292 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa00db920 con 0x7f1aa00dacd0 2022-04-03T19:44:28.945 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.908+0000 7f1aa659c700 1 --2- 172.21.15.77:0/1258161292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa000e2e0 0x7f1aa00e35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.946 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.904+0000 7f280d646700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2808172c50 con 0x7f28080692e0 2022-04-03T19:44:28.946 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.904+0000 7f280d646700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2808158c10 con 0x7f28080c6920 2022-04-03T19:44:28.946 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.904+0000 7f280d646700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28081594b0 con 0x7f2808068f10 2022-04-03T19:44:28.947 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.904+0000 7f28067fc700 1 -- 172.21.15.77:0/74086498 learned_addr learned my addr 172.21.15.77:0/74086498 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:28.947 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.905+0000 7f28067fc700 1 -- 172.21.15.77:0/74086498 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2808068f10 msgr2=0x7f2808172710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-04-03T19:44:28.947 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.905+0000 7f28067fc700 1 --2- 172.21.15.77:0/74086498 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2808068f10 0x7f2808172710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.947 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.905+0000 7f28067fc700 1 -- 172.21.15.77:0/74086498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28080c6920 msgr2=0x7f28080ae560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.947 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.905+0000 7f28067fc700 1 --2- 172.21.15.77:0/74086498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28080c6920 0x7f28080ae560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.948 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.905+0000 7f28067fc700 1 -- 172.21.15.77:0/74086498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2800009ae0 con 0x7f28080692e0 2022-04-03T19:44:28.948 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.907+0000 7f28067fc700 1 --2- 172.21.15.77:0/74086498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28080692e0 0x7f2808154700 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f27f800b810 tx=0x7f27f8009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:28.948 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.907+0000 7ffa3b7fe700 1 -- 172.21.15.77:0/596656967 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffa3c0c5930 msgr2=0x7ffa3c157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.949 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.908+0000 7ffa3b7fe700 1 --2- 172.21.15.77:0/596656967 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffa3c0c5930 0x7ffa3c157f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.949 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.908+0000 7ffa3b7fe700 1 -- 172.21.15.77:0/596656967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa3c156120 msgr2=0x7ffa3c14a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.949 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.908+0000 7ffa3b7fe700 1 --2- 172.21.15.77:0/596656967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa3c156120 0x7ffa3c14a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.949 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.908+0000 7ffa3b7fe700 1 -- 172.21.15.77:0/596656967 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa240092a0 con 0x7ffa3c06d3f0 2022-04-03T19:44:28.950 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.907+0000 7f6feb114700 1 Processor -- start 2022-04-03T19:44:28.950 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.907+0000 7f6feb114700 1 -- start start 2022-04-03T19:44:28.950 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.908+0000 7fdb7389c700 1 Processor -- start 2022-04-03T19:44:28.951 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.909+0000 7ffa3b7fe700 1 --2- 172.21.15.77:0/596656967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa3c06d3f0 0x7ffa3c1579e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffa24011c30 tx=0x7ffa24017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:28.951 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.906+0000 7f23b5321700 1 -- 172.21.15.77:0/2217993779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23a80d58a0 msgr2=0x7f23a804f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.952 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.906+0000 7f23b5321700 1 --2- 172.21.15.77:0/2217993779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23a80d58a0 0x7f23a804f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.952 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.906+0000 7f23b5321700 1 -- 172.21.15.77:0/2217993779 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a00092a0 con 0x7f23a8049df0 2022-04-03T19:44:28.952 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.909+0000 7f1aa5d9b700 1 -- 172.21.15.77:0/1258161292 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1aa00dacd0 msgr2=0x7f1aa00ddc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.953 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.909+0000 7f1aa5d9b700 1 --2- 172.21.15.77:0/1258161292 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1aa00dacd0 0x7f1aa00ddc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.953 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.909+0000 7f1aa5d9b700 1 -- 172.21.15.77:0/1258161292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa000e2e0 msgr2=0x7f1aa00e35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.953 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.907+0000 7f23b5321700 1 --2- 172.21.15.77:0/2217993779 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23a8049df0 0x7f23a80084c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f23ac00c510 tx=0x7f23ac007d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:28.954 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.907+0000 7f239d7fa700 1 -- 172.21.15.77:0/2217993779 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac0052e0 con 0x7f23a8049df0 2022-04-03T19:44:28.954 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.907+0000 7f23b7585700 1 -- 172.21.15.77:0/2217993779 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a80d7070 con 0x7f23a8049df0 2022-04-03T19:44:28.954 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.908+0000 7f23b7585700 1 -- 172.21.15.77:0/2217993779 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a80d7630 con 0x7f23a8049df0 2022-04-03T19:44:28.955 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.909+0000 7f27feffd700 1 -- 172.21.15.77:0/74086498 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f8017480 con 0x7f28080692e0 2022-04-03T19:44:28.955 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.909+0000 7f1aa5d9b700 1 --2- 172.21.15.77:0/1258161292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa000e2e0 0x7f1aa00e35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.955 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.909+0000 7f1aa5d9b700 1 -- 172.21.15.77:0/1258161292 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9c0092a0 con 0x7f1aa0037cb0 2022-04-03T19:44:28.956 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.909+0000 7ffa337fe700 1 -- 172.21.15.77:0/596656967 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24003960 con 0x7ffa3c06d3f0 2022-04-03T19:44:28.956 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.909+0000 7ffa337fe700 1 -- 172.21.15.77:0/596656967 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa24003ac0 con 0x7ffa3c06d3f0 2022-04-03T19:44:28.957 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.909+0000 7ffa337fe700 1 -- 172.21.15.77:0/596656967 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa24003960 con 0x7ffa3c06d3f0 2022-04-03T19:44:28.957 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.910+0000 7ffa40c63700 1 -- 172.21.15.77:0/596656967 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa3c0c1300 con 0x7ffa3c06d3f0 2022-04-03T19:44:28.957 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.910+0000 7ffa40c63700 1 -- 172.21.15.77:0/596656967 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa3c0c18c0 con 0x7ffa3c06d3f0 2022-04-03T19:44:28.957 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.910+0000 7ffa3bfff700 1 --2- 172.21.15.77:0/596656967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa3c156120 0x7ffa3c14a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-04-03T19:44:28.958 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.909+0000 7f6feb114700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6fe4162230 0x7f6fe4162680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.958 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.909+0000 7f6feb114700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fe40c7200 0x7f6fe40c7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.958 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.909+0000 7f6feb114700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe415c620 con 0x7f6fe4087a30 2022-04-03T19:44:28.959 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.909+0000 7f6feb114700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe4162bc0 con 0x7f6fe4162230 2022-04-03T19:44:28.959 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.909+0000 7f6feb114700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe40691b0 con 0x7f6fe40c7200 2022-04-03T19:44:28.959 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.910+0000 7f6fe9911700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6fe4162230 0x7f6fe4162680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.959 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.910+0000 7f6fe9911700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6fe4162230 0x7f6fe4162680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42540/0 (socket says 172.21.15.77:42540) 2022-04-03T19:44:28.959 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.910+0000 7f6fe9911700 1 -- 172.21.15.77:0/3620378658 learned_addr learned my addr 172.21.15.77:0/3620378658 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:28.960 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.908+0000 7fdb7389c700 1 -- start start 2022-04-03T19:44:28.960 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.910+0000 7fdb7389c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdb6c0c7580 0x7fdb6c0c2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.960 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.910+0000 7fdb7389c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb6c06a000 0x7fdb6c153d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.960 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.910+0000 7fdb7389c700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb6c069460 con 0x7fdb6c06ab60 2022-04-03T19:44:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.910+0000 7fdb7389c700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb6c0c4d80 con 0x7fdb6c0c7580 2022-04-03T19:44:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.910+0000 7fdb7389c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c06c8e0 con 0x7fdb6c06a000 2022-04-03T19:44:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.909+0000 7f27feffd700 1 -- 172.21.15.77:0/74086498 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f80175e0 con 0x7f28080692e0 2022-04-03T19:44:28.962 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.909+0000 7f27feffd700 1 -- 172.21.15.77:0/74086498 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f8017980 con 0x7f28080692e0 2022-04-03T19:44:28.962 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.911+0000 7f280d646700 1 -- 172.21.15.77:0/74086498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28080a5490 con 0x7f28080692e0 2022-04-03T19:44:28.963 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.911+0000 7fdb71638700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdb6c0c7580 0x7fdb6c0c2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.963 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.911+0000 7fdb71638700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdb6c0c7580 0x7fdb6c0c2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37726/0 (socket says 172.21.15.77:37726) 2022-04-03T19:44:28.963 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.911+0000 7fdb71638700 1 -- 172.21.15.77:0/2642765641 learned_addr learned my addr 172.21.15.77:0/2642765641 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:28.963 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.911+0000 7fdb71e39700 1 --2- 172.21.15.77:0/2642765641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb6c06a000 0x7fdb6c153d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.964 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.911+0000 7f6fea913700 1 --2- 172.21.15.77:0/3620378658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fe40c7200 0x7f6fe40c7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.911+0000 7f6fea913700 1 -- 172.21.15.77:0/3620378658 >> v2:172.21.15.77:3300/0 conn(0x7f6fe4162230 msgr2=0x7f6fe4162680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:28.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.911+0000 7f6fea913700 1 --2- 172.21.15.77:0/3620378658 >> v2:172.21.15.77:3300/0 conn(0x7f6fe4162230 0x7f6fe4162680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.911+0000 7f6fea913700 1 -- 172.21.15.77:0/3620378658 >> v1:172.21.15.77:6789/0 conn(0x7f6fe4087a30 legacy=0x7f6fe415bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.911+0000 7f6fea913700 1 -- 172.21.15.77:0/3620378658 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe4069490 con 0x7f6fe40c7200 2022-04-03T19:44:28.966 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.911+0000 7fdb70e37700 1 --1- 172.21.15.77:0/2642765641 >> v1:172.21.15.77:6789/0 conn(0x7fdb6c06ab60 0x7fdb6c0698f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38310/0 (socket says 172.21.15.77:38310) 2022-04-03T19:44:28.966 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.911+0000 7f280d646700 1 -- 172.21.15.77:0/74086498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28080a59d0 con 0x7f28080692e0 2022-04-03T19:44:28.967 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.912+0000 7f239d7fa700 1 -- 172.21.15.77:0/2217993779 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ac0034c0 con 0x7f23a8049df0 2022-04-03T19:44:28.967 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.912+0000 7fdb71e39700 1 -- 172.21.15.77:0/2642765641 >> v2:172.21.15.186:3300/0 conn(0x7fdb6c0c7580 msgr2=0x7fdb6c0c2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:28.967 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.912+0000 7fdb71e39700 1 --2- 172.21.15.77:0/2642765641 >> v2:172.21.15.186:3300/0 conn(0x7fdb6c0c7580 0x7fdb6c0c2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.912+0000 7fdb71e39700 1 -- 172.21.15.77:0/2642765641 >> v1:172.21.15.77:6789/0 conn(0x7fdb6c06ab60 legacy=0x7fdb6c0698f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.912+0000 7fdb71e39700 1 -- 172.21.15.77:0/2642765641 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6c0c4200 con 0x7fdb6c06a000 2022-04-03T19:44:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.912+0000 7f6fea913700 1 --2- 172.21.15.77:0/3620378658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fe40c7200 0x7f6fe40c7640 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4009760 tx=0x7f6fd4009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2af2acd71898aa80 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:28.969 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.912+0000 7f239d7fa700 1 -- 172.21.15.77:0/2217993779 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac003880 con 0x7f23a8049df0 2022-04-03T19:44:28.969 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.913+0000 7f6fe9110700 1 -- 172.21.15.77:0/3620378658 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4011630 con 0x7f6fe40c7200 2022-04-03T19:44:28.970 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.913+0000 7f6fe9110700 1 -- 172.21.15.77:0/3620378658 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd4011790 con 0x7f6fe40c7200 2022-04-03T19:44:28.970 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.913+0000 7f6fe9110700 1 -- 172.21.15.77:0/3620378658 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4011b10 con 0x7f6fe40c7200 2022-04-03T19:44:28.970 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.913+0000 7fdb71e39700 1 --2- 172.21.15.77:0/2642765641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb6c06a000 0x7fdb6c153d40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdb64009e60 tx=0x7fdb64009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1b4f848f960ea2b server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:28.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.913+0000 7fdb5bfff700 1 -- 172.21.15.77:0/2642765641 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6400e070 con 0x7fdb6c06a000 2022-04-03T19:44:28.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.913+0000 7fdb5bfff700 1 -- 172.21.15.77:0/2642765641 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb6400bb20 con 0x7fdb6c06a000 2022-04-03T19:44:28.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.911+0000 7ffa40c63700 1 -- 172.21.15.77:0/596656967 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffa3c0c2860 con 0x7ffa3c06d3f0 2022-04-03T19:44:28.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.909+0000 7f1aa659c700 1 --2- 172.21.15.77:0/1258161292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1aa000e2e0 0x7f1aa00e35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:28.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.910+0000 7f1aa5d9b700 1 --2- 172.21.15.77:0/1258161292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1aa0037cb0 0x7f1aa00dd6d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8060620 tx=0x7f1aa804bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:28.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.913+0000 7f1a967fc700 1 -- 172.21.15.77:0/1258161292 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa806b070 con 0x7f1aa0037cb0 2022-04-03T19:44:28.973 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.913+0000 7f1a967fc700 1 -- 172.21.15.77:0/1258161292 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa806b1d0 con 0x7f1aa0037cb0 2022-04-03T19:44:28.973 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.913+0000 7f1a967fc700 1 -- 172.21.15.77:0/1258161292 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa806cea0 con 0x7f1aa0037cb0 2022-04-03T19:44:28.973 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.913+0000 7f4ba2405700 1 Processor -- start 2022-04-03T19:44:28.974 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.914+0000 7f1aa759e700 1 -- 172.21.15.77:0/1258161292 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa00dbbc0 con 0x7f1aa0037cb0 2022-04-03T19:44:28.974 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.914+0000 7f6feb114700 1 -- 172.21.15.77:0/3620378658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fe40c7200 msgr2=0x7f6fe40c7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.914+0000 7f6feb114700 1 --2- 172.21.15.77:0/3620378658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fe40c7200 0x7f6fe40c7640 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4009760 tx=0x7f6fd4009060 comp rx=0 tx=0).stop 2022-04-03T19:44:28.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.914+0000 7f1aa759e700 1 -- 172.21.15.77:0/1258161292 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa00dbe80 con 0x7f1aa0037cb0 2022-04-03T19:44:28.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.914+0000 7fdb7389c700 1 -- 172.21.15.77:0/2642765641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb6c06a000 msgr2=0x7fdb6c153d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.914+0000 7fdb7389c700 1 --2- 172.21.15.77:0/2642765641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb6c06a000 0x7fdb6c153d40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdb64009e60 tx=0x7fdb64009780 comp rx=0 tx=0).stop 2022-04-03T19:44:28.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.914+0000 7f6feb114700 1 -- 172.21.15.77:0/3620378658 shutdown_connections 2022-04-03T19:44:28.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.914+0000 7f6feb114700 1 --2- 172.21.15.77:0/3620378658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fe40c7200 0x7f6fe40c7640 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.914+0000 7f6feb114700 1 --2- 172.21.15.77:0/3620378658 >> v2:172.21.15.77:3300/0 conn(0x7f6fe4162230 0x7f6fe4162680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.914+0000 7f4ba2405700 1 -- start start 2022-04-03T19:44:28.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.914+0000 7f6feb114700 1 -- 172.21.15.77:0/3620378658 >> 172.21.15.77:0/3620378658 conn(0x7f6fe4066da0 msgr2=0x7f6fe40671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:28.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.914+0000 7fdb7389c700 1 -- 172.21.15.77:0/2642765641 shutdown_connections 2022-04-03T19:44:28.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.914+0000 7fdb7389c700 1 --2- 172.21.15.77:0/2642765641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb6c06a000 0x7fdb6c153d40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.914+0000 7fdb7389c700 1 --2- 172.21.15.77:0/2642765641 >> v2:172.21.15.186:3300/0 conn(0x7fdb6c0c7580 0x7fdb6c0c2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.914+0000 7fdb7389c700 1 -- 172.21.15.77:0/2642765641 >> 172.21.15.77:0/2642765641 conn(0x7fdb6c06d900 msgr2=0x7fdb6c0626d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:28.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.915+0000 7ffa337fe700 1 -- 172.21.15.77:0/596656967 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7ffa2400f050 con 0x7ffa3c06d3f0 2022-04-03T19:44:28.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.915+0000 7fdb7389c700 1 -- 172.21.15.77:0/2642765641 shutdown_connections 2022-04-03T19:44:28.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.915+0000 7f1aa759e700 1 -- 172.21.15.77:0/1258161292 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1aa0005710 con 0x7f1aa0037cb0 2022-04-03T19:44:28.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.915+0000 7f6feb114700 1 -- 172.21.15.77:0/3620378658 shutdown_connections 2022-04-03T19:44:28.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.915+0000 7f27fcff9700 1 -- 172.21.15.77:0/74086498 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2808064600 con 0x7f28080692e0 2022-04-03T19:44:28.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.915+0000 7f6feb114700 1 -- 172.21.15.77:0/3620378658 wait complete. 2022-04-03T19:44:28.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.916+0000 7f23b7585700 1 -- 172.21.15.77:0/2217993779 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f238c000fd0 con 0x7f23a8049df0 2022-04-03T19:44:28.984 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.916+0000 7f4ba2405700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4b9c0678b0 0x7f4b9c152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.984 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.916+0000 7f4ba2405700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4b9c15d450 0x7f4b9c153410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.984 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.916+0000 7f4ba2405700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b9c069d00 con 0x7f4b9c0ba320 2022-04-03T19:44:28.984 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.916+0000 7f4ba2405700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b9c069110 con 0x7f4b9c15d450 2022-04-03T19:44:28.984 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.916+0000 7f4ba2405700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b9c06e460 con 0x7f4b9c0678b0 2022-04-03T19:44:28.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.917+0000 7f4ba0c02700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4b9c0678b0 0x7f4b9c152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.917+0000 7f4ba0c02700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4b9c0678b0 0x7f4b9c152ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37728/0 (socket says 172.21.15.77:37728) 2022-04-03T19:44:28.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.917+0000 7f4ba0c02700 1 -- 172.21.15.77:0/3176635912 learned_addr learned my addr 172.21.15.77:0/3176635912 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:28.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.917+0000 7f4ba0c02700 1 -- 172.21.15.77:0/3176635912 >> v2:172.21.15.77:3300/0 conn(0x7f4b9c15d450 msgr2=0x7f4b9c153410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:28.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.917+0000 7f4ba0c02700 1 --2- 172.21.15.77:0/3176635912 >> v2:172.21.15.77:3300/0 conn(0x7f4b9c15d450 0x7f4b9c153410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.986 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.917+0000 7f4ba0c02700 1 -- 172.21.15.77:0/3176635912 >> v1:172.21.15.77:6789/0 conn(0x7f4b9c0ba320 legacy=0x7f4b9c15cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:28.986 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.917+0000 7f4ba0c02700 1 -- 172.21.15.77:0/3176635912 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9c15a2a0 con 0x7f4b9c0678b0 2022-04-03T19:44:28.986 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.915+0000 7fdb7389c700 1 -- 172.21.15.77:0/2642765641 wait complete. 2022-04-03T19:44:28.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.917+0000 7f1a967fc700 1 -- 172.21.15.77:0/1258161292 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f1aa8063a30 con 0x7f1aa0037cb0 2022-04-03T19:44:28.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.918+0000 7fdb7389c700 1 Processor -- start 2022-04-03T19:44:28.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.918+0000 7fa194efe700 1 Processor -- start 2022-04-03T19:44:28.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.918+0000 7f4ba0c02700 1 --2- 172.21.15.77:0/3176635912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b9c0678b0 0x7f4b9c152ed0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c06fb00 tx=0x7f4b94009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f5dd5ca1587d475 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:28.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.918+0000 7f4b8bfff700 1 -- 172.21.15.77:0/3176635912 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9400e070 con 0x7f4b9c0678b0 2022-04-03T19:44:28.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.918+0000 7f4b8bfff700 1 -- 172.21.15.77:0/3176635912 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9400baa0 con 0x7f4b9c0678b0 2022-04-03T19:44:28.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.918+0000 7f4b8bfff700 1 -- 172.21.15.77:0/3176635912 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b940092e0 con 0x7f4b9c0678b0 2022-04-03T19:44:28.990 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.918+0000 7f239d7fa700 1 -- 172.21.15.77:0/2217993779 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f23ac019030 con 0x7f23a8049df0 2022-04-03T19:44:28.990 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.918+0000 7f6feb114700 1 Processor -- start 2022-04-03T19:44:28.991 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.919+0000 7f1a967fc700 1 --2- 172.21.15.77:0/1258161292 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a84063de0 0x7f1a84066290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.991 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.919+0000 7f6feb114700 1 -- start start 2022-04-03T19:44:28.992 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.919+0000 7ffa337fe700 1 --2- 172.21.15.77:0/596656967 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ffa20063da0 0x7ffa20066250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.992 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.919+0000 7ffa3affd700 1 --2- 172.21.15.77:0/596656967 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ffa20063da0 0x7ffa20066250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.919+0000 7f1a967fc700 1 -- 172.21.15.77:0/1258161292 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f1aa80f7bc0 con 0x7f1aa0037cb0 2022-04-03T19:44:28.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.919+0000 7f1a967fc700 1 --2- 172.21.15.77:0/1258161292 >> [v2:172.21.15.77:6823/20252,v1:172.21.15.77:6825/20252] conn(0x7f1a8406c7b0 0x7f1a8406ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.919+0000 7f1a967fc700 1 -- 172.21.15.77:0/1258161292 --> [v2:172.21.15.77:6823/20252,v1:172.21.15.77:6825/20252] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a8406f270 con 0x7f1a8406c7b0 2022-04-03T19:44:28.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.919+0000 7f1a967fc700 1 -- 172.21.15.77:0/1258161292 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1aa80b1070 con 0x7f1aa0037cb0 2022-04-03T19:44:28.994 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.919+0000 7f4ba2405700 1 -- 172.21.15.77:0/3176635912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b9c0678b0 msgr2=0x7f4b9c152ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:28.994 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.919+0000 7f4ba2405700 1 --2- 172.21.15.77:0/3176635912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b9c0678b0 0x7f4b9c152ed0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c06fb00 tx=0x7f4b94009840 comp rx=0 tx=0).stop 2022-04-03T19:44:28.995 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.919+0000 7ffa337fe700 1 -- 172.21.15.77:0/596656967 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7ffa24055370 con 0x7ffa3c06d3f0 2022-04-03T19:44:28.995 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.918+0000 7fa194efe700 1 -- start start 2022-04-03T19:44:28.996 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.920+0000 7f4ba2405700 1 -- 172.21.15.77:0/3176635912 shutdown_connections 2022-04-03T19:44:28.996 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.920+0000 7f4ba2405700 1 --2- 172.21.15.77:0/3176635912 >> v2:172.21.15.77:3300/0 conn(0x7f4b9c15d450 0x7f4b9c153410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.996 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.920+0000 7f4ba2405700 1 --2- 172.21.15.77:0/3176635912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b9c0678b0 0x7f4b9c152ed0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:28.996 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.920+0000 7f4ba2405700 1 -- 172.21.15.77:0/3176635912 >> 172.21.15.77:0/3176635912 conn(0x7f4b9c066da0 msgr2=0x7f4b9c0671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:28.997 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.920+0000 7f1aa659c700 1 --2- 172.21.15.77:0/1258161292 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a84063de0 0x7f1a84066290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:28.997 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.920+0000 7ffa337fe700 1 --2- 172.21.15.77:0/596656967 >> [v2:172.21.15.77:6811/20250,v1:172.21.15.77:6814/20250] conn(0x7ffa2006c6f0 0x7ffa2006eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.998 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.920+0000 7ffa337fe700 1 -- 172.21.15.77:0/596656967 --> [v2:172.21.15.77:6811/20250,v1:172.21.15.77:6814/20250] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa2006f1b0 con 0x7ffa2006c6f0 2022-04-03T19:44:28.998 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.920+0000 7ffa337fe700 1 -- 172.21.15.77:0/596656967 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa24003960 con 0x7ffa3c06d3f0 2022-04-03T19:44:28.999 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.918+0000 7fdb7389c700 1 -- start start 2022-04-03T19:44:28.999 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.920+0000 7fdb7389c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb6c06a000 0x7fdb6c1527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.999 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.920+0000 7fdb7389c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb6c06ab60 0x7fdb6c0bdde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:28.999 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.920+0000 7fdb7389c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb6c0c7580 0x7fdb6c0be320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.000 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7fdb7389c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c0bf7e0 con 0x7fdb6c06ab60 2022-04-03T19:44:29.000 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7fdb7389c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c154880 con 0x7fdb6c0c7580 2022-04-03T19:44:29.000 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7fdb7389c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c153cb0 con 0x7fdb6c06a000 2022-04-03T19:44:29.000 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.920+0000 7fa194efe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1901538c0 0x7fa1900c3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.001 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7f6feb114700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fe4087a30 0x7f6fe40c0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.001 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.920+0000 7fa194efe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa190062f40 0x7fa19015b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7fa194efe700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa19015acb0 con 0x7fa190153c90 2022-04-03T19:44:29.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7fa194efe700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa190159750 con 0x7fa190062f40 2022-04-03T19:44:29.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7fa194efe700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1901599d0 con 0x7fa1901538c0 2022-04-03T19:44:29.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7f6feb114700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fe40c7200 0x7f6fe40c1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.920+0000 7f4ba2405700 1 -- 172.21.15.77:0/3176635912 shutdown_connections 2022-04-03T19:44:29.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7f4ba2405700 1 -- 172.21.15.77:0/3176635912 wait complete. 2022-04-03T19:44:29.004 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7ffa3bfff700 1 --2- 172.21.15.77:0/596656967 >> [v2:172.21.15.77:6811/20250,v1:172.21.15.77:6814/20250] conn(0x7ffa2006c6f0 0x7ffa2006eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.004 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.920+0000 7f27feffd700 1 -- 172.21.15.77:0/74086498 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f27f8017cf0 con 0x7f28080692e0 2022-04-03T19:44:29.005 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7f27feffd700 1 --2- 172.21.15.77:0/74086498 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f27e0063e30 0x7f27e00662e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.005 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7f6feb114700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fe4162230 0x7f6fe40bf650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.006 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7f6feb114700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe415b910 con 0x7f6fe4087a30 2022-04-03T19:44:29.006 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7f6feb114700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe4172050 con 0x7f6fe4162230 2022-04-03T19:44:29.006 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7f6feb114700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe41722f0 con 0x7f6fe40c7200 2022-04-03T19:44:29.006 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7f6fe9911700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fe40c7200 0x7f6fe40c1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7f6fe9911700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fe40c7200 0x7f6fe40c1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59840/0 (socket says 172.21.15.77:59840) 2022-04-03T19:44:29.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7f6fe9911700 1 -- 172.21.15.77:0/2842604191 learned_addr learned my addr 172.21.15.77:0/2842604191 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:29.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7f6fea112700 1 --2- 172.21.15.77:0/2842604191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fe4087a30 0x7f6fe40c0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7f6fea913700 1 --2- 172.21.15.77:0/2842604191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fe4162230 0x7f6fe40bf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.008 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.922+0000 7fdb70e37700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb6c06ab60 0x7fdb6c0bdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.008 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.922+0000 7fa18ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa190062f40 0x7fa19015b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.922+0000 7fa18f7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1901538c0 0x7fa1900c3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.922+0000 7fa18f7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1901538c0 0x7fa1900c3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59838/0 (socket says 172.21.15.77:59838) 2022-04-03T19:44:29.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.922+0000 7fa18f7fe700 1 -- 172.21.15.77:0/3302084906 learned_addr learned my addr 172.21.15.77:0/3302084906 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:29.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.922+0000 7ffa3bfff700 1 --2- 172.21.15.77:0/596656967 >> [v2:172.21.15.77:6811/20250,v1:172.21.15.77:6814/20250] conn(0x7ffa2006c6f0 0x7ffa2006eb00 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.922+0000 7fdb70e37700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb6c06ab60 0x7fdb6c0bdde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42546/0 (socket says 172.21.15.77:42546) 2022-04-03T19:44:29.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.922+0000 7fdb70e37700 1 -- 172.21.15.77:0/3984074504 learned_addr learned my addr 172.21.15.77:0/3984074504 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:29.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.922+0000 7fdb71638700 1 --2- 172.21.15.77:0/3984074504 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb6c06a000 0x7fdb6c1527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.922+0000 7f239d7fa700 1 --2- 172.21.15.77:0/2217993779 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2390063e60 0x7f2390066310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.922+0000 7f2806ffd700 1 --2- 172.21.15.77:0/74086498 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f27e0063e30 0x7f27e00662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.923+0000 7f6fea913700 1 -- 172.21.15.77:0/2842604191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fe40c7200 msgr2=0x7f6fe40c1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.923+0000 7f239d7fa700 1 -- 172.21.15.77:0/2217993779 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f23ac080410 con 0x7f23a8049df0 2022-04-03T19:44:29.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.923+0000 7f239d7fa700 1 --2- 172.21.15.77:0/2217993779 >> [v2:172.21.15.77:6804/20249,v1:172.21.15.77:6805/20249] conn(0x7f239006c830 0x7f239006ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.923+0000 7f6fea913700 1 --2- 172.21.15.77:0/2842604191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fe40c7200 0x7f6fe40c1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.923+0000 7f6fea913700 1 -- 172.21.15.77:0/2842604191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fe4087a30 msgr2=0x7f6fe40c0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.923+0000 7f6fea913700 1 --2- 172.21.15.77:0/2842604191 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fe4087a30 0x7f6fe40c0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.923+0000 7f6fea913700 1 -- 172.21.15.77:0/2842604191 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd40092a0 con 0x7f6fe4162230 2022-04-03T19:44:29.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.923+0000 7f6fe9911700 1 --2- 172.21.15.77:0/2842604191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fe40c7200 0x7f6fe40c1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:29.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.922+0000 7fdb71e39700 1 --2- 172.21.15.77:0/3984074504 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb6c0c7580 0x7fdb6c0be320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.923+0000 7fdb71638700 1 -- 172.21.15.77:0/3984074504 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb6c0c7580 msgr2=0x7fdb6c0be320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.923+0000 7fdb71638700 1 --2- 172.21.15.77:0/3984074504 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb6c0c7580 0x7fdb6c0be320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.923+0000 7fdb71638700 1 -- 172.21.15.77:0/3984074504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb6c06ab60 msgr2=0x7fdb6c0bdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.923+0000 7fdb71638700 1 --2- 172.21.15.77:0/3984074504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb6c06ab60 0x7fdb6c0bdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.923+0000 7fdb71638700 1 -- 172.21.15.77:0/3984074504 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb64009a20 con 0x7fdb6c06a000 2022-04-03T19:44:29.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.923+0000 7f6fea913700 1 --2- 172.21.15.77:0/2842604191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fe4162230 0x7f6fe40bf650 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4016000 tx=0x7f6fd4017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.923+0000 7f239d7fa700 1 -- 172.21.15.77:0/2217993779 --> [v2:172.21.15.77:6804/20249,v1:172.21.15.77:6805/20249] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f239006f2f0 con 0x7f239006c830 2022-04-03T19:44:29.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.923+0000 7f239d7fa700 1 -- 172.21.15.77:0/2217993779 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23ac080740 con 0x7f23a8049df0 2022-04-03T19:44:29.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.923+0000 7f4ba2405700 1 Processor -- start 2022-04-03T19:44:29.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.924+0000 7f23b5b22700 1 --2- 172.21.15.77:0/2217993779 >> [v2:172.21.15.77:6804/20249,v1:172.21.15.77:6805/20249] conn(0x7f239006c830 0x7f239006ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.920+0000 7f1aa6d9d700 1 --2- 172.21.15.77:0/1258161292 >> [v2:172.21.15.77:6823/20252,v1:172.21.15.77:6825/20252] conn(0x7f1a8406c7b0 0x7f1a8406ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.921+0000 7f1aa659c700 1 --2- 172.21.15.77:0/1258161292 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a84063de0 0x7f1a84066290 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009fd0 tx=0x7f1a9c017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.019 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.924+0000 7f1aa6d9d700 1 --2- 172.21.15.77:0/1258161292 >> [v2:172.21.15.77:6823/20252,v1:172.21.15.77:6825/20252] conn(0x7f1a8406c7b0 0x7f1a8406ebc0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.019 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.924+0000 7f4ba2405700 1 -- start start 2022-04-03T19:44:29.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.924+0000 7fa18f7fe700 1 -- 172.21.15.77:0/3302084906 >> v2:172.21.15.77:3300/0 conn(0x7fa190062f40 msgr2=0x7fa19015b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:29.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.924+0000 7fa18f7fe700 1 --2- 172.21.15.77:0/3302084906 >> v2:172.21.15.77:3300/0 conn(0x7fa190062f40 0x7fa19015b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.924+0000 7fa18f7fe700 1 -- 172.21.15.77:0/3302084906 >> v1:172.21.15.77:6789/0 conn(0x7fa190153c90 legacy=0x7fa190069070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:29.021 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.924+0000 7f6fd27fc700 1 -- 172.21.15.77:0/2842604191 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4011e80 con 0x7f6fe4162230 2022-04-03T19:44:29.021 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.924+0000 7fdb71638700 1 --2- 172.21.15.77:0/3984074504 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb6c06a000 0x7fdb6c1527b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdb68006fd0 tx=0x7fdb6800d040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.924+0000 7fa18f7fe700 1 -- 172.21.15.77:0/3302084906 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa190159cb0 con 0x7fa1901538c0 2022-04-03T19:44:29.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.924+0000 7f27feffd700 1 -- 172.21.15.77:0/74086498 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f27f8012070 con 0x7f28080692e0 2022-04-03T19:44:29.023 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.924+0000 7fa18ffff700 1 --2- 172.21.15.77:0/3302084906 >> v2:172.21.15.77:3300/0 conn(0x7fa190062f40 0x7fa19015b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:44:29.023 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.924+0000 7fa18f7fe700 1 --2- 172.21.15.77:0/3302084906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1901538c0 0x7fa1900c3e00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa180006fd0 tx=0x7fa1800090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df573ef2347200e server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.023 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.925+0000 7fdb597fa700 1 -- 172.21.15.77:0/3984074504 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb68010040 con 0x7fdb6c06a000 2022-04-03T19:44:29.024 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.925+0000 7fdb597fa700 1 -- 172.21.15.77:0/3984074504 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb68009e90 con 0x7fdb6c06a000 2022-04-03T19:44:29.024 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.924+0000 7ffa337fe700 1 -- 172.21.15.77:0/596656967 <== osd.3 v2:172.21.15.77:6811/20250 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7ffa2006f1b0 con 0x7ffa2006c6f0 2022-04-03T19:44:29.025 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.925+0000 7fdb597fa700 1 -- 172.21.15.77:0/3984074504 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6800ea60 con 0x7fdb6c06a000 2022-04-03T19:44:29.025 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.925+0000 7fa18effd700 1 -- 172.21.15.77:0/3302084906 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa180004030 con 0x7fa1901538c0 2022-04-03T19:44:29.025 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.925+0000 7fa18effd700 1 -- 172.21.15.77:0/3302084906 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa180004190 con 0x7fa1901538c0 2022-04-03T19:44:29.026 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.925+0000 7fa18effd700 1 -- 172.21.15.77:0/3302084906 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa180011630 con 0x7fa1901538c0 2022-04-03T19:44:29.026 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.925+0000 7fdb7389c700 1 -- 172.21.15.77:0/3984074504 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb6c153f50 con 0x7fdb6c06a000 2022-04-03T19:44:29.026 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.925+0000 7f23b5b22700 1 --2- 172.21.15.77:0/2217993779 >> [v2:172.21.15.77:6804/20249,v1:172.21.15.77:6805/20249] conn(0x7f239006c830 0x7f239006ec40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.027 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.925+0000 7f2806ffd700 1 --2- 172.21.15.77:0/74086498 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f27e0063e30 0x7f27e00662e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f27f0001440 tx=0x7f27f0007980 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.027 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.924+0000 7f6fd27fc700 1 -- 172.21.15.77:0/2842604191 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd40044f0 con 0x7f6fe4162230 2022-04-03T19:44:29.028 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.924+0000 7f6fd27fc700 1 -- 172.21.15.77:0/2842604191 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4003e50 con 0x7f6fe4162230 2022-04-03T19:44:29.028 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.925+0000 7fdb7389c700 1 -- 172.21.15.77:0/3984074504 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb6c154510 con 0x7fdb6c06a000 2022-04-03T19:44:29.029 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.925+0000 7f6feb114700 1 -- 172.21.15.77:0/2842604191 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe40be5b0 con 0x7f6fe4162230 2022-04-03T19:44:29.029 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.925+0000 7f4ba2405700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4b9c0678b0 0x7f4b9c0c5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.029 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.926+0000 7f6feb114700 1 -- 172.21.15.77:0/2842604191 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe40beb70 con 0x7f6fe4162230 2022-04-03T19:44:29.030 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.926+0000 7f1a967fc700 1 -- 172.21.15.77:0/1258161292 <== osd.0 v2:172.21.15.77:6823/20252 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7f1a8406f270 con 0x7f1a8406c7b0 2022-04-03T19:44:29.031 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.926+0000 7f23b4b20700 1 --2- 172.21.15.77:0/2217993779 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2390063e60 0x7f2390066310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.031 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.926+0000 7ffa3affd700 1 --2- 172.21.15.77:0/596656967 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ffa20063da0 0x7ffa20066250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c009fd0 tx=0x7ffa2c00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.032 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.926+0000 7fa194efe700 1 -- 172.21.15.77:0/3302084906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1901538c0 msgr2=0x7fa1900c3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.032 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.926+0000 7fa194efe700 1 --2- 172.21.15.77:0/3302084906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1901538c0 0x7fa1900c3e00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa180006fd0 tx=0x7fa1800090f0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.032 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.926+0000 7f4ba2405700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b9c0ba320 0x7f4b9c0c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.926+0000 7f4ba2405700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b9c15d450 0x7f4b9c16d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.926+0000 7fa194efe700 1 -- 172.21.15.77:0/3302084906 shutdown_connections 2022-04-03T19:44:29.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.926+0000 7fa194efe700 1 --2- 172.21.15.77:0/3302084906 >> v2:172.21.15.77:3300/0 conn(0x7fa190062f40 0x7fa19015b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.926+0000 7fa194efe700 1 --2- 172.21.15.77:0/3302084906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1901538c0 0x7fa1900c3e00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.926+0000 7fa194efe700 1 -- 172.21.15.77:0/3302084906 >> 172.21.15.77:0/3302084906 conn(0x7fa190067960 msgr2=0x7fa190067d60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:29.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.927+0000 7fdb52ffd700 1 -- 172.21.15.77:0/3984074504 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdb6c04b890 con 0x7fdb6c06a000 2022-04-03T19:44:29.035 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.927+0000 7f4ba0c02700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b9c0ba320 0x7f4b9c0c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.035 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.927+0000 7f4ba2405700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c0c6990 con 0x7f4b9c15d450 2022-04-03T19:44:29.035 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.927+0000 7fa194efe700 1 -- 172.21.15.77:0/3302084906 shutdown_connections 2022-04-03T19:44:29.036 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.927+0000 7f239d7fa700 1 -- 172.21.15.77:0/2217993779 <== osd.1 v2:172.21.15.77:6804/20249 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7f239006f2f0 con 0x7f239006c830 2022-04-03T19:44:29.036 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.927+0000 7f27feffd700 1 --2- 172.21.15.77:0/74086498 >> [v2:172.21.15.77:6800/20248,v1:172.21.15.77:6801/20248] conn(0x7f27e006c800 0x7f27e006ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.927+0000 7f27feffd700 1 -- 172.21.15.77:0/74086498 --> [v2:172.21.15.77:6800/20248,v1:172.21.15.77:6801/20248] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27e006f2c0 con 0x7f27e006c800 2022-04-03T19:44:29.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.927+0000 7f27feffd700 1 -- 172.21.15.77:0/74086498 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27f8082be0 con 0x7f28080692e0 2022-04-03T19:44:29.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.927+0000 7f4ba1c04700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b9c15d450 0x7f4b9c16d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.038 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.927+0000 7f4ba1c04700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b9c15d450 0x7f4b9c16d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42552/0 (socket says 172.21.15.77:42552) 2022-04-03T19:44:29.038 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.927+0000 7f4ba1c04700 1 -- 172.21.15.77:0/964205966 learned_addr learned my addr 172.21.15.77:0/964205966 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:29.038 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.927+0000 7f4ba2405700 1 -- 172.21.15.77:0/964205966 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c0c3c40 con 0x7f4b9c0ba320 2022-04-03T19:44:29.038 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.927+0000 7f4ba2405700 1 -- 172.21.15.77:0/964205966 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c0bbef0 con 0x7f4b9c0678b0 2022-04-03T19:44:29.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.928+0000 7f23b4b20700 1 --2- 172.21.15.77:0/2217993779 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2390063e60 0x7f2390066310 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f23b002af70 tx=0x7f23b006e040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.928+0000 7f28077fe700 1 --2- 172.21.15.77:0/74086498 >> [v2:172.21.15.77:6800/20248,v1:172.21.15.77:6801/20248] conn(0x7f27e006c800 0x7f27e006ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.927+0000 7f6feb114700 1 -- 172.21.15.77:0/2842604191 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6fe40be3a0 con 0x7f6fe4162230 2022-04-03T19:44:29.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.929+0000 7f28077fe700 1 --2- 172.21.15.77:0/74086498 >> [v2:172.21.15.77:6800/20248,v1:172.21.15.77:6801/20248] conn(0x7f27e006c800 0x7f27e006ec10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.041 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.930+0000 7f6fd27fc700 1 -- 172.21.15.77:0/2842604191 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f6fd400f050 con 0x7f6fe4162230 2022-04-03T19:44:29.041 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.929+0000 7fdb597fa700 1 -- 172.21.15.77:0/3984074504 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fdb68016410 con 0x7fdb6c06a000 2022-04-03T19:44:29.042 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.928+0000 7f4ba1c04700 1 -- 172.21.15.77:0/964205966 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4b9c0678b0 msgr2=0x7f4b9c0c5b30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2022-04-03T19:44:29.042 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.928+0000 7f4ba1c04700 1 --2- 172.21.15.77:0/964205966 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4b9c0678b0 0x7f4b9c0c5b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.042 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.929+0000 7f4ba1c04700 1 -- 172.21.15.77:0/964205966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b9c0ba320 msgr2=0x7f4b9c0c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.042 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.929+0000 7f4ba1c04700 1 --2- 172.21.15.77:0/964205966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b9c0ba320 0x7f4b9c0c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.929+0000 7f4ba1c04700 1 -- 172.21.15.77:0/964205966 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b94009ae0 con 0x7f4b9c15d450 2022-04-03T19:44:29.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.929+0000 7f4ba1c04700 1 --2- 172.21.15.77:0/964205966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b9c15d450 0x7f4b9c16d0b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c00b810 tx=0x7f4b8c006f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.930+0000 7f4b897fa700 1 -- 172.21.15.77:0/964205966 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c017400 con 0x7f4b9c15d450 2022-04-03T19:44:29.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.930+0000 7f4b897fa700 1 -- 172.21.15.77:0/964205966 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8c017580 con 0x7f4b9c15d450 2022-04-03T19:44:29.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.931+0000 7f4ba2405700 1 -- 172.21.15.77:0/964205966 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b9c0bc1d0 con 0x7f4b9c15d450 2022-04-03T19:44:29.044 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.927+0000 7fa194efe700 1 -- 172.21.15.77:0/3302084906 wait complete. 2022-04-03T19:44:29.044 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.931+0000 7f27feffd700 1 -- 172.21.15.77:0/74086498 <== osd.2 v2:172.21.15.77:6800/20248 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7f27e006f2c0 con 0x7f27e006c800 2022-04-03T19:44:29.045 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.931+0000 7f6fd27fc700 1 --2- 172.21.15.77:0/2842604191 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6fc4063e30 0x7f6fc40662e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.045 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.931+0000 7f6fd27fc700 1 -- 172.21.15.77:0/2842604191 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f6fd4081e70 con 0x7f6fe4162230 2022-04-03T19:44:29.046 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.931+0000 7fdb597fa700 1 --2- 172.21.15.77:0/3984074504 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdb4c063d10 0x7fdb4c0661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.046 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.931+0000 7fdb597fa700 1 -- 172.21.15.77:0/3984074504 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fdb68080c30 con 0x7fdb6c06a000 2022-04-03T19:44:29.046 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.931+0000 7fdb597fa700 1 --2- 172.21.15.77:0/3984074504 >> [v2:172.21.15.186:6808/25541,v1:172.21.15.186:6809/25541] conn(0x7fdb4c06c6e0 0x7fdb4c06eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.046 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.931+0000 7fdb597fa700 1 -- 172.21.15.77:0/3984074504 --> [v2:172.21.15.186:6808/25541,v1:172.21.15.186:6809/25541] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb4c06f1a0 con 0x7fdb4c06c6e0 2022-04-03T19:44:29.047 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.931+0000 7fdb597fa700 1 -- 172.21.15.77:0/3984074504 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb68086230 con 0x7fdb6c06a000 2022-04-03T19:44:29.047 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.931+0000 7f6fd27fc700 1 --2- 172.21.15.77:0/2842604191 >> [v2:172.21.15.186:6820/25551,v1:172.21.15.186:6822/25551] conn(0x7f6fc406c780 0x7f6fc406eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.047 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.932+0000 7f6fd27fc700 1 -- 172.21.15.77:0/2842604191 --> [v2:172.21.15.186:6820/25551,v1:172.21.15.186:6822/25551] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6fc406f240 con 0x7f6fc406c780 2022-04-03T19:44:29.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.932+0000 7f6fd27fc700 1 -- 172.21.15.77:0/2842604191 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fd40509e0 con 0x7f6fe4162230 2022-04-03T19:44:29.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.932+0000 7f4ba2405700 1 -- 172.21.15.77:0/964205966 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b9c169ed0 con 0x7f4b9c15d450 2022-04-03T19:44:29.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.932+0000 7f6fea112700 1 --2- 172.21.15.77:0/2842604191 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6fc4063e30 0x7f6fc40662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.932+0000 7f6fe9911700 1 --2- 172.21.15.77:0/2842604191 >> [v2:172.21.15.186:6820/25551,v1:172.21.15.186:6822/25551] conn(0x7f6fc406c780 0x7f6fc406eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.932+0000 7f4b897fa700 1 -- 172.21.15.77:0/964205966 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c017940 con 0x7f4b9c15d450 2022-04-03T19:44:29.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.933+0000 7fdb71e39700 1 --2- 172.21.15.77:0/3984074504 >> [v2:172.21.15.186:6808/25541,v1:172.21.15.186:6809/25541] conn(0x7fdb4c06c6e0 0x7fdb4c06eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.934+0000 7f6fe9911700 1 --2- 172.21.15.77:0/2842604191 >> [v2:172.21.15.186:6820/25551,v1:172.21.15.186:6822/25551] conn(0x7f6fc406c780 0x7f6fc406eb90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.930+0000 7fa194efe700 1 Processor -- start 2022-04-03T19:44:29.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.931+0000 7fa194efe700 1 -- start start 2022-04-03T19:44:29.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.932+0000 7fa194efe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa190062f40 0x7fa19016a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.932+0000 7fa194efe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1901538c0 0x7fa19015c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.932+0000 7fa194efe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa190153c90 0x7fa1900c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.932+0000 7fa194efe700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa19016a220 con 0x7fa190153c90 2022-04-03T19:44:29.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.932+0000 7fa194efe700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa19015c8e0 con 0x7fa1901538c0 2022-04-03T19:44:29.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.933+0000 7fa194efe700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa19015cbc0 con 0x7fa190062f40 2022-04-03T19:44:29.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.933+0000 7fa18ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa190153c90 0x7fa1900c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.933+0000 7fa18f7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa190062f40 0x7fa19016a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.933+0000 7fa18ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa190153c90 0x7fa1900c4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42554/0 (socket says 172.21.15.77:42554) 2022-04-03T19:44:29.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.933+0000 7fa18ffff700 1 -- 172.21.15.77:0/3870356874 learned_addr learned my addr 172.21.15.77:0/3870356874 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:29.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.934+0000 7fa18f7fe700 1 -- 172.21.15.77:0/3870356874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1901538c0 msgr2=0x7fa19015c2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-04-03T19:44:29.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.934+0000 7fa186ffd700 1 --2- 172.21.15.77:0/3870356874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1901538c0 0x7fa19015c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.934+0000 7fa18f7fe700 1 --2- 172.21.15.77:0/3870356874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1901538c0 0x7fa19015c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.055 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.934+0000 7fa18f7fe700 1 -- 172.21.15.77:0/3870356874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa190153c90 msgr2=0x7fa1900c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.055 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.934+0000 7fa18f7fe700 1 --2- 172.21.15.77:0/3870356874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa190153c90 0x7fa1900c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.055 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.934+0000 7fa18f7fe700 1 -- 172.21.15.77:0/3870356874 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa180009370 con 0x7fa190062f40 2022-04-03T19:44:29.055 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.934+0000 7fa18ffff700 1 --2- 172.21.15.77:0/3870356874 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa190153c90 0x7fa1900c4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:29.056 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.934+0000 7f6fea112700 1 --2- 172.21.15.77:0/2842604191 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6fc4063e30 0x7f6fc40662e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0001840 tx=0x7f6fe0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.056 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.935+0000 7f6fd27fc700 1 -- 172.21.15.77:0/2842604191 <== osd.4 v2:172.21.15.186:6820/25551 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7f6fc406f240 con 0x7f6fc406c780 2022-04-03T19:44:29.057 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.935+0000 7fdb70e37700 1 --2- 172.21.15.77:0/3984074504 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdb4c063d10 0x7fdb4c0661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.057 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.935+0000 7fdb71e39700 1 --2- 172.21.15.77:0/3984074504 >> [v2:172.21.15.186:6808/25541,v1:172.21.15.186:6809/25541] conn(0x7fdb4c06c6e0 0x7fdb4c06eaf0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.057 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.935+0000 7f4ba2405700 1 -- 172.21.15.77:0/964205966 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4b78000fd0 con 0x7f4b9c15d450 2022-04-03T19:44:29.058 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.935+0000 7fa18f7fe700 1 --2- 172.21.15.77:0/3870356874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa190062f40 0x7fa19016a6b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa180009340 tx=0x7fa180016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.058 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.936+0000 7fa187fff700 1 -- 172.21.15.77:0/3870356874 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1800037c0 con 0x7fa190062f40 2022-04-03T19:44:29.059 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.936+0000 7fa194efe700 1 -- 172.21.15.77:0/3870356874 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa190169bd0 con 0x7fa190062f40 2022-04-03T19:44:29.059 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.936+0000 7fa194efe700 1 -- 172.21.15.77:0/3870356874 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa19015cea0 con 0x7fa190062f40 2022-04-03T19:44:29.059 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.937+0000 7fdb597fa700 1 -- 172.21.15.77:0/3984074504 <== osd.5 v2:172.21.15.186:6808/25541 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7fdb4c06f1a0 con 0x7fdb4c06c6e0 2022-04-03T19:44:29.060 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.936+0000 7f4b897fa700 1 -- 172.21.15.77:0/964205966 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f4b8c00d040 con 0x7f4b9c15d450 2022-04-03T19:44:29.060 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.938+0000 7f4b897fa700 1 --2- 172.21.15.77:0/964205966 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4b7c063db0 0x7f4b7c066260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.060 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.939+0000 7f4b897fa700 1 -- 172.21.15.77:0/964205966 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f4b8c0802a0 con 0x7f4b9c15d450 2022-04-03T19:44:29.060 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.939+0000 7f4b897fa700 1 --2- 172.21.15.77:0/964205966 >> [v2:172.21.15.186:6800/25542,v1:172.21.15.186:6801/25542] conn(0x7f4b7c06c700 0x7f4b7c06eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.061 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.939+0000 7f4b897fa700 1 -- 172.21.15.77:0/964205966 --> [v2:172.21.15.186:6800/25542,v1:172.21.15.186:6801/25542] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b7c06f1c0 con 0x7f4b7c06c700 2022-04-03T19:44:29.061 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.940+0000 7f4b897fa700 1 -- 172.21.15.77:0/964205966 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b8c04ef00 con 0x7f4b9c15d450 2022-04-03T19:44:29.061 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.937+0000 7fa187fff700 1 -- 172.21.15.77:0/3870356874 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa180003960 con 0x7fa190062f40 2022-04-03T19:44:29.062 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.939+0000 7fa187fff700 1 -- 172.21.15.77:0/3870356874 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa180003d20 con 0x7fa190062f40 2022-04-03T19:44:29.062 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.940+0000 7fa194efe700 1 -- 172.21.15.77:0/3870356874 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa168000fd0 con 0x7fa190062f40 2022-04-03T19:44:29.062 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.940+0000 7f4ba0c02700 1 --2- 172.21.15.77:0/964205966 >> [v2:172.21.15.186:6800/25542,v1:172.21.15.186:6801/25542] conn(0x7f4b7c06c700 0x7f4b7c06eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.063 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.940+0000 7fa187fff700 1 -- 172.21.15.77:0/3870356874 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fa18002b030 con 0x7fa190062f40 2022-04-03T19:44:29.063 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.941+0000 7f4ba1403700 1 --2- 172.21.15.77:0/964205966 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4b7c063db0 0x7f4b7c066260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.064 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.942+0000 7f4ba1403700 1 --2- 172.21.15.77:0/964205966 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4b7c063db0 0x7f4b7c066260 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4b98001440 tx=0x7f4b98007980 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.065 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.942+0000 7fa187fff700 1 --2- 172.21.15.77:0/3870356874 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa174063db0 0x7fa174066260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.066 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.943+0000 7fa187fff700 1 -- 172.21.15.77:0/3870356874 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fa180082260 con 0x7fa190062f40 2022-04-03T19:44:29.066 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.943+0000 7fa186ffd700 1 --2- 172.21.15.77:0/3870356874 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa174063db0 0x7fa174066260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.066 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.943+0000 7fa187fff700 1 --2- 172.21.15.77:0/3870356874 >> [v2:172.21.15.186:6816/25572,v1:172.21.15.186:6817/25572] conn(0x7fa17406c610 0x7fa17406ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.066 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.944+0000 7fa186ffd700 1 --2- 172.21.15.77:0/3870356874 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa174063db0 0x7fa174066260 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa190154060 tx=0x7fa178009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.067 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.944+0000 7fa187fff700 1 -- 172.21.15.77:0/3870356874 --> [v2:172.21.15.186:6816/25572,v1:172.21.15.186:6817/25572] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa17406f0d0 con 0x7fa17406c610 2022-04-03T19:44:29.067 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.945+0000 7fa18ffff700 1 --2- 172.21.15.77:0/3870356874 >> [v2:172.21.15.186:6816/25572,v1:172.21.15.186:6817/25572] conn(0x7fa17406c610 0x7fa17406ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.067 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.945+0000 7fa187fff700 1 -- 172.21.15.77:0/3870356874 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa180084800 con 0x7fa190062f40 2022-04-03T19:44:29.068 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.946+0000 7f4ba0c02700 1 --2- 172.21.15.77:0/964205966 >> [v2:172.21.15.186:6800/25542,v1:172.21.15.186:6801/25542] conn(0x7f4b7c06c700 0x7f4b7c06eb10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.068 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.945+0000 7fa18ffff700 1 --2- 172.21.15.77:0/3870356874 >> [v2:172.21.15.186:6816/25572,v1:172.21.15.186:6817/25572] conn(0x7fa17406c610 0x7fa17406ea20 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.069 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.946+0000 7fa187fff700 1 -- 172.21.15.77:0/3870356874 <== osd.7 v2:172.21.15.186:6816/25572 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7fa17406f0d0 con 0x7fa17406c610 2022-04-03T19:44:29.069 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.942+0000 7fdb70e37700 1 --2- 172.21.15.77:0/3984074504 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdb4c063d10 0x7fdb4c0661c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c0bfae0 tx=0x7fdb5c007e10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.070 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.951+0000 7f4b897fa700 1 -- 172.21.15.77:0/964205966 <== osd.6 v2:172.21.15.186:6800/25542 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24062 (crc 0 0 0) 0x7f4b7c06f1c0 con 0x7f4b7c06c700 2022-04-03T19:44:29.070 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.959+0000 7f23b7585700 1 -- 172.21.15.77:0/2217993779 --> [v2:172.21.15.77:6804/20249,v1:172.21.15.77:6805/20249] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f238c002bb0 con 0x7f239006c830 2022-04-03T19:44:29.071 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.959+0000 7f239d7fa700 1 -- 172.21.15.77:0/2217993779 <== osd.1 v2:172.21.15.77:6804/20249 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f238c002bb0 con 0x7f239006c830 2022-04-03T19:44:29.071 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.961+0000 7f2396ffd700 1 -- 172.21.15.77:0/2217993779 >> [v2:172.21.15.77:6804/20249,v1:172.21.15.77:6805/20249] conn(0x7f239006c830 msgr2=0x7f239006ec40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.072 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.961+0000 7f2396ffd700 1 --2- 172.21.15.77:0/2217993779 >> [v2:172.21.15.77:6804/20249,v1:172.21.15.77:6805/20249] conn(0x7f239006c830 0x7f239006ec40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.072 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.961+0000 7f1aa759e700 1 -- 172.21.15.77:0/1258161292 --> [v2:172.21.15.77:6823/20252,v1:172.21.15.77:6825/20252] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1aa0006b20 con 0x7f1a8406c7b0 2022-04-03T19:44:29.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.962+0000 7f2396ffd700 1 -- 172.21.15.77:0/2217993779 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2390063e60 msgr2=0x7f2390066310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.962+0000 7f2396ffd700 1 --2- 172.21.15.77:0/2217993779 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2390063e60 0x7f2390066310 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f23b002af70 tx=0x7f23b006e040 comp rx=0 tx=0).stop 2022-04-03T19:44:29.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.962+0000 7f2396ffd700 1 -- 172.21.15.77:0/2217993779 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23a8049df0 msgr2=0x7f23a80084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.074 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.962+0000 7f2396ffd700 1 --2- 172.21.15.77:0/2217993779 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23a8049df0 0x7f23a80084c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f23ac00c510 tx=0x7f23ac007d60 comp rx=0 tx=0).stop 2022-04-03T19:44:29.074 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.962+0000 7f23b5b22700 1 -- 172.21.15.77:0/2217993779 reap_dead start 2022-04-03T19:44:29.075 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.962+0000 7f2396ffd700 1 -- 172.21.15.77:0/2217993779 shutdown_connections 2022-04-03T19:44:29.075 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.963+0000 7f2396ffd700 1 -- 172.21.15.77:0/2217993779 >> 172.21.15.77:0/2217993779 conn(0x7f23a8051fe0 msgr2=0x7f23a80d4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:29.075 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.962+0000 7f1a967fc700 1 -- 172.21.15.77:0/1258161292 <== osd.0 v2:172.21.15.77:6823/20252 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1aa0006b20 con 0x7f1a8406c7b0 2022-04-03T19:44:29.076 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.963+0000 7ffa40c63700 1 -- 172.21.15.77:0/596656967 --> [v2:172.21.15.77:6811/20250,v1:172.21.15.77:6814/20250] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ffa3c04b050 con 0x7ffa2006c6f0 2022-04-03T19:44:29.076 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.964+0000 7f2396ffd700 1 -- 172.21.15.77:0/2217993779 shutdown_connections 2022-04-03T19:44:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.964+0000 7f2396ffd700 1 -- 172.21.15.77:0/2217993779 wait complete. 2022-04-03T19:44:29.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.964+0000 7ffa337fe700 1 -- 172.21.15.77:0/596656967 <== osd.3 v2:172.21.15.77:6811/20250 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ffa3c04b050 con 0x7ffa2006c6f0 2022-04-03T19:44:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.964+0000 7f1a83fff700 1 -- 172.21.15.77:0/1258161292 >> [v2:172.21.15.77:6823/20252,v1:172.21.15.77:6825/20252] conn(0x7f1a8406c7b0 msgr2=0x7f1a8406ebc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.964+0000 7f1a83fff700 1 --2- 172.21.15.77:0/1258161292 >> [v2:172.21.15.77:6823/20252,v1:172.21.15.77:6825/20252] conn(0x7f1a8406c7b0 0x7f1a8406ebc0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.964+0000 7f1a83fff700 1 -- 172.21.15.77:0/1258161292 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a84063de0 msgr2=0x7f1a84066290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.964+0000 7f1a83fff700 1 --2- 172.21.15.77:0/1258161292 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a84063de0 0x7f1a84066290 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c009fd0 tx=0x7f1a9c017000 comp rx=0 tx=0).stop 2022-04-03T19:44:29.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.964+0000 7f1a83fff700 1 -- 172.21.15.77:0/1258161292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1aa0037cb0 msgr2=0x7f1aa00dd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.964+0000 7f1a83fff700 1 --2- 172.21.15.77:0/1258161292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1aa0037cb0 0x7f1aa00dd6d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8060620 tx=0x7f1aa804bea0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.965+0000 7f1aa6d9d700 1 -- 172.21.15.77:0/1258161292 reap_dead start 2022-04-03T19:44:29.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.965+0000 7f1a83fff700 1 -- 172.21.15.77:0/1258161292 shutdown_connections 2022-04-03T19:44:29.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.965+0000 7f1a83fff700 1 -- 172.21.15.77:0/1258161292 >> 172.21.15.77:0/1258161292 conn(0x7f1aa0054290 msgr2=0x7f1aa0007450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:29.081 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.965+0000 7ffa40c63700 1 -- 172.21.15.77:0/596656967 >> [v2:172.21.15.77:6811/20250,v1:172.21.15.77:6814/20250] conn(0x7ffa2006c6f0 msgr2=0x7ffa2006eb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.966+0000 7f1a83fff700 1 -- 172.21.15.77:0/1258161292 shutdown_connections 2022-04-03T19:44:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.966+0000 7f27fcff9700 1 -- 172.21.15.77:0/74086498 --> [v2:172.21.15.77:6800/20248,v1:172.21.15.77:6801/20248] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f280806a370 con 0x7f27e006c800 2022-04-03T19:44:29.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.965+0000 7ffa40c63700 1 --2- 172.21.15.77:0/596656967 >> [v2:172.21.15.77:6811/20250,v1:172.21.15.77:6814/20250] conn(0x7ffa2006c6f0 0x7ffa2006eb00 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.083 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.966+0000 7f1a83fff700 1 -- 172.21.15.77:0/1258161292 wait complete. 2022-04-03T19:44:29.084 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.966+0000 7ffa40c63700 1 -- 172.21.15.77:0/596656967 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ffa20063da0 msgr2=0x7ffa20066250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.084 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.966+0000 7ffa40c63700 1 --2- 172.21.15.77:0/596656967 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ffa20063da0 0x7ffa20066250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c009fd0 tx=0x7ffa2c00b040 comp rx=0 tx=0).stop 2022-04-03T19:44:29.084 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.966+0000 7ffa40c63700 1 -- 172.21.15.77:0/596656967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa3c06d3f0 msgr2=0x7ffa3c1579e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.084 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.966+0000 7ffa40c63700 1 --2- 172.21.15.77:0/596656967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa3c06d3f0 0x7ffa3c1579e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffa24011c30 tx=0x7ffa24017000 comp rx=0 tx=0).stop 2022-04-03T19:44:29.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.966+0000 7ffa3bfff700 1 -- 172.21.15.77:0/596656967 reap_dead start 2022-04-03T19:44:29.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.967+0000 7f27feffd700 1 -- 172.21.15.77:0/74086498 <== osd.2 v2:172.21.15.77:6800/20248 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f280806a370 con 0x7f27e006c800 2022-04-03T19:44:29.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.966+0000 7ffa40c63700 1 -- 172.21.15.77:0/596656967 shutdown_connections 2022-04-03T19:44:29.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.967+0000 7ffa40c63700 1 -- 172.21.15.77:0/596656967 >> 172.21.15.77:0/596656967 conn(0x7ffa3c0cb720 msgr2=0x7ffa3c0c9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:29.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.967+0000 7f27fcff9700 1 -- 172.21.15.77:0/74086498 >> [v2:172.21.15.77:6800/20248,v1:172.21.15.77:6801/20248] conn(0x7f27e006c800 msgr2=0x7f27e006ec10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.967+0000 7f27fcff9700 1 --2- 172.21.15.77:0/74086498 >> [v2:172.21.15.77:6800/20248,v1:172.21.15.77:6801/20248] conn(0x7f27e006c800 0x7f27e006ec10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.968+0000 7f27fcff9700 1 -- 172.21.15.77:0/74086498 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f27e0063e30 msgr2=0x7f27e00662e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.968+0000 7f27fcff9700 1 --2- 172.21.15.77:0/74086498 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f27e0063e30 0x7f27e00662e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f27f0001440 tx=0x7f27f0007980 comp rx=0 tx=0).stop 2022-04-03T19:44:29.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.968+0000 7f27fcff9700 1 -- 172.21.15.77:0/74086498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28080692e0 msgr2=0x7f2808154700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.968+0000 7f27fcff9700 1 --2- 172.21.15.77:0/74086498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f28080692e0 0x7f2808154700 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f27f800b810 tx=0x7f27f8009060 comp rx=0 tx=0).stop 2022-04-03T19:44:29.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.968+0000 7f28077fe700 1 -- 172.21.15.77:0/74086498 reap_dead start 2022-04-03T19:44:29.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.968+0000 7f27fcff9700 1 -- 172.21.15.77:0/74086498 shutdown_connections 2022-04-03T19:44:29.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.968+0000 7f27fcff9700 1 -- 172.21.15.77:0/74086498 >> 172.21.15.77:0/74086498 conn(0x7f28080c9fd0 msgr2=0x7f28080c2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:29.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.968+0000 7fdb52ffd700 1 -- 172.21.15.77:0/3984074504 --> [v2:172.21.15.186:6808/25541,v1:172.21.15.186:6809/25541] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdb6c0c1950 con 0x7fdb4c06c6e0 2022-04-03T19:44:29.090 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.969+0000 7f6feb114700 1 -- 172.21.15.77:0/2842604191 --> [v2:172.21.15.186:6820/25551,v1:172.21.15.186:6822/25551] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6fe4163650 con 0x7f6fc406c780 2022-04-03T19:44:29.090 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.969+0000 7ffa40c63700 1 -- 172.21.15.77:0/596656967 shutdown_connections 2022-04-03T19:44:29.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.969+0000 7fdb597fa700 1 -- 172.21.15.77:0/3984074504 <== osd.5 v2:172.21.15.186:6808/25541 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdb6400bed0 con 0x7fdb4c06c6e0 2022-04-03T19:44:29.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.969+0000 7f6fd27fc700 1 -- 172.21.15.77:0/2842604191 <== osd.4 v2:172.21.15.186:6820/25551 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6fe4163650 con 0x7f6fc406c780 2022-04-03T19:44:29.092 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.969+0000 7ffa40c63700 1 -- 172.21.15.77:0/596656967 wait complete. 2022-04-03T19:44:29.093 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.971+0000 7f6feb114700 1 -- 172.21.15.77:0/2842604191 >> [v2:172.21.15.186:6820/25551,v1:172.21.15.186:6822/25551] conn(0x7f6fc406c780 msgr2=0x7f6fc406eb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.093 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.971+0000 7f6feb114700 1 --2- 172.21.15.77:0/2842604191 >> [v2:172.21.15.186:6820/25551,v1:172.21.15.186:6822/25551] conn(0x7f6fc406c780 0x7f6fc406eb90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.971+0000 7f6feb114700 1 -- 172.21.15.77:0/2842604191 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6fc4063e30 msgr2=0x7f6fc40662e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.971+0000 7f6feb114700 1 --2- 172.21.15.77:0/2842604191 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6fc4063e30 0x7f6fc40662e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0001840 tx=0x7f6fe0009040 comp rx=0 tx=0).stop 2022-04-03T19:44:29.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.971+0000 7f6feb114700 1 -- 172.21.15.77:0/2842604191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fe4162230 msgr2=0x7f6fe40bf650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.971+0000 7f6feb114700 1 --2- 172.21.15.77:0/2842604191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fe4162230 0x7f6fe40bf650 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4016000 tx=0x7f6fd4017000 comp rx=0 tx=0).stop 2022-04-03T19:44:29.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.972+0000 7f6fea913700 1 -- 172.21.15.77:0/2842604191 reap_dead start 2022-04-03T19:44:29.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.972+0000 7f6feb114700 1 -- 172.21.15.77:0/2842604191 shutdown_connections 2022-04-03T19:44:29.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.972+0000 7f6feb114700 1 -- 172.21.15.77:0/2842604191 >> 172.21.15.77:0/2842604191 conn(0x7f6fe4066da0 msgr2=0x7f6fe4161380 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:29.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.972+0000 7f6feb114700 1 -- 172.21.15.77:0/2842604191 shutdown_connections 2022-04-03T19:44:29.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.972+0000 7f6feb114700 1 -- 172.21.15.77:0/2842604191 wait complete. 2022-04-03T19:44:29.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.973+0000 7f27fcff9700 1 -- 172.21.15.77:0/74086498 shutdown_connections 2022-04-03T19:44:29.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.973+0000 7fdb52ffd700 1 -- 172.21.15.77:0/3984074504 >> [v2:172.21.15.186:6808/25541,v1:172.21.15.186:6809/25541] conn(0x7fdb4c06c6e0 msgr2=0x7fdb4c06eaf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.973+0000 7fdb52ffd700 1 --2- 172.21.15.77:0/3984074504 >> [v2:172.21.15.186:6808/25541,v1:172.21.15.186:6809/25541] conn(0x7fdb4c06c6e0 0x7fdb4c06eaf0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.973+0000 7f27fcff9700 1 -- 172.21.15.77:0/74086498 wait complete. 2022-04-03T19:44:29.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.973+0000 7fdb52ffd700 1 -- 172.21.15.77:0/3984074504 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdb4c063d10 msgr2=0x7fdb4c0661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.973+0000 7fdb52ffd700 1 --2- 172.21.15.77:0/3984074504 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdb4c063d10 0x7fdb4c0661c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c0bfae0 tx=0x7fdb5c007e10 comp rx=0 tx=0).stop 2022-04-03T19:44:29.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.973+0000 7fdb52ffd700 1 -- 172.21.15.77:0/3984074504 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb6c06a000 msgr2=0x7fdb6c1527b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.973+0000 7fdb52ffd700 1 --2- 172.21.15.77:0/3984074504 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb6c06a000 0x7fdb6c1527b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdb68006fd0 tx=0x7fdb6800d040 comp rx=0 tx=0).stop 2022-04-03T19:44:29.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.973+0000 7fdb71e39700 1 -- 172.21.15.77:0/3984074504 reap_dead start 2022-04-03T19:44:29.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.978+0000 7fa194efe700 1 -- 172.21.15.77:0/3870356874 --> [v2:172.21.15.186:6816/25572,v1:172.21.15.186:6817/25572] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa168002bb0 con 0x7fa17406c610 2022-04-03T19:44:29.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.979+0000 7fa187fff700 1 -- 172.21.15.77:0/3870356874 <== osd.7 v2:172.21.15.186:6816/25572 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa168002bb0 con 0x7fa17406c610 2022-04-03T19:44:29.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.979+0000 7fdb52ffd700 1 -- 172.21.15.77:0/3984074504 shutdown_connections 2022-04-03T19:44:29.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.979+0000 7fdb52ffd700 1 -- 172.21.15.77:0/3984074504 >> 172.21.15.77:0/3984074504 conn(0x7fdb6c06d900 msgr2=0x7fdb6c0698f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:29.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.979+0000 7f4ba2405700 1 -- 172.21.15.77:0/964205966 --> [v2:172.21.15.186:6800/25542,v1:172.21.15.186:6801/25542] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4b78002ba0 con 0x7f4b7c06c700 2022-04-03T19:44:29.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.980+0000 7fa194efe700 1 -- 172.21.15.77:0/3870356874 >> [v2:172.21.15.186:6816/25572,v1:172.21.15.186:6817/25572] conn(0x7fa17406c610 msgr2=0x7fa17406ea20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.980+0000 7fa194efe700 1 --2- 172.21.15.77:0/3870356874 >> [v2:172.21.15.186:6816/25572,v1:172.21.15.186:6817/25572] conn(0x7fa17406c610 0x7fa17406ea20 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.980+0000 7fa194efe700 1 -- 172.21.15.77:0/3870356874 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa174063db0 msgr2=0x7fa174066260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.980+0000 7fa194efe700 1 --2- 172.21.15.77:0/3870356874 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa174063db0 0x7fa174066260 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa190154060 tx=0x7fa178009040 comp rx=0 tx=0).stop 2022-04-03T19:44:29.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.980+0000 7fa194efe700 1 -- 172.21.15.77:0/3870356874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa190062f40 msgr2=0x7fa19016a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.980+0000 7fa194efe700 1 --2- 172.21.15.77:0/3870356874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa190062f40 0x7fa19016a6b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa180009340 tx=0x7fa180016000 comp rx=0 tx=0).stop 2022-04-03T19:44:29.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.980+0000 7fa18ffff700 1 -- 172.21.15.77:0/3870356874 reap_dead start 2022-04-03T19:44:29.104 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.981+0000 7f4b897fa700 1 -- 172.21.15.77:0/964205966 <== osd.6 v2:172.21.15.186:6800/25542 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4b78002ba0 con 0x7f4b7c06c700 2022-04-03T19:44:29.104 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.981+0000 7fa194efe700 1 -- 172.21.15.77:0/3870356874 shutdown_connections 2022-04-03T19:44:29.104 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.981+0000 7fa194efe700 1 -- 172.21.15.77:0/3870356874 >> 172.21.15.77:0/3870356874 conn(0x7fa190067960 msgr2=0x7fa190157d50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:29.105 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.981+0000 7fdb52ffd700 1 -- 172.21.15.77:0/3984074504 shutdown_connections 2022-04-03T19:44:29.105 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.981+0000 7fdb52ffd700 1 -- 172.21.15.77:0/3984074504 wait complete. 2022-04-03T19:44:29.106 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.981+0000 7fa194efe700 1 -- 172.21.15.77:0/3870356874 shutdown_connections 2022-04-03T19:44:29.106 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.981+0000 7fa194efe700 1 -- 172.21.15.77:0/3870356874 wait complete. 2022-04-03T19:44:29.108 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.982+0000 7f4ba2405700 1 -- 172.21.15.77:0/964205966 >> [v2:172.21.15.186:6800/25542,v1:172.21.15.186:6801/25542] conn(0x7f4b7c06c700 msgr2=0x7f4b7c06eb10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.109 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.982+0000 7f4ba2405700 1 --2- 172.21.15.77:0/964205966 >> [v2:172.21.15.186:6800/25542,v1:172.21.15.186:6801/25542] conn(0x7f4b7c06c700 0x7f4b7c06eb10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.109 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.983+0000 7f4ba2405700 1 -- 172.21.15.77:0/964205966 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4b7c063db0 msgr2=0x7f4b7c066260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.109 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.983+0000 7f4ba2405700 1 --2- 172.21.15.77:0/964205966 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4b7c063db0 0x7f4b7c066260 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4b98001440 tx=0x7f4b98007980 comp rx=0 tx=0).stop 2022-04-03T19:44:29.109 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.983+0000 7f4ba2405700 1 -- 172.21.15.77:0/964205966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b9c15d450 msgr2=0x7f4b9c16d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.984+0000 7f4ba2405700 1 --2- 172.21.15.77:0/964205966 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b9c15d450 0x7f4b9c16d0b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c00b810 tx=0x7f4b8c006f50 comp rx=0 tx=0).stop 2022-04-03T19:44:29.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.984+0000 7f4ba1c04700 1 -- 172.21.15.77:0/964205966 reap_dead start 2022-04-03T19:44:29.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.985+0000 7f4ba2405700 1 -- 172.21.15.77:0/964205966 shutdown_connections 2022-04-03T19:44:29.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.985+0000 7f4ba2405700 1 -- 172.21.15.77:0/964205966 >> 172.21.15.77:0/964205966 conn(0x7f4b9c066da0 msgr2=0x7f4b9c151610 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:29.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.986+0000 7f4ba2405700 1 -- 172.21.15.77:0/964205966 shutdown_connections 2022-04-03T19:44:29.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:28.986+0000 7f4ba2405700 1 -- 172.21.15.77:0/964205966 wait complete. 2022-04-03T19:44:29.111 INFO:teuthology.orchestra.run.smithi077.stdout:55834574857 2022-04-03T19:44:29.112 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2022-04-03T19:44:29.113 INFO:teuthology.orchestra.run.smithi077.stdout:55834574856 2022-04-03T19:44:29.113 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2022-04-03T19:44:29.114 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2022-04-03T19:44:29.115 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2022-04-03T19:44:29.116 INFO:teuthology.orchestra.run.smithi077.stdout:55834574856 2022-04-03T19:44:29.116 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2022-04-03T19:44:29.117 INFO:teuthology.orchestra.run.smithi077.stdout:55834574856 2022-04-03T19:44:29.118 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2022-04-03T19:44:29.118 INFO:teuthology.orchestra.run.smithi077.stdout:55834574856 2022-04-03T19:44:29.119 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2022-04-03T19:44:29.120 INFO:teuthology.orchestra.run.smithi077.stdout:55834574858 2022-04-03T19:44:29.120 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2022-04-03T19:44:29.121 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2022-04-03T19:44:29.122 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2022-04-03T19:44:29.510 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.506+0000 7f373d64e700 1 Processor -- start 2022-04-03T19:44:29.511 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.506+0000 7f373d64e700 1 -- start start 2022-04-03T19:44:29.511 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.508+0000 7f373d64e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3738157340 0x7f373815a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.511 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.508+0000 7f373d64e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f373806a440 0x7f373806aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.511 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.508+0000 7f373d64e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37380c2bd0 con 0x7f3738156f70 2022-04-03T19:44:29.512 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.508+0000 7f373d64e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37380bf2c0 con 0x7f373806a440 2022-04-03T19:44:29.512 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.508+0000 7f373d64e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37380c8310 con 0x7f3738157340 2022-04-03T19:44:29.512 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.509+0000 7f37377fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f373806a440 0x7f373806aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.512 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.509+0000 7f37377fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f373806a440 0x7f373806aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37738/0 (socket says 172.21.15.77:37738) 2022-04-03T19:44:29.513 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.509+0000 7f37377fe700 1 -- 172.21.15.77:0/919284038 learned_addr learned my addr 172.21.15.77:0/919284038 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:29.513 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.510+0000 7f37367fc700 1 --2- 172.21.15.77:0/919284038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3738157340 0x7f373815a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.514 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.510+0000 7f37367fc700 1 -- 172.21.15.77:0/919284038 >> v2:172.21.15.186:3300/0 conn(0x7f373806a440 msgr2=0x7f373806aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:29.514 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.510+0000 7f37367fc700 1 --2- 172.21.15.77:0/919284038 >> v2:172.21.15.186:3300/0 conn(0x7f373806a440 0x7f373806aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.515 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.510+0000 7f37367fc700 1 -- 172.21.15.77:0/919284038 >> v1:172.21.15.77:6789/0 conn(0x7f3738156f70 legacy=0x7f37380c3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:29.515 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.510+0000 7f37367fc700 1 -- 172.21.15.77:0/919284038 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3738069ce0 con 0x7f3738157340 2022-04-03T19:44:29.515 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.512+0000 7f855be21700 1 Processor -- start 2022-04-03T19:44:29.516 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.512+0000 7f5bbcfbf700 1 Processor -- start 2022-04-03T19:44:29.516 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.512+0000 7f5bbcfbf700 1 -- start start 2022-04-03T19:44:29.517 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.513+0000 7f37367fc700 1 --2- 172.21.15.77:0/919284038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3738157340 0x7f373815a360 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3728009760 tx=0x7f3728009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b7401cb0c7425f6 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.518 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.513+0000 7f3735ffb700 1 -- 172.21.15.77:0/919284038 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3728004030 con 0x7f3738157340 2022-04-03T19:44:29.518 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.512+0000 7f855be21700 1 -- start start 2022-04-03T19:44:29.519 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.513+0000 7f3735ffb700 1 -- 172.21.15.77:0/919284038 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3728004190 con 0x7f3738157340 2022-04-03T19:44:29.519 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.513+0000 7f3735ffb700 1 -- 172.21.15.77:0/919284038 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3728011630 con 0x7f3738157340 2022-04-03T19:44:29.520 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.514+0000 7f855be21700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85540c60b0 0x7f855406dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.515+0000 7f5bbcfbf700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5bb80c2680 0x7f5bb806bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.515+0000 7f5bbcfbf700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb815cb70 0x7f5bb806c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.515+0000 7f5bbcfbf700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb8069380 con 0x7f5bb80c1ba0 2022-04-03T19:44:29.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.515+0000 7f5bbcfbf700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bb806bb50 con 0x7f5bb80c2680 2022-04-03T19:44:29.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.515+0000 7f5bbcfbf700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb80c4010 con 0x7f5bb815cb70 2022-04-03T19:44:29.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.515+0000 7f5bb77fe700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f5bb80c1ba0 0x7f5bb80c1f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44114/0 (socket says 172.21.15.77:44114) 2022-04-03T19:44:29.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.514+0000 7f855be21700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f855406c800 0x7f85540c3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.516+0000 7f855be21700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85540c30c0 con 0x7f855406c430 2022-04-03T19:44:29.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.516+0000 7f855be21700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85540c3a90 con 0x7f855406c800 2022-04-03T19:44:29.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.516+0000 7f855be21700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8554155bf0 con 0x7f85540c60b0 2022-04-03T19:44:29.525 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.515+0000 7f5bb77fe700 1 -- 172.21.15.77:0/1273075013 learned_addr learned my addr 172.21.15.77:0/1273075013 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:29.525 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.516+0000 7f5bb7fff700 1 --2- 172.21.15.77:0/1273075013 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb815cb70 0x7f5bb806c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.517+0000 7f8559bbd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f855406c430 0x7f85541553b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38318/0 (socket says 172.21.15.77:38318) 2022-04-03T19:44:29.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.517+0000 7f8559bbd700 1 -- 172.21.15.77:0/246846733 learned_addr learned my addr 172.21.15.77:0/246846733 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:29.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.516+0000 7f5bb6ffd700 1 --2- 172.21.15.77:0/1273075013 >> v2:172.21.15.77:3300/0 conn(0x7f5bb80c2680 0x7f5bb806bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.515+0000 7f373d64e700 1 -- 172.21.15.77:0/919284038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3738157340 msgr2=0x7f373815a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.528 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.517+0000 7f5bb7fff700 1 -- 172.21.15.77:0/1273075013 >> v2:172.21.15.77:3300/0 conn(0x7f5bb80c2680 msgr2=0x7f5bb806bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:29.528 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.517+0000 7f5bb7fff700 1 --2- 172.21.15.77:0/1273075013 >> v2:172.21.15.77:3300/0 conn(0x7f5bb80c2680 0x7f5bb806bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.528 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.517+0000 7f5bb7fff700 1 -- 172.21.15.77:0/1273075013 >> v1:172.21.15.186:6789/0 conn(0x7f5bb80c1ba0 legacy=0x7f5bb80c1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.517+0000 7f5bb7fff700 1 -- 172.21.15.77:0/1273075013 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb80c42f0 con 0x7f5bb815cb70 2022-04-03T19:44:29.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.517+0000 7f5bb6ffd700 1 --2- 172.21.15.77:0/1273075013 >> v2:172.21.15.77:3300/0 conn(0x7f5bb80c2680 0x7f5bb806bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:29.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.518+0000 7f85593bc700 1 --2- 172.21.15.77:0/246846733 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85540c60b0 0x7f855406dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.518+0000 7f855a3be700 1 --2- 172.21.15.77:0/246846733 >> v2:172.21.15.186:3300/0 conn(0x7f855406c800 0x7f85540c3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.515+0000 7f373d64e700 1 --2- 172.21.15.77:0/919284038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3738157340 0x7f373815a360 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3728009760 tx=0x7f3728009060 comp rx=0 tx=0).stop 2022-04-03T19:44:29.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.518+0000 7f373d64e700 1 -- 172.21.15.77:0/919284038 shutdown_connections 2022-04-03T19:44:29.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.518+0000 7f373d64e700 1 --2- 172.21.15.77:0/919284038 >> v2:172.21.15.186:3300/0 conn(0x7f373806a440 0x7f373806aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.518+0000 7f373d64e700 1 --2- 172.21.15.77:0/919284038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3738157340 0x7f373815a360 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.518+0000 7f373d64e700 1 -- 172.21.15.77:0/919284038 >> 172.21.15.77:0/919284038 conn(0x7f3738068280 msgr2=0x7f37380c7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:29.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.518+0000 7f373d64e700 1 -- 172.21.15.77:0/919284038 shutdown_connections 2022-04-03T19:44:29.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.519+0000 7f373d64e700 1 -- 172.21.15.77:0/919284038 wait complete. 2022-04-03T19:44:29.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.519+0000 7f5bb7fff700 1 --2- 172.21.15.77:0/1273075013 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb815cb70 0x7f5bb806c4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5bac009760 tx=0x7f5bac009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fd288ee8eee54cb server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.519+0000 7f5bb67fc700 1 -- 172.21.15.77:0/1273075013 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac011630 con 0x7f5bb815cb70 2022-04-03T19:44:29.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.519+0000 7f5bb67fc700 1 -- 172.21.15.77:0/1273075013 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bac011790 con 0x7f5bb815cb70 2022-04-03T19:44:29.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.519+0000 7f5bb67fc700 1 -- 172.21.15.77:0/1273075013 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac011b10 con 0x7f5bb815cb70 2022-04-03T19:44:29.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.519+0000 7f855a3be700 1 -- 172.21.15.77:0/246846733 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85540c60b0 msgr2=0x7f855406dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:29.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.519+0000 7f855a3be700 1 --2- 172.21.15.77:0/246846733 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85540c60b0 0x7f855406dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.519+0000 7f8558bbb700 1 -- 172.21.15.77:0/246846733 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039582290 0 0) 0x7f85540c30c0 con 0x7f855406c430 2022-04-03T19:44:29.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.519+0000 7f855a3be700 1 -- 172.21.15.77:0/246846733 >> v1:172.21.15.77:6789/0 conn(0x7f855406c430 legacy=0x7f85541553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.519+0000 7f855a3be700 1 -- 172.21.15.77:0/246846733 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8554155ed0 con 0x7f855406c800 2022-04-03T19:44:29.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.520+0000 7f855a3be700 1 --2- 172.21.15.77:0/246846733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f855406c800 0x7f85540c3550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f854c009e60 tx=0x7f854c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=992b84af8b7c91e3 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.520+0000 7f5bbcfbf700 1 -- 172.21.15.77:0/1273075013 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb815cb70 msgr2=0x7f5bb806c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.521+0000 7f8558bbb700 1 -- 172.21.15.77:0/246846733 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c00e070 con 0x7f855406c800 2022-04-03T19:44:29.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.521+0000 7f8558bbb700 1 -- 172.21.15.77:0/246846733 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854c00bb60 con 0x7f855406c800 2022-04-03T19:44:29.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.521+0000 7f8558bbb700 1 -- 172.21.15.77:0/246846733 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c012430 con 0x7f855406c800 2022-04-03T19:44:29.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.520+0000 7f5bbcfbf700 1 --2- 172.21.15.77:0/1273075013 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb815cb70 0x7f5bb806c4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5bac009760 tx=0x7f5bac009060 comp rx=0 tx=0).stop 2022-04-03T19:44:29.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.521+0000 7f5bbcfbf700 1 -- 172.21.15.77:0/1273075013 shutdown_connections 2022-04-03T19:44:29.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.521+0000 7f5bbcfbf700 1 --2- 172.21.15.77:0/1273075013 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb815cb70 0x7f5bb806c4e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.521+0000 7f5bbcfbf700 1 --2- 172.21.15.77:0/1273075013 >> v2:172.21.15.77:3300/0 conn(0x7f5bb80c2680 0x7f5bb806bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.521+0000 7f5bbcfbf700 1 -- 172.21.15.77:0/1273075013 >> 172.21.15.77:0/1273075013 conn(0x7f5bb8066da0 msgr2=0x7f5bb80671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:29.540 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.522+0000 7f373d64e700 1 Processor -- start 2022-04-03T19:44:29.540 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.522+0000 7f373d64e700 1 -- start start 2022-04-03T19:44:29.540 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.522+0000 7f855be21700 1 -- 172.21.15.77:0/246846733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f855406c800 msgr2=0x7f85540c3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.541 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.522+0000 7f855be21700 1 --2- 172.21.15.77:0/246846733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f855406c800 0x7f85540c3550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f854c009e60 tx=0x7f854c009780 comp rx=0 tx=0).stop 2022-04-03T19:44:29.541 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.521+0000 7f5bbcfbf700 1 -- 172.21.15.77:0/1273075013 shutdown_connections 2022-04-03T19:44:29.542 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.523+0000 7f855be21700 1 -- 172.21.15.77:0/246846733 shutdown_connections 2022-04-03T19:44:29.542 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.523+0000 7f855be21700 1 --2- 172.21.15.77:0/246846733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f855406c800 0x7f85540c3550 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.543 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.523+0000 7f855be21700 1 --2- 172.21.15.77:0/246846733 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85540c60b0 0x7f855406dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.543 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.523+0000 7f855be21700 1 -- 172.21.15.77:0/246846733 >> 172.21.15.77:0/246846733 conn(0x7f85540cb700 msgr2=0x7f85540cbb00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:29.543 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.522+0000 7f5bbcfbf700 1 -- 172.21.15.77:0/1273075013 wait complete. 2022-04-03T19:44:29.544 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.523+0000 7f855be21700 1 -- 172.21.15.77:0/246846733 shutdown_connections 2022-04-03T19:44:29.546 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.523+0000 7f373d64e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f373806a440 0x7f373815f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.546 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.524+0000 7f373d64e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3738156f70 0x7f3738164820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.547 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.524+0000 7f373d64e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3738157340 0x7f37381542f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.547 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.524+0000 7f373d64e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3738163a90 con 0x7f3738157340 2022-04-03T19:44:29.547 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.524+0000 7f373d64e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373815ee50 con 0x7f3738156f70 2022-04-03T19:44:29.547 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.524+0000 7f373d64e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f373814d360 con 0x7f373806a440 2022-04-03T19:44:29.548 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.524+0000 7f37377fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3738157340 0x7f37381542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.548 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.525+0000 7f37377fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3738157340 0x7f37381542f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42558/0 (socket says 172.21.15.77:42558) 2022-04-03T19:44:29.548 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.525+0000 7f37377fe700 1 -- 172.21.15.77:0/2551445156 learned_addr learned my addr 172.21.15.77:0/2551445156 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:29.548 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.525+0000 7f37377fe700 1 -- 172.21.15.77:0/2551445156 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f373806a440 msgr2=0x7f373815f840 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2022-04-03T19:44:29.549 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.524+0000 7f5bbcfbf700 1 Processor -- start 2022-04-03T19:44:29.549 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.524+0000 7f5bbcfbf700 1 -- start start 2022-04-03T19:44:29.549 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.526+0000 7f5bbcfbf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bb80c1ba0 0x7f5bb80b9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.550 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.525+0000 7f37377fe700 1 --2- 172.21.15.77:0/2551445156 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f373806a440 0x7f373815f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.551 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.526+0000 7f5bbcfbf700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bb80c2680 0x7f5bb80b5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.551 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.525+0000 7f37377fe700 1 -- 172.21.15.77:0/2551445156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3738156f70 msgr2=0x7f3738164820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-04-03T19:44:29.552 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.525+0000 7f37377fe700 1 --2- 172.21.15.77:0/2551445156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3738156f70 0x7f3738164820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.552 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.525+0000 7f37377fe700 1 -- 172.21.15.77:0/2551445156 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37280092a0 con 0x7f3738157340 2022-04-03T19:44:29.553 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.526+0000 7f5bbcfbf700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb815cb70 0x7f5bb80b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.553 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.526+0000 7f37377fe700 1 --2- 172.21.15.77:0/2551445156 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3738157340 0x7f37381542f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f373000c0f0 tx=0x7f373000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.554 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.526+0000 7f5bb77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bb80c1ba0 0x7f5bb80b9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.554 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.526+0000 7f5bb77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bb80c1ba0 0x7f5bb80b9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42560/0 (socket says 172.21.15.77:42560) 2022-04-03T19:44:29.555 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.527+0000 7f5bbcfbf700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb80bb830 con 0x7f5bb80c1ba0 2022-04-03T19:44:29.555 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.527+0000 7f5bbcfbf700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb80b5500 con 0x7f5bb80c2680 2022-04-03T19:44:29.555 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.527+0000 7f5bbcfbf700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb8169690 con 0x7f5bb815cb70 2022-04-03T19:44:29.556 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.527+0000 7f5bb77fe700 1 -- 172.21.15.77:0/1713341074 learned_addr learned my addr 172.21.15.77:0/1713341074 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:29.556 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.527+0000 7f855be21700 1 -- 172.21.15.77:0/246846733 wait complete. 2022-04-03T19:44:29.557 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.528+0000 7f5bb77fe700 1 -- 172.21.15.77:0/1713341074 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb815cb70 msgr2=0x7f5bb80b5e90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2022-04-03T19:44:29.557 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.528+0000 7f5bb77fe700 1 --2- 172.21.15.77:0/1713341074 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb815cb70 0x7f5bb80b5e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.557 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.528+0000 7f5bb77fe700 1 -- 172.21.15.77:0/1713341074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bb80c2680 msgr2=0x7f5bb80b5950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2022-04-03T19:44:29.558 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.528+0000 7f5bb77fe700 1 --2- 172.21.15.77:0/1713341074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bb80c2680 0x7f5bb80b5950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.558 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.528+0000 7f5bb77fe700 1 -- 172.21.15.77:0/1713341074 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb0009a20 con 0x7f5bb80c1ba0 2022-04-03T19:44:29.558 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.529+0000 7f5bb77fe700 1 --2- 172.21.15.77:0/1713341074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bb80c1ba0 0x7f5bb80b9fb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8007fd0 tx=0x7f5ba8007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.527+0000 7f371effd700 1 -- 172.21.15.77:0/2551445156 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3730009110 con 0x7f3738157340 2022-04-03T19:44:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.527+0000 7f371effd700 1 -- 172.21.15.77:0/2551445156 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373000f040 con 0x7f3738157340 2022-04-03T19:44:29.559 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.527+0000 7f371effd700 1 -- 172.21.15.77:0/2551445156 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373000cd90 con 0x7f3738157340 2022-04-03T19:44:29.560 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.528+0000 7f373d64e700 1 -- 172.21.15.77:0/2551445156 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f373815f2e0 con 0x7f3738157340 2022-04-03T19:44:29.560 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.529+0000 7f373d64e700 1 -- 172.21.15.77:0/2551445156 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3738160420 con 0x7f3738157340 2022-04-03T19:44:29.560 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.530+0000 7f5b9f7fe700 1 -- 172.21.15.77:0/1713341074 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba80072e0 con 0x7f5bb80c1ba0 2022-04-03T19:44:29.561 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.529+0000 7f855be21700 1 Processor -- start 2022-04-03T19:44:29.561 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.530+0000 7f5bbcfbf700 1 -- 172.21.15.77:0/1713341074 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bac0092a0 con 0x7f5bb80c1ba0 2022-04-03T19:44:29.562 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.530+0000 7f855be21700 1 -- start start 2022-04-03T19:44:29.562 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.530+0000 7f371cff9700 1 -- 172.21.15.77:0/2551445156 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f370c005130 con 0x7f3738157340 2022-04-03T19:44:29.563 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.531+0000 7f5bbcfbf700 1 -- 172.21.15.77:0/1713341074 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb8169c50 con 0x7f5bb80c1ba0 2022-04-03T19:44:29.563 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.531+0000 7f5b9f7fe700 1 -- 172.21.15.77:0/1713341074 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba8007440 con 0x7f5bb80c1ba0 2022-04-03T19:44:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.532+0000 7f855be21700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f855406c430 0x7f8554151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.532+0000 7f855be21700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f855406c800 0x7f85541522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.533+0000 7f8559bbd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f855406c430 0x7f8554151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.533+0000 7f8559bbd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f855406c430 0x7f8554151d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37744/0 (socket says 172.21.15.77:37744) 2022-04-03T19:44:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.533+0000 7f85593bc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f855406c800 0x7f85541522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.533+0000 7f8559bbd700 1 -- 172.21.15.77:0/3051162908 learned_addr learned my addr 172.21.15.77:0/3051162908 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.533+0000 7f855be21700 1 --2- 172.21.15.77:0/3051162908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85541682d0 0x7f8554168740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.533+0000 7f855be21700 1 -- 172.21.15.77:0/3051162908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8554155440 con 0x7f85541682d0 2022-04-03T19:44:29.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.534+0000 7f855be21700 1 -- 172.21.15.77:0/3051162908 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85540bf510 con 0x7f855406c430 2022-04-03T19:44:29.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.533+0000 7fa712930700 1 Processor -- start 2022-04-03T19:44:29.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.534+0000 7f855be21700 1 -- 172.21.15.77:0/3051162908 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8554154ec0 con 0x7f855406c800 2022-04-03T19:44:29.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.532+0000 7f5b9f7fe700 1 -- 172.21.15.77:0/1713341074 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8005780 con 0x7f5bb80c1ba0 2022-04-03T19:44:29.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.534+0000 7f5b9d7fa700 1 -- 172.21.15.77:0/1713341074 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb8067d90 con 0x7f5bb80c1ba0 2022-04-03T19:44:29.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.535+0000 7f855a3be700 1 --2- 172.21.15.77:0/3051162908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85541682d0 0x7f8554168740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.535+0000 7f371effd700 1 -- 172.21.15.77:0/2551445156 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f3730003a70 con 0x7f3738157340 2022-04-03T19:44:29.570 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.536+0000 7f5b9f7fe700 1 -- 172.21.15.77:0/1713341074 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f5ba8019030 con 0x7f5bb80c1ba0 2022-04-03T19:44:29.570 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.537+0000 7f5b9f7fe700 1 --2- 172.21.15.77:0/1713341074 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5b98063de0 0x7f5b98066290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.570 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.537+0000 7f5b9f7fe700 1 -- 172.21.15.77:0/1713341074 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f5ba807f960 con 0x7f5bb80c1ba0 2022-04-03T19:44:29.571 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.536+0000 7f855a3be700 1 -- 172.21.15.77:0/3051162908 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f855406c800 msgr2=0x7f85541522a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.571 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.536+0000 7f855a3be700 1 --2- 172.21.15.77:0/3051162908 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f855406c800 0x7f85541522a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.571 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.536+0000 7f855a3be700 1 -- 172.21.15.77:0/3051162908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f855406c430 msgr2=0x7f8554151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.536+0000 7f855a3be700 1 --2- 172.21.15.77:0/3051162908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f855406c430 0x7f8554151d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.536+0000 7f855a3be700 1 -- 172.21.15.77:0/3051162908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c009a20 con 0x7f85541682d0 2022-04-03T19:44:29.573 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.535+0000 7fa712930700 1 -- start start 2022-04-03T19:44:29.573 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.537+0000 7fa712930700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa70c15d450 0x7fa70c1535e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.574 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.537+0000 7fa712930700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa70c069d00 con 0x7fa70c0ba320 2022-04-03T19:44:29.574 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.537+0000 7fa712930700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa70c069110 con 0x7fa70c0678b0 2022-04-03T19:44:29.574 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.537+0000 7fa712930700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c06e460 con 0x7fa70c15d450 2022-04-03T19:44:29.574 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.538+0000 7fa71212f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa70c15d450 0x7fa70c1535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.574 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.538+0000 7fa71192e700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa70c0ba320 0x7fa70c15cd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38320/0 (socket says 172.21.15.77:38320) 2022-04-03T19:44:29.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.538+0000 7fa71192e700 1 -- 172.21.15.77:0/1762966569 learned_addr learned my addr 172.21.15.77:0/1762966569 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:29.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.539+0000 7fa71212f700 1 -- 172.21.15.77:0/1762966569 >> v1:172.21.15.186:6789/0 conn(0x7fa70c0678b0 legacy=0x7fa70c152ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.539+0000 7fa71092c700 1 -- 172.21.15.77:0/1762966569 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720743450 0 0) 0x7fa70c069110 con 0x7fa70c0678b0 2022-04-03T19:44:29.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.539+0000 7fa71212f700 1 -- 172.21.15.77:0/1762966569 >> v1:172.21.15.77:6789/0 conn(0x7fa70c0ba320 legacy=0x7fa70c15cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.539+0000 7fa71212f700 1 -- 172.21.15.77:0/1762966569 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa70c15a2a0 con 0x7fa70c15d450 2022-04-03T19:44:29.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.537+0000 7f371effd700 1 --2- 172.21.15.77:0/2551445156 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3718063e30 0x7f37180662e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.537+0000 7f371effd700 1 -- 172.21.15.77:0/2551445156 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f3730080f50 con 0x7f3738157340 2022-04-03T19:44:29.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.537+0000 7f3736ffd700 1 --2- 172.21.15.77:0/2551445156 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3718063e30 0x7f37180662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.538+0000 7f5bb6ffd700 1 --2- 172.21.15.77:0/1713341074 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5b98063de0 0x7f5b98066290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.539+0000 7f3736ffd700 1 --2- 172.21.15.77:0/2551445156 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3718063e30 0x7f37180662e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f372c0095a0 tx=0x7f372c006a60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.539+0000 7fa71212f700 1 --2- 172.21.15.77:0/1762966569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa70c15d450 0x7fa70c1535e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa708009790 tx=0x7fa708009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90de8c92c2463ba3 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.579 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.539+0000 7f855a3be700 1 --2- 172.21.15.77:0/3051162908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85541682d0 0x7f8554168740 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f854c016000 tx=0x7f854c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.579 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.540+0000 7fa71092c700 1 -- 172.21.15.77:0/1762966569 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa708011630 con 0x7fa70c15d450 2022-04-03T19:44:29.580 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.540+0000 7fa71092c700 1 -- 172.21.15.77:0/1762966569 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa708011790 con 0x7fa70c15d450 2022-04-03T19:44:29.580 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.540+0000 7fa71092c700 1 -- 172.21.15.77:0/1762966569 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa708011630 con 0x7fa70c15d450 2022-04-03T19:44:29.581 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.540+0000 7fcccee49700 1 Processor -- start 2022-04-03T19:44:29.581 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.540+0000 7fcccee49700 1 -- start start 2022-04-03T19:44:29.581 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.540+0000 7f92c3562700 1 Processor -- start 2022-04-03T19:44:29.582 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.541+0000 7fa712930700 1 -- 172.21.15.77:0/1762966569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa70c15d450 msgr2=0x7fa70c1535e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.582 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.541+0000 7fa712930700 1 --2- 172.21.15.77:0/1762966569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa70c15d450 0x7fa70c1535e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa708009790 tx=0x7fa708009060 comp rx=0 tx=0).stop 2022-04-03T19:44:29.583 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.541+0000 7f92c3562700 1 -- start start 2022-04-03T19:44:29.583 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.541+0000 7fa712930700 1 -- 172.21.15.77:0/1762966569 shutdown_connections 2022-04-03T19:44:29.583 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.541+0000 7fa712930700 1 --2- 172.21.15.77:0/1762966569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa70c15d450 0x7fa70c1535e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.584 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.541+0000 7fa712930700 1 -- 172.21.15.77:0/1762966569 >> 172.21.15.77:0/1762966569 conn(0x7fa70c066da0 msgr2=0x7fa70c0671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:29.584 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.541+0000 7f5bb6ffd700 1 --2- 172.21.15.77:0/1713341074 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5b98063de0 0x7f5b98066290 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8124710 tx=0x7f5bb000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.585 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.541+0000 7fa712930700 1 -- 172.21.15.77:0/1762966569 shutdown_connections 2022-04-03T19:44:29.585 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.541+0000 7fa712930700 1 -- 172.21.15.77:0/1762966569 wait complete. 2022-04-03T19:44:29.586 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.542+0000 7fcccee49700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccc8069370 0x7fccc8159160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.587 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.542+0000 7fcccee49700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fccc80c74f0 0x7fccc80c7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.587 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.542+0000 7fcccee49700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc80c1f70 con 0x7fccc80c1b80 2022-04-03T19:44:29.587 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.542+0000 7fcccd3e6700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fccc80c74f0 0x7fccc80c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.587 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.542+0000 7fccbffff700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fccc80c1b80 0x7fccc8069740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44118/0 (socket says 172.21.15.77:44118) 2022-04-03T19:44:29.587 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.542+0000 7fccbffff700 1 -- 172.21.15.77:0/3469559783 learned_addr learned my addr 172.21.15.77:0/3469559783 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:29.588 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.543+0000 7fcccee49700 1 -- 172.21.15.77:0/3469559783 --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc8068e90 con 0x7fccc80c74f0 2022-04-03T19:44:29.588 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.543+0000 7fcccee49700 1 -- 172.21.15.77:0/3469559783 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc8156030 con 0x7fccc8069370 2022-04-03T19:44:29.589 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.543+0000 7f92c2560700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f92bc06daf0 0x7f92bc06ba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38322/0 (socket says 172.21.15.77:38322) 2022-04-03T19:44:29.589 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.543+0000 7f92c2560700 1 -- 172.21.15.77:0/361366402 learned_addr learned my addr 172.21.15.77:0/361366402 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:29.589 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.544+0000 7fccbf7fe700 1 -- 172.21.15.77:0/3469559783 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187593435 0 0) 0x7fccc80c1f70 con 0x7fccc80c1b80 2022-04-03T19:44:29.589 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.544+0000 7fccbf7fe700 1 -- 172.21.15.77:0/3469559783 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccb40033f0 con 0x7fccc80c1b80 2022-04-03T19:44:29.590 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.543+0000 7f92c3562700 1 --2- 172.21.15.77:0/361366402 >> v2:172.21.15.77:3300/0 conn(0x7f92bc0c36a0 0x7f92bc06cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.590 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.544+0000 7f92c2d61700 1 --2- 172.21.15.77:0/361366402 >> v2:172.21.15.77:3300/0 conn(0x7f92bc0c36a0 0x7f92bc06cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.591 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.544+0000 7f92c3562700 1 -- 172.21.15.77:0/361366402 --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92bc14fa50 con 0x7f92bc06daf0 2022-04-03T19:44:29.591 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.544+0000 7f33f4079700 1 Processor -- start 2022-04-03T19:44:29.592 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.544+0000 7f92c3562700 1 -- 172.21.15.77:0/361366402 --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92bc0c3210 con 0x7f92bc0c5f80 2022-04-03T19:44:29.592 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.544+0000 7f92c3562700 1 -- 172.21.15.77:0/361366402 --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92bc1561b0 con 0x7f92bc0c36a0 2022-04-03T19:44:29.592 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.544+0000 7fccbf7fe700 1 -- 172.21.15.77:0/3469559783 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3195909552 0 0) 0x7fccb40033f0 con 0x7fccc80c1b80 2022-04-03T19:44:29.593 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.544+0000 7fccbf7fe700 1 -- 172.21.15.77:0/3469559783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccc8069370 msgr2=0x7fccc8159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:29.593 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.544+0000 7fccccbe5700 1 --2- 172.21.15.77:0/3469559783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccc8069370 0x7fccc8159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.593 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.544+0000 7f33f4079700 1 -- start start 2022-04-03T19:44:29.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.545+0000 7f92c2d61700 1 -- 172.21.15.77:0/361366402 >> v1:172.21.15.186:6789/0 conn(0x7f92bc0c5f80 legacy=0x7f92bc0c5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.545+0000 7f92c2d61700 1 -- 172.21.15.77:0/361366402 >> v1:172.21.15.77:6789/0 conn(0x7f92bc06daf0 legacy=0x7f92bc06ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.545+0000 7f92c2d61700 1 -- 172.21.15.77:0/361366402 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92bc06c4d0 con 0x7f92bc0c36a0 2022-04-03T19:44:29.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.545+0000 7fa712930700 1 Processor -- start 2022-04-03T19:44:29.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.544+0000 7fccbf7fe700 1 --2- 172.21.15.77:0/3469559783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccc8069370 0x7fccc8159160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.544+0000 7fccbf7fe700 1 -- 172.21.15.77:0/3469559783 >> v2:172.21.15.186:3300/0 conn(0x7fccc80c74f0 msgr2=0x7fccc80c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:29.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.544+0000 7fccbf7fe700 1 --2- 172.21.15.77:0/3469559783 >> v2:172.21.15.186:3300/0 conn(0x7fccc80c74f0 0x7fccc80c7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.544+0000 7fccbf7fe700 1 -- 172.21.15.77:0/3469559783 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc80c7ed0 con 0x7fccc80c1b80 2022-04-03T19:44:29.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.544+0000 7fccbf7fe700 1 -- 172.21.15.77:0/3469559783 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4219302955 0 0) 0x7fccb8001470 con 0x7fccc80c1b80 2022-04-03T19:44:29.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.544+0000 7fccbf7fe700 1 -- 172.21.15.77:0/3469559783 >> v1:172.21.15.186:6789/0 conn(0x7fccc80c1b80 legacy=0x7fccc8069740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.544+0000 7fccbf7fe700 1 --2- 172.21.15.77:0/3469559783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccb4003960 0x7fccb4003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.545+0000 7fccbf7fe700 1 --2- 172.21.15.77:0/3469559783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb4004690 0x7fccb4004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.545+0000 7fccbf7fe700 1 --2- 172.21.15.77:0/3469559783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccb4005090 0x7fccb40140f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.545+0000 7fccbf7fe700 1 -- 172.21.15.77:0/3469559783 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc8156030 con 0x7fccb4005090 2022-04-03T19:44:29.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.545+0000 7fccbf7fe700 1 -- 172.21.15.77:0/3469559783 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb4014770 con 0x7fccb4004690 2022-04-03T19:44:29.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.545+0000 7fccbf7fe700 1 -- 172.21.15.77:0/3469559783 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb4014930 con 0x7fccb4003960 2022-04-03T19:44:29.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.545+0000 7f92c2d61700 1 --2- 172.21.15.77:0/361366402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92bc0c36a0 0x7f92bc06cf40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f92b4009e60 tx=0x7f92b4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a879eb4916a7181b server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.545+0000 7fccccbe5700 1 --2- 172.21.15.77:0/3469559783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccc8069370 0x7fccc8159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-04-03T19:44:29.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.545+0000 7fccbffff700 1 --2- 172.21.15.77:0/3469559783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb4004690 0x7fccb4004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.545+0000 7fccccbe5700 1 --2- 172.21.15.77:0/3469559783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccb4003960 0x7fccb4003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.545+0000 7fa712930700 1 -- start start 2022-04-03T19:44:29.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.546+0000 7fccccbe5700 1 -- 172.21.15.77:0/3469559783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb4004690 msgr2=0x7fccb4004b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.546+0000 7fcccd3e6700 1 --2- 172.21.15.77:0/3469559783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccb4005090 0x7fccb40140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.546+0000 7fccccbe5700 1 --2- 172.21.15.77:0/3469559783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb4004690 0x7fccb4004b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.546+0000 7fccccbe5700 1 -- 172.21.15.77:0/3469559783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccb4005090 msgr2=0x7fccb40140f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:29.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.546+0000 7fccccbe5700 1 --2- 172.21.15.77:0/3469559783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccb4005090 0x7fccb40140f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.546+0000 7fccccbe5700 1 -- 172.21.15.77:0/3469559783 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb40033f0 con 0x7fccb4003960 2022-04-03T19:44:29.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.546+0000 7f33f4079700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f33ec14f630 0x7f33ec155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.546+0000 7f33f4079700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33ec06d280 con 0x7f33ec0c67e0 2022-04-03T19:44:29.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.546+0000 7f33f1e15700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f33ec0c5410 0x7f33ec0c60d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44122/0 (socket says 172.21.15.77:44122) 2022-04-03T19:44:29.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.546+0000 7f33f1e15700 1 -- 172.21.15.77:0/940753733 learned_addr learned my addr 172.21.15.77:0/940753733 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:29.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.547+0000 7fccccbe5700 1 --2- 172.21.15.77:0/3469559783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccb4003960 0x7fccb4003d70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fccc80715c0 tx=0x7fccc40075a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f55670442677698 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.547+0000 7fcccd3e6700 1 --2- 172.21.15.77:0/3469559783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccb4005090 0x7fccb40140f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-04-03T19:44:29.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.547+0000 7f33f4079700 1 -- 172.21.15.77:0/940753733 --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33ec06d560 con 0x7f33ec0c5410 2022-04-03T19:44:29.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.547+0000 7f33f4079700 1 -- 172.21.15.77:0/940753733 --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33ec06d840 con 0x7f33ec14f630 2022-04-03T19:44:29.611 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.547+0000 7fccbf7fe700 1 -- 172.21.15.77:0/3469559783 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc40078d0 con 0x7fccb4003960 2022-04-03T19:44:29.611 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.547+0000 7fccbf7fe700 1 -- 172.21.15.77:0/3469559783 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc400cc30 con 0x7fccb4003960 2022-04-03T19:44:29.611 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.547+0000 7fccbf7fe700 1 -- 172.21.15.77:0/3469559783 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc400d440 con 0x7fccb4003960 2022-04-03T19:44:29.612 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.547+0000 7f33f2616700 1 --2- 172.21.15.77:0/940753733 >> v2:172.21.15.186:3300/0 conn(0x7f33ec14f630 0x7f33ec155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.612 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.547+0000 7fccbffff700 1 --2- 172.21.15.77:0/3469559783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccb4004690 0x7fccb4004b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:29.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.547+0000 7fa712930700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa70c0678b0 0x7fa70c16d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.547+0000 7fa712930700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa70c0ba320 0x7fa70c155710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.547+0000 7fa712930700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa70c15d450 0x7fa70c155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.547+0000 7fa712930700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c16a110 con 0x7fa70c0678b0 2022-04-03T19:44:29.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.547+0000 7fa712930700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c16cdf0 con 0x7fa70c0ba320 2022-04-03T19:44:29.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.547+0000 7fa712930700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c16c810 con 0x7fa70c15d450 2022-04-03T19:44:29.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.547+0000 7fa71192e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa70c0678b0 0x7fa70c16d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.547+0000 7fa71192e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa70c0678b0 0x7fa70c16d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42568/0 (socket says 172.21.15.77:42568) 2022-04-03T19:44:29.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.547+0000 7fa71192e700 1 -- 172.21.15.77:0/1739124100 learned_addr learned my addr 172.21.15.77:0/1739124100 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:29.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.547+0000 7f33f2616700 1 -- 172.21.15.77:0/940753733 >> v1:172.21.15.186:6789/0 conn(0x7f33ec0c5410 legacy=0x7f33ec0c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.547+0000 7f33f2616700 1 -- 172.21.15.77:0/940753733 >> v1:172.21.15.77:6789/0 conn(0x7f33ec0c67e0 legacy=0x7f33ec14ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.548+0000 7f33f2616700 1 -- 172.21.15.77:0/940753733 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ec06db20 con 0x7f33ec14f630 2022-04-03T19:44:29.617 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.548+0000 7fa71112d700 1 --2- 172.21.15.77:0/1739124100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa70c0ba320 0x7fa70c155710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.617 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.548+0000 7fa71212f700 1 --2- 172.21.15.77:0/1739124100 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa70c15d450 0x7fa70c155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.618 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.548+0000 7f33f2616700 1 --2- 172.21.15.77:0/940753733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33ec14f630 0x7f33ec155ad0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f33e4009e90 tx=0x7f33e4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8725c201e3332f1a server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.618 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.548+0000 7f33f0e13700 1 -- 172.21.15.77:0/940753733 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e400e070 con 0x7f33ec14f630 2022-04-03T19:44:29.618 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.548+0000 7f300a3f3700 1 Processor -- start 2022-04-03T19:44:29.619 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.548+0000 7fcccee49700 1 -- 172.21.15.77:0/3469559783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccb4003960 msgr2=0x7fccb4003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.619 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.548+0000 7fcccee49700 1 --2- 172.21.15.77:0/3469559783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccb4003960 0x7fccb4003d70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fccc80715c0 tx=0x7fccc40075a0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.548+0000 7f33f0e13700 1 -- 172.21.15.77:0/940753733 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e400bc10 con 0x7f33ec14f630 2022-04-03T19:44:29.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.548+0000 7f33f0e13700 1 -- 172.21.15.77:0/940753733 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e4003860 con 0x7f33ec14f630 2022-04-03T19:44:29.621 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.548+0000 7f300a3f3700 1 -- start start 2022-04-03T19:44:29.621 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.549+0000 7fcccd3e6700 1 -- 172.21.15.77:0/3469559783 reap_dead start 2022-04-03T19:44:29.622 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.549+0000 7fa71112d700 1 -- 172.21.15.77:0/1739124100 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa70c15d450 msgr2=0x7fa70c155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.622 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.549+0000 7fa71112d700 1 --2- 172.21.15.77:0/1739124100 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa70c15d450 0x7fa70c155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.623 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.549+0000 7fa71112d700 1 -- 172.21.15.77:0/1739124100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa70c0678b0 msgr2=0x7fa70c16d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.549+0000 7fa71112d700 1 --2- 172.21.15.77:0/1739124100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa70c0678b0 0x7fa70c16d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.549+0000 7fa71112d700 1 -- 172.21.15.77:0/1739124100 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7080092a0 con 0x7fa70c0ba320 2022-04-03T19:44:29.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.549+0000 7fcccee49700 1 -- 172.21.15.77:0/3469559783 shutdown_connections 2022-04-03T19:44:29.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.549+0000 7fcccee49700 1 -- 172.21.15.77:0/3469559783 >> 172.21.15.77:0/3469559783 conn(0x7fccc8066da0 msgr2=0x7fccc80671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:29.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.549+0000 7fcccee49700 1 -- 172.21.15.77:0/3469559783 shutdown_connections 2022-04-03T19:44:29.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.549+0000 7fcccee49700 1 -- 172.21.15.77:0/3469559783 wait complete. 2022-04-03T19:44:29.626 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.549+0000 7f371effd700 1 -- 172.21.15.77:0/2551445156 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f373004fac0 con 0x7f3738157340 2022-04-03T19:44:29.627 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.550+0000 7f33f4079700 1 -- 172.21.15.77:0/940753733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33ec14f630 msgr2=0x7f33ec155ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.627 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.550+0000 7f33f4079700 1 --2- 172.21.15.77:0/940753733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33ec14f630 0x7f33ec155ad0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f33e4009e90 tx=0x7f33e4009780 comp rx=0 tx=0).stop 2022-04-03T19:44:29.627 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.550+0000 7f33f4079700 1 -- 172.21.15.77:0/940753733 shutdown_connections 2022-04-03T19:44:29.628 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.550+0000 7f33f4079700 1 --2- 172.21.15.77:0/940753733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33ec14f630 0x7f33ec155ad0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.628 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.550+0000 7f33f4079700 1 -- 172.21.15.77:0/940753733 >> 172.21.15.77:0/940753733 conn(0x7f33ec0cb800 msgr2=0x7f33ec0cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:29.628 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.550+0000 7f33f4079700 1 -- 172.21.15.77:0/940753733 shutdown_connections 2022-04-03T19:44:29.629 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.551+0000 7f300a3f3700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f30040b7410 0x7f30040c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.629 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.551+0000 7f300a3f3700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3004152340 con 0x7f30040c2870 2022-04-03T19:44:29.630 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.551+0000 7f300a3f3700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3004152620 con 0x7f300406a1a0 2022-04-03T19:44:29.630 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.551+0000 7f300a3f3700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3004152900 con 0x7f30040b7410 2022-04-03T19:44:29.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.551+0000 7fcccee49700 1 Processor -- start 2022-04-03T19:44:29.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.551+0000 7f3003fff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f30040b7410 0x7f30040c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.551+0000 7f3008990700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f300406a1a0 0x7f30040c01d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44124/0 (socket says 172.21.15.77:44124) 2022-04-03T19:44:29.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.551+0000 7f3008990700 1 -- 172.21.15.77:0/2629659491 learned_addr learned my addr 172.21.15.77:0/2629659491 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:29.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.551+0000 7fcccee49700 1 -- start start 2022-04-03T19:44:29.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.552+0000 7f3003fff700 1 -- 172.21.15.77:0/2629659491 >> v1:172.21.15.186:6789/0 conn(0x7f300406a1a0 legacy=0x7f30040c01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.552+0000 7f3003fff700 1 -- 172.21.15.77:0/2629659491 >> v1:172.21.15.77:6789/0 conn(0x7f30040c2870 legacy=0x7f3004069a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-04-03T19:44:29.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.552+0000 7f3003fff700 1 -- 172.21.15.77:0/2629659491 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3004152be0 con 0x7f30040b7410 2022-04-03T19:44:29.634 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.552+0000 7f33f4079700 1 -- 172.21.15.77:0/940753733 wait complete. 2022-04-03T19:44:29.635 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.552+0000 7f3003fff700 1 --2- 172.21.15.77:0/2629659491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30040b7410 0x7f30040c73c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800ba00 tx=0x7f2ff800b300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4ae89bdeaf385fb server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.635 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.552+0000 7f3002ffd700 1 -- 172.21.15.77:0/2629659491 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8004030 con 0x7f30040b7410 2022-04-03T19:44:29.635 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.552+0000 7f3002ffd700 1 -- 172.21.15.77:0/2629659491 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff8004190 con 0x7f30040b7410 2022-04-03T19:44:29.635 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.552+0000 7f3002ffd700 1 -- 172.21.15.77:0/2629659491 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff800da30 con 0x7f30040b7410 2022-04-03T19:44:29.636 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.550+0000 7fa71112d700 1 --2- 172.21.15.77:0/1739124100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa70c0ba320 0x7fa70c155710 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa70400c840 tx=0x7fa704007f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.636 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.550+0000 7fa6f9ffb700 1 -- 172.21.15.77:0/1739124100 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa704003a70 con 0x7fa70c0ba320 2022-04-03T19:44:29.637 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.550+0000 7fa6f9ffb700 1 -- 172.21.15.77:0/1739124100 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa704003bf0 con 0x7fa70c0ba320 2022-04-03T19:44:29.637 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.550+0000 7fa712930700 1 -- 172.21.15.77:0/1739124100 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa70c068b30 con 0x7fa70c0ba320 2022-04-03T19:44:29.637 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.550+0000 7fa6f9ffb700 1 -- 172.21.15.77:0/1739124100 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7040050d0 con 0x7fa70c0ba320 2022-04-03T19:44:29.637 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.550+0000 7fa712930700 1 -- 172.21.15.77:0/1739124100 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa70c16ca80 con 0x7fa70c0ba320 2022-04-03T19:44:29.639 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.553+0000 7fcccee49700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccb4003960 0x7fccc81611e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.639 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.553+0000 7fcccee49700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccc814ddd0 0x7fccc80b9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.640 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.553+0000 7fcccee49700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccc814e220 0x7fccc80b9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.640 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.553+0000 7fcccee49700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc814e9e0 con 0x7fccb4003960 2022-04-03T19:44:29.640 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.553+0000 7fcccee49700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc80c4370 con 0x7fccc814e220 2022-04-03T19:44:29.640 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.553+0000 7fcccee49700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc80c45d0 con 0x7fccc814ddd0 2022-04-03T19:44:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.553+0000 7fccccbe5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccb4003960 0x7fccc81611e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.553+0000 7fccccbe5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccb4003960 0x7fccc81611e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42570/0 (socket says 172.21.15.77:42570) 2022-04-03T19:44:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.553+0000 7fccccbe5700 1 -- 172.21.15.77:0/2505112689 learned_addr learned my addr 172.21.15.77:0/2505112689 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.553+0000 7fccbffff700 1 --2- 172.21.15.77:0/2505112689 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccc814ddd0 0x7fccc80b9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.641 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.554+0000 7fcccd3e6700 1 --2- 172.21.15.77:0/2505112689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccc814e220 0x7fccc80b9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.642 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.554+0000 7fccbffff700 1 -- 172.21.15.77:0/2505112689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccc814e220 msgr2=0x7fccc80b9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.642 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.554+0000 7fccbffff700 1 --2- 172.21.15.77:0/2505112689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccc814e220 0x7fccc80b9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.642 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.554+0000 7fccbffff700 1 -- 172.21.15.77:0/2505112689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccb4003960 msgr2=0x7fccc81611e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.642 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.554+0000 7fccbffff700 1 --2- 172.21.15.77:0/2505112689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccb4003960 0x7fccc81611e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.643 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.554+0000 7fccbffff700 1 -- 172.21.15.77:0/2505112689 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc80c4890 con 0x7fccc814ddd0 2022-04-03T19:44:29.643 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.554+0000 7fccccbe5700 1 --2- 172.21.15.77:0/2505112689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccb4003960 0x7fccc81611e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:29.643 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.554+0000 7f33f4079700 1 Processor -- start 2022-04-03T19:44:29.644 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.554+0000 7fccbffff700 1 --2- 172.21.15.77:0/2505112689 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccc814ddd0 0x7fccc80b9920 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fccc80b7b30 tx=0x7fccb8001da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.645 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.554+0000 7f5b9f7fe700 1 -- 172.21.15.77:0/1713341074 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f5ba8048ae0 con 0x7f5bb80c1ba0 2022-04-03T19:44:29.645 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.554+0000 7f33f4079700 1 -- start start 2022-04-03T19:44:29.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.555+0000 7f92c155e700 1 -- 172.21.15.77:0/361366402 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b400e070 con 0x7f92bc0c36a0 2022-04-03T19:44:29.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.555+0000 7f8541ffb700 1 -- 172.21.15.77:0/3051162908 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c003e40 con 0x7f85541682d0 2022-04-03T19:44:29.647 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.555+0000 7f92c155e700 1 -- 172.21.15.77:0/361366402 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b400bcc0 con 0x7f92bc0c36a0 2022-04-03T19:44:29.647 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.555+0000 7f92c155e700 1 -- 172.21.15.77:0/361366402 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b4012640 con 0x7f92bc0c36a0 2022-04-03T19:44:29.648 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.555+0000 7f8541ffb700 1 -- 172.21.15.77:0/3051162908 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854c0129b0 con 0x7f85541682d0 2022-04-03T19:44:29.648 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.556+0000 7f8541ffb700 1 -- 172.21.15.77:0/3051162908 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c012d70 con 0x7f85541682d0 2022-04-03T19:44:29.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.556+0000 7f92c3562700 1 -- 172.21.15.77:0/361366402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92bc0c36a0 msgr2=0x7f92bc06cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.556+0000 7f92c3562700 1 --2- 172.21.15.77:0/361366402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92bc0c36a0 0x7f92bc06cf40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f92b4009e60 tx=0x7f92b4009780 comp rx=0 tx=0).stop 2022-04-03T19:44:29.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.556+0000 7f33f4079700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33ec0c5410 0x7f33ec167e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.556+0000 7f855be21700 1 -- 172.21.15.77:0/3051162908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85540bf820 con 0x7f85541682d0 2022-04-03T19:44:29.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.554+0000 7fa6f9ffb700 1 -- 172.21.15.77:0/1739124100 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fa704019030 con 0x7fa70c0ba320 2022-04-03T19:44:29.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.556+0000 7fa6f9ffb700 1 --2- 172.21.15.77:0/1739124100 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa6ec063e60 0x7fa6ec066310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.556+0000 7fa6f9ffb700 1 -- 172.21.15.77:0/1739124100 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fa70407fdd0 con 0x7fa70c0ba320 2022-04-03T19:44:29.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.556+0000 7f33f4079700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33ec0c67e0 0x7f33ec168aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.556+0000 7f300a3f3700 1 -- 172.21.15.77:0/2629659491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30040b7410 msgr2=0x7f30040c73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.653 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.556+0000 7fa712930700 1 -- 172.21.15.77:0/1739124100 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa70c16cc10 con 0x7fa70c0ba320 2022-04-03T19:44:29.653 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.556+0000 7f33f4079700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33ec14f630 0x7f33ec168fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.654 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.556+0000 7f855be21700 1 -- 172.21.15.77:0/3051162908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8554151280 con 0x7f85541682d0 2022-04-03T19:44:29.654 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.556+0000 7f300a3f3700 1 --2- 172.21.15.77:0/2629659491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30040b7410 0x7f30040c73c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800ba00 tx=0x7f2ff800b300 comp rx=0 tx=0).stop 2022-04-03T19:44:29.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.556+0000 7f33f4079700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec1671b0 con 0x7f33ec14f630 2022-04-03T19:44:29.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.556+0000 7f33f4079700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec0b13a0 con 0x7f33ec0c5410 2022-04-03T19:44:29.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.556+0000 7f33f4079700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec163300 con 0x7f33ec0c67e0 2022-04-03T19:44:29.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.557+0000 7f33f1614700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33ec0c67e0 0x7f33ec168aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.656 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.557+0000 7f33f1614700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33ec0c67e0 0x7f33ec168aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59876/0 (socket says 172.21.15.77:59876) 2022-04-03T19:44:29.656 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.557+0000 7f33f1614700 1 -- 172.21.15.77:0/3761572414 learned_addr learned my addr 172.21.15.77:0/3761572414 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:29.656 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.557+0000 7f300a3f3700 1 -- 172.21.15.77:0/2629659491 shutdown_connections 2022-04-03T19:44:29.657 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.557+0000 7fa71192e700 1 --2- 172.21.15.77:0/1739124100 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa6ec063e60 0x7fa6ec066310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.657 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.557+0000 7f300a3f3700 1 --2- 172.21.15.77:0/2629659491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30040b7410 0x7f30040c73c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.658 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.557+0000 7f300a3f3700 1 -- 172.21.15.77:0/2629659491 >> 172.21.15.77:0/2629659491 conn(0x7f300406d900 msgr2=0x7f30040626d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:29.658 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.557+0000 7f33f1e15700 1 --2- 172.21.15.77:0/3761572414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33ec0c5410 0x7f33ec167e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.659 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.555+0000 7fccbcff9700 1 -- 172.21.15.77:0/2505112689 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb8018030 con 0x7fccc814ddd0 2022-04-03T19:44:29.659 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.555+0000 7fccbcff9700 1 -- 172.21.15.77:0/2505112689 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb800f960 con 0x7fccc814ddd0 2022-04-03T19:44:29.660 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.555+0000 7fccbcff9700 1 -- 172.21.15.77:0/2505112689 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb800fce0 con 0x7fccc814ddd0 2022-04-03T19:44:29.660 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.555+0000 7fcccee49700 1 -- 172.21.15.77:0/2505112689 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc80c2810 con 0x7fccc814ddd0 2022-04-03T19:44:29.660 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.555+0000 7fcccee49700 1 -- 172.21.15.77:0/2505112689 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc80c2dd0 con 0x7fccc814ddd0 2022-04-03T19:44:29.660 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.558+0000 7fa71192e700 1 --2- 172.21.15.77:0/1739124100 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa6ec063e60 0x7fa6ec066310 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc007fd0 tx=0x7fa6fc007df0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.661 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.558+0000 7f855be21700 1 -- 172.21.15.77:0/3051162908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85541561b0 con 0x7f85541682d0 2022-04-03T19:44:29.662 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.558+0000 7fcccee49700 1 -- 172.21.15.77:0/2505112689 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca4005130 con 0x7fccc814ddd0 2022-04-03T19:44:29.662 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.556+0000 7f92c3562700 1 -- 172.21.15.77:0/361366402 shutdown_connections 2022-04-03T19:44:29.662 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.556+0000 7f92c3562700 1 --2- 172.21.15.77:0/361366402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92bc0c36a0 0x7f92bc06cf40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.663 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.556+0000 7f92c3562700 1 -- 172.21.15.77:0/361366402 >> 172.21.15.77:0/361366402 conn(0x7f92bc0cb6b0 msgr2=0x7f92bc0cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:29.663 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.558+0000 7f33f1614700 1 -- 172.21.15.77:0/3761572414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33ec0c5410 msgr2=0x7f33ec167e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.558+0000 7f33f1614700 1 --2- 172.21.15.77:0/3761572414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33ec0c5410 0x7f33ec167e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.558+0000 7f33f1614700 1 -- 172.21.15.77:0/3761572414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33ec14f630 msgr2=0x7f33ec168fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.558+0000 7f33f1614700 1 --2- 172.21.15.77:0/3761572414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33ec14f630 0x7f33ec168fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.558+0000 7f33f1614700 1 -- 172.21.15.77:0/3761572414 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e4009a20 con 0x7f33ec0c67e0 2022-04-03T19:44:29.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.558+0000 7f33f1614700 1 --2- 172.21.15.77:0/3761572414 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33ec0c67e0 0x7f33ec168aa0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f33dc00c4f0 tx=0x7f33dc007e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.559+0000 7f300a3f3700 1 -- 172.21.15.77:0/2629659491 shutdown_connections 2022-04-03T19:44:29.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.559+0000 7f92c3562700 1 -- 172.21.15.77:0/361366402 shutdown_connections 2022-04-03T19:44:29.666 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.559+0000 7f33d9ffb700 1 -- 172.21.15.77:0/3761572414 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc0050d0 con 0x7f33ec0c67e0 2022-04-03T19:44:29.666 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.559+0000 7f33f4079700 1 -- 172.21.15.77:0/3761572414 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33ec0b0e20 con 0x7f33ec0c67e0 2022-04-03T19:44:29.667 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.559+0000 7f300a3f3700 1 -- 172.21.15.77:0/2629659491 wait complete. 2022-04-03T19:44:29.667 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.559+0000 7f33f4079700 1 -- 172.21.15.77:0/3761572414 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33ec0b10c0 con 0x7f33ec0c67e0 2022-04-03T19:44:29.668 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.560+0000 7f33d9ffb700 1 -- 172.21.15.77:0/3761572414 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33dc005230 con 0x7f33ec0c67e0 2022-04-03T19:44:29.668 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.560+0000 7f33d9ffb700 1 -- 172.21.15.77:0/3761572414 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc0117a0 con 0x7f33ec0c67e0 2022-04-03T19:44:29.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.560+0000 7fccbcff9700 1 -- 172.21.15.77:0/2505112689 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fccb8013600 con 0x7fccc814ddd0 2022-04-03T19:44:29.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.561+0000 7fccbcff9700 1 --2- 172.21.15.77:0/2505112689 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcca8063e60 0x7fcca8066310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.561+0000 7fccbcff9700 1 -- 172.21.15.77:0/2505112689 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fccb8019070 con 0x7fccc814ddd0 2022-04-03T19:44:29.670 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.562+0000 7f33d9ffb700 1 -- 172.21.15.77:0/3761572414 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f33dc019030 con 0x7f33ec0c67e0 2022-04-03T19:44:29.670 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.561+0000 7f8541ffb700 1 -- 172.21.15.77:0/3051162908 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f854c004250 con 0x7f85541682d0 2022-04-03T19:44:29.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.563+0000 7fa6f9ffb700 1 -- 172.21.15.77:0/1739124100 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fa704017020 con 0x7fa70c0ba320 2022-04-03T19:44:29.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.562+0000 7fccccbe5700 1 --2- 172.21.15.77:0/2505112689 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcca8063e60 0x7fcca8066310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.559+0000 7f92c3562700 1 -- 172.21.15.77:0/361366402 wait complete. 2022-04-03T19:44:29.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.561+0000 7f92c3562700 1 Processor -- start 2022-04-03T19:44:29.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.563+0000 7f92c3562700 1 -- start start 2022-04-03T19:44:29.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.564+0000 7f92c3562700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92bc06daf0 0x7f92bc0bf290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.673 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.563+0000 7f33f4079700 1 -- 172.21.15.77:0/3761572414 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c8005130 con 0x7f33ec0c67e0 2022-04-03T19:44:29.673 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.564+0000 7f33d9ffb700 1 --2- 172.21.15.77:0/3761572414 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f33cc063db0 0x7f33cc066260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.674 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.563+0000 7f300a3f3700 1 Processor -- start 2022-04-03T19:44:29.674 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.563+0000 7f300a3f3700 1 -- start start 2022-04-03T19:44:29.674 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.565+0000 7f300a3f3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30040b7410 0x7f3004164d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.562+0000 7f8541ffb700 1 --2- 172.21.15.77:0/3051162908 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8534063db0 0x7f8534066260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.563+0000 7f8541ffb700 1 -- 172.21.15.77:0/3051162908 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f854c083090 con 0x7f85541682d0 2022-04-03T19:44:29.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.563+0000 7f8559bbd700 1 --2- 172.21.15.77:0/3051162908 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8534063db0 0x7f8534066260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.676 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.565+0000 7f300a3f3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30040c2870 0x7f300415ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.676 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.565+0000 7f3003fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30040b7410 0x7f3004164d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.676 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.566+0000 7fccbcff9700 1 -- 172.21.15.77:0/2505112689 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fccb804f820 con 0x7fccc814ddd0 2022-04-03T19:44:29.677 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.565+0000 7f92c3562700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92bc0c36a0 0x7f92bc0c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.677 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.565+0000 7f92c3562700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92bc0c5f80 0x7f92bc1695b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.565+0000 7f92c3562700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc169120 con 0x7f92bc06daf0 2022-04-03T19:44:29.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.566+0000 7fccccbe5700 1 --2- 172.21.15.77:0/2505112689 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcca8063e60 0x7fcca8066310 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fccc4001190 tx=0x7fccc4017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.565+0000 7f3003fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30040b7410 0x7f3004164d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37764/0 (socket says 172.21.15.77:37764) 2022-04-03T19:44:29.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.566+0000 7f30037fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30040c2870 0x7f300415ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.680 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.566+0000 7f30037fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30040c2870 0x7f300415ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42576/0 (socket says 172.21.15.77:42576) 2022-04-03T19:44:29.680 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.566+0000 7f92c3562700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc16ce50 con 0x7f92bc0c5f80 2022-04-03T19:44:29.680 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.566+0000 7f92c3562700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92bc168dd0 con 0x7f92bc0c36a0 2022-04-03T19:44:29.681 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.566+0000 7f92c2560700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92bc06daf0 0x7f92bc0bf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.681 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.566+0000 7f92c2560700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92bc06daf0 0x7f92bc0bf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42574/0 (socket says 172.21.15.77:42574) 2022-04-03T19:44:29.681 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.566+0000 7f92c2560700 1 -- 172.21.15.77:0/49991705 learned_addr learned my addr 172.21.15.77:0/49991705 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:29.681 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.566+0000 7f92c1d5f700 1 --2- 172.21.15.77:0/49991705 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92bc0c36a0 0x7f92bc0c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.682 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.565+0000 7f33f1e15700 1 --2- 172.21.15.77:0/3761572414 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f33cc063db0 0x7f33cc066260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.682 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.565+0000 7f33d9ffb700 1 -- 172.21.15.77:0/3761572414 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f33dc0586c0 con 0x7f33ec0c67e0 2022-04-03T19:44:29.683 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.567+0000 7f92c2d61700 1 --2- 172.21.15.77:0/49991705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92bc0c5f80 0x7f92bc1695b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.684 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.566+0000 7f300a3f3700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3004160cb0 0x7f30040b65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.684 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.566+0000 7f300a3f3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3004146e10 con 0x7f30040c2870 2022-04-03T19:44:29.684 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.566+0000 7f300a3f3700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30041648f0 con 0x7f30040b7410 2022-04-03T19:44:29.684 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.566+0000 7f300a3f3700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3004161510 con 0x7f3004160cb0 2022-04-03T19:44:29.685 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.567+0000 7f3003fff700 1 -- 172.21.15.77:0/3588541447 learned_addr learned my addr 172.21.15.77:0/3588541447 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:29.685 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.567+0000 7f3003fff700 1 -- 172.21.15.77:0/3588541447 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3004160cb0 msgr2=0x7f30040b65a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2022-04-03T19:44:29.685 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.567+0000 7f3003fff700 1 --2- 172.21.15.77:0/3588541447 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3004160cb0 0x7f30040b65a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.685 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.567+0000 7f3003fff700 1 -- 172.21.15.77:0/3588541447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30040c2870 msgr2=0x7f300415ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.686 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.567+0000 7f3003fff700 1 --2- 172.21.15.77:0/3588541447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30040c2870 0x7f300415ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.686 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.567+0000 7f3003fff700 1 -- 172.21.15.77:0/3588541447 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff800b540 con 0x7f30040b7410 2022-04-03T19:44:29.686 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.567+0000 7f30037fe700 1 --2- 172.21.15.77:0/3588541447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30040c2870 0x7f300415ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2022-04-03T19:44:29.686 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.568+0000 7f3003fff700 1 --2- 172.21.15.77:0/3588541447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30040b7410 0x7f3004164d80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8004000 tx=0x7f2ff801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.687 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.568+0000 7f92c2560700 1 -- 172.21.15.77:0/49991705 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92bc0c36a0 msgr2=0x7f92bc0c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.687 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.568+0000 7f92c2560700 1 --2- 172.21.15.77:0/49991705 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92bc0c36a0 0x7f92bc0c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.568+0000 7f92c2560700 1 -- 172.21.15.77:0/49991705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92bc0c5f80 msgr2=0x7f92bc1695b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.568+0000 7f92c2560700 1 --2- 172.21.15.77:0/49991705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92bc0c5f80 0x7f92bc1695b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.568+0000 7f92c2560700 1 -- 172.21.15.77:0/49991705 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b4009a20 con 0x7f92bc06daf0 2022-04-03T19:44:29.689 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.568+0000 7f2febfff700 1 -- 172.21.15.77:0/3588541447 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8004300 con 0x7f30040b7410 2022-04-03T19:44:29.689 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.568+0000 7f2febfff700 1 -- 172.21.15.77:0/3588541447 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff8004480 con 0x7f30040b7410 2022-04-03T19:44:29.690 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.568+0000 7f2febfff700 1 -- 172.21.15.77:0/3588541447 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff80106a0 con 0x7f30040b7410 2022-04-03T19:44:29.690 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.569+0000 7f92c2560700 1 --2- 172.21.15.77:0/49991705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92bc06daf0 0x7f92bc0bf290 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f92b800c810 tx=0x7f92b8007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.691 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.570+0000 7f8559bbd700 1 --2- 172.21.15.77:0/3051162908 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8534063db0 0x7f8534066260 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8550007fd0 tx=0x7f8550007b80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.691 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.570+0000 7f33f1e15700 1 --2- 172.21.15.77:0/3761572414 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f33cc063db0 0x7f33cc066260 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f33e8007fd0 tx=0x7f33e8007db0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.692 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.570+0000 7f300a3f3700 1 -- 172.21.15.77:0/3588541447 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30041617f0 con 0x7f30040b7410 2022-04-03T19:44:29.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.570+0000 7f300a3f3700 1 -- 172.21.15.77:0/3588541447 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30041642a0 con 0x7f30040b7410 2022-04-03T19:44:29.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.571+0000 7f92aa7fc700 1 -- 172.21.15.77:0/49991705 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b80070f0 con 0x7f92bc06daf0 2022-04-03T19:44:29.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.571+0000 7f92aa7fc700 1 -- 172.21.15.77:0/49991705 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b8007250 con 0x7f92bc06daf0 2022-04-03T19:44:29.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.571+0000 7f92aa7fc700 1 -- 172.21.15.77:0/49991705 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b80075d0 con 0x7f92bc06daf0 2022-04-03T19:44:29.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.571+0000 7f92c3562700 1 -- 172.21.15.77:0/49991705 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92bc0bfca0 con 0x7f92bc06daf0 2022-04-03T19:44:29.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.572+0000 7f8541ffb700 1 -- 172.21.15.77:0/3051162908 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f854c055630 con 0x7f85541682d0 2022-04-03T19:44:29.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.571+0000 7f92c3562700 1 -- 172.21.15.77:0/49991705 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92bc16a2a0 con 0x7f92bc06daf0 2022-04-03T19:44:29.696 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.572+0000 7f300a3f3700 1 -- 172.21.15.77:0/3588541447 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f300404b890 con 0x7f30040b7410 2022-04-03T19:44:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.573+0000 7f2febfff700 1 -- 172.21.15.77:0/3588541447 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f2ff8023420 con 0x7f30040b7410 2022-04-03T19:44:29.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.573+0000 7f33d9ffb700 1 -- 172.21.15.77:0/3761572414 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f33dc048ae0 con 0x7f33ec0c67e0 2022-04-03T19:44:29.698 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.574+0000 7f92c3562700 1 -- 172.21.15.77:0/49991705 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a0005130 con 0x7f92bc06daf0 2022-04-03T19:44:29.698 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.574+0000 7f92aa7fc700 1 -- 172.21.15.77:0/49991705 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f92b8010070 con 0x7f92bc06daf0 2022-04-03T19:44:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.575+0000 7f2febfff700 1 --2- 172.21.15.77:0/3588541447 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2fe4063d90 0x7f2fe4066240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.699 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.576+0000 7f92aa7fc700 1 --2- 172.21.15.77:0/49991705 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f929c063d10 0x7f929c0661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:29.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.576+0000 7f92aa7fc700 1 -- 172.21.15.77:0/49991705 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f92b807ff70 con 0x7f92bc06daf0 2022-04-03T19:44:29.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.577+0000 7f92c1d5f700 1 --2- 172.21.15.77:0/49991705 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f929c063d10 0x7f929c0661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.576+0000 7f30037fe700 1 --2- 172.21.15.77:0/3588541447 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2fe4063d90 0x7f2fe4066240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:29.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.576+0000 7f2febfff700 1 -- 172.21.15.77:0/3588541447 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f2ff8014070 con 0x7f30040b7410 2022-04-03T19:44:29.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.578+0000 7f30037fe700 1 --2- 172.21.15.77:0/3588541447 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2fe4063d90 0x7f2fe4066240 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3004152880 tx=0x7f2ff4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.579+0000 7f92c1d5f700 1 --2- 172.21.15.77:0/49991705 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f929c063d10 0x7f929c0661c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f92ac007fd0 tx=0x7f92ac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:29.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.582+0000 7f2febfff700 1 -- 172.21.15.77:0/3588541447 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f2ff8053510 con 0x7f30040b7410 2022-04-03T19:44:29.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.584+0000 7f92aa7fc700 1 -- 172.21.15.77:0/49991705 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f92b804ebd0 con 0x7f92bc06daf0 2022-04-03T19:44:29.966 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.966+0000 7f371cff9700 1 -- 172.21.15.77:0/2551445156 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f370c0053e0 con 0x7f3738157340 2022-04-03T19:44:29.967 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.967+0000 7f371effd700 1 -- 172.21.15.77:0/2551445156 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f37300530e0 con 0x7f3738157340 2022-04-03T19:44:29.968 INFO:teuthology.orchestra.run.smithi077.stdout:55834574856 2022-04-03T19:44:29.969 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.968+0000 7f5b9d7fa700 1 -- 172.21.15.77:0/1713341074 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f5bb8070c50 con 0x7f5bb80c1ba0 2022-04-03T19:44:29.970 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.969+0000 7f5b9f7fe700 1 -- 172.21.15.77:0/1713341074 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5ba8017070 con 0x7f5bb80c1ba0 2022-04-03T19:44:29.970 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2022-04-03T19:44:29.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.972+0000 7f373d64e700 1 -- 172.21.15.77:0/2551445156 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3718063e30 msgr2=0x7f37180662e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.973 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.972+0000 7f373d64e700 1 --2- 172.21.15.77:0/2551445156 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3718063e30 0x7f37180662e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f372c0095a0 tx=0x7f372c006a60 comp rx=0 tx=0).stop 2022-04-03T19:44:29.974 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.973+0000 7f373d64e700 1 -- 172.21.15.77:0/2551445156 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3738157340 msgr2=0x7f37381542f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.974 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.973+0000 7f373d64e700 1 --2- 172.21.15.77:0/2551445156 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3738157340 0x7f37381542f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f373000c0f0 tx=0x7f373000b040 comp rx=0 tx=0).stop 2022-04-03T19:44:29.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.974+0000 7f373d64e700 1 -- 172.21.15.77:0/2551445156 shutdown_connections 2022-04-03T19:44:29.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.974+0000 7f373d64e700 1 --2- 172.21.15.77:0/2551445156 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3738157340 0x7f37381542f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.974+0000 7f373d64e700 1 --2- 172.21.15.77:0/2551445156 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3718063e30 0x7f37180662e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.974+0000 7f373d64e700 1 --2- 172.21.15.77:0/2551445156 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f373806a440 0x7f373815f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.974+0000 7f373d64e700 1 --2- 172.21.15.77:0/2551445156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3738156f70 0x7f3738164820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.974+0000 7f373d64e700 1 -- 172.21.15.77:0/2551445156 >> 172.21.15.77:0/2551445156 conn(0x7f3738068280 msgr2=0x7f37380c5fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:29.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.974+0000 7f373d64e700 1 -- 172.21.15.77:0/2551445156 shutdown_connections 2022-04-03T19:44:29.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.974+0000 7f373d64e700 1 -- 172.21.15.77:0/2551445156 wait complete. 2022-04-03T19:44:29.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.979+0000 7f5bbcfbf700 1 -- 172.21.15.77:0/1713341074 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5b98063de0 msgr2=0x7f5b98066290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.979+0000 7f5bbcfbf700 1 --2- 172.21.15.77:0/1713341074 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5b98063de0 0x7f5b98066290 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8124710 tx=0x7f5bb000b040 comp rx=0 tx=0).stop 2022-04-03T19:44:29.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.979+0000 7f5bbcfbf700 1 -- 172.21.15.77:0/1713341074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bb80c1ba0 msgr2=0x7f5bb80b9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.979+0000 7f5bbcfbf700 1 --2- 172.21.15.77:0/1713341074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bb80c1ba0 0x7f5bb80b9fb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8007fd0 tx=0x7f5ba8007df0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.980+0000 7f5bbcfbf700 1 -- 172.21.15.77:0/1713341074 shutdown_connections 2022-04-03T19:44:29.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.980+0000 7f5bbcfbf700 1 --2- 172.21.15.77:0/1713341074 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5b98063de0 0x7f5b98066290 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.980+0000 7f5bbcfbf700 1 --2- 172.21.15.77:0/1713341074 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb815cb70 0x7f5bb80b5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.981+0000 7f5bbcfbf700 1 --2- 172.21.15.77:0/1713341074 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5bb80c1ba0 0x7f5bb80b9fb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.981+0000 7f5bbcfbf700 1 --2- 172.21.15.77:0/1713341074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bb80c2680 0x7f5bb80b5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.981+0000 7f5bbcfbf700 1 -- 172.21.15.77:0/1713341074 >> 172.21.15.77:0/1713341074 conn(0x7f5bb8066da0 msgr2=0x7f5bb80c1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:29.983 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.981+0000 7f5bbcfbf700 1 -- 172.21.15.77:0/1713341074 shutdown_connections 2022-04-03T19:44:29.983 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.981+0000 7f5bbcfbf700 1 -- 172.21.15.77:0/1713341074 wait complete. 2022-04-03T19:44:29.984 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.982+0000 7fcccee49700 1 -- 172.21.15.77:0/2505112689 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fcca40053e0 con 0x7fccc814ddd0 2022-04-03T19:44:29.984 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.983+0000 7fccbcff9700 1 -- 172.21.15.77:0/2505112689 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fccb8052e40 con 0x7fccc814ddd0 2022-04-03T19:44:29.985 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2022-04-03T19:44:29.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.987+0000 7fcccee49700 1 -- 172.21.15.77:0/2505112689 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcca8063e60 msgr2=0x7fcca8066310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.988+0000 7fcccee49700 1 --2- 172.21.15.77:0/2505112689 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcca8063e60 0x7fcca8066310 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fccc4001190 tx=0x7fccc4017000 comp rx=0 tx=0).stop 2022-04-03T19:44:29.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.988+0000 7fcccee49700 1 -- 172.21.15.77:0/2505112689 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccc814ddd0 msgr2=0x7fccc80b9920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:29.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.988+0000 7fcccee49700 1 --2- 172.21.15.77:0/2505112689 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccc814ddd0 0x7fccc80b9920 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fccc80b7b30 tx=0x7fccb8001da0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.992 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.991+0000 7fcccee49700 1 -- 172.21.15.77:0/2505112689 shutdown_connections 2022-04-03T19:44:29.992 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.991+0000 7fa712930700 1 -- 172.21.15.77:0/1739124100 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa70c16cf30 con 0x7fa70c0ba320 2022-04-03T19:44:29.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.991+0000 7fcccee49700 1 --2- 172.21.15.77:0/2505112689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccc814e220 0x7fccc80b9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.991+0000 7fcccee49700 1 --2- 172.21.15.77:0/2505112689 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fccb4003960 0x7fccc81611e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.994 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.991+0000 7fcccee49700 1 --2- 172.21.15.77:0/2505112689 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcca8063e60 0x7fcca8066310 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fccc4001190 tx=0x7fccc4017000 comp rx=0 tx=0).stop 2022-04-03T19:44:29.995 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.991+0000 7fcccee49700 1 --2- 172.21.15.77:0/2505112689 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccc814ddd0 0x7fccc80b9920 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:29.995 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.991+0000 7fcccee49700 1 -- 172.21.15.77:0/2505112689 >> 172.21.15.77:0/2505112689 conn(0x7fccc8066da0 msgr2=0x7fccc8156350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:29.995 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.992+0000 7fcccee49700 1 -- 172.21.15.77:0/2505112689 shutdown_connections 2022-04-03T19:44:29.995 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.992+0000 7fcccee49700 1 -- 172.21.15.77:0/2505112689 wait complete. 2022-04-03T19:44:29.996 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.992+0000 7f855be21700 1 -- 172.21.15.77:0/3051162908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f855404b050 con 0x7f85541682d0 2022-04-03T19:44:29.996 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2022-04-03T19:44:29.997 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.993+0000 7fa6f9ffb700 1 -- 172.21.15.77:0/1739124100 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa704052370 con 0x7fa70c0ba320 2022-04-03T19:44:29.998 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.993+0000 7f8541ffb700 1 -- 172.21.15.77:0/3051162908 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f854c055220 con 0x7f85541682d0 2022-04-03T19:44:29.998 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2022-04-03T19:44:29.999 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.994+0000 7f300a3f3700 1 -- 172.21.15.77:0/3588541447 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f300406a620 con 0x7f30040b7410 2022-04-03T19:44:29.999 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.994+0000 7f2febfff700 1 -- 172.21.15.77:0/3588541447 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2ff8084030 con 0x7f30040b7410 2022-04-03T19:44:30.000 INFO:teuthology.orchestra.run.smithi077.stdout:55834574854 2022-04-03T19:44:30.001 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.996+0000 7fa712930700 1 -- 172.21.15.77:0/1739124100 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa6ec063e60 msgr2=0x7fa6ec066310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:30.001 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.996+0000 7fa712930700 1 --2- 172.21.15.77:0/1739124100 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa6ec063e60 0x7fa6ec066310 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc007fd0 tx=0x7fa6fc007df0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.997+0000 7fa712930700 1 -- 172.21.15.77:0/1739124100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa70c0ba320 msgr2=0x7fa70c155710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:30.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.997+0000 7fa712930700 1 --2- 172.21.15.77:0/1739124100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa70c0ba320 0x7fa70c155710 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa70400c840 tx=0x7fa704007f50 comp rx=0 tx=0).stop 2022-04-03T19:44:30.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.997+0000 7fa712930700 1 -- 172.21.15.77:0/1739124100 shutdown_connections 2022-04-03T19:44:30.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.997+0000 7fa712930700 1 --2- 172.21.15.77:0/1739124100 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa6ec063e60 0x7fa6ec066310 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.997+0000 7fa712930700 1 --2- 172.21.15.77:0/1739124100 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa70c15d450 0x7fa70c155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.997+0000 7fa712930700 1 --2- 172.21.15.77:0/1739124100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa70c0678b0 0x7fa70c16d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.997+0000 7fa712930700 1 --2- 172.21.15.77:0/1739124100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa70c0ba320 0x7fa70c155710 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.997+0000 7fa712930700 1 -- 172.21.15.77:0/1739124100 >> 172.21.15.77:0/1739124100 conn(0x7fa70c066da0 msgr2=0x7fa70c14c770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:30.004 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.997+0000 7f855be21700 1 -- 172.21.15.77:0/3051162908 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8534063db0 msgr2=0x7f8534066260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:30.004 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.997+0000 7f855be21700 1 --2- 172.21.15.77:0/3051162908 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8534063db0 0x7f8534066260 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8550007fd0 tx=0x7f8550007b80 comp rx=0 tx=0).stop 2022-04-03T19:44:30.004 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.997+0000 7fa712930700 1 -- 172.21.15.77:0/1739124100 shutdown_connections 2022-04-03T19:44:30.005 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.998+0000 7f855be21700 1 -- 172.21.15.77:0/3051162908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85541682d0 msgr2=0x7f8554168740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:30.005 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.998+0000 7f855be21700 1 --2- 172.21.15.77:0/3051162908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85541682d0 0x7f8554168740 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f854c016000 tx=0x7f854c019000 comp rx=0 tx=0).stop 2022-04-03T19:44:30.006 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.997+0000 7fa712930700 1 -- 172.21.15.77:0/1739124100 wait complete. 2022-04-03T19:44:30.006 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.998+0000 7f855be21700 1 -- 172.21.15.77:0/3051162908 shutdown_connections 2022-04-03T19:44:30.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.998+0000 7f855be21700 1 --2- 172.21.15.77:0/3051162908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85541682d0 0x7f8554168740 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.998+0000 7f855be21700 1 --2- 172.21.15.77:0/3051162908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f855406c430 0x7f8554151d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.998+0000 7f855be21700 1 --2- 172.21.15.77:0/3051162908 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8534063db0 0x7f8534066260 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.998+0000 7f855be21700 1 --2- 172.21.15.77:0/3051162908 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f855406c800 0x7f85541522a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.008 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.998+0000 7f855be21700 1 -- 172.21.15.77:0/3051162908 >> 172.21.15.77:0/3051162908 conn(0x7f85540cb700 msgr2=0x7f85540c9690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:30.008 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.999+0000 7f2fe9ffb700 1 -- 172.21.15.77:0/3588541447 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2fe4063d90 msgr2=0x7f2fe4066240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:30.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.999+0000 7f855be21700 1 -- 172.21.15.77:0/3051162908 shutdown_connections 2022-04-03T19:44:30.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.000+0000 7f855be21700 1 -- 172.21.15.77:0/3051162908 wait complete. 2022-04-03T19:44:30.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.999+0000 7f2fe9ffb700 1 --2- 172.21.15.77:0/3588541447 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2fe4063d90 0x7f2fe4066240 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3004152880 tx=0x7f2ff4008040 comp rx=0 tx=0).stop 2022-04-03T19:44:30.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.999+0000 7f2fe9ffb700 1 -- 172.21.15.77:0/3588541447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30040b7410 msgr2=0x7f3004164d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:30.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:29.999+0000 7f2fe9ffb700 1 --2- 172.21.15.77:0/3588541447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30040b7410 0x7f3004164d80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8004000 tx=0x7f2ff801a000 comp rx=0 tx=0).stop 2022-04-03T19:44:30.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.000+0000 7f2fe9ffb700 1 -- 172.21.15.77:0/3588541447 shutdown_connections 2022-04-03T19:44:30.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.000+0000 7f2fe9ffb700 1 --2- 172.21.15.77:0/3588541447 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2fe4063d90 0x7f2fe4066240 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.000+0000 7f2fe9ffb700 1 --2- 172.21.15.77:0/3588541447 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3004160cb0 0x7f30040b65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.000+0000 7f2fe9ffb700 1 --2- 172.21.15.77:0/3588541447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30040b7410 0x7f3004164d80 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8004000 tx=0x7f2ff801a000 comp rx=0 tx=0).stop 2022-04-03T19:44:30.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.000+0000 7f2fe9ffb700 1 --2- 172.21.15.77:0/3588541447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30040c2870 0x7f300415ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.000+0000 7f2fe9ffb700 1 -- 172.21.15.77:0/3588541447 >> 172.21.15.77:0/3588541447 conn(0x7f300406d900 msgr2=0x7f30040c2da0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:30.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.000+0000 7f33f4079700 1 -- 172.21.15.77:0/3761572414 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f33c80053e0 con 0x7f33ec0c67e0 2022-04-03T19:44:30.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.001+0000 7f2fe9ffb700 1 -- 172.21.15.77:0/3588541447 shutdown_connections 2022-04-03T19:44:30.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.001+0000 7f2fe9ffb700 1 -- 172.21.15.77:0/3588541447 wait complete. 2022-04-03T19:44:30.014 INFO:teuthology.orchestra.run.smithi077.stdout:55834574857 2022-04-03T19:44:30.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.001+0000 7f33d9ffb700 1 -- 172.21.15.77:0/3761572414 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f33dc017070 con 0x7f33ec0c67e0 2022-04-03T19:44:30.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.005+0000 7f33f4079700 1 -- 172.21.15.77:0/3761572414 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f33cc063db0 msgr2=0x7f33cc066260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:30.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.005+0000 7f33f4079700 1 --2- 172.21.15.77:0/3761572414 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f33cc063db0 0x7f33cc066260 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f33e8007fd0 tx=0x7f33e8007db0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.005+0000 7f33f4079700 1 -- 172.21.15.77:0/3761572414 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33ec0c67e0 msgr2=0x7f33ec168aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:30.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.005+0000 7f33f4079700 1 --2- 172.21.15.77:0/3761572414 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33ec0c67e0 0x7f33ec168aa0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f33dc00c4f0 tx=0x7f33dc007e00 comp rx=0 tx=0).stop 2022-04-03T19:44:30.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.005+0000 7f92c3562700 1 -- 172.21.15.77:0/49991705 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f92a00053e0 con 0x7f92bc06daf0 2022-04-03T19:44:30.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.010+0000 7f33f4079700 1 -- 172.21.15.77:0/3761572414 shutdown_connections 2022-04-03T19:44:30.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.010+0000 7f33f4079700 1 --2- 172.21.15.77:0/3761572414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33ec14f630 0x7f33ec168fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.010+0000 7f33f4079700 1 --2- 172.21.15.77:0/3761572414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33ec0c5410 0x7f33ec167e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.010+0000 7f33f4079700 1 --2- 172.21.15.77:0/3761572414 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f33cc063db0 0x7f33cc066260 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.010+0000 7f33f4079700 1 --2- 172.21.15.77:0/3761572414 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33ec0c67e0 0x7f33ec168aa0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.010+0000 7f33f4079700 1 -- 172.21.15.77:0/3761572414 >> 172.21.15.77:0/3761572414 conn(0x7f33ec0cb800 msgr2=0x7f33ec1548b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:30.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.011+0000 7f33f4079700 1 -- 172.21.15.77:0/3761572414 shutdown_connections 2022-04-03T19:44:30.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.011+0000 7f33f4079700 1 -- 172.21.15.77:0/3761572414 wait complete. 2022-04-03T19:44:30.019 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.012+0000 7f92aa7fc700 1 -- 172.21.15.77:0/49991705 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f92b80521f0 con 0x7f92bc06daf0 2022-04-03T19:44:30.019 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2022-04-03T19:44:30.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.017+0000 7f92c3562700 1 -- 172.21.15.77:0/49991705 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f929c063d10 msgr2=0x7f929c0661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:30.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.017+0000 7f92c3562700 1 --2- 172.21.15.77:0/49991705 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f929c063d10 0x7f929c0661c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f92ac007fd0 tx=0x7f92ac009040 comp rx=0 tx=0).stop 2022-04-03T19:44:30.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.017+0000 7f92c3562700 1 -- 172.21.15.77:0/49991705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92bc06daf0 msgr2=0x7f92bc0bf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:30.021 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.017+0000 7f92c3562700 1 --2- 172.21.15.77:0/49991705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92bc06daf0 0x7f92bc0bf290 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f92b800c810 tx=0x7f92b8007df0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.021 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.017+0000 7f92c3562700 1 -- 172.21.15.77:0/49991705 shutdown_connections 2022-04-03T19:44:30.021 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.017+0000 7f92c3562700 1 --2- 172.21.15.77:0/49991705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92bc0c5f80 0x7f92bc1695b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.021 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.017+0000 7f92c3562700 1 --2- 172.21.15.77:0/49991705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92bc06daf0 0x7f92bc0bf290 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.017+0000 7f92c3562700 1 --2- 172.21.15.77:0/49991705 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f929c063d10 0x7f929c0661c0 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f92ac007fd0 tx=0x7f92ac009040 comp rx=0 tx=0).stop 2022-04-03T19:44:30.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.017+0000 7f92c3562700 1 --2- 172.21.15.77:0/49991705 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92bc0c36a0 0x7f92bc0c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:30.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.017+0000 7f92c3562700 1 -- 172.21.15.77:0/49991705 >> 172.21.15.77:0/49991705 conn(0x7f92bc0cb6b0 msgr2=0x7f92bc1532b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:30.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.018+0000 7f92c3562700 1 -- 172.21.15.77:0/49991705 shutdown_connections 2022-04-03T19:44:30.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:30.019+0000 7f92c3562700 1 -- 172.21.15.77:0/49991705 wait complete. 2022-04-03T19:44:30.023 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.1 2022-04-03T19:44:30.025 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.0 2022-04-03T19:44:30.039 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.6 2022-04-03T19:44:30.040 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.5 2022-04-03T19:44:30.041 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.3 2022-04-03T19:44:30.043 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.4 2022-04-03T19:44:30.052 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.2 2022-04-03T19:44:30.057 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.7 2022-04-03T19:44:31.025 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2022-04-03T19:44:31.025 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2022-04-03T19:44:31.039 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2022-04-03T19:44:31.040 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2022-04-03T19:44:31.041 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2022-04-03T19:44:31.044 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2022-04-03T19:44:31.053 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2022-04-03T19:44:31.057 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2022-04-03T19:44:31.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.368+0000 7f59fbfff700 1 Processor -- start 2022-04-03T19:44:31.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.369+0000 7f59fbfff700 1 -- start start 2022-04-03T19:44:31.373 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.370+0000 7f59fbfff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f59f40dcd40 0x7f59f40dee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.373 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.370+0000 7f59fbfff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f40dfd20 con 0x7f59f404f3e0 2022-04-03T19:44:31.373 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.371+0000 7f59fbfff700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f40dbd80 con 0x7f59f4049640 2022-04-03T19:44:31.374 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.371+0000 7f59fbfff700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59f404ac60 con 0x7f59f40dcd40 2022-04-03T19:44:31.374 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.372+0000 7f59fa7fc700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f59f40dcd40 0x7f59f40dee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.374 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.372+0000 7f59fa7fc700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f59f40dcd40 0x7f59f40dee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37768/0 (socket says 172.21.15.77:37768) 2022-04-03T19:44:31.374 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.373+0000 7f877cb2c700 1 Processor -- start 2022-04-03T19:44:31.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.372+0000 7f59fa7fc700 1 -- 172.21.15.77:0/640775837 learned_addr learned my addr 172.21.15.77:0/640775837 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:31.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.373+0000 7f59fa7fc700 1 -- 172.21.15.77:0/640775837 >> v1:172.21.15.186:6789/0 conn(0x7f59f4049640 legacy=0x7f59f400d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-04-03T19:44:31.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.373+0000 7f59fa7fc700 1 -- 172.21.15.77:0/640775837 >> v1:172.21.15.77:6789/0 conn(0x7f59f404f3e0 legacy=0x7f59f40dc210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-04-03T19:44:31.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.373+0000 7f59fa7fc700 1 -- 172.21.15.77:0/640775837 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f404f7b0 con 0x7f59f40dcd40 2022-04-03T19:44:31.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.373+0000 7f877cb2c700 1 -- start start 2022-04-03T19:44:31.379 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.375+0000 7f59fa7fc700 1 --2- 172.21.15.77:0/640775837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59f40dcd40 0x7f59f40dee40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f59fc06f650 tx=0x7f59fc04bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b2da16ab4cac776 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.379 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.375+0000 7f59f9ffb700 1 -- 172.21.15.77:0/640775837 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc06b070 con 0x7f59f40dcd40 2022-04-03T19:44:31.380 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.375+0000 7f877cb2c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f877000c790 0x7f877000b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.381 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.375+0000 7f877cb2c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f877004af90 0x7f87700da6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.381 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.375+0000 7f877cb2c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f877004a840 0x7f8770006d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.382 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.375+0000 7f877cb2c700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87700d9d30 con 0x7f877004a840 2022-04-03T19:44:31.382 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.375+0000 7f877cb2c700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87700dd3b0 con 0x7f877000c790 2022-04-03T19:44:31.382 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.375+0000 7f877cb2c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87700dac10 con 0x7f877004af90 2022-04-03T19:44:31.383 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.376+0000 7f59f9ffb700 1 -- 172.21.15.77:0/640775837 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59fc06f140 con 0x7f59f40dcd40 2022-04-03T19:44:31.385 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.376+0000 7f8775d9b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f877004af90 0x7f87700da6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.385 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.376+0000 7f8775d9b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f877004af90 0x7f87700da6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59882/0 (socket says 172.21.15.77:59882) 2022-04-03T19:44:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.376+0000 7f8775d9b700 1 -- 172.21.15.77:0/3296218230 learned_addr learned my addr 172.21.15.77:0/3296218230 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:31.386 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.377+0000 7f59fbfff700 1 -- 172.21.15.77:0/640775837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59f40dcd40 msgr2=0x7f59f40dee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.387 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.377+0000 7f59fbfff700 1 --2- 172.21.15.77:0/640775837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59f40dcd40 0x7f59f40dee40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f59fc06f650 tx=0x7f59fc04bea0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.388 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.377+0000 7f8775d9b700 1 -- 172.21.15.77:0/3296218230 >> v2:172.21.15.186:3300/0 conn(0x7f877000c790 msgr2=0x7f877000b190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-04-03T19:44:31.389 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.377+0000 7f8775d9b700 1 --2- 172.21.15.77:0/3296218230 >> v2:172.21.15.186:3300/0 conn(0x7f877000c790 0x7f877000b190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.389 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.377+0000 7f8775d9b700 1 -- 172.21.15.77:0/3296218230 >> v2:172.21.15.77:3300/0 conn(0x7f877004a840 msgr2=0x7f8770006d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-04-03T19:44:31.389 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.377+0000 7f8775d9b700 1 --2- 172.21.15.77:0/3296218230 >> v2:172.21.15.77:3300/0 conn(0x7f877004a840 0x7f8770006d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.390 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.377+0000 7f8775d9b700 1 -- 172.21.15.77:0/3296218230 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f877004b480 con 0x7f877004af90 2022-04-03T19:44:31.391 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.377+0000 7f59fbfff700 1 -- 172.21.15.77:0/640775837 shutdown_connections 2022-04-03T19:44:31.391 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.377+0000 7f59fbfff700 1 --2- 172.21.15.77:0/640775837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59f40dcd40 0x7f59f40dee40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.391 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.377+0000 7f59fbfff700 1 -- 172.21.15.77:0/640775837 >> 172.21.15.77:0/640775837 conn(0x7f59f4009d10 msgr2=0x7f59f400a110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:31.392 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.377+0000 7f8775d9b700 1 --2- 172.21.15.77:0/3296218230 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f877004af90 0x7f87700da6d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8778060620 tx=0x7f877804b5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bae6e0662b9d17c server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.393 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.377+0000 7f877559a700 1 -- 172.21.15.77:0/3296218230 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877806b070 con 0x7f877004af90 2022-04-03T19:44:31.393 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.378+0000 7f877559a700 1 -- 172.21.15.77:0/3296218230 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877806f130 con 0x7f877004af90 2022-04-03T19:44:31.393 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.378+0000 7f877559a700 1 -- 172.21.15.77:0/3296218230 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8778064620 con 0x7f877004af90 2022-04-03T19:44:31.394 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.378+0000 7f59fbfff700 1 -- 172.21.15.77:0/640775837 shutdown_connections 2022-04-03T19:44:31.395 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.379+0000 7f877cb2c700 1 -- 172.21.15.77:0/3296218230 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f877004af90 msgr2=0x7f87700da6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.396 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.379+0000 7f877cb2c700 1 --2- 172.21.15.77:0/3296218230 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f877004af90 0x7f87700da6d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8778060620 tx=0x7f877804b5b0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.396 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.379+0000 7f877cb2c700 1 -- 172.21.15.77:0/3296218230 shutdown_connections 2022-04-03T19:44:31.396 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.379+0000 7f877cb2c700 1 --2- 172.21.15.77:0/3296218230 >> v2:172.21.15.77:3300/0 conn(0x7f877004a840 0x7f8770006d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.397 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.379+0000 7f877cb2c700 1 --2- 172.21.15.77:0/3296218230 >> v2:172.21.15.186:3300/0 conn(0x7f877000c790 0x7f877000b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.398 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.379+0000 7f877cb2c700 1 --2- 172.21.15.77:0/3296218230 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f877004af90 0x7f87700da6d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.398 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.379+0000 7f877cb2c700 1 -- 172.21.15.77:0/3296218230 >> 172.21.15.77:0/3296218230 conn(0x7f877000bc80 msgr2=0x7f877000c080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:31.398 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.380+0000 7f877cb2c700 1 -- 172.21.15.77:0/3296218230 shutdown_connections 2022-04-03T19:44:31.400 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.378+0000 7f59fbfff700 1 -- 172.21.15.77:0/640775837 wait complete. 2022-04-03T19:44:31.400 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.380+0000 7f59fbfff700 1 Processor -- start 2022-04-03T19:44:31.401 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.380+0000 7f877cb2c700 1 -- 172.21.15.77:0/3296218230 wait complete. 2022-04-03T19:44:31.402 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.381+0000 7f59fbfff700 1 -- start start 2022-04-03T19:44:31.403 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.383+0000 7f877cb2c700 1 Processor -- start 2022-04-03T19:44:31.403 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.383+0000 7f59fbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59f4049640 0x7f59f40d3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.404 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.383+0000 7f877cb2c700 1 -- start start 2022-04-03T19:44:31.406 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.383+0000 7f59fbfff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59f404f3e0 0x7f59f40d0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.407 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.384+0000 7f59faffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59f4049640 0x7f59f40d3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.407 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.384+0000 7f59faffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59f4049640 0x7f59f40d3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42578/0 (socket says 172.21.15.77:42578) 2022-04-03T19:44:31.407 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.384+0000 7f59fbfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59f40dcd40 0x7f59f4007b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.408 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.384+0000 7f59fbfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f40d3c90 con 0x7f59f4049640 2022-04-03T19:44:31.408 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.384+0000 7f59fbfff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f40d32c0 con 0x7f59f40dcd40 2022-04-03T19:44:31.408 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.384+0000 7f59fbfff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59f4045710 con 0x7f59f404f3e0 2022-04-03T19:44:31.409 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.385+0000 7f59faffd700 1 -- 172.21.15.77:0/2180923786 learned_addr learned my addr 172.21.15.77:0/2180923786 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:31.410 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.385+0000 7f877cb2c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f877000c790 0x7f87700d8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.411 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.385+0000 7f59fa7fc700 1 --2- 172.21.15.77:0/2180923786 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59f404f3e0 0x7f59f40d0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.411 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.385+0000 7f59fb7fe700 1 --2- 172.21.15.77:0/2180923786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59f40dcd40 0x7f59f4007b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.411 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.386+0000 7f59faffd700 1 -- 172.21.15.77:0/2180923786 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59f404f3e0 msgr2=0x7f59f40d0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.412 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.386+0000 7f59faffd700 1 --2- 172.21.15.77:0/2180923786 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59f404f3e0 0x7f59f40d0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.412 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.386+0000 7f59faffd700 1 -- 172.21.15.77:0/2180923786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59f40dcd40 msgr2=0x7f59f4007b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.412 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.386+0000 7f59faffd700 1 --2- 172.21.15.77:0/2180923786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59f40dcd40 0x7f59f4007b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.413 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.386+0000 7f59faffd700 1 -- 172.21.15.77:0/2180923786 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f4008870 con 0x7f59f4049640 2022-04-03T19:44:31.413 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.386+0000 7f877659c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f877000c790 0x7f87700d8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.414 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.386+0000 7f877659c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f877000c790 0x7f87700d8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37772/0 (socket says 172.21.15.77:37772) 2022-04-03T19:44:31.414 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.386+0000 7f877659c700 1 -- 172.21.15.77:0/3038807238 learned_addr learned my addr 172.21.15.77:0/3038807238 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:31.415 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.386+0000 7f59fb7fe700 1 --2- 172.21.15.77:0/2180923786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59f40dcd40 0x7f59f4007b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:31.416 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.386+0000 7f877cb2c700 1 --2- 172.21.15.77:0/3038807238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f877004a840 0x7f87700c4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.416 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.386+0000 7f59faffd700 1 --2- 172.21.15.77:0/2180923786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59f4049640 0x7f59f40d3750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f59ec00b4c0 tx=0x7f59ec009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.417 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.387+0000 7f8775d9b700 1 --2- 172.21.15.77:0/3038807238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f877004a840 0x7f87700c4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.417 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.387+0000 7f2cbd8b1700 1 Processor -- start 2022-04-03T19:44:31.418 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.387+0000 7f2cbd8b1700 1 -- start start 2022-04-03T19:44:31.418 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.387+0000 7f877cb2c700 1 --2- 172.21.15.77:0/3038807238 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f877004af90 0x7f87700c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.419 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.387+0000 7f877cb2c700 1 -- 172.21.15.77:0/3038807238 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8770005740 con 0x7f877004a840 2022-04-03T19:44:31.419 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.387+0000 7f877cb2c700 1 -- 172.21.15.77:0/3038807238 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87700065c0 con 0x7f877000c790 2022-04-03T19:44:31.419 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.387+0000 7f877cb2c700 1 -- 172.21.15.77:0/3038807238 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87700c4790 con 0x7f877004af90 2022-04-03T19:44:31.420 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.388+0000 7f59e2ffd700 1 -- 172.21.15.77:0/2180923786 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ec00be60 con 0x7f59f4049640 2022-04-03T19:44:31.420 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.388+0000 7f59e2ffd700 1 -- 172.21.15.77:0/2180923786 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59ec011630 con 0x7f59f4049640 2022-04-03T19:44:31.421 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.389+0000 7f59fbfff700 1 -- 172.21.15.77:0/2180923786 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f40d0c50 con 0x7f59f4049640 2022-04-03T19:44:31.422 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.389+0000 7f8775d9b700 1 -- 172.21.15.77:0/3038807238 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f877004af90 msgr2=0x7f87700c8be0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2022-04-03T19:44:31.422 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.389+0000 7f8775d9b700 1 --2- 172.21.15.77:0/3038807238 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f877004af90 0x7f87700c8be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.423 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.389+0000 7f8775d9b700 1 -- 172.21.15.77:0/3038807238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f877000c790 msgr2=0x7f87700d8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.423 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.389+0000 7f8775d9b700 1 --2- 172.21.15.77:0/3038807238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f877000c790 0x7f87700d8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.423 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.389+0000 7f8775d9b700 1 -- 172.21.15.77:0/3038807238 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8778063850 con 0x7f877004a840 2022-04-03T19:44:31.423 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.389+0000 7f8775d9b700 1 --2- 172.21.15.77:0/3038807238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f877004a840 0x7f87700c4250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f877804c3f0 tx=0x7f877808b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.424 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.391+0000 7f87667fc700 1 -- 172.21.15.77:0/3038807238 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877806b6e0 con 0x7f877004a840 2022-04-03T19:44:31.425 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.389+0000 7f2cbd8b1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2cb806c650 0x7f2cb806ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.425 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.391+0000 7f87667fc700 1 -- 172.21.15.77:0/3038807238 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8778063f90 con 0x7f877004a840 2022-04-03T19:44:31.425 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.391+0000 7f877cb2c700 1 -- 172.21.15.77:0/3038807238 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8770005f90 con 0x7f877004a840 2022-04-03T19:44:31.426 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.389+0000 7f2cbd8b1700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2cb806bf60 0x7f2cb8159c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.389+0000 7f2cbd8b1700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb8069430 con 0x7f2cb80a9a60 2022-04-03T19:44:31.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.389+0000 7f2cbd8b1700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb80c2b90 con 0x7f2cb806bf60 2022-04-03T19:44:31.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.389+0000 7f2cbd8b1700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb80696f0 con 0x7f2cb806c650 2022-04-03T19:44:31.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.390+0000 7f2cb77fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2cb806bf60 0x7f2cb8159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.390+0000 7f2cb77fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2cb806bf60 0x7f2cb8159c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37774/0 (socket says 172.21.15.77:37774) 2022-04-03T19:44:31.428 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.390+0000 7f2cb6ffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2cb80a9a60 0x7f2cb806a4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38326/0 (socket says 172.21.15.77:38326) 2022-04-03T19:44:31.428 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.390+0000 7f2cb77fe700 1 -- 172.21.15.77:0/1200853865 learned_addr learned my addr 172.21.15.77:0/1200853865 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:31.428 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.390+0000 7f2cb67fc700 1 --2- 172.21.15.77:0/1200853865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2cb806c650 0x7f2cb806ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.428 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.390+0000 7f2cb77fe700 1 -- 172.21.15.77:0/1200853865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2cb806c650 msgr2=0x7f2cb806ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:31.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.390+0000 7f2cb77fe700 1 --2- 172.21.15.77:0/1200853865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2cb806c650 0x7f2cb806ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.390+0000 7f2cb77fe700 1 -- 172.21.15.77:0/1200853865 >> v1:172.21.15.77:6789/0 conn(0x7f2cb80a9a60 legacy=0x7f2cb806a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.390+0000 7f2cb77fe700 1 -- 172.21.15.77:0/1200853865 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb80699d0 con 0x7f2cb806bf60 2022-04-03T19:44:31.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.392+0000 7f2cb77fe700 1 --2- 172.21.15.77:0/1200853865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cb806bf60 0x7f2cb8159c10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0009f20 tx=0x7f2cb0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4be802f9aa97dd14 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.391+0000 7f87667fc700 1 -- 172.21.15.77:0/3038807238 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87780906a0 con 0x7f877004a840 2022-04-03T19:44:31.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.392+0000 7f877cb2c700 1 -- 172.21.15.77:0/3038807238 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8770005c70 con 0x7f877004a840 2022-04-03T19:44:31.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.390+0000 7f59e2ffd700 1 -- 172.21.15.77:0/2180923786 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ec0119d0 con 0x7f59f4049640 2022-04-03T19:44:31.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.392+0000 7f59fbfff700 1 -- 172.21.15.77:0/2180923786 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f4009790 con 0x7f59f4049640 2022-04-03T19:44:31.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.390+0000 7f77e495b700 1 Processor -- start 2022-04-03T19:44:31.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.390+0000 7f77e495b700 1 -- start start 2022-04-03T19:44:31.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.392+0000 7f77e495b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77d8006b40 0x7f77d80e1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.392+0000 7f77e495b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f77d804ee70 0x7f77d80de340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.392+0000 7f77e495b700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f77d80de880 0x7f77d800f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.392+0000 7f77e495b700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d804b1e0 con 0x7f77d804ee70 2022-04-03T19:44:31.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.392+0000 7f77e495b700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d80ddc10 con 0x7f77d80de880 2022-04-03T19:44:31.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.392+0000 7f77e495b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d80075f0 con 0x7f77d8006b40 2022-04-03T19:44:31.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.392+0000 7f77ded9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77d8006b40 0x7f77d80e1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.392+0000 7f77df59e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f77d80de880 0x7f77d800f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.392+0000 7f77df59e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f77d80de880 0x7f77d800f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37776/0 (socket says 172.21.15.77:37776) 2022-04-03T19:44:31.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.392+0000 7f77df59e700 1 -- 172.21.15.77:0/1134135259 learned_addr learned my addr 172.21.15.77:0/1134135259 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:31.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.392+0000 7f77de59c700 1 --2- 172.21.15.77:0/1134135259 >> v2:172.21.15.77:3300/0 conn(0x7f77d804ee70 0x7f77d80de340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.393+0000 7f77ded9d700 1 -- 172.21.15.77:0/1134135259 >> v2:172.21.15.186:3300/0 conn(0x7f77d80de880 msgr2=0x7f77d800f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:31.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.393+0000 7f77ded9d700 1 --2- 172.21.15.77:0/1134135259 >> v2:172.21.15.186:3300/0 conn(0x7f77d80de880 0x7f77d800f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.393+0000 7f77ded9d700 1 -- 172.21.15.77:0/1134135259 >> v2:172.21.15.77:3300/0 conn(0x7f77d804ee70 msgr2=0x7f77d80de340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:31.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.393+0000 7f77ded9d700 1 --2- 172.21.15.77:0/1134135259 >> v2:172.21.15.77:3300/0 conn(0x7f77d804ee70 0x7f77d80de340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.393+0000 7f77ded9d700 1 -- 172.21.15.77:0/1134135259 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d8005e90 con 0x7f77d8006b40 2022-04-03T19:44:31.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.394+0000 7f77ded9d700 1 --2- 172.21.15.77:0/1134135259 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77d8006b40 0x7f77d80e1ba0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f77e002af70 tx=0x7f77e004bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19ab78472a1db727 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.394+0000 7f77ddd9b700 1 -- 172.21.15.77:0/1134135259 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e006b070 con 0x7f77d8006b40 2022-04-03T19:44:31.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.394+0000 7f77ddd9b700 1 -- 172.21.15.77:0/1134135259 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e004b890 con 0x7f77d8006b40 2022-04-03T19:44:31.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.394+0000 7f77ddd9b700 1 -- 172.21.15.77:0/1134135259 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e0064660 con 0x7f77d8006b40 2022-04-03T19:44:31.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.395+0000 7f877cb2c700 1 -- 172.21.15.77:0/3038807238 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87700051f0 con 0x7f877004a840 2022-04-03T19:44:31.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.395+0000 7f87667fc700 1 -- 172.21.15.77:0/3038807238 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f8778099070 con 0x7f877004a840 2022-04-03T19:44:31.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.395+0000 7f2cb5ffb700 1 -- 172.21.15.77:0/1200853865 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb000e070 con 0x7f2cb806bf60 2022-04-03T19:44:31.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.395+0000 7f2cb5ffb700 1 -- 172.21.15.77:0/1200853865 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb000ba60 con 0x7f2cb806bf60 2022-04-03T19:44:31.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.396+0000 7f2cb5ffb700 1 -- 172.21.15.77:0/1200853865 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb00092e0 con 0x7f2cb806bf60 2022-04-03T19:44:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.396+0000 7f87667fc700 1 --2- 172.21.15.77:0/3038807238 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8754063db0 0x7f8754066260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.396+0000 7f2cbd8b1700 1 -- 172.21.15.77:0/1200853865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cb806bf60 msgr2=0x7f2cb8159c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.397+0000 7f87667fc700 1 -- 172.21.15.77:0/3038807238 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f87780f8250 con 0x7f877004a840 2022-04-03T19:44:31.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.397+0000 7f77e495b700 1 -- 172.21.15.77:0/1134135259 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77d8006b40 msgr2=0x7f77d80e1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.397+0000 7f77e495b700 1 --2- 172.21.15.77:0/1134135259 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77d8006b40 0x7f77d80e1ba0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f77e002af70 tx=0x7f77e004bea0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.441 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.396+0000 7f59e2ffd700 1 -- 172.21.15.77:0/2180923786 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f59ec021070 con 0x7f59f4049640 2022-04-03T19:44:31.441 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.396+0000 7f2cbd8b1700 1 --2- 172.21.15.77:0/1200853865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cb806bf60 0x7f2cb8159c10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0009f20 tx=0x7f2cb0009840 comp rx=0 tx=0).stop 2022-04-03T19:44:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.397+0000 7f77e495b700 1 -- 172.21.15.77:0/1134135259 shutdown_connections 2022-04-03T19:44:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.397+0000 7f77e495b700 1 --2- 172.21.15.77:0/1134135259 >> v2:172.21.15.186:3300/0 conn(0x7f77d80de880 0x7f77d800f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.442 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.397+0000 7f77e495b700 1 --2- 172.21.15.77:0/1134135259 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77d8006b40 0x7f77d80e1ba0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.443 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.397+0000 7f59e2ffd700 1 --2- 172.21.15.77:0/2180923786 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f59dc063e00 0x7f59dc0662b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.443 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.397+0000 7f59e2ffd700 1 -- 172.21.15.77:0/2180923786 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f59ec0806a0 con 0x7f59f4049640 2022-04-03T19:44:31.444 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.397+0000 7f77e495b700 1 --2- 172.21.15.77:0/1134135259 >> v2:172.21.15.77:3300/0 conn(0x7f77d804ee70 0x7f77d80de340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.444 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.397+0000 7f77e495b700 1 -- 172.21.15.77:0/1134135259 >> 172.21.15.77:0/1134135259 conn(0x7f77d8010d00 msgr2=0x7f77d8050d50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:31.444 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.398+0000 7f77e495b700 1 -- 172.21.15.77:0/1134135259 shutdown_connections 2022-04-03T19:44:31.444 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.398+0000 7f77e495b700 1 -- 172.21.15.77:0/1134135259 wait complete. 2022-04-03T19:44:31.445 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.398+0000 7f877659c700 1 --2- 172.21.15.77:0/3038807238 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8754063db0 0x7f8754066260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.445 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.399+0000 7f59fa7fc700 1 --2- 172.21.15.77:0/2180923786 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f59dc063e00 0x7f59dc0662b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.446 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.398+0000 7f2cbd8b1700 1 -- 172.21.15.77:0/1200853865 shutdown_connections 2022-04-03T19:44:31.446 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.398+0000 7f2cbd8b1700 1 --2- 172.21.15.77:0/1200853865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cb806bf60 0x7f2cb8159c10 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.398+0000 7f2cbd8b1700 1 --2- 172.21.15.77:0/1200853865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2cb806c650 0x7f2cb806ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.398+0000 7f2cbd8b1700 1 -- 172.21.15.77:0/1200853865 >> 172.21.15.77:0/1200853865 conn(0x7f2cb80c71c0 msgr2=0x7f2cb80c75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:31.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.399+0000 7f2cbd8b1700 1 -- 172.21.15.77:0/1200853865 shutdown_connections 2022-04-03T19:44:31.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.399+0000 7f2cbd8b1700 1 -- 172.21.15.77:0/1200853865 wait complete. 2022-04-03T19:44:31.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.399+0000 7f7e2bfff700 1 Processor -- start 2022-04-03T19:44:31.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.400+0000 7f877659c700 1 --2- 172.21.15.77:0/3038807238 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8754063db0 0x7f8754066260 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8768006fd0 tx=0x7f8768008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.400+0000 7f7e2bfff700 1 -- start start 2022-04-03T19:44:31.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.400+0000 7f77e495b700 1 Processor -- start 2022-04-03T19:44:31.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.401+0000 7f77e495b700 1 -- start start 2022-04-03T19:44:31.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.402+0000 7f59fa7fc700 1 --2- 172.21.15.77:0/2180923786 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f59dc063e00 0x7f59dc0662b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f59f400f480 tx=0x7f59fc088000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.402+0000 7f59fbfff700 1 -- 172.21.15.77:0/2180923786 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f40d35a0 con 0x7f59f4049640 2022-04-03T19:44:31.452 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.403+0000 7f7e2bfff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7e240d3f70 0x7f7e24047bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.452 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.403+0000 7f77e495b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77d8006b40 0x7f77d80d0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.453 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.401+0000 7f2cbd8b1700 1 Processor -- start 2022-04-03T19:44:31.453 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.402+0000 7f2cbd8b1700 1 -- start start 2022-04-03T19:44:31.454 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.403+0000 7f7e2bfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7e24009a20 0x7f7e240e4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.454 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f7e2bfff700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e24049f20 con 0x7f7e240da780 2022-04-03T19:44:31.454 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f7e2affd700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7e240d3f70 0x7f7e24047bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.454 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f7e2bfff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e24049960 con 0x7f7e24009a20 2022-04-03T19:44:31.455 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f77e495b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77d804ee70 0x7f77d80eee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.455 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f77e495b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77d80de880 0x7f77d80dede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.455 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f77e495b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d80d30a0 con 0x7f77d8006b40 2022-04-03T19:44:31.455 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f77e495b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d80d0da0 con 0x7f77d80de880 2022-04-03T19:44:31.456 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f77e495b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d80df320 con 0x7f77d804ee70 2022-04-03T19:44:31.456 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f7e2bfff700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e24049c40 con 0x7f7e240d3f70 2022-04-03T19:44:31.457 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f77ded9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77d8006b40 0x7f77d80d0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.457 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.403+0000 7f2cbd8b1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cb806bf60 0x7f2cb8161760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.458 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.405+0000 7f77ded9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77d8006b40 0x7f77d80d0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42584/0 (socket says 172.21.15.77:42584) 2022-04-03T19:44:31.458 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.405+0000 7f77ded9d700 1 -- 172.21.15.77:0/3172866390 learned_addr learned my addr 172.21.15.77:0/3172866390 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:31.459 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.403+0000 7f2cbd8b1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2cb806c650 0x7f2cb814ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.459 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.405+0000 7f77de59c700 1 --2- 172.21.15.77:0/3172866390 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77d804ee70 0x7f77d80eee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.460 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f2cb67fc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2cb806c650 0x7f2cb814ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.460 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.405+0000 7f77df59e700 1 --2- 172.21.15.77:0/3172866390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77d80de880 0x7f77d80dede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.461 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f2cb6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cb806bf60 0x7f2cb8161760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.461 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f2cb67fc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2cb806c650 0x7f2cb814ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59896/0 (socket says 172.21.15.77:59896) 2022-04-03T19:44:31.462 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f2cbd8b1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cb80a9a60 0x7f2cb80abac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.462 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f2cbd8b1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb814c6c0 con 0x7f2cb806bf60 2022-04-03T19:44:31.462 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f2cbd8b1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb81512e0 con 0x7f2cb80a9a60 2022-04-03T19:44:31.462 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.406+0000 7f77de59c700 1 -- 172.21.15.77:0/3172866390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77d80de880 msgr2=0x7f77d80dede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.463 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f2cbd8b1700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb8165160 con 0x7f2cb806c650 2022-04-03T19:44:31.463 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f2cb6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cb806bf60 0x7f2cb8161760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42586/0 (socket says 172.21.15.77:42586) 2022-04-03T19:44:31.464 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f2cb6ffd700 1 -- 172.21.15.77:0/2707584730 learned_addr learned my addr 172.21.15.77:0/2707584730 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:31.464 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.406+0000 7f77de59c700 1 --2- 172.21.15.77:0/3172866390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77d80de880 0x7f77d80dede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.464 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.406+0000 7f77de59c700 1 -- 172.21.15.77:0/3172866390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77d8006b40 msgr2=0x7f77d80d0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.464 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.406+0000 7f77de59c700 1 --2- 172.21.15.77:0/3172866390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77d8006b40 0x7f77d80d0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.465 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.406+0000 7f77de59c700 1 -- 172.21.15.77:0/3172866390 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d80d2490 con 0x7f77d804ee70 2022-04-03T19:44:31.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.405+0000 7f2cb77fe700 1 --2- 172.21.15.77:0/2707584730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cb80a9a60 0x7f2cb80abac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.405+0000 7f2cb6ffd700 1 -- 172.21.15.77:0/2707584730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2cb806c650 msgr2=0x7f2cb814ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.405+0000 7f2cb6ffd700 1 --2- 172.21.15.77:0/2707584730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2cb806c650 0x7f2cb814ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.405+0000 7f2cb6ffd700 1 -- 172.21.15.77:0/2707584730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cb80a9a60 msgr2=0x7f2cb80abac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.405+0000 7f2cb6ffd700 1 --2- 172.21.15.77:0/2707584730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cb80a9a60 0x7f2cb80abac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.405+0000 7f2cb6ffd700 1 -- 172.21.15.77:0/2707584730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb0009ae0 con 0x7f2cb806bf60 2022-04-03T19:44:31.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.405+0000 7f2cb67fc700 1 --2- 172.21.15.77:0/2707584730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2cb806c650 0x7f2cb814ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:44:31.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.406+0000 7f2cb6ffd700 1 --2- 172.21.15.77:0/2707584730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cb806bf60 0x7f2cb8161760 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2cac00c4a0 tx=0x7f2cac007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f7e2affd700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7e240d3f70 0x7f7e24047bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37778/0 (socket says 172.21.15.77:37778) 2022-04-03T19:44:31.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.404+0000 7f7e2affd700 1 -- 172.21.15.77:0/511419133 learned_addr learned my addr 172.21.15.77:0/511419133 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:31.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.405+0000 7f7e29ffb700 1 -- 172.21.15.77:0/511419133 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250490375 0 0) 0x7f7e24049f20 con 0x7f7e240da780 2022-04-03T19:44:31.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.406+0000 7f7e2b7fe700 1 --2- 172.21.15.77:0/511419133 >> v2:172.21.15.77:3300/0 conn(0x7f7e24009a20 0x7f7e240e4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.406+0000 7f77de59c700 1 --2- 172.21.15.77:0/3172866390 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77d804ee70 0x7f77d80eee50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f77d8012100 tx=0x7f77d0009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.406+0000 7f7e29ffb700 1 -- 172.21.15.77:0/511419133 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e180033f0 con 0x7f7e240da780 2022-04-03T19:44:31.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.407+0000 7f87667fc700 1 -- 172.21.15.77:0/3038807238 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f87780fd060 con 0x7f877004a840 2022-04-03T19:44:31.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.407+0000 7f7e2b7fe700 1 -- 172.21.15.77:0/511419133 >> v2:172.21.15.186:3300/0 conn(0x7f7e240d3f70 msgr2=0x7f7e24047bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:31.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.407+0000 7f7e2b7fe700 1 --2- 172.21.15.77:0/511419133 >> v2:172.21.15.186:3300/0 conn(0x7f7e240d3f70 0x7f7e24047bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.407+0000 7f7e2b7fe700 1 -- 172.21.15.77:0/511419133 >> v1:172.21.15.186:6789/0 conn(0x7f7e240da780 legacy=0x7f7e24006d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.407+0000 7f7e2b7fe700 1 -- 172.21.15.77:0/511419133 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e240d9ca0 con 0x7f7e24009a20 2022-04-03T19:44:31.472 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.406+0000 7f77c6ffd700 1 -- 172.21.15.77:0/3172866390 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d0013680 con 0x7f77d804ee70 2022-04-03T19:44:31.472 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.407+0000 7f77c6ffd700 1 -- 172.21.15.77:0/3172866390 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d00137e0 con 0x7f77d804ee70 2022-04-03T19:44:31.472 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.407+0000 7f77c6ffd700 1 -- 172.21.15.77:0/3172866390 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d0013b60 con 0x7f77d804ee70 2022-04-03T19:44:31.473 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.407+0000 7f77e495b700 1 -- 172.21.15.77:0/3172866390 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d80d2770 con 0x7f77d804ee70 2022-04-03T19:44:31.473 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.407+0000 7f77e495b700 1 -- 172.21.15.77:0/3172866390 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d80d2d30 con 0x7f77d804ee70 2022-04-03T19:44:31.473 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.408+0000 7f7e2b7fe700 1 --2- 172.21.15.77:0/511419133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e24009a20 0x7f7e240e4470 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c064160 tx=0x7f7e2c06f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88bef5acbeeca12c server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.474 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.409+0000 7f8c35846700 1 Processor -- start 2022-04-03T19:44:31.474 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.409+0000 7f8c35846700 1 -- start start 2022-04-03T19:44:31.475 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.410+0000 7f77e495b700 1 -- 172.21.15.77:0/3172866390 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77bc005130 con 0x7f77d804ee70 2022-04-03T19:44:31.475 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.410+0000 7f77c6ffd700 1 -- 172.21.15.77:0/3172866390 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f77d0019030 con 0x7f77d804ee70 2022-04-03T19:44:31.476 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.411+0000 7f8c35846700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8c30068080 0x7f8c30068ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.477 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.411+0000 7f8c35846700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c30069400 0x7f8c300c08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.477 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.411+0000 7f8c2effd700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f8c30067cb0 0x7f8c300c5800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44130/0 (socket says 172.21.15.77:44130) 2022-04-03T19:44:31.477 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.411+0000 7f8c2effd700 1 -- 172.21.15.77:0/3333879421 learned_addr learned my addr 172.21.15.77:0/3333879421 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:31.477 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.411+0000 7f8c35846700 1 -- 172.21.15.77:0/3333879421 --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c300c5f10 con 0x7f8c30067cb0 2022-04-03T19:44:31.477 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.411+0000 7f8c35846700 1 -- 172.21.15.77:0/3333879421 --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c300c1a40 con 0x7f8c30068080 2022-04-03T19:44:31.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.411+0000 7f8c2e7fc700 1 --2- 172.21.15.77:0/3333879421 >> v2:172.21.15.186:3300/0 conn(0x7f8c30068080 0x7f8c30068ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.411+0000 7f8c2dffb700 1 -- 172.21.15.77:0/3333879421 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133381640 0 0) 0x7f8c300c5f10 con 0x7f8c30067cb0 2022-04-03T19:44:31.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.411+0000 7f77c6ffd700 1 --2- 172.21.15.77:0/3172866390 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f77c0063db0 0x7f77c0066260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.411+0000 7f77c6ffd700 1 -- 172.21.15.77:0/3172866390 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f77d00801f0 con 0x7f77d804ee70 2022-04-03T19:44:31.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.412+0000 7f77ded9d700 1 --2- 172.21.15.77:0/3172866390 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f77c0063db0 0x7f77c0066260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.412+0000 7f8c2f7fe700 1 --2- 172.21.15.77:0/3333879421 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c30069400 0x7f8c300c08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.412+0000 7f8c35846700 1 -- 172.21.15.77:0/3333879421 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c300c32b0 con 0x7f8c30069400 2022-04-03T19:44:31.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.412+0000 7f77ded9d700 1 --2- 172.21.15.77:0/3172866390 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f77c0063db0 0x7f77c0066260 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f77e006f650 tx=0x7f77e0077000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.482 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.412+0000 7f8c2dffb700 1 -- 172.21.15.77:0/3333879421 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c1c0033f0 con 0x7f8c30067cb0 2022-04-03T19:44:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.412+0000 7f8c2dffb700 1 -- 172.21.15.77:0/3333879421 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3264339462 0 0) 0x7f8c1c0033f0 con 0x7f8c30067cb0 2022-04-03T19:44:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.412+0000 7f8c2dffb700 1 -- 172.21.15.77:0/3333879421 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c30069400 msgr2=0x7f8c300c08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.412+0000 7f8c2dffb700 1 --2- 172.21.15.77:0/3333879421 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c30069400 0x7f8c300c08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.412+0000 7f8c2dffb700 1 -- 172.21.15.77:0/3333879421 >> v2:172.21.15.186:3300/0 conn(0x7f8c30068080 msgr2=0x7f8c30068ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:31.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.412+0000 7f8c2dffb700 1 --2- 172.21.15.77:0/3333879421 >> v2:172.21.15.186:3300/0 conn(0x7f8c30068080 0x7f8c30068ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.413+0000 7f8c2dffb700 1 -- 172.21.15.77:0/3333879421 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c300c4b30 con 0x7f8c30067cb0 2022-04-03T19:44:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.413+0000 7f8c2e7fc700 1 --2- 172.21.15.77:0/3333879421 >> v2:172.21.15.186:3300/0 conn(0x7f8c30068080 0x7f8c30068ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.413+0000 7f8c2dffb700 1 -- 172.21.15.77:0/3333879421 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4219302955 0 0) 0x7f8c24004d60 con 0x7f8c30067cb0 2022-04-03T19:44:31.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.413+0000 7f8c2dffb700 1 -- 172.21.15.77:0/3333879421 >> v1:172.21.15.186:6789/0 conn(0x7f8c30067cb0 legacy=0x7f8c300c5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.413+0000 7f8c2dffb700 1 --2- 172.21.15.77:0/3333879421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c1c0038e0 0x7f8c1c003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.413+0000 7f8c2dffb700 1 --2- 172.21.15.77:0/3333879421 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c1c004690 0x7f8c1c004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.413+0000 7f8c2dffb700 1 --2- 172.21.15.77:0/3333879421 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c30069400 0x7f8c1c005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.413+0000 7f8c2dffb700 1 -- 172.21.15.77:0/3333879421 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c300c1a40 con 0x7f8c1c0038e0 2022-04-03T19:44:31.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.413+0000 7f8c2dffb700 1 -- 172.21.15.77:0/3333879421 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c300c32b0 con 0x7f8c1c004690 2022-04-03T19:44:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.413+0000 7f8c2dffb700 1 -- 172.21.15.77:0/3333879421 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c00b990 con 0x7f8c30069400 2022-04-03T19:44:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.413+0000 7f8c2effd700 1 --2- 172.21.15.77:0/3333879421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c1c0038e0 0x7f8c1c003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.414+0000 7f7e29ffb700 1 -- 172.21.15.77:0/511419133 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c06b070 con 0x7f7e24009a20 2022-04-03T19:44:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.414+0000 7f59e2ffd700 1 -- 172.21.15.77:0/2180923786 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f59ec015080 con 0x7f59f4049640 2022-04-03T19:44:31.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.414+0000 7f7e29ffb700 1 -- 172.21.15.77:0/511419133 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2c04b960 con 0x7f7e24009a20 2022-04-03T19:44:31.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.414+0000 7f7e29ffb700 1 -- 172.21.15.77:0/511419133 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c064600 con 0x7f7e24009a20 2022-04-03T19:44:31.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.414+0000 7f2c9effd700 1 -- 172.21.15.77:0/2707584730 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cac0072e0 con 0x7f2cb806bf60 2022-04-03T19:44:31.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.414+0000 7f2c9effd700 1 -- 172.21.15.77:0/2707584730 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cac007440 con 0x7f2cb806bf60 2022-04-03T19:44:31.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.415+0000 7f7e2bfff700 1 -- 172.21.15.77:0/511419133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e24009a20 msgr2=0x7f7e240e4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.415+0000 7f7e2bfff700 1 --2- 172.21.15.77:0/511419133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e24009a20 0x7f7e240e4470 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c064160 tx=0x7f7e2c06f390 comp rx=0 tx=0).stop 2022-04-03T19:44:31.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.415+0000 7f77c6ffd700 1 -- 172.21.15.77:0/3172866390 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f77d004ee90 con 0x7f77d804ee70 2022-04-03T19:44:31.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.414+0000 7f2cbd8b1700 1 -- 172.21.15.77:0/2707584730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb8164cd0 con 0x7f2cb806bf60 2022-04-03T19:44:31.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.414+0000 7f8c2f7fe700 1 --2- 172.21.15.77:0/3333879421 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c30069400 0x7f8c1c005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.492 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.414+0000 7f8c2effd700 1 -- 172.21.15.77:0/3333879421 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c30069400 msgr2=0x7f8c1c005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:31.492 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.414+0000 7f8c2effd700 1 --2- 172.21.15.77:0/3333879421 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c30069400 0x7f8c1c005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.492 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.414+0000 7f8c2effd700 1 -- 172.21.15.77:0/3333879421 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c1c004690 msgr2=0x7f8c1c004b00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2022-04-03T19:44:31.492 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.414+0000 7f8c2effd700 1 --2- 172.21.15.77:0/3333879421 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c1c004690 0x7f8c1c004b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.414+0000 7f8c2effd700 1 -- 172.21.15.77:0/3333879421 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1c0033f0 con 0x7f8c1c0038e0 2022-04-03T19:44:31.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.415+0000 7f8c2effd700 1 --2- 172.21.15.77:0/3333879421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c1c0038e0 0x7f8c1c003cf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8c24014a20 tx=0x7f8c240077c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c08ab5ae2313e449 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.415+0000 7f8c2dffb700 1 -- 172.21.15.77:0/3333879421 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c24014e00 con 0x7f8c1c0038e0 2022-04-03T19:44:31.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.416+0000 7f8c2dffb700 1 -- 172.21.15.77:0/3333879421 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c240070f0 con 0x7f8c1c0038e0 2022-04-03T19:44:31.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.416+0000 7f8c2dffb700 1 -- 172.21.15.77:0/3333879421 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c240074b0 con 0x7f8c1c0038e0 2022-04-03T19:44:31.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.416+0000 7f7e2bfff700 1 -- 172.21.15.77:0/511419133 shutdown_connections 2022-04-03T19:44:31.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.416+0000 7f7e2bfff700 1 --2- 172.21.15.77:0/511419133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e24009a20 0x7f7e240e4470 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.416+0000 7f7e2bfff700 1 --2- 172.21.15.77:0/511419133 >> v2:172.21.15.186:3300/0 conn(0x7f7e240d3f70 0x7f7e24047bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.416+0000 7f7e2bfff700 1 -- 172.21.15.77:0/511419133 >> 172.21.15.77:0/511419133 conn(0x7f7e2400bc70 msgr2=0x7f7e2400c070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:31.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.417+0000 7f8c35846700 1 -- 172.21.15.77:0/3333879421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c1c0038e0 msgr2=0x7f8c1c003cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.417+0000 7f8c35846700 1 --2- 172.21.15.77:0/3333879421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c1c0038e0 0x7f8c1c003cf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8c24014a20 tx=0x7f8c240077c0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.417+0000 7f8c2f7fe700 1 -- 172.21.15.77:0/3333879421 reap_dead start 2022-04-03T19:44:31.497 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.418+0000 7f8c35846700 1 -- 172.21.15.77:0/3333879421 shutdown_connections 2022-04-03T19:44:31.497 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.418+0000 7f8c35846700 1 -- 172.21.15.77:0/3333879421 >> 172.21.15.77:0/3333879421 conn(0x7f8c300626d0 msgr2=0x7f8c30062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:31.498 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.418+0000 7f7e2bfff700 1 -- 172.21.15.77:0/511419133 shutdown_connections 2022-04-03T19:44:31.498 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.418+0000 7f7e2bfff700 1 -- 172.21.15.77:0/511419133 wait complete. 2022-04-03T19:44:31.498 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.414+0000 7f2c9effd700 1 -- 172.21.15.77:0/2707584730 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cac004900 con 0x7f2cb806bf60 2022-04-03T19:44:31.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.414+0000 7f2cbd8b1700 1 -- 172.21.15.77:0/2707584730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb81643a0 con 0x7f2cb806bf60 2022-04-03T19:44:31.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.421+0000 7f8c35846700 1 -- 172.21.15.77:0/3333879421 shutdown_connections 2022-04-03T19:44:31.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.416+0000 7f2cbd8b1700 1 -- 172.21.15.77:0/2707584730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c94005130 con 0x7f2cb806bf60 2022-04-03T19:44:31.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.419+0000 7f2c9effd700 1 -- 172.21.15.77:0/2707584730 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f2cac010070 con 0x7f2cb806bf60 2022-04-03T19:44:31.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.420+0000 7f2c9effd700 1 --2- 172.21.15.77:0/2707584730 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2c98063e30 0x7f2c980662e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.420+0000 7f2c9effd700 1 -- 172.21.15.77:0/2707584730 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f2cac07f900 con 0x7f2cb806bf60 2022-04-03T19:44:31.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.422+0000 7f7e2bfff700 1 Processor -- start 2022-04-03T19:44:31.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.421+0000 7f8c35846700 1 -- 172.21.15.77:0/3333879421 wait complete. 2022-04-03T19:44:31.502 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.422+0000 7f7e2bfff700 1 -- start start 2022-04-03T19:44:31.502 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.424+0000 7f7e2bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e24009a20 0x7f7e240f1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.503 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.425+0000 7f8c35846700 1 Processor -- start 2022-04-03T19:44:31.504 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.424+0000 7f7e2bfff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7e240d3f70 0x7f7e240d6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.424+0000 7f7e2bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7e240da780 0x7f7e2404e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.424+0000 7f7e2bfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e24042d20 con 0x7f7e24009a20 2022-04-03T19:44:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.424+0000 7f7e2bfff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e240d5bd0 con 0x7f7e240da780 2022-04-03T19:44:31.505 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.424+0000 7f7e2bfff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e240d58d0 con 0x7f7e240d3f70 2022-04-03T19:44:31.506 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.425+0000 7f7e2a7fc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7e240d3f70 0x7f7e240d6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.506 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.425+0000 7f7e2a7fc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7e240d3f70 0x7f7e240d6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59904/0 (socket says 172.21.15.77:59904) 2022-04-03T19:44:31.506 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.425+0000 7f7e2a7fc700 1 -- 172.21.15.77:0/2124168297 learned_addr learned my addr 172.21.15.77:0/2124168297 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:31.506 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.425+0000 7f7e2affd700 1 --2- 172.21.15.77:0/2124168297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e24009a20 0x7f7e240f1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.507 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.425+0000 7f7e2a7fc700 1 -- 172.21.15.77:0/2124168297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7e240da780 msgr2=0x7f7e2404e8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2022-04-03T19:44:31.507 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.425+0000 7f7e2a7fc700 1 --2- 172.21.15.77:0/2124168297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7e240da780 0x7f7e2404e8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.507 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.425+0000 7f7e2a7fc700 1 -- 172.21.15.77:0/2124168297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e24009a20 msgr2=0x7f7e240f1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.507 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.425+0000 7f7e2a7fc700 1 --2- 172.21.15.77:0/2124168297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e24009a20 0x7f7e240f1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.508 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.425+0000 7f8c35846700 1 -- start start 2022-04-03T19:44:31.508 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.426+0000 7f2cb67fc700 1 --2- 172.21.15.77:0/2707584730 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2c98063e30 0x7f2c980662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.425+0000 7f7e2a7fc700 1 -- 172.21.15.77:0/2124168297 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c063850 con 0x7f7e240d3f70 2022-04-03T19:44:31.509 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.427+0000 7f7e2a7fc700 1 --2- 172.21.15.77:0/2124168297 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7e240d3f70 0x7f7e240d6060 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c007fd0 tx=0x7f7e1c007c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.427+0000 7f8c35846700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c1c0038e0 0x7f8c301619d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.427+0000 7f8c35846700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c3014e0b0 0x7f8c300be080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.510 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.428+0000 7f8c2effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c1c0038e0 0x7f8c301619d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.428+0000 7f8c35846700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c3014e520 0x7f8c300c8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.428+0000 7f2c9effd700 1 -- 172.21.15.77:0/2707584730 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f2cac048ae0 con 0x7f2cb806bf60 2022-04-03T19:44:31.511 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.428+0000 7f2cb67fc700 1 --2- 172.21.15.77:0/2707584730 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2c98063e30 0x7f2c980662e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2cb806bf30 tx=0x7f2ca8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.428+0000 7f8c2effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c1c0038e0 0x7f8c301619d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42594/0 (socket says 172.21.15.77:42594) 2022-04-03T19:44:31.512 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.428+0000 7f8c2e7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c3014e0b0 0x7f8c300be080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.513 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.428+0000 7f7e12ffd700 1 -- 172.21.15.77:0/2124168297 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c00ce40 con 0x7f7e240d3f70 2022-04-03T19:44:31.513 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.428+0000 7f7e2bfff700 1 -- 172.21.15.77:0/2124168297 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e240d7ce0 con 0x7f7e240d3f70 2022-04-03T19:44:31.514 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.428+0000 7f7e2bfff700 1 -- 172.21.15.77:0/2124168297 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e24043250 con 0x7f7e240d3f70 2022-04-03T19:44:31.515 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.428+0000 7f8c2e7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c3014e0b0 0x7f8c300be080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37790/0 (socket says 172.21.15.77:37790) 2022-04-03T19:44:31.515 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.428+0000 7f8c2f7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c3014e520 0x7f8c300c8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.516 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.428+0000 7f8c2f7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c3014e520 0x7f8c300c8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59906/0 (socket says 172.21.15.77:59906) 2022-04-03T19:44:31.516 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.429+0000 7f8c2effd700 1 -- 172.21.15.77:0/2140680340 learned_addr learned my addr 172.21.15.77:0/2140680340 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:31.516 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.429+0000 7f8c35846700 1 -- 172.21.15.77:0/2140680340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c30164e70 con 0x7f8c1c0038e0 2022-04-03T19:44:31.516 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.430+0000 7f8c35846700 1 -- 172.21.15.77:0/2140680340 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c30165300 con 0x7f8c3014e0b0 2022-04-03T19:44:31.517 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.430+0000 7f8c35846700 1 -- 172.21.15.77:0/2140680340 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c300bd930 con 0x7f8c3014e520 2022-04-03T19:44:31.517 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.429+0000 7f7e12ffd700 1 -- 172.21.15.77:0/2124168297 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1c011590 con 0x7f7e240d3f70 2022-04-03T19:44:31.517 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.429+0000 7f7e12ffd700 1 -- 172.21.15.77:0/2124168297 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c011950 con 0x7f7e240d3f70 2022-04-03T19:44:31.518 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.430+0000 7f7e2bfff700 1 -- 172.21.15.77:0/2124168297 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e240d5a10 con 0x7f7e240d3f70 2022-04-03T19:44:31.519 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.431+0000 7f8c2e7fc700 1 -- 172.21.15.77:0/2140680340 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c3014e520 msgr2=0x7f8c300c8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.520 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.431+0000 7f8c2e7fc700 1 --2- 172.21.15.77:0/2140680340 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c3014e520 0x7f8c300c8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.520 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.431+0000 7f8c2e7fc700 1 -- 172.21.15.77:0/2140680340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c1c0038e0 msgr2=0x7f8c301619d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.520 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.431+0000 7f8c2e7fc700 1 --2- 172.21.15.77:0/2140680340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c1c0038e0 0x7f8c301619d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.520 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.432+0000 7f8c2e7fc700 1 -- 172.21.15.77:0/2140680340 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c30164540 con 0x7f8c3014e0b0 2022-04-03T19:44:31.521 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.432+0000 7f8c2e7fc700 1 --2- 172.21.15.77:0/2140680340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c3014e0b0 0x7f8c300be080 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8c2000ba20 tx=0x7f8c2000d780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.521 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.433+0000 7f8c16ffd700 1 -- 172.21.15.77:0/2140680340 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c20004030 con 0x7f8c3014e0b0 2022-04-03T19:44:31.521 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.433+0000 7f8c16ffd700 1 -- 172.21.15.77:0/2140680340 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c20004190 con 0x7f8c3014e0b0 2022-04-03T19:44:31.521 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.434+0000 7f8c35846700 1 -- 172.21.15.77:0/2140680340 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c30164820 con 0x7f8c3014e0b0 2022-04-03T19:44:31.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.434+0000 7f8c16ffd700 1 -- 172.21.15.77:0/2140680340 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2000a410 con 0x7f8c3014e0b0 2022-04-03T19:44:31.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.434+0000 7f8c35846700 1 -- 172.21.15.77:0/2140680340 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c300bdc10 con 0x7f8c3014e0b0 2022-04-03T19:44:31.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.436+0000 7f8c35846700 1 -- 172.21.15.77:0/2140680340 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c30165020 con 0x7f8c3014e0b0 2022-04-03T19:44:31.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.437+0000 7f7e12ffd700 1 -- 172.21.15.77:0/2124168297 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f7e1c010050 con 0x7f7e240d3f70 2022-04-03T19:44:31.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.439+0000 7f7e12ffd700 1 --2- 172.21.15.77:0/2124168297 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7e0c063d60 0x7f7e0c066210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.440+0000 7f7e12ffd700 1 -- 172.21.15.77:0/2124168297 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f7e1c080b30 con 0x7f7e240d3f70 2022-04-03T19:44:31.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.440+0000 7f7e12ffd700 1 -- 172.21.15.77:0/2124168297 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f7e1c0a8b80 con 0x7f7e240d3f70 2022-04-03T19:44:31.525 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.439+0000 7f8c16ffd700 1 -- 172.21.15.77:0/2140680340 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f8c2003a080 con 0x7f8c3014e0b0 2022-04-03T19:44:31.525 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.440+0000 7f8c16ffd700 1 --2- 172.21.15.77:0/2140680340 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8c10063db0 0x7f8c10066260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.440+0000 7f8c16ffd700 1 -- 172.21.15.77:0/2140680340 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f8c200807a0 con 0x7f8c3014e0b0 2022-04-03T19:44:31.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.442+0000 7f8c2effd700 1 --2- 172.21.15.77:0/2140680340 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8c10063db0 0x7f8c10066260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.443+0000 7f8c16ffd700 1 -- 172.21.15.77:0/2140680340 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f8c2004f330 con 0x7f8c3014e0b0 2022-04-03T19:44:31.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.445+0000 7f7e2affd700 1 --2- 172.21.15.77:0/2124168297 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7e0c063d60 0x7f7e0c066210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.447+0000 7f8c2effd700 1 --2- 172.21.15.77:0/2140680340 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8c10063db0 0x7f8c10066260 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8c24005670 tx=0x7f8c24019000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.447+0000 7f7e2affd700 1 --2- 172.21.15.77:0/2124168297 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7e0c063d60 0x7f7e0c066210 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7e20006bc0 tx=0x7f7e200090f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.472+0000 7f4f5c37b700 1 Processor -- start 2022-04-03T19:44:31.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.472+0000 7f4f5c37b700 1 -- start start 2022-04-03T19:44:31.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.474+0000 7f4f5c37b700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4f540c73a0 0x7f4f54159820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.474+0000 7f4f5c37b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4f540c2a00 0x7f4f540b0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.474+0000 7f4f5c37b700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f540aff70 con 0x7f4f540c7770 2022-04-03T19:44:31.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.473+0000 7fe4decb6700 1 Processor -- start 2022-04-03T19:44:31.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.473+0000 7fe4decb6700 1 -- start start 2022-04-03T19:44:31.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.474+0000 7f4f5c37b700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f540af9f0 con 0x7f4f540c2a00 2022-04-03T19:44:31.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.474+0000 7f4f5c37b700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f5415a000 con 0x7f4f540c73a0 2022-04-03T19:44:31.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.475+0000 7f4f5a117700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4f540c73a0 0x7f4f54159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.475+0000 7f4f5a918700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4f540c7770 0x7f4f5415cd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38328/0 (socket says 172.21.15.77:38328) 2022-04-03T19:44:31.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.475+0000 7f4f5a918700 1 -- 172.21.15.77:0/2214157319 learned_addr learned my addr 172.21.15.77:0/2214157319 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:31.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.475+0000 7f4f59916700 1 --2- 172.21.15.77:0/2214157319 >> v2:172.21.15.77:3300/0 conn(0x7f4f540c2a00 0x7f4f540b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.476+0000 7f4f59115700 1 -- 172.21.15.77:0/2214157319 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704742496 0 0) 0x7f4f540aff70 con 0x7f4f540c7770 2022-04-03T19:44:31.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.476+0000 7f4f59115700 1 -- 172.21.15.77:0/2214157319 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f3c0033f0 con 0x7f4f540c7770 2022-04-03T19:44:31.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.475+0000 7fe4decb6700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe4d000df60 0x7fe4d00e5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.476+0000 7fe4decb6700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe4d004c840 0x7fe4d00dbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.476+0000 7fe4decb6700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d000c630 con 0x7fe4d000e330 2022-04-03T19:44:31.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.476+0000 7fe4decb6700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4d004c220 con 0x7fe4d004c840 2022-04-03T19:44:31.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.476+0000 7fe4decb6700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4d00e2270 con 0x7fe4d000df60 2022-04-03T19:44:31.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.477+0000 7fe4dca52700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe4d000df60 0x7fe4d00e5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.477+0000 7fe4dd253700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe4d000e330 0x7fe4d000cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38330/0 (socket says 172.21.15.77:38330) 2022-04-03T19:44:31.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.477+0000 7fe4dd253700 1 -- 172.21.15.77:0/3364518953 learned_addr learned my addr 172.21.15.77:0/3364518953 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:31.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7fe4dca52700 1 -- 172.21.15.77:0/3364518953 >> v2:172.21.15.77:3300/0 conn(0x7fe4d004c840 msgr2=0x7fe4d00dbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:31.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7fe4dca52700 1 --2- 172.21.15.77:0/3364518953 >> v2:172.21.15.77:3300/0 conn(0x7fe4d004c840 0x7fe4d00dbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7fe4dca52700 1 -- 172.21.15.77:0/3364518953 >> v1:172.21.15.77:6789/0 conn(0x7fe4d000e330 legacy=0x7fe4d000cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7fe4dca52700 1 -- 172.21.15.77:0/3364518953 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d00e2550 con 0x7fe4d000df60 2022-04-03T19:44:31.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7fe4dca52700 1 --2- 172.21.15.77:0/3364518953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4d000df60 0x7fe4d00e5e60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8065170 tx=0x7fe4d806f390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66de241368ae19e6 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.479+0000 7fe4d77fe700 1 -- 172.21.15.77:0/3364518953 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d806b070 con 0x7fe4d000df60 2022-04-03T19:44:31.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.479+0000 7fe4d77fe700 1 -- 172.21.15.77:0/3364518953 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d806f140 con 0x7fe4d000df60 2022-04-03T19:44:31.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.479+0000 7fe4d77fe700 1 -- 172.21.15.77:0/3364518953 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8064660 con 0x7fe4d000df60 2022-04-03T19:44:31.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.480+0000 7fe4decb6700 1 -- 172.21.15.77:0/3364518953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4d000df60 msgr2=0x7fe4d00e5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.480+0000 7fe4decb6700 1 --2- 172.21.15.77:0/3364518953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4d000df60 0x7fe4d00e5e60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8065170 tx=0x7fe4d806f390 comp rx=0 tx=0).stop 2022-04-03T19:44:31.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.480+0000 7fe4decb6700 1 -- 172.21.15.77:0/3364518953 shutdown_connections 2022-04-03T19:44:31.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.481+0000 7fe4decb6700 1 --2- 172.21.15.77:0/3364518953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4d000df60 0x7fe4d00e5e60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.481+0000 7fe4decb6700 1 --2- 172.21.15.77:0/3364518953 >> v2:172.21.15.77:3300/0 conn(0x7fe4d004c840 0x7fe4d00dbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.481+0000 7fe4decb6700 1 -- 172.21.15.77:0/3364518953 >> 172.21.15.77:0/3364518953 conn(0x7fe4d0051f90 msgr2=0x7fe4d0052390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:31.540 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.481+0000 7fe4decb6700 1 -- 172.21.15.77:0/3364518953 shutdown_connections 2022-04-03T19:44:31.540 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.481+0000 7fe4decb6700 1 -- 172.21.15.77:0/3364518953 wait complete. 2022-04-03T19:44:31.541 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7f4f59115700 1 -- 172.21.15.77:0/2214157319 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2871563768 0 0) 0x7f4f3c0033f0 con 0x7f4f540c7770 2022-04-03T19:44:31.541 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7f4f59115700 1 -- 172.21.15.77:0/2214157319 >> v2:172.21.15.186:3300/0 conn(0x7f4f540c73a0 msgr2=0x7f4f54159820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:31.542 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7f4f59115700 1 --2- 172.21.15.77:0/2214157319 >> v2:172.21.15.186:3300/0 conn(0x7f4f540c73a0 0x7f4f54159820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.542 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7f4f59115700 1 -- 172.21.15.77:0/2214157319 >> v2:172.21.15.77:3300/0 conn(0x7f4f540c2a00 msgr2=0x7f4f540b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:31.542 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.483+0000 7fe4decb6700 1 Processor -- start 2022-04-03T19:44:31.544 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7f4f59115700 1 --2- 172.21.15.77:0/2214157319 >> v2:172.21.15.77:3300/0 conn(0x7f4f540c2a00 0x7f4f540b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.544 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7f4f59115700 1 -- 172.21.15.77:0/2214157319 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f54159d60 con 0x7f4f540c7770 2022-04-03T19:44:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7f4f59115700 1 -- 172.21.15.77:0/2214157319 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4219302955 0 0) 0x7f4f50001470 con 0x7f4f540c7770 2022-04-03T19:44:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7f4f59115700 1 -- 172.21.15.77:0/2214157319 >> v1:172.21.15.77:6789/0 conn(0x7f4f540c7770 legacy=0x7f4f5415cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7f4f59115700 1 --2- 172.21.15.77:0/2214157319 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f3c0039b0 0x7f4f3c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.545 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7f4f59115700 1 --2- 172.21.15.77:0/2214157319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f3c004690 0x7f4f3c004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.546 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7f4f59115700 1 --2- 172.21.15.77:0/2214157319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f3c005040 0x7f4f3c00b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.546 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7f4f59115700 1 -- 172.21.15.77:0/2214157319 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f540af9f0 con 0x7f4f3c004690 2022-04-03T19:44:31.546 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.484+0000 7fe4decb6700 1 -- start start 2022-04-03T19:44:31.547 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.485+0000 7fe4decb6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4d000df60 0x7fe4d0048060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.547 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7f4f59115700 1 -- 172.21.15.77:0/2214157319 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5415a000 con 0x7f4f3c005040 2022-04-03T19:44:31.548 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.485+0000 7fe4decb6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d000e330 0x7fe4d004a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.548 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.486+0000 7fe4decb6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4d004c840 0x7fe4d00db360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7f4f59115700 1 -- 172.21.15.77:0/2214157319 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c00bf20 con 0x7f4f3c0039b0 2022-04-03T19:44:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.486+0000 7fe4dca52700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4d000df60 0x7fe4d0048060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.549 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.486+0000 7fe4dca52700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4d000df60 0x7fe4d0048060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59910/0 (socket says 172.21.15.77:59910) 2022-04-03T19:44:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7f4f5a117700 1 --2- 172.21.15.77:0/2214157319 >> v2:172.21.15.186:3300/0 conn(0x7f4f540c73a0 0x7f4f54159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:31.550 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.478+0000 7f4f5a918700 1 --2- 172.21.15.77:0/2214157319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f3c005040 0x7f4f3c00b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.552 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.479+0000 7f4f5a117700 1 --2- 172.21.15.77:0/2214157319 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f3c0039b0 0x7f4f3c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.552 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.479+0000 7f4f5a918700 1 -- 172.21.15.77:0/2214157319 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f3c0039b0 msgr2=0x7f4f3c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:31.552 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.479+0000 7f4f5a918700 1 --2- 172.21.15.77:0/2214157319 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f3c0039b0 0x7f4f3c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.552 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.479+0000 7f4f5a918700 1 -- 172.21.15.77:0/2214157319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f3c004690 msgr2=0x7f4f3c004b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:31.553 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.479+0000 7f4f5a918700 1 --2- 172.21.15.77:0/2214157319 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f3c004690 0x7f4f3c004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.553 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.479+0000 7f4f5a918700 1 -- 172.21.15.77:0/2214157319 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3c0033f0 con 0x7f4f3c005040 2022-04-03T19:44:31.553 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.479+0000 7f4f5a918700 1 -- 172.21.15.77:0/2214157319 reap_dead start 2022-04-03T19:44:31.553 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.479+0000 7f4f5a918700 1 --2- 172.21.15.77:0/2214157319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f3c005040 0x7f4f3c00b8e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4f5000c0e0 tx=0x7f4f50001da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac290b48c4a61a3d server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.553 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.480+0000 7f4f59115700 1 -- 172.21.15.77:0/2214157319 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5000ca10 con 0x7f4f3c005040 2022-04-03T19:44:31.554 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.480+0000 7f4f59115700 1 -- 172.21.15.77:0/2214157319 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5000cb70 con 0x7f4f3c005040 2022-04-03T19:44:31.554 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.481+0000 7f4f59115700 1 -- 172.21.15.77:0/2214157319 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f500070d0 con 0x7f4f3c005040 2022-04-03T19:44:31.554 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.482+0000 7f4f5c37b700 1 -- 172.21.15.77:0/2214157319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f3c005040 msgr2=0x7f4f3c00b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.554 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.482+0000 7f4f5c37b700 1 --2- 172.21.15.77:0/2214157319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f3c005040 0x7f4f3c00b8e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4f5000c0e0 tx=0x7f4f50001da0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.555 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.482+0000 7f4f5c37b700 1 -- 172.21.15.77:0/2214157319 shutdown_connections 2022-04-03T19:44:31.555 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.482+0000 7f4f5c37b700 1 --2- 172.21.15.77:0/2214157319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f3c005040 0x7f4f3c00b8e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.555 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.482+0000 7f4f5c37b700 1 -- 172.21.15.77:0/2214157319 >> 172.21.15.77:0/2214157319 conn(0x7f4f540cc300 msgr2=0x7f4f540c82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:31.557 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.486+0000 7fe4dd253700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4d004c840 0x7fe4d00db360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.557 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.486+0000 7fe4dd253700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4d004c840 0x7fe4d00db360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37798/0 (socket says 172.21.15.77:37798) 2022-04-03T19:44:31.557 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.487+0000 7fe4dca52700 1 -- 172.21.15.77:0/4154063053 learned_addr learned my addr 172.21.15.77:0/4154063053 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:31.557 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.487+0000 7fe4decb6700 1 -- 172.21.15.77:0/4154063053 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d00f6240 con 0x7fe4d000e330 2022-04-03T19:44:31.557 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.487+0000 7fe4decb6700 1 -- 172.21.15.77:0/4154063053 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d00f5db0 con 0x7fe4d004c840 2022-04-03T19:44:31.558 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.487+0000 7fe4decb6700 1 -- 172.21.15.77:0/4154063053 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d00ddb70 con 0x7fe4d000df60 2022-04-03T19:44:31.558 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.489+0000 7fe4d7fff700 1 --2- 172.21.15.77:0/4154063053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d000e330 0x7fe4d004a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.559 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.483+0000 7f4f5c37b700 1 -- 172.21.15.77:0/2214157319 shutdown_connections 2022-04-03T19:44:31.559 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.483+0000 7f4f5c37b700 1 -- 172.21.15.77:0/2214157319 wait complete. 2022-04-03T19:44:31.559 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.489+0000 7f4f5c37b700 1 Processor -- start 2022-04-03T19:44:31.559 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.490+0000 7fe4dca52700 1 -- 172.21.15.77:0/4154063053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4d004c840 msgr2=0x7fe4d00db360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.560 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.490+0000 7fe4dca52700 1 --2- 172.21.15.77:0/4154063053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4d004c840 0x7fe4d00db360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.560 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.490+0000 7fe4dca52700 1 -- 172.21.15.77:0/4154063053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d000e330 msgr2=0x7fe4d004a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.561 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.490+0000 7f4f5c37b700 1 -- start start 2022-04-03T19:44:31.561 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.490+0000 7fe4dca52700 1 --2- 172.21.15.77:0/4154063053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d000e330 0x7fe4d004a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.561 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.490+0000 7fe4dca52700 1 -- 172.21.15.77:0/4154063053 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d8063850 con 0x7fe4d000df60 2022-04-03T19:44:31.562 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.490+0000 7fe4dd253700 1 --2- 172.21.15.77:0/4154063053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4d004c840 0x7fe4d00db360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:31.562 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.491+0000 7fe4d7fff700 1 --2- 172.21.15.77:0/4154063053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d000e330 0x7fe4d004a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:31.564 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.491+0000 7f4f5c37b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f3c005040 0x7f4f540c4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.564 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.491+0000 7f4f5c37b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f540c51e0 0x7f4f5414e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.564 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.491+0000 7f4f5c37b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f5414ecd0 0x7f4f541640e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.564 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.491+0000 7f4f5c37b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f540c5630 con 0x7f4f540c51e0 2022-04-03T19:44:31.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.491+0000 7f4f5c37b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f540bf640 con 0x7f4f5414ecd0 2022-04-03T19:44:31.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.491+0000 7f4f5c37b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5414f140 con 0x7f4f3c005040 2022-04-03T19:44:31.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.492+0000 7f4f59916700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f540c51e0 0x7f4f5414e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.492+0000 7f4f59916700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f540c51e0 0x7f4f5414e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42604/0 (socket says 172.21.15.77:42604) 2022-04-03T19:44:31.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.492+0000 7f4f59916700 1 -- 172.21.15.77:0/4036955797 learned_addr learned my addr 172.21.15.77:0/4036955797 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:31.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.492+0000 7f4f5a117700 1 --2- 172.21.15.77:0/4036955797 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f3c005040 0x7f4f540c4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.493+0000 7f4f59916700 1 -- 172.21.15.77:0/4036955797 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f3c005040 msgr2=0x7f4f540c4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.493+0000 7f4f59916700 1 --2- 172.21.15.77:0/4036955797 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f3c005040 0x7f4f540c4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.493+0000 7f4f59916700 1 -- 172.21.15.77:0/4036955797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f5414ecd0 msgr2=0x7f4f541640e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2022-04-03T19:44:31.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.493+0000 7f4f59916700 1 --2- 172.21.15.77:0/4036955797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f5414ecd0 0x7f4f541640e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.493+0000 7f4f59916700 1 -- 172.21.15.77:0/4036955797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5414fd30 con 0x7f4f540c51e0 2022-04-03T19:44:31.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.493+0000 7f4f59916700 1 --2- 172.21.15.77:0/4036955797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f540c51e0 0x7f4f5414e790 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c009de0 tx=0x7f4f4c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.494+0000 7fe4dca52700 1 --2- 172.21.15.77:0/4154063053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4d000df60 0x7fe4d0048060 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8060620 tx=0x7fe4d8088000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.494+0000 7fe4d4ff9700 1 -- 172.21.15.77:0/4154063053 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8097070 con 0x7fe4d000df60 2022-04-03T19:44:31.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.495+0000 7fe4d4ff9700 1 -- 172.21.15.77:0/4154063053 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d806b070 con 0x7fe4d000df60 2022-04-03T19:44:31.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.495+0000 7fe4d4ff9700 1 -- 172.21.15.77:0/4154063053 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d806b410 con 0x7fe4d000df60 2022-04-03T19:44:31.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.495+0000 7fe4decb6700 1 -- 172.21.15.77:0/4154063053 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d00f5480 con 0x7fe4d000df60 2022-04-03T19:44:31.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.495+0000 7fe4decb6700 1 -- 172.21.15.77:0/4154063053 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d00f5a40 con 0x7fe4d000df60 2022-04-03T19:44:31.571 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.495+0000 7f4f427fc700 1 -- 172.21.15.77:0/4036955797 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c00caf0 con 0x7f4f540c51e0 2022-04-03T19:44:31.571 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.496+0000 7f4f5c37b700 1 -- 172.21.15.77:0/4036955797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f540c2330 con 0x7f4f540c51e0 2022-04-03T19:44:31.571 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.496+0000 7f4f5c37b700 1 -- 172.21.15.77:0/4036955797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f540c3230 con 0x7f4f540c51e0 2022-04-03T19:44:31.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.497+0000 7f4f427fc700 1 -- 172.21.15.77:0/4036955797 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4c00f040 con 0x7f4f540c51e0 2022-04-03T19:44:31.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.497+0000 7f4f427fc700 1 -- 172.21.15.77:0/4036955797 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c00cea0 con 0x7f4f540c51e0 2022-04-03T19:44:31.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.498+0000 7f4f5c37b700 1 -- 172.21.15.77:0/4036955797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f38005130 con 0x7f4f540c51e0 2022-04-03T19:44:31.573 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.500+0000 7fe4decb6700 1 -- 172.21.15.77:0/4154063053 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d0005640 con 0x7fe4d000df60 2022-04-03T19:44:31.574 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.503+0000 7f4f427fc700 1 -- 172.21.15.77:0/4036955797 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f4f4c024070 con 0x7f4f540c51e0 2022-04-03T19:44:31.574 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.504+0000 7f4f427fc700 1 --2- 172.21.15.77:0/4036955797 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4f34063e60 0x7f4f34066310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.574 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.504+0000 7f4f427fc700 1 -- 172.21.15.77:0/4036955797 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f4f4c081370 con 0x7f4f540c51e0 2022-04-03T19:44:31.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.501+0000 7fe4d4ff9700 1 -- 172.21.15.77:0/4154063053 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fe4d809b450 con 0x7fe4d000df60 2022-04-03T19:44:31.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.502+0000 7fe4d4ff9700 1 --2- 172.21.15.77:0/4154063053 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe4c0063e60 0x7fe4c0066310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:31.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.503+0000 7fe4d4ff9700 1 -- 172.21.15.77:0/4154063053 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fe4d80f8b20 con 0x7fe4d000df60 2022-04-03T19:44:31.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.507+0000 7fe4d4ff9700 1 -- 172.21.15.77:0/4154063053 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fe4d80c7630 con 0x7fe4d000df60 2022-04-03T19:44:31.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.507+0000 7fe4d7fff700 1 --2- 172.21.15.77:0/4154063053 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe4c0063e60 0x7fe4c0066310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.508+0000 7f4f5a117700 1 --2- 172.21.15.77:0/4036955797 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4f34063e60 0x7f4f34066310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:31.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.510+0000 7f4f427fc700 1 -- 172.21.15.77:0/4036955797 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f4f4c04fee0 con 0x7f4f540c51e0 2022-04-03T19:44:31.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.511+0000 7f4f5a117700 1 --2- 172.21.15.77:0/4036955797 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4f34063e60 0x7f4f34066310 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4f44009270 tx=0x7f4f44008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.519+0000 7fe4d7fff700 1 --2- 172.21.15.77:0/4154063053 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe4c0063e60 0x7fe4c0066310 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe4d000ec80 tx=0x7fe4c4006f10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:31.821 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.821+0000 7f59fbfff700 1 -- 172.21.15.77:0/2180923786 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f59f4005710 con 0x7f59f4049640 2022-04-03T19:44:31.823 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.822+0000 7f59e2ffd700 1 -- 172.21.15.77:0/2180923786 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f59ec055e60 con 0x7f59f4049640 2022-04-03T19:44:31.823 INFO:teuthology.orchestra.run.smithi077.stdout:55834574857 2022-04-03T19:44:31.827 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.827+0000 7f59e0ff9700 1 -- 172.21.15.77:0/2180923786 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f59dc063e00 msgr2=0x7f59dc0662b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.828 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.827+0000 7f59e0ff9700 1 --2- 172.21.15.77:0/2180923786 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f59dc063e00 0x7f59dc0662b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f59f400f480 tx=0x7f59fc088000 comp rx=0 tx=0).stop 2022-04-03T19:44:31.829 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.828+0000 7f59e0ff9700 1 -- 172.21.15.77:0/2180923786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59f4049640 msgr2=0x7f59f40d3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.829 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.828+0000 7f59e0ff9700 1 --2- 172.21.15.77:0/2180923786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59f4049640 0x7f59f40d3750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f59ec00b4c0 tx=0x7f59ec009060 comp rx=0 tx=0).stop 2022-04-03T19:44:31.831 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.830+0000 7f59e0ff9700 1 -- 172.21.15.77:0/2180923786 shutdown_connections 2022-04-03T19:44:31.832 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.830+0000 7f59e0ff9700 1 --2- 172.21.15.77:0/2180923786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59f40dcd40 0x7f59f4007b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.832 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.830+0000 7f59e0ff9700 1 --2- 172.21.15.77:0/2180923786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59f4049640 0x7f59f40d3750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.832 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.830+0000 7f59e0ff9700 1 --2- 172.21.15.77:0/2180923786 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f59dc063e00 0x7f59dc0662b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.832 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.831+0000 7f59e0ff9700 1 --2- 172.21.15.77:0/2180923786 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59f404f3e0 0x7f59f40d0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.832 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.831+0000 7f59e0ff9700 1 -- 172.21.15.77:0/2180923786 >> 172.21.15.77:0/2180923786 conn(0x7f59f4009d10 msgr2=0x7f59f40de1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:31.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.832+0000 7f59e0ff9700 1 -- 172.21.15.77:0/2180923786 shutdown_connections 2022-04-03T19:44:31.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.832+0000 7f59e0ff9700 1 -- 172.21.15.77:0/2180923786 wait complete. 2022-04-03T19:44:31.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.833+0000 7f77e495b700 1 -- 172.21.15.77:0/3172866390 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f77bc0053e0 con 0x7f77d804ee70 2022-04-03T19:44:31.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.833+0000 7f77c6ffd700 1 -- 172.21.15.77:0/3172866390 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f77d0021090 con 0x7f77d804ee70 2022-04-03T19:44:31.835 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2022-04-03T19:44:31.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.838+0000 7f77e495b700 1 -- 172.21.15.77:0/3172866390 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f77c0063db0 msgr2=0x7f77c0066260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.839+0000 7f77e495b700 1 --2- 172.21.15.77:0/3172866390 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f77c0063db0 0x7f77c0066260 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f77e006f650 tx=0x7f77e0077000 comp rx=0 tx=0).stop 2022-04-03T19:44:31.840 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.838+0000 7f2cbd8b1700 1 -- 172.21.15.77:0/2707584730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f2c940053e0 con 0x7f2cb806bf60 2022-04-03T19:44:31.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.839+0000 7f77e495b700 1 -- 172.21.15.77:0/3172866390 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77d804ee70 msgr2=0x7f77d80eee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.839+0000 7f77e495b700 1 --2- 172.21.15.77:0/3172866390 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77d804ee70 0x7f77d80eee50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f77d8012100 tx=0x7f77d0009060 comp rx=0 tx=0).stop 2022-04-03T19:44:31.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.839+0000 7f77e495b700 1 -- 172.21.15.77:0/3172866390 shutdown_connections 2022-04-03T19:44:31.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.840+0000 7f77e495b700 1 --2- 172.21.15.77:0/3172866390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77d80de880 0x7f77d80dede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.840+0000 7f77e495b700 1 --2- 172.21.15.77:0/3172866390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77d8006b40 0x7f77d80d0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.843 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.840+0000 7f2c9effd700 1 -- 172.21.15.77:0/2707584730 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2cac04e470 con 0x7f2cb806bf60 2022-04-03T19:44:31.843 INFO:teuthology.orchestra.run.smithi077.stdout:55834574855 2022-04-03T19:44:31.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.840+0000 7f77e495b700 1 --2- 172.21.15.77:0/3172866390 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f77c0063db0 0x7f77c0066260 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.840+0000 7f77e495b700 1 --2- 172.21.15.77:0/3172866390 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77d804ee70 0x7f77d80eee50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.841+0000 7f77e495b700 1 -- 172.21.15.77:0/3172866390 >> 172.21.15.77:0/3172866390 conn(0x7f77d8010d00 msgr2=0x7f77d80dcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:31.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.841+0000 7f77e495b700 1 -- 172.21.15.77:0/3172866390 shutdown_connections 2022-04-03T19:44:31.846 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.842+0000 7f77e495b700 1 -- 172.21.15.77:0/3172866390 wait complete. 2022-04-03T19:44:31.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.844+0000 7f2c9cff9700 1 -- 172.21.15.77:0/2707584730 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2c98063e30 msgr2=0x7f2c980662e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.844+0000 7f2c9cff9700 1 --2- 172.21.15.77:0/2707584730 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2c98063e30 0x7f2c980662e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2cb806bf30 tx=0x7f2ca8008040 comp rx=0 tx=0).stop 2022-04-03T19:44:31.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.845+0000 7f2c9cff9700 1 -- 172.21.15.77:0/2707584730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cb806bf60 msgr2=0x7f2cb8161760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.845+0000 7f2c9cff9700 1 --2- 172.21.15.77:0/2707584730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cb806bf60 0x7f2cb8161760 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2cac00c4a0 tx=0x7f2cac007db0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.845+0000 7f2c9cff9700 1 -- 172.21.15.77:0/2707584730 shutdown_connections 2022-04-03T19:44:31.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.845+0000 7f2c9cff9700 1 --2- 172.21.15.77:0/2707584730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cb80a9a60 0x7f2cb80abac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.845+0000 7f2c9cff9700 1 --2- 172.21.15.77:0/2707584730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cb806bf60 0x7f2cb8161760 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.845+0000 7f2c9cff9700 1 --2- 172.21.15.77:0/2707584730 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2c98063e30 0x7f2c980662e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.845+0000 7f2c9cff9700 1 --2- 172.21.15.77:0/2707584730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2cb806c650 0x7f2cb814ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.845+0000 7f2c9cff9700 1 -- 172.21.15.77:0/2707584730 >> 172.21.15.77:0/2707584730 conn(0x7f2cb80c71c0 msgr2=0x7f2cb80c4fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:31.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.846+0000 7f2c9cff9700 1 -- 172.21.15.77:0/2707584730 shutdown_connections 2022-04-03T19:44:31.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.846+0000 7f2c9cff9700 1 -- 172.21.15.77:0/2707584730 wait complete. 2022-04-03T19:44:31.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.845+0000 7f877cb2c700 1 -- 172.21.15.77:0/3038807238 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f87700c8a30 con 0x7f877004a840 2022-04-03T19:44:31.851 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.847+0000 7f87667fc700 1 -- 172.21.15.77:0/3038807238 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f87780c6e50 con 0x7f877004a840 2022-04-03T19:44:31.851 INFO:teuthology.orchestra.run.smithi077.stdout:55834574856 2022-04-03T19:44:31.852 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.851+0000 7f877cb2c700 1 -- 172.21.15.77:0/3038807238 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8754063db0 msgr2=0x7f8754066260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.853 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.851+0000 7f877cb2c700 1 --2- 172.21.15.77:0/3038807238 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8754063db0 0x7f8754066260 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8768006fd0 tx=0x7f8768008040 comp rx=0 tx=0).stop 2022-04-03T19:44:31.853 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.851+0000 7f877cb2c700 1 -- 172.21.15.77:0/3038807238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f877004a840 msgr2=0x7f87700c4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.853 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.851+0000 7f877cb2c700 1 --2- 172.21.15.77:0/3038807238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f877004a840 0x7f87700c4250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f877804c3f0 tx=0x7f877808b000 comp rx=0 tx=0).stop 2022-04-03T19:44:31.855 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.853+0000 7f877cb2c700 1 -- 172.21.15.77:0/3038807238 shutdown_connections 2022-04-03T19:44:31.855 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.853+0000 7f877cb2c700 1 --2- 172.21.15.77:0/3038807238 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8754063db0 0x7f8754066260 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.853+0000 7f877cb2c700 1 --2- 172.21.15.77:0/3038807238 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f877004af90 0x7f87700c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.854+0000 7f877cb2c700 1 --2- 172.21.15.77:0/3038807238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f877000c790 0x7f87700d8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.854+0000 7f877cb2c700 1 --2- 172.21.15.77:0/3038807238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f877004a840 0x7f87700c4250 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.854+0000 7f877cb2c700 1 -- 172.21.15.77:0/3038807238 >> 172.21.15.77:0/3038807238 conn(0x7f877000bc80 msgr2=0x7f877004dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:31.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.855+0000 7f877cb2c700 1 -- 172.21.15.77:0/3038807238 shutdown_connections 2022-04-03T19:44:31.857 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.855+0000 7f877cb2c700 1 -- 172.21.15.77:0/3038807238 wait complete. 2022-04-03T19:44:31.858 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.858+0000 7f7e2bfff700 1 -- 172.21.15.77:0/2124168297 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f7e240d5a10 con 0x7f7e240d3f70 2022-04-03T19:44:31.859 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.859+0000 7f7e12ffd700 1 -- 172.21.15.77:0/2124168297 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7e240d5a10 con 0x7f7e240d3f70 2022-04-03T19:44:31.860 INFO:teuthology.orchestra.run.smithi077.stdout:55834574856 2022-04-03T19:44:31.865 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.864+0000 7f7e10ff9700 1 -- 172.21.15.77:0/2124168297 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7e0c063d60 msgr2=0x7f7e0c066210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.866 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.864+0000 7f7e10ff9700 1 --2- 172.21.15.77:0/2124168297 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7e0c063d60 0x7f7e0c066210 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7e20006bc0 tx=0x7f7e200090f0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.866 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.864+0000 7f7e10ff9700 1 -- 172.21.15.77:0/2124168297 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7e240d3f70 msgr2=0x7f7e240d6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.866 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.864+0000 7f7e10ff9700 1 --2- 172.21.15.77:0/2124168297 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7e240d3f70 0x7f7e240d6060 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c007fd0 tx=0x7f7e1c007c70 comp rx=0 tx=0).stop 2022-04-03T19:44:31.866 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.865+0000 7f7e10ff9700 1 -- 172.21.15.77:0/2124168297 shutdown_connections 2022-04-03T19:44:31.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.865+0000 7f7e10ff9700 1 --2- 172.21.15.77:0/2124168297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7e240da780 0x7f7e2404e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.865+0000 7f7e10ff9700 1 --2- 172.21.15.77:0/2124168297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7e24009a20 0x7f7e240f1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.865+0000 7f7e10ff9700 1 --2- 172.21.15.77:0/2124168297 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7e0c063d60 0x7f7e0c066210 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.865+0000 7f7e10ff9700 1 --2- 172.21.15.77:0/2124168297 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7e240d3f70 0x7f7e240d6060 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.865+0000 7f7e10ff9700 1 -- 172.21.15.77:0/2124168297 >> 172.21.15.77:0/2124168297 conn(0x7f7e2400bc70 msgr2=0x7f7e240d6ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:31.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.866+0000 7f7e10ff9700 1 -- 172.21.15.77:0/2124168297 shutdown_connections 2022-04-03T19:44:31.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.866+0000 7f7e10ff9700 1 -- 172.21.15.77:0/2124168297 wait complete. 2022-04-03T19:44:31.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.867+0000 7f8c35846700 1 -- 172.21.15.77:0/2140680340 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f8c3004b890 con 0x7f8c3014e0b0 2022-04-03T19:44:31.870 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.868+0000 7f8c16ffd700 1 -- 172.21.15.77:0/2140680340 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8c20052950 con 0x7f8c3014e0b0 2022-04-03T19:44:31.872 INFO:teuthology.orchestra.run.smithi077.stdout:55834574856 2022-04-03T19:44:31.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.872+0000 7f8c14ff9700 1 -- 172.21.15.77:0/2140680340 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8c10063db0 msgr2=0x7f8c10066260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.873 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.872+0000 7f8c14ff9700 1 --2- 172.21.15.77:0/2140680340 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8c10063db0 0x7f8c10066260 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8c24005670 tx=0x7f8c24019000 comp rx=0 tx=0).stop 2022-04-03T19:44:31.873 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.872+0000 7f8c14ff9700 1 -- 172.21.15.77:0/2140680340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c3014e0b0 msgr2=0x7f8c300be080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.873 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.872+0000 7f8c14ff9700 1 --2- 172.21.15.77:0/2140680340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c3014e0b0 0x7f8c300be080 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8c2000ba20 tx=0x7f8c2000d780 comp rx=0 tx=0).stop 2022-04-03T19:44:31.873 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.872+0000 7f8c14ff9700 1 -- 172.21.15.77:0/2140680340 shutdown_connections 2022-04-03T19:44:31.873 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.872+0000 7f8c14ff9700 1 --2- 172.21.15.77:0/2140680340 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8c10063db0 0x7f8c10066260 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8c24005670 tx=0x7f8c24019000 comp rx=0 tx=0).stop 2022-04-03T19:44:31.874 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.872+0000 7f8c14ff9700 1 --2- 172.21.15.77:0/2140680340 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c3014e520 0x7f8c300c8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.874 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.872+0000 7f8c14ff9700 1 --2- 172.21.15.77:0/2140680340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c1c0038e0 0x7f8c301619d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.874 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.872+0000 7f8c14ff9700 1 --2- 172.21.15.77:0/2140680340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c3014e0b0 0x7f8c300be080 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8c2000ba20 tx=0x7f8c2000d780 comp rx=0 tx=0).stop 2022-04-03T19:44:31.874 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.872+0000 7f8c14ff9700 1 -- 172.21.15.77:0/2140680340 >> 172.21.15.77:0/2140680340 conn(0x7f8c300626d0 msgr2=0x7f8c300c7480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:31.875 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.875+0000 7f8c14ff9700 1 -- 172.21.15.77:0/2140680340 shutdown_connections 2022-04-03T19:44:31.875 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.875+0000 7f8c14ff9700 1 -- 172.21.15.77:0/2140680340 wait complete. 2022-04-03T19:44:31.881 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.1 2022-04-03T19:44:31.882 DEBUG:teuthology.parallel:result is None 2022-04-03T19:44:31.888 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.6 2022-04-03T19:44:31.888 DEBUG:teuthology.parallel:result is None 2022-04-03T19:44:31.890 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2022-04-03T19:44:31.890 DEBUG:teuthology.parallel:result is None 2022-04-03T19:44:31.896 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.0 2022-04-03T19:44:31.896 DEBUG:teuthology.parallel:result is None 2022-04-03T19:44:31.911 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.5 2022-04-03T19:44:31.911 DEBUG:teuthology.parallel:result is None 2022-04-03T19:44:31.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.917+0000 7fe4decb6700 1 -- 172.21.15.77:0/4154063053 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fe4d00f6090 con 0x7fe4d000df60 2022-04-03T19:44:31.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.917+0000 7fe4d4ff9700 1 -- 172.21.15.77:0/4154063053 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe4d808b090 con 0x7fe4d000df60 2022-04-03T19:44:31.918 INFO:teuthology.orchestra.run.smithi077.stdout:55834574856 2022-04-03T19:44:31.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.920+0000 7fe4decb6700 1 -- 172.21.15.77:0/4154063053 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe4c0063e60 msgr2=0x7fe4c0066310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.920+0000 7fe4decb6700 1 --2- 172.21.15.77:0/4154063053 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe4c0063e60 0x7fe4c0066310 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe4d000ec80 tx=0x7fe4c4006f10 comp rx=0 tx=0).stop 2022-04-03T19:44:31.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.920+0000 7fe4decb6700 1 -- 172.21.15.77:0/4154063053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4d000df60 msgr2=0x7fe4d0048060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.920+0000 7fe4decb6700 1 --2- 172.21.15.77:0/4154063053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4d000df60 0x7fe4d0048060 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8060620 tx=0x7fe4d8088000 comp rx=0 tx=0).stop 2022-04-03T19:44:31.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.920+0000 7fe4decb6700 1 -- 172.21.15.77:0/4154063053 shutdown_connections 2022-04-03T19:44:31.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.920+0000 7fe4decb6700 1 --2- 172.21.15.77:0/4154063053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4d004c840 0x7fe4d00db360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.920+0000 7fe4decb6700 1 --2- 172.21.15.77:0/4154063053 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe4c0063e60 0x7fe4c0066310 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.920+0000 7fe4decb6700 1 --2- 172.21.15.77:0/4154063053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4d000df60 0x7fe4d0048060 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.920+0000 7fe4decb6700 1 --2- 172.21.15.77:0/4154063053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d000e330 0x7fe4d004a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.920+0000 7fe4decb6700 1 -- 172.21.15.77:0/4154063053 >> 172.21.15.77:0/4154063053 conn(0x7fe4d0051f90 msgr2=0x7fe4d00e3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:31.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.920+0000 7fe4decb6700 1 -- 172.21.15.77:0/4154063053 shutdown_connections 2022-04-03T19:44:31.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.920+0000 7fe4decb6700 1 -- 172.21.15.77:0/4154063053 wait complete. 2022-04-03T19:44:31.924 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2022-04-03T19:44:31.925 DEBUG:teuthology.parallel:result is None 2022-04-03T19:44:31.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.930+0000 7f4f5c37b700 1 -- 172.21.15.77:0/4036955797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f4f380053e0 con 0x7f4f540c51e0 2022-04-03T19:44:31.931 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.931+0000 7f4f427fc700 1 -- 172.21.15.77:0/4036955797 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4f4c018020 con 0x7f4f540c51e0 2022-04-03T19:44:31.931 INFO:teuthology.orchestra.run.smithi077.stdout:55834574858 2022-04-03T19:44:31.934 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.933+0000 7f4f5c37b700 1 -- 172.21.15.77:0/4036955797 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4f34063e60 msgr2=0x7f4f34066310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.934 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.933+0000 7f4f5c37b700 1 --2- 172.21.15.77:0/4036955797 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4f34063e60 0x7f4f34066310 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4f44009270 tx=0x7f4f44008040 comp rx=0 tx=0).stop 2022-04-03T19:44:31.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.933+0000 7f4f5c37b700 1 -- 172.21.15.77:0/4036955797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f540c51e0 msgr2=0x7f4f5414e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:31.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.933+0000 7f4f5c37b700 1 --2- 172.21.15.77:0/4036955797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f540c51e0 0x7f4f5414e790 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c009de0 tx=0x7f4f4c00b040 comp rx=0 tx=0).stop 2022-04-03T19:44:31.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.934+0000 7f4f5c37b700 1 -- 172.21.15.77:0/4036955797 shutdown_connections 2022-04-03T19:44:31.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.934+0000 7f4f5c37b700 1 --2- 172.21.15.77:0/4036955797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f5414ecd0 0x7f4f541640e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.934+0000 7f4f5c37b700 1 --2- 172.21.15.77:0/4036955797 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4f34063e60 0x7f4f34066310 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.934+0000 7f4f5c37b700 1 --2- 172.21.15.77:0/4036955797 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f3c005040 0x7f4f540c4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.934+0000 7f4f5c37b700 1 --2- 172.21.15.77:0/4036955797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4f540c51e0 0x7f4f5414e790 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:31.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.934+0000 7f4f5c37b700 1 -- 172.21.15.77:0/4036955797 >> 172.21.15.77:0/4036955797 conn(0x7f4f540cc300 msgr2=0x7f4f5406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:31.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.934+0000 7f4f5c37b700 1 -- 172.21.15.77:0/4036955797 shutdown_connections 2022-04-03T19:44:31.937 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:31.934+0000 7f4f5c37b700 1 -- 172.21.15.77:0/4036955797 wait complete. 2022-04-03T19:44:31.945 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.7 2022-04-03T19:44:31.945 DEBUG:teuthology.parallel:result is None 2022-04-03T19:44:31.958 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.2 2022-04-03T19:44:31.958 DEBUG:teuthology.parallel:result is None 2022-04-03T19:44:31.958 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2022-04-03T19:44:31.959 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-04-03T19:44:32.195 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.195+0000 7f08bbfff700 1 Processor -- start 2022-04-03T19:44:32.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.195+0000 7f08bbfff700 1 -- start start 2022-04-03T19:44:32.198 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.196+0000 7f08bbfff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08bc06bfb0 0x7f08bc1584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:32.198 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.196+0000 7f08bbfff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f08bc06c500 0x7f08bc14eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:32.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.196+0000 7f08bbfff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08bc06d260 con 0x7f08bc06bbe0 2022-04-03T19:44:32.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.196+0000 7f08bbfff700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08bc159450 con 0x7f08bc06c500 2022-04-03T19:44:32.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.196+0000 7f08bbfff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc0c6330 con 0x7f08bc06bfb0 2022-04-03T19:44:32.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.197+0000 7f08baffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f08bc06bbe0 0x7f08bc06d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38332/0 (socket says 172.21.15.77:38332) 2022-04-03T19:44:32.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.197+0000 7f08baffd700 1 -- 172.21.15.77:0/9006175 learned_addr learned my addr 172.21.15.77:0/9006175 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:32.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.197+0000 7f08bb7fe700 1 --2- 172.21.15.77:0/9006175 >> v2:172.21.15.186:3300/0 conn(0x7f08bc06c500 0x7f08bc14eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:32.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.197+0000 7f08ba7fc700 1 --2- 172.21.15.77:0/9006175 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08bc06bfb0 0x7f08bc1584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:32.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.197+0000 7f08b9ffb700 1 -- 172.21.15.77:0/9006175 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943523508 0 0) 0x7f08bc06d260 con 0x7f08bc06bbe0 2022-04-03T19:44:32.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.197+0000 7f08b9ffb700 1 -- 172.21.15.77:0/9006175 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08a80033f0 con 0x7f08bc06bbe0 2022-04-03T19:44:32.201 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.197+0000 7f08bb7fe700 1 -- 172.21.15.77:0/9006175 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08bc06bfb0 msgr2=0x7f08bc1584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:32.201 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.197+0000 7f08bb7fe700 1 --2- 172.21.15.77:0/9006175 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08bc06bfb0 0x7f08bc1584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:32.201 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.197+0000 7f08bb7fe700 1 -- 172.21.15.77:0/9006175 >> v1:172.21.15.77:6789/0 conn(0x7f08bc06bbe0 legacy=0x7f08bc06d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:32.201 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.197+0000 7f08bb7fe700 1 -- 172.21.15.77:0/9006175 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08bc158bb0 con 0x7f08bc06c500 2022-04-03T19:44:32.201 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.197+0000 7f08ba7fc700 1 --2- 172.21.15.77:0/9006175 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08bc06bfb0 0x7f08bc1584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:32.202 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.198+0000 7f08bb7fe700 1 --2- 172.21.15.77:0/9006175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08bc06c500 0x7f08bc14eea0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f08b4009e60 tx=0x7f08b4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0988f239d5c53b0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:32.202 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.198+0000 7f08b9ffb700 1 -- 172.21.15.77:0/9006175 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b400e070 con 0x7f08bc06c500 2022-04-03T19:44:32.202 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.198+0000 7f08b9ffb700 1 -- 172.21.15.77:0/9006175 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b400bab0 con 0x7f08bc06c500 2022-04-03T19:44:32.202 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.198+0000 7f08b9ffb700 1 -- 172.21.15.77:0/9006175 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b400be70 con 0x7f08bc06c500 2022-04-03T19:44:32.206 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.198+0000 7f08bbfff700 1 -- 172.21.15.77:0/9006175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08bc06c500 msgr2=0x7f08bc14eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:32.206 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.198+0000 7f08bbfff700 1 --2- 172.21.15.77:0/9006175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08bc06c500 0x7f08bc14eea0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f08b4009e60 tx=0x7f08b4009780 comp rx=0 tx=0).stop 2022-04-03T19:44:32.206 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.198+0000 7f08bbfff700 1 -- 172.21.15.77:0/9006175 shutdown_connections 2022-04-03T19:44:32.206 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.198+0000 7f08bbfff700 1 --2- 172.21.15.77:0/9006175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08bc06c500 0x7f08bc14eea0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:32.206 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.198+0000 7f08bbfff700 1 --2- 172.21.15.77:0/9006175 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08bc06bfb0 0x7f08bc1584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:32.207 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.198+0000 7f08bbfff700 1 -- 172.21.15.77:0/9006175 >> 172.21.15.77:0/9006175 conn(0x7f08bc0cb6b0 msgr2=0x7f08bc0cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:32.207 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.198+0000 7f08bbfff700 1 -- 172.21.15.77:0/9006175 shutdown_connections 2022-04-03T19:44:32.207 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.199+0000 7f08bbfff700 1 -- 172.21.15.77:0/9006175 wait complete. 2022-04-03T19:44:32.207 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.200+0000 7f08bbfff700 1 Processor -- start 2022-04-03T19:44:32.208 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.200+0000 7f08bbfff700 1 -- start start 2022-04-03T19:44:32.208 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.201+0000 7f08bbfff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08bc06bbe0 0x7f08bc165ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:32.208 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.201+0000 7f08bbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08bc06bfb0 0x7f08bc167da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:32.208 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.201+0000 7f08bbfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08bc06c500 0x7f08bc1682e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.201+0000 7f08bbfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc151e80 con 0x7f08bc06bfb0 2022-04-03T19:44:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.201+0000 7f08bbfff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc155790 con 0x7f08bc06c500 2022-04-03T19:44:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.201+0000 7f08bbfff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc168c50 con 0x7f08bc06bbe0 2022-04-03T19:44:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.201+0000 7f08ba7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08bc06bfb0 0x7f08bc167da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:32.209 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.201+0000 7f08ba7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08bc06bfb0 0x7f08bc167da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42606/0 (socket says 172.21.15.77:42606) 2022-04-03T19:44:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.201+0000 7f08ba7fc700 1 -- 172.21.15.77:0/539897179 learned_addr learned my addr 172.21.15.77:0/539897179 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.201+0000 7f08bb7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08bc06c500 0x7f08bc1682e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.201+0000 7f08baffd700 1 --2- 172.21.15.77:0/539897179 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08bc06bbe0 0x7f08bc165ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.202+0000 7f08ba7fc700 1 -- 172.21.15.77:0/539897179 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08bc06bbe0 msgr2=0x7f08bc165ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:32.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.202+0000 7f08ba7fc700 1 --2- 172.21.15.77:0/539897179 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08bc06bbe0 0x7f08bc165ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:32.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.202+0000 7f08ba7fc700 1 -- 172.21.15.77:0/539897179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08bc06c500 msgr2=0x7f08bc1682e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:32.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.202+0000 7f08ba7fc700 1 --2- 172.21.15.77:0/539897179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08bc06c500 0x7f08bc1682e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:32.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.202+0000 7f08ba7fc700 1 -- 172.21.15.77:0/539897179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b00092a0 con 0x7f08bc06bfb0 2022-04-03T19:44:32.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.202+0000 7f08bb7fe700 1 --2- 172.21.15.77:0/539897179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08bc06c500 0x7f08bc1682e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:32.212 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.202+0000 7f08ba7fc700 1 --2- 172.21.15.77:0/539897179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08bc06bfb0 0x7f08bc167da0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f08b000ea50 tx=0x7f08b0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:32.212 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.202+0000 7f08a2ffd700 1 -- 172.21.15.77:0/539897179 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b0009600 con 0x7f08bc06bfb0 2022-04-03T19:44:32.212 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.202+0000 7f08a2ffd700 1 -- 172.21.15.77:0/539897179 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b000c850 con 0x7f08bc06bfb0 2022-04-03T19:44:32.212 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.202+0000 7f08a2ffd700 1 -- 172.21.15.77:0/539897179 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b000cbf0 con 0x7f08bc06bfb0 2022-04-03T19:44:32.212 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.202+0000 7f08bbfff700 1 -- 172.21.15.77:0/539897179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b4009a20 con 0x7f08bc06bfb0 2022-04-03T19:44:32.213 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.203+0000 7f08bbfff700 1 -- 172.21.15.77:0/539897179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08bc151240 con 0x7f08bc06bfb0 2022-04-03T19:44:32.213 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.203+0000 7f08bbfff700 1 -- 172.21.15.77:0/539897179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08bc0c3610 con 0x7f08bc06bfb0 2022-04-03T19:44:32.213 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.204+0000 7f08a2ffd700 1 -- 172.21.15.77:0/539897179 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f08b0019030 con 0x7f08bc06bfb0 2022-04-03T19:44:32.213 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.205+0000 7f08a2ffd700 1 --2- 172.21.15.77:0/539897179 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f089c063d10 0x7f089c0661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:32.213 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.205+0000 7f08a2ffd700 1 -- 172.21.15.77:0/539897179 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f08b00807c0 con 0x7f08bc06bfb0 2022-04-03T19:44:32.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.206+0000 7f08baffd700 1 --2- 172.21.15.77:0/539897179 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f089c063d10 0x7f089c0661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:32.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.206+0000 7f08baffd700 1 --2- 172.21.15.77:0/539897179 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f089c063d10 0x7f089c0661c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f08ac005750 tx=0x7f08ac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:32.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.209+0000 7f08a2ffd700 1 -- 172.21.15.77:0/539897179 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f08b004f3a0 con 0x7f08bc06bfb0 2022-04-03T19:44:32.403 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.403+0000 7f08bbfff700 1 -- 172.21.15.77:0/539897179 --> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f08bc04b050 con 0x7f089c063d10 2022-04-03T19:44:32.413 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.412+0000 7f08a2ffd700 1 -- 172.21.15.77:0/539897179 <== mgr.4106 v2:172.21.15.186:6832/24948 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284624 (secure 0 0 0) 0x7f08bc04b050 con 0x7f089c063d10 2022-04-03T19:44:32.416 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:32.419 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.417+0000 7f08bbfff700 1 -- 172.21.15.77:0/539897179 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f089c063d10 msgr2=0x7f089c0661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:32.419 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.417+0000 7f08bbfff700 1 --2- 172.21.15.77:0/539897179 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f089c063d10 0x7f089c0661c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f08ac005750 tx=0x7f08ac009040 comp rx=0 tx=0).stop 2022-04-03T19:44:32.419 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.417+0000 7f08bbfff700 1 -- 172.21.15.77:0/539897179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08bc06bfb0 msgr2=0x7f08bc167da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:32.420 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.417+0000 7f08bbfff700 1 --2- 172.21.15.77:0/539897179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08bc06bfb0 0x7f08bc167da0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f08b000ea50 tx=0x7f08b0009060 comp rx=0 tx=0).stop 2022-04-03T19:44:32.420 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.417+0000 7f08bbfff700 1 -- 172.21.15.77:0/539897179 shutdown_connections 2022-04-03T19:44:32.420 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.417+0000 7f08bbfff700 1 --2- 172.21.15.77:0/539897179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08bc06c500 0x7f08bc1682e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:32.420 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.417+0000 7f08bbfff700 1 --2- 172.21.15.77:0/539897179 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f089c063d10 0x7f089c0661c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:32.421 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.417+0000 7f08bbfff700 1 --2- 172.21.15.77:0/539897179 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f08bc06bbe0 0x7f08bc165ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:32.421 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.417+0000 7f08bbfff700 1 --2- 172.21.15.77:0/539897179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08bc06bfb0 0x7f08bc167da0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:32.421 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.417+0000 7f08bbfff700 1 -- 172.21.15.77:0/539897179 >> 172.21.15.77:0/539897179 conn(0x7f08bc0cb6b0 msgr2=0x7f08bc14d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:32.421 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.417+0000 7f08bbfff700 1 -- 172.21.15.77:0/539897179 shutdown_connections 2022-04-03T19:44:32.421 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.417+0000 7f08bbfff700 1 -- 172.21.15.77:0/539897179 wait complete. 2022-04-03T19:44:32.422 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2022-04-03T19:44:32.442 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2022-04-03T19:44:31.490856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":65808,"kb_used_data":3088,"kb_used_omap":0,"kb_used_meta":62656,"kb_avail":754908912,"statfs":{"total":773094113280,"available":773026725888,"internally_reserved":0,"allocated":3162112,"data_stored":1814308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64159744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002741"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302418+0000","last_change":"2022-04-03T19:44:21.300867+0000","last_active":"2022-04-03T19:44:21.302418+0000","last_peered":"2022-04-03T19:44:21.302418+0000","last_clean":"2022-04-03T19:44:21.302418+0000","last_became_active":"2022-04-03T19:44:21.300569+0000","last_became_peered":"2022-04-03T19:44:21.300569+0000","last_unstale":"2022-04-03T19:44:21.302418+0000","last_undegraded":"2022-04-03T19:44:21.302418+0000","last_fullsized":"2022-04-03T19:44:21.302418+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:10:32.827314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305911+0000","last_change":"2022-04-03T19:44:21.302410+0000","last_active":"2022-04-03T19:44:21.305911+0000","last_peered":"2022-04-03T19:44:21.305911+0000","last_clean":"2022-04-03T19:44:21.305911+0000","last_became_active":"2022-04-03T19:44:21.302142+0000","last_became_peered":"2022-04-03T19:44:21.302142+0000","last_unstale":"2022-04-03T19:44:21.305911+0000","last_undegraded":"2022-04-03T19:44:21.305911+0000","last_fullsized":"2022-04-03T19:44:21.305911+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:01:46.327996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.298511+0000","last_change":"2022-04-03T19:44:21.293080+0000","last_active":"2022-04-03T19:44:21.298511+0000","last_peered":"2022-04-03T19:44:21.298511+0000","last_clean":"2022-04-03T19:44:21.298511+0000","last_became_active":"2022-04-03T19:44:21.292762+0000","last_became_peered":"2022-04-03T19:44:21.292762+0000","last_unstale":"2022-04-03T19:44:21.298511+0000","last_undegraded":"2022-04-03T19:44:21.298511+0000","last_fullsized":"2022-04-03T19:44:21.298511+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:28:36.726593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300326+0000","last_change":"2022-04-03T19:44:21.294448+0000","last_active":"2022-04-03T19:44:21.300326+0000","last_peered":"2022-04-03T19:44:21.300326+0000","last_clean":"2022-04-03T19:44:21.300326+0000","last_became_active":"2022-04-03T19:44:21.294083+0000","last_became_peered":"2022-04-03T19:44:21.294083+0000","last_unstale":"2022-04-03T19:44:21.300326+0000","last_undegraded":"2022-04-03T19:44:21.300326+0000","last_fullsized":"2022-04-03T19:44:21.300326+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:57:02.374826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.308268+0000","last_change":"2022-04-03T19:44:21.299551+0000","last_active":"2022-04-03T19:44:21.308268+0000","last_peered":"2022-04-03T19:44:21.308268+0000","last_clean":"2022-04-03T19:44:21.308268+0000","last_became_active":"2022-04-03T19:44:21.299123+0000","last_became_peered":"2022-04-03T19:44:21.299123+0000","last_unstale":"2022-04-03T19:44:21.308268+0000","last_undegraded":"2022-04-03T19:44:21.308268+0000","last_fullsized":"2022-04-03T19:44:21.308268+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:08:24.911647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300209+0000","last_change":"2022-04-03T19:44:21.293129+0000","last_active":"2022-04-03T19:44:21.300209+0000","last_peered":"2022-04-03T19:44:21.300209+0000","last_clean":"2022-04-03T19:44:21.300209+0000","last_became_active":"2022-04-03T19:44:21.292843+0000","last_became_peered":"2022-04-03T19:44:21.292843+0000","last_unstale":"2022-04-03T19:44:21.300209+0000","last_undegraded":"2022-04-03T19:44:21.300209+0000","last_fullsized":"2022-04-03T19:44:21.300209+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:38:32.432609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302772+0000","last_change":"2022-04-03T19:44:21.299112+0000","last_active":"2022-04-03T19:44:21.302772+0000","last_peered":"2022-04-03T19:44:21.302772+0000","last_clean":"2022-04-03T19:44:21.302772+0000","last_became_active":"2022-04-03T19:44:21.298803+0000","last_became_peered":"2022-04-03T19:44:21.298803+0000","last_unstale":"2022-04-03T19:44:21.302772+0000","last_undegraded":"2022-04-03T19:44:21.302772+0000","last_fullsized":"2022-04-03T19:44:21.302772+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:29:43.577843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299160+0000","last_change":"2022-04-03T19:44:21.293638+0000","last_active":"2022-04-03T19:44:21.299160+0000","last_peered":"2022-04-03T19:44:21.299160+0000","last_clean":"2022-04-03T19:44:21.299160+0000","last_became_active":"2022-04-03T19:44:21.293359+0000","last_became_peered":"2022-04-03T19:44:21.293359+0000","last_unstale":"2022-04-03T19:44:21.299160+0000","last_undegraded":"2022-04-03T19:44:21.299160+0000","last_fullsized":"2022-04-03T19:44:21.299160+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:28:50.113792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.306107+0000","last_change":"2022-04-03T19:44:21.299916+0000","last_active":"2022-04-03T19:44:21.306107+0000","last_peered":"2022-04-03T19:44:21.306107+0000","last_clean":"2022-04-03T19:44:21.306107+0000","last_became_active":"2022-04-03T19:44:21.299578+0000","last_became_peered":"2022-04-03T19:44:21.299578+0000","last_unstale":"2022-04-03T19:44:21.306107+0000","last_undegraded":"2022-04-03T19:44:21.306107+0000","last_fullsized":"2022-04-03T19:44:21.306107+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:38:40.281496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.298833+0000","last_change":"2022-04-03T19:44:21.294003+0000","last_active":"2022-04-03T19:44:21.298833+0000","last_peered":"2022-04-03T19:44:21.298833+0000","last_clean":"2022-04-03T19:44:21.298833+0000","last_became_active":"2022-04-03T19:44:21.293640+0000","last_became_peered":"2022-04-03T19:44:21.293640+0000","last_unstale":"2022-04-03T19:44:21.298833+0000","last_undegraded":"2022-04-03T19:44:21.298833+0000","last_fullsized":"2022-04-03T19:44:21.298833+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:40:13.436484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302745+0000","last_change":"2022-04-03T19:44:21.301158+0000","last_active":"2022-04-03T19:44:21.302745+0000","last_peered":"2022-04-03T19:44:21.302745+0000","last_clean":"2022-04-03T19:44:21.302745+0000","last_became_active":"2022-04-03T19:44:21.300823+0000","last_became_peered":"2022-04-03T19:44:21.300823+0000","last_unstale":"2022-04-03T19:44:21.302745+0000","last_undegraded":"2022-04-03T19:44:21.302745+0000","last_fullsized":"2022-04-03T19:44:21.302745+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:09:28.365828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299985+0000","last_change":"2022-04-03T19:44:21.295301+0000","last_active":"2022-04-03T19:44:21.299985+0000","last_peered":"2022-04-03T19:44:21.299985+0000","last_clean":"2022-04-03T19:44:21.299985+0000","last_became_active":"2022-04-03T19:44:21.295032+0000","last_became_peered":"2022-04-03T19:44:21.295032+0000","last_unstale":"2022-04-03T19:44:21.299985+0000","last_undegraded":"2022-04-03T19:44:21.299985+0000","last_fullsized":"2022-04-03T19:44:21.299985+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:55:55.637194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.301541+0000","last_change":"2022-04-03T19:44:21.296250+0000","last_active":"2022-04-03T19:44:21.301541+0000","last_peered":"2022-04-03T19:44:21.301541+0000","last_clean":"2022-04-03T19:44:21.301541+0000","last_became_active":"2022-04-03T19:44:21.295967+0000","last_became_peered":"2022-04-03T19:44:21.295967+0000","last_unstale":"2022-04-03T19:44:21.301541+0000","last_undegraded":"2022-04-03T19:44:21.301541+0000","last_fullsized":"2022-04-03T19:44:21.301541+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:43:34.301454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299466+0000","last_change":"2022-04-03T19:44:21.297229+0000","last_active":"2022-04-03T19:44:21.299466+0000","last_peered":"2022-04-03T19:44:21.299466+0000","last_clean":"2022-04-03T19:44:21.299466+0000","last_became_active":"2022-04-03T19:44:21.296993+0000","last_became_peered":"2022-04-03T19:44:21.296993+0000","last_unstale":"2022-04-03T19:44:21.299466+0000","last_undegraded":"2022-04-03T19:44:21.299466+0000","last_fullsized":"2022-04-03T19:44:21.299466+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:27:17.280580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303252+0000","last_change":"2022-04-03T19:44:21.302427+0000","last_active":"2022-04-03T19:44:21.303252+0000","last_peered":"2022-04-03T19:44:21.303252+0000","last_clean":"2022-04-03T19:44:21.303252+0000","last_became_active":"2022-04-03T19:44:21.302157+0000","last_became_peered":"2022-04-03T19:44:21.302157+0000","last_unstale":"2022-04-03T19:44:21.303252+0000","last_undegraded":"2022-04-03T19:44:21.303252+0000","last_fullsized":"2022-04-03T19:44:21.303252+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:57:23.292801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.298518+0000","last_change":"2022-04-03T19:44:21.293096+0000","last_active":"2022-04-03T19:44:21.298518+0000","last_peered":"2022-04-03T19:44:21.298518+0000","last_clean":"2022-04-03T19:44:21.298518+0000","last_became_active":"2022-04-03T19:44:21.292785+0000","last_became_peered":"2022-04-03T19:44:21.292785+0000","last_unstale":"2022-04-03T19:44:21.298518+0000","last_undegraded":"2022-04-03T19:44:21.298518+0000","last_fullsized":"2022-04-03T19:44:21.298518+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:27:06.759308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.306127+0000","last_change":"2022-04-03T19:44:21.300957+0000","last_active":"2022-04-03T19:44:21.306127+0000","last_peered":"2022-04-03T19:44:21.306127+0000","last_clean":"2022-04-03T19:44:21.306127+0000","last_became_active":"2022-04-03T19:44:21.300342+0000","last_became_peered":"2022-04-03T19:44:21.300342+0000","last_unstale":"2022-04-03T19:44:21.306127+0000","last_undegraded":"2022-04-03T19:44:21.306127+0000","last_fullsized":"2022-04-03T19:44:21.306127+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:43:53.194230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302142+0000","last_change":"2022-04-03T19:44:21.295996+0000","last_active":"2022-04-03T19:44:21.302142+0000","last_peered":"2022-04-03T19:44:21.302142+0000","last_clean":"2022-04-03T19:44:21.302142+0000","last_became_active":"2022-04-03T19:44:21.295766+0000","last_became_peered":"2022-04-03T19:44:21.295766+0000","last_unstale":"2022-04-03T19:44:21.302142+0000","last_undegraded":"2022-04-03T19:44:21.302142+0000","last_fullsized":"2022-04-03T19:44:21.302142+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:21:31.628478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302310+0000","last_change":"2022-04-03T19:44:21.299840+0000","last_active":"2022-04-03T19:44:21.302310+0000","last_peered":"2022-04-03T19:44:21.302310+0000","last_clean":"2022-04-03T19:44:21.302310+0000","last_became_active":"2022-04-03T19:44:21.299487+0000","last_became_peered":"2022-04-03T19:44:21.299487+0000","last_unstale":"2022-04-03T19:44:21.302310+0000","last_undegraded":"2022-04-03T19:44:21.302310+0000","last_fullsized":"2022-04-03T19:44:21.302310+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:24:39.583019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.295324+0000","last_change":"2022-04-03T19:44:21.291748+0000","last_active":"2022-04-03T19:44:21.295324+0000","last_peered":"2022-04-03T19:44:21.295324+0000","last_clean":"2022-04-03T19:44:21.295324+0000","last_became_active":"2022-04-03T19:44:21.290939+0000","last_became_peered":"2022-04-03T19:44:21.290939+0000","last_unstale":"2022-04-03T19:44:21.295324+0000","last_undegraded":"2022-04-03T19:44:21.295324+0000","last_fullsized":"2022-04-03T19:44:21.295324+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:28:45.710805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302521+0000","last_change":"2022-04-03T19:44:21.299442+0000","last_active":"2022-04-03T19:44:21.302521+0000","last_peered":"2022-04-03T19:44:21.302521+0000","last_clean":"2022-04-03T19:44:21.302521+0000","last_became_active":"2022-04-03T19:44:21.299166+0000","last_became_peered":"2022-04-03T19:44:21.299166+0000","last_unstale":"2022-04-03T19:44:21.302521+0000","last_undegraded":"2022-04-03T19:44:21.302521+0000","last_fullsized":"2022-04-03T19:44:21.302521+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:31:30.905517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.311290+0000","last_change":"2022-04-03T19:44:21.307007+0000","last_active":"2022-04-03T19:44:21.311290+0000","last_peered":"2022-04-03T19:44:21.311290+0000","last_clean":"2022-04-03T19:44:21.311290+0000","last_became_active":"2022-04-03T19:44:21.306650+0000","last_became_peered":"2022-04-03T19:44:21.306650+0000","last_unstale":"2022-04-03T19:44:21.311290+0000","last_undegraded":"2022-04-03T19:44:21.311290+0000","last_fullsized":"2022-04-03T19:44:21.311290+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:03:26.043624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299849+0000","last_change":"2022-04-03T19:44:21.293566+0000","last_active":"2022-04-03T19:44:21.299849+0000","last_peered":"2022-04-03T19:44:21.299849+0000","last_clean":"2022-04-03T19:44:21.299849+0000","last_became_active":"2022-04-03T19:44:21.293263+0000","last_became_peered":"2022-04-03T19:44:21.293263+0000","last_unstale":"2022-04-03T19:44:21.299849+0000","last_undegraded":"2022-04-03T19:44:21.299849+0000","last_fullsized":"2022-04-03T19:44:21.299849+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:39:52.523944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302610+0000","last_change":"2022-04-03T19:44:21.300329+0000","last_active":"2022-04-03T19:44:21.302610+0000","last_peered":"2022-04-03T19:44:21.302610+0000","last_clean":"2022-04-03T19:44:21.302610+0000","last_became_active":"2022-04-03T19:44:21.300000+0000","last_became_peered":"2022-04-03T19:44:21.300000+0000","last_unstale":"2022-04-03T19:44:21.302610+0000","last_undegraded":"2022-04-03T19:44:21.302610+0000","last_fullsized":"2022-04-03T19:44:21.302610+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:15:10.839571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.312922+0000","last_change":"2022-04-03T19:44:21.302506+0000","last_active":"2022-04-03T19:44:21.312922+0000","last_peered":"2022-04-03T19:44:21.312922+0000","last_clean":"2022-04-03T19:44:21.312922+0000","last_became_active":"2022-04-03T19:44:21.302235+0000","last_became_peered":"2022-04-03T19:44:21.302235+0000","last_unstale":"2022-04-03T19:44:21.312922+0000","last_undegraded":"2022-04-03T19:44:21.312922+0000","last_fullsized":"2022-04-03T19:44:21.312922+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:25:25.801754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313258+0000","last_change":"2022-04-03T19:44:21.306670+0000","last_active":"2022-04-03T19:44:21.313258+0000","last_peered":"2022-04-03T19:44:21.313258+0000","last_clean":"2022-04-03T19:44:21.313258+0000","last_became_active":"2022-04-03T19:44:21.305722+0000","last_became_peered":"2022-04-03T19:44:21.305722+0000","last_unstale":"2022-04-03T19:44:21.313258+0000","last_undegraded":"2022-04-03T19:44:21.313258+0000","last_fullsized":"2022-04-03T19:44:21.313258+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:44:09.515207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303081+0000","last_change":"2022-04-03T19:44:21.300024+0000","last_active":"2022-04-03T19:44:21.303081+0000","last_peered":"2022-04-03T19:44:21.303081+0000","last_clean":"2022-04-03T19:44:21.303081+0000","last_became_active":"2022-04-03T19:44:21.299753+0000","last_became_peered":"2022-04-03T19:44:21.299753+0000","last_unstale":"2022-04-03T19:44:21.303081+0000","last_undegraded":"2022-04-03T19:44:21.303081+0000","last_fullsized":"2022-04-03T19:44:21.303081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:15:31.808623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.390304+0000","last_change":"2022-04-03T19:44:20.783898+0000","last_active":"2022-04-03T19:44:22.390304+0000","last_peered":"2022-04-03T19:44:22.390304+0000","last_clean":"2022-04-03T19:44:22.390304+0000","last_became_active":"2022-04-03T19:44:20.783612+0000","last_became_peered":"2022-04-03T19:44:20.783612+0000","last_unstale":"2022-04-03T19:44:22.390304+0000","last_undegraded":"2022-04-03T19:44:22.390304+0000","last_fullsized":"2022-04-03T19:44:22.390304+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:13:32.863927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.310259+0000","last_change":"2022-04-03T19:44:21.306904+0000","last_active":"2022-04-03T19:44:21.310259+0000","last_peered":"2022-04-03T19:44:21.310259+0000","last_clean":"2022-04-03T19:44:21.310259+0000","last_became_active":"2022-04-03T19:44:21.306616+0000","last_became_peered":"2022-04-03T19:44:21.306616+0000","last_unstale":"2022-04-03T19:44:21.310259+0000","last_undegraded":"2022-04-03T19:44:21.310259+0000","last_fullsized":"2022-04-03T19:44:21.310259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:49:35.783912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.296341+0000","last_change":"2022-04-03T19:44:20.788125+0000","last_active":"2022-04-03T19:44:21.296341+0000","last_peered":"2022-04-03T19:44:21.296341+0000","last_clean":"2022-04-03T19:44:21.296341+0000","last_became_active":"2022-04-03T19:44:20.787792+0000","last_became_peered":"2022-04-03T19:44:20.787792+0000","last_unstale":"2022-04-03T19:44:21.296341+0000","last_undegraded":"2022-04-03T19:44:21.296341+0000","last_fullsized":"2022-04-03T19:44:21.296341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:11:51.126774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305572+0000","last_change":"2022-04-03T19:44:21.302904+0000","last_active":"2022-04-03T19:44:21.305572+0000","last_peered":"2022-04-03T19:44:21.305572+0000","last_clean":"2022-04-03T19:44:21.305572+0000","last_became_active":"2022-04-03T19:44:21.301564+0000","last_became_peered":"2022-04-03T19:44:21.301564+0000","last_unstale":"2022-04-03T19:44:21.305572+0000","last_undegraded":"2022-04-03T19:44:21.305572+0000","last_fullsized":"2022-04-03T19:44:21.305572+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:25:51.611789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.283649+0000","last_change":"2022-04-03T19:44:20.792529+0000","last_active":"2022-04-03T19:44:21.283649+0000","last_peered":"2022-04-03T19:44:21.283649+0000","last_clean":"2022-04-03T19:44:21.283649+0000","last_became_active":"2022-04-03T19:44:20.792014+0000","last_became_peered":"2022-04-03T19:44:20.792014+0000","last_unstale":"2022-04-03T19:44:21.283649+0000","last_undegraded":"2022-04-03T19:44:21.283649+0000","last_fullsized":"2022-04-03T19:44:21.283649+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:01:38.319693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302637+0000","last_change":"2022-04-03T19:44:21.299302+0000","last_active":"2022-04-03T19:44:21.302637+0000","last_peered":"2022-04-03T19:44:21.302637+0000","last_clean":"2022-04-03T19:44:21.302637+0000","last_became_active":"2022-04-03T19:44:21.299007+0000","last_became_peered":"2022-04-03T19:44:21.299007+0000","last_unstale":"2022-04-03T19:44:21.302637+0000","last_undegraded":"2022-04-03T19:44:21.302637+0000","last_fullsized":"2022-04-03T19:44:21.302637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:59:48.767335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.282232+0000","last_change":"2022-04-03T19:44:20.790672+0000","last_active":"2022-04-03T19:44:21.282232+0000","last_peered":"2022-04-03T19:44:21.282232+0000","last_clean":"2022-04-03T19:44:21.282232+0000","last_became_active":"2022-04-03T19:44:20.788526+0000","last_became_peered":"2022-04-03T19:44:20.788526+0000","last_unstale":"2022-04-03T19:44:21.282232+0000","last_undegraded":"2022-04-03T19:44:21.282232+0000","last_fullsized":"2022-04-03T19:44:21.282232+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:14:50.058917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.309960+0000","last_change":"2022-04-03T19:44:21.306472+0000","last_active":"2022-04-03T19:44:21.309960+0000","last_peered":"2022-04-03T19:44:21.309960+0000","last_clean":"2022-04-03T19:44:21.309960+0000","last_became_active":"2022-04-03T19:44:21.306071+0000","last_became_peered":"2022-04-03T19:44:21.306071+0000","last_unstale":"2022-04-03T19:44:21.309960+0000","last_undegraded":"2022-04-03T19:44:21.309960+0000","last_fullsized":"2022-04-03T19:44:21.309960+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:43:22.221014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286837+0000","last_change":"2022-04-03T19:44:20.785638+0000","last_active":"2022-04-03T19:44:21.286837+0000","last_peered":"2022-04-03T19:44:21.286837+0000","last_clean":"2022-04-03T19:44:21.286837+0000","last_became_active":"2022-04-03T19:44:20.784963+0000","last_became_peered":"2022-04-03T19:44:20.784963+0000","last_unstale":"2022-04-03T19:44:21.286837+0000","last_undegraded":"2022-04-03T19:44:21.286837+0000","last_fullsized":"2022-04-03T19:44:21.286837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:39:26.933298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.309328+0000","last_change":"2022-04-03T19:44:21.299583+0000","last_active":"2022-04-03T19:44:21.309328+0000","last_peered":"2022-04-03T19:44:21.309328+0000","last_clean":"2022-04-03T19:44:21.309328+0000","last_became_active":"2022-04-03T19:44:21.299301+0000","last_became_peered":"2022-04-03T19:44:21.299301+0000","last_unstale":"2022-04-03T19:44:21.309328+0000","last_undegraded":"2022-04-03T19:44:21.309328+0000","last_fullsized":"2022-04-03T19:44:21.309328+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:37:49.567741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"24'3","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.290849+0000","last_change":"2022-04-03T19:44:20.795890+0000","last_active":"2022-04-03T19:44:26.290849+0000","last_peered":"2022-04-03T19:44:26.290849+0000","last_clean":"2022-04-03T19:44:26.290849+0000","last_became_active":"2022-04-03T19:44:20.795611+0000","last_became_peered":"2022-04-03T19:44:20.795611+0000","last_unstale":"2022-04-03T19:44:26.290849+0000","last_undegraded":"2022-04-03T19:44:26.290849+0000","last_fullsized":"2022-04-03T19:44:26.290849+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:08:33.216934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313173+0000","last_change":"2022-04-03T19:44:21.304724+0000","last_active":"2022-04-03T19:44:21.313173+0000","last_peered":"2022-04-03T19:44:21.313173+0000","last_clean":"2022-04-03T19:44:21.313173+0000","last_became_active":"2022-04-03T19:44:21.302708+0000","last_became_peered":"2022-04-03T19:44:21.302708+0000","last_unstale":"2022-04-03T19:44:21.313173+0000","last_undegraded":"2022-04-03T19:44:21.313173+0000","last_fullsized":"2022-04-03T19:44:21.313173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:04:54.970100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"24'6","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.292663+0000","last_change":"2022-04-03T19:44:20.785659+0000","last_active":"2022-04-03T19:44:26.292663+0000","last_peered":"2022-04-03T19:44:26.292663+0000","last_clean":"2022-04-03T19:44:26.292663+0000","last_became_active":"2022-04-03T19:44:20.785308+0000","last_became_peered":"2022-04-03T19:44:20.785308+0000","last_unstale":"2022-04-03T19:44:26.292663+0000","last_undegraded":"2022-04-03T19:44:26.292663+0000","last_fullsized":"2022-04-03T19:44:26.292663+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:52:08.268532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293514+0000","last_change":"2022-04-03T19:44:21.291033+0000","last_active":"2022-04-03T19:44:21.293514+0000","last_peered":"2022-04-03T19:44:21.293514+0000","last_clean":"2022-04-03T19:44:21.293514+0000","last_became_active":"2022-04-03T19:44:21.290713+0000","last_became_peered":"2022-04-03T19:44:21.290713+0000","last_unstale":"2022-04-03T19:44:21.293514+0000","last_undegraded":"2022-04-03T19:44:21.293514+0000","last_fullsized":"2022-04-03T19:44:21.293514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:19:41.873591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.290200+0000","last_change":"2022-04-03T19:44:20.787511+0000","last_active":"2022-04-03T19:44:26.290200+0000","last_peered":"2022-04-03T19:44:26.290200+0000","last_clean":"2022-04-03T19:44:26.290200+0000","last_became_active":"2022-04-03T19:44:20.787189+0000","last_became_peered":"2022-04-03T19:44:20.787189+0000","last_unstale":"2022-04-03T19:44:26.290200+0000","last_undegraded":"2022-04-03T19:44:26.290200+0000","last_fullsized":"2022-04-03T19:44:26.290200+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:11:56.297263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313518+0000","last_change":"2022-04-03T19:44:21.307846+0000","last_active":"2022-04-03T19:44:21.313518+0000","last_peered":"2022-04-03T19:44:21.313518+0000","last_clean":"2022-04-03T19:44:21.313518+0000","last_became_active":"2022-04-03T19:44:21.307513+0000","last_became_peered":"2022-04-03T19:44:21.307513+0000","last_unstale":"2022-04-03T19:44:21.313518+0000","last_undegraded":"2022-04-03T19:44:21.313518+0000","last_fullsized":"2022-04-03T19:44:21.313518+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:58:09.091062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.281910+0000","last_change":"2022-04-03T19:44:20.790212+0000","last_active":"2022-04-03T19:44:21.281910+0000","last_peered":"2022-04-03T19:44:21.281910+0000","last_clean":"2022-04-03T19:44:21.281910+0000","last_became_active":"2022-04-03T19:44:20.789863+0000","last_became_peered":"2022-04-03T19:44:20.789863+0000","last_unstale":"2022-04-03T19:44:21.281910+0000","last_undegraded":"2022-04-03T19:44:21.281910+0000","last_fullsized":"2022-04-03T19:44:21.281910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:48:13.489309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303061+0000","last_change":"2022-04-03T19:44:21.299953+0000","last_active":"2022-04-03T19:44:21.303061+0000","last_peered":"2022-04-03T19:44:21.303061+0000","last_clean":"2022-04-03T19:44:21.303061+0000","last_became_active":"2022-04-03T19:44:21.299667+0000","last_became_peered":"2022-04-03T19:44:21.299667+0000","last_unstale":"2022-04-03T19:44:21.303061+0000","last_undegraded":"2022-04-03T19:44:21.303061+0000","last_fullsized":"2022-04-03T19:44:21.303061+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:51:16.125160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286253+0000","last_change":"2022-04-03T19:44:20.791054+0000","last_active":"2022-04-03T19:44:21.286253+0000","last_peered":"2022-04-03T19:44:21.286253+0000","last_clean":"2022-04-03T19:44:21.286253+0000","last_became_active":"2022-04-03T19:44:20.790664+0000","last_became_peered":"2022-04-03T19:44:20.790664+0000","last_unstale":"2022-04-03T19:44:21.286253+0000","last_undegraded":"2022-04-03T19:44:21.286253+0000","last_fullsized":"2022-04-03T19:44:21.286253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:07:27.501728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.304407+0000","last_change":"2022-04-03T19:44:21.295028+0000","last_active":"2022-04-03T19:44:21.304407+0000","last_peered":"2022-04-03T19:44:21.304407+0000","last_clean":"2022-04-03T19:44:21.304407+0000","last_became_active":"2022-04-03T19:44:21.294723+0000","last_became_peered":"2022-04-03T19:44:21.294723+0000","last_unstale":"2022-04-03T19:44:21.304407+0000","last_undegraded":"2022-04-03T19:44:21.304407+0000","last_fullsized":"2022-04-03T19:44:21.304407+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:32:45.455842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"24'8","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.294387+0000","last_change":"2022-04-03T19:44:20.792133+0000","last_active":"2022-04-03T19:44:26.294387+0000","last_peered":"2022-04-03T19:44:26.294387+0000","last_clean":"2022-04-03T19:44:26.294387+0000","last_became_active":"2022-04-03T19:44:20.791829+0000","last_became_peered":"2022-04-03T19:44:20.791829+0000","last_unstale":"2022-04-03T19:44:26.294387+0000","last_undegraded":"2022-04-03T19:44:26.294387+0000","last_fullsized":"2022-04-03T19:44:26.294387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:01:21.966734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.310014+0000","last_change":"2022-04-03T19:44:21.300267+0000","last_active":"2022-04-03T19:44:21.310014+0000","last_peered":"2022-04-03T19:44:21.310014+0000","last_clean":"2022-04-03T19:44:21.310014+0000","last_became_active":"2022-04-03T19:44:21.299948+0000","last_became_peered":"2022-04-03T19:44:21.299948+0000","last_unstale":"2022-04-03T19:44:21.310014+0000","last_undegraded":"2022-04-03T19:44:21.310014+0000","last_fullsized":"2022-04-03T19:44:21.310014+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:06:54.481379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.391229+0000","last_change":"2022-04-03T19:44:20.789793+0000","last_active":"2022-04-03T19:44:22.391229+0000","last_peered":"2022-04-03T19:44:22.391229+0000","last_clean":"2022-04-03T19:44:22.391229+0000","last_became_active":"2022-04-03T19:44:20.789371+0000","last_became_peered":"2022-04-03T19:44:20.789371+0000","last_unstale":"2022-04-03T19:44:22.391229+0000","last_undegraded":"2022-04-03T19:44:22.391229+0000","last_fullsized":"2022-04-03T19:44:22.391229+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:15:52.060059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.301800+0000","last_change":"2022-04-03T19:44:21.298477+0000","last_active":"2022-04-03T19:44:21.301800+0000","last_peered":"2022-04-03T19:44:21.301800+0000","last_clean":"2022-04-03T19:44:21.301800+0000","last_became_active":"2022-04-03T19:44:21.298165+0000","last_became_peered":"2022-04-03T19:44:21.298165+0000","last_unstale":"2022-04-03T19:44:21.301800+0000","last_undegraded":"2022-04-03T19:44:21.301800+0000","last_fullsized":"2022-04-03T19:44:21.301800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:23:11.042589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.290125+0000","last_change":"2022-04-03T19:44:20.787949+0000","last_active":"2022-04-03T19:44:26.290125+0000","last_peered":"2022-04-03T19:44:26.290125+0000","last_clean":"2022-04-03T19:44:26.290125+0000","last_became_active":"2022-04-03T19:44:20.787592+0000","last_became_peered":"2022-04-03T19:44:20.787592+0000","last_unstale":"2022-04-03T19:44:26.290125+0000","last_undegraded":"2022-04-03T19:44:26.290125+0000","last_fullsized":"2022-04-03T19:44:26.290125+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:50:09.411246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302765+0000","last_change":"2022-04-03T19:44:21.301200+0000","last_active":"2022-04-03T19:44:21.302765+0000","last_peered":"2022-04-03T19:44:21.302765+0000","last_clean":"2022-04-03T19:44:21.302765+0000","last_became_active":"2022-04-03T19:44:21.300878+0000","last_became_peered":"2022-04-03T19:44:21.300878+0000","last_unstale":"2022-04-03T19:44:21.302765+0000","last_undegraded":"2022-04-03T19:44:21.302765+0000","last_fullsized":"2022-04-03T19:44:21.302765+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:54:36.089411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.391128+0000","last_change":"2022-04-03T19:44:20.783711+0000","last_active":"2022-04-03T19:44:22.391128+0000","last_peered":"2022-04-03T19:44:22.391128+0000","last_clean":"2022-04-03T19:44:22.391128+0000","last_became_active":"2022-04-03T19:44:20.783423+0000","last_became_peered":"2022-04-03T19:44:20.783423+0000","last_unstale":"2022-04-03T19:44:22.391128+0000","last_undegraded":"2022-04-03T19:44:22.391128+0000","last_fullsized":"2022-04-03T19:44:22.391128+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:22:37.665229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303123+0000","last_change":"2022-04-03T19:44:21.300815+0000","last_active":"2022-04-03T19:44:21.303123+0000","last_peered":"2022-04-03T19:44:21.303123+0000","last_clean":"2022-04-03T19:44:21.303123+0000","last_became_active":"2022-04-03T19:44:21.300545+0000","last_became_peered":"2022-04-03T19:44:21.300545+0000","last_unstale":"2022-04-03T19:44:21.303123+0000","last_undegraded":"2022-04-03T19:44:21.303123+0000","last_fullsized":"2022-04-03T19:44:21.303123+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:33:13.941191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.285374+0000","last_change":"2022-04-03T19:44:20.785480+0000","last_active":"2022-04-03T19:44:21.285374+0000","last_peered":"2022-04-03T19:44:21.285374+0000","last_clean":"2022-04-03T19:44:21.285374+0000","last_became_active":"2022-04-03T19:44:20.785178+0000","last_became_peered":"2022-04-03T19:44:20.785178+0000","last_unstale":"2022-04-03T19:44:21.285374+0000","last_undegraded":"2022-04-03T19:44:21.285374+0000","last_fullsized":"2022-04-03T19:44:21.285374+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:15:37.181452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.308247+0000","last_change":"2022-04-03T19:44:21.296509+0000","last_active":"2022-04-03T19:44:21.308247+0000","last_peered":"2022-04-03T19:44:21.308247+0000","last_clean":"2022-04-03T19:44:21.308247+0000","last_became_active":"2022-04-03T19:44:21.296240+0000","last_became_peered":"2022-04-03T19:44:21.296240+0000","last_unstale":"2022-04-03T19:44:21.308247+0000","last_undegraded":"2022-04-03T19:44:21.308247+0000","last_fullsized":"2022-04-03T19:44:21.308247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:50:45.955195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.288396+0000","last_change":"2022-04-03T19:44:20.792550+0000","last_active":"2022-04-03T19:44:26.288396+0000","last_peered":"2022-04-03T19:44:26.288396+0000","last_clean":"2022-04-03T19:44:26.288396+0000","last_became_active":"2022-04-03T19:44:20.792191+0000","last_became_peered":"2022-04-03T19:44:20.792191+0000","last_unstale":"2022-04-03T19:44:26.288396+0000","last_undegraded":"2022-04-03T19:44:26.288396+0000","last_fullsized":"2022-04-03T19:44:26.288396+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:51:30.297728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299612+0000","last_change":"2022-04-03T19:44:21.294898+0000","last_active":"2022-04-03T19:44:21.299612+0000","last_peered":"2022-04-03T19:44:21.299612+0000","last_clean":"2022-04-03T19:44:21.299612+0000","last_became_active":"2022-04-03T19:44:21.294635+0000","last_became_peered":"2022-04-03T19:44:21.294635+0000","last_unstale":"2022-04-03T19:44:21.299612+0000","last_undegraded":"2022-04-03T19:44:21.299612+0000","last_fullsized":"2022-04-03T19:44:21.299612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:42:52.629552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.289106+0000","last_change":"2022-04-03T19:44:20.783711+0000","last_active":"2022-04-03T19:44:21.289106+0000","last_peered":"2022-04-03T19:44:21.289106+0000","last_clean":"2022-04-03T19:44:21.289106+0000","last_became_active":"2022-04-03T19:44:20.783426+0000","last_became_peered":"2022-04-03T19:44:20.783426+0000","last_unstale":"2022-04-03T19:44:21.289106+0000","last_undegraded":"2022-04-03T19:44:21.289106+0000","last_fullsized":"2022-04-03T19:44:21.289106+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:25:26.219558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302205+0000","last_change":"2022-04-03T19:44:21.296081+0000","last_active":"2022-04-03T19:44:21.302205+0000","last_peered":"2022-04-03T19:44:21.302205+0000","last_clean":"2022-04-03T19:44:21.302205+0000","last_became_active":"2022-04-03T19:44:21.295784+0000","last_became_peered":"2022-04-03T19:44:21.295784+0000","last_unstale":"2022-04-03T19:44:21.302205+0000","last_undegraded":"2022-04-03T19:44:21.302205+0000","last_fullsized":"2022-04-03T19:44:21.302205+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:23:40.568168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.283775+0000","last_change":"2022-04-03T19:44:20.789868+0000","last_active":"2022-04-03T19:44:21.283775+0000","last_peered":"2022-04-03T19:44:21.283775+0000","last_clean":"2022-04-03T19:44:21.283775+0000","last_became_active":"2022-04-03T19:44:20.789563+0000","last_became_peered":"2022-04-03T19:44:20.789563+0000","last_unstale":"2022-04-03T19:44:21.283775+0000","last_undegraded":"2022-04-03T19:44:21.283775+0000","last_fullsized":"2022-04-03T19:44:21.283775+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:04:22.814932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293542+0000","last_change":"2022-04-03T19:44:21.291212+0000","last_active":"2022-04-03T19:44:21.293542+0000","last_peered":"2022-04-03T19:44:21.293542+0000","last_clean":"2022-04-03T19:44:21.293542+0000","last_became_active":"2022-04-03T19:44:21.290888+0000","last_became_peered":"2022-04-03T19:44:21.290888+0000","last_unstale":"2022-04-03T19:44:21.293542+0000","last_undegraded":"2022-04-03T19:44:21.293542+0000","last_fullsized":"2022-04-03T19:44:21.293542+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:29:34.850679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.281344+0000","last_change":"2022-04-03T19:44:20.789319+0000","last_active":"2022-04-03T19:44:21.281344+0000","last_peered":"2022-04-03T19:44:21.281344+0000","last_clean":"2022-04-03T19:44:21.281344+0000","last_became_active":"2022-04-03T19:44:20.788840+0000","last_became_peered":"2022-04-03T19:44:20.788840+0000","last_unstale":"2022-04-03T19:44:21.281344+0000","last_undegraded":"2022-04-03T19:44:21.281344+0000","last_fullsized":"2022-04-03T19:44:21.281344+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:50:21.536926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313655+0000","last_change":"2022-04-03T19:44:21.311886+0000","last_active":"2022-04-03T19:44:21.313655+0000","last_peered":"2022-04-03T19:44:21.313655+0000","last_clean":"2022-04-03T19:44:21.313655+0000","last_became_active":"2022-04-03T19:44:21.311521+0000","last_became_peered":"2022-04-03T19:44:21.311521+0000","last_unstale":"2022-04-03T19:44:21.313655+0000","last_undegraded":"2022-04-03T19:44:21.313655+0000","last_fullsized":"2022-04-03T19:44:21.313655+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:25:47.761166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.297426+0000","last_change":"2022-04-03T19:44:14.754674+0000","last_active":"2022-04-03T19:44:21.297426+0000","last_peered":"2022-04-03T19:44:21.297426+0000","last_clean":"2022-04-03T19:44:21.297426+0000","last_became_active":"2022-04-03T19:44:14.754210+0000","last_became_peered":"2022-04-03T19:44:14.754210+0000","last_unstale":"2022-04-03T19:44:21.297426+0000","last_undegraded":"2022-04-03T19:44:21.297426+0000","last_fullsized":"2022-04-03T19:44:21.297426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:23:14.624494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286308+0000","last_change":"2022-04-03T19:44:20.785400+0000","last_active":"2022-04-03T19:44:21.286308+0000","last_peered":"2022-04-03T19:44:21.286308+0000","last_clean":"2022-04-03T19:44:21.286308+0000","last_became_active":"2022-04-03T19:44:20.785125+0000","last_became_peered":"2022-04-03T19:44:20.785125+0000","last_unstale":"2022-04-03T19:44:21.286308+0000","last_undegraded":"2022-04-03T19:44:21.286308+0000","last_fullsized":"2022-04-03T19:44:21.286308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:24:02.690646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313197+0000","last_change":"2022-04-03T19:44:21.305321+0000","last_active":"2022-04-03T19:44:21.313197+0000","last_peered":"2022-04-03T19:44:21.313197+0000","last_clean":"2022-04-03T19:44:21.313197+0000","last_became_active":"2022-04-03T19:44:21.304017+0000","last_became_peered":"2022-04-03T19:44:21.304017+0000","last_unstale":"2022-04-03T19:44:21.313197+0000","last_undegraded":"2022-04-03T19:44:21.313197+0000","last_fullsized":"2022-04-03T19:44:21.313197+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:59:12.619058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.285445+0000","last_change":"2022-04-03T19:44:14.755580+0000","last_active":"2022-04-03T19:44:21.285445+0000","last_peered":"2022-04-03T19:44:21.285445+0000","last_clean":"2022-04-03T19:44:21.285445+0000","last_became_active":"2022-04-03T19:44:14.755222+0000","last_became_peered":"2022-04-03T19:44:14.755222+0000","last_unstale":"2022-04-03T19:44:21.285445+0000","last_undegraded":"2022-04-03T19:44:21.285445+0000","last_fullsized":"2022-04-03T19:44:21.285445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:08:18.314326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.283116+0000","last_change":"2022-04-03T19:44:20.791540+0000","last_active":"2022-04-03T19:44:21.283116+0000","last_peered":"2022-04-03T19:44:21.283116+0000","last_clean":"2022-04-03T19:44:21.283116+0000","last_became_active":"2022-04-03T19:44:20.791192+0000","last_became_peered":"2022-04-03T19:44:20.791192+0000","last_unstale":"2022-04-03T19:44:21.283116+0000","last_undegraded":"2022-04-03T19:44:21.283116+0000","last_fullsized":"2022-04-03T19:44:21.283116+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:58:59.674606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313218+0000","last_change":"2022-04-03T19:44:21.305930+0000","last_active":"2022-04-03T19:44:21.313218+0000","last_peered":"2022-04-03T19:44:21.313218+0000","last_clean":"2022-04-03T19:44:21.313218+0000","last_became_active":"2022-04-03T19:44:21.305659+0000","last_became_peered":"2022-04-03T19:44:21.305659+0000","last_unstale":"2022-04-03T19:44:21.313218+0000","last_undegraded":"2022-04-03T19:44:21.313218+0000","last_fullsized":"2022-04-03T19:44:21.313218+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:14:34.958569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.290570+0000","last_change":"2022-04-03T19:44:14.754598+0000","last_active":"2022-04-03T19:44:21.290570+0000","last_peered":"2022-04-03T19:44:21.290570+0000","last_clean":"2022-04-03T19:44:21.290570+0000","last_became_active":"2022-04-03T19:44:14.754287+0000","last_became_peered":"2022-04-03T19:44:14.754287+0000","last_unstale":"2022-04-03T19:44:21.290570+0000","last_undegraded":"2022-04-03T19:44:21.290570+0000","last_fullsized":"2022-04-03T19:44:21.290570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:32:16.232689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.296054+0000","last_change":"2022-04-03T19:44:20.789567+0000","last_active":"2022-04-03T19:44:21.296054+0000","last_peered":"2022-04-03T19:44:21.296054+0000","last_clean":"2022-04-03T19:44:21.296054+0000","last_became_active":"2022-04-03T19:44:20.788664+0000","last_became_peered":"2022-04-03T19:44:20.788664+0000","last_unstale":"2022-04-03T19:44:21.296054+0000","last_undegraded":"2022-04-03T19:44:21.296054+0000","last_fullsized":"2022-04-03T19:44:21.296054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:15:37.098494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.312944+0000","last_change":"2022-04-03T19:44:21.303079+0000","last_active":"2022-04-03T19:44:21.312944+0000","last_peered":"2022-04-03T19:44:21.312944+0000","last_clean":"2022-04-03T19:44:21.312944+0000","last_became_active":"2022-04-03T19:44:21.302014+0000","last_became_peered":"2022-04-03T19:44:21.302014+0000","last_unstale":"2022-04-03T19:44:21.312944+0000","last_undegraded":"2022-04-03T19:44:21.312944+0000","last_fullsized":"2022-04-03T19:44:21.312944+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:15:58.860701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.285641+0000","last_change":"2022-04-03T19:44:14.754385+0000","last_active":"2022-04-03T19:44:21.285641+0000","last_peered":"2022-04-03T19:44:21.285641+0000","last_clean":"2022-04-03T19:44:21.285641+0000","last_became_active":"2022-04-03T19:44:14.754003+0000","last_became_peered":"2022-04-03T19:44:14.754003+0000","last_unstale":"2022-04-03T19:44:21.285641+0000","last_undegraded":"2022-04-03T19:44:21.285641+0000","last_fullsized":"2022-04-03T19:44:21.285641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:05:22.716933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":166,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286290+0000","last_change":"2022-04-03T19:44:08.726758+0000","last_active":"2022-04-03T19:44:21.286290+0000","last_peered":"2022-04-03T19:44:21.286290+0000","last_clean":"2022-04-03T19:44:21.286290+0000","last_became_active":"2022-04-03T19:44:08.725694+0000","last_became_peered":"2022-04-03T19:44:08.725694+0000","last_unstale":"2022-04-03T19:44:21.286290+0000","last_undegraded":"2022-04-03T19:44:21.286290+0000","last_fullsized":"2022-04-03T19:44:21.286290+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:07.708129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:07.708129+0000","last_clean_scrub_stamp":"2022-04-03T19:44:07.708129+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:27:43.354267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.282879+0000","last_change":"2022-04-03T19:44:20.781585+0000","last_active":"2022-04-03T19:44:21.282879+0000","last_peered":"2022-04-03T19:44:21.282879+0000","last_clean":"2022-04-03T19:44:21.282879+0000","last_became_active":"2022-04-03T19:44:20.781274+0000","last_became_peered":"2022-04-03T19:44:20.781274+0000","last_unstale":"2022-04-03T19:44:21.282879+0000","last_undegraded":"2022-04-03T19:44:21.282879+0000","last_fullsized":"2022-04-03T19:44:21.282879+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:49:54.368001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.301298+0000","last_change":"2022-04-03T19:44:21.296620+0000","last_active":"2022-04-03T19:44:21.301298+0000","last_peered":"2022-04-03T19:44:21.301298+0000","last_clean":"2022-04-03T19:44:21.301298+0000","last_became_active":"2022-04-03T19:44:21.296314+0000","last_became_peered":"2022-04-03T19:44:21.296314+0000","last_unstale":"2022-04-03T19:44:21.301298+0000","last_undegraded":"2022-04-03T19:44:21.301298+0000","last_fullsized":"2022-04-03T19:44:21.301298+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:09:18.338683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286626+0000","last_change":"2022-04-03T19:44:14.754664+0000","last_active":"2022-04-03T19:44:21.286626+0000","last_peered":"2022-04-03T19:44:21.286626+0000","last_clean":"2022-04-03T19:44:21.286626+0000","last_became_active":"2022-04-03T19:44:14.754328+0000","last_became_peered":"2022-04-03T19:44:14.754328+0000","last_unstale":"2022-04-03T19:44:21.286626+0000","last_undegraded":"2022-04-03T19:44:21.286626+0000","last_fullsized":"2022-04-03T19:44:21.286626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:39:58.022353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.281559+0000","last_change":"2022-04-03T19:44:20.787271+0000","last_active":"2022-04-03T19:44:21.281559+0000","last_peered":"2022-04-03T19:44:21.281559+0000","last_clean":"2022-04-03T19:44:21.281559+0000","last_became_active":"2022-04-03T19:44:20.786957+0000","last_became_peered":"2022-04-03T19:44:20.786957+0000","last_unstale":"2022-04-03T19:44:21.281559+0000","last_undegraded":"2022-04-03T19:44:21.281559+0000","last_fullsized":"2022-04-03T19:44:21.281559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:21:45.992744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.311227+0000","last_change":"2022-04-03T19:44:21.306857+0000","last_active":"2022-04-03T19:44:21.311227+0000","last_peered":"2022-04-03T19:44:21.311227+0000","last_clean":"2022-04-03T19:44:21.311227+0000","last_became_active":"2022-04-03T19:44:21.306521+0000","last_became_peered":"2022-04-03T19:44:21.306521+0000","last_unstale":"2022-04-03T19:44:21.311227+0000","last_undegraded":"2022-04-03T19:44:21.311227+0000","last_fullsized":"2022-04-03T19:44:21.311227+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:50:03.912834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.287006+0000","last_change":"2022-04-03T19:44:14.754657+0000","last_active":"2022-04-03T19:44:21.287006+0000","last_peered":"2022-04-03T19:44:21.287006+0000","last_clean":"2022-04-03T19:44:21.287006+0000","last_became_active":"2022-04-03T19:44:14.754310+0000","last_became_peered":"2022-04-03T19:44:14.754310+0000","last_unstale":"2022-04-03T19:44:21.287006+0000","last_undegraded":"2022-04-03T19:44:21.287006+0000","last_fullsized":"2022-04-03T19:44:21.287006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:42:05.279424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.297553+0000","last_change":"2022-04-03T19:44:20.794454+0000","last_active":"2022-04-03T19:44:21.297553+0000","last_peered":"2022-04-03T19:44:21.297553+0000","last_clean":"2022-04-03T19:44:21.297553+0000","last_became_active":"2022-04-03T19:44:20.789468+0000","last_became_peered":"2022-04-03T19:44:20.789468+0000","last_unstale":"2022-04-03T19:44:21.297553+0000","last_undegraded":"2022-04-03T19:44:21.297553+0000","last_fullsized":"2022-04-03T19:44:21.297553+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:02:49.184838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.308286+0000","last_change":"2022-04-03T19:44:21.303669+0000","last_active":"2022-04-03T19:44:21.308286+0000","last_peered":"2022-04-03T19:44:21.308286+0000","last_clean":"2022-04-03T19:44:21.308286+0000","last_became_active":"2022-04-03T19:44:21.303381+0000","last_became_peered":"2022-04-03T19:44:21.303381+0000","last_unstale":"2022-04-03T19:44:21.308286+0000","last_undegraded":"2022-04-03T19:44:21.308286+0000","last_fullsized":"2022-04-03T19:44:21.308286+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:43:22.320070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.289548+0000","last_change":"2022-04-03T19:44:20.791594+0000","last_active":"2022-04-03T19:44:26.289548+0000","last_peered":"2022-04-03T19:44:26.289548+0000","last_clean":"2022-04-03T19:44:26.289548+0000","last_became_active":"2022-04-03T19:44:20.789693+0000","last_became_peered":"2022-04-03T19:44:20.789693+0000","last_unstale":"2022-04-03T19:44:26.289548+0000","last_undegraded":"2022-04-03T19:44:26.289548+0000","last_fullsized":"2022-04-03T19:44:26.289548+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:18:21.832066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.297820+0000","last_change":"2022-04-03T19:44:14.755096+0000","last_active":"2022-04-03T19:44:21.297820+0000","last_peered":"2022-04-03T19:44:21.297820+0000","last_clean":"2022-04-03T19:44:21.297820+0000","last_became_active":"2022-04-03T19:44:14.754799+0000","last_became_peered":"2022-04-03T19:44:14.754799+0000","last_unstale":"2022-04-03T19:44:21.297820+0000","last_undegraded":"2022-04-03T19:44:21.297820+0000","last_fullsized":"2022-04-03T19:44:21.297820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:04:17.179114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.309672+0000","last_change":"2022-04-03T19:44:21.299649+0000","last_active":"2022-04-03T19:44:21.309672+0000","last_peered":"2022-04-03T19:44:21.309672+0000","last_clean":"2022-04-03T19:44:21.309672+0000","last_became_active":"2022-04-03T19:44:21.299399+0000","last_became_peered":"2022-04-03T19:44:21.299399+0000","last_unstale":"2022-04-03T19:44:21.309672+0000","last_undegraded":"2022-04-03T19:44:21.309672+0000","last_fullsized":"2022-04-03T19:44:21.309672+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:13:06.313729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.288717+0000","last_change":"2022-04-03T19:44:14.754659+0000","last_active":"2022-04-03T19:44:21.288717+0000","last_peered":"2022-04-03T19:44:21.288717+0000","last_clean":"2022-04-03T19:44:21.288717+0000","last_became_active":"2022-04-03T19:44:14.754386+0000","last_became_peered":"2022-04-03T19:44:14.754386+0000","last_unstale":"2022-04-03T19:44:21.288717+0000","last_undegraded":"2022-04-03T19:44:21.288717+0000","last_fullsized":"2022-04-03T19:44:21.288717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:33:33.013536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.283339+0000","last_change":"2022-04-03T19:44:20.784840+0000","last_active":"2022-04-03T19:44:21.283339+0000","last_peered":"2022-04-03T19:44:21.283339+0000","last_clean":"2022-04-03T19:44:21.283339+0000","last_became_active":"2022-04-03T19:44:20.784521+0000","last_became_peered":"2022-04-03T19:44:20.784521+0000","last_unstale":"2022-04-03T19:44:21.283339+0000","last_undegraded":"2022-04-03T19:44:21.283339+0000","last_fullsized":"2022-04-03T19:44:21.283339+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:15:38.240804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293467+0000","last_change":"2022-04-03T19:44:21.290769+0000","last_active":"2022-04-03T19:44:21.293467+0000","last_peered":"2022-04-03T19:44:21.293467+0000","last_clean":"2022-04-03T19:44:21.293467+0000","last_became_active":"2022-04-03T19:44:21.290394+0000","last_became_peered":"2022-04-03T19:44:21.290394+0000","last_unstale":"2022-04-03T19:44:21.293467+0000","last_undegraded":"2022-04-03T19:44:21.293467+0000","last_fullsized":"2022-04-03T19:44:21.293467+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:57:39.528871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.285708+0000","last_change":"2022-04-03T19:44:20.789662+0000","last_active":"2022-04-03T19:44:22.285708+0000","last_peered":"2022-04-03T19:44:22.285708+0000","last_clean":"2022-04-03T19:44:22.285708+0000","last_became_active":"2022-04-03T19:44:20.787868+0000","last_became_peered":"2022-04-03T19:44:20.787868+0000","last_unstale":"2022-04-03T19:44:22.285708+0000","last_undegraded":"2022-04-03T19:44:22.285708+0000","last_fullsized":"2022-04-03T19:44:22.285708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:13:16.092180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305589+0000","last_change":"2022-04-03T19:44:21.304539+0000","last_active":"2022-04-03T19:44:21.305589+0000","last_peered":"2022-04-03T19:44:21.305589+0000","last_clean":"2022-04-03T19:44:21.305589+0000","last_became_active":"2022-04-03T19:44:21.304008+0000","last_became_peered":"2022-04-03T19:44:21.304008+0000","last_unstale":"2022-04-03T19:44:21.305589+0000","last_undegraded":"2022-04-03T19:44:21.305589+0000","last_fullsized":"2022-04-03T19:44:21.305589+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:45:01.346479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.292841+0000","last_change":"2022-04-03T19:44:20.786813+0000","last_active":"2022-04-03T19:44:26.292841+0000","last_peered":"2022-04-03T19:44:26.292841+0000","last_clean":"2022-04-03T19:44:26.292841+0000","last_became_active":"2022-04-03T19:44:20.786548+0000","last_became_peered":"2022-04-03T19:44:20.786548+0000","last_unstale":"2022-04-03T19:44:26.292841+0000","last_undegraded":"2022-04-03T19:44:26.292841+0000","last_fullsized":"2022-04-03T19:44:26.292841+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:36:40.611720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302733+0000","last_change":"2022-04-03T19:44:21.299513+0000","last_active":"2022-04-03T19:44:21.302733+0000","last_peered":"2022-04-03T19:44:21.302733+0000","last_clean":"2022-04-03T19:44:21.302733+0000","last_became_active":"2022-04-03T19:44:21.299270+0000","last_became_peered":"2022-04-03T19:44:21.299270+0000","last_unstale":"2022-04-03T19:44:21.302733+0000","last_undegraded":"2022-04-03T19:44:21.302733+0000","last_fullsized":"2022-04-03T19:44:21.302733+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:54:16.302815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.290239+0000","last_change":"2022-04-03T19:44:20.786750+0000","last_active":"2022-04-03T19:44:26.290239+0000","last_peered":"2022-04-03T19:44:26.290239+0000","last_clean":"2022-04-03T19:44:26.290239+0000","last_became_active":"2022-04-03T19:44:20.786490+0000","last_became_peered":"2022-04-03T19:44:20.786490+0000","last_unstale":"2022-04-03T19:44:26.290239+0000","last_undegraded":"2022-04-03T19:44:26.290239+0000","last_fullsized":"2022-04-03T19:44:26.290239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:30:31.980806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300228+0000","last_change":"2022-04-03T19:44:21.294849+0000","last_active":"2022-04-03T19:44:21.300228+0000","last_peered":"2022-04-03T19:44:21.300228+0000","last_clean":"2022-04-03T19:44:21.300228+0000","last_became_active":"2022-04-03T19:44:21.294586+0000","last_became_peered":"2022-04-03T19:44:21.294586+0000","last_unstale":"2022-04-03T19:44:21.300228+0000","last_undegraded":"2022-04-03T19:44:21.300228+0000","last_fullsized":"2022-04-03T19:44:21.300228+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:24:11.050193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.289167+0000","last_change":"2022-04-03T19:44:20.781619+0000","last_active":"2022-04-03T19:44:22.289167+0000","last_peered":"2022-04-03T19:44:22.289167+0000","last_clean":"2022-04-03T19:44:22.289167+0000","last_became_active":"2022-04-03T19:44:20.781288+0000","last_became_peered":"2022-04-03T19:44:20.781288+0000","last_unstale":"2022-04-03T19:44:22.289167+0000","last_undegraded":"2022-04-03T19:44:22.289167+0000","last_fullsized":"2022-04-03T19:44:22.289167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:05:56.179441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305553+0000","last_change":"2022-04-03T19:44:21.296332+0000","last_active":"2022-04-03T19:44:21.305553+0000","last_peered":"2022-04-03T19:44:21.305553+0000","last_clean":"2022-04-03T19:44:21.305553+0000","last_became_active":"2022-04-03T19:44:21.296026+0000","last_became_peered":"2022-04-03T19:44:21.296026+0000","last_unstale":"2022-04-03T19:44:21.305553+0000","last_undegraded":"2022-04-03T19:44:21.305553+0000","last_fullsized":"2022-04-03T19:44:21.305553+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:43:33.822187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"24'4","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.293148+0000","last_change":"2022-04-03T19:44:20.789448+0000","last_active":"2022-04-03T19:44:26.293148+0000","last_peered":"2022-04-03T19:44:26.293148+0000","last_clean":"2022-04-03T19:44:26.293148+0000","last_became_active":"2022-04-03T19:44:20.789131+0000","last_became_peered":"2022-04-03T19:44:20.789131+0000","last_unstale":"2022-04-03T19:44:26.293148+0000","last_undegraded":"2022-04-03T19:44:26.293148+0000","last_fullsized":"2022-04-03T19:44:26.293148+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:46:20.830676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293972+0000","last_change":"2022-04-03T19:44:21.291681+0000","last_active":"2022-04-03T19:44:21.293972+0000","last_peered":"2022-04-03T19:44:21.293972+0000","last_clean":"2022-04-03T19:44:21.293972+0000","last_became_active":"2022-04-03T19:44:21.290817+0000","last_became_peered":"2022-04-03T19:44:21.290817+0000","last_unstale":"2022-04-03T19:44:21.293972+0000","last_undegraded":"2022-04-03T19:44:21.293972+0000","last_fullsized":"2022-04-03T19:44:21.293972+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:37:30.833818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300290+0000","last_change":"2022-04-03T19:44:21.295604+0000","last_active":"2022-04-03T19:44:21.300290+0000","last_peered":"2022-04-03T19:44:21.300290+0000","last_clean":"2022-04-03T19:44:21.300290+0000","last_became_active":"2022-04-03T19:44:21.295297+0000","last_became_peered":"2022-04-03T19:44:21.295297+0000","last_unstale":"2022-04-03T19:44:21.300290+0000","last_undegraded":"2022-04-03T19:44:21.300290+0000","last_fullsized":"2022-04-03T19:44:21.300290+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:26:34.341290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.309922+0000","last_change":"2022-04-03T19:44:21.306547+0000","last_active":"2022-04-03T19:44:21.309922+0000","last_peered":"2022-04-03T19:44:21.309922+0000","last_clean":"2022-04-03T19:44:21.309922+0000","last_became_active":"2022-04-03T19:44:21.306179+0000","last_became_peered":"2022-04-03T19:44:21.306179+0000","last_unstale":"2022-04-03T19:44:21.309922+0000","last_undegraded":"2022-04-03T19:44:21.309922+0000","last_fullsized":"2022-04-03T19:44:21.309922+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:34:58.854671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293642+0000","last_change":"2022-04-03T19:44:21.291341+0000","last_active":"2022-04-03T19:44:21.293642+0000","last_peered":"2022-04-03T19:44:21.293642+0000","last_clean":"2022-04-03T19:44:21.293642+0000","last_became_active":"2022-04-03T19:44:21.290943+0000","last_became_peered":"2022-04-03T19:44:21.290943+0000","last_unstale":"2022-04-03T19:44:21.293642+0000","last_undegraded":"2022-04-03T19:44:21.293642+0000","last_fullsized":"2022-04-03T19:44:21.293642+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:50:56.877454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313543+0000","last_change":"2022-04-03T19:44:21.307815+0000","last_active":"2022-04-03T19:44:21.313543+0000","last_peered":"2022-04-03T19:44:21.313543+0000","last_clean":"2022-04-03T19:44:21.313543+0000","last_became_active":"2022-04-03T19:44:21.307459+0000","last_became_peered":"2022-04-03T19:44:21.307459+0000","last_unstale":"2022-04-03T19:44:21.313543+0000","last_undegraded":"2022-04-03T19:44:21.313543+0000","last_fullsized":"2022-04-03T19:44:21.313543+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:16:52.723409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.312899+0000","last_change":"2022-04-03T19:44:21.303095+0000","last_active":"2022-04-03T19:44:21.312899+0000","last_peered":"2022-04-03T19:44:21.312899+0000","last_clean":"2022-04-03T19:44:21.312899+0000","last_became_active":"2022-04-03T19:44:21.302363+0000","last_became_peered":"2022-04-03T19:44:21.302363+0000","last_unstale":"2022-04-03T19:44:21.312899+0000","last_undegraded":"2022-04-03T19:44:21.312899+0000","last_fullsized":"2022-04-03T19:44:21.312899+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:42:45.617661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8972,"kb_used_data":772,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94362868,"statfs":{"total":96636764160,"available":96627576832,"internally_reserved":0,"allocated":790528,"data_stored":623468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90700000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]}]},{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8996,"kb_used_data":796,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94362844,"statfs":{"total":96636764160,"available":96627552256,"internally_reserved":0,"allocated":815104,"data_stored":624666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7936,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363904,"statfs":{"total":96636764160,"available":96628637696,"internally_reserved":0,"allocated":253952,"data_stored":93974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8000,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363840,"statfs":{"total":96636764160,"available":96628572160,"internally_reserved":0,"allocated":253952,"data_stored":93974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8004,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363836,"statfs":{"total":96636764160,"available":96628568064,"internally_reserved":0,"allocated":258048,"data_stored":94064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8020,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363820,"statfs":{"total":96636764160,"available":96628551680,"internally_reserved":0,"allocated":274432,"data_stored":95578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7944,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363896,"statfs":{"total":96636764160,"available":96628629504,"internally_reserved":0,"allocated":262144,"data_stored":94610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7936,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363904,"statfs":{"total":96636764160,"available":96628637696,"internally_reserved":0,"allocated":253952,"data_stored":93974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-03T19:44:32.447 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-04-03T19:44:32.643 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.643+0000 7f5a08d7f700 1 Processor -- start 2022-04-03T19:44:32.644 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.643+0000 7f5a08d7f700 1 -- start start 2022-04-03T19:44:32.645 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.644+0000 7f5a08d7f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5a0406bbe0 0x7f5a041584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:32.645 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.644+0000 7f5a08d7f700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5a0406c500 0x7f5a0414eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:32.645 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.645+0000 7f5a08d7f700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0406d260 con 0x7f5a0406bfb0 2022-04-03T19:44:32.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.645+0000 7f5a08d7f700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a04159450 con 0x7f5a0406bbe0 2022-04-03T19:44:32.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.645+0000 7f5a08d7f700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a040c6330 con 0x7f5a0406c500 2022-04-03T19:44:32.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.645+0000 7f5a037fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5a0406bbe0 0x7f5a041584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:32.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.645+0000 7f5a03fff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5a0406c500 0x7f5a0414eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:32.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.645+0000 7f5a037fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5a0406bbe0 0x7f5a041584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42608/0 (socket says 172.21.15.77:42608) 2022-04-03T19:44:32.647 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.645+0000 7f5a037fe700 1 -- 172.21.15.77:0/3228452200 learned_addr learned my addr 172.21.15.77:0/3228452200 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:32.648 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.645+0000 7f5a037fe700 1 -- 172.21.15.77:0/3228452200 >> v2:172.21.15.186:3300/0 conn(0x7f5a0406c500 msgr2=0x7f5a0414eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:32.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.645+0000 7f5a037fe700 1 --2- 172.21.15.77:0/3228452200 >> v2:172.21.15.186:3300/0 conn(0x7f5a0406c500 0x7f5a0414eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:32.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.645+0000 7f5a037fe700 1 -- 172.21.15.77:0/3228452200 >> v1:172.21.15.186:6789/0 conn(0x7f5a0406bfb0 legacy=0x7f5a0406d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:32.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.645+0000 7f5a037fe700 1 -- 172.21.15.77:0/3228452200 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a04158bb0 con 0x7f5a0406bbe0 2022-04-03T19:44:32.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.646+0000 7f5a037fe700 1 --2- 172.21.15.77:0/3228452200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a0406bbe0 0x7f5a041584b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5a040cc6b0 tx=0x7f59f40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48b6b99b85cd8aa2 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:32.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.646+0000 7f5a027fc700 1 -- 172.21.15.77:0/3228452200 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4004030 con 0x7f5a0406bbe0 2022-04-03T19:44:32.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.646+0000 7f5a027fc700 1 -- 172.21.15.77:0/3228452200 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f4004190 con 0x7f5a0406bbe0 2022-04-03T19:44:32.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.646+0000 7f5a027fc700 1 -- 172.21.15.77:0/3228452200 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4011630 con 0x7f5a0406bbe0 2022-04-03T19:44:32.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.646+0000 7f5a08d7f700 1 -- 172.21.15.77:0/3228452200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a0406bbe0 msgr2=0x7f5a041584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:32.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.646+0000 7f5a08d7f700 1 --2- 172.21.15.77:0/3228452200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a0406bbe0 0x7f5a041584b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5a040cc6b0 tx=0x7f59f40090f0 comp rx=0 tx=0).stop 2022-04-03T19:44:32.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.647+0000 7f5a08d7f700 1 -- 172.21.15.77:0/3228452200 shutdown_connections 2022-04-03T19:44:32.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.647+0000 7f5a08d7f700 1 --2- 172.21.15.77:0/3228452200 >> v2:172.21.15.186:3300/0 conn(0x7f5a0406c500 0x7f5a0414eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:32.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.647+0000 7f5a08d7f700 1 --2- 172.21.15.77:0/3228452200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a0406bbe0 0x7f5a041584b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:32.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.647+0000 7f5a08d7f700 1 -- 172.21.15.77:0/3228452200 >> 172.21.15.77:0/3228452200 conn(0x7f5a040cb6b0 msgr2=0x7f5a040cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:32.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.647+0000 7f5a08d7f700 1 -- 172.21.15.77:0/3228452200 shutdown_connections 2022-04-03T19:44:32.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.647+0000 7f5a08d7f700 1 -- 172.21.15.77:0/3228452200 wait complete. 2022-04-03T19:44:32.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.648+0000 7f5a08d7f700 1 Processor -- start 2022-04-03T19:44:32.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.648+0000 7f5a08d7f700 1 -- start start 2022-04-03T19:44:32.654 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.649+0000 7f5a08d7f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a0406bbe0 0x7f5a04158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:32.654 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.649+0000 7f5a08d7f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5a0406bfb0 0x7f5a04154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:32.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.649+0000 7f5a08d7f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a0406c500 0x7f5a04150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:32.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.649+0000 7f5a08d7f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0414af90 con 0x7f5a0406bbe0 2022-04-03T19:44:32.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.649+0000 7f5a08d7f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a041502d0 con 0x7f5a0406c500 2022-04-03T19:44:32.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.649+0000 7f5a08d7f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a04166310 con 0x7f5a0406bfb0 2022-04-03T19:44:32.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.649+0000 7f5a037fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a0406bbe0 0x7f5a04158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:32.656 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.649+0000 7f5a037fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a0406bbe0 0x7f5a04158250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42610/0 (socket says 172.21.15.77:42610) 2022-04-03T19:44:32.656 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.650+0000 7f5a03fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a0406c500 0x7f5a04150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:32.656 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.650+0000 7f5a037fe700 1 -- 172.21.15.77:0/940993029 learned_addr learned my addr 172.21.15.77:0/940993029 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:32.656 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.650+0000 7f5a02ffd700 1 --2- 172.21.15.77:0/940993029 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5a0406bfb0 0x7f5a04154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:32.656 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.650+0000 7f5a037fe700 1 -- 172.21.15.77:0/940993029 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5a0406bfb0 msgr2=0x7f5a04154ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:32.657 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.650+0000 7f5a037fe700 1 --2- 172.21.15.77:0/940993029 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5a0406bfb0 0x7f5a04154ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:32.657 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.650+0000 7f5a037fe700 1 -- 172.21.15.77:0/940993029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a0406c500 msgr2=0x7f5a04150760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:32.657 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.650+0000 7f5a037fe700 1 --2- 172.21.15.77:0/940993029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a0406c500 0x7f5a04150760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:32.657 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.650+0000 7f5a037fe700 1 -- 172.21.15.77:0/940993029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f4009370 con 0x7f5a0406bbe0 2022-04-03T19:44:32.658 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.650+0000 7f5a037fe700 1 --2- 172.21.15.77:0/940993029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a0406bbe0 0x7f5a04158250 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5a040632c0 tx=0x7f59f4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:32.658 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.650+0000 7f59eb7fe700 1 -- 172.21.15.77:0/940993029 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f40036f0 con 0x7f5a0406bbe0 2022-04-03T19:44:32.658 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.651+0000 7f59eb7fe700 1 -- 172.21.15.77:0/940993029 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f4003850 con 0x7f5a0406bbe0 2022-04-03T19:44:32.658 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.651+0000 7f59eb7fe700 1 -- 172.21.15.77:0/940993029 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4003c10 con 0x7f5a0406bbe0 2022-04-03T19:44:32.658 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.651+0000 7f5a08d7f700 1 -- 172.21.15.77:0/940993029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0414f9e0 con 0x7f5a0406bbe0 2022-04-03T19:44:32.659 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.651+0000 7f5a08d7f700 1 -- 172.21.15.77:0/940993029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a0414fc80 con 0x7f5a0406bbe0 2022-04-03T19:44:32.659 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.652+0000 7f5a08d7f700 1 -- 172.21.15.77:0/940993029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a041505b0 con 0x7f5a0406bbe0 2022-04-03T19:44:32.659 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.652+0000 7f59eb7fe700 1 -- 172.21.15.77:0/940993029 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f59f4017020 con 0x7f5a0406bbe0 2022-04-03T19:44:32.659 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.653+0000 7f59eb7fe700 1 --2- 172.21.15.77:0/940993029 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f59e4063d10 0x7f59e40661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:32.659 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.653+0000 7f59eb7fe700 1 -- 172.21.15.77:0/940993029 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f59f40820c0 con 0x7f5a0406bbe0 2022-04-03T19:44:32.660 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.653+0000 7f5a02ffd700 1 --2- 172.21.15.77:0/940993029 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f59e4063d10 0x7f59e40661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:32.660 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.654+0000 7f5a02ffd700 1 --2- 172.21.15.77:0/940993029 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f59e4063d10 0x7f59e40661c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5a0414efd0 tx=0x7f59f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:32.660 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.657+0000 7f59eb7fe700 1 -- 172.21.15.77:0/940993029 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f59f4050bb0 con 0x7f5a0406bbe0 2022-04-03T19:44:32.851 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.851+0000 7f5a08d7f700 1 -- 172.21.15.77:0/940993029 --> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5a040c7ec0 con 0x7f59e4063d10 2022-04-03T19:44:32.860 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.860+0000 7f59eb7fe700 1 -- 172.21.15.77:0/940993029 <== mgr.4106 v2:172.21.15.186:6832/24948 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284624 (secure 0 0 0) 0x7f5a040c7ec0 con 0x7f59e4063d10 2022-04-03T19:44:32.864 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:32.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.864+0000 7f5a08d7f700 1 -- 172.21.15.77:0/940993029 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f59e4063d10 msgr2=0x7f59e40661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:32.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.864+0000 7f5a08d7f700 1 --2- 172.21.15.77:0/940993029 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f59e4063d10 0x7f59e40661c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5a0414efd0 tx=0x7f59f8009040 comp rx=0 tx=0).stop 2022-04-03T19:44:32.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.864+0000 7f5a08d7f700 1 -- 172.21.15.77:0/940993029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a0406bbe0 msgr2=0x7f5a04158250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:32.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.864+0000 7f5a08d7f700 1 --2- 172.21.15.77:0/940993029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a0406bbe0 0x7f5a04158250 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5a040632c0 tx=0x7f59f4016000 comp rx=0 tx=0).stop 2022-04-03T19:44:32.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.864+0000 7f5a08d7f700 1 -- 172.21.15.77:0/940993029 shutdown_connections 2022-04-03T19:44:32.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.864+0000 7f5a08d7f700 1 --2- 172.21.15.77:0/940993029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a0406c500 0x7f5a04150760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:32.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.864+0000 7f5a08d7f700 1 --2- 172.21.15.77:0/940993029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a0406bbe0 0x7f5a04158250 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:32.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.864+0000 7f5a08d7f700 1 --2- 172.21.15.77:0/940993029 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f59e4063d10 0x7f59e40661c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:32.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.864+0000 7f5a08d7f700 1 --2- 172.21.15.77:0/940993029 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5a0406bfb0 0x7f5a04154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:32.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.864+0000 7f5a08d7f700 1 -- 172.21.15.77:0/940993029 >> 172.21.15.77:0/940993029 conn(0x7f5a040cb6b0 msgr2=0x7f5a040cba80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:32.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.864+0000 7f5a08d7f700 1 -- 172.21.15.77:0/940993029 shutdown_connections 2022-04-03T19:44:32.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:32.864+0000 7f5a08d7f700 1 -- 172.21.15.77:0/940993029 wait complete. 2022-04-03T19:44:32.870 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2022-04-03T19:44:32.889 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2022-04-03T19:44:31.490856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592698,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":65808,"kb_used_data":3088,"kb_used_omap":0,"kb_used_meta":62656,"kb_avail":754908912,"statfs":{"total":773094113280,"available":773026725888,"internally_reserved":0,"allocated":3162112,"data_stored":1814308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64159744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002741"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302418+0000","last_change":"2022-04-03T19:44:21.300867+0000","last_active":"2022-04-03T19:44:21.302418+0000","last_peered":"2022-04-03T19:44:21.302418+0000","last_clean":"2022-04-03T19:44:21.302418+0000","last_became_active":"2022-04-03T19:44:21.300569+0000","last_became_peered":"2022-04-03T19:44:21.300569+0000","last_unstale":"2022-04-03T19:44:21.302418+0000","last_undegraded":"2022-04-03T19:44:21.302418+0000","last_fullsized":"2022-04-03T19:44:21.302418+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:10:32.827314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305911+0000","last_change":"2022-04-03T19:44:21.302410+0000","last_active":"2022-04-03T19:44:21.305911+0000","last_peered":"2022-04-03T19:44:21.305911+0000","last_clean":"2022-04-03T19:44:21.305911+0000","last_became_active":"2022-04-03T19:44:21.302142+0000","last_became_peered":"2022-04-03T19:44:21.302142+0000","last_unstale":"2022-04-03T19:44:21.305911+0000","last_undegraded":"2022-04-03T19:44:21.305911+0000","last_fullsized":"2022-04-03T19:44:21.305911+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:01:46.327996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.298511+0000","last_change":"2022-04-03T19:44:21.293080+0000","last_active":"2022-04-03T19:44:21.298511+0000","last_peered":"2022-04-03T19:44:21.298511+0000","last_clean":"2022-04-03T19:44:21.298511+0000","last_became_active":"2022-04-03T19:44:21.292762+0000","last_became_peered":"2022-04-03T19:44:21.292762+0000","last_unstale":"2022-04-03T19:44:21.298511+0000","last_undegraded":"2022-04-03T19:44:21.298511+0000","last_fullsized":"2022-04-03T19:44:21.298511+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:28:36.726593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300326+0000","last_change":"2022-04-03T19:44:21.294448+0000","last_active":"2022-04-03T19:44:21.300326+0000","last_peered":"2022-04-03T19:44:21.300326+0000","last_clean":"2022-04-03T19:44:21.300326+0000","last_became_active":"2022-04-03T19:44:21.294083+0000","last_became_peered":"2022-04-03T19:44:21.294083+0000","last_unstale":"2022-04-03T19:44:21.300326+0000","last_undegraded":"2022-04-03T19:44:21.300326+0000","last_fullsized":"2022-04-03T19:44:21.300326+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:57:02.374826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.308268+0000","last_change":"2022-04-03T19:44:21.299551+0000","last_active":"2022-04-03T19:44:21.308268+0000","last_peered":"2022-04-03T19:44:21.308268+0000","last_clean":"2022-04-03T19:44:21.308268+0000","last_became_active":"2022-04-03T19:44:21.299123+0000","last_became_peered":"2022-04-03T19:44:21.299123+0000","last_unstale":"2022-04-03T19:44:21.308268+0000","last_undegraded":"2022-04-03T19:44:21.308268+0000","last_fullsized":"2022-04-03T19:44:21.308268+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:08:24.911647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300209+0000","last_change":"2022-04-03T19:44:21.293129+0000","last_active":"2022-04-03T19:44:21.300209+0000","last_peered":"2022-04-03T19:44:21.300209+0000","last_clean":"2022-04-03T19:44:21.300209+0000","last_became_active":"2022-04-03T19:44:21.292843+0000","last_became_peered":"2022-04-03T19:44:21.292843+0000","last_unstale":"2022-04-03T19:44:21.300209+0000","last_undegraded":"2022-04-03T19:44:21.300209+0000","last_fullsized":"2022-04-03T19:44:21.300209+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:38:32.432609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302772+0000","last_change":"2022-04-03T19:44:21.299112+0000","last_active":"2022-04-03T19:44:21.302772+0000","last_peered":"2022-04-03T19:44:21.302772+0000","last_clean":"2022-04-03T19:44:21.302772+0000","last_became_active":"2022-04-03T19:44:21.298803+0000","last_became_peered":"2022-04-03T19:44:21.298803+0000","last_unstale":"2022-04-03T19:44:21.302772+0000","last_undegraded":"2022-04-03T19:44:21.302772+0000","last_fullsized":"2022-04-03T19:44:21.302772+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:29:43.577843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299160+0000","last_change":"2022-04-03T19:44:21.293638+0000","last_active":"2022-04-03T19:44:21.299160+0000","last_peered":"2022-04-03T19:44:21.299160+0000","last_clean":"2022-04-03T19:44:21.299160+0000","last_became_active":"2022-04-03T19:44:21.293359+0000","last_became_peered":"2022-04-03T19:44:21.293359+0000","last_unstale":"2022-04-03T19:44:21.299160+0000","last_undegraded":"2022-04-03T19:44:21.299160+0000","last_fullsized":"2022-04-03T19:44:21.299160+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:28:50.113792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.306107+0000","last_change":"2022-04-03T19:44:21.299916+0000","last_active":"2022-04-03T19:44:21.306107+0000","last_peered":"2022-04-03T19:44:21.306107+0000","last_clean":"2022-04-03T19:44:21.306107+0000","last_became_active":"2022-04-03T19:44:21.299578+0000","last_became_peered":"2022-04-03T19:44:21.299578+0000","last_unstale":"2022-04-03T19:44:21.306107+0000","last_undegraded":"2022-04-03T19:44:21.306107+0000","last_fullsized":"2022-04-03T19:44:21.306107+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:38:40.281496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.298833+0000","last_change":"2022-04-03T19:44:21.294003+0000","last_active":"2022-04-03T19:44:21.298833+0000","last_peered":"2022-04-03T19:44:21.298833+0000","last_clean":"2022-04-03T19:44:21.298833+0000","last_became_active":"2022-04-03T19:44:21.293640+0000","last_became_peered":"2022-04-03T19:44:21.293640+0000","last_unstale":"2022-04-03T19:44:21.298833+0000","last_undegraded":"2022-04-03T19:44:21.298833+0000","last_fullsized":"2022-04-03T19:44:21.298833+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:40:13.436484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302745+0000","last_change":"2022-04-03T19:44:21.301158+0000","last_active":"2022-04-03T19:44:21.302745+0000","last_peered":"2022-04-03T19:44:21.302745+0000","last_clean":"2022-04-03T19:44:21.302745+0000","last_became_active":"2022-04-03T19:44:21.300823+0000","last_became_peered":"2022-04-03T19:44:21.300823+0000","last_unstale":"2022-04-03T19:44:21.302745+0000","last_undegraded":"2022-04-03T19:44:21.302745+0000","last_fullsized":"2022-04-03T19:44:21.302745+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:09:28.365828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299985+0000","last_change":"2022-04-03T19:44:21.295301+0000","last_active":"2022-04-03T19:44:21.299985+0000","last_peered":"2022-04-03T19:44:21.299985+0000","last_clean":"2022-04-03T19:44:21.299985+0000","last_became_active":"2022-04-03T19:44:21.295032+0000","last_became_peered":"2022-04-03T19:44:21.295032+0000","last_unstale":"2022-04-03T19:44:21.299985+0000","last_undegraded":"2022-04-03T19:44:21.299985+0000","last_fullsized":"2022-04-03T19:44:21.299985+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:55:55.637194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.301541+0000","last_change":"2022-04-03T19:44:21.296250+0000","last_active":"2022-04-03T19:44:21.301541+0000","last_peered":"2022-04-03T19:44:21.301541+0000","last_clean":"2022-04-03T19:44:21.301541+0000","last_became_active":"2022-04-03T19:44:21.295967+0000","last_became_peered":"2022-04-03T19:44:21.295967+0000","last_unstale":"2022-04-03T19:44:21.301541+0000","last_undegraded":"2022-04-03T19:44:21.301541+0000","last_fullsized":"2022-04-03T19:44:21.301541+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:43:34.301454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299466+0000","last_change":"2022-04-03T19:44:21.297229+0000","last_active":"2022-04-03T19:44:21.299466+0000","last_peered":"2022-04-03T19:44:21.299466+0000","last_clean":"2022-04-03T19:44:21.299466+0000","last_became_active":"2022-04-03T19:44:21.296993+0000","last_became_peered":"2022-04-03T19:44:21.296993+0000","last_unstale":"2022-04-03T19:44:21.299466+0000","last_undegraded":"2022-04-03T19:44:21.299466+0000","last_fullsized":"2022-04-03T19:44:21.299466+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:27:17.280580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303252+0000","last_change":"2022-04-03T19:44:21.302427+0000","last_active":"2022-04-03T19:44:21.303252+0000","last_peered":"2022-04-03T19:44:21.303252+0000","last_clean":"2022-04-03T19:44:21.303252+0000","last_became_active":"2022-04-03T19:44:21.302157+0000","last_became_peered":"2022-04-03T19:44:21.302157+0000","last_unstale":"2022-04-03T19:44:21.303252+0000","last_undegraded":"2022-04-03T19:44:21.303252+0000","last_fullsized":"2022-04-03T19:44:21.303252+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:57:23.292801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.298518+0000","last_change":"2022-04-03T19:44:21.293096+0000","last_active":"2022-04-03T19:44:21.298518+0000","last_peered":"2022-04-03T19:44:21.298518+0000","last_clean":"2022-04-03T19:44:21.298518+0000","last_became_active":"2022-04-03T19:44:21.292785+0000","last_became_peered":"2022-04-03T19:44:21.292785+0000","last_unstale":"2022-04-03T19:44:21.298518+0000","last_undegraded":"2022-04-03T19:44:21.298518+0000","last_fullsized":"2022-04-03T19:44:21.298518+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:27:06.759308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.306127+0000","last_change":"2022-04-03T19:44:21.300957+0000","last_active":"2022-04-03T19:44:21.306127+0000","last_peered":"2022-04-03T19:44:21.306127+0000","last_clean":"2022-04-03T19:44:21.306127+0000","last_became_active":"2022-04-03T19:44:21.300342+0000","last_became_peered":"2022-04-03T19:44:21.300342+0000","last_unstale":"2022-04-03T19:44:21.306127+0000","last_undegraded":"2022-04-03T19:44:21.306127+0000","last_fullsized":"2022-04-03T19:44:21.306127+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:43:53.194230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302142+0000","last_change":"2022-04-03T19:44:21.295996+0000","last_active":"2022-04-03T19:44:21.302142+0000","last_peered":"2022-04-03T19:44:21.302142+0000","last_clean":"2022-04-03T19:44:21.302142+0000","last_became_active":"2022-04-03T19:44:21.295766+0000","last_became_peered":"2022-04-03T19:44:21.295766+0000","last_unstale":"2022-04-03T19:44:21.302142+0000","last_undegraded":"2022-04-03T19:44:21.302142+0000","last_fullsized":"2022-04-03T19:44:21.302142+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:21:31.628478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302310+0000","last_change":"2022-04-03T19:44:21.299840+0000","last_active":"2022-04-03T19:44:21.302310+0000","last_peered":"2022-04-03T19:44:21.302310+0000","last_clean":"2022-04-03T19:44:21.302310+0000","last_became_active":"2022-04-03T19:44:21.299487+0000","last_became_peered":"2022-04-03T19:44:21.299487+0000","last_unstale":"2022-04-03T19:44:21.302310+0000","last_undegraded":"2022-04-03T19:44:21.302310+0000","last_fullsized":"2022-04-03T19:44:21.302310+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:24:39.583019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.295324+0000","last_change":"2022-04-03T19:44:21.291748+0000","last_active":"2022-04-03T19:44:21.295324+0000","last_peered":"2022-04-03T19:44:21.295324+0000","last_clean":"2022-04-03T19:44:21.295324+0000","last_became_active":"2022-04-03T19:44:21.290939+0000","last_became_peered":"2022-04-03T19:44:21.290939+0000","last_unstale":"2022-04-03T19:44:21.295324+0000","last_undegraded":"2022-04-03T19:44:21.295324+0000","last_fullsized":"2022-04-03T19:44:21.295324+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:28:45.710805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302521+0000","last_change":"2022-04-03T19:44:21.299442+0000","last_active":"2022-04-03T19:44:21.302521+0000","last_peered":"2022-04-03T19:44:21.302521+0000","last_clean":"2022-04-03T19:44:21.302521+0000","last_became_active":"2022-04-03T19:44:21.299166+0000","last_became_peered":"2022-04-03T19:44:21.299166+0000","last_unstale":"2022-04-03T19:44:21.302521+0000","last_undegraded":"2022-04-03T19:44:21.302521+0000","last_fullsized":"2022-04-03T19:44:21.302521+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:31:30.905517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.311290+0000","last_change":"2022-04-03T19:44:21.307007+0000","last_active":"2022-04-03T19:44:21.311290+0000","last_peered":"2022-04-03T19:44:21.311290+0000","last_clean":"2022-04-03T19:44:21.311290+0000","last_became_active":"2022-04-03T19:44:21.306650+0000","last_became_peered":"2022-04-03T19:44:21.306650+0000","last_unstale":"2022-04-03T19:44:21.311290+0000","last_undegraded":"2022-04-03T19:44:21.311290+0000","last_fullsized":"2022-04-03T19:44:21.311290+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:03:26.043624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299849+0000","last_change":"2022-04-03T19:44:21.293566+0000","last_active":"2022-04-03T19:44:21.299849+0000","last_peered":"2022-04-03T19:44:21.299849+0000","last_clean":"2022-04-03T19:44:21.299849+0000","last_became_active":"2022-04-03T19:44:21.293263+0000","last_became_peered":"2022-04-03T19:44:21.293263+0000","last_unstale":"2022-04-03T19:44:21.299849+0000","last_undegraded":"2022-04-03T19:44:21.299849+0000","last_fullsized":"2022-04-03T19:44:21.299849+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:39:52.523944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302610+0000","last_change":"2022-04-03T19:44:21.300329+0000","last_active":"2022-04-03T19:44:21.302610+0000","last_peered":"2022-04-03T19:44:21.302610+0000","last_clean":"2022-04-03T19:44:21.302610+0000","last_became_active":"2022-04-03T19:44:21.300000+0000","last_became_peered":"2022-04-03T19:44:21.300000+0000","last_unstale":"2022-04-03T19:44:21.302610+0000","last_undegraded":"2022-04-03T19:44:21.302610+0000","last_fullsized":"2022-04-03T19:44:21.302610+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:15:10.839571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.312922+0000","last_change":"2022-04-03T19:44:21.302506+0000","last_active":"2022-04-03T19:44:21.312922+0000","last_peered":"2022-04-03T19:44:21.312922+0000","last_clean":"2022-04-03T19:44:21.312922+0000","last_became_active":"2022-04-03T19:44:21.302235+0000","last_became_peered":"2022-04-03T19:44:21.302235+0000","last_unstale":"2022-04-03T19:44:21.312922+0000","last_undegraded":"2022-04-03T19:44:21.312922+0000","last_fullsized":"2022-04-03T19:44:21.312922+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:25:25.801754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313258+0000","last_change":"2022-04-03T19:44:21.306670+0000","last_active":"2022-04-03T19:44:21.313258+0000","last_peered":"2022-04-03T19:44:21.313258+0000","last_clean":"2022-04-03T19:44:21.313258+0000","last_became_active":"2022-04-03T19:44:21.305722+0000","last_became_peered":"2022-04-03T19:44:21.305722+0000","last_unstale":"2022-04-03T19:44:21.313258+0000","last_undegraded":"2022-04-03T19:44:21.313258+0000","last_fullsized":"2022-04-03T19:44:21.313258+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:44:09.515207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303081+0000","last_change":"2022-04-03T19:44:21.300024+0000","last_active":"2022-04-03T19:44:21.303081+0000","last_peered":"2022-04-03T19:44:21.303081+0000","last_clean":"2022-04-03T19:44:21.303081+0000","last_became_active":"2022-04-03T19:44:21.299753+0000","last_became_peered":"2022-04-03T19:44:21.299753+0000","last_unstale":"2022-04-03T19:44:21.303081+0000","last_undegraded":"2022-04-03T19:44:21.303081+0000","last_fullsized":"2022-04-03T19:44:21.303081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:15:31.808623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.390304+0000","last_change":"2022-04-03T19:44:20.783898+0000","last_active":"2022-04-03T19:44:22.390304+0000","last_peered":"2022-04-03T19:44:22.390304+0000","last_clean":"2022-04-03T19:44:22.390304+0000","last_became_active":"2022-04-03T19:44:20.783612+0000","last_became_peered":"2022-04-03T19:44:20.783612+0000","last_unstale":"2022-04-03T19:44:22.390304+0000","last_undegraded":"2022-04-03T19:44:22.390304+0000","last_fullsized":"2022-04-03T19:44:22.390304+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:13:32.863927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.310259+0000","last_change":"2022-04-03T19:44:21.306904+0000","last_active":"2022-04-03T19:44:21.310259+0000","last_peered":"2022-04-03T19:44:21.310259+0000","last_clean":"2022-04-03T19:44:21.310259+0000","last_became_active":"2022-04-03T19:44:21.306616+0000","last_became_peered":"2022-04-03T19:44:21.306616+0000","last_unstale":"2022-04-03T19:44:21.310259+0000","last_undegraded":"2022-04-03T19:44:21.310259+0000","last_fullsized":"2022-04-03T19:44:21.310259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:49:35.783912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.296341+0000","last_change":"2022-04-03T19:44:20.788125+0000","last_active":"2022-04-03T19:44:21.296341+0000","last_peered":"2022-04-03T19:44:21.296341+0000","last_clean":"2022-04-03T19:44:21.296341+0000","last_became_active":"2022-04-03T19:44:20.787792+0000","last_became_peered":"2022-04-03T19:44:20.787792+0000","last_unstale":"2022-04-03T19:44:21.296341+0000","last_undegraded":"2022-04-03T19:44:21.296341+0000","last_fullsized":"2022-04-03T19:44:21.296341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:11:51.126774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305572+0000","last_change":"2022-04-03T19:44:21.302904+0000","last_active":"2022-04-03T19:44:21.305572+0000","last_peered":"2022-04-03T19:44:21.305572+0000","last_clean":"2022-04-03T19:44:21.305572+0000","last_became_active":"2022-04-03T19:44:21.301564+0000","last_became_peered":"2022-04-03T19:44:21.301564+0000","last_unstale":"2022-04-03T19:44:21.305572+0000","last_undegraded":"2022-04-03T19:44:21.305572+0000","last_fullsized":"2022-04-03T19:44:21.305572+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:25:51.611789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.283649+0000","last_change":"2022-04-03T19:44:20.792529+0000","last_active":"2022-04-03T19:44:21.283649+0000","last_peered":"2022-04-03T19:44:21.283649+0000","last_clean":"2022-04-03T19:44:21.283649+0000","last_became_active":"2022-04-03T19:44:20.792014+0000","last_became_peered":"2022-04-03T19:44:20.792014+0000","last_unstale":"2022-04-03T19:44:21.283649+0000","last_undegraded":"2022-04-03T19:44:21.283649+0000","last_fullsized":"2022-04-03T19:44:21.283649+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:01:38.319693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302637+0000","last_change":"2022-04-03T19:44:21.299302+0000","last_active":"2022-04-03T19:44:21.302637+0000","last_peered":"2022-04-03T19:44:21.302637+0000","last_clean":"2022-04-03T19:44:21.302637+0000","last_became_active":"2022-04-03T19:44:21.299007+0000","last_became_peered":"2022-04-03T19:44:21.299007+0000","last_unstale":"2022-04-03T19:44:21.302637+0000","last_undegraded":"2022-04-03T19:44:21.302637+0000","last_fullsized":"2022-04-03T19:44:21.302637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:59:48.767335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.282232+0000","last_change":"2022-04-03T19:44:20.790672+0000","last_active":"2022-04-03T19:44:21.282232+0000","last_peered":"2022-04-03T19:44:21.282232+0000","last_clean":"2022-04-03T19:44:21.282232+0000","last_became_active":"2022-04-03T19:44:20.788526+0000","last_became_peered":"2022-04-03T19:44:20.788526+0000","last_unstale":"2022-04-03T19:44:21.282232+0000","last_undegraded":"2022-04-03T19:44:21.282232+0000","last_fullsized":"2022-04-03T19:44:21.282232+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:14:50.058917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.309960+0000","last_change":"2022-04-03T19:44:21.306472+0000","last_active":"2022-04-03T19:44:21.309960+0000","last_peered":"2022-04-03T19:44:21.309960+0000","last_clean":"2022-04-03T19:44:21.309960+0000","last_became_active":"2022-04-03T19:44:21.306071+0000","last_became_peered":"2022-04-03T19:44:21.306071+0000","last_unstale":"2022-04-03T19:44:21.309960+0000","last_undegraded":"2022-04-03T19:44:21.309960+0000","last_fullsized":"2022-04-03T19:44:21.309960+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:43:22.221014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286837+0000","last_change":"2022-04-03T19:44:20.785638+0000","last_active":"2022-04-03T19:44:21.286837+0000","last_peered":"2022-04-03T19:44:21.286837+0000","last_clean":"2022-04-03T19:44:21.286837+0000","last_became_active":"2022-04-03T19:44:20.784963+0000","last_became_peered":"2022-04-03T19:44:20.784963+0000","last_unstale":"2022-04-03T19:44:21.286837+0000","last_undegraded":"2022-04-03T19:44:21.286837+0000","last_fullsized":"2022-04-03T19:44:21.286837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:39:26.933298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.309328+0000","last_change":"2022-04-03T19:44:21.299583+0000","last_active":"2022-04-03T19:44:21.309328+0000","last_peered":"2022-04-03T19:44:21.309328+0000","last_clean":"2022-04-03T19:44:21.309328+0000","last_became_active":"2022-04-03T19:44:21.299301+0000","last_became_peered":"2022-04-03T19:44:21.299301+0000","last_unstale":"2022-04-03T19:44:21.309328+0000","last_undegraded":"2022-04-03T19:44:21.309328+0000","last_fullsized":"2022-04-03T19:44:21.309328+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:37:49.567741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"24'3","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.290849+0000","last_change":"2022-04-03T19:44:20.795890+0000","last_active":"2022-04-03T19:44:26.290849+0000","last_peered":"2022-04-03T19:44:26.290849+0000","last_clean":"2022-04-03T19:44:26.290849+0000","last_became_active":"2022-04-03T19:44:20.795611+0000","last_became_peered":"2022-04-03T19:44:20.795611+0000","last_unstale":"2022-04-03T19:44:26.290849+0000","last_undegraded":"2022-04-03T19:44:26.290849+0000","last_fullsized":"2022-04-03T19:44:26.290849+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:08:33.216934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313173+0000","last_change":"2022-04-03T19:44:21.304724+0000","last_active":"2022-04-03T19:44:21.313173+0000","last_peered":"2022-04-03T19:44:21.313173+0000","last_clean":"2022-04-03T19:44:21.313173+0000","last_became_active":"2022-04-03T19:44:21.302708+0000","last_became_peered":"2022-04-03T19:44:21.302708+0000","last_unstale":"2022-04-03T19:44:21.313173+0000","last_undegraded":"2022-04-03T19:44:21.313173+0000","last_fullsized":"2022-04-03T19:44:21.313173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:04:54.970100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"24'6","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.292663+0000","last_change":"2022-04-03T19:44:20.785659+0000","last_active":"2022-04-03T19:44:26.292663+0000","last_peered":"2022-04-03T19:44:26.292663+0000","last_clean":"2022-04-03T19:44:26.292663+0000","last_became_active":"2022-04-03T19:44:20.785308+0000","last_became_peered":"2022-04-03T19:44:20.785308+0000","last_unstale":"2022-04-03T19:44:26.292663+0000","last_undegraded":"2022-04-03T19:44:26.292663+0000","last_fullsized":"2022-04-03T19:44:26.292663+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:52:08.268532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1548,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293514+0000","last_change":"2022-04-03T19:44:21.291033+0000","last_active":"2022-04-03T19:44:21.293514+0000","last_peered":"2022-04-03T19:44:21.293514+0000","last_clean":"2022-04-03T19:44:21.293514+0000","last_became_active":"2022-04-03T19:44:21.290713+0000","last_became_peered":"2022-04-03T19:44:21.290713+0000","last_unstale":"2022-04-03T19:44:21.293514+0000","last_undegraded":"2022-04-03T19:44:21.293514+0000","last_fullsized":"2022-04-03T19:44:21.293514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:19:41.873591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.290200+0000","last_change":"2022-04-03T19:44:20.787511+0000","last_active":"2022-04-03T19:44:26.290200+0000","last_peered":"2022-04-03T19:44:26.290200+0000","last_clean":"2022-04-03T19:44:26.290200+0000","last_became_active":"2022-04-03T19:44:20.787189+0000","last_became_peered":"2022-04-03T19:44:20.787189+0000","last_unstale":"2022-04-03T19:44:26.290200+0000","last_undegraded":"2022-04-03T19:44:26.290200+0000","last_fullsized":"2022-04-03T19:44:26.290200+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:11:56.297263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313518+0000","last_change":"2022-04-03T19:44:21.307846+0000","last_active":"2022-04-03T19:44:21.313518+0000","last_peered":"2022-04-03T19:44:21.313518+0000","last_clean":"2022-04-03T19:44:21.313518+0000","last_became_active":"2022-04-03T19:44:21.307513+0000","last_became_peered":"2022-04-03T19:44:21.307513+0000","last_unstale":"2022-04-03T19:44:21.313518+0000","last_undegraded":"2022-04-03T19:44:21.313518+0000","last_fullsized":"2022-04-03T19:44:21.313518+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:58:09.091062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.281910+0000","last_change":"2022-04-03T19:44:20.790212+0000","last_active":"2022-04-03T19:44:21.281910+0000","last_peered":"2022-04-03T19:44:21.281910+0000","last_clean":"2022-04-03T19:44:21.281910+0000","last_became_active":"2022-04-03T19:44:20.789863+0000","last_became_peered":"2022-04-03T19:44:20.789863+0000","last_unstale":"2022-04-03T19:44:21.281910+0000","last_undegraded":"2022-04-03T19:44:21.281910+0000","last_fullsized":"2022-04-03T19:44:21.281910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:48:13.489309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303061+0000","last_change":"2022-04-03T19:44:21.299953+0000","last_active":"2022-04-03T19:44:21.303061+0000","last_peered":"2022-04-03T19:44:21.303061+0000","last_clean":"2022-04-03T19:44:21.303061+0000","last_became_active":"2022-04-03T19:44:21.299667+0000","last_became_peered":"2022-04-03T19:44:21.299667+0000","last_unstale":"2022-04-03T19:44:21.303061+0000","last_undegraded":"2022-04-03T19:44:21.303061+0000","last_fullsized":"2022-04-03T19:44:21.303061+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:51:16.125160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286253+0000","last_change":"2022-04-03T19:44:20.791054+0000","last_active":"2022-04-03T19:44:21.286253+0000","last_peered":"2022-04-03T19:44:21.286253+0000","last_clean":"2022-04-03T19:44:21.286253+0000","last_became_active":"2022-04-03T19:44:20.790664+0000","last_became_peered":"2022-04-03T19:44:20.790664+0000","last_unstale":"2022-04-03T19:44:21.286253+0000","last_undegraded":"2022-04-03T19:44:21.286253+0000","last_fullsized":"2022-04-03T19:44:21.286253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:07:27.501728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.304407+0000","last_change":"2022-04-03T19:44:21.295028+0000","last_active":"2022-04-03T19:44:21.304407+0000","last_peered":"2022-04-03T19:44:21.304407+0000","last_clean":"2022-04-03T19:44:21.304407+0000","last_became_active":"2022-04-03T19:44:21.294723+0000","last_became_peered":"2022-04-03T19:44:21.294723+0000","last_unstale":"2022-04-03T19:44:21.304407+0000","last_undegraded":"2022-04-03T19:44:21.304407+0000","last_fullsized":"2022-04-03T19:44:21.304407+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:32:45.455842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"24'8","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.294387+0000","last_change":"2022-04-03T19:44:20.792133+0000","last_active":"2022-04-03T19:44:26.294387+0000","last_peered":"2022-04-03T19:44:26.294387+0000","last_clean":"2022-04-03T19:44:26.294387+0000","last_became_active":"2022-04-03T19:44:20.791829+0000","last_became_peered":"2022-04-03T19:44:20.791829+0000","last_unstale":"2022-04-03T19:44:26.294387+0000","last_undegraded":"2022-04-03T19:44:26.294387+0000","last_fullsized":"2022-04-03T19:44:26.294387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:01:21.966734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.310014+0000","last_change":"2022-04-03T19:44:21.300267+0000","last_active":"2022-04-03T19:44:21.310014+0000","last_peered":"2022-04-03T19:44:21.310014+0000","last_clean":"2022-04-03T19:44:21.310014+0000","last_became_active":"2022-04-03T19:44:21.299948+0000","last_became_peered":"2022-04-03T19:44:21.299948+0000","last_unstale":"2022-04-03T19:44:21.310014+0000","last_undegraded":"2022-04-03T19:44:21.310014+0000","last_fullsized":"2022-04-03T19:44:21.310014+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:06:54.481379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.391229+0000","last_change":"2022-04-03T19:44:20.789793+0000","last_active":"2022-04-03T19:44:22.391229+0000","last_peered":"2022-04-03T19:44:22.391229+0000","last_clean":"2022-04-03T19:44:22.391229+0000","last_became_active":"2022-04-03T19:44:20.789371+0000","last_became_peered":"2022-04-03T19:44:20.789371+0000","last_unstale":"2022-04-03T19:44:22.391229+0000","last_undegraded":"2022-04-03T19:44:22.391229+0000","last_fullsized":"2022-04-03T19:44:22.391229+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:15:52.060059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.301800+0000","last_change":"2022-04-03T19:44:21.298477+0000","last_active":"2022-04-03T19:44:21.301800+0000","last_peered":"2022-04-03T19:44:21.301800+0000","last_clean":"2022-04-03T19:44:21.301800+0000","last_became_active":"2022-04-03T19:44:21.298165+0000","last_became_peered":"2022-04-03T19:44:21.298165+0000","last_unstale":"2022-04-03T19:44:21.301800+0000","last_undegraded":"2022-04-03T19:44:21.301800+0000","last_fullsized":"2022-04-03T19:44:21.301800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:23:11.042589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.290125+0000","last_change":"2022-04-03T19:44:20.787949+0000","last_active":"2022-04-03T19:44:26.290125+0000","last_peered":"2022-04-03T19:44:26.290125+0000","last_clean":"2022-04-03T19:44:26.290125+0000","last_became_active":"2022-04-03T19:44:20.787592+0000","last_became_peered":"2022-04-03T19:44:20.787592+0000","last_unstale":"2022-04-03T19:44:26.290125+0000","last_undegraded":"2022-04-03T19:44:26.290125+0000","last_fullsized":"2022-04-03T19:44:26.290125+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:50:09.411246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302765+0000","last_change":"2022-04-03T19:44:21.301200+0000","last_active":"2022-04-03T19:44:21.302765+0000","last_peered":"2022-04-03T19:44:21.302765+0000","last_clean":"2022-04-03T19:44:21.302765+0000","last_became_active":"2022-04-03T19:44:21.300878+0000","last_became_peered":"2022-04-03T19:44:21.300878+0000","last_unstale":"2022-04-03T19:44:21.302765+0000","last_undegraded":"2022-04-03T19:44:21.302765+0000","last_fullsized":"2022-04-03T19:44:21.302765+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:54:36.089411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.391128+0000","last_change":"2022-04-03T19:44:20.783711+0000","last_active":"2022-04-03T19:44:22.391128+0000","last_peered":"2022-04-03T19:44:22.391128+0000","last_clean":"2022-04-03T19:44:22.391128+0000","last_became_active":"2022-04-03T19:44:20.783423+0000","last_became_peered":"2022-04-03T19:44:20.783423+0000","last_unstale":"2022-04-03T19:44:22.391128+0000","last_undegraded":"2022-04-03T19:44:22.391128+0000","last_fullsized":"2022-04-03T19:44:22.391128+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:22:37.665229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303123+0000","last_change":"2022-04-03T19:44:21.300815+0000","last_active":"2022-04-03T19:44:21.303123+0000","last_peered":"2022-04-03T19:44:21.303123+0000","last_clean":"2022-04-03T19:44:21.303123+0000","last_became_active":"2022-04-03T19:44:21.300545+0000","last_became_peered":"2022-04-03T19:44:21.300545+0000","last_unstale":"2022-04-03T19:44:21.303123+0000","last_undegraded":"2022-04-03T19:44:21.303123+0000","last_fullsized":"2022-04-03T19:44:21.303123+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:33:13.941191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.285374+0000","last_change":"2022-04-03T19:44:20.785480+0000","last_active":"2022-04-03T19:44:21.285374+0000","last_peered":"2022-04-03T19:44:21.285374+0000","last_clean":"2022-04-03T19:44:21.285374+0000","last_became_active":"2022-04-03T19:44:20.785178+0000","last_became_peered":"2022-04-03T19:44:20.785178+0000","last_unstale":"2022-04-03T19:44:21.285374+0000","last_undegraded":"2022-04-03T19:44:21.285374+0000","last_fullsized":"2022-04-03T19:44:21.285374+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:15:37.181452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.308247+0000","last_change":"2022-04-03T19:44:21.296509+0000","last_active":"2022-04-03T19:44:21.308247+0000","last_peered":"2022-04-03T19:44:21.308247+0000","last_clean":"2022-04-03T19:44:21.308247+0000","last_became_active":"2022-04-03T19:44:21.296240+0000","last_became_peered":"2022-04-03T19:44:21.296240+0000","last_unstale":"2022-04-03T19:44:21.308247+0000","last_undegraded":"2022-04-03T19:44:21.308247+0000","last_fullsized":"2022-04-03T19:44:21.308247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:50:45.955195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.288396+0000","last_change":"2022-04-03T19:44:20.792550+0000","last_active":"2022-04-03T19:44:26.288396+0000","last_peered":"2022-04-03T19:44:26.288396+0000","last_clean":"2022-04-03T19:44:26.288396+0000","last_became_active":"2022-04-03T19:44:20.792191+0000","last_became_peered":"2022-04-03T19:44:20.792191+0000","last_unstale":"2022-04-03T19:44:26.288396+0000","last_undegraded":"2022-04-03T19:44:26.288396+0000","last_fullsized":"2022-04-03T19:44:26.288396+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:51:30.297728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299612+0000","last_change":"2022-04-03T19:44:21.294898+0000","last_active":"2022-04-03T19:44:21.299612+0000","last_peered":"2022-04-03T19:44:21.299612+0000","last_clean":"2022-04-03T19:44:21.299612+0000","last_became_active":"2022-04-03T19:44:21.294635+0000","last_became_peered":"2022-04-03T19:44:21.294635+0000","last_unstale":"2022-04-03T19:44:21.299612+0000","last_undegraded":"2022-04-03T19:44:21.299612+0000","last_fullsized":"2022-04-03T19:44:21.299612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:42:52.629552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.289106+0000","last_change":"2022-04-03T19:44:20.783711+0000","last_active":"2022-04-03T19:44:21.289106+0000","last_peered":"2022-04-03T19:44:21.289106+0000","last_clean":"2022-04-03T19:44:21.289106+0000","last_became_active":"2022-04-03T19:44:20.783426+0000","last_became_peered":"2022-04-03T19:44:20.783426+0000","last_unstale":"2022-04-03T19:44:21.289106+0000","last_undegraded":"2022-04-03T19:44:21.289106+0000","last_fullsized":"2022-04-03T19:44:21.289106+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:25:26.219558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302205+0000","last_change":"2022-04-03T19:44:21.296081+0000","last_active":"2022-04-03T19:44:21.302205+0000","last_peered":"2022-04-03T19:44:21.302205+0000","last_clean":"2022-04-03T19:44:21.302205+0000","last_became_active":"2022-04-03T19:44:21.295784+0000","last_became_peered":"2022-04-03T19:44:21.295784+0000","last_unstale":"2022-04-03T19:44:21.302205+0000","last_undegraded":"2022-04-03T19:44:21.302205+0000","last_fullsized":"2022-04-03T19:44:21.302205+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:23:40.568168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.283775+0000","last_change":"2022-04-03T19:44:20.789868+0000","last_active":"2022-04-03T19:44:21.283775+0000","last_peered":"2022-04-03T19:44:21.283775+0000","last_clean":"2022-04-03T19:44:21.283775+0000","last_became_active":"2022-04-03T19:44:20.789563+0000","last_became_peered":"2022-04-03T19:44:20.789563+0000","last_unstale":"2022-04-03T19:44:21.283775+0000","last_undegraded":"2022-04-03T19:44:21.283775+0000","last_fullsized":"2022-04-03T19:44:21.283775+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:04:22.814932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293542+0000","last_change":"2022-04-03T19:44:21.291212+0000","last_active":"2022-04-03T19:44:21.293542+0000","last_peered":"2022-04-03T19:44:21.293542+0000","last_clean":"2022-04-03T19:44:21.293542+0000","last_became_active":"2022-04-03T19:44:21.290888+0000","last_became_peered":"2022-04-03T19:44:21.290888+0000","last_unstale":"2022-04-03T19:44:21.293542+0000","last_undegraded":"2022-04-03T19:44:21.293542+0000","last_fullsized":"2022-04-03T19:44:21.293542+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:29:34.850679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.281344+0000","last_change":"2022-04-03T19:44:20.789319+0000","last_active":"2022-04-03T19:44:21.281344+0000","last_peered":"2022-04-03T19:44:21.281344+0000","last_clean":"2022-04-03T19:44:21.281344+0000","last_became_active":"2022-04-03T19:44:20.788840+0000","last_became_peered":"2022-04-03T19:44:20.788840+0000","last_unstale":"2022-04-03T19:44:21.281344+0000","last_undegraded":"2022-04-03T19:44:21.281344+0000","last_fullsized":"2022-04-03T19:44:21.281344+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:50:21.536926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313655+0000","last_change":"2022-04-03T19:44:21.311886+0000","last_active":"2022-04-03T19:44:21.313655+0000","last_peered":"2022-04-03T19:44:21.313655+0000","last_clean":"2022-04-03T19:44:21.313655+0000","last_became_active":"2022-04-03T19:44:21.311521+0000","last_became_peered":"2022-04-03T19:44:21.311521+0000","last_unstale":"2022-04-03T19:44:21.313655+0000","last_undegraded":"2022-04-03T19:44:21.313655+0000","last_fullsized":"2022-04-03T19:44:21.313655+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:25:47.761166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.297426+0000","last_change":"2022-04-03T19:44:14.754674+0000","last_active":"2022-04-03T19:44:21.297426+0000","last_peered":"2022-04-03T19:44:21.297426+0000","last_clean":"2022-04-03T19:44:21.297426+0000","last_became_active":"2022-04-03T19:44:14.754210+0000","last_became_peered":"2022-04-03T19:44:14.754210+0000","last_unstale":"2022-04-03T19:44:21.297426+0000","last_undegraded":"2022-04-03T19:44:21.297426+0000","last_fullsized":"2022-04-03T19:44:21.297426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:23:14.624494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286308+0000","last_change":"2022-04-03T19:44:20.785400+0000","last_active":"2022-04-03T19:44:21.286308+0000","last_peered":"2022-04-03T19:44:21.286308+0000","last_clean":"2022-04-03T19:44:21.286308+0000","last_became_active":"2022-04-03T19:44:20.785125+0000","last_became_peered":"2022-04-03T19:44:20.785125+0000","last_unstale":"2022-04-03T19:44:21.286308+0000","last_undegraded":"2022-04-03T19:44:21.286308+0000","last_fullsized":"2022-04-03T19:44:21.286308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:24:02.690646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313197+0000","last_change":"2022-04-03T19:44:21.305321+0000","last_active":"2022-04-03T19:44:21.313197+0000","last_peered":"2022-04-03T19:44:21.313197+0000","last_clean":"2022-04-03T19:44:21.313197+0000","last_became_active":"2022-04-03T19:44:21.304017+0000","last_became_peered":"2022-04-03T19:44:21.304017+0000","last_unstale":"2022-04-03T19:44:21.313197+0000","last_undegraded":"2022-04-03T19:44:21.313197+0000","last_fullsized":"2022-04-03T19:44:21.313197+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:59:12.619058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.285445+0000","last_change":"2022-04-03T19:44:14.755580+0000","last_active":"2022-04-03T19:44:21.285445+0000","last_peered":"2022-04-03T19:44:21.285445+0000","last_clean":"2022-04-03T19:44:21.285445+0000","last_became_active":"2022-04-03T19:44:14.755222+0000","last_became_peered":"2022-04-03T19:44:14.755222+0000","last_unstale":"2022-04-03T19:44:21.285445+0000","last_undegraded":"2022-04-03T19:44:21.285445+0000","last_fullsized":"2022-04-03T19:44:21.285445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:08:18.314326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.283116+0000","last_change":"2022-04-03T19:44:20.791540+0000","last_active":"2022-04-03T19:44:21.283116+0000","last_peered":"2022-04-03T19:44:21.283116+0000","last_clean":"2022-04-03T19:44:21.283116+0000","last_became_active":"2022-04-03T19:44:20.791192+0000","last_became_peered":"2022-04-03T19:44:20.791192+0000","last_unstale":"2022-04-03T19:44:21.283116+0000","last_undegraded":"2022-04-03T19:44:21.283116+0000","last_fullsized":"2022-04-03T19:44:21.283116+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:58:59.674606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313218+0000","last_change":"2022-04-03T19:44:21.305930+0000","last_active":"2022-04-03T19:44:21.313218+0000","last_peered":"2022-04-03T19:44:21.313218+0000","last_clean":"2022-04-03T19:44:21.313218+0000","last_became_active":"2022-04-03T19:44:21.305659+0000","last_became_peered":"2022-04-03T19:44:21.305659+0000","last_unstale":"2022-04-03T19:44:21.313218+0000","last_undegraded":"2022-04-03T19:44:21.313218+0000","last_fullsized":"2022-04-03T19:44:21.313218+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:14:34.958569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.290570+0000","last_change":"2022-04-03T19:44:14.754598+0000","last_active":"2022-04-03T19:44:21.290570+0000","last_peered":"2022-04-03T19:44:21.290570+0000","last_clean":"2022-04-03T19:44:21.290570+0000","last_became_active":"2022-04-03T19:44:14.754287+0000","last_became_peered":"2022-04-03T19:44:14.754287+0000","last_unstale":"2022-04-03T19:44:21.290570+0000","last_undegraded":"2022-04-03T19:44:21.290570+0000","last_fullsized":"2022-04-03T19:44:21.290570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:32:16.232689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.296054+0000","last_change":"2022-04-03T19:44:20.789567+0000","last_active":"2022-04-03T19:44:21.296054+0000","last_peered":"2022-04-03T19:44:21.296054+0000","last_clean":"2022-04-03T19:44:21.296054+0000","last_became_active":"2022-04-03T19:44:20.788664+0000","last_became_peered":"2022-04-03T19:44:20.788664+0000","last_unstale":"2022-04-03T19:44:21.296054+0000","last_undegraded":"2022-04-03T19:44:21.296054+0000","last_fullsized":"2022-04-03T19:44:21.296054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:15:37.098494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.312944+0000","last_change":"2022-04-03T19:44:21.303079+0000","last_active":"2022-04-03T19:44:21.312944+0000","last_peered":"2022-04-03T19:44:21.312944+0000","last_clean":"2022-04-03T19:44:21.312944+0000","last_became_active":"2022-04-03T19:44:21.302014+0000","last_became_peered":"2022-04-03T19:44:21.302014+0000","last_unstale":"2022-04-03T19:44:21.312944+0000","last_undegraded":"2022-04-03T19:44:21.312944+0000","last_fullsized":"2022-04-03T19:44:21.312944+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:15:58.860701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.285641+0000","last_change":"2022-04-03T19:44:14.754385+0000","last_active":"2022-04-03T19:44:21.285641+0000","last_peered":"2022-04-03T19:44:21.285641+0000","last_clean":"2022-04-03T19:44:21.285641+0000","last_became_active":"2022-04-03T19:44:14.754003+0000","last_became_peered":"2022-04-03T19:44:14.754003+0000","last_unstale":"2022-04-03T19:44:21.285641+0000","last_undegraded":"2022-04-03T19:44:21.285641+0000","last_fullsized":"2022-04-03T19:44:21.285641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:05:22.716933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":166,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286290+0000","last_change":"2022-04-03T19:44:08.726758+0000","last_active":"2022-04-03T19:44:21.286290+0000","last_peered":"2022-04-03T19:44:21.286290+0000","last_clean":"2022-04-03T19:44:21.286290+0000","last_became_active":"2022-04-03T19:44:08.725694+0000","last_became_peered":"2022-04-03T19:44:08.725694+0000","last_unstale":"2022-04-03T19:44:21.286290+0000","last_undegraded":"2022-04-03T19:44:21.286290+0000","last_fullsized":"2022-04-03T19:44:21.286290+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:07.708129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:07.708129+0000","last_clean_scrub_stamp":"2022-04-03T19:44:07.708129+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:27:43.354267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.282879+0000","last_change":"2022-04-03T19:44:20.781585+0000","last_active":"2022-04-03T19:44:21.282879+0000","last_peered":"2022-04-03T19:44:21.282879+0000","last_clean":"2022-04-03T19:44:21.282879+0000","last_became_active":"2022-04-03T19:44:20.781274+0000","last_became_peered":"2022-04-03T19:44:20.781274+0000","last_unstale":"2022-04-03T19:44:21.282879+0000","last_undegraded":"2022-04-03T19:44:21.282879+0000","last_fullsized":"2022-04-03T19:44:21.282879+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:49:54.368001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.301298+0000","last_change":"2022-04-03T19:44:21.296620+0000","last_active":"2022-04-03T19:44:21.301298+0000","last_peered":"2022-04-03T19:44:21.301298+0000","last_clean":"2022-04-03T19:44:21.301298+0000","last_became_active":"2022-04-03T19:44:21.296314+0000","last_became_peered":"2022-04-03T19:44:21.296314+0000","last_unstale":"2022-04-03T19:44:21.301298+0000","last_undegraded":"2022-04-03T19:44:21.301298+0000","last_fullsized":"2022-04-03T19:44:21.301298+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:09:18.338683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286626+0000","last_change":"2022-04-03T19:44:14.754664+0000","last_active":"2022-04-03T19:44:21.286626+0000","last_peered":"2022-04-03T19:44:21.286626+0000","last_clean":"2022-04-03T19:44:21.286626+0000","last_became_active":"2022-04-03T19:44:14.754328+0000","last_became_peered":"2022-04-03T19:44:14.754328+0000","last_unstale":"2022-04-03T19:44:21.286626+0000","last_undegraded":"2022-04-03T19:44:21.286626+0000","last_fullsized":"2022-04-03T19:44:21.286626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:39:58.022353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.281559+0000","last_change":"2022-04-03T19:44:20.787271+0000","last_active":"2022-04-03T19:44:21.281559+0000","last_peered":"2022-04-03T19:44:21.281559+0000","last_clean":"2022-04-03T19:44:21.281559+0000","last_became_active":"2022-04-03T19:44:20.786957+0000","last_became_peered":"2022-04-03T19:44:20.786957+0000","last_unstale":"2022-04-03T19:44:21.281559+0000","last_undegraded":"2022-04-03T19:44:21.281559+0000","last_fullsized":"2022-04-03T19:44:21.281559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:21:45.992744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.311227+0000","last_change":"2022-04-03T19:44:21.306857+0000","last_active":"2022-04-03T19:44:21.311227+0000","last_peered":"2022-04-03T19:44:21.311227+0000","last_clean":"2022-04-03T19:44:21.311227+0000","last_became_active":"2022-04-03T19:44:21.306521+0000","last_became_peered":"2022-04-03T19:44:21.306521+0000","last_unstale":"2022-04-03T19:44:21.311227+0000","last_undegraded":"2022-04-03T19:44:21.311227+0000","last_fullsized":"2022-04-03T19:44:21.311227+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:50:03.912834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.287006+0000","last_change":"2022-04-03T19:44:14.754657+0000","last_active":"2022-04-03T19:44:21.287006+0000","last_peered":"2022-04-03T19:44:21.287006+0000","last_clean":"2022-04-03T19:44:21.287006+0000","last_became_active":"2022-04-03T19:44:14.754310+0000","last_became_peered":"2022-04-03T19:44:14.754310+0000","last_unstale":"2022-04-03T19:44:21.287006+0000","last_undegraded":"2022-04-03T19:44:21.287006+0000","last_fullsized":"2022-04-03T19:44:21.287006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:42:05.279424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.297553+0000","last_change":"2022-04-03T19:44:20.794454+0000","last_active":"2022-04-03T19:44:21.297553+0000","last_peered":"2022-04-03T19:44:21.297553+0000","last_clean":"2022-04-03T19:44:21.297553+0000","last_became_active":"2022-04-03T19:44:20.789468+0000","last_became_peered":"2022-04-03T19:44:20.789468+0000","last_unstale":"2022-04-03T19:44:21.297553+0000","last_undegraded":"2022-04-03T19:44:21.297553+0000","last_fullsized":"2022-04-03T19:44:21.297553+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:02:49.184838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.308286+0000","last_change":"2022-04-03T19:44:21.303669+0000","last_active":"2022-04-03T19:44:21.308286+0000","last_peered":"2022-04-03T19:44:21.308286+0000","last_clean":"2022-04-03T19:44:21.308286+0000","last_became_active":"2022-04-03T19:44:21.303381+0000","last_became_peered":"2022-04-03T19:44:21.303381+0000","last_unstale":"2022-04-03T19:44:21.308286+0000","last_undegraded":"2022-04-03T19:44:21.308286+0000","last_fullsized":"2022-04-03T19:44:21.308286+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:43:22.320070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.289548+0000","last_change":"2022-04-03T19:44:20.791594+0000","last_active":"2022-04-03T19:44:26.289548+0000","last_peered":"2022-04-03T19:44:26.289548+0000","last_clean":"2022-04-03T19:44:26.289548+0000","last_became_active":"2022-04-03T19:44:20.789693+0000","last_became_peered":"2022-04-03T19:44:20.789693+0000","last_unstale":"2022-04-03T19:44:26.289548+0000","last_undegraded":"2022-04-03T19:44:26.289548+0000","last_fullsized":"2022-04-03T19:44:26.289548+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:18:21.832066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.297820+0000","last_change":"2022-04-03T19:44:14.755096+0000","last_active":"2022-04-03T19:44:21.297820+0000","last_peered":"2022-04-03T19:44:21.297820+0000","last_clean":"2022-04-03T19:44:21.297820+0000","last_became_active":"2022-04-03T19:44:14.754799+0000","last_became_peered":"2022-04-03T19:44:14.754799+0000","last_unstale":"2022-04-03T19:44:21.297820+0000","last_undegraded":"2022-04-03T19:44:21.297820+0000","last_fullsized":"2022-04-03T19:44:21.297820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:04:17.179114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.309672+0000","last_change":"2022-04-03T19:44:21.299649+0000","last_active":"2022-04-03T19:44:21.309672+0000","last_peered":"2022-04-03T19:44:21.309672+0000","last_clean":"2022-04-03T19:44:21.309672+0000","last_became_active":"2022-04-03T19:44:21.299399+0000","last_became_peered":"2022-04-03T19:44:21.299399+0000","last_unstale":"2022-04-03T19:44:21.309672+0000","last_undegraded":"2022-04-03T19:44:21.309672+0000","last_fullsized":"2022-04-03T19:44:21.309672+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:13:06.313729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.288717+0000","last_change":"2022-04-03T19:44:14.754659+0000","last_active":"2022-04-03T19:44:21.288717+0000","last_peered":"2022-04-03T19:44:21.288717+0000","last_clean":"2022-04-03T19:44:21.288717+0000","last_became_active":"2022-04-03T19:44:14.754386+0000","last_became_peered":"2022-04-03T19:44:14.754386+0000","last_unstale":"2022-04-03T19:44:21.288717+0000","last_undegraded":"2022-04-03T19:44:21.288717+0000","last_fullsized":"2022-04-03T19:44:21.288717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:33:33.013536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.283339+0000","last_change":"2022-04-03T19:44:20.784840+0000","last_active":"2022-04-03T19:44:21.283339+0000","last_peered":"2022-04-03T19:44:21.283339+0000","last_clean":"2022-04-03T19:44:21.283339+0000","last_became_active":"2022-04-03T19:44:20.784521+0000","last_became_peered":"2022-04-03T19:44:20.784521+0000","last_unstale":"2022-04-03T19:44:21.283339+0000","last_undegraded":"2022-04-03T19:44:21.283339+0000","last_fullsized":"2022-04-03T19:44:21.283339+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:15:38.240804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293467+0000","last_change":"2022-04-03T19:44:21.290769+0000","last_active":"2022-04-03T19:44:21.293467+0000","last_peered":"2022-04-03T19:44:21.293467+0000","last_clean":"2022-04-03T19:44:21.293467+0000","last_became_active":"2022-04-03T19:44:21.290394+0000","last_became_peered":"2022-04-03T19:44:21.290394+0000","last_unstale":"2022-04-03T19:44:21.293467+0000","last_undegraded":"2022-04-03T19:44:21.293467+0000","last_fullsized":"2022-04-03T19:44:21.293467+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:57:39.528871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.285708+0000","last_change":"2022-04-03T19:44:20.789662+0000","last_active":"2022-04-03T19:44:22.285708+0000","last_peered":"2022-04-03T19:44:22.285708+0000","last_clean":"2022-04-03T19:44:22.285708+0000","last_became_active":"2022-04-03T19:44:20.787868+0000","last_became_peered":"2022-04-03T19:44:20.787868+0000","last_unstale":"2022-04-03T19:44:22.285708+0000","last_undegraded":"2022-04-03T19:44:22.285708+0000","last_fullsized":"2022-04-03T19:44:22.285708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:13:16.092180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305589+0000","last_change":"2022-04-03T19:44:21.304539+0000","last_active":"2022-04-03T19:44:21.305589+0000","last_peered":"2022-04-03T19:44:21.305589+0000","last_clean":"2022-04-03T19:44:21.305589+0000","last_became_active":"2022-04-03T19:44:21.304008+0000","last_became_peered":"2022-04-03T19:44:21.304008+0000","last_unstale":"2022-04-03T19:44:21.305589+0000","last_undegraded":"2022-04-03T19:44:21.305589+0000","last_fullsized":"2022-04-03T19:44:21.305589+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:45:01.346479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.292841+0000","last_change":"2022-04-03T19:44:20.786813+0000","last_active":"2022-04-03T19:44:26.292841+0000","last_peered":"2022-04-03T19:44:26.292841+0000","last_clean":"2022-04-03T19:44:26.292841+0000","last_became_active":"2022-04-03T19:44:20.786548+0000","last_became_peered":"2022-04-03T19:44:20.786548+0000","last_unstale":"2022-04-03T19:44:26.292841+0000","last_undegraded":"2022-04-03T19:44:26.292841+0000","last_fullsized":"2022-04-03T19:44:26.292841+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:36:40.611720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302733+0000","last_change":"2022-04-03T19:44:21.299513+0000","last_active":"2022-04-03T19:44:21.302733+0000","last_peered":"2022-04-03T19:44:21.302733+0000","last_clean":"2022-04-03T19:44:21.302733+0000","last_became_active":"2022-04-03T19:44:21.299270+0000","last_became_peered":"2022-04-03T19:44:21.299270+0000","last_unstale":"2022-04-03T19:44:21.302733+0000","last_undegraded":"2022-04-03T19:44:21.302733+0000","last_fullsized":"2022-04-03T19:44:21.302733+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:54:16.302815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.290239+0000","last_change":"2022-04-03T19:44:20.786750+0000","last_active":"2022-04-03T19:44:26.290239+0000","last_peered":"2022-04-03T19:44:26.290239+0000","last_clean":"2022-04-03T19:44:26.290239+0000","last_became_active":"2022-04-03T19:44:20.786490+0000","last_became_peered":"2022-04-03T19:44:20.786490+0000","last_unstale":"2022-04-03T19:44:26.290239+0000","last_undegraded":"2022-04-03T19:44:26.290239+0000","last_fullsized":"2022-04-03T19:44:26.290239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:30:31.980806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300228+0000","last_change":"2022-04-03T19:44:21.294849+0000","last_active":"2022-04-03T19:44:21.300228+0000","last_peered":"2022-04-03T19:44:21.300228+0000","last_clean":"2022-04-03T19:44:21.300228+0000","last_became_active":"2022-04-03T19:44:21.294586+0000","last_became_peered":"2022-04-03T19:44:21.294586+0000","last_unstale":"2022-04-03T19:44:21.300228+0000","last_undegraded":"2022-04-03T19:44:21.300228+0000","last_fullsized":"2022-04-03T19:44:21.300228+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:24:11.050193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.289167+0000","last_change":"2022-04-03T19:44:20.781619+0000","last_active":"2022-04-03T19:44:22.289167+0000","last_peered":"2022-04-03T19:44:22.289167+0000","last_clean":"2022-04-03T19:44:22.289167+0000","last_became_active":"2022-04-03T19:44:20.781288+0000","last_became_peered":"2022-04-03T19:44:20.781288+0000","last_unstale":"2022-04-03T19:44:22.289167+0000","last_undegraded":"2022-04-03T19:44:22.289167+0000","last_fullsized":"2022-04-03T19:44:22.289167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:05:56.179441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305553+0000","last_change":"2022-04-03T19:44:21.296332+0000","last_active":"2022-04-03T19:44:21.305553+0000","last_peered":"2022-04-03T19:44:21.305553+0000","last_clean":"2022-04-03T19:44:21.305553+0000","last_became_active":"2022-04-03T19:44:21.296026+0000","last_became_peered":"2022-04-03T19:44:21.296026+0000","last_unstale":"2022-04-03T19:44:21.305553+0000","last_undegraded":"2022-04-03T19:44:21.305553+0000","last_fullsized":"2022-04-03T19:44:21.305553+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:43:33.822187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"24'4","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.293148+0000","last_change":"2022-04-03T19:44:20.789448+0000","last_active":"2022-04-03T19:44:26.293148+0000","last_peered":"2022-04-03T19:44:26.293148+0000","last_clean":"2022-04-03T19:44:26.293148+0000","last_became_active":"2022-04-03T19:44:20.789131+0000","last_became_peered":"2022-04-03T19:44:20.789131+0000","last_unstale":"2022-04-03T19:44:26.293148+0000","last_undegraded":"2022-04-03T19:44:26.293148+0000","last_fullsized":"2022-04-03T19:44:26.293148+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:46:20.830676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293972+0000","last_change":"2022-04-03T19:44:21.291681+0000","last_active":"2022-04-03T19:44:21.293972+0000","last_peered":"2022-04-03T19:44:21.293972+0000","last_clean":"2022-04-03T19:44:21.293972+0000","last_became_active":"2022-04-03T19:44:21.290817+0000","last_became_peered":"2022-04-03T19:44:21.290817+0000","last_unstale":"2022-04-03T19:44:21.293972+0000","last_undegraded":"2022-04-03T19:44:21.293972+0000","last_fullsized":"2022-04-03T19:44:21.293972+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:37:30.833818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300290+0000","last_change":"2022-04-03T19:44:21.295604+0000","last_active":"2022-04-03T19:44:21.300290+0000","last_peered":"2022-04-03T19:44:21.300290+0000","last_clean":"2022-04-03T19:44:21.300290+0000","last_became_active":"2022-04-03T19:44:21.295297+0000","last_became_peered":"2022-04-03T19:44:21.295297+0000","last_unstale":"2022-04-03T19:44:21.300290+0000","last_undegraded":"2022-04-03T19:44:21.300290+0000","last_fullsized":"2022-04-03T19:44:21.300290+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:26:34.341290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.309922+0000","last_change":"2022-04-03T19:44:21.306547+0000","last_active":"2022-04-03T19:44:21.309922+0000","last_peered":"2022-04-03T19:44:21.309922+0000","last_clean":"2022-04-03T19:44:21.309922+0000","last_became_active":"2022-04-03T19:44:21.306179+0000","last_became_peered":"2022-04-03T19:44:21.306179+0000","last_unstale":"2022-04-03T19:44:21.309922+0000","last_undegraded":"2022-04-03T19:44:21.309922+0000","last_fullsized":"2022-04-03T19:44:21.309922+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:34:58.854671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293642+0000","last_change":"2022-04-03T19:44:21.291341+0000","last_active":"2022-04-03T19:44:21.293642+0000","last_peered":"2022-04-03T19:44:21.293642+0000","last_clean":"2022-04-03T19:44:21.293642+0000","last_became_active":"2022-04-03T19:44:21.290943+0000","last_became_peered":"2022-04-03T19:44:21.290943+0000","last_unstale":"2022-04-03T19:44:21.293642+0000","last_undegraded":"2022-04-03T19:44:21.293642+0000","last_fullsized":"2022-04-03T19:44:21.293642+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:50:56.877454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313543+0000","last_change":"2022-04-03T19:44:21.307815+0000","last_active":"2022-04-03T19:44:21.313543+0000","last_peered":"2022-04-03T19:44:21.313543+0000","last_clean":"2022-04-03T19:44:21.313543+0000","last_became_active":"2022-04-03T19:44:21.307459+0000","last_became_peered":"2022-04-03T19:44:21.307459+0000","last_unstale":"2022-04-03T19:44:21.313543+0000","last_undegraded":"2022-04-03T19:44:21.313543+0000","last_fullsized":"2022-04-03T19:44:21.313543+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:16:52.723409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.312899+0000","last_change":"2022-04-03T19:44:21.303095+0000","last_active":"2022-04-03T19:44:21.312899+0000","last_peered":"2022-04-03T19:44:21.312899+0000","last_clean":"2022-04-03T19:44:21.312899+0000","last_became_active":"2022-04-03T19:44:21.302363+0000","last_became_peered":"2022-04-03T19:44:21.302363+0000","last_unstale":"2022-04-03T19:44:21.312899+0000","last_undegraded":"2022-04-03T19:44:21.312899+0000","last_fullsized":"2022-04-03T19:44:21.312899+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:42:45.617661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2330,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8972,"kb_used_data":772,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94362868,"statfs":{"total":96636764160,"available":96627576832,"internally_reserved":0,"allocated":790528,"data_stored":623468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90700000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]}]},{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8996,"kb_used_data":796,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94362844,"statfs":{"total":96636764160,"available":96627552256,"internally_reserved":0,"allocated":815104,"data_stored":624666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7936,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363904,"statfs":{"total":96636764160,"available":96628637696,"internally_reserved":0,"allocated":253952,"data_stored":93974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8000,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363840,"statfs":{"total":96636764160,"available":96628572160,"internally_reserved":0,"allocated":253952,"data_stored":93974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8004,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363836,"statfs":{"total":96636764160,"available":96628568064,"internally_reserved":0,"allocated":258048,"data_stored":94064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8020,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363820,"statfs":{"total":96636764160,"available":96628551680,"internally_reserved":0,"allocated":274432,"data_stored":95578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7944,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363896,"statfs":{"total":96636764160,"available":96628629504,"internally_reserved":0,"allocated":262144,"data_stored":94610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7936,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363904,"statfs":{"total":96636764160,"available":96628637696,"internally_reserved":0,"allocated":253952,"data_stored":93974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-03T19:44:32.895 INFO:tasks.ceph.ceph_manager.ceph:clean! 2022-04-03T19:44:32.895 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-04-03T19:44:32.895 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2022-04-03T19:44:32.896 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2022-04-03T19:44:33.092 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.092+0000 7f0fee70f700 1 Processor -- start 2022-04-03T19:44:33.093 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.092+0000 7f0fee70f700 1 -- start start 2022-04-03T19:44:33.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.093+0000 7f0fee70f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0fe806bbe0 0x7f0fe80c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:33.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.093+0000 7f0fee70f700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0fe8152b80 0x7f0fe8151140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:33.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.093+0000 7f0fee70f700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe8069d10 con 0x7f0fe806bfb0 2022-04-03T19:44:33.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.093+0000 7f0fee70f700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fe80c7fc0 con 0x7f0fe806bbe0 2022-04-03T19:44:33.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.093+0000 7f0fee70f700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fe806cbc0 con 0x7f0fe8152b80 2022-04-03T19:44:33.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.094+0000 7f0fed70d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0fe806bbe0 0x7f0fe80c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:33.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.094+0000 7f0fedf0e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0fe8152b80 0x7f0fe8151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:33.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.094+0000 7f0fed70d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0fe806bbe0 0x7f0fe80c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42612/0 (socket says 172.21.15.77:42612) 2022-04-03T19:44:33.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.094+0000 7f0fecf0c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0fe806bfb0 0x7f0fe806a1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38334/0 (socket says 172.21.15.77:38334) 2022-04-03T19:44:33.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.094+0000 7f0fed70d700 1 -- 172.21.15.77:0/2409190327 learned_addr learned my addr 172.21.15.77:0/2409190327 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:33.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.094+0000 7f0fed70d700 1 -- 172.21.15.77:0/2409190327 >> v2:172.21.15.186:3300/0 conn(0x7f0fe8152b80 msgr2=0x7f0fe8151140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:33.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.094+0000 7f0fed70d700 1 --2- 172.21.15.77:0/2409190327 >> v2:172.21.15.186:3300/0 conn(0x7f0fe8152b80 0x7f0fe8151140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.094+0000 7f0fed70d700 1 -- 172.21.15.77:0/2409190327 >> v1:172.21.15.77:6789/0 conn(0x7f0fe806bfb0 legacy=0x7f0fe806a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:33.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.094+0000 7f0fd7fff700 1 -- 172.21.15.77:0/2409190327 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615217412 0 0) 0x7f0fe8069d10 con 0x7f0fe806bfb0 2022-04-03T19:44:33.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.094+0000 7f0fed70d700 1 -- 172.21.15.77:0/2409190327 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe8069100 con 0x7f0fe806bbe0 2022-04-03T19:44:33.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.094+0000 7f0fed70d700 1 --2- 172.21.15.77:0/2409190327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe806bbe0 0x7f0fe80c79e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0fe80c3d20 tx=0x7f0fe40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b30c1726715a297a server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:33.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.095+0000 7f0fd7fff700 1 -- 172.21.15.77:0/2409190327 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4004030 con 0x7f0fe806bbe0 2022-04-03T19:44:33.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.095+0000 7f0fd7fff700 1 -- 172.21.15.77:0/2409190327 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe4004190 con 0x7f0fe806bbe0 2022-04-03T19:44:33.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.095+0000 7f0fd7fff700 1 -- 172.21.15.77:0/2409190327 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4011630 con 0x7f0fe806bbe0 2022-04-03T19:44:33.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.095+0000 7f0fee70f700 1 -- 172.21.15.77:0/2409190327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe806bbe0 msgr2=0x7f0fe80c79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:33.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.095+0000 7f0fee70f700 1 --2- 172.21.15.77:0/2409190327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe806bbe0 0x7f0fe80c79e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0fe80c3d20 tx=0x7f0fe40090f0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.095+0000 7f0fee70f700 1 -- 172.21.15.77:0/2409190327 shutdown_connections 2022-04-03T19:44:33.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.095+0000 7f0fee70f700 1 --2- 172.21.15.77:0/2409190327 >> v2:172.21.15.186:3300/0 conn(0x7f0fe8152b80 0x7f0fe8151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.095+0000 7f0fee70f700 1 --2- 172.21.15.77:0/2409190327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe806bbe0 0x7f0fe80c79e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.095+0000 7f0fee70f700 1 -- 172.21.15.77:0/2409190327 >> 172.21.15.77:0/2409190327 conn(0x7f0fe8067870 msgr2=0x7f0fe80c72d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:33.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.095+0000 7f0fee70f700 1 -- 172.21.15.77:0/2409190327 shutdown_connections 2022-04-03T19:44:33.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.095+0000 7f0fee70f700 1 -- 172.21.15.77:0/2409190327 wait complete. 2022-04-03T19:44:33.104 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.097+0000 7f0fee70f700 1 Processor -- start 2022-04-03T19:44:33.104 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.097+0000 7f0fee70f700 1 -- start start 2022-04-03T19:44:33.104 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.098+0000 7f0fee70f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fe806bbe0 0x7f0fe80c3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:33.104 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.098+0000 7f0fee70f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe806bfb0 0x7f0fe80abaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:33.104 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.098+0000 7f0fee70f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fe8152b80 0x7f0fe80ac030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:33.105 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.098+0000 7f0fee70f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe8151430 con 0x7f0fe806bfb0 2022-04-03T19:44:33.105 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.098+0000 7f0fee70f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe8151ef0 con 0x7f0fe806bbe0 2022-04-03T19:44:33.105 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.098+0000 7f0fee70f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe8152190 con 0x7f0fe8152b80 2022-04-03T19:44:33.105 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.098+0000 7f0fed70d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fe806bbe0 0x7f0fe80c3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:33.105 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.098+0000 7f0fecf0c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe806bfb0 0x7f0fe80abaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:33.106 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.098+0000 7f0fed70d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fe806bbe0 0x7f0fe80c3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37812/0 (socket says 172.21.15.77:37812) 2022-04-03T19:44:33.106 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.098+0000 7f0fedf0e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fe8152b80 0x7f0fe80ac030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:33.106 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.098+0000 7f0fed70d700 1 -- 172.21.15.77:0/418176476 learned_addr learned my addr 172.21.15.77:0/418176476 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:33.106 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.099+0000 7f0fed70d700 1 -- 172.21.15.77:0/418176476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fe8152b80 msgr2=0x7f0fe80ac030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:33.106 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.099+0000 7f0fed70d700 1 --2- 172.21.15.77:0/418176476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fe8152b80 0x7f0fe80ac030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.107 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.099+0000 7f0fed70d700 1 -- 172.21.15.77:0/418176476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe806bfb0 msgr2=0x7f0fe80abaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:33.107 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.099+0000 7f0fed70d700 1 --2- 172.21.15.77:0/418176476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe806bfb0 0x7f0fe80abaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.107 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.099+0000 7f0fed70d700 1 -- 172.21.15.77:0/418176476 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe4009370 con 0x7f0fe806bbe0 2022-04-03T19:44:33.107 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.099+0000 7f0fecf0c700 1 --2- 172.21.15.77:0/418176476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe806bfb0 0x7f0fe80abaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:33.107 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.099+0000 7f0fed70d700 1 --2- 172.21.15.77:0/418176476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fe806bbe0 0x7f0fe80c3250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8065630 tx=0x7f0fe4016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:33.108 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.099+0000 7f0fd57fa700 1 -- 172.21.15.77:0/418176476 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe40036f0 con 0x7f0fe806bbe0 2022-04-03T19:44:33.108 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.100+0000 7f0fee70f700 1 -- 172.21.15.77:0/418176476 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe80ab3b0 con 0x7f0fe806bbe0 2022-04-03T19:44:33.108 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.100+0000 7f0fd57fa700 1 -- 172.21.15.77:0/418176476 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe4003870 con 0x7f0fe806bbe0 2022-04-03T19:44:33.108 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.100+0000 7f0fd57fa700 1 -- 172.21.15.77:0/418176476 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4003c30 con 0x7f0fe806bbe0 2022-04-03T19:44:33.108 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.100+0000 7f0fee70f700 1 -- 172.21.15.77:0/418176476 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe8154190 con 0x7f0fe806bbe0 2022-04-03T19:44:33.109 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.101+0000 7f0fee70f700 1 -- 172.21.15.77:0/418176476 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe80ab940 con 0x7f0fe806bbe0 2022-04-03T19:44:33.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.102+0000 7f0fd57fa700 1 -- 172.21.15.77:0/418176476 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f0fe402b030 con 0x7f0fe806bbe0 2022-04-03T19:44:33.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.102+0000 7f0fd57fa700 1 --2- 172.21.15.77:0/418176476 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0fc8063d10 0x7f0fc80661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:33.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.102+0000 7f0fd57fa700 1 -- 172.21.15.77:0/418176476 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f0fe40825f0 con 0x7f0fe806bbe0 2022-04-03T19:44:33.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.103+0000 7f0fecf0c700 1 --2- 172.21.15.77:0/418176476 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0fc8063d10 0x7f0fc80661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:33.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.103+0000 7f0fecf0c700 1 --2- 172.21.15.77:0/418176476 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0fc8063d10 0x7f0fc80661c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8007fd0 tx=0x7f0fd8007bb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:33.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.106+0000 7f0fd57fa700 1 -- 172.21.15.77:0/418176476 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f0fe4051170 con 0x7f0fe806bbe0 2022-04-03T19:44:33.338 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.338+0000 7f0fee70f700 1 -- 172.21.15.77:0/418176476 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0fe80693e0 con 0x7f0fe806bbe0 2022-04-03T19:44:33.339 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.338+0000 7f0fd57fa700 1 -- 172.21.15.77:0/418176476 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f0fe4054790 con 0x7f0fe806bbe0 2022-04-03T19:44:33.339 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:33.339 INFO:teuthology.orchestra.run.smithi077.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-04-03T19:44:33.342 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.341+0000 7f0fee70f700 1 -- 172.21.15.77:0/418176476 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0fc8063d10 msgr2=0x7f0fc80661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:33.342 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.341+0000 7f0fee70f700 1 --2- 172.21.15.77:0/418176476 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0fc8063d10 0x7f0fc80661c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8007fd0 tx=0x7f0fd8007bb0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.342 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.341+0000 7f0fee70f700 1 -- 172.21.15.77:0/418176476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fe806bbe0 msgr2=0x7f0fe80c3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:33.342 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.341+0000 7f0fee70f700 1 --2- 172.21.15.77:0/418176476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fe806bbe0 0x7f0fe80c3250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8065630 tx=0x7f0fe4016000 comp rx=0 tx=0).stop 2022-04-03T19:44:33.343 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.341+0000 7f0fee70f700 1 -- 172.21.15.77:0/418176476 shutdown_connections 2022-04-03T19:44:33.343 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.341+0000 7f0fee70f700 1 --2- 172.21.15.77:0/418176476 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0fc8063d10 0x7f0fc80661c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.343 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.341+0000 7f0fee70f700 1 --2- 172.21.15.77:0/418176476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fe8152b80 0x7f0fe80ac030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.343 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.341+0000 7f0fee70f700 1 --2- 172.21.15.77:0/418176476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fe806bbe0 0x7f0fe80c3250 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.343 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.341+0000 7f0fee70f700 1 --2- 172.21.15.77:0/418176476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0fe806bfb0 0x7f0fe80abaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.344 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.341+0000 7f0fee70f700 1 -- 172.21.15.77:0/418176476 >> 172.21.15.77:0/418176476 conn(0x7f0fe8067870 msgr2=0x7f0fe81507f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:33.344 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.341+0000 7f0fee70f700 1 -- 172.21.15.77:0/418176476 shutdown_connections 2022-04-03T19:44:33.344 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.341+0000 7f0fee70f700 1 -- 172.21.15.77:0/418176476 wait complete. 2022-04-03T19:44:33.365 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2022-04-03T19:44:33.366 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-04-03T19:44:33.563 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.563+0000 7f3f411d7700 1 Processor -- start 2022-04-03T19:44:33.564 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.563+0000 7f3f411d7700 1 -- start start 2022-04-03T19:44:33.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.564+0000 7f3f411d7700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3f3c06d3f0 0x7f3f3c06d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:33.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.564+0000 7f3f411d7700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f3c0c2a90 0x7f3f3c06c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:33.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.564+0000 7f3f411d7700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3c1562f0 con 0x7f3f3c0c5930 2022-04-03T19:44:33.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.564+0000 7f3f411d7700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f3c0c2f80 con 0x7f3f3c06d3f0 2022-04-03T19:44:33.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.564+0000 7f3f411d7700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c0c3260 con 0x7f3f3c0c2a90 2022-04-03T19:44:33.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.564+0000 7f3f3bfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3f3c06d3f0 0x7f3f3c06d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:33.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.565+0000 7f3f3bfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3f3c06d3f0 0x7f3f3c06d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42616/0 (socket says 172.21.15.77:42616) 2022-04-03T19:44:33.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.565+0000 7f3f3bfff700 1 -- 172.21.15.77:0/4237483240 learned_addr learned my addr 172.21.15.77:0/4237483240 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:33.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.565+0000 7f3f409d6700 1 --2- 172.21.15.77:0/4237483240 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f3c0c2a90 0x7f3f3c06c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:33.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.565+0000 7f3f3bfff700 1 -- 172.21.15.77:0/4237483240 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f3c0c2a90 msgr2=0x7f3f3c06c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:33.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.565+0000 7f3f3bfff700 1 --2- 172.21.15.77:0/4237483240 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f3c0c2a90 0x7f3f3c06c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.565+0000 7f3f3bfff700 1 -- 172.21.15.77:0/4237483240 >> v1:172.21.15.186:6789/0 conn(0x7f3f3c0c5930 legacy=0x7f3f3c155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:33.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.565+0000 7f3f3bfff700 1 -- 172.21.15.77:0/4237483240 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c0c3540 con 0x7f3f3c06d3f0 2022-04-03T19:44:33.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.565+0000 7f3f3affd700 1 -- 172.21.15.77:0/4237483240 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110200413 0 0) 0x7f3f3c1562f0 con 0x7f3f3c0c5930 2022-04-03T19:44:33.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.565+0000 7f3f3bfff700 1 --2- 172.21.15.77:0/4237483240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f3c06d3f0 0x7f3f3c06d7c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c009830 tx=0x7f3f2c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ebf36f6dc68023e server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:33.570 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.565+0000 7f3f3affd700 1 -- 172.21.15.77:0/4237483240 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c004030 con 0x7f3f3c06d3f0 2022-04-03T19:44:33.570 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.565+0000 7f3f3affd700 1 -- 172.21.15.77:0/4237483240 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2c004190 con 0x7f3f3c06d3f0 2022-04-03T19:44:33.570 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.566+0000 7f3f3affd700 1 -- 172.21.15.77:0/4237483240 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c00b690 con 0x7f3f3c06d3f0 2022-04-03T19:44:33.570 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.566+0000 7f3f411d7700 1 -- 172.21.15.77:0/4237483240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f3c06d3f0 msgr2=0x7f3f3c06d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:33.570 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.566+0000 7f3f411d7700 1 --2- 172.21.15.77:0/4237483240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f3c06d3f0 0x7f3f3c06d7c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c009830 tx=0x7f3f2c0090f0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.571 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.566+0000 7f3f411d7700 1 -- 172.21.15.77:0/4237483240 shutdown_connections 2022-04-03T19:44:33.571 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.566+0000 7f3f411d7700 1 --2- 172.21.15.77:0/4237483240 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f3c0c2a90 0x7f3f3c06c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.571 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.566+0000 7f3f411d7700 1 --2- 172.21.15.77:0/4237483240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f3c06d3f0 0x7f3f3c06d7c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.571 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.566+0000 7f3f411d7700 1 -- 172.21.15.77:0/4237483240 >> 172.21.15.77:0/4237483240 conn(0x7f3f3c0cb720 msgr2=0x7f3f3c0cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:33.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.566+0000 7f3f411d7700 1 -- 172.21.15.77:0/4237483240 shutdown_connections 2022-04-03T19:44:33.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.566+0000 7f3f411d7700 1 -- 172.21.15.77:0/4237483240 wait complete. 2022-04-03T19:44:33.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.568+0000 7f3f411d7700 1 Processor -- start 2022-04-03T19:44:33.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.568+0000 7f3f411d7700 1 -- start start 2022-04-03T19:44:33.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.569+0000 7f3f411d7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f3c06d3f0 0x7f3f3c159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:33.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.569+0000 7f3f411d7700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f3c0c2a90 0x7f3f3c0c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:33.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.569+0000 7f3f411d7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f3c0c5930 0x7f3f3c168de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:33.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.569+0000 7f3f411d7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c0c7200 con 0x7f3f3c0c5930 2022-04-03T19:44:33.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.569+0000 7f3f411d7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c1664b0 con 0x7f3f3c06d3f0 2022-04-03T19:44:33.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.569+0000 7f3f411d7700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c150ba0 con 0x7f3f3c0c2a90 2022-04-03T19:44:33.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.569+0000 7f3f409d6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f3c0c5930 0x7f3f3c168de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:33.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.569+0000 7f3f3bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f3c06d3f0 0x7f3f3c159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:33.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.569+0000 7f3f409d6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f3c0c5930 0x7f3f3c168de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42618/0 (socket says 172.21.15.77:42618) 2022-04-03T19:44:33.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.569+0000 7f3f409d6700 1 -- 172.21.15.77:0/1169508359 learned_addr learned my addr 172.21.15.77:0/1169508359 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:33.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.569+0000 7f3f3b7fe700 1 --2- 172.21.15.77:0/1169508359 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f3c0c2a90 0x7f3f3c0c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:33.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.569+0000 7f3f3bfff700 1 -- 172.21.15.77:0/1169508359 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f3c0c2a90 msgr2=0x7f3f3c0c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:33.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.569+0000 7f3f3bfff700 1 --2- 172.21.15.77:0/1169508359 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f3c0c2a90 0x7f3f3c0c85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.569+0000 7f3f3bfff700 1 -- 172.21.15.77:0/1169508359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f3c0c5930 msgr2=0x7f3f3c168de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:33.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.569+0000 7f3f3bfff700 1 --2- 172.21.15.77:0/1169508359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f3c0c5930 0x7f3f3c168de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.579 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.569+0000 7f3f3bfff700 1 -- 172.21.15.77:0/1169508359 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2c009370 con 0x7f3f3c06d3f0 2022-04-03T19:44:33.579 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.569+0000 7f3f409d6700 1 --2- 172.21.15.77:0/1169508359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f3c0c5930 0x7f3f3c168de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:33.579 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.570+0000 7f3f3bfff700 1 --2- 172.21.15.77:0/1169508359 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f3c06d3f0 0x7f3f3c159430 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c009340 tx=0x7f3f2c016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:33.579 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.570+0000 7f3f23fff700 1 -- 172.21.15.77:0/1169508359 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c0037c0 con 0x7f3f3c06d3f0 2022-04-03T19:44:33.580 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.570+0000 7f3f23fff700 1 -- 172.21.15.77:0/1169508359 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2c003920 con 0x7f3f3c06d3f0 2022-04-03T19:44:33.580 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.570+0000 7f3f411d7700 1 -- 172.21.15.77:0/1169508359 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3c1658a0 con 0x7f3f3c06d3f0 2022-04-03T19:44:33.580 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.570+0000 7f3f23fff700 1 -- 172.21.15.77:0/1169508359 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c003d20 con 0x7f3f3c06d3f0 2022-04-03T19:44:33.580 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.570+0000 7f3f411d7700 1 -- 172.21.15.77:0/1169508359 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3c165e60 con 0x7f3f3c06d3f0 2022-04-03T19:44:33.580 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.571+0000 7f3f411d7700 1 -- 172.21.15.77:0/1169508359 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f3c04b050 con 0x7f3f3c06d3f0 2022-04-03T19:44:33.581 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.572+0000 7f3f23fff700 1 -- 172.21.15.77:0/1169508359 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f3f2c02b030 con 0x7f3f3c06d3f0 2022-04-03T19:44:33.581 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.573+0000 7f3f23fff700 1 --2- 172.21.15.77:0/1169508359 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3f1c063ce0 0x7f3f1c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:33.581 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.573+0000 7f3f23fff700 1 -- 172.21.15.77:0/1169508359 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f3f2c081ff0 con 0x7f3f3c06d3f0 2022-04-03T19:44:33.581 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.573+0000 7f3f3b7fe700 1 --2- 172.21.15.77:0/1169508359 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3f1c063ce0 0x7f3f1c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:33.581 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.574+0000 7f3f3b7fe700 1 --2- 172.21.15.77:0/1169508359 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3f1c063ce0 0x7f3f1c066190 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3f30001810 tx=0x7f3f30009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:33.582 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.577+0000 7f3f23fff700 1 -- 172.21.15.77:0/1169508359 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f3f2c050b60 con 0x7f3f3c06d3f0 2022-04-03T19:44:33.773 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.773+0000 7f3f411d7700 1 -- 172.21.15.77:0/1169508359 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f3c06d010 con 0x7f3f3c06d3f0 2022-04-03T19:44:33.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.774+0000 7f3f23fff700 1 -- 172.21.15.77:0/1169508359 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16845 (secure 0 0 0) 0x7f3f2c029090 con 0x7f3f3c06d3f0 2022-04-03T19:44:33.775 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:33.775 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"89c3a3f4-599d-4c4a-8f5a-7180eafb7292","created":"2022-04-03T19:43:54.101559+0000","modified":"2022-04-03T19:44:22.278254+0000","last_up_change":"2022-04-03T19:44:05.589174+0000","last_in_change":"2022-04-03T19:43:56.729760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-03T19:44:07.567638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-04-03T19:44:13.149481+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-04-03T19:44:18.823926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-04-03T19:44:20.275479+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f35f5195-3729-4d78-85d1-0539f259283f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6823","nonce":20252},{"type":"v1","addr":"172.21.15.77:6825","nonce":20252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":20252},{"type":"v1","addr":"172.21.15.77:6827","nonce":20252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":20252},{"type":"v1","addr":"172.21.15.77:6831","nonce":20252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":20252},{"type":"v1","addr":"172.21.15.77:6829","nonce":20252}]},"public_addr":"172.21.15.77:6825/20252","cluster_addr":"172.21.15.77:6827/20252","heartbeat_back_addr":"172.21.15.77:6831/20252","heartbeat_front_addr":"172.21.15.77:6829/20252","state":["exists","up"]},{"osd":1,"uuid":"ab1c89dc-57ce-4cc4-b8c7-9b0c19939601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":20249},{"type":"v1","addr":"172.21.15.77:6805","nonce":20249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":20249},{"type":"v1","addr":"172.21.15.77:6809","nonce":20249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":20249},{"type":"v1","addr":"172.21.15.77:6819","nonce":20249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":20249},{"type":"v1","addr":"172.21.15.77:6815","nonce":20249}]},"public_addr":"172.21.15.77:6805/20249","cluster_addr":"172.21.15.77:6809/20249","heartbeat_back_addr":"172.21.15.77:6819/20249","heartbeat_front_addr":"172.21.15.77:6815/20249","state":["exists","up"]},{"osd":2,"uuid":"351e218b-7749-4ee9-b44a-0a5b52fb9be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":20248},{"type":"v1","addr":"172.21.15.77:6801","nonce":20248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":20248},{"type":"v1","addr":"172.21.15.77:6803","nonce":20248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":20248},{"type":"v1","addr":"172.21.15.77:6813","nonce":20248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":20248},{"type":"v1","addr":"172.21.15.77:6808","nonce":20248}]},"public_addr":"172.21.15.77:6801/20248","cluster_addr":"172.21.15.77:6803/20248","heartbeat_back_addr":"172.21.15.77:6813/20248","heartbeat_front_addr":"172.21.15.77:6808/20248","state":["exists","up"]},{"osd":3,"uuid":"1ba6ec8a-7cbf-4c12-8e21-c3097a263ae8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":20250},{"type":"v1","addr":"172.21.15.77:6814","nonce":20250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":20250},{"type":"v1","addr":"172.21.15.77:6818","nonce":20250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":20250},{"type":"v1","addr":"172.21.15.77:6824","nonce":20250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":20250},{"type":"v1","addr":"172.21.15.77:6821","nonce":20250}]},"public_addr":"172.21.15.77:6814/20250","cluster_addr":"172.21.15.77:6818/20250","heartbeat_back_addr":"172.21.15.77:6824/20250","heartbeat_front_addr":"172.21.15.77:6821/20250","state":["exists","up"]},{"osd":4,"uuid":"3faf4c09-846c-4133-856b-7f53c535a67d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":25551},{"type":"v1","addr":"172.21.15.186:6822","nonce":25551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":25551},{"type":"v1","addr":"172.21.15.186:6826","nonce":25551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":25551},{"type":"v1","addr":"172.21.15.186:6831","nonce":25551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":25551},{"type":"v1","addr":"172.21.15.186:6829","nonce":25551}]},"public_addr":"172.21.15.186:6822/25551","cluster_addr":"172.21.15.186:6826/25551","heartbeat_back_addr":"172.21.15.186:6831/25551","heartbeat_front_addr":"172.21.15.186:6829/25551","state":["exists","up"]},{"osd":5,"uuid":"ffe41d00-d0aa-42f4-82c1-5cced13c1bc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":25541},{"type":"v1","addr":"172.21.15.186:6809","nonce":25541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":25541},{"type":"v1","addr":"172.21.15.186:6811","nonce":25541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":25541},{"type":"v1","addr":"172.21.15.186:6815","nonce":25541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":25541},{"type":"v1","addr":"172.21.15.186:6813","nonce":25541}]},"public_addr":"172.21.15.186:6809/25541","cluster_addr":"172.21.15.186:6811/25541","heartbeat_back_addr":"172.21.15.186:6815/25541","heartbeat_front_addr":"172.21.15.186:6813/25541","state":["exists","up"]},{"osd":6,"uuid":"b0344ee8-deca-4e9b-934c-5866d90d0073","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":25542},{"type":"v1","addr":"172.21.15.186:6801","nonce":25542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":25542},{"type":"v1","addr":"172.21.15.186:6803","nonce":25542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":25542},{"type":"v1","addr":"172.21.15.186:6807","nonce":25542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":25542},{"type":"v1","addr":"172.21.15.186:6805","nonce":25542}]},"public_addr":"172.21.15.186:6801/25542","cluster_addr":"172.21.15.186:6803/25542","heartbeat_back_addr":"172.21.15.186:6807/25542","heartbeat_front_addr":"172.21.15.186:6805/25542","state":["exists","up"]},{"osd":7,"uuid":"0eb0e5d3-3dad-4f49-9b50-3b24a3924bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":25572},{"type":"v1","addr":"172.21.15.186:6817","nonce":25572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":25572},{"type":"v1","addr":"172.21.15.186:6819","nonce":25572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6825","nonce":25572},{"type":"v1","addr":"172.21.15.186:6827","nonce":25572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":25572},{"type":"v1","addr":"172.21.15.186:6823","nonce":25572}]},"public_addr":"172.21.15.186:6817/25572","cluster_addr":"172.21.15.186:6819/25572","heartbeat_back_addr":"172.21.15.186:6827/25572","heartbeat_front_addr":"172.21.15.186:6823/25572","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.143602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.490366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.361292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.343544+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.905429+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969298+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.949524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-03T19:44:33.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.777+0000 7f3f411d7700 1 -- 172.21.15.77:0/1169508359 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3f1c063ce0 msgr2=0x7f3f1c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:33.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.777+0000 7f3f411d7700 1 --2- 172.21.15.77:0/1169508359 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3f1c063ce0 0x7f3f1c066190 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3f30001810 tx=0x7f3f30009040 comp rx=0 tx=0).stop 2022-04-03T19:44:33.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.777+0000 7f3f411d7700 1 -- 172.21.15.77:0/1169508359 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f3c06d3f0 msgr2=0x7f3f3c159430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:33.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.777+0000 7f3f411d7700 1 --2- 172.21.15.77:0/1169508359 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f3c06d3f0 0x7f3f3c159430 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c009340 tx=0x7f3f2c016000 comp rx=0 tx=0).stop 2022-04-03T19:44:33.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.777+0000 7f3f411d7700 1 -- 172.21.15.77:0/1169508359 shutdown_connections 2022-04-03T19:44:33.779 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.777+0000 7f3f411d7700 1 --2- 172.21.15.77:0/1169508359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3f3c0c5930 0x7f3f3c168de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.779 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.777+0000 7f3f411d7700 1 --2- 172.21.15.77:0/1169508359 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f3c06d3f0 0x7f3f3c159430 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.779 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.777+0000 7f3f411d7700 1 --2- 172.21.15.77:0/1169508359 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3f1c063ce0 0x7f3f1c066190 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.779 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.777+0000 7f3f411d7700 1 --2- 172.21.15.77:0/1169508359 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f3c0c2a90 0x7f3f3c0c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:33.779 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.777+0000 7f3f411d7700 1 -- 172.21.15.77:0/1169508359 >> 172.21.15.77:0/1169508359 conn(0x7f3f3c0cb720 msgr2=0x7f3f3c154d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:33.780 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.777+0000 7f3f411d7700 1 -- 172.21.15.77:0/1169508359 shutdown_connections 2022-04-03T19:44:33.780 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:33.777+0000 7f3f411d7700 1 -- 172.21.15.77:0/1169508359 wait complete. 2022-04-03T19:44:33.801 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-04-03T19:44:07.567638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-04-03T19:44:13.149481+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-04-03T19:44:18.823926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-04-03T19:44:20.275479+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-04-03T19:44:33.802 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-04-03T19:44:34.001 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.001+0000 7f5bfffff700 1 Processor -- start 2022-04-03T19:44:34.001 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.001+0000 7f5bfffff700 1 -- start start 2022-04-03T19:44:34.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.002+0000 7f5bfffff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5c0014f460 0x7f5c000c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:34.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.002+0000 7f5bfffff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c000c7e80 con 0x7f5c000c52b0 2022-04-03T19:44:34.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.002+0000 7f5bfffff700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c000c6ea0 con 0x7f5c000c5680 2022-04-03T19:44:34.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.002+0000 7f5bfffff700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c000c4260 con 0x7f5c0014f460 2022-04-03T19:44:34.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.002+0000 7f5bfeffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5c000c52b0 0x7f5c000c8310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38338/0 (socket says 172.21.15.77:38338) 2022-04-03T19:44:34.004 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.002+0000 7f5bfeffd700 1 -- 172.21.15.77:0/3456038304 learned_addr learned my addr 172.21.15.77:0/3456038304 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:34.004 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.002+0000 7f5bff7fe700 1 --2- 172.21.15.77:0/3456038304 >> v2:172.21.15.186:3300/0 conn(0x7f5c0014f460 0x7f5c000c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:34.006 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.003+0000 7f5bfdffb700 1 -- 172.21.15.77:0/3456038304 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275493568 0 0) 0x7f5c000c7e80 con 0x7f5c000c52b0 2022-04-03T19:44:34.006 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.003+0000 7f5bfdffb700 1 -- 172.21.15.77:0/3456038304 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bec0033f0 con 0x7f5c000c52b0 2022-04-03T19:44:34.006 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.003+0000 7f5bfdffb700 1 -- 172.21.15.77:0/3456038304 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416846732 0 0) 0x7f5c000c6ea0 con 0x7f5c000c5680 2022-04-03T19:44:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.003+0000 7f5bff7fe700 1 -- 172.21.15.77:0/3456038304 >> v1:172.21.15.186:6789/0 conn(0x7f5c000c5680 legacy=0x7f5c0014ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.003+0000 7f5bff7fe700 1 -- 172.21.15.77:0/3456038304 >> v1:172.21.15.77:6789/0 conn(0x7f5c000c52b0 legacy=0x7f5c000c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.003+0000 7f5bff7fe700 1 -- 172.21.15.77:0/3456038304 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c000c6600 con 0x7f5c0014f460 2022-04-03T19:44:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.003+0000 7f5bff7fe700 1 --2- 172.21.15.77:0/3456038304 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c0014f460 0x7f5c000c5f70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009e60 tx=0x7f5bf8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4494fb4c0fdc181 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:34.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.003+0000 7f5bfdffb700 1 -- 172.21.15.77:0/3456038304 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf800e070 con 0x7f5c0014f460 2022-04-03T19:44:34.008 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.003+0000 7f5bfdffb700 1 -- 172.21.15.77:0/3456038304 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf800bbf0 con 0x7f5c0014f460 2022-04-03T19:44:34.008 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.003+0000 7f5bfdffb700 1 -- 172.21.15.77:0/3456038304 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf8012430 con 0x7f5c0014f460 2022-04-03T19:44:34.008 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.004+0000 7f5bfffff700 1 -- 172.21.15.77:0/3456038304 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c0014f460 msgr2=0x7f5c000c5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:34.008 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.004+0000 7f5bfffff700 1 --2- 172.21.15.77:0/3456038304 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c0014f460 0x7f5c000c5f70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009e60 tx=0x7f5bf8009780 comp rx=0 tx=0).stop 2022-04-03T19:44:34.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.004+0000 7f5bfffff700 1 -- 172.21.15.77:0/3456038304 shutdown_connections 2022-04-03T19:44:34.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.004+0000 7f5bfffff700 1 --2- 172.21.15.77:0/3456038304 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c0014f460 0x7f5c000c5f70 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.004+0000 7f5bfffff700 1 -- 172.21.15.77:0/3456038304 >> 172.21.15.77:0/3456038304 conn(0x7f5c000626d0 msgr2=0x7f5c00062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:34.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.004+0000 7f5bfffff700 1 -- 172.21.15.77:0/3456038304 shutdown_connections 2022-04-03T19:44:34.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.004+0000 7f5bfffff700 1 -- 172.21.15.77:0/3456038304 wait complete. 2022-04-03T19:44:34.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.005+0000 7f5bfffff700 1 Processor -- start 2022-04-03T19:44:34.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.005+0000 7f5bfffff700 1 -- start start 2022-04-03T19:44:34.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.006+0000 7f5bfffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c000c52b0 0x7f5c00164b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:34.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.006+0000 7f5bfffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c000c5680 0x7f5c00148250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:34.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.006+0000 7f5bfffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c0014f460 0x7f5c00148790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:34.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.006+0000 7f5bfffff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c00149080 con 0x7f5c000c5680 2022-04-03T19:44:34.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.006+0000 7f5bfffff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c00149270 con 0x7f5c000c52b0 2022-04-03T19:44:34.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.006+0000 7f5bfffff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c00149550 con 0x7f5c0014f460 2022-04-03T19:44:34.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.007+0000 7f5bfe7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c000c5680 0x7f5c00148250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:34.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.007+0000 7f5bfe7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c000c5680 0x7f5c00148250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42620/0 (socket says 172.21.15.77:42620) 2022-04-03T19:44:34.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.007+0000 7f5bfe7fc700 1 -- 172.21.15.77:0/30960169 learned_addr learned my addr 172.21.15.77:0/30960169 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:34.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.007+0000 7f5bff7fe700 1 --2- 172.21.15.77:0/30960169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c0014f460 0x7f5c00148790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:34.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.007+0000 7f5bfeffd700 1 --2- 172.21.15.77:0/30960169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c000c52b0 0x7f5c00164b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:34.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.007+0000 7f5bfe7fc700 1 -- 172.21.15.77:0/30960169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c0014f460 msgr2=0x7f5c00148790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:34.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.007+0000 7f5bfe7fc700 1 --2- 172.21.15.77:0/30960169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c0014f460 0x7f5c00148790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.007+0000 7f5bfe7fc700 1 -- 172.21.15.77:0/30960169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c000c52b0 msgr2=0x7f5c00164b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:34.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.007+0000 7f5bfe7fc700 1 --2- 172.21.15.77:0/30960169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c000c52b0 0x7f5c00164b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.007+0000 7f5bfe7fc700 1 -- 172.21.15.77:0/30960169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf8009a20 con 0x7f5c000c5680 2022-04-03T19:44:34.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.007+0000 7f5bfe7fc700 1 --2- 172.21.15.77:0/30960169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c000c5680 0x7f5c00148250 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400c4c0 tx=0x7f5bf4007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:34.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.008+0000 7f5be6ffd700 1 -- 172.21.15.77:0/30960169 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf400ce70 con 0x7f5c000c5680 2022-04-03T19:44:34.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.008+0000 7f5be6ffd700 1 -- 172.21.15.77:0/30960169 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf40052e0 con 0x7f5c000c5680 2022-04-03T19:44:34.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.008+0000 7f5bfffff700 1 -- 172.21.15.77:0/30960169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c00149830 con 0x7f5c000c5680 2022-04-03T19:44:34.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.008+0000 7f5be6ffd700 1 -- 172.21.15.77:0/30960169 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4003730 con 0x7f5c000c5680 2022-04-03T19:44:34.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.008+0000 7f5bfffff700 1 -- 172.21.15.77:0/30960169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c00149df0 con 0x7f5c000c5680 2022-04-03T19:44:34.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.009+0000 7f5bfffff700 1 -- 172.21.15.77:0/30960169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c00088e10 con 0x7f5c000c5680 2022-04-03T19:44:34.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.009+0000 7f5be6ffd700 1 -- 172.21.15.77:0/30960169 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f5bf4019030 con 0x7f5c000c5680 2022-04-03T19:44:34.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.010+0000 7f5be6ffd700 1 --2- 172.21.15.77:0/30960169 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5be0063d10 0x7f5be00661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:34.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.010+0000 7f5be6ffd700 1 -- 172.21.15.77:0/30960169 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f5bf407f9c0 con 0x7f5c000c5680 2022-04-03T19:44:34.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.011+0000 7f5bfeffd700 1 --2- 172.21.15.77:0/30960169 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5be0063d10 0x7f5be00661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:34.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.011+0000 7f5bfeffd700 1 --2- 172.21.15.77:0/30960169 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5be0063d10 0x7f5be00661c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0005750 tx=0x7f5bf0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:34.019 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.014+0000 7f5be6ffd700 1 -- 172.21.15.77:0/30960169 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f5bf404e540 con 0x7f5c000c5680 2022-04-03T19:44:34.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.209+0000 7f5bfffff700 1 -- 172.21.15.77:0/30960169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5c00156960 con 0x7f5c000c5680 2022-04-03T19:44:34.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.210+0000 7f5be6ffd700 1 -- 172.21.15.77:0/30960169 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5bf4017070 con 0x7f5c000c5680 2022-04-03T19:44:34.211 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2022-04-03T19:44:34.213 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.212+0000 7f5bfffff700 1 -- 172.21.15.77:0/30960169 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5be0063d10 msgr2=0x7f5be00661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:34.213 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.212+0000 7f5bfffff700 1 --2- 172.21.15.77:0/30960169 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5be0063d10 0x7f5be00661c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0005750 tx=0x7f5bf0009040 comp rx=0 tx=0).stop 2022-04-03T19:44:34.213 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.212+0000 7f5bfffff700 1 -- 172.21.15.77:0/30960169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c000c5680 msgr2=0x7f5c00148250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:34.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.212+0000 7f5bfffff700 1 --2- 172.21.15.77:0/30960169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c000c5680 0x7f5c00148250 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400c4c0 tx=0x7f5bf4007c70 comp rx=0 tx=0).stop 2022-04-03T19:44:34.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.212+0000 7f5bfffff700 1 -- 172.21.15.77:0/30960169 shutdown_connections 2022-04-03T19:44:34.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.212+0000 7f5bfffff700 1 --2- 172.21.15.77:0/30960169 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5be0063d10 0x7f5be00661c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.212+0000 7f5bfffff700 1 --2- 172.21.15.77:0/30960169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c0014f460 0x7f5c00148790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.212+0000 7f5bfffff700 1 --2- 172.21.15.77:0/30960169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c000c52b0 0x7f5c00164b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.212+0000 7f5bfffff700 1 --2- 172.21.15.77:0/30960169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c000c5680 0x7f5c00148250 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.212+0000 7f5bfffff700 1 -- 172.21.15.77:0/30960169 >> 172.21.15.77:0/30960169 conn(0x7f5c000626d0 msgr2=0x7f5c0014e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:34.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.212+0000 7f5bfffff700 1 -- 172.21.15.77:0/30960169 shutdown_connections 2022-04-03T19:44:34.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.213+0000 7f5bfffff700 1 -- 172.21.15.77:0/30960169 wait complete. 2022-04-03T19:44:34.237 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-04-03T19:44:34.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.436+0000 7fb33d1fe700 1 Processor -- start 2022-04-03T19:44:34.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.436+0000 7fb33d1fe700 1 -- start start 2022-04-03T19:44:34.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.437+0000 7fb33d1fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3380c52b0 0x7fb3380c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:34.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.437+0000 7fb33d1fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb3380c5680 0x7fb338158e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:34.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.437+0000 7fb33d1fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb3380c8310 0x7fb33814ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:34.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.437+0000 7fb33d1fe700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3380c4260 con 0x7fb3380c5680 2022-04-03T19:44:34.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.437+0000 7fb33d1fe700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3380c7e80 con 0x7fb3380c8310 2022-04-03T19:44:34.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.437+0000 7fb33d1fe700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3380c6ea0 con 0x7fb3380c52b0 2022-04-03T19:44:34.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.438+0000 7fb33c9fd700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb3380c8310 0x7fb33814ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:34.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.438+0000 7fb337fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3380c52b0 0x7fb3380c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:34.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.438+0000 7fb32ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb3380c5680 0x7fb338158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:34.441 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.438+0000 7fb33c9fd700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb3380c8310 0x7fb33814ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37820/0 (socket says 172.21.15.77:37820) 2022-04-03T19:44:34.441 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.438+0000 7fb33c9fd700 1 -- 172.21.15.77:0/1500866498 learned_addr learned my addr 172.21.15.77:0/1500866498 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:34.441 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.438+0000 7fb33c9fd700 1 -- 172.21.15.77:0/1500866498 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3380c52b0 msgr2=0x7fb3380c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:34.441 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.438+0000 7fb33c9fd700 1 --2- 172.21.15.77:0/1500866498 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3380c52b0 0x7fb3380c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.441 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.438+0000 7fb33c9fd700 1 -- 172.21.15.77:0/1500866498 >> v2:172.21.15.77:3300/0 conn(0x7fb3380c5680 msgr2=0x7fb338158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:34.442 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.438+0000 7fb33c9fd700 1 --2- 172.21.15.77:0/1500866498 >> v2:172.21.15.77:3300/0 conn(0x7fb3380c5680 0x7fb338158e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.442 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.438+0000 7fb33c9fd700 1 -- 172.21.15.77:0/1500866498 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3380c6600 con 0x7fb3380c8310 2022-04-03T19:44:34.442 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.438+0000 7fb33c9fd700 1 --2- 172.21.15.77:0/1500866498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3380c8310 0x7fb33814ed50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb3380cc780 tx=0x7fb330009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85ba8cb91df1ad31 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:34.446 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.439+0000 7fb3377fe700 1 -- 172.21.15.77:0/1500866498 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33000e070 con 0x7fb3380c8310 2022-04-03T19:44:34.446 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.439+0000 7fb3377fe700 1 -- 172.21.15.77:0/1500866498 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33000ba00 con 0x7fb3380c8310 2022-04-03T19:44:34.446 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.439+0000 7fb3377fe700 1 -- 172.21.15.77:0/1500866498 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3300092e0 con 0x7fb3380c8310 2022-04-03T19:44:34.446 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.439+0000 7fb33d1fe700 1 -- 172.21.15.77:0/1500866498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3380c8310 msgr2=0x7fb33814ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:34.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.439+0000 7fb33d1fe700 1 --2- 172.21.15.77:0/1500866498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3380c8310 0x7fb33814ed50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb3380cc780 tx=0x7fb330009840 comp rx=0 tx=0).stop 2022-04-03T19:44:34.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.439+0000 7fb33d1fe700 1 -- 172.21.15.77:0/1500866498 shutdown_connections 2022-04-03T19:44:34.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.439+0000 7fb33d1fe700 1 --2- 172.21.15.77:0/1500866498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3380c8310 0x7fb33814ed50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.439+0000 7fb33d1fe700 1 --2- 172.21.15.77:0/1500866498 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3380c52b0 0x7fb3380c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.439+0000 7fb33d1fe700 1 --2- 172.21.15.77:0/1500866498 >> v2:172.21.15.77:3300/0 conn(0x7fb3380c5680 0x7fb338158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.439+0000 7fb33d1fe700 1 -- 172.21.15.77:0/1500866498 >> 172.21.15.77:0/1500866498 conn(0x7fb3380626d0 msgr2=0x7fb338062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:34.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.439+0000 7fb33d1fe700 1 -- 172.21.15.77:0/1500866498 shutdown_connections 2022-04-03T19:44:34.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.439+0000 7fb33d1fe700 1 -- 172.21.15.77:0/1500866498 wait complete. 2022-04-03T19:44:34.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.441+0000 7fb33d1fe700 1 Processor -- start 2022-04-03T19:44:34.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.441+0000 7fb33d1fe700 1 -- start start 2022-04-03T19:44:34.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.442+0000 7fb33d1fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3380c52b0 0x7fb338151aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:34.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.442+0000 7fb33d1fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3380c5680 0x7fb33814f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:34.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.442+0000 7fb33d1fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3380c8310 0x7fb33814fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:34.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.442+0000 7fb33d1fe700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb338151510 con 0x7fb3380c5680 2022-04-03T19:44:34.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.442+0000 7fb33d1fe700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3381682d0 con 0x7fb3380c52b0 2022-04-03T19:44:34.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.442+0000 7fb33d1fe700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb338168590 con 0x7fb3380c8310 2022-04-03T19:44:34.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.442+0000 7fb32ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3380c5680 0x7fb33814f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:34.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.442+0000 7fb32ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3380c5680 0x7fb33814f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42624/0 (socket says 172.21.15.77:42624) 2022-04-03T19:44:34.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.442+0000 7fb32ffff700 1 -- 172.21.15.77:0/985569080 learned_addr learned my addr 172.21.15.77:0/985569080 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:34.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.442+0000 7fb33c9fd700 1 --2- 172.21.15.77:0/985569080 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3380c8310 0x7fb33814fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:34.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.442+0000 7fb337fff700 1 --2- 172.21.15.77:0/985569080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3380c52b0 0x7fb338151aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:34.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.443+0000 7fb32ffff700 1 -- 172.21.15.77:0/985569080 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3380c8310 msgr2=0x7fb33814fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:34.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.443+0000 7fb32ffff700 1 --2- 172.21.15.77:0/985569080 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3380c8310 0x7fb33814fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.452 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.443+0000 7fb32ffff700 1 -- 172.21.15.77:0/985569080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3380c52b0 msgr2=0x7fb338151aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:34.452 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.443+0000 7fb32ffff700 1 --2- 172.21.15.77:0/985569080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3380c52b0 0x7fb338151aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.452 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.443+0000 7fb32ffff700 1 -- 172.21.15.77:0/985569080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb330009ae0 con 0x7fb3380c5680 2022-04-03T19:44:34.452 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.443+0000 7fb32ffff700 1 --2- 172.21.15.77:0/985569080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3380c5680 0x7fb33814f650 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb32000e900 tx=0x7fb320009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:34.453 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.443+0000 7fb334ff9700 1 -- 172.21.15.77:0/985569080 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb320013680 con 0x7fb3380c5680 2022-04-03T19:44:34.453 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.443+0000 7fb334ff9700 1 -- 172.21.15.77:0/985569080 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3200137e0 con 0x7fb3380c5680 2022-04-03T19:44:34.453 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.443+0000 7fb33d1fe700 1 -- 172.21.15.77:0/985569080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb338168870 con 0x7fb3380c5680 2022-04-03T19:44:34.453 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.443+0000 7fb334ff9700 1 -- 172.21.15.77:0/985569080 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb320013b60 con 0x7fb3380c5680 2022-04-03T19:44:34.453 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.444+0000 7fb33d1fe700 1 -- 172.21.15.77:0/985569080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb338168e30 con 0x7fb3380c5680 2022-04-03T19:44:34.454 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.444+0000 7fb33d1fe700 1 -- 172.21.15.77:0/985569080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb338088e10 con 0x7fb3380c5680 2022-04-03T19:44:34.454 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.445+0000 7fb334ff9700 1 -- 172.21.15.77:0/985569080 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fb320019030 con 0x7fb3380c5680 2022-04-03T19:44:34.455 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.446+0000 7fb334ff9700 1 --2- 172.21.15.77:0/985569080 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb31c063ce0 0x7fb31c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:34.455 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.446+0000 7fb334ff9700 1 -- 172.21.15.77:0/985569080 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fb320080920 con 0x7fb3380c5680 2022-04-03T19:44:34.455 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.447+0000 7fb337fff700 1 --2- 172.21.15.77:0/985569080 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb31c063ce0 0x7fb31c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:34.456 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.447+0000 7fb337fff700 1 --2- 172.21.15.77:0/985569080 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb31c063ce0 0x7fb31c066190 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb328009c30 tx=0x7fb328009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:34.456 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.451+0000 7fb334ff9700 1 -- 172.21.15.77:0/985569080 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fb32004f410 con 0x7fb3380c5680 2022-04-03T19:44:34.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.645+0000 7fb33d1fe700 1 -- 172.21.15.77:0/985569080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb338158680 con 0x7fb3380c5680 2022-04-03T19:44:34.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.646+0000 7fb334ff9700 1 -- 172.21.15.77:0/985569080 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb3200210a0 con 0x7fb3380c5680 2022-04-03T19:44:34.647 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2022-04-03T19:44:34.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.648+0000 7fb33d1fe700 1 -- 172.21.15.77:0/985569080 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb31c063ce0 msgr2=0x7fb31c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:34.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.648+0000 7fb33d1fe700 1 --2- 172.21.15.77:0/985569080 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb31c063ce0 0x7fb31c066190 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb328009c30 tx=0x7fb328009060 comp rx=0 tx=0).stop 2022-04-03T19:44:34.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.648+0000 7fb33d1fe700 1 -- 172.21.15.77:0/985569080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3380c5680 msgr2=0x7fb33814f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:34.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.648+0000 7fb33d1fe700 1 --2- 172.21.15.77:0/985569080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3380c5680 0x7fb33814f650 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb32000e900 tx=0x7fb320009060 comp rx=0 tx=0).stop 2022-04-03T19:44:34.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.648+0000 7fb33d1fe700 1 -- 172.21.15.77:0/985569080 shutdown_connections 2022-04-03T19:44:34.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.648+0000 7fb33d1fe700 1 --2- 172.21.15.77:0/985569080 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb31c063ce0 0x7fb31c066190 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.648+0000 7fb33d1fe700 1 --2- 172.21.15.77:0/985569080 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3380c8310 0x7fb33814fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.648+0000 7fb33d1fe700 1 --2- 172.21.15.77:0/985569080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3380c52b0 0x7fb338151aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.648+0000 7fb33d1fe700 1 --2- 172.21.15.77:0/985569080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3380c5680 0x7fb33814f650 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.648+0000 7fb33d1fe700 1 -- 172.21.15.77:0/985569080 >> 172.21.15.77:0/985569080 conn(0x7fb3380626d0 msgr2=0x7fb33814bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:34.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.648+0000 7fb33d1fe700 1 -- 172.21.15.77:0/985569080 shutdown_connections 2022-04-03T19:44:34.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.648+0000 7fb33d1fe700 1 -- 172.21.15.77:0/985569080 wait complete. 2022-04-03T19:44:34.673 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-04-03T19:44:34.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.871+0000 7f86be0c8700 1 Processor -- start 2022-04-03T19:44:34.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.872+0000 7f86be0c8700 1 -- start start 2022-04-03T19:44:34.874 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.873+0000 7f86be0c8700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f86b806bdb0 0x7f86b81559d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:34.874 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.873+0000 7f86be0c8700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b80c5170 0x7f86b80c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:34.875 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.873+0000 7f86be0c8700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b806df20 con 0x7f86b814ef20 2022-04-03T19:44:34.875 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.873+0000 7f86be0c8700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86b8155f10 con 0x7f86b806bdb0 2022-04-03T19:44:34.875 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.873+0000 7f86be0c8700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b80c6690 con 0x7f86b80c5170 2022-04-03T19:44:34.875 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.873+0000 7f86bd8c7700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b80c5170 0x7f86b80c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:34.875 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.873+0000 7f86bc8c5700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f86b806bdb0 0x7f86b81559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:34.876 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.873+0000 7f86bd0c6700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f86b814ef20 0x7f86b80c5f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44138/0 (socket says 172.21.15.77:44138) 2022-04-03T19:44:34.876 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.873+0000 7f86bd0c6700 1 -- 172.21.15.77:0/1907973196 learned_addr learned my addr 172.21.15.77:0/1907973196 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:34.876 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.874+0000 7f86a7fff700 1 -- 172.21.15.77:0/1907973196 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501487909 0 0) 0x7f86b806df20 con 0x7f86b814ef20 2022-04-03T19:44:34.876 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.874+0000 7f86a7fff700 1 -- 172.21.15.77:0/1907973196 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86a00033f0 con 0x7f86b814ef20 2022-04-03T19:44:34.876 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.874+0000 7f86bd8c7700 1 -- 172.21.15.77:0/1907973196 >> v2:172.21.15.186:3300/0 conn(0x7f86b806bdb0 msgr2=0x7f86b81559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:34.877 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.874+0000 7f86bd8c7700 1 --2- 172.21.15.77:0/1907973196 >> v2:172.21.15.186:3300/0 conn(0x7f86b806bdb0 0x7f86b81559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.877 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.874+0000 7f86bd8c7700 1 -- 172.21.15.77:0/1907973196 >> v1:172.21.15.186:6789/0 conn(0x7f86b814ef20 legacy=0x7f86b80c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:34.877 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.874+0000 7f86bd8c7700 1 -- 172.21.15.77:0/1907973196 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b806cd90 con 0x7f86b80c5170 2022-04-03T19:44:34.877 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.874+0000 7f86bd8c7700 1 --2- 172.21.15.77:0/1907973196 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b80c5170 0x7f86b80c55f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f86b0009e60 tx=0x7f86b0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=108baac1c0c0f4b4 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:34.881 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.874+0000 7f86a7fff700 1 -- 172.21.15.77:0/1907973196 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b000e070 con 0x7f86b80c5170 2022-04-03T19:44:34.881 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.874+0000 7f86a7fff700 1 -- 172.21.15.77:0/1907973196 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b000bb40 con 0x7f86b80c5170 2022-04-03T19:44:34.881 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.874+0000 7f86a7fff700 1 -- 172.21.15.77:0/1907973196 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b0012430 con 0x7f86b80c5170 2022-04-03T19:44:34.881 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.875+0000 7f86be0c8700 1 -- 172.21.15.77:0/1907973196 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b80c5170 msgr2=0x7f86b80c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:34.882 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.875+0000 7f86be0c8700 1 --2- 172.21.15.77:0/1907973196 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b80c5170 0x7f86b80c55f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f86b0009e60 tx=0x7f86b0009780 comp rx=0 tx=0).stop 2022-04-03T19:44:34.882 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.875+0000 7f86be0c8700 1 -- 172.21.15.77:0/1907973196 shutdown_connections 2022-04-03T19:44:34.882 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.875+0000 7f86be0c8700 1 --2- 172.21.15.77:0/1907973196 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b80c5170 0x7f86b80c55f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.882 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.875+0000 7f86be0c8700 1 --2- 172.21.15.77:0/1907973196 >> v2:172.21.15.186:3300/0 conn(0x7f86b806bdb0 0x7f86b81559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.882 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.875+0000 7f86be0c8700 1 -- 172.21.15.77:0/1907973196 >> 172.21.15.77:0/1907973196 conn(0x7f86b80cb6d0 msgr2=0x7f86b80cbad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:34.883 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.875+0000 7f86be0c8700 1 -- 172.21.15.77:0/1907973196 shutdown_connections 2022-04-03T19:44:34.883 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.875+0000 7f86be0c8700 1 -- 172.21.15.77:0/1907973196 wait complete. 2022-04-03T19:44:34.883 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.876+0000 7f86be0c8700 1 Processor -- start 2022-04-03T19:44:34.883 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.876+0000 7f86be0c8700 1 -- start start 2022-04-03T19:44:34.883 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.877+0000 7f86be0c8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86b806bdb0 0x7f86b8150f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:34.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.877+0000 7f86be0c8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86b80c5170 0x7f86b80c1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:34.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.877+0000 7f86be0c8700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b814ef20 0x7f86b80c2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:34.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.877+0000 7f86be0c8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b8168470 con 0x7f86b806bdb0 2022-04-03T19:44:34.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.877+0000 7f86be0c8700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b8168750 con 0x7f86b80c5170 2022-04-03T19:44:34.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.877+0000 7f86be0c8700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b8168a30 con 0x7f86b814ef20 2022-04-03T19:44:34.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.878+0000 7f86bd0c6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86b806bdb0 0x7f86b8150f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:34.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.878+0000 7f86bc8c5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86b80c5170 0x7f86b80c1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:34.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.878+0000 7f86bd8c7700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b814ef20 0x7f86b80c2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:34.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.878+0000 7f86bd8c7700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b814ef20 0x7f86b80c2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59938/0 (socket says 172.21.15.77:59938) 2022-04-03T19:44:34.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.878+0000 7f86bc8c5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86b80c5170 0x7f86b80c1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37828/0 (socket says 172.21.15.77:37828) 2022-04-03T19:44:34.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.878+0000 7f86bd8c7700 1 -- 172.21.15.77:0/4277479564 learned_addr learned my addr 172.21.15.77:0/4277479564 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:34.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.878+0000 7f86bd8c7700 1 -- 172.21.15.77:0/4277479564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86b80c5170 msgr2=0x7f86b80c1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:34.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.878+0000 7f86bd8c7700 1 --2- 172.21.15.77:0/4277479564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86b80c5170 0x7f86b80c1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.878+0000 7f86bd8c7700 1 -- 172.21.15.77:0/4277479564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86b806bdb0 msgr2=0x7f86b8150f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:34.887 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.878+0000 7f86bd8c7700 1 --2- 172.21.15.77:0/4277479564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86b806bdb0 0x7f86b8150f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:34.887 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.878+0000 7f86bd8c7700 1 -- 172.21.15.77:0/4277479564 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b0009a20 con 0x7f86b814ef20 2022-04-03T19:44:34.887 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.878+0000 7f86bd0c6700 1 --2- 172.21.15.77:0/4277479564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86b806bdb0 0x7f86b8150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:34.887 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.878+0000 7f86bc8c5700 1 --2- 172.21.15.77:0/4277479564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86b80c5170 0x7f86b80c1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:34.887 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.879+0000 7f86bd8c7700 1 --2- 172.21.15.77:0/4277479564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b814ef20 0x7f86b80c2120 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f86b80c67d0 tx=0x7f86b0019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:34.888 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.879+0000 7f86a57fa700 1 -- 172.21.15.77:0/4277479564 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b0003e60 con 0x7f86b814ef20 2022-04-03T19:44:34.888 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.879+0000 7f86a57fa700 1 -- 172.21.15.77:0/4277479564 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b00129b0 con 0x7f86b814ef20 2022-04-03T19:44:34.888 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.879+0000 7f86be0c8700 1 -- 172.21.15.77:0/4277479564 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b8168d10 con 0x7f86b814ef20 2022-04-03T19:44:34.888 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.879+0000 7f86a57fa700 1 -- 172.21.15.77:0/4277479564 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b0012d50 con 0x7f86b814ef20 2022-04-03T19:44:34.888 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.879+0000 7f86be0c8700 1 -- 172.21.15.77:0/4277479564 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86b80c0cf0 con 0x7f86b814ef20 2022-04-03T19:44:34.889 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.880+0000 7f86be0c8700 1 -- 172.21.15.77:0/4277479564 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b8088dd0 con 0x7f86b814ef20 2022-04-03T19:44:34.889 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.881+0000 7f86a57fa700 1 -- 172.21.15.77:0/4277479564 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f86b0004250 con 0x7f86b814ef20 2022-04-03T19:44:34.890 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.881+0000 7f86a57fa700 1 --2- 172.21.15.77:0/4277479564 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8698063ce0 0x7f8698066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:34.890 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.882+0000 7f86a57fa700 1 -- 172.21.15.77:0/4277479564 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f86b0083150 con 0x7f86b814ef20 2022-04-03T19:44:34.890 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.882+0000 7f86bd0c6700 1 --2- 172.21.15.77:0/4277479564 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8698063ce0 0x7f8698066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:34.891 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.882+0000 7f86bd0c6700 1 --2- 172.21.15.77:0/4277479564 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8698063ce0 0x7f8698066190 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f86b4007fd0 tx=0x7f86b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:34.891 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:34.886+0000 7f86a57fa700 1 -- 172.21.15.77:0/4277479564 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f86b0051cd0 con 0x7f86b814ef20 2022-04-03T19:44:35.081 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.081+0000 7f86be0c8700 1 -- 172.21.15.77:0/4277479564 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f86b81691b0 con 0x7f86b814ef20 2022-04-03T19:44:35.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.082+0000 7f86a57fa700 1 -- 172.21.15.77:0/4277479564 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f86b00552f0 con 0x7f86b814ef20 2022-04-03T19:44:35.083 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2022-04-03T19:44:35.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.084+0000 7f86be0c8700 1 -- 172.21.15.77:0/4277479564 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8698063ce0 msgr2=0x7f8698066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:35.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.084+0000 7f86be0c8700 1 --2- 172.21.15.77:0/4277479564 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8698063ce0 0x7f8698066190 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f86b4007fd0 tx=0x7f86b4009040 comp rx=0 tx=0).stop 2022-04-03T19:44:35.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.084+0000 7f86be0c8700 1 -- 172.21.15.77:0/4277479564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b814ef20 msgr2=0x7f86b80c2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:35.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.084+0000 7f86be0c8700 1 --2- 172.21.15.77:0/4277479564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b814ef20 0x7f86b80c2120 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f86b80c67d0 tx=0x7f86b0019000 comp rx=0 tx=0).stop 2022-04-03T19:44:35.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.084+0000 7f86be0c8700 1 -- 172.21.15.77:0/4277479564 shutdown_connections 2022-04-03T19:44:35.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.084+0000 7f86be0c8700 1 --2- 172.21.15.77:0/4277479564 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8698063ce0 0x7f8698066190 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.084+0000 7f86be0c8700 1 --2- 172.21.15.77:0/4277479564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86b814ef20 0x7f86b80c2120 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.084+0000 7f86be0c8700 1 --2- 172.21.15.77:0/4277479564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f86b806bdb0 0x7f86b8150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.084+0000 7f86be0c8700 1 --2- 172.21.15.77:0/4277479564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86b80c5170 0x7f86b80c1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.084+0000 7f86be0c8700 1 -- 172.21.15.77:0/4277479564 >> 172.21.15.77:0/4277479564 conn(0x7f86b80cb6d0 msgr2=0x7f86b80c71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:35.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.084+0000 7f86be0c8700 1 -- 172.21.15.77:0/4277479564 shutdown_connections 2022-04-03T19:44:35.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.084+0000 7f86be0c8700 1 -- 172.21.15.77:0/4277479564 wait complete. 2022-04-03T19:44:35.109 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-04-03T19:44:35.310 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.310+0000 7f98ed5c0700 1 Processor -- start 2022-04-03T19:44:35.311 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.310+0000 7f98ed5c0700 1 -- start start 2022-04-03T19:44:35.312 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.311+0000 7f98ed5c0700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f98e80c52b0 0x7f98e80c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:35.312 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.311+0000 7f98ed5c0700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e80c7e80 con 0x7f98e8158e10 2022-04-03T19:44:35.312 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.311+0000 7f98ed5c0700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e80c6ea0 con 0x7f98e80c5680 2022-04-03T19:44:35.313 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.311+0000 7f98ed5c0700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98e80c4260 con 0x7f98e80c52b0 2022-04-03T19:44:35.313 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.311+0000 7f98e7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f98e80c52b0 0x7f98e80c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:35.313 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.311+0000 7f98e7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f98e80c52b0 0x7f98e80c5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42628/0 (socket says 172.21.15.77:42628) 2022-04-03T19:44:35.313 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.311+0000 7f98e7fff700 1 -- 172.21.15.77:0/1967406590 learned_addr learned my addr 172.21.15.77:0/1967406590 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:35.314 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.311+0000 7f98ecdbf700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f98e8158e10 0x7f98e814ed50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38340/0 (socket says 172.21.15.77:38340) 2022-04-03T19:44:35.314 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.312+0000 7f98e7fff700 1 -- 172.21.15.77:0/1967406590 >> v1:172.21.15.186:6789/0 conn(0x7f98e80c5680 legacy=0x7f98e80c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:35.314 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.312+0000 7f98e7fff700 1 -- 172.21.15.77:0/1967406590 >> v1:172.21.15.77:6789/0 conn(0x7f98e8158e10 legacy=0x7f98e814ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:35.314 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.312+0000 7f98e7fff700 1 -- 172.21.15.77:0/1967406590 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e80c6600 con 0x7f98e80c52b0 2022-04-03T19:44:35.317 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.312+0000 7f98e7fff700 1 --2- 172.21.15.77:0/1967406590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e80c52b0 0x7f98e80c5f70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f98d8006fd0 tx=0x7f98d80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11586b454273e609 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:35.318 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.312+0000 7f98e6ffd700 1 -- 172.21.15.77:0/1967406590 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d8004030 con 0x7f98e80c52b0 2022-04-03T19:44:35.318 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.312+0000 7f98e6ffd700 1 -- 172.21.15.77:0/1967406590 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d80041b0 con 0x7f98e80c52b0 2022-04-03T19:44:35.318 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.312+0000 7f98e6ffd700 1 -- 172.21.15.77:0/1967406590 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d800b6b0 con 0x7f98e80c52b0 2022-04-03T19:44:35.318 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.313+0000 7f98ed5c0700 1 -- 172.21.15.77:0/1967406590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e80c52b0 msgr2=0x7f98e80c5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:35.319 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.313+0000 7f98ed5c0700 1 --2- 172.21.15.77:0/1967406590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e80c52b0 0x7f98e80c5f70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f98d8006fd0 tx=0x7f98d80090f0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.319 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.313+0000 7f98ed5c0700 1 -- 172.21.15.77:0/1967406590 shutdown_connections 2022-04-03T19:44:35.319 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.313+0000 7f98ed5c0700 1 --2- 172.21.15.77:0/1967406590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e80c52b0 0x7f98e80c5f70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.319 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.313+0000 7f98ed5c0700 1 -- 172.21.15.77:0/1967406590 >> 172.21.15.77:0/1967406590 conn(0x7f98e80626d0 msgr2=0x7f98e8062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:35.319 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.313+0000 7f98ed5c0700 1 -- 172.21.15.77:0/1967406590 shutdown_connections 2022-04-03T19:44:35.320 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.313+0000 7f98ed5c0700 1 -- 172.21.15.77:0/1967406590 wait complete. 2022-04-03T19:44:35.320 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.315+0000 7f98ed5c0700 1 Processor -- start 2022-04-03T19:44:35.320 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.315+0000 7f98ed5c0700 1 -- start start 2022-04-03T19:44:35.320 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.316+0000 7f98ed5c0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e80c52b0 0x7f98e8155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:35.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.316+0000 7f98ed5c0700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98e80c5680 0x7f98e8151ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:35.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.316+0000 7f98ed5c0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98e8158e10 0x7f98e81529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:35.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.316+0000 7f98ed5c0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e80c5ed0 con 0x7f98e80c52b0 2022-04-03T19:44:35.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.316+0000 7f98ed5c0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e8154350 con 0x7f98e8158e10 2022-04-03T19:44:35.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.316+0000 7f98ed5c0700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e81651d0 con 0x7f98e80c5680 2022-04-03T19:44:35.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.316+0000 7f98e7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e80c52b0 0x7f98e8155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:35.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.316+0000 7f98e7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e80c52b0 0x7f98e8155ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42630/0 (socket says 172.21.15.77:42630) 2022-04-03T19:44:35.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.316+0000 7f98e7fff700 1 -- 172.21.15.77:0/3751883985 learned_addr learned my addr 172.21.15.77:0/3751883985 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:35.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.316+0000 7f98e77fe700 1 --2- 172.21.15.77:0/3751883985 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98e80c5680 0x7f98e8151ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:35.323 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.316+0000 7f98ecdbf700 1 --2- 172.21.15.77:0/3751883985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98e8158e10 0x7f98e81529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:35.323 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.316+0000 7f98e7fff700 1 -- 172.21.15.77:0/3751883985 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98e80c5680 msgr2=0x7f98e8151ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:35.323 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.316+0000 7f98e7fff700 1 --2- 172.21.15.77:0/3751883985 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98e80c5680 0x7f98e8151ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.323 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.316+0000 7f98e7fff700 1 -- 172.21.15.77:0/3751883985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98e8158e10 msgr2=0x7f98e81529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:35.323 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.316+0000 7f98e7fff700 1 --2- 172.21.15.77:0/3751883985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98e8158e10 0x7f98e81529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.324 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.316+0000 7f98e7fff700 1 -- 172.21.15.77:0/3751883985 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d8009370 con 0x7f98e80c52b0 2022-04-03T19:44:35.324 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.316+0000 7f98ecdbf700 1 --2- 172.21.15.77:0/3751883985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98e8158e10 0x7f98e81529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:44:35.324 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.317+0000 7f98e7fff700 1 --2- 172.21.15.77:0/3751883985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e80c52b0 0x7f98e8155ce0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f98e80632a0 tx=0x7f98d8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:35.324 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.317+0000 7f98cffff700 1 -- 172.21.15.77:0/3751883985 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d8003840 con 0x7f98e80c52b0 2022-04-03T19:44:35.325 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.317+0000 7f98cffff700 1 -- 172.21.15.77:0/3751883985 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d80039a0 con 0x7f98e80c52b0 2022-04-03T19:44:35.325 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.317+0000 7f98cffff700 1 -- 172.21.15.77:0/3751883985 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d8003d40 con 0x7f98e80c52b0 2022-04-03T19:44:35.325 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.317+0000 7f98ed5c0700 1 -- 172.21.15.77:0/3751883985 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e81654b0 con 0x7f98e80c52b0 2022-04-03T19:44:35.325 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.317+0000 7f98ed5c0700 1 -- 172.21.15.77:0/3751883985 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e8165a70 con 0x7f98e80c52b0 2022-04-03T19:44:35.326 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.318+0000 7f98ed5c0700 1 -- 172.21.15.77:0/3751883985 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e8165da0 con 0x7f98e80c52b0 2022-04-03T19:44:35.326 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.319+0000 7f98cffff700 1 -- 172.21.15.77:0/3751883985 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f98d8017020 con 0x7f98e80c52b0 2022-04-03T19:44:35.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.320+0000 7f98cffff700 1 --2- 172.21.15.77:0/3751883985 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f98c8063ce0 0x7f98c8066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:35.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.320+0000 7f98cffff700 1 -- 172.21.15.77:0/3751883985 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f98d8082460 con 0x7f98e80c52b0 2022-04-03T19:44:35.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.320+0000 7f98e77fe700 1 --2- 172.21.15.77:0/3751883985 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f98c8063ce0 0x7f98c8066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:35.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.321+0000 7f98e77fe700 1 --2- 172.21.15.77:0/3751883985 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f98c8063ce0 0x7f98c8066190 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f98e806d9a0 tx=0x7f98dc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:35.328 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.324+0000 7f98cffff700 1 -- 172.21.15.77:0/3751883985 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f98d80510c0 con 0x7f98e80c52b0 2022-04-03T19:44:35.519 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.518+0000 7f98ed5c0700 1 -- 172.21.15.77:0/3751883985 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f98e804b050 con 0x7f98e80c52b0 2022-04-03T19:44:35.519 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.519+0000 7f98cffff700 1 -- 172.21.15.77:0/3751883985 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f98d802a020 con 0x7f98e80c52b0 2022-04-03T19:44:35.520 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2022-04-03T19:44:35.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.521+0000 7f98ed5c0700 1 -- 172.21.15.77:0/3751883985 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f98c8063ce0 msgr2=0x7f98c8066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:35.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.521+0000 7f98ed5c0700 1 --2- 172.21.15.77:0/3751883985 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f98c8063ce0 0x7f98c8066190 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f98e806d9a0 tx=0x7f98dc009040 comp rx=0 tx=0).stop 2022-04-03T19:44:35.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.521+0000 7f98ed5c0700 1 -- 172.21.15.77:0/3751883985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e80c52b0 msgr2=0x7f98e8155ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:35.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.521+0000 7f98ed5c0700 1 --2- 172.21.15.77:0/3751883985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e80c52b0 0x7f98e8155ce0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f98e80632a0 tx=0x7f98d8016000 comp rx=0 tx=0).stop 2022-04-03T19:44:35.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.521+0000 7f98ed5c0700 1 -- 172.21.15.77:0/3751883985 shutdown_connections 2022-04-03T19:44:35.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.521+0000 7f98ed5c0700 1 --2- 172.21.15.77:0/3751883985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98e8158e10 0x7f98e81529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.521+0000 7f98ed5c0700 1 --2- 172.21.15.77:0/3751883985 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98e80c52b0 0x7f98e8155ce0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.521+0000 7f98ed5c0700 1 --2- 172.21.15.77:0/3751883985 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f98c8063ce0 0x7f98c8066190 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.521+0000 7f98ed5c0700 1 --2- 172.21.15.77:0/3751883985 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98e80c5680 0x7f98e8151ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.522+0000 7f98ed5c0700 1 -- 172.21.15.77:0/3751883985 >> 172.21.15.77:0/3751883985 conn(0x7f98e80626d0 msgr2=0x7f98e8156a00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:35.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.522+0000 7f98ed5c0700 1 -- 172.21.15.77:0/3751883985 shutdown_connections 2022-04-03T19:44:35.525 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.522+0000 7f98ed5c0700 1 -- 172.21.15.77:0/3751883985 wait complete. 2022-04-03T19:44:35.546 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:44:35.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.744+0000 7f995329d700 1 Processor -- start 2022-04-03T19:44:35.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.744+0000 7f995329d700 1 -- start start 2022-04-03T19:44:35.747 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.745+0000 7f995329d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f994c06d3f0 0x7f994c06d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:35.747 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.745+0000 7f995329d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f994c0c5930 0x7f994c155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:35.748 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.745+0000 7f995329d700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994c0c2f80 con 0x7f994c156120 2022-04-03T19:44:35.748 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.745+0000 7f995329d700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c0c3260 con 0x7f994c06d3f0 2022-04-03T19:44:35.748 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.745+0000 7f995329d700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c0c3540 con 0x7f994c0c5930 2022-04-03T19:44:35.748 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.746+0000 7f995229b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f994c06d3f0 0x7f994c06d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:35.749 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.746+0000 7f9952a9c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f994c156120 0x7f994c06c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38342/0 (socket says 172.21.15.77:38342) 2022-04-03T19:44:35.749 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.746+0000 7f9952a9c700 1 -- 172.21.15.77:0/3164922724 learned_addr learned my addr 172.21.15.77:0/3164922724 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:35.749 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.746+0000 7f995229b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f994c06d3f0 0x7f994c06d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42632/0 (socket says 172.21.15.77:42632) 2022-04-03T19:44:35.749 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.746+0000 7f9951a9a700 1 --2- 172.21.15.77:0/3164922724 >> v2:172.21.15.186:3300/0 conn(0x7f994c0c5930 0x7f994c155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:35.749 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.746+0000 7f995229b700 1 -- 172.21.15.77:0/3164922724 >> v2:172.21.15.186:3300/0 conn(0x7f994c0c5930 msgr2=0x7f994c155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:35.750 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.746+0000 7f995229b700 1 --2- 172.21.15.77:0/3164922724 >> v2:172.21.15.186:3300/0 conn(0x7f994c0c5930 0x7f994c155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.750 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.746+0000 7f995229b700 1 -- 172.21.15.77:0/3164922724 >> v1:172.21.15.77:6789/0 conn(0x7f994c156120 legacy=0x7f994c06c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:35.750 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.746+0000 7f995229b700 1 -- 172.21.15.77:0/3164922724 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c0c3820 con 0x7f994c06d3f0 2022-04-03T19:44:35.750 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.746+0000 7f9951299700 1 -- 172.21.15.77:0/3164922724 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433659689 0 0) 0x7f994c0c2f80 con 0x7f994c156120 2022-04-03T19:44:35.750 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.747+0000 7f995229b700 1 --2- 172.21.15.77:0/3164922724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c06d3f0 0x7f994c06d7c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9948009760 tx=0x7f9948009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88b994a6fcb8c510 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:35.751 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.747+0000 7f9951299700 1 -- 172.21.15.77:0/3164922724 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9948011630 con 0x7f994c06d3f0 2022-04-03T19:44:35.751 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.747+0000 7f9951299700 1 -- 172.21.15.77:0/3164922724 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9948011790 con 0x7f994c06d3f0 2022-04-03T19:44:35.751 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.747+0000 7f9951299700 1 -- 172.21.15.77:0/3164922724 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9948011b30 con 0x7f994c06d3f0 2022-04-03T19:44:35.755 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.747+0000 7f995329d700 1 -- 172.21.15.77:0/3164922724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c06d3f0 msgr2=0x7f994c06d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:35.755 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.747+0000 7f995329d700 1 --2- 172.21.15.77:0/3164922724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c06d3f0 0x7f994c06d7c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9948009760 tx=0x7f9948009060 comp rx=0 tx=0).stop 2022-04-03T19:44:35.755 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.747+0000 7f995329d700 1 -- 172.21.15.77:0/3164922724 shutdown_connections 2022-04-03T19:44:35.756 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.747+0000 7f995329d700 1 --2- 172.21.15.77:0/3164922724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c06d3f0 0x7f994c06d7c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.756 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.747+0000 7f995329d700 1 --2- 172.21.15.77:0/3164922724 >> v2:172.21.15.186:3300/0 conn(0x7f994c0c5930 0x7f994c155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.756 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.747+0000 7f995329d700 1 -- 172.21.15.77:0/3164922724 >> 172.21.15.77:0/3164922724 conn(0x7f994c0cb720 msgr2=0x7f994c0cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:35.756 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.748+0000 7f995329d700 1 -- 172.21.15.77:0/3164922724 shutdown_connections 2022-04-03T19:44:35.757 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.748+0000 7f995329d700 1 -- 172.21.15.77:0/3164922724 wait complete. 2022-04-03T19:44:35.757 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.749+0000 7f995329d700 1 Processor -- start 2022-04-03T19:44:35.757 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.749+0000 7f995329d700 1 -- start start 2022-04-03T19:44:35.757 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.750+0000 7f995329d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f994c06d3f0 0x7f994c156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:35.758 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.750+0000 7f995329d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f994c0c5930 0x7f994c14ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:35.758 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.750+0000 7f995329d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c156120 0x7f994c14d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:35.758 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.750+0000 7f995329d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c1510e0 con 0x7f994c156120 2022-04-03T19:44:35.758 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.750+0000 7f995329d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c156f90 con 0x7f994c06d3f0 2022-04-03T19:44:35.758 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.750+0000 7f995329d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c1505c0 con 0x7f994c0c5930 2022-04-03T19:44:35.759 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.750+0000 7f9952a9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c156120 0x7f994c14d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:35.759 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.750+0000 7f995229b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f994c06d3f0 0x7f994c156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:35.759 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.750+0000 7f9951a9a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f994c0c5930 0x7f994c14ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:35.759 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.750+0000 7f9952a9c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c156120 0x7f994c14d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42636/0 (socket says 172.21.15.77:42636) 2022-04-03T19:44:35.760 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.750+0000 7f995229b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f994c06d3f0 0x7f994c156540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37834/0 (socket says 172.21.15.77:37834) 2022-04-03T19:44:35.760 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.750+0000 7f9952a9c700 1 -- 172.21.15.77:0/2842379852 learned_addr learned my addr 172.21.15.77:0/2842379852 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:35.760 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.751+0000 7f995229b700 1 -- 172.21.15.77:0/2842379852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f994c0c5930 msgr2=0x7f994c14ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:35.760 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.751+0000 7f995229b700 1 --2- 172.21.15.77:0/2842379852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f994c0c5930 0x7f994c14ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.760 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.751+0000 7f995229b700 1 -- 172.21.15.77:0/2842379852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c156120 msgr2=0x7f994c14d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:35.761 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.751+0000 7f995229b700 1 --2- 172.21.15.77:0/2842379852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c156120 0x7f994c14d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.761 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.751+0000 7f995229b700 1 -- 172.21.15.77:0/2842379852 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99480092a0 con 0x7f994c06d3f0 2022-04-03T19:44:35.761 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.751+0000 7f995229b700 1 --2- 172.21.15.77:0/2842379852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f994c06d3f0 0x7f994c156540 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f994c0c4710 tx=0x7f9948017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:35.761 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.751+0000 7f993a7fc700 1 -- 172.21.15.77:0/2842379852 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9948003770 con 0x7f994c06d3f0 2022-04-03T19:44:35.762 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.751+0000 7f993a7fc700 1 -- 172.21.15.77:0/2842379852 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99480038d0 con 0x7f994c06d3f0 2022-04-03T19:44:35.762 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.751+0000 7f995329d700 1 -- 172.21.15.77:0/2842379852 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994c169060 con 0x7f994c06d3f0 2022-04-03T19:44:35.762 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.751+0000 7f993a7fc700 1 -- 172.21.15.77:0/2842379852 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9948003c90 con 0x7f994c06d3f0 2022-04-03T19:44:35.762 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.752+0000 7f995329d700 1 -- 172.21.15.77:0/2842379852 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994c156a80 con 0x7f994c06d3f0 2022-04-03T19:44:35.762 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.752+0000 7f995329d700 1 -- 172.21.15.77:0/2842379852 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994c14fdd0 con 0x7f994c06d3f0 2022-04-03T19:44:35.763 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.753+0000 7f993a7fc700 1 -- 172.21.15.77:0/2842379852 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f994800f050 con 0x7f994c06d3f0 2022-04-03T19:44:35.763 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.754+0000 7f993a7fc700 1 --2- 172.21.15.77:0/2842379852 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f992c063ce0 0x7f992c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:35.763 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.754+0000 7f993a7fc700 1 -- 172.21.15.77:0/2842379852 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f9948082270 con 0x7f994c06d3f0 2022-04-03T19:44:35.763 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.755+0000 7f9951a9a700 1 --2- 172.21.15.77:0/2842379852 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f992c063ce0 0x7f992c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:35.764 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.755+0000 7f9951a9a700 1 --2- 172.21.15.77:0/2842379852 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f992c063ce0 0x7f992c066190 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f994c14f900 tx=0x7f993c009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:35.764 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.758+0000 7f993a7fc700 1 -- 172.21.15.77:0/2842379852 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f9948050eb0 con 0x7f994c06d3f0 2022-04-03T19:44:35.914 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T19:44:35.913+0000 7f565c377700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-04-03T19:44:35.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.987+0000 7f995329d700 1 -- 172.21.15.77:0/2842379852 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f994c04b050 con 0x7f994c06d3f0 2022-04-03T19:44:35.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.988+0000 7f993a7fc700 1 -- 172.21.15.77:0/2842379852 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6107 (secure 0 0 0) 0x7f9948015020 con 0x7f994c06d3f0 2022-04-03T19:44:35.989 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:35.989 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4303,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:44:27.295743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4327":{"gid":4327,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6837/3665969042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3665969042},{"type":"v1","addr":"172.21.15.77:6837","nonce":3665969042}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:44:35.991 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.991+0000 7f995329d700 1 -- 172.21.15.77:0/2842379852 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f992c063ce0 msgr2=0x7f992c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:35.991 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.991+0000 7f995329d700 1 --2- 172.21.15.77:0/2842379852 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f992c063ce0 0x7f992c066190 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f994c14f900 tx=0x7f993c009060 comp rx=0 tx=0).stop 2022-04-03T19:44:35.992 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.991+0000 7f995329d700 1 -- 172.21.15.77:0/2842379852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f994c06d3f0 msgr2=0x7f994c156540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:35.992 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.991+0000 7f995329d700 1 --2- 172.21.15.77:0/2842379852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f994c06d3f0 0x7f994c156540 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f994c0c4710 tx=0x7f9948017000 comp rx=0 tx=0).stop 2022-04-03T19:44:35.992 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.991+0000 7f995329d700 1 -- 172.21.15.77:0/2842379852 shutdown_connections 2022-04-03T19:44:35.992 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.991+0000 7f995329d700 1 --2- 172.21.15.77:0/2842379852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c156120 0x7f994c14d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.991+0000 7f995329d700 1 --2- 172.21.15.77:0/2842379852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f994c06d3f0 0x7f994c156540 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.991+0000 7f995329d700 1 --2- 172.21.15.77:0/2842379852 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f992c063ce0 0x7f992c066190 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.991+0000 7f995329d700 1 --2- 172.21.15.77:0/2842379852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f994c0c5930 0x7f994c14ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:35.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.991+0000 7f995329d700 1 -- 172.21.15.77:0/2842379852 >> 172.21.15.77:0/2842379852 conn(0x7f994c0cb720 msgr2=0x7f994c154510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:35.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.991+0000 7f995329d700 1 -- 172.21.15.77:0/2842379852 shutdown_connections 2022-04-03T19:44:35.994 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:35.991+0000 7f995329d700 1 -- 172.21.15.77:0/2842379852 wait complete. 2022-04-03T19:44:35.994 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2022-04-03T19:44:36.619 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-04-03T19:44:21.275716+0000', 'modified': '2022-04-03T19:44:27.295743+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4327}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4327': {'gid': 4327, 'name': 'e', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.77:6837/3665969042', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.77:6836', 'nonce': 3665969042}, {'type': 'v1', 'addr': '172.21.15.77:6837', 'nonce': 3665969042}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138303579357183, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-04-03T19:44:36.619 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-04-03T19:44:36.619 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2022-04-03T19:44:36.823 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.822+0000 7fea9a12b700 1 Processor -- start 2022-04-03T19:44:37.067 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.822+0000 7fea9a12b700 1 -- start start 2022-04-03T19:44:37.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.823+0000 7fea9a12b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea9406bbe0 0x7fea941584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.823+0000 7fea9a12b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fea9406bfb0 0x7fea9406d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.823+0000 7fea9a12b700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fea9406c500 0x7fea9414eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.823+0000 7fea9a12b700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea94159450 con 0x7fea9406bfb0 2022-04-03T19:44:37.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.823+0000 7fea9a12b700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea9406d260 con 0x7fea9406c500 2022-04-03T19:44:37.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.823+0000 7fea9a12b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea940c6330 con 0x7fea9406bbe0 2022-04-03T19:44:37.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.824+0000 7fea98928700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fea9406bfb0 0x7fea9406d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.824+0000 7fea99129700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea9406bbe0 0x7fea941584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.824+0000 7fea98928700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fea9406bfb0 0x7fea9406d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42638/0 (socket says 172.21.15.77:42638) 2022-04-03T19:44:37.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.824+0000 7fea98928700 1 -- 172.21.15.77:0/1073438080 learned_addr learned my addr 172.21.15.77:0/1073438080 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:37.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.824+0000 7fea9992a700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fea9406c500 0x7fea9414eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.824+0000 7fea99129700 1 -- 172.21.15.77:0/1073438080 >> v2:172.21.15.186:3300/0 conn(0x7fea9406c500 msgr2=0x7fea9414eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:37.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.824+0000 7fea99129700 1 --2- 172.21.15.77:0/1073438080 >> v2:172.21.15.186:3300/0 conn(0x7fea9406c500 0x7fea9414eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.824+0000 7fea99129700 1 -- 172.21.15.77:0/1073438080 >> v2:172.21.15.77:3300/0 conn(0x7fea9406bfb0 msgr2=0x7fea9406d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:37.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.824+0000 7fea99129700 1 --2- 172.21.15.77:0/1073438080 >> v2:172.21.15.77:3300/0 conn(0x7fea9406bfb0 0x7fea9406d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.824+0000 7fea99129700 1 -- 172.21.15.77:0/1073438080 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea94158bb0 con 0x7fea9406bbe0 2022-04-03T19:44:37.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.825+0000 7fea99129700 1 --2- 172.21.15.77:0/1073438080 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea9406bbe0 0x7fea941584b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fea940c4fe0 tx=0x7fea90009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2db611e061c7be95 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:37.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.825+0000 7fea83fff700 1 -- 172.21.15.77:0/1073438080 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea900105b0 con 0x7fea9406bbe0 2022-04-03T19:44:37.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.825+0000 7fea83fff700 1 -- 172.21.15.77:0/1073438080 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea90010710 con 0x7fea9406bbe0 2022-04-03T19:44:37.090 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.825+0000 7fea83fff700 1 -- 172.21.15.77:0/1073438080 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea90010ad0 con 0x7fea9406bbe0 2022-04-03T19:44:37.090 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.825+0000 7fea9a12b700 1 -- 172.21.15.77:0/1073438080 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea9406bbe0 msgr2=0x7fea941584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:37.090 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.825+0000 7fea9a12b700 1 --2- 172.21.15.77:0/1073438080 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea9406bbe0 0x7fea941584b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fea940c4fe0 tx=0x7fea90009060 comp rx=0 tx=0).stop 2022-04-03T19:44:37.090 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.825+0000 7fea9a12b700 1 -- 172.21.15.77:0/1073438080 shutdown_connections 2022-04-03T19:44:37.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.825+0000 7fea9a12b700 1 --2- 172.21.15.77:0/1073438080 >> v2:172.21.15.186:3300/0 conn(0x7fea9406c500 0x7fea9414eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.825+0000 7fea9a12b700 1 --2- 172.21.15.77:0/1073438080 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea9406bbe0 0x7fea941584b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.825+0000 7fea9a12b700 1 --2- 172.21.15.77:0/1073438080 >> v2:172.21.15.77:3300/0 conn(0x7fea9406bfb0 0x7fea9406d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.825+0000 7fea9a12b700 1 -- 172.21.15.77:0/1073438080 >> 172.21.15.77:0/1073438080 conn(0x7fea940cb6b0 msgr2=0x7fea940cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:37.092 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.826+0000 7fea9a12b700 1 -- 172.21.15.77:0/1073438080 shutdown_connections 2022-04-03T19:44:37.092 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.826+0000 7fea9a12b700 1 -- 172.21.15.77:0/1073438080 wait complete. 2022-04-03T19:44:37.092 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.827+0000 7fea9a12b700 1 Processor -- start 2022-04-03T19:44:37.092 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.827+0000 7fea9a12b700 1 -- start start 2022-04-03T19:44:37.092 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.828+0000 7fea9a12b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea9406bbe0 0x7fea94164bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.093 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.828+0000 7fea9a12b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea9406bfb0 0x7fea940c0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.093 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.828+0000 7fea9a12b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea9406c500 0x7fea940ca650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.093 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.828+0000 7fea9a12b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea940bfd90 con 0x7fea9406bbe0 2022-04-03T19:44:37.093 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.828+0000 7fea9a12b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea941523a0 con 0x7fea9406c500 2022-04-03T19:44:37.093 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.828+0000 7fea9a12b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea94151de0 con 0x7fea9406bfb0 2022-04-03T19:44:37.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.828+0000 7fea99129700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea9406bbe0 0x7fea94164bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.828+0000 7fea9992a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea9406c500 0x7fea940ca650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.829+0000 7fea98928700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea9406bfb0 0x7fea940c0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.828+0000 7fea99129700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea9406bbe0 0x7fea94164bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42640/0 (socket says 172.21.15.77:42640) 2022-04-03T19:44:37.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.829+0000 7fea99129700 1 -- 172.21.15.77:0/2615677343 learned_addr learned my addr 172.21.15.77:0/2615677343 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:37.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.829+0000 7fea99129700 1 -- 172.21.15.77:0/2615677343 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea9406bfb0 msgr2=0x7fea940c0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:37.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.829+0000 7fea99129700 1 --2- 172.21.15.77:0/2615677343 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea9406bfb0 0x7fea940c0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.829+0000 7fea99129700 1 -- 172.21.15.77:0/2615677343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea9406c500 msgr2=0x7fea940ca650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:37.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.829+0000 7fea99129700 1 --2- 172.21.15.77:0/2615677343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea9406c500 0x7fea940ca650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.829+0000 7fea99129700 1 -- 172.21.15.77:0/2615677343 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea900092a0 con 0x7fea9406bbe0 2022-04-03T19:44:37.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.829+0000 7fea99129700 1 --2- 172.21.15.77:0/2615677343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea9406bbe0 0x7fea94164bf0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fea90010630 tx=0x7fea90016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:37.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.829+0000 7fea817fa700 1 -- 172.21.15.77:0/2615677343 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea90010a90 con 0x7fea9406bbe0 2022-04-03T19:44:37.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.830+0000 7fea817fa700 1 -- 172.21.15.77:0/2615677343 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea90000c90 con 0x7fea9406bbe0 2022-04-03T19:44:37.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.830+0000 7fea817fa700 1 -- 172.21.15.77:0/2615677343 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea90004280 con 0x7fea9406bbe0 2022-04-03T19:44:37.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.830+0000 7fea9a12b700 1 -- 172.21.15.77:0/2615677343 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea941520c0 con 0x7fea9406bbe0 2022-04-03T19:44:37.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.830+0000 7fea9a12b700 1 -- 172.21.15.77:0/2615677343 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea940bfa20 con 0x7fea9406bbe0 2022-04-03T19:44:37.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.832+0000 7fea817fa700 1 -- 172.21.15.77:0/2615677343 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fea9002a020 con 0x7fea9406bbe0 2022-04-03T19:44:37.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.832+0000 7fea817fa700 1 --2- 172.21.15.77:0/2615677343 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fea74063ce0 0x7fea74066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.833+0000 7fea817fa700 1 -- 172.21.15.77:0/2615677343 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fea90081df0 con 0x7fea9406bbe0 2022-04-03T19:44:37.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.833+0000 7fea98928700 1 --2- 172.21.15.77:0/2615677343 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fea74063ce0 0x7fea74066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.833+0000 7fea98928700 1 --2- 172.21.15.77:0/2615677343 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fea74063ce0 0x7fea74066190 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fea9414b650 tx=0x7fea840090f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:37.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.835+0000 7feaa4cffb80 1 Processor -- start 2022-04-03T19:44:37.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.835+0000 7feaa4cffb80 1 -- start start 2022-04-03T19:44:37.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.836+0000 7feaa4cffb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd001f30 0x55f2bd047b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.836+0000 7feaa4cffb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002b40 0x55f2bd039f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.836+0000 7feaa4cffb80 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd002300 0x55f2bd03a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.836+0000 7feaa4cffb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2bcf76980 con 0x55f2bd002b40 2022-04-03T19:44:37.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.836+0000 7feaa4cffb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2bced1f30 con 0x55f2bd001f30 2022-04-03T19:44:37.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.836+0000 7feaa4cffb80 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f2bcf58450 con 0x55f2bd002300 2022-04-03T19:44:37.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.836+0000 7fea98928700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002b40 0x55f2bd039f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.836+0000 7fea98928700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002b40 0x55f2bd039f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42642/0 (socket says 172.21.15.77:42642) 2022-04-03T19:44:37.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.836+0000 7fea98928700 1 -- 172.21.15.77:0/741927684 learned_addr learned my addr 172.21.15.77:0/741927684 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:37.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.836+0000 7fea99129700 1 --2- 172.21.15.77:0/741927684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd001f30 0x55f2bd047b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.836+0000 7fea9992a700 1 --2- 172.21.15.77:0/741927684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd002300 0x55f2bd03a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.836+0000 7fea98928700 1 -- 172.21.15.77:0/741927684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd002300 msgr2=0x55f2bd03a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:37.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.836+0000 7fea98928700 1 --2- 172.21.15.77:0/741927684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd002300 0x55f2bd03a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.836+0000 7fea98928700 1 -- 172.21.15.77:0/741927684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd001f30 msgr2=0x55f2bd047b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:37.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.836+0000 7fea98928700 1 --2- 172.21.15.77:0/741927684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd001f30 0x55f2bd047b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.836+0000 7fea98928700 1 -- 172.21.15.77:0/741927684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2bcde8360 con 0x55f2bd002b40 2022-04-03T19:44:37.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.836+0000 7fea9992a700 1 --2- 172.21.15.77:0/741927684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd002300 0x55f2bd03a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:44:37.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.837+0000 7fea98928700 1 --2- 172.21.15.77:0/741927684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002b40 0x55f2bd039f50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fea84006fd0 tx=0x7fea84019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b6c06495486a18 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:37.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.837+0000 7fea7a7fc700 1 -- 172.21.15.77:0/741927684 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea84014c60 con 0x55f2bd002b40 2022-04-03T19:44:37.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.837+0000 7fea7a7fc700 1 -- 172.21.15.77:0/741927684 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea84014dc0 con 0x55f2bd002b40 2022-04-03T19:44:37.104 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.837+0000 7fea7a7fc700 1 -- 172.21.15.77:0/741927684 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea84029620 con 0x55f2bd002b40 2022-04-03T19:44:37.104 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.837+0000 7feaa4cffb80 1 -- 172.21.15.77:0/741927684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002b40 msgr2=0x55f2bd039f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:37.104 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.837+0000 7feaa4cffb80 1 --2- 172.21.15.77:0/741927684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002b40 0x55f2bd039f50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fea84006fd0 tx=0x7fea84019000 comp rx=0 tx=0).stop 2022-04-03T19:44:37.104 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.838+0000 7feaa4cffb80 1 -- 172.21.15.77:0/741927684 shutdown_connections 2022-04-03T19:44:37.104 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.838+0000 7feaa4cffb80 1 --2- 172.21.15.77:0/741927684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd002300 0x55f2bd03a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.105 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.838+0000 7feaa4cffb80 1 --2- 172.21.15.77:0/741927684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd001f30 0x55f2bd047b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.105 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.838+0000 7feaa4cffb80 1 --2- 172.21.15.77:0/741927684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002b40 0x55f2bd039f50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.105 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.838+0000 7feaa4cffb80 1 -- 172.21.15.77:0/741927684 >> 172.21.15.77:0/741927684 conn(0x55f2bd001990 msgr2=0x55f2bd047450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:37.105 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.838+0000 7feaa4cffb80 1 -- 172.21.15.77:0/741927684 shutdown_connections 2022-04-03T19:44:37.106 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.838+0000 7feaa4cffb80 1 -- 172.21.15.77:0/741927684 wait complete. 2022-04-03T19:44:37.106 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.838+0000 7feaa4cffb80 1 -- start start 2022-04-03T19:44:37.106 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.839+0000 7feaa4cffb80 1 Processor -- start 2022-04-03T19:44:37.106 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.840+0000 7feaa4cffb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd001f30 0x55f2bd040a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.106 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.840+0000 7feaa4cffb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 0x55f2bd050bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.107 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.840+0000 7feaa4cffb80 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd002b40 0x55f2bd0346c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.107 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.840+0000 7feaa4cffb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2bcf52530 con 0x55f2bd002300 2022-04-03T19:44:37.107 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.840+0000 7feaa4cffb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2bcdea650 con 0x55f2bd001f30 2022-04-03T19:44:37.107 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.840+0000 7feaa4cffb80 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f2bd03e220 con 0x55f2bd002b40 2022-04-03T19:44:37.108 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.840+0000 7feaa4cffb80 10 client.0 _ll_register_callbacks cb 0x55f2bce423d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-04-03T19:44:37.108 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.840+0000 7fea98928700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 0x55f2bd050bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.108 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.840+0000 7fea98928700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 0x55f2bd050bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42646/0 (socket says 172.21.15.77:42646) 2022-04-03T19:44:37.108 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.840+0000 7fea98928700 1 -- 172.21.15.77:0/291543013 learned_addr learned my addr 172.21.15.77:0/291543013 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:37.108 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.840+0000 7fea99129700 1 --2- 172.21.15.77:0/291543013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd001f30 0x55f2bd040a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.109 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.840+0000 7fea9992a700 1 --2- 172.21.15.77:0/291543013 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd002b40 0x55f2bd0346c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.109 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.840+0000 7fea98928700 1 -- 172.21.15.77:0/291543013 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd002b40 msgr2=0x55f2bd0346c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:37.109 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.840+0000 7fea98928700 1 --2- 172.21.15.77:0/291543013 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd002b40 0x55f2bd0346c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.109 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.840+0000 7fea98928700 1 -- 172.21.15.77:0/291543013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd001f30 msgr2=0x55f2bd040a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:37.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.840+0000 7fea98928700 1 --2- 172.21.15.77:0/291543013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd001f30 0x55f2bd040a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.840+0000 7fea98928700 1 -- 172.21.15.77:0/291543013 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea84011a20 con 0x55f2bd002300 2022-04-03T19:44:37.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.841+0000 7fea98928700 1 --2- 172.21.15.77:0/291543013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 0x55f2bd050bb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fea84012010 tx=0x7fea84023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea0137eb43fb2d18 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:37.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.841+0000 7fea6f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2022-04-03T19:44:37.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.841+0000 7fea6f7fe700 1 -- 172.21.15.77:0/291543013 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8402f070 con 0x55f2bd002300 2022-04-03T19:44:37.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.841+0000 7fea6f7fe700 1 -- 172.21.15.77:0/291543013 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fea8401a040 con 0x55f2bd002300 2022-04-03T19:44:37.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.841+0000 7feaa4cffb80 1 -- 172.21.15.77:0/291543013 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f2bcfd3630 con 0x55f2bd002300 2022-04-03T19:44:37.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.841+0000 7fea6f7fe700 1 -- 172.21.15.77:0/291543013 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea84004550 con 0x55f2bd002300 2022-04-03T19:44:37.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.841+0000 7feaa4cffb80 10 client.4590 fetch_fsmap learned FSMap version 8 2022-04-03T19:44:37.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.841+0000 7feaa4cffb80 1 -- 172.21.15.77:0/291543013 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55f2bcd92fe0 con 0x55f2bd002300 2022-04-03T19:44:37.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.842+0000 7fea6f7fe700 1 -- 172.21.15.77:0/291543013 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f2bcfd3630 con 0x55f2bd002300 2022-04-03T19:44:37.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.842+0000 7fea6f7fe700 1 -- 172.21.15.77:0/291543013 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 8) v1 ==== 4190+0+0 (secure 0 0 0) 0x7fea84034470 con 0x55f2bd002300 2022-04-03T19:44:37.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.842+0000 7feaa4cffb80 10 client.4590 fetch_fsmap finished waiting for FSMap version 8 2022-04-03T19:44:37.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.842+0000 7feaa4cffb80 10 client.4590 resolve_mds: resolved name 'e' to daemon mds.e 2022-04-03T19:44:37.113 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.842+0000 7feaa4cffb80 20 client.4590 populate_metadata read hostname 'smithi077' 2022-04-03T19:44:37.113 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.842+0000 7feaa4cffb80 1 --2- 172.21.15.77:0/291543013 >> [v2:172.21.15.77:6836/3665969042,v1:172.21.15.77:6837/3665969042] conn(0x55f2bd055d20 0x55f2bd052f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.113 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.842+0000 7feaa4cffb80 4 client.4590 mds_command: new command op to 4327 tid=0[{"prefix": "get_command_descriptions"}] 2022-04-03T19:44:37.113 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.842+0000 7feaa4cffb80 1 -- 172.21.15.77:0/291543013 --> [v2:172.21.15.77:6836/3665969042,v1:172.21.15.77:6837/3665969042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f2bcef56e0 con 0x55f2bd055d20 2022-04-03T19:44:37.113 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.842+0000 7fea99129700 1 --2- 172.21.15.77:0/291543013 >> [v2:172.21.15.77:6836/3665969042,v1:172.21.15.77:6837/3665969042] conn(0x55f2bd055d20 0x55f2bd052f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.843+0000 7fea6f7fe700 10 client.4590 ms_handle_connect on v2:172.21.15.77:6836/3665969042 2022-04-03T19:44:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.843+0000 7fea99129700 1 --2- 172.21.15.77:0/291543013 >> [v2:172.21.15.77:6836/3665969042,v1:172.21.15.77:6837/3665969042] conn(0x55f2bd055d20 0x55f2bd052f50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d9d09032b022803 server_cookie=95441efa018d46eb in_seq=0 out_seq=0 2022-04-03T19:44:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.843+0000 7fea6f7fe700 1 -- 172.21.15.77:0/291543013 <== mds.0 v2:172.21.15.77:6836/3665969042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17246 (crc 0 0 0) 0x55f2bcef56e0 con 0x55f2bd055d20 2022-04-03T19:44:37.114 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.843+0000 7fea6f7fe700 10 client.4590 handle_command_reply: tid=0 2022-04-03T19:44:37.115 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.844+0000 7feaa4cffb80 1 client.4590 shutdown 2022-04-03T19:44:37.115 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.844+0000 7feaa4cffb80 1 -- 172.21.15.77:0/291543013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 msgr2=0x55f2bd050bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:37.115 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.844+0000 7feaa4cffb80 1 --2- 172.21.15.77:0/291543013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 0x55f2bd050bb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fea84012010 tx=0x7fea84023000 comp rx=0 tx=0).stop 2022-04-03T19:44:37.115 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.844+0000 7feaa4cffb80 1 -- 172.21.15.77:0/291543013 shutdown_connections 2022-04-03T19:44:37.116 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.844+0000 7feaa4cffb80 1 --2- 172.21.15.77:0/291543013 >> [v2:172.21.15.77:6836/3665969042,v1:172.21.15.77:6837/3665969042] conn(0x55f2bd055d20 0x55f2bd052f50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.116 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.844+0000 7feaa4cffb80 1 --2- 172.21.15.77:0/291543013 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd002b40 0x55f2bd0346c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.116 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.844+0000 7feaa4cffb80 1 --2- 172.21.15.77:0/291543013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd001f30 0x55f2bd040a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.116 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.844+0000 7fea6f7fe700 0 client.4590 ms_handle_reset on v2:172.21.15.77:6836/3665969042 2022-04-03T19:44:37.116 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.844+0000 7feaa4cffb80 1 --2- 172.21.15.77:0/291543013 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 0x55f2bd050bb0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.844+0000 7feaa4cffb80 1 -- 172.21.15.77:0/291543013 >> 172.21.15.77:0/291543013 conn(0x55f2bd001990 msgr2=0x55f2bcfff250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:37.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.844+0000 7feaa4cffb80 1 -- 172.21.15.77:0/291543013 shutdown_connections 2022-04-03T19:44:37.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.844+0000 7feaa4cffb80 1 -- 172.21.15.77:0/291543013 wait complete. 2022-04-03T19:44:37.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.844+0000 7feaa4cffb80 20 client.4590 trim_cache size 0 max 16384 2022-04-03T19:44:37.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.862+0000 7feaa4cffb80 1 Processor -- start 2022-04-03T19:44:37.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.862+0000 7feaa4cffb80 1 -- start start 2022-04-03T19:44:37.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.863+0000 7feaa4cffb80 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd001f30 0x55f2bd038c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.863+0000 7feaa4cffb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 0x55f2bcfff020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.863+0000 7feaa4cffb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd002b40 0x55f2bd051160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.863+0000 7feaa4cffb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2bd047870 con 0x55f2bd002300 2022-04-03T19:44:37.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.863+0000 7feaa4cffb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2bd0016e0 con 0x55f2bd002b40 2022-04-03T19:44:37.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.863+0000 7feaa4cffb80 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f2bd047e40 con 0x55f2bd001f30 2022-04-03T19:44:37.120 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.863+0000 7fea98928700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 0x55f2bcfff020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.120 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.863+0000 7fea98928700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 0x55f2bcfff020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42648/0 (socket says 172.21.15.77:42648) 2022-04-03T19:44:37.120 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.863+0000 7fea98928700 1 -- 172.21.15.77:0/3115849946 learned_addr learned my addr 172.21.15.77:0/3115849946 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:37.120 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.863+0000 7fea99129700 1 --2- 172.21.15.77:0/3115849946 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd001f30 0x55f2bd038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.121 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.863+0000 7fea9992a700 1 --2- 172.21.15.77:0/3115849946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd002b40 0x55f2bd051160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.121 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.863+0000 7fea98928700 1 -- 172.21.15.77:0/3115849946 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd001f30 msgr2=0x55f2bd038c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:37.121 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.863+0000 7fea98928700 1 --2- 172.21.15.77:0/3115849946 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd001f30 0x55f2bd038c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.121 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.863+0000 7fea98928700 1 -- 172.21.15.77:0/3115849946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd002b40 msgr2=0x55f2bd051160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:37.121 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.863+0000 7fea98928700 1 --2- 172.21.15.77:0/3115849946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd002b40 0x55f2bd051160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.122 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.863+0000 7fea98928700 1 -- 172.21.15.77:0/3115849946 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2bcdbaf80 con 0x55f2bd002300 2022-04-03T19:44:37.122 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.864+0000 7fea9992a700 1 --2- 172.21.15.77:0/3115849946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd002b40 0x55f2bd051160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:44:37.122 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.864+0000 7fea98928700 1 --2- 172.21.15.77:0/3115849946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 0x55f2bcfff020 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fea84006bc0 tx=0x7fea8402c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac2aff2344ff66e8 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:37.122 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.864+0000 7fea7affd700 1 -- 172.21.15.77:0/3115849946 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8402f070 con 0x55f2bd002300 2022-04-03T19:44:37.123 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.864+0000 7fea7affd700 1 -- 172.21.15.77:0/3115849946 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea84034930 con 0x55f2bd002300 2022-04-03T19:44:37.123 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.864+0000 7fea7affd700 1 -- 172.21.15.77:0/3115849946 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea84004550 con 0x55f2bd002300 2022-04-03T19:44:37.123 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.865+0000 7feaa4cffb80 1 -- 172.21.15.77:0/3115849946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 msgr2=0x55f2bcfff020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:37.123 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.865+0000 7feaa4cffb80 1 --2- 172.21.15.77:0/3115849946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 0x55f2bcfff020 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fea84006bc0 tx=0x7fea8402c070 comp rx=0 tx=0).stop 2022-04-03T19:44:37.123 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.865+0000 7feaa4cffb80 1 -- 172.21.15.77:0/3115849946 shutdown_connections 2022-04-03T19:44:37.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.865+0000 7feaa4cffb80 1 --2- 172.21.15.77:0/3115849946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd002b40 0x55f2bd051160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.865+0000 7feaa4cffb80 1 --2- 172.21.15.77:0/3115849946 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd001f30 0x55f2bd038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.865+0000 7feaa4cffb80 1 --2- 172.21.15.77:0/3115849946 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 0x55f2bcfff020 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.865+0000 7feaa4cffb80 1 -- 172.21.15.77:0/3115849946 >> 172.21.15.77:0/3115849946 conn(0x55f2bd001990 msgr2=0x55f2bd041030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:37.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.865+0000 7feaa4cffb80 1 -- 172.21.15.77:0/3115849946 shutdown_connections 2022-04-03T19:44:37.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.865+0000 7feaa4cffb80 1 -- 172.21.15.77:0/3115849946 wait complete. 2022-04-03T19:44:37.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.865+0000 7feaa4cffb80 1 -- start start 2022-04-03T19:44:37.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.866+0000 7feaa4cffb80 1 Processor -- start 2022-04-03T19:44:37.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.867+0000 7feaa4cffb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd001f30 0x55f2bd051480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.867+0000 7feaa4cffb80 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 0x55f2bd0abfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.867+0000 7feaa4cffb80 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd002b40 0x55f2bd0a26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.867+0000 7feaa4cffb80 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2bd0519c0 con 0x55f2bd002300 2022-04-03T19:44:37.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.867+0000 7feaa4cffb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2bd0ac790 con 0x55f2bd001f30 2022-04-03T19:44:37.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.867+0000 7feaa4cffb80 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f2bcffd570 con 0x55f2bd002b40 2022-04-03T19:44:37.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.867+0000 7feaa4cffb80 10 client.0 _ll_register_callbacks cb 0x55f2bce423d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-04-03T19:44:37.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.867+0000 7fea98928700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 0x55f2bd0abfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.867+0000 7fea98928700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 0x55f2bd0abfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42650/0 (socket says 172.21.15.77:42650) 2022-04-03T19:44:37.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.867+0000 7fea98928700 1 -- 172.21.15.77:0/2344817897 learned_addr learned my addr 172.21.15.77:0/2344817897 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:37.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.867+0000 7fea9992a700 1 --2- 172.21.15.77:0/2344817897 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd002b40 0x55f2bd0a26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.867+0000 7fea99129700 1 --2- 172.21.15.77:0/2344817897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd001f30 0x55f2bd051480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.867+0000 7fea98928700 1 -- 172.21.15.77:0/2344817897 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd002b40 msgr2=0x55f2bd0a26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:37.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.867+0000 7fea98928700 1 --2- 172.21.15.77:0/2344817897 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd002b40 0x55f2bd0a26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.867+0000 7fea98928700 1 -- 172.21.15.77:0/2344817897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd001f30 msgr2=0x55f2bd051480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:37.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.867+0000 7fea98928700 1 --2- 172.21.15.77:0/2344817897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd001f30 0x55f2bd051480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.867+0000 7fea98928700 1 -- 172.21.15.77:0/2344817897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea84029e70 con 0x55f2bd002300 2022-04-03T19:44:37.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.867+0000 7fea99129700 1 --2- 172.21.15.77:0/2344817897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd001f30 0x55f2bd051480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:44:37.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.867+0000 7fea9992a700 1 --2- 172.21.15.77:0/2344817897 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd002b40 0x55f2bd0a26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:44:37.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.868+0000 7fea98928700 1 --2- 172.21.15.77:0/2344817897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 0x55f2bd0abfd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fea84034ce0 tx=0x7fea8402f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22657d7d608de3b6 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:37.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.868+0000 7fea6f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.77:3300/0 2022-04-03T19:44:37.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.868+0000 7fea6f7fe700 1 -- 172.21.15.77:0/2344817897 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea84009210 con 0x55f2bd002300 2022-04-03T19:44:37.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.868+0000 7fea6f7fe700 1 -- 172.21.15.77:0/2344817897 <== mon.0 v2:172.21.15.77:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fea8401a030 con 0x55f2bd002300 2022-04-03T19:44:37.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.868+0000 7feaa4cffb80 1 -- 172.21.15.77:0/2344817897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f2bcff6c10 con 0x55f2bd002300 2022-04-03T19:44:37.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.868+0000 7fea6f7fe700 1 -- 172.21.15.77:0/2344817897 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea8404b640 con 0x55f2bd002300 2022-04-03T19:44:37.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.868+0000 7feaa4cffb80 10 client.4596 fetch_fsmap learned FSMap version 8 2022-04-03T19:44:37.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.868+0000 7feaa4cffb80 1 -- 172.21.15.77:0/2344817897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55f2bcd94b10 con 0x55f2bd002300 2022-04-03T19:44:37.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.868+0000 7fea6f7fe700 1 -- 172.21.15.77:0/2344817897 <== mon.0 v2:172.21.15.77:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f2bcff6c10 con 0x55f2bd002300 2022-04-03T19:44:37.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.869+0000 7fea6f7fe700 1 -- 172.21.15.77:0/2344817897 <== mon.0 v2:172.21.15.77:3300/0 5 ==== fsmap(e 8) v1 ==== 4190+0+0 (secure 0 0 0) 0x7fea8404b7a0 con 0x55f2bd002300 2022-04-03T19:44:37.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.869+0000 7feaa4cffb80 10 client.4596 fetch_fsmap finished waiting for FSMap version 8 2022-04-03T19:44:37.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.869+0000 7feaa4cffb80 10 client.4596 resolve_mds: resolved name 'e' to daemon mds.e 2022-04-03T19:44:37.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.869+0000 7feaa4cffb80 20 client.4596 populate_metadata read hostname 'smithi077' 2022-04-03T19:44:37.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.869+0000 7feaa4cffb80 1 --2- 172.21.15.77:0/2344817897 >> [v2:172.21.15.77:6836/3665969042,v1:172.21.15.77:6837/3665969042] conn(0x55f2bd0a2bf0 0x55f2bd04f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.869+0000 7feaa4cffb80 4 client.4596 mds_command: new command op to 4327 tid=0[{"prefix": "status"}] 2022-04-03T19:44:37.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.869+0000 7feaa4cffb80 1 -- 172.21.15.77:0/2344817897 --> [v2:172.21.15.77:6836/3665969042,v1:172.21.15.77:6837/3665969042] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f2bcef76b0 con 0x55f2bd0a2bf0 2022-04-03T19:44:37.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.869+0000 7fea99129700 1 --2- 172.21.15.77:0/2344817897 >> [v2:172.21.15.77:6836/3665969042,v1:172.21.15.77:6837/3665969042] conn(0x55f2bd0a2bf0 0x55f2bd04f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.869+0000 7fea6f7fe700 10 client.4596 ms_handle_connect on v2:172.21.15.77:6836/3665969042 2022-04-03T19:44:37.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.869+0000 7fea99129700 1 --2- 172.21.15.77:0/2344817897 >> [v2:172.21.15.77:6836/3665969042,v1:172.21.15.77:6837/3665969042] conn(0x55f2bd0a2bf0 0x55f2bd04f6e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5df587264d232b9b server_cookie=33905f13c8636ff9 in_seq=0 out_seq=0 2022-04-03T19:44:37.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.870+0000 7fea6f7fe700 1 -- 172.21.15.77:0/2344817897 <== mds.0 v2:172.21.15.77:6836/3665969042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x55f2bcef76b0 con 0x55f2bd0a2bf0 2022-04-03T19:44:37.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.870+0000 7fea6f7fe700 10 client.4596 handle_command_reply: tid=0 2022-04-03T19:44:37.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.870+0000 7feaa4cffb80 1 client.4596 shutdown 2022-04-03T19:44:37.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.870+0000 7fea99129700 1 -- 172.21.15.77:0/2344817897 >> [v2:172.21.15.77:6836/3665969042,v1:172.21.15.77:6837/3665969042] conn(0x55f2bd0a2bf0 msgr2=0x55f2bd04f6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2022-04-03T19:44:37.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.870+0000 7fea99129700 1 -- 172.21.15.77:0/2344817897 >> [v2:172.21.15.77:6836/3665969042,v1:172.21.15.77:6837/3665969042] conn(0x55f2bd0a2bf0 msgr2=0x55f2bd04f6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-04-03T19:44:37.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.870+0000 7fea99129700 1 --2- 172.21.15.77:0/2344817897 >> [v2:172.21.15.77:6836/3665969042,v1:172.21.15.77:6837/3665969042] conn(0x55f2bd0a2bf0 0x55f2bd04f6e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-04-03T19:44:37.137 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.870+0000 7feaa4cffb80 1 -- 172.21.15.77:0/2344817897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 msgr2=0x55f2bd0abfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:37.137 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.870+0000 7feaa4cffb80 1 --2- 172.21.15.77:0/2344817897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 0x55f2bd0abfd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fea84034ce0 tx=0x7fea8402f000 comp rx=0 tx=0).stop 2022-04-03T19:44:37.137 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.870+0000 7fea99129700 1 --2- 172.21.15.77:0/2344817897 >> [v2:172.21.15.77:6836/3665969042,v1:172.21.15.77:6837/3665969042] conn(0x55f2bd0a2bf0 0x55f2bd04f6e0 unknown :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2022-04-03T19:44:37.137 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.870+0000 7feaa4cffb80 1 -- 172.21.15.77:0/2344817897 shutdown_connections 2022-04-03T19:44:37.137 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.870+0000 7feaa4cffb80 1 --2- 172.21.15.77:0/2344817897 >> [v2:172.21.15.77:6836/3665969042,v1:172.21.15.77:6837/3665969042] conn(0x55f2bd0a2bf0 0x55f2bd04f6e0 unknown :-1 s=BANNER_CONNECTING pgs=13 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.138 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.870+0000 7feaa4cffb80 1 --2- 172.21.15.77:0/2344817897 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55f2bd002b40 0x55f2bd0a26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.138 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.870+0000 7feaa4cffb80 1 --2- 172.21.15.77:0/2344817897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f2bd001f30 0x55f2bd051480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.138 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.870+0000 7feaa4cffb80 1 --2- 172.21.15.77:0/2344817897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f2bd002300 0x55f2bd0abfd0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.138 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.870+0000 7fea6f7fe700 0 client.4596 ms_handle_reset on v2:172.21.15.77:6836/3665969042 2022-04-03T19:44:37.139 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.870+0000 7feaa4cffb80 1 -- 172.21.15.77:0/2344817897 >> 172.21.15.77:0/2344817897 conn(0x55f2bd001990 msgr2=0x55f2bcffefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:37.139 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.870+0000 7feaa4cffb80 1 -- 172.21.15.77:0/2344817897 shutdown_connections 2022-04-03T19:44:37.139 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.870+0000 7feaa4cffb80 1 -- 172.21.15.77:0/2344817897 wait complete. 2022-04-03T19:44:37.139 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.870+0000 7feaa4cffb80 20 client.4596 trim_cache size 0 max 16384 2022-04-03T19:44:37.140 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2022-04-03T19:44:37.140 INFO:teuthology.orchestra.run.smithi077.stdout: "cluster_fsid": "89c3a3f4-599d-4c4a-8f5a-7180eafb7292", 2022-04-03T19:44:37.140 INFO:teuthology.orchestra.run.smithi077.stdout: "whoami": 0, 2022-04-03T19:44:37.141 INFO:teuthology.orchestra.run.smithi077.stdout: "id": 4327, 2022-04-03T19:44:37.141 INFO:teuthology.orchestra.run.smithi077.stdout: "want_state": "up:active", 2022-04-03T19:44:37.141 INFO:teuthology.orchestra.run.smithi077.stdout: "state": "up:active", 2022-04-03T19:44:37.141 INFO:teuthology.orchestra.run.smithi077.stdout: "fs_name": "cephfs", 2022-04-03T19:44:37.141 INFO:teuthology.orchestra.run.smithi077.stdout: "rank_uptime": 15.591807492999999, 2022-04-03T19:44:37.142 INFO:teuthology.orchestra.run.smithi077.stdout: "mdsmap_epoch": 8, 2022-04-03T19:44:37.142 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch": 24, 2022-04-03T19:44:37.142 INFO:teuthology.orchestra.run.smithi077.stdout: "osdmap_epoch_barrier": 24, 2022-04-03T19:44:37.142 INFO:teuthology.orchestra.run.smithi077.stdout: "uptime": 21.906183117000001 2022-04-03T19:44:37.143 INFO:teuthology.orchestra.run.smithi077.stdout:} 2022-04-03T19:44:37.144 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.871+0000 7fea9a12b700 1 -- 172.21.15.77:0/2615677343 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fea74063ce0 msgr2=0x7fea74066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:37.145 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.871+0000 7fea9a12b700 1 --2- 172.21.15.77:0/2615677343 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fea74063ce0 0x7fea74066190 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fea9414b650 tx=0x7fea840090f0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.145 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.871+0000 7fea9a12b700 1 -- 172.21.15.77:0/2615677343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea9406bbe0 msgr2=0x7fea94164bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:37.145 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.871+0000 7fea9a12b700 1 --2- 172.21.15.77:0/2615677343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea9406bbe0 0x7fea94164bf0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fea90010630 tx=0x7fea90016000 comp rx=0 tx=0).stop 2022-04-03T19:44:37.145 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.871+0000 7fea9a12b700 1 -- 172.21.15.77:0/2615677343 shutdown_connections 2022-04-03T19:44:37.146 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.871+0000 7fea9a12b700 1 --2- 172.21.15.77:0/2615677343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea9406c500 0x7fea940ca650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.146 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.871+0000 7fea9a12b700 1 --2- 172.21.15.77:0/2615677343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea9406bbe0 0x7fea94164bf0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.146 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.871+0000 7fea9a12b700 1 --2- 172.21.15.77:0/2615677343 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fea74063ce0 0x7fea74066190 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.146 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.871+0000 7fea9a12b700 1 --2- 172.21.15.77:0/2615677343 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea9406bfb0 0x7fea940c0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.146 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.871+0000 7fea9a12b700 1 -- 172.21.15.77:0/2615677343 >> 172.21.15.77:0/2615677343 conn(0x7fea940cb6b0 msgr2=0x7fea940cbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:37.147 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.871+0000 7fea9a12b700 1 -- 172.21.15.77:0/2615677343 shutdown_connections 2022-04-03T19:44:37.147 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:36.871+0000 7fea9a12b700 1 -- 172.21.15.77:0/2615677343 wait complete. 2022-04-03T19:44:37.147 INFO:teuthology.run_tasks:Running task kclient... 2022-04-03T19:44:37.168 INFO:tasks.kclient:Mounting kernel clients... 2022-04-03T19:44:37.169 INFO:tasks.kclient:config is {'client.0': {}} 2022-04-03T19:44:37.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-04-03T19:44:37.170 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2022-04-03T19:44:37.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-04-03T19:44:37.216 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link delete ceph-brx 2022-04-03T19:44:37.332 INFO:teuthology.orchestra.run.smithi077.stderr:Cannot find device "ceph-brx" 2022-04-03T19:44:37.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-04-03T19:44:37.336 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2022-04-03T19:44:37.337 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2022-04-03T19:44:37.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.534+0000 7fa7711a6700 1 Processor -- start 2022-04-03T19:44:37.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.534+0000 7fa7711a6700 1 -- start start 2022-04-03T19:44:37.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.535+0000 7fa7711a6700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa76c06bbe0 0x7fa76c1584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.535+0000 7fa7711a6700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa76c06c500 0x7fa76c14eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.535+0000 7fa7711a6700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76c06d260 con 0x7fa76c06bfb0 2022-04-03T19:44:37.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.535+0000 7fa7711a6700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa76c159450 con 0x7fa76c06c500 2022-04-03T19:44:37.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.535+0000 7fa7711a6700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa76c0c6330 con 0x7fa76c06bbe0 2022-04-03T19:44:37.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.535+0000 7fa76bfff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa76c06bbe0 0x7fa76c1584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.535+0000 7fa7709a5700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa76c06c500 0x7fa76c14eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.535+0000 7fa76b7fe700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fa76c06bfb0 0x7fa76c06d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44142/0 (socket says 172.21.15.77:44142) 2022-04-03T19:44:37.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.535+0000 7fa76b7fe700 1 -- 172.21.15.77:0/503609446 learned_addr learned my addr 172.21.15.77:0/503609446 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:37.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.536+0000 7fa76affd700 1 -- 172.21.15.77:0/503609446 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608900870 0 0) 0x7fa76c06d260 con 0x7fa76c06bfb0 2022-04-03T19:44:37.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.536+0000 7fa76affd700 1 -- 172.21.15.77:0/503609446 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7580033f0 con 0x7fa76c06bfb0 2022-04-03T19:44:37.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.536+0000 7fa7709a5700 1 -- 172.21.15.77:0/503609446 >> v2:172.21.15.186:3300/0 conn(0x7fa76c06bbe0 msgr2=0x7fa76c1584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:37.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.536+0000 7fa7709a5700 1 --2- 172.21.15.77:0/503609446 >> v2:172.21.15.186:3300/0 conn(0x7fa76c06bbe0 0x7fa76c1584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.540 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.536+0000 7fa7709a5700 1 -- 172.21.15.77:0/503609446 >> v1:172.21.15.186:6789/0 conn(0x7fa76c06bfb0 legacy=0x7fa76c06d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:37.540 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.536+0000 7fa7709a5700 1 -- 172.21.15.77:0/503609446 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c158bb0 con 0x7fa76c06c500 2022-04-03T19:44:37.540 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.536+0000 7fa7709a5700 1 --2- 172.21.15.77:0/503609446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa76c06c500 0x7fa76c14eea0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa76c088d40 tx=0x7fa764009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=811c5ab1549d1a24 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:37.540 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.536+0000 7fa76affd700 1 -- 172.21.15.77:0/503609446 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76400e070 con 0x7fa76c06c500 2022-04-03T19:44:37.540 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.536+0000 7fa76affd700 1 -- 172.21.15.77:0/503609446 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76400bab0 con 0x7fa76c06c500 2022-04-03T19:44:37.541 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.536+0000 7fa76affd700 1 -- 172.21.15.77:0/503609446 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76400be70 con 0x7fa76c06c500 2022-04-03T19:44:37.544 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.537+0000 7fa7711a6700 1 -- 172.21.15.77:0/503609446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa76c06c500 msgr2=0x7fa76c14eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:37.544 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.537+0000 7fa7711a6700 1 --2- 172.21.15.77:0/503609446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa76c06c500 0x7fa76c14eea0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa76c088d40 tx=0x7fa764009780 comp rx=0 tx=0).stop 2022-04-03T19:44:37.545 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.537+0000 7fa7711a6700 1 -- 172.21.15.77:0/503609446 shutdown_connections 2022-04-03T19:44:37.545 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.537+0000 7fa7711a6700 1 --2- 172.21.15.77:0/503609446 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa76c06c500 0x7fa76c14eea0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.545 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.537+0000 7fa7711a6700 1 --2- 172.21.15.77:0/503609446 >> v2:172.21.15.186:3300/0 conn(0x7fa76c06bbe0 0x7fa76c1584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.545 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.537+0000 7fa7711a6700 1 -- 172.21.15.77:0/503609446 >> 172.21.15.77:0/503609446 conn(0x7fa76c0cb6b0 msgr2=0x7fa76c0cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:37.545 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.537+0000 7fa7711a6700 1 -- 172.21.15.77:0/503609446 shutdown_connections 2022-04-03T19:44:37.546 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.537+0000 7fa7711a6700 1 -- 172.21.15.77:0/503609446 wait complete. 2022-04-03T19:44:37.546 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.539+0000 7fa7711a6700 1 Processor -- start 2022-04-03T19:44:37.546 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.539+0000 7fa7711a6700 1 -- start start 2022-04-03T19:44:37.546 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.540+0000 7fa7711a6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa76c06bbe0 0x7fa76c166230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.547 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.540+0000 7fa7711a6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa76c06bfb0 0x7fa76c166770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.547 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.540+0000 7fa7711a6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa76c06c500 0x7fa76c14f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.547 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.540+0000 7fa7711a6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c151c00 con 0x7fa76c06bbe0 2022-04-03T19:44:37.547 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.540+0000 7fa7711a6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c151680 con 0x7fa76c06c500 2022-04-03T19:44:37.548 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.540+0000 7fa7711a6700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c1507b0 con 0x7fa76c06bfb0 2022-04-03T19:44:37.548 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.540+0000 7fa76bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa76c06bbe0 0x7fa76c166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.548 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.540+0000 7fa76bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa76c06bbe0 0x7fa76c166230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42654/0 (socket says 172.21.15.77:42654) 2022-04-03T19:44:37.548 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.540+0000 7fa76bfff700 1 -- 172.21.15.77:0/272338888 learned_addr learned my addr 172.21.15.77:0/272338888 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:37.548 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.540+0000 7fa76b7fe700 1 --2- 172.21.15.77:0/272338888 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa76c06bfb0 0x7fa76c166770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.549 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.540+0000 7fa7709a5700 1 --2- 172.21.15.77:0/272338888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa76c06c500 0x7fa76c14f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.549 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.541+0000 7fa76bfff700 1 -- 172.21.15.77:0/272338888 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa76c06bfb0 msgr2=0x7fa76c166770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:37.549 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.541+0000 7fa76bfff700 1 --2- 172.21.15.77:0/272338888 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa76c06bfb0 0x7fa76c166770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.549 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.541+0000 7fa76bfff700 1 -- 172.21.15.77:0/272338888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa76c06c500 msgr2=0x7fa76c14f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:37.549 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.541+0000 7fa76bfff700 1 --2- 172.21.15.77:0/272338888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa76c06c500 0x7fa76c14f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.550 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.541+0000 7fa76bfff700 1 -- 172.21.15.77:0/272338888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa764009a20 con 0x7fa76c06bbe0 2022-04-03T19:44:37.550 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.541+0000 7fa7709a5700 1 --2- 172.21.15.77:0/272338888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa76c06c500 0x7fa76c14f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:37.550 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.541+0000 7fa76bfff700 1 --2- 172.21.15.77:0/272338888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa76c06bbe0 0x7fa76c166230 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa75c00e900 tx=0x7fa75c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:37.550 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.541+0000 7fa753fff700 1 -- 172.21.15.77:0/272338888 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c013680 con 0x7fa76c06bbe0 2022-04-03T19:44:37.551 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.541+0000 7fa7711a6700 1 -- 172.21.15.77:0/272338888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa76c152650 con 0x7fa76c06bbe0 2022-04-03T19:44:37.551 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.541+0000 7fa753fff700 1 -- 172.21.15.77:0/272338888 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75c0137e0 con 0x7fa76c06bbe0 2022-04-03T19:44:37.551 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.541+0000 7fa753fff700 1 -- 172.21.15.77:0/272338888 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c013b80 con 0x7fa76c06bbe0 2022-04-03T19:44:37.551 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.541+0000 7fa7711a6700 1 -- 172.21.15.77:0/272338888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa76c150ce0 con 0x7fa76c06bbe0 2022-04-03T19:44:37.551 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.542+0000 7fa7711a6700 1 -- 172.21.15.77:0/272338888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa76c166080 con 0x7fa76c06bbe0 2022-04-03T19:44:37.552 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.543+0000 7fa753fff700 1 -- 172.21.15.77:0/272338888 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fa75c023070 con 0x7fa76c06bbe0 2022-04-03T19:44:37.552 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.544+0000 7fa753fff700 1 --2- 172.21.15.77:0/272338888 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa74c063ce0 0x7fa74c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:37.552 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.544+0000 7fa753fff700 1 -- 172.21.15.77:0/272338888 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fa75c0809e0 con 0x7fa76c06bbe0 2022-04-03T19:44:37.553 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.544+0000 7fa76b7fe700 1 --2- 172.21.15.77:0/272338888 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa74c063ce0 0x7fa74c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:37.553 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.545+0000 7fa76b7fe700 1 --2- 172.21.15.77:0/272338888 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa74c063ce0 0x7fa74c066190 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa76c14a030 tx=0x7fa760009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:37.553 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.548+0000 7fa753fff700 1 -- 172.21.15.77:0/272338888 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fa75c04f5c0 con 0x7fa76c06bbe0 2022-04-03T19:44:37.785 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.785+0000 7fa7711a6700 1 -- 172.21.15.77:0/272338888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa76c04b050 con 0x7fa76c06bbe0 2022-04-03T19:44:37.786 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.785+0000 7fa753fff700 1 -- 172.21.15.77:0/272338888 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa75c017070 con 0x7fa76c06bbe0 2022-04-03T19:44:37.786 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2022-04-03T19:44:37.788 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.787+0000 7fa7711a6700 1 -- 172.21.15.77:0/272338888 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa74c063ce0 msgr2=0x7fa74c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:37.788 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.787+0000 7fa7711a6700 1 --2- 172.21.15.77:0/272338888 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa74c063ce0 0x7fa74c066190 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa76c14a030 tx=0x7fa760009040 comp rx=0 tx=0).stop 2022-04-03T19:44:37.789 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.787+0000 7fa7711a6700 1 -- 172.21.15.77:0/272338888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa76c06bbe0 msgr2=0x7fa76c166230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:37.789 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.787+0000 7fa7711a6700 1 --2- 172.21.15.77:0/272338888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa76c06bbe0 0x7fa76c166230 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa75c00e900 tx=0x7fa75c009060 comp rx=0 tx=0).stop 2022-04-03T19:44:37.789 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.787+0000 7fa7711a6700 1 -- 172.21.15.77:0/272338888 shutdown_connections 2022-04-03T19:44:37.789 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.787+0000 7fa7711a6700 1 --2- 172.21.15.77:0/272338888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa76c06c500 0x7fa76c14f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.789 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.787+0000 7fa7711a6700 1 --2- 172.21.15.77:0/272338888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa76c06bbe0 0x7fa76c166230 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.790 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.787+0000 7fa7711a6700 1 --2- 172.21.15.77:0/272338888 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa74c063ce0 0x7fa74c066190 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.790 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.787+0000 7fa7711a6700 1 --2- 172.21.15.77:0/272338888 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa76c06bfb0 0x7fa76c166770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:37.790 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.788+0000 7fa7711a6700 1 -- 172.21.15.77:0/272338888 >> 172.21.15.77:0/272338888 conn(0x7fa76c0cb6b0 msgr2=0x7fa76c0c7680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:37.790 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.788+0000 7fa7711a6700 1 -- 172.21.15.77:0/272338888 shutdown_connections 2022-04-03T19:44:37.791 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:37.788+0000 7fa7711a6700 1 -- 172.21.15.77:0/272338888 wait complete. 2022-04-03T19:44:37.813 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2022-04-03T19:44:37.813 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2022-04-03T19:44:37.814 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2022-04-03T19:44:37.814 INFO:tasks.cephfs.mount:self.client.name = client.0 2022-04-03T19:44:37.814 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2022-04-03T19:44:37.814 INFO:tasks.cephfs.mount:self.cephfs_name = None 2022-04-03T19:44:37.815 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2022-04-03T19:44:37.815 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2022-04-03T19:44:37.815 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2022-04-03T19:44:37.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-04-03T19:44:37.816 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2022-04-03T19:44:37.860 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-04-03T19:44:37.860 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-04-03T19:44:37.860 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2022-04-03T19:44:37.860 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2022-04-03T19:44:37.861 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2022-04-03T19:44:37.861 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2022-04-03T19:44:37.861 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-04-03T19:44:37.861 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2022-04-03T19:44:37.862 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-04-03T19:44:37.862 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2022-04-03T19:44:37.862 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-04-03T19:44:37.862 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2022-04-03T19:44:37.862 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-04-03T19:44:37.863 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2022-04-03T19:44:37.863 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2022-04-03T19:44:37.863 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42606sec preferred_lft 42606sec 2022-04-03T19:44:37.863 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link noprefixroute 2022-04-03T19:44:37.863 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2022-04-03T19:44:37.865 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2022-04-03T19:44:37.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-04-03T19:44:37.865 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2022-04-03T19:44:37.865 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2022-04-03T19:44:37.866 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add name ceph-brx type bridge 2022-04-03T19:44:37.866 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip addr flush dev ceph-brx 2022-04-03T19:44:37.867 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set ceph-brx up 2022-04-03T19:44:37.867 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2022-04-03T19:44:37.867 DEBUG:teuthology.orchestra.run.smithi077:> ') 2022-04-03T19:44:38.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-04-03T19:44:38.326 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2022-04-03T19:44:38.465 INFO:teuthology.orchestra.run.smithi077.stdout:1 2022-04-03T19:44:38.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-04-03T19:44:38.471 DEBUG:teuthology.orchestra.run.smithi077:> route 2022-04-03T19:44:38.534 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2022-04-03T19:44:38.534 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2022-04-03T19:44:38.534 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2022-04-03T19:44:38.535 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2022-04-03T19:44:38.535 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2022-04-03T19:44:38.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-04-03T19:44:38.537 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2022-04-03T19:44:38.537 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2022-04-03T19:44:38.537 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2022-04-03T19:44:38.538 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2022-04-03T19:44:38.538 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2022-04-03T19:44:38.538 DEBUG:teuthology.orchestra.run.smithi077:> ') 2022-04-03T19:44:39.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-04-03T19:44:39.218 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2022-04-03T19:44:39.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-04-03T19:44:39.265 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list-id 2022-04-03T19:44:39.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-04-03T19:44:39.349 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2022-04-03T19:44:39.349 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2022-04-03T19:44:39.350 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2022-04-03T19:44:39.350 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2022-04-03T19:44:39.350 DEBUG:teuthology.orchestra.run.smithi077:> ') 2022-04-03T19:44:39.496 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2022-04-03T19:44:39.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-04-03T19:44:39.497 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2022-04-03T19:44:39.497 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2022-04-03T19:44:39.497 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2022-04-03T19:44:39.498 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2022-04-03T19:44:39.498 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2022-04-03T19:44:39.499 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2022-04-03T19:44:39.499 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2022-04-03T19:44:39.499 DEBUG:teuthology.orchestra.run.smithi077:> ') 2022-04-03T19:44:39.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-04-03T19:44:39.785 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2022-04-03T19:44:39.786 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2022-04-03T19:44:39.786 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2022-04-03T19:44:39.786 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set dev brx.0 master ceph-brx 2022-04-03T19:44:39.787 DEBUG:teuthology.orchestra.run.smithi077:> ') 2022-04-03T19:44:39.966 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-04-03T19:44:39.966 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2022-04-03T19:44:40.010 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2022-04-03T19:44:40.011 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-04-03T19:44:40.011 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2022-04-03T19:44:40.094 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2022-04-03T19:44:40.095 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-04-03T19:44:40.095 DEBUG:teuthology.orchestra.run.smithi077:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2022-04-03T19:44:40.206 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2022-04-03T19:44:40.207 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2022-04-03T19:44:40.207 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2022-04-03T19:44:40.651 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2022-04-03T19:44:40.651 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2022-04-03T19:44:40.651 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2022-04-03T19:44:40.651 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2022-04-03T19:44:40.681 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2022-04-03T19:44:40.681 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2022-04-03T19:44:40.682 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2022-04-03T19:44:40.682 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2022-04-03T19:44:40.682 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2022-04-03T19:44:40.682 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=89c3a3f4-599d-4c4a-8f5a-7180eafb7292" will pass to kernel 2022-04-03T19:44:40.691 INFO:tasks.cephfs.kernel_mount:mount command passed 2022-04-03T19:44:40.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-04-03T19:44:40.692 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2022-04-03T19:44:40.763 INFO:teuthology.run_tasks:Running task mds_thrash... 2022-04-03T19:44:40.778 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-04-03T19:44:40.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.981+0000 7fe336af0700 1 Processor -- start 2022-04-03T19:44:40.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.981+0000 7fe336af0700 1 -- start start 2022-04-03T19:44:40.983 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.982+0000 7fe336af0700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe33006bbe0 0x7fe3301584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:40.983 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.982+0000 7fe336af0700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe33006bfb0 0x7fe33006d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:40.983 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.982+0000 7fe336af0700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe33006d260 con 0x7fe33006c500 2022-04-03T19:44:40.983 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.982+0000 7fe336af0700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe330159450 con 0x7fe33006bfb0 2022-04-03T19:44:40.984 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.982+0000 7fe336af0700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3300c6330 con 0x7fe33006bbe0 2022-04-03T19:44:40.984 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.982+0000 7fe3352ed700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe33006bfb0 0x7fe33006d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:40.984 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.982+0000 7fe335aee700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe33006bbe0 0x7fe3301584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:40.984 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.983+0000 7fe3352ed700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe33006bfb0 0x7fe33006d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42656/0 (socket says 172.21.15.77:42656) 2022-04-03T19:44:40.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.983+0000 7fe3352ed700 1 -- 172.21.15.77:0/1396168965 learned_addr learned my addr 172.21.15.77:0/1396168965 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:40.986 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.983+0000 7fe335aee700 1 -- 172.21.15.77:0/1396168965 >> v2:172.21.15.77:3300/0 conn(0x7fe33006bfb0 msgr2=0x7fe33006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:40.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.983+0000 7fe335aee700 1 --2- 172.21.15.77:0/1396168965 >> v2:172.21.15.77:3300/0 conn(0x7fe33006bfb0 0x7fe33006d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:40.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.983+0000 7fe335aee700 1 -- 172.21.15.77:0/1396168965 >> v1:172.21.15.186:6789/0 conn(0x7fe33006c500 legacy=0x7fe33014eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:40.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.983+0000 7fe335aee700 1 -- 172.21.15.77:0/1396168965 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe330158bb0 con 0x7fe33006bbe0 2022-04-03T19:44:40.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.983+0000 7fe335aee700 1 --2- 172.21.15.77:0/1396168965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe33006bbe0 0x7fe3301584b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe330088d40 tx=0x7fe32c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=874a5189233702eb server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:40.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.984+0000 7fe334aec700 1 -- 172.21.15.77:0/1396168965 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32c011630 con 0x7fe33006bbe0 2022-04-03T19:44:40.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.984+0000 7fe334aec700 1 -- 172.21.15.77:0/1396168965 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe32c011790 con 0x7fe33006bbe0 2022-04-03T19:44:40.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.984+0000 7fe334aec700 1 -- 172.21.15.77:0/1396168965 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32c011b30 con 0x7fe33006bbe0 2022-04-03T19:44:40.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.984+0000 7fe336af0700 1 -- 172.21.15.77:0/1396168965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe33006bbe0 msgr2=0x7fe3301584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:40.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.984+0000 7fe336af0700 1 --2- 172.21.15.77:0/1396168965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe33006bbe0 0x7fe3301584b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe330088d40 tx=0x7fe32c009060 comp rx=0 tx=0).stop 2022-04-03T19:44:40.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.984+0000 7fe336af0700 1 -- 172.21.15.77:0/1396168965 shutdown_connections 2022-04-03T19:44:40.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.984+0000 7fe336af0700 1 --2- 172.21.15.77:0/1396168965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe33006bbe0 0x7fe3301584b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:40.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.984+0000 7fe336af0700 1 --2- 172.21.15.77:0/1396168965 >> v2:172.21.15.77:3300/0 conn(0x7fe33006bfb0 0x7fe33006d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:40.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.984+0000 7fe336af0700 1 -- 172.21.15.77:0/1396168965 >> 172.21.15.77:0/1396168965 conn(0x7fe3300cb6b0 msgr2=0x7fe3300cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:40.990 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.984+0000 7fe336af0700 1 -- 172.21.15.77:0/1396168965 shutdown_connections 2022-04-03T19:44:40.990 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.984+0000 7fe336af0700 1 -- 172.21.15.77:0/1396168965 wait complete. 2022-04-03T19:44:40.990 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.986+0000 7fe336af0700 1 Processor -- start 2022-04-03T19:44:40.990 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.986+0000 7fe336af0700 1 -- start start 2022-04-03T19:44:40.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.987+0000 7fe336af0700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe33006bbe0 0x7fe330150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:40.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.987+0000 7fe336af0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe33006bfb0 0x7fe3301546c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:40.994 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.987+0000 7fe336af0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe33006c500 0x7fe330154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:40.994 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.987+0000 7fe336af0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe330152c20 con 0x7fe33006c500 2022-04-03T19:44:40.994 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.987+0000 7fe336af0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe330154230 con 0x7fe33006bfb0 2022-04-03T19:44:40.994 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.987+0000 7fe336af0700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe330152380 con 0x7fe33006bbe0 2022-04-03T19:44:40.995 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.988+0000 7fe3362ef700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe33006c500 0x7fe330154c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:40.995 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.988+0000 7fe3352ed700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe33006bfb0 0x7fe3301546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:40.995 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.988+0000 7fe3362ef700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe33006c500 0x7fe330154c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42658/0 (socket says 172.21.15.77:42658) 2022-04-03T19:44:40.995 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.988+0000 7fe3362ef700 1 -- 172.21.15.77:0/747955271 learned_addr learned my addr 172.21.15.77:0/747955271 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:40.995 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.988+0000 7fe335aee700 1 --2- 172.21.15.77:0/747955271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe33006bbe0 0x7fe330150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:40.996 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.988+0000 7fe3362ef700 1 -- 172.21.15.77:0/747955271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe33006bbe0 msgr2=0x7fe330150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:40.996 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.988+0000 7fe3362ef700 1 --2- 172.21.15.77:0/747955271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe33006bbe0 0x7fe330150970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:40.996 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.988+0000 7fe3362ef700 1 -- 172.21.15.77:0/747955271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe33006bfb0 msgr2=0x7fe3301546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:40.996 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.988+0000 7fe3362ef700 1 --2- 172.21.15.77:0/747955271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe33006bfb0 0x7fe3301546c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:40.996 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.988+0000 7fe3362ef700 1 -- 172.21.15.77:0/747955271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32c0092a0 con 0x7fe33006c500 2022-04-03T19:44:40.997 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.988+0000 7fe3352ed700 1 --2- 172.21.15.77:0/747955271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe33006bfb0 0x7fe3301546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:40.997 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.988+0000 7fe3362ef700 1 --2- 172.21.15.77:0/747955271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe33006c500 0x7fe330154c00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe32800c790 tx=0x7fe3280015e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:40.997 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.989+0000 7fe31dffb700 1 -- 172.21.15.77:0/747955271 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe328003840 con 0x7fe33006c500 2022-04-03T19:44:40.997 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.989+0000 7fe31dffb700 1 -- 172.21.15.77:0/747955271 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3280039a0 con 0x7fe33006c500 2022-04-03T19:44:40.998 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.989+0000 7fe336af0700 1 -- 172.21.15.77:0/747955271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe330152660 con 0x7fe33006c500 2022-04-03T19:44:40.998 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.989+0000 7fe31dffb700 1 -- 172.21.15.77:0/747955271 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe328003d20 con 0x7fe33006c500 2022-04-03T19:44:40.998 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.989+0000 7fe336af0700 1 -- 172.21.15.77:0/747955271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe330153ec0 con 0x7fe33006c500 2022-04-03T19:44:40.998 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.990+0000 7fe336af0700 1 -- 172.21.15.77:0/747955271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe330154370 con 0x7fe33006c500 2022-04-03T19:44:40.998 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.991+0000 7fe31dffb700 1 -- 172.21.15.77:0/747955271 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fe328019030 con 0x7fe33006c500 2022-04-03T19:44:40.999 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.992+0000 7fe31dffb700 1 --2- 172.21.15.77:0/747955271 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe310063d10 0x7fe3100661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:40.999 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.992+0000 7fe31dffb700 1 -- 172.21.15.77:0/747955271 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fe32807faf0 con 0x7fe33006c500 2022-04-03T19:44:40.999 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.993+0000 7fe335aee700 1 --2- 172.21.15.77:0/747955271 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe310063d10 0x7fe3100661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:41.000 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.993+0000 7fe335aee700 1 --2- 172.21.15.77:0/747955271 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe310063d10 0x7fe3100661c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe32c011fd0 tx=0x7fe32c017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:41.000 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:40.997+0000 7fe31dffb700 1 -- 172.21.15.77:0/747955271 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fe328048ae0 con 0x7fe33006c500 2022-04-03T19:44:41.191 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.191+0000 7fe336af0700 1 -- 172.21.15.77:0/747955271 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3300c7ec0 con 0x7fe33006c500 2022-04-03T19:44:41.192 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.192+0000 7fe31dffb700 1 -- 172.21.15.77:0/747955271 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16845 (secure 0 0 0) 0x7fe328017020 con 0x7fe33006c500 2022-04-03T19:44:41.193 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:41.193 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"89c3a3f4-599d-4c4a-8f5a-7180eafb7292","created":"2022-04-03T19:43:54.101559+0000","modified":"2022-04-03T19:44:22.278254+0000","last_up_change":"2022-04-03T19:44:05.589174+0000","last_in_change":"2022-04-03T19:43:56.729760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-03T19:44:07.567638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-04-03T19:44:13.149481+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-04-03T19:44:18.823926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-04-03T19:44:20.275479+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f35f5195-3729-4d78-85d1-0539f259283f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6823","nonce":20252},{"type":"v1","addr":"172.21.15.77:6825","nonce":20252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":20252},{"type":"v1","addr":"172.21.15.77:6827","nonce":20252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":20252},{"type":"v1","addr":"172.21.15.77:6831","nonce":20252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":20252},{"type":"v1","addr":"172.21.15.77:6829","nonce":20252}]},"public_addr":"172.21.15.77:6825/20252","cluster_addr":"172.21.15.77:6827/20252","heartbeat_back_addr":"172.21.15.77:6831/20252","heartbeat_front_addr":"172.21.15.77:6829/20252","state":["exists","up"]},{"osd":1,"uuid":"ab1c89dc-57ce-4cc4-b8c7-9b0c19939601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":20249},{"type":"v1","addr":"172.21.15.77:6805","nonce":20249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":20249},{"type":"v1","addr":"172.21.15.77:6809","nonce":20249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":20249},{"type":"v1","addr":"172.21.15.77:6819","nonce":20249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":20249},{"type":"v1","addr":"172.21.15.77:6815","nonce":20249}]},"public_addr":"172.21.15.77:6805/20249","cluster_addr":"172.21.15.77:6809/20249","heartbeat_back_addr":"172.21.15.77:6819/20249","heartbeat_front_addr":"172.21.15.77:6815/20249","state":["exists","up"]},{"osd":2,"uuid":"351e218b-7749-4ee9-b44a-0a5b52fb9be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":20248},{"type":"v1","addr":"172.21.15.77:6801","nonce":20248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":20248},{"type":"v1","addr":"172.21.15.77:6803","nonce":20248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":20248},{"type":"v1","addr":"172.21.15.77:6813","nonce":20248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":20248},{"type":"v1","addr":"172.21.15.77:6808","nonce":20248}]},"public_addr":"172.21.15.77:6801/20248","cluster_addr":"172.21.15.77:6803/20248","heartbeat_back_addr":"172.21.15.77:6813/20248","heartbeat_front_addr":"172.21.15.77:6808/20248","state":["exists","up"]},{"osd":3,"uuid":"1ba6ec8a-7cbf-4c12-8e21-c3097a263ae8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":20250},{"type":"v1","addr":"172.21.15.77:6814","nonce":20250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":20250},{"type":"v1","addr":"172.21.15.77:6818","nonce":20250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":20250},{"type":"v1","addr":"172.21.15.77:6824","nonce":20250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":20250},{"type":"v1","addr":"172.21.15.77:6821","nonce":20250}]},"public_addr":"172.21.15.77:6814/20250","cluster_addr":"172.21.15.77:6818/20250","heartbeat_back_addr":"172.21.15.77:6824/20250","heartbeat_front_addr":"172.21.15.77:6821/20250","state":["exists","up"]},{"osd":4,"uuid":"3faf4c09-846c-4133-856b-7f53c535a67d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":25551},{"type":"v1","addr":"172.21.15.186:6822","nonce":25551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":25551},{"type":"v1","addr":"172.21.15.186:6826","nonce":25551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":25551},{"type":"v1","addr":"172.21.15.186:6831","nonce":25551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":25551},{"type":"v1","addr":"172.21.15.186:6829","nonce":25551}]},"public_addr":"172.21.15.186:6822/25551","cluster_addr":"172.21.15.186:6826/25551","heartbeat_back_addr":"172.21.15.186:6831/25551","heartbeat_front_addr":"172.21.15.186:6829/25551","state":["exists","up"]},{"osd":5,"uuid":"ffe41d00-d0aa-42f4-82c1-5cced13c1bc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":25541},{"type":"v1","addr":"172.21.15.186:6809","nonce":25541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":25541},{"type":"v1","addr":"172.21.15.186:6811","nonce":25541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":25541},{"type":"v1","addr":"172.21.15.186:6815","nonce":25541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":25541},{"type":"v1","addr":"172.21.15.186:6813","nonce":25541}]},"public_addr":"172.21.15.186:6809/25541","cluster_addr":"172.21.15.186:6811/25541","heartbeat_back_addr":"172.21.15.186:6815/25541","heartbeat_front_addr":"172.21.15.186:6813/25541","state":["exists","up"]},{"osd":6,"uuid":"b0344ee8-deca-4e9b-934c-5866d90d0073","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":25542},{"type":"v1","addr":"172.21.15.186:6801","nonce":25542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":25542},{"type":"v1","addr":"172.21.15.186:6803","nonce":25542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":25542},{"type":"v1","addr":"172.21.15.186:6807","nonce":25542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":25542},{"type":"v1","addr":"172.21.15.186:6805","nonce":25542}]},"public_addr":"172.21.15.186:6801/25542","cluster_addr":"172.21.15.186:6803/25542","heartbeat_back_addr":"172.21.15.186:6807/25542","heartbeat_front_addr":"172.21.15.186:6805/25542","state":["exists","up"]},{"osd":7,"uuid":"0eb0e5d3-3dad-4f49-9b50-3b24a3924bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":25572},{"type":"v1","addr":"172.21.15.186:6817","nonce":25572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":25572},{"type":"v1","addr":"172.21.15.186:6819","nonce":25572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6825","nonce":25572},{"type":"v1","addr":"172.21.15.186:6827","nonce":25572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":25572},{"type":"v1","addr":"172.21.15.186:6823","nonce":25572}]},"public_addr":"172.21.15.186:6817/25572","cluster_addr":"172.21.15.186:6819/25572","heartbeat_back_addr":"172.21.15.186:6827/25572","heartbeat_front_addr":"172.21.15.186:6823/25572","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.143602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.490366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.361292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.343544+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.905429+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969298+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.949524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-03T19:44:41.195 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.194+0000 7fe336af0700 1 -- 172.21.15.77:0/747955271 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe310063d10 msgr2=0x7fe3100661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:41.195 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.194+0000 7fe336af0700 1 --2- 172.21.15.77:0/747955271 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe310063d10 0x7fe3100661c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe32c011fd0 tx=0x7fe32c017000 comp rx=0 tx=0).stop 2022-04-03T19:44:41.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.194+0000 7fe336af0700 1 -- 172.21.15.77:0/747955271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe33006c500 msgr2=0x7fe330154c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:41.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.194+0000 7fe336af0700 1 --2- 172.21.15.77:0/747955271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe33006c500 0x7fe330154c00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe32800c790 tx=0x7fe3280015e0 comp rx=0 tx=0).stop 2022-04-03T19:44:41.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.194+0000 7fe336af0700 1 -- 172.21.15.77:0/747955271 shutdown_connections 2022-04-03T19:44:41.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.195+0000 7fe336af0700 1 --2- 172.21.15.77:0/747955271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe33006c500 0x7fe330154c00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:41.197 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.195+0000 7fe336af0700 1 --2- 172.21.15.77:0/747955271 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe310063d10 0x7fe3100661c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:41.197 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.195+0000 7fe336af0700 1 --2- 172.21.15.77:0/747955271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe33006bbe0 0x7fe330150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:41.197 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.195+0000 7fe336af0700 1 --2- 172.21.15.77:0/747955271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe33006bfb0 0x7fe3301546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:41.197 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.195+0000 7fe336af0700 1 -- 172.21.15.77:0/747955271 >> 172.21.15.77:0/747955271 conn(0x7fe3300cb6b0 msgr2=0x7fe33014da80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:41.197 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.195+0000 7fe336af0700 1 -- 172.21.15.77:0/747955271 shutdown_connections 2022-04-03T19:44:41.198 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.195+0000 7fe336af0700 1 -- 172.21.15.77:0/747955271 wait complete. 2022-04-03T19:44:41.220 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-04-03T19:44:07.567638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-04-03T19:44:13.149481+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-04-03T19:44:18.823926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-04-03T19:44:20.275479+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-04-03T19:44:41.220 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-04-03T19:44:41.422 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.421+0000 7f8190e81700 1 Processor -- start 2022-04-03T19:44:41.422 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.421+0000 7f8190e81700 1 -- start start 2022-04-03T19:44:41.423 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.422+0000 7f8190e81700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f818c0c7410 0x7f818c0c1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:41.423 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.422+0000 7f8190e81700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f818c0c0af0 0x7f818c0c29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:41.424 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.422+0000 7f8190e81700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f818c155a50 con 0x7f818c06b0e0 2022-04-03T19:44:41.424 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.423+0000 7f8190e81700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f818c154e40 con 0x7f818c0c7410 2022-04-03T19:44:41.424 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.423+0000 7f8190e81700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f818c155120 con 0x7f818c0c0af0 2022-04-03T19:44:41.424 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.423+0000 7f818b7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f818c0c7410 0x7f818c0c1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:41.424 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.423+0000 7f818b7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f818c0c7410 0x7f818c0c1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42660/0 (socket says 172.21.15.77:42660) 2022-04-03T19:44:41.425 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.423+0000 7f818b7fe700 1 -- 172.21.15.77:0/2501604229 learned_addr learned my addr 172.21.15.77:0/2501604229 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:41.425 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.423+0000 7f818bfff700 1 --2- 172.21.15.77:0/2501604229 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f818c0c0af0 0x7f818c0c29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:41.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.423+0000 7f818b7fe700 1 -- 172.21.15.77:0/2501604229 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f818c0c0af0 msgr2=0x7f818c0c29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:41.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.423+0000 7f818b7fe700 1 --2- 172.21.15.77:0/2501604229 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f818c0c0af0 0x7f818c0c29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:41.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.423+0000 7f818b7fe700 1 -- 172.21.15.77:0/2501604229 >> v1:172.21.15.186:6789/0 conn(0x7f818c06b0e0 legacy=0x7f818c0c03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:41.428 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.423+0000 7f818a7fc700 1 -- 172.21.15.77:0/2501604229 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078546085 0 0) 0x7f818c155a50 con 0x7f818c06b0e0 2022-04-03T19:44:41.428 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.423+0000 7f818b7fe700 1 -- 172.21.15.77:0/2501604229 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f818c155400 con 0x7f818c0c7410 2022-04-03T19:44:41.428 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.424+0000 7f818b7fe700 1 --2- 172.21.15.77:0/2501604229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f818c0c7410 0x7f818c0c1b90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f817c006fd0 tx=0x7f817c00b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d30eac4f046da4d1 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:41.428 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.424+0000 7f818a7fc700 1 -- 172.21.15.77:0/2501604229 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c004030 con 0x7f818c0c7410 2022-04-03T19:44:41.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.424+0000 7f818a7fc700 1 -- 172.21.15.77:0/2501604229 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817c004190 con 0x7f818c0c7410 2022-04-03T19:44:41.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.424+0000 7f818a7fc700 1 -- 172.21.15.77:0/2501604229 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c00da70 con 0x7f818c0c7410 2022-04-03T19:44:41.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.424+0000 7f8190e81700 1 -- 172.21.15.77:0/2501604229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f818c0c7410 msgr2=0x7f818c0c1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:41.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.424+0000 7f8190e81700 1 --2- 172.21.15.77:0/2501604229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f818c0c7410 0x7f818c0c1b90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f817c006fd0 tx=0x7f817c00b3b0 comp rx=0 tx=0).stop 2022-04-03T19:44:41.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.425+0000 7f8190e81700 1 -- 172.21.15.77:0/2501604229 shutdown_connections 2022-04-03T19:44:41.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.425+0000 7f8190e81700 1 --2- 172.21.15.77:0/2501604229 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f818c0c0af0 0x7f818c0c29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:41.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.425+0000 7f8190e81700 1 --2- 172.21.15.77:0/2501604229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f818c0c7410 0x7f818c0c1b90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:41.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.425+0000 7f8190e81700 1 -- 172.21.15.77:0/2501604229 >> 172.21.15.77:0/2501604229 conn(0x7f818c0626d0 msgr2=0x7f818c062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:41.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.425+0000 7f8190e81700 1 -- 172.21.15.77:0/2501604229 shutdown_connections 2022-04-03T19:44:41.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.425+0000 7f8190e81700 1 -- 172.21.15.77:0/2501604229 wait complete. 2022-04-03T19:44:41.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.426+0000 7f8190e81700 1 Processor -- start 2022-04-03T19:44:41.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.426+0000 7f8190e81700 1 -- start start 2022-04-03T19:44:41.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.427+0000 7f8190e81700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f818c06b0e0 0x7f818c0c64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:41.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.427+0000 7f8190e81700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f818c0c0af0 0x7f818c0bc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:41.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.427+0000 7f8190e81700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f818c0c7410 0x7f818c150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:41.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.427+0000 7f8190e81700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818c0bac90 con 0x7f818c0c0af0 2022-04-03T19:44:41.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.427+0000 7f8190e81700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818c0bd830 con 0x7f818c0c7410 2022-04-03T19:44:41.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.427+0000 7f8190e81700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f818c0c69e0 con 0x7f818c06b0e0 2022-04-03T19:44:41.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.428+0000 7f818b7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f818c06b0e0 0x7f818c0c64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:41.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.428+0000 7f818affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f818c0c0af0 0x7f818c0bc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:41.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.428+0000 7f818b7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f818c06b0e0 0x7f818c0c64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59964/0 (socket says 172.21.15.77:59964) 2022-04-03T19:44:41.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.428+0000 7f818b7fe700 1 -- 172.21.15.77:0/1985071078 learned_addr learned my addr 172.21.15.77:0/1985071078 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:41.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.428+0000 7f818bfff700 1 --2- 172.21.15.77:0/1985071078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f818c0c7410 0x7f818c150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:41.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.428+0000 7f818affd700 1 -- 172.21.15.77:0/1985071078 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f818c06b0e0 msgr2=0x7f818c0c64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:41.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.428+0000 7f818affd700 1 --2- 172.21.15.77:0/1985071078 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f818c06b0e0 0x7f818c0c64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:41.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.428+0000 7f818affd700 1 -- 172.21.15.77:0/1985071078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f818c0c7410 msgr2=0x7f818c150c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:41.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.428+0000 7f818affd700 1 --2- 172.21.15.77:0/1985071078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f818c0c7410 0x7f818c150c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:41.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.428+0000 7f818affd700 1 -- 172.21.15.77:0/1985071078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817c00b610 con 0x7f818c0c0af0 2022-04-03T19:44:41.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.428+0000 7f818affd700 1 --2- 172.21.15.77:0/1985071078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f818c0c0af0 0x7f818c0bc460 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f818000c470 tx=0x7f8180007bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:41.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.429+0000 7f81737fe700 1 -- 172.21.15.77:0/1985071078 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818000ce30 con 0x7f818c0c0af0 2022-04-03T19:44:41.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.429+0000 7f81737fe700 1 -- 172.21.15.77:0/1985071078 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81800052e0 con 0x7f818c0c0af0 2022-04-03T19:44:41.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.429+0000 7f81737fe700 1 -- 172.21.15.77:0/1985071078 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8180003710 con 0x7f818c0c0af0 2022-04-03T19:44:41.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.429+0000 7f8190e81700 1 -- 172.21.15.77:0/1985071078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f818c0ba9b0 con 0x7f818c0c0af0 2022-04-03T19:44:41.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.429+0000 7f8190e81700 1 -- 172.21.15.77:0/1985071078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f818c0bd1e0 con 0x7f818c0c0af0 2022-04-03T19:44:41.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.430+0000 7f8190e81700 1 -- 172.21.15.77:0/1985071078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f818c165770 con 0x7f818c0c0af0 2022-04-03T19:44:41.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.431+0000 7f81737fe700 1 -- 172.21.15.77:0/1985071078 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f8180019030 con 0x7f818c0c0af0 2022-04-03T19:44:41.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.432+0000 7f81737fe700 1 --2- 172.21.15.77:0/1985071078 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f816c063ce0 0x7f816c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:41.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.432+0000 7f81737fe700 1 -- 172.21.15.77:0/1985071078 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f818007fa50 con 0x7f818c0c0af0 2022-04-03T19:44:41.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.432+0000 7f818b7fe700 1 --2- 172.21.15.77:0/1985071078 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f816c063ce0 0x7f816c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:41.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.433+0000 7f818b7fe700 1 --2- 172.21.15.77:0/1985071078 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f816c063ce0 0x7f816c066190 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f817c00b3f0 tx=0x7f817c01a000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:41.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.436+0000 7f81737fe700 1 -- 172.21.15.77:0/1985071078 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f818c165770 con 0x7f818c0c0af0 2022-04-03T19:44:41.626 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.626+0000 7f8190e81700 1 -- 172.21.15.77:0/1985071078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f818c165770 con 0x7f818c0c0af0 2022-04-03T19:44:41.627 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.626+0000 7f81737fe700 1 -- 172.21.15.77:0/1985071078 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f818c165770 con 0x7f818c0c0af0 2022-04-03T19:44:41.627 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2022-04-03T19:44:41.630 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.629+0000 7f8190e81700 1 -- 172.21.15.77:0/1985071078 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f816c063ce0 msgr2=0x7f816c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:41.630 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.629+0000 7f8190e81700 1 --2- 172.21.15.77:0/1985071078 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f816c063ce0 0x7f816c066190 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f817c00b3f0 tx=0x7f817c01a000 comp rx=0 tx=0).stop 2022-04-03T19:44:41.630 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.629+0000 7f8190e81700 1 -- 172.21.15.77:0/1985071078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f818c0c0af0 msgr2=0x7f818c0bc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:41.630 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.629+0000 7f8190e81700 1 --2- 172.21.15.77:0/1985071078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f818c0c0af0 0x7f818c0bc460 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f818000c470 tx=0x7f8180007bb0 comp rx=0 tx=0).stop 2022-04-03T19:44:41.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.629+0000 7f8190e81700 1 -- 172.21.15.77:0/1985071078 shutdown_connections 2022-04-03T19:44:41.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.629+0000 7f8190e81700 1 --2- 172.21.15.77:0/1985071078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f818c0c7410 0x7f818c150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:41.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.629+0000 7f8190e81700 1 --2- 172.21.15.77:0/1985071078 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f816c063ce0 0x7f816c066190 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:41.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.629+0000 7f8190e81700 1 --2- 172.21.15.77:0/1985071078 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f818c06b0e0 0x7f818c0c64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:41.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.629+0000 7f8190e81700 1 --2- 172.21.15.77:0/1985071078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f818c0c0af0 0x7f818c0bc460 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:41.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.629+0000 7f8190e81700 1 -- 172.21.15.77:0/1985071078 >> 172.21.15.77:0/1985071078 conn(0x7f818c0626d0 msgr2=0x7f818c152db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:41.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.629+0000 7f8190e81700 1 -- 172.21.15.77:0/1985071078 shutdown_connections 2022-04-03T19:44:41.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.629+0000 7f8190e81700 1 -- 172.21.15.77:0/1985071078 wait complete. 2022-04-03T19:44:41.653 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-04-03T19:44:41.853 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.853+0000 7f9530dd1700 1 Processor -- start 2022-04-03T19:44:41.854 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.853+0000 7f9530dd1700 1 -- start start 2022-04-03T19:44:41.855 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.854+0000 7f9530dd1700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f952c14c1e0 0x7f952c152790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:41.855 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.854+0000 7f9530dd1700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f952c0c1b30 con 0x7f952c0c6010 2022-04-03T19:44:41.855 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.854+0000 7f9530dd1700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f952c0c6840 con 0x7f952c14be10 2022-04-03T19:44:41.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.854+0000 7f9530dd1700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f952c152cd0 con 0x7f952c14c1e0 2022-04-03T19:44:41.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.855+0000 7f952affd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f952c0c6010 0x7f952c0c3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38344/0 (socket says 172.21.15.77:38344) 2022-04-03T19:44:41.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.855+0000 7f952affd700 1 -- 172.21.15.77:0/3975215340 learned_addr learned my addr 172.21.15.77:0/3975215340 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:41.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.855+0000 7f952b7fe700 1 --1- 172.21.15.77:0/3975215340 >> v1:172.21.15.186:6789/0 conn(0x7f952c14be10 0x7f952c0c2060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44148/0 (socket says 172.21.15.77:44148) 2022-04-03T19:44:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.855+0000 7f952bfff700 1 --2- 172.21.15.77:0/3975215340 >> v2:172.21.15.186:3300/0 conn(0x7f952c14c1e0 0x7f952c152790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:41.858 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.855+0000 7f952a7fc700 1 -- 172.21.15.77:0/3975215340 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1565014483 0 0) 0x7f952c0c6840 con 0x7f952c14be10 2022-04-03T19:44:41.859 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.855+0000 7f952a7fc700 1 -- 172.21.15.77:0/3975215340 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95180033f0 con 0x7f952c14be10 2022-04-03T19:44:41.859 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.855+0000 7f952bfff700 1 -- 172.21.15.77:0/3975215340 >> v1:172.21.15.186:6789/0 conn(0x7f952c14be10 legacy=0x7f952c0c2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:41.859 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.855+0000 7f952bfff700 1 -- 172.21.15.77:0/3975215340 >> v1:172.21.15.77:6789/0 conn(0x7f952c0c6010 legacy=0x7f952c0c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:41.859 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.855+0000 7f952a7fc700 1 -- 172.21.15.77:0/3975215340 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808358028 0 0) 0x7f952c0c1b30 con 0x7f952c0c6010 2022-04-03T19:44:41.860 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.855+0000 7f952bfff700 1 -- 172.21.15.77:0/3975215340 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952c0c5250 con 0x7f952c14c1e0 2022-04-03T19:44:41.860 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.856+0000 7f952bfff700 1 --2- 172.21.15.77:0/3975215340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f952c14c1e0 0x7f952c152790 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9524009e60 tx=0x7f9524009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=896ed1b0ef5888ae server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:41.860 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.856+0000 7f952a7fc700 1 -- 172.21.15.77:0/3975215340 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952400e070 con 0x7f952c14c1e0 2022-04-03T19:44:41.860 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.856+0000 7f952a7fc700 1 -- 172.21.15.77:0/3975215340 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f952400bbf0 con 0x7f952c14c1e0 2022-04-03T19:44:41.860 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.856+0000 7f952a7fc700 1 -- 172.21.15.77:0/3975215340 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9524012430 con 0x7f952c14c1e0 2022-04-03T19:44:41.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.856+0000 7f9530dd1700 1 -- 172.21.15.77:0/3975215340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f952c14c1e0 msgr2=0x7f952c152790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:41.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.856+0000 7f9530dd1700 1 --2- 172.21.15.77:0/3975215340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f952c14c1e0 0x7f952c152790 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9524009e60 tx=0x7f9524009780 comp rx=0 tx=0).stop 2022-04-03T19:44:41.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.856+0000 7f9530dd1700 1 -- 172.21.15.77:0/3975215340 shutdown_connections 2022-04-03T19:44:41.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.856+0000 7f9530dd1700 1 --2- 172.21.15.77:0/3975215340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f952c14c1e0 0x7f952c152790 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:41.862 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.856+0000 7f9530dd1700 1 -- 172.21.15.77:0/3975215340 >> 172.21.15.77:0/3975215340 conn(0x7f952c066da0 msgr2=0x7f952c0671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:41.862 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.856+0000 7f9530dd1700 1 -- 172.21.15.77:0/3975215340 shutdown_connections 2022-04-03T19:44:41.862 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.857+0000 7f9530dd1700 1 -- 172.21.15.77:0/3975215340 wait complete. 2022-04-03T19:44:41.862 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.858+0000 7f9530dd1700 1 Processor -- start 2022-04-03T19:44:41.862 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.858+0000 7f9530dd1700 1 -- start start 2022-04-03T19:44:41.865 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.859+0000 7f9530dd1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f952c0c6010 0x7f952c069820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:41.866 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.859+0000 7f9530dd1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f952c14be10 0x7f952c148440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:41.866 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.859+0000 7f9530dd1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f952c14c1e0 0x7f952c165200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:41.866 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.859+0000 7f9530dd1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c164af0 con 0x7f952c0c6010 2022-04-03T19:44:41.866 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.859+0000 7f9530dd1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952c164d70 con 0x7f952c14c1e0 2022-04-03T19:44:41.866 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.859+0000 7f9530dd1700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952c14a530 con 0x7f952c14be10 2022-04-03T19:44:41.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.859+0000 7f952b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f952c0c6010 0x7f952c069820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:41.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.859+0000 7f952affd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f952c14be10 0x7f952c148440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:41.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.859+0000 7f952bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f952c14c1e0 0x7f952c165200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:41.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.859+0000 7f952b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f952c0c6010 0x7f952c069820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42664/0 (socket says 172.21.15.77:42664) 2022-04-03T19:44:41.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.859+0000 7f952b7fe700 1 -- 172.21.15.77:0/3239527605 learned_addr learned my addr 172.21.15.77:0/3239527605 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:41.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.860+0000 7f952affd700 1 -- 172.21.15.77:0/3239527605 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f952c14c1e0 msgr2=0x7f952c165200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:41.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.860+0000 7f952affd700 1 --2- 172.21.15.77:0/3239527605 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f952c14c1e0 0x7f952c165200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:41.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.860+0000 7f952affd700 1 -- 172.21.15.77:0/3239527605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f952c0c6010 msgr2=0x7f952c069820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:41.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.860+0000 7f952affd700 1 --2- 172.21.15.77:0/3239527605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f952c0c6010 0x7f952c069820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:41.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.860+0000 7f952affd700 1 -- 172.21.15.77:0/3239527605 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9524009a20 con 0x7f952c14be10 2022-04-03T19:44:41.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.860+0000 7f952affd700 1 --2- 172.21.15.77:0/3239527605 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f952c14be10 0x7f952c148440 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f952000c450 tx=0x7f9520007bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:41.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.860+0000 7f95137fe700 1 -- 172.21.15.77:0/3239527605 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952000ce50 con 0x7f952c14be10 2022-04-03T19:44:41.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.860+0000 7f95137fe700 1 -- 172.21.15.77:0/3239527605 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9520007360 con 0x7f952c14be10 2022-04-03T19:44:41.870 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.860+0000 7f9530dd1700 1 -- 172.21.15.77:0/3239527605 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f952c168410 con 0x7f952c14be10 2022-04-03T19:44:41.870 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.860+0000 7f95137fe700 1 -- 172.21.15.77:0/3239527605 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95200046b0 con 0x7f952c14be10 2022-04-03T19:44:41.870 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.861+0000 7f9530dd1700 1 -- 172.21.15.77:0/3239527605 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f952c1689d0 con 0x7f952c14be10 2022-04-03T19:44:41.870 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.862+0000 7f9530dd1700 1 -- 172.21.15.77:0/3239527605 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f952c164f00 con 0x7f952c14be10 2022-04-03T19:44:41.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.863+0000 7f95137fe700 1 -- 172.21.15.77:0/3239527605 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f9520010030 con 0x7f952c14be10 2022-04-03T19:44:41.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.863+0000 7f95137fe700 1 --2- 172.21.15.77:0/3239527605 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f950c063ce0 0x7f950c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:41.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.863+0000 7f95137fe700 1 -- 172.21.15.77:0/3239527605 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f9520080390 con 0x7f952c14be10 2022-04-03T19:44:41.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.864+0000 7f952b7fe700 1 --2- 172.21.15.77:0/3239527605 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f950c063ce0 0x7f950c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:41.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.864+0000 7f952b7fe700 1 --2- 172.21.15.77:0/3239527605 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f950c063ce0 0x7f950c066190 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f951c007fd0 tx=0x7f951c007c70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:41.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:41.868+0000 7f95137fe700 1 -- 172.21.15.77:0/3239527605 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f952004ef00 con 0x7f952c14be10 2022-04-03T19:44:42.060 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.059+0000 7f9530dd1700 1 -- 172.21.15.77:0/3239527605 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f952c04b050 con 0x7f952c14be10 2022-04-03T19:44:42.061 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.060+0000 7f95137fe700 1 -- 172.21.15.77:0/3239527605 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9520052520 con 0x7f952c14be10 2022-04-03T19:44:42.061 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2022-04-03T19:44:42.064 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.063+0000 7f9530dd1700 1 -- 172.21.15.77:0/3239527605 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f950c063ce0 msgr2=0x7f950c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:42.064 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.063+0000 7f9530dd1700 1 --2- 172.21.15.77:0/3239527605 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f950c063ce0 0x7f950c066190 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f951c007fd0 tx=0x7f951c007c70 comp rx=0 tx=0).stop 2022-04-03T19:44:42.064 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.063+0000 7f9530dd1700 1 -- 172.21.15.77:0/3239527605 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f952c14be10 msgr2=0x7f952c148440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:42.064 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.063+0000 7f9530dd1700 1 --2- 172.21.15.77:0/3239527605 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f952c14be10 0x7f952c148440 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f952000c450 tx=0x7f9520007bb0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.065 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.063+0000 7f9530dd1700 1 -- 172.21.15.77:0/3239527605 shutdown_connections 2022-04-03T19:44:42.065 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.063+0000 7f9530dd1700 1 --2- 172.21.15.77:0/3239527605 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f952c14c1e0 0x7f952c165200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.065 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.063+0000 7f9530dd1700 1 --2- 172.21.15.77:0/3239527605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f952c0c6010 0x7f952c069820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.065 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.063+0000 7f9530dd1700 1 --2- 172.21.15.77:0/3239527605 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f950c063ce0 0x7f950c066190 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.066 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.063+0000 7f9530dd1700 1 --2- 172.21.15.77:0/3239527605 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f952c14be10 0x7f952c148440 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.066 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.063+0000 7f9530dd1700 1 -- 172.21.15.77:0/3239527605 >> 172.21.15.77:0/3239527605 conn(0x7f952c066da0 msgr2=0x7f952c147250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:42.066 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.063+0000 7f9530dd1700 1 -- 172.21.15.77:0/3239527605 shutdown_connections 2022-04-03T19:44:42.066 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.063+0000 7f9530dd1700 1 -- 172.21.15.77:0/3239527605 wait complete. 2022-04-03T19:44:42.089 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-04-03T19:44:42.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.289+0000 7f7f26d11700 1 Processor -- start 2022-04-03T19:44:42.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.289+0000 7f7f26d11700 1 -- start start 2022-04-03T19:44:42.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.290+0000 7f7f26d11700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f200c52b0 0x7f7f200c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:42.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.290+0000 7f7f26d11700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7f200c5680 0x7f7f20158e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:42.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.290+0000 7f7f26d11700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f200c7e80 con 0x7f7f200c8310 2022-04-03T19:44:42.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.290+0000 7f7f26d11700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f200c4260 con 0x7f7f200c5680 2022-04-03T19:44:42.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.290+0000 7f7f26d11700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f200c6ea0 con 0x7f7f200c52b0 2022-04-03T19:44:42.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.290+0000 7f7f25d0f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f200c52b0 0x7f7f200c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:42.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.290+0000 7f7f2550e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7f200c5680 0x7f7f20158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:42.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.290+0000 7f7f25d0f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f200c52b0 0x7f7f200c5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59968/0 (socket says 172.21.15.77:59968) 2022-04-03T19:44:42.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.290+0000 7f7f25d0f700 1 -- 172.21.15.77:0/2763995891 learned_addr learned my addr 172.21.15.77:0/2763995891 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:42.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.291+0000 7f7f25d0f700 1 -- 172.21.15.77:0/2763995891 >> v2:172.21.15.186:3300/0 conn(0x7f7f200c5680 msgr2=0x7f7f20158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:42.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.291+0000 7f7f25d0f700 1 --2- 172.21.15.77:0/2763995891 >> v2:172.21.15.186:3300/0 conn(0x7f7f200c5680 0x7f7f20158e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.291+0000 7f7f25d0f700 1 -- 172.21.15.77:0/2763995891 >> v1:172.21.15.186:6789/0 conn(0x7f7f200c8310 legacy=0x7f7f2014ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:42.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.291+0000 7f7f25d0f700 1 -- 172.21.15.77:0/2763995891 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f200c6600 con 0x7f7f200c52b0 2022-04-03T19:44:42.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.291+0000 7f7f2550e700 1 --2- 172.21.15.77:0/2763995891 >> v2:172.21.15.186:3300/0 conn(0x7f7f200c5680 0x7f7f20158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:42.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.291+0000 7f7f25d0f700 1 --2- 172.21.15.77:0/2763995891 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f200c52b0 0x7f7f200c5f70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c006fd0 tx=0x7f7f1c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecc93a86eb21c50b server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:42.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.291+0000 7f7f24d0d700 1 -- 172.21.15.77:0/2763995891 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c004030 con 0x7f7f200c52b0 2022-04-03T19:44:42.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.291+0000 7f7f24d0d700 1 -- 172.21.15.77:0/2763995891 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1c0041d0 con 0x7f7f200c52b0 2022-04-03T19:44:42.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.291+0000 7f7f24d0d700 1 -- 172.21.15.77:0/2763995891 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c011630 con 0x7f7f200c52b0 2022-04-03T19:44:42.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.292+0000 7f7f26d11700 1 -- 172.21.15.77:0/2763995891 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f200c52b0 msgr2=0x7f7f200c5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:42.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.292+0000 7f7f26d11700 1 --2- 172.21.15.77:0/2763995891 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f200c52b0 0x7f7f200c5f70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c006fd0 tx=0x7f7f1c0090f0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.292+0000 7f7f26d11700 1 -- 172.21.15.77:0/2763995891 shutdown_connections 2022-04-03T19:44:42.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.292+0000 7f7f26d11700 1 --2- 172.21.15.77:0/2763995891 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f200c52b0 0x7f7f200c5f70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.292+0000 7f7f26d11700 1 --2- 172.21.15.77:0/2763995891 >> v2:172.21.15.186:3300/0 conn(0x7f7f200c5680 0x7f7f20158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.292+0000 7f7f26d11700 1 -- 172.21.15.77:0/2763995891 >> 172.21.15.77:0/2763995891 conn(0x7f7f200626d0 msgr2=0x7f7f20062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:42.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.292+0000 7f7f26d11700 1 -- 172.21.15.77:0/2763995891 shutdown_connections 2022-04-03T19:44:42.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.292+0000 7f7f26d11700 1 -- 172.21.15.77:0/2763995891 wait complete. 2022-04-03T19:44:42.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.294+0000 7f7f26d11700 1 Processor -- start 2022-04-03T19:44:42.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.294+0000 7f7f26d11700 1 -- start start 2022-04-03T19:44:42.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.295+0000 7f7f26d11700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f200c52b0 0x7f7f20168280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:42.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.295+0000 7f7f26d11700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f200c5680 0x7f7f20167200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:42.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.295+0000 7f7f26d11700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f200c8310 0x7f7f20167740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:42.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.295+0000 7f7f26d11700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f200bf990 con 0x7f7f200c52b0 2022-04-03T19:44:42.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.295+0000 7f7f26d11700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f20168ed0 con 0x7f7f200c8310 2022-04-03T19:44:42.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.295+0000 7f7f26d11700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f20168c30 con 0x7f7f200c5680 2022-04-03T19:44:42.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.295+0000 7f7f25d0f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f200c52b0 0x7f7f20168280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:42.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.295+0000 7f7f2550e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f200c5680 0x7f7f20167200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:42.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.295+0000 7f7f26510700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f200c8310 0x7f7f20167740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:42.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.295+0000 7f7f2550e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f200c5680 0x7f7f20167200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59970/0 (socket says 172.21.15.77:59970) 2022-04-03T19:44:42.302 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.295+0000 7f7f2550e700 1 -- 172.21.15.77:0/2595770257 learned_addr learned my addr 172.21.15.77:0/2595770257 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:42.304 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.296+0000 7f7f2550e700 1 -- 172.21.15.77:0/2595770257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f200c8310 msgr2=0x7f7f20167740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:42.304 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.296+0000 7f7f2550e700 1 --2- 172.21.15.77:0/2595770257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f200c8310 0x7f7f20167740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.304 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.296+0000 7f7f2550e700 1 -- 172.21.15.77:0/2595770257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f200c52b0 msgr2=0x7f7f20168280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:42.304 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.296+0000 7f7f2550e700 1 --2- 172.21.15.77:0/2595770257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f200c52b0 0x7f7f20168280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.305 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.296+0000 7f7f2550e700 1 -- 172.21.15.77:0/2595770257 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f100092a0 con 0x7f7f200c5680 2022-04-03T19:44:42.305 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.296+0000 7f7f2550e700 1 --2- 172.21.15.77:0/2595770257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f200c5680 0x7f7f20167200 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7f1000ea00 tx=0x7f7f10009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:42.305 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.296+0000 7f7f0dffb700 1 -- 172.21.15.77:0/2595770257 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f10013680 con 0x7f7f200c5680 2022-04-03T19:44:42.305 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.296+0000 7f7f0dffb700 1 -- 172.21.15.77:0/2595770257 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f100137e0 con 0x7f7f200c5680 2022-04-03T19:44:42.306 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.296+0000 7f7f26d11700 1 -- 172.21.15.77:0/2595770257 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f1c009370 con 0x7f7f200c5680 2022-04-03T19:44:42.306 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.296+0000 7f7f0dffb700 1 -- 172.21.15.77:0/2595770257 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f10013b80 con 0x7f7f200c5680 2022-04-03T19:44:42.306 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.296+0000 7f7f26d11700 1 -- 172.21.15.77:0/2595770257 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f20155a80 con 0x7f7f200c5680 2022-04-03T19:44:42.306 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.297+0000 7f7f26d11700 1 -- 172.21.15.77:0/2595770257 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f20088e10 con 0x7f7f200c5680 2022-04-03T19:44:42.306 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.298+0000 7f7f0dffb700 1 -- 172.21.15.77:0/2595770257 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f7f10019030 con 0x7f7f200c5680 2022-04-03T19:44:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.299+0000 7f7f0dffb700 1 --2- 172.21.15.77:0/2595770257 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7f00063d10 0x7f7f000661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.299+0000 7f7f0dffb700 1 -- 172.21.15.77:0/2595770257 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f7f10080830 con 0x7f7f200c5680 2022-04-03T19:44:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.299+0000 7f7f25d0f700 1 --2- 172.21.15.77:0/2595770257 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7f00063d10 0x7f7f000661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.300+0000 7f7f25d0f700 1 --2- 172.21.15.77:0/2595770257 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7f00063d10 0x7f7f000661c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c009340 tx=0x7f7f1c016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:42.308 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.304+0000 7f7f0dffb700 1 -- 172.21.15.77:0/2595770257 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f7f1004f410 con 0x7f7f200c5680 2022-04-03T19:44:42.498 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.498+0000 7f7f26d11700 1 -- 172.21.15.77:0/2595770257 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7f2006b460 con 0x7f7f200c5680 2022-04-03T19:44:42.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.499+0000 7f7f0dffb700 1 -- 172.21.15.77:0/2595770257 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7f10021020 con 0x7f7f200c5680 2022-04-03T19:44:42.499 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2022-04-03T19:44:42.502 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.501+0000 7f7f26d11700 1 -- 172.21.15.77:0/2595770257 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7f00063d10 msgr2=0x7f7f000661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:42.502 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.501+0000 7f7f26d11700 1 --2- 172.21.15.77:0/2595770257 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7f00063d10 0x7f7f000661c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c009340 tx=0x7f7f1c016000 comp rx=0 tx=0).stop 2022-04-03T19:44:42.502 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.501+0000 7f7f26d11700 1 -- 172.21.15.77:0/2595770257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f200c5680 msgr2=0x7f7f20167200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:42.502 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.501+0000 7f7f26d11700 1 --2- 172.21.15.77:0/2595770257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f200c5680 0x7f7f20167200 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7f1000ea00 tx=0x7f7f10009060 comp rx=0 tx=0).stop 2022-04-03T19:44:42.502 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.501+0000 7f7f26d11700 1 -- 172.21.15.77:0/2595770257 shutdown_connections 2022-04-03T19:44:42.503 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.501+0000 7f7f26d11700 1 --2- 172.21.15.77:0/2595770257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f200c8310 0x7f7f20167740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.503 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.501+0000 7f7f26d11700 1 --2- 172.21.15.77:0/2595770257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f200c52b0 0x7f7f20168280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.503 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.501+0000 7f7f26d11700 1 --2- 172.21.15.77:0/2595770257 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7f00063d10 0x7f7f000661c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.503 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.501+0000 7f7f26d11700 1 --2- 172.21.15.77:0/2595770257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f200c5680 0x7f7f20167200 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.504 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.501+0000 7f7f26d11700 1 -- 172.21.15.77:0/2595770257 >> 172.21.15.77:0/2595770257 conn(0x7f7f200626d0 msgr2=0x7f7f2006cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:42.504 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.501+0000 7f7f26d11700 1 -- 172.21.15.77:0/2595770257 shutdown_connections 2022-04-03T19:44:42.504 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.501+0000 7f7f26d11700 1 -- 172.21.15.77:0/2595770257 wait complete. 2022-04-03T19:44:42.525 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-04-03T19:44:42.725 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.725+0000 7ff6b8bad700 1 Processor -- start 2022-04-03T19:44:42.726 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.725+0000 7ff6b8bad700 1 -- start start 2022-04-03T19:44:42.727 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.726+0000 7ff6b8bad700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6b40c52b0 0x7ff6b40c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:42.727 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.726+0000 7ff6b8bad700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff6b40c5680 0x7ff6b4158e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:42.728 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.726+0000 7ff6b8bad700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6b40c7e80 con 0x7ff6b40c8310 2022-04-03T19:44:42.728 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.726+0000 7ff6b8bad700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6b40c4260 con 0x7ff6b40c5680 2022-04-03T19:44:42.728 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.726+0000 7ff6b8bad700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b40c6ea0 con 0x7ff6b40c52b0 2022-04-03T19:44:42.728 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.727+0000 7ff6b3fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff6b40c8310 0x7ff6b414ed50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38346/0 (socket says 172.21.15.77:38346) 2022-04-03T19:44:42.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.727+0000 7ff6b3fff700 1 -- 172.21.15.77:0/2385081991 learned_addr learned my addr 172.21.15.77:0/2385081991 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:42.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.727+0000 7ff6b37fe700 1 --2- 172.21.15.77:0/2385081991 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6b40c52b0 0x7ff6b40c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:42.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.727+0000 7ff6b2ffd700 1 --2- 172.21.15.77:0/2385081991 >> v2:172.21.15.186:3300/0 conn(0x7ff6b40c5680 0x7ff6b4158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:42.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.727+0000 7ff6b2ffd700 1 -- 172.21.15.77:0/2385081991 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6b40c52b0 msgr2=0x7ff6b40c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:42.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.727+0000 7ff6b2ffd700 1 --2- 172.21.15.77:0/2385081991 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6b40c52b0 0x7ff6b40c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.727+0000 7ff6b2ffd700 1 -- 172.21.15.77:0/2385081991 >> v1:172.21.15.77:6789/0 conn(0x7ff6b40c8310 legacy=0x7ff6b414ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:42.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.727+0000 7ff6b2ffd700 1 -- 172.21.15.77:0/2385081991 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b40c6600 con 0x7ff6b40c5680 2022-04-03T19:44:42.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.727+0000 7ff6b27fc700 1 -- 172.21.15.77:0/2385081991 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143618058 0 0) 0x7ff6b40c7e80 con 0x7ff6b40c8310 2022-04-03T19:44:42.734 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.727+0000 7ff6b2ffd700 1 --2- 172.21.15.77:0/2385081991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6b40c5680 0x7ff6b4158e10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009760 tx=0x7ff6a8009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=113df55e30785a82 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:42.734 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.728+0000 7ff6b27fc700 1 -- 172.21.15.77:0/2385081991 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8011630 con 0x7ff6b40c5680 2022-04-03T19:44:42.735 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.728+0000 7ff6b27fc700 1 -- 172.21.15.77:0/2385081991 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a8011790 con 0x7ff6b40c5680 2022-04-03T19:44:42.735 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.728+0000 7ff6b27fc700 1 -- 172.21.15.77:0/2385081991 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8011b50 con 0x7ff6b40c5680 2022-04-03T19:44:42.735 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.728+0000 7ff6b8bad700 1 -- 172.21.15.77:0/2385081991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6b40c5680 msgr2=0x7ff6b4158e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:42.735 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.728+0000 7ff6b8bad700 1 --2- 172.21.15.77:0/2385081991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6b40c5680 0x7ff6b4158e10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009760 tx=0x7ff6a8009060 comp rx=0 tx=0).stop 2022-04-03T19:44:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.728+0000 7ff6b8bad700 1 -- 172.21.15.77:0/2385081991 shutdown_connections 2022-04-03T19:44:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.728+0000 7ff6b8bad700 1 --2- 172.21.15.77:0/2385081991 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6b40c52b0 0x7ff6b40c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.728+0000 7ff6b8bad700 1 --2- 172.21.15.77:0/2385081991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6b40c5680 0x7ff6b4158e10 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.728+0000 7ff6b8bad700 1 -- 172.21.15.77:0/2385081991 >> 172.21.15.77:0/2385081991 conn(0x7ff6b40626d0 msgr2=0x7ff6b4062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.728+0000 7ff6b8bad700 1 -- 172.21.15.77:0/2385081991 shutdown_connections 2022-04-03T19:44:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.728+0000 7ff6b8bad700 1 -- 172.21.15.77:0/2385081991 wait complete. 2022-04-03T19:44:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.730+0000 7ff6b8bad700 1 Processor -- start 2022-04-03T19:44:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.730+0000 7ff6b8bad700 1 -- start start 2022-04-03T19:44:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.731+0000 7ff6b8bad700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6b40c52b0 0x7ff6b4165bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.731+0000 7ff6b8bad700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6b40c5680 0x7ff6b4149d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.731+0000 7ff6b8bad700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6b40c8310 0x7ff6b414a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.731+0000 7ff6b8bad700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b4165740 con 0x7ff6b40c52b0 2022-04-03T19:44:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.731+0000 7ff6b8bad700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b4153350 con 0x7ff6b40c5680 2022-04-03T19:44:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.731+0000 7ff6b8bad700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b4153630 con 0x7ff6b40c8310 2022-04-03T19:44:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.731+0000 7ff6b37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6b40c52b0 0x7ff6b4165bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.731+0000 7ff6b2ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6b40c5680 0x7ff6b4149d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.731+0000 7ff6b2ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6b40c5680 0x7ff6b4149d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37870/0 (socket says 172.21.15.77:37870) 2022-04-03T19:44:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.731+0000 7ff6b2ffd700 1 -- 172.21.15.77:0/603554097 learned_addr learned my addr 172.21.15.77:0/603554097 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:42.740 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.731+0000 7ff6b3fff700 1 --2- 172.21.15.77:0/603554097 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6b40c8310 0x7ff6b414a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:42.740 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.732+0000 7ff6b2ffd700 1 -- 172.21.15.77:0/603554097 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6b40c8310 msgr2=0x7ff6b414a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:42.740 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.732+0000 7ff6b2ffd700 1 --2- 172.21.15.77:0/603554097 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6b40c8310 0x7ff6b414a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.740 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.732+0000 7ff6b2ffd700 1 -- 172.21.15.77:0/603554097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6b40c52b0 msgr2=0x7ff6b4165bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:42.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.732+0000 7ff6b2ffd700 1 --2- 172.21.15.77:0/603554097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6b40c52b0 0x7ff6b4165bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.732+0000 7ff6b2ffd700 1 -- 172.21.15.77:0/603554097 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a80092a0 con 0x7ff6b40c5680 2022-04-03T19:44:42.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.732+0000 7ff6b37fe700 1 --2- 172.21.15.77:0/603554097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6b40c52b0 0x7ff6b4165bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:42.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.732+0000 7ff6b2ffd700 1 --2- 172.21.15.77:0/603554097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6b40c5680 0x7ff6b4149d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff6b40632a0 tx=0x7ff6a8017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:42.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.732+0000 7ff69b7fe700 1 -- 172.21.15.77:0/603554097 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8000bc0 con 0x7ff6b40c5680 2022-04-03T19:44:42.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.732+0000 7ff69b7fe700 1 -- 172.21.15.77:0/603554097 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a8011900 con 0x7ff6b40c5680 2022-04-03T19:44:42.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.732+0000 7ff6b8bad700 1 -- 172.21.15.77:0/603554097 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b4153910 con 0x7ff6b40c5680 2022-04-03T19:44:42.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.732+0000 7ff69b7fe700 1 -- 172.21.15.77:0/603554097 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8025420 con 0x7ff6b40c5680 2022-04-03T19:44:42.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.732+0000 7ff6b8bad700 1 -- 172.21.15.77:0/603554097 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b41653d0 con 0x7ff6b40c5680 2022-04-03T19:44:42.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.733+0000 7ff6b8bad700 1 -- 172.21.15.77:0/603554097 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b4088e10 con 0x7ff6b40c5680 2022-04-03T19:44:42.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.735+0000 7ff69b7fe700 1 -- 172.21.15.77:0/603554097 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7ff6a802c030 con 0x7ff6b40c5680 2022-04-03T19:44:42.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.735+0000 7ff69b7fe700 1 --2- 172.21.15.77:0/603554097 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff694063d10 0x7ff6940661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:42.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.735+0000 7ff69b7fe700 1 -- 172.21.15.77:0/603554097 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7ff6a80821e0 con 0x7ff6b40c5680 2022-04-03T19:44:42.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.736+0000 7ff6b37fe700 1 --2- 172.21.15.77:0/603554097 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff694063d10 0x7ff6940661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:42.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.736+0000 7ff6b37fe700 1 --2- 172.21.15.77:0/603554097 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff694063d10 0x7ff6940661c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4011310 tx=0x7ff6a4009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:42.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.738+0000 7ff69b7fe700 1 -- 172.21.15.77:0/603554097 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7ff6a8050d50 con 0x7ff6b40c5680 2022-04-03T19:44:42.931 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.931+0000 7ff6b8bad700 1 -- 172.21.15.77:0/603554097 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff6b4156270 con 0x7ff6b40c5680 2022-04-03T19:44:42.932 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.932+0000 7ff69b7fe700 1 -- 172.21.15.77:0/603554097 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff6a802a0a0 con 0x7ff6b40c5680 2022-04-03T19:44:42.933 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2022-04-03T19:44:42.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.934+0000 7ff6b8bad700 1 -- 172.21.15.77:0/603554097 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff694063d10 msgr2=0x7ff6940661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:42.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.934+0000 7ff6b8bad700 1 --2- 172.21.15.77:0/603554097 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff694063d10 0x7ff6940661c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4011310 tx=0x7ff6a4009060 comp rx=0 tx=0).stop 2022-04-03T19:44:42.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.934+0000 7ff6b8bad700 1 -- 172.21.15.77:0/603554097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6b40c5680 msgr2=0x7ff6b4149d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:42.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.934+0000 7ff6b8bad700 1 --2- 172.21.15.77:0/603554097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6b40c5680 0x7ff6b4149d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff6b40632a0 tx=0x7ff6a8017000 comp rx=0 tx=0).stop 2022-04-03T19:44:42.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.934+0000 7ff6b8bad700 1 -- 172.21.15.77:0/603554097 shutdown_connections 2022-04-03T19:44:42.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.934+0000 7ff6b8bad700 1 --2- 172.21.15.77:0/603554097 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff694063d10 0x7ff6940661c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.934+0000 7ff6b8bad700 1 --2- 172.21.15.77:0/603554097 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6b40c8310 0x7ff6b414a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.937 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.934+0000 7ff6b8bad700 1 --2- 172.21.15.77:0/603554097 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6b40c52b0 0x7ff6b4165bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.937 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.934+0000 7ff6b8bad700 1 --2- 172.21.15.77:0/603554097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6b40c5680 0x7ff6b4149d40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:42.937 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.934+0000 7ff6b8bad700 1 -- 172.21.15.77:0/603554097 >> 172.21.15.77:0/603554097 conn(0x7ff6b40626d0 msgr2=0x7ff6b40c8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:42.937 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.934+0000 7ff6b8bad700 1 -- 172.21.15.77:0/603554097 shutdown_connections 2022-04-03T19:44:42.938 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:42.934+0000 7ff6b8bad700 1 -- 172.21.15.77:0/603554097 wait complete. 2022-04-03T19:44:42.959 INFO:tasks.mds_thrash:mds thrasher using random seed: 1649015082 2022-04-03T19:44:42.960 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-04-03T19:44:43.158 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.157+0000 7f93c1a3f700 1 Processor -- start 2022-04-03T19:44:43.158 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.157+0000 7f93c1a3f700 1 -- start start 2022-04-03T19:44:43.160 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.158+0000 7f93c1a3f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f93bc06d3f0 0x7f93bc06d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:43.161 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.158+0000 7f93c1a3f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93bc0c5930 0x7f93bc155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:43.161 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.159+0000 7f93c1a3f700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93bc0c2f80 con 0x7f93bc156120 2022-04-03T19:44:43.161 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.159+0000 7f93c1a3f700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93bc0c3260 con 0x7f93bc06d3f0 2022-04-03T19:44:43.161 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.159+0000 7f93c1a3f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc0c3540 con 0x7f93bc0c5930 2022-04-03T19:44:43.161 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.159+0000 7f93c0a3d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f93bc06d3f0 0x7f93bc06d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:43.162 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.159+0000 7f93bbfff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93bc0c5930 0x7f93bc155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:43.162 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.159+0000 7f93c123e700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f93bc156120 0x7f93bc06c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38350/0 (socket says 172.21.15.77:38350) 2022-04-03T19:44:43.162 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.159+0000 7f93c123e700 1 -- 172.21.15.77:0/1458577159 learned_addr learned my addr 172.21.15.77:0/1458577159 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:43.162 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.159+0000 7f93bbfff700 1 -- 172.21.15.77:0/1458577159 >> v2:172.21.15.77:3300/0 conn(0x7f93bc06d3f0 msgr2=0x7f93bc06d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:43.163 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.159+0000 7f93bbfff700 1 --2- 172.21.15.77:0/1458577159 >> v2:172.21.15.77:3300/0 conn(0x7f93bc06d3f0 0x7f93bc06d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:43.163 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.159+0000 7f93bbfff700 1 -- 172.21.15.77:0/1458577159 >> v1:172.21.15.77:6789/0 conn(0x7f93bc156120 legacy=0x7f93bc06c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:43.163 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.160+0000 7f93bbfff700 1 -- 172.21.15.77:0/1458577159 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93bc0c3820 con 0x7f93bc0c5930 2022-04-03T19:44:43.163 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.160+0000 7f93c0a3d700 1 --2- 172.21.15.77:0/1458577159 >> v2:172.21.15.77:3300/0 conn(0x7f93bc06d3f0 0x7f93bc06d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:43.164 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.160+0000 7f93bbfff700 1 --2- 172.21.15.77:0/1458577159 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93bc0c5930 0x7f93bc155be0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f93a8009790 tx=0x7f93a8009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=773bbad407e03fd4 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:43.164 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.160+0000 7f93bb7fe700 1 -- 172.21.15.77:0/1458577159 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a8004030 con 0x7f93bc0c5930 2022-04-03T19:44:43.164 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.160+0000 7f93bb7fe700 1 -- 172.21.15.77:0/1458577159 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93a8004190 con 0x7f93bc0c5930 2022-04-03T19:44:43.164 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.160+0000 7f93bb7fe700 1 -- 172.21.15.77:0/1458577159 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a80105b0 con 0x7f93bc0c5930 2022-04-03T19:44:43.168 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.161+0000 7f93c1a3f700 1 -- 172.21.15.77:0/1458577159 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93bc0c5930 msgr2=0x7f93bc155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:43.168 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.161+0000 7f93c1a3f700 1 --2- 172.21.15.77:0/1458577159 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93bc0c5930 0x7f93bc155be0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f93a8009790 tx=0x7f93a8009060 comp rx=0 tx=0).stop 2022-04-03T19:44:43.168 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.161+0000 7f93c1a3f700 1 -- 172.21.15.77:0/1458577159 shutdown_connections 2022-04-03T19:44:43.169 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.161+0000 7f93c1a3f700 1 --2- 172.21.15.77:0/1458577159 >> v2:172.21.15.77:3300/0 conn(0x7f93bc06d3f0 0x7f93bc06d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:43.169 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.161+0000 7f93c1a3f700 1 --2- 172.21.15.77:0/1458577159 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93bc0c5930 0x7f93bc155be0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:43.169 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.161+0000 7f93c1a3f700 1 -- 172.21.15.77:0/1458577159 >> 172.21.15.77:0/1458577159 conn(0x7f93bc0cb720 msgr2=0x7f93bc0cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:43.169 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.161+0000 7f93c1a3f700 1 -- 172.21.15.77:0/1458577159 shutdown_connections 2022-04-03T19:44:43.170 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.161+0000 7f93c1a3f700 1 -- 172.21.15.77:0/1458577159 wait complete. 2022-04-03T19:44:43.170 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.162+0000 7f93c1a3f700 1 Processor -- start 2022-04-03T19:44:43.170 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.162+0000 7f93c1a3f700 1 -- start start 2022-04-03T19:44:43.170 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.163+0000 7f93c1a3f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93bc06d3f0 0x7f93bc149fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:43.171 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.163+0000 7f93c1a3f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93bc0c5930 0x7f93bc14fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:43.171 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.163+0000 7f93c1a3f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93bc156120 0x7f93bc168e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:43.171 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.163+0000 7f93c1a3f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc152e40 con 0x7f93bc156120 2022-04-03T19:44:43.171 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.163+0000 7f93c1a3f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc169c90 con 0x7f93bc06d3f0 2022-04-03T19:44:43.172 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.163+0000 7f93c1a3f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc0c0ac0 con 0x7f93bc0c5930 2022-04-03T19:44:43.172 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.164+0000 7f93c123e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93bc156120 0x7f93bc168e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:43.172 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.164+0000 7f93bbfff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93bc0c5930 0x7f93bc14fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:43.172 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.164+0000 7f93c0a3d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93bc06d3f0 0x7f93bc149fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:43.172 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.164+0000 7f93c123e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93bc156120 0x7f93bc168e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42672/0 (socket says 172.21.15.77:42672) 2022-04-03T19:44:43.173 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.164+0000 7f93c123e700 1 -- 172.21.15.77:0/3042782227 learned_addr learned my addr 172.21.15.77:0/3042782227 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:43.173 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.164+0000 7f93c123e700 1 -- 172.21.15.77:0/3042782227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93bc0c5930 msgr2=0x7f93bc14fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:43.173 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.164+0000 7f93c123e700 1 --2- 172.21.15.77:0/3042782227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93bc0c5930 0x7f93bc14fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:43.173 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.164+0000 7f93c123e700 1 -- 172.21.15.77:0/3042782227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93bc06d3f0 msgr2=0x7f93bc149fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:43.174 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.164+0000 7f93c123e700 1 --2- 172.21.15.77:0/3042782227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93bc06d3f0 0x7f93bc149fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:43.174 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.164+0000 7f93c123e700 1 -- 172.21.15.77:0/3042782227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b00092a0 con 0x7f93bc156120 2022-04-03T19:44:43.174 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.164+0000 7f93c123e700 1 --2- 172.21.15.77:0/3042782227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93bc156120 0x7f93bc168e40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f93b400c960 tx=0x7f93b40016c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:43.174 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.165+0000 7f93b8ff9700 1 -- 172.21.15.77:0/3042782227 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b4004c40 con 0x7f93bc156120 2022-04-03T19:44:43.175 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.165+0000 7f93b8ff9700 1 -- 172.21.15.77:0/3042782227 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b4004da0 con 0x7f93bc156120 2022-04-03T19:44:43.175 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.165+0000 7f93c1a3f700 1 -- 172.21.15.77:0/3042782227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a80092a0 con 0x7f93bc156120 2022-04-03T19:44:43.175 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.165+0000 7f93b8ff9700 1 -- 172.21.15.77:0/3042782227 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b40073a0 con 0x7f93bc156120 2022-04-03T19:44:43.175 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.165+0000 7f93c1a3f700 1 -- 172.21.15.77:0/3042782227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93bc152880 con 0x7f93bc156120 2022-04-03T19:44:43.175 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.166+0000 7f93c1a3f700 1 -- 172.21.15.77:0/3042782227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93bc0c05f0 con 0x7f93bc156120 2022-04-03T19:44:43.176 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.167+0000 7f93b8ff9700 1 -- 172.21.15.77:0/3042782227 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f93b401a030 con 0x7f93bc156120 2022-04-03T19:44:43.176 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.167+0000 7f93b8ff9700 1 --2- 172.21.15.77:0/3042782227 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f93a4063d10 0x7f93a40661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:43.176 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.167+0000 7f93b8ff9700 1 -- 172.21.15.77:0/3042782227 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f93b407fd20 con 0x7f93bc156120 2022-04-03T19:44:43.176 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.168+0000 7f93c0a3d700 1 --2- 172.21.15.77:0/3042782227 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f93a4063d10 0x7f93a40661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:43.177 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.168+0000 7f93c0a3d700 1 --2- 172.21.15.77:0/3042782227 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f93a4063d10 0x7f93a40661c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f93b0009d20 tx=0x7f93b0009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:43.177 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.171+0000 7f93b8ff9700 1 -- 172.21.15.77:0/3042782227 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f93bc0c05f0 con 0x7f93bc156120 2022-04-03T19:44:43.366 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.366+0000 7f93c1a3f700 1 -- 172.21.15.77:0/3042782227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93bc0c05f0 con 0x7f93bc156120 2022-04-03T19:44:43.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.367+0000 7f93b8ff9700 1 -- 172.21.15.77:0/3042782227 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16845 (secure 0 0 0) 0x7f93bc0c05f0 con 0x7f93bc156120 2022-04-03T19:44:43.368 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:43.368 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"89c3a3f4-599d-4c4a-8f5a-7180eafb7292","created":"2022-04-03T19:43:54.101559+0000","modified":"2022-04-03T19:44:22.278254+0000","last_up_change":"2022-04-03T19:44:05.589174+0000","last_in_change":"2022-04-03T19:43:56.729760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-03T19:44:07.567638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-04-03T19:44:13.149481+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-04-03T19:44:18.823926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-04-03T19:44:20.275479+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f35f5195-3729-4d78-85d1-0539f259283f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6823","nonce":20252},{"type":"v1","addr":"172.21.15.77:6825","nonce":20252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":20252},{"type":"v1","addr":"172.21.15.77:6827","nonce":20252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":20252},{"type":"v1","addr":"172.21.15.77:6831","nonce":20252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":20252},{"type":"v1","addr":"172.21.15.77:6829","nonce":20252}]},"public_addr":"172.21.15.77:6825/20252","cluster_addr":"172.21.15.77:6827/20252","heartbeat_back_addr":"172.21.15.77:6831/20252","heartbeat_front_addr":"172.21.15.77:6829/20252","state":["exists","up"]},{"osd":1,"uuid":"ab1c89dc-57ce-4cc4-b8c7-9b0c19939601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":20249},{"type":"v1","addr":"172.21.15.77:6805","nonce":20249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":20249},{"type":"v1","addr":"172.21.15.77:6809","nonce":20249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":20249},{"type":"v1","addr":"172.21.15.77:6819","nonce":20249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":20249},{"type":"v1","addr":"172.21.15.77:6815","nonce":20249}]},"public_addr":"172.21.15.77:6805/20249","cluster_addr":"172.21.15.77:6809/20249","heartbeat_back_addr":"172.21.15.77:6819/20249","heartbeat_front_addr":"172.21.15.77:6815/20249","state":["exists","up"]},{"osd":2,"uuid":"351e218b-7749-4ee9-b44a-0a5b52fb9be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":20248},{"type":"v1","addr":"172.21.15.77:6801","nonce":20248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":20248},{"type":"v1","addr":"172.21.15.77:6803","nonce":20248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":20248},{"type":"v1","addr":"172.21.15.77:6813","nonce":20248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":20248},{"type":"v1","addr":"172.21.15.77:6808","nonce":20248}]},"public_addr":"172.21.15.77:6801/20248","cluster_addr":"172.21.15.77:6803/20248","heartbeat_back_addr":"172.21.15.77:6813/20248","heartbeat_front_addr":"172.21.15.77:6808/20248","state":["exists","up"]},{"osd":3,"uuid":"1ba6ec8a-7cbf-4c12-8e21-c3097a263ae8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":20250},{"type":"v1","addr":"172.21.15.77:6814","nonce":20250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":20250},{"type":"v1","addr":"172.21.15.77:6818","nonce":20250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":20250},{"type":"v1","addr":"172.21.15.77:6824","nonce":20250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":20250},{"type":"v1","addr":"172.21.15.77:6821","nonce":20250}]},"public_addr":"172.21.15.77:6814/20250","cluster_addr":"172.21.15.77:6818/20250","heartbeat_back_addr":"172.21.15.77:6824/20250","heartbeat_front_addr":"172.21.15.77:6821/20250","state":["exists","up"]},{"osd":4,"uuid":"3faf4c09-846c-4133-856b-7f53c535a67d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":25551},{"type":"v1","addr":"172.21.15.186:6822","nonce":25551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":25551},{"type":"v1","addr":"172.21.15.186:6826","nonce":25551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":25551},{"type":"v1","addr":"172.21.15.186:6831","nonce":25551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":25551},{"type":"v1","addr":"172.21.15.186:6829","nonce":25551}]},"public_addr":"172.21.15.186:6822/25551","cluster_addr":"172.21.15.186:6826/25551","heartbeat_back_addr":"172.21.15.186:6831/25551","heartbeat_front_addr":"172.21.15.186:6829/25551","state":["exists","up"]},{"osd":5,"uuid":"ffe41d00-d0aa-42f4-82c1-5cced13c1bc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":25541},{"type":"v1","addr":"172.21.15.186:6809","nonce":25541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":25541},{"type":"v1","addr":"172.21.15.186:6811","nonce":25541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":25541},{"type":"v1","addr":"172.21.15.186:6815","nonce":25541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":25541},{"type":"v1","addr":"172.21.15.186:6813","nonce":25541}]},"public_addr":"172.21.15.186:6809/25541","cluster_addr":"172.21.15.186:6811/25541","heartbeat_back_addr":"172.21.15.186:6815/25541","heartbeat_front_addr":"172.21.15.186:6813/25541","state":["exists","up"]},{"osd":6,"uuid":"b0344ee8-deca-4e9b-934c-5866d90d0073","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":25542},{"type":"v1","addr":"172.21.15.186:6801","nonce":25542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":25542},{"type":"v1","addr":"172.21.15.186:6803","nonce":25542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":25542},{"type":"v1","addr":"172.21.15.186:6807","nonce":25542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":25542},{"type":"v1","addr":"172.21.15.186:6805","nonce":25542}]},"public_addr":"172.21.15.186:6801/25542","cluster_addr":"172.21.15.186:6803/25542","heartbeat_back_addr":"172.21.15.186:6807/25542","heartbeat_front_addr":"172.21.15.186:6805/25542","state":["exists","up"]},{"osd":7,"uuid":"0eb0e5d3-3dad-4f49-9b50-3b24a3924bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":25572},{"type":"v1","addr":"172.21.15.186:6817","nonce":25572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":25572},{"type":"v1","addr":"172.21.15.186:6819","nonce":25572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6825","nonce":25572},{"type":"v1","addr":"172.21.15.186:6827","nonce":25572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":25572},{"type":"v1","addr":"172.21.15.186:6823","nonce":25572}]},"public_addr":"172.21.15.186:6817/25572","cluster_addr":"172.21.15.186:6819/25572","heartbeat_back_addr":"172.21.15.186:6827/25572","heartbeat_front_addr":"172.21.15.186:6823/25572","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.143602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.490366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.361292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.343544+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.905429+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969298+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.949524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-03T19:44:43.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.370+0000 7f93c1a3f700 1 -- 172.21.15.77:0/3042782227 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f93a4063d10 msgr2=0x7f93a40661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:43.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.370+0000 7f93c1a3f700 1 --2- 172.21.15.77:0/3042782227 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f93a4063d10 0x7f93a40661c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f93b0009d20 tx=0x7f93b0009060 comp rx=0 tx=0).stop 2022-04-03T19:44:43.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.370+0000 7f93c1a3f700 1 -- 172.21.15.77:0/3042782227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93bc156120 msgr2=0x7f93bc168e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:43.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.370+0000 7f93c1a3f700 1 --2- 172.21.15.77:0/3042782227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93bc156120 0x7f93bc168e40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f93b400c960 tx=0x7f93b40016c0 comp rx=0 tx=0).stop 2022-04-03T19:44:43.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.370+0000 7f93c1a3f700 1 -- 172.21.15.77:0/3042782227 shutdown_connections 2022-04-03T19:44:43.372 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.370+0000 7f93c1a3f700 1 --2- 172.21.15.77:0/3042782227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93bc156120 0x7f93bc168e40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:43.372 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.370+0000 7f93c1a3f700 1 --2- 172.21.15.77:0/3042782227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93bc06d3f0 0x7f93bc149fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:43.373 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.370+0000 7f93c1a3f700 1 --2- 172.21.15.77:0/3042782227 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f93a4063d10 0x7f93a40661c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:43.373 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.370+0000 7f93c1a3f700 1 --2- 172.21.15.77:0/3042782227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93bc0c5930 0x7f93bc14fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:43.374 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.370+0000 7f93c1a3f700 1 -- 172.21.15.77:0/3042782227 >> 172.21.15.77:0/3042782227 conn(0x7f93bc0cb720 msgr2=0x7f93bc1531f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:43.374 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.370+0000 7f93c1a3f700 1 -- 172.21.15.77:0/3042782227 shutdown_connections 2022-04-03T19:44:43.374 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.370+0000 7f93c1a3f700 1 -- 172.21.15.77:0/3042782227 wait complete. 2022-04-03T19:44:43.395 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-04-03T19:44:07.567638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-04-03T19:44:13.149481+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-04-03T19:44:18.823926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-04-03T19:44:20.275479+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-04-03T19:44:43.396 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-04-03T19:44:43.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.596+0000 7f1966047700 1 Processor -- start 2022-04-03T19:44:43.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.596+0000 7f1966047700 1 -- start start 2022-04-03T19:44:43.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.597+0000 7f1966047700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f196014f250 0x7f1960155b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:43.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.597+0000 7f1966047700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19600c3c60 0x7f1960066da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:43.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.597+0000 7f1966047700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1960069f60 con 0x7f1960156050 2022-04-03T19:44:43.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.597+0000 7f1966047700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1960152e40 con 0x7f196014f250 2022-04-03T19:44:43.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.597+0000 7f1966047700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1960155680 con 0x7f19600c3c60 2022-04-03T19:44:43.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.598+0000 7f1965846700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1960156050 0x7f19600c7250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38352/0 (socket says 172.21.15.77:38352) 2022-04-03T19:44:43.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.598+0000 7f1965045700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f196014f250 0x7f1960155b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:43.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.598+0000 7f1965846700 1 -- 172.21.15.77:0/102156407 learned_addr learned my addr 172.21.15.77:0/102156407 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:43.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.598+0000 7f1964844700 1 --2- 172.21.15.77:0/102156407 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19600c3c60 0x7f1960066da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.598+0000 7f1965045700 1 -- 172.21.15.77:0/102156407 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19600c3c60 msgr2=0x7f1960066da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.598+0000 7f1965045700 1 --2- 172.21.15.77:0/102156407 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19600c3c60 0x7f1960066da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.598+0000 7f194ffff700 1 -- 172.21.15.77:0/102156407 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54777241 0 0) 0x7f1960069f60 con 0x7f1960156050 2022-04-03T19:44:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.598+0000 7f1965045700 1 -- 172.21.15.77:0/102156407 >> v1:172.21.15.77:6789/0 conn(0x7f1960156050 legacy=0x7f19600c7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:43.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.598+0000 7f1965045700 1 -- 172.21.15.77:0/102156407 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19601522c0 con 0x7f196014f250 2022-04-03T19:44:43.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.599+0000 7f1965045700 1 --2- 172.21.15.77:0/102156407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f196014f250 0x7f1960155b10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f195c009760 tx=0x7f195c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c19c3aca6941b379 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:43.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.599+0000 7f194ffff700 1 -- 172.21.15.77:0/102156407 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c00f6b0 con 0x7f196014f250 2022-04-03T19:44:43.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.599+0000 7f194ffff700 1 -- 172.21.15.77:0/102156407 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195c00f810 con 0x7f196014f250 2022-04-03T19:44:43.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.599+0000 7f194ffff700 1 -- 172.21.15.77:0/102156407 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c00fbd0 con 0x7f196014f250 2022-04-03T19:44:43.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.599+0000 7f1966047700 1 -- 172.21.15.77:0/102156407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f196014f250 msgr2=0x7f1960155b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:43.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.599+0000 7f1966047700 1 --2- 172.21.15.77:0/102156407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f196014f250 0x7f1960155b10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f195c009760 tx=0x7f195c009060 comp rx=0 tx=0).stop 2022-04-03T19:44:43.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.600+0000 7f1966047700 1 -- 172.21.15.77:0/102156407 shutdown_connections 2022-04-03T19:44:43.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.600+0000 7f1966047700 1 --2- 172.21.15.77:0/102156407 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f196014f250 0x7f1960155b10 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:43.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.600+0000 7f1966047700 1 --2- 172.21.15.77:0/102156407 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19600c3c60 0x7f1960066da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:43.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.600+0000 7f1966047700 1 -- 172.21.15.77:0/102156407 >> 172.21.15.77:0/102156407 conn(0x7f19600626d0 msgr2=0x7f1960062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:43.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.600+0000 7f1966047700 1 -- 172.21.15.77:0/102156407 shutdown_connections 2022-04-03T19:44:43.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.600+0000 7f1966047700 1 -- 172.21.15.77:0/102156407 wait complete. 2022-04-03T19:44:43.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.601+0000 7f1966047700 1 Processor -- start 2022-04-03T19:44:43.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.601+0000 7f1966047700 1 -- start start 2022-04-03T19:44:43.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.602+0000 7f1966047700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19600c3c60 0x7f196006bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:43.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.602+0000 7f1966047700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f196014f250 0x7f19600c50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:43.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.602+0000 7f1966047700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1960156050 0x7f1960154090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:43.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.602+0000 7f1966047700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1960152060 con 0x7f19600c3c60 2022-04-03T19:44:43.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.602+0000 7f1966047700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19600c8160 con 0x7f196014f250 2022-04-03T19:44:43.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.602+0000 7f1966047700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1960148880 con 0x7f1960156050 2022-04-03T19:44:43.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.602+0000 7f1965045700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19600c3c60 0x7f196006bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:43.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.603+0000 7f1965846700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1960156050 0x7f1960154090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:43.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.603+0000 7f1964844700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f196014f250 0x7f19600c50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:43.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.603+0000 7f1965846700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1960156050 0x7f1960154090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59984/0 (socket says 172.21.15.77:59984) 2022-04-03T19:44:43.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.603+0000 7f1964844700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f196014f250 0x7f19600c50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37874/0 (socket says 172.21.15.77:37874) 2022-04-03T19:44:43.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.603+0000 7f1965846700 1 -- 172.21.15.77:0/578679692 learned_addr learned my addr 172.21.15.77:0/578679692 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:43.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.603+0000 7f1965045700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19600c3c60 0x7f196006bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42676/0 (socket says 172.21.15.77:42676) 2022-04-03T19:44:43.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.603+0000 7f1965045700 1 -- 172.21.15.77:0/578679692 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1960156050 msgr2=0x7f1960154090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:43.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.603+0000 7f1965045700 1 --2- 172.21.15.77:0/578679692 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1960156050 0x7f1960154090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:43.617 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.603+0000 7f1965045700 1 -- 172.21.15.77:0/578679692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f196014f250 msgr2=0x7f19600c50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:43.617 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.603+0000 7f1965045700 1 --2- 172.21.15.77:0/578679692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f196014f250 0x7f19600c50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:43.617 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.603+0000 7f1965045700 1 -- 172.21.15.77:0/578679692 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f195c0092a0 con 0x7f19600c3c60 2022-04-03T19:44:43.617 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.603+0000 7f1965045700 1 --2- 172.21.15.77:0/578679692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19600c3c60 0x7f196006bc70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f19600b7b30 tx=0x7f195c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:43.618 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.604+0000 7f194d7fa700 1 -- 172.21.15.77:0/578679692 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c00f980 con 0x7f19600c3c60 2022-04-03T19:44:43.618 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.604+0000 7f194d7fa700 1 -- 172.21.15.77:0/578679692 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195c00fae0 con 0x7f19600c3c60 2022-04-03T19:44:43.618 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.604+0000 7f194d7fa700 1 -- 172.21.15.77:0/578679692 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c004240 con 0x7f19600c3c60 2022-04-03T19:44:43.618 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.604+0000 7f1966047700 1 -- 172.21.15.77:0/578679692 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1960151210 con 0x7f19600c3c60 2022-04-03T19:44:43.619 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.604+0000 7f1966047700 1 -- 172.21.15.77:0/578679692 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1960151ac0 con 0x7f19600c3c60 2022-04-03T19:44:43.619 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.605+0000 7f1966047700 1 -- 172.21.15.77:0/578679692 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1960150fe0 con 0x7f19600c3c60 2022-04-03T19:44:43.619 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.606+0000 7f194d7fa700 1 -- 172.21.15.77:0/578679692 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f195c0036f0 con 0x7f19600c3c60 2022-04-03T19:44:43.619 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.607+0000 7f194d7fa700 1 --2- 172.21.15.77:0/578679692 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1940063d10 0x7f19400661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:43.619 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.607+0000 7f194d7fa700 1 -- 172.21.15.77:0/578679692 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f195c014070 con 0x7f19600c3c60 2022-04-03T19:44:43.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.607+0000 7f1964844700 1 --2- 172.21.15.77:0/578679692 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1940063d10 0x7f19400661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:43.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.608+0000 7f1964844700 1 --2- 172.21.15.77:0/578679692 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1940063d10 0x7f19400661c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f19600badc0 tx=0x7f1950009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:43.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.612+0000 7f194d7fa700 1 -- 172.21.15.77:0/578679692 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f1960150fe0 con 0x7f19600c3c60 2022-04-03T19:44:43.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.804+0000 7f1966047700 1 -- 172.21.15.77:0/578679692 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f19601514f0 con 0x7f19600c3c60 2022-04-03T19:44:43.805 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.804+0000 7f194d7fa700 1 -- 172.21.15.77:0/578679692 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f195c050c50 con 0x7f19600c3c60 2022-04-03T19:44:43.805 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2022-04-03T19:44:43.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.807+0000 7f1966047700 1 -- 172.21.15.77:0/578679692 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1940063d10 msgr2=0x7f19400661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:43.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.807+0000 7f1966047700 1 --2- 172.21.15.77:0/578679692 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1940063d10 0x7f19400661c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f19600badc0 tx=0x7f1950009060 comp rx=0 tx=0).stop 2022-04-03T19:44:43.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.807+0000 7f1966047700 1 -- 172.21.15.77:0/578679692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19600c3c60 msgr2=0x7f196006bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:43.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.807+0000 7f1966047700 1 --2- 172.21.15.77:0/578679692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19600c3c60 0x7f196006bc70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f19600b7b30 tx=0x7f195c019000 comp rx=0 tx=0).stop 2022-04-03T19:44:43.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.807+0000 7f1966047700 1 -- 172.21.15.77:0/578679692 shutdown_connections 2022-04-03T19:44:43.809 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.807+0000 7f1966047700 1 --2- 172.21.15.77:0/578679692 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1940063d10 0x7f19400661c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:43.809 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.807+0000 7f1966047700 1 --2- 172.21.15.77:0/578679692 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1960156050 0x7f1960154090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:43.809 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.807+0000 7f1966047700 1 --2- 172.21.15.77:0/578679692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19600c3c60 0x7f196006bc70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:43.809 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.807+0000 7f1966047700 1 --2- 172.21.15.77:0/578679692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f196014f250 0x7f19600c50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:43.809 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.807+0000 7f1966047700 1 -- 172.21.15.77:0/578679692 >> 172.21.15.77:0/578679692 conn(0x7f19600626d0 msgr2=0x7f19600c1960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:43.810 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.807+0000 7f1966047700 1 -- 172.21.15.77:0/578679692 shutdown_connections 2022-04-03T19:44:43.810 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:43.807+0000 7f1966047700 1 -- 172.21.15.77:0/578679692 wait complete. 2022-04-03T19:44:43.832 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-04-03T19:44:44.031 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.031+0000 7fe510b64700 1 Processor -- start 2022-04-03T19:44:44.032 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.031+0000 7fe510b64700 1 -- start start 2022-04-03T19:44:44.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.032+0000 7fe510b64700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe50c0c5680 0x7fe50c0c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:44.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.032+0000 7fe510b64700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe50c0c7e80 con 0x7fe50c158e10 2022-04-03T19:44:44.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.032+0000 7fe510b64700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe50c0c6ea0 con 0x7fe50c0c52b0 2022-04-03T19:44:44.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.032+0000 7fe510b64700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe50c0c4260 con 0x7fe50c0c5680 2022-04-03T19:44:44.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.033+0000 7fe50affd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe50c0c5680 0x7fe50c0c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:44.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.033+0000 7fe50b7fe700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fe50c0c52b0 0x7fe50c0c8310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44152/0 (socket says 172.21.15.77:44152) 2022-04-03T19:44:44.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.033+0000 7fe50b7fe700 1 -- 172.21.15.77:0/3216746036 learned_addr learned my addr 172.21.15.77:0/3216746036 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:44.035 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.033+0000 7fe50bfff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe50c158e10 0x7fe50c14ed50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38354/0 (socket says 172.21.15.77:38354) 2022-04-03T19:44:44.036 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.033+0000 7fe50a7fc700 1 -- 172.21.15.77:0/3216746036 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167044014 0 0) 0x7fe50c0c6ea0 con 0x7fe50c0c52b0 2022-04-03T19:44:44.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.033+0000 7fe50affd700 1 -- 172.21.15.77:0/3216746036 >> v1:172.21.15.186:6789/0 conn(0x7fe50c0c52b0 legacy=0x7fe50c0c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:44.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.033+0000 7fe50affd700 1 -- 172.21.15.77:0/3216746036 >> v1:172.21.15.77:6789/0 conn(0x7fe50c158e10 legacy=0x7fe50c14ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:44.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.033+0000 7fe50affd700 1 -- 172.21.15.77:0/3216746036 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe50c0c6600 con 0x7fe50c0c5680 2022-04-03T19:44:44.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.034+0000 7fe50affd700 1 --2- 172.21.15.77:0/3216746036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe50c0c5680 0x7fe50c0c5f70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe500009760 tx=0x7fe500009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f2d3acfc836e91 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:44.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.034+0000 7fe50a7fc700 1 -- 172.21.15.77:0/3216746036 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe500011630 con 0x7fe50c0c5680 2022-04-03T19:44:44.038 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.034+0000 7fe50a7fc700 1 -- 172.21.15.77:0/3216746036 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe500011790 con 0x7fe50c0c5680 2022-04-03T19:44:44.038 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.034+0000 7fe50a7fc700 1 -- 172.21.15.77:0/3216746036 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe500011b10 con 0x7fe50c0c5680 2022-04-03T19:44:44.038 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.034+0000 7fe510b64700 1 -- 172.21.15.77:0/3216746036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe50c0c5680 msgr2=0x7fe50c0c5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:44.038 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.034+0000 7fe510b64700 1 --2- 172.21.15.77:0/3216746036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe50c0c5680 0x7fe50c0c5f70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe500009760 tx=0x7fe500009060 comp rx=0 tx=0).stop 2022-04-03T19:44:44.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.034+0000 7fe510b64700 1 -- 172.21.15.77:0/3216746036 shutdown_connections 2022-04-03T19:44:44.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.034+0000 7fe510b64700 1 --2- 172.21.15.77:0/3216746036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe50c0c5680 0x7fe50c0c5f70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:44.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.034+0000 7fe510b64700 1 -- 172.21.15.77:0/3216746036 >> 172.21.15.77:0/3216746036 conn(0x7fe50c0626d0 msgr2=0x7fe50c062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:44.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.034+0000 7fe510b64700 1 -- 172.21.15.77:0/3216746036 shutdown_connections 2022-04-03T19:44:44.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.034+0000 7fe510b64700 1 -- 172.21.15.77:0/3216746036 wait complete. 2022-04-03T19:44:44.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.036+0000 7fe510b64700 1 Processor -- start 2022-04-03T19:44:44.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.036+0000 7fe510b64700 1 -- start start 2022-04-03T19:44:44.042 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.037+0000 7fe510b64700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe50c0c52b0 0x7fe50c154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:44.042 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.037+0000 7fe510b64700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe50c0c5680 0x7fe50c06cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:44.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.037+0000 7fe510b64700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe50c158e10 0x7fe50c06d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:44.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.037+0000 7fe510b64700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50c150900 con 0x7fe50c158e10 2022-04-03T19:44:44.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.037+0000 7fe510b64700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50c150270 con 0x7fe50c0c5680 2022-04-03T19:44:44.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.037+0000 7fe510b64700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe50c14fde0 con 0x7fe50c0c52b0 2022-04-03T19:44:44.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.037+0000 7fe50bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe50c158e10 0x7fe50c06d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:44.044 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.037+0000 7fe50b7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe50c0c52b0 0x7fe50c154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:44.044 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.037+0000 7fe50affd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe50c0c5680 0x7fe50c06cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:44.044 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.037+0000 7fe50bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe50c158e10 0x7fe50c06d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42680/0 (socket says 172.21.15.77:42680) 2022-04-03T19:44:44.044 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.037+0000 7fe50bfff700 1 -- 172.21.15.77:0/897098453 learned_addr learned my addr 172.21.15.77:0/897098453 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:44.044 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.038+0000 7fe50affd700 1 -- 172.21.15.77:0/897098453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe50c0c52b0 msgr2=0x7fe50c154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:44.045 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.038+0000 7fe50affd700 1 --2- 172.21.15.77:0/897098453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe50c0c52b0 0x7fe50c154380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:44.045 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.038+0000 7fe50affd700 1 -- 172.21.15.77:0/897098453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe50c158e10 msgr2=0x7fe50c06d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:44.045 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.038+0000 7fe50affd700 1 --2- 172.21.15.77:0/897098453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe50c158e10 0x7fe50c06d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:44.045 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.038+0000 7fe50affd700 1 -- 172.21.15.77:0/897098453 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5000092a0 con 0x7fe50c0c5680 2022-04-03T19:44:44.046 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.038+0000 7fe50affd700 1 --2- 172.21.15.77:0/897098453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe50c0c5680 0x7fe50c06cfe0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe500016000 tx=0x7fe500017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:44.046 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.038+0000 7fe4f37fe700 1 -- 172.21.15.77:0/897098453 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe500000bc0 con 0x7fe50c0c5680 2022-04-03T19:44:44.046 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.039+0000 7fe4f37fe700 1 -- 172.21.15.77:0/897098453 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe500011900 con 0x7fe50c0c5680 2022-04-03T19:44:44.046 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.039+0000 7fe4f37fe700 1 -- 172.21.15.77:0/897098453 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe500025670 con 0x7fe50c0c5680 2022-04-03T19:44:44.046 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.039+0000 7fe510b64700 1 -- 172.21.15.77:0/897098453 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe50c151d20 con 0x7fe50c0c5680 2022-04-03T19:44:44.047 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.039+0000 7fe510b64700 1 -- 172.21.15.77:0/897098453 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe50c14f790 con 0x7fe50c0c5680 2022-04-03T19:44:44.047 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.040+0000 7fe510b64700 1 -- 172.21.15.77:0/897098453 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe50c14f5f0 con 0x7fe50c0c5680 2022-04-03T19:44:44.047 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.041+0000 7fe4f37fe700 1 -- 172.21.15.77:0/897098453 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fe50000f050 con 0x7fe50c0c5680 2022-04-03T19:44:44.047 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.041+0000 7fe4f37fe700 1 --2- 172.21.15.77:0/897098453 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe4ec063ce0 0x7fe4ec066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:44.047 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.042+0000 7fe4f37fe700 1 -- 172.21.15.77:0/897098453 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fe500082160 con 0x7fe50c0c5680 2022-04-03T19:44:44.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.042+0000 7fe50b7fe700 1 --2- 172.21.15.77:0/897098453 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe4ec063ce0 0x7fe4ec066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:44.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.042+0000 7fe50b7fe700 1 --2- 172.21.15.77:0/897098453 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe4ec063ce0 0x7fe4ec066190 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc0047d0 tx=0x7fe4fc007bb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:44.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.045+0000 7fe4f37fe700 1 -- 172.21.15.77:0/897098453 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fe500050d80 con 0x7fe50c0c5680 2022-04-03T19:44:44.242 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.242+0000 7fe510b64700 1 -- 172.21.15.77:0/897098453 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe50c14f5f0 con 0x7fe50c0c5680 2022-04-03T19:44:44.243 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.243+0000 7fe4f37fe700 1 -- 172.21.15.77:0/897098453 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe50002a020 con 0x7fe50c0c5680 2022-04-03T19:44:44.243 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2022-04-03T19:44:44.246 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.245+0000 7fe510b64700 1 -- 172.21.15.77:0/897098453 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe4ec063ce0 msgr2=0x7fe4ec066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:44.246 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.245+0000 7fe510b64700 1 --2- 172.21.15.77:0/897098453 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe4ec063ce0 0x7fe4ec066190 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc0047d0 tx=0x7fe4fc007bb0 comp rx=0 tx=0).stop 2022-04-03T19:44:44.246 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.245+0000 7fe510b64700 1 -- 172.21.15.77:0/897098453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe50c0c5680 msgr2=0x7fe50c06cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:44.247 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.245+0000 7fe510b64700 1 --2- 172.21.15.77:0/897098453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe50c0c5680 0x7fe50c06cfe0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe500016000 tx=0x7fe500017000 comp rx=0 tx=0).stop 2022-04-03T19:44:44.247 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.245+0000 7fe510b64700 1 -- 172.21.15.77:0/897098453 shutdown_connections 2022-04-03T19:44:44.247 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.245+0000 7fe510b64700 1 --2- 172.21.15.77:0/897098453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe50c158e10 0x7fe50c06d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:44.247 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.245+0000 7fe510b64700 1 --2- 172.21.15.77:0/897098453 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe4ec063ce0 0x7fe4ec066190 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:44.247 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.245+0000 7fe510b64700 1 --2- 172.21.15.77:0/897098453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe50c0c52b0 0x7fe50c154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:44.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.245+0000 7fe510b64700 1 --2- 172.21.15.77:0/897098453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe50c0c5680 0x7fe50c06cfe0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:44.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.245+0000 7fe510b64700 1 -- 172.21.15.77:0/897098453 >> 172.21.15.77:0/897098453 conn(0x7fe50c0626d0 msgr2=0x7fe50c157790 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:44.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.245+0000 7fe510b64700 1 -- 172.21.15.77:0/897098453 shutdown_connections 2022-04-03T19:44:44.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.245+0000 7fe510b64700 1 -- 172.21.15.77:0/897098453 wait complete. 2022-04-03T19:44:44.270 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-04-03T19:44:44.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.470+0000 7f2057fff700 1 Processor -- start 2022-04-03T19:44:44.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.470+0000 7f2057fff700 1 -- start start 2022-04-03T19:44:44.472 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.472+0000 7f2057fff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f205806bdb0 0x7f20581559d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:44.472 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.472+0000 7f2057fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f205806df20 con 0x7f205814ef20 2022-04-03T19:44:44.473 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.472+0000 7f2057fff700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20580c6690 con 0x7f20580c5170 2022-04-03T19:44:44.473 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.472+0000 7f2057fff700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2058155f10 con 0x7f205806bdb0 2022-04-03T19:44:44.473 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.472+0000 7f2056ffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f205814ef20 0x7f20580c5f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38356/0 (socket says 172.21.15.77:38356) 2022-04-03T19:44:44.473 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.472+0000 7f2056ffd700 1 -- 172.21.15.77:0/2999118413 learned_addr learned my addr 172.21.15.77:0/2999118413 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:44.473 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.472+0000 7f20567fc700 1 --2- 172.21.15.77:0/2999118413 >> v2:172.21.15.186:3300/0 conn(0x7f205806bdb0 0x7f20581559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:44.475 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.472+0000 7f2055ffb700 1 -- 172.21.15.77:0/2999118413 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107091826 0 0) 0x7f205806df20 con 0x7f205814ef20 2022-04-03T19:44:44.475 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.472+0000 7f2055ffb700 1 -- 172.21.15.77:0/2999118413 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20440033f0 con 0x7f205814ef20 2022-04-03T19:44:44.476 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.473+0000 7f2055ffb700 1 -- 172.21.15.77:0/2999118413 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982794086 0 0) 0x7f20580c6690 con 0x7f20580c5170 2022-04-03T19:44:44.476 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.473+0000 7f2055ffb700 1 -- 172.21.15.77:0/2999118413 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f205806df20 con 0x7f20580c5170 2022-04-03T19:44:44.476 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.473+0000 7f20567fc700 1 -- 172.21.15.77:0/2999118413 >> v1:172.21.15.186:6789/0 conn(0x7f20580c5170 legacy=0x7f205815f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:44.476 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.473+0000 7f20567fc700 1 -- 172.21.15.77:0/2999118413 >> v1:172.21.15.77:6789/0 conn(0x7f205814ef20 legacy=0x7f20580c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:44.477 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.473+0000 7f20567fc700 1 -- 172.21.15.77:0/2999118413 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205806cd90 con 0x7f205806bdb0 2022-04-03T19:44:44.477 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.473+0000 7f20567fc700 1 --2- 172.21.15.77:0/2999118413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f205806bdb0 0x7f20581559d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f204c009760 tx=0x7f204c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10178a07326fcd2e server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:44.477 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.473+0000 7f2055ffb700 1 -- 172.21.15.77:0/2999118413 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204c011630 con 0x7f205806bdb0 2022-04-03T19:44:44.477 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.473+0000 7f2055ffb700 1 -- 172.21.15.77:0/2999118413 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f204c011790 con 0x7f205806bdb0 2022-04-03T19:44:44.477 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.473+0000 7f2055ffb700 1 -- 172.21.15.77:0/2999118413 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f204c011b10 con 0x7f205806bdb0 2022-04-03T19:44:44.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.474+0000 7f2057fff700 1 -- 172.21.15.77:0/2999118413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f205806bdb0 msgr2=0x7f20581559d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:44.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.474+0000 7f2057fff700 1 --2- 172.21.15.77:0/2999118413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f205806bdb0 0x7f20581559d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f204c009760 tx=0x7f204c009060 comp rx=0 tx=0).stop 2022-04-03T19:44:44.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.474+0000 7f2057fff700 1 -- 172.21.15.77:0/2999118413 shutdown_connections 2022-04-03T19:44:44.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.474+0000 7f2057fff700 1 --2- 172.21.15.77:0/2999118413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f205806bdb0 0x7f20581559d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:44.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.474+0000 7f2057fff700 1 -- 172.21.15.77:0/2999118413 >> 172.21.15.77:0/2999118413 conn(0x7f20580cb6d0 msgr2=0x7f20580cbad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:44.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.474+0000 7f2057fff700 1 -- 172.21.15.77:0/2999118413 shutdown_connections 2022-04-03T19:44:44.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.474+0000 7f2057fff700 1 -- 172.21.15.77:0/2999118413 wait complete. 2022-04-03T19:44:44.481 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.475+0000 7f2057fff700 1 Processor -- start 2022-04-03T19:44:44.482 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.476+0000 7f2057fff700 1 -- start start 2022-04-03T19:44:44.482 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.476+0000 7f2057fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f205806bdb0 0x7f2058166440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:44.482 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.476+0000 7f2057fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20580c5170 0x7f20580c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:44.482 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.476+0000 7f2057fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f205814ef20 0x7f205814b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:44.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.477+0000 7f2057fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205815bad0 con 0x7f205814ef20 2022-04-03T19:44:44.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.477+0000 7f2057fff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20581511d0 con 0x7f205806bdb0 2022-04-03T19:44:44.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.477+0000 7f2057fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f205815d080 con 0x7f20580c5170 2022-04-03T19:44:44.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.477+0000 7f20577fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f205814ef20 0x7f205814b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:44.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.477+0000 7f2056ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f205806bdb0 0x7f2058166440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:44.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.477+0000 7f20577fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f205814ef20 0x7f205814b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42682/0 (socket says 172.21.15.77:42682) 2022-04-03T19:44:44.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.477+0000 7f20577fe700 1 -- 172.21.15.77:0/2576697999 learned_addr learned my addr 172.21.15.77:0/2576697999 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:44.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.477+0000 7f20567fc700 1 --2- 172.21.15.77:0/2576697999 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20580c5170 0x7f20580c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:44.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.477+0000 7f20577fe700 1 -- 172.21.15.77:0/2576697999 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20580c5170 msgr2=0x7f20580c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:44.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.477+0000 7f20577fe700 1 --2- 172.21.15.77:0/2576697999 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20580c5170 0x7f20580c9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:44.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.477+0000 7f20577fe700 1 -- 172.21.15.77:0/2576697999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f205806bdb0 msgr2=0x7f2058166440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:44.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.477+0000 7f20577fe700 1 --2- 172.21.15.77:0/2576697999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f205806bdb0 0x7f2058166440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:44.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.477+0000 7f20577fe700 1 -- 172.21.15.77:0/2576697999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204c0092a0 con 0x7f205814ef20 2022-04-03T19:44:44.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.478+0000 7f20577fe700 1 --2- 172.21.15.77:0/2576697999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f205814ef20 0x7f205814b3b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f205000c7c0 tx=0x7f2050007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:44.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.478+0000 7f203effd700 1 -- 172.21.15.77:0/2576697999 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2050003840 con 0x7f205814ef20 2022-04-03T19:44:44.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.478+0000 7f203effd700 1 -- 172.21.15.77:0/2576697999 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20500039a0 con 0x7f205814ef20 2022-04-03T19:44:44.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.478+0000 7f2057fff700 1 -- 172.21.15.77:0/2576697999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f205815d320 con 0x7f205814ef20 2022-04-03T19:44:44.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.478+0000 7f203effd700 1 -- 172.21.15.77:0/2576697999 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2050003d60 con 0x7f205814ef20 2022-04-03T19:44:44.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.478+0000 7f2057fff700 1 -- 172.21.15.77:0/2576697999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2058150b80 con 0x7f205814ef20 2022-04-03T19:44:44.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.479+0000 7f2057fff700 1 -- 172.21.15.77:0/2576697999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2058151360 con 0x7f205814ef20 2022-04-03T19:44:44.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.480+0000 7f203effd700 1 -- 172.21.15.77:0/2576697999 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f2050019030 con 0x7f205814ef20 2022-04-03T19:44:44.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.481+0000 7f203effd700 1 --2- 172.21.15.77:0/2576697999 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2038063ce0 0x7f2038066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:44.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.481+0000 7f203effd700 1 -- 172.21.15.77:0/2576697999 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f205007fcb0 con 0x7f205814ef20 2022-04-03T19:44:44.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.482+0000 7f2056ffd700 1 --2- 172.21.15.77:0/2576697999 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2038063ce0 0x7f2038066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:44.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.482+0000 7f2056ffd700 1 --2- 172.21.15.77:0/2576697999 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2038063ce0 0x7f2038066190 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2048005750 tx=0x7f2048009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:44.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.486+0000 7f203effd700 1 -- 172.21.15.77:0/2576697999 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f205004e890 con 0x7f205814ef20 2022-04-03T19:44:44.569 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:44:44.574 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:44:44.684 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.684+0000 7f2057fff700 1 -- 172.21.15.77:0/2576697999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2058088dd0 con 0x7f205814ef20 2022-04-03T19:44:44.685 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.684+0000 7f203effd700 1 -- 172.21.15.77:0/2576697999 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2050051eb0 con 0x7f205814ef20 2022-04-03T19:44:44.685 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2022-04-03T19:44:44.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.687+0000 7f2057fff700 1 -- 172.21.15.77:0/2576697999 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2038063ce0 msgr2=0x7f2038066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:44.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.687+0000 7f2057fff700 1 --2- 172.21.15.77:0/2576697999 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2038063ce0 0x7f2038066190 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2048005750 tx=0x7f2048009040 comp rx=0 tx=0).stop 2022-04-03T19:44:44.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.687+0000 7f2057fff700 1 -- 172.21.15.77:0/2576697999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f205814ef20 msgr2=0x7f205814b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:44.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.687+0000 7f2057fff700 1 --2- 172.21.15.77:0/2576697999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f205814ef20 0x7f205814b3b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f205000c7c0 tx=0x7f2050007f70 comp rx=0 tx=0).stop 2022-04-03T19:44:44.689 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.687+0000 7f2057fff700 1 -- 172.21.15.77:0/2576697999 shutdown_connections 2022-04-03T19:44:44.689 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.687+0000 7f2057fff700 1 --2- 172.21.15.77:0/2576697999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f205814ef20 0x7f205814b3b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:44.689 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.687+0000 7f2057fff700 1 --2- 172.21.15.77:0/2576697999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f205806bdb0 0x7f2058166440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:44.689 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.687+0000 7f2057fff700 1 --2- 172.21.15.77:0/2576697999 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2038063ce0 0x7f2038066190 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:44.689 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.687+0000 7f2057fff700 1 --2- 172.21.15.77:0/2576697999 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20580c5170 0x7f20580c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:44.690 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.687+0000 7f2057fff700 1 -- 172.21.15.77:0/2576697999 >> 172.21.15.77:0/2576697999 conn(0x7f20580cb6d0 msgr2=0x7f205814d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:44.690 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.687+0000 7f2057fff700 1 -- 172.21.15.77:0/2576697999 shutdown_connections 2022-04-03T19:44:44.690 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.687+0000 7f2057fff700 1 -- 172.21.15.77:0/2576697999 wait complete. 2022-04-03T19:44:44.712 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-04-03T19:44:44.913 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.913+0000 7f1741900700 1 Processor -- start 2022-04-03T19:44:44.913 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.913+0000 7f1741900700 1 -- start start 2022-04-03T19:44:44.915 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.914+0000 7f1741900700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f173c1559d0 0x7f173c160910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:44.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.914+0000 7f1741900700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f173c06df20 con 0x7f173c06bdb0 2022-04-03T19:44:44.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.914+0000 7f1741900700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f173c0c6690 con 0x7f173c14ef20 2022-04-03T19:44:44.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.914+0000 7f1741900700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f173c06cd90 con 0x7f173c1559d0 2022-04-03T19:44:44.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.914+0000 7f17410ff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f173c1559d0 0x7f173c160910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:44.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.914+0000 7f17410ff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f173c1559d0 0x7f173c160910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59990/0 (socket says 172.21.15.77:59990) 2022-04-03T19:44:44.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.914+0000 7f17408fe700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f173c14ef20 0x7f173c0c5f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44156/0 (socket says 172.21.15.77:44156) 2022-04-03T19:44:44.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.914+0000 7f17410ff700 1 -- 172.21.15.77:0/615530848 learned_addr learned my addr 172.21.15.77:0/615530848 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:44.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.914+0000 7f173bfff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f173c06bdb0 0x7f173c0c5170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38358/0 (socket says 172.21.15.77:38358) 2022-04-03T19:44:44.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.915+0000 7f17410ff700 1 -- 172.21.15.77:0/615530848 >> v1:172.21.15.186:6789/0 conn(0x7f173c14ef20 legacy=0x7f173c0c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:44.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.915+0000 7f17410ff700 1 -- 172.21.15.77:0/615530848 >> v1:172.21.15.77:6789/0 conn(0x7f173c06bdb0 legacy=0x7f173c0c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:44.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.915+0000 7f173b7fe700 1 -- 172.21.15.77:0/615530848 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297047759 0 0) 0x7f173c06df20 con 0x7f173c06bdb0 2022-04-03T19:44:44.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.915+0000 7f17410ff700 1 -- 172.21.15.77:0/615530848 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f173c155e50 con 0x7f173c1559d0 2022-04-03T19:44:44.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.915+0000 7f17410ff700 1 --2- 172.21.15.77:0/615530848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f173c1559d0 0x7f173c160910 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1734009f50 tx=0x7f1734009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9454dd72cddc3ff0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:44.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.916+0000 7f173b7fe700 1 -- 172.21.15.77:0/615530848 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173400e070 con 0x7f173c1559d0 2022-04-03T19:44:44.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.916+0000 7f173b7fe700 1 -- 172.21.15.77:0/615530848 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173400bb40 con 0x7f173c1559d0 2022-04-03T19:44:44.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.916+0000 7f173b7fe700 1 -- 172.21.15.77:0/615530848 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17340092e0 con 0x7f173c1559d0 2022-04-03T19:44:44.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.916+0000 7f1741900700 1 -- 172.21.15.77:0/615530848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f173c1559d0 msgr2=0x7f173c160910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:44.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.916+0000 7f1741900700 1 --2- 172.21.15.77:0/615530848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f173c1559d0 0x7f173c160910 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1734009f50 tx=0x7f1734009840 comp rx=0 tx=0).stop 2022-04-03T19:44:44.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.916+0000 7f1741900700 1 -- 172.21.15.77:0/615530848 shutdown_connections 2022-04-03T19:44:44.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.916+0000 7f1741900700 1 --2- 172.21.15.77:0/615530848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f173c1559d0 0x7f173c160910 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:44.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.916+0000 7f1741900700 1 -- 172.21.15.77:0/615530848 >> 172.21.15.77:0/615530848 conn(0x7f173c0cb6d0 msgr2=0x7f173c0cbad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:44.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.916+0000 7f1741900700 1 -- 172.21.15.77:0/615530848 shutdown_connections 2022-04-03T19:44:44.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.916+0000 7f1741900700 1 -- 172.21.15.77:0/615530848 wait complete. 2022-04-03T19:44:44.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.918+0000 7f1741900700 1 Processor -- start 2022-04-03T19:44:44.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.918+0000 7f1741900700 1 -- start start 2022-04-03T19:44:44.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.919+0000 7f1741900700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f173c06bdb0 0x7f173c150980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:44.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.919+0000 7f1741900700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f173c14ef20 0x7f173c165710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:44.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.919+0000 7f1741900700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f173c1559d0 0x7f173c148470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:44.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.919+0000 7f1741900700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173c166450 con 0x7f173c1559d0 2022-04-03T19:44:44.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.919+0000 7f1741900700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173c1504f0 con 0x7f173c14ef20 2022-04-03T19:44:44.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.919+0000 7f1741900700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f173c152900 con 0x7f173c06bdb0 2022-04-03T19:44:44.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.919+0000 7f17410ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f173c1559d0 0x7f173c148470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:44.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.919+0000 7f17408fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f173c06bdb0 0x7f173c150980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:44.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.919+0000 7f173bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f173c14ef20 0x7f173c165710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:44.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.919+0000 7f17410ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f173c1559d0 0x7f173c148470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42684/0 (socket says 172.21.15.77:42684) 2022-04-03T19:44:44.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.919+0000 7f17410ff700 1 -- 172.21.15.77:0/2932164237 learned_addr learned my addr 172.21.15.77:0/2932164237 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:44.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.919+0000 7f17410ff700 1 -- 172.21.15.77:0/2932164237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f173c06bdb0 msgr2=0x7f173c150980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:44.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.919+0000 7f17410ff700 1 --2- 172.21.15.77:0/2932164237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f173c06bdb0 0x7f173c150980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:44.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.919+0000 7f17410ff700 1 -- 172.21.15.77:0/2932164237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f173c14ef20 msgr2=0x7f173c165710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:44.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.919+0000 7f17410ff700 1 --2- 172.21.15.77:0/2932164237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f173c14ef20 0x7f173c165710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:44.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.919+0000 7f17410ff700 1 -- 172.21.15.77:0/2932164237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1734009ae0 con 0x7f173c1559d0 2022-04-03T19:44:44.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.920+0000 7f17410ff700 1 --2- 172.21.15.77:0/2932164237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f173c1559d0 0x7f173c148470 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f173400b770 tx=0x7f1734018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:44.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.920+0000 7f1738ff9700 1 -- 172.21.15.77:0/2932164237 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1734009940 con 0x7f173c1559d0 2022-04-03T19:44:44.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.920+0000 7f1738ff9700 1 -- 172.21.15.77:0/2932164237 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1734007d00 con 0x7f173c1559d0 2022-04-03T19:44:44.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.920+0000 7f1738ff9700 1 -- 172.21.15.77:0/2932164237 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173401d670 con 0x7f173c1559d0 2022-04-03T19:44:44.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.920+0000 7f1741900700 1 -- 172.21.15.77:0/2932164237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f173c152660 con 0x7f173c1559d0 2022-04-03T19:44:44.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.920+0000 7f1741900700 1 -- 172.21.15.77:0/2932164237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f173c150180 con 0x7f173c1559d0 2022-04-03T19:44:44.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.921+0000 7f1741900700 1 -- 172.21.15.77:0/2932164237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f173c150680 con 0x7f173c1559d0 2022-04-03T19:44:44.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.922+0000 7f1738ff9700 1 -- 172.21.15.77:0/2932164237 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f173402e070 con 0x7f173c1559d0 2022-04-03T19:44:44.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.923+0000 7f1738ff9700 1 --2- 172.21.15.77:0/2932164237 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1724063ce0 0x7f1724066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:44.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.923+0000 7f1738ff9700 1 -- 172.21.15.77:0/2932164237 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f1734082420 con 0x7f173c1559d0 2022-04-03T19:44:44.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.923+0000 7f17408fe700 1 --2- 172.21.15.77:0/2932164237 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1724063ce0 0x7f1724066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:44.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.924+0000 7f17408fe700 1 --2- 172.21.15.77:0/2932164237 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1724063ce0 0x7f1724066190 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1730007fd0 tx=0x7f1730007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:44.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:44.927+0000 7f1738ff9700 1 -- 172.21.15.77:0/2932164237 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f1734051000 con 0x7f173c1559d0 2022-04-03T19:44:45.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.125+0000 7f1741900700 1 -- 172.21.15.77:0/2932164237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f173c06d030 con 0x7f173c1559d0 2022-04-03T19:44:45.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.125+0000 7f1738ff9700 1 -- 172.21.15.77:0/2932164237 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f17340220a0 con 0x7f173c1559d0 2022-04-03T19:44:45.126 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2022-04-03T19:44:45.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.128+0000 7f1741900700 1 -- 172.21.15.77:0/2932164237 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1724063ce0 msgr2=0x7f1724066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:45.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.128+0000 7f1741900700 1 --2- 172.21.15.77:0/2932164237 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1724063ce0 0x7f1724066190 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1730007fd0 tx=0x7f1730007d30 comp rx=0 tx=0).stop 2022-04-03T19:44:45.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.128+0000 7f1741900700 1 -- 172.21.15.77:0/2932164237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f173c1559d0 msgr2=0x7f173c148470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:45.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.128+0000 7f1741900700 1 --2- 172.21.15.77:0/2932164237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f173c1559d0 0x7f173c148470 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f173400b770 tx=0x7f1734018000 comp rx=0 tx=0).stop 2022-04-03T19:44:45.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.128+0000 7f1741900700 1 -- 172.21.15.77:0/2932164237 shutdown_connections 2022-04-03T19:44:45.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.128+0000 7f1741900700 1 --2- 172.21.15.77:0/2932164237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f173c1559d0 0x7f173c148470 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.128+0000 7f1741900700 1 --2- 172.21.15.77:0/2932164237 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1724063ce0 0x7f1724066190 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.128+0000 7f1741900700 1 --2- 172.21.15.77:0/2932164237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f173c06bdb0 0x7f173c150980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.128+0000 7f1741900700 1 --2- 172.21.15.77:0/2932164237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f173c14ef20 0x7f173c165710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.128+0000 7f1741900700 1 -- 172.21.15.77:0/2932164237 >> 172.21.15.77:0/2932164237 conn(0x7f173c0cb6d0 msgr2=0x7f173c14d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:45.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.128+0000 7f1741900700 1 -- 172.21.15.77:0/2932164237 shutdown_connections 2022-04-03T19:44:45.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.128+0000 7f1741900700 1 -- 172.21.15.77:0/2932164237 wait complete. 2022-04-03T19:44:45.153 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2022-04-03T19:44:45.153 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:44:45.356 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.355+0000 7f4ae1c02700 1 Processor -- start 2022-04-03T19:44:45.356 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.355+0000 7f4ae1c02700 1 -- start start 2022-04-03T19:44:45.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.357+0000 7f4ae1c02700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4adc06d3f0 0x7f4adc06d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:45.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.357+0000 7f4ae1c02700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4adc0c2a90 0x7f4adc06c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:45.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.357+0000 7f4ae1c02700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4adc1562f0 con 0x7f4adc0c5930 2022-04-03T19:44:45.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.357+0000 7f4ae1c02700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4adc0c2f80 con 0x7f4adc0c2a90 2022-04-03T19:44:45.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.357+0000 7f4ae1c02700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc0c3260 con 0x7f4adc06d3f0 2022-04-03T19:44:45.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.357+0000 7f4ae0c00700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4adc06d3f0 0x7f4adc06d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:45.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.357+0000 7f4ae1401700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4adc0c2a90 0x7f4adc06c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:45.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.357+0000 7f4ae0c00700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4adc06d3f0 0x7f4adc06d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59994/0 (socket says 172.21.15.77:59994) 2022-04-03T19:44:45.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.357+0000 7f4ae1401700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4adc0c2a90 0x7f4adc06c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37884/0 (socket says 172.21.15.77:37884) 2022-04-03T19:44:45.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.357+0000 7f4ae0c00700 1 -- 172.21.15.77:0/170279545 learned_addr learned my addr 172.21.15.77:0/170279545 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:45.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.357+0000 7f4ad3fff700 1 --1- 172.21.15.77:0/170279545 >> v1:172.21.15.186:6789/0 conn(0x7f4adc0c5930 0x7f4adc155be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44158/0 (socket says 172.21.15.77:44158) 2022-04-03T19:44:45.361 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.357+0000 7f4ae0c00700 1 -- 172.21.15.77:0/170279545 >> v2:172.21.15.186:3300/0 conn(0x7f4adc0c2a90 msgr2=0x7f4adc06c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:45.361 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.357+0000 7f4ae0c00700 1 --2- 172.21.15.77:0/170279545 >> v2:172.21.15.186:3300/0 conn(0x7f4adc0c2a90 0x7f4adc06c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.362 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.358+0000 7f4ae0c00700 1 -- 172.21.15.77:0/170279545 >> v1:172.21.15.186:6789/0 conn(0x7f4adc0c5930 legacy=0x7f4adc155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:45.362 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.358+0000 7f4ae0c00700 1 -- 172.21.15.77:0/170279545 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4adc0c3540 con 0x7f4adc06d3f0 2022-04-03T19:44:45.362 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.358+0000 7f4ae0c00700 1 --2- 172.21.15.77:0/170279545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4adc06d3f0 0x7f4adc06d7c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8006fd0 tx=0x7f4ad80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec00582f653d9072 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:45.362 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.358+0000 7f4ad37fe700 1 -- 172.21.15.77:0/170279545 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8004030 con 0x7f4adc06d3f0 2022-04-03T19:44:45.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.358+0000 7f4ad37fe700 1 -- 172.21.15.77:0/170279545 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad80041d0 con 0x7f4adc06d3f0 2022-04-03T19:44:45.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.358+0000 7f4ad37fe700 1 -- 172.21.15.77:0/170279545 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8011630 con 0x7f4adc06d3f0 2022-04-03T19:44:45.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.358+0000 7f4ae1c02700 1 -- 172.21.15.77:0/170279545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4adc06d3f0 msgr2=0x7f4adc06d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:45.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.358+0000 7f4ae1c02700 1 --2- 172.21.15.77:0/170279545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4adc06d3f0 0x7f4adc06d7c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8006fd0 tx=0x7f4ad80090f0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.359+0000 7f4ae1c02700 1 -- 172.21.15.77:0/170279545 shutdown_connections 2022-04-03T19:44:45.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.359+0000 7f4ae1c02700 1 --2- 172.21.15.77:0/170279545 >> v2:172.21.15.186:3300/0 conn(0x7f4adc0c2a90 0x7f4adc06c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.359+0000 7f4ae1c02700 1 --2- 172.21.15.77:0/170279545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4adc06d3f0 0x7f4adc06d7c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.359+0000 7f4ae1c02700 1 -- 172.21.15.77:0/170279545 >> 172.21.15.77:0/170279545 conn(0x7f4adc0cb720 msgr2=0x7f4adc0cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:45.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.359+0000 7f4ae1c02700 1 -- 172.21.15.77:0/170279545 shutdown_connections 2022-04-03T19:44:45.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.359+0000 7f4ae1c02700 1 -- 172.21.15.77:0/170279545 wait complete. 2022-04-03T19:44:45.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.360+0000 7f4ae1c02700 1 Processor -- start 2022-04-03T19:44:45.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.360+0000 7f4ae1c02700 1 -- start start 2022-04-03T19:44:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.361+0000 7f4ae1c02700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4adc06d3f0 0x7f4adc0c0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.361+0000 7f4ae1c02700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4adc0c2a90 0x7f4adc0c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.361+0000 7f4ae1c02700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4adc0c5930 0x7f4adc14aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.361+0000 7f4ae1c02700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc1507d0 con 0x7f4adc06d3f0 2022-04-03T19:44:45.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.361+0000 7f4ae1c02700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc1666a0 con 0x7f4adc0c2a90 2022-04-03T19:44:45.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.361+0000 7f4ae1c02700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4adc0c0b50 con 0x7f4adc0c5930 2022-04-03T19:44:45.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.362+0000 7f4ae0c00700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4adc06d3f0 0x7f4adc0c0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:45.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.362+0000 7f4ae0c00700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4adc06d3f0 0x7f4adc0c0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42686/0 (socket says 172.21.15.77:42686) 2022-04-03T19:44:45.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.362+0000 7f4ae0c00700 1 -- 172.21.15.77:0/1947543126 learned_addr learned my addr 172.21.15.77:0/1947543126 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:45.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.362+0000 7f4ad3fff700 1 --2- 172.21.15.77:0/1947543126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4adc0c2a90 0x7f4adc0c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:45.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.362+0000 7f4ae1401700 1 --2- 172.21.15.77:0/1947543126 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4adc0c5930 0x7f4adc14aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:45.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.362+0000 7f4ae0c00700 1 -- 172.21.15.77:0/1947543126 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4adc0c5930 msgr2=0x7f4adc14aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:45.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.362+0000 7f4ae0c00700 1 --2- 172.21.15.77:0/1947543126 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4adc0c5930 0x7f4adc14aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.362+0000 7f4ae0c00700 1 -- 172.21.15.77:0/1947543126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4adc0c2a90 msgr2=0x7f4adc0c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:45.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.362+0000 7f4ae0c00700 1 --2- 172.21.15.77:0/1947543126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4adc0c2a90 0x7f4adc0c6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.362+0000 7f4ae0c00700 1 -- 172.21.15.77:0/1947543126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad8009370 con 0x7f4adc06d3f0 2022-04-03T19:44:45.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.363+0000 7f4ae0c00700 1 --2- 172.21.15.77:0/1947543126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4adc06d3f0 0x7f4adc0c0fe0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8004000 tx=0x7f4ad8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:45.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.363+0000 7f4ad0ff9700 1 -- 172.21.15.77:0/1947543126 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad80036f0 con 0x7f4adc06d3f0 2022-04-03T19:44:45.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.363+0000 7f4ad0ff9700 1 -- 172.21.15.77:0/1947543126 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad8003850 con 0x7f4adc06d3f0 2022-04-03T19:44:45.372 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.363+0000 7f4ae1c02700 1 -- 172.21.15.77:0/1947543126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4adc150210 con 0x7f4adc06d3f0 2022-04-03T19:44:45.372 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.363+0000 7f4ad0ff9700 1 -- 172.21.15.77:0/1947543126 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8003c10 con 0x7f4adc06d3f0 2022-04-03T19:44:45.372 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.363+0000 7f4ae1c02700 1 -- 172.21.15.77:0/1947543126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4adc0c07e0 con 0x7f4adc06d3f0 2022-04-03T19:44:45.372 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.364+0000 7f4ae1c02700 1 -- 172.21.15.77:0/1947543126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4adc0c0640 con 0x7f4adc06d3f0 2022-04-03T19:44:45.373 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.365+0000 7f4ad0ff9700 1 -- 172.21.15.77:0/1947543126 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f4ad802b030 con 0x7f4adc06d3f0 2022-04-03T19:44:45.373 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.366+0000 7f4ad0ff9700 1 --2- 172.21.15.77:0/1947543126 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4abc063ce0 0x7f4abc066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:45.373 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.366+0000 7f4ad0ff9700 1 -- 172.21.15.77:0/1947543126 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f4ad8082060 con 0x7f4adc06d3f0 2022-04-03T19:44:45.373 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.366+0000 7f4ad3fff700 1 --2- 172.21.15.77:0/1947543126 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4abc063ce0 0x7f4abc066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:45.373 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.367+0000 7f4ad3fff700 1 --2- 172.21.15.77:0/1947543126 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4abc063ce0 0x7f4abc066190 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4adc15ffd0 tx=0x7f4acc007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:45.374 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.370+0000 7f4ad0ff9700 1 -- 172.21.15.77:0/1947543126 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f4adc0c0640 con 0x7f4adc06d3f0 2022-04-03T19:44:45.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.605+0000 7f4ae1c02700 1 -- 172.21.15.77:0/1947543126 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4adc0c0640 con 0x7f4adc06d3f0 2022-04-03T19:44:45.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.605+0000 7f4ad0ff9700 1 -- 172.21.15.77:0/1947543126 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6107 (secure 0 0 0) 0x7f4adc0c0640 con 0x7f4adc06d3f0 2022-04-03T19:44:45.606 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:45.607 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4303,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:44:27.295743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4327":{"gid":4327,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6837/3665969042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3665969042},{"type":"v1","addr":"172.21.15.77:6837","nonce":3665969042}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:44:45.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.608+0000 7f4ae1c02700 1 -- 172.21.15.77:0/1947543126 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4abc063ce0 msgr2=0x7f4abc066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:45.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.608+0000 7f4ae1c02700 1 --2- 172.21.15.77:0/1947543126 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4abc063ce0 0x7f4abc066190 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4adc15ffd0 tx=0x7f4acc007d30 comp rx=0 tx=0).stop 2022-04-03T19:44:45.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.608+0000 7f4ae1c02700 1 -- 172.21.15.77:0/1947543126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4adc06d3f0 msgr2=0x7f4adc0c0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:45.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.608+0000 7f4ae1c02700 1 --2- 172.21.15.77:0/1947543126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4adc06d3f0 0x7f4adc0c0fe0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8004000 tx=0x7f4ad8016000 comp rx=0 tx=0).stop 2022-04-03T19:44:45.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.608+0000 7f4ae1c02700 1 -- 172.21.15.77:0/1947543126 shutdown_connections 2022-04-03T19:44:45.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.608+0000 7f4ae1c02700 1 --2- 172.21.15.77:0/1947543126 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4abc063ce0 0x7f4abc066190 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.608+0000 7f4ae1c02700 1 --2- 172.21.15.77:0/1947543126 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4adc0c5930 0x7f4adc14aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.608+0000 7f4ae1c02700 1 --2- 172.21.15.77:0/1947543126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4adc06d3f0 0x7f4adc0c0fe0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.608+0000 7f4ae1c02700 1 --2- 172.21.15.77:0/1947543126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4adc0c2a90 0x7f4adc0c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.608+0000 7f4ae1c02700 1 -- 172.21.15.77:0/1947543126 >> 172.21.15.77:0/1947543126 conn(0x7f4adc0cb720 msgr2=0x7f4adc1531f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:45.611 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.608+0000 7f4ae1c02700 1 -- 172.21.15.77:0/1947543126 shutdown_connections 2022-04-03T19:44:45.611 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.608+0000 7f4ae1c02700 1 -- 172.21.15.77:0/1947543126 wait complete. 2022-04-03T19:44:45.611 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2022-04-03T19:44:45.632 INFO:tasks.mds_thrash:Ready to start thrashing 2022-04-03T19:44:45.633 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2022-04-03T19:44:45.633 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-04-03T19:44:45.831 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.831+0000 7f5ccc8c9700 1 Processor -- start 2022-04-03T19:44:45.832 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.831+0000 7f5ccc8c9700 1 -- start start 2022-04-03T19:44:45.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.832+0000 7f5ccc8c9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cc80c1170 0x7f5cc81527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:45.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.832+0000 7f5ccc8c9700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5cc8155370 0x7f5cc8069110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:45.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.832+0000 7f5ccc8c9700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5cc80c3b10 0x7f5cc8069650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:45.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.832+0000 7f5ccc8c9700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cc80c3710 con 0x7f5cc80c3b10 2022-04-03T19:44:45.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.832+0000 7f5ccc8c9700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cc8152330 con 0x7f5cc8155370 2022-04-03T19:44:45.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.832+0000 7f5ccc8c9700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc80c1540 con 0x7f5cc80c1170 2022-04-03T19:44:45.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.833+0000 7f5cc77fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cc80c1170 0x7f5cc81527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:45.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.833+0000 7f5cc6ffd700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5cc8155370 0x7f5cc8069110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:45.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.833+0000 7f5cc7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5cc80c3b10 0x7f5cc8069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:45.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.833+0000 7f5cc77fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cc80c1170 0x7f5cc81527c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:59998/0 (socket says 172.21.15.77:59998) 2022-04-03T19:44:45.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.833+0000 7f5cc77fe700 1 -- 172.21.15.77:0/2235799564 learned_addr learned my addr 172.21.15.77:0/2235799564 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:45.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.833+0000 7f5cc77fe700 1 -- 172.21.15.77:0/2235799564 >> v2:172.21.15.186:3300/0 conn(0x7f5cc8155370 msgr2=0x7f5cc8069110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:45.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.833+0000 7f5cc77fe700 1 --2- 172.21.15.77:0/2235799564 >> v2:172.21.15.186:3300/0 conn(0x7f5cc8155370 0x7f5cc8069110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.833+0000 7f5cc77fe700 1 -- 172.21.15.77:0/2235799564 >> v2:172.21.15.77:3300/0 conn(0x7f5cc80c3b10 msgr2=0x7f5cc8069650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:45.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.833+0000 7f5cc77fe700 1 --2- 172.21.15.77:0/2235799564 >> v2:172.21.15.77:3300/0 conn(0x7f5cc80c3b10 0x7f5cc8069650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.833+0000 7f5cc77fe700 1 -- 172.21.15.77:0/2235799564 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc80c4270 con 0x7f5cc80c1170 2022-04-03T19:44:45.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.834+0000 7f5cc77fe700 1 --2- 172.21.15.77:0/2235799564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cc80c1170 0x7f5cc81527c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8065600 tx=0x7f5cb80090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13dc363a6d67f26f server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:45.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.834+0000 7f5cc67fc700 1 -- 172.21.15.77:0/2235799564 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8004030 con 0x7f5cc80c1170 2022-04-03T19:44:45.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.834+0000 7f5cc67fc700 1 -- 172.21.15.77:0/2235799564 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb8004190 con 0x7f5cc80c1170 2022-04-03T19:44:45.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.834+0000 7f5cc67fc700 1 -- 172.21.15.77:0/2235799564 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8011630 con 0x7f5cc80c1170 2022-04-03T19:44:45.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.834+0000 7f5ccc8c9700 1 -- 172.21.15.77:0/2235799564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cc80c1170 msgr2=0x7f5cc81527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:45.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.834+0000 7f5ccc8c9700 1 --2- 172.21.15.77:0/2235799564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cc80c1170 0x7f5cc81527c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8065600 tx=0x7f5cb80090f0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.834+0000 7f5ccc8c9700 1 -- 172.21.15.77:0/2235799564 shutdown_connections 2022-04-03T19:44:45.840 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.834+0000 7f5ccc8c9700 1 --2- 172.21.15.77:0/2235799564 >> v2:172.21.15.77:3300/0 conn(0x7f5cc80c3b10 0x7f5cc8069650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.840 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.834+0000 7f5ccc8c9700 1 --2- 172.21.15.77:0/2235799564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cc80c1170 0x7f5cc81527c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.840 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.834+0000 7f5ccc8c9700 1 --2- 172.21.15.77:0/2235799564 >> v2:172.21.15.186:3300/0 conn(0x7f5cc8155370 0x7f5cc8069110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.840 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.834+0000 7f5ccc8c9700 1 -- 172.21.15.77:0/2235799564 >> 172.21.15.77:0/2235799564 conn(0x7f5cc8066da0 msgr2=0x7f5cc80671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:45.840 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.835+0000 7f5ccc8c9700 1 -- 172.21.15.77:0/2235799564 shutdown_connections 2022-04-03T19:44:45.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.835+0000 7f5ccc8c9700 1 -- 172.21.15.77:0/2235799564 wait complete. 2022-04-03T19:44:45.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.836+0000 7f5ccc8c9700 1 Processor -- start 2022-04-03T19:44:45.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.836+0000 7f5ccc8c9700 1 -- start start 2022-04-03T19:44:45.844 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.837+0000 7f5ccc8c9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc80c1170 0x7f5cc81548a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:45.844 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.837+0000 7f5ccc8c9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cc80c3b10 0x7f5cc8144ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:45.844 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.837+0000 7f5ccc8c9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cc8155370 0x7f5cc80c1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:45.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.837+0000 7f5ccc8c9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc80bd510 con 0x7f5cc80c1170 2022-04-03T19:44:45.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.837+0000 7f5ccc8c9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc80bcf30 con 0x7f5cc8155370 2022-04-03T19:44:45.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.837+0000 7f5ccc8c9700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc8154de0 con 0x7f5cc80c3b10 2022-04-03T19:44:45.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.838+0000 7f5cc77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc80c1170 0x7f5cc81548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:45.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.838+0000 7f5cc6ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cc80c3b10 0x7f5cc8144ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:45.846 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.838+0000 7f5cc7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cc8155370 0x7f5cc80c1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:45.846 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.838+0000 7f5cc6ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cc80c3b10 0x7f5cc8144ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60000/0 (socket says 172.21.15.77:60000) 2022-04-03T19:44:45.846 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.838+0000 7f5cc7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cc8155370 0x7f5cc80c1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37890/0 (socket says 172.21.15.77:37890) 2022-04-03T19:44:45.846 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.838+0000 7f5cc6ffd700 1 -- 172.21.15.77:0/2880334580 learned_addr learned my addr 172.21.15.77:0/2880334580 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:45.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.838+0000 7f5cc7fff700 1 -- 172.21.15.77:0/2880334580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cc80c3b10 msgr2=0x7f5cc8144ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:45.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.838+0000 7f5cc7fff700 1 --2- 172.21.15.77:0/2880334580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cc80c3b10 0x7f5cc8144ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.838+0000 7f5cc7fff700 1 -- 172.21.15.77:0/2880334580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc80c1170 msgr2=0x7f5cc81548a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:45.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.838+0000 7f5cc7fff700 1 --2- 172.21.15.77:0/2880334580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc80c1170 0x7f5cc81548a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:45.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.838+0000 7f5cc7fff700 1 -- 172.21.15.77:0/2880334580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb8009370 con 0x7f5cc8155370 2022-04-03T19:44:45.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.838+0000 7f5cc7fff700 1 --2- 172.21.15.77:0/2880334580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cc8155370 0x7f5cc80c1fd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0009e50 tx=0x7f5cc000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:45.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.839+0000 7f5caf7fe700 1 -- 172.21.15.77:0/2880334580 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc000caa0 con 0x7f5cc8155370 2022-04-03T19:44:45.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.839+0000 7f5caf7fe700 1 -- 172.21.15.77:0/2880334580 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc000f040 con 0x7f5cc8155370 2022-04-03T19:44:45.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.839+0000 7f5caf7fe700 1 -- 172.21.15.77:0/2880334580 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc000ce50 con 0x7f5cc8155370 2022-04-03T19:44:45.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.839+0000 7f5ccc8c9700 1 -- 172.21.15.77:0/2880334580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc80c7b30 con 0x7f5cc8155370 2022-04-03T19:44:45.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.839+0000 7f5ccc8c9700 1 -- 172.21.15.77:0/2880334580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc80bd1a0 con 0x7f5cc8155370 2022-04-03T19:44:45.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.840+0000 7f5ccc8c9700 1 -- 172.21.15.77:0/2880334580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cc80bd6c0 con 0x7f5cc8155370 2022-04-03T19:44:45.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.841+0000 7f5caf7fe700 1 -- 172.21.15.77:0/2880334580 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f5cc0014690 con 0x7f5cc8155370 2022-04-03T19:44:45.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.841+0000 7f5caf7fe700 1 --2- 172.21.15.77:0/2880334580 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5ca8063de0 0x7f5ca8066290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:45.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.842+0000 7f5caf7fe700 1 -- 172.21.15.77:0/2880334580 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f5cc0081ad0 con 0x7f5cc8155370 2022-04-03T19:44:45.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.842+0000 7f5cc77fe700 1 --2- 172.21.15.77:0/2880334580 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5ca8063de0 0x7f5ca8066290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:45.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.842+0000 7f5cc77fe700 1 --2- 172.21.15.77:0/2880334580 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5ca8063de0 0x7f5ca8066290 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8065600 tx=0x7f5cb8016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:45.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:45.845+0000 7f5caf7fe700 1 -- 172.21.15.77:0/2880334580 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f5cc00506c0 con 0x7f5cc8155370 2022-04-03T19:44:46.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.033+0000 7f5ccc8c9700 1 -- 172.21.15.77:0/2880334580 --> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5cc80bd6c0 con 0x7f5ca8063de0 2022-04-03T19:44:46.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.043+0000 7f5caf7fe700 1 -- 172.21.15.77:0/2880334580 <== mgr.4106 v2:172.21.15.186:6832/24948 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284674 (secure 0 0 0) 0x7f5cc80bd6c0 con 0x7f5ca8063de0 2022-04-03T19:44:46.047 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:46.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.047+0000 7f5ccc8c9700 1 -- 172.21.15.77:0/2880334580 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5ca8063de0 msgr2=0x7f5ca8066290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:46.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.047+0000 7f5ccc8c9700 1 --2- 172.21.15.77:0/2880334580 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5ca8063de0 0x7f5ca8066290 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8065600 tx=0x7f5cb8016000 comp rx=0 tx=0).stop 2022-04-03T19:44:46.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.047+0000 7f5ccc8c9700 1 -- 172.21.15.77:0/2880334580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cc8155370 msgr2=0x7f5cc80c1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:46.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.047+0000 7f5ccc8c9700 1 --2- 172.21.15.77:0/2880334580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cc8155370 0x7f5cc80c1fd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0009e50 tx=0x7f5cc000b040 comp rx=0 tx=0).stop 2022-04-03T19:44:46.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.047+0000 7f5ccc8c9700 1 -- 172.21.15.77:0/2880334580 shutdown_connections 2022-04-03T19:44:46.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.047+0000 7f5ccc8c9700 1 --2- 172.21.15.77:0/2880334580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cc8155370 0x7f5cc80c1fd0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.047+0000 7f5ccc8c9700 1 --2- 172.21.15.77:0/2880334580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cc80c1170 0x7f5cc81548a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.047+0000 7f5ccc8c9700 1 --2- 172.21.15.77:0/2880334580 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5ca8063de0 0x7f5ca8066290 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.047+0000 7f5ccc8c9700 1 --2- 172.21.15.77:0/2880334580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cc80c3b10 0x7f5cc8144ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.047+0000 7f5ccc8c9700 1 -- 172.21.15.77:0/2880334580 >> 172.21.15.77:0/2880334580 conn(0x7f5cc8066da0 msgr2=0x7f5cc80c00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:46.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.047+0000 7f5ccc8c9700 1 -- 172.21.15.77:0/2880334580 shutdown_connections 2022-04-03T19:44:46.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.047+0000 7f5ccc8c9700 1 -- 172.21.15.77:0/2880334580 wait complete. 2022-04-03T19:44:46.052 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2022-04-03T19:44:46.072 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2022-04-03T19:44:45.494643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592962,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":205,"num_write_kb":2431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":65808,"kb_used_data":3088,"kb_used_omap":0,"kb_used_meta":62656,"kb_avail":754908912,"statfs":{"total":773094113280,"available":773026725888,"internally_reserved":0,"allocated":3162112,"data_stored":1814836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64159744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":264,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003270"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302418+0000","last_change":"2022-04-03T19:44:21.300867+0000","last_active":"2022-04-03T19:44:21.302418+0000","last_peered":"2022-04-03T19:44:21.302418+0000","last_clean":"2022-04-03T19:44:21.302418+0000","last_became_active":"2022-04-03T19:44:21.300569+0000","last_became_peered":"2022-04-03T19:44:21.300569+0000","last_unstale":"2022-04-03T19:44:21.302418+0000","last_undegraded":"2022-04-03T19:44:21.302418+0000","last_fullsized":"2022-04-03T19:44:21.302418+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:10:32.827314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305911+0000","last_change":"2022-04-03T19:44:21.302410+0000","last_active":"2022-04-03T19:44:21.305911+0000","last_peered":"2022-04-03T19:44:21.305911+0000","last_clean":"2022-04-03T19:44:21.305911+0000","last_became_active":"2022-04-03T19:44:21.302142+0000","last_became_peered":"2022-04-03T19:44:21.302142+0000","last_unstale":"2022-04-03T19:44:21.305911+0000","last_undegraded":"2022-04-03T19:44:21.305911+0000","last_fullsized":"2022-04-03T19:44:21.305911+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:01:46.327996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.298511+0000","last_change":"2022-04-03T19:44:21.293080+0000","last_active":"2022-04-03T19:44:21.298511+0000","last_peered":"2022-04-03T19:44:21.298511+0000","last_clean":"2022-04-03T19:44:21.298511+0000","last_became_active":"2022-04-03T19:44:21.292762+0000","last_became_peered":"2022-04-03T19:44:21.292762+0000","last_unstale":"2022-04-03T19:44:21.298511+0000","last_undegraded":"2022-04-03T19:44:21.298511+0000","last_fullsized":"2022-04-03T19:44:21.298511+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:28:36.726593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300326+0000","last_change":"2022-04-03T19:44:21.294448+0000","last_active":"2022-04-03T19:44:21.300326+0000","last_peered":"2022-04-03T19:44:21.300326+0000","last_clean":"2022-04-03T19:44:21.300326+0000","last_became_active":"2022-04-03T19:44:21.294083+0000","last_became_peered":"2022-04-03T19:44:21.294083+0000","last_unstale":"2022-04-03T19:44:21.300326+0000","last_undegraded":"2022-04-03T19:44:21.300326+0000","last_fullsized":"2022-04-03T19:44:21.300326+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:57:02.374826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.308268+0000","last_change":"2022-04-03T19:44:21.299551+0000","last_active":"2022-04-03T19:44:21.308268+0000","last_peered":"2022-04-03T19:44:21.308268+0000","last_clean":"2022-04-03T19:44:21.308268+0000","last_became_active":"2022-04-03T19:44:21.299123+0000","last_became_peered":"2022-04-03T19:44:21.299123+0000","last_unstale":"2022-04-03T19:44:21.308268+0000","last_undegraded":"2022-04-03T19:44:21.308268+0000","last_fullsized":"2022-04-03T19:44:21.308268+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:08:24.911647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300209+0000","last_change":"2022-04-03T19:44:21.293129+0000","last_active":"2022-04-03T19:44:21.300209+0000","last_peered":"2022-04-03T19:44:21.300209+0000","last_clean":"2022-04-03T19:44:21.300209+0000","last_became_active":"2022-04-03T19:44:21.292843+0000","last_became_peered":"2022-04-03T19:44:21.292843+0000","last_unstale":"2022-04-03T19:44:21.300209+0000","last_undegraded":"2022-04-03T19:44:21.300209+0000","last_fullsized":"2022-04-03T19:44:21.300209+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:38:32.432609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302772+0000","last_change":"2022-04-03T19:44:21.299112+0000","last_active":"2022-04-03T19:44:21.302772+0000","last_peered":"2022-04-03T19:44:21.302772+0000","last_clean":"2022-04-03T19:44:21.302772+0000","last_became_active":"2022-04-03T19:44:21.298803+0000","last_became_peered":"2022-04-03T19:44:21.298803+0000","last_unstale":"2022-04-03T19:44:21.302772+0000","last_undegraded":"2022-04-03T19:44:21.302772+0000","last_fullsized":"2022-04-03T19:44:21.302772+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:29:43.577843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299160+0000","last_change":"2022-04-03T19:44:21.293638+0000","last_active":"2022-04-03T19:44:21.299160+0000","last_peered":"2022-04-03T19:44:21.299160+0000","last_clean":"2022-04-03T19:44:21.299160+0000","last_became_active":"2022-04-03T19:44:21.293359+0000","last_became_peered":"2022-04-03T19:44:21.293359+0000","last_unstale":"2022-04-03T19:44:21.299160+0000","last_undegraded":"2022-04-03T19:44:21.299160+0000","last_fullsized":"2022-04-03T19:44:21.299160+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:28:50.113792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.306107+0000","last_change":"2022-04-03T19:44:21.299916+0000","last_active":"2022-04-03T19:44:21.306107+0000","last_peered":"2022-04-03T19:44:21.306107+0000","last_clean":"2022-04-03T19:44:21.306107+0000","last_became_active":"2022-04-03T19:44:21.299578+0000","last_became_peered":"2022-04-03T19:44:21.299578+0000","last_unstale":"2022-04-03T19:44:21.306107+0000","last_undegraded":"2022-04-03T19:44:21.306107+0000","last_fullsized":"2022-04-03T19:44:21.306107+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:38:40.281496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.298833+0000","last_change":"2022-04-03T19:44:21.294003+0000","last_active":"2022-04-03T19:44:21.298833+0000","last_peered":"2022-04-03T19:44:21.298833+0000","last_clean":"2022-04-03T19:44:21.298833+0000","last_became_active":"2022-04-03T19:44:21.293640+0000","last_became_peered":"2022-04-03T19:44:21.293640+0000","last_unstale":"2022-04-03T19:44:21.298833+0000","last_undegraded":"2022-04-03T19:44:21.298833+0000","last_fullsized":"2022-04-03T19:44:21.298833+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:40:13.436484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302745+0000","last_change":"2022-04-03T19:44:21.301158+0000","last_active":"2022-04-03T19:44:21.302745+0000","last_peered":"2022-04-03T19:44:21.302745+0000","last_clean":"2022-04-03T19:44:21.302745+0000","last_became_active":"2022-04-03T19:44:21.300823+0000","last_became_peered":"2022-04-03T19:44:21.300823+0000","last_unstale":"2022-04-03T19:44:21.302745+0000","last_undegraded":"2022-04-03T19:44:21.302745+0000","last_fullsized":"2022-04-03T19:44:21.302745+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:09:28.365828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299985+0000","last_change":"2022-04-03T19:44:21.295301+0000","last_active":"2022-04-03T19:44:21.299985+0000","last_peered":"2022-04-03T19:44:21.299985+0000","last_clean":"2022-04-03T19:44:21.299985+0000","last_became_active":"2022-04-03T19:44:21.295032+0000","last_became_peered":"2022-04-03T19:44:21.295032+0000","last_unstale":"2022-04-03T19:44:21.299985+0000","last_undegraded":"2022-04-03T19:44:21.299985+0000","last_fullsized":"2022-04-03T19:44:21.299985+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:55:55.637194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.301541+0000","last_change":"2022-04-03T19:44:21.296250+0000","last_active":"2022-04-03T19:44:21.301541+0000","last_peered":"2022-04-03T19:44:21.301541+0000","last_clean":"2022-04-03T19:44:21.301541+0000","last_became_active":"2022-04-03T19:44:21.295967+0000","last_became_peered":"2022-04-03T19:44:21.295967+0000","last_unstale":"2022-04-03T19:44:21.301541+0000","last_undegraded":"2022-04-03T19:44:21.301541+0000","last_fullsized":"2022-04-03T19:44:21.301541+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:43:34.301454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299466+0000","last_change":"2022-04-03T19:44:21.297229+0000","last_active":"2022-04-03T19:44:21.299466+0000","last_peered":"2022-04-03T19:44:21.299466+0000","last_clean":"2022-04-03T19:44:21.299466+0000","last_became_active":"2022-04-03T19:44:21.296993+0000","last_became_peered":"2022-04-03T19:44:21.296993+0000","last_unstale":"2022-04-03T19:44:21.299466+0000","last_undegraded":"2022-04-03T19:44:21.299466+0000","last_fullsized":"2022-04-03T19:44:21.299466+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:27:17.280580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303252+0000","last_change":"2022-04-03T19:44:21.302427+0000","last_active":"2022-04-03T19:44:21.303252+0000","last_peered":"2022-04-03T19:44:21.303252+0000","last_clean":"2022-04-03T19:44:21.303252+0000","last_became_active":"2022-04-03T19:44:21.302157+0000","last_became_peered":"2022-04-03T19:44:21.302157+0000","last_unstale":"2022-04-03T19:44:21.303252+0000","last_undegraded":"2022-04-03T19:44:21.303252+0000","last_fullsized":"2022-04-03T19:44:21.303252+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:57:23.292801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.298518+0000","last_change":"2022-04-03T19:44:21.293096+0000","last_active":"2022-04-03T19:44:21.298518+0000","last_peered":"2022-04-03T19:44:21.298518+0000","last_clean":"2022-04-03T19:44:21.298518+0000","last_became_active":"2022-04-03T19:44:21.292785+0000","last_became_peered":"2022-04-03T19:44:21.292785+0000","last_unstale":"2022-04-03T19:44:21.298518+0000","last_undegraded":"2022-04-03T19:44:21.298518+0000","last_fullsized":"2022-04-03T19:44:21.298518+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:27:06.759308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.306127+0000","last_change":"2022-04-03T19:44:21.300957+0000","last_active":"2022-04-03T19:44:21.306127+0000","last_peered":"2022-04-03T19:44:21.306127+0000","last_clean":"2022-04-03T19:44:21.306127+0000","last_became_active":"2022-04-03T19:44:21.300342+0000","last_became_peered":"2022-04-03T19:44:21.300342+0000","last_unstale":"2022-04-03T19:44:21.306127+0000","last_undegraded":"2022-04-03T19:44:21.306127+0000","last_fullsized":"2022-04-03T19:44:21.306127+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:43:53.194230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302142+0000","last_change":"2022-04-03T19:44:21.295996+0000","last_active":"2022-04-03T19:44:21.302142+0000","last_peered":"2022-04-03T19:44:21.302142+0000","last_clean":"2022-04-03T19:44:21.302142+0000","last_became_active":"2022-04-03T19:44:21.295766+0000","last_became_peered":"2022-04-03T19:44:21.295766+0000","last_unstale":"2022-04-03T19:44:21.302142+0000","last_undegraded":"2022-04-03T19:44:21.302142+0000","last_fullsized":"2022-04-03T19:44:21.302142+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:21:31.628478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302310+0000","last_change":"2022-04-03T19:44:21.299840+0000","last_active":"2022-04-03T19:44:21.302310+0000","last_peered":"2022-04-03T19:44:21.302310+0000","last_clean":"2022-04-03T19:44:21.302310+0000","last_became_active":"2022-04-03T19:44:21.299487+0000","last_became_peered":"2022-04-03T19:44:21.299487+0000","last_unstale":"2022-04-03T19:44:21.302310+0000","last_undegraded":"2022-04-03T19:44:21.302310+0000","last_fullsized":"2022-04-03T19:44:21.302310+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:24:39.583019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.295324+0000","last_change":"2022-04-03T19:44:21.291748+0000","last_active":"2022-04-03T19:44:21.295324+0000","last_peered":"2022-04-03T19:44:21.295324+0000","last_clean":"2022-04-03T19:44:21.295324+0000","last_became_active":"2022-04-03T19:44:21.290939+0000","last_became_peered":"2022-04-03T19:44:21.290939+0000","last_unstale":"2022-04-03T19:44:21.295324+0000","last_undegraded":"2022-04-03T19:44:21.295324+0000","last_fullsized":"2022-04-03T19:44:21.295324+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:28:45.710805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302521+0000","last_change":"2022-04-03T19:44:21.299442+0000","last_active":"2022-04-03T19:44:21.302521+0000","last_peered":"2022-04-03T19:44:21.302521+0000","last_clean":"2022-04-03T19:44:21.302521+0000","last_became_active":"2022-04-03T19:44:21.299166+0000","last_became_peered":"2022-04-03T19:44:21.299166+0000","last_unstale":"2022-04-03T19:44:21.302521+0000","last_undegraded":"2022-04-03T19:44:21.302521+0000","last_fullsized":"2022-04-03T19:44:21.302521+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:31:30.905517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.311290+0000","last_change":"2022-04-03T19:44:21.307007+0000","last_active":"2022-04-03T19:44:21.311290+0000","last_peered":"2022-04-03T19:44:21.311290+0000","last_clean":"2022-04-03T19:44:21.311290+0000","last_became_active":"2022-04-03T19:44:21.306650+0000","last_became_peered":"2022-04-03T19:44:21.306650+0000","last_unstale":"2022-04-03T19:44:21.311290+0000","last_undegraded":"2022-04-03T19:44:21.311290+0000","last_fullsized":"2022-04-03T19:44:21.311290+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:03:26.043624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299849+0000","last_change":"2022-04-03T19:44:21.293566+0000","last_active":"2022-04-03T19:44:21.299849+0000","last_peered":"2022-04-03T19:44:21.299849+0000","last_clean":"2022-04-03T19:44:21.299849+0000","last_became_active":"2022-04-03T19:44:21.293263+0000","last_became_peered":"2022-04-03T19:44:21.293263+0000","last_unstale":"2022-04-03T19:44:21.299849+0000","last_undegraded":"2022-04-03T19:44:21.299849+0000","last_fullsized":"2022-04-03T19:44:21.299849+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:39:52.523944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302610+0000","last_change":"2022-04-03T19:44:21.300329+0000","last_active":"2022-04-03T19:44:21.302610+0000","last_peered":"2022-04-03T19:44:21.302610+0000","last_clean":"2022-04-03T19:44:21.302610+0000","last_became_active":"2022-04-03T19:44:21.300000+0000","last_became_peered":"2022-04-03T19:44:21.300000+0000","last_unstale":"2022-04-03T19:44:21.302610+0000","last_undegraded":"2022-04-03T19:44:21.302610+0000","last_fullsized":"2022-04-03T19:44:21.302610+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:15:10.839571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.312922+0000","last_change":"2022-04-03T19:44:21.302506+0000","last_active":"2022-04-03T19:44:21.312922+0000","last_peered":"2022-04-03T19:44:21.312922+0000","last_clean":"2022-04-03T19:44:21.312922+0000","last_became_active":"2022-04-03T19:44:21.302235+0000","last_became_peered":"2022-04-03T19:44:21.302235+0000","last_unstale":"2022-04-03T19:44:21.312922+0000","last_undegraded":"2022-04-03T19:44:21.312922+0000","last_fullsized":"2022-04-03T19:44:21.312922+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:25:25.801754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313258+0000","last_change":"2022-04-03T19:44:21.306670+0000","last_active":"2022-04-03T19:44:21.313258+0000","last_peered":"2022-04-03T19:44:21.313258+0000","last_clean":"2022-04-03T19:44:21.313258+0000","last_became_active":"2022-04-03T19:44:21.305722+0000","last_became_peered":"2022-04-03T19:44:21.305722+0000","last_unstale":"2022-04-03T19:44:21.313258+0000","last_undegraded":"2022-04-03T19:44:21.313258+0000","last_fullsized":"2022-04-03T19:44:21.313258+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:44:09.515207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303081+0000","last_change":"2022-04-03T19:44:21.300024+0000","last_active":"2022-04-03T19:44:21.303081+0000","last_peered":"2022-04-03T19:44:21.303081+0000","last_clean":"2022-04-03T19:44:21.303081+0000","last_became_active":"2022-04-03T19:44:21.299753+0000","last_became_peered":"2022-04-03T19:44:21.299753+0000","last_unstale":"2022-04-03T19:44:21.303081+0000","last_undegraded":"2022-04-03T19:44:21.303081+0000","last_fullsized":"2022-04-03T19:44:21.303081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:15:31.808623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.390304+0000","last_change":"2022-04-03T19:44:20.783898+0000","last_active":"2022-04-03T19:44:22.390304+0000","last_peered":"2022-04-03T19:44:22.390304+0000","last_clean":"2022-04-03T19:44:22.390304+0000","last_became_active":"2022-04-03T19:44:20.783612+0000","last_became_peered":"2022-04-03T19:44:20.783612+0000","last_unstale":"2022-04-03T19:44:22.390304+0000","last_undegraded":"2022-04-03T19:44:22.390304+0000","last_fullsized":"2022-04-03T19:44:22.390304+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:13:32.863927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.310259+0000","last_change":"2022-04-03T19:44:21.306904+0000","last_active":"2022-04-03T19:44:21.310259+0000","last_peered":"2022-04-03T19:44:21.310259+0000","last_clean":"2022-04-03T19:44:21.310259+0000","last_became_active":"2022-04-03T19:44:21.306616+0000","last_became_peered":"2022-04-03T19:44:21.306616+0000","last_unstale":"2022-04-03T19:44:21.310259+0000","last_undegraded":"2022-04-03T19:44:21.310259+0000","last_fullsized":"2022-04-03T19:44:21.310259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:49:35.783912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.296341+0000","last_change":"2022-04-03T19:44:20.788125+0000","last_active":"2022-04-03T19:44:21.296341+0000","last_peered":"2022-04-03T19:44:21.296341+0000","last_clean":"2022-04-03T19:44:21.296341+0000","last_became_active":"2022-04-03T19:44:20.787792+0000","last_became_peered":"2022-04-03T19:44:20.787792+0000","last_unstale":"2022-04-03T19:44:21.296341+0000","last_undegraded":"2022-04-03T19:44:21.296341+0000","last_fullsized":"2022-04-03T19:44:21.296341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:11:51.126774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305572+0000","last_change":"2022-04-03T19:44:21.302904+0000","last_active":"2022-04-03T19:44:21.305572+0000","last_peered":"2022-04-03T19:44:21.305572+0000","last_clean":"2022-04-03T19:44:21.305572+0000","last_became_active":"2022-04-03T19:44:21.301564+0000","last_became_peered":"2022-04-03T19:44:21.301564+0000","last_unstale":"2022-04-03T19:44:21.305572+0000","last_undegraded":"2022-04-03T19:44:21.305572+0000","last_fullsized":"2022-04-03T19:44:21.305572+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:25:51.611789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.283649+0000","last_change":"2022-04-03T19:44:20.792529+0000","last_active":"2022-04-03T19:44:21.283649+0000","last_peered":"2022-04-03T19:44:21.283649+0000","last_clean":"2022-04-03T19:44:21.283649+0000","last_became_active":"2022-04-03T19:44:20.792014+0000","last_became_peered":"2022-04-03T19:44:20.792014+0000","last_unstale":"2022-04-03T19:44:21.283649+0000","last_undegraded":"2022-04-03T19:44:21.283649+0000","last_fullsized":"2022-04-03T19:44:21.283649+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:01:38.319693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302637+0000","last_change":"2022-04-03T19:44:21.299302+0000","last_active":"2022-04-03T19:44:21.302637+0000","last_peered":"2022-04-03T19:44:21.302637+0000","last_clean":"2022-04-03T19:44:21.302637+0000","last_became_active":"2022-04-03T19:44:21.299007+0000","last_became_peered":"2022-04-03T19:44:21.299007+0000","last_unstale":"2022-04-03T19:44:21.302637+0000","last_undegraded":"2022-04-03T19:44:21.302637+0000","last_fullsized":"2022-04-03T19:44:21.302637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:59:48.767335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.282232+0000","last_change":"2022-04-03T19:44:20.790672+0000","last_active":"2022-04-03T19:44:21.282232+0000","last_peered":"2022-04-03T19:44:21.282232+0000","last_clean":"2022-04-03T19:44:21.282232+0000","last_became_active":"2022-04-03T19:44:20.788526+0000","last_became_peered":"2022-04-03T19:44:20.788526+0000","last_unstale":"2022-04-03T19:44:21.282232+0000","last_undegraded":"2022-04-03T19:44:21.282232+0000","last_fullsized":"2022-04-03T19:44:21.282232+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:14:50.058917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.309960+0000","last_change":"2022-04-03T19:44:21.306472+0000","last_active":"2022-04-03T19:44:21.309960+0000","last_peered":"2022-04-03T19:44:21.309960+0000","last_clean":"2022-04-03T19:44:21.309960+0000","last_became_active":"2022-04-03T19:44:21.306071+0000","last_became_peered":"2022-04-03T19:44:21.306071+0000","last_unstale":"2022-04-03T19:44:21.309960+0000","last_undegraded":"2022-04-03T19:44:21.309960+0000","last_fullsized":"2022-04-03T19:44:21.309960+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:43:22.221014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286837+0000","last_change":"2022-04-03T19:44:20.785638+0000","last_active":"2022-04-03T19:44:21.286837+0000","last_peered":"2022-04-03T19:44:21.286837+0000","last_clean":"2022-04-03T19:44:21.286837+0000","last_became_active":"2022-04-03T19:44:20.784963+0000","last_became_peered":"2022-04-03T19:44:20.784963+0000","last_unstale":"2022-04-03T19:44:21.286837+0000","last_undegraded":"2022-04-03T19:44:21.286837+0000","last_fullsized":"2022-04-03T19:44:21.286837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:39:26.933298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.309328+0000","last_change":"2022-04-03T19:44:21.299583+0000","last_active":"2022-04-03T19:44:21.309328+0000","last_peered":"2022-04-03T19:44:21.309328+0000","last_clean":"2022-04-03T19:44:21.309328+0000","last_became_active":"2022-04-03T19:44:21.299301+0000","last_became_peered":"2022-04-03T19:44:21.299301+0000","last_unstale":"2022-04-03T19:44:21.309328+0000","last_undegraded":"2022-04-03T19:44:21.309328+0000","last_fullsized":"2022-04-03T19:44:21.309328+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:37:49.567741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"24'3","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.290849+0000","last_change":"2022-04-03T19:44:20.795890+0000","last_active":"2022-04-03T19:44:26.290849+0000","last_peered":"2022-04-03T19:44:26.290849+0000","last_clean":"2022-04-03T19:44:26.290849+0000","last_became_active":"2022-04-03T19:44:20.795611+0000","last_became_peered":"2022-04-03T19:44:20.795611+0000","last_unstale":"2022-04-03T19:44:26.290849+0000","last_undegraded":"2022-04-03T19:44:26.290849+0000","last_fullsized":"2022-04-03T19:44:26.290849+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:08:33.216934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313173+0000","last_change":"2022-04-03T19:44:21.304724+0000","last_active":"2022-04-03T19:44:21.313173+0000","last_peered":"2022-04-03T19:44:21.313173+0000","last_clean":"2022-04-03T19:44:21.313173+0000","last_became_active":"2022-04-03T19:44:21.302708+0000","last_became_peered":"2022-04-03T19:44:21.302708+0000","last_unstale":"2022-04-03T19:44:21.313173+0000","last_undegraded":"2022-04-03T19:44:21.313173+0000","last_fullsized":"2022-04-03T19:44:21.313173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:04:54.970100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"24'7","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.294405+0000","last_change":"2022-04-03T19:44:20.785659+0000","last_active":"2022-04-03T19:44:26.294405+0000","last_peered":"2022-04-03T19:44:26.294405+0000","last_clean":"2022-04-03T19:44:26.294405+0000","last_became_active":"2022-04-03T19:44:20.785308+0000","last_became_peered":"2022-04-03T19:44:20.785308+0000","last_unstale":"2022-04-03T19:44:26.294405+0000","last_undegraded":"2022-04-03T19:44:26.294405+0000","last_fullsized":"2022-04-03T19:44:26.294405+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:52:08.268532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1812,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293514+0000","last_change":"2022-04-03T19:44:21.291033+0000","last_active":"2022-04-03T19:44:21.293514+0000","last_peered":"2022-04-03T19:44:21.293514+0000","last_clean":"2022-04-03T19:44:21.293514+0000","last_became_active":"2022-04-03T19:44:21.290713+0000","last_became_peered":"2022-04-03T19:44:21.290713+0000","last_unstale":"2022-04-03T19:44:21.293514+0000","last_undegraded":"2022-04-03T19:44:21.293514+0000","last_fullsized":"2022-04-03T19:44:21.293514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:19:41.873591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.290200+0000","last_change":"2022-04-03T19:44:20.787511+0000","last_active":"2022-04-03T19:44:26.290200+0000","last_peered":"2022-04-03T19:44:26.290200+0000","last_clean":"2022-04-03T19:44:26.290200+0000","last_became_active":"2022-04-03T19:44:20.787189+0000","last_became_peered":"2022-04-03T19:44:20.787189+0000","last_unstale":"2022-04-03T19:44:26.290200+0000","last_undegraded":"2022-04-03T19:44:26.290200+0000","last_fullsized":"2022-04-03T19:44:26.290200+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:11:56.297263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313518+0000","last_change":"2022-04-03T19:44:21.307846+0000","last_active":"2022-04-03T19:44:21.313518+0000","last_peered":"2022-04-03T19:44:21.313518+0000","last_clean":"2022-04-03T19:44:21.313518+0000","last_became_active":"2022-04-03T19:44:21.307513+0000","last_became_peered":"2022-04-03T19:44:21.307513+0000","last_unstale":"2022-04-03T19:44:21.313518+0000","last_undegraded":"2022-04-03T19:44:21.313518+0000","last_fullsized":"2022-04-03T19:44:21.313518+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:58:09.091062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.281910+0000","last_change":"2022-04-03T19:44:20.790212+0000","last_active":"2022-04-03T19:44:21.281910+0000","last_peered":"2022-04-03T19:44:21.281910+0000","last_clean":"2022-04-03T19:44:21.281910+0000","last_became_active":"2022-04-03T19:44:20.789863+0000","last_became_peered":"2022-04-03T19:44:20.789863+0000","last_unstale":"2022-04-03T19:44:21.281910+0000","last_undegraded":"2022-04-03T19:44:21.281910+0000","last_fullsized":"2022-04-03T19:44:21.281910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:48:13.489309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303061+0000","last_change":"2022-04-03T19:44:21.299953+0000","last_active":"2022-04-03T19:44:21.303061+0000","last_peered":"2022-04-03T19:44:21.303061+0000","last_clean":"2022-04-03T19:44:21.303061+0000","last_became_active":"2022-04-03T19:44:21.299667+0000","last_became_peered":"2022-04-03T19:44:21.299667+0000","last_unstale":"2022-04-03T19:44:21.303061+0000","last_undegraded":"2022-04-03T19:44:21.303061+0000","last_fullsized":"2022-04-03T19:44:21.303061+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:51:16.125160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286253+0000","last_change":"2022-04-03T19:44:20.791054+0000","last_active":"2022-04-03T19:44:21.286253+0000","last_peered":"2022-04-03T19:44:21.286253+0000","last_clean":"2022-04-03T19:44:21.286253+0000","last_became_active":"2022-04-03T19:44:20.790664+0000","last_became_peered":"2022-04-03T19:44:20.790664+0000","last_unstale":"2022-04-03T19:44:21.286253+0000","last_undegraded":"2022-04-03T19:44:21.286253+0000","last_fullsized":"2022-04-03T19:44:21.286253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:07:27.501728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.304407+0000","last_change":"2022-04-03T19:44:21.295028+0000","last_active":"2022-04-03T19:44:21.304407+0000","last_peered":"2022-04-03T19:44:21.304407+0000","last_clean":"2022-04-03T19:44:21.304407+0000","last_became_active":"2022-04-03T19:44:21.294723+0000","last_became_peered":"2022-04-03T19:44:21.294723+0000","last_unstale":"2022-04-03T19:44:21.304407+0000","last_undegraded":"2022-04-03T19:44:21.304407+0000","last_fullsized":"2022-04-03T19:44:21.304407+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:32:45.455842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"24'8","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.294387+0000","last_change":"2022-04-03T19:44:20.792133+0000","last_active":"2022-04-03T19:44:26.294387+0000","last_peered":"2022-04-03T19:44:26.294387+0000","last_clean":"2022-04-03T19:44:26.294387+0000","last_became_active":"2022-04-03T19:44:20.791829+0000","last_became_peered":"2022-04-03T19:44:20.791829+0000","last_unstale":"2022-04-03T19:44:26.294387+0000","last_undegraded":"2022-04-03T19:44:26.294387+0000","last_fullsized":"2022-04-03T19:44:26.294387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:01:21.966734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.310014+0000","last_change":"2022-04-03T19:44:21.300267+0000","last_active":"2022-04-03T19:44:21.310014+0000","last_peered":"2022-04-03T19:44:21.310014+0000","last_clean":"2022-04-03T19:44:21.310014+0000","last_became_active":"2022-04-03T19:44:21.299948+0000","last_became_peered":"2022-04-03T19:44:21.299948+0000","last_unstale":"2022-04-03T19:44:21.310014+0000","last_undegraded":"2022-04-03T19:44:21.310014+0000","last_fullsized":"2022-04-03T19:44:21.310014+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:06:54.481379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.391229+0000","last_change":"2022-04-03T19:44:20.789793+0000","last_active":"2022-04-03T19:44:22.391229+0000","last_peered":"2022-04-03T19:44:22.391229+0000","last_clean":"2022-04-03T19:44:22.391229+0000","last_became_active":"2022-04-03T19:44:20.789371+0000","last_became_peered":"2022-04-03T19:44:20.789371+0000","last_unstale":"2022-04-03T19:44:22.391229+0000","last_undegraded":"2022-04-03T19:44:22.391229+0000","last_fullsized":"2022-04-03T19:44:22.391229+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:15:52.060059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.301800+0000","last_change":"2022-04-03T19:44:21.298477+0000","last_active":"2022-04-03T19:44:21.301800+0000","last_peered":"2022-04-03T19:44:21.301800+0000","last_clean":"2022-04-03T19:44:21.301800+0000","last_became_active":"2022-04-03T19:44:21.298165+0000","last_became_peered":"2022-04-03T19:44:21.298165+0000","last_unstale":"2022-04-03T19:44:21.301800+0000","last_undegraded":"2022-04-03T19:44:21.301800+0000","last_fullsized":"2022-04-03T19:44:21.301800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:23:11.042589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.290125+0000","last_change":"2022-04-03T19:44:20.787949+0000","last_active":"2022-04-03T19:44:26.290125+0000","last_peered":"2022-04-03T19:44:26.290125+0000","last_clean":"2022-04-03T19:44:26.290125+0000","last_became_active":"2022-04-03T19:44:20.787592+0000","last_became_peered":"2022-04-03T19:44:20.787592+0000","last_unstale":"2022-04-03T19:44:26.290125+0000","last_undegraded":"2022-04-03T19:44:26.290125+0000","last_fullsized":"2022-04-03T19:44:26.290125+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:50:09.411246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302765+0000","last_change":"2022-04-03T19:44:21.301200+0000","last_active":"2022-04-03T19:44:21.302765+0000","last_peered":"2022-04-03T19:44:21.302765+0000","last_clean":"2022-04-03T19:44:21.302765+0000","last_became_active":"2022-04-03T19:44:21.300878+0000","last_became_peered":"2022-04-03T19:44:21.300878+0000","last_unstale":"2022-04-03T19:44:21.302765+0000","last_undegraded":"2022-04-03T19:44:21.302765+0000","last_fullsized":"2022-04-03T19:44:21.302765+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:54:36.089411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.391128+0000","last_change":"2022-04-03T19:44:20.783711+0000","last_active":"2022-04-03T19:44:22.391128+0000","last_peered":"2022-04-03T19:44:22.391128+0000","last_clean":"2022-04-03T19:44:22.391128+0000","last_became_active":"2022-04-03T19:44:20.783423+0000","last_became_peered":"2022-04-03T19:44:20.783423+0000","last_unstale":"2022-04-03T19:44:22.391128+0000","last_undegraded":"2022-04-03T19:44:22.391128+0000","last_fullsized":"2022-04-03T19:44:22.391128+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:22:37.665229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303123+0000","last_change":"2022-04-03T19:44:21.300815+0000","last_active":"2022-04-03T19:44:21.303123+0000","last_peered":"2022-04-03T19:44:21.303123+0000","last_clean":"2022-04-03T19:44:21.303123+0000","last_became_active":"2022-04-03T19:44:21.300545+0000","last_became_peered":"2022-04-03T19:44:21.300545+0000","last_unstale":"2022-04-03T19:44:21.303123+0000","last_undegraded":"2022-04-03T19:44:21.303123+0000","last_fullsized":"2022-04-03T19:44:21.303123+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:33:13.941191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.285374+0000","last_change":"2022-04-03T19:44:20.785480+0000","last_active":"2022-04-03T19:44:21.285374+0000","last_peered":"2022-04-03T19:44:21.285374+0000","last_clean":"2022-04-03T19:44:21.285374+0000","last_became_active":"2022-04-03T19:44:20.785178+0000","last_became_peered":"2022-04-03T19:44:20.785178+0000","last_unstale":"2022-04-03T19:44:21.285374+0000","last_undegraded":"2022-04-03T19:44:21.285374+0000","last_fullsized":"2022-04-03T19:44:21.285374+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:15:37.181452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.308247+0000","last_change":"2022-04-03T19:44:21.296509+0000","last_active":"2022-04-03T19:44:21.308247+0000","last_peered":"2022-04-03T19:44:21.308247+0000","last_clean":"2022-04-03T19:44:21.308247+0000","last_became_active":"2022-04-03T19:44:21.296240+0000","last_became_peered":"2022-04-03T19:44:21.296240+0000","last_unstale":"2022-04-03T19:44:21.308247+0000","last_undegraded":"2022-04-03T19:44:21.308247+0000","last_fullsized":"2022-04-03T19:44:21.308247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:50:45.955195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.288396+0000","last_change":"2022-04-03T19:44:20.792550+0000","last_active":"2022-04-03T19:44:26.288396+0000","last_peered":"2022-04-03T19:44:26.288396+0000","last_clean":"2022-04-03T19:44:26.288396+0000","last_became_active":"2022-04-03T19:44:20.792191+0000","last_became_peered":"2022-04-03T19:44:20.792191+0000","last_unstale":"2022-04-03T19:44:26.288396+0000","last_undegraded":"2022-04-03T19:44:26.288396+0000","last_fullsized":"2022-04-03T19:44:26.288396+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:51:30.297728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299612+0000","last_change":"2022-04-03T19:44:21.294898+0000","last_active":"2022-04-03T19:44:21.299612+0000","last_peered":"2022-04-03T19:44:21.299612+0000","last_clean":"2022-04-03T19:44:21.299612+0000","last_became_active":"2022-04-03T19:44:21.294635+0000","last_became_peered":"2022-04-03T19:44:21.294635+0000","last_unstale":"2022-04-03T19:44:21.299612+0000","last_undegraded":"2022-04-03T19:44:21.299612+0000","last_fullsized":"2022-04-03T19:44:21.299612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:42:52.629552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.289106+0000","last_change":"2022-04-03T19:44:20.783711+0000","last_active":"2022-04-03T19:44:21.289106+0000","last_peered":"2022-04-03T19:44:21.289106+0000","last_clean":"2022-04-03T19:44:21.289106+0000","last_became_active":"2022-04-03T19:44:20.783426+0000","last_became_peered":"2022-04-03T19:44:20.783426+0000","last_unstale":"2022-04-03T19:44:21.289106+0000","last_undegraded":"2022-04-03T19:44:21.289106+0000","last_fullsized":"2022-04-03T19:44:21.289106+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:25:26.219558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302205+0000","last_change":"2022-04-03T19:44:21.296081+0000","last_active":"2022-04-03T19:44:21.302205+0000","last_peered":"2022-04-03T19:44:21.302205+0000","last_clean":"2022-04-03T19:44:21.302205+0000","last_became_active":"2022-04-03T19:44:21.295784+0000","last_became_peered":"2022-04-03T19:44:21.295784+0000","last_unstale":"2022-04-03T19:44:21.302205+0000","last_undegraded":"2022-04-03T19:44:21.302205+0000","last_fullsized":"2022-04-03T19:44:21.302205+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:23:40.568168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.283775+0000","last_change":"2022-04-03T19:44:20.789868+0000","last_active":"2022-04-03T19:44:21.283775+0000","last_peered":"2022-04-03T19:44:21.283775+0000","last_clean":"2022-04-03T19:44:21.283775+0000","last_became_active":"2022-04-03T19:44:20.789563+0000","last_became_peered":"2022-04-03T19:44:20.789563+0000","last_unstale":"2022-04-03T19:44:21.283775+0000","last_undegraded":"2022-04-03T19:44:21.283775+0000","last_fullsized":"2022-04-03T19:44:21.283775+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:04:22.814932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293542+0000","last_change":"2022-04-03T19:44:21.291212+0000","last_active":"2022-04-03T19:44:21.293542+0000","last_peered":"2022-04-03T19:44:21.293542+0000","last_clean":"2022-04-03T19:44:21.293542+0000","last_became_active":"2022-04-03T19:44:21.290888+0000","last_became_peered":"2022-04-03T19:44:21.290888+0000","last_unstale":"2022-04-03T19:44:21.293542+0000","last_undegraded":"2022-04-03T19:44:21.293542+0000","last_fullsized":"2022-04-03T19:44:21.293542+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:29:34.850679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.281344+0000","last_change":"2022-04-03T19:44:20.789319+0000","last_active":"2022-04-03T19:44:21.281344+0000","last_peered":"2022-04-03T19:44:21.281344+0000","last_clean":"2022-04-03T19:44:21.281344+0000","last_became_active":"2022-04-03T19:44:20.788840+0000","last_became_peered":"2022-04-03T19:44:20.788840+0000","last_unstale":"2022-04-03T19:44:21.281344+0000","last_undegraded":"2022-04-03T19:44:21.281344+0000","last_fullsized":"2022-04-03T19:44:21.281344+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:50:21.536926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313655+0000","last_change":"2022-04-03T19:44:21.311886+0000","last_active":"2022-04-03T19:44:21.313655+0000","last_peered":"2022-04-03T19:44:21.313655+0000","last_clean":"2022-04-03T19:44:21.313655+0000","last_became_active":"2022-04-03T19:44:21.311521+0000","last_became_peered":"2022-04-03T19:44:21.311521+0000","last_unstale":"2022-04-03T19:44:21.313655+0000","last_undegraded":"2022-04-03T19:44:21.313655+0000","last_fullsized":"2022-04-03T19:44:21.313655+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:25:47.761166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.297426+0000","last_change":"2022-04-03T19:44:14.754674+0000","last_active":"2022-04-03T19:44:21.297426+0000","last_peered":"2022-04-03T19:44:21.297426+0000","last_clean":"2022-04-03T19:44:21.297426+0000","last_became_active":"2022-04-03T19:44:14.754210+0000","last_became_peered":"2022-04-03T19:44:14.754210+0000","last_unstale":"2022-04-03T19:44:21.297426+0000","last_undegraded":"2022-04-03T19:44:21.297426+0000","last_fullsized":"2022-04-03T19:44:21.297426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:23:14.624494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286308+0000","last_change":"2022-04-03T19:44:20.785400+0000","last_active":"2022-04-03T19:44:21.286308+0000","last_peered":"2022-04-03T19:44:21.286308+0000","last_clean":"2022-04-03T19:44:21.286308+0000","last_became_active":"2022-04-03T19:44:20.785125+0000","last_became_peered":"2022-04-03T19:44:20.785125+0000","last_unstale":"2022-04-03T19:44:21.286308+0000","last_undegraded":"2022-04-03T19:44:21.286308+0000","last_fullsized":"2022-04-03T19:44:21.286308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:24:02.690646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313197+0000","last_change":"2022-04-03T19:44:21.305321+0000","last_active":"2022-04-03T19:44:21.313197+0000","last_peered":"2022-04-03T19:44:21.313197+0000","last_clean":"2022-04-03T19:44:21.313197+0000","last_became_active":"2022-04-03T19:44:21.304017+0000","last_became_peered":"2022-04-03T19:44:21.304017+0000","last_unstale":"2022-04-03T19:44:21.313197+0000","last_undegraded":"2022-04-03T19:44:21.313197+0000","last_fullsized":"2022-04-03T19:44:21.313197+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:59:12.619058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.285445+0000","last_change":"2022-04-03T19:44:14.755580+0000","last_active":"2022-04-03T19:44:21.285445+0000","last_peered":"2022-04-03T19:44:21.285445+0000","last_clean":"2022-04-03T19:44:21.285445+0000","last_became_active":"2022-04-03T19:44:14.755222+0000","last_became_peered":"2022-04-03T19:44:14.755222+0000","last_unstale":"2022-04-03T19:44:21.285445+0000","last_undegraded":"2022-04-03T19:44:21.285445+0000","last_fullsized":"2022-04-03T19:44:21.285445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:08:18.314326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.283116+0000","last_change":"2022-04-03T19:44:20.791540+0000","last_active":"2022-04-03T19:44:21.283116+0000","last_peered":"2022-04-03T19:44:21.283116+0000","last_clean":"2022-04-03T19:44:21.283116+0000","last_became_active":"2022-04-03T19:44:20.791192+0000","last_became_peered":"2022-04-03T19:44:20.791192+0000","last_unstale":"2022-04-03T19:44:21.283116+0000","last_undegraded":"2022-04-03T19:44:21.283116+0000","last_fullsized":"2022-04-03T19:44:21.283116+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:58:59.674606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313218+0000","last_change":"2022-04-03T19:44:21.305930+0000","last_active":"2022-04-03T19:44:21.313218+0000","last_peered":"2022-04-03T19:44:21.313218+0000","last_clean":"2022-04-03T19:44:21.313218+0000","last_became_active":"2022-04-03T19:44:21.305659+0000","last_became_peered":"2022-04-03T19:44:21.305659+0000","last_unstale":"2022-04-03T19:44:21.313218+0000","last_undegraded":"2022-04-03T19:44:21.313218+0000","last_fullsized":"2022-04-03T19:44:21.313218+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:14:34.958569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.290570+0000","last_change":"2022-04-03T19:44:14.754598+0000","last_active":"2022-04-03T19:44:21.290570+0000","last_peered":"2022-04-03T19:44:21.290570+0000","last_clean":"2022-04-03T19:44:21.290570+0000","last_became_active":"2022-04-03T19:44:14.754287+0000","last_became_peered":"2022-04-03T19:44:14.754287+0000","last_unstale":"2022-04-03T19:44:21.290570+0000","last_undegraded":"2022-04-03T19:44:21.290570+0000","last_fullsized":"2022-04-03T19:44:21.290570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:32:16.232689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.296054+0000","last_change":"2022-04-03T19:44:20.789567+0000","last_active":"2022-04-03T19:44:21.296054+0000","last_peered":"2022-04-03T19:44:21.296054+0000","last_clean":"2022-04-03T19:44:21.296054+0000","last_became_active":"2022-04-03T19:44:20.788664+0000","last_became_peered":"2022-04-03T19:44:20.788664+0000","last_unstale":"2022-04-03T19:44:21.296054+0000","last_undegraded":"2022-04-03T19:44:21.296054+0000","last_fullsized":"2022-04-03T19:44:21.296054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:15:37.098494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.312944+0000","last_change":"2022-04-03T19:44:21.303079+0000","last_active":"2022-04-03T19:44:21.312944+0000","last_peered":"2022-04-03T19:44:21.312944+0000","last_clean":"2022-04-03T19:44:21.312944+0000","last_became_active":"2022-04-03T19:44:21.302014+0000","last_became_peered":"2022-04-03T19:44:21.302014+0000","last_unstale":"2022-04-03T19:44:21.312944+0000","last_undegraded":"2022-04-03T19:44:21.312944+0000","last_fullsized":"2022-04-03T19:44:21.312944+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:15:58.860701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.285641+0000","last_change":"2022-04-03T19:44:14.754385+0000","last_active":"2022-04-03T19:44:21.285641+0000","last_peered":"2022-04-03T19:44:21.285641+0000","last_clean":"2022-04-03T19:44:21.285641+0000","last_became_active":"2022-04-03T19:44:14.754003+0000","last_became_peered":"2022-04-03T19:44:14.754003+0000","last_unstale":"2022-04-03T19:44:21.285641+0000","last_undegraded":"2022-04-03T19:44:21.285641+0000","last_fullsized":"2022-04-03T19:44:21.285641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:05:22.716933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":166,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286290+0000","last_change":"2022-04-03T19:44:08.726758+0000","last_active":"2022-04-03T19:44:21.286290+0000","last_peered":"2022-04-03T19:44:21.286290+0000","last_clean":"2022-04-03T19:44:21.286290+0000","last_became_active":"2022-04-03T19:44:08.725694+0000","last_became_peered":"2022-04-03T19:44:08.725694+0000","last_unstale":"2022-04-03T19:44:21.286290+0000","last_undegraded":"2022-04-03T19:44:21.286290+0000","last_fullsized":"2022-04-03T19:44:21.286290+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:07.708129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:07.708129+0000","last_clean_scrub_stamp":"2022-04-03T19:44:07.708129+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:27:43.354267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.282879+0000","last_change":"2022-04-03T19:44:20.781585+0000","last_active":"2022-04-03T19:44:21.282879+0000","last_peered":"2022-04-03T19:44:21.282879+0000","last_clean":"2022-04-03T19:44:21.282879+0000","last_became_active":"2022-04-03T19:44:20.781274+0000","last_became_peered":"2022-04-03T19:44:20.781274+0000","last_unstale":"2022-04-03T19:44:21.282879+0000","last_undegraded":"2022-04-03T19:44:21.282879+0000","last_fullsized":"2022-04-03T19:44:21.282879+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:49:54.368001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.301298+0000","last_change":"2022-04-03T19:44:21.296620+0000","last_active":"2022-04-03T19:44:21.301298+0000","last_peered":"2022-04-03T19:44:21.301298+0000","last_clean":"2022-04-03T19:44:21.301298+0000","last_became_active":"2022-04-03T19:44:21.296314+0000","last_became_peered":"2022-04-03T19:44:21.296314+0000","last_unstale":"2022-04-03T19:44:21.301298+0000","last_undegraded":"2022-04-03T19:44:21.301298+0000","last_fullsized":"2022-04-03T19:44:21.301298+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:09:18.338683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286626+0000","last_change":"2022-04-03T19:44:14.754664+0000","last_active":"2022-04-03T19:44:21.286626+0000","last_peered":"2022-04-03T19:44:21.286626+0000","last_clean":"2022-04-03T19:44:21.286626+0000","last_became_active":"2022-04-03T19:44:14.754328+0000","last_became_peered":"2022-04-03T19:44:14.754328+0000","last_unstale":"2022-04-03T19:44:21.286626+0000","last_undegraded":"2022-04-03T19:44:21.286626+0000","last_fullsized":"2022-04-03T19:44:21.286626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:39:58.022353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.281559+0000","last_change":"2022-04-03T19:44:20.787271+0000","last_active":"2022-04-03T19:44:21.281559+0000","last_peered":"2022-04-03T19:44:21.281559+0000","last_clean":"2022-04-03T19:44:21.281559+0000","last_became_active":"2022-04-03T19:44:20.786957+0000","last_became_peered":"2022-04-03T19:44:20.786957+0000","last_unstale":"2022-04-03T19:44:21.281559+0000","last_undegraded":"2022-04-03T19:44:21.281559+0000","last_fullsized":"2022-04-03T19:44:21.281559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:21:45.992744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.311227+0000","last_change":"2022-04-03T19:44:21.306857+0000","last_active":"2022-04-03T19:44:21.311227+0000","last_peered":"2022-04-03T19:44:21.311227+0000","last_clean":"2022-04-03T19:44:21.311227+0000","last_became_active":"2022-04-03T19:44:21.306521+0000","last_became_peered":"2022-04-03T19:44:21.306521+0000","last_unstale":"2022-04-03T19:44:21.311227+0000","last_undegraded":"2022-04-03T19:44:21.311227+0000","last_fullsized":"2022-04-03T19:44:21.311227+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:50:03.912834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.287006+0000","last_change":"2022-04-03T19:44:14.754657+0000","last_active":"2022-04-03T19:44:21.287006+0000","last_peered":"2022-04-03T19:44:21.287006+0000","last_clean":"2022-04-03T19:44:21.287006+0000","last_became_active":"2022-04-03T19:44:14.754310+0000","last_became_peered":"2022-04-03T19:44:14.754310+0000","last_unstale":"2022-04-03T19:44:21.287006+0000","last_undegraded":"2022-04-03T19:44:21.287006+0000","last_fullsized":"2022-04-03T19:44:21.287006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:42:05.279424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.297553+0000","last_change":"2022-04-03T19:44:20.794454+0000","last_active":"2022-04-03T19:44:21.297553+0000","last_peered":"2022-04-03T19:44:21.297553+0000","last_clean":"2022-04-03T19:44:21.297553+0000","last_became_active":"2022-04-03T19:44:20.789468+0000","last_became_peered":"2022-04-03T19:44:20.789468+0000","last_unstale":"2022-04-03T19:44:21.297553+0000","last_undegraded":"2022-04-03T19:44:21.297553+0000","last_fullsized":"2022-04-03T19:44:21.297553+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:02:49.184838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.308286+0000","last_change":"2022-04-03T19:44:21.303669+0000","last_active":"2022-04-03T19:44:21.308286+0000","last_peered":"2022-04-03T19:44:21.308286+0000","last_clean":"2022-04-03T19:44:21.308286+0000","last_became_active":"2022-04-03T19:44:21.303381+0000","last_became_peered":"2022-04-03T19:44:21.303381+0000","last_unstale":"2022-04-03T19:44:21.308286+0000","last_undegraded":"2022-04-03T19:44:21.308286+0000","last_fullsized":"2022-04-03T19:44:21.308286+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:43:22.320070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.289548+0000","last_change":"2022-04-03T19:44:20.791594+0000","last_active":"2022-04-03T19:44:26.289548+0000","last_peered":"2022-04-03T19:44:26.289548+0000","last_clean":"2022-04-03T19:44:26.289548+0000","last_became_active":"2022-04-03T19:44:20.789693+0000","last_became_peered":"2022-04-03T19:44:20.789693+0000","last_unstale":"2022-04-03T19:44:26.289548+0000","last_undegraded":"2022-04-03T19:44:26.289548+0000","last_fullsized":"2022-04-03T19:44:26.289548+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:18:21.832066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.297820+0000","last_change":"2022-04-03T19:44:14.755096+0000","last_active":"2022-04-03T19:44:21.297820+0000","last_peered":"2022-04-03T19:44:21.297820+0000","last_clean":"2022-04-03T19:44:21.297820+0000","last_became_active":"2022-04-03T19:44:14.754799+0000","last_became_peered":"2022-04-03T19:44:14.754799+0000","last_unstale":"2022-04-03T19:44:21.297820+0000","last_undegraded":"2022-04-03T19:44:21.297820+0000","last_fullsized":"2022-04-03T19:44:21.297820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:04:17.179114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.309672+0000","last_change":"2022-04-03T19:44:21.299649+0000","last_active":"2022-04-03T19:44:21.309672+0000","last_peered":"2022-04-03T19:44:21.309672+0000","last_clean":"2022-04-03T19:44:21.309672+0000","last_became_active":"2022-04-03T19:44:21.299399+0000","last_became_peered":"2022-04-03T19:44:21.299399+0000","last_unstale":"2022-04-03T19:44:21.309672+0000","last_undegraded":"2022-04-03T19:44:21.309672+0000","last_fullsized":"2022-04-03T19:44:21.309672+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:13:06.313729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.288717+0000","last_change":"2022-04-03T19:44:14.754659+0000","last_active":"2022-04-03T19:44:21.288717+0000","last_peered":"2022-04-03T19:44:21.288717+0000","last_clean":"2022-04-03T19:44:21.288717+0000","last_became_active":"2022-04-03T19:44:14.754386+0000","last_became_peered":"2022-04-03T19:44:14.754386+0000","last_unstale":"2022-04-03T19:44:21.288717+0000","last_undegraded":"2022-04-03T19:44:21.288717+0000","last_fullsized":"2022-04-03T19:44:21.288717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:33:33.013536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.285800+0000","last_change":"2022-04-03T19:44:20.784840+0000","last_active":"2022-04-03T19:44:22.285800+0000","last_peered":"2022-04-03T19:44:22.285800+0000","last_clean":"2022-04-03T19:44:22.285800+0000","last_became_active":"2022-04-03T19:44:20.784521+0000","last_became_peered":"2022-04-03T19:44:20.784521+0000","last_unstale":"2022-04-03T19:44:22.285800+0000","last_undegraded":"2022-04-03T19:44:22.285800+0000","last_fullsized":"2022-04-03T19:44:22.285800+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:15:38.240804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293467+0000","last_change":"2022-04-03T19:44:21.290769+0000","last_active":"2022-04-03T19:44:21.293467+0000","last_peered":"2022-04-03T19:44:21.293467+0000","last_clean":"2022-04-03T19:44:21.293467+0000","last_became_active":"2022-04-03T19:44:21.290394+0000","last_became_peered":"2022-04-03T19:44:21.290394+0000","last_unstale":"2022-04-03T19:44:21.293467+0000","last_undegraded":"2022-04-03T19:44:21.293467+0000","last_fullsized":"2022-04-03T19:44:21.293467+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:57:39.528871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.285708+0000","last_change":"2022-04-03T19:44:20.789662+0000","last_active":"2022-04-03T19:44:22.285708+0000","last_peered":"2022-04-03T19:44:22.285708+0000","last_clean":"2022-04-03T19:44:22.285708+0000","last_became_active":"2022-04-03T19:44:20.787868+0000","last_became_peered":"2022-04-03T19:44:20.787868+0000","last_unstale":"2022-04-03T19:44:22.285708+0000","last_undegraded":"2022-04-03T19:44:22.285708+0000","last_fullsized":"2022-04-03T19:44:22.285708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:13:16.092180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305589+0000","last_change":"2022-04-03T19:44:21.304539+0000","last_active":"2022-04-03T19:44:21.305589+0000","last_peered":"2022-04-03T19:44:21.305589+0000","last_clean":"2022-04-03T19:44:21.305589+0000","last_became_active":"2022-04-03T19:44:21.304008+0000","last_became_peered":"2022-04-03T19:44:21.304008+0000","last_unstale":"2022-04-03T19:44:21.305589+0000","last_undegraded":"2022-04-03T19:44:21.305589+0000","last_fullsized":"2022-04-03T19:44:21.305589+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:45:01.346479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.292841+0000","last_change":"2022-04-03T19:44:20.786813+0000","last_active":"2022-04-03T19:44:26.292841+0000","last_peered":"2022-04-03T19:44:26.292841+0000","last_clean":"2022-04-03T19:44:26.292841+0000","last_became_active":"2022-04-03T19:44:20.786548+0000","last_became_peered":"2022-04-03T19:44:20.786548+0000","last_unstale":"2022-04-03T19:44:26.292841+0000","last_undegraded":"2022-04-03T19:44:26.292841+0000","last_fullsized":"2022-04-03T19:44:26.292841+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:36:40.611720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302733+0000","last_change":"2022-04-03T19:44:21.299513+0000","last_active":"2022-04-03T19:44:21.302733+0000","last_peered":"2022-04-03T19:44:21.302733+0000","last_clean":"2022-04-03T19:44:21.302733+0000","last_became_active":"2022-04-03T19:44:21.299270+0000","last_became_peered":"2022-04-03T19:44:21.299270+0000","last_unstale":"2022-04-03T19:44:21.302733+0000","last_undegraded":"2022-04-03T19:44:21.302733+0000","last_fullsized":"2022-04-03T19:44:21.302733+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:54:16.302815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.290239+0000","last_change":"2022-04-03T19:44:20.786750+0000","last_active":"2022-04-03T19:44:26.290239+0000","last_peered":"2022-04-03T19:44:26.290239+0000","last_clean":"2022-04-03T19:44:26.290239+0000","last_became_active":"2022-04-03T19:44:20.786490+0000","last_became_peered":"2022-04-03T19:44:20.786490+0000","last_unstale":"2022-04-03T19:44:26.290239+0000","last_undegraded":"2022-04-03T19:44:26.290239+0000","last_fullsized":"2022-04-03T19:44:26.290239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:30:31.980806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300228+0000","last_change":"2022-04-03T19:44:21.294849+0000","last_active":"2022-04-03T19:44:21.300228+0000","last_peered":"2022-04-03T19:44:21.300228+0000","last_clean":"2022-04-03T19:44:21.300228+0000","last_became_active":"2022-04-03T19:44:21.294586+0000","last_became_peered":"2022-04-03T19:44:21.294586+0000","last_unstale":"2022-04-03T19:44:21.300228+0000","last_undegraded":"2022-04-03T19:44:21.300228+0000","last_fullsized":"2022-04-03T19:44:21.300228+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:24:11.050193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.289167+0000","last_change":"2022-04-03T19:44:20.781619+0000","last_active":"2022-04-03T19:44:22.289167+0000","last_peered":"2022-04-03T19:44:22.289167+0000","last_clean":"2022-04-03T19:44:22.289167+0000","last_became_active":"2022-04-03T19:44:20.781288+0000","last_became_peered":"2022-04-03T19:44:20.781288+0000","last_unstale":"2022-04-03T19:44:22.289167+0000","last_undegraded":"2022-04-03T19:44:22.289167+0000","last_fullsized":"2022-04-03T19:44:22.289167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:05:56.179441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305553+0000","last_change":"2022-04-03T19:44:21.296332+0000","last_active":"2022-04-03T19:44:21.305553+0000","last_peered":"2022-04-03T19:44:21.305553+0000","last_clean":"2022-04-03T19:44:21.305553+0000","last_became_active":"2022-04-03T19:44:21.296026+0000","last_became_peered":"2022-04-03T19:44:21.296026+0000","last_unstale":"2022-04-03T19:44:21.305553+0000","last_undegraded":"2022-04-03T19:44:21.305553+0000","last_fullsized":"2022-04-03T19:44:21.305553+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:43:33.822187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"24'4","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.293148+0000","last_change":"2022-04-03T19:44:20.789448+0000","last_active":"2022-04-03T19:44:26.293148+0000","last_peered":"2022-04-03T19:44:26.293148+0000","last_clean":"2022-04-03T19:44:26.293148+0000","last_became_active":"2022-04-03T19:44:20.789131+0000","last_became_peered":"2022-04-03T19:44:20.789131+0000","last_unstale":"2022-04-03T19:44:26.293148+0000","last_undegraded":"2022-04-03T19:44:26.293148+0000","last_fullsized":"2022-04-03T19:44:26.293148+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:46:20.830676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293972+0000","last_change":"2022-04-03T19:44:21.291681+0000","last_active":"2022-04-03T19:44:21.293972+0000","last_peered":"2022-04-03T19:44:21.293972+0000","last_clean":"2022-04-03T19:44:21.293972+0000","last_became_active":"2022-04-03T19:44:21.290817+0000","last_became_peered":"2022-04-03T19:44:21.290817+0000","last_unstale":"2022-04-03T19:44:21.293972+0000","last_undegraded":"2022-04-03T19:44:21.293972+0000","last_fullsized":"2022-04-03T19:44:21.293972+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:37:30.833818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300290+0000","last_change":"2022-04-03T19:44:21.295604+0000","last_active":"2022-04-03T19:44:21.300290+0000","last_peered":"2022-04-03T19:44:21.300290+0000","last_clean":"2022-04-03T19:44:21.300290+0000","last_became_active":"2022-04-03T19:44:21.295297+0000","last_became_peered":"2022-04-03T19:44:21.295297+0000","last_unstale":"2022-04-03T19:44:21.300290+0000","last_undegraded":"2022-04-03T19:44:21.300290+0000","last_fullsized":"2022-04-03T19:44:21.300290+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:26:34.341290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.309922+0000","last_change":"2022-04-03T19:44:21.306547+0000","last_active":"2022-04-03T19:44:21.309922+0000","last_peered":"2022-04-03T19:44:21.309922+0000","last_clean":"2022-04-03T19:44:21.309922+0000","last_became_active":"2022-04-03T19:44:21.306179+0000","last_became_peered":"2022-04-03T19:44:21.306179+0000","last_unstale":"2022-04-03T19:44:21.309922+0000","last_undegraded":"2022-04-03T19:44:21.309922+0000","last_fullsized":"2022-04-03T19:44:21.309922+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:34:58.854671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293642+0000","last_change":"2022-04-03T19:44:21.291341+0000","last_active":"2022-04-03T19:44:21.293642+0000","last_peered":"2022-04-03T19:44:21.293642+0000","last_clean":"2022-04-03T19:44:21.293642+0000","last_became_active":"2022-04-03T19:44:21.290943+0000","last_became_peered":"2022-04-03T19:44:21.290943+0000","last_unstale":"2022-04-03T19:44:21.293642+0000","last_undegraded":"2022-04-03T19:44:21.293642+0000","last_fullsized":"2022-04-03T19:44:21.293642+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:50:56.877454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313543+0000","last_change":"2022-04-03T19:44:21.307815+0000","last_active":"2022-04-03T19:44:21.313543+0000","last_peered":"2022-04-03T19:44:21.313543+0000","last_clean":"2022-04-03T19:44:21.313543+0000","last_became_active":"2022-04-03T19:44:21.307459+0000","last_became_peered":"2022-04-03T19:44:21.307459+0000","last_unstale":"2022-04-03T19:44:21.313543+0000","last_undegraded":"2022-04-03T19:44:21.313543+0000","last_fullsized":"2022-04-03T19:44:21.313543+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:16:52.723409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.312899+0000","last_change":"2022-04-03T19:44:21.303095+0000","last_active":"2022-04-03T19:44:21.312899+0000","last_peered":"2022-04-03T19:44:21.312899+0000","last_clean":"2022-04-03T19:44:21.312899+0000","last_became_active":"2022-04-03T19:44:21.302363+0000","last_became_peered":"2022-04-03T19:44:21.302363+0000","last_unstale":"2022-04-03T19:44:21.312899+0000","last_undegraded":"2022-04-03T19:44:21.312899+0000","last_fullsized":"2022-04-03T19:44:21.312899+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:42:45.617661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2594,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8972,"kb_used_data":772,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94362868,"statfs":{"total":96636764160,"available":96627576832,"internally_reserved":0,"allocated":790528,"data_stored":623468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]}]},{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8996,"kb_used_data":796,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94362844,"statfs":{"total":96636764160,"available":96627552256,"internally_reserved":0,"allocated":815104,"data_stored":624930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7936,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363904,"statfs":{"total":96636764160,"available":96628637696,"internally_reserved":0,"allocated":253952,"data_stored":93974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8000,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363840,"statfs":{"total":96636764160,"available":96628572160,"internally_reserved":0,"allocated":253952,"data_stored":93974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8004,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363836,"statfs":{"total":96636764160,"available":96628568064,"internally_reserved":0,"allocated":258048,"data_stored":94064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8020,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363820,"statfs":{"total":96636764160,"available":96628551680,"internally_reserved":0,"allocated":274432,"data_stored":95842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7944,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363896,"statfs":{"total":96636764160,"available":96628629504,"internally_reserved":0,"allocated":262144,"data_stored":94610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7936,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363904,"statfs":{"total":96636764160,"available":96628637696,"internally_reserved":0,"allocated":253952,"data_stored":93974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-03T19:44:46.077 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-04-03T19:44:46.273 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.273+0000 7fb38e069700 1 Processor -- start 2022-04-03T19:44:46.274 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.273+0000 7fb38e069700 1 -- start start 2022-04-03T19:44:46.276 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.274+0000 7fb38e069700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb3880c1170 0x7fb3881527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:46.276 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.274+0000 7fb38e069700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb388155370 0x7fb388069110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:46.276 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.274+0000 7fb38e069700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3880c3710 con 0x7fb3880c3b10 2022-04-03T19:44:46.276 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.274+0000 7fb38e069700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb388152330 con 0x7fb3880c1170 2022-04-03T19:44:46.277 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.274+0000 7fb38e069700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3880c1540 con 0x7fb388155370 2022-04-03T19:44:46.277 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.275+0000 7fb38d067700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb3880c1170 0x7fb3881527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:46.277 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.275+0000 7fb38d868700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb3880c3b10 0x7fb3880c71f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38362/0 (socket says 172.21.15.77:38362) 2022-04-03T19:44:46.277 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.275+0000 7fb38d868700 1 -- 172.21.15.77:0/3314392613 learned_addr learned my addr 172.21.15.77:0/3314392613 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:46.277 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.275+0000 7fb38c866700 1 --2- 172.21.15.77:0/3314392613 >> v2:172.21.15.186:3300/0 conn(0x7fb388155370 0x7fb388069110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:46.278 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.275+0000 7fb377fff700 1 -- 172.21.15.77:0/3314392613 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507419769 0 0) 0x7fb3880c3710 con 0x7fb3880c3b10 2022-04-03T19:44:46.278 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.275+0000 7fb377fff700 1 -- 172.21.15.77:0/3314392613 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3700033f0 con 0x7fb3880c3b10 2022-04-03T19:44:46.278 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.275+0000 7fb38d067700 1 -- 172.21.15.77:0/3314392613 >> v2:172.21.15.186:3300/0 conn(0x7fb388155370 msgr2=0x7fb388069110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:46.278 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.275+0000 7fb38d067700 1 --2- 172.21.15.77:0/3314392613 >> v2:172.21.15.186:3300/0 conn(0x7fb388155370 0x7fb388069110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.279 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.275+0000 7fb38d067700 1 -- 172.21.15.77:0/3314392613 >> v1:172.21.15.77:6789/0 conn(0x7fb3880c3b10 legacy=0x7fb3880c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:46.279 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.275+0000 7fb38d067700 1 -- 172.21.15.77:0/3314392613 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3880c4270 con 0x7fb3880c1170 2022-04-03T19:44:46.279 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.276+0000 7fb38d067700 1 --2- 172.21.15.77:0/3314392613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3880c1170 0x7fb3881527c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb3880badf0 tx=0x7fb384009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38338d1da01de19f server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:46.282 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.276+0000 7fb377fff700 1 -- 172.21.15.77:0/3314392613 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb384011630 con 0x7fb3880c1170 2022-04-03T19:44:46.282 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.276+0000 7fb377fff700 1 -- 172.21.15.77:0/3314392613 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb384011790 con 0x7fb3880c1170 2022-04-03T19:44:46.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.276+0000 7fb377fff700 1 -- 172.21.15.77:0/3314392613 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb384011b30 con 0x7fb3880c1170 2022-04-03T19:44:46.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.276+0000 7fb38e069700 1 -- 172.21.15.77:0/3314392613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3880c1170 msgr2=0x7fb3881527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:46.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.276+0000 7fb38e069700 1 --2- 172.21.15.77:0/3314392613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3880c1170 0x7fb3881527c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb3880badf0 tx=0x7fb384009060 comp rx=0 tx=0).stop 2022-04-03T19:44:46.284 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.277+0000 7fb38e069700 1 -- 172.21.15.77:0/3314392613 shutdown_connections 2022-04-03T19:44:46.284 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.277+0000 7fb38e069700 1 --2- 172.21.15.77:0/3314392613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3880c1170 0x7fb3881527c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.284 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.277+0000 7fb38e069700 1 --2- 172.21.15.77:0/3314392613 >> v2:172.21.15.186:3300/0 conn(0x7fb388155370 0x7fb388069110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.284 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.277+0000 7fb38e069700 1 -- 172.21.15.77:0/3314392613 >> 172.21.15.77:0/3314392613 conn(0x7fb388066da0 msgr2=0x7fb3880671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:46.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.277+0000 7fb38e069700 1 -- 172.21.15.77:0/3314392613 shutdown_connections 2022-04-03T19:44:46.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.277+0000 7fb38e069700 1 -- 172.21.15.77:0/3314392613 wait complete. 2022-04-03T19:44:46.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.278+0000 7fb38e069700 1 Processor -- start 2022-04-03T19:44:46.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.278+0000 7fb38e069700 1 -- start start 2022-04-03T19:44:46.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.279+0000 7fb38e069700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3880c1170 0x7fb3880c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:46.286 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.279+0000 7fb38e069700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3880c3b10 0x7fb3880c5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:46.286 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.279+0000 7fb38e069700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb388155370 0x7fb38815cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:46.286 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.279+0000 7fb38e069700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb388149050 con 0x7fb388155370 2022-04-03T19:44:46.286 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.279+0000 7fb38e069700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38814a880 con 0x7fb3880c1170 2022-04-03T19:44:46.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.279+0000 7fb38e069700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3880c57c0 con 0x7fb3880c3b10 2022-04-03T19:44:46.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.279+0000 7fb38d868700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb388155370 0x7fb38815cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:46.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.279+0000 7fb38d868700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb388155370 0x7fb38815cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42694/0 (socket says 172.21.15.77:42694) 2022-04-03T19:44:46.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.280+0000 7fb38d868700 1 -- 172.21.15.77:0/4112526818 learned_addr learned my addr 172.21.15.77:0/4112526818 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:46.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.280+0000 7fb38d067700 1 --2- 172.21.15.77:0/4112526818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3880c1170 0x7fb3880c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:46.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.280+0000 7fb38c866700 1 --2- 172.21.15.77:0/4112526818 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3880c3b10 0x7fb3880c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:46.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.280+0000 7fb38d868700 1 -- 172.21.15.77:0/4112526818 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3880c3b10 msgr2=0x7fb3880c5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:46.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.280+0000 7fb38d868700 1 --2- 172.21.15.77:0/4112526818 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3880c3b10 0x7fb3880c5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.280+0000 7fb38d868700 1 -- 172.21.15.77:0/4112526818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3880c1170 msgr2=0x7fb3880c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:46.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.280+0000 7fb38d868700 1 --2- 172.21.15.77:0/4112526818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3880c1170 0x7fb3880c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.280+0000 7fb38d868700 1 -- 172.21.15.77:0/4112526818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3840092a0 con 0x7fb388155370 2022-04-03T19:44:46.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.280+0000 7fb38d868700 1 --2- 172.21.15.77:0/4112526818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb388155370 0x7fb38815cee0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb380005ad0 tx=0x7fb380001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:46.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.280+0000 7fb3757fa700 1 -- 172.21.15.77:0/4112526818 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb380005d30 con 0x7fb388155370 2022-04-03T19:44:46.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.281+0000 7fb3757fa700 1 -- 172.21.15.77:0/4112526818 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb380005e90 con 0x7fb388155370 2022-04-03T19:44:46.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.281+0000 7fb3757fa700 1 -- 172.21.15.77:0/4112526818 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb380007300 con 0x7fb388155370 2022-04-03T19:44:46.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.281+0000 7fb38e069700 1 -- 172.21.15.77:0/4112526818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb388169970 con 0x7fb388155370 2022-04-03T19:44:46.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.281+0000 7fb38e069700 1 -- 172.21.15.77:0/4112526818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3881492a0 con 0x7fb388155370 2022-04-03T19:44:46.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.282+0000 7fb38e069700 1 -- 172.21.15.77:0/4112526818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb38814a9f0 con 0x7fb388155370 2022-04-03T19:44:46.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.283+0000 7fb3757fa700 1 -- 172.21.15.77:0/4112526818 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fb380010070 con 0x7fb388155370 2022-04-03T19:44:46.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.283+0000 7fb3757fa700 1 --2- 172.21.15.77:0/4112526818 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb368063ce0 0x7fb368066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:46.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.283+0000 7fb3757fa700 1 -- 172.21.15.77:0/4112526818 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fb380080280 con 0x7fb388155370 2022-04-03T19:44:46.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.284+0000 7fb38d067700 1 --2- 172.21.15.77:0/4112526818 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb368063ce0 0x7fb368066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:46.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.285+0000 7fb38d067700 1 --2- 172.21.15.77:0/4112526818 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb368063ce0 0x7fb368066190 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb384009fd0 tx=0x7fb384017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:46.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.288+0000 7fb3757fa700 1 -- 172.21.15.77:0/4112526818 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fb38004eee0 con 0x7fb388155370 2022-04-03T19:44:46.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.482+0000 7fb38e069700 1 -- 172.21.15.77:0/4112526818 --> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb388067d70 con 0x7fb368063ce0 2022-04-03T19:44:46.492 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.491+0000 7fb3757fa700 1 -- 172.21.15.77:0/4112526818 <== mgr.4106 v2:172.21.15.186:6832/24948 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284674 (secure 0 0 0) 0x7fb388067d70 con 0x7fb368063ce0 2022-04-03T19:44:46.496 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:46.498 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.496+0000 7fb38e069700 1 -- 172.21.15.77:0/4112526818 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb368063ce0 msgr2=0x7fb368066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.496+0000 7fb38e069700 1 --2- 172.21.15.77:0/4112526818 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb368063ce0 0x7fb368066190 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb384009fd0 tx=0x7fb384017000 comp rx=0 tx=0).stop 2022-04-03T19:44:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.496+0000 7fb38e069700 1 -- 172.21.15.77:0/4112526818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb388155370 msgr2=0x7fb38815cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.496+0000 7fb38e069700 1 --2- 172.21.15.77:0/4112526818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb388155370 0x7fb38815cee0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb380005ad0 tx=0x7fb380001650 comp rx=0 tx=0).stop 2022-04-03T19:44:46.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.496+0000 7fb38e069700 1 -- 172.21.15.77:0/4112526818 shutdown_connections 2022-04-03T19:44:46.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.496+0000 7fb38e069700 1 --2- 172.21.15.77:0/4112526818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb388155370 0x7fb38815cee0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.496+0000 7fb38e069700 1 --2- 172.21.15.77:0/4112526818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3880c1170 0x7fb3880c70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.496+0000 7fb38e069700 1 --2- 172.21.15.77:0/4112526818 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb368063ce0 0x7fb368066190 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.496+0000 7fb38e069700 1 --2- 172.21.15.77:0/4112526818 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3880c3b10 0x7fb3880c5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.496+0000 7fb38e069700 1 -- 172.21.15.77:0/4112526818 >> 172.21.15.77:0/4112526818 conn(0x7fb388066da0 msgr2=0x7fb388155780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:46.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.496+0000 7fb38e069700 1 -- 172.21.15.77:0/4112526818 shutdown_connections 2022-04-03T19:44:46.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.496+0000 7fb38e069700 1 -- 172.21.15.77:0/4112526818 wait complete. 2022-04-03T19:44:46.501 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2022-04-03T19:44:46.520 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2022-04-03T19:44:45.494643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592962,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":205,"num_write_kb":2431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":65808,"kb_used_data":3088,"kb_used_omap":0,"kb_used_meta":62656,"kb_avail":754908912,"statfs":{"total":773094113280,"available":773026725888,"internally_reserved":0,"allocated":3162112,"data_stored":1814836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64159744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":264,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003270"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302418+0000","last_change":"2022-04-03T19:44:21.300867+0000","last_active":"2022-04-03T19:44:21.302418+0000","last_peered":"2022-04-03T19:44:21.302418+0000","last_clean":"2022-04-03T19:44:21.302418+0000","last_became_active":"2022-04-03T19:44:21.300569+0000","last_became_peered":"2022-04-03T19:44:21.300569+0000","last_unstale":"2022-04-03T19:44:21.302418+0000","last_undegraded":"2022-04-03T19:44:21.302418+0000","last_fullsized":"2022-04-03T19:44:21.302418+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:10:32.827314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305911+0000","last_change":"2022-04-03T19:44:21.302410+0000","last_active":"2022-04-03T19:44:21.305911+0000","last_peered":"2022-04-03T19:44:21.305911+0000","last_clean":"2022-04-03T19:44:21.305911+0000","last_became_active":"2022-04-03T19:44:21.302142+0000","last_became_peered":"2022-04-03T19:44:21.302142+0000","last_unstale":"2022-04-03T19:44:21.305911+0000","last_undegraded":"2022-04-03T19:44:21.305911+0000","last_fullsized":"2022-04-03T19:44:21.305911+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:01:46.327996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.298511+0000","last_change":"2022-04-03T19:44:21.293080+0000","last_active":"2022-04-03T19:44:21.298511+0000","last_peered":"2022-04-03T19:44:21.298511+0000","last_clean":"2022-04-03T19:44:21.298511+0000","last_became_active":"2022-04-03T19:44:21.292762+0000","last_became_peered":"2022-04-03T19:44:21.292762+0000","last_unstale":"2022-04-03T19:44:21.298511+0000","last_undegraded":"2022-04-03T19:44:21.298511+0000","last_fullsized":"2022-04-03T19:44:21.298511+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:28:36.726593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300326+0000","last_change":"2022-04-03T19:44:21.294448+0000","last_active":"2022-04-03T19:44:21.300326+0000","last_peered":"2022-04-03T19:44:21.300326+0000","last_clean":"2022-04-03T19:44:21.300326+0000","last_became_active":"2022-04-03T19:44:21.294083+0000","last_became_peered":"2022-04-03T19:44:21.294083+0000","last_unstale":"2022-04-03T19:44:21.300326+0000","last_undegraded":"2022-04-03T19:44:21.300326+0000","last_fullsized":"2022-04-03T19:44:21.300326+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:57:02.374826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.308268+0000","last_change":"2022-04-03T19:44:21.299551+0000","last_active":"2022-04-03T19:44:21.308268+0000","last_peered":"2022-04-03T19:44:21.308268+0000","last_clean":"2022-04-03T19:44:21.308268+0000","last_became_active":"2022-04-03T19:44:21.299123+0000","last_became_peered":"2022-04-03T19:44:21.299123+0000","last_unstale":"2022-04-03T19:44:21.308268+0000","last_undegraded":"2022-04-03T19:44:21.308268+0000","last_fullsized":"2022-04-03T19:44:21.308268+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:08:24.911647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300209+0000","last_change":"2022-04-03T19:44:21.293129+0000","last_active":"2022-04-03T19:44:21.300209+0000","last_peered":"2022-04-03T19:44:21.300209+0000","last_clean":"2022-04-03T19:44:21.300209+0000","last_became_active":"2022-04-03T19:44:21.292843+0000","last_became_peered":"2022-04-03T19:44:21.292843+0000","last_unstale":"2022-04-03T19:44:21.300209+0000","last_undegraded":"2022-04-03T19:44:21.300209+0000","last_fullsized":"2022-04-03T19:44:21.300209+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:38:32.432609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302772+0000","last_change":"2022-04-03T19:44:21.299112+0000","last_active":"2022-04-03T19:44:21.302772+0000","last_peered":"2022-04-03T19:44:21.302772+0000","last_clean":"2022-04-03T19:44:21.302772+0000","last_became_active":"2022-04-03T19:44:21.298803+0000","last_became_peered":"2022-04-03T19:44:21.298803+0000","last_unstale":"2022-04-03T19:44:21.302772+0000","last_undegraded":"2022-04-03T19:44:21.302772+0000","last_fullsized":"2022-04-03T19:44:21.302772+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:29:43.577843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299160+0000","last_change":"2022-04-03T19:44:21.293638+0000","last_active":"2022-04-03T19:44:21.299160+0000","last_peered":"2022-04-03T19:44:21.299160+0000","last_clean":"2022-04-03T19:44:21.299160+0000","last_became_active":"2022-04-03T19:44:21.293359+0000","last_became_peered":"2022-04-03T19:44:21.293359+0000","last_unstale":"2022-04-03T19:44:21.299160+0000","last_undegraded":"2022-04-03T19:44:21.299160+0000","last_fullsized":"2022-04-03T19:44:21.299160+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:28:50.113792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.306107+0000","last_change":"2022-04-03T19:44:21.299916+0000","last_active":"2022-04-03T19:44:21.306107+0000","last_peered":"2022-04-03T19:44:21.306107+0000","last_clean":"2022-04-03T19:44:21.306107+0000","last_became_active":"2022-04-03T19:44:21.299578+0000","last_became_peered":"2022-04-03T19:44:21.299578+0000","last_unstale":"2022-04-03T19:44:21.306107+0000","last_undegraded":"2022-04-03T19:44:21.306107+0000","last_fullsized":"2022-04-03T19:44:21.306107+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:38:40.281496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.298833+0000","last_change":"2022-04-03T19:44:21.294003+0000","last_active":"2022-04-03T19:44:21.298833+0000","last_peered":"2022-04-03T19:44:21.298833+0000","last_clean":"2022-04-03T19:44:21.298833+0000","last_became_active":"2022-04-03T19:44:21.293640+0000","last_became_peered":"2022-04-03T19:44:21.293640+0000","last_unstale":"2022-04-03T19:44:21.298833+0000","last_undegraded":"2022-04-03T19:44:21.298833+0000","last_fullsized":"2022-04-03T19:44:21.298833+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:40:13.436484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302745+0000","last_change":"2022-04-03T19:44:21.301158+0000","last_active":"2022-04-03T19:44:21.302745+0000","last_peered":"2022-04-03T19:44:21.302745+0000","last_clean":"2022-04-03T19:44:21.302745+0000","last_became_active":"2022-04-03T19:44:21.300823+0000","last_became_peered":"2022-04-03T19:44:21.300823+0000","last_unstale":"2022-04-03T19:44:21.302745+0000","last_undegraded":"2022-04-03T19:44:21.302745+0000","last_fullsized":"2022-04-03T19:44:21.302745+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:09:28.365828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299985+0000","last_change":"2022-04-03T19:44:21.295301+0000","last_active":"2022-04-03T19:44:21.299985+0000","last_peered":"2022-04-03T19:44:21.299985+0000","last_clean":"2022-04-03T19:44:21.299985+0000","last_became_active":"2022-04-03T19:44:21.295032+0000","last_became_peered":"2022-04-03T19:44:21.295032+0000","last_unstale":"2022-04-03T19:44:21.299985+0000","last_undegraded":"2022-04-03T19:44:21.299985+0000","last_fullsized":"2022-04-03T19:44:21.299985+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:55:55.637194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.301541+0000","last_change":"2022-04-03T19:44:21.296250+0000","last_active":"2022-04-03T19:44:21.301541+0000","last_peered":"2022-04-03T19:44:21.301541+0000","last_clean":"2022-04-03T19:44:21.301541+0000","last_became_active":"2022-04-03T19:44:21.295967+0000","last_became_peered":"2022-04-03T19:44:21.295967+0000","last_unstale":"2022-04-03T19:44:21.301541+0000","last_undegraded":"2022-04-03T19:44:21.301541+0000","last_fullsized":"2022-04-03T19:44:21.301541+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:43:34.301454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299466+0000","last_change":"2022-04-03T19:44:21.297229+0000","last_active":"2022-04-03T19:44:21.299466+0000","last_peered":"2022-04-03T19:44:21.299466+0000","last_clean":"2022-04-03T19:44:21.299466+0000","last_became_active":"2022-04-03T19:44:21.296993+0000","last_became_peered":"2022-04-03T19:44:21.296993+0000","last_unstale":"2022-04-03T19:44:21.299466+0000","last_undegraded":"2022-04-03T19:44:21.299466+0000","last_fullsized":"2022-04-03T19:44:21.299466+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:27:17.280580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303252+0000","last_change":"2022-04-03T19:44:21.302427+0000","last_active":"2022-04-03T19:44:21.303252+0000","last_peered":"2022-04-03T19:44:21.303252+0000","last_clean":"2022-04-03T19:44:21.303252+0000","last_became_active":"2022-04-03T19:44:21.302157+0000","last_became_peered":"2022-04-03T19:44:21.302157+0000","last_unstale":"2022-04-03T19:44:21.303252+0000","last_undegraded":"2022-04-03T19:44:21.303252+0000","last_fullsized":"2022-04-03T19:44:21.303252+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:57:23.292801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.298518+0000","last_change":"2022-04-03T19:44:21.293096+0000","last_active":"2022-04-03T19:44:21.298518+0000","last_peered":"2022-04-03T19:44:21.298518+0000","last_clean":"2022-04-03T19:44:21.298518+0000","last_became_active":"2022-04-03T19:44:21.292785+0000","last_became_peered":"2022-04-03T19:44:21.292785+0000","last_unstale":"2022-04-03T19:44:21.298518+0000","last_undegraded":"2022-04-03T19:44:21.298518+0000","last_fullsized":"2022-04-03T19:44:21.298518+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:27:06.759308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.306127+0000","last_change":"2022-04-03T19:44:21.300957+0000","last_active":"2022-04-03T19:44:21.306127+0000","last_peered":"2022-04-03T19:44:21.306127+0000","last_clean":"2022-04-03T19:44:21.306127+0000","last_became_active":"2022-04-03T19:44:21.300342+0000","last_became_peered":"2022-04-03T19:44:21.300342+0000","last_unstale":"2022-04-03T19:44:21.306127+0000","last_undegraded":"2022-04-03T19:44:21.306127+0000","last_fullsized":"2022-04-03T19:44:21.306127+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:43:53.194230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302142+0000","last_change":"2022-04-03T19:44:21.295996+0000","last_active":"2022-04-03T19:44:21.302142+0000","last_peered":"2022-04-03T19:44:21.302142+0000","last_clean":"2022-04-03T19:44:21.302142+0000","last_became_active":"2022-04-03T19:44:21.295766+0000","last_became_peered":"2022-04-03T19:44:21.295766+0000","last_unstale":"2022-04-03T19:44:21.302142+0000","last_undegraded":"2022-04-03T19:44:21.302142+0000","last_fullsized":"2022-04-03T19:44:21.302142+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:21:31.628478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302310+0000","last_change":"2022-04-03T19:44:21.299840+0000","last_active":"2022-04-03T19:44:21.302310+0000","last_peered":"2022-04-03T19:44:21.302310+0000","last_clean":"2022-04-03T19:44:21.302310+0000","last_became_active":"2022-04-03T19:44:21.299487+0000","last_became_peered":"2022-04-03T19:44:21.299487+0000","last_unstale":"2022-04-03T19:44:21.302310+0000","last_undegraded":"2022-04-03T19:44:21.302310+0000","last_fullsized":"2022-04-03T19:44:21.302310+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:24:39.583019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.295324+0000","last_change":"2022-04-03T19:44:21.291748+0000","last_active":"2022-04-03T19:44:21.295324+0000","last_peered":"2022-04-03T19:44:21.295324+0000","last_clean":"2022-04-03T19:44:21.295324+0000","last_became_active":"2022-04-03T19:44:21.290939+0000","last_became_peered":"2022-04-03T19:44:21.290939+0000","last_unstale":"2022-04-03T19:44:21.295324+0000","last_undegraded":"2022-04-03T19:44:21.295324+0000","last_fullsized":"2022-04-03T19:44:21.295324+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:28:45.710805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302521+0000","last_change":"2022-04-03T19:44:21.299442+0000","last_active":"2022-04-03T19:44:21.302521+0000","last_peered":"2022-04-03T19:44:21.302521+0000","last_clean":"2022-04-03T19:44:21.302521+0000","last_became_active":"2022-04-03T19:44:21.299166+0000","last_became_peered":"2022-04-03T19:44:21.299166+0000","last_unstale":"2022-04-03T19:44:21.302521+0000","last_undegraded":"2022-04-03T19:44:21.302521+0000","last_fullsized":"2022-04-03T19:44:21.302521+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:31:30.905517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.311290+0000","last_change":"2022-04-03T19:44:21.307007+0000","last_active":"2022-04-03T19:44:21.311290+0000","last_peered":"2022-04-03T19:44:21.311290+0000","last_clean":"2022-04-03T19:44:21.311290+0000","last_became_active":"2022-04-03T19:44:21.306650+0000","last_became_peered":"2022-04-03T19:44:21.306650+0000","last_unstale":"2022-04-03T19:44:21.311290+0000","last_undegraded":"2022-04-03T19:44:21.311290+0000","last_fullsized":"2022-04-03T19:44:21.311290+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:03:26.043624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299849+0000","last_change":"2022-04-03T19:44:21.293566+0000","last_active":"2022-04-03T19:44:21.299849+0000","last_peered":"2022-04-03T19:44:21.299849+0000","last_clean":"2022-04-03T19:44:21.299849+0000","last_became_active":"2022-04-03T19:44:21.293263+0000","last_became_peered":"2022-04-03T19:44:21.293263+0000","last_unstale":"2022-04-03T19:44:21.299849+0000","last_undegraded":"2022-04-03T19:44:21.299849+0000","last_fullsized":"2022-04-03T19:44:21.299849+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:39:52.523944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302610+0000","last_change":"2022-04-03T19:44:21.300329+0000","last_active":"2022-04-03T19:44:21.302610+0000","last_peered":"2022-04-03T19:44:21.302610+0000","last_clean":"2022-04-03T19:44:21.302610+0000","last_became_active":"2022-04-03T19:44:21.300000+0000","last_became_peered":"2022-04-03T19:44:21.300000+0000","last_unstale":"2022-04-03T19:44:21.302610+0000","last_undegraded":"2022-04-03T19:44:21.302610+0000","last_fullsized":"2022-04-03T19:44:21.302610+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:15:10.839571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.312922+0000","last_change":"2022-04-03T19:44:21.302506+0000","last_active":"2022-04-03T19:44:21.312922+0000","last_peered":"2022-04-03T19:44:21.312922+0000","last_clean":"2022-04-03T19:44:21.312922+0000","last_became_active":"2022-04-03T19:44:21.302235+0000","last_became_peered":"2022-04-03T19:44:21.302235+0000","last_unstale":"2022-04-03T19:44:21.312922+0000","last_undegraded":"2022-04-03T19:44:21.312922+0000","last_fullsized":"2022-04-03T19:44:21.312922+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:25:25.801754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313258+0000","last_change":"2022-04-03T19:44:21.306670+0000","last_active":"2022-04-03T19:44:21.313258+0000","last_peered":"2022-04-03T19:44:21.313258+0000","last_clean":"2022-04-03T19:44:21.313258+0000","last_became_active":"2022-04-03T19:44:21.305722+0000","last_became_peered":"2022-04-03T19:44:21.305722+0000","last_unstale":"2022-04-03T19:44:21.313258+0000","last_undegraded":"2022-04-03T19:44:21.313258+0000","last_fullsized":"2022-04-03T19:44:21.313258+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:44:09.515207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303081+0000","last_change":"2022-04-03T19:44:21.300024+0000","last_active":"2022-04-03T19:44:21.303081+0000","last_peered":"2022-04-03T19:44:21.303081+0000","last_clean":"2022-04-03T19:44:21.303081+0000","last_became_active":"2022-04-03T19:44:21.299753+0000","last_became_peered":"2022-04-03T19:44:21.299753+0000","last_unstale":"2022-04-03T19:44:21.303081+0000","last_undegraded":"2022-04-03T19:44:21.303081+0000","last_fullsized":"2022-04-03T19:44:21.303081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:15:31.808623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.390304+0000","last_change":"2022-04-03T19:44:20.783898+0000","last_active":"2022-04-03T19:44:22.390304+0000","last_peered":"2022-04-03T19:44:22.390304+0000","last_clean":"2022-04-03T19:44:22.390304+0000","last_became_active":"2022-04-03T19:44:20.783612+0000","last_became_peered":"2022-04-03T19:44:20.783612+0000","last_unstale":"2022-04-03T19:44:22.390304+0000","last_undegraded":"2022-04-03T19:44:22.390304+0000","last_fullsized":"2022-04-03T19:44:22.390304+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:13:32.863927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.310259+0000","last_change":"2022-04-03T19:44:21.306904+0000","last_active":"2022-04-03T19:44:21.310259+0000","last_peered":"2022-04-03T19:44:21.310259+0000","last_clean":"2022-04-03T19:44:21.310259+0000","last_became_active":"2022-04-03T19:44:21.306616+0000","last_became_peered":"2022-04-03T19:44:21.306616+0000","last_unstale":"2022-04-03T19:44:21.310259+0000","last_undegraded":"2022-04-03T19:44:21.310259+0000","last_fullsized":"2022-04-03T19:44:21.310259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:49:35.783912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.296341+0000","last_change":"2022-04-03T19:44:20.788125+0000","last_active":"2022-04-03T19:44:21.296341+0000","last_peered":"2022-04-03T19:44:21.296341+0000","last_clean":"2022-04-03T19:44:21.296341+0000","last_became_active":"2022-04-03T19:44:20.787792+0000","last_became_peered":"2022-04-03T19:44:20.787792+0000","last_unstale":"2022-04-03T19:44:21.296341+0000","last_undegraded":"2022-04-03T19:44:21.296341+0000","last_fullsized":"2022-04-03T19:44:21.296341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:11:51.126774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305572+0000","last_change":"2022-04-03T19:44:21.302904+0000","last_active":"2022-04-03T19:44:21.305572+0000","last_peered":"2022-04-03T19:44:21.305572+0000","last_clean":"2022-04-03T19:44:21.305572+0000","last_became_active":"2022-04-03T19:44:21.301564+0000","last_became_peered":"2022-04-03T19:44:21.301564+0000","last_unstale":"2022-04-03T19:44:21.305572+0000","last_undegraded":"2022-04-03T19:44:21.305572+0000","last_fullsized":"2022-04-03T19:44:21.305572+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:25:51.611789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.283649+0000","last_change":"2022-04-03T19:44:20.792529+0000","last_active":"2022-04-03T19:44:21.283649+0000","last_peered":"2022-04-03T19:44:21.283649+0000","last_clean":"2022-04-03T19:44:21.283649+0000","last_became_active":"2022-04-03T19:44:20.792014+0000","last_became_peered":"2022-04-03T19:44:20.792014+0000","last_unstale":"2022-04-03T19:44:21.283649+0000","last_undegraded":"2022-04-03T19:44:21.283649+0000","last_fullsized":"2022-04-03T19:44:21.283649+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:01:38.319693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302637+0000","last_change":"2022-04-03T19:44:21.299302+0000","last_active":"2022-04-03T19:44:21.302637+0000","last_peered":"2022-04-03T19:44:21.302637+0000","last_clean":"2022-04-03T19:44:21.302637+0000","last_became_active":"2022-04-03T19:44:21.299007+0000","last_became_peered":"2022-04-03T19:44:21.299007+0000","last_unstale":"2022-04-03T19:44:21.302637+0000","last_undegraded":"2022-04-03T19:44:21.302637+0000","last_fullsized":"2022-04-03T19:44:21.302637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:59:48.767335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.282232+0000","last_change":"2022-04-03T19:44:20.790672+0000","last_active":"2022-04-03T19:44:21.282232+0000","last_peered":"2022-04-03T19:44:21.282232+0000","last_clean":"2022-04-03T19:44:21.282232+0000","last_became_active":"2022-04-03T19:44:20.788526+0000","last_became_peered":"2022-04-03T19:44:20.788526+0000","last_unstale":"2022-04-03T19:44:21.282232+0000","last_undegraded":"2022-04-03T19:44:21.282232+0000","last_fullsized":"2022-04-03T19:44:21.282232+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:14:50.058917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.309960+0000","last_change":"2022-04-03T19:44:21.306472+0000","last_active":"2022-04-03T19:44:21.309960+0000","last_peered":"2022-04-03T19:44:21.309960+0000","last_clean":"2022-04-03T19:44:21.309960+0000","last_became_active":"2022-04-03T19:44:21.306071+0000","last_became_peered":"2022-04-03T19:44:21.306071+0000","last_unstale":"2022-04-03T19:44:21.309960+0000","last_undegraded":"2022-04-03T19:44:21.309960+0000","last_fullsized":"2022-04-03T19:44:21.309960+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:43:22.221014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286837+0000","last_change":"2022-04-03T19:44:20.785638+0000","last_active":"2022-04-03T19:44:21.286837+0000","last_peered":"2022-04-03T19:44:21.286837+0000","last_clean":"2022-04-03T19:44:21.286837+0000","last_became_active":"2022-04-03T19:44:20.784963+0000","last_became_peered":"2022-04-03T19:44:20.784963+0000","last_unstale":"2022-04-03T19:44:21.286837+0000","last_undegraded":"2022-04-03T19:44:21.286837+0000","last_fullsized":"2022-04-03T19:44:21.286837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:39:26.933298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.309328+0000","last_change":"2022-04-03T19:44:21.299583+0000","last_active":"2022-04-03T19:44:21.309328+0000","last_peered":"2022-04-03T19:44:21.309328+0000","last_clean":"2022-04-03T19:44:21.309328+0000","last_became_active":"2022-04-03T19:44:21.299301+0000","last_became_peered":"2022-04-03T19:44:21.299301+0000","last_unstale":"2022-04-03T19:44:21.309328+0000","last_undegraded":"2022-04-03T19:44:21.309328+0000","last_fullsized":"2022-04-03T19:44:21.309328+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:37:49.567741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"24'3","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.290849+0000","last_change":"2022-04-03T19:44:20.795890+0000","last_active":"2022-04-03T19:44:26.290849+0000","last_peered":"2022-04-03T19:44:26.290849+0000","last_clean":"2022-04-03T19:44:26.290849+0000","last_became_active":"2022-04-03T19:44:20.795611+0000","last_became_peered":"2022-04-03T19:44:20.795611+0000","last_unstale":"2022-04-03T19:44:26.290849+0000","last_undegraded":"2022-04-03T19:44:26.290849+0000","last_fullsized":"2022-04-03T19:44:26.290849+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:08:33.216934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313173+0000","last_change":"2022-04-03T19:44:21.304724+0000","last_active":"2022-04-03T19:44:21.313173+0000","last_peered":"2022-04-03T19:44:21.313173+0000","last_clean":"2022-04-03T19:44:21.313173+0000","last_became_active":"2022-04-03T19:44:21.302708+0000","last_became_peered":"2022-04-03T19:44:21.302708+0000","last_unstale":"2022-04-03T19:44:21.313173+0000","last_undegraded":"2022-04-03T19:44:21.313173+0000","last_fullsized":"2022-04-03T19:44:21.313173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:04:54.970100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"24'7","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.294405+0000","last_change":"2022-04-03T19:44:20.785659+0000","last_active":"2022-04-03T19:44:26.294405+0000","last_peered":"2022-04-03T19:44:26.294405+0000","last_clean":"2022-04-03T19:44:26.294405+0000","last_became_active":"2022-04-03T19:44:20.785308+0000","last_became_peered":"2022-04-03T19:44:20.785308+0000","last_unstale":"2022-04-03T19:44:26.294405+0000","last_undegraded":"2022-04-03T19:44:26.294405+0000","last_fullsized":"2022-04-03T19:44:26.294405+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:52:08.268532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1812,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293514+0000","last_change":"2022-04-03T19:44:21.291033+0000","last_active":"2022-04-03T19:44:21.293514+0000","last_peered":"2022-04-03T19:44:21.293514+0000","last_clean":"2022-04-03T19:44:21.293514+0000","last_became_active":"2022-04-03T19:44:21.290713+0000","last_became_peered":"2022-04-03T19:44:21.290713+0000","last_unstale":"2022-04-03T19:44:21.293514+0000","last_undegraded":"2022-04-03T19:44:21.293514+0000","last_fullsized":"2022-04-03T19:44:21.293514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:19:41.873591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.290200+0000","last_change":"2022-04-03T19:44:20.787511+0000","last_active":"2022-04-03T19:44:26.290200+0000","last_peered":"2022-04-03T19:44:26.290200+0000","last_clean":"2022-04-03T19:44:26.290200+0000","last_became_active":"2022-04-03T19:44:20.787189+0000","last_became_peered":"2022-04-03T19:44:20.787189+0000","last_unstale":"2022-04-03T19:44:26.290200+0000","last_undegraded":"2022-04-03T19:44:26.290200+0000","last_fullsized":"2022-04-03T19:44:26.290200+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:11:56.297263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313518+0000","last_change":"2022-04-03T19:44:21.307846+0000","last_active":"2022-04-03T19:44:21.313518+0000","last_peered":"2022-04-03T19:44:21.313518+0000","last_clean":"2022-04-03T19:44:21.313518+0000","last_became_active":"2022-04-03T19:44:21.307513+0000","last_became_peered":"2022-04-03T19:44:21.307513+0000","last_unstale":"2022-04-03T19:44:21.313518+0000","last_undegraded":"2022-04-03T19:44:21.313518+0000","last_fullsized":"2022-04-03T19:44:21.313518+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:58:09.091062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.281910+0000","last_change":"2022-04-03T19:44:20.790212+0000","last_active":"2022-04-03T19:44:21.281910+0000","last_peered":"2022-04-03T19:44:21.281910+0000","last_clean":"2022-04-03T19:44:21.281910+0000","last_became_active":"2022-04-03T19:44:20.789863+0000","last_became_peered":"2022-04-03T19:44:20.789863+0000","last_unstale":"2022-04-03T19:44:21.281910+0000","last_undegraded":"2022-04-03T19:44:21.281910+0000","last_fullsized":"2022-04-03T19:44:21.281910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:48:13.489309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303061+0000","last_change":"2022-04-03T19:44:21.299953+0000","last_active":"2022-04-03T19:44:21.303061+0000","last_peered":"2022-04-03T19:44:21.303061+0000","last_clean":"2022-04-03T19:44:21.303061+0000","last_became_active":"2022-04-03T19:44:21.299667+0000","last_became_peered":"2022-04-03T19:44:21.299667+0000","last_unstale":"2022-04-03T19:44:21.303061+0000","last_undegraded":"2022-04-03T19:44:21.303061+0000","last_fullsized":"2022-04-03T19:44:21.303061+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:51:16.125160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286253+0000","last_change":"2022-04-03T19:44:20.791054+0000","last_active":"2022-04-03T19:44:21.286253+0000","last_peered":"2022-04-03T19:44:21.286253+0000","last_clean":"2022-04-03T19:44:21.286253+0000","last_became_active":"2022-04-03T19:44:20.790664+0000","last_became_peered":"2022-04-03T19:44:20.790664+0000","last_unstale":"2022-04-03T19:44:21.286253+0000","last_undegraded":"2022-04-03T19:44:21.286253+0000","last_fullsized":"2022-04-03T19:44:21.286253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:07:27.501728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.304407+0000","last_change":"2022-04-03T19:44:21.295028+0000","last_active":"2022-04-03T19:44:21.304407+0000","last_peered":"2022-04-03T19:44:21.304407+0000","last_clean":"2022-04-03T19:44:21.304407+0000","last_became_active":"2022-04-03T19:44:21.294723+0000","last_became_peered":"2022-04-03T19:44:21.294723+0000","last_unstale":"2022-04-03T19:44:21.304407+0000","last_undegraded":"2022-04-03T19:44:21.304407+0000","last_fullsized":"2022-04-03T19:44:21.304407+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:32:45.455842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"24'8","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.294387+0000","last_change":"2022-04-03T19:44:20.792133+0000","last_active":"2022-04-03T19:44:26.294387+0000","last_peered":"2022-04-03T19:44:26.294387+0000","last_clean":"2022-04-03T19:44:26.294387+0000","last_became_active":"2022-04-03T19:44:20.791829+0000","last_became_peered":"2022-04-03T19:44:20.791829+0000","last_unstale":"2022-04-03T19:44:26.294387+0000","last_undegraded":"2022-04-03T19:44:26.294387+0000","last_fullsized":"2022-04-03T19:44:26.294387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:01:21.966734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.310014+0000","last_change":"2022-04-03T19:44:21.300267+0000","last_active":"2022-04-03T19:44:21.310014+0000","last_peered":"2022-04-03T19:44:21.310014+0000","last_clean":"2022-04-03T19:44:21.310014+0000","last_became_active":"2022-04-03T19:44:21.299948+0000","last_became_peered":"2022-04-03T19:44:21.299948+0000","last_unstale":"2022-04-03T19:44:21.310014+0000","last_undegraded":"2022-04-03T19:44:21.310014+0000","last_fullsized":"2022-04-03T19:44:21.310014+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:06:54.481379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.391229+0000","last_change":"2022-04-03T19:44:20.789793+0000","last_active":"2022-04-03T19:44:22.391229+0000","last_peered":"2022-04-03T19:44:22.391229+0000","last_clean":"2022-04-03T19:44:22.391229+0000","last_became_active":"2022-04-03T19:44:20.789371+0000","last_became_peered":"2022-04-03T19:44:20.789371+0000","last_unstale":"2022-04-03T19:44:22.391229+0000","last_undegraded":"2022-04-03T19:44:22.391229+0000","last_fullsized":"2022-04-03T19:44:22.391229+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:15:52.060059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.301800+0000","last_change":"2022-04-03T19:44:21.298477+0000","last_active":"2022-04-03T19:44:21.301800+0000","last_peered":"2022-04-03T19:44:21.301800+0000","last_clean":"2022-04-03T19:44:21.301800+0000","last_became_active":"2022-04-03T19:44:21.298165+0000","last_became_peered":"2022-04-03T19:44:21.298165+0000","last_unstale":"2022-04-03T19:44:21.301800+0000","last_undegraded":"2022-04-03T19:44:21.301800+0000","last_fullsized":"2022-04-03T19:44:21.301800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:23:11.042589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.290125+0000","last_change":"2022-04-03T19:44:20.787949+0000","last_active":"2022-04-03T19:44:26.290125+0000","last_peered":"2022-04-03T19:44:26.290125+0000","last_clean":"2022-04-03T19:44:26.290125+0000","last_became_active":"2022-04-03T19:44:20.787592+0000","last_became_peered":"2022-04-03T19:44:20.787592+0000","last_unstale":"2022-04-03T19:44:26.290125+0000","last_undegraded":"2022-04-03T19:44:26.290125+0000","last_fullsized":"2022-04-03T19:44:26.290125+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:50:09.411246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302765+0000","last_change":"2022-04-03T19:44:21.301200+0000","last_active":"2022-04-03T19:44:21.302765+0000","last_peered":"2022-04-03T19:44:21.302765+0000","last_clean":"2022-04-03T19:44:21.302765+0000","last_became_active":"2022-04-03T19:44:21.300878+0000","last_became_peered":"2022-04-03T19:44:21.300878+0000","last_unstale":"2022-04-03T19:44:21.302765+0000","last_undegraded":"2022-04-03T19:44:21.302765+0000","last_fullsized":"2022-04-03T19:44:21.302765+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:54:36.089411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.391128+0000","last_change":"2022-04-03T19:44:20.783711+0000","last_active":"2022-04-03T19:44:22.391128+0000","last_peered":"2022-04-03T19:44:22.391128+0000","last_clean":"2022-04-03T19:44:22.391128+0000","last_became_active":"2022-04-03T19:44:20.783423+0000","last_became_peered":"2022-04-03T19:44:20.783423+0000","last_unstale":"2022-04-03T19:44:22.391128+0000","last_undegraded":"2022-04-03T19:44:22.391128+0000","last_fullsized":"2022-04-03T19:44:22.391128+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:22:37.665229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303123+0000","last_change":"2022-04-03T19:44:21.300815+0000","last_active":"2022-04-03T19:44:21.303123+0000","last_peered":"2022-04-03T19:44:21.303123+0000","last_clean":"2022-04-03T19:44:21.303123+0000","last_became_active":"2022-04-03T19:44:21.300545+0000","last_became_peered":"2022-04-03T19:44:21.300545+0000","last_unstale":"2022-04-03T19:44:21.303123+0000","last_undegraded":"2022-04-03T19:44:21.303123+0000","last_fullsized":"2022-04-03T19:44:21.303123+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:33:13.941191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.285374+0000","last_change":"2022-04-03T19:44:20.785480+0000","last_active":"2022-04-03T19:44:21.285374+0000","last_peered":"2022-04-03T19:44:21.285374+0000","last_clean":"2022-04-03T19:44:21.285374+0000","last_became_active":"2022-04-03T19:44:20.785178+0000","last_became_peered":"2022-04-03T19:44:20.785178+0000","last_unstale":"2022-04-03T19:44:21.285374+0000","last_undegraded":"2022-04-03T19:44:21.285374+0000","last_fullsized":"2022-04-03T19:44:21.285374+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:15:37.181452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.308247+0000","last_change":"2022-04-03T19:44:21.296509+0000","last_active":"2022-04-03T19:44:21.308247+0000","last_peered":"2022-04-03T19:44:21.308247+0000","last_clean":"2022-04-03T19:44:21.308247+0000","last_became_active":"2022-04-03T19:44:21.296240+0000","last_became_peered":"2022-04-03T19:44:21.296240+0000","last_unstale":"2022-04-03T19:44:21.308247+0000","last_undegraded":"2022-04-03T19:44:21.308247+0000","last_fullsized":"2022-04-03T19:44:21.308247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:50:45.955195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.288396+0000","last_change":"2022-04-03T19:44:20.792550+0000","last_active":"2022-04-03T19:44:26.288396+0000","last_peered":"2022-04-03T19:44:26.288396+0000","last_clean":"2022-04-03T19:44:26.288396+0000","last_became_active":"2022-04-03T19:44:20.792191+0000","last_became_peered":"2022-04-03T19:44:20.792191+0000","last_unstale":"2022-04-03T19:44:26.288396+0000","last_undegraded":"2022-04-03T19:44:26.288396+0000","last_fullsized":"2022-04-03T19:44:26.288396+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:51:30.297728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299612+0000","last_change":"2022-04-03T19:44:21.294898+0000","last_active":"2022-04-03T19:44:21.299612+0000","last_peered":"2022-04-03T19:44:21.299612+0000","last_clean":"2022-04-03T19:44:21.299612+0000","last_became_active":"2022-04-03T19:44:21.294635+0000","last_became_peered":"2022-04-03T19:44:21.294635+0000","last_unstale":"2022-04-03T19:44:21.299612+0000","last_undegraded":"2022-04-03T19:44:21.299612+0000","last_fullsized":"2022-04-03T19:44:21.299612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:42:52.629552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.289106+0000","last_change":"2022-04-03T19:44:20.783711+0000","last_active":"2022-04-03T19:44:21.289106+0000","last_peered":"2022-04-03T19:44:21.289106+0000","last_clean":"2022-04-03T19:44:21.289106+0000","last_became_active":"2022-04-03T19:44:20.783426+0000","last_became_peered":"2022-04-03T19:44:20.783426+0000","last_unstale":"2022-04-03T19:44:21.289106+0000","last_undegraded":"2022-04-03T19:44:21.289106+0000","last_fullsized":"2022-04-03T19:44:21.289106+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:25:26.219558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302205+0000","last_change":"2022-04-03T19:44:21.296081+0000","last_active":"2022-04-03T19:44:21.302205+0000","last_peered":"2022-04-03T19:44:21.302205+0000","last_clean":"2022-04-03T19:44:21.302205+0000","last_became_active":"2022-04-03T19:44:21.295784+0000","last_became_peered":"2022-04-03T19:44:21.295784+0000","last_unstale":"2022-04-03T19:44:21.302205+0000","last_undegraded":"2022-04-03T19:44:21.302205+0000","last_fullsized":"2022-04-03T19:44:21.302205+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:23:40.568168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.283775+0000","last_change":"2022-04-03T19:44:20.789868+0000","last_active":"2022-04-03T19:44:21.283775+0000","last_peered":"2022-04-03T19:44:21.283775+0000","last_clean":"2022-04-03T19:44:21.283775+0000","last_became_active":"2022-04-03T19:44:20.789563+0000","last_became_peered":"2022-04-03T19:44:20.789563+0000","last_unstale":"2022-04-03T19:44:21.283775+0000","last_undegraded":"2022-04-03T19:44:21.283775+0000","last_fullsized":"2022-04-03T19:44:21.283775+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:04:22.814932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293542+0000","last_change":"2022-04-03T19:44:21.291212+0000","last_active":"2022-04-03T19:44:21.293542+0000","last_peered":"2022-04-03T19:44:21.293542+0000","last_clean":"2022-04-03T19:44:21.293542+0000","last_became_active":"2022-04-03T19:44:21.290888+0000","last_became_peered":"2022-04-03T19:44:21.290888+0000","last_unstale":"2022-04-03T19:44:21.293542+0000","last_undegraded":"2022-04-03T19:44:21.293542+0000","last_fullsized":"2022-04-03T19:44:21.293542+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:29:34.850679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.281344+0000","last_change":"2022-04-03T19:44:20.789319+0000","last_active":"2022-04-03T19:44:21.281344+0000","last_peered":"2022-04-03T19:44:21.281344+0000","last_clean":"2022-04-03T19:44:21.281344+0000","last_became_active":"2022-04-03T19:44:20.788840+0000","last_became_peered":"2022-04-03T19:44:20.788840+0000","last_unstale":"2022-04-03T19:44:21.281344+0000","last_undegraded":"2022-04-03T19:44:21.281344+0000","last_fullsized":"2022-04-03T19:44:21.281344+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:50:21.536926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313655+0000","last_change":"2022-04-03T19:44:21.311886+0000","last_active":"2022-04-03T19:44:21.313655+0000","last_peered":"2022-04-03T19:44:21.313655+0000","last_clean":"2022-04-03T19:44:21.313655+0000","last_became_active":"2022-04-03T19:44:21.311521+0000","last_became_peered":"2022-04-03T19:44:21.311521+0000","last_unstale":"2022-04-03T19:44:21.313655+0000","last_undegraded":"2022-04-03T19:44:21.313655+0000","last_fullsized":"2022-04-03T19:44:21.313655+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:25:47.761166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.297426+0000","last_change":"2022-04-03T19:44:14.754674+0000","last_active":"2022-04-03T19:44:21.297426+0000","last_peered":"2022-04-03T19:44:21.297426+0000","last_clean":"2022-04-03T19:44:21.297426+0000","last_became_active":"2022-04-03T19:44:14.754210+0000","last_became_peered":"2022-04-03T19:44:14.754210+0000","last_unstale":"2022-04-03T19:44:21.297426+0000","last_undegraded":"2022-04-03T19:44:21.297426+0000","last_fullsized":"2022-04-03T19:44:21.297426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:23:14.624494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286308+0000","last_change":"2022-04-03T19:44:20.785400+0000","last_active":"2022-04-03T19:44:21.286308+0000","last_peered":"2022-04-03T19:44:21.286308+0000","last_clean":"2022-04-03T19:44:21.286308+0000","last_became_active":"2022-04-03T19:44:20.785125+0000","last_became_peered":"2022-04-03T19:44:20.785125+0000","last_unstale":"2022-04-03T19:44:21.286308+0000","last_undegraded":"2022-04-03T19:44:21.286308+0000","last_fullsized":"2022-04-03T19:44:21.286308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:24:02.690646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313197+0000","last_change":"2022-04-03T19:44:21.305321+0000","last_active":"2022-04-03T19:44:21.313197+0000","last_peered":"2022-04-03T19:44:21.313197+0000","last_clean":"2022-04-03T19:44:21.313197+0000","last_became_active":"2022-04-03T19:44:21.304017+0000","last_became_peered":"2022-04-03T19:44:21.304017+0000","last_unstale":"2022-04-03T19:44:21.313197+0000","last_undegraded":"2022-04-03T19:44:21.313197+0000","last_fullsized":"2022-04-03T19:44:21.313197+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:59:12.619058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.285445+0000","last_change":"2022-04-03T19:44:14.755580+0000","last_active":"2022-04-03T19:44:21.285445+0000","last_peered":"2022-04-03T19:44:21.285445+0000","last_clean":"2022-04-03T19:44:21.285445+0000","last_became_active":"2022-04-03T19:44:14.755222+0000","last_became_peered":"2022-04-03T19:44:14.755222+0000","last_unstale":"2022-04-03T19:44:21.285445+0000","last_undegraded":"2022-04-03T19:44:21.285445+0000","last_fullsized":"2022-04-03T19:44:21.285445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:08:18.314326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.283116+0000","last_change":"2022-04-03T19:44:20.791540+0000","last_active":"2022-04-03T19:44:21.283116+0000","last_peered":"2022-04-03T19:44:21.283116+0000","last_clean":"2022-04-03T19:44:21.283116+0000","last_became_active":"2022-04-03T19:44:20.791192+0000","last_became_peered":"2022-04-03T19:44:20.791192+0000","last_unstale":"2022-04-03T19:44:21.283116+0000","last_undegraded":"2022-04-03T19:44:21.283116+0000","last_fullsized":"2022-04-03T19:44:21.283116+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:58:59.674606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313218+0000","last_change":"2022-04-03T19:44:21.305930+0000","last_active":"2022-04-03T19:44:21.313218+0000","last_peered":"2022-04-03T19:44:21.313218+0000","last_clean":"2022-04-03T19:44:21.313218+0000","last_became_active":"2022-04-03T19:44:21.305659+0000","last_became_peered":"2022-04-03T19:44:21.305659+0000","last_unstale":"2022-04-03T19:44:21.313218+0000","last_undegraded":"2022-04-03T19:44:21.313218+0000","last_fullsized":"2022-04-03T19:44:21.313218+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:14:34.958569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.290570+0000","last_change":"2022-04-03T19:44:14.754598+0000","last_active":"2022-04-03T19:44:21.290570+0000","last_peered":"2022-04-03T19:44:21.290570+0000","last_clean":"2022-04-03T19:44:21.290570+0000","last_became_active":"2022-04-03T19:44:14.754287+0000","last_became_peered":"2022-04-03T19:44:14.754287+0000","last_unstale":"2022-04-03T19:44:21.290570+0000","last_undegraded":"2022-04-03T19:44:21.290570+0000","last_fullsized":"2022-04-03T19:44:21.290570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:32:16.232689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.296054+0000","last_change":"2022-04-03T19:44:20.789567+0000","last_active":"2022-04-03T19:44:21.296054+0000","last_peered":"2022-04-03T19:44:21.296054+0000","last_clean":"2022-04-03T19:44:21.296054+0000","last_became_active":"2022-04-03T19:44:20.788664+0000","last_became_peered":"2022-04-03T19:44:20.788664+0000","last_unstale":"2022-04-03T19:44:21.296054+0000","last_undegraded":"2022-04-03T19:44:21.296054+0000","last_fullsized":"2022-04-03T19:44:21.296054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:15:37.098494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.312944+0000","last_change":"2022-04-03T19:44:21.303079+0000","last_active":"2022-04-03T19:44:21.312944+0000","last_peered":"2022-04-03T19:44:21.312944+0000","last_clean":"2022-04-03T19:44:21.312944+0000","last_became_active":"2022-04-03T19:44:21.302014+0000","last_became_peered":"2022-04-03T19:44:21.302014+0000","last_unstale":"2022-04-03T19:44:21.312944+0000","last_undegraded":"2022-04-03T19:44:21.312944+0000","last_fullsized":"2022-04-03T19:44:21.312944+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:15:58.860701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.285641+0000","last_change":"2022-04-03T19:44:14.754385+0000","last_active":"2022-04-03T19:44:21.285641+0000","last_peered":"2022-04-03T19:44:21.285641+0000","last_clean":"2022-04-03T19:44:21.285641+0000","last_became_active":"2022-04-03T19:44:14.754003+0000","last_became_peered":"2022-04-03T19:44:14.754003+0000","last_unstale":"2022-04-03T19:44:21.285641+0000","last_undegraded":"2022-04-03T19:44:21.285641+0000","last_fullsized":"2022-04-03T19:44:21.285641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:05:22.716933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":166,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286290+0000","last_change":"2022-04-03T19:44:08.726758+0000","last_active":"2022-04-03T19:44:21.286290+0000","last_peered":"2022-04-03T19:44:21.286290+0000","last_clean":"2022-04-03T19:44:21.286290+0000","last_became_active":"2022-04-03T19:44:08.725694+0000","last_became_peered":"2022-04-03T19:44:08.725694+0000","last_unstale":"2022-04-03T19:44:21.286290+0000","last_undegraded":"2022-04-03T19:44:21.286290+0000","last_fullsized":"2022-04-03T19:44:21.286290+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:07.708129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:07.708129+0000","last_clean_scrub_stamp":"2022-04-03T19:44:07.708129+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:27:43.354267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.282879+0000","last_change":"2022-04-03T19:44:20.781585+0000","last_active":"2022-04-03T19:44:21.282879+0000","last_peered":"2022-04-03T19:44:21.282879+0000","last_clean":"2022-04-03T19:44:21.282879+0000","last_became_active":"2022-04-03T19:44:20.781274+0000","last_became_peered":"2022-04-03T19:44:20.781274+0000","last_unstale":"2022-04-03T19:44:21.282879+0000","last_undegraded":"2022-04-03T19:44:21.282879+0000","last_fullsized":"2022-04-03T19:44:21.282879+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:49:54.368001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.301298+0000","last_change":"2022-04-03T19:44:21.296620+0000","last_active":"2022-04-03T19:44:21.301298+0000","last_peered":"2022-04-03T19:44:21.301298+0000","last_clean":"2022-04-03T19:44:21.301298+0000","last_became_active":"2022-04-03T19:44:21.296314+0000","last_became_peered":"2022-04-03T19:44:21.296314+0000","last_unstale":"2022-04-03T19:44:21.301298+0000","last_undegraded":"2022-04-03T19:44:21.301298+0000","last_fullsized":"2022-04-03T19:44:21.301298+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:09:18.338683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286626+0000","last_change":"2022-04-03T19:44:14.754664+0000","last_active":"2022-04-03T19:44:21.286626+0000","last_peered":"2022-04-03T19:44:21.286626+0000","last_clean":"2022-04-03T19:44:21.286626+0000","last_became_active":"2022-04-03T19:44:14.754328+0000","last_became_peered":"2022-04-03T19:44:14.754328+0000","last_unstale":"2022-04-03T19:44:21.286626+0000","last_undegraded":"2022-04-03T19:44:21.286626+0000","last_fullsized":"2022-04-03T19:44:21.286626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:39:58.022353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.281559+0000","last_change":"2022-04-03T19:44:20.787271+0000","last_active":"2022-04-03T19:44:21.281559+0000","last_peered":"2022-04-03T19:44:21.281559+0000","last_clean":"2022-04-03T19:44:21.281559+0000","last_became_active":"2022-04-03T19:44:20.786957+0000","last_became_peered":"2022-04-03T19:44:20.786957+0000","last_unstale":"2022-04-03T19:44:21.281559+0000","last_undegraded":"2022-04-03T19:44:21.281559+0000","last_fullsized":"2022-04-03T19:44:21.281559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:21:45.992744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.311227+0000","last_change":"2022-04-03T19:44:21.306857+0000","last_active":"2022-04-03T19:44:21.311227+0000","last_peered":"2022-04-03T19:44:21.311227+0000","last_clean":"2022-04-03T19:44:21.311227+0000","last_became_active":"2022-04-03T19:44:21.306521+0000","last_became_peered":"2022-04-03T19:44:21.306521+0000","last_unstale":"2022-04-03T19:44:21.311227+0000","last_undegraded":"2022-04-03T19:44:21.311227+0000","last_fullsized":"2022-04-03T19:44:21.311227+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:50:03.912834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.287006+0000","last_change":"2022-04-03T19:44:14.754657+0000","last_active":"2022-04-03T19:44:21.287006+0000","last_peered":"2022-04-03T19:44:21.287006+0000","last_clean":"2022-04-03T19:44:21.287006+0000","last_became_active":"2022-04-03T19:44:14.754310+0000","last_became_peered":"2022-04-03T19:44:14.754310+0000","last_unstale":"2022-04-03T19:44:21.287006+0000","last_undegraded":"2022-04-03T19:44:21.287006+0000","last_fullsized":"2022-04-03T19:44:21.287006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:42:05.279424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.297553+0000","last_change":"2022-04-03T19:44:20.794454+0000","last_active":"2022-04-03T19:44:21.297553+0000","last_peered":"2022-04-03T19:44:21.297553+0000","last_clean":"2022-04-03T19:44:21.297553+0000","last_became_active":"2022-04-03T19:44:20.789468+0000","last_became_peered":"2022-04-03T19:44:20.789468+0000","last_unstale":"2022-04-03T19:44:21.297553+0000","last_undegraded":"2022-04-03T19:44:21.297553+0000","last_fullsized":"2022-04-03T19:44:21.297553+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:02:49.184838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.308286+0000","last_change":"2022-04-03T19:44:21.303669+0000","last_active":"2022-04-03T19:44:21.308286+0000","last_peered":"2022-04-03T19:44:21.308286+0000","last_clean":"2022-04-03T19:44:21.308286+0000","last_became_active":"2022-04-03T19:44:21.303381+0000","last_became_peered":"2022-04-03T19:44:21.303381+0000","last_unstale":"2022-04-03T19:44:21.308286+0000","last_undegraded":"2022-04-03T19:44:21.308286+0000","last_fullsized":"2022-04-03T19:44:21.308286+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:43:22.320070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.289548+0000","last_change":"2022-04-03T19:44:20.791594+0000","last_active":"2022-04-03T19:44:26.289548+0000","last_peered":"2022-04-03T19:44:26.289548+0000","last_clean":"2022-04-03T19:44:26.289548+0000","last_became_active":"2022-04-03T19:44:20.789693+0000","last_became_peered":"2022-04-03T19:44:20.789693+0000","last_unstale":"2022-04-03T19:44:26.289548+0000","last_undegraded":"2022-04-03T19:44:26.289548+0000","last_fullsized":"2022-04-03T19:44:26.289548+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:18:21.832066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.297820+0000","last_change":"2022-04-03T19:44:14.755096+0000","last_active":"2022-04-03T19:44:21.297820+0000","last_peered":"2022-04-03T19:44:21.297820+0000","last_clean":"2022-04-03T19:44:21.297820+0000","last_became_active":"2022-04-03T19:44:14.754799+0000","last_became_peered":"2022-04-03T19:44:14.754799+0000","last_unstale":"2022-04-03T19:44:21.297820+0000","last_undegraded":"2022-04-03T19:44:21.297820+0000","last_fullsized":"2022-04-03T19:44:21.297820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:04:17.179114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.309672+0000","last_change":"2022-04-03T19:44:21.299649+0000","last_active":"2022-04-03T19:44:21.309672+0000","last_peered":"2022-04-03T19:44:21.309672+0000","last_clean":"2022-04-03T19:44:21.309672+0000","last_became_active":"2022-04-03T19:44:21.299399+0000","last_became_peered":"2022-04-03T19:44:21.299399+0000","last_unstale":"2022-04-03T19:44:21.309672+0000","last_undegraded":"2022-04-03T19:44:21.309672+0000","last_fullsized":"2022-04-03T19:44:21.309672+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:13:06.313729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.288717+0000","last_change":"2022-04-03T19:44:14.754659+0000","last_active":"2022-04-03T19:44:21.288717+0000","last_peered":"2022-04-03T19:44:21.288717+0000","last_clean":"2022-04-03T19:44:21.288717+0000","last_became_active":"2022-04-03T19:44:14.754386+0000","last_became_peered":"2022-04-03T19:44:14.754386+0000","last_unstale":"2022-04-03T19:44:21.288717+0000","last_undegraded":"2022-04-03T19:44:21.288717+0000","last_fullsized":"2022-04-03T19:44:21.288717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:33:33.013536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.285800+0000","last_change":"2022-04-03T19:44:20.784840+0000","last_active":"2022-04-03T19:44:22.285800+0000","last_peered":"2022-04-03T19:44:22.285800+0000","last_clean":"2022-04-03T19:44:22.285800+0000","last_became_active":"2022-04-03T19:44:20.784521+0000","last_became_peered":"2022-04-03T19:44:20.784521+0000","last_unstale":"2022-04-03T19:44:22.285800+0000","last_undegraded":"2022-04-03T19:44:22.285800+0000","last_fullsized":"2022-04-03T19:44:22.285800+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:15:38.240804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293467+0000","last_change":"2022-04-03T19:44:21.290769+0000","last_active":"2022-04-03T19:44:21.293467+0000","last_peered":"2022-04-03T19:44:21.293467+0000","last_clean":"2022-04-03T19:44:21.293467+0000","last_became_active":"2022-04-03T19:44:21.290394+0000","last_became_peered":"2022-04-03T19:44:21.290394+0000","last_unstale":"2022-04-03T19:44:21.293467+0000","last_undegraded":"2022-04-03T19:44:21.293467+0000","last_fullsized":"2022-04-03T19:44:21.293467+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:57:39.528871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.285708+0000","last_change":"2022-04-03T19:44:20.789662+0000","last_active":"2022-04-03T19:44:22.285708+0000","last_peered":"2022-04-03T19:44:22.285708+0000","last_clean":"2022-04-03T19:44:22.285708+0000","last_became_active":"2022-04-03T19:44:20.787868+0000","last_became_peered":"2022-04-03T19:44:20.787868+0000","last_unstale":"2022-04-03T19:44:22.285708+0000","last_undegraded":"2022-04-03T19:44:22.285708+0000","last_fullsized":"2022-04-03T19:44:22.285708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:13:16.092180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305589+0000","last_change":"2022-04-03T19:44:21.304539+0000","last_active":"2022-04-03T19:44:21.305589+0000","last_peered":"2022-04-03T19:44:21.305589+0000","last_clean":"2022-04-03T19:44:21.305589+0000","last_became_active":"2022-04-03T19:44:21.304008+0000","last_became_peered":"2022-04-03T19:44:21.304008+0000","last_unstale":"2022-04-03T19:44:21.305589+0000","last_undegraded":"2022-04-03T19:44:21.305589+0000","last_fullsized":"2022-04-03T19:44:21.305589+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:45:01.346479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.292841+0000","last_change":"2022-04-03T19:44:20.786813+0000","last_active":"2022-04-03T19:44:26.292841+0000","last_peered":"2022-04-03T19:44:26.292841+0000","last_clean":"2022-04-03T19:44:26.292841+0000","last_became_active":"2022-04-03T19:44:20.786548+0000","last_became_peered":"2022-04-03T19:44:20.786548+0000","last_unstale":"2022-04-03T19:44:26.292841+0000","last_undegraded":"2022-04-03T19:44:26.292841+0000","last_fullsized":"2022-04-03T19:44:26.292841+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:36:40.611720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302733+0000","last_change":"2022-04-03T19:44:21.299513+0000","last_active":"2022-04-03T19:44:21.302733+0000","last_peered":"2022-04-03T19:44:21.302733+0000","last_clean":"2022-04-03T19:44:21.302733+0000","last_became_active":"2022-04-03T19:44:21.299270+0000","last_became_peered":"2022-04-03T19:44:21.299270+0000","last_unstale":"2022-04-03T19:44:21.302733+0000","last_undegraded":"2022-04-03T19:44:21.302733+0000","last_fullsized":"2022-04-03T19:44:21.302733+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:54:16.302815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.290239+0000","last_change":"2022-04-03T19:44:20.786750+0000","last_active":"2022-04-03T19:44:26.290239+0000","last_peered":"2022-04-03T19:44:26.290239+0000","last_clean":"2022-04-03T19:44:26.290239+0000","last_became_active":"2022-04-03T19:44:20.786490+0000","last_became_peered":"2022-04-03T19:44:20.786490+0000","last_unstale":"2022-04-03T19:44:26.290239+0000","last_undegraded":"2022-04-03T19:44:26.290239+0000","last_fullsized":"2022-04-03T19:44:26.290239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:30:31.980806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300228+0000","last_change":"2022-04-03T19:44:21.294849+0000","last_active":"2022-04-03T19:44:21.300228+0000","last_peered":"2022-04-03T19:44:21.300228+0000","last_clean":"2022-04-03T19:44:21.300228+0000","last_became_active":"2022-04-03T19:44:21.294586+0000","last_became_peered":"2022-04-03T19:44:21.294586+0000","last_unstale":"2022-04-03T19:44:21.300228+0000","last_undegraded":"2022-04-03T19:44:21.300228+0000","last_fullsized":"2022-04-03T19:44:21.300228+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:24:11.050193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.289167+0000","last_change":"2022-04-03T19:44:20.781619+0000","last_active":"2022-04-03T19:44:22.289167+0000","last_peered":"2022-04-03T19:44:22.289167+0000","last_clean":"2022-04-03T19:44:22.289167+0000","last_became_active":"2022-04-03T19:44:20.781288+0000","last_became_peered":"2022-04-03T19:44:20.781288+0000","last_unstale":"2022-04-03T19:44:22.289167+0000","last_undegraded":"2022-04-03T19:44:22.289167+0000","last_fullsized":"2022-04-03T19:44:22.289167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:05:56.179441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305553+0000","last_change":"2022-04-03T19:44:21.296332+0000","last_active":"2022-04-03T19:44:21.305553+0000","last_peered":"2022-04-03T19:44:21.305553+0000","last_clean":"2022-04-03T19:44:21.305553+0000","last_became_active":"2022-04-03T19:44:21.296026+0000","last_became_peered":"2022-04-03T19:44:21.296026+0000","last_unstale":"2022-04-03T19:44:21.305553+0000","last_undegraded":"2022-04-03T19:44:21.305553+0000","last_fullsized":"2022-04-03T19:44:21.305553+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:43:33.822187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"24'4","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.293148+0000","last_change":"2022-04-03T19:44:20.789448+0000","last_active":"2022-04-03T19:44:26.293148+0000","last_peered":"2022-04-03T19:44:26.293148+0000","last_clean":"2022-04-03T19:44:26.293148+0000","last_became_active":"2022-04-03T19:44:20.789131+0000","last_became_peered":"2022-04-03T19:44:20.789131+0000","last_unstale":"2022-04-03T19:44:26.293148+0000","last_undegraded":"2022-04-03T19:44:26.293148+0000","last_fullsized":"2022-04-03T19:44:26.293148+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:46:20.830676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293972+0000","last_change":"2022-04-03T19:44:21.291681+0000","last_active":"2022-04-03T19:44:21.293972+0000","last_peered":"2022-04-03T19:44:21.293972+0000","last_clean":"2022-04-03T19:44:21.293972+0000","last_became_active":"2022-04-03T19:44:21.290817+0000","last_became_peered":"2022-04-03T19:44:21.290817+0000","last_unstale":"2022-04-03T19:44:21.293972+0000","last_undegraded":"2022-04-03T19:44:21.293972+0000","last_fullsized":"2022-04-03T19:44:21.293972+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:37:30.833818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300290+0000","last_change":"2022-04-03T19:44:21.295604+0000","last_active":"2022-04-03T19:44:21.300290+0000","last_peered":"2022-04-03T19:44:21.300290+0000","last_clean":"2022-04-03T19:44:21.300290+0000","last_became_active":"2022-04-03T19:44:21.295297+0000","last_became_peered":"2022-04-03T19:44:21.295297+0000","last_unstale":"2022-04-03T19:44:21.300290+0000","last_undegraded":"2022-04-03T19:44:21.300290+0000","last_fullsized":"2022-04-03T19:44:21.300290+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:26:34.341290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.309922+0000","last_change":"2022-04-03T19:44:21.306547+0000","last_active":"2022-04-03T19:44:21.309922+0000","last_peered":"2022-04-03T19:44:21.309922+0000","last_clean":"2022-04-03T19:44:21.309922+0000","last_became_active":"2022-04-03T19:44:21.306179+0000","last_became_peered":"2022-04-03T19:44:21.306179+0000","last_unstale":"2022-04-03T19:44:21.309922+0000","last_undegraded":"2022-04-03T19:44:21.309922+0000","last_fullsized":"2022-04-03T19:44:21.309922+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:34:58.854671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293642+0000","last_change":"2022-04-03T19:44:21.291341+0000","last_active":"2022-04-03T19:44:21.293642+0000","last_peered":"2022-04-03T19:44:21.293642+0000","last_clean":"2022-04-03T19:44:21.293642+0000","last_became_active":"2022-04-03T19:44:21.290943+0000","last_became_peered":"2022-04-03T19:44:21.290943+0000","last_unstale":"2022-04-03T19:44:21.293642+0000","last_undegraded":"2022-04-03T19:44:21.293642+0000","last_fullsized":"2022-04-03T19:44:21.293642+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:50:56.877454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313543+0000","last_change":"2022-04-03T19:44:21.307815+0000","last_active":"2022-04-03T19:44:21.313543+0000","last_peered":"2022-04-03T19:44:21.313543+0000","last_clean":"2022-04-03T19:44:21.313543+0000","last_became_active":"2022-04-03T19:44:21.307459+0000","last_became_peered":"2022-04-03T19:44:21.307459+0000","last_unstale":"2022-04-03T19:44:21.313543+0000","last_undegraded":"2022-04-03T19:44:21.313543+0000","last_fullsized":"2022-04-03T19:44:21.313543+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:16:52.723409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.312899+0000","last_change":"2022-04-03T19:44:21.303095+0000","last_active":"2022-04-03T19:44:21.312899+0000","last_peered":"2022-04-03T19:44:21.312899+0000","last_clean":"2022-04-03T19:44:21.312899+0000","last_became_active":"2022-04-03T19:44:21.302363+0000","last_became_peered":"2022-04-03T19:44:21.302363+0000","last_unstale":"2022-04-03T19:44:21.312899+0000","last_undegraded":"2022-04-03T19:44:21.312899+0000","last_fullsized":"2022-04-03T19:44:21.312899+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:42:45.617661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2594,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8972,"kb_used_data":772,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94362868,"statfs":{"total":96636764160,"available":96627576832,"internally_reserved":0,"allocated":790528,"data_stored":623468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]}]},{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8996,"kb_used_data":796,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94362844,"statfs":{"total":96636764160,"available":96627552256,"internally_reserved":0,"allocated":815104,"data_stored":624930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7936,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363904,"statfs":{"total":96636764160,"available":96628637696,"internally_reserved":0,"allocated":253952,"data_stored":93974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8000,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363840,"statfs":{"total":96636764160,"available":96628572160,"internally_reserved":0,"allocated":253952,"data_stored":93974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8004,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363836,"statfs":{"total":96636764160,"available":96628568064,"internally_reserved":0,"allocated":258048,"data_stored":94064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8020,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363820,"statfs":{"total":96636764160,"available":96628551680,"internally_reserved":0,"allocated":274432,"data_stored":95842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7944,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363896,"statfs":{"total":96636764160,"available":96628629504,"internally_reserved":0,"allocated":262144,"data_stored":94610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7936,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363904,"statfs":{"total":96636764160,"available":96628637696,"internally_reserved":0,"allocated":253952,"data_stored":93974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-03T19:44:46.526 INFO:tasks.mds_thrash.ceph_manager:clean! 2022-04-03T19:44:46.527 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-04-03T19:44:46.723 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.723+0000 7fc9a7fff700 1 Processor -- start 2022-04-03T19:44:46.724 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.723+0000 7fc9a7fff700 1 -- start start 2022-04-03T19:44:46.724 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.724+0000 7fc9a7fff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc9a806d3f0 0x7fc9a806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:46.725 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.724+0000 7fc9a7fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc9a80c2a90 0x7fc9a806c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:46.725 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.724+0000 7fc9a7fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a81562f0 con 0x7fc9a80c5930 2022-04-03T19:44:46.725 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.724+0000 7fc9a7fff700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9a80c2f80 con 0x7fc9a806d3f0 2022-04-03T19:44:46.725 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.724+0000 7fc9a7fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a80c3260 con 0x7fc9a80c2a90 2022-04-03T19:44:46.728 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.725+0000 7fc9a67fc700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc9a80c5930 0x7fc9a8155be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38364/0 (socket says 172.21.15.77:38364) 2022-04-03T19:44:46.728 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.725+0000 7fc9a67fc700 1 -- 172.21.15.77:0/3468778249 learned_addr learned my addr 172.21.15.77:0/3468778249 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:46.728 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.725+0000 7fc9a6ffd700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc9a806d3f0 0x7fc9a806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:46.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.725+0000 7fc9a77fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc9a80c2a90 0x7fc9a806c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:46.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.725+0000 7fc9a6ffd700 1 -- 172.21.15.77:0/3468778249 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc9a80c2a90 msgr2=0x7fc9a806c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:46.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.725+0000 7fc9a6ffd700 1 --2- 172.21.15.77:0/3468778249 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc9a80c2a90 0x7fc9a806c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.725+0000 7fc9a6ffd700 1 -- 172.21.15.77:0/3468778249 >> v1:172.21.15.77:6789/0 conn(0x7fc9a80c5930 legacy=0x7fc9a8155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:46.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.725+0000 7fc9a6ffd700 1 -- 172.21.15.77:0/3468778249 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a80c3540 con 0x7fc9a806d3f0 2022-04-03T19:44:46.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.725+0000 7fc9a5ffb700 1 -- 172.21.15.77:0/3468778249 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676034156 0 0) 0x7fc9a81562f0 con 0x7fc9a80c5930 2022-04-03T19:44:46.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.725+0000 7fc9a6ffd700 1 --2- 172.21.15.77:0/3468778249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9a806d3f0 0x7fc9a806d7c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc998009790 tx=0x7fc998009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c58aedc431d761c6 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:46.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.726+0000 7fc9a5ffb700 1 -- 172.21.15.77:0/3468778249 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc998011630 con 0x7fc9a806d3f0 2022-04-03T19:44:46.731 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.726+0000 7fc9a5ffb700 1 -- 172.21.15.77:0/3468778249 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc998011790 con 0x7fc9a806d3f0 2022-04-03T19:44:46.731 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.726+0000 7fc9a5ffb700 1 -- 172.21.15.77:0/3468778249 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc998011b50 con 0x7fc9a806d3f0 2022-04-03T19:44:46.731 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.726+0000 7fc9a7fff700 1 -- 172.21.15.77:0/3468778249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9a806d3f0 msgr2=0x7fc9a806d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:46.731 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.726+0000 7fc9a7fff700 1 --2- 172.21.15.77:0/3468778249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9a806d3f0 0x7fc9a806d7c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc998009790 tx=0x7fc998009060 comp rx=0 tx=0).stop 2022-04-03T19:44:46.732 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.726+0000 7fc9a7fff700 1 -- 172.21.15.77:0/3468778249 shutdown_connections 2022-04-03T19:44:46.732 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.726+0000 7fc9a7fff700 1 --2- 172.21.15.77:0/3468778249 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc9a80c2a90 0x7fc9a806c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.732 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.726+0000 7fc9a7fff700 1 --2- 172.21.15.77:0/3468778249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9a806d3f0 0x7fc9a806d7c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.732 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.726+0000 7fc9a7fff700 1 -- 172.21.15.77:0/3468778249 >> 172.21.15.77:0/3468778249 conn(0x7fc9a80cb720 msgr2=0x7fc9a80cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:46.732 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.726+0000 7fc9a7fff700 1 -- 172.21.15.77:0/3468778249 shutdown_connections 2022-04-03T19:44:46.733 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.726+0000 7fc9a7fff700 1 -- 172.21.15.77:0/3468778249 wait complete. 2022-04-03T19:44:46.733 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.728+0000 7fc9a7fff700 1 Processor -- start 2022-04-03T19:44:46.733 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.728+0000 7fc9a7fff700 1 -- start start 2022-04-03T19:44:46.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.729+0000 7fc9a7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9a806d3f0 0x7fc9a81652b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:46.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.729+0000 7fc9a7fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc9a80c2a90 0x7fc9a8166790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:46.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.729+0000 7fc9a7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9a80c5930 0x7fc9a8166cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:46.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.729+0000 7fc9a7fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a8152440 con 0x7fc9a80c5930 2022-04-03T19:44:46.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.729+0000 7fc9a7fff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a8151550 con 0x7fc9a806d3f0 2022-04-03T19:44:46.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.729+0000 7fc9a7fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a8151830 con 0x7fc9a80c2a90 2022-04-03T19:44:46.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.729+0000 7fc9a77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9a80c5930 0x7fc9a8166cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:46.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.729+0000 7fc9a6ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9a806d3f0 0x7fc9a81652b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:46.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.729+0000 7fc9a6ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9a806d3f0 0x7fc9a81652b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37898/0 (socket says 172.21.15.77:37898) 2022-04-03T19:44:46.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.729+0000 7fc9a6ffd700 1 -- 172.21.15.77:0/2463399229 learned_addr learned my addr 172.21.15.77:0/2463399229 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:46.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.729+0000 7fc9a67fc700 1 --2- 172.21.15.77:0/2463399229 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc9a80c2a90 0x7fc9a8166790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:46.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.730+0000 7fc9a6ffd700 1 -- 172.21.15.77:0/2463399229 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc9a80c2a90 msgr2=0x7fc9a8166790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:46.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.730+0000 7fc9a6ffd700 1 --2- 172.21.15.77:0/2463399229 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc9a80c2a90 0x7fc9a8166790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.730+0000 7fc9a6ffd700 1 -- 172.21.15.77:0/2463399229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9a80c5930 msgr2=0x7fc9a8166cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.730+0000 7fc9a6ffd700 1 --2- 172.21.15.77:0/2463399229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9a80c5930 0x7fc9a8166cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.730+0000 7fc9a6ffd700 1 -- 172.21.15.77:0/2463399229 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9980092a0 con 0x7fc9a806d3f0 2022-04-03T19:44:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.730+0000 7fc9a77fe700 1 --2- 172.21.15.77:0/2463399229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9a80c5930 0x7fc9a8166cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:46.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.730+0000 7fc9a6ffd700 1 --2- 172.21.15.77:0/2463399229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9a806d3f0 0x7fc9a81652b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc998009e30 tx=0x7fc998017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:46.740 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.730+0000 7fc98effd700 1 -- 172.21.15.77:0/2463399229 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc998000bc0 con 0x7fc9a806d3f0 2022-04-03T19:44:46.740 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.730+0000 7fc98effd700 1 -- 172.21.15.77:0/2463399229 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc998011900 con 0x7fc9a806d3f0 2022-04-03T19:44:46.740 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.730+0000 7fc98effd700 1 -- 172.21.15.77:0/2463399229 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc998025420 con 0x7fc9a806d3f0 2022-04-03T19:44:46.740 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.730+0000 7fc9a7fff700 1 -- 172.21.15.77:0/2463399229 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a8151b10 con 0x7fc9a806d3f0 2022-04-03T19:44:46.740 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.730+0000 7fc9a7fff700 1 -- 172.21.15.77:0/2463399229 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a81520d0 con 0x7fc9a806d3f0 2022-04-03T19:44:46.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.731+0000 7fc9a7fff700 1 -- 172.21.15.77:0/2463399229 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a8153f40 con 0x7fc9a806d3f0 2022-04-03T19:44:46.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.732+0000 7fc98effd700 1 -- 172.21.15.77:0/2463399229 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fc99802c030 con 0x7fc9a806d3f0 2022-04-03T19:44:46.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.733+0000 7fc98effd700 1 --2- 172.21.15.77:0/2463399229 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc988063ce0 0x7fc988066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:46.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.733+0000 7fc98effd700 1 -- 172.21.15.77:0/2463399229 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fc9980823c0 con 0x7fc9a806d3f0 2022-04-03T19:44:46.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.733+0000 7fc9a67fc700 1 --2- 172.21.15.77:0/2463399229 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc988063ce0 0x7fc988066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:46.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.734+0000 7fc9a67fc700 1 --2- 172.21.15.77:0/2463399229 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc988063ce0 0x7fc988066190 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc9a80c27c0 tx=0x7fc99c007cd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:46.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.737+0000 7fc98effd700 1 -- 172.21.15.77:0/2463399229 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fc998050ed0 con 0x7fc9a806d3f0 2022-04-03T19:44:46.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.926+0000 7fc9a7fff700 1 -- 172.21.15.77:0/2463399229 --> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc9a804b050 con 0x7fc988063ce0 2022-04-03T19:44:46.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.935+0000 7fc98effd700 1 -- 172.21.15.77:0/2463399229 <== mgr.4106 v2:172.21.15.186:6832/24948 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284674 (secure 0 0 0) 0x7fc9a804b050 con 0x7fc988063ce0 2022-04-03T19:44:46.939 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:46.942 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.939+0000 7fc9a7fff700 1 -- 172.21.15.77:0/2463399229 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc988063ce0 msgr2=0x7fc988066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:46.943 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.939+0000 7fc9a7fff700 1 --2- 172.21.15.77:0/2463399229 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc988063ce0 0x7fc988066190 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc9a80c27c0 tx=0x7fc99c007cd0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.943 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.939+0000 7fc9a7fff700 1 -- 172.21.15.77:0/2463399229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9a806d3f0 msgr2=0x7fc9a81652b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:46.943 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.939+0000 7fc9a7fff700 1 --2- 172.21.15.77:0/2463399229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9a806d3f0 0x7fc9a81652b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc998009e30 tx=0x7fc998017000 comp rx=0 tx=0).stop 2022-04-03T19:44:46.943 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.939+0000 7fc9a7fff700 1 -- 172.21.15.77:0/2463399229 shutdown_connections 2022-04-03T19:44:46.944 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.939+0000 7fc9a7fff700 1 --2- 172.21.15.77:0/2463399229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9a80c5930 0x7fc9a8166cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.944 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.939+0000 7fc9a7fff700 1 --2- 172.21.15.77:0/2463399229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9a806d3f0 0x7fc9a81652b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.944 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.939+0000 7fc9a7fff700 1 --2- 172.21.15.77:0/2463399229 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc988063ce0 0x7fc988066190 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.944 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.939+0000 7fc9a7fff700 1 --2- 172.21.15.77:0/2463399229 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc9a80c2a90 0x7fc9a8166790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:46.944 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.939+0000 7fc9a7fff700 1 -- 172.21.15.77:0/2463399229 >> 172.21.15.77:0/2463399229 conn(0x7fc9a80cb720 msgr2=0x7fc9a814d650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:46.945 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.939+0000 7fc9a7fff700 1 -- 172.21.15.77:0/2463399229 shutdown_connections 2022-04-03T19:44:46.945 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:46.939+0000 7fc9a7fff700 1 -- 172.21.15.77:0/2463399229 wait complete. 2022-04-03T19:44:46.945 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2022-04-03T19:44:46.964 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2022-04-03T19:44:45.494643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592962,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":205,"num_write_kb":2431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":65808,"kb_used_data":3088,"kb_used_omap":0,"kb_used_meta":62656,"kb_avail":754908912,"statfs":{"total":773094113280,"available":773026725888,"internally_reserved":0,"allocated":3162112,"data_stored":1814836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64159744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":264,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003270"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302418+0000","last_change":"2022-04-03T19:44:21.300867+0000","last_active":"2022-04-03T19:44:21.302418+0000","last_peered":"2022-04-03T19:44:21.302418+0000","last_clean":"2022-04-03T19:44:21.302418+0000","last_became_active":"2022-04-03T19:44:21.300569+0000","last_became_peered":"2022-04-03T19:44:21.300569+0000","last_unstale":"2022-04-03T19:44:21.302418+0000","last_undegraded":"2022-04-03T19:44:21.302418+0000","last_fullsized":"2022-04-03T19:44:21.302418+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:10:32.827314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305911+0000","last_change":"2022-04-03T19:44:21.302410+0000","last_active":"2022-04-03T19:44:21.305911+0000","last_peered":"2022-04-03T19:44:21.305911+0000","last_clean":"2022-04-03T19:44:21.305911+0000","last_became_active":"2022-04-03T19:44:21.302142+0000","last_became_peered":"2022-04-03T19:44:21.302142+0000","last_unstale":"2022-04-03T19:44:21.305911+0000","last_undegraded":"2022-04-03T19:44:21.305911+0000","last_fullsized":"2022-04-03T19:44:21.305911+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:01:46.327996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.298511+0000","last_change":"2022-04-03T19:44:21.293080+0000","last_active":"2022-04-03T19:44:21.298511+0000","last_peered":"2022-04-03T19:44:21.298511+0000","last_clean":"2022-04-03T19:44:21.298511+0000","last_became_active":"2022-04-03T19:44:21.292762+0000","last_became_peered":"2022-04-03T19:44:21.292762+0000","last_unstale":"2022-04-03T19:44:21.298511+0000","last_undegraded":"2022-04-03T19:44:21.298511+0000","last_fullsized":"2022-04-03T19:44:21.298511+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:28:36.726593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300326+0000","last_change":"2022-04-03T19:44:21.294448+0000","last_active":"2022-04-03T19:44:21.300326+0000","last_peered":"2022-04-03T19:44:21.300326+0000","last_clean":"2022-04-03T19:44:21.300326+0000","last_became_active":"2022-04-03T19:44:21.294083+0000","last_became_peered":"2022-04-03T19:44:21.294083+0000","last_unstale":"2022-04-03T19:44:21.300326+0000","last_undegraded":"2022-04-03T19:44:21.300326+0000","last_fullsized":"2022-04-03T19:44:21.300326+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:57:02.374826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.308268+0000","last_change":"2022-04-03T19:44:21.299551+0000","last_active":"2022-04-03T19:44:21.308268+0000","last_peered":"2022-04-03T19:44:21.308268+0000","last_clean":"2022-04-03T19:44:21.308268+0000","last_became_active":"2022-04-03T19:44:21.299123+0000","last_became_peered":"2022-04-03T19:44:21.299123+0000","last_unstale":"2022-04-03T19:44:21.308268+0000","last_undegraded":"2022-04-03T19:44:21.308268+0000","last_fullsized":"2022-04-03T19:44:21.308268+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:08:24.911647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300209+0000","last_change":"2022-04-03T19:44:21.293129+0000","last_active":"2022-04-03T19:44:21.300209+0000","last_peered":"2022-04-03T19:44:21.300209+0000","last_clean":"2022-04-03T19:44:21.300209+0000","last_became_active":"2022-04-03T19:44:21.292843+0000","last_became_peered":"2022-04-03T19:44:21.292843+0000","last_unstale":"2022-04-03T19:44:21.300209+0000","last_undegraded":"2022-04-03T19:44:21.300209+0000","last_fullsized":"2022-04-03T19:44:21.300209+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:38:32.432609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302772+0000","last_change":"2022-04-03T19:44:21.299112+0000","last_active":"2022-04-03T19:44:21.302772+0000","last_peered":"2022-04-03T19:44:21.302772+0000","last_clean":"2022-04-03T19:44:21.302772+0000","last_became_active":"2022-04-03T19:44:21.298803+0000","last_became_peered":"2022-04-03T19:44:21.298803+0000","last_unstale":"2022-04-03T19:44:21.302772+0000","last_undegraded":"2022-04-03T19:44:21.302772+0000","last_fullsized":"2022-04-03T19:44:21.302772+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:29:43.577843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299160+0000","last_change":"2022-04-03T19:44:21.293638+0000","last_active":"2022-04-03T19:44:21.299160+0000","last_peered":"2022-04-03T19:44:21.299160+0000","last_clean":"2022-04-03T19:44:21.299160+0000","last_became_active":"2022-04-03T19:44:21.293359+0000","last_became_peered":"2022-04-03T19:44:21.293359+0000","last_unstale":"2022-04-03T19:44:21.299160+0000","last_undegraded":"2022-04-03T19:44:21.299160+0000","last_fullsized":"2022-04-03T19:44:21.299160+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:28:50.113792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.306107+0000","last_change":"2022-04-03T19:44:21.299916+0000","last_active":"2022-04-03T19:44:21.306107+0000","last_peered":"2022-04-03T19:44:21.306107+0000","last_clean":"2022-04-03T19:44:21.306107+0000","last_became_active":"2022-04-03T19:44:21.299578+0000","last_became_peered":"2022-04-03T19:44:21.299578+0000","last_unstale":"2022-04-03T19:44:21.306107+0000","last_undegraded":"2022-04-03T19:44:21.306107+0000","last_fullsized":"2022-04-03T19:44:21.306107+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:38:40.281496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.298833+0000","last_change":"2022-04-03T19:44:21.294003+0000","last_active":"2022-04-03T19:44:21.298833+0000","last_peered":"2022-04-03T19:44:21.298833+0000","last_clean":"2022-04-03T19:44:21.298833+0000","last_became_active":"2022-04-03T19:44:21.293640+0000","last_became_peered":"2022-04-03T19:44:21.293640+0000","last_unstale":"2022-04-03T19:44:21.298833+0000","last_undegraded":"2022-04-03T19:44:21.298833+0000","last_fullsized":"2022-04-03T19:44:21.298833+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:40:13.436484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302745+0000","last_change":"2022-04-03T19:44:21.301158+0000","last_active":"2022-04-03T19:44:21.302745+0000","last_peered":"2022-04-03T19:44:21.302745+0000","last_clean":"2022-04-03T19:44:21.302745+0000","last_became_active":"2022-04-03T19:44:21.300823+0000","last_became_peered":"2022-04-03T19:44:21.300823+0000","last_unstale":"2022-04-03T19:44:21.302745+0000","last_undegraded":"2022-04-03T19:44:21.302745+0000","last_fullsized":"2022-04-03T19:44:21.302745+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:09:28.365828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299985+0000","last_change":"2022-04-03T19:44:21.295301+0000","last_active":"2022-04-03T19:44:21.299985+0000","last_peered":"2022-04-03T19:44:21.299985+0000","last_clean":"2022-04-03T19:44:21.299985+0000","last_became_active":"2022-04-03T19:44:21.295032+0000","last_became_peered":"2022-04-03T19:44:21.295032+0000","last_unstale":"2022-04-03T19:44:21.299985+0000","last_undegraded":"2022-04-03T19:44:21.299985+0000","last_fullsized":"2022-04-03T19:44:21.299985+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:55:55.637194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.301541+0000","last_change":"2022-04-03T19:44:21.296250+0000","last_active":"2022-04-03T19:44:21.301541+0000","last_peered":"2022-04-03T19:44:21.301541+0000","last_clean":"2022-04-03T19:44:21.301541+0000","last_became_active":"2022-04-03T19:44:21.295967+0000","last_became_peered":"2022-04-03T19:44:21.295967+0000","last_unstale":"2022-04-03T19:44:21.301541+0000","last_undegraded":"2022-04-03T19:44:21.301541+0000","last_fullsized":"2022-04-03T19:44:21.301541+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:43:34.301454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299466+0000","last_change":"2022-04-03T19:44:21.297229+0000","last_active":"2022-04-03T19:44:21.299466+0000","last_peered":"2022-04-03T19:44:21.299466+0000","last_clean":"2022-04-03T19:44:21.299466+0000","last_became_active":"2022-04-03T19:44:21.296993+0000","last_became_peered":"2022-04-03T19:44:21.296993+0000","last_unstale":"2022-04-03T19:44:21.299466+0000","last_undegraded":"2022-04-03T19:44:21.299466+0000","last_fullsized":"2022-04-03T19:44:21.299466+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:27:17.280580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303252+0000","last_change":"2022-04-03T19:44:21.302427+0000","last_active":"2022-04-03T19:44:21.303252+0000","last_peered":"2022-04-03T19:44:21.303252+0000","last_clean":"2022-04-03T19:44:21.303252+0000","last_became_active":"2022-04-03T19:44:21.302157+0000","last_became_peered":"2022-04-03T19:44:21.302157+0000","last_unstale":"2022-04-03T19:44:21.303252+0000","last_undegraded":"2022-04-03T19:44:21.303252+0000","last_fullsized":"2022-04-03T19:44:21.303252+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:57:23.292801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.298518+0000","last_change":"2022-04-03T19:44:21.293096+0000","last_active":"2022-04-03T19:44:21.298518+0000","last_peered":"2022-04-03T19:44:21.298518+0000","last_clean":"2022-04-03T19:44:21.298518+0000","last_became_active":"2022-04-03T19:44:21.292785+0000","last_became_peered":"2022-04-03T19:44:21.292785+0000","last_unstale":"2022-04-03T19:44:21.298518+0000","last_undegraded":"2022-04-03T19:44:21.298518+0000","last_fullsized":"2022-04-03T19:44:21.298518+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:27:06.759308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.306127+0000","last_change":"2022-04-03T19:44:21.300957+0000","last_active":"2022-04-03T19:44:21.306127+0000","last_peered":"2022-04-03T19:44:21.306127+0000","last_clean":"2022-04-03T19:44:21.306127+0000","last_became_active":"2022-04-03T19:44:21.300342+0000","last_became_peered":"2022-04-03T19:44:21.300342+0000","last_unstale":"2022-04-03T19:44:21.306127+0000","last_undegraded":"2022-04-03T19:44:21.306127+0000","last_fullsized":"2022-04-03T19:44:21.306127+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:43:53.194230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302142+0000","last_change":"2022-04-03T19:44:21.295996+0000","last_active":"2022-04-03T19:44:21.302142+0000","last_peered":"2022-04-03T19:44:21.302142+0000","last_clean":"2022-04-03T19:44:21.302142+0000","last_became_active":"2022-04-03T19:44:21.295766+0000","last_became_peered":"2022-04-03T19:44:21.295766+0000","last_unstale":"2022-04-03T19:44:21.302142+0000","last_undegraded":"2022-04-03T19:44:21.302142+0000","last_fullsized":"2022-04-03T19:44:21.302142+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:21:31.628478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302310+0000","last_change":"2022-04-03T19:44:21.299840+0000","last_active":"2022-04-03T19:44:21.302310+0000","last_peered":"2022-04-03T19:44:21.302310+0000","last_clean":"2022-04-03T19:44:21.302310+0000","last_became_active":"2022-04-03T19:44:21.299487+0000","last_became_peered":"2022-04-03T19:44:21.299487+0000","last_unstale":"2022-04-03T19:44:21.302310+0000","last_undegraded":"2022-04-03T19:44:21.302310+0000","last_fullsized":"2022-04-03T19:44:21.302310+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:24:39.583019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.295324+0000","last_change":"2022-04-03T19:44:21.291748+0000","last_active":"2022-04-03T19:44:21.295324+0000","last_peered":"2022-04-03T19:44:21.295324+0000","last_clean":"2022-04-03T19:44:21.295324+0000","last_became_active":"2022-04-03T19:44:21.290939+0000","last_became_peered":"2022-04-03T19:44:21.290939+0000","last_unstale":"2022-04-03T19:44:21.295324+0000","last_undegraded":"2022-04-03T19:44:21.295324+0000","last_fullsized":"2022-04-03T19:44:21.295324+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:28:45.710805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302521+0000","last_change":"2022-04-03T19:44:21.299442+0000","last_active":"2022-04-03T19:44:21.302521+0000","last_peered":"2022-04-03T19:44:21.302521+0000","last_clean":"2022-04-03T19:44:21.302521+0000","last_became_active":"2022-04-03T19:44:21.299166+0000","last_became_peered":"2022-04-03T19:44:21.299166+0000","last_unstale":"2022-04-03T19:44:21.302521+0000","last_undegraded":"2022-04-03T19:44:21.302521+0000","last_fullsized":"2022-04-03T19:44:21.302521+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:31:30.905517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.311290+0000","last_change":"2022-04-03T19:44:21.307007+0000","last_active":"2022-04-03T19:44:21.311290+0000","last_peered":"2022-04-03T19:44:21.311290+0000","last_clean":"2022-04-03T19:44:21.311290+0000","last_became_active":"2022-04-03T19:44:21.306650+0000","last_became_peered":"2022-04-03T19:44:21.306650+0000","last_unstale":"2022-04-03T19:44:21.311290+0000","last_undegraded":"2022-04-03T19:44:21.311290+0000","last_fullsized":"2022-04-03T19:44:21.311290+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:03:26.043624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299849+0000","last_change":"2022-04-03T19:44:21.293566+0000","last_active":"2022-04-03T19:44:21.299849+0000","last_peered":"2022-04-03T19:44:21.299849+0000","last_clean":"2022-04-03T19:44:21.299849+0000","last_became_active":"2022-04-03T19:44:21.293263+0000","last_became_peered":"2022-04-03T19:44:21.293263+0000","last_unstale":"2022-04-03T19:44:21.299849+0000","last_undegraded":"2022-04-03T19:44:21.299849+0000","last_fullsized":"2022-04-03T19:44:21.299849+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:39:52.523944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302610+0000","last_change":"2022-04-03T19:44:21.300329+0000","last_active":"2022-04-03T19:44:21.302610+0000","last_peered":"2022-04-03T19:44:21.302610+0000","last_clean":"2022-04-03T19:44:21.302610+0000","last_became_active":"2022-04-03T19:44:21.300000+0000","last_became_peered":"2022-04-03T19:44:21.300000+0000","last_unstale":"2022-04-03T19:44:21.302610+0000","last_undegraded":"2022-04-03T19:44:21.302610+0000","last_fullsized":"2022-04-03T19:44:21.302610+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:15:10.839571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.312922+0000","last_change":"2022-04-03T19:44:21.302506+0000","last_active":"2022-04-03T19:44:21.312922+0000","last_peered":"2022-04-03T19:44:21.312922+0000","last_clean":"2022-04-03T19:44:21.312922+0000","last_became_active":"2022-04-03T19:44:21.302235+0000","last_became_peered":"2022-04-03T19:44:21.302235+0000","last_unstale":"2022-04-03T19:44:21.312922+0000","last_undegraded":"2022-04-03T19:44:21.312922+0000","last_fullsized":"2022-04-03T19:44:21.312922+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:25:25.801754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313258+0000","last_change":"2022-04-03T19:44:21.306670+0000","last_active":"2022-04-03T19:44:21.313258+0000","last_peered":"2022-04-03T19:44:21.313258+0000","last_clean":"2022-04-03T19:44:21.313258+0000","last_became_active":"2022-04-03T19:44:21.305722+0000","last_became_peered":"2022-04-03T19:44:21.305722+0000","last_unstale":"2022-04-03T19:44:21.313258+0000","last_undegraded":"2022-04-03T19:44:21.313258+0000","last_fullsized":"2022-04-03T19:44:21.313258+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:44:09.515207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303081+0000","last_change":"2022-04-03T19:44:21.300024+0000","last_active":"2022-04-03T19:44:21.303081+0000","last_peered":"2022-04-03T19:44:21.303081+0000","last_clean":"2022-04-03T19:44:21.303081+0000","last_became_active":"2022-04-03T19:44:21.299753+0000","last_became_peered":"2022-04-03T19:44:21.299753+0000","last_unstale":"2022-04-03T19:44:21.303081+0000","last_undegraded":"2022-04-03T19:44:21.303081+0000","last_fullsized":"2022-04-03T19:44:21.303081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:15:31.808623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.390304+0000","last_change":"2022-04-03T19:44:20.783898+0000","last_active":"2022-04-03T19:44:22.390304+0000","last_peered":"2022-04-03T19:44:22.390304+0000","last_clean":"2022-04-03T19:44:22.390304+0000","last_became_active":"2022-04-03T19:44:20.783612+0000","last_became_peered":"2022-04-03T19:44:20.783612+0000","last_unstale":"2022-04-03T19:44:22.390304+0000","last_undegraded":"2022-04-03T19:44:22.390304+0000","last_fullsized":"2022-04-03T19:44:22.390304+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:13:32.863927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.310259+0000","last_change":"2022-04-03T19:44:21.306904+0000","last_active":"2022-04-03T19:44:21.310259+0000","last_peered":"2022-04-03T19:44:21.310259+0000","last_clean":"2022-04-03T19:44:21.310259+0000","last_became_active":"2022-04-03T19:44:21.306616+0000","last_became_peered":"2022-04-03T19:44:21.306616+0000","last_unstale":"2022-04-03T19:44:21.310259+0000","last_undegraded":"2022-04-03T19:44:21.310259+0000","last_fullsized":"2022-04-03T19:44:21.310259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:49:35.783912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.296341+0000","last_change":"2022-04-03T19:44:20.788125+0000","last_active":"2022-04-03T19:44:21.296341+0000","last_peered":"2022-04-03T19:44:21.296341+0000","last_clean":"2022-04-03T19:44:21.296341+0000","last_became_active":"2022-04-03T19:44:20.787792+0000","last_became_peered":"2022-04-03T19:44:20.787792+0000","last_unstale":"2022-04-03T19:44:21.296341+0000","last_undegraded":"2022-04-03T19:44:21.296341+0000","last_fullsized":"2022-04-03T19:44:21.296341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:11:51.126774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305572+0000","last_change":"2022-04-03T19:44:21.302904+0000","last_active":"2022-04-03T19:44:21.305572+0000","last_peered":"2022-04-03T19:44:21.305572+0000","last_clean":"2022-04-03T19:44:21.305572+0000","last_became_active":"2022-04-03T19:44:21.301564+0000","last_became_peered":"2022-04-03T19:44:21.301564+0000","last_unstale":"2022-04-03T19:44:21.305572+0000","last_undegraded":"2022-04-03T19:44:21.305572+0000","last_fullsized":"2022-04-03T19:44:21.305572+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:25:51.611789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.283649+0000","last_change":"2022-04-03T19:44:20.792529+0000","last_active":"2022-04-03T19:44:21.283649+0000","last_peered":"2022-04-03T19:44:21.283649+0000","last_clean":"2022-04-03T19:44:21.283649+0000","last_became_active":"2022-04-03T19:44:20.792014+0000","last_became_peered":"2022-04-03T19:44:20.792014+0000","last_unstale":"2022-04-03T19:44:21.283649+0000","last_undegraded":"2022-04-03T19:44:21.283649+0000","last_fullsized":"2022-04-03T19:44:21.283649+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:01:38.319693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302637+0000","last_change":"2022-04-03T19:44:21.299302+0000","last_active":"2022-04-03T19:44:21.302637+0000","last_peered":"2022-04-03T19:44:21.302637+0000","last_clean":"2022-04-03T19:44:21.302637+0000","last_became_active":"2022-04-03T19:44:21.299007+0000","last_became_peered":"2022-04-03T19:44:21.299007+0000","last_unstale":"2022-04-03T19:44:21.302637+0000","last_undegraded":"2022-04-03T19:44:21.302637+0000","last_fullsized":"2022-04-03T19:44:21.302637+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:59:48.767335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.282232+0000","last_change":"2022-04-03T19:44:20.790672+0000","last_active":"2022-04-03T19:44:21.282232+0000","last_peered":"2022-04-03T19:44:21.282232+0000","last_clean":"2022-04-03T19:44:21.282232+0000","last_became_active":"2022-04-03T19:44:20.788526+0000","last_became_peered":"2022-04-03T19:44:20.788526+0000","last_unstale":"2022-04-03T19:44:21.282232+0000","last_undegraded":"2022-04-03T19:44:21.282232+0000","last_fullsized":"2022-04-03T19:44:21.282232+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:14:50.058917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.309960+0000","last_change":"2022-04-03T19:44:21.306472+0000","last_active":"2022-04-03T19:44:21.309960+0000","last_peered":"2022-04-03T19:44:21.309960+0000","last_clean":"2022-04-03T19:44:21.309960+0000","last_became_active":"2022-04-03T19:44:21.306071+0000","last_became_peered":"2022-04-03T19:44:21.306071+0000","last_unstale":"2022-04-03T19:44:21.309960+0000","last_undegraded":"2022-04-03T19:44:21.309960+0000","last_fullsized":"2022-04-03T19:44:21.309960+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:43:22.221014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286837+0000","last_change":"2022-04-03T19:44:20.785638+0000","last_active":"2022-04-03T19:44:21.286837+0000","last_peered":"2022-04-03T19:44:21.286837+0000","last_clean":"2022-04-03T19:44:21.286837+0000","last_became_active":"2022-04-03T19:44:20.784963+0000","last_became_peered":"2022-04-03T19:44:20.784963+0000","last_unstale":"2022-04-03T19:44:21.286837+0000","last_undegraded":"2022-04-03T19:44:21.286837+0000","last_fullsized":"2022-04-03T19:44:21.286837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:39:26.933298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.309328+0000","last_change":"2022-04-03T19:44:21.299583+0000","last_active":"2022-04-03T19:44:21.309328+0000","last_peered":"2022-04-03T19:44:21.309328+0000","last_clean":"2022-04-03T19:44:21.309328+0000","last_became_active":"2022-04-03T19:44:21.299301+0000","last_became_peered":"2022-04-03T19:44:21.299301+0000","last_unstale":"2022-04-03T19:44:21.309328+0000","last_undegraded":"2022-04-03T19:44:21.309328+0000","last_fullsized":"2022-04-03T19:44:21.309328+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:37:49.567741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"24'3","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.290849+0000","last_change":"2022-04-03T19:44:20.795890+0000","last_active":"2022-04-03T19:44:26.290849+0000","last_peered":"2022-04-03T19:44:26.290849+0000","last_clean":"2022-04-03T19:44:26.290849+0000","last_became_active":"2022-04-03T19:44:20.795611+0000","last_became_peered":"2022-04-03T19:44:20.795611+0000","last_unstale":"2022-04-03T19:44:26.290849+0000","last_undegraded":"2022-04-03T19:44:26.290849+0000","last_fullsized":"2022-04-03T19:44:26.290849+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:08:33.216934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313173+0000","last_change":"2022-04-03T19:44:21.304724+0000","last_active":"2022-04-03T19:44:21.313173+0000","last_peered":"2022-04-03T19:44:21.313173+0000","last_clean":"2022-04-03T19:44:21.313173+0000","last_became_active":"2022-04-03T19:44:21.302708+0000","last_became_peered":"2022-04-03T19:44:21.302708+0000","last_unstale":"2022-04-03T19:44:21.313173+0000","last_undegraded":"2022-04-03T19:44:21.313173+0000","last_fullsized":"2022-04-03T19:44:21.313173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:04:54.970100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"24'7","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.294405+0000","last_change":"2022-04-03T19:44:20.785659+0000","last_active":"2022-04-03T19:44:26.294405+0000","last_peered":"2022-04-03T19:44:26.294405+0000","last_clean":"2022-04-03T19:44:26.294405+0000","last_became_active":"2022-04-03T19:44:20.785308+0000","last_became_peered":"2022-04-03T19:44:20.785308+0000","last_unstale":"2022-04-03T19:44:26.294405+0000","last_undegraded":"2022-04-03T19:44:26.294405+0000","last_fullsized":"2022-04-03T19:44:26.294405+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:52:08.268532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1812,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293514+0000","last_change":"2022-04-03T19:44:21.291033+0000","last_active":"2022-04-03T19:44:21.293514+0000","last_peered":"2022-04-03T19:44:21.293514+0000","last_clean":"2022-04-03T19:44:21.293514+0000","last_became_active":"2022-04-03T19:44:21.290713+0000","last_became_peered":"2022-04-03T19:44:21.290713+0000","last_unstale":"2022-04-03T19:44:21.293514+0000","last_undegraded":"2022-04-03T19:44:21.293514+0000","last_fullsized":"2022-04-03T19:44:21.293514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:19:41.873591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.290200+0000","last_change":"2022-04-03T19:44:20.787511+0000","last_active":"2022-04-03T19:44:26.290200+0000","last_peered":"2022-04-03T19:44:26.290200+0000","last_clean":"2022-04-03T19:44:26.290200+0000","last_became_active":"2022-04-03T19:44:20.787189+0000","last_became_peered":"2022-04-03T19:44:20.787189+0000","last_unstale":"2022-04-03T19:44:26.290200+0000","last_undegraded":"2022-04-03T19:44:26.290200+0000","last_fullsized":"2022-04-03T19:44:26.290200+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:11:56.297263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313518+0000","last_change":"2022-04-03T19:44:21.307846+0000","last_active":"2022-04-03T19:44:21.313518+0000","last_peered":"2022-04-03T19:44:21.313518+0000","last_clean":"2022-04-03T19:44:21.313518+0000","last_became_active":"2022-04-03T19:44:21.307513+0000","last_became_peered":"2022-04-03T19:44:21.307513+0000","last_unstale":"2022-04-03T19:44:21.313518+0000","last_undegraded":"2022-04-03T19:44:21.313518+0000","last_fullsized":"2022-04-03T19:44:21.313518+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:58:09.091062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.281910+0000","last_change":"2022-04-03T19:44:20.790212+0000","last_active":"2022-04-03T19:44:21.281910+0000","last_peered":"2022-04-03T19:44:21.281910+0000","last_clean":"2022-04-03T19:44:21.281910+0000","last_became_active":"2022-04-03T19:44:20.789863+0000","last_became_peered":"2022-04-03T19:44:20.789863+0000","last_unstale":"2022-04-03T19:44:21.281910+0000","last_undegraded":"2022-04-03T19:44:21.281910+0000","last_fullsized":"2022-04-03T19:44:21.281910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:48:13.489309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303061+0000","last_change":"2022-04-03T19:44:21.299953+0000","last_active":"2022-04-03T19:44:21.303061+0000","last_peered":"2022-04-03T19:44:21.303061+0000","last_clean":"2022-04-03T19:44:21.303061+0000","last_became_active":"2022-04-03T19:44:21.299667+0000","last_became_peered":"2022-04-03T19:44:21.299667+0000","last_unstale":"2022-04-03T19:44:21.303061+0000","last_undegraded":"2022-04-03T19:44:21.303061+0000","last_fullsized":"2022-04-03T19:44:21.303061+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:51:16.125160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286253+0000","last_change":"2022-04-03T19:44:20.791054+0000","last_active":"2022-04-03T19:44:21.286253+0000","last_peered":"2022-04-03T19:44:21.286253+0000","last_clean":"2022-04-03T19:44:21.286253+0000","last_became_active":"2022-04-03T19:44:20.790664+0000","last_became_peered":"2022-04-03T19:44:20.790664+0000","last_unstale":"2022-04-03T19:44:21.286253+0000","last_undegraded":"2022-04-03T19:44:21.286253+0000","last_fullsized":"2022-04-03T19:44:21.286253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:07:27.501728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.304407+0000","last_change":"2022-04-03T19:44:21.295028+0000","last_active":"2022-04-03T19:44:21.304407+0000","last_peered":"2022-04-03T19:44:21.304407+0000","last_clean":"2022-04-03T19:44:21.304407+0000","last_became_active":"2022-04-03T19:44:21.294723+0000","last_became_peered":"2022-04-03T19:44:21.294723+0000","last_unstale":"2022-04-03T19:44:21.304407+0000","last_undegraded":"2022-04-03T19:44:21.304407+0000","last_fullsized":"2022-04-03T19:44:21.304407+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:32:45.455842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"24'8","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.294387+0000","last_change":"2022-04-03T19:44:20.792133+0000","last_active":"2022-04-03T19:44:26.294387+0000","last_peered":"2022-04-03T19:44:26.294387+0000","last_clean":"2022-04-03T19:44:26.294387+0000","last_became_active":"2022-04-03T19:44:20.791829+0000","last_became_peered":"2022-04-03T19:44:20.791829+0000","last_unstale":"2022-04-03T19:44:26.294387+0000","last_undegraded":"2022-04-03T19:44:26.294387+0000","last_fullsized":"2022-04-03T19:44:26.294387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:01:21.966734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.310014+0000","last_change":"2022-04-03T19:44:21.300267+0000","last_active":"2022-04-03T19:44:21.310014+0000","last_peered":"2022-04-03T19:44:21.310014+0000","last_clean":"2022-04-03T19:44:21.310014+0000","last_became_active":"2022-04-03T19:44:21.299948+0000","last_became_peered":"2022-04-03T19:44:21.299948+0000","last_unstale":"2022-04-03T19:44:21.310014+0000","last_undegraded":"2022-04-03T19:44:21.310014+0000","last_fullsized":"2022-04-03T19:44:21.310014+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:06:54.481379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.391229+0000","last_change":"2022-04-03T19:44:20.789793+0000","last_active":"2022-04-03T19:44:22.391229+0000","last_peered":"2022-04-03T19:44:22.391229+0000","last_clean":"2022-04-03T19:44:22.391229+0000","last_became_active":"2022-04-03T19:44:20.789371+0000","last_became_peered":"2022-04-03T19:44:20.789371+0000","last_unstale":"2022-04-03T19:44:22.391229+0000","last_undegraded":"2022-04-03T19:44:22.391229+0000","last_fullsized":"2022-04-03T19:44:22.391229+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:15:52.060059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.301800+0000","last_change":"2022-04-03T19:44:21.298477+0000","last_active":"2022-04-03T19:44:21.301800+0000","last_peered":"2022-04-03T19:44:21.301800+0000","last_clean":"2022-04-03T19:44:21.301800+0000","last_became_active":"2022-04-03T19:44:21.298165+0000","last_became_peered":"2022-04-03T19:44:21.298165+0000","last_unstale":"2022-04-03T19:44:21.301800+0000","last_undegraded":"2022-04-03T19:44:21.301800+0000","last_fullsized":"2022-04-03T19:44:21.301800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:23:11.042589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.290125+0000","last_change":"2022-04-03T19:44:20.787949+0000","last_active":"2022-04-03T19:44:26.290125+0000","last_peered":"2022-04-03T19:44:26.290125+0000","last_clean":"2022-04-03T19:44:26.290125+0000","last_became_active":"2022-04-03T19:44:20.787592+0000","last_became_peered":"2022-04-03T19:44:20.787592+0000","last_unstale":"2022-04-03T19:44:26.290125+0000","last_undegraded":"2022-04-03T19:44:26.290125+0000","last_fullsized":"2022-04-03T19:44:26.290125+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:50:09.411246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302765+0000","last_change":"2022-04-03T19:44:21.301200+0000","last_active":"2022-04-03T19:44:21.302765+0000","last_peered":"2022-04-03T19:44:21.302765+0000","last_clean":"2022-04-03T19:44:21.302765+0000","last_became_active":"2022-04-03T19:44:21.300878+0000","last_became_peered":"2022-04-03T19:44:21.300878+0000","last_unstale":"2022-04-03T19:44:21.302765+0000","last_undegraded":"2022-04-03T19:44:21.302765+0000","last_fullsized":"2022-04-03T19:44:21.302765+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:54:36.089411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.391128+0000","last_change":"2022-04-03T19:44:20.783711+0000","last_active":"2022-04-03T19:44:22.391128+0000","last_peered":"2022-04-03T19:44:22.391128+0000","last_clean":"2022-04-03T19:44:22.391128+0000","last_became_active":"2022-04-03T19:44:20.783423+0000","last_became_peered":"2022-04-03T19:44:20.783423+0000","last_unstale":"2022-04-03T19:44:22.391128+0000","last_undegraded":"2022-04-03T19:44:22.391128+0000","last_fullsized":"2022-04-03T19:44:22.391128+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:22:37.665229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.303123+0000","last_change":"2022-04-03T19:44:21.300815+0000","last_active":"2022-04-03T19:44:21.303123+0000","last_peered":"2022-04-03T19:44:21.303123+0000","last_clean":"2022-04-03T19:44:21.303123+0000","last_became_active":"2022-04-03T19:44:21.300545+0000","last_became_peered":"2022-04-03T19:44:21.300545+0000","last_unstale":"2022-04-03T19:44:21.303123+0000","last_undegraded":"2022-04-03T19:44:21.303123+0000","last_fullsized":"2022-04-03T19:44:21.303123+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:33:13.941191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.285374+0000","last_change":"2022-04-03T19:44:20.785480+0000","last_active":"2022-04-03T19:44:21.285374+0000","last_peered":"2022-04-03T19:44:21.285374+0000","last_clean":"2022-04-03T19:44:21.285374+0000","last_became_active":"2022-04-03T19:44:20.785178+0000","last_became_peered":"2022-04-03T19:44:20.785178+0000","last_unstale":"2022-04-03T19:44:21.285374+0000","last_undegraded":"2022-04-03T19:44:21.285374+0000","last_fullsized":"2022-04-03T19:44:21.285374+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:15:37.181452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.308247+0000","last_change":"2022-04-03T19:44:21.296509+0000","last_active":"2022-04-03T19:44:21.308247+0000","last_peered":"2022-04-03T19:44:21.308247+0000","last_clean":"2022-04-03T19:44:21.308247+0000","last_became_active":"2022-04-03T19:44:21.296240+0000","last_became_peered":"2022-04-03T19:44:21.296240+0000","last_unstale":"2022-04-03T19:44:21.308247+0000","last_undegraded":"2022-04-03T19:44:21.308247+0000","last_fullsized":"2022-04-03T19:44:21.308247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:50:45.955195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.288396+0000","last_change":"2022-04-03T19:44:20.792550+0000","last_active":"2022-04-03T19:44:26.288396+0000","last_peered":"2022-04-03T19:44:26.288396+0000","last_clean":"2022-04-03T19:44:26.288396+0000","last_became_active":"2022-04-03T19:44:20.792191+0000","last_became_peered":"2022-04-03T19:44:20.792191+0000","last_unstale":"2022-04-03T19:44:26.288396+0000","last_undegraded":"2022-04-03T19:44:26.288396+0000","last_fullsized":"2022-04-03T19:44:26.288396+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:51:30.297728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.299612+0000","last_change":"2022-04-03T19:44:21.294898+0000","last_active":"2022-04-03T19:44:21.299612+0000","last_peered":"2022-04-03T19:44:21.299612+0000","last_clean":"2022-04-03T19:44:21.299612+0000","last_became_active":"2022-04-03T19:44:21.294635+0000","last_became_peered":"2022-04-03T19:44:21.294635+0000","last_unstale":"2022-04-03T19:44:21.299612+0000","last_undegraded":"2022-04-03T19:44:21.299612+0000","last_fullsized":"2022-04-03T19:44:21.299612+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:42:52.629552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.289106+0000","last_change":"2022-04-03T19:44:20.783711+0000","last_active":"2022-04-03T19:44:21.289106+0000","last_peered":"2022-04-03T19:44:21.289106+0000","last_clean":"2022-04-03T19:44:21.289106+0000","last_became_active":"2022-04-03T19:44:20.783426+0000","last_became_peered":"2022-04-03T19:44:20.783426+0000","last_unstale":"2022-04-03T19:44:21.289106+0000","last_undegraded":"2022-04-03T19:44:21.289106+0000","last_fullsized":"2022-04-03T19:44:21.289106+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:25:26.219558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302205+0000","last_change":"2022-04-03T19:44:21.296081+0000","last_active":"2022-04-03T19:44:21.302205+0000","last_peered":"2022-04-03T19:44:21.302205+0000","last_clean":"2022-04-03T19:44:21.302205+0000","last_became_active":"2022-04-03T19:44:21.295784+0000","last_became_peered":"2022-04-03T19:44:21.295784+0000","last_unstale":"2022-04-03T19:44:21.302205+0000","last_undegraded":"2022-04-03T19:44:21.302205+0000","last_fullsized":"2022-04-03T19:44:21.302205+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:23:40.568168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.283775+0000","last_change":"2022-04-03T19:44:20.789868+0000","last_active":"2022-04-03T19:44:21.283775+0000","last_peered":"2022-04-03T19:44:21.283775+0000","last_clean":"2022-04-03T19:44:21.283775+0000","last_became_active":"2022-04-03T19:44:20.789563+0000","last_became_peered":"2022-04-03T19:44:20.789563+0000","last_unstale":"2022-04-03T19:44:21.283775+0000","last_undegraded":"2022-04-03T19:44:21.283775+0000","last_fullsized":"2022-04-03T19:44:21.283775+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:04:22.814932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293542+0000","last_change":"2022-04-03T19:44:21.291212+0000","last_active":"2022-04-03T19:44:21.293542+0000","last_peered":"2022-04-03T19:44:21.293542+0000","last_clean":"2022-04-03T19:44:21.293542+0000","last_became_active":"2022-04-03T19:44:21.290888+0000","last_became_peered":"2022-04-03T19:44:21.290888+0000","last_unstale":"2022-04-03T19:44:21.293542+0000","last_undegraded":"2022-04-03T19:44:21.293542+0000","last_fullsized":"2022-04-03T19:44:21.293542+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:29:34.850679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.281344+0000","last_change":"2022-04-03T19:44:20.789319+0000","last_active":"2022-04-03T19:44:21.281344+0000","last_peered":"2022-04-03T19:44:21.281344+0000","last_clean":"2022-04-03T19:44:21.281344+0000","last_became_active":"2022-04-03T19:44:20.788840+0000","last_became_peered":"2022-04-03T19:44:20.788840+0000","last_unstale":"2022-04-03T19:44:21.281344+0000","last_undegraded":"2022-04-03T19:44:21.281344+0000","last_fullsized":"2022-04-03T19:44:21.281344+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:50:21.536926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313655+0000","last_change":"2022-04-03T19:44:21.311886+0000","last_active":"2022-04-03T19:44:21.313655+0000","last_peered":"2022-04-03T19:44:21.313655+0000","last_clean":"2022-04-03T19:44:21.313655+0000","last_became_active":"2022-04-03T19:44:21.311521+0000","last_became_peered":"2022-04-03T19:44:21.311521+0000","last_unstale":"2022-04-03T19:44:21.313655+0000","last_undegraded":"2022-04-03T19:44:21.313655+0000","last_fullsized":"2022-04-03T19:44:21.313655+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T07:25:47.761166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.297426+0000","last_change":"2022-04-03T19:44:14.754674+0000","last_active":"2022-04-03T19:44:21.297426+0000","last_peered":"2022-04-03T19:44:21.297426+0000","last_clean":"2022-04-03T19:44:21.297426+0000","last_became_active":"2022-04-03T19:44:14.754210+0000","last_became_peered":"2022-04-03T19:44:14.754210+0000","last_unstale":"2022-04-03T19:44:21.297426+0000","last_undegraded":"2022-04-03T19:44:21.297426+0000","last_fullsized":"2022-04-03T19:44:21.297426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:23:14.624494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286308+0000","last_change":"2022-04-03T19:44:20.785400+0000","last_active":"2022-04-03T19:44:21.286308+0000","last_peered":"2022-04-03T19:44:21.286308+0000","last_clean":"2022-04-03T19:44:21.286308+0000","last_became_active":"2022-04-03T19:44:20.785125+0000","last_became_peered":"2022-04-03T19:44:20.785125+0000","last_unstale":"2022-04-03T19:44:21.286308+0000","last_undegraded":"2022-04-03T19:44:21.286308+0000","last_fullsized":"2022-04-03T19:44:21.286308+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:24:02.690646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313197+0000","last_change":"2022-04-03T19:44:21.305321+0000","last_active":"2022-04-03T19:44:21.313197+0000","last_peered":"2022-04-03T19:44:21.313197+0000","last_clean":"2022-04-03T19:44:21.313197+0000","last_became_active":"2022-04-03T19:44:21.304017+0000","last_became_peered":"2022-04-03T19:44:21.304017+0000","last_unstale":"2022-04-03T19:44:21.313197+0000","last_undegraded":"2022-04-03T19:44:21.313197+0000","last_fullsized":"2022-04-03T19:44:21.313197+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:59:12.619058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.285445+0000","last_change":"2022-04-03T19:44:14.755580+0000","last_active":"2022-04-03T19:44:21.285445+0000","last_peered":"2022-04-03T19:44:21.285445+0000","last_clean":"2022-04-03T19:44:21.285445+0000","last_became_active":"2022-04-03T19:44:14.755222+0000","last_became_peered":"2022-04-03T19:44:14.755222+0000","last_unstale":"2022-04-03T19:44:21.285445+0000","last_undegraded":"2022-04-03T19:44:21.285445+0000","last_fullsized":"2022-04-03T19:44:21.285445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:08:18.314326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.283116+0000","last_change":"2022-04-03T19:44:20.791540+0000","last_active":"2022-04-03T19:44:21.283116+0000","last_peered":"2022-04-03T19:44:21.283116+0000","last_clean":"2022-04-03T19:44:21.283116+0000","last_became_active":"2022-04-03T19:44:20.791192+0000","last_became_peered":"2022-04-03T19:44:20.791192+0000","last_unstale":"2022-04-03T19:44:21.283116+0000","last_undegraded":"2022-04-03T19:44:21.283116+0000","last_fullsized":"2022-04-03T19:44:21.283116+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T22:58:59.674606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313218+0000","last_change":"2022-04-03T19:44:21.305930+0000","last_active":"2022-04-03T19:44:21.313218+0000","last_peered":"2022-04-03T19:44:21.313218+0000","last_clean":"2022-04-03T19:44:21.313218+0000","last_became_active":"2022-04-03T19:44:21.305659+0000","last_became_peered":"2022-04-03T19:44:21.305659+0000","last_unstale":"2022-04-03T19:44:21.313218+0000","last_undegraded":"2022-04-03T19:44:21.313218+0000","last_fullsized":"2022-04-03T19:44:21.313218+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:14:34.958569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.290570+0000","last_change":"2022-04-03T19:44:14.754598+0000","last_active":"2022-04-03T19:44:21.290570+0000","last_peered":"2022-04-03T19:44:21.290570+0000","last_clean":"2022-04-03T19:44:21.290570+0000","last_became_active":"2022-04-03T19:44:14.754287+0000","last_became_peered":"2022-04-03T19:44:14.754287+0000","last_unstale":"2022-04-03T19:44:21.290570+0000","last_undegraded":"2022-04-03T19:44:21.290570+0000","last_fullsized":"2022-04-03T19:44:21.290570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:32:16.232689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.296054+0000","last_change":"2022-04-03T19:44:20.789567+0000","last_active":"2022-04-03T19:44:21.296054+0000","last_peered":"2022-04-03T19:44:21.296054+0000","last_clean":"2022-04-03T19:44:21.296054+0000","last_became_active":"2022-04-03T19:44:20.788664+0000","last_became_peered":"2022-04-03T19:44:20.788664+0000","last_unstale":"2022-04-03T19:44:21.296054+0000","last_undegraded":"2022-04-03T19:44:21.296054+0000","last_fullsized":"2022-04-03T19:44:21.296054+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:15:37.098494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.312944+0000","last_change":"2022-04-03T19:44:21.303079+0000","last_active":"2022-04-03T19:44:21.312944+0000","last_peered":"2022-04-03T19:44:21.312944+0000","last_clean":"2022-04-03T19:44:21.312944+0000","last_became_active":"2022-04-03T19:44:21.302014+0000","last_became_peered":"2022-04-03T19:44:21.302014+0000","last_unstale":"2022-04-03T19:44:21.312944+0000","last_undegraded":"2022-04-03T19:44:21.312944+0000","last_fullsized":"2022-04-03T19:44:21.312944+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:15:58.860701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.285641+0000","last_change":"2022-04-03T19:44:14.754385+0000","last_active":"2022-04-03T19:44:21.285641+0000","last_peered":"2022-04-03T19:44:21.285641+0000","last_clean":"2022-04-03T19:44:21.285641+0000","last_became_active":"2022-04-03T19:44:14.754003+0000","last_became_peered":"2022-04-03T19:44:14.754003+0000","last_unstale":"2022-04-03T19:44:21.285641+0000","last_undegraded":"2022-04-03T19:44:21.285641+0000","last_fullsized":"2022-04-03T19:44:21.285641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:05:22.716933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":166,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286290+0000","last_change":"2022-04-03T19:44:08.726758+0000","last_active":"2022-04-03T19:44:21.286290+0000","last_peered":"2022-04-03T19:44:21.286290+0000","last_clean":"2022-04-03T19:44:21.286290+0000","last_became_active":"2022-04-03T19:44:08.725694+0000","last_became_peered":"2022-04-03T19:44:08.725694+0000","last_unstale":"2022-04-03T19:44:21.286290+0000","last_undegraded":"2022-04-03T19:44:21.286290+0000","last_fullsized":"2022-04-03T19:44:21.286290+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:07.708129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:07.708129+0000","last_clean_scrub_stamp":"2022-04-03T19:44:07.708129+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:27:43.354267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.282879+0000","last_change":"2022-04-03T19:44:20.781585+0000","last_active":"2022-04-03T19:44:21.282879+0000","last_peered":"2022-04-03T19:44:21.282879+0000","last_clean":"2022-04-03T19:44:21.282879+0000","last_became_active":"2022-04-03T19:44:20.781274+0000","last_became_peered":"2022-04-03T19:44:20.781274+0000","last_unstale":"2022-04-03T19:44:21.282879+0000","last_undegraded":"2022-04-03T19:44:21.282879+0000","last_fullsized":"2022-04-03T19:44:21.282879+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:49:54.368001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.301298+0000","last_change":"2022-04-03T19:44:21.296620+0000","last_active":"2022-04-03T19:44:21.301298+0000","last_peered":"2022-04-03T19:44:21.301298+0000","last_clean":"2022-04-03T19:44:21.301298+0000","last_became_active":"2022-04-03T19:44:21.296314+0000","last_became_peered":"2022-04-03T19:44:21.296314+0000","last_unstale":"2022-04-03T19:44:21.301298+0000","last_undegraded":"2022-04-03T19:44:21.301298+0000","last_fullsized":"2022-04-03T19:44:21.301298+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:09:18.338683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.286626+0000","last_change":"2022-04-03T19:44:14.754664+0000","last_active":"2022-04-03T19:44:21.286626+0000","last_peered":"2022-04-03T19:44:21.286626+0000","last_clean":"2022-04-03T19:44:21.286626+0000","last_became_active":"2022-04-03T19:44:14.754328+0000","last_became_peered":"2022-04-03T19:44:14.754328+0000","last_unstale":"2022-04-03T19:44:21.286626+0000","last_undegraded":"2022-04-03T19:44:21.286626+0000","last_fullsized":"2022-04-03T19:44:21.286626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:39:58.022353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.281559+0000","last_change":"2022-04-03T19:44:20.787271+0000","last_active":"2022-04-03T19:44:21.281559+0000","last_peered":"2022-04-03T19:44:21.281559+0000","last_clean":"2022-04-03T19:44:21.281559+0000","last_became_active":"2022-04-03T19:44:20.786957+0000","last_became_peered":"2022-04-03T19:44:20.786957+0000","last_unstale":"2022-04-03T19:44:21.281559+0000","last_undegraded":"2022-04-03T19:44:21.281559+0000","last_fullsized":"2022-04-03T19:44:21.281559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:21:45.992744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.311227+0000","last_change":"2022-04-03T19:44:21.306857+0000","last_active":"2022-04-03T19:44:21.311227+0000","last_peered":"2022-04-03T19:44:21.311227+0000","last_clean":"2022-04-03T19:44:21.311227+0000","last_became_active":"2022-04-03T19:44:21.306521+0000","last_became_peered":"2022-04-03T19:44:21.306521+0000","last_unstale":"2022-04-03T19:44:21.311227+0000","last_undegraded":"2022-04-03T19:44:21.311227+0000","last_fullsized":"2022-04-03T19:44:21.311227+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:50:03.912834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.287006+0000","last_change":"2022-04-03T19:44:14.754657+0000","last_active":"2022-04-03T19:44:21.287006+0000","last_peered":"2022-04-03T19:44:21.287006+0000","last_clean":"2022-04-03T19:44:21.287006+0000","last_became_active":"2022-04-03T19:44:14.754310+0000","last_became_peered":"2022-04-03T19:44:14.754310+0000","last_unstale":"2022-04-03T19:44:21.287006+0000","last_undegraded":"2022-04-03T19:44:21.287006+0000","last_fullsized":"2022-04-03T19:44:21.287006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:42:05.279424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.297553+0000","last_change":"2022-04-03T19:44:20.794454+0000","last_active":"2022-04-03T19:44:21.297553+0000","last_peered":"2022-04-03T19:44:21.297553+0000","last_clean":"2022-04-03T19:44:21.297553+0000","last_became_active":"2022-04-03T19:44:20.789468+0000","last_became_peered":"2022-04-03T19:44:20.789468+0000","last_unstale":"2022-04-03T19:44:21.297553+0000","last_undegraded":"2022-04-03T19:44:21.297553+0000","last_fullsized":"2022-04-03T19:44:21.297553+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:02:49.184838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.308286+0000","last_change":"2022-04-03T19:44:21.303669+0000","last_active":"2022-04-03T19:44:21.308286+0000","last_peered":"2022-04-03T19:44:21.308286+0000","last_clean":"2022-04-03T19:44:21.308286+0000","last_became_active":"2022-04-03T19:44:21.303381+0000","last_became_peered":"2022-04-03T19:44:21.303381+0000","last_unstale":"2022-04-03T19:44:21.308286+0000","last_undegraded":"2022-04-03T19:44:21.308286+0000","last_fullsized":"2022-04-03T19:44:21.308286+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:43:22.320070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.289548+0000","last_change":"2022-04-03T19:44:20.791594+0000","last_active":"2022-04-03T19:44:26.289548+0000","last_peered":"2022-04-03T19:44:26.289548+0000","last_clean":"2022-04-03T19:44:26.289548+0000","last_became_active":"2022-04-03T19:44:20.789693+0000","last_became_peered":"2022-04-03T19:44:20.789693+0000","last_unstale":"2022-04-03T19:44:26.289548+0000","last_undegraded":"2022-04-03T19:44:26.289548+0000","last_fullsized":"2022-04-03T19:44:26.289548+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:18:21.832066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.297820+0000","last_change":"2022-04-03T19:44:14.755096+0000","last_active":"2022-04-03T19:44:21.297820+0000","last_peered":"2022-04-03T19:44:21.297820+0000","last_clean":"2022-04-03T19:44:21.297820+0000","last_became_active":"2022-04-03T19:44:14.754799+0000","last_became_peered":"2022-04-03T19:44:14.754799+0000","last_unstale":"2022-04-03T19:44:21.297820+0000","last_undegraded":"2022-04-03T19:44:21.297820+0000","last_fullsized":"2022-04-03T19:44:21.297820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:04:17.179114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.309672+0000","last_change":"2022-04-03T19:44:21.299649+0000","last_active":"2022-04-03T19:44:21.309672+0000","last_peered":"2022-04-03T19:44:21.309672+0000","last_clean":"2022-04-03T19:44:21.309672+0000","last_became_active":"2022-04-03T19:44:21.299399+0000","last_became_peered":"2022-04-03T19:44:21.299399+0000","last_unstale":"2022-04-03T19:44:21.309672+0000","last_undegraded":"2022-04-03T19:44:21.309672+0000","last_fullsized":"2022-04-03T19:44:21.309672+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:13:06.313729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.288717+0000","last_change":"2022-04-03T19:44:14.754659+0000","last_active":"2022-04-03T19:44:21.288717+0000","last_peered":"2022-04-03T19:44:21.288717+0000","last_clean":"2022-04-03T19:44:21.288717+0000","last_became_active":"2022-04-03T19:44:14.754386+0000","last_became_peered":"2022-04-03T19:44:14.754386+0000","last_unstale":"2022-04-03T19:44:21.288717+0000","last_undegraded":"2022-04-03T19:44:21.288717+0000","last_fullsized":"2022-04-03T19:44:21.288717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:13.729521+0000","last_clean_scrub_stamp":"2022-04-03T19:44:13.729521+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:33:33.013536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.285800+0000","last_change":"2022-04-03T19:44:20.784840+0000","last_active":"2022-04-03T19:44:22.285800+0000","last_peered":"2022-04-03T19:44:22.285800+0000","last_clean":"2022-04-03T19:44:22.285800+0000","last_became_active":"2022-04-03T19:44:20.784521+0000","last_became_peered":"2022-04-03T19:44:20.784521+0000","last_unstale":"2022-04-03T19:44:22.285800+0000","last_undegraded":"2022-04-03T19:44:22.285800+0000","last_fullsized":"2022-04-03T19:44:22.285800+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T06:15:38.240804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293467+0000","last_change":"2022-04-03T19:44:21.290769+0000","last_active":"2022-04-03T19:44:21.293467+0000","last_peered":"2022-04-03T19:44:21.293467+0000","last_clean":"2022-04-03T19:44:21.293467+0000","last_became_active":"2022-04-03T19:44:21.290394+0000","last_became_peered":"2022-04-03T19:44:21.290394+0000","last_unstale":"2022-04-03T19:44:21.293467+0000","last_undegraded":"2022-04-03T19:44:21.293467+0000","last_fullsized":"2022-04-03T19:44:21.293467+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:57:39.528871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.285708+0000","last_change":"2022-04-03T19:44:20.789662+0000","last_active":"2022-04-03T19:44:22.285708+0000","last_peered":"2022-04-03T19:44:22.285708+0000","last_clean":"2022-04-03T19:44:22.285708+0000","last_became_active":"2022-04-03T19:44:20.787868+0000","last_became_peered":"2022-04-03T19:44:20.787868+0000","last_unstale":"2022-04-03T19:44:22.285708+0000","last_undegraded":"2022-04-03T19:44:22.285708+0000","last_fullsized":"2022-04-03T19:44:22.285708+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:13:16.092180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305589+0000","last_change":"2022-04-03T19:44:21.304539+0000","last_active":"2022-04-03T19:44:21.305589+0000","last_peered":"2022-04-03T19:44:21.305589+0000","last_clean":"2022-04-03T19:44:21.305589+0000","last_became_active":"2022-04-03T19:44:21.304008+0000","last_became_peered":"2022-04-03T19:44:21.304008+0000","last_unstale":"2022-04-03T19:44:21.305589+0000","last_undegraded":"2022-04-03T19:44:21.305589+0000","last_fullsized":"2022-04-03T19:44:21.305589+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:45:01.346479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.292841+0000","last_change":"2022-04-03T19:44:20.786813+0000","last_active":"2022-04-03T19:44:26.292841+0000","last_peered":"2022-04-03T19:44:26.292841+0000","last_clean":"2022-04-03T19:44:26.292841+0000","last_became_active":"2022-04-03T19:44:20.786548+0000","last_became_peered":"2022-04-03T19:44:20.786548+0000","last_unstale":"2022-04-03T19:44:26.292841+0000","last_undegraded":"2022-04-03T19:44:26.292841+0000","last_fullsized":"2022-04-03T19:44:26.292841+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T05:36:40.611720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.302733+0000","last_change":"2022-04-03T19:44:21.299513+0000","last_active":"2022-04-03T19:44:21.302733+0000","last_peered":"2022-04-03T19:44:21.302733+0000","last_clean":"2022-04-03T19:44:21.302733+0000","last_became_active":"2022-04-03T19:44:21.299270+0000","last_became_peered":"2022-04-03T19:44:21.299270+0000","last_unstale":"2022-04-03T19:44:21.302733+0000","last_undegraded":"2022-04-03T19:44:21.302733+0000","last_fullsized":"2022-04-03T19:44:21.302733+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T19:54:16.302815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"24'2","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.290239+0000","last_change":"2022-04-03T19:44:20.786750+0000","last_active":"2022-04-03T19:44:26.290239+0000","last_peered":"2022-04-03T19:44:26.290239+0000","last_clean":"2022-04-03T19:44:26.290239+0000","last_became_active":"2022-04-03T19:44:20.786490+0000","last_became_peered":"2022-04-03T19:44:20.786490+0000","last_unstale":"2022-04-03T19:44:26.290239+0000","last_undegraded":"2022-04-03T19:44:26.290239+0000","last_fullsized":"2022-04-03T19:44:26.290239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:30:31.980806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300228+0000","last_change":"2022-04-03T19:44:21.294849+0000","last_active":"2022-04-03T19:44:21.300228+0000","last_peered":"2022-04-03T19:44:21.300228+0000","last_clean":"2022-04-03T19:44:21.300228+0000","last_became_active":"2022-04-03T19:44:21.294586+0000","last_became_peered":"2022-04-03T19:44:21.294586+0000","last_unstale":"2022-04-03T19:44:21.300228+0000","last_undegraded":"2022-04-03T19:44:21.300228+0000","last_fullsized":"2022-04-03T19:44:21.300228+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T20:24:11.050193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"24'1","reported_seq":11,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:22.289167+0000","last_change":"2022-04-03T19:44:20.781619+0000","last_active":"2022-04-03T19:44:22.289167+0000","last_peered":"2022-04-03T19:44:22.289167+0000","last_clean":"2022-04-03T19:44:22.289167+0000","last_became_active":"2022-04-03T19:44:20.781288+0000","last_became_peered":"2022-04-03T19:44:20.781288+0000","last_unstale":"2022-04-03T19:44:22.289167+0000","last_undegraded":"2022-04-03T19:44:22.289167+0000","last_fullsized":"2022-04-03T19:44:22.289167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T03:05:56.179441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.305553+0000","last_change":"2022-04-03T19:44:21.296332+0000","last_active":"2022-04-03T19:44:21.305553+0000","last_peered":"2022-04-03T19:44:21.305553+0000","last_clean":"2022-04-03T19:44:21.305553+0000","last_became_active":"2022-04-03T19:44:21.296026+0000","last_became_peered":"2022-04-03T19:44:21.296026+0000","last_unstale":"2022-04-03T19:44:21.305553+0000","last_undegraded":"2022-04-03T19:44:21.305553+0000","last_fullsized":"2022-04-03T19:44:21.305553+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:43:33.822187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"24'4","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2022-04-03T19:44:26.293148+0000","last_change":"2022-04-03T19:44:20.789448+0000","last_active":"2022-04-03T19:44:26.293148+0000","last_peered":"2022-04-03T19:44:26.293148+0000","last_clean":"2022-04-03T19:44:26.293148+0000","last_became_active":"2022-04-03T19:44:20.789131+0000","last_became_peered":"2022-04-03T19:44:20.789131+0000","last_unstale":"2022-04-03T19:44:26.293148+0000","last_undegraded":"2022-04-03T19:44:26.293148+0000","last_fullsized":"2022-04-03T19:44:26.293148+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:19.756581+0000","last_clean_scrub_stamp":"2022-04-03T19:44:19.756581+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T02:46:20.830676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293972+0000","last_change":"2022-04-03T19:44:21.291681+0000","last_active":"2022-04-03T19:44:21.293972+0000","last_peered":"2022-04-03T19:44:21.293972+0000","last_clean":"2022-04-03T19:44:21.293972+0000","last_became_active":"2022-04-03T19:44:21.290817+0000","last_became_peered":"2022-04-03T19:44:21.290817+0000","last_unstale":"2022-04-03T19:44:21.293972+0000","last_undegraded":"2022-04-03T19:44:21.293972+0000","last_fullsized":"2022-04-03T19:44:21.293972+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T00:37:30.833818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.300290+0000","last_change":"2022-04-03T19:44:21.295604+0000","last_active":"2022-04-03T19:44:21.300290+0000","last_peered":"2022-04-03T19:44:21.300290+0000","last_clean":"2022-04-03T19:44:21.300290+0000","last_became_active":"2022-04-03T19:44:21.295297+0000","last_became_peered":"2022-04-03T19:44:21.295297+0000","last_unstale":"2022-04-03T19:44:21.300290+0000","last_undegraded":"2022-04-03T19:44:21.300290+0000","last_fullsized":"2022-04-03T19:44:21.300290+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:26:34.341290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.309922+0000","last_change":"2022-04-03T19:44:21.306547+0000","last_active":"2022-04-03T19:44:21.309922+0000","last_peered":"2022-04-03T19:44:21.309922+0000","last_clean":"2022-04-03T19:44:21.309922+0000","last_became_active":"2022-04-03T19:44:21.306179+0000","last_became_peered":"2022-04-03T19:44:21.306179+0000","last_unstale":"2022-04-03T19:44:21.309922+0000","last_undegraded":"2022-04-03T19:44:21.309922+0000","last_fullsized":"2022-04-03T19:44:21.309922+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T23:34:58.854671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.293642+0000","last_change":"2022-04-03T19:44:21.291341+0000","last_active":"2022-04-03T19:44:21.293642+0000","last_peered":"2022-04-03T19:44:21.293642+0000","last_clean":"2022-04-03T19:44:21.293642+0000","last_became_active":"2022-04-03T19:44:21.290943+0000","last_became_peered":"2022-04-03T19:44:21.290943+0000","last_unstale":"2022-04-03T19:44:21.293642+0000","last_undegraded":"2022-04-03T19:44:21.293642+0000","last_fullsized":"2022-04-03T19:44:21.293642+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T01:50:56.877454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.313543+0000","last_change":"2022-04-03T19:44:21.307815+0000","last_active":"2022-04-03T19:44:21.313543+0000","last_peered":"2022-04-03T19:44:21.313543+0000","last_clean":"2022-04-03T19:44:21.313543+0000","last_became_active":"2022-04-03T19:44:21.307459+0000","last_became_peered":"2022-04-03T19:44:21.307459+0000","last_unstale":"2022-04-03T19:44:21.313543+0000","last_undegraded":"2022-04-03T19:44:21.313543+0000","last_fullsized":"2022-04-03T19:44:21.313543+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-04T21:16:52.723409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2022-04-03T19:44:21.312899+0000","last_change":"2022-04-03T19:44:21.303095+0000","last_active":"2022-04-03T19:44:21.312899+0000","last_peered":"2022-04-03T19:44:21.312899+0000","last_clean":"2022-04-03T19:44:21.312899+0000","last_became_active":"2022-04-03T19:44:21.302363+0000","last_became_peered":"2022-04-03T19:44:21.302363+0000","last_unstale":"2022-04-03T19:44:21.312899+0000","last_undegraded":"2022-04-03T19:44:21.312899+0000","last_fullsized":"2022-04-03T19:44:21.312899+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-04-03T19:44:20.762513+0000","last_clean_scrub_stamp":"2022-04-03T19:44:20.762513+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-04-05T04:42:45.617661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2594,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8972,"kb_used_data":772,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94362868,"statfs":{"total":96636764160,"available":96627576832,"internally_reserved":0,"allocated":790528,"data_stored":623468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]}]},{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8996,"kb_used_data":796,"kb_used_omap":0,"kb_used_meta":8192,"kb_avail":94362844,"statfs":{"total":96636764160,"available":96627552256,"internally_reserved":0,"allocated":815104,"data_stored":624930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8388608},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7936,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363904,"statfs":{"total":96636764160,"available":96628637696,"internally_reserved":0,"allocated":253952,"data_stored":93974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8000,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363840,"statfs":{"total":96636764160,"available":96628572160,"internally_reserved":0,"allocated":253952,"data_stored":93974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8004,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363836,"statfs":{"total":96636764160,"available":96628568064,"internally_reserved":0,"allocated":258048,"data_stored":94064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8020,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":7744,"kb_avail":94363820,"statfs":{"total":96636764160,"available":96628551680,"internally_reserved":0,"allocated":274432,"data_stored":95842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7929856},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7944,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363896,"statfs":{"total":96636764160,"available":96628629504,"internally_reserved":0,"allocated":262144,"data_stored":94610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7936,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":7680,"kb_avail":94363904,"statfs":{"total":96636764160,"available":96628637696,"internally_reserved":0,"allocated":253952,"data_stored":93974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7864320},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-04-03T19:44:46.970 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-04-03T19:44:47.170 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.169+0000 7f0250fd1700 1 Processor -- start 2022-04-03T19:44:47.170 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.170+0000 7f0250fd1700 1 -- start start 2022-04-03T19:44:47.171 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.171+0000 7f0250fd1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f024c06bfb0 0x7f024c1584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:47.171 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.171+0000 7f0250fd1700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c06d260 con 0x7f024c06c500 2022-04-03T19:44:47.172 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.171+0000 7f0250fd1700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c0c6330 con 0x7f024c06bbe0 2022-04-03T19:44:47.172 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.171+0000 7f0250fd1700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f024c159450 con 0x7f024c06bfb0 2022-04-03T19:44:47.172 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.171+0000 7f024bfff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f024c06c500 0x7f024c14eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38366/0 (socket says 172.21.15.77:38366) 2022-04-03T19:44:47.173 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.171+0000 7f024bfff700 1 -- 172.21.15.77:0/2501668796 learned_addr learned my addr 172.21.15.77:0/2501668796 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:47.173 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.171+0000 7f024affd700 1 --2- 172.21.15.77:0/2501668796 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f024c06bfb0 0x7f024c1584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:47.175 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.172+0000 7f024a7fc700 1 -- 172.21.15.77:0/2501668796 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919929032 0 0) 0x7f024c0c6330 con 0x7f024c06bbe0 2022-04-03T19:44:47.175 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.172+0000 7f024a7fc700 1 -- 172.21.15.77:0/2501668796 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02380033f0 con 0x7f024c06bbe0 2022-04-03T19:44:47.176 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.172+0000 7f024a7fc700 1 -- 172.21.15.77:0/2501668796 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971274603 0 0) 0x7f024c06d260 con 0x7f024c06c500 2022-04-03T19:44:47.176 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.172+0000 7f024affd700 1 -- 172.21.15.77:0/2501668796 >> v1:172.21.15.186:6789/0 conn(0x7f024c06bbe0 legacy=0x7f024c06d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:47.176 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.172+0000 7f024affd700 1 -- 172.21.15.77:0/2501668796 >> v1:172.21.15.77:6789/0 conn(0x7f024c06c500 legacy=0x7f024c14eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:47.176 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.172+0000 7f024affd700 1 -- 172.21.15.77:0/2501668796 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024c158bb0 con 0x7f024c06bfb0 2022-04-03T19:44:47.177 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.172+0000 7f024affd700 1 --2- 172.21.15.77:0/2501668796 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f024c06bfb0 0x7f024c1584b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0240009790 tx=0x7f0240009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89b1b115bafd897f server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:47.177 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.172+0000 7f024a7fc700 1 -- 172.21.15.77:0/2501668796 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0240011630 con 0x7f024c06bfb0 2022-04-03T19:44:47.177 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.172+0000 7f024a7fc700 1 -- 172.21.15.77:0/2501668796 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0240011790 con 0x7f024c06bfb0 2022-04-03T19:44:47.177 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.172+0000 7f024a7fc700 1 -- 172.21.15.77:0/2501668796 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0240011b10 con 0x7f024c06bfb0 2022-04-03T19:44:47.178 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.173+0000 7f0250fd1700 1 -- 172.21.15.77:0/2501668796 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f024c06bfb0 msgr2=0x7f024c1584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:47.178 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.173+0000 7f0250fd1700 1 --2- 172.21.15.77:0/2501668796 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f024c06bfb0 0x7f024c1584b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0240009790 tx=0x7f0240009060 comp rx=0 tx=0).stop 2022-04-03T19:44:47.178 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.173+0000 7f0250fd1700 1 -- 172.21.15.77:0/2501668796 shutdown_connections 2022-04-03T19:44:47.178 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.173+0000 7f0250fd1700 1 --2- 172.21.15.77:0/2501668796 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f024c06bfb0 0x7f024c1584b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:47.179 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.173+0000 7f0250fd1700 1 -- 172.21.15.77:0/2501668796 >> 172.21.15.77:0/2501668796 conn(0x7f024c0cb6b0 msgr2=0x7f024c0cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:47.179 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.173+0000 7f0250fd1700 1 -- 172.21.15.77:0/2501668796 shutdown_connections 2022-04-03T19:44:47.179 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.173+0000 7f0250fd1700 1 -- 172.21.15.77:0/2501668796 wait complete. 2022-04-03T19:44:47.179 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.174+0000 7f0250fd1700 1 Processor -- start 2022-04-03T19:44:47.180 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.175+0000 7f0250fd1700 1 -- start start 2022-04-03T19:44:47.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.175+0000 7f0250fd1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f024c06bbe0 0x7f024c165550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:47.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.175+0000 7f0250fd1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f024c06bfb0 0x7f024c1573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:47.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.175+0000 7f0250fd1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f024c06c500 0x7f024c0bff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:47.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.175+0000 7f0250fd1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024c165a90 con 0x7f024c06bbe0 2022-04-03T19:44:47.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.175+0000 7f0250fd1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024c0c35f0 con 0x7f024c06c500 2022-04-03T19:44:47.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.175+0000 7f0250fd1700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f024c0c2720 con 0x7f024c06bfb0 2022-04-03T19:44:47.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.176+0000 7f024b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f024c06bbe0 0x7f024c165550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:47.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.176+0000 7f024affd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f024c06bfb0 0x7f024c1573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.176+0000 7f024bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f024c06c500 0x7f024c0bff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.176+0000 7f024b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f024c06bbe0 0x7f024c165550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42700/0 (socket says 172.21.15.77:42700) 2022-04-03T19:44:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.176+0000 7f024b7fe700 1 -- 172.21.15.77:0/361945712 learned_addr learned my addr 172.21.15.77:0/361945712 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.176+0000 7f024affd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f024c06bfb0 0x7f024c1573b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60010/0 (socket says 172.21.15.77:60010) 2022-04-03T19:44:47.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.176+0000 7f024b7fe700 1 -- 172.21.15.77:0/361945712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f024c06bfb0 msgr2=0x7f024c1573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:47.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.176+0000 7f024b7fe700 1 --2- 172.21.15.77:0/361945712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f024c06bfb0 0x7f024c1573b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:47.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.176+0000 7f024b7fe700 1 -- 172.21.15.77:0/361945712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f024c06c500 msgr2=0x7f024c0bff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:47.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.176+0000 7f024b7fe700 1 --2- 172.21.15.77:0/361945712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f024c06c500 0x7f024c0bff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:47.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.176+0000 7f024b7fe700 1 -- 172.21.15.77:0/361945712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02400092a0 con 0x7f024c06bbe0 2022-04-03T19:44:47.187 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.177+0000 7f024b7fe700 1 --2- 172.21.15.77:0/361945712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f024c06bbe0 0x7f024c165550 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f023c00c4c0 tx=0x7f023c007c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:47.187 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.177+0000 7f02337fe700 1 -- 172.21.15.77:0/361945712 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023c0050d0 con 0x7f024c06bbe0 2022-04-03T19:44:47.187 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.177+0000 7f02337fe700 1 -- 172.21.15.77:0/361945712 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023c005250 con 0x7f024c06bbe0 2022-04-03T19:44:47.187 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.177+0000 7f0250fd1700 1 -- 172.21.15.77:0/361945712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f024c0c29e0 con 0x7f024c06bbe0 2022-04-03T19:44:47.188 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.177+0000 7f02337fe700 1 -- 172.21.15.77:0/361945712 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023c0053c0 con 0x7f024c06bbe0 2022-04-03T19:44:47.188 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.177+0000 7f0250fd1700 1 -- 172.21.15.77:0/361945712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f024c0c2fa0 con 0x7f024c06bbe0 2022-04-03T19:44:47.188 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.178+0000 7f0250fd1700 1 -- 172.21.15.77:0/361945712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f024c04b050 con 0x7f024c06bbe0 2022-04-03T19:44:47.188 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.179+0000 7f02337fe700 1 -- 172.21.15.77:0/361945712 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f023c019030 con 0x7f024c06bbe0 2022-04-03T19:44:47.189 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.180+0000 7f02337fe700 1 --2- 172.21.15.77:0/361945712 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f022c063d10 0x7f022c0661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:47.189 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.180+0000 7f02337fe700 1 -- 172.21.15.77:0/361945712 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f023c07f940 con 0x7f024c06bbe0 2022-04-03T19:44:47.189 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.180+0000 7f024affd700 1 --2- 172.21.15.77:0/361945712 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f022c063d10 0x7f022c0661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:47.189 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.181+0000 7f024affd700 1 --2- 172.21.15.77:0/361945712 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f022c063d10 0x7f022c0661c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f02400096e0 tx=0x7f0240017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:47.190 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.184+0000 7f02337fe700 1 -- 172.21.15.77:0/361945712 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f023c04e4c0 con 0x7f024c06bbe0 2022-04-03T19:44:47.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.378+0000 7f0250fd1700 1 -- 172.21.15.77:0/361945712 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f024c063d10 con 0x7f024c06bbe0 2022-04-03T19:44:47.379 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.379+0000 7f02337fe700 1 -- 172.21.15.77:0/361945712 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16845 (secure 0 0 0) 0x7f023c017090 con 0x7f024c06bbe0 2022-04-03T19:44:47.380 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:47.380 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"89c3a3f4-599d-4c4a-8f5a-7180eafb7292","created":"2022-04-03T19:43:54.101559+0000","modified":"2022-04-03T19:44:22.278254+0000","last_up_change":"2022-04-03T19:44:05.589174+0000","last_in_change":"2022-04-03T19:43:56.729760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-03T19:44:07.567638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-04-03T19:44:13.149481+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-04-03T19:44:18.823926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-04-03T19:44:20.275479+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f35f5195-3729-4d78-85d1-0539f259283f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6823","nonce":20252},{"type":"v1","addr":"172.21.15.77:6825","nonce":20252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":20252},{"type":"v1","addr":"172.21.15.77:6827","nonce":20252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":20252},{"type":"v1","addr":"172.21.15.77:6831","nonce":20252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":20252},{"type":"v1","addr":"172.21.15.77:6829","nonce":20252}]},"public_addr":"172.21.15.77:6825/20252","cluster_addr":"172.21.15.77:6827/20252","heartbeat_back_addr":"172.21.15.77:6831/20252","heartbeat_front_addr":"172.21.15.77:6829/20252","state":["exists","up"]},{"osd":1,"uuid":"ab1c89dc-57ce-4cc4-b8c7-9b0c19939601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":20249},{"type":"v1","addr":"172.21.15.77:6805","nonce":20249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":20249},{"type":"v1","addr":"172.21.15.77:6809","nonce":20249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":20249},{"type":"v1","addr":"172.21.15.77:6819","nonce":20249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":20249},{"type":"v1","addr":"172.21.15.77:6815","nonce":20249}]},"public_addr":"172.21.15.77:6805/20249","cluster_addr":"172.21.15.77:6809/20249","heartbeat_back_addr":"172.21.15.77:6819/20249","heartbeat_front_addr":"172.21.15.77:6815/20249","state":["exists","up"]},{"osd":2,"uuid":"351e218b-7749-4ee9-b44a-0a5b52fb9be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":20248},{"type":"v1","addr":"172.21.15.77:6801","nonce":20248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":20248},{"type":"v1","addr":"172.21.15.77:6803","nonce":20248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":20248},{"type":"v1","addr":"172.21.15.77:6813","nonce":20248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":20248},{"type":"v1","addr":"172.21.15.77:6808","nonce":20248}]},"public_addr":"172.21.15.77:6801/20248","cluster_addr":"172.21.15.77:6803/20248","heartbeat_back_addr":"172.21.15.77:6813/20248","heartbeat_front_addr":"172.21.15.77:6808/20248","state":["exists","up"]},{"osd":3,"uuid":"1ba6ec8a-7cbf-4c12-8e21-c3097a263ae8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":20250},{"type":"v1","addr":"172.21.15.77:6814","nonce":20250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":20250},{"type":"v1","addr":"172.21.15.77:6818","nonce":20250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":20250},{"type":"v1","addr":"172.21.15.77:6824","nonce":20250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":20250},{"type":"v1","addr":"172.21.15.77:6821","nonce":20250}]},"public_addr":"172.21.15.77:6814/20250","cluster_addr":"172.21.15.77:6818/20250","heartbeat_back_addr":"172.21.15.77:6824/20250","heartbeat_front_addr":"172.21.15.77:6821/20250","state":["exists","up"]},{"osd":4,"uuid":"3faf4c09-846c-4133-856b-7f53c535a67d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":25551},{"type":"v1","addr":"172.21.15.186:6822","nonce":25551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":25551},{"type":"v1","addr":"172.21.15.186:6826","nonce":25551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":25551},{"type":"v1","addr":"172.21.15.186:6831","nonce":25551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":25551},{"type":"v1","addr":"172.21.15.186:6829","nonce":25551}]},"public_addr":"172.21.15.186:6822/25551","cluster_addr":"172.21.15.186:6826/25551","heartbeat_back_addr":"172.21.15.186:6831/25551","heartbeat_front_addr":"172.21.15.186:6829/25551","state":["exists","up"]},{"osd":5,"uuid":"ffe41d00-d0aa-42f4-82c1-5cced13c1bc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":25541},{"type":"v1","addr":"172.21.15.186:6809","nonce":25541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":25541},{"type":"v1","addr":"172.21.15.186:6811","nonce":25541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":25541},{"type":"v1","addr":"172.21.15.186:6815","nonce":25541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":25541},{"type":"v1","addr":"172.21.15.186:6813","nonce":25541}]},"public_addr":"172.21.15.186:6809/25541","cluster_addr":"172.21.15.186:6811/25541","heartbeat_back_addr":"172.21.15.186:6815/25541","heartbeat_front_addr":"172.21.15.186:6813/25541","state":["exists","up"]},{"osd":6,"uuid":"b0344ee8-deca-4e9b-934c-5866d90d0073","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":25542},{"type":"v1","addr":"172.21.15.186:6801","nonce":25542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":25542},{"type":"v1","addr":"172.21.15.186:6803","nonce":25542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":25542},{"type":"v1","addr":"172.21.15.186:6807","nonce":25542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":25542},{"type":"v1","addr":"172.21.15.186:6805","nonce":25542}]},"public_addr":"172.21.15.186:6801/25542","cluster_addr":"172.21.15.186:6803/25542","heartbeat_back_addr":"172.21.15.186:6807/25542","heartbeat_front_addr":"172.21.15.186:6805/25542","state":["exists","up"]},{"osd":7,"uuid":"0eb0e5d3-3dad-4f49-9b50-3b24a3924bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":25572},{"type":"v1","addr":"172.21.15.186:6817","nonce":25572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":25572},{"type":"v1","addr":"172.21.15.186:6819","nonce":25572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6825","nonce":25572},{"type":"v1","addr":"172.21.15.186:6827","nonce":25572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":25572},{"type":"v1","addr":"172.21.15.186:6823","nonce":25572}]},"public_addr":"172.21.15.186:6817/25572","cluster_addr":"172.21.15.186:6819/25572","heartbeat_back_addr":"172.21.15.186:6827/25572","heartbeat_front_addr":"172.21.15.186:6823/25572","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.143602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.490366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.361292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.343544+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.905429+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969298+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.949524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-03T19:44:47.382 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.381+0000 7f0250fd1700 1 -- 172.21.15.77:0/361945712 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f022c063d10 msgr2=0x7f022c0661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:47.383 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.381+0000 7f0250fd1700 1 --2- 172.21.15.77:0/361945712 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f022c063d10 0x7f022c0661c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f02400096e0 tx=0x7f0240017000 comp rx=0 tx=0).stop 2022-04-03T19:44:47.383 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.381+0000 7f0250fd1700 1 -- 172.21.15.77:0/361945712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f024c06bbe0 msgr2=0x7f024c165550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:47.383 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.381+0000 7f0250fd1700 1 --2- 172.21.15.77:0/361945712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f024c06bbe0 0x7f024c165550 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f023c00c4c0 tx=0x7f023c007c30 comp rx=0 tx=0).stop 2022-04-03T19:44:47.383 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.381+0000 7f0250fd1700 1 -- 172.21.15.77:0/361945712 shutdown_connections 2022-04-03T19:44:47.384 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.381+0000 7f0250fd1700 1 --2- 172.21.15.77:0/361945712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f024c06c500 0x7f024c0bff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:47.384 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.381+0000 7f0250fd1700 1 --2- 172.21.15.77:0/361945712 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f024c06bbe0 0x7f024c165550 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:47.384 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.381+0000 7f0250fd1700 1 --2- 172.21.15.77:0/361945712 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f022c063d10 0x7f022c0661c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:47.384 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.381+0000 7f0250fd1700 1 --2- 172.21.15.77:0/361945712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f024c06bfb0 0x7f024c1573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:47.384 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.382+0000 7f0250fd1700 1 -- 172.21.15.77:0/361945712 >> 172.21.15.77:0/361945712 conn(0x7f024c0cb6b0 msgr2=0x7f024c14d430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.382+0000 7f0250fd1700 1 -- 172.21.15.77:0/361945712 shutdown_connections 2022-04-03T19:44:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.382+0000 7f0250fd1700 1 -- 172.21.15.77:0/361945712 wait complete. 2022-04-03T19:44:47.407 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-04-03T19:44:07.567638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-04-03T19:44:13.149481+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-04-03T19:44:18.823926+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-04-03T19:44:20.275479+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-04-03T19:44:47.408 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-04-03T19:44:47.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.606+0000 7f31cbfff700 1 Processor -- start 2022-04-03T19:44:47.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.607+0000 7f31cbfff700 1 -- start start 2022-04-03T19:44:47.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.608+0000 7f31cbfff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31cc0c0af0 0x7f31cc0c1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:47.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.608+0000 7f31cbfff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31cc155a50 con 0x7f31cc06b0e0 2022-04-03T19:44:47.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.608+0000 7f31cbfff700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31cc154e40 con 0x7f31cc0c7410 2022-04-03T19:44:47.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.608+0000 7f31cbfff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc155120 con 0x7f31cc0c0af0 2022-04-03T19:44:47.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.608+0000 7f31caffd700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f31cc0c7410 0x7f31cc0c03e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44162/0 (socket says 172.21.15.77:44162) 2022-04-03T19:44:47.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.608+0000 7f31ca7fc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31cc0c0af0 0x7f31cc0c1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:47.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.608+0000 7f31caffd700 1 -- 172.21.15.77:0/3894323629 learned_addr learned my addr 172.21.15.77:0/3894323629 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:47.612 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.608+0000 7f31c9ffb700 1 -- 172.21.15.77:0/3894323629 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024055077 0 0) 0x7f31cc154e40 con 0x7f31cc0c7410 2022-04-03T19:44:47.612 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.609+0000 7f31c9ffb700 1 -- 172.21.15.77:0/3894323629 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31b80033f0 con 0x7f31cc0c7410 2022-04-03T19:44:47.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.609+0000 7f31ca7fc700 1 -- 172.21.15.77:0/3894323629 >> v1:172.21.15.186:6789/0 conn(0x7f31cc0c7410 legacy=0x7f31cc0c03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:47.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.609+0000 7f31ca7fc700 1 -- 172.21.15.77:0/3894323629 >> v1:172.21.15.77:6789/0 conn(0x7f31cc06b0e0 legacy=0x7f31cc0c29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:47.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.609+0000 7f31ca7fc700 1 -- 172.21.15.77:0/3894323629 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31cc155400 con 0x7f31cc0c0af0 2022-04-03T19:44:47.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.609+0000 7f31ca7fc700 1 --2- 172.21.15.77:0/3894323629 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31cc0c0af0 0x7f31cc0c1b90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f31c0009790 tx=0x7f31c0009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cd82fc3f6ef79e5 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:47.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.609+0000 7f31c9ffb700 1 -- 172.21.15.77:0/3894323629 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c0004030 con 0x7f31cc0c0af0 2022-04-03T19:44:47.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.609+0000 7f31c9ffb700 1 -- 172.21.15.77:0/3894323629 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c0004190 con 0x7f31cc0c0af0 2022-04-03T19:44:47.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.609+0000 7f31c9ffb700 1 -- 172.21.15.77:0/3894323629 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c0011630 con 0x7f31cc0c0af0 2022-04-03T19:44:47.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.610+0000 7f31cbfff700 1 -- 172.21.15.77:0/3894323629 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31cc0c0af0 msgr2=0x7f31cc0c1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:47.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.610+0000 7f31cbfff700 1 --2- 172.21.15.77:0/3894323629 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31cc0c0af0 0x7f31cc0c1b90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f31c0009790 tx=0x7f31c0009060 comp rx=0 tx=0).stop 2022-04-03T19:44:47.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.610+0000 7f31cbfff700 1 -- 172.21.15.77:0/3894323629 shutdown_connections 2022-04-03T19:44:47.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.610+0000 7f31cbfff700 1 --2- 172.21.15.77:0/3894323629 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31cc0c0af0 0x7f31cc0c1b90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:47.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.610+0000 7f31cbfff700 1 -- 172.21.15.77:0/3894323629 >> 172.21.15.77:0/3894323629 conn(0x7f31cc0626d0 msgr2=0x7f31cc062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:47.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.610+0000 7f31cbfff700 1 -- 172.21.15.77:0/3894323629 shutdown_connections 2022-04-03T19:44:47.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.610+0000 7f31cbfff700 1 -- 172.21.15.77:0/3894323629 wait complete. 2022-04-03T19:44:47.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.611+0000 7f31cbfff700 1 Processor -- start 2022-04-03T19:44:47.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.612+0000 7f31cbfff700 1 -- start start 2022-04-03T19:44:47.619 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.613+0000 7f31cbfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31cc06b0e0 0x7f31cc14e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:47.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.613+0000 7f31cbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31cc0c0af0 0x7f31cc0bc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:47.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.613+0000 7f31cbfff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31cc0c7410 0x7f31cc14fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:47.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.613+0000 7f31cbfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc14d780 con 0x7f31cc0c0af0 2022-04-03T19:44:47.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.613+0000 7f31cbfff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc14dd00 con 0x7f31cc06b0e0 2022-04-03T19:44:47.621 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.613+0000 7f31cbfff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc0c7010 con 0x7f31cc0c7410 2022-04-03T19:44:47.621 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.613+0000 7f31ca7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31cc0c0af0 0x7f31cc0bc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:47.621 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.613+0000 7f31ca7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31cc0c0af0 0x7f31cc0bc720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42702/0 (socket says 172.21.15.77:42702) 2022-04-03T19:44:47.621 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.613+0000 7f31ca7fc700 1 -- 172.21.15.77:0/2336883581 learned_addr learned my addr 172.21.15.77:0/2336883581 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:47.622 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.613+0000 7f31caffd700 1 --2- 172.21.15.77:0/2336883581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31cc06b0e0 0x7f31cc14e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:47.622 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.613+0000 7f31cb7fe700 1 --2- 172.21.15.77:0/2336883581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31cc0c7410 0x7f31cc14fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:47.622 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.613+0000 7f31ca7fc700 1 -- 172.21.15.77:0/2336883581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31cc0c7410 msgr2=0x7f31cc14fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:47.622 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.613+0000 7f31ca7fc700 1 --2- 172.21.15.77:0/2336883581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31cc0c7410 0x7f31cc14fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:47.623 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.613+0000 7f31ca7fc700 1 -- 172.21.15.77:0/2336883581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31cc06b0e0 msgr2=0x7f31cc14e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:47.623 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.613+0000 7f31ca7fc700 1 --2- 172.21.15.77:0/2336883581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31cc06b0e0 0x7f31cc14e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:47.623 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.613+0000 7f31ca7fc700 1 -- 172.21.15.77:0/2336883581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c00092a0 con 0x7f31cc0c0af0 2022-04-03T19:44:47.623 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.613+0000 7f31caffd700 1 --2- 172.21.15.77:0/2336883581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31cc06b0e0 0x7f31cc14e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:44:47.623 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.614+0000 7f31ca7fc700 1 --2- 172.21.15.77:0/2336883581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31cc0c0af0 0x7f31cc0bc720 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f31c0004630 tx=0x7f31c0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:47.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.614+0000 7f31b2ffd700 1 -- 172.21.15.77:0/2336883581 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c0003660 con 0x7f31cc0c0af0 2022-04-03T19:44:47.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.614+0000 7f31b2ffd700 1 -- 172.21.15.77:0/2336883581 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c0020070 con 0x7f31cc0c0af0 2022-04-03T19:44:47.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.614+0000 7f31b2ffd700 1 -- 172.21.15.77:0/2336883581 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c00039f0 con 0x7f31cc0c0af0 2022-04-03T19:44:47.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.614+0000 7f31cbfff700 1 -- 172.21.15.77:0/2336883581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31cc14f610 con 0x7f31cc0c0af0 2022-04-03T19:44:47.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.614+0000 7f31cbfff700 1 -- 172.21.15.77:0/2336883581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31cc166530 con 0x7f31cc0c0af0 2022-04-03T19:44:47.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.615+0000 7f31cbfff700 1 -- 172.21.15.77:0/2336883581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31cc0c1f10 con 0x7f31cc0c0af0 2022-04-03T19:44:47.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.616+0000 7f31b2ffd700 1 -- 172.21.15.77:0/2336883581 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f31c0017020 con 0x7f31cc0c0af0 2022-04-03T19:44:47.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.616+0000 7f31b2ffd700 1 --2- 172.21.15.77:0/2336883581 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f31ac063ce0 0x7f31ac066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:47.626 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.617+0000 7f31b2ffd700 1 -- 172.21.15.77:0/2336883581 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f31c0081f50 con 0x7f31cc0c0af0 2022-04-03T19:44:47.626 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.617+0000 7f31caffd700 1 --2- 172.21.15.77:0/2336883581 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f31ac063ce0 0x7f31ac066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:47.626 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.617+0000 7f31caffd700 1 --2- 172.21.15.77:0/2336883581 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f31ac063ce0 0x7f31ac066190 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f31bc000c00 tx=0x7f31bc007c50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:47.627 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.621+0000 7f31b2ffd700 1 -- 172.21.15.77:0/2336883581 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f31c0050a40 con 0x7f31cc0c0af0 2022-04-03T19:44:47.810 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.810+0000 7f31cbfff700 1 -- 172.21.15.77:0/2336883581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f31cc14e6d0 con 0x7f31cc0c0af0 2022-04-03T19:44:47.811 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.811+0000 7f31b2ffd700 1 -- 172.21.15.77:0/2336883581 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f31c002b020 con 0x7f31cc0c0af0 2022-04-03T19:44:47.812 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2022-04-03T19:44:47.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.813+0000 7f31cbfff700 1 -- 172.21.15.77:0/2336883581 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f31ac063ce0 msgr2=0x7f31ac066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:47.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.813+0000 7f31cbfff700 1 --2- 172.21.15.77:0/2336883581 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f31ac063ce0 0x7f31ac066190 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f31bc000c00 tx=0x7f31bc007c50 comp rx=0 tx=0).stop 2022-04-03T19:44:47.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.813+0000 7f31cbfff700 1 -- 172.21.15.77:0/2336883581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31cc0c0af0 msgr2=0x7f31cc0bc720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:47.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.813+0000 7f31cbfff700 1 --2- 172.21.15.77:0/2336883581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31cc0c0af0 0x7f31cc0bc720 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f31c0004630 tx=0x7f31c0016000 comp rx=0 tx=0).stop 2022-04-03T19:44:47.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.813+0000 7f31cbfff700 1 -- 172.21.15.77:0/2336883581 shutdown_connections 2022-04-03T19:44:47.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.813+0000 7f31cbfff700 1 --2- 172.21.15.77:0/2336883581 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f31ac063ce0 0x7f31ac066190 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:47.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.813+0000 7f31cbfff700 1 --2- 172.21.15.77:0/2336883581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31cc0c7410 0x7f31cc14fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:47.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.813+0000 7f31cbfff700 1 --2- 172.21.15.77:0/2336883581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31cc06b0e0 0x7f31cc14e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:47.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.813+0000 7f31cbfff700 1 --2- 172.21.15.77:0/2336883581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31cc0c0af0 0x7f31cc0bc720 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:47.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.813+0000 7f31cbfff700 1 -- 172.21.15.77:0/2336883581 >> 172.21.15.77:0/2336883581 conn(0x7f31cc0626d0 msgr2=0x7f31cc0c77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:47.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.813+0000 7f31cbfff700 1 -- 172.21.15.77:0/2336883581 shutdown_connections 2022-04-03T19:44:47.817 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:47.813+0000 7f31cbfff700 1 -- 172.21.15.77:0/2336883581 wait complete. 2022-04-03T19:44:47.838 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-04-03T19:44:48.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.036+0000 7f6902629700 1 Processor -- start 2022-04-03T19:44:48.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.036+0000 7f6902629700 1 -- start start 2022-04-03T19:44:48.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.037+0000 7f6902629700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f68fc0c45c0 0x7f68fc14f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:48.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.037+0000 7f6902629700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f68fc0682c0 0x7f68fc0c4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:48.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.037+0000 7f6902629700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68fc066f40 con 0x7f68fc14f2c0 2022-04-03T19:44:48.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.038+0000 7f6902629700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68fc06e6c0 con 0x7f68fc0682c0 2022-04-03T19:44:48.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.038+0000 7f6902629700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68fc067220 con 0x7f68fc0c45c0 2022-04-03T19:44:48.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.038+0000 7f6901627700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f68fc14f2c0 0x7f68fc06cb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44164/0 (socket says 172.21.15.77:44164) 2022-04-03T19:44:48.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.038+0000 7f6901627700 1 -- 172.21.15.77:0/3420080920 learned_addr learned my addr 172.21.15.77:0/3420080920 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:48.041 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.038+0000 7f6900e26700 1 --2- 172.21.15.77:0/3420080920 >> v2:172.21.15.186:3300/0 conn(0x7f68fc0c45c0 0x7f68fc14f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:48.041 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.038+0000 7f6901e28700 1 --2- 172.21.15.77:0/3420080920 >> v2:172.21.15.77:3300/0 conn(0x7f68fc0682c0 0x7f68fc0c4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:48.041 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.038+0000 7f68ebfff700 1 -- 172.21.15.77:0/3420080920 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236156840 0 0) 0x7f68fc066f40 con 0x7f68fc14f2c0 2022-04-03T19:44:48.041 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.038+0000 7f68ebfff700 1 -- 172.21.15.77:0/3420080920 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68e40033f0 con 0x7f68fc14f2c0 2022-04-03T19:44:48.041 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.038+0000 7f6900e26700 1 -- 172.21.15.77:0/3420080920 >> v2:172.21.15.77:3300/0 conn(0x7f68fc0682c0 msgr2=0x7f68fc0c4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.038+0000 7f6900e26700 1 --2- 172.21.15.77:0/3420080920 >> v2:172.21.15.77:3300/0 conn(0x7f68fc0682c0 0x7f68fc0c4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.038+0000 7f6900e26700 1 -- 172.21.15.77:0/3420080920 >> v1:172.21.15.186:6789/0 conn(0x7f68fc14f2c0 legacy=0x7f68fc06cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.038+0000 7f6900e26700 1 -- 172.21.15.77:0/3420080920 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68fc067500 con 0x7f68fc0c45c0 2022-04-03T19:44:48.042 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.039+0000 7f6900e26700 1 --2- 172.21.15.77:0/3420080920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68fc0c45c0 0x7f68fc14f690 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f68fc06b060 tx=0x7f68ec009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e52cefdebdbfef07 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:48.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.039+0000 7f68ebfff700 1 -- 172.21.15.77:0/3420080920 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec0105b0 con 0x7f68fc0c45c0 2022-04-03T19:44:48.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.039+0000 7f68ebfff700 1 -- 172.21.15.77:0/3420080920 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ec010710 con 0x7f68fc0c45c0 2022-04-03T19:44:48.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.039+0000 7f68ebfff700 1 -- 172.21.15.77:0/3420080920 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec010ad0 con 0x7f68fc0c45c0 2022-04-03T19:44:48.046 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.039+0000 7f6902629700 1 -- 172.21.15.77:0/3420080920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68fc0c45c0 msgr2=0x7f68fc14f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:48.046 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.039+0000 7f6902629700 1 --2- 172.21.15.77:0/3420080920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68fc0c45c0 0x7f68fc14f690 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f68fc06b060 tx=0x7f68ec009060 comp rx=0 tx=0).stop 2022-04-03T19:44:48.046 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.039+0000 7f6902629700 1 -- 172.21.15.77:0/3420080920 shutdown_connections 2022-04-03T19:44:48.046 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.039+0000 7f6902629700 1 --2- 172.21.15.77:0/3420080920 >> v2:172.21.15.77:3300/0 conn(0x7f68fc0682c0 0x7f68fc0c4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.047 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.039+0000 7f6902629700 1 --2- 172.21.15.77:0/3420080920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68fc0c45c0 0x7f68fc14f690 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.047 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.039+0000 7f6902629700 1 -- 172.21.15.77:0/3420080920 >> 172.21.15.77:0/3420080920 conn(0x7f68fc069920 msgr2=0x7f68fc0626d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:48.047 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.040+0000 7f6902629700 1 -- 172.21.15.77:0/3420080920 shutdown_connections 2022-04-03T19:44:48.047 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.040+0000 7f6902629700 1 -- 172.21.15.77:0/3420080920 wait complete. 2022-04-03T19:44:48.047 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.041+0000 7f6902629700 1 Processor -- start 2022-04-03T19:44:48.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.041+0000 7f6902629700 1 -- start start 2022-04-03T19:44:48.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.042+0000 7f6902629700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68fc0682c0 0x7f68fc151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:48.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.042+0000 7f6902629700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68fc0c45c0 0x7f68fc16d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:48.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.042+0000 7f6902629700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68fc14f2c0 0x7f68fc14b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:48.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.042+0000 7f6902629700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68fc16cbd0 con 0x7f68fc0682c0 2022-04-03T19:44:48.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.042+0000 7f6902629700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68fc0c2960 con 0x7f68fc0c45c0 2022-04-03T19:44:48.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.042+0000 7f6902629700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68fc0c3170 con 0x7f68fc14f2c0 2022-04-03T19:44:48.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.042+0000 7f6901627700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68fc0682c0 0x7f68fc151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:48.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.043+0000 7f6901e28700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68fc14f2c0 0x7f68fc14b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:48.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.043+0000 7f6901627700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68fc0682c0 0x7f68fc151860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42706/0 (socket says 172.21.15.77:42706) 2022-04-03T19:44:48.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.043+0000 7f6901627700 1 -- 172.21.15.77:0/3150926251 learned_addr learned my addr 172.21.15.77:0/3150926251 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:48.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.043+0000 7f6900e26700 1 --2- 172.21.15.77:0/3150926251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68fc0c45c0 0x7f68fc16d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:48.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.043+0000 7f6901627700 1 -- 172.21.15.77:0/3150926251 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68fc14f2c0 msgr2=0x7f68fc14b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:48.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.043+0000 7f6901627700 1 --2- 172.21.15.77:0/3150926251 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68fc14f2c0 0x7f68fc14b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.043+0000 7f6901627700 1 -- 172.21.15.77:0/3150926251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68fc0c45c0 msgr2=0x7f68fc16d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:48.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.043+0000 7f6901627700 1 --2- 172.21.15.77:0/3150926251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68fc0c45c0 0x7f68fc16d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.043+0000 7f6901627700 1 -- 172.21.15.77:0/3150926251 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ec0092a0 con 0x7f68fc0682c0 2022-04-03T19:44:48.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.043+0000 7f6901627700 1 --2- 172.21.15.77:0/3150926251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68fc0682c0 0x7f68fc151860 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f68f800c490 tx=0x7f68f8007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:48.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.043+0000 7f68e97fa700 1 -- 172.21.15.77:0/3150926251 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f800ce40 con 0x7f68fc0682c0 2022-04-03T19:44:48.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.044+0000 7f68e97fa700 1 -- 172.21.15.77:0/3150926251 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f80072e0 con 0x7f68fc0682c0 2022-04-03T19:44:48.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.044+0000 7f6902629700 1 -- 172.21.15.77:0/3150926251 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68fc0c2420 con 0x7f68fc0682c0 2022-04-03T19:44:48.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.044+0000 7f68e97fa700 1 -- 172.21.15.77:0/3150926251 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f8005780 con 0x7f68fc0682c0 2022-04-03T19:44:48.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.044+0000 7f6902629700 1 -- 172.21.15.77:0/3150926251 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68fc0c2e90 con 0x7f68fc0682c0 2022-04-03T19:44:48.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.045+0000 7f6902629700 1 -- 172.21.15.77:0/3150926251 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68fc16ce70 con 0x7f68fc0682c0 2022-04-03T19:44:48.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.046+0000 7f68e97fa700 1 -- 172.21.15.77:0/3150926251 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f68f8019030 con 0x7f68fc0682c0 2022-04-03T19:44:48.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.047+0000 7f68e97fa700 1 --2- 172.21.15.77:0/3150926251 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f68dc063ce0 0x7f68dc066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:48.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.047+0000 7f68e97fa700 1 -- 172.21.15.77:0/3150926251 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f68f807ff80 con 0x7f68fc0682c0 2022-04-03T19:44:48.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.047+0000 7f6900e26700 1 --2- 172.21.15.77:0/3150926251 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f68dc063ce0 0x7f68dc066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:48.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.048+0000 7f6900e26700 1 --2- 172.21.15.77:0/3150926251 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f68dc063ce0 0x7f68dc066190 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f68ec009fd0 tx=0x7f68ec016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:48.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.050+0000 7f68e97fa700 1 -- 172.21.15.77:0/3150926251 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f68f804eaf0 con 0x7f68fc0682c0 2022-04-03T19:44:48.247 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.246+0000 7f6902629700 1 -- 172.21.15.77:0/3150926251 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f68fc064640 con 0x7f68fc0682c0 2022-04-03T19:44:48.247 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.247+0000 7f68e97fa700 1 -- 172.21.15.77:0/3150926251 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7f68f8017070 con 0x7f68fc0682c0 2022-04-03T19:44:48.248 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2022-04-03T19:44:48.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.249+0000 7f6902629700 1 -- 172.21.15.77:0/3150926251 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f68dc063ce0 msgr2=0x7f68dc066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:48.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.250+0000 7f6902629700 1 --2- 172.21.15.77:0/3150926251 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f68dc063ce0 0x7f68dc066190 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f68ec009fd0 tx=0x7f68ec016000 comp rx=0 tx=0).stop 2022-04-03T19:44:48.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.250+0000 7f6902629700 1 -- 172.21.15.77:0/3150926251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68fc0682c0 msgr2=0x7f68fc151860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:48.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.250+0000 7f6902629700 1 --2- 172.21.15.77:0/3150926251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68fc0682c0 0x7f68fc151860 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f68f800c490 tx=0x7f68f8007c70 comp rx=0 tx=0).stop 2022-04-03T19:44:48.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.250+0000 7f6902629700 1 -- 172.21.15.77:0/3150926251 shutdown_connections 2022-04-03T19:44:48.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.250+0000 7f6902629700 1 --2- 172.21.15.77:0/3150926251 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f68dc063ce0 0x7f68dc066190 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.250+0000 7f6902629700 1 --2- 172.21.15.77:0/3150926251 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68fc14f2c0 0x7f68fc14b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.250+0000 7f6902629700 1 --2- 172.21.15.77:0/3150926251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68fc0682c0 0x7f68fc151860 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.250+0000 7f6902629700 1 --2- 172.21.15.77:0/3150926251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68fc0c45c0 0x7f68fc16d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.250+0000 7f6902629700 1 -- 172.21.15.77:0/3150926251 >> 172.21.15.77:0/3150926251 conn(0x7f68fc069920 msgr2=0x7f68fc156f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:48.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.250+0000 7f6902629700 1 -- 172.21.15.77:0/3150926251 shutdown_connections 2022-04-03T19:44:48.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.250+0000 7f6902629700 1 -- 172.21.15.77:0/3150926251 wait complete. 2022-04-03T19:44:48.274 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-04-03T19:44:48.474 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.473+0000 7f6a05a8f700 1 Processor -- start 2022-04-03T19:44:48.474 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.473+0000 7f6a05a8f700 1 -- start start 2022-04-03T19:44:48.475 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.474+0000 7f6a05a8f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a000c5680 0x7f6a000c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:48.475 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.475+0000 7f6a05a8f700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6a00158e10 0x7f6a0014ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:48.475 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.475+0000 7f6a05a8f700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a000c7e80 con 0x7f6a000c52b0 2022-04-03T19:44:48.476 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.475+0000 7f6a05a8f700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a000c4260 con 0x7f6a00158e10 2022-04-03T19:44:48.476 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.475+0000 7f6a05a8f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a000c8a20 con 0x7f6a000c5680 2022-04-03T19:44:48.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.475+0000 7f6a04a8d700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f6a000c52b0 0x7f6a000c8310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44166/0 (socket says 172.21.15.77:44166) 2022-04-03T19:44:48.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.475+0000 7f6a04a8d700 1 -- 172.21.15.77:0/4194204656 learned_addr learned my addr 172.21.15.77:0/4194204656 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:48.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.475+0000 7f69fffff700 1 --2- 172.21.15.77:0/4194204656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a000c5680 0x7f6a000c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:48.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.475+0000 7f6a0528e700 1 --2- 172.21.15.77:0/4194204656 >> v2:172.21.15.186:3300/0 conn(0x7f6a00158e10 0x7f6a0014ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:48.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.476+0000 7f69ff7fe700 1 -- 172.21.15.77:0/4194204656 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205960869 0 0) 0x7f6a000c7e80 con 0x7f6a000c52b0 2022-04-03T19:44:48.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.476+0000 7f69ff7fe700 1 -- 172.21.15.77:0/4194204656 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69e40033f0 con 0x7f6a000c52b0 2022-04-03T19:44:48.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.476+0000 7f6a0528e700 1 -- 172.21.15.77:0/4194204656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a000c5680 msgr2=0x7f6a000c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:48.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.476+0000 7f6a0528e700 1 --2- 172.21.15.77:0/4194204656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a000c5680 0x7f6a000c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.476+0000 7f6a0528e700 1 -- 172.21.15.77:0/4194204656 >> v1:172.21.15.186:6789/0 conn(0x7f6a000c52b0 legacy=0x7f6a000c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:48.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.476+0000 7f6a0528e700 1 -- 172.21.15.77:0/4194204656 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a000c6ea0 con 0x7f6a00158e10 2022-04-03T19:44:48.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.476+0000 7f6a0528e700 1 --2- 172.21.15.77:0/4194204656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a00158e10 0x7f6a0014ed50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f69f8009e60 tx=0x7f69f8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e69602d36ad7680 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:48.481 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.476+0000 7f69ff7fe700 1 -- 172.21.15.77:0/4194204656 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f800e070 con 0x7f6a00158e10 2022-04-03T19:44:48.481 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.476+0000 7f69ff7fe700 1 -- 172.21.15.77:0/4194204656 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f800bb40 con 0x7f6a00158e10 2022-04-03T19:44:48.481 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.476+0000 7f69ff7fe700 1 -- 172.21.15.77:0/4194204656 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f8012430 con 0x7f6a00158e10 2022-04-03T19:44:48.481 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.477+0000 7f6a05a8f700 1 -- 172.21.15.77:0/4194204656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a00158e10 msgr2=0x7f6a0014ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:48.481 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.477+0000 7f6a05a8f700 1 --2- 172.21.15.77:0/4194204656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a00158e10 0x7f6a0014ed50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f69f8009e60 tx=0x7f69f8009780 comp rx=0 tx=0).stop 2022-04-03T19:44:48.482 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.477+0000 7f6a05a8f700 1 -- 172.21.15.77:0/4194204656 shutdown_connections 2022-04-03T19:44:48.482 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.477+0000 7f6a05a8f700 1 --2- 172.21.15.77:0/4194204656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a00158e10 0x7f6a0014ed50 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.482 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.477+0000 7f6a05a8f700 1 --2- 172.21.15.77:0/4194204656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a000c5680 0x7f6a000c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.482 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.477+0000 7f6a05a8f700 1 -- 172.21.15.77:0/4194204656 >> 172.21.15.77:0/4194204656 conn(0x7f6a000626d0 msgr2=0x7f6a00062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:48.482 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.477+0000 7f6a05a8f700 1 -- 172.21.15.77:0/4194204656 shutdown_connections 2022-04-03T19:44:48.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.477+0000 7f6a05a8f700 1 -- 172.21.15.77:0/4194204656 wait complete. 2022-04-03T19:44:48.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.479+0000 7f6a05a8f700 1 Processor -- start 2022-04-03T19:44:48.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.479+0000 7f6a05a8f700 1 -- start start 2022-04-03T19:44:48.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.480+0000 7f6a05a8f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a000c52b0 0x7f6a00164d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:48.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.480+0000 7f6a05a8f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a000c5680 0x7f6a000c9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:48.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.480+0000 7f6a05a8f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a00158e10 0x7f6a00157670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:48.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.480+0000 7f6a05a8f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a000c99d0 con 0x7f6a00158e10 2022-04-03T19:44:48.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.480+0000 7f6a05a8f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a00165c00 con 0x7f6a000c5680 2022-04-03T19:44:48.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.480+0000 7f6a05a8f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a00165360 con 0x7f6a000c52b0 2022-04-03T19:44:48.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.480+0000 7f6a0528e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a00158e10 0x7f6a00157670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:48.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.480+0000 7f6a04a8d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a000c52b0 0x7f6a00164d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.480+0000 7f6a0528e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a00158e10 0x7f6a00157670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42708/0 (socket says 172.21.15.77:42708) 2022-04-03T19:44:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.480+0000 7f6a0528e700 1 -- 172.21.15.77:0/2918652214 learned_addr learned my addr 172.21.15.77:0/2918652214 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.480+0000 7f6a04a8d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a000c52b0 0x7f6a00164d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60022/0 (socket says 172.21.15.77:60022) 2022-04-03T19:44:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.480+0000 7f69fffff700 1 --2- 172.21.15.77:0/2918652214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a000c5680 0x7f6a000c9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:48.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.480+0000 7f69fffff700 1 -- 172.21.15.77:0/2918652214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a000c52b0 msgr2=0x7f6a00164d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:48.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.480+0000 7f69fffff700 1 --2- 172.21.15.77:0/2918652214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a000c52b0 0x7f6a00164d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.480+0000 7f69fffff700 1 -- 172.21.15.77:0/2918652214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a00158e10 msgr2=0x7f6a00157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:48.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.480+0000 7f69fffff700 1 --2- 172.21.15.77:0/2918652214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a00158e10 0x7f6a00157670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.480+0000 7f69fffff700 1 -- 172.21.15.77:0/2918652214 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f8009a20 con 0x7f6a000c5680 2022-04-03T19:44:48.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.481+0000 7f69fffff700 1 --2- 172.21.15.77:0/2918652214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a000c5680 0x7f6a000c9e60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f69ec00e900 tx=0x7f69ec009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:48.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.481+0000 7f69fcff9700 1 -- 172.21.15.77:0/2918652214 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec013680 con 0x7f6a000c5680 2022-04-03T19:44:48.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.481+0000 7f69fcff9700 1 -- 172.21.15.77:0/2918652214 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ec0137e0 con 0x7f6a000c5680 2022-04-03T19:44:48.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.481+0000 7f6a05a8f700 1 -- 172.21.15.77:0/2918652214 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a00165640 con 0x7f6a000c5680 2022-04-03T19:44:48.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.481+0000 7f69fcff9700 1 -- 172.21.15.77:0/2918652214 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec013b60 con 0x7f6a000c5680 2022-04-03T19:44:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.481+0000 7f6a05a8f700 1 -- 172.21.15.77:0/2918652214 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a000c9660 con 0x7f6a000c5680 2022-04-03T19:44:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.482+0000 7f6a05a8f700 1 -- 172.21.15.77:0/2918652214 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a00088e10 con 0x7f6a000c5680 2022-04-03T19:44:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.483+0000 7f69fcff9700 1 -- 172.21.15.77:0/2918652214 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f69ec019030 con 0x7f6a000c5680 2022-04-03T19:44:48.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.484+0000 7f69fcff9700 1 --2- 172.21.15.77:0/2918652214 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f69e8063d10 0x7f69e80661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:48.492 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.484+0000 7f69fcff9700 1 -- 172.21.15.77:0/2918652214 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f69ec080770 con 0x7f6a000c5680 2022-04-03T19:44:48.492 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.484+0000 7f6a04a8d700 1 --2- 172.21.15.77:0/2918652214 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f69e8063d10 0x7f69e80661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:48.492 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.485+0000 7f6a04a8d700 1 --2- 172.21.15.77:0/2918652214 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f69e8063d10 0x7f69e80661c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f69f4005750 tx=0x7f69f4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:48.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.489+0000 7f69fcff9700 1 -- 172.21.15.77:0/2918652214 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f69ec04f350 con 0x7f6a000c5680 2022-04-03T19:44:48.683 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.682+0000 7f6a05a8f700 1 -- 172.21.15.77:0/2918652214 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6a0004b050 con 0x7f6a000c5680 2022-04-03T19:44:48.683 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.683+0000 7f69fcff9700 1 -- 172.21.15.77:0/2918652214 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f69ec017020 con 0x7f6a000c5680 2022-04-03T19:44:48.684 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2022-04-03T19:44:48.687 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.686+0000 7f6a05a8f700 1 -- 172.21.15.77:0/2918652214 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f69e8063d10 msgr2=0x7f69e80661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:48.687 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.686+0000 7f6a05a8f700 1 --2- 172.21.15.77:0/2918652214 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f69e8063d10 0x7f69e80661c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f69f4005750 tx=0x7f69f4009040 comp rx=0 tx=0).stop 2022-04-03T19:44:48.687 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.686+0000 7f6a05a8f700 1 -- 172.21.15.77:0/2918652214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a000c5680 msgr2=0x7f6a000c9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:48.687 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.686+0000 7f6a05a8f700 1 --2- 172.21.15.77:0/2918652214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a000c5680 0x7f6a000c9e60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f69ec00e900 tx=0x7f69ec009060 comp rx=0 tx=0).stop 2022-04-03T19:44:48.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.686+0000 7f6a05a8f700 1 -- 172.21.15.77:0/2918652214 shutdown_connections 2022-04-03T19:44:48.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.686+0000 7f6a05a8f700 1 --2- 172.21.15.77:0/2918652214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a00158e10 0x7f6a00157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.686+0000 7f6a05a8f700 1 --2- 172.21.15.77:0/2918652214 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f69e8063d10 0x7f69e80661c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.686+0000 7f6a05a8f700 1 --2- 172.21.15.77:0/2918652214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a000c52b0 0x7f6a00164d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.686+0000 7f6a05a8f700 1 --2- 172.21.15.77:0/2918652214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a000c5680 0x7f6a000c9e60 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.689 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.686+0000 7f6a05a8f700 1 -- 172.21.15.77:0/2918652214 >> 172.21.15.77:0/2918652214 conn(0x7f6a000626d0 msgr2=0x7f6a00062aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:48.689 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.686+0000 7f6a05a8f700 1 -- 172.21.15.77:0/2918652214 shutdown_connections 2022-04-03T19:44:48.689 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.686+0000 7f6a05a8f700 1 -- 172.21.15.77:0/2918652214 wait complete. 2022-04-03T19:44:48.710 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-04-03T19:44:48.914 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.914+0000 7f47d1357700 1 Processor -- start 2022-04-03T19:44:48.914 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.914+0000 7f47d1357700 1 -- start start 2022-04-03T19:44:48.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.915+0000 7f47d1357700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47cc0c0af0 0x7f47cc0c1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:48.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.915+0000 7f47d1357700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f47cc06b0e0 0x7f47cc0c29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:48.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.915+0000 7f47d1357700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47cc155a50 con 0x7f47cc0c7410 2022-04-03T19:44:48.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.915+0000 7f47d1357700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47cc154e40 con 0x7f47cc06b0e0 2022-04-03T19:44:48.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.915+0000 7f47d1357700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc155120 con 0x7f47cc0c0af0 2022-04-03T19:44:48.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.915+0000 7f47d0b56700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f47cc06b0e0 0x7f47cc0c29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:48.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.915+0000 7f47cbfff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f47cc0c7410 0x7f47cc0c03e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38372/0 (socket says 172.21.15.77:38372) 2022-04-03T19:44:48.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.915+0000 7f47c3fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47cc0c0af0 0x7f47cc0c1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:48.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.915+0000 7f47cbfff700 1 -- 172.21.15.77:0/516853026 learned_addr learned my addr 172.21.15.77:0/516853026 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:48.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.916+0000 7f47c3fff700 1 -- 172.21.15.77:0/516853026 >> v2:172.21.15.77:3300/0 conn(0x7f47cc06b0e0 msgr2=0x7f47cc0c29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:48.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.916+0000 7f47c3fff700 1 --2- 172.21.15.77:0/516853026 >> v2:172.21.15.77:3300/0 conn(0x7f47cc06b0e0 0x7f47cc0c29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.916+0000 7f47c3fff700 1 -- 172.21.15.77:0/516853026 >> v1:172.21.15.77:6789/0 conn(0x7f47cc0c7410 legacy=0x7f47cc0c03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:48.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.916+0000 7f47c3fff700 1 -- 172.21.15.77:0/516853026 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47cc155400 con 0x7f47cc0c0af0 2022-04-03T19:44:48.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.916+0000 7f47c3fff700 1 --2- 172.21.15.77:0/516853026 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47cc0c0af0 0x7f47cc0c1b90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f47b400a880 tx=0x7f47b400a120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c895f516b7e9dc2 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:48.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.917+0000 7f47cb7fe700 1 -- 172.21.15.77:0/516853026 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b4012630 con 0x7f47cc0c0af0 2022-04-03T19:44:48.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.917+0000 7f47cb7fe700 1 -- 172.21.15.77:0/516853026 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b4012790 con 0x7f47cc0c0af0 2022-04-03T19:44:48.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.917+0000 7f47cb7fe700 1 -- 172.21.15.77:0/516853026 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b4012b30 con 0x7f47cc0c0af0 2022-04-03T19:44:48.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.917+0000 7f47d1357700 1 -- 172.21.15.77:0/516853026 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47cc0c0af0 msgr2=0x7f47cc0c1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:48.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.917+0000 7f47d1357700 1 --2- 172.21.15.77:0/516853026 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47cc0c0af0 0x7f47cc0c1b90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f47b400a880 tx=0x7f47b400a120 comp rx=0 tx=0).stop 2022-04-03T19:44:48.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.917+0000 7f47d1357700 1 -- 172.21.15.77:0/516853026 shutdown_connections 2022-04-03T19:44:48.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.917+0000 7f47d1357700 1 --2- 172.21.15.77:0/516853026 >> v2:172.21.15.77:3300/0 conn(0x7f47cc06b0e0 0x7f47cc0c29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.917+0000 7f47d1357700 1 --2- 172.21.15.77:0/516853026 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47cc0c0af0 0x7f47cc0c1b90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.917+0000 7f47d1357700 1 -- 172.21.15.77:0/516853026 >> 172.21.15.77:0/516853026 conn(0x7f47cc0626d0 msgr2=0x7f47cc062ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:48.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.917+0000 7f47d1357700 1 -- 172.21.15.77:0/516853026 shutdown_connections 2022-04-03T19:44:48.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.917+0000 7f47d1357700 1 -- 172.21.15.77:0/516853026 wait complete. 2022-04-03T19:44:48.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.919+0000 7f47d1357700 1 Processor -- start 2022-04-03T19:44:48.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.919+0000 7f47d1357700 1 -- start start 2022-04-03T19:44:48.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.920+0000 7f47d1357700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47cc06b0e0 0x7f47cc0c3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:48.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.920+0000 7f47d1357700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47cc0c0af0 0x7f47cc169ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:48.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.920+0000 7f47d1357700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47cc0c7410 0x7f47cc0b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:48.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.920+0000 7f47d1357700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc0c3800 con 0x7f47cc06b0e0 2022-04-03T19:44:48.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.920+0000 7f47d1357700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc0c3b40 con 0x7f47cc0c7410 2022-04-03T19:44:48.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.920+0000 7f47d1357700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc0c3de0 con 0x7f47cc0c0af0 2022-04-03T19:44:48.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.920+0000 7f47cbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47cc06b0e0 0x7f47cc0c3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:48.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.920+0000 7f47c3fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47cc0c0af0 0x7f47cc169ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:48.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.920+0000 7f47c3fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47cc0c0af0 0x7f47cc169ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60026/0 (socket says 172.21.15.77:60026) 2022-04-03T19:44:48.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.920+0000 7f47c3fff700 1 -- 172.21.15.77:0/940082421 learned_addr learned my addr 172.21.15.77:0/940082421 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:48.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.921+0000 7f47d0b56700 1 --2- 172.21.15.77:0/940082421 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47cc0c7410 0x7f47cc0b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:48.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.921+0000 7f47c3fff700 1 -- 172.21.15.77:0/940082421 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47cc0c7410 msgr2=0x7f47cc0b40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:48.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.921+0000 7f47c3fff700 1 --2- 172.21.15.77:0/940082421 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47cc0c7410 0x7f47cc0b40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.921+0000 7f47c3fff700 1 -- 172.21.15.77:0/940082421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47cc06b0e0 msgr2=0x7f47cc0c3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:48.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.921+0000 7f47c3fff700 1 --2- 172.21.15.77:0/940082421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47cc06b0e0 0x7f47cc0c3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:48.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.921+0000 7f47c3fff700 1 -- 172.21.15.77:0/940082421 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b400a3c0 con 0x7f47cc0c0af0 2022-04-03T19:44:48.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.921+0000 7f47c3fff700 1 --2- 172.21.15.77:0/940082421 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47cc0c0af0 0x7f47cc169ae0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f47b400a160 tx=0x7f47b4018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:48.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.921+0000 7f47c8ff9700 1 -- 172.21.15.77:0/940082421 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b4012630 con 0x7f47cc0c0af0 2022-04-03T19:44:48.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.922+0000 7f47c8ff9700 1 -- 172.21.15.77:0/940082421 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b4005a90 con 0x7f47cc0c0af0 2022-04-03T19:44:48.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.922+0000 7f47d1357700 1 -- 172.21.15.77:0/940082421 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47cc0c40c0 con 0x7f47cc0c0af0 2022-04-03T19:44:48.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.922+0000 7f47c8ff9700 1 -- 172.21.15.77:0/940082421 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b4005e30 con 0x7f47cc0c0af0 2022-04-03T19:44:48.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.922+0000 7f47d1357700 1 -- 172.21.15.77:0/940082421 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47cc151210 con 0x7f47cc0c0af0 2022-04-03T19:44:48.931 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.923+0000 7f47d1357700 1 -- 172.21.15.77:0/940082421 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47cc0c48c0 con 0x7f47cc0c0af0 2022-04-03T19:44:48.932 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.924+0000 7f47c8ff9700 1 -- 172.21.15.77:0/940082421 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f47b4010050 con 0x7f47cc0c0af0 2022-04-03T19:44:48.932 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.924+0000 7f47c8ff9700 1 --2- 172.21.15.77:0/940082421 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f47b0063ce0 0x7f47b0066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:48.932 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.925+0000 7f47c8ff9700 1 -- 172.21.15.77:0/940082421 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f47b4081d80 con 0x7f47cc0c0af0 2022-04-03T19:44:48.932 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.925+0000 7f47cbfff700 1 --2- 172.21.15.77:0/940082421 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f47b0063ce0 0x7f47b0066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:48.933 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.926+0000 7f47cbfff700 1 --2- 172.21.15.77:0/940082421 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f47b0063ce0 0x7f47b0066190 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f47bc007fd0 tx=0x7f47bc007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:48.933 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:48.928+0000 7f47c8ff9700 1 -- 172.21.15.77:0/940082421 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f47b4050980 con 0x7f47cc0c0af0 2022-04-03T19:44:49.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.126+0000 7f47d1357700 1 -- 172.21.15.77:0/940082421 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f47cc1556e0 con 0x7f47cc0c0af0 2022-04-03T19:44:49.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.127+0000 7f47c8ff9700 1 -- 172.21.15.77:0/940082421 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f47b40160a0 con 0x7f47cc0c0af0 2022-04-03T19:44:49.128 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 64 2022-04-03T19:44:49.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.129+0000 7f47d1357700 1 -- 172.21.15.77:0/940082421 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f47b0063ce0 msgr2=0x7f47b0066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:49.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.129+0000 7f47d1357700 1 --2- 172.21.15.77:0/940082421 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f47b0063ce0 0x7f47b0066190 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f47bc007fd0 tx=0x7f47bc007d30 comp rx=0 tx=0).stop 2022-04-03T19:44:49.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.129+0000 7f47d1357700 1 -- 172.21.15.77:0/940082421 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47cc0c0af0 msgr2=0x7f47cc169ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:49.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.129+0000 7f47d1357700 1 --2- 172.21.15.77:0/940082421 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47cc0c0af0 0x7f47cc169ae0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f47b400a160 tx=0x7f47b4018000 comp rx=0 tx=0).stop 2022-04-03T19:44:49.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.130+0000 7f47d1357700 1 -- 172.21.15.77:0/940082421 shutdown_connections 2022-04-03T19:44:49.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.130+0000 7f47d1357700 1 --2- 172.21.15.77:0/940082421 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47cc0c7410 0x7f47cc0b40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:49.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.130+0000 7f47d1357700 1 --2- 172.21.15.77:0/940082421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47cc06b0e0 0x7f47cc0c3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:49.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.130+0000 7f47d1357700 1 --2- 172.21.15.77:0/940082421 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f47b0063ce0 0x7f47b0066190 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:49.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.130+0000 7f47d1357700 1 --2- 172.21.15.77:0/940082421 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47cc0c0af0 0x7f47cc169ae0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:49.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.130+0000 7f47d1357700 1 -- 172.21.15.77:0/940082421 >> 172.21.15.77:0/940082421 conn(0x7f47cc0626d0 msgr2=0x7f47cc06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:49.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.130+0000 7f47d1357700 1 -- 172.21.15.77:0/940082421 shutdown_connections 2022-04-03T19:44:49.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.130+0000 7f47d1357700 1 -- 172.21.15.77:0/940082421 wait complete. 2022-04-03T19:44:49.154 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:44:49.351 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.351+0000 7f42e759e700 1 Processor -- start 2022-04-03T19:44:49.351 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.351+0000 7f42e759e700 1 -- start start 2022-04-03T19:44:49.353 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.352+0000 7f42e759e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42e80c3270 0x7f42e80c2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:49.353 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.352+0000 7f42e759e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f42e8069d10 0x7f42e8151760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:49.353 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.352+0000 7f42e759e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e80c4b90 con 0x7f42e80c3640 2022-04-03T19:44:49.353 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.352+0000 7f42e759e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42e80c3d10 con 0x7f42e8069d10 2022-04-03T19:44:49.354 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.352+0000 7f42e759e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e80c2a40 con 0x7f42e80c3270 2022-04-03T19:44:49.354 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.353+0000 7f42e659c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42e80c3270 0x7f42e80c2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:49.354 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.353+0000 7f42e5d9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f42e80c3640 0x7f42e8069600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38374/0 (socket says 172.21.15.77:38374) 2022-04-03T19:44:49.354 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.353+0000 7f42e5d9b700 1 -- 172.21.15.77:0/2937294838 learned_addr learned my addr 172.21.15.77:0/2937294838 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:44:49.355 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.353+0000 7f42e6d9d700 1 --2- 172.21.15.77:0/2937294838 >> v2:172.21.15.77:3300/0 conn(0x7f42e8069d10 0x7f42e8151760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:49.356 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.353+0000 7f42e659c700 1 -- 172.21.15.77:0/2937294838 >> v2:172.21.15.77:3300/0 conn(0x7f42e8069d10 msgr2=0x7f42e8151760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:49.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.353+0000 7f42e659c700 1 --2- 172.21.15.77:0/2937294838 >> v2:172.21.15.77:3300/0 conn(0x7f42e8069d10 0x7f42e8151760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:49.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.353+0000 7f42e659c700 1 -- 172.21.15.77:0/2937294838 >> v1:172.21.15.77:6789/0 conn(0x7f42e80c3640 legacy=0x7f42e8069600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:49.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.353+0000 7f42e659c700 1 -- 172.21.15.77:0/2937294838 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e80c3f80 con 0x7f42e80c3270 2022-04-03T19:44:49.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.353+0000 7f42e659c700 1 --2- 172.21.15.77:0/2937294838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42e80c3270 0x7f42e80c2500 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f42d800ba00 tx=0x7f42d800b300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d53aa1eca3f76e2 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:49.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.354+0000 7f42e559a700 1 -- 172.21.15.77:0/2937294838 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d800da70 con 0x7f42e80c3270 2022-04-03T19:44:49.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.354+0000 7f42e559a700 1 -- 172.21.15.77:0/2937294838 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d800dbd0 con 0x7f42e80c3270 2022-04-03T19:44:49.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.354+0000 7f42e559a700 1 -- 172.21.15.77:0/2937294838 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d8019480 con 0x7f42e80c3270 2022-04-03T19:44:49.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.354+0000 7f42e759e700 1 -- 172.21.15.77:0/2937294838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42e80c3270 msgr2=0x7f42e80c2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:49.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.354+0000 7f42e759e700 1 --2- 172.21.15.77:0/2937294838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42e80c3270 0x7f42e80c2500 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f42d800ba00 tx=0x7f42d800b300 comp rx=0 tx=0).stop 2022-04-03T19:44:49.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.354+0000 7f42e759e700 1 -- 172.21.15.77:0/2937294838 shutdown_connections 2022-04-03T19:44:49.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.354+0000 7f42e759e700 1 --2- 172.21.15.77:0/2937294838 >> v2:172.21.15.77:3300/0 conn(0x7f42e8069d10 0x7f42e8151760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:49.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.354+0000 7f42e759e700 1 --2- 172.21.15.77:0/2937294838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42e80c3270 0x7f42e80c2500 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:49.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.354+0000 7f42e759e700 1 -- 172.21.15.77:0/2937294838 >> 172.21.15.77:0/2937294838 conn(0x7f42e8066da0 msgr2=0x7f42e80671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:49.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.354+0000 7f42e759e700 1 -- 172.21.15.77:0/2937294838 shutdown_connections 2022-04-03T19:44:49.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.354+0000 7f42e759e700 1 -- 172.21.15.77:0/2937294838 wait complete. 2022-04-03T19:44:49.360 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.356+0000 7f42e759e700 1 Processor -- start 2022-04-03T19:44:49.360 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.356+0000 7f42e759e700 1 -- start start 2022-04-03T19:44:49.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.357+0000 7f42e759e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42e8069d10 0x7f42e8165d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:49.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.357+0000 7f42e759e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42e80c3270 0x7f42e80ba090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:49.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.357+0000 7f42e759e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42e80c3640 0x7f42e80b8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:49.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.357+0000 7f42e759e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e80bb000 con 0x7f42e8069d10 2022-04-03T19:44:49.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.357+0000 7f42e759e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e80ba5d0 con 0x7f42e80c3270 2022-04-03T19:44:49.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.357+0000 7f42e759e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e80b88c0 con 0x7f42e80c3640 2022-04-03T19:44:49.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.357+0000 7f42e659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42e8069d10 0x7f42e8165d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:49.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.358+0000 7f42e659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42e8069d10 0x7f42e8165d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42716/0 (socket says 172.21.15.77:42716) 2022-04-03T19:44:49.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.358+0000 7f42e659c700 1 -- 172.21.15.77:0/1869005320 learned_addr learned my addr 172.21.15.77:0/1869005320 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:49.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.358+0000 7f42e5d9b700 1 --2- 172.21.15.77:0/1869005320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42e80c3270 0x7f42e80ba090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:49.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.358+0000 7f42e6d9d700 1 --2- 172.21.15.77:0/1869005320 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42e80c3640 0x7f42e80b8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:49.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.358+0000 7f42e659c700 1 -- 172.21.15.77:0/1869005320 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42e80c3640 msgr2=0x7f42e80b8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:49.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.358+0000 7f42e659c700 1 --2- 172.21.15.77:0/1869005320 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42e80c3640 0x7f42e80b8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:49.366 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.358+0000 7f42e659c700 1 -- 172.21.15.77:0/1869005320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42e80c3270 msgr2=0x7f42e80ba090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:49.366 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.358+0000 7f42e659c700 1 --2- 172.21.15.77:0/1869005320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42e80c3270 0x7f42e80ba090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:49.366 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.358+0000 7f42e659c700 1 -- 172.21.15.77:0/1869005320 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d800b540 con 0x7f42e8069d10 2022-04-03T19:44:49.366 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.358+0000 7f42e659c700 1 --2- 172.21.15.77:0/1869005320 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42e8069d10 0x7f42e8165d10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f42d800d5d0 tx=0x7f42d801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:49.366 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.359+0000 7f42d67fc700 1 -- 172.21.15.77:0/1869005320 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d800da70 con 0x7f42e8069d10 2022-04-03T19:44:49.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.359+0000 7f42d67fc700 1 -- 172.21.15.77:0/1869005320 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d8010470 con 0x7f42e8069d10 2022-04-03T19:44:49.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.359+0000 7f42e759e700 1 -- 172.21.15.77:0/1869005320 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e80b8300 con 0x7f42e8069d10 2022-04-03T19:44:49.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.359+0000 7f42d67fc700 1 -- 172.21.15.77:0/1869005320 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d8010810 con 0x7f42e8069d10 2022-04-03T19:44:49.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.359+0000 7f42e759e700 1 -- 172.21.15.77:0/1869005320 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e80bb4f0 con 0x7f42e8069d10 2022-04-03T19:44:49.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.360+0000 7f42e759e700 1 -- 172.21.15.77:0/1869005320 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e8166250 con 0x7f42e8069d10 2022-04-03T19:44:49.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.361+0000 7f42d67fc700 1 -- 172.21.15.77:0/1869005320 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f42d802c430 con 0x7f42e8069d10 2022-04-03T19:44:49.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.361+0000 7f42d67fc700 1 --2- 172.21.15.77:0/1869005320 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f42c4063ce0 0x7f42c4066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:49.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.361+0000 7f42d67fc700 1 -- 172.21.15.77:0/1869005320 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f42d8014070 con 0x7f42e8069d10 2022-04-03T19:44:49.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.362+0000 7f42e5d9b700 1 --2- 172.21.15.77:0/1869005320 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f42c4063ce0 0x7f42c4066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:49.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.362+0000 7f42e5d9b700 1 --2- 172.21.15.77:0/1869005320 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f42c4063ce0 0x7f42c4066190 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f42e80bbf40 tx=0x7f42dc007dc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:49.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.366+0000 7f42d67fc700 1 -- 172.21.15.77:0/1869005320 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f42d8051270 con 0x7f42e8069d10 2022-04-03T19:44:49.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.597+0000 7f42e759e700 1 -- 172.21.15.77:0/1869005320 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42e80c4260 con 0x7f42e8069d10 2022-04-03T19:44:49.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.598+0000 7f42d67fc700 1 -- 172.21.15.77:0/1869005320 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6107 (secure 0 0 0) 0x7f42d8054890 con 0x7f42e8069d10 2022-04-03T19:44:49.599 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:49.599 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4303,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:44:27.295743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4327":{"gid":4327,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6837/3665969042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3665969042},{"type":"v1","addr":"172.21.15.77:6837","nonce":3665969042}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:44:49.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.600+0000 7f42e759e700 1 -- 172.21.15.77:0/1869005320 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f42c4063ce0 msgr2=0x7f42c4066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:49.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.600+0000 7f42e759e700 1 --2- 172.21.15.77:0/1869005320 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f42c4063ce0 0x7f42c4066190 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f42e80bbf40 tx=0x7f42dc007dc0 comp rx=0 tx=0).stop 2022-04-03T19:44:49.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.600+0000 7f42e759e700 1 -- 172.21.15.77:0/1869005320 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42e8069d10 msgr2=0x7f42e8165d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:49.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.600+0000 7f42e759e700 1 --2- 172.21.15.77:0/1869005320 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42e8069d10 0x7f42e8165d10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f42d800d5d0 tx=0x7f42d801b000 comp rx=0 tx=0).stop 2022-04-03T19:44:49.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.600+0000 7f42e759e700 1 -- 172.21.15.77:0/1869005320 shutdown_connections 2022-04-03T19:44:49.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.600+0000 7f42e759e700 1 --2- 172.21.15.77:0/1869005320 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f42c4063ce0 0x7f42c4066190 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:49.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.600+0000 7f42e759e700 1 --2- 172.21.15.77:0/1869005320 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42e80c3640 0x7f42e80b8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:49.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.600+0000 7f42e759e700 1 --2- 172.21.15.77:0/1869005320 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42e8069d10 0x7f42e8165d10 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:49.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.600+0000 7f42e759e700 1 --2- 172.21.15.77:0/1869005320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42e80c3270 0x7f42e80ba090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:49.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.600+0000 7f42e759e700 1 -- 172.21.15.77:0/1869005320 >> 172.21.15.77:0/1869005320 conn(0x7f42e8066da0 msgr2=0x7f42e80c59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:49.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.601+0000 7f42e759e700 1 -- 172.21.15.77:0/1869005320 shutdown_connections 2022-04-03T19:44:49.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.601+0000 7f42e759e700 1 -- 172.21.15.77:0/1869005320 wait complete. 2022-04-03T19:44:49.604 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2022-04-03T19:44:49.626 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-04-03T19:44:49.823 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.823+0000 7f5fd8a24700 1 Processor -- start 2022-04-03T19:44:49.823 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.823+0000 7f5fd8a24700 1 -- start start 2022-04-03T19:44:49.825 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.824+0000 7f5fd8a24700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5fd406d3f0 0x7f5fd406d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:49.825 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.824+0000 7f5fd8a24700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fd40c5930 0x7f5fd4155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:49.825 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.824+0000 7f5fd8a24700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd40c2f80 con 0x7f5fd4156120 2022-04-03T19:44:49.826 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.824+0000 7f5fd8a24700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fd40c3260 con 0x7f5fd406d3f0 2022-04-03T19:44:49.826 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.824+0000 7f5fd8a24700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd40c3540 con 0x7f5fd40c5930 2022-04-03T19:44:49.826 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.825+0000 7f5fd37fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5fd406d3f0 0x7f5fd406d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:49.826 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.825+0000 7f5fd2ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fd40c5930 0x7f5fd4155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:49.827 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.825+0000 7f5fd37fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5fd406d3f0 0x7f5fd406d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42718/0 (socket says 172.21.15.77:42718) 2022-04-03T19:44:49.827 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.825+0000 7f5fd37fe700 1 -- 172.21.15.77:0/3751899451 learned_addr learned my addr 172.21.15.77:0/3751899451 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:49.830 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.825+0000 7f5fd37fe700 1 -- 172.21.15.77:0/3751899451 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fd40c5930 msgr2=0x7f5fd4155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:44:49.830 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.825+0000 7f5fd37fe700 1 --2- 172.21.15.77:0/3751899451 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fd40c5930 0x7f5fd4155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:49.830 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.825+0000 7f5fd37fe700 1 -- 172.21.15.77:0/3751899451 >> v1:172.21.15.77:6789/0 conn(0x7f5fd4156120 legacy=0x7f5fd406c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:49.830 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.825+0000 7f5fd37fe700 1 -- 172.21.15.77:0/3751899451 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd40c3820 con 0x7f5fd406d3f0 2022-04-03T19:44:49.831 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.825+0000 7f5fd27fc700 1 -- 172.21.15.77:0/3751899451 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836436161 0 0) 0x7f5fd40c2f80 con 0x7f5fd4156120 2022-04-03T19:44:49.831 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.825+0000 7f5fd2ffd700 1 --2- 172.21.15.77:0/3751899451 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fd40c5930 0x7f5fd4155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:49.831 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.825+0000 7f5fd37fe700 1 --2- 172.21.15.77:0/3751899451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fd406d3f0 0x7f5fd406d7c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4009830 tx=0x7f5fc40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9caaa12c99cf0a41 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:49.831 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.826+0000 7f5fd27fc700 1 -- 172.21.15.77:0/3751899451 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4004030 con 0x7f5fd406d3f0 2022-04-03T19:44:49.832 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.826+0000 7f5fd27fc700 1 -- 172.21.15.77:0/3751899451 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc4004190 con 0x7f5fd406d3f0 2022-04-03T19:44:49.832 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.826+0000 7f5fd27fc700 1 -- 172.21.15.77:0/3751899451 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4011630 con 0x7f5fd406d3f0 2022-04-03T19:44:49.832 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.826+0000 7f5fd8a24700 1 -- 172.21.15.77:0/3751899451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fd406d3f0 msgr2=0x7f5fd406d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:49.832 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.826+0000 7f5fd8a24700 1 --2- 172.21.15.77:0/3751899451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fd406d3f0 0x7f5fd406d7c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4009830 tx=0x7f5fc40090f0 comp rx=0 tx=0).stop 2022-04-03T19:44:49.832 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.826+0000 7f5fd8a24700 1 -- 172.21.15.77:0/3751899451 shutdown_connections 2022-04-03T19:44:49.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.826+0000 7f5fd8a24700 1 --2- 172.21.15.77:0/3751899451 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fd406d3f0 0x7f5fd406d7c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:49.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.826+0000 7f5fd8a24700 1 --2- 172.21.15.77:0/3751899451 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fd40c5930 0x7f5fd4155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:49.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.826+0000 7f5fd8a24700 1 -- 172.21.15.77:0/3751899451 >> 172.21.15.77:0/3751899451 conn(0x7f5fd40cb720 msgr2=0x7f5fd40cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:49.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.826+0000 7f5fd8a24700 1 -- 172.21.15.77:0/3751899451 shutdown_connections 2022-04-03T19:44:49.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.826+0000 7f5fd8a24700 1 -- 172.21.15.77:0/3751899451 wait complete. 2022-04-03T19:44:49.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.828+0000 7f5fd8a24700 1 Processor -- start 2022-04-03T19:44:49.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.828+0000 7f5fd8a24700 1 -- start start 2022-04-03T19:44:49.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.829+0000 7f5fd8a24700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5fd406d3f0 0x7f5fd414af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:49.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.829+0000 7f5fd8a24700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fd40c5930 0x7f5fd4150b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:49.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.829+0000 7f5fd8a24700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fd4156120 0x7f5fd40ca150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:49.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.829+0000 7f5fd8a24700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd4169350 con 0x7f5fd40c5930 2022-04-03T19:44:49.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.829+0000 7f5fd8a24700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd414b470 con 0x7f5fd406d3f0 2022-04-03T19:44:49.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.829+0000 7f5fd8a24700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd41510d0 con 0x7f5fd4156120 2022-04-03T19:44:49.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.829+0000 7f5fd2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fd40c5930 0x7f5fd4150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:49.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.829+0000 7f5fd37fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5fd406d3f0 0x7f5fd414af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:49.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.829+0000 7f5fd37fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5fd406d3f0 0x7f5fd414af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37916/0 (socket says 172.21.15.77:37916) 2022-04-03T19:44:49.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.829+0000 7f5fd37fe700 1 -- 172.21.15.77:0/10764563 learned_addr learned my addr 172.21.15.77:0/10764563 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:44:49.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.829+0000 7f5fd3fff700 1 --2- 172.21.15.77:0/10764563 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fd4156120 0x7f5fd40ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:49.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.830+0000 7f5fd2ffd700 1 -- 172.21.15.77:0/10764563 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fd4156120 msgr2=0x7f5fd40ca150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:49.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.830+0000 7f5fd2ffd700 1 --2- 172.21.15.77:0/10764563 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fd4156120 0x7f5fd40ca150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:49.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.830+0000 7f5fd2ffd700 1 -- 172.21.15.77:0/10764563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5fd406d3f0 msgr2=0x7f5fd414af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:49.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.830+0000 7f5fd2ffd700 1 --2- 172.21.15.77:0/10764563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5fd406d3f0 0x7f5fd414af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:49.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.830+0000 7f5fd2ffd700 1 -- 172.21.15.77:0/10764563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc80092a0 con 0x7f5fd40c5930 2022-04-03T19:44:49.840 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.830+0000 7f5fd37fe700 1 --2- 172.21.15.77:0/10764563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5fd406d3f0 0x7f5fd414af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:44:49.840 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.830+0000 7f5fd2ffd700 1 --2- 172.21.15.77:0/10764563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fd40c5930 0x7f5fd4150b90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5fc800e950 tx=0x7f5fc8009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:49.840 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.830+0000 7f5fbb7fe700 1 -- 172.21.15.77:0/10764563 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8013680 con 0x7f5fd40c5930 2022-04-03T19:44:49.840 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.830+0000 7f5fbb7fe700 1 -- 172.21.15.77:0/10764563 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc80137e0 con 0x7f5fd40c5930 2022-04-03T19:44:49.840 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.830+0000 7f5fd8a24700 1 -- 172.21.15.77:0/10764563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc4009370 con 0x7f5fd40c5930 2022-04-03T19:44:49.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.830+0000 7f5fbb7fe700 1 -- 172.21.15.77:0/10764563 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8013ba0 con 0x7f5fd40c5930 2022-04-03T19:44:49.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.831+0000 7f5fd8a24700 1 -- 172.21.15.77:0/10764563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd41686d0 con 0x7f5fd40c5930 2022-04-03T19:44:49.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.832+0000 7f5fd8a24700 1 -- 172.21.15.77:0/10764563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fd4169490 con 0x7f5fd40c5930 2022-04-03T19:44:49.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.832+0000 7f5fbb7fe700 1 -- 172.21.15.77:0/10764563 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f5fc8019030 con 0x7f5fd40c5930 2022-04-03T19:44:49.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.833+0000 7f5fbb7fe700 1 --2- 172.21.15.77:0/10764563 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5fb4063ce0 0x7f5fb4066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:44:49.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.833+0000 7f5fbb7fe700 1 -- 172.21.15.77:0/10764563 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f5fc8080810 con 0x7f5fd40c5930 2022-04-03T19:44:49.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.834+0000 7f5fd37fe700 1 --2- 172.21.15.77:0/10764563 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5fb4063ce0 0x7f5fb4066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:44:49.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.834+0000 7f5fd37fe700 1 --2- 172.21.15.77:0/10764563 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5fb4063ce0 0x7f5fb4066190 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4009310 tx=0x7f5fc4016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:44:49.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:49.837+0000 7f5fbb7fe700 1 -- 172.21.15.77:0/10764563 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f5fd4169490 con 0x7f5fd40c5930 2022-04-03T19:44:50.032 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:50.031+0000 7f5fd8a24700 1 -- 172.21.15.77:0/10764563 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fd4169490 con 0x7f5fd40c5930 2022-04-03T19:44:50.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:50.032+0000 7f5fbb7fe700 1 -- 172.21.15.77:0/10764563 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16845 (secure 0 0 0) 0x7f5fd4169490 con 0x7f5fd40c5930 2022-04-03T19:44:50.033 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:44:50.034 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"fsid":"89c3a3f4-599d-4c4a-8f5a-7180eafb7292","created":"2022-04-03T19:43:54.101559+0000","modified":"2022-04-03T19:44:22.278254+0000","last_up_change":"2022-04-03T19:44:05.589174+0000","last_in_change":"2022-04-03T19:43:56.729760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-04-03T19:44:07.567638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-04-03T19:44:13.149481+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-04-03T19:44:18.823926+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-04-03T19:44:20.275479+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f35f5195-3729-4d78-85d1-0539f259283f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6823","nonce":20252},{"type":"v1","addr":"172.21.15.77:6825","nonce":20252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":20252},{"type":"v1","addr":"172.21.15.77:6827","nonce":20252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":20252},{"type":"v1","addr":"172.21.15.77:6831","nonce":20252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":20252},{"type":"v1","addr":"172.21.15.77:6829","nonce":20252}]},"public_addr":"172.21.15.77:6825/20252","cluster_addr":"172.21.15.77:6827/20252","heartbeat_back_addr":"172.21.15.77:6831/20252","heartbeat_front_addr":"172.21.15.77:6829/20252","state":["exists","up"]},{"osd":1,"uuid":"ab1c89dc-57ce-4cc4-b8c7-9b0c19939601","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":20249},{"type":"v1","addr":"172.21.15.77:6805","nonce":20249}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":20249},{"type":"v1","addr":"172.21.15.77:6809","nonce":20249}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6817","nonce":20249},{"type":"v1","addr":"172.21.15.77:6819","nonce":20249}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":20249},{"type":"v1","addr":"172.21.15.77:6815","nonce":20249}]},"public_addr":"172.21.15.77:6805/20249","cluster_addr":"172.21.15.77:6809/20249","heartbeat_back_addr":"172.21.15.77:6819/20249","heartbeat_front_addr":"172.21.15.77:6815/20249","state":["exists","up"]},{"osd":2,"uuid":"351e218b-7749-4ee9-b44a-0a5b52fb9be1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":20248},{"type":"v1","addr":"172.21.15.77:6801","nonce":20248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":20248},{"type":"v1","addr":"172.21.15.77:6803","nonce":20248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":20248},{"type":"v1","addr":"172.21.15.77:6813","nonce":20248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":20248},{"type":"v1","addr":"172.21.15.77:6808","nonce":20248}]},"public_addr":"172.21.15.77:6801/20248","cluster_addr":"172.21.15.77:6803/20248","heartbeat_back_addr":"172.21.15.77:6813/20248","heartbeat_front_addr":"172.21.15.77:6808/20248","state":["exists","up"]},{"osd":3,"uuid":"1ba6ec8a-7cbf-4c12-8e21-c3097a263ae8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":20250},{"type":"v1","addr":"172.21.15.77:6814","nonce":20250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":20250},{"type":"v1","addr":"172.21.15.77:6818","nonce":20250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":20250},{"type":"v1","addr":"172.21.15.77:6824","nonce":20250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":20250},{"type":"v1","addr":"172.21.15.77:6821","nonce":20250}]},"public_addr":"172.21.15.77:6814/20250","cluster_addr":"172.21.15.77:6818/20250","heartbeat_back_addr":"172.21.15.77:6824/20250","heartbeat_front_addr":"172.21.15.77:6821/20250","state":["exists","up"]},{"osd":4,"uuid":"3faf4c09-846c-4133-856b-7f53c535a67d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":25551},{"type":"v1","addr":"172.21.15.186:6822","nonce":25551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":25551},{"type":"v1","addr":"172.21.15.186:6826","nonce":25551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":25551},{"type":"v1","addr":"172.21.15.186:6831","nonce":25551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":25551},{"type":"v1","addr":"172.21.15.186:6829","nonce":25551}]},"public_addr":"172.21.15.186:6822/25551","cluster_addr":"172.21.15.186:6826/25551","heartbeat_back_addr":"172.21.15.186:6831/25551","heartbeat_front_addr":"172.21.15.186:6829/25551","state":["exists","up"]},{"osd":5,"uuid":"ffe41d00-d0aa-42f4-82c1-5cced13c1bc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":25541},{"type":"v1","addr":"172.21.15.186:6809","nonce":25541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":25541},{"type":"v1","addr":"172.21.15.186:6811","nonce":25541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":25541},{"type":"v1","addr":"172.21.15.186:6815","nonce":25541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":25541},{"type":"v1","addr":"172.21.15.186:6813","nonce":25541}]},"public_addr":"172.21.15.186:6809/25541","cluster_addr":"172.21.15.186:6811/25541","heartbeat_back_addr":"172.21.15.186:6815/25541","heartbeat_front_addr":"172.21.15.186:6813/25541","state":["exists","up"]},{"osd":6,"uuid":"b0344ee8-deca-4e9b-934c-5866d90d0073","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":25542},{"type":"v1","addr":"172.21.15.186:6801","nonce":25542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":25542},{"type":"v1","addr":"172.21.15.186:6803","nonce":25542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":25542},{"type":"v1","addr":"172.21.15.186:6807","nonce":25542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":25542},{"type":"v1","addr":"172.21.15.186:6805","nonce":25542}]},"public_addr":"172.21.15.186:6801/25542","cluster_addr":"172.21.15.186:6803/25542","heartbeat_back_addr":"172.21.15.186:6807/25542","heartbeat_front_addr":"172.21.15.186:6805/25542","state":["exists","up"]},{"osd":7,"uuid":"0eb0e5d3-3dad-4f49-9b50-3b24a3924bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":25572},{"type":"v1","addr":"172.21.15.186:6817","nonce":25572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":25572},{"type":"v1","addr":"172.21.15.186:6819","nonce":25572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6825","nonce":25572},{"type":"v1","addr":"172.21.15.186:6827","nonce":25572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6821","nonce":25572},{"type":"v1","addr":"172.21.15.186:6823","nonce":25572}]},"public_addr":"172.21.15.186:6817/25572","cluster_addr":"172.21.15.186:6819/25572","heartbeat_back_addr":"172.21.15.186:6827/25572","heartbeat_front_addr":"172.21.15.186:6823/25572","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.143602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.490366+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.361292+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:02.343544+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969619+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.905429+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.969298+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2022-04-03T19:44:01.949524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-04-03T19:44:50.036 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:50.035+0000 7f5fd8a24700 1 -- 172.21.15.77:0/10764563 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5fb4063ce0 msgr2=0x7f5fb4066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:50.036 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:50.035+0000 7f5fd8a24700 1 --2- 172.21.15.77:0/10764563 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5fb4063ce0 0x7f5fb4066190 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4009310 tx=0x7f5fc4016000 comp rx=0 tx=0).stop 2022-04-03T19:44:50.036 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:50.035+0000 7f5fd8a24700 1 -- 172.21.15.77:0/10764563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fd40c5930 msgr2=0x7f5fd4150b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:44:50.036 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:50.035+0000 7f5fd8a24700 1 --2- 172.21.15.77:0/10764563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fd40c5930 0x7f5fd4150b90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5fc800e950 tx=0x7f5fc8009060 comp rx=0 tx=0).stop 2022-04-03T19:44:50.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:50.035+0000 7f5fd8a24700 1 -- 172.21.15.77:0/10764563 shutdown_connections 2022-04-03T19:44:50.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:50.035+0000 7f5fd8a24700 1 --2- 172.21.15.77:0/10764563 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5fb4063ce0 0x7f5fb4066190 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:50.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:50.035+0000 7f5fd8a24700 1 --2- 172.21.15.77:0/10764563 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fd4156120 0x7f5fd40ca150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:50.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:50.035+0000 7f5fd8a24700 1 --2- 172.21.15.77:0/10764563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5fd406d3f0 0x7f5fd414af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:50.038 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:50.035+0000 7f5fd8a24700 1 --2- 172.21.15.77:0/10764563 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5fd40c5930 0x7f5fd4150b90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:44:50.038 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:50.035+0000 7f5fd8a24700 1 -- 172.21.15.77:0/10764563 >> 172.21.15.77:0/10764563 conn(0x7f5fd40cb720 msgr2=0x7f5fd4153210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:44:50.038 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:50.035+0000 7f5fd8a24700 1 -- 172.21.15.77:0/10764563 shutdown_connections 2022-04-03T19:44:50.038 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:44:50.035+0000 7f5fd8a24700 1 -- 172.21.15.77:0/10764563 wait complete. 2022-04-03T19:44:50.060 DEBUG:tasks.mds_thrash:Yielding 2022-04-03T19:44:50.060 INFO:teuthology.run_tasks:Running task workunit... 2022-04-03T19:44:50.075 INFO:tasks.workunit:Pulling workunits from ref cfb8f943163b374162da0d7b0240f267dd46e4e1 2022-04-03T19:44:50.076 INFO:tasks.workunit:Making a separate scratch dir for every client... 2022-04-03T19:44:50.076 INFO:tasks.workunit:timeout=3h 2022-04-03T19:44:50.077 INFO:tasks.workunit:cleanup=True 2022-04-03T19:44:50.077 DEBUG:teuthology.orchestra.run.smithi077:> stat -- /home/ubuntu/cephtest/mnt.0 2022-04-03T19:44:50.078 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2022-04-03T19:44:50.078 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 96 secs before thrashing 2022-04-03T19:44:50.125 INFO:teuthology.orchestra.run.smithi077.stdout: File: /home/ubuntu/cephtest/mnt.0 2022-04-03T19:44:50.125 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2022-04-03T19:44:50.126 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 2ah/42d Inode: 1 Links: 2 2022-04-03T19:44:50.126 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2022-04-03T19:44:50.126 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:cephfs_t:s0 2022-04-03T19:44:50.126 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2022-04-03T19:44:50.126 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2022-04-03 19:44:26.280436771 +0000 2022-04-03T19:44:50.127 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2022-04-03 19:44:40.758272188 +0000 2022-04-03T19:44:50.127 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2022-04-03T19:44:50.127 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2022-04-03T19:44:50.127 DEBUG:teuthology.orchestra.run.smithi077:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2022-04-03T19:44:50.203 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout cfb8f943163b374162da0d7b0240f267dd46e4e1 2022-04-03T19:44:50.250 INFO:tasks.workunit.client.0.smithi077.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2022-04-03T19:45:14.722 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:45:14.727 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:45:37.938 INFO:tasks.workunit.client.0.smithi077.stderr:Updating files: 64% (6277/9786) Updating files: 65% (6361/9786) Updating files: 66% (6459/9786) Updating files: 67% (6557/9786) Updating files: 68% (6655/9786) Updating files: 69% (6753/9786) Updating files: 70% (6851/9786) Updating files: 71% (6949/9786) Updating files: 72% (7046/9786) Updating files: 73% (7144/9786) Updating files: 74% (7242/9786) Updating files: 75% (7340/9786) Updating files: 76% (7438/9786) Updating files: 77% (7536/9786) Updating files: 78% (7634/9786) Updating files: 79% (7731/9786) Updating files: 80% (7829/9786) Updating files: 81% (7927/9786) Updating files: 82% (8025/9786) Updating files: 83% (8123/9786) Updating files: 84% (8221/9786) Updating files: 85% (8319/9786) Updating files: 86% (8416/9786) Updating files: 87% (8514/9786) Updating files: 88% (8612/9786) Updating files: 89% (8710/9786) Updating files: 90% (8808/9786) Updating files: 91% (8906/9786) Updating files: 92% (9004/9786) Updating files: 93% (9101/9786) Updating files: 94% (9199/9786) Updating files: 95% (9297/9786) Updating files: 96% (9395/9786) Updating files: 97% (9493/9786) Updating files: 98% (9591/9786) Updating files: 99% (9689/9786) Updating files: 100% (9786/9786) Updating files: 100% (9786/9786), done. 2022-04-03T19:45:39.915 INFO:tasks.workunit.client.0.smithi077.stderr:Note: switching to 'cfb8f943163b374162da0d7b0240f267dd46e4e1'. 2022-04-03T19:45:39.916 INFO:tasks.workunit.client.0.smithi077.stderr: 2022-04-03T19:45:39.916 INFO:tasks.workunit.client.0.smithi077.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-04-03T19:45:39.916 INFO:tasks.workunit.client.0.smithi077.stderr:changes and commit them, and you can discard any commits you make in this 2022-04-03T19:45:39.916 INFO:tasks.workunit.client.0.smithi077.stderr:state without impacting any branches by switching back to a branch. 2022-04-03T19:45:39.917 INFO:tasks.workunit.client.0.smithi077.stderr: 2022-04-03T19:45:39.917 INFO:tasks.workunit.client.0.smithi077.stderr:If you want to create a new branch to retain commits you create, you may 2022-04-03T19:45:39.917 INFO:tasks.workunit.client.0.smithi077.stderr:do so (now or later) by using -c with the switch command. Example: 2022-04-03T19:45:39.917 INFO:tasks.workunit.client.0.smithi077.stderr: 2022-04-03T19:45:39.918 INFO:tasks.workunit.client.0.smithi077.stderr: git switch -c 2022-04-03T19:45:39.918 INFO:tasks.workunit.client.0.smithi077.stderr: 2022-04-03T19:45:39.918 INFO:tasks.workunit.client.0.smithi077.stderr:Or undo this operation with: 2022-04-03T19:45:39.918 INFO:tasks.workunit.client.0.smithi077.stderr: 2022-04-03T19:45:39.918 INFO:tasks.workunit.client.0.smithi077.stderr: git switch - 2022-04-03T19:45:39.919 INFO:tasks.workunit.client.0.smithi077.stderr: 2022-04-03T19:45:39.919 INFO:tasks.workunit.client.0.smithi077.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-04-03T19:45:39.919 INFO:tasks.workunit.client.0.smithi077.stderr: 2022-04-03T19:45:39.919 INFO:tasks.workunit.client.0.smithi077.stderr:HEAD is now at cfb8f943163 mount.ceph: remove `ms_mode' mount option when switching to old-syntax 2022-04-03T19:45:39.959 DEBUG:teuthology.orchestra.run.smithi077:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2022-04-03T19:45:40.010 INFO:tasks.workunit.client.0.smithi077.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2022-04-03T19:45:40.016 INFO:tasks.workunit.client.0.smithi077.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-04-03T19:45:40.016 INFO:tasks.workunit.client.0.smithi077.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2022-04-03T19:45:40.626 INFO:tasks.workunit.client.0.smithi077.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2022-04-03T19:45:40.733 INFO:tasks.workunit.client.0.smithi077.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2022-04-03T19:45:40.832 INFO:tasks.workunit.client.0.smithi077.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-04-03T19:45:40.837 INFO:tasks.workunit.client.0.smithi077.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-04-03T19:45:40.837 INFO:tasks.workunit.client.0.smithi077.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2022-04-03T19:45:40.916 INFO:tasks.workunit.client.0.smithi077.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-04-03T19:45:40.926 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2022-04-03T19:45:40.927 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2022-04-03T19:45:40.971 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2022-04-03T19:45:40.972 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2022-04-03T19:45:40.972 DEBUG:teuthology.orchestra.run.smithi077:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2022-04-03T19:45:41.069 INFO:tasks.workunit.client.0.smithi077.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2022-04-03T19:45:44.943 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:45:44.948 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:45:46.128 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir foo 2022-04-03T19:45:46.136 INFO:tasks.workunit.client.0.smithi077.stderr:+ cp linux-2.6.33.tar.bz2 foo 2022-04-03T19:45:46.258 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir foo/.snap/barsnap 2022-04-03T19:45:46.269 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm foo/linux-2.6.33.tar.bz2 2022-04-03T19:45:46.278 INFO:tasks.workunit.client.0.smithi077.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2022-04-03T19:45:46.927 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo 'passed: files are identical' 2022-04-03T19:45:46.928 INFO:tasks.workunit.client.0.smithi077.stdout:passed: files are identical 2022-04-03T19:45:46.928 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir foo/.snap/barsnap 2022-04-03T19:45:46.980 INFO:tasks.workunit.client.0.smithi077.stdout:OK 2022-04-03T19:45:46.981 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo OK 2022-04-03T19:45:46.982 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-03T19:45:46.982 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-03T19:45:47.063 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2022-04-03T19:45:47.063 DEBUG:teuthology.orchestra.run.smithi077:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2022-04-03T19:45:47.124 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo 1 2022-04-03T19:45:47.125 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo 2 2022-04-03T19:45:47.126 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo 3 2022-04-03T19:45:47.126 INFO:tasks.workunit.client.0.smithi077.stderr:+ '[' -e file4 ']' 2022-04-03T19:45:47.126 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir .snap/snap1 2022-04-03T19:45:47.139 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo 4 2022-04-03T19:45:47.140 INFO:tasks.workunit.client.0.smithi077.stderr:++ ls 2022-04-03T19:45:47.146 INFO:tasks.workunit.client.0.smithi077.stderr:+ now='file1 2022-04-03T19:45:47.147 INFO:tasks.workunit.client.0.smithi077.stderr:file2 2022-04-03T19:45:47.147 INFO:tasks.workunit.client.0.smithi077.stderr:file3 2022-04-03T19:45:47.147 INFO:tasks.workunit.client.0.smithi077.stderr:file4' 2022-04-03T19:45:47.148 INFO:tasks.workunit.client.0.smithi077.stderr:++ ls .snap/snap1 2022-04-03T19:45:47.152 INFO:tasks.workunit.client.0.smithi077.stderr:+ then='file1 2022-04-03T19:45:47.152 INFO:tasks.workunit.client.0.smithi077.stderr:file2 2022-04-03T19:45:47.152 INFO:tasks.workunit.client.0.smithi077.stderr:file3' 2022-04-03T19:45:47.153 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir .snap/snap1 2022-04-03T19:45:47.159 INFO:tasks.workunit.client.0.smithi077.stderr:+ '[' 'file1 2022-04-03T19:45:47.159 INFO:tasks.workunit.client.0.smithi077.stderr:file2 2022-04-03T19:45:47.159 INFO:tasks.workunit.client.0.smithi077.stderr:file3 2022-04-03T19:45:47.160 INFO:tasks.workunit.client.0.smithi077.stderr:file4' = 'file1 2022-04-03T19:45:47.160 INFO:tasks.workunit.client.0.smithi077.stderr:file2 2022-04-03T19:45:47.160 INFO:tasks.workunit.client.0.smithi077.stderr:file3' ']' 2022-04-03T19:45:47.160 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo 1 2022-04-03T19:45:47.166 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo 2 2022-04-03T19:45:47.294 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo 3 2022-04-03T19:45:47.357 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir .snap/snap1 2022-04-03T19:45:48.150 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo 4 2022-04-03T19:45:48.159 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir .snap/snap1 2022-04-03T19:45:48.757 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm file1 file2 file3 file4 2022-04-03T19:45:48.761 INFO:tasks.workunit.client.0.smithi077.stdout:OK 2022-04-03T19:45:48.762 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo OK 2022-04-03T19:45:48.764 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-03T19:45:48.764 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-03T19:45:48.849 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2022-04-03T19:45:48.849 DEBUG:teuthology.orchestra.run.smithi077:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2022-04-03T19:45:48.911 INFO:tasks.workunit.client.0.smithi077.stdout:Create dir 100 to 199 ... 2022-04-03T19:45:48.914 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 100 2022-04-03T19:45:48.924 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 101 2022-04-03T19:45:48.934 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 102 2022-04-03T19:45:48.944 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 103 2022-04-03T19:45:48.955 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 104 2022-04-03T19:45:49.154 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 105 2022-04-03T19:45:49.165 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 106 2022-04-03T19:45:49.176 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 107 2022-04-03T19:45:49.185 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 108 2022-04-03T19:45:49.197 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 109 2022-04-03T19:45:49.206 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 110 2022-04-03T19:45:49.218 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 111 2022-04-03T19:45:49.228 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 112 2022-04-03T19:45:49.237 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 113 2022-04-03T19:45:49.247 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 114 2022-04-03T19:45:49.257 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 115 2022-04-03T19:45:49.268 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 116 2022-04-03T19:45:49.278 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 117 2022-04-03T19:45:49.288 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 118 2022-04-03T19:45:49.298 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 119 2022-04-03T19:45:49.309 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 120 2022-04-03T19:45:49.320 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 121 2022-04-03T19:45:49.331 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 122 2022-04-03T19:45:49.342 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 123 2022-04-03T19:45:49.354 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 124 2022-04-03T19:45:49.363 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 125 2022-04-03T19:45:49.373 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 126 2022-04-03T19:45:49.383 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 127 2022-04-03T19:45:49.393 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 128 2022-04-03T19:45:49.410 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 129 2022-04-03T19:45:49.684 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 130 2022-04-03T19:45:49.954 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 131 2022-04-03T19:45:49.967 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 132 2022-04-03T19:45:49.981 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 133 2022-04-03T19:45:49.995 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 134 2022-04-03T19:45:50.006 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 135 2022-04-03T19:45:50.016 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 136 2022-04-03T19:45:50.416 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 137 2022-04-03T19:45:50.425 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 138 2022-04-03T19:45:50.871 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 139 2022-04-03T19:45:50.881 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 140 2022-04-03T19:45:50.894 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 141 2022-04-03T19:45:50.904 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 142 2022-04-03T19:45:50.915 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 143 2022-04-03T19:45:50.926 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 144 2022-04-03T19:45:50.935 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 145 2022-04-03T19:45:50.946 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 146 2022-04-03T19:45:50.956 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 147 2022-04-03T19:45:50.966 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 148 2022-04-03T19:45:50.976 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 149 2022-04-03T19:45:50.986 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 150 2022-04-03T19:45:50.995 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 151 2022-04-03T19:45:51.005 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 152 2022-04-03T19:45:51.022 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 153 2022-04-03T19:45:51.285 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 154 2022-04-03T19:45:51.294 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 155 2022-04-03T19:45:51.304 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 156 2022-04-03T19:45:51.314 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 157 2022-04-03T19:45:51.323 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 158 2022-04-03T19:45:51.334 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 159 2022-04-03T19:45:51.346 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 160 2022-04-03T19:45:51.358 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 161 2022-04-03T19:45:51.370 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 162 2022-04-03T19:45:51.381 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 163 2022-04-03T19:45:51.392 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 164 2022-04-03T19:45:51.403 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 165 2022-04-03T19:45:51.414 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 166 2022-04-03T19:45:51.425 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 167 2022-04-03T19:45:51.436 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 168 2022-04-03T19:45:51.447 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 169 2022-04-03T19:45:51.458 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 170 2022-04-03T19:45:51.469 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 171 2022-04-03T19:45:51.480 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 172 2022-04-03T19:45:51.491 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 173 2022-04-03T19:45:51.502 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 174 2022-04-03T19:45:51.513 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 175 2022-04-03T19:45:51.524 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 176 2022-04-03T19:45:51.535 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 177 2022-04-03T19:45:51.546 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 178 2022-04-03T19:45:51.557 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 179 2022-04-03T19:45:51.567 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 180 2022-04-03T19:45:51.577 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 181 2022-04-03T19:45:51.586 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 182 2022-04-03T19:45:51.596 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 183 2022-04-03T19:45:52.149 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 184 2022-04-03T19:45:52.358 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 185 2022-04-03T19:45:52.374 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 186 2022-04-03T19:45:52.390 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 187 2022-04-03T19:45:52.407 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 188 2022-04-03T19:45:52.423 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 189 2022-04-03T19:45:52.437 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 190 2022-04-03T19:45:52.453 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 191 2022-04-03T19:45:52.467 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 192 2022-04-03T19:45:52.480 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 193 2022-04-03T19:45:52.494 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 194 2022-04-03T19:45:52.508 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 195 2022-04-03T19:45:52.789 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 196 2022-04-03T19:45:53.766 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 197 2022-04-03T19:45:53.777 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 198 2022-04-03T19:45:53.791 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 199 2022-04-03T19:45:53.806 INFO:tasks.workunit.client.0.smithi077.stdout:Take first snapshot .snap/test1 2022-04-03T19:45:54.777 INFO:tasks.workunit.client.0.smithi077.stdout:Create dir 200 to 299 ... 2022-04-03T19:45:54.780 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 200 2022-04-03T19:45:54.791 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 201 2022-04-03T19:45:54.806 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 202 2022-04-03T19:45:54.821 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 203 2022-04-03T19:45:54.835 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 204 2022-04-03T19:45:54.849 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 205 2022-04-03T19:45:55.808 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 206 2022-04-03T19:45:55.822 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 207 2022-04-03T19:45:55.835 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 208 2022-04-03T19:45:55.849 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 209 2022-04-03T19:45:55.861 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 210 2022-04-03T19:45:55.870 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 211 2022-04-03T19:45:55.882 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 212 2022-04-03T19:45:55.891 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 213 2022-04-03T19:45:55.900 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 214 2022-04-03T19:45:55.910 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 215 2022-04-03T19:45:55.923 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 216 2022-04-03T19:45:55.936 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 217 2022-04-03T19:45:55.949 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 218 2022-04-03T19:45:55.958 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 219 2022-04-03T19:45:55.967 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 220 2022-04-03T19:45:55.976 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 221 2022-04-03T19:45:55.986 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 222 2022-04-03T19:45:55.996 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 223 2022-04-03T19:45:56.008 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 224 2022-04-03T19:45:56.018 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 225 2022-04-03T19:45:56.027 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 226 2022-04-03T19:45:56.037 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 227 2022-04-03T19:45:56.700 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 228 2022-04-03T19:45:56.711 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 229 2022-04-03T19:45:56.720 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 230 2022-04-03T19:45:56.730 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 231 2022-04-03T19:45:57.006 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 232 2022-04-03T19:45:57.016 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 233 2022-04-03T19:45:57.026 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 234 2022-04-03T19:45:57.036 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 235 2022-04-03T19:45:57.045 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 236 2022-04-03T19:45:57.055 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 237 2022-04-03T19:45:57.071 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 238 2022-04-03T19:45:57.796 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 239 2022-04-03T19:45:57.807 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 240 2022-04-03T19:45:57.818 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 241 2022-04-03T19:45:57.828 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 242 2022-04-03T19:45:57.839 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 243 2022-04-03T19:45:57.850 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 244 2022-04-03T19:45:57.860 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 245 2022-04-03T19:45:57.871 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 246 2022-04-03T19:45:57.886 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 247 2022-04-03T19:45:57.899 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 248 2022-04-03T19:45:57.913 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 249 2022-04-03T19:45:57.926 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 250 2022-04-03T19:45:57.969 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 251 2022-04-03T19:46:03.010 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 252 2022-04-03T19:46:03.021 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 253 2022-04-03T19:46:03.033 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 254 2022-04-03T19:46:03.042 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 255 2022-04-03T19:46:03.053 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 256 2022-04-03T19:46:03.560 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 257 2022-04-03T19:46:03.571 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 258 2022-04-03T19:46:03.584 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 259 2022-04-03T19:46:03.600 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 260 2022-04-03T19:46:03.615 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 261 2022-04-03T19:46:03.628 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 262 2022-04-03T19:46:03.642 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 263 2022-04-03T19:46:03.657 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 264 2022-04-03T19:46:03.671 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 265 2022-04-03T19:46:03.684 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 266 2022-04-03T19:46:03.697 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 267 2022-04-03T19:46:03.717 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 268 2022-04-03T19:46:03.983 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 269 2022-04-03T19:46:03.995 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 270 2022-04-03T19:46:04.763 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 271 2022-04-03T19:46:04.776 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 272 2022-04-03T19:46:04.790 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 273 2022-04-03T19:46:04.802 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 274 2022-04-03T19:46:04.812 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 275 2022-04-03T19:46:04.823 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 276 2022-04-03T19:46:04.833 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 277 2022-04-03T19:46:04.844 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 278 2022-04-03T19:46:04.854 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 279 2022-04-03T19:46:04.865 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 280 2022-04-03T19:46:05.197 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 281 2022-04-03T19:46:05.207 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 282 2022-04-03T19:46:05.217 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 283 2022-04-03T19:46:05.228 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 284 2022-04-03T19:46:05.238 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 285 2022-04-03T19:46:05.249 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 286 2022-04-03T19:46:05.260 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 287 2022-04-03T19:46:05.270 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 288 2022-04-03T19:46:05.694 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 289 2022-04-03T19:46:05.704 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 290 2022-04-03T19:46:05.715 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 291 2022-04-03T19:46:05.726 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 292 2022-04-03T19:46:05.737 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 293 2022-04-03T19:46:05.748 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 294 2022-04-03T19:46:06.384 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 295 2022-04-03T19:46:06.395 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 296 2022-04-03T19:46:07.214 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 297 2022-04-03T19:46:07.225 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 298 2022-04-03T19:46:07.236 INFO:tasks.workunit.client.0.smithi077.stdout: create dir 299 2022-04-03T19:46:07.248 INFO:tasks.workunit.client.0.smithi077.stdout:Create a snapshot in every first level dir ... 2022-04-03T19:46:07.402 INFO:tasks.workunit.client.0.smithi077.stdout: create 100/.snap/snap-subdir-test 2022-04-03T19:46:07.423 INFO:tasks.workunit.client.0.smithi077.stdout: create 100/30 file after the snapshot 2022-04-03T19:46:07.426 INFO:tasks.workunit.client.0.smithi077.stdout: create 100/31 file after the snapshot 2022-04-03T19:46:07.426 INFO:tasks.workunit.client.0.smithi077.stdout: create 100/32 file after the snapshot 2022-04-03T19:46:07.427 INFO:tasks.workunit.client.0.smithi077.stdout: create 100/33 file after the snapshot 2022-04-03T19:46:07.427 INFO:tasks.workunit.client.0.smithi077.stdout: create 100/34 file after the snapshot 2022-04-03T19:46:07.427 INFO:tasks.workunit.client.0.smithi077.stdout: create 100/35 file after the snapshot 2022-04-03T19:46:07.427 INFO:tasks.workunit.client.0.smithi077.stdout: create 100/36 file after the snapshot 2022-04-03T19:46:07.428 INFO:tasks.workunit.client.0.smithi077.stdout: create 100/37 file after the snapshot 2022-04-03T19:46:07.428 INFO:tasks.workunit.client.0.smithi077.stdout: create 100/38 file after the snapshot 2022-04-03T19:46:07.428 INFO:tasks.workunit.client.0.smithi077.stdout: create 100/39 file after the snapshot 2022-04-03T19:46:07.428 INFO:tasks.workunit.client.0.smithi077.stdout: create 101/.snap/snap-subdir-test 2022-04-03T19:46:07.456 INFO:tasks.workunit.client.0.smithi077.stdout: create 101/30 file after the snapshot 2022-04-03T19:46:07.459 INFO:tasks.workunit.client.0.smithi077.stdout: create 101/31 file after the snapshot 2022-04-03T19:46:07.459 INFO:tasks.workunit.client.0.smithi077.stdout: create 101/32 file after the snapshot 2022-04-03T19:46:07.459 INFO:tasks.workunit.client.0.smithi077.stdout: create 101/33 file after the snapshot 2022-04-03T19:46:07.459 INFO:tasks.workunit.client.0.smithi077.stdout: create 101/34 file after the snapshot 2022-04-03T19:46:07.460 INFO:tasks.workunit.client.0.smithi077.stdout: create 101/35 file after the snapshot 2022-04-03T19:46:07.460 INFO:tasks.workunit.client.0.smithi077.stdout: create 101/36 file after the snapshot 2022-04-03T19:46:07.460 INFO:tasks.workunit.client.0.smithi077.stdout: create 101/37 file after the snapshot 2022-04-03T19:46:07.460 INFO:tasks.workunit.client.0.smithi077.stdout: create 101/38 file after the snapshot 2022-04-03T19:46:07.460 INFO:tasks.workunit.client.0.smithi077.stdout: create 101/39 file after the snapshot 2022-04-03T19:46:07.461 INFO:tasks.workunit.client.0.smithi077.stdout: create 102/.snap/snap-subdir-test 2022-04-03T19:46:08.139 INFO:tasks.workunit.client.0.smithi077.stdout: create 102/30 file after the snapshot 2022-04-03T19:46:08.141 INFO:tasks.workunit.client.0.smithi077.stdout: create 102/31 file after the snapshot 2022-04-03T19:46:08.141 INFO:tasks.workunit.client.0.smithi077.stdout: create 102/32 file after the snapshot 2022-04-03T19:46:08.142 INFO:tasks.workunit.client.0.smithi077.stdout: create 102/33 file after the snapshot 2022-04-03T19:46:08.143 INFO:tasks.workunit.client.0.smithi077.stdout: create 102/34 file after the snapshot 2022-04-03T19:46:08.143 INFO:tasks.workunit.client.0.smithi077.stdout: create 102/35 file after the snapshot 2022-04-03T19:46:08.143 INFO:tasks.workunit.client.0.smithi077.stdout: create 102/36 file after the snapshot 2022-04-03T19:46:08.143 INFO:tasks.workunit.client.0.smithi077.stdout: create 102/37 file after the snapshot 2022-04-03T19:46:08.144 INFO:tasks.workunit.client.0.smithi077.stdout: create 102/38 file after the snapshot 2022-04-03T19:46:08.144 INFO:tasks.workunit.client.0.smithi077.stdout: create 102/39 file after the snapshot 2022-04-03T19:46:08.144 INFO:tasks.workunit.client.0.smithi077.stdout: create 103/.snap/snap-subdir-test 2022-04-03T19:46:08.169 INFO:tasks.workunit.client.0.smithi077.stdout: create 103/30 file after the snapshot 2022-04-03T19:46:08.170 INFO:tasks.workunit.client.0.smithi077.stdout: create 103/31 file after the snapshot 2022-04-03T19:46:08.171 INFO:tasks.workunit.client.0.smithi077.stdout: create 103/32 file after the snapshot 2022-04-03T19:46:08.172 INFO:tasks.workunit.client.0.smithi077.stdout: create 103/33 file after the snapshot 2022-04-03T19:46:08.172 INFO:tasks.workunit.client.0.smithi077.stdout: create 103/34 file after the snapshot 2022-04-03T19:46:08.172 INFO:tasks.workunit.client.0.smithi077.stdout: create 103/35 file after the snapshot 2022-04-03T19:46:08.173 INFO:tasks.workunit.client.0.smithi077.stdout: create 103/36 file after the snapshot 2022-04-03T19:46:08.173 INFO:tasks.workunit.client.0.smithi077.stdout: create 103/37 file after the snapshot 2022-04-03T19:46:08.173 INFO:tasks.workunit.client.0.smithi077.stdout: create 103/38 file after the snapshot 2022-04-03T19:46:08.173 INFO:tasks.workunit.client.0.smithi077.stdout: create 103/39 file after the snapshot 2022-04-03T19:46:08.173 INFO:tasks.workunit.client.0.smithi077.stdout: create 104/.snap/snap-subdir-test 2022-04-03T19:46:08.197 INFO:tasks.workunit.client.0.smithi077.stdout: create 104/30 file after the snapshot 2022-04-03T19:46:08.199 INFO:tasks.workunit.client.0.smithi077.stdout: create 104/31 file after the snapshot 2022-04-03T19:46:08.199 INFO:tasks.workunit.client.0.smithi077.stdout: create 104/32 file after the snapshot 2022-04-03T19:46:08.200 INFO:tasks.workunit.client.0.smithi077.stdout: create 104/33 file after the snapshot 2022-04-03T19:46:08.200 INFO:tasks.workunit.client.0.smithi077.stdout: create 104/34 file after the snapshot 2022-04-03T19:46:08.200 INFO:tasks.workunit.client.0.smithi077.stdout: create 104/35 file after the snapshot 2022-04-03T19:46:08.200 INFO:tasks.workunit.client.0.smithi077.stdout: create 104/36 file after the snapshot 2022-04-03T19:46:08.201 INFO:tasks.workunit.client.0.smithi077.stdout: create 104/37 file after the snapshot 2022-04-03T19:46:08.201 INFO:tasks.workunit.client.0.smithi077.stdout: create 104/38 file after the snapshot 2022-04-03T19:46:08.201 INFO:tasks.workunit.client.0.smithi077.stdout: create 104/39 file after the snapshot 2022-04-03T19:46:08.201 INFO:tasks.workunit.client.0.smithi077.stdout: create 105/.snap/snap-subdir-test 2022-04-03T19:46:08.230 INFO:tasks.workunit.client.0.smithi077.stdout: create 105/30 file after the snapshot 2022-04-03T19:46:08.232 INFO:tasks.workunit.client.0.smithi077.stdout: create 105/31 file after the snapshot 2022-04-03T19:46:08.232 INFO:tasks.workunit.client.0.smithi077.stdout: create 105/32 file after the snapshot 2022-04-03T19:46:08.232 INFO:tasks.workunit.client.0.smithi077.stdout: create 105/33 file after the snapshot 2022-04-03T19:46:08.232 INFO:tasks.workunit.client.0.smithi077.stdout: create 105/34 file after the snapshot 2022-04-03T19:46:08.233 INFO:tasks.workunit.client.0.smithi077.stdout: create 105/35 file after the snapshot 2022-04-03T19:46:08.234 INFO:tasks.workunit.client.0.smithi077.stdout: create 105/36 file after the snapshot 2022-04-03T19:46:08.234 INFO:tasks.workunit.client.0.smithi077.stdout: create 105/37 file after the snapshot 2022-04-03T19:46:08.234 INFO:tasks.workunit.client.0.smithi077.stdout: create 105/38 file after the snapshot 2022-04-03T19:46:08.234 INFO:tasks.workunit.client.0.smithi077.stdout: create 105/39 file after the snapshot 2022-04-03T19:46:08.235 INFO:tasks.workunit.client.0.smithi077.stdout: create 106/.snap/snap-subdir-test 2022-04-03T19:46:08.259 INFO:tasks.workunit.client.0.smithi077.stdout: create 106/30 file after the snapshot 2022-04-03T19:46:08.261 INFO:tasks.workunit.client.0.smithi077.stdout: create 106/31 file after the snapshot 2022-04-03T19:46:08.261 INFO:tasks.workunit.client.0.smithi077.stdout: create 106/32 file after the snapshot 2022-04-03T19:46:08.262 INFO:tasks.workunit.client.0.smithi077.stdout: create 106/33 file after the snapshot 2022-04-03T19:46:08.262 INFO:tasks.workunit.client.0.smithi077.stdout: create 106/34 file after the snapshot 2022-04-03T19:46:08.262 INFO:tasks.workunit.client.0.smithi077.stdout: create 106/35 file after the snapshot 2022-04-03T19:46:08.262 INFO:tasks.workunit.client.0.smithi077.stdout: create 106/36 file after the snapshot 2022-04-03T19:46:08.263 INFO:tasks.workunit.client.0.smithi077.stdout: create 106/37 file after the snapshot 2022-04-03T19:46:08.263 INFO:tasks.workunit.client.0.smithi077.stdout: create 106/38 file after the snapshot 2022-04-03T19:46:08.263 INFO:tasks.workunit.client.0.smithi077.stdout: create 106/39 file after the snapshot 2022-04-03T19:46:08.263 INFO:tasks.workunit.client.0.smithi077.stdout: create 107/.snap/snap-subdir-test 2022-04-03T19:46:08.287 INFO:tasks.workunit.client.0.smithi077.stdout: create 107/30 file after the snapshot 2022-04-03T19:46:08.289 INFO:tasks.workunit.client.0.smithi077.stdout: create 107/31 file after the snapshot 2022-04-03T19:46:08.290 INFO:tasks.workunit.client.0.smithi077.stdout: create 107/32 file after the snapshot 2022-04-03T19:46:08.290 INFO:tasks.workunit.client.0.smithi077.stdout: create 107/33 file after the snapshot 2022-04-03T19:46:08.290 INFO:tasks.workunit.client.0.smithi077.stdout: create 107/34 file after the snapshot 2022-04-03T19:46:08.290 INFO:tasks.workunit.client.0.smithi077.stdout: create 107/35 file after the snapshot 2022-04-03T19:46:08.291 INFO:tasks.workunit.client.0.smithi077.stdout: create 107/36 file after the snapshot 2022-04-03T19:46:08.291 INFO:tasks.workunit.client.0.smithi077.stdout: create 107/37 file after the snapshot 2022-04-03T19:46:08.291 INFO:tasks.workunit.client.0.smithi077.stdout: create 107/38 file after the snapshot 2022-04-03T19:46:08.291 INFO:tasks.workunit.client.0.smithi077.stdout: create 107/39 file after the snapshot 2022-04-03T19:46:08.292 INFO:tasks.workunit.client.0.smithi077.stdout: create 108/.snap/snap-subdir-test 2022-04-03T19:46:08.318 INFO:tasks.workunit.client.0.smithi077.stdout: create 108/30 file after the snapshot 2022-04-03T19:46:08.320 INFO:tasks.workunit.client.0.smithi077.stdout: create 108/31 file after the snapshot 2022-04-03T19:46:08.320 INFO:tasks.workunit.client.0.smithi077.stdout: create 108/32 file after the snapshot 2022-04-03T19:46:08.320 INFO:tasks.workunit.client.0.smithi077.stdout: create 108/33 file after the snapshot 2022-04-03T19:46:08.321 INFO:tasks.workunit.client.0.smithi077.stdout: create 108/34 file after the snapshot 2022-04-03T19:46:08.321 INFO:tasks.workunit.client.0.smithi077.stdout: create 108/35 file after the snapshot 2022-04-03T19:46:08.321 INFO:tasks.workunit.client.0.smithi077.stdout: create 108/36 file after the snapshot 2022-04-03T19:46:08.321 INFO:tasks.workunit.client.0.smithi077.stdout: create 108/37 file after the snapshot 2022-04-03T19:46:08.321 INFO:tasks.workunit.client.0.smithi077.stdout: create 108/38 file after the snapshot 2022-04-03T19:46:08.322 INFO:tasks.workunit.client.0.smithi077.stdout: create 108/39 file after the snapshot 2022-04-03T19:46:08.322 INFO:tasks.workunit.client.0.smithi077.stdout: create 109/.snap/snap-subdir-test 2022-04-03T19:46:08.346 INFO:tasks.workunit.client.0.smithi077.stdout: create 109/30 file after the snapshot 2022-04-03T19:46:08.348 INFO:tasks.workunit.client.0.smithi077.stdout: create 109/31 file after the snapshot 2022-04-03T19:46:08.348 INFO:tasks.workunit.client.0.smithi077.stdout: create 109/32 file after the snapshot 2022-04-03T19:46:08.348 INFO:tasks.workunit.client.0.smithi077.stdout: create 109/33 file after the snapshot 2022-04-03T19:46:08.348 INFO:tasks.workunit.client.0.smithi077.stdout: create 109/34 file after the snapshot 2022-04-03T19:46:08.349 INFO:tasks.workunit.client.0.smithi077.stdout: create 109/35 file after the snapshot 2022-04-03T19:46:08.349 INFO:tasks.workunit.client.0.smithi077.stdout: create 109/36 file after the snapshot 2022-04-03T19:46:08.349 INFO:tasks.workunit.client.0.smithi077.stdout: create 109/37 file after the snapshot 2022-04-03T19:46:08.349 INFO:tasks.workunit.client.0.smithi077.stdout: create 109/38 file after the snapshot 2022-04-03T19:46:08.349 INFO:tasks.workunit.client.0.smithi077.stdout: create 109/39 file after the snapshot 2022-04-03T19:46:08.350 INFO:tasks.workunit.client.0.smithi077.stdout: create 110/.snap/snap-subdir-test 2022-04-03T19:46:08.374 INFO:tasks.workunit.client.0.smithi077.stdout: create 110/30 file after the snapshot 2022-04-03T19:46:08.376 INFO:tasks.workunit.client.0.smithi077.stdout: create 110/31 file after the snapshot 2022-04-03T19:46:08.376 INFO:tasks.workunit.client.0.smithi077.stdout: create 110/32 file after the snapshot 2022-04-03T19:46:08.376 INFO:tasks.workunit.client.0.smithi077.stdout: create 110/33 file after the snapshot 2022-04-03T19:46:08.376 INFO:tasks.workunit.client.0.smithi077.stdout: create 110/34 file after the snapshot 2022-04-03T19:46:08.377 INFO:tasks.workunit.client.0.smithi077.stdout: create 110/35 file after the snapshot 2022-04-03T19:46:08.377 INFO:tasks.workunit.client.0.smithi077.stdout: create 110/36 file after the snapshot 2022-04-03T19:46:08.377 INFO:tasks.workunit.client.0.smithi077.stdout: create 110/37 file after the snapshot 2022-04-03T19:46:08.377 INFO:tasks.workunit.client.0.smithi077.stdout: create 110/38 file after the snapshot 2022-04-03T19:46:08.377 INFO:tasks.workunit.client.0.smithi077.stdout: create 110/39 file after the snapshot 2022-04-03T19:46:08.378 INFO:tasks.workunit.client.0.smithi077.stdout: create 111/.snap/snap-subdir-test 2022-04-03T19:46:08.665 INFO:tasks.workunit.client.0.smithi077.stdout: create 111/30 file after the snapshot 2022-04-03T19:46:08.667 INFO:tasks.workunit.client.0.smithi077.stdout: create 111/31 file after the snapshot 2022-04-03T19:46:08.667 INFO:tasks.workunit.client.0.smithi077.stdout: create 111/32 file after the snapshot 2022-04-03T19:46:08.667 INFO:tasks.workunit.client.0.smithi077.stdout: create 111/33 file after the snapshot 2022-04-03T19:46:08.667 INFO:tasks.workunit.client.0.smithi077.stdout: create 111/34 file after the snapshot 2022-04-03T19:46:08.668 INFO:tasks.workunit.client.0.smithi077.stdout: create 111/35 file after the snapshot 2022-04-03T19:46:08.668 INFO:tasks.workunit.client.0.smithi077.stdout: create 111/36 file after the snapshot 2022-04-03T19:46:08.668 INFO:tasks.workunit.client.0.smithi077.stdout: create 111/37 file after the snapshot 2022-04-03T19:46:08.668 INFO:tasks.workunit.client.0.smithi077.stdout: create 111/38 file after the snapshot 2022-04-03T19:46:08.668 INFO:tasks.workunit.client.0.smithi077.stdout: create 111/39 file after the snapshot 2022-04-03T19:46:08.669 INFO:tasks.workunit.client.0.smithi077.stdout: create 112/.snap/snap-subdir-test 2022-04-03T19:46:08.693 INFO:tasks.workunit.client.0.smithi077.stdout: create 112/30 file after the snapshot 2022-04-03T19:46:08.695 INFO:tasks.workunit.client.0.smithi077.stdout: create 112/31 file after the snapshot 2022-04-03T19:46:08.695 INFO:tasks.workunit.client.0.smithi077.stdout: create 112/32 file after the snapshot 2022-04-03T19:46:08.695 INFO:tasks.workunit.client.0.smithi077.stdout: create 112/33 file after the snapshot 2022-04-03T19:46:08.696 INFO:tasks.workunit.client.0.smithi077.stdout: create 112/34 file after the snapshot 2022-04-03T19:46:08.696 INFO:tasks.workunit.client.0.smithi077.stdout: create 112/35 file after the snapshot 2022-04-03T19:46:08.696 INFO:tasks.workunit.client.0.smithi077.stdout: create 112/36 file after the snapshot 2022-04-03T19:46:08.696 INFO:tasks.workunit.client.0.smithi077.stdout: create 112/37 file after the snapshot 2022-04-03T19:46:08.696 INFO:tasks.workunit.client.0.smithi077.stdout: create 112/38 file after the snapshot 2022-04-03T19:46:08.697 INFO:tasks.workunit.client.0.smithi077.stdout: create 112/39 file after the snapshot 2022-04-03T19:46:08.697 INFO:tasks.workunit.client.0.smithi077.stdout: create 113/.snap/snap-subdir-test 2022-04-03T19:46:08.815 INFO:tasks.workunit.client.0.smithi077.stdout: create 113/30 file after the snapshot 2022-04-03T19:46:08.817 INFO:tasks.workunit.client.0.smithi077.stdout: create 113/31 file after the snapshot 2022-04-03T19:46:08.817 INFO:tasks.workunit.client.0.smithi077.stdout: create 113/32 file after the snapshot 2022-04-03T19:46:08.818 INFO:tasks.workunit.client.0.smithi077.stdout: create 113/33 file after the snapshot 2022-04-03T19:46:08.818 INFO:tasks.workunit.client.0.smithi077.stdout: create 113/34 file after the snapshot 2022-04-03T19:46:08.818 INFO:tasks.workunit.client.0.smithi077.stdout: create 113/35 file after the snapshot 2022-04-03T19:46:08.818 INFO:tasks.workunit.client.0.smithi077.stdout: create 113/36 file after the snapshot 2022-04-03T19:46:08.819 INFO:tasks.workunit.client.0.smithi077.stdout: create 113/37 file after the snapshot 2022-04-03T19:46:08.819 INFO:tasks.workunit.client.0.smithi077.stdout: create 113/38 file after the snapshot 2022-04-03T19:46:08.819 INFO:tasks.workunit.client.0.smithi077.stdout: create 113/39 file after the snapshot 2022-04-03T19:46:08.819 INFO:tasks.workunit.client.0.smithi077.stdout: create 114/.snap/snap-subdir-test 2022-04-03T19:46:08.843 INFO:tasks.workunit.client.0.smithi077.stdout: create 114/30 file after the snapshot 2022-04-03T19:46:08.845 INFO:tasks.workunit.client.0.smithi077.stdout: create 114/31 file after the snapshot 2022-04-03T19:46:08.845 INFO:tasks.workunit.client.0.smithi077.stdout: create 114/32 file after the snapshot 2022-04-03T19:46:08.846 INFO:tasks.workunit.client.0.smithi077.stdout: create 114/33 file after the snapshot 2022-04-03T19:46:08.846 INFO:tasks.workunit.client.0.smithi077.stdout: create 114/34 file after the snapshot 2022-04-03T19:46:08.846 INFO:tasks.workunit.client.0.smithi077.stdout: create 114/35 file after the snapshot 2022-04-03T19:46:08.846 INFO:tasks.workunit.client.0.smithi077.stdout: create 114/36 file after the snapshot 2022-04-03T19:46:08.847 INFO:tasks.workunit.client.0.smithi077.stdout: create 114/37 file after the snapshot 2022-04-03T19:46:08.847 INFO:tasks.workunit.client.0.smithi077.stdout: create 114/38 file after the snapshot 2022-04-03T19:46:08.847 INFO:tasks.workunit.client.0.smithi077.stdout: create 114/39 file after the snapshot 2022-04-03T19:46:08.847 INFO:tasks.workunit.client.0.smithi077.stdout: create 115/.snap/snap-subdir-test 2022-04-03T19:46:08.871 INFO:tasks.workunit.client.0.smithi077.stdout: create 115/30 file after the snapshot 2022-04-03T19:46:08.873 INFO:tasks.workunit.client.0.smithi077.stdout: create 115/31 file after the snapshot 2022-04-03T19:46:08.873 INFO:tasks.workunit.client.0.smithi077.stdout: create 115/32 file after the snapshot 2022-04-03T19:46:08.873 INFO:tasks.workunit.client.0.smithi077.stdout: create 115/33 file after the snapshot 2022-04-03T19:46:08.874 INFO:tasks.workunit.client.0.smithi077.stdout: create 115/34 file after the snapshot 2022-04-03T19:46:08.874 INFO:tasks.workunit.client.0.smithi077.stdout: create 115/35 file after the snapshot 2022-04-03T19:46:08.874 INFO:tasks.workunit.client.0.smithi077.stdout: create 115/36 file after the snapshot 2022-04-03T19:46:08.874 INFO:tasks.workunit.client.0.smithi077.stdout: create 115/37 file after the snapshot 2022-04-03T19:46:08.874 INFO:tasks.workunit.client.0.smithi077.stdout: create 115/38 file after the snapshot 2022-04-03T19:46:08.875 INFO:tasks.workunit.client.0.smithi077.stdout: create 115/39 file after the snapshot 2022-04-03T19:46:08.875 INFO:tasks.workunit.client.0.smithi077.stdout: create 116/.snap/snap-subdir-test 2022-04-03T19:46:08.898 INFO:tasks.workunit.client.0.smithi077.stdout: create 116/30 file after the snapshot 2022-04-03T19:46:08.900 INFO:tasks.workunit.client.0.smithi077.stdout: create 116/31 file after the snapshot 2022-04-03T19:46:08.901 INFO:tasks.workunit.client.0.smithi077.stdout: create 116/32 file after the snapshot 2022-04-03T19:46:08.901 INFO:tasks.workunit.client.0.smithi077.stdout: create 116/33 file after the snapshot 2022-04-03T19:46:08.901 INFO:tasks.workunit.client.0.smithi077.stdout: create 116/34 file after the snapshot 2022-04-03T19:46:08.901 INFO:tasks.workunit.client.0.smithi077.stdout: create 116/35 file after the snapshot 2022-04-03T19:46:08.901 INFO:tasks.workunit.client.0.smithi077.stdout: create 116/36 file after the snapshot 2022-04-03T19:46:08.902 INFO:tasks.workunit.client.0.smithi077.stdout: create 116/37 file after the snapshot 2022-04-03T19:46:08.902 INFO:tasks.workunit.client.0.smithi077.stdout: create 116/38 file after the snapshot 2022-04-03T19:46:08.902 INFO:tasks.workunit.client.0.smithi077.stdout: create 116/39 file after the snapshot 2022-04-03T19:46:08.902 INFO:tasks.workunit.client.0.smithi077.stdout: create 117/.snap/snap-subdir-test 2022-04-03T19:46:08.926 INFO:tasks.workunit.client.0.smithi077.stdout: create 117/30 file after the snapshot 2022-04-03T19:46:08.928 INFO:tasks.workunit.client.0.smithi077.stdout: create 117/31 file after the snapshot 2022-04-03T19:46:08.928 INFO:tasks.workunit.client.0.smithi077.stdout: create 117/32 file after the snapshot 2022-04-03T19:46:08.928 INFO:tasks.workunit.client.0.smithi077.stdout: create 117/33 file after the snapshot 2022-04-03T19:46:08.929 INFO:tasks.workunit.client.0.smithi077.stdout: create 117/34 file after the snapshot 2022-04-03T19:46:08.929 INFO:tasks.workunit.client.0.smithi077.stdout: create 117/35 file after the snapshot 2022-04-03T19:46:08.929 INFO:tasks.workunit.client.0.smithi077.stdout: create 117/36 file after the snapshot 2022-04-03T19:46:08.929 INFO:tasks.workunit.client.0.smithi077.stdout: create 117/37 file after the snapshot 2022-04-03T19:46:08.929 INFO:tasks.workunit.client.0.smithi077.stdout: create 117/38 file after the snapshot 2022-04-03T19:46:08.930 INFO:tasks.workunit.client.0.smithi077.stdout: create 117/39 file after the snapshot 2022-04-03T19:46:08.930 INFO:tasks.workunit.client.0.smithi077.stdout: create 118/.snap/snap-subdir-test 2022-04-03T19:46:08.953 INFO:tasks.workunit.client.0.smithi077.stdout: create 118/30 file after the snapshot 2022-04-03T19:46:08.956 INFO:tasks.workunit.client.0.smithi077.stdout: create 118/31 file after the snapshot 2022-04-03T19:46:08.956 INFO:tasks.workunit.client.0.smithi077.stdout: create 118/32 file after the snapshot 2022-04-03T19:46:08.957 INFO:tasks.workunit.client.0.smithi077.stdout: create 118/33 file after the snapshot 2022-04-03T19:46:08.957 INFO:tasks.workunit.client.0.smithi077.stdout: create 118/34 file after the snapshot 2022-04-03T19:46:08.957 INFO:tasks.workunit.client.0.smithi077.stdout: create 118/35 file after the snapshot 2022-04-03T19:46:08.957 INFO:tasks.workunit.client.0.smithi077.stdout: create 118/36 file after the snapshot 2022-04-03T19:46:08.957 INFO:tasks.workunit.client.0.smithi077.stdout: create 118/37 file after the snapshot 2022-04-03T19:46:08.958 INFO:tasks.workunit.client.0.smithi077.stdout: create 118/38 file after the snapshot 2022-04-03T19:46:08.958 INFO:tasks.workunit.client.0.smithi077.stdout: create 118/39 file after the snapshot 2022-04-03T19:46:08.958 INFO:tasks.workunit.client.0.smithi077.stdout: create 119/.snap/snap-subdir-test 2022-04-03T19:46:08.982 INFO:tasks.workunit.client.0.smithi077.stdout: create 119/30 file after the snapshot 2022-04-03T19:46:09.104 INFO:tasks.workunit.client.0.smithi077.stdout: create 119/31 file after the snapshot 2022-04-03T19:46:09.104 INFO:tasks.workunit.client.0.smithi077.stdout: create 119/32 file after the snapshot 2022-04-03T19:46:09.105 INFO:tasks.workunit.client.0.smithi077.stdout: create 119/33 file after the snapshot 2022-04-03T19:46:09.105 INFO:tasks.workunit.client.0.smithi077.stdout: create 119/34 file after the snapshot 2022-04-03T19:46:09.105 INFO:tasks.workunit.client.0.smithi077.stdout: create 119/35 file after the snapshot 2022-04-03T19:46:09.105 INFO:tasks.workunit.client.0.smithi077.stdout: create 119/36 file after the snapshot 2022-04-03T19:46:09.105 INFO:tasks.workunit.client.0.smithi077.stdout: create 119/37 file after the snapshot 2022-04-03T19:46:09.106 INFO:tasks.workunit.client.0.smithi077.stdout: create 119/38 file after the snapshot 2022-04-03T19:46:09.106 INFO:tasks.workunit.client.0.smithi077.stdout: create 119/39 file after the snapshot 2022-04-03T19:46:09.106 INFO:tasks.workunit.client.0.smithi077.stdout: create 120/.snap/snap-subdir-test 2022-04-03T19:46:09.106 INFO:tasks.workunit.client.0.smithi077.stdout: create 120/30 file after the snapshot 2022-04-03T19:46:09.107 INFO:tasks.workunit.client.0.smithi077.stdout: create 120/31 file after the snapshot 2022-04-03T19:46:09.107 INFO:tasks.workunit.client.0.smithi077.stdout: create 120/32 file after the snapshot 2022-04-03T19:46:09.107 INFO:tasks.workunit.client.0.smithi077.stdout: create 120/33 file after the snapshot 2022-04-03T19:46:09.107 INFO:tasks.workunit.client.0.smithi077.stdout: create 120/34 file after the snapshot 2022-04-03T19:46:09.108 INFO:tasks.workunit.client.0.smithi077.stdout: create 120/35 file after the snapshot 2022-04-03T19:46:09.108 INFO:tasks.workunit.client.0.smithi077.stdout: create 120/36 file after the snapshot 2022-04-03T19:46:09.108 INFO:tasks.workunit.client.0.smithi077.stdout: create 120/37 file after the snapshot 2022-04-03T19:46:09.108 INFO:tasks.workunit.client.0.smithi077.stdout: create 120/38 file after the snapshot 2022-04-03T19:46:09.108 INFO:tasks.workunit.client.0.smithi077.stdout: create 120/39 file after the snapshot 2022-04-03T19:46:09.109 INFO:tasks.workunit.client.0.smithi077.stdout: create 121/.snap/snap-subdir-test 2022-04-03T19:46:09.187 INFO:tasks.workunit.client.0.smithi077.stdout: create 121/30 file after the snapshot 2022-04-03T19:46:09.189 INFO:tasks.workunit.client.0.smithi077.stdout: create 121/31 file after the snapshot 2022-04-03T19:46:09.189 INFO:tasks.workunit.client.0.smithi077.stdout: create 121/32 file after the snapshot 2022-04-03T19:46:09.189 INFO:tasks.workunit.client.0.smithi077.stdout: create 121/33 file after the snapshot 2022-04-03T19:46:09.190 INFO:tasks.workunit.client.0.smithi077.stdout: create 121/34 file after the snapshot 2022-04-03T19:46:09.191 INFO:tasks.workunit.client.0.smithi077.stdout: create 121/35 file after the snapshot 2022-04-03T19:46:09.191 INFO:tasks.workunit.client.0.smithi077.stdout: create 121/36 file after the snapshot 2022-04-03T19:46:09.191 INFO:tasks.workunit.client.0.smithi077.stdout: create 121/37 file after the snapshot 2022-04-03T19:46:09.191 INFO:tasks.workunit.client.0.smithi077.stdout: create 121/38 file after the snapshot 2022-04-03T19:46:09.192 INFO:tasks.workunit.client.0.smithi077.stdout: create 121/39 file after the snapshot 2022-04-03T19:46:09.192 INFO:tasks.workunit.client.0.smithi077.stdout: create 122/.snap/snap-subdir-test 2022-04-03T19:46:09.223 INFO:tasks.workunit.client.0.smithi077.stdout: create 122/30 file after the snapshot 2022-04-03T19:46:09.225 INFO:tasks.workunit.client.0.smithi077.stdout: create 122/31 file after the snapshot 2022-04-03T19:46:09.226 INFO:tasks.workunit.client.0.smithi077.stdout: create 122/32 file after the snapshot 2022-04-03T19:46:09.226 INFO:tasks.workunit.client.0.smithi077.stdout: create 122/33 file after the snapshot 2022-04-03T19:46:09.226 INFO:tasks.workunit.client.0.smithi077.stdout: create 122/34 file after the snapshot 2022-04-03T19:46:09.226 INFO:tasks.workunit.client.0.smithi077.stdout: create 122/35 file after the snapshot 2022-04-03T19:46:09.227 INFO:tasks.workunit.client.0.smithi077.stdout: create 122/36 file after the snapshot 2022-04-03T19:46:09.227 INFO:tasks.workunit.client.0.smithi077.stdout: create 122/37 file after the snapshot 2022-04-03T19:46:09.227 INFO:tasks.workunit.client.0.smithi077.stdout: create 122/38 file after the snapshot 2022-04-03T19:46:09.227 INFO:tasks.workunit.client.0.smithi077.stdout: create 122/39 file after the snapshot 2022-04-03T19:46:09.227 INFO:tasks.workunit.client.0.smithi077.stdout: create 123/.snap/snap-subdir-test 2022-04-03T19:46:09.262 INFO:tasks.workunit.client.0.smithi077.stdout: create 123/30 file after the snapshot 2022-04-03T19:46:09.264 INFO:tasks.workunit.client.0.smithi077.stdout: create 123/31 file after the snapshot 2022-04-03T19:46:09.264 INFO:tasks.workunit.client.0.smithi077.stdout: create 123/32 file after the snapshot 2022-04-03T19:46:09.265 INFO:tasks.workunit.client.0.smithi077.stdout: create 123/33 file after the snapshot 2022-04-03T19:46:09.265 INFO:tasks.workunit.client.0.smithi077.stdout: create 123/34 file after the snapshot 2022-04-03T19:46:09.265 INFO:tasks.workunit.client.0.smithi077.stdout: create 123/35 file after the snapshot 2022-04-03T19:46:09.265 INFO:tasks.workunit.client.0.smithi077.stdout: create 123/36 file after the snapshot 2022-04-03T19:46:09.265 INFO:tasks.workunit.client.0.smithi077.stdout: create 123/37 file after the snapshot 2022-04-03T19:46:09.266 INFO:tasks.workunit.client.0.smithi077.stdout: create 123/38 file after the snapshot 2022-04-03T19:46:09.266 INFO:tasks.workunit.client.0.smithi077.stdout: create 123/39 file after the snapshot 2022-04-03T19:46:09.266 INFO:tasks.workunit.client.0.smithi077.stdout: create 124/.snap/snap-subdir-test 2022-04-03T19:46:09.535 INFO:tasks.workunit.client.0.smithi077.stdout: create 124/30 file after the snapshot 2022-04-03T19:46:09.537 INFO:tasks.workunit.client.0.smithi077.stdout: create 124/31 file after the snapshot 2022-04-03T19:46:09.537 INFO:tasks.workunit.client.0.smithi077.stdout: create 124/32 file after the snapshot 2022-04-03T19:46:09.537 INFO:tasks.workunit.client.0.smithi077.stdout: create 124/33 file after the snapshot 2022-04-03T19:46:09.538 INFO:tasks.workunit.client.0.smithi077.stdout: create 124/34 file after the snapshot 2022-04-03T19:46:09.538 INFO:tasks.workunit.client.0.smithi077.stdout: create 124/35 file after the snapshot 2022-04-03T19:46:09.539 INFO:tasks.workunit.client.0.smithi077.stdout: create 124/36 file after the snapshot 2022-04-03T19:46:09.539 INFO:tasks.workunit.client.0.smithi077.stdout: create 124/37 file after the snapshot 2022-04-03T19:46:09.539 INFO:tasks.workunit.client.0.smithi077.stdout: create 124/38 file after the snapshot 2022-04-03T19:46:09.539 INFO:tasks.workunit.client.0.smithi077.stdout: create 124/39 file after the snapshot 2022-04-03T19:46:09.540 INFO:tasks.workunit.client.0.smithi077.stdout: create 125/.snap/snap-subdir-test 2022-04-03T19:46:09.768 INFO:tasks.workunit.client.0.smithi077.stdout: create 125/30 file after the snapshot 2022-04-03T19:46:09.770 INFO:tasks.workunit.client.0.smithi077.stdout: create 125/31 file after the snapshot 2022-04-03T19:46:09.770 INFO:tasks.workunit.client.0.smithi077.stdout: create 125/32 file after the snapshot 2022-04-03T19:46:09.771 INFO:tasks.workunit.client.0.smithi077.stdout: create 125/33 file after the snapshot 2022-04-03T19:46:09.771 INFO:tasks.workunit.client.0.smithi077.stdout: create 125/34 file after the snapshot 2022-04-03T19:46:09.771 INFO:tasks.workunit.client.0.smithi077.stdout: create 125/35 file after the snapshot 2022-04-03T19:46:09.771 INFO:tasks.workunit.client.0.smithi077.stdout: create 125/36 file after the snapshot 2022-04-03T19:46:09.771 INFO:tasks.workunit.client.0.smithi077.stdout: create 125/37 file after the snapshot 2022-04-03T19:46:09.772 INFO:tasks.workunit.client.0.smithi077.stdout: create 125/38 file after the snapshot 2022-04-03T19:46:09.772 INFO:tasks.workunit.client.0.smithi077.stdout: create 125/39 file after the snapshot 2022-04-03T19:46:09.772 INFO:tasks.workunit.client.0.smithi077.stdout: create 126/.snap/snap-subdir-test 2022-04-03T19:46:09.796 INFO:tasks.workunit.client.0.smithi077.stdout: create 126/30 file after the snapshot 2022-04-03T19:46:09.798 INFO:tasks.workunit.client.0.smithi077.stdout: create 126/31 file after the snapshot 2022-04-03T19:46:09.799 INFO:tasks.workunit.client.0.smithi077.stdout: create 126/32 file after the snapshot 2022-04-03T19:46:09.799 INFO:tasks.workunit.client.0.smithi077.stdout: create 126/33 file after the snapshot 2022-04-03T19:46:09.799 INFO:tasks.workunit.client.0.smithi077.stdout: create 126/34 file after the snapshot 2022-04-03T19:46:09.799 INFO:tasks.workunit.client.0.smithi077.stdout: create 126/35 file after the snapshot 2022-04-03T19:46:09.799 INFO:tasks.workunit.client.0.smithi077.stdout: create 126/36 file after the snapshot 2022-04-03T19:46:09.800 INFO:tasks.workunit.client.0.smithi077.stdout: create 126/37 file after the snapshot 2022-04-03T19:46:09.800 INFO:tasks.workunit.client.0.smithi077.stdout: create 126/38 file after the snapshot 2022-04-03T19:46:09.800 INFO:tasks.workunit.client.0.smithi077.stdout: create 126/39 file after the snapshot 2022-04-03T19:46:09.801 INFO:tasks.workunit.client.0.smithi077.stdout: create 127/.snap/snap-subdir-test 2022-04-03T19:46:09.823 INFO:tasks.workunit.client.0.smithi077.stdout: create 127/30 file after the snapshot 2022-04-03T19:46:09.826 INFO:tasks.workunit.client.0.smithi077.stdout: create 127/31 file after the snapshot 2022-04-03T19:46:09.826 INFO:tasks.workunit.client.0.smithi077.stdout: create 127/32 file after the snapshot 2022-04-03T19:46:09.826 INFO:tasks.workunit.client.0.smithi077.stdout: create 127/33 file after the snapshot 2022-04-03T19:46:09.826 INFO:tasks.workunit.client.0.smithi077.stdout: create 127/34 file after the snapshot 2022-04-03T19:46:09.827 INFO:tasks.workunit.client.0.smithi077.stdout: create 127/35 file after the snapshot 2022-04-03T19:46:09.827 INFO:tasks.workunit.client.0.smithi077.stdout: create 127/36 file after the snapshot 2022-04-03T19:46:09.827 INFO:tasks.workunit.client.0.smithi077.stdout: create 127/37 file after the snapshot 2022-04-03T19:46:09.827 INFO:tasks.workunit.client.0.smithi077.stdout: create 127/38 file after the snapshot 2022-04-03T19:46:09.828 INFO:tasks.workunit.client.0.smithi077.stdout: create 127/39 file after the snapshot 2022-04-03T19:46:09.828 INFO:tasks.workunit.client.0.smithi077.stdout: create 128/.snap/snap-subdir-test 2022-04-03T19:46:09.851 INFO:tasks.workunit.client.0.smithi077.stdout: create 128/30 file after the snapshot 2022-04-03T19:46:09.853 INFO:tasks.workunit.client.0.smithi077.stdout: create 128/31 file after the snapshot 2022-04-03T19:46:09.854 INFO:tasks.workunit.client.0.smithi077.stdout: create 128/32 file after the snapshot 2022-04-03T19:46:09.854 INFO:tasks.workunit.client.0.smithi077.stdout: create 128/33 file after the snapshot 2022-04-03T19:46:09.854 INFO:tasks.workunit.client.0.smithi077.stdout: create 128/34 file after the snapshot 2022-04-03T19:46:09.854 INFO:tasks.workunit.client.0.smithi077.stdout: create 128/35 file after the snapshot 2022-04-03T19:46:09.855 INFO:tasks.workunit.client.0.smithi077.stdout: create 128/36 file after the snapshot 2022-04-03T19:46:09.855 INFO:tasks.workunit.client.0.smithi077.stdout: create 128/37 file after the snapshot 2022-04-03T19:46:09.855 INFO:tasks.workunit.client.0.smithi077.stdout: create 128/38 file after the snapshot 2022-04-03T19:46:09.855 INFO:tasks.workunit.client.0.smithi077.stdout: create 128/39 file after the snapshot 2022-04-03T19:46:09.856 INFO:tasks.workunit.client.0.smithi077.stdout: create 129/.snap/snap-subdir-test 2022-04-03T19:46:09.880 INFO:tasks.workunit.client.0.smithi077.stdout: create 129/30 file after the snapshot 2022-04-03T19:46:09.882 INFO:tasks.workunit.client.0.smithi077.stdout: create 129/31 file after the snapshot 2022-04-03T19:46:09.882 INFO:tasks.workunit.client.0.smithi077.stdout: create 129/32 file after the snapshot 2022-04-03T19:46:09.882 INFO:tasks.workunit.client.0.smithi077.stdout: create 129/33 file after the snapshot 2022-04-03T19:46:09.883 INFO:tasks.workunit.client.0.smithi077.stdout: create 129/34 file after the snapshot 2022-04-03T19:46:09.883 INFO:tasks.workunit.client.0.smithi077.stdout: create 129/35 file after the snapshot 2022-04-03T19:46:09.883 INFO:tasks.workunit.client.0.smithi077.stdout: create 129/36 file after the snapshot 2022-04-03T19:46:09.883 INFO:tasks.workunit.client.0.smithi077.stdout: create 129/37 file after the snapshot 2022-04-03T19:46:09.883 INFO:tasks.workunit.client.0.smithi077.stdout: create 129/38 file after the snapshot 2022-04-03T19:46:09.884 INFO:tasks.workunit.client.0.smithi077.stdout: create 129/39 file after the snapshot 2022-04-03T19:46:09.884 INFO:tasks.workunit.client.0.smithi077.stdout: create 130/.snap/snap-subdir-test 2022-04-03T19:46:09.911 INFO:tasks.workunit.client.0.smithi077.stdout: create 130/30 file after the snapshot 2022-04-03T19:46:09.914 INFO:tasks.workunit.client.0.smithi077.stdout: create 130/31 file after the snapshot 2022-04-03T19:46:09.914 INFO:tasks.workunit.client.0.smithi077.stdout: create 130/32 file after the snapshot 2022-04-03T19:46:09.914 INFO:tasks.workunit.client.0.smithi077.stdout: create 130/33 file after the snapshot 2022-04-03T19:46:09.914 INFO:tasks.workunit.client.0.smithi077.stdout: create 130/34 file after the snapshot 2022-04-03T19:46:09.915 INFO:tasks.workunit.client.0.smithi077.stdout: create 130/35 file after the snapshot 2022-04-03T19:46:09.915 INFO:tasks.workunit.client.0.smithi077.stdout: create 130/36 file after the snapshot 2022-04-03T19:46:09.915 INFO:tasks.workunit.client.0.smithi077.stdout: create 130/37 file after the snapshot 2022-04-03T19:46:09.915 INFO:tasks.workunit.client.0.smithi077.stdout: create 130/38 file after the snapshot 2022-04-03T19:46:09.916 INFO:tasks.workunit.client.0.smithi077.stdout: create 130/39 file after the snapshot 2022-04-03T19:46:09.916 INFO:tasks.workunit.client.0.smithi077.stdout: create 131/.snap/snap-subdir-test 2022-04-03T19:46:09.943 INFO:tasks.workunit.client.0.smithi077.stdout: create 131/30 file after the snapshot 2022-04-03T19:46:09.945 INFO:tasks.workunit.client.0.smithi077.stdout: create 131/31 file after the snapshot 2022-04-03T19:46:09.946 INFO:tasks.workunit.client.0.smithi077.stdout: create 131/32 file after the snapshot 2022-04-03T19:46:09.946 INFO:tasks.workunit.client.0.smithi077.stdout: create 131/33 file after the snapshot 2022-04-03T19:46:09.946 INFO:tasks.workunit.client.0.smithi077.stdout: create 131/34 file after the snapshot 2022-04-03T19:46:09.946 INFO:tasks.workunit.client.0.smithi077.stdout: create 131/35 file after the snapshot 2022-04-03T19:46:09.947 INFO:tasks.workunit.client.0.smithi077.stdout: create 131/36 file after the snapshot 2022-04-03T19:46:09.947 INFO:tasks.workunit.client.0.smithi077.stdout: create 131/37 file after the snapshot 2022-04-03T19:46:09.948 INFO:tasks.workunit.client.0.smithi077.stdout: create 131/38 file after the snapshot 2022-04-03T19:46:09.948 INFO:tasks.workunit.client.0.smithi077.stdout: create 131/39 file after the snapshot 2022-04-03T19:46:09.948 INFO:tasks.workunit.client.0.smithi077.stdout: create 132/.snap/snap-subdir-test 2022-04-03T19:46:09.973 INFO:tasks.workunit.client.0.smithi077.stdout: create 132/30 file after the snapshot 2022-04-03T19:46:09.975 INFO:tasks.workunit.client.0.smithi077.stdout: create 132/31 file after the snapshot 2022-04-03T19:46:09.975 INFO:tasks.workunit.client.0.smithi077.stdout: create 132/32 file after the snapshot 2022-04-03T19:46:09.975 INFO:tasks.workunit.client.0.smithi077.stdout: create 132/33 file after the snapshot 2022-04-03T19:46:09.976 INFO:tasks.workunit.client.0.smithi077.stdout: create 132/34 file after the snapshot 2022-04-03T19:46:09.976 INFO:tasks.workunit.client.0.smithi077.stdout: create 132/35 file after the snapshot 2022-04-03T19:46:09.976 INFO:tasks.workunit.client.0.smithi077.stdout: create 132/36 file after the snapshot 2022-04-03T19:46:09.976 INFO:tasks.workunit.client.0.smithi077.stdout: create 132/37 file after the snapshot 2022-04-03T19:46:09.977 INFO:tasks.workunit.client.0.smithi077.stdout: create 132/38 file after the snapshot 2022-04-03T19:46:09.977 INFO:tasks.workunit.client.0.smithi077.stdout: create 132/39 file after the snapshot 2022-04-03T19:46:09.977 INFO:tasks.workunit.client.0.smithi077.stdout: create 133/.snap/snap-subdir-test 2022-04-03T19:46:10.001 INFO:tasks.workunit.client.0.smithi077.stdout: create 133/30 file after the snapshot 2022-04-03T19:46:10.003 INFO:tasks.workunit.client.0.smithi077.stdout: create 133/31 file after the snapshot 2022-04-03T19:46:10.003 INFO:tasks.workunit.client.0.smithi077.stdout: create 133/32 file after the snapshot 2022-04-03T19:46:10.004 INFO:tasks.workunit.client.0.smithi077.stdout: create 133/33 file after the snapshot 2022-04-03T19:46:10.004 INFO:tasks.workunit.client.0.smithi077.stdout: create 133/34 file after the snapshot 2022-04-03T19:46:10.004 INFO:tasks.workunit.client.0.smithi077.stdout: create 133/35 file after the snapshot 2022-04-03T19:46:10.004 INFO:tasks.workunit.client.0.smithi077.stdout: create 133/36 file after the snapshot 2022-04-03T19:46:10.004 INFO:tasks.workunit.client.0.smithi077.stdout: create 133/37 file after the snapshot 2022-04-03T19:46:10.005 INFO:tasks.workunit.client.0.smithi077.stdout: create 133/38 file after the snapshot 2022-04-03T19:46:10.005 INFO:tasks.workunit.client.0.smithi077.stdout: create 133/39 file after the snapshot 2022-04-03T19:46:10.005 INFO:tasks.workunit.client.0.smithi077.stdout: create 134/.snap/snap-subdir-test 2022-04-03T19:46:10.030 INFO:tasks.workunit.client.0.smithi077.stdout: create 134/30 file after the snapshot 2022-04-03T19:46:10.032 INFO:tasks.workunit.client.0.smithi077.stdout: create 134/31 file after the snapshot 2022-04-03T19:46:10.032 INFO:tasks.workunit.client.0.smithi077.stdout: create 134/32 file after the snapshot 2022-04-03T19:46:10.032 INFO:tasks.workunit.client.0.smithi077.stdout: create 134/33 file after the snapshot 2022-04-03T19:46:10.033 INFO:tasks.workunit.client.0.smithi077.stdout: create 134/34 file after the snapshot 2022-04-03T19:46:10.033 INFO:tasks.workunit.client.0.smithi077.stdout: create 134/35 file after the snapshot 2022-04-03T19:46:10.033 INFO:tasks.workunit.client.0.smithi077.stdout: create 134/36 file after the snapshot 2022-04-03T19:46:10.033 INFO:tasks.workunit.client.0.smithi077.stdout: create 134/37 file after the snapshot 2022-04-03T19:46:10.033 INFO:tasks.workunit.client.0.smithi077.stdout: create 134/38 file after the snapshot 2022-04-03T19:46:10.034 INFO:tasks.workunit.client.0.smithi077.stdout: create 134/39 file after the snapshot 2022-04-03T19:46:10.034 INFO:tasks.workunit.client.0.smithi077.stdout: create 135/.snap/snap-subdir-test 2022-04-03T19:46:10.058 INFO:tasks.workunit.client.0.smithi077.stdout: create 135/30 file after the snapshot 2022-04-03T19:46:10.060 INFO:tasks.workunit.client.0.smithi077.stdout: create 135/31 file after the snapshot 2022-04-03T19:46:10.060 INFO:tasks.workunit.client.0.smithi077.stdout: create 135/32 file after the snapshot 2022-04-03T19:46:10.060 INFO:tasks.workunit.client.0.smithi077.stdout: create 135/33 file after the snapshot 2022-04-03T19:46:10.060 INFO:tasks.workunit.client.0.smithi077.stdout: create 135/34 file after the snapshot 2022-04-03T19:46:10.061 INFO:tasks.workunit.client.0.smithi077.stdout: create 135/35 file after the snapshot 2022-04-03T19:46:10.061 INFO:tasks.workunit.client.0.smithi077.stdout: create 135/36 file after the snapshot 2022-04-03T19:46:10.061 INFO:tasks.workunit.client.0.smithi077.stdout: create 135/37 file after the snapshot 2022-04-03T19:46:10.061 INFO:tasks.workunit.client.0.smithi077.stdout: create 135/38 file after the snapshot 2022-04-03T19:46:10.062 INFO:tasks.workunit.client.0.smithi077.stdout: create 135/39 file after the snapshot 2022-04-03T19:46:10.062 INFO:tasks.workunit.client.0.smithi077.stdout: create 136/.snap/snap-subdir-test 2022-04-03T19:46:10.421 INFO:tasks.workunit.client.0.smithi077.stdout: create 136/30 file after the snapshot 2022-04-03T19:46:10.423 INFO:tasks.workunit.client.0.smithi077.stdout: create 136/31 file after the snapshot 2022-04-03T19:46:10.423 INFO:tasks.workunit.client.0.smithi077.stdout: create 136/32 file after the snapshot 2022-04-03T19:46:10.424 INFO:tasks.workunit.client.0.smithi077.stdout: create 136/33 file after the snapshot 2022-04-03T19:46:10.424 INFO:tasks.workunit.client.0.smithi077.stdout: create 136/34 file after the snapshot 2022-04-03T19:46:10.424 INFO:tasks.workunit.client.0.smithi077.stdout: create 136/35 file after the snapshot 2022-04-03T19:46:10.424 INFO:tasks.workunit.client.0.smithi077.stdout: create 136/36 file after the snapshot 2022-04-03T19:46:10.425 INFO:tasks.workunit.client.0.smithi077.stdout: create 136/37 file after the snapshot 2022-04-03T19:46:10.425 INFO:tasks.workunit.client.0.smithi077.stdout: create 136/38 file after the snapshot 2022-04-03T19:46:10.425 INFO:tasks.workunit.client.0.smithi077.stdout: create 136/39 file after the snapshot 2022-04-03T19:46:10.425 INFO:tasks.workunit.client.0.smithi077.stdout: create 137/.snap/snap-subdir-test 2022-04-03T19:46:10.460 INFO:tasks.workunit.client.0.smithi077.stdout: create 137/30 file after the snapshot 2022-04-03T19:46:10.463 INFO:tasks.workunit.client.0.smithi077.stdout: create 137/31 file after the snapshot 2022-04-03T19:46:10.463 INFO:tasks.workunit.client.0.smithi077.stdout: create 137/32 file after the snapshot 2022-04-03T19:46:10.463 INFO:tasks.workunit.client.0.smithi077.stdout: create 137/33 file after the snapshot 2022-04-03T19:46:10.463 INFO:tasks.workunit.client.0.smithi077.stdout: create 137/34 file after the snapshot 2022-04-03T19:46:10.464 INFO:tasks.workunit.client.0.smithi077.stdout: create 137/35 file after the snapshot 2022-04-03T19:46:10.464 INFO:tasks.workunit.client.0.smithi077.stdout: create 137/36 file after the snapshot 2022-04-03T19:46:10.464 INFO:tasks.workunit.client.0.smithi077.stdout: create 137/37 file after the snapshot 2022-04-03T19:46:10.464 INFO:tasks.workunit.client.0.smithi077.stdout: create 137/38 file after the snapshot 2022-04-03T19:46:10.464 INFO:tasks.workunit.client.0.smithi077.stdout: create 137/39 file after the snapshot 2022-04-03T19:46:10.465 INFO:tasks.workunit.client.0.smithi077.stdout: create 138/.snap/snap-subdir-test 2022-04-03T19:46:10.492 INFO:tasks.workunit.client.0.smithi077.stdout: create 138/30 file after the snapshot 2022-04-03T19:46:10.494 INFO:tasks.workunit.client.0.smithi077.stdout: create 138/31 file after the snapshot 2022-04-03T19:46:10.495 INFO:tasks.workunit.client.0.smithi077.stdout: create 138/32 file after the snapshot 2022-04-03T19:46:10.495 INFO:tasks.workunit.client.0.smithi077.stdout: create 138/33 file after the snapshot 2022-04-03T19:46:10.495 INFO:tasks.workunit.client.0.smithi077.stdout: create 138/34 file after the snapshot 2022-04-03T19:46:10.495 INFO:tasks.workunit.client.0.smithi077.stdout: create 138/35 file after the snapshot 2022-04-03T19:46:10.496 INFO:tasks.workunit.client.0.smithi077.stdout: create 138/36 file after the snapshot 2022-04-03T19:46:10.496 INFO:tasks.workunit.client.0.smithi077.stdout: create 138/37 file after the snapshot 2022-04-03T19:46:10.496 INFO:tasks.workunit.client.0.smithi077.stdout: create 138/38 file after the snapshot 2022-04-03T19:46:10.496 INFO:tasks.workunit.client.0.smithi077.stdout: create 138/39 file after the snapshot 2022-04-03T19:46:10.497 INFO:tasks.workunit.client.0.smithi077.stdout: create 139/.snap/snap-subdir-test 2022-04-03T19:46:10.523 INFO:tasks.workunit.client.0.smithi077.stdout: create 139/30 file after the snapshot 2022-04-03T19:46:10.526 INFO:tasks.workunit.client.0.smithi077.stdout: create 139/31 file after the snapshot 2022-04-03T19:46:10.526 INFO:tasks.workunit.client.0.smithi077.stdout: create 139/32 file after the snapshot 2022-04-03T19:46:10.526 INFO:tasks.workunit.client.0.smithi077.stdout: create 139/33 file after the snapshot 2022-04-03T19:46:10.526 INFO:tasks.workunit.client.0.smithi077.stdout: create 139/34 file after the snapshot 2022-04-03T19:46:10.527 INFO:tasks.workunit.client.0.smithi077.stdout: create 139/35 file after the snapshot 2022-04-03T19:46:10.527 INFO:tasks.workunit.client.0.smithi077.stdout: create 139/36 file after the snapshot 2022-04-03T19:46:10.527 INFO:tasks.workunit.client.0.smithi077.stdout: create 139/37 file after the snapshot 2022-04-03T19:46:10.527 INFO:tasks.workunit.client.0.smithi077.stdout: create 139/38 file after the snapshot 2022-04-03T19:46:10.528 INFO:tasks.workunit.client.0.smithi077.stdout: create 139/39 file after the snapshot 2022-04-03T19:46:10.528 INFO:tasks.workunit.client.0.smithi077.stdout: create 140/.snap/snap-subdir-test 2022-04-03T19:46:10.551 INFO:tasks.workunit.client.0.smithi077.stdout: create 140/30 file after the snapshot 2022-04-03T19:46:10.553 INFO:tasks.workunit.client.0.smithi077.stdout: create 140/31 file after the snapshot 2022-04-03T19:46:10.553 INFO:tasks.workunit.client.0.smithi077.stdout: create 140/32 file after the snapshot 2022-04-03T19:46:10.553 INFO:tasks.workunit.client.0.smithi077.stdout: create 140/33 file after the snapshot 2022-04-03T19:46:10.553 INFO:tasks.workunit.client.0.smithi077.stdout: create 140/34 file after the snapshot 2022-04-03T19:46:10.554 INFO:tasks.workunit.client.0.smithi077.stdout: create 140/35 file after the snapshot 2022-04-03T19:46:10.554 INFO:tasks.workunit.client.0.smithi077.stdout: create 140/36 file after the snapshot 2022-04-03T19:46:10.554 INFO:tasks.workunit.client.0.smithi077.stdout: create 140/37 file after the snapshot 2022-04-03T19:46:10.554 INFO:tasks.workunit.client.0.smithi077.stdout: create 140/38 file after the snapshot 2022-04-03T19:46:10.555 INFO:tasks.workunit.client.0.smithi077.stdout: create 140/39 file after the snapshot 2022-04-03T19:46:10.555 INFO:tasks.workunit.client.0.smithi077.stdout: create 141/.snap/snap-subdir-test 2022-04-03T19:46:10.577 INFO:tasks.workunit.client.0.smithi077.stdout: create 141/30 file after the snapshot 2022-04-03T19:46:10.579 INFO:tasks.workunit.client.0.smithi077.stdout: create 141/31 file after the snapshot 2022-04-03T19:46:10.580 INFO:tasks.workunit.client.0.smithi077.stdout: create 141/32 file after the snapshot 2022-04-03T19:46:10.580 INFO:tasks.workunit.client.0.smithi077.stdout: create 141/33 file after the snapshot 2022-04-03T19:46:10.580 INFO:tasks.workunit.client.0.smithi077.stdout: create 141/34 file after the snapshot 2022-04-03T19:46:10.580 INFO:tasks.workunit.client.0.smithi077.stdout: create 141/35 file after the snapshot 2022-04-03T19:46:10.581 INFO:tasks.workunit.client.0.smithi077.stdout: create 141/36 file after the snapshot 2022-04-03T19:46:10.581 INFO:tasks.workunit.client.0.smithi077.stdout: create 141/37 file after the snapshot 2022-04-03T19:46:10.581 INFO:tasks.workunit.client.0.smithi077.stdout: create 141/38 file after the snapshot 2022-04-03T19:46:10.581 INFO:tasks.workunit.client.0.smithi077.stdout: create 141/39 file after the snapshot 2022-04-03T19:46:10.582 INFO:tasks.workunit.client.0.smithi077.stdout: create 142/.snap/snap-subdir-test 2022-04-03T19:46:10.604 INFO:tasks.workunit.client.0.smithi077.stdout: create 142/30 file after the snapshot 2022-04-03T19:46:10.607 INFO:tasks.workunit.client.0.smithi077.stdout: create 142/31 file after the snapshot 2022-04-03T19:46:10.607 INFO:tasks.workunit.client.0.smithi077.stdout: create 142/32 file after the snapshot 2022-04-03T19:46:10.607 INFO:tasks.workunit.client.0.smithi077.stdout: create 142/33 file after the snapshot 2022-04-03T19:46:10.607 INFO:tasks.workunit.client.0.smithi077.stdout: create 142/34 file after the snapshot 2022-04-03T19:46:10.608 INFO:tasks.workunit.client.0.smithi077.stdout: create 142/35 file after the snapshot 2022-04-03T19:46:10.608 INFO:tasks.workunit.client.0.smithi077.stdout: create 142/36 file after the snapshot 2022-04-03T19:46:10.608 INFO:tasks.workunit.client.0.smithi077.stdout: create 142/37 file after the snapshot 2022-04-03T19:46:10.608 INFO:tasks.workunit.client.0.smithi077.stdout: create 142/38 file after the snapshot 2022-04-03T19:46:10.608 INFO:tasks.workunit.client.0.smithi077.stdout: create 142/39 file after the snapshot 2022-04-03T19:46:10.609 INFO:tasks.workunit.client.0.smithi077.stdout: create 143/.snap/snap-subdir-test 2022-04-03T19:46:10.633 INFO:tasks.workunit.client.0.smithi077.stdout: create 143/30 file after the snapshot 2022-04-03T19:46:10.636 INFO:tasks.workunit.client.0.smithi077.stdout: create 143/31 file after the snapshot 2022-04-03T19:46:10.636 INFO:tasks.workunit.client.0.smithi077.stdout: create 143/32 file after the snapshot 2022-04-03T19:46:10.636 INFO:tasks.workunit.client.0.smithi077.stdout: create 143/33 file after the snapshot 2022-04-03T19:46:10.636 INFO:tasks.workunit.client.0.smithi077.stdout: create 143/34 file after the snapshot 2022-04-03T19:46:10.637 INFO:tasks.workunit.client.0.smithi077.stdout: create 143/35 file after the snapshot 2022-04-03T19:46:10.637 INFO:tasks.workunit.client.0.smithi077.stdout: create 143/36 file after the snapshot 2022-04-03T19:46:10.637 INFO:tasks.workunit.client.0.smithi077.stdout: create 143/37 file after the snapshot 2022-04-03T19:46:10.637 INFO:tasks.workunit.client.0.smithi077.stdout: create 143/38 file after the snapshot 2022-04-03T19:46:10.638 INFO:tasks.workunit.client.0.smithi077.stdout: create 143/39 file after the snapshot 2022-04-03T19:46:10.638 INFO:tasks.workunit.client.0.smithi077.stdout: create 144/.snap/snap-subdir-test 2022-04-03T19:46:10.663 INFO:tasks.workunit.client.0.smithi077.stdout: create 144/30 file after the snapshot 2022-04-03T19:46:10.665 INFO:tasks.workunit.client.0.smithi077.stdout: create 144/31 file after the snapshot 2022-04-03T19:46:10.665 INFO:tasks.workunit.client.0.smithi077.stdout: create 144/32 file after the snapshot 2022-04-03T19:46:10.665 INFO:tasks.workunit.client.0.smithi077.stdout: create 144/33 file after the snapshot 2022-04-03T19:46:10.666 INFO:tasks.workunit.client.0.smithi077.stdout: create 144/34 file after the snapshot 2022-04-03T19:46:10.666 INFO:tasks.workunit.client.0.smithi077.stdout: create 144/35 file after the snapshot 2022-04-03T19:46:10.666 INFO:tasks.workunit.client.0.smithi077.stdout: create 144/36 file after the snapshot 2022-04-03T19:46:10.666 INFO:tasks.workunit.client.0.smithi077.stdout: create 144/37 file after the snapshot 2022-04-03T19:46:10.667 INFO:tasks.workunit.client.0.smithi077.stdout: create 144/38 file after the snapshot 2022-04-03T19:46:10.667 INFO:tasks.workunit.client.0.smithi077.stdout: create 144/39 file after the snapshot 2022-04-03T19:46:10.667 INFO:tasks.workunit.client.0.smithi077.stdout: create 145/.snap/snap-subdir-test 2022-04-03T19:46:10.691 INFO:tasks.workunit.client.0.smithi077.stdout: create 145/30 file after the snapshot 2022-04-03T19:46:10.694 INFO:tasks.workunit.client.0.smithi077.stdout: create 145/31 file after the snapshot 2022-04-03T19:46:10.694 INFO:tasks.workunit.client.0.smithi077.stdout: create 145/32 file after the snapshot 2022-04-03T19:46:10.694 INFO:tasks.workunit.client.0.smithi077.stdout: create 145/33 file after the snapshot 2022-04-03T19:46:10.695 INFO:tasks.workunit.client.0.smithi077.stdout: create 145/34 file after the snapshot 2022-04-03T19:46:10.695 INFO:tasks.workunit.client.0.smithi077.stdout: create 145/35 file after the snapshot 2022-04-03T19:46:10.695 INFO:tasks.workunit.client.0.smithi077.stdout: create 145/36 file after the snapshot 2022-04-03T19:46:10.695 INFO:tasks.workunit.client.0.smithi077.stdout: create 145/37 file after the snapshot 2022-04-03T19:46:10.696 INFO:tasks.workunit.client.0.smithi077.stdout: create 145/38 file after the snapshot 2022-04-03T19:46:10.696 INFO:tasks.workunit.client.0.smithi077.stdout: create 145/39 file after the snapshot 2022-04-03T19:46:10.696 INFO:tasks.workunit.client.0.smithi077.stdout: create 146/.snap/snap-subdir-test 2022-04-03T19:46:10.722 INFO:tasks.workunit.client.0.smithi077.stdout: create 146/30 file after the snapshot 2022-04-03T19:46:10.724 INFO:tasks.workunit.client.0.smithi077.stdout: create 146/31 file after the snapshot 2022-04-03T19:46:10.724 INFO:tasks.workunit.client.0.smithi077.stdout: create 146/32 file after the snapshot 2022-04-03T19:46:10.725 INFO:tasks.workunit.client.0.smithi077.stdout: create 146/33 file after the snapshot 2022-04-03T19:46:10.726 INFO:tasks.workunit.client.0.smithi077.stdout: create 146/34 file after the snapshot 2022-04-03T19:46:10.726 INFO:tasks.workunit.client.0.smithi077.stdout: create 146/35 file after the snapshot 2022-04-03T19:46:10.727 INFO:tasks.workunit.client.0.smithi077.stdout: create 146/36 file after the snapshot 2022-04-03T19:46:10.727 INFO:tasks.workunit.client.0.smithi077.stdout: create 146/37 file after the snapshot 2022-04-03T19:46:10.727 INFO:tasks.workunit.client.0.smithi077.stdout: create 146/38 file after the snapshot 2022-04-03T19:46:10.727 INFO:tasks.workunit.client.0.smithi077.stdout: create 146/39 file after the snapshot 2022-04-03T19:46:10.727 INFO:tasks.workunit.client.0.smithi077.stdout: create 147/.snap/snap-subdir-test 2022-04-03T19:46:10.752 INFO:tasks.workunit.client.0.smithi077.stdout: create 147/30 file after the snapshot 2022-04-03T19:46:10.755 INFO:tasks.workunit.client.0.smithi077.stdout: create 147/31 file after the snapshot 2022-04-03T19:46:10.755 INFO:tasks.workunit.client.0.smithi077.stdout: create 147/32 file after the snapshot 2022-04-03T19:46:10.755 INFO:tasks.workunit.client.0.smithi077.stdout: create 147/33 file after the snapshot 2022-04-03T19:46:10.756 INFO:tasks.workunit.client.0.smithi077.stdout: create 147/34 file after the snapshot 2022-04-03T19:46:10.756 INFO:tasks.workunit.client.0.smithi077.stdout: create 147/35 file after the snapshot 2022-04-03T19:46:10.756 INFO:tasks.workunit.client.0.smithi077.stdout: create 147/36 file after the snapshot 2022-04-03T19:46:10.756 INFO:tasks.workunit.client.0.smithi077.stdout: create 147/37 file after the snapshot 2022-04-03T19:46:10.757 INFO:tasks.workunit.client.0.smithi077.stdout: create 147/38 file after the snapshot 2022-04-03T19:46:10.757 INFO:tasks.workunit.client.0.smithi077.stdout: create 147/39 file after the snapshot 2022-04-03T19:46:10.757 INFO:tasks.workunit.client.0.smithi077.stdout: create 148/.snap/snap-subdir-test 2022-04-03T19:46:10.780 INFO:tasks.workunit.client.0.smithi077.stdout: create 148/30 file after the snapshot 2022-04-03T19:46:10.782 INFO:tasks.workunit.client.0.smithi077.stdout: create 148/31 file after the snapshot 2022-04-03T19:46:10.783 INFO:tasks.workunit.client.0.smithi077.stdout: create 148/32 file after the snapshot 2022-04-03T19:46:10.783 INFO:tasks.workunit.client.0.smithi077.stdout: create 148/33 file after the snapshot 2022-04-03T19:46:10.783 INFO:tasks.workunit.client.0.smithi077.stdout: create 148/34 file after the snapshot 2022-04-03T19:46:10.783 INFO:tasks.workunit.client.0.smithi077.stdout: create 148/35 file after the snapshot 2022-04-03T19:46:10.784 INFO:tasks.workunit.client.0.smithi077.stdout: create 148/36 file after the snapshot 2022-04-03T19:46:10.784 INFO:tasks.workunit.client.0.smithi077.stdout: create 148/37 file after the snapshot 2022-04-03T19:46:10.784 INFO:tasks.workunit.client.0.smithi077.stdout: create 148/38 file after the snapshot 2022-04-03T19:46:10.784 INFO:tasks.workunit.client.0.smithi077.stdout: create 148/39 file after the snapshot 2022-04-03T19:46:10.785 INFO:tasks.workunit.client.0.smithi077.stdout: create 149/.snap/snap-subdir-test 2022-04-03T19:46:10.807 INFO:tasks.workunit.client.0.smithi077.stdout: create 149/30 file after the snapshot 2022-04-03T19:46:10.809 INFO:tasks.workunit.client.0.smithi077.stdout: create 149/31 file after the snapshot 2022-04-03T19:46:10.809 INFO:tasks.workunit.client.0.smithi077.stdout: create 149/32 file after the snapshot 2022-04-03T19:46:10.810 INFO:tasks.workunit.client.0.smithi077.stdout: create 149/33 file after the snapshot 2022-04-03T19:46:10.810 INFO:tasks.workunit.client.0.smithi077.stdout: create 149/34 file after the snapshot 2022-04-03T19:46:10.810 INFO:tasks.workunit.client.0.smithi077.stdout: create 149/35 file after the snapshot 2022-04-03T19:46:10.810 INFO:tasks.workunit.client.0.smithi077.stdout: create 149/36 file after the snapshot 2022-04-03T19:46:10.811 INFO:tasks.workunit.client.0.smithi077.stdout: create 149/37 file after the snapshot 2022-04-03T19:46:10.811 INFO:tasks.workunit.client.0.smithi077.stdout: create 149/38 file after the snapshot 2022-04-03T19:46:10.811 INFO:tasks.workunit.client.0.smithi077.stdout: create 149/39 file after the snapshot 2022-04-03T19:46:10.811 INFO:tasks.workunit.client.0.smithi077.stdout: create 150/.snap/snap-subdir-test 2022-04-03T19:46:10.835 INFO:tasks.workunit.client.0.smithi077.stdout: create 150/30 file after the snapshot 2022-04-03T19:46:10.837 INFO:tasks.workunit.client.0.smithi077.stdout: create 150/31 file after the snapshot 2022-04-03T19:46:10.837 INFO:tasks.workunit.client.0.smithi077.stdout: create 150/32 file after the snapshot 2022-04-03T19:46:10.837 INFO:tasks.workunit.client.0.smithi077.stdout: create 150/33 file after the snapshot 2022-04-03T19:46:10.837 INFO:tasks.workunit.client.0.smithi077.stdout: create 150/34 file after the snapshot 2022-04-03T19:46:10.838 INFO:tasks.workunit.client.0.smithi077.stdout: create 150/35 file after the snapshot 2022-04-03T19:46:10.838 INFO:tasks.workunit.client.0.smithi077.stdout: create 150/36 file after the snapshot 2022-04-03T19:46:10.838 INFO:tasks.workunit.client.0.smithi077.stdout: create 150/37 file after the snapshot 2022-04-03T19:46:10.838 INFO:tasks.workunit.client.0.smithi077.stdout: create 150/38 file after the snapshot 2022-04-03T19:46:10.839 INFO:tasks.workunit.client.0.smithi077.stdout: create 150/39 file after the snapshot 2022-04-03T19:46:10.839 INFO:tasks.workunit.client.0.smithi077.stdout: create 151/.snap/snap-subdir-test 2022-04-03T19:46:10.862 INFO:tasks.workunit.client.0.smithi077.stdout: create 151/30 file after the snapshot 2022-04-03T19:46:10.864 INFO:tasks.workunit.client.0.smithi077.stdout: create 151/31 file after the snapshot 2022-04-03T19:46:10.864 INFO:tasks.workunit.client.0.smithi077.stdout: create 151/32 file after the snapshot 2022-04-03T19:46:10.864 INFO:tasks.workunit.client.0.smithi077.stdout: create 151/33 file after the snapshot 2022-04-03T19:46:10.865 INFO:tasks.workunit.client.0.smithi077.stdout: create 151/34 file after the snapshot 2022-04-03T19:46:10.865 INFO:tasks.workunit.client.0.smithi077.stdout: create 151/35 file after the snapshot 2022-04-03T19:46:10.866 INFO:tasks.workunit.client.0.smithi077.stdout: create 151/36 file after the snapshot 2022-04-03T19:46:10.866 INFO:tasks.workunit.client.0.smithi077.stdout: create 151/37 file after the snapshot 2022-04-03T19:46:10.866 INFO:tasks.workunit.client.0.smithi077.stdout: create 151/38 file after the snapshot 2022-04-03T19:46:10.867 INFO:tasks.workunit.client.0.smithi077.stdout: create 151/39 file after the snapshot 2022-04-03T19:46:10.867 INFO:tasks.workunit.client.0.smithi077.stdout: create 152/.snap/snap-subdir-test 2022-04-03T19:46:10.889 INFO:tasks.workunit.client.0.smithi077.stdout: create 152/30 file after the snapshot 2022-04-03T19:46:10.891 INFO:tasks.workunit.client.0.smithi077.stdout: create 152/31 file after the snapshot 2022-04-03T19:46:10.891 INFO:tasks.workunit.client.0.smithi077.stdout: create 152/32 file after the snapshot 2022-04-03T19:46:10.891 INFO:tasks.workunit.client.0.smithi077.stdout: create 152/33 file after the snapshot 2022-04-03T19:46:10.892 INFO:tasks.workunit.client.0.smithi077.stdout: create 152/34 file after the snapshot 2022-04-03T19:46:10.892 INFO:tasks.workunit.client.0.smithi077.stdout: create 152/35 file after the snapshot 2022-04-03T19:46:10.893 INFO:tasks.workunit.client.0.smithi077.stdout: create 152/36 file after the snapshot 2022-04-03T19:46:10.893 INFO:tasks.workunit.client.0.smithi077.stdout: create 152/37 file after the snapshot 2022-04-03T19:46:10.893 INFO:tasks.workunit.client.0.smithi077.stdout: create 152/38 file after the snapshot 2022-04-03T19:46:10.894 INFO:tasks.workunit.client.0.smithi077.stdout: create 152/39 file after the snapshot 2022-04-03T19:46:10.894 INFO:tasks.workunit.client.0.smithi077.stdout: create 153/.snap/snap-subdir-test 2022-04-03T19:46:10.920 INFO:tasks.workunit.client.0.smithi077.stdout: create 153/30 file after the snapshot 2022-04-03T19:46:10.921 INFO:tasks.workunit.client.0.smithi077.stdout: create 153/31 file after the snapshot 2022-04-03T19:46:10.921 INFO:tasks.workunit.client.0.smithi077.stdout: create 153/32 file after the snapshot 2022-04-03T19:46:10.922 INFO:tasks.workunit.client.0.smithi077.stdout: create 153/33 file after the snapshot 2022-04-03T19:46:10.922 INFO:tasks.workunit.client.0.smithi077.stdout: create 153/34 file after the snapshot 2022-04-03T19:46:10.923 INFO:tasks.workunit.client.0.smithi077.stdout: create 153/35 file after the snapshot 2022-04-03T19:46:10.923 INFO:tasks.workunit.client.0.smithi077.stdout: create 153/36 file after the snapshot 2022-04-03T19:46:10.924 INFO:tasks.workunit.client.0.smithi077.stdout: create 153/37 file after the snapshot 2022-04-03T19:46:10.924 INFO:tasks.workunit.client.0.smithi077.stdout: create 153/38 file after the snapshot 2022-04-03T19:46:10.924 INFO:tasks.workunit.client.0.smithi077.stdout: create 153/39 file after the snapshot 2022-04-03T19:46:10.924 INFO:tasks.workunit.client.0.smithi077.stdout: create 154/.snap/snap-subdir-test 2022-04-03T19:46:10.950 INFO:tasks.workunit.client.0.smithi077.stdout: create 154/30 file after the snapshot 2022-04-03T19:46:10.952 INFO:tasks.workunit.client.0.smithi077.stdout: create 154/31 file after the snapshot 2022-04-03T19:46:10.953 INFO:tasks.workunit.client.0.smithi077.stdout: create 154/32 file after the snapshot 2022-04-03T19:46:10.953 INFO:tasks.workunit.client.0.smithi077.stdout: create 154/33 file after the snapshot 2022-04-03T19:46:10.953 INFO:tasks.workunit.client.0.smithi077.stdout: create 154/34 file after the snapshot 2022-04-03T19:46:10.954 INFO:tasks.workunit.client.0.smithi077.stdout: create 154/35 file after the snapshot 2022-04-03T19:46:10.954 INFO:tasks.workunit.client.0.smithi077.stdout: create 154/36 file after the snapshot 2022-04-03T19:46:10.955 INFO:tasks.workunit.client.0.smithi077.stdout: create 154/37 file after the snapshot 2022-04-03T19:46:10.955 INFO:tasks.workunit.client.0.smithi077.stdout: create 154/38 file after the snapshot 2022-04-03T19:46:10.955 INFO:tasks.workunit.client.0.smithi077.stdout: create 154/39 file after the snapshot 2022-04-03T19:46:10.955 INFO:tasks.workunit.client.0.smithi077.stdout: create 155/.snap/snap-subdir-test 2022-04-03T19:46:11.404 INFO:tasks.workunit.client.0.smithi077.stdout: create 155/30 file after the snapshot 2022-04-03T19:46:11.406 INFO:tasks.workunit.client.0.smithi077.stdout: create 155/31 file after the snapshot 2022-04-03T19:46:11.406 INFO:tasks.workunit.client.0.smithi077.stdout: create 155/32 file after the snapshot 2022-04-03T19:46:11.406 INFO:tasks.workunit.client.0.smithi077.stdout: create 155/33 file after the snapshot 2022-04-03T19:46:11.408 INFO:tasks.workunit.client.0.smithi077.stdout: create 155/34 file after the snapshot 2022-04-03T19:46:11.408 INFO:tasks.workunit.client.0.smithi077.stdout: create 155/35 file after the snapshot 2022-04-03T19:46:11.408 INFO:tasks.workunit.client.0.smithi077.stdout: create 155/36 file after the snapshot 2022-04-03T19:46:11.408 INFO:tasks.workunit.client.0.smithi077.stdout: create 155/37 file after the snapshot 2022-04-03T19:46:11.409 INFO:tasks.workunit.client.0.smithi077.stdout: create 155/38 file after the snapshot 2022-04-03T19:46:11.409 INFO:tasks.workunit.client.0.smithi077.stdout: create 155/39 file after the snapshot 2022-04-03T19:46:11.409 INFO:tasks.workunit.client.0.smithi077.stdout: create 156/.snap/snap-subdir-test 2022-04-03T19:46:11.437 INFO:tasks.workunit.client.0.smithi077.stdout: create 156/30 file after the snapshot 2022-04-03T19:46:11.439 INFO:tasks.workunit.client.0.smithi077.stdout: create 156/31 file after the snapshot 2022-04-03T19:46:11.439 INFO:tasks.workunit.client.0.smithi077.stdout: create 156/32 file after the snapshot 2022-04-03T19:46:11.440 INFO:tasks.workunit.client.0.smithi077.stdout: create 156/33 file after the snapshot 2022-04-03T19:46:11.440 INFO:tasks.workunit.client.0.smithi077.stdout: create 156/34 file after the snapshot 2022-04-03T19:46:11.440 INFO:tasks.workunit.client.0.smithi077.stdout: create 156/35 file after the snapshot 2022-04-03T19:46:11.441 INFO:tasks.workunit.client.0.smithi077.stdout: create 156/36 file after the snapshot 2022-04-03T19:46:11.441 INFO:tasks.workunit.client.0.smithi077.stdout: create 156/37 file after the snapshot 2022-04-03T19:46:11.441 INFO:tasks.workunit.client.0.smithi077.stdout: create 156/38 file after the snapshot 2022-04-03T19:46:11.442 INFO:tasks.workunit.client.0.smithi077.stdout: create 156/39 file after the snapshot 2022-04-03T19:46:11.442 INFO:tasks.workunit.client.0.smithi077.stdout: create 157/.snap/snap-subdir-test 2022-04-03T19:46:11.469 INFO:tasks.workunit.client.0.smithi077.stdout: create 157/30 file after the snapshot 2022-04-03T19:46:11.471 INFO:tasks.workunit.client.0.smithi077.stdout: create 157/31 file after the snapshot 2022-04-03T19:46:11.472 INFO:tasks.workunit.client.0.smithi077.stdout: create 157/32 file after the snapshot 2022-04-03T19:46:11.472 INFO:tasks.workunit.client.0.smithi077.stdout: create 157/33 file after the snapshot 2022-04-03T19:46:11.472 INFO:tasks.workunit.client.0.smithi077.stdout: create 157/34 file after the snapshot 2022-04-03T19:46:11.472 INFO:tasks.workunit.client.0.smithi077.stdout: create 157/35 file after the snapshot 2022-04-03T19:46:11.473 INFO:tasks.workunit.client.0.smithi077.stdout: create 157/36 file after the snapshot 2022-04-03T19:46:11.474 INFO:tasks.workunit.client.0.smithi077.stdout: create 157/37 file after the snapshot 2022-04-03T19:46:11.474 INFO:tasks.workunit.client.0.smithi077.stdout: create 157/38 file after the snapshot 2022-04-03T19:46:11.474 INFO:tasks.workunit.client.0.smithi077.stdout: create 157/39 file after the snapshot 2022-04-03T19:46:11.474 INFO:tasks.workunit.client.0.smithi077.stdout: create 158/.snap/snap-subdir-test 2022-04-03T19:46:12.325 INFO:tasks.workunit.client.0.smithi077.stdout: create 158/30 file after the snapshot 2022-04-03T19:46:12.427 INFO:tasks.workunit.client.0.smithi077.stdout: create 158/31 file after the snapshot 2022-04-03T19:46:12.427 INFO:tasks.workunit.client.0.smithi077.stdout: create 158/32 file after the snapshot 2022-04-03T19:46:12.428 INFO:tasks.workunit.client.0.smithi077.stdout: create 158/33 file after the snapshot 2022-04-03T19:46:12.428 INFO:tasks.workunit.client.0.smithi077.stdout: create 158/34 file after the snapshot 2022-04-03T19:46:12.428 INFO:tasks.workunit.client.0.smithi077.stdout: create 158/35 file after the snapshot 2022-04-03T19:46:12.429 INFO:tasks.workunit.client.0.smithi077.stdout: create 158/36 file after the snapshot 2022-04-03T19:46:12.429 INFO:tasks.workunit.client.0.smithi077.stdout: create 158/37 file after the snapshot 2022-04-03T19:46:12.429 INFO:tasks.workunit.client.0.smithi077.stdout: create 158/38 file after the snapshot 2022-04-03T19:46:12.429 INFO:tasks.workunit.client.0.smithi077.stdout: create 158/39 file after the snapshot 2022-04-03T19:46:12.429 INFO:tasks.workunit.client.0.smithi077.stdout: create 159/.snap/snap-subdir-test 2022-04-03T19:46:12.454 INFO:tasks.workunit.client.0.smithi077.stdout: create 159/30 file after the snapshot 2022-04-03T19:46:12.456 INFO:tasks.workunit.client.0.smithi077.stdout: create 159/31 file after the snapshot 2022-04-03T19:46:12.457 INFO:tasks.workunit.client.0.smithi077.stdout: create 159/32 file after the snapshot 2022-04-03T19:46:12.457 INFO:tasks.workunit.client.0.smithi077.stdout: create 159/33 file after the snapshot 2022-04-03T19:46:12.457 INFO:tasks.workunit.client.0.smithi077.stdout: create 159/34 file after the snapshot 2022-04-03T19:46:12.458 INFO:tasks.workunit.client.0.smithi077.stdout: create 159/35 file after the snapshot 2022-04-03T19:46:12.458 INFO:tasks.workunit.client.0.smithi077.stdout: create 159/36 file after the snapshot 2022-04-03T19:46:12.458 INFO:tasks.workunit.client.0.smithi077.stdout: create 159/37 file after the snapshot 2022-04-03T19:46:12.458 INFO:tasks.workunit.client.0.smithi077.stdout: create 159/38 file after the snapshot 2022-04-03T19:46:12.458 INFO:tasks.workunit.client.0.smithi077.stdout: create 159/39 file after the snapshot 2022-04-03T19:46:12.459 INFO:tasks.workunit.client.0.smithi077.stdout: create 160/.snap/snap-subdir-test 2022-04-03T19:46:12.485 INFO:tasks.workunit.client.0.smithi077.stdout: create 160/30 file after the snapshot 2022-04-03T19:46:12.488 INFO:tasks.workunit.client.0.smithi077.stdout: create 160/31 file after the snapshot 2022-04-03T19:46:12.488 INFO:tasks.workunit.client.0.smithi077.stdout: create 160/32 file after the snapshot 2022-04-03T19:46:12.489 INFO:tasks.workunit.client.0.smithi077.stdout: create 160/33 file after the snapshot 2022-04-03T19:46:12.489 INFO:tasks.workunit.client.0.smithi077.stdout: create 160/34 file after the snapshot 2022-04-03T19:46:12.489 INFO:tasks.workunit.client.0.smithi077.stdout: create 160/35 file after the snapshot 2022-04-03T19:46:12.489 INFO:tasks.workunit.client.0.smithi077.stdout: create 160/36 file after the snapshot 2022-04-03T19:46:12.490 INFO:tasks.workunit.client.0.smithi077.stdout: create 160/37 file after the snapshot 2022-04-03T19:46:12.490 INFO:tasks.workunit.client.0.smithi077.stdout: create 160/38 file after the snapshot 2022-04-03T19:46:12.490 INFO:tasks.workunit.client.0.smithi077.stdout: create 160/39 file after the snapshot 2022-04-03T19:46:12.490 INFO:tasks.workunit.client.0.smithi077.stdout: create 161/.snap/snap-subdir-test 2022-04-03T19:46:12.516 INFO:tasks.workunit.client.0.smithi077.stdout: create 161/30 file after the snapshot 2022-04-03T19:46:12.519 INFO:tasks.workunit.client.0.smithi077.stdout: create 161/31 file after the snapshot 2022-04-03T19:46:12.519 INFO:tasks.workunit.client.0.smithi077.stdout: create 161/32 file after the snapshot 2022-04-03T19:46:12.519 INFO:tasks.workunit.client.0.smithi077.stdout: create 161/33 file after the snapshot 2022-04-03T19:46:12.519 INFO:tasks.workunit.client.0.smithi077.stdout: create 161/34 file after the snapshot 2022-04-03T19:46:12.520 INFO:tasks.workunit.client.0.smithi077.stdout: create 161/35 file after the snapshot 2022-04-03T19:46:12.520 INFO:tasks.workunit.client.0.smithi077.stdout: create 161/36 file after the snapshot 2022-04-03T19:46:12.520 INFO:tasks.workunit.client.0.smithi077.stdout: create 161/37 file after the snapshot 2022-04-03T19:46:12.520 INFO:tasks.workunit.client.0.smithi077.stdout: create 161/38 file after the snapshot 2022-04-03T19:46:12.520 INFO:tasks.workunit.client.0.smithi077.stdout: create 161/39 file after the snapshot 2022-04-03T19:46:12.521 INFO:tasks.workunit.client.0.smithi077.stdout: create 162/.snap/snap-subdir-test 2022-04-03T19:46:12.547 INFO:tasks.workunit.client.0.smithi077.stdout: create 162/30 file after the snapshot 2022-04-03T19:46:12.549 INFO:tasks.workunit.client.0.smithi077.stdout: create 162/31 file after the snapshot 2022-04-03T19:46:12.550 INFO:tasks.workunit.client.0.smithi077.stdout: create 162/32 file after the snapshot 2022-04-03T19:46:12.550 INFO:tasks.workunit.client.0.smithi077.stdout: create 162/33 file after the snapshot 2022-04-03T19:46:12.550 INFO:tasks.workunit.client.0.smithi077.stdout: create 162/34 file after the snapshot 2022-04-03T19:46:12.550 INFO:tasks.workunit.client.0.smithi077.stdout: create 162/35 file after the snapshot 2022-04-03T19:46:12.551 INFO:tasks.workunit.client.0.smithi077.stdout: create 162/36 file after the snapshot 2022-04-03T19:46:12.551 INFO:tasks.workunit.client.0.smithi077.stdout: create 162/37 file after the snapshot 2022-04-03T19:46:12.551 INFO:tasks.workunit.client.0.smithi077.stdout: create 162/38 file after the snapshot 2022-04-03T19:46:12.551 INFO:tasks.workunit.client.0.smithi077.stdout: create 162/39 file after the snapshot 2022-04-03T19:46:12.552 INFO:tasks.workunit.client.0.smithi077.stdout: create 163/.snap/snap-subdir-test 2022-04-03T19:46:12.577 INFO:tasks.workunit.client.0.smithi077.stdout: create 163/30 file after the snapshot 2022-04-03T19:46:12.580 INFO:tasks.workunit.client.0.smithi077.stdout: create 163/31 file after the snapshot 2022-04-03T19:46:12.580 INFO:tasks.workunit.client.0.smithi077.stdout: create 163/32 file after the snapshot 2022-04-03T19:46:12.580 INFO:tasks.workunit.client.0.smithi077.stdout: create 163/33 file after the snapshot 2022-04-03T19:46:12.581 INFO:tasks.workunit.client.0.smithi077.stdout: create 163/34 file after the snapshot 2022-04-03T19:46:12.581 INFO:tasks.workunit.client.0.smithi077.stdout: create 163/35 file after the snapshot 2022-04-03T19:46:12.581 INFO:tasks.workunit.client.0.smithi077.stdout: create 163/36 file after the snapshot 2022-04-03T19:46:12.581 INFO:tasks.workunit.client.0.smithi077.stdout: create 163/37 file after the snapshot 2022-04-03T19:46:12.582 INFO:tasks.workunit.client.0.smithi077.stdout: create 163/38 file after the snapshot 2022-04-03T19:46:12.582 INFO:tasks.workunit.client.0.smithi077.stdout: create 163/39 file after the snapshot 2022-04-03T19:46:12.582 INFO:tasks.workunit.client.0.smithi077.stdout: create 164/.snap/snap-subdir-test 2022-04-03T19:46:12.608 INFO:tasks.workunit.client.0.smithi077.stdout: create 164/30 file after the snapshot 2022-04-03T19:46:12.610 INFO:tasks.workunit.client.0.smithi077.stdout: create 164/31 file after the snapshot 2022-04-03T19:46:12.610 INFO:tasks.workunit.client.0.smithi077.stdout: create 164/32 file after the snapshot 2022-04-03T19:46:12.611 INFO:tasks.workunit.client.0.smithi077.stdout: create 164/33 file after the snapshot 2022-04-03T19:46:12.611 INFO:tasks.workunit.client.0.smithi077.stdout: create 164/34 file after the snapshot 2022-04-03T19:46:12.611 INFO:tasks.workunit.client.0.smithi077.stdout: create 164/35 file after the snapshot 2022-04-03T19:46:12.611 INFO:tasks.workunit.client.0.smithi077.stdout: create 164/36 file after the snapshot 2022-04-03T19:46:12.612 INFO:tasks.workunit.client.0.smithi077.stdout: create 164/37 file after the snapshot 2022-04-03T19:46:12.612 INFO:tasks.workunit.client.0.smithi077.stdout: create 164/38 file after the snapshot 2022-04-03T19:46:12.613 INFO:tasks.workunit.client.0.smithi077.stdout: create 164/39 file after the snapshot 2022-04-03T19:46:12.613 INFO:tasks.workunit.client.0.smithi077.stdout: create 165/.snap/snap-subdir-test 2022-04-03T19:46:12.633 INFO:tasks.workunit.client.0.smithi077.stdout: create 165/30 file after the snapshot 2022-04-03T19:46:12.636 INFO:tasks.workunit.client.0.smithi077.stdout: create 165/31 file after the snapshot 2022-04-03T19:46:12.637 INFO:tasks.workunit.client.0.smithi077.stdout: create 165/32 file after the snapshot 2022-04-03T19:46:12.637 INFO:tasks.workunit.client.0.smithi077.stdout: create 165/33 file after the snapshot 2022-04-03T19:46:12.637 INFO:tasks.workunit.client.0.smithi077.stdout: create 165/34 file after the snapshot 2022-04-03T19:46:12.637 INFO:tasks.workunit.client.0.smithi077.stdout: create 165/35 file after the snapshot 2022-04-03T19:46:12.637 INFO:tasks.workunit.client.0.smithi077.stdout: create 165/36 file after the snapshot 2022-04-03T19:46:12.638 INFO:tasks.workunit.client.0.smithi077.stdout: create 165/37 file after the snapshot 2022-04-03T19:46:12.638 INFO:tasks.workunit.client.0.smithi077.stdout: create 165/38 file after the snapshot 2022-04-03T19:46:12.638 INFO:tasks.workunit.client.0.smithi077.stdout: create 165/39 file after the snapshot 2022-04-03T19:46:12.638 INFO:tasks.workunit.client.0.smithi077.stdout: create 166/.snap/snap-subdir-test 2022-04-03T19:46:13.412 INFO:tasks.workunit.client.0.smithi077.stdout: create 166/30 file after the snapshot 2022-04-03T19:46:13.414 INFO:tasks.workunit.client.0.smithi077.stdout: create 166/31 file after the snapshot 2022-04-03T19:46:13.414 INFO:tasks.workunit.client.0.smithi077.stdout: create 166/32 file after the snapshot 2022-04-03T19:46:13.414 INFO:tasks.workunit.client.0.smithi077.stdout: create 166/33 file after the snapshot 2022-04-03T19:46:13.415 INFO:tasks.workunit.client.0.smithi077.stdout: create 166/34 file after the snapshot 2022-04-03T19:46:13.415 INFO:tasks.workunit.client.0.smithi077.stdout: create 166/35 file after the snapshot 2022-04-03T19:46:13.415 INFO:tasks.workunit.client.0.smithi077.stdout: create 166/36 file after the snapshot 2022-04-03T19:46:13.415 INFO:tasks.workunit.client.0.smithi077.stdout: create 166/37 file after the snapshot 2022-04-03T19:46:13.415 INFO:tasks.workunit.client.0.smithi077.stdout: create 166/38 file after the snapshot 2022-04-03T19:46:13.416 INFO:tasks.workunit.client.0.smithi077.stdout: create 166/39 file after the snapshot 2022-04-03T19:46:13.416 INFO:tasks.workunit.client.0.smithi077.stdout: create 167/.snap/snap-subdir-test 2022-04-03T19:46:13.438 INFO:tasks.workunit.client.0.smithi077.stdout: create 167/30 file after the snapshot 2022-04-03T19:46:13.441 INFO:tasks.workunit.client.0.smithi077.stdout: create 167/31 file after the snapshot 2022-04-03T19:46:13.441 INFO:tasks.workunit.client.0.smithi077.stdout: create 167/32 file after the snapshot 2022-04-03T19:46:13.441 INFO:tasks.workunit.client.0.smithi077.stdout: create 167/33 file after the snapshot 2022-04-03T19:46:13.442 INFO:tasks.workunit.client.0.smithi077.stdout: create 167/34 file after the snapshot 2022-04-03T19:46:13.442 INFO:tasks.workunit.client.0.smithi077.stdout: create 167/35 file after the snapshot 2022-04-03T19:46:13.442 INFO:tasks.workunit.client.0.smithi077.stdout: create 167/36 file after the snapshot 2022-04-03T19:46:13.442 INFO:tasks.workunit.client.0.smithi077.stdout: create 167/37 file after the snapshot 2022-04-03T19:46:13.443 INFO:tasks.workunit.client.0.smithi077.stdout: create 167/38 file after the snapshot 2022-04-03T19:46:13.443 INFO:tasks.workunit.client.0.smithi077.stdout: create 167/39 file after the snapshot 2022-04-03T19:46:13.443 INFO:tasks.workunit.client.0.smithi077.stdout: create 168/.snap/snap-subdir-test 2022-04-03T19:46:13.465 INFO:tasks.workunit.client.0.smithi077.stdout: create 168/30 file after the snapshot 2022-04-03T19:46:13.467 INFO:tasks.workunit.client.0.smithi077.stdout: create 168/31 file after the snapshot 2022-04-03T19:46:13.467 INFO:tasks.workunit.client.0.smithi077.stdout: create 168/32 file after the snapshot 2022-04-03T19:46:13.468 INFO:tasks.workunit.client.0.smithi077.stdout: create 168/33 file after the snapshot 2022-04-03T19:46:13.468 INFO:tasks.workunit.client.0.smithi077.stdout: create 168/34 file after the snapshot 2022-04-03T19:46:13.468 INFO:tasks.workunit.client.0.smithi077.stdout: create 168/35 file after the snapshot 2022-04-03T19:46:13.468 INFO:tasks.workunit.client.0.smithi077.stdout: create 168/36 file after the snapshot 2022-04-03T19:46:13.468 INFO:tasks.workunit.client.0.smithi077.stdout: create 168/37 file after the snapshot 2022-04-03T19:46:13.469 INFO:tasks.workunit.client.0.smithi077.stdout: create 168/38 file after the snapshot 2022-04-03T19:46:13.469 INFO:tasks.workunit.client.0.smithi077.stdout: create 168/39 file after the snapshot 2022-04-03T19:46:13.469 INFO:tasks.workunit.client.0.smithi077.stdout: create 169/.snap/snap-subdir-test 2022-04-03T19:46:13.491 INFO:tasks.workunit.client.0.smithi077.stdout: create 169/30 file after the snapshot 2022-04-03T19:46:13.493 INFO:tasks.workunit.client.0.smithi077.stdout: create 169/31 file after the snapshot 2022-04-03T19:46:13.494 INFO:tasks.workunit.client.0.smithi077.stdout: create 169/32 file after the snapshot 2022-04-03T19:46:13.494 INFO:tasks.workunit.client.0.smithi077.stdout: create 169/33 file after the snapshot 2022-04-03T19:46:13.494 INFO:tasks.workunit.client.0.smithi077.stdout: create 169/34 file after the snapshot 2022-04-03T19:46:13.495 INFO:tasks.workunit.client.0.smithi077.stdout: create 169/35 file after the snapshot 2022-04-03T19:46:13.495 INFO:tasks.workunit.client.0.smithi077.stdout: create 169/36 file after the snapshot 2022-04-03T19:46:13.495 INFO:tasks.workunit.client.0.smithi077.stdout: create 169/37 file after the snapshot 2022-04-03T19:46:13.495 INFO:tasks.workunit.client.0.smithi077.stdout: create 169/38 file after the snapshot 2022-04-03T19:46:13.495 INFO:tasks.workunit.client.0.smithi077.stdout: create 169/39 file after the snapshot 2022-04-03T19:46:13.496 INFO:tasks.workunit.client.0.smithi077.stdout: create 170/.snap/snap-subdir-test 2022-04-03T19:46:13.518 INFO:tasks.workunit.client.0.smithi077.stdout: create 170/30 file after the snapshot 2022-04-03T19:46:13.520 INFO:tasks.workunit.client.0.smithi077.stdout: create 170/31 file after the snapshot 2022-04-03T19:46:13.520 INFO:tasks.workunit.client.0.smithi077.stdout: create 170/32 file after the snapshot 2022-04-03T19:46:13.520 INFO:tasks.workunit.client.0.smithi077.stdout: create 170/33 file after the snapshot 2022-04-03T19:46:13.521 INFO:tasks.workunit.client.0.smithi077.stdout: create 170/34 file after the snapshot 2022-04-03T19:46:13.521 INFO:tasks.workunit.client.0.smithi077.stdout: create 170/35 file after the snapshot 2022-04-03T19:46:13.521 INFO:tasks.workunit.client.0.smithi077.stdout: create 170/36 file after the snapshot 2022-04-03T19:46:13.521 INFO:tasks.workunit.client.0.smithi077.stdout: create 170/37 file after the snapshot 2022-04-03T19:46:13.521 INFO:tasks.workunit.client.0.smithi077.stdout: create 170/38 file after the snapshot 2022-04-03T19:46:13.522 INFO:tasks.workunit.client.0.smithi077.stdout: create 170/39 file after the snapshot 2022-04-03T19:46:13.522 INFO:tasks.workunit.client.0.smithi077.stdout: create 171/.snap/snap-subdir-test 2022-04-03T19:46:13.544 INFO:tasks.workunit.client.0.smithi077.stdout: create 171/30 file after the snapshot 2022-04-03T19:46:13.546 INFO:tasks.workunit.client.0.smithi077.stdout: create 171/31 file after the snapshot 2022-04-03T19:46:13.546 INFO:tasks.workunit.client.0.smithi077.stdout: create 171/32 file after the snapshot 2022-04-03T19:46:13.546 INFO:tasks.workunit.client.0.smithi077.stdout: create 171/33 file after the snapshot 2022-04-03T19:46:13.547 INFO:tasks.workunit.client.0.smithi077.stdout: create 171/34 file after the snapshot 2022-04-03T19:46:13.547 INFO:tasks.workunit.client.0.smithi077.stdout: create 171/35 file after the snapshot 2022-04-03T19:46:13.547 INFO:tasks.workunit.client.0.smithi077.stdout: create 171/36 file after the snapshot 2022-04-03T19:46:13.547 INFO:tasks.workunit.client.0.smithi077.stdout: create 171/37 file after the snapshot 2022-04-03T19:46:13.547 INFO:tasks.workunit.client.0.smithi077.stdout: create 171/38 file after the snapshot 2022-04-03T19:46:13.548 INFO:tasks.workunit.client.0.smithi077.stdout: create 171/39 file after the snapshot 2022-04-03T19:46:13.548 INFO:tasks.workunit.client.0.smithi077.stdout: create 172/.snap/snap-subdir-test 2022-04-03T19:46:13.571 INFO:tasks.workunit.client.0.smithi077.stdout: create 172/30 file after the snapshot 2022-04-03T19:46:13.573 INFO:tasks.workunit.client.0.smithi077.stdout: create 172/31 file after the snapshot 2022-04-03T19:46:13.574 INFO:tasks.workunit.client.0.smithi077.stdout: create 172/32 file after the snapshot 2022-04-03T19:46:13.574 INFO:tasks.workunit.client.0.smithi077.stdout: create 172/33 file after the snapshot 2022-04-03T19:46:13.574 INFO:tasks.workunit.client.0.smithi077.stdout: create 172/34 file after the snapshot 2022-04-03T19:46:13.574 INFO:tasks.workunit.client.0.smithi077.stdout: create 172/35 file after the snapshot 2022-04-03T19:46:13.575 INFO:tasks.workunit.client.0.smithi077.stdout: create 172/36 file after the snapshot 2022-04-03T19:46:13.575 INFO:tasks.workunit.client.0.smithi077.stdout: create 172/37 file after the snapshot 2022-04-03T19:46:13.575 INFO:tasks.workunit.client.0.smithi077.stdout: create 172/38 file after the snapshot 2022-04-03T19:46:13.575 INFO:tasks.workunit.client.0.smithi077.stdout: create 172/39 file after the snapshot 2022-04-03T19:46:13.576 INFO:tasks.workunit.client.0.smithi077.stdout: create 173/.snap/snap-subdir-test 2022-04-03T19:46:14.532 INFO:tasks.workunit.client.0.smithi077.stdout: create 173/30 file after the snapshot 2022-04-03T19:46:14.534 INFO:tasks.workunit.client.0.smithi077.stdout: create 173/31 file after the snapshot 2022-04-03T19:46:14.534 INFO:tasks.workunit.client.0.smithi077.stdout: create 173/32 file after the snapshot 2022-04-03T19:46:14.534 INFO:tasks.workunit.client.0.smithi077.stdout: create 173/33 file after the snapshot 2022-04-03T19:46:14.535 INFO:tasks.workunit.client.0.smithi077.stdout: create 173/34 file after the snapshot 2022-04-03T19:46:14.535 INFO:tasks.workunit.client.0.smithi077.stdout: create 173/35 file after the snapshot 2022-04-03T19:46:14.535 INFO:tasks.workunit.client.0.smithi077.stdout: create 173/36 file after the snapshot 2022-04-03T19:46:14.535 INFO:tasks.workunit.client.0.smithi077.stdout: create 173/37 file after the snapshot 2022-04-03T19:46:14.536 INFO:tasks.workunit.client.0.smithi077.stdout: create 173/38 file after the snapshot 2022-04-03T19:46:14.536 INFO:tasks.workunit.client.0.smithi077.stdout: create 173/39 file after the snapshot 2022-04-03T19:46:14.536 INFO:tasks.workunit.client.0.smithi077.stdout: create 174/.snap/snap-subdir-test 2022-04-03T19:46:14.569 INFO:tasks.workunit.client.0.smithi077.stdout: create 174/30 file after the snapshot 2022-04-03T19:46:14.571 INFO:tasks.workunit.client.0.smithi077.stdout: create 174/31 file after the snapshot 2022-04-03T19:46:14.571 INFO:tasks.workunit.client.0.smithi077.stdout: create 174/32 file after the snapshot 2022-04-03T19:46:14.571 INFO:tasks.workunit.client.0.smithi077.stdout: create 174/33 file after the snapshot 2022-04-03T19:46:14.572 INFO:tasks.workunit.client.0.smithi077.stdout: create 174/34 file after the snapshot 2022-04-03T19:46:14.572 INFO:tasks.workunit.client.0.smithi077.stdout: create 174/35 file after the snapshot 2022-04-03T19:46:14.572 INFO:tasks.workunit.client.0.smithi077.stdout: create 174/36 file after the snapshot 2022-04-03T19:46:14.572 INFO:tasks.workunit.client.0.smithi077.stdout: create 174/37 file after the snapshot 2022-04-03T19:46:14.572 INFO:tasks.workunit.client.0.smithi077.stdout: create 174/38 file after the snapshot 2022-04-03T19:46:14.573 INFO:tasks.workunit.client.0.smithi077.stdout: create 174/39 file after the snapshot 2022-04-03T19:46:14.573 INFO:tasks.workunit.client.0.smithi077.stdout: create 175/.snap/snap-subdir-test 2022-04-03T19:46:14.604 INFO:tasks.workunit.client.0.smithi077.stdout: create 175/30 file after the snapshot 2022-04-03T19:46:14.606 INFO:tasks.workunit.client.0.smithi077.stdout: create 175/31 file after the snapshot 2022-04-03T19:46:14.606 INFO:tasks.workunit.client.0.smithi077.stdout: create 175/32 file after the snapshot 2022-04-03T19:46:14.606 INFO:tasks.workunit.client.0.smithi077.stdout: create 175/33 file after the snapshot 2022-04-03T19:46:14.607 INFO:tasks.workunit.client.0.smithi077.stdout: create 175/34 file after the snapshot 2022-04-03T19:46:14.607 INFO:tasks.workunit.client.0.smithi077.stdout: create 175/35 file after the snapshot 2022-04-03T19:46:14.607 INFO:tasks.workunit.client.0.smithi077.stdout: create 175/36 file after the snapshot 2022-04-03T19:46:14.607 INFO:tasks.workunit.client.0.smithi077.stdout: create 175/37 file after the snapshot 2022-04-03T19:46:14.607 INFO:tasks.workunit.client.0.smithi077.stdout: create 175/38 file after the snapshot 2022-04-03T19:46:14.608 INFO:tasks.workunit.client.0.smithi077.stdout: create 175/39 file after the snapshot 2022-04-03T19:46:14.608 INFO:tasks.workunit.client.0.smithi077.stdout: create 176/.snap/snap-subdir-test 2022-04-03T19:46:14.640 INFO:tasks.workunit.client.0.smithi077.stdout: create 176/30 file after the snapshot 2022-04-03T19:46:14.642 INFO:tasks.workunit.client.0.smithi077.stdout: create 176/31 file after the snapshot 2022-04-03T19:46:14.642 INFO:tasks.workunit.client.0.smithi077.stdout: create 176/32 file after the snapshot 2022-04-03T19:46:14.643 INFO:tasks.workunit.client.0.smithi077.stdout: create 176/33 file after the snapshot 2022-04-03T19:46:14.643 INFO:tasks.workunit.client.0.smithi077.stdout: create 176/34 file after the snapshot 2022-04-03T19:46:14.643 INFO:tasks.workunit.client.0.smithi077.stdout: create 176/35 file after the snapshot 2022-04-03T19:46:14.643 INFO:tasks.workunit.client.0.smithi077.stdout: create 176/36 file after the snapshot 2022-04-03T19:46:14.643 INFO:tasks.workunit.client.0.smithi077.stdout: create 176/37 file after the snapshot 2022-04-03T19:46:14.644 INFO:tasks.workunit.client.0.smithi077.stdout: create 176/38 file after the snapshot 2022-04-03T19:46:14.644 INFO:tasks.workunit.client.0.smithi077.stdout: create 176/39 file after the snapshot 2022-04-03T19:46:14.644 INFO:tasks.workunit.client.0.smithi077.stdout: create 177/.snap/snap-subdir-test 2022-04-03T19:46:14.674 INFO:tasks.workunit.client.0.smithi077.stdout: create 177/30 file after the snapshot 2022-04-03T19:46:14.679 INFO:tasks.workunit.client.0.smithi077.stdout: create 177/31 file after the snapshot 2022-04-03T19:46:14.679 INFO:tasks.workunit.client.0.smithi077.stdout: create 177/32 file after the snapshot 2022-04-03T19:46:14.679 INFO:tasks.workunit.client.0.smithi077.stdout: create 177/33 file after the snapshot 2022-04-03T19:46:14.679 INFO:tasks.workunit.client.0.smithi077.stdout: create 177/34 file after the snapshot 2022-04-03T19:46:14.680 INFO:tasks.workunit.client.0.smithi077.stdout: create 177/35 file after the snapshot 2022-04-03T19:46:14.680 INFO:tasks.workunit.client.0.smithi077.stdout: create 177/36 file after the snapshot 2022-04-03T19:46:14.680 INFO:tasks.workunit.client.0.smithi077.stdout: create 177/37 file after the snapshot 2022-04-03T19:46:14.680 INFO:tasks.workunit.client.0.smithi077.stdout: create 177/38 file after the snapshot 2022-04-03T19:46:14.681 INFO:tasks.workunit.client.0.smithi077.stdout: create 177/39 file after the snapshot 2022-04-03T19:46:14.681 INFO:tasks.workunit.client.0.smithi077.stdout: create 178/.snap/snap-subdir-test 2022-04-03T19:46:14.710 INFO:tasks.workunit.client.0.smithi077.stdout: create 178/30 file after the snapshot 2022-04-03T19:46:14.712 INFO:tasks.workunit.client.0.smithi077.stdout: create 178/31 file after the snapshot 2022-04-03T19:46:14.712 INFO:tasks.workunit.client.0.smithi077.stdout: create 178/32 file after the snapshot 2022-04-03T19:46:14.712 INFO:tasks.workunit.client.0.smithi077.stdout: create 178/33 file after the snapshot 2022-04-03T19:46:14.713 INFO:tasks.workunit.client.0.smithi077.stdout: create 178/34 file after the snapshot 2022-04-03T19:46:14.713 INFO:tasks.workunit.client.0.smithi077.stdout: create 178/35 file after the snapshot 2022-04-03T19:46:14.713 INFO:tasks.workunit.client.0.smithi077.stdout: create 178/36 file after the snapshot 2022-04-03T19:46:14.713 INFO:tasks.workunit.client.0.smithi077.stdout: create 178/37 file after the snapshot 2022-04-03T19:46:14.713 INFO:tasks.workunit.client.0.smithi077.stdout: create 178/38 file after the snapshot 2022-04-03T19:46:14.714 INFO:tasks.workunit.client.0.smithi077.stdout: create 178/39 file after the snapshot 2022-04-03T19:46:14.714 INFO:tasks.workunit.client.0.smithi077.stdout: create 179/.snap/snap-subdir-test 2022-04-03T19:46:15.048 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:46:15.053 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:46:15.464 INFO:tasks.workunit.client.0.smithi077.stdout: create 179/30 file after the snapshot 2022-04-03T19:46:15.466 INFO:tasks.workunit.client.0.smithi077.stdout: create 179/31 file after the snapshot 2022-04-03T19:46:15.466 INFO:tasks.workunit.client.0.smithi077.stdout: create 179/32 file after the snapshot 2022-04-03T19:46:15.466 INFO:tasks.workunit.client.0.smithi077.stdout: create 179/33 file after the snapshot 2022-04-03T19:46:15.466 INFO:tasks.workunit.client.0.smithi077.stdout: create 179/34 file after the snapshot 2022-04-03T19:46:15.467 INFO:tasks.workunit.client.0.smithi077.stdout: create 179/35 file after the snapshot 2022-04-03T19:46:15.467 INFO:tasks.workunit.client.0.smithi077.stdout: create 179/36 file after the snapshot 2022-04-03T19:46:15.467 INFO:tasks.workunit.client.0.smithi077.stdout: create 179/37 file after the snapshot 2022-04-03T19:46:15.467 INFO:tasks.workunit.client.0.smithi077.stdout: create 179/38 file after the snapshot 2022-04-03T19:46:15.467 INFO:tasks.workunit.client.0.smithi077.stdout: create 179/39 file after the snapshot 2022-04-03T19:46:15.468 INFO:tasks.workunit.client.0.smithi077.stdout: create 180/.snap/snap-subdir-test 2022-04-03T19:46:15.496 INFO:tasks.workunit.client.0.smithi077.stdout: create 180/30 file after the snapshot 2022-04-03T19:46:15.498 INFO:tasks.workunit.client.0.smithi077.stdout: create 180/31 file after the snapshot 2022-04-03T19:46:15.498 INFO:tasks.workunit.client.0.smithi077.stdout: create 180/32 file after the snapshot 2022-04-03T19:46:15.498 INFO:tasks.workunit.client.0.smithi077.stdout: create 180/33 file after the snapshot 2022-04-03T19:46:15.499 INFO:tasks.workunit.client.0.smithi077.stdout: create 180/34 file after the snapshot 2022-04-03T19:46:15.499 INFO:tasks.workunit.client.0.smithi077.stdout: create 180/35 file after the snapshot 2022-04-03T19:46:15.499 INFO:tasks.workunit.client.0.smithi077.stdout: create 180/36 file after the snapshot 2022-04-03T19:46:15.499 INFO:tasks.workunit.client.0.smithi077.stdout: create 180/37 file after the snapshot 2022-04-03T19:46:15.500 INFO:tasks.workunit.client.0.smithi077.stdout: create 180/38 file after the snapshot 2022-04-03T19:46:15.500 INFO:tasks.workunit.client.0.smithi077.stdout: create 180/39 file after the snapshot 2022-04-03T19:46:15.500 INFO:tasks.workunit.client.0.smithi077.stdout: create 181/.snap/snap-subdir-test 2022-04-03T19:46:15.802 INFO:tasks.workunit.client.0.smithi077.stdout: create 181/30 file after the snapshot 2022-04-03T19:46:15.804 INFO:tasks.workunit.client.0.smithi077.stdout: create 181/31 file after the snapshot 2022-04-03T19:46:15.804 INFO:tasks.workunit.client.0.smithi077.stdout: create 181/32 file after the snapshot 2022-04-03T19:46:15.805 INFO:tasks.workunit.client.0.smithi077.stdout: create 181/33 file after the snapshot 2022-04-03T19:46:15.805 INFO:tasks.workunit.client.0.smithi077.stdout: create 181/34 file after the snapshot 2022-04-03T19:46:15.805 INFO:tasks.workunit.client.0.smithi077.stdout: create 181/35 file after the snapshot 2022-04-03T19:46:15.805 INFO:tasks.workunit.client.0.smithi077.stdout: create 181/36 file after the snapshot 2022-04-03T19:46:15.805 INFO:tasks.workunit.client.0.smithi077.stdout: create 181/37 file after the snapshot 2022-04-03T19:46:15.806 INFO:tasks.workunit.client.0.smithi077.stdout: create 181/38 file after the snapshot 2022-04-03T19:46:15.806 INFO:tasks.workunit.client.0.smithi077.stdout: create 181/39 file after the snapshot 2022-04-03T19:46:15.806 INFO:tasks.workunit.client.0.smithi077.stdout: create 182/.snap/snap-subdir-test 2022-04-03T19:46:15.836 INFO:tasks.workunit.client.0.smithi077.stdout: create 182/30 file after the snapshot 2022-04-03T19:46:15.838 INFO:tasks.workunit.client.0.smithi077.stdout: create 182/31 file after the snapshot 2022-04-03T19:46:15.838 INFO:tasks.workunit.client.0.smithi077.stdout: create 182/32 file after the snapshot 2022-04-03T19:46:15.838 INFO:tasks.workunit.client.0.smithi077.stdout: create 182/33 file after the snapshot 2022-04-03T19:46:15.839 INFO:tasks.workunit.client.0.smithi077.stdout: create 182/34 file after the snapshot 2022-04-03T19:46:15.839 INFO:tasks.workunit.client.0.smithi077.stdout: create 182/35 file after the snapshot 2022-04-03T19:46:15.839 INFO:tasks.workunit.client.0.smithi077.stdout: create 182/36 file after the snapshot 2022-04-03T19:46:15.839 INFO:tasks.workunit.client.0.smithi077.stdout: create 182/37 file after the snapshot 2022-04-03T19:46:15.840 INFO:tasks.workunit.client.0.smithi077.stdout: create 182/38 file after the snapshot 2022-04-03T19:46:15.840 INFO:tasks.workunit.client.0.smithi077.stdout: create 182/39 file after the snapshot 2022-04-03T19:46:15.840 INFO:tasks.workunit.client.0.smithi077.stdout: create 183/.snap/snap-subdir-test 2022-04-03T19:46:15.864 INFO:tasks.workunit.client.0.smithi077.stdout: create 183/30 file after the snapshot 2022-04-03T19:46:15.865 INFO:tasks.workunit.client.0.smithi077.stdout: create 183/31 file after the snapshot 2022-04-03T19:46:15.866 INFO:tasks.workunit.client.0.smithi077.stdout: create 183/32 file after the snapshot 2022-04-03T19:46:15.866 INFO:tasks.workunit.client.0.smithi077.stdout: create 183/33 file after the snapshot 2022-04-03T19:46:15.866 INFO:tasks.workunit.client.0.smithi077.stdout: create 183/34 file after the snapshot 2022-04-03T19:46:15.866 INFO:tasks.workunit.client.0.smithi077.stdout: create 183/35 file after the snapshot 2022-04-03T19:46:15.867 INFO:tasks.workunit.client.0.smithi077.stdout: create 183/36 file after the snapshot 2022-04-03T19:46:15.867 INFO:tasks.workunit.client.0.smithi077.stdout: create 183/37 file after the snapshot 2022-04-03T19:46:15.867 INFO:tasks.workunit.client.0.smithi077.stdout: create 183/38 file after the snapshot 2022-04-03T19:46:15.867 INFO:tasks.workunit.client.0.smithi077.stdout: create 183/39 file after the snapshot 2022-04-03T19:46:15.868 INFO:tasks.workunit.client.0.smithi077.stdout: create 184/.snap/snap-subdir-test 2022-04-03T19:46:15.897 INFO:tasks.workunit.client.0.smithi077.stdout: create 184/30 file after the snapshot 2022-04-03T19:46:15.899 INFO:tasks.workunit.client.0.smithi077.stdout: create 184/31 file after the snapshot 2022-04-03T19:46:15.900 INFO:tasks.workunit.client.0.smithi077.stdout: create 184/32 file after the snapshot 2022-04-03T19:46:15.900 INFO:tasks.workunit.client.0.smithi077.stdout: create 184/33 file after the snapshot 2022-04-03T19:46:15.900 INFO:tasks.workunit.client.0.smithi077.stdout: create 184/34 file after the snapshot 2022-04-03T19:46:15.900 INFO:tasks.workunit.client.0.smithi077.stdout: create 184/35 file after the snapshot 2022-04-03T19:46:15.900 INFO:tasks.workunit.client.0.smithi077.stdout: create 184/36 file after the snapshot 2022-04-03T19:46:15.901 INFO:tasks.workunit.client.0.smithi077.stdout: create 184/37 file after the snapshot 2022-04-03T19:46:15.901 INFO:tasks.workunit.client.0.smithi077.stdout: create 184/38 file after the snapshot 2022-04-03T19:46:15.901 INFO:tasks.workunit.client.0.smithi077.stdout: create 184/39 file after the snapshot 2022-04-03T19:46:15.901 INFO:tasks.workunit.client.0.smithi077.stdout: create 185/.snap/snap-subdir-test 2022-04-03T19:46:15.931 INFO:tasks.workunit.client.0.smithi077.stdout: create 185/30 file after the snapshot 2022-04-03T19:46:15.933 INFO:tasks.workunit.client.0.smithi077.stdout: create 185/31 file after the snapshot 2022-04-03T19:46:15.933 INFO:tasks.workunit.client.0.smithi077.stdout: create 185/32 file after the snapshot 2022-04-03T19:46:15.933 INFO:tasks.workunit.client.0.smithi077.stdout: create 185/33 file after the snapshot 2022-04-03T19:46:15.933 INFO:tasks.workunit.client.0.smithi077.stdout: create 185/34 file after the snapshot 2022-04-03T19:46:15.934 INFO:tasks.workunit.client.0.smithi077.stdout: create 185/35 file after the snapshot 2022-04-03T19:46:15.934 INFO:tasks.workunit.client.0.smithi077.stdout: create 185/36 file after the snapshot 2022-04-03T19:46:15.934 INFO:tasks.workunit.client.0.smithi077.stdout: create 185/37 file after the snapshot 2022-04-03T19:46:15.934 INFO:tasks.workunit.client.0.smithi077.stdout: create 185/38 file after the snapshot 2022-04-03T19:46:15.935 INFO:tasks.workunit.client.0.smithi077.stdout: create 185/39 file after the snapshot 2022-04-03T19:46:15.935 INFO:tasks.workunit.client.0.smithi077.stdout: create 186/.snap/snap-subdir-test 2022-04-03T19:46:15.964 INFO:tasks.workunit.client.0.smithi077.stdout: create 186/30 file after the snapshot 2022-04-03T19:46:15.966 INFO:tasks.workunit.client.0.smithi077.stdout: create 186/31 file after the snapshot 2022-04-03T19:46:15.967 INFO:tasks.workunit.client.0.smithi077.stdout: create 186/32 file after the snapshot 2022-04-03T19:46:15.967 INFO:tasks.workunit.client.0.smithi077.stdout: create 186/33 file after the snapshot 2022-04-03T19:46:15.967 INFO:tasks.workunit.client.0.smithi077.stdout: create 186/34 file after the snapshot 2022-04-03T19:46:15.967 INFO:tasks.workunit.client.0.smithi077.stdout: create 186/35 file after the snapshot 2022-04-03T19:46:15.968 INFO:tasks.workunit.client.0.smithi077.stdout: create 186/36 file after the snapshot 2022-04-03T19:46:15.968 INFO:tasks.workunit.client.0.smithi077.stdout: create 186/37 file after the snapshot 2022-04-03T19:46:15.968 INFO:tasks.workunit.client.0.smithi077.stdout: create 186/38 file after the snapshot 2022-04-03T19:46:15.968 INFO:tasks.workunit.client.0.smithi077.stdout: create 186/39 file after the snapshot 2022-04-03T19:46:15.968 INFO:tasks.workunit.client.0.smithi077.stdout: create 187/.snap/snap-subdir-test 2022-04-03T19:46:15.990 INFO:tasks.workunit.client.0.smithi077.stdout: create 187/30 file after the snapshot 2022-04-03T19:46:15.992 INFO:tasks.workunit.client.0.smithi077.stdout: create 187/31 file after the snapshot 2022-04-03T19:46:15.992 INFO:tasks.workunit.client.0.smithi077.stdout: create 187/32 file after the snapshot 2022-04-03T19:46:15.992 INFO:tasks.workunit.client.0.smithi077.stdout: create 187/33 file after the snapshot 2022-04-03T19:46:15.992 INFO:tasks.workunit.client.0.smithi077.stdout: create 187/34 file after the snapshot 2022-04-03T19:46:15.993 INFO:tasks.workunit.client.0.smithi077.stdout: create 187/35 file after the snapshot 2022-04-03T19:46:15.993 INFO:tasks.workunit.client.0.smithi077.stdout: create 187/36 file after the snapshot 2022-04-03T19:46:15.993 INFO:tasks.workunit.client.0.smithi077.stdout: create 187/37 file after the snapshot 2022-04-03T19:46:15.993 INFO:tasks.workunit.client.0.smithi077.stdout: create 187/38 file after the snapshot 2022-04-03T19:46:15.994 INFO:tasks.workunit.client.0.smithi077.stdout: create 187/39 file after the snapshot 2022-04-03T19:46:15.994 INFO:tasks.workunit.client.0.smithi077.stdout: create 188/.snap/snap-subdir-test 2022-04-03T19:46:16.020 INFO:tasks.workunit.client.0.smithi077.stdout: create 188/30 file after the snapshot 2022-04-03T19:46:16.021 INFO:tasks.workunit.client.0.smithi077.stdout: create 188/31 file after the snapshot 2022-04-03T19:46:16.022 INFO:tasks.workunit.client.0.smithi077.stdout: create 188/32 file after the snapshot 2022-04-03T19:46:16.023 INFO:tasks.workunit.client.0.smithi077.stdout: create 188/33 file after the snapshot 2022-04-03T19:46:16.023 INFO:tasks.workunit.client.0.smithi077.stdout: create 188/34 file after the snapshot 2022-04-03T19:46:16.023 INFO:tasks.workunit.client.0.smithi077.stdout: create 188/35 file after the snapshot 2022-04-03T19:46:16.024 INFO:tasks.workunit.client.0.smithi077.stdout: create 188/36 file after the snapshot 2022-04-03T19:46:16.024 INFO:tasks.workunit.client.0.smithi077.stdout: create 188/37 file after the snapshot 2022-04-03T19:46:16.024 INFO:tasks.workunit.client.0.smithi077.stdout: create 188/38 file after the snapshot 2022-04-03T19:46:16.024 INFO:tasks.workunit.client.0.smithi077.stdout: create 188/39 file after the snapshot 2022-04-03T19:46:16.025 INFO:tasks.workunit.client.0.smithi077.stdout: create 189/.snap/snap-subdir-test 2022-04-03T19:46:16.791 INFO:tasks.workunit.client.0.smithi077.stdout: create 189/30 file after the snapshot 2022-04-03T19:46:16.793 INFO:tasks.workunit.client.0.smithi077.stdout: create 189/31 file after the snapshot 2022-04-03T19:46:16.794 INFO:tasks.workunit.client.0.smithi077.stdout: create 189/32 file after the snapshot 2022-04-03T19:46:16.794 INFO:tasks.workunit.client.0.smithi077.stdout: create 189/33 file after the snapshot 2022-04-03T19:46:16.794 INFO:tasks.workunit.client.0.smithi077.stdout: create 189/34 file after the snapshot 2022-04-03T19:46:16.794 INFO:tasks.workunit.client.0.smithi077.stdout: create 189/35 file after the snapshot 2022-04-03T19:46:16.795 INFO:tasks.workunit.client.0.smithi077.stdout: create 189/36 file after the snapshot 2022-04-03T19:46:16.795 INFO:tasks.workunit.client.0.smithi077.stdout: create 189/37 file after the snapshot 2022-04-03T19:46:16.795 INFO:tasks.workunit.client.0.smithi077.stdout: create 189/38 file after the snapshot 2022-04-03T19:46:16.795 INFO:tasks.workunit.client.0.smithi077.stdout: create 189/39 file after the snapshot 2022-04-03T19:46:16.795 INFO:tasks.workunit.client.0.smithi077.stdout: create 190/.snap/snap-subdir-test 2022-04-03T19:46:16.821 INFO:tasks.workunit.client.0.smithi077.stdout: create 190/30 file after the snapshot 2022-04-03T19:46:16.823 INFO:tasks.workunit.client.0.smithi077.stdout: create 190/31 file after the snapshot 2022-04-03T19:46:16.823 INFO:tasks.workunit.client.0.smithi077.stdout: create 190/32 file after the snapshot 2022-04-03T19:46:16.824 INFO:tasks.workunit.client.0.smithi077.stdout: create 190/33 file after the snapshot 2022-04-03T19:46:16.824 INFO:tasks.workunit.client.0.smithi077.stdout: create 190/34 file after the snapshot 2022-04-03T19:46:16.824 INFO:tasks.workunit.client.0.smithi077.stdout: create 190/35 file after the snapshot 2022-04-03T19:46:16.824 INFO:tasks.workunit.client.0.smithi077.stdout: create 190/36 file after the snapshot 2022-04-03T19:46:16.824 INFO:tasks.workunit.client.0.smithi077.stdout: create 190/37 file after the snapshot 2022-04-03T19:46:16.825 INFO:tasks.workunit.client.0.smithi077.stdout: create 190/38 file after the snapshot 2022-04-03T19:46:16.825 INFO:tasks.workunit.client.0.smithi077.stdout: create 190/39 file after the snapshot 2022-04-03T19:46:16.825 INFO:tasks.workunit.client.0.smithi077.stdout: create 191/.snap/snap-subdir-test 2022-04-03T19:46:16.850 INFO:tasks.workunit.client.0.smithi077.stdout: create 191/30 file after the snapshot 2022-04-03T19:46:16.852 INFO:tasks.workunit.client.0.smithi077.stdout: create 191/31 file after the snapshot 2022-04-03T19:46:16.852 INFO:tasks.workunit.client.0.smithi077.stdout: create 191/32 file after the snapshot 2022-04-03T19:46:16.853 INFO:tasks.workunit.client.0.smithi077.stdout: create 191/33 file after the snapshot 2022-04-03T19:46:16.853 INFO:tasks.workunit.client.0.smithi077.stdout: create 191/34 file after the snapshot 2022-04-03T19:46:16.853 INFO:tasks.workunit.client.0.smithi077.stdout: create 191/35 file after the snapshot 2022-04-03T19:46:16.853 INFO:tasks.workunit.client.0.smithi077.stdout: create 191/36 file after the snapshot 2022-04-03T19:46:16.854 INFO:tasks.workunit.client.0.smithi077.stdout: create 191/37 file after the snapshot 2022-04-03T19:46:16.854 INFO:tasks.workunit.client.0.smithi077.stdout: create 191/38 file after the snapshot 2022-04-03T19:46:16.854 INFO:tasks.workunit.client.0.smithi077.stdout: create 191/39 file after the snapshot 2022-04-03T19:46:16.854 INFO:tasks.workunit.client.0.smithi077.stdout: create 192/.snap/snap-subdir-test 2022-04-03T19:46:16.879 INFO:tasks.workunit.client.0.smithi077.stdout: create 192/30 file after the snapshot 2022-04-03T19:46:16.881 INFO:tasks.workunit.client.0.smithi077.stdout: create 192/31 file after the snapshot 2022-04-03T19:46:16.881 INFO:tasks.workunit.client.0.smithi077.stdout: create 192/32 file after the snapshot 2022-04-03T19:46:16.881 INFO:tasks.workunit.client.0.smithi077.stdout: create 192/33 file after the snapshot 2022-04-03T19:46:16.881 INFO:tasks.workunit.client.0.smithi077.stdout: create 192/34 file after the snapshot 2022-04-03T19:46:16.882 INFO:tasks.workunit.client.0.smithi077.stdout: create 192/35 file after the snapshot 2022-04-03T19:46:16.882 INFO:tasks.workunit.client.0.smithi077.stdout: create 192/36 file after the snapshot 2022-04-03T19:46:16.882 INFO:tasks.workunit.client.0.smithi077.stdout: create 192/37 file after the snapshot 2022-04-03T19:46:16.882 INFO:tasks.workunit.client.0.smithi077.stdout: create 192/38 file after the snapshot 2022-04-03T19:46:16.883 INFO:tasks.workunit.client.0.smithi077.stdout: create 192/39 file after the snapshot 2022-04-03T19:46:16.883 INFO:tasks.workunit.client.0.smithi077.stdout: create 193/.snap/snap-subdir-test 2022-04-03T19:46:16.907 INFO:tasks.workunit.client.0.smithi077.stdout: create 193/30 file after the snapshot 2022-04-03T19:46:16.909 INFO:tasks.workunit.client.0.smithi077.stdout: create 193/31 file after the snapshot 2022-04-03T19:46:16.909 INFO:tasks.workunit.client.0.smithi077.stdout: create 193/32 file after the snapshot 2022-04-03T19:46:16.910 INFO:tasks.workunit.client.0.smithi077.stdout: create 193/33 file after the snapshot 2022-04-03T19:46:16.910 INFO:tasks.workunit.client.0.smithi077.stdout: create 193/34 file after the snapshot 2022-04-03T19:46:16.910 INFO:tasks.workunit.client.0.smithi077.stdout: create 193/35 file after the snapshot 2022-04-03T19:46:16.910 INFO:tasks.workunit.client.0.smithi077.stdout: create 193/36 file after the snapshot 2022-04-03T19:46:16.911 INFO:tasks.workunit.client.0.smithi077.stdout: create 193/37 file after the snapshot 2022-04-03T19:46:16.911 INFO:tasks.workunit.client.0.smithi077.stdout: create 193/38 file after the snapshot 2022-04-03T19:46:16.911 INFO:tasks.workunit.client.0.smithi077.stdout: create 193/39 file after the snapshot 2022-04-03T19:46:16.911 INFO:tasks.workunit.client.0.smithi077.stdout: create 194/.snap/snap-subdir-test 2022-04-03T19:46:16.935 INFO:tasks.workunit.client.0.smithi077.stdout: create 194/30 file after the snapshot 2022-04-03T19:46:16.937 INFO:tasks.workunit.client.0.smithi077.stdout: create 194/31 file after the snapshot 2022-04-03T19:46:16.937 INFO:tasks.workunit.client.0.smithi077.stdout: create 194/32 file after the snapshot 2022-04-03T19:46:16.938 INFO:tasks.workunit.client.0.smithi077.stdout: create 194/33 file after the snapshot 2022-04-03T19:46:16.938 INFO:tasks.workunit.client.0.smithi077.stdout: create 194/34 file after the snapshot 2022-04-03T19:46:16.938 INFO:tasks.workunit.client.0.smithi077.stdout: create 194/35 file after the snapshot 2022-04-03T19:46:16.938 INFO:tasks.workunit.client.0.smithi077.stdout: create 194/36 file after the snapshot 2022-04-03T19:46:16.939 INFO:tasks.workunit.client.0.smithi077.stdout: create 194/37 file after the snapshot 2022-04-03T19:46:16.939 INFO:tasks.workunit.client.0.smithi077.stdout: create 194/38 file after the snapshot 2022-04-03T19:46:16.939 INFO:tasks.workunit.client.0.smithi077.stdout: create 194/39 file after the snapshot 2022-04-03T19:46:16.939 INFO:tasks.workunit.client.0.smithi077.stdout: create 195/.snap/snap-subdir-test 2022-04-03T19:46:16.962 INFO:tasks.workunit.client.0.smithi077.stdout: create 195/30 file after the snapshot 2022-04-03T19:46:16.964 INFO:tasks.workunit.client.0.smithi077.stdout: create 195/31 file after the snapshot 2022-04-03T19:46:16.965 INFO:tasks.workunit.client.0.smithi077.stdout: create 195/32 file after the snapshot 2022-04-03T19:46:16.965 INFO:tasks.workunit.client.0.smithi077.stdout: create 195/33 file after the snapshot 2022-04-03T19:46:16.965 INFO:tasks.workunit.client.0.smithi077.stdout: create 195/34 file after the snapshot 2022-04-03T19:46:16.965 INFO:tasks.workunit.client.0.smithi077.stdout: create 195/35 file after the snapshot 2022-04-03T19:46:16.966 INFO:tasks.workunit.client.0.smithi077.stdout: create 195/36 file after the snapshot 2022-04-03T19:46:16.966 INFO:tasks.workunit.client.0.smithi077.stdout: create 195/37 file after the snapshot 2022-04-03T19:46:16.966 INFO:tasks.workunit.client.0.smithi077.stdout: create 195/38 file after the snapshot 2022-04-03T19:46:16.966 INFO:tasks.workunit.client.0.smithi077.stdout: create 195/39 file after the snapshot 2022-04-03T19:46:16.967 INFO:tasks.workunit.client.0.smithi077.stdout: create 196/.snap/snap-subdir-test 2022-04-03T19:46:16.992 INFO:tasks.workunit.client.0.smithi077.stdout: create 196/30 file after the snapshot 2022-04-03T19:46:16.994 INFO:tasks.workunit.client.0.smithi077.stdout: create 196/31 file after the snapshot 2022-04-03T19:46:16.994 INFO:tasks.workunit.client.0.smithi077.stdout: create 196/32 file after the snapshot 2022-04-03T19:46:16.994 INFO:tasks.workunit.client.0.smithi077.stdout: create 196/33 file after the snapshot 2022-04-03T19:46:16.994 INFO:tasks.workunit.client.0.smithi077.stdout: create 196/34 file after the snapshot 2022-04-03T19:46:16.995 INFO:tasks.workunit.client.0.smithi077.stdout: create 196/35 file after the snapshot 2022-04-03T19:46:16.995 INFO:tasks.workunit.client.0.smithi077.stdout: create 196/36 file after the snapshot 2022-04-03T19:46:16.995 INFO:tasks.workunit.client.0.smithi077.stdout: create 196/37 file after the snapshot 2022-04-03T19:46:16.995 INFO:tasks.workunit.client.0.smithi077.stdout: create 196/38 file after the snapshot 2022-04-03T19:46:16.996 INFO:tasks.workunit.client.0.smithi077.stdout: create 196/39 file after the snapshot 2022-04-03T19:46:16.996 INFO:tasks.workunit.client.0.smithi077.stdout: create 197/.snap/snap-subdir-test 2022-04-03T19:46:17.022 INFO:tasks.workunit.client.0.smithi077.stdout: create 197/30 file after the snapshot 2022-04-03T19:46:17.024 INFO:tasks.workunit.client.0.smithi077.stdout: create 197/31 file after the snapshot 2022-04-03T19:46:17.024 INFO:tasks.workunit.client.0.smithi077.stdout: create 197/32 file after the snapshot 2022-04-03T19:46:17.024 INFO:tasks.workunit.client.0.smithi077.stdout: create 197/33 file after the snapshot 2022-04-03T19:46:17.025 INFO:tasks.workunit.client.0.smithi077.stdout: create 197/34 file after the snapshot 2022-04-03T19:46:17.025 INFO:tasks.workunit.client.0.smithi077.stdout: create 197/35 file after the snapshot 2022-04-03T19:46:17.025 INFO:tasks.workunit.client.0.smithi077.stdout: create 197/36 file after the snapshot 2022-04-03T19:46:17.025 INFO:tasks.workunit.client.0.smithi077.stdout: create 197/37 file after the snapshot 2022-04-03T19:46:17.026 INFO:tasks.workunit.client.0.smithi077.stdout: create 197/38 file after the snapshot 2022-04-03T19:46:17.026 INFO:tasks.workunit.client.0.smithi077.stdout: create 197/39 file after the snapshot 2022-04-03T19:46:17.026 INFO:tasks.workunit.client.0.smithi077.stdout: create 198/.snap/snap-subdir-test 2022-04-03T19:46:17.052 INFO:tasks.workunit.client.0.smithi077.stdout: create 198/30 file after the snapshot 2022-04-03T19:46:17.054 INFO:tasks.workunit.client.0.smithi077.stdout: create 198/31 file after the snapshot 2022-04-03T19:46:17.054 INFO:tasks.workunit.client.0.smithi077.stdout: create 198/32 file after the snapshot 2022-04-03T19:46:17.054 INFO:tasks.workunit.client.0.smithi077.stdout: create 198/33 file after the snapshot 2022-04-03T19:46:17.055 INFO:tasks.workunit.client.0.smithi077.stdout: create 198/34 file after the snapshot 2022-04-03T19:46:17.055 INFO:tasks.workunit.client.0.smithi077.stdout: create 198/35 file after the snapshot 2022-04-03T19:46:17.056 INFO:tasks.workunit.client.0.smithi077.stdout: create 198/36 file after the snapshot 2022-04-03T19:46:17.056 INFO:tasks.workunit.client.0.smithi077.stdout: create 198/37 file after the snapshot 2022-04-03T19:46:17.056 INFO:tasks.workunit.client.0.smithi077.stdout: create 198/38 file after the snapshot 2022-04-03T19:46:17.056 INFO:tasks.workunit.client.0.smithi077.stdout: create 198/39 file after the snapshot 2022-04-03T19:46:17.057 INFO:tasks.workunit.client.0.smithi077.stdout: create 199/.snap/snap-subdir-test 2022-04-03T19:46:17.080 INFO:tasks.workunit.client.0.smithi077.stdout: create 199/30 file after the snapshot 2022-04-03T19:46:17.082 INFO:tasks.workunit.client.0.smithi077.stdout: create 199/31 file after the snapshot 2022-04-03T19:46:17.082 INFO:tasks.workunit.client.0.smithi077.stdout: create 199/32 file after the snapshot 2022-04-03T19:46:17.082 INFO:tasks.workunit.client.0.smithi077.stdout: create 199/33 file after the snapshot 2022-04-03T19:46:17.083 INFO:tasks.workunit.client.0.smithi077.stdout: create 199/34 file after the snapshot 2022-04-03T19:46:17.083 INFO:tasks.workunit.client.0.smithi077.stdout: create 199/35 file after the snapshot 2022-04-03T19:46:17.083 INFO:tasks.workunit.client.0.smithi077.stdout: create 199/36 file after the snapshot 2022-04-03T19:46:17.083 INFO:tasks.workunit.client.0.smithi077.stdout: create 199/37 file after the snapshot 2022-04-03T19:46:17.084 INFO:tasks.workunit.client.0.smithi077.stdout: create 199/38 file after the snapshot 2022-04-03T19:46:17.084 INFO:tasks.workunit.client.0.smithi077.stdout: create 199/39 file after the snapshot 2022-04-03T19:46:17.084 INFO:tasks.workunit.client.0.smithi077.stdout: create 200/.snap/snap-subdir-test 2022-04-03T19:46:17.106 INFO:tasks.workunit.client.0.smithi077.stdout: create 200/30 file after the snapshot 2022-04-03T19:46:17.108 INFO:tasks.workunit.client.0.smithi077.stdout: create 200/31 file after the snapshot 2022-04-03T19:46:17.108 INFO:tasks.workunit.client.0.smithi077.stdout: create 200/32 file after the snapshot 2022-04-03T19:46:17.108 INFO:tasks.workunit.client.0.smithi077.stdout: create 200/33 file after the snapshot 2022-04-03T19:46:17.108 INFO:tasks.workunit.client.0.smithi077.stdout: create 200/34 file after the snapshot 2022-04-03T19:46:17.109 INFO:tasks.workunit.client.0.smithi077.stdout: create 200/35 file after the snapshot 2022-04-03T19:46:17.109 INFO:tasks.workunit.client.0.smithi077.stdout: create 200/36 file after the snapshot 2022-04-03T19:46:17.109 INFO:tasks.workunit.client.0.smithi077.stdout: create 200/37 file after the snapshot 2022-04-03T19:46:17.109 INFO:tasks.workunit.client.0.smithi077.stdout: create 200/38 file after the snapshot 2022-04-03T19:46:17.109 INFO:tasks.workunit.client.0.smithi077.stdout: create 200/39 file after the snapshot 2022-04-03T19:46:17.110 INFO:tasks.workunit.client.0.smithi077.stdout: create 201/.snap/snap-subdir-test 2022-04-03T19:46:17.131 INFO:tasks.workunit.client.0.smithi077.stdout: create 201/30 file after the snapshot 2022-04-03T19:46:17.133 INFO:tasks.workunit.client.0.smithi077.stdout: create 201/31 file after the snapshot 2022-04-03T19:46:17.133 INFO:tasks.workunit.client.0.smithi077.stdout: create 201/32 file after the snapshot 2022-04-03T19:46:17.133 INFO:tasks.workunit.client.0.smithi077.stdout: create 201/33 file after the snapshot 2022-04-03T19:46:17.133 INFO:tasks.workunit.client.0.smithi077.stdout: create 201/34 file after the snapshot 2022-04-03T19:46:17.133 INFO:tasks.workunit.client.0.smithi077.stdout: create 201/35 file after the snapshot 2022-04-03T19:46:17.134 INFO:tasks.workunit.client.0.smithi077.stdout: create 201/36 file after the snapshot 2022-04-03T19:46:17.134 INFO:tasks.workunit.client.0.smithi077.stdout: create 201/37 file after the snapshot 2022-04-03T19:46:17.134 INFO:tasks.workunit.client.0.smithi077.stdout: create 201/38 file after the snapshot 2022-04-03T19:46:17.134 INFO:tasks.workunit.client.0.smithi077.stdout: create 201/39 file after the snapshot 2022-04-03T19:46:17.135 INFO:tasks.workunit.client.0.smithi077.stdout: create 202/.snap/snap-subdir-test 2022-04-03T19:46:17.156 INFO:tasks.workunit.client.0.smithi077.stdout: create 202/30 file after the snapshot 2022-04-03T19:46:17.158 INFO:tasks.workunit.client.0.smithi077.stdout: create 202/31 file after the snapshot 2022-04-03T19:46:17.158 INFO:tasks.workunit.client.0.smithi077.stdout: create 202/32 file after the snapshot 2022-04-03T19:46:17.158 INFO:tasks.workunit.client.0.smithi077.stdout: create 202/33 file after the snapshot 2022-04-03T19:46:17.158 INFO:tasks.workunit.client.0.smithi077.stdout: create 202/34 file after the snapshot 2022-04-03T19:46:17.158 INFO:tasks.workunit.client.0.smithi077.stdout: create 202/35 file after the snapshot 2022-04-03T19:46:17.159 INFO:tasks.workunit.client.0.smithi077.stdout: create 202/36 file after the snapshot 2022-04-03T19:46:17.159 INFO:tasks.workunit.client.0.smithi077.stdout: create 202/37 file after the snapshot 2022-04-03T19:46:17.159 INFO:tasks.workunit.client.0.smithi077.stdout: create 202/38 file after the snapshot 2022-04-03T19:46:17.159 INFO:tasks.workunit.client.0.smithi077.stdout: create 202/39 file after the snapshot 2022-04-03T19:46:17.160 INFO:tasks.workunit.client.0.smithi077.stdout: create 203/.snap/snap-subdir-test 2022-04-03T19:46:17.241 INFO:tasks.workunit.client.0.smithi077.stdout: create 203/30 file after the snapshot 2022-04-03T19:46:17.243 INFO:tasks.workunit.client.0.smithi077.stdout: create 203/31 file after the snapshot 2022-04-03T19:46:17.243 INFO:tasks.workunit.client.0.smithi077.stdout: create 203/32 file after the snapshot 2022-04-03T19:46:17.243 INFO:tasks.workunit.client.0.smithi077.stdout: create 203/33 file after the snapshot 2022-04-03T19:46:17.244 INFO:tasks.workunit.client.0.smithi077.stdout: create 203/34 file after the snapshot 2022-04-03T19:46:17.245 INFO:tasks.workunit.client.0.smithi077.stdout: create 203/35 file after the snapshot 2022-04-03T19:46:17.245 INFO:tasks.workunit.client.0.smithi077.stdout: create 203/36 file after the snapshot 2022-04-03T19:46:17.245 INFO:tasks.workunit.client.0.smithi077.stdout: create 203/37 file after the snapshot 2022-04-03T19:46:17.245 INFO:tasks.workunit.client.0.smithi077.stdout: create 203/38 file after the snapshot 2022-04-03T19:46:17.246 INFO:tasks.workunit.client.0.smithi077.stdout: create 203/39 file after the snapshot 2022-04-03T19:46:17.246 INFO:tasks.workunit.client.0.smithi077.stdout: create 204/.snap/snap-subdir-test 2022-04-03T19:46:17.268 INFO:tasks.workunit.client.0.smithi077.stdout: create 204/30 file after the snapshot 2022-04-03T19:46:17.269 INFO:tasks.workunit.client.0.smithi077.stdout: create 204/31 file after the snapshot 2022-04-03T19:46:17.269 INFO:tasks.workunit.client.0.smithi077.stdout: create 204/32 file after the snapshot 2022-04-03T19:46:17.270 INFO:tasks.workunit.client.0.smithi077.stdout: create 204/33 file after the snapshot 2022-04-03T19:46:17.271 INFO:tasks.workunit.client.0.smithi077.stdout: create 204/34 file after the snapshot 2022-04-03T19:46:17.271 INFO:tasks.workunit.client.0.smithi077.stdout: create 204/35 file after the snapshot 2022-04-03T19:46:17.271 INFO:tasks.workunit.client.0.smithi077.stdout: create 204/36 file after the snapshot 2022-04-03T19:46:17.272 INFO:tasks.workunit.client.0.smithi077.stdout: create 204/37 file after the snapshot 2022-04-03T19:46:17.272 INFO:tasks.workunit.client.0.smithi077.stdout: create 204/38 file after the snapshot 2022-04-03T19:46:17.272 INFO:tasks.workunit.client.0.smithi077.stdout: create 204/39 file after the snapshot 2022-04-03T19:46:17.272 INFO:tasks.workunit.client.0.smithi077.stdout: create 205/.snap/snap-subdir-test 2022-04-03T19:46:17.937 INFO:tasks.workunit.client.0.smithi077.stdout: create 205/30 file after the snapshot 2022-04-03T19:46:17.938 INFO:tasks.workunit.client.0.smithi077.stdout: create 205/31 file after the snapshot 2022-04-03T19:46:17.939 INFO:tasks.workunit.client.0.smithi077.stdout: create 205/32 file after the snapshot 2022-04-03T19:46:17.939 INFO:tasks.workunit.client.0.smithi077.stdout: create 205/33 file after the snapshot 2022-04-03T19:46:17.940 INFO:tasks.workunit.client.0.smithi077.stdout: create 205/34 file after the snapshot 2022-04-03T19:46:17.940 INFO:tasks.workunit.client.0.smithi077.stdout: create 205/35 file after the snapshot 2022-04-03T19:46:17.940 INFO:tasks.workunit.client.0.smithi077.stdout: create 205/36 file after the snapshot 2022-04-03T19:46:17.941 INFO:tasks.workunit.client.0.smithi077.stdout: create 205/37 file after the snapshot 2022-04-03T19:46:17.941 INFO:tasks.workunit.client.0.smithi077.stdout: create 205/38 file after the snapshot 2022-04-03T19:46:17.942 INFO:tasks.workunit.client.0.smithi077.stdout: create 205/39 file after the snapshot 2022-04-03T19:46:17.942 INFO:tasks.workunit.client.0.smithi077.stdout: create 206/.snap/snap-subdir-test 2022-04-03T19:46:17.963 INFO:tasks.workunit.client.0.smithi077.stdout: create 206/30 file after the snapshot 2022-04-03T19:46:17.965 INFO:tasks.workunit.client.0.smithi077.stdout: create 206/31 file after the snapshot 2022-04-03T19:46:17.965 INFO:tasks.workunit.client.0.smithi077.stdout: create 206/32 file after the snapshot 2022-04-03T19:46:17.965 INFO:tasks.workunit.client.0.smithi077.stdout: create 206/33 file after the snapshot 2022-04-03T19:46:17.966 INFO:tasks.workunit.client.0.smithi077.stdout: create 206/34 file after the snapshot 2022-04-03T19:46:17.967 INFO:tasks.workunit.client.0.smithi077.stdout: create 206/35 file after the snapshot 2022-04-03T19:46:17.967 INFO:tasks.workunit.client.0.smithi077.stdout: create 206/36 file after the snapshot 2022-04-03T19:46:17.967 INFO:tasks.workunit.client.0.smithi077.stdout: create 206/37 file after the snapshot 2022-04-03T19:46:17.967 INFO:tasks.workunit.client.0.smithi077.stdout: create 206/38 file after the snapshot 2022-04-03T19:46:17.968 INFO:tasks.workunit.client.0.smithi077.stdout: create 206/39 file after the snapshot 2022-04-03T19:46:17.968 INFO:tasks.workunit.client.0.smithi077.stdout: create 207/.snap/snap-subdir-test 2022-04-03T19:46:17.989 INFO:tasks.workunit.client.0.smithi077.stdout: create 207/30 file after the snapshot 2022-04-03T19:46:17.992 INFO:tasks.workunit.client.0.smithi077.stdout: create 207/31 file after the snapshot 2022-04-03T19:46:17.992 INFO:tasks.workunit.client.0.smithi077.stdout: create 207/32 file after the snapshot 2022-04-03T19:46:17.992 INFO:tasks.workunit.client.0.smithi077.stdout: create 207/33 file after the snapshot 2022-04-03T19:46:17.993 INFO:tasks.workunit.client.0.smithi077.stdout: create 207/34 file after the snapshot 2022-04-03T19:46:17.993 INFO:tasks.workunit.client.0.smithi077.stdout: create 207/35 file after the snapshot 2022-04-03T19:46:17.993 INFO:tasks.workunit.client.0.smithi077.stdout: create 207/36 file after the snapshot 2022-04-03T19:46:17.993 INFO:tasks.workunit.client.0.smithi077.stdout: create 207/37 file after the snapshot 2022-04-03T19:46:17.994 INFO:tasks.workunit.client.0.smithi077.stdout: create 207/38 file after the snapshot 2022-04-03T19:46:17.994 INFO:tasks.workunit.client.0.smithi077.stdout: create 207/39 file after the snapshot 2022-04-03T19:46:17.994 INFO:tasks.workunit.client.0.smithi077.stdout: create 208/.snap/snap-subdir-test 2022-04-03T19:46:18.019 INFO:tasks.workunit.client.0.smithi077.stdout: create 208/30 file after the snapshot 2022-04-03T19:46:18.022 INFO:tasks.workunit.client.0.smithi077.stdout: create 208/31 file after the snapshot 2022-04-03T19:46:18.022 INFO:tasks.workunit.client.0.smithi077.stdout: create 208/32 file after the snapshot 2022-04-03T19:46:18.022 INFO:tasks.workunit.client.0.smithi077.stdout: create 208/33 file after the snapshot 2022-04-03T19:46:18.023 INFO:tasks.workunit.client.0.smithi077.stdout: create 208/34 file after the snapshot 2022-04-03T19:46:18.023 INFO:tasks.workunit.client.0.smithi077.stdout: create 208/35 file after the snapshot 2022-04-03T19:46:18.023 INFO:tasks.workunit.client.0.smithi077.stdout: create 208/36 file after the snapshot 2022-04-03T19:46:18.023 INFO:tasks.workunit.client.0.smithi077.stdout: create 208/37 file after the snapshot 2022-04-03T19:46:18.023 INFO:tasks.workunit.client.0.smithi077.stdout: create 208/38 file after the snapshot 2022-04-03T19:46:18.024 INFO:tasks.workunit.client.0.smithi077.stdout: create 208/39 file after the snapshot 2022-04-03T19:46:18.024 INFO:tasks.workunit.client.0.smithi077.stdout: create 209/.snap/snap-subdir-test 2022-04-03T19:46:18.053 INFO:tasks.workunit.client.0.smithi077.stdout: create 209/30 file after the snapshot 2022-04-03T19:46:18.055 INFO:tasks.workunit.client.0.smithi077.stdout: create 209/31 file after the snapshot 2022-04-03T19:46:18.056 INFO:tasks.workunit.client.0.smithi077.stdout: create 209/32 file after the snapshot 2022-04-03T19:46:18.056 INFO:tasks.workunit.client.0.smithi077.stdout: create 209/33 file after the snapshot 2022-04-03T19:46:18.056 INFO:tasks.workunit.client.0.smithi077.stdout: create 209/34 file after the snapshot 2022-04-03T19:46:18.056 INFO:tasks.workunit.client.0.smithi077.stdout: create 209/35 file after the snapshot 2022-04-03T19:46:18.056 INFO:tasks.workunit.client.0.smithi077.stdout: create 209/36 file after the snapshot 2022-04-03T19:46:18.057 INFO:tasks.workunit.client.0.smithi077.stdout: create 209/37 file after the snapshot 2022-04-03T19:46:18.057 INFO:tasks.workunit.client.0.smithi077.stdout: create 209/38 file after the snapshot 2022-04-03T19:46:18.057 INFO:tasks.workunit.client.0.smithi077.stdout: create 209/39 file after the snapshot 2022-04-03T19:46:18.057 INFO:tasks.workunit.client.0.smithi077.stdout: create 210/.snap/snap-subdir-test 2022-04-03T19:46:18.913 INFO:tasks.workunit.client.0.smithi077.stdout: create 210/30 file after the snapshot 2022-04-03T19:46:18.916 INFO:tasks.workunit.client.0.smithi077.stdout: create 210/31 file after the snapshot 2022-04-03T19:46:18.916 INFO:tasks.workunit.client.0.smithi077.stdout: create 210/32 file after the snapshot 2022-04-03T19:46:18.916 INFO:tasks.workunit.client.0.smithi077.stdout: create 210/33 file after the snapshot 2022-04-03T19:46:18.917 INFO:tasks.workunit.client.0.smithi077.stdout: create 210/34 file after the snapshot 2022-04-03T19:46:18.917 INFO:tasks.workunit.client.0.smithi077.stdout: create 210/35 file after the snapshot 2022-04-03T19:46:18.917 INFO:tasks.workunit.client.0.smithi077.stdout: create 210/36 file after the snapshot 2022-04-03T19:46:18.917 INFO:tasks.workunit.client.0.smithi077.stdout: create 210/37 file after the snapshot 2022-04-03T19:46:18.917 INFO:tasks.workunit.client.0.smithi077.stdout: create 210/38 file after the snapshot 2022-04-03T19:46:18.918 INFO:tasks.workunit.client.0.smithi077.stdout: create 210/39 file after the snapshot 2022-04-03T19:46:18.918 INFO:tasks.workunit.client.0.smithi077.stdout: create 211/.snap/snap-subdir-test 2022-04-03T19:46:18.945 INFO:tasks.workunit.client.0.smithi077.stdout: create 211/30 file after the snapshot 2022-04-03T19:46:18.947 INFO:tasks.workunit.client.0.smithi077.stdout: create 211/31 file after the snapshot 2022-04-03T19:46:18.948 INFO:tasks.workunit.client.0.smithi077.stdout: create 211/32 file after the snapshot 2022-04-03T19:46:18.948 INFO:tasks.workunit.client.0.smithi077.stdout: create 211/33 file after the snapshot 2022-04-03T19:46:18.948 INFO:tasks.workunit.client.0.smithi077.stdout: create 211/34 file after the snapshot 2022-04-03T19:46:18.948 INFO:tasks.workunit.client.0.smithi077.stdout: create 211/35 file after the snapshot 2022-04-03T19:46:18.949 INFO:tasks.workunit.client.0.smithi077.stdout: create 211/36 file after the snapshot 2022-04-03T19:46:18.949 INFO:tasks.workunit.client.0.smithi077.stdout: create 211/37 file after the snapshot 2022-04-03T19:46:18.949 INFO:tasks.workunit.client.0.smithi077.stdout: create 211/38 file after the snapshot 2022-04-03T19:46:18.949 INFO:tasks.workunit.client.0.smithi077.stdout: create 211/39 file after the snapshot 2022-04-03T19:46:18.949 INFO:tasks.workunit.client.0.smithi077.stdout: create 212/.snap/snap-subdir-test 2022-04-03T19:46:18.974 INFO:tasks.workunit.client.0.smithi077.stdout: create 212/30 file after the snapshot 2022-04-03T19:46:18.975 INFO:tasks.workunit.client.0.smithi077.stdout: create 212/31 file after the snapshot 2022-04-03T19:46:18.975 INFO:tasks.workunit.client.0.smithi077.stdout: create 212/32 file after the snapshot 2022-04-03T19:46:18.976 INFO:tasks.workunit.client.0.smithi077.stdout: create 212/33 file after the snapshot 2022-04-03T19:46:18.976 INFO:tasks.workunit.client.0.smithi077.stdout: create 212/34 file after the snapshot 2022-04-03T19:46:18.977 INFO:tasks.workunit.client.0.smithi077.stdout: create 212/35 file after the snapshot 2022-04-03T19:46:18.977 INFO:tasks.workunit.client.0.smithi077.stdout: create 212/36 file after the snapshot 2022-04-03T19:46:18.978 INFO:tasks.workunit.client.0.smithi077.stdout: create 212/37 file after the snapshot 2022-04-03T19:46:18.978 INFO:tasks.workunit.client.0.smithi077.stdout: create 212/38 file after the snapshot 2022-04-03T19:46:18.978 INFO:tasks.workunit.client.0.smithi077.stdout: create 212/39 file after the snapshot 2022-04-03T19:46:18.978 INFO:tasks.workunit.client.0.smithi077.stdout: create 213/.snap/snap-subdir-test 2022-04-03T19:46:19.000 INFO:tasks.workunit.client.0.smithi077.stdout: create 213/30 file after the snapshot 2022-04-03T19:46:19.002 INFO:tasks.workunit.client.0.smithi077.stdout: create 213/31 file after the snapshot 2022-04-03T19:46:19.002 INFO:tasks.workunit.client.0.smithi077.stdout: create 213/32 file after the snapshot 2022-04-03T19:46:19.003 INFO:tasks.workunit.client.0.smithi077.stdout: create 213/33 file after the snapshot 2022-04-03T19:46:19.003 INFO:tasks.workunit.client.0.smithi077.stdout: create 213/34 file after the snapshot 2022-04-03T19:46:19.004 INFO:tasks.workunit.client.0.smithi077.stdout: create 213/35 file after the snapshot 2022-04-03T19:46:19.004 INFO:tasks.workunit.client.0.smithi077.stdout: create 213/36 file after the snapshot 2022-04-03T19:46:19.004 INFO:tasks.workunit.client.0.smithi077.stdout: create 213/37 file after the snapshot 2022-04-03T19:46:19.005 INFO:tasks.workunit.client.0.smithi077.stdout: create 213/38 file after the snapshot 2022-04-03T19:46:19.005 INFO:tasks.workunit.client.0.smithi077.stdout: create 213/39 file after the snapshot 2022-04-03T19:46:19.005 INFO:tasks.workunit.client.0.smithi077.stdout: create 214/.snap/snap-subdir-test 2022-04-03T19:46:19.026 INFO:tasks.workunit.client.0.smithi077.stdout: create 214/30 file after the snapshot 2022-04-03T19:46:19.027 INFO:tasks.workunit.client.0.smithi077.stdout: create 214/31 file after the snapshot 2022-04-03T19:46:19.028 INFO:tasks.workunit.client.0.smithi077.stdout: create 214/32 file after the snapshot 2022-04-03T19:46:19.029 INFO:tasks.workunit.client.0.smithi077.stdout: create 214/33 file after the snapshot 2022-04-03T19:46:19.029 INFO:tasks.workunit.client.0.smithi077.stdout: create 214/34 file after the snapshot 2022-04-03T19:46:19.029 INFO:tasks.workunit.client.0.smithi077.stdout: create 214/35 file after the snapshot 2022-04-03T19:46:19.029 INFO:tasks.workunit.client.0.smithi077.stdout: create 214/36 file after the snapshot 2022-04-03T19:46:19.029 INFO:tasks.workunit.client.0.smithi077.stdout: create 214/37 file after the snapshot 2022-04-03T19:46:19.030 INFO:tasks.workunit.client.0.smithi077.stdout: create 214/38 file after the snapshot 2022-04-03T19:46:19.030 INFO:tasks.workunit.client.0.smithi077.stdout: create 214/39 file after the snapshot 2022-04-03T19:46:19.031 INFO:tasks.workunit.client.0.smithi077.stdout: create 215/.snap/snap-subdir-test 2022-04-03T19:46:19.051 INFO:tasks.workunit.client.0.smithi077.stdout: create 215/30 file after the snapshot 2022-04-03T19:46:19.052 INFO:tasks.workunit.client.0.smithi077.stdout: create 215/31 file after the snapshot 2022-04-03T19:46:19.052 INFO:tasks.workunit.client.0.smithi077.stdout: create 215/32 file after the snapshot 2022-04-03T19:46:19.053 INFO:tasks.workunit.client.0.smithi077.stdout: create 215/33 file after the snapshot 2022-04-03T19:46:19.053 INFO:tasks.workunit.client.0.smithi077.stdout: create 215/34 file after the snapshot 2022-04-03T19:46:19.053 INFO:tasks.workunit.client.0.smithi077.stdout: create 215/35 file after the snapshot 2022-04-03T19:46:19.054 INFO:tasks.workunit.client.0.smithi077.stdout: create 215/36 file after the snapshot 2022-04-03T19:46:19.055 INFO:tasks.workunit.client.0.smithi077.stdout: create 215/37 file after the snapshot 2022-04-03T19:46:19.055 INFO:tasks.workunit.client.0.smithi077.stdout: create 215/38 file after the snapshot 2022-04-03T19:46:19.055 INFO:tasks.workunit.client.0.smithi077.stdout: create 215/39 file after the snapshot 2022-04-03T19:46:19.055 INFO:tasks.workunit.client.0.smithi077.stdout: create 216/.snap/snap-subdir-test 2022-04-03T19:46:19.075 INFO:tasks.workunit.client.0.smithi077.stdout: create 216/30 file after the snapshot 2022-04-03T19:46:19.077 INFO:tasks.workunit.client.0.smithi077.stdout: create 216/31 file after the snapshot 2022-04-03T19:46:19.078 INFO:tasks.workunit.client.0.smithi077.stdout: create 216/32 file after the snapshot 2022-04-03T19:46:19.078 INFO:tasks.workunit.client.0.smithi077.stdout: create 216/33 file after the snapshot 2022-04-03T19:46:19.078 INFO:tasks.workunit.client.0.smithi077.stdout: create 216/34 file after the snapshot 2022-04-03T19:46:19.078 INFO:tasks.workunit.client.0.smithi077.stdout: create 216/35 file after the snapshot 2022-04-03T19:46:19.078 INFO:tasks.workunit.client.0.smithi077.stdout: create 216/36 file after the snapshot 2022-04-03T19:46:19.079 INFO:tasks.workunit.client.0.smithi077.stdout: create 216/37 file after the snapshot 2022-04-03T19:46:19.079 INFO:tasks.workunit.client.0.smithi077.stdout: create 216/38 file after the snapshot 2022-04-03T19:46:19.079 INFO:tasks.workunit.client.0.smithi077.stdout: create 216/39 file after the snapshot 2022-04-03T19:46:19.079 INFO:tasks.workunit.client.0.smithi077.stdout: create 217/.snap/snap-subdir-test 2022-04-03T19:46:19.103 INFO:tasks.workunit.client.0.smithi077.stdout: create 217/30 file after the snapshot 2022-04-03T19:46:19.105 INFO:tasks.workunit.client.0.smithi077.stdout: create 217/31 file after the snapshot 2022-04-03T19:46:19.105 INFO:tasks.workunit.client.0.smithi077.stdout: create 217/32 file after the snapshot 2022-04-03T19:46:19.105 INFO:tasks.workunit.client.0.smithi077.stdout: create 217/33 file after the snapshot 2022-04-03T19:46:19.106 INFO:tasks.workunit.client.0.smithi077.stdout: create 217/34 file after the snapshot 2022-04-03T19:46:19.106 INFO:tasks.workunit.client.0.smithi077.stdout: create 217/35 file after the snapshot 2022-04-03T19:46:19.106 INFO:tasks.workunit.client.0.smithi077.stdout: create 217/36 file after the snapshot 2022-04-03T19:46:19.106 INFO:tasks.workunit.client.0.smithi077.stdout: create 217/37 file after the snapshot 2022-04-03T19:46:19.107 INFO:tasks.workunit.client.0.smithi077.stdout: create 217/38 file after the snapshot 2022-04-03T19:46:19.107 INFO:tasks.workunit.client.0.smithi077.stdout: create 217/39 file after the snapshot 2022-04-03T19:46:19.107 INFO:tasks.workunit.client.0.smithi077.stdout: create 218/.snap/snap-subdir-test 2022-04-03T19:46:19.134 INFO:tasks.workunit.client.0.smithi077.stdout: create 218/30 file after the snapshot 2022-04-03T19:46:19.136 INFO:tasks.workunit.client.0.smithi077.stdout: create 218/31 file after the snapshot 2022-04-03T19:46:19.137 INFO:tasks.workunit.client.0.smithi077.stdout: create 218/32 file after the snapshot 2022-04-03T19:46:19.137 INFO:tasks.workunit.client.0.smithi077.stdout: create 218/33 file after the snapshot 2022-04-03T19:46:19.137 INFO:tasks.workunit.client.0.smithi077.stdout: create 218/34 file after the snapshot 2022-04-03T19:46:19.137 INFO:tasks.workunit.client.0.smithi077.stdout: create 218/35 file after the snapshot 2022-04-03T19:46:19.138 INFO:tasks.workunit.client.0.smithi077.stdout: create 218/36 file after the snapshot 2022-04-03T19:46:19.138 INFO:tasks.workunit.client.0.smithi077.stdout: create 218/37 file after the snapshot 2022-04-03T19:46:19.138 INFO:tasks.workunit.client.0.smithi077.stdout: create 218/38 file after the snapshot 2022-04-03T19:46:19.138 INFO:tasks.workunit.client.0.smithi077.stdout: create 218/39 file after the snapshot 2022-04-03T19:46:19.139 INFO:tasks.workunit.client.0.smithi077.stdout: create 219/.snap/snap-subdir-test 2022-04-03T19:46:19.165 INFO:tasks.workunit.client.0.smithi077.stdout: create 219/30 file after the snapshot 2022-04-03T19:46:19.167 INFO:tasks.workunit.client.0.smithi077.stdout: create 219/31 file after the snapshot 2022-04-03T19:46:19.167 INFO:tasks.workunit.client.0.smithi077.stdout: create 219/32 file after the snapshot 2022-04-03T19:46:19.168 INFO:tasks.workunit.client.0.smithi077.stdout: create 219/33 file after the snapshot 2022-04-03T19:46:19.168 INFO:tasks.workunit.client.0.smithi077.stdout: create 219/34 file after the snapshot 2022-04-03T19:46:19.168 INFO:tasks.workunit.client.0.smithi077.stdout: create 219/35 file after the snapshot 2022-04-03T19:46:19.168 INFO:tasks.workunit.client.0.smithi077.stdout: create 219/36 file after the snapshot 2022-04-03T19:46:19.169 INFO:tasks.workunit.client.0.smithi077.stdout: create 219/37 file after the snapshot 2022-04-03T19:46:19.169 INFO:tasks.workunit.client.0.smithi077.stdout: create 219/38 file after the snapshot 2022-04-03T19:46:19.170 INFO:tasks.workunit.client.0.smithi077.stdout: create 219/39 file after the snapshot 2022-04-03T19:46:19.170 INFO:tasks.workunit.client.0.smithi077.stdout: create 220/.snap/snap-subdir-test 2022-04-03T19:46:19.196 INFO:tasks.workunit.client.0.smithi077.stdout: create 220/30 file after the snapshot 2022-04-03T19:46:19.198 INFO:tasks.workunit.client.0.smithi077.stdout: create 220/31 file after the snapshot 2022-04-03T19:46:19.198 INFO:tasks.workunit.client.0.smithi077.stdout: create 220/32 file after the snapshot 2022-04-03T19:46:19.198 INFO:tasks.workunit.client.0.smithi077.stdout: create 220/33 file after the snapshot 2022-04-03T19:46:19.199 INFO:tasks.workunit.client.0.smithi077.stdout: create 220/34 file after the snapshot 2022-04-03T19:46:19.199 INFO:tasks.workunit.client.0.smithi077.stdout: create 220/35 file after the snapshot 2022-04-03T19:46:19.199 INFO:tasks.workunit.client.0.smithi077.stdout: create 220/36 file after the snapshot 2022-04-03T19:46:19.200 INFO:tasks.workunit.client.0.smithi077.stdout: create 220/37 file after the snapshot 2022-04-03T19:46:19.200 INFO:tasks.workunit.client.0.smithi077.stdout: create 220/38 file after the snapshot 2022-04-03T19:46:19.200 INFO:tasks.workunit.client.0.smithi077.stdout: create 220/39 file after the snapshot 2022-04-03T19:46:19.201 INFO:tasks.workunit.client.0.smithi077.stdout: create 221/.snap/snap-subdir-test 2022-04-03T19:46:19.220 INFO:tasks.workunit.client.0.smithi077.stdout: create 221/30 file after the snapshot 2022-04-03T19:46:19.221 INFO:tasks.workunit.client.0.smithi077.stdout: create 221/31 file after the snapshot 2022-04-03T19:46:19.222 INFO:tasks.workunit.client.0.smithi077.stdout: create 221/32 file after the snapshot 2022-04-03T19:46:19.222 INFO:tasks.workunit.client.0.smithi077.stdout: create 221/33 file after the snapshot 2022-04-03T19:46:19.223 INFO:tasks.workunit.client.0.smithi077.stdout: create 221/34 file after the snapshot 2022-04-03T19:46:19.223 INFO:tasks.workunit.client.0.smithi077.stdout: create 221/35 file after the snapshot 2022-04-03T19:46:19.223 INFO:tasks.workunit.client.0.smithi077.stdout: create 221/36 file after the snapshot 2022-04-03T19:46:19.224 INFO:tasks.workunit.client.0.smithi077.stdout: create 221/37 file after the snapshot 2022-04-03T19:46:19.224 INFO:tasks.workunit.client.0.smithi077.stdout: create 221/38 file after the snapshot 2022-04-03T19:46:19.224 INFO:tasks.workunit.client.0.smithi077.stdout: create 221/39 file after the snapshot 2022-04-03T19:46:19.225 INFO:tasks.workunit.client.0.smithi077.stdout: create 222/.snap/snap-subdir-test 2022-04-03T19:46:19.251 INFO:tasks.workunit.client.0.smithi077.stdout: create 222/30 file after the snapshot 2022-04-03T19:46:19.253 INFO:tasks.workunit.client.0.smithi077.stdout: create 222/31 file after the snapshot 2022-04-03T19:46:19.253 INFO:tasks.workunit.client.0.smithi077.stdout: create 222/32 file after the snapshot 2022-04-03T19:46:19.253 INFO:tasks.workunit.client.0.smithi077.stdout: create 222/33 file after the snapshot 2022-04-03T19:46:19.253 INFO:tasks.workunit.client.0.smithi077.stdout: create 222/34 file after the snapshot 2022-04-03T19:46:19.254 INFO:tasks.workunit.client.0.smithi077.stdout: create 222/35 file after the snapshot 2022-04-03T19:46:19.254 INFO:tasks.workunit.client.0.smithi077.stdout: create 222/36 file after the snapshot 2022-04-03T19:46:19.254 INFO:tasks.workunit.client.0.smithi077.stdout: create 222/37 file after the snapshot 2022-04-03T19:46:19.254 INFO:tasks.workunit.client.0.smithi077.stdout: create 222/38 file after the snapshot 2022-04-03T19:46:19.254 INFO:tasks.workunit.client.0.smithi077.stdout: create 222/39 file after the snapshot 2022-04-03T19:46:19.255 INFO:tasks.workunit.client.0.smithi077.stdout: create 223/.snap/snap-subdir-test 2022-04-03T19:46:19.278 INFO:tasks.workunit.client.0.smithi077.stdout: create 223/30 file after the snapshot 2022-04-03T19:46:19.280 INFO:tasks.workunit.client.0.smithi077.stdout: create 223/31 file after the snapshot 2022-04-03T19:46:19.280 INFO:tasks.workunit.client.0.smithi077.stdout: create 223/32 file after the snapshot 2022-04-03T19:46:19.280 INFO:tasks.workunit.client.0.smithi077.stdout: create 223/33 file after the snapshot 2022-04-03T19:46:19.281 INFO:tasks.workunit.client.0.smithi077.stdout: create 223/34 file after the snapshot 2022-04-03T19:46:19.281 INFO:tasks.workunit.client.0.smithi077.stdout: create 223/35 file after the snapshot 2022-04-03T19:46:19.281 INFO:tasks.workunit.client.0.smithi077.stdout: create 223/36 file after the snapshot 2022-04-03T19:46:19.281 INFO:tasks.workunit.client.0.smithi077.stdout: create 223/37 file after the snapshot 2022-04-03T19:46:19.281 INFO:tasks.workunit.client.0.smithi077.stdout: create 223/38 file after the snapshot 2022-04-03T19:46:19.282 INFO:tasks.workunit.client.0.smithi077.stdout: create 223/39 file after the snapshot 2022-04-03T19:46:19.282 INFO:tasks.workunit.client.0.smithi077.stdout: create 224/.snap/snap-subdir-test 2022-04-03T19:46:19.304 INFO:tasks.workunit.client.0.smithi077.stdout: create 224/30 file after the snapshot 2022-04-03T19:46:19.306 INFO:tasks.workunit.client.0.smithi077.stdout: create 224/31 file after the snapshot 2022-04-03T19:46:19.306 INFO:tasks.workunit.client.0.smithi077.stdout: create 224/32 file after the snapshot 2022-04-03T19:46:19.306 INFO:tasks.workunit.client.0.smithi077.stdout: create 224/33 file after the snapshot 2022-04-03T19:46:19.307 INFO:tasks.workunit.client.0.smithi077.stdout: create 224/34 file after the snapshot 2022-04-03T19:46:19.307 INFO:tasks.workunit.client.0.smithi077.stdout: create 224/35 file after the snapshot 2022-04-03T19:46:19.307 INFO:tasks.workunit.client.0.smithi077.stdout: create 224/36 file after the snapshot 2022-04-03T19:46:19.307 INFO:tasks.workunit.client.0.smithi077.stdout: create 224/37 file after the snapshot 2022-04-03T19:46:19.307 INFO:tasks.workunit.client.0.smithi077.stdout: create 224/38 file after the snapshot 2022-04-03T19:46:19.308 INFO:tasks.workunit.client.0.smithi077.stdout: create 224/39 file after the snapshot 2022-04-03T19:46:19.308 INFO:tasks.workunit.client.0.smithi077.stdout: create 225/.snap/snap-subdir-test 2022-04-03T19:46:19.330 INFO:tasks.workunit.client.0.smithi077.stdout: create 225/30 file after the snapshot 2022-04-03T19:46:19.332 INFO:tasks.workunit.client.0.smithi077.stdout: create 225/31 file after the snapshot 2022-04-03T19:46:19.332 INFO:tasks.workunit.client.0.smithi077.stdout: create 225/32 file after the snapshot 2022-04-03T19:46:19.333 INFO:tasks.workunit.client.0.smithi077.stdout: create 225/33 file after the snapshot 2022-04-03T19:46:19.333 INFO:tasks.workunit.client.0.smithi077.stdout: create 225/34 file after the snapshot 2022-04-03T19:46:19.333 INFO:tasks.workunit.client.0.smithi077.stdout: create 225/35 file after the snapshot 2022-04-03T19:46:19.333 INFO:tasks.workunit.client.0.smithi077.stdout: create 225/36 file after the snapshot 2022-04-03T19:46:19.334 INFO:tasks.workunit.client.0.smithi077.stdout: create 225/37 file after the snapshot 2022-04-03T19:46:19.334 INFO:tasks.workunit.client.0.smithi077.stdout: create 225/38 file after the snapshot 2022-04-03T19:46:19.334 INFO:tasks.workunit.client.0.smithi077.stdout: create 225/39 file after the snapshot 2022-04-03T19:46:19.334 INFO:tasks.workunit.client.0.smithi077.stdout: create 226/.snap/snap-subdir-test 2022-04-03T19:46:19.356 INFO:tasks.workunit.client.0.smithi077.stdout: create 226/30 file after the snapshot 2022-04-03T19:46:19.358 INFO:tasks.workunit.client.0.smithi077.stdout: create 226/31 file after the snapshot 2022-04-03T19:46:19.359 INFO:tasks.workunit.client.0.smithi077.stdout: create 226/32 file after the snapshot 2022-04-03T19:46:19.359 INFO:tasks.workunit.client.0.smithi077.stdout: create 226/33 file after the snapshot 2022-04-03T19:46:19.359 INFO:tasks.workunit.client.0.smithi077.stdout: create 226/34 file after the snapshot 2022-04-03T19:46:19.359 INFO:tasks.workunit.client.0.smithi077.stdout: create 226/35 file after the snapshot 2022-04-03T19:46:19.360 INFO:tasks.workunit.client.0.smithi077.stdout: create 226/36 file after the snapshot 2022-04-03T19:46:19.360 INFO:tasks.workunit.client.0.smithi077.stdout: create 226/37 file after the snapshot 2022-04-03T19:46:19.360 INFO:tasks.workunit.client.0.smithi077.stdout: create 226/38 file after the snapshot 2022-04-03T19:46:19.360 INFO:tasks.workunit.client.0.smithi077.stdout: create 226/39 file after the snapshot 2022-04-03T19:46:19.360 INFO:tasks.workunit.client.0.smithi077.stdout: create 227/.snap/snap-subdir-test 2022-04-03T19:46:19.384 INFO:tasks.workunit.client.0.smithi077.stdout: create 227/30 file after the snapshot 2022-04-03T19:46:19.386 INFO:tasks.workunit.client.0.smithi077.stdout: create 227/31 file after the snapshot 2022-04-03T19:46:19.386 INFO:tasks.workunit.client.0.smithi077.stdout: create 227/32 file after the snapshot 2022-04-03T19:46:19.387 INFO:tasks.workunit.client.0.smithi077.stdout: create 227/33 file after the snapshot 2022-04-03T19:46:19.387 INFO:tasks.workunit.client.0.smithi077.stdout: create 227/34 file after the snapshot 2022-04-03T19:46:19.387 INFO:tasks.workunit.client.0.smithi077.stdout: create 227/35 file after the snapshot 2022-04-03T19:46:19.387 INFO:tasks.workunit.client.0.smithi077.stdout: create 227/36 file after the snapshot 2022-04-03T19:46:19.387 INFO:tasks.workunit.client.0.smithi077.stdout: create 227/37 file after the snapshot 2022-04-03T19:46:19.388 INFO:tasks.workunit.client.0.smithi077.stdout: create 227/38 file after the snapshot 2022-04-03T19:46:19.388 INFO:tasks.workunit.client.0.smithi077.stdout: create 227/39 file after the snapshot 2022-04-03T19:46:19.388 INFO:tasks.workunit.client.0.smithi077.stdout: create 228/.snap/snap-subdir-test 2022-04-03T19:46:19.736 INFO:tasks.workunit.client.0.smithi077.stdout: create 228/30 file after the snapshot 2022-04-03T19:46:19.738 INFO:tasks.workunit.client.0.smithi077.stdout: create 228/31 file after the snapshot 2022-04-03T19:46:19.738 INFO:tasks.workunit.client.0.smithi077.stdout: create 228/32 file after the snapshot 2022-04-03T19:46:19.738 INFO:tasks.workunit.client.0.smithi077.stdout: create 228/33 file after the snapshot 2022-04-03T19:46:19.738 INFO:tasks.workunit.client.0.smithi077.stdout: create 228/34 file after the snapshot 2022-04-03T19:46:19.739 INFO:tasks.workunit.client.0.smithi077.stdout: create 228/35 file after the snapshot 2022-04-03T19:46:19.739 INFO:tasks.workunit.client.0.smithi077.stdout: create 228/36 file after the snapshot 2022-04-03T19:46:19.739 INFO:tasks.workunit.client.0.smithi077.stdout: create 228/37 file after the snapshot 2022-04-03T19:46:19.739 INFO:tasks.workunit.client.0.smithi077.stdout: create 228/38 file after the snapshot 2022-04-03T19:46:19.739 INFO:tasks.workunit.client.0.smithi077.stdout: create 228/39 file after the snapshot 2022-04-03T19:46:19.740 INFO:tasks.workunit.client.0.smithi077.stdout: create 229/.snap/snap-subdir-test 2022-04-03T19:46:19.767 INFO:tasks.workunit.client.0.smithi077.stdout: create 229/30 file after the snapshot 2022-04-03T19:46:19.769 INFO:tasks.workunit.client.0.smithi077.stdout: create 229/31 file after the snapshot 2022-04-03T19:46:19.769 INFO:tasks.workunit.client.0.smithi077.stdout: create 229/32 file after the snapshot 2022-04-03T19:46:19.769 INFO:tasks.workunit.client.0.smithi077.stdout: create 229/33 file after the snapshot 2022-04-03T19:46:19.769 INFO:tasks.workunit.client.0.smithi077.stdout: create 229/34 file after the snapshot 2022-04-03T19:46:19.770 INFO:tasks.workunit.client.0.smithi077.stdout: create 229/35 file after the snapshot 2022-04-03T19:46:19.770 INFO:tasks.workunit.client.0.smithi077.stdout: create 229/36 file after the snapshot 2022-04-03T19:46:19.770 INFO:tasks.workunit.client.0.smithi077.stdout: create 229/37 file after the snapshot 2022-04-03T19:46:19.770 INFO:tasks.workunit.client.0.smithi077.stdout: create 229/38 file after the snapshot 2022-04-03T19:46:19.771 INFO:tasks.workunit.client.0.smithi077.stdout: create 229/39 file after the snapshot 2022-04-03T19:46:19.771 INFO:tasks.workunit.client.0.smithi077.stdout: create 230/.snap/snap-subdir-test 2022-04-03T19:46:20.239 INFO:tasks.workunit.client.0.smithi077.stdout: create 230/30 file after the snapshot 2022-04-03T19:46:20.242 INFO:tasks.workunit.client.0.smithi077.stdout: create 230/31 file after the snapshot 2022-04-03T19:46:20.242 INFO:tasks.workunit.client.0.smithi077.stdout: create 230/32 file after the snapshot 2022-04-03T19:46:20.243 INFO:tasks.workunit.client.0.smithi077.stdout: create 230/33 file after the snapshot 2022-04-03T19:46:20.243 INFO:tasks.workunit.client.0.smithi077.stdout: create 230/34 file after the snapshot 2022-04-03T19:46:20.244 INFO:tasks.workunit.client.0.smithi077.stdout: create 230/35 file after the snapshot 2022-04-03T19:46:20.244 INFO:tasks.workunit.client.0.smithi077.stdout: create 230/36 file after the snapshot 2022-04-03T19:46:20.245 INFO:tasks.workunit.client.0.smithi077.stdout: create 230/37 file after the snapshot 2022-04-03T19:46:20.245 INFO:tasks.workunit.client.0.smithi077.stdout: create 230/38 file after the snapshot 2022-04-03T19:46:20.245 INFO:tasks.workunit.client.0.smithi077.stdout: create 230/39 file after the snapshot 2022-04-03T19:46:20.246 INFO:tasks.workunit.client.0.smithi077.stdout: create 231/.snap/snap-subdir-test 2022-04-03T19:46:20.268 INFO:tasks.workunit.client.0.smithi077.stdout: create 231/30 file after the snapshot 2022-04-03T19:46:20.270 INFO:tasks.workunit.client.0.smithi077.stdout: create 231/31 file after the snapshot 2022-04-03T19:46:20.270 INFO:tasks.workunit.client.0.smithi077.stdout: create 231/32 file after the snapshot 2022-04-03T19:46:20.271 INFO:tasks.workunit.client.0.smithi077.stdout: create 231/33 file after the snapshot 2022-04-03T19:46:20.271 INFO:tasks.workunit.client.0.smithi077.stdout: create 231/34 file after the snapshot 2022-04-03T19:46:20.271 INFO:tasks.workunit.client.0.smithi077.stdout: create 231/35 file after the snapshot 2022-04-03T19:46:20.272 INFO:tasks.workunit.client.0.smithi077.stdout: create 231/36 file after the snapshot 2022-04-03T19:46:20.272 INFO:tasks.workunit.client.0.smithi077.stdout: create 231/37 file after the snapshot 2022-04-03T19:46:20.273 INFO:tasks.workunit.client.0.smithi077.stdout: create 231/38 file after the snapshot 2022-04-03T19:46:20.273 INFO:tasks.workunit.client.0.smithi077.stdout: create 231/39 file after the snapshot 2022-04-03T19:46:20.273 INFO:tasks.workunit.client.0.smithi077.stdout: create 232/.snap/snap-subdir-test 2022-04-03T19:46:20.294 INFO:tasks.workunit.client.0.smithi077.stdout: create 232/30 file after the snapshot 2022-04-03T19:46:20.295 INFO:tasks.workunit.client.0.smithi077.stdout: create 232/31 file after the snapshot 2022-04-03T19:46:20.295 INFO:tasks.workunit.client.0.smithi077.stdout: create 232/32 file after the snapshot 2022-04-03T19:46:20.296 INFO:tasks.workunit.client.0.smithi077.stdout: create 232/33 file after the snapshot 2022-04-03T19:46:20.297 INFO:tasks.workunit.client.0.smithi077.stdout: create 232/34 file after the snapshot 2022-04-03T19:46:20.297 INFO:tasks.workunit.client.0.smithi077.stdout: create 232/35 file after the snapshot 2022-04-03T19:46:20.297 INFO:tasks.workunit.client.0.smithi077.stdout: create 232/36 file after the snapshot 2022-04-03T19:46:20.297 INFO:tasks.workunit.client.0.smithi077.stdout: create 232/37 file after the snapshot 2022-04-03T19:46:20.298 INFO:tasks.workunit.client.0.smithi077.stdout: create 232/38 file after the snapshot 2022-04-03T19:46:20.298 INFO:tasks.workunit.client.0.smithi077.stdout: create 232/39 file after the snapshot 2022-04-03T19:46:20.298 INFO:tasks.workunit.client.0.smithi077.stdout: create 233/.snap/snap-subdir-test 2022-04-03T19:46:20.318 INFO:tasks.workunit.client.0.smithi077.stdout: create 233/30 file after the snapshot 2022-04-03T19:46:20.320 INFO:tasks.workunit.client.0.smithi077.stdout: create 233/31 file after the snapshot 2022-04-03T19:46:20.320 INFO:tasks.workunit.client.0.smithi077.stdout: create 233/32 file after the snapshot 2022-04-03T19:46:20.320 INFO:tasks.workunit.client.0.smithi077.stdout: create 233/33 file after the snapshot 2022-04-03T19:46:20.321 INFO:tasks.workunit.client.0.smithi077.stdout: create 233/34 file after the snapshot 2022-04-03T19:46:20.322 INFO:tasks.workunit.client.0.smithi077.stdout: create 233/35 file after the snapshot 2022-04-03T19:46:20.322 INFO:tasks.workunit.client.0.smithi077.stdout: create 233/36 file after the snapshot 2022-04-03T19:46:20.322 INFO:tasks.workunit.client.0.smithi077.stdout: create 233/37 file after the snapshot 2022-04-03T19:46:20.323 INFO:tasks.workunit.client.0.smithi077.stdout: create 233/38 file after the snapshot 2022-04-03T19:46:20.323 INFO:tasks.workunit.client.0.smithi077.stdout: create 233/39 file after the snapshot 2022-04-03T19:46:20.323 INFO:tasks.workunit.client.0.smithi077.stdout: create 234/.snap/snap-subdir-test 2022-04-03T19:46:20.346 INFO:tasks.workunit.client.0.smithi077.stdout: create 234/30 file after the snapshot 2022-04-03T19:46:20.347 INFO:tasks.workunit.client.0.smithi077.stdout: create 234/31 file after the snapshot 2022-04-03T19:46:20.347 INFO:tasks.workunit.client.0.smithi077.stdout: create 234/32 file after the snapshot 2022-04-03T19:46:20.347 INFO:tasks.workunit.client.0.smithi077.stdout: create 234/33 file after the snapshot 2022-04-03T19:46:20.347 INFO:tasks.workunit.client.0.smithi077.stdout: create 234/34 file after the snapshot 2022-04-03T19:46:20.348 INFO:tasks.workunit.client.0.smithi077.stdout: create 234/35 file after the snapshot 2022-04-03T19:46:20.349 INFO:tasks.workunit.client.0.smithi077.stdout: create 234/36 file after the snapshot 2022-04-03T19:46:20.349 INFO:tasks.workunit.client.0.smithi077.stdout: create 234/37 file after the snapshot 2022-04-03T19:46:20.349 INFO:tasks.workunit.client.0.smithi077.stdout: create 234/38 file after the snapshot 2022-04-03T19:46:20.349 INFO:tasks.workunit.client.0.smithi077.stdout: create 234/39 file after the snapshot 2022-04-03T19:46:20.350 INFO:tasks.workunit.client.0.smithi077.stdout: create 235/.snap/snap-subdir-test 2022-04-03T19:46:20.368 INFO:tasks.workunit.client.0.smithi077.stdout: create 235/30 file after the snapshot 2022-04-03T19:46:20.369 INFO:tasks.workunit.client.0.smithi077.stdout: create 235/31 file after the snapshot 2022-04-03T19:46:20.370 INFO:tasks.workunit.client.0.smithi077.stdout: create 235/32 file after the snapshot 2022-04-03T19:46:20.370 INFO:tasks.workunit.client.0.smithi077.stdout: create 235/33 file after the snapshot 2022-04-03T19:46:20.371 INFO:tasks.workunit.client.0.smithi077.stdout: create 235/34 file after the snapshot 2022-04-03T19:46:20.371 INFO:tasks.workunit.client.0.smithi077.stdout: create 235/35 file after the snapshot 2022-04-03T19:46:20.372 INFO:tasks.workunit.client.0.smithi077.stdout: create 235/36 file after the snapshot 2022-04-03T19:46:20.372 INFO:tasks.workunit.client.0.smithi077.stdout: create 235/37 file after the snapshot 2022-04-03T19:46:20.372 INFO:tasks.workunit.client.0.smithi077.stdout: create 235/38 file after the snapshot 2022-04-03T19:46:20.373 INFO:tasks.workunit.client.0.smithi077.stdout: create 235/39 file after the snapshot 2022-04-03T19:46:20.373 INFO:tasks.workunit.client.0.smithi077.stdout: create 236/.snap/snap-subdir-test 2022-04-03T19:46:21.064 INFO:tasks.workunit.client.0.smithi077.stdout: create 236/30 file after the snapshot 2022-04-03T19:46:21.066 INFO:tasks.workunit.client.0.smithi077.stdout: create 236/31 file after the snapshot 2022-04-03T19:46:21.066 INFO:tasks.workunit.client.0.smithi077.stdout: create 236/32 file after the snapshot 2022-04-03T19:46:21.066 INFO:tasks.workunit.client.0.smithi077.stdout: create 236/33 file after the snapshot 2022-04-03T19:46:21.066 INFO:tasks.workunit.client.0.smithi077.stdout: create 236/34 file after the snapshot 2022-04-03T19:46:21.067 INFO:tasks.workunit.client.0.smithi077.stdout: create 236/35 file after the snapshot 2022-04-03T19:46:21.067 INFO:tasks.workunit.client.0.smithi077.stdout: create 236/36 file after the snapshot 2022-04-03T19:46:21.067 INFO:tasks.workunit.client.0.smithi077.stdout: create 236/37 file after the snapshot 2022-04-03T19:46:21.067 INFO:tasks.workunit.client.0.smithi077.stdout: create 236/38 file after the snapshot 2022-04-03T19:46:21.068 INFO:tasks.workunit.client.0.smithi077.stdout: create 236/39 file after the snapshot 2022-04-03T19:46:21.068 INFO:tasks.workunit.client.0.smithi077.stdout: create 237/.snap/snap-subdir-test 2022-04-03T19:46:21.088 INFO:tasks.workunit.client.0.smithi077.stdout: create 237/30 file after the snapshot 2022-04-03T19:46:21.090 INFO:tasks.workunit.client.0.smithi077.stdout: create 237/31 file after the snapshot 2022-04-03T19:46:21.091 INFO:tasks.workunit.client.0.smithi077.stdout: create 237/32 file after the snapshot 2022-04-03T19:46:21.091 INFO:tasks.workunit.client.0.smithi077.stdout: create 237/33 file after the snapshot 2022-04-03T19:46:21.091 INFO:tasks.workunit.client.0.smithi077.stdout: create 237/34 file after the snapshot 2022-04-03T19:46:21.091 INFO:tasks.workunit.client.0.smithi077.stdout: create 237/35 file after the snapshot 2022-04-03T19:46:21.091 INFO:tasks.workunit.client.0.smithi077.stdout: create 237/36 file after the snapshot 2022-04-03T19:46:21.092 INFO:tasks.workunit.client.0.smithi077.stdout: create 237/37 file after the snapshot 2022-04-03T19:46:21.092 INFO:tasks.workunit.client.0.smithi077.stdout: create 237/38 file after the snapshot 2022-04-03T19:46:21.092 INFO:tasks.workunit.client.0.smithi077.stdout: create 237/39 file after the snapshot 2022-04-03T19:46:21.092 INFO:tasks.workunit.client.0.smithi077.stdout: create 238/.snap/snap-subdir-test 2022-04-03T19:46:21.113 INFO:tasks.workunit.client.0.smithi077.stdout: create 238/30 file after the snapshot 2022-04-03T19:46:21.115 INFO:tasks.workunit.client.0.smithi077.stdout: create 238/31 file after the snapshot 2022-04-03T19:46:21.115 INFO:tasks.workunit.client.0.smithi077.stdout: create 238/32 file after the snapshot 2022-04-03T19:46:21.115 INFO:tasks.workunit.client.0.smithi077.stdout: create 238/33 file after the snapshot 2022-04-03T19:46:21.115 INFO:tasks.workunit.client.0.smithi077.stdout: create 238/34 file after the snapshot 2022-04-03T19:46:21.116 INFO:tasks.workunit.client.0.smithi077.stdout: create 238/35 file after the snapshot 2022-04-03T19:46:21.116 INFO:tasks.workunit.client.0.smithi077.stdout: create 238/36 file after the snapshot 2022-04-03T19:46:21.116 INFO:tasks.workunit.client.0.smithi077.stdout: create 238/37 file after the snapshot 2022-04-03T19:46:21.116 INFO:tasks.workunit.client.0.smithi077.stdout: create 238/38 file after the snapshot 2022-04-03T19:46:21.117 INFO:tasks.workunit.client.0.smithi077.stdout: create 238/39 file after the snapshot 2022-04-03T19:46:21.117 INFO:tasks.workunit.client.0.smithi077.stdout: create 239/.snap/snap-subdir-test 2022-04-03T19:46:21.857 INFO:tasks.workunit.client.0.smithi077.stdout: create 239/30 file after the snapshot 2022-04-03T19:46:21.859 INFO:tasks.workunit.client.0.smithi077.stdout: create 239/31 file after the snapshot 2022-04-03T19:46:21.860 INFO:tasks.workunit.client.0.smithi077.stdout: create 239/32 file after the snapshot 2022-04-03T19:46:21.860 INFO:tasks.workunit.client.0.smithi077.stdout: create 239/33 file after the snapshot 2022-04-03T19:46:21.860 INFO:tasks.workunit.client.0.smithi077.stdout: create 239/34 file after the snapshot 2022-04-03T19:46:21.860 INFO:tasks.workunit.client.0.smithi077.stdout: create 239/35 file after the snapshot 2022-04-03T19:46:21.861 INFO:tasks.workunit.client.0.smithi077.stdout: create 239/36 file after the snapshot 2022-04-03T19:46:21.861 INFO:tasks.workunit.client.0.smithi077.stdout: create 239/37 file after the snapshot 2022-04-03T19:46:21.861 INFO:tasks.workunit.client.0.smithi077.stdout: create 239/38 file after the snapshot 2022-04-03T19:46:21.861 INFO:tasks.workunit.client.0.smithi077.stdout: create 239/39 file after the snapshot 2022-04-03T19:46:21.862 INFO:tasks.workunit.client.0.smithi077.stdout: create 240/.snap/snap-subdir-test 2022-04-03T19:46:23.268 INFO:tasks.workunit.client.0.smithi077.stdout: create 240/30 file after the snapshot 2022-04-03T19:46:24.417 INFO:tasks.workunit.client.0.smithi077.stdout: create 240/31 file after the snapshot 2022-04-03T19:46:24.418 INFO:tasks.workunit.client.0.smithi077.stdout: create 240/32 file after the snapshot 2022-04-03T19:46:24.418 INFO:tasks.workunit.client.0.smithi077.stdout: create 240/33 file after the snapshot 2022-04-03T19:46:24.418 INFO:tasks.workunit.client.0.smithi077.stdout: create 240/34 file after the snapshot 2022-04-03T19:46:24.418 INFO:tasks.workunit.client.0.smithi077.stdout: create 240/35 file after the snapshot 2022-04-03T19:46:24.419 INFO:tasks.workunit.client.0.smithi077.stdout: create 240/36 file after the snapshot 2022-04-03T19:46:24.419 INFO:tasks.workunit.client.0.smithi077.stdout: create 240/37 file after the snapshot 2022-04-03T19:46:24.419 INFO:tasks.workunit.client.0.smithi077.stdout: create 240/38 file after the snapshot 2022-04-03T19:46:24.419 INFO:tasks.workunit.client.0.smithi077.stdout: create 240/39 file after the snapshot 2022-04-03T19:46:24.420 INFO:tasks.workunit.client.0.smithi077.stdout: create 241/.snap/snap-subdir-test 2022-04-03T19:46:25.214 INFO:tasks.workunit.client.0.smithi077.stdout: create 241/30 file after the snapshot 2022-04-03T19:46:25.216 INFO:tasks.workunit.client.0.smithi077.stdout: create 241/31 file after the snapshot 2022-04-03T19:46:25.216 INFO:tasks.workunit.client.0.smithi077.stdout: create 241/32 file after the snapshot 2022-04-03T19:46:25.216 INFO:tasks.workunit.client.0.smithi077.stdout: create 241/33 file after the snapshot 2022-04-03T19:46:25.217 INFO:tasks.workunit.client.0.smithi077.stdout: create 241/34 file after the snapshot 2022-04-03T19:46:25.217 INFO:tasks.workunit.client.0.smithi077.stdout: create 241/35 file after the snapshot 2022-04-03T19:46:25.217 INFO:tasks.workunit.client.0.smithi077.stdout: create 241/36 file after the snapshot 2022-04-03T19:46:25.217 INFO:tasks.workunit.client.0.smithi077.stdout: create 241/37 file after the snapshot 2022-04-03T19:46:25.217 INFO:tasks.workunit.client.0.smithi077.stdout: create 241/38 file after the snapshot 2022-04-03T19:46:25.218 INFO:tasks.workunit.client.0.smithi077.stdout: create 241/39 file after the snapshot 2022-04-03T19:46:25.218 INFO:tasks.workunit.client.0.smithi077.stdout: create 242/.snap/snap-subdir-test 2022-04-03T19:46:25.238 INFO:tasks.workunit.client.0.smithi077.stdout: create 242/30 file after the snapshot 2022-04-03T19:46:25.240 INFO:tasks.workunit.client.0.smithi077.stdout: create 242/31 file after the snapshot 2022-04-03T19:46:25.241 INFO:tasks.workunit.client.0.smithi077.stdout: create 242/32 file after the snapshot 2022-04-03T19:46:25.241 INFO:tasks.workunit.client.0.smithi077.stdout: create 242/33 file after the snapshot 2022-04-03T19:46:25.241 INFO:tasks.workunit.client.0.smithi077.stdout: create 242/34 file after the snapshot 2022-04-03T19:46:25.241 INFO:tasks.workunit.client.0.smithi077.stdout: create 242/35 file after the snapshot 2022-04-03T19:46:25.242 INFO:tasks.workunit.client.0.smithi077.stdout: create 242/36 file after the snapshot 2022-04-03T19:46:25.242 INFO:tasks.workunit.client.0.smithi077.stdout: create 242/37 file after the snapshot 2022-04-03T19:46:25.242 INFO:tasks.workunit.client.0.smithi077.stdout: create 242/38 file after the snapshot 2022-04-03T19:46:25.242 INFO:tasks.workunit.client.0.smithi077.stdout: create 242/39 file after the snapshot 2022-04-03T19:46:25.242 INFO:tasks.workunit.client.0.smithi077.stdout: create 243/.snap/snap-subdir-test 2022-04-03T19:46:25.519 INFO:tasks.workunit.client.0.smithi077.stdout: create 243/30 file after the snapshot 2022-04-03T19:46:25.521 INFO:tasks.workunit.client.0.smithi077.stdout: create 243/31 file after the snapshot 2022-04-03T19:46:25.521 INFO:tasks.workunit.client.0.smithi077.stdout: create 243/32 file after the snapshot 2022-04-03T19:46:25.521 INFO:tasks.workunit.client.0.smithi077.stdout: create 243/33 file after the snapshot 2022-04-03T19:46:25.522 INFO:tasks.workunit.client.0.smithi077.stdout: create 243/34 file after the snapshot 2022-04-03T19:46:25.522 INFO:tasks.workunit.client.0.smithi077.stdout: create 243/35 file after the snapshot 2022-04-03T19:46:25.522 INFO:tasks.workunit.client.0.smithi077.stdout: create 243/36 file after the snapshot 2022-04-03T19:46:25.522 INFO:tasks.workunit.client.0.smithi077.stdout: create 243/37 file after the snapshot 2022-04-03T19:46:25.523 INFO:tasks.workunit.client.0.smithi077.stdout: create 243/38 file after the snapshot 2022-04-03T19:46:25.523 INFO:tasks.workunit.client.0.smithi077.stdout: create 243/39 file after the snapshot 2022-04-03T19:46:25.523 INFO:tasks.workunit.client.0.smithi077.stdout: create 244/.snap/snap-subdir-test 2022-04-03T19:46:25.543 INFO:tasks.workunit.client.0.smithi077.stdout: create 244/30 file after the snapshot 2022-04-03T19:46:25.545 INFO:tasks.workunit.client.0.smithi077.stdout: create 244/31 file after the snapshot 2022-04-03T19:46:25.546 INFO:tasks.workunit.client.0.smithi077.stdout: create 244/32 file after the snapshot 2022-04-03T19:46:25.546 INFO:tasks.workunit.client.0.smithi077.stdout: create 244/33 file after the snapshot 2022-04-03T19:46:25.546 INFO:tasks.workunit.client.0.smithi077.stdout: create 244/34 file after the snapshot 2022-04-03T19:46:25.546 INFO:tasks.workunit.client.0.smithi077.stdout: create 244/35 file after the snapshot 2022-04-03T19:46:25.547 INFO:tasks.workunit.client.0.smithi077.stdout: create 244/36 file after the snapshot 2022-04-03T19:46:25.547 INFO:tasks.workunit.client.0.smithi077.stdout: create 244/37 file after the snapshot 2022-04-03T19:46:25.547 INFO:tasks.workunit.client.0.smithi077.stdout: create 244/38 file after the snapshot 2022-04-03T19:46:25.547 INFO:tasks.workunit.client.0.smithi077.stdout: create 244/39 file after the snapshot 2022-04-03T19:46:25.547 INFO:tasks.workunit.client.0.smithi077.stdout: create 245/.snap/snap-subdir-test 2022-04-03T19:46:25.567 INFO:tasks.workunit.client.0.smithi077.stdout: create 245/30 file after the snapshot 2022-04-03T19:46:25.569 INFO:tasks.workunit.client.0.smithi077.stdout: create 245/31 file after the snapshot 2022-04-03T19:46:25.569 INFO:tasks.workunit.client.0.smithi077.stdout: create 245/32 file after the snapshot 2022-04-03T19:46:25.570 INFO:tasks.workunit.client.0.smithi077.stdout: create 245/33 file after the snapshot 2022-04-03T19:46:25.570 INFO:tasks.workunit.client.0.smithi077.stdout: create 245/34 file after the snapshot 2022-04-03T19:46:25.570 INFO:tasks.workunit.client.0.smithi077.stdout: create 245/35 file after the snapshot 2022-04-03T19:46:25.570 INFO:tasks.workunit.client.0.smithi077.stdout: create 245/36 file after the snapshot 2022-04-03T19:46:25.571 INFO:tasks.workunit.client.0.smithi077.stdout: create 245/37 file after the snapshot 2022-04-03T19:46:25.571 INFO:tasks.workunit.client.0.smithi077.stdout: create 245/38 file after the snapshot 2022-04-03T19:46:25.571 INFO:tasks.workunit.client.0.smithi077.stdout: create 245/39 file after the snapshot 2022-04-03T19:46:25.571 INFO:tasks.workunit.client.0.smithi077.stdout: create 246/.snap/snap-subdir-test 2022-04-03T19:46:25.591 INFO:tasks.workunit.client.0.smithi077.stdout: create 246/30 file after the snapshot 2022-04-03T19:46:25.593 INFO:tasks.workunit.client.0.smithi077.stdout: create 246/31 file after the snapshot 2022-04-03T19:46:25.594 INFO:tasks.workunit.client.0.smithi077.stdout: create 246/32 file after the snapshot 2022-04-03T19:46:25.594 INFO:tasks.workunit.client.0.smithi077.stdout: create 246/33 file after the snapshot 2022-04-03T19:46:25.594 INFO:tasks.workunit.client.0.smithi077.stdout: create 246/34 file after the snapshot 2022-04-03T19:46:25.594 INFO:tasks.workunit.client.0.smithi077.stdout: create 246/35 file after the snapshot 2022-04-03T19:46:25.594 INFO:tasks.workunit.client.0.smithi077.stdout: create 246/36 file after the snapshot 2022-04-03T19:46:25.595 INFO:tasks.workunit.client.0.smithi077.stdout: create 246/37 file after the snapshot 2022-04-03T19:46:25.595 INFO:tasks.workunit.client.0.smithi077.stdout: create 246/38 file after the snapshot 2022-04-03T19:46:25.595 INFO:tasks.workunit.client.0.smithi077.stdout: create 246/39 file after the snapshot 2022-04-03T19:46:25.595 INFO:tasks.workunit.client.0.smithi077.stdout: create 247/.snap/snap-subdir-test 2022-04-03T19:46:25.615 INFO:tasks.workunit.client.0.smithi077.stdout: create 247/30 file after the snapshot 2022-04-03T19:46:25.617 INFO:tasks.workunit.client.0.smithi077.stdout: create 247/31 file after the snapshot 2022-04-03T19:46:25.617 INFO:tasks.workunit.client.0.smithi077.stdout: create 247/32 file after the snapshot 2022-04-03T19:46:25.617 INFO:tasks.workunit.client.0.smithi077.stdout: create 247/33 file after the snapshot 2022-04-03T19:46:25.617 INFO:tasks.workunit.client.0.smithi077.stdout: create 247/34 file after the snapshot 2022-04-03T19:46:25.618 INFO:tasks.workunit.client.0.smithi077.stdout: create 247/35 file after the snapshot 2022-04-03T19:46:25.618 INFO:tasks.workunit.client.0.smithi077.stdout: create 247/36 file after the snapshot 2022-04-03T19:46:25.618 INFO:tasks.workunit.client.0.smithi077.stdout: create 247/37 file after the snapshot 2022-04-03T19:46:25.618 INFO:tasks.workunit.client.0.smithi077.stdout: create 247/38 file after the snapshot 2022-04-03T19:46:25.618 INFO:tasks.workunit.client.0.smithi077.stdout: create 247/39 file after the snapshot 2022-04-03T19:46:25.619 INFO:tasks.workunit.client.0.smithi077.stdout: create 248/.snap/snap-subdir-test 2022-04-03T19:46:25.638 INFO:tasks.workunit.client.0.smithi077.stdout: create 248/30 file after the snapshot 2022-04-03T19:46:25.640 INFO:tasks.workunit.client.0.smithi077.stdout: create 248/31 file after the snapshot 2022-04-03T19:46:25.640 INFO:tasks.workunit.client.0.smithi077.stdout: create 248/32 file after the snapshot 2022-04-03T19:46:25.641 INFO:tasks.workunit.client.0.smithi077.stdout: create 248/33 file after the snapshot 2022-04-03T19:46:25.641 INFO:tasks.workunit.client.0.smithi077.stdout: create 248/34 file after the snapshot 2022-04-03T19:46:25.641 INFO:tasks.workunit.client.0.smithi077.stdout: create 248/35 file after the snapshot 2022-04-03T19:46:25.641 INFO:tasks.workunit.client.0.smithi077.stdout: create 248/36 file after the snapshot 2022-04-03T19:46:25.641 INFO:tasks.workunit.client.0.smithi077.stdout: create 248/37 file after the snapshot 2022-04-03T19:46:25.642 INFO:tasks.workunit.client.0.smithi077.stdout: create 248/38 file after the snapshot 2022-04-03T19:46:25.642 INFO:tasks.workunit.client.0.smithi077.stdout: create 248/39 file after the snapshot 2022-04-03T19:46:25.642 INFO:tasks.workunit.client.0.smithi077.stdout: create 249/.snap/snap-subdir-test 2022-04-03T19:46:25.667 INFO:tasks.workunit.client.0.smithi077.stdout: create 249/30 file after the snapshot 2022-04-03T19:46:25.669 INFO:tasks.workunit.client.0.smithi077.stdout: create 249/31 file after the snapshot 2022-04-03T19:46:25.669 INFO:tasks.workunit.client.0.smithi077.stdout: create 249/32 file after the snapshot 2022-04-03T19:46:25.669 INFO:tasks.workunit.client.0.smithi077.stdout: create 249/33 file after the snapshot 2022-04-03T19:46:25.670 INFO:tasks.workunit.client.0.smithi077.stdout: create 249/34 file after the snapshot 2022-04-03T19:46:25.670 INFO:tasks.workunit.client.0.smithi077.stdout: create 249/35 file after the snapshot 2022-04-03T19:46:25.670 INFO:tasks.workunit.client.0.smithi077.stdout: create 249/36 file after the snapshot 2022-04-03T19:46:25.670 INFO:tasks.workunit.client.0.smithi077.stdout: create 249/37 file after the snapshot 2022-04-03T19:46:25.670 INFO:tasks.workunit.client.0.smithi077.stdout: create 249/38 file after the snapshot 2022-04-03T19:46:25.671 INFO:tasks.workunit.client.0.smithi077.stdout: create 249/39 file after the snapshot 2022-04-03T19:46:25.671 INFO:tasks.workunit.client.0.smithi077.stdout: create 250/.snap/snap-subdir-test 2022-04-03T19:46:25.695 INFO:tasks.workunit.client.0.smithi077.stdout: create 250/30 file after the snapshot 2022-04-03T19:46:25.697 INFO:tasks.workunit.client.0.smithi077.stdout: create 250/31 file after the snapshot 2022-04-03T19:46:25.697 INFO:tasks.workunit.client.0.smithi077.stdout: create 250/32 file after the snapshot 2022-04-03T19:46:25.698 INFO:tasks.workunit.client.0.smithi077.stdout: create 250/33 file after the snapshot 2022-04-03T19:46:25.698 INFO:tasks.workunit.client.0.smithi077.stdout: create 250/34 file after the snapshot 2022-04-03T19:46:25.698 INFO:tasks.workunit.client.0.smithi077.stdout: create 250/35 file after the snapshot 2022-04-03T19:46:25.698 INFO:tasks.workunit.client.0.smithi077.stdout: create 250/36 file after the snapshot 2022-04-03T19:46:25.699 INFO:tasks.workunit.client.0.smithi077.stdout: create 250/37 file after the snapshot 2022-04-03T19:46:25.699 INFO:tasks.workunit.client.0.smithi077.stdout: create 250/38 file after the snapshot 2022-04-03T19:46:25.699 INFO:tasks.workunit.client.0.smithi077.stdout: create 250/39 file after the snapshot 2022-04-03T19:46:25.699 INFO:tasks.workunit.client.0.smithi077.stdout: create 251/.snap/snap-subdir-test 2022-04-03T19:46:25.725 INFO:tasks.workunit.client.0.smithi077.stdout: create 251/30 file after the snapshot 2022-04-03T19:46:25.727 INFO:tasks.workunit.client.0.smithi077.stdout: create 251/31 file after the snapshot 2022-04-03T19:46:25.727 INFO:tasks.workunit.client.0.smithi077.stdout: create 251/32 file after the snapshot 2022-04-03T19:46:25.727 INFO:tasks.workunit.client.0.smithi077.stdout: create 251/33 file after the snapshot 2022-04-03T19:46:25.727 INFO:tasks.workunit.client.0.smithi077.stdout: create 251/34 file after the snapshot 2022-04-03T19:46:25.728 INFO:tasks.workunit.client.0.smithi077.stdout: create 251/35 file after the snapshot 2022-04-03T19:46:25.728 INFO:tasks.workunit.client.0.smithi077.stdout: create 251/36 file after the snapshot 2022-04-03T19:46:25.728 INFO:tasks.workunit.client.0.smithi077.stdout: create 251/37 file after the snapshot 2022-04-03T19:46:25.728 INFO:tasks.workunit.client.0.smithi077.stdout: create 251/38 file after the snapshot 2022-04-03T19:46:25.728 INFO:tasks.workunit.client.0.smithi077.stdout: create 251/39 file after the snapshot 2022-04-03T19:46:25.729 INFO:tasks.workunit.client.0.smithi077.stdout: create 252/.snap/snap-subdir-test 2022-04-03T19:46:25.755 INFO:tasks.workunit.client.0.smithi077.stdout: create 252/30 file after the snapshot 2022-04-03T19:46:25.757 INFO:tasks.workunit.client.0.smithi077.stdout: create 252/31 file after the snapshot 2022-04-03T19:46:25.758 INFO:tasks.workunit.client.0.smithi077.stdout: create 252/32 file after the snapshot 2022-04-03T19:46:25.758 INFO:tasks.workunit.client.0.smithi077.stdout: create 252/33 file after the snapshot 2022-04-03T19:46:25.758 INFO:tasks.workunit.client.0.smithi077.stdout: create 252/34 file after the snapshot 2022-04-03T19:46:25.758 INFO:tasks.workunit.client.0.smithi077.stdout: create 252/35 file after the snapshot 2022-04-03T19:46:25.759 INFO:tasks.workunit.client.0.smithi077.stdout: create 252/36 file after the snapshot 2022-04-03T19:46:25.759 INFO:tasks.workunit.client.0.smithi077.stdout: create 252/37 file after the snapshot 2022-04-03T19:46:25.759 INFO:tasks.workunit.client.0.smithi077.stdout: create 252/38 file after the snapshot 2022-04-03T19:46:25.759 INFO:tasks.workunit.client.0.smithi077.stdout: create 252/39 file after the snapshot 2022-04-03T19:46:25.759 INFO:tasks.workunit.client.0.smithi077.stdout: create 253/.snap/snap-subdir-test 2022-04-03T19:46:26.079 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:46:26.279 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.278+0000 7f11bb59e700 1 Processor -- start 2022-04-03T19:46:26.279 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.278+0000 7f11bb59e700 1 -- start start 2022-04-03T19:46:26.281 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.279+0000 7f11bb59e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f11bc0af230 0x7f11bc067360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:26.281 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.280+0000 7f11bb59e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f11bc0c5760 0x7f11bc0678a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:26.282 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.280+0000 7f11bb59e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11bc066ed0 con 0x7f11bc06ccf0 2022-04-03T19:46:26.282 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.280+0000 7f11bb59e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11bc06d930 con 0x7f11bc0af230 2022-04-03T19:46:26.282 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.280+0000 7f11bb59e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11bc06e380 con 0x7f11bc0c5760 2022-04-03T19:46:26.282 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.280+0000 7f11b3fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f11bc0af230 0x7f11bc067360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:26.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.280+0000 7f11bad9d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f11bc0c5760 0x7f11bc0678a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:26.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.280+0000 7f11b3fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f11bc0af230 0x7f11bc067360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42722/0 (socket says 172.21.15.77:42722) 2022-04-03T19:46:26.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.280+0000 7f11b3fff700 1 -- 172.21.15.77:0/2807269883 learned_addr learned my addr 172.21.15.77:0/2807269883 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:26.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.280+0000 7f11ba59c700 1 --1- 172.21.15.77:0/2807269883 >> v1:172.21.15.77:6789/0 conn(0x7f11bc06ccf0 0x7f11bc0aeb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38384/0 (socket says 172.21.15.77:38384) 2022-04-03T19:46:26.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.280+0000 7f11b3fff700 1 -- 172.21.15.77:0/2807269883 >> v2:172.21.15.186:3300/0 conn(0x7f11bc0c5760 msgr2=0x7f11bc0678a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:46:26.284 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.280+0000 7f11b3fff700 1 --2- 172.21.15.77:0/2807269883 >> v2:172.21.15.186:3300/0 conn(0x7f11bc0c5760 0x7f11bc0678a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:26.284 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.280+0000 7f11b3fff700 1 -- 172.21.15.77:0/2807269883 >> v1:172.21.15.77:6789/0 conn(0x7f11bc06ccf0 legacy=0x7f11bc0aeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:26.284 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.281+0000 7f11b3fff700 1 -- 172.21.15.77:0/2807269883 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc0c6790 con 0x7f11bc0af230 2022-04-03T19:46:26.284 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.280+0000 7f11b9d9b700 1 -- 172.21.15.77:0/2807269883 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155074946 0 0) 0x7f11bc066ed0 con 0x7f11bc06ccf0 2022-04-03T19:46:26.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.281+0000 7f11bad9d700 1 --2- 172.21.15.77:0/2807269883 >> v2:172.21.15.186:3300/0 conn(0x7f11bc0c5760 0x7f11bc0678a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:46:26.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.281+0000 7f11b3fff700 1 --2- 172.21.15.77:0/2807269883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11bc0af230 0x7f11bc067360 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f11bc0c6fa0 tx=0x7f11a40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f442a166b3f1652 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:26.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.281+0000 7f11b9d9b700 1 -- 172.21.15.77:0/2807269883 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a4004030 con 0x7f11bc0af230 2022-04-03T19:46:26.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.281+0000 7f11b9d9b700 1 -- 172.21.15.77:0/2807269883 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a4004190 con 0x7f11bc0af230 2022-04-03T19:46:26.286 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.281+0000 7f11b9d9b700 1 -- 172.21.15.77:0/2807269883 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a4011630 con 0x7f11bc0af230 2022-04-03T19:46:26.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.282+0000 7f11bb59e700 1 -- 172.21.15.77:0/2807269883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11bc0af230 msgr2=0x7f11bc067360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:26.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.282+0000 7f11bb59e700 1 --2- 172.21.15.77:0/2807269883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11bc0af230 0x7f11bc067360 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f11bc0c6fa0 tx=0x7f11a40090f0 comp rx=0 tx=0).stop 2022-04-03T19:46:26.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.282+0000 7f11bb59e700 1 -- 172.21.15.77:0/2807269883 shutdown_connections 2022-04-03T19:46:26.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.282+0000 7f11bb59e700 1 --2- 172.21.15.77:0/2807269883 >> v2:172.21.15.186:3300/0 conn(0x7f11bc0c5760 0x7f11bc0678a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:26.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.282+0000 7f11bb59e700 1 --2- 172.21.15.77:0/2807269883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11bc0af230 0x7f11bc067360 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:26.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.282+0000 7f11bb59e700 1 -- 172.21.15.77:0/2807269883 >> 172.21.15.77:0/2807269883 conn(0x7f11bc069990 msgr2=0x7f11bc0626d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:26.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.282+0000 7f11bb59e700 1 -- 172.21.15.77:0/2807269883 shutdown_connections 2022-04-03T19:46:26.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.282+0000 7f11bb59e700 1 -- 172.21.15.77:0/2807269883 wait complete. 2022-04-03T19:46:26.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.283+0000 7f11bb59e700 1 Processor -- start 2022-04-03T19:46:26.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.284+0000 7f11bb59e700 1 -- start start 2022-04-03T19:46:26.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.284+0000 7f11bb59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11bc06ccf0 0x7f11bc165a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:26.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.284+0000 7f11bb59e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11bc0af230 0x7f11bc152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:26.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.284+0000 7f11bb59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11bc0c5760 0x7f11bc0bf670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:26.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.284+0000 7f11bb59e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc0aa4a0 con 0x7f11bc06ccf0 2022-04-03T19:46:26.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.284+0000 7f11bb59e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc0a9940 con 0x7f11bc0af230 2022-04-03T19:46:26.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.284+0000 7f11bb59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc153290 con 0x7f11bc0c5760 2022-04-03T19:46:26.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.285+0000 7f11ba59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11bc06ccf0 0x7f11bc165a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:26.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.285+0000 7f11ba59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11bc06ccf0 0x7f11bc165a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42724/0 (socket says 172.21.15.77:42724) 2022-04-03T19:46:26.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.285+0000 7f11ba59c700 1 -- 172.21.15.77:0/4126212848 learned_addr learned my addr 172.21.15.77:0/4126212848 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:26.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.285+0000 7f11b3fff700 1 --2- 172.21.15.77:0/4126212848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11bc0af230 0x7f11bc152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:26.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.285+0000 7f11bad9d700 1 --2- 172.21.15.77:0/4126212848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11bc0c5760 0x7f11bc0bf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:26.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.285+0000 7f11ba59c700 1 -- 172.21.15.77:0/4126212848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11bc0c5760 msgr2=0x7f11bc0bf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:26.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.285+0000 7f11ba59c700 1 --2- 172.21.15.77:0/4126212848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11bc0c5760 0x7f11bc0bf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:26.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.285+0000 7f11ba59c700 1 -- 172.21.15.77:0/4126212848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11bc0af230 msgr2=0x7f11bc152d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:26.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.285+0000 7f11ba59c700 1 --2- 172.21.15.77:0/4126212848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11bc0af230 0x7f11bc152d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:26.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.285+0000 7f11ba59c700 1 -- 172.21.15.77:0/4126212848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b4009a20 con 0x7f11bc06ccf0 2022-04-03T19:46:26.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.286+0000 7f11ba59c700 1 --2- 172.21.15.77:0/4126212848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11bc06ccf0 0x7f11bc165a10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f11ac00c4e0 tx=0x7f11ac007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:26.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.286+0000 7f11b27fc700 1 -- 172.21.15.77:0/4126212848 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac007360 con 0x7f11bc06ccf0 2022-04-03T19:46:26.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.286+0000 7f11b27fc700 1 -- 172.21.15.77:0/4126212848 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ac0074c0 con 0x7f11bc06ccf0 2022-04-03T19:46:26.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.286+0000 7f11b27fc700 1 -- 172.21.15.77:0/4126212848 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac004900 con 0x7f11bc06ccf0 2022-04-03T19:46:26.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.286+0000 7f11bb59e700 1 -- 172.21.15.77:0/4126212848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a4009370 con 0x7f11bc06ccf0 2022-04-03T19:46:26.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.286+0000 7f11bb59e700 1 -- 172.21.15.77:0/4126212848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11bc0a9ee0 con 0x7f11bc06ccf0 2022-04-03T19:46:26.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.287+0000 7f11bb59e700 1 -- 172.21.15.77:0/4126212848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11bc069f50 con 0x7f11bc06ccf0 2022-04-03T19:46:26.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.288+0000 7f11b27fc700 1 -- 172.21.15.77:0/4126212848 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f11ac010030 con 0x7f11bc06ccf0 2022-04-03T19:46:26.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.289+0000 7f11b27fc700 1 --2- 172.21.15.77:0/4126212848 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f11a0063d10 0x7f11a00661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:26.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.289+0000 7f11b27fc700 1 -- 172.21.15.77:0/4126212848 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f11ac080380 con 0x7f11bc06ccf0 2022-04-03T19:46:26.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.289+0000 7f11b3fff700 1 --2- 172.21.15.77:0/4126212848 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f11a0063d10 0x7f11a00661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:26.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.290+0000 7f11b3fff700 1 --2- 172.21.15.77:0/4126212848 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f11a0063d10 0x7f11a00661c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f11bc0c6fa0 tx=0x7f11a4016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:26.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.293+0000 7f11b27fc700 1 -- 172.21.15.77:0/4126212848 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f11ac04ef80 con 0x7f11bc06ccf0 2022-04-03T19:46:26.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.529+0000 7f11bb59e700 1 -- 172.21.15.77:0/4126212848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11bc0bd0a0 con 0x7f11bc06ccf0 2022-04-03T19:46:26.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.530+0000 7f11b27fc700 1 -- 172.21.15.77:0/4126212848 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6107 (secure 0 0 0) 0x7f11ac0525a0 con 0x7f11bc06ccf0 2022-04-03T19:46:26.530 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:46:26.531 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4303,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:44:27.295743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4327":{"gid":4327,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6837/3665969042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3665969042},{"type":"v1","addr":"172.21.15.77:6837","nonce":3665969042}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:46:26.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.532+0000 7f11bb59e700 1 -- 172.21.15.77:0/4126212848 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f11a0063d10 msgr2=0x7f11a00661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:26.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.532+0000 7f11bb59e700 1 --2- 172.21.15.77:0/4126212848 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f11a0063d10 0x7f11a00661c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f11bc0c6fa0 tx=0x7f11a4016000 comp rx=0 tx=0).stop 2022-04-03T19:46:26.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.532+0000 7f11bb59e700 1 -- 172.21.15.77:0/4126212848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11bc06ccf0 msgr2=0x7f11bc165a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:26.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.532+0000 7f11bb59e700 1 --2- 172.21.15.77:0/4126212848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11bc06ccf0 0x7f11bc165a10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f11ac00c4e0 tx=0x7f11ac007c70 comp rx=0 tx=0).stop 2022-04-03T19:46:26.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.533+0000 7f11bb59e700 1 -- 172.21.15.77:0/4126212848 shutdown_connections 2022-04-03T19:46:26.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.533+0000 7f11bb59e700 1 --2- 172.21.15.77:0/4126212848 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f11a0063d10 0x7f11a00661c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:26.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.533+0000 7f11bb59e700 1 --2- 172.21.15.77:0/4126212848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11bc0c5760 0x7f11bc0bf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:26.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.533+0000 7f11bb59e700 1 --2- 172.21.15.77:0/4126212848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11bc06ccf0 0x7f11bc165a10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:26.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.533+0000 7f11bb59e700 1 --2- 172.21.15.77:0/4126212848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11bc0af230 0x7f11bc152d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:26.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.533+0000 7f11bb59e700 1 -- 172.21.15.77:0/4126212848 >> 172.21.15.77:0/4126212848 conn(0x7f11bc069990 msgr2=0x7f11bc0c4590 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:26.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.533+0000 7f11bb59e700 1 -- 172.21.15.77:0/4126212848 shutdown_connections 2022-04-03T19:46:26.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.533+0000 7f11bb59e700 1 -- 172.21.15.77:0/4126212848 wait complete. 2022-04-03T19:46:26.536 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2022-04-03T19:46:26.557 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2022-04-03T19:46:26.558 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2022-04-03T19:46:26.558 INFO:teuthology.orchestra.run:waiting for 300 2022-04-03T19:46:26.597 INFO:tasks.ceph.mds.e:Stopped 2022-04-03T19:46:26.598 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:46:26.795 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.795+0000 7f078db9f700 1 Processor -- start 2022-04-03T19:46:26.796 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.795+0000 7f078db9f700 1 -- start start 2022-04-03T19:46:26.797 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.796+0000 7f078db9f700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f07880678b0 0x7f078814cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:26.797 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.796+0000 7f078db9f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f07880c1b30 0x7f078814d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:26.798 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.796+0000 7f078db9f700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07880c3c50 con 0x7f07880ba3c0 2022-04-03T19:46:26.798 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.796+0000 7f078db9f700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07880c8020 con 0x7f07880c1b30 2022-04-03T19:46:26.798 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.796+0000 7f078db9f700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0788156690 con 0x7f07880678b0 2022-04-03T19:46:26.798 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.797+0000 7f078cb9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f07880ba3c0 0x7f0788156b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38386/0 (socket says 172.21.15.77:38386) 2022-04-03T19:46:26.798 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.797+0000 7f078cb9d700 1 -- 172.21.15.77:0/1586353499 learned_addr learned my addr 172.21.15.77:0/1586353499 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:46:26.799 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.797+0000 7f078d39e700 1 --2- 172.21.15.77:0/1586353499 >> v2:172.21.15.77:3300/0 conn(0x7f07880c1b30 0x7f078814d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:26.799 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.797+0000 7f0787fff700 1 --2- 172.21.15.77:0/1586353499 >> v2:172.21.15.186:3300/0 conn(0x7f07880678b0 0x7f078814cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:26.801 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.797+0000 7f07877fe700 1 -- 172.21.15.77:0/1586353499 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167633956 0 0) 0x7f07880c3c50 con 0x7f07880ba3c0 2022-04-03T19:46:26.801 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.797+0000 7f078d39e700 1 -- 172.21.15.77:0/1586353499 >> v2:172.21.15.186:3300/0 conn(0x7f07880678b0 msgr2=0x7f078814cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:46:26.801 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.797+0000 7f078d39e700 1 --2- 172.21.15.77:0/1586353499 >> v2:172.21.15.186:3300/0 conn(0x7f07880678b0 0x7f078814cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:26.801 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.797+0000 7f078d39e700 1 -- 172.21.15.77:0/1586353499 >> v1:172.21.15.77:6789/0 conn(0x7f07880ba3c0 legacy=0x7f0788156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:26.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.797+0000 7f078d39e700 1 -- 172.21.15.77:0/1586353499 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07880bf350 con 0x7f07880c1b30 2022-04-03T19:46:26.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.798+0000 7f078d39e700 1 --2- 172.21.15.77:0/1586353499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07880c1b30 0x7f078814d040 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f07880c3a40 tx=0x7f0780009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=446e56fe3e6a6c36 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:26.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.798+0000 7f07877fe700 1 -- 172.21.15.77:0/1586353499 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078000e070 con 0x7f07880c1b30 2022-04-03T19:46:26.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.798+0000 7f07877fe700 1 -- 172.21.15.77:0/1586353499 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078000bbf0 con 0x7f07880c1b30 2022-04-03T19:46:26.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.798+0000 7f07877fe700 1 -- 172.21.15.77:0/1586353499 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0780012430 con 0x7f07880c1b30 2022-04-03T19:46:26.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.798+0000 7f078db9f700 1 -- 172.21.15.77:0/1586353499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07880c1b30 msgr2=0x7f078814d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:26.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.798+0000 7f078db9f700 1 --2- 172.21.15.77:0/1586353499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07880c1b30 0x7f078814d040 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f07880c3a40 tx=0x7f0780009780 comp rx=0 tx=0).stop 2022-04-03T19:46:26.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.799+0000 7f078db9f700 1 -- 172.21.15.77:0/1586353499 shutdown_connections 2022-04-03T19:46:26.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.799+0000 7f078db9f700 1 --2- 172.21.15.77:0/1586353499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07880c1b30 0x7f078814d040 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:26.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.799+0000 7f078db9f700 1 --2- 172.21.15.77:0/1586353499 >> v2:172.21.15.186:3300/0 conn(0x7f07880678b0 0x7f078814cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:26.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.799+0000 7f078db9f700 1 -- 172.21.15.77:0/1586353499 >> 172.21.15.77:0/1586353499 conn(0x7f0788066da0 msgr2=0x7f07880671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:26.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.799+0000 7f078db9f700 1 -- 172.21.15.77:0/1586353499 shutdown_connections 2022-04-03T19:46:26.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.799+0000 7f078db9f700 1 -- 172.21.15.77:0/1586353499 wait complete. 2022-04-03T19:46:26.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.800+0000 7f078db9f700 1 Processor -- start 2022-04-03T19:46:26.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.800+0000 7f078db9f700 1 -- start start 2022-04-03T19:46:26.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.801+0000 7f078db9f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07880678b0 0x7f0788154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:26.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.801+0000 7f078db9f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07880ba3c0 0x7f07881689d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:26.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.801+0000 7f078db9f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07880c1b30 0x7f0788165400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:26.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.801+0000 7f078db9f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078806a8b0 con 0x7f07880678b0 2022-04-03T19:46:26.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.801+0000 7f078db9f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0788168540 con 0x7f07880ba3c0 2022-04-03T19:46:26.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.801+0000 7f078db9f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0788168f10 con 0x7f07880c1b30 2022-04-03T19:46:26.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.801+0000 7f078cb9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07880678b0 0x7f0788154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:26.809 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.802+0000 7f078cb9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07880678b0 0x7f0788154a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42728/0 (socket says 172.21.15.77:42728) 2022-04-03T19:46:26.809 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.802+0000 7f078cb9d700 1 -- 172.21.15.77:0/634332882 learned_addr learned my addr 172.21.15.77:0/634332882 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:26.809 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.802+0000 7f078d39e700 1 --2- 172.21.15.77:0/634332882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07880c1b30 0x7f0788165400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:26.809 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.802+0000 7f0787fff700 1 --2- 172.21.15.77:0/634332882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07880ba3c0 0x7f07881689d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:26.810 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.802+0000 7f078cb9d700 1 -- 172.21.15.77:0/634332882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07880c1b30 msgr2=0x7f0788165400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:26.810 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.802+0000 7f078cb9d700 1 --2- 172.21.15.77:0/634332882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07880c1b30 0x7f0788165400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:26.810 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.802+0000 7f078cb9d700 1 -- 172.21.15.77:0/634332882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07880ba3c0 msgr2=0x7f07881689d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:26.810 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.802+0000 7f078cb9d700 1 --2- 172.21.15.77:0/634332882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07880ba3c0 0x7f07881689d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:26.811 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.802+0000 7f078cb9d700 1 -- 172.21.15.77:0/634332882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0780009a20 con 0x7f07880678b0 2022-04-03T19:46:26.811 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.802+0000 7f078d39e700 1 --2- 172.21.15.77:0/634332882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07880c1b30 0x7f0788165400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:46:26.811 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.802+0000 7f078cb9d700 1 --2- 172.21.15.77:0/634332882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07880678b0 0x7f0788154a30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f077c00c510 tx=0x7f077c007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:26.811 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.803+0000 7f0784ff9700 1 -- 172.21.15.77:0/634332882 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c00ce20 con 0x7f07880678b0 2022-04-03T19:46:26.811 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.803+0000 7f0784ff9700 1 -- 172.21.15.77:0/634332882 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077c007360 con 0x7f07880678b0 2022-04-03T19:46:26.812 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.803+0000 7f0784ff9700 1 -- 172.21.15.77:0/634332882 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c0046b0 con 0x7f07880678b0 2022-04-03T19:46:26.812 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.803+0000 7f078db9f700 1 -- 172.21.15.77:0/634332882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07880c52c0 con 0x7f07880678b0 2022-04-03T19:46:26.812 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.803+0000 7f078db9f700 1 -- 172.21.15.77:0/634332882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07881681d0 con 0x7f07880678b0 2022-04-03T19:46:26.812 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.804+0000 7f078db9f700 1 -- 172.21.15.77:0/634332882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0788168820 con 0x7f07880678b0 2022-04-03T19:46:26.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.805+0000 7f0784ff9700 1 -- 172.21.15.77:0/634332882 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f077c010030 con 0x7f07880678b0 2022-04-03T19:46:26.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.806+0000 7f0784ff9700 1 --2- 172.21.15.77:0/634332882 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0770063ce0 0x7f0770066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:26.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.806+0000 7f0784ff9700 1 -- 172.21.15.77:0/634332882 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f077c080330 con 0x7f07880678b0 2022-04-03T19:46:26.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.806+0000 7f0787fff700 1 --2- 172.21.15.77:0/634332882 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0770063ce0 0x7f0770066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:26.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.807+0000 7f0787fff700 1 --2- 172.21.15.77:0/634332882 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0770063ce0 0x7f0770066190 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f07880be350 tx=0x7f0774009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:26.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:26.810+0000 7f0784ff9700 1 -- 172.21.15.77:0/634332882 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f077c04ef40 con 0x7f07880678b0 2022-04-03T19:46:27.047 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:27.047+0000 7f078db9f700 1 -- 172.21.15.77:0/634332882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07880c74a0 con 0x7f07880678b0 2022-04-03T19:46:27.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:27.048+0000 7f0784ff9700 1 -- 172.21.15.77:0/634332882 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6107 (secure 0 0 0) 0x7f077c052560 con 0x7f07880678b0 2022-04-03T19:46:27.048 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:46:27.049 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4303,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:44:27.295743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4327":{"gid":4327,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6837/3665969042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3665969042},{"type":"v1","addr":"172.21.15.77:6837","nonce":3665969042}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:46:27.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:27.050+0000 7f078db9f700 1 -- 172.21.15.77:0/634332882 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0770063ce0 msgr2=0x7f0770066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:27.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:27.050+0000 7f078db9f700 1 --2- 172.21.15.77:0/634332882 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0770063ce0 0x7f0770066190 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f07880be350 tx=0x7f0774009060 comp rx=0 tx=0).stop 2022-04-03T19:46:27.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:27.050+0000 7f078db9f700 1 -- 172.21.15.77:0/634332882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07880678b0 msgr2=0x7f0788154a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:27.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:27.050+0000 7f078db9f700 1 --2- 172.21.15.77:0/634332882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07880678b0 0x7f0788154a30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f077c00c510 tx=0x7f077c007c70 comp rx=0 tx=0).stop 2022-04-03T19:46:27.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:27.050+0000 7f078db9f700 1 -- 172.21.15.77:0/634332882 shutdown_connections 2022-04-03T19:46:27.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:27.050+0000 7f078db9f700 1 --2- 172.21.15.77:0/634332882 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0770063ce0 0x7f0770066190 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:27.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:27.050+0000 7f078db9f700 1 --2- 172.21.15.77:0/634332882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07880c1b30 0x7f0788165400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:27.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:27.050+0000 7f078db9f700 1 --2- 172.21.15.77:0/634332882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07880678b0 0x7f0788154a30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:27.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:27.050+0000 7f078db9f700 1 --2- 172.21.15.77:0/634332882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07880ba3c0 0x7f07881689d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:27.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:27.051+0000 7f078db9f700 1 -- 172.21.15.77:0/634332882 >> 172.21.15.77:0/634332882 conn(0x7f0788066da0 msgr2=0x7f078814be60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:27.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:27.051+0000 7f078db9f700 1 -- 172.21.15.77:0/634332882 shutdown_connections 2022-04-03T19:46:27.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:27.051+0000 7f078db9f700 1 -- 172.21.15.77:0/634332882 wait complete. 2022-04-03T19:46:27.055 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2022-04-03T19:46:27.075 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:46:29.077 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:46:29.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.284+0000 7fc76612c700 1 Processor -- start 2022-04-03T19:46:29.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.284+0000 7fc76612c700 1 -- start start 2022-04-03T19:46:29.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.285+0000 7fc76612c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc76006bbe0 0x7fc7601584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:29.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.285+0000 7fc76612c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc76006c500 0x7fc76014eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:29.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.285+0000 7fc76612c700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc76006d260 con 0x7fc76006bfb0 2022-04-03T19:46:29.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.285+0000 7fc76612c700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc760159450 con 0x7fc76006c500 2022-04-03T19:46:29.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.285+0000 7fc76612c700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7600c6330 con 0x7fc76006bbe0 2022-04-03T19:46:29.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.285+0000 7fc76592b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc76006c500 0x7fc76014eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:29.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.285+0000 7fc76592b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc76006c500 0x7fc76014eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42730/0 (socket says 172.21.15.77:42730) 2022-04-03T19:46:29.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.285+0000 7fc76592b700 1 -- 172.21.15.77:0/1556297235 learned_addr learned my addr 172.21.15.77:0/1556297235 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:29.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.285+0000 7fc76512a700 1 --2- 172.21.15.77:0/1556297235 >> v2:172.21.15.186:3300/0 conn(0x7fc76006bbe0 0x7fc7601584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:29.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.286+0000 7fc76592b700 1 -- 172.21.15.77:0/1556297235 >> v2:172.21.15.186:3300/0 conn(0x7fc76006bbe0 msgr2=0x7fc7601584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:46:29.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.286+0000 7fc76592b700 1 --2- 172.21.15.77:0/1556297235 >> v2:172.21.15.186:3300/0 conn(0x7fc76006bbe0 0x7fc7601584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:29.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.286+0000 7fc76592b700 1 -- 172.21.15.77:0/1556297235 >> v1:172.21.15.186:6789/0 conn(0x7fc76006bfb0 legacy=0x7fc76006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:29.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.286+0000 7fc76592b700 1 -- 172.21.15.77:0/1556297235 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc760158bb0 con 0x7fc76006c500 2022-04-03T19:46:29.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.286+0000 7fc76512a700 1 --2- 172.21.15.77:0/1556297235 >> v2:172.21.15.186:3300/0 conn(0x7fc76006bbe0 0x7fc7601584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:46:29.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.286+0000 7fc76592b700 1 --2- 172.21.15.77:0/1556297235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc76006c500 0x7fc76014eea0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc760088d40 tx=0x7fc758009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=174f5a92d849c10e server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:29.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.286+0000 7fc74ffff700 1 -- 172.21.15.77:0/1556297235 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75800e070 con 0x7fc76006c500 2022-04-03T19:46:29.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.286+0000 7fc74ffff700 1 -- 172.21.15.77:0/1556297235 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc75800ba50 con 0x7fc76006c500 2022-04-03T19:46:29.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.286+0000 7fc74ffff700 1 -- 172.21.15.77:0/1556297235 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7580092e0 con 0x7fc76006c500 2022-04-03T19:46:29.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.287+0000 7fc76612c700 1 -- 172.21.15.77:0/1556297235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc76006c500 msgr2=0x7fc76014eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:29.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.287+0000 7fc76612c700 1 --2- 172.21.15.77:0/1556297235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc76006c500 0x7fc76014eea0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc760088d40 tx=0x7fc758009840 comp rx=0 tx=0).stop 2022-04-03T19:46:29.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.287+0000 7fc76612c700 1 -- 172.21.15.77:0/1556297235 shutdown_connections 2022-04-03T19:46:29.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.287+0000 7fc76612c700 1 --2- 172.21.15.77:0/1556297235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc76006c500 0x7fc76014eea0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:29.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.287+0000 7fc76612c700 1 --2- 172.21.15.77:0/1556297235 >> v2:172.21.15.186:3300/0 conn(0x7fc76006bbe0 0x7fc7601584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:29.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.287+0000 7fc76612c700 1 -- 172.21.15.77:0/1556297235 >> 172.21.15.77:0/1556297235 conn(0x7fc7600cb6b0 msgr2=0x7fc7600cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:29.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.287+0000 7fc76612c700 1 -- 172.21.15.77:0/1556297235 shutdown_connections 2022-04-03T19:46:29.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.287+0000 7fc76612c700 1 -- 172.21.15.77:0/1556297235 wait complete. 2022-04-03T19:46:29.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.289+0000 7fc76612c700 1 Processor -- start 2022-04-03T19:46:29.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.289+0000 7fc76612c700 1 -- start start 2022-04-03T19:46:29.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.289+0000 7fc76612c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc76006bbe0 0x7fc760153fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:29.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.289+0000 7fc76612c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc76006bfb0 0x7fc7601544f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:29.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.289+0000 7fc76612c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc76006c500 0x7fc760150b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:29.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.289+0000 7fc76612c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc760164ee0 con 0x7fc76006bbe0 2022-04-03T19:46:29.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.290+0000 7fc76612c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc760165cd0 con 0x7fc76006bfb0 2022-04-03T19:46:29.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.290+0000 7fc76612c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc760165150 con 0x7fc76006c500 2022-04-03T19:46:29.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.290+0000 7fc76512a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc76006bbe0 0x7fc760153fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:29.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.290+0000 7fc76512a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc76006bbe0 0x7fc760153fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42732/0 (socket says 172.21.15.77:42732) 2022-04-03T19:46:29.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.290+0000 7fc76512a700 1 -- 172.21.15.77:0/2897673795 learned_addr learned my addr 172.21.15.77:0/2897673795 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:29.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.290+0000 7fc764929700 1 --2- 172.21.15.77:0/2897673795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc76006bfb0 0x7fc7601544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:29.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.290+0000 7fc76592b700 1 --2- 172.21.15.77:0/2897673795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc76006c500 0x7fc760150b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:29.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.290+0000 7fc76512a700 1 -- 172.21.15.77:0/2897673795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc76006c500 msgr2=0x7fc760150b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:29.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.290+0000 7fc76512a700 1 --2- 172.21.15.77:0/2897673795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc76006c500 0x7fc760150b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:29.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.290+0000 7fc76512a700 1 -- 172.21.15.77:0/2897673795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc76006bfb0 msgr2=0x7fc7601544f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:29.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.290+0000 7fc76512a700 1 --2- 172.21.15.77:0/2897673795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc76006bfb0 0x7fc7601544f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:29.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.290+0000 7fc76512a700 1 -- 172.21.15.77:0/2897673795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc758009ae0 con 0x7fc76006bbe0 2022-04-03T19:46:29.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.290+0000 7fc764929700 1 --2- 172.21.15.77:0/2897673795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc76006bfb0 0x7fc7601544f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:46:29.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.291+0000 7fc76512a700 1 --2- 172.21.15.77:0/2897673795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc76006bbe0 0x7fc760153fb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc75c00d890 tx=0x7fc75c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:29.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.291+0000 7fc74d7fa700 1 -- 172.21.15.77:0/2897673795 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75c004030 con 0x7fc76006bbe0 2022-04-03T19:46:29.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.291+0000 7fc74d7fa700 1 -- 172.21.15.77:0/2897673795 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc75c004190 con 0x7fc76006bbe0 2022-04-03T19:46:29.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.291+0000 7fc76612c700 1 -- 172.21.15.77:0/2897673795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc760165430 con 0x7fc76006bbe0 2022-04-03T19:46:29.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.291+0000 7fc74d7fa700 1 -- 172.21.15.77:0/2897673795 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75c0125f0 con 0x7fc76006bbe0 2022-04-03T19:46:29.302 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.291+0000 7fc76612c700 1 -- 172.21.15.77:0/2897673795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7601659f0 con 0x7fc76006bbe0 2022-04-03T19:46:29.302 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.292+0000 7fc76612c700 1 -- 172.21.15.77:0/2897673795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7600b1720 con 0x7fc76006bbe0 2022-04-03T19:46:29.302 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.293+0000 7fc74d7fa700 1 -- 172.21.15.77:0/2897673795 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fc75c0129a0 con 0x7fc76006bbe0 2022-04-03T19:46:29.303 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.294+0000 7fc74d7fa700 1 --2- 172.21.15.77:0/2897673795 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc740063ce0 0x7fc740066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:29.303 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.294+0000 7fc74d7fa700 1 -- 172.21.15.77:0/2897673795 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6144+0+0 (secure 0 0 0) 0x7fc75c080b10 con 0x7fc76006bbe0 2022-04-03T19:46:29.304 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.294+0000 7fc764929700 1 --2- 172.21.15.77:0/2897673795 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc740063ce0 0x7fc740066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:29.304 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.295+0000 7fc764929700 1 --2- 172.21.15.77:0/2897673795 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc740063ce0 0x7fc740066190 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc76004b8b0 tx=0x7fc750009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:29.304 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.299+0000 7fc74d7fa700 1 -- 172.21.15.77:0/2897673795 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fc75c04f5d0 con 0x7fc76006bbe0 2022-04-03T19:46:29.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.529+0000 7fc76612c700 1 -- 172.21.15.77:0/2897673795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc76004b050 con 0x7fc76006bbe0 2022-04-03T19:46:29.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.529+0000 7fc74d7fa700 1 -- 172.21.15.77:0/2897673795 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6107 (secure 0 0 0) 0x7fc75c052bf0 con 0x7fc76006bbe0 2022-04-03T19:46:29.531 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:46:29.531 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4303,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:44:27.295743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4327":{"gid":4327,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6837/3665969042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3665969042},{"type":"v1","addr":"172.21.15.77:6837","nonce":3665969042}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:46:29.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.532+0000 7fc76612c700 1 -- 172.21.15.77:0/2897673795 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc740063ce0 msgr2=0x7fc740066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:29.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.532+0000 7fc76612c700 1 --2- 172.21.15.77:0/2897673795 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc740063ce0 0x7fc740066190 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc76004b8b0 tx=0x7fc750009040 comp rx=0 tx=0).stop 2022-04-03T19:46:29.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.532+0000 7fc76612c700 1 -- 172.21.15.77:0/2897673795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc76006bbe0 msgr2=0x7fc760153fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:29.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.532+0000 7fc76612c700 1 --2- 172.21.15.77:0/2897673795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc76006bbe0 0x7fc760153fb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc75c00d890 tx=0x7fc75c008040 comp rx=0 tx=0).stop 2022-04-03T19:46:29.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.532+0000 7fc76612c700 1 -- 172.21.15.77:0/2897673795 shutdown_connections 2022-04-03T19:46:29.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.532+0000 7fc76612c700 1 --2- 172.21.15.77:0/2897673795 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc740063ce0 0x7fc740066190 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:29.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.532+0000 7fc76612c700 1 --2- 172.21.15.77:0/2897673795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc76006c500 0x7fc760150b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:29.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.532+0000 7fc76612c700 1 --2- 172.21.15.77:0/2897673795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc76006bbe0 0x7fc760153fb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:29.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.532+0000 7fc76612c700 1 --2- 172.21.15.77:0/2897673795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc76006bfb0 0x7fc7601544f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:29.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.532+0000 7fc76612c700 1 -- 172.21.15.77:0/2897673795 >> 172.21.15.77:0/2897673795 conn(0x7fc7600cb6b0 msgr2=0x7fc76014b480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:29.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.532+0000 7fc76612c700 1 -- 172.21.15.77:0/2897673795 shutdown_connections 2022-04-03T19:46:29.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:29.532+0000 7fc76612c700 1 -- 172.21.15.77:0/2897673795 wait complete. 2022-04-03T19:46:29.536 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2022-04-03T19:46:29.557 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:46:31.558 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:46:31.763 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.761+0000 7f20f2911700 1 Processor -- start 2022-04-03T19:46:31.763 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.761+0000 7f20f2911700 1 -- start start 2022-04-03T19:46:31.765 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.763+0000 7f20f2911700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f20ec06bbe0 0x7f20ec1584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:31.765 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.763+0000 7f20f2911700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20ec06bfb0 0x7f20ec06d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:31.766 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.763+0000 7f20f2911700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ec06d260 con 0x7f20ec06c500 2022-04-03T19:46:31.766 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.763+0000 7f20f2911700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20ec159450 con 0x7f20ec06bbe0 2022-04-03T19:46:31.766 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.763+0000 7f20f2911700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec0c6330 con 0x7f20ec06bfb0 2022-04-03T19:46:31.766 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.763+0000 7f20f190f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f20ec06bbe0 0x7f20ec1584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:31.766 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.763+0000 7f20f190f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f20ec06bbe0 0x7f20ec1584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42734/0 (socket says 172.21.15.77:42734) 2022-04-03T19:46:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.763+0000 7f20f190f700 1 -- 172.21.15.77:0/120237724 learned_addr learned my addr 172.21.15.77:0/120237724 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.763+0000 7f20f2110700 1 --1- 172.21.15.77:0/120237724 >> v1:172.21.15.77:6789/0 conn(0x7f20ec06c500 0x7f20ec14eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38388/0 (socket says 172.21.15.77:38388) 2022-04-03T19:46:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.763+0000 7f20f110e700 1 --2- 172.21.15.77:0/120237724 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20ec06bfb0 0x7f20ec06d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.763+0000 7f20f190f700 1 -- 172.21.15.77:0/120237724 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20ec06bfb0 msgr2=0x7f20ec06d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:46:31.767 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.763+0000 7f20f190f700 1 --2- 172.21.15.77:0/120237724 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20ec06bfb0 0x7f20ec06d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.763+0000 7f20f190f700 1 -- 172.21.15.77:0/120237724 >> v1:172.21.15.77:6789/0 conn(0x7f20ec06c500 legacy=0x7f20ec14eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.763+0000 7f20f190f700 1 -- 172.21.15.77:0/120237724 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ec158bb0 con 0x7f20ec06bbe0 2022-04-03T19:46:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.764+0000 7f20f190f700 1 --2- 172.21.15.77:0/120237724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20ec06bbe0 0x7f20ec1584b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f20e8006fd0 tx=0x7f20e80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8671b32f0b912e09 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:31.768 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.764+0000 7f20f090d700 1 -- 172.21.15.77:0/120237724 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e8004030 con 0x7f20ec06bbe0 2022-04-03T19:46:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.764+0000 7f20f090d700 1 -- 172.21.15.77:0/120237724 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e80041d0 con 0x7f20ec06bbe0 2022-04-03T19:46:31.769 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.764+0000 7f20f090d700 1 -- 172.21.15.77:0/120237724 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e8011630 con 0x7f20ec06bbe0 2022-04-03T19:46:31.772 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.764+0000 7f20f2911700 1 -- 172.21.15.77:0/120237724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20ec06bbe0 msgr2=0x7f20ec1584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:31.772 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.765+0000 7f20f2911700 1 --2- 172.21.15.77:0/120237724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20ec06bbe0 0x7f20ec1584b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f20e8006fd0 tx=0x7f20e80090f0 comp rx=0 tx=0).stop 2022-04-03T19:46:31.772 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.765+0000 7f20f2911700 1 -- 172.21.15.77:0/120237724 shutdown_connections 2022-04-03T19:46:31.773 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.765+0000 7f20f2911700 1 --2- 172.21.15.77:0/120237724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20ec06bbe0 0x7f20ec1584b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:31.773 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.765+0000 7f20f2911700 1 --2- 172.21.15.77:0/120237724 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20ec06bfb0 0x7f20ec06d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:31.773 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.765+0000 7f20f2911700 1 -- 172.21.15.77:0/120237724 >> 172.21.15.77:0/120237724 conn(0x7f20ec0cb6b0 msgr2=0x7f20ec0cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:31.773 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.765+0000 7f20f2911700 1 -- 172.21.15.77:0/120237724 shutdown_connections 2022-04-03T19:46:31.773 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.765+0000 7f20f2911700 1 -- 172.21.15.77:0/120237724 wait complete. 2022-04-03T19:46:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.766+0000 7f20f2911700 1 Processor -- start 2022-04-03T19:46:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.766+0000 7f20f2911700 1 -- start start 2022-04-03T19:46:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.767+0000 7f20f2911700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20ec06bbe0 0x7f20ec1652b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.767+0000 7f20f2911700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20ec06bfb0 0x7f20ec169150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:31.774 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.767+0000 7f20f2911700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20ec06c500 0x7f20ec157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:31.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.767+0000 7f20f2911700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec0c1170 con 0x7f20ec06bbe0 2022-04-03T19:46:31.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.767+0000 7f20f2911700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec166190 con 0x7f20ec06bfb0 2022-04-03T19:46:31.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.767+0000 7f20f2911700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec168cc0 con 0x7f20ec06c500 2022-04-03T19:46:31.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.768+0000 7f20f190f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20ec06bbe0 0x7f20ec1652b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:31.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.768+0000 7f20f2110700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20ec06c500 0x7f20ec157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:31.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.768+0000 7f20f110e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20ec06bfb0 0x7f20ec169150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:31.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.768+0000 7f20f110e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20ec06bfb0 0x7f20ec169150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37930/0 (socket says 172.21.15.77:37930) 2022-04-03T19:46:31.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.768+0000 7f20f110e700 1 -- 172.21.15.77:0/2409976581 learned_addr learned my addr 172.21.15.77:0/2409976581 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:31.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.768+0000 7f20f110e700 1 -- 172.21.15.77:0/2409976581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20ec06c500 msgr2=0x7f20ec157a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:31.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.768+0000 7f20f110e700 1 --2- 172.21.15.77:0/2409976581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20ec06c500 0x7f20ec157a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:31.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.768+0000 7f20f110e700 1 -- 172.21.15.77:0/2409976581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20ec06bbe0 msgr2=0x7f20ec1652b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:31.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.768+0000 7f20f110e700 1 --2- 172.21.15.77:0/2409976581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20ec06bbe0 0x7f20ec1652b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:31.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.768+0000 7f20f110e700 1 -- 172.21.15.77:0/2409976581 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e8009370 con 0x7f20ec06bfb0 2022-04-03T19:46:31.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.768+0000 7f20f190f700 1 --2- 172.21.15.77:0/2409976581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20ec06bbe0 0x7f20ec1652b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:46:31.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.768+0000 7f20f110e700 1 --2- 172.21.15.77:0/2409976581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20ec06bfb0 0x7f20ec169150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f20dc00e960 tx=0x7f20dc009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:31.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.769+0000 7f20d9ffb700 1 -- 172.21.15.77:0/2409976581 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20dc013680 con 0x7f20ec06bfb0 2022-04-03T19:46:31.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.769+0000 7f20d9ffb700 1 -- 172.21.15.77:0/2409976581 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20dc0137e0 con 0x7f20ec06bfb0 2022-04-03T19:46:31.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.769+0000 7f20d9ffb700 1 -- 172.21.15.77:0/2409976581 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20dc013b60 con 0x7f20ec06bfb0 2022-04-03T19:46:31.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.769+0000 7f20f2911700 1 -- 172.21.15.77:0/2409976581 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ec1658f0 con 0x7f20ec06bfb0 2022-04-03T19:46:31.779 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.769+0000 7f20f2911700 1 -- 172.21.15.77:0/2409976581 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ec165eb0 con 0x7f20ec06bfb0 2022-04-03T19:46:31.779 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.770+0000 7f20f2911700 1 -- 172.21.15.77:0/2409976581 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20ec0b1720 con 0x7f20ec06bfb0 2022-04-03T19:46:31.779 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.771+0000 7f20d9ffb700 1 -- 172.21.15.77:0/2409976581 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f20dc023070 con 0x7f20ec06bfb0 2022-04-03T19:46:31.780 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.772+0000 7f20d9ffb700 1 --2- 172.21.15.77:0/2409976581 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f20cc063ce0 0x7f20cc066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:31.780 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.772+0000 7f20d9ffb700 1 -- 172.21.15.77:0/2409976581 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f20dc080b90 con 0x7f20ec06bfb0 2022-04-03T19:46:31.780 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.772+0000 7f20f190f700 1 --2- 172.21.15.77:0/2409976581 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f20cc063ce0 0x7f20cc066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:31.781 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.773+0000 7f20f190f700 1 --2- 172.21.15.77:0/2409976581 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f20cc063ce0 0x7f20cc066190 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f20e8009340 tx=0x7f20e8016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:31.781 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:31.776+0000 7f20d9ffb700 1 -- 172.21.15.77:0/2409976581 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f20dc04f680 con 0x7f20ec06bfb0 2022-04-03T19:46:32.005 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:32.004+0000 7f20f2911700 1 -- 172.21.15.77:0/2409976581 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20ec04b050 con 0x7f20ec06bfb0 2022-04-03T19:46:32.006 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:32.005+0000 7f20d9ffb700 1 -- 172.21.15.77:0/2409976581 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6107 (secure 0 0 0) 0x7f20dc052ca0 con 0x7f20ec06bfb0 2022-04-03T19:46:32.006 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:46:32.007 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4303,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:44:27.295743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4327":{"gid":4327,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6837/3665969042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3665969042},{"type":"v1","addr":"172.21.15.77:6837","nonce":3665969042}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:46:32.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:32.007+0000 7f20f2911700 1 -- 172.21.15.77:0/2409976581 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f20cc063ce0 msgr2=0x7f20cc066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:32.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:32.007+0000 7f20f2911700 1 --2- 172.21.15.77:0/2409976581 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f20cc063ce0 0x7f20cc066190 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f20e8009340 tx=0x7f20e8016000 comp rx=0 tx=0).stop 2022-04-03T19:46:32.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:32.007+0000 7f20f2911700 1 -- 172.21.15.77:0/2409976581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20ec06bfb0 msgr2=0x7f20ec169150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:32.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:32.007+0000 7f20f2911700 1 --2- 172.21.15.77:0/2409976581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20ec06bfb0 0x7f20ec169150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f20dc00e960 tx=0x7f20dc009060 comp rx=0 tx=0).stop 2022-04-03T19:46:32.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:32.007+0000 7f20f2911700 1 -- 172.21.15.77:0/2409976581 shutdown_connections 2022-04-03T19:46:32.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:32.007+0000 7f20f2911700 1 --2- 172.21.15.77:0/2409976581 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f20cc063ce0 0x7f20cc066190 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:32.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:32.007+0000 7f20f2911700 1 --2- 172.21.15.77:0/2409976581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f20ec06c500 0x7f20ec157a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:32.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:32.007+0000 7f20f2911700 1 --2- 172.21.15.77:0/2409976581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20ec06bbe0 0x7f20ec1652b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:32.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:32.007+0000 7f20f2911700 1 --2- 172.21.15.77:0/2409976581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20ec06bfb0 0x7f20ec169150 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:32.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:32.007+0000 7f20f2911700 1 -- 172.21.15.77:0/2409976581 >> 172.21.15.77:0/2409976581 conn(0x7f20ec0cb6b0 msgr2=0x7f20ec149ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:32.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:32.007+0000 7f20f2911700 1 -- 172.21.15.77:0/2409976581 shutdown_connections 2022-04-03T19:46:32.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:32.007+0000 7f20f2911700 1 -- 172.21.15.77:0/2409976581 wait complete. 2022-04-03T19:46:32.012 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2022-04-03T19:46:32.033 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:46:34.034 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:46:34.244 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.242+0000 7fe17930f700 1 Processor -- start 2022-04-03T19:46:34.244 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.243+0000 7fe17930f700 1 -- start start 2022-04-03T19:46:34.247 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.244+0000 7fe17930f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe17406bbe0 0x7fe1741584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:34.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.244+0000 7fe17930f700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe17406bfb0 0x7fe17406d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:34.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.244+0000 7fe17930f700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe17406d260 con 0x7fe17406c500 2022-04-03T19:46:34.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.244+0000 7fe17930f700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe174159450 con 0x7fe17406bbe0 2022-04-03T19:46:34.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.244+0000 7fe17930f700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1740c6330 con 0x7fe17406bfb0 2022-04-03T19:46:34.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.244+0000 7fe173fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe17406bbe0 0x7fe1741584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:34.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.244+0000 7fe1737fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe17406bfb0 0x7fe17406d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:34.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.244+0000 7fe173fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe17406bbe0 0x7fe1741584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42738/0 (socket says 172.21.15.77:42738) 2022-04-03T19:46:34.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.244+0000 7fe1737fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe17406bfb0 0x7fe17406d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37932/0 (socket says 172.21.15.77:37932) 2022-04-03T19:46:34.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.244+0000 7fe173fff700 1 -- 172.21.15.77:0/3014185728 learned_addr learned my addr 172.21.15.77:0/3014185728 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:34.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.244+0000 7fe178b0e700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe17406c500 0x7fe17414eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38390/0 (socket says 172.21.15.77:38390) 2022-04-03T19:46:34.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.245+0000 7fe173fff700 1 -- 172.21.15.77:0/3014185728 >> v2:172.21.15.186:3300/0 conn(0x7fe17406bfb0 msgr2=0x7fe17406d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:46:34.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.245+0000 7fe173fff700 1 --2- 172.21.15.77:0/3014185728 >> v2:172.21.15.186:3300/0 conn(0x7fe17406bfb0 0x7fe17406d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:34.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.245+0000 7fe173fff700 1 -- 172.21.15.77:0/3014185728 >> v1:172.21.15.77:6789/0 conn(0x7fe17406c500 legacy=0x7fe17414eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:34.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.245+0000 7fe173fff700 1 -- 172.21.15.77:0/3014185728 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe174158bb0 con 0x7fe17406bbe0 2022-04-03T19:46:34.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.245+0000 7fe173fff700 1 --2- 172.21.15.77:0/3014185728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe17406bbe0 0x7fe1741584b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe1740898c0 tx=0x7fe1640090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=270b85efaedef114 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:34.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.245+0000 7fe172ffd700 1 -- 172.21.15.77:0/3014185728 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe164004030 con 0x7fe17406bbe0 2022-04-03T19:46:34.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.245+0000 7fe172ffd700 1 -- 172.21.15.77:0/3014185728 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe164004190 con 0x7fe17406bbe0 2022-04-03T19:46:34.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.245+0000 7fe172ffd700 1 -- 172.21.15.77:0/3014185728 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe164011630 con 0x7fe17406bbe0 2022-04-03T19:46:34.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.246+0000 7fe17930f700 1 -- 172.21.15.77:0/3014185728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe17406bbe0 msgr2=0x7fe1741584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:34.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.246+0000 7fe17930f700 1 --2- 172.21.15.77:0/3014185728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe17406bbe0 0x7fe1741584b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe1740898c0 tx=0x7fe1640090f0 comp rx=0 tx=0).stop 2022-04-03T19:46:34.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.246+0000 7fe17930f700 1 -- 172.21.15.77:0/3014185728 shutdown_connections 2022-04-03T19:46:34.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.246+0000 7fe17930f700 1 --2- 172.21.15.77:0/3014185728 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe17406bbe0 0x7fe1741584b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:34.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.246+0000 7fe17930f700 1 --2- 172.21.15.77:0/3014185728 >> v2:172.21.15.186:3300/0 conn(0x7fe17406bfb0 0x7fe17406d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:34.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.246+0000 7fe17930f700 1 -- 172.21.15.77:0/3014185728 >> 172.21.15.77:0/3014185728 conn(0x7fe1740cb6b0 msgr2=0x7fe1740cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:34.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.246+0000 7fe17930f700 1 -- 172.21.15.77:0/3014185728 shutdown_connections 2022-04-03T19:46:34.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.246+0000 7fe17930f700 1 -- 172.21.15.77:0/3014185728 wait complete. 2022-04-03T19:46:34.256 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.247+0000 7fe17930f700 1 Processor -- start 2022-04-03T19:46:34.257 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.248+0000 7fe17930f700 1 -- start start 2022-04-03T19:46:34.257 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.248+0000 7fe17930f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe17406bbe0 0x7fe174151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:34.257 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.248+0000 7fe17930f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe17406bfb0 0x7fe174163080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:34.257 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.248+0000 7fe17930f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe17406c500 0x7fe174166560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:34.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.248+0000 7fe17930f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe174155730 con 0x7fe17406c500 2022-04-03T19:46:34.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.248+0000 7fe17930f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe174168390 con 0x7fe17406bfb0 2022-04-03T19:46:34.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.248+0000 7fe17930f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe174167010 con 0x7fe17406bbe0 2022-04-03T19:46:34.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.249+0000 7fe178b0e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe17406c500 0x7fe174166560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:34.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.249+0000 7fe173fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe17406bbe0 0x7fe174151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:34.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.249+0000 7fe178b0e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe17406c500 0x7fe174166560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42740/0 (socket says 172.21.15.77:42740) 2022-04-03T19:46:34.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.249+0000 7fe178b0e700 1 -- 172.21.15.77:0/1920483515 learned_addr learned my addr 172.21.15.77:0/1920483515 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:34.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.249+0000 7fe1737fe700 1 --2- 172.21.15.77:0/1920483515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe17406bfb0 0x7fe174163080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:34.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.249+0000 7fe178b0e700 1 -- 172.21.15.77:0/1920483515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe17406bbe0 msgr2=0x7fe174151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:34.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.249+0000 7fe178b0e700 1 --2- 172.21.15.77:0/1920483515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe17406bbe0 0x7fe174151d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:34.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.249+0000 7fe178b0e700 1 -- 172.21.15.77:0/1920483515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe17406bfb0 msgr2=0x7fe174163080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:34.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.249+0000 7fe178b0e700 1 --2- 172.21.15.77:0/1920483515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe17406bfb0 0x7fe174163080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:34.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.249+0000 7fe178b0e700 1 -- 172.21.15.77:0/1920483515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe164009370 con 0x7fe17406c500 2022-04-03T19:46:34.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.249+0000 7fe1737fe700 1 --2- 172.21.15.77:0/1920483515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe17406bfb0 0x7fe174163080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:46:34.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.250+0000 7fe178b0e700 1 --2- 172.21.15.77:0/1920483515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe17406c500 0x7fe174166560 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe16c00c790 tx=0x7fe16c0015e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:34.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.250+0000 7fe15bfff700 1 -- 172.21.15.77:0/1920483515 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c005c70 con 0x7fe17406c500 2022-04-03T19:46:34.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.250+0000 7fe15bfff700 1 -- 172.21.15.77:0/1920483515 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe16c005dd0 con 0x7fe17406c500 2022-04-03T19:46:34.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.250+0000 7fe15bfff700 1 -- 172.21.15.77:0/1920483515 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c007300 con 0x7fe17406c500 2022-04-03T19:46:34.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.250+0000 7fe17930f700 1 -- 172.21.15.77:0/1920483515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1741672f0 con 0x7fe17406c500 2022-04-03T19:46:34.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.250+0000 7fe17930f700 1 -- 172.21.15.77:0/1920483515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1741678b0 con 0x7fe17406c500 2022-04-03T19:46:34.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.251+0000 7fe17930f700 1 -- 172.21.15.77:0/1920483515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe17404b050 con 0x7fe17406c500 2022-04-03T19:46:34.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.252+0000 7fe15bfff700 1 -- 172.21.15.77:0/1920483515 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fe16c010070 con 0x7fe17406c500 2022-04-03T19:46:34.263 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.253+0000 7fe15bfff700 1 --2- 172.21.15.77:0/1920483515 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe154063ce0 0x7fe154066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:34.263 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.253+0000 7fe15bfff700 1 -- 172.21.15.77:0/1920483515 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6144+0+0 (secure 0 0 0) 0x7fe16c0802a0 con 0x7fe17406c500 2022-04-03T19:46:34.263 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.253+0000 7fe173fff700 1 --2- 172.21.15.77:0/1920483515 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe154063ce0 0x7fe154066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:34.263 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.254+0000 7fe173fff700 1 --2- 172.21.15.77:0/1920483515 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe154063ce0 0x7fe154066190 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe1740898c0 tx=0x7fe164016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:34.264 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.257+0000 7fe15bfff700 1 -- 172.21.15.77:0/1920483515 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fe16c04ee30 con 0x7fe17406c500 2022-04-03T19:46:34.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.487+0000 7fe17930f700 1 -- 172.21.15.77:0/1920483515 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe174063d10 con 0x7fe17406c500 2022-04-03T19:46:34.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.488+0000 7fe15bfff700 1 -- 172.21.15.77:0/1920483515 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6107 (secure 0 0 0) 0x7fe16c052450 con 0x7fe17406c500 2022-04-03T19:46:34.490 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:46:34.490 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4303,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:44:27.295743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4327":{"gid":4327,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6837/3665969042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3665969042},{"type":"v1","addr":"172.21.15.77:6837","nonce":3665969042}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:46:34.492 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.490+0000 7fe17930f700 1 -- 172.21.15.77:0/1920483515 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe154063ce0 msgr2=0x7fe154066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:34.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.490+0000 7fe17930f700 1 --2- 172.21.15.77:0/1920483515 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe154063ce0 0x7fe154066190 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe1740898c0 tx=0x7fe164016000 comp rx=0 tx=0).stop 2022-04-03T19:46:34.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.490+0000 7fe17930f700 1 -- 172.21.15.77:0/1920483515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe17406c500 msgr2=0x7fe174166560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:34.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.490+0000 7fe17930f700 1 --2- 172.21.15.77:0/1920483515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe17406c500 0x7fe174166560 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe16c00c790 tx=0x7fe16c0015e0 comp rx=0 tx=0).stop 2022-04-03T19:46:34.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.491+0000 7fe17930f700 1 -- 172.21.15.77:0/1920483515 shutdown_connections 2022-04-03T19:46:34.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.491+0000 7fe17930f700 1 --2- 172.21.15.77:0/1920483515 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe17406c500 0x7fe174166560 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:34.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.491+0000 7fe17930f700 1 --2- 172.21.15.77:0/1920483515 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe154063ce0 0x7fe154066190 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:34.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.491+0000 7fe17930f700 1 --2- 172.21.15.77:0/1920483515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe17406bbe0 0x7fe174151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:34.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.491+0000 7fe17930f700 1 --2- 172.21.15.77:0/1920483515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe17406bfb0 0x7fe174163080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:34.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.491+0000 7fe17930f700 1 -- 172.21.15.77:0/1920483515 >> 172.21.15.77:0/1920483515 conn(0x7fe1740cb6b0 msgr2=0x7fe1740c5050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:34.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.491+0000 7fe17930f700 1 -- 172.21.15.77:0/1920483515 shutdown_connections 2022-04-03T19:46:34.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:34.491+0000 7fe17930f700 1 -- 172.21.15.77:0/1920483515 wait complete. 2022-04-03T19:46:34.496 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2022-04-03T19:46:34.516 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:46:36.517 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:46:36.724 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.722+0000 7f114eb4d700 1 Processor -- start 2022-04-03T19:46:36.724 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.722+0000 7f114eb4d700 1 -- start start 2022-04-03T19:46:36.727 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.723+0000 7f114eb4d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f11480c5930 0x7f1148155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:36.727 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.723+0000 7f114eb4d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1148156120 0x7f114806c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:36.727 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.723+0000 7f114eb4d700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11480c2f80 con 0x7f114806d3f0 2022-04-03T19:46:36.728 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.723+0000 7f114eb4d700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11480c3260 con 0x7f11480c5930 2022-04-03T19:46:36.728 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.723+0000 7f114eb4d700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11480c3540 con 0x7f1148156120 2022-04-03T19:46:36.728 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.724+0000 7f114d34a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f11480c5930 0x7f1148155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:36.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.724+0000 7f114d34a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f11480c5930 0x7f1148155be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42742/0 (socket says 172.21.15.77:42742) 2022-04-03T19:46:36.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.724+0000 7f114d34a700 1 -- 172.21.15.77:0/1996471413 learned_addr learned my addr 172.21.15.77:0/1996471413 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:36.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.724+0000 7f114e34c700 1 --2- 172.21.15.77:0/1996471413 >> v2:172.21.15.186:3300/0 conn(0x7f1148156120 0x7f114806c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:36.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.724+0000 7f114d34a700 1 -- 172.21.15.77:0/1996471413 >> v2:172.21.15.186:3300/0 conn(0x7f1148156120 msgr2=0x7f114806c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:46:36.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.724+0000 7f114d34a700 1 --2- 172.21.15.77:0/1996471413 >> v2:172.21.15.186:3300/0 conn(0x7f1148156120 0x7f114806c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:36.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.724+0000 7f114d34a700 1 -- 172.21.15.77:0/1996471413 >> v1:172.21.15.186:6789/0 conn(0x7f114806d3f0 legacy=0x7f114806d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:36.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.724+0000 7f114d34a700 1 -- 172.21.15.77:0/1996471413 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11480c3820 con 0x7f11480c5930 2022-04-03T19:46:36.731 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.725+0000 7f114d34a700 1 --2- 172.21.15.77:0/1996471413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11480c5930 0x7f1148155be0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1138006fd0 tx=0x7f11380090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=695a1d4b110b8cfd server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:36.731 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.725+0000 7f114cb49700 1 -- 172.21.15.77:0/1996471413 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1138004030 con 0x7f11480c5930 2022-04-03T19:46:36.731 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.725+0000 7f114cb49700 1 -- 172.21.15.77:0/1996471413 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1138004190 con 0x7f11480c5930 2022-04-03T19:46:36.731 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.725+0000 7f114cb49700 1 -- 172.21.15.77:0/1996471413 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1138011630 con 0x7f11480c5930 2022-04-03T19:46:36.732 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.725+0000 7f114eb4d700 1 -- 172.21.15.77:0/1996471413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11480c5930 msgr2=0x7f1148155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:36.732 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.725+0000 7f114eb4d700 1 --2- 172.21.15.77:0/1996471413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11480c5930 0x7f1148155be0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1138006fd0 tx=0x7f11380090f0 comp rx=0 tx=0).stop 2022-04-03T19:46:36.732 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.725+0000 7f114eb4d700 1 -- 172.21.15.77:0/1996471413 shutdown_connections 2022-04-03T19:46:36.732 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.725+0000 7f114eb4d700 1 --2- 172.21.15.77:0/1996471413 >> v2:172.21.15.186:3300/0 conn(0x7f1148156120 0x7f114806c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:36.732 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.725+0000 7f114eb4d700 1 --2- 172.21.15.77:0/1996471413 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11480c5930 0x7f1148155be0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:36.733 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.725+0000 7f114eb4d700 1 -- 172.21.15.77:0/1996471413 >> 172.21.15.77:0/1996471413 conn(0x7f11480cb720 msgr2=0x7f11480cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:36.733 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.726+0000 7f114eb4d700 1 -- 172.21.15.77:0/1996471413 shutdown_connections 2022-04-03T19:46:36.733 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.726+0000 7f114eb4d700 1 -- 172.21.15.77:0/1996471413 wait complete. 2022-04-03T19:46:36.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.727+0000 7f114eb4d700 1 Processor -- start 2022-04-03T19:46:36.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.727+0000 7f114eb4d700 1 -- start start 2022-04-03T19:46:36.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.728+0000 7f114eb4d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f114806d3f0 0x7f11481573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:36.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.728+0000 7f114eb4d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11480c5930 0x7f1148157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:36.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.728+0000 7f114eb4d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1148156120 0x7f11480c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:36.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.728+0000 7f114eb4d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1148156f40 con 0x7f1148156120 2022-04-03T19:46:36.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.728+0000 7f114eb4d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1148156c60 con 0x7f114806d3f0 2022-04-03T19:46:36.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.728+0000 7f114eb4d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1148158e10 con 0x7f11480c5930 2022-04-03T19:46:36.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.728+0000 7f114d34a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11480c5930 0x7f1148157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:36.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.728+0000 7f114db4b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f114806d3f0 0x7f11481573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:36.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.728+0000 7f114db4b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f114806d3f0 0x7f11481573d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37938/0 (socket says 172.21.15.77:37938) 2022-04-03T19:46:36.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.728+0000 7f114db4b700 1 -- 172.21.15.77:0/452787210 learned_addr learned my addr 172.21.15.77:0/452787210 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:36.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.728+0000 7f114d34a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11480c5930 0x7f1148157910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60048/0 (socket says 172.21.15.77:60048) 2022-04-03T19:46:36.740 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.728+0000 7f114e34c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1148156120 0x7f11480c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:36.740 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.729+0000 7f114db4b700 1 -- 172.21.15.77:0/452787210 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11480c5930 msgr2=0x7f1148157910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:36.740 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.729+0000 7f114db4b700 1 --2- 172.21.15.77:0/452787210 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11480c5930 0x7f1148157910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:36.740 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.729+0000 7f114db4b700 1 -- 172.21.15.77:0/452787210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1148156120 msgr2=0x7f11480c18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:36.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.729+0000 7f114db4b700 1 --2- 172.21.15.77:0/452787210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1148156120 0x7f11480c18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:36.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.729+0000 7f114db4b700 1 -- 172.21.15.77:0/452787210 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1138009370 con 0x7f114806d3f0 2022-04-03T19:46:36.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.729+0000 7f114e34c700 1 --2- 172.21.15.77:0/452787210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1148156120 0x7f11480c18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:46:36.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.729+0000 7f114db4b700 1 --2- 172.21.15.77:0/452787210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f114806d3f0 0x7f11481573d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f114400c520 tx=0x7f1144007e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:36.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.729+0000 7f1135ffb700 1 -- 172.21.15.77:0/452787210 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11440050d0 con 0x7f114806d3f0 2022-04-03T19:46:36.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.730+0000 7f1135ffb700 1 -- 172.21.15.77:0/452787210 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1144005230 con 0x7f114806d3f0 2022-04-03T19:46:36.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.730+0000 7f1135ffb700 1 -- 172.21.15.77:0/452787210 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11440117e0 con 0x7f114806d3f0 2022-04-03T19:46:36.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.730+0000 7f114eb4d700 1 -- 172.21.15.77:0/452787210 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1148159090 con 0x7f114806d3f0 2022-04-03T19:46:36.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.730+0000 7f114eb4d700 1 -- 172.21.15.77:0/452787210 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1148152680 con 0x7f114806d3f0 2022-04-03T19:46:36.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.731+0000 7f114eb4d700 1 -- 172.21.15.77:0/452787210 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f114804b050 con 0x7f114806d3f0 2022-04-03T19:46:36.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.732+0000 7f1135ffb700 1 -- 172.21.15.77:0/452787210 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f1144019030 con 0x7f114806d3f0 2022-04-03T19:46:36.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.732+0000 7f1135ffb700 1 --2- 172.21.15.77:0/452787210 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1128063ce0 0x7f1128066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:36.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.733+0000 7f1135ffb700 1 -- 172.21.15.77:0/452787210 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f114407f9d0 con 0x7f114806d3f0 2022-04-03T19:46:36.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.733+0000 7f114d34a700 1 --2- 172.21.15.77:0/452787210 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1128063ce0 0x7f1128066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:36.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.733+0000 7f114d34a700 1 --2- 172.21.15.77:0/452787210 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1128063ce0 0x7f1128066190 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1138006fd0 tx=0x7f1138013000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:36.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.737+0000 7f1135ffb700 1 -- 172.21.15.77:0/452787210 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f114404e560 con 0x7f114806d3f0 2022-04-03T19:46:36.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.974+0000 7f114eb4d700 1 -- 172.21.15.77:0/452787210 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f114806e1e0 con 0x7f114806d3f0 2022-04-03T19:46:36.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.976+0000 7f1135ffb700 1 -- 172.21.15.77:0/452787210 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6107 (secure 0 0 0) 0x7f1144017020 con 0x7f114806d3f0 2022-04-03T19:46:36.977 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:46:36.977 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4303,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:44:27.295743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4327":{"gid":4327,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6837/3665969042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3665969042},{"type":"v1","addr":"172.21.15.77:6837","nonce":3665969042}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:46:36.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.977+0000 7f114eb4d700 1 -- 172.21.15.77:0/452787210 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1128063ce0 msgr2=0x7f1128066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:36.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.977+0000 7f114eb4d700 1 --2- 172.21.15.77:0/452787210 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1128063ce0 0x7f1128066190 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1138006fd0 tx=0x7f1138013000 comp rx=0 tx=0).stop 2022-04-03T19:46:36.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.977+0000 7f114eb4d700 1 -- 172.21.15.77:0/452787210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f114806d3f0 msgr2=0x7f11481573d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:36.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.977+0000 7f114eb4d700 1 --2- 172.21.15.77:0/452787210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f114806d3f0 0x7f11481573d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f114400c520 tx=0x7f1144007e00 comp rx=0 tx=0).stop 2022-04-03T19:46:36.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.978+0000 7f114eb4d700 1 -- 172.21.15.77:0/452787210 shutdown_connections 2022-04-03T19:46:36.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.978+0000 7f114eb4d700 1 --2- 172.21.15.77:0/452787210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1148156120 0x7f11480c18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:36.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.978+0000 7f114eb4d700 1 --2- 172.21.15.77:0/452787210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f114806d3f0 0x7f11481573d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:36.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.978+0000 7f114eb4d700 1 --2- 172.21.15.77:0/452787210 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1128063ce0 0x7f1128066190 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:36.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.978+0000 7f114eb4d700 1 --2- 172.21.15.77:0/452787210 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11480c5930 0x7f1148157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:36.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.978+0000 7f114eb4d700 1 -- 172.21.15.77:0/452787210 >> 172.21.15.77:0/452787210 conn(0x7f11480cb720 msgr2=0x7f114806d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:36.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.978+0000 7f114eb4d700 1 -- 172.21.15.77:0/452787210 shutdown_connections 2022-04-03T19:46:36.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:36.978+0000 7f114eb4d700 1 -- 172.21.15.77:0/452787210 wait complete. 2022-04-03T19:46:36.983 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2022-04-03T19:46:37.003 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:46:39.004 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:46:39.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.213+0000 7f486f59e700 1 Processor -- start 2022-04-03T19:46:39.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.214+0000 7f486f59e700 1 -- start start 2022-04-03T19:46:39.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.215+0000 7f486f59e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f487006d3f0 0x7f487006d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:39.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.215+0000 7f486f59e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48701562f0 con 0x7f48700c2a90 2022-04-03T19:46:39.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.215+0000 7f486f59e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48700c2f80 con 0x7f48700c5930 2022-04-03T19:46:39.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.215+0000 7f486f59e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48700c3260 con 0x7f487006d3f0 2022-04-03T19:46:39.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.215+0000 7f486ed9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f48700c2a90 0x7f487006c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38392/0 (socket says 172.21.15.77:38392) 2022-04-03T19:46:39.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.215+0000 7f486ed9d700 1 -- 172.21.15.77:0/77685630 learned_addr learned my addr 172.21.15.77:0/77685630 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:46:39.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.215+0000 7f486e59c700 1 --2- 172.21.15.77:0/77685630 >> v2:172.21.15.186:3300/0 conn(0x7f487006d3f0 0x7f487006d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:39.219 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.216+0000 7f486e59c700 1 -- 172.21.15.77:0/77685630 >> v1:172.21.15.186:6789/0 conn(0x7f48700c5930 legacy=0x7f4870155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:39.219 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.216+0000 7f486e59c700 1 -- 172.21.15.77:0/77685630 >> v1:172.21.15.77:6789/0 conn(0x7f48700c2a90 legacy=0x7f487006c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:39.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.216+0000 7f486d59a700 1 -- 172.21.15.77:0/77685630 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958599648 0 0) 0x7f48701562f0 con 0x7f48700c2a90 2022-04-03T19:46:39.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.216+0000 7f486e59c700 1 -- 172.21.15.77:0/77685630 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48700c3540 con 0x7f487006d3f0 2022-04-03T19:46:39.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.216+0000 7f486e59c700 1 --2- 172.21.15.77:0/77685630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f487006d3f0 0x7f487006d7c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4860009790 tx=0x7f4860009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=277d8ca8a948e38 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:39.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.216+0000 7f486d59a700 1 -- 172.21.15.77:0/77685630 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860011630 con 0x7f487006d3f0 2022-04-03T19:46:39.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.216+0000 7f486d59a700 1 -- 172.21.15.77:0/77685630 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4860011790 con 0x7f487006d3f0 2022-04-03T19:46:39.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.216+0000 7f486d59a700 1 -- 172.21.15.77:0/77685630 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860011b30 con 0x7f487006d3f0 2022-04-03T19:46:39.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.217+0000 7f486f59e700 1 -- 172.21.15.77:0/77685630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f487006d3f0 msgr2=0x7f487006d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:39.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.217+0000 7f486f59e700 1 --2- 172.21.15.77:0/77685630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f487006d3f0 0x7f487006d7c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4860009790 tx=0x7f4860009060 comp rx=0 tx=0).stop 2022-04-03T19:46:39.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.217+0000 7f486f59e700 1 -- 172.21.15.77:0/77685630 shutdown_connections 2022-04-03T19:46:39.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.217+0000 7f486f59e700 1 --2- 172.21.15.77:0/77685630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f487006d3f0 0x7f487006d7c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:39.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.217+0000 7f486f59e700 1 -- 172.21.15.77:0/77685630 >> 172.21.15.77:0/77685630 conn(0x7f48700cb720 msgr2=0x7f48700cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:39.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.217+0000 7f486f59e700 1 -- 172.21.15.77:0/77685630 shutdown_connections 2022-04-03T19:46:39.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.217+0000 7f486f59e700 1 -- 172.21.15.77:0/77685630 wait complete. 2022-04-03T19:46:39.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.219+0000 7f486f59e700 1 Processor -- start 2022-04-03T19:46:39.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.219+0000 7f486f59e700 1 -- start start 2022-04-03T19:46:39.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.220+0000 7f486f59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f487006d3f0 0x7f4870151420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:39.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.220+0000 7f486f59e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48700c2a90 0x7f48700c1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:39.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.220+0000 7f486f59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48700c5930 0x7f48700c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:39.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.220+0000 7f486f59e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4870157510 con 0x7f487006d3f0 2022-04-03T19:46:39.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.220+0000 7f486f59e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4870169b60 con 0x7f48700c2a90 2022-04-03T19:46:39.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.220+0000 7f486f59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4870150f90 con 0x7f48700c5930 2022-04-03T19:46:39.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.220+0000 7f486e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f487006d3f0 0x7f4870151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:39.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.220+0000 7f486ed9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48700c5930 0x7f48700c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:39.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.220+0000 7f486e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f487006d3f0 0x7f4870151420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42746/0 (socket says 172.21.15.77:42746) 2022-04-03T19:46:39.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.220+0000 7f486e59c700 1 -- 172.21.15.77:0/1568669850 learned_addr learned my addr 172.21.15.77:0/1568669850 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:39.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.220+0000 7f486dd9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48700c2a90 0x7f48700c1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:39.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.221+0000 7f486e59c700 1 -- 172.21.15.77:0/1568669850 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48700c5930 msgr2=0x7f48700c21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:39.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.221+0000 7f486e59c700 1 --2- 172.21.15.77:0/1568669850 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48700c5930 0x7f48700c21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:39.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.221+0000 7f486e59c700 1 -- 172.21.15.77:0/1568669850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48700c2a90 msgr2=0x7f48700c1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:39.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.221+0000 7f486e59c700 1 --2- 172.21.15.77:0/1568669850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48700c2a90 0x7f48700c1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:39.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.221+0000 7f486e59c700 1 -- 172.21.15.77:0/1568669850 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48600092a0 con 0x7f487006d3f0 2022-04-03T19:46:39.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.221+0000 7f486ed9d700 1 --2- 172.21.15.77:0/1568669850 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48700c5930 0x7f48700c21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:46:39.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.221+0000 7f486e59c700 1 --2- 172.21.15.77:0/1568669850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f487006d3f0 0x7f4870151420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f48700890b0 tx=0x7f4860017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:39.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.221+0000 7f485e7fc700 1 -- 172.21.15.77:0/1568669850 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860000bc0 con 0x7f487006d3f0 2022-04-03T19:46:39.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.221+0000 7f485e7fc700 1 -- 172.21.15.77:0/1568669850 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4860011900 con 0x7f487006d3f0 2022-04-03T19:46:39.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.221+0000 7f486f59e700 1 -- 172.21.15.77:0/1568669850 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4870169880 con 0x7f487006d3f0 2022-04-03T19:46:39.230 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.221+0000 7f485e7fc700 1 -- 172.21.15.77:0/1568669850 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860025420 con 0x7f487006d3f0 2022-04-03T19:46:39.230 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.221+0000 7f486f59e700 1 -- 172.21.15.77:0/1568669850 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4870150c20 con 0x7f487006d3f0 2022-04-03T19:46:39.230 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.222+0000 7f486f59e700 1 -- 172.21.15.77:0/1568669850 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f487004b050 con 0x7f487006d3f0 2022-04-03T19:46:39.230 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.224+0000 7f485e7fc700 1 -- 172.21.15.77:0/1568669850 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f486002c030 con 0x7f487006d3f0 2022-04-03T19:46:39.231 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.225+0000 7f485e7fc700 1 --2- 172.21.15.77:0/1568669850 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f484c063ce0 0x7f484c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:39.231 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.225+0000 7f485e7fc700 1 -- 172.21.15.77:0/1568669850 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f4860082390 con 0x7f487006d3f0 2022-04-03T19:46:39.232 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.225+0000 7f486dd9b700 1 --2- 172.21.15.77:0/1568669850 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f484c063ce0 0x7f484c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:39.232 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.226+0000 7f486dd9b700 1 --2- 172.21.15.77:0/1568669850 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f484c063ce0 0x7f484c066190 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f487014e650 tx=0x7f4864009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:39.232 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.229+0000 7f485e7fc700 1 -- 172.21.15.77:0/1568669850 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f4860050ed0 con 0x7f487006d3f0 2022-04-03T19:46:39.461 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.461+0000 7f486f59e700 1 -- 172.21.15.77:0/1568669850 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f487006e1e0 con 0x7f487006d3f0 2022-04-03T19:46:39.462 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.462+0000 7f485e7fc700 1 -- 172.21.15.77:0/1568669850 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6107 (secure 0 0 0) 0x7f486002a090 con 0x7f487006d3f0 2022-04-03T19:46:39.463 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:46:39.463 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4303,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:44:27.295743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4327":{"gid":4327,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6837/3665969042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3665969042},{"type":"v1","addr":"172.21.15.77:6837","nonce":3665969042}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:46:39.465 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.464+0000 7f486f59e700 1 -- 172.21.15.77:0/1568669850 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f484c063ce0 msgr2=0x7f484c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:39.465 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.464+0000 7f486f59e700 1 --2- 172.21.15.77:0/1568669850 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f484c063ce0 0x7f484c066190 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f487014e650 tx=0x7f4864009040 comp rx=0 tx=0).stop 2022-04-03T19:46:39.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.464+0000 7f486f59e700 1 -- 172.21.15.77:0/1568669850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f487006d3f0 msgr2=0x7f4870151420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:39.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.464+0000 7f486f59e700 1 --2- 172.21.15.77:0/1568669850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f487006d3f0 0x7f4870151420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f48700890b0 tx=0x7f4860017000 comp rx=0 tx=0).stop 2022-04-03T19:46:39.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.464+0000 7f486f59e700 1 -- 172.21.15.77:0/1568669850 shutdown_connections 2022-04-03T19:46:39.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.464+0000 7f486f59e700 1 --2- 172.21.15.77:0/1568669850 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f484c063ce0 0x7f484c066190 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:39.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.464+0000 7f486f59e700 1 --2- 172.21.15.77:0/1568669850 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48700c5930 0x7f48700c21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:39.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.464+0000 7f486f59e700 1 --2- 172.21.15.77:0/1568669850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f487006d3f0 0x7f4870151420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:39.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.464+0000 7f486f59e700 1 --2- 172.21.15.77:0/1568669850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48700c2a90 0x7f48700c1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:39.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.464+0000 7f486f59e700 1 -- 172.21.15.77:0/1568669850 >> 172.21.15.77:0/1568669850 conn(0x7f48700cb720 msgr2=0x7f487014eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:39.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.465+0000 7f486f59e700 1 -- 172.21.15.77:0/1568669850 shutdown_connections 2022-04-03T19:46:39.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:39.465+0000 7f486f59e700 1 -- 172.21.15.77:0/1568669850 wait complete. 2022-04-03T19:46:39.468 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2022-04-03T19:46:39.490 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:46:41.491 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:46:41.691 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.690+0000 7f51f0ebf700 1 Processor -- start 2022-04-03T19:46:41.691 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.690+0000 7f51f0ebf700 1 -- start start 2022-04-03T19:46:41.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.691+0000 7f51f0ebf700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51ec06d3f0 0x7f51ec06d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:41.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.691+0000 7f51f0ebf700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f51ec0c5930 0x7f51ec155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:41.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.691+0000 7f51f0ebf700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51ec0c2f80 con 0x7f51ec156120 2022-04-03T19:46:41.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.691+0000 7f51f0ebf700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51ec0c3260 con 0x7f51ec0c5930 2022-04-03T19:46:41.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.691+0000 7f51f0ebf700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec0c3540 con 0x7f51ec06d3f0 2022-04-03T19:46:41.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.692+0000 7f51eb7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51ec06d3f0 0x7f51ec06d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:41.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.692+0000 7f51eaffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f51ec0c5930 0x7f51ec155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:41.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.692+0000 7f51ebfff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f51ec156120 0x7f51ec06c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38394/0 (socket says 172.21.15.77:38394) 2022-04-03T19:46:41.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.692+0000 7f51ebfff700 1 -- 172.21.15.77:0/1651882834 learned_addr learned my addr 172.21.15.77:0/1651882834 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:46:41.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.692+0000 7f51eb7fe700 1 -- 172.21.15.77:0/1651882834 >> v2:172.21.15.77:3300/0 conn(0x7f51ec0c5930 msgr2=0x7f51ec155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:46:41.696 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.692+0000 7f51eb7fe700 1 --2- 172.21.15.77:0/1651882834 >> v2:172.21.15.77:3300/0 conn(0x7f51ec0c5930 0x7f51ec155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:41.696 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.692+0000 7f51eb7fe700 1 -- 172.21.15.77:0/1651882834 >> v1:172.21.15.77:6789/0 conn(0x7f51ec156120 legacy=0x7f51ec06c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:41.696 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.692+0000 7f51eb7fe700 1 -- 172.21.15.77:0/1651882834 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51ec0c3820 con 0x7f51ec06d3f0 2022-04-03T19:46:41.696 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.692+0000 7f51eaffd700 1 --2- 172.21.15.77:0/1651882834 >> v2:172.21.15.77:3300/0 conn(0x7f51ec0c5930 0x7f51ec155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:46:41.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.693+0000 7f51eb7fe700 1 --2- 172.21.15.77:0/1651882834 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51ec06d3f0 0x7f51ec06d7c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009790 tx=0x7f51dc009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=992733d7d4deed14 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:41.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.693+0000 7f51ea7fc700 1 -- 172.21.15.77:0/1651882834 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc011630 con 0x7f51ec06d3f0 2022-04-03T19:46:41.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.693+0000 7f51ea7fc700 1 -- 172.21.15.77:0/1651882834 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc011790 con 0x7f51ec06d3f0 2022-04-03T19:46:41.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.693+0000 7f51ea7fc700 1 -- 172.21.15.77:0/1651882834 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc011b50 con 0x7f51ec06d3f0 2022-04-03T19:46:41.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.693+0000 7f51f0ebf700 1 -- 172.21.15.77:0/1651882834 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51ec06d3f0 msgr2=0x7f51ec06d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:41.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.693+0000 7f51f0ebf700 1 --2- 172.21.15.77:0/1651882834 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51ec06d3f0 0x7f51ec06d7c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009790 tx=0x7f51dc009060 comp rx=0 tx=0).stop 2022-04-03T19:46:41.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.694+0000 7f51f0ebf700 1 -- 172.21.15.77:0/1651882834 shutdown_connections 2022-04-03T19:46:41.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.694+0000 7f51f0ebf700 1 --2- 172.21.15.77:0/1651882834 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51ec06d3f0 0x7f51ec06d7c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:41.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.694+0000 7f51f0ebf700 1 --2- 172.21.15.77:0/1651882834 >> v2:172.21.15.77:3300/0 conn(0x7f51ec0c5930 0x7f51ec155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:41.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.694+0000 7f51f0ebf700 1 -- 172.21.15.77:0/1651882834 >> 172.21.15.77:0/1651882834 conn(0x7f51ec0cb720 msgr2=0x7f51ec0cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:41.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.694+0000 7f51f0ebf700 1 -- 172.21.15.77:0/1651882834 shutdown_connections 2022-04-03T19:46:41.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.694+0000 7f51f0ebf700 1 -- 172.21.15.77:0/1651882834 wait complete. 2022-04-03T19:46:41.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.695+0000 7f51f0ebf700 1 Processor -- start 2022-04-03T19:46:41.704 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.695+0000 7f51f0ebf700 1 -- start start 2022-04-03T19:46:41.704 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.696+0000 7f51f0ebf700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51ec06d3f0 0x7f51ec0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:41.704 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.696+0000 7f51f0ebf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51ec0c5930 0x7f51ec1566f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:41.704 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.696+0000 7f51f0ebf700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51ec156120 0x7f51ec156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:41.705 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.696+0000 7f51f0ebf700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec0c1ce0 con 0x7f51ec0c5930 2022-04-03T19:46:41.705 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.696+0000 7f51f0ebf700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec0c1a00 con 0x7f51ec06d3f0 2022-04-03T19:46:41.705 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.696+0000 7f51f0ebf700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec0c0df0 con 0x7f51ec156120 2022-04-03T19:46:41.705 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.696+0000 7f51eb7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51ec06d3f0 0x7f51ec0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:41.705 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.696+0000 7f51eaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51ec0c5930 0x7f51ec1566f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:41.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.696+0000 7f51ebfff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51ec156120 0x7f51ec156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:41.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.696+0000 7f51eb7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51ec06d3f0 0x7f51ec0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37944/0 (socket says 172.21.15.77:37944) 2022-04-03T19:46:41.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.696+0000 7f51eb7fe700 1 -- 172.21.15.77:0/1458001298 learned_addr learned my addr 172.21.15.77:0/1458001298 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:41.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.697+0000 7f51eb7fe700 1 -- 172.21.15.77:0/1458001298 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51ec156120 msgr2=0x7f51ec156c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:41.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.697+0000 7f51eb7fe700 1 --2- 172.21.15.77:0/1458001298 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51ec156120 0x7f51ec156c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:41.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.697+0000 7f51eb7fe700 1 -- 172.21.15.77:0/1458001298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51ec0c5930 msgr2=0x7f51ec1566f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:41.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.697+0000 7f51eb7fe700 1 --2- 172.21.15.77:0/1458001298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51ec0c5930 0x7f51ec1566f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:41.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.697+0000 7f51eb7fe700 1 -- 172.21.15.77:0/1458001298 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e00092a0 con 0x7f51ec06d3f0 2022-04-03T19:46:41.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.697+0000 7f51eaffd700 1 --2- 172.21.15.77:0/1458001298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51ec0c5930 0x7f51ec1566f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:46:41.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.697+0000 7f51eb7fe700 1 --2- 172.21.15.77:0/1458001298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51ec06d3f0 0x7f51ec0c0400 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0116b0 tx=0x7f51dc017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:41.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.697+0000 7f51d37fe700 1 -- 172.21.15.77:0/1458001298 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc000c90 con 0x7f51ec06d3f0 2022-04-03T19:46:41.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.698+0000 7f51d37fe700 1 -- 172.21.15.77:0/1458001298 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc011900 con 0x7f51ec06d3f0 2022-04-03T19:46:41.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.698+0000 7f51f0ebf700 1 -- 172.21.15.77:0/1458001298 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51dc0092a0 con 0x7f51ec06d3f0 2022-04-03T19:46:41.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.698+0000 7f51d37fe700 1 -- 172.21.15.77:0/1458001298 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc025420 con 0x7f51ec06d3f0 2022-04-03T19:46:41.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.698+0000 7f51f0ebf700 1 -- 172.21.15.77:0/1458001298 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51ec0c13b0 con 0x7f51ec06d3f0 2022-04-03T19:46:41.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.699+0000 7f51f0ebf700 1 -- 172.21.15.77:0/1458001298 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51ec04b050 con 0x7f51ec06d3f0 2022-04-03T19:46:41.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.700+0000 7f51d37fe700 1 -- 172.21.15.77:0/1458001298 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f51dc02c030 con 0x7f51ec06d3f0 2022-04-03T19:46:41.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.700+0000 7f51d37fe700 1 --2- 172.21.15.77:0/1458001298 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f51cc063ce0 0x7f51cc066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:41.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.701+0000 7f51d37fe700 1 -- 172.21.15.77:0/1458001298 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f51dc081df0 con 0x7f51ec06d3f0 2022-04-03T19:46:41.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.701+0000 7f51eaffd700 1 --2- 172.21.15.77:0/1458001298 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f51cc063ce0 0x7f51cc066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:41.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.701+0000 7f51eaffd700 1 --2- 172.21.15.77:0/1458001298 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f51cc063ce0 0x7f51cc066190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f51ec162f80 tx=0x7f51e0009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:41.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.705+0000 7f51d37fe700 1 -- 172.21.15.77:0/1458001298 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f51dc0509a0 con 0x7f51ec06d3f0 2022-04-03T19:46:41.938 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.938+0000 7f51f0ebf700 1 -- 172.21.15.77:0/1458001298 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51ec06e1e0 con 0x7f51ec06d3f0 2022-04-03T19:46:41.939 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.939+0000 7f51d37fe700 1 -- 172.21.15.77:0/1458001298 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6107 (secure 0 0 0) 0x7f51dc02a090 con 0x7f51ec06d3f0 2022-04-03T19:46:41.940 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:46:41.940 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4303,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:44:27.295743+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4327":{"gid":4327,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6837/3665969042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3665969042},{"type":"v1","addr":"172.21.15.77:6837","nonce":3665969042}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:46:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.941+0000 7f51f0ebf700 1 -- 172.21.15.77:0/1458001298 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f51cc063ce0 msgr2=0x7f51cc066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:41.942 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.941+0000 7f51f0ebf700 1 --2- 172.21.15.77:0/1458001298 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f51cc063ce0 0x7f51cc066190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f51ec162f80 tx=0x7f51e0009060 comp rx=0 tx=0).stop 2022-04-03T19:46:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.941+0000 7f51f0ebf700 1 -- 172.21.15.77:0/1458001298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51ec06d3f0 msgr2=0x7f51ec0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.941+0000 7f51f0ebf700 1 --2- 172.21.15.77:0/1458001298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51ec06d3f0 0x7f51ec0c0400 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0116b0 tx=0x7f51dc017000 comp rx=0 tx=0).stop 2022-04-03T19:46:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.941+0000 7f51f0ebf700 1 -- 172.21.15.77:0/1458001298 shutdown_connections 2022-04-03T19:46:41.943 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.941+0000 7f51f0ebf700 1 --2- 172.21.15.77:0/1458001298 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f51cc063ce0 0x7f51cc066190 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.941+0000 7f51f0ebf700 1 --2- 172.21.15.77:0/1458001298 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51ec156120 0x7f51ec156c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.941+0000 7f51f0ebf700 1 --2- 172.21.15.77:0/1458001298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51ec06d3f0 0x7f51ec0c0400 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:41.944 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.941+0000 7f51f0ebf700 1 --2- 172.21.15.77:0/1458001298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51ec0c5930 0x7f51ec1566f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.941+0000 7f51f0ebf700 1 -- 172.21.15.77:0/1458001298 >> 172.21.15.77:0/1458001298 conn(0x7f51ec0cb720 msgr2=0x7f51ec0c8140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.941+0000 7f51f0ebf700 1 -- 172.21.15.77:0/1458001298 shutdown_connections 2022-04-03T19:46:41.945 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:41.941+0000 7f51f0ebf700 1 -- 172.21.15.77:0/1458001298 wait complete. 2022-04-03T19:46:41.946 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 8 2022-04-03T19:46:41.965 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:46:43.967 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:46:44.179 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.179+0000 7f2b2d1ce700 1 Processor -- start 2022-04-03T19:46:44.180 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.179+0000 7f2b2d1ce700 1 -- start start 2022-04-03T19:46:44.182 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.180+0000 7f2b2d1ce700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2b2806d3f0 0x7f2b2806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:44.182 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.180+0000 7f2b2d1ce700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2b280c5930 0x7f2b28155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:44.182 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.180+0000 7f2b2d1ce700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b280c2f80 con 0x7f2b28156120 2022-04-03T19:46:44.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.180+0000 7f2b2d1ce700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b280c3260 con 0x7f2b280c5930 2022-04-03T19:46:44.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.180+0000 7f2b2d1ce700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b280c3540 con 0x7f2b2806d3f0 2022-04-03T19:46:44.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.181+0000 7f2b277fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2b280c5930 0x7f2b28155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:44.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.181+0000 7f2b27fff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2b2806d3f0 0x7f2b2806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:44.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.181+0000 7f2b27fff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2b2806d3f0 0x7f2b2806d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37946/0 (socket says 172.21.15.77:37946) 2022-04-03T19:46:44.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.181+0000 7f2b2c9cd700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f2b28156120 0x7f2b2806c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44180/0 (socket says 172.21.15.77:44180) 2022-04-03T19:46:44.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.181+0000 7f2b27fff700 1 -- 172.21.15.77:0/3301306964 learned_addr learned my addr 172.21.15.77:0/3301306964 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:44.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.181+0000 7f2b27fff700 1 -- 172.21.15.77:0/3301306964 >> v2:172.21.15.77:3300/0 conn(0x7f2b280c5930 msgr2=0x7f2b28155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:46:44.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.181+0000 7f2b27fff700 1 --2- 172.21.15.77:0/3301306964 >> v2:172.21.15.77:3300/0 conn(0x7f2b280c5930 0x7f2b28155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:44.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.181+0000 7f2b27fff700 1 -- 172.21.15.77:0/3301306964 >> v1:172.21.15.186:6789/0 conn(0x7f2b28156120 legacy=0x7f2b2806c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:44.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.181+0000 7f2b27fff700 1 -- 172.21.15.77:0/3301306964 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b280c3820 con 0x7f2b2806d3f0 2022-04-03T19:46:44.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.181+0000 7f2b26ffd700 1 -- 172.21.15.77:0/3301306964 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326620199 0 0) 0x7f2b280c2f80 con 0x7f2b28156120 2022-04-03T19:46:44.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.182+0000 7f2b27fff700 1 --2- 172.21.15.77:0/3301306964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b2806d3f0 0x7f2b2806d7c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2b18006fd0 tx=0x7f2b180090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98bb1ee3e6e8021d server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:44.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.182+0000 7f2b26ffd700 1 -- 172.21.15.77:0/3301306964 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b18004030 con 0x7f2b2806d3f0 2022-04-03T19:46:44.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.182+0000 7f2b26ffd700 1 -- 172.21.15.77:0/3301306964 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b18004190 con 0x7f2b2806d3f0 2022-04-03T19:46:44.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.182+0000 7f2b26ffd700 1 -- 172.21.15.77:0/3301306964 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b18011630 con 0x7f2b2806d3f0 2022-04-03T19:46:44.189 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.182+0000 7f2b2d1ce700 1 -- 172.21.15.77:0/3301306964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b2806d3f0 msgr2=0x7f2b2806d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:44.190 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.182+0000 7f2b2d1ce700 1 --2- 172.21.15.77:0/3301306964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b2806d3f0 0x7f2b2806d7c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2b18006fd0 tx=0x7f2b180090f0 comp rx=0 tx=0).stop 2022-04-03T19:46:44.190 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.182+0000 7f2b2d1ce700 1 -- 172.21.15.77:0/3301306964 shutdown_connections 2022-04-03T19:46:44.190 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.182+0000 7f2b2d1ce700 1 --2- 172.21.15.77:0/3301306964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b2806d3f0 0x7f2b2806d7c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:44.190 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.182+0000 7f2b2d1ce700 1 --2- 172.21.15.77:0/3301306964 >> v2:172.21.15.77:3300/0 conn(0x7f2b280c5930 0x7f2b28155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:44.191 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.182+0000 7f2b2d1ce700 1 -- 172.21.15.77:0/3301306964 >> 172.21.15.77:0/3301306964 conn(0x7f2b280cb720 msgr2=0x7f2b280cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:44.191 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.182+0000 7f2b2d1ce700 1 -- 172.21.15.77:0/3301306964 shutdown_connections 2022-04-03T19:46:44.191 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.182+0000 7f2b2d1ce700 1 -- 172.21.15.77:0/3301306964 wait complete. 2022-04-03T19:46:44.191 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.184+0000 7f2b2d1ce700 1 Processor -- start 2022-04-03T19:46:44.192 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.184+0000 7f2b2d1ce700 1 -- start start 2022-04-03T19:46:44.192 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.185+0000 7f2b2d1ce700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b2806d3f0 0x7f2b2814aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:44.192 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.185+0000 7f2b2d1ce700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b280c5930 0x7f2b280c1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:44.192 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.185+0000 7f2b2d1ce700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b28156120 0x7f2b280c1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:44.192 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.185+0000 7f2b2d1ce700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b28166190 con 0x7f2b2806d3f0 2022-04-03T19:46:44.193 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.185+0000 7f2b2d1ce700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b280c0bf0 con 0x7f2b28156120 2022-04-03T19:46:44.193 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.185+0000 7f2b2d1ce700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b280c0e90 con 0x7f2b280c5930 2022-04-03T19:46:44.193 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.185+0000 7f2b27fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b2806d3f0 0x7f2b2814aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:44.193 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.185+0000 7f2b27fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b2806d3f0 0x7f2b2814aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42756/0 (socket says 172.21.15.77:42756) 2022-04-03T19:46:44.194 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.185+0000 7f2b27fff700 1 -- 172.21.15.77:0/906068724 learned_addr learned my addr 172.21.15.77:0/906068724 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:44.194 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.185+0000 7f2b277fe700 1 --2- 172.21.15.77:0/906068724 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b280c5930 0x7f2b280c1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:44.194 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.186+0000 7f2b2c9cd700 1 --2- 172.21.15.77:0/906068724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b28156120 0x7f2b280c1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:44.194 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.186+0000 7f2b27fff700 1 -- 172.21.15.77:0/906068724 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b280c5930 msgr2=0x7f2b280c1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:44.194 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.186+0000 7f2b27fff700 1 --2- 172.21.15.77:0/906068724 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b280c5930 0x7f2b280c1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.186+0000 7f2b27fff700 1 -- 172.21.15.77:0/906068724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b28156120 msgr2=0x7f2b280c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.186+0000 7f2b27fff700 1 --2- 172.21.15.77:0/906068724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b28156120 0x7f2b280c1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.186+0000 7f2b27fff700 1 -- 172.21.15.77:0/906068724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b18009370 con 0x7f2b2806d3f0 2022-04-03T19:46:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.186+0000 7f2b2c9cd700 1 --2- 172.21.15.77:0/906068724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b28156120 0x7f2b280c1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:46:44.195 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.186+0000 7f2b27fff700 1 --2- 172.21.15.77:0/906068724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b2806d3f0 0x7f2b2814aea0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2b280c3680 tx=0x7f2b18013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:44.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.186+0000 7f2b0ffff700 1 -- 172.21.15.77:0/906068724 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b180036f0 con 0x7f2b2806d3f0 2022-04-03T19:46:44.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.187+0000 7f2b0ffff700 1 -- 172.21.15.77:0/906068724 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b18003850 con 0x7f2b2806d3f0 2022-04-03T19:46:44.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.187+0000 7f2b2d1ce700 1 -- 172.21.15.77:0/906068724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b280c24d0 con 0x7f2b2806d3f0 2022-04-03T19:46:44.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.187+0000 7f2b0ffff700 1 -- 172.21.15.77:0/906068724 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b18003bd0 con 0x7f2b2806d3f0 2022-04-03T19:46:44.197 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.187+0000 7f2b2d1ce700 1 -- 172.21.15.77:0/906068724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b280bff20 con 0x7f2b2806d3f0 2022-04-03T19:46:44.197 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.188+0000 7f2b2d1ce700 1 -- 172.21.15.77:0/906068724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b2804b050 con 0x7f2b2806d3f0 2022-04-03T19:46:44.197 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.189+0000 7f2b0ffff700 1 -- 172.21.15.77:0/906068724 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f2b18017020 con 0x7f2b2806d3f0 2022-04-03T19:46:44.198 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.190+0000 7f2b0ffff700 1 --2- 172.21.15.77:0/906068724 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2b08063ce0 0x7f2b08066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:44.198 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.190+0000 7f2b0ffff700 1 -- 172.21.15.77:0/906068724 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f2b18082120 con 0x7f2b2806d3f0 2022-04-03T19:46:44.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.190+0000 7f2b277fe700 1 --2- 172.21.15.77:0/906068724 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2b08063ce0 0x7f2b08066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:44.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.191+0000 7f2b277fe700 1 --2- 172.21.15.77:0/906068724 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2b08063ce0 0x7f2b08066190 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2b28152f90 tx=0x7f2b1c009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:44.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.195+0000 7f2b0ffff700 1 -- 172.21.15.77:0/906068724 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f2b18050c70 con 0x7f2b2806d3f0 2022-04-03T19:46:44.423 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.422+0000 7f2b2d1ce700 1 -- 172.21.15.77:0/906068724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b2806d010 con 0x7f2b2806d3f0 2022-04-03T19:46:44.423 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.423+0000 7f2b0ffff700 1 -- 172.21.15.77:0/906068724 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+5352 (secure 0 0 0) 0x7f2b1802a020 con 0x7f2b2806d3f0 2022-04-03T19:46:44.424 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:46:44.424 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:46:44.097538+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:replay","state_seq":1,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:46:44.426 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.425+0000 7f2b2d1ce700 1 -- 172.21.15.77:0/906068724 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2b08063ce0 msgr2=0x7f2b08066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:44.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.425+0000 7f2b2d1ce700 1 --2- 172.21.15.77:0/906068724 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2b08063ce0 0x7f2b08066190 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2b28152f90 tx=0x7f2b1c009060 comp rx=0 tx=0).stop 2022-04-03T19:46:44.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.425+0000 7f2b2d1ce700 1 -- 172.21.15.77:0/906068724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b2806d3f0 msgr2=0x7f2b2814aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:44.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.425+0000 7f2b2d1ce700 1 --2- 172.21.15.77:0/906068724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b2806d3f0 0x7f2b2814aea0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2b280c3680 tx=0x7f2b18013000 comp rx=0 tx=0).stop 2022-04-03T19:46:44.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.426+0000 7f2b2d1ce700 1 -- 172.21.15.77:0/906068724 shutdown_connections 2022-04-03T19:46:44.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.426+0000 7f2b2d1ce700 1 --2- 172.21.15.77:0/906068724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b28156120 0x7f2b280c1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:44.428 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.426+0000 7f2b2d1ce700 1 --2- 172.21.15.77:0/906068724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b2806d3f0 0x7f2b2814aea0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:44.428 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.426+0000 7f2b2d1ce700 1 --2- 172.21.15.77:0/906068724 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2b08063ce0 0x7f2b08066190 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:44.428 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.426+0000 7f2b2d1ce700 1 --2- 172.21.15.77:0/906068724 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b280c5930 0x7f2b280c1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:44.428 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.426+0000 7f2b2d1ce700 1 -- 172.21.15.77:0/906068724 >> 172.21.15.77:0/906068724 conn(0x7f2b280cb720 msgr2=0x7f2b280c8140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:44.428 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.426+0000 7f2b2d1ce700 1 -- 172.21.15.77:0/906068724 shutdown_connections 2022-04-03T19:46:44.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.426+0000 7f2b2d1ce700 1 -- 172.21.15.77:0/906068724 wait complete. 2022-04-03T19:46:44.429 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 9 2022-04-03T19:46:44.451 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2022-04-03T19:46:44.451 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-04-03T19:46:44.451 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:46:44.648 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.648+0000 7f5cfdbd9700 1 Processor -- start 2022-04-03T19:46:44.648 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.648+0000 7f5cfdbd9700 1 -- start start 2022-04-03T19:46:44.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.649+0000 7f5cfdbd9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cf80ba3c0 0x7f5cf8156b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:44.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.649+0000 7f5cfdbd9700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5cf8157060 0x7f5cf814cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:44.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.649+0000 7f5cfdbd9700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf80c3c50 con 0x7f5cf80678b0 2022-04-03T19:46:44.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.649+0000 7f5cfdbd9700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cf80c8020 con 0x7f5cf8157060 2022-04-03T19:46:44.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.649+0000 7f5cfdbd9700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf8156690 con 0x7f5cf80ba3c0 2022-04-03T19:46:44.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.649+0000 7f5cf7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5cf8157060 0x7f5cf814cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:44.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.650+0000 7f5cfd3d8700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5cf80678b0 0x7f5cf814d040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38400/0 (socket says 172.21.15.77:38400) 2022-04-03T19:46:44.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.650+0000 7f5cfd3d8700 1 -- 172.21.15.77:0/3241454981 learned_addr learned my addr 172.21.15.77:0/3241454981 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:46:44.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.650+0000 7f5cfcbd7700 1 --2- 172.21.15.77:0/3241454981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cf80ba3c0 0x7f5cf8156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:44.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.650+0000 7f5cfcbd7700 1 -- 172.21.15.77:0/3241454981 >> v2:172.21.15.77:3300/0 conn(0x7f5cf8157060 msgr2=0x7f5cf814cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:46:44.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.650+0000 7f5cfcbd7700 1 --2- 172.21.15.77:0/3241454981 >> v2:172.21.15.77:3300/0 conn(0x7f5cf8157060 0x7f5cf814cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:44.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.650+0000 7f5cfcbd7700 1 -- 172.21.15.77:0/3241454981 >> v1:172.21.15.77:6789/0 conn(0x7f5cf80678b0 legacy=0x7f5cf814d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:44.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.650+0000 7f5cfcbd7700 1 -- 172.21.15.77:0/3241454981 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf80bf350 con 0x7f5cf80ba3c0 2022-04-03T19:46:44.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.650+0000 7f5cf7fff700 1 --2- 172.21.15.77:0/3241454981 >> v2:172.21.15.77:3300/0 conn(0x7f5cf8157060 0x7f5cf814cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:46:44.656 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.650+0000 7f5cf77fe700 1 -- 172.21.15.77:0/3241454981 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008902561 0 0) 0x7f5cf80c3c50 con 0x7f5cf80678b0 2022-04-03T19:46:44.656 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.650+0000 7f5cfcbd7700 1 --2- 172.21.15.77:0/3241454981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cf80ba3c0 0x7f5cf8156b20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5cec009760 tx=0x7f5cec009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f07502f6e25b092f server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:44.656 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.651+0000 7f5cf77fe700 1 -- 172.21.15.77:0/3241454981 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec011630 con 0x7f5cf80ba3c0 2022-04-03T19:46:44.656 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.651+0000 7f5cf77fe700 1 -- 172.21.15.77:0/3241454981 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cec011790 con 0x7f5cf80ba3c0 2022-04-03T19:46:44.657 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.651+0000 7f5cf77fe700 1 -- 172.21.15.77:0/3241454981 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cec011b50 con 0x7f5cf80ba3c0 2022-04-03T19:46:44.657 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.651+0000 7f5cfdbd9700 1 -- 172.21.15.77:0/3241454981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cf80ba3c0 msgr2=0x7f5cf8156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:44.657 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.651+0000 7f5cfdbd9700 1 --2- 172.21.15.77:0/3241454981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cf80ba3c0 0x7f5cf8156b20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5cec009760 tx=0x7f5cec009060 comp rx=0 tx=0).stop 2022-04-03T19:46:44.657 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.651+0000 7f5cfdbd9700 1 -- 172.21.15.77:0/3241454981 shutdown_connections 2022-04-03T19:46:44.657 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.651+0000 7f5cfdbd9700 1 --2- 172.21.15.77:0/3241454981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cf80ba3c0 0x7f5cf8156b20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:44.658 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.651+0000 7f5cfdbd9700 1 --2- 172.21.15.77:0/3241454981 >> v2:172.21.15.77:3300/0 conn(0x7f5cf8157060 0x7f5cf814cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:44.658 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.651+0000 7f5cfdbd9700 1 -- 172.21.15.77:0/3241454981 >> 172.21.15.77:0/3241454981 conn(0x7f5cf8066da0 msgr2=0x7f5cf80671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:44.658 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.652+0000 7f5cfdbd9700 1 -- 172.21.15.77:0/3241454981 shutdown_connections 2022-04-03T19:46:44.658 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.652+0000 7f5cfdbd9700 1 -- 172.21.15.77:0/3241454981 wait complete. 2022-04-03T19:46:44.658 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.653+0000 7f5cfdbd9700 1 Processor -- start 2022-04-03T19:46:44.659 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.653+0000 7f5cfdbd9700 1 -- start start 2022-04-03T19:46:44.659 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.654+0000 7f5cfdbd9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cf80678b0 0x7f5cf81661a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:44.659 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.654+0000 7f5cfdbd9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cf80ba3c0 0x7f5cf8142ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:44.659 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.654+0000 7f5cfdbd9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cf8157060 0x7f5cf81549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:44.659 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.654+0000 7f5cfdbd9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf80c5070 con 0x7f5cf8157060 2022-04-03T19:46:44.660 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.654+0000 7f5cfdbd9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf814d7d0 con 0x7f5cf80678b0 2022-04-03T19:46:44.660 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.654+0000 7f5cfdbd9700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf80c5280 con 0x7f5cf80ba3c0 2022-04-03T19:46:44.660 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.654+0000 7f5cf7fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cf80ba3c0 0x7f5cf8142ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:44.660 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.654+0000 7f5cfcbd7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cf80678b0 0x7f5cf81661a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:44.660 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.654+0000 7f5cfd3d8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cf8157060 0x7f5cf81549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:44.661 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.654+0000 7f5cf7fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cf80ba3c0 0x7f5cf8142ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60060/0 (socket says 172.21.15.77:60060) 2022-04-03T19:46:44.661 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.654+0000 7f5cf7fff700 1 -- 172.21.15.77:0/2072580452 learned_addr learned my addr 172.21.15.77:0/2072580452 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:44.663 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.655+0000 7f5cf7fff700 1 -- 172.21.15.77:0/2072580452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cf80678b0 msgr2=0x7f5cf81661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:44.663 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.655+0000 7f5cf7fff700 1 --2- 172.21.15.77:0/2072580452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cf80678b0 0x7f5cf81661a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:44.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.655+0000 7f5cf7fff700 1 -- 172.21.15.77:0/2072580452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cf8157060 msgr2=0x7f5cf81549e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:44.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.655+0000 7f5cf7fff700 1 --2- 172.21.15.77:0/2072580452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cf8157060 0x7f5cf81549e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:44.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.655+0000 7f5cf7fff700 1 -- 172.21.15.77:0/2072580452 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce40092a0 con 0x7f5cf80ba3c0 2022-04-03T19:46:44.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.655+0000 7f5cfcbd7700 1 --2- 172.21.15.77:0/2072580452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cf80678b0 0x7f5cf81661a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:46:44.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.655+0000 7f5cfd3d8700 1 --2- 172.21.15.77:0/2072580452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cf8157060 0x7f5cf81549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:46:44.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.655+0000 7f5cf7fff700 1 --2- 172.21.15.77:0/2072580452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cf80ba3c0 0x7f5cf8142ec0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5ce400e900 tx=0x7f5ce4009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:44.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.655+0000 7f5cf4ff9700 1 -- 172.21.15.77:0/2072580452 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4013680 con 0x7f5cf80ba3c0 2022-04-03T19:46:44.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.656+0000 7f5cf4ff9700 1 -- 172.21.15.77:0/2072580452 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce40137e0 con 0x7f5cf80ba3c0 2022-04-03T19:46:44.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.656+0000 7f5cfdbd9700 1 -- 172.21.15.77:0/2072580452 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cec0092a0 con 0x7f5cf80ba3c0 2022-04-03T19:46:44.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.656+0000 7f5cf4ff9700 1 -- 172.21.15.77:0/2072580452 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4013ba0 con 0x7f5cf80ba3c0 2022-04-03T19:46:44.666 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.656+0000 7f5cfdbd9700 1 -- 172.21.15.77:0/2072580452 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cf814da20 con 0x7f5cf80ba3c0 2022-04-03T19:46:44.666 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.657+0000 7f5cfdbd9700 1 -- 172.21.15.77:0/2072580452 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cf80c74a0 con 0x7f5cf80ba3c0 2022-04-03T19:46:44.666 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.658+0000 7f5cf4ff9700 1 -- 172.21.15.77:0/2072580452 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f5ce4019030 con 0x7f5cf80ba3c0 2022-04-03T19:46:44.666 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.658+0000 7f5cf4ff9700 1 --2- 172.21.15.77:0/2072580452 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5ce0063ce0 0x7f5ce0066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:44.667 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.659+0000 7f5cf4ff9700 1 -- 172.21.15.77:0/2072580452 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f5ce4080c70 con 0x7f5cf80ba3c0 2022-04-03T19:46:44.667 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.659+0000 7f5cfcbd7700 1 --2- 172.21.15.77:0/2072580452 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5ce0063ce0 0x7f5ce0066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:44.667 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.659+0000 7f5cfcbd7700 1 --2- 172.21.15.77:0/2072580452 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5ce0063ce0 0x7f5ce0066190 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5cec009760 tx=0x7f5cec017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:44.667 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.662+0000 7f5cf4ff9700 1 -- 172.21.15.77:0/2072580452 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f5ce404f750 con 0x7f5cf80ba3c0 2022-04-03T19:46:44.895 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.895+0000 7f5cfdbd9700 1 -- 172.21.15.77:0/2072580452 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5cf80bfaf0 con 0x7f5cf80ba3c0 2022-04-03T19:46:44.896 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.896+0000 7f5cf4ff9700 1 -- 172.21.15.77:0/2072580452 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+5352 (secure 0 0 0) 0x7f5ce4021090 con 0x7f5cf80ba3c0 2022-04-03T19:46:44.896 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:46:44.897 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:46:44.097538+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:replay","state_seq":1,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:46:44.899 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.898+0000 7f5cfdbd9700 1 -- 172.21.15.77:0/2072580452 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5ce0063ce0 msgr2=0x7f5ce0066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:44.899 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.898+0000 7f5cfdbd9700 1 --2- 172.21.15.77:0/2072580452 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5ce0063ce0 0x7f5ce0066190 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5cec009760 tx=0x7f5cec017000 comp rx=0 tx=0).stop 2022-04-03T19:46:44.899 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.898+0000 7f5cfdbd9700 1 -- 172.21.15.77:0/2072580452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cf80ba3c0 msgr2=0x7f5cf8142ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:44.900 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.898+0000 7f5cfdbd9700 1 --2- 172.21.15.77:0/2072580452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cf80ba3c0 0x7f5cf8142ec0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5ce400e900 tx=0x7f5ce4009060 comp rx=0 tx=0).stop 2022-04-03T19:46:44.900 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.898+0000 7f5cfdbd9700 1 -- 172.21.15.77:0/2072580452 shutdown_connections 2022-04-03T19:46:44.900 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.898+0000 7f5cfdbd9700 1 --2- 172.21.15.77:0/2072580452 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5cf8157060 0x7f5cf81549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:44.900 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.898+0000 7f5cfdbd9700 1 --2- 172.21.15.77:0/2072580452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cf80678b0 0x7f5cf81661a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:44.900 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.898+0000 7f5cfdbd9700 1 --2- 172.21.15.77:0/2072580452 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5ce0063ce0 0x7f5ce0066190 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:44.901 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.898+0000 7f5cfdbd9700 1 --2- 172.21.15.77:0/2072580452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5cf80ba3c0 0x7f5cf8142ec0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:44.901 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.898+0000 7f5cfdbd9700 1 -- 172.21.15.77:0/2072580452 >> 172.21.15.77:0/2072580452 conn(0x7f5cf8066da0 msgr2=0x7f5cf814bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:44.901 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.898+0000 7f5cfdbd9700 1 -- 172.21.15.77:0/2072580452 shutdown_connections 2022-04-03T19:46:44.901 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:44.898+0000 7f5cfdbd9700 1 -- 172.21.15.77:0/2072580452 wait complete. 2022-04-03T19:46:44.902 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 9 2022-04-03T19:46:44.924 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 9, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-04-03T19:44:21.275716+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 9, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4303": { "addr": "172.21.15.186:6837/1407029827", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 1407029827, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 1407029827, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4303, "incarnation": 9, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-04-03T19:46:44.097538+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4303 } } } ], "standbys": [ { "addr": "172.21.15.77:6834/2116223509", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6832", "nonce": 2116223509, "type": "v2" }, { "addr": "172.21.15.77:6834", "nonce": 2116223509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4245, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6836/2374276747", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2374276747, "type": "v2" }, { "addr": "172.21.15.186:6836", "nonce": 2374276747, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4259, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/610990853", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6838", "nonce": 610990853, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 610990853, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4265, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.77:6835/170053412", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6833", "nonce": 170053412, "type": "v2" }, { "addr": "172.21.15.77:6835", "nonce": 170053412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4271, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-04-03T19:46:45.147 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:46:45.152 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:46:46.926 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:46:47.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.132+0000 7f40f7fff700 1 Processor -- start 2022-04-03T19:46:47.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.132+0000 7f40f7fff700 1 -- start start 2022-04-03T19:46:47.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.133+0000 7f40f7fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40f80c5930 0x7f40f8155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:47.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.133+0000 7f40f7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f40f8156120 0x7f40f806c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:47.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.134+0000 7f40f7fff700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f80c2f80 con 0x7f40f806d3f0 2022-04-03T19:46:47.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.134+0000 7f40f7fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40f80c3260 con 0x7f40f8156120 2022-04-03T19:46:47.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.134+0000 7f40f7fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f80c3540 con 0x7f40f80c5930 2022-04-03T19:46:47.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.134+0000 7f40effff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40f80c5930 0x7f40f8155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:47.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.134+0000 7f40f77fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f40f8156120 0x7f40f806c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:47.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.134+0000 7f40f6ffd700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f40f806d3f0 0x7f40f806d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44182/0 (socket says 172.21.15.77:44182) 2022-04-03T19:46:47.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.134+0000 7f40f6ffd700 1 -- 172.21.15.77:0/3911972971 learned_addr learned my addr 172.21.15.77:0/3911972971 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:46:47.138 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.134+0000 7f40effff700 1 -- 172.21.15.77:0/3911972971 >> v2:172.21.15.77:3300/0 conn(0x7f40f8156120 msgr2=0x7f40f806c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:46:47.138 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.134+0000 7f40effff700 1 --2- 172.21.15.77:0/3911972971 >> v2:172.21.15.77:3300/0 conn(0x7f40f8156120 0x7f40f806c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:47.139 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.134+0000 7f40effff700 1 -- 172.21.15.77:0/3911972971 >> v1:172.21.15.186:6789/0 conn(0x7f40f806d3f0 legacy=0x7f40f806d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:47.139 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.134+0000 7f40f67fc700 1 -- 172.21.15.77:0/3911972971 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088706631 0 0) 0x7f40f80c2f80 con 0x7f40f806d3f0 2022-04-03T19:46:47.139 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.134+0000 7f40effff700 1 -- 172.21.15.77:0/3911972971 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f80c3820 con 0x7f40f80c5930 2022-04-03T19:46:47.139 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.135+0000 7f40effff700 1 --2- 172.21.15.77:0/3911972971 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40f80c5930 0x7f40f8155be0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f40e0009790 tx=0x7f40e0009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=677bdcea4bdd17a server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:47.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.135+0000 7f40f67fc700 1 -- 172.21.15.77:0/3911972971 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e0011630 con 0x7f40f80c5930 2022-04-03T19:46:47.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.135+0000 7f40f67fc700 1 -- 172.21.15.77:0/3911972971 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e0011790 con 0x7f40f80c5930 2022-04-03T19:46:47.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.135+0000 7f40f67fc700 1 -- 172.21.15.77:0/3911972971 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e0011b50 con 0x7f40f80c5930 2022-04-03T19:46:47.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.136+0000 7f40f7fff700 1 -- 172.21.15.77:0/3911972971 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40f80c5930 msgr2=0x7f40f8155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:47.141 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.136+0000 7f40f7fff700 1 --2- 172.21.15.77:0/3911972971 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40f80c5930 0x7f40f8155be0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f40e0009790 tx=0x7f40e0009060 comp rx=0 tx=0).stop 2022-04-03T19:46:47.141 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.136+0000 7f40f7fff700 1 -- 172.21.15.77:0/3911972971 shutdown_connections 2022-04-03T19:46:47.141 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.136+0000 7f40f7fff700 1 --2- 172.21.15.77:0/3911972971 >> v2:172.21.15.77:3300/0 conn(0x7f40f8156120 0x7f40f806c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:47.141 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.136+0000 7f40f7fff700 1 --2- 172.21.15.77:0/3911972971 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40f80c5930 0x7f40f8155be0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:47.142 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.136+0000 7f40f7fff700 1 -- 172.21.15.77:0/3911972971 >> 172.21.15.77:0/3911972971 conn(0x7f40f80cb720 msgr2=0x7f40f80cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:47.142 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.136+0000 7f40f7fff700 1 -- 172.21.15.77:0/3911972971 shutdown_connections 2022-04-03T19:46:47.142 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.136+0000 7f40f7fff700 1 -- 172.21.15.77:0/3911972971 wait complete. 2022-04-03T19:46:47.142 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.137+0000 7f40f7fff700 1 Processor -- start 2022-04-03T19:46:47.143 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.138+0000 7f40f7fff700 1 -- start start 2022-04-03T19:46:47.146 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.138+0000 7f40f7fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40f806d3f0 0x7f40f8158900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:47.146 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.138+0000 7f40f7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40f80c5930 0x7f40f80c05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:47.146 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.138+0000 7f40f7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40f8156120 0x7f40f80c1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:47.146 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.138+0000 7f40f7fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f81597d0 con 0x7f40f8156120 2022-04-03T19:46:47.147 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.138+0000 7f40f7fff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f8165f60 con 0x7f40f80c5930 2022-04-03T19:46:47.147 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.138+0000 7f40f7fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f80c0b20 con 0x7f40f806d3f0 2022-04-03T19:46:47.147 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.139+0000 7f40f77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40f8156120 0x7f40f80c1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:47.147 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.139+0000 7f40f77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40f8156120 0x7f40f80c1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42766/0 (socket says 172.21.15.77:42766) 2022-04-03T19:46:47.148 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.139+0000 7f40f77fe700 1 -- 172.21.15.77:0/1892962009 learned_addr learned my addr 172.21.15.77:0/1892962009 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:47.148 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.139+0000 7f40f6ffd700 1 --2- 172.21.15.77:0/1892962009 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40f806d3f0 0x7f40f8158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:47.148 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.139+0000 7f40effff700 1 --2- 172.21.15.77:0/1892962009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40f80c5930 0x7f40f80c05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:47.148 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.139+0000 7f40f77fe700 1 -- 172.21.15.77:0/1892962009 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40f806d3f0 msgr2=0x7f40f8158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:47.148 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.139+0000 7f40f77fe700 1 --2- 172.21.15.77:0/1892962009 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40f806d3f0 0x7f40f8158900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:47.149 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.139+0000 7f40f77fe700 1 -- 172.21.15.77:0/1892962009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40f80c5930 msgr2=0x7f40f80c05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:47.149 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.139+0000 7f40f77fe700 1 --2- 172.21.15.77:0/1892962009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40f80c5930 0x7f40f80c05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:47.149 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.139+0000 7f40f77fe700 1 -- 172.21.15.77:0/1892962009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e00092a0 con 0x7f40f8156120 2022-04-03T19:46:47.149 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.139+0000 7f40effff700 1 --2- 172.21.15.77:0/1892962009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40f80c5930 0x7f40f80c05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:46:47.150 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.140+0000 7f40f77fe700 1 --2- 172.21.15.77:0/1892962009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40f8156120 0x7f40f80c1780 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009d70 tx=0x7f40f000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:47.150 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.140+0000 7f40eeffd700 1 -- 172.21.15.77:0/1892962009 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f000caa0 con 0x7f40f8156120 2022-04-03T19:46:47.150 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.140+0000 7f40eeffd700 1 -- 172.21.15.77:0/1892962009 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f000f040 con 0x7f40f8156120 2022-04-03T19:46:47.150 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.140+0000 7f40eeffd700 1 -- 172.21.15.77:0/1892962009 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f000ce50 con 0x7f40f8156120 2022-04-03T19:46:47.150 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.140+0000 7f40f7fff700 1 -- 172.21.15.77:0/1892962009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f8159210 con 0x7f40f8156120 2022-04-03T19:46:47.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.140+0000 7f40f7fff700 1 -- 172.21.15.77:0/1892962009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f80bfb00 con 0x7f40f8156120 2022-04-03T19:46:47.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.141+0000 7f40f7fff700 1 -- 172.21.15.77:0/1892962009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f804b050 con 0x7f40f8156120 2022-04-03T19:46:47.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.142+0000 7f40eeffd700 1 -- 172.21.15.77:0/1892962009 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f40f0014690 con 0x7f40f8156120 2022-04-03T19:46:47.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.143+0000 7f40eeffd700 1 --2- 172.21.15.77:0/1892962009 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f40dc063ce0 0x7f40dc066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:47.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.143+0000 7f40eeffd700 1 -- 172.21.15.77:0/1892962009 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f40f0081750 con 0x7f40f8156120 2022-04-03T19:46:47.152 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.144+0000 7f40f6ffd700 1 --2- 172.21.15.77:0/1892962009 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f40dc063ce0 0x7f40dc066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:47.152 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.144+0000 7f40f6ffd700 1 --2- 172.21.15.77:0/1892962009 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f40dc063ce0 0x7f40dc066190 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f40e8007690 tx=0x7f40e8007bb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:47.152 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.148+0000 7f40eeffd700 1 -- 172.21.15.77:0/1892962009 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f40f00502a0 con 0x7f40f8156120 2022-04-03T19:46:47.382 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.381+0000 7f40f7fff700 1 -- 172.21.15.77:0/1892962009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40f806e1e0 con 0x7f40f8156120 2022-04-03T19:46:47.383 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.382+0000 7f40eeffd700 1 -- 172.21.15.77:0/1892962009 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+5352 (secure 0 0 0) 0x7f40f00538c0 con 0x7f40f8156120 2022-04-03T19:46:47.383 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:46:47.383 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:46:44.097538+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:replay","state_seq":1,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:46:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.384+0000 7f40f7fff700 1 -- 172.21.15.77:0/1892962009 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f40dc063ce0 msgr2=0x7f40dc066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.384+0000 7f40f7fff700 1 --2- 172.21.15.77:0/1892962009 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f40dc063ce0 0x7f40dc066190 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f40e8007690 tx=0x7f40e8007bb0 comp rx=0 tx=0).stop 2022-04-03T19:46:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.384+0000 7f40f7fff700 1 -- 172.21.15.77:0/1892962009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40f8156120 msgr2=0x7f40f80c1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:47.386 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.384+0000 7f40f7fff700 1 --2- 172.21.15.77:0/1892962009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40f8156120 0x7f40f80c1780 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009d70 tx=0x7f40f000b040 comp rx=0 tx=0).stop 2022-04-03T19:46:47.386 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.384+0000 7f40f7fff700 1 -- 172.21.15.77:0/1892962009 shutdown_connections 2022-04-03T19:46:47.386 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.384+0000 7f40f7fff700 1 --2- 172.21.15.77:0/1892962009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40f8156120 0x7f40f80c1780 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:47.386 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.384+0000 7f40f7fff700 1 --2- 172.21.15.77:0/1892962009 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f40dc063ce0 0x7f40dc066190 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:47.387 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.384+0000 7f40f7fff700 1 --2- 172.21.15.77:0/1892962009 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40f806d3f0 0x7f40f8158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:47.387 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.384+0000 7f40f7fff700 1 --2- 172.21.15.77:0/1892962009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40f80c5930 0x7f40f80c05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:47.387 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.384+0000 7f40f7fff700 1 -- 172.21.15.77:0/1892962009 >> 172.21.15.77:0/1892962009 conn(0x7f40f80cb720 msgr2=0x7f40f80c8140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:47.387 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.384+0000 7f40f7fff700 1 -- 172.21.15.77:0/1892962009 shutdown_connections 2022-04-03T19:46:47.388 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:47.385+0000 7f40f7fff700 1 -- 172.21.15.77:0/1892962009 wait complete. 2022-04-03T19:46:47.388 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 9 2022-04-03T19:46:47.409 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:46:49.411 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:46:49.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.615+0000 7f120e907700 1 Processor -- start 2022-04-03T19:46:49.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.615+0000 7f120e907700 1 -- start start 2022-04-03T19:46:49.618 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.616+0000 7f120e907700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f12080ba3c0 0x7f1208156b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:49.618 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.616+0000 7f120e907700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f12080678b0 0x7f120814d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:49.618 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.616+0000 7f120e907700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12080c3c50 con 0x7f1208157060 2022-04-03T19:46:49.619 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.616+0000 7f120e907700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12080c8020 con 0x7f12080678b0 2022-04-03T19:46:49.619 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.616+0000 7f120e907700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1208156690 con 0x7f12080ba3c0 2022-04-03T19:46:49.619 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.617+0000 7f120d905700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f12080ba3c0 0x7f1208156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:49.619 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.617+0000 7f120d905700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f12080ba3c0 0x7f1208156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37954/0 (socket says 172.21.15.77:37954) 2022-04-03T19:46:49.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.617+0000 7f120d905700 1 -- 172.21.15.77:0/128701190 learned_addr learned my addr 172.21.15.77:0/128701190 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:49.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.617+0000 7f120e106700 1 --2- 172.21.15.77:0/128701190 >> v2:172.21.15.77:3300/0 conn(0x7f12080678b0 0x7f120814d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:49.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.617+0000 7f120d905700 1 -- 172.21.15.77:0/128701190 >> v2:172.21.15.77:3300/0 conn(0x7f12080678b0 msgr2=0x7f120814d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:46:49.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.617+0000 7f120d905700 1 --2- 172.21.15.77:0/128701190 >> v2:172.21.15.77:3300/0 conn(0x7f12080678b0 0x7f120814d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:49.621 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.617+0000 7f120d905700 1 -- 172.21.15.77:0/128701190 >> v1:172.21.15.77:6789/0 conn(0x7f1208157060 legacy=0x7f120814cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:49.621 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.617+0000 7f120d905700 1 -- 172.21.15.77:0/128701190 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12080bf350 con 0x7f12080ba3c0 2022-04-03T19:46:49.621 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.617+0000 7f120e106700 1 --2- 172.21.15.77:0/128701190 >> v2:172.21.15.77:3300/0 conn(0x7f12080678b0 0x7f120814d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:46:49.621 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.618+0000 7f120d905700 1 --2- 172.21.15.77:0/128701190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12080ba3c0 0x7f1208156b20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1208068280 tx=0x7f12040090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d95d00956c3b14e server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:49.621 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.618+0000 7f120c903700 1 -- 172.21.15.77:0/128701190 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1204004030 con 0x7f12080ba3c0 2022-04-03T19:46:49.622 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.618+0000 7f120c903700 1 -- 172.21.15.77:0/128701190 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12040041d0 con 0x7f12080ba3c0 2022-04-03T19:46:49.622 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.618+0000 7f120c903700 1 -- 172.21.15.77:0/128701190 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120400b690 con 0x7f12080ba3c0 2022-04-03T19:46:49.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.618+0000 7f120e907700 1 -- 172.21.15.77:0/128701190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12080ba3c0 msgr2=0x7f1208156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:49.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.618+0000 7f120e907700 1 --2- 172.21.15.77:0/128701190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12080ba3c0 0x7f1208156b20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1208068280 tx=0x7f12040090f0 comp rx=0 tx=0).stop 2022-04-03T19:46:49.626 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.618+0000 7f120e907700 1 -- 172.21.15.77:0/128701190 shutdown_connections 2022-04-03T19:46:49.626 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.618+0000 7f120e907700 1 --2- 172.21.15.77:0/128701190 >> v2:172.21.15.77:3300/0 conn(0x7f12080678b0 0x7f120814d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:49.626 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.618+0000 7f120e907700 1 --2- 172.21.15.77:0/128701190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12080ba3c0 0x7f1208156b20 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:49.626 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.618+0000 7f120e907700 1 -- 172.21.15.77:0/128701190 >> 172.21.15.77:0/128701190 conn(0x7f1208066da0 msgr2=0x7f12080671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:49.627 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.618+0000 7f120e907700 1 -- 172.21.15.77:0/128701190 shutdown_connections 2022-04-03T19:46:49.627 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.619+0000 7f120e907700 1 -- 172.21.15.77:0/128701190 wait complete. 2022-04-03T19:46:49.627 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.620+0000 7f120e907700 1 Processor -- start 2022-04-03T19:46:49.627 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.620+0000 7f120e907700 1 -- start start 2022-04-03T19:46:49.627 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.621+0000 7f120e907700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12080678b0 0x7f1208168cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:49.628 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.621+0000 7f120e907700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12080ba3c0 0x7f12080bce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:49.628 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.621+0000 7f120e907700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1208157060 0x7f1208149930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:49.628 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.621+0000 7f120e907700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12080bd800 con 0x7f1208157060 2022-04-03T19:46:49.628 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.621+0000 7f120e907700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12080bca20 con 0x7f12080ba3c0 2022-04-03T19:46:49.629 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.621+0000 7f120e907700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12080bb460 con 0x7f12080678b0 2022-04-03T19:46:49.629 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.621+0000 7f120d104700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12080ba3c0 0x7f12080bce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:49.629 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.621+0000 7f120e106700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1208157060 0x7f1208149930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:49.629 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.621+0000 7f120d905700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12080678b0 0x7f1208168cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:49.629 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.621+0000 7f120d104700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12080ba3c0 0x7f12080bce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37956/0 (socket says 172.21.15.77:37956) 2022-04-03T19:46:49.630 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.621+0000 7f120d104700 1 -- 172.21.15.77:0/1295366993 learned_addr learned my addr 172.21.15.77:0/1295366993 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:49.630 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.622+0000 7f120d104700 1 -- 172.21.15.77:0/1295366993 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12080678b0 msgr2=0x7f1208168cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:49.630 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.622+0000 7f120d104700 1 --2- 172.21.15.77:0/1295366993 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12080678b0 0x7f1208168cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:49.630 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.622+0000 7f120d104700 1 -- 172.21.15.77:0/1295366993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1208157060 msgr2=0x7f1208149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:49.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.622+0000 7f120d104700 1 --2- 172.21.15.77:0/1295366993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1208157060 0x7f1208149930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:49.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.622+0000 7f120d104700 1 -- 172.21.15.77:0/1295366993 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1204009370 con 0x7f12080ba3c0 2022-04-03T19:46:49.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.622+0000 7f120e106700 1 --2- 172.21.15.77:0/1295366993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1208157060 0x7f1208149930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:46:49.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.622+0000 7f120d104700 1 --2- 172.21.15.77:0/1295366993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12080ba3c0 0x7f12080bce40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f11f800c440 tx=0x7f11f8007dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:49.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.622+0000 7f11f5ffb700 1 -- 172.21.15.77:0/1295366993 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f80050d0 con 0x7f12080ba3c0 2022-04-03T19:46:49.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.622+0000 7f11f5ffb700 1 -- 172.21.15.77:0/1295366993 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f8005230 con 0x7f12080ba3c0 2022-04-03T19:46:49.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.622+0000 7f120e907700 1 -- 172.21.15.77:0/1295366993 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12080bb670 con 0x7f12080ba3c0 2022-04-03T19:46:49.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.623+0000 7f11f5ffb700 1 -- 172.21.15.77:0/1295366993 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f8003710 con 0x7f12080ba3c0 2022-04-03T19:46:49.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.623+0000 7f120e907700 1 -- 172.21.15.77:0/1295366993 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12080bc360 con 0x7f12080ba3c0 2022-04-03T19:46:49.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.624+0000 7f120e907700 1 -- 172.21.15.77:0/1295366993 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12080c74a0 con 0x7f12080ba3c0 2022-04-03T19:46:49.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.625+0000 7f11f5ffb700 1 -- 172.21.15.77:0/1295366993 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f11f8019030 con 0x7f12080ba3c0 2022-04-03T19:46:49.634 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.625+0000 7f11f5ffb700 1 --2- 172.21.15.77:0/1295366993 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f11e8063ce0 0x7f11e8066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:49.634 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.625+0000 7f11f5ffb700 1 -- 172.21.15.77:0/1295366993 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f11f807f950 con 0x7f12080ba3c0 2022-04-03T19:46:49.635 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.626+0000 7f120d905700 1 --2- 172.21.15.77:0/1295366993 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f11e8063ce0 0x7f11e8066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:49.635 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.626+0000 7f120d905700 1 --2- 172.21.15.77:0/1295366993 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f11e8063ce0 0x7f11e8066190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1204009340 tx=0x7f1204016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:49.635 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.630+0000 7f11f5ffb700 1 -- 172.21.15.77:0/1295366993 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f11f804e4a0 con 0x7f12080ba3c0 2022-04-03T19:46:49.862 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.862+0000 7f120e907700 1 -- 172.21.15.77:0/1295366993 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f120804b050 con 0x7f12080ba3c0 2022-04-03T19:46:49.863 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.863+0000 7f11f5ffb700 1 -- 172.21.15.77:0/1295366993 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+5352 (secure 0 0 0) 0x7f11f8017020 con 0x7f12080ba3c0 2022-04-03T19:46:49.864 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:46:49.864 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:46:44.097538+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:replay","state_seq":1,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:46:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.865+0000 7f120e907700 1 -- 172.21.15.77:0/1295366993 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f11e8063ce0 msgr2=0x7f11e8066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:49.866 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.865+0000 7f120e907700 1 --2- 172.21.15.77:0/1295366993 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f11e8063ce0 0x7f11e8066190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1204009340 tx=0x7f1204016000 comp rx=0 tx=0).stop 2022-04-03T19:46:49.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.865+0000 7f120e907700 1 -- 172.21.15.77:0/1295366993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12080ba3c0 msgr2=0x7f12080bce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:49.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.865+0000 7f120e907700 1 --2- 172.21.15.77:0/1295366993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12080ba3c0 0x7f12080bce40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f11f800c440 tx=0x7f11f8007dd0 comp rx=0 tx=0).stop 2022-04-03T19:46:49.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.865+0000 7f120e907700 1 -- 172.21.15.77:0/1295366993 shutdown_connections 2022-04-03T19:46:49.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.865+0000 7f120e907700 1 --2- 172.21.15.77:0/1295366993 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1208157060 0x7f1208149930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:49.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.865+0000 7f120e907700 1 --2- 172.21.15.77:0/1295366993 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f11e8063ce0 0x7f11e8066190 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:49.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.865+0000 7f120e907700 1 --2- 172.21.15.77:0/1295366993 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12080678b0 0x7f1208168cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:49.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.865+0000 7f120e907700 1 --2- 172.21.15.77:0/1295366993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12080ba3c0 0x7f12080bce40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:49.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.865+0000 7f120e907700 1 -- 172.21.15.77:0/1295366993 >> 172.21.15.77:0/1295366993 conn(0x7f1208066da0 msgr2=0x7f120814c070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:49.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.865+0000 7f120e907700 1 -- 172.21.15.77:0/1295366993 shutdown_connections 2022-04-03T19:46:49.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:49.866+0000 7f120e907700 1 -- 172.21.15.77:0/1295366993 wait complete. 2022-04-03T19:46:49.869 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 9 2022-04-03T19:46:49.890 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:46:51.891 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:46:52.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.096+0000 7fcb0d66c700 1 Processor -- start 2022-04-03T19:46:52.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.096+0000 7fcb0d66c700 1 -- start start 2022-04-03T19:46:52.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.097+0000 7fcb0d66c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb0806d3f0 0x7fcb0806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:52.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.098+0000 7fcb0d66c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcb080c2a90 0x7fcb0806c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:52.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.098+0000 7fcb0d66c700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb081562f0 con 0x7fcb080c5930 2022-04-03T19:46:52.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.098+0000 7fcb0d66c700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb080c2f80 con 0x7fcb080c2a90 2022-04-03T19:46:52.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.098+0000 7fcb0d66c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb080c3260 con 0x7fcb0806d3f0 2022-04-03T19:46:52.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.098+0000 7fcb0ce6b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcb080c2a90 0x7fcb0806c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:52.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.098+0000 7fcb07fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb0806d3f0 0x7fcb0806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:52.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.098+0000 7fcb07fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb0806d3f0 0x7fcb0806d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60070/0 (socket says 172.21.15.77:60070) 2022-04-03T19:46:52.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.098+0000 7fcb07fff700 1 -- 172.21.15.77:0/2460279870 learned_addr learned my addr 172.21.15.77:0/2460279870 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:52.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.098+0000 7fcb0ce6b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcb080c2a90 0x7fcb0806c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42772/0 (socket says 172.21.15.77:42772) 2022-04-03T19:46:52.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.099+0000 7fcb0ce6b700 1 -- 172.21.15.77:0/2460279870 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb0806d3f0 msgr2=0x7fcb0806d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:46:52.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.099+0000 7fcb0ce6b700 1 --2- 172.21.15.77:0/2460279870 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb0806d3f0 0x7fcb0806d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:52.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.099+0000 7fcb0ce6b700 1 -- 172.21.15.77:0/2460279870 >> v1:172.21.15.77:6789/0 conn(0x7fcb080c5930 legacy=0x7fcb08155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:52.104 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.099+0000 7fcb0ce6b700 1 -- 172.21.15.77:0/2460279870 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb080c3540 con 0x7fcb080c2a90 2022-04-03T19:46:52.104 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.099+0000 7fcb0ce6b700 1 --2- 172.21.15.77:0/2460279870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb080c2a90 0x7fcb0806c040 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009e60 tx=0x7fcb00009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8908b16b8f4fa3ae server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:52.104 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.099+0000 7fcb06ffd700 1 -- 172.21.15.77:0/2460279870 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0000e070 con 0x7fcb080c2a90 2022-04-03T19:46:52.104 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.099+0000 7fcb06ffd700 1 -- 172.21.15.77:0/2460279870 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0000bbf0 con 0x7fcb080c2a90 2022-04-03T19:46:52.105 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.099+0000 7fcb06ffd700 1 -- 172.21.15.77:0/2460279870 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb00012430 con 0x7fcb080c2a90 2022-04-03T19:46:52.105 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.100+0000 7fcb0d66c700 1 -- 172.21.15.77:0/2460279870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb080c2a90 msgr2=0x7fcb0806c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:52.105 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.100+0000 7fcb0d66c700 1 --2- 172.21.15.77:0/2460279870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb080c2a90 0x7fcb0806c040 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009e60 tx=0x7fcb00009780 comp rx=0 tx=0).stop 2022-04-03T19:46:52.105 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.100+0000 7fcb0d66c700 1 -- 172.21.15.77:0/2460279870 shutdown_connections 2022-04-03T19:46:52.106 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.100+0000 7fcb0d66c700 1 --2- 172.21.15.77:0/2460279870 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb080c2a90 0x7fcb0806c040 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:52.106 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.100+0000 7fcb0d66c700 1 --2- 172.21.15.77:0/2460279870 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb0806d3f0 0x7fcb0806d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:52.106 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.100+0000 7fcb0d66c700 1 -- 172.21.15.77:0/2460279870 >> 172.21.15.77:0/2460279870 conn(0x7fcb080cb720 msgr2=0x7fcb080cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:52.106 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.100+0000 7fcb0d66c700 1 -- 172.21.15.77:0/2460279870 shutdown_connections 2022-04-03T19:46:52.106 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.100+0000 7fcb0d66c700 1 -- 172.21.15.77:0/2460279870 wait complete. 2022-04-03T19:46:52.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.102+0000 7fcb0d66c700 1 Processor -- start 2022-04-03T19:46:52.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.102+0000 7fcb0d66c700 1 -- start start 2022-04-03T19:46:52.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.103+0000 7fcb0d66c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb0806d3f0 0x7fcb08165ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:52.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.103+0000 7fcb0d66c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb080c2a90 0x7fcb080ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:52.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.103+0000 7fcb0d66c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb080c5930 0x7fcb0814d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:52.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.103+0000 7fcb0d66c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb08169810 con 0x7fcb0806d3f0 2022-04-03T19:46:52.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.103+0000 7fcb0d66c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb08165a40 con 0x7fcb080c5930 2022-04-03T19:46:52.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.103+0000 7fcb0d66c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb080ca220 con 0x7fcb080c2a90 2022-04-03T19:46:52.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.103+0000 7fcb07fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb0806d3f0 0x7fcb08165ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:52.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.103+0000 7fcb07fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb0806d3f0 0x7fcb08165ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42774/0 (socket says 172.21.15.77:42774) 2022-04-03T19:46:52.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.103+0000 7fcb07fff700 1 -- 172.21.15.77:0/3455649652 learned_addr learned my addr 172.21.15.77:0/3455649652 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:52.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.103+0000 7fcb077fe700 1 --2- 172.21.15.77:0/3455649652 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb080c2a90 0x7fcb080ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:52.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.103+0000 7fcb0ce6b700 1 --2- 172.21.15.77:0/3455649652 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb080c5930 0x7fcb0814d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:52.113 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.104+0000 7fcb07fff700 1 -- 172.21.15.77:0/3455649652 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb080c2a90 msgr2=0x7fcb080ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:52.113 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.104+0000 7fcb07fff700 1 --2- 172.21.15.77:0/3455649652 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb080c2a90 0x7fcb080ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:52.113 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.104+0000 7fcb07fff700 1 -- 172.21.15.77:0/3455649652 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb080c5930 msgr2=0x7fcb0814d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:52.113 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.104+0000 7fcb07fff700 1 --2- 172.21.15.77:0/3455649652 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb080c5930 0x7fcb0814d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:52.113 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.104+0000 7fcb07fff700 1 -- 172.21.15.77:0/3455649652 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb00009a20 con 0x7fcb0806d3f0 2022-04-03T19:46:52.114 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.104+0000 7fcb07fff700 1 --2- 172.21.15.77:0/3455649652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb0806d3f0 0x7fcb08165ed0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcaf800e900 tx=0x7fcaf80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:52.114 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.104+0000 7fcaeffff700 1 -- 172.21.15.77:0/3455649652 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf8013680 con 0x7fcb0806d3f0 2022-04-03T19:46:52.114 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.104+0000 7fcaeffff700 1 -- 172.21.15.77:0/3455649652 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaf80137e0 con 0x7fcb0806d3f0 2022-04-03T19:46:52.114 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.104+0000 7fcb0d66c700 1 -- 172.21.15.77:0/3455649652 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb080ca4c0 con 0x7fcb0806d3f0 2022-04-03T19:46:52.115 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.104+0000 7fcaeffff700 1 -- 172.21.15.77:0/3455649652 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf8013b80 con 0x7fcb0806d3f0 2022-04-03T19:46:52.115 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.104+0000 7fcb0d66c700 1 -- 172.21.15.77:0/3455649652 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb081653c0 con 0x7fcb0806d3f0 2022-04-03T19:46:52.115 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.106+0000 7fcb0d66c700 1 -- 172.21.15.77:0/3455649652 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb08165d20 con 0x7fcb0806d3f0 2022-04-03T19:46:52.115 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.106+0000 7fcaeffff700 1 -- 172.21.15.77:0/3455649652 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fcaf8023070 con 0x7fcb0806d3f0 2022-04-03T19:46:52.115 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.107+0000 7fcaeffff700 1 --2- 172.21.15.77:0/3455649652 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcae8063d10 0x7fcae80661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:52.116 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.107+0000 7fcaeffff700 1 -- 172.21.15.77:0/3455649652 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6202+0+0 (secure 0 0 0) 0x7fcaf8080c30 con 0x7fcb0806d3f0 2022-04-03T19:46:52.116 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.107+0000 7fcb077fe700 1 --2- 172.21.15.77:0/3455649652 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcae8063d10 0x7fcae80661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:52.116 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.108+0000 7fcb077fe700 1 --2- 172.21.15.77:0/3455649652 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcae8063d10 0x7fcae80661c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcb0814ed30 tx=0x7fcafc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:52.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.112+0000 7fcaeffff700 1 -- 172.21.15.77:0/3455649652 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fcaf804f780 con 0x7fcb0806d3f0 2022-04-03T19:46:52.343 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.342+0000 7fcb0d66c700 1 -- 172.21.15.77:0/3455649652 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb080ca650 con 0x7fcb0806d3f0 2022-04-03T19:46:52.344 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.343+0000 7fcaeffff700 1 -- 172.21.15.77:0/3455649652 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+5358 (secure 0 0 0) 0x7fcaf8052da0 con 0x7fcb0806d3f0 2022-04-03T19:46:52.344 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:46:52.344 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:46:50.162574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:reconnect","state_seq":40,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:46:52.346 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.345+0000 7fcb0d66c700 1 -- 172.21.15.77:0/3455649652 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcae8063d10 msgr2=0x7fcae80661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:52.347 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.345+0000 7fcb0d66c700 1 --2- 172.21.15.77:0/3455649652 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcae8063d10 0x7fcae80661c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcb0814ed30 tx=0x7fcafc009040 comp rx=0 tx=0).stop 2022-04-03T19:46:52.347 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.345+0000 7fcb0d66c700 1 -- 172.21.15.77:0/3455649652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb0806d3f0 msgr2=0x7fcb08165ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:52.347 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.345+0000 7fcb0d66c700 1 --2- 172.21.15.77:0/3455649652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb0806d3f0 0x7fcb08165ed0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcaf800e900 tx=0x7fcaf80090f0 comp rx=0 tx=0).stop 2022-04-03T19:46:52.347 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.346+0000 7fcb0d66c700 1 -- 172.21.15.77:0/3455649652 shutdown_connections 2022-04-03T19:46:52.348 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.346+0000 7fcb0d66c700 1 --2- 172.21.15.77:0/3455649652 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb080c5930 0x7fcb0814d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:52.348 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.346+0000 7fcb0d66c700 1 --2- 172.21.15.77:0/3455649652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcb0806d3f0 0x7fcb08165ed0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:52.348 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.346+0000 7fcb0d66c700 1 --2- 172.21.15.77:0/3455649652 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcae8063d10 0x7fcae80661c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:52.348 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.346+0000 7fcb0d66c700 1 --2- 172.21.15.77:0/3455649652 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb080c2a90 0x7fcb080ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:52.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.346+0000 7fcb0d66c700 1 -- 172.21.15.77:0/3455649652 >> 172.21.15.77:0/3455649652 conn(0x7fcb080cb720 msgr2=0x7fcb0814f280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:52.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.346+0000 7fcb0d66c700 1 -- 172.21.15.77:0/3455649652 shutdown_connections 2022-04-03T19:46:52.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:52.346+0000 7fcb0d66c700 1 -- 172.21.15.77:0/3455649652 wait complete. 2022-04-03T19:46:52.350 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 10 2022-04-03T19:46:52.372 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:46:54.374 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:46:54.580 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.579+0000 7fd3c292e700 1 Processor -- start 2022-04-03T19:46:54.580 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.580+0000 7fd3c292e700 1 -- start start 2022-04-03T19:46:54.581 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.581+0000 7fd3c292e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3bc1584b0 0x7fd3bc14eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:54.582 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.581+0000 7fd3c292e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3bc06d260 con 0x7fd3bc06bbe0 2022-04-03T19:46:54.582 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.581+0000 7fd3c292e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3bc0c6330 con 0x7fd3bc06bfb0 2022-04-03T19:46:54.582 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.581+0000 7fd3c292e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc159450 con 0x7fd3bc1584b0 2022-04-03T19:46:54.582 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.581+0000 7fd3c192c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd3bc06bbe0 0x7fd3bc06d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38406/0 (socket says 172.21.15.77:38406) 2022-04-03T19:46:54.582 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.581+0000 7fd3c192c700 1 -- 172.21.15.77:0/1310440929 learned_addr learned my addr 172.21.15.77:0/1310440929 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:46:54.583 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.581+0000 7fd3c212d700 1 --2- 172.21.15.77:0/1310440929 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3bc1584b0 0x7fd3bc14eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:54.585 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.581+0000 7fd3c092a700 1 -- 172.21.15.77:0/1310440929 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16452605 0 0) 0x7fd3bc06d260 con 0x7fd3bc06bbe0 2022-04-03T19:46:54.585 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.582+0000 7fd3c092a700 1 -- 172.21.15.77:0/1310440929 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3a40033f0 con 0x7fd3bc06bbe0 2022-04-03T19:46:54.585 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.582+0000 7fd3c092a700 1 -- 172.21.15.77:0/1310440929 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902332475 0 0) 0x7fd3bc0c6330 con 0x7fd3bc06bfb0 2022-04-03T19:46:54.586 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.582+0000 7fd3c092a700 1 -- 172.21.15.77:0/1310440929 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3bc06d260 con 0x7fd3bc06bfb0 2022-04-03T19:46:54.586 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.582+0000 7fd3c212d700 1 -- 172.21.15.77:0/1310440929 >> v1:172.21.15.186:6789/0 conn(0x7fd3bc06bfb0 legacy=0x7fd3bc06c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:54.586 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.582+0000 7fd3c212d700 1 -- 172.21.15.77:0/1310440929 >> v1:172.21.15.77:6789/0 conn(0x7fd3bc06bbe0 legacy=0x7fd3bc06d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:54.586 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.582+0000 7fd3c212d700 1 -- 172.21.15.77:0/1310440929 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3bc158910 con 0x7fd3bc1584b0 2022-04-03T19:46:54.586 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.582+0000 7fd3c212d700 1 --2- 172.21.15.77:0/1310440929 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3bc1584b0 0x7fd3bc14eea0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4009e90 tx=0x7fd3b4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7ceaced443dfcd3 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:54.587 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.582+0000 7fd3c092a700 1 -- 172.21.15.77:0/1310440929 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b400e070 con 0x7fd3bc1584b0 2022-04-03T19:46:54.587 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.582+0000 7fd3c092a700 1 -- 172.21.15.77:0/1310440929 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b400bb80 con 0x7fd3bc1584b0 2022-04-03T19:46:54.587 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.582+0000 7fd3c092a700 1 -- 172.21.15.77:0/1310440929 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4012430 con 0x7fd3bc1584b0 2022-04-03T19:46:54.587 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.583+0000 7fd3c292e700 1 -- 172.21.15.77:0/1310440929 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3bc1584b0 msgr2=0x7fd3bc14eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:54.588 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.583+0000 7fd3c292e700 1 --2- 172.21.15.77:0/1310440929 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3bc1584b0 0x7fd3bc14eea0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4009e90 tx=0x7fd3b4009780 comp rx=0 tx=0).stop 2022-04-03T19:46:54.588 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.583+0000 7fd3c292e700 1 -- 172.21.15.77:0/1310440929 shutdown_connections 2022-04-03T19:46:54.588 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.583+0000 7fd3c292e700 1 --2- 172.21.15.77:0/1310440929 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3bc1584b0 0x7fd3bc14eea0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:54.588 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.583+0000 7fd3c292e700 1 -- 172.21.15.77:0/1310440929 >> 172.21.15.77:0/1310440929 conn(0x7fd3bc0cb6b0 msgr2=0x7fd3bc0cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:54.588 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.583+0000 7fd3c292e700 1 -- 172.21.15.77:0/1310440929 shutdown_connections 2022-04-03T19:46:54.589 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.583+0000 7fd3c292e700 1 -- 172.21.15.77:0/1310440929 wait complete. 2022-04-03T19:46:54.589 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.585+0000 7fd3c292e700 1 Processor -- start 2022-04-03T19:46:54.589 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.585+0000 7fd3c292e700 1 -- start start 2022-04-03T19:46:54.592 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.586+0000 7fd3c292e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3bc06bbe0 0x7fd3bc1653c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:54.592 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.586+0000 7fd3c292e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3bc06bfb0 0x7fd3bc165900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:54.592 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.586+0000 7fd3c292e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3bc1584b0 0x7fd3bc0bf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:54.592 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.586+0000 7fd3c292e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc164f70 con 0x7fd3bc06bbe0 2022-04-03T19:46:54.593 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.586+0000 7fd3c292e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc0c2d30 con 0x7fd3bc06bfb0 2022-04-03T19:46:54.593 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.586+0000 7fd3c292e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc150920 con 0x7fd3bc1584b0 2022-04-03T19:46:54.593 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.586+0000 7fd3c192c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3bc06bbe0 0x7fd3bc1653c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:54.593 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.586+0000 7fd3c112b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3bc06bfb0 0x7fd3bc165900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:54.593 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.586+0000 7fd3c192c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3bc06bbe0 0x7fd3bc1653c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42776/0 (socket says 172.21.15.77:42776) 2022-04-03T19:46:54.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.586+0000 7fd3c192c700 1 -- 172.21.15.77:0/4001606047 learned_addr learned my addr 172.21.15.77:0/4001606047 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:54.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.586+0000 7fd3c212d700 1 --2- 172.21.15.77:0/4001606047 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3bc1584b0 0x7fd3bc0bf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:54.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.586+0000 7fd3c192c700 1 -- 172.21.15.77:0/4001606047 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3bc1584b0 msgr2=0x7fd3bc0bf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:54.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.586+0000 7fd3c192c700 1 --2- 172.21.15.77:0/4001606047 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3bc1584b0 0x7fd3bc0bf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:54.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.586+0000 7fd3c192c700 1 -- 172.21.15.77:0/4001606047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3bc06bfb0 msgr2=0x7fd3bc165900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:54.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.586+0000 7fd3c192c700 1 --2- 172.21.15.77:0/4001606047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3bc06bfb0 0x7fd3bc165900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:54.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.586+0000 7fd3c192c700 1 -- 172.21.15.77:0/4001606047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b4009a20 con 0x7fd3bc06bbe0 2022-04-03T19:46:54.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.587+0000 7fd3c192c700 1 --2- 172.21.15.77:0/4001606047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3bc06bbe0 0x7fd3bc1653c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd3b800c450 tx=0x7fd3b8007ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:54.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.587+0000 7fd3a9ffb700 1 -- 172.21.15.77:0/4001606047 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b80070d0 con 0x7fd3bc06bbe0 2022-04-03T19:46:54.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.587+0000 7fd3a9ffb700 1 -- 172.21.15.77:0/4001606047 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b8007250 con 0x7fd3bc06bbe0 2022-04-03T19:46:54.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.587+0000 7fd3a9ffb700 1 -- 172.21.15.77:0/4001606047 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b8007610 con 0x7fd3bc06bbe0 2022-04-03T19:46:54.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.587+0000 7fd3c292e700 1 -- 172.21.15.77:0/4001606047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3bc0c2fa0 con 0x7fd3bc06bbe0 2022-04-03T19:46:54.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.587+0000 7fd3c292e700 1 -- 172.21.15.77:0/4001606047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3bc0c3560 con 0x7fd3bc06bbe0 2022-04-03T19:46:54.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.588+0000 7fd3c292e700 1 -- 172.21.15.77:0/4001606047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3bc158bb0 con 0x7fd3bc06bbe0 2022-04-03T19:46:54.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.589+0000 7fd3a9ffb700 1 -- 172.21.15.77:0/4001606047 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fd3b8019030 con 0x7fd3bc06bbe0 2022-04-03T19:46:54.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.590+0000 7fd3a9ffb700 1 --2- 172.21.15.77:0/4001606047 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd39c063d10 0x7fd39c0661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:54.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.590+0000 7fd3a9ffb700 1 -- 172.21.15.77:0/4001606047 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6202+0+0 (secure 0 0 0) 0x7fd3b807fa90 con 0x7fd3bc06bbe0 2022-04-03T19:46:54.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.590+0000 7fd3c112b700 1 --2- 172.21.15.77:0/4001606047 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd39c063d10 0x7fd39c0661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:54.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.591+0000 7fd3c112b700 1 --2- 172.21.15.77:0/4001606047 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd39c063d10 0x7fd39c0661c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc14a340 tx=0x7fd3ac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:54.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.594+0000 7fd3a9ffb700 1 -- 172.21.15.77:0/4001606047 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fd3b804e5e0 con 0x7fd3bc06bbe0 2022-04-03T19:46:54.829 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.829+0000 7fd3c292e700 1 -- 172.21.15.77:0/4001606047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3bc04b050 con 0x7fd3bc06bbe0 2022-04-03T19:46:54.830 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.829+0000 7fd3a9ffb700 1 -- 172.21.15.77:0/4001606047 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+5358 (secure 0 0 0) 0x7fd3b8017070 con 0x7fd3bc06bbe0 2022-04-03T19:46:54.830 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:46:54.830 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:46:50.162574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:reconnect","state_seq":40,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:46:54.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.832+0000 7fd3c292e700 1 -- 172.21.15.77:0/4001606047 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd39c063d10 msgr2=0x7fd39c0661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:54.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.832+0000 7fd3c292e700 1 --2- 172.21.15.77:0/4001606047 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd39c063d10 0x7fd39c0661c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc14a340 tx=0x7fd3ac009040 comp rx=0 tx=0).stop 2022-04-03T19:46:54.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.832+0000 7fd3c292e700 1 -- 172.21.15.77:0/4001606047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3bc06bbe0 msgr2=0x7fd3bc1653c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:54.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.832+0000 7fd3c292e700 1 --2- 172.21.15.77:0/4001606047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3bc06bbe0 0x7fd3bc1653c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd3b800c450 tx=0x7fd3b8007ca0 comp rx=0 tx=0).stop 2022-04-03T19:46:54.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.832+0000 7fd3c292e700 1 -- 172.21.15.77:0/4001606047 shutdown_connections 2022-04-03T19:46:54.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.832+0000 7fd3c292e700 1 --2- 172.21.15.77:0/4001606047 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd39c063d10 0x7fd39c0661c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:54.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.832+0000 7fd3c292e700 1 --2- 172.21.15.77:0/4001606047 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3bc1584b0 0x7fd3bc0bf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:54.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.832+0000 7fd3c292e700 1 --2- 172.21.15.77:0/4001606047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3bc06bbe0 0x7fd3bc1653c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:54.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.832+0000 7fd3c292e700 1 --2- 172.21.15.77:0/4001606047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3bc06bfb0 0x7fd3bc165900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:54.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.832+0000 7fd3c292e700 1 -- 172.21.15.77:0/4001606047 >> 172.21.15.77:0/4001606047 conn(0x7fd3bc0cb6b0 msgr2=0x7fd3bc1495c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:54.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.832+0000 7fd3c292e700 1 -- 172.21.15.77:0/4001606047 shutdown_connections 2022-04-03T19:46:54.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:54.832+0000 7fd3c292e700 1 -- 172.21.15.77:0/4001606047 wait complete. 2022-04-03T19:46:54.835 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 10 2022-04-03T19:46:54.856 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:46:56.857 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:46:57.063 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.063+0000 7f8969595700 1 Processor -- start 2022-04-03T19:46:57.064 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.063+0000 7f8969595700 1 -- start start 2022-04-03T19:46:57.066 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.064+0000 7f8969595700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f896406bbe0 0x7f89641584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:57.066 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.064+0000 7f8969595700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f896406c500 0x7f896414eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:57.066 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.064+0000 7f8969595700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f896406d260 con 0x7f896406bfb0 2022-04-03T19:46:57.067 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.064+0000 7f8969595700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8964159450 con 0x7f896406c500 2022-04-03T19:46:57.067 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.064+0000 7f8969595700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89640c6330 con 0x7f896406bbe0 2022-04-03T19:46:57.067 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.064+0000 7f8968d94700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f896406c500 0x7f896414eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:57.067 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.065+0000 7f8968d94700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f896406c500 0x7f896414eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42778/0 (socket says 172.21.15.77:42778) 2022-04-03T19:46:57.067 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.065+0000 7f8963fff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f896406bbe0 0x7f89641584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:57.068 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.065+0000 7f8968d94700 1 -- 172.21.15.77:0/592088200 learned_addr learned my addr 172.21.15.77:0/592088200 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:57.068 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.065+0000 7f8968d94700 1 -- 172.21.15.77:0/592088200 >> v2:172.21.15.186:3300/0 conn(0x7f896406bbe0 msgr2=0x7f89641584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:46:57.068 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.065+0000 7f8968d94700 1 --2- 172.21.15.77:0/592088200 >> v2:172.21.15.186:3300/0 conn(0x7f896406bbe0 0x7f89641584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:57.068 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.065+0000 7f8968d94700 1 -- 172.21.15.77:0/592088200 >> v1:172.21.15.186:6789/0 conn(0x7f896406bfb0 legacy=0x7f896406d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:57.069 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.065+0000 7f8968d94700 1 -- 172.21.15.77:0/592088200 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8964158bb0 con 0x7f896406c500 2022-04-03T19:46:57.069 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.065+0000 7f8962ffd700 1 -- 172.21.15.77:0/592088200 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074290916 0 0) 0x7f896406d260 con 0x7f896406bfb0 2022-04-03T19:46:57.069 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.065+0000 7f8968d94700 1 --2- 172.21.15.77:0/592088200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f896406c500 0x7f896414eea0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8964088d40 tx=0x7f895c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a3b186272c4b49 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:57.069 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.066+0000 7f8962ffd700 1 -- 172.21.15.77:0/592088200 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895c00e070 con 0x7f896406c500 2022-04-03T19:46:57.069 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.066+0000 7f8962ffd700 1 -- 172.21.15.77:0/592088200 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f895c00ba20 con 0x7f896406c500 2022-04-03T19:46:57.070 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.066+0000 7f8962ffd700 1 -- 172.21.15.77:0/592088200 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895c0092e0 con 0x7f896406c500 2022-04-03T19:46:57.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.066+0000 7f8969595700 1 -- 172.21.15.77:0/592088200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f896406c500 msgr2=0x7f896414eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:57.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.066+0000 7f8969595700 1 --2- 172.21.15.77:0/592088200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f896406c500 0x7f896414eea0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8964088d40 tx=0x7f895c009840 comp rx=0 tx=0).stop 2022-04-03T19:46:57.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.066+0000 7f8969595700 1 -- 172.21.15.77:0/592088200 shutdown_connections 2022-04-03T19:46:57.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.066+0000 7f8969595700 1 --2- 172.21.15.77:0/592088200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f896406c500 0x7f896414eea0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:57.074 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.066+0000 7f8969595700 1 --2- 172.21.15.77:0/592088200 >> v2:172.21.15.186:3300/0 conn(0x7f896406bbe0 0x7f89641584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:57.074 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.066+0000 7f8969595700 1 -- 172.21.15.77:0/592088200 >> 172.21.15.77:0/592088200 conn(0x7f89640cb6b0 msgr2=0x7f89640cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:57.074 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.066+0000 7f8969595700 1 -- 172.21.15.77:0/592088200 shutdown_connections 2022-04-03T19:46:57.074 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.066+0000 7f8969595700 1 -- 172.21.15.77:0/592088200 wait complete. 2022-04-03T19:46:57.075 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.068+0000 7f8969595700 1 Processor -- start 2022-04-03T19:46:57.075 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.068+0000 7f8969595700 1 -- start start 2022-04-03T19:46:57.075 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.069+0000 7f8969595700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f896406bbe0 0x7f8964166cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:57.075 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.069+0000 7f8969595700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f896406bfb0 0x7f89641634b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:57.075 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.069+0000 7f8969595700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f896406c500 0x7f89641639f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:57.076 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.069+0000 7f8969595700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8964163020 con 0x7f896406c500 2022-04-03T19:46:57.076 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.069+0000 7f8969595700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896414a640 con 0x7f896406bfb0 2022-04-03T19:46:57.076 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.069+0000 7f8969595700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89641676d0 con 0x7f896406bbe0 2022-04-03T19:46:57.076 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.069+0000 7f8963fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f896406bbe0 0x7f8964166cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:57.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.069+0000 7f89637fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f896406bfb0 0x7f89641634b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:57.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.069+0000 7f8963fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f896406bbe0 0x7f8964166cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60078/0 (socket says 172.21.15.77:60078) 2022-04-03T19:46:57.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.069+0000 7f8963fff700 1 -- 172.21.15.77:0/3377613935 learned_addr learned my addr 172.21.15.77:0/3377613935 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:57.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.069+0000 7f8968d94700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f896406c500 0x7f89641639f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:57.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.070+0000 7f8963fff700 1 -- 172.21.15.77:0/3377613935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f896406bfb0 msgr2=0x7f89641634b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.070+0000 7f8963fff700 1 --2- 172.21.15.77:0/3377613935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f896406bfb0 0x7f89641634b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.070+0000 7f8963fff700 1 -- 172.21.15.77:0/3377613935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f896406c500 msgr2=0x7f89641639f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.070+0000 7f8963fff700 1 --2- 172.21.15.77:0/3377613935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f896406c500 0x7f89641639f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:57.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.070+0000 7f8963fff700 1 -- 172.21.15.77:0/3377613935 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f895c009ae0 con 0x7f896406bbe0 2022-04-03T19:46:57.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.070+0000 7f8963fff700 1 --2- 172.21.15.77:0/3377613935 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f896406bbe0 0x7f8964166cc0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f895400e900 tx=0x7f8954009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:57.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.070+0000 7f894bfff700 1 -- 172.21.15.77:0/3377613935 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8954013680 con 0x7f896406bbe0 2022-04-03T19:46:57.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.070+0000 7f894bfff700 1 -- 172.21.15.77:0/3377613935 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89540137e0 con 0x7f896406bbe0 2022-04-03T19:46:57.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.070+0000 7f894bfff700 1 -- 172.21.15.77:0/3377613935 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8954013b80 con 0x7f896406bbe0 2022-04-03T19:46:57.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.070+0000 7f8969595700 1 -- 172.21.15.77:0/3377613935 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89641679b0 con 0x7f896406bbe0 2022-04-03T19:46:57.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.071+0000 7f8969595700 1 -- 172.21.15.77:0/3377613935 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8964167f70 con 0x7f896406bbe0 2022-04-03T19:46:57.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.071+0000 7f8969595700 1 -- 172.21.15.77:0/3377613935 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8964163300 con 0x7f896406bbe0 2022-04-03T19:46:57.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.073+0000 7f894bfff700 1 -- 172.21.15.77:0/3377613935 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f8954019030 con 0x7f896406bbe0 2022-04-03T19:46:57.081 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.073+0000 7f894bfff700 1 --2- 172.21.15.77:0/3377613935 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8944063d10 0x7f89440661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:57.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.073+0000 7f894bfff700 1 -- 172.21.15.77:0/3377613935 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f8954080ac0 con 0x7f896406bbe0 2022-04-03T19:46:57.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.074+0000 7f89637fe700 1 --2- 172.21.15.77:0/3377613935 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8944063d10 0x7f89440661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:57.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.074+0000 7f89637fe700 1 --2- 172.21.15.77:0/3377613935 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8944063d10 0x7f89440661c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f896414e280 tx=0x7f8958009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:57.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.077+0000 7f894bfff700 1 -- 172.21.15.77:0/3377613935 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f895404f610 con 0x7f896406bbe0 2022-04-03T19:46:57.315 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.314+0000 7f8969595700 1 -- 172.21.15.77:0/3377613935 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8964168190 con 0x7f896406bbe0 2022-04-03T19:46:57.316 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.316+0000 7f894bfff700 1 -- 172.21.15.77:0/3377613935 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+5355 (secure 0 0 0) 0x7f8954021090 con 0x7f896406bbe0 2022-04-03T19:46:57.316 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:46:57.317 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:46:56.175844+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:rejoin","state_seq":42,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:46:57.319 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.318+0000 7f8969595700 1 -- 172.21.15.77:0/3377613935 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8944063d10 msgr2=0x7f89440661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:57.319 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.318+0000 7f8969595700 1 --2- 172.21.15.77:0/3377613935 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8944063d10 0x7f89440661c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f896414e280 tx=0x7f8958009040 comp rx=0 tx=0).stop 2022-04-03T19:46:57.319 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.318+0000 7f8969595700 1 -- 172.21.15.77:0/3377613935 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f896406bbe0 msgr2=0x7f8964166cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:57.319 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.318+0000 7f8969595700 1 --2- 172.21.15.77:0/3377613935 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f896406bbe0 0x7f8964166cc0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f895400e900 tx=0x7f8954009060 comp rx=0 tx=0).stop 2022-04-03T19:46:57.320 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.318+0000 7f8969595700 1 -- 172.21.15.77:0/3377613935 shutdown_connections 2022-04-03T19:46:57.320 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.318+0000 7f8969595700 1 --2- 172.21.15.77:0/3377613935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f896406c500 0x7f89641639f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:57.320 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.318+0000 7f8969595700 1 --2- 172.21.15.77:0/3377613935 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8944063d10 0x7f89440661c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:57.320 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.318+0000 7f8969595700 1 --2- 172.21.15.77:0/3377613935 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f896406bbe0 0x7f8964166cc0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:57.320 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.318+0000 7f8969595700 1 --2- 172.21.15.77:0/3377613935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f896406bfb0 0x7f89641634b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:57.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.318+0000 7f8969595700 1 -- 172.21.15.77:0/3377613935 >> 172.21.15.77:0/3377613935 conn(0x7f89640cb6b0 msgr2=0x7f896414b430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:57.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.318+0000 7f8969595700 1 -- 172.21.15.77:0/3377613935 shutdown_connections 2022-04-03T19:46:57.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:57.318+0000 7f8969595700 1 -- 172.21.15.77:0/3377613935 wait complete. 2022-04-03T19:46:57.322 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 11 2022-04-03T19:46:57.343 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 11, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-04-03T19:44:21.275716+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 11, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4303": { "addr": "172.21.15.186:6837/1407029827", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 1407029827, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 1407029827, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4303, "incarnation": 9, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 42 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-04-03T19:46:56.175844+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4303 } } } ], "standbys": [ { "addr": "172.21.15.77:6834/2116223509", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6832", "nonce": 2116223509, "type": "v2" }, { "addr": "172.21.15.77:6834", "nonce": 2116223509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4245, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6836/2374276747", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2374276747, "type": "v2" }, { "addr": "172.21.15.186:6836", "nonce": 2374276747, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4259, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/610990853", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6838", "nonce": 610990853, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 610990853, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4265, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.77:6835/170053412", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6833", "nonce": 170053412, "type": "v2" }, { "addr": "172.21.15.77:6835", "nonce": 170053412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4271, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-04-03T19:46:59.344 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:46:59.551 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.551+0000 7f9951ec9700 1 Processor -- start 2022-04-03T19:46:59.551 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.551+0000 7f9951ec9700 1 -- start start 2022-04-03T19:46:59.553 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.552+0000 7f9951ec9700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f994c0c1170 0x7f994c1527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:59.553 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.552+0000 7f9951ec9700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f994c0c3b10 0x7f994c15c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:59.553 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.552+0000 7f9951ec9700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994c0c3710 con 0x7f994c155370 2022-04-03T19:46:59.553 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.552+0000 7f9951ec9700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c152330 con 0x7f994c0c1170 2022-04-03T19:46:59.554 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.552+0000 7f9951ec9700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c0c1540 con 0x7f994c0c3b10 2022-04-03T19:46:59.554 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.552+0000 7f9950ec7700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f994c0c1170 0x7f994c1527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:59.554 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.552+0000 7f9950ec7700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f994c0c1170 0x7f994c1527c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42782/0 (socket says 172.21.15.77:42782) 2022-04-03T19:46:59.554 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.552+0000 7f9950ec7700 1 -- 172.21.15.77:0/359026444 learned_addr learned my addr 172.21.15.77:0/359026444 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:59.554 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.552+0000 7f99516c8700 1 --2- 172.21.15.77:0/359026444 >> v2:172.21.15.186:3300/0 conn(0x7f994c0c3b10 0x7f994c15c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:59.556 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.553+0000 7f9950ec7700 1 -- 172.21.15.77:0/359026444 >> v2:172.21.15.186:3300/0 conn(0x7f994c0c3b10 msgr2=0x7f994c15c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:46:59.557 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.553+0000 7f9950ec7700 1 --2- 172.21.15.77:0/359026444 >> v2:172.21.15.186:3300/0 conn(0x7f994c0c3b10 0x7f994c15c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:59.557 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.553+0000 7f9950ec7700 1 -- 172.21.15.77:0/359026444 >> v1:172.21.15.77:6789/0 conn(0x7f994c155370 legacy=0x7f994c069110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:59.557 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.553+0000 7f9950ec7700 1 -- 172.21.15.77:0/359026444 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c0c4270 con 0x7f994c0c1170 2022-04-03T19:46:59.557 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.553+0000 7f9950ec7700 1 --2- 172.21.15.77:0/359026444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c0c1170 0x7f994c1527c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f994c0690a0 tx=0x7f99400090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92467e645b10583 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:59.558 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.553+0000 7f994b7fe700 1 -- 172.21.15.77:0/359026444 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9940004030 con 0x7f994c0c1170 2022-04-03T19:46:59.558 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.554+0000 7f994b7fe700 1 -- 172.21.15.77:0/359026444 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9940004190 con 0x7f994c0c1170 2022-04-03T19:46:59.558 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.554+0000 7f994b7fe700 1 -- 172.21.15.77:0/359026444 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9940011630 con 0x7f994c0c1170 2022-04-03T19:46:59.558 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.554+0000 7f9951ec9700 1 -- 172.21.15.77:0/359026444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c0c1170 msgr2=0x7f994c1527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:59.559 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.554+0000 7f9951ec9700 1 --2- 172.21.15.77:0/359026444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c0c1170 0x7f994c1527c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f994c0690a0 tx=0x7f99400090f0 comp rx=0 tx=0).stop 2022-04-03T19:46:59.559 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.554+0000 7f9951ec9700 1 -- 172.21.15.77:0/359026444 shutdown_connections 2022-04-03T19:46:59.559 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.554+0000 7f9951ec9700 1 --2- 172.21.15.77:0/359026444 >> v2:172.21.15.186:3300/0 conn(0x7f994c0c3b10 0x7f994c15c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:59.559 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.554+0000 7f9951ec9700 1 --2- 172.21.15.77:0/359026444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c0c1170 0x7f994c1527c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:59.559 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.554+0000 7f9951ec9700 1 -- 172.21.15.77:0/359026444 >> 172.21.15.77:0/359026444 conn(0x7f994c066da0 msgr2=0x7f994c0671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:59.560 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.554+0000 7f9951ec9700 1 -- 172.21.15.77:0/359026444 shutdown_connections 2022-04-03T19:46:59.560 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.554+0000 7f9951ec9700 1 -- 172.21.15.77:0/359026444 wait complete. 2022-04-03T19:46:59.560 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.556+0000 7f9951ec9700 1 Processor -- start 2022-04-03T19:46:59.560 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.556+0000 7f9951ec9700 1 -- start start 2022-04-03T19:46:59.563 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.557+0000 7f9951ec9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c0c1170 0x7f994c0c5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:59.563 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.557+0000 7f9951ec9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f994c0c3b10 0x7f994c146100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:59.564 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.557+0000 7f9951ec9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f994c155370 0x7f994c13c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:59.564 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.557+0000 7f9951ec9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c150a30 con 0x7f994c0c1170 2022-04-03T19:46:59.564 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.557+0000 7f9951ec9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c14fa30 con 0x7f994c0c3b10 2022-04-03T19:46:59.564 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.557+0000 7f9951ec9700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c1425f0 con 0x7f994c155370 2022-04-03T19:46:59.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.557+0000 7f9950ec7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c0c1170 0x7f994c0c5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:59.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.557+0000 7f9950ec7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c0c1170 0x7f994c0c5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42784/0 (socket says 172.21.15.77:42784) 2022-04-03T19:46:59.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.557+0000 7f9950ec7700 1 -- 172.21.15.77:0/824518645 learned_addr learned my addr 172.21.15.77:0/824518645 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:46:59.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.557+0000 7f99516c8700 1 --2- 172.21.15.77:0/824518645 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f994c155370 0x7f994c13c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:59.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.557+0000 7f994bfff700 1 --2- 172.21.15.77:0/824518645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f994c0c3b10 0x7f994c146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:59.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.557+0000 7f9950ec7700 1 -- 172.21.15.77:0/824518645 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f994c155370 msgr2=0x7f994c13c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:59.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.557+0000 7f9950ec7700 1 --2- 172.21.15.77:0/824518645 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f994c155370 0x7f994c13c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:59.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.557+0000 7f9950ec7700 1 -- 172.21.15.77:0/824518645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f994c0c3b10 msgr2=0x7f994c146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:59.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.557+0000 7f9950ec7700 1 --2- 172.21.15.77:0/824518645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f994c0c3b10 0x7f994c146100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:59.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.557+0000 7f9950ec7700 1 -- 172.21.15.77:0/824518645 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9940009370 con 0x7f994c0c1170 2022-04-03T19:46:59.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.558+0000 7f9950ec7700 1 --2- 172.21.15.77:0/824518645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c0c1170 0x7f994c0c5160 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9940006fd0 tx=0x7f9940016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:59.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.558+0000 7f9948ff9700 1 -- 172.21.15.77:0/824518645 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99400037c0 con 0x7f994c0c1170 2022-04-03T19:46:59.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.558+0000 7f9948ff9700 1 -- 172.21.15.77:0/824518645 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9940003940 con 0x7f994c0c1170 2022-04-03T19:46:59.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.558+0000 7f9951ec9700 1 -- 172.21.15.77:0/824518645 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994c14fc80 con 0x7f994c0c1170 2022-04-03T19:46:59.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.558+0000 7f9948ff9700 1 -- 172.21.15.77:0/824518645 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9940003d00 con 0x7f994c0c1170 2022-04-03T19:46:59.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.558+0000 7f9951ec9700 1 -- 172.21.15.77:0/824518645 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994c141fa0 con 0x7f994c0c1170 2022-04-03T19:46:59.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.559+0000 7f9951ec9700 1 -- 172.21.15.77:0/824518645 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994c06c880 con 0x7f994c0c1170 2022-04-03T19:46:59.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.560+0000 7f9948ff9700 1 -- 172.21.15.77:0/824518645 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f994002b030 con 0x7f994c0c1170 2022-04-03T19:46:59.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.561+0000 7f9948ff9700 1 --2- 172.21.15.77:0/824518645 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9934063ce0 0x7f9934066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:46:59.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.561+0000 7f9948ff9700 1 -- 172.21.15.77:0/824518645 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f99400823a0 con 0x7f994c0c1170 2022-04-03T19:46:59.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.562+0000 7f994bfff700 1 --2- 172.21.15.77:0/824518645 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9934063ce0 0x7f9934066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:46:59.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.562+0000 7f994bfff700 1 --2- 172.21.15.77:0/824518645 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9934063ce0 0x7f9934066190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f994c0c8240 tx=0x7f9938009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:46:59.570 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.566+0000 7f9948ff9700 1 -- 172.21.15.77:0/824518645 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f9940050e00 con 0x7f994c0c1170 2022-04-03T19:46:59.797 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.796+0000 7f9951ec9700 1 -- 172.21.15.77:0/824518645 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f994c04b050 con 0x7f994c0c1170 2022-04-03T19:46:59.797 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.797+0000 7f9948ff9700 1 -- 172.21.15.77:0/824518645 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+5355 (secure 0 0 0) 0x7f9940003e60 con 0x7f994c0c1170 2022-04-03T19:46:59.798 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:46:59.798 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:46:56.175844+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:rejoin","state_seq":42,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:46:59.800 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.799+0000 7f9951ec9700 1 -- 172.21.15.77:0/824518645 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9934063ce0 msgr2=0x7f9934066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:59.800 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.799+0000 7f9951ec9700 1 --2- 172.21.15.77:0/824518645 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9934063ce0 0x7f9934066190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f994c0c8240 tx=0x7f9938009040 comp rx=0 tx=0).stop 2022-04-03T19:46:59.801 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.799+0000 7f9951ec9700 1 -- 172.21.15.77:0/824518645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c0c1170 msgr2=0x7f994c0c5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:46:59.801 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.799+0000 7f9951ec9700 1 --2- 172.21.15.77:0/824518645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c0c1170 0x7f994c0c5160 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9940006fd0 tx=0x7f9940016000 comp rx=0 tx=0).stop 2022-04-03T19:46:59.801 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.799+0000 7f9951ec9700 1 -- 172.21.15.77:0/824518645 shutdown_connections 2022-04-03T19:46:59.801 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.799+0000 7f9951ec9700 1 --2- 172.21.15.77:0/824518645 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9934063ce0 0x7f9934066190 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:59.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.799+0000 7f9951ec9700 1 --2- 172.21.15.77:0/824518645 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f994c155370 0x7f994c13c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:59.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.799+0000 7f9951ec9700 1 --2- 172.21.15.77:0/824518645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f994c0c1170 0x7f994c0c5160 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:59.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.799+0000 7f9951ec9700 1 --2- 172.21.15.77:0/824518645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f994c0c3b10 0x7f994c146100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:46:59.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.799+0000 7f9951ec9700 1 -- 172.21.15.77:0/824518645 >> 172.21.15.77:0/824518645 conn(0x7f994c066da0 msgr2=0x7f994c1557c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:46:59.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.799+0000 7f9951ec9700 1 -- 172.21.15.77:0/824518645 shutdown_connections 2022-04-03T19:46:59.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:46:59.800+0000 7f9951ec9700 1 -- 172.21.15.77:0/824518645 wait complete. 2022-04-03T19:46:59.803 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 11 2022-04-03T19:46:59.825 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:47:01.826 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:47:02.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.032+0000 7f02969bc700 1 Processor -- start 2022-04-03T19:47:02.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.032+0000 7f02969bc700 1 -- start start 2022-04-03T19:47:02.036 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.034+0000 7f02969bc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02900c1170 0x7f02901527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:02.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.034+0000 7f02969bc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0290155370 0x7f0290069110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:02.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.034+0000 7f02969bc700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02900c3710 con 0x7f02900c3b10 2022-04-03T19:47:02.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.034+0000 7f02969bc700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0290152330 con 0x7f0290155370 2022-04-03T19:47:02.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.034+0000 7f02969bc700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02900c1540 con 0x7f02900c1170 2022-04-03T19:47:02.038 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.034+0000 7f02951b9700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0290155370 0x7f0290069110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:02.038 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.034+0000 7f02959ba700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02900c1170 0x7f02901527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:02.038 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.034+0000 7f02951b9700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0290155370 0x7f0290069110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42786/0 (socket says 172.21.15.77:42786) 2022-04-03T19:47:02.038 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.034+0000 7f02951b9700 1 -- 172.21.15.77:0/3630790802 learned_addr learned my addr 172.21.15.77:0/3630790802 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:47:02.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.034+0000 7f02961bb700 1 --1- 172.21.15.77:0/3630790802 >> v1:172.21.15.77:6789/0 conn(0x7f02900c3b10 0x7f02900c71f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38410/0 (socket says 172.21.15.77:38410) 2022-04-03T19:47:02.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.034+0000 7f02951b9700 1 -- 172.21.15.77:0/3630790802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02900c1170 msgr2=0x7f02901527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:47:02.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.034+0000 7f02951b9700 1 --2- 172.21.15.77:0/3630790802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02900c1170 0x7f02901527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:02.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.034+0000 7f02951b9700 1 -- 172.21.15.77:0/3630790802 >> v1:172.21.15.77:6789/0 conn(0x7f02900c3b10 legacy=0x7f02900c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:02.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.034+0000 7f02951b9700 1 -- 172.21.15.77:0/3630790802 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02900c4270 con 0x7f0290155370 2022-04-03T19:47:02.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.035+0000 7f02951b9700 1 --2- 172.21.15.77:0/3630790802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0290155370 0x7f0290069110 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0280006fd0 tx=0x7f02800090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b00db673e856db4f server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:02.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.035+0000 7f02949b8700 1 -- 172.21.15.77:0/3630790802 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0280004030 con 0x7f0290155370 2022-04-03T19:47:02.041 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.035+0000 7f02949b8700 1 -- 172.21.15.77:0/3630790802 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02800041d0 con 0x7f0290155370 2022-04-03T19:47:02.041 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.035+0000 7f02949b8700 1 -- 172.21.15.77:0/3630790802 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0280011630 con 0x7f0290155370 2022-04-03T19:47:02.041 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.035+0000 7f02969bc700 1 -- 172.21.15.77:0/3630790802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0290155370 msgr2=0x7f0290069110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:02.041 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.035+0000 7f02969bc700 1 --2- 172.21.15.77:0/3630790802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0290155370 0x7f0290069110 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0280006fd0 tx=0x7f02800090f0 comp rx=0 tx=0).stop 2022-04-03T19:47:02.042 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.036+0000 7f02969bc700 1 -- 172.21.15.77:0/3630790802 shutdown_connections 2022-04-03T19:47:02.042 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.036+0000 7f02969bc700 1 --2- 172.21.15.77:0/3630790802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02900c1170 0x7f02901527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:02.042 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.036+0000 7f02969bc700 1 --2- 172.21.15.77:0/3630790802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0290155370 0x7f0290069110 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:02.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.036+0000 7f02969bc700 1 -- 172.21.15.77:0/3630790802 >> 172.21.15.77:0/3630790802 conn(0x7f0290066da0 msgr2=0x7f02900671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:47:02.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.036+0000 7f02969bc700 1 -- 172.21.15.77:0/3630790802 shutdown_connections 2022-04-03T19:47:02.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.036+0000 7f02969bc700 1 -- 172.21.15.77:0/3630790802 wait complete. 2022-04-03T19:47:02.047 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.037+0000 7f02969bc700 1 Processor -- start 2022-04-03T19:47:02.047 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.037+0000 7f02969bc700 1 -- start start 2022-04-03T19:47:02.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.038+0000 7f02969bc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02900c1170 0x7f02900c7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:02.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.038+0000 7f02969bc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02900c3b10 0x7f029014d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:02.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.038+0000 7f02969bc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0290155370 0x7f029014d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:02.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.038+0000 7f02969bc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029014c710 con 0x7f0290155370 2022-04-03T19:47:02.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.038+0000 7f02969bc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02900bc0b0 con 0x7f02900c1170 2022-04-03T19:47:02.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.038+0000 7f02969bc700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02900bc390 con 0x7f02900c3b10 2022-04-03T19:47:02.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.039+0000 7f02961bb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0290155370 0x7f029014d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:02.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.039+0000 7f02961bb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0290155370 0x7f029014d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42788/0 (socket says 172.21.15.77:42788) 2022-04-03T19:47:02.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.039+0000 7f02961bb700 1 -- 172.21.15.77:0/3175561029 learned_addr learned my addr 172.21.15.77:0/3175561029 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:47:02.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.039+0000 7f02951b9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02900c3b10 0x7f029014d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:02.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.039+0000 7f02959ba700 1 --2- 172.21.15.77:0/3175561029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02900c1170 0x7f02900c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:02.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.039+0000 7f02961bb700 1 -- 172.21.15.77:0/3175561029 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02900c3b10 msgr2=0x7f029014d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:02.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.039+0000 7f02961bb700 1 --2- 172.21.15.77:0/3175561029 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02900c3b10 0x7f029014d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:02.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.039+0000 7f02961bb700 1 -- 172.21.15.77:0/3175561029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02900c1170 msgr2=0x7f02900c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:02.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.039+0000 7f02961bb700 1 --2- 172.21.15.77:0/3175561029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02900c1170 0x7f02900c7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:02.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.039+0000 7f02961bb700 1 -- 172.21.15.77:0/3175561029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0280009370 con 0x7f0290155370 2022-04-03T19:47:02.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.039+0000 7f02959ba700 1 --2- 172.21.15.77:0/3175561029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02900c1170 0x7f02900c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:47:02.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.039+0000 7f02961bb700 1 --2- 172.21.15.77:0/3175561029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0290155370 0x7f029014d980 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f028800c790 tx=0x7f02880015e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:02.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.040+0000 7f027dffb700 1 -- 172.21.15.77:0/3175561029 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0288005c70 con 0x7f0290155370 2022-04-03T19:47:02.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.040+0000 7f027dffb700 1 -- 172.21.15.77:0/3175561029 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0288005dd0 con 0x7f0290155370 2022-04-03T19:47:02.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.040+0000 7f02969bc700 1 -- 172.21.15.77:0/3175561029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f029014c960 con 0x7f0290155370 2022-04-03T19:47:02.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.040+0000 7f027dffb700 1 -- 172.21.15.77:0/3175561029 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0288007300 con 0x7f0290155370 2022-04-03T19:47:02.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.040+0000 7f02969bc700 1 -- 172.21.15.77:0/3175561029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029014cf20 con 0x7f0290155370 2022-04-03T19:47:02.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.041+0000 7f02969bc700 1 -- 172.21.15.77:0/3175561029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02900be3a0 con 0x7f0290155370 2022-04-03T19:47:02.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.042+0000 7f027dffb700 1 -- 172.21.15.77:0/3175561029 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f0288010070 con 0x7f0290155370 2022-04-03T19:47:02.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.042+0000 7f027dffb700 1 --2- 172.21.15.77:0/3175561029 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0270063ce0 0x7f0270066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:02.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.042+0000 7f027dffb700 1 -- 172.21.15.77:0/3175561029 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f0288080370 con 0x7f0290155370 2022-04-03T19:47:02.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.043+0000 7f02959ba700 1 --2- 172.21.15.77:0/3175561029 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0270063ce0 0x7f0270066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:02.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.043+0000 7f02959ba700 1 --2- 172.21.15.77:0/3175561029 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0270063ce0 0x7f0270066190 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f028c009c00 tx=0x7f028c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:02.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.047+0000 7f027dffb700 1 -- 172.21.15.77:0/3175561029 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f028804ef40 con 0x7f0290155370 2022-04-03T19:47:02.278 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.278+0000 7f02969bc700 1 -- 172.21.15.77:0/3175561029 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02900bcb00 con 0x7f0290155370 2022-04-03T19:47:02.279 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.279+0000 7f027dffb700 1 -- 172.21.15.77:0/3175561029 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5361 (secure 0 0 0) 0x7f0288052560 con 0x7f0290155370 2022-04-03T19:47:02.280 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:47:02.280 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:47:02.202801+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:clientreplay","state_seq":44,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:47:02.282 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.281+0000 7f02969bc700 1 -- 172.21.15.77:0/3175561029 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0270063ce0 msgr2=0x7f0270066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:02.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.281+0000 7f02969bc700 1 --2- 172.21.15.77:0/3175561029 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0270063ce0 0x7f0270066190 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f028c009c00 tx=0x7f028c008040 comp rx=0 tx=0).stop 2022-04-03T19:47:02.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.281+0000 7f02969bc700 1 -- 172.21.15.77:0/3175561029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0290155370 msgr2=0x7f029014d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:02.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.281+0000 7f02969bc700 1 --2- 172.21.15.77:0/3175561029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0290155370 0x7f029014d980 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f028800c790 tx=0x7f02880015e0 comp rx=0 tx=0).stop 2022-04-03T19:47:02.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.281+0000 7f02969bc700 1 -- 172.21.15.77:0/3175561029 shutdown_connections 2022-04-03T19:47:02.284 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.281+0000 7f02969bc700 1 --2- 172.21.15.77:0/3175561029 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0290155370 0x7f029014d980 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:02.284 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.281+0000 7f02969bc700 1 --2- 172.21.15.77:0/3175561029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02900c1170 0x7f02900c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:02.284 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.281+0000 7f02969bc700 1 --2- 172.21.15.77:0/3175561029 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0270063ce0 0x7f0270066190 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:02.284 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.281+0000 7f02969bc700 1 --2- 172.21.15.77:0/3175561029 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02900c3b10 0x7f029014d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:02.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.281+0000 7f02969bc700 1 -- 172.21.15.77:0/3175561029 >> 172.21.15.77:0/3175561029 conn(0x7f0290066da0 msgr2=0x7f02900ba570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:47:02.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.281+0000 7f02969bc700 1 -- 172.21.15.77:0/3175561029 shutdown_connections 2022-04-03T19:47:02.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:02.281+0000 7f02969bc700 1 -- 172.21.15.77:0/3175561029 wait complete. 2022-04-03T19:47:02.286 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 12 2022-04-03T19:47:02.307 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:47:04.308 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:47:04.513 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.513+0000 7f7bb9fbb700 1 Processor -- start 2022-04-03T19:47:04.514 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.513+0000 7f7bb9fbb700 1 -- start start 2022-04-03T19:47:04.518 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.514+0000 7f7bb9fbb700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7bb40c1170 0x7f7bb41527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:04.519 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.514+0000 7f7bb9fbb700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bb4155370 0x7f7bb4069110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:04.519 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.514+0000 7f7bb9fbb700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb40c3710 con 0x7f7bb40c3b10 2022-04-03T19:47:04.519 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.514+0000 7f7bb9fbb700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bb4152330 con 0x7f7bb40c1170 2022-04-03T19:47:04.520 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.514+0000 7f7bb9fbb700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb40c1540 con 0x7f7bb4155370 2022-04-03T19:47:04.520 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.514+0000 7f7bb8fb9700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7bb40c1170 0x7f7bb41527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:04.520 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.514+0000 7f7bb3fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bb4155370 0x7f7bb4069110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:04.521 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.514+0000 7f7bb8fb9700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7bb40c1170 0x7f7bb41527c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42790/0 (socket says 172.21.15.77:42790) 2022-04-03T19:47:04.521 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.514+0000 7f7bb8fb9700 1 -- 172.21.15.77:0/1408263571 learned_addr learned my addr 172.21.15.77:0/1408263571 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:47:04.521 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.514+0000 7f7bb97ba700 1 --1- 172.21.15.77:0/1408263571 >> v1:172.21.15.186:6789/0 conn(0x7f7bb40c3b10 0x7f7bb40c71f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44192/0 (socket says 172.21.15.77:44192) 2022-04-03T19:47:04.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.515+0000 7f7bb8fb9700 1 -- 172.21.15.77:0/1408263571 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bb4155370 msgr2=0x7f7bb4069110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:47:04.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.515+0000 7f7bb8fb9700 1 --2- 172.21.15.77:0/1408263571 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bb4155370 0x7f7bb4069110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:04.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.515+0000 7f7bb37fe700 1 -- 172.21.15.77:0/1408263571 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041806669 0 0) 0x7f7bb40c3710 con 0x7f7bb40c3b10 2022-04-03T19:47:04.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.515+0000 7f7bb8fb9700 1 -- 172.21.15.77:0/1408263571 >> v1:172.21.15.186:6789/0 conn(0x7f7bb40c3b10 legacy=0x7f7bb40c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:04.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.515+0000 7f7bb8fb9700 1 -- 172.21.15.77:0/1408263571 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb40c4270 con 0x7f7bb40c1170 2022-04-03T19:47:04.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.515+0000 7f7bb8fb9700 1 --2- 172.21.15.77:0/1408263571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bb40c1170 0x7f7bb41527c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8006fd0 tx=0x7f7ba80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85f9553350ae0eb5 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:04.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.515+0000 7f7bb37fe700 1 -- 172.21.15.77:0/1408263571 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba8004030 con 0x7f7bb40c1170 2022-04-03T19:47:04.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.515+0000 7f7bb37fe700 1 -- 172.21.15.77:0/1408263571 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba80041d0 con 0x7f7bb40c1170 2022-04-03T19:47:04.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.515+0000 7f7bb37fe700 1 -- 172.21.15.77:0/1408263571 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba8011630 con 0x7f7bb40c1170 2022-04-03T19:47:04.525 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.516+0000 7f7bb9fbb700 1 -- 172.21.15.77:0/1408263571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bb40c1170 msgr2=0x7f7bb41527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:04.525 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.516+0000 7f7bb9fbb700 1 --2- 172.21.15.77:0/1408263571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bb40c1170 0x7f7bb41527c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8006fd0 tx=0x7f7ba80090f0 comp rx=0 tx=0).stop 2022-04-03T19:47:04.525 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.516+0000 7f7bb9fbb700 1 -- 172.21.15.77:0/1408263571 shutdown_connections 2022-04-03T19:47:04.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.516+0000 7f7bb9fbb700 1 --2- 172.21.15.77:0/1408263571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bb40c1170 0x7f7bb41527c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:04.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.516+0000 7f7bb9fbb700 1 --2- 172.21.15.77:0/1408263571 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bb4155370 0x7f7bb4069110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:04.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.516+0000 7f7bb9fbb700 1 -- 172.21.15.77:0/1408263571 >> 172.21.15.77:0/1408263571 conn(0x7f7bb4066da0 msgr2=0x7f7bb40671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:47:04.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.516+0000 7f7bb9fbb700 1 -- 172.21.15.77:0/1408263571 shutdown_connections 2022-04-03T19:47:04.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.516+0000 7f7bb9fbb700 1 -- 172.21.15.77:0/1408263571 wait complete. 2022-04-03T19:47:04.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.518+0000 7f7bb9fbb700 1 Processor -- start 2022-04-03T19:47:04.528 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.518+0000 7f7bb9fbb700 1 -- start start 2022-04-03T19:47:04.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.518+0000 7f7bb9fbb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bb40c1170 0x7f7bb41662e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:04.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.518+0000 7f7bb9fbb700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bb40c3b10 0x7f7bb40c7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:04.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.519+0000 7f7bb9fbb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bb4155370 0x7f7bb40bd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:04.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.519+0000 7f7bb9fbb700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb40bc1c0 con 0x7f7bb4155370 2022-04-03T19:47:04.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.519+0000 7f7bb9fbb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb40c7fc0 con 0x7f7bb40c1170 2022-04-03T19:47:04.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.519+0000 7f7bb9fbb700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb40bb310 con 0x7f7bb40c3b10 2022-04-03T19:47:04.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.519+0000 7f7bb97ba700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bb4155370 0x7f7bb40bd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:04.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.519+0000 7f7bb3fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bb40c3b10 0x7f7bb40c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:04.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.519+0000 7f7bb8fb9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bb40c1170 0x7f7bb41662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:04.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.519+0000 7f7bb3fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bb40c3b10 0x7f7bb40c7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60088/0 (socket says 172.21.15.77:60088) 2022-04-03T19:47:04.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.519+0000 7f7bb3fff700 1 -- 172.21.15.77:0/2745252181 learned_addr learned my addr 172.21.15.77:0/2745252181 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:47:04.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.519+0000 7f7bb3fff700 1 -- 172.21.15.77:0/2745252181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bb40c1170 msgr2=0x7f7bb41662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:04.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.519+0000 7f7bb3fff700 1 --2- 172.21.15.77:0/2745252181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bb40c1170 0x7f7bb41662e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:04.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.519+0000 7f7bb3fff700 1 -- 172.21.15.77:0/2745252181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bb4155370 msgr2=0x7f7bb40bd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:04.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.519+0000 7f7bb3fff700 1 --2- 172.21.15.77:0/2745252181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bb4155370 0x7f7bb40bd860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:04.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.519+0000 7f7bb3fff700 1 -- 172.21.15.77:0/2745252181 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba8009370 con 0x7f7bb40c3b10 2022-04-03T19:47:04.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.519+0000 7f7bb8fb9700 1 --2- 172.21.15.77:0/2745252181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bb40c1170 0x7f7bb41662e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:47:04.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.519+0000 7f7bb97ba700 1 --2- 172.21.15.77:0/2745252181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bb4155370 0x7f7bb40bd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:47:04.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.520+0000 7f7bb3fff700 1 --2- 172.21.15.77:0/2745252181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bb40c3b10 0x7f7bb40c7050 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7ba000e900 tx=0x7f7ba0009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:04.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.520+0000 7f7bb0ff9700 1 -- 172.21.15.77:0/2745252181 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0013680 con 0x7f7bb40c3b10 2022-04-03T19:47:04.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.520+0000 7f7bb0ff9700 1 -- 172.21.15.77:0/2745252181 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba00137e0 con 0x7f7bb40c3b10 2022-04-03T19:47:04.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.520+0000 7f7bb9fbb700 1 -- 172.21.15.77:0/2745252181 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb40bb5b0 con 0x7f7bb40c3b10 2022-04-03T19:47:04.540 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.520+0000 7f7bb0ff9700 1 -- 172.21.15.77:0/2745252181 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0013b80 con 0x7f7bb40c3b10 2022-04-03T19:47:04.540 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.520+0000 7f7bb9fbb700 1 -- 172.21.15.77:0/2745252181 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb40bbb70 con 0x7f7bb40c3b10 2022-04-03T19:47:04.540 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.521+0000 7f7bb9fbb700 1 -- 172.21.15.77:0/2745252181 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb40c2130 con 0x7f7bb40c3b10 2022-04-03T19:47:04.541 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.522+0000 7f7bb0ff9700 1 -- 172.21.15.77:0/2745252181 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f7ba0019030 con 0x7f7bb40c3b10 2022-04-03T19:47:04.541 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.523+0000 7f7bb0ff9700 1 --2- 172.21.15.77:0/2745252181 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7b9c063ce0 0x7f7b9c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:04.541 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.523+0000 7f7bb0ff9700 1 -- 172.21.15.77:0/2745252181 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f7ba00806b0 con 0x7f7bb40c3b10 2022-04-03T19:47:04.542 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.523+0000 7f7bb8fb9700 1 --2- 172.21.15.77:0/2745252181 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7b9c063ce0 0x7f7b9c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:04.542 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.524+0000 7f7bb8fb9700 1 --2- 172.21.15.77:0/2745252181 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7b9c063ce0 0x7f7b9c066190 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8009340 tx=0x7f7ba8016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:04.542 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.527+0000 7f7bb0ff9700 1 -- 172.21.15.77:0/2745252181 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f7ba004f200 con 0x7f7bb40c3b10 2022-04-03T19:47:04.760 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.760+0000 7f7bb9fbb700 1 -- 172.21.15.77:0/2745252181 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bb404b050 con 0x7f7bb40c3b10 2022-04-03T19:47:04.761 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.761+0000 7f7bb0ff9700 1 -- 172.21.15.77:0/2745252181 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5361 (secure 0 0 0) 0x7f7ba0052820 con 0x7f7bb40c3b10 2022-04-03T19:47:04.762 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:47:04.762 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:47:02.202801+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:clientreplay","state_seq":44,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:47:04.764 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.763+0000 7f7bb9fbb700 1 -- 172.21.15.77:0/2745252181 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7b9c063ce0 msgr2=0x7f7b9c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:04.765 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.763+0000 7f7bb9fbb700 1 --2- 172.21.15.77:0/2745252181 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7b9c063ce0 0x7f7b9c066190 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8009340 tx=0x7f7ba8016000 comp rx=0 tx=0).stop 2022-04-03T19:47:04.765 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.763+0000 7f7bb9fbb700 1 -- 172.21.15.77:0/2745252181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bb40c3b10 msgr2=0x7f7bb40c7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:04.765 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.763+0000 7f7bb9fbb700 1 --2- 172.21.15.77:0/2745252181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bb40c3b10 0x7f7bb40c7050 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7ba000e900 tx=0x7f7ba0009060 comp rx=0 tx=0).stop 2022-04-03T19:47:04.765 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.763+0000 7f7bb9fbb700 1 -- 172.21.15.77:0/2745252181 shutdown_connections 2022-04-03T19:47:04.766 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.764+0000 7f7bb9fbb700 1 --2- 172.21.15.77:0/2745252181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bb4155370 0x7f7bb40bd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:04.766 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.764+0000 7f7bb9fbb700 1 --2- 172.21.15.77:0/2745252181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bb40c1170 0x7f7bb41662e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:04.766 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.764+0000 7f7bb9fbb700 1 --2- 172.21.15.77:0/2745252181 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7b9c063ce0 0x7f7b9c066190 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:04.766 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.764+0000 7f7bb9fbb700 1 --2- 172.21.15.77:0/2745252181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bb40c3b10 0x7f7bb40c7050 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:04.767 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.764+0000 7f7bb9fbb700 1 -- 172.21.15.77:0/2745252181 >> 172.21.15.77:0/2745252181 conn(0x7f7bb4066da0 msgr2=0x7f7bb406c370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:47:04.767 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.764+0000 7f7bb9fbb700 1 -- 172.21.15.77:0/2745252181 shutdown_connections 2022-04-03T19:47:04.767 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:04.764+0000 7f7bb9fbb700 1 -- 172.21.15.77:0/2745252181 wait complete. 2022-04-03T19:47:04.768 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 12 2022-04-03T19:47:04.788 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:47:06.789 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:47:06.997 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.996+0000 7fed91006700 1 Processor -- start 2022-04-03T19:47:06.997 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.997+0000 7fed91006700 1 -- start start 2022-04-03T19:47:06.999 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.998+0000 7fed91006700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fed8c06bbe0 0x7fed8c1584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:07.000 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.998+0000 7fed91006700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed8c06bfb0 0x7fed8c06d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:07.000 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.998+0000 7fed91006700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed8c06d260 con 0x7fed8c06c500 2022-04-03T19:47:07.000 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.998+0000 7fed91006700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed8c159450 con 0x7fed8c06bbe0 2022-04-03T19:47:07.000 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.998+0000 7fed91006700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c0c6330 con 0x7fed8c06bfb0 2022-04-03T19:47:07.001 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.998+0000 7fed90805700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fed8c06c500 0x7fed8c14eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38414/0 (socket says 172.21.15.77:38414) 2022-04-03T19:47:07.001 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.998+0000 7fed90805700 1 -- 172.21.15.77:0/1376631466 learned_addr learned my addr 172.21.15.77:0/1376631466 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:47:07.001 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.998+0000 7fed83fff700 1 --2- 172.21.15.77:0/1376631466 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed8c06bfb0 0x7fed8c06d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:07.001 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.998+0000 7fed8bfff700 1 --2- 172.21.15.77:0/1376631466 >> v2:172.21.15.186:3300/0 conn(0x7fed8c06bbe0 0x7fed8c1584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:07.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.999+0000 7fed8b7fe700 1 -- 172.21.15.77:0/1376631466 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052591971 0 0) 0x7fed8c06d260 con 0x7fed8c06c500 2022-04-03T19:47:07.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.999+0000 7fed8b7fe700 1 -- 172.21.15.77:0/1376631466 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed6c0033f0 con 0x7fed8c06c500 2022-04-03T19:47:07.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.999+0000 7fed83fff700 1 -- 172.21.15.77:0/1376631466 >> v2:172.21.15.186:3300/0 conn(0x7fed8c06bbe0 msgr2=0x7fed8c1584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:47:07.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.999+0000 7fed83fff700 1 --2- 172.21.15.77:0/1376631466 >> v2:172.21.15.186:3300/0 conn(0x7fed8c06bbe0 0x7fed8c1584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:07.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.999+0000 7fed83fff700 1 -- 172.21.15.77:0/1376631466 >> v1:172.21.15.77:6789/0 conn(0x7fed8c06c500 legacy=0x7fed8c14eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:07.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.999+0000 7fed83fff700 1 -- 172.21.15.77:0/1376631466 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8c158bb0 con 0x7fed8c06bfb0 2022-04-03T19:47:07.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.999+0000 7fed8bfff700 1 --2- 172.21.15.77:0/1376631466 >> v2:172.21.15.186:3300/0 conn(0x7fed8c06bbe0 0x7fed8c1584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:47:07.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.999+0000 7fed83fff700 1 --2- 172.21.15.77:0/1376631466 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed8c06bfb0 0x7fed8c06d6f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fed74009760 tx=0x7fed74009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b0a587c1c46ad79 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:07.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.999+0000 7fed8b7fe700 1 -- 172.21.15.77:0/1376631466 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed740105b0 con 0x7fed8c06bfb0 2022-04-03T19:47:07.004 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.999+0000 7fed8b7fe700 1 -- 172.21.15.77:0/1376631466 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed74010710 con 0x7fed8c06bfb0 2022-04-03T19:47:07.004 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:06.999+0000 7fed8b7fe700 1 -- 172.21.15.77:0/1376631466 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed74010ab0 con 0x7fed8c06bfb0 2022-04-03T19:47:07.008 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.000+0000 7fed91006700 1 -- 172.21.15.77:0/1376631466 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed8c06bfb0 msgr2=0x7fed8c06d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:07.008 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.000+0000 7fed91006700 1 --2- 172.21.15.77:0/1376631466 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed8c06bfb0 0x7fed8c06d6f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fed74009760 tx=0x7fed74009060 comp rx=0 tx=0).stop 2022-04-03T19:47:07.008 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.000+0000 7fed91006700 1 -- 172.21.15.77:0/1376631466 shutdown_connections 2022-04-03T19:47:07.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.000+0000 7fed91006700 1 --2- 172.21.15.77:0/1376631466 >> v2:172.21.15.186:3300/0 conn(0x7fed8c06bbe0 0x7fed8c1584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:07.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.000+0000 7fed91006700 1 --2- 172.21.15.77:0/1376631466 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed8c06bfb0 0x7fed8c06d6f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:07.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.000+0000 7fed91006700 1 -- 172.21.15.77:0/1376631466 >> 172.21.15.77:0/1376631466 conn(0x7fed8c0cb6b0 msgr2=0x7fed8c0cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:47:07.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.000+0000 7fed91006700 1 -- 172.21.15.77:0/1376631466 shutdown_connections 2022-04-03T19:47:07.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.000+0000 7fed91006700 1 -- 172.21.15.77:0/1376631466 wait complete. 2022-04-03T19:47:07.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.002+0000 7fed91006700 1 Processor -- start 2022-04-03T19:47:07.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.002+0000 7fed91006700 1 -- start start 2022-04-03T19:47:07.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.003+0000 7fed91006700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed8c06bbe0 0x7fed8c147d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:07.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.003+0000 7fed91006700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed8c06bfb0 0x7fed8c157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:07.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.003+0000 7fed91006700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed8c06c500 0x7fed8c1570f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:07.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.003+0000 7fed91006700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c0c6b70 con 0x7fed8c06bbe0 2022-04-03T19:47:07.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.003+0000 7fed91006700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c165e60 con 0x7fed8c06c500 2022-04-03T19:47:07.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.003+0000 7fed91006700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c1482a0 con 0x7fed8c06bfb0 2022-04-03T19:47:07.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.003+0000 7fed8bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed8c06bbe0 0x7fed8c147d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:07.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.003+0000 7fed90805700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed8c06c500 0x7fed8c1570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:07.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.003+0000 7fed8bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed8c06bbe0 0x7fed8c147d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42796/0 (socket says 172.21.15.77:42796) 2022-04-03T19:47:07.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.003+0000 7fed8bfff700 1 -- 172.21.15.77:0/3982933001 learned_addr learned my addr 172.21.15.77:0/3982933001 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:47:07.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.003+0000 7fed83fff700 1 --2- 172.21.15.77:0/3982933001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed8c06bfb0 0x7fed8c157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:07.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.004+0000 7fed8bfff700 1 -- 172.21.15.77:0/3982933001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed8c06bfb0 msgr2=0x7fed8c157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:07.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.004+0000 7fed8bfff700 1 --2- 172.21.15.77:0/3982933001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed8c06bfb0 0x7fed8c157f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:07.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.004+0000 7fed8bfff700 1 -- 172.21.15.77:0/3982933001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed8c06c500 msgr2=0x7fed8c1570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:07.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.004+0000 7fed8bfff700 1 --2- 172.21.15.77:0/3982933001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed8c06c500 0x7fed8c1570f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:07.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.004+0000 7fed8bfff700 1 -- 172.21.15.77:0/3982933001 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7c0092a0 con 0x7fed8c06bbe0 2022-04-03T19:47:07.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.004+0000 7fed83fff700 1 --2- 172.21.15.77:0/3982933001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed8c06bfb0 0x7fed8c157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:47:07.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.004+0000 7fed8bfff700 1 --2- 172.21.15.77:0/3982933001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed8c06bbe0 0x7fed8c147d60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00ea50 tx=0x7fed7c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:07.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.004+0000 7fed88ff9700 1 -- 172.21.15.77:0/3982933001 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c009600 con 0x7fed8c06bbe0 2022-04-03T19:47:07.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.004+0000 7fed88ff9700 1 -- 172.21.15.77:0/3982933001 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7c00c850 con 0x7fed8c06bbe0 2022-04-03T19:47:07.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.004+0000 7fed91006700 1 -- 172.21.15.77:0/3982933001 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed740092a0 con 0x7fed8c06bbe0 2022-04-03T19:47:07.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.004+0000 7fed88ff9700 1 -- 172.21.15.77:0/3982933001 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c00cbf0 con 0x7fed8c06bbe0 2022-04-03T19:47:07.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.004+0000 7fed91006700 1 -- 172.21.15.77:0/3982933001 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed8c1655c0 con 0x7fed8c06bbe0 2022-04-03T19:47:07.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.005+0000 7fed91006700 1 -- 172.21.15.77:0/3982933001 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed8c0b1720 con 0x7fed8c06bbe0 2022-04-03T19:47:07.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.006+0000 7fed88ff9700 1 -- 172.21.15.77:0/3982933001 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fed7c019030 con 0x7fed8c06bbe0 2022-04-03T19:47:07.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.007+0000 7fed88ff9700 1 --2- 172.21.15.77:0/3982933001 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fed70063ce0 0x7fed70066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:07.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.007+0000 7fed88ff9700 1 -- 172.21.15.77:0/3982933001 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6202+0+0 (secure 0 0 0) 0x7fed7c0809b0 con 0x7fed8c06bbe0 2022-04-03T19:47:07.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.007+0000 7fed83fff700 1 --2- 172.21.15.77:0/3982933001 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fed70063ce0 0x7fed70066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:07.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.008+0000 7fed83fff700 1 --2- 172.21.15.77:0/3982933001 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fed70063ce0 0x7fed70066190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fed8c0c6fe0 tx=0x7fed74016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:07.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.012+0000 7fed88ff9700 1 -- 172.21.15.77:0/3982933001 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fed7c04f440 con 0x7fed8c06bbe0 2022-04-03T19:47:07.242 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.241+0000 7fed91006700 1 -- 172.21.15.77:0/3982933001 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed8c04b050 con 0x7fed8c06bbe0 2022-04-03T19:47:07.243 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.242+0000 7fed88ff9700 1 -- 172.21.15.77:0/3982933001 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5361 (secure 0 0 0) 0x7fed7c017090 con 0x7fed8c06bbe0 2022-04-03T19:47:07.243 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:47:07.243 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:47:02.202801+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:clientreplay","state_seq":44,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:47:07.245 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.244+0000 7fed91006700 1 -- 172.21.15.77:0/3982933001 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fed70063ce0 msgr2=0x7fed70066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:07.246 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.244+0000 7fed91006700 1 --2- 172.21.15.77:0/3982933001 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fed70063ce0 0x7fed70066190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fed8c0c6fe0 tx=0x7fed74016000 comp rx=0 tx=0).stop 2022-04-03T19:47:07.246 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.245+0000 7fed91006700 1 -- 172.21.15.77:0/3982933001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed8c06bbe0 msgr2=0x7fed8c147d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:07.246 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.245+0000 7fed91006700 1 --2- 172.21.15.77:0/3982933001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed8c06bbe0 0x7fed8c147d60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00ea50 tx=0x7fed7c009060 comp rx=0 tx=0).stop 2022-04-03T19:47:07.246 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.245+0000 7fed91006700 1 -- 172.21.15.77:0/3982933001 shutdown_connections 2022-04-03T19:47:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.245+0000 7fed91006700 1 --2- 172.21.15.77:0/3982933001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed8c06c500 0x7fed8c1570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.245+0000 7fed91006700 1 --2- 172.21.15.77:0/3982933001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed8c06bbe0 0x7fed8c147d60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.245+0000 7fed91006700 1 --2- 172.21.15.77:0/3982933001 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fed70063ce0 0x7fed70066190 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.245+0000 7fed91006700 1 --2- 172.21.15.77:0/3982933001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed8c06bfb0 0x7fed8c157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:07.247 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.245+0000 7fed91006700 1 -- 172.21.15.77:0/3982933001 >> 172.21.15.77:0/3982933001 conn(0x7fed8c0cb6b0 msgr2=0x7fed8c148dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:47:07.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.245+0000 7fed91006700 1 -- 172.21.15.77:0/3982933001 shutdown_connections 2022-04-03T19:47:07.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:07.245+0000 7fed91006700 1 -- 172.21.15.77:0/3982933001 wait complete. 2022-04-03T19:47:07.249 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 12 2022-04-03T19:47:07.269 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:47:09.270 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:47:09.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.469+0000 7efddffff700 1 Processor -- start 2022-04-03T19:47:09.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.470+0000 7efddffff700 1 -- start start 2022-04-03T19:47:09.473 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.471+0000 7efddffff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7efde006d3f0 0x7efde006d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:09.474 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.471+0000 7efddffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efde00c2a90 0x7efde006c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:09.474 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.471+0000 7efddffff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efde01562f0 con 0x7efde00c5930 2022-04-03T19:47:09.474 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.471+0000 7efddffff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde00c2f80 con 0x7efde00c2a90 2022-04-03T19:47:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.471+0000 7efddffff700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde00c3260 con 0x7efde006d3f0 2022-04-03T19:47:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.471+0000 7efddeffd700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7efde006d3f0 0x7efde006d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.471+0000 7efddf7fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7efde00c2a90 0x7efde006c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.471+0000 7efdde7fc700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7efde00c5930 0x7efde0155be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38416/0 (socket says 172.21.15.77:38416) 2022-04-03T19:47:09.475 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.471+0000 7efdde7fc700 1 -- 172.21.15.77:0/2317308787 learned_addr learned my addr 172.21.15.77:0/2317308787 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:47:09.476 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.472+0000 7efddeffd700 1 -- 172.21.15.77:0/2317308787 >> v2:172.21.15.77:3300/0 conn(0x7efde00c2a90 msgr2=0x7efde006c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:47:09.476 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.472+0000 7efddeffd700 1 --2- 172.21.15.77:0/2317308787 >> v2:172.21.15.77:3300/0 conn(0x7efde00c2a90 0x7efde006c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:09.476 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.472+0000 7efddeffd700 1 -- 172.21.15.77:0/2317308787 >> v1:172.21.15.77:6789/0 conn(0x7efde00c5930 legacy=0x7efde0155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:09.476 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.472+0000 7efddeffd700 1 -- 172.21.15.77:0/2317308787 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde00c3540 con 0x7efde006d3f0 2022-04-03T19:47:09.477 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.472+0000 7efddf7fe700 1 --2- 172.21.15.77:0/2317308787 >> v2:172.21.15.77:3300/0 conn(0x7efde00c2a90 0x7efde006c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:47:09.477 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.472+0000 7efddeffd700 1 --2- 172.21.15.77:0/2317308787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde006d3f0 0x7efde006d7c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009760 tx=0x7efdd0009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e550a3428cfeca server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:09.477 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.472+0000 7efdddffb700 1 -- 172.21.15.77:0/2317308787 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd0011630 con 0x7efde006d3f0 2022-04-03T19:47:09.477 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.472+0000 7efdddffb700 1 -- 172.21.15.77:0/2317308787 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd0011790 con 0x7efde006d3f0 2022-04-03T19:47:09.477 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.472+0000 7efdddffb700 1 -- 172.21.15.77:0/2317308787 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd0011b30 con 0x7efde006d3f0 2022-04-03T19:47:09.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.473+0000 7efddffff700 1 -- 172.21.15.77:0/2317308787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde006d3f0 msgr2=0x7efde006d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:09.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.473+0000 7efddffff700 1 --2- 172.21.15.77:0/2317308787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde006d3f0 0x7efde006d7c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009760 tx=0x7efdd0009060 comp rx=0 tx=0).stop 2022-04-03T19:47:09.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.473+0000 7efddffff700 1 -- 172.21.15.77:0/2317308787 shutdown_connections 2022-04-03T19:47:09.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.473+0000 7efddffff700 1 --2- 172.21.15.77:0/2317308787 >> v2:172.21.15.77:3300/0 conn(0x7efde00c2a90 0x7efde006c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:09.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.473+0000 7efddffff700 1 --2- 172.21.15.77:0/2317308787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde006d3f0 0x7efde006d7c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:09.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.473+0000 7efddffff700 1 -- 172.21.15.77:0/2317308787 >> 172.21.15.77:0/2317308787 conn(0x7efde00cb720 msgr2=0x7efde00cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:47:09.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.473+0000 7efddffff700 1 -- 172.21.15.77:0/2317308787 shutdown_connections 2022-04-03T19:47:09.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.473+0000 7efddffff700 1 -- 172.21.15.77:0/2317308787 wait complete. 2022-04-03T19:47:09.482 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.475+0000 7efddffff700 1 Processor -- start 2022-04-03T19:47:09.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.475+0000 7efddffff700 1 -- start start 2022-04-03T19:47:09.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.476+0000 7efddffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde006d3f0 0x7efde014ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:09.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.476+0000 7efddffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde00c2a90 0x7efde01578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:09.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.476+0000 7efddffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde00c5930 0x7efde0157e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:09.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.476+0000 7efddffff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde01695e0 con 0x7efde00c2a90 2022-04-03T19:47:09.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.476+0000 7efddffff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde0168960 con 0x7efde00c5930 2022-04-03T19:47:09.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.476+0000 7efddffff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde0168c40 con 0x7efde006d3f0 2022-04-03T19:47:09.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.476+0000 7efddeffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde006d3f0 0x7efde014ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:09.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.476+0000 7efdde7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde00c2a90 0x7efde01578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:09.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.476+0000 7efddeffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde006d3f0 0x7efde014ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60094/0 (socket says 172.21.15.77:60094) 2022-04-03T19:47:09.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.476+0000 7efddf7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde00c5930 0x7efde0157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:09.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.476+0000 7efddeffd700 1 -- 172.21.15.77:0/199468362 learned_addr learned my addr 172.21.15.77:0/199468362 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:47:09.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.476+0000 7efddeffd700 1 -- 172.21.15.77:0/199468362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde00c5930 msgr2=0x7efde0157e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:09.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.476+0000 7efddeffd700 1 --2- 172.21.15.77:0/199468362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde00c5930 0x7efde0157e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:09.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.476+0000 7efddeffd700 1 -- 172.21.15.77:0/199468362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde00c2a90 msgr2=0x7efde01578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:09.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.476+0000 7efddeffd700 1 --2- 172.21.15.77:0/199468362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde00c2a90 0x7efde01578c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:09.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.477+0000 7efddeffd700 1 -- 172.21.15.77:0/199468362 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd8009a20 con 0x7efde006d3f0 2022-04-03T19:47:09.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.477+0000 7efddf7fe700 1 --2- 172.21.15.77:0/199468362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde00c5930 0x7efde0157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:47:09.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.477+0000 7efddeffd700 1 --2- 172.21.15.77:0/199468362 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde006d3f0 0x7efde014ddb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efde00c33a0 tx=0x7efdd0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:09.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.477+0000 7efdc6ffd700 1 -- 172.21.15.77:0/199468362 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd0003770 con 0x7efde006d3f0 2022-04-03T19:47:09.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.477+0000 7efdc6ffd700 1 -- 172.21.15.77:0/199468362 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd00038d0 con 0x7efde006d3f0 2022-04-03T19:47:09.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.477+0000 7efdc6ffd700 1 -- 172.21.15.77:0/199468362 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd0003c90 con 0x7efde006d3f0 2022-04-03T19:47:09.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.477+0000 7efddffff700 1 -- 172.21.15.77:0/199468362 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd00092a0 con 0x7efde006d3f0 2022-04-03T19:47:09.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.477+0000 7efddffff700 1 -- 172.21.15.77:0/199468362 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde0169200 con 0x7efde006d3f0 2022-04-03T19:47:09.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.478+0000 7efddffff700 1 -- 172.21.15.77:0/199468362 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde004b050 con 0x7efde006d3f0 2022-04-03T19:47:09.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.479+0000 7efdc6ffd700 1 -- 172.21.15.77:0/199468362 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7efdd000f050 con 0x7efde006d3f0 2022-04-03T19:47:09.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.480+0000 7efdc6ffd700 1 --2- 172.21.15.77:0/199468362 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7efdc0063ce0 0x7efdc0066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:09.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.480+0000 7efdc6ffd700 1 -- 172.21.15.77:0/199468362 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6202+0+0 (secure 0 0 0) 0x7efdd0081de0 con 0x7efde006d3f0 2022-04-03T19:47:09.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.480+0000 7efdde7fc700 1 --2- 172.21.15.77:0/199468362 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7efdc0063ce0 0x7efdc0066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:09.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.481+0000 7efdde7fc700 1 --2- 172.21.15.77:0/199468362 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7efdc0063ce0 0x7efdc0066190 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7efde00cb5c0 tx=0x7efdd4007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:09.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.484+0000 7efdc6ffd700 1 -- 172.21.15.77:0/199468362 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7efdd0050990 con 0x7efde006d3f0 2022-04-03T19:47:09.721 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.721+0000 7efddffff700 1 -- 172.21.15.77:0/199468362 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efde006d010 con 0x7efde006d3f0 2022-04-03T19:47:09.722 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.722+0000 7efdc6ffd700 1 -- 172.21.15.77:0/199468362 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5361 (secure 0 0 0) 0x7efdd0053fb0 con 0x7efde006d3f0 2022-04-03T19:47:09.723 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:47:09.723 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:47:02.202801+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:clientreplay","state_seq":44,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:47:09.725 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.724+0000 7efddffff700 1 -- 172.21.15.77:0/199468362 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7efdc0063ce0 msgr2=0x7efdc0066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:09.725 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.724+0000 7efddffff700 1 --2- 172.21.15.77:0/199468362 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7efdc0063ce0 0x7efdc0066190 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7efde00cb5c0 tx=0x7efdd4007d30 comp rx=0 tx=0).stop 2022-04-03T19:47:09.726 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.724+0000 7efddffff700 1 -- 172.21.15.77:0/199468362 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde006d3f0 msgr2=0x7efde014ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:09.726 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.724+0000 7efddffff700 1 --2- 172.21.15.77:0/199468362 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde006d3f0 0x7efde014ddb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efde00c33a0 tx=0x7efdd0017000 comp rx=0 tx=0).stop 2022-04-03T19:47:09.726 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.724+0000 7efddffff700 1 -- 172.21.15.77:0/199468362 shutdown_connections 2022-04-03T19:47:09.726 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.724+0000 7efddffff700 1 --2- 172.21.15.77:0/199468362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde00c5930 0x7efde0157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.724+0000 7efddffff700 1 --2- 172.21.15.77:0/199468362 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7efdc0063ce0 0x7efdc0066190 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.724+0000 7efddffff700 1 --2- 172.21.15.77:0/199468362 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde006d3f0 0x7efde014ddb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.724+0000 7efddffff700 1 --2- 172.21.15.77:0/199468362 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efde00c2a90 0x7efde01578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:09.727 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.724+0000 7efddffff700 1 -- 172.21.15.77:0/199468362 >> 172.21.15.77:0/199468362 conn(0x7efde00cb720 msgr2=0x7efde0154b60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:47:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.725+0000 7efddffff700 1 -- 172.21.15.77:0/199468362 shutdown_connections 2022-04-03T19:47:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:09.725+0000 7efddffff700 1 -- 172.21.15.77:0/199468362 wait complete. 2022-04-03T19:47:09.728 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 12 2022-04-03T19:47:09.750 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 12, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-04-03T19:44:21.275716+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 12, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4303": { "addr": "172.21.15.186:6837/1407029827", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 1407029827, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 1407029827, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4303, "incarnation": 9, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:clientreplay", "state_seq": 44 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-04-03T19:47:02.202801+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4303 } } } ], "standbys": [ { "addr": "172.21.15.77:6834/2116223509", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6832", "nonce": 2116223509, "type": "v2" }, { "addr": "172.21.15.77:6834", "nonce": 2116223509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4245, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6836/2374276747", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2374276747, "type": "v2" }, { "addr": "172.21.15.186:6836", "nonce": 2374276747, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4259, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/610990853", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6838", "nonce": 610990853, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 610990853, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4265, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.77:6835/170053412", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6833", "nonce": 170053412, "type": "v2" }, { "addr": "172.21.15.77:6835", "nonce": 170053412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4271, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-04-03T19:47:11.751 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:47:11.958 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.957+0000 7f5f47fff700 1 Processor -- start 2022-04-03T19:47:11.958 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.957+0000 7f5f47fff700 1 -- start start 2022-04-03T19:47:11.961 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.958+0000 7f5f47fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f4806bbe0 0x7f5f481584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:11.961 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.959+0000 7f5f47fff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5f4806bfb0 0x7f5f4806d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.959+0000 7f5f47fff700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4806d260 con 0x7f5f4806c500 2022-04-03T19:47:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.959+0000 7f5f47fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f48159450 con 0x7f5f4806bbe0 2022-04-03T19:47:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.959+0000 7f5f47fff700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f480c6330 con 0x7f5f4806bfb0 2022-04-03T19:47:11.962 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.959+0000 7f5f46ffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f4806bbe0 0x7f5f481584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:11.963 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.959+0000 7f5f46ffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f4806bbe0 0x7f5f481584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42802/0 (socket says 172.21.15.77:42802) 2022-04-03T19:47:11.963 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.959+0000 7f5f46ffd700 1 -- 172.21.15.77:0/4153796657 learned_addr learned my addr 172.21.15.77:0/4153796657 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:47:11.963 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.959+0000 7f5f3ffff700 1 --2- 172.21.15.77:0/4153796657 >> v2:172.21.15.186:3300/0 conn(0x7f5f4806bfb0 0x7f5f4806d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:11.963 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.959+0000 7f5f46ffd700 1 -- 172.21.15.77:0/4153796657 >> v2:172.21.15.186:3300/0 conn(0x7f5f4806bfb0 msgr2=0x7f5f4806d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:47:11.964 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.959+0000 7f5f46ffd700 1 --2- 172.21.15.77:0/4153796657 >> v2:172.21.15.186:3300/0 conn(0x7f5f4806bfb0 0x7f5f4806d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:11.964 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.959+0000 7f5f46ffd700 1 -- 172.21.15.77:0/4153796657 >> v1:172.21.15.186:6789/0 conn(0x7f5f4806c500 legacy=0x7f5f4814eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:11.964 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.960+0000 7f5f46ffd700 1 -- 172.21.15.77:0/4153796657 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f48158bb0 con 0x7f5f4806bbe0 2022-04-03T19:47:11.964 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.960+0000 7f5f46ffd700 1 --2- 172.21.15.77:0/4153796657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f4806bbe0 0x7f5f481584b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5f480898c0 tx=0x7f5f380090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd4a3e77b570460b server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:11.964 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.960+0000 7f5f467fc700 1 -- 172.21.15.77:0/4153796657 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f38004030 con 0x7f5f4806bbe0 2022-04-03T19:47:11.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.960+0000 7f5f467fc700 1 -- 172.21.15.77:0/4153796657 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f38004190 con 0x7f5f4806bbe0 2022-04-03T19:47:11.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.960+0000 7f5f467fc700 1 -- 172.21.15.77:0/4153796657 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f38011630 con 0x7f5f4806bbe0 2022-04-03T19:47:11.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.961+0000 7f5f47fff700 1 -- 172.21.15.77:0/4153796657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f4806bbe0 msgr2=0x7f5f481584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:11.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.961+0000 7f5f47fff700 1 --2- 172.21.15.77:0/4153796657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f4806bbe0 0x7f5f481584b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5f480898c0 tx=0x7f5f380090f0 comp rx=0 tx=0).stop 2022-04-03T19:47:11.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.961+0000 7f5f47fff700 1 -- 172.21.15.77:0/4153796657 shutdown_connections 2022-04-03T19:47:11.966 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.961+0000 7f5f47fff700 1 --2- 172.21.15.77:0/4153796657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f4806bbe0 0x7f5f481584b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:11.966 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.961+0000 7f5f47fff700 1 --2- 172.21.15.77:0/4153796657 >> v2:172.21.15.186:3300/0 conn(0x7f5f4806bfb0 0x7f5f4806d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:11.966 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.961+0000 7f5f47fff700 1 -- 172.21.15.77:0/4153796657 >> 172.21.15.77:0/4153796657 conn(0x7f5f480cb6b0 msgr2=0x7f5f480cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:47:11.966 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.961+0000 7f5f47fff700 1 -- 172.21.15.77:0/4153796657 shutdown_connections 2022-04-03T19:47:11.967 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.961+0000 7f5f47fff700 1 -- 172.21.15.77:0/4153796657 wait complete. 2022-04-03T19:47:11.970 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.962+0000 7f5f47fff700 1 Processor -- start 2022-04-03T19:47:11.970 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.963+0000 7f5f47fff700 1 -- start start 2022-04-03T19:47:11.970 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.963+0000 7f5f47fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f4806bbe0 0x7f5f48155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:11.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.963+0000 7f5f47fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f4806bfb0 0x7f5f4814a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:11.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.964+0000 7f5f47fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f4806c500 0x7f5f48168e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:11.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.964+0000 7f5f47fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f48168a20 con 0x7f5f4806c500 2022-04-03T19:47:11.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.964+0000 7f5f47fff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f48166760 con 0x7f5f4806bfb0 2022-04-03T19:47:11.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.964+0000 7f5f47fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4814a9b0 con 0x7f5f4806bbe0 2022-04-03T19:47:11.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.964+0000 7f5f477fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f4806c500 0x7f5f48168e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:11.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.964+0000 7f5f3ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f4806bfb0 0x7f5f4814a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:11.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.964+0000 7f5f477fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f4806c500 0x7f5f48168e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42804/0 (socket says 172.21.15.77:42804) 2022-04-03T19:47:11.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.964+0000 7f5f477fe700 1 -- 172.21.15.77:0/168192971 learned_addr learned my addr 172.21.15.77:0/168192971 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:47:11.973 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.964+0000 7f5f46ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f4806bbe0 0x7f5f48155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:11.973 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.964+0000 7f5f477fe700 1 -- 172.21.15.77:0/168192971 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f4806bbe0 msgr2=0x7f5f48155200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:11.973 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.964+0000 7f5f477fe700 1 --2- 172.21.15.77:0/168192971 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f4806bbe0 0x7f5f48155200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:11.973 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.964+0000 7f5f477fe700 1 -- 172.21.15.77:0/168192971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f4806bfb0 msgr2=0x7f5f4814a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:11.973 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.964+0000 7f5f477fe700 1 --2- 172.21.15.77:0/168192971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f4806bfb0 0x7f5f4814a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:11.974 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.964+0000 7f5f477fe700 1 -- 172.21.15.77:0/168192971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f38009370 con 0x7f5f4806c500 2022-04-03T19:47:11.974 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.964+0000 7f5f3ffff700 1 --2- 172.21.15.77:0/168192971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f4806bfb0 0x7f5f4814a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:47:11.974 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.965+0000 7f5f477fe700 1 --2- 172.21.15.77:0/168192971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f4806c500 0x7f5f48168e70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5f4000c790 tx=0x7f5f400015e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:11.974 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.965+0000 7f5f3effd700 1 -- 172.21.15.77:0/168192971 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f40003840 con 0x7f5f4806c500 2022-04-03T19:47:11.974 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.965+0000 7f5f3effd700 1 -- 172.21.15.77:0/168192971 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f400039a0 con 0x7f5f4806c500 2022-04-03T19:47:11.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.965+0000 7f5f3effd700 1 -- 172.21.15.77:0/168192971 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f40003d20 con 0x7f5f4806c500 2022-04-03T19:47:11.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.965+0000 7f5f47fff700 1 -- 172.21.15.77:0/168192971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f48168750 con 0x7f5f4806c500 2022-04-03T19:47:11.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.965+0000 7f5f47fff700 1 -- 172.21.15.77:0/168192971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f48151240 con 0x7f5f4806c500 2022-04-03T19:47:11.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.966+0000 7f5f47fff700 1 -- 172.21.15.77:0/168192971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f48168b90 con 0x7f5f4806c500 2022-04-03T19:47:11.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.967+0000 7f5f3effd700 1 -- 172.21.15.77:0/168192971 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f5f40019030 con 0x7f5f4806c500 2022-04-03T19:47:11.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.968+0000 7f5f3effd700 1 --2- 172.21.15.77:0/168192971 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5f2c063ce0 0x7f5f2c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:11.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.968+0000 7f5f3effd700 1 -- 172.21.15.77:0/168192971 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f5f4007f950 con 0x7f5f4806c500 2022-04-03T19:47:11.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.968+0000 7f5f46ffd700 1 --2- 172.21.15.77:0/168192971 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5f2c063ce0 0x7f5f2c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:11.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.969+0000 7f5f46ffd700 1 --2- 172.21.15.77:0/168192971 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5f2c063ce0 0x7f5f2c066190 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5f480898c0 tx=0x7f5f38016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:11.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:11.972+0000 7f5f3effd700 1 -- 172.21.15.77:0/168192971 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f5f40048ae0 con 0x7f5f4806c500 2022-04-03T19:47:12.207 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:12.207+0000 7f5f47fff700 1 -- 172.21.15.77:0/168192971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f480c7ec0 con 0x7f5f4806c500 2022-04-03T19:47:12.208 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:12.208+0000 7f5f3effd700 1 -- 172.21.15.77:0/168192971 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+5355 (secure 0 0 0) 0x7f5f40017020 con 0x7f5f4806c500 2022-04-03T19:47:12.209 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:47:12.209 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:47:10.107928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:active","state_seq":47,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:47:12.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:12.210+0000 7f5f47fff700 1 -- 172.21.15.77:0/168192971 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5f2c063ce0 msgr2=0x7f5f2c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:12.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:12.210+0000 7f5f47fff700 1 --2- 172.21.15.77:0/168192971 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5f2c063ce0 0x7f5f2c066190 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5f480898c0 tx=0x7f5f38016000 comp rx=0 tx=0).stop 2022-04-03T19:47:12.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:12.210+0000 7f5f47fff700 1 -- 172.21.15.77:0/168192971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f4806c500 msgr2=0x7f5f48168e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:12.212 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:12.210+0000 7f5f47fff700 1 --2- 172.21.15.77:0/168192971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f4806c500 0x7f5f48168e70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5f4000c790 tx=0x7f5f400015e0 comp rx=0 tx=0).stop 2022-04-03T19:47:12.212 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:12.210+0000 7f5f47fff700 1 -- 172.21.15.77:0/168192971 shutdown_connections 2022-04-03T19:47:12.212 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:12.210+0000 7f5f47fff700 1 --2- 172.21.15.77:0/168192971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f4806c500 0x7f5f48168e70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:12.212 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:12.210+0000 7f5f47fff700 1 --2- 172.21.15.77:0/168192971 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5f2c063ce0 0x7f5f2c066190 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:12.213 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:12.210+0000 7f5f47fff700 1 --2- 172.21.15.77:0/168192971 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f4806bbe0 0x7f5f48155200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:12.213 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:12.210+0000 7f5f47fff700 1 --2- 172.21.15.77:0/168192971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f4806bfb0 0x7f5f4814a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:12.213 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:12.210+0000 7f5f47fff700 1 -- 172.21.15.77:0/168192971 >> 172.21.15.77:0/168192971 conn(0x7f5f480cb6b0 msgr2=0x7f5f4814d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:47:12.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:12.210+0000 7f5f47fff700 1 -- 172.21.15.77:0/168192971 shutdown_connections 2022-04-03T19:47:12.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:12.211+0000 7f5f47fff700 1 -- 172.21.15.77:0/168192971 wait complete. 2022-04-03T19:47:12.214 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 13 2022-04-03T19:47:12.235 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=4327 2022-04-03T19:47:12.236 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.e 2022-04-03T19:47:15.255 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:47:15.261 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:47:16.471 INFO:tasks.workunit.client.0.smithi077.stdout: create 253/30 file after the snapshot 2022-04-03T19:47:16.475 INFO:tasks.workunit.client.0.smithi077.stdout: create 253/31 file after the snapshot 2022-04-03T19:47:16.476 INFO:tasks.workunit.client.0.smithi077.stdout: create 253/32 file after the snapshot 2022-04-03T19:47:16.477 INFO:tasks.workunit.client.0.smithi077.stdout: create 253/33 file after the snapshot 2022-04-03T19:47:16.478 INFO:tasks.workunit.client.0.smithi077.stdout: create 253/34 file after the snapshot 2022-04-03T19:47:16.480 INFO:tasks.workunit.client.0.smithi077.stdout: create 253/35 file after the snapshot 2022-04-03T19:47:16.481 INFO:tasks.workunit.client.0.smithi077.stdout: create 253/36 file after the snapshot 2022-04-03T19:47:16.482 INFO:tasks.workunit.client.0.smithi077.stdout: create 253/37 file after the snapshot 2022-04-03T19:47:16.483 INFO:tasks.workunit.client.0.smithi077.stdout: create 253/38 file after the snapshot 2022-04-03T19:47:16.484 INFO:tasks.workunit.client.0.smithi077.stdout: create 253/39 file after the snapshot 2022-04-03T19:47:16.486 INFO:tasks.workunit.client.0.smithi077.stdout: create 254/.snap/snap-subdir-test 2022-04-03T19:47:16.509 INFO:tasks.workunit.client.0.smithi077.stdout: create 254/30 file after the snapshot 2022-04-03T19:47:16.511 INFO:tasks.workunit.client.0.smithi077.stdout: create 254/31 file after the snapshot 2022-04-03T19:47:16.512 INFO:tasks.workunit.client.0.smithi077.stdout: create 254/32 file after the snapshot 2022-04-03T19:47:16.513 INFO:tasks.workunit.client.0.smithi077.stdout: create 254/33 file after the snapshot 2022-04-03T19:47:16.514 INFO:tasks.workunit.client.0.smithi077.stdout: create 254/34 file after the snapshot 2022-04-03T19:47:16.515 INFO:tasks.workunit.client.0.smithi077.stdout: create 254/35 file after the snapshot 2022-04-03T19:47:16.516 INFO:tasks.workunit.client.0.smithi077.stdout: create 254/36 file after the snapshot 2022-04-03T19:47:16.518 INFO:tasks.workunit.client.0.smithi077.stdout: create 254/37 file after the snapshot 2022-04-03T19:47:16.519 INFO:tasks.workunit.client.0.smithi077.stdout: create 254/38 file after the snapshot 2022-04-03T19:47:16.520 INFO:tasks.workunit.client.0.smithi077.stdout: create 254/39 file after the snapshot 2022-04-03T19:47:16.521 INFO:tasks.workunit.client.0.smithi077.stdout: create 255/.snap/snap-subdir-test 2022-04-03T19:47:16.807 INFO:tasks.workunit.client.0.smithi077.stdout: create 255/30 file after the snapshot 2022-04-03T19:47:16.809 INFO:tasks.workunit.client.0.smithi077.stdout: create 255/31 file after the snapshot 2022-04-03T19:47:16.810 INFO:tasks.workunit.client.0.smithi077.stdout: create 255/32 file after the snapshot 2022-04-03T19:47:16.811 INFO:tasks.workunit.client.0.smithi077.stdout: create 255/33 file after the snapshot 2022-04-03T19:47:16.812 INFO:tasks.workunit.client.0.smithi077.stdout: create 255/34 file after the snapshot 2022-04-03T19:47:16.814 INFO:tasks.workunit.client.0.smithi077.stdout: create 255/35 file after the snapshot 2022-04-03T19:47:16.815 INFO:tasks.workunit.client.0.smithi077.stdout: create 255/36 file after the snapshot 2022-04-03T19:47:16.816 INFO:tasks.workunit.client.0.smithi077.stdout: create 255/37 file after the snapshot 2022-04-03T19:47:16.817 INFO:tasks.workunit.client.0.smithi077.stdout: create 255/38 file after the snapshot 2022-04-03T19:47:16.819 INFO:tasks.workunit.client.0.smithi077.stdout: create 255/39 file after the snapshot 2022-04-03T19:47:16.820 INFO:tasks.workunit.client.0.smithi077.stdout: create 256/.snap/snap-subdir-test 2022-04-03T19:47:16.841 INFO:tasks.workunit.client.0.smithi077.stdout: create 256/30 file after the snapshot 2022-04-03T19:47:16.843 INFO:tasks.workunit.client.0.smithi077.stdout: create 256/31 file after the snapshot 2022-04-03T19:47:16.844 INFO:tasks.workunit.client.0.smithi077.stdout: create 256/32 file after the snapshot 2022-04-03T19:47:16.845 INFO:tasks.workunit.client.0.smithi077.stdout: create 256/33 file after the snapshot 2022-04-03T19:47:16.846 INFO:tasks.workunit.client.0.smithi077.stdout: create 256/34 file after the snapshot 2022-04-03T19:47:16.847 INFO:tasks.workunit.client.0.smithi077.stdout: create 256/35 file after the snapshot 2022-04-03T19:47:16.848 INFO:tasks.workunit.client.0.smithi077.stdout: create 256/36 file after the snapshot 2022-04-03T19:47:16.850 INFO:tasks.workunit.client.0.smithi077.stdout: create 256/37 file after the snapshot 2022-04-03T19:47:16.851 INFO:tasks.workunit.client.0.smithi077.stdout: create 256/38 file after the snapshot 2022-04-03T19:47:16.852 INFO:tasks.workunit.client.0.smithi077.stdout: create 256/39 file after the snapshot 2022-04-03T19:47:16.853 INFO:tasks.workunit.client.0.smithi077.stdout: create 257/.snap/snap-subdir-test 2022-04-03T19:47:16.873 INFO:tasks.workunit.client.0.smithi077.stdout: create 257/30 file after the snapshot 2022-04-03T19:47:16.874 INFO:tasks.workunit.client.0.smithi077.stdout: create 257/31 file after the snapshot 2022-04-03T19:47:16.875 INFO:tasks.workunit.client.0.smithi077.stdout: create 257/32 file after the snapshot 2022-04-03T19:47:16.877 INFO:tasks.workunit.client.0.smithi077.stdout: create 257/33 file after the snapshot 2022-04-03T19:47:16.877 INFO:tasks.workunit.client.0.smithi077.stdout: create 257/34 file after the snapshot 2022-04-03T19:47:16.879 INFO:tasks.workunit.client.0.smithi077.stdout: create 257/35 file after the snapshot 2022-04-03T19:47:16.880 INFO:tasks.workunit.client.0.smithi077.stdout: create 257/36 file after the snapshot 2022-04-03T19:47:16.881 INFO:tasks.workunit.client.0.smithi077.stdout: create 257/37 file after the snapshot 2022-04-03T19:47:16.882 INFO:tasks.workunit.client.0.smithi077.stdout: create 257/38 file after the snapshot 2022-04-03T19:47:16.883 INFO:tasks.workunit.client.0.smithi077.stdout: create 257/39 file after the snapshot 2022-04-03T19:47:16.885 INFO:tasks.workunit.client.0.smithi077.stdout: create 258/.snap/snap-subdir-test 2022-04-03T19:47:16.903 INFO:tasks.workunit.client.0.smithi077.stdout: create 258/30 file after the snapshot 2022-04-03T19:47:16.904 INFO:tasks.workunit.client.0.smithi077.stdout: create 258/31 file after the snapshot 2022-04-03T19:47:16.905 INFO:tasks.workunit.client.0.smithi077.stdout: create 258/32 file after the snapshot 2022-04-03T19:47:16.906 INFO:tasks.workunit.client.0.smithi077.stdout: create 258/33 file after the snapshot 2022-04-03T19:47:16.907 INFO:tasks.workunit.client.0.smithi077.stdout: create 258/34 file after the snapshot 2022-04-03T19:47:16.908 INFO:tasks.workunit.client.0.smithi077.stdout: create 258/35 file after the snapshot 2022-04-03T19:47:16.909 INFO:tasks.workunit.client.0.smithi077.stdout: create 258/36 file after the snapshot 2022-04-03T19:47:16.910 INFO:tasks.workunit.client.0.smithi077.stdout: create 258/37 file after the snapshot 2022-04-03T19:47:16.911 INFO:tasks.workunit.client.0.smithi077.stdout: create 258/38 file after the snapshot 2022-04-03T19:47:16.912 INFO:tasks.workunit.client.0.smithi077.stdout: create 258/39 file after the snapshot 2022-04-03T19:47:16.914 INFO:tasks.workunit.client.0.smithi077.stdout: create 259/.snap/snap-subdir-test 2022-04-03T19:47:16.932 INFO:tasks.workunit.client.0.smithi077.stdout: create 259/30 file after the snapshot 2022-04-03T19:47:16.933 INFO:tasks.workunit.client.0.smithi077.stdout: create 259/31 file after the snapshot 2022-04-03T19:47:16.934 INFO:tasks.workunit.client.0.smithi077.stdout: create 259/32 file after the snapshot 2022-04-03T19:47:16.935 INFO:tasks.workunit.client.0.smithi077.stdout: create 259/33 file after the snapshot 2022-04-03T19:47:16.936 INFO:tasks.workunit.client.0.smithi077.stdout: create 259/34 file after the snapshot 2022-04-03T19:47:16.937 INFO:tasks.workunit.client.0.smithi077.stdout: create 259/35 file after the snapshot 2022-04-03T19:47:16.937 INFO:tasks.workunit.client.0.smithi077.stdout: create 259/36 file after the snapshot 2022-04-03T19:47:16.938 INFO:tasks.workunit.client.0.smithi077.stdout: create 259/37 file after the snapshot 2022-04-03T19:47:16.939 INFO:tasks.workunit.client.0.smithi077.stdout: create 259/38 file after the snapshot 2022-04-03T19:47:16.940 INFO:tasks.workunit.client.0.smithi077.stdout: create 259/39 file after the snapshot 2022-04-03T19:47:16.941 INFO:tasks.workunit.client.0.smithi077.stdout: create 260/.snap/snap-subdir-test 2022-04-03T19:47:16.959 INFO:tasks.workunit.client.0.smithi077.stdout: create 260/30 file after the snapshot 2022-04-03T19:47:16.960 INFO:tasks.workunit.client.0.smithi077.stdout: create 260/31 file after the snapshot 2022-04-03T19:47:16.962 INFO:tasks.workunit.client.0.smithi077.stdout: create 260/32 file after the snapshot 2022-04-03T19:47:16.963 INFO:tasks.workunit.client.0.smithi077.stdout: create 260/33 file after the snapshot 2022-04-03T19:47:16.964 INFO:tasks.workunit.client.0.smithi077.stdout: create 260/34 file after the snapshot 2022-04-03T19:47:16.965 INFO:tasks.workunit.client.0.smithi077.stdout: create 260/35 file after the snapshot 2022-04-03T19:47:16.966 INFO:tasks.workunit.client.0.smithi077.stdout: create 260/36 file after the snapshot 2022-04-03T19:47:16.967 INFO:tasks.workunit.client.0.smithi077.stdout: create 260/37 file after the snapshot 2022-04-03T19:47:16.968 INFO:tasks.workunit.client.0.smithi077.stdout: create 260/38 file after the snapshot 2022-04-03T19:47:16.969 INFO:tasks.workunit.client.0.smithi077.stdout: create 260/39 file after the snapshot 2022-04-03T19:47:16.970 INFO:tasks.workunit.client.0.smithi077.stdout: create 261/.snap/snap-subdir-test 2022-04-03T19:47:16.988 INFO:tasks.workunit.client.0.smithi077.stdout: create 261/30 file after the snapshot 2022-04-03T19:47:16.989 INFO:tasks.workunit.client.0.smithi077.stdout: create 261/31 file after the snapshot 2022-04-03T19:47:16.990 INFO:tasks.workunit.client.0.smithi077.stdout: create 261/32 file after the snapshot 2022-04-03T19:47:16.991 INFO:tasks.workunit.client.0.smithi077.stdout: create 261/33 file after the snapshot 2022-04-03T19:47:16.992 INFO:tasks.workunit.client.0.smithi077.stdout: create 261/34 file after the snapshot 2022-04-03T19:47:16.993 INFO:tasks.workunit.client.0.smithi077.stdout: create 261/35 file after the snapshot 2022-04-03T19:47:16.995 INFO:tasks.workunit.client.0.smithi077.stdout: create 261/36 file after the snapshot 2022-04-03T19:47:16.996 INFO:tasks.workunit.client.0.smithi077.stdout: create 261/37 file after the snapshot 2022-04-03T19:47:16.997 INFO:tasks.workunit.client.0.smithi077.stdout: create 261/38 file after the snapshot 2022-04-03T19:47:16.998 INFO:tasks.workunit.client.0.smithi077.stdout: create 261/39 file after the snapshot 2022-04-03T19:47:16.999 INFO:tasks.workunit.client.0.smithi077.stdout: create 262/.snap/snap-subdir-test 2022-04-03T19:47:17.016 INFO:tasks.workunit.client.0.smithi077.stdout: create 262/30 file after the snapshot 2022-04-03T19:47:17.017 INFO:tasks.workunit.client.0.smithi077.stdout: create 262/31 file after the snapshot 2022-04-03T19:47:17.019 INFO:tasks.workunit.client.0.smithi077.stdout: create 262/32 file after the snapshot 2022-04-03T19:47:17.019 INFO:tasks.workunit.client.0.smithi077.stdout: create 262/33 file after the snapshot 2022-04-03T19:47:17.020 INFO:tasks.workunit.client.0.smithi077.stdout: create 262/34 file after the snapshot 2022-04-03T19:47:17.021 INFO:tasks.workunit.client.0.smithi077.stdout: create 262/35 file after the snapshot 2022-04-03T19:47:17.022 INFO:tasks.workunit.client.0.smithi077.stdout: create 262/36 file after the snapshot 2022-04-03T19:47:17.023 INFO:tasks.workunit.client.0.smithi077.stdout: create 262/37 file after the snapshot 2022-04-03T19:47:17.024 INFO:tasks.workunit.client.0.smithi077.stdout: create 262/38 file after the snapshot 2022-04-03T19:47:17.025 INFO:tasks.workunit.client.0.smithi077.stdout: create 262/39 file after the snapshot 2022-04-03T19:47:17.026 INFO:tasks.workunit.client.0.smithi077.stdout: create 263/.snap/snap-subdir-test 2022-04-03T19:47:17.719 INFO:tasks.workunit.client.0.smithi077.stdout: create 263/30 file after the snapshot 2022-04-03T19:47:17.720 INFO:tasks.workunit.client.0.smithi077.stdout: create 263/31 file after the snapshot 2022-04-03T19:47:17.721 INFO:tasks.workunit.client.0.smithi077.stdout: create 263/32 file after the snapshot 2022-04-03T19:47:17.723 INFO:tasks.workunit.client.0.smithi077.stdout: create 263/33 file after the snapshot 2022-04-03T19:47:17.724 INFO:tasks.workunit.client.0.smithi077.stdout: create 263/34 file after the snapshot 2022-04-03T19:47:17.725 INFO:tasks.workunit.client.0.smithi077.stdout: create 263/35 file after the snapshot 2022-04-03T19:47:17.727 INFO:tasks.workunit.client.0.smithi077.stdout: create 263/36 file after the snapshot 2022-04-03T19:47:17.728 INFO:tasks.workunit.client.0.smithi077.stdout: create 263/37 file after the snapshot 2022-04-03T19:47:17.729 INFO:tasks.workunit.client.0.smithi077.stdout: create 263/38 file after the snapshot 2022-04-03T19:47:17.730 INFO:tasks.workunit.client.0.smithi077.stdout: create 263/39 file after the snapshot 2022-04-03T19:47:17.732 INFO:tasks.workunit.client.0.smithi077.stdout: create 264/.snap/snap-subdir-test 2022-04-03T19:47:17.752 INFO:tasks.workunit.client.0.smithi077.stdout: create 264/30 file after the snapshot 2022-04-03T19:47:17.754 INFO:tasks.workunit.client.0.smithi077.stdout: create 264/31 file after the snapshot 2022-04-03T19:47:17.756 INFO:tasks.workunit.client.0.smithi077.stdout: create 264/32 file after the snapshot 2022-04-03T19:47:17.757 INFO:tasks.workunit.client.0.smithi077.stdout: create 264/33 file after the snapshot 2022-04-03T19:47:17.759 INFO:tasks.workunit.client.0.smithi077.stdout: create 264/34 file after the snapshot 2022-04-03T19:47:17.761 INFO:tasks.workunit.client.0.smithi077.stdout: create 264/35 file after the snapshot 2022-04-03T19:47:17.762 INFO:tasks.workunit.client.0.smithi077.stdout: create 264/36 file after the snapshot 2022-04-03T19:47:17.763 INFO:tasks.workunit.client.0.smithi077.stdout: create 264/37 file after the snapshot 2022-04-03T19:47:17.765 INFO:tasks.workunit.client.0.smithi077.stdout: create 264/38 file after the snapshot 2022-04-03T19:47:17.766 INFO:tasks.workunit.client.0.smithi077.stdout: create 264/39 file after the snapshot 2022-04-03T19:47:17.768 INFO:tasks.workunit.client.0.smithi077.stdout: create 265/.snap/snap-subdir-test 2022-04-03T19:47:18.623 INFO:tasks.workunit.client.0.smithi077.stdout: create 265/30 file after the snapshot 2022-04-03T19:47:18.624 INFO:tasks.workunit.client.0.smithi077.stdout: create 265/31 file after the snapshot 2022-04-03T19:47:18.626 INFO:tasks.workunit.client.0.smithi077.stdout: create 265/32 file after the snapshot 2022-04-03T19:47:18.628 INFO:tasks.workunit.client.0.smithi077.stdout: create 265/33 file after the snapshot 2022-04-03T19:47:18.629 INFO:tasks.workunit.client.0.smithi077.stdout: create 265/34 file after the snapshot 2022-04-03T19:47:18.631 INFO:tasks.workunit.client.0.smithi077.stdout: create 265/35 file after the snapshot 2022-04-03T19:47:18.632 INFO:tasks.workunit.client.0.smithi077.stdout: create 265/36 file after the snapshot 2022-04-03T19:47:18.634 INFO:tasks.workunit.client.0.smithi077.stdout: create 265/37 file after the snapshot 2022-04-03T19:47:18.635 INFO:tasks.workunit.client.0.smithi077.stdout: create 265/38 file after the snapshot 2022-04-03T19:47:18.637 INFO:tasks.workunit.client.0.smithi077.stdout: create 265/39 file after the snapshot 2022-04-03T19:47:18.638 INFO:tasks.workunit.client.0.smithi077.stdout: create 266/.snap/snap-subdir-test 2022-04-03T19:47:18.659 INFO:tasks.workunit.client.0.smithi077.stdout: create 266/30 file after the snapshot 2022-04-03T19:47:18.662 INFO:tasks.workunit.client.0.smithi077.stdout: create 266/31 file after the snapshot 2022-04-03T19:47:18.663 INFO:tasks.workunit.client.0.smithi077.stdout: create 266/32 file after the snapshot 2022-04-03T19:47:18.665 INFO:tasks.workunit.client.0.smithi077.stdout: create 266/33 file after the snapshot 2022-04-03T19:47:18.666 INFO:tasks.workunit.client.0.smithi077.stdout: create 266/34 file after the snapshot 2022-04-03T19:47:18.668 INFO:tasks.workunit.client.0.smithi077.stdout: create 266/35 file after the snapshot 2022-04-03T19:47:18.669 INFO:tasks.workunit.client.0.smithi077.stdout: create 266/36 file after the snapshot 2022-04-03T19:47:18.670 INFO:tasks.workunit.client.0.smithi077.stdout: create 266/37 file after the snapshot 2022-04-03T19:47:18.671 INFO:tasks.workunit.client.0.smithi077.stdout: create 266/38 file after the snapshot 2022-04-03T19:47:18.672 INFO:tasks.workunit.client.0.smithi077.stdout: create 266/39 file after the snapshot 2022-04-03T19:47:18.673 INFO:tasks.workunit.client.0.smithi077.stdout: create 267/.snap/snap-subdir-test 2022-04-03T19:47:18.693 INFO:tasks.workunit.client.0.smithi077.stdout: create 267/30 file after the snapshot 2022-04-03T19:47:18.695 INFO:tasks.workunit.client.0.smithi077.stdout: create 267/31 file after the snapshot 2022-04-03T19:47:18.696 INFO:tasks.workunit.client.0.smithi077.stdout: create 267/32 file after the snapshot 2022-04-03T19:47:18.739 INFO:tasks.workunit.client.0.smithi077.stdout: create 267/33 file after the snapshot 2022-04-03T19:47:19.579 INFO:tasks.workunit.client.0.smithi077.stdout: create 267/34 file after the snapshot 2022-04-03T19:47:19.581 INFO:tasks.workunit.client.0.smithi077.stdout: create 267/35 file after the snapshot 2022-04-03T19:47:19.582 INFO:tasks.workunit.client.0.smithi077.stdout: create 267/36 file after the snapshot 2022-04-03T19:47:19.584 INFO:tasks.workunit.client.0.smithi077.stdout: create 267/37 file after the snapshot 2022-04-03T19:47:19.585 INFO:tasks.workunit.client.0.smithi077.stdout: create 267/38 file after the snapshot 2022-04-03T19:47:19.587 INFO:tasks.workunit.client.0.smithi077.stdout: create 267/39 file after the snapshot 2022-04-03T19:47:19.589 INFO:tasks.workunit.client.0.smithi077.stdout: create 268/.snap/snap-subdir-test 2022-04-03T19:47:19.613 INFO:tasks.workunit.client.0.smithi077.stdout: create 268/30 file after the snapshot 2022-04-03T19:47:19.615 INFO:tasks.workunit.client.0.smithi077.stdout: create 268/31 file after the snapshot 2022-04-03T19:47:19.617 INFO:tasks.workunit.client.0.smithi077.stdout: create 268/32 file after the snapshot 2022-04-03T19:47:19.618 INFO:tasks.workunit.client.0.smithi077.stdout: create 268/33 file after the snapshot 2022-04-03T19:47:19.620 INFO:tasks.workunit.client.0.smithi077.stdout: create 268/34 file after the snapshot 2022-04-03T19:47:19.621 INFO:tasks.workunit.client.0.smithi077.stdout: create 268/35 file after the snapshot 2022-04-03T19:47:19.623 INFO:tasks.workunit.client.0.smithi077.stdout: create 268/36 file after the snapshot 2022-04-03T19:47:19.624 INFO:tasks.workunit.client.0.smithi077.stdout: create 268/37 file after the snapshot 2022-04-03T19:47:19.626 INFO:tasks.workunit.client.0.smithi077.stdout: create 268/38 file after the snapshot 2022-04-03T19:47:19.627 INFO:tasks.workunit.client.0.smithi077.stdout: create 268/39 file after the snapshot 2022-04-03T19:47:19.629 INFO:tasks.workunit.client.0.smithi077.stdout: create 269/.snap/snap-subdir-test 2022-04-03T19:47:19.649 INFO:tasks.workunit.client.0.smithi077.stdout: create 269/30 file after the snapshot 2022-04-03T19:47:19.651 INFO:tasks.workunit.client.0.smithi077.stdout: create 269/31 file after the snapshot 2022-04-03T19:47:19.653 INFO:tasks.workunit.client.0.smithi077.stdout: create 269/32 file after the snapshot 2022-04-03T19:47:19.654 INFO:tasks.workunit.client.0.smithi077.stdout: create 269/33 file after the snapshot 2022-04-03T19:47:19.656 INFO:tasks.workunit.client.0.smithi077.stdout: create 269/34 file after the snapshot 2022-04-03T19:47:19.657 INFO:tasks.workunit.client.0.smithi077.stdout: create 269/35 file after the snapshot 2022-04-03T19:47:19.659 INFO:tasks.workunit.client.0.smithi077.stdout: create 269/36 file after the snapshot 2022-04-03T19:47:19.660 INFO:tasks.workunit.client.0.smithi077.stdout: create 269/37 file after the snapshot 2022-04-03T19:47:19.661 INFO:tasks.workunit.client.0.smithi077.stdout: create 269/38 file after the snapshot 2022-04-03T19:47:19.663 INFO:tasks.workunit.client.0.smithi077.stdout: create 269/39 file after the snapshot 2022-04-03T19:47:19.664 INFO:tasks.workunit.client.0.smithi077.stdout: create 270/.snap/snap-subdir-test 2022-04-03T19:47:19.685 INFO:tasks.workunit.client.0.smithi077.stdout: create 270/30 file after the snapshot 2022-04-03T19:47:19.686 INFO:tasks.workunit.client.0.smithi077.stdout: create 270/31 file after the snapshot 2022-04-03T19:47:19.687 INFO:tasks.workunit.client.0.smithi077.stdout: create 270/32 file after the snapshot 2022-04-03T19:47:19.688 INFO:tasks.workunit.client.0.smithi077.stdout: create 270/33 file after the snapshot 2022-04-03T19:47:19.690 INFO:tasks.workunit.client.0.smithi077.stdout: create 270/34 file after the snapshot 2022-04-03T19:47:19.691 INFO:tasks.workunit.client.0.smithi077.stdout: create 270/35 file after the snapshot 2022-04-03T19:47:19.692 INFO:tasks.workunit.client.0.smithi077.stdout: create 270/36 file after the snapshot 2022-04-03T19:47:19.693 INFO:tasks.workunit.client.0.smithi077.stdout: create 270/37 file after the snapshot 2022-04-03T19:47:19.694 INFO:tasks.workunit.client.0.smithi077.stdout: create 270/38 file after the snapshot 2022-04-03T19:47:19.695 INFO:tasks.workunit.client.0.smithi077.stdout: create 270/39 file after the snapshot 2022-04-03T19:47:19.697 INFO:tasks.workunit.client.0.smithi077.stdout: create 271/.snap/snap-subdir-test 2022-04-03T19:47:19.717 INFO:tasks.workunit.client.0.smithi077.stdout: create 271/30 file after the snapshot 2022-04-03T19:47:19.719 INFO:tasks.workunit.client.0.smithi077.stdout: create 271/31 file after the snapshot 2022-04-03T19:47:19.729 INFO:tasks.workunit.client.0.smithi077.stdout: create 271/32 file after the snapshot 2022-04-03T19:47:19.729 INFO:tasks.workunit.client.0.smithi077.stdout: create 271/33 file after the snapshot 2022-04-03T19:47:19.729 INFO:tasks.workunit.client.0.smithi077.stdout: create 271/34 file after the snapshot 2022-04-03T19:47:19.729 INFO:tasks.workunit.client.0.smithi077.stdout: create 271/35 file after the snapshot 2022-04-03T19:47:19.730 INFO:tasks.workunit.client.0.smithi077.stdout: create 271/36 file after the snapshot 2022-04-03T19:47:19.730 INFO:tasks.workunit.client.0.smithi077.stdout: create 271/37 file after the snapshot 2022-04-03T19:47:19.731 INFO:tasks.workunit.client.0.smithi077.stdout: create 271/38 file after the snapshot 2022-04-03T19:47:19.731 INFO:tasks.workunit.client.0.smithi077.stdout: create 271/39 file after the snapshot 2022-04-03T19:47:19.732 INFO:tasks.workunit.client.0.smithi077.stdout: create 272/.snap/snap-subdir-test 2022-04-03T19:47:19.752 INFO:tasks.workunit.client.0.smithi077.stdout: create 272/30 file after the snapshot 2022-04-03T19:47:19.754 INFO:tasks.workunit.client.0.smithi077.stdout: create 272/31 file after the snapshot 2022-04-03T19:47:19.755 INFO:tasks.workunit.client.0.smithi077.stdout: create 272/32 file after the snapshot 2022-04-03T19:47:19.757 INFO:tasks.workunit.client.0.smithi077.stdout: create 272/33 file after the snapshot 2022-04-03T19:47:19.758 INFO:tasks.workunit.client.0.smithi077.stdout: create 272/34 file after the snapshot 2022-04-03T19:47:19.760 INFO:tasks.workunit.client.0.smithi077.stdout: create 272/35 file after the snapshot 2022-04-03T19:47:19.761 INFO:tasks.workunit.client.0.smithi077.stdout: create 272/36 file after the snapshot 2022-04-03T19:47:19.762 INFO:tasks.workunit.client.0.smithi077.stdout: create 272/37 file after the snapshot 2022-04-03T19:47:19.764 INFO:tasks.workunit.client.0.smithi077.stdout: create 272/38 file after the snapshot 2022-04-03T19:47:19.765 INFO:tasks.workunit.client.0.smithi077.stdout: create 272/39 file after the snapshot 2022-04-03T19:47:19.766 INFO:tasks.workunit.client.0.smithi077.stdout: create 273/.snap/snap-subdir-test 2022-04-03T19:47:19.786 INFO:tasks.workunit.client.0.smithi077.stdout: create 273/30 file after the snapshot 2022-04-03T19:47:19.788 INFO:tasks.workunit.client.0.smithi077.stdout: create 273/31 file after the snapshot 2022-04-03T19:47:19.789 INFO:tasks.workunit.client.0.smithi077.stdout: create 273/32 file after the snapshot 2022-04-03T19:47:19.791 INFO:tasks.workunit.client.0.smithi077.stdout: create 273/33 file after the snapshot 2022-04-03T19:47:19.792 INFO:tasks.workunit.client.0.smithi077.stdout: create 273/34 file after the snapshot 2022-04-03T19:47:19.793 INFO:tasks.workunit.client.0.smithi077.stdout: create 273/35 file after the snapshot 2022-04-03T19:47:19.795 INFO:tasks.workunit.client.0.smithi077.stdout: create 273/36 file after the snapshot 2022-04-03T19:47:19.796 INFO:tasks.workunit.client.0.smithi077.stdout: create 273/37 file after the snapshot 2022-04-03T19:47:19.798 INFO:tasks.workunit.client.0.smithi077.stdout: create 273/38 file after the snapshot 2022-04-03T19:47:19.799 INFO:tasks.workunit.client.0.smithi077.stdout: create 273/39 file after the snapshot 2022-04-03T19:47:19.801 INFO:tasks.workunit.client.0.smithi077.stdout: create 274/.snap/snap-subdir-test 2022-04-03T19:47:19.820 INFO:tasks.workunit.client.0.smithi077.stdout: create 274/30 file after the snapshot 2022-04-03T19:47:19.821 INFO:tasks.workunit.client.0.smithi077.stdout: create 274/31 file after the snapshot 2022-04-03T19:47:19.823 INFO:tasks.workunit.client.0.smithi077.stdout: create 274/32 file after the snapshot 2022-04-03T19:47:19.824 INFO:tasks.workunit.client.0.smithi077.stdout: create 274/33 file after the snapshot 2022-04-03T19:47:19.825 INFO:tasks.workunit.client.0.smithi077.stdout: create 274/34 file after the snapshot 2022-04-03T19:47:19.827 INFO:tasks.workunit.client.0.smithi077.stdout: create 274/35 file after the snapshot 2022-04-03T19:47:19.828 INFO:tasks.workunit.client.0.smithi077.stdout: create 274/36 file after the snapshot 2022-04-03T19:47:19.830 INFO:tasks.workunit.client.0.smithi077.stdout: create 274/37 file after the snapshot 2022-04-03T19:47:19.831 INFO:tasks.workunit.client.0.smithi077.stdout: create 274/38 file after the snapshot 2022-04-03T19:47:19.832 INFO:tasks.workunit.client.0.smithi077.stdout: create 274/39 file after the snapshot 2022-04-03T19:47:19.833 INFO:tasks.workunit.client.0.smithi077.stdout: create 275/.snap/snap-subdir-test 2022-04-03T19:47:19.852 INFO:tasks.workunit.client.0.smithi077.stdout: create 275/30 file after the snapshot 2022-04-03T19:47:19.854 INFO:tasks.workunit.client.0.smithi077.stdout: create 275/31 file after the snapshot 2022-04-03T19:47:19.856 INFO:tasks.workunit.client.0.smithi077.stdout: create 275/32 file after the snapshot 2022-04-03T19:47:19.857 INFO:tasks.workunit.client.0.smithi077.stdout: create 275/33 file after the snapshot 2022-04-03T19:47:19.859 INFO:tasks.workunit.client.0.smithi077.stdout: create 275/34 file after the snapshot 2022-04-03T19:47:19.860 INFO:tasks.workunit.client.0.smithi077.stdout: create 275/35 file after the snapshot 2022-04-03T19:47:19.861 INFO:tasks.workunit.client.0.smithi077.stdout: create 275/36 file after the snapshot 2022-04-03T19:47:19.862 INFO:tasks.workunit.client.0.smithi077.stdout: create 275/37 file after the snapshot 2022-04-03T19:47:19.864 INFO:tasks.workunit.client.0.smithi077.stdout: create 275/38 file after the snapshot 2022-04-03T19:47:19.865 INFO:tasks.workunit.client.0.smithi077.stdout: create 275/39 file after the snapshot 2022-04-03T19:47:19.866 INFO:tasks.workunit.client.0.smithi077.stdout: create 276/.snap/snap-subdir-test 2022-04-03T19:47:19.914 INFO:tasks.ceph.mds.d.smithi186.stderr:2022-04-03T19:47:19.914+0000 7f0968b08700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-04-03T19:47:20.236 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2022-04-03T19:47:20.237 INFO:tasks.ceph.mds.e:Restarting daemon 2022-04-03T19:47:20.237 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2022-04-03T19:47:20.242 INFO:tasks.ceph.mds.e:Started 2022-04-03T19:47:20.243 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:47:20.398 INFO:tasks.ceph.mds.e.smithi077.stdout:starting mds.e at 2022-04-03T19:47:20.453 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.451+0000 7f76d2947700 1 Processor -- start 2022-04-03T19:47:20.453 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.452+0000 7f76d2947700 1 -- start start 2022-04-03T19:47:20.455 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.453+0000 7f76d2947700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f76cc0c2a90 0x7f76cc06c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:20.456 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.453+0000 7f76d2947700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc1562f0 con 0x7f76cc0c5930 2022-04-03T19:47:20.456 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.453+0000 7f76d2947700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc0c2f80 con 0x7f76cc06d3f0 2022-04-03T19:47:20.456 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.453+0000 7f76d2947700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76cc0c3260 con 0x7f76cc0c2a90 2022-04-03T19:47:20.456 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.453+0000 7f76d2146700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f76cc0c2a90 0x7f76cc06c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:20.457 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.453+0000 7f76d1144700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f76cc0c5930 0x7f76cc155be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38420/0 (socket says 172.21.15.77:38420) 2022-04-03T19:47:20.457 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.453+0000 7f76d1144700 1 -- 172.21.15.77:0/2767046961 learned_addr learned my addr 172.21.15.77:0/2767046961 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:47:20.457 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.453+0000 7f76d1945700 1 --1- 172.21.15.77:0/2767046961 >> v1:172.21.15.186:6789/0 conn(0x7f76cc06d3f0 0x7f76cc06d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44196/0 (socket says 172.21.15.77:44196) 2022-04-03T19:47:20.457 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.454+0000 7f76d0943700 1 -- 172.21.15.77:0/2767046961 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066865161 0 0) 0x7f76cc0c2f80 con 0x7f76cc06d3f0 2022-04-03T19:47:20.457 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.454+0000 7f76d0943700 1 -- 172.21.15.77:0/2767046961 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b40033f0 con 0x7f76cc06d3f0 2022-04-03T19:47:20.458 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.454+0000 7f76d2146700 1 -- 172.21.15.77:0/2767046961 >> v1:172.21.15.186:6789/0 conn(0x7f76cc06d3f0 legacy=0x7f76cc06d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:20.458 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.454+0000 7f76d0943700 1 -- 172.21.15.77:0/2767046961 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398571698 0 0) 0x7f76cc1562f0 con 0x7f76cc0c5930 2022-04-03T19:47:20.458 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.454+0000 7f76d2146700 1 -- 172.21.15.77:0/2767046961 >> v1:172.21.15.77:6789/0 conn(0x7f76cc0c5930 legacy=0x7f76cc155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:20.459 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.454+0000 7f76d2146700 1 -- 172.21.15.77:0/2767046961 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc0c3540 con 0x7f76cc0c2a90 2022-04-03T19:47:20.459 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.454+0000 7f76d2146700 1 --2- 172.21.15.77:0/2767046961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76cc0c2a90 0x7f76cc06c040 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009e90 tx=0x7f76c4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dbc8f9535fc78e0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:20.459 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.454+0000 7f76d0943700 1 -- 172.21.15.77:0/2767046961 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c400e070 con 0x7f76cc0c2a90 2022-04-03T19:47:20.459 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.454+0000 7f76d0943700 1 -- 172.21.15.77:0/2767046961 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c400bcd0 con 0x7f76cc0c2a90 2022-04-03T19:47:20.460 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.454+0000 7f76d0943700 1 -- 172.21.15.77:0/2767046961 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c4012680 con 0x7f76cc0c2a90 2022-04-03T19:47:20.463 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.455+0000 7f76d2947700 1 -- 172.21.15.77:0/2767046961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76cc0c2a90 msgr2=0x7f76cc06c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:20.463 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.455+0000 7f76d2947700 1 --2- 172.21.15.77:0/2767046961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76cc0c2a90 0x7f76cc06c040 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009e90 tx=0x7f76c4009780 comp rx=0 tx=0).stop 2022-04-03T19:47:20.464 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.455+0000 7f76d2947700 1 -- 172.21.15.77:0/2767046961 shutdown_connections 2022-04-03T19:47:20.464 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.455+0000 7f76d2947700 1 --2- 172.21.15.77:0/2767046961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76cc0c2a90 0x7f76cc06c040 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:20.464 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.455+0000 7f76d2947700 1 -- 172.21.15.77:0/2767046961 >> 172.21.15.77:0/2767046961 conn(0x7f76cc0cb720 msgr2=0x7f76cc0cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:47:20.464 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.455+0000 7f76d2947700 1 -- 172.21.15.77:0/2767046961 shutdown_connections 2022-04-03T19:47:20.465 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.455+0000 7f76d2947700 1 -- 172.21.15.77:0/2767046961 wait complete. 2022-04-03T19:47:20.465 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.456+0000 7f76d2947700 1 Processor -- start 2022-04-03T19:47:20.465 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.457+0000 7f76d2947700 1 -- start start 2022-04-03T19:47:20.465 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.457+0000 7f76d2947700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76cc06d3f0 0x7f76cc166740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:20.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.457+0000 7f76d2947700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76cc0c2a90 0x7f76cc166c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:20.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.458+0000 7f76d2947700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76cc0c5930 0x7f76cc158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:20.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.458+0000 7f76d2947700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc0ca2d0 con 0x7f76cc0c5930 2022-04-03T19:47:20.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.458+0000 7f76d2947700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc0ca030 con 0x7f76cc0c2a90 2022-04-03T19:47:20.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.458+0000 7f76d2947700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc150aa0 con 0x7f76cc06d3f0 2022-04-03T19:47:20.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.458+0000 7f76d1144700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76cc0c2a90 0x7f76cc166c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:20.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.458+0000 7f76d2146700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76cc0c5930 0x7f76cc158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:20.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.458+0000 7f76d1945700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76cc06d3f0 0x7f76cc166740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:20.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.458+0000 7f76d1144700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76cc0c2a90 0x7f76cc166c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37996/0 (socket says 172.21.15.77:37996) 2022-04-03T19:47:20.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.458+0000 7f76d1144700 1 -- 172.21.15.77:0/4116120640 learned_addr learned my addr 172.21.15.77:0/4116120640 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:47:20.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.458+0000 7f76d1144700 1 -- 172.21.15.77:0/4116120640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76cc06d3f0 msgr2=0x7f76cc166740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:20.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.458+0000 7f76d1144700 1 --2- 172.21.15.77:0/4116120640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76cc06d3f0 0x7f76cc166740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:20.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.458+0000 7f76d1144700 1 -- 172.21.15.77:0/4116120640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76cc0c5930 msgr2=0x7f76cc158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:20.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.458+0000 7f76d1144700 1 --2- 172.21.15.77:0/4116120640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76cc0c5930 0x7f76cc158790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:20.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.458+0000 7f76d1144700 1 -- 172.21.15.77:0/4116120640 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c4009a20 con 0x7f76cc0c2a90 2022-04-03T19:47:20.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.458+0000 7f76d1945700 1 --2- 172.21.15.77:0/4116120640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76cc06d3f0 0x7f76cc166740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:47:20.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.459+0000 7f76d1144700 1 --2- 172.21.15.77:0/4116120640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76cc0c2a90 0x7f76cc166c80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f76bc00c440 tx=0x7f76bc007d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:20.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.459+0000 7f76b9ffb700 1 -- 172.21.15.77:0/4116120640 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76bc007360 con 0x7f76cc0c2a90 2022-04-03T19:47:20.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.459+0000 7f76b9ffb700 1 -- 172.21.15.77:0/4116120640 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76bc0074c0 con 0x7f76cc0c2a90 2022-04-03T19:47:20.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.459+0000 7f76d2947700 1 -- 172.21.15.77:0/4116120640 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76cc150d80 con 0x7f76cc0c2a90 2022-04-03T19:47:20.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.459+0000 7f76b9ffb700 1 -- 172.21.15.77:0/4116120640 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76bc004900 con 0x7f76cc0c2a90 2022-04-03T19:47:20.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.459+0000 7f76d2947700 1 -- 172.21.15.77:0/4116120640 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76cc151340 con 0x7f76cc0c2a90 2022-04-03T19:47:20.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.460+0000 7f76d2947700 1 -- 172.21.15.77:0/4116120640 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76cc151560 con 0x7f76cc0c2a90 2022-04-03T19:47:20.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.461+0000 7f76b9ffb700 1 -- 172.21.15.77:0/4116120640 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f76bc010030 con 0x7f76cc0c2a90 2022-04-03T19:47:20.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.462+0000 7f76b9ffb700 1 --2- 172.21.15.77:0/4116120640 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f76ac063ce0 0x7f76ac066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:20.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.462+0000 7f76b9ffb700 1 -- 172.21.15.77:0/4116120640 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f76bc080310 con 0x7f76cc0c2a90 2022-04-03T19:47:20.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.462+0000 7f76d1945700 1 --2- 172.21.15.77:0/4116120640 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f76ac063ce0 0x7f76ac066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:20.472 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.463+0000 7f76d1945700 1 --2- 172.21.15.77:0/4116120640 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f76ac063ce0 0x7f76ac066190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f76c8007990 tx=0x7f76c8007bb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:20.472 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.466+0000 7f76b9ffb700 1 -- 172.21.15.77:0/4116120640 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f76bc04eee0 con 0x7f76cc0c2a90 2022-04-03T19:47:20.541 INFO:tasks.workunit.client.0.smithi077.stdout: create 276/30 file after the snapshot 2022-04-03T19:47:20.542 INFO:tasks.workunit.client.0.smithi077.stdout: create 276/31 file after the snapshot 2022-04-03T19:47:20.543 INFO:tasks.workunit.client.0.smithi077.stdout: create 276/32 file after the snapshot 2022-04-03T19:47:20.544 INFO:tasks.workunit.client.0.smithi077.stdout: create 276/33 file after the snapshot 2022-04-03T19:47:20.545 INFO:tasks.workunit.client.0.smithi077.stdout: create 276/34 file after the snapshot 2022-04-03T19:47:20.546 INFO:tasks.workunit.client.0.smithi077.stdout: create 276/35 file after the snapshot 2022-04-03T19:47:20.548 INFO:tasks.workunit.client.0.smithi077.stdout: create 276/36 file after the snapshot 2022-04-03T19:47:20.549 INFO:tasks.workunit.client.0.smithi077.stdout: create 276/37 file after the snapshot 2022-04-03T19:47:20.550 INFO:tasks.workunit.client.0.smithi077.stdout: create 276/38 file after the snapshot 2022-04-03T19:47:20.551 INFO:tasks.workunit.client.0.smithi077.stdout: create 276/39 file after the snapshot 2022-04-03T19:47:20.552 INFO:tasks.workunit.client.0.smithi077.stdout: create 277/.snap/snap-subdir-test 2022-04-03T19:47:20.571 INFO:tasks.workunit.client.0.smithi077.stdout: create 277/30 file after the snapshot 2022-04-03T19:47:20.573 INFO:tasks.workunit.client.0.smithi077.stdout: create 277/31 file after the snapshot 2022-04-03T19:47:20.574 INFO:tasks.workunit.client.0.smithi077.stdout: create 277/32 file after the snapshot 2022-04-03T19:47:20.576 INFO:tasks.workunit.client.0.smithi077.stdout: create 277/33 file after the snapshot 2022-04-03T19:47:20.577 INFO:tasks.workunit.client.0.smithi077.stdout: create 277/34 file after the snapshot 2022-04-03T19:47:20.577 INFO:tasks.workunit.client.0.smithi077.stdout: create 277/35 file after the snapshot 2022-04-03T19:47:20.579 INFO:tasks.workunit.client.0.smithi077.stdout: create 277/36 file after the snapshot 2022-04-03T19:47:20.580 INFO:tasks.workunit.client.0.smithi077.stdout: create 277/37 file after the snapshot 2022-04-03T19:47:20.581 INFO:tasks.workunit.client.0.smithi077.stdout: create 277/38 file after the snapshot 2022-04-03T19:47:20.582 INFO:tasks.workunit.client.0.smithi077.stdout: create 277/39 file after the snapshot 2022-04-03T19:47:20.583 INFO:tasks.workunit.client.0.smithi077.stdout: create 278/.snap/snap-subdir-test 2022-04-03T19:47:20.602 INFO:tasks.workunit.client.0.smithi077.stdout: create 278/30 file after the snapshot 2022-04-03T19:47:20.604 INFO:tasks.workunit.client.0.smithi077.stdout: create 278/31 file after the snapshot 2022-04-03T19:47:20.605 INFO:tasks.workunit.client.0.smithi077.stdout: create 278/32 file after the snapshot 2022-04-03T19:47:20.606 INFO:tasks.workunit.client.0.smithi077.stdout: create 278/33 file after the snapshot 2022-04-03T19:47:20.607 INFO:tasks.workunit.client.0.smithi077.stdout: create 278/34 file after the snapshot 2022-04-03T19:47:20.608 INFO:tasks.workunit.client.0.smithi077.stdout: create 278/35 file after the snapshot 2022-04-03T19:47:20.609 INFO:tasks.workunit.client.0.smithi077.stdout: create 278/36 file after the snapshot 2022-04-03T19:47:20.610 INFO:tasks.workunit.client.0.smithi077.stdout: create 278/37 file after the snapshot 2022-04-03T19:47:20.611 INFO:tasks.workunit.client.0.smithi077.stdout: create 278/38 file after the snapshot 2022-04-03T19:47:20.612 INFO:tasks.workunit.client.0.smithi077.stdout: create 278/39 file after the snapshot 2022-04-03T19:47:20.613 INFO:tasks.workunit.client.0.smithi077.stdout: create 279/.snap/snap-subdir-test 2022-04-03T19:47:20.631 INFO:tasks.workunit.client.0.smithi077.stdout: create 279/30 file after the snapshot 2022-04-03T19:47:20.632 INFO:tasks.workunit.client.0.smithi077.stdout: create 279/31 file after the snapshot 2022-04-03T19:47:20.633 INFO:tasks.workunit.client.0.smithi077.stdout: create 279/32 file after the snapshot 2022-04-03T19:47:20.634 INFO:tasks.workunit.client.0.smithi077.stdout: create 279/33 file after the snapshot 2022-04-03T19:47:20.635 INFO:tasks.workunit.client.0.smithi077.stdout: create 279/34 file after the snapshot 2022-04-03T19:47:20.636 INFO:tasks.workunit.client.0.smithi077.stdout: create 279/35 file after the snapshot 2022-04-03T19:47:20.637 INFO:tasks.workunit.client.0.smithi077.stdout: create 279/36 file after the snapshot 2022-04-03T19:47:20.638 INFO:tasks.workunit.client.0.smithi077.stdout: create 279/37 file after the snapshot 2022-04-03T19:47:20.638 INFO:tasks.workunit.client.0.smithi077.stdout: create 279/38 file after the snapshot 2022-04-03T19:47:20.639 INFO:tasks.workunit.client.0.smithi077.stdout: create 279/39 file after the snapshot 2022-04-03T19:47:20.640 INFO:tasks.workunit.client.0.smithi077.stdout: create 280/.snap/snap-subdir-test 2022-04-03T19:47:20.659 INFO:tasks.workunit.client.0.smithi077.stdout: create 280/30 file after the snapshot 2022-04-03T19:47:20.660 INFO:tasks.workunit.client.0.smithi077.stdout: create 280/31 file after the snapshot 2022-04-03T19:47:20.661 INFO:tasks.workunit.client.0.smithi077.stdout: create 280/32 file after the snapshot 2022-04-03T19:47:20.662 INFO:tasks.workunit.client.0.smithi077.stdout: create 280/33 file after the snapshot 2022-04-03T19:47:20.663 INFO:tasks.workunit.client.0.smithi077.stdout: create 280/34 file after the snapshot 2022-04-03T19:47:20.664 INFO:tasks.workunit.client.0.smithi077.stdout: create 280/35 file after the snapshot 2022-04-03T19:47:20.665 INFO:tasks.workunit.client.0.smithi077.stdout: create 280/36 file after the snapshot 2022-04-03T19:47:20.666 INFO:tasks.workunit.client.0.smithi077.stdout: create 280/37 file after the snapshot 2022-04-03T19:47:20.667 INFO:tasks.workunit.client.0.smithi077.stdout: create 280/38 file after the snapshot 2022-04-03T19:47:20.668 INFO:tasks.workunit.client.0.smithi077.stdout: create 280/39 file after the snapshot 2022-04-03T19:47:20.669 INFO:tasks.workunit.client.0.smithi077.stdout: create 281/.snap/snap-subdir-test 2022-04-03T19:47:20.687 INFO:tasks.workunit.client.0.smithi077.stdout: create 281/30 file after the snapshot 2022-04-03T19:47:20.689 INFO:tasks.workunit.client.0.smithi077.stdout: create 281/31 file after the snapshot 2022-04-03T19:47:20.690 INFO:tasks.workunit.client.0.smithi077.stdout: create 281/32 file after the snapshot 2022-04-03T19:47:20.691 INFO:tasks.workunit.client.0.smithi077.stdout: create 281/33 file after the snapshot 2022-04-03T19:47:20.692 INFO:tasks.workunit.client.0.smithi077.stdout: create 281/34 file after the snapshot 2022-04-03T19:47:20.693 INFO:tasks.workunit.client.0.smithi077.stdout: create 281/35 file after the snapshot 2022-04-03T19:47:20.694 INFO:tasks.workunit.client.0.smithi077.stdout: create 281/36 file after the snapshot 2022-04-03T19:47:20.695 INFO:tasks.workunit.client.0.smithi077.stdout: create 281/37 file after the snapshot 2022-04-03T19:47:20.695 INFO:tasks.workunit.client.0.smithi077.stdout: create 281/38 file after the snapshot 2022-04-03T19:47:20.696 INFO:tasks.workunit.client.0.smithi077.stdout: create 281/39 file after the snapshot 2022-04-03T19:47:20.697 INFO:tasks.workunit.client.0.smithi077.stdout: create 282/.snap/snap-subdir-test 2022-04-03T19:47:20.716 INFO:tasks.workunit.client.0.smithi077.stdout: create 282/30 file after the snapshot 2022-04-03T19:47:20.717 INFO:tasks.workunit.client.0.smithi077.stdout: create 282/31 file after the snapshot 2022-04-03T19:47:20.718 INFO:tasks.workunit.client.0.smithi077.stdout: create 282/32 file after the snapshot 2022-04-03T19:47:20.719 INFO:tasks.workunit.client.0.smithi077.stdout: create 282/33 file after the snapshot 2022-04-03T19:47:20.720 INFO:tasks.workunit.client.0.smithi077.stdout: create 282/34 file after the snapshot 2022-04-03T19:47:20.721 INFO:tasks.workunit.client.0.smithi077.stdout: create 282/35 file after the snapshot 2022-04-03T19:47:20.722 INFO:tasks.workunit.client.0.smithi077.stdout: create 282/36 file after the snapshot 2022-04-03T19:47:20.723 INFO:tasks.workunit.client.0.smithi077.stdout: create 282/37 file after the snapshot 2022-04-03T19:47:20.724 INFO:tasks.workunit.client.0.smithi077.stdout: create 282/38 file after the snapshot 2022-04-03T19:47:20.725 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.724+0000 7f76d2947700 1 -- 172.21.15.77:0/4116120640 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76cc155de0 con 0x7f76cc0c2a90 2022-04-03T19:47:20.725 INFO:tasks.workunit.client.0.smithi077.stdout: create 282/39 file after the snapshot 2022-04-03T19:47:20.726 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.724+0000 7f76b9ffb700 1 -- 172.21.15.77:0/4116120640 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+5355 (secure 0 0 0) 0x7f76bc052500 con 0x7f76cc0c2a90 2022-04-03T19:47:20.726 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:47:20.727 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:47:10.107928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:active","state_seq":47,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:47:20.727 INFO:tasks.workunit.client.0.smithi077.stdout: create 283/.snap/snap-subdir-test 2022-04-03T19:47:20.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.727+0000 7f76d2947700 1 -- 172.21.15.77:0/4116120640 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f76ac063ce0 msgr2=0x7f76ac066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:20.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.727+0000 7f76d2947700 1 --2- 172.21.15.77:0/4116120640 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f76ac063ce0 0x7f76ac066190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f76c8007990 tx=0x7f76c8007bb0 comp rx=0 tx=0).stop 2022-04-03T19:47:20.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.727+0000 7f76d2947700 1 -- 172.21.15.77:0/4116120640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76cc0c2a90 msgr2=0x7f76cc166c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:20.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.727+0000 7f76d2947700 1 --2- 172.21.15.77:0/4116120640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76cc0c2a90 0x7f76cc166c80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f76bc00c440 tx=0x7f76bc007d90 comp rx=0 tx=0).stop 2022-04-03T19:47:20.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.727+0000 7f76d2947700 1 -- 172.21.15.77:0/4116120640 shutdown_connections 2022-04-03T19:47:20.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.727+0000 7f76d2947700 1 --2- 172.21.15.77:0/4116120640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76cc0c5930 0x7f76cc158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:20.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.727+0000 7f76d2947700 1 --2- 172.21.15.77:0/4116120640 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f76ac063ce0 0x7f76ac066190 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:20.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.727+0000 7f76d2947700 1 --2- 172.21.15.77:0/4116120640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76cc06d3f0 0x7f76cc166740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:20.731 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.727+0000 7f76d2947700 1 --2- 172.21.15.77:0/4116120640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76cc0c2a90 0x7f76cc166c80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:20.731 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.727+0000 7f76d2947700 1 -- 172.21.15.77:0/4116120640 >> 172.21.15.77:0/4116120640 conn(0x7f76cc0cb720 msgr2=0x7f76cc14dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:47:20.731 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.727+0000 7f76d2947700 1 -- 172.21.15.77:0/4116120640 shutdown_connections 2022-04-03T19:47:20.731 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:20.727+0000 7f76d2947700 1 -- 172.21.15.77:0/4116120640 wait complete. 2022-04-03T19:47:20.732 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 13 2022-04-03T19:47:20.753 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2022-04-03T19:47:21.159 INFO:tasks.workunit.client.0.smithi077.stdout: create 283/30 file after the snapshot 2022-04-03T19:47:21.160 INFO:tasks.workunit.client.0.smithi077.stdout: create 283/31 file after the snapshot 2022-04-03T19:47:21.161 INFO:tasks.workunit.client.0.smithi077.stdout: create 283/32 file after the snapshot 2022-04-03T19:47:21.162 INFO:tasks.workunit.client.0.smithi077.stdout: create 283/33 file after the snapshot 2022-04-03T19:47:21.163 INFO:tasks.workunit.client.0.smithi077.stdout: create 283/34 file after the snapshot 2022-04-03T19:47:21.164 INFO:tasks.workunit.client.0.smithi077.stdout: create 283/35 file after the snapshot 2022-04-03T19:47:21.165 INFO:tasks.workunit.client.0.smithi077.stdout: create 283/36 file after the snapshot 2022-04-03T19:47:21.166 INFO:tasks.workunit.client.0.smithi077.stdout: create 283/37 file after the snapshot 2022-04-03T19:47:21.168 INFO:tasks.workunit.client.0.smithi077.stdout: create 283/38 file after the snapshot 2022-04-03T19:47:21.169 INFO:tasks.workunit.client.0.smithi077.stdout: create 283/39 file after the snapshot 2022-04-03T19:47:21.170 INFO:tasks.workunit.client.0.smithi077.stdout: create 284/.snap/snap-subdir-test 2022-04-03T19:47:21.187 INFO:tasks.workunit.client.0.smithi077.stdout: create 284/30 file after the snapshot 2022-04-03T19:47:21.188 INFO:tasks.workunit.client.0.smithi077.stdout: create 284/31 file after the snapshot 2022-04-03T19:47:21.190 INFO:tasks.workunit.client.0.smithi077.stdout: create 284/32 file after the snapshot 2022-04-03T19:47:21.191 INFO:tasks.workunit.client.0.smithi077.stdout: create 284/33 file after the snapshot 2022-04-03T19:47:21.192 INFO:tasks.workunit.client.0.smithi077.stdout: create 284/34 file after the snapshot 2022-04-03T19:47:21.193 INFO:tasks.workunit.client.0.smithi077.stdout: create 284/35 file after the snapshot 2022-04-03T19:47:21.194 INFO:tasks.workunit.client.0.smithi077.stdout: create 284/36 file after the snapshot 2022-04-03T19:47:21.195 INFO:tasks.workunit.client.0.smithi077.stdout: create 284/37 file after the snapshot 2022-04-03T19:47:21.196 INFO:tasks.workunit.client.0.smithi077.stdout: create 284/38 file after the snapshot 2022-04-03T19:47:21.197 INFO:tasks.workunit.client.0.smithi077.stdout: create 284/39 file after the snapshot 2022-04-03T19:47:21.198 INFO:tasks.workunit.client.0.smithi077.stdout: create 285/.snap/snap-subdir-test 2022-04-03T19:47:21.215 INFO:tasks.workunit.client.0.smithi077.stdout: create 285/30 file after the snapshot 2022-04-03T19:47:21.217 INFO:tasks.workunit.client.0.smithi077.stdout: create 285/31 file after the snapshot 2022-04-03T19:47:21.218 INFO:tasks.workunit.client.0.smithi077.stdout: create 285/32 file after the snapshot 2022-04-03T19:47:21.219 INFO:tasks.workunit.client.0.smithi077.stdout: create 285/33 file after the snapshot 2022-04-03T19:47:21.220 INFO:tasks.workunit.client.0.smithi077.stdout: create 285/34 file after the snapshot 2022-04-03T19:47:21.221 INFO:tasks.workunit.client.0.smithi077.stdout: create 285/35 file after the snapshot 2022-04-03T19:47:21.222 INFO:tasks.workunit.client.0.smithi077.stdout: create 285/36 file after the snapshot 2022-04-03T19:47:21.224 INFO:tasks.workunit.client.0.smithi077.stdout: create 285/37 file after the snapshot 2022-04-03T19:47:21.225 INFO:tasks.workunit.client.0.smithi077.stdout: create 285/38 file after the snapshot 2022-04-03T19:47:21.226 INFO:tasks.workunit.client.0.smithi077.stdout: create 285/39 file after the snapshot 2022-04-03T19:47:21.227 INFO:tasks.workunit.client.0.smithi077.stdout: create 286/.snap/snap-subdir-test 2022-04-03T19:47:21.244 INFO:tasks.workunit.client.0.smithi077.stdout: create 286/30 file after the snapshot 2022-04-03T19:47:21.245 INFO:tasks.workunit.client.0.smithi077.stdout: create 286/31 file after the snapshot 2022-04-03T19:47:21.246 INFO:tasks.workunit.client.0.smithi077.stdout: create 286/32 file after the snapshot 2022-04-03T19:47:21.247 INFO:tasks.workunit.client.0.smithi077.stdout: create 286/33 file after the snapshot 2022-04-03T19:47:21.248 INFO:tasks.workunit.client.0.smithi077.stdout: create 286/34 file after the snapshot 2022-04-03T19:47:21.249 INFO:tasks.workunit.client.0.smithi077.stdout: create 286/35 file after the snapshot 2022-04-03T19:47:21.250 INFO:tasks.workunit.client.0.smithi077.stdout: create 286/36 file after the snapshot 2022-04-03T19:47:21.251 INFO:tasks.workunit.client.0.smithi077.stdout: create 286/37 file after the snapshot 2022-04-03T19:47:21.252 INFO:tasks.workunit.client.0.smithi077.stdout: create 286/38 file after the snapshot 2022-04-03T19:47:21.253 INFO:tasks.workunit.client.0.smithi077.stdout: create 286/39 file after the snapshot 2022-04-03T19:47:21.255 INFO:tasks.workunit.client.0.smithi077.stdout: create 287/.snap/snap-subdir-test 2022-04-03T19:47:21.273 INFO:tasks.workunit.client.0.smithi077.stdout: create 287/30 file after the snapshot 2022-04-03T19:47:21.274 INFO:tasks.workunit.client.0.smithi077.stdout: create 287/31 file after the snapshot 2022-04-03T19:47:21.275 INFO:tasks.workunit.client.0.smithi077.stdout: create 287/32 file after the snapshot 2022-04-03T19:47:21.276 INFO:tasks.workunit.client.0.smithi077.stdout: create 287/33 file after the snapshot 2022-04-03T19:47:21.277 INFO:tasks.workunit.client.0.smithi077.stdout: create 287/34 file after the snapshot 2022-04-03T19:47:21.278 INFO:tasks.workunit.client.0.smithi077.stdout: create 287/35 file after the snapshot 2022-04-03T19:47:21.279 INFO:tasks.workunit.client.0.smithi077.stdout: create 287/36 file after the snapshot 2022-04-03T19:47:21.280 INFO:tasks.workunit.client.0.smithi077.stdout: create 287/37 file after the snapshot 2022-04-03T19:47:21.281 INFO:tasks.workunit.client.0.smithi077.stdout: create 287/38 file after the snapshot 2022-04-03T19:47:21.547 INFO:tasks.workunit.client.0.smithi077.stdout: create 287/39 file after the snapshot 2022-04-03T19:47:21.548 INFO:tasks.workunit.client.0.smithi077.stdout: create 288/.snap/snap-subdir-test 2022-04-03T19:47:21.567 INFO:tasks.workunit.client.0.smithi077.stdout: create 288/30 file after the snapshot 2022-04-03T19:47:21.568 INFO:tasks.workunit.client.0.smithi077.stdout: create 288/31 file after the snapshot 2022-04-03T19:47:21.569 INFO:tasks.workunit.client.0.smithi077.stdout: create 288/32 file after the snapshot 2022-04-03T19:47:21.570 INFO:tasks.workunit.client.0.smithi077.stdout: create 288/33 file after the snapshot 2022-04-03T19:47:21.571 INFO:tasks.workunit.client.0.smithi077.stdout: create 288/34 file after the snapshot 2022-04-03T19:47:21.572 INFO:tasks.workunit.client.0.smithi077.stdout: create 288/35 file after the snapshot 2022-04-03T19:47:21.573 INFO:tasks.workunit.client.0.smithi077.stdout: create 288/36 file after the snapshot 2022-04-03T19:47:21.574 INFO:tasks.workunit.client.0.smithi077.stdout: create 288/37 file after the snapshot 2022-04-03T19:47:21.575 INFO:tasks.workunit.client.0.smithi077.stdout: create 288/38 file after the snapshot 2022-04-03T19:47:22.365 INFO:tasks.workunit.client.0.smithi077.stdout: create 288/39 file after the snapshot 2022-04-03T19:47:22.403 INFO:tasks.workunit.client.0.smithi077.stdout: create 289/.snap/snap-subdir-test 2022-04-03T19:47:22.490 INFO:tasks.workunit.client.0.smithi077.stdout: create 289/30 file after the snapshot 2022-04-03T19:47:22.491 INFO:tasks.workunit.client.0.smithi077.stdout: create 289/31 file after the snapshot 2022-04-03T19:47:22.492 INFO:tasks.workunit.client.0.smithi077.stdout: create 289/32 file after the snapshot 2022-04-03T19:47:22.493 INFO:tasks.workunit.client.0.smithi077.stdout: create 289/33 file after the snapshot 2022-04-03T19:47:22.494 INFO:tasks.workunit.client.0.smithi077.stdout: create 289/34 file after the snapshot 2022-04-03T19:47:22.496 INFO:tasks.workunit.client.0.smithi077.stdout: create 289/35 file after the snapshot 2022-04-03T19:47:22.497 INFO:tasks.workunit.client.0.smithi077.stdout: create 289/36 file after the snapshot 2022-04-03T19:47:22.498 INFO:tasks.workunit.client.0.smithi077.stdout: create 289/37 file after the snapshot 2022-04-03T19:47:22.499 INFO:tasks.workunit.client.0.smithi077.stdout: create 289/38 file after the snapshot 2022-04-03T19:47:22.500 INFO:tasks.workunit.client.0.smithi077.stdout: create 289/39 file after the snapshot 2022-04-03T19:47:22.501 INFO:tasks.workunit.client.0.smithi077.stdout: create 290/.snap/snap-subdir-test 2022-04-03T19:47:22.517 INFO:tasks.workunit.client.0.smithi077.stdout: create 290/30 file after the snapshot 2022-04-03T19:47:22.518 INFO:tasks.workunit.client.0.smithi077.stdout: create 290/31 file after the snapshot 2022-04-03T19:47:22.520 INFO:tasks.workunit.client.0.smithi077.stdout: create 290/32 file after the snapshot 2022-04-03T19:47:22.521 INFO:tasks.workunit.client.0.smithi077.stdout: create 290/33 file after the snapshot 2022-04-03T19:47:22.522 INFO:tasks.workunit.client.0.smithi077.stdout: create 290/34 file after the snapshot 2022-04-03T19:47:22.523 INFO:tasks.workunit.client.0.smithi077.stdout: create 290/35 file after the snapshot 2022-04-03T19:47:22.524 INFO:tasks.workunit.client.0.smithi077.stdout: create 290/36 file after the snapshot 2022-04-03T19:47:22.525 INFO:tasks.workunit.client.0.smithi077.stdout: create 290/37 file after the snapshot 2022-04-03T19:47:22.526 INFO:tasks.workunit.client.0.smithi077.stdout: create 290/38 file after the snapshot 2022-04-03T19:47:22.528 INFO:tasks.workunit.client.0.smithi077.stdout: create 290/39 file after the snapshot 2022-04-03T19:47:22.529 INFO:tasks.workunit.client.0.smithi077.stdout: create 291/.snap/snap-subdir-test 2022-04-03T19:47:22.545 INFO:tasks.workunit.client.0.smithi077.stdout: create 291/30 file after the snapshot 2022-04-03T19:47:22.547 INFO:tasks.workunit.client.0.smithi077.stdout: create 291/31 file after the snapshot 2022-04-03T19:47:22.548 INFO:tasks.workunit.client.0.smithi077.stdout: create 291/32 file after the snapshot 2022-04-03T19:47:22.549 INFO:tasks.workunit.client.0.smithi077.stdout: create 291/33 file after the snapshot 2022-04-03T19:47:22.550 INFO:tasks.workunit.client.0.smithi077.stdout: create 291/34 file after the snapshot 2022-04-03T19:47:22.551 INFO:tasks.workunit.client.0.smithi077.stdout: create 291/35 file after the snapshot 2022-04-03T19:47:22.552 INFO:tasks.workunit.client.0.smithi077.stdout: create 291/36 file after the snapshot 2022-04-03T19:47:22.553 INFO:tasks.workunit.client.0.smithi077.stdout: create 291/37 file after the snapshot 2022-04-03T19:47:22.554 INFO:tasks.workunit.client.0.smithi077.stdout: create 291/38 file after the snapshot 2022-04-03T19:47:22.555 INFO:tasks.workunit.client.0.smithi077.stdout: create 291/39 file after the snapshot 2022-04-03T19:47:22.556 INFO:tasks.workunit.client.0.smithi077.stdout: create 292/.snap/snap-subdir-test 2022-04-03T19:47:22.573 INFO:tasks.workunit.client.0.smithi077.stdout: create 292/30 file after the snapshot 2022-04-03T19:47:22.574 INFO:tasks.workunit.client.0.smithi077.stdout: create 292/31 file after the snapshot 2022-04-03T19:47:22.575 INFO:tasks.workunit.client.0.smithi077.stdout: create 292/32 file after the snapshot 2022-04-03T19:47:22.576 INFO:tasks.workunit.client.0.smithi077.stdout: create 292/33 file after the snapshot 2022-04-03T19:47:22.577 INFO:tasks.workunit.client.0.smithi077.stdout: create 292/34 file after the snapshot 2022-04-03T19:47:22.578 INFO:tasks.workunit.client.0.smithi077.stdout: create 292/35 file after the snapshot 2022-04-03T19:47:22.579 INFO:tasks.workunit.client.0.smithi077.stdout: create 292/36 file after the snapshot 2022-04-03T19:47:22.580 INFO:tasks.workunit.client.0.smithi077.stdout: create 292/37 file after the snapshot 2022-04-03T19:47:22.582 INFO:tasks.workunit.client.0.smithi077.stdout: create 292/38 file after the snapshot 2022-04-03T19:47:22.583 INFO:tasks.workunit.client.0.smithi077.stdout: create 292/39 file after the snapshot 2022-04-03T19:47:22.584 INFO:tasks.workunit.client.0.smithi077.stdout: create 293/.snap/snap-subdir-test 2022-04-03T19:47:22.601 INFO:tasks.workunit.client.0.smithi077.stdout: create 293/30 file after the snapshot 2022-04-03T19:47:22.602 INFO:tasks.workunit.client.0.smithi077.stdout: create 293/31 file after the snapshot 2022-04-03T19:47:22.604 INFO:tasks.workunit.client.0.smithi077.stdout: create 293/32 file after the snapshot 2022-04-03T19:47:22.604 INFO:tasks.workunit.client.0.smithi077.stdout: create 293/33 file after the snapshot 2022-04-03T19:47:22.605 INFO:tasks.workunit.client.0.smithi077.stdout: create 293/34 file after the snapshot 2022-04-03T19:47:22.607 INFO:tasks.workunit.client.0.smithi077.stdout: create 293/35 file after the snapshot 2022-04-03T19:47:22.608 INFO:tasks.workunit.client.0.smithi077.stdout: create 293/36 file after the snapshot 2022-04-03T19:47:22.609 INFO:tasks.workunit.client.0.smithi077.stdout: create 293/37 file after the snapshot 2022-04-03T19:47:22.610 INFO:tasks.workunit.client.0.smithi077.stdout: create 293/38 file after the snapshot 2022-04-03T19:47:22.611 INFO:tasks.workunit.client.0.smithi077.stdout: create 293/39 file after the snapshot 2022-04-03T19:47:22.612 INFO:tasks.workunit.client.0.smithi077.stdout: create 294/.snap/snap-subdir-test 2022-04-03T19:47:22.629 INFO:tasks.workunit.client.0.smithi077.stdout: create 294/30 file after the snapshot 2022-04-03T19:47:22.631 INFO:tasks.workunit.client.0.smithi077.stdout: create 294/31 file after the snapshot 2022-04-03T19:47:22.632 INFO:tasks.workunit.client.0.smithi077.stdout: create 294/32 file after the snapshot 2022-04-03T19:47:22.633 INFO:tasks.workunit.client.0.smithi077.stdout: create 294/33 file after the snapshot 2022-04-03T19:47:22.634 INFO:tasks.workunit.client.0.smithi077.stdout: create 294/34 file after the snapshot 2022-04-03T19:47:22.635 INFO:tasks.workunit.client.0.smithi077.stdout: create 294/35 file after the snapshot 2022-04-03T19:47:22.636 INFO:tasks.workunit.client.0.smithi077.stdout: create 294/36 file after the snapshot 2022-04-03T19:47:22.637 INFO:tasks.workunit.client.0.smithi077.stdout: create 294/37 file after the snapshot 2022-04-03T19:47:22.639 INFO:tasks.workunit.client.0.smithi077.stdout: create 294/38 file after the snapshot 2022-04-03T19:47:22.640 INFO:tasks.workunit.client.0.smithi077.stdout: create 294/39 file after the snapshot 2022-04-03T19:47:22.641 INFO:tasks.workunit.client.0.smithi077.stdout: create 295/.snap/snap-subdir-test 2022-04-03T19:47:22.658 INFO:tasks.workunit.client.0.smithi077.stdout: create 295/30 file after the snapshot 2022-04-03T19:47:22.660 INFO:tasks.workunit.client.0.smithi077.stdout: create 295/31 file after the snapshot 2022-04-03T19:47:22.661 INFO:tasks.workunit.client.0.smithi077.stdout: create 295/32 file after the snapshot 2022-04-03T19:47:22.663 INFO:tasks.workunit.client.0.smithi077.stdout: create 295/33 file after the snapshot 2022-04-03T19:47:22.664 INFO:tasks.workunit.client.0.smithi077.stdout: create 295/34 file after the snapshot 2022-04-03T19:47:22.665 INFO:tasks.workunit.client.0.smithi077.stdout: create 295/35 file after the snapshot 2022-04-03T19:47:22.667 INFO:tasks.workunit.client.0.smithi077.stdout: create 295/36 file after the snapshot 2022-04-03T19:47:22.668 INFO:tasks.workunit.client.0.smithi077.stdout: create 295/37 file after the snapshot 2022-04-03T19:47:22.669 INFO:tasks.workunit.client.0.smithi077.stdout: create 295/38 file after the snapshot 2022-04-03T19:47:22.671 INFO:tasks.workunit.client.0.smithi077.stdout: create 295/39 file after the snapshot 2022-04-03T19:47:22.672 INFO:tasks.workunit.client.0.smithi077.stdout: create 296/.snap/snap-subdir-test 2022-04-03T19:47:22.690 INFO:tasks.workunit.client.0.smithi077.stdout: create 296/30 file after the snapshot 2022-04-03T19:47:22.692 INFO:tasks.workunit.client.0.smithi077.stdout: create 296/31 file after the snapshot 2022-04-03T19:47:22.693 INFO:tasks.workunit.client.0.smithi077.stdout: create 296/32 file after the snapshot 2022-04-03T19:47:22.695 INFO:tasks.workunit.client.0.smithi077.stdout: create 296/33 file after the snapshot 2022-04-03T19:47:22.696 INFO:tasks.workunit.client.0.smithi077.stdout: create 296/34 file after the snapshot 2022-04-03T19:47:22.698 INFO:tasks.workunit.client.0.smithi077.stdout: create 296/35 file after the snapshot 2022-04-03T19:47:22.699 INFO:tasks.workunit.client.0.smithi077.stdout: create 296/36 file after the snapshot 2022-04-03T19:47:22.700 INFO:tasks.workunit.client.0.smithi077.stdout: create 296/37 file after the snapshot 2022-04-03T19:47:22.701 INFO:tasks.workunit.client.0.smithi077.stdout: create 296/38 file after the snapshot 2022-04-03T19:47:22.703 INFO:tasks.workunit.client.0.smithi077.stdout: create 296/39 file after the snapshot 2022-04-03T19:47:22.704 INFO:tasks.workunit.client.0.smithi077.stdout: create 297/.snap/snap-subdir-test 2022-04-03T19:47:22.721 INFO:tasks.workunit.client.0.smithi077.stdout: create 297/30 file after the snapshot 2022-04-03T19:47:22.722 INFO:tasks.workunit.client.0.smithi077.stdout: create 297/31 file after the snapshot 2022-04-03T19:47:22.723 INFO:tasks.workunit.client.0.smithi077.stdout: create 297/32 file after the snapshot 2022-04-03T19:47:22.724 INFO:tasks.workunit.client.0.smithi077.stdout: create 297/33 file after the snapshot 2022-04-03T19:47:22.725 INFO:tasks.workunit.client.0.smithi077.stdout: create 297/34 file after the snapshot 2022-04-03T19:47:22.726 INFO:tasks.workunit.client.0.smithi077.stdout: create 297/35 file after the snapshot 2022-04-03T19:47:22.727 INFO:tasks.workunit.client.0.smithi077.stdout: create 297/36 file after the snapshot 2022-04-03T19:47:22.728 INFO:tasks.workunit.client.0.smithi077.stdout: create 297/37 file after the snapshot 2022-04-03T19:47:22.729 INFO:tasks.workunit.client.0.smithi077.stdout: create 297/38 file after the snapshot 2022-04-03T19:47:22.730 INFO:tasks.workunit.client.0.smithi077.stdout: create 297/39 file after the snapshot 2022-04-03T19:47:22.731 INFO:tasks.workunit.client.0.smithi077.stdout: create 298/.snap/snap-subdir-test 2022-04-03T19:47:22.749 INFO:tasks.workunit.client.0.smithi077.stdout: create 298/30 file after the snapshot 2022-04-03T19:47:22.750 INFO:tasks.workunit.client.0.smithi077.stdout: create 298/31 file after the snapshot 2022-04-03T19:47:22.751 INFO:tasks.workunit.client.0.smithi077.stdout: create 298/32 file after the snapshot 2022-04-03T19:47:22.753 INFO:tasks.workunit.client.0.smithi077.stdout: create 298/33 file after the snapshot 2022-04-03T19:47:22.754 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:47:22.755 INFO:tasks.workunit.client.0.smithi077.stdout: create 298/34 file after the snapshot 2022-04-03T19:47:22.756 INFO:tasks.workunit.client.0.smithi077.stdout: create 298/35 file after the snapshot 2022-04-03T19:47:22.756 INFO:tasks.workunit.client.0.smithi077.stdout: create 298/36 file after the snapshot 2022-04-03T19:47:22.760 INFO:tasks.workunit.client.0.smithi077.stdout: create 298/37 file after the snapshot 2022-04-03T19:47:22.761 INFO:tasks.workunit.client.0.smithi077.stdout: create 298/38 file after the snapshot 2022-04-03T19:47:22.761 INFO:tasks.workunit.client.0.smithi077.stdout: create 298/39 file after the snapshot 2022-04-03T19:47:22.761 INFO:tasks.workunit.client.0.smithi077.stdout: create 299/.snap/snap-subdir-test 2022-04-03T19:47:22.781 INFO:tasks.workunit.client.0.smithi077.stdout: create 299/30 file after the snapshot 2022-04-03T19:47:22.782 INFO:tasks.workunit.client.0.smithi077.stdout: create 299/31 file after the snapshot 2022-04-03T19:47:22.783 INFO:tasks.workunit.client.0.smithi077.stdout: create 299/32 file after the snapshot 2022-04-03T19:47:22.784 INFO:tasks.workunit.client.0.smithi077.stdout: create 299/33 file after the snapshot 2022-04-03T19:47:22.786 INFO:tasks.workunit.client.0.smithi077.stdout: create 299/34 file after the snapshot 2022-04-03T19:47:22.787 INFO:tasks.workunit.client.0.smithi077.stdout: create 299/35 file after the snapshot 2022-04-03T19:47:22.788 INFO:tasks.workunit.client.0.smithi077.stdout: create 299/36 file after the snapshot 2022-04-03T19:47:22.789 INFO:tasks.workunit.client.0.smithi077.stdout: create 299/37 file after the snapshot 2022-04-03T19:47:22.790 INFO:tasks.workunit.client.0.smithi077.stdout: create 299/38 file after the snapshot 2022-04-03T19:47:22.791 INFO:tasks.workunit.client.0.smithi077.stdout: create 299/39 file after the snapshot 2022-04-03T19:47:22.793 INFO:tasks.workunit.client.0.smithi077.stdout:Take second snapshot .snap/test2 2022-04-03T19:47:22.819 INFO:tasks.workunit.client.0.smithi077.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2022-04-03T19:47:22.960 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.958+0000 7f536ffff700 1 Processor -- start 2022-04-03T19:47:22.961 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.959+0000 7f536ffff700 1 -- start start 2022-04-03T19:47:22.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.960+0000 7f536ffff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f53700ba3c0 0x7f5370156b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:22.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.960+0000 7f536ffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5370157060 0x7f537014cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:22.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.960+0000 7f536ffff700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53700c3c50 con 0x7f53700678b0 2022-04-03T19:47:22.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.960+0000 7f536ffff700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53700c8020 con 0x7f53700ba3c0 2022-04-03T19:47:22.966 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.960+0000 7f536ffff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5370156690 con 0x7f5370157060 2022-04-03T19:47:22.966 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.960+0000 7f536effd700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f53700ba3c0 0x7f5370156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:22.966 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.960+0000 7f536e7fc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5370157060 0x7f537014cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:22.966 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.960+0000 7f536e7fc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5370157060 0x7f537014cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60104/0 (socket says 172.21.15.77:60104) 2022-04-03T19:47:22.967 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.960+0000 7f536f7fe700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f53700678b0 0x7f537014d040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44200/0 (socket says 172.21.15.77:44200) 2022-04-03T19:47:22.967 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.960+0000 7f536e7fc700 1 -- 172.21.15.77:0/1761108941 learned_addr learned my addr 172.21.15.77:0/1761108941 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:47:22.967 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.960+0000 7f536effd700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f53700ba3c0 0x7f5370156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:37998/0 (socket says 172.21.15.77:37998) 2022-04-03T19:47:22.967 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.961+0000 7f536e7fc700 1 -- 172.21.15.77:0/1761108941 >> v2:172.21.15.186:3300/0 conn(0x7f53700ba3c0 msgr2=0x7f5370156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:47:22.968 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.961+0000 7f536e7fc700 1 --2- 172.21.15.77:0/1761108941 >> v2:172.21.15.186:3300/0 conn(0x7f53700ba3c0 0x7f5370156b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:22.968 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.961+0000 7f536e7fc700 1 -- 172.21.15.77:0/1761108941 >> v1:172.21.15.186:6789/0 conn(0x7f53700678b0 legacy=0x7f537014d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:22.968 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.961+0000 7f536e7fc700 1 -- 172.21.15.77:0/1761108941 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53700bf350 con 0x7f5370157060 2022-04-03T19:47:22.968 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.961+0000 7f536dffb700 1 -- 172.21.15.77:0/1761108941 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068123918 0 0) 0x7f53700c3c50 con 0x7f53700678b0 2022-04-03T19:47:22.968 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.961+0000 7f536effd700 1 --2- 172.21.15.77:0/1761108941 >> v2:172.21.15.186:3300/0 conn(0x7f53700ba3c0 0x7f5370156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:47:22.969 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.961+0000 7f536e7fc700 1 --2- 172.21.15.77:0/1761108941 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5370157060 0x7f537014cb00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5364006fd0 tx=0x7f53640090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8a1d39eae155a22 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:22.969 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.961+0000 7f536dffb700 1 -- 172.21.15.77:0/1761108941 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5364004030 con 0x7f5370157060 2022-04-03T19:47:22.969 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.961+0000 7f536dffb700 1 -- 172.21.15.77:0/1761108941 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53640041d0 con 0x7f5370157060 2022-04-03T19:47:22.969 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.961+0000 7f536dffb700 1 -- 172.21.15.77:0/1761108941 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5364011630 con 0x7f5370157060 2022-04-03T19:47:22.970 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.962+0000 7f536ffff700 1 -- 172.21.15.77:0/1761108941 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5370157060 msgr2=0x7f537014cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:22.970 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.962+0000 7f536ffff700 1 --2- 172.21.15.77:0/1761108941 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5370157060 0x7f537014cb00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5364006fd0 tx=0x7f53640090f0 comp rx=0 tx=0).stop 2022-04-03T19:47:22.970 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.962+0000 7f536ffff700 1 -- 172.21.15.77:0/1761108941 shutdown_connections 2022-04-03T19:47:22.970 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.962+0000 7f536ffff700 1 --2- 172.21.15.77:0/1761108941 >> v2:172.21.15.186:3300/0 conn(0x7f53700ba3c0 0x7f5370156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:22.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.962+0000 7f536ffff700 1 --2- 172.21.15.77:0/1761108941 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5370157060 0x7f537014cb00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:22.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.962+0000 7f536ffff700 1 -- 172.21.15.77:0/1761108941 >> 172.21.15.77:0/1761108941 conn(0x7f5370066da0 msgr2=0x7f53700671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:47:22.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.962+0000 7f536ffff700 1 -- 172.21.15.77:0/1761108941 shutdown_connections 2022-04-03T19:47:22.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.962+0000 7f536ffff700 1 -- 172.21.15.77:0/1761108941 wait complete. 2022-04-03T19:47:22.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.964+0000 7f536ffff700 1 Processor -- start 2022-04-03T19:47:22.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.964+0000 7f536ffff700 1 -- start start 2022-04-03T19:47:22.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.965+0000 7f536ffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53700678b0 0x7f53700c3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:22.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.965+0000 7f536ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53700ba3c0 0x7f5370164c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:22.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.965+0000 7f536ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5370157060 0x7f537014d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:22.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.965+0000 7f536ffff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537014e8b0 con 0x7f5370157060 2022-04-03T19:47:22.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.965+0000 7f536ffff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53701644f0 con 0x7f53700ba3c0 2022-04-03T19:47:22.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.965+0000 7f536ffff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53701647d0 con 0x7f53700678b0 2022-04-03T19:47:22.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.965+0000 7f536f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5370157060 0x7f537014d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:22.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.965+0000 7f536effd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53700678b0 0x7f53700c3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:22.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.965+0000 7f536e7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53700ba3c0 0x7f5370164c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:22.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.965+0000 7f536f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5370157060 0x7f537014d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42814/0 (socket says 172.21.15.77:42814) 2022-04-03T19:47:22.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.965+0000 7f536f7fe700 1 -- 172.21.15.77:0/1016189740 learned_addr learned my addr 172.21.15.77:0/1016189740 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:47:22.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.965+0000 7f536effd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53700678b0 0x7f53700c3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60106/0 (socket says 172.21.15.77:60106) 2022-04-03T19:47:22.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.965+0000 7f536f7fe700 1 -- 172.21.15.77:0/1016189740 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53700678b0 msgr2=0x7f53700c3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:22.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.965+0000 7f536f7fe700 1 --2- 172.21.15.77:0/1016189740 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53700678b0 0x7f53700c3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:22.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.965+0000 7f536f7fe700 1 -- 172.21.15.77:0/1016189740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53700ba3c0 msgr2=0x7f5370164c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:22.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.965+0000 7f536f7fe700 1 --2- 172.21.15.77:0/1016189740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53700ba3c0 0x7f5370164c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:22.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.965+0000 7f536f7fe700 1 -- 172.21.15.77:0/1016189740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536000d740 con 0x7f5370157060 2022-04-03T19:47:22.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.965+0000 7f536e7fc700 1 --2- 172.21.15.77:0/1016189740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53700ba3c0 0x7f5370164c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:47:22.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.966+0000 7f536f7fe700 1 --2- 172.21.15.77:0/1016189740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5370157060 0x7f537014d7d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f536800c7e0 tx=0x7f5368007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:22.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.966+0000 7f5356ffd700 1 -- 172.21.15.77:0/1016189740 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5368004c40 con 0x7f5370157060 2022-04-03T19:47:22.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.966+0000 7f5356ffd700 1 -- 172.21.15.77:0/1016189740 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5368004da0 con 0x7f5370157060 2022-04-03T19:47:22.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.966+0000 7f5356ffd700 1 -- 172.21.15.77:0/1016189740 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53680073a0 con 0x7f5370157060 2022-04-03T19:47:22.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.966+0000 7f536ffff700 1 -- 172.21.15.77:0/1016189740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5364009370 con 0x7f5370157060 2022-04-03T19:47:22.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.966+0000 7f536ffff700 1 -- 172.21.15.77:0/1016189740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f537014f360 con 0x7f5370157060 2022-04-03T19:47:22.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.967+0000 7f536ffff700 1 -- 172.21.15.77:0/1016189740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5370164910 con 0x7f5370157060 2022-04-03T19:47:22.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.968+0000 7f5356ffd700 1 -- 172.21.15.77:0/1016189740 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f536801a030 con 0x7f5370157060 2022-04-03T19:47:22.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.969+0000 7f5356ffd700 1 --2- 172.21.15.77:0/1016189740 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5350063ce0 0x7f5350066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:47:22.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.969+0000 7f5356ffd700 1 -- 172.21.15.77:0/1016189740 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f536807fc40 con 0x7f5370157060 2022-04-03T19:47:22.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.969+0000 7f536effd700 1 --2- 172.21.15.77:0/1016189740 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5350063ce0 0x7f5350066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:47:22.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.970+0000 7f536effd700 1 --2- 172.21.15.77:0/1016189740 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5350063ce0 0x7f5350066190 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5360015490 tx=0x7f536000f040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:47:22.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:22.973+0000 7f5356ffd700 1 -- 172.21.15.77:0/1016189740 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f536804e810 con 0x7f5370157060 2022-04-03T19:47:23.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:23.209+0000 7f536ffff700 1 -- 172.21.15.77:0/1016189740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f537014f360 con 0x7f5370157060 2022-04-03T19:47:23.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:23.210+0000 7f5356ffd700 1 -- 172.21.15.77:0/1016189740 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6112 (secure 0 0 0) 0x7f537014f360 con 0x7f5370157060 2022-04-03T19:47:23.212 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:47:23.212 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4960,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:47:10.107928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:active","state_seq":47,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:47:23.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:23.213+0000 7f536ffff700 1 -- 172.21.15.77:0/1016189740 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5350063ce0 msgr2=0x7f5350066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:23.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:23.213+0000 7f536ffff700 1 --2- 172.21.15.77:0/1016189740 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5350063ce0 0x7f5350066190 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5360015490 tx=0x7f536000f040 comp rx=0 tx=0).stop 2022-04-03T19:47:23.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:23.213+0000 7f536ffff700 1 -- 172.21.15.77:0/1016189740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5370157060 msgr2=0x7f537014d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:47:23.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:23.213+0000 7f536ffff700 1 --2- 172.21.15.77:0/1016189740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5370157060 0x7f537014d7d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f536800c7e0 tx=0x7f5368007f40 comp rx=0 tx=0).stop 2022-04-03T19:47:23.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:23.213+0000 7f536ffff700 1 -- 172.21.15.77:0/1016189740 shutdown_connections 2022-04-03T19:47:23.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:23.213+0000 7f536ffff700 1 --2- 172.21.15.77:0/1016189740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5370157060 0x7f537014d7d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:23.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:23.213+0000 7f536ffff700 1 --2- 172.21.15.77:0/1016189740 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5350063ce0 0x7f5350066190 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:23.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:23.213+0000 7f536ffff700 1 --2- 172.21.15.77:0/1016189740 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53700678b0 0x7f53700c3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:23.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:23.213+0000 7f536ffff700 1 --2- 172.21.15.77:0/1016189740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53700ba3c0 0x7f5370164c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:47:23.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:23.213+0000 7f536ffff700 1 -- 172.21.15.77:0/1016189740 >> 172.21.15.77:0/1016189740 conn(0x7f5370066da0 msgr2=0x7f537014c070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:47:23.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:23.213+0000 7f536ffff700 1 -- 172.21.15.77:0/1016189740 shutdown_connections 2022-04-03T19:47:23.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:47:23.213+0000 7f536ffff700 1 -- 172.21.15.77:0/1016189740 wait complete. 2022-04-03T19:47:23.218 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 14 2022-04-03T19:47:23.240 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2022-04-03T19:47:23.241 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 39 secs before thrashing 2022-04-03T19:47:23.785 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2022-04-03T19:47:23.785 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2022-04-03T19:47:23.786 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2022-04-03T19:47:23.786 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2022-04-03T19:47:23.786 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2022-04-03T19:47:23.786 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2022-04-03T19:47:23.787 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2022-04-03T19:47:23.787 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2022-04-03T19:47:23.787 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2022-04-03T19:47:23.787 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2022-04-03T19:47:23.787 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2022-04-03T19:47:23.788 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2022-04-03T19:47:23.788 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2022-04-03T19:47:23.788 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2022-04-03T19:47:23.788 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2022-04-03T19:47:23.789 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2022-04-03T19:47:23.789 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2022-04-03T19:47:23.789 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2022-04-03T19:47:23.789 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2022-04-03T19:47:23.790 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2022-04-03T19:47:23.790 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2022-04-03T19:47:23.790 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2022-04-03T19:47:23.790 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2022-04-03T19:47:23.791 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2022-04-03T19:47:23.791 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2022-04-03T19:47:23.791 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2022-04-03T19:47:23.791 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2022-04-03T19:47:23.791 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2022-04-03T19:47:23.792 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2022-04-03T19:47:23.792 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2022-04-03T19:47:23.792 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2022-04-03T19:47:23.792 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2022-04-03T19:47:23.793 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2022-04-03T19:47:23.793 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2022-04-03T19:47:23.793 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2022-04-03T19:47:23.793 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2022-04-03T19:47:23.793 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2022-04-03T19:47:23.794 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2022-04-03T19:47:23.794 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2022-04-03T19:47:23.794 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2022-04-03T19:47:23.794 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2022-04-03T19:47:23.795 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2022-04-03T19:47:23.795 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2022-04-03T19:47:23.795 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2022-04-03T19:47:23.795 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2022-04-03T19:47:23.796 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2022-04-03T19:47:23.796 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2022-04-03T19:47:23.796 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2022-04-03T19:47:23.796 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2022-04-03T19:47:23.796 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2022-04-03T19:47:23.797 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2022-04-03T19:47:23.797 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2022-04-03T19:47:23.797 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2022-04-03T19:47:23.797 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2022-04-03T19:47:23.798 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2022-04-03T19:47:23.798 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2022-04-03T19:47:23.798 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2022-04-03T19:47:23.798 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2022-04-03T19:47:23.799 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2022-04-03T19:47:23.799 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2022-04-03T19:47:23.799 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2022-04-03T19:47:23.799 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2022-04-03T19:47:23.800 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2022-04-03T19:47:23.800 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2022-04-03T19:47:23.800 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2022-04-03T19:47:23.800 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2022-04-03T19:47:23.800 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2022-04-03T19:47:23.801 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2022-04-03T19:47:23.801 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2022-04-03T19:47:23.801 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2022-04-03T19:47:23.801 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2022-04-03T19:47:23.802 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2022-04-03T19:47:23.802 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2022-04-03T19:47:23.802 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2022-04-03T19:47:23.802 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2022-04-03T19:47:23.802 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2022-04-03T19:47:23.803 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2022-04-03T19:47:23.803 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2022-04-03T19:47:23.803 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2022-04-03T19:47:23.803 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2022-04-03T19:47:23.804 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2022-04-03T19:47:24.846 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2022-04-03T19:47:24.847 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2022-04-03T19:47:24.847 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2022-04-03T19:47:24.847 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2022-04-03T19:47:24.848 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2022-04-03T19:47:24.848 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2022-04-03T19:47:24.848 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2022-04-03T19:47:24.848 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2022-04-03T19:47:24.849 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2022-04-03T19:47:24.849 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2022-04-03T19:47:24.849 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2022-04-03T19:47:24.849 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2022-04-03T19:47:24.849 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2022-04-03T19:47:24.850 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2022-04-03T19:47:24.850 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2022-04-03T19:47:24.850 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2022-04-03T19:47:24.850 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2022-04-03T19:47:24.851 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2022-04-03T19:47:24.851 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2022-04-03T19:47:24.851 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2022-04-03T19:47:24.851 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2022-04-03T19:47:24.851 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2022-04-03T19:47:24.852 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2022-04-03T19:47:24.852 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2022-04-03T19:47:24.852 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2022-04-03T19:47:24.852 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2022-04-03T19:47:24.853 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2022-04-03T19:47:24.853 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2022-04-03T19:47:24.853 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2022-04-03T19:47:24.853 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2022-04-03T19:47:24.853 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2022-04-03T19:47:24.854 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2022-04-03T19:47:24.854 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2022-04-03T19:47:24.854 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2022-04-03T19:47:24.854 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2022-04-03T19:47:24.855 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2022-04-03T19:47:24.855 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2022-04-03T19:47:24.855 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2022-04-03T19:47:24.855 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2022-04-03T19:47:24.855 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2022-04-03T19:47:24.856 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2022-04-03T19:47:24.856 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2022-04-03T19:47:24.856 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2022-04-03T19:47:24.856 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2022-04-03T19:47:24.857 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2022-04-03T19:47:24.857 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2022-04-03T19:47:24.857 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2022-04-03T19:47:24.857 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2022-04-03T19:47:24.857 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2022-04-03T19:47:24.858 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2022-04-03T19:47:24.858 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2022-04-03T19:47:24.858 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2022-04-03T19:47:24.858 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2022-04-03T19:47:24.859 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2022-04-03T19:47:24.859 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2022-04-03T19:47:24.859 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2022-04-03T19:47:24.859 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2022-04-03T19:47:24.859 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2022-04-03T19:47:24.860 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2022-04-03T19:47:24.860 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2022-04-03T19:47:24.860 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2022-04-03T19:47:24.860 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2022-04-03T19:47:24.860 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2022-04-03T19:47:24.861 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2022-04-03T19:47:24.861 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2022-04-03T19:47:24.861 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2022-04-03T19:47:24.861 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2022-04-03T19:47:24.862 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2022-04-03T19:47:24.862 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2022-04-03T19:47:24.862 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2022-04-03T19:47:24.862 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2022-04-03T19:47:24.862 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2022-04-03T19:47:24.863 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2022-04-03T19:47:24.863 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2022-04-03T19:47:24.863 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2022-04-03T19:47:24.863 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2022-04-03T19:47:24.863 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2022-04-03T19:47:24.864 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2022-04-03T19:47:24.864 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2022-04-03T19:47:24.864 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2022-04-03T19:47:24.864 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2022-04-03T19:47:25.851 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2022-04-03T19:47:25.851 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2022-04-03T19:47:25.851 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2022-04-03T19:47:25.852 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2022-04-03T19:47:25.852 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2022-04-03T19:47:25.852 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2022-04-03T19:47:25.852 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2022-04-03T19:47:25.853 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2022-04-03T19:47:25.853 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2022-04-03T19:47:25.853 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2022-04-03T19:47:25.853 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2022-04-03T19:47:25.853 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2022-04-03T19:47:25.854 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2022-04-03T19:47:25.854 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2022-04-03T19:47:25.854 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2022-04-03T19:47:25.854 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2022-04-03T19:47:25.855 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2022-04-03T19:47:25.855 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2022-04-03T19:47:25.855 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2022-04-03T19:47:25.855 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2022-04-03T19:47:25.855 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2022-04-03T19:47:25.856 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2022-04-03T19:47:25.856 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2022-04-03T19:47:25.856 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2022-04-03T19:47:25.857 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2022-04-03T19:47:25.857 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2022-04-03T19:47:25.857 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2022-04-03T19:47:25.857 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2022-04-03T19:47:25.857 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2022-04-03T19:47:25.858 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2022-04-03T19:47:25.858 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2022-04-03T19:47:25.858 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2022-04-03T19:47:25.858 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2022-04-03T19:47:25.859 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2022-04-03T19:47:25.859 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2022-04-03T19:47:25.859 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2022-04-03T19:47:25.859 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2022-04-03T19:47:25.859 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2022-04-03T19:47:25.860 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2022-04-03T19:47:25.860 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2022-04-03T19:47:25.860 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2022-04-03T19:47:25.860 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2022-04-03T19:47:25.861 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2022-04-03T19:47:25.861 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2022-04-03T19:47:25.861 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2022-04-03T19:47:25.861 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2022-04-03T19:47:25.861 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2022-04-03T19:47:25.862 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2022-04-03T19:47:25.862 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2022-04-03T19:47:25.862 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2022-04-03T19:47:25.862 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2022-04-03T19:47:25.863 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2022-04-03T19:47:25.863 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2022-04-03T19:47:25.863 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2022-04-03T19:47:25.863 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2022-04-03T19:47:25.863 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2022-04-03T19:47:25.864 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2022-04-03T19:47:25.864 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2022-04-03T19:47:25.864 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2022-04-03T19:47:25.864 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2022-04-03T19:47:25.865 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2022-04-03T19:47:25.865 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2022-04-03T19:47:25.865 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2022-04-03T19:47:25.865 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2022-04-03T19:47:25.865 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2022-04-03T19:47:25.866 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2022-04-03T19:47:25.866 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2022-04-03T19:47:25.866 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2022-04-03T19:47:25.866 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2022-04-03T19:47:25.867 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2022-04-03T19:47:25.867 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2022-04-03T19:47:25.867 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2022-04-03T19:47:25.867 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2022-04-03T19:47:25.868 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2022-04-03T19:47:25.868 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2022-04-03T19:47:25.868 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2022-04-03T19:47:25.868 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2022-04-03T19:47:25.868 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2022-04-03T19:47:25.869 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2022-04-03T19:47:25.869 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2022-04-03T19:47:25.869 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2022-04-03T19:47:26.276 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2022-04-03T19:47:26.276 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2022-04-03T19:47:26.276 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2022-04-03T19:47:26.277 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2022-04-03T19:47:26.277 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2022-04-03T19:47:26.277 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2022-04-03T19:47:26.277 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2022-04-03T19:47:26.277 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2022-04-03T19:47:26.278 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2022-04-03T19:47:26.278 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2022-04-03T19:47:26.278 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2022-04-03T19:47:26.278 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2022-04-03T19:47:26.279 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2022-04-03T19:47:26.279 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2022-04-03T19:47:26.279 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2022-04-03T19:47:26.279 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2022-04-03T19:47:26.279 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2022-04-03T19:47:26.280 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2022-04-03T19:47:26.280 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2022-04-03T19:47:26.280 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2022-04-03T19:47:26.280 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2022-04-03T19:47:26.280 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2022-04-03T19:47:26.281 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2022-04-03T19:47:26.281 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2022-04-03T19:47:26.281 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2022-04-03T19:47:26.281 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2022-04-03T19:47:26.282 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2022-04-03T19:47:26.282 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2022-04-03T19:47:26.282 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2022-04-03T19:47:26.282 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2022-04-03T19:47:26.283 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2022-04-03T19:47:26.283 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2022-04-03T19:47:26.283 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2022-04-03T19:47:26.283 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2022-04-03T19:47:26.283 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2022-04-03T19:47:26.284 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2022-04-03T19:47:26.284 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2022-04-03T19:47:26.284 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2022-04-03T19:47:26.284 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2022-04-03T19:47:26.285 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2022-04-03T19:47:26.285 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2022-04-03T19:47:26.285 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2022-04-03T19:47:26.285 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2022-04-03T19:47:26.286 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2022-04-03T19:47:26.286 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2022-04-03T19:47:26.286 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2022-04-03T19:47:26.286 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2022-04-03T19:47:26.286 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2022-04-03T19:47:26.287 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2022-04-03T19:47:26.287 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2022-04-03T19:47:26.287 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2022-04-03T19:47:26.287 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2022-04-03T19:47:26.288 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2022-04-03T19:47:26.288 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2022-04-03T19:47:26.288 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2022-04-03T19:47:26.288 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2022-04-03T19:47:26.288 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2022-04-03T19:47:26.289 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2022-04-03T19:47:26.289 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2022-04-03T19:47:26.289 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2022-04-03T19:47:26.289 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2022-04-03T19:47:26.289 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2022-04-03T19:47:26.290 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2022-04-03T19:47:26.290 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2022-04-03T19:47:26.290 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2022-04-03T19:47:26.290 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2022-04-03T19:47:26.291 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2022-04-03T19:47:26.291 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2022-04-03T19:47:26.291 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2022-04-03T19:47:26.291 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2022-04-03T19:47:26.291 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2022-04-03T19:47:26.292 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2022-04-03T19:47:26.292 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2022-04-03T19:47:26.292 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2022-04-03T19:47:26.292 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2022-04-03T19:47:26.292 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2022-04-03T19:47:26.293 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2022-04-03T19:47:26.293 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2022-04-03T19:47:26.293 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2022-04-03T19:47:26.293 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2022-04-03T19:47:26.293 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2022-04-03T19:47:26.976 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2022-04-03T19:47:26.977 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2022-04-03T19:47:26.977 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2022-04-03T19:47:26.977 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2022-04-03T19:47:26.977 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2022-04-03T19:47:26.977 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2022-04-03T19:47:26.978 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2022-04-03T19:47:26.978 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2022-04-03T19:47:26.978 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2022-04-03T19:47:26.978 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2022-04-03T19:47:26.979 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2022-04-03T19:47:26.979 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2022-04-03T19:47:26.979 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2022-04-03T19:47:26.979 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2022-04-03T19:47:26.979 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2022-04-03T19:47:26.980 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2022-04-03T19:47:26.980 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2022-04-03T19:47:26.980 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2022-04-03T19:47:26.980 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2022-04-03T19:47:26.980 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2022-04-03T19:47:26.981 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2022-04-03T19:47:26.981 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2022-04-03T19:47:26.981 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2022-04-03T19:47:26.981 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2022-04-03T19:47:26.981 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2022-04-03T19:47:26.982 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2022-04-03T19:47:26.982 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2022-04-03T19:47:26.982 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2022-04-03T19:47:26.982 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2022-04-03T19:47:26.983 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2022-04-03T19:47:26.983 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2022-04-03T19:47:26.983 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2022-04-03T19:47:26.983 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2022-04-03T19:47:26.983 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2022-04-03T19:47:26.984 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2022-04-03T19:47:26.984 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2022-04-03T19:47:26.984 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2022-04-03T19:47:26.984 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2022-04-03T19:47:26.985 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2022-04-03T19:47:26.985 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2022-04-03T19:47:26.985 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2022-04-03T19:47:26.985 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2022-04-03T19:47:26.985 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2022-04-03T19:47:26.986 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2022-04-03T19:47:26.986 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2022-04-03T19:47:26.986 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2022-04-03T19:47:26.986 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2022-04-03T19:47:26.987 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2022-04-03T19:47:26.987 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2022-04-03T19:47:26.987 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2022-04-03T19:47:26.987 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2022-04-03T19:47:26.987 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2022-04-03T19:47:26.988 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2022-04-03T19:47:26.988 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2022-04-03T19:47:26.988 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2022-04-03T19:47:26.988 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2022-04-03T19:47:26.989 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2022-04-03T19:47:26.989 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2022-04-03T19:47:26.989 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2022-04-03T19:47:26.989 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2022-04-03T19:47:26.989 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2022-04-03T19:47:26.990 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2022-04-03T19:47:26.990 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2022-04-03T19:47:26.991 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2022-04-03T19:47:26.991 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2022-04-03T19:47:26.991 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2022-04-03T19:47:26.991 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2022-04-03T19:47:26.991 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2022-04-03T19:47:26.992 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2022-04-03T19:47:26.992 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2022-04-03T19:47:26.992 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2022-04-03T19:47:26.992 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2022-04-03T19:47:26.993 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2022-04-03T19:47:26.993 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2022-04-03T19:47:26.993 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2022-04-03T19:47:26.993 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2022-04-03T19:47:26.993 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2022-04-03T19:47:26.994 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2022-04-03T19:47:26.994 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2022-04-03T19:47:26.994 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2022-04-03T19:47:26.994 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2022-04-03T19:47:29.216 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2022-04-03T19:47:29.217 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2022-04-03T19:47:29.217 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2022-04-03T19:47:29.217 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2022-04-03T19:47:29.217 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2022-04-03T19:47:29.218 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2022-04-03T19:47:29.218 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2022-04-03T19:47:29.218 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2022-04-03T19:47:29.218 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2022-04-03T19:47:29.219 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2022-04-03T19:47:29.219 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2022-04-03T19:47:29.219 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2022-04-03T19:47:29.219 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2022-04-03T19:47:29.220 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2022-04-03T19:47:29.220 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2022-04-03T19:47:29.220 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2022-04-03T19:47:29.220 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2022-04-03T19:47:29.221 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2022-04-03T19:47:29.221 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2022-04-03T19:47:29.221 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2022-04-03T19:47:29.221 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2022-04-03T19:47:29.222 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2022-04-03T19:47:29.222 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2022-04-03T19:47:29.222 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2022-04-03T19:47:29.222 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2022-04-03T19:47:29.223 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2022-04-03T19:47:29.223 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2022-04-03T19:47:29.223 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2022-04-03T19:47:29.223 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2022-04-03T19:47:29.224 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2022-04-03T19:47:29.224 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2022-04-03T19:47:29.224 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2022-04-03T19:47:29.224 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2022-04-03T19:47:29.225 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2022-04-03T19:47:29.225 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2022-04-03T19:47:29.225 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2022-04-03T19:47:29.225 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2022-04-03T19:47:29.226 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2022-04-03T19:47:29.226 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2022-04-03T19:47:29.226 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2022-04-03T19:47:29.226 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2022-04-03T19:47:29.226 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2022-04-03T19:47:29.227 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2022-04-03T19:47:29.227 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2022-04-03T19:47:29.227 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2022-04-03T19:47:29.227 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2022-04-03T19:47:29.228 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2022-04-03T19:47:29.228 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2022-04-03T19:47:29.228 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2022-04-03T19:47:29.228 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2022-04-03T19:47:29.228 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2022-04-03T19:47:29.229 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2022-04-03T19:47:29.229 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2022-04-03T19:47:29.229 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2022-04-03T19:47:29.229 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2022-04-03T19:47:29.230 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2022-04-03T19:47:29.230 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2022-04-03T19:47:29.230 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2022-04-03T19:47:29.230 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2022-04-03T19:47:29.231 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2022-04-03T19:47:29.231 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2022-04-03T19:47:29.231 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2022-04-03T19:47:29.231 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2022-04-03T19:47:29.231 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2022-04-03T19:47:29.232 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2022-04-03T19:47:29.232 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2022-04-03T19:47:29.232 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2022-04-03T19:47:29.232 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2022-04-03T19:47:29.232 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2022-04-03T19:47:29.233 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2022-04-03T19:47:29.233 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2022-04-03T19:47:29.233 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2022-04-03T19:47:29.233 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2022-04-03T19:47:29.233 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2022-04-03T19:47:29.234 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2022-04-03T19:47:29.234 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2022-04-03T19:47:29.234 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2022-04-03T19:47:29.234 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2022-04-03T19:47:29.234 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2022-04-03T19:47:29.235 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2022-04-03T19:47:29.235 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2022-04-03T19:47:32.120 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2022-04-03T19:47:32.121 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2022-04-03T19:47:32.121 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2022-04-03T19:47:32.121 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2022-04-03T19:47:32.121 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2022-04-03T19:47:32.122 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2022-04-03T19:47:32.122 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2022-04-03T19:47:32.122 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2022-04-03T19:47:32.122 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2022-04-03T19:47:32.122 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2022-04-03T19:47:32.123 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2022-04-03T19:47:32.123 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2022-04-03T19:47:32.123 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2022-04-03T19:47:32.123 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2022-04-03T19:47:32.124 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2022-04-03T19:47:32.124 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2022-04-03T19:47:32.124 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2022-04-03T19:47:32.124 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2022-04-03T19:47:32.124 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2022-04-03T19:47:32.125 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2022-04-03T19:47:32.125 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2022-04-03T19:47:32.125 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2022-04-03T19:47:32.125 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2022-04-03T19:47:32.126 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2022-04-03T19:47:32.126 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2022-04-03T19:47:32.126 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2022-04-03T19:47:32.126 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2022-04-03T19:47:32.126 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2022-04-03T19:47:32.127 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2022-04-03T19:47:32.127 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2022-04-03T19:47:32.127 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2022-04-03T19:47:32.127 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2022-04-03T19:47:32.128 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2022-04-03T19:47:32.128 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2022-04-03T19:47:32.128 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2022-04-03T19:47:32.128 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2022-04-03T19:47:32.128 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2022-04-03T19:47:32.129 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2022-04-03T19:47:32.129 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2022-04-03T19:47:32.129 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2022-04-03T19:47:32.129 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2022-04-03T19:47:32.130 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2022-04-03T19:47:32.130 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2022-04-03T19:47:32.130 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2022-04-03T19:47:32.130 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2022-04-03T19:47:32.130 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2022-04-03T19:47:32.131 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2022-04-03T19:47:32.131 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2022-04-03T19:47:32.131 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2022-04-03T19:47:32.131 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2022-04-03T19:47:32.132 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2022-04-03T19:47:32.132 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2022-04-03T19:47:32.132 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2022-04-03T19:47:32.132 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2022-04-03T19:47:32.132 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2022-04-03T19:47:32.133 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2022-04-03T19:47:32.133 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2022-04-03T19:47:32.133 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2022-04-03T19:47:32.133 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2022-04-03T19:47:32.133 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2022-04-03T19:47:32.134 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2022-04-03T19:47:32.134 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2022-04-03T19:47:32.134 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2022-04-03T19:47:32.134 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2022-04-03T19:47:32.135 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2022-04-03T19:47:32.135 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2022-04-03T19:47:32.135 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2022-04-03T19:47:32.135 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2022-04-03T19:47:32.136 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2022-04-03T19:47:32.136 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2022-04-03T19:47:32.136 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2022-04-03T19:47:32.136 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2022-04-03T19:47:32.136 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2022-04-03T19:47:32.137 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2022-04-03T19:47:32.137 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2022-04-03T19:47:32.137 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2022-04-03T19:47:32.137 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2022-04-03T19:47:32.137 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2022-04-03T19:47:32.138 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2022-04-03T19:47:32.138 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2022-04-03T19:47:32.138 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2022-04-03T19:47:32.138 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2022-04-03T19:47:33.383 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2022-04-03T19:47:33.383 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2022-04-03T19:47:33.384 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2022-04-03T19:47:33.384 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2022-04-03T19:47:33.384 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2022-04-03T19:47:33.384 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2022-04-03T19:47:33.384 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2022-04-03T19:47:33.385 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2022-04-03T19:47:33.385 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2022-04-03T19:47:33.385 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2022-04-03T19:47:33.385 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2022-04-03T19:47:33.386 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2022-04-03T19:47:33.386 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2022-04-03T19:47:33.386 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2022-04-03T19:47:33.386 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2022-04-03T19:47:33.386 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2022-04-03T19:47:33.387 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2022-04-03T19:47:33.387 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2022-04-03T19:47:33.387 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2022-04-03T19:47:33.387 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2022-04-03T19:47:33.388 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2022-04-03T19:47:33.388 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2022-04-03T19:47:33.388 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2022-04-03T19:47:33.388 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2022-04-03T19:47:33.388 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2022-04-03T19:47:33.389 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2022-04-03T19:47:33.389 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2022-04-03T19:47:33.389 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2022-04-03T19:47:33.389 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2022-04-03T19:47:33.390 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2022-04-03T19:47:33.390 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2022-04-03T19:47:33.390 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2022-04-03T19:47:33.390 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2022-04-03T19:47:33.390 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2022-04-03T19:47:33.391 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2022-04-03T19:47:33.391 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2022-04-03T19:47:33.391 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2022-04-03T19:47:33.391 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2022-04-03T19:47:33.391 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2022-04-03T19:47:33.392 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2022-04-03T19:47:33.392 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2022-04-03T19:47:33.392 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2022-04-03T19:47:33.392 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2022-04-03T19:47:33.393 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2022-04-03T19:47:33.393 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2022-04-03T19:47:33.393 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2022-04-03T19:47:33.393 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2022-04-03T19:47:33.393 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2022-04-03T19:47:33.394 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2022-04-03T19:47:33.394 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2022-04-03T19:47:33.394 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2022-04-03T19:47:33.394 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2022-04-03T19:47:33.395 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2022-04-03T19:47:33.395 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2022-04-03T19:47:33.395 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2022-04-03T19:47:33.395 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2022-04-03T19:47:33.395 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2022-04-03T19:47:33.396 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2022-04-03T19:47:33.396 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2022-04-03T19:47:33.396 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2022-04-03T19:47:33.396 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2022-04-03T19:47:33.396 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2022-04-03T19:47:33.397 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2022-04-03T19:47:33.397 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2022-04-03T19:47:33.397 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2022-04-03T19:47:33.397 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2022-04-03T19:47:33.398 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2022-04-03T19:47:33.398 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2022-04-03T19:47:33.398 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2022-04-03T19:47:33.399 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2022-04-03T19:47:33.399 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2022-04-03T19:47:33.399 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2022-04-03T19:47:33.399 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2022-04-03T19:47:33.399 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2022-04-03T19:47:33.400 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2022-04-03T19:47:33.400 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2022-04-03T19:47:33.400 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2022-04-03T19:47:33.400 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2022-04-03T19:47:33.401 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2022-04-03T19:47:33.401 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2022-04-03T19:47:33.401 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2022-04-03T19:47:34.138 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2022-04-03T19:47:34.139 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2022-04-03T19:47:34.139 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2022-04-03T19:47:34.139 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2022-04-03T19:47:34.139 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2022-04-03T19:47:34.140 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2022-04-03T19:47:34.140 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2022-04-03T19:47:34.140 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2022-04-03T19:47:34.140 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2022-04-03T19:47:34.141 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2022-04-03T19:47:34.141 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2022-04-03T19:47:34.141 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2022-04-03T19:47:34.141 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2022-04-03T19:47:34.142 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2022-04-03T19:47:34.142 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2022-04-03T19:47:34.142 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2022-04-03T19:47:34.142 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2022-04-03T19:47:34.142 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2022-04-03T19:47:34.143 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2022-04-03T19:47:34.143 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2022-04-03T19:47:34.143 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2022-04-03T19:47:34.143 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2022-04-03T19:47:34.143 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2022-04-03T19:47:34.144 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2022-04-03T19:47:34.144 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2022-04-03T19:47:34.144 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2022-04-03T19:47:34.144 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2022-04-03T19:47:34.145 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2022-04-03T19:47:34.145 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2022-04-03T19:47:34.145 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2022-04-03T19:47:34.145 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2022-04-03T19:47:34.145 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2022-04-03T19:47:34.146 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2022-04-03T19:47:34.146 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2022-04-03T19:47:34.146 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2022-04-03T19:47:34.146 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2022-04-03T19:47:34.146 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2022-04-03T19:47:34.147 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2022-04-03T19:47:34.147 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2022-04-03T19:47:34.147 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2022-04-03T19:47:34.147 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2022-04-03T19:47:34.147 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2022-04-03T19:47:34.148 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2022-04-03T19:47:34.148 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2022-04-03T19:47:34.148 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2022-04-03T19:47:34.148 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2022-04-03T19:47:34.149 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2022-04-03T19:47:34.149 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2022-04-03T19:47:34.149 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2022-04-03T19:47:34.149 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2022-04-03T19:47:34.149 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2022-04-03T19:47:34.150 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2022-04-03T19:47:34.150 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2022-04-03T19:47:34.150 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2022-04-03T19:47:34.150 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2022-04-03T19:47:34.150 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2022-04-03T19:47:34.151 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2022-04-03T19:47:34.151 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2022-04-03T19:47:34.151 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2022-04-03T19:47:34.151 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2022-04-03T19:47:34.152 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2022-04-03T19:47:34.152 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2022-04-03T19:47:34.152 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2022-04-03T19:47:34.152 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2022-04-03T19:47:34.152 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2022-04-03T19:47:34.153 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2022-04-03T19:47:34.153 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2022-04-03T19:47:34.153 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2022-04-03T19:47:34.153 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2022-04-03T19:47:34.154 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2022-04-03T19:47:34.154 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2022-04-03T19:47:34.154 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2022-04-03T19:47:34.154 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2022-04-03T19:47:34.154 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2022-04-03T19:47:34.155 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2022-04-03T19:47:34.155 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2022-04-03T19:47:34.160 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2022-04-03T19:47:34.160 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2022-04-03T19:47:34.161 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2022-04-03T19:47:34.161 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2022-04-03T19:47:34.161 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2022-04-03T19:47:34.576 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2022-04-03T19:47:34.576 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2022-04-03T19:47:34.576 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2022-04-03T19:47:34.577 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2022-04-03T19:47:34.577 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2022-04-03T19:47:34.577 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2022-04-03T19:47:34.577 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2022-04-03T19:47:34.578 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2022-04-03T19:47:34.578 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2022-04-03T19:47:34.578 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2022-04-03T19:47:34.578 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2022-04-03T19:47:34.578 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2022-04-03T19:47:34.579 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2022-04-03T19:47:34.579 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2022-04-03T19:47:34.579 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2022-04-03T19:47:34.579 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2022-04-03T19:47:34.580 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2022-04-03T19:47:34.580 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2022-04-03T19:47:34.580 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2022-04-03T19:47:34.580 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2022-04-03T19:47:34.580 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2022-04-03T19:47:34.581 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2022-04-03T19:47:34.581 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2022-04-03T19:47:34.581 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2022-04-03T19:47:34.581 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2022-04-03T19:47:34.581 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2022-04-03T19:47:34.582 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2022-04-03T19:47:34.582 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2022-04-03T19:47:34.582 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2022-04-03T19:47:34.582 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2022-04-03T19:47:34.582 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2022-04-03T19:47:34.583 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2022-04-03T19:47:34.583 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2022-04-03T19:47:34.583 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2022-04-03T19:47:34.583 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2022-04-03T19:47:34.584 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2022-04-03T19:47:34.584 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2022-04-03T19:47:34.584 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2022-04-03T19:47:34.584 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2022-04-03T19:47:34.584 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2022-04-03T19:47:34.585 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2022-04-03T19:47:34.585 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2022-04-03T19:47:34.585 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2022-04-03T19:47:34.585 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2022-04-03T19:47:34.585 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2022-04-03T19:47:34.586 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2022-04-03T19:47:34.586 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2022-04-03T19:47:34.586 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2022-04-03T19:47:34.586 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2022-04-03T19:47:34.587 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2022-04-03T19:47:34.587 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2022-04-03T19:47:34.587 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2022-04-03T19:47:34.587 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2022-04-03T19:47:34.587 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2022-04-03T19:47:34.588 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2022-04-03T19:47:34.588 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2022-04-03T19:47:34.588 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2022-04-03T19:47:34.588 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2022-04-03T19:47:34.589 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2022-04-03T19:47:34.589 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2022-04-03T19:47:34.589 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2022-04-03T19:47:34.589 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2022-04-03T19:47:34.589 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2022-04-03T19:47:34.590 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2022-04-03T19:47:34.590 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2022-04-03T19:47:34.590 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2022-04-03T19:47:34.591 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2022-04-03T19:47:34.591 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2022-04-03T19:47:34.591 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2022-04-03T19:47:34.591 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2022-04-03T19:47:34.592 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2022-04-03T19:47:34.592 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2022-04-03T19:47:34.592 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2022-04-03T19:47:34.592 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2022-04-03T19:47:34.592 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2022-04-03T19:47:34.593 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2022-04-03T19:47:34.593 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2022-04-03T19:47:34.593 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2022-04-03T19:47:34.593 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2022-04-03T19:47:34.593 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2022-04-03T19:47:34.594 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2022-04-03T19:47:35.193 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2022-04-03T19:47:35.194 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2022-04-03T19:47:35.194 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2022-04-03T19:47:35.194 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2022-04-03T19:47:35.195 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2022-04-03T19:47:35.195 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2022-04-03T19:47:35.195 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2022-04-03T19:47:35.195 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2022-04-03T19:47:35.195 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2022-04-03T19:47:35.196 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2022-04-03T19:47:35.196 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2022-04-03T19:47:35.196 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2022-04-03T19:47:35.196 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2022-04-03T19:47:35.197 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2022-04-03T19:47:35.197 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2022-04-03T19:47:35.197 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2022-04-03T19:47:35.197 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2022-04-03T19:47:35.197 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2022-04-03T19:47:35.198 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2022-04-03T19:47:35.198 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2022-04-03T19:47:35.198 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2022-04-03T19:47:35.198 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2022-04-03T19:47:35.199 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2022-04-03T19:47:35.199 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2022-04-03T19:47:35.199 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2022-04-03T19:47:35.199 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2022-04-03T19:47:35.199 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2022-04-03T19:47:35.200 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2022-04-03T19:47:35.200 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2022-04-03T19:47:35.200 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2022-04-03T19:47:35.200 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2022-04-03T19:47:35.200 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2022-04-03T19:47:35.201 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2022-04-03T19:47:35.201 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2022-04-03T19:47:35.201 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2022-04-03T19:47:35.201 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2022-04-03T19:47:35.201 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2022-04-03T19:47:35.202 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2022-04-03T19:47:35.202 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2022-04-03T19:47:35.202 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2022-04-03T19:47:35.202 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2022-04-03T19:47:35.203 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2022-04-03T19:47:35.203 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2022-04-03T19:47:35.203 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2022-04-03T19:47:35.203 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2022-04-03T19:47:35.203 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2022-04-03T19:47:35.204 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2022-04-03T19:47:35.204 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2022-04-03T19:47:35.204 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2022-04-03T19:47:35.204 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2022-04-03T19:47:35.205 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2022-04-03T19:47:35.205 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2022-04-03T19:47:35.205 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2022-04-03T19:47:35.205 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2022-04-03T19:47:35.205 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2022-04-03T19:47:35.206 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2022-04-03T19:47:35.206 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2022-04-03T19:47:35.206 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2022-04-03T19:47:35.206 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2022-04-03T19:47:35.206 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2022-04-03T19:47:35.207 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2022-04-03T19:47:35.207 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2022-04-03T19:47:35.207 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2022-04-03T19:47:35.207 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2022-04-03T19:47:35.207 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2022-04-03T19:47:35.208 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2022-04-03T19:47:35.208 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2022-04-03T19:47:35.208 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2022-04-03T19:47:35.208 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2022-04-03T19:47:35.209 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2022-04-03T19:47:35.209 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2022-04-03T19:47:35.209 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2022-04-03T19:47:35.209 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2022-04-03T19:47:35.209 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2022-04-03T19:47:35.210 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2022-04-03T19:47:35.210 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2022-04-03T19:47:35.210 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2022-04-03T19:47:35.210 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2022-04-03T19:47:35.211 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2022-04-03T19:47:35.211 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2022-04-03T19:47:35.211 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2022-04-03T19:47:35.629 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2022-04-03T19:47:35.629 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2022-04-03T19:47:35.630 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2022-04-03T19:47:35.630 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2022-04-03T19:47:35.630 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2022-04-03T19:47:35.630 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2022-04-03T19:47:35.630 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2022-04-03T19:47:35.631 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2022-04-03T19:47:35.631 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2022-04-03T19:47:35.631 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2022-04-03T19:47:35.631 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2022-04-03T19:47:35.632 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2022-04-03T19:47:35.632 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2022-04-03T19:47:35.632 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2022-04-03T19:47:35.632 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2022-04-03T19:47:35.632 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2022-04-03T19:47:35.633 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2022-04-03T19:47:35.633 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2022-04-03T19:47:35.633 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2022-04-03T19:47:35.633 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2022-04-03T19:47:35.633 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2022-04-03T19:47:35.634 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2022-04-03T19:47:35.634 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2022-04-03T19:47:35.634 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2022-04-03T19:47:35.634 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2022-04-03T19:47:35.635 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2022-04-03T19:47:35.635 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2022-04-03T19:47:35.635 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2022-04-03T19:47:35.635 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2022-04-03T19:47:35.635 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2022-04-03T19:47:35.636 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2022-04-03T19:47:35.636 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2022-04-03T19:47:35.636 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2022-04-03T19:47:35.636 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2022-04-03T19:47:35.636 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2022-04-03T19:47:35.637 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2022-04-03T19:47:35.637 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2022-04-03T19:47:35.637 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2022-04-03T19:47:35.637 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2022-04-03T19:47:35.637 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2022-04-03T19:47:35.638 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2022-04-03T19:47:35.638 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2022-04-03T19:47:35.638 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2022-04-03T19:47:35.638 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2022-04-03T19:47:35.638 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2022-04-03T19:47:35.639 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2022-04-03T19:47:35.639 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2022-04-03T19:47:35.639 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2022-04-03T19:47:35.639 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2022-04-03T19:47:35.639 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2022-04-03T19:47:35.640 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2022-04-03T19:47:35.640 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2022-04-03T19:47:35.640 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2022-04-03T19:47:35.640 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2022-04-03T19:47:35.641 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2022-04-03T19:47:35.641 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2022-04-03T19:47:35.641 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2022-04-03T19:47:35.641 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2022-04-03T19:47:35.641 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2022-04-03T19:47:35.642 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2022-04-03T19:47:35.642 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2022-04-03T19:47:35.642 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2022-04-03T19:47:35.642 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2022-04-03T19:47:35.643 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2022-04-03T19:47:35.643 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2022-04-03T19:47:35.643 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2022-04-03T19:47:35.643 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2022-04-03T19:47:35.643 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2022-04-03T19:47:35.644 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2022-04-03T19:47:35.644 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2022-04-03T19:47:35.644 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2022-04-03T19:47:35.644 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2022-04-03T19:47:35.644 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2022-04-03T19:47:35.645 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2022-04-03T19:47:35.645 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2022-04-03T19:47:35.645 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2022-04-03T19:47:35.645 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2022-04-03T19:47:35.646 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2022-04-03T19:47:35.646 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2022-04-03T19:47:35.646 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2022-04-03T19:47:35.646 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2022-04-03T19:47:37.189 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2022-04-03T19:47:37.189 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2022-04-03T19:47:37.189 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2022-04-03T19:47:37.190 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2022-04-03T19:47:37.190 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2022-04-03T19:47:37.190 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2022-04-03T19:47:37.191 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2022-04-03T19:47:37.191 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2022-04-03T19:47:37.191 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2022-04-03T19:47:37.191 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2022-04-03T19:47:37.192 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2022-04-03T19:47:37.192 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2022-04-03T19:47:37.192 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2022-04-03T19:47:37.193 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2022-04-03T19:47:37.193 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2022-04-03T19:47:37.193 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2022-04-03T19:47:37.194 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2022-04-03T19:47:37.194 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2022-04-03T19:47:37.194 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2022-04-03T19:47:37.194 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2022-04-03T19:47:37.195 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2022-04-03T19:47:37.195 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2022-04-03T19:47:37.195 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2022-04-03T19:47:37.196 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2022-04-03T19:47:37.196 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2022-04-03T19:47:37.196 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2022-04-03T19:47:37.196 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2022-04-03T19:47:37.197 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2022-04-03T19:47:37.197 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2022-04-03T19:47:37.197 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2022-04-03T19:47:37.197 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2022-04-03T19:47:37.198 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2022-04-03T19:47:37.198 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2022-04-03T19:47:37.198 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2022-04-03T19:47:37.198 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2022-04-03T19:47:37.199 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2022-04-03T19:47:37.199 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2022-04-03T19:47:37.199 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2022-04-03T19:47:37.199 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2022-04-03T19:47:37.200 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2022-04-03T19:47:37.200 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2022-04-03T19:47:37.200 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2022-04-03T19:47:37.200 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2022-04-03T19:47:37.201 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2022-04-03T19:47:37.201 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2022-04-03T19:47:37.201 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2022-04-03T19:47:37.201 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2022-04-03T19:47:37.202 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2022-04-03T19:47:37.202 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2022-04-03T19:47:37.202 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2022-04-03T19:47:37.202 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2022-04-03T19:47:37.203 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2022-04-03T19:47:37.203 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2022-04-03T19:47:37.203 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2022-04-03T19:47:37.203 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2022-04-03T19:47:37.204 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2022-04-03T19:47:37.204 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2022-04-03T19:47:37.204 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2022-04-03T19:47:37.204 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2022-04-03T19:47:37.205 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2022-04-03T19:47:37.205 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2022-04-03T19:47:37.205 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2022-04-03T19:47:37.205 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2022-04-03T19:47:37.205 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2022-04-03T19:47:37.206 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2022-04-03T19:47:37.206 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2022-04-03T19:47:37.206 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2022-04-03T19:47:37.206 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2022-04-03T19:47:37.207 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2022-04-03T19:47:37.207 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2022-04-03T19:47:37.207 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2022-04-03T19:47:37.207 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2022-04-03T19:47:37.208 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2022-04-03T19:47:37.208 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2022-04-03T19:47:37.208 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2022-04-03T19:47:37.208 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2022-04-03T19:47:37.209 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2022-04-03T19:47:37.209 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2022-04-03T19:47:37.209 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2022-04-03T19:47:37.209 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2022-04-03T19:47:37.210 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2022-04-03T19:47:39.469 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2022-04-03T19:47:39.470 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2022-04-03T19:47:39.470 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2022-04-03T19:47:39.470 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2022-04-03T19:47:39.470 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2022-04-03T19:47:39.471 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2022-04-03T19:47:39.471 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2022-04-03T19:47:39.471 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2022-04-03T19:47:39.471 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2022-04-03T19:47:39.471 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2022-04-03T19:47:39.472 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2022-04-03T19:47:39.472 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2022-04-03T19:47:39.472 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2022-04-03T19:47:39.472 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2022-04-03T19:47:39.472 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2022-04-03T19:47:39.473 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2022-04-03T19:47:39.473 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2022-04-03T19:47:39.473 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2022-04-03T19:47:39.473 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2022-04-03T19:47:39.473 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2022-04-03T19:47:39.474 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2022-04-03T19:47:39.474 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2022-04-03T19:47:39.474 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2022-04-03T19:47:39.474 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2022-04-03T19:47:39.475 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2022-04-03T19:47:39.475 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2022-04-03T19:47:39.475 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2022-04-03T19:47:39.475 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2022-04-03T19:47:39.475 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2022-04-03T19:47:39.476 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2022-04-03T19:47:39.476 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2022-04-03T19:47:39.476 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2022-04-03T19:47:39.476 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2022-04-03T19:47:39.476 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2022-04-03T19:47:39.476 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2022-04-03T19:47:39.477 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2022-04-03T19:47:39.477 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2022-04-03T19:47:39.477 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2022-04-03T19:47:39.477 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2022-04-03T19:47:39.477 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2022-04-03T19:47:39.478 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2022-04-03T19:47:39.478 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2022-04-03T19:47:39.478 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2022-04-03T19:47:39.478 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2022-04-03T19:47:39.478 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2022-04-03T19:47:39.479 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2022-04-03T19:47:39.479 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2022-04-03T19:47:39.479 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2022-04-03T19:47:39.479 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2022-04-03T19:47:39.479 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2022-04-03T19:47:39.480 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2022-04-03T19:47:39.480 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2022-04-03T19:47:39.480 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2022-04-03T19:47:39.480 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2022-04-03T19:47:39.481 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2022-04-03T19:47:39.481 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2022-04-03T19:47:39.481 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2022-04-03T19:47:39.481 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2022-04-03T19:47:39.481 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2022-04-03T19:47:39.482 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2022-04-03T19:47:39.482 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2022-04-03T19:47:39.482 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2022-04-03T19:47:39.482 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2022-04-03T19:47:39.482 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2022-04-03T19:47:39.483 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2022-04-03T19:47:39.483 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2022-04-03T19:47:39.483 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2022-04-03T19:47:39.483 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2022-04-03T19:47:39.484 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2022-04-03T19:47:39.484 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2022-04-03T19:47:39.484 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2022-04-03T19:47:39.484 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2022-04-03T19:47:39.484 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2022-04-03T19:47:39.485 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2022-04-03T19:47:39.485 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2022-04-03T19:47:39.485 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2022-04-03T19:47:39.485 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2022-04-03T19:47:39.485 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2022-04-03T19:47:39.486 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2022-04-03T19:47:39.486 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2022-04-03T19:47:39.486 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2022-04-03T19:47:39.821 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2022-04-03T19:47:39.821 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2022-04-03T19:47:39.821 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2022-04-03T19:47:39.822 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2022-04-03T19:47:39.822 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2022-04-03T19:47:39.822 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2022-04-03T19:47:39.822 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2022-04-03T19:47:39.823 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2022-04-03T19:47:39.823 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2022-04-03T19:47:39.823 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2022-04-03T19:47:39.823 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2022-04-03T19:47:39.823 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2022-04-03T19:47:39.824 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2022-04-03T19:47:39.824 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2022-04-03T19:47:39.824 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2022-04-03T19:47:39.824 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2022-04-03T19:47:39.825 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2022-04-03T19:47:39.825 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2022-04-03T19:47:39.825 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2022-04-03T19:47:39.825 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2022-04-03T19:47:39.825 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2022-04-03T19:47:39.826 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2022-04-03T19:47:39.826 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2022-04-03T19:47:39.826 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2022-04-03T19:47:39.826 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2022-04-03T19:47:39.826 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2022-04-03T19:47:39.827 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2022-04-03T19:47:39.827 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2022-04-03T19:47:39.827 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2022-04-03T19:47:39.827 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2022-04-03T19:47:39.827 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2022-04-03T19:47:39.828 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2022-04-03T19:47:39.828 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2022-04-03T19:47:39.828 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2022-04-03T19:47:39.828 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2022-04-03T19:47:39.829 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2022-04-03T19:47:39.829 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2022-04-03T19:47:39.829 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2022-04-03T19:47:39.829 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2022-04-03T19:47:39.829 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2022-04-03T19:47:39.830 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2022-04-03T19:47:39.830 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2022-04-03T19:47:39.830 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2022-04-03T19:47:39.830 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2022-04-03T19:47:39.831 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2022-04-03T19:47:39.831 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2022-04-03T19:47:39.831 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2022-04-03T19:47:39.831 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2022-04-03T19:47:39.832 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2022-04-03T19:47:39.832 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2022-04-03T19:47:39.832 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2022-04-03T19:47:39.832 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2022-04-03T19:47:39.833 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2022-04-03T19:47:39.833 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2022-04-03T19:47:39.833 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2022-04-03T19:47:39.833 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2022-04-03T19:47:39.834 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2022-04-03T19:47:39.834 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2022-04-03T19:47:39.834 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2022-04-03T19:47:39.834 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2022-04-03T19:47:39.834 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2022-04-03T19:47:39.835 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2022-04-03T19:47:39.835 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2022-04-03T19:47:39.835 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2022-04-03T19:47:39.835 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2022-04-03T19:47:39.836 INFO:tasks.workunit.client.0.smithi077.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2022-04-03T19:47:39.836 INFO:tasks.workunit.client.0.smithi077.stdout:Take third snapshot .snap/test3 2022-04-03T19:47:39.836 INFO:tasks.workunit.client.0.smithi077.stdout:Delete the snapshots... 2022-04-03T19:47:42.254 INFO:tasks.workunit.client.0.smithi077.stdout:.//.snap/test1 2022-04-03T19:47:42.255 INFO:tasks.workunit.client.0.smithi077.stdout:.//.snap/test2 2022-04-03T19:47:42.255 INFO:tasks.workunit.client.0.smithi077.stdout:.//.snap/test3 2022-04-03T19:47:42.262 INFO:tasks.workunit.client.0.smithi077.stdout:./274/.snap/snap-subdir-test 2022-04-03T19:47:42.273 INFO:tasks.workunit.client.0.smithi077.stdout:./166/.snap/snap-subdir-test 2022-04-03T19:47:42.281 INFO:tasks.workunit.client.0.smithi077.stdout:./256/.snap/snap-subdir-test 2022-04-03T19:47:42.290 INFO:tasks.workunit.client.0.smithi077.stdout:./247/.snap/snap-subdir-test 2022-04-03T19:47:43.079 INFO:tasks.workunit.client.0.smithi077.stdout:./248/.snap/snap-subdir-test 2022-04-03T19:47:43.087 INFO:tasks.workunit.client.0.smithi077.stdout:./147/.snap/snap-subdir-test 2022-04-03T19:47:43.098 INFO:tasks.workunit.client.0.smithi077.stdout:./151/.snap/snap-subdir-test 2022-04-03T19:47:43.108 INFO:tasks.workunit.client.0.smithi077.stdout:./170/.snap/snap-subdir-test 2022-04-03T19:47:43.118 INFO:tasks.workunit.client.0.smithi077.stdout:./167/.snap/snap-subdir-test 2022-04-03T19:47:43.128 INFO:tasks.workunit.client.0.smithi077.stdout:./140/.snap/snap-subdir-test 2022-04-03T19:47:43.137 INFO:tasks.workunit.client.0.smithi077.stdout:./142/.snap/snap-subdir-test 2022-04-03T19:47:43.147 INFO:tasks.workunit.client.0.smithi077.stdout:./150/.snap/snap-subdir-test 2022-04-03T19:47:43.154 INFO:tasks.workunit.client.0.smithi077.stdout:./273/.snap/snap-subdir-test 2022-04-03T19:47:43.423 INFO:tasks.workunit.client.0.smithi077.stdout:./269/.snap/snap-subdir-test 2022-04-03T19:47:43.435 INFO:tasks.workunit.client.0.smithi077.stdout:./105/.snap/snap-subdir-test 2022-04-03T19:47:43.447 INFO:tasks.workunit.client.0.smithi077.stdout:./210/.snap/snap-subdir-test 2022-04-03T19:47:43.717 INFO:tasks.workunit.client.0.smithi077.stdout:./265/.snap/snap-subdir-test 2022-04-03T19:47:43.729 INFO:tasks.workunit.client.0.smithi077.stdout:./218/.snap/snap-subdir-test 2022-04-03T19:47:43.737 INFO:tasks.workunit.client.0.smithi077.stdout:./290/.snap/snap-subdir-test 2022-04-03T19:47:43.747 INFO:tasks.workunit.client.0.smithi077.stdout:./149/.snap/snap-subdir-test 2022-04-03T19:47:43.757 INFO:tasks.workunit.client.0.smithi077.stdout:./109/.snap/snap-subdir-test 2022-04-03T19:47:43.765 INFO:tasks.workunit.client.0.smithi077.stdout:./292/.snap/snap-subdir-test 2022-04-03T19:47:44.016 INFO:tasks.workunit.client.0.smithi077.stdout:./242/.snap/snap-subdir-test 2022-04-03T19:47:44.029 INFO:tasks.workunit.client.0.smithi077.stdout:./158/.snap/snap-subdir-test 2022-04-03T19:47:44.041 INFO:tasks.workunit.client.0.smithi077.stdout:./243/.snap/snap-subdir-test 2022-04-03T19:47:44.053 INFO:tasks.workunit.client.0.smithi077.stdout:./106/.snap/snap-subdir-test 2022-04-03T19:47:44.064 INFO:tasks.workunit.client.0.smithi077.stdout:./250/.snap/snap-subdir-test 2022-04-03T19:47:44.075 INFO:tasks.workunit.client.0.smithi077.stdout:./221/.snap/snap-subdir-test 2022-04-03T19:47:44.086 INFO:tasks.workunit.client.0.smithi077.stdout:./244/.snap/snap-subdir-test 2022-04-03T19:47:44.095 INFO:tasks.workunit.client.0.smithi077.stdout:./258/.snap/snap-subdir-test 2022-04-03T19:47:44.103 INFO:tasks.workunit.client.0.smithi077.stdout:./268/.snap/snap-subdir-test 2022-04-03T19:47:44.112 INFO:tasks.workunit.client.0.smithi077.stdout:./189/.snap/snap-subdir-test 2022-04-03T19:47:44.121 INFO:tasks.workunit.client.0.smithi077.stdout:./285/.snap/snap-subdir-test 2022-04-03T19:47:44.608 INFO:tasks.workunit.client.0.smithi077.stdout:./208/.snap/snap-subdir-test 2022-04-03T19:47:44.617 INFO:tasks.workunit.client.0.smithi077.stdout:./288/.snap/snap-subdir-test 2022-04-03T19:47:44.627 INFO:tasks.workunit.client.0.smithi077.stdout:./209/.snap/snap-subdir-test 2022-04-03T19:47:44.636 INFO:tasks.workunit.client.0.smithi077.stdout:./253/.snap/snap-subdir-test 2022-04-03T19:47:44.646 INFO:tasks.workunit.client.0.smithi077.stdout:./249/.snap/snap-subdir-test 2022-04-03T19:47:44.657 INFO:tasks.workunit.client.0.smithi077.stdout:./133/.snap/snap-subdir-test 2022-04-03T19:47:44.666 INFO:tasks.workunit.client.0.smithi077.stdout:./291/.snap/snap-subdir-test 2022-04-03T19:47:44.676 INFO:tasks.workunit.client.0.smithi077.stdout:./226/.snap/snap-subdir-test 2022-04-03T19:47:44.686 INFO:tasks.workunit.client.0.smithi077.stdout:./173/.snap/snap-subdir-test 2022-04-03T19:47:44.695 INFO:tasks.workunit.client.0.smithi077.stdout:./174/.snap/snap-subdir-test 2022-04-03T19:47:44.704 INFO:tasks.workunit.client.0.smithi077.stdout:./299/.snap/snap-subdir-test 2022-04-03T19:47:44.712 INFO:tasks.workunit.client.0.smithi077.stdout:./262/.snap/snap-subdir-test 2022-04-03T19:47:44.722 INFO:tasks.workunit.client.0.smithi077.stdout:./231/.snap/snap-subdir-test 2022-04-03T19:47:44.732 INFO:tasks.workunit.client.0.smithi077.stdout:./214/.snap/snap-subdir-test 2022-04-03T19:47:44.740 INFO:tasks.workunit.client.0.smithi077.stdout:./281/.snap/snap-subdir-test 2022-04-03T19:47:44.750 INFO:tasks.workunit.client.0.smithi077.stdout:./289/.snap/snap-subdir-test 2022-04-03T19:47:44.756 INFO:tasks.workunit.client.0.smithi077.stdout:./283/.snap/snap-subdir-test 2022-04-03T19:47:44.766 INFO:tasks.workunit.client.0.smithi077.stdout:./232/.snap/snap-subdir-test 2022-04-03T19:47:44.773 INFO:tasks.workunit.client.0.smithi077.stdout:./298/.snap/snap-subdir-test 2022-04-03T19:47:44.782 INFO:tasks.workunit.client.0.smithi077.stdout:./233/.snap/snap-subdir-test 2022-04-03T19:47:44.791 INFO:tasks.workunit.client.0.smithi077.stdout:./155/.snap/snap-subdir-test 2022-04-03T19:47:44.800 INFO:tasks.workunit.client.0.smithi077.stdout:./114/.snap/snap-subdir-test 2022-04-03T19:47:44.809 INFO:tasks.workunit.client.0.smithi077.stdout:./159/.snap/snap-subdir-test 2022-04-03T19:47:44.818 INFO:tasks.workunit.client.0.smithi077.stdout:./187/.snap/snap-subdir-test 2022-04-03T19:47:44.828 INFO:tasks.workunit.client.0.smithi077.stdout:./185/.snap/snap-subdir-test 2022-04-03T19:47:44.836 INFO:tasks.workunit.client.0.smithi077.stdout:./293/.snap/snap-subdir-test 2022-04-03T19:47:44.844 INFO:tasks.workunit.client.0.smithi077.stdout:./228/.snap/snap-subdir-test 2022-04-03T19:47:44.853 INFO:tasks.workunit.client.0.smithi077.stdout:./120/.snap/snap-subdir-test 2022-04-03T19:47:44.861 INFO:tasks.workunit.client.0.smithi077.stdout:./110/.snap/snap-subdir-test 2022-04-03T19:47:44.869 INFO:tasks.workunit.client.0.smithi077.stdout:./267/.snap/snap-subdir-test 2022-04-03T19:47:44.879 INFO:tasks.workunit.client.0.smithi077.stdout:./202/.snap/snap-subdir-test 2022-04-03T19:47:45.150 INFO:tasks.workunit.client.0.smithi077.stdout:./137/.snap/snap-subdir-test 2022-04-03T19:47:45.159 INFO:tasks.workunit.client.0.smithi077.stdout:./246/.snap/snap-subdir-test 2022-04-03T19:47:45.167 INFO:tasks.workunit.client.0.smithi077.stdout:./219/.snap/snap-subdir-test 2022-04-03T19:47:45.177 INFO:tasks.workunit.client.0.smithi077.stdout:./198/.snap/snap-subdir-test 2022-04-03T19:47:45.185 INFO:tasks.workunit.client.0.smithi077.stdout:./136/.snap/snap-subdir-test 2022-04-03T19:47:45.193 INFO:tasks.workunit.client.0.smithi077.stdout:./270/.snap/snap-subdir-test 2022-04-03T19:47:45.201 INFO:tasks.workunit.client.0.smithi077.stdout:./125/.snap/snap-subdir-test 2022-04-03T19:47:45.210 INFO:tasks.workunit.client.0.smithi077.stdout:./194/.snap/snap-subdir-test 2022-04-03T19:47:45.218 INFO:tasks.workunit.client.0.smithi077.stdout:./119/.snap/snap-subdir-test 2022-04-03T19:47:45.226 INFO:tasks.workunit.client.0.smithi077.stdout:./297/.snap/snap-subdir-test 2022-04-03T19:47:45.234 INFO:tasks.workunit.client.0.smithi077.stdout:./230/.snap/snap-subdir-test 2022-04-03T19:47:45.241 INFO:tasks.workunit.client.0.smithi077.stdout:./260/.snap/snap-subdir-test 2022-04-03T19:47:45.250 INFO:tasks.workunit.client.0.smithi077.stdout:./157/.snap/snap-subdir-test 2022-04-03T19:47:45.261 INFO:tasks.workunit.client.0.smithi077.stdout:./172/.snap/snap-subdir-test 2022-04-03T19:47:45.268 INFO:tasks.workunit.client.0.smithi077.stdout:./277/.snap/snap-subdir-test 2022-04-03T19:47:45.277 INFO:tasks.workunit.client.0.smithi077.stdout:./186/.snap/snap-subdir-test 2022-04-03T19:47:45.286 INFO:tasks.workunit.client.0.smithi077.stdout:./237/.snap/snap-subdir-test 2022-04-03T19:47:45.295 INFO:tasks.workunit.client.0.smithi077.stdout:./190/.snap/snap-subdir-test 2022-04-03T19:47:45.304 INFO:tasks.workunit.client.0.smithi077.stdout:./213/.snap/snap-subdir-test 2022-04-03T19:47:45.313 INFO:tasks.workunit.client.0.smithi077.stdout:./104/.snap/snap-subdir-test 2022-04-03T19:47:45.322 INFO:tasks.workunit.client.0.smithi077.stdout:./164/.snap/snap-subdir-test 2022-04-03T19:47:45.331 INFO:tasks.workunit.client.0.smithi077.stdout:./112/.snap/snap-subdir-test 2022-04-03T19:47:45.339 INFO:tasks.workunit.client.0.smithi077.stdout:./295/.snap/snap-subdir-test 2022-04-03T19:47:45.346 INFO:tasks.workunit.client.0.smithi077.stdout:./272/.snap/snap-subdir-test 2022-04-03T19:47:45.354 INFO:tasks.workunit.client.0.smithi077.stdout:./100/.snap/snap-subdir-test 2022-04-03T19:47:45.364 INFO:tasks.workunit.client.0.smithi077.stdout:./134/.snap/snap-subdir-test 2022-04-03T19:47:45.374 INFO:tasks.workunit.client.0.smithi077.stdout:./200/.snap/snap-subdir-test 2022-04-03T19:47:45.384 INFO:tasks.workunit.client.0.smithi077.stdout:./115/.snap/snap-subdir-test 2022-04-03T19:47:45.392 INFO:tasks.workunit.client.0.smithi077.stdout:./284/.snap/snap-subdir-test 2022-04-03T19:47:45.396 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:47:45.401 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:47:45.402 INFO:tasks.workunit.client.0.smithi077.stdout:./280/.snap/snap-subdir-test 2022-04-03T19:47:45.409 INFO:tasks.workunit.client.0.smithi077.stdout:./116/.snap/snap-subdir-test 2022-04-03T19:47:45.418 INFO:tasks.workunit.client.0.smithi077.stdout:./245/.snap/snap-subdir-test 2022-04-03T19:47:45.426 INFO:tasks.workunit.client.0.smithi077.stdout:./195/.snap/snap-subdir-test 2022-04-03T19:47:45.435 INFO:tasks.workunit.client.0.smithi077.stdout:./138/.snap/snap-subdir-test 2022-04-03T19:47:45.444 INFO:tasks.workunit.client.0.smithi077.stdout:./135/.snap/snap-subdir-test 2022-04-03T19:47:45.453 INFO:tasks.workunit.client.0.smithi077.stdout:./117/.snap/snap-subdir-test 2022-04-03T19:47:45.462 INFO:tasks.workunit.client.0.smithi077.stdout:./241/.snap/snap-subdir-test 2022-04-03T19:47:45.472 INFO:tasks.workunit.client.0.smithi077.stdout:./143/.snap/snap-subdir-test 2022-04-03T19:47:45.481 INFO:tasks.workunit.client.0.smithi077.stdout:./180/.snap/snap-subdir-test 2022-04-03T19:47:45.489 INFO:tasks.workunit.client.0.smithi077.stdout:./126/.snap/snap-subdir-test 2022-04-03T19:47:45.498 INFO:tasks.workunit.client.0.smithi077.stdout:./161/.snap/snap-subdir-test 2022-04-03T19:47:45.507 INFO:tasks.workunit.client.0.smithi077.stdout:./204/.snap/snap-subdir-test 2022-04-03T19:47:45.819 INFO:tasks.workunit.client.0.smithi077.stdout:./176/.snap/snap-subdir-test 2022-04-03T19:47:45.827 INFO:tasks.workunit.client.0.smithi077.stdout:./255/.snap/snap-subdir-test 2022-04-03T19:47:45.951 INFO:tasks.workunit.client.0.smithi077.stdout:./215/.snap/snap-subdir-test 2022-04-03T19:47:45.960 INFO:tasks.workunit.client.0.smithi077.stdout:./111/.snap/snap-subdir-test 2022-04-03T19:47:45.968 INFO:tasks.workunit.client.0.smithi077.stdout:./238/.snap/snap-subdir-test 2022-04-03T19:47:45.978 INFO:tasks.workunit.client.0.smithi077.stdout:./191/.snap/snap-subdir-test 2022-04-03T19:47:45.986 INFO:tasks.workunit.client.0.smithi077.stdout:./184/.snap/snap-subdir-test 2022-04-03T19:47:46.976 INFO:tasks.workunit.client.0.smithi077.stdout:./192/.snap/snap-subdir-test 2022-04-03T19:47:46.987 INFO:tasks.workunit.client.0.smithi077.stdout:./103/.snap/snap-subdir-test 2022-04-03T19:47:46.997 INFO:tasks.workunit.client.0.smithi077.stdout:./128/.snap/snap-subdir-test 2022-04-03T19:47:47.008 INFO:tasks.workunit.client.0.smithi077.stdout:./229/.snap/snap-subdir-test 2022-04-03T19:47:47.018 INFO:tasks.workunit.client.0.smithi077.stdout:./144/.snap/snap-subdir-test 2022-04-03T19:47:47.029 INFO:tasks.workunit.client.0.smithi077.stdout:./146/.snap/snap-subdir-test 2022-04-03T19:47:47.038 INFO:tasks.workunit.client.0.smithi077.stdout:./227/.snap/snap-subdir-test 2022-04-03T19:47:47.047 INFO:tasks.workunit.client.0.smithi077.stdout:./278/.snap/snap-subdir-test 2022-04-03T19:47:47.055 INFO:tasks.workunit.client.0.smithi077.stdout:./141/.snap/snap-subdir-test 2022-04-03T19:47:47.064 INFO:tasks.workunit.client.0.smithi077.stdout:./234/.snap/snap-subdir-test 2022-04-03T19:47:47.074 INFO:tasks.workunit.client.0.smithi077.stdout:./225/.snap/snap-subdir-test 2022-04-03T19:47:47.083 INFO:tasks.workunit.client.0.smithi077.stdout:./182/.snap/snap-subdir-test 2022-04-03T19:47:47.092 INFO:tasks.workunit.client.0.smithi077.stdout:./107/.snap/snap-subdir-test 2022-04-03T19:47:47.102 INFO:tasks.workunit.client.0.smithi077.stdout:./178/.snap/snap-subdir-test 2022-04-03T19:47:47.109 INFO:tasks.workunit.client.0.smithi077.stdout:./266/.snap/snap-subdir-test 2022-04-03T19:47:47.118 INFO:tasks.workunit.client.0.smithi077.stdout:./240/.snap/snap-subdir-test 2022-04-03T19:47:47.127 INFO:tasks.workunit.client.0.smithi077.stdout:./156/.snap/snap-subdir-test 2022-04-03T19:47:47.136 INFO:tasks.workunit.client.0.smithi077.stdout:./131/.snap/snap-subdir-test 2022-04-03T19:47:47.145 INFO:tasks.workunit.client.0.smithi077.stdout:./123/.snap/snap-subdir-test 2022-04-03T19:47:47.154 INFO:tasks.workunit.client.0.smithi077.stdout:./160/.snap/snap-subdir-test 2022-04-03T19:47:47.163 INFO:tasks.workunit.client.0.smithi077.stdout:./165/.snap/snap-subdir-test 2022-04-03T19:47:47.172 INFO:tasks.workunit.client.0.smithi077.stdout:./101/.snap/snap-subdir-test 2022-04-03T19:47:47.181 INFO:tasks.workunit.client.0.smithi077.stdout:./148/.snap/snap-subdir-test 2022-04-03T19:47:47.190 INFO:tasks.workunit.client.0.smithi077.stdout:./168/.snap/snap-subdir-test 2022-04-03T19:47:47.199 INFO:tasks.workunit.client.0.smithi077.stdout:./179/.snap/snap-subdir-test 2022-04-03T19:47:47.206 INFO:tasks.workunit.client.0.smithi077.stdout:./257/.snap/snap-subdir-test 2022-04-03T19:47:47.215 INFO:tasks.workunit.client.0.smithi077.stdout:./252/.snap/snap-subdir-test 2022-04-03T19:47:47.224 INFO:tasks.workunit.client.0.smithi077.stdout:./132/.snap/snap-subdir-test 2022-04-03T19:47:47.233 INFO:tasks.workunit.client.0.smithi077.stdout:./236/.snap/snap-subdir-test 2022-04-03T19:47:47.241 INFO:tasks.workunit.client.0.smithi077.stdout:./211/.snap/snap-subdir-test 2022-04-03T19:47:47.251 INFO:tasks.workunit.client.0.smithi077.stdout:./199/.snap/snap-subdir-test 2022-04-03T19:47:47.259 INFO:tasks.workunit.client.0.smithi077.stdout:./205/.snap/snap-subdir-test 2022-04-03T19:47:47.268 INFO:tasks.workunit.client.0.smithi077.stdout:./163/.snap/snap-subdir-test 2022-04-03T19:47:47.275 INFO:tasks.workunit.client.0.smithi077.stdout:./261/.snap/snap-subdir-test 2022-04-03T19:47:47.284 INFO:tasks.workunit.client.0.smithi077.stdout:./201/.snap/snap-subdir-test 2022-04-03T19:47:47.292 INFO:tasks.workunit.client.0.smithi077.stdout:./282/.snap/snap-subdir-test 2022-04-03T19:47:47.564 INFO:tasks.workunit.client.0.smithi077.stdout:./296/.snap/snap-subdir-test 2022-04-03T19:47:47.573 INFO:tasks.workunit.client.0.smithi077.stdout:./217/.snap/snap-subdir-test 2022-04-03T19:47:47.581 INFO:tasks.workunit.client.0.smithi077.stdout:./153/.snap/snap-subdir-test 2022-04-03T19:47:47.589 INFO:tasks.workunit.client.0.smithi077.stdout:./206/.snap/snap-subdir-test 2022-04-03T19:47:47.598 INFO:tasks.workunit.client.0.smithi077.stdout:./122/.snap/snap-subdir-test 2022-04-03T19:47:47.607 INFO:tasks.workunit.client.0.smithi077.stdout:./162/.snap/snap-subdir-test 2022-04-03T19:47:47.616 INFO:tasks.workunit.client.0.smithi077.stdout:./197/.snap/snap-subdir-test 2022-04-03T19:47:47.623 INFO:tasks.workunit.client.0.smithi077.stdout:./216/.snap/snap-subdir-test 2022-04-03T19:47:50.509 INFO:tasks.workunit.client.0.smithi077.stdout:./254/.snap/snap-subdir-test 2022-04-03T19:47:50.517 INFO:tasks.workunit.client.0.smithi077.stdout:./113/.snap/snap-subdir-test 2022-04-03T19:47:51.461 INFO:tasks.workunit.client.0.smithi077.stdout:./152/.snap/snap-subdir-test 2022-04-03T19:47:51.472 INFO:tasks.workunit.client.0.smithi077.stdout:./121/.snap/snap-subdir-test 2022-04-03T19:47:51.483 INFO:tasks.workunit.client.0.smithi077.stdout:./118/.snap/snap-subdir-test 2022-04-03T19:47:51.494 INFO:tasks.workunit.client.0.smithi077.stdout:./196/.snap/snap-subdir-test 2022-04-03T19:47:51.770 INFO:tasks.workunit.client.0.smithi077.stdout:./124/.snap/snap-subdir-test 2022-04-03T19:47:51.781 INFO:tasks.workunit.client.0.smithi077.stdout:./239/.snap/snap-subdir-test 2022-04-03T19:47:51.793 INFO:tasks.workunit.client.0.smithi077.stdout:./188/.snap/snap-subdir-test 2022-04-03T19:47:51.802 INFO:tasks.workunit.client.0.smithi077.stdout:./287/.snap/snap-subdir-test 2022-04-03T19:47:52.072 INFO:tasks.workunit.client.0.smithi077.stdout:./275/.snap/snap-subdir-test 2022-04-03T19:47:52.081 INFO:tasks.workunit.client.0.smithi077.stdout:./259/.snap/snap-subdir-test 2022-04-03T19:47:52.093 INFO:tasks.workunit.client.0.smithi077.stdout:./224/.snap/snap-subdir-test 2022-04-03T19:47:52.102 INFO:tasks.workunit.client.0.smithi077.stdout:./264/.snap/snap-subdir-test 2022-04-03T19:47:52.113 INFO:tasks.workunit.client.0.smithi077.stdout:./145/.snap/snap-subdir-test 2022-04-03T19:47:52.122 INFO:tasks.workunit.client.0.smithi077.stdout:./279/.snap/snap-subdir-test 2022-04-03T19:47:52.130 INFO:tasks.workunit.client.0.smithi077.stdout:./263/.snap/snap-subdir-test 2022-04-03T19:47:52.141 INFO:tasks.workunit.client.0.smithi077.stdout:./130/.snap/snap-subdir-test 2022-04-03T19:47:52.151 INFO:tasks.workunit.client.0.smithi077.stdout:./193/.snap/snap-subdir-test 2022-04-03T19:47:52.162 INFO:tasks.workunit.client.0.smithi077.stdout:./223/.snap/snap-subdir-test 2022-04-03T19:47:52.172 INFO:tasks.workunit.client.0.smithi077.stdout:./207/.snap/snap-subdir-test 2022-04-03T19:47:52.179 INFO:tasks.workunit.client.0.smithi077.stdout:./276/.snap/snap-subdir-test 2022-04-03T19:47:52.188 INFO:tasks.workunit.client.0.smithi077.stdout:./271/.snap/snap-subdir-test 2022-04-03T19:47:52.196 INFO:tasks.workunit.client.0.smithi077.stdout:./251/.snap/snap-subdir-test 2022-04-03T19:47:52.206 INFO:tasks.workunit.client.0.smithi077.stdout:./175/.snap/snap-subdir-test 2022-04-03T19:47:52.215 INFO:tasks.workunit.client.0.smithi077.stdout:./222/.snap/snap-subdir-test 2022-04-03T19:47:52.224 INFO:tasks.workunit.client.0.smithi077.stdout:./220/.snap/snap-subdir-test 2022-04-03T19:47:52.233 INFO:tasks.workunit.client.0.smithi077.stdout:./203/.snap/snap-subdir-test 2022-04-03T19:47:52.242 INFO:tasks.workunit.client.0.smithi077.stdout:./181/.snap/snap-subdir-test 2022-04-03T19:47:52.913 INFO:tasks.workunit.client.0.smithi077.stdout:./286/.snap/snap-subdir-test 2022-04-03T19:47:53.534 INFO:tasks.workunit.client.0.smithi077.stdout:./127/.snap/snap-subdir-test 2022-04-03T19:47:54.086 INFO:tasks.workunit.client.0.smithi077.stdout:./108/.snap/snap-subdir-test 2022-04-03T19:47:54.096 INFO:tasks.workunit.client.0.smithi077.stdout:./183/.snap/snap-subdir-test 2022-04-03T19:47:54.104 INFO:tasks.workunit.client.0.smithi077.stdout:./294/.snap/snap-subdir-test 2022-04-03T19:47:54.380 INFO:tasks.workunit.client.0.smithi077.stdout:./235/.snap/snap-subdir-test 2022-04-03T19:47:54.393 INFO:tasks.workunit.client.0.smithi077.stdout:./177/.snap/snap-subdir-test 2022-04-03T19:47:54.405 INFO:tasks.workunit.client.0.smithi077.stdout:./169/.snap/snap-subdir-test 2022-04-03T19:47:54.416 INFO:tasks.workunit.client.0.smithi077.stdout:./212/.snap/snap-subdir-test 2022-04-03T19:47:54.428 INFO:tasks.workunit.client.0.smithi077.stdout:./154/.snap/snap-subdir-test 2022-04-03T19:47:54.702 INFO:tasks.workunit.client.0.smithi077.stdout:./171/.snap/snap-subdir-test 2022-04-03T19:47:54.711 INFO:tasks.workunit.client.0.smithi077.stdout:./139/.snap/snap-subdir-test 2022-04-03T19:47:54.720 INFO:tasks.workunit.client.0.smithi077.stdout:./102/.snap/snap-subdir-test 2022-04-03T19:47:54.731 INFO:tasks.workunit.client.0.smithi077.stdout:./129/.snap/snap-subdir-test 2022-04-03T19:47:59.120 INFO:tasks.workunit.client.0.smithi077.stdout:Delete all the files and directories ... 2022-04-03T19:48:01.818 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/19' 2022-04-03T19:48:01.819 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/15' 2022-04-03T19:48:01.820 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/31' 2022-04-03T19:48:01.820 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/36' 2022-04-03T19:48:01.820 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/20' 2022-04-03T19:48:01.821 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/12' 2022-04-03T19:48:01.821 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/39' 2022-04-03T19:48:01.822 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/10' 2022-04-03T19:48:01.822 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/34' 2022-04-03T19:48:01.823 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/38' 2022-04-03T19:48:01.823 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/11' 2022-04-03T19:48:01.824 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/16' 2022-04-03T19:48:01.824 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/32' 2022-04-03T19:48:01.825 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/30' 2022-04-03T19:48:01.825 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/14' 2022-04-03T19:48:01.826 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/18' 2022-04-03T19:48:01.826 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/13' 2022-04-03T19:48:01.826 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/33' 2022-04-03T19:48:01.827 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/17' 2022-04-03T19:48:01.828 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/35' 2022-04-03T19:48:01.828 INFO:tasks.workunit.client.0.smithi077.stdout:removed './100/37' 2022-04-03T19:48:01.828 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './100' 2022-04-03T19:48:01.829 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/19' 2022-04-03T19:48:01.829 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/15' 2022-04-03T19:48:01.830 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/31' 2022-04-03T19:48:01.830 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/36' 2022-04-03T19:48:01.831 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/20' 2022-04-03T19:48:01.831 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/12' 2022-04-03T19:48:01.831 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/39' 2022-04-03T19:48:01.832 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/10' 2022-04-03T19:48:01.832 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/34' 2022-04-03T19:48:01.833 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/38' 2022-04-03T19:48:01.833 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/11' 2022-04-03T19:48:01.834 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/16' 2022-04-03T19:48:01.834 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/32' 2022-04-03T19:48:01.835 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/30' 2022-04-03T19:48:01.835 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/14' 2022-04-03T19:48:01.835 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/18' 2022-04-03T19:48:01.836 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/13' 2022-04-03T19:48:01.836 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/33' 2022-04-03T19:48:01.837 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/17' 2022-04-03T19:48:01.837 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/35' 2022-04-03T19:48:01.838 INFO:tasks.workunit.client.0.smithi077.stdout:removed './101/37' 2022-04-03T19:48:01.838 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './101' 2022-04-03T19:48:01.839 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/19' 2022-04-03T19:48:01.839 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/15' 2022-04-03T19:48:01.839 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/31' 2022-04-03T19:48:01.839 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/36' 2022-04-03T19:48:01.840 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/20' 2022-04-03T19:48:01.840 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/12' 2022-04-03T19:48:01.840 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/39' 2022-04-03T19:48:01.840 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/10' 2022-04-03T19:48:01.841 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/34' 2022-04-03T19:48:01.841 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/38' 2022-04-03T19:48:01.841 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/11' 2022-04-03T19:48:01.841 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/16' 2022-04-03T19:48:01.842 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/32' 2022-04-03T19:48:01.842 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/30' 2022-04-03T19:48:01.842 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/14' 2022-04-03T19:48:01.842 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/18' 2022-04-03T19:48:01.842 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/13' 2022-04-03T19:48:01.843 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/33' 2022-04-03T19:48:01.843 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/17' 2022-04-03T19:48:01.843 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/35' 2022-04-03T19:48:01.843 INFO:tasks.workunit.client.0.smithi077.stdout:removed './102/37' 2022-04-03T19:48:01.844 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './102' 2022-04-03T19:48:01.844 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/19' 2022-04-03T19:48:01.844 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/15' 2022-04-03T19:48:01.844 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/31' 2022-04-03T19:48:01.845 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/36' 2022-04-03T19:48:01.845 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/20' 2022-04-03T19:48:01.845 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/12' 2022-04-03T19:48:01.845 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/39' 2022-04-03T19:48:01.845 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/10' 2022-04-03T19:48:01.846 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/34' 2022-04-03T19:48:01.846 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/38' 2022-04-03T19:48:01.846 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/11' 2022-04-03T19:48:01.846 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/16' 2022-04-03T19:48:01.847 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/32' 2022-04-03T19:48:01.847 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/30' 2022-04-03T19:48:01.847 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/14' 2022-04-03T19:48:01.847 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/18' 2022-04-03T19:48:01.847 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/13' 2022-04-03T19:48:01.848 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/33' 2022-04-03T19:48:01.848 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/17' 2022-04-03T19:48:01.848 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/35' 2022-04-03T19:48:01.848 INFO:tasks.workunit.client.0.smithi077.stdout:removed './103/37' 2022-04-03T19:48:01.848 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './103' 2022-04-03T19:48:01.849 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/19' 2022-04-03T19:48:01.849 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/15' 2022-04-03T19:48:01.849 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/31' 2022-04-03T19:48:01.849 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/36' 2022-04-03T19:48:01.850 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/20' 2022-04-03T19:48:01.850 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/12' 2022-04-03T19:48:01.850 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/39' 2022-04-03T19:48:01.850 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/10' 2022-04-03T19:48:01.851 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/34' 2022-04-03T19:48:01.851 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/38' 2022-04-03T19:48:01.851 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/11' 2022-04-03T19:48:01.851 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/16' 2022-04-03T19:48:01.851 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/32' 2022-04-03T19:48:01.852 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/30' 2022-04-03T19:48:01.852 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/14' 2022-04-03T19:48:01.852 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/18' 2022-04-03T19:48:01.852 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/13' 2022-04-03T19:48:01.853 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/33' 2022-04-03T19:48:01.853 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/17' 2022-04-03T19:48:01.853 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/35' 2022-04-03T19:48:01.853 INFO:tasks.workunit.client.0.smithi077.stdout:removed './104/37' 2022-04-03T19:48:01.853 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './104' 2022-04-03T19:48:01.854 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/19' 2022-04-03T19:48:01.854 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/15' 2022-04-03T19:48:01.854 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/31' 2022-04-03T19:48:01.854 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/36' 2022-04-03T19:48:01.855 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/20' 2022-04-03T19:48:01.855 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/12' 2022-04-03T19:48:01.855 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/39' 2022-04-03T19:48:01.855 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/10' 2022-04-03T19:48:01.855 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/34' 2022-04-03T19:48:01.856 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/38' 2022-04-03T19:48:01.856 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/11' 2022-04-03T19:48:01.856 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/16' 2022-04-03T19:48:01.856 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/32' 2022-04-03T19:48:01.857 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/30' 2022-04-03T19:48:01.857 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/14' 2022-04-03T19:48:01.857 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/18' 2022-04-03T19:48:01.857 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/13' 2022-04-03T19:48:01.857 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/33' 2022-04-03T19:48:01.858 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/17' 2022-04-03T19:48:01.858 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/35' 2022-04-03T19:48:01.858 INFO:tasks.workunit.client.0.smithi077.stdout:removed './105/37' 2022-04-03T19:48:01.858 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './105' 2022-04-03T19:48:01.859 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/19' 2022-04-03T19:48:01.859 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/15' 2022-04-03T19:48:01.859 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/31' 2022-04-03T19:48:01.859 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/36' 2022-04-03T19:48:01.859 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/20' 2022-04-03T19:48:01.860 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/12' 2022-04-03T19:48:01.860 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/39' 2022-04-03T19:48:01.860 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/10' 2022-04-03T19:48:01.860 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/34' 2022-04-03T19:48:01.861 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/38' 2022-04-03T19:48:01.861 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/11' 2022-04-03T19:48:01.861 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/16' 2022-04-03T19:48:01.861 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/32' 2022-04-03T19:48:01.861 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/30' 2022-04-03T19:48:01.862 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/14' 2022-04-03T19:48:01.862 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/18' 2022-04-03T19:48:01.862 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/13' 2022-04-03T19:48:01.862 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/33' 2022-04-03T19:48:01.863 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/17' 2022-04-03T19:48:01.863 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/35' 2022-04-03T19:48:01.863 INFO:tasks.workunit.client.0.smithi077.stdout:removed './106/37' 2022-04-03T19:48:01.863 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './106' 2022-04-03T19:48:01.863 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/19' 2022-04-03T19:48:01.864 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/15' 2022-04-03T19:48:01.864 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/31' 2022-04-03T19:48:01.864 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/36' 2022-04-03T19:48:01.864 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/20' 2022-04-03T19:48:01.864 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/12' 2022-04-03T19:48:01.865 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/39' 2022-04-03T19:48:01.865 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/10' 2022-04-03T19:48:01.865 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/34' 2022-04-03T19:48:01.865 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/38' 2022-04-03T19:48:01.866 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/11' 2022-04-03T19:48:01.866 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/16' 2022-04-03T19:48:01.866 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/32' 2022-04-03T19:48:01.866 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/30' 2022-04-03T19:48:01.866 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/14' 2022-04-03T19:48:01.867 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/18' 2022-04-03T19:48:01.867 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/13' 2022-04-03T19:48:01.867 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/33' 2022-04-03T19:48:01.867 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/17' 2022-04-03T19:48:01.868 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/35' 2022-04-03T19:48:01.868 INFO:tasks.workunit.client.0.smithi077.stdout:removed './107/37' 2022-04-03T19:48:01.868 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './107' 2022-04-03T19:48:01.868 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/19' 2022-04-03T19:48:01.868 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/15' 2022-04-03T19:48:01.869 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/31' 2022-04-03T19:48:01.869 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/36' 2022-04-03T19:48:01.869 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/20' 2022-04-03T19:48:01.869 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/12' 2022-04-03T19:48:01.869 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/39' 2022-04-03T19:48:01.870 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/10' 2022-04-03T19:48:01.870 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/34' 2022-04-03T19:48:01.870 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/38' 2022-04-03T19:48:01.870 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/11' 2022-04-03T19:48:01.871 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/16' 2022-04-03T19:48:01.871 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/32' 2022-04-03T19:48:01.871 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/30' 2022-04-03T19:48:01.871 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/14' 2022-04-03T19:48:01.872 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/18' 2022-04-03T19:48:01.872 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/13' 2022-04-03T19:48:01.872 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/33' 2022-04-03T19:48:01.872 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/17' 2022-04-03T19:48:01.872 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/35' 2022-04-03T19:48:01.873 INFO:tasks.workunit.client.0.smithi077.stdout:removed './108/37' 2022-04-03T19:48:01.873 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './108' 2022-04-03T19:48:01.873 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/19' 2022-04-03T19:48:01.873 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/15' 2022-04-03T19:48:01.873 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/31' 2022-04-03T19:48:01.874 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/36' 2022-04-03T19:48:01.874 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/20' 2022-04-03T19:48:01.874 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/12' 2022-04-03T19:48:01.874 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/39' 2022-04-03T19:48:01.875 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/10' 2022-04-03T19:48:01.875 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/34' 2022-04-03T19:48:01.875 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/38' 2022-04-03T19:48:01.875 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/11' 2022-04-03T19:48:01.875 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/16' 2022-04-03T19:48:01.876 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/32' 2022-04-03T19:48:01.876 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/30' 2022-04-03T19:48:02.242 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:02.441 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.440+0000 7f1e7d79c700 1 Processor -- start 2022-04-03T19:48:02.442 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.440+0000 7f1e7d79c700 1 -- start start 2022-04-03T19:48:02.443 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.441+0000 7f1e7d79c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1e780c1b30 0x7f1e7814d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:02.443 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.441+0000 7f1e7d79c700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e780c3c50 con 0x7f1e780ba3c0 2022-04-03T19:48:02.443 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.441+0000 7f1e7d79c700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e78156690 con 0x7f1e780678b0 2022-04-03T19:48:02.444 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.441+0000 7f1e7d79c700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e780c8020 con 0x7f1e780c1b30 2022-04-03T19:48:02.444 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.442+0000 7f1e77fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1e780ba3c0 0x7f1e78156b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38422/0 (socket says 172.21.15.77:38422) 2022-04-03T19:48:02.444 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.442+0000 7f1e77fff700 1 -- 172.21.15.77:0/3180364432 learned_addr learned my addr 172.21.15.77:0/3180364432 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:48:02.444 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.442+0000 7f1e7cf9b700 1 --2- 172.21.15.77:0/3180364432 >> v2:172.21.15.77:3300/0 conn(0x7f1e780c1b30 0x7f1e7814d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:02.446 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.442+0000 7f1e76ffd700 1 -- 172.21.15.77:0/3180364432 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 279803694 0 0) 0x7f1e78156690 con 0x7f1e780678b0 2022-04-03T19:48:02.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.442+0000 7f1e76ffd700 1 -- 172.21.15.77:0/3180364432 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e640033f0 con 0x7f1e780678b0 2022-04-03T19:48:02.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.442+0000 7f1e76ffd700 1 -- 172.21.15.77:0/3180364432 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464594415 0 0) 0x7f1e780c3c50 con 0x7f1e780ba3c0 2022-04-03T19:48:02.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.442+0000 7f1e76ffd700 1 -- 172.21.15.77:0/3180364432 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e78156690 con 0x7f1e780ba3c0 2022-04-03T19:48:02.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.442+0000 7f1e7cf9b700 1 -- 172.21.15.77:0/3180364432 >> v1:172.21.15.186:6789/0 conn(0x7f1e780678b0 legacy=0x7f1e7814cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:02.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.442+0000 7f1e7cf9b700 1 -- 172.21.15.77:0/3180364432 >> v1:172.21.15.77:6789/0 conn(0x7f1e780ba3c0 legacy=0x7f1e78156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:02.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.442+0000 7f1e7cf9b700 1 -- 172.21.15.77:0/3180364432 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e780bf350 con 0x7f1e780c1b30 2022-04-03T19:48:02.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.443+0000 7f1e7cf9b700 1 --2- 172.21.15.77:0/3180364432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e780c1b30 0x7f1e7814d210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1e70009e60 tx=0x7f1e70009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82196f329f434d17 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:02.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.443+0000 7f1e76ffd700 1 -- 172.21.15.77:0/3180364432 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7000e070 con 0x7f1e780c1b30 2022-04-03T19:48:02.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.443+0000 7f1e76ffd700 1 -- 172.21.15.77:0/3180364432 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7000bb40 con 0x7f1e780c1b30 2022-04-03T19:48:02.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.443+0000 7f1e76ffd700 1 -- 172.21.15.77:0/3180364432 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e70012430 con 0x7f1e780c1b30 2022-04-03T19:48:02.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.443+0000 7f1e7d79c700 1 -- 172.21.15.77:0/3180364432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e780c1b30 msgr2=0x7f1e7814d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:02.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.443+0000 7f1e7d79c700 1 --2- 172.21.15.77:0/3180364432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e780c1b30 0x7f1e7814d210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1e70009e60 tx=0x7f1e70009780 comp rx=0 tx=0).stop 2022-04-03T19:48:02.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.443+0000 7f1e7d79c700 1 -- 172.21.15.77:0/3180364432 shutdown_connections 2022-04-03T19:48:02.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.443+0000 7f1e7d79c700 1 --2- 172.21.15.77:0/3180364432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e780c1b30 0x7f1e7814d210 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:02.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.443+0000 7f1e7d79c700 1 -- 172.21.15.77:0/3180364432 >> 172.21.15.77:0/3180364432 conn(0x7f1e78066da0 msgr2=0x7f1e780671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:02.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.444+0000 7f1e7d79c700 1 -- 172.21.15.77:0/3180364432 shutdown_connections 2022-04-03T19:48:02.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.444+0000 7f1e7d79c700 1 -- 172.21.15.77:0/3180364432 wait complete. 2022-04-03T19:48:02.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.445+0000 7f1e7d79c700 1 Processor -- start 2022-04-03T19:48:02.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.445+0000 7f1e7d79c700 1 -- start start 2022-04-03T19:48:02.453 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.446+0000 7f1e7d79c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1e780678b0 0x7f1e78155470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:02.454 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.446+0000 7f1e7d79c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e780ba3c0 0x7f1e78165840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:02.454 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.446+0000 7f1e7d79c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e780c1b30 0x7f1e780bb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:02.454 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.446+0000 7f1e7d79c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e780c5140 con 0x7f1e780c1b30 2022-04-03T19:48:02.454 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.446+0000 7f1e7d79c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e780bb880 con 0x7f1e780ba3c0 2022-04-03T19:48:02.455 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.446+0000 7f1e7d79c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e780c6260 con 0x7f1e780678b0 2022-04-03T19:48:02.455 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.446+0000 7f1e77fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1e780678b0 0x7f1e78155470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:02.455 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.446+0000 7f1e777fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e780ba3c0 0x7f1e78165840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:02.455 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.446+0000 7f1e7cf9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e780c1b30 0x7f1e780bb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:02.456 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.446+0000 7f1e777fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e780ba3c0 0x7f1e78165840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38002/0 (socket says 172.21.15.77:38002) 2022-04-03T19:48:02.456 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.446+0000 7f1e77fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1e780678b0 0x7f1e78155470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60108/0 (socket says 172.21.15.77:60108) 2022-04-03T19:48:02.456 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.446+0000 7f1e777fe700 1 -- 172.21.15.77:0/2206164294 learned_addr learned my addr 172.21.15.77:0/2206164294 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:02.456 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.447+0000 7f1e777fe700 1 -- 172.21.15.77:0/2206164294 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1e780678b0 msgr2=0x7f1e78155470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:02.456 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.447+0000 7f1e777fe700 1 --2- 172.21.15.77:0/2206164294 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1e780678b0 0x7f1e78155470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:02.457 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.447+0000 7f1e777fe700 1 -- 172.21.15.77:0/2206164294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e780c1b30 msgr2=0x7f1e780bb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:02.457 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.447+0000 7f1e777fe700 1 --2- 172.21.15.77:0/2206164294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e780c1b30 0x7f1e780bb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:02.457 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.447+0000 7f1e777fe700 1 -- 172.21.15.77:0/2206164294 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e70009a20 con 0x7f1e780ba3c0 2022-04-03T19:48:02.457 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.447+0000 7f1e77fff700 1 --2- 172.21.15.77:0/2206164294 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1e780678b0 0x7f1e78155470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:48:02.458 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.447+0000 7f1e777fe700 1 --2- 172.21.15.77:0/2206164294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e780ba3c0 0x7f1e78165840 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c00c440 tx=0x7f1e6c007bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:02.458 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.447+0000 7f1e5ffff700 1 -- 172.21.15.77:0/2206164294 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c0050d0 con 0x7f1e780ba3c0 2022-04-03T19:48:02.458 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.447+0000 7f1e5ffff700 1 -- 172.21.15.77:0/2206164294 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e6c005230 con 0x7f1e780ba3c0 2022-04-03T19:48:02.458 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.447+0000 7f1e7d79c700 1 -- 172.21.15.77:0/2206164294 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e780c5370 con 0x7f1e780ba3c0 2022-04-03T19:48:02.458 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.447+0000 7f1e5ffff700 1 -- 172.21.15.77:0/2206164294 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c003710 con 0x7f1e780ba3c0 2022-04-03T19:48:02.459 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.448+0000 7f1e7d79c700 1 -- 172.21.15.77:0/2206164294 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e780c5930 con 0x7f1e780ba3c0 2022-04-03T19:48:02.459 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.449+0000 7f1e7d79c700 1 -- 172.21.15.77:0/2206164294 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e780c74a0 con 0x7f1e780ba3c0 2022-04-03T19:48:02.459 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.449+0000 7f1e5ffff700 1 -- 172.21.15.77:0/2206164294 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f1e6c019030 con 0x7f1e780ba3c0 2022-04-03T19:48:02.459 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.450+0000 7f1e5ffff700 1 --2- 172.21.15.77:0/2206164294 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1e58063ce0 0x7f1e58066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:02.459 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.450+0000 7f1e5ffff700 1 -- 172.21.15.77:0/2206164294 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6482+0+0 (secure 0 0 0) 0x7f1e6c07fa50 con 0x7f1e780ba3c0 2022-04-03T19:48:02.460 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.451+0000 7f1e77fff700 1 --2- 172.21.15.77:0/2206164294 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1e58063ce0 0x7f1e58066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:02.460 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.451+0000 7f1e77fff700 1 --2- 172.21.15.77:0/2206164294 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1e58063ce0 0x7f1e58066190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1e68007fd0 tx=0x7f1e68009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:02.460 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.455+0000 7f1e5ffff700 1 -- 172.21.15.77:0/2206164294 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f1e6c04e480 con 0x7f1e780ba3c0 2022-04-03T19:48:02.687 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.686+0000 7f1e7d79c700 1 -- 172.21.15.77:0/2206164294 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e7804b050 con 0x7f1e780ba3c0 2022-04-03T19:48:02.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.687+0000 7f1e5ffff700 1 -- 172.21.15.77:0/2206164294 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6112 (secure 0 0 0) 0x7f1e6c017020 con 0x7f1e780ba3c0 2022-04-03T19:48:02.688 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:02.689 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4960,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:47:10.107928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:active","state_seq":47,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:02.691 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.689+0000 7f1e7d79c700 1 -- 172.21.15.77:0/2206164294 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1e58063ce0 msgr2=0x7f1e58066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:02.691 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.689+0000 7f1e7d79c700 1 --2- 172.21.15.77:0/2206164294 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1e58063ce0 0x7f1e58066190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1e68007fd0 tx=0x7f1e68009040 comp rx=0 tx=0).stop 2022-04-03T19:48:02.691 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.689+0000 7f1e7d79c700 1 -- 172.21.15.77:0/2206164294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e780ba3c0 msgr2=0x7f1e78165840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:02.692 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.689+0000 7f1e7d79c700 1 --2- 172.21.15.77:0/2206164294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e780ba3c0 0x7f1e78165840 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c00c440 tx=0x7f1e6c007bf0 comp rx=0 tx=0).stop 2022-04-03T19:48:02.692 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.689+0000 7f1e7d79c700 1 -- 172.21.15.77:0/2206164294 shutdown_connections 2022-04-03T19:48:02.692 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.689+0000 7f1e7d79c700 1 --2- 172.21.15.77:0/2206164294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e780c1b30 0x7f1e780bb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:02.692 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.689+0000 7f1e7d79c700 1 --2- 172.21.15.77:0/2206164294 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1e58063ce0 0x7f1e58066190 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:02.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.689+0000 7f1e7d79c700 1 --2- 172.21.15.77:0/2206164294 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1e780678b0 0x7f1e78155470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:02.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.689+0000 7f1e7d79c700 1 --2- 172.21.15.77:0/2206164294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e780ba3c0 0x7f1e78165840 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:02.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.689+0000 7f1e7d79c700 1 -- 172.21.15.77:0/2206164294 >> 172.21.15.77:0/2206164294 conn(0x7f1e78066da0 msgr2=0x7f1e7814bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:02.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.689+0000 7f1e7d79c700 1 -- 172.21.15.77:0/2206164294 shutdown_connections 2022-04-03T19:48:02.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.689+0000 7f1e7d79c700 1 -- 172.21.15.77:0/2206164294 wait complete. 2022-04-03T19:48:02.694 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 14 2022-04-03T19:48:02.715 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2022-04-03T19:48:02.716 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2022-04-03T19:48:02.716 INFO:teuthology.orchestra.run:waiting for 300 2022-04-03T19:48:02.728 INFO:tasks.ceph.mds.d:Stopped 2022-04-03T19:48:02.728 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:02.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.924+0000 7f45fb59e700 1 Processor -- start 2022-04-03T19:48:02.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.924+0000 7f45fb59e700 1 -- start start 2022-04-03T19:48:02.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.925+0000 7f45fb59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45fc1584b0 0x7f45fc14eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:02.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.925+0000 7f45fb59e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45fc06d260 con 0x7f45fc06bfb0 2022-04-03T19:48:02.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.925+0000 7f45fb59e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45fc0c6330 con 0x7f45fc06bbe0 2022-04-03T19:48:02.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.925+0000 7f45fb59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc159450 con 0x7f45fc1584b0 2022-04-03T19:48:02.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.925+0000 7f45fa59c700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f45fc06bbe0 0x7f45fc06d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44218/0 (socket says 172.21.15.77:44218) 2022-04-03T19:48:02.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.925+0000 7f45fa59c700 1 -- 172.21.15.77:0/3831987304 learned_addr learned my addr 172.21.15.77:0/3831987304 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:48:02.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.925+0000 7f45fad9d700 1 --2- 172.21.15.77:0/3831987304 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45fc1584b0 0x7f45fc14eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:02.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.926+0000 7f45f9d9b700 1 -- 172.21.15.77:0/3831987304 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983810176 0 0) 0x7f45fc0c6330 con 0x7f45fc06bbe0 2022-04-03T19:48:02.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.926+0000 7f45f9d9b700 1 -- 172.21.15.77:0/3831987304 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45dc0033f0 con 0x7f45fc06bbe0 2022-04-03T19:48:02.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.926+0000 7f45f9d9b700 1 -- 172.21.15.77:0/3831987304 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252117574 0 0) 0x7f45fc06d260 con 0x7f45fc06bfb0 2022-04-03T19:48:02.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.926+0000 7f45fad9d700 1 -- 172.21.15.77:0/3831987304 >> v1:172.21.15.186:6789/0 conn(0x7f45fc06bbe0 legacy=0x7f45fc06d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:02.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.926+0000 7f45fad9d700 1 -- 172.21.15.77:0/3831987304 >> v1:172.21.15.77:6789/0 conn(0x7f45fc06bfb0 legacy=0x7f45fc06c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:02.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.926+0000 7f45fad9d700 1 -- 172.21.15.77:0/3831987304 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45fc158910 con 0x7f45fc1584b0 2022-04-03T19:48:02.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.926+0000 7f45fad9d700 1 --2- 172.21.15.77:0/3831987304 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45fc1584b0 0x7f45fc14eea0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f45f4009e90 tx=0x7f45f4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba1e54afd21d56fa server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:02.933 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.926+0000 7f45f9d9b700 1 -- 172.21.15.77:0/3831987304 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f400e070 con 0x7f45fc1584b0 2022-04-03T19:48:02.934 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.926+0000 7f45f9d9b700 1 -- 172.21.15.77:0/3831987304 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45f400bc30 con 0x7f45fc1584b0 2022-04-03T19:48:02.934 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.926+0000 7f45f9d9b700 1 -- 172.21.15.77:0/3831987304 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f4012430 con 0x7f45fc1584b0 2022-04-03T19:48:02.934 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.927+0000 7f45fb59e700 1 -- 172.21.15.77:0/3831987304 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45fc1584b0 msgr2=0x7f45fc14eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:02.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.927+0000 7f45fb59e700 1 --2- 172.21.15.77:0/3831987304 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45fc1584b0 0x7f45fc14eea0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f45f4009e90 tx=0x7f45f4009780 comp rx=0 tx=0).stop 2022-04-03T19:48:02.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.927+0000 7f45fb59e700 1 -- 172.21.15.77:0/3831987304 shutdown_connections 2022-04-03T19:48:02.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.927+0000 7f45fb59e700 1 --2- 172.21.15.77:0/3831987304 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45fc1584b0 0x7f45fc14eea0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:02.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.927+0000 7f45fb59e700 1 -- 172.21.15.77:0/3831987304 >> 172.21.15.77:0/3831987304 conn(0x7f45fc0cb6b0 msgr2=0x7f45fc0cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:02.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.927+0000 7f45fb59e700 1 -- 172.21.15.77:0/3831987304 shutdown_connections 2022-04-03T19:48:02.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.927+0000 7f45fb59e700 1 -- 172.21.15.77:0/3831987304 wait complete. 2022-04-03T19:48:02.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.929+0000 7f45fb59e700 1 Processor -- start 2022-04-03T19:48:02.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.929+0000 7f45fb59e700 1 -- start start 2022-04-03T19:48:02.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.930+0000 7f45fb59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45fc06bbe0 0x7f45fc15a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:02.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.930+0000 7f45fb59e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45fc06bfb0 0x7f45fc1507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:02.937 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.930+0000 7f45fb59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45fc1584b0 0x7f45fc150ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:02.937 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.930+0000 7f45fb59e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc150360 con 0x7f45fc06bbe0 2022-04-03T19:48:02.937 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.930+0000 7f45fb59e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc0bff60 con 0x7f45fc06bfb0 2022-04-03T19:48:02.937 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.930+0000 7f45fb59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc168ae0 con 0x7f45fc1584b0 2022-04-03T19:48:02.937 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.930+0000 7f45fa59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45fc06bbe0 0x7f45fc15a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:02.938 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.930+0000 7f45fa59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45fc06bbe0 0x7f45fc15a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42822/0 (socket says 172.21.15.77:42822) 2022-04-03T19:48:02.938 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.930+0000 7f45fa59c700 1 -- 172.21.15.77:0/3196418539 learned_addr learned my addr 172.21.15.77:0/3196418539 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:02.938 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.930+0000 7f45f3fff700 1 --2- 172.21.15.77:0/3196418539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45fc06bfb0 0x7f45fc1507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:02.938 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.930+0000 7f45fad9d700 1 --2- 172.21.15.77:0/3196418539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45fc1584b0 0x7f45fc150ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:02.939 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.931+0000 7f45fa59c700 1 -- 172.21.15.77:0/3196418539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45fc1584b0 msgr2=0x7f45fc150ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:02.939 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.931+0000 7f45fa59c700 1 --2- 172.21.15.77:0/3196418539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45fc1584b0 0x7f45fc150ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:02.939 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.931+0000 7f45fa59c700 1 -- 172.21.15.77:0/3196418539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45fc06bfb0 msgr2=0x7f45fc1507a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:02.939 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.931+0000 7f45fa59c700 1 --2- 172.21.15.77:0/3196418539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45fc06bfb0 0x7f45fc1507a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:02.939 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.931+0000 7f45fa59c700 1 -- 172.21.15.77:0/3196418539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f4009a20 con 0x7f45fc06bbe0 2022-04-03T19:48:02.940 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.931+0000 7f45f3fff700 1 --2- 172.21.15.77:0/3196418539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45fc06bfb0 0x7f45fc1507a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:48:02.940 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.931+0000 7f45fa59c700 1 --2- 172.21.15.77:0/3196418539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45fc06bbe0 0x7f45fc15a9a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f45ec00c4a0 tx=0x7f45ec007c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:02.940 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.931+0000 7f45f27fc700 1 -- 172.21.15.77:0/3196418539 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec0070d0 con 0x7f45fc06bbe0 2022-04-03T19:48:02.940 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.931+0000 7f45f27fc700 1 -- 172.21.15.77:0/3196418539 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ec007230 con 0x7f45fc06bbe0 2022-04-03T19:48:02.940 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.931+0000 7f45fb59e700 1 -- 172.21.15.77:0/3196418539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45fc167ed0 con 0x7f45fc06bbe0 2022-04-03T19:48:02.941 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.931+0000 7f45f27fc700 1 -- 172.21.15.77:0/3196418539 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec0075f0 con 0x7f45fc06bbe0 2022-04-03T19:48:02.941 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.932+0000 7f45fb59e700 1 -- 172.21.15.77:0/3196418539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45fc168490 con 0x7f45fc06bbe0 2022-04-03T19:48:02.942 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.933+0000 7f45fb59e700 1 -- 172.21.15.77:0/3196418539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45fc158bb0 con 0x7f45fc06bbe0 2022-04-03T19:48:02.942 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.934+0000 7f45f27fc700 1 -- 172.21.15.77:0/3196418539 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f45ec019030 con 0x7f45fc06bbe0 2022-04-03T19:48:02.943 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.934+0000 7f45f27fc700 1 --2- 172.21.15.77:0/3196418539 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f45e0063d10 0x7f45e00661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:02.943 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.935+0000 7f45f27fc700 1 -- 172.21.15.77:0/3196418539 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6482+0+0 (secure 0 0 0) 0x7f45ec07fbc0 con 0x7f45fc06bbe0 2022-04-03T19:48:02.943 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.935+0000 7f45f3fff700 1 --2- 172.21.15.77:0/3196418539 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f45e0063d10 0x7f45e00661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:02.943 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.935+0000 7f45f3fff700 1 --2- 172.21.15.77:0/3196418539 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f45e0063d10 0x7f45e00661c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f45fc15b430 tx=0x7f45e4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:02.943 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:02.939+0000 7f45f27fc700 1 -- 172.21.15.77:0/3196418539 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f45ec04e5f0 con 0x7f45fc06bbe0 2022-04-03T19:48:03.177 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:03.175+0000 7f45fb59e700 1 -- 172.21.15.77:0/3196418539 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45fc1505f0 con 0x7f45fc06bbe0 2022-04-03T19:48:03.178 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:03.176+0000 7f45f27fc700 1 -- 172.21.15.77:0/3196418539 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6112 (secure 0 0 0) 0x7f45ec017070 con 0x7f45fc06bbe0 2022-04-03T19:48:03.178 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:03.178 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4960,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:47:10.107928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:active","state_seq":47,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:03.180 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:03.178+0000 7f45fb59e700 1 -- 172.21.15.77:0/3196418539 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f45e0063d10 msgr2=0x7f45e00661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:03.178+0000 7f45fb59e700 1 --2- 172.21.15.77:0/3196418539 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f45e0063d10 0x7f45e00661c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f45fc15b430 tx=0x7f45e4009040 comp rx=0 tx=0).stop 2022-04-03T19:48:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:03.178+0000 7f45fb59e700 1 -- 172.21.15.77:0/3196418539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45fc06bbe0 msgr2=0x7f45fc15a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:03.178+0000 7f45fb59e700 1 --2- 172.21.15.77:0/3196418539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45fc06bbe0 0x7f45fc15a9a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f45ec00c4a0 tx=0x7f45ec007c30 comp rx=0 tx=0).stop 2022-04-03T19:48:03.181 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:03.179+0000 7f45fb59e700 1 -- 172.21.15.77:0/3196418539 shutdown_connections 2022-04-03T19:48:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:03.179+0000 7f45fb59e700 1 --2- 172.21.15.77:0/3196418539 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f45e0063d10 0x7f45e00661c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:03.179+0000 7f45fb59e700 1 --2- 172.21.15.77:0/3196418539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45fc1584b0 0x7f45fc150ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:03.179+0000 7f45fb59e700 1 --2- 172.21.15.77:0/3196418539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45fc06bbe0 0x7f45fc15a9a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:03.182 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:03.179+0000 7f45fb59e700 1 --2- 172.21.15.77:0/3196418539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45fc06bfb0 0x7f45fc1507a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:03.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:03.179+0000 7f45fb59e700 1 -- 172.21.15.77:0/3196418539 >> 172.21.15.77:0/3196418539 conn(0x7f45fc0cb6b0 msgr2=0x7f45fc0cba80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:03.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:03.179+0000 7f45fb59e700 1 -- 172.21.15.77:0/3196418539 shutdown_connections 2022-04-03T19:48:03.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:03.179+0000 7f45fb59e700 1 -- 172.21.15.77:0/3196418539 wait complete. 2022-04-03T19:48:03.183 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 14 2022-04-03T19:48:03.205 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:48:05.206 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:05.416 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.414+0000 7f16bcac5700 1 Processor -- start 2022-04-03T19:48:05.416 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.414+0000 7f16bcac5700 1 -- start start 2022-04-03T19:48:05.419 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.416+0000 7f16bcac5700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f16b80c3b10 0x7f16b815d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:05.420 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.416+0000 7f16bcac5700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b80c3710 con 0x7f16b8155370 2022-04-03T19:48:05.420 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.416+0000 7f16bcac5700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b8152330 con 0x7f16b80c1170 2022-04-03T19:48:05.420 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.416+0000 7f16bcac5700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b80c1540 con 0x7f16b80c3b10 2022-04-03T19:48:05.421 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.416+0000 7f16b6ffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f16b8155370 0x7f16b8069110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38426/0 (socket says 172.21.15.77:38426) 2022-04-03T19:48:05.421 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.416+0000 7f16b6ffd700 1 -- 172.21.15.77:0/223118675 learned_addr learned my addr 172.21.15.77:0/223118675 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:48:05.421 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.416+0000 7f16b7fff700 1 --2- 172.21.15.77:0/223118675 >> v2:172.21.15.186:3300/0 conn(0x7f16b80c3b10 0x7f16b815d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:05.421 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.416+0000 7f16b67fc700 1 -- 172.21.15.77:0/223118675 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122568437 0 0) 0x7f16b8152330 con 0x7f16b80c1170 2022-04-03T19:48:05.422 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.417+0000 7f16b67fc700 1 -- 172.21.15.77:0/223118675 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a40033f0 con 0x7f16b80c1170 2022-04-03T19:48:05.422 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.417+0000 7f16b7fff700 1 -- 172.21.15.77:0/223118675 >> v1:172.21.15.186:6789/0 conn(0x7f16b80c1170 legacy=0x7f16b81527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:05.422 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.417+0000 7f16b7fff700 1 -- 172.21.15.77:0/223118675 >> v1:172.21.15.77:6789/0 conn(0x7f16b8155370 legacy=0x7f16b8069110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:05.422 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.417+0000 7f16b67fc700 1 -- 172.21.15.77:0/223118675 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561005098 0 0) 0x7f16b80c3710 con 0x7f16b8155370 2022-04-03T19:48:05.423 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.417+0000 7f16b7fff700 1 -- 172.21.15.77:0/223118675 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b80c4270 con 0x7f16b80c3b10 2022-04-03T19:48:05.423 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.417+0000 7f16b7fff700 1 --2- 172.21.15.77:0/223118675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16b80c3b10 0x7f16b815d700 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009e60 tx=0x7f16b0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c09e38e7a4de3f7c server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:05.423 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.417+0000 7f16b67fc700 1 -- 172.21.15.77:0/223118675 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b000e070 con 0x7f16b80c3b10 2022-04-03T19:48:05.423 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.417+0000 7f16b67fc700 1 -- 172.21.15.77:0/223118675 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b000bb60 con 0x7f16b80c3b10 2022-04-03T19:48:05.424 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.417+0000 7f16b67fc700 1 -- 172.21.15.77:0/223118675 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b000bee0 con 0x7f16b80c3b10 2022-04-03T19:48:05.424 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.418+0000 7f16bcac5700 1 -- 172.21.15.77:0/223118675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16b80c3b10 msgr2=0x7f16b815d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:05.424 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.418+0000 7f16bcac5700 1 --2- 172.21.15.77:0/223118675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16b80c3b10 0x7f16b815d700 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009e60 tx=0x7f16b0009780 comp rx=0 tx=0).stop 2022-04-03T19:48:05.424 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.418+0000 7f16bcac5700 1 -- 172.21.15.77:0/223118675 shutdown_connections 2022-04-03T19:48:05.425 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.418+0000 7f16bcac5700 1 --2- 172.21.15.77:0/223118675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16b80c3b10 0x7f16b815d700 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:05.425 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.418+0000 7f16bcac5700 1 -- 172.21.15.77:0/223118675 >> 172.21.15.77:0/223118675 conn(0x7f16b8066da0 msgr2=0x7f16b80671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:05.425 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.418+0000 7f16bcac5700 1 -- 172.21.15.77:0/223118675 shutdown_connections 2022-04-03T19:48:05.425 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.418+0000 7f16bcac5700 1 -- 172.21.15.77:0/223118675 wait complete. 2022-04-03T19:48:05.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.419+0000 7f16bcac5700 1 Processor -- start 2022-04-03T19:48:05.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.419+0000 7f16bcac5700 1 -- start start 2022-04-03T19:48:05.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.420+0000 7f16bcac5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16b80c1170 0x7f16b814da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:05.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.420+0000 7f16bcac5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16b80c3b10 0x7f16b814baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:05.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.420+0000 7f16bcac5700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16b8155370 0x7f16b8152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:05.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.420+0000 7f16bcac5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b81545a0 con 0x7f16b80c3b10 2022-04-03T19:48:05.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.420+0000 7f16bcac5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b81653e0 con 0x7f16b80c1170 2022-04-03T19:48:05.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.420+0000 7f16bcac5700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b8154090 con 0x7f16b8155370 2022-04-03T19:48:05.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.421+0000 7f16b6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16b80c3b10 0x7f16b814baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:05.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.421+0000 7f16b6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16b80c3b10 0x7f16b814baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42824/0 (socket says 172.21.15.77:42824) 2022-04-03T19:48:05.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.421+0000 7f16b6ffd700 1 -- 172.21.15.77:0/1144774787 learned_addr learned my addr 172.21.15.77:0/1144774787 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:05.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.421+0000 7f16b7fff700 1 --2- 172.21.15.77:0/1144774787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16b8155370 0x7f16b8152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:05.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.421+0000 7f16b77fe700 1 --2- 172.21.15.77:0/1144774787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16b80c1170 0x7f16b814da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:05.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.421+0000 7f16b6ffd700 1 -- 172.21.15.77:0/1144774787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16b8155370 msgr2=0x7f16b8152ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:05.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.421+0000 7f16b6ffd700 1 --2- 172.21.15.77:0/1144774787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16b8155370 0x7f16b8152ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:05.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.421+0000 7f16b6ffd700 1 -- 172.21.15.77:0/1144774787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16b80c1170 msgr2=0x7f16b814da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:05.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.421+0000 7f16b6ffd700 1 --2- 172.21.15.77:0/1144774787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16b80c1170 0x7f16b814da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:05.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.421+0000 7f16b6ffd700 1 -- 172.21.15.77:0/1144774787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b0009a20 con 0x7f16b80c3b10 2022-04-03T19:48:05.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.421+0000 7f16b77fe700 1 --2- 172.21.15.77:0/1144774787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16b80c1170 0x7f16b814da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:48:05.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.422+0000 7f16b6ffd700 1 --2- 172.21.15.77:0/1144774787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16b80c3b10 0x7f16b814baf0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00c470 tx=0x7f16ac007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:05.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.422+0000 7f169f7fe700 1 -- 172.21.15.77:0/1144774787 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac007150 con 0x7f16b80c3b10 2022-04-03T19:48:05.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.422+0000 7f169f7fe700 1 -- 172.21.15.77:0/1144774787 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ac0072b0 con 0x7f16b80c3b10 2022-04-03T19:48:05.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.422+0000 7f169f7fe700 1 -- 172.21.15.77:0/1144774787 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac0046b0 con 0x7f16b80c3b10 2022-04-03T19:48:05.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.422+0000 7f16bcac5700 1 -- 172.21.15.77:0/1144774787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b8164f50 con 0x7f16b80c3b10 2022-04-03T19:48:05.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.422+0000 7f16bcac5700 1 -- 172.21.15.77:0/1144774787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b8164620 con 0x7f16b80c3b10 2022-04-03T19:48:05.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.423+0000 7f16bcac5700 1 -- 172.21.15.77:0/1144774787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b80c4020 con 0x7f16b80c3b10 2022-04-03T19:48:05.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.424+0000 7f169f7fe700 1 -- 172.21.15.77:0/1144774787 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f16ac010030 con 0x7f16b80c3b10 2022-04-03T19:48:05.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.425+0000 7f169f7fe700 1 --2- 172.21.15.77:0/1144774787 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1698063ce0 0x7f1698066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:05.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.425+0000 7f169f7fe700 1 -- 172.21.15.77:0/1144774787 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6482+0+0 (secure 0 0 0) 0x7f16ac080590 con 0x7f16b80c3b10 2022-04-03T19:48:05.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.425+0000 7f16b77fe700 1 --2- 172.21.15.77:0/1144774787 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1698063ce0 0x7f1698066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:05.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.426+0000 7f16b77fe700 1 --2- 172.21.15.77:0/1144774787 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1698063ce0 0x7f1698066190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f16a8007980 tx=0x7f16a8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:05.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.430+0000 7f169f7fe700 1 -- 172.21.15.77:0/1144774787 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f16ac04efe0 con 0x7f16b80c3b10 2022-04-03T19:48:05.663 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.662+0000 7f16bcac5700 1 -- 172.21.15.77:0/1144774787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16b80c2e90 con 0x7f16b80c3b10 2022-04-03T19:48:05.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.662+0000 7f169f7fe700 1 -- 172.21.15.77:0/1144774787 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6112 (secure 0 0 0) 0x7f16ac004cb0 con 0x7f16b80c3b10 2022-04-03T19:48:05.664 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:05.664 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4960,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:47:10.107928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:active","state_seq":47,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:05.667 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.665+0000 7f16bcac5700 1 -- 172.21.15.77:0/1144774787 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1698063ce0 msgr2=0x7f1698066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:05.667 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.665+0000 7f16bcac5700 1 --2- 172.21.15.77:0/1144774787 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1698063ce0 0x7f1698066190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f16a8007980 tx=0x7f16a8009040 comp rx=0 tx=0).stop 2022-04-03T19:48:05.667 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.665+0000 7f16bcac5700 1 -- 172.21.15.77:0/1144774787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16b80c3b10 msgr2=0x7f16b814baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:05.667 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.665+0000 7f16bcac5700 1 --2- 172.21.15.77:0/1144774787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16b80c3b10 0x7f16b814baf0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00c470 tx=0x7f16ac007d90 comp rx=0 tx=0).stop 2022-04-03T19:48:05.668 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.665+0000 7f16bcac5700 1 -- 172.21.15.77:0/1144774787 shutdown_connections 2022-04-03T19:48:05.668 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.665+0000 7f16bcac5700 1 --2- 172.21.15.77:0/1144774787 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1698063ce0 0x7f1698066190 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:05.668 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.665+0000 7f16bcac5700 1 --2- 172.21.15.77:0/1144774787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16b8155370 0x7f16b8152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:05.668 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.665+0000 7f16bcac5700 1 --2- 172.21.15.77:0/1144774787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16b80c1170 0x7f16b814da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:05.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.665+0000 7f16bcac5700 1 --2- 172.21.15.77:0/1144774787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16b80c3b10 0x7f16b814baf0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:05.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.665+0000 7f16bcac5700 1 -- 172.21.15.77:0/1144774787 >> 172.21.15.77:0/1144774787 conn(0x7f16b8066da0 msgr2=0x7f16b8155780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:05.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.665+0000 7f16bcac5700 1 -- 172.21.15.77:0/1144774787 shutdown_connections 2022-04-03T19:48:05.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:05.665+0000 7f16bcac5700 1 -- 172.21.15.77:0/1144774787 wait complete. 2022-04-03T19:48:05.670 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 14 2022-04-03T19:48:05.691 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:48:07.692 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:07.898 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.896+0000 7fc4315f2700 1 Processor -- start 2022-04-03T19:48:07.898 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.896+0000 7fc4315f2700 1 -- start start 2022-04-03T19:48:07.900 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.897+0000 7fc4315f2700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc42c06d3f0 0x7fc42c06d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:07.901 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.897+0000 7fc4315f2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc42c0c2a90 0x7fc42c06c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:07.901 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.897+0000 7fc4315f2700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc42c1562f0 con 0x7fc42c0c5930 2022-04-03T19:48:07.901 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.897+0000 7fc4315f2700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc42c0c2f80 con 0x7fc42c06d3f0 2022-04-03T19:48:07.901 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.897+0000 7fc4315f2700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c0c3260 con 0x7fc42c0c2a90 2022-04-03T19:48:07.902 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.898+0000 7fc42bfff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc42c06d3f0 0x7fc42c06d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:07.902 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.898+0000 7fc430df1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc42c0c2a90 0x7fc42c06c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:07.902 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.898+0000 7fc42b7fe700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fc42c0c5930 0x7fc42c155be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44222/0 (socket says 172.21.15.77:44222) 2022-04-03T19:48:07.902 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.898+0000 7fc42b7fe700 1 -- 172.21.15.77:0/2952840231 learned_addr learned my addr 172.21.15.77:0/2952840231 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:48:07.903 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.898+0000 7fc42bfff700 1 -- 172.21.15.77:0/2952840231 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc42c0c2a90 msgr2=0x7fc42c06c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:48:07.903 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.898+0000 7fc42bfff700 1 --2- 172.21.15.77:0/2952840231 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc42c0c2a90 0x7fc42c06c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:07.903 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.898+0000 7fc42bfff700 1 -- 172.21.15.77:0/2952840231 >> v1:172.21.15.186:6789/0 conn(0x7fc42c0c5930 legacy=0x7fc42c155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:07.903 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.898+0000 7fc42affd700 1 -- 172.21.15.77:0/2952840231 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218427964 0 0) 0x7fc42c1562f0 con 0x7fc42c0c5930 2022-04-03T19:48:07.904 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.898+0000 7fc42bfff700 1 -- 172.21.15.77:0/2952840231 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42c0c3540 con 0x7fc42c06d3f0 2022-04-03T19:48:07.904 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.899+0000 7fc42bfff700 1 --2- 172.21.15.77:0/2952840231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc42c06d3f0 0x7fc42c06d7c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc41c009790 tx=0x7fc41c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9601431201eaaaf server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:07.904 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.899+0000 7fc42affd700 1 -- 172.21.15.77:0/2952840231 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41c011630 con 0x7fc42c06d3f0 2022-04-03T19:48:07.904 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.899+0000 7fc42affd700 1 -- 172.21.15.77:0/2952840231 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41c011790 con 0x7fc42c06d3f0 2022-04-03T19:48:07.904 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.899+0000 7fc42affd700 1 -- 172.21.15.77:0/2952840231 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41c011b50 con 0x7fc42c06d3f0 2022-04-03T19:48:07.909 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.899+0000 7fc4315f2700 1 -- 172.21.15.77:0/2952840231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc42c06d3f0 msgr2=0x7fc42c06d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:07.910 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.899+0000 7fc4315f2700 1 --2- 172.21.15.77:0/2952840231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc42c06d3f0 0x7fc42c06d7c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc41c009790 tx=0x7fc41c009060 comp rx=0 tx=0).stop 2022-04-03T19:48:07.910 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.900+0000 7fc4315f2700 1 -- 172.21.15.77:0/2952840231 shutdown_connections 2022-04-03T19:48:07.910 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.900+0000 7fc4315f2700 1 --2- 172.21.15.77:0/2952840231 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc42c0c2a90 0x7fc42c06c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:07.911 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.900+0000 7fc4315f2700 1 --2- 172.21.15.77:0/2952840231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc42c06d3f0 0x7fc42c06d7c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:07.911 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.900+0000 7fc4315f2700 1 -- 172.21.15.77:0/2952840231 >> 172.21.15.77:0/2952840231 conn(0x7fc42c0cb720 msgr2=0x7fc42c0cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:07.911 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.900+0000 7fc4315f2700 1 -- 172.21.15.77:0/2952840231 shutdown_connections 2022-04-03T19:48:07.912 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.900+0000 7fc4315f2700 1 -- 172.21.15.77:0/2952840231 wait complete. 2022-04-03T19:48:07.912 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.901+0000 7fc4315f2700 1 Processor -- start 2022-04-03T19:48:07.912 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.901+0000 7fc4315f2700 1 -- start start 2022-04-03T19:48:07.912 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.902+0000 7fc4315f2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc42c06d3f0 0x7fc42c1652b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:07.913 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.902+0000 7fc4315f2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc42c0c2a90 0x7fc42c166790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:07.913 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.902+0000 7fc4315f2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc42c0c5930 0x7fc42c166cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:07.913 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.902+0000 7fc4315f2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c152440 con 0x7fc42c0c5930 2022-04-03T19:48:07.913 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.902+0000 7fc4315f2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c151550 con 0x7fc42c06d3f0 2022-04-03T19:48:07.914 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.902+0000 7fc4315f2700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c151830 con 0x7fc42c0c2a90 2022-04-03T19:48:07.914 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.903+0000 7fc42b7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc42c0c2a90 0x7fc42c166790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:07.914 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.903+0000 7fc430df1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc42c0c5930 0x7fc42c166cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:07.914 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.903+0000 7fc42bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc42c06d3f0 0x7fc42c1652b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:07.915 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.903+0000 7fc42b7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc42c0c2a90 0x7fc42c166790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60118/0 (socket says 172.21.15.77:60118) 2022-04-03T19:48:07.915 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.903+0000 7fc42b7fe700 1 -- 172.21.15.77:0/1720416825 learned_addr learned my addr 172.21.15.77:0/1720416825 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:07.915 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.903+0000 7fc42bfff700 1 -- 172.21.15.77:0/1720416825 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc42c0c2a90 msgr2=0x7fc42c166790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:07.915 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.903+0000 7fc42bfff700 1 --2- 172.21.15.77:0/1720416825 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc42c0c2a90 0x7fc42c166790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:07.915 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.903+0000 7fc42bfff700 1 -- 172.21.15.77:0/1720416825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc42c0c5930 msgr2=0x7fc42c166cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:07.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.903+0000 7fc42bfff700 1 --2- 172.21.15.77:0/1720416825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc42c0c5930 0x7fc42c166cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:07.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.903+0000 7fc42bfff700 1 -- 172.21.15.77:0/1720416825 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41c0092a0 con 0x7fc42c06d3f0 2022-04-03T19:48:07.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.903+0000 7fc42b7fe700 1 --2- 172.21.15.77:0/1720416825 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc42c0c2a90 0x7fc42c166790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:48:07.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.904+0000 7fc42bfff700 1 --2- 172.21.15.77:0/1720416825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc42c06d3f0 0x7fc42c1652b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc41c016000 tx=0x7fc41c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:07.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.904+0000 7fc413fff700 1 -- 172.21.15.77:0/1720416825 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41c000bc0 con 0x7fc42c06d3f0 2022-04-03T19:48:07.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.904+0000 7fc413fff700 1 -- 172.21.15.77:0/1720416825 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc41c011900 con 0x7fc42c06d3f0 2022-04-03T19:48:07.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.904+0000 7fc413fff700 1 -- 172.21.15.77:0/1720416825 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41c025420 con 0x7fc42c06d3f0 2022-04-03T19:48:07.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.904+0000 7fc4315f2700 1 -- 172.21.15.77:0/1720416825 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc42c151b10 con 0x7fc42c06d3f0 2022-04-03T19:48:07.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.904+0000 7fc4315f2700 1 -- 172.21.15.77:0/1720416825 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc42c1520d0 con 0x7fc42c06d3f0 2022-04-03T19:48:07.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.905+0000 7fc4315f2700 1 -- 172.21.15.77:0/1720416825 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc42c153f40 con 0x7fc42c06d3f0 2022-04-03T19:48:07.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.906+0000 7fc413fff700 1 -- 172.21.15.77:0/1720416825 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fc41c00f050 con 0x7fc42c06d3f0 2022-04-03T19:48:07.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.907+0000 7fc413fff700 1 --2- 172.21.15.77:0/1720416825 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc40c063ce0 0x7fc40c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:07.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.907+0000 7fc413fff700 1 -- 172.21.15.77:0/1720416825 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6482+0+0 (secure 0 0 0) 0x7fc41c082420 con 0x7fc42c06d3f0 2022-04-03T19:48:07.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.907+0000 7fc42b7fe700 1 --2- 172.21.15.77:0/1720416825 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc40c063ce0 0x7fc40c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:07.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.908+0000 7fc42b7fe700 1 --2- 172.21.15.77:0/1720416825 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc40c063ce0 0x7fc40c066190 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc42c0c27c0 tx=0x7fc420007d90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:07.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:07.910+0000 7fc413fff700 1 -- 172.21.15.77:0/1720416825 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fc41c050db0 con 0x7fc42c06d3f0 2022-04-03T19:48:08.145 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:08.144+0000 7fc4315f2700 1 -- 172.21.15.77:0/1720416825 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc42c04b050 con 0x7fc42c06d3f0 2022-04-03T19:48:08.146 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:08.144+0000 7fc413fff700 1 -- 172.21.15.77:0/1720416825 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6112 (secure 0 0 0) 0x7fc41c0543d0 con 0x7fc42c06d3f0 2022-04-03T19:48:08.146 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:08.147 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4960,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:47:10.107928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:active","state_seq":47,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:08.149 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:08.147+0000 7fc4315f2700 1 -- 172.21.15.77:0/1720416825 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc40c063ce0 msgr2=0x7fc40c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:08.149 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:08.147+0000 7fc4315f2700 1 --2- 172.21.15.77:0/1720416825 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc40c063ce0 0x7fc40c066190 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc42c0c27c0 tx=0x7fc420007d90 comp rx=0 tx=0).stop 2022-04-03T19:48:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:08.147+0000 7fc4315f2700 1 -- 172.21.15.77:0/1720416825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc42c06d3f0 msgr2=0x7fc42c1652b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:08.147+0000 7fc4315f2700 1 --2- 172.21.15.77:0/1720416825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc42c06d3f0 0x7fc42c1652b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc41c016000 tx=0x7fc41c017000 comp rx=0 tx=0).stop 2022-04-03T19:48:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:08.147+0000 7fc4315f2700 1 -- 172.21.15.77:0/1720416825 shutdown_connections 2022-04-03T19:48:08.150 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:08.147+0000 7fc4315f2700 1 --2- 172.21.15.77:0/1720416825 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc42c0c5930 0x7fc42c166cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:08.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:08.147+0000 7fc4315f2700 1 --2- 172.21.15.77:0/1720416825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc42c06d3f0 0x7fc42c1652b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:08.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:08.147+0000 7fc4315f2700 1 --2- 172.21.15.77:0/1720416825 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc40c063ce0 0x7fc40c066190 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:08.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:08.147+0000 7fc4315f2700 1 --2- 172.21.15.77:0/1720416825 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc42c0c2a90 0x7fc42c166790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:08.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:08.147+0000 7fc4315f2700 1 -- 172.21.15.77:0/1720416825 >> 172.21.15.77:0/1720416825 conn(0x7fc42c0cb720 msgr2=0x7fc42c14d650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:08.147+0000 7fc4315f2700 1 -- 172.21.15.77:0/1720416825 shutdown_connections 2022-04-03T19:48:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:08.148+0000 7fc4315f2700 1 -- 172.21.15.77:0/1720416825 wait complete. 2022-04-03T19:48:08.152 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 14 2022-04-03T19:48:08.174 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:48:10.175 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:10.380 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.379+0000 7fa384d85700 1 Processor -- start 2022-04-03T19:48:10.381 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.379+0000 7fa384d85700 1 -- start start 2022-04-03T19:48:10.383 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.380+0000 7fa384d85700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa38006d3f0 0x7fa38006d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:10.384 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.380+0000 7fa384d85700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3801562f0 con 0x7fa3800c5930 2022-04-03T19:48:10.384 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.380+0000 7fa384d85700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3800c2f80 con 0x7fa3800c2a90 2022-04-03T19:48:10.384 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.380+0000 7fa384d85700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3800c3260 con 0x7fa38006d3f0 2022-04-03T19:48:10.384 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.381+0000 7fa377fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa3800c5930 0x7fa380155be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38428/0 (socket says 172.21.15.77:38428) 2022-04-03T19:48:10.389 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.381+0000 7fa377fff700 1 -- 172.21.15.77:0/96416223 learned_addr learned my addr 172.21.15.77:0/96416223 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:48:10.389 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.381+0000 7fa37f7fe700 1 --2- 172.21.15.77:0/96416223 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa38006d3f0 0x7fa38006d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:10.389 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.381+0000 7fa37f7fe700 1 -- 172.21.15.77:0/96416223 >> v1:172.21.15.186:6789/0 conn(0x7fa3800c2a90 legacy=0x7fa38006c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:10.389 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.381+0000 7fa37f7fe700 1 -- 172.21.15.77:0/96416223 >> v1:172.21.15.77:6789/0 conn(0x7fa3800c5930 legacy=0x7fa380155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:10.390 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.381+0000 7fa37f7fe700 1 -- 172.21.15.77:0/96416223 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3800c3540 con 0x7fa38006d3f0 2022-04-03T19:48:10.390 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.381+0000 7fa37effd700 1 -- 172.21.15.77:0/96416223 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2805812118 0 0) 0x7fa3801562f0 con 0x7fa3800c5930 2022-04-03T19:48:10.390 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.382+0000 7fa37f7fe700 1 --2- 172.21.15.77:0/96416223 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa38006d3f0 0x7fa38006d7c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa370009790 tx=0x7fa370009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf3243c91d3ad81f server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:10.390 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.382+0000 7fa37effd700 1 -- 172.21.15.77:0/96416223 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa370011630 con 0x7fa38006d3f0 2022-04-03T19:48:10.391 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.382+0000 7fa37effd700 1 -- 172.21.15.77:0/96416223 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa370011790 con 0x7fa38006d3f0 2022-04-03T19:48:10.391 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.382+0000 7fa37effd700 1 -- 172.21.15.77:0/96416223 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa370011b30 con 0x7fa38006d3f0 2022-04-03T19:48:10.394 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.382+0000 7fa384d85700 1 -- 172.21.15.77:0/96416223 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa38006d3f0 msgr2=0x7fa38006d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:10.395 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.382+0000 7fa384d85700 1 --2- 172.21.15.77:0/96416223 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa38006d3f0 0x7fa38006d7c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa370009790 tx=0x7fa370009060 comp rx=0 tx=0).stop 2022-04-03T19:48:10.395 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.383+0000 7fa384d85700 1 -- 172.21.15.77:0/96416223 shutdown_connections 2022-04-03T19:48:10.395 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.383+0000 7fa384d85700 1 --2- 172.21.15.77:0/96416223 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa38006d3f0 0x7fa38006d7c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:10.397 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.383+0000 7fa384d85700 1 -- 172.21.15.77:0/96416223 >> 172.21.15.77:0/96416223 conn(0x7fa3800cb720 msgr2=0x7fa3800cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:10.397 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.383+0000 7fa384d85700 1 -- 172.21.15.77:0/96416223 shutdown_connections 2022-04-03T19:48:10.397 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.383+0000 7fa384d85700 1 -- 172.21.15.77:0/96416223 wait complete. 2022-04-03T19:48:10.397 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.384+0000 7fa384d85700 1 Processor -- start 2022-04-03T19:48:10.398 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.384+0000 7fa384d85700 1 -- start start 2022-04-03T19:48:10.398 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.385+0000 7fa384d85700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa38006d3f0 0x7fa380165f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:10.398 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.385+0000 7fa384d85700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3800c2a90 0x7fa3800c07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:10.399 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.385+0000 7fa384d85700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3800c5930 0x7fa3800c16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:10.399 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.385+0000 7fa384d85700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa380156c80 con 0x7fa3800c2a90 2022-04-03T19:48:10.399 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.385+0000 7fa384d85700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3801664c0 con 0x7fa38006d3f0 2022-04-03T19:48:10.399 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.385+0000 7fa384d85700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa380152680 con 0x7fa3800c5930 2022-04-03T19:48:10.400 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.385+0000 7fa37f7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa38006d3f0 0x7fa380165f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:10.400 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.385+0000 7fa37ffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3800c5930 0x7fa3800c16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:10.400 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.385+0000 7fa377fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3800c2a90 0x7fa3800c07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:10.400 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.385+0000 7fa37ffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3800c5930 0x7fa3800c16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60122/0 (socket says 172.21.15.77:60122) 2022-04-03T19:48:10.400 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.385+0000 7fa37ffff700 1 -- 172.21.15.77:0/1575249117 learned_addr learned my addr 172.21.15.77:0/1575249117 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:10.401 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.385+0000 7fa37f7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa38006d3f0 0x7fa380165f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38018/0 (socket says 172.21.15.77:38018) 2022-04-03T19:48:10.401 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.386+0000 7fa37ffff700 1 -- 172.21.15.77:0/1575249117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa38006d3f0 msgr2=0x7fa380165f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:10.401 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.386+0000 7fa37ffff700 1 --2- 172.21.15.77:0/1575249117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa38006d3f0 0x7fa380165f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:10.402 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.386+0000 7fa37ffff700 1 -- 172.21.15.77:0/1575249117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3800c2a90 msgr2=0x7fa3800c07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:10.402 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.386+0000 7fa37ffff700 1 --2- 172.21.15.77:0/1575249117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3800c2a90 0x7fa3800c07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:10.402 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.386+0000 7fa37ffff700 1 -- 172.21.15.77:0/1575249117 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3700092a0 con 0x7fa3800c5930 2022-04-03T19:48:10.402 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.386+0000 7fa37ffff700 1 --2- 172.21.15.77:0/1575249117 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3800c5930 0x7fa3800c16e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa37800c7c0 tx=0x7fa378001530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:10.403 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.386+0000 7fa3777fe700 1 -- 172.21.15.77:0/1575249117 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa378003840 con 0x7fa3800c5930 2022-04-03T19:48:10.403 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.387+0000 7fa3777fe700 1 -- 172.21.15.77:0/1575249117 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3780039a0 con 0x7fa3800c5930 2022-04-03T19:48:10.403 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.387+0000 7fa3777fe700 1 -- 172.21.15.77:0/1575249117 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa378003d20 con 0x7fa3800c5930 2022-04-03T19:48:10.404 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.387+0000 7fa384d85700 1 -- 172.21.15.77:0/1575249117 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3800bfbe0 con 0x7fa3800c5930 2022-04-03T19:48:10.404 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.387+0000 7fa384d85700 1 -- 172.21.15.77:0/1575249117 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3800bfe30 con 0x7fa3800c5930 2022-04-03T19:48:10.404 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.388+0000 7fa384d85700 1 -- 172.21.15.77:0/1575249117 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa38004b050 con 0x7fa3800c5930 2022-04-03T19:48:10.404 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.388+0000 7fa3777fe700 1 -- 172.21.15.77:0/1575249117 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fa378019030 con 0x7fa3800c5930 2022-04-03T19:48:10.405 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.389+0000 7fa3777fe700 1 --2- 172.21.15.77:0/1575249117 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa364063ce0 0x7fa364066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:10.405 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.389+0000 7fa3777fe700 1 -- 172.21.15.77:0/1575249117 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6450+0+0 (secure 0 0 0) 0x7fa37807fba0 con 0x7fa3800c5930 2022-04-03T19:48:10.405 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.390+0000 7fa37f7fe700 1 --2- 172.21.15.77:0/1575249117 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa364063ce0 0x7fa364066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:10.405 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.390+0000 7fa37f7fe700 1 --2- 172.21.15.77:0/1575249117 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa364063ce0 0x7fa364066190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa3700096e0 tx=0x7fa370014000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:10.406 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.394+0000 7fa3777fe700 1 -- 172.21.15.77:0/1575249117 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fa378048ae0 con 0x7fa3800c5930 2022-04-03T19:48:10.627 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.626+0000 7fa384d85700 1 -- 172.21.15.77:0/1575249117 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa38006e1e0 con 0x7fa3800c5930 2022-04-03T19:48:10.628 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.627+0000 7fa3777fe700 1 -- 172.21.15.77:0/1575249117 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6112 (secure 0 0 0) 0x7fa378017020 con 0x7fa3800c5930 2022-04-03T19:48:10.629 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:10.629 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4960,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:47:10.107928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:active","state_seq":47,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:10.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.629+0000 7fa384d85700 1 -- 172.21.15.77:0/1575249117 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa364063ce0 msgr2=0x7fa364066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:10.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.629+0000 7fa384d85700 1 --2- 172.21.15.77:0/1575249117 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa364063ce0 0x7fa364066190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa3700096e0 tx=0x7fa370014000 comp rx=0 tx=0).stop 2022-04-03T19:48:10.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.629+0000 7fa384d85700 1 -- 172.21.15.77:0/1575249117 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3800c5930 msgr2=0x7fa3800c16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:10.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.629+0000 7fa384d85700 1 --2- 172.21.15.77:0/1575249117 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3800c5930 0x7fa3800c16e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa37800c7c0 tx=0x7fa378001530 comp rx=0 tx=0).stop 2022-04-03T19:48:10.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.629+0000 7fa384d85700 1 -- 172.21.15.77:0/1575249117 shutdown_connections 2022-04-03T19:48:10.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.629+0000 7fa384d85700 1 --2- 172.21.15.77:0/1575249117 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa364063ce0 0x7fa364066190 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:10.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.629+0000 7fa384d85700 1 --2- 172.21.15.77:0/1575249117 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3800c5930 0x7fa3800c16e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:10.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.629+0000 7fa384d85700 1 --2- 172.21.15.77:0/1575249117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa38006d3f0 0x7fa380165f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:10.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.629+0000 7fa384d85700 1 --2- 172.21.15.77:0/1575249117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3800c2a90 0x7fa3800c07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:10.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.630+0000 7fa384d85700 1 -- 172.21.15.77:0/1575249117 >> 172.21.15.77:0/1575249117 conn(0x7fa3800cb720 msgr2=0x7fa3800c8140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:10.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.630+0000 7fa384d85700 1 -- 172.21.15.77:0/1575249117 shutdown_connections 2022-04-03T19:48:10.634 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:10.630+0000 7fa384d85700 1 -- 172.21.15.77:0/1575249117 wait complete. 2022-04-03T19:48:10.634 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 14 2022-04-03T19:48:10.655 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:48:12.656 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:12.864 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.863+0000 7f8235454700 1 Processor -- start 2022-04-03T19:48:12.864 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.863+0000 7f8235454700 1 -- start start 2022-04-03T19:48:12.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.864+0000 7f8235454700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f823006bfb0 0x7f82301584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:12.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.864+0000 7f8235454700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f823006c500 0x7f823014eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:12.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.864+0000 7f8235454700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f823006d260 con 0x7f823006bbe0 2022-04-03T19:48:12.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.864+0000 7f8235454700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8230159450 con 0x7f823006c500 2022-04-03T19:48:12.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.864+0000 7f8235454700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82300c6330 con 0x7f823006bfb0 2022-04-03T19:48:12.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.865+0000 7f822ffff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f823006bbe0 0x7f823006d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38430/0 (socket says 172.21.15.77:38430) 2022-04-03T19:48:12.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.865+0000 7f822ffff700 1 -- 172.21.15.77:0/4287062639 learned_addr learned my addr 172.21.15.77:0/4287062639 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:48:12.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.865+0000 7f8234c53700 1 --2- 172.21.15.77:0/4287062639 >> v2:172.21.15.77:3300/0 conn(0x7f823006c500 0x7f823014eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:12.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.865+0000 7f822f7fe700 1 --2- 172.21.15.77:0/4287062639 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f823006bfb0 0x7f82301584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:12.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.865+0000 7f822effd700 1 -- 172.21.15.77:0/4287062639 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1620808586 0 0) 0x7f823006d260 con 0x7f823006bbe0 2022-04-03T19:48:12.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.865+0000 7f822effd700 1 -- 172.21.15.77:0/4287062639 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f821c0033f0 con 0x7f823006bbe0 2022-04-03T19:48:12.870 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.865+0000 7f822f7fe700 1 -- 172.21.15.77:0/4287062639 >> v2:172.21.15.77:3300/0 conn(0x7f823006c500 msgr2=0x7f823014eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:48:12.870 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.865+0000 7f822f7fe700 1 --2- 172.21.15.77:0/4287062639 >> v2:172.21.15.77:3300/0 conn(0x7f823006c500 0x7f823014eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:12.870 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.865+0000 7f822f7fe700 1 -- 172.21.15.77:0/4287062639 >> v1:172.21.15.77:6789/0 conn(0x7f823006bbe0 legacy=0x7f823006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:12.870 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.865+0000 7f822f7fe700 1 -- 172.21.15.77:0/4287062639 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8230158bb0 con 0x7f823006bfb0 2022-04-03T19:48:12.870 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.866+0000 7f822f7fe700 1 --2- 172.21.15.77:0/4287062639 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f823006bfb0 0x7f82301584b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8224009760 tx=0x7f8224009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5ed6cfb1568b95b server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:12.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.866+0000 7f822effd700 1 -- 172.21.15.77:0/4287062639 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82240105b0 con 0x7f823006bfb0 2022-04-03T19:48:12.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.866+0000 7f822effd700 1 -- 172.21.15.77:0/4287062639 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8224010710 con 0x7f823006bfb0 2022-04-03T19:48:12.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.866+0000 7f822effd700 1 -- 172.21.15.77:0/4287062639 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8224010a90 con 0x7f823006bfb0 2022-04-03T19:48:12.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.867+0000 7f8235454700 1 -- 172.21.15.77:0/4287062639 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f823006bfb0 msgr2=0x7f82301584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:12.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.867+0000 7f8235454700 1 --2- 172.21.15.77:0/4287062639 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f823006bfb0 0x7f82301584b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8224009760 tx=0x7f8224009060 comp rx=0 tx=0).stop 2022-04-03T19:48:12.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.867+0000 7f8235454700 1 -- 172.21.15.77:0/4287062639 shutdown_connections 2022-04-03T19:48:12.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.867+0000 7f8235454700 1 --2- 172.21.15.77:0/4287062639 >> v2:172.21.15.77:3300/0 conn(0x7f823006c500 0x7f823014eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:12.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.867+0000 7f8235454700 1 --2- 172.21.15.77:0/4287062639 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f823006bfb0 0x7f82301584b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:12.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.867+0000 7f8235454700 1 -- 172.21.15.77:0/4287062639 >> 172.21.15.77:0/4287062639 conn(0x7f82300cb6b0 msgr2=0x7f82300cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:12.873 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.867+0000 7f8235454700 1 -- 172.21.15.77:0/4287062639 shutdown_connections 2022-04-03T19:48:12.873 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.867+0000 7f8235454700 1 -- 172.21.15.77:0/4287062639 wait complete. 2022-04-03T19:48:12.876 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.868+0000 7f8235454700 1 Processor -- start 2022-04-03T19:48:12.876 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.868+0000 7f8235454700 1 -- start start 2022-04-03T19:48:12.876 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.869+0000 7f8235454700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f823006bbe0 0x7f823014a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:12.877 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.869+0000 7f8235454700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f823006bfb0 0x7f8230165f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:12.877 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.869+0000 7f8235454700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f823006c500 0x7f8230151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:12.877 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.869+0000 7f8235454700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82300c24d0 con 0x7f823006bbe0 2022-04-03T19:48:12.877 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.869+0000 7f8235454700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82300c2d70 con 0x7f823006bfb0 2022-04-03T19:48:12.878 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.869+0000 7f8235454700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8230168b40 con 0x7f823006c500 2022-04-03T19:48:12.878 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.870+0000 7f822ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f823006bbe0 0x7f823014a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:12.878 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.870+0000 7f822ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f823006bbe0 0x7f823014a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42832/0 (socket says 172.21.15.77:42832) 2022-04-03T19:48:12.878 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.870+0000 7f822ffff700 1 -- 172.21.15.77:0/2363678935 learned_addr learned my addr 172.21.15.77:0/2363678935 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:12.879 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.870+0000 7f822f7fe700 1 --2- 172.21.15.77:0/2363678935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f823006bfb0 0x7f8230165f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:12.879 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.870+0000 7f8234c53700 1 --2- 172.21.15.77:0/2363678935 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f823006c500 0x7f8230151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:12.879 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.870+0000 7f822ffff700 1 -- 172.21.15.77:0/2363678935 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f823006c500 msgr2=0x7f8230151c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:12.879 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.870+0000 7f822ffff700 1 --2- 172.21.15.77:0/2363678935 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f823006c500 0x7f8230151c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:12.879 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.870+0000 7f822ffff700 1 -- 172.21.15.77:0/2363678935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f823006bfb0 msgr2=0x7f8230165f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:12.880 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.870+0000 7f822ffff700 1 --2- 172.21.15.77:0/2363678935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f823006bfb0 0x7f8230165f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:12.880 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.870+0000 7f822ffff700 1 -- 172.21.15.77:0/2363678935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82240092a0 con 0x7f823006bbe0 2022-04-03T19:48:12.880 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.870+0000 7f822f7fe700 1 --2- 172.21.15.77:0/2363678935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f823006bfb0 0x7f8230165f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:48:12.880 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.870+0000 7f8234c53700 1 --2- 172.21.15.77:0/2363678935 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f823006c500 0x7f8230151c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:48:12.881 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.870+0000 7f822ffff700 1 --2- 172.21.15.77:0/2363678935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f823006bbe0 0x7f823014a6e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f822000c4a0 tx=0x7f8220007c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:12.881 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.871+0000 7f8217fff700 1 -- 172.21.15.77:0/2363678935 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82200072e0 con 0x7f823006bbe0 2022-04-03T19:48:12.881 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.871+0000 7f8217fff700 1 -- 172.21.15.77:0/2363678935 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8220007440 con 0x7f823006bbe0 2022-04-03T19:48:12.881 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.871+0000 7f8217fff700 1 -- 172.21.15.77:0/2363678935 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8220005780 con 0x7f823006bbe0 2022-04-03T19:48:12.881 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.871+0000 7f8235454700 1 -- 172.21.15.77:0/2363678935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82301664d0 con 0x7f823006bbe0 2022-04-03T19:48:12.882 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.871+0000 7f8235454700 1 -- 172.21.15.77:0/2363678935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82300c2720 con 0x7f823006bbe0 2022-04-03T19:48:12.882 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.872+0000 7f8235454700 1 -- 172.21.15.77:0/2363678935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82300c5d50 con 0x7f823006bbe0 2022-04-03T19:48:12.882 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.873+0000 7f8217fff700 1 -- 172.21.15.77:0/2363678935 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f8220019030 con 0x7f823006bbe0 2022-04-03T19:48:12.882 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.873+0000 7f8217fff700 1 --2- 172.21.15.77:0/2363678935 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8210063ce0 0x7f8210066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:12.882 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.874+0000 7f8217fff700 1 -- 172.21.15.77:0/2363678935 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6450+0+0 (secure 0 0 0) 0x7f822007fcd0 con 0x7f823006bbe0 2022-04-03T19:48:12.883 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.874+0000 7f822f7fe700 1 --2- 172.21.15.77:0/2363678935 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8210063ce0 0x7f8210066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:12.883 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.875+0000 7f822f7fe700 1 --2- 172.21.15.77:0/2363678935 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8210063ce0 0x7f8210066190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f823014cf90 tx=0x7f8224016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:12.883 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:12.878+0000 7f8217fff700 1 -- 172.21.15.77:0/2363678935 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f822004e6c0 con 0x7f823006bbe0 2022-04-03T19:48:13.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:13.110+0000 7f8235454700 1 -- 172.21.15.77:0/2363678935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f823004b050 con 0x7f823006bbe0 2022-04-03T19:48:13.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:13.110+0000 7f8217fff700 1 -- 172.21.15.77:0/2363678935 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6112 (secure 0 0 0) 0x7f8220017020 con 0x7f823006bbe0 2022-04-03T19:48:13.112 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:13.112 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4960,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:47:10.107928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:active","state_seq":47,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:13.114 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:13.113+0000 7f8235454700 1 -- 172.21.15.77:0/2363678935 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8210063ce0 msgr2=0x7f8210066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:13.114 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:13.113+0000 7f8235454700 1 --2- 172.21.15.77:0/2363678935 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8210063ce0 0x7f8210066190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f823014cf90 tx=0x7f8224016000 comp rx=0 tx=0).stop 2022-04-03T19:48:13.114 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:13.113+0000 7f8235454700 1 -- 172.21.15.77:0/2363678935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f823006bbe0 msgr2=0x7f823014a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:13.114 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:13.113+0000 7f8235454700 1 --2- 172.21.15.77:0/2363678935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f823006bbe0 0x7f823014a6e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f822000c4a0 tx=0x7f8220007c30 comp rx=0 tx=0).stop 2022-04-03T19:48:13.115 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:13.113+0000 7f8235454700 1 -- 172.21.15.77:0/2363678935 shutdown_connections 2022-04-03T19:48:13.115 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:13.113+0000 7f8235454700 1 --2- 172.21.15.77:0/2363678935 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8210063ce0 0x7f8210066190 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:13.115 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:13.113+0000 7f8235454700 1 --2- 172.21.15.77:0/2363678935 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f823006c500 0x7f8230151c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:13.115 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:13.113+0000 7f8235454700 1 --2- 172.21.15.77:0/2363678935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f823006bbe0 0x7f823014a6e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:13.116 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:13.113+0000 7f8235454700 1 --2- 172.21.15.77:0/2363678935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f823006bfb0 0x7f8230165f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:13.116 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:13.113+0000 7f8235454700 1 -- 172.21.15.77:0/2363678935 >> 172.21.15.77:0/2363678935 conn(0x7f82300cb6b0 msgr2=0x7f82300cba80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:13.116 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:13.113+0000 7f8235454700 1 -- 172.21.15.77:0/2363678935 shutdown_connections 2022-04-03T19:48:13.116 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:13.113+0000 7f8235454700 1 -- 172.21.15.77:0/2363678935 wait complete. 2022-04-03T19:48:13.117 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 14 2022-04-03T19:48:13.138 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:48:15.139 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:15.362 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.361+0000 7fa71359e700 1 Processor -- start 2022-04-03T19:48:15.362 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.361+0000 7fa71359e700 1 -- start start 2022-04-03T19:48:15.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.362+0000 7fa71359e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa7140c5930 0x7fa714155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:15.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.362+0000 7fa71359e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa714156120 0x7fa71406c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:15.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.362+0000 7fa71359e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7140c2f80 con 0x7fa71406d3f0 2022-04-03T19:48:15.366 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.362+0000 7fa71359e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7140c3260 con 0x7fa7140c5930 2022-04-03T19:48:15.366 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.362+0000 7fa71359e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7140c3540 con 0x7fa714156120 2022-04-03T19:48:15.366 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.363+0000 7fa71259c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa71406d3f0 0x7fa71406d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38432/0 (socket says 172.21.15.77:38432) 2022-04-03T19:48:15.366 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.363+0000 7fa71259c700 1 -- 172.21.15.77:0/2822833432 learned_addr learned my addr 172.21.15.77:0/2822833432 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:48:15.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.363+0000 7fa70bfff700 1 --2- 172.21.15.77:0/2822833432 >> v2:172.21.15.77:3300/0 conn(0x7fa7140c5930 0x7fa714155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:15.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.363+0000 7fa712d9d700 1 --2- 172.21.15.77:0/2822833432 >> v2:172.21.15.186:3300/0 conn(0x7fa714156120 0x7fa71406c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:15.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.363+0000 7fa711d9b700 1 -- 172.21.15.77:0/2822833432 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789808671 0 0) 0x7fa7140c2f80 con 0x7fa71406d3f0 2022-04-03T19:48:15.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.363+0000 7fa711d9b700 1 -- 172.21.15.77:0/2822833432 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6f40033f0 con 0x7fa71406d3f0 2022-04-03T19:48:15.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.363+0000 7fa70bfff700 1 -- 172.21.15.77:0/2822833432 >> v2:172.21.15.186:3300/0 conn(0x7fa714156120 msgr2=0x7fa71406c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:48:15.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.363+0000 7fa70bfff700 1 --2- 172.21.15.77:0/2822833432 >> v2:172.21.15.186:3300/0 conn(0x7fa714156120 0x7fa71406c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:15.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.363+0000 7fa70bfff700 1 -- 172.21.15.77:0/2822833432 >> v1:172.21.15.77:6789/0 conn(0x7fa71406d3f0 legacy=0x7fa71406d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:15.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.363+0000 7fa70bfff700 1 -- 172.21.15.77:0/2822833432 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7140c3820 con 0x7fa7140c5930 2022-04-03T19:48:15.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.364+0000 7fa70bfff700 1 --2- 172.21.15.77:0/2822833432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7140c5930 0x7fa714155be0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc009760 tx=0x7fa6fc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff66b4256207859 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:15.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.364+0000 7fa711d9b700 1 -- 172.21.15.77:0/2822833432 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc011630 con 0x7fa7140c5930 2022-04-03T19:48:15.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.364+0000 7fa711d9b700 1 -- 172.21.15.77:0/2822833432 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6fc011790 con 0x7fa7140c5930 2022-04-03T19:48:15.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.364+0000 7fa711d9b700 1 -- 172.21.15.77:0/2822833432 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc011b30 con 0x7fa7140c5930 2022-04-03T19:48:15.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.364+0000 7fa71359e700 1 -- 172.21.15.77:0/2822833432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7140c5930 msgr2=0x7fa714155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:15.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.364+0000 7fa71359e700 1 --2- 172.21.15.77:0/2822833432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7140c5930 0x7fa714155be0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc009760 tx=0x7fa6fc009060 comp rx=0 tx=0).stop 2022-04-03T19:48:15.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.365+0000 7fa71359e700 1 -- 172.21.15.77:0/2822833432 shutdown_connections 2022-04-03T19:48:15.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.365+0000 7fa71359e700 1 --2- 172.21.15.77:0/2822833432 >> v2:172.21.15.186:3300/0 conn(0x7fa714156120 0x7fa71406c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:15.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.365+0000 7fa71359e700 1 --2- 172.21.15.77:0/2822833432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7140c5930 0x7fa714155be0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:15.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.365+0000 7fa71359e700 1 -- 172.21.15.77:0/2822833432 >> 172.21.15.77:0/2822833432 conn(0x7fa7140cb720 msgr2=0x7fa7140cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:15.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.365+0000 7fa71359e700 1 -- 172.21.15.77:0/2822833432 shutdown_connections 2022-04-03T19:48:15.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.365+0000 7fa71359e700 1 -- 172.21.15.77:0/2822833432 wait complete. 2022-04-03T19:48:15.374 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.366+0000 7fa71359e700 1 Processor -- start 2022-04-03T19:48:15.374 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.366+0000 7fa71359e700 1 -- start start 2022-04-03T19:48:15.375 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.367+0000 7fa71359e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa71406d3f0 0x7fa714152260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:15.375 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.367+0000 7fa71359e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7140c5930 0x7fa714157af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:15.375 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.367+0000 7fa71359e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa714156120 0x7fa714168c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:15.375 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.367+0000 7fa71359e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa714158030 con 0x7fa7140c5930 2022-04-03T19:48:15.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.367+0000 7fa71359e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa714156980 con 0x7fa714156120 2022-04-03T19:48:15.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.367+0000 7fa71359e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa714154490 con 0x7fa71406d3f0 2022-04-03T19:48:15.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.368+0000 7fa70bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7140c5930 0x7fa714157af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:15.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.368+0000 7fa70bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7140c5930 0x7fa714157af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42836/0 (socket says 172.21.15.77:42836) 2022-04-03T19:48:15.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.368+0000 7fa70bfff700 1 -- 172.21.15.77:0/1192264861 learned_addr learned my addr 172.21.15.77:0/1192264861 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:15.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.368+0000 7fa71259c700 1 --2- 172.21.15.77:0/1192264861 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa71406d3f0 0x7fa714152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:15.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.368+0000 7fa712d9d700 1 --2- 172.21.15.77:0/1192264861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa714156120 0x7fa714168c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:15.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.368+0000 7fa70bfff700 1 -- 172.21.15.77:0/1192264861 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa71406d3f0 msgr2=0x7fa714152260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:15.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.368+0000 7fa70bfff700 1 --2- 172.21.15.77:0/1192264861 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa71406d3f0 0x7fa714152260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:15.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.368+0000 7fa70bfff700 1 -- 172.21.15.77:0/1192264861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa714156120 msgr2=0x7fa714168c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:15.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.368+0000 7fa70bfff700 1 --2- 172.21.15.77:0/1192264861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa714156120 0x7fa714168c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:15.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.368+0000 7fa70bfff700 1 -- 172.21.15.77:0/1192264861 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6fc0092a0 con 0x7fa7140c5930 2022-04-03T19:48:15.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.368+0000 7fa70bfff700 1 --2- 172.21.15.77:0/1192264861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7140c5930 0x7fa714157af0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc013000 tx=0x7fa6fc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:15.379 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.369+0000 7fa70a7fc700 1 -- 172.21.15.77:0/1192264861 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc0036f0 con 0x7fa7140c5930 2022-04-03T19:48:15.379 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.369+0000 7fa70a7fc700 1 -- 172.21.15.77:0/1192264861 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6fc003850 con 0x7fa7140c5930 2022-04-03T19:48:15.379 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.369+0000 7fa70a7fc700 1 -- 172.21.15.77:0/1192264861 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc003c10 con 0x7fa7140c5930 2022-04-03T19:48:15.379 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.369+0000 7fa71359e700 1 -- 172.21.15.77:0/1192264861 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa714151dd0 con 0x7fa7140c5930 2022-04-03T19:48:15.379 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.369+0000 7fa71359e700 1 -- 172.21.15.77:0/1192264861 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7141517c0 con 0x7fa7140c5930 2022-04-03T19:48:15.380 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.370+0000 7fa71359e700 1 -- 172.21.15.77:0/1192264861 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa714151f60 con 0x7fa7140c5930 2022-04-03T19:48:15.380 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.371+0000 7fa70a7fc700 1 -- 172.21.15.77:0/1192264861 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fa6fc00f050 con 0x7fa7140c5930 2022-04-03T19:48:15.380 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.372+0000 7fa70a7fc700 1 --2- 172.21.15.77:0/1192264861 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa6f8063d10 0x7fa6f80661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:15.380 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.372+0000 7fa70a7fc700 1 -- 172.21.15.77:0/1192264861 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6310+0+0 (secure 0 0 0) 0x7fa6fc081d80 con 0x7fa7140c5930 2022-04-03T19:48:15.381 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.372+0000 7fa71259c700 1 --2- 172.21.15.77:0/1192264861 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa6f8063d10 0x7fa6f80661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:15.381 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.373+0000 7fa71259c700 1 --2- 172.21.15.77:0/1192264861 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa6f8063d10 0x7fa6f80661c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa704005750 tx=0x7fa704009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:15.381 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.377+0000 7fa70a7fc700 1 -- 172.21.15.77:0/1192264861 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fa6fc050860 con 0x7fa7140c5930 2022-04-03T19:48:15.526 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:48:15.530 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:48:15.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.615+0000 7fa71359e700 1 -- 172.21.15.77:0/1192264861 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa714155ae0 con 0x7fa7140c5930 2022-04-03T19:48:15.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.616+0000 7fa70a7fc700 1 -- 172.21.15.77:0/1192264861 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6112 (secure 0 0 0) 0x7fa6fc083090 con 0x7fa7140c5930 2022-04-03T19:48:15.616 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:15.617 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4960,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:47:10.107928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:active","state_seq":47,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:15.619 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.618+0000 7fa71359e700 1 -- 172.21.15.77:0/1192264861 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa6f8063d10 msgr2=0x7fa6f80661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:15.619 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.618+0000 7fa71359e700 1 --2- 172.21.15.77:0/1192264861 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa6f8063d10 0x7fa6f80661c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa704005750 tx=0x7fa704009040 comp rx=0 tx=0).stop 2022-04-03T19:48:15.619 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.618+0000 7fa71359e700 1 -- 172.21.15.77:0/1192264861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7140c5930 msgr2=0x7fa714157af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:15.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.618+0000 7fa71359e700 1 --2- 172.21.15.77:0/1192264861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7140c5930 0x7fa714157af0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc013000 tx=0x7fa6fc017000 comp rx=0 tx=0).stop 2022-04-03T19:48:15.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.618+0000 7fa71359e700 1 -- 172.21.15.77:0/1192264861 shutdown_connections 2022-04-03T19:48:15.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.618+0000 7fa71359e700 1 --2- 172.21.15.77:0/1192264861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa714156120 0x7fa714168c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:15.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.618+0000 7fa71359e700 1 --2- 172.21.15.77:0/1192264861 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa6f8063d10 0x7fa6f80661c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:15.621 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.618+0000 7fa71359e700 1 --2- 172.21.15.77:0/1192264861 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa71406d3f0 0x7fa714152260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:15.621 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.618+0000 7fa71359e700 1 --2- 172.21.15.77:0/1192264861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa7140c5930 0x7fa714157af0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:15.621 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.618+0000 7fa71359e700 1 -- 172.21.15.77:0/1192264861 >> 172.21.15.77:0/1192264861 conn(0x7fa7140cb720 msgr2=0x7fa7141531f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:15.621 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.618+0000 7fa71359e700 1 -- 172.21.15.77:0/1192264861 shutdown_connections 2022-04-03T19:48:15.621 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:15.619+0000 7fa71359e700 1 -- 172.21.15.77:0/1192264861 wait complete. 2022-04-03T19:48:15.622 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 14 2022-04-03T19:48:15.643 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:48:17.644 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:17.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.848+0000 7f0cd539f700 1 Processor -- start 2022-04-03T19:48:17.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.848+0000 7f0cd539f700 1 -- start start 2022-04-03T19:48:17.852 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.850+0000 7f0cd539f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cd006c410 0x7f0cd006c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:17.852 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.850+0000 7f0cd539f700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0cd0069a00 0x7f0cd0069e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:17.852 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.850+0000 7f0cd539f700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd006bd20 con 0x7f0cd0155cf0 2022-04-03T19:48:17.852 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.850+0000 7f0cd539f700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd006d600 con 0x7f0cd0069a00 2022-04-03T19:48:17.853 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.850+0000 7f0cd539f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd01568a0 con 0x7f0cd006c410 2022-04-03T19:48:17.853 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.850+0000 7f0cd4b9e700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0cd0155cf0 0x7f0cd0156190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38434/0 (socket says 172.21.15.77:38434) 2022-04-03T19:48:17.853 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.850+0000 7f0cd4b9e700 1 -- 172.21.15.77:0/595592492 learned_addr learned my addr 172.21.15.77:0/595592492 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:48:17.853 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.850+0000 7f0ccffff700 1 --2- 172.21.15.77:0/595592492 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cd006c410 0x7f0cd006c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:17.854 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.850+0000 7f0ccf7fe700 1 --2- 172.21.15.77:0/595592492 >> v2:172.21.15.186:3300/0 conn(0x7f0cd0069a00 0x7f0cd0069e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:17.854 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.850+0000 7f0cceffd700 1 -- 172.21.15.77:0/595592492 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190268000 0 0) 0x7f0cd006bd20 con 0x7f0cd0155cf0 2022-04-03T19:48:17.854 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.851+0000 7f0cceffd700 1 -- 172.21.15.77:0/595592492 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cbc0033f0 con 0x7f0cd0155cf0 2022-04-03T19:48:17.854 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.851+0000 7f0ccffff700 1 -- 172.21.15.77:0/595592492 >> v2:172.21.15.186:3300/0 conn(0x7f0cd0069a00 msgr2=0x7f0cd0069e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:48:17.855 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.851+0000 7f0ccffff700 1 --2- 172.21.15.77:0/595592492 >> v2:172.21.15.186:3300/0 conn(0x7f0cd0069a00 0x7f0cd0069e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:17.855 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.851+0000 7f0ccffff700 1 -- 172.21.15.77:0/595592492 >> v1:172.21.15.77:6789/0 conn(0x7f0cd0155cf0 legacy=0x7f0cd0156190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:17.855 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.851+0000 7f0cceffd700 1 -- 172.21.15.77:0/595592492 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1342054313 0 0) 0x7f0cbc0033f0 con 0x7f0cd0155cf0 2022-04-03T19:48:17.855 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.851+0000 7f0ccffff700 1 -- 172.21.15.77:0/595592492 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd00c2bf0 con 0x7f0cd006c410 2022-04-03T19:48:17.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.851+0000 7f0ccffff700 1 --2- 172.21.15.77:0/595592492 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cd006c410 0x7f0cd006c7e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0009760 tx=0x7f0cc0009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64bc64d9861222f2 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:17.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.851+0000 7f0cceffd700 1 -- 172.21.15.77:0/595592492 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0011630 con 0x7f0cd006c410 2022-04-03T19:48:17.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.851+0000 7f0cceffd700 1 -- 172.21.15.77:0/595592492 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc0011790 con 0x7f0cd006c410 2022-04-03T19:48:17.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.851+0000 7f0cceffd700 1 -- 172.21.15.77:0/595592492 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0011b30 con 0x7f0cd006c410 2022-04-03T19:48:17.860 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.852+0000 7f0cd539f700 1 -- 172.21.15.77:0/595592492 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cd006c410 msgr2=0x7f0cd006c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:17.860 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.852+0000 7f0cd539f700 1 --2- 172.21.15.77:0/595592492 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cd006c410 0x7f0cd006c7e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0009760 tx=0x7f0cc0009060 comp rx=0 tx=0).stop 2022-04-03T19:48:17.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.852+0000 7f0cd539f700 1 -- 172.21.15.77:0/595592492 shutdown_connections 2022-04-03T19:48:17.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.852+0000 7f0cd539f700 1 --2- 172.21.15.77:0/595592492 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cd006c410 0x7f0cd006c7e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:17.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.852+0000 7f0cd539f700 1 --2- 172.21.15.77:0/595592492 >> v2:172.21.15.186:3300/0 conn(0x7f0cd0069a00 0x7f0cd0069e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:17.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.852+0000 7f0cd539f700 1 -- 172.21.15.77:0/595592492 >> 172.21.15.77:0/595592492 conn(0x7f0cd00c7b60 msgr2=0x7f0cd00c7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:17.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.852+0000 7f0cd539f700 1 -- 172.21.15.77:0/595592492 shutdown_connections 2022-04-03T19:48:17.862 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.852+0000 7f0cd539f700 1 -- 172.21.15.77:0/595592492 wait complete. 2022-04-03T19:48:17.862 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.854+0000 7f0cd539f700 1 Processor -- start 2022-04-03T19:48:17.862 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.854+0000 7f0cd539f700 1 -- start start 2022-04-03T19:48:17.862 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.855+0000 7f0cd539f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cd0069a00 0x7f0cd0166220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:17.863 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.855+0000 7f0cd539f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cd006c410 0x7f0cd0154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:17.863 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.855+0000 7f0cd539f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cd0155cf0 0x7f0cd014be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:17.863 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.855+0000 7f0cd539f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd00beef0 con 0x7f0cd0155cf0 2022-04-03T19:48:17.864 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.855+0000 7f0cd539f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd00be930 con 0x7f0cd0069a00 2022-04-03T19:48:17.864 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.855+0000 7f0cd539f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd00bec10 con 0x7f0cd006c410 2022-04-03T19:48:17.864 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.855+0000 7f0cd4b9e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cd0155cf0 0x7f0cd014be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:17.865 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.855+0000 7f0ccffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cd0069a00 0x7f0cd0166220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:17.865 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.855+0000 7f0ccf7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cd006c410 0x7f0cd0154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:17.866 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.855+0000 7f0ccf7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cd006c410 0x7f0cd0154190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60132/0 (socket says 172.21.15.77:60132) 2022-04-03T19:48:17.866 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.855+0000 7f0cd4b9e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cd0155cf0 0x7f0cd014be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42838/0 (socket says 172.21.15.77:42838) 2022-04-03T19:48:17.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.855+0000 7f0ccf7fe700 1 -- 172.21.15.77:0/2750863889 learned_addr learned my addr 172.21.15.77:0/2750863889 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:17.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.855+0000 7f0ccffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cd0069a00 0x7f0cd0166220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38032/0 (socket says 172.21.15.77:38032) 2022-04-03T19:48:17.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.856+0000 7f0cd4b9e700 1 -- 172.21.15.77:0/2750863889 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cd006c410 msgr2=0x7f0cd0154190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:17.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.856+0000 7f0cd4b9e700 1 --2- 172.21.15.77:0/2750863889 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cd006c410 0x7f0cd0154190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:17.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.856+0000 7f0cd4b9e700 1 -- 172.21.15.77:0/2750863889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cd0069a00 msgr2=0x7f0cd0166220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:17.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.856+0000 7f0cd4b9e700 1 --2- 172.21.15.77:0/2750863889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cd0069a00 0x7f0cd0166220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:17.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.856+0000 7f0cd4b9e700 1 -- 172.21.15.77:0/2750863889 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc00092a0 con 0x7f0cd0155cf0 2022-04-03T19:48:17.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.856+0000 7f0cd4b9e700 1 --2- 172.21.15.77:0/2750863889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cd0155cf0 0x7f0cd014be30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0cc800c810 tx=0x7f0cc8007e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:17.870 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.856+0000 7f0cb7fff700 1 -- 172.21.15.77:0/2750863889 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8005d30 con 0x7f0cd0155cf0 2022-04-03T19:48:17.870 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.856+0000 7f0cb7fff700 1 -- 172.21.15.77:0/2750863889 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc8005e90 con 0x7f0cd0155cf0 2022-04-03T19:48:17.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.856+0000 7f0cd539f700 1 -- 172.21.15.77:0/2750863889 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd0152360 con 0x7f0cd0155cf0 2022-04-03T19:48:17.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.856+0000 7f0cb7fff700 1 -- 172.21.15.77:0/2750863889 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8007300 con 0x7f0cd0155cf0 2022-04-03T19:48:17.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.856+0000 7f0cd539f700 1 -- 172.21.15.77:0/2750863889 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd00ad760 con 0x7f0cd0155cf0 2022-04-03T19:48:17.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.857+0000 7f0cd539f700 1 -- 172.21.15.77:0/2750863889 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd00ad090 con 0x7f0cd0155cf0 2022-04-03T19:48:17.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.858+0000 7f0cb7fff700 1 -- 172.21.15.77:0/2750863889 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f0cc8019030 con 0x7f0cd0155cf0 2022-04-03T19:48:17.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.859+0000 7f0cb7fff700 1 --2- 172.21.15.77:0/2750863889 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0cb0063d10 0x7f0cb00661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:17.873 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.859+0000 7f0cb7fff700 1 -- 172.21.15.77:0/2750863889 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6310+0+0 (secure 0 0 0) 0x7f0cc807fc60 con 0x7f0cd0155cf0 2022-04-03T19:48:17.873 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.859+0000 7f0ccffff700 1 --2- 172.21.15.77:0/2750863889 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0cb0063d10 0x7f0cb00661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:17.874 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.860+0000 7f0ccffff700 1 --2- 172.21.15.77:0/2750863889 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0cb0063d10 0x7f0cb00661c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0009760 tx=0x7f0cc0017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:17.874 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:17.863+0000 7f0cb7fff700 1 -- 172.21.15.77:0/2750863889 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f0cc804e6e0 con 0x7f0cd0155cf0 2022-04-03T19:48:18.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:18.096+0000 7f0cd539f700 1 -- 172.21.15.77:0/2750863889 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cd0152fa0 con 0x7f0cd0155cf0 2022-04-03T19:48:18.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:18.096+0000 7f0cb7fff700 1 -- 172.21.15.77:0/2750863889 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+6112 (secure 0 0 0) 0x7f0cc8017070 con 0x7f0cd0155cf0 2022-04-03T19:48:18.097 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:18.097 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4960,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:47:10.107928+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4303":{"gid":4303,"name":"d","rank":0,"incarnation":9,"state":"up:active","state_seq":47,"addr":"172.21.15.186:6837/1407029827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":1407029827},{"type":"v1","addr":"172.21.15.186:6837","nonce":1407029827}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:18.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:18.099+0000 7f0cd539f700 1 -- 172.21.15.77:0/2750863889 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0cb0063d10 msgr2=0x7f0cb00661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:18.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:18.099+0000 7f0cd539f700 1 --2- 172.21.15.77:0/2750863889 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0cb0063d10 0x7f0cb00661c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0009760 tx=0x7f0cc0017000 comp rx=0 tx=0).stop 2022-04-03T19:48:18.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:18.099+0000 7f0cd539f700 1 -- 172.21.15.77:0/2750863889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cd0155cf0 msgr2=0x7f0cd014be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:18.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:18.099+0000 7f0cd539f700 1 --2- 172.21.15.77:0/2750863889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cd0155cf0 0x7f0cd014be30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0cc800c810 tx=0x7f0cc8007e30 comp rx=0 tx=0).stop 2022-04-03T19:48:18.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:18.099+0000 7f0cd539f700 1 -- 172.21.15.77:0/2750863889 shutdown_connections 2022-04-03T19:48:18.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:18.099+0000 7f0cd539f700 1 --2- 172.21.15.77:0/2750863889 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cd0155cf0 0x7f0cd014be30 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:18.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:18.099+0000 7f0cd539f700 1 --2- 172.21.15.77:0/2750863889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cd0069a00 0x7f0cd0166220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:18.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:18.099+0000 7f0cd539f700 1 --2- 172.21.15.77:0/2750863889 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0cb0063d10 0x7f0cb00661c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:18.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:18.099+0000 7f0cd539f700 1 --2- 172.21.15.77:0/2750863889 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cd006c410 0x7f0cd0154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:18.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:18.099+0000 7f0cd539f700 1 -- 172.21.15.77:0/2750863889 >> 172.21.15.77:0/2750863889 conn(0x7f0cd00c7b60 msgr2=0x7f0cd00c1040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:18.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:18.099+0000 7f0cd539f700 1 -- 172.21.15.77:0/2750863889 shutdown_connections 2022-04-03T19:48:18.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:18.099+0000 7f0cd539f700 1 -- 172.21.15.77:0/2750863889 wait complete. 2022-04-03T19:48:18.103 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 14 2022-04-03T19:48:18.123 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:48:20.124 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:20.330 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.330+0000 7fa58359e700 1 Processor -- start 2022-04-03T19:48:20.331 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.330+0000 7fa58359e700 1 -- start start 2022-04-03T19:48:20.333 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.331+0000 7fa58359e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa58406d3f0 0x7fa58406d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:20.334 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.331+0000 7fa58359e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5840c5930 0x7fa584155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:20.334 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.331+0000 7fa58359e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5840c2f80 con 0x7fa584156120 2022-04-03T19:48:20.334 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.331+0000 7fa58359e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5840c3260 con 0x7fa58406d3f0 2022-04-03T19:48:20.335 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.331+0000 7fa58359e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5840c3540 con 0x7fa5840c5930 2022-04-03T19:48:20.335 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.331+0000 7fa58259c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa58406d3f0 0x7fa58406d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:20.335 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.331+0000 7fa582d9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa584156120 0x7fa58406c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38436/0 (socket says 172.21.15.77:38436) 2022-04-03T19:48:20.335 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.331+0000 7fa582d9d700 1 -- 172.21.15.77:0/2698510944 learned_addr learned my addr 172.21.15.77:0/2698510944 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:48:20.335 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.332+0000 7fa57bfff700 1 --2- 172.21.15.77:0/2698510944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5840c5930 0x7fa584155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:20.336 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.332+0000 7fa58259c700 1 -- 172.21.15.77:0/2698510944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5840c5930 msgr2=0x7fa584155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:48:20.336 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.332+0000 7fa58259c700 1 --2- 172.21.15.77:0/2698510944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5840c5930 0x7fa584155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:20.336 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.332+0000 7fa58259c700 1 -- 172.21.15.77:0/2698510944 >> v1:172.21.15.77:6789/0 conn(0x7fa584156120 legacy=0x7fa58406c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:20.337 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.332+0000 7fa58259c700 1 -- 172.21.15.77:0/2698510944 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5840c3820 con 0x7fa58406d3f0 2022-04-03T19:48:20.337 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.332+0000 7fa57bfff700 1 --2- 172.21.15.77:0/2698510944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5840c5930 0x7fa584155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:48:20.337 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.332+0000 7fa58259c700 1 --2- 172.21.15.77:0/2698510944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa58406d3f0 0x7fa58406d7c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa574009790 tx=0x7fa574009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ef5d2b1dde489b server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:20.337 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.332+0000 7fa581d9b700 1 -- 172.21.15.77:0/2698510944 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa574011630 con 0x7fa58406d3f0 2022-04-03T19:48:20.338 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.333+0000 7fa581d9b700 1 -- 172.21.15.77:0/2698510944 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa574011790 con 0x7fa58406d3f0 2022-04-03T19:48:20.338 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.333+0000 7fa581d9b700 1 -- 172.21.15.77:0/2698510944 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa574011b50 con 0x7fa58406d3f0 2022-04-03T19:48:20.338 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.333+0000 7fa58359e700 1 -- 172.21.15.77:0/2698510944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa58406d3f0 msgr2=0x7fa58406d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:20.338 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.333+0000 7fa58359e700 1 --2- 172.21.15.77:0/2698510944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa58406d3f0 0x7fa58406d7c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa574009790 tx=0x7fa574009060 comp rx=0 tx=0).stop 2022-04-03T19:48:20.339 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.333+0000 7fa58359e700 1 -- 172.21.15.77:0/2698510944 shutdown_connections 2022-04-03T19:48:20.339 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.333+0000 7fa58359e700 1 --2- 172.21.15.77:0/2698510944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa58406d3f0 0x7fa58406d7c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:20.339 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.333+0000 7fa58359e700 1 --2- 172.21.15.77:0/2698510944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5840c5930 0x7fa584155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:20.339 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.333+0000 7fa58359e700 1 -- 172.21.15.77:0/2698510944 >> 172.21.15.77:0/2698510944 conn(0x7fa5840cb720 msgr2=0x7fa5840cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:20.340 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.333+0000 7fa58359e700 1 -- 172.21.15.77:0/2698510944 shutdown_connections 2022-04-03T19:48:20.340 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.333+0000 7fa58359e700 1 -- 172.21.15.77:0/2698510944 wait complete. 2022-04-03T19:48:20.343 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.335+0000 7fa58359e700 1 Processor -- start 2022-04-03T19:48:20.343 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.335+0000 7fa58359e700 1 -- start start 2022-04-03T19:48:20.343 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.336+0000 7fa58359e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa58406d3f0 0x7fa5840c1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:20.344 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.336+0000 7fa58359e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5840c5930 0x7fa5841510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:20.344 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.336+0000 7fa58359e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa584156120 0x7fa5841515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:20.344 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.336+0000 7fa58359e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5840bffc0 con 0x7fa5840c5930 2022-04-03T19:48:20.344 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.336+0000 7fa58359e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa584150e70 con 0x7fa58406d3f0 2022-04-03T19:48:20.345 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.336+0000 7fa58359e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa584150860 con 0x7fa584156120 2022-04-03T19:48:20.345 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.336+0000 7fa57bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5840c5930 0x7fa5841510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:20.345 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.336+0000 7fa57bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5840c5930 0x7fa5841510b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42842/0 (socket says 172.21.15.77:42842) 2022-04-03T19:48:20.345 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.336+0000 7fa57bfff700 1 -- 172.21.15.77:0/4244039130 learned_addr learned my addr 172.21.15.77:0/4244039130 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:20.346 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.336+0000 7fa58259c700 1 --2- 172.21.15.77:0/4244039130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa58406d3f0 0x7fa5840c1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:20.346 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.336+0000 7fa582d9d700 1 --2- 172.21.15.77:0/4244039130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa584156120 0x7fa5841515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:20.346 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.336+0000 7fa57bfff700 1 -- 172.21.15.77:0/4244039130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa584156120 msgr2=0x7fa5841515f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:20.346 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.336+0000 7fa57bfff700 1 --2- 172.21.15.77:0/4244039130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa584156120 0x7fa5841515f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:20.347 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.336+0000 7fa57bfff700 1 -- 172.21.15.77:0/4244039130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa58406d3f0 msgr2=0x7fa5840c1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:20.347 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.336+0000 7fa57bfff700 1 --2- 172.21.15.77:0/4244039130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa58406d3f0 0x7fa5840c1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:20.347 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.336+0000 7fa57bfff700 1 -- 172.21.15.77:0/4244039130 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5740092a0 con 0x7fa5840c5930 2022-04-03T19:48:20.347 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.337+0000 7fa57bfff700 1 --2- 172.21.15.77:0/4244039130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5840c5930 0x7fa5841510b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00d860 tx=0x7fa56c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:20.347 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.337+0000 7fa57a7fc700 1 -- 172.21.15.77:0/4244039130 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c012630 con 0x7fa5840c5930 2022-04-03T19:48:20.348 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.337+0000 7fa57a7fc700 1 -- 172.21.15.77:0/4244039130 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56c012790 con 0x7fa5840c5930 2022-04-03T19:48:20.348 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.337+0000 7fa57a7fc700 1 -- 172.21.15.77:0/4244039130 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c012b10 con 0x7fa5840c5930 2022-04-03T19:48:20.348 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.337+0000 7fa58359e700 1 -- 172.21.15.77:0/4244039130 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa584150270 con 0x7fa5840c5930 2022-04-03T19:48:20.348 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.337+0000 7fa58359e700 1 -- 172.21.15.77:0/4244039130 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5841529c0 con 0x7fa5840c5930 2022-04-03T19:48:20.348 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.338+0000 7fa58359e700 1 -- 172.21.15.77:0/4244039130 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa584151dd0 con 0x7fa5840c5930 2022-04-03T19:48:20.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.339+0000 7fa57a7fc700 1 -- 172.21.15.77:0/4244039130 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fa56c01e460 con 0x7fa5840c5930 2022-04-03T19:48:20.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.340+0000 7fa57a7fc700 1 --2- 172.21.15.77:0/4244039130 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa568063ce0 0x7fa568066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:20.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.340+0000 7fa57a7fc700 1 -- 172.21.15.77:0/4244039130 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6288+0+0 (secure 0 0 0) 0x7fa56c081000 con 0x7fa5840c5930 2022-04-03T19:48:20.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.340+0000 7fa58259c700 1 --2- 172.21.15.77:0/4244039130 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa568063ce0 0x7fa568066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:20.350 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.341+0000 7fa58259c700 1 --2- 172.21.15.77:0/4244039130 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa568063ce0 0x7fa568066190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa574009fd0 tx=0x7fa574017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:20.350 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.344+0000 7fa57a7fc700 1 -- 172.21.15.77:0/4244039130 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fa56c04faa0 con 0x7fa5840c5930 2022-04-03T19:48:20.573 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.573+0000 7fa58359e700 1 -- 172.21.15.77:0/4244039130 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5840c44e0 con 0x7fa5840c5930 2022-04-03T19:48:20.574 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.573+0000 7fa57a7fc700 1 -- 172.21.15.77:0/4244039130 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5352 (secure 0 0 0) 0x7fa56c0530c0 con 0x7fa5840c5930 2022-04-03T19:48:20.574 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:20.575 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:48:19.127461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4960},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4960":{"gid":4960,"name":"e","rank":0,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:20.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.576+0000 7fa58359e700 1 -- 172.21.15.77:0/4244039130 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa568063ce0 msgr2=0x7fa568066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:20.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.576+0000 7fa58359e700 1 --2- 172.21.15.77:0/4244039130 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa568063ce0 0x7fa568066190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa574009fd0 tx=0x7fa574017000 comp rx=0 tx=0).stop 2022-04-03T19:48:20.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.576+0000 7fa58359e700 1 -- 172.21.15.77:0/4244039130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5840c5930 msgr2=0x7fa5841510b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:20.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.576+0000 7fa58359e700 1 --2- 172.21.15.77:0/4244039130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5840c5930 0x7fa5841510b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00d860 tx=0x7fa56c008040 comp rx=0 tx=0).stop 2022-04-03T19:48:20.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.576+0000 7fa58359e700 1 -- 172.21.15.77:0/4244039130 shutdown_connections 2022-04-03T19:48:20.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.576+0000 7fa58359e700 1 --2- 172.21.15.77:0/4244039130 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa568063ce0 0x7fa568066190 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:20.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.576+0000 7fa58359e700 1 --2- 172.21.15.77:0/4244039130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa584156120 0x7fa5841515f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:20.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.576+0000 7fa58359e700 1 --2- 172.21.15.77:0/4244039130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa58406d3f0 0x7fa5840c1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:20.579 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.576+0000 7fa58359e700 1 --2- 172.21.15.77:0/4244039130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa5840c5930 0x7fa5841510b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:20.579 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.576+0000 7fa58359e700 1 -- 172.21.15.77:0/4244039130 >> 172.21.15.77:0/4244039130 conn(0x7fa5840cb720 msgr2=0x7fa584154b60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:20.579 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.576+0000 7fa58359e700 1 -- 172.21.15.77:0/4244039130 shutdown_connections 2022-04-03T19:48:20.579 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.576+0000 7fa58359e700 1 -- 172.21.15.77:0/4244039130 wait complete. 2022-04-03T19:48:20.580 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 15 2022-04-03T19:48:20.600 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2022-04-03T19:48:20.601 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-04-03T19:48:20.601 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:20.798 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.798+0000 7f7f86a5a700 1 Processor -- start 2022-04-03T19:48:20.799 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.798+0000 7f7f86a5a700 1 -- start start 2022-04-03T19:48:20.800 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.799+0000 7f7f86a5a700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7f8006d3f0 0x7f7f8006d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:20.800 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.799+0000 7f7f86a5a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f800c5930 0x7f7f80155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:20.800 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.799+0000 7f7f86a5a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7f80156120 0x7f7f8006c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:20.801 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.799+0000 7f7f86a5a700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f800c2f80 con 0x7f7f80156120 2022-04-03T19:48:20.801 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.799+0000 7f7f86a5a700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f800c3260 con 0x7f7f8006d3f0 2022-04-03T19:48:20.801 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.799+0000 7f7f86a5a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f800c3540 con 0x7f7f800c5930 2022-04-03T19:48:20.801 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.799+0000 7f7f86259700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7f80156120 0x7f7f8006c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:20.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.800+0000 7f7f85257700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f800c5930 0x7f7f80155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:20.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.800+0000 7f7f85a58700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7f8006d3f0 0x7f7f8006d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:20.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.800+0000 7f7f86259700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7f80156120 0x7f7f8006c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42844/0 (socket says 172.21.15.77:42844) 2022-04-03T19:48:20.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.800+0000 7f7f86259700 1 -- 172.21.15.77:0/2384438349 learned_addr learned my addr 172.21.15.77:0/2384438349 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:20.806 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.800+0000 7f7f85257700 1 -- 172.21.15.77:0/2384438349 >> v2:172.21.15.186:3300/0 conn(0x7f7f8006d3f0 msgr2=0x7f7f8006d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:48:20.806 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.800+0000 7f7f85257700 1 --2- 172.21.15.77:0/2384438349 >> v2:172.21.15.186:3300/0 conn(0x7f7f8006d3f0 0x7f7f8006d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:20.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.800+0000 7f7f85257700 1 -- 172.21.15.77:0/2384438349 >> v2:172.21.15.77:3300/0 conn(0x7f7f80156120 msgr2=0x7f7f8006c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:48:20.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.800+0000 7f7f85257700 1 --2- 172.21.15.77:0/2384438349 >> v2:172.21.15.77:3300/0 conn(0x7f7f80156120 0x7f7f8006c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:20.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.800+0000 7f7f85257700 1 -- 172.21.15.77:0/2384438349 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f800c3820 con 0x7f7f800c5930 2022-04-03T19:48:20.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.800+0000 7f7f86259700 1 --2- 172.21.15.77:0/2384438349 >> v2:172.21.15.77:3300/0 conn(0x7f7f80156120 0x7f7f8006c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:48:20.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.800+0000 7f7f85257700 1 --2- 172.21.15.77:0/2384438349 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f800c5930 0x7f7f80155be0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7f70009760 tx=0x7f7f70009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b142efc87e324b1 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:20.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.801+0000 7f7f84a56700 1 -- 172.21.15.77:0/2384438349 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f70011630 con 0x7f7f800c5930 2022-04-03T19:48:20.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.801+0000 7f7f84a56700 1 -- 172.21.15.77:0/2384438349 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f70011790 con 0x7f7f800c5930 2022-04-03T19:48:20.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.801+0000 7f7f84a56700 1 -- 172.21.15.77:0/2384438349 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f70011b50 con 0x7f7f800c5930 2022-04-03T19:48:20.809 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.801+0000 7f7f86a5a700 1 -- 172.21.15.77:0/2384438349 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f800c5930 msgr2=0x7f7f80155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:20.809 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.801+0000 7f7f86a5a700 1 --2- 172.21.15.77:0/2384438349 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f800c5930 0x7f7f80155be0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7f70009760 tx=0x7f7f70009060 comp rx=0 tx=0).stop 2022-04-03T19:48:20.809 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.801+0000 7f7f86a5a700 1 -- 172.21.15.77:0/2384438349 shutdown_connections 2022-04-03T19:48:20.809 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.801+0000 7f7f86a5a700 1 --2- 172.21.15.77:0/2384438349 >> v2:172.21.15.77:3300/0 conn(0x7f7f80156120 0x7f7f8006c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:20.809 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.801+0000 7f7f86a5a700 1 --2- 172.21.15.77:0/2384438349 >> v2:172.21.15.186:3300/0 conn(0x7f7f8006d3f0 0x7f7f8006d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:20.810 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.801+0000 7f7f86a5a700 1 --2- 172.21.15.77:0/2384438349 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f800c5930 0x7f7f80155be0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:20.810 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.801+0000 7f7f86a5a700 1 -- 172.21.15.77:0/2384438349 >> 172.21.15.77:0/2384438349 conn(0x7f7f800cb720 msgr2=0x7f7f800cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:20.810 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.801+0000 7f7f86a5a700 1 -- 172.21.15.77:0/2384438349 shutdown_connections 2022-04-03T19:48:20.810 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.801+0000 7f7f86a5a700 1 -- 172.21.15.77:0/2384438349 wait complete. 2022-04-03T19:48:20.811 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.803+0000 7f7f86a5a700 1 Processor -- start 2022-04-03T19:48:20.811 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.803+0000 7f7f86a5a700 1 -- start start 2022-04-03T19:48:20.811 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.804+0000 7f7f86a5a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f8006d3f0 0x7f7f80165d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:20.811 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.804+0000 7f7f86a5a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f800c5930 0x7f7f800c1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:20.811 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.804+0000 7f7f86a5a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f80156120 0x7f7f80156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:20.812 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.804+0000 7f7f86a5a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f801591e0 con 0x7f7f8006d3f0 2022-04-03T19:48:20.812 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.804+0000 7f7f86a5a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f80158050 con 0x7f7f800c5930 2022-04-03T19:48:20.812 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.804+0000 7f7f86a5a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f801582a0 con 0x7f7f80156120 2022-04-03T19:48:20.812 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.804+0000 7f7f85a58700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f8006d3f0 0x7f7f80165d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:20.812 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.804+0000 7f7f85a58700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f8006d3f0 0x7f7f80165d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42846/0 (socket says 172.21.15.77:42846) 2022-04-03T19:48:20.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.804+0000 7f7f85a58700 1 -- 172.21.15.77:0/3615336232 learned_addr learned my addr 172.21.15.77:0/3615336232 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:20.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.804+0000 7f7f86259700 1 --2- 172.21.15.77:0/3615336232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f80156120 0x7f7f80156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:20.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.804+0000 7f7f85257700 1 --2- 172.21.15.77:0/3615336232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f800c5930 0x7f7f800c1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:20.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.805+0000 7f7f85a58700 1 -- 172.21.15.77:0/3615336232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f80156120 msgr2=0x7f7f80156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:20.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.805+0000 7f7f85a58700 1 --2- 172.21.15.77:0/3615336232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f80156120 0x7f7f80156640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:20.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.805+0000 7f7f85a58700 1 -- 172.21.15.77:0/3615336232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f800c5930 msgr2=0x7f7f800c1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:20.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.805+0000 7f7f85a58700 1 --2- 172.21.15.77:0/3615336232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f800c5930 0x7f7f800c1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:20.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.805+0000 7f7f85a58700 1 -- 172.21.15.77:0/3615336232 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f78009ae0 con 0x7f7f8006d3f0 2022-04-03T19:48:20.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.805+0000 7f7f85a58700 1 --2- 172.21.15.77:0/3615336232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f8006d3f0 0x7f7f80165d90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c00e900 tx=0x7f7f7c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:20.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.805+0000 7f7f6dffb700 1 -- 172.21.15.77:0/3615336232 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c013680 con 0x7f7f8006d3f0 2022-04-03T19:48:20.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.805+0000 7f7f6dffb700 1 -- 172.21.15.77:0/3615336232 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7c0137e0 con 0x7f7f8006d3f0 2022-04-03T19:48:20.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.805+0000 7f7f86a5a700 1 -- 172.21.15.77:0/3615336232 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f700092a0 con 0x7f7f8006d3f0 2022-04-03T19:48:20.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.805+0000 7f7f6dffb700 1 -- 172.21.15.77:0/3615336232 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c013b80 con 0x7f7f8006d3f0 2022-04-03T19:48:20.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.806+0000 7f7f86a5a700 1 -- 172.21.15.77:0/3615336232 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f80158840 con 0x7f7f8006d3f0 2022-04-03T19:48:20.817 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.807+0000 7f7f86a5a700 1 -- 172.21.15.77:0/3615336232 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f80159450 con 0x7f7f8006d3f0 2022-04-03T19:48:20.817 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.807+0000 7f7f6dffb700 1 -- 172.21.15.77:0/3615336232 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f7f7c019030 con 0x7f7f8006d3f0 2022-04-03T19:48:20.817 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.808+0000 7f7f6dffb700 1 --2- 172.21.15.77:0/3615336232 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7f60063ce0 0x7f7f60066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:20.818 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.808+0000 7f7f6dffb700 1 -- 172.21.15.77:0/3615336232 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6288+0+0 (secure 0 0 0) 0x7f7f7c080700 con 0x7f7f8006d3f0 2022-04-03T19:48:20.818 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.809+0000 7f7f85257700 1 --2- 172.21.15.77:0/3615336232 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7f60063ce0 0x7f7f60066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:20.818 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.809+0000 7f7f85257700 1 --2- 172.21.15.77:0/3615336232 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7f60063ce0 0x7f7f60066190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7f70009760 tx=0x7f7f70017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:20.818 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:20.812+0000 7f7f6dffb700 1 -- 172.21.15.77:0/3615336232 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f7f7c04f200 con 0x7f7f8006d3f0 2022-04-03T19:48:21.047 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:21.046+0000 7f7f86a5a700 1 -- 172.21.15.77:0/3615336232 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f8004b050 con 0x7f7f8006d3f0 2022-04-03T19:48:21.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:21.047+0000 7f7f6dffb700 1 -- 172.21.15.77:0/3615336232 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5352 (secure 0 0 0) 0x7f7f7c052820 con 0x7f7f8006d3f0 2022-04-03T19:48:21.048 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:21.049 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:48:19.127461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4960},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4960":{"gid":4960,"name":"e","rank":0,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:21.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:21.049+0000 7f7f86a5a700 1 -- 172.21.15.77:0/3615336232 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7f60063ce0 msgr2=0x7f7f60066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:21.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:21.049+0000 7f7f86a5a700 1 --2- 172.21.15.77:0/3615336232 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7f60063ce0 0x7f7f60066190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7f70009760 tx=0x7f7f70017000 comp rx=0 tx=0).stop 2022-04-03T19:48:21.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:21.049+0000 7f7f86a5a700 1 -- 172.21.15.77:0/3615336232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f8006d3f0 msgr2=0x7f7f80165d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:21.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:21.049+0000 7f7f86a5a700 1 --2- 172.21.15.77:0/3615336232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f8006d3f0 0x7f7f80165d90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c00e900 tx=0x7f7f7c009060 comp rx=0 tx=0).stop 2022-04-03T19:48:21.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:21.049+0000 7f7f86a5a700 1 -- 172.21.15.77:0/3615336232 shutdown_connections 2022-04-03T19:48:21.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:21.049+0000 7f7f86a5a700 1 --2- 172.21.15.77:0/3615336232 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7f60063ce0 0x7f7f60066190 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:21.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:21.049+0000 7f7f86a5a700 1 --2- 172.21.15.77:0/3615336232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f80156120 0x7f7f80156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:21.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:21.049+0000 7f7f86a5a700 1 --2- 172.21.15.77:0/3615336232 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f8006d3f0 0x7f7f80165d90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:21.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:21.049+0000 7f7f86a5a700 1 --2- 172.21.15.77:0/3615336232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f800c5930 0x7f7f800c1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:21.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:21.049+0000 7f7f86a5a700 1 -- 172.21.15.77:0/3615336232 >> 172.21.15.77:0/3615336232 conn(0x7f7f800cb720 msgr2=0x7f7f800c8110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:21.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:21.050+0000 7f7f86a5a700 1 -- 172.21.15.77:0/3615336232 shutdown_connections 2022-04-03T19:48:21.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:21.050+0000 7f7f86a5a700 1 -- 172.21.15.77:0/3615336232 wait complete. 2022-04-03T19:48:21.053 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 15 2022-04-03T19:48:21.074 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 15, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-04-03T19:44:21.275716+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 15, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4960": { "addr": "172.21.15.77:6837/2256447697", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6836", "nonce": 2256447697, "type": "v2" }, { "addr": "172.21.15.77:6837", "nonce": 2256447697, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4960, "incarnation": 15, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-04-03T19:48:19.127461+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4960 } } } ], "standbys": [ { "addr": "172.21.15.77:6834/2116223509", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6832", "nonce": 2116223509, "type": "v2" }, { "addr": "172.21.15.77:6834", "nonce": 2116223509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4245, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6836/2374276747", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2374276747, "type": "v2" }, { "addr": "172.21.15.186:6836", "nonce": 2374276747, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4259, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/610990853", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6838", "nonce": 610990853, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 610990853, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4265, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.77:6835/170053412", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6833", "nonce": 170053412, "type": "v2" }, { "addr": "172.21.15.77:6835", "nonce": 170053412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4271, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-04-03T19:48:23.076 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:23.281 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.281+0000 7fac0751d700 1 Processor -- start 2022-04-03T19:48:23.282 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.281+0000 7fac0751d700 1 -- start start 2022-04-03T19:48:23.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.282+0000 7fac0751d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac000ba3c0 0x7fac00156b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:23.286 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.282+0000 7fac0751d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fac000678b0 0x7fac0014d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:23.286 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.282+0000 7fac0751d700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac000c3c50 con 0x7fac00157060 2022-04-03T19:48:23.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.282+0000 7fac0751d700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac000c8020 con 0x7fac000678b0 2022-04-03T19:48:23.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.282+0000 7fac0751d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac00156690 con 0x7fac000ba3c0 2022-04-03T19:48:23.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.283+0000 7fac06d1c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fac000678b0 0x7fac0014d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:23.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.283+0000 7fac0651b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac000ba3c0 0x7fac00156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:23.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.283+0000 7fac06d1c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fac000678b0 0x7fac0014d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42848/0 (socket says 172.21.15.77:42848) 2022-04-03T19:48:23.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.283+0000 7fac06d1c700 1 -- 172.21.15.77:0/2142541504 learned_addr learned my addr 172.21.15.77:0/2142541504 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:23.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.283+0000 7fac0651b700 1 -- 172.21.15.77:0/2142541504 >> v2:172.21.15.77:3300/0 conn(0x7fac000678b0 msgr2=0x7fac0014d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:48:23.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.283+0000 7fac0651b700 1 --2- 172.21.15.77:0/2142541504 >> v2:172.21.15.77:3300/0 conn(0x7fac000678b0 0x7fac0014d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:23.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.283+0000 7fac0651b700 1 -- 172.21.15.77:0/2142541504 >> v1:172.21.15.186:6789/0 conn(0x7fac00157060 legacy=0x7fac0014cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:23.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.283+0000 7fac0651b700 1 -- 172.21.15.77:0/2142541504 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac000bf350 con 0x7fac000ba3c0 2022-04-03T19:48:23.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.283+0000 7fac06d1c700 1 --2- 172.21.15.77:0/2142541504 >> v2:172.21.15.77:3300/0 conn(0x7fac000678b0 0x7fac0014d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:48:23.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.283+0000 7fac0651b700 1 --2- 172.21.15.77:0/2142541504 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac000ba3c0 0x7fac00156b20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fabfc009760 tx=0x7fabfc009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75d78ebcde5bbac1 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:23.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.284+0000 7fac05519700 1 -- 172.21.15.77:0/2142541504 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc011630 con 0x7fac000ba3c0 2022-04-03T19:48:23.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.284+0000 7fac05519700 1 -- 172.21.15.77:0/2142541504 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabfc011790 con 0x7fac000ba3c0 2022-04-03T19:48:23.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.284+0000 7fac05519700 1 -- 172.21.15.77:0/2142541504 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc011b50 con 0x7fac000ba3c0 2022-04-03T19:48:23.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.284+0000 7fac0751d700 1 -- 172.21.15.77:0/2142541504 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac000ba3c0 msgr2=0x7fac00156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:23.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.284+0000 7fac0751d700 1 --2- 172.21.15.77:0/2142541504 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac000ba3c0 0x7fac00156b20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fabfc009760 tx=0x7fabfc009060 comp rx=0 tx=0).stop 2022-04-03T19:48:23.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.284+0000 7fac0751d700 1 -- 172.21.15.77:0/2142541504 shutdown_connections 2022-04-03T19:48:23.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.284+0000 7fac0751d700 1 --2- 172.21.15.77:0/2142541504 >> v2:172.21.15.77:3300/0 conn(0x7fac000678b0 0x7fac0014d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:23.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.284+0000 7fac0751d700 1 --2- 172.21.15.77:0/2142541504 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac000ba3c0 0x7fac00156b20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:23.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.284+0000 7fac0751d700 1 -- 172.21.15.77:0/2142541504 >> 172.21.15.77:0/2142541504 conn(0x7fac00066da0 msgr2=0x7fac000671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:23.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.284+0000 7fac0751d700 1 -- 172.21.15.77:0/2142541504 shutdown_connections 2022-04-03T19:48:23.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.284+0000 7fac0751d700 1 -- 172.21.15.77:0/2142541504 wait complete. 2022-04-03T19:48:23.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.286+0000 7fac0751d700 1 Processor -- start 2022-04-03T19:48:23.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.286+0000 7fac0751d700 1 -- start start 2022-04-03T19:48:23.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.287+0000 7fac0751d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac000678b0 0x7fac000c6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:23.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.287+0000 7fac0751d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac000ba3c0 0x7fac0014fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:23.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.287+0000 7fac0751d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac00157060 0x7fac0014ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:23.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.287+0000 7fac0751d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac001504a0 con 0x7fac00157060 2022-04-03T19:48:23.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.287+0000 7fac0751d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0006b140 con 0x7fac000678b0 2022-04-03T19:48:23.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.287+0000 7fac0751d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac000c71e0 con 0x7fac000ba3c0 2022-04-03T19:48:23.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.287+0000 7fac06d1c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac00157060 0x7fac0014ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:23.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.287+0000 7fac06d1c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac00157060 0x7fac0014ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42850/0 (socket says 172.21.15.77:42850) 2022-04-03T19:48:23.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.287+0000 7fac06d1c700 1 -- 172.21.15.77:0/336969864 learned_addr learned my addr 172.21.15.77:0/336969864 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:23.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.287+0000 7fac05d1a700 1 --2- 172.21.15.77:0/336969864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac000ba3c0 0x7fac0014fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:23.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.287+0000 7fac0651b700 1 --2- 172.21.15.77:0/336969864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac000678b0 0x7fac000c6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:23.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.288+0000 7fac06d1c700 1 -- 172.21.15.77:0/336969864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac000ba3c0 msgr2=0x7fac0014fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:23.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.288+0000 7fac06d1c700 1 --2- 172.21.15.77:0/336969864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac000ba3c0 0x7fac0014fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:23.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.288+0000 7fac06d1c700 1 -- 172.21.15.77:0/336969864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac000678b0 msgr2=0x7fac000c6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:23.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.288+0000 7fac06d1c700 1 --2- 172.21.15.77:0/336969864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac000678b0 0x7fac000c6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:23.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.288+0000 7fac06d1c700 1 -- 172.21.15.77:0/336969864 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf8009ae0 con 0x7fac00157060 2022-04-03T19:48:23.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.288+0000 7fac06d1c700 1 --2- 172.21.15.77:0/336969864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac00157060 0x7fac0014ff60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fabf8009fd0 tx=0x7fabf800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:23.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.288+0000 7fabee7fc700 1 -- 172.21.15.77:0/336969864 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf8018070 con 0x7fac00157060 2022-04-03T19:48:23.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.288+0000 7fabee7fc700 1 -- 172.21.15.77:0/336969864 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf800f040 con 0x7fac00157060 2022-04-03T19:48:23.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.288+0000 7fac0751d700 1 -- 172.21.15.77:0/336969864 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabfc0092a0 con 0x7fac00157060 2022-04-03T19:48:23.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.288+0000 7fabee7fc700 1 -- 172.21.15.77:0/336969864 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf80092e0 con 0x7fac00157060 2022-04-03T19:48:23.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.288+0000 7fac0751d700 1 -- 172.21.15.77:0/336969864 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac00168eb0 con 0x7fac00157060 2022-04-03T19:48:23.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.289+0000 7fac0751d700 1 -- 172.21.15.77:0/336969864 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac00066ba0 con 0x7fac00157060 2022-04-03T19:48:23.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.290+0000 7fabee7fc700 1 -- 172.21.15.77:0/336969864 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fabf8014480 con 0x7fac00157060 2022-04-03T19:48:23.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.291+0000 7fabee7fc700 1 --2- 172.21.15.77:0/336969864 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fabe0063ce0 0x7fabe0066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:23.302 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.292+0000 7fabee7fc700 1 -- 172.21.15.77:0/336969864 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6288+0+0 (secure 0 0 0) 0x7fabf8080db0 con 0x7fac00157060 2022-04-03T19:48:23.302 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.292+0000 7fac0651b700 1 --2- 172.21.15.77:0/336969864 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fabe0063ce0 0x7fabe0066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:23.302 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.293+0000 7fac0651b700 1 --2- 172.21.15.77:0/336969864 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fabe0063ce0 0x7fabe0066190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fabfc009760 tx=0x7fabfc017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:23.303 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.296+0000 7fabee7fc700 1 -- 172.21.15.77:0/336969864 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fabf804f850 con 0x7fac00157060 2022-04-03T19:48:23.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.526+0000 7fac0751d700 1 -- 172.21.15.77:0/336969864 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac000c74a0 con 0x7fac00157060 2022-04-03T19:48:23.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.526+0000 7fabee7fc700 1 -- 172.21.15.77:0/336969864 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5352 (secure 0 0 0) 0x7fabf8052e70 con 0x7fac00157060 2022-04-03T19:48:23.527 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:23.528 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:48:19.127461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4960},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4960":{"gid":4960,"name":"e","rank":0,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:23.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.528+0000 7fac0751d700 1 -- 172.21.15.77:0/336969864 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fabe0063ce0 msgr2=0x7fabe0066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:23.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.528+0000 7fac0751d700 1 --2- 172.21.15.77:0/336969864 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fabe0063ce0 0x7fabe0066190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fabfc009760 tx=0x7fabfc017000 comp rx=0 tx=0).stop 2022-04-03T19:48:23.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.529+0000 7fac0751d700 1 -- 172.21.15.77:0/336969864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac00157060 msgr2=0x7fac0014ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:23.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.529+0000 7fac0751d700 1 --2- 172.21.15.77:0/336969864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac00157060 0x7fac0014ff60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fabf8009fd0 tx=0x7fabf800b040 comp rx=0 tx=0).stop 2022-04-03T19:48:23.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.529+0000 7fac0751d700 1 -- 172.21.15.77:0/336969864 shutdown_connections 2022-04-03T19:48:23.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.529+0000 7fac0751d700 1 --2- 172.21.15.77:0/336969864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac00157060 0x7fac0014ff60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:23.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.529+0000 7fac0751d700 1 --2- 172.21.15.77:0/336969864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac000678b0 0x7fac000c6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:23.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.529+0000 7fac0751d700 1 --2- 172.21.15.77:0/336969864 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fabe0063ce0 0x7fabe0066190 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:23.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.529+0000 7fac0751d700 1 --2- 172.21.15.77:0/336969864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac000ba3c0 0x7fac0014fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:23.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.529+0000 7fac0751d700 1 -- 172.21.15.77:0/336969864 >> 172.21.15.77:0/336969864 conn(0x7fac00066da0 msgr2=0x7fac000c24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:23.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.529+0000 7fac0751d700 1 -- 172.21.15.77:0/336969864 shutdown_connections 2022-04-03T19:48:23.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:23.529+0000 7fac0751d700 1 -- 172.21.15.77:0/336969864 wait complete. 2022-04-03T19:48:23.533 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 15 2022-04-03T19:48:23.553 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:48:25.554 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:25.762 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.761+0000 7f5b8f44b700 1 Processor -- start 2022-04-03T19:48:25.762 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.761+0000 7f5b8f44b700 1 -- start start 2022-04-03T19:48:25.764 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.763+0000 7f5b8f44b700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5b8806bfb0 0x7f5b881584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:25.764 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.763+0000 7f5b8f44b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5b8806c500 0x7f5b8814eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:25.764 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.763+0000 7f5b8f44b700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b8806d260 con 0x7f5b8806bbe0 2022-04-03T19:48:25.765 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.763+0000 7f5b8f44b700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b88159450 con 0x7f5b8806c500 2022-04-03T19:48:25.765 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.763+0000 7f5b8f44b700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b880c6330 con 0x7f5b8806bfb0 2022-04-03T19:48:25.765 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.763+0000 7f5b8ec4a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5b8806c500 0x7f5b8814eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:25.765 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.763+0000 7f5b8ec4a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5b8806c500 0x7f5b8814eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42852/0 (socket says 172.21.15.77:42852) 2022-04-03T19:48:25.766 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.763+0000 7f5b8dc48700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5b8806bfb0 0x7f5b881584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:25.766 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.763+0000 7f5b8ec4a700 1 -- 172.21.15.77:0/1886305399 learned_addr learned my addr 172.21.15.77:0/1886305399 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:25.766 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.764+0000 7f5b8ec4a700 1 -- 172.21.15.77:0/1886305399 >> v2:172.21.15.186:3300/0 conn(0x7f5b8806bfb0 msgr2=0x7f5b881584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:48:25.766 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.764+0000 7f5b8ec4a700 1 --2- 172.21.15.77:0/1886305399 >> v2:172.21.15.186:3300/0 conn(0x7f5b8806bfb0 0x7f5b881584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:25.767 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.764+0000 7f5b8ec4a700 1 -- 172.21.15.77:0/1886305399 >> v1:172.21.15.186:6789/0 conn(0x7f5b8806bbe0 legacy=0x7f5b8806d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:25.767 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.764+0000 7f5b8ec4a700 1 -- 172.21.15.77:0/1886305399 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b88158bb0 con 0x7f5b8806c500 2022-04-03T19:48:25.767 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.764+0000 7f5b8ec4a700 1 --2- 172.21.15.77:0/1886305399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b8806c500 0x7f5b8814eea0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5b880898c0 tx=0x7f5b80009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90d43ce67889021 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:25.771 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.764+0000 7f5b8d447700 1 -- 172.21.15.77:0/1886305399 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8000e070 con 0x7f5b8806c500 2022-04-03T19:48:25.771 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.764+0000 7f5b8d447700 1 -- 172.21.15.77:0/1886305399 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8000ba20 con 0x7f5b8806c500 2022-04-03T19:48:25.771 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.764+0000 7f5b8d447700 1 -- 172.21.15.77:0/1886305399 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b800092e0 con 0x7f5b8806c500 2022-04-03T19:48:25.772 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.765+0000 7f5b8f44b700 1 -- 172.21.15.77:0/1886305399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b8806c500 msgr2=0x7f5b8814eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:25.772 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.765+0000 7f5b8f44b700 1 --2- 172.21.15.77:0/1886305399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b8806c500 0x7f5b8814eea0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5b880898c0 tx=0x7f5b80009840 comp rx=0 tx=0).stop 2022-04-03T19:48:25.772 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.765+0000 7f5b8f44b700 1 -- 172.21.15.77:0/1886305399 shutdown_connections 2022-04-03T19:48:25.772 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.765+0000 7f5b8f44b700 1 --2- 172.21.15.77:0/1886305399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b8806c500 0x7f5b8814eea0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:25.773 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.765+0000 7f5b8f44b700 1 --2- 172.21.15.77:0/1886305399 >> v2:172.21.15.186:3300/0 conn(0x7f5b8806bfb0 0x7f5b881584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:25.773 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.765+0000 7f5b8f44b700 1 -- 172.21.15.77:0/1886305399 >> 172.21.15.77:0/1886305399 conn(0x7f5b880cb6b0 msgr2=0x7f5b880cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:25.773 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.765+0000 7f5b8f44b700 1 -- 172.21.15.77:0/1886305399 shutdown_connections 2022-04-03T19:48:25.773 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.765+0000 7f5b8f44b700 1 -- 172.21.15.77:0/1886305399 wait complete. 2022-04-03T19:48:25.774 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.766+0000 7f5b8f44b700 1 Processor -- start 2022-04-03T19:48:25.774 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.766+0000 7f5b8f44b700 1 -- start start 2022-04-03T19:48:25.774 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.767+0000 7f5b8f44b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b8806bbe0 0x7f5b88148f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:25.774 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.767+0000 7f5b8f44b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b8806bfb0 0x7f5b88165ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:25.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.767+0000 7f5b8f44b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b8806c500 0x7f5b88164710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:25.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.767+0000 7f5b8f44b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b88152370 con 0x7f5b8806c500 2022-04-03T19:48:25.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.767+0000 7f5b8f44b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b88151290 con 0x7f5b8806bbe0 2022-04-03T19:48:25.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.767+0000 7f5b8f44b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881555a0 con 0x7f5b8806bfb0 2022-04-03T19:48:25.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.768+0000 7f5b8ec4a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b8806c500 0x7f5b88164710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:25.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.768+0000 7f5b8ec4a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b8806c500 0x7f5b88164710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42854/0 (socket says 172.21.15.77:42854) 2022-04-03T19:48:25.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.768+0000 7f5b8ec4a700 1 -- 172.21.15.77:0/1031626352 learned_addr learned my addr 172.21.15.77:0/1031626352 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:25.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.768+0000 7f5b8dc48700 1 --2- 172.21.15.77:0/1031626352 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b8806bfb0 0x7f5b88165ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:25.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.768+0000 7f5b8e449700 1 --2- 172.21.15.77:0/1031626352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b8806bbe0 0x7f5b88148f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:25.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.768+0000 7f5b8ec4a700 1 -- 172.21.15.77:0/1031626352 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b8806bfb0 msgr2=0x7f5b88165ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:25.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.768+0000 7f5b8ec4a700 1 --2- 172.21.15.77:0/1031626352 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b8806bfb0 0x7f5b88165ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:25.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.768+0000 7f5b8ec4a700 1 -- 172.21.15.77:0/1031626352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b8806bbe0 msgr2=0x7f5b88148f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:25.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.768+0000 7f5b8ec4a700 1 --2- 172.21.15.77:0/1031626352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b8806bbe0 0x7f5b88148f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:25.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.768+0000 7f5b8ec4a700 1 -- 172.21.15.77:0/1031626352 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b80009ae0 con 0x7f5b8806c500 2022-04-03T19:48:25.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.768+0000 7f5b8ec4a700 1 --2- 172.21.15.77:0/1031626352 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b8806c500 0x7f5b88164710 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5b800038e0 tx=0x7f5b80018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:25.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.769+0000 7f5b767fc700 1 -- 172.21.15.77:0/1031626352 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b80003da0 con 0x7f5b8806c500 2022-04-03T19:48:25.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.769+0000 7f5b767fc700 1 -- 172.21.15.77:0/1031626352 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b80007610 con 0x7f5b8806c500 2022-04-03T19:48:25.779 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.769+0000 7f5b8f44b700 1 -- 172.21.15.77:0/1031626352 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b88165470 con 0x7f5b8806c500 2022-04-03T19:48:25.779 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.769+0000 7f5b767fc700 1 -- 172.21.15.77:0/1031626352 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b800079b0 con 0x7f5b8806c500 2022-04-03T19:48:25.779 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.769+0000 7f5b8f44b700 1 -- 172.21.15.77:0/1031626352 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b88151780 con 0x7f5b8806c500 2022-04-03T19:48:25.779 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.770+0000 7f5b8f44b700 1 -- 172.21.15.77:0/1031626352 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b8804b050 con 0x7f5b8806c500 2022-04-03T19:48:25.779 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.771+0000 7f5b767fc700 1 -- 172.21.15.77:0/1031626352 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f5b80004040 con 0x7f5b8806c500 2022-04-03T19:48:25.780 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.771+0000 7f5b767fc700 1 --2- 172.21.15.77:0/1031626352 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5b68063d10 0x7f5b680661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:25.781 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.772+0000 7f5b767fc700 1 -- 172.21.15.77:0/1031626352 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6288+0+0 (secure 0 0 0) 0x7f5b80082760 con 0x7f5b8806c500 2022-04-03T19:48:25.781 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.772+0000 7f5b8e449700 1 --2- 172.21.15.77:0/1031626352 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5b68063d10 0x7f5b680661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:25.781 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.773+0000 7f5b8e449700 1 --2- 172.21.15.77:0/1031626352 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5b68063d10 0x7f5b680661c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5b840017a0 tx=0x7f5b84009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:25.782 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:25.776+0000 7f5b767fc700 1 -- 172.21.15.77:0/1031626352 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f5b80051260 con 0x7f5b8806c500 2022-04-03T19:48:26.008 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:26.007+0000 7f5b8f44b700 1 -- 172.21.15.77:0/1031626352 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b88063d10 con 0x7f5b8806c500 2022-04-03T19:48:26.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:26.007+0000 7f5b767fc700 1 -- 172.21.15.77:0/1031626352 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5352 (secure 0 0 0) 0x7f5b80080e60 con 0x7f5b8806c500 2022-04-03T19:48:26.009 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:26.010 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:48:19.127461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4960},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4960":{"gid":4960,"name":"e","rank":0,"incarnation":15,"state":"up:replay","state_seq":1,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:26.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:26.010+0000 7f5b8f44b700 1 -- 172.21.15.77:0/1031626352 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5b68063d10 msgr2=0x7f5b680661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:26.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:26.010+0000 7f5b8f44b700 1 --2- 172.21.15.77:0/1031626352 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5b68063d10 0x7f5b680661c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5b840017a0 tx=0x7f5b84009040 comp rx=0 tx=0).stop 2022-04-03T19:48:26.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:26.010+0000 7f5b8f44b700 1 -- 172.21.15.77:0/1031626352 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b8806c500 msgr2=0x7f5b88164710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:26.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:26.010+0000 7f5b8f44b700 1 --2- 172.21.15.77:0/1031626352 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b8806c500 0x7f5b88164710 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5b800038e0 tx=0x7f5b80018000 comp rx=0 tx=0).stop 2022-04-03T19:48:26.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:26.010+0000 7f5b8f44b700 1 -- 172.21.15.77:0/1031626352 shutdown_connections 2022-04-03T19:48:26.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:26.010+0000 7f5b8f44b700 1 --2- 172.21.15.77:0/1031626352 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b8806c500 0x7f5b88164710 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:26.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:26.010+0000 7f5b8f44b700 1 --2- 172.21.15.77:0/1031626352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b8806bbe0 0x7f5b88148f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:26.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:26.010+0000 7f5b8f44b700 1 --2- 172.21.15.77:0/1031626352 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5b68063d10 0x7f5b680661c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:26.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:26.010+0000 7f5b8f44b700 1 --2- 172.21.15.77:0/1031626352 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b8806bfb0 0x7f5b88165ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:26.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:26.010+0000 7f5b8f44b700 1 -- 172.21.15.77:0/1031626352 >> 172.21.15.77:0/1031626352 conn(0x7f5b880cb6b0 msgr2=0x7f5b8814a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:26.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:26.010+0000 7f5b8f44b700 1 -- 172.21.15.77:0/1031626352 shutdown_connections 2022-04-03T19:48:26.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:26.010+0000 7f5b8f44b700 1 -- 172.21.15.77:0/1031626352 wait complete. 2022-04-03T19:48:26.014 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 15 2022-04-03T19:48:26.035 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:48:28.036 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:28.239 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.238+0000 7f42d939a700 1 Processor -- start 2022-04-03T19:48:28.240 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.238+0000 7f42d939a700 1 -- start start 2022-04-03T19:48:28.246 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.239+0000 7f42d939a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42d406bbe0 0x7f42d41584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:28.246 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.239+0000 7f42d939a700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d406d260 con 0x7f42d406c500 2022-04-03T19:48:28.246 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.239+0000 7f42d939a700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d40c6330 con 0x7f42d406bfb0 2022-04-03T19:48:28.246 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.239+0000 7f42d939a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d4159450 con 0x7f42d406bbe0 2022-04-03T19:48:28.247 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.240+0000 7f42d8b99700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f42d406c500 0x7f42d414eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38438/0 (socket says 172.21.15.77:38438) 2022-04-03T19:48:28.247 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.240+0000 7f42d8b99700 1 -- 172.21.15.77:0/1244282781 learned_addr learned my addr 172.21.15.77:0/1244282781 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:48:28.247 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.240+0000 7f42d3fff700 1 --2- 172.21.15.77:0/1244282781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42d406bbe0 0x7f42d41584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:28.247 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.240+0000 7f42d3fff700 1 -- 172.21.15.77:0/1244282781 >> v1:172.21.15.186:6789/0 conn(0x7f42d406bfb0 legacy=0x7f42d406d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:28.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.240+0000 7f42d2ffd700 1 -- 172.21.15.77:0/1244282781 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016544672 0 0) 0x7f42d40c6330 con 0x7f42d406bfb0 2022-04-03T19:48:28.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.240+0000 7f42d3fff700 1 -- 172.21.15.77:0/1244282781 >> v1:172.21.15.77:6789/0 conn(0x7f42d406c500 legacy=0x7f42d414eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:28.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.240+0000 7f42d3fff700 1 -- 172.21.15.77:0/1244282781 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d4158bb0 con 0x7f42d406bbe0 2022-04-03T19:48:28.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.241+0000 7f42d3fff700 1 --2- 172.21.15.77:0/1244282781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42d406bbe0 0x7f42d41584b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f42c4009790 tx=0x7f42c4009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f08ee6eea97e9554 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:28.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.241+0000 7f42d2ffd700 1 -- 172.21.15.77:0/1244282781 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c4004030 con 0x7f42d406bbe0 2022-04-03T19:48:28.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.241+0000 7f42d2ffd700 1 -- 172.21.15.77:0/1244282781 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c4004190 con 0x7f42d406bbe0 2022-04-03T19:48:28.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.241+0000 7f42d2ffd700 1 -- 172.21.15.77:0/1244282781 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c4011630 con 0x7f42d406bbe0 2022-04-03T19:48:28.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.241+0000 7f42d939a700 1 -- 172.21.15.77:0/1244282781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42d406bbe0 msgr2=0x7f42d41584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:28.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.241+0000 7f42d939a700 1 --2- 172.21.15.77:0/1244282781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42d406bbe0 0x7f42d41584b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f42c4009790 tx=0x7f42c4009060 comp rx=0 tx=0).stop 2022-04-03T19:48:28.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.242+0000 7f42d939a700 1 -- 172.21.15.77:0/1244282781 shutdown_connections 2022-04-03T19:48:28.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.242+0000 7f42d939a700 1 --2- 172.21.15.77:0/1244282781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42d406bbe0 0x7f42d41584b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:28.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.242+0000 7f42d939a700 1 -- 172.21.15.77:0/1244282781 >> 172.21.15.77:0/1244282781 conn(0x7f42d40cb6b0 msgr2=0x7f42d40cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:28.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.242+0000 7f42d939a700 1 -- 172.21.15.77:0/1244282781 shutdown_connections 2022-04-03T19:48:28.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.242+0000 7f42d939a700 1 -- 172.21.15.77:0/1244282781 wait complete. 2022-04-03T19:48:28.254 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.243+0000 7f42d939a700 1 Processor -- start 2022-04-03T19:48:28.254 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.243+0000 7f42d939a700 1 -- start start 2022-04-03T19:48:28.255 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.244+0000 7f42d939a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42d406bbe0 0x7f42d4168d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:28.255 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.244+0000 7f42d939a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42d406bfb0 0x7f42d4167120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:28.255 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.244+0000 7f42d939a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42d406c500 0x7f42d4167660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:28.256 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.244+0000 7f42d939a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d41550c0 con 0x7f42d406bbe0 2022-04-03T19:48:28.256 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.244+0000 7f42d939a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d4166c90 con 0x7f42d406c500 2022-04-03T19:48:28.257 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.244+0000 7f42d939a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d4169780 con 0x7f42d406bfb0 2022-04-03T19:48:28.257 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.245+0000 7f42d3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42d406bbe0 0x7f42d4168d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:28.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.245+0000 7f42d3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42d406bbe0 0x7f42d4168d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42856/0 (socket says 172.21.15.77:42856) 2022-04-03T19:48:28.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.245+0000 7f42d3fff700 1 -- 172.21.15.77:0/3900620107 learned_addr learned my addr 172.21.15.77:0/3900620107 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:28.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.245+0000 7f42d8b99700 1 --2- 172.21.15.77:0/3900620107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42d406c500 0x7f42d4167660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:28.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.245+0000 7f42d37fe700 1 --2- 172.21.15.77:0/3900620107 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42d406bfb0 0x7f42d4167120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:28.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.245+0000 7f42d3fff700 1 -- 172.21.15.77:0/3900620107 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42d406bfb0 msgr2=0x7f42d4167120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:28.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.245+0000 7f42d3fff700 1 --2- 172.21.15.77:0/3900620107 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42d406bfb0 0x7f42d4167120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:28.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.245+0000 7f42d3fff700 1 -- 172.21.15.77:0/3900620107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42d406c500 msgr2=0x7f42d4167660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:28.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.245+0000 7f42d3fff700 1 --2- 172.21.15.77:0/3900620107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42d406c500 0x7f42d4167660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:28.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.245+0000 7f42d3fff700 1 -- 172.21.15.77:0/3900620107 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c40092a0 con 0x7f42d406bbe0 2022-04-03T19:48:28.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.246+0000 7f42d3fff700 1 --2- 172.21.15.77:0/3900620107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42d406bbe0 0x7f42d4168d20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f42c4004630 tx=0x7f42c4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:28.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.246+0000 7f42bbfff700 1 -- 172.21.15.77:0/3900620107 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c4003840 con 0x7f42d406bbe0 2022-04-03T19:48:28.263 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.246+0000 7f42bbfff700 1 -- 172.21.15.77:0/3900620107 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c40039a0 con 0x7f42d406bbe0 2022-04-03T19:48:28.263 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.246+0000 7f42bbfff700 1 -- 172.21.15.77:0/3900620107 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c4003d60 con 0x7f42d406bbe0 2022-04-03T19:48:28.264 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.246+0000 7f42d939a700 1 -- 172.21.15.77:0/3900620107 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42d41552d0 con 0x7f42d406bbe0 2022-04-03T19:48:28.264 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.246+0000 7f42d939a700 1 -- 172.21.15.77:0/3900620107 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d40ca550 con 0x7f42d406bbe0 2022-04-03T19:48:28.264 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.247+0000 7f42d939a700 1 -- 172.21.15.77:0/3900620107 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42d4166a60 con 0x7f42d406bbe0 2022-04-03T19:48:28.265 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.248+0000 7f42bbfff700 1 -- 172.21.15.77:0/3900620107 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f42c4017020 con 0x7f42d406bbe0 2022-04-03T19:48:28.265 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.249+0000 7f42bbfff700 1 --2- 172.21.15.77:0/3900620107 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f42b4063ce0 0x7f42b4066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:28.266 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.249+0000 7f42bbfff700 1 -- 172.21.15.77:0/3900620107 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6288+0+0 (secure 0 0 0) 0x7f42c40822f0 con 0x7f42d406bbe0 2022-04-03T19:48:28.266 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.249+0000 7f42d37fe700 1 --2- 172.21.15.77:0/3900620107 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f42b4063ce0 0x7f42b4066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:28.267 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.250+0000 7f42d37fe700 1 --2- 172.21.15.77:0/3900620107 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f42b4063ce0 0x7f42b4066190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f42d4155eb0 tx=0x7f42c8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:28.267 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.253+0000 7f42bbfff700 1 -- 172.21.15.77:0/3900620107 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f42d4166a60 con 0x7f42d406bbe0 2022-04-03T19:48:28.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.488+0000 7f42d939a700 1 -- 172.21.15.77:0/3900620107 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42d4166a60 con 0x7f42d406bbe0 2022-04-03T19:48:28.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.489+0000 7f42bbfff700 1 -- 172.21.15.77:0/3900620107 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5356 (secure 0 0 0) 0x7f42d4166a60 con 0x7f42d406bbe0 2022-04-03T19:48:28.490 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:28.490 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:48:26.176265+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4960},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4960":{"gid":4960,"name":"e","rank":0,"incarnation":15,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:28.492 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.491+0000 7f42d939a700 1 -- 172.21.15.77:0/3900620107 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f42b4063ce0 msgr2=0x7f42b4066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:28.492 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.491+0000 7f42d939a700 1 --2- 172.21.15.77:0/3900620107 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f42b4063ce0 0x7f42b4066190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f42d4155eb0 tx=0x7f42c8009040 comp rx=0 tx=0).stop 2022-04-03T19:48:28.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.491+0000 7f42d939a700 1 -- 172.21.15.77:0/3900620107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42d406bbe0 msgr2=0x7f42d4168d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:28.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.491+0000 7f42d939a700 1 --2- 172.21.15.77:0/3900620107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42d406bbe0 0x7f42d4168d20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f42c4004630 tx=0x7f42c4013000 comp rx=0 tx=0).stop 2022-04-03T19:48:28.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.491+0000 7f42d939a700 1 -- 172.21.15.77:0/3900620107 shutdown_connections 2022-04-03T19:48:28.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.491+0000 7f42d939a700 1 --2- 172.21.15.77:0/3900620107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42d406c500 0x7f42d4167660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:28.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.491+0000 7f42d939a700 1 --2- 172.21.15.77:0/3900620107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42d406bbe0 0x7f42d4168d20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:28.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.491+0000 7f42d939a700 1 --2- 172.21.15.77:0/3900620107 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f42b4063ce0 0x7f42b4066190 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:28.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.491+0000 7f42d939a700 1 --2- 172.21.15.77:0/3900620107 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42d406bfb0 0x7f42d4167120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:28.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.491+0000 7f42d939a700 1 -- 172.21.15.77:0/3900620107 >> 172.21.15.77:0/3900620107 conn(0x7f42d40cb6b0 msgr2=0x7f42d414a130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:28.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.491+0000 7f42d939a700 1 -- 172.21.15.77:0/3900620107 shutdown_connections 2022-04-03T19:48:28.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:28.492+0000 7f42d939a700 1 -- 172.21.15.77:0/3900620107 wait complete. 2022-04-03T19:48:28.495 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 16 2022-04-03T19:48:28.516 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:48:30.516 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:30.720 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.720+0000 7f7f6bfff700 1 Processor -- start 2022-04-03T19:48:30.721 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.720+0000 7f7f6bfff700 1 -- start start 2022-04-03T19:48:30.722 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.721+0000 7f7f6bfff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7f6c0ba3c0 0x7f7f6c156b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:30.722 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.721+0000 7f7f6bfff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f6c157060 0x7f7f6c14cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:30.722 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.721+0000 7f7f6bfff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f6c0c3c50 con 0x7f7f6c0678b0 2022-04-03T19:48:30.723 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.721+0000 7f7f6bfff700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f6c0c8020 con 0x7f7f6c0ba3c0 2022-04-03T19:48:30.723 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.721+0000 7f7f6bfff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c156690 con 0x7f7f6c157060 2022-04-03T19:48:30.723 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.722+0000 7f7f6b7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7f6c0678b0 0x7f7f6c14d040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38440/0 (socket says 172.21.15.77:38440) 2022-04-03T19:48:30.723 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.722+0000 7f7f6b7fe700 1 -- 172.21.15.77:0/3396106544 learned_addr learned my addr 172.21.15.77:0/3396106544 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:48:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.722+0000 7f7f6a7fc700 1 --2- 172.21.15.77:0/3396106544 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f6c157060 0x7f7f6c14cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:30.724 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.722+0000 7f7f6affd700 1 --2- 172.21.15.77:0/3396106544 >> v2:172.21.15.186:3300/0 conn(0x7f7f6c0ba3c0 0x7f7f6c156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:30.726 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.722+0000 7f7f69ffb700 1 -- 172.21.15.77:0/3396106544 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041645656 0 0) 0x7f7f6c0c3c50 con 0x7f7f6c0678b0 2022-04-03T19:48:30.726 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.722+0000 7f7f69ffb700 1 -- 172.21.15.77:0/3396106544 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f580033f0 con 0x7f7f6c0678b0 2022-04-03T19:48:30.726 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.722+0000 7f7f6a7fc700 1 -- 172.21.15.77:0/3396106544 >> v2:172.21.15.186:3300/0 conn(0x7f7f6c0ba3c0 msgr2=0x7f7f6c156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:48:30.726 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.722+0000 7f7f6a7fc700 1 --2- 172.21.15.77:0/3396106544 >> v2:172.21.15.186:3300/0 conn(0x7f7f6c0ba3c0 0x7f7f6c156b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:30.727 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.722+0000 7f7f6a7fc700 1 -- 172.21.15.77:0/3396106544 >> v1:172.21.15.77:6789/0 conn(0x7f7f6c0678b0 legacy=0x7f7f6c14d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:30.727 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.722+0000 7f7f6a7fc700 1 -- 172.21.15.77:0/3396106544 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f6c0bf350 con 0x7f7f6c157060 2022-04-03T19:48:30.727 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.722+0000 7f7f6affd700 1 --2- 172.21.15.77:0/3396106544 >> v2:172.21.15.186:3300/0 conn(0x7f7f6c0ba3c0 0x7f7f6c156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:48:30.727 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.723+0000 7f7f6a7fc700 1 --2- 172.21.15.77:0/3396106544 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f6c157060 0x7f7f6c14cb00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7f60009760 tx=0x7f7f60009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61c9767cdbdfa209 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:30.728 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.723+0000 7f7f69ffb700 1 -- 172.21.15.77:0/3396106544 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f60011630 con 0x7f7f6c157060 2022-04-03T19:48:30.728 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.723+0000 7f7f69ffb700 1 -- 172.21.15.77:0/3396106544 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f60011790 con 0x7f7f6c157060 2022-04-03T19:48:30.728 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.723+0000 7f7f69ffb700 1 -- 172.21.15.77:0/3396106544 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f60011b30 con 0x7f7f6c157060 2022-04-03T19:48:30.728 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.723+0000 7f7f6bfff700 1 -- 172.21.15.77:0/3396106544 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f6c157060 msgr2=0x7f7f6c14cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:30.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.723+0000 7f7f6bfff700 1 --2- 172.21.15.77:0/3396106544 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f6c157060 0x7f7f6c14cb00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7f60009760 tx=0x7f7f60009060 comp rx=0 tx=0).stop 2022-04-03T19:48:30.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.724+0000 7f7f6bfff700 1 -- 172.21.15.77:0/3396106544 shutdown_connections 2022-04-03T19:48:30.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.724+0000 7f7f6bfff700 1 --2- 172.21.15.77:0/3396106544 >> v2:172.21.15.186:3300/0 conn(0x7f7f6c0ba3c0 0x7f7f6c156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:30.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.724+0000 7f7f6bfff700 1 --2- 172.21.15.77:0/3396106544 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f6c157060 0x7f7f6c14cb00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:30.729 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.724+0000 7f7f6bfff700 1 -- 172.21.15.77:0/3396106544 >> 172.21.15.77:0/3396106544 conn(0x7f7f6c066da0 msgr2=0x7f7f6c0671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:30.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.724+0000 7f7f6bfff700 1 -- 172.21.15.77:0/3396106544 shutdown_connections 2022-04-03T19:48:30.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.724+0000 7f7f6bfff700 1 -- 172.21.15.77:0/3396106544 wait complete. 2022-04-03T19:48:30.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.725+0000 7f7f6bfff700 1 Processor -- start 2022-04-03T19:48:30.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.725+0000 7f7f6bfff700 1 -- start start 2022-04-03T19:48:30.733 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.726+0000 7f7f6bfff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f6c0678b0 0x7f7f6c0c0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:30.733 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.726+0000 7f7f6bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f6c0ba3c0 0x7f7f6c1680e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:30.733 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.726+0000 7f7f6bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f6c157060 0x7f7f6c168620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:30.734 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.726+0000 7f7f6bfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c0696b0 con 0x7f7f6c157060 2022-04-03T19:48:30.734 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.726+0000 7f7f6bfff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c0c4b20 con 0x7f7f6c0ba3c0 2022-04-03T19:48:30.734 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.726+0000 7f7f6bfff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c165770 con 0x7f7f6c0678b0 2022-04-03T19:48:30.734 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.727+0000 7f7f6affd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f6c0678b0 0x7f7f6c0c0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:30.735 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.727+0000 7f7f6b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f6c157060 0x7f7f6c168620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:30.735 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.727+0000 7f7f6a7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f6c0ba3c0 0x7f7f6c1680e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:30.735 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.727+0000 7f7f6affd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f6c0678b0 0x7f7f6c0c0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60154/0 (socket says 172.21.15.77:60154) 2022-04-03T19:48:30.735 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.727+0000 7f7f6a7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f6c0ba3c0 0x7f7f6c1680e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38052/0 (socket says 172.21.15.77:38052) 2022-04-03T19:48:30.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.727+0000 7f7f6affd700 1 -- 172.21.15.77:0/1882420048 learned_addr learned my addr 172.21.15.77:0/1882420048 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:30.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.727+0000 7f7f6affd700 1 -- 172.21.15.77:0/1882420048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f6c0ba3c0 msgr2=0x7f7f6c1680e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:30.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.727+0000 7f7f6affd700 1 --2- 172.21.15.77:0/1882420048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f6c0ba3c0 0x7f7f6c1680e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:30.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.727+0000 7f7f6affd700 1 -- 172.21.15.77:0/1882420048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f6c157060 msgr2=0x7f7f6c168620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:30.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.727+0000 7f7f6affd700 1 --2- 172.21.15.77:0/1882420048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f6c157060 0x7f7f6c168620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:30.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.727+0000 7f7f6affd700 1 -- 172.21.15.77:0/1882420048 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c0092a0 con 0x7f7f6c0678b0 2022-04-03T19:48:30.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.727+0000 7f7f6b7fe700 1 --2- 172.21.15.77:0/1882420048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f6c157060 0x7f7f6c168620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:48:30.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.727+0000 7f7f6affd700 1 --2- 172.21.15.77:0/1882420048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f6c0678b0 0x7f7f6c0c0dd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00ea50 tx=0x7f7f5c009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:30.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.728+0000 7f7f52ffd700 1 -- 172.21.15.77:0/1882420048 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c00ee90 con 0x7f7f6c0678b0 2022-04-03T19:48:30.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.728+0000 7f7f52ffd700 1 -- 172.21.15.77:0/1882420048 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5c00c8e0 con 0x7f7f6c0678b0 2022-04-03T19:48:30.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.728+0000 7f7f6bfff700 1 -- 172.21.15.77:0/1882420048 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f600092a0 con 0x7f7f6c0678b0 2022-04-03T19:48:30.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.728+0000 7f7f52ffd700 1 -- 172.21.15.77:0/1882420048 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c00cc60 con 0x7f7f6c0678b0 2022-04-03T19:48:30.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.728+0000 7f7f6bfff700 1 -- 172.21.15.77:0/1882420048 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f6c164b60 con 0x7f7f6c0678b0 2022-04-03T19:48:30.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.729+0000 7f7f6bfff700 1 -- 172.21.15.77:0/1882420048 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f6c0c74a0 con 0x7f7f6c0678b0 2022-04-03T19:48:30.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.730+0000 7f7f52ffd700 1 -- 172.21.15.77:0/1882420048 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f7f5c019030 con 0x7f7f6c0678b0 2022-04-03T19:48:30.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.730+0000 7f7f52ffd700 1 --2- 172.21.15.77:0/1882420048 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7f4c063d10 0x7f7f4c0661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:30.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.730+0000 7f7f52ffd700 1 -- 172.21.15.77:0/1882420048 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6288+0+0 (secure 0 0 0) 0x7f7f5c080a70 con 0x7f7f6c0678b0 2022-04-03T19:48:30.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.731+0000 7f7f6a7fc700 1 --2- 172.21.15.77:0/1882420048 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7f4c063d10 0x7f7f4c0661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:30.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.732+0000 7f7f6a7fc700 1 --2- 172.21.15.77:0/1882420048 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7f4c063d10 0x7f7f4c0661c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c157470 tx=0x7f7f60017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:30.740 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.735+0000 7f7f52ffd700 1 -- 172.21.15.77:0/1882420048 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f7f5c082e60 con 0x7f7f6c0678b0 2022-04-03T19:48:30.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.971+0000 7f7f6bfff700 1 -- 172.21.15.77:0/1882420048 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f6c04b050 con 0x7f7f6c0678b0 2022-04-03T19:48:30.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.971+0000 7f7f52ffd700 1 -- 172.21.15.77:0/1882420048 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5356 (secure 0 0 0) 0x7f7f5c017090 con 0x7f7f6c0678b0 2022-04-03T19:48:30.972 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:30.972 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:48:26.176265+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4960},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4960":{"gid":4960,"name":"e","rank":0,"incarnation":15,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:30.974 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.974+0000 7f7f6bfff700 1 -- 172.21.15.77:0/1882420048 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7f4c063d10 msgr2=0x7f7f4c0661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:30.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.974+0000 7f7f6bfff700 1 --2- 172.21.15.77:0/1882420048 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7f4c063d10 0x7f7f4c0661c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c157470 tx=0x7f7f60017000 comp rx=0 tx=0).stop 2022-04-03T19:48:30.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.974+0000 7f7f6bfff700 1 -- 172.21.15.77:0/1882420048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f6c0678b0 msgr2=0x7f7f6c0c0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:30.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.974+0000 7f7f6bfff700 1 --2- 172.21.15.77:0/1882420048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f6c0678b0 0x7f7f6c0c0dd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00ea50 tx=0x7f7f5c009060 comp rx=0 tx=0).stop 2022-04-03T19:48:30.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.974+0000 7f7f6bfff700 1 -- 172.21.15.77:0/1882420048 shutdown_connections 2022-04-03T19:48:30.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.974+0000 7f7f6bfff700 1 --2- 172.21.15.77:0/1882420048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7f6c157060 0x7f7f6c168620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:30.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.974+0000 7f7f6bfff700 1 --2- 172.21.15.77:0/1882420048 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7f4c063d10 0x7f7f4c0661c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:30.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.974+0000 7f7f6bfff700 1 --2- 172.21.15.77:0/1882420048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f6c0678b0 0x7f7f6c0c0dd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:30.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.974+0000 7f7f6bfff700 1 --2- 172.21.15.77:0/1882420048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f6c0ba3c0 0x7f7f6c1680e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:30.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.974+0000 7f7f6bfff700 1 -- 172.21.15.77:0/1882420048 >> 172.21.15.77:0/1882420048 conn(0x7f7f6c066da0 msgr2=0x7f7f6c0bebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:30.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.974+0000 7f7f6bfff700 1 -- 172.21.15.77:0/1882420048 shutdown_connections 2022-04-03T19:48:30.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:30.974+0000 7f7f6bfff700 1 -- 172.21.15.77:0/1882420048 wait complete. 2022-04-03T19:48:30.978 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 16 2022-04-03T19:48:30.999 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:48:33.000 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:33.206 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.205+0000 7f720359e700 1 Processor -- start 2022-04-03T19:48:33.206 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.206+0000 7f720359e700 1 -- start start 2022-04-03T19:48:33.209 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.207+0000 7f720359e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72040ba3c0 0x7f7204156b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:33.209 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.207+0000 7f720359e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72040678b0 0x7f720414d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:33.209 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.207+0000 7f720359e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72040c3c50 con 0x7f7204157060 2022-04-03T19:48:33.209 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.207+0000 7f720359e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72040c8020 con 0x7f72040678b0 2022-04-03T19:48:33.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.207+0000 7f720359e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7204156690 con 0x7f72040ba3c0 2022-04-03T19:48:33.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.207+0000 7f7202d9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72040678b0 0x7f720414d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:33.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.207+0000 7f720259c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72040ba3c0 0x7f7204156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:33.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.207+0000 7f7202d9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72040678b0 0x7f720414d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42860/0 (socket says 172.21.15.77:42860) 2022-04-03T19:48:33.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.207+0000 7f7201d9b700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f7204157060 0x7f720414cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44234/0 (socket says 172.21.15.77:44234) 2022-04-03T19:48:33.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.207+0000 7f7202d9d700 1 -- 172.21.15.77:0/3473317756 learned_addr learned my addr 172.21.15.77:0/3473317756 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:33.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.207+0000 7f720259c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72040ba3c0 0x7f7204156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60156/0 (socket says 172.21.15.77:60156) 2022-04-03T19:48:33.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.208+0000 7f720259c700 1 -- 172.21.15.77:0/3473317756 >> v2:172.21.15.77:3300/0 conn(0x7f72040678b0 msgr2=0x7f720414d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:48:33.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.208+0000 7f720259c700 1 --2- 172.21.15.77:0/3473317756 >> v2:172.21.15.77:3300/0 conn(0x7f72040678b0 0x7f720414d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:33.212 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.208+0000 7f720159a700 1 -- 172.21.15.77:0/3473317756 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825610481 0 0) 0x7f72040c3c50 con 0x7f7204157060 2022-04-03T19:48:33.212 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.208+0000 7f720259c700 1 -- 172.21.15.77:0/3473317756 >> v1:172.21.15.186:6789/0 conn(0x7f7204157060 legacy=0x7f720414cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:33.212 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.208+0000 7f720259c700 1 -- 172.21.15.77:0/3473317756 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72040bf350 con 0x7f72040ba3c0 2022-04-03T19:48:33.212 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.208+0000 7f7202d9d700 1 --2- 172.21.15.77:0/3473317756 >> v2:172.21.15.77:3300/0 conn(0x7f72040678b0 0x7f720414d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:48:33.213 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.208+0000 7f720259c700 1 --2- 172.21.15.77:0/3473317756 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72040ba3c0 0x7f7204156b20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f71f4009760 tx=0x7f71f4009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16521160729dbff1 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:33.213 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.208+0000 7f720159a700 1 -- 172.21.15.77:0/3473317756 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f4011630 con 0x7f72040ba3c0 2022-04-03T19:48:33.213 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.208+0000 7f720159a700 1 -- 172.21.15.77:0/3473317756 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f4011790 con 0x7f72040ba3c0 2022-04-03T19:48:33.213 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.208+0000 7f720159a700 1 -- 172.21.15.77:0/3473317756 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f4011b50 con 0x7f72040ba3c0 2022-04-03T19:48:33.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.209+0000 7f720359e700 1 -- 172.21.15.77:0/3473317756 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72040ba3c0 msgr2=0x7f7204156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:33.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.209+0000 7f720359e700 1 --2- 172.21.15.77:0/3473317756 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72040ba3c0 0x7f7204156b20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f71f4009760 tx=0x7f71f4009060 comp rx=0 tx=0).stop 2022-04-03T19:48:33.218 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.209+0000 7f720359e700 1 -- 172.21.15.77:0/3473317756 shutdown_connections 2022-04-03T19:48:33.218 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.209+0000 7f720359e700 1 --2- 172.21.15.77:0/3473317756 >> v2:172.21.15.77:3300/0 conn(0x7f72040678b0 0x7f720414d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:33.218 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.209+0000 7f720359e700 1 --2- 172.21.15.77:0/3473317756 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72040ba3c0 0x7f7204156b20 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:33.218 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.209+0000 7f720359e700 1 -- 172.21.15.77:0/3473317756 >> 172.21.15.77:0/3473317756 conn(0x7f7204066da0 msgr2=0x7f72040671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:33.219 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.209+0000 7f720359e700 1 -- 172.21.15.77:0/3473317756 shutdown_connections 2022-04-03T19:48:33.219 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.209+0000 7f720359e700 1 -- 172.21.15.77:0/3473317756 wait complete. 2022-04-03T19:48:33.219 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.211+0000 7f720359e700 1 Processor -- start 2022-04-03T19:48:33.219 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.211+0000 7f720359e700 1 -- start start 2022-04-03T19:48:33.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.212+0000 7f720359e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72040678b0 0x7f72040c3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:33.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.212+0000 7f720359e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72040ba3c0 0x7f7204149c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:33.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.212+0000 7f720359e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7204157060 0x7f720414a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:33.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.212+0000 7f720359e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72040b9880 con 0x7f72040678b0 2022-04-03T19:48:33.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.212+0000 7f720359e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72040c3610 con 0x7f72040ba3c0 2022-04-03T19:48:33.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.212+0000 7f720359e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7204165330 con 0x7f7204157060 2022-04-03T19:48:33.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.212+0000 7f720259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72040678b0 0x7f72040c3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:33.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.212+0000 7f720259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72040678b0 0x7f72040c3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42862/0 (socket says 172.21.15.77:42862) 2022-04-03T19:48:33.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.212+0000 7f720259c700 1 -- 172.21.15.77:0/3925020793 learned_addr learned my addr 172.21.15.77:0/3925020793 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:33.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.212+0000 7f7201d9b700 1 --2- 172.21.15.77:0/3925020793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72040ba3c0 0x7f7204149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:33.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.212+0000 7f7202d9d700 1 --2- 172.21.15.77:0/3925020793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7204157060 0x7f720414a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:33.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.212+0000 7f720259c700 1 -- 172.21.15.77:0/3925020793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7204157060 msgr2=0x7f720414a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:33.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.212+0000 7f720259c700 1 --2- 172.21.15.77:0/3925020793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7204157060 0x7f720414a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:33.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.212+0000 7f720259c700 1 -- 172.21.15.77:0/3925020793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72040ba3c0 msgr2=0x7f7204149c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:33.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.212+0000 7f720259c700 1 --2- 172.21.15.77:0/3925020793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72040ba3c0 0x7f7204149c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:33.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.212+0000 7f720259c700 1 -- 172.21.15.77:0/3925020793 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71fc009ae0 con 0x7f72040678b0 2022-04-03T19:48:33.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.212+0000 7f7202d9d700 1 --2- 172.21.15.77:0/3925020793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7204157060 0x7f720414a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:48:33.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.212+0000 7f7201d9b700 1 --2- 172.21.15.77:0/3925020793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72040ba3c0 0x7f7204149c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:48:33.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.213+0000 7f720259c700 1 --2- 172.21.15.77:0/3925020793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72040678b0 0x7f72040c3c30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f71f40091e0 tx=0x7f71f4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:33.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.213+0000 7f71f27fc700 1 -- 172.21.15.77:0/3925020793 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f4000c90 con 0x7f72040678b0 2022-04-03T19:48:33.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.213+0000 7f71f27fc700 1 -- 172.21.15.77:0/3925020793 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f4011900 con 0x7f72040678b0 2022-04-03T19:48:33.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.213+0000 7f71f27fc700 1 -- 172.21.15.77:0/3925020793 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f4025420 con 0x7f72040678b0 2022-04-03T19:48:33.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.213+0000 7f720359e700 1 -- 172.21.15.77:0/3925020793 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f40092a0 con 0x7f72040678b0 2022-04-03T19:48:33.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.213+0000 7f720359e700 1 -- 172.21.15.77:0/3925020793 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72040b8c00 con 0x7f72040678b0 2022-04-03T19:48:33.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.214+0000 7f720359e700 1 -- 172.21.15.77:0/3925020793 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72040c74a0 con 0x7f72040678b0 2022-04-03T19:48:33.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.215+0000 7f71f27fc700 1 -- 172.21.15.77:0/3925020793 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f71f402c030 con 0x7f72040678b0 2022-04-03T19:48:33.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.216+0000 7f71f27fc700 1 --2- 172.21.15.77:0/3925020793 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f71e0063ce0 0x7f71e0066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:33.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.216+0000 7f71f27fc700 1 -- 172.21.15.77:0/3925020793 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6288+0+0 (secure 0 0 0) 0x7f71f4082080 con 0x7f72040678b0 2022-04-03T19:48:33.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.216+0000 7f7201d9b700 1 --2- 172.21.15.77:0/3925020793 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f71e0063ce0 0x7f71e0066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:33.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.217+0000 7f7201d9b700 1 --2- 172.21.15.77:0/3925020793 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f71e0063ce0 0x7f71e0066190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7204150730 tx=0x7f71f8007bf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:33.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.220+0000 7f71f27fc700 1 -- 172.21.15.77:0/3925020793 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f71f4050b80 con 0x7f72040678b0 2022-04-03T19:48:33.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.448+0000 7f720359e700 1 -- 172.21.15.77:0/3925020793 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7204154270 con 0x7f72040678b0 2022-04-03T19:48:33.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.449+0000 7f71f27fc700 1 -- 172.21.15.77:0/3925020793 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5353 (secure 0 0 0) 0x7f71f402a090 con 0x7f72040678b0 2022-04-03T19:48:33.450 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:33.450 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:48:31.242953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4960},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4960":{"gid":4960,"name":"e","rank":0,"incarnation":15,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:33.452 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.451+0000 7f720359e700 1 -- 172.21.15.77:0/3925020793 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f71e0063ce0 msgr2=0x7f71e0066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:33.453 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.451+0000 7f720359e700 1 --2- 172.21.15.77:0/3925020793 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f71e0063ce0 0x7f71e0066190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7204150730 tx=0x7f71f8007bf0 comp rx=0 tx=0).stop 2022-04-03T19:48:33.453 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.452+0000 7f720359e700 1 -- 172.21.15.77:0/3925020793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72040678b0 msgr2=0x7f72040c3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:33.453 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.452+0000 7f720359e700 1 --2- 172.21.15.77:0/3925020793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72040678b0 0x7f72040c3c30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f71f40091e0 tx=0x7f71f4017000 comp rx=0 tx=0).stop 2022-04-03T19:48:33.453 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.452+0000 7f720359e700 1 -- 172.21.15.77:0/3925020793 shutdown_connections 2022-04-03T19:48:33.454 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.452+0000 7f720359e700 1 --2- 172.21.15.77:0/3925020793 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f71e0063ce0 0x7f71e0066190 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:33.454 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.452+0000 7f720359e700 1 --2- 172.21.15.77:0/3925020793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7204157060 0x7f720414a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:33.454 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.452+0000 7f720359e700 1 --2- 172.21.15.77:0/3925020793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72040678b0 0x7f72040c3c30 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:33.454 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.452+0000 7f720359e700 1 --2- 172.21.15.77:0/3925020793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72040ba3c0 0x7f7204149c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:33.455 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.452+0000 7f720359e700 1 -- 172.21.15.77:0/3925020793 >> 172.21.15.77:0/3925020793 conn(0x7f7204066da0 msgr2=0x7f720414c000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:33.455 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.452+0000 7f720359e700 1 -- 172.21.15.77:0/3925020793 shutdown_connections 2022-04-03T19:48:33.455 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:33.452+0000 7f720359e700 1 -- 172.21.15.77:0/3925020793 wait complete. 2022-04-03T19:48:33.456 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 17 2022-04-03T19:48:33.477 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-04-03T19:44:21.275716+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4960": { "addr": "172.21.15.77:6837/2256447697", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6836", "nonce": 2256447697, "type": "v2" }, { "addr": "172.21.15.77:6837", "nonce": 2256447697, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4960, "incarnation": 15, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:rejoin", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-04-03T19:48:31.242953+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4960 } } } ], "standbys": [ { "addr": "172.21.15.77:6834/2116223509", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6832", "nonce": 2116223509, "type": "v2" }, { "addr": "172.21.15.77:6834", "nonce": 2116223509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4245, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6836/2374276747", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2374276747, "type": "v2" }, { "addr": "172.21.15.186:6836", "nonce": 2374276747, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4259, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/610990853", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6838", "nonce": 610990853, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 610990853, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4265, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.77:6835/170053412", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6833", "nonce": 170053412, "type": "v2" }, { "addr": "172.21.15.77:6835", "nonce": 170053412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4271, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-04-03T19:48:35.478 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:35.684 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.683+0000 7fd61ffff700 1 Processor -- start 2022-04-03T19:48:35.684 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.683+0000 7fd61ffff700 1 -- start start 2022-04-03T19:48:35.686 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.684+0000 7fd61ffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd62006d3f0 0x7fd62006d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:35.686 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.684+0000 7fd61ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd6200c5930 0x7fd620155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:35.686 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.684+0000 7fd61ffff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6200c2f80 con 0x7fd620156120 2022-04-03T19:48:35.686 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.684+0000 7fd61ffff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6200c3260 con 0x7fd6200c5930 2022-04-03T19:48:35.687 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.684+0000 7fd61ffff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6200c3540 con 0x7fd62006d3f0 2022-04-03T19:48:35.687 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.685+0000 7fd617fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd6200c5930 0x7fd620155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:35.687 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.685+0000 7fd617fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd6200c5930 0x7fd620155be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42864/0 (socket says 172.21.15.77:42864) 2022-04-03T19:48:35.687 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.685+0000 7fd617fff700 1 -- 172.21.15.77:0/190698223 learned_addr learned my addr 172.21.15.77:0/190698223 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:35.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.685+0000 7fd61effd700 1 --2- 172.21.15.77:0/190698223 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd62006d3f0 0x7fd62006d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:35.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.685+0000 7fd61f7fe700 1 --1- 172.21.15.77:0/190698223 >> v1:172.21.15.77:6789/0 conn(0x7fd620156120 0x7fd62006c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38442/0 (socket says 172.21.15.77:38442) 2022-04-03T19:48:35.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.685+0000 7fd617fff700 1 -- 172.21.15.77:0/190698223 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd62006d3f0 msgr2=0x7fd62006d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:48:35.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.685+0000 7fd617fff700 1 --2- 172.21.15.77:0/190698223 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd62006d3f0 0x7fd62006d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:35.689 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.685+0000 7fd617fff700 1 -- 172.21.15.77:0/190698223 >> v1:172.21.15.77:6789/0 conn(0x7fd620156120 legacy=0x7fd62006c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:35.689 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.685+0000 7fd617fff700 1 -- 172.21.15.77:0/190698223 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6200c3820 con 0x7fd6200c5930 2022-04-03T19:48:35.689 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.686+0000 7fd617fff700 1 --2- 172.21.15.77:0/190698223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6200c5930 0x7fd620155be0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd608009830 tx=0x7fd6080090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9af7484bd57c3478 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:35.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.686+0000 7fd61e7fc700 1 -- 172.21.15.77:0/190698223 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd608004030 con 0x7fd6200c5930 2022-04-03T19:48:35.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.686+0000 7fd61e7fc700 1 -- 172.21.15.77:0/190698223 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd608004190 con 0x7fd6200c5930 2022-04-03T19:48:35.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.686+0000 7fd61e7fc700 1 -- 172.21.15.77:0/190698223 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd608011630 con 0x7fd6200c5930 2022-04-03T19:48:35.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.686+0000 7fd61ffff700 1 -- 172.21.15.77:0/190698223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6200c5930 msgr2=0x7fd620155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:35.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.686+0000 7fd61ffff700 1 --2- 172.21.15.77:0/190698223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6200c5930 0x7fd620155be0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd608009830 tx=0x7fd6080090f0 comp rx=0 tx=0).stop 2022-04-03T19:48:35.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.686+0000 7fd61ffff700 1 -- 172.21.15.77:0/190698223 shutdown_connections 2022-04-03T19:48:35.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.686+0000 7fd61ffff700 1 --2- 172.21.15.77:0/190698223 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd62006d3f0 0x7fd62006d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:35.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.686+0000 7fd61ffff700 1 --2- 172.21.15.77:0/190698223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6200c5930 0x7fd620155be0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:35.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.686+0000 7fd61ffff700 1 -- 172.21.15.77:0/190698223 >> 172.21.15.77:0/190698223 conn(0x7fd6200cb720 msgr2=0x7fd6200cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:35.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.687+0000 7fd61ffff700 1 -- 172.21.15.77:0/190698223 shutdown_connections 2022-04-03T19:48:35.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.687+0000 7fd61ffff700 1 -- 172.21.15.77:0/190698223 wait complete. 2022-04-03T19:48:35.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.688+0000 7fd61ffff700 1 Processor -- start 2022-04-03T19:48:35.696 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.688+0000 7fd61ffff700 1 -- start start 2022-04-03T19:48:35.696 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.689+0000 7fd61ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd62006d3f0 0x7fd620157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:35.696 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.689+0000 7fd61ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6200c5930 0x7fd620168ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:35.696 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.689+0000 7fd61ffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd620156120 0x7fd62014a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:35.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.689+0000 7fd61ffff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6201569e0 con 0x7fd62006d3f0 2022-04-03T19:48:35.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.689+0000 7fd61ffff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6200c1900 con 0x7fd6200c5930 2022-04-03T19:48:35.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.689+0000 7fd61ffff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6201699e0 con 0x7fd620156120 2022-04-03T19:48:35.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.689+0000 7fd61effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd62006d3f0 0x7fd620157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:35.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.689+0000 7fd61f7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd620156120 0x7fd62014a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:35.698 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.689+0000 7fd61effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd62006d3f0 0x7fd620157b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42866/0 (socket says 172.21.15.77:42866) 2022-04-03T19:48:35.698 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.689+0000 7fd61effd700 1 -- 172.21.15.77:0/1022910834 learned_addr learned my addr 172.21.15.77:0/1022910834 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:35.698 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.689+0000 7fd617fff700 1 --2- 172.21.15.77:0/1022910834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6200c5930 0x7fd620168ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:35.698 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.690+0000 7fd61effd700 1 -- 172.21.15.77:0/1022910834 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd620156120 msgr2=0x7fd62014a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:35.699 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.690+0000 7fd61effd700 1 --2- 172.21.15.77:0/1022910834 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd620156120 0x7fd62014a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:35.699 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.690+0000 7fd61effd700 1 -- 172.21.15.77:0/1022910834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6200c5930 msgr2=0x7fd620168ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:35.699 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.690+0000 7fd61effd700 1 --2- 172.21.15.77:0/1022910834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6200c5930 0x7fd620168ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:35.699 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.690+0000 7fd61effd700 1 -- 172.21.15.77:0/1022910834 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd608009370 con 0x7fd62006d3f0 2022-04-03T19:48:35.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.690+0000 7fd61f7fe700 1 --2- 172.21.15.77:0/1022910834 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd620156120 0x7fd62014a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:48:35.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.690+0000 7fd61effd700 1 --2- 172.21.15.77:0/1022910834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd62006d3f0 0x7fd620157b40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd61000e960 tx=0x7fd610009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:35.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.690+0000 7fd616ffd700 1 -- 172.21.15.77:0/1022910834 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd610013680 con 0x7fd62006d3f0 2022-04-03T19:48:35.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.690+0000 7fd616ffd700 1 -- 172.21.15.77:0/1022910834 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6100137e0 con 0x7fd62006d3f0 2022-04-03T19:48:35.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.691+0000 7fd61ffff700 1 -- 172.21.15.77:0/1022910834 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6200c1d90 con 0x7fd62006d3f0 2022-04-03T19:48:35.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.691+0000 7fd616ffd700 1 -- 172.21.15.77:0/1022910834 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd610013b60 con 0x7fd62006d3f0 2022-04-03T19:48:35.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.691+0000 7fd61ffff700 1 -- 172.21.15.77:0/1022910834 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd620169790 con 0x7fd62006d3f0 2022-04-03T19:48:35.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.692+0000 7fd61ffff700 1 -- 172.21.15.77:0/1022910834 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6200c1be0 con 0x7fd62006d3f0 2022-04-03T19:48:35.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.693+0000 7fd616ffd700 1 -- 172.21.15.77:0/1022910834 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fd610019030 con 0x7fd62006d3f0 2022-04-03T19:48:35.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.693+0000 7fd616ffd700 1 --2- 172.21.15.77:0/1022910834 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd604063ce0 0x7fd604066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:35.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.694+0000 7fd616ffd700 1 -- 172.21.15.77:0/1022910834 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6288+0+0 (secure 0 0 0) 0x7fd610080980 con 0x7fd62006d3f0 2022-04-03T19:48:35.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.694+0000 7fd617fff700 1 --2- 172.21.15.77:0/1022910834 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd604063ce0 0x7fd604066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:35.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.695+0000 7fd617fff700 1 --2- 172.21.15.77:0/1022910834 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd604063ce0 0x7fd604066190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd608009830 tx=0x7fd608016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:35.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.698+0000 7fd616ffd700 1 -- 172.21.15.77:0/1022910834 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fd61004f420 con 0x7fd62006d3f0 2022-04-03T19:48:35.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.930+0000 7fd61ffff700 1 -- 172.21.15.77:0/1022910834 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd620155c90 con 0x7fd62006d3f0 2022-04-03T19:48:35.931 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.930+0000 7fd616ffd700 1 -- 172.21.15.77:0/1022910834 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5353 (secure 0 0 0) 0x7fd610017090 con 0x7fd62006d3f0 2022-04-03T19:48:35.931 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:35.932 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:48:31.242953+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4960},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4960":{"gid":4960,"name":"e","rank":0,"incarnation":15,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:35.934 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.933+0000 7fd61ffff700 1 -- 172.21.15.77:0/1022910834 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd604063ce0 msgr2=0x7fd604066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:35.934 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.933+0000 7fd61ffff700 1 --2- 172.21.15.77:0/1022910834 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd604063ce0 0x7fd604066190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd608009830 tx=0x7fd608016000 comp rx=0 tx=0).stop 2022-04-03T19:48:35.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.933+0000 7fd61ffff700 1 -- 172.21.15.77:0/1022910834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd62006d3f0 msgr2=0x7fd620157b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:35.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.933+0000 7fd61ffff700 1 --2- 172.21.15.77:0/1022910834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd62006d3f0 0x7fd620157b40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd61000e960 tx=0x7fd610009060 comp rx=0 tx=0).stop 2022-04-03T19:48:35.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.933+0000 7fd61ffff700 1 -- 172.21.15.77:0/1022910834 shutdown_connections 2022-04-03T19:48:35.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.933+0000 7fd61ffff700 1 --2- 172.21.15.77:0/1022910834 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd604063ce0 0x7fd604066190 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:35.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.933+0000 7fd61ffff700 1 --2- 172.21.15.77:0/1022910834 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd620156120 0x7fd62014a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:35.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.933+0000 7fd61ffff700 1 --2- 172.21.15.77:0/1022910834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd62006d3f0 0x7fd620157b40 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:35.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.933+0000 7fd61ffff700 1 --2- 172.21.15.77:0/1022910834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6200c5930 0x7fd620168ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:35.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.933+0000 7fd61ffff700 1 -- 172.21.15.77:0/1022910834 >> 172.21.15.77:0/1022910834 conn(0x7fd6200cb720 msgr2=0x7fd6201531f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:35.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.933+0000 7fd61ffff700 1 -- 172.21.15.77:0/1022910834 shutdown_connections 2022-04-03T19:48:35.937 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:35.933+0000 7fd61ffff700 1 -- 172.21.15.77:0/1022910834 wait complete. 2022-04-03T19:48:35.937 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 17 2022-04-03T19:48:35.958 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:48:37.959 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:38.164 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.163+0000 7f4565fa9700 1 Processor -- start 2022-04-03T19:48:38.164 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.164+0000 7f4565fa9700 1 -- start start 2022-04-03T19:48:38.165 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.165+0000 7f4565fa9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45600ba3c0 0x7f4560156b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:38.166 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.165+0000 7f4565fa9700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f45600678b0 0x7f456014d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:38.166 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.165+0000 7f4565fa9700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45600c3c50 con 0x7f4560157060 2022-04-03T19:48:38.166 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.165+0000 7f4565fa9700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45600c8020 con 0x7f45600678b0 2022-04-03T19:48:38.166 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.165+0000 7f4565fa9700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4560156690 con 0x7f45600ba3c0 2022-04-03T19:48:38.167 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.165+0000 7f4564fa7700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45600ba3c0 0x7f4560156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:38.167 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.165+0000 7f45657a8700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f45600678b0 0x7f456014d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:38.167 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.165+0000 7f455ffff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f4560157060 0x7f456014cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38444/0 (socket says 172.21.15.77:38444) 2022-04-03T19:48:38.167 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.165+0000 7f455ffff700 1 -- 172.21.15.77:0/1031669342 learned_addr learned my addr 172.21.15.77:0/1031669342 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:48:38.170 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.166+0000 7f4564fa7700 1 -- 172.21.15.77:0/1031669342 >> v2:172.21.15.186:3300/0 conn(0x7f45600678b0 msgr2=0x7f456014d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:48:38.170 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.166+0000 7f4564fa7700 1 --2- 172.21.15.77:0/1031669342 >> v2:172.21.15.186:3300/0 conn(0x7f45600678b0 0x7f456014d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:38.170 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.166+0000 7f455f7fe700 1 -- 172.21.15.77:0/1031669342 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473040242 0 0) 0x7f45600c3c50 con 0x7f4560157060 2022-04-03T19:48:38.170 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.166+0000 7f4564fa7700 1 -- 172.21.15.77:0/1031669342 >> v1:172.21.15.77:6789/0 conn(0x7f4560157060 legacy=0x7f456014cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:38.171 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.166+0000 7f4564fa7700 1 -- 172.21.15.77:0/1031669342 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45600bf350 con 0x7f45600ba3c0 2022-04-03T19:48:38.171 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.166+0000 7f45657a8700 1 --2- 172.21.15.77:0/1031669342 >> v2:172.21.15.186:3300/0 conn(0x7f45600678b0 0x7f456014d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:48:38.171 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.166+0000 7f4564fa7700 1 --2- 172.21.15.77:0/1031669342 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45600ba3c0 0x7f4560156b20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4554009760 tx=0x7f4554009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e655ff9c96b5f28 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:38.171 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.166+0000 7f455f7fe700 1 -- 172.21.15.77:0/1031669342 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554011630 con 0x7f45600ba3c0 2022-04-03T19:48:38.171 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.166+0000 7f455f7fe700 1 -- 172.21.15.77:0/1031669342 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4554011790 con 0x7f45600ba3c0 2022-04-03T19:48:38.172 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.166+0000 7f455f7fe700 1 -- 172.21.15.77:0/1031669342 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554011b50 con 0x7f45600ba3c0 2022-04-03T19:48:38.172 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.167+0000 7f4565fa9700 1 -- 172.21.15.77:0/1031669342 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45600ba3c0 msgr2=0x7f4560156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:38.172 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.167+0000 7f4565fa9700 1 --2- 172.21.15.77:0/1031669342 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45600ba3c0 0x7f4560156b20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4554009760 tx=0x7f4554009060 comp rx=0 tx=0).stop 2022-04-03T19:48:38.172 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.167+0000 7f4565fa9700 1 -- 172.21.15.77:0/1031669342 shutdown_connections 2022-04-03T19:48:38.173 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.167+0000 7f4565fa9700 1 --2- 172.21.15.77:0/1031669342 >> v2:172.21.15.186:3300/0 conn(0x7f45600678b0 0x7f456014d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:38.173 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.167+0000 7f4565fa9700 1 --2- 172.21.15.77:0/1031669342 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45600ba3c0 0x7f4560156b20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:38.173 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.167+0000 7f4565fa9700 1 -- 172.21.15.77:0/1031669342 >> 172.21.15.77:0/1031669342 conn(0x7f4560066da0 msgr2=0x7f45600671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:38.173 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.167+0000 7f4565fa9700 1 -- 172.21.15.77:0/1031669342 shutdown_connections 2022-04-03T19:48:38.173 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.167+0000 7f4565fa9700 1 -- 172.21.15.77:0/1031669342 wait complete. 2022-04-03T19:48:38.174 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.168+0000 7f4565fa9700 1 Processor -- start 2022-04-03T19:48:38.174 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.169+0000 7f4565fa9700 1 -- start start 2022-04-03T19:48:38.174 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.169+0000 7f4565fa9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45600678b0 0x7f4560069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:38.174 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.169+0000 7f4565fa9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45600ba3c0 0x7f45600c3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:38.175 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.169+0000 7f4565fa9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4560157060 0x7f456006aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:38.175 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.170+0000 7f4565fa9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456014f330 con 0x7f45600678b0 2022-04-03T19:48:38.175 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.170+0000 7f4565fa9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45600c3650 con 0x7f45600ba3c0 2022-04-03T19:48:38.175 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.170+0000 7f4565fa9700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f456006a590 con 0x7f4560157060 2022-04-03T19:48:38.178 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.170+0000 7f4564fa7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45600678b0 0x7f4560069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:38.178 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.170+0000 7f4564fa7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45600678b0 0x7f4560069410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42868/0 (socket says 172.21.15.77:42868) 2022-04-03T19:48:38.178 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.170+0000 7f4564fa7700 1 -- 172.21.15.77:0/2565461133 learned_addr learned my addr 172.21.15.77:0/2565461133 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:38.178 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.170+0000 7f45657a8700 1 --2- 172.21.15.77:0/2565461133 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4560157060 0x7f456006aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:38.179 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.170+0000 7f455ffff700 1 --2- 172.21.15.77:0/2565461133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45600ba3c0 0x7f45600c3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:38.179 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.170+0000 7f4564fa7700 1 -- 172.21.15.77:0/2565461133 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4560157060 msgr2=0x7f456006aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:38.179 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.170+0000 7f4564fa7700 1 --2- 172.21.15.77:0/2565461133 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4560157060 0x7f456006aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:38.179 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.170+0000 7f4564fa7700 1 -- 172.21.15.77:0/2565461133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45600ba3c0 msgr2=0x7f45600c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:38.179 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.170+0000 7f4564fa7700 1 --2- 172.21.15.77:0/2565461133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45600ba3c0 0x7f45600c3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:38.180 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.170+0000 7f4564fa7700 1 -- 172.21.15.77:0/2565461133 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4558009a20 con 0x7f45600678b0 2022-04-03T19:48:38.180 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.171+0000 7f4564fa7700 1 --2- 172.21.15.77:0/2565461133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45600678b0 0x7f4560069410 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f45540091e0 tx=0x7f4554017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:38.180 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.171+0000 7f455cff9700 1 -- 172.21.15.77:0/2565461133 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554000bc0 con 0x7f45600678b0 2022-04-03T19:48:38.180 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.171+0000 7f455cff9700 1 -- 172.21.15.77:0/2565461133 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4554011900 con 0x7f45600678b0 2022-04-03T19:48:38.181 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.171+0000 7f4565fa9700 1 -- 172.21.15.77:0/2565461133 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45540092a0 con 0x7f45600678b0 2022-04-03T19:48:38.181 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.171+0000 7f455cff9700 1 -- 172.21.15.77:0/2565461133 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554025420 con 0x7f45600678b0 2022-04-03T19:48:38.181 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.171+0000 7f4565fa9700 1 -- 172.21.15.77:0/2565461133 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f456016d0d0 con 0x7f45600678b0 2022-04-03T19:48:38.181 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.172+0000 7f4565fa9700 1 -- 172.21.15.77:0/2565461133 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f456006a870 con 0x7f45600678b0 2022-04-03T19:48:38.181 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.173+0000 7f455cff9700 1 -- 172.21.15.77:0/2565461133 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f455402c030 con 0x7f45600678b0 2022-04-03T19:48:38.182 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.174+0000 7f455cff9700 1 --2- 172.21.15.77:0/2565461133 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4548063ce0 0x7f4548066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:38.182 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.174+0000 7f455cff9700 1 -- 172.21.15.77:0/2565461133 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6288+0+0 (secure 0 0 0) 0x7f4554081fe0 con 0x7f45600678b0 2022-04-03T19:48:38.182 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.174+0000 7f455ffff700 1 --2- 172.21.15.77:0/2565461133 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4548063ce0 0x7f4548066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:38.182 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.175+0000 7f455ffff700 1 --2- 172.21.15.77:0/2565461133 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4548063ce0 0x7f4548066190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4560163c20 tx=0x7f454c007cd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:38.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.178+0000 7f455cff9700 1 -- 172.21.15.77:0/2565461133 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f4554050b10 con 0x7f45600678b0 2022-04-03T19:48:38.407 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.407+0000 7f4565fa9700 1 -- 172.21.15.77:0/2565461133 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45600c74a0 con 0x7f45600678b0 2022-04-03T19:48:38.408 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.407+0000 7f455cff9700 1 -- 172.21.15.77:0/2565461133 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+5359 (secure 0 0 0) 0x7f455402a070 con 0x7f45600678b0 2022-04-03T19:48:38.408 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:38.408 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:48:37.143133+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4960},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4960":{"gid":4960,"name":"e","rank":0,"incarnation":15,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:38.410 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.410+0000 7f4565fa9700 1 -- 172.21.15.77:0/2565461133 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4548063ce0 msgr2=0x7f4548066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:38.411 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.410+0000 7f4565fa9700 1 --2- 172.21.15.77:0/2565461133 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4548063ce0 0x7f4548066190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4560163c20 tx=0x7f454c007cd0 comp rx=0 tx=0).stop 2022-04-03T19:48:38.411 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.410+0000 7f4565fa9700 1 -- 172.21.15.77:0/2565461133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45600678b0 msgr2=0x7f4560069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:38.411 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.410+0000 7f4565fa9700 1 --2- 172.21.15.77:0/2565461133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45600678b0 0x7f4560069410 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f45540091e0 tx=0x7f4554017000 comp rx=0 tx=0).stop 2022-04-03T19:48:38.411 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.410+0000 7f4565fa9700 1 -- 172.21.15.77:0/2565461133 shutdown_connections 2022-04-03T19:48:38.412 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.410+0000 7f4565fa9700 1 --2- 172.21.15.77:0/2565461133 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4548063ce0 0x7f4548066190 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:38.412 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.410+0000 7f4565fa9700 1 --2- 172.21.15.77:0/2565461133 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4560157060 0x7f456006aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:38.412 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.410+0000 7f4565fa9700 1 --2- 172.21.15.77:0/2565461133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45600678b0 0x7f4560069410 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:38.412 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.410+0000 7f4565fa9700 1 --2- 172.21.15.77:0/2565461133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45600ba3c0 0x7f45600c3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:38.412 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.410+0000 7f4565fa9700 1 -- 172.21.15.77:0/2565461133 >> 172.21.15.77:0/2565461133 conn(0x7f4560066da0 msgr2=0x7f456014c000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:38.413 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.410+0000 7f4565fa9700 1 -- 172.21.15.77:0/2565461133 shutdown_connections 2022-04-03T19:48:38.413 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:38.410+0000 7f4565fa9700 1 -- 172.21.15.77:0/2565461133 wait complete. 2022-04-03T19:48:38.413 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 18 2022-04-03T19:48:38.434 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:48:40.435 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:40.640 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.639+0000 7f78a5bd5700 1 Processor -- start 2022-04-03T19:48:40.640 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.639+0000 7f78a5bd5700 1 -- start start 2022-04-03T19:48:40.642 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.640+0000 7f78a5bd5700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78a006bbe0 0x7f78a01584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:40.642 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.641+0000 7f78a5bd5700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f78a006c500 0x7f78a014eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:40.642 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.641+0000 7f78a5bd5700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a006d260 con 0x7f78a006bfb0 2022-04-03T19:48:40.642 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.641+0000 7f78a5bd5700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a0159450 con 0x7f78a006c500 2022-04-03T19:48:40.643 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.641+0000 7f78a5bd5700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a00c6330 con 0x7f78a006bbe0 2022-04-03T19:48:40.643 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.641+0000 7f789ffff700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f78a006bfb0 0x7f78a006d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44236/0 (socket says 172.21.15.77:44236) 2022-04-03T19:48:40.643 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.641+0000 7f78a53d4700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f78a006c500 0x7f78a014eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:40.643 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.641+0000 7f78a4bd3700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78a006bbe0 0x7f78a01584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:40.644 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.641+0000 7f789ffff700 1 -- 172.21.15.77:0/1112809840 learned_addr learned my addr 172.21.15.77:0/1112809840 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:48:40.644 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.641+0000 7f789f7fe700 1 -- 172.21.15.77:0/1112809840 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4046656796 0 0) 0x7f78a006d260 con 0x7f78a006bfb0 2022-04-03T19:48:40.644 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.641+0000 7f789f7fe700 1 -- 172.21.15.77:0/1112809840 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78840033f0 con 0x7f78a006bfb0 2022-04-03T19:48:40.644 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.641+0000 7f78a53d4700 1 -- 172.21.15.77:0/1112809840 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78a006bbe0 msgr2=0x7f78a01584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:48:40.645 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.641+0000 7f78a53d4700 1 --2- 172.21.15.77:0/1112809840 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78a006bbe0 0x7f78a01584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:40.645 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.641+0000 7f78a53d4700 1 -- 172.21.15.77:0/1112809840 >> v1:172.21.15.186:6789/0 conn(0x7f78a006bfb0 legacy=0x7f78a006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:40.645 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.641+0000 7f78a53d4700 1 -- 172.21.15.77:0/1112809840 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a0158bb0 con 0x7f78a006c500 2022-04-03T19:48:40.645 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.642+0000 7f78a53d4700 1 --2- 172.21.15.77:0/1112809840 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78a006c500 0x7f78a014eea0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7898009e60 tx=0x7f7898009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=234c07d11bfadceb server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:40.648 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.642+0000 7f789f7fe700 1 -- 172.21.15.77:0/1112809840 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789800e070 con 0x7f78a006c500 2022-04-03T19:48:40.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.642+0000 7f789f7fe700 1 -- 172.21.15.77:0/1112809840 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789800bab0 con 0x7f78a006c500 2022-04-03T19:48:40.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.642+0000 7f789f7fe700 1 -- 172.21.15.77:0/1112809840 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789800be30 con 0x7f78a006c500 2022-04-03T19:48:40.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.642+0000 7f78a5bd5700 1 -- 172.21.15.77:0/1112809840 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78a006c500 msgr2=0x7f78a014eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:40.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.642+0000 7f78a5bd5700 1 --2- 172.21.15.77:0/1112809840 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78a006c500 0x7f78a014eea0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7898009e60 tx=0x7f7898009780 comp rx=0 tx=0).stop 2022-04-03T19:48:40.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.643+0000 7f78a5bd5700 1 -- 172.21.15.77:0/1112809840 shutdown_connections 2022-04-03T19:48:40.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.643+0000 7f78a5bd5700 1 --2- 172.21.15.77:0/1112809840 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78a006c500 0x7f78a014eea0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:40.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.643+0000 7f78a5bd5700 1 --2- 172.21.15.77:0/1112809840 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78a006bbe0 0x7f78a01584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:40.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.643+0000 7f78a5bd5700 1 -- 172.21.15.77:0/1112809840 >> 172.21.15.77:0/1112809840 conn(0x7f78a00cb6b0 msgr2=0x7f78a00cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:40.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.643+0000 7f78a5bd5700 1 -- 172.21.15.77:0/1112809840 shutdown_connections 2022-04-03T19:48:40.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.643+0000 7f78a5bd5700 1 -- 172.21.15.77:0/1112809840 wait complete. 2022-04-03T19:48:40.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.644+0000 7f78a5bd5700 1 Processor -- start 2022-04-03T19:48:40.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.644+0000 7f78a5bd5700 1 -- start start 2022-04-03T19:48:40.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.645+0000 7f78a5bd5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78a006bbe0 0x7f78a0154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:40.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.645+0000 7f78a5bd5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78a006bfb0 0x7f78a01682d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:40.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.645+0000 7f78a5bd5700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78a006c500 0x7f78a0148bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:40.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.645+0000 7f78a5bd5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a01546d0 con 0x7f78a006bbe0 2022-04-03T19:48:40.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.645+0000 7f78a5bd5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a0153d00 con 0x7f78a006bfb0 2022-04-03T19:48:40.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.645+0000 7f78a5bd5700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a014a350 con 0x7f78a006c500 2022-04-03T19:48:40.653 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.645+0000 7f78a4bd3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78a006bbe0 0x7f78a0154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:40.653 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.646+0000 7f78a4bd3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78a006bbe0 0x7f78a0154190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42872/0 (socket says 172.21.15.77:42872) 2022-04-03T19:48:40.653 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.646+0000 7f78a4bd3700 1 -- 172.21.15.77:0/2017341957 learned_addr learned my addr 172.21.15.77:0/2017341957 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:40.653 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.646+0000 7f789ffff700 1 --2- 172.21.15.77:0/2017341957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78a006bfb0 0x7f78a01682d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:40.654 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.646+0000 7f78a53d4700 1 --2- 172.21.15.77:0/2017341957 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78a006c500 0x7f78a0148bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:40.654 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.646+0000 7f78a4bd3700 1 -- 172.21.15.77:0/2017341957 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78a006c500 msgr2=0x7f78a0148bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:40.654 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.646+0000 7f78a4bd3700 1 --2- 172.21.15.77:0/2017341957 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78a006c500 0x7f78a0148bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:40.654 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.646+0000 7f78a4bd3700 1 -- 172.21.15.77:0/2017341957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78a006bfb0 msgr2=0x7f78a01682d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:40.654 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.646+0000 7f78a4bd3700 1 --2- 172.21.15.77:0/2017341957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78a006bfb0 0x7f78a01682d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:40.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.646+0000 7f78a4bd3700 1 -- 172.21.15.77:0/2017341957 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7898009a20 con 0x7f78a006bbe0 2022-04-03T19:48:40.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.646+0000 7f78a4bd3700 1 --2- 172.21.15.77:0/2017341957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78a006bbe0 0x7f78a0154190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f789400e930 tx=0x7f7894009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:40.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.647+0000 7f789cff9700 1 -- 172.21.15.77:0/2017341957 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7894013680 con 0x7f78a006bbe0 2022-04-03T19:48:40.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.647+0000 7f789cff9700 1 -- 172.21.15.77:0/2017341957 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78940137e0 con 0x7f78a006bbe0 2022-04-03T19:48:40.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.647+0000 7f78a5bd5700 1 -- 172.21.15.77:0/2017341957 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a0158270 con 0x7f78a006bbe0 2022-04-03T19:48:40.656 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.647+0000 7f789cff9700 1 -- 172.21.15.77:0/2017341957 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7894013b60 con 0x7f78a006bbe0 2022-04-03T19:48:40.656 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.647+0000 7f78a5bd5700 1 -- 172.21.15.77:0/2017341957 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a0168c50 con 0x7f78a006bbe0 2022-04-03T19:48:40.656 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.648+0000 7f78a5bd5700 1 -- 172.21.15.77:0/2017341957 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a01584a0 con 0x7f78a006bbe0 2022-04-03T19:48:40.656 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.648+0000 7f789cff9700 1 -- 172.21.15.77:0/2017341957 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f7894019030 con 0x7f78a006bbe0 2022-04-03T19:48:40.657 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.649+0000 7f789cff9700 1 --2- 172.21.15.77:0/2017341957 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7888063d10 0x7f78880661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:40.658 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.649+0000 7f789cff9700 1 -- 172.21.15.77:0/2017341957 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6288+0+0 (secure 0 0 0) 0x7f7894080880 con 0x7f78a006bbe0 2022-04-03T19:48:40.658 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.650+0000 7f789ffff700 1 --2- 172.21.15.77:0/2017341957 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7888063d10 0x7f78880661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:40.658 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.650+0000 7f789ffff700 1 --2- 172.21.15.77:0/2017341957 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7888063d10 0x7f78880661c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f78a014aef0 tx=0x7f788c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:40.658 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.654+0000 7f789cff9700 1 -- 172.21.15.77:0/2017341957 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f789404f2b0 con 0x7f78a006bbe0 2022-04-03T19:48:40.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.885+0000 7f78a5bd5700 1 -- 172.21.15.77:0/2017341957 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78a006d960 con 0x7f78a006bbe0 2022-04-03T19:48:40.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.886+0000 7f789cff9700 1 -- 172.21.15.77:0/2017341957 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+5359 (secure 0 0 0) 0x7f78940528d0 con 0x7f78a006bbe0 2022-04-03T19:48:40.887 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:40.887 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:48:37.143133+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4960},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4960":{"gid":4960,"name":"e","rank":0,"incarnation":15,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:40.889 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.888+0000 7f78a5bd5700 1 -- 172.21.15.77:0/2017341957 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7888063d10 msgr2=0x7f78880661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:40.889 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.888+0000 7f78a5bd5700 1 --2- 172.21.15.77:0/2017341957 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7888063d10 0x7f78880661c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f78a014aef0 tx=0x7f788c009040 comp rx=0 tx=0).stop 2022-04-03T19:48:40.889 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.888+0000 7f78a5bd5700 1 -- 172.21.15.77:0/2017341957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78a006bbe0 msgr2=0x7f78a0154190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:40.889 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.888+0000 7f78a5bd5700 1 --2- 172.21.15.77:0/2017341957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78a006bbe0 0x7f78a0154190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f789400e930 tx=0x7f7894009060 comp rx=0 tx=0).stop 2022-04-03T19:48:40.890 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.888+0000 7f78a5bd5700 1 -- 172.21.15.77:0/2017341957 shutdown_connections 2022-04-03T19:48:40.890 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.888+0000 7f78a5bd5700 1 --2- 172.21.15.77:0/2017341957 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7888063d10 0x7f78880661c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:40.890 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.888+0000 7f78a5bd5700 1 --2- 172.21.15.77:0/2017341957 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78a006c500 0x7f78a0148bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:40.891 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.888+0000 7f78a5bd5700 1 --2- 172.21.15.77:0/2017341957 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78a006bbe0 0x7f78a0154190 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:40.891 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.888+0000 7f78a5bd5700 1 --2- 172.21.15.77:0/2017341957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78a006bfb0 0x7f78a01682d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:40.891 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.888+0000 7f78a5bd5700 1 -- 172.21.15.77:0/2017341957 >> 172.21.15.77:0/2017341957 conn(0x7f78a00cb6b0 msgr2=0x7f78a014b550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:40.891 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.888+0000 7f78a5bd5700 1 -- 172.21.15.77:0/2017341957 shutdown_connections 2022-04-03T19:48:40.892 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:40.888+0000 7f78a5bd5700 1 -- 172.21.15.77:0/2017341957 wait complete. 2022-04-03T19:48:40.892 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 18 2022-04-03T19:48:40.912 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:48:42.914 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:43.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.111+0000 7f1baa7b3700 1 Processor -- start 2022-04-03T19:48:43.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.111+0000 7f1baa7b3700 1 -- start start 2022-04-03T19:48:43.115 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.112+0000 7f1baa7b3700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1ba406d3f0 0x7f1ba406d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:43.115 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.112+0000 7f1baa7b3700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1ba40c5930 0x7f1ba4155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:43.116 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.112+0000 7f1baa7b3700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba40c2f80 con 0x7f1ba4156120 2022-04-03T19:48:43.116 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.112+0000 7f1baa7b3700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba40c3260 con 0x7f1ba40c5930 2022-04-03T19:48:43.116 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.112+0000 7f1baa7b3700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba40c3540 con 0x7f1ba406d3f0 2022-04-03T19:48:43.116 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.113+0000 7f1ba8fb0700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1ba40c5930 0x7f1ba4155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:43.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.113+0000 7f1ba8fb0700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1ba40c5930 0x7f1ba4155be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42874/0 (socket says 172.21.15.77:42874) 2022-04-03T19:48:43.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.113+0000 7f1ba8fb0700 1 -- 172.21.15.77:0/911948159 learned_addr learned my addr 172.21.15.77:0/911948159 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:43.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.113+0000 7f1ba97b1700 1 --2- 172.21.15.77:0/911948159 >> v2:172.21.15.186:3300/0 conn(0x7f1ba406d3f0 0x7f1ba406d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:43.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.113+0000 7f1ba8fb0700 1 -- 172.21.15.77:0/911948159 >> v2:172.21.15.186:3300/0 conn(0x7f1ba406d3f0 msgr2=0x7f1ba406d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:48:43.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.113+0000 7f1ba8fb0700 1 --2- 172.21.15.77:0/911948159 >> v2:172.21.15.186:3300/0 conn(0x7f1ba406d3f0 0x7f1ba406d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:43.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.113+0000 7f1b93fff700 1 -- 172.21.15.77:0/911948159 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056124223 0 0) 0x7f1ba40c2f80 con 0x7f1ba4156120 2022-04-03T19:48:43.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.113+0000 7f1ba8fb0700 1 -- 172.21.15.77:0/911948159 >> v1:172.21.15.77:6789/0 conn(0x7f1ba4156120 legacy=0x7f1ba406c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:43.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.114+0000 7f1ba8fb0700 1 -- 172.21.15.77:0/911948159 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba40c3820 con 0x7f1ba40c5930 2022-04-03T19:48:43.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.114+0000 7f1ba8fb0700 1 --2- 172.21.15.77:0/911948159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ba40c5930 0x7f1ba4155be0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1b94006fd0 tx=0x7f1b940090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d29b466c56040ef server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:43.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.114+0000 7f1b93fff700 1 -- 172.21.15.77:0/911948159 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b94004030 con 0x7f1ba40c5930 2022-04-03T19:48:43.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.114+0000 7f1b93fff700 1 -- 172.21.15.77:0/911948159 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b94004190 con 0x7f1ba40c5930 2022-04-03T19:48:43.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.114+0000 7f1b93fff700 1 -- 172.21.15.77:0/911948159 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b94011630 con 0x7f1ba40c5930 2022-04-03T19:48:43.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.115+0000 7f1baa7b3700 1 -- 172.21.15.77:0/911948159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ba40c5930 msgr2=0x7f1ba4155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:43.120 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.115+0000 7f1baa7b3700 1 --2- 172.21.15.77:0/911948159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ba40c5930 0x7f1ba4155be0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1b94006fd0 tx=0x7f1b940090f0 comp rx=0 tx=0).stop 2022-04-03T19:48:43.120 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.115+0000 7f1baa7b3700 1 -- 172.21.15.77:0/911948159 shutdown_connections 2022-04-03T19:48:43.120 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.115+0000 7f1baa7b3700 1 --2- 172.21.15.77:0/911948159 >> v2:172.21.15.186:3300/0 conn(0x7f1ba406d3f0 0x7f1ba406d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:43.120 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.115+0000 7f1baa7b3700 1 --2- 172.21.15.77:0/911948159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ba40c5930 0x7f1ba4155be0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:43.121 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.115+0000 7f1baa7b3700 1 -- 172.21.15.77:0/911948159 >> 172.21.15.77:0/911948159 conn(0x7f1ba40cb720 msgr2=0x7f1ba40cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:43.121 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.115+0000 7f1baa7b3700 1 -- 172.21.15.77:0/911948159 shutdown_connections 2022-04-03T19:48:43.121 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.115+0000 7f1baa7b3700 1 -- 172.21.15.77:0/911948159 wait complete. 2022-04-03T19:48:43.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.116+0000 7f1baa7b3700 1 Processor -- start 2022-04-03T19:48:43.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.117+0000 7f1baa7b3700 1 -- start start 2022-04-03T19:48:43.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.117+0000 7f1baa7b3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ba406d3f0 0x7f1ba41591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:43.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.118+0000 7f1baa7b3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ba40c5930 0x7f1ba4150020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:43.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.118+0000 7f1baa7b3700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ba4156120 0x7f1ba4150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:43.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.118+0000 7f1baa7b3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba4169440 con 0x7f1ba40c5930 2022-04-03T19:48:43.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.118+0000 7f1baa7b3700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba41515b0 con 0x7f1ba406d3f0 2022-04-03T19:48:43.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.118+0000 7f1baa7b3700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba4155340 con 0x7f1ba4156120 2022-04-03T19:48:43.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.118+0000 7f1ba8fb0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ba40c5930 0x7f1ba4150020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:43.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.118+0000 7f1ba97b1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ba406d3f0 0x7f1ba41591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:43.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.118+0000 7f1ba9fb2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ba4156120 0x7f1ba4150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:43.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.118+0000 7f1ba8fb0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ba40c5930 0x7f1ba4150020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42876/0 (socket says 172.21.15.77:42876) 2022-04-03T19:48:43.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.118+0000 7f1ba8fb0700 1 -- 172.21.15.77:0/2565606981 learned_addr learned my addr 172.21.15.77:0/2565606981 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:43.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.118+0000 7f1ba97b1700 1 -- 172.21.15.77:0/2565606981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ba4156120 msgr2=0x7f1ba4150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:43.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.118+0000 7f1ba97b1700 1 --2- 172.21.15.77:0/2565606981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ba4156120 0x7f1ba4150560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:43.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.118+0000 7f1ba97b1700 1 -- 172.21.15.77:0/2565606981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ba40c5930 msgr2=0x7f1ba4150020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:43.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.118+0000 7f1ba97b1700 1 --2- 172.21.15.77:0/2565606981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ba40c5930 0x7f1ba4150020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:43.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.118+0000 7f1ba97b1700 1 -- 172.21.15.77:0/2565606981 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b94009370 con 0x7f1ba406d3f0 2022-04-03T19:48:43.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.119+0000 7f1ba97b1700 1 --2- 172.21.15.77:0/2565606981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ba406d3f0 0x7f1ba41591e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1ba000e6d0 tx=0x7f1ba0009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:43.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.119+0000 7f1b917fa700 1 -- 172.21.15.77:0/2565606981 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0013680 con 0x7f1ba406d3f0 2022-04-03T19:48:43.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.119+0000 7f1b917fa700 1 -- 172.21.15.77:0/2565606981 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba0013800 con 0x7f1ba406d3f0 2022-04-03T19:48:43.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.119+0000 7f1b917fa700 1 -- 172.21.15.77:0/2565606981 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0013bc0 con 0x7f1ba406d3f0 2022-04-03T19:48:43.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.119+0000 7f1baa7b3700 1 -- 172.21.15.77:0/2565606981 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba4158b20 con 0x7f1ba406d3f0 2022-04-03T19:48:43.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.119+0000 7f1baa7b3700 1 -- 172.21.15.77:0/2565606981 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba41521d0 con 0x7f1ba406d3f0 2022-04-03T19:48:43.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.120+0000 7f1baa7b3700 1 -- 172.21.15.77:0/2565606981 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba414fe70 con 0x7f1ba406d3f0 2022-04-03T19:48:43.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.121+0000 7f1b917fa700 1 -- 172.21.15.77:0/2565606981 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f1ba0019030 con 0x7f1ba406d3f0 2022-04-03T19:48:43.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.122+0000 7f1b917fa700 1 --2- 172.21.15.77:0/2565606981 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1b84063ce0 0x7f1b84066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:43.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.122+0000 7f1b917fa700 1 -- 172.21.15.77:0/2565606981 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6288+0+0 (secure 0 0 0) 0x7f1ba0080390 con 0x7f1ba406d3f0 2022-04-03T19:48:43.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.122+0000 7f1ba8fb0700 1 --2- 172.21.15.77:0/2565606981 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1b84063ce0 0x7f1b84066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:43.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.123+0000 7f1ba8fb0700 1 --2- 172.21.15.77:0/2565606981 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1b84063ce0 0x7f1b84066190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1ba40c20b0 tx=0x7f1b94013000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:43.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.126+0000 7f1b917fa700 1 -- 172.21.15.77:0/2565606981 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f1ba004ee00 con 0x7f1ba406d3f0 2022-04-03T19:48:43.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.365+0000 7f1baa7b3700 1 -- 172.21.15.77:0/2565606981 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ba404b050 con 0x7f1ba406d3f0 2022-04-03T19:48:43.366 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.366+0000 7f1b917fa700 1 -- 172.21.15.77:0/2565606981 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+5353 (secure 0 0 0) 0x7f1ba0052420 con 0x7f1ba406d3f0 2022-04-03T19:48:43.366 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:43.367 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:48:43.168138+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4960},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4960":{"gid":4960,"name":"e","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:43.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.368+0000 7f1baa7b3700 1 -- 172.21.15.77:0/2565606981 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1b84063ce0 msgr2=0x7f1b84066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:43.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.368+0000 7f1baa7b3700 1 --2- 172.21.15.77:0/2565606981 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1b84063ce0 0x7f1b84066190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1ba40c20b0 tx=0x7f1b94013000 comp rx=0 tx=0).stop 2022-04-03T19:48:43.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.368+0000 7f1baa7b3700 1 -- 172.21.15.77:0/2565606981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ba406d3f0 msgr2=0x7f1ba41591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:43.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.368+0000 7f1baa7b3700 1 --2- 172.21.15.77:0/2565606981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ba406d3f0 0x7f1ba41591e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1ba000e6d0 tx=0x7f1ba0009060 comp rx=0 tx=0).stop 2022-04-03T19:48:43.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.368+0000 7f1baa7b3700 1 -- 172.21.15.77:0/2565606981 shutdown_connections 2022-04-03T19:48:43.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.368+0000 7f1baa7b3700 1 --2- 172.21.15.77:0/2565606981 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1b84063ce0 0x7f1b84066190 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:43.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.368+0000 7f1baa7b3700 1 --2- 172.21.15.77:0/2565606981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ba4156120 0x7f1ba4150560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:43.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.368+0000 7f1baa7b3700 1 --2- 172.21.15.77:0/2565606981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ba406d3f0 0x7f1ba41591e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:43.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.368+0000 7f1baa7b3700 1 --2- 172.21.15.77:0/2565606981 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ba40c5930 0x7f1ba4150020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:43.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.368+0000 7f1baa7b3700 1 -- 172.21.15.77:0/2565606981 >> 172.21.15.77:0/2565606981 conn(0x7f1ba40cb720 msgr2=0x7f1ba4153fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:43.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.368+0000 7f1baa7b3700 1 -- 172.21.15.77:0/2565606981 shutdown_connections 2022-04-03T19:48:43.372 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:43.368+0000 7f1baa7b3700 1 -- 172.21.15.77:0/2565606981 wait complete. 2022-04-03T19:48:43.372 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 19 2022-04-03T19:48:43.393 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=4303 2022-04-03T19:48:43.394 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.d 2022-04-03T19:48:45.652 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:48:45.658 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:48:48.555 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/14' 2022-04-03T19:48:48.555 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/18' 2022-04-03T19:48:48.555 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/13' 2022-04-03T19:48:48.555 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/33' 2022-04-03T19:48:48.556 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/17' 2022-04-03T19:48:48.556 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/35' 2022-04-03T19:48:48.556 INFO:tasks.workunit.client.0.smithi077.stdout:removed './109/37' 2022-04-03T19:48:48.556 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './109' 2022-04-03T19:48:48.557 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/19' 2022-04-03T19:48:48.557 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/15' 2022-04-03T19:48:48.557 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/31' 2022-04-03T19:48:48.557 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/36' 2022-04-03T19:48:48.557 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/20' 2022-04-03T19:48:48.558 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/12' 2022-04-03T19:48:48.558 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/39' 2022-04-03T19:48:48.558 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/10' 2022-04-03T19:48:48.558 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/34' 2022-04-03T19:48:48.559 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/38' 2022-04-03T19:48:48.559 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/11' 2022-04-03T19:48:48.559 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/16' 2022-04-03T19:48:48.559 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/32' 2022-04-03T19:48:48.559 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/30' 2022-04-03T19:48:48.560 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/14' 2022-04-03T19:48:48.560 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/18' 2022-04-03T19:48:48.560 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/13' 2022-04-03T19:48:48.560 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/33' 2022-04-03T19:48:48.561 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/17' 2022-04-03T19:48:48.561 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/35' 2022-04-03T19:48:48.561 INFO:tasks.workunit.client.0.smithi077.stdout:removed './110/37' 2022-04-03T19:48:48.561 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './110' 2022-04-03T19:48:48.561 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/19' 2022-04-03T19:48:48.562 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/15' 2022-04-03T19:48:48.562 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/31' 2022-04-03T19:48:48.562 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/36' 2022-04-03T19:48:48.562 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/20' 2022-04-03T19:48:48.562 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/12' 2022-04-03T19:48:48.563 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/39' 2022-04-03T19:48:48.563 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/10' 2022-04-03T19:48:48.563 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/34' 2022-04-03T19:48:48.563 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/38' 2022-04-03T19:48:48.564 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/11' 2022-04-03T19:48:48.564 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/16' 2022-04-03T19:48:48.564 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/32' 2022-04-03T19:48:48.564 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/30' 2022-04-03T19:48:48.565 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/14' 2022-04-03T19:48:48.565 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/18' 2022-04-03T19:48:48.565 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/13' 2022-04-03T19:48:48.565 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/33' 2022-04-03T19:48:48.565 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/17' 2022-04-03T19:48:48.566 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/35' 2022-04-03T19:48:48.566 INFO:tasks.workunit.client.0.smithi077.stdout:removed './111/37' 2022-04-03T19:48:48.566 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './111' 2022-04-03T19:48:48.566 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/19' 2022-04-03T19:48:48.567 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/15' 2022-04-03T19:48:48.567 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/31' 2022-04-03T19:48:48.567 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/36' 2022-04-03T19:48:48.567 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/20' 2022-04-03T19:48:48.567 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/12' 2022-04-03T19:48:48.568 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/39' 2022-04-03T19:48:48.568 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/10' 2022-04-03T19:48:48.568 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/34' 2022-04-03T19:48:48.568 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/38' 2022-04-03T19:48:48.569 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/11' 2022-04-03T19:48:48.569 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/16' 2022-04-03T19:48:48.569 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/32' 2022-04-03T19:48:48.569 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/30' 2022-04-03T19:48:48.569 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/14' 2022-04-03T19:48:48.570 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/18' 2022-04-03T19:48:48.570 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/13' 2022-04-03T19:48:48.570 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/33' 2022-04-03T19:48:48.570 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/17' 2022-04-03T19:48:48.570 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/35' 2022-04-03T19:48:48.571 INFO:tasks.workunit.client.0.smithi077.stdout:removed './112/37' 2022-04-03T19:48:48.571 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './112' 2022-04-03T19:48:48.571 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/19' 2022-04-03T19:48:48.571 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/15' 2022-04-03T19:48:48.571 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/31' 2022-04-03T19:48:48.572 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/36' 2022-04-03T19:48:48.572 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/20' 2022-04-03T19:48:48.572 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/12' 2022-04-03T19:48:48.572 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/39' 2022-04-03T19:48:48.573 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/10' 2022-04-03T19:48:48.573 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/34' 2022-04-03T19:48:48.573 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/38' 2022-04-03T19:48:48.573 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/11' 2022-04-03T19:48:48.573 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/16' 2022-04-03T19:48:48.574 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/32' 2022-04-03T19:48:48.574 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/30' 2022-04-03T19:48:48.574 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/14' 2022-04-03T19:48:48.574 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/18' 2022-04-03T19:48:48.575 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/13' 2022-04-03T19:48:48.575 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/33' 2022-04-03T19:48:48.575 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/17' 2022-04-03T19:48:48.575 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/35' 2022-04-03T19:48:48.575 INFO:tasks.workunit.client.0.smithi077.stdout:removed './113/37' 2022-04-03T19:48:48.576 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './113' 2022-04-03T19:48:48.576 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/19' 2022-04-03T19:48:48.576 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/15' 2022-04-03T19:48:48.576 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/31' 2022-04-03T19:48:48.577 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/36' 2022-04-03T19:48:48.577 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/20' 2022-04-03T19:48:48.577 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/12' 2022-04-03T19:48:48.577 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/39' 2022-04-03T19:48:48.577 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/10' 2022-04-03T19:48:48.578 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/34' 2022-04-03T19:48:48.578 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/38' 2022-04-03T19:48:48.578 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/11' 2022-04-03T19:48:48.578 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/16' 2022-04-03T19:48:48.579 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/32' 2022-04-03T19:48:48.579 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/30' 2022-04-03T19:48:48.579 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/14' 2022-04-03T19:48:48.579 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/18' 2022-04-03T19:48:48.579 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/13' 2022-04-03T19:48:48.580 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/33' 2022-04-03T19:48:48.580 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/17' 2022-04-03T19:48:48.580 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/35' 2022-04-03T19:48:48.580 INFO:tasks.workunit.client.0.smithi077.stdout:removed './114/37' 2022-04-03T19:48:48.581 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './114' 2022-04-03T19:48:48.581 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/19' 2022-04-03T19:48:48.581 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/15' 2022-04-03T19:48:48.581 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/31' 2022-04-03T19:48:48.581 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/36' 2022-04-03T19:48:48.582 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/20' 2022-04-03T19:48:48.582 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/12' 2022-04-03T19:48:48.582 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/39' 2022-04-03T19:48:48.582 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/10' 2022-04-03T19:48:48.582 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/34' 2022-04-03T19:48:48.583 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/38' 2022-04-03T19:48:48.583 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/11' 2022-04-03T19:48:48.583 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/16' 2022-04-03T19:48:48.583 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/32' 2022-04-03T19:48:48.584 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/30' 2022-04-03T19:48:48.584 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/14' 2022-04-03T19:48:48.584 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/18' 2022-04-03T19:48:48.584 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/13' 2022-04-03T19:48:48.584 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/33' 2022-04-03T19:48:48.585 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/17' 2022-04-03T19:48:48.585 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/35' 2022-04-03T19:48:48.585 INFO:tasks.workunit.client.0.smithi077.stdout:removed './115/37' 2022-04-03T19:48:48.585 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './115' 2022-04-03T19:48:48.586 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/19' 2022-04-03T19:48:48.587 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/15' 2022-04-03T19:48:48.587 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/31' 2022-04-03T19:48:48.587 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/36' 2022-04-03T19:48:48.588 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/20' 2022-04-03T19:48:48.588 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/12' 2022-04-03T19:48:48.588 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/39' 2022-04-03T19:48:48.588 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/10' 2022-04-03T19:48:48.588 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/34' 2022-04-03T19:48:48.589 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/38' 2022-04-03T19:48:48.589 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/11' 2022-04-03T19:48:48.589 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/16' 2022-04-03T19:48:48.590 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/32' 2022-04-03T19:48:48.590 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/30' 2022-04-03T19:48:48.590 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/14' 2022-04-03T19:48:48.591 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/18' 2022-04-03T19:48:48.591 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/13' 2022-04-03T19:48:48.591 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/33' 2022-04-03T19:48:48.592 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/17' 2022-04-03T19:48:48.592 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/35' 2022-04-03T19:48:48.592 INFO:tasks.workunit.client.0.smithi077.stdout:removed './116/37' 2022-04-03T19:48:48.593 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './116' 2022-04-03T19:48:48.593 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/19' 2022-04-03T19:48:48.593 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/15' 2022-04-03T19:48:48.594 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/31' 2022-04-03T19:48:48.594 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/36' 2022-04-03T19:48:48.595 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/20' 2022-04-03T19:48:48.595 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/12' 2022-04-03T19:48:48.596 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/39' 2022-04-03T19:48:48.596 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/10' 2022-04-03T19:48:48.596 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/34' 2022-04-03T19:48:48.597 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/38' 2022-04-03T19:48:48.597 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/11' 2022-04-03T19:48:48.597 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/16' 2022-04-03T19:48:48.597 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/32' 2022-04-03T19:48:48.597 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/30' 2022-04-03T19:48:48.598 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/14' 2022-04-03T19:48:48.598 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/18' 2022-04-03T19:48:48.598 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/13' 2022-04-03T19:48:48.598 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/33' 2022-04-03T19:48:48.599 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/17' 2022-04-03T19:48:48.599 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/35' 2022-04-03T19:48:48.599 INFO:tasks.workunit.client.0.smithi077.stdout:removed './117/37' 2022-04-03T19:48:48.599 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './117' 2022-04-03T19:48:48.599 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/19' 2022-04-03T19:48:48.600 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/15' 2022-04-03T19:48:48.600 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/31' 2022-04-03T19:48:48.600 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/36' 2022-04-03T19:48:48.600 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/20' 2022-04-03T19:48:48.601 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/12' 2022-04-03T19:48:48.601 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/39' 2022-04-03T19:48:48.601 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/10' 2022-04-03T19:48:48.601 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/34' 2022-04-03T19:48:48.601 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/38' 2022-04-03T19:48:48.602 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/11' 2022-04-03T19:48:48.602 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/16' 2022-04-03T19:48:48.602 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/32' 2022-04-03T19:48:48.602 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/30' 2022-04-03T19:48:48.603 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/14' 2022-04-03T19:48:48.603 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/18' 2022-04-03T19:48:48.603 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/13' 2022-04-03T19:48:48.603 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/33' 2022-04-03T19:48:48.603 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/17' 2022-04-03T19:48:48.604 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/35' 2022-04-03T19:48:48.604 INFO:tasks.workunit.client.0.smithi077.stdout:removed './118/37' 2022-04-03T19:48:48.604 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './118' 2022-04-03T19:48:48.604 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/19' 2022-04-03T19:48:48.604 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/15' 2022-04-03T19:48:48.605 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/31' 2022-04-03T19:48:48.605 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/36' 2022-04-03T19:48:48.605 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/20' 2022-04-03T19:48:48.605 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/12' 2022-04-03T19:48:49.394 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2022-04-03T19:48:49.395 INFO:tasks.ceph.mds.d:Restarting daemon 2022-04-03T19:48:49.395 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2022-04-03T19:48:49.398 INFO:tasks.ceph.mds.d:Started 2022-04-03T19:48:49.398 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:49.477 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2022-04-03T19:48:49.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.593+0000 7f73b4bc3700 1 Processor -- start 2022-04-03T19:48:49.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.594+0000 7f73b4bc3700 1 -- start start 2022-04-03T19:48:49.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.595+0000 7f73b4bc3700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f73b006bbe0 0x7f73b01584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:49.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.595+0000 7f73b4bc3700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f73b006bfb0 0x7f73b006d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:49.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.595+0000 7f73b4bc3700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b006d260 con 0x7f73b006c500 2022-04-03T19:48:49.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.595+0000 7f73b4bc3700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b0159450 con 0x7f73b006bfb0 2022-04-03T19:48:49.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.595+0000 7f73b4bc3700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b00c6330 con 0x7f73b006bbe0 2022-04-03T19:48:49.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.595+0000 7f73aeffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f73b006bfb0 0x7f73b006d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:49.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.595+0000 7f73af7fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f73b006bbe0 0x7f73b01584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:49.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.595+0000 7f73af7fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f73b006bbe0 0x7f73b01584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38070/0 (socket says 172.21.15.77:38070) 2022-04-03T19:48:49.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.595+0000 7f73affff700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f73b006c500 0x7f73b014eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44238/0 (socket says 172.21.15.77:44238) 2022-04-03T19:48:49.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.595+0000 7f73af7fe700 1 -- 172.21.15.77:0/3459967170 learned_addr learned my addr 172.21.15.77:0/3459967170 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:49.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.596+0000 7f73af7fe700 1 -- 172.21.15.77:0/3459967170 >> v2:172.21.15.77:3300/0 conn(0x7f73b006bfb0 msgr2=0x7f73b006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:48:49.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.596+0000 7f73af7fe700 1 --2- 172.21.15.77:0/3459967170 >> v2:172.21.15.77:3300/0 conn(0x7f73b006bfb0 0x7f73b006d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:49.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.596+0000 7f73af7fe700 1 -- 172.21.15.77:0/3459967170 >> v1:172.21.15.186:6789/0 conn(0x7f73b006c500 legacy=0x7f73b014eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:49.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.596+0000 7f73ae7fc700 1 -- 172.21.15.77:0/3459967170 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630783014 0 0) 0x7f73b006d260 con 0x7f73b006c500 2022-04-03T19:48:49.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.596+0000 7f73af7fe700 1 -- 172.21.15.77:0/3459967170 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b0158bb0 con 0x7f73b006bbe0 2022-04-03T19:48:49.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.596+0000 7f73aeffd700 1 --2- 172.21.15.77:0/3459967170 >> v2:172.21.15.77:3300/0 conn(0x7f73b006bfb0 0x7f73b006d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:48:49.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.596+0000 7f73af7fe700 1 --2- 172.21.15.77:0/3459967170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73b006bbe0 0x7f73b01584b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f73b0088d40 tx=0x7f73a00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f48c2af2ea829039 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:49.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.596+0000 7f73ae7fc700 1 -- 172.21.15.77:0/3459967170 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0004030 con 0x7f73b006bbe0 2022-04-03T19:48:49.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.596+0000 7f73ae7fc700 1 -- 172.21.15.77:0/3459967170 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a0004190 con 0x7f73b006bbe0 2022-04-03T19:48:49.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.596+0000 7f73ae7fc700 1 -- 172.21.15.77:0/3459967170 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a0011630 con 0x7f73b006bbe0 2022-04-03T19:48:49.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.597+0000 7f73b4bc3700 1 -- 172.21.15.77:0/3459967170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73b006bbe0 msgr2=0x7f73b01584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:49.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.597+0000 7f73b4bc3700 1 --2- 172.21.15.77:0/3459967170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73b006bbe0 0x7f73b01584b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f73b0088d40 tx=0x7f73a00090f0 comp rx=0 tx=0).stop 2022-04-03T19:48:49.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.597+0000 7f73b4bc3700 1 -- 172.21.15.77:0/3459967170 shutdown_connections 2022-04-03T19:48:49.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.597+0000 7f73b4bc3700 1 --2- 172.21.15.77:0/3459967170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73b006bbe0 0x7f73b01584b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:49.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.597+0000 7f73b4bc3700 1 --2- 172.21.15.77:0/3459967170 >> v2:172.21.15.77:3300/0 conn(0x7f73b006bfb0 0x7f73b006d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:49.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.597+0000 7f73b4bc3700 1 -- 172.21.15.77:0/3459967170 >> 172.21.15.77:0/3459967170 conn(0x7f73b00cb6b0 msgr2=0x7f73b00cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:49.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.597+0000 7f73b4bc3700 1 -- 172.21.15.77:0/3459967170 shutdown_connections 2022-04-03T19:48:49.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.597+0000 7f73b4bc3700 1 -- 172.21.15.77:0/3459967170 wait complete. 2022-04-03T19:48:49.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.599+0000 7f73b4bc3700 1 Processor -- start 2022-04-03T19:48:49.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.599+0000 7f73b4bc3700 1 -- start start 2022-04-03T19:48:49.611 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.600+0000 7f73b4bc3700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73b006bbe0 0x7f73b0163460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:49.611 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.600+0000 7f73b4bc3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73b006bfb0 0x7f73b00c28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:49.611 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.600+0000 7f73b4bc3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73b006c500 0x7f73b00c2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:49.611 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.600+0000 7f73b4bc3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b00c3360 con 0x7f73b006c500 2022-04-03T19:48:49.612 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.600+0000 7f73b4bc3700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b01682e0 con 0x7f73b006bfb0 2022-04-03T19:48:49.612 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.600+0000 7f73b4bc3700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b0168770 con 0x7f73b006bbe0 2022-04-03T19:48:49.612 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.600+0000 7f73aeffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73b006bfb0 0x7f73b00c28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:49.612 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.600+0000 7f73affff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73b006c500 0x7f73b00c2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:49.612 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.600+0000 7f73aeffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73b006bfb0 0x7f73b00c28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38072/0 (socket says 172.21.15.77:38072) 2022-04-03T19:48:49.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.600+0000 7f73aeffd700 1 -- 172.21.15.77:0/4209988709 learned_addr learned my addr 172.21.15.77:0/4209988709 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:49.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.600+0000 7f73af7fe700 1 --2- 172.21.15.77:0/4209988709 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73b006bbe0 0x7f73b0163460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:49.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.600+0000 7f73aeffd700 1 -- 172.21.15.77:0/4209988709 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73b006bbe0 msgr2=0x7f73b0163460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:49.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.600+0000 7f73aeffd700 1 --2- 172.21.15.77:0/4209988709 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73b006bbe0 0x7f73b0163460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:49.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.600+0000 7f73aeffd700 1 -- 172.21.15.77:0/4209988709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73b006c500 msgr2=0x7f73b00c2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:49.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.600+0000 7f73aeffd700 1 --2- 172.21.15.77:0/4209988709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73b006c500 0x7f73b00c2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:49.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.600+0000 7f73aeffd700 1 -- 172.21.15.77:0/4209988709 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a40092a0 con 0x7f73b006bfb0 2022-04-03T19:48:49.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.601+0000 7f73aeffd700 1 --2- 172.21.15.77:0/4209988709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73b006bfb0 0x7f73b00c28e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f73a400ea50 tx=0x7f73a4009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:49.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.601+0000 7f73977fe700 1 -- 172.21.15.77:0/4209988709 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a400ee90 con 0x7f73b006bfb0 2022-04-03T19:48:49.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.601+0000 7f73977fe700 1 -- 172.21.15.77:0/4209988709 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a400c850 con 0x7f73b006bfb0 2022-04-03T19:48:49.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.601+0000 7f73977fe700 1 -- 172.21.15.77:0/4209988709 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a400cbd0 con 0x7f73b006bfb0 2022-04-03T19:48:49.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.601+0000 7f73b4bc3700 1 -- 172.21.15.77:0/4209988709 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a0009370 con 0x7f73b006bfb0 2022-04-03T19:48:49.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.601+0000 7f73b4bc3700 1 -- 172.21.15.77:0/4209988709 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b01676d0 con 0x7f73b006bfb0 2022-04-03T19:48:49.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.602+0000 7f73b4bc3700 1 -- 172.21.15.77:0/4209988709 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b00cb4d0 con 0x7f73b006bfb0 2022-04-03T19:48:49.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.603+0000 7f73977fe700 1 -- 172.21.15.77:0/4209988709 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f73a4023070 con 0x7f73b006bfb0 2022-04-03T19:48:49.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.604+0000 7f73977fe700 1 --2- 172.21.15.77:0/4209988709 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7390063ce0 0x7f7390066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:49.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.604+0000 7f73977fe700 1 -- 172.21.15.77:0/4209988709 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6288+0+0 (secure 0 0 0) 0x7f73a4080dc0 con 0x7f73b006bfb0 2022-04-03T19:48:49.617 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.604+0000 7f73af7fe700 1 --2- 172.21.15.77:0/4209988709 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7390063ce0 0x7f7390066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:49.617 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.605+0000 7f73af7fe700 1 --2- 172.21.15.77:0/4209988709 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7390063ce0 0x7f7390066190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f73b0088d40 tx=0x7f73a0016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:49.617 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.608+0000 7f73977fe700 1 -- 172.21.15.77:0/4209988709 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f73a404f860 con 0x7f73b006bfb0 2022-04-03T19:48:49.843 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.841+0000 7f73b4bc3700 1 -- 172.21.15.77:0/4209988709 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73b006cb90 con 0x7f73b006bfb0 2022-04-03T19:48:49.844 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.843+0000 7f73977fe700 1 -- 172.21.15.77:0/4209988709 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+5353 (secure 0 0 0) 0x7f73a4017090 con 0x7f73b006bfb0 2022-04-03T19:48:49.844 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:49.844 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:48:43.168138+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4960},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4960":{"gid":4960,"name":"e","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:49.846 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.844+0000 7f73b4bc3700 1 -- 172.21.15.77:0/4209988709 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7390063ce0 msgr2=0x7f7390066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.845+0000 7f73b4bc3700 1 --2- 172.21.15.77:0/4209988709 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7390063ce0 0x7f7390066190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f73b0088d40 tx=0x7f73a0016000 comp rx=0 tx=0).stop 2022-04-03T19:48:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.845+0000 7f73b4bc3700 1 -- 172.21.15.77:0/4209988709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73b006bfb0 msgr2=0x7f73b00c28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.845+0000 7f73b4bc3700 1 --2- 172.21.15.77:0/4209988709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73b006bfb0 0x7f73b00c28e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f73a400ea50 tx=0x7f73a4009060 comp rx=0 tx=0).stop 2022-04-03T19:48:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.845+0000 7f73b4bc3700 1 -- 172.21.15.77:0/4209988709 shutdown_connections 2022-04-03T19:48:49.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.845+0000 7f73b4bc3700 1 --2- 172.21.15.77:0/4209988709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f73b006c500 0x7f73b00c2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:49.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.845+0000 7f73b4bc3700 1 --2- 172.21.15.77:0/4209988709 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7390063ce0 0x7f7390066190 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:49.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.845+0000 7f73b4bc3700 1 --2- 172.21.15.77:0/4209988709 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73b006bbe0 0x7f73b0163460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:49.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.845+0000 7f73b4bc3700 1 --2- 172.21.15.77:0/4209988709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73b006bfb0 0x7f73b00c28e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:49.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.845+0000 7f73b4bc3700 1 -- 172.21.15.77:0/4209988709 >> 172.21.15.77:0/4209988709 conn(0x7f73b00cb6b0 msgr2=0x7f73b00c50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:49.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.845+0000 7f73b4bc3700 1 -- 172.21.15.77:0/4209988709 shutdown_connections 2022-04-03T19:48:49.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:49.845+0000 7f73b4bc3700 1 -- 172.21.15.77:0/4209988709 wait complete. 2022-04-03T19:48:49.850 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 19 2022-04-03T19:48:49.869 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2022-04-03T19:48:51.733 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/39' 2022-04-03T19:48:51.733 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/10' 2022-04-03T19:48:51.734 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/34' 2022-04-03T19:48:51.734 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/38' 2022-04-03T19:48:51.734 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/11' 2022-04-03T19:48:51.734 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/16' 2022-04-03T19:48:51.735 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/32' 2022-04-03T19:48:51.735 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/30' 2022-04-03T19:48:51.735 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/14' 2022-04-03T19:48:51.735 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/18' 2022-04-03T19:48:51.736 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/13' 2022-04-03T19:48:51.736 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/33' 2022-04-03T19:48:51.736 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/17' 2022-04-03T19:48:51.736 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/35' 2022-04-03T19:48:51.736 INFO:tasks.workunit.client.0.smithi077.stdout:removed './119/37' 2022-04-03T19:48:51.737 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './119' 2022-04-03T19:48:51.737 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/19' 2022-04-03T19:48:51.737 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/15' 2022-04-03T19:48:51.737 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/31' 2022-04-03T19:48:51.738 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/36' 2022-04-03T19:48:51.738 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/20' 2022-04-03T19:48:51.738 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/12' 2022-04-03T19:48:51.738 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/39' 2022-04-03T19:48:51.738 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/10' 2022-04-03T19:48:51.739 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/34' 2022-04-03T19:48:51.739 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/38' 2022-04-03T19:48:51.739 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/11' 2022-04-03T19:48:51.739 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/16' 2022-04-03T19:48:51.740 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/32' 2022-04-03T19:48:51.740 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/30' 2022-04-03T19:48:51.740 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/14' 2022-04-03T19:48:51.740 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/18' 2022-04-03T19:48:51.740 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/13' 2022-04-03T19:48:51.741 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/33' 2022-04-03T19:48:51.741 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/17' 2022-04-03T19:48:51.741 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/35' 2022-04-03T19:48:51.741 INFO:tasks.workunit.client.0.smithi077.stdout:removed './120/37' 2022-04-03T19:48:51.742 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './120' 2022-04-03T19:48:51.742 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/19' 2022-04-03T19:48:51.742 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/15' 2022-04-03T19:48:51.742 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/31' 2022-04-03T19:48:51.742 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/36' 2022-04-03T19:48:51.743 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/20' 2022-04-03T19:48:51.743 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/12' 2022-04-03T19:48:51.743 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/39' 2022-04-03T19:48:51.743 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/10' 2022-04-03T19:48:51.744 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/34' 2022-04-03T19:48:51.744 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/38' 2022-04-03T19:48:51.744 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/11' 2022-04-03T19:48:51.744 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/16' 2022-04-03T19:48:51.744 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/32' 2022-04-03T19:48:51.745 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/30' 2022-04-03T19:48:51.745 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/14' 2022-04-03T19:48:51.745 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/18' 2022-04-03T19:48:51.745 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/13' 2022-04-03T19:48:51.745 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/33' 2022-04-03T19:48:51.746 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/17' 2022-04-03T19:48:51.746 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/35' 2022-04-03T19:48:51.746 INFO:tasks.workunit.client.0.smithi077.stdout:removed './121/37' 2022-04-03T19:48:51.746 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './121' 2022-04-03T19:48:51.747 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/19' 2022-04-03T19:48:51.747 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/15' 2022-04-03T19:48:51.747 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/31' 2022-04-03T19:48:51.747 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/36' 2022-04-03T19:48:51.748 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/20' 2022-04-03T19:48:51.748 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/12' 2022-04-03T19:48:51.748 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/39' 2022-04-03T19:48:51.748 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/10' 2022-04-03T19:48:51.748 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/34' 2022-04-03T19:48:51.749 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/38' 2022-04-03T19:48:51.749 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/11' 2022-04-03T19:48:51.749 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/16' 2022-04-03T19:48:51.749 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/32' 2022-04-03T19:48:51.749 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/30' 2022-04-03T19:48:51.750 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/14' 2022-04-03T19:48:51.750 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/18' 2022-04-03T19:48:51.750 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/13' 2022-04-03T19:48:51.750 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/33' 2022-04-03T19:48:51.750 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/17' 2022-04-03T19:48:51.751 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/35' 2022-04-03T19:48:51.751 INFO:tasks.workunit.client.0.smithi077.stdout:removed './122/37' 2022-04-03T19:48:51.751 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './122' 2022-04-03T19:48:51.751 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/19' 2022-04-03T19:48:51.751 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/15' 2022-04-03T19:48:51.752 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/31' 2022-04-03T19:48:51.752 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/36' 2022-04-03T19:48:51.752 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/20' 2022-04-03T19:48:51.752 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/12' 2022-04-03T19:48:51.752 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/39' 2022-04-03T19:48:51.753 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/10' 2022-04-03T19:48:51.753 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/34' 2022-04-03T19:48:51.753 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/38' 2022-04-03T19:48:51.753 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/11' 2022-04-03T19:48:51.753 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/16' 2022-04-03T19:48:51.753 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/32' 2022-04-03T19:48:51.754 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/30' 2022-04-03T19:48:51.754 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/14' 2022-04-03T19:48:51.754 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/18' 2022-04-03T19:48:51.754 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/13' 2022-04-03T19:48:51.755 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/33' 2022-04-03T19:48:51.755 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/17' 2022-04-03T19:48:51.755 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/35' 2022-04-03T19:48:51.755 INFO:tasks.workunit.client.0.smithi077.stdout:removed './123/37' 2022-04-03T19:48:51.755 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './123' 2022-04-03T19:48:51.756 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/19' 2022-04-03T19:48:51.756 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/15' 2022-04-03T19:48:51.756 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/31' 2022-04-03T19:48:51.756 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/36' 2022-04-03T19:48:51.757 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/20' 2022-04-03T19:48:51.757 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/12' 2022-04-03T19:48:51.757 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/39' 2022-04-03T19:48:51.757 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/10' 2022-04-03T19:48:51.757 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/34' 2022-04-03T19:48:51.758 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/38' 2022-04-03T19:48:51.758 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/11' 2022-04-03T19:48:51.758 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/16' 2022-04-03T19:48:51.758 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/32' 2022-04-03T19:48:51.758 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/30' 2022-04-03T19:48:51.759 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/14' 2022-04-03T19:48:51.759 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/18' 2022-04-03T19:48:51.759 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/13' 2022-04-03T19:48:51.759 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/33' 2022-04-03T19:48:51.759 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/17' 2022-04-03T19:48:51.760 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/35' 2022-04-03T19:48:51.760 INFO:tasks.workunit.client.0.smithi077.stdout:removed './124/37' 2022-04-03T19:48:51.760 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './124' 2022-04-03T19:48:51.760 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/19' 2022-04-03T19:48:51.760 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/15' 2022-04-03T19:48:51.761 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/31' 2022-04-03T19:48:51.761 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/36' 2022-04-03T19:48:51.761 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/20' 2022-04-03T19:48:51.761 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/12' 2022-04-03T19:48:51.761 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/39' 2022-04-03T19:48:51.762 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/10' 2022-04-03T19:48:51.762 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/34' 2022-04-03T19:48:51.762 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/38' 2022-04-03T19:48:51.762 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/11' 2022-04-03T19:48:51.763 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/16' 2022-04-03T19:48:51.763 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/32' 2022-04-03T19:48:51.763 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/30' 2022-04-03T19:48:51.763 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/14' 2022-04-03T19:48:51.763 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/18' 2022-04-03T19:48:51.764 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/13' 2022-04-03T19:48:51.764 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/33' 2022-04-03T19:48:51.764 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/17' 2022-04-03T19:48:51.764 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/35' 2022-04-03T19:48:51.764 INFO:tasks.workunit.client.0.smithi077.stdout:removed './125/37' 2022-04-03T19:48:51.765 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './125' 2022-04-03T19:48:51.765 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/19' 2022-04-03T19:48:51.765 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/15' 2022-04-03T19:48:51.765 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/31' 2022-04-03T19:48:51.765 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/36' 2022-04-03T19:48:51.766 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/20' 2022-04-03T19:48:51.766 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/12' 2022-04-03T19:48:51.766 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/39' 2022-04-03T19:48:51.766 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/10' 2022-04-03T19:48:51.766 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/34' 2022-04-03T19:48:51.767 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/38' 2022-04-03T19:48:51.767 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/11' 2022-04-03T19:48:51.767 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/16' 2022-04-03T19:48:51.767 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/32' 2022-04-03T19:48:51.767 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/30' 2022-04-03T19:48:51.768 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/14' 2022-04-03T19:48:51.768 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/18' 2022-04-03T19:48:51.768 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/13' 2022-04-03T19:48:51.768 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/33' 2022-04-03T19:48:51.768 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/17' 2022-04-03T19:48:51.769 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/35' 2022-04-03T19:48:51.769 INFO:tasks.workunit.client.0.smithi077.stdout:removed './126/37' 2022-04-03T19:48:51.769 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './126' 2022-04-03T19:48:51.769 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/19' 2022-04-03T19:48:51.769 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/15' 2022-04-03T19:48:51.770 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/31' 2022-04-03T19:48:51.770 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/36' 2022-04-03T19:48:51.770 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/20' 2022-04-03T19:48:51.770 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/12' 2022-04-03T19:48:51.770 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/39' 2022-04-03T19:48:51.771 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/10' 2022-04-03T19:48:51.771 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/34' 2022-04-03T19:48:51.771 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/38' 2022-04-03T19:48:51.771 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/11' 2022-04-03T19:48:51.771 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/16' 2022-04-03T19:48:51.772 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/32' 2022-04-03T19:48:51.772 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/30' 2022-04-03T19:48:51.772 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/14' 2022-04-03T19:48:51.772 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/18' 2022-04-03T19:48:51.772 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/13' 2022-04-03T19:48:51.773 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/33' 2022-04-03T19:48:51.773 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/17' 2022-04-03T19:48:51.773 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/35' 2022-04-03T19:48:51.773 INFO:tasks.workunit.client.0.smithi077.stdout:removed './127/37' 2022-04-03T19:48:51.773 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './127' 2022-04-03T19:48:51.774 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/19' 2022-04-03T19:48:51.774 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/15' 2022-04-03T19:48:51.774 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/31' 2022-04-03T19:48:51.774 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/36' 2022-04-03T19:48:51.774 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/20' 2022-04-03T19:48:51.775 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/12' 2022-04-03T19:48:51.775 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/39' 2022-04-03T19:48:51.775 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/10' 2022-04-03T19:48:51.775 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/34' 2022-04-03T19:48:51.776 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/38' 2022-04-03T19:48:51.776 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/11' 2022-04-03T19:48:51.776 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/16' 2022-04-03T19:48:51.776 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/32' 2022-04-03T19:48:51.777 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/30' 2022-04-03T19:48:51.777 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/14' 2022-04-03T19:48:51.777 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/18' 2022-04-03T19:48:51.777 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/13' 2022-04-03T19:48:51.777 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/33' 2022-04-03T19:48:51.778 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/17' 2022-04-03T19:48:51.778 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/35' 2022-04-03T19:48:51.871 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:48:52.070 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.068+0000 7f529084c700 1 Processor -- start 2022-04-03T19:48:52.070 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.069+0000 7f529084c700 1 -- start start 2022-04-03T19:48:52.071 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.070+0000 7f529084c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f528c0ba3c0 0x7f528c156b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:52.072 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.070+0000 7f529084c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f528c0678b0 0x7f528c14d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:52.072 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.070+0000 7f529084c700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528c0c3c50 con 0x7f528c157060 2022-04-03T19:48:52.072 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.070+0000 7f529084c700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f528c0c8020 con 0x7f528c0ba3c0 2022-04-03T19:48:52.072 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.070+0000 7f529084c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f528c156690 con 0x7f528c0678b0 2022-04-03T19:48:52.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.070+0000 7f528ad9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f528c0ba3c0 0x7f528c156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:52.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.070+0000 7f528ad9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f528c0ba3c0 0x7f528c156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42882/0 (socket says 172.21.15.77:42882) 2022-04-03T19:48:52.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.070+0000 7f528ad9d700 1 -- 172.21.15.77:0/3797638730 learned_addr learned my addr 172.21.15.77:0/3797638730 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:52.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.070+0000 7f528b59e700 1 --2- 172.21.15.77:0/3797638730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f528c0678b0 0x7f528c14d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:52.075 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.071+0000 7f528ad9d700 1 -- 172.21.15.77:0/3797638730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f528c0678b0 msgr2=0x7f528c14d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:48:52.075 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.071+0000 7f528ad9d700 1 --2- 172.21.15.77:0/3797638730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f528c0678b0 0x7f528c14d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:52.076 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.071+0000 7f528ad9d700 1 -- 172.21.15.77:0/3797638730 >> v1:172.21.15.186:6789/0 conn(0x7f528c157060 legacy=0x7f528c14cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:52.076 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.071+0000 7f528ad9d700 1 -- 172.21.15.77:0/3797638730 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528c0bf350 con 0x7f528c0ba3c0 2022-04-03T19:48:52.076 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.071+0000 7f528ad9d700 1 --2- 172.21.15.77:0/3797638730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f528c0ba3c0 0x7f528c156b20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f527c006fd0 tx=0x7f527c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1849224171abc5ff server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:52.076 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.071+0000 7f5289d9b700 1 -- 172.21.15.77:0/3797638730 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527c004030 con 0x7f528c0ba3c0 2022-04-03T19:48:52.076 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.071+0000 7f5289d9b700 1 -- 172.21.15.77:0/3797638730 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f527c0041d0 con 0x7f528c0ba3c0 2022-04-03T19:48:52.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.071+0000 7f5289d9b700 1 -- 172.21.15.77:0/3797638730 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f527c011630 con 0x7f528c0ba3c0 2022-04-03T19:48:52.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.072+0000 7f529084c700 1 -- 172.21.15.77:0/3797638730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f528c0ba3c0 msgr2=0x7f528c156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:52.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.072+0000 7f529084c700 1 --2- 172.21.15.77:0/3797638730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f528c0ba3c0 0x7f528c156b20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f527c006fd0 tx=0x7f527c0090f0 comp rx=0 tx=0).stop 2022-04-03T19:48:52.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.072+0000 7f529084c700 1 -- 172.21.15.77:0/3797638730 shutdown_connections 2022-04-03T19:48:52.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.072+0000 7f529084c700 1 --2- 172.21.15.77:0/3797638730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f528c0678b0 0x7f528c14d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:52.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.072+0000 7f529084c700 1 --2- 172.21.15.77:0/3797638730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f528c0ba3c0 0x7f528c156b20 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:52.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.072+0000 7f529084c700 1 -- 172.21.15.77:0/3797638730 >> 172.21.15.77:0/3797638730 conn(0x7f528c066da0 msgr2=0x7f528c0671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:52.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.072+0000 7f529084c700 1 -- 172.21.15.77:0/3797638730 shutdown_connections 2022-04-03T19:48:52.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.072+0000 7f529084c700 1 -- 172.21.15.77:0/3797638730 wait complete. 2022-04-03T19:48:52.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.073+0000 7f529084c700 1 Processor -- start 2022-04-03T19:48:52.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.074+0000 7f529084c700 1 -- start start 2022-04-03T19:48:52.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.074+0000 7f529084c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f528c0678b0 0x7f528c1548f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:52.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.074+0000 7f529084c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f528c0ba3c0 0x7f528c159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:52.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.074+0000 7f529084c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f528c157060 0x7f528c0c2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:52.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.074+0000 7f529084c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c155690 con 0x7f528c0ba3c0 2022-04-03T19:48:52.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.074+0000 7f529084c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528c165630 con 0x7f528c157060 2022-04-03T19:48:52.083 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.074+0000 7f529084c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f528c1658f0 con 0x7f528c0678b0 2022-04-03T19:48:52.083 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.075+0000 7f528a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f528c0ba3c0 0x7f528c159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:52.083 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.075+0000 7f528a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f528c0ba3c0 0x7f528c159930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42884/0 (socket says 172.21.15.77:42884) 2022-04-03T19:48:52.083 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.075+0000 7f528a59c700 1 -- 172.21.15.77:0/1777916680 learned_addr learned my addr 172.21.15.77:0/1777916680 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:48:52.084 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.075+0000 7f528ad9d700 1 --2- 172.21.15.77:0/1777916680 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f528c0678b0 0x7f528c1548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:52.084 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.075+0000 7f528b59e700 1 --2- 172.21.15.77:0/1777916680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f528c157060 0x7f528c0c2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:52.084 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.075+0000 7f528a59c700 1 -- 172.21.15.77:0/1777916680 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f528c0678b0 msgr2=0x7f528c1548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:52.084 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.075+0000 7f528a59c700 1 --2- 172.21.15.77:0/1777916680 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f528c0678b0 0x7f528c1548f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:52.084 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.075+0000 7f528a59c700 1 -- 172.21.15.77:0/1777916680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f528c157060 msgr2=0x7f528c0c2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:52.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.075+0000 7f528a59c700 1 --2- 172.21.15.77:0/1777916680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f528c157060 0x7f528c0c2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:52.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.075+0000 7f528a59c700 1 -- 172.21.15.77:0/1777916680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527c009370 con 0x7f528c0ba3c0 2022-04-03T19:48:52.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.076+0000 7f528a59c700 1 --2- 172.21.15.77:0/1777916680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f528c0ba3c0 0x7f528c159930 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f528000c440 tx=0x7f5280007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:52.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.076+0000 7f5272ffd700 1 -- 172.21.15.77:0/1777916680 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52800050d0 con 0x7f528c0ba3c0 2022-04-03T19:48:52.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.076+0000 7f5272ffd700 1 -- 172.21.15.77:0/1777916680 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5280005230 con 0x7f528c0ba3c0 2022-04-03T19:48:52.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.076+0000 7f5272ffd700 1 -- 172.21.15.77:0/1777916680 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5280003710 con 0x7f528c0ba3c0 2022-04-03T19:48:52.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.076+0000 7f529084c700 1 -- 172.21.15.77:0/1777916680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f528c165bd0 con 0x7f528c0ba3c0 2022-04-03T19:48:52.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.076+0000 7f529084c700 1 -- 172.21.15.77:0/1777916680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f528c166190 con 0x7f528c0ba3c0 2022-04-03T19:48:52.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.077+0000 7f529084c700 1 -- 172.21.15.77:0/1777916680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f528c068d80 con 0x7f528c0ba3c0 2022-04-03T19:48:52.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.078+0000 7f5272ffd700 1 -- 172.21.15.77:0/1777916680 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f5280019030 con 0x7f528c0ba3c0 2022-04-03T19:48:52.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.079+0000 7f5272ffd700 1 --2- 172.21.15.77:0/1777916680 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f526c063ce0 0x7f526c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:48:52.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.079+0000 7f5272ffd700 1 -- 172.21.15.77:0/1777916680 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6288+0+0 (secure 0 0 0) 0x7f528007f920 con 0x7f528c0ba3c0 2022-04-03T19:48:52.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.079+0000 7f528ad9d700 1 --2- 172.21.15.77:0/1777916680 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f526c063ce0 0x7f526c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:48:52.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.080+0000 7f528ad9d700 1 --2- 172.21.15.77:0/1777916680 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f526c063ce0 0x7f526c066190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f527c009340 tx=0x7f527c016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:48:52.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.083+0000 7f5272ffd700 1 -- 172.21.15.77:0/1777916680 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f528004e420 con 0x7f528c0ba3c0 2022-04-03T19:48:52.142 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T19:48:52.141+0000 7feb780af700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-04-03T19:48:52.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.362+0000 7f529084c700 1 -- 172.21.15.77:0/1777916680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f528c0c74a0 con 0x7f528c0ba3c0 2022-04-03T19:48:52.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.363+0000 7f5272ffd700 1 -- 172.21.15.77:0/1777916680 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6113 (secure 0 0 0) 0x7f5280017020 con 0x7f528c0ba3c0 2022-04-03T19:48:52.364 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:48:52.365 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4850,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:48:43.168138+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4960},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4960":{"gid":4960,"name":"e","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:48:52.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.365+0000 7f529084c700 1 -- 172.21.15.77:0/1777916680 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f526c063ce0 msgr2=0x7f526c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:52.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.365+0000 7f529084c700 1 --2- 172.21.15.77:0/1777916680 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f526c063ce0 0x7f526c066190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f527c009340 tx=0x7f527c016000 comp rx=0 tx=0).stop 2022-04-03T19:48:52.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.365+0000 7f529084c700 1 -- 172.21.15.77:0/1777916680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f528c0ba3c0 msgr2=0x7f528c159930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:48:52.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.365+0000 7f529084c700 1 --2- 172.21.15.77:0/1777916680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f528c0ba3c0 0x7f528c159930 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f528000c440 tx=0x7f5280007dd0 comp rx=0 tx=0).stop 2022-04-03T19:48:52.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.365+0000 7f529084c700 1 -- 172.21.15.77:0/1777916680 shutdown_connections 2022-04-03T19:48:52.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.365+0000 7f529084c700 1 --2- 172.21.15.77:0/1777916680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f528c157060 0x7f528c0c2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:52.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.365+0000 7f529084c700 1 --2- 172.21.15.77:0/1777916680 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f526c063ce0 0x7f526c066190 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:52.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.365+0000 7f529084c700 1 --2- 172.21.15.77:0/1777916680 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f528c0678b0 0x7f528c1548f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:52.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.365+0000 7f529084c700 1 --2- 172.21.15.77:0/1777916680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f528c0ba3c0 0x7f528c159930 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:48:52.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.365+0000 7f529084c700 1 -- 172.21.15.77:0/1777916680 >> 172.21.15.77:0/1777916680 conn(0x7f528c066da0 msgr2=0x7f528c0b8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:48:52.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.365+0000 7f529084c700 1 -- 172.21.15.77:0/1777916680 shutdown_connections 2022-04-03T19:48:52.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:48:52.365+0000 7f529084c700 1 -- 172.21.15.77:0/1777916680 wait complete. 2022-04-03T19:48:52.370 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 20 2022-04-03T19:48:52.390 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2022-04-03T19:48:52.391 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 52 secs before thrashing 2022-04-03T19:48:55.311 INFO:tasks.workunit.client.0.smithi077.stdout:removed './128/37' 2022-04-03T19:48:55.312 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './128' 2022-04-03T19:48:55.312 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/19' 2022-04-03T19:48:55.312 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/15' 2022-04-03T19:48:55.312 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/31' 2022-04-03T19:48:55.313 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/36' 2022-04-03T19:48:55.313 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/20' 2022-04-03T19:48:55.313 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/12' 2022-04-03T19:48:55.313 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/39' 2022-04-03T19:48:55.314 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/10' 2022-04-03T19:48:55.314 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/34' 2022-04-03T19:48:55.314 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/38' 2022-04-03T19:48:55.315 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/11' 2022-04-03T19:48:55.315 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/16' 2022-04-03T19:48:55.315 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/32' 2022-04-03T19:48:55.315 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/30' 2022-04-03T19:48:55.316 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/14' 2022-04-03T19:48:55.316 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/18' 2022-04-03T19:48:55.316 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/13' 2022-04-03T19:48:55.316 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/33' 2022-04-03T19:48:55.317 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/17' 2022-04-03T19:48:55.317 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/35' 2022-04-03T19:48:55.317 INFO:tasks.workunit.client.0.smithi077.stdout:removed './129/37' 2022-04-03T19:48:55.317 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './129' 2022-04-03T19:48:55.318 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/19' 2022-04-03T19:48:55.318 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/15' 2022-04-03T19:48:55.318 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/31' 2022-04-03T19:48:55.318 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/36' 2022-04-03T19:48:55.319 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/20' 2022-04-03T19:48:55.319 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/12' 2022-04-03T19:48:55.319 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/39' 2022-04-03T19:48:55.319 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/10' 2022-04-03T19:48:55.320 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/34' 2022-04-03T19:48:55.320 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/38' 2022-04-03T19:48:55.320 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/11' 2022-04-03T19:48:55.320 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/16' 2022-04-03T19:48:55.321 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/32' 2022-04-03T19:48:55.321 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/30' 2022-04-03T19:48:55.321 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/14' 2022-04-03T19:48:55.321 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/18' 2022-04-03T19:48:55.322 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/13' 2022-04-03T19:48:55.322 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/33' 2022-04-03T19:48:55.322 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/17' 2022-04-03T19:48:55.322 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/35' 2022-04-03T19:48:55.322 INFO:tasks.workunit.client.0.smithi077.stdout:removed './130/37' 2022-04-03T19:48:55.323 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './130' 2022-04-03T19:48:55.323 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/19' 2022-04-03T19:48:55.323 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/15' 2022-04-03T19:48:55.324 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/31' 2022-04-03T19:48:55.324 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/36' 2022-04-03T19:48:55.324 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/20' 2022-04-03T19:48:55.324 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/12' 2022-04-03T19:48:55.325 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/39' 2022-04-03T19:48:55.325 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/10' 2022-04-03T19:48:55.325 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/34' 2022-04-03T19:48:55.326 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/38' 2022-04-03T19:48:55.326 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/11' 2022-04-03T19:48:55.326 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/16' 2022-04-03T19:48:55.326 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/32' 2022-04-03T19:48:55.327 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/30' 2022-04-03T19:48:55.327 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/14' 2022-04-03T19:48:55.327 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/18' 2022-04-03T19:48:55.327 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/13' 2022-04-03T19:48:55.328 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/33' 2022-04-03T19:48:55.328 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/17' 2022-04-03T19:48:55.328 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/35' 2022-04-03T19:48:55.329 INFO:tasks.workunit.client.0.smithi077.stdout:removed './131/37' 2022-04-03T19:48:55.329 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './131' 2022-04-03T19:48:55.329 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/19' 2022-04-03T19:48:55.329 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/15' 2022-04-03T19:48:55.330 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/31' 2022-04-03T19:48:55.330 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/36' 2022-04-03T19:48:55.330 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/20' 2022-04-03T19:48:55.330 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/12' 2022-04-03T19:48:55.331 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/39' 2022-04-03T19:48:55.331 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/10' 2022-04-03T19:48:55.331 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/34' 2022-04-03T19:48:55.331 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/38' 2022-04-03T19:48:55.332 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/11' 2022-04-03T19:48:55.332 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/16' 2022-04-03T19:48:55.332 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/32' 2022-04-03T19:48:55.332 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/30' 2022-04-03T19:48:55.333 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/14' 2022-04-03T19:48:55.333 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/18' 2022-04-03T19:48:55.333 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/13' 2022-04-03T19:48:55.333 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/33' 2022-04-03T19:48:55.334 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/17' 2022-04-03T19:48:55.334 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/35' 2022-04-03T19:48:55.334 INFO:tasks.workunit.client.0.smithi077.stdout:removed './132/37' 2022-04-03T19:48:55.334 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './132' 2022-04-03T19:48:55.335 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/19' 2022-04-03T19:48:55.335 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/15' 2022-04-03T19:48:55.336 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/31' 2022-04-03T19:48:55.336 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/36' 2022-04-03T19:48:55.336 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/20' 2022-04-03T19:48:55.336 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/12' 2022-04-03T19:48:55.337 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/39' 2022-04-03T19:48:55.337 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/10' 2022-04-03T19:48:55.337 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/34' 2022-04-03T19:48:55.337 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/38' 2022-04-03T19:48:55.338 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/11' 2022-04-03T19:48:55.338 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/16' 2022-04-03T19:48:55.338 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/32' 2022-04-03T19:48:55.339 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/30' 2022-04-03T19:48:55.339 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/14' 2022-04-03T19:48:55.339 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/18' 2022-04-03T19:48:55.339 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/13' 2022-04-03T19:48:55.340 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/33' 2022-04-03T19:48:55.340 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/17' 2022-04-03T19:48:55.340 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/35' 2022-04-03T19:48:55.340 INFO:tasks.workunit.client.0.smithi077.stdout:removed './133/37' 2022-04-03T19:48:55.341 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './133' 2022-04-03T19:48:55.341 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/19' 2022-04-03T19:48:55.341 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/15' 2022-04-03T19:48:55.341 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/31' 2022-04-03T19:48:55.342 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/36' 2022-04-03T19:48:55.342 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/20' 2022-04-03T19:48:55.342 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/12' 2022-04-03T19:48:55.342 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/39' 2022-04-03T19:48:55.343 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/10' 2022-04-03T19:48:55.343 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/34' 2022-04-03T19:48:55.343 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/38' 2022-04-03T19:48:55.344 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/11' 2022-04-03T19:48:55.344 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/16' 2022-04-03T19:48:55.344 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/32' 2022-04-03T19:48:55.344 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/30' 2022-04-03T19:48:55.345 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/14' 2022-04-03T19:48:55.345 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/18' 2022-04-03T19:48:55.345 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/13' 2022-04-03T19:48:55.345 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/33' 2022-04-03T19:48:55.346 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/17' 2022-04-03T19:48:55.346 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/35' 2022-04-03T19:48:55.346 INFO:tasks.workunit.client.0.smithi077.stdout:removed './134/37' 2022-04-03T19:48:55.347 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './134' 2022-04-03T19:48:55.347 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/19' 2022-04-03T19:48:55.347 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/15' 2022-04-03T19:48:55.348 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/31' 2022-04-03T19:48:55.348 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/36' 2022-04-03T19:48:55.348 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/20' 2022-04-03T19:48:55.349 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/12' 2022-04-03T19:48:55.349 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/39' 2022-04-03T19:48:55.349 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/10' 2022-04-03T19:48:55.350 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/34' 2022-04-03T19:48:55.350 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/38' 2022-04-03T19:48:55.350 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/11' 2022-04-03T19:48:55.350 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/16' 2022-04-03T19:48:55.351 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/32' 2022-04-03T19:48:55.351 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/30' 2022-04-03T19:48:55.351 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/14' 2022-04-03T19:48:55.351 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/18' 2022-04-03T19:48:55.352 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/13' 2022-04-03T19:48:55.352 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/33' 2022-04-03T19:48:55.352 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/17' 2022-04-03T19:48:55.353 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/35' 2022-04-03T19:48:55.353 INFO:tasks.workunit.client.0.smithi077.stdout:removed './135/37' 2022-04-03T19:48:55.353 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './135' 2022-04-03T19:48:55.353 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/19' 2022-04-03T19:48:55.354 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/15' 2022-04-03T19:48:55.354 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/31' 2022-04-03T19:48:55.354 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/36' 2022-04-03T19:48:55.354 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/20' 2022-04-03T19:48:55.355 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/12' 2022-04-03T19:48:55.355 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/39' 2022-04-03T19:48:55.355 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/10' 2022-04-03T19:48:55.355 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/34' 2022-04-03T19:48:55.355 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/38' 2022-04-03T19:48:55.356 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/11' 2022-04-03T19:48:55.356 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/16' 2022-04-03T19:48:55.356 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/32' 2022-04-03T19:48:55.357 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/30' 2022-04-03T19:48:55.357 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/14' 2022-04-03T19:48:55.357 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/18' 2022-04-03T19:48:55.357 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/13' 2022-04-03T19:48:55.358 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/33' 2022-04-03T19:48:55.358 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/17' 2022-04-03T19:48:55.358 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/35' 2022-04-03T19:48:55.358 INFO:tasks.workunit.client.0.smithi077.stdout:removed './136/37' 2022-04-03T19:48:55.359 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './136' 2022-04-03T19:48:55.359 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/19' 2022-04-03T19:48:55.359 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/15' 2022-04-03T19:48:55.360 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/31' 2022-04-03T19:48:55.360 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/36' 2022-04-03T19:48:55.360 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/20' 2022-04-03T19:48:55.360 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/12' 2022-04-03T19:48:55.361 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/39' 2022-04-03T19:48:55.361 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/10' 2022-04-03T19:48:55.361 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/34' 2022-04-03T19:48:55.362 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/38' 2022-04-03T19:48:55.362 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/11' 2022-04-03T19:48:55.362 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/16' 2022-04-03T19:48:55.362 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/32' 2022-04-03T19:48:55.363 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/30' 2022-04-03T19:48:55.363 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/14' 2022-04-03T19:48:55.363 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/18' 2022-04-03T19:48:55.363 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/13' 2022-04-03T19:48:55.363 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/33' 2022-04-03T19:48:55.364 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/17' 2022-04-03T19:48:55.364 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/35' 2022-04-03T19:48:55.364 INFO:tasks.workunit.client.0.smithi077.stdout:removed './137/37' 2022-04-03T19:48:55.364 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './137' 2022-04-03T19:48:55.365 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/19' 2022-04-03T19:48:55.365 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/15' 2022-04-03T19:48:55.365 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/31' 2022-04-03T19:48:55.366 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/36' 2022-04-03T19:48:55.366 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/20' 2022-04-03T19:48:55.366 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/12' 2022-04-03T19:48:55.367 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/39' 2022-04-03T19:48:55.367 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/10' 2022-04-03T19:48:55.367 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/34' 2022-04-03T19:48:55.367 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/38' 2022-04-03T19:48:55.368 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/11' 2022-04-03T19:48:55.368 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/16' 2022-04-03T19:48:57.042 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/32' 2022-04-03T19:48:57.042 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/30' 2022-04-03T19:48:57.042 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/14' 2022-04-03T19:48:57.043 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/18' 2022-04-03T19:48:57.043 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/13' 2022-04-03T19:48:57.043 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/33' 2022-04-03T19:48:57.043 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/17' 2022-04-03T19:48:57.043 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/35' 2022-04-03T19:48:57.044 INFO:tasks.workunit.client.0.smithi077.stdout:removed './138/37' 2022-04-03T19:48:57.044 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './138' 2022-04-03T19:48:57.044 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/19' 2022-04-03T19:48:57.044 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/15' 2022-04-03T19:48:57.045 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/31' 2022-04-03T19:48:57.045 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/36' 2022-04-03T19:48:57.045 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/20' 2022-04-03T19:48:57.045 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/12' 2022-04-03T19:48:57.045 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/39' 2022-04-03T19:48:57.046 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/10' 2022-04-03T19:48:57.046 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/34' 2022-04-03T19:48:57.046 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/38' 2022-04-03T19:48:57.046 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/11' 2022-04-03T19:48:57.046 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/16' 2022-04-03T19:48:57.047 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/32' 2022-04-03T19:48:57.047 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/30' 2022-04-03T19:48:57.047 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/14' 2022-04-03T19:48:57.048 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/18' 2022-04-03T19:48:57.048 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/13' 2022-04-03T19:48:57.048 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/33' 2022-04-03T19:48:57.049 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/17' 2022-04-03T19:48:57.049 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/35' 2022-04-03T19:48:57.049 INFO:tasks.workunit.client.0.smithi077.stdout:removed './139/37' 2022-04-03T19:48:57.049 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './139' 2022-04-03T19:48:57.049 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/19' 2022-04-03T19:48:57.050 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/15' 2022-04-03T19:48:57.050 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/31' 2022-04-03T19:48:57.050 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/36' 2022-04-03T19:48:57.050 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/20' 2022-04-03T19:48:57.051 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/12' 2022-04-03T19:48:57.051 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/39' 2022-04-03T19:48:57.051 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/10' 2022-04-03T19:48:57.051 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/34' 2022-04-03T19:48:57.052 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/38' 2022-04-03T19:48:57.052 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/11' 2022-04-03T19:48:57.053 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/16' 2022-04-03T19:48:57.053 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/32' 2022-04-03T19:48:57.053 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/30' 2022-04-03T19:48:57.054 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/14' 2022-04-03T19:48:57.054 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/18' 2022-04-03T19:48:57.054 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/13' 2022-04-03T19:48:57.055 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/33' 2022-04-03T19:48:57.055 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/17' 2022-04-03T19:48:57.055 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/35' 2022-04-03T19:48:57.056 INFO:tasks.workunit.client.0.smithi077.stdout:removed './140/37' 2022-04-03T19:48:57.056 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './140' 2022-04-03T19:48:57.056 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/19' 2022-04-03T19:48:57.056 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/15' 2022-04-03T19:48:57.057 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/31' 2022-04-03T19:48:57.057 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/36' 2022-04-03T19:48:57.057 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/20' 2022-04-03T19:48:57.057 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/12' 2022-04-03T19:48:57.057 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/39' 2022-04-03T19:48:57.058 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/10' 2022-04-03T19:48:57.058 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/34' 2022-04-03T19:48:57.058 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/38' 2022-04-03T19:48:57.058 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/11' 2022-04-03T19:48:57.059 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/16' 2022-04-03T19:48:57.059 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/32' 2022-04-03T19:48:57.059 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/30' 2022-04-03T19:48:57.059 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/14' 2022-04-03T19:48:57.060 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/18' 2022-04-03T19:48:57.060 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/13' 2022-04-03T19:48:57.060 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/33' 2022-04-03T19:48:57.060 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/17' 2022-04-03T19:48:57.061 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/35' 2022-04-03T19:48:57.061 INFO:tasks.workunit.client.0.smithi077.stdout:removed './141/37' 2022-04-03T19:48:57.061 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './141' 2022-04-03T19:48:57.061 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/19' 2022-04-03T19:48:57.061 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/15' 2022-04-03T19:48:57.062 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/31' 2022-04-03T19:48:57.062 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/36' 2022-04-03T19:48:57.062 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/20' 2022-04-03T19:48:57.062 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/12' 2022-04-03T19:48:57.063 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/39' 2022-04-03T19:48:57.063 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/10' 2022-04-03T19:48:57.063 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/34' 2022-04-03T19:48:57.063 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/38' 2022-04-03T19:48:57.063 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/11' 2022-04-03T19:48:57.064 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/16' 2022-04-03T19:48:57.064 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/32' 2022-04-03T19:48:57.064 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/30' 2022-04-03T19:48:57.065 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/14' 2022-04-03T19:48:57.065 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/18' 2022-04-03T19:48:57.066 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/13' 2022-04-03T19:48:57.066 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/33' 2022-04-03T19:48:57.066 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/17' 2022-04-03T19:48:57.066 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/35' 2022-04-03T19:48:57.067 INFO:tasks.workunit.client.0.smithi077.stdout:removed './142/37' 2022-04-03T19:48:57.067 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './142' 2022-04-03T19:48:57.067 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/19' 2022-04-03T19:48:57.067 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/15' 2022-04-03T19:48:57.068 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/31' 2022-04-03T19:48:57.068 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/36' 2022-04-03T19:48:57.068 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/20' 2022-04-03T19:48:57.068 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/12' 2022-04-03T19:48:57.068 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/39' 2022-04-03T19:48:57.069 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/10' 2022-04-03T19:48:57.069 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/34' 2022-04-03T19:48:57.069 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/38' 2022-04-03T19:48:57.069 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/11' 2022-04-03T19:48:57.069 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/16' 2022-04-03T19:48:57.070 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/32' 2022-04-03T19:48:57.070 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/30' 2022-04-03T19:48:57.070 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/14' 2022-04-03T19:48:57.070 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/18' 2022-04-03T19:48:57.071 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/13' 2022-04-03T19:48:57.071 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/33' 2022-04-03T19:48:57.071 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/17' 2022-04-03T19:48:57.071 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/35' 2022-04-03T19:48:57.071 INFO:tasks.workunit.client.0.smithi077.stdout:removed './143/37' 2022-04-03T19:48:57.072 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './143' 2022-04-03T19:48:57.072 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/19' 2022-04-03T19:48:57.072 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/15' 2022-04-03T19:48:57.072 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/31' 2022-04-03T19:48:57.073 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/36' 2022-04-03T19:48:57.073 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/20' 2022-04-03T19:48:57.073 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/12' 2022-04-03T19:48:57.073 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/39' 2022-04-03T19:48:57.073 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/10' 2022-04-03T19:48:57.074 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/34' 2022-04-03T19:48:57.074 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/38' 2022-04-03T19:48:57.074 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/11' 2022-04-03T19:48:57.074 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/16' 2022-04-03T19:48:57.075 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/32' 2022-04-03T19:48:57.075 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/30' 2022-04-03T19:48:57.075 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/14' 2022-04-03T19:48:57.075 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/18' 2022-04-03T19:48:57.076 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/13' 2022-04-03T19:48:57.076 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/33' 2022-04-03T19:48:57.076 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/17' 2022-04-03T19:48:57.076 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/35' 2022-04-03T19:48:57.076 INFO:tasks.workunit.client.0.smithi077.stdout:removed './144/37' 2022-04-03T19:48:57.077 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './144' 2022-04-03T19:48:57.077 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/19' 2022-04-03T19:48:57.077 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/15' 2022-04-03T19:48:57.078 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/31' 2022-04-03T19:48:57.078 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/36' 2022-04-03T19:48:57.079 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/20' 2022-04-03T19:48:57.079 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/12' 2022-04-03T19:48:57.079 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/39' 2022-04-03T19:48:57.080 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/10' 2022-04-03T19:48:57.080 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/34' 2022-04-03T19:48:57.081 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/38' 2022-04-03T19:48:57.081 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/11' 2022-04-03T19:48:57.081 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/16' 2022-04-03T19:48:57.081 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/32' 2022-04-03T19:48:57.082 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/30' 2022-04-03T19:48:57.082 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/14' 2022-04-03T19:48:57.082 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/18' 2022-04-03T19:48:57.082 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/13' 2022-04-03T19:48:57.083 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/33' 2022-04-03T19:48:57.083 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/17' 2022-04-03T19:48:57.083 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/35' 2022-04-03T19:48:57.083 INFO:tasks.workunit.client.0.smithi077.stdout:removed './145/37' 2022-04-03T19:48:57.083 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './145' 2022-04-03T19:48:57.084 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/19' 2022-04-03T19:48:57.084 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/15' 2022-04-03T19:48:57.084 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/31' 2022-04-03T19:48:57.084 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/36' 2022-04-03T19:48:57.084 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/20' 2022-04-03T19:48:57.085 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/12' 2022-04-03T19:48:57.085 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/39' 2022-04-03T19:48:57.085 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/10' 2022-04-03T19:48:57.085 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/34' 2022-04-03T19:48:57.086 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/38' 2022-04-03T19:48:57.086 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/11' 2022-04-03T19:48:57.086 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/16' 2022-04-03T19:48:57.086 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/32' 2022-04-03T19:48:57.086 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/30' 2022-04-03T19:48:57.087 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/14' 2022-04-03T19:48:57.087 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/18' 2022-04-03T19:48:57.087 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/13' 2022-04-03T19:48:57.087 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/33' 2022-04-03T19:48:57.088 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/17' 2022-04-03T19:48:57.088 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/35' 2022-04-03T19:48:57.088 INFO:tasks.workunit.client.0.smithi077.stdout:removed './146/37' 2022-04-03T19:48:57.088 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './146' 2022-04-03T19:48:57.088 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/19' 2022-04-03T19:48:57.089 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/15' 2022-04-03T19:48:57.089 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/31' 2022-04-03T19:48:57.089 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/36' 2022-04-03T19:48:57.089 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/20' 2022-04-03T19:48:57.090 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/12' 2022-04-03T19:48:57.090 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/39' 2022-04-03T19:48:57.090 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/10' 2022-04-03T19:48:57.090 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/34' 2022-04-03T19:48:57.090 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/38' 2022-04-03T19:48:57.091 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/11' 2022-04-03T19:48:57.091 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/16' 2022-04-03T19:48:57.091 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/32' 2022-04-03T19:48:57.091 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/30' 2022-04-03T19:48:57.092 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/14' 2022-04-03T19:48:57.092 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/18' 2022-04-03T19:48:57.092 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/13' 2022-04-03T19:48:57.092 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/33' 2022-04-03T19:48:57.092 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/17' 2022-04-03T19:48:57.093 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/35' 2022-04-03T19:48:57.093 INFO:tasks.workunit.client.0.smithi077.stdout:removed './147/37' 2022-04-03T19:48:57.093 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './147' 2022-04-03T19:48:57.093 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/19' 2022-04-03T19:48:57.094 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/15' 2022-04-03T19:48:57.094 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/31' 2022-04-03T19:48:57.094 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/36' 2022-04-03T19:48:58.224 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/20' 2022-04-03T19:48:58.224 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/12' 2022-04-03T19:48:58.224 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/39' 2022-04-03T19:48:58.225 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/10' 2022-04-03T19:48:58.225 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/34' 2022-04-03T19:48:58.225 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/38' 2022-04-03T19:48:58.225 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/11' 2022-04-03T19:48:58.225 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/16' 2022-04-03T19:48:58.226 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/32' 2022-04-03T19:48:58.226 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/30' 2022-04-03T19:48:58.226 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/14' 2022-04-03T19:48:58.226 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/18' 2022-04-03T19:48:58.227 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/13' 2022-04-03T19:48:58.227 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/33' 2022-04-03T19:48:58.227 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/17' 2022-04-03T19:48:58.227 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/35' 2022-04-03T19:48:58.227 INFO:tasks.workunit.client.0.smithi077.stdout:removed './148/37' 2022-04-03T19:48:58.228 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './148' 2022-04-03T19:48:58.228 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/19' 2022-04-03T19:48:58.228 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/15' 2022-04-03T19:48:58.228 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/31' 2022-04-03T19:48:58.228 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/36' 2022-04-03T19:48:58.229 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/20' 2022-04-03T19:48:58.229 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/12' 2022-04-03T19:48:58.229 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/39' 2022-04-03T19:48:58.229 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/10' 2022-04-03T19:48:58.230 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/34' 2022-04-03T19:48:58.230 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/38' 2022-04-03T19:48:58.230 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/11' 2022-04-03T19:48:58.230 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/16' 2022-04-03T19:48:58.230 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/32' 2022-04-03T19:48:58.231 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/30' 2022-04-03T19:48:58.231 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/14' 2022-04-03T19:48:58.231 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/18' 2022-04-03T19:48:58.231 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/13' 2022-04-03T19:48:58.231 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/33' 2022-04-03T19:48:58.232 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/17' 2022-04-03T19:48:58.232 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/35' 2022-04-03T19:48:58.232 INFO:tasks.workunit.client.0.smithi077.stdout:removed './149/37' 2022-04-03T19:48:58.232 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './149' 2022-04-03T19:48:58.232 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/19' 2022-04-03T19:48:58.233 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/15' 2022-04-03T19:48:58.233 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/31' 2022-04-03T19:48:58.233 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/36' 2022-04-03T19:48:58.233 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/20' 2022-04-03T19:48:58.234 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/12' 2022-04-03T19:48:58.234 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/39' 2022-04-03T19:48:58.234 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/10' 2022-04-03T19:48:58.234 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/34' 2022-04-03T19:48:58.235 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/38' 2022-04-03T19:48:58.235 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/11' 2022-04-03T19:48:58.235 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/16' 2022-04-03T19:48:58.235 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/32' 2022-04-03T19:48:58.235 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/30' 2022-04-03T19:48:58.236 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/14' 2022-04-03T19:48:58.236 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/18' 2022-04-03T19:48:58.236 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/13' 2022-04-03T19:48:58.236 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/33' 2022-04-03T19:48:58.236 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/17' 2022-04-03T19:48:58.237 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/35' 2022-04-03T19:48:58.237 INFO:tasks.workunit.client.0.smithi077.stdout:removed './150/37' 2022-04-03T19:48:58.237 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './150' 2022-04-03T19:48:58.237 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/19' 2022-04-03T19:48:58.238 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/15' 2022-04-03T19:48:58.238 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/31' 2022-04-03T19:48:58.238 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/36' 2022-04-03T19:48:58.238 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/20' 2022-04-03T19:48:58.238 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/12' 2022-04-03T19:48:58.239 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/39' 2022-04-03T19:48:58.239 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/10' 2022-04-03T19:48:58.239 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/34' 2022-04-03T19:48:58.239 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/38' 2022-04-03T19:48:58.239 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/11' 2022-04-03T19:48:58.240 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/16' 2022-04-03T19:48:58.240 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/32' 2022-04-03T19:48:58.240 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/30' 2022-04-03T19:48:58.240 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/14' 2022-04-03T19:48:58.241 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/18' 2022-04-03T19:48:58.241 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/13' 2022-04-03T19:48:58.241 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/33' 2022-04-03T19:48:58.241 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/17' 2022-04-03T19:48:58.241 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/35' 2022-04-03T19:48:58.242 INFO:tasks.workunit.client.0.smithi077.stdout:removed './151/37' 2022-04-03T19:48:58.242 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './151' 2022-04-03T19:48:58.242 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/19' 2022-04-03T19:48:58.242 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/15' 2022-04-03T19:48:58.243 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/31' 2022-04-03T19:48:58.243 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/36' 2022-04-03T19:48:58.243 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/20' 2022-04-03T19:48:58.243 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/12' 2022-04-03T19:48:58.243 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/39' 2022-04-03T19:48:58.244 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/10' 2022-04-03T19:48:58.244 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/34' 2022-04-03T19:48:58.244 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/38' 2022-04-03T19:48:58.244 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/11' 2022-04-03T19:48:58.244 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/16' 2022-04-03T19:48:58.245 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/32' 2022-04-03T19:48:58.245 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/30' 2022-04-03T19:48:58.245 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/14' 2022-04-03T19:48:58.245 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/18' 2022-04-03T19:48:58.245 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/13' 2022-04-03T19:48:58.246 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/33' 2022-04-03T19:48:58.246 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/17' 2022-04-03T19:48:58.246 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/35' 2022-04-03T19:48:58.246 INFO:tasks.workunit.client.0.smithi077.stdout:removed './152/37' 2022-04-03T19:48:58.247 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './152' 2022-04-03T19:48:58.247 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/19' 2022-04-03T19:48:58.247 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/15' 2022-04-03T19:48:58.247 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/31' 2022-04-03T19:48:58.247 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/36' 2022-04-03T19:48:58.248 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/20' 2022-04-03T19:48:58.248 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/12' 2022-04-03T19:48:58.248 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/39' 2022-04-03T19:48:58.248 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/10' 2022-04-03T19:48:58.248 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/34' 2022-04-03T19:48:58.249 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/38' 2022-04-03T19:48:58.249 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/11' 2022-04-03T19:48:58.249 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/16' 2022-04-03T19:48:58.249 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/32' 2022-04-03T19:48:58.249 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/30' 2022-04-03T19:48:58.250 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/14' 2022-04-03T19:48:58.250 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/18' 2022-04-03T19:48:58.250 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/13' 2022-04-03T19:48:58.250 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/33' 2022-04-03T19:48:58.251 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/17' 2022-04-03T19:48:58.251 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/35' 2022-04-03T19:48:58.251 INFO:tasks.workunit.client.0.smithi077.stdout:removed './153/37' 2022-04-03T19:48:58.251 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './153' 2022-04-03T19:48:58.251 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/19' 2022-04-03T19:48:58.252 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/15' 2022-04-03T19:48:58.252 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/31' 2022-04-03T19:48:58.252 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/36' 2022-04-03T19:48:58.252 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/20' 2022-04-03T19:48:58.253 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/12' 2022-04-03T19:48:58.253 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/39' 2022-04-03T19:48:58.253 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/10' 2022-04-03T19:48:58.253 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/34' 2022-04-03T19:48:58.253 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/38' 2022-04-03T19:48:58.254 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/11' 2022-04-03T19:48:58.254 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/16' 2022-04-03T19:48:58.254 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/32' 2022-04-03T19:48:58.254 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/30' 2022-04-03T19:48:58.254 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/14' 2022-04-03T19:48:58.255 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/18' 2022-04-03T19:48:58.255 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/13' 2022-04-03T19:48:58.255 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/33' 2022-04-03T19:48:58.255 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/17' 2022-04-03T19:48:58.256 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/35' 2022-04-03T19:48:58.256 INFO:tasks.workunit.client.0.smithi077.stdout:removed './154/37' 2022-04-03T19:48:58.256 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './154' 2022-04-03T19:48:58.256 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/19' 2022-04-03T19:48:58.256 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/15' 2022-04-03T19:48:58.257 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/31' 2022-04-03T19:48:58.257 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/36' 2022-04-03T19:48:58.257 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/20' 2022-04-03T19:48:58.257 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/12' 2022-04-03T19:48:58.258 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/39' 2022-04-03T19:48:58.258 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/10' 2022-04-03T19:48:58.258 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/34' 2022-04-03T19:48:58.258 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/38' 2022-04-03T19:48:58.259 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/11' 2022-04-03T19:48:58.259 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/16' 2022-04-03T19:48:58.259 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/32' 2022-04-03T19:48:58.259 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/30' 2022-04-03T19:48:58.260 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/14' 2022-04-03T19:48:58.260 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/18' 2022-04-03T19:48:58.260 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/13' 2022-04-03T19:48:58.260 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/33' 2022-04-03T19:48:58.260 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/17' 2022-04-03T19:48:58.261 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/35' 2022-04-03T19:48:58.261 INFO:tasks.workunit.client.0.smithi077.stdout:removed './155/37' 2022-04-03T19:48:58.261 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './155' 2022-04-03T19:48:58.261 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/19' 2022-04-03T19:48:58.261 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/15' 2022-04-03T19:48:58.262 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/31' 2022-04-03T19:48:58.262 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/36' 2022-04-03T19:48:58.262 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/20' 2022-04-03T19:48:58.262 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/12' 2022-04-03T19:48:58.263 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/39' 2022-04-03T19:48:58.263 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/10' 2022-04-03T19:48:58.263 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/34' 2022-04-03T19:48:58.263 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/38' 2022-04-03T19:48:58.263 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/11' 2022-04-03T19:48:58.264 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/16' 2022-04-03T19:48:58.264 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/32' 2022-04-03T19:48:58.264 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/30' 2022-04-03T19:48:58.264 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/14' 2022-04-03T19:48:58.264 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/18' 2022-04-03T19:48:58.265 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/13' 2022-04-03T19:48:58.265 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/33' 2022-04-03T19:48:58.265 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/17' 2022-04-03T19:48:58.265 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/35' 2022-04-03T19:48:58.265 INFO:tasks.workunit.client.0.smithi077.stdout:removed './156/37' 2022-04-03T19:48:58.266 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './156' 2022-04-03T19:48:58.266 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/19' 2022-04-03T19:48:58.266 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/15' 2022-04-03T19:48:58.266 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/31' 2022-04-03T19:48:58.267 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/36' 2022-04-03T19:48:58.267 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/20' 2022-04-03T19:48:58.267 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/12' 2022-04-03T19:48:58.267 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/39' 2022-04-03T19:48:58.267 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/10' 2022-04-03T19:48:58.268 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/34' 2022-04-03T19:48:58.268 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/38' 2022-04-03T19:48:58.268 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/11' 2022-04-03T19:48:58.268 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/16' 2022-04-03T19:48:58.268 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/32' 2022-04-03T19:48:58.269 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/30' 2022-04-03T19:48:58.269 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/14' 2022-04-03T19:48:58.269 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/18' 2022-04-03T19:48:58.269 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/13' 2022-04-03T19:48:58.269 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/33' 2022-04-03T19:48:59.930 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/17' 2022-04-03T19:48:59.931 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/35' 2022-04-03T19:48:59.931 INFO:tasks.workunit.client.0.smithi077.stdout:removed './157/37' 2022-04-03T19:48:59.931 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './157' 2022-04-03T19:48:59.931 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/19' 2022-04-03T19:48:59.932 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/15' 2022-04-03T19:48:59.932 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/31' 2022-04-03T19:48:59.932 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/36' 2022-04-03T19:48:59.932 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/20' 2022-04-03T19:48:59.932 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/12' 2022-04-03T19:48:59.933 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/39' 2022-04-03T19:48:59.933 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/10' 2022-04-03T19:48:59.933 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/34' 2022-04-03T19:48:59.933 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/38' 2022-04-03T19:48:59.934 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/11' 2022-04-03T19:48:59.934 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/16' 2022-04-03T19:48:59.934 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/32' 2022-04-03T19:48:59.934 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/30' 2022-04-03T19:48:59.934 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/14' 2022-04-03T19:48:59.935 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/18' 2022-04-03T19:48:59.935 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/13' 2022-04-03T19:48:59.935 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/33' 2022-04-03T19:48:59.935 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/17' 2022-04-03T19:48:59.936 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/35' 2022-04-03T19:48:59.936 INFO:tasks.workunit.client.0.smithi077.stdout:removed './158/37' 2022-04-03T19:48:59.936 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './158' 2022-04-03T19:48:59.936 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/19' 2022-04-03T19:48:59.937 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/15' 2022-04-03T19:48:59.937 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/31' 2022-04-03T19:48:59.937 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/36' 2022-04-03T19:48:59.937 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/20' 2022-04-03T19:48:59.938 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/12' 2022-04-03T19:48:59.938 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/39' 2022-04-03T19:48:59.938 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/10' 2022-04-03T19:48:59.938 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/34' 2022-04-03T19:48:59.938 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/38' 2022-04-03T19:48:59.939 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/11' 2022-04-03T19:48:59.939 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/16' 2022-04-03T19:48:59.939 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/32' 2022-04-03T19:48:59.939 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/30' 2022-04-03T19:48:59.939 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/14' 2022-04-03T19:48:59.940 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/18' 2022-04-03T19:48:59.940 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/13' 2022-04-03T19:48:59.940 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/33' 2022-04-03T19:48:59.940 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/17' 2022-04-03T19:48:59.941 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/35' 2022-04-03T19:48:59.941 INFO:tasks.workunit.client.0.smithi077.stdout:removed './159/37' 2022-04-03T19:48:59.941 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './159' 2022-04-03T19:48:59.941 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/19' 2022-04-03T19:48:59.941 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/15' 2022-04-03T19:48:59.942 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/31' 2022-04-03T19:48:59.942 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/36' 2022-04-03T19:48:59.942 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/20' 2022-04-03T19:48:59.942 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/12' 2022-04-03T19:48:59.943 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/39' 2022-04-03T19:48:59.943 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/10' 2022-04-03T19:48:59.943 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/34' 2022-04-03T19:48:59.943 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/38' 2022-04-03T19:48:59.943 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/11' 2022-04-03T19:48:59.944 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/16' 2022-04-03T19:48:59.944 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/32' 2022-04-03T19:48:59.944 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/30' 2022-04-03T19:48:59.944 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/14' 2022-04-03T19:48:59.944 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/18' 2022-04-03T19:48:59.945 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/13' 2022-04-03T19:48:59.945 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/33' 2022-04-03T19:48:59.945 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/17' 2022-04-03T19:48:59.945 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/35' 2022-04-03T19:48:59.946 INFO:tasks.workunit.client.0.smithi077.stdout:removed './160/37' 2022-04-03T19:48:59.946 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './160' 2022-04-03T19:48:59.946 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/19' 2022-04-03T19:48:59.946 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/15' 2022-04-03T19:48:59.946 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/31' 2022-04-03T19:48:59.947 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/36' 2022-04-03T19:48:59.947 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/20' 2022-04-03T19:48:59.947 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/12' 2022-04-03T19:48:59.947 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/39' 2022-04-03T19:48:59.947 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/10' 2022-04-03T19:48:59.948 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/34' 2022-04-03T19:48:59.948 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/38' 2022-04-03T19:48:59.948 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/11' 2022-04-03T19:48:59.948 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/16' 2022-04-03T19:48:59.949 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/32' 2022-04-03T19:48:59.949 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/30' 2022-04-03T19:48:59.949 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/14' 2022-04-03T19:48:59.949 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/18' 2022-04-03T19:48:59.949 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/13' 2022-04-03T19:48:59.950 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/33' 2022-04-03T19:48:59.950 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/17' 2022-04-03T19:48:59.950 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/35' 2022-04-03T19:48:59.950 INFO:tasks.workunit.client.0.smithi077.stdout:removed './161/37' 2022-04-03T19:48:59.950 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './161' 2022-04-03T19:48:59.951 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/19' 2022-04-03T19:48:59.951 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/15' 2022-04-03T19:48:59.951 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/31' 2022-04-03T19:48:59.951 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/36' 2022-04-03T19:48:59.952 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/20' 2022-04-03T19:48:59.952 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/12' 2022-04-03T19:48:59.952 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/39' 2022-04-03T19:48:59.952 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/10' 2022-04-03T19:48:59.952 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/34' 2022-04-03T19:48:59.953 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/38' 2022-04-03T19:48:59.953 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/11' 2022-04-03T19:48:59.953 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/16' 2022-04-03T19:48:59.953 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/32' 2022-04-03T19:48:59.953 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/30' 2022-04-03T19:48:59.954 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/14' 2022-04-03T19:48:59.954 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/18' 2022-04-03T19:48:59.954 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/13' 2022-04-03T19:48:59.954 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/33' 2022-04-03T19:48:59.954 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/17' 2022-04-03T19:48:59.955 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/35' 2022-04-03T19:48:59.955 INFO:tasks.workunit.client.0.smithi077.stdout:removed './162/37' 2022-04-03T19:48:59.955 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './162' 2022-04-03T19:48:59.955 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/19' 2022-04-03T19:48:59.956 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/15' 2022-04-03T19:48:59.956 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/31' 2022-04-03T19:48:59.956 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/36' 2022-04-03T19:48:59.956 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/20' 2022-04-03T19:48:59.956 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/12' 2022-04-03T19:48:59.957 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/39' 2022-04-03T19:48:59.957 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/10' 2022-04-03T19:48:59.957 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/34' 2022-04-03T19:48:59.957 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/38' 2022-04-03T19:48:59.957 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/11' 2022-04-03T19:48:59.958 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/16' 2022-04-03T19:48:59.958 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/32' 2022-04-03T19:48:59.958 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/30' 2022-04-03T19:48:59.958 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/14' 2022-04-03T19:48:59.959 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/18' 2022-04-03T19:48:59.959 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/13' 2022-04-03T19:48:59.959 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/33' 2022-04-03T19:48:59.959 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/17' 2022-04-03T19:48:59.959 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/35' 2022-04-03T19:48:59.960 INFO:tasks.workunit.client.0.smithi077.stdout:removed './163/37' 2022-04-03T19:48:59.960 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './163' 2022-04-03T19:48:59.960 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/19' 2022-04-03T19:48:59.960 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/15' 2022-04-03T19:48:59.960 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/31' 2022-04-03T19:48:59.961 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/36' 2022-04-03T19:48:59.961 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/20' 2022-04-03T19:48:59.961 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/12' 2022-04-03T19:48:59.961 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/39' 2022-04-03T19:48:59.962 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/10' 2022-04-03T19:48:59.962 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/34' 2022-04-03T19:48:59.962 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/38' 2022-04-03T19:48:59.962 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/11' 2022-04-03T19:48:59.962 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/16' 2022-04-03T19:48:59.963 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/32' 2022-04-03T19:48:59.963 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/30' 2022-04-03T19:48:59.963 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/14' 2022-04-03T19:48:59.963 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/18' 2022-04-03T19:48:59.963 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/13' 2022-04-03T19:48:59.964 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/33' 2022-04-03T19:48:59.964 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/17' 2022-04-03T19:48:59.964 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/35' 2022-04-03T19:48:59.964 INFO:tasks.workunit.client.0.smithi077.stdout:removed './164/37' 2022-04-03T19:48:59.965 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './164' 2022-04-03T19:48:59.965 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/19' 2022-04-03T19:48:59.965 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/15' 2022-04-03T19:48:59.965 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/31' 2022-04-03T19:48:59.965 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/36' 2022-04-03T19:48:59.966 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/20' 2022-04-03T19:48:59.966 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/12' 2022-04-03T19:48:59.966 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/39' 2022-04-03T19:48:59.966 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/10' 2022-04-03T19:48:59.966 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/34' 2022-04-03T19:48:59.967 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/38' 2022-04-03T19:48:59.967 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/11' 2022-04-03T19:48:59.967 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/16' 2022-04-03T19:48:59.967 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/32' 2022-04-03T19:48:59.968 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/30' 2022-04-03T19:48:59.968 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/14' 2022-04-03T19:48:59.968 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/18' 2022-04-03T19:48:59.968 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/13' 2022-04-03T19:48:59.968 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/33' 2022-04-03T19:48:59.969 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/17' 2022-04-03T19:48:59.969 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/35' 2022-04-03T19:48:59.969 INFO:tasks.workunit.client.0.smithi077.stdout:removed './165/37' 2022-04-03T19:48:59.969 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './165' 2022-04-03T19:48:59.969 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/19' 2022-04-03T19:48:59.970 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/15' 2022-04-03T19:48:59.970 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/31' 2022-04-03T19:48:59.970 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/36' 2022-04-03T19:48:59.970 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/20' 2022-04-03T19:48:59.971 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/12' 2022-04-03T19:48:59.971 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/39' 2022-04-03T19:48:59.971 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/10' 2022-04-03T19:48:59.971 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/34' 2022-04-03T19:48:59.971 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/38' 2022-04-03T19:48:59.972 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/11' 2022-04-03T19:48:59.972 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/16' 2022-04-03T19:48:59.972 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/32' 2022-04-03T19:48:59.972 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/30' 2022-04-03T19:48:59.972 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/14' 2022-04-03T19:48:59.973 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/18' 2022-04-03T19:48:59.973 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/13' 2022-04-03T19:48:59.973 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/33' 2022-04-03T19:48:59.973 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/17' 2022-04-03T19:48:59.973 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/35' 2022-04-03T19:48:59.974 INFO:tasks.workunit.client.0.smithi077.stdout:removed './166/37' 2022-04-03T19:48:59.974 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './166' 2022-04-03T19:48:59.974 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/19' 2022-04-03T19:48:59.974 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/15' 2022-04-03T19:48:59.975 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/31' 2022-04-03T19:48:59.975 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/36' 2022-04-03T19:48:59.975 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/20' 2022-04-03T19:48:59.975 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/12' 2022-04-03T19:48:59.975 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/39' 2022-04-03T19:48:59.976 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/10' 2022-04-03T19:48:59.976 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/34' 2022-04-03T19:48:59.976 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/38' 2022-04-03T19:49:00.927 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/11' 2022-04-03T19:49:00.927 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/16' 2022-04-03T19:49:00.927 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/32' 2022-04-03T19:49:00.928 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/30' 2022-04-03T19:49:00.928 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/14' 2022-04-03T19:49:00.928 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/18' 2022-04-03T19:49:00.928 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/13' 2022-04-03T19:49:00.929 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/33' 2022-04-03T19:49:00.929 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/17' 2022-04-03T19:49:00.929 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/35' 2022-04-03T19:49:00.929 INFO:tasks.workunit.client.0.smithi077.stdout:removed './167/37' 2022-04-03T19:49:00.929 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './167' 2022-04-03T19:49:00.930 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/19' 2022-04-03T19:49:00.930 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/15' 2022-04-03T19:49:00.930 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/31' 2022-04-03T19:49:00.930 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/36' 2022-04-03T19:49:00.931 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/20' 2022-04-03T19:49:00.931 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/12' 2022-04-03T19:49:00.931 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/39' 2022-04-03T19:49:00.932 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/10' 2022-04-03T19:49:00.932 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/34' 2022-04-03T19:49:00.933 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/38' 2022-04-03T19:49:00.933 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/11' 2022-04-03T19:49:00.934 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/16' 2022-04-03T19:49:00.934 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/32' 2022-04-03T19:49:00.934 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/30' 2022-04-03T19:49:00.935 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/14' 2022-04-03T19:49:00.935 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/18' 2022-04-03T19:49:00.936 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/13' 2022-04-03T19:49:00.936 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/33' 2022-04-03T19:49:00.937 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/17' 2022-04-03T19:49:00.937 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/35' 2022-04-03T19:49:00.938 INFO:tasks.workunit.client.0.smithi077.stdout:removed './168/37' 2022-04-03T19:49:00.938 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './168' 2022-04-03T19:49:00.938 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/19' 2022-04-03T19:49:00.939 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/15' 2022-04-03T19:49:00.939 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/31' 2022-04-03T19:49:00.939 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/36' 2022-04-03T19:49:00.939 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/20' 2022-04-03T19:49:00.940 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/12' 2022-04-03T19:49:00.940 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/39' 2022-04-03T19:49:00.940 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/10' 2022-04-03T19:49:00.941 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/34' 2022-04-03T19:49:00.941 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/38' 2022-04-03T19:49:00.941 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/11' 2022-04-03T19:49:00.942 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/16' 2022-04-03T19:49:00.942 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/32' 2022-04-03T19:49:00.943 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/30' 2022-04-03T19:49:00.943 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/14' 2022-04-03T19:49:00.944 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/18' 2022-04-03T19:49:00.944 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/13' 2022-04-03T19:49:00.944 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/33' 2022-04-03T19:49:00.945 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/17' 2022-04-03T19:49:00.945 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/35' 2022-04-03T19:49:00.946 INFO:tasks.workunit.client.0.smithi077.stdout:removed './169/37' 2022-04-03T19:49:00.946 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './169' 2022-04-03T19:49:00.946 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/19' 2022-04-03T19:49:00.947 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/15' 2022-04-03T19:49:00.947 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/31' 2022-04-03T19:49:00.947 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/36' 2022-04-03T19:49:00.948 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/20' 2022-04-03T19:49:00.948 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/12' 2022-04-03T19:49:00.948 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/39' 2022-04-03T19:49:00.948 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/10' 2022-04-03T19:49:00.949 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/34' 2022-04-03T19:49:00.949 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/38' 2022-04-03T19:49:00.949 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/11' 2022-04-03T19:49:00.949 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/16' 2022-04-03T19:49:00.949 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/32' 2022-04-03T19:49:00.950 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/30' 2022-04-03T19:49:00.950 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/14' 2022-04-03T19:49:00.950 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/18' 2022-04-03T19:49:00.950 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/13' 2022-04-03T19:49:00.951 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/33' 2022-04-03T19:49:00.951 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/17' 2022-04-03T19:49:00.951 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/35' 2022-04-03T19:49:00.951 INFO:tasks.workunit.client.0.smithi077.stdout:removed './170/37' 2022-04-03T19:49:00.951 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './170' 2022-04-03T19:49:00.952 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/19' 2022-04-03T19:49:00.952 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/15' 2022-04-03T19:49:00.952 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/31' 2022-04-03T19:49:00.952 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/36' 2022-04-03T19:49:00.952 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/20' 2022-04-03T19:49:00.953 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/12' 2022-04-03T19:49:00.953 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/39' 2022-04-03T19:49:00.953 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/10' 2022-04-03T19:49:00.953 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/34' 2022-04-03T19:49:00.953 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/38' 2022-04-03T19:49:00.954 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/11' 2022-04-03T19:49:00.954 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/16' 2022-04-03T19:49:00.954 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/32' 2022-04-03T19:49:00.954 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/30' 2022-04-03T19:49:00.955 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/14' 2022-04-03T19:49:00.955 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/18' 2022-04-03T19:49:00.955 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/13' 2022-04-03T19:49:00.955 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/33' 2022-04-03T19:49:00.955 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/17' 2022-04-03T19:49:00.956 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/35' 2022-04-03T19:49:00.956 INFO:tasks.workunit.client.0.smithi077.stdout:removed './171/37' 2022-04-03T19:49:00.956 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './171' 2022-04-03T19:49:00.956 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/19' 2022-04-03T19:49:00.956 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/15' 2022-04-03T19:49:00.957 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/31' 2022-04-03T19:49:00.957 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/36' 2022-04-03T19:49:00.957 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/20' 2022-04-03T19:49:00.957 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/12' 2022-04-03T19:49:00.958 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/39' 2022-04-03T19:49:00.958 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/10' 2022-04-03T19:49:00.958 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/34' 2022-04-03T19:49:00.958 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/38' 2022-04-03T19:49:00.958 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/11' 2022-04-03T19:49:00.959 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/16' 2022-04-03T19:49:00.959 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/32' 2022-04-03T19:49:00.959 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/30' 2022-04-03T19:49:00.959 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/14' 2022-04-03T19:49:00.959 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/18' 2022-04-03T19:49:00.960 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/13' 2022-04-03T19:49:00.960 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/33' 2022-04-03T19:49:00.960 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/17' 2022-04-03T19:49:00.960 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/35' 2022-04-03T19:49:00.961 INFO:tasks.workunit.client.0.smithi077.stdout:removed './172/37' 2022-04-03T19:49:00.961 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './172' 2022-04-03T19:49:00.961 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/19' 2022-04-03T19:49:00.961 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/15' 2022-04-03T19:49:00.961 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/31' 2022-04-03T19:49:00.962 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/36' 2022-04-03T19:49:00.962 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/20' 2022-04-03T19:49:00.962 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/12' 2022-04-03T19:49:00.962 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/39' 2022-04-03T19:49:00.962 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/10' 2022-04-03T19:49:00.963 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/34' 2022-04-03T19:49:00.963 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/38' 2022-04-03T19:49:00.963 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/11' 2022-04-03T19:49:00.963 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/16' 2022-04-03T19:49:00.964 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/32' 2022-04-03T19:49:00.964 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/30' 2022-04-03T19:49:00.964 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/14' 2022-04-03T19:49:00.964 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/18' 2022-04-03T19:49:00.964 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/13' 2022-04-03T19:49:00.965 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/33' 2022-04-03T19:49:00.965 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/17' 2022-04-03T19:49:00.965 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/35' 2022-04-03T19:49:00.965 INFO:tasks.workunit.client.0.smithi077.stdout:removed './173/37' 2022-04-03T19:49:00.965 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './173' 2022-04-03T19:49:00.966 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/19' 2022-04-03T19:49:00.966 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/15' 2022-04-03T19:49:00.966 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/31' 2022-04-03T19:49:00.966 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/36' 2022-04-03T19:49:00.967 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/20' 2022-04-03T19:49:00.967 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/12' 2022-04-03T19:49:00.967 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/39' 2022-04-03T19:49:00.967 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/10' 2022-04-03T19:49:00.967 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/34' 2022-04-03T19:49:00.968 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/38' 2022-04-03T19:49:00.968 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/11' 2022-04-03T19:49:00.968 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/16' 2022-04-03T19:49:00.968 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/32' 2022-04-03T19:49:00.969 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/30' 2022-04-03T19:49:00.969 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/14' 2022-04-03T19:49:00.969 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/18' 2022-04-03T19:49:00.969 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/13' 2022-04-03T19:49:00.969 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/33' 2022-04-03T19:49:00.970 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/17' 2022-04-03T19:49:00.970 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/35' 2022-04-03T19:49:00.970 INFO:tasks.workunit.client.0.smithi077.stdout:removed './174/37' 2022-04-03T19:49:00.970 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './174' 2022-04-03T19:49:00.971 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/19' 2022-04-03T19:49:00.971 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/15' 2022-04-03T19:49:00.971 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/31' 2022-04-03T19:49:00.971 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/36' 2022-04-03T19:49:00.971 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/20' 2022-04-03T19:49:00.972 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/12' 2022-04-03T19:49:00.972 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/39' 2022-04-03T19:49:00.972 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/10' 2022-04-03T19:49:00.972 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/34' 2022-04-03T19:49:00.972 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/38' 2022-04-03T19:49:00.973 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/11' 2022-04-03T19:49:00.973 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/16' 2022-04-03T19:49:00.973 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/32' 2022-04-03T19:49:00.973 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/30' 2022-04-03T19:49:00.974 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/14' 2022-04-03T19:49:00.974 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/18' 2022-04-03T19:49:00.974 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/13' 2022-04-03T19:49:00.974 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/33' 2022-04-03T19:49:00.974 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/17' 2022-04-03T19:49:00.975 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/35' 2022-04-03T19:49:00.975 INFO:tasks.workunit.client.0.smithi077.stdout:removed './175/37' 2022-04-03T19:49:00.975 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './175' 2022-04-03T19:49:00.975 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/19' 2022-04-03T19:49:00.976 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/15' 2022-04-03T19:49:00.976 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/31' 2022-04-03T19:49:00.976 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/36' 2022-04-03T19:49:00.976 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/20' 2022-04-03T19:49:00.976 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/12' 2022-04-03T19:49:00.977 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/39' 2022-04-03T19:49:00.977 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/10' 2022-04-03T19:49:00.977 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/34' 2022-04-03T19:49:00.977 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/38' 2022-04-03T19:49:00.977 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/11' 2022-04-03T19:49:00.978 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/16' 2022-04-03T19:49:00.978 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/32' 2022-04-03T19:49:00.978 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/30' 2022-04-03T19:49:00.978 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/14' 2022-04-03T19:49:00.979 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/18' 2022-04-03T19:49:00.979 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/13' 2022-04-03T19:49:00.979 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/33' 2022-04-03T19:49:00.979 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/17' 2022-04-03T19:49:00.979 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/35' 2022-04-03T19:49:00.980 INFO:tasks.workunit.client.0.smithi077.stdout:removed './176/37' 2022-04-03T19:49:00.980 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './176' 2022-04-03T19:49:00.980 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/19' 2022-04-03T19:49:00.980 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/15' 2022-04-03T19:49:03.606 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/31' 2022-04-03T19:49:03.606 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/36' 2022-04-03T19:49:03.606 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/20' 2022-04-03T19:49:03.607 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/12' 2022-04-03T19:49:03.607 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/39' 2022-04-03T19:49:03.607 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/10' 2022-04-03T19:49:03.607 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/34' 2022-04-03T19:49:03.608 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/38' 2022-04-03T19:49:03.608 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/11' 2022-04-03T19:49:03.608 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/16' 2022-04-03T19:49:03.608 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/32' 2022-04-03T19:49:03.608 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/30' 2022-04-03T19:49:03.609 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/14' 2022-04-03T19:49:03.609 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/18' 2022-04-03T19:49:03.609 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/13' 2022-04-03T19:49:03.609 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/33' 2022-04-03T19:49:03.610 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/17' 2022-04-03T19:49:03.610 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/35' 2022-04-03T19:49:03.610 INFO:tasks.workunit.client.0.smithi077.stdout:removed './177/37' 2022-04-03T19:49:03.610 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './177' 2022-04-03T19:49:03.611 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/19' 2022-04-03T19:49:03.611 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/15' 2022-04-03T19:49:03.611 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/31' 2022-04-03T19:49:03.611 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/36' 2022-04-03T19:49:03.612 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/20' 2022-04-03T19:49:03.612 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/12' 2022-04-03T19:49:03.612 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/39' 2022-04-03T19:49:03.612 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/10' 2022-04-03T19:49:03.613 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/34' 2022-04-03T19:49:03.613 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/38' 2022-04-03T19:49:03.613 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/11' 2022-04-03T19:49:03.613 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/16' 2022-04-03T19:49:03.614 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/32' 2022-04-03T19:49:03.614 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/30' 2022-04-03T19:49:03.614 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/14' 2022-04-03T19:49:03.614 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/18' 2022-04-03T19:49:03.615 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/13' 2022-04-03T19:49:03.615 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/33' 2022-04-03T19:49:03.615 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/17' 2022-04-03T19:49:03.615 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/35' 2022-04-03T19:49:03.616 INFO:tasks.workunit.client.0.smithi077.stdout:removed './178/37' 2022-04-03T19:49:03.616 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './178' 2022-04-03T19:49:03.616 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/19' 2022-04-03T19:49:03.616 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/15' 2022-04-03T19:49:03.617 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/31' 2022-04-03T19:49:03.617 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/36' 2022-04-03T19:49:03.617 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/20' 2022-04-03T19:49:03.617 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/12' 2022-04-03T19:49:03.618 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/39' 2022-04-03T19:49:03.618 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/10' 2022-04-03T19:49:03.618 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/34' 2022-04-03T19:49:03.618 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/38' 2022-04-03T19:49:03.619 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/11' 2022-04-03T19:49:03.619 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/16' 2022-04-03T19:49:03.619 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/32' 2022-04-03T19:49:03.619 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/30' 2022-04-03T19:49:03.619 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/14' 2022-04-03T19:49:03.620 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/18' 2022-04-03T19:49:03.620 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/13' 2022-04-03T19:49:03.620 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/33' 2022-04-03T19:49:03.620 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/17' 2022-04-03T19:49:03.621 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/35' 2022-04-03T19:49:03.621 INFO:tasks.workunit.client.0.smithi077.stdout:removed './179/37' 2022-04-03T19:49:03.621 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './179' 2022-04-03T19:49:03.621 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/19' 2022-04-03T19:49:03.621 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/15' 2022-04-03T19:49:03.622 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/31' 2022-04-03T19:49:03.622 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/36' 2022-04-03T19:49:03.622 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/20' 2022-04-03T19:49:03.622 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/12' 2022-04-03T19:49:03.622 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/39' 2022-04-03T19:49:03.623 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/10' 2022-04-03T19:49:03.623 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/34' 2022-04-03T19:49:03.623 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/38' 2022-04-03T19:49:03.623 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/11' 2022-04-03T19:49:03.624 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/16' 2022-04-03T19:49:03.624 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/32' 2022-04-03T19:49:03.624 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/30' 2022-04-03T19:49:03.624 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/14' 2022-04-03T19:49:03.624 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/18' 2022-04-03T19:49:03.625 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/13' 2022-04-03T19:49:03.625 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/33' 2022-04-03T19:49:03.625 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/17' 2022-04-03T19:49:03.625 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/35' 2022-04-03T19:49:03.625 INFO:tasks.workunit.client.0.smithi077.stdout:removed './180/37' 2022-04-03T19:49:03.626 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './180' 2022-04-03T19:49:03.626 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/19' 2022-04-03T19:49:03.626 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/15' 2022-04-03T19:49:03.626 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/31' 2022-04-03T19:49:03.627 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/36' 2022-04-03T19:49:03.627 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/20' 2022-04-03T19:49:03.627 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/12' 2022-04-03T19:49:03.627 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/39' 2022-04-03T19:49:03.628 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/10' 2022-04-03T19:49:03.628 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/34' 2022-04-03T19:49:03.628 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/38' 2022-04-03T19:49:03.629 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/11' 2022-04-03T19:49:03.629 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/16' 2022-04-03T19:49:03.629 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/32' 2022-04-03T19:49:03.629 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/30' 2022-04-03T19:49:03.629 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/14' 2022-04-03T19:49:03.630 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/18' 2022-04-03T19:49:03.630 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/13' 2022-04-03T19:49:03.630 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/33' 2022-04-03T19:49:03.631 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/17' 2022-04-03T19:49:03.631 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/35' 2022-04-03T19:49:03.631 INFO:tasks.workunit.client.0.smithi077.stdout:removed './181/37' 2022-04-03T19:49:03.631 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './181' 2022-04-03T19:49:03.632 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/19' 2022-04-03T19:49:03.632 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/15' 2022-04-03T19:49:03.632 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/31' 2022-04-03T19:49:03.632 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/36' 2022-04-03T19:49:03.633 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/20' 2022-04-03T19:49:03.633 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/12' 2022-04-03T19:49:03.633 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/39' 2022-04-03T19:49:03.633 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/10' 2022-04-03T19:49:03.633 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/34' 2022-04-03T19:49:03.634 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/38' 2022-04-03T19:49:03.634 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/11' 2022-04-03T19:49:03.634 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/16' 2022-04-03T19:49:03.634 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/32' 2022-04-03T19:49:03.635 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/30' 2022-04-03T19:49:03.635 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/14' 2022-04-03T19:49:03.635 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/18' 2022-04-03T19:49:03.635 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/13' 2022-04-03T19:49:03.635 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/33' 2022-04-03T19:49:03.636 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/17' 2022-04-03T19:49:03.636 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/35' 2022-04-03T19:49:03.636 INFO:tasks.workunit.client.0.smithi077.stdout:removed './182/37' 2022-04-03T19:49:03.636 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './182' 2022-04-03T19:49:03.637 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/19' 2022-04-03T19:49:03.637 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/15' 2022-04-03T19:49:03.637 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/31' 2022-04-03T19:49:03.637 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/36' 2022-04-03T19:49:03.637 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/20' 2022-04-03T19:49:03.638 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/12' 2022-04-03T19:49:03.638 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/39' 2022-04-03T19:49:03.638 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/10' 2022-04-03T19:49:03.638 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/34' 2022-04-03T19:49:03.639 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/38' 2022-04-03T19:49:03.639 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/11' 2022-04-03T19:49:03.639 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/16' 2022-04-03T19:49:03.639 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/32' 2022-04-03T19:49:03.639 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/30' 2022-04-03T19:49:03.640 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/14' 2022-04-03T19:49:03.640 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/18' 2022-04-03T19:49:03.640 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/13' 2022-04-03T19:49:03.640 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/33' 2022-04-03T19:49:03.640 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/17' 2022-04-03T19:49:03.641 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/35' 2022-04-03T19:49:03.641 INFO:tasks.workunit.client.0.smithi077.stdout:removed './183/37' 2022-04-03T19:49:03.641 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './183' 2022-04-03T19:49:03.641 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/19' 2022-04-03T19:49:03.642 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/15' 2022-04-03T19:49:03.642 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/31' 2022-04-03T19:49:03.642 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/36' 2022-04-03T19:49:03.642 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/20' 2022-04-03T19:49:03.642 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/12' 2022-04-03T19:49:03.643 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/39' 2022-04-03T19:49:03.643 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/10' 2022-04-03T19:49:03.643 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/34' 2022-04-03T19:49:03.643 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/38' 2022-04-03T19:49:03.644 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/11' 2022-04-03T19:49:03.644 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/16' 2022-04-03T19:49:03.644 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/32' 2022-04-03T19:49:03.644 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/30' 2022-04-03T19:49:03.644 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/14' 2022-04-03T19:49:03.645 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/18' 2022-04-03T19:49:03.645 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/13' 2022-04-03T19:49:03.645 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/33' 2022-04-03T19:49:03.645 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/17' 2022-04-03T19:49:03.646 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/35' 2022-04-03T19:49:03.646 INFO:tasks.workunit.client.0.smithi077.stdout:removed './184/37' 2022-04-03T19:49:03.646 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './184' 2022-04-03T19:49:03.646 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/19' 2022-04-03T19:49:03.646 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/15' 2022-04-03T19:49:03.647 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/31' 2022-04-03T19:49:03.647 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/36' 2022-04-03T19:49:03.647 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/20' 2022-04-03T19:49:03.647 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/12' 2022-04-03T19:49:03.648 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/39' 2022-04-03T19:49:03.648 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/10' 2022-04-03T19:49:03.648 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/34' 2022-04-03T19:49:03.648 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/38' 2022-04-03T19:49:03.648 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/11' 2022-04-03T19:49:03.649 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/16' 2022-04-03T19:49:03.649 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/32' 2022-04-03T19:49:03.649 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/30' 2022-04-03T19:49:03.649 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/14' 2022-04-03T19:49:03.650 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/18' 2022-04-03T19:49:03.650 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/13' 2022-04-03T19:49:03.650 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/33' 2022-04-03T19:49:03.650 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/17' 2022-04-03T19:49:03.650 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/35' 2022-04-03T19:49:03.651 INFO:tasks.workunit.client.0.smithi077.stdout:removed './185/37' 2022-04-03T19:49:03.651 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './185' 2022-04-03T19:49:03.651 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/19' 2022-04-03T19:49:03.651 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/15' 2022-04-03T19:49:03.652 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/31' 2022-04-03T19:49:03.652 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/36' 2022-04-03T19:49:03.652 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/20' 2022-04-03T19:49:03.652 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/12' 2022-04-03T19:49:03.652 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/39' 2022-04-03T19:49:03.653 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/10' 2022-04-03T19:49:03.653 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/34' 2022-04-03T19:49:03.653 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/38' 2022-04-03T19:49:03.653 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/11' 2022-04-03T19:49:03.653 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/16' 2022-04-03T19:49:03.654 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/32' 2022-04-03T19:49:03.654 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/30' 2022-04-03T19:49:03.654 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/14' 2022-04-03T19:49:03.654 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/18' 2022-04-03T19:49:05.184 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/13' 2022-04-03T19:49:05.184 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/33' 2022-04-03T19:49:05.185 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/17' 2022-04-03T19:49:05.185 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/35' 2022-04-03T19:49:05.185 INFO:tasks.workunit.client.0.smithi077.stdout:removed './186/37' 2022-04-03T19:49:05.185 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './186' 2022-04-03T19:49:05.186 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/19' 2022-04-03T19:49:05.186 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/15' 2022-04-03T19:49:05.186 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/31' 2022-04-03T19:49:05.186 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/36' 2022-04-03T19:49:05.187 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/20' 2022-04-03T19:49:05.187 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/12' 2022-04-03T19:49:05.187 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/39' 2022-04-03T19:49:05.187 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/10' 2022-04-03T19:49:05.188 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/34' 2022-04-03T19:49:05.188 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/38' 2022-04-03T19:49:05.188 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/11' 2022-04-03T19:49:05.189 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/16' 2022-04-03T19:49:05.189 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/32' 2022-04-03T19:49:05.189 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/30' 2022-04-03T19:49:05.189 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/14' 2022-04-03T19:49:05.190 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/18' 2022-04-03T19:49:05.190 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/13' 2022-04-03T19:49:05.190 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/33' 2022-04-03T19:49:05.190 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/17' 2022-04-03T19:49:05.191 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/35' 2022-04-03T19:49:05.191 INFO:tasks.workunit.client.0.smithi077.stdout:removed './187/37' 2022-04-03T19:49:05.191 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './187' 2022-04-03T19:49:05.191 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/19' 2022-04-03T19:49:05.192 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/15' 2022-04-03T19:49:05.192 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/31' 2022-04-03T19:49:05.192 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/36' 2022-04-03T19:49:05.193 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/20' 2022-04-03T19:49:05.193 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/12' 2022-04-03T19:49:05.193 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/39' 2022-04-03T19:49:05.193 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/10' 2022-04-03T19:49:05.194 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/34' 2022-04-03T19:49:05.194 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/38' 2022-04-03T19:49:05.194 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/11' 2022-04-03T19:49:05.195 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/16' 2022-04-03T19:49:05.195 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/32' 2022-04-03T19:49:05.195 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/30' 2022-04-03T19:49:05.196 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/14' 2022-04-03T19:49:05.196 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/18' 2022-04-03T19:49:05.196 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/13' 2022-04-03T19:49:05.196 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/33' 2022-04-03T19:49:05.197 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/17' 2022-04-03T19:49:05.197 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/35' 2022-04-03T19:49:05.197 INFO:tasks.workunit.client.0.smithi077.stdout:removed './188/37' 2022-04-03T19:49:05.197 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './188' 2022-04-03T19:49:05.198 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/19' 2022-04-03T19:49:05.198 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/15' 2022-04-03T19:49:05.198 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/31' 2022-04-03T19:49:05.198 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/36' 2022-04-03T19:49:05.199 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/20' 2022-04-03T19:49:05.199 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/12' 2022-04-03T19:49:05.199 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/39' 2022-04-03T19:49:05.199 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/10' 2022-04-03T19:49:05.200 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/34' 2022-04-03T19:49:05.200 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/38' 2022-04-03T19:49:05.200 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/11' 2022-04-03T19:49:05.201 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/16' 2022-04-03T19:49:05.201 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/32' 2022-04-03T19:49:05.201 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/30' 2022-04-03T19:49:05.201 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/14' 2022-04-03T19:49:05.202 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/18' 2022-04-03T19:49:05.202 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/13' 2022-04-03T19:49:05.202 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/33' 2022-04-03T19:49:05.202 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/17' 2022-04-03T19:49:05.203 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/35' 2022-04-03T19:49:05.203 INFO:tasks.workunit.client.0.smithi077.stdout:removed './189/37' 2022-04-03T19:49:05.203 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './189' 2022-04-03T19:49:05.203 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/19' 2022-04-03T19:49:05.204 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/15' 2022-04-03T19:49:05.204 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/31' 2022-04-03T19:49:05.204 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/36' 2022-04-03T19:49:05.204 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/20' 2022-04-03T19:49:05.205 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/12' 2022-04-03T19:49:05.205 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/39' 2022-04-03T19:49:05.205 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/10' 2022-04-03T19:49:05.205 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/34' 2022-04-03T19:49:05.206 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/38' 2022-04-03T19:49:05.206 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/11' 2022-04-03T19:49:05.206 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/16' 2022-04-03T19:49:05.206 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/32' 2022-04-03T19:49:05.207 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/30' 2022-04-03T19:49:05.207 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/14' 2022-04-03T19:49:05.207 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/18' 2022-04-03T19:49:05.207 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/13' 2022-04-03T19:49:05.207 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/33' 2022-04-03T19:49:05.208 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/17' 2022-04-03T19:49:05.208 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/35' 2022-04-03T19:49:05.208 INFO:tasks.workunit.client.0.smithi077.stdout:removed './190/37' 2022-04-03T19:49:05.208 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './190' 2022-04-03T19:49:05.209 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/19' 2022-04-03T19:49:05.209 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/15' 2022-04-03T19:49:05.209 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/31' 2022-04-03T19:49:05.209 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/36' 2022-04-03T19:49:05.210 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/20' 2022-04-03T19:49:05.210 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/12' 2022-04-03T19:49:05.210 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/39' 2022-04-03T19:49:05.210 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/10' 2022-04-03T19:49:05.211 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/34' 2022-04-03T19:49:05.211 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/38' 2022-04-03T19:49:05.211 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/11' 2022-04-03T19:49:05.211 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/16' 2022-04-03T19:49:05.211 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/32' 2022-04-03T19:49:05.212 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/30' 2022-04-03T19:49:05.212 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/14' 2022-04-03T19:49:05.212 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/18' 2022-04-03T19:49:05.212 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/13' 2022-04-03T19:49:05.213 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/33' 2022-04-03T19:49:05.213 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/17' 2022-04-03T19:49:05.213 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/35' 2022-04-03T19:49:05.213 INFO:tasks.workunit.client.0.smithi077.stdout:removed './191/37' 2022-04-03T19:49:05.214 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './191' 2022-04-03T19:49:05.214 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/19' 2022-04-03T19:49:05.214 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/15' 2022-04-03T19:49:05.214 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/31' 2022-04-03T19:49:05.214 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/36' 2022-04-03T19:49:05.215 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/20' 2022-04-03T19:49:05.215 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/12' 2022-04-03T19:49:05.215 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/39' 2022-04-03T19:49:05.215 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/10' 2022-04-03T19:49:05.216 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/34' 2022-04-03T19:49:05.216 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/38' 2022-04-03T19:49:05.216 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/11' 2022-04-03T19:49:05.216 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/16' 2022-04-03T19:49:05.217 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/32' 2022-04-03T19:49:05.217 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/30' 2022-04-03T19:49:05.217 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/14' 2022-04-03T19:49:05.217 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/18' 2022-04-03T19:49:05.217 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/13' 2022-04-03T19:49:05.218 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/33' 2022-04-03T19:49:05.218 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/17' 2022-04-03T19:49:05.218 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/35' 2022-04-03T19:49:05.218 INFO:tasks.workunit.client.0.smithi077.stdout:removed './192/37' 2022-04-03T19:49:05.219 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './192' 2022-04-03T19:49:05.219 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/19' 2022-04-03T19:49:05.219 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/15' 2022-04-03T19:49:05.219 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/31' 2022-04-03T19:49:05.220 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/36' 2022-04-03T19:49:05.220 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/20' 2022-04-03T19:49:05.220 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/12' 2022-04-03T19:49:05.220 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/39' 2022-04-03T19:49:05.221 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/10' 2022-04-03T19:49:05.221 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/34' 2022-04-03T19:49:05.221 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/38' 2022-04-03T19:49:05.221 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/11' 2022-04-03T19:49:05.222 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/16' 2022-04-03T19:49:05.222 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/32' 2022-04-03T19:49:05.222 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/30' 2022-04-03T19:49:05.222 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/14' 2022-04-03T19:49:05.223 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/18' 2022-04-03T19:49:05.223 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/13' 2022-04-03T19:49:05.223 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/33' 2022-04-03T19:49:05.223 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/17' 2022-04-03T19:49:05.224 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/35' 2022-04-03T19:49:05.224 INFO:tasks.workunit.client.0.smithi077.stdout:removed './193/37' 2022-04-03T19:49:05.224 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './193' 2022-04-03T19:49:05.224 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/19' 2022-04-03T19:49:05.224 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/15' 2022-04-03T19:49:05.225 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/31' 2022-04-03T19:49:05.225 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/36' 2022-04-03T19:49:05.225 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/20' 2022-04-03T19:49:05.226 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/12' 2022-04-03T19:49:05.226 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/39' 2022-04-03T19:49:05.226 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/10' 2022-04-03T19:49:05.226 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/34' 2022-04-03T19:49:05.227 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/38' 2022-04-03T19:49:05.227 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/11' 2022-04-03T19:49:05.227 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/16' 2022-04-03T19:49:05.227 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/32' 2022-04-03T19:49:05.227 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/30' 2022-04-03T19:49:05.228 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/14' 2022-04-03T19:49:05.228 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/18' 2022-04-03T19:49:05.228 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/13' 2022-04-03T19:49:05.228 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/33' 2022-04-03T19:49:05.229 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/17' 2022-04-03T19:49:05.229 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/35' 2022-04-03T19:49:05.229 INFO:tasks.workunit.client.0.smithi077.stdout:removed './194/37' 2022-04-03T19:49:05.229 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './194' 2022-04-03T19:49:05.230 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/19' 2022-04-03T19:49:05.230 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/15' 2022-04-03T19:49:05.230 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/31' 2022-04-03T19:49:05.230 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/36' 2022-04-03T19:49:05.231 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/20' 2022-04-03T19:49:05.231 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/12' 2022-04-03T19:49:05.231 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/39' 2022-04-03T19:49:05.231 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/10' 2022-04-03T19:49:05.231 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/34' 2022-04-03T19:49:05.232 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/38' 2022-04-03T19:49:05.232 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/11' 2022-04-03T19:49:05.232 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/16' 2022-04-03T19:49:05.232 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/32' 2022-04-03T19:49:05.233 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/30' 2022-04-03T19:49:05.233 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/14' 2022-04-03T19:49:05.233 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/18' 2022-04-03T19:49:05.233 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/13' 2022-04-03T19:49:05.234 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/33' 2022-04-03T19:49:05.234 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/17' 2022-04-03T19:49:05.234 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/35' 2022-04-03T19:49:05.234 INFO:tasks.workunit.client.0.smithi077.stdout:removed './195/37' 2022-04-03T19:49:05.234 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './195' 2022-04-03T19:49:05.235 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/19' 2022-04-03T19:49:05.235 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/15' 2022-04-03T19:49:05.235 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/31' 2022-04-03T19:49:05.235 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/36' 2022-04-03T19:49:05.236 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/20' 2022-04-03T19:49:05.236 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/12' 2022-04-03T19:49:05.236 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/39' 2022-04-03T19:49:05.236 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/10' 2022-04-03T19:49:09.004 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/34' 2022-04-03T19:49:09.005 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/38' 2022-04-03T19:49:09.005 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/11' 2022-04-03T19:49:09.005 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/16' 2022-04-03T19:49:09.006 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/32' 2022-04-03T19:49:09.006 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/30' 2022-04-03T19:49:09.006 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/14' 2022-04-03T19:49:09.006 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/18' 2022-04-03T19:49:09.007 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/13' 2022-04-03T19:49:09.007 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/33' 2022-04-03T19:49:09.007 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/17' 2022-04-03T19:49:09.007 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/35' 2022-04-03T19:49:09.007 INFO:tasks.workunit.client.0.smithi077.stdout:removed './196/37' 2022-04-03T19:49:09.008 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './196' 2022-04-03T19:49:09.008 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/19' 2022-04-03T19:49:09.008 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/15' 2022-04-03T19:49:09.008 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/31' 2022-04-03T19:49:09.009 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/36' 2022-04-03T19:49:09.009 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/20' 2022-04-03T19:49:09.009 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/12' 2022-04-03T19:49:09.009 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/39' 2022-04-03T19:49:09.009 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/10' 2022-04-03T19:49:09.010 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/34' 2022-04-03T19:49:09.010 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/38' 2022-04-03T19:49:09.010 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/11' 2022-04-03T19:49:09.010 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/16' 2022-04-03T19:49:09.011 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/32' 2022-04-03T19:49:09.011 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/30' 2022-04-03T19:49:09.011 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/14' 2022-04-03T19:49:09.011 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/18' 2022-04-03T19:49:09.012 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/13' 2022-04-03T19:49:09.012 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/33' 2022-04-03T19:49:09.012 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/17' 2022-04-03T19:49:09.012 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/35' 2022-04-03T19:49:09.012 INFO:tasks.workunit.client.0.smithi077.stdout:removed './197/37' 2022-04-03T19:49:09.013 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './197' 2022-04-03T19:49:09.013 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/19' 2022-04-03T19:49:09.013 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/15' 2022-04-03T19:49:09.013 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/31' 2022-04-03T19:49:09.014 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/36' 2022-04-03T19:49:09.014 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/20' 2022-04-03T19:49:09.014 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/12' 2022-04-03T19:49:09.014 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/39' 2022-04-03T19:49:09.014 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/10' 2022-04-03T19:49:09.015 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/34' 2022-04-03T19:49:09.015 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/38' 2022-04-03T19:49:09.015 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/11' 2022-04-03T19:49:09.015 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/16' 2022-04-03T19:49:09.016 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/32' 2022-04-03T19:49:09.016 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/30' 2022-04-03T19:49:09.016 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/14' 2022-04-03T19:49:09.016 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/18' 2022-04-03T19:49:09.016 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/13' 2022-04-03T19:49:09.017 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/33' 2022-04-03T19:49:09.017 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/17' 2022-04-03T19:49:09.017 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/35' 2022-04-03T19:49:09.017 INFO:tasks.workunit.client.0.smithi077.stdout:removed './198/37' 2022-04-03T19:49:09.018 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './198' 2022-04-03T19:49:09.018 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/19' 2022-04-03T19:49:09.018 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/15' 2022-04-03T19:49:09.018 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/31' 2022-04-03T19:49:09.018 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/36' 2022-04-03T19:49:09.019 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/20' 2022-04-03T19:49:09.019 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/12' 2022-04-03T19:49:09.019 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/39' 2022-04-03T19:49:09.019 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/10' 2022-04-03T19:49:09.020 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/34' 2022-04-03T19:49:09.020 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/38' 2022-04-03T19:49:09.020 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/11' 2022-04-03T19:49:09.020 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/16' 2022-04-03T19:49:09.020 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/32' 2022-04-03T19:49:09.021 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/30' 2022-04-03T19:49:09.021 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/14' 2022-04-03T19:49:09.021 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/18' 2022-04-03T19:49:09.021 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/13' 2022-04-03T19:49:09.022 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/33' 2022-04-03T19:49:09.022 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/17' 2022-04-03T19:49:09.022 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/35' 2022-04-03T19:49:09.022 INFO:tasks.workunit.client.0.smithi077.stdout:removed './199/37' 2022-04-03T19:49:09.022 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './199' 2022-04-03T19:49:09.023 INFO:tasks.workunit.client.0.smithi077.stdout:removed './200/21' 2022-04-03T19:49:09.023 INFO:tasks.workunit.client.0.smithi077.stdout:removed './200/27' 2022-04-03T19:49:09.023 INFO:tasks.workunit.client.0.smithi077.stdout:removed './200/31' 2022-04-03T19:49:09.023 INFO:tasks.workunit.client.0.smithi077.stdout:removed './200/36' 2022-04-03T19:49:09.023 INFO:tasks.workunit.client.0.smithi077.stdout:removed './200/20' 2022-04-03T19:49:09.024 INFO:tasks.workunit.client.0.smithi077.stdout:removed './200/39' 2022-04-03T19:49:09.024 INFO:tasks.workunit.client.0.smithi077.stdout:removed './200/34' 2022-04-03T19:49:09.024 INFO:tasks.workunit.client.0.smithi077.stdout:removed './200/38' 2022-04-03T19:49:09.024 INFO:tasks.workunit.client.0.smithi077.stdout:removed './200/28' 2022-04-03T19:49:09.025 INFO:tasks.workunit.client.0.smithi077.stdout:removed './200/24' 2022-04-03T19:49:09.025 INFO:tasks.workunit.client.0.smithi077.stdout:removed './200/29' 2022-04-03T19:49:09.025 INFO:tasks.workunit.client.0.smithi077.stdout:removed './200/32' 2022-04-03T19:49:09.025 INFO:tasks.workunit.client.0.smithi077.stdout:removed './200/22' 2022-04-03T19:49:09.025 INFO:tasks.workunit.client.0.smithi077.stdout:removed './200/25' 2022-04-03T19:49:09.026 INFO:tasks.workunit.client.0.smithi077.stdout:removed './200/30' 2022-04-03T19:49:09.026 INFO:tasks.workunit.client.0.smithi077.stdout:removed './200/33' 2022-04-03T19:49:09.026 INFO:tasks.workunit.client.0.smithi077.stdout:removed './200/23' 2022-04-03T19:49:09.026 INFO:tasks.workunit.client.0.smithi077.stdout:removed './200/35' 2022-04-03T19:49:09.027 INFO:tasks.workunit.client.0.smithi077.stdout:removed './200/26' 2022-04-03T19:49:09.027 INFO:tasks.workunit.client.0.smithi077.stdout:removed './200/37' 2022-04-03T19:49:09.027 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './200' 2022-04-03T19:49:09.027 INFO:tasks.workunit.client.0.smithi077.stdout:removed './201/21' 2022-04-03T19:49:09.027 INFO:tasks.workunit.client.0.smithi077.stdout:removed './201/27' 2022-04-03T19:49:09.028 INFO:tasks.workunit.client.0.smithi077.stdout:removed './201/31' 2022-04-03T19:49:09.028 INFO:tasks.workunit.client.0.smithi077.stdout:removed './201/36' 2022-04-03T19:49:09.028 INFO:tasks.workunit.client.0.smithi077.stdout:removed './201/20' 2022-04-03T19:49:09.028 INFO:tasks.workunit.client.0.smithi077.stdout:removed './201/39' 2022-04-03T19:49:09.029 INFO:tasks.workunit.client.0.smithi077.stdout:removed './201/34' 2022-04-03T19:49:09.029 INFO:tasks.workunit.client.0.smithi077.stdout:removed './201/38' 2022-04-03T19:49:09.029 INFO:tasks.workunit.client.0.smithi077.stdout:removed './201/28' 2022-04-03T19:49:09.029 INFO:tasks.workunit.client.0.smithi077.stdout:removed './201/24' 2022-04-03T19:49:09.029 INFO:tasks.workunit.client.0.smithi077.stdout:removed './201/29' 2022-04-03T19:49:09.030 INFO:tasks.workunit.client.0.smithi077.stdout:removed './201/32' 2022-04-03T19:49:09.030 INFO:tasks.workunit.client.0.smithi077.stdout:removed './201/22' 2022-04-03T19:49:09.030 INFO:tasks.workunit.client.0.smithi077.stdout:removed './201/25' 2022-04-03T19:49:09.030 INFO:tasks.workunit.client.0.smithi077.stdout:removed './201/30' 2022-04-03T19:49:09.031 INFO:tasks.workunit.client.0.smithi077.stdout:removed './201/33' 2022-04-03T19:49:09.031 INFO:tasks.workunit.client.0.smithi077.stdout:removed './201/23' 2022-04-03T19:49:09.031 INFO:tasks.workunit.client.0.smithi077.stdout:removed './201/35' 2022-04-03T19:49:09.031 INFO:tasks.workunit.client.0.smithi077.stdout:removed './201/26' 2022-04-03T19:49:09.031 INFO:tasks.workunit.client.0.smithi077.stdout:removed './201/37' 2022-04-03T19:49:09.032 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './201' 2022-04-03T19:49:09.032 INFO:tasks.workunit.client.0.smithi077.stdout:removed './202/21' 2022-04-03T19:49:09.032 INFO:tasks.workunit.client.0.smithi077.stdout:removed './202/27' 2022-04-03T19:49:09.032 INFO:tasks.workunit.client.0.smithi077.stdout:removed './202/31' 2022-04-03T19:49:09.033 INFO:tasks.workunit.client.0.smithi077.stdout:removed './202/36' 2022-04-03T19:49:09.033 INFO:tasks.workunit.client.0.smithi077.stdout:removed './202/20' 2022-04-03T19:49:09.033 INFO:tasks.workunit.client.0.smithi077.stdout:removed './202/39' 2022-04-03T19:49:09.033 INFO:tasks.workunit.client.0.smithi077.stdout:removed './202/34' 2022-04-03T19:49:09.034 INFO:tasks.workunit.client.0.smithi077.stdout:removed './202/38' 2022-04-03T19:49:09.034 INFO:tasks.workunit.client.0.smithi077.stdout:removed './202/28' 2022-04-03T19:49:09.034 INFO:tasks.workunit.client.0.smithi077.stdout:removed './202/24' 2022-04-03T19:49:09.034 INFO:tasks.workunit.client.0.smithi077.stdout:removed './202/29' 2022-04-03T19:49:09.034 INFO:tasks.workunit.client.0.smithi077.stdout:removed './202/32' 2022-04-03T19:49:09.035 INFO:tasks.workunit.client.0.smithi077.stdout:removed './202/22' 2022-04-03T19:49:09.035 INFO:tasks.workunit.client.0.smithi077.stdout:removed './202/25' 2022-04-03T19:49:09.035 INFO:tasks.workunit.client.0.smithi077.stdout:removed './202/30' 2022-04-03T19:49:09.035 INFO:tasks.workunit.client.0.smithi077.stdout:removed './202/33' 2022-04-03T19:49:09.036 INFO:tasks.workunit.client.0.smithi077.stdout:removed './202/23' 2022-04-03T19:49:09.036 INFO:tasks.workunit.client.0.smithi077.stdout:removed './202/35' 2022-04-03T19:49:09.036 INFO:tasks.workunit.client.0.smithi077.stdout:removed './202/26' 2022-04-03T19:49:09.036 INFO:tasks.workunit.client.0.smithi077.stdout:removed './202/37' 2022-04-03T19:49:09.036 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './202' 2022-04-03T19:49:09.037 INFO:tasks.workunit.client.0.smithi077.stdout:removed './203/21' 2022-04-03T19:49:09.037 INFO:tasks.workunit.client.0.smithi077.stdout:removed './203/27' 2022-04-03T19:49:09.037 INFO:tasks.workunit.client.0.smithi077.stdout:removed './203/31' 2022-04-03T19:49:09.037 INFO:tasks.workunit.client.0.smithi077.stdout:removed './203/36' 2022-04-03T19:49:09.037 INFO:tasks.workunit.client.0.smithi077.stdout:removed './203/20' 2022-04-03T19:49:09.038 INFO:tasks.workunit.client.0.smithi077.stdout:removed './203/39' 2022-04-03T19:49:09.038 INFO:tasks.workunit.client.0.smithi077.stdout:removed './203/34' 2022-04-03T19:49:09.038 INFO:tasks.workunit.client.0.smithi077.stdout:removed './203/38' 2022-04-03T19:49:09.038 INFO:tasks.workunit.client.0.smithi077.stdout:removed './203/28' 2022-04-03T19:49:09.039 INFO:tasks.workunit.client.0.smithi077.stdout:removed './203/24' 2022-04-03T19:49:09.039 INFO:tasks.workunit.client.0.smithi077.stdout:removed './203/29' 2022-04-03T19:49:09.039 INFO:tasks.workunit.client.0.smithi077.stdout:removed './203/32' 2022-04-03T19:49:09.039 INFO:tasks.workunit.client.0.smithi077.stdout:removed './203/22' 2022-04-03T19:49:09.039 INFO:tasks.workunit.client.0.smithi077.stdout:removed './203/25' 2022-04-03T19:49:09.040 INFO:tasks.workunit.client.0.smithi077.stdout:removed './203/30' 2022-04-03T19:49:09.040 INFO:tasks.workunit.client.0.smithi077.stdout:removed './203/33' 2022-04-03T19:49:09.040 INFO:tasks.workunit.client.0.smithi077.stdout:removed './203/23' 2022-04-03T19:49:09.040 INFO:tasks.workunit.client.0.smithi077.stdout:removed './203/35' 2022-04-03T19:49:09.041 INFO:tasks.workunit.client.0.smithi077.stdout:removed './203/26' 2022-04-03T19:49:09.041 INFO:tasks.workunit.client.0.smithi077.stdout:removed './203/37' 2022-04-03T19:49:09.041 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './203' 2022-04-03T19:49:09.041 INFO:tasks.workunit.client.0.smithi077.stdout:removed './204/21' 2022-04-03T19:49:09.041 INFO:tasks.workunit.client.0.smithi077.stdout:removed './204/27' 2022-04-03T19:49:09.042 INFO:tasks.workunit.client.0.smithi077.stdout:removed './204/31' 2022-04-03T19:49:09.042 INFO:tasks.workunit.client.0.smithi077.stdout:removed './204/36' 2022-04-03T19:49:09.042 INFO:tasks.workunit.client.0.smithi077.stdout:removed './204/20' 2022-04-03T19:49:09.042 INFO:tasks.workunit.client.0.smithi077.stdout:removed './204/39' 2022-04-03T19:49:09.043 INFO:tasks.workunit.client.0.smithi077.stdout:removed './204/34' 2022-04-03T19:49:09.043 INFO:tasks.workunit.client.0.smithi077.stdout:removed './204/38' 2022-04-03T19:49:09.043 INFO:tasks.workunit.client.0.smithi077.stdout:removed './204/28' 2022-04-03T19:49:09.043 INFO:tasks.workunit.client.0.smithi077.stdout:removed './204/24' 2022-04-03T19:49:09.043 INFO:tasks.workunit.client.0.smithi077.stdout:removed './204/29' 2022-04-03T19:49:09.044 INFO:tasks.workunit.client.0.smithi077.stdout:removed './204/32' 2022-04-03T19:49:09.044 INFO:tasks.workunit.client.0.smithi077.stdout:removed './204/22' 2022-04-03T19:49:09.044 INFO:tasks.workunit.client.0.smithi077.stdout:removed './204/25' 2022-04-03T19:49:09.044 INFO:tasks.workunit.client.0.smithi077.stdout:removed './204/30' 2022-04-03T19:49:09.045 INFO:tasks.workunit.client.0.smithi077.stdout:removed './204/33' 2022-04-03T19:49:09.045 INFO:tasks.workunit.client.0.smithi077.stdout:removed './204/23' 2022-04-03T19:49:09.045 INFO:tasks.workunit.client.0.smithi077.stdout:removed './204/35' 2022-04-03T19:49:09.045 INFO:tasks.workunit.client.0.smithi077.stdout:removed './204/26' 2022-04-03T19:49:09.045 INFO:tasks.workunit.client.0.smithi077.stdout:removed './204/37' 2022-04-03T19:49:09.046 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './204' 2022-04-03T19:49:09.046 INFO:tasks.workunit.client.0.smithi077.stdout:removed './205/21' 2022-04-03T19:49:09.046 INFO:tasks.workunit.client.0.smithi077.stdout:removed './205/27' 2022-04-03T19:49:09.046 INFO:tasks.workunit.client.0.smithi077.stdout:removed './205/31' 2022-04-03T19:49:09.046 INFO:tasks.workunit.client.0.smithi077.stdout:removed './205/36' 2022-04-03T19:49:09.047 INFO:tasks.workunit.client.0.smithi077.stdout:removed './205/20' 2022-04-03T19:49:09.047 INFO:tasks.workunit.client.0.smithi077.stdout:removed './205/39' 2022-04-03T19:49:09.047 INFO:tasks.workunit.client.0.smithi077.stdout:removed './205/34' 2022-04-03T19:49:09.047 INFO:tasks.workunit.client.0.smithi077.stdout:removed './205/38' 2022-04-03T19:49:09.048 INFO:tasks.workunit.client.0.smithi077.stdout:removed './205/28' 2022-04-03T19:49:09.048 INFO:tasks.workunit.client.0.smithi077.stdout:removed './205/24' 2022-04-03T19:49:09.048 INFO:tasks.workunit.client.0.smithi077.stdout:removed './205/29' 2022-04-03T19:49:09.048 INFO:tasks.workunit.client.0.smithi077.stdout:removed './205/32' 2022-04-03T19:49:09.048 INFO:tasks.workunit.client.0.smithi077.stdout:removed './205/22' 2022-04-03T19:49:09.049 INFO:tasks.workunit.client.0.smithi077.stdout:removed './205/25' 2022-04-03T19:49:09.049 INFO:tasks.workunit.client.0.smithi077.stdout:removed './205/30' 2022-04-03T19:49:09.049 INFO:tasks.workunit.client.0.smithi077.stdout:removed './205/33' 2022-04-03T19:49:09.049 INFO:tasks.workunit.client.0.smithi077.stdout:removed './205/23' 2022-04-03T19:49:09.050 INFO:tasks.workunit.client.0.smithi077.stdout:removed './205/35' 2022-04-03T19:49:09.050 INFO:tasks.workunit.client.0.smithi077.stdout:removed './205/26' 2022-04-03T19:49:09.050 INFO:tasks.workunit.client.0.smithi077.stdout:removed './205/37' 2022-04-03T19:49:09.050 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './205' 2022-04-03T19:49:09.050 INFO:tasks.workunit.client.0.smithi077.stdout:removed './206/21' 2022-04-03T19:49:09.051 INFO:tasks.workunit.client.0.smithi077.stdout:removed './206/27' 2022-04-03T19:49:09.051 INFO:tasks.workunit.client.0.smithi077.stdout:removed './206/31' 2022-04-03T19:49:09.051 INFO:tasks.workunit.client.0.smithi077.stdout:removed './206/36' 2022-04-03T19:49:09.051 INFO:tasks.workunit.client.0.smithi077.stdout:removed './206/20' 2022-04-03T19:49:09.052 INFO:tasks.workunit.client.0.smithi077.stdout:removed './206/39' 2022-04-03T19:49:12.211 INFO:tasks.workunit.client.0.smithi077.stdout:removed './206/34' 2022-04-03T19:49:12.212 INFO:tasks.workunit.client.0.smithi077.stdout:removed './206/38' 2022-04-03T19:49:12.212 INFO:tasks.workunit.client.0.smithi077.stdout:removed './206/28' 2022-04-03T19:49:12.212 INFO:tasks.workunit.client.0.smithi077.stdout:removed './206/24' 2022-04-03T19:49:12.212 INFO:tasks.workunit.client.0.smithi077.stdout:removed './206/29' 2022-04-03T19:49:12.213 INFO:tasks.workunit.client.0.smithi077.stdout:removed './206/32' 2022-04-03T19:49:12.213 INFO:tasks.workunit.client.0.smithi077.stdout:removed './206/22' 2022-04-03T19:49:12.213 INFO:tasks.workunit.client.0.smithi077.stdout:removed './206/25' 2022-04-03T19:49:12.213 INFO:tasks.workunit.client.0.smithi077.stdout:removed './206/30' 2022-04-03T19:49:12.214 INFO:tasks.workunit.client.0.smithi077.stdout:removed './206/33' 2022-04-03T19:49:12.214 INFO:tasks.workunit.client.0.smithi077.stdout:removed './206/23' 2022-04-03T19:49:12.214 INFO:tasks.workunit.client.0.smithi077.stdout:removed './206/35' 2022-04-03T19:49:12.214 INFO:tasks.workunit.client.0.smithi077.stdout:removed './206/26' 2022-04-03T19:49:12.214 INFO:tasks.workunit.client.0.smithi077.stdout:removed './206/37' 2022-04-03T19:49:12.215 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './206' 2022-04-03T19:49:12.215 INFO:tasks.workunit.client.0.smithi077.stdout:removed './207/21' 2022-04-03T19:49:12.215 INFO:tasks.workunit.client.0.smithi077.stdout:removed './207/27' 2022-04-03T19:49:12.215 INFO:tasks.workunit.client.0.smithi077.stdout:removed './207/31' 2022-04-03T19:49:12.216 INFO:tasks.workunit.client.0.smithi077.stdout:removed './207/36' 2022-04-03T19:49:12.216 INFO:tasks.workunit.client.0.smithi077.stdout:removed './207/20' 2022-04-03T19:49:12.216 INFO:tasks.workunit.client.0.smithi077.stdout:removed './207/39' 2022-04-03T19:49:12.216 INFO:tasks.workunit.client.0.smithi077.stdout:removed './207/34' 2022-04-03T19:49:12.216 INFO:tasks.workunit.client.0.smithi077.stdout:removed './207/38' 2022-04-03T19:49:12.217 INFO:tasks.workunit.client.0.smithi077.stdout:removed './207/28' 2022-04-03T19:49:12.217 INFO:tasks.workunit.client.0.smithi077.stdout:removed './207/24' 2022-04-03T19:49:12.217 INFO:tasks.workunit.client.0.smithi077.stdout:removed './207/29' 2022-04-03T19:49:12.217 INFO:tasks.workunit.client.0.smithi077.stdout:removed './207/32' 2022-04-03T19:49:12.218 INFO:tasks.workunit.client.0.smithi077.stdout:removed './207/22' 2022-04-03T19:49:12.218 INFO:tasks.workunit.client.0.smithi077.stdout:removed './207/25' 2022-04-03T19:49:12.218 INFO:tasks.workunit.client.0.smithi077.stdout:removed './207/30' 2022-04-03T19:49:12.218 INFO:tasks.workunit.client.0.smithi077.stdout:removed './207/33' 2022-04-03T19:49:12.219 INFO:tasks.workunit.client.0.smithi077.stdout:removed './207/23' 2022-04-03T19:49:12.219 INFO:tasks.workunit.client.0.smithi077.stdout:removed './207/35' 2022-04-03T19:49:12.219 INFO:tasks.workunit.client.0.smithi077.stdout:removed './207/26' 2022-04-03T19:49:12.219 INFO:tasks.workunit.client.0.smithi077.stdout:removed './207/37' 2022-04-03T19:49:12.219 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './207' 2022-04-03T19:49:12.220 INFO:tasks.workunit.client.0.smithi077.stdout:removed './208/21' 2022-04-03T19:49:12.220 INFO:tasks.workunit.client.0.smithi077.stdout:removed './208/27' 2022-04-03T19:49:12.220 INFO:tasks.workunit.client.0.smithi077.stdout:removed './208/31' 2022-04-03T19:49:12.220 INFO:tasks.workunit.client.0.smithi077.stdout:removed './208/36' 2022-04-03T19:49:12.221 INFO:tasks.workunit.client.0.smithi077.stdout:removed './208/20' 2022-04-03T19:49:12.221 INFO:tasks.workunit.client.0.smithi077.stdout:removed './208/39' 2022-04-03T19:49:12.221 INFO:tasks.workunit.client.0.smithi077.stdout:removed './208/34' 2022-04-03T19:49:12.221 INFO:tasks.workunit.client.0.smithi077.stdout:removed './208/38' 2022-04-03T19:49:12.222 INFO:tasks.workunit.client.0.smithi077.stdout:removed './208/28' 2022-04-03T19:49:12.222 INFO:tasks.workunit.client.0.smithi077.stdout:removed './208/24' 2022-04-03T19:49:12.222 INFO:tasks.workunit.client.0.smithi077.stdout:removed './208/29' 2022-04-03T19:49:12.222 INFO:tasks.workunit.client.0.smithi077.stdout:removed './208/32' 2022-04-03T19:49:12.222 INFO:tasks.workunit.client.0.smithi077.stdout:removed './208/22' 2022-04-03T19:49:12.223 INFO:tasks.workunit.client.0.smithi077.stdout:removed './208/25' 2022-04-03T19:49:12.223 INFO:tasks.workunit.client.0.smithi077.stdout:removed './208/30' 2022-04-03T19:49:12.223 INFO:tasks.workunit.client.0.smithi077.stdout:removed './208/33' 2022-04-03T19:49:12.223 INFO:tasks.workunit.client.0.smithi077.stdout:removed './208/23' 2022-04-03T19:49:12.224 INFO:tasks.workunit.client.0.smithi077.stdout:removed './208/35' 2022-04-03T19:49:12.224 INFO:tasks.workunit.client.0.smithi077.stdout:removed './208/26' 2022-04-03T19:49:12.224 INFO:tasks.workunit.client.0.smithi077.stdout:removed './208/37' 2022-04-03T19:49:12.224 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './208' 2022-04-03T19:49:12.224 INFO:tasks.workunit.client.0.smithi077.stdout:removed './209/21' 2022-04-03T19:49:12.225 INFO:tasks.workunit.client.0.smithi077.stdout:removed './209/27' 2022-04-03T19:49:12.225 INFO:tasks.workunit.client.0.smithi077.stdout:removed './209/31' 2022-04-03T19:49:12.225 INFO:tasks.workunit.client.0.smithi077.stdout:removed './209/36' 2022-04-03T19:49:12.225 INFO:tasks.workunit.client.0.smithi077.stdout:removed './209/20' 2022-04-03T19:49:12.226 INFO:tasks.workunit.client.0.smithi077.stdout:removed './209/39' 2022-04-03T19:49:12.226 INFO:tasks.workunit.client.0.smithi077.stdout:removed './209/34' 2022-04-03T19:49:12.226 INFO:tasks.workunit.client.0.smithi077.stdout:removed './209/38' 2022-04-03T19:49:12.226 INFO:tasks.workunit.client.0.smithi077.stdout:removed './209/28' 2022-04-03T19:49:12.226 INFO:tasks.workunit.client.0.smithi077.stdout:removed './209/24' 2022-04-03T19:49:12.227 INFO:tasks.workunit.client.0.smithi077.stdout:removed './209/29' 2022-04-03T19:49:12.227 INFO:tasks.workunit.client.0.smithi077.stdout:removed './209/32' 2022-04-03T19:49:12.227 INFO:tasks.workunit.client.0.smithi077.stdout:removed './209/22' 2022-04-03T19:49:12.227 INFO:tasks.workunit.client.0.smithi077.stdout:removed './209/25' 2022-04-03T19:49:12.228 INFO:tasks.workunit.client.0.smithi077.stdout:removed './209/30' 2022-04-03T19:49:12.228 INFO:tasks.workunit.client.0.smithi077.stdout:removed './209/33' 2022-04-03T19:49:12.228 INFO:tasks.workunit.client.0.smithi077.stdout:removed './209/23' 2022-04-03T19:49:12.228 INFO:tasks.workunit.client.0.smithi077.stdout:removed './209/35' 2022-04-03T19:49:12.228 INFO:tasks.workunit.client.0.smithi077.stdout:removed './209/26' 2022-04-03T19:49:12.229 INFO:tasks.workunit.client.0.smithi077.stdout:removed './209/37' 2022-04-03T19:49:12.229 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './209' 2022-04-03T19:49:12.229 INFO:tasks.workunit.client.0.smithi077.stdout:removed './210/21' 2022-04-03T19:49:12.229 INFO:tasks.workunit.client.0.smithi077.stdout:removed './210/27' 2022-04-03T19:49:12.230 INFO:tasks.workunit.client.0.smithi077.stdout:removed './210/31' 2022-04-03T19:49:12.230 INFO:tasks.workunit.client.0.smithi077.stdout:removed './210/36' 2022-04-03T19:49:12.230 INFO:tasks.workunit.client.0.smithi077.stdout:removed './210/20' 2022-04-03T19:49:12.230 INFO:tasks.workunit.client.0.smithi077.stdout:removed './210/39' 2022-04-03T19:49:12.230 INFO:tasks.workunit.client.0.smithi077.stdout:removed './210/34' 2022-04-03T19:49:12.231 INFO:tasks.workunit.client.0.smithi077.stdout:removed './210/38' 2022-04-03T19:49:12.231 INFO:tasks.workunit.client.0.smithi077.stdout:removed './210/28' 2022-04-03T19:49:12.231 INFO:tasks.workunit.client.0.smithi077.stdout:removed './210/24' 2022-04-03T19:49:12.231 INFO:tasks.workunit.client.0.smithi077.stdout:removed './210/29' 2022-04-03T19:49:12.232 INFO:tasks.workunit.client.0.smithi077.stdout:removed './210/32' 2022-04-03T19:49:12.232 INFO:tasks.workunit.client.0.smithi077.stdout:removed './210/22' 2022-04-03T19:49:12.232 INFO:tasks.workunit.client.0.smithi077.stdout:removed './210/25' 2022-04-03T19:49:12.232 INFO:tasks.workunit.client.0.smithi077.stdout:removed './210/30' 2022-04-03T19:49:12.232 INFO:tasks.workunit.client.0.smithi077.stdout:removed './210/33' 2022-04-03T19:49:12.233 INFO:tasks.workunit.client.0.smithi077.stdout:removed './210/23' 2022-04-03T19:49:12.233 INFO:tasks.workunit.client.0.smithi077.stdout:removed './210/35' 2022-04-03T19:49:12.233 INFO:tasks.workunit.client.0.smithi077.stdout:removed './210/26' 2022-04-03T19:49:12.233 INFO:tasks.workunit.client.0.smithi077.stdout:removed './210/37' 2022-04-03T19:49:12.234 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './210' 2022-04-03T19:49:12.234 INFO:tasks.workunit.client.0.smithi077.stdout:removed './211/21' 2022-04-03T19:49:12.234 INFO:tasks.workunit.client.0.smithi077.stdout:removed './211/27' 2022-04-03T19:49:12.234 INFO:tasks.workunit.client.0.smithi077.stdout:removed './211/31' 2022-04-03T19:49:12.234 INFO:tasks.workunit.client.0.smithi077.stdout:removed './211/36' 2022-04-03T19:49:12.235 INFO:tasks.workunit.client.0.smithi077.stdout:removed './211/20' 2022-04-03T19:49:12.235 INFO:tasks.workunit.client.0.smithi077.stdout:removed './211/39' 2022-04-03T19:49:12.235 INFO:tasks.workunit.client.0.smithi077.stdout:removed './211/34' 2022-04-03T19:49:12.235 INFO:tasks.workunit.client.0.smithi077.stdout:removed './211/38' 2022-04-03T19:49:12.236 INFO:tasks.workunit.client.0.smithi077.stdout:removed './211/28' 2022-04-03T19:49:12.236 INFO:tasks.workunit.client.0.smithi077.stdout:removed './211/24' 2022-04-03T19:49:12.236 INFO:tasks.workunit.client.0.smithi077.stdout:removed './211/29' 2022-04-03T19:49:12.236 INFO:tasks.workunit.client.0.smithi077.stdout:removed './211/32' 2022-04-03T19:49:12.236 INFO:tasks.workunit.client.0.smithi077.stdout:removed './211/22' 2022-04-03T19:49:12.237 INFO:tasks.workunit.client.0.smithi077.stdout:removed './211/25' 2022-04-03T19:49:12.237 INFO:tasks.workunit.client.0.smithi077.stdout:removed './211/30' 2022-04-03T19:49:12.237 INFO:tasks.workunit.client.0.smithi077.stdout:removed './211/33' 2022-04-03T19:49:12.237 INFO:tasks.workunit.client.0.smithi077.stdout:removed './211/23' 2022-04-03T19:49:12.237 INFO:tasks.workunit.client.0.smithi077.stdout:removed './211/35' 2022-04-03T19:49:12.238 INFO:tasks.workunit.client.0.smithi077.stdout:removed './211/26' 2022-04-03T19:49:12.238 INFO:tasks.workunit.client.0.smithi077.stdout:removed './211/37' 2022-04-03T19:49:12.238 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './211' 2022-04-03T19:49:12.238 INFO:tasks.workunit.client.0.smithi077.stdout:removed './212/21' 2022-04-03T19:49:12.239 INFO:tasks.workunit.client.0.smithi077.stdout:removed './212/27' 2022-04-03T19:49:12.239 INFO:tasks.workunit.client.0.smithi077.stdout:removed './212/31' 2022-04-03T19:49:12.239 INFO:tasks.workunit.client.0.smithi077.stdout:removed './212/36' 2022-04-03T19:49:12.239 INFO:tasks.workunit.client.0.smithi077.stdout:removed './212/20' 2022-04-03T19:49:12.239 INFO:tasks.workunit.client.0.smithi077.stdout:removed './212/39' 2022-04-03T19:49:12.240 INFO:tasks.workunit.client.0.smithi077.stdout:removed './212/34' 2022-04-03T19:49:12.240 INFO:tasks.workunit.client.0.smithi077.stdout:removed './212/38' 2022-04-03T19:49:12.240 INFO:tasks.workunit.client.0.smithi077.stdout:removed './212/28' 2022-04-03T19:49:12.240 INFO:tasks.workunit.client.0.smithi077.stdout:removed './212/24' 2022-04-03T19:49:12.241 INFO:tasks.workunit.client.0.smithi077.stdout:removed './212/29' 2022-04-03T19:49:12.241 INFO:tasks.workunit.client.0.smithi077.stdout:removed './212/32' 2022-04-03T19:49:12.241 INFO:tasks.workunit.client.0.smithi077.stdout:removed './212/22' 2022-04-03T19:49:12.241 INFO:tasks.workunit.client.0.smithi077.stdout:removed './212/25' 2022-04-03T19:49:12.241 INFO:tasks.workunit.client.0.smithi077.stdout:removed './212/30' 2022-04-03T19:49:12.242 INFO:tasks.workunit.client.0.smithi077.stdout:removed './212/33' 2022-04-03T19:49:12.242 INFO:tasks.workunit.client.0.smithi077.stdout:removed './212/23' 2022-04-03T19:49:12.242 INFO:tasks.workunit.client.0.smithi077.stdout:removed './212/35' 2022-04-03T19:49:12.242 INFO:tasks.workunit.client.0.smithi077.stdout:removed './212/26' 2022-04-03T19:49:12.243 INFO:tasks.workunit.client.0.smithi077.stdout:removed './212/37' 2022-04-03T19:49:12.243 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './212' 2022-04-03T19:49:12.243 INFO:tasks.workunit.client.0.smithi077.stdout:removed './213/21' 2022-04-03T19:49:12.243 INFO:tasks.workunit.client.0.smithi077.stdout:removed './213/27' 2022-04-03T19:49:12.243 INFO:tasks.workunit.client.0.smithi077.stdout:removed './213/31' 2022-04-03T19:49:12.244 INFO:tasks.workunit.client.0.smithi077.stdout:removed './213/36' 2022-04-03T19:49:12.244 INFO:tasks.workunit.client.0.smithi077.stdout:removed './213/20' 2022-04-03T19:49:12.244 INFO:tasks.workunit.client.0.smithi077.stdout:removed './213/39' 2022-04-03T19:49:12.244 INFO:tasks.workunit.client.0.smithi077.stdout:removed './213/34' 2022-04-03T19:49:12.245 INFO:tasks.workunit.client.0.smithi077.stdout:removed './213/38' 2022-04-03T19:49:12.245 INFO:tasks.workunit.client.0.smithi077.stdout:removed './213/28' 2022-04-03T19:49:12.245 INFO:tasks.workunit.client.0.smithi077.stdout:removed './213/24' 2022-04-03T19:49:12.245 INFO:tasks.workunit.client.0.smithi077.stdout:removed './213/29' 2022-04-03T19:49:12.245 INFO:tasks.workunit.client.0.smithi077.stdout:removed './213/32' 2022-04-03T19:49:12.246 INFO:tasks.workunit.client.0.smithi077.stdout:removed './213/22' 2022-04-03T19:49:12.246 INFO:tasks.workunit.client.0.smithi077.stdout:removed './213/25' 2022-04-03T19:49:12.246 INFO:tasks.workunit.client.0.smithi077.stdout:removed './213/30' 2022-04-03T19:49:12.246 INFO:tasks.workunit.client.0.smithi077.stdout:removed './213/33' 2022-04-03T19:49:12.247 INFO:tasks.workunit.client.0.smithi077.stdout:removed './213/23' 2022-04-03T19:49:12.247 INFO:tasks.workunit.client.0.smithi077.stdout:removed './213/35' 2022-04-03T19:49:12.247 INFO:tasks.workunit.client.0.smithi077.stdout:removed './213/26' 2022-04-03T19:49:12.247 INFO:tasks.workunit.client.0.smithi077.stdout:removed './213/37' 2022-04-03T19:49:12.247 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './213' 2022-04-03T19:49:12.248 INFO:tasks.workunit.client.0.smithi077.stdout:removed './214/21' 2022-04-03T19:49:12.248 INFO:tasks.workunit.client.0.smithi077.stdout:removed './214/27' 2022-04-03T19:49:12.248 INFO:tasks.workunit.client.0.smithi077.stdout:removed './214/31' 2022-04-03T19:49:12.248 INFO:tasks.workunit.client.0.smithi077.stdout:removed './214/36' 2022-04-03T19:49:12.248 INFO:tasks.workunit.client.0.smithi077.stdout:removed './214/20' 2022-04-03T19:49:12.249 INFO:tasks.workunit.client.0.smithi077.stdout:removed './214/39' 2022-04-03T19:49:12.249 INFO:tasks.workunit.client.0.smithi077.stdout:removed './214/34' 2022-04-03T19:49:12.249 INFO:tasks.workunit.client.0.smithi077.stdout:removed './214/38' 2022-04-03T19:49:12.249 INFO:tasks.workunit.client.0.smithi077.stdout:removed './214/28' 2022-04-03T19:49:12.250 INFO:tasks.workunit.client.0.smithi077.stdout:removed './214/24' 2022-04-03T19:49:12.250 INFO:tasks.workunit.client.0.smithi077.stdout:removed './214/29' 2022-04-03T19:49:12.250 INFO:tasks.workunit.client.0.smithi077.stdout:removed './214/32' 2022-04-03T19:49:12.250 INFO:tasks.workunit.client.0.smithi077.stdout:removed './214/22' 2022-04-03T19:49:12.250 INFO:tasks.workunit.client.0.smithi077.stdout:removed './214/25' 2022-04-03T19:49:12.251 INFO:tasks.workunit.client.0.smithi077.stdout:removed './214/30' 2022-04-03T19:49:12.251 INFO:tasks.workunit.client.0.smithi077.stdout:removed './214/33' 2022-04-03T19:49:12.251 INFO:tasks.workunit.client.0.smithi077.stdout:removed './214/23' 2022-04-03T19:49:12.251 INFO:tasks.workunit.client.0.smithi077.stdout:removed './214/35' 2022-04-03T19:49:12.251 INFO:tasks.workunit.client.0.smithi077.stdout:removed './214/26' 2022-04-03T19:49:12.252 INFO:tasks.workunit.client.0.smithi077.stdout:removed './214/37' 2022-04-03T19:49:12.252 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './214' 2022-04-03T19:49:12.252 INFO:tasks.workunit.client.0.smithi077.stdout:removed './215/21' 2022-04-03T19:49:12.252 INFO:tasks.workunit.client.0.smithi077.stdout:removed './215/27' 2022-04-03T19:49:12.253 INFO:tasks.workunit.client.0.smithi077.stdout:removed './215/31' 2022-04-03T19:49:12.253 INFO:tasks.workunit.client.0.smithi077.stdout:removed './215/36' 2022-04-03T19:49:12.253 INFO:tasks.workunit.client.0.smithi077.stdout:removed './215/20' 2022-04-03T19:49:12.253 INFO:tasks.workunit.client.0.smithi077.stdout:removed './215/39' 2022-04-03T19:49:12.253 INFO:tasks.workunit.client.0.smithi077.stdout:removed './215/34' 2022-04-03T19:49:12.254 INFO:tasks.workunit.client.0.smithi077.stdout:removed './215/38' 2022-04-03T19:49:12.254 INFO:tasks.workunit.client.0.smithi077.stdout:removed './215/28' 2022-04-03T19:49:12.254 INFO:tasks.workunit.client.0.smithi077.stdout:removed './215/24' 2022-04-03T19:49:12.254 INFO:tasks.workunit.client.0.smithi077.stdout:removed './215/29' 2022-04-03T19:49:12.254 INFO:tasks.workunit.client.0.smithi077.stdout:removed './215/32' 2022-04-03T19:49:12.255 INFO:tasks.workunit.client.0.smithi077.stdout:removed './215/22' 2022-04-03T19:49:12.255 INFO:tasks.workunit.client.0.smithi077.stdout:removed './215/25' 2022-04-03T19:49:12.255 INFO:tasks.workunit.client.0.smithi077.stdout:removed './215/30' 2022-04-03T19:49:12.255 INFO:tasks.workunit.client.0.smithi077.stdout:removed './215/33' 2022-04-03T19:49:12.256 INFO:tasks.workunit.client.0.smithi077.stdout:removed './215/23' 2022-04-03T19:49:12.256 INFO:tasks.workunit.client.0.smithi077.stdout:removed './215/35' 2022-04-03T19:49:12.256 INFO:tasks.workunit.client.0.smithi077.stdout:removed './215/26' 2022-04-03T19:49:12.256 INFO:tasks.workunit.client.0.smithi077.stdout:removed './215/37' 2022-04-03T19:49:12.256 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './215' 2022-04-03T19:49:12.257 INFO:tasks.workunit.client.0.smithi077.stdout:removed './216/21' 2022-04-03T19:49:12.257 INFO:tasks.workunit.client.0.smithi077.stdout:removed './216/27' 2022-04-03T19:49:12.257 INFO:tasks.workunit.client.0.smithi077.stdout:removed './216/31' 2022-04-03T19:49:12.257 INFO:tasks.workunit.client.0.smithi077.stdout:removed './216/36' 2022-04-03T19:49:12.257 INFO:tasks.workunit.client.0.smithi077.stdout:removed './216/20' 2022-04-03T19:49:12.258 INFO:tasks.workunit.client.0.smithi077.stdout:removed './216/39' 2022-04-03T19:49:12.258 INFO:tasks.workunit.client.0.smithi077.stdout:removed './216/34' 2022-04-03T19:49:12.258 INFO:tasks.workunit.client.0.smithi077.stdout:removed './216/38' 2022-04-03T19:49:14.191 INFO:tasks.workunit.client.0.smithi077.stdout:removed './216/28' 2022-04-03T19:49:14.192 INFO:tasks.workunit.client.0.smithi077.stdout:removed './216/24' 2022-04-03T19:49:14.192 INFO:tasks.workunit.client.0.smithi077.stdout:removed './216/29' 2022-04-03T19:49:14.192 INFO:tasks.workunit.client.0.smithi077.stdout:removed './216/32' 2022-04-03T19:49:14.192 INFO:tasks.workunit.client.0.smithi077.stdout:removed './216/22' 2022-04-03T19:49:14.193 INFO:tasks.workunit.client.0.smithi077.stdout:removed './216/25' 2022-04-03T19:49:14.193 INFO:tasks.workunit.client.0.smithi077.stdout:removed './216/30' 2022-04-03T19:49:14.193 INFO:tasks.workunit.client.0.smithi077.stdout:removed './216/33' 2022-04-03T19:49:14.193 INFO:tasks.workunit.client.0.smithi077.stdout:removed './216/23' 2022-04-03T19:49:14.194 INFO:tasks.workunit.client.0.smithi077.stdout:removed './216/35' 2022-04-03T19:49:14.194 INFO:tasks.workunit.client.0.smithi077.stdout:removed './216/26' 2022-04-03T19:49:14.194 INFO:tasks.workunit.client.0.smithi077.stdout:removed './216/37' 2022-04-03T19:49:14.194 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './216' 2022-04-03T19:49:14.195 INFO:tasks.workunit.client.0.smithi077.stdout:removed './217/21' 2022-04-03T19:49:14.195 INFO:tasks.workunit.client.0.smithi077.stdout:removed './217/27' 2022-04-03T19:49:14.195 INFO:tasks.workunit.client.0.smithi077.stdout:removed './217/31' 2022-04-03T19:49:14.195 INFO:tasks.workunit.client.0.smithi077.stdout:removed './217/36' 2022-04-03T19:49:14.195 INFO:tasks.workunit.client.0.smithi077.stdout:removed './217/20' 2022-04-03T19:49:14.196 INFO:tasks.workunit.client.0.smithi077.stdout:removed './217/39' 2022-04-03T19:49:14.196 INFO:tasks.workunit.client.0.smithi077.stdout:removed './217/34' 2022-04-03T19:49:14.196 INFO:tasks.workunit.client.0.smithi077.stdout:removed './217/38' 2022-04-03T19:49:14.196 INFO:tasks.workunit.client.0.smithi077.stdout:removed './217/28' 2022-04-03T19:49:14.196 INFO:tasks.workunit.client.0.smithi077.stdout:removed './217/24' 2022-04-03T19:49:14.197 INFO:tasks.workunit.client.0.smithi077.stdout:removed './217/29' 2022-04-03T19:49:14.197 INFO:tasks.workunit.client.0.smithi077.stdout:removed './217/32' 2022-04-03T19:49:14.197 INFO:tasks.workunit.client.0.smithi077.stdout:removed './217/22' 2022-04-03T19:49:14.197 INFO:tasks.workunit.client.0.smithi077.stdout:removed './217/25' 2022-04-03T19:49:14.198 INFO:tasks.workunit.client.0.smithi077.stdout:removed './217/30' 2022-04-03T19:49:14.198 INFO:tasks.workunit.client.0.smithi077.stdout:removed './217/33' 2022-04-03T19:49:14.198 INFO:tasks.workunit.client.0.smithi077.stdout:removed './217/23' 2022-04-03T19:49:14.198 INFO:tasks.workunit.client.0.smithi077.stdout:removed './217/35' 2022-04-03T19:49:14.199 INFO:tasks.workunit.client.0.smithi077.stdout:removed './217/26' 2022-04-03T19:49:14.199 INFO:tasks.workunit.client.0.smithi077.stdout:removed './217/37' 2022-04-03T19:49:14.199 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './217' 2022-04-03T19:49:14.199 INFO:tasks.workunit.client.0.smithi077.stdout:removed './218/21' 2022-04-03T19:49:14.199 INFO:tasks.workunit.client.0.smithi077.stdout:removed './218/27' 2022-04-03T19:49:14.200 INFO:tasks.workunit.client.0.smithi077.stdout:removed './218/31' 2022-04-03T19:49:14.200 INFO:tasks.workunit.client.0.smithi077.stdout:removed './218/36' 2022-04-03T19:49:14.200 INFO:tasks.workunit.client.0.smithi077.stdout:removed './218/20' 2022-04-03T19:49:14.200 INFO:tasks.workunit.client.0.smithi077.stdout:removed './218/39' 2022-04-03T19:49:14.201 INFO:tasks.workunit.client.0.smithi077.stdout:removed './218/34' 2022-04-03T19:49:14.201 INFO:tasks.workunit.client.0.smithi077.stdout:removed './218/38' 2022-04-03T19:49:14.201 INFO:tasks.workunit.client.0.smithi077.stdout:removed './218/28' 2022-04-03T19:49:14.201 INFO:tasks.workunit.client.0.smithi077.stdout:removed './218/24' 2022-04-03T19:49:14.201 INFO:tasks.workunit.client.0.smithi077.stdout:removed './218/29' 2022-04-03T19:49:14.202 INFO:tasks.workunit.client.0.smithi077.stdout:removed './218/32' 2022-04-03T19:49:14.202 INFO:tasks.workunit.client.0.smithi077.stdout:removed './218/22' 2022-04-03T19:49:14.202 INFO:tasks.workunit.client.0.smithi077.stdout:removed './218/25' 2022-04-03T19:49:14.202 INFO:tasks.workunit.client.0.smithi077.stdout:removed './218/30' 2022-04-03T19:49:14.203 INFO:tasks.workunit.client.0.smithi077.stdout:removed './218/33' 2022-04-03T19:49:14.203 INFO:tasks.workunit.client.0.smithi077.stdout:removed './218/23' 2022-04-03T19:49:14.203 INFO:tasks.workunit.client.0.smithi077.stdout:removed './218/35' 2022-04-03T19:49:14.203 INFO:tasks.workunit.client.0.smithi077.stdout:removed './218/26' 2022-04-03T19:49:14.203 INFO:tasks.workunit.client.0.smithi077.stdout:removed './218/37' 2022-04-03T19:49:14.204 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './218' 2022-04-03T19:49:14.204 INFO:tasks.workunit.client.0.smithi077.stdout:removed './219/21' 2022-04-03T19:49:14.204 INFO:tasks.workunit.client.0.smithi077.stdout:removed './219/27' 2022-04-03T19:49:14.205 INFO:tasks.workunit.client.0.smithi077.stdout:removed './219/31' 2022-04-03T19:49:14.205 INFO:tasks.workunit.client.0.smithi077.stdout:removed './219/36' 2022-04-03T19:49:14.205 INFO:tasks.workunit.client.0.smithi077.stdout:removed './219/20' 2022-04-03T19:49:14.205 INFO:tasks.workunit.client.0.smithi077.stdout:removed './219/39' 2022-04-03T19:49:14.206 INFO:tasks.workunit.client.0.smithi077.stdout:removed './219/34' 2022-04-03T19:49:14.206 INFO:tasks.workunit.client.0.smithi077.stdout:removed './219/38' 2022-04-03T19:49:14.206 INFO:tasks.workunit.client.0.smithi077.stdout:removed './219/28' 2022-04-03T19:49:14.206 INFO:tasks.workunit.client.0.smithi077.stdout:removed './219/24' 2022-04-03T19:49:14.207 INFO:tasks.workunit.client.0.smithi077.stdout:removed './219/29' 2022-04-03T19:49:14.207 INFO:tasks.workunit.client.0.smithi077.stdout:removed './219/32' 2022-04-03T19:49:14.207 INFO:tasks.workunit.client.0.smithi077.stdout:removed './219/22' 2022-04-03T19:49:14.207 INFO:tasks.workunit.client.0.smithi077.stdout:removed './219/25' 2022-04-03T19:49:14.207 INFO:tasks.workunit.client.0.smithi077.stdout:removed './219/30' 2022-04-03T19:49:14.208 INFO:tasks.workunit.client.0.smithi077.stdout:removed './219/33' 2022-04-03T19:49:14.208 INFO:tasks.workunit.client.0.smithi077.stdout:removed './219/23' 2022-04-03T19:49:14.208 INFO:tasks.workunit.client.0.smithi077.stdout:removed './219/35' 2022-04-03T19:49:14.208 INFO:tasks.workunit.client.0.smithi077.stdout:removed './219/26' 2022-04-03T19:49:14.209 INFO:tasks.workunit.client.0.smithi077.stdout:removed './219/37' 2022-04-03T19:49:14.209 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './219' 2022-04-03T19:49:14.209 INFO:tasks.workunit.client.0.smithi077.stdout:removed './220/21' 2022-04-03T19:49:14.209 INFO:tasks.workunit.client.0.smithi077.stdout:removed './220/27' 2022-04-03T19:49:14.209 INFO:tasks.workunit.client.0.smithi077.stdout:removed './220/31' 2022-04-03T19:49:14.210 INFO:tasks.workunit.client.0.smithi077.stdout:removed './220/36' 2022-04-03T19:49:14.210 INFO:tasks.workunit.client.0.smithi077.stdout:removed './220/20' 2022-04-03T19:49:14.210 INFO:tasks.workunit.client.0.smithi077.stdout:removed './220/39' 2022-04-03T19:49:14.210 INFO:tasks.workunit.client.0.smithi077.stdout:removed './220/34' 2022-04-03T19:49:14.211 INFO:tasks.workunit.client.0.smithi077.stdout:removed './220/38' 2022-04-03T19:49:14.211 INFO:tasks.workunit.client.0.smithi077.stdout:removed './220/28' 2022-04-03T19:49:14.211 INFO:tasks.workunit.client.0.smithi077.stdout:removed './220/24' 2022-04-03T19:49:14.211 INFO:tasks.workunit.client.0.smithi077.stdout:removed './220/29' 2022-04-03T19:49:14.211 INFO:tasks.workunit.client.0.smithi077.stdout:removed './220/32' 2022-04-03T19:49:14.212 INFO:tasks.workunit.client.0.smithi077.stdout:removed './220/22' 2022-04-03T19:49:14.212 INFO:tasks.workunit.client.0.smithi077.stdout:removed './220/25' 2022-04-03T19:49:14.212 INFO:tasks.workunit.client.0.smithi077.stdout:removed './220/30' 2022-04-03T19:49:14.212 INFO:tasks.workunit.client.0.smithi077.stdout:removed './220/33' 2022-04-03T19:49:14.212 INFO:tasks.workunit.client.0.smithi077.stdout:removed './220/23' 2022-04-03T19:49:14.213 INFO:tasks.workunit.client.0.smithi077.stdout:removed './220/35' 2022-04-03T19:49:14.213 INFO:tasks.workunit.client.0.smithi077.stdout:removed './220/26' 2022-04-03T19:49:14.213 INFO:tasks.workunit.client.0.smithi077.stdout:removed './220/37' 2022-04-03T19:49:14.213 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './220' 2022-04-03T19:49:14.214 INFO:tasks.workunit.client.0.smithi077.stdout:removed './221/21' 2022-04-03T19:49:14.214 INFO:tasks.workunit.client.0.smithi077.stdout:removed './221/27' 2022-04-03T19:49:14.214 INFO:tasks.workunit.client.0.smithi077.stdout:removed './221/31' 2022-04-03T19:49:14.214 INFO:tasks.workunit.client.0.smithi077.stdout:removed './221/36' 2022-04-03T19:49:14.214 INFO:tasks.workunit.client.0.smithi077.stdout:removed './221/20' 2022-04-03T19:49:14.215 INFO:tasks.workunit.client.0.smithi077.stdout:removed './221/39' 2022-04-03T19:49:14.215 INFO:tasks.workunit.client.0.smithi077.stdout:removed './221/34' 2022-04-03T19:49:14.215 INFO:tasks.workunit.client.0.smithi077.stdout:removed './221/38' 2022-04-03T19:49:14.215 INFO:tasks.workunit.client.0.smithi077.stdout:removed './221/28' 2022-04-03T19:49:14.216 INFO:tasks.workunit.client.0.smithi077.stdout:removed './221/24' 2022-04-03T19:49:14.216 INFO:tasks.workunit.client.0.smithi077.stdout:removed './221/29' 2022-04-03T19:49:14.216 INFO:tasks.workunit.client.0.smithi077.stdout:removed './221/32' 2022-04-03T19:49:14.216 INFO:tasks.workunit.client.0.smithi077.stdout:removed './221/22' 2022-04-03T19:49:14.217 INFO:tasks.workunit.client.0.smithi077.stdout:removed './221/25' 2022-04-03T19:49:14.217 INFO:tasks.workunit.client.0.smithi077.stdout:removed './221/30' 2022-04-03T19:49:14.217 INFO:tasks.workunit.client.0.smithi077.stdout:removed './221/33' 2022-04-03T19:49:14.217 INFO:tasks.workunit.client.0.smithi077.stdout:removed './221/23' 2022-04-03T19:49:14.218 INFO:tasks.workunit.client.0.smithi077.stdout:removed './221/35' 2022-04-03T19:49:14.218 INFO:tasks.workunit.client.0.smithi077.stdout:removed './221/26' 2022-04-03T19:49:14.218 INFO:tasks.workunit.client.0.smithi077.stdout:removed './221/37' 2022-04-03T19:49:14.218 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './221' 2022-04-03T19:49:14.219 INFO:tasks.workunit.client.0.smithi077.stdout:removed './222/21' 2022-04-03T19:49:14.219 INFO:tasks.workunit.client.0.smithi077.stdout:removed './222/27' 2022-04-03T19:49:14.219 INFO:tasks.workunit.client.0.smithi077.stdout:removed './222/31' 2022-04-03T19:49:14.219 INFO:tasks.workunit.client.0.smithi077.stdout:removed './222/36' 2022-04-03T19:49:14.220 INFO:tasks.workunit.client.0.smithi077.stdout:removed './222/20' 2022-04-03T19:49:14.220 INFO:tasks.workunit.client.0.smithi077.stdout:removed './222/39' 2022-04-03T19:49:14.220 INFO:tasks.workunit.client.0.smithi077.stdout:removed './222/34' 2022-04-03T19:49:14.220 INFO:tasks.workunit.client.0.smithi077.stdout:removed './222/38' 2022-04-03T19:49:14.221 INFO:tasks.workunit.client.0.smithi077.stdout:removed './222/28' 2022-04-03T19:49:14.221 INFO:tasks.workunit.client.0.smithi077.stdout:removed './222/24' 2022-04-03T19:49:14.221 INFO:tasks.workunit.client.0.smithi077.stdout:removed './222/29' 2022-04-03T19:49:14.221 INFO:tasks.workunit.client.0.smithi077.stdout:removed './222/32' 2022-04-03T19:49:14.221 INFO:tasks.workunit.client.0.smithi077.stdout:removed './222/22' 2022-04-03T19:49:14.222 INFO:tasks.workunit.client.0.smithi077.stdout:removed './222/25' 2022-04-03T19:49:14.222 INFO:tasks.workunit.client.0.smithi077.stdout:removed './222/30' 2022-04-03T19:49:14.222 INFO:tasks.workunit.client.0.smithi077.stdout:removed './222/33' 2022-04-03T19:49:14.222 INFO:tasks.workunit.client.0.smithi077.stdout:removed './222/23' 2022-04-03T19:49:14.223 INFO:tasks.workunit.client.0.smithi077.stdout:removed './222/35' 2022-04-03T19:49:14.223 INFO:tasks.workunit.client.0.smithi077.stdout:removed './222/26' 2022-04-03T19:49:14.223 INFO:tasks.workunit.client.0.smithi077.stdout:removed './222/37' 2022-04-03T19:49:14.223 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './222' 2022-04-03T19:49:14.223 INFO:tasks.workunit.client.0.smithi077.stdout:removed './223/21' 2022-04-03T19:49:14.224 INFO:tasks.workunit.client.0.smithi077.stdout:removed './223/27' 2022-04-03T19:49:14.224 INFO:tasks.workunit.client.0.smithi077.stdout:removed './223/31' 2022-04-03T19:49:14.224 INFO:tasks.workunit.client.0.smithi077.stdout:removed './223/36' 2022-04-03T19:49:14.224 INFO:tasks.workunit.client.0.smithi077.stdout:removed './223/20' 2022-04-03T19:49:14.224 INFO:tasks.workunit.client.0.smithi077.stdout:removed './223/39' 2022-04-03T19:49:14.225 INFO:tasks.workunit.client.0.smithi077.stdout:removed './223/34' 2022-04-03T19:49:14.225 INFO:tasks.workunit.client.0.smithi077.stdout:removed './223/38' 2022-04-03T19:49:14.225 INFO:tasks.workunit.client.0.smithi077.stdout:removed './223/28' 2022-04-03T19:49:14.225 INFO:tasks.workunit.client.0.smithi077.stdout:removed './223/24' 2022-04-03T19:49:14.226 INFO:tasks.workunit.client.0.smithi077.stdout:removed './223/29' 2022-04-03T19:49:14.226 INFO:tasks.workunit.client.0.smithi077.stdout:removed './223/32' 2022-04-03T19:49:14.226 INFO:tasks.workunit.client.0.smithi077.stdout:removed './223/22' 2022-04-03T19:49:14.226 INFO:tasks.workunit.client.0.smithi077.stdout:removed './223/25' 2022-04-03T19:49:14.226 INFO:tasks.workunit.client.0.smithi077.stdout:removed './223/30' 2022-04-03T19:49:14.227 INFO:tasks.workunit.client.0.smithi077.stdout:removed './223/33' 2022-04-03T19:49:14.227 INFO:tasks.workunit.client.0.smithi077.stdout:removed './223/23' 2022-04-03T19:49:14.227 INFO:tasks.workunit.client.0.smithi077.stdout:removed './223/35' 2022-04-03T19:49:14.228 INFO:tasks.workunit.client.0.smithi077.stdout:removed './223/26' 2022-04-03T19:49:14.228 INFO:tasks.workunit.client.0.smithi077.stdout:removed './223/37' 2022-04-03T19:49:14.228 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './223' 2022-04-03T19:49:14.228 INFO:tasks.workunit.client.0.smithi077.stdout:removed './224/21' 2022-04-03T19:49:14.228 INFO:tasks.workunit.client.0.smithi077.stdout:removed './224/27' 2022-04-03T19:49:14.229 INFO:tasks.workunit.client.0.smithi077.stdout:removed './224/31' 2022-04-03T19:49:14.229 INFO:tasks.workunit.client.0.smithi077.stdout:removed './224/36' 2022-04-03T19:49:14.229 INFO:tasks.workunit.client.0.smithi077.stdout:removed './224/20' 2022-04-03T19:49:14.229 INFO:tasks.workunit.client.0.smithi077.stdout:removed './224/39' 2022-04-03T19:49:14.230 INFO:tasks.workunit.client.0.smithi077.stdout:removed './224/34' 2022-04-03T19:49:14.230 INFO:tasks.workunit.client.0.smithi077.stdout:removed './224/38' 2022-04-03T19:49:14.230 INFO:tasks.workunit.client.0.smithi077.stdout:removed './224/28' 2022-04-03T19:49:14.230 INFO:tasks.workunit.client.0.smithi077.stdout:removed './224/24' 2022-04-03T19:49:14.231 INFO:tasks.workunit.client.0.smithi077.stdout:removed './224/29' 2022-04-03T19:49:14.231 INFO:tasks.workunit.client.0.smithi077.stdout:removed './224/32' 2022-04-03T19:49:14.231 INFO:tasks.workunit.client.0.smithi077.stdout:removed './224/22' 2022-04-03T19:49:14.231 INFO:tasks.workunit.client.0.smithi077.stdout:removed './224/25' 2022-04-03T19:49:14.231 INFO:tasks.workunit.client.0.smithi077.stdout:removed './224/30' 2022-04-03T19:49:14.232 INFO:tasks.workunit.client.0.smithi077.stdout:removed './224/33' 2022-04-03T19:49:14.232 INFO:tasks.workunit.client.0.smithi077.stdout:removed './224/23' 2022-04-03T19:49:14.232 INFO:tasks.workunit.client.0.smithi077.stdout:removed './224/35' 2022-04-03T19:49:14.232 INFO:tasks.workunit.client.0.smithi077.stdout:removed './224/26' 2022-04-03T19:49:14.233 INFO:tasks.workunit.client.0.smithi077.stdout:removed './224/37' 2022-04-03T19:49:14.233 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './224' 2022-04-03T19:49:14.233 INFO:tasks.workunit.client.0.smithi077.stdout:removed './225/21' 2022-04-03T19:49:14.233 INFO:tasks.workunit.client.0.smithi077.stdout:removed './225/27' 2022-04-03T19:49:14.234 INFO:tasks.workunit.client.0.smithi077.stdout:removed './225/31' 2022-04-03T19:49:14.234 INFO:tasks.workunit.client.0.smithi077.stdout:removed './225/36' 2022-04-03T19:49:14.234 INFO:tasks.workunit.client.0.smithi077.stdout:removed './225/20' 2022-04-03T19:49:14.234 INFO:tasks.workunit.client.0.smithi077.stdout:removed './225/39' 2022-04-03T19:49:14.235 INFO:tasks.workunit.client.0.smithi077.stdout:removed './225/34' 2022-04-03T19:49:14.235 INFO:tasks.workunit.client.0.smithi077.stdout:removed './225/38' 2022-04-03T19:49:14.235 INFO:tasks.workunit.client.0.smithi077.stdout:removed './225/28' 2022-04-03T19:49:14.235 INFO:tasks.workunit.client.0.smithi077.stdout:removed './225/24' 2022-04-03T19:49:14.236 INFO:tasks.workunit.client.0.smithi077.stdout:removed './225/29' 2022-04-03T19:49:14.236 INFO:tasks.workunit.client.0.smithi077.stdout:removed './225/32' 2022-04-03T19:49:14.236 INFO:tasks.workunit.client.0.smithi077.stdout:removed './225/22' 2022-04-03T19:49:14.236 INFO:tasks.workunit.client.0.smithi077.stdout:removed './225/25' 2022-04-03T19:49:14.236 INFO:tasks.workunit.client.0.smithi077.stdout:removed './225/30' 2022-04-03T19:49:14.237 INFO:tasks.workunit.client.0.smithi077.stdout:removed './225/33' 2022-04-03T19:49:14.237 INFO:tasks.workunit.client.0.smithi077.stdout:removed './225/23' 2022-04-03T19:49:14.237 INFO:tasks.workunit.client.0.smithi077.stdout:removed './225/35' 2022-04-03T19:49:14.237 INFO:tasks.workunit.client.0.smithi077.stdout:removed './225/26' 2022-04-03T19:49:14.238 INFO:tasks.workunit.client.0.smithi077.stdout:removed './225/37' 2022-04-03T19:49:14.238 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './225' 2022-04-03T19:49:14.238 INFO:tasks.workunit.client.0.smithi077.stdout:removed './226/21' 2022-04-03T19:49:14.238 INFO:tasks.workunit.client.0.smithi077.stdout:removed './226/27' 2022-04-03T19:49:14.238 INFO:tasks.workunit.client.0.smithi077.stdout:removed './226/31' 2022-04-03T19:49:14.239 INFO:tasks.workunit.client.0.smithi077.stdout:removed './226/36' 2022-04-03T19:49:14.239 INFO:tasks.workunit.client.0.smithi077.stdout:removed './226/20' 2022-04-03T19:49:14.239 INFO:tasks.workunit.client.0.smithi077.stdout:removed './226/39' 2022-04-03T19:49:14.239 INFO:tasks.workunit.client.0.smithi077.stdout:removed './226/34' 2022-04-03T19:49:14.240 INFO:tasks.workunit.client.0.smithi077.stdout:removed './226/38' 2022-04-03T19:49:14.240 INFO:tasks.workunit.client.0.smithi077.stdout:removed './226/28' 2022-04-03T19:49:14.240 INFO:tasks.workunit.client.0.smithi077.stdout:removed './226/24' 2022-04-03T19:49:15.688 INFO:tasks.workunit.client.0.smithi077.stdout:removed './226/29' 2022-04-03T19:49:15.689 INFO:tasks.workunit.client.0.smithi077.stdout:removed './226/32' 2022-04-03T19:49:15.689 INFO:tasks.workunit.client.0.smithi077.stdout:removed './226/22' 2022-04-03T19:49:15.689 INFO:tasks.workunit.client.0.smithi077.stdout:removed './226/25' 2022-04-03T19:49:15.689 INFO:tasks.workunit.client.0.smithi077.stdout:removed './226/30' 2022-04-03T19:49:15.690 INFO:tasks.workunit.client.0.smithi077.stdout:removed './226/33' 2022-04-03T19:49:15.690 INFO:tasks.workunit.client.0.smithi077.stdout:removed './226/23' 2022-04-03T19:49:15.690 INFO:tasks.workunit.client.0.smithi077.stdout:removed './226/35' 2022-04-03T19:49:15.690 INFO:tasks.workunit.client.0.smithi077.stdout:removed './226/26' 2022-04-03T19:49:15.690 INFO:tasks.workunit.client.0.smithi077.stdout:removed './226/37' 2022-04-03T19:49:15.691 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './226' 2022-04-03T19:49:15.691 INFO:tasks.workunit.client.0.smithi077.stdout:removed './227/21' 2022-04-03T19:49:15.691 INFO:tasks.workunit.client.0.smithi077.stdout:removed './227/27' 2022-04-03T19:49:15.691 INFO:tasks.workunit.client.0.smithi077.stdout:removed './227/31' 2022-04-03T19:49:15.692 INFO:tasks.workunit.client.0.smithi077.stdout:removed './227/36' 2022-04-03T19:49:15.692 INFO:tasks.workunit.client.0.smithi077.stdout:removed './227/20' 2022-04-03T19:49:15.692 INFO:tasks.workunit.client.0.smithi077.stdout:removed './227/39' 2022-04-03T19:49:15.692 INFO:tasks.workunit.client.0.smithi077.stdout:removed './227/34' 2022-04-03T19:49:15.693 INFO:tasks.workunit.client.0.smithi077.stdout:removed './227/38' 2022-04-03T19:49:15.693 INFO:tasks.workunit.client.0.smithi077.stdout:removed './227/28' 2022-04-03T19:49:15.693 INFO:tasks.workunit.client.0.smithi077.stdout:removed './227/24' 2022-04-03T19:49:15.693 INFO:tasks.workunit.client.0.smithi077.stdout:removed './227/29' 2022-04-03T19:49:15.693 INFO:tasks.workunit.client.0.smithi077.stdout:removed './227/32' 2022-04-03T19:49:15.694 INFO:tasks.workunit.client.0.smithi077.stdout:removed './227/22' 2022-04-03T19:49:15.694 INFO:tasks.workunit.client.0.smithi077.stdout:removed './227/25' 2022-04-03T19:49:15.694 INFO:tasks.workunit.client.0.smithi077.stdout:removed './227/30' 2022-04-03T19:49:15.694 INFO:tasks.workunit.client.0.smithi077.stdout:removed './227/33' 2022-04-03T19:49:15.695 INFO:tasks.workunit.client.0.smithi077.stdout:removed './227/23' 2022-04-03T19:49:15.695 INFO:tasks.workunit.client.0.smithi077.stdout:removed './227/35' 2022-04-03T19:49:15.695 INFO:tasks.workunit.client.0.smithi077.stdout:removed './227/26' 2022-04-03T19:49:15.695 INFO:tasks.workunit.client.0.smithi077.stdout:removed './227/37' 2022-04-03T19:49:15.696 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './227' 2022-04-03T19:49:15.696 INFO:tasks.workunit.client.0.smithi077.stdout:removed './228/21' 2022-04-03T19:49:15.696 INFO:tasks.workunit.client.0.smithi077.stdout:removed './228/27' 2022-04-03T19:49:15.696 INFO:tasks.workunit.client.0.smithi077.stdout:removed './228/31' 2022-04-03T19:49:15.697 INFO:tasks.workunit.client.0.smithi077.stdout:removed './228/36' 2022-04-03T19:49:15.697 INFO:tasks.workunit.client.0.smithi077.stdout:removed './228/20' 2022-04-03T19:49:15.697 INFO:tasks.workunit.client.0.smithi077.stdout:removed './228/39' 2022-04-03T19:49:15.697 INFO:tasks.workunit.client.0.smithi077.stdout:removed './228/34' 2022-04-03T19:49:15.698 INFO:tasks.workunit.client.0.smithi077.stdout:removed './228/38' 2022-04-03T19:49:15.698 INFO:tasks.workunit.client.0.smithi077.stdout:removed './228/28' 2022-04-03T19:49:15.698 INFO:tasks.workunit.client.0.smithi077.stdout:removed './228/24' 2022-04-03T19:49:15.698 INFO:tasks.workunit.client.0.smithi077.stdout:removed './228/29' 2022-04-03T19:49:15.698 INFO:tasks.workunit.client.0.smithi077.stdout:removed './228/32' 2022-04-03T19:49:15.699 INFO:tasks.workunit.client.0.smithi077.stdout:removed './228/22' 2022-04-03T19:49:15.699 INFO:tasks.workunit.client.0.smithi077.stdout:removed './228/25' 2022-04-03T19:49:15.699 INFO:tasks.workunit.client.0.smithi077.stdout:removed './228/30' 2022-04-03T19:49:15.699 INFO:tasks.workunit.client.0.smithi077.stdout:removed './228/33' 2022-04-03T19:49:15.700 INFO:tasks.workunit.client.0.smithi077.stdout:removed './228/23' 2022-04-03T19:49:15.700 INFO:tasks.workunit.client.0.smithi077.stdout:removed './228/35' 2022-04-03T19:49:15.700 INFO:tasks.workunit.client.0.smithi077.stdout:removed './228/26' 2022-04-03T19:49:15.700 INFO:tasks.workunit.client.0.smithi077.stdout:removed './228/37' 2022-04-03T19:49:15.701 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './228' 2022-04-03T19:49:15.701 INFO:tasks.workunit.client.0.smithi077.stdout:removed './229/21' 2022-04-03T19:49:15.701 INFO:tasks.workunit.client.0.smithi077.stdout:removed './229/27' 2022-04-03T19:49:15.701 INFO:tasks.workunit.client.0.smithi077.stdout:removed './229/31' 2022-04-03T19:49:15.701 INFO:tasks.workunit.client.0.smithi077.stdout:removed './229/36' 2022-04-03T19:49:15.702 INFO:tasks.workunit.client.0.smithi077.stdout:removed './229/20' 2022-04-03T19:49:15.702 INFO:tasks.workunit.client.0.smithi077.stdout:removed './229/39' 2022-04-03T19:49:15.702 INFO:tasks.workunit.client.0.smithi077.stdout:removed './229/34' 2022-04-03T19:49:15.702 INFO:tasks.workunit.client.0.smithi077.stdout:removed './229/38' 2022-04-03T19:49:15.703 INFO:tasks.workunit.client.0.smithi077.stdout:removed './229/28' 2022-04-03T19:49:15.703 INFO:tasks.workunit.client.0.smithi077.stdout:removed './229/24' 2022-04-03T19:49:15.703 INFO:tasks.workunit.client.0.smithi077.stdout:removed './229/29' 2022-04-03T19:49:15.703 INFO:tasks.workunit.client.0.smithi077.stdout:removed './229/32' 2022-04-03T19:49:15.704 INFO:tasks.workunit.client.0.smithi077.stdout:removed './229/22' 2022-04-03T19:49:15.704 INFO:tasks.workunit.client.0.smithi077.stdout:removed './229/25' 2022-04-03T19:49:15.704 INFO:tasks.workunit.client.0.smithi077.stdout:removed './229/30' 2022-04-03T19:49:15.704 INFO:tasks.workunit.client.0.smithi077.stdout:removed './229/33' 2022-04-03T19:49:15.704 INFO:tasks.workunit.client.0.smithi077.stdout:removed './229/23' 2022-04-03T19:49:15.705 INFO:tasks.workunit.client.0.smithi077.stdout:removed './229/35' 2022-04-03T19:49:15.705 INFO:tasks.workunit.client.0.smithi077.stdout:removed './229/26' 2022-04-03T19:49:15.705 INFO:tasks.workunit.client.0.smithi077.stdout:removed './229/37' 2022-04-03T19:49:15.705 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './229' 2022-04-03T19:49:15.706 INFO:tasks.workunit.client.0.smithi077.stdout:removed './230/21' 2022-04-03T19:49:15.706 INFO:tasks.workunit.client.0.smithi077.stdout:removed './230/27' 2022-04-03T19:49:15.706 INFO:tasks.workunit.client.0.smithi077.stdout:removed './230/31' 2022-04-03T19:49:15.706 INFO:tasks.workunit.client.0.smithi077.stdout:removed './230/36' 2022-04-03T19:49:15.706 INFO:tasks.workunit.client.0.smithi077.stdout:removed './230/20' 2022-04-03T19:49:15.707 INFO:tasks.workunit.client.0.smithi077.stdout:removed './230/39' 2022-04-03T19:49:15.707 INFO:tasks.workunit.client.0.smithi077.stdout:removed './230/34' 2022-04-03T19:49:15.707 INFO:tasks.workunit.client.0.smithi077.stdout:removed './230/38' 2022-04-03T19:49:15.707 INFO:tasks.workunit.client.0.smithi077.stdout:removed './230/28' 2022-04-03T19:49:15.708 INFO:tasks.workunit.client.0.smithi077.stdout:removed './230/24' 2022-04-03T19:49:15.708 INFO:tasks.workunit.client.0.smithi077.stdout:removed './230/29' 2022-04-03T19:49:15.708 INFO:tasks.workunit.client.0.smithi077.stdout:removed './230/32' 2022-04-03T19:49:15.708 INFO:tasks.workunit.client.0.smithi077.stdout:removed './230/22' 2022-04-03T19:49:15.708 INFO:tasks.workunit.client.0.smithi077.stdout:removed './230/25' 2022-04-03T19:49:15.709 INFO:tasks.workunit.client.0.smithi077.stdout:removed './230/30' 2022-04-03T19:49:15.709 INFO:tasks.workunit.client.0.smithi077.stdout:removed './230/33' 2022-04-03T19:49:15.709 INFO:tasks.workunit.client.0.smithi077.stdout:removed './230/23' 2022-04-03T19:49:15.709 INFO:tasks.workunit.client.0.smithi077.stdout:removed './230/35' 2022-04-03T19:49:15.710 INFO:tasks.workunit.client.0.smithi077.stdout:removed './230/26' 2022-04-03T19:49:15.710 INFO:tasks.workunit.client.0.smithi077.stdout:removed './230/37' 2022-04-03T19:49:15.710 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './230' 2022-04-03T19:49:15.710 INFO:tasks.workunit.client.0.smithi077.stdout:removed './231/21' 2022-04-03T19:49:15.710 INFO:tasks.workunit.client.0.smithi077.stdout:removed './231/27' 2022-04-03T19:49:15.711 INFO:tasks.workunit.client.0.smithi077.stdout:removed './231/31' 2022-04-03T19:49:15.711 INFO:tasks.workunit.client.0.smithi077.stdout:removed './231/36' 2022-04-03T19:49:15.711 INFO:tasks.workunit.client.0.smithi077.stdout:removed './231/20' 2022-04-03T19:49:15.711 INFO:tasks.workunit.client.0.smithi077.stdout:removed './231/39' 2022-04-03T19:49:15.712 INFO:tasks.workunit.client.0.smithi077.stdout:removed './231/34' 2022-04-03T19:49:15.712 INFO:tasks.workunit.client.0.smithi077.stdout:removed './231/38' 2022-04-03T19:49:15.712 INFO:tasks.workunit.client.0.smithi077.stdout:removed './231/28' 2022-04-03T19:49:15.712 INFO:tasks.workunit.client.0.smithi077.stdout:removed './231/24' 2022-04-03T19:49:15.712 INFO:tasks.workunit.client.0.smithi077.stdout:removed './231/29' 2022-04-03T19:49:15.713 INFO:tasks.workunit.client.0.smithi077.stdout:removed './231/32' 2022-04-03T19:49:15.713 INFO:tasks.workunit.client.0.smithi077.stdout:removed './231/22' 2022-04-03T19:49:15.713 INFO:tasks.workunit.client.0.smithi077.stdout:removed './231/25' 2022-04-03T19:49:15.713 INFO:tasks.workunit.client.0.smithi077.stdout:removed './231/30' 2022-04-03T19:49:15.713 INFO:tasks.workunit.client.0.smithi077.stdout:removed './231/33' 2022-04-03T19:49:15.714 INFO:tasks.workunit.client.0.smithi077.stdout:removed './231/23' 2022-04-03T19:49:15.714 INFO:tasks.workunit.client.0.smithi077.stdout:removed './231/35' 2022-04-03T19:49:15.714 INFO:tasks.workunit.client.0.smithi077.stdout:removed './231/26' 2022-04-03T19:49:15.714 INFO:tasks.workunit.client.0.smithi077.stdout:removed './231/37' 2022-04-03T19:49:15.715 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './231' 2022-04-03T19:49:15.715 INFO:tasks.workunit.client.0.smithi077.stdout:removed './232/21' 2022-04-03T19:49:15.715 INFO:tasks.workunit.client.0.smithi077.stdout:removed './232/27' 2022-04-03T19:49:15.715 INFO:tasks.workunit.client.0.smithi077.stdout:removed './232/31' 2022-04-03T19:49:15.716 INFO:tasks.workunit.client.0.smithi077.stdout:removed './232/36' 2022-04-03T19:49:15.716 INFO:tasks.workunit.client.0.smithi077.stdout:removed './232/20' 2022-04-03T19:49:15.716 INFO:tasks.workunit.client.0.smithi077.stdout:removed './232/39' 2022-04-03T19:49:15.716 INFO:tasks.workunit.client.0.smithi077.stdout:removed './232/34' 2022-04-03T19:49:15.716 INFO:tasks.workunit.client.0.smithi077.stdout:removed './232/38' 2022-04-03T19:49:15.717 INFO:tasks.workunit.client.0.smithi077.stdout:removed './232/28' 2022-04-03T19:49:15.717 INFO:tasks.workunit.client.0.smithi077.stdout:removed './232/24' 2022-04-03T19:49:15.717 INFO:tasks.workunit.client.0.smithi077.stdout:removed './232/29' 2022-04-03T19:49:15.717 INFO:tasks.workunit.client.0.smithi077.stdout:removed './232/32' 2022-04-03T19:49:15.718 INFO:tasks.workunit.client.0.smithi077.stdout:removed './232/22' 2022-04-03T19:49:15.718 INFO:tasks.workunit.client.0.smithi077.stdout:removed './232/25' 2022-04-03T19:49:15.718 INFO:tasks.workunit.client.0.smithi077.stdout:removed './232/30' 2022-04-03T19:49:15.718 INFO:tasks.workunit.client.0.smithi077.stdout:removed './232/33' 2022-04-03T19:49:15.718 INFO:tasks.workunit.client.0.smithi077.stdout:removed './232/23' 2022-04-03T19:49:15.719 INFO:tasks.workunit.client.0.smithi077.stdout:removed './232/35' 2022-04-03T19:49:15.719 INFO:tasks.workunit.client.0.smithi077.stdout:removed './232/26' 2022-04-03T19:49:15.719 INFO:tasks.workunit.client.0.smithi077.stdout:removed './232/37' 2022-04-03T19:49:15.719 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './232' 2022-04-03T19:49:15.720 INFO:tasks.workunit.client.0.smithi077.stdout:removed './233/21' 2022-04-03T19:49:15.720 INFO:tasks.workunit.client.0.smithi077.stdout:removed './233/27' 2022-04-03T19:49:15.720 INFO:tasks.workunit.client.0.smithi077.stdout:removed './233/31' 2022-04-03T19:49:15.720 INFO:tasks.workunit.client.0.smithi077.stdout:removed './233/36' 2022-04-03T19:49:15.720 INFO:tasks.workunit.client.0.smithi077.stdout:removed './233/20' 2022-04-03T19:49:15.721 INFO:tasks.workunit.client.0.smithi077.stdout:removed './233/39' 2022-04-03T19:49:15.721 INFO:tasks.workunit.client.0.smithi077.stdout:removed './233/34' 2022-04-03T19:49:15.721 INFO:tasks.workunit.client.0.smithi077.stdout:removed './233/38' 2022-04-03T19:49:15.721 INFO:tasks.workunit.client.0.smithi077.stdout:removed './233/28' 2022-04-03T19:49:15.722 INFO:tasks.workunit.client.0.smithi077.stdout:removed './233/24' 2022-04-03T19:49:15.722 INFO:tasks.workunit.client.0.smithi077.stdout:removed './233/29' 2022-04-03T19:49:15.722 INFO:tasks.workunit.client.0.smithi077.stdout:removed './233/32' 2022-04-03T19:49:15.722 INFO:tasks.workunit.client.0.smithi077.stdout:removed './233/22' 2022-04-03T19:49:15.723 INFO:tasks.workunit.client.0.smithi077.stdout:removed './233/25' 2022-04-03T19:49:15.723 INFO:tasks.workunit.client.0.smithi077.stdout:removed './233/30' 2022-04-03T19:49:15.723 INFO:tasks.workunit.client.0.smithi077.stdout:removed './233/33' 2022-04-03T19:49:15.723 INFO:tasks.workunit.client.0.smithi077.stdout:removed './233/23' 2022-04-03T19:49:15.723 INFO:tasks.workunit.client.0.smithi077.stdout:removed './233/35' 2022-04-03T19:49:15.724 INFO:tasks.workunit.client.0.smithi077.stdout:removed './233/26' 2022-04-03T19:49:15.724 INFO:tasks.workunit.client.0.smithi077.stdout:removed './233/37' 2022-04-03T19:49:15.724 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './233' 2022-04-03T19:49:15.724 INFO:tasks.workunit.client.0.smithi077.stdout:removed './234/21' 2022-04-03T19:49:15.725 INFO:tasks.workunit.client.0.smithi077.stdout:removed './234/27' 2022-04-03T19:49:15.725 INFO:tasks.workunit.client.0.smithi077.stdout:removed './234/31' 2022-04-03T19:49:15.725 INFO:tasks.workunit.client.0.smithi077.stdout:removed './234/36' 2022-04-03T19:49:15.725 INFO:tasks.workunit.client.0.smithi077.stdout:removed './234/20' 2022-04-03T19:49:15.726 INFO:tasks.workunit.client.0.smithi077.stdout:removed './234/39' 2022-04-03T19:49:15.726 INFO:tasks.workunit.client.0.smithi077.stdout:removed './234/34' 2022-04-03T19:49:15.726 INFO:tasks.workunit.client.0.smithi077.stdout:removed './234/38' 2022-04-03T19:49:15.727 INFO:tasks.workunit.client.0.smithi077.stdout:removed './234/28' 2022-04-03T19:49:15.727 INFO:tasks.workunit.client.0.smithi077.stdout:removed './234/24' 2022-04-03T19:49:15.727 INFO:tasks.workunit.client.0.smithi077.stdout:removed './234/29' 2022-04-03T19:49:15.727 INFO:tasks.workunit.client.0.smithi077.stdout:removed './234/32' 2022-04-03T19:49:15.727 INFO:tasks.workunit.client.0.smithi077.stdout:removed './234/22' 2022-04-03T19:49:15.728 INFO:tasks.workunit.client.0.smithi077.stdout:removed './234/25' 2022-04-03T19:49:15.728 INFO:tasks.workunit.client.0.smithi077.stdout:removed './234/30' 2022-04-03T19:49:15.728 INFO:tasks.workunit.client.0.smithi077.stdout:removed './234/33' 2022-04-03T19:49:15.728 INFO:tasks.workunit.client.0.smithi077.stdout:removed './234/23' 2022-04-03T19:49:15.729 INFO:tasks.workunit.client.0.smithi077.stdout:removed './234/35' 2022-04-03T19:49:15.729 INFO:tasks.workunit.client.0.smithi077.stdout:removed './234/26' 2022-04-03T19:49:15.729 INFO:tasks.workunit.client.0.smithi077.stdout:removed './234/37' 2022-04-03T19:49:15.729 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './234' 2022-04-03T19:49:15.729 INFO:tasks.workunit.client.0.smithi077.stdout:removed './235/21' 2022-04-03T19:49:15.730 INFO:tasks.workunit.client.0.smithi077.stdout:removed './235/27' 2022-04-03T19:49:15.730 INFO:tasks.workunit.client.0.smithi077.stdout:removed './235/31' 2022-04-03T19:49:15.730 INFO:tasks.workunit.client.0.smithi077.stdout:removed './235/36' 2022-04-03T19:49:15.731 INFO:tasks.workunit.client.0.smithi077.stdout:removed './235/20' 2022-04-03T19:49:15.731 INFO:tasks.workunit.client.0.smithi077.stdout:removed './235/39' 2022-04-03T19:49:15.731 INFO:tasks.workunit.client.0.smithi077.stdout:removed './235/34' 2022-04-03T19:49:15.731 INFO:tasks.workunit.client.0.smithi077.stdout:removed './235/38' 2022-04-03T19:49:15.732 INFO:tasks.workunit.client.0.smithi077.stdout:removed './235/28' 2022-04-03T19:49:15.732 INFO:tasks.workunit.client.0.smithi077.stdout:removed './235/24' 2022-04-03T19:49:15.732 INFO:tasks.workunit.client.0.smithi077.stdout:removed './235/29' 2022-04-03T19:49:15.732 INFO:tasks.workunit.client.0.smithi077.stdout:removed './235/32' 2022-04-03T19:49:15.732 INFO:tasks.workunit.client.0.smithi077.stdout:removed './235/22' 2022-04-03T19:49:15.733 INFO:tasks.workunit.client.0.smithi077.stdout:removed './235/25' 2022-04-03T19:49:15.733 INFO:tasks.workunit.client.0.smithi077.stdout:removed './235/30' 2022-04-03T19:49:15.733 INFO:tasks.workunit.client.0.smithi077.stdout:removed './235/33' 2022-04-03T19:49:15.733 INFO:tasks.workunit.client.0.smithi077.stdout:removed './235/23' 2022-04-03T19:49:15.733 INFO:tasks.workunit.client.0.smithi077.stdout:removed './235/35' 2022-04-03T19:49:15.734 INFO:tasks.workunit.client.0.smithi077.stdout:removed './235/26' 2022-04-03T19:49:15.734 INFO:tasks.workunit.client.0.smithi077.stdout:removed './235/37' 2022-04-03T19:49:15.734 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './235' 2022-04-03T19:49:15.734 INFO:tasks.workunit.client.0.smithi077.stdout:removed './236/21' 2022-04-03T19:49:15.735 INFO:tasks.workunit.client.0.smithi077.stdout:removed './236/27' 2022-04-03T19:49:15.735 INFO:tasks.workunit.client.0.smithi077.stdout:removed './236/31' 2022-04-03T19:49:15.735 INFO:tasks.workunit.client.0.smithi077.stdout:removed './236/36' 2022-04-03T19:49:15.735 INFO:tasks.workunit.client.0.smithi077.stdout:removed './236/20' 2022-04-03T19:49:15.736 INFO:tasks.workunit.client.0.smithi077.stdout:removed './236/39' 2022-04-03T19:49:15.736 INFO:tasks.workunit.client.0.smithi077.stdout:removed './236/34' 2022-04-03T19:49:15.736 INFO:tasks.workunit.client.0.smithi077.stdout:removed './236/38' 2022-04-03T19:49:15.736 INFO:tasks.workunit.client.0.smithi077.stdout:removed './236/28' 2022-04-03T19:49:15.736 INFO:tasks.workunit.client.0.smithi077.stdout:removed './236/24' 2022-04-03T19:49:15.737 INFO:tasks.workunit.client.0.smithi077.stdout:removed './236/29' 2022-04-03T19:49:15.737 INFO:tasks.workunit.client.0.smithi077.stdout:removed './236/32' 2022-04-03T19:49:15.769 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:49:15.773 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:49:18.087 INFO:tasks.workunit.client.0.smithi077.stdout:removed './236/22' 2022-04-03T19:49:18.088 INFO:tasks.workunit.client.0.smithi077.stdout:removed './236/25' 2022-04-03T19:49:18.088 INFO:tasks.workunit.client.0.smithi077.stdout:removed './236/30' 2022-04-03T19:49:18.088 INFO:tasks.workunit.client.0.smithi077.stdout:removed './236/33' 2022-04-03T19:49:18.089 INFO:tasks.workunit.client.0.smithi077.stdout:removed './236/23' 2022-04-03T19:49:18.089 INFO:tasks.workunit.client.0.smithi077.stdout:removed './236/35' 2022-04-03T19:49:18.089 INFO:tasks.workunit.client.0.smithi077.stdout:removed './236/26' 2022-04-03T19:49:18.089 INFO:tasks.workunit.client.0.smithi077.stdout:removed './236/37' 2022-04-03T19:49:18.090 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './236' 2022-04-03T19:49:18.090 INFO:tasks.workunit.client.0.smithi077.stdout:removed './237/21' 2022-04-03T19:49:18.090 INFO:tasks.workunit.client.0.smithi077.stdout:removed './237/27' 2022-04-03T19:49:18.090 INFO:tasks.workunit.client.0.smithi077.stdout:removed './237/31' 2022-04-03T19:49:18.091 INFO:tasks.workunit.client.0.smithi077.stdout:removed './237/36' 2022-04-03T19:49:18.091 INFO:tasks.workunit.client.0.smithi077.stdout:removed './237/20' 2022-04-03T19:49:18.091 INFO:tasks.workunit.client.0.smithi077.stdout:removed './237/39' 2022-04-03T19:49:18.091 INFO:tasks.workunit.client.0.smithi077.stdout:removed './237/34' 2022-04-03T19:49:18.091 INFO:tasks.workunit.client.0.smithi077.stdout:removed './237/38' 2022-04-03T19:49:18.092 INFO:tasks.workunit.client.0.smithi077.stdout:removed './237/28' 2022-04-03T19:49:18.092 INFO:tasks.workunit.client.0.smithi077.stdout:removed './237/24' 2022-04-03T19:49:18.092 INFO:tasks.workunit.client.0.smithi077.stdout:removed './237/29' 2022-04-03T19:49:18.092 INFO:tasks.workunit.client.0.smithi077.stdout:removed './237/32' 2022-04-03T19:49:18.092 INFO:tasks.workunit.client.0.smithi077.stdout:removed './237/22' 2022-04-03T19:49:18.093 INFO:tasks.workunit.client.0.smithi077.stdout:removed './237/25' 2022-04-03T19:49:18.093 INFO:tasks.workunit.client.0.smithi077.stdout:removed './237/30' 2022-04-03T19:49:18.093 INFO:tasks.workunit.client.0.smithi077.stdout:removed './237/33' 2022-04-03T19:49:18.093 INFO:tasks.workunit.client.0.smithi077.stdout:removed './237/23' 2022-04-03T19:49:18.094 INFO:tasks.workunit.client.0.smithi077.stdout:removed './237/35' 2022-04-03T19:49:18.094 INFO:tasks.workunit.client.0.smithi077.stdout:removed './237/26' 2022-04-03T19:49:18.094 INFO:tasks.workunit.client.0.smithi077.stdout:removed './237/37' 2022-04-03T19:49:18.094 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './237' 2022-04-03T19:49:18.094 INFO:tasks.workunit.client.0.smithi077.stdout:removed './238/21' 2022-04-03T19:49:18.095 INFO:tasks.workunit.client.0.smithi077.stdout:removed './238/27' 2022-04-03T19:49:18.095 INFO:tasks.workunit.client.0.smithi077.stdout:removed './238/31' 2022-04-03T19:49:18.095 INFO:tasks.workunit.client.0.smithi077.stdout:removed './238/36' 2022-04-03T19:49:18.095 INFO:tasks.workunit.client.0.smithi077.stdout:removed './238/20' 2022-04-03T19:49:18.096 INFO:tasks.workunit.client.0.smithi077.stdout:removed './238/39' 2022-04-03T19:49:18.096 INFO:tasks.workunit.client.0.smithi077.stdout:removed './238/34' 2022-04-03T19:49:18.096 INFO:tasks.workunit.client.0.smithi077.stdout:removed './238/38' 2022-04-03T19:49:18.096 INFO:tasks.workunit.client.0.smithi077.stdout:removed './238/28' 2022-04-03T19:49:18.096 INFO:tasks.workunit.client.0.smithi077.stdout:removed './238/24' 2022-04-03T19:49:18.097 INFO:tasks.workunit.client.0.smithi077.stdout:removed './238/29' 2022-04-03T19:49:18.097 INFO:tasks.workunit.client.0.smithi077.stdout:removed './238/32' 2022-04-03T19:49:18.097 INFO:tasks.workunit.client.0.smithi077.stdout:removed './238/22' 2022-04-03T19:49:18.097 INFO:tasks.workunit.client.0.smithi077.stdout:removed './238/25' 2022-04-03T19:49:18.098 INFO:tasks.workunit.client.0.smithi077.stdout:removed './238/30' 2022-04-03T19:49:18.098 INFO:tasks.workunit.client.0.smithi077.stdout:removed './238/33' 2022-04-03T19:49:18.098 INFO:tasks.workunit.client.0.smithi077.stdout:removed './238/23' 2022-04-03T19:49:18.098 INFO:tasks.workunit.client.0.smithi077.stdout:removed './238/35' 2022-04-03T19:49:18.098 INFO:tasks.workunit.client.0.smithi077.stdout:removed './238/26' 2022-04-03T19:49:18.099 INFO:tasks.workunit.client.0.smithi077.stdout:removed './238/37' 2022-04-03T19:49:18.099 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './238' 2022-04-03T19:49:18.099 INFO:tasks.workunit.client.0.smithi077.stdout:removed './239/21' 2022-04-03T19:49:18.099 INFO:tasks.workunit.client.0.smithi077.stdout:removed './239/27' 2022-04-03T19:49:18.100 INFO:tasks.workunit.client.0.smithi077.stdout:removed './239/31' 2022-04-03T19:49:18.100 INFO:tasks.workunit.client.0.smithi077.stdout:removed './239/36' 2022-04-03T19:49:18.100 INFO:tasks.workunit.client.0.smithi077.stdout:removed './239/20' 2022-04-03T19:49:18.100 INFO:tasks.workunit.client.0.smithi077.stdout:removed './239/39' 2022-04-03T19:49:18.100 INFO:tasks.workunit.client.0.smithi077.stdout:removed './239/34' 2022-04-03T19:49:18.101 INFO:tasks.workunit.client.0.smithi077.stdout:removed './239/38' 2022-04-03T19:49:18.101 INFO:tasks.workunit.client.0.smithi077.stdout:removed './239/28' 2022-04-03T19:49:18.101 INFO:tasks.workunit.client.0.smithi077.stdout:removed './239/24' 2022-04-03T19:49:18.101 INFO:tasks.workunit.client.0.smithi077.stdout:removed './239/29' 2022-04-03T19:49:18.102 INFO:tasks.workunit.client.0.smithi077.stdout:removed './239/32' 2022-04-03T19:49:18.102 INFO:tasks.workunit.client.0.smithi077.stdout:removed './239/22' 2022-04-03T19:49:18.102 INFO:tasks.workunit.client.0.smithi077.stdout:removed './239/25' 2022-04-03T19:49:18.102 INFO:tasks.workunit.client.0.smithi077.stdout:removed './239/30' 2022-04-03T19:49:18.102 INFO:tasks.workunit.client.0.smithi077.stdout:removed './239/33' 2022-04-03T19:49:18.103 INFO:tasks.workunit.client.0.smithi077.stdout:removed './239/23' 2022-04-03T19:49:18.103 INFO:tasks.workunit.client.0.smithi077.stdout:removed './239/35' 2022-04-03T19:49:18.103 INFO:tasks.workunit.client.0.smithi077.stdout:removed './239/26' 2022-04-03T19:49:18.103 INFO:tasks.workunit.client.0.smithi077.stdout:removed './239/37' 2022-04-03T19:49:18.103 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './239' 2022-04-03T19:49:18.104 INFO:tasks.workunit.client.0.smithi077.stdout:removed './240/21' 2022-04-03T19:49:18.104 INFO:tasks.workunit.client.0.smithi077.stdout:removed './240/27' 2022-04-03T19:49:18.104 INFO:tasks.workunit.client.0.smithi077.stdout:removed './240/31' 2022-04-03T19:49:18.104 INFO:tasks.workunit.client.0.smithi077.stdout:removed './240/36' 2022-04-03T19:49:18.105 INFO:tasks.workunit.client.0.smithi077.stdout:removed './240/20' 2022-04-03T19:49:18.105 INFO:tasks.workunit.client.0.smithi077.stdout:removed './240/39' 2022-04-03T19:49:18.105 INFO:tasks.workunit.client.0.smithi077.stdout:removed './240/34' 2022-04-03T19:49:18.105 INFO:tasks.workunit.client.0.smithi077.stdout:removed './240/38' 2022-04-03T19:49:18.105 INFO:tasks.workunit.client.0.smithi077.stdout:removed './240/28' 2022-04-03T19:49:18.106 INFO:tasks.workunit.client.0.smithi077.stdout:removed './240/24' 2022-04-03T19:49:18.106 INFO:tasks.workunit.client.0.smithi077.stdout:removed './240/29' 2022-04-03T19:49:18.106 INFO:tasks.workunit.client.0.smithi077.stdout:removed './240/32' 2022-04-03T19:49:18.106 INFO:tasks.workunit.client.0.smithi077.stdout:removed './240/22' 2022-04-03T19:49:18.107 INFO:tasks.workunit.client.0.smithi077.stdout:removed './240/25' 2022-04-03T19:49:18.107 INFO:tasks.workunit.client.0.smithi077.stdout:removed './240/30' 2022-04-03T19:49:18.107 INFO:tasks.workunit.client.0.smithi077.stdout:removed './240/33' 2022-04-03T19:49:18.107 INFO:tasks.workunit.client.0.smithi077.stdout:removed './240/23' 2022-04-03T19:49:18.107 INFO:tasks.workunit.client.0.smithi077.stdout:removed './240/35' 2022-04-03T19:49:18.108 INFO:tasks.workunit.client.0.smithi077.stdout:removed './240/26' 2022-04-03T19:49:18.108 INFO:tasks.workunit.client.0.smithi077.stdout:removed './240/37' 2022-04-03T19:49:18.108 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './240' 2022-04-03T19:49:18.108 INFO:tasks.workunit.client.0.smithi077.stdout:removed './241/21' 2022-04-03T19:49:18.109 INFO:tasks.workunit.client.0.smithi077.stdout:removed './241/27' 2022-04-03T19:49:18.109 INFO:tasks.workunit.client.0.smithi077.stdout:removed './241/31' 2022-04-03T19:49:18.109 INFO:tasks.workunit.client.0.smithi077.stdout:removed './241/36' 2022-04-03T19:49:18.109 INFO:tasks.workunit.client.0.smithi077.stdout:removed './241/20' 2022-04-03T19:49:18.110 INFO:tasks.workunit.client.0.smithi077.stdout:removed './241/39' 2022-04-03T19:49:18.110 INFO:tasks.workunit.client.0.smithi077.stdout:removed './241/34' 2022-04-03T19:49:18.110 INFO:tasks.workunit.client.0.smithi077.stdout:removed './241/38' 2022-04-03T19:49:18.110 INFO:tasks.workunit.client.0.smithi077.stdout:removed './241/28' 2022-04-03T19:49:18.111 INFO:tasks.workunit.client.0.smithi077.stdout:removed './241/24' 2022-04-03T19:49:18.111 INFO:tasks.workunit.client.0.smithi077.stdout:removed './241/29' 2022-04-03T19:49:18.111 INFO:tasks.workunit.client.0.smithi077.stdout:removed './241/32' 2022-04-03T19:49:18.111 INFO:tasks.workunit.client.0.smithi077.stdout:removed './241/22' 2022-04-03T19:49:18.111 INFO:tasks.workunit.client.0.smithi077.stdout:removed './241/25' 2022-04-03T19:49:18.112 INFO:tasks.workunit.client.0.smithi077.stdout:removed './241/30' 2022-04-03T19:49:18.112 INFO:tasks.workunit.client.0.smithi077.stdout:removed './241/33' 2022-04-03T19:49:18.112 INFO:tasks.workunit.client.0.smithi077.stdout:removed './241/23' 2022-04-03T19:49:18.112 INFO:tasks.workunit.client.0.smithi077.stdout:removed './241/35' 2022-04-03T19:49:18.113 INFO:tasks.workunit.client.0.smithi077.stdout:removed './241/26' 2022-04-03T19:49:18.113 INFO:tasks.workunit.client.0.smithi077.stdout:removed './241/37' 2022-04-03T19:49:18.113 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './241' 2022-04-03T19:49:18.113 INFO:tasks.workunit.client.0.smithi077.stdout:removed './242/21' 2022-04-03T19:49:18.113 INFO:tasks.workunit.client.0.smithi077.stdout:removed './242/27' 2022-04-03T19:49:18.114 INFO:tasks.workunit.client.0.smithi077.stdout:removed './242/31' 2022-04-03T19:49:18.114 INFO:tasks.workunit.client.0.smithi077.stdout:removed './242/36' 2022-04-03T19:49:18.114 INFO:tasks.workunit.client.0.smithi077.stdout:removed './242/20' 2022-04-03T19:49:18.114 INFO:tasks.workunit.client.0.smithi077.stdout:removed './242/39' 2022-04-03T19:49:18.114 INFO:tasks.workunit.client.0.smithi077.stdout:removed './242/34' 2022-04-03T19:49:18.115 INFO:tasks.workunit.client.0.smithi077.stdout:removed './242/38' 2022-04-03T19:49:18.115 INFO:tasks.workunit.client.0.smithi077.stdout:removed './242/28' 2022-04-03T19:49:18.115 INFO:tasks.workunit.client.0.smithi077.stdout:removed './242/24' 2022-04-03T19:49:18.115 INFO:tasks.workunit.client.0.smithi077.stdout:removed './242/29' 2022-04-03T19:49:18.115 INFO:tasks.workunit.client.0.smithi077.stdout:removed './242/32' 2022-04-03T19:49:18.116 INFO:tasks.workunit.client.0.smithi077.stdout:removed './242/22' 2022-04-03T19:49:18.116 INFO:tasks.workunit.client.0.smithi077.stdout:removed './242/25' 2022-04-03T19:49:18.116 INFO:tasks.workunit.client.0.smithi077.stdout:removed './242/30' 2022-04-03T19:49:18.116 INFO:tasks.workunit.client.0.smithi077.stdout:removed './242/33' 2022-04-03T19:49:18.117 INFO:tasks.workunit.client.0.smithi077.stdout:removed './242/23' 2022-04-03T19:49:18.117 INFO:tasks.workunit.client.0.smithi077.stdout:removed './242/35' 2022-04-03T19:49:18.117 INFO:tasks.workunit.client.0.smithi077.stdout:removed './242/26' 2022-04-03T19:49:18.117 INFO:tasks.workunit.client.0.smithi077.stdout:removed './242/37' 2022-04-03T19:49:18.117 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './242' 2022-04-03T19:49:18.118 INFO:tasks.workunit.client.0.smithi077.stdout:removed './243/21' 2022-04-03T19:49:18.118 INFO:tasks.workunit.client.0.smithi077.stdout:removed './243/27' 2022-04-03T19:49:18.118 INFO:tasks.workunit.client.0.smithi077.stdout:removed './243/31' 2022-04-03T19:49:18.118 INFO:tasks.workunit.client.0.smithi077.stdout:removed './243/36' 2022-04-03T19:49:18.119 INFO:tasks.workunit.client.0.smithi077.stdout:removed './243/20' 2022-04-03T19:49:18.119 INFO:tasks.workunit.client.0.smithi077.stdout:removed './243/39' 2022-04-03T19:49:18.120 INFO:tasks.workunit.client.0.smithi077.stdout:removed './243/34' 2022-04-03T19:49:18.120 INFO:tasks.workunit.client.0.smithi077.stdout:removed './243/38' 2022-04-03T19:49:18.120 INFO:tasks.workunit.client.0.smithi077.stdout:removed './243/28' 2022-04-03T19:49:18.120 INFO:tasks.workunit.client.0.smithi077.stdout:removed './243/24' 2022-04-03T19:49:18.121 INFO:tasks.workunit.client.0.smithi077.stdout:removed './243/29' 2022-04-03T19:49:18.121 INFO:tasks.workunit.client.0.smithi077.stdout:removed './243/32' 2022-04-03T19:49:18.121 INFO:tasks.workunit.client.0.smithi077.stdout:removed './243/22' 2022-04-03T19:49:18.121 INFO:tasks.workunit.client.0.smithi077.stdout:removed './243/25' 2022-04-03T19:49:18.121 INFO:tasks.workunit.client.0.smithi077.stdout:removed './243/30' 2022-04-03T19:49:18.122 INFO:tasks.workunit.client.0.smithi077.stdout:removed './243/33' 2022-04-03T19:49:18.122 INFO:tasks.workunit.client.0.smithi077.stdout:removed './243/23' 2022-04-03T19:49:18.122 INFO:tasks.workunit.client.0.smithi077.stdout:removed './243/35' 2022-04-03T19:49:18.122 INFO:tasks.workunit.client.0.smithi077.stdout:removed './243/26' 2022-04-03T19:49:18.122 INFO:tasks.workunit.client.0.smithi077.stdout:removed './243/37' 2022-04-03T19:49:18.123 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './243' 2022-04-03T19:49:18.123 INFO:tasks.workunit.client.0.smithi077.stdout:removed './244/21' 2022-04-03T19:49:18.123 INFO:tasks.workunit.client.0.smithi077.stdout:removed './244/27' 2022-04-03T19:49:18.123 INFO:tasks.workunit.client.0.smithi077.stdout:removed './244/31' 2022-04-03T19:49:18.124 INFO:tasks.workunit.client.0.smithi077.stdout:removed './244/36' 2022-04-03T19:49:18.124 INFO:tasks.workunit.client.0.smithi077.stdout:removed './244/20' 2022-04-03T19:49:18.124 INFO:tasks.workunit.client.0.smithi077.stdout:removed './244/39' 2022-04-03T19:49:18.124 INFO:tasks.workunit.client.0.smithi077.stdout:removed './244/34' 2022-04-03T19:49:18.124 INFO:tasks.workunit.client.0.smithi077.stdout:removed './244/38' 2022-04-03T19:49:18.125 INFO:tasks.workunit.client.0.smithi077.stdout:removed './244/28' 2022-04-03T19:49:18.125 INFO:tasks.workunit.client.0.smithi077.stdout:removed './244/24' 2022-04-03T19:49:18.125 INFO:tasks.workunit.client.0.smithi077.stdout:removed './244/29' 2022-04-03T19:49:18.125 INFO:tasks.workunit.client.0.smithi077.stdout:removed './244/32' 2022-04-03T19:49:18.126 INFO:tasks.workunit.client.0.smithi077.stdout:removed './244/22' 2022-04-03T19:49:18.126 INFO:tasks.workunit.client.0.smithi077.stdout:removed './244/25' 2022-04-03T19:49:18.126 INFO:tasks.workunit.client.0.smithi077.stdout:removed './244/30' 2022-04-03T19:49:18.126 INFO:tasks.workunit.client.0.smithi077.stdout:removed './244/33' 2022-04-03T19:49:18.126 INFO:tasks.workunit.client.0.smithi077.stdout:removed './244/23' 2022-04-03T19:49:18.127 INFO:tasks.workunit.client.0.smithi077.stdout:removed './244/35' 2022-04-03T19:49:18.127 INFO:tasks.workunit.client.0.smithi077.stdout:removed './244/26' 2022-04-03T19:49:18.127 INFO:tasks.workunit.client.0.smithi077.stdout:removed './244/37' 2022-04-03T19:49:18.127 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './244' 2022-04-03T19:49:18.127 INFO:tasks.workunit.client.0.smithi077.stdout:removed './245/21' 2022-04-03T19:49:18.128 INFO:tasks.workunit.client.0.smithi077.stdout:removed './245/27' 2022-04-03T19:49:18.128 INFO:tasks.workunit.client.0.smithi077.stdout:removed './245/31' 2022-04-03T19:49:18.128 INFO:tasks.workunit.client.0.smithi077.stdout:removed './245/36' 2022-04-03T19:49:18.128 INFO:tasks.workunit.client.0.smithi077.stdout:removed './245/20' 2022-04-03T19:49:18.129 INFO:tasks.workunit.client.0.smithi077.stdout:removed './245/39' 2022-04-03T19:49:18.129 INFO:tasks.workunit.client.0.smithi077.stdout:removed './245/34' 2022-04-03T19:49:18.129 INFO:tasks.workunit.client.0.smithi077.stdout:removed './245/38' 2022-04-03T19:49:18.129 INFO:tasks.workunit.client.0.smithi077.stdout:removed './245/28' 2022-04-03T19:49:18.129 INFO:tasks.workunit.client.0.smithi077.stdout:removed './245/24' 2022-04-03T19:49:18.130 INFO:tasks.workunit.client.0.smithi077.stdout:removed './245/29' 2022-04-03T19:49:18.130 INFO:tasks.workunit.client.0.smithi077.stdout:removed './245/32' 2022-04-03T19:49:18.130 INFO:tasks.workunit.client.0.smithi077.stdout:removed './245/22' 2022-04-03T19:49:18.130 INFO:tasks.workunit.client.0.smithi077.stdout:removed './245/25' 2022-04-03T19:49:18.130 INFO:tasks.workunit.client.0.smithi077.stdout:removed './245/30' 2022-04-03T19:49:18.131 INFO:tasks.workunit.client.0.smithi077.stdout:removed './245/33' 2022-04-03T19:49:18.131 INFO:tasks.workunit.client.0.smithi077.stdout:removed './245/23' 2022-04-03T19:49:18.131 INFO:tasks.workunit.client.0.smithi077.stdout:removed './245/35' 2022-04-03T19:49:18.131 INFO:tasks.workunit.client.0.smithi077.stdout:removed './245/26' 2022-04-03T19:49:18.132 INFO:tasks.workunit.client.0.smithi077.stdout:removed './245/37' 2022-04-03T19:49:18.132 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './245' 2022-04-03T19:49:18.132 INFO:tasks.workunit.client.0.smithi077.stdout:removed './246/21' 2022-04-03T19:49:18.132 INFO:tasks.workunit.client.0.smithi077.stdout:removed './246/27' 2022-04-03T19:49:18.132 INFO:tasks.workunit.client.0.smithi077.stdout:removed './246/31' 2022-04-03T19:49:18.133 INFO:tasks.workunit.client.0.smithi077.stdout:removed './246/36' 2022-04-03T19:49:18.133 INFO:tasks.workunit.client.0.smithi077.stdout:removed './246/20' 2022-04-03T19:49:18.133 INFO:tasks.workunit.client.0.smithi077.stdout:removed './246/39' 2022-04-03T19:49:18.133 INFO:tasks.workunit.client.0.smithi077.stdout:removed './246/34' 2022-04-03T19:49:18.134 INFO:tasks.workunit.client.0.smithi077.stdout:removed './246/38' 2022-04-03T19:49:18.134 INFO:tasks.workunit.client.0.smithi077.stdout:removed './246/28' 2022-04-03T19:49:18.134 INFO:tasks.workunit.client.0.smithi077.stdout:removed './246/24' 2022-04-03T19:49:18.134 INFO:tasks.workunit.client.0.smithi077.stdout:removed './246/29' 2022-04-03T19:49:18.134 INFO:tasks.workunit.client.0.smithi077.stdout:removed './246/32' 2022-04-03T19:49:18.135 INFO:tasks.workunit.client.0.smithi077.stdout:removed './246/22' 2022-04-03T19:49:20.625 INFO:tasks.workunit.client.0.smithi077.stdout:removed './246/25' 2022-04-03T19:49:20.626 INFO:tasks.workunit.client.0.smithi077.stdout:removed './246/30' 2022-04-03T19:49:20.626 INFO:tasks.workunit.client.0.smithi077.stdout:removed './246/33' 2022-04-03T19:49:20.626 INFO:tasks.workunit.client.0.smithi077.stdout:removed './246/23' 2022-04-03T19:49:20.626 INFO:tasks.workunit.client.0.smithi077.stdout:removed './246/35' 2022-04-03T19:49:20.627 INFO:tasks.workunit.client.0.smithi077.stdout:removed './246/26' 2022-04-03T19:49:20.627 INFO:tasks.workunit.client.0.smithi077.stdout:removed './246/37' 2022-04-03T19:49:20.627 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './246' 2022-04-03T19:49:20.627 INFO:tasks.workunit.client.0.smithi077.stdout:removed './247/21' 2022-04-03T19:49:20.628 INFO:tasks.workunit.client.0.smithi077.stdout:removed './247/27' 2022-04-03T19:49:20.628 INFO:tasks.workunit.client.0.smithi077.stdout:removed './247/31' 2022-04-03T19:49:20.628 INFO:tasks.workunit.client.0.smithi077.stdout:removed './247/36' 2022-04-03T19:49:20.628 INFO:tasks.workunit.client.0.smithi077.stdout:removed './247/20' 2022-04-03T19:49:20.628 INFO:tasks.workunit.client.0.smithi077.stdout:removed './247/39' 2022-04-03T19:49:20.629 INFO:tasks.workunit.client.0.smithi077.stdout:removed './247/34' 2022-04-03T19:49:20.629 INFO:tasks.workunit.client.0.smithi077.stdout:removed './247/38' 2022-04-03T19:49:20.629 INFO:tasks.workunit.client.0.smithi077.stdout:removed './247/28' 2022-04-03T19:49:20.629 INFO:tasks.workunit.client.0.smithi077.stdout:removed './247/24' 2022-04-03T19:49:20.630 INFO:tasks.workunit.client.0.smithi077.stdout:removed './247/29' 2022-04-03T19:49:20.630 INFO:tasks.workunit.client.0.smithi077.stdout:removed './247/32' 2022-04-03T19:49:20.630 INFO:tasks.workunit.client.0.smithi077.stdout:removed './247/22' 2022-04-03T19:49:20.630 INFO:tasks.workunit.client.0.smithi077.stdout:removed './247/25' 2022-04-03T19:49:20.630 INFO:tasks.workunit.client.0.smithi077.stdout:removed './247/30' 2022-04-03T19:49:20.631 INFO:tasks.workunit.client.0.smithi077.stdout:removed './247/33' 2022-04-03T19:49:20.631 INFO:tasks.workunit.client.0.smithi077.stdout:removed './247/23' 2022-04-03T19:49:20.631 INFO:tasks.workunit.client.0.smithi077.stdout:removed './247/35' 2022-04-03T19:49:20.631 INFO:tasks.workunit.client.0.smithi077.stdout:removed './247/26' 2022-04-03T19:49:20.631 INFO:tasks.workunit.client.0.smithi077.stdout:removed './247/37' 2022-04-03T19:49:20.632 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './247' 2022-04-03T19:49:20.632 INFO:tasks.workunit.client.0.smithi077.stdout:removed './248/21' 2022-04-03T19:49:20.632 INFO:tasks.workunit.client.0.smithi077.stdout:removed './248/27' 2022-04-03T19:49:20.632 INFO:tasks.workunit.client.0.smithi077.stdout:removed './248/31' 2022-04-03T19:49:20.632 INFO:tasks.workunit.client.0.smithi077.stdout:removed './248/36' 2022-04-03T19:49:20.633 INFO:tasks.workunit.client.0.smithi077.stdout:removed './248/20' 2022-04-03T19:49:20.633 INFO:tasks.workunit.client.0.smithi077.stdout:removed './248/39' 2022-04-03T19:49:20.633 INFO:tasks.workunit.client.0.smithi077.stdout:removed './248/34' 2022-04-03T19:49:20.633 INFO:tasks.workunit.client.0.smithi077.stdout:removed './248/38' 2022-04-03T19:49:20.634 INFO:tasks.workunit.client.0.smithi077.stdout:removed './248/28' 2022-04-03T19:49:20.634 INFO:tasks.workunit.client.0.smithi077.stdout:removed './248/24' 2022-04-03T19:49:20.634 INFO:tasks.workunit.client.0.smithi077.stdout:removed './248/29' 2022-04-03T19:49:20.634 INFO:tasks.workunit.client.0.smithi077.stdout:removed './248/32' 2022-04-03T19:49:20.634 INFO:tasks.workunit.client.0.smithi077.stdout:removed './248/22' 2022-04-03T19:49:20.635 INFO:tasks.workunit.client.0.smithi077.stdout:removed './248/25' 2022-04-03T19:49:20.635 INFO:tasks.workunit.client.0.smithi077.stdout:removed './248/30' 2022-04-03T19:49:20.635 INFO:tasks.workunit.client.0.smithi077.stdout:removed './248/33' 2022-04-03T19:49:20.635 INFO:tasks.workunit.client.0.smithi077.stdout:removed './248/23' 2022-04-03T19:49:20.636 INFO:tasks.workunit.client.0.smithi077.stdout:removed './248/35' 2022-04-03T19:49:20.636 INFO:tasks.workunit.client.0.smithi077.stdout:removed './248/26' 2022-04-03T19:49:20.636 INFO:tasks.workunit.client.0.smithi077.stdout:removed './248/37' 2022-04-03T19:49:20.636 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './248' 2022-04-03T19:49:20.636 INFO:tasks.workunit.client.0.smithi077.stdout:removed './249/21' 2022-04-03T19:49:20.637 INFO:tasks.workunit.client.0.smithi077.stdout:removed './249/27' 2022-04-03T19:49:20.637 INFO:tasks.workunit.client.0.smithi077.stdout:removed './249/31' 2022-04-03T19:49:20.637 INFO:tasks.workunit.client.0.smithi077.stdout:removed './249/36' 2022-04-03T19:49:20.637 INFO:tasks.workunit.client.0.smithi077.stdout:removed './249/20' 2022-04-03T19:49:20.637 INFO:tasks.workunit.client.0.smithi077.stdout:removed './249/39' 2022-04-03T19:49:20.638 INFO:tasks.workunit.client.0.smithi077.stdout:removed './249/34' 2022-04-03T19:49:20.638 INFO:tasks.workunit.client.0.smithi077.stdout:removed './249/38' 2022-04-03T19:49:20.638 INFO:tasks.workunit.client.0.smithi077.stdout:removed './249/28' 2022-04-03T19:49:20.638 INFO:tasks.workunit.client.0.smithi077.stdout:removed './249/24' 2022-04-03T19:49:20.638 INFO:tasks.workunit.client.0.smithi077.stdout:removed './249/29' 2022-04-03T19:49:20.639 INFO:tasks.workunit.client.0.smithi077.stdout:removed './249/32' 2022-04-03T19:49:20.639 INFO:tasks.workunit.client.0.smithi077.stdout:removed './249/22' 2022-04-03T19:49:20.639 INFO:tasks.workunit.client.0.smithi077.stdout:removed './249/25' 2022-04-03T19:49:20.639 INFO:tasks.workunit.client.0.smithi077.stdout:removed './249/30' 2022-04-03T19:49:20.640 INFO:tasks.workunit.client.0.smithi077.stdout:removed './249/33' 2022-04-03T19:49:20.640 INFO:tasks.workunit.client.0.smithi077.stdout:removed './249/23' 2022-04-03T19:49:20.640 INFO:tasks.workunit.client.0.smithi077.stdout:removed './249/35' 2022-04-03T19:49:20.640 INFO:tasks.workunit.client.0.smithi077.stdout:removed './249/26' 2022-04-03T19:49:20.640 INFO:tasks.workunit.client.0.smithi077.stdout:removed './249/37' 2022-04-03T19:49:20.641 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './249' 2022-04-03T19:49:20.641 INFO:tasks.workunit.client.0.smithi077.stdout:removed './250/21' 2022-04-03T19:49:20.641 INFO:tasks.workunit.client.0.smithi077.stdout:removed './250/27' 2022-04-03T19:49:20.641 INFO:tasks.workunit.client.0.smithi077.stdout:removed './250/31' 2022-04-03T19:49:20.642 INFO:tasks.workunit.client.0.smithi077.stdout:removed './250/36' 2022-04-03T19:49:20.642 INFO:tasks.workunit.client.0.smithi077.stdout:removed './250/20' 2022-04-03T19:49:20.642 INFO:tasks.workunit.client.0.smithi077.stdout:removed './250/39' 2022-04-03T19:49:20.642 INFO:tasks.workunit.client.0.smithi077.stdout:removed './250/34' 2022-04-03T19:49:20.642 INFO:tasks.workunit.client.0.smithi077.stdout:removed './250/38' 2022-04-03T19:49:20.643 INFO:tasks.workunit.client.0.smithi077.stdout:removed './250/28' 2022-04-03T19:49:20.643 INFO:tasks.workunit.client.0.smithi077.stdout:removed './250/24' 2022-04-03T19:49:20.643 INFO:tasks.workunit.client.0.smithi077.stdout:removed './250/29' 2022-04-03T19:49:20.643 INFO:tasks.workunit.client.0.smithi077.stdout:removed './250/32' 2022-04-03T19:49:20.643 INFO:tasks.workunit.client.0.smithi077.stdout:removed './250/22' 2022-04-03T19:49:20.643 INFO:tasks.workunit.client.0.smithi077.stdout:removed './250/25' 2022-04-03T19:49:20.644 INFO:tasks.workunit.client.0.smithi077.stdout:removed './250/30' 2022-04-03T19:49:20.644 INFO:tasks.workunit.client.0.smithi077.stdout:removed './250/33' 2022-04-03T19:49:20.644 INFO:tasks.workunit.client.0.smithi077.stdout:removed './250/23' 2022-04-03T19:49:20.644 INFO:tasks.workunit.client.0.smithi077.stdout:removed './250/35' 2022-04-03T19:49:20.644 INFO:tasks.workunit.client.0.smithi077.stdout:removed './250/26' 2022-04-03T19:49:20.645 INFO:tasks.workunit.client.0.smithi077.stdout:removed './250/37' 2022-04-03T19:49:20.645 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './250' 2022-04-03T19:49:20.645 INFO:tasks.workunit.client.0.smithi077.stdout:removed './251/21' 2022-04-03T19:49:20.645 INFO:tasks.workunit.client.0.smithi077.stdout:removed './251/27' 2022-04-03T19:49:20.645 INFO:tasks.workunit.client.0.smithi077.stdout:removed './251/31' 2022-04-03T19:49:20.646 INFO:tasks.workunit.client.0.smithi077.stdout:removed './251/36' 2022-04-03T19:49:20.646 INFO:tasks.workunit.client.0.smithi077.stdout:removed './251/20' 2022-04-03T19:49:20.646 INFO:tasks.workunit.client.0.smithi077.stdout:removed './251/39' 2022-04-03T19:49:20.646 INFO:tasks.workunit.client.0.smithi077.stdout:removed './251/34' 2022-04-03T19:49:20.647 INFO:tasks.workunit.client.0.smithi077.stdout:removed './251/38' 2022-04-03T19:49:20.647 INFO:tasks.workunit.client.0.smithi077.stdout:removed './251/28' 2022-04-03T19:49:20.647 INFO:tasks.workunit.client.0.smithi077.stdout:removed './251/24' 2022-04-03T19:49:20.647 INFO:tasks.workunit.client.0.smithi077.stdout:removed './251/29' 2022-04-03T19:49:20.647 INFO:tasks.workunit.client.0.smithi077.stdout:removed './251/32' 2022-04-03T19:49:20.648 INFO:tasks.workunit.client.0.smithi077.stdout:removed './251/22' 2022-04-03T19:49:20.648 INFO:tasks.workunit.client.0.smithi077.stdout:removed './251/25' 2022-04-03T19:49:20.648 INFO:tasks.workunit.client.0.smithi077.stdout:removed './251/30' 2022-04-03T19:49:20.648 INFO:tasks.workunit.client.0.smithi077.stdout:removed './251/33' 2022-04-03T19:49:20.648 INFO:tasks.workunit.client.0.smithi077.stdout:removed './251/23' 2022-04-03T19:49:20.649 INFO:tasks.workunit.client.0.smithi077.stdout:removed './251/35' 2022-04-03T19:49:20.649 INFO:tasks.workunit.client.0.smithi077.stdout:removed './251/26' 2022-04-03T19:49:20.649 INFO:tasks.workunit.client.0.smithi077.stdout:removed './251/37' 2022-04-03T19:49:20.649 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './251' 2022-04-03T19:49:20.649 INFO:tasks.workunit.client.0.smithi077.stdout:removed './252/21' 2022-04-03T19:49:20.650 INFO:tasks.workunit.client.0.smithi077.stdout:removed './252/27' 2022-04-03T19:49:20.650 INFO:tasks.workunit.client.0.smithi077.stdout:removed './252/31' 2022-04-03T19:49:20.650 INFO:tasks.workunit.client.0.smithi077.stdout:removed './252/36' 2022-04-03T19:49:20.650 INFO:tasks.workunit.client.0.smithi077.stdout:removed './252/20' 2022-04-03T19:49:20.650 INFO:tasks.workunit.client.0.smithi077.stdout:removed './252/39' 2022-04-03T19:49:20.651 INFO:tasks.workunit.client.0.smithi077.stdout:removed './252/34' 2022-04-03T19:49:20.651 INFO:tasks.workunit.client.0.smithi077.stdout:removed './252/38' 2022-04-03T19:49:20.651 INFO:tasks.workunit.client.0.smithi077.stdout:removed './252/28' 2022-04-03T19:49:20.651 INFO:tasks.workunit.client.0.smithi077.stdout:removed './252/24' 2022-04-03T19:49:20.652 INFO:tasks.workunit.client.0.smithi077.stdout:removed './252/29' 2022-04-03T19:49:20.652 INFO:tasks.workunit.client.0.smithi077.stdout:removed './252/32' 2022-04-03T19:49:20.652 INFO:tasks.workunit.client.0.smithi077.stdout:removed './252/22' 2022-04-03T19:49:20.652 INFO:tasks.workunit.client.0.smithi077.stdout:removed './252/25' 2022-04-03T19:49:20.652 INFO:tasks.workunit.client.0.smithi077.stdout:removed './252/30' 2022-04-03T19:49:20.653 INFO:tasks.workunit.client.0.smithi077.stdout:removed './252/33' 2022-04-03T19:49:20.653 INFO:tasks.workunit.client.0.smithi077.stdout:removed './252/23' 2022-04-03T19:49:20.653 INFO:tasks.workunit.client.0.smithi077.stdout:removed './252/35' 2022-04-03T19:49:20.653 INFO:tasks.workunit.client.0.smithi077.stdout:removed './252/26' 2022-04-03T19:49:20.653 INFO:tasks.workunit.client.0.smithi077.stdout:removed './252/37' 2022-04-03T19:49:20.653 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './252' 2022-04-03T19:49:20.654 INFO:tasks.workunit.client.0.smithi077.stdout:removed './253/21' 2022-04-03T19:49:20.654 INFO:tasks.workunit.client.0.smithi077.stdout:removed './253/27' 2022-04-03T19:49:20.654 INFO:tasks.workunit.client.0.smithi077.stdout:removed './253/31' 2022-04-03T19:49:20.654 INFO:tasks.workunit.client.0.smithi077.stdout:removed './253/36' 2022-04-03T19:49:20.655 INFO:tasks.workunit.client.0.smithi077.stdout:removed './253/20' 2022-04-03T19:49:20.655 INFO:tasks.workunit.client.0.smithi077.stdout:removed './253/39' 2022-04-03T19:49:20.655 INFO:tasks.workunit.client.0.smithi077.stdout:removed './253/34' 2022-04-03T19:49:20.655 INFO:tasks.workunit.client.0.smithi077.stdout:removed './253/38' 2022-04-03T19:49:20.655 INFO:tasks.workunit.client.0.smithi077.stdout:removed './253/28' 2022-04-03T19:49:20.656 INFO:tasks.workunit.client.0.smithi077.stdout:removed './253/24' 2022-04-03T19:49:20.656 INFO:tasks.workunit.client.0.smithi077.stdout:removed './253/29' 2022-04-03T19:49:20.656 INFO:tasks.workunit.client.0.smithi077.stdout:removed './253/32' 2022-04-03T19:49:20.656 INFO:tasks.workunit.client.0.smithi077.stdout:removed './253/22' 2022-04-03T19:49:20.656 INFO:tasks.workunit.client.0.smithi077.stdout:removed './253/25' 2022-04-03T19:49:20.657 INFO:tasks.workunit.client.0.smithi077.stdout:removed './253/30' 2022-04-03T19:49:20.657 INFO:tasks.workunit.client.0.smithi077.stdout:removed './253/33' 2022-04-03T19:49:20.657 INFO:tasks.workunit.client.0.smithi077.stdout:removed './253/23' 2022-04-03T19:49:20.657 INFO:tasks.workunit.client.0.smithi077.stdout:removed './253/35' 2022-04-03T19:49:20.658 INFO:tasks.workunit.client.0.smithi077.stdout:removed './253/26' 2022-04-03T19:49:20.658 INFO:tasks.workunit.client.0.smithi077.stdout:removed './253/37' 2022-04-03T19:49:20.658 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './253' 2022-04-03T19:49:20.658 INFO:tasks.workunit.client.0.smithi077.stdout:removed './254/21' 2022-04-03T19:49:20.658 INFO:tasks.workunit.client.0.smithi077.stdout:removed './254/27' 2022-04-03T19:49:20.659 INFO:tasks.workunit.client.0.smithi077.stdout:removed './254/31' 2022-04-03T19:49:20.659 INFO:tasks.workunit.client.0.smithi077.stdout:removed './254/36' 2022-04-03T19:49:20.659 INFO:tasks.workunit.client.0.smithi077.stdout:removed './254/20' 2022-04-03T19:49:20.659 INFO:tasks.workunit.client.0.smithi077.stdout:removed './254/39' 2022-04-03T19:49:20.659 INFO:tasks.workunit.client.0.smithi077.stdout:removed './254/34' 2022-04-03T19:49:20.660 INFO:tasks.workunit.client.0.smithi077.stdout:removed './254/38' 2022-04-03T19:49:20.660 INFO:tasks.workunit.client.0.smithi077.stdout:removed './254/28' 2022-04-03T19:49:20.660 INFO:tasks.workunit.client.0.smithi077.stdout:removed './254/24' 2022-04-03T19:49:20.660 INFO:tasks.workunit.client.0.smithi077.stdout:removed './254/29' 2022-04-03T19:49:20.661 INFO:tasks.workunit.client.0.smithi077.stdout:removed './254/32' 2022-04-03T19:49:20.661 INFO:tasks.workunit.client.0.smithi077.stdout:removed './254/22' 2022-04-03T19:49:20.661 INFO:tasks.workunit.client.0.smithi077.stdout:removed './254/25' 2022-04-03T19:49:20.661 INFO:tasks.workunit.client.0.smithi077.stdout:removed './254/30' 2022-04-03T19:49:20.661 INFO:tasks.workunit.client.0.smithi077.stdout:removed './254/33' 2022-04-03T19:49:20.662 INFO:tasks.workunit.client.0.smithi077.stdout:removed './254/23' 2022-04-03T19:49:20.662 INFO:tasks.workunit.client.0.smithi077.stdout:removed './254/35' 2022-04-03T19:49:20.662 INFO:tasks.workunit.client.0.smithi077.stdout:removed './254/26' 2022-04-03T19:49:20.662 INFO:tasks.workunit.client.0.smithi077.stdout:removed './254/37' 2022-04-03T19:49:20.662 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './254' 2022-04-03T19:49:20.663 INFO:tasks.workunit.client.0.smithi077.stdout:removed './255/21' 2022-04-03T19:49:20.663 INFO:tasks.workunit.client.0.smithi077.stdout:removed './255/27' 2022-04-03T19:49:20.663 INFO:tasks.workunit.client.0.smithi077.stdout:removed './255/31' 2022-04-03T19:49:20.663 INFO:tasks.workunit.client.0.smithi077.stdout:removed './255/36' 2022-04-03T19:49:20.663 INFO:tasks.workunit.client.0.smithi077.stdout:removed './255/20' 2022-04-03T19:49:20.664 INFO:tasks.workunit.client.0.smithi077.stdout:removed './255/39' 2022-04-03T19:49:20.664 INFO:tasks.workunit.client.0.smithi077.stdout:removed './255/34' 2022-04-03T19:49:20.664 INFO:tasks.workunit.client.0.smithi077.stdout:removed './255/38' 2022-04-03T19:49:20.664 INFO:tasks.workunit.client.0.smithi077.stdout:removed './255/28' 2022-04-03T19:49:20.665 INFO:tasks.workunit.client.0.smithi077.stdout:removed './255/24' 2022-04-03T19:49:20.665 INFO:tasks.workunit.client.0.smithi077.stdout:removed './255/29' 2022-04-03T19:49:20.665 INFO:tasks.workunit.client.0.smithi077.stdout:removed './255/32' 2022-04-03T19:49:20.665 INFO:tasks.workunit.client.0.smithi077.stdout:removed './255/22' 2022-04-03T19:49:20.665 INFO:tasks.workunit.client.0.smithi077.stdout:removed './255/25' 2022-04-03T19:49:20.666 INFO:tasks.workunit.client.0.smithi077.stdout:removed './255/30' 2022-04-03T19:49:20.666 INFO:tasks.workunit.client.0.smithi077.stdout:removed './255/33' 2022-04-03T19:49:20.666 INFO:tasks.workunit.client.0.smithi077.stdout:removed './255/23' 2022-04-03T19:49:20.666 INFO:tasks.workunit.client.0.smithi077.stdout:removed './255/35' 2022-04-03T19:49:20.666 INFO:tasks.workunit.client.0.smithi077.stdout:removed './255/26' 2022-04-03T19:49:20.666 INFO:tasks.workunit.client.0.smithi077.stdout:removed './255/37' 2022-04-03T19:49:20.667 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './255' 2022-04-03T19:49:20.667 INFO:tasks.workunit.client.0.smithi077.stdout:removed './256/21' 2022-04-03T19:49:20.667 INFO:tasks.workunit.client.0.smithi077.stdout:removed './256/27' 2022-04-03T19:49:20.667 INFO:tasks.workunit.client.0.smithi077.stdout:removed './256/31' 2022-04-03T19:49:20.667 INFO:tasks.workunit.client.0.smithi077.stdout:removed './256/36' 2022-04-03T19:49:20.668 INFO:tasks.workunit.client.0.smithi077.stdout:removed './256/20' 2022-04-03T19:49:20.668 INFO:tasks.workunit.client.0.smithi077.stdout:removed './256/39' 2022-04-03T19:49:20.668 INFO:tasks.workunit.client.0.smithi077.stdout:removed './256/34' 2022-04-03T19:49:20.668 INFO:tasks.workunit.client.0.smithi077.stdout:removed './256/38' 2022-04-03T19:49:20.668 INFO:tasks.workunit.client.0.smithi077.stdout:removed './256/28' 2022-04-03T19:49:20.669 INFO:tasks.workunit.client.0.smithi077.stdout:removed './256/24' 2022-04-03T19:49:20.669 INFO:tasks.workunit.client.0.smithi077.stdout:removed './256/29' 2022-04-03T19:49:20.669 INFO:tasks.workunit.client.0.smithi077.stdout:removed './256/32' 2022-04-03T19:49:20.669 INFO:tasks.workunit.client.0.smithi077.stdout:removed './256/22' 2022-04-03T19:49:20.669 INFO:tasks.workunit.client.0.smithi077.stdout:removed './256/25' 2022-04-03T19:49:20.670 INFO:tasks.workunit.client.0.smithi077.stdout:removed './256/30' 2022-04-03T19:49:22.658 INFO:tasks.workunit.client.0.smithi077.stdout:removed './256/33' 2022-04-03T19:49:22.658 INFO:tasks.workunit.client.0.smithi077.stdout:removed './256/23' 2022-04-03T19:49:22.659 INFO:tasks.workunit.client.0.smithi077.stdout:removed './256/35' 2022-04-03T19:49:22.659 INFO:tasks.workunit.client.0.smithi077.stdout:removed './256/26' 2022-04-03T19:49:22.659 INFO:tasks.workunit.client.0.smithi077.stdout:removed './256/37' 2022-04-03T19:49:22.659 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './256' 2022-04-03T19:49:22.660 INFO:tasks.workunit.client.0.smithi077.stdout:removed './257/21' 2022-04-03T19:49:22.660 INFO:tasks.workunit.client.0.smithi077.stdout:removed './257/27' 2022-04-03T19:49:22.660 INFO:tasks.workunit.client.0.smithi077.stdout:removed './257/31' 2022-04-03T19:49:22.660 INFO:tasks.workunit.client.0.smithi077.stdout:removed './257/36' 2022-04-03T19:49:22.661 INFO:tasks.workunit.client.0.smithi077.stdout:removed './257/20' 2022-04-03T19:49:22.661 INFO:tasks.workunit.client.0.smithi077.stdout:removed './257/39' 2022-04-03T19:49:22.661 INFO:tasks.workunit.client.0.smithi077.stdout:removed './257/34' 2022-04-03T19:49:22.661 INFO:tasks.workunit.client.0.smithi077.stdout:removed './257/38' 2022-04-03T19:49:22.661 INFO:tasks.workunit.client.0.smithi077.stdout:removed './257/28' 2022-04-03T19:49:22.662 INFO:tasks.workunit.client.0.smithi077.stdout:removed './257/24' 2022-04-03T19:49:22.662 INFO:tasks.workunit.client.0.smithi077.stdout:removed './257/29' 2022-04-03T19:49:22.662 INFO:tasks.workunit.client.0.smithi077.stdout:removed './257/32' 2022-04-03T19:49:22.662 INFO:tasks.workunit.client.0.smithi077.stdout:removed './257/22' 2022-04-03T19:49:22.663 INFO:tasks.workunit.client.0.smithi077.stdout:removed './257/25' 2022-04-03T19:49:22.663 INFO:tasks.workunit.client.0.smithi077.stdout:removed './257/30' 2022-04-03T19:49:22.663 INFO:tasks.workunit.client.0.smithi077.stdout:removed './257/33' 2022-04-03T19:49:22.663 INFO:tasks.workunit.client.0.smithi077.stdout:removed './257/23' 2022-04-03T19:49:22.664 INFO:tasks.workunit.client.0.smithi077.stdout:removed './257/35' 2022-04-03T19:49:22.664 INFO:tasks.workunit.client.0.smithi077.stdout:removed './257/26' 2022-04-03T19:49:22.664 INFO:tasks.workunit.client.0.smithi077.stdout:removed './257/37' 2022-04-03T19:49:22.664 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './257' 2022-04-03T19:49:22.664 INFO:tasks.workunit.client.0.smithi077.stdout:removed './258/21' 2022-04-03T19:49:22.665 INFO:tasks.workunit.client.0.smithi077.stdout:removed './258/27' 2022-04-03T19:49:22.665 INFO:tasks.workunit.client.0.smithi077.stdout:removed './258/31' 2022-04-03T19:49:22.665 INFO:tasks.workunit.client.0.smithi077.stdout:removed './258/36' 2022-04-03T19:49:22.665 INFO:tasks.workunit.client.0.smithi077.stdout:removed './258/20' 2022-04-03T19:49:22.666 INFO:tasks.workunit.client.0.smithi077.stdout:removed './258/39' 2022-04-03T19:49:22.666 INFO:tasks.workunit.client.0.smithi077.stdout:removed './258/34' 2022-04-03T19:49:22.666 INFO:tasks.workunit.client.0.smithi077.stdout:removed './258/38' 2022-04-03T19:49:22.666 INFO:tasks.workunit.client.0.smithi077.stdout:removed './258/28' 2022-04-03T19:49:22.666 INFO:tasks.workunit.client.0.smithi077.stdout:removed './258/24' 2022-04-03T19:49:22.667 INFO:tasks.workunit.client.0.smithi077.stdout:removed './258/29' 2022-04-03T19:49:22.667 INFO:tasks.workunit.client.0.smithi077.stdout:removed './258/32' 2022-04-03T19:49:22.667 INFO:tasks.workunit.client.0.smithi077.stdout:removed './258/22' 2022-04-03T19:49:22.667 INFO:tasks.workunit.client.0.smithi077.stdout:removed './258/25' 2022-04-03T19:49:22.668 INFO:tasks.workunit.client.0.smithi077.stdout:removed './258/30' 2022-04-03T19:49:22.668 INFO:tasks.workunit.client.0.smithi077.stdout:removed './258/33' 2022-04-03T19:49:22.668 INFO:tasks.workunit.client.0.smithi077.stdout:removed './258/23' 2022-04-03T19:49:22.668 INFO:tasks.workunit.client.0.smithi077.stdout:removed './258/35' 2022-04-03T19:49:22.669 INFO:tasks.workunit.client.0.smithi077.stdout:removed './258/26' 2022-04-03T19:49:22.669 INFO:tasks.workunit.client.0.smithi077.stdout:removed './258/37' 2022-04-03T19:49:22.669 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './258' 2022-04-03T19:49:22.669 INFO:tasks.workunit.client.0.smithi077.stdout:removed './259/21' 2022-04-03T19:49:22.669 INFO:tasks.workunit.client.0.smithi077.stdout:removed './259/27' 2022-04-03T19:49:22.670 INFO:tasks.workunit.client.0.smithi077.stdout:removed './259/31' 2022-04-03T19:49:22.670 INFO:tasks.workunit.client.0.smithi077.stdout:removed './259/36' 2022-04-03T19:49:22.670 INFO:tasks.workunit.client.0.smithi077.stdout:removed './259/20' 2022-04-03T19:49:22.670 INFO:tasks.workunit.client.0.smithi077.stdout:removed './259/39' 2022-04-03T19:49:22.671 INFO:tasks.workunit.client.0.smithi077.stdout:removed './259/34' 2022-04-03T19:49:22.671 INFO:tasks.workunit.client.0.smithi077.stdout:removed './259/38' 2022-04-03T19:49:22.671 INFO:tasks.workunit.client.0.smithi077.stdout:removed './259/28' 2022-04-03T19:49:22.671 INFO:tasks.workunit.client.0.smithi077.stdout:removed './259/24' 2022-04-03T19:49:22.671 INFO:tasks.workunit.client.0.smithi077.stdout:removed './259/29' 2022-04-03T19:49:22.672 INFO:tasks.workunit.client.0.smithi077.stdout:removed './259/32' 2022-04-03T19:49:22.672 INFO:tasks.workunit.client.0.smithi077.stdout:removed './259/22' 2022-04-03T19:49:22.672 INFO:tasks.workunit.client.0.smithi077.stdout:removed './259/25' 2022-04-03T19:49:22.672 INFO:tasks.workunit.client.0.smithi077.stdout:removed './259/30' 2022-04-03T19:49:22.673 INFO:tasks.workunit.client.0.smithi077.stdout:removed './259/33' 2022-04-03T19:49:22.673 INFO:tasks.workunit.client.0.smithi077.stdout:removed './259/23' 2022-04-03T19:49:22.673 INFO:tasks.workunit.client.0.smithi077.stdout:removed './259/35' 2022-04-03T19:49:22.673 INFO:tasks.workunit.client.0.smithi077.stdout:removed './259/26' 2022-04-03T19:49:22.673 INFO:tasks.workunit.client.0.smithi077.stdout:removed './259/37' 2022-04-03T19:49:22.674 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './259' 2022-04-03T19:49:22.674 INFO:tasks.workunit.client.0.smithi077.stdout:removed './260/21' 2022-04-03T19:49:22.674 INFO:tasks.workunit.client.0.smithi077.stdout:removed './260/27' 2022-04-03T19:49:22.674 INFO:tasks.workunit.client.0.smithi077.stdout:removed './260/31' 2022-04-03T19:49:22.675 INFO:tasks.workunit.client.0.smithi077.stdout:removed './260/36' 2022-04-03T19:49:22.675 INFO:tasks.workunit.client.0.smithi077.stdout:removed './260/20' 2022-04-03T19:49:22.675 INFO:tasks.workunit.client.0.smithi077.stdout:removed './260/39' 2022-04-03T19:49:22.675 INFO:tasks.workunit.client.0.smithi077.stdout:removed './260/34' 2022-04-03T19:49:22.675 INFO:tasks.workunit.client.0.smithi077.stdout:removed './260/38' 2022-04-03T19:49:22.676 INFO:tasks.workunit.client.0.smithi077.stdout:removed './260/28' 2022-04-03T19:49:22.676 INFO:tasks.workunit.client.0.smithi077.stdout:removed './260/24' 2022-04-03T19:49:22.676 INFO:tasks.workunit.client.0.smithi077.stdout:removed './260/29' 2022-04-03T19:49:22.676 INFO:tasks.workunit.client.0.smithi077.stdout:removed './260/32' 2022-04-03T19:49:22.677 INFO:tasks.workunit.client.0.smithi077.stdout:removed './260/22' 2022-04-03T19:49:22.677 INFO:tasks.workunit.client.0.smithi077.stdout:removed './260/25' 2022-04-03T19:49:22.677 INFO:tasks.workunit.client.0.smithi077.stdout:removed './260/30' 2022-04-03T19:49:22.677 INFO:tasks.workunit.client.0.smithi077.stdout:removed './260/33' 2022-04-03T19:49:22.678 INFO:tasks.workunit.client.0.smithi077.stdout:removed './260/23' 2022-04-03T19:49:22.679 INFO:tasks.workunit.client.0.smithi077.stdout:removed './260/35' 2022-04-03T19:49:22.679 INFO:tasks.workunit.client.0.smithi077.stdout:removed './260/26' 2022-04-03T19:49:22.679 INFO:tasks.workunit.client.0.smithi077.stdout:removed './260/37' 2022-04-03T19:49:22.679 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './260' 2022-04-03T19:49:22.680 INFO:tasks.workunit.client.0.smithi077.stdout:removed './261/21' 2022-04-03T19:49:22.680 INFO:tasks.workunit.client.0.smithi077.stdout:removed './261/27' 2022-04-03T19:49:22.680 INFO:tasks.workunit.client.0.smithi077.stdout:removed './261/31' 2022-04-03T19:49:22.680 INFO:tasks.workunit.client.0.smithi077.stdout:removed './261/36' 2022-04-03T19:49:22.681 INFO:tasks.workunit.client.0.smithi077.stdout:removed './261/20' 2022-04-03T19:49:22.681 INFO:tasks.workunit.client.0.smithi077.stdout:removed './261/39' 2022-04-03T19:49:22.681 INFO:tasks.workunit.client.0.smithi077.stdout:removed './261/34' 2022-04-03T19:49:22.681 INFO:tasks.workunit.client.0.smithi077.stdout:removed './261/38' 2022-04-03T19:49:22.681 INFO:tasks.workunit.client.0.smithi077.stdout:removed './261/28' 2022-04-03T19:49:22.682 INFO:tasks.workunit.client.0.smithi077.stdout:removed './261/24' 2022-04-03T19:49:22.682 INFO:tasks.workunit.client.0.smithi077.stdout:removed './261/29' 2022-04-03T19:49:22.682 INFO:tasks.workunit.client.0.smithi077.stdout:removed './261/32' 2022-04-03T19:49:22.682 INFO:tasks.workunit.client.0.smithi077.stdout:removed './261/22' 2022-04-03T19:49:22.683 INFO:tasks.workunit.client.0.smithi077.stdout:removed './261/25' 2022-04-03T19:49:22.683 INFO:tasks.workunit.client.0.smithi077.stdout:removed './261/30' 2022-04-03T19:49:22.683 INFO:tasks.workunit.client.0.smithi077.stdout:removed './261/33' 2022-04-03T19:49:22.683 INFO:tasks.workunit.client.0.smithi077.stdout:removed './261/23' 2022-04-03T19:49:22.683 INFO:tasks.workunit.client.0.smithi077.stdout:removed './261/35' 2022-04-03T19:49:22.684 INFO:tasks.workunit.client.0.smithi077.stdout:removed './261/26' 2022-04-03T19:49:22.684 INFO:tasks.workunit.client.0.smithi077.stdout:removed './261/37' 2022-04-03T19:49:22.684 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './261' 2022-04-03T19:49:22.684 INFO:tasks.workunit.client.0.smithi077.stdout:removed './262/21' 2022-04-03T19:49:22.685 INFO:tasks.workunit.client.0.smithi077.stdout:removed './262/27' 2022-04-03T19:49:22.685 INFO:tasks.workunit.client.0.smithi077.stdout:removed './262/31' 2022-04-03T19:49:22.685 INFO:tasks.workunit.client.0.smithi077.stdout:removed './262/36' 2022-04-03T19:49:22.685 INFO:tasks.workunit.client.0.smithi077.stdout:removed './262/20' 2022-04-03T19:49:22.685 INFO:tasks.workunit.client.0.smithi077.stdout:removed './262/39' 2022-04-03T19:49:22.686 INFO:tasks.workunit.client.0.smithi077.stdout:removed './262/34' 2022-04-03T19:49:22.686 INFO:tasks.workunit.client.0.smithi077.stdout:removed './262/38' 2022-04-03T19:49:22.686 INFO:tasks.workunit.client.0.smithi077.stdout:removed './262/28' 2022-04-03T19:49:22.686 INFO:tasks.workunit.client.0.smithi077.stdout:removed './262/24' 2022-04-03T19:49:22.687 INFO:tasks.workunit.client.0.smithi077.stdout:removed './262/29' 2022-04-03T19:49:22.687 INFO:tasks.workunit.client.0.smithi077.stdout:removed './262/32' 2022-04-03T19:49:22.687 INFO:tasks.workunit.client.0.smithi077.stdout:removed './262/22' 2022-04-03T19:49:22.687 INFO:tasks.workunit.client.0.smithi077.stdout:removed './262/25' 2022-04-03T19:49:22.687 INFO:tasks.workunit.client.0.smithi077.stdout:removed './262/30' 2022-04-03T19:49:22.688 INFO:tasks.workunit.client.0.smithi077.stdout:removed './262/33' 2022-04-03T19:49:22.688 INFO:tasks.workunit.client.0.smithi077.stdout:removed './262/23' 2022-04-03T19:49:22.688 INFO:tasks.workunit.client.0.smithi077.stdout:removed './262/35' 2022-04-03T19:49:22.688 INFO:tasks.workunit.client.0.smithi077.stdout:removed './262/26' 2022-04-03T19:49:22.689 INFO:tasks.workunit.client.0.smithi077.stdout:removed './262/37' 2022-04-03T19:49:22.689 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './262' 2022-04-03T19:49:22.689 INFO:tasks.workunit.client.0.smithi077.stdout:removed './263/21' 2022-04-03T19:49:22.689 INFO:tasks.workunit.client.0.smithi077.stdout:removed './263/27' 2022-04-03T19:49:22.689 INFO:tasks.workunit.client.0.smithi077.stdout:removed './263/31' 2022-04-03T19:49:22.690 INFO:tasks.workunit.client.0.smithi077.stdout:removed './263/36' 2022-04-03T19:49:22.690 INFO:tasks.workunit.client.0.smithi077.stdout:removed './263/20' 2022-04-03T19:49:22.690 INFO:tasks.workunit.client.0.smithi077.stdout:removed './263/39' 2022-04-03T19:49:22.690 INFO:tasks.workunit.client.0.smithi077.stdout:removed './263/34' 2022-04-03T19:49:22.691 INFO:tasks.workunit.client.0.smithi077.stdout:removed './263/38' 2022-04-03T19:49:22.691 INFO:tasks.workunit.client.0.smithi077.stdout:removed './263/28' 2022-04-03T19:49:22.691 INFO:tasks.workunit.client.0.smithi077.stdout:removed './263/24' 2022-04-03T19:49:22.691 INFO:tasks.workunit.client.0.smithi077.stdout:removed './263/29' 2022-04-03T19:49:22.691 INFO:tasks.workunit.client.0.smithi077.stdout:removed './263/32' 2022-04-03T19:49:22.692 INFO:tasks.workunit.client.0.smithi077.stdout:removed './263/22' 2022-04-03T19:49:22.692 INFO:tasks.workunit.client.0.smithi077.stdout:removed './263/25' 2022-04-03T19:49:22.692 INFO:tasks.workunit.client.0.smithi077.stdout:removed './263/30' 2022-04-03T19:49:22.692 INFO:tasks.workunit.client.0.smithi077.stdout:removed './263/33' 2022-04-03T19:49:22.693 INFO:tasks.workunit.client.0.smithi077.stdout:removed './263/23' 2022-04-03T19:49:22.693 INFO:tasks.workunit.client.0.smithi077.stdout:removed './263/35' 2022-04-03T19:49:22.693 INFO:tasks.workunit.client.0.smithi077.stdout:removed './263/26' 2022-04-03T19:49:22.693 INFO:tasks.workunit.client.0.smithi077.stdout:removed './263/37' 2022-04-03T19:49:22.693 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './263' 2022-04-03T19:49:22.694 INFO:tasks.workunit.client.0.smithi077.stdout:removed './264/21' 2022-04-03T19:49:22.694 INFO:tasks.workunit.client.0.smithi077.stdout:removed './264/27' 2022-04-03T19:49:22.694 INFO:tasks.workunit.client.0.smithi077.stdout:removed './264/31' 2022-04-03T19:49:22.694 INFO:tasks.workunit.client.0.smithi077.stdout:removed './264/36' 2022-04-03T19:49:22.694 INFO:tasks.workunit.client.0.smithi077.stdout:removed './264/20' 2022-04-03T19:49:22.695 INFO:tasks.workunit.client.0.smithi077.stdout:removed './264/39' 2022-04-03T19:49:22.695 INFO:tasks.workunit.client.0.smithi077.stdout:removed './264/34' 2022-04-03T19:49:22.695 INFO:tasks.workunit.client.0.smithi077.stdout:removed './264/38' 2022-04-03T19:49:22.695 INFO:tasks.workunit.client.0.smithi077.stdout:removed './264/28' 2022-04-03T19:49:22.696 INFO:tasks.workunit.client.0.smithi077.stdout:removed './264/24' 2022-04-03T19:49:22.696 INFO:tasks.workunit.client.0.smithi077.stdout:removed './264/29' 2022-04-03T19:49:22.696 INFO:tasks.workunit.client.0.smithi077.stdout:removed './264/32' 2022-04-03T19:49:22.696 INFO:tasks.workunit.client.0.smithi077.stdout:removed './264/22' 2022-04-03T19:49:22.696 INFO:tasks.workunit.client.0.smithi077.stdout:removed './264/25' 2022-04-03T19:49:22.697 INFO:tasks.workunit.client.0.smithi077.stdout:removed './264/30' 2022-04-03T19:49:22.697 INFO:tasks.workunit.client.0.smithi077.stdout:removed './264/33' 2022-04-03T19:49:22.697 INFO:tasks.workunit.client.0.smithi077.stdout:removed './264/23' 2022-04-03T19:49:22.697 INFO:tasks.workunit.client.0.smithi077.stdout:removed './264/35' 2022-04-03T19:49:22.698 INFO:tasks.workunit.client.0.smithi077.stdout:removed './264/26' 2022-04-03T19:49:22.698 INFO:tasks.workunit.client.0.smithi077.stdout:removed './264/37' 2022-04-03T19:49:22.698 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './264' 2022-04-03T19:49:22.698 INFO:tasks.workunit.client.0.smithi077.stdout:removed './265/21' 2022-04-03T19:49:22.698 INFO:tasks.workunit.client.0.smithi077.stdout:removed './265/27' 2022-04-03T19:49:22.699 INFO:tasks.workunit.client.0.smithi077.stdout:removed './265/31' 2022-04-03T19:49:22.699 INFO:tasks.workunit.client.0.smithi077.stdout:removed './265/36' 2022-04-03T19:49:22.699 INFO:tasks.workunit.client.0.smithi077.stdout:removed './265/20' 2022-04-03T19:49:22.699 INFO:tasks.workunit.client.0.smithi077.stdout:removed './265/39' 2022-04-03T19:49:22.700 INFO:tasks.workunit.client.0.smithi077.stdout:removed './265/34' 2022-04-03T19:49:22.700 INFO:tasks.workunit.client.0.smithi077.stdout:removed './265/38' 2022-04-03T19:49:22.700 INFO:tasks.workunit.client.0.smithi077.stdout:removed './265/28' 2022-04-03T19:49:22.700 INFO:tasks.workunit.client.0.smithi077.stdout:removed './265/24' 2022-04-03T19:49:22.700 INFO:tasks.workunit.client.0.smithi077.stdout:removed './265/29' 2022-04-03T19:49:22.701 INFO:tasks.workunit.client.0.smithi077.stdout:removed './265/32' 2022-04-03T19:49:22.701 INFO:tasks.workunit.client.0.smithi077.stdout:removed './265/22' 2022-04-03T19:49:22.701 INFO:tasks.workunit.client.0.smithi077.stdout:removed './265/25' 2022-04-03T19:49:22.701 INFO:tasks.workunit.client.0.smithi077.stdout:removed './265/30' 2022-04-03T19:49:22.702 INFO:tasks.workunit.client.0.smithi077.stdout:removed './265/33' 2022-04-03T19:49:22.702 INFO:tasks.workunit.client.0.smithi077.stdout:removed './265/23' 2022-04-03T19:49:22.702 INFO:tasks.workunit.client.0.smithi077.stdout:removed './265/35' 2022-04-03T19:49:22.702 INFO:tasks.workunit.client.0.smithi077.stdout:removed './265/26' 2022-04-03T19:49:22.702 INFO:tasks.workunit.client.0.smithi077.stdout:removed './265/37' 2022-04-03T19:49:22.703 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './265' 2022-04-03T19:49:22.703 INFO:tasks.workunit.client.0.smithi077.stdout:removed './266/21' 2022-04-03T19:49:22.703 INFO:tasks.workunit.client.0.smithi077.stdout:removed './266/27' 2022-04-03T19:49:22.703 INFO:tasks.workunit.client.0.smithi077.stdout:removed './266/31' 2022-04-03T19:49:22.704 INFO:tasks.workunit.client.0.smithi077.stdout:removed './266/36' 2022-04-03T19:49:22.704 INFO:tasks.workunit.client.0.smithi077.stdout:removed './266/20' 2022-04-03T19:49:22.704 INFO:tasks.workunit.client.0.smithi077.stdout:removed './266/39' 2022-04-03T19:49:22.704 INFO:tasks.workunit.client.0.smithi077.stdout:removed './266/34' 2022-04-03T19:49:22.705 INFO:tasks.workunit.client.0.smithi077.stdout:removed './266/38' 2022-04-03T19:49:22.705 INFO:tasks.workunit.client.0.smithi077.stdout:removed './266/28' 2022-04-03T19:49:22.705 INFO:tasks.workunit.client.0.smithi077.stdout:removed './266/24' 2022-04-03T19:49:22.705 INFO:tasks.workunit.client.0.smithi077.stdout:removed './266/29' 2022-04-03T19:49:22.705 INFO:tasks.workunit.client.0.smithi077.stdout:removed './266/32' 2022-04-03T19:49:22.706 INFO:tasks.workunit.client.0.smithi077.stdout:removed './266/22' 2022-04-03T19:49:22.706 INFO:tasks.workunit.client.0.smithi077.stdout:removed './266/25' 2022-04-03T19:49:22.706 INFO:tasks.workunit.client.0.smithi077.stdout:removed './266/30' 2022-04-03T19:49:22.706 INFO:tasks.workunit.client.0.smithi077.stdout:removed './266/33' 2022-04-03T19:49:22.707 INFO:tasks.workunit.client.0.smithi077.stdout:removed './266/23' 2022-04-03T19:49:26.101 INFO:tasks.workunit.client.0.smithi077.stdout:removed './266/35' 2022-04-03T19:49:26.102 INFO:tasks.workunit.client.0.smithi077.stdout:removed './266/26' 2022-04-03T19:49:26.102 INFO:tasks.workunit.client.0.smithi077.stdout:removed './266/37' 2022-04-03T19:49:26.102 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './266' 2022-04-03T19:49:26.103 INFO:tasks.workunit.client.0.smithi077.stdout:removed './267/21' 2022-04-03T19:49:26.103 INFO:tasks.workunit.client.0.smithi077.stdout:removed './267/27' 2022-04-03T19:49:26.103 INFO:tasks.workunit.client.0.smithi077.stdout:removed './267/31' 2022-04-03T19:49:26.103 INFO:tasks.workunit.client.0.smithi077.stdout:removed './267/36' 2022-04-03T19:49:26.104 INFO:tasks.workunit.client.0.smithi077.stdout:removed './267/20' 2022-04-03T19:49:26.104 INFO:tasks.workunit.client.0.smithi077.stdout:removed './267/39' 2022-04-03T19:49:26.104 INFO:tasks.workunit.client.0.smithi077.stdout:removed './267/34' 2022-04-03T19:49:26.104 INFO:tasks.workunit.client.0.smithi077.stdout:removed './267/38' 2022-04-03T19:49:26.105 INFO:tasks.workunit.client.0.smithi077.stdout:removed './267/28' 2022-04-03T19:49:26.105 INFO:tasks.workunit.client.0.smithi077.stdout:removed './267/24' 2022-04-03T19:49:26.105 INFO:tasks.workunit.client.0.smithi077.stdout:removed './267/29' 2022-04-03T19:49:26.105 INFO:tasks.workunit.client.0.smithi077.stdout:removed './267/32' 2022-04-03T19:49:26.105 INFO:tasks.workunit.client.0.smithi077.stdout:removed './267/22' 2022-04-03T19:49:26.106 INFO:tasks.workunit.client.0.smithi077.stdout:removed './267/25' 2022-04-03T19:49:26.106 INFO:tasks.workunit.client.0.smithi077.stdout:removed './267/30' 2022-04-03T19:49:26.106 INFO:tasks.workunit.client.0.smithi077.stdout:removed './267/33' 2022-04-03T19:49:26.106 INFO:tasks.workunit.client.0.smithi077.stdout:removed './267/23' 2022-04-03T19:49:26.107 INFO:tasks.workunit.client.0.smithi077.stdout:removed './267/35' 2022-04-03T19:49:26.107 INFO:tasks.workunit.client.0.smithi077.stdout:removed './267/26' 2022-04-03T19:49:26.107 INFO:tasks.workunit.client.0.smithi077.stdout:removed './267/37' 2022-04-03T19:49:26.107 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './267' 2022-04-03T19:49:26.108 INFO:tasks.workunit.client.0.smithi077.stdout:removed './268/21' 2022-04-03T19:49:26.108 INFO:tasks.workunit.client.0.smithi077.stdout:removed './268/27' 2022-04-03T19:49:26.108 INFO:tasks.workunit.client.0.smithi077.stdout:removed './268/31' 2022-04-03T19:49:26.108 INFO:tasks.workunit.client.0.smithi077.stdout:removed './268/36' 2022-04-03T19:49:26.108 INFO:tasks.workunit.client.0.smithi077.stdout:removed './268/20' 2022-04-03T19:49:26.109 INFO:tasks.workunit.client.0.smithi077.stdout:removed './268/39' 2022-04-03T19:49:26.109 INFO:tasks.workunit.client.0.smithi077.stdout:removed './268/34' 2022-04-03T19:49:26.109 INFO:tasks.workunit.client.0.smithi077.stdout:removed './268/38' 2022-04-03T19:49:26.109 INFO:tasks.workunit.client.0.smithi077.stdout:removed './268/28' 2022-04-03T19:49:26.110 INFO:tasks.workunit.client.0.smithi077.stdout:removed './268/24' 2022-04-03T19:49:26.110 INFO:tasks.workunit.client.0.smithi077.stdout:removed './268/29' 2022-04-03T19:49:26.110 INFO:tasks.workunit.client.0.smithi077.stdout:removed './268/32' 2022-04-03T19:49:26.110 INFO:tasks.workunit.client.0.smithi077.stdout:removed './268/22' 2022-04-03T19:49:26.111 INFO:tasks.workunit.client.0.smithi077.stdout:removed './268/25' 2022-04-03T19:49:26.111 INFO:tasks.workunit.client.0.smithi077.stdout:removed './268/30' 2022-04-03T19:49:26.111 INFO:tasks.workunit.client.0.smithi077.stdout:removed './268/33' 2022-04-03T19:49:26.111 INFO:tasks.workunit.client.0.smithi077.stdout:removed './268/23' 2022-04-03T19:49:26.111 INFO:tasks.workunit.client.0.smithi077.stdout:removed './268/35' 2022-04-03T19:49:26.112 INFO:tasks.workunit.client.0.smithi077.stdout:removed './268/26' 2022-04-03T19:49:26.112 INFO:tasks.workunit.client.0.smithi077.stdout:removed './268/37' 2022-04-03T19:49:26.112 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './268' 2022-04-03T19:49:26.112 INFO:tasks.workunit.client.0.smithi077.stdout:removed './269/21' 2022-04-03T19:49:26.112 INFO:tasks.workunit.client.0.smithi077.stdout:removed './269/27' 2022-04-03T19:49:26.113 INFO:tasks.workunit.client.0.smithi077.stdout:removed './269/31' 2022-04-03T19:49:26.113 INFO:tasks.workunit.client.0.smithi077.stdout:removed './269/36' 2022-04-03T19:49:26.113 INFO:tasks.workunit.client.0.smithi077.stdout:removed './269/20' 2022-04-03T19:49:26.113 INFO:tasks.workunit.client.0.smithi077.stdout:removed './269/39' 2022-04-03T19:49:26.114 INFO:tasks.workunit.client.0.smithi077.stdout:removed './269/34' 2022-04-03T19:49:26.114 INFO:tasks.workunit.client.0.smithi077.stdout:removed './269/38' 2022-04-03T19:49:26.114 INFO:tasks.workunit.client.0.smithi077.stdout:removed './269/28' 2022-04-03T19:49:26.114 INFO:tasks.workunit.client.0.smithi077.stdout:removed './269/24' 2022-04-03T19:49:26.115 INFO:tasks.workunit.client.0.smithi077.stdout:removed './269/29' 2022-04-03T19:49:26.115 INFO:tasks.workunit.client.0.smithi077.stdout:removed './269/32' 2022-04-03T19:49:26.115 INFO:tasks.workunit.client.0.smithi077.stdout:removed './269/22' 2022-04-03T19:49:26.115 INFO:tasks.workunit.client.0.smithi077.stdout:removed './269/25' 2022-04-03T19:49:26.116 INFO:tasks.workunit.client.0.smithi077.stdout:removed './269/30' 2022-04-03T19:49:26.116 INFO:tasks.workunit.client.0.smithi077.stdout:removed './269/33' 2022-04-03T19:49:26.116 INFO:tasks.workunit.client.0.smithi077.stdout:removed './269/23' 2022-04-03T19:49:26.116 INFO:tasks.workunit.client.0.smithi077.stdout:removed './269/35' 2022-04-03T19:49:26.116 INFO:tasks.workunit.client.0.smithi077.stdout:removed './269/26' 2022-04-03T19:49:26.117 INFO:tasks.workunit.client.0.smithi077.stdout:removed './269/37' 2022-04-03T19:49:26.117 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './269' 2022-04-03T19:49:26.118 INFO:tasks.workunit.client.0.smithi077.stdout:removed './270/21' 2022-04-03T19:49:26.119 INFO:tasks.workunit.client.0.smithi077.stdout:removed './270/27' 2022-04-03T19:49:26.119 INFO:tasks.workunit.client.0.smithi077.stdout:removed './270/31' 2022-04-03T19:49:26.119 INFO:tasks.workunit.client.0.smithi077.stdout:removed './270/36' 2022-04-03T19:49:26.120 INFO:tasks.workunit.client.0.smithi077.stdout:removed './270/20' 2022-04-03T19:49:26.120 INFO:tasks.workunit.client.0.smithi077.stdout:removed './270/39' 2022-04-03T19:49:26.120 INFO:tasks.workunit.client.0.smithi077.stdout:removed './270/34' 2022-04-03T19:49:26.120 INFO:tasks.workunit.client.0.smithi077.stdout:removed './270/38' 2022-04-03T19:49:26.121 INFO:tasks.workunit.client.0.smithi077.stdout:removed './270/28' 2022-04-03T19:49:26.121 INFO:tasks.workunit.client.0.smithi077.stdout:removed './270/24' 2022-04-03T19:49:26.121 INFO:tasks.workunit.client.0.smithi077.stdout:removed './270/29' 2022-04-03T19:49:26.121 INFO:tasks.workunit.client.0.smithi077.stdout:removed './270/32' 2022-04-03T19:49:26.122 INFO:tasks.workunit.client.0.smithi077.stdout:removed './270/22' 2022-04-03T19:49:26.122 INFO:tasks.workunit.client.0.smithi077.stdout:removed './270/25' 2022-04-03T19:49:26.122 INFO:tasks.workunit.client.0.smithi077.stdout:removed './270/30' 2022-04-03T19:49:26.122 INFO:tasks.workunit.client.0.smithi077.stdout:removed './270/33' 2022-04-03T19:49:26.122 INFO:tasks.workunit.client.0.smithi077.stdout:removed './270/23' 2022-04-03T19:49:26.123 INFO:tasks.workunit.client.0.smithi077.stdout:removed './270/35' 2022-04-03T19:49:26.123 INFO:tasks.workunit.client.0.smithi077.stdout:removed './270/26' 2022-04-03T19:49:26.123 INFO:tasks.workunit.client.0.smithi077.stdout:removed './270/37' 2022-04-03T19:49:26.123 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './270' 2022-04-03T19:49:26.124 INFO:tasks.workunit.client.0.smithi077.stdout:removed './271/21' 2022-04-03T19:49:26.124 INFO:tasks.workunit.client.0.smithi077.stdout:removed './271/27' 2022-04-03T19:49:26.124 INFO:tasks.workunit.client.0.smithi077.stdout:removed './271/31' 2022-04-03T19:49:26.124 INFO:tasks.workunit.client.0.smithi077.stdout:removed './271/36' 2022-04-03T19:49:26.125 INFO:tasks.workunit.client.0.smithi077.stdout:removed './271/20' 2022-04-03T19:49:26.125 INFO:tasks.workunit.client.0.smithi077.stdout:removed './271/39' 2022-04-03T19:49:26.125 INFO:tasks.workunit.client.0.smithi077.stdout:removed './271/34' 2022-04-03T19:49:26.125 INFO:tasks.workunit.client.0.smithi077.stdout:removed './271/38' 2022-04-03T19:49:26.126 INFO:tasks.workunit.client.0.smithi077.stdout:removed './271/28' 2022-04-03T19:49:26.126 INFO:tasks.workunit.client.0.smithi077.stdout:removed './271/24' 2022-04-03T19:49:26.126 INFO:tasks.workunit.client.0.smithi077.stdout:removed './271/29' 2022-04-03T19:49:26.126 INFO:tasks.workunit.client.0.smithi077.stdout:removed './271/32' 2022-04-03T19:49:26.126 INFO:tasks.workunit.client.0.smithi077.stdout:removed './271/22' 2022-04-03T19:49:26.127 INFO:tasks.workunit.client.0.smithi077.stdout:removed './271/25' 2022-04-03T19:49:26.127 INFO:tasks.workunit.client.0.smithi077.stdout:removed './271/30' 2022-04-03T19:49:26.127 INFO:tasks.workunit.client.0.smithi077.stdout:removed './271/33' 2022-04-03T19:49:26.127 INFO:tasks.workunit.client.0.smithi077.stdout:removed './271/23' 2022-04-03T19:49:26.128 INFO:tasks.workunit.client.0.smithi077.stdout:removed './271/35' 2022-04-03T19:49:26.128 INFO:tasks.workunit.client.0.smithi077.stdout:removed './271/26' 2022-04-03T19:49:26.128 INFO:tasks.workunit.client.0.smithi077.stdout:removed './271/37' 2022-04-03T19:49:26.128 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './271' 2022-04-03T19:49:26.129 INFO:tasks.workunit.client.0.smithi077.stdout:removed './272/21' 2022-04-03T19:49:26.129 INFO:tasks.workunit.client.0.smithi077.stdout:removed './272/27' 2022-04-03T19:49:26.129 INFO:tasks.workunit.client.0.smithi077.stdout:removed './272/31' 2022-04-03T19:49:26.129 INFO:tasks.workunit.client.0.smithi077.stdout:removed './272/36' 2022-04-03T19:49:26.130 INFO:tasks.workunit.client.0.smithi077.stdout:removed './272/20' 2022-04-03T19:49:26.130 INFO:tasks.workunit.client.0.smithi077.stdout:removed './272/39' 2022-04-03T19:49:26.130 INFO:tasks.workunit.client.0.smithi077.stdout:removed './272/34' 2022-04-03T19:49:26.130 INFO:tasks.workunit.client.0.smithi077.stdout:removed './272/38' 2022-04-03T19:49:26.131 INFO:tasks.workunit.client.0.smithi077.stdout:removed './272/28' 2022-04-03T19:49:26.131 INFO:tasks.workunit.client.0.smithi077.stdout:removed './272/24' 2022-04-03T19:49:26.131 INFO:tasks.workunit.client.0.smithi077.stdout:removed './272/29' 2022-04-03T19:49:26.131 INFO:tasks.workunit.client.0.smithi077.stdout:removed './272/32' 2022-04-03T19:49:26.131 INFO:tasks.workunit.client.0.smithi077.stdout:removed './272/22' 2022-04-03T19:49:26.132 INFO:tasks.workunit.client.0.smithi077.stdout:removed './272/25' 2022-04-03T19:49:26.132 INFO:tasks.workunit.client.0.smithi077.stdout:removed './272/30' 2022-04-03T19:49:26.132 INFO:tasks.workunit.client.0.smithi077.stdout:removed './272/33' 2022-04-03T19:49:26.132 INFO:tasks.workunit.client.0.smithi077.stdout:removed './272/23' 2022-04-03T19:49:26.133 INFO:tasks.workunit.client.0.smithi077.stdout:removed './272/35' 2022-04-03T19:49:26.133 INFO:tasks.workunit.client.0.smithi077.stdout:removed './272/26' 2022-04-03T19:49:26.133 INFO:tasks.workunit.client.0.smithi077.stdout:removed './272/37' 2022-04-03T19:49:26.133 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './272' 2022-04-03T19:49:26.134 INFO:tasks.workunit.client.0.smithi077.stdout:removed './273/21' 2022-04-03T19:49:26.134 INFO:tasks.workunit.client.0.smithi077.stdout:removed './273/27' 2022-04-03T19:49:26.134 INFO:tasks.workunit.client.0.smithi077.stdout:removed './273/31' 2022-04-03T19:49:26.134 INFO:tasks.workunit.client.0.smithi077.stdout:removed './273/36' 2022-04-03T19:49:26.135 INFO:tasks.workunit.client.0.smithi077.stdout:removed './273/20' 2022-04-03T19:49:26.135 INFO:tasks.workunit.client.0.smithi077.stdout:removed './273/39' 2022-04-03T19:49:26.135 INFO:tasks.workunit.client.0.smithi077.stdout:removed './273/34' 2022-04-03T19:49:26.135 INFO:tasks.workunit.client.0.smithi077.stdout:removed './273/38' 2022-04-03T19:49:26.135 INFO:tasks.workunit.client.0.smithi077.stdout:removed './273/28' 2022-04-03T19:49:26.136 INFO:tasks.workunit.client.0.smithi077.stdout:removed './273/24' 2022-04-03T19:49:26.136 INFO:tasks.workunit.client.0.smithi077.stdout:removed './273/29' 2022-04-03T19:49:26.136 INFO:tasks.workunit.client.0.smithi077.stdout:removed './273/32' 2022-04-03T19:49:26.136 INFO:tasks.workunit.client.0.smithi077.stdout:removed './273/22' 2022-04-03T19:49:26.137 INFO:tasks.workunit.client.0.smithi077.stdout:removed './273/25' 2022-04-03T19:49:26.137 INFO:tasks.workunit.client.0.smithi077.stdout:removed './273/30' 2022-04-03T19:49:26.137 INFO:tasks.workunit.client.0.smithi077.stdout:removed './273/33' 2022-04-03T19:49:26.137 INFO:tasks.workunit.client.0.smithi077.stdout:removed './273/23' 2022-04-03T19:49:26.137 INFO:tasks.workunit.client.0.smithi077.stdout:removed './273/35' 2022-04-03T19:49:26.138 INFO:tasks.workunit.client.0.smithi077.stdout:removed './273/26' 2022-04-03T19:49:26.138 INFO:tasks.workunit.client.0.smithi077.stdout:removed './273/37' 2022-04-03T19:49:26.138 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './273' 2022-04-03T19:49:26.138 INFO:tasks.workunit.client.0.smithi077.stdout:removed './274/21' 2022-04-03T19:49:26.139 INFO:tasks.workunit.client.0.smithi077.stdout:removed './274/27' 2022-04-03T19:49:26.139 INFO:tasks.workunit.client.0.smithi077.stdout:removed './274/31' 2022-04-03T19:49:26.139 INFO:tasks.workunit.client.0.smithi077.stdout:removed './274/36' 2022-04-03T19:49:26.139 INFO:tasks.workunit.client.0.smithi077.stdout:removed './274/20' 2022-04-03T19:49:26.139 INFO:tasks.workunit.client.0.smithi077.stdout:removed './274/39' 2022-04-03T19:49:26.140 INFO:tasks.workunit.client.0.smithi077.stdout:removed './274/34' 2022-04-03T19:49:26.140 INFO:tasks.workunit.client.0.smithi077.stdout:removed './274/38' 2022-04-03T19:49:26.140 INFO:tasks.workunit.client.0.smithi077.stdout:removed './274/28' 2022-04-03T19:49:26.140 INFO:tasks.workunit.client.0.smithi077.stdout:removed './274/24' 2022-04-03T19:49:26.141 INFO:tasks.workunit.client.0.smithi077.stdout:removed './274/29' 2022-04-03T19:49:26.141 INFO:tasks.workunit.client.0.smithi077.stdout:removed './274/32' 2022-04-03T19:49:26.141 INFO:tasks.workunit.client.0.smithi077.stdout:removed './274/22' 2022-04-03T19:49:26.141 INFO:tasks.workunit.client.0.smithi077.stdout:removed './274/25' 2022-04-03T19:49:26.141 INFO:tasks.workunit.client.0.smithi077.stdout:removed './274/30' 2022-04-03T19:49:26.142 INFO:tasks.workunit.client.0.smithi077.stdout:removed './274/33' 2022-04-03T19:49:26.142 INFO:tasks.workunit.client.0.smithi077.stdout:removed './274/23' 2022-04-03T19:49:26.142 INFO:tasks.workunit.client.0.smithi077.stdout:removed './274/35' 2022-04-03T19:49:26.142 INFO:tasks.workunit.client.0.smithi077.stdout:removed './274/26' 2022-04-03T19:49:26.143 INFO:tasks.workunit.client.0.smithi077.stdout:removed './274/37' 2022-04-03T19:49:26.143 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './274' 2022-04-03T19:49:26.143 INFO:tasks.workunit.client.0.smithi077.stdout:removed './275/21' 2022-04-03T19:49:26.143 INFO:tasks.workunit.client.0.smithi077.stdout:removed './275/27' 2022-04-03T19:49:26.143 INFO:tasks.workunit.client.0.smithi077.stdout:removed './275/31' 2022-04-03T19:49:26.144 INFO:tasks.workunit.client.0.smithi077.stdout:removed './275/36' 2022-04-03T19:49:26.144 INFO:tasks.workunit.client.0.smithi077.stdout:removed './275/20' 2022-04-03T19:49:26.144 INFO:tasks.workunit.client.0.smithi077.stdout:removed './275/39' 2022-04-03T19:49:26.144 INFO:tasks.workunit.client.0.smithi077.stdout:removed './275/34' 2022-04-03T19:49:26.145 INFO:tasks.workunit.client.0.smithi077.stdout:removed './275/38' 2022-04-03T19:49:26.145 INFO:tasks.workunit.client.0.smithi077.stdout:removed './275/28' 2022-04-03T19:49:26.145 INFO:tasks.workunit.client.0.smithi077.stdout:removed './275/24' 2022-04-03T19:49:26.145 INFO:tasks.workunit.client.0.smithi077.stdout:removed './275/29' 2022-04-03T19:49:26.146 INFO:tasks.workunit.client.0.smithi077.stdout:removed './275/32' 2022-04-03T19:49:26.146 INFO:tasks.workunit.client.0.smithi077.stdout:removed './275/22' 2022-04-03T19:49:26.146 INFO:tasks.workunit.client.0.smithi077.stdout:removed './275/25' 2022-04-03T19:49:26.146 INFO:tasks.workunit.client.0.smithi077.stdout:removed './275/30' 2022-04-03T19:49:26.146 INFO:tasks.workunit.client.0.smithi077.stdout:removed './275/33' 2022-04-03T19:49:26.147 INFO:tasks.workunit.client.0.smithi077.stdout:removed './275/23' 2022-04-03T19:49:26.147 INFO:tasks.workunit.client.0.smithi077.stdout:removed './275/35' 2022-04-03T19:49:26.147 INFO:tasks.workunit.client.0.smithi077.stdout:removed './275/26' 2022-04-03T19:49:26.147 INFO:tasks.workunit.client.0.smithi077.stdout:removed './275/37' 2022-04-03T19:49:26.148 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './275' 2022-04-03T19:49:26.148 INFO:tasks.workunit.client.0.smithi077.stdout:removed './276/21' 2022-04-03T19:49:26.148 INFO:tasks.workunit.client.0.smithi077.stdout:removed './276/27' 2022-04-03T19:49:26.148 INFO:tasks.workunit.client.0.smithi077.stdout:removed './276/31' 2022-04-03T19:49:26.148 INFO:tasks.workunit.client.0.smithi077.stdout:removed './276/36' 2022-04-03T19:49:26.149 INFO:tasks.workunit.client.0.smithi077.stdout:removed './276/20' 2022-04-03T19:49:26.149 INFO:tasks.workunit.client.0.smithi077.stdout:removed './276/39' 2022-04-03T19:49:26.149 INFO:tasks.workunit.client.0.smithi077.stdout:removed './276/34' 2022-04-03T19:49:26.149 INFO:tasks.workunit.client.0.smithi077.stdout:removed './276/38' 2022-04-03T19:49:26.149 INFO:tasks.workunit.client.0.smithi077.stdout:removed './276/28' 2022-04-03T19:49:26.150 INFO:tasks.workunit.client.0.smithi077.stdout:removed './276/24' 2022-04-03T19:49:26.150 INFO:tasks.workunit.client.0.smithi077.stdout:removed './276/29' 2022-04-03T19:49:26.150 INFO:tasks.workunit.client.0.smithi077.stdout:removed './276/32' 2022-04-03T19:49:26.150 INFO:tasks.workunit.client.0.smithi077.stdout:removed './276/22' 2022-04-03T19:49:26.151 INFO:tasks.workunit.client.0.smithi077.stdout:removed './276/25' 2022-04-03T19:49:26.151 INFO:tasks.workunit.client.0.smithi077.stdout:removed './276/30' 2022-04-03T19:49:26.151 INFO:tasks.workunit.client.0.smithi077.stdout:removed './276/33' 2022-04-03T19:49:26.151 INFO:tasks.workunit.client.0.smithi077.stdout:removed './276/23' 2022-04-03T19:49:26.152 INFO:tasks.workunit.client.0.smithi077.stdout:removed './276/35' 2022-04-03T19:49:26.152 INFO:tasks.workunit.client.0.smithi077.stdout:removed './276/26' 2022-04-03T19:49:30.143 INFO:tasks.workunit.client.0.smithi077.stdout:removed './276/37' 2022-04-03T19:49:30.143 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './276' 2022-04-03T19:49:30.144 INFO:tasks.workunit.client.0.smithi077.stdout:removed './277/21' 2022-04-03T19:49:30.144 INFO:tasks.workunit.client.0.smithi077.stdout:removed './277/27' 2022-04-03T19:49:30.144 INFO:tasks.workunit.client.0.smithi077.stdout:removed './277/31' 2022-04-03T19:49:30.144 INFO:tasks.workunit.client.0.smithi077.stdout:removed './277/36' 2022-04-03T19:49:30.145 INFO:tasks.workunit.client.0.smithi077.stdout:removed './277/20' 2022-04-03T19:49:30.145 INFO:tasks.workunit.client.0.smithi077.stdout:removed './277/39' 2022-04-03T19:49:30.145 INFO:tasks.workunit.client.0.smithi077.stdout:removed './277/34' 2022-04-03T19:49:30.145 INFO:tasks.workunit.client.0.smithi077.stdout:removed './277/38' 2022-04-03T19:49:30.145 INFO:tasks.workunit.client.0.smithi077.stdout:removed './277/28' 2022-04-03T19:49:30.146 INFO:tasks.workunit.client.0.smithi077.stdout:removed './277/24' 2022-04-03T19:49:30.146 INFO:tasks.workunit.client.0.smithi077.stdout:removed './277/29' 2022-04-03T19:49:30.146 INFO:tasks.workunit.client.0.smithi077.stdout:removed './277/32' 2022-04-03T19:49:30.146 INFO:tasks.workunit.client.0.smithi077.stdout:removed './277/22' 2022-04-03T19:49:30.146 INFO:tasks.workunit.client.0.smithi077.stdout:removed './277/25' 2022-04-03T19:49:30.147 INFO:tasks.workunit.client.0.smithi077.stdout:removed './277/30' 2022-04-03T19:49:30.147 INFO:tasks.workunit.client.0.smithi077.stdout:removed './277/33' 2022-04-03T19:49:30.147 INFO:tasks.workunit.client.0.smithi077.stdout:removed './277/23' 2022-04-03T19:49:30.147 INFO:tasks.workunit.client.0.smithi077.stdout:removed './277/35' 2022-04-03T19:49:30.148 INFO:tasks.workunit.client.0.smithi077.stdout:removed './277/26' 2022-04-03T19:49:30.148 INFO:tasks.workunit.client.0.smithi077.stdout:removed './277/37' 2022-04-03T19:49:30.148 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './277' 2022-04-03T19:49:30.148 INFO:tasks.workunit.client.0.smithi077.stdout:removed './278/21' 2022-04-03T19:49:30.148 INFO:tasks.workunit.client.0.smithi077.stdout:removed './278/27' 2022-04-03T19:49:30.149 INFO:tasks.workunit.client.0.smithi077.stdout:removed './278/31' 2022-04-03T19:49:30.149 INFO:tasks.workunit.client.0.smithi077.stdout:removed './278/36' 2022-04-03T19:49:30.149 INFO:tasks.workunit.client.0.smithi077.stdout:removed './278/20' 2022-04-03T19:49:30.149 INFO:tasks.workunit.client.0.smithi077.stdout:removed './278/39' 2022-04-03T19:49:30.149 INFO:tasks.workunit.client.0.smithi077.stdout:removed './278/34' 2022-04-03T19:49:30.150 INFO:tasks.workunit.client.0.smithi077.stdout:removed './278/38' 2022-04-03T19:49:30.150 INFO:tasks.workunit.client.0.smithi077.stdout:removed './278/28' 2022-04-03T19:49:30.150 INFO:tasks.workunit.client.0.smithi077.stdout:removed './278/24' 2022-04-03T19:49:30.150 INFO:tasks.workunit.client.0.smithi077.stdout:removed './278/29' 2022-04-03T19:49:30.150 INFO:tasks.workunit.client.0.smithi077.stdout:removed './278/32' 2022-04-03T19:49:30.151 INFO:tasks.workunit.client.0.smithi077.stdout:removed './278/22' 2022-04-03T19:49:30.151 INFO:tasks.workunit.client.0.smithi077.stdout:removed './278/25' 2022-04-03T19:49:30.151 INFO:tasks.workunit.client.0.smithi077.stdout:removed './278/30' 2022-04-03T19:49:30.151 INFO:tasks.workunit.client.0.smithi077.stdout:removed './278/33' 2022-04-03T19:49:30.151 INFO:tasks.workunit.client.0.smithi077.stdout:removed './278/23' 2022-04-03T19:49:30.152 INFO:tasks.workunit.client.0.smithi077.stdout:removed './278/35' 2022-04-03T19:49:30.152 INFO:tasks.workunit.client.0.smithi077.stdout:removed './278/26' 2022-04-03T19:49:30.152 INFO:tasks.workunit.client.0.smithi077.stdout:removed './278/37' 2022-04-03T19:49:30.152 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './278' 2022-04-03T19:49:30.152 INFO:tasks.workunit.client.0.smithi077.stdout:removed './279/21' 2022-04-03T19:49:30.153 INFO:tasks.workunit.client.0.smithi077.stdout:removed './279/27' 2022-04-03T19:49:30.153 INFO:tasks.workunit.client.0.smithi077.stdout:removed './279/31' 2022-04-03T19:49:30.153 INFO:tasks.workunit.client.0.smithi077.stdout:removed './279/36' 2022-04-03T19:49:30.153 INFO:tasks.workunit.client.0.smithi077.stdout:removed './279/20' 2022-04-03T19:49:30.153 INFO:tasks.workunit.client.0.smithi077.stdout:removed './279/39' 2022-04-03T19:49:30.154 INFO:tasks.workunit.client.0.smithi077.stdout:removed './279/34' 2022-04-03T19:49:30.154 INFO:tasks.workunit.client.0.smithi077.stdout:removed './279/38' 2022-04-03T19:49:30.154 INFO:tasks.workunit.client.0.smithi077.stdout:removed './279/28' 2022-04-03T19:49:30.154 INFO:tasks.workunit.client.0.smithi077.stdout:removed './279/24' 2022-04-03T19:49:30.154 INFO:tasks.workunit.client.0.smithi077.stdout:removed './279/29' 2022-04-03T19:49:30.155 INFO:tasks.workunit.client.0.smithi077.stdout:removed './279/32' 2022-04-03T19:49:30.155 INFO:tasks.workunit.client.0.smithi077.stdout:removed './279/22' 2022-04-03T19:49:30.155 INFO:tasks.workunit.client.0.smithi077.stdout:removed './279/25' 2022-04-03T19:49:30.155 INFO:tasks.workunit.client.0.smithi077.stdout:removed './279/30' 2022-04-03T19:49:30.155 INFO:tasks.workunit.client.0.smithi077.stdout:removed './279/33' 2022-04-03T19:49:30.156 INFO:tasks.workunit.client.0.smithi077.stdout:removed './279/23' 2022-04-03T19:49:30.156 INFO:tasks.workunit.client.0.smithi077.stdout:removed './279/35' 2022-04-03T19:49:30.156 INFO:tasks.workunit.client.0.smithi077.stdout:removed './279/26' 2022-04-03T19:49:30.156 INFO:tasks.workunit.client.0.smithi077.stdout:removed './279/37' 2022-04-03T19:49:30.156 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './279' 2022-04-03T19:49:30.157 INFO:tasks.workunit.client.0.smithi077.stdout:removed './280/21' 2022-04-03T19:49:30.157 INFO:tasks.workunit.client.0.smithi077.stdout:removed './280/27' 2022-04-03T19:49:30.157 INFO:tasks.workunit.client.0.smithi077.stdout:removed './280/31' 2022-04-03T19:49:30.157 INFO:tasks.workunit.client.0.smithi077.stdout:removed './280/36' 2022-04-03T19:49:30.157 INFO:tasks.workunit.client.0.smithi077.stdout:removed './280/20' 2022-04-03T19:49:30.158 INFO:tasks.workunit.client.0.smithi077.stdout:removed './280/39' 2022-04-03T19:49:30.158 INFO:tasks.workunit.client.0.smithi077.stdout:removed './280/34' 2022-04-03T19:49:30.158 INFO:tasks.workunit.client.0.smithi077.stdout:removed './280/38' 2022-04-03T19:49:30.158 INFO:tasks.workunit.client.0.smithi077.stdout:removed './280/28' 2022-04-03T19:49:30.158 INFO:tasks.workunit.client.0.smithi077.stdout:removed './280/24' 2022-04-03T19:49:30.159 INFO:tasks.workunit.client.0.smithi077.stdout:removed './280/29' 2022-04-03T19:49:30.159 INFO:tasks.workunit.client.0.smithi077.stdout:removed './280/32' 2022-04-03T19:49:30.159 INFO:tasks.workunit.client.0.smithi077.stdout:removed './280/22' 2022-04-03T19:49:30.159 INFO:tasks.workunit.client.0.smithi077.stdout:removed './280/25' 2022-04-03T19:49:30.159 INFO:tasks.workunit.client.0.smithi077.stdout:removed './280/30' 2022-04-03T19:49:30.160 INFO:tasks.workunit.client.0.smithi077.stdout:removed './280/33' 2022-04-03T19:49:30.160 INFO:tasks.workunit.client.0.smithi077.stdout:removed './280/23' 2022-04-03T19:49:30.160 INFO:tasks.workunit.client.0.smithi077.stdout:removed './280/35' 2022-04-03T19:49:30.160 INFO:tasks.workunit.client.0.smithi077.stdout:removed './280/26' 2022-04-03T19:49:30.160 INFO:tasks.workunit.client.0.smithi077.stdout:removed './280/37' 2022-04-03T19:49:30.161 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './280' 2022-04-03T19:49:30.161 INFO:tasks.workunit.client.0.smithi077.stdout:removed './281/21' 2022-04-03T19:49:30.161 INFO:tasks.workunit.client.0.smithi077.stdout:removed './281/27' 2022-04-03T19:49:30.161 INFO:tasks.workunit.client.0.smithi077.stdout:removed './281/31' 2022-04-03T19:49:30.161 INFO:tasks.workunit.client.0.smithi077.stdout:removed './281/36' 2022-04-03T19:49:30.162 INFO:tasks.workunit.client.0.smithi077.stdout:removed './281/20' 2022-04-03T19:49:30.162 INFO:tasks.workunit.client.0.smithi077.stdout:removed './281/39' 2022-04-03T19:49:30.162 INFO:tasks.workunit.client.0.smithi077.stdout:removed './281/34' 2022-04-03T19:49:30.162 INFO:tasks.workunit.client.0.smithi077.stdout:removed './281/38' 2022-04-03T19:49:30.162 INFO:tasks.workunit.client.0.smithi077.stdout:removed './281/28' 2022-04-03T19:49:30.163 INFO:tasks.workunit.client.0.smithi077.stdout:removed './281/24' 2022-04-03T19:49:30.163 INFO:tasks.workunit.client.0.smithi077.stdout:removed './281/29' 2022-04-03T19:49:30.163 INFO:tasks.workunit.client.0.smithi077.stdout:removed './281/32' 2022-04-03T19:49:30.163 INFO:tasks.workunit.client.0.smithi077.stdout:removed './281/22' 2022-04-03T19:49:30.164 INFO:tasks.workunit.client.0.smithi077.stdout:removed './281/25' 2022-04-03T19:49:30.164 INFO:tasks.workunit.client.0.smithi077.stdout:removed './281/30' 2022-04-03T19:49:30.164 INFO:tasks.workunit.client.0.smithi077.stdout:removed './281/33' 2022-04-03T19:49:30.164 INFO:tasks.workunit.client.0.smithi077.stdout:removed './281/23' 2022-04-03T19:49:30.164 INFO:tasks.workunit.client.0.smithi077.stdout:removed './281/35' 2022-04-03T19:49:30.165 INFO:tasks.workunit.client.0.smithi077.stdout:removed './281/26' 2022-04-03T19:49:30.165 INFO:tasks.workunit.client.0.smithi077.stdout:removed './281/37' 2022-04-03T19:49:30.165 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './281' 2022-04-03T19:49:30.165 INFO:tasks.workunit.client.0.smithi077.stdout:removed './282/21' 2022-04-03T19:49:30.165 INFO:tasks.workunit.client.0.smithi077.stdout:removed './282/27' 2022-04-03T19:49:30.165 INFO:tasks.workunit.client.0.smithi077.stdout:removed './282/31' 2022-04-03T19:49:30.166 INFO:tasks.workunit.client.0.smithi077.stdout:removed './282/36' 2022-04-03T19:49:30.166 INFO:tasks.workunit.client.0.smithi077.stdout:removed './282/20' 2022-04-03T19:49:30.166 INFO:tasks.workunit.client.0.smithi077.stdout:removed './282/39' 2022-04-03T19:49:30.166 INFO:tasks.workunit.client.0.smithi077.stdout:removed './282/34' 2022-04-03T19:49:30.166 INFO:tasks.workunit.client.0.smithi077.stdout:removed './282/38' 2022-04-03T19:49:30.167 INFO:tasks.workunit.client.0.smithi077.stdout:removed './282/28' 2022-04-03T19:49:30.167 INFO:tasks.workunit.client.0.smithi077.stdout:removed './282/24' 2022-04-03T19:49:30.167 INFO:tasks.workunit.client.0.smithi077.stdout:removed './282/29' 2022-04-03T19:49:30.167 INFO:tasks.workunit.client.0.smithi077.stdout:removed './282/32' 2022-04-03T19:49:30.167 INFO:tasks.workunit.client.0.smithi077.stdout:removed './282/22' 2022-04-03T19:49:30.168 INFO:tasks.workunit.client.0.smithi077.stdout:removed './282/25' 2022-04-03T19:49:30.168 INFO:tasks.workunit.client.0.smithi077.stdout:removed './282/30' 2022-04-03T19:49:30.168 INFO:tasks.workunit.client.0.smithi077.stdout:removed './282/33' 2022-04-03T19:49:30.168 INFO:tasks.workunit.client.0.smithi077.stdout:removed './282/23' 2022-04-03T19:49:30.168 INFO:tasks.workunit.client.0.smithi077.stdout:removed './282/35' 2022-04-03T19:49:30.169 INFO:tasks.workunit.client.0.smithi077.stdout:removed './282/26' 2022-04-03T19:49:30.169 INFO:tasks.workunit.client.0.smithi077.stdout:removed './282/37' 2022-04-03T19:49:30.169 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './282' 2022-04-03T19:49:30.169 INFO:tasks.workunit.client.0.smithi077.stdout:removed './283/21' 2022-04-03T19:49:30.169 INFO:tasks.workunit.client.0.smithi077.stdout:removed './283/27' 2022-04-03T19:49:30.170 INFO:tasks.workunit.client.0.smithi077.stdout:removed './283/31' 2022-04-03T19:49:30.170 INFO:tasks.workunit.client.0.smithi077.stdout:removed './283/36' 2022-04-03T19:49:30.170 INFO:tasks.workunit.client.0.smithi077.stdout:removed './283/20' 2022-04-03T19:49:30.170 INFO:tasks.workunit.client.0.smithi077.stdout:removed './283/39' 2022-04-03T19:49:30.170 INFO:tasks.workunit.client.0.smithi077.stdout:removed './283/34' 2022-04-03T19:49:30.171 INFO:tasks.workunit.client.0.smithi077.stdout:removed './283/38' 2022-04-03T19:49:30.171 INFO:tasks.workunit.client.0.smithi077.stdout:removed './283/28' 2022-04-03T19:49:30.171 INFO:tasks.workunit.client.0.smithi077.stdout:removed './283/24' 2022-04-03T19:49:30.171 INFO:tasks.workunit.client.0.smithi077.stdout:removed './283/29' 2022-04-03T19:49:30.171 INFO:tasks.workunit.client.0.smithi077.stdout:removed './283/32' 2022-04-03T19:49:30.172 INFO:tasks.workunit.client.0.smithi077.stdout:removed './283/22' 2022-04-03T19:49:30.172 INFO:tasks.workunit.client.0.smithi077.stdout:removed './283/25' 2022-04-03T19:49:30.172 INFO:tasks.workunit.client.0.smithi077.stdout:removed './283/30' 2022-04-03T19:49:30.172 INFO:tasks.workunit.client.0.smithi077.stdout:removed './283/33' 2022-04-03T19:49:30.172 INFO:tasks.workunit.client.0.smithi077.stdout:removed './283/23' 2022-04-03T19:49:30.173 INFO:tasks.workunit.client.0.smithi077.stdout:removed './283/35' 2022-04-03T19:49:30.173 INFO:tasks.workunit.client.0.smithi077.stdout:removed './283/26' 2022-04-03T19:49:30.173 INFO:tasks.workunit.client.0.smithi077.stdout:removed './283/37' 2022-04-03T19:49:30.173 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './283' 2022-04-03T19:49:30.173 INFO:tasks.workunit.client.0.smithi077.stdout:removed './284/21' 2022-04-03T19:49:30.174 INFO:tasks.workunit.client.0.smithi077.stdout:removed './284/27' 2022-04-03T19:49:30.174 INFO:tasks.workunit.client.0.smithi077.stdout:removed './284/31' 2022-04-03T19:49:30.174 INFO:tasks.workunit.client.0.smithi077.stdout:removed './284/36' 2022-04-03T19:49:30.174 INFO:tasks.workunit.client.0.smithi077.stdout:removed './284/20' 2022-04-03T19:49:30.174 INFO:tasks.workunit.client.0.smithi077.stdout:removed './284/39' 2022-04-03T19:49:30.175 INFO:tasks.workunit.client.0.smithi077.stdout:removed './284/34' 2022-04-03T19:49:30.175 INFO:tasks.workunit.client.0.smithi077.stdout:removed './284/38' 2022-04-03T19:49:30.175 INFO:tasks.workunit.client.0.smithi077.stdout:removed './284/28' 2022-04-03T19:49:30.175 INFO:tasks.workunit.client.0.smithi077.stdout:removed './284/24' 2022-04-03T19:49:30.176 INFO:tasks.workunit.client.0.smithi077.stdout:removed './284/29' 2022-04-03T19:49:30.176 INFO:tasks.workunit.client.0.smithi077.stdout:removed './284/32' 2022-04-03T19:49:30.176 INFO:tasks.workunit.client.0.smithi077.stdout:removed './284/22' 2022-04-03T19:49:30.176 INFO:tasks.workunit.client.0.smithi077.stdout:removed './284/25' 2022-04-03T19:49:30.176 INFO:tasks.workunit.client.0.smithi077.stdout:removed './284/30' 2022-04-03T19:49:30.177 INFO:tasks.workunit.client.0.smithi077.stdout:removed './284/33' 2022-04-03T19:49:30.177 INFO:tasks.workunit.client.0.smithi077.stdout:removed './284/23' 2022-04-03T19:49:30.177 INFO:tasks.workunit.client.0.smithi077.stdout:removed './284/35' 2022-04-03T19:49:30.177 INFO:tasks.workunit.client.0.smithi077.stdout:removed './284/26' 2022-04-03T19:49:30.177 INFO:tasks.workunit.client.0.smithi077.stdout:removed './284/37' 2022-04-03T19:49:30.177 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './284' 2022-04-03T19:49:30.178 INFO:tasks.workunit.client.0.smithi077.stdout:removed './285/21' 2022-04-03T19:49:30.178 INFO:tasks.workunit.client.0.smithi077.stdout:removed './285/27' 2022-04-03T19:49:30.178 INFO:tasks.workunit.client.0.smithi077.stdout:removed './285/31' 2022-04-03T19:49:30.178 INFO:tasks.workunit.client.0.smithi077.stdout:removed './285/36' 2022-04-03T19:49:30.178 INFO:tasks.workunit.client.0.smithi077.stdout:removed './285/20' 2022-04-03T19:49:30.179 INFO:tasks.workunit.client.0.smithi077.stdout:removed './285/39' 2022-04-03T19:49:30.179 INFO:tasks.workunit.client.0.smithi077.stdout:removed './285/34' 2022-04-03T19:49:30.179 INFO:tasks.workunit.client.0.smithi077.stdout:removed './285/38' 2022-04-03T19:49:30.179 INFO:tasks.workunit.client.0.smithi077.stdout:removed './285/28' 2022-04-03T19:49:30.180 INFO:tasks.workunit.client.0.smithi077.stdout:removed './285/24' 2022-04-03T19:49:30.180 INFO:tasks.workunit.client.0.smithi077.stdout:removed './285/29' 2022-04-03T19:49:30.180 INFO:tasks.workunit.client.0.smithi077.stdout:removed './285/32' 2022-04-03T19:49:30.180 INFO:tasks.workunit.client.0.smithi077.stdout:removed './285/22' 2022-04-03T19:49:30.180 INFO:tasks.workunit.client.0.smithi077.stdout:removed './285/25' 2022-04-03T19:49:30.181 INFO:tasks.workunit.client.0.smithi077.stdout:removed './285/30' 2022-04-03T19:49:30.181 INFO:tasks.workunit.client.0.smithi077.stdout:removed './285/33' 2022-04-03T19:49:30.181 INFO:tasks.workunit.client.0.smithi077.stdout:removed './285/23' 2022-04-03T19:49:30.181 INFO:tasks.workunit.client.0.smithi077.stdout:removed './285/35' 2022-04-03T19:49:30.181 INFO:tasks.workunit.client.0.smithi077.stdout:removed './285/26' 2022-04-03T19:49:30.182 INFO:tasks.workunit.client.0.smithi077.stdout:removed './285/37' 2022-04-03T19:49:30.182 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './285' 2022-04-03T19:49:30.182 INFO:tasks.workunit.client.0.smithi077.stdout:removed './286/21' 2022-04-03T19:49:30.182 INFO:tasks.workunit.client.0.smithi077.stdout:removed './286/27' 2022-04-03T19:49:30.182 INFO:tasks.workunit.client.0.smithi077.stdout:removed './286/31' 2022-04-03T19:49:30.183 INFO:tasks.workunit.client.0.smithi077.stdout:removed './286/36' 2022-04-03T19:49:30.183 INFO:tasks.workunit.client.0.smithi077.stdout:removed './286/20' 2022-04-03T19:49:30.183 INFO:tasks.workunit.client.0.smithi077.stdout:removed './286/39' 2022-04-03T19:49:30.183 INFO:tasks.workunit.client.0.smithi077.stdout:removed './286/34' 2022-04-03T19:49:30.183 INFO:tasks.workunit.client.0.smithi077.stdout:removed './286/38' 2022-04-03T19:49:30.184 INFO:tasks.workunit.client.0.smithi077.stdout:removed './286/28' 2022-04-03T19:49:30.184 INFO:tasks.workunit.client.0.smithi077.stdout:removed './286/24' 2022-04-03T19:49:30.184 INFO:tasks.workunit.client.0.smithi077.stdout:removed './286/29' 2022-04-03T19:49:30.184 INFO:tasks.workunit.client.0.smithi077.stdout:removed './286/32' 2022-04-03T19:49:30.184 INFO:tasks.workunit.client.0.smithi077.stdout:removed './286/22' 2022-04-03T19:49:30.185 INFO:tasks.workunit.client.0.smithi077.stdout:removed './286/25' 2022-04-03T19:49:30.185 INFO:tasks.workunit.client.0.smithi077.stdout:removed './286/30' 2022-04-03T19:49:30.185 INFO:tasks.workunit.client.0.smithi077.stdout:removed './286/33' 2022-04-03T19:49:30.185 INFO:tasks.workunit.client.0.smithi077.stdout:removed './286/23' 2022-04-03T19:49:30.185 INFO:tasks.workunit.client.0.smithi077.stdout:removed './286/35' 2022-04-03T19:49:30.186 INFO:tasks.workunit.client.0.smithi077.stdout:removed './286/26' 2022-04-03T19:49:30.186 INFO:tasks.workunit.client.0.smithi077.stdout:removed './286/37' 2022-04-03T19:49:30.186 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './286' 2022-04-03T19:49:31.709 INFO:tasks.workunit.client.0.smithi077.stdout:removed './287/21' 2022-04-03T19:49:31.709 INFO:tasks.workunit.client.0.smithi077.stdout:removed './287/27' 2022-04-03T19:49:31.710 INFO:tasks.workunit.client.0.smithi077.stdout:removed './287/31' 2022-04-03T19:49:31.710 INFO:tasks.workunit.client.0.smithi077.stdout:removed './287/36' 2022-04-03T19:49:31.710 INFO:tasks.workunit.client.0.smithi077.stdout:removed './287/20' 2022-04-03T19:49:31.711 INFO:tasks.workunit.client.0.smithi077.stdout:removed './287/39' 2022-04-03T19:49:31.711 INFO:tasks.workunit.client.0.smithi077.stdout:removed './287/34' 2022-04-03T19:49:31.711 INFO:tasks.workunit.client.0.smithi077.stdout:removed './287/38' 2022-04-03T19:49:31.711 INFO:tasks.workunit.client.0.smithi077.stdout:removed './287/28' 2022-04-03T19:49:31.712 INFO:tasks.workunit.client.0.smithi077.stdout:removed './287/24' 2022-04-03T19:49:31.712 INFO:tasks.workunit.client.0.smithi077.stdout:removed './287/29' 2022-04-03T19:49:31.712 INFO:tasks.workunit.client.0.smithi077.stdout:removed './287/32' 2022-04-03T19:49:31.712 INFO:tasks.workunit.client.0.smithi077.stdout:removed './287/22' 2022-04-03T19:49:31.713 INFO:tasks.workunit.client.0.smithi077.stdout:removed './287/25' 2022-04-03T19:49:31.713 INFO:tasks.workunit.client.0.smithi077.stdout:removed './287/30' 2022-04-03T19:49:31.713 INFO:tasks.workunit.client.0.smithi077.stdout:removed './287/33' 2022-04-03T19:49:31.713 INFO:tasks.workunit.client.0.smithi077.stdout:removed './287/23' 2022-04-03T19:49:31.713 INFO:tasks.workunit.client.0.smithi077.stdout:removed './287/35' 2022-04-03T19:49:31.714 INFO:tasks.workunit.client.0.smithi077.stdout:removed './287/26' 2022-04-03T19:49:31.714 INFO:tasks.workunit.client.0.smithi077.stdout:removed './287/37' 2022-04-03T19:49:31.714 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './287' 2022-04-03T19:49:31.714 INFO:tasks.workunit.client.0.smithi077.stdout:removed './288/21' 2022-04-03T19:49:31.714 INFO:tasks.workunit.client.0.smithi077.stdout:removed './288/27' 2022-04-03T19:49:31.715 INFO:tasks.workunit.client.0.smithi077.stdout:removed './288/31' 2022-04-03T19:49:31.715 INFO:tasks.workunit.client.0.smithi077.stdout:removed './288/36' 2022-04-03T19:49:31.715 INFO:tasks.workunit.client.0.smithi077.stdout:removed './288/20' 2022-04-03T19:49:31.715 INFO:tasks.workunit.client.0.smithi077.stdout:removed './288/39' 2022-04-03T19:49:31.716 INFO:tasks.workunit.client.0.smithi077.stdout:removed './288/34' 2022-04-03T19:49:31.716 INFO:tasks.workunit.client.0.smithi077.stdout:removed './288/38' 2022-04-03T19:49:31.716 INFO:tasks.workunit.client.0.smithi077.stdout:removed './288/28' 2022-04-03T19:49:31.716 INFO:tasks.workunit.client.0.smithi077.stdout:removed './288/24' 2022-04-03T19:49:31.716 INFO:tasks.workunit.client.0.smithi077.stdout:removed './288/29' 2022-04-03T19:49:31.717 INFO:tasks.workunit.client.0.smithi077.stdout:removed './288/32' 2022-04-03T19:49:31.717 INFO:tasks.workunit.client.0.smithi077.stdout:removed './288/22' 2022-04-03T19:49:31.717 INFO:tasks.workunit.client.0.smithi077.stdout:removed './288/25' 2022-04-03T19:49:31.717 INFO:tasks.workunit.client.0.smithi077.stdout:removed './288/30' 2022-04-03T19:49:31.718 INFO:tasks.workunit.client.0.smithi077.stdout:removed './288/33' 2022-04-03T19:49:31.718 INFO:tasks.workunit.client.0.smithi077.stdout:removed './288/23' 2022-04-03T19:49:31.718 INFO:tasks.workunit.client.0.smithi077.stdout:removed './288/35' 2022-04-03T19:49:31.718 INFO:tasks.workunit.client.0.smithi077.stdout:removed './288/26' 2022-04-03T19:49:31.718 INFO:tasks.workunit.client.0.smithi077.stdout:removed './288/37' 2022-04-03T19:49:31.719 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './288' 2022-04-03T19:49:31.719 INFO:tasks.workunit.client.0.smithi077.stdout:removed './289/21' 2022-04-03T19:49:31.719 INFO:tasks.workunit.client.0.smithi077.stdout:removed './289/27' 2022-04-03T19:49:31.719 INFO:tasks.workunit.client.0.smithi077.stdout:removed './289/31' 2022-04-03T19:49:31.719 INFO:tasks.workunit.client.0.smithi077.stdout:removed './289/36' 2022-04-03T19:49:31.720 INFO:tasks.workunit.client.0.smithi077.stdout:removed './289/20' 2022-04-03T19:49:31.720 INFO:tasks.workunit.client.0.smithi077.stdout:removed './289/39' 2022-04-03T19:49:31.720 INFO:tasks.workunit.client.0.smithi077.stdout:removed './289/34' 2022-04-03T19:49:31.721 INFO:tasks.workunit.client.0.smithi077.stdout:removed './289/38' 2022-04-03T19:49:31.721 INFO:tasks.workunit.client.0.smithi077.stdout:removed './289/28' 2022-04-03T19:49:31.721 INFO:tasks.workunit.client.0.smithi077.stdout:removed './289/24' 2022-04-03T19:49:31.722 INFO:tasks.workunit.client.0.smithi077.stdout:removed './289/29' 2022-04-03T19:49:31.722 INFO:tasks.workunit.client.0.smithi077.stdout:removed './289/32' 2022-04-03T19:49:31.722 INFO:tasks.workunit.client.0.smithi077.stdout:removed './289/22' 2022-04-03T19:49:31.722 INFO:tasks.workunit.client.0.smithi077.stdout:removed './289/25' 2022-04-03T19:49:31.723 INFO:tasks.workunit.client.0.smithi077.stdout:removed './289/30' 2022-04-03T19:49:31.723 INFO:tasks.workunit.client.0.smithi077.stdout:removed './289/33' 2022-04-03T19:49:31.723 INFO:tasks.workunit.client.0.smithi077.stdout:removed './289/23' 2022-04-03T19:49:31.723 INFO:tasks.workunit.client.0.smithi077.stdout:removed './289/35' 2022-04-03T19:49:31.724 INFO:tasks.workunit.client.0.smithi077.stdout:removed './289/26' 2022-04-03T19:49:31.724 INFO:tasks.workunit.client.0.smithi077.stdout:removed './289/37' 2022-04-03T19:49:31.724 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './289' 2022-04-03T19:49:31.724 INFO:tasks.workunit.client.0.smithi077.stdout:removed './290/21' 2022-04-03T19:49:31.725 INFO:tasks.workunit.client.0.smithi077.stdout:removed './290/27' 2022-04-03T19:49:31.725 INFO:tasks.workunit.client.0.smithi077.stdout:removed './290/31' 2022-04-03T19:49:31.725 INFO:tasks.workunit.client.0.smithi077.stdout:removed './290/36' 2022-04-03T19:49:31.726 INFO:tasks.workunit.client.0.smithi077.stdout:removed './290/20' 2022-04-03T19:49:31.726 INFO:tasks.workunit.client.0.smithi077.stdout:removed './290/39' 2022-04-03T19:49:31.726 INFO:tasks.workunit.client.0.smithi077.stdout:removed './290/34' 2022-04-03T19:49:31.726 INFO:tasks.workunit.client.0.smithi077.stdout:removed './290/38' 2022-04-03T19:49:31.727 INFO:tasks.workunit.client.0.smithi077.stdout:removed './290/28' 2022-04-03T19:49:31.727 INFO:tasks.workunit.client.0.smithi077.stdout:removed './290/24' 2022-04-03T19:49:31.727 INFO:tasks.workunit.client.0.smithi077.stdout:removed './290/29' 2022-04-03T19:49:31.727 INFO:tasks.workunit.client.0.smithi077.stdout:removed './290/32' 2022-04-03T19:49:31.728 INFO:tasks.workunit.client.0.smithi077.stdout:removed './290/22' 2022-04-03T19:49:31.728 INFO:tasks.workunit.client.0.smithi077.stdout:removed './290/25' 2022-04-03T19:49:31.728 INFO:tasks.workunit.client.0.smithi077.stdout:removed './290/30' 2022-04-03T19:49:31.728 INFO:tasks.workunit.client.0.smithi077.stdout:removed './290/33' 2022-04-03T19:49:31.729 INFO:tasks.workunit.client.0.smithi077.stdout:removed './290/23' 2022-04-03T19:49:31.729 INFO:tasks.workunit.client.0.smithi077.stdout:removed './290/35' 2022-04-03T19:49:31.729 INFO:tasks.workunit.client.0.smithi077.stdout:removed './290/26' 2022-04-03T19:49:31.729 INFO:tasks.workunit.client.0.smithi077.stdout:removed './290/37' 2022-04-03T19:49:31.729 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './290' 2022-04-03T19:49:31.730 INFO:tasks.workunit.client.0.smithi077.stdout:removed './291/21' 2022-04-03T19:49:31.730 INFO:tasks.workunit.client.0.smithi077.stdout:removed './291/27' 2022-04-03T19:49:31.730 INFO:tasks.workunit.client.0.smithi077.stdout:removed './291/31' 2022-04-03T19:49:31.730 INFO:tasks.workunit.client.0.smithi077.stdout:removed './291/36' 2022-04-03T19:49:31.731 INFO:tasks.workunit.client.0.smithi077.stdout:removed './291/20' 2022-04-03T19:49:31.731 INFO:tasks.workunit.client.0.smithi077.stdout:removed './291/39' 2022-04-03T19:49:31.731 INFO:tasks.workunit.client.0.smithi077.stdout:removed './291/34' 2022-04-03T19:49:31.731 INFO:tasks.workunit.client.0.smithi077.stdout:removed './291/38' 2022-04-03T19:49:31.731 INFO:tasks.workunit.client.0.smithi077.stdout:removed './291/28' 2022-04-03T19:49:31.732 INFO:tasks.workunit.client.0.smithi077.stdout:removed './291/24' 2022-04-03T19:49:31.732 INFO:tasks.workunit.client.0.smithi077.stdout:removed './291/29' 2022-04-03T19:49:31.732 INFO:tasks.workunit.client.0.smithi077.stdout:removed './291/32' 2022-04-03T19:49:31.732 INFO:tasks.workunit.client.0.smithi077.stdout:removed './291/22' 2022-04-03T19:49:31.732 INFO:tasks.workunit.client.0.smithi077.stdout:removed './291/25' 2022-04-03T19:49:31.733 INFO:tasks.workunit.client.0.smithi077.stdout:removed './291/30' 2022-04-03T19:49:31.733 INFO:tasks.workunit.client.0.smithi077.stdout:removed './291/33' 2022-04-03T19:49:31.733 INFO:tasks.workunit.client.0.smithi077.stdout:removed './291/23' 2022-04-03T19:49:31.733 INFO:tasks.workunit.client.0.smithi077.stdout:removed './291/35' 2022-04-03T19:49:31.734 INFO:tasks.workunit.client.0.smithi077.stdout:removed './291/26' 2022-04-03T19:49:31.734 INFO:tasks.workunit.client.0.smithi077.stdout:removed './291/37' 2022-04-03T19:49:31.734 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './291' 2022-04-03T19:49:31.734 INFO:tasks.workunit.client.0.smithi077.stdout:removed './292/21' 2022-04-03T19:49:31.735 INFO:tasks.workunit.client.0.smithi077.stdout:removed './292/27' 2022-04-03T19:49:31.735 INFO:tasks.workunit.client.0.smithi077.stdout:removed './292/31' 2022-04-03T19:49:31.735 INFO:tasks.workunit.client.0.smithi077.stdout:removed './292/36' 2022-04-03T19:49:31.735 INFO:tasks.workunit.client.0.smithi077.stdout:removed './292/20' 2022-04-03T19:49:31.735 INFO:tasks.workunit.client.0.smithi077.stdout:removed './292/39' 2022-04-03T19:49:31.736 INFO:tasks.workunit.client.0.smithi077.stdout:removed './292/34' 2022-04-03T19:49:31.736 INFO:tasks.workunit.client.0.smithi077.stdout:removed './292/38' 2022-04-03T19:49:31.736 INFO:tasks.workunit.client.0.smithi077.stdout:removed './292/28' 2022-04-03T19:49:31.736 INFO:tasks.workunit.client.0.smithi077.stdout:removed './292/24' 2022-04-03T19:49:31.737 INFO:tasks.workunit.client.0.smithi077.stdout:removed './292/29' 2022-04-03T19:49:31.737 INFO:tasks.workunit.client.0.smithi077.stdout:removed './292/32' 2022-04-03T19:49:31.737 INFO:tasks.workunit.client.0.smithi077.stdout:removed './292/22' 2022-04-03T19:49:31.737 INFO:tasks.workunit.client.0.smithi077.stdout:removed './292/25' 2022-04-03T19:49:31.737 INFO:tasks.workunit.client.0.smithi077.stdout:removed './292/30' 2022-04-03T19:49:31.738 INFO:tasks.workunit.client.0.smithi077.stdout:removed './292/33' 2022-04-03T19:49:31.738 INFO:tasks.workunit.client.0.smithi077.stdout:removed './292/23' 2022-04-03T19:49:31.738 INFO:tasks.workunit.client.0.smithi077.stdout:removed './292/35' 2022-04-03T19:49:31.738 INFO:tasks.workunit.client.0.smithi077.stdout:removed './292/26' 2022-04-03T19:49:31.739 INFO:tasks.workunit.client.0.smithi077.stdout:removed './292/37' 2022-04-03T19:49:31.739 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './292' 2022-04-03T19:49:31.739 INFO:tasks.workunit.client.0.smithi077.stdout:removed './293/21' 2022-04-03T19:49:31.739 INFO:tasks.workunit.client.0.smithi077.stdout:removed './293/27' 2022-04-03T19:49:31.740 INFO:tasks.workunit.client.0.smithi077.stdout:removed './293/31' 2022-04-03T19:49:31.740 INFO:tasks.workunit.client.0.smithi077.stdout:removed './293/36' 2022-04-03T19:49:31.740 INFO:tasks.workunit.client.0.smithi077.stdout:removed './293/20' 2022-04-03T19:49:31.740 INFO:tasks.workunit.client.0.smithi077.stdout:removed './293/39' 2022-04-03T19:49:31.740 INFO:tasks.workunit.client.0.smithi077.stdout:removed './293/34' 2022-04-03T19:49:31.741 INFO:tasks.workunit.client.0.smithi077.stdout:removed './293/38' 2022-04-03T19:49:31.741 INFO:tasks.workunit.client.0.smithi077.stdout:removed './293/28' 2022-04-03T19:49:31.741 INFO:tasks.workunit.client.0.smithi077.stdout:removed './293/24' 2022-04-03T19:49:31.741 INFO:tasks.workunit.client.0.smithi077.stdout:removed './293/29' 2022-04-03T19:49:31.742 INFO:tasks.workunit.client.0.smithi077.stdout:removed './293/32' 2022-04-03T19:49:31.742 INFO:tasks.workunit.client.0.smithi077.stdout:removed './293/22' 2022-04-03T19:49:31.742 INFO:tasks.workunit.client.0.smithi077.stdout:removed './293/25' 2022-04-03T19:49:31.742 INFO:tasks.workunit.client.0.smithi077.stdout:removed './293/30' 2022-04-03T19:49:31.743 INFO:tasks.workunit.client.0.smithi077.stdout:removed './293/33' 2022-04-03T19:49:31.743 INFO:tasks.workunit.client.0.smithi077.stdout:removed './293/23' 2022-04-03T19:49:31.743 INFO:tasks.workunit.client.0.smithi077.stdout:removed './293/35' 2022-04-03T19:49:31.744 INFO:tasks.workunit.client.0.smithi077.stdout:removed './293/26' 2022-04-03T19:49:31.744 INFO:tasks.workunit.client.0.smithi077.stdout:removed './293/37' 2022-04-03T19:49:31.744 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './293' 2022-04-03T19:49:31.745 INFO:tasks.workunit.client.0.smithi077.stdout:removed './294/21' 2022-04-03T19:49:31.745 INFO:tasks.workunit.client.0.smithi077.stdout:removed './294/27' 2022-04-03T19:49:31.745 INFO:tasks.workunit.client.0.smithi077.stdout:removed './294/31' 2022-04-03T19:49:31.745 INFO:tasks.workunit.client.0.smithi077.stdout:removed './294/36' 2022-04-03T19:49:31.746 INFO:tasks.workunit.client.0.smithi077.stdout:removed './294/20' 2022-04-03T19:49:31.746 INFO:tasks.workunit.client.0.smithi077.stdout:removed './294/39' 2022-04-03T19:49:31.746 INFO:tasks.workunit.client.0.smithi077.stdout:removed './294/34' 2022-04-03T19:49:31.747 INFO:tasks.workunit.client.0.smithi077.stdout:removed './294/38' 2022-04-03T19:49:31.747 INFO:tasks.workunit.client.0.smithi077.stdout:removed './294/28' 2022-04-03T19:49:31.747 INFO:tasks.workunit.client.0.smithi077.stdout:removed './294/24' 2022-04-03T19:49:31.747 INFO:tasks.workunit.client.0.smithi077.stdout:removed './294/29' 2022-04-03T19:49:31.748 INFO:tasks.workunit.client.0.smithi077.stdout:removed './294/32' 2022-04-03T19:49:31.748 INFO:tasks.workunit.client.0.smithi077.stdout:removed './294/22' 2022-04-03T19:49:31.748 INFO:tasks.workunit.client.0.smithi077.stdout:removed './294/25' 2022-04-03T19:49:31.748 INFO:tasks.workunit.client.0.smithi077.stdout:removed './294/30' 2022-04-03T19:49:31.749 INFO:tasks.workunit.client.0.smithi077.stdout:removed './294/33' 2022-04-03T19:49:31.749 INFO:tasks.workunit.client.0.smithi077.stdout:removed './294/23' 2022-04-03T19:49:31.749 INFO:tasks.workunit.client.0.smithi077.stdout:removed './294/35' 2022-04-03T19:49:31.749 INFO:tasks.workunit.client.0.smithi077.stdout:removed './294/26' 2022-04-03T19:49:31.749 INFO:tasks.workunit.client.0.smithi077.stdout:removed './294/37' 2022-04-03T19:49:31.750 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './294' 2022-04-03T19:49:31.750 INFO:tasks.workunit.client.0.smithi077.stdout:removed './295/21' 2022-04-03T19:49:31.750 INFO:tasks.workunit.client.0.smithi077.stdout:removed './295/27' 2022-04-03T19:49:31.750 INFO:tasks.workunit.client.0.smithi077.stdout:removed './295/31' 2022-04-03T19:49:31.751 INFO:tasks.workunit.client.0.smithi077.stdout:removed './295/36' 2022-04-03T19:49:31.751 INFO:tasks.workunit.client.0.smithi077.stdout:removed './295/20' 2022-04-03T19:49:31.751 INFO:tasks.workunit.client.0.smithi077.stdout:removed './295/39' 2022-04-03T19:49:31.751 INFO:tasks.workunit.client.0.smithi077.stdout:removed './295/34' 2022-04-03T19:49:31.752 INFO:tasks.workunit.client.0.smithi077.stdout:removed './295/38' 2022-04-03T19:49:31.752 INFO:tasks.workunit.client.0.smithi077.stdout:removed './295/28' 2022-04-03T19:49:31.752 INFO:tasks.workunit.client.0.smithi077.stdout:removed './295/24' 2022-04-03T19:49:31.752 INFO:tasks.workunit.client.0.smithi077.stdout:removed './295/29' 2022-04-03T19:49:31.753 INFO:tasks.workunit.client.0.smithi077.stdout:removed './295/32' 2022-04-03T19:49:31.753 INFO:tasks.workunit.client.0.smithi077.stdout:removed './295/22' 2022-04-03T19:49:31.753 INFO:tasks.workunit.client.0.smithi077.stdout:removed './295/25' 2022-04-03T19:49:31.753 INFO:tasks.workunit.client.0.smithi077.stdout:removed './295/30' 2022-04-03T19:49:31.754 INFO:tasks.workunit.client.0.smithi077.stdout:removed './295/33' 2022-04-03T19:49:31.754 INFO:tasks.workunit.client.0.smithi077.stdout:removed './295/23' 2022-04-03T19:49:31.754 INFO:tasks.workunit.client.0.smithi077.stdout:removed './295/35' 2022-04-03T19:49:31.754 INFO:tasks.workunit.client.0.smithi077.stdout:removed './295/26' 2022-04-03T19:49:31.755 INFO:tasks.workunit.client.0.smithi077.stdout:removed './295/37' 2022-04-03T19:49:31.755 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './295' 2022-04-03T19:49:31.755 INFO:tasks.workunit.client.0.smithi077.stdout:removed './296/21' 2022-04-03T19:49:31.755 INFO:tasks.workunit.client.0.smithi077.stdout:removed './296/27' 2022-04-03T19:49:31.756 INFO:tasks.workunit.client.0.smithi077.stdout:removed './296/31' 2022-04-03T19:49:31.756 INFO:tasks.workunit.client.0.smithi077.stdout:removed './296/36' 2022-04-03T19:49:31.756 INFO:tasks.workunit.client.0.smithi077.stdout:removed './296/20' 2022-04-03T19:49:31.756 INFO:tasks.workunit.client.0.smithi077.stdout:removed './296/39' 2022-04-03T19:49:31.757 INFO:tasks.workunit.client.0.smithi077.stdout:removed './296/34' 2022-04-03T19:49:31.757 INFO:tasks.workunit.client.0.smithi077.stdout:removed './296/38' 2022-04-03T19:49:31.757 INFO:tasks.workunit.client.0.smithi077.stdout:removed './296/28' 2022-04-03T19:49:31.757 INFO:tasks.workunit.client.0.smithi077.stdout:removed './296/24' 2022-04-03T19:49:31.758 INFO:tasks.workunit.client.0.smithi077.stdout:removed './296/29' 2022-04-03T19:49:31.758 INFO:tasks.workunit.client.0.smithi077.stdout:removed './296/32' 2022-04-03T19:49:31.758 INFO:tasks.workunit.client.0.smithi077.stdout:removed './296/22' 2022-04-03T19:49:31.759 INFO:tasks.workunit.client.0.smithi077.stdout:removed './296/25' 2022-04-03T19:49:31.759 INFO:tasks.workunit.client.0.smithi077.stdout:removed './296/30' 2022-04-03T19:49:31.759 INFO:tasks.workunit.client.0.smithi077.stdout:removed './296/33' 2022-04-03T19:49:31.759 INFO:tasks.workunit.client.0.smithi077.stdout:removed './296/23' 2022-04-03T19:49:31.760 INFO:tasks.workunit.client.0.smithi077.stdout:removed './296/35' 2022-04-03T19:49:31.760 INFO:tasks.workunit.client.0.smithi077.stdout:removed './296/26' 2022-04-03T19:49:31.760 INFO:tasks.workunit.client.0.smithi077.stdout:removed './296/37' 2022-04-03T19:49:31.760 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './296' 2022-04-03T19:49:31.761 INFO:tasks.workunit.client.0.smithi077.stdout:removed './297/21' 2022-04-03T19:49:31.761 INFO:tasks.workunit.client.0.smithi077.stdout:removed './297/27' 2022-04-03T19:49:34.643 INFO:tasks.workunit.client.0.smithi077.stdout:removed './297/31' 2022-04-03T19:49:34.643 INFO:tasks.workunit.client.0.smithi077.stdout:removed './297/36' 2022-04-03T19:49:34.643 INFO:tasks.workunit.client.0.smithi077.stdout:removed './297/20' 2022-04-03T19:49:34.644 INFO:tasks.workunit.client.0.smithi077.stdout:removed './297/39' 2022-04-03T19:49:34.644 INFO:tasks.workunit.client.0.smithi077.stdout:removed './297/34' 2022-04-03T19:49:34.644 INFO:tasks.workunit.client.0.smithi077.stdout:removed './297/38' 2022-04-03T19:49:34.644 INFO:tasks.workunit.client.0.smithi077.stdout:removed './297/28' 2022-04-03T19:49:34.645 INFO:tasks.workunit.client.0.smithi077.stdout:removed './297/24' 2022-04-03T19:49:34.645 INFO:tasks.workunit.client.0.smithi077.stdout:removed './297/29' 2022-04-03T19:49:34.645 INFO:tasks.workunit.client.0.smithi077.stdout:removed './297/32' 2022-04-03T19:49:34.645 INFO:tasks.workunit.client.0.smithi077.stdout:removed './297/22' 2022-04-03T19:49:34.645 INFO:tasks.workunit.client.0.smithi077.stdout:removed './297/25' 2022-04-03T19:49:34.646 INFO:tasks.workunit.client.0.smithi077.stdout:removed './297/30' 2022-04-03T19:49:34.646 INFO:tasks.workunit.client.0.smithi077.stdout:removed './297/33' 2022-04-03T19:49:34.646 INFO:tasks.workunit.client.0.smithi077.stdout:removed './297/23' 2022-04-03T19:49:34.646 INFO:tasks.workunit.client.0.smithi077.stdout:removed './297/35' 2022-04-03T19:49:34.647 INFO:tasks.workunit.client.0.smithi077.stdout:removed './297/26' 2022-04-03T19:49:34.647 INFO:tasks.workunit.client.0.smithi077.stdout:removed './297/37' 2022-04-03T19:49:34.647 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './297' 2022-04-03T19:49:34.647 INFO:tasks.workunit.client.0.smithi077.stdout:removed './298/21' 2022-04-03T19:49:34.647 INFO:tasks.workunit.client.0.smithi077.stdout:removed './298/27' 2022-04-03T19:49:34.648 INFO:tasks.workunit.client.0.smithi077.stdout:removed './298/31' 2022-04-03T19:49:34.648 INFO:tasks.workunit.client.0.smithi077.stdout:removed './298/36' 2022-04-03T19:49:34.648 INFO:tasks.workunit.client.0.smithi077.stdout:removed './298/20' 2022-04-03T19:49:34.648 INFO:tasks.workunit.client.0.smithi077.stdout:removed './298/39' 2022-04-03T19:49:34.649 INFO:tasks.workunit.client.0.smithi077.stdout:removed './298/34' 2022-04-03T19:49:34.649 INFO:tasks.workunit.client.0.smithi077.stdout:removed './298/38' 2022-04-03T19:49:34.649 INFO:tasks.workunit.client.0.smithi077.stdout:removed './298/28' 2022-04-03T19:49:34.649 INFO:tasks.workunit.client.0.smithi077.stdout:removed './298/24' 2022-04-03T19:49:34.649 INFO:tasks.workunit.client.0.smithi077.stdout:removed './298/29' 2022-04-03T19:49:34.650 INFO:tasks.workunit.client.0.smithi077.stdout:removed './298/32' 2022-04-03T19:49:34.650 INFO:tasks.workunit.client.0.smithi077.stdout:removed './298/22' 2022-04-03T19:49:34.650 INFO:tasks.workunit.client.0.smithi077.stdout:removed './298/25' 2022-04-03T19:49:34.650 INFO:tasks.workunit.client.0.smithi077.stdout:removed './298/30' 2022-04-03T19:49:34.651 INFO:tasks.workunit.client.0.smithi077.stdout:removed './298/33' 2022-04-03T19:49:34.651 INFO:tasks.workunit.client.0.smithi077.stdout:removed './298/23' 2022-04-03T19:49:34.651 INFO:tasks.workunit.client.0.smithi077.stdout:removed './298/35' 2022-04-03T19:49:34.651 INFO:tasks.workunit.client.0.smithi077.stdout:removed './298/26' 2022-04-03T19:49:34.652 INFO:tasks.workunit.client.0.smithi077.stdout:removed './298/37' 2022-04-03T19:49:34.652 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './298' 2022-04-03T19:49:34.652 INFO:tasks.workunit.client.0.smithi077.stdout:removed './299/21' 2022-04-03T19:49:34.652 INFO:tasks.workunit.client.0.smithi077.stdout:removed './299/27' 2022-04-03T19:49:34.653 INFO:tasks.workunit.client.0.smithi077.stdout:removed './299/31' 2022-04-03T19:49:34.653 INFO:tasks.workunit.client.0.smithi077.stdout:removed './299/36' 2022-04-03T19:49:34.653 INFO:tasks.workunit.client.0.smithi077.stdout:removed './299/20' 2022-04-03T19:49:34.653 INFO:tasks.workunit.client.0.smithi077.stdout:removed './299/39' 2022-04-03T19:49:34.654 INFO:tasks.workunit.client.0.smithi077.stdout:removed './299/34' 2022-04-03T19:49:34.654 INFO:tasks.workunit.client.0.smithi077.stdout:removed './299/38' 2022-04-03T19:49:34.654 INFO:tasks.workunit.client.0.smithi077.stdout:removed './299/28' 2022-04-03T19:49:34.655 INFO:tasks.workunit.client.0.smithi077.stdout:removed './299/24' 2022-04-03T19:49:34.655 INFO:tasks.workunit.client.0.smithi077.stdout:removed './299/29' 2022-04-03T19:49:34.655 INFO:tasks.workunit.client.0.smithi077.stdout:removed './299/32' 2022-04-03T19:49:34.655 INFO:tasks.workunit.client.0.smithi077.stdout:removed './299/22' 2022-04-03T19:49:34.655 INFO:tasks.workunit.client.0.smithi077.stdout:removed './299/25' 2022-04-03T19:49:34.656 INFO:tasks.workunit.client.0.smithi077.stdout:removed './299/30' 2022-04-03T19:49:34.656 INFO:tasks.workunit.client.0.smithi077.stdout:removed './299/33' 2022-04-03T19:49:34.656 INFO:tasks.workunit.client.0.smithi077.stdout:removed './299/23' 2022-04-03T19:49:34.656 INFO:tasks.workunit.client.0.smithi077.stdout:removed './299/35' 2022-04-03T19:49:34.657 INFO:tasks.workunit.client.0.smithi077.stdout:removed './299/26' 2022-04-03T19:49:34.657 INFO:tasks.workunit.client.0.smithi077.stdout:removed './299/37' 2022-04-03T19:49:34.657 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './299' 2022-04-03T19:49:34.657 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/166/19' 2022-04-03T19:49:34.657 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/166/15' 2022-04-03T19:49:34.658 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/166/20' 2022-04-03T19:49:34.658 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/166/12' 2022-04-03T19:49:34.658 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/166/10' 2022-04-03T19:49:34.658 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/166/11' 2022-04-03T19:49:34.659 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/166/16' 2022-04-03T19:49:34.659 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/166/14' 2022-04-03T19:49:34.659 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/166/18' 2022-04-03T19:49:34.659 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/166/13' 2022-04-03T19:49:34.659 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/166/17' 2022-04-03T19:49:34.660 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/166' 2022-04-03T19:49:34.660 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/147/19' 2022-04-03T19:49:34.660 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/147/15' 2022-04-03T19:49:34.660 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/147/20' 2022-04-03T19:49:34.661 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/147/12' 2022-04-03T19:49:34.661 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/147/10' 2022-04-03T19:49:34.661 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/147/11' 2022-04-03T19:49:34.661 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/147/16' 2022-04-03T19:49:34.661 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/147/14' 2022-04-03T19:49:34.662 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/147/18' 2022-04-03T19:49:34.662 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/147/13' 2022-04-03T19:49:34.662 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/147/17' 2022-04-03T19:49:34.662 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/147' 2022-04-03T19:49:34.663 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/151/19' 2022-04-03T19:49:34.663 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/151/15' 2022-04-03T19:49:34.663 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/151/20' 2022-04-03T19:49:34.663 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/151/12' 2022-04-03T19:49:34.663 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/151/10' 2022-04-03T19:49:34.664 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/151/11' 2022-04-03T19:49:34.664 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/151/16' 2022-04-03T19:49:34.664 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/151/14' 2022-04-03T19:49:34.665 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/151/18' 2022-04-03T19:49:34.665 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/151/13' 2022-04-03T19:49:34.665 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/151/17' 2022-04-03T19:49:34.665 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/151' 2022-04-03T19:49:34.666 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/170/19' 2022-04-03T19:49:34.666 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/170/15' 2022-04-03T19:49:34.666 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/170/20' 2022-04-03T19:49:34.667 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/170/12' 2022-04-03T19:49:34.667 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/170/10' 2022-04-03T19:49:34.667 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/170/11' 2022-04-03T19:49:34.667 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/170/16' 2022-04-03T19:49:34.668 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/170/14' 2022-04-03T19:49:34.668 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/170/18' 2022-04-03T19:49:34.668 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/170/13' 2022-04-03T19:49:34.668 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/170/17' 2022-04-03T19:49:34.669 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/170' 2022-04-03T19:49:34.669 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/167/19' 2022-04-03T19:49:34.669 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/167/15' 2022-04-03T19:49:34.669 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/167/20' 2022-04-03T19:49:34.670 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/167/12' 2022-04-03T19:49:34.670 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/167/10' 2022-04-03T19:49:34.670 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/167/11' 2022-04-03T19:49:34.670 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/167/16' 2022-04-03T19:49:34.670 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/167/14' 2022-04-03T19:49:34.671 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/167/18' 2022-04-03T19:49:34.671 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/167/13' 2022-04-03T19:49:34.671 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/167/17' 2022-04-03T19:49:34.671 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/167' 2022-04-03T19:49:34.672 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/140/19' 2022-04-03T19:49:34.672 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/140/15' 2022-04-03T19:49:34.672 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/140/20' 2022-04-03T19:49:34.672 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/140/12' 2022-04-03T19:49:34.673 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/140/10' 2022-04-03T19:49:34.673 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/140/11' 2022-04-03T19:49:34.673 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/140/16' 2022-04-03T19:49:34.673 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/140/14' 2022-04-03T19:49:34.673 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/140/18' 2022-04-03T19:49:34.674 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/140/13' 2022-04-03T19:49:34.674 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/140/17' 2022-04-03T19:49:34.674 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/140' 2022-04-03T19:49:34.674 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/142/19' 2022-04-03T19:49:34.674 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/142/15' 2022-04-03T19:49:34.675 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/142/20' 2022-04-03T19:49:34.675 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/142/12' 2022-04-03T19:49:34.675 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/142/10' 2022-04-03T19:49:35.960 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/142/11' 2022-04-03T19:49:35.960 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/142/16' 2022-04-03T19:49:35.961 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/142/14' 2022-04-03T19:49:35.961 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/142/18' 2022-04-03T19:49:35.961 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/142/13' 2022-04-03T19:49:35.961 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/142/17' 2022-04-03T19:49:35.961 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/142' 2022-04-03T19:49:35.962 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/150/19' 2022-04-03T19:49:35.962 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/150/15' 2022-04-03T19:49:35.962 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/150/20' 2022-04-03T19:49:35.962 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/150/12' 2022-04-03T19:49:35.963 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/150/10' 2022-04-03T19:49:35.963 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/150/11' 2022-04-03T19:49:35.963 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/150/16' 2022-04-03T19:49:35.963 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/150/14' 2022-04-03T19:49:35.963 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/150/18' 2022-04-03T19:49:35.964 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/150/13' 2022-04-03T19:49:35.964 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/150/17' 2022-04-03T19:49:35.964 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/150' 2022-04-03T19:49:35.965 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/105/19' 2022-04-03T19:49:35.965 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/105/15' 2022-04-03T19:49:35.965 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/105/20' 2022-04-03T19:49:35.965 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/105/12' 2022-04-03T19:49:35.966 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/105/10' 2022-04-03T19:49:35.966 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/105/11' 2022-04-03T19:49:35.966 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/105/16' 2022-04-03T19:49:35.967 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/105/14' 2022-04-03T19:49:35.967 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/105/18' 2022-04-03T19:49:35.967 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/105/13' 2022-04-03T19:49:35.967 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/105/17' 2022-04-03T19:49:35.968 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/105' 2022-04-03T19:49:35.968 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/149/19' 2022-04-03T19:49:35.968 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/149/15' 2022-04-03T19:49:35.968 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/149/20' 2022-04-03T19:49:35.969 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/149/12' 2022-04-03T19:49:35.969 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/149/10' 2022-04-03T19:49:35.969 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/149/11' 2022-04-03T19:49:35.969 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/149/16' 2022-04-03T19:49:35.969 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/149/14' 2022-04-03T19:49:35.970 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/149/18' 2022-04-03T19:49:35.970 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/149/13' 2022-04-03T19:49:35.970 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/149/17' 2022-04-03T19:49:35.971 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/149' 2022-04-03T19:49:35.971 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/109/19' 2022-04-03T19:49:35.971 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/109/15' 2022-04-03T19:49:35.971 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/109/20' 2022-04-03T19:49:35.972 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/109/12' 2022-04-03T19:49:35.972 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/109/10' 2022-04-03T19:49:35.972 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/109/11' 2022-04-03T19:49:35.972 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/109/16' 2022-04-03T19:49:35.972 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/109/14' 2022-04-03T19:49:35.973 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/109/18' 2022-04-03T19:49:35.973 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/109/13' 2022-04-03T19:49:35.974 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/109/17' 2022-04-03T19:49:35.974 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/109' 2022-04-03T19:49:35.974 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/158/19' 2022-04-03T19:49:35.974 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/158/15' 2022-04-03T19:49:35.975 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/158/20' 2022-04-03T19:49:35.975 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/158/12' 2022-04-03T19:49:35.975 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/158/10' 2022-04-03T19:49:35.975 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/158/11' 2022-04-03T19:49:35.975 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/158/16' 2022-04-03T19:49:35.976 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/158/14' 2022-04-03T19:49:35.976 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/158/18' 2022-04-03T19:49:35.976 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/158/13' 2022-04-03T19:49:35.976 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/158/17' 2022-04-03T19:49:35.977 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/158' 2022-04-03T19:49:35.977 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/106/19' 2022-04-03T19:49:35.977 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/106/15' 2022-04-03T19:49:35.977 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/106/20' 2022-04-03T19:49:35.977 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/106/12' 2022-04-03T19:49:35.978 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/106/10' 2022-04-03T19:49:35.978 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/106/11' 2022-04-03T19:49:35.978 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/106/16' 2022-04-03T19:49:35.978 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/106/14' 2022-04-03T19:49:35.978 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/106/18' 2022-04-03T19:49:35.979 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/106/13' 2022-04-03T19:49:35.979 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/106/17' 2022-04-03T19:49:35.979 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/106' 2022-04-03T19:49:35.979 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/189/19' 2022-04-03T19:49:35.979 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/189/15' 2022-04-03T19:49:35.980 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/189/20' 2022-04-03T19:49:35.980 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/189/12' 2022-04-03T19:49:35.980 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/189/10' 2022-04-03T19:49:35.980 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/189/11' 2022-04-03T19:49:35.980 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/189/16' 2022-04-03T19:49:35.981 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/189/14' 2022-04-03T19:49:35.981 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/189/18' 2022-04-03T19:49:35.981 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/189/13' 2022-04-03T19:49:35.981 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/189/17' 2022-04-03T19:49:35.981 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/189' 2022-04-03T19:49:35.982 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/133/19' 2022-04-03T19:49:35.982 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/133/15' 2022-04-03T19:49:35.982 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/133/20' 2022-04-03T19:49:35.982 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/133/12' 2022-04-03T19:49:35.982 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/133/10' 2022-04-03T19:49:35.983 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/133/11' 2022-04-03T19:49:35.983 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/133/16' 2022-04-03T19:49:35.983 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/133/14' 2022-04-03T19:49:35.984 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/133/18' 2022-04-03T19:49:35.984 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/133/13' 2022-04-03T19:49:35.984 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/133/17' 2022-04-03T19:49:35.984 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/133' 2022-04-03T19:49:35.985 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/173/19' 2022-04-03T19:49:35.985 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/173/15' 2022-04-03T19:49:35.985 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/173/20' 2022-04-03T19:49:35.985 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/173/12' 2022-04-03T19:49:35.986 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/173/10' 2022-04-03T19:49:35.986 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/173/11' 2022-04-03T19:49:36.834 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/173/16' 2022-04-03T19:49:36.834 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/173/14' 2022-04-03T19:49:36.834 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/173/18' 2022-04-03T19:49:36.835 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/173/13' 2022-04-03T19:49:36.835 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/173/17' 2022-04-03T19:49:36.835 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/173' 2022-04-03T19:49:36.835 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/174/19' 2022-04-03T19:49:36.836 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/174/15' 2022-04-03T19:49:36.836 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/174/20' 2022-04-03T19:49:36.836 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/174/12' 2022-04-03T19:49:36.836 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/174/10' 2022-04-03T19:49:36.836 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/174/11' 2022-04-03T19:49:36.837 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/174/16' 2022-04-03T19:49:36.837 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/174/14' 2022-04-03T19:49:36.837 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/174/18' 2022-04-03T19:49:36.837 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/174/13' 2022-04-03T19:49:36.838 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/174/17' 2022-04-03T19:49:36.838 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/174' 2022-04-03T19:49:36.838 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/155/19' 2022-04-03T19:49:36.838 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/155/15' 2022-04-03T19:49:36.838 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/155/20' 2022-04-03T19:49:36.839 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/155/12' 2022-04-03T19:49:36.839 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/155/10' 2022-04-03T19:49:36.839 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/155/11' 2022-04-03T19:49:36.839 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/155/16' 2022-04-03T19:49:36.840 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/155/14' 2022-04-03T19:49:36.840 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/155/18' 2022-04-03T19:49:36.840 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/155/13' 2022-04-03T19:49:36.840 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/155/17' 2022-04-03T19:49:36.840 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/155' 2022-04-03T19:49:36.841 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/114/19' 2022-04-03T19:49:36.841 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/114/15' 2022-04-03T19:49:36.841 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/114/20' 2022-04-03T19:49:36.841 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/114/12' 2022-04-03T19:49:36.842 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/114/10' 2022-04-03T19:49:36.842 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/114/11' 2022-04-03T19:49:36.842 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/114/16' 2022-04-03T19:49:36.842 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/114/14' 2022-04-03T19:49:36.843 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/114/18' 2022-04-03T19:49:36.843 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/114/13' 2022-04-03T19:49:36.843 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/114/17' 2022-04-03T19:49:36.843 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/114' 2022-04-03T19:49:36.843 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/159/19' 2022-04-03T19:49:36.844 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/159/15' 2022-04-03T19:49:36.844 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/159/20' 2022-04-03T19:49:36.844 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/159/12' 2022-04-03T19:49:36.844 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/159/10' 2022-04-03T19:49:36.845 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/159/11' 2022-04-03T19:49:36.845 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/159/16' 2022-04-03T19:49:36.845 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/159/14' 2022-04-03T19:49:36.845 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/159/18' 2022-04-03T19:49:36.845 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/159/13' 2022-04-03T19:49:36.846 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/159/17' 2022-04-03T19:49:36.846 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/159' 2022-04-03T19:49:36.846 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/187/19' 2022-04-03T19:49:36.846 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/187/15' 2022-04-03T19:49:36.847 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/187/20' 2022-04-03T19:49:36.847 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/187/12' 2022-04-03T19:49:36.847 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/187/10' 2022-04-03T19:49:36.847 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/187/11' 2022-04-03T19:49:36.847 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/187/16' 2022-04-03T19:49:36.848 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/187/14' 2022-04-03T19:49:36.848 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/187/18' 2022-04-03T19:49:36.848 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/187/13' 2022-04-03T19:49:36.848 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/187/17' 2022-04-03T19:49:36.849 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/187' 2022-04-03T19:49:36.849 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/185/19' 2022-04-03T19:49:36.849 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/185/15' 2022-04-03T19:49:36.849 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/185/20' 2022-04-03T19:49:36.849 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/185/12' 2022-04-03T19:49:36.850 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/185/10' 2022-04-03T19:49:36.850 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/185/11' 2022-04-03T19:49:36.850 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/185/16' 2022-04-03T19:49:36.850 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/185/14' 2022-04-03T19:49:36.851 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/185/18' 2022-04-03T19:49:36.851 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/185/13' 2022-04-03T19:49:36.851 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/185/17' 2022-04-03T19:49:36.851 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/185' 2022-04-03T19:49:36.851 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/120/19' 2022-04-03T19:49:36.852 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/120/15' 2022-04-03T19:49:36.852 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/120/20' 2022-04-03T19:49:36.852 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/120/12' 2022-04-03T19:49:36.852 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/120/10' 2022-04-03T19:49:36.853 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/120/11' 2022-04-03T19:49:36.853 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/120/16' 2022-04-03T19:49:36.853 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/120/14' 2022-04-03T19:49:36.853 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/120/18' 2022-04-03T19:49:36.853 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/120/13' 2022-04-03T19:49:36.854 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/120/17' 2022-04-03T19:49:36.854 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/120' 2022-04-03T19:49:36.854 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/110/19' 2022-04-03T19:49:36.854 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/110/15' 2022-04-03T19:49:36.855 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/110/20' 2022-04-03T19:49:36.855 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/110/12' 2022-04-03T19:49:36.855 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/110/10' 2022-04-03T19:49:36.855 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/110/11' 2022-04-03T19:49:36.856 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/110/16' 2022-04-03T19:49:36.856 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/110/14' 2022-04-03T19:49:36.856 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/110/18' 2022-04-03T19:49:36.856 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/110/13' 2022-04-03T19:49:36.856 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/110/17' 2022-04-03T19:49:36.857 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/110' 2022-04-03T19:49:36.857 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/137/19' 2022-04-03T19:49:36.857 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/137/15' 2022-04-03T19:49:36.857 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/137/20' 2022-04-03T19:49:36.858 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/137/12' 2022-04-03T19:49:36.858 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/137/10' 2022-04-03T19:49:36.858 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/137/11' 2022-04-03T19:49:36.858 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/137/16' 2022-04-03T19:49:39.751 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/137/14' 2022-04-03T19:49:39.751 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/137/18' 2022-04-03T19:49:39.752 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/137/13' 2022-04-03T19:49:39.752 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/137/17' 2022-04-03T19:49:39.752 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/137' 2022-04-03T19:49:39.752 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/198/19' 2022-04-03T19:49:39.753 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/198/15' 2022-04-03T19:49:39.753 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/198/20' 2022-04-03T19:49:39.753 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/198/12' 2022-04-03T19:49:39.753 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/198/10' 2022-04-03T19:49:39.753 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/198/11' 2022-04-03T19:49:39.754 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/198/16' 2022-04-03T19:49:39.754 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/198/14' 2022-04-03T19:49:39.754 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/198/18' 2022-04-03T19:49:39.754 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/198/13' 2022-04-03T19:49:39.755 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/198/17' 2022-04-03T19:49:39.755 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/198' 2022-04-03T19:49:39.755 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/136/19' 2022-04-03T19:49:39.755 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/136/15' 2022-04-03T19:49:39.756 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/136/20' 2022-04-03T19:49:39.756 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/136/12' 2022-04-03T19:49:39.756 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/136/10' 2022-04-03T19:49:39.756 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/136/11' 2022-04-03T19:49:39.757 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/136/16' 2022-04-03T19:49:39.757 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/136/14' 2022-04-03T19:49:39.757 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/136/18' 2022-04-03T19:49:39.757 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/136/13' 2022-04-03T19:49:39.757 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/136/17' 2022-04-03T19:49:39.758 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/136' 2022-04-03T19:49:39.758 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/125/19' 2022-04-03T19:49:39.758 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/125/15' 2022-04-03T19:49:39.758 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/125/20' 2022-04-03T19:49:39.759 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/125/12' 2022-04-03T19:49:39.759 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/125/10' 2022-04-03T19:49:39.760 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/125/11' 2022-04-03T19:49:39.760 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/125/16' 2022-04-03T19:49:39.760 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/125/14' 2022-04-03T19:49:39.760 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/125/18' 2022-04-03T19:49:39.761 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/125/13' 2022-04-03T19:49:39.761 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/125/17' 2022-04-03T19:49:39.761 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/125' 2022-04-03T19:49:39.761 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/194/19' 2022-04-03T19:49:39.762 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/194/15' 2022-04-03T19:49:39.762 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/194/20' 2022-04-03T19:49:39.762 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/194/12' 2022-04-03T19:49:39.763 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/194/10' 2022-04-03T19:49:39.763 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/194/11' 2022-04-03T19:49:39.763 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/194/16' 2022-04-03T19:49:39.763 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/194/14' 2022-04-03T19:49:39.763 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/194/18' 2022-04-03T19:49:39.764 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/194/13' 2022-04-03T19:49:39.764 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/194/17' 2022-04-03T19:49:39.764 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/194' 2022-04-03T19:49:39.764 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/119/19' 2022-04-03T19:49:39.765 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/119/15' 2022-04-03T19:49:39.765 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/119/20' 2022-04-03T19:49:39.765 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/119/12' 2022-04-03T19:49:39.765 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/119/10' 2022-04-03T19:49:39.765 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/119/11' 2022-04-03T19:49:39.766 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/119/16' 2022-04-03T19:49:39.766 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/119/14' 2022-04-03T19:49:39.766 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/119/18' 2022-04-03T19:49:39.766 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/119/13' 2022-04-03T19:49:39.767 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/119/17' 2022-04-03T19:49:39.767 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/119' 2022-04-03T19:49:39.767 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/157/19' 2022-04-03T19:49:39.767 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/157/15' 2022-04-03T19:49:39.767 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/157/20' 2022-04-03T19:49:39.768 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/157/12' 2022-04-03T19:49:39.768 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/157/10' 2022-04-03T19:49:39.768 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/157/11' 2022-04-03T19:49:39.768 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/157/16' 2022-04-03T19:49:39.769 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/157/14' 2022-04-03T19:49:39.769 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/157/18' 2022-04-03T19:49:39.769 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/157/13' 2022-04-03T19:49:39.769 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/157/17' 2022-04-03T19:49:39.769 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/157' 2022-04-03T19:49:39.770 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/172/19' 2022-04-03T19:49:39.770 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/172/15' 2022-04-03T19:49:39.770 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/172/20' 2022-04-03T19:49:39.770 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/172/12' 2022-04-03T19:49:39.771 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/172/10' 2022-04-03T19:49:39.771 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/172/11' 2022-04-03T19:49:39.771 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/172/16' 2022-04-03T19:49:39.771 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/172/14' 2022-04-03T19:49:39.771 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/172/18' 2022-04-03T19:49:39.772 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/172/13' 2022-04-03T19:49:39.772 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/172/17' 2022-04-03T19:49:39.772 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/172' 2022-04-03T19:49:39.772 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/186/19' 2022-04-03T19:49:39.773 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/186/15' 2022-04-03T19:49:39.773 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/186/20' 2022-04-03T19:49:39.773 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/186/12' 2022-04-03T19:49:39.773 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/186/10' 2022-04-03T19:49:39.773 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/186/11' 2022-04-03T19:49:39.774 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/186/16' 2022-04-03T19:49:39.774 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/186/14' 2022-04-03T19:49:39.774 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/186/18' 2022-04-03T19:49:39.774 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/186/13' 2022-04-03T19:49:39.775 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/186/17' 2022-04-03T19:49:39.775 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/186' 2022-04-03T19:49:39.775 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/190/19' 2022-04-03T19:49:39.775 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/190/15' 2022-04-03T19:49:39.775 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/190/20' 2022-04-03T19:49:39.776 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/190/12' 2022-04-03T19:49:39.776 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/190/10' 2022-04-03T19:49:39.776 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/190/11' 2022-04-03T19:49:39.776 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/190/16' 2022-04-03T19:49:39.777 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/190/14' 2022-04-03T19:49:40.457 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/190/18' 2022-04-03T19:49:40.458 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/190/13' 2022-04-03T19:49:40.458 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/190/17' 2022-04-03T19:49:40.459 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/190' 2022-04-03T19:49:40.460 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/104/19' 2022-04-03T19:49:40.460 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/104/15' 2022-04-03T19:49:40.461 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/104/20' 2022-04-03T19:49:40.461 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/104/12' 2022-04-03T19:49:40.462 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/104/10' 2022-04-03T19:49:40.462 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/104/11' 2022-04-03T19:49:40.463 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/104/16' 2022-04-03T19:49:40.463 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/104/14' 2022-04-03T19:49:40.464 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/104/18' 2022-04-03T19:49:40.464 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/104/13' 2022-04-03T19:49:40.465 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/104/17' 2022-04-03T19:49:40.465 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/104' 2022-04-03T19:49:40.466 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/164/19' 2022-04-03T19:49:40.466 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/164/15' 2022-04-03T19:49:40.467 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/164/20' 2022-04-03T19:49:40.467 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/164/12' 2022-04-03T19:49:40.468 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/164/10' 2022-04-03T19:49:40.468 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/164/11' 2022-04-03T19:49:40.469 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/164/16' 2022-04-03T19:49:40.469 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/164/14' 2022-04-03T19:49:40.470 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/164/18' 2022-04-03T19:49:40.470 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/164/13' 2022-04-03T19:49:40.471 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/164/17' 2022-04-03T19:49:40.471 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/164' 2022-04-03T19:49:40.472 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/112/19' 2022-04-03T19:49:40.472 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/112/15' 2022-04-03T19:49:40.473 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/112/20' 2022-04-03T19:49:40.473 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/112/12' 2022-04-03T19:49:40.474 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/112/10' 2022-04-03T19:49:40.474 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/112/11' 2022-04-03T19:49:40.475 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/112/16' 2022-04-03T19:49:40.475 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/112/14' 2022-04-03T19:49:40.476 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/112/18' 2022-04-03T19:49:40.476 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/112/13' 2022-04-03T19:49:40.477 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/112/17' 2022-04-03T19:49:40.477 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/112' 2022-04-03T19:49:40.477 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/100/19' 2022-04-03T19:49:40.478 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/100/15' 2022-04-03T19:49:40.478 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/100/20' 2022-04-03T19:49:40.479 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/100/12' 2022-04-03T19:49:40.479 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/100/10' 2022-04-03T19:49:40.480 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/100/11' 2022-04-03T19:49:40.480 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/100/16' 2022-04-03T19:49:40.481 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/100/14' 2022-04-03T19:49:40.481 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/100/18' 2022-04-03T19:49:40.482 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/100/13' 2022-04-03T19:49:40.482 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/100/17' 2022-04-03T19:49:40.483 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/100' 2022-04-03T19:49:40.483 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/134/19' 2022-04-03T19:49:40.484 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/134/15' 2022-04-03T19:49:40.484 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/134/20' 2022-04-03T19:49:40.485 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/134/12' 2022-04-03T19:49:40.485 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/134/10' 2022-04-03T19:49:40.486 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/134/11' 2022-04-03T19:49:40.486 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/134/16' 2022-04-03T19:49:40.487 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/134/14' 2022-04-03T19:49:40.487 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/134/18' 2022-04-03T19:49:40.488 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/134/13' 2022-04-03T19:49:40.488 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/134/17' 2022-04-03T19:49:40.489 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/134' 2022-04-03T19:49:40.489 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/115/19' 2022-04-03T19:49:40.490 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/115/15' 2022-04-03T19:49:40.490 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/115/20' 2022-04-03T19:49:40.491 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/115/12' 2022-04-03T19:49:40.491 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/115/10' 2022-04-03T19:49:40.492 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/115/11' 2022-04-03T19:49:40.492 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/115/16' 2022-04-03T19:49:40.492 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/115/14' 2022-04-03T19:49:40.493 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/115/18' 2022-04-03T19:49:40.493 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/115/13' 2022-04-03T19:49:40.494 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/115/17' 2022-04-03T19:49:40.494 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/115' 2022-04-03T19:49:40.495 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/116/19' 2022-04-03T19:49:40.495 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/116/15' 2022-04-03T19:49:40.495 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/116/20' 2022-04-03T19:49:40.496 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/116/12' 2022-04-03T19:49:40.496 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/116/10' 2022-04-03T19:49:40.496 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/116/11' 2022-04-03T19:49:40.496 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/116/16' 2022-04-03T19:49:40.497 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/116/14' 2022-04-03T19:49:40.497 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/116/18' 2022-04-03T19:49:40.497 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/116/13' 2022-04-03T19:49:40.497 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/116/17' 2022-04-03T19:49:40.498 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/116' 2022-04-03T19:49:40.498 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/195/19' 2022-04-03T19:49:40.498 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/195/15' 2022-04-03T19:49:40.498 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/195/20' 2022-04-03T19:49:40.499 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/195/12' 2022-04-03T19:49:40.499 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/195/10' 2022-04-03T19:49:40.499 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/195/11' 2022-04-03T19:49:40.499 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/195/16' 2022-04-03T19:49:40.500 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/195/14' 2022-04-03T19:49:40.500 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/195/18' 2022-04-03T19:49:40.500 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/195/13' 2022-04-03T19:49:40.500 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/195/17' 2022-04-03T19:49:40.501 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/195' 2022-04-03T19:49:40.501 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/138/19' 2022-04-03T19:49:40.501 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/138/15' 2022-04-03T19:49:40.501 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/138/20' 2022-04-03T19:49:40.502 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/138/12' 2022-04-03T19:49:40.502 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/138/10' 2022-04-03T19:49:40.502 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/138/11' 2022-04-03T19:49:40.502 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/138/16' 2022-04-03T19:49:40.503 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/138/14' 2022-04-03T19:49:40.503 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/138/18' 2022-04-03T19:49:42.342 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/138/13' 2022-04-03T19:49:42.343 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/138/17' 2022-04-03T19:49:42.343 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/138' 2022-04-03T19:49:42.343 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/135/19' 2022-04-03T19:49:42.343 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/135/15' 2022-04-03T19:49:42.344 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/135/20' 2022-04-03T19:49:42.344 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/135/12' 2022-04-03T19:49:42.344 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/135/10' 2022-04-03T19:49:42.344 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/135/11' 2022-04-03T19:49:42.345 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/135/16' 2022-04-03T19:49:42.345 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/135/14' 2022-04-03T19:49:42.345 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/135/18' 2022-04-03T19:49:42.345 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/135/13' 2022-04-03T19:49:42.345 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/135/17' 2022-04-03T19:49:42.346 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/135' 2022-04-03T19:49:42.346 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/117/19' 2022-04-03T19:49:42.346 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/117/15' 2022-04-03T19:49:42.346 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/117/20' 2022-04-03T19:49:42.347 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/117/12' 2022-04-03T19:49:42.347 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/117/10' 2022-04-03T19:49:42.347 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/117/11' 2022-04-03T19:49:42.347 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/117/16' 2022-04-03T19:49:42.348 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/117/14' 2022-04-03T19:49:42.348 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/117/18' 2022-04-03T19:49:42.348 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/117/13' 2022-04-03T19:49:42.348 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/117/17' 2022-04-03T19:49:42.349 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/117' 2022-04-03T19:49:42.349 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/143/19' 2022-04-03T19:49:42.349 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/143/15' 2022-04-03T19:49:42.349 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/143/20' 2022-04-03T19:49:42.350 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/143/12' 2022-04-03T19:49:42.350 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/143/10' 2022-04-03T19:49:42.350 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/143/11' 2022-04-03T19:49:42.350 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/143/16' 2022-04-03T19:49:42.351 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/143/14' 2022-04-03T19:49:42.351 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/143/18' 2022-04-03T19:49:42.351 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/143/13' 2022-04-03T19:49:42.351 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/143/17' 2022-04-03T19:49:42.351 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/143' 2022-04-03T19:49:42.352 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/180/19' 2022-04-03T19:49:42.352 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/180/15' 2022-04-03T19:49:42.352 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/180/20' 2022-04-03T19:49:42.352 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/180/12' 2022-04-03T19:49:42.353 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/180/10' 2022-04-03T19:49:42.353 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/180/11' 2022-04-03T19:49:42.353 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/180/16' 2022-04-03T19:49:42.353 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/180/14' 2022-04-03T19:49:42.354 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/180/18' 2022-04-03T19:49:42.354 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/180/13' 2022-04-03T19:49:42.354 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/180/17' 2022-04-03T19:49:42.354 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/180' 2022-04-03T19:49:42.354 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/126/19' 2022-04-03T19:49:42.355 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/126/15' 2022-04-03T19:49:42.355 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/126/20' 2022-04-03T19:49:42.355 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/126/12' 2022-04-03T19:49:42.355 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/126/10' 2022-04-03T19:49:42.356 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/126/11' 2022-04-03T19:49:42.356 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/126/16' 2022-04-03T19:49:42.356 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/126/14' 2022-04-03T19:49:42.356 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/126/18' 2022-04-03T19:49:42.356 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/126/13' 2022-04-03T19:49:42.357 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/126/17' 2022-04-03T19:49:42.357 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/126' 2022-04-03T19:49:42.357 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/161/19' 2022-04-03T19:49:42.357 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/161/15' 2022-04-03T19:49:42.357 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/161/20' 2022-04-03T19:49:42.358 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/161/12' 2022-04-03T19:49:42.358 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/161/10' 2022-04-03T19:49:42.358 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/161/11' 2022-04-03T19:49:42.358 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/161/16' 2022-04-03T19:49:42.358 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/161/14' 2022-04-03T19:49:42.359 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/161/18' 2022-04-03T19:49:42.359 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/161/13' 2022-04-03T19:49:42.359 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/161/17' 2022-04-03T19:49:42.359 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/161' 2022-04-03T19:49:42.359 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/176/19' 2022-04-03T19:49:42.360 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/176/15' 2022-04-03T19:49:42.360 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/176/20' 2022-04-03T19:49:42.360 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/176/12' 2022-04-03T19:49:42.360 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/176/10' 2022-04-03T19:49:42.361 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/176/11' 2022-04-03T19:49:42.361 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/176/16' 2022-04-03T19:49:42.361 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/176/14' 2022-04-03T19:49:42.361 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/176/18' 2022-04-03T19:49:42.361 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/176/13' 2022-04-03T19:49:42.362 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/176/17' 2022-04-03T19:49:42.362 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/176' 2022-04-03T19:49:42.362 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/111/19' 2022-04-03T19:49:42.362 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/111/15' 2022-04-03T19:49:42.363 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/111/20' 2022-04-03T19:49:42.363 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/111/12' 2022-04-03T19:49:42.363 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/111/10' 2022-04-03T19:49:42.363 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/111/11' 2022-04-03T19:49:42.364 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/111/16' 2022-04-03T19:49:42.364 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/111/14' 2022-04-03T19:49:42.364 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/111/18' 2022-04-03T19:49:42.364 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/111/13' 2022-04-03T19:49:42.364 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/111/17' 2022-04-03T19:49:42.365 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/111' 2022-04-03T19:49:42.365 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/191/19' 2022-04-03T19:49:42.365 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/191/15' 2022-04-03T19:49:42.365 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/191/20' 2022-04-03T19:49:42.366 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/191/12' 2022-04-03T19:49:42.366 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/191/10' 2022-04-03T19:49:42.366 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/191/11' 2022-04-03T19:49:42.366 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/191/16' 2022-04-03T19:49:42.367 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/191/14' 2022-04-03T19:49:42.367 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/191/18' 2022-04-03T19:49:42.367 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/191/13' 2022-04-03T19:49:43.020 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/191/17' 2022-04-03T19:49:43.020 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/191' 2022-04-03T19:49:43.021 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/184/19' 2022-04-03T19:49:43.021 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/184/15' 2022-04-03T19:49:43.021 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/184/20' 2022-04-03T19:49:43.021 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/184/12' 2022-04-03T19:49:43.021 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/184/10' 2022-04-03T19:49:43.022 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/184/11' 2022-04-03T19:49:43.022 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/184/16' 2022-04-03T19:49:43.022 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/184/14' 2022-04-03T19:49:43.022 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/184/18' 2022-04-03T19:49:43.023 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/184/13' 2022-04-03T19:49:43.023 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/184/17' 2022-04-03T19:49:43.023 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/184' 2022-04-03T19:49:43.023 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/192/19' 2022-04-03T19:49:43.023 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/192/15' 2022-04-03T19:49:43.024 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/192/20' 2022-04-03T19:49:43.024 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/192/12' 2022-04-03T19:49:43.024 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/192/10' 2022-04-03T19:49:43.024 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/192/11' 2022-04-03T19:49:43.025 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/192/16' 2022-04-03T19:49:43.025 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/192/14' 2022-04-03T19:49:43.025 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/192/18' 2022-04-03T19:49:43.025 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/192/13' 2022-04-03T19:49:43.026 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/192/17' 2022-04-03T19:49:43.026 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/192' 2022-04-03T19:49:43.026 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/103/19' 2022-04-03T19:49:43.026 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/103/15' 2022-04-03T19:49:43.026 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/103/20' 2022-04-03T19:49:43.027 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/103/12' 2022-04-03T19:49:43.027 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/103/10' 2022-04-03T19:49:43.027 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/103/11' 2022-04-03T19:49:43.027 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/103/16' 2022-04-03T19:49:43.028 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/103/14' 2022-04-03T19:49:43.028 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/103/18' 2022-04-03T19:49:43.028 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/103/13' 2022-04-03T19:49:43.028 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/103/17' 2022-04-03T19:49:43.029 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/103' 2022-04-03T19:49:43.029 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/128/19' 2022-04-03T19:49:43.029 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/128/15' 2022-04-03T19:49:43.029 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/128/20' 2022-04-03T19:49:43.029 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/128/12' 2022-04-03T19:49:43.030 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/128/10' 2022-04-03T19:49:43.030 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/128/11' 2022-04-03T19:49:43.030 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/128/16' 2022-04-03T19:49:43.030 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/128/14' 2022-04-03T19:49:43.030 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/128/18' 2022-04-03T19:49:43.031 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/128/13' 2022-04-03T19:49:43.031 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/128/17' 2022-04-03T19:49:43.031 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/128' 2022-04-03T19:49:43.031 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/144/19' 2022-04-03T19:49:43.031 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/144/15' 2022-04-03T19:49:43.032 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/144/20' 2022-04-03T19:49:43.032 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/144/12' 2022-04-03T19:49:43.032 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/144/10' 2022-04-03T19:49:43.032 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/144/11' 2022-04-03T19:49:43.032 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/144/16' 2022-04-03T19:49:43.033 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/144/14' 2022-04-03T19:49:43.033 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/144/18' 2022-04-03T19:49:43.033 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/144/13' 2022-04-03T19:49:43.033 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/144/17' 2022-04-03T19:49:43.033 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/144' 2022-04-03T19:49:43.034 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/146/19' 2022-04-03T19:49:43.034 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/146/15' 2022-04-03T19:49:43.034 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/146/20' 2022-04-03T19:49:43.034 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/146/12' 2022-04-03T19:49:43.035 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/146/10' 2022-04-03T19:49:43.035 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/146/11' 2022-04-03T19:49:43.035 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/146/16' 2022-04-03T19:49:43.035 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/146/14' 2022-04-03T19:49:43.035 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/146/18' 2022-04-03T19:49:43.036 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/146/13' 2022-04-03T19:49:43.036 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/146/17' 2022-04-03T19:49:43.036 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/146' 2022-04-03T19:49:43.036 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/141/19' 2022-04-03T19:49:43.037 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/141/15' 2022-04-03T19:49:43.037 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/141/20' 2022-04-03T19:49:43.037 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/141/12' 2022-04-03T19:49:43.037 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/141/10' 2022-04-03T19:49:43.037 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/141/11' 2022-04-03T19:49:43.038 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/141/16' 2022-04-03T19:49:43.038 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/141/14' 2022-04-03T19:49:43.038 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/141/18' 2022-04-03T19:49:43.038 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/141/13' 2022-04-03T19:49:43.039 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/141/17' 2022-04-03T19:49:43.039 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/141' 2022-04-03T19:49:43.039 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/182/19' 2022-04-03T19:49:43.039 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/182/15' 2022-04-03T19:49:43.040 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/182/20' 2022-04-03T19:49:43.040 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/182/12' 2022-04-03T19:49:43.040 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/182/10' 2022-04-03T19:49:43.040 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/182/11' 2022-04-03T19:49:43.040 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/182/16' 2022-04-03T19:49:43.041 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/182/14' 2022-04-03T19:49:43.041 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/182/18' 2022-04-03T19:49:43.041 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/182/13' 2022-04-03T19:49:43.041 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/182/17' 2022-04-03T19:49:43.042 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/182' 2022-04-03T19:49:43.042 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/107/19' 2022-04-03T19:49:43.042 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/107/15' 2022-04-03T19:49:43.042 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/107/20' 2022-04-03T19:49:43.042 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/107/12' 2022-04-03T19:49:43.043 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/107/10' 2022-04-03T19:49:43.043 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/107/11' 2022-04-03T19:49:43.043 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/107/16' 2022-04-03T19:49:43.043 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/107/14' 2022-04-03T19:49:43.043 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/107/18' 2022-04-03T19:49:43.044 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/107/13' 2022-04-03T19:49:43.044 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/107/17' 2022-04-03T19:49:44.392 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:49:44.592 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.590+0000 7f9af359e700 1 Processor -- start 2022-04-03T19:49:44.592 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.590+0000 7f9af359e700 1 -- start start 2022-04-03T19:49:44.593 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.592+0000 7f9af359e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9af40c1b30 0x7f9af414d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:44.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.592+0000 7f9af359e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af40c3c50 con 0x7f9af40ba3c0 2022-04-03T19:49:44.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.592+0000 7f9af359e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af4156690 con 0x7f9af40678b0 2022-04-03T19:49:44.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.592+0000 7f9af359e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9af40c8020 con 0x7f9af40c1b30 2022-04-03T19:49:44.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.592+0000 7f9af259c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9af40ba3c0 0x7f9af4156b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38448/0 (socket says 172.21.15.77:38448) 2022-04-03T19:49:44.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.592+0000 7f9af259c700 1 -- 172.21.15.77:0/240881829 learned_addr learned my addr 172.21.15.77:0/240881829 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:49:44.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.592+0000 7f9af2d9d700 1 --2- 172.21.15.77:0/240881829 >> v2:172.21.15.186:3300/0 conn(0x7f9af40c1b30 0x7f9af414d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:44.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.593+0000 7f9af1d9b700 1 -- 172.21.15.77:0/240881829 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457084314 0 0) 0x7f9af40c3c50 con 0x7f9af40ba3c0 2022-04-03T19:49:44.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.593+0000 7f9af1d9b700 1 -- 172.21.15.77:0/240881829 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad40033f0 con 0x7f9af40ba3c0 2022-04-03T19:49:44.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.593+0000 7f9af1d9b700 1 -- 172.21.15.77:0/240881829 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269269227 0 0) 0x7f9af4156690 con 0x7f9af40678b0 2022-04-03T19:49:44.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.593+0000 7f9af1d9b700 1 -- 172.21.15.77:0/240881829 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af40c3c50 con 0x7f9af40678b0 2022-04-03T19:49:44.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.593+0000 7f9af2d9d700 1 -- 172.21.15.77:0/240881829 >> v1:172.21.15.186:6789/0 conn(0x7f9af40678b0 legacy=0x7f9af414cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:44.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.593+0000 7f9af2d9d700 1 -- 172.21.15.77:0/240881829 >> v1:172.21.15.77:6789/0 conn(0x7f9af40ba3c0 legacy=0x7f9af4156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:44.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.593+0000 7f9af2d9d700 1 -- 172.21.15.77:0/240881829 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af40bf350 con 0x7f9af40c1b30 2022-04-03T19:49:44.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.593+0000 7f9af2d9d700 1 --2- 172.21.15.77:0/240881829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9af40c1b30 0x7f9af414d210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009e60 tx=0x7f9aec009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e78239f28f4f9734 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:44.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.593+0000 7f9af1d9b700 1 -- 172.21.15.77:0/240881829 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec00e070 con 0x7f9af40c1b30 2022-04-03T19:49:44.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.594+0000 7f9af1d9b700 1 -- 172.21.15.77:0/240881829 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aec00bb40 con 0x7f9af40c1b30 2022-04-03T19:49:44.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.594+0000 7f9af1d9b700 1 -- 172.21.15.77:0/240881829 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec012430 con 0x7f9af40c1b30 2022-04-03T19:49:44.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.594+0000 7f9af359e700 1 -- 172.21.15.77:0/240881829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9af40c1b30 msgr2=0x7f9af414d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:44.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.594+0000 7f9af359e700 1 --2- 172.21.15.77:0/240881829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9af40c1b30 0x7f9af414d210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009e60 tx=0x7f9aec009780 comp rx=0 tx=0).stop 2022-04-03T19:49:44.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.594+0000 7f9af359e700 1 -- 172.21.15.77:0/240881829 shutdown_connections 2022-04-03T19:49:44.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.594+0000 7f9af359e700 1 --2- 172.21.15.77:0/240881829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9af40c1b30 0x7f9af414d210 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:44.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.594+0000 7f9af359e700 1 -- 172.21.15.77:0/240881829 >> 172.21.15.77:0/240881829 conn(0x7f9af4066da0 msgr2=0x7f9af40671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:49:44.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.594+0000 7f9af359e700 1 -- 172.21.15.77:0/240881829 shutdown_connections 2022-04-03T19:49:44.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.594+0000 7f9af359e700 1 -- 172.21.15.77:0/240881829 wait complete. 2022-04-03T19:49:44.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.596+0000 7f9af359e700 1 Processor -- start 2022-04-03T19:49:44.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.596+0000 7f9af359e700 1 -- start start 2022-04-03T19:49:44.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.597+0000 7f9af359e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9af40678b0 0x7f9af41480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:44.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.597+0000 7f9af359e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9af40ba3c0 0x7f9af416c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:44.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.597+0000 7f9af359e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9af40c1b30 0x7f9af40c5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:44.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.597+0000 7f9af359e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af40c6c90 con 0x7f9af40678b0 2022-04-03T19:49:44.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.597+0000 7f9af359e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af416cf20 con 0x7f9af40c1b30 2022-04-03T19:49:44.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.597+0000 7f9af359e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af4069f10 con 0x7f9af40ba3c0 2022-04-03T19:49:44.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.597+0000 7f9af259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9af40678b0 0x7f9af41480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:44.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.597+0000 7f9af259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9af40678b0 0x7f9af41480f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42886/0 (socket says 172.21.15.77:42886) 2022-04-03T19:49:44.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.597+0000 7f9af259c700 1 -- 172.21.15.77:0/503471721 learned_addr learned my addr 172.21.15.77:0/503471721 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:49:44.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.597+0000 7f9aebfff700 1 --2- 172.21.15.77:0/503471721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9af40ba3c0 0x7f9af416c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:44.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.597+0000 7f9af2d9d700 1 --2- 172.21.15.77:0/503471721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9af40c1b30 0x7f9af40c5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:44.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.598+0000 7f9af259c700 1 -- 172.21.15.77:0/503471721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9af40ba3c0 msgr2=0x7f9af416c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:44.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.598+0000 7f9af259c700 1 --2- 172.21.15.77:0/503471721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9af40ba3c0 0x7f9af416c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:44.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.598+0000 7f9af259c700 1 -- 172.21.15.77:0/503471721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9af40c1b30 msgr2=0x7f9af40c5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:44.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.598+0000 7f9af259c700 1 --2- 172.21.15.77:0/503471721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9af40c1b30 0x7f9af40c5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:44.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.598+0000 7f9af259c700 1 -- 172.21.15.77:0/503471721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aec009a20 con 0x7f9af40678b0 2022-04-03T19:49:44.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.598+0000 7f9af259c700 1 --2- 172.21.15.77:0/503471721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9af40678b0 0x7f9af41480f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9ae400c520 tx=0x7f9ae4007ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:44.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.598+0000 7f9aea7fc700 1 -- 172.21.15.77:0/503471721 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae400ce50 con 0x7f9af40678b0 2022-04-03T19:49:44.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.598+0000 7f9aea7fc700 1 -- 172.21.15.77:0/503471721 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae40072e0 con 0x7f9af40678b0 2022-04-03T19:49:44.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.598+0000 7f9aea7fc700 1 -- 172.21.15.77:0/503471721 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae4005780 con 0x7f9af40678b0 2022-04-03T19:49:44.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.598+0000 7f9af359e700 1 -- 172.21.15.77:0/503471721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af406aa70 con 0x7f9af40678b0 2022-04-03T19:49:44.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.598+0000 7f9af359e700 1 -- 172.21.15.77:0/503471721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af406a140 con 0x7f9af40678b0 2022-04-03T19:49:44.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.599+0000 7f9af359e700 1 -- 172.21.15.77:0/503471721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af40c74a0 con 0x7f9af40678b0 2022-04-03T19:49:44.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.600+0000 7f9aea7fc700 1 -- 172.21.15.77:0/503471721 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f9ae4019030 con 0x7f9af40678b0 2022-04-03T19:49:44.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.601+0000 7f9aea7fc700 1 --2- 172.21.15.77:0/503471721 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9ad8063ce0 0x7f9ad8066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:44.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.601+0000 7f9aea7fc700 1 -- 172.21.15.77:0/503471721 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6288+0+0 (secure 0 0 0) 0x7f9ae407fb90 con 0x7f9af40678b0 2022-04-03T19:49:44.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.601+0000 7f9aebfff700 1 --2- 172.21.15.77:0/503471721 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9ad8063ce0 0x7f9ad8066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:44.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.602+0000 7f9aebfff700 1 --2- 172.21.15.77:0/503471721 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9ad8063ce0 0x7f9ad8066190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9af414ae50 tx=0x7f9adc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:44.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.605+0000 7f9aea7fc700 1 -- 172.21.15.77:0/503471721 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f9ae404e690 con 0x7f9af40678b0 2022-04-03T19:49:44.701 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/107' 2022-04-03T19:49:44.701 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/178/19' 2022-04-03T19:49:44.701 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/178/15' 2022-04-03T19:49:44.702 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/178/20' 2022-04-03T19:49:44.702 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/178/12' 2022-04-03T19:49:44.702 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/178/10' 2022-04-03T19:49:44.702 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/178/11' 2022-04-03T19:49:44.702 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/178/16' 2022-04-03T19:49:44.703 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/178/14' 2022-04-03T19:49:44.703 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/178/18' 2022-04-03T19:49:44.703 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/178/13' 2022-04-03T19:49:44.703 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/178/17' 2022-04-03T19:49:44.704 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/178' 2022-04-03T19:49:44.704 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/156/19' 2022-04-03T19:49:44.704 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/156/15' 2022-04-03T19:49:44.704 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/156/20' 2022-04-03T19:49:44.704 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/156/12' 2022-04-03T19:49:44.705 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/156/10' 2022-04-03T19:49:44.705 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/156/11' 2022-04-03T19:49:44.705 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/156/16' 2022-04-03T19:49:44.705 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/156/14' 2022-04-03T19:49:44.705 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/156/18' 2022-04-03T19:49:44.706 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/156/13' 2022-04-03T19:49:44.706 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/156/17' 2022-04-03T19:49:44.706 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/156' 2022-04-03T19:49:44.706 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/131/19' 2022-04-03T19:49:44.706 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/131/15' 2022-04-03T19:49:44.707 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/131/20' 2022-04-03T19:49:44.707 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/131/12' 2022-04-03T19:49:44.707 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/131/10' 2022-04-03T19:49:44.707 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/131/11' 2022-04-03T19:49:44.707 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/131/16' 2022-04-03T19:49:44.708 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/131/14' 2022-04-03T19:49:44.708 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/131/18' 2022-04-03T19:49:44.708 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/131/13' 2022-04-03T19:49:44.708 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/131/17' 2022-04-03T19:49:44.708 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/131' 2022-04-03T19:49:44.709 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/123/19' 2022-04-03T19:49:44.709 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/123/15' 2022-04-03T19:49:44.709 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/123/20' 2022-04-03T19:49:44.709 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/123/12' 2022-04-03T19:49:44.709 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/123/10' 2022-04-03T19:49:44.709 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/123/11' 2022-04-03T19:49:44.710 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/123/16' 2022-04-03T19:49:44.710 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/123/14' 2022-04-03T19:49:44.710 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/123/18' 2022-04-03T19:49:44.710 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/123/13' 2022-04-03T19:49:44.710 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/123/17' 2022-04-03T19:49:44.711 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/123' 2022-04-03T19:49:44.711 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/160/19' 2022-04-03T19:49:44.711 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/160/15' 2022-04-03T19:49:44.711 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/160/20' 2022-04-03T19:49:44.711 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/160/12' 2022-04-03T19:49:44.712 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/160/10' 2022-04-03T19:49:44.712 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/160/11' 2022-04-03T19:49:44.712 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/160/16' 2022-04-03T19:49:44.712 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/160/14' 2022-04-03T19:49:44.712 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/160/18' 2022-04-03T19:49:44.713 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/160/13' 2022-04-03T19:49:44.713 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/160/17' 2022-04-03T19:49:44.713 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/160' 2022-04-03T19:49:44.713 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/165/19' 2022-04-03T19:49:44.713 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/165/15' 2022-04-03T19:49:44.714 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/165/20' 2022-04-03T19:49:44.714 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/165/12' 2022-04-03T19:49:44.714 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/165/10' 2022-04-03T19:49:44.714 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/165/11' 2022-04-03T19:49:44.714 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/165/16' 2022-04-03T19:49:44.715 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/165/14' 2022-04-03T19:49:44.715 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/165/18' 2022-04-03T19:49:44.715 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/165/13' 2022-04-03T19:49:44.715 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/165/17' 2022-04-03T19:49:44.715 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/165' 2022-04-03T19:49:44.716 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/101/19' 2022-04-03T19:49:44.716 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/101/15' 2022-04-03T19:49:44.716 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/101/20' 2022-04-03T19:49:44.716 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/101/12' 2022-04-03T19:49:44.717 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/101/10' 2022-04-03T19:49:44.717 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/101/11' 2022-04-03T19:49:44.717 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/101/16' 2022-04-03T19:49:44.718 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/101/14' 2022-04-03T19:49:44.718 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/101/18' 2022-04-03T19:49:44.718 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/101/13' 2022-04-03T19:49:44.718 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/101/17' 2022-04-03T19:49:44.719 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/101' 2022-04-03T19:49:44.719 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/148/19' 2022-04-03T19:49:44.719 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/148/15' 2022-04-03T19:49:44.720 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/148/20' 2022-04-03T19:49:44.720 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/148/12' 2022-04-03T19:49:44.720 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/148/10' 2022-04-03T19:49:44.720 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/148/11' 2022-04-03T19:49:44.721 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/148/16' 2022-04-03T19:49:44.721 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/148/14' 2022-04-03T19:49:44.721 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/148/18' 2022-04-03T19:49:44.721 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/148/13' 2022-04-03T19:49:44.721 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/148/17' 2022-04-03T19:49:44.722 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/148' 2022-04-03T19:49:44.722 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/168/19' 2022-04-03T19:49:44.722 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/168/15' 2022-04-03T19:49:44.722 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/168/20' 2022-04-03T19:49:44.723 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/168/12' 2022-04-03T19:49:44.723 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/168/10' 2022-04-03T19:49:44.723 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/168/11' 2022-04-03T19:49:44.723 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/168/16' 2022-04-03T19:49:44.723 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/168/14' 2022-04-03T19:49:44.724 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/168/18' 2022-04-03T19:49:44.724 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/168/13' 2022-04-03T19:49:44.724 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/168/17' 2022-04-03T19:49:44.724 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/168' 2022-04-03T19:49:44.948 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.947+0000 7f9af359e700 1 -- 172.21.15.77:0/503471721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9af404b050 con 0x7f9af40678b0 2022-04-03T19:49:44.949 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.948+0000 7f9aea7fc700 1 -- 172.21.15.77:0/503471721 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6113 (secure 0 0 0) 0x7f9ae4017070 con 0x7f9af40678b0 2022-04-03T19:49:44.950 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:49:44.950 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4850,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:48:43.168138+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4960},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4960":{"gid":4960,"name":"e","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:49:44.952 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.950+0000 7f9af359e700 1 -- 172.21.15.77:0/503471721 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9ad8063ce0 msgr2=0x7f9ad8066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:44.952 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.950+0000 7f9af359e700 1 --2- 172.21.15.77:0/503471721 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9ad8063ce0 0x7f9ad8066190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9af414ae50 tx=0x7f9adc009040 comp rx=0 tx=0).stop 2022-04-03T19:49:44.953 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.950+0000 7f9af359e700 1 -- 172.21.15.77:0/503471721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9af40678b0 msgr2=0x7f9af41480f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:44.953 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.950+0000 7f9af359e700 1 --2- 172.21.15.77:0/503471721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9af40678b0 0x7f9af41480f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9ae400c520 tx=0x7f9ae4007ca0 comp rx=0 tx=0).stop 2022-04-03T19:49:44.953 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.950+0000 7f9af359e700 1 -- 172.21.15.77:0/503471721 shutdown_connections 2022-04-03T19:49:44.953 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.950+0000 7f9af359e700 1 --2- 172.21.15.77:0/503471721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9af40c1b30 0x7f9af40c5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:44.954 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.950+0000 7f9af359e700 1 --2- 172.21.15.77:0/503471721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9af40678b0 0x7f9af41480f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:44.954 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.950+0000 7f9af359e700 1 --2- 172.21.15.77:0/503471721 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9ad8063ce0 0x7f9ad8066190 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:44.954 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.950+0000 7f9af359e700 1 --2- 172.21.15.77:0/503471721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9af40ba3c0 0x7f9af416c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:44.954 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.950+0000 7f9af359e700 1 -- 172.21.15.77:0/503471721 >> 172.21.15.77:0/503471721 conn(0x7f9af4066da0 msgr2=0x7f9af40c3420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:49:44.954 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.950+0000 7f9af359e700 1 -- 172.21.15.77:0/503471721 shutdown_connections 2022-04-03T19:49:44.955 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:44.950+0000 7f9af359e700 1 -- 172.21.15.77:0/503471721 wait complete. 2022-04-03T19:49:44.955 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 20 2022-04-03T19:49:44.980 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2022-04-03T19:49:44.981 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2022-04-03T19:49:44.981 INFO:teuthology.orchestra.run:waiting for 300 2022-04-03T19:49:45.020 INFO:tasks.ceph.mds.e:Stopped 2022-04-03T19:49:45.021 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:49:45.219 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.217+0000 7f45bb00b700 1 Processor -- start 2022-04-03T19:49:45.219 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.217+0000 7f45bb00b700 1 -- start start 2022-04-03T19:49:45.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.219+0000 7f45bb00b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f45b40678b0 0x7f45b414cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:45.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.219+0000 7f45bb00b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b40c1b30 0x7f45b414d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:45.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.219+0000 7f45bb00b700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b40c3c50 con 0x7f45b40ba3c0 2022-04-03T19:49:45.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.219+0000 7f45bb00b700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b40c8020 con 0x7f45b40678b0 2022-04-03T19:49:45.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.219+0000 7f45bb00b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b4156690 con 0x7f45b40c1b30 2022-04-03T19:49:45.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.219+0000 7f45ba009700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f45b40ba3c0 0x7f45b4156b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44244/0 (socket says 172.21.15.77:44244) 2022-04-03T19:49:45.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.219+0000 7f45b9808700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f45b40678b0 0x7f45b414cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:45.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.219+0000 7f45ba009700 1 -- 172.21.15.77:0/430427233 learned_addr learned my addr 172.21.15.77:0/430427233 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:49:45.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.219+0000 7f45ba80a700 1 --2- 172.21.15.77:0/430427233 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b40c1b30 0x7f45b414d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:45.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.220+0000 7f45b9808700 1 -- 172.21.15.77:0/430427233 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b40c1b30 msgr2=0x7f45b414d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:49:45.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.220+0000 7f45b9808700 1 --2- 172.21.15.77:0/430427233 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b40c1b30 0x7f45b414d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:45.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.220+0000 7f45b9808700 1 -- 172.21.15.77:0/430427233 >> v1:172.21.15.186:6789/0 conn(0x7f45b40ba3c0 legacy=0x7f45b4156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:45.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.219+0000 7f45b9007700 1 -- 172.21.15.77:0/430427233 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1614990254 0 0) 0x7f45b40c3c50 con 0x7f45b40ba3c0 2022-04-03T19:49:45.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.220+0000 7f45b9808700 1 -- 172.21.15.77:0/430427233 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b40bf350 con 0x7f45b40678b0 2022-04-03T19:49:45.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.220+0000 7f45b9808700 1 --2- 172.21.15.77:0/430427233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45b40678b0 0x7f45b414cb00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f45a4009760 tx=0x7f45a4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=100b3d9263c215f5 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:45.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.220+0000 7f45b9007700 1 -- 172.21.15.77:0/430427233 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a40108a0 con 0x7f45b40678b0 2022-04-03T19:49:45.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.220+0000 7f45b9007700 1 -- 172.21.15.77:0/430427233 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a4010a00 con 0x7f45b40678b0 2022-04-03T19:49:45.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.220+0000 7f45b9007700 1 -- 172.21.15.77:0/430427233 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4010dc0 con 0x7f45b40678b0 2022-04-03T19:49:45.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.221+0000 7f45bb00b700 1 -- 172.21.15.77:0/430427233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45b40678b0 msgr2=0x7f45b414cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:45.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.221+0000 7f45bb00b700 1 --2- 172.21.15.77:0/430427233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45b40678b0 0x7f45b414cb00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f45a4009760 tx=0x7f45a4009060 comp rx=0 tx=0).stop 2022-04-03T19:49:45.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.221+0000 7f45bb00b700 1 -- 172.21.15.77:0/430427233 shutdown_connections 2022-04-03T19:49:45.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.221+0000 7f45bb00b700 1 --2- 172.21.15.77:0/430427233 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b40c1b30 0x7f45b414d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:45.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.221+0000 7f45bb00b700 1 --2- 172.21.15.77:0/430427233 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45b40678b0 0x7f45b414cb00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:45.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.221+0000 7f45bb00b700 1 -- 172.21.15.77:0/430427233 >> 172.21.15.77:0/430427233 conn(0x7f45b4066da0 msgr2=0x7f45b40671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:49:45.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.221+0000 7f45bb00b700 1 -- 172.21.15.77:0/430427233 shutdown_connections 2022-04-03T19:49:45.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.221+0000 7f45bb00b700 1 -- 172.21.15.77:0/430427233 wait complete. 2022-04-03T19:49:45.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.222+0000 7f45bb00b700 1 Processor -- start 2022-04-03T19:49:45.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.222+0000 7f45bb00b700 1 -- start start 2022-04-03T19:49:45.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.223+0000 7f45bb00b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b40678b0 0x7f45b4143030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:45.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.223+0000 7f45bb00b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45b40ba3c0 0x7f45b414fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:45.230 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.223+0000 7f45bb00b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45b40c1b30 0x7f45b4150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:45.230 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.223+0000 7f45bb00b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b4143b00 con 0x7f45b40ba3c0 2022-04-03T19:49:45.230 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.223+0000 7f45bb00b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b4143570 con 0x7f45b40c1b30 2022-04-03T19:49:45.230 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.223+0000 7f45bb00b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b41506c0 con 0x7f45b40678b0 2022-04-03T19:49:45.230 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.224+0000 7f45b9808700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45b40ba3c0 0x7f45b414fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:45.231 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.224+0000 7f45ba80a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45b40c1b30 0x7f45b4150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:45.231 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.224+0000 7f45b9808700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45b40ba3c0 0x7f45b414fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42892/0 (socket says 172.21.15.77:42892) 2022-04-03T19:49:45.231 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.224+0000 7f45ba009700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b40678b0 0x7f45b4143030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:45.231 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.224+0000 7f45b9808700 1 -- 172.21.15.77:0/4278872377 learned_addr learned my addr 172.21.15.77:0/4278872377 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:49:45.231 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.224+0000 7f45b9808700 1 -- 172.21.15.77:0/4278872377 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b40678b0 msgr2=0x7f45b4143030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:45.232 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.224+0000 7f45b9808700 1 --2- 172.21.15.77:0/4278872377 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b40678b0 0x7f45b4143030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:45.232 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.224+0000 7f45b9808700 1 -- 172.21.15.77:0/4278872377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45b40c1b30 msgr2=0x7f45b4150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:45.232 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.224+0000 7f45b9808700 1 --2- 172.21.15.77:0/4278872377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45b40c1b30 0x7f45b4150010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:45.232 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.224+0000 7f45b9808700 1 -- 172.21.15.77:0/4278872377 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a40092a0 con 0x7f45b40ba3c0 2022-04-03T19:49:45.232 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.225+0000 7f45b9808700 1 --2- 172.21.15.77:0/4278872377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45b40ba3c0 0x7f45b414fad0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f45b4067fa0 tx=0x7f45a4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:45.234 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.227+0000 7f45a27fc700 1 -- 172.21.15.77:0/4278872377 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4019e30 con 0x7f45b40ba3c0 2022-04-03T19:49:45.234 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.227+0000 7f45a27fc700 1 -- 172.21.15.77:0/4278872377 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a4004050 con 0x7f45b40ba3c0 2022-04-03T19:49:45.234 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.227+0000 7f45bb00b700 1 -- 172.21.15.77:0/4278872377 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b414e280 con 0x7f45b40ba3c0 2022-04-03T19:49:45.234 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.227+0000 7f45a27fc700 1 -- 172.21.15.77:0/4278872377 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4004410 con 0x7f45b40ba3c0 2022-04-03T19:49:45.235 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.227+0000 7f45bb00b700 1 -- 172.21.15.77:0/4278872377 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b414d950 con 0x7f45b40ba3c0 2022-04-03T19:49:45.235 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.228+0000 7f45bb00b700 1 -- 172.21.15.77:0/4278872377 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b40c74a0 con 0x7f45b40ba3c0 2022-04-03T19:49:45.235 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.229+0000 7f45a27fc700 1 -- 172.21.15.77:0/4278872377 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f45a402a030 con 0x7f45b40ba3c0 2022-04-03T19:49:45.235 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.230+0000 7f45a27fc700 1 --2- 172.21.15.77:0/4278872377 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4594063ce0 0x7f4594066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.230+0000 7f45a27fc700 1 -- 172.21.15.77:0/4278872377 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6288+0+0 (secure 0 0 0) 0x7f45a4085670 con 0x7f45b40ba3c0 2022-04-03T19:49:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.230+0000 7f45ba009700 1 --2- 172.21.15.77:0/4278872377 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4594063ce0 0x7f4594066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.231+0000 7f45ba009700 1 --2- 172.21.15.77:0/4278872377 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4594063ce0 0x7f4594066190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f45b0007fd0 tx=0x7f45b0007bb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.234+0000 7f45a27fc700 1 -- 172.21.15.77:0/4278872377 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f45a40540a0 con 0x7f45b40ba3c0 2022-04-03T19:49:45.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.465+0000 7f45bb00b700 1 -- 172.21.15.77:0/4278872377 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45b40c1620 con 0x7f45b40ba3c0 2022-04-03T19:49:45.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.466+0000 7f45a27fc700 1 -- 172.21.15.77:0/4278872377 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6113 (secure 0 0 0) 0x7f45a40576c0 con 0x7f45b40ba3c0 2022-04-03T19:49:45.467 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:49:45.468 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4850,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:48:43.168138+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4960},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4960":{"gid":4960,"name":"e","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:49:45.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.468+0000 7f45bb00b700 1 -- 172.21.15.77:0/4278872377 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4594063ce0 msgr2=0x7f4594066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:45.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.468+0000 7f45bb00b700 1 --2- 172.21.15.77:0/4278872377 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4594063ce0 0x7f4594066190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f45b0007fd0 tx=0x7f45b0007bb0 comp rx=0 tx=0).stop 2022-04-03T19:49:45.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.468+0000 7f45bb00b700 1 -- 172.21.15.77:0/4278872377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45b40ba3c0 msgr2=0x7f45b414fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:45.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.468+0000 7f45bb00b700 1 --2- 172.21.15.77:0/4278872377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45b40ba3c0 0x7f45b414fad0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f45b4067fa0 tx=0x7f45a4018000 comp rx=0 tx=0).stop 2022-04-03T19:49:45.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.468+0000 7f45bb00b700 1 -- 172.21.15.77:0/4278872377 shutdown_connections 2022-04-03T19:49:45.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.468+0000 7f45bb00b700 1 --2- 172.21.15.77:0/4278872377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45b40c1b30 0x7f45b4150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:45.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.468+0000 7f45bb00b700 1 --2- 172.21.15.77:0/4278872377 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4594063ce0 0x7f4594066190 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:45.472 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.468+0000 7f45bb00b700 1 --2- 172.21.15.77:0/4278872377 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b40678b0 0x7f45b4143030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:45.472 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.468+0000 7f45bb00b700 1 --2- 172.21.15.77:0/4278872377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45b40ba3c0 0x7f45b414fad0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:45.472 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.468+0000 7f45bb00b700 1 -- 172.21.15.77:0/4278872377 >> 172.21.15.77:0/4278872377 conn(0x7f45b4066da0 msgr2=0x7f45b40be5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:49:45.472 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.469+0000 7f45bb00b700 1 -- 172.21.15.77:0/4278872377 shutdown_connections 2022-04-03T19:49:45.473 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:45.469+0000 7f45bb00b700 1 -- 172.21.15.77:0/4278872377 wait complete. 2022-04-03T19:49:45.473 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 20 2022-04-03T19:49:45.494 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:49:45.875 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:49:45.880 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:49:47.495 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:49:47.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.700+0000 7ff91b59e700 1 Processor -- start 2022-04-03T19:49:47.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.700+0000 7ff91b59e700 1 -- start start 2022-04-03T19:49:47.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.701+0000 7ff91b59e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff91c14f360 0x7ff91c0c2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:47.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.701+0000 7ff91b59e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff91c0c5050 con 0x7ff91c0c1fa0 2022-04-03T19:49:47.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.701+0000 7ff91b59e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff91c1559b0 con 0x7ff91c14ef90 2022-04-03T19:49:47.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.701+0000 7ff91b59e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff91c0c4330 con 0x7ff91c14f360 2022-04-03T19:49:47.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.702+0000 7ff91ad9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff91c0c1fa0 0x7ff91c0626d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38450/0 (socket says 172.21.15.77:38450) 2022-04-03T19:49:47.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.702+0000 7ff91ad9d700 1 -- 172.21.15.77:0/2603607298 learned_addr learned my addr 172.21.15.77:0/2603607298 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:49:47.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.702+0000 7ff919d9b700 1 --2- 172.21.15.77:0/2603607298 >> v2:172.21.15.186:3300/0 conn(0x7ff91c14f360 0x7ff91c0c2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:47.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.702+0000 7ff91959a700 1 -- 172.21.15.77:0/2603607298 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545881044 0 0) 0x7ff91c0c5050 con 0x7ff91c0c1fa0 2022-04-03T19:49:47.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.702+0000 7ff91959a700 1 -- 172.21.15.77:0/2603607298 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9000033f0 con 0x7ff91c0c1fa0 2022-04-03T19:49:47.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.702+0000 7ff91959a700 1 -- 172.21.15.77:0/2603607298 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424982632 0 0) 0x7ff91c1559b0 con 0x7ff91c14ef90 2022-04-03T19:49:47.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.702+0000 7ff91959a700 1 -- 172.21.15.77:0/2603607298 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff91c0c5050 con 0x7ff91c14ef90 2022-04-03T19:49:47.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.702+0000 7ff919d9b700 1 -- 172.21.15.77:0/2603607298 >> v1:172.21.15.186:6789/0 conn(0x7ff91c14ef90 legacy=0x7ff91c0c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:47.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.702+0000 7ff919d9b700 1 -- 172.21.15.77:0/2603607298 >> v1:172.21.15.77:6789/0 conn(0x7ff91c0c1fa0 legacy=0x7ff91c0626d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:47.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.703+0000 7ff919d9b700 1 -- 172.21.15.77:0/2603607298 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91c0c1bb0 con 0x7ff91c14f360 2022-04-03T19:49:47.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.703+0000 7ff919d9b700 1 --2- 172.21.15.77:0/2603607298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff91c14f360 0x7ff91c0c2c90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff910009760 tx=0x7ff910009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecff8525c3027293 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:47.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.703+0000 7ff91959a700 1 -- 172.21.15.77:0/2603607298 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff910011630 con 0x7ff91c14f360 2022-04-03T19:49:47.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.703+0000 7ff91959a700 1 -- 172.21.15.77:0/2603607298 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff910011790 con 0x7ff91c14f360 2022-04-03T19:49:47.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.703+0000 7ff91959a700 1 -- 172.21.15.77:0/2603607298 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff910011b10 con 0x7ff91c14f360 2022-04-03T19:49:47.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.704+0000 7ff91b59e700 1 -- 172.21.15.77:0/2603607298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff91c14f360 msgr2=0x7ff91c0c2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:47.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.704+0000 7ff91b59e700 1 --2- 172.21.15.77:0/2603607298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff91c14f360 0x7ff91c0c2c90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff910009760 tx=0x7ff910009060 comp rx=0 tx=0).stop 2022-04-03T19:49:47.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.704+0000 7ff91b59e700 1 -- 172.21.15.77:0/2603607298 shutdown_connections 2022-04-03T19:49:47.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.704+0000 7ff91b59e700 1 --2- 172.21.15.77:0/2603607298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff91c14f360 0x7ff91c0c2c90 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:47.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.704+0000 7ff91b59e700 1 -- 172.21.15.77:0/2603607298 >> 172.21.15.77:0/2603607298 conn(0x7ff91c06e340 msgr2=0x7ff91c0677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:49:47.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.704+0000 7ff91b59e700 1 -- 172.21.15.77:0/2603607298 shutdown_connections 2022-04-03T19:49:47.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.704+0000 7ff91b59e700 1 -- 172.21.15.77:0/2603607298 wait complete. 2022-04-03T19:49:47.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.705+0000 7ff91b59e700 1 Processor -- start 2022-04-03T19:49:47.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.706+0000 7ff91b59e700 1 -- start start 2022-04-03T19:49:47.712 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.706+0000 7ff91b59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff91c0c1fa0 0x7ff91c164cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:47.712 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.706+0000 7ff91b59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff91c14ef90 0x7ff91c0c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:47.713 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.706+0000 7ff91b59e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff91c14f360 0x7ff91c14d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:47.713 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.706+0000 7ff91b59e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c0c7fb0 con 0x7ff91c0c1fa0 2022-04-03T19:49:47.713 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.706+0000 7ff91b59e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c14e220 con 0x7ff91c14f360 2022-04-03T19:49:47.713 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.706+0000 7ff91b59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c14dc60 con 0x7ff91c14ef90 2022-04-03T19:49:47.713 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.707+0000 7ff91a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff91c0c1fa0 0x7ff91c164cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:47.714 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.707+0000 7ff91a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff91c0c1fa0 0x7ff91c164cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42894/0 (socket says 172.21.15.77:42894) 2022-04-03T19:49:47.714 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.707+0000 7ff91a59c700 1 -- 172.21.15.77:0/2164263925 learned_addr learned my addr 172.21.15.77:0/2164263925 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:49:47.714 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.707+0000 7ff919d9b700 1 --2- 172.21.15.77:0/2164263925 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff91c14ef90 0x7ff91c0c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:47.714 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.707+0000 7ff91ad9d700 1 --2- 172.21.15.77:0/2164263925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff91c14f360 0x7ff91c14d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:47.714 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.707+0000 7ff91a59c700 1 -- 172.21.15.77:0/2164263925 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff91c14ef90 msgr2=0x7ff91c0c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:47.715 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.707+0000 7ff91a59c700 1 --2- 172.21.15.77:0/2164263925 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff91c14ef90 0x7ff91c0c65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:47.715 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.707+0000 7ff91a59c700 1 -- 172.21.15.77:0/2164263925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff91c14f360 msgr2=0x7ff91c14d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:47.715 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.707+0000 7ff91a59c700 1 --2- 172.21.15.77:0/2164263925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff91c14f360 0x7ff91c14d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:47.715 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.707+0000 7ff91a59c700 1 -- 172.21.15.77:0/2164263925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9100092a0 con 0x7ff91c0c1fa0 2022-04-03T19:49:47.715 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.707+0000 7ff919d9b700 1 --2- 172.21.15.77:0/2164263925 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff91c14ef90 0x7ff91c0c65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:49:47.716 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.707+0000 7ff91a59c700 1 --2- 172.21.15.77:0/2164263925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff91c0c1fa0 0x7ff91c164cf0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff90c00c490 tx=0x7ff90c007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:47.716 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.708+0000 7ff90a7fc700 1 -- 172.21.15.77:0/2164263925 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90c0050d0 con 0x7ff91c0c1fa0 2022-04-03T19:49:47.716 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.708+0000 7ff90a7fc700 1 -- 172.21.15.77:0/2164263925 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff90c005250 con 0x7ff91c0c1fa0 2022-04-03T19:49:47.716 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.708+0000 7ff90a7fc700 1 -- 172.21.15.77:0/2164263925 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90c0053c0 con 0x7ff91c0c1fa0 2022-04-03T19:49:47.716 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.708+0000 7ff91b59e700 1 -- 172.21.15.77:0/2164263925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff91c14df40 con 0x7ff91c0c1fa0 2022-04-03T19:49:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.708+0000 7ff91b59e700 1 -- 172.21.15.77:0/2164263925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff91c0c5dd0 con 0x7ff91c0c1fa0 2022-04-03T19:49:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.709+0000 7ff91b59e700 1 -- 172.21.15.77:0/2164263925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff91c0c6b10 con 0x7ff91c0c1fa0 2022-04-03T19:49:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.710+0000 7ff90a7fc700 1 -- 172.21.15.77:0/2164263925 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7ff90c019030 con 0x7ff91c0c1fa0 2022-04-03T19:49:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.711+0000 7ff90a7fc700 1 --2- 172.21.15.77:0/2164263925 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff8f8063d10 0x7ff8f80661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.711+0000 7ff90a7fc700 1 -- 172.21.15.77:0/2164263925 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6288+0+0 (secure 0 0 0) 0x7ff90c07fa40 con 0x7ff91c0c1fa0 2022-04-03T19:49:47.718 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.711+0000 7ff919d9b700 1 --2- 172.21.15.77:0/2164263925 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff8f8063d10 0x7ff8f80661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:47.718 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.712+0000 7ff919d9b700 1 --2- 172.21.15.77:0/2164263925 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff8f8063d10 0x7ff8f80661c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff91c15aca0 tx=0x7ff910017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:47.718 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.715+0000 7ff90a7fc700 1 -- 172.21.15.77:0/2164263925 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7ff90c04e540 con 0x7ff91c0c1fa0 2022-04-03T19:49:47.943 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.943+0000 7ff91b59e700 1 -- 172.21.15.77:0/2164263925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff91c04b050 con 0x7ff91c0c1fa0 2022-04-03T19:49:47.944 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.944+0000 7ff90a7fc700 1 -- 172.21.15.77:0/2164263925 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6113 (secure 0 0 0) 0x7ff90c017090 con 0x7ff91c0c1fa0 2022-04-03T19:49:47.945 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:49:47.945 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4850,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:48:43.168138+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4960},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4960":{"gid":4960,"name":"e","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:49:47.947 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.946+0000 7ff91b59e700 1 -- 172.21.15.77:0/2164263925 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff8f8063d10 msgr2=0x7ff8f80661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:47.947 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.946+0000 7ff91b59e700 1 --2- 172.21.15.77:0/2164263925 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff8f8063d10 0x7ff8f80661c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff91c15aca0 tx=0x7ff910017000 comp rx=0 tx=0).stop 2022-04-03T19:49:47.947 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.946+0000 7ff91b59e700 1 -- 172.21.15.77:0/2164263925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff91c0c1fa0 msgr2=0x7ff91c164cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:47.948 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.946+0000 7ff91b59e700 1 --2- 172.21.15.77:0/2164263925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff91c0c1fa0 0x7ff91c164cf0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff90c00c490 tx=0x7ff90c007c70 comp rx=0 tx=0).stop 2022-04-03T19:49:47.948 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.946+0000 7ff91b59e700 1 -- 172.21.15.77:0/2164263925 shutdown_connections 2022-04-03T19:49:47.948 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.946+0000 7ff91b59e700 1 --2- 172.21.15.77:0/2164263925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff91c14f360 0x7ff91c14d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:47.948 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.946+0000 7ff91b59e700 1 --2- 172.21.15.77:0/2164263925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff91c0c1fa0 0x7ff91c164cf0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:47.949 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.946+0000 7ff91b59e700 1 --2- 172.21.15.77:0/2164263925 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff8f8063d10 0x7ff8f80661c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:47.949 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.946+0000 7ff91b59e700 1 --2- 172.21.15.77:0/2164263925 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff91c14ef90 0x7ff91c0c65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:47.949 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.946+0000 7ff91b59e700 1 -- 172.21.15.77:0/2164263925 >> 172.21.15.77:0/2164263925 conn(0x7ff91c06e340 msgr2=0x7ff91c14e860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:49:47.949 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.946+0000 7ff91b59e700 1 -- 172.21.15.77:0/2164263925 shutdown_connections 2022-04-03T19:49:47.950 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:47.946+0000 7ff91b59e700 1 -- 172.21.15.77:0/2164263925 wait complete. 2022-04-03T19:49:47.950 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 20 2022-04-03T19:49:47.971 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:49:49.973 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:49:50.181 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.180+0000 7f19a7fff700 1 Processor -- start 2022-04-03T19:49:50.182 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.180+0000 7f19a7fff700 1 -- start start 2022-04-03T19:49:50.188 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.181+0000 7f19a7fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19a80c1170 0x7f19a81527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:50.188 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.181+0000 7f19a7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f19a80c3b10 0x7f19a815c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:50.189 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.181+0000 7f19a7fff700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19a80c3710 con 0x7f19a8155370 2022-04-03T19:49:50.189 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.181+0000 7f19a7fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a8152330 con 0x7f19a80c3b10 2022-04-03T19:49:50.189 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.181+0000 7f19a7fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a80c1540 con 0x7f19a80c1170 2022-04-03T19:49:50.190 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.182+0000 7f19a77fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f19a80c3b10 0x7f19a815c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:50.192 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.182+0000 7f19a6ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19a80c1170 0x7f19a81527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:50.192 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.182+0000 7f19a6ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19a80c1170 0x7f19a81527c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60190/0 (socket says 172.21.15.77:60190) 2022-04-03T19:49:50.192 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.182+0000 7f19a6ffd700 1 -- 172.21.15.77:0/232849712 learned_addr learned my addr 172.21.15.77:0/232849712 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:49:50.193 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.182+0000 7f19a67fc700 1 --1- 172.21.15.77:0/232849712 >> v1:172.21.15.186:6789/0 conn(0x7f19a8155370 0x7f19a8069110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44250/0 (socket says 172.21.15.77:44250) 2022-04-03T19:49:50.193 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.182+0000 7f19a77fe700 1 --2- 172.21.15.77:0/232849712 >> v2:172.21.15.77:3300/0 conn(0x7f19a80c3b10 0x7f19a815c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42896/0 (socket says 172.21.15.77:42896) 2022-04-03T19:49:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.182+0000 7f19a6ffd700 1 -- 172.21.15.77:0/232849712 >> v2:172.21.15.77:3300/0 conn(0x7f19a80c3b10 msgr2=0x7f19a815c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:49:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.182+0000 7f19a6ffd700 1 --2- 172.21.15.77:0/232849712 >> v2:172.21.15.77:3300/0 conn(0x7f19a80c3b10 0x7f19a815c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.182+0000 7f19a6ffd700 1 -- 172.21.15.77:0/232849712 >> v1:172.21.15.186:6789/0 conn(0x7f19a8155370 legacy=0x7f19a8069110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.182+0000 7f19a6ffd700 1 -- 172.21.15.77:0/232849712 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a80c4270 con 0x7f19a80c1170 2022-04-03T19:49:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.182+0000 7f19a5ffb700 1 -- 172.21.15.77:0/232849712 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665347995 0 0) 0x7f19a80c3710 con 0x7f19a8155370 2022-04-03T19:49:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.183+0000 7f19a6ffd700 1 --2- 172.21.15.77:0/232849712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19a80c1170 0x7f19a81527c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1998006fd0 tx=0x7f199800b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce59503859e17dfc server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.183+0000 7f19a5ffb700 1 -- 172.21.15.77:0/232849712 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199800db00 con 0x7f19a80c1170 2022-04-03T19:49:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.183+0000 7f19a5ffb700 1 -- 172.21.15.77:0/232849712 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199800dc60 con 0x7f19a80c1170 2022-04-03T19:49:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.183+0000 7f19a5ffb700 1 -- 172.21.15.77:0/232849712 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1998004030 con 0x7f19a80c1170 2022-04-03T19:49:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.183+0000 7f19a7fff700 1 -- 172.21.15.77:0/232849712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19a80c1170 msgr2=0x7f19a81527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.183+0000 7f19a7fff700 1 --2- 172.21.15.77:0/232849712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19a80c1170 0x7f19a81527c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1998006fd0 tx=0x7f199800b3b0 comp rx=0 tx=0).stop 2022-04-03T19:49:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.183+0000 7f19a7fff700 1 -- 172.21.15.77:0/232849712 shutdown_connections 2022-04-03T19:49:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.183+0000 7f19a7fff700 1 --2- 172.21.15.77:0/232849712 >> v2:172.21.15.77:3300/0 conn(0x7f19a80c3b10 0x7f19a815c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:50.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.183+0000 7f19a7fff700 1 --2- 172.21.15.77:0/232849712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19a80c1170 0x7f19a81527c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:50.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.183+0000 7f19a7fff700 1 -- 172.21.15.77:0/232849712 >> 172.21.15.77:0/232849712 conn(0x7f19a8066da0 msgr2=0x7f19a80671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:49:50.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.184+0000 7f19a7fff700 1 -- 172.21.15.77:0/232849712 shutdown_connections 2022-04-03T19:49:50.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.184+0000 7f19a7fff700 1 -- 172.21.15.77:0/232849712 wait complete. 2022-04-03T19:49:50.201 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.185+0000 7f19a7fff700 1 Processor -- start 2022-04-03T19:49:50.202 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.185+0000 7f19a7fff700 1 -- start start 2022-04-03T19:49:50.203 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.186+0000 7f19a7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a80c1170 0x7f19a8067330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:50.203 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.186+0000 7f19a7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19a80c3b10 0x7f19a814ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:50.203 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.186+0000 7f19a7fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19a8155370 0x7f19a814e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:50.204 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.186+0000 7f19a7fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a814cbd0 con 0x7f19a80c3b10 2022-04-03T19:49:50.204 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.186+0000 7f19a7fff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a8157fc0 con 0x7f19a80c1170 2022-04-03T19:49:50.204 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.186+0000 7f19a7fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a8155130 con 0x7f19a8155370 2022-04-03T19:49:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.186+0000 7f19a67fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19a80c3b10 0x7f19a814ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.186+0000 7f19a6ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a80c1170 0x7f19a8067330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.187+0000 7f19a6ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a80c1170 0x7f19a8067330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38088/0 (socket says 172.21.15.77:38088) 2022-04-03T19:49:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.187+0000 7f19a77fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19a8155370 0x7f19a814e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.187+0000 7f19a6ffd700 1 -- 172.21.15.77:0/1939554357 learned_addr learned my addr 172.21.15.77:0/1939554357 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:49:50.207 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.187+0000 7f19a67fc700 1 -- 172.21.15.77:0/1939554357 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19a8155370 msgr2=0x7f19a814e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:50.207 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.187+0000 7f19a67fc700 1 --2- 172.21.15.77:0/1939554357 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19a8155370 0x7f19a814e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:50.207 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.187+0000 7f19a67fc700 1 -- 172.21.15.77:0/1939554357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a80c1170 msgr2=0x7f19a8067330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:50.208 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.187+0000 7f19a67fc700 1 --2- 172.21.15.77:0/1939554357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a80c1170 0x7f19a8067330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:50.209 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.187+0000 7f19a67fc700 1 -- 172.21.15.77:0/1939554357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199800b610 con 0x7f19a80c3b10 2022-04-03T19:49:50.209 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.187+0000 7f19a6ffd700 1 --2- 172.21.15.77:0/1939554357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a80c1170 0x7f19a8067330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:49:50.209 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.187+0000 7f19a77fe700 1 --2- 172.21.15.77:0/1939554357 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19a8155370 0x7f19a814e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:49:50.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.187+0000 7f19a67fc700 1 --2- 172.21.15.77:0/1939554357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19a80c3b10 0x7f19a814ddd0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f199c00c440 tx=0x7f199c007cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:50.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.187+0000 7f198effd700 1 -- 172.21.15.77:0/1939554357 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c007150 con 0x7f19a80c3b10 2022-04-03T19:49:50.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.188+0000 7f198effd700 1 -- 172.21.15.77:0/1939554357 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199c0072b0 con 0x7f19a80c3b10 2022-04-03T19:49:50.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.188+0000 7f198effd700 1 -- 172.21.15.77:0/1939554357 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c0046b0 con 0x7f19a80c3b10 2022-04-03T19:49:50.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.188+0000 7f19a7fff700 1 -- 172.21.15.77:0/1939554357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a81619d0 con 0x7f19a80c3b10 2022-04-03T19:49:50.212 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.188+0000 7f19a7fff700 1 -- 172.21.15.77:0/1939554357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a81573f0 con 0x7f19a80c3b10 2022-04-03T19:49:50.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.189+0000 7f19a7fff700 1 -- 172.21.15.77:0/1939554357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a806c460 con 0x7f19a80c3b10 2022-04-03T19:49:50.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.189+0000 7f198effd700 1 -- 172.21.15.77:0/1939554357 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f199c010030 con 0x7f19a80c3b10 2022-04-03T19:49:50.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.190+0000 7f198effd700 1 --2- 172.21.15.77:0/1939554357 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1988063ce0 0x7f1988066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:50.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.190+0000 7f198effd700 1 -- 172.21.15.77:0/1939554357 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6288+0+0 (secure 0 0 0) 0x7f199c0801c0 con 0x7f19a80c3b10 2022-04-03T19:49:50.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.191+0000 7f19a6ffd700 1 --2- 172.21.15.77:0/1939554357 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1988063ce0 0x7f1988066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:50.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.191+0000 7f19a6ffd700 1 --2- 172.21.15.77:0/1939554357 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1988063ce0 0x7f1988066190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f199800b3f0 tx=0x7f199801a000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:50.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.194+0000 7f198effd700 1 -- 172.21.15.77:0/1939554357 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f199c04ed40 con 0x7f19a80c3b10 2022-04-03T19:49:50.425 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.425+0000 7f19a7fff700 1 -- 172.21.15.77:0/1939554357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19a804b050 con 0x7f19a80c3b10 2022-04-03T19:49:50.426 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.425+0000 7f198effd700 1 -- 172.21.15.77:0/1939554357 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6113 (secure 0 0 0) 0x7f199c052360 con 0x7f19a80c3b10 2022-04-03T19:49:50.426 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:49:50.427 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4850,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:48:43.168138+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4960},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4960":{"gid":4960,"name":"e","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:49:50.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.428+0000 7f19a7fff700 1 -- 172.21.15.77:0/1939554357 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1988063ce0 msgr2=0x7f1988066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:50.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.428+0000 7f19a7fff700 1 --2- 172.21.15.77:0/1939554357 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1988063ce0 0x7f1988066190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f199800b3f0 tx=0x7f199801a000 comp rx=0 tx=0).stop 2022-04-03T19:49:50.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.428+0000 7f19a7fff700 1 -- 172.21.15.77:0/1939554357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19a80c3b10 msgr2=0x7f19a814ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:50.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.428+0000 7f19a7fff700 1 --2- 172.21.15.77:0/1939554357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19a80c3b10 0x7f19a814ddd0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f199c00c440 tx=0x7f199c007cd0 comp rx=0 tx=0).stop 2022-04-03T19:49:50.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.428+0000 7f19a7fff700 1 -- 172.21.15.77:0/1939554357 shutdown_connections 2022-04-03T19:49:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.428+0000 7f19a7fff700 1 --2- 172.21.15.77:0/1939554357 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1988063ce0 0x7f1988066190 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.428+0000 7f19a7fff700 1 --2- 172.21.15.77:0/1939554357 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19a8155370 0x7f19a814e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.428+0000 7f19a7fff700 1 --2- 172.21.15.77:0/1939554357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a80c1170 0x7f19a8067330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.428+0000 7f19a7fff700 1 --2- 172.21.15.77:0/1939554357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f19a80c3b10 0x7f19a814ddd0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:50.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.428+0000 7f19a7fff700 1 -- 172.21.15.77:0/1939554357 >> 172.21.15.77:0/1939554357 conn(0x7f19a8066da0 msgr2=0x7f19a80bf7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:49:50.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.428+0000 7f19a7fff700 1 -- 172.21.15.77:0/1939554357 shutdown_connections 2022-04-03T19:49:50.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:50.428+0000 7f19a7fff700 1 -- 172.21.15.77:0/1939554357 wait complete. 2022-04-03T19:49:50.432 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 20 2022-04-03T19:49:50.452 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:49:52.454 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:49:52.659 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.659+0000 7fcff4d2d700 1 Processor -- start 2022-04-03T19:49:52.660 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.659+0000 7fcff4d2d700 1 -- start start 2022-04-03T19:49:52.661 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.660+0000 7fcff4d2d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcff006bfb0 0x7fcff01584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:52.661 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.660+0000 7fcff4d2d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcff006c500 0x7fcff014eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:52.662 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.660+0000 7fcff4d2d700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff006d260 con 0x7fcff006bbe0 2022-04-03T19:49:52.662 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.660+0000 7fcff4d2d700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcff0159450 con 0x7fcff006bfb0 2022-04-03T19:49:52.662 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.660+0000 7fcff4d2d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff00c6330 con 0x7fcff006c500 2022-04-03T19:49:52.662 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.661+0000 7fcfeeffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcff006bfb0 0x7fcff01584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:52.662 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.661+0000 7fcfeeffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fcff006bfb0 0x7fcff01584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42900/0 (socket says 172.21.15.77:42900) 2022-04-03T19:49:52.663 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.661+0000 7fcfeeffd700 1 -- 172.21.15.77:0/3444396477 learned_addr learned my addr 172.21.15.77:0/3444396477 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:49:52.663 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.661+0000 7fcfef7fe700 1 --1- 172.21.15.77:0/3444396477 >> v1:172.21.15.77:6789/0 conn(0x7fcff006bbe0 0x7fcff006d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38452/0 (socket says 172.21.15.77:38452) 2022-04-03T19:49:52.663 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.661+0000 7fcfeffff700 1 --2- 172.21.15.77:0/3444396477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcff006c500 0x7fcff014eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:52.667 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.661+0000 7fcfeeffd700 1 -- 172.21.15.77:0/3444396477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcff006c500 msgr2=0x7fcff014eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:49:52.667 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.661+0000 7fcfeeffd700 1 --2- 172.21.15.77:0/3444396477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcff006c500 0x7fcff014eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:52.668 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.661+0000 7fcfeeffd700 1 -- 172.21.15.77:0/3444396477 >> v1:172.21.15.77:6789/0 conn(0x7fcff006bbe0 legacy=0x7fcff006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:52.668 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.661+0000 7fcfeeffd700 1 -- 172.21.15.77:0/3444396477 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff0158bb0 con 0x7fcff006bfb0 2022-04-03T19:49:52.668 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.661+0000 7fcfee7fc700 1 -- 172.21.15.77:0/3444396477 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826189048 0 0) 0x7fcff006d260 con 0x7fcff006bbe0 2022-04-03T19:49:52.668 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.662+0000 7fcfeeffd700 1 --2- 172.21.15.77:0/3444396477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff006bfb0 0x7fcff01584b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4006fd0 tx=0x7fcfe40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b34b30f74de6731 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:52.668 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.662+0000 7fcfee7fc700 1 -- 172.21.15.77:0/3444396477 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4004030 con 0x7fcff006bfb0 2022-04-03T19:49:52.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.662+0000 7fcfee7fc700 1 -- 172.21.15.77:0/3444396477 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfe40041d0 con 0x7fcff006bfb0 2022-04-03T19:49:52.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.662+0000 7fcfee7fc700 1 -- 172.21.15.77:0/3444396477 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4011630 con 0x7fcff006bfb0 2022-04-03T19:49:52.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.662+0000 7fcff4d2d700 1 -- 172.21.15.77:0/3444396477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff006bfb0 msgr2=0x7fcff01584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:52.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.662+0000 7fcff4d2d700 1 --2- 172.21.15.77:0/3444396477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff006bfb0 0x7fcff01584b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4006fd0 tx=0x7fcfe40090f0 comp rx=0 tx=0).stop 2022-04-03T19:49:52.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.662+0000 7fcff4d2d700 1 -- 172.21.15.77:0/3444396477 shutdown_connections 2022-04-03T19:49:52.670 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.662+0000 7fcff4d2d700 1 --2- 172.21.15.77:0/3444396477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcff006c500 0x7fcff014eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:52.670 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.662+0000 7fcff4d2d700 1 --2- 172.21.15.77:0/3444396477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff006bfb0 0x7fcff01584b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:52.670 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.662+0000 7fcff4d2d700 1 -- 172.21.15.77:0/3444396477 >> 172.21.15.77:0/3444396477 conn(0x7fcff00cb6b0 msgr2=0x7fcff00cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:49:52.670 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.663+0000 7fcff4d2d700 1 -- 172.21.15.77:0/3444396477 shutdown_connections 2022-04-03T19:49:52.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.663+0000 7fcff4d2d700 1 -- 172.21.15.77:0/3444396477 wait complete. 2022-04-03T19:49:52.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.664+0000 7fcff4d2d700 1 Processor -- start 2022-04-03T19:49:52.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.664+0000 7fcff4d2d700 1 -- start start 2022-04-03T19:49:52.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.665+0000 7fcff4d2d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcff006bbe0 0x7fcff0154910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:52.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.665+0000 7fcff4d2d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff006bfb0 0x7fcff0165200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:52.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.665+0000 7fcff4d2d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcff006c500 0x7fcff01535c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:52.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.665+0000 7fcff4d2d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff0153130 con 0x7fcff006bfb0 2022-04-03T19:49:52.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.665+0000 7fcff4d2d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff0165740 con 0x7fcff006c500 2022-04-03T19:49:52.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.665+0000 7fcff4d2d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff0165990 con 0x7fcff006bbe0 2022-04-03T19:49:52.673 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.665+0000 7fcfef7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcff006bbe0 0x7fcff0154910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:52.673 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.665+0000 7fcfef7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcff006bbe0 0x7fcff0154910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60196/0 (socket says 172.21.15.77:60196) 2022-04-03T19:49:52.673 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.665+0000 7fcfef7fe700 1 -- 172.21.15.77:0/1082774983 learned_addr learned my addr 172.21.15.77:0/1082774983 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:49:52.673 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.665+0000 7fcfeffff700 1 --2- 172.21.15.77:0/1082774983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcff006c500 0x7fcff01535c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:52.674 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.665+0000 7fcfeeffd700 1 --2- 172.21.15.77:0/1082774983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff006bfb0 0x7fcff0165200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:52.674 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.666+0000 7fcfef7fe700 1 -- 172.21.15.77:0/1082774983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcff006c500 msgr2=0x7fcff01535c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:52.674 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.666+0000 7fcfef7fe700 1 --2- 172.21.15.77:0/1082774983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcff006c500 0x7fcff01535c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:52.674 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.666+0000 7fcfef7fe700 1 -- 172.21.15.77:0/1082774983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff006bfb0 msgr2=0x7fcff0165200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:52.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.666+0000 7fcfef7fe700 1 --2- 172.21.15.77:0/1082774983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff006bfb0 0x7fcff0165200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:52.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.666+0000 7fcfef7fe700 1 -- 172.21.15.77:0/1082774983 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe4009370 con 0x7fcff006bbe0 2022-04-03T19:49:52.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.666+0000 7fcfef7fe700 1 --2- 172.21.15.77:0/1082774983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcff006bbe0 0x7fcff0154910 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcfe000c440 tx=0x7fcfe0007d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:52.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.666+0000 7fcfd77fe700 1 -- 172.21.15.77:0/1082774983 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe0007360 con 0x7fcff006bbe0 2022-04-03T19:49:52.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.666+0000 7fcfd77fe700 1 -- 172.21.15.77:0/1082774983 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfe00074c0 con 0x7fcff006bbe0 2022-04-03T19:49:52.676 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.666+0000 7fcff4d2d700 1 -- 172.21.15.77:0/1082774983 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff0165c70 con 0x7fcff006bbe0 2022-04-03T19:49:52.676 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.666+0000 7fcfd77fe700 1 -- 172.21.15.77:0/1082774983 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe0004900 con 0x7fcff006bbe0 2022-04-03T19:49:52.676 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.667+0000 7fcff4d2d700 1 -- 172.21.15.77:0/1082774983 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff0152dc0 con 0x7fcff006bbe0 2022-04-03T19:49:52.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.667+0000 7fcff4d2d700 1 -- 172.21.15.77:0/1082774983 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff0166190 con 0x7fcff006bbe0 2022-04-03T19:49:52.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.669+0000 7fcfd77fe700 1 -- 172.21.15.77:0/1082774983 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fcfe0038080 con 0x7fcff006bbe0 2022-04-03T19:49:52.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.669+0000 7fcfd77fe700 1 --2- 172.21.15.77:0/1082774983 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcfd0063d10 0x7fcfd00661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:52.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.669+0000 7fcfd77fe700 1 -- 172.21.15.77:0/1082774983 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6288+0+0 (secure 0 0 0) 0x7fcfe0080a60 con 0x7fcff006bbe0 2022-04-03T19:49:52.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.670+0000 7fcfeeffd700 1 --2- 172.21.15.77:0/1082774983 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcfd0063d10 0x7fcfd00661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:52.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.670+0000 7fcfeeffd700 1 --2- 172.21.15.77:0/1082774983 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcfd0063d10 0x7fcfd00661c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4009340 tx=0x7fcfe4016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:52.680 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.673+0000 7fcfd77fe700 1 -- 172.21.15.77:0/1082774983 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fcfe004f590 con 0x7fcff006bbe0 2022-04-03T19:49:52.906 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.906+0000 7fcff4d2d700 1 -- 172.21.15.77:0/1082774983 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcff0166190 con 0x7fcff006bbe0 2022-04-03T19:49:52.907 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.907+0000 7fcfd77fe700 1 -- 172.21.15.77:0/1082774983 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6113 (secure 0 0 0) 0x7fcff0166190 con 0x7fcff006bbe0 2022-04-03T19:49:52.907 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:49:52.908 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4850,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:48:43.168138+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4960},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4960":{"gid":4960,"name":"e","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:49:52.910 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.909+0000 7fcff4d2d700 1 -- 172.21.15.77:0/1082774983 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcfd0063d10 msgr2=0x7fcfd00661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:52.910 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.909+0000 7fcff4d2d700 1 --2- 172.21.15.77:0/1082774983 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcfd0063d10 0x7fcfd00661c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4009340 tx=0x7fcfe4016000 comp rx=0 tx=0).stop 2022-04-03T19:49:52.910 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.909+0000 7fcff4d2d700 1 -- 172.21.15.77:0/1082774983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcff006bbe0 msgr2=0x7fcff0154910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:52.911 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.909+0000 7fcff4d2d700 1 --2- 172.21.15.77:0/1082774983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcff006bbe0 0x7fcff0154910 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcfe000c440 tx=0x7fcfe0007d90 comp rx=0 tx=0).stop 2022-04-03T19:49:52.911 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.909+0000 7fcff4d2d700 1 -- 172.21.15.77:0/1082774983 shutdown_connections 2022-04-03T19:49:52.911 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.909+0000 7fcff4d2d700 1 --2- 172.21.15.77:0/1082774983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcff006c500 0x7fcff01535c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:52.911 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.909+0000 7fcff4d2d700 1 --2- 172.21.15.77:0/1082774983 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fcfd0063d10 0x7fcfd00661c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:52.912 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.909+0000 7fcff4d2d700 1 --2- 172.21.15.77:0/1082774983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcff006bbe0 0x7fcff0154910 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:52.912 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.909+0000 7fcff4d2d700 1 --2- 172.21.15.77:0/1082774983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcff006bfb0 0x7fcff0165200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:52.912 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.909+0000 7fcff4d2d700 1 -- 172.21.15.77:0/1082774983 >> 172.21.15.77:0/1082774983 conn(0x7fcff00cb6b0 msgr2=0x7fcff00cba80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:49:52.912 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.909+0000 7fcff4d2d700 1 -- 172.21.15.77:0/1082774983 shutdown_connections 2022-04-03T19:49:52.913 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:52.909+0000 7fcff4d2d700 1 -- 172.21.15.77:0/1082774983 wait complete. 2022-04-03T19:49:52.913 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 20 2022-04-03T19:49:52.934 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:49:54.935 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:49:55.141 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.140+0000 7f675c92c700 1 Processor -- start 2022-04-03T19:49:55.141 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.140+0000 7f675c92c700 1 -- start start 2022-04-03T19:49:55.142 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.142+0000 7f675c92c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f67580c3640 0x7f6758069600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:55.143 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.142+0000 7f675c92c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f67580c7af0 0x7f6758069b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:55.143 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.142+0000 7f675c92c700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67580c4b90 con 0x7f67580c3270 2022-04-03T19:49:55.143 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.142+0000 7f675c92c700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67580c3d10 con 0x7f67580c3640 2022-04-03T19:49:55.143 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.142+0000 7f675c92c700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67580c3f80 con 0x7f67580c7af0 2022-04-03T19:49:55.144 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.142+0000 7f675659c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f67580c3640 0x7f6758069600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:55.144 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.142+0000 7f675659c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f67580c3640 0x7f6758069600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42904/0 (socket says 172.21.15.77:42904) 2022-04-03T19:49:55.144 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.142+0000 7f675659c700 1 -- 172.21.15.77:0/1025343311 learned_addr learned my addr 172.21.15.77:0/1025343311 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:49:55.144 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.142+0000 7f675759e700 1 --2- 172.21.15.77:0/1025343311 >> v2:172.21.15.186:3300/0 conn(0x7f67580c7af0 0x7f6758069b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:55.148 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.143+0000 7f675659c700 1 -- 172.21.15.77:0/1025343311 >> v2:172.21.15.186:3300/0 conn(0x7f67580c7af0 msgr2=0x7f6758069b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:49:55.149 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.143+0000 7f675659c700 1 --2- 172.21.15.77:0/1025343311 >> v2:172.21.15.186:3300/0 conn(0x7f67580c7af0 0x7f6758069b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:55.149 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.143+0000 7f675659c700 1 -- 172.21.15.77:0/1025343311 >> v1:172.21.15.186:6789/0 conn(0x7f67580c3270 legacy=0x7f67580c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:55.149 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.143+0000 7f675659c700 1 -- 172.21.15.77:0/1025343311 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67580c4260 con 0x7f67580c3640 2022-04-03T19:49:55.150 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.143+0000 7f675659c700 1 --2- 172.21.15.77:0/1025343311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67580c3640 0x7f6758069600 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f67580c4b00 tx=0x7f674c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=508d807fb1a9bc94 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:55.150 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.143+0000 7f6755d9b700 1 -- 172.21.15.77:0/1025343311 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c004030 con 0x7f67580c3640 2022-04-03T19:49:55.150 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.143+0000 7f6755d9b700 1 -- 172.21.15.77:0/1025343311 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674c004190 con 0x7f67580c3640 2022-04-03T19:49:55.150 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.143+0000 7f6755d9b700 1 -- 172.21.15.77:0/1025343311 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c00f6b0 con 0x7f67580c3640 2022-04-03T19:49:55.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.144+0000 7f675c92c700 1 -- 172.21.15.77:0/1025343311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67580c3640 msgr2=0x7f6758069600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:55.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.144+0000 7f675c92c700 1 --2- 172.21.15.77:0/1025343311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67580c3640 0x7f6758069600 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f67580c4b00 tx=0x7f674c0090f0 comp rx=0 tx=0).stop 2022-04-03T19:49:55.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.144+0000 7f675c92c700 1 -- 172.21.15.77:0/1025343311 shutdown_connections 2022-04-03T19:49:55.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.144+0000 7f675c92c700 1 --2- 172.21.15.77:0/1025343311 >> v2:172.21.15.186:3300/0 conn(0x7f67580c7af0 0x7f6758069b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:55.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.144+0000 7f675c92c700 1 --2- 172.21.15.77:0/1025343311 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67580c3640 0x7f6758069600 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:55.152 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.144+0000 7f675c92c700 1 -- 172.21.15.77:0/1025343311 >> 172.21.15.77:0/1025343311 conn(0x7f6758066da0 msgr2=0x7f67580671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:49:55.152 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.144+0000 7f675c92c700 1 -- 172.21.15.77:0/1025343311 shutdown_connections 2022-04-03T19:49:55.152 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.144+0000 7f675c92c700 1 -- 172.21.15.77:0/1025343311 wait complete. 2022-04-03T19:49:55.152 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.145+0000 7f675c92c700 1 Processor -- start 2022-04-03T19:49:55.153 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.146+0000 7f675c92c700 1 -- start start 2022-04-03T19:49:55.153 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.146+0000 7f675c92c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67580c3270 0x7f67581516b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:55.153 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.146+0000 7f675c92c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67580c3640 0x7f67580b91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:55.153 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.146+0000 7f675c92c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67580c7af0 0x7f67580b96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:55.154 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.146+0000 7f675c92c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6758165a20 con 0x7f67580c3270 2022-04-03T19:49:55.154 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.146+0000 7f675c92c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67580b87a0 con 0x7f67580c7af0 2022-04-03T19:49:55.154 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.146+0000 7f675c92c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67580b8be0 con 0x7f67580c3640 2022-04-03T19:49:55.154 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.147+0000 7f6756d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67580c3270 0x7f67581516b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:55.154 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.147+0000 7f6756d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67580c3270 0x7f67581516b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42906/0 (socket says 172.21.15.77:42906) 2022-04-03T19:49:55.155 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.147+0000 7f6756d9d700 1 -- 172.21.15.77:0/671354802 learned_addr learned my addr 172.21.15.77:0/671354802 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:49:55.155 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.147+0000 7f675659c700 1 --2- 172.21.15.77:0/671354802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67580c3640 0x7f67580b91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:55.155 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.147+0000 7f675759e700 1 --2- 172.21.15.77:0/671354802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67580c7af0 0x7f67580b96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:55.155 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.147+0000 7f6756d9d700 1 -- 172.21.15.77:0/671354802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67580c3640 msgr2=0x7f67580b91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:55.155 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.147+0000 7f6756d9d700 1 --2- 172.21.15.77:0/671354802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67580c3640 0x7f67580b91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:55.156 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.147+0000 7f6756d9d700 1 -- 172.21.15.77:0/671354802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67580c7af0 msgr2=0x7f67580b96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:55.156 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.147+0000 7f6756d9d700 1 --2- 172.21.15.77:0/671354802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67580c7af0 0x7f67580b96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:55.156 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.147+0000 7f6756d9d700 1 -- 172.21.15.77:0/671354802 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c009370 con 0x7f67580c3270 2022-04-03T19:49:55.156 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.148+0000 7f6756d9d700 1 --2- 172.21.15.77:0/671354802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67580c3270 0x7f67581516b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f674800c470 tx=0x7f6748007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:55.157 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.148+0000 7f673effd700 1 -- 172.21.15.77:0/671354802 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67480050d0 con 0x7f67580c3270 2022-04-03T19:49:55.157 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.148+0000 7f673effd700 1 -- 172.21.15.77:0/671354802 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6748005230 con 0x7f67580c3270 2022-04-03T19:49:55.157 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.148+0000 7f675c92c700 1 -- 172.21.15.77:0/671354802 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6758165740 con 0x7f67580c3270 2022-04-03T19:49:55.157 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.148+0000 7f673effd700 1 -- 172.21.15.77:0/671354802 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6748003710 con 0x7f67580c3270 2022-04-03T19:49:55.158 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.148+0000 7f675c92c700 1 -- 172.21.15.77:0/671354802 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6758154670 con 0x7f67580c3270 2022-04-03T19:49:55.159 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.149+0000 7f675c92c700 1 -- 172.21.15.77:0/671354802 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f675804b050 con 0x7f67580c3270 2022-04-03T19:49:55.159 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.150+0000 7f673effd700 1 -- 172.21.15.77:0/671354802 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f6748019030 con 0x7f67580c3270 2022-04-03T19:49:55.159 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.151+0000 7f673effd700 1 --2- 172.21.15.77:0/671354802 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6738063d10 0x7f67380661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:55.160 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.151+0000 7f673effd700 1 -- 172.21.15.77:0/671354802 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6288+0+0 (secure 0 0 0) 0x7f674807f8d0 con 0x7f67580c3270 2022-04-03T19:49:55.160 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.151+0000 7f675659c700 1 --2- 172.21.15.77:0/671354802 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6738063d10 0x7f67380661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:55.160 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.152+0000 7f675659c700 1 --2- 172.21.15.77:0/671354802 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6738063d10 0x7f67380661c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f67580c4b00 tx=0x7f674c018000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:55.160 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.155+0000 7f673effd700 1 -- 172.21.15.77:0/671354802 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f674804e370 con 0x7f67580c3270 2022-04-03T19:49:55.387 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.387+0000 7f675c92c700 1 -- 172.21.15.77:0/671354802 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67580b9000 con 0x7f67580c3270 2022-04-03T19:49:55.388 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.388+0000 7f673effd700 1 -- 172.21.15.77:0/671354802 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6113 (secure 0 0 0) 0x7f6748051990 con 0x7f67580c3270 2022-04-03T19:49:55.389 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:49:55.389 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4850,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:48:43.168138+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4960},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4960":{"gid":4960,"name":"e","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:49:55.391 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.390+0000 7f675c92c700 1 -- 172.21.15.77:0/671354802 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6738063d10 msgr2=0x7f67380661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:55.391 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.390+0000 7f675c92c700 1 --2- 172.21.15.77:0/671354802 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6738063d10 0x7f67380661c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f67580c4b00 tx=0x7f674c018000 comp rx=0 tx=0).stop 2022-04-03T19:49:55.392 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.390+0000 7f675c92c700 1 -- 172.21.15.77:0/671354802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67580c3270 msgr2=0x7f67581516b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:55.392 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.390+0000 7f675c92c700 1 --2- 172.21.15.77:0/671354802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67580c3270 0x7f67581516b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f674800c470 tx=0x7f6748007e00 comp rx=0 tx=0).stop 2022-04-03T19:49:55.392 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.390+0000 7f675c92c700 1 -- 172.21.15.77:0/671354802 shutdown_connections 2022-04-03T19:49:55.392 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.390+0000 7f675c92c700 1 --2- 172.21.15.77:0/671354802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67580c7af0 0x7f67580b96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:55.392 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.390+0000 7f675c92c700 1 --2- 172.21.15.77:0/671354802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67580c3270 0x7f67581516b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:55.393 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.390+0000 7f675c92c700 1 --2- 172.21.15.77:0/671354802 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6738063d10 0x7f67380661c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:55.393 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.390+0000 7f675c92c700 1 --2- 172.21.15.77:0/671354802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67580c3640 0x7f67580b91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:55.393 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.390+0000 7f675c92c700 1 -- 172.21.15.77:0/671354802 >> 172.21.15.77:0/671354802 conn(0x7f6758066da0 msgr2=0x7f6758153740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:49:55.394 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.390+0000 7f675c92c700 1 -- 172.21.15.77:0/671354802 shutdown_connections 2022-04-03T19:49:55.394 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:55.390+0000 7f675c92c700 1 -- 172.21.15.77:0/671354802 wait complete. 2022-04-03T19:49:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 20 2022-04-03T19:49:55.415 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:49:57.418 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:49:57.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.625+0000 7f791326a700 1 Processor -- start 2022-04-03T19:49:57.626 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.625+0000 7f791326a700 1 -- start start 2022-04-03T19:49:57.628 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.626+0000 7f791326a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f790c0c2010 0x7f790c159050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:57.629 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.626+0000 7f791326a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f790c06a280 0x7f790c155910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:57.629 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.626+0000 7f791326a700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f790c0c4dc0 0x7f790c0c5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:57.629 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.626+0000 7f791326a700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f790c0c4200 con 0x7f790c06a280 2022-04-03T19:49:57.630 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.626+0000 7f791326a700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f790c159590 con 0x7f790c0c4dc0 2022-04-03T19:49:57.630 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.626+0000 7f791326a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790c155480 con 0x7f790c0c2010 2022-04-03T19:49:57.630 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.626+0000 7f7912268700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f790c0c2010 0x7f790c159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:57.630 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.626+0000 7f7912a69700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f790c0c4dc0 0x7f790c0c5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:57.630 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.626+0000 7f7911a67700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f790c06a280 0x7f790c155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:57.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.626+0000 7f7912268700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f790c0c2010 0x7f790c159050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60200/0 (socket says 172.21.15.77:60200) 2022-04-03T19:49:57.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.626+0000 7f7912268700 1 -- 172.21.15.77:0/1124950313 learned_addr learned my addr 172.21.15.77:0/1124950313 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:49:57.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.627+0000 7f7911a67700 1 -- 172.21.15.77:0/1124950313 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f790c0c2010 msgr2=0x7f790c159050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:49:57.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.627+0000 7f7911a67700 1 --2- 172.21.15.77:0/1124950313 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f790c0c2010 0x7f790c159050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:57.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.627+0000 7f7911a67700 1 -- 172.21.15.77:0/1124950313 >> v2:172.21.15.186:3300/0 conn(0x7f790c0c4dc0 msgr2=0x7f790c0c5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:49:57.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.627+0000 7f7911a67700 1 --2- 172.21.15.77:0/1124950313 >> v2:172.21.15.186:3300/0 conn(0x7f790c0c4dc0 0x7f790c0c5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:57.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.627+0000 7f7911a67700 1 -- 172.21.15.77:0/1124950313 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790c0c2d70 con 0x7f790c06a280 2022-04-03T19:49:57.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.627+0000 7f7911a67700 1 --2- 172.21.15.77:0/1124950313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f790c06a280 0x7f790c155910 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009760 tx=0x7f78fc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab9a92e833ddec46 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:57.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.627+0000 7f7911266700 1 -- 172.21.15.77:0/1124950313 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc0105b0 con 0x7f790c06a280 2022-04-03T19:49:57.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.627+0000 7f7911266700 1 -- 172.21.15.77:0/1124950313 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78fc010710 con 0x7f790c06a280 2022-04-03T19:49:57.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.627+0000 7f7911266700 1 -- 172.21.15.77:0/1124950313 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc010ad0 con 0x7f790c06a280 2022-04-03T19:49:57.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.628+0000 7f791326a700 1 -- 172.21.15.77:0/1124950313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f790c06a280 msgr2=0x7f790c155910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:57.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.628+0000 7f791326a700 1 --2- 172.21.15.77:0/1124950313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f790c06a280 0x7f790c155910 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009760 tx=0x7f78fc009060 comp rx=0 tx=0).stop 2022-04-03T19:49:57.634 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.628+0000 7f791326a700 1 -- 172.21.15.77:0/1124950313 shutdown_connections 2022-04-03T19:49:57.634 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.628+0000 7f791326a700 1 --2- 172.21.15.77:0/1124950313 >> v2:172.21.15.186:3300/0 conn(0x7f790c0c4dc0 0x7f790c0c5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:57.634 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.628+0000 7f791326a700 1 --2- 172.21.15.77:0/1124950313 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f790c0c2010 0x7f790c159050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:57.634 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.628+0000 7f791326a700 1 --2- 172.21.15.77:0/1124950313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f790c06a280 0x7f790c155910 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:57.634 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.628+0000 7f791326a700 1 -- 172.21.15.77:0/1124950313 >> 172.21.15.77:0/1124950313 conn(0x7f790c06cb30 msgr2=0x7f790c06cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:49:57.635 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.628+0000 7f791326a700 1 -- 172.21.15.77:0/1124950313 shutdown_connections 2022-04-03T19:49:57.635 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.628+0000 7f791326a700 1 -- 172.21.15.77:0/1124950313 wait complete. 2022-04-03T19:49:57.638 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.630+0000 7f791326a700 1 Processor -- start 2022-04-03T19:49:57.638 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.630+0000 7f791326a700 1 -- start start 2022-04-03T19:49:57.639 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.631+0000 7f791326a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f790c06a280 0x7f790c0bb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:57.639 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.631+0000 7f791326a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f790c0c2010 0x7f790c152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:57.639 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.631+0000 7f791326a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f790c0c4dc0 0x7f790c164ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:57.639 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.631+0000 7f791326a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790c14d160 con 0x7f790c0c2010 2022-04-03T19:49:57.639 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.631+0000 7f791326a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790c1505c0 con 0x7f790c06a280 2022-04-03T19:49:57.640 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.631+0000 7f791326a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790c14ce80 con 0x7f790c0c4dc0 2022-04-03T19:49:57.640 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.631+0000 7f7912268700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f790c06a280 0x7f790c0bb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:57.640 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.631+0000 7f7912268700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f790c06a280 0x7f790c0bb750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38098/0 (socket says 172.21.15.77:38098) 2022-04-03T19:49:57.640 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.631+0000 7f7912268700 1 -- 172.21.15.77:0/1336863267 learned_addr learned my addr 172.21.15.77:0/1336863267 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:49:57.641 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.631+0000 7f7911a67700 1 --2- 172.21.15.77:0/1336863267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f790c0c2010 0x7f790c152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:57.641 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.631+0000 7f7912a69700 1 --2- 172.21.15.77:0/1336863267 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f790c0c4dc0 0x7f790c164ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:57.641 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.631+0000 7f7912268700 1 -- 172.21.15.77:0/1336863267 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f790c0c4dc0 msgr2=0x7f790c164ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:57.641 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.631+0000 7f7912268700 1 --2- 172.21.15.77:0/1336863267 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f790c0c4dc0 0x7f790c164ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:57.641 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.631+0000 7f7912268700 1 -- 172.21.15.77:0/1336863267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f790c0c2010 msgr2=0x7f790c152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:57.642 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.631+0000 7f7912268700 1 --2- 172.21.15.77:0/1336863267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f790c0c2010 0x7f790c152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:57.642 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.631+0000 7f7912268700 1 -- 172.21.15.77:0/1336863267 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc0092a0 con 0x7f790c06a280 2022-04-03T19:49:57.642 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.632+0000 7f7912268700 1 --2- 172.21.15.77:0/1336863267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f790c06a280 0x7f790c0bb750 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f790800e930 tx=0x7f79080090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:57.642 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.632+0000 7f78fa7fc700 1 -- 172.21.15.77:0/1336863267 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7908013680 con 0x7f790c06a280 2022-04-03T19:49:57.642 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.632+0000 7f78fa7fc700 1 -- 172.21.15.77:0/1336863267 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79080137e0 con 0x7f790c06a280 2022-04-03T19:49:57.643 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.632+0000 7f78fa7fc700 1 -- 172.21.15.77:0/1336863267 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7908013b80 con 0x7f790c06a280 2022-04-03T19:49:57.643 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.632+0000 7f791326a700 1 -- 172.21.15.77:0/1336863267 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f790c14fc20 con 0x7f790c06a280 2022-04-03T19:49:57.643 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.632+0000 7f791326a700 1 -- 172.21.15.77:0/1336863267 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f790c1501e0 con 0x7f790c06a280 2022-04-03T19:49:57.643 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.633+0000 7f791326a700 1 -- 172.21.15.77:0/1336863267 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f790c1547a0 con 0x7f790c06a280 2022-04-03T19:49:57.644 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.634+0000 7f78fa7fc700 1 -- 172.21.15.77:0/1336863267 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f7908023070 con 0x7f790c06a280 2022-04-03T19:49:57.644 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.635+0000 7f78fa7fc700 1 --2- 172.21.15.77:0/1336863267 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f78ec063ce0 0x7f78ec066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:49:57.644 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.635+0000 7f78fa7fc700 1 -- 172.21.15.77:0/1336863267 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6288+0+0 (secure 0 0 0) 0x7f7908080cd0 con 0x7f790c06a280 2022-04-03T19:49:57.644 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.635+0000 7f7911a67700 1 --2- 172.21.15.77:0/1336863267 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f78ec063ce0 0x7f78ec066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:49:57.644 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.636+0000 7f7911a67700 1 --2- 172.21.15.77:0/1336863267 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f78ec063ce0 0x7f78ec066190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009760 tx=0x7f78fc016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:49:57.645 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.639+0000 7f78fa7fc700 1 -- 172.21.15.77:0/1336863267 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f790804f700 con 0x7f790c06a280 2022-04-03T19:49:57.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.871+0000 7f791326a700 1 -- 172.21.15.77:0/1336863267 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f790c04b050 con 0x7f790c06a280 2022-04-03T19:49:57.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.872+0000 7f78fa7fc700 1 -- 172.21.15.77:0/1336863267 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+6113 (secure 0 0 0) 0x7f7908052d20 con 0x7f790c06a280 2022-04-03T19:49:57.873 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:49:57.873 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4850,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:48:43.168138+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4960},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4960":{"gid":4960,"name":"e","rank":0,"incarnation":15,"state":"up:active","state_seq":24,"addr":"172.21.15.77:6837/2256447697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2256447697},{"type":"v1","addr":"172.21.15.77:6837","nonce":2256447697}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:49:57.875 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.874+0000 7f791326a700 1 -- 172.21.15.77:0/1336863267 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f78ec063ce0 msgr2=0x7f78ec066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:57.876 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.874+0000 7f791326a700 1 --2- 172.21.15.77:0/1336863267 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f78ec063ce0 0x7f78ec066190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009760 tx=0x7f78fc016000 comp rx=0 tx=0).stop 2022-04-03T19:49:57.876 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.874+0000 7f791326a700 1 -- 172.21.15.77:0/1336863267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f790c06a280 msgr2=0x7f790c0bb750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:49:57.876 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.874+0000 7f791326a700 1 --2- 172.21.15.77:0/1336863267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f790c06a280 0x7f790c0bb750 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f790800e930 tx=0x7f79080090f0 comp rx=0 tx=0).stop 2022-04-03T19:49:57.876 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.875+0000 7f791326a700 1 -- 172.21.15.77:0/1336863267 shutdown_connections 2022-04-03T19:49:57.877 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.875+0000 7f791326a700 1 --2- 172.21.15.77:0/1336863267 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f78ec063ce0 0x7f78ec066190 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:57.877 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.875+0000 7f791326a700 1 --2- 172.21.15.77:0/1336863267 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f790c0c4dc0 0x7f790c164ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:57.877 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.875+0000 7f791326a700 1 --2- 172.21.15.77:0/1336863267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f790c06a280 0x7f790c0bb750 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:57.877 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.875+0000 7f791326a700 1 --2- 172.21.15.77:0/1336863267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f790c0c2010 0x7f790c152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:49:57.877 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.875+0000 7f791326a700 1 -- 172.21.15.77:0/1336863267 >> 172.21.15.77:0/1336863267 conn(0x7f790c06cb30 msgr2=0x7f790c154090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:49:57.878 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.875+0000 7f791326a700 1 -- 172.21.15.77:0/1336863267 shutdown_connections 2022-04-03T19:49:57.878 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:49:57.875+0000 7f791326a700 1 -- 172.21.15.77:0/1336863267 wait complete. 2022-04-03T19:49:57.879 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 20 2022-04-03T19:49:57.899 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:49:59.901 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:50:00.107 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.106+0000 7f0b0887d700 1 Processor -- start 2022-04-03T19:50:00.107 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.106+0000 7f0b0887d700 1 -- start start 2022-04-03T19:50:00.109 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.107+0000 7f0b0887d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0b040c1170 0x7f0b041527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:00.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.107+0000 7f0b0887d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0b04155370 0x7f0b04069110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:00.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.107+0000 7f0b0887d700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b040c3710 con 0x7f0b040c3b10 2022-04-03T19:50:00.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.108+0000 7f0b0887d700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b04152330 con 0x7f0b040c1170 2022-04-03T19:50:00.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.108+0000 7f0b0887d700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b040c1540 con 0x7f0b04155370 2022-04-03T19:50:00.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.108+0000 7f0b037fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0b040c1170 0x7f0b041527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:00.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.108+0000 7f0b02ffd700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0b04155370 0x7f0b04069110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:00.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.108+0000 7f0b03fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0b040c3b10 0x7f0b040c71f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38454/0 (socket says 172.21.15.77:38454) 2022-04-03T19:50:00.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.108+0000 7f0b03fff700 1 -- 172.21.15.77:0/2259003506 learned_addr learned my addr 172.21.15.77:0/2259003506 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:50:00.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.108+0000 7f0b02ffd700 1 -- 172.21.15.77:0/2259003506 >> v2:172.21.15.77:3300/0 conn(0x7f0b040c1170 msgr2=0x7f0b041527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:50:00.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.108+0000 7f0b02ffd700 1 --2- 172.21.15.77:0/2259003506 >> v2:172.21.15.77:3300/0 conn(0x7f0b040c1170 0x7f0b041527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:00.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.108+0000 7f0b02ffd700 1 -- 172.21.15.77:0/2259003506 >> v1:172.21.15.77:6789/0 conn(0x7f0b040c3b10 legacy=0x7f0b040c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:00.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.108+0000 7f0b027fc700 1 -- 172.21.15.77:0/2259003506 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417571033 0 0) 0x7f0b040c3710 con 0x7f0b040c3b10 2022-04-03T19:50:00.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.109+0000 7f0b02ffd700 1 -- 172.21.15.77:0/2259003506 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b040c4270 con 0x7f0b04155370 2022-04-03T19:50:00.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.109+0000 7f0b037fe700 1 --2- 172.21.15.77:0/2259003506 >> v2:172.21.15.77:3300/0 conn(0x7f0b040c1170 0x7f0b041527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:50:00.113 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.109+0000 7f0b02ffd700 1 --2- 172.21.15.77:0/2259003506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b04155370 0x7f0b04069110 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0b040badf0 tx=0x7f0af8009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=638395e599bf7329 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:00.113 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.109+0000 7f0b027fc700 1 -- 172.21.15.77:0/2259003506 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af8011630 con 0x7f0b04155370 2022-04-03T19:50:00.113 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.109+0000 7f0b027fc700 1 -- 172.21.15.77:0/2259003506 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af8011790 con 0x7f0b04155370 2022-04-03T19:50:00.113 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.109+0000 7f0b027fc700 1 -- 172.21.15.77:0/2259003506 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af8011b30 con 0x7f0b04155370 2022-04-03T19:50:00.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.110+0000 7f0b0887d700 1 -- 172.21.15.77:0/2259003506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b04155370 msgr2=0x7f0b04069110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:00.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.110+0000 7f0b0887d700 1 --2- 172.21.15.77:0/2259003506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b04155370 0x7f0b04069110 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0b040badf0 tx=0x7f0af8009060 comp rx=0 tx=0).stop 2022-04-03T19:50:00.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.110+0000 7f0b0887d700 1 -- 172.21.15.77:0/2259003506 shutdown_connections 2022-04-03T19:50:00.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.110+0000 7f0b0887d700 1 --2- 172.21.15.77:0/2259003506 >> v2:172.21.15.77:3300/0 conn(0x7f0b040c1170 0x7f0b041527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:00.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.110+0000 7f0b0887d700 1 --2- 172.21.15.77:0/2259003506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b04155370 0x7f0b04069110 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:00.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.110+0000 7f0b0887d700 1 -- 172.21.15.77:0/2259003506 >> 172.21.15.77:0/2259003506 conn(0x7f0b04066da0 msgr2=0x7f0b040671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:00.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.110+0000 7f0b0887d700 1 -- 172.21.15.77:0/2259003506 shutdown_connections 2022-04-03T19:50:00.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.110+0000 7f0b0887d700 1 -- 172.21.15.77:0/2259003506 wait complete. 2022-04-03T19:50:00.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.111+0000 7f0b0887d700 1 Processor -- start 2022-04-03T19:50:00.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.111+0000 7f0b0887d700 1 -- start start 2022-04-03T19:50:00.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.112+0000 7f0b0887d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b040c1170 0x7f0b0414c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:00.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.112+0000 7f0b0887d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b040c3b10 0x7f0b040b9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:00.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.112+0000 7f0b0887d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b04155370 0x7f0b04147040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:00.120 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.112+0000 7f0b0887d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b04150960 con 0x7f0b04155370 2022-04-03T19:50:00.120 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.112+0000 7f0b0887d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b04169010 con 0x7f0b040c3b10 2022-04-03T19:50:00.120 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.112+0000 7f0b0887d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b040692d0 con 0x7f0b040c1170 2022-04-03T19:50:00.120 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.113+0000 7f0b03fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b04155370 0x7f0b04147040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:00.120 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.113+0000 7f0b02ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b040c3b10 0x7f0b040b9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:00.121 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.113+0000 7f0b03fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b04155370 0x7f0b04147040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42914/0 (socket says 172.21.15.77:42914) 2022-04-03T19:50:00.121 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.113+0000 7f0b03fff700 1 -- 172.21.15.77:0/1410381033 learned_addr learned my addr 172.21.15.77:0/1410381033 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:50:00.121 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.113+0000 7f0b037fe700 1 --2- 172.21.15.77:0/1410381033 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b040c1170 0x7f0b0414c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:00.121 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.113+0000 7f0b03fff700 1 -- 172.21.15.77:0/1410381033 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b040c1170 msgr2=0x7f0b0414c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:00.122 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.113+0000 7f0b03fff700 1 --2- 172.21.15.77:0/1410381033 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b040c1170 0x7f0b0414c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:00.122 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.113+0000 7f0b03fff700 1 -- 172.21.15.77:0/1410381033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b040c3b10 msgr2=0x7f0b040b9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:00.122 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.113+0000 7f0b03fff700 1 --2- 172.21.15.77:0/1410381033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b040c3b10 0x7f0b040b9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:00.122 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.113+0000 7f0b03fff700 1 -- 172.21.15.77:0/1410381033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af40092a0 con 0x7f0b04155370 2022-04-03T19:50:00.122 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.113+0000 7f0b03fff700 1 --2- 172.21.15.77:0/1410381033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b04155370 0x7f0b04147040 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0afc00c790 tx=0x7f0afc007f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:00.123 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.114+0000 7f0aeb7fe700 1 -- 172.21.15.77:0/1410381033 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc004c40 con 0x7f0b04155370 2022-04-03T19:50:00.123 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.114+0000 7f0aeb7fe700 1 -- 172.21.15.77:0/1410381033 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0afc004da0 con 0x7f0b04155370 2022-04-03T19:50:00.123 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.114+0000 7f0aeb7fe700 1 -- 172.21.15.77:0/1410381033 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc0073a0 con 0x7f0b04155370 2022-04-03T19:50:00.123 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.114+0000 7f0b0887d700 1 -- 172.21.15.77:0/1410381033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0af80092a0 con 0x7f0b04155370 2022-04-03T19:50:00.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.114+0000 7f0b0887d700 1 -- 172.21.15.77:0/1410381033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b0414ffd0 con 0x7f0b04155370 2022-04-03T19:50:00.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.115+0000 7f0b0887d700 1 -- 172.21.15.77:0/1410381033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b04066a80 con 0x7f0b04155370 2022-04-03T19:50:00.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.116+0000 7f0aeb7fe700 1 -- 172.21.15.77:0/1410381033 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f0afc01a030 con 0x7f0b04155370 2022-04-03T19:50:00.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.116+0000 7f0aeb7fe700 1 --2- 172.21.15.77:0/1410381033 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0ae4063d10 0x7f0ae40661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:00.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.117+0000 7f0aeb7fe700 1 -- 172.21.15.77:0/1410381033 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7f0afc07fe90 con 0x7f0b04155370 2022-04-03T19:50:00.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.117+0000 7f0b037fe700 1 --2- 172.21.15.77:0/1410381033 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0ae4063d10 0x7f0ae40661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:00.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.118+0000 7f0b037fe700 1 --2- 172.21.15.77:0/1410381033 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0ae4063d10 0x7f0ae40661c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0af4009dc0 tx=0x7f0af4009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:00.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.120+0000 7f0aeb7fe700 1 -- 172.21.15.77:0/1410381033 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f0afc04e9b0 con 0x7f0b04155370 2022-04-03T19:50:00.360 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.359+0000 7f0b0887d700 1 -- 172.21.15.77:0/1410381033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b04169150 con 0x7f0b04155370 2022-04-03T19:50:00.360 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.360+0000 7f0aeb7fe700 1 -- 172.21.15.77:0/1410381033 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5355 (secure 0 0 0) 0x7f0afc018030 con 0x7f0b04155370 2022-04-03T19:50:00.361 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:50:00.361 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:49:59.156875+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:50:00.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.362+0000 7f0b0887d700 1 -- 172.21.15.77:0/1410381033 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0ae4063d10 msgr2=0x7f0ae40661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:00.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.362+0000 7f0b0887d700 1 --2- 172.21.15.77:0/1410381033 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0ae4063d10 0x7f0ae40661c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0af4009dc0 tx=0x7f0af4009060 comp rx=0 tx=0).stop 2022-04-03T19:50:00.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.362+0000 7f0b0887d700 1 -- 172.21.15.77:0/1410381033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b04155370 msgr2=0x7f0b04147040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:00.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.362+0000 7f0b0887d700 1 --2- 172.21.15.77:0/1410381033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b04155370 0x7f0b04147040 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0afc00c790 tx=0x7f0afc007f10 comp rx=0 tx=0).stop 2022-04-03T19:50:00.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.362+0000 7f0b0887d700 1 -- 172.21.15.77:0/1410381033 shutdown_connections 2022-04-03T19:50:00.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.362+0000 7f0b0887d700 1 --2- 172.21.15.77:0/1410381033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b04155370 0x7f0b04147040 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:00.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.362+0000 7f0b0887d700 1 --2- 172.21.15.77:0/1410381033 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0ae4063d10 0x7f0ae40661c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:00.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.362+0000 7f0b0887d700 1 --2- 172.21.15.77:0/1410381033 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b040c1170 0x7f0b0414c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:00.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.362+0000 7f0b0887d700 1 --2- 172.21.15.77:0/1410381033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b040c3b10 0x7f0b040b9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:00.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.362+0000 7f0b0887d700 1 -- 172.21.15.77:0/1410381033 >> 172.21.15.77:0/1410381033 conn(0x7f0b04066da0 msgr2=0x7f0b040ba250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:00.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.362+0000 7f0b0887d700 1 -- 172.21.15.77:0/1410381033 shutdown_connections 2022-04-03T19:50:00.366 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.362+0000 7f0b0887d700 1 -- 172.21.15.77:0/1410381033 wait complete. 2022-04-03T19:50:00.366 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2022-04-03T19:50:00.387 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2022-04-03T19:50:00.388 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-04-03T19:50:00.388 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:50:00.588 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.587+0000 7fe1f9aea700 1 Processor -- start 2022-04-03T19:50:00.588 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.587+0000 7fe1f9aea700 1 -- start start 2022-04-03T19:50:00.591 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.588+0000 7fe1f9aea700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe1f406bfb0 0x7fe1f41584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:00.591 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.588+0000 7fe1f9aea700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1f406c500 0x7fe1f414eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:00.591 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.588+0000 7fe1f9aea700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f406d260 con 0x7fe1f406bbe0 2022-04-03T19:50:00.592 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.588+0000 7fe1f9aea700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f4159450 con 0x7fe1f406bfb0 2022-04-03T19:50:00.592 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.588+0000 7fe1f9aea700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f40c6330 con 0x7fe1f406c500 2022-04-03T19:50:00.592 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.589+0000 7fe1f3fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe1f406bfb0 0x7fe1f41584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:00.593 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.589+0000 7fe1f3fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe1f406bfb0 0x7fe1f41584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42916/0 (socket says 172.21.15.77:42916) 2022-04-03T19:50:00.593 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.589+0000 7fe1f3fff700 1 -- 172.21.15.77:0/94130422 learned_addr learned my addr 172.21.15.77:0/94130422 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:50:00.593 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.589+0000 7fe1f8ae8700 1 --1- 172.21.15.77:0/94130422 >> v1:172.21.15.186:6789/0 conn(0x7fe1f406bbe0 0x7fe1f406d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44254/0 (socket says 172.21.15.77:44254) 2022-04-03T19:50:00.593 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.589+0000 7fe1f92e9700 1 --2- 172.21.15.77:0/94130422 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1f406c500 0x7fe1f414eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:00.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.589+0000 7fe1f3fff700 1 -- 172.21.15.77:0/94130422 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1f406c500 msgr2=0x7fe1f414eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:50:00.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.589+0000 7fe1f3fff700 1 --2- 172.21.15.77:0/94130422 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1f406c500 0x7fe1f414eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:00.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.589+0000 7fe1f3fff700 1 -- 172.21.15.77:0/94130422 >> v1:172.21.15.186:6789/0 conn(0x7fe1f406bbe0 legacy=0x7fe1f406d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:00.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.589+0000 7fe1f3fff700 1 -- 172.21.15.77:0/94130422 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f4158bb0 con 0x7fe1f406bfb0 2022-04-03T19:50:00.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.590+0000 7fe1f3fff700 1 --2- 172.21.15.77:0/94130422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1f406bfb0 0x7fe1f41584b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0006fd0 tx=0x7fe1e00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4fb37cc1e2eefec server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:00.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.590+0000 7fe1f37fe700 1 -- 172.21.15.77:0/94130422 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e0004030 con 0x7fe1f406bfb0 2022-04-03T19:50:00.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.590+0000 7fe1f37fe700 1 -- 172.21.15.77:0/94130422 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e00041d0 con 0x7fe1f406bfb0 2022-04-03T19:50:00.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.590+0000 7fe1f37fe700 1 -- 172.21.15.77:0/94130422 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e0011630 con 0x7fe1f406bfb0 2022-04-03T19:50:00.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.590+0000 7fe1f9aea700 1 -- 172.21.15.77:0/94130422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1f406bfb0 msgr2=0x7fe1f41584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:00.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.590+0000 7fe1f9aea700 1 --2- 172.21.15.77:0/94130422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1f406bfb0 0x7fe1f41584b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0006fd0 tx=0x7fe1e00090f0 comp rx=0 tx=0).stop 2022-04-03T19:50:00.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.590+0000 7fe1f9aea700 1 -- 172.21.15.77:0/94130422 shutdown_connections 2022-04-03T19:50:00.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.590+0000 7fe1f9aea700 1 --2- 172.21.15.77:0/94130422 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1f406c500 0x7fe1f414eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:00.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.590+0000 7fe1f9aea700 1 --2- 172.21.15.77:0/94130422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1f406bfb0 0x7fe1f41584b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:00.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.590+0000 7fe1f9aea700 1 -- 172.21.15.77:0/94130422 >> 172.21.15.77:0/94130422 conn(0x7fe1f40cb6b0 msgr2=0x7fe1f40cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:00.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.591+0000 7fe1f9aea700 1 -- 172.21.15.77:0/94130422 shutdown_connections 2022-04-03T19:50:00.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.591+0000 7fe1f9aea700 1 -- 172.21.15.77:0/94130422 wait complete. 2022-04-03T19:50:00.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.592+0000 7fe1f9aea700 1 Processor -- start 2022-04-03T19:50:00.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.592+0000 7fe1f9aea700 1 -- start start 2022-04-03T19:50:00.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.593+0000 7fe1f9aea700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1f406bbe0 0x7fe1f41659c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:00.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.593+0000 7fe1f9aea700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1f406bfb0 0x7fe1f4158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:00.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.593+0000 7fe1f9aea700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1f406c500 0x7fe1f4151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:00.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.593+0000 7fe1f9aea700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f40c0cf0 con 0x7fe1f406bfb0 2022-04-03T19:50:00.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.593+0000 7fe1f9aea700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f4165530 con 0x7fe1f406bbe0 2022-04-03T19:50:00.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.593+0000 7fe1f9aea700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f40bf420 con 0x7fe1f406c500 2022-04-03T19:50:00.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.593+0000 7fe1f3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1f406bfb0 0x7fe1f4158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:00.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.593+0000 7fe1f92e9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1f406c500 0x7fe1f4151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:00.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.593+0000 7fe1f8ae8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1f406bbe0 0x7fe1f41659c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:00.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.594+0000 7fe1f92e9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1f406c500 0x7fe1f4151a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60208/0 (socket says 172.21.15.77:60208) 2022-04-03T19:50:00.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.594+0000 7fe1f92e9700 1 -- 172.21.15.77:0/713939099 learned_addr learned my addr 172.21.15.77:0/713939099 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:50:00.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.594+0000 7fe1f92e9700 1 -- 172.21.15.77:0/713939099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1f406bbe0 msgr2=0x7fe1f41659c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:00.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.594+0000 7fe1f92e9700 1 --2- 172.21.15.77:0/713939099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1f406bbe0 0x7fe1f41659c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:00.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.594+0000 7fe1f92e9700 1 -- 172.21.15.77:0/713939099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1f406bfb0 msgr2=0x7fe1f4158080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:00.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.594+0000 7fe1f92e9700 1 --2- 172.21.15.77:0/713939099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1f406bfb0 0x7fe1f4158080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:00.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.594+0000 7fe1f92e9700 1 -- 172.21.15.77:0/713939099 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e0009370 con 0x7fe1f406c500 2022-04-03T19:50:00.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.594+0000 7fe1f3fff700 1 --2- 172.21.15.77:0/713939099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1f406bfb0 0x7fe1f4158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:50:00.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.594+0000 7fe1f92e9700 1 --2- 172.21.15.77:0/713939099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1f406c500 0x7fe1f4151a40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009ec0 tx=0x7fe1ec00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:00.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.595+0000 7fe1f0ff9700 1 -- 172.21.15.77:0/713939099 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec00cb70 con 0x7fe1f406c500 2022-04-03T19:50:00.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.595+0000 7fe1f0ff9700 1 -- 172.21.15.77:0/713939099 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ec00f040 con 0x7fe1f406c500 2022-04-03T19:50:00.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.595+0000 7fe1f0ff9700 1 -- 172.21.15.77:0/713939099 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec014480 con 0x7fe1f406c500 2022-04-03T19:50:00.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.595+0000 7fe1f9aea700 1 -- 172.21.15.77:0/713939099 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f40bf6e0 con 0x7fe1f406c500 2022-04-03T19:50:00.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.595+0000 7fe1f9aea700 1 -- 172.21.15.77:0/713939099 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f40bfca0 con 0x7fe1f406c500 2022-04-03T19:50:00.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.596+0000 7fe1f9aea700 1 -- 172.21.15.77:0/713939099 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f41656e0 con 0x7fe1f406c500 2022-04-03T19:50:00.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.597+0000 7fe1f0ff9700 1 -- 172.21.15.77:0/713939099 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fe1ec014830 con 0x7fe1f406c500 2022-04-03T19:50:00.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.597+0000 7fe1f0ff9700 1 --2- 172.21.15.77:0/713939099 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe1dc063d10 0x7fe1dc0661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:00.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.597+0000 7fe1f0ff9700 1 -- 172.21.15.77:0/713939099 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7fe1ec081510 con 0x7fe1f406c500 2022-04-03T19:50:00.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.598+0000 7fe1f8ae8700 1 --2- 172.21.15.77:0/713939099 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe1dc063d10 0x7fe1dc0661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:00.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.598+0000 7fe1f8ae8700 1 --2- 172.21.15.77:0/713939099 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe1dc063d10 0x7fe1dc0661c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8007fd0 tx=0x7fe1e8007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:00.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.602+0000 7fe1f0ff9700 1 -- 172.21.15.77:0/713939099 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fe1f41656e0 con 0x7fe1f406c500 2022-04-03T19:50:00.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.835+0000 7fe1f9aea700 1 -- 172.21.15.77:0/713939099 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1f41656e0 con 0x7fe1f406c500 2022-04-03T19:50:00.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.836+0000 7fe1f0ff9700 1 -- 172.21.15.77:0/713939099 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5355 (secure 0 0 0) 0x7fe1f41656e0 con 0x7fe1f406c500 2022-04-03T19:50:00.836 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:50:00.837 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:49:59.156875+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:50:00.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.838+0000 7fe1f9aea700 1 -- 172.21.15.77:0/713939099 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe1dc063d10 msgr2=0x7fe1dc0661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:00.840 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.838+0000 7fe1f9aea700 1 --2- 172.21.15.77:0/713939099 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe1dc063d10 0x7fe1dc0661c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8007fd0 tx=0x7fe1e8007d30 comp rx=0 tx=0).stop 2022-04-03T19:50:00.840 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.838+0000 7fe1f9aea700 1 -- 172.21.15.77:0/713939099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1f406c500 msgr2=0x7fe1f4151a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:00.840 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.838+0000 7fe1f9aea700 1 --2- 172.21.15.77:0/713939099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1f406c500 0x7fe1f4151a40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009ec0 tx=0x7fe1ec00b040 comp rx=0 tx=0).stop 2022-04-03T19:50:00.840 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.838+0000 7fe1f9aea700 1 -- 172.21.15.77:0/713939099 shutdown_connections 2022-04-03T19:50:00.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.838+0000 7fe1f9aea700 1 --2- 172.21.15.77:0/713939099 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe1dc063d10 0x7fe1dc0661c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:00.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.838+0000 7fe1f9aea700 1 --2- 172.21.15.77:0/713939099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1f406c500 0x7fe1f4151a40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:00.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.838+0000 7fe1f9aea700 1 --2- 172.21.15.77:0/713939099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1f406bbe0 0x7fe1f41659c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:00.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.838+0000 7fe1f9aea700 1 --2- 172.21.15.77:0/713939099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1f406bfb0 0x7fe1f4158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:00.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.838+0000 7fe1f9aea700 1 -- 172.21.15.77:0/713939099 >> 172.21.15.77:0/713939099 conn(0x7fe1f40cb6b0 msgr2=0x7fe1f40cba80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:00.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.839+0000 7fe1f9aea700 1 -- 172.21.15.77:0/713939099 shutdown_connections 2022-04-03T19:50:00.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:00.839+0000 7fe1f9aea700 1 -- 172.21.15.77:0/713939099 wait complete. 2022-04-03T19:50:00.843 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2022-04-03T19:50:00.864 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 21, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-04-03T19:44:21.275716+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 21, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4850": { "addr": "172.21.15.186:6837/3568325507", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 3568325507, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 3568325507, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4850, "incarnation": 21, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-04-03T19:49:59.156875+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4850 } } } ], "standbys": [ { "addr": "172.21.15.77:6834/2116223509", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6832", "nonce": 2116223509, "type": "v2" }, { "addr": "172.21.15.77:6834", "nonce": 2116223509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4245, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6836/2374276747", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2374276747, "type": "v2" }, { "addr": "172.21.15.186:6836", "nonce": 2374276747, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4259, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/610990853", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6838", "nonce": 610990853, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 610990853, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4265, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.77:6835/170053412", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6833", "nonce": 170053412, "type": "v2" }, { "addr": "172.21.15.77:6835", "nonce": 170053412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4271, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-04-03T19:50:02.865 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:50:03.072 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.072+0000 7ff483fff700 1 Processor -- start 2022-04-03T19:50:03.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.072+0000 7ff483fff700 1 -- start start 2022-04-03T19:50:03.076 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.073+0000 7ff483fff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff48406bbe0 0x7ff4841584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:03.076 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.073+0000 7ff483fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff48406bfb0 0x7ff48406d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:03.076 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.073+0000 7ff483fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff48406d260 con 0x7ff48406c500 2022-04-03T19:50:03.076 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.073+0000 7ff483fff700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff484159450 con 0x7ff48406bbe0 2022-04-03T19:50:03.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.073+0000 7ff483fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4840c6330 con 0x7ff48406bfb0 2022-04-03T19:50:03.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.073+0000 7ff4837fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff48406c500 0x7ff48414eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38456/0 (socket says 172.21.15.77:38456) 2022-04-03T19:50:03.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.073+0000 7ff4837fe700 1 -- 172.21.15.77:0/3767034187 learned_addr learned my addr 172.21.15.77:0/3767034187 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:50:03.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.073+0000 7ff482ffd700 1 --2- 172.21.15.77:0/3767034187 >> v2:172.21.15.186:3300/0 conn(0x7ff48406bbe0 0x7ff4841584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:03.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.073+0000 7ff47bfff700 1 --2- 172.21.15.77:0/3767034187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff48406bfb0 0x7ff48406d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:03.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.074+0000 7ff4827fc700 1 -- 172.21.15.77:0/3767034187 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878655173 0 0) 0x7ff48406d260 con 0x7ff48406c500 2022-04-03T19:50:03.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.074+0000 7ff4827fc700 1 -- 172.21.15.77:0/3767034187 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4640033f0 con 0x7ff48406c500 2022-04-03T19:50:03.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.074+0000 7ff47bfff700 1 -- 172.21.15.77:0/3767034187 >> v2:172.21.15.186:3300/0 conn(0x7ff48406bbe0 msgr2=0x7ff4841584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:50:03.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.074+0000 7ff47bfff700 1 --2- 172.21.15.77:0/3767034187 >> v2:172.21.15.186:3300/0 conn(0x7ff48406bbe0 0x7ff4841584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:03.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.074+0000 7ff47bfff700 1 -- 172.21.15.77:0/3767034187 >> v1:172.21.15.77:6789/0 conn(0x7ff48406c500 legacy=0x7ff48414eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:03.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.074+0000 7ff47bfff700 1 -- 172.21.15.77:0/3767034187 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff484158bb0 con 0x7ff48406bfb0 2022-04-03T19:50:03.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.074+0000 7ff482ffd700 1 --2- 172.21.15.77:0/3767034187 >> v2:172.21.15.186:3300/0 conn(0x7ff48406bbe0 0x7ff4841584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:50:03.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.074+0000 7ff47bfff700 1 --2- 172.21.15.77:0/3767034187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff48406bfb0 0x7ff48406d6f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009760 tx=0x7ff46c009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83a6018e00ca6fa1 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:03.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.075+0000 7ff4827fc700 1 -- 172.21.15.77:0/3767034187 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c0105b0 con 0x7ff48406bfb0 2022-04-03T19:50:03.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.075+0000 7ff4827fc700 1 -- 172.21.15.77:0/3767034187 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46c010710 con 0x7ff48406bfb0 2022-04-03T19:50:03.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.075+0000 7ff4827fc700 1 -- 172.21.15.77:0/3767034187 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c010ab0 con 0x7ff48406bfb0 2022-04-03T19:50:03.081 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.075+0000 7ff483fff700 1 -- 172.21.15.77:0/3767034187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff48406bfb0 msgr2=0x7ff48406d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:03.081 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.075+0000 7ff483fff700 1 --2- 172.21.15.77:0/3767034187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff48406bfb0 0x7ff48406d6f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009760 tx=0x7ff46c009060 comp rx=0 tx=0).stop 2022-04-03T19:50:03.081 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.075+0000 7ff483fff700 1 -- 172.21.15.77:0/3767034187 shutdown_connections 2022-04-03T19:50:03.081 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.075+0000 7ff483fff700 1 --2- 172.21.15.77:0/3767034187 >> v2:172.21.15.186:3300/0 conn(0x7ff48406bbe0 0x7ff4841584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:03.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.075+0000 7ff483fff700 1 --2- 172.21.15.77:0/3767034187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff48406bfb0 0x7ff48406d6f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:03.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.075+0000 7ff483fff700 1 -- 172.21.15.77:0/3767034187 >> 172.21.15.77:0/3767034187 conn(0x7ff4840cb6b0 msgr2=0x7ff4840cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:03.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.075+0000 7ff483fff700 1 -- 172.21.15.77:0/3767034187 shutdown_connections 2022-04-03T19:50:03.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.075+0000 7ff483fff700 1 -- 172.21.15.77:0/3767034187 wait complete. 2022-04-03T19:50:03.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.077+0000 7ff483fff700 1 Processor -- start 2022-04-03T19:50:03.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.077+0000 7ff483fff700 1 -- start start 2022-04-03T19:50:03.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.078+0000 7ff483fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff48406bbe0 0x7ff484147d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:03.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.078+0000 7ff483fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff48406bfb0 0x7ff484157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:03.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.078+0000 7ff483fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff48406c500 0x7ff4841570f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:03.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.078+0000 7ff483fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4840c6b70 con 0x7ff48406bbe0 2022-04-03T19:50:03.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.078+0000 7ff483fff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff484165e60 con 0x7ff48406c500 2022-04-03T19:50:03.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.078+0000 7ff483fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4841482a0 con 0x7ff48406bfb0 2022-04-03T19:50:03.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.078+0000 7ff482ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff48406bbe0 0x7ff484147d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:03.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.078+0000 7ff482ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff48406bbe0 0x7ff484147d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42920/0 (socket says 172.21.15.77:42920) 2022-04-03T19:50:03.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.078+0000 7ff482ffd700 1 -- 172.21.15.77:0/2103787487 learned_addr learned my addr 172.21.15.77:0/2103787487 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:50:03.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.078+0000 7ff4837fe700 1 --2- 172.21.15.77:0/2103787487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff48406c500 0x7ff4841570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:03.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.078+0000 7ff47bfff700 1 --2- 172.21.15.77:0/2103787487 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff48406bfb0 0x7ff484157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:03.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.078+0000 7ff482ffd700 1 -- 172.21.15.77:0/2103787487 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff48406bfb0 msgr2=0x7ff484157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:03.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.078+0000 7ff482ffd700 1 --2- 172.21.15.77:0/2103787487 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff48406bfb0 0x7ff484157f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:03.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.078+0000 7ff482ffd700 1 -- 172.21.15.77:0/2103787487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff48406c500 msgr2=0x7ff4841570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:03.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.078+0000 7ff482ffd700 1 --2- 172.21.15.77:0/2103787487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff48406c500 0x7ff4841570f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:03.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.078+0000 7ff482ffd700 1 -- 172.21.15.77:0/2103787487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4740092a0 con 0x7ff48406bbe0 2022-04-03T19:50:03.090 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.079+0000 7ff482ffd700 1 --2- 172.21.15.77:0/2103787487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff48406bbe0 0x7ff484147d60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff47400ea50 tx=0x7ff474009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:03.090 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.079+0000 7ff47affd700 1 -- 172.21.15.77:0/2103787487 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff474009600 con 0x7ff48406bbe0 2022-04-03T19:50:03.090 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.079+0000 7ff47affd700 1 -- 172.21.15.77:0/2103787487 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47400c850 con 0x7ff48406bbe0 2022-04-03T19:50:03.090 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.079+0000 7ff483fff700 1 -- 172.21.15.77:0/2103787487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff46c0092a0 con 0x7ff48406bbe0 2022-04-03T19:50:03.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.079+0000 7ff47affd700 1 -- 172.21.15.77:0/2103787487 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47400cbf0 con 0x7ff48406bbe0 2022-04-03T19:50:03.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.079+0000 7ff483fff700 1 -- 172.21.15.77:0/2103787487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4841655c0 con 0x7ff48406bbe0 2022-04-03T19:50:03.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.080+0000 7ff483fff700 1 -- 172.21.15.77:0/2103787487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4840b1720 con 0x7ff48406bbe0 2022-04-03T19:50:03.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.081+0000 7ff47affd700 1 -- 172.21.15.77:0/2103787487 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7ff474019030 con 0x7ff48406bbe0 2022-04-03T19:50:03.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.082+0000 7ff47affd700 1 --2- 172.21.15.77:0/2103787487 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff468063ce0 0x7ff468066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:03.092 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.082+0000 7ff47affd700 1 -- 172.21.15.77:0/2103787487 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7ff474080a50 con 0x7ff48406bbe0 2022-04-03T19:50:03.092 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.082+0000 7ff47bfff700 1 --2- 172.21.15.77:0/2103787487 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff468063ce0 0x7ff468066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:03.092 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.083+0000 7ff47bfff700 1 --2- 172.21.15.77:0/2103787487 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff468063ce0 0x7ff468066190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff4840c6fe0 tx=0x7ff46c016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:03.093 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.086+0000 7ff47affd700 1 -- 172.21.15.77:0/2103787487 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7ff47404f3a0 con 0x7ff48406bbe0 2022-04-03T19:50:03.323 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.322+0000 7ff483fff700 1 -- 172.21.15.77:0/2103787487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff48404b050 con 0x7ff48406bbe0 2022-04-03T19:50:03.323 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.323+0000 7ff47affd700 1 -- 172.21.15.77:0/2103787487 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5355 (secure 0 0 0) 0x7ff474017090 con 0x7ff48406bbe0 2022-04-03T19:50:03.324 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:50:03.324 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:49:59.156875+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:50:03.326 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.325+0000 7ff483fff700 1 -- 172.21.15.77:0/2103787487 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff468063ce0 msgr2=0x7ff468066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:03.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.325+0000 7ff483fff700 1 --2- 172.21.15.77:0/2103787487 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff468063ce0 0x7ff468066190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff4840c6fe0 tx=0x7ff46c016000 comp rx=0 tx=0).stop 2022-04-03T19:50:03.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.325+0000 7ff483fff700 1 -- 172.21.15.77:0/2103787487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff48406bbe0 msgr2=0x7ff484147d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:03.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.325+0000 7ff483fff700 1 --2- 172.21.15.77:0/2103787487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff48406bbe0 0x7ff484147d60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff47400ea50 tx=0x7ff474009060 comp rx=0 tx=0).stop 2022-04-03T19:50:03.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.325+0000 7ff483fff700 1 -- 172.21.15.77:0/2103787487 shutdown_connections 2022-04-03T19:50:03.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.325+0000 7ff483fff700 1 --2- 172.21.15.77:0/2103787487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff48406c500 0x7ff4841570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:03.328 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.325+0000 7ff483fff700 1 --2- 172.21.15.77:0/2103787487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff48406bbe0 0x7ff484147d60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:03.328 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.325+0000 7ff483fff700 1 --2- 172.21.15.77:0/2103787487 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff468063ce0 0x7ff468066190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:03.328 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.325+0000 7ff483fff700 1 --2- 172.21.15.77:0/2103787487 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff48406bfb0 0x7ff484157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:03.328 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.326+0000 7ff483fff700 1 -- 172.21.15.77:0/2103787487 >> 172.21.15.77:0/2103787487 conn(0x7ff4840cb6b0 msgr2=0x7ff484148dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:03.329 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.326+0000 7ff483fff700 1 -- 172.21.15.77:0/2103787487 shutdown_connections 2022-04-03T19:50:03.329 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:03.326+0000 7ff483fff700 1 -- 172.21.15.77:0/2103787487 wait complete. 2022-04-03T19:50:03.329 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2022-04-03T19:50:03.352 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:50:05.353 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:50:05.561 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.560+0000 7f1f5f058700 1 Processor -- start 2022-04-03T19:50:05.561 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.560+0000 7f1f5f058700 1 -- start start 2022-04-03T19:50:05.563 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.561+0000 7f1f5f058700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1f580ba3c0 0x7f1f58156b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:05.564 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.561+0000 7f1f5f058700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f580678b0 0x7f1f5814d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:05.564 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.561+0000 7f1f5f058700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f580c3c50 con 0x7f1f58157060 2022-04-03T19:50:05.564 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.561+0000 7f1f5f058700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f580c8020 con 0x7f1f580ba3c0 2022-04-03T19:50:05.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.561+0000 7f1f5f058700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f58156690 con 0x7f1f580678b0 2022-04-03T19:50:05.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.562+0000 7f1f5e056700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1f580ba3c0 0x7f1f58156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:05.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.562+0000 7f1f5e857700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f580678b0 0x7f1f5814d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:05.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.562+0000 7f1f5e857700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f580678b0 0x7f1f5814d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60214/0 (socket says 172.21.15.77:60214) 2022-04-03T19:50:05.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.562+0000 7f1f5e857700 1 -- 172.21.15.77:0/3783846289 learned_addr learned my addr 172.21.15.77:0/3783846289 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:50:05.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.562+0000 7f1f5e056700 1 -- 172.21.15.77:0/3783846289 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f580678b0 msgr2=0x7f1f5814d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:50:05.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.562+0000 7f1f5e056700 1 --2- 172.21.15.77:0/3783846289 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f580678b0 0x7f1f5814d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:05.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.562+0000 7f1f5e056700 1 -- 172.21.15.77:0/3783846289 >> v1:172.21.15.77:6789/0 conn(0x7f1f58157060 legacy=0x7f1f5814cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:05.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.562+0000 7f1f5e056700 1 -- 172.21.15.77:0/3783846289 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f580bf350 con 0x7f1f580ba3c0 2022-04-03T19:50:05.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.563+0000 7f1f5e056700 1 --2- 172.21.15.77:0/3783846289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f580ba3c0 0x7f1f58156b20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1f54009760 tx=0x7f1f54009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a3ee2ee245311a server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:05.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.563+0000 7f1f5d054700 1 -- 172.21.15.77:0/3783846289 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f54011630 con 0x7f1f580ba3c0 2022-04-03T19:50:05.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.563+0000 7f1f5d054700 1 -- 172.21.15.77:0/3783846289 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f54011790 con 0x7f1f580ba3c0 2022-04-03T19:50:05.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.563+0000 7f1f5d054700 1 -- 172.21.15.77:0/3783846289 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f54011b50 con 0x7f1f580ba3c0 2022-04-03T19:50:05.571 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.563+0000 7f1f5f058700 1 -- 172.21.15.77:0/3783846289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f580ba3c0 msgr2=0x7f1f58156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:05.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.563+0000 7f1f5f058700 1 --2- 172.21.15.77:0/3783846289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f580ba3c0 0x7f1f58156b20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1f54009760 tx=0x7f1f54009060 comp rx=0 tx=0).stop 2022-04-03T19:50:05.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.563+0000 7f1f5f058700 1 -- 172.21.15.77:0/3783846289 shutdown_connections 2022-04-03T19:50:05.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.563+0000 7f1f5f058700 1 --2- 172.21.15.77:0/3783846289 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f580678b0 0x7f1f5814d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:05.573 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.563+0000 7f1f5f058700 1 --2- 172.21.15.77:0/3783846289 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f580ba3c0 0x7f1f58156b20 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:05.573 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.564+0000 7f1f5f058700 1 -- 172.21.15.77:0/3783846289 >> 172.21.15.77:0/3783846289 conn(0x7f1f58066da0 msgr2=0x7f1f580671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:05.573 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.564+0000 7f1f5f058700 1 -- 172.21.15.77:0/3783846289 shutdown_connections 2022-04-03T19:50:05.573 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.564+0000 7f1f5f058700 1 -- 172.21.15.77:0/3783846289 wait complete. 2022-04-03T19:50:05.574 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.565+0000 7f1f5f058700 1 Processor -- start 2022-04-03T19:50:05.574 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.565+0000 7f1f5f058700 1 -- start start 2022-04-03T19:50:05.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.566+0000 7f1f5f058700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f580678b0 0x7f1f58158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:05.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.566+0000 7f1f5f058700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f580ba3c0 0x7f1f580c6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:05.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.566+0000 7f1f5f058700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f58157060 0x7f1f58149560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:05.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.566+0000 7f1f5f058700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f58143be0 con 0x7f1f58157060 2022-04-03T19:50:05.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.566+0000 7f1f5f058700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f58168a80 con 0x7f1f580678b0 2022-04-03T19:50:05.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.566+0000 7f1f5f058700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5806add0 con 0x7f1f580ba3c0 2022-04-03T19:50:05.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.566+0000 7f1f5e056700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f580678b0 0x7f1f58158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:05.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.566+0000 7f1f5e857700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f58157060 0x7f1f58149560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:05.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.566+0000 7f1f5d855700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f580ba3c0 0x7f1f580c6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:05.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.566+0000 7f1f5e056700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f580678b0 0x7f1f58158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38110/0 (socket says 172.21.15.77:38110) 2022-04-03T19:50:05.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.566+0000 7f1f5e056700 1 -- 172.21.15.77:0/1977146473 learned_addr learned my addr 172.21.15.77:0/1977146473 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:50:05.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.567+0000 7f1f5d855700 1 -- 172.21.15.77:0/1977146473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f580678b0 msgr2=0x7f1f58158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:05.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.567+0000 7f1f5d855700 1 --2- 172.21.15.77:0/1977146473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f580678b0 0x7f1f58158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:05.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.567+0000 7f1f5d855700 1 -- 172.21.15.77:0/1977146473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f58157060 msgr2=0x7f1f58149560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:05.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.567+0000 7f1f5d855700 1 --2- 172.21.15.77:0/1977146473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f58157060 0x7f1f58149560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:05.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.567+0000 7f1f5d855700 1 -- 172.21.15.77:0/1977146473 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f540092a0 con 0x7f1f580ba3c0 2022-04-03T19:50:05.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.567+0000 7f1f5d855700 1 --2- 172.21.15.77:0/1977146473 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f580ba3c0 0x7f1f580c6150 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1f4800c410 tx=0x7f1f48007d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:05.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.567+0000 7f1f467fc700 1 -- 172.21.15.77:0/1977146473 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f480050d0 con 0x7f1f580ba3c0 2022-04-03T19:50:05.579 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.567+0000 7f1f467fc700 1 -- 172.21.15.77:0/1977146473 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f48005230 con 0x7f1f580ba3c0 2022-04-03T19:50:05.579 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.567+0000 7f1f5f058700 1 -- 172.21.15.77:0/1977146473 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f58069d50 con 0x7f1f580ba3c0 2022-04-03T19:50:05.579 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.567+0000 7f1f467fc700 1 -- 172.21.15.77:0/1977146473 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f48003710 con 0x7f1f580ba3c0 2022-04-03T19:50:05.579 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.568+0000 7f1f5f058700 1 -- 172.21.15.77:0/1977146473 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f5806a010 con 0x7f1f580ba3c0 2022-04-03T19:50:05.580 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.568+0000 7f1f5f058700 1 -- 172.21.15.77:0/1977146473 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f580c74a0 con 0x7f1f580ba3c0 2022-04-03T19:50:05.580 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.569+0000 7f1f467fc700 1 -- 172.21.15.77:0/1977146473 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f1f48019030 con 0x7f1f580ba3c0 2022-04-03T19:50:05.580 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.570+0000 7f1f467fc700 1 --2- 172.21.15.77:0/1977146473 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1f38063ce0 0x7f1f38066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:05.580 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.570+0000 7f1f467fc700 1 -- 172.21.15.77:0/1977146473 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7f1f4807f980 con 0x7f1f580ba3c0 2022-04-03T19:50:05.580 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.570+0000 7f1f5e056700 1 --2- 172.21.15.77:0/1977146473 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1f38063ce0 0x7f1f38066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:05.581 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.571+0000 7f1f5e056700 1 --2- 172.21.15.77:0/1977146473 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1f38063ce0 0x7f1f38066190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1f58067fa0 tx=0x7f1f54017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:05.581 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.574+0000 7f1f467fc700 1 -- 172.21.15.77:0/1977146473 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f1f4804e420 con 0x7f1f580ba3c0 2022-04-03T19:50:05.810 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.810+0000 7f1f5f058700 1 -- 172.21.15.77:0/1977146473 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f5806d5e0 con 0x7f1f580ba3c0 2022-04-03T19:50:05.811 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.811+0000 7f1f467fc700 1 -- 172.21.15.77:0/1977146473 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5355 (secure 0 0 0) 0x7f1f48017020 con 0x7f1f580ba3c0 2022-04-03T19:50:05.812 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:50:05.812 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:49:59.156875+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:50:05.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.813+0000 7f1f5f058700 1 -- 172.21.15.77:0/1977146473 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1f38063ce0 msgr2=0x7f1f38066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:05.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.813+0000 7f1f5f058700 1 --2- 172.21.15.77:0/1977146473 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1f38063ce0 0x7f1f38066190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1f58067fa0 tx=0x7f1f54017000 comp rx=0 tx=0).stop 2022-04-03T19:50:05.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.813+0000 7f1f5f058700 1 -- 172.21.15.77:0/1977146473 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f580ba3c0 msgr2=0x7f1f580c6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:05.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.813+0000 7f1f5f058700 1 --2- 172.21.15.77:0/1977146473 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f580ba3c0 0x7f1f580c6150 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1f4800c410 tx=0x7f1f48007d40 comp rx=0 tx=0).stop 2022-04-03T19:50:05.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.813+0000 7f1f5f058700 1 -- 172.21.15.77:0/1977146473 shutdown_connections 2022-04-03T19:50:05.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.813+0000 7f1f5f058700 1 --2- 172.21.15.77:0/1977146473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f58157060 0x7f1f58149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:05.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.813+0000 7f1f5f058700 1 --2- 172.21.15.77:0/1977146473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f580678b0 0x7f1f58158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.813+0000 7f1f5f058700 1 --2- 172.21.15.77:0/1977146473 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1f38063ce0 0x7f1f38066190 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.813+0000 7f1f5f058700 1 --2- 172.21.15.77:0/1977146473 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f580ba3c0 0x7f1f580c6150 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.813+0000 7f1f5f058700 1 -- 172.21.15.77:0/1977146473 >> 172.21.15.77:0/1977146473 conn(0x7f1f58066da0 msgr2=0x7f1f5814be60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.814+0000 7f1f5f058700 1 -- 172.21.15.77:0/1977146473 shutdown_connections 2022-04-03T19:50:05.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:05.814+0000 7f1f5f058700 1 -- 172.21.15.77:0/1977146473 wait complete. 2022-04-03T19:50:05.817 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 21 2022-04-03T19:50:05.838 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:50:07.839 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:50:08.045 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.044+0000 7f5659733700 1 Processor -- start 2022-04-03T19:50:08.045 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.045+0000 7f5659733700 1 -- start start 2022-04-03T19:50:08.047 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.046+0000 7f5659733700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f565406d3f0 0x7f565406d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:08.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.046+0000 7f5659733700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f56540c2a90 0x7f565406c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:08.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.046+0000 7f5659733700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56541562f0 con 0x7f56540c5930 2022-04-03T19:50:08.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.046+0000 7f5659733700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56540c2f80 con 0x7f56540c2a90 2022-04-03T19:50:08.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.046+0000 7f5659733700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56540c3260 con 0x7f565406d3f0 2022-04-03T19:50:08.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.046+0000 7f5658f32700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f56540c2a90 0x7f565406c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:08.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.046+0000 7f5658f32700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f56540c2a90 0x7f565406c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42926/0 (socket says 172.21.15.77:42926) 2022-04-03T19:50:08.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.046+0000 7f5658f32700 1 -- 172.21.15.77:0/2430494374 learned_addr learned my addr 172.21.15.77:0/2430494374 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:50:08.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.046+0000 7f5653fff700 1 --2- 172.21.15.77:0/2430494374 >> v2:172.21.15.186:3300/0 conn(0x7f565406d3f0 0x7f565406d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:08.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.047+0000 7f5658f32700 1 -- 172.21.15.77:0/2430494374 >> v2:172.21.15.186:3300/0 conn(0x7f565406d3f0 msgr2=0x7f565406d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:50:08.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.047+0000 7f5658f32700 1 --2- 172.21.15.77:0/2430494374 >> v2:172.21.15.186:3300/0 conn(0x7f565406d3f0 0x7f565406d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:08.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.047+0000 7f5658f32700 1 -- 172.21.15.77:0/2430494374 >> v1:172.21.15.186:6789/0 conn(0x7f56540c5930 legacy=0x7f5654155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:08.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.047+0000 7f5658f32700 1 -- 172.21.15.77:0/2430494374 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56540c3540 con 0x7f56540c2a90 2022-04-03T19:50:08.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.047+0000 7f5653fff700 1 --2- 172.21.15.77:0/2430494374 >> v2:172.21.15.186:3300/0 conn(0x7f565406d3f0 0x7f565406d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:50:08.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.047+0000 7f5658f32700 1 --2- 172.21.15.77:0/2430494374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56540c2a90 0x7f565406c040 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f564c009f20 tx=0x7f564c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38f4e0f745e86bc4 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:08.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.047+0000 7f5652ffd700 1 -- 172.21.15.77:0/2430494374 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c00e070 con 0x7f56540c2a90 2022-04-03T19:50:08.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.047+0000 7f5652ffd700 1 -- 172.21.15.77:0/2430494374 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564c00b990 con 0x7f56540c2a90 2022-04-03T19:50:08.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.047+0000 7f5652ffd700 1 -- 172.21.15.77:0/2430494374 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c0092e0 con 0x7f56540c2a90 2022-04-03T19:50:08.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.048+0000 7f5659733700 1 -- 172.21.15.77:0/2430494374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56540c2a90 msgr2=0x7f565406c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:08.055 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.048+0000 7f5659733700 1 --2- 172.21.15.77:0/2430494374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56540c2a90 0x7f565406c040 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f564c009f20 tx=0x7f564c009840 comp rx=0 tx=0).stop 2022-04-03T19:50:08.055 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.048+0000 7f5659733700 1 -- 172.21.15.77:0/2430494374 shutdown_connections 2022-04-03T19:50:08.055 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.048+0000 7f5659733700 1 --2- 172.21.15.77:0/2430494374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f56540c2a90 0x7f565406c040 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:08.055 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.048+0000 7f5659733700 1 --2- 172.21.15.77:0/2430494374 >> v2:172.21.15.186:3300/0 conn(0x7f565406d3f0 0x7f565406d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:08.055 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.048+0000 7f5659733700 1 -- 172.21.15.77:0/2430494374 >> 172.21.15.77:0/2430494374 conn(0x7f56540cb720 msgr2=0x7f56540cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:08.056 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.048+0000 7f5659733700 1 -- 172.21.15.77:0/2430494374 shutdown_connections 2022-04-03T19:50:08.056 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.048+0000 7f5659733700 1 -- 172.21.15.77:0/2430494374 wait complete. 2022-04-03T19:50:08.056 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.050+0000 7f5659733700 1 Processor -- start 2022-04-03T19:50:08.056 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.050+0000 7f5659733700 1 -- start start 2022-04-03T19:50:08.057 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.051+0000 7f5659733700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565406d3f0 0x7f5654165dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:08.057 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.051+0000 7f5659733700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56540c2a90 0x7f56540bf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:08.057 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.051+0000 7f5659733700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56540c5930 0x7f56540ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:08.058 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.051+0000 7f5659733700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56541597b0 con 0x7f565406d3f0 2022-04-03T19:50:08.058 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.051+0000 7f5659733700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56540c1cd0 con 0x7f56540c2a90 2022-04-03T19:50:08.058 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.051+0000 7f5659733700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56540bfa00 con 0x7f56540c5930 2022-04-03T19:50:08.058 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.051+0000 7f5653fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565406d3f0 0x7f5654165dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:08.059 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.051+0000 7f5653fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565406d3f0 0x7f5654165dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42928/0 (socket says 172.21.15.77:42928) 2022-04-03T19:50:08.059 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.051+0000 7f5653fff700 1 -- 172.21.15.77:0/3400535852 learned_addr learned my addr 172.21.15.77:0/3400535852 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:50:08.059 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.051+0000 7f5658f32700 1 --2- 172.21.15.77:0/3400535852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56540c5930 0x7f56540ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:08.059 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.051+0000 7f56537fe700 1 --2- 172.21.15.77:0/3400535852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56540c2a90 0x7f56540bf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:08.059 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.051+0000 7f5653fff700 1 -- 172.21.15.77:0/3400535852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56540c5930 msgr2=0x7f56540ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:08.060 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.051+0000 7f5653fff700 1 --2- 172.21.15.77:0/3400535852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56540c5930 0x7f56540ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:08.060 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.051+0000 7f5653fff700 1 -- 172.21.15.77:0/3400535852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56540c2a90 msgr2=0x7f56540bf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:08.060 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.051+0000 7f5653fff700 1 --2- 172.21.15.77:0/3400535852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56540c2a90 0x7f56540bf4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:08.060 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.051+0000 7f5653fff700 1 -- 172.21.15.77:0/3400535852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56440092a0 con 0x7f565406d3f0 2022-04-03T19:50:08.061 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.051+0000 7f56537fe700 1 --2- 172.21.15.77:0/3400535852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56540c2a90 0x7f56540bf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:50:08.061 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.052+0000 7f5653fff700 1 --2- 172.21.15.77:0/3400535852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565406d3f0 0x7f5654165dc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f564400ea80 tx=0x7f5644009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:08.061 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.052+0000 7f563bfff700 1 -- 172.21.15.77:0/3400535852 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564400c850 con 0x7f565406d3f0 2022-04-03T19:50:08.061 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.052+0000 7f563bfff700 1 -- 172.21.15.77:0/3400535852 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564400c9b0 con 0x7f565406d3f0 2022-04-03T19:50:08.061 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.052+0000 7f5659733700 1 -- 172.21.15.77:0/3400535852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f564c009ae0 con 0x7f565406d3f0 2022-04-03T19:50:08.062 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.052+0000 7f563bfff700 1 -- 172.21.15.77:0/3400535852 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564400cd30 con 0x7f565406d3f0 2022-04-03T19:50:08.062 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.052+0000 7f5659733700 1 -- 172.21.15.77:0/3400535852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56541594d0 con 0x7f565406d3f0 2022-04-03T19:50:08.062 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.053+0000 7f5659733700 1 -- 172.21.15.77:0/3400535852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565404b050 con 0x7f565406d3f0 2022-04-03T19:50:08.062 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.054+0000 7f563bfff700 1 -- 172.21.15.77:0/3400535852 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f5644023070 con 0x7f565406d3f0 2022-04-03T19:50:08.063 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.055+0000 7f563bfff700 1 --2- 172.21.15.77:0/3400535852 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5634063ce0 0x7f5634066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:08.064 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.055+0000 7f563bfff700 1 -- 172.21.15.77:0/3400535852 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7f5644080f00 con 0x7f565406d3f0 2022-04-03T19:50:08.064 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.055+0000 7f56537fe700 1 --2- 172.21.15.77:0/3400535852 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5634063ce0 0x7f5634066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:08.064 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.056+0000 7f56537fe700 1 --2- 172.21.15.77:0/3400535852 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5634063ce0 0x7f5634066190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f56540beef0 tx=0x7f5648009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:08.064 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.059+0000 7f563bfff700 1 -- 172.21.15.77:0/3400535852 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f564404f850 con 0x7f565406d3f0 2022-04-03T19:50:08.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.291+0000 7f5659733700 1 -- 172.21.15.77:0/3400535852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f565406e1e0 con 0x7f565406d3f0 2022-04-03T19:50:08.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.291+0000 7f563bfff700 1 -- 172.21.15.77:0/3400535852 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5359 (secure 0 0 0) 0x7f5644052e70 con 0x7f565406d3f0 2022-04-03T19:50:08.293 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:50:08.293 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:50:07.201387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:50:08.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.294+0000 7f5659733700 1 -- 172.21.15.77:0/3400535852 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5634063ce0 msgr2=0x7f5634066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:08.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.294+0000 7f5659733700 1 --2- 172.21.15.77:0/3400535852 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5634063ce0 0x7f5634066190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f56540beef0 tx=0x7f5648009040 comp rx=0 tx=0).stop 2022-04-03T19:50:08.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.294+0000 7f5659733700 1 -- 172.21.15.77:0/3400535852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565406d3f0 msgr2=0x7f5654165dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:08.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.294+0000 7f5659733700 1 --2- 172.21.15.77:0/3400535852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565406d3f0 0x7f5654165dc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f564400ea80 tx=0x7f5644009060 comp rx=0 tx=0).stop 2022-04-03T19:50:08.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.294+0000 7f5659733700 1 -- 172.21.15.77:0/3400535852 shutdown_connections 2022-04-03T19:50:08.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.294+0000 7f5659733700 1 --2- 172.21.15.77:0/3400535852 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5634063ce0 0x7f5634066190 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:08.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.294+0000 7f5659733700 1 --2- 172.21.15.77:0/3400535852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56540c5930 0x7f56540ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:08.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.294+0000 7f5659733700 1 --2- 172.21.15.77:0/3400535852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f565406d3f0 0x7f5654165dc0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:08.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.294+0000 7f5659733700 1 --2- 172.21.15.77:0/3400535852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56540c2a90 0x7f56540bf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:08.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.294+0000 7f5659733700 1 -- 172.21.15.77:0/3400535852 >> 172.21.15.77:0/3400535852 conn(0x7f56540cb720 msgr2=0x7f56540c8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:08.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.294+0000 7f5659733700 1 -- 172.21.15.77:0/3400535852 shutdown_connections 2022-04-03T19:50:08.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:08.294+0000 7f5659733700 1 -- 172.21.15.77:0/3400535852 wait complete. 2022-04-03T19:50:08.298 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 22 2022-04-03T19:50:08.319 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:50:10.320 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:50:10.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.526+0000 7f1b1f3e5700 1 Processor -- start 2022-04-03T19:50:10.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.526+0000 7f1b1f3e5700 1 -- start start 2022-04-03T19:50:10.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.527+0000 7f1b1f3e5700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1b181584b0 0x7f1b1814eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:10.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.527+0000 7f1b1f3e5700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b1806d260 con 0x7f1b1806bfb0 2022-04-03T19:50:10.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.527+0000 7f1b1f3e5700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b180c6330 con 0x7f1b1806bbe0 2022-04-03T19:50:10.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.527+0000 7f1b1f3e5700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b18159450 con 0x7f1b181584b0 2022-04-03T19:50:10.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.528+0000 7f1b1dbe2700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1b1806bfb0 0x7f1b1806c500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38460/0 (socket says 172.21.15.77:38460) 2022-04-03T19:50:10.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.528+0000 7f1b1dbe2700 1 -- 172.21.15.77:0/1127500098 learned_addr learned my addr 172.21.15.77:0/1127500098 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:50:10.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.528+0000 7f1b1ebe4700 1 --2- 172.21.15.77:0/1127500098 >> v2:172.21.15.186:3300/0 conn(0x7f1b181584b0 0x7f1b1814eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:10.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.528+0000 7f1b1d3e1700 1 -- 172.21.15.77:0/1127500098 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028107011 0 0) 0x7f1b1806d260 con 0x7f1b1806bfb0 2022-04-03T19:50:10.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.528+0000 7f1b1d3e1700 1 -- 172.21.15.77:0/1127500098 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b000033f0 con 0x7f1b1806bfb0 2022-04-03T19:50:10.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.528+0000 7f1b1d3e1700 1 -- 172.21.15.77:0/1127500098 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090902706 0 0) 0x7f1b180c6330 con 0x7f1b1806bbe0 2022-04-03T19:50:10.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.528+0000 7f1b1ebe4700 1 -- 172.21.15.77:0/1127500098 >> v1:172.21.15.186:6789/0 conn(0x7f1b1806bbe0 legacy=0x7f1b1806d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:10.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.528+0000 7f1b1ebe4700 1 -- 172.21.15.77:0/1127500098 >> v1:172.21.15.77:6789/0 conn(0x7f1b1806bfb0 legacy=0x7f1b1806c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:10.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.528+0000 7f1b1ebe4700 1 -- 172.21.15.77:0/1127500098 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b18158910 con 0x7f1b181584b0 2022-04-03T19:50:10.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.529+0000 7f1b1ebe4700 1 --2- 172.21.15.77:0/1127500098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b181584b0 0x7f1b1814eea0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009e60 tx=0x7f1b10009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=343b9f099ba1605 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:10.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.529+0000 7f1b1d3e1700 1 -- 172.21.15.77:0/1127500098 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1000e070 con 0x7f1b181584b0 2022-04-03T19:50:10.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.529+0000 7f1b1d3e1700 1 -- 172.21.15.77:0/1127500098 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b1000bb60 con 0x7f1b181584b0 2022-04-03T19:50:10.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.529+0000 7f1b1d3e1700 1 -- 172.21.15.77:0/1127500098 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b10012430 con 0x7f1b181584b0 2022-04-03T19:50:10.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.529+0000 7f1b1f3e5700 1 -- 172.21.15.77:0/1127500098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b181584b0 msgr2=0x7f1b1814eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:10.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.529+0000 7f1b1f3e5700 1 --2- 172.21.15.77:0/1127500098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b181584b0 0x7f1b1814eea0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009e60 tx=0x7f1b10009780 comp rx=0 tx=0).stop 2022-04-03T19:50:10.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.530+0000 7f1b1f3e5700 1 -- 172.21.15.77:0/1127500098 shutdown_connections 2022-04-03T19:50:10.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.530+0000 7f1b1f3e5700 1 --2- 172.21.15.77:0/1127500098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b181584b0 0x7f1b1814eea0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:10.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.530+0000 7f1b1f3e5700 1 -- 172.21.15.77:0/1127500098 >> 172.21.15.77:0/1127500098 conn(0x7f1b180cb6b0 msgr2=0x7f1b180cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:10.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.530+0000 7f1b1f3e5700 1 -- 172.21.15.77:0/1127500098 shutdown_connections 2022-04-03T19:50:10.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.530+0000 7f1b1f3e5700 1 -- 172.21.15.77:0/1127500098 wait complete. 2022-04-03T19:50:10.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.531+0000 7f1b1f3e5700 1 Processor -- start 2022-04-03T19:50:10.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.531+0000 7f1b1f3e5700 1 -- start start 2022-04-03T19:50:10.541 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.532+0000 7f1b1f3e5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b1806bbe0 0x7f1b18156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:10.542 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.532+0000 7f1b1f3e5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b1806bfb0 0x7f1b18146890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:10.542 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.532+0000 7f1b1f3e5700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b181584b0 0x7f1b18146dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:10.542 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.532+0000 7f1b1f3e5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b18156f40 con 0x7f1b1806bbe0 2022-04-03T19:50:10.542 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.532+0000 7f1b1f3e5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b18152c40 con 0x7f1b1806bfb0 2022-04-03T19:50:10.543 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.532+0000 7f1b1f3e5700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b18147310 con 0x7f1b181584b0 2022-04-03T19:50:10.543 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.533+0000 7f1b1e3e3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b1806bbe0 0x7f1b18156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:10.543 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.533+0000 7f1b1e3e3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b1806bbe0 0x7f1b18156a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42930/0 (socket says 172.21.15.77:42930) 2022-04-03T19:50:10.543 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.533+0000 7f1b1e3e3700 1 -- 172.21.15.77:0/1197019354 learned_addr learned my addr 172.21.15.77:0/1197019354 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:50:10.544 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.533+0000 7f1b1ebe4700 1 --2- 172.21.15.77:0/1197019354 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b181584b0 0x7f1b18146dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:10.544 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.533+0000 7f1b1dbe2700 1 --2- 172.21.15.77:0/1197019354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b1806bfb0 0x7f1b18146890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:10.544 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.533+0000 7f1b1e3e3700 1 -- 172.21.15.77:0/1197019354 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b181584b0 msgr2=0x7f1b18146dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.533+0000 7f1b1e3e3700 1 --2- 172.21.15.77:0/1197019354 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b181584b0 0x7f1b18146dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.533+0000 7f1b1e3e3700 1 -- 172.21.15.77:0/1197019354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b1806bfb0 msgr2=0x7f1b18146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.533+0000 7f1b1e3e3700 1 --2- 172.21.15.77:0/1197019354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b1806bfb0 0x7f1b18146890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.533+0000 7f1b1e3e3700 1 -- 172.21.15.77:0/1197019354 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b10009a20 con 0x7f1b1806bbe0 2022-04-03T19:50:10.545 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.534+0000 7f1b1e3e3700 1 --2- 172.21.15.77:0/1197019354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b1806bbe0 0x7f1b18156a00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1b1400c4c0 tx=0x7f1b14007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.534+0000 7f1b067fc700 1 -- 172.21.15.77:0/1197019354 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1400ce70 con 0x7f1b1806bbe0 2022-04-03T19:50:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.534+0000 7f1b067fc700 1 -- 172.21.15.77:0/1197019354 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b140052e0 con 0x7f1b1806bbe0 2022-04-03T19:50:10.546 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.534+0000 7f1b067fc700 1 -- 172.21.15.77:0/1197019354 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b14003730 con 0x7f1b1806bbe0 2022-04-03T19:50:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.534+0000 7f1b1f3e5700 1 -- 172.21.15.77:0/1197019354 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b180c39b0 con 0x7f1b1806bbe0 2022-04-03T19:50:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.534+0000 7f1b1f3e5700 1 -- 172.21.15.77:0/1197019354 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b181531c0 con 0x7f1b1806bbe0 2022-04-03T19:50:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.535+0000 7f1b1f3e5700 1 -- 172.21.15.77:0/1197019354 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b18158bb0 con 0x7f1b1806bbe0 2022-04-03T19:50:10.547 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.536+0000 7f1b067fc700 1 -- 172.21.15.77:0/1197019354 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f1b14019030 con 0x7f1b1806bbe0 2022-04-03T19:50:10.548 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.537+0000 7f1b067fc700 1 --2- 172.21.15.77:0/1197019354 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1af8063ce0 0x7f1af8066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:10.548 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.537+0000 7f1b067fc700 1 -- 172.21.15.77:0/1197019354 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7f1b1407fbf0 con 0x7f1b1806bbe0 2022-04-03T19:50:10.548 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.537+0000 7f1b1dbe2700 1 --2- 172.21.15.77:0/1197019354 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1af8063ce0 0x7f1af8066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:10.548 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.538+0000 7f1b1dbe2700 1 --2- 172.21.15.77:0/1197019354 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1af8063ce0 0x7f1af8066190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1b180c7b80 tx=0x7f1b08009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:10.549 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.541+0000 7f1b067fc700 1 -- 172.21.15.77:0/1197019354 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f1b1404e690 con 0x7f1b1806bbe0 2022-04-03T19:50:10.772 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.771+0000 7f1b1f3e5700 1 -- 172.21.15.77:0/1197019354 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b180c7ec0 con 0x7f1b1806bbe0 2022-04-03T19:50:10.772 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.772+0000 7f1b067fc700 1 -- 172.21.15.77:0/1197019354 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5359 (secure 0 0 0) 0x7f1b14017070 con 0x7f1b1806bbe0 2022-04-03T19:50:10.773 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:50:10.773 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:50:07.201387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:50:10.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.774+0000 7f1b1f3e5700 1 -- 172.21.15.77:0/1197019354 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1af8063ce0 msgr2=0x7f1af8066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:10.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.774+0000 7f1b1f3e5700 1 --2- 172.21.15.77:0/1197019354 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1af8063ce0 0x7f1af8066190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1b180c7b80 tx=0x7f1b08009040 comp rx=0 tx=0).stop 2022-04-03T19:50:10.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.774+0000 7f1b1f3e5700 1 -- 172.21.15.77:0/1197019354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b1806bbe0 msgr2=0x7f1b18156a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:10.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.774+0000 7f1b1f3e5700 1 --2- 172.21.15.77:0/1197019354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b1806bbe0 0x7f1b18156a00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1b1400c4c0 tx=0x7f1b14007c70 comp rx=0 tx=0).stop 2022-04-03T19:50:10.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.774+0000 7f1b1f3e5700 1 -- 172.21.15.77:0/1197019354 shutdown_connections 2022-04-03T19:50:10.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.774+0000 7f1b1f3e5700 1 --2- 172.21.15.77:0/1197019354 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1af8063ce0 0x7f1af8066190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:10.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.774+0000 7f1b1f3e5700 1 --2- 172.21.15.77:0/1197019354 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b181584b0 0x7f1b18146dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:10.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.774+0000 7f1b1f3e5700 1 --2- 172.21.15.77:0/1197019354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b1806bbe0 0x7f1b18156a00 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:10.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.774+0000 7f1b1f3e5700 1 --2- 172.21.15.77:0/1197019354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b1806bfb0 0x7f1b18146890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:10.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.774+0000 7f1b1f3e5700 1 -- 172.21.15.77:0/1197019354 >> 172.21.15.77:0/1197019354 conn(0x7f1b180cb6b0 msgr2=0x7f1b18149990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:10.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.774+0000 7f1b1f3e5700 1 -- 172.21.15.77:0/1197019354 shutdown_connections 2022-04-03T19:50:10.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:10.774+0000 7f1b1f3e5700 1 -- 172.21.15.77:0/1197019354 wait complete. 2022-04-03T19:50:10.778 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 22 2022-04-03T19:50:10.799 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:50:12.800 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:50:13.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.006+0000 7fdf098c3700 1 Processor -- start 2022-04-03T19:50:13.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.006+0000 7fdf098c3700 1 -- start start 2022-04-03T19:50:13.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.008+0000 7fdf098c3700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdf0406bfb0 0x7fdf041584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:13.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.008+0000 7fdf098c3700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fdf0406c500 0x7fdf0414eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:13.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.008+0000 7fdf098c3700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0406d260 con 0x7fdf0406bbe0 2022-04-03T19:50:13.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.008+0000 7fdf098c3700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf04159450 con 0x7fdf0406c500 2022-04-03T19:50:13.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.008+0000 7fdf098c3700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf040c6330 con 0x7fdf0406bfb0 2022-04-03T19:50:13.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.008+0000 7fdf088c1700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdf0406bbe0 0x7fdf0406d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38462/0 (socket says 172.21.15.77:38462) 2022-04-03T19:50:13.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.008+0000 7fdf088c1700 1 -- 172.21.15.77:0/4282690515 learned_addr learned my addr 172.21.15.77:0/4282690515 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:50:13.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.008+0000 7fdefbfff700 1 --2- 172.21.15.77:0/4282690515 >> v2:172.21.15.186:3300/0 conn(0x7fdf0406bfb0 0x7fdf041584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:13.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.008+0000 7fdf090c2700 1 --2- 172.21.15.77:0/4282690515 >> v2:172.21.15.77:3300/0 conn(0x7fdf0406c500 0x7fdf0414eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:13.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.009+0000 7fdefb7fe700 1 -- 172.21.15.77:0/4282690515 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4207265300 0 0) 0x7fdf0406d260 con 0x7fdf0406bbe0 2022-04-03T19:50:13.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.009+0000 7fdefb7fe700 1 -- 172.21.15.77:0/4282690515 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdef00033f0 con 0x7fdf0406bbe0 2022-04-03T19:50:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.009+0000 7fdefbfff700 1 -- 172.21.15.77:0/4282690515 >> v2:172.21.15.77:3300/0 conn(0x7fdf0406c500 msgr2=0x7fdf0414eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:50:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.009+0000 7fdefbfff700 1 --2- 172.21.15.77:0/4282690515 >> v2:172.21.15.77:3300/0 conn(0x7fdf0406c500 0x7fdf0414eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.009+0000 7fdefbfff700 1 -- 172.21.15.77:0/4282690515 >> v1:172.21.15.77:6789/0 conn(0x7fdf0406bbe0 legacy=0x7fdf0406d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.009+0000 7fdefbfff700 1 -- 172.21.15.77:0/4282690515 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf04158bb0 con 0x7fdf0406bfb0 2022-04-03T19:50:13.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.009+0000 7fdefbfff700 1 --2- 172.21.15.77:0/4282690515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf0406bfb0 0x7fdf041584b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdf040898c0 tx=0x7fdef4009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67c663d78471d57c server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:13.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.009+0000 7fdefb7fe700 1 -- 172.21.15.77:0/4282690515 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef40105b0 con 0x7fdf0406bfb0 2022-04-03T19:50:13.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.009+0000 7fdefb7fe700 1 -- 172.21.15.77:0/4282690515 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef4010710 con 0x7fdf0406bfb0 2022-04-03T19:50:13.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.009+0000 7fdefb7fe700 1 -- 172.21.15.77:0/4282690515 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef4010ad0 con 0x7fdf0406bfb0 2022-04-03T19:50:13.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.010+0000 7fdf098c3700 1 -- 172.21.15.77:0/4282690515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf0406bfb0 msgr2=0x7fdf041584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:13.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.010+0000 7fdf098c3700 1 --2- 172.21.15.77:0/4282690515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf0406bfb0 0x7fdf041584b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdf040898c0 tx=0x7fdef4009060 comp rx=0 tx=0).stop 2022-04-03T19:50:13.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.010+0000 7fdf098c3700 1 -- 172.21.15.77:0/4282690515 shutdown_connections 2022-04-03T19:50:13.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.010+0000 7fdf098c3700 1 --2- 172.21.15.77:0/4282690515 >> v2:172.21.15.77:3300/0 conn(0x7fdf0406c500 0x7fdf0414eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:13.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.010+0000 7fdf098c3700 1 --2- 172.21.15.77:0/4282690515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf0406bfb0 0x7fdf041584b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:13.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.010+0000 7fdf098c3700 1 -- 172.21.15.77:0/4282690515 >> 172.21.15.77:0/4282690515 conn(0x7fdf040cb6b0 msgr2=0x7fdf040cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:13.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.010+0000 7fdf098c3700 1 -- 172.21.15.77:0/4282690515 shutdown_connections 2022-04-03T19:50:13.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.010+0000 7fdf098c3700 1 -- 172.21.15.77:0/4282690515 wait complete. 2022-04-03T19:50:13.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.011+0000 7fdf098c3700 1 Processor -- start 2022-04-03T19:50:13.019 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.012+0000 7fdf098c3700 1 -- start start 2022-04-03T19:50:13.019 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.012+0000 7fdf098c3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf0406bbe0 0x7fdf04149460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:13.019 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.012+0000 7fdf098c3700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf0406bfb0 0x7fdf041540c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:13.019 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.012+0000 7fdf098c3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf0406c500 0x7fdf04154600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:13.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.013+0000 7fdf098c3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf04150d30 con 0x7fdf0406c500 2022-04-03T19:50:13.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.013+0000 7fdf098c3700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf04155d10 con 0x7fdf0406bbe0 2022-04-03T19:50:13.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.013+0000 7fdf098c3700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf040c3740 con 0x7fdf0406bfb0 2022-04-03T19:50:13.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.013+0000 7fdf090c2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf0406c500 0x7fdf04154600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:13.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.013+0000 7fdf090c2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf0406c500 0x7fdf04154600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42934/0 (socket says 172.21.15.77:42934) 2022-04-03T19:50:13.021 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.013+0000 7fdf090c2700 1 -- 172.21.15.77:0/389493009 learned_addr learned my addr 172.21.15.77:0/389493009 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:50:13.021 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.013+0000 7fdefbfff700 1 --2- 172.21.15.77:0/389493009 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf0406bfb0 0x7fdf041540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:13.021 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.013+0000 7fdf088c1700 1 --2- 172.21.15.77:0/389493009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf0406bbe0 0x7fdf04149460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:13.021 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.013+0000 7fdf090c2700 1 -- 172.21.15.77:0/389493009 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf0406bfb0 msgr2=0x7fdf041540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:13.021 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.013+0000 7fdf090c2700 1 --2- 172.21.15.77:0/389493009 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf0406bfb0 0x7fdf041540c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:13.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.013+0000 7fdf090c2700 1 -- 172.21.15.77:0/389493009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf0406bbe0 msgr2=0x7fdf04149460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:13.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.013+0000 7fdf090c2700 1 --2- 172.21.15.77:0/389493009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf0406bbe0 0x7fdf04149460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:13.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.013+0000 7fdf090c2700 1 -- 172.21.15.77:0/389493009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef40092a0 con 0x7fdf0406c500 2022-04-03T19:50:13.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.014+0000 7fdf090c2700 1 --2- 172.21.15.77:0/389493009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf0406c500 0x7fdf04154600 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fdefc009e90 tx=0x7fdefc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:13.023 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.014+0000 7fdef8ff9700 1 -- 172.21.15.77:0/389493009 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc00caa0 con 0x7fdf0406c500 2022-04-03T19:50:13.023 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.014+0000 7fdef8ff9700 1 -- 172.21.15.77:0/389493009 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdefc00f040 con 0x7fdf0406c500 2022-04-03T19:50:13.023 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.014+0000 7fdf098c3700 1 -- 172.21.15.77:0/389493009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf040c3a20 con 0x7fdf0406c500 2022-04-03T19:50:13.023 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.014+0000 7fdef8ff9700 1 -- 172.21.15.77:0/389493009 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc00ce50 con 0x7fdf0406c500 2022-04-03T19:50:13.023 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.014+0000 7fdf098c3700 1 -- 172.21.15.77:0/389493009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf041506e0 con 0x7fdf0406c500 2022-04-03T19:50:13.024 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.015+0000 7fdf098c3700 1 -- 172.21.15.77:0/389493009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf040b1720 con 0x7fdf0406c500 2022-04-03T19:50:13.025 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.016+0000 7fdef8ff9700 1 -- 172.21.15.77:0/389493009 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fdefc014690 con 0x7fdf0406c500 2022-04-03T19:50:13.025 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.017+0000 7fdef8ff9700 1 --2- 172.21.15.77:0/389493009 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdee4063ce0 0x7fdee4066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:13.025 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.017+0000 7fdef8ff9700 1 -- 172.21.15.77:0/389493009 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7fdefc081680 con 0x7fdf0406c500 2022-04-03T19:50:13.025 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.017+0000 7fdf088c1700 1 --2- 172.21.15.77:0/389493009 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdee4063ce0 0x7fdee4066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:13.026 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.018+0000 7fdf088c1700 1 --2- 172.21.15.77:0/389493009 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdee4063ce0 0x7fdee4066190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdf00005750 tx=0x7fdf00009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:13.026 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.021+0000 7fdef8ff9700 1 -- 172.21.15.77:0/389493009 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fdefc050030 con 0x7fdf0406c500 2022-04-03T19:50:13.255 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.254+0000 7fdf098c3700 1 -- 172.21.15.77:0/389493009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf0404b050 con 0x7fdf0406c500 2022-04-03T19:50:13.255 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.255+0000 7fdef8ff9700 1 -- 172.21.15.77:0/389493009 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5356 (secure 0 0 0) 0x7fdefc018020 con 0x7fdf0406c500 2022-04-03T19:50:13.256 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:50:13.256 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:50:12.222121+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:50:13.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.257+0000 7fdf098c3700 1 -- 172.21.15.77:0/389493009 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdee4063ce0 msgr2=0x7fdee4066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:13.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.257+0000 7fdf098c3700 1 --2- 172.21.15.77:0/389493009 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdee4063ce0 0x7fdee4066190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdf00005750 tx=0x7fdf00009040 comp rx=0 tx=0).stop 2022-04-03T19:50:13.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.257+0000 7fdf098c3700 1 -- 172.21.15.77:0/389493009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf0406c500 msgr2=0x7fdf04154600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:13.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.257+0000 7fdf098c3700 1 --2- 172.21.15.77:0/389493009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf0406c500 0x7fdf04154600 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fdefc009e90 tx=0x7fdefc00b040 comp rx=0 tx=0).stop 2022-04-03T19:50:13.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.257+0000 7fdf098c3700 1 -- 172.21.15.77:0/389493009 shutdown_connections 2022-04-03T19:50:13.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.257+0000 7fdf098c3700 1 --2- 172.21.15.77:0/389493009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf0406c500 0x7fdf04154600 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.257+0000 7fdf098c3700 1 --2- 172.21.15.77:0/389493009 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf0406bbe0 0x7fdf04149460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.257+0000 7fdf098c3700 1 --2- 172.21.15.77:0/389493009 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdee4063ce0 0x7fdee4066190 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.257+0000 7fdf098c3700 1 --2- 172.21.15.77:0/389493009 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf0406bfb0 0x7fdf041540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:13.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.257+0000 7fdf098c3700 1 -- 172.21.15.77:0/389493009 >> 172.21.15.77:0/389493009 conn(0x7fdf040cb6b0 msgr2=0x7fdf0414b550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:13.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.257+0000 7fdf098c3700 1 -- 172.21.15.77:0/389493009 shutdown_connections 2022-04-03T19:50:13.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:13.258+0000 7fdf098c3700 1 -- 172.21.15.77:0/389493009 wait complete. 2022-04-03T19:50:13.261 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 23 2022-04-03T19:50:13.283 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 23, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-04-03T19:44:21.275716+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 23, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4850": { "addr": "172.21.15.186:6837/3568325507", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 3568325507, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 3568325507, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4850, "incarnation": 21, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-04-03T19:50:12.222121+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4850 } } } ], "standbys": [ { "addr": "172.21.15.77:6834/2116223509", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6832", "nonce": 2116223509, "type": "v2" }, { "addr": "172.21.15.77:6834", "nonce": 2116223509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4245, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6836/2374276747", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2374276747, "type": "v2" }, { "addr": "172.21.15.186:6836", "nonce": 2374276747, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4259, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/610990853", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6838", "nonce": 610990853, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 610990853, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4265, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.77:6835/170053412", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6833", "nonce": 170053412, "type": "v2" }, { "addr": "172.21.15.77:6835", "nonce": 170053412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4271, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-04-03T19:50:15.285 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:50:15.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.490+0000 7f9b3752a700 1 Processor -- start 2022-04-03T19:50:15.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.491+0000 7f9b3752a700 1 -- start start 2022-04-03T19:50:15.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.492+0000 7f9b3752a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9b3006bfb0 0x7f9b301584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:15.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.492+0000 7f9b3752a700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9b3006c500 0x7f9b3014eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:15.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.492+0000 7f9b3752a700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3006d260 con 0x7f9b3006bbe0 2022-04-03T19:50:15.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.492+0000 7f9b3752a700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b30159450 con 0x7f9b3006bfb0 2022-04-03T19:50:15.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.492+0000 7f9b3752a700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b300c6330 con 0x7f9b3006c500 2022-04-03T19:50:15.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.492+0000 7f9b35d27700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9b3006bfb0 0x7f9b301584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:15.497 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.492+0000 7f9b36528700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9b3006bbe0 0x7f9b3006d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38464/0 (socket says 172.21.15.77:38464) 2022-04-03T19:50:15.497 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.492+0000 7f9b36528700 1 -- 172.21.15.77:0/1159075879 learned_addr learned my addr 172.21.15.77:0/1159075879 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:50:15.497 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.492+0000 7f9b36d29700 1 --2- 172.21.15.77:0/1159075879 >> v2:172.21.15.186:3300/0 conn(0x7f9b3006c500 0x7f9b3014eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:15.497 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.493+0000 7f9b35d27700 1 -- 172.21.15.77:0/1159075879 >> v2:172.21.15.186:3300/0 conn(0x7f9b3006c500 msgr2=0x7f9b3014eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:50:15.498 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.493+0000 7f9b35d27700 1 --2- 172.21.15.77:0/1159075879 >> v2:172.21.15.186:3300/0 conn(0x7f9b3006c500 0x7f9b3014eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:15.498 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.493+0000 7f9b35d27700 1 -- 172.21.15.77:0/1159075879 >> v1:172.21.15.77:6789/0 conn(0x7f9b3006bbe0 legacy=0x7f9b3006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:15.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.493+0000 7f9b35d27700 1 -- 172.21.15.77:0/1159075879 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b30158bb0 con 0x7f9b3006bfb0 2022-04-03T19:50:15.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.493+0000 7f9b35526700 1 -- 172.21.15.77:0/1159075879 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036165041 0 0) 0x7f9b3006d260 con 0x7f9b3006bbe0 2022-04-03T19:50:15.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.493+0000 7f9b35d27700 1 --2- 172.21.15.77:0/1159075879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b3006bfb0 0x7f9b301584b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9b300cc6b0 tx=0x7f9b20009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9abac8474e7ce56d server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:15.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.493+0000 7f9b35526700 1 -- 172.21.15.77:0/1159075879 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b20011630 con 0x7f9b3006bfb0 2022-04-03T19:50:15.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.493+0000 7f9b35526700 1 -- 172.21.15.77:0/1159075879 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b20011790 con 0x7f9b3006bfb0 2022-04-03T19:50:15.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.493+0000 7f9b35526700 1 -- 172.21.15.77:0/1159075879 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b20011b30 con 0x7f9b3006bfb0 2022-04-03T19:50:15.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.494+0000 7f9b3752a700 1 -- 172.21.15.77:0/1159075879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b3006bfb0 msgr2=0x7f9b301584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:15.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.494+0000 7f9b3752a700 1 --2- 172.21.15.77:0/1159075879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b3006bfb0 0x7f9b301584b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9b300cc6b0 tx=0x7f9b20009060 comp rx=0 tx=0).stop 2022-04-03T19:50:15.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.494+0000 7f9b3752a700 1 -- 172.21.15.77:0/1159075879 shutdown_connections 2022-04-03T19:50:15.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.494+0000 7f9b3752a700 1 --2- 172.21.15.77:0/1159075879 >> v2:172.21.15.186:3300/0 conn(0x7f9b3006c500 0x7f9b3014eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:15.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.494+0000 7f9b3752a700 1 --2- 172.21.15.77:0/1159075879 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b3006bfb0 0x7f9b301584b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:15.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.494+0000 7f9b3752a700 1 -- 172.21.15.77:0/1159075879 >> 172.21.15.77:0/1159075879 conn(0x7f9b300cb6b0 msgr2=0x7f9b300cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:15.502 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.494+0000 7f9b3752a700 1 -- 172.21.15.77:0/1159075879 shutdown_connections 2022-04-03T19:50:15.502 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.494+0000 7f9b3752a700 1 -- 172.21.15.77:0/1159075879 wait complete. 2022-04-03T19:50:15.505 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.495+0000 7f9b3752a700 1 Processor -- start 2022-04-03T19:50:15.505 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.496+0000 7f9b3752a700 1 -- start start 2022-04-03T19:50:15.506 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.497+0000 7f9b3752a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b3006bbe0 0x7f9b30168e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:15.506 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.497+0000 7f9b3752a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b3006bfb0 0x7f9b30154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:15.506 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.497+0000 7f9b3752a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b3006c500 0x7f9b30154de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:15.506 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.497+0000 7f9b3752a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b301693a0 con 0x7f9b3006c500 2022-04-03T19:50:15.507 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.497+0000 7f9b3752a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b30153bd0 con 0x7f9b3006bbe0 2022-04-03T19:50:15.507 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.497+0000 7f9b3752a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b30152ce0 con 0x7f9b3006bfb0 2022-04-03T19:50:15.507 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.497+0000 7f9b36d29700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b3006c500 0x7f9b30154de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:15.507 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.497+0000 7f9b36d29700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b3006c500 0x7f9b30154de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42940/0 (socket says 172.21.15.77:42940) 2022-04-03T19:50:15.508 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.497+0000 7f9b36d29700 1 -- 172.21.15.77:0/477299154 learned_addr learned my addr 172.21.15.77:0/477299154 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:50:15.508 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.497+0000 7f9b35d27700 1 --2- 172.21.15.77:0/477299154 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b3006bfb0 0x7f9b30154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:15.508 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.497+0000 7f9b36528700 1 --2- 172.21.15.77:0/477299154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b3006bbe0 0x7f9b30168e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:15.509 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.497+0000 7f9b36d29700 1 -- 172.21.15.77:0/477299154 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b3006bfb0 msgr2=0x7f9b30154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:15.509 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.497+0000 7f9b36d29700 1 --2- 172.21.15.77:0/477299154 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b3006bfb0 0x7f9b30154060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:15.509 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.497+0000 7f9b36d29700 1 -- 172.21.15.77:0/477299154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b3006bbe0 msgr2=0x7f9b30168e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:15.509 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.497+0000 7f9b36d29700 1 --2- 172.21.15.77:0/477299154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b3006bbe0 0x7f9b30168e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:15.510 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.497+0000 7f9b36d29700 1 -- 172.21.15.77:0/477299154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b200092a0 con 0x7f9b3006c500 2022-04-03T19:50:15.510 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.497+0000 7f9b36528700 1 --2- 172.21.15.77:0/477299154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b3006bbe0 0x7f9b30168e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:50:15.511 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.498+0000 7f9b36d29700 1 --2- 172.21.15.77:0/477299154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b3006c500 0x7f9b30154de0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9b28009e90 tx=0x7f9b2800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:15.511 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.498+0000 7f9b1e7fc700 1 -- 172.21.15.77:0/477299154 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2800cb50 con 0x7f9b3006c500 2022-04-03T19:50:15.512 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.498+0000 7f9b1e7fc700 1 -- 172.21.15.77:0/477299154 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2800f040 con 0x7f9b3006c500 2022-04-03T19:50:15.512 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.498+0000 7f9b3752a700 1 -- 172.21.15.77:0/477299154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b30152fc0 con 0x7f9b3006c500 2022-04-03T19:50:15.512 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.498+0000 7f9b1e7fc700 1 -- 172.21.15.77:0/477299154 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b28014480 con 0x7f9b3006c500 2022-04-03T19:50:15.513 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.498+0000 7f9b3752a700 1 -- 172.21.15.77:0/477299154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b30153580 con 0x7f9b3006c500 2022-04-03T19:50:15.513 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.499+0000 7f9b3752a700 1 -- 172.21.15.77:0/477299154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b30153d60 con 0x7f9b3006c500 2022-04-03T19:50:15.513 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.500+0000 7f9b1e7fc700 1 -- 172.21.15.77:0/477299154 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f9b280147f0 con 0x7f9b3006c500 2022-04-03T19:50:15.514 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.501+0000 7f9b1e7fc700 1 --2- 172.21.15.77:0/477299154 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9b10063ce0 0x7f9b10066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:15.514 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.501+0000 7f9b1e7fc700 1 -- 172.21.15.77:0/477299154 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7f9b280819a0 con 0x7f9b3006c500 2022-04-03T19:50:15.514 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.501+0000 7f9b36528700 1 --2- 172.21.15.77:0/477299154 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9b10063ce0 0x7f9b10066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:15.515 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.502+0000 7f9b36528700 1 --2- 172.21.15.77:0/477299154 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9b10063ce0 0x7f9b10066190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c0076b0 tx=0x7f9b2c007bf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:15.515 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.505+0000 7f9b1e7fc700 1 -- 172.21.15.77:0/477299154 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f9b280503e0 con 0x7f9b3006c500 2022-04-03T19:50:15.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.736+0000 7f9b3752a700 1 -- 172.21.15.77:0/477299154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b300c7ec0 con 0x7f9b3006c500 2022-04-03T19:50:15.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.737+0000 7f9b1e7fc700 1 -- 172.21.15.77:0/477299154 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5356 (secure 0 0 0) 0x7f9b28018020 con 0x7f9b3006c500 2022-04-03T19:50:15.738 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:50:15.738 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:50:12.222121+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:50:15.740 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.740+0000 7f9b3752a700 1 -- 172.21.15.77:0/477299154 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9b10063ce0 msgr2=0x7f9b10066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:15.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.740+0000 7f9b3752a700 1 --2- 172.21.15.77:0/477299154 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9b10063ce0 0x7f9b10066190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c0076b0 tx=0x7f9b2c007bf0 comp rx=0 tx=0).stop 2022-04-03T19:50:15.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.740+0000 7f9b3752a700 1 -- 172.21.15.77:0/477299154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b3006c500 msgr2=0x7f9b30154de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:15.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.740+0000 7f9b3752a700 1 --2- 172.21.15.77:0/477299154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b3006c500 0x7f9b30154de0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9b28009e90 tx=0x7f9b2800b040 comp rx=0 tx=0).stop 2022-04-03T19:50:15.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.740+0000 7f9b3752a700 1 -- 172.21.15.77:0/477299154 shutdown_connections 2022-04-03T19:50:15.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.740+0000 7f9b3752a700 1 --2- 172.21.15.77:0/477299154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b3006c500 0x7f9b30154de0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:15.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.740+0000 7f9b3752a700 1 --2- 172.21.15.77:0/477299154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b3006bbe0 0x7f9b30168e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:15.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.740+0000 7f9b3752a700 1 --2- 172.21.15.77:0/477299154 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9b10063ce0 0x7f9b10066190 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:15.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.740+0000 7f9b3752a700 1 --2- 172.21.15.77:0/477299154 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b3006bfb0 0x7f9b30154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:15.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.740+0000 7f9b3752a700 1 -- 172.21.15.77:0/477299154 >> 172.21.15.77:0/477299154 conn(0x7f9b300cb6b0 msgr2=0x7f9b3006d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:15.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.740+0000 7f9b3752a700 1 -- 172.21.15.77:0/477299154 shutdown_connections 2022-04-03T19:50:15.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:15.740+0000 7f9b3752a700 1 -- 172.21.15.77:0/477299154 wait complete. 2022-04-03T19:50:15.744 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 23 2022-04-03T19:50:15.764 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:50:16.017 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:50:16.022 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:50:17.765 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:50:17.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.971+0000 7fbcea057700 1 Processor -- start 2022-04-03T19:50:17.973 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.972+0000 7fbcea057700 1 -- start start 2022-04-03T19:50:17.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.973+0000 7fbcea057700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbce40c2010 0x7fbce4159050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:17.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.973+0000 7fbcea057700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbce406a280 0x7fbce4155910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:17.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.973+0000 7fbcea057700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbce4155480 con 0x7fbce40c4dc0 2022-04-03T19:50:17.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.973+0000 7fbcea057700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbce40c4200 con 0x7fbce40c2010 2022-04-03T19:50:17.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.973+0000 7fbcea057700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbce4159590 con 0x7fbce406a280 2022-04-03T19:50:17.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.973+0000 7fbce9055700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbce40c2010 0x7fbce4159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:17.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.973+0000 7fbce9856700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fbce40c4dc0 0x7fbce406ba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38468/0 (socket says 172.21.15.77:38468) 2022-04-03T19:50:17.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.973+0000 7fbce9856700 1 -- 172.21.15.77:0/3538128891 learned_addr learned my addr 172.21.15.77:0/3538128891 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:50:17.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.973+0000 7fbce8854700 1 --2- 172.21.15.77:0/3538128891 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbce406a280 0x7fbce4155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:17.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.974+0000 7fbce9055700 1 -- 172.21.15.77:0/3538128891 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbce406a280 msgr2=0x7fbce4155910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:50:17.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.974+0000 7fbce9055700 1 --2- 172.21.15.77:0/3538128891 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbce406a280 0x7fbce4155910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:17.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.974+0000 7fbce9055700 1 -- 172.21.15.77:0/3538128891 >> v1:172.21.15.77:6789/0 conn(0x7fbce40c4dc0 legacy=0x7fbce406ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:17.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.974+0000 7fbce9055700 1 -- 172.21.15.77:0/3538128891 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce40c2d70 con 0x7fbce40c2010 2022-04-03T19:50:17.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.974+0000 7fbcd3fff700 1 -- 172.21.15.77:0/3538128891 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751701007 0 0) 0x7fbce4155480 con 0x7fbce40c4dc0 2022-04-03T19:50:17.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.974+0000 7fbce8854700 1 --2- 172.21.15.77:0/3538128891 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbce406a280 0x7fbce4155910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:50:17.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.974+0000 7fbce9055700 1 --2- 172.21.15.77:0/3538128891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbce40c2010 0x7fbce4159050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbce0009760 tx=0x7fbce0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c93e7265216225 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:17.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.974+0000 7fbcd3fff700 1 -- 172.21.15.77:0/3538128891 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce0011630 con 0x7fbce40c2010 2022-04-03T19:50:17.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.974+0000 7fbcd3fff700 1 -- 172.21.15.77:0/3538128891 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce0011790 con 0x7fbce40c2010 2022-04-03T19:50:17.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.974+0000 7fbcd3fff700 1 -- 172.21.15.77:0/3538128891 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce0011b50 con 0x7fbce40c2010 2022-04-03T19:50:17.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.975+0000 7fbcea057700 1 -- 172.21.15.77:0/3538128891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbce40c2010 msgr2=0x7fbce4159050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:17.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.975+0000 7fbcea057700 1 --2- 172.21.15.77:0/3538128891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbce40c2010 0x7fbce4159050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbce0009760 tx=0x7fbce0009060 comp rx=0 tx=0).stop 2022-04-03T19:50:17.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.975+0000 7fbcea057700 1 -- 172.21.15.77:0/3538128891 shutdown_connections 2022-04-03T19:50:17.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.975+0000 7fbcea057700 1 --2- 172.21.15.77:0/3538128891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbce40c2010 0x7fbce4159050 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:17.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.975+0000 7fbcea057700 1 --2- 172.21.15.77:0/3538128891 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbce406a280 0x7fbce4155910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:17.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.975+0000 7fbcea057700 1 -- 172.21.15.77:0/3538128891 >> 172.21.15.77:0/3538128891 conn(0x7fbce406cb30 msgr2=0x7fbce406cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:17.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.975+0000 7fbcea057700 1 -- 172.21.15.77:0/3538128891 shutdown_connections 2022-04-03T19:50:17.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.975+0000 7fbcea057700 1 -- 172.21.15.77:0/3538128891 wait complete. 2022-04-03T19:50:17.984 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.977+0000 7fbcea057700 1 Processor -- start 2022-04-03T19:50:17.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.977+0000 7fbcea057700 1 -- start start 2022-04-03T19:50:17.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.978+0000 7fbcea057700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbce406a280 0x7fbce414c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:17.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.978+0000 7fbcea057700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbce40c2010 0x7fbce414fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:17.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.978+0000 7fbcea057700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbce40c4dc0 0x7fbce4143e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:17.986 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.978+0000 7fbcea057700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce40c16a0 con 0x7fbce40c4dc0 2022-04-03T19:50:17.986 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.978+0000 7fbcea057700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce40bb950 con 0x7fbce40c2010 2022-04-03T19:50:17.986 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.978+0000 7fbcea057700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbce40c1150 con 0x7fbce406a280 2022-04-03T19:50:17.986 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.978+0000 7fbce9856700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbce40c4dc0 0x7fbce4143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:17.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.978+0000 7fbce9856700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbce40c4dc0 0x7fbce4143e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42946/0 (socket says 172.21.15.77:42946) 2022-04-03T19:50:17.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.978+0000 7fbce9856700 1 -- 172.21.15.77:0/826340184 learned_addr learned my addr 172.21.15.77:0/826340184 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:50:17.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.978+0000 7fbce9055700 1 --2- 172.21.15.77:0/826340184 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbce406a280 0x7fbce414c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:17.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.978+0000 7fbce8854700 1 --2- 172.21.15.77:0/826340184 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbce40c2010 0x7fbce414fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:17.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.978+0000 7fbce9856700 1 -- 172.21.15.77:0/826340184 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbce406a280 msgr2=0x7fbce414c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:17.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.978+0000 7fbce9856700 1 --2- 172.21.15.77:0/826340184 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbce406a280 0x7fbce414c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:17.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.978+0000 7fbce9856700 1 -- 172.21.15.77:0/826340184 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbce40c2010 msgr2=0x7fbce414fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:17.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.978+0000 7fbce9856700 1 --2- 172.21.15.77:0/826340184 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbce40c2010 0x7fbce414fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:17.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.978+0000 7fbce9856700 1 -- 172.21.15.77:0/826340184 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd40092a0 con 0x7fbce40c4dc0 2022-04-03T19:50:17.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.979+0000 7fbce9856700 1 --2- 172.21.15.77:0/826340184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbce40c4dc0 0x7fbce4143e30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc00c790 tx=0x7fbcdc001540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:17.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.979+0000 7fbcd17fa700 1 -- 172.21.15.77:0/826340184 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc004c40 con 0x7fbce40c4dc0 2022-04-03T19:50:17.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.979+0000 7fbcd17fa700 1 -- 172.21.15.77:0/826340184 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcdc004da0 con 0x7fbce40c4dc0 2022-04-03T19:50:17.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.979+0000 7fbcea057700 1 -- 172.21.15.77:0/826340184 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbce00092a0 con 0x7fbce40c4dc0 2022-04-03T19:50:17.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.979+0000 7fbcd17fa700 1 -- 172.21.15.77:0/826340184 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc007380 con 0x7fbce40c4dc0 2022-04-03T19:50:17.990 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.979+0000 7fbcea057700 1 -- 172.21.15.77:0/826340184 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbce40c13c0 con 0x7fbce40c4dc0 2022-04-03T19:50:17.990 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.980+0000 7fbcea057700 1 -- 172.21.15.77:0/826340184 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbce40bba90 con 0x7fbce40c4dc0 2022-04-03T19:50:17.990 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.981+0000 7fbcd17fa700 1 -- 172.21.15.77:0/826340184 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fbcdc01a030 con 0x7fbce40c4dc0 2022-04-03T19:50:17.990 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.982+0000 7fbcd17fa700 1 --2- 172.21.15.77:0/826340184 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbcc4063ce0 0x7fbcc4066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:17.991 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.982+0000 7fbcd17fa700 1 -- 172.21.15.77:0/826340184 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7fbcdc07ff00 con 0x7fbce40c4dc0 2022-04-03T19:50:17.991 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.982+0000 7fbce9055700 1 --2- 172.21.15.77:0/826340184 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbcc4063ce0 0x7fbcc4066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:17.991 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.983+0000 7fbce9055700 1 --2- 172.21.15.77:0/826340184 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbcc4063ce0 0x7fbcc4066190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbce0009760 tx=0x7fbce0017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:17.991 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:17.986+0000 7fbcd17fa700 1 -- 172.21.15.77:0/826340184 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fbcdc04e9c0 con 0x7fbce40c4dc0 2022-04-03T19:50:18.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:18.222+0000 7fbcea057700 1 -- 172.21.15.77:0/826340184 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbce4152b20 con 0x7fbce40c4dc0 2022-04-03T19:50:18.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:18.223+0000 7fbcd17fa700 1 -- 172.21.15.77:0/826340184 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5356 (secure 0 0 0) 0x7fbcdc018030 con 0x7fbce40c4dc0 2022-04-03T19:50:18.224 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:50:18.224 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:50:12.222121+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:50:18.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:18.225+0000 7fbcea057700 1 -- 172.21.15.77:0/826340184 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbcc4063ce0 msgr2=0x7fbcc4066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:18.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:18.225+0000 7fbcea057700 1 --2- 172.21.15.77:0/826340184 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbcc4063ce0 0x7fbcc4066190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbce0009760 tx=0x7fbce0017000 comp rx=0 tx=0).stop 2022-04-03T19:50:18.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:18.225+0000 7fbcea057700 1 -- 172.21.15.77:0/826340184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbce40c4dc0 msgr2=0x7fbce4143e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:18.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:18.225+0000 7fbcea057700 1 --2- 172.21.15.77:0/826340184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbce40c4dc0 0x7fbce4143e30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc00c790 tx=0x7fbcdc001540 comp rx=0 tx=0).stop 2022-04-03T19:50:18.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:18.225+0000 7fbcea057700 1 -- 172.21.15.77:0/826340184 shutdown_connections 2022-04-03T19:50:18.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:18.225+0000 7fbcea057700 1 --2- 172.21.15.77:0/826340184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbce40c4dc0 0x7fbce4143e30 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:18.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:18.225+0000 7fbcea057700 1 --2- 172.21.15.77:0/826340184 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbcc4063ce0 0x7fbcc4066190 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:18.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:18.225+0000 7fbcea057700 1 --2- 172.21.15.77:0/826340184 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbce406a280 0x7fbce414c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:18.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:18.225+0000 7fbcea057700 1 --2- 172.21.15.77:0/826340184 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbce40c2010 0x7fbce414fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:18.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:18.225+0000 7fbcea057700 1 -- 172.21.15.77:0/826340184 >> 172.21.15.77:0/826340184 conn(0x7fbce406cb30 msgr2=0x7fbce414ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:18.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:18.225+0000 7fbcea057700 1 -- 172.21.15.77:0/826340184 shutdown_connections 2022-04-03T19:50:18.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:18.225+0000 7fbcea057700 1 -- 172.21.15.77:0/826340184 wait complete. 2022-04-03T19:50:18.229 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 23 2022-04-03T19:50:18.251 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:50:20.254 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:50:20.461 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.460+0000 7fca15fc6700 1 Processor -- start 2022-04-03T19:50:20.462 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.460+0000 7fca15fc6700 1 -- start start 2022-04-03T19:50:20.464 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.461+0000 7fca15fc6700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fca1006bbe0 0x7fca101584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:20.465 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.461+0000 7fca15fc6700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca1006d260 con 0x7fca1006c500 2022-04-03T19:50:20.465 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.461+0000 7fca15fc6700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca100c6330 con 0x7fca1006bfb0 2022-04-03T19:50:20.465 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.461+0000 7fca15fc6700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca10159450 con 0x7fca1006bbe0 2022-04-03T19:50:20.465 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.462+0000 7fca0ffff700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fca1006bfb0 0x7fca1006d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44260/0 (socket says 172.21.15.77:44260) 2022-04-03T19:50:20.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.462+0000 7fca14fc4700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fca1006bbe0 0x7fca101584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:20.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.462+0000 7fca0ffff700 1 -- 172.21.15.77:0/4020978612 learned_addr learned my addr 172.21.15.77:0/4020978612 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:50:20.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.462+0000 7fca157c5700 1 --1- 172.21.15.77:0/4020978612 >> v1:172.21.15.77:6789/0 conn(0x7fca1006c500 0x7fca1014eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38470/0 (socket says 172.21.15.77:38470) 2022-04-03T19:50:20.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.462+0000 7fca0f7fe700 1 -- 172.21.15.77:0/4020978612 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184543609 0 0) 0x7fca100c6330 con 0x7fca1006bfb0 2022-04-03T19:50:20.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.462+0000 7fca0f7fe700 1 -- 172.21.15.77:0/4020978612 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9f40033f0 con 0x7fca1006bfb0 2022-04-03T19:50:20.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.462+0000 7fca0f7fe700 1 -- 172.21.15.77:0/4020978612 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590646573 0 0) 0x7fca1006d260 con 0x7fca1006c500 2022-04-03T19:50:20.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.462+0000 7fca14fc4700 1 -- 172.21.15.77:0/4020978612 >> v1:172.21.15.186:6789/0 conn(0x7fca1006bfb0 legacy=0x7fca1006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:20.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.462+0000 7fca14fc4700 1 -- 172.21.15.77:0/4020978612 >> v1:172.21.15.77:6789/0 conn(0x7fca1006c500 legacy=0x7fca1014eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:20.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.462+0000 7fca14fc4700 1 -- 172.21.15.77:0/4020978612 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca10158bb0 con 0x7fca1006bbe0 2022-04-03T19:50:20.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.463+0000 7fca14fc4700 1 --2- 172.21.15.77:0/4020978612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca1006bbe0 0x7fca101584b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fca04009760 tx=0x7fca04009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe69f28dc1fc52 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:20.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.463+0000 7fca0f7fe700 1 -- 172.21.15.77:0/4020978612 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca04011630 con 0x7fca1006bbe0 2022-04-03T19:50:20.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.463+0000 7fca0f7fe700 1 -- 172.21.15.77:0/4020978612 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca04011790 con 0x7fca1006bbe0 2022-04-03T19:50:20.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.463+0000 7fca0f7fe700 1 -- 172.21.15.77:0/4020978612 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca04011b10 con 0x7fca1006bbe0 2022-04-03T19:50:20.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.463+0000 7fca15fc6700 1 -- 172.21.15.77:0/4020978612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca1006bbe0 msgr2=0x7fca101584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:20.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.463+0000 7fca15fc6700 1 --2- 172.21.15.77:0/4020978612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca1006bbe0 0x7fca101584b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fca04009760 tx=0x7fca04009060 comp rx=0 tx=0).stop 2022-04-03T19:50:20.471 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.464+0000 7fca15fc6700 1 -- 172.21.15.77:0/4020978612 shutdown_connections 2022-04-03T19:50:20.472 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.464+0000 7fca15fc6700 1 --2- 172.21.15.77:0/4020978612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca1006bbe0 0x7fca101584b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:20.472 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.464+0000 7fca15fc6700 1 -- 172.21.15.77:0/4020978612 >> 172.21.15.77:0/4020978612 conn(0x7fca100cb6b0 msgr2=0x7fca100cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:20.472 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.464+0000 7fca15fc6700 1 -- 172.21.15.77:0/4020978612 shutdown_connections 2022-04-03T19:50:20.473 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.464+0000 7fca15fc6700 1 -- 172.21.15.77:0/4020978612 wait complete. 2022-04-03T19:50:20.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.465+0000 7fca15fc6700 1 Processor -- start 2022-04-03T19:50:20.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.465+0000 7fca15fc6700 1 -- start start 2022-04-03T19:50:20.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.466+0000 7fca15fc6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca1006bbe0 0x7fca10168e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:20.481 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.466+0000 7fca15fc6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca1006bfb0 0x7fca10157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:20.481 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.466+0000 7fca15fc6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca1006c500 0x7fca10153370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:20.482 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.466+0000 7fca15fc6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca10152570 con 0x7fca1006bbe0 2022-04-03T19:50:20.482 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.466+0000 7fca15fc6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca10158700 con 0x7fca1006bfb0 2022-04-03T19:50:20.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.466+0000 7fca15fc6700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca101653a0 con 0x7fca1006c500 2022-04-03T19:50:20.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.467+0000 7fca14fc4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca1006bbe0 0x7fca10168e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:20.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.467+0000 7fca157c5700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca1006c500 0x7fca10153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:20.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.467+0000 7fca0ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca1006bfb0 0x7fca10157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:20.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.467+0000 7fca14fc4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca1006bbe0 0x7fca10168e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42952/0 (socket says 172.21.15.77:42952) 2022-04-03T19:50:20.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.467+0000 7fca14fc4700 1 -- 172.21.15.77:0/3419510453 learned_addr learned my addr 172.21.15.77:0/3419510453 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:50:20.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.467+0000 7fca14fc4700 1 -- 172.21.15.77:0/3419510453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca1006c500 msgr2=0x7fca10153370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:20.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.467+0000 7fca14fc4700 1 --2- 172.21.15.77:0/3419510453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca1006c500 0x7fca10153370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:20.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.467+0000 7fca14fc4700 1 -- 172.21.15.77:0/3419510453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca1006bfb0 msgr2=0x7fca10157750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:20.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.467+0000 7fca14fc4700 1 --2- 172.21.15.77:0/3419510453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca1006bfb0 0x7fca10157750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:20.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.467+0000 7fca14fc4700 1 -- 172.21.15.77:0/3419510453 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca040092a0 con 0x7fca1006bbe0 2022-04-03T19:50:20.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.467+0000 7fca14fc4700 1 --2- 172.21.15.77:0/3419510453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca1006bbe0 0x7fca10168e50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fca04009e00 tx=0x7fca04017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:20.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.468+0000 7fca0cff9700 1 -- 172.21.15.77:0/3419510453 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca04011b10 con 0x7fca1006bbe0 2022-04-03T19:50:20.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.468+0000 7fca0cff9700 1 -- 172.21.15.77:0/3419510453 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0401be00 con 0x7fca1006bbe0 2022-04-03T19:50:20.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.468+0000 7fca15fc6700 1 -- 172.21.15.77:0/3419510453 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca10165680 con 0x7fca1006bbe0 2022-04-03T19:50:20.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.468+0000 7fca0cff9700 1 -- 172.21.15.77:0/3419510453 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca040042a0 con 0x7fca1006bbe0 2022-04-03T19:50:20.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.468+0000 7fca15fc6700 1 -- 172.21.15.77:0/3419510453 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca10165c40 con 0x7fca1006bbe0 2022-04-03T19:50:20.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.469+0000 7fca15fc6700 1 -- 172.21.15.77:0/3419510453 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca1014a2b0 con 0x7fca1006bbe0 2022-04-03T19:50:20.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.470+0000 7fca0cff9700 1 -- 172.21.15.77:0/3419510453 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fca0400f050 con 0x7fca1006bbe0 2022-04-03T19:50:20.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.471+0000 7fca0cff9700 1 --2- 172.21.15.77:0/3419510453 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc9f8063d10 0x7fc9f80661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:20.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.471+0000 7fca0cff9700 1 -- 172.21.15.77:0/3419510453 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7fca04082070 con 0x7fca1006bbe0 2022-04-03T19:50:20.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.471+0000 7fca0ffff700 1 --2- 172.21.15.77:0/3419510453 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc9f8063d10 0x7fc9f80661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:20.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.472+0000 7fca0ffff700 1 --2- 172.21.15.77:0/3419510453 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc9f8063d10 0x7fc9f80661c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fca10162ad0 tx=0x7fc9fc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:20.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.475+0000 7fca0cff9700 1 -- 172.21.15.77:0/3419510453 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fca040b0ae0 con 0x7fca1006bbe0 2022-04-03T19:50:20.705 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.705+0000 7fca15fc6700 1 -- 172.21.15.77:0/3419510453 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca1004b050 con 0x7fca1006bbe0 2022-04-03T19:50:20.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.706+0000 7fca0cff9700 1 -- 172.21.15.77:0/3419510453 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5362 (secure 0 0 0) 0x7fca04015090 con 0x7fca1006bbe0 2022-04-03T19:50:20.707 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:50:20.707 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:50:18.243174+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:clientreplay","state_seq":25,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:50:20.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.708+0000 7fca15fc6700 1 -- 172.21.15.77:0/3419510453 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc9f8063d10 msgr2=0x7fc9f80661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:20.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.708+0000 7fca15fc6700 1 --2- 172.21.15.77:0/3419510453 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc9f8063d10 0x7fc9f80661c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fca10162ad0 tx=0x7fc9fc009040 comp rx=0 tx=0).stop 2022-04-03T19:50:20.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.708+0000 7fca15fc6700 1 -- 172.21.15.77:0/3419510453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca1006bbe0 msgr2=0x7fca10168e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:20.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.708+0000 7fca15fc6700 1 --2- 172.21.15.77:0/3419510453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca1006bbe0 0x7fca10168e50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fca04009e00 tx=0x7fca04017000 comp rx=0 tx=0).stop 2022-04-03T19:50:20.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.708+0000 7fca15fc6700 1 -- 172.21.15.77:0/3419510453 shutdown_connections 2022-04-03T19:50:20.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.708+0000 7fca15fc6700 1 --2- 172.21.15.77:0/3419510453 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fc9f8063d10 0x7fc9f80661c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:20.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.708+0000 7fca15fc6700 1 --2- 172.21.15.77:0/3419510453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca1006c500 0x7fca10153370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:20.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.708+0000 7fca15fc6700 1 --2- 172.21.15.77:0/3419510453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fca1006bbe0 0x7fca10168e50 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:20.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.708+0000 7fca15fc6700 1 --2- 172.21.15.77:0/3419510453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca1006bfb0 0x7fca10157750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:20.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.708+0000 7fca15fc6700 1 -- 172.21.15.77:0/3419510453 >> 172.21.15.77:0/3419510453 conn(0x7fca100cb6b0 msgr2=0x7fca10149a80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:20.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.708+0000 7fca15fc6700 1 -- 172.21.15.77:0/3419510453 shutdown_connections 2022-04-03T19:50:20.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:20.708+0000 7fca15fc6700 1 -- 172.21.15.77:0/3419510453 wait complete. 2022-04-03T19:50:20.712 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 24 2022-04-03T19:50:20.733 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:50:22.734 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:50:22.941 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.939+0000 7f0bcffff700 1 Processor -- start 2022-04-03T19:50:22.941 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.939+0000 7f0bcffff700 1 -- start start 2022-04-03T19:50:22.943 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.940+0000 7f0bcffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0bd00c5930 0x7f0bd0155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:22.943 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.940+0000 7f0bcffff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0bd0156120 0x7f0bd006c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:22.944 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.940+0000 7f0bcffff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd00c2f80 con 0x7f0bd006d3f0 2022-04-03T19:50:22.944 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.940+0000 7f0bcffff700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd00c3260 con 0x7f0bd0156120 2022-04-03T19:50:22.944 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.940+0000 7f0bcffff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd00c3540 con 0x7f0bd00c5930 2022-04-03T19:50:22.944 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.941+0000 7f0bceffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0bd006d3f0 0x7f0bd006d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38472/0 (socket says 172.21.15.77:38472) 2022-04-03T19:50:22.945 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.941+0000 7f0bceffd700 1 -- 172.21.15.77:0/917868461 learned_addr learned my addr 172.21.15.77:0/917868461 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:50:22.945 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.941+0000 7f0bce7fc700 1 --2- 172.21.15.77:0/917868461 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0bd00c5930 0x7f0bd0155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:22.945 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.941+0000 7f0bcf7fe700 1 --2- 172.21.15.77:0/917868461 >> v2:172.21.15.186:3300/0 conn(0x7f0bd0156120 0x7f0bd006c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:22.945 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.941+0000 7f0bcdffb700 1 -- 172.21.15.77:0/917868461 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001610461 0 0) 0x7f0bd00c2f80 con 0x7f0bd006d3f0 2022-04-03T19:50:22.945 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.941+0000 7f0bcdffb700 1 -- 172.21.15.77:0/917868461 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bbc0033f0 con 0x7f0bd006d3f0 2022-04-03T19:50:22.946 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.941+0000 7f0bce7fc700 1 -- 172.21.15.77:0/917868461 >> v2:172.21.15.186:3300/0 conn(0x7f0bd0156120 msgr2=0x7f0bd006c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:50:22.946 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.941+0000 7f0bce7fc700 1 --2- 172.21.15.77:0/917868461 >> v2:172.21.15.186:3300/0 conn(0x7f0bd0156120 0x7f0bd006c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:22.946 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.941+0000 7f0bce7fc700 1 -- 172.21.15.77:0/917868461 >> v1:172.21.15.77:6789/0 conn(0x7f0bd006d3f0 legacy=0x7f0bd006d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:22.946 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.941+0000 7f0bce7fc700 1 -- 172.21.15.77:0/917868461 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd00c3820 con 0x7f0bd00c5930 2022-04-03T19:50:22.947 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.941+0000 7f0bcf7fe700 1 --2- 172.21.15.77:0/917868461 >> v2:172.21.15.186:3300/0 conn(0x7f0bd0156120 0x7f0bd006c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:50:22.947 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.942+0000 7f0bce7fc700 1 --2- 172.21.15.77:0/917868461 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0bd00c5930 0x7f0bd0155be0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4009790 tx=0x7f0bc4009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b22fb064690f2e84 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:22.947 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.942+0000 7f0bcdffb700 1 -- 172.21.15.77:0/917868461 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4011630 con 0x7f0bd00c5930 2022-04-03T19:50:22.947 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.942+0000 7f0bcdffb700 1 -- 172.21.15.77:0/917868461 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc4011790 con 0x7f0bd00c5930 2022-04-03T19:50:22.947 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.942+0000 7f0bcdffb700 1 -- 172.21.15.77:0/917868461 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4011b50 con 0x7f0bd00c5930 2022-04-03T19:50:22.951 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.942+0000 7f0bcffff700 1 -- 172.21.15.77:0/917868461 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0bd00c5930 msgr2=0x7f0bd0155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:22.951 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.942+0000 7f0bcffff700 1 --2- 172.21.15.77:0/917868461 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0bd00c5930 0x7f0bd0155be0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4009790 tx=0x7f0bc4009060 comp rx=0 tx=0).stop 2022-04-03T19:50:22.951 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.943+0000 7f0bcffff700 1 -- 172.21.15.77:0/917868461 shutdown_connections 2022-04-03T19:50:22.952 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.943+0000 7f0bcffff700 1 --2- 172.21.15.77:0/917868461 >> v2:172.21.15.186:3300/0 conn(0x7f0bd0156120 0x7f0bd006c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:22.952 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.943+0000 7f0bcffff700 1 --2- 172.21.15.77:0/917868461 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0bd00c5930 0x7f0bd0155be0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:22.952 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.943+0000 7f0bcffff700 1 -- 172.21.15.77:0/917868461 >> 172.21.15.77:0/917868461 conn(0x7f0bd00cb720 msgr2=0x7f0bd00cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:22.952 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.943+0000 7f0bcffff700 1 -- 172.21.15.77:0/917868461 shutdown_connections 2022-04-03T19:50:22.953 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.943+0000 7f0bcffff700 1 -- 172.21.15.77:0/917868461 wait complete. 2022-04-03T19:50:22.953 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.944+0000 7f0bcffff700 1 Processor -- start 2022-04-03T19:50:22.953 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.944+0000 7f0bcffff700 1 -- start start 2022-04-03T19:50:22.953 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.945+0000 7f0bcffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0bd006d3f0 0x7f0bd00c5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:22.953 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.945+0000 7f0bcffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0bd00c5930 0x7f0bd0165450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:22.954 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.945+0000 7f0bcffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0bd0156120 0x7f0bd0165990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:22.954 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.945+0000 7f0bcffff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd0157ba0 con 0x7f0bd00c5930 2022-04-03T19:50:22.954 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.945+0000 7f0bcffff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd01590d0 con 0x7f0bd006d3f0 2022-04-03T19:50:22.955 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.945+0000 7f0bcffff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd0156e50 con 0x7f0bd0156120 2022-04-03T19:50:22.955 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.945+0000 7f0bce7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0bd00c5930 0x7f0bd0165450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:22.955 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.945+0000 7f0bce7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0bd00c5930 0x7f0bd0165450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42954/0 (socket says 172.21.15.77:42954) 2022-04-03T19:50:22.955 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.946+0000 7f0bce7fc700 1 -- 172.21.15.77:0/1348812532 learned_addr learned my addr 172.21.15.77:0/1348812532 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:50:22.955 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.946+0000 7f0bcf7fe700 1 --2- 172.21.15.77:0/1348812532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0bd0156120 0x7f0bd0165990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:22.956 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.946+0000 7f0bceffd700 1 --2- 172.21.15.77:0/1348812532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0bd006d3f0 0x7f0bd00c5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:22.956 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.946+0000 7f0bce7fc700 1 -- 172.21.15.77:0/1348812532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0bd0156120 msgr2=0x7f0bd0165990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:22.956 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.946+0000 7f0bce7fc700 1 --2- 172.21.15.77:0/1348812532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0bd0156120 0x7f0bd0165990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:22.956 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.946+0000 7f0bce7fc700 1 -- 172.21.15.77:0/1348812532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0bd006d3f0 msgr2=0x7f0bd00c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:22.957 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.946+0000 7f0bce7fc700 1 --2- 172.21.15.77:0/1348812532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0bd006d3f0 0x7f0bd00c5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:22.957 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.946+0000 7f0bce7fc700 1 -- 172.21.15.77:0/1348812532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc8009a20 con 0x7f0bd00c5930 2022-04-03T19:50:22.957 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.946+0000 7f0bce7fc700 1 --2- 172.21.15.77:0/1348812532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0bd00c5930 0x7f0bd0165450 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0bd00c3680 tx=0x7f0bc4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:22.957 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.946+0000 7f0bb6ffd700 1 -- 172.21.15.77:0/1348812532 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4011b10 con 0x7f0bd00c5930 2022-04-03T19:50:22.958 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.947+0000 7f0bb6ffd700 1 -- 172.21.15.77:0/1348812532 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc4000c90 con 0x7f0bd00c5930 2022-04-03T19:50:22.958 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.947+0000 7f0bcffff700 1 -- 172.21.15.77:0/1348812532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc40092a0 con 0x7f0bd00c5930 2022-04-03T19:50:22.958 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.947+0000 7f0bb6ffd700 1 -- 172.21.15.77:0/1348812532 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4025660 con 0x7f0bd00c5930 2022-04-03T19:50:22.958 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.947+0000 7f0bcffff700 1 -- 172.21.15.77:0/1348812532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd01570c0 con 0x7f0bd00c5930 2022-04-03T19:50:22.959 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.948+0000 7f0bcffff700 1 -- 172.21.15.77:0/1348812532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd00c5570 con 0x7f0bd00c5930 2022-04-03T19:50:22.959 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.948+0000 7f0bb6ffd700 1 -- 172.21.15.77:0/1348812532 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f0bc400f050 con 0x7f0bd00c5930 2022-04-03T19:50:22.959 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.949+0000 7f0bb6ffd700 1 --2- 172.21.15.77:0/1348812532 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0bb0063ce0 0x7f0bb0066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:22.959 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.949+0000 7f0bb6ffd700 1 -- 172.21.15.77:0/1348812532 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7f0bc4081e00 con 0x7f0bd00c5930 2022-04-03T19:50:22.960 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.950+0000 7f0bceffd700 1 --2- 172.21.15.77:0/1348812532 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0bb0063ce0 0x7f0bb0066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:22.960 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.950+0000 7f0bceffd700 1 --2- 172.21.15.77:0/1348812532 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0bb0063ce0 0x7f0bb0066190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0005750 tx=0x7f0bc0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:22.961 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:22.953+0000 7f0bb6ffd700 1 -- 172.21.15.77:0/1348812532 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f0bc4050900 con 0x7f0bd00c5930 2022-04-03T19:50:23.187 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:23.185+0000 7f0bcffff700 1 -- 172.21.15.77:0/1348812532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bd00c74b0 con 0x7f0bd00c5930 2022-04-03T19:50:23.187 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:23.186+0000 7f0bb6ffd700 1 -- 172.21.15.77:0/1348812532 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5362 (secure 0 0 0) 0x7f0bc4015070 con 0x7f0bd00c5930 2022-04-03T19:50:23.188 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:50:23.188 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:50:18.243174+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:clientreplay","state_seq":25,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:50:23.190 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:23.188+0000 7f0bcffff700 1 -- 172.21.15.77:0/1348812532 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0bb0063ce0 msgr2=0x7f0bb0066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:23.191 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:23.188+0000 7f0bcffff700 1 --2- 172.21.15.77:0/1348812532 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0bb0063ce0 0x7f0bb0066190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0005750 tx=0x7f0bc0009040 comp rx=0 tx=0).stop 2022-04-03T19:50:23.191 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:23.188+0000 7f0bcffff700 1 -- 172.21.15.77:0/1348812532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0bd00c5930 msgr2=0x7f0bd0165450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:23.191 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:23.188+0000 7f0bcffff700 1 --2- 172.21.15.77:0/1348812532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0bd00c5930 0x7f0bd0165450 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0bd00c3680 tx=0x7f0bc4017000 comp rx=0 tx=0).stop 2022-04-03T19:50:23.191 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:23.189+0000 7f0bcffff700 1 -- 172.21.15.77:0/1348812532 shutdown_connections 2022-04-03T19:50:23.192 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:23.189+0000 7f0bcffff700 1 --2- 172.21.15.77:0/1348812532 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0bb0063ce0 0x7f0bb0066190 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:23.192 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:23.189+0000 7f0bcffff700 1 --2- 172.21.15.77:0/1348812532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0bd0156120 0x7f0bd0165990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:23.192 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:23.189+0000 7f0bcffff700 1 --2- 172.21.15.77:0/1348812532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0bd006d3f0 0x7f0bd00c5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:23.192 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:23.189+0000 7f0bcffff700 1 --2- 172.21.15.77:0/1348812532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0bd00c5930 0x7f0bd0165450 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:23.193 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:23.189+0000 7f0bcffff700 1 -- 172.21.15.77:0/1348812532 >> 172.21.15.77:0/1348812532 conn(0x7f0bd00cb720 msgr2=0x7f0bd0154710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:23.193 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:23.189+0000 7f0bcffff700 1 -- 172.21.15.77:0/1348812532 shutdown_connections 2022-04-03T19:50:23.193 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:23.189+0000 7f0bcffff700 1 -- 172.21.15.77:0/1348812532 wait complete. 2022-04-03T19:50:23.193 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 24 2022-04-03T19:50:23.214 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:50:25.216 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:50:25.421 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.420+0000 7fde0e33d700 1 Processor -- start 2022-04-03T19:50:25.422 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.420+0000 7fde0e33d700 1 -- start start 2022-04-03T19:50:25.423 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.421+0000 7fde0e33d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fde0806bbe0 0x7fde081584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:25.423 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.421+0000 7fde0e33d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fde0806c500 0x7fde0814eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:25.423 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.421+0000 7fde0e33d700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0806d260 con 0x7fde0806bfb0 2022-04-03T19:50:25.424 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.421+0000 7fde0e33d700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde08159450 con 0x7fde0806c500 2022-04-03T19:50:25.424 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.421+0000 7fde0e33d700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde080c6330 con 0x7fde0806bbe0 2022-04-03T19:50:25.424 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.422+0000 7fde0db3c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fde0806c500 0x7fde0814eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:25.424 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.422+0000 7fde0d33b700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fde0806bbe0 0x7fde081584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:25.425 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.422+0000 7fde0d33b700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fde0806bbe0 0x7fde081584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38144/0 (socket says 172.21.15.77:38144) 2022-04-03T19:50:25.425 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.422+0000 7fde0d33b700 1 -- 172.21.15.77:0/4276882039 learned_addr learned my addr 172.21.15.77:0/4276882039 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:50:25.425 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.422+0000 7fde0cb3a700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fde0806bfb0 0x7fde0806d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44262/0 (socket says 172.21.15.77:44262) 2022-04-03T19:50:25.428 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.422+0000 7fde0d33b700 1 -- 172.21.15.77:0/4276882039 >> v2:172.21.15.77:3300/0 conn(0x7fde0806c500 msgr2=0x7fde0814eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:50:25.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.422+0000 7fde0d33b700 1 --2- 172.21.15.77:0/4276882039 >> v2:172.21.15.77:3300/0 conn(0x7fde0806c500 0x7fde0814eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:25.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.422+0000 7fde0d33b700 1 -- 172.21.15.77:0/4276882039 >> v1:172.21.15.186:6789/0 conn(0x7fde0806bfb0 legacy=0x7fde0806d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:25.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.422+0000 7fde0d33b700 1 -- 172.21.15.77:0/4276882039 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde08158bb0 con 0x7fde0806bbe0 2022-04-03T19:50:25.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.422+0000 7fde0d33b700 1 --2- 172.21.15.77:0/4276882039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde0806bbe0 0x7fde081584b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fde08088d40 tx=0x7fde040090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7572ed07c2855d08 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:25.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.423+0000 7fddf7fff700 1 -- 172.21.15.77:0/4276882039 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04004030 con 0x7fde0806bbe0 2022-04-03T19:50:25.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.423+0000 7fddf7fff700 1 -- 172.21.15.77:0/4276882039 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde04004190 con 0x7fde0806bbe0 2022-04-03T19:50:25.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.423+0000 7fddf7fff700 1 -- 172.21.15.77:0/4276882039 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04011630 con 0x7fde0806bbe0 2022-04-03T19:50:25.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.423+0000 7fde0e33d700 1 -- 172.21.15.77:0/4276882039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde0806bbe0 msgr2=0x7fde081584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:25.430 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.423+0000 7fde0e33d700 1 --2- 172.21.15.77:0/4276882039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde0806bbe0 0x7fde081584b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fde08088d40 tx=0x7fde040090f0 comp rx=0 tx=0).stop 2022-04-03T19:50:25.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.423+0000 7fde0e33d700 1 -- 172.21.15.77:0/4276882039 shutdown_connections 2022-04-03T19:50:25.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.423+0000 7fde0e33d700 1 --2- 172.21.15.77:0/4276882039 >> v2:172.21.15.77:3300/0 conn(0x7fde0806c500 0x7fde0814eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:25.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.423+0000 7fde0e33d700 1 --2- 172.21.15.77:0/4276882039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde0806bbe0 0x7fde081584b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:25.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.423+0000 7fde0e33d700 1 -- 172.21.15.77:0/4276882039 >> 172.21.15.77:0/4276882039 conn(0x7fde080cb6b0 msgr2=0x7fde080cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:25.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.423+0000 7fde0e33d700 1 -- 172.21.15.77:0/4276882039 shutdown_connections 2022-04-03T19:50:25.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.424+0000 7fde0e33d700 1 -- 172.21.15.77:0/4276882039 wait complete. 2022-04-03T19:50:25.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.425+0000 7fde0e33d700 1 Processor -- start 2022-04-03T19:50:25.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.425+0000 7fde0e33d700 1 -- start start 2022-04-03T19:50:25.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.426+0000 7fde0e33d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde0806bbe0 0x7fde08166490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:25.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.426+0000 7fde0e33d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde0806bfb0 0x7fde08154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:25.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.426+0000 7fde0e33d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde0806c500 0x7fde08168f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:25.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.426+0000 7fde0e33d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde080bf830 con 0x7fde0806bbe0 2022-04-03T19:50:25.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.426+0000 7fde0e33d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde081545f0 con 0x7fde0806c500 2022-04-03T19:50:25.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.426+0000 7fde0e33d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde08155770 con 0x7fde0806bfb0 2022-04-03T19:50:25.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.426+0000 7fde0d33b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde0806bbe0 0x7fde08166490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:25.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.426+0000 7fde0cb3a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde0806bfb0 0x7fde08154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:25.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.426+0000 7fde0db3c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde0806c500 0x7fde08168f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:25.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.426+0000 7fde0d33b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde0806bbe0 0x7fde08166490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42960/0 (socket says 172.21.15.77:42960) 2022-04-03T19:50:25.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.426+0000 7fde0d33b700 1 -- 172.21.15.77:0/123380820 learned_addr learned my addr 172.21.15.77:0/123380820 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:50:25.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.427+0000 7fde0db3c700 1 -- 172.21.15.77:0/123380820 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde0806bfb0 msgr2=0x7fde08154a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:25.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.427+0000 7fde0db3c700 1 --2- 172.21.15.77:0/123380820 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde0806bfb0 0x7fde08154a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:25.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.427+0000 7fde0db3c700 1 -- 172.21.15.77:0/123380820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde0806bbe0 msgr2=0x7fde08166490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:25.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.427+0000 7fde0db3c700 1 --2- 172.21.15.77:0/123380820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde0806bbe0 0x7fde08166490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:25.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.427+0000 7fde0db3c700 1 -- 172.21.15.77:0/123380820 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde04009370 con 0x7fde0806c500 2022-04-03T19:50:25.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.427+0000 7fde0d33b700 1 --2- 172.21.15.77:0/123380820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde0806bbe0 0x7fde08166490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:50:25.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.427+0000 7fde0db3c700 1 --2- 172.21.15.77:0/123380820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde0806c500 0x7fde08168f60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fde00009e80 tx=0x7fde0000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:25.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.427+0000 7fddf57fa700 1 -- 172.21.15.77:0/123380820 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0000caa0 con 0x7fde0806c500 2022-04-03T19:50:25.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.427+0000 7fddf57fa700 1 -- 172.21.15.77:0/123380820 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0000f040 con 0x7fde0806c500 2022-04-03T19:50:25.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.428+0000 7fddf57fa700 1 -- 172.21.15.77:0/123380820 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0000ce50 con 0x7fde0806c500 2022-04-03T19:50:25.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.428+0000 7fde0e33d700 1 -- 172.21.15.77:0/123380820 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde08155a50 con 0x7fde0806c500 2022-04-03T19:50:25.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.428+0000 7fde0e33d700 1 -- 172.21.15.77:0/123380820 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde08154280 con 0x7fde0806c500 2022-04-03T19:50:25.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.429+0000 7fde0e33d700 1 -- 172.21.15.77:0/123380820 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde08154780 con 0x7fde0806c500 2022-04-03T19:50:25.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.429+0000 7fddf57fa700 1 -- 172.21.15.77:0/123380820 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fde00014690 con 0x7fde0806c500 2022-04-03T19:50:25.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.430+0000 7fddf57fa700 1 --2- 172.21.15.77:0/123380820 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdde8063ce0 0x7fdde8066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:25.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.430+0000 7fddf57fa700 1 -- 172.21.15.77:0/123380820 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7fde00081890 con 0x7fde0806c500 2022-04-03T19:50:25.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.431+0000 7fde0d33b700 1 --2- 172.21.15.77:0/123380820 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdde8063ce0 0x7fdde8066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:25.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.431+0000 7fde0d33b700 1 --2- 172.21.15.77:0/123380820 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdde8063ce0 0x7fdde8066190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fde04009310 tx=0x7fde04016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:25.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.435+0000 7fddf57fa700 1 -- 172.21.15.77:0/123380820 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fde00050280 con 0x7fde0806c500 2022-04-03T19:50:25.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.663+0000 7fde0e33d700 1 -- 172.21.15.77:0/123380820 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde080894e0 con 0x7fde0806c500 2022-04-03T19:50:25.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.664+0000 7fddf57fa700 1 -- 172.21.15.77:0/123380820 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5356 (secure 0 0 0) 0x7fde000538a0 con 0x7fde0806c500 2022-04-03T19:50:25.666 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:50:25.666 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:50:23.308013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:50:25.668 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.666+0000 7fde0e33d700 1 -- 172.21.15.77:0/123380820 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdde8063ce0 msgr2=0x7fdde8066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:25.668 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.666+0000 7fde0e33d700 1 --2- 172.21.15.77:0/123380820 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdde8063ce0 0x7fdde8066190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fde04009310 tx=0x7fde04016000 comp rx=0 tx=0).stop 2022-04-03T19:50:25.668 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.666+0000 7fde0e33d700 1 -- 172.21.15.77:0/123380820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde0806c500 msgr2=0x7fde08168f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:25.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.666+0000 7fde0e33d700 1 --2- 172.21.15.77:0/123380820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde0806c500 0x7fde08168f60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fde00009e80 tx=0x7fde0000b040 comp rx=0 tx=0).stop 2022-04-03T19:50:25.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.666+0000 7fde0e33d700 1 -- 172.21.15.77:0/123380820 shutdown_connections 2022-04-03T19:50:25.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.666+0000 7fde0e33d700 1 --2- 172.21.15.77:0/123380820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde0806c500 0x7fde08168f60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:25.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.666+0000 7fde0e33d700 1 --2- 172.21.15.77:0/123380820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fde0806bbe0 0x7fde08166490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:25.670 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.666+0000 7fde0e33d700 1 --2- 172.21.15.77:0/123380820 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdde8063ce0 0x7fdde8066190 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:25.670 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.666+0000 7fde0e33d700 1 --2- 172.21.15.77:0/123380820 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde0806bfb0 0x7fde08154a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:25.670 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.666+0000 7fde0e33d700 1 -- 172.21.15.77:0/123380820 >> 172.21.15.77:0/123380820 conn(0x7fde080cb6b0 msgr2=0x7fde0814ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:25.670 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.666+0000 7fde0e33d700 1 -- 172.21.15.77:0/123380820 shutdown_connections 2022-04-03T19:50:25.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:25.667+0000 7fde0e33d700 1 -- 172.21.15.77:0/123380820 wait complete. 2022-04-03T19:50:25.671 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 25 2022-04-03T19:50:25.693 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=4960 2022-04-03T19:50:25.693 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.e 2022-04-03T19:50:28.694 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2022-04-03T19:50:28.694 INFO:tasks.ceph.mds.e:Restarting daemon 2022-04-03T19:50:28.695 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2022-04-03T19:50:28.700 INFO:tasks.ceph.mds.e:Started 2022-04-03T19:50:28.700 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:50:28.785 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/179/19' 2022-04-03T19:50:28.785 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/179/15' 2022-04-03T19:50:28.786 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/179/20' 2022-04-03T19:50:28.786 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/179/12' 2022-04-03T19:50:28.786 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/179/10' 2022-04-03T19:50:28.787 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/179/11' 2022-04-03T19:50:28.787 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/179/16' 2022-04-03T19:50:28.787 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/179/14' 2022-04-03T19:50:28.787 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/179/18' 2022-04-03T19:50:28.787 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/179/13' 2022-04-03T19:50:28.788 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/179/17' 2022-04-03T19:50:28.788 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/179' 2022-04-03T19:50:28.788 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/132/19' 2022-04-03T19:50:28.788 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/132/15' 2022-04-03T19:50:28.788 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/132/20' 2022-04-03T19:50:28.789 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/132/12' 2022-04-03T19:50:28.789 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/132/10' 2022-04-03T19:50:28.789 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/132/11' 2022-04-03T19:50:28.789 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/132/16' 2022-04-03T19:50:28.790 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/132/14' 2022-04-03T19:50:28.790 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/132/18' 2022-04-03T19:50:28.790 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/132/13' 2022-04-03T19:50:28.790 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/132/17' 2022-04-03T19:50:28.791 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/132' 2022-04-03T19:50:28.791 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/199/19' 2022-04-03T19:50:28.791 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/199/15' 2022-04-03T19:50:28.791 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/199/20' 2022-04-03T19:50:28.792 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/199/12' 2022-04-03T19:50:28.792 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/199/10' 2022-04-03T19:50:28.792 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/199/11' 2022-04-03T19:50:28.792 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/199/16' 2022-04-03T19:50:28.792 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/199/14' 2022-04-03T19:50:28.793 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/199/18' 2022-04-03T19:50:28.793 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/199/13' 2022-04-03T19:50:28.793 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/199/17' 2022-04-03T19:50:28.793 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/199' 2022-04-03T19:50:28.793 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/163/19' 2022-04-03T19:50:28.794 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/163/15' 2022-04-03T19:50:28.794 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/163/20' 2022-04-03T19:50:28.794 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/163/12' 2022-04-03T19:50:28.794 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/163/10' 2022-04-03T19:50:28.795 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/163/11' 2022-04-03T19:50:28.795 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/163/16' 2022-04-03T19:50:28.795 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/163/14' 2022-04-03T19:50:28.795 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/163/18' 2022-04-03T19:50:28.795 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/163/13' 2022-04-03T19:50:28.796 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/163/17' 2022-04-03T19:50:28.796 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/163' 2022-04-03T19:50:28.796 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/153/19' 2022-04-03T19:50:28.796 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/153/15' 2022-04-03T19:50:28.796 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/153/20' 2022-04-03T19:50:28.797 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/153/12' 2022-04-03T19:50:28.797 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/153/10' 2022-04-03T19:50:28.797 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/153/11' 2022-04-03T19:50:28.797 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/153/16' 2022-04-03T19:50:28.798 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/153/14' 2022-04-03T19:50:28.798 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/153/18' 2022-04-03T19:50:28.798 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/153/13' 2022-04-03T19:50:28.798 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/153/17' 2022-04-03T19:50:28.798 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/153' 2022-04-03T19:50:28.799 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/122/19' 2022-04-03T19:50:28.799 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/122/15' 2022-04-03T19:50:28.799 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/122/20' 2022-04-03T19:50:28.799 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/122/12' 2022-04-03T19:50:28.800 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/122/10' 2022-04-03T19:50:28.800 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/122/11' 2022-04-03T19:50:28.800 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/122/16' 2022-04-03T19:50:28.800 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/122/14' 2022-04-03T19:50:28.800 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/122/18' 2022-04-03T19:50:28.801 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/122/13' 2022-04-03T19:50:28.801 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/122/17' 2022-04-03T19:50:28.801 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/122' 2022-04-03T19:50:28.801 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/162/19' 2022-04-03T19:50:28.802 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/162/15' 2022-04-03T19:50:28.802 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/162/20' 2022-04-03T19:50:28.802 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/162/12' 2022-04-03T19:50:28.802 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/162/10' 2022-04-03T19:50:28.802 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/162/11' 2022-04-03T19:50:28.803 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/162/16' 2022-04-03T19:50:28.803 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/162/14' 2022-04-03T19:50:28.803 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/162/18' 2022-04-03T19:50:28.803 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/162/13' 2022-04-03T19:50:28.803 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/162/17' 2022-04-03T19:50:28.804 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/162' 2022-04-03T19:50:28.804 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/197/19' 2022-04-03T19:50:28.804 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/197/15' 2022-04-03T19:50:28.804 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/197/20' 2022-04-03T19:50:28.804 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/197/12' 2022-04-03T19:50:28.805 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/197/10' 2022-04-03T19:50:28.805 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/197/11' 2022-04-03T19:50:28.805 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/197/16' 2022-04-03T19:50:28.805 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/197/14' 2022-04-03T19:50:28.805 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/197/18' 2022-04-03T19:50:28.806 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/197/13' 2022-04-03T19:50:28.806 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/197/17' 2022-04-03T19:50:28.806 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/197' 2022-04-03T19:50:28.806 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/113/19' 2022-04-03T19:50:28.806 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/113/15' 2022-04-03T19:50:28.807 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/113/20' 2022-04-03T19:50:28.807 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/113/12' 2022-04-03T19:50:28.807 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/113/10' 2022-04-03T19:50:28.807 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/113/11' 2022-04-03T19:50:28.808 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/113/16' 2022-04-03T19:50:28.808 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/113/14' 2022-04-03T19:50:28.808 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/113/18' 2022-04-03T19:50:28.808 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/113/13' 2022-04-03T19:50:28.808 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/113/17' 2022-04-03T19:50:28.809 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/113' 2022-04-03T19:50:28.809 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/152/19' 2022-04-03T19:50:28.860 INFO:tasks.ceph.mds.e.smithi077.stdout:starting mds.e at 2022-04-03T19:50:28.915 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.913+0000 7f3fcf130700 1 Processor -- start 2022-04-03T19:50:28.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.914+0000 7f3fcf130700 1 -- start start 2022-04-03T19:50:28.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.915+0000 7f3fcf130700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3fc80c5930 0x7f3fc8155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:28.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.915+0000 7f3fcf130700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc80c2f80 con 0x7f3fc806d3f0 2022-04-03T19:50:28.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.915+0000 7f3fcf130700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc80c3260 con 0x7f3fc8156120 2022-04-03T19:50:28.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.915+0000 7f3fcf130700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fc80c3540 con 0x7f3fc80c5930 2022-04-03T19:50:28.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.915+0000 7f3fce12e700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3fc806d3f0 0x7f3fc806d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38476/0 (socket says 172.21.15.77:38476) 2022-04-03T19:50:28.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.915+0000 7f3fce12e700 1 -- 172.21.15.77:0/1810354287 learned_addr learned my addr 172.21.15.77:0/1810354287 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:50:28.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.915+0000 7f3fcd92d700 1 --2- 172.21.15.77:0/1810354287 >> v2:172.21.15.77:3300/0 conn(0x7f3fc80c5930 0x7f3fc8155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:28.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.915+0000 7f3fcd12c700 1 -- 172.21.15.77:0/1810354287 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670575644 0 0) 0x7f3fc80c3260 con 0x7f3fc8156120 2022-04-03T19:50:28.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.916+0000 7f3fcd12c700 1 -- 172.21.15.77:0/1810354287 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fb00033f0 con 0x7f3fc8156120 2022-04-03T19:50:28.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.916+0000 7f3fcd92d700 1 -- 172.21.15.77:0/1810354287 >> v1:172.21.15.186:6789/0 conn(0x7f3fc8156120 legacy=0x7f3fc806c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:28.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.916+0000 7f3fcd92d700 1 -- 172.21.15.77:0/1810354287 >> v1:172.21.15.77:6789/0 conn(0x7f3fc806d3f0 legacy=0x7f3fc806d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:28.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.916+0000 7f3fcd92d700 1 -- 172.21.15.77:0/1810354287 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc80c3820 con 0x7f3fc80c5930 2022-04-03T19:50:28.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.916+0000 7f3fcd92d700 1 --2- 172.21.15.77:0/1810354287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fc80c5930 0x7f3fc8155be0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8009790 tx=0x7f3fb8009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7145ccecf60c3ff1 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:28.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.916+0000 7f3fcd12c700 1 -- 172.21.15.77:0/1810354287 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8011630 con 0x7f3fc80c5930 2022-04-03T19:50:28.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.916+0000 7f3fcd12c700 1 -- 172.21.15.77:0/1810354287 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb8011790 con 0x7f3fc80c5930 2022-04-03T19:50:28.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.916+0000 7f3fcd12c700 1 -- 172.21.15.77:0/1810354287 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8011b30 con 0x7f3fc80c5930 2022-04-03T19:50:28.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.917+0000 7f3fcf130700 1 -- 172.21.15.77:0/1810354287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fc80c5930 msgr2=0x7f3fc8155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:28.931 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.917+0000 7f3fcf130700 1 --2- 172.21.15.77:0/1810354287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fc80c5930 0x7f3fc8155be0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8009790 tx=0x7f3fb8009060 comp rx=0 tx=0).stop 2022-04-03T19:50:28.931 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.917+0000 7f3fcf130700 1 -- 172.21.15.77:0/1810354287 shutdown_connections 2022-04-03T19:50:28.931 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.917+0000 7f3fcf130700 1 --2- 172.21.15.77:0/1810354287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fc80c5930 0x7f3fc8155be0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:28.932 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.917+0000 7f3fcf130700 1 -- 172.21.15.77:0/1810354287 >> 172.21.15.77:0/1810354287 conn(0x7f3fc80cb720 msgr2=0x7f3fc80cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:28.932 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.917+0000 7f3fcf130700 1 -- 172.21.15.77:0/1810354287 shutdown_connections 2022-04-03T19:50:28.932 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.917+0000 7f3fcf130700 1 -- 172.21.15.77:0/1810354287 wait complete. 2022-04-03T19:50:28.933 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.918+0000 7f3fcf130700 1 Processor -- start 2022-04-03T19:50:28.933 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.919+0000 7f3fcf130700 1 -- start start 2022-04-03T19:50:28.934 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.919+0000 7f3fcf130700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fc806d3f0 0x7f3fc814ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:28.934 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.919+0000 7f3fcf130700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3fc80c5930 0x7f3fc8157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:28.934 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.919+0000 7f3fcf130700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fc8156120 0x7f3fc8168d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:28.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.919+0000 7f3fcf130700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc8152240 con 0x7f3fc8156120 2022-04-03T19:50:28.935 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.919+0000 7f3fcf130700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc8151690 con 0x7f3fc806d3f0 2022-04-03T19:50:28.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.919+0000 7f3fcf130700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc8169a70 con 0x7f3fc80c5930 2022-04-03T19:50:28.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.920+0000 7f3fce12e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fc806d3f0 0x7f3fc814ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:28.936 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.920+0000 7f3fcd92d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3fc80c5930 0x7f3fc8157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:28.937 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.920+0000 7f3fce12e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fc806d3f0 0x7f3fc814ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38152/0 (socket says 172.21.15.77:38152) 2022-04-03T19:50:28.937 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.920+0000 7f3fce12e700 1 -- 172.21.15.77:0/1106698731 learned_addr learned my addr 172.21.15.77:0/1106698731 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:50:28.938 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.920+0000 7f3fce92f700 1 --2- 172.21.15.77:0/1106698731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fc8156120 0x7f3fc8168d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:28.938 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.920+0000 7f3fce12e700 1 -- 172.21.15.77:0/1106698731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3fc80c5930 msgr2=0x7f3fc8157c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:28.938 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.920+0000 7f3fce12e700 1 --2- 172.21.15.77:0/1106698731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3fc80c5930 0x7f3fc8157c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:28.939 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.920+0000 7f3fce12e700 1 -- 172.21.15.77:0/1106698731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fc8156120 msgr2=0x7f3fc8168d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:28.939 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.920+0000 7f3fce12e700 1 --2- 172.21.15.77:0/1106698731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fc8156120 0x7f3fc8168d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:28.940 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.920+0000 7f3fce12e700 1 -- 172.21.15.77:0/1106698731 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb80092a0 con 0x7f3fc806d3f0 2022-04-03T19:50:28.940 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.920+0000 7f3fcd92d700 1 --2- 172.21.15.77:0/1106698731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3fc80c5930 0x7f3fc8157c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:50:28.940 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.921+0000 7f3fce12e700 1 --2- 172.21.15.77:0/1106698731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fc806d3f0 0x7f3fc814ac30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3fc400c490 tx=0x7f3fc4007df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:28.941 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.921+0000 7f3fb67fc700 1 -- 172.21.15.77:0/1106698731 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc400ce90 con 0x7f3fc806d3f0 2022-04-03T19:50:28.941 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.921+0000 7f3fb67fc700 1 -- 172.21.15.77:0/1106698731 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc40072e0 con 0x7f3fc806d3f0 2022-04-03T19:50:28.941 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.921+0000 7f3fcf130700 1 -- 172.21.15.77:0/1106698731 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc81518a0 con 0x7f3fc806d3f0 2022-04-03T19:50:28.942 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.921+0000 7f3fb67fc700 1 -- 172.21.15.77:0/1106698731 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4007660 con 0x7f3fc806d3f0 2022-04-03T19:50:28.942 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.921+0000 7f3fcf130700 1 -- 172.21.15.77:0/1106698731 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc8151e60 con 0x7f3fc806d3f0 2022-04-03T19:50:28.942 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.922+0000 7f3fcf130700 1 -- 172.21.15.77:0/1106698731 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc804b050 con 0x7f3fc806d3f0 2022-04-03T19:50:28.944 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.923+0000 7f3fb67fc700 1 -- 172.21.15.77:0/1106698731 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f3fc4019030 con 0x7f3fc806d3f0 2022-04-03T19:50:28.945 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.924+0000 7f3fb67fc700 1 --2- 172.21.15.77:0/1106698731 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3fa8063d10 0x7f3fa80661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:28.945 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.924+0000 7f3fb67fc700 1 -- 172.21.15.77:0/1106698731 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7f3fc407fb10 con 0x7f3fc806d3f0 2022-04-03T19:50:28.945 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.924+0000 7f3fcd92d700 1 --2- 172.21.15.77:0/1106698731 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3fa8063d10 0x7f3fa80661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:28.946 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.925+0000 7f3fcd92d700 1 --2- 172.21.15.77:0/1106698731 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3fa8063d10 0x7f3fa80661c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8006d30 tx=0x7f3fb8017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:28.946 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:28.930+0000 7f3fb67fc700 1 -- 172.21.15.77:0/1106698731 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f3fc4048ae0 con 0x7f3fc806d3f0 2022-04-03T19:50:29.163 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:29.162+0000 7f3fcf130700 1 -- 172.21.15.77:0/1106698731 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fc8155400 con 0x7f3fc806d3f0 2022-04-03T19:50:29.164 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:29.162+0000 7f3fb67fc700 1 -- 172.21.15.77:0/1106698731 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5356 (secure 0 0 0) 0x7f3fc4017020 con 0x7f3fc806d3f0 2022-04-03T19:50:29.164 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:50:29.164 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:50:23.308013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:50:29.167 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:29.165+0000 7f3fcf130700 1 -- 172.21.15.77:0/1106698731 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3fa8063d10 msgr2=0x7f3fa80661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:29.167 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:29.165+0000 7f3fcf130700 1 --2- 172.21.15.77:0/1106698731 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3fa8063d10 0x7f3fa80661c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8006d30 tx=0x7f3fb8017000 comp rx=0 tx=0).stop 2022-04-03T19:50:29.168 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:29.165+0000 7f3fcf130700 1 -- 172.21.15.77:0/1106698731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fc806d3f0 msgr2=0x7f3fc814ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:29.168 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:29.165+0000 7f3fcf130700 1 --2- 172.21.15.77:0/1106698731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fc806d3f0 0x7f3fc814ac30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3fc400c490 tx=0x7f3fc4007df0 comp rx=0 tx=0).stop 2022-04-03T19:50:29.168 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:29.165+0000 7f3fcf130700 1 -- 172.21.15.77:0/1106698731 shutdown_connections 2022-04-03T19:50:29.168 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:29.165+0000 7f3fcf130700 1 --2- 172.21.15.77:0/1106698731 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3fc8156120 0x7f3fc8168d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:29.169 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:29.165+0000 7f3fcf130700 1 --2- 172.21.15.77:0/1106698731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fc806d3f0 0x7f3fc814ac30 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:29.169 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:29.165+0000 7f3fcf130700 1 --2- 172.21.15.77:0/1106698731 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3fa8063d10 0x7f3fa80661c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:29.169 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:29.165+0000 7f3fcf130700 1 --2- 172.21.15.77:0/1106698731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3fc80c5930 0x7f3fc8157c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:29.169 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:29.165+0000 7f3fcf130700 1 -- 172.21.15.77:0/1106698731 >> 172.21.15.77:0/1106698731 conn(0x7f3fc80cb720 msgr2=0x7f3fc806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:29.169 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:29.165+0000 7f3fcf130700 1 -- 172.21.15.77:0/1106698731 shutdown_connections 2022-04-03T19:50:29.170 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:29.165+0000 7f3fcf130700 1 -- 172.21.15.77:0/1106698731 wait complete. 2022-04-03T19:50:29.170 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 25 2022-04-03T19:50:29.190 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2022-04-03T19:50:30.198 INFO:tasks.ceph.mds.d.smithi186.stderr:2022-04-03T19:50:30.198+0000 7f193547a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-04-03T19:50:31.192 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:50:31.397 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.396+0000 7f006b59e700 1 Processor -- start 2022-04-03T19:50:31.398 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.396+0000 7f006b59e700 1 -- start start 2022-04-03T19:50:31.401 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.397+0000 7f006b59e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f006c06bbe0 0x7f006c1584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:31.401 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.397+0000 7f006b59e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f006c06d260 con 0x7f006c06c500 2022-04-03T19:50:31.401 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.397+0000 7f006b59e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f006c0c6330 con 0x7f006c06bfb0 2022-04-03T19:50:31.401 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.397+0000 7f006b59e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f006c159450 con 0x7f006c06bbe0 2022-04-03T19:50:31.402 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.397+0000 7f006ad9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f006c06c500 0x7f006c14eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38478/0 (socket says 172.21.15.77:38478) 2022-04-03T19:50:31.402 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.397+0000 7f006ad9d700 1 -- 172.21.15.77:0/1076005250 learned_addr learned my addr 172.21.15.77:0/1076005250 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:50:31.402 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.397+0000 7f006a59c700 1 --2- 172.21.15.77:0/1076005250 >> v2:172.21.15.186:3300/0 conn(0x7f006c06bbe0 0x7f006c1584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:31.402 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.398+0000 7f0069d9b700 1 -- 172.21.15.77:0/1076005250 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599260692 0 0) 0x7f006c06d260 con 0x7f006c06c500 2022-04-03T19:50:31.403 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.398+0000 7f0069d9b700 1 -- 172.21.15.77:0/1076005250 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f004c0033f0 con 0x7f006c06c500 2022-04-03T19:50:31.403 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.398+0000 7f0069d9b700 1 -- 172.21.15.77:0/1076005250 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591174801 0 0) 0x7f006c0c6330 con 0x7f006c06bfb0 2022-04-03T19:50:31.403 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.398+0000 7f0069d9b700 1 -- 172.21.15.77:0/1076005250 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f006c06d260 con 0x7f006c06bfb0 2022-04-03T19:50:31.403 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.398+0000 7f006a59c700 1 -- 172.21.15.77:0/1076005250 >> v1:172.21.15.186:6789/0 conn(0x7f006c06bfb0 legacy=0x7f006c06d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:31.404 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.398+0000 7f006a59c700 1 -- 172.21.15.77:0/1076005250 >> v1:172.21.15.77:6789/0 conn(0x7f006c06c500 legacy=0x7f006c14eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:31.404 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.398+0000 7f006a59c700 1 -- 172.21.15.77:0/1076005250 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c158bb0 con 0x7f006c06bbe0 2022-04-03T19:50:31.404 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.398+0000 7f006a59c700 1 --2- 172.21.15.77:0/1076005250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f006c06bbe0 0x7f006c1584b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f005c009760 tx=0x7f005c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3f4c4dac580c4ae server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:31.404 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.399+0000 7f0069d9b700 1 -- 172.21.15.77:0/1076005250 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c0105b0 con 0x7f006c06bbe0 2022-04-03T19:50:31.405 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.399+0000 7f0069d9b700 1 -- 172.21.15.77:0/1076005250 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005c010710 con 0x7f006c06bbe0 2022-04-03T19:50:31.405 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.399+0000 7f0069d9b700 1 -- 172.21.15.77:0/1076005250 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c010a90 con 0x7f006c06bbe0 2022-04-03T19:50:31.405 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.399+0000 7f006b59e700 1 -- 172.21.15.77:0/1076005250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f006c06bbe0 msgr2=0x7f006c1584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:31.406 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.399+0000 7f006b59e700 1 --2- 172.21.15.77:0/1076005250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f006c06bbe0 0x7f006c1584b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f005c009760 tx=0x7f005c009060 comp rx=0 tx=0).stop 2022-04-03T19:50:31.406 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.399+0000 7f006b59e700 1 -- 172.21.15.77:0/1076005250 shutdown_connections 2022-04-03T19:50:31.406 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.399+0000 7f006b59e700 1 --2- 172.21.15.77:0/1076005250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f006c06bbe0 0x7f006c1584b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:31.406 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.399+0000 7f006b59e700 1 -- 172.21.15.77:0/1076005250 >> 172.21.15.77:0/1076005250 conn(0x7f006c0cb6b0 msgr2=0x7f006c0cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:31.407 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.399+0000 7f006b59e700 1 -- 172.21.15.77:0/1076005250 shutdown_connections 2022-04-03T19:50:31.407 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.399+0000 7f006b59e700 1 -- 172.21.15.77:0/1076005250 wait complete. 2022-04-03T19:50:31.410 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.401+0000 7f006b59e700 1 Processor -- start 2022-04-03T19:50:31.410 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.401+0000 7f006b59e700 1 -- start start 2022-04-03T19:50:31.410 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.402+0000 7f006b59e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f006c06bbe0 0x7f006c0bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:31.411 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.402+0000 7f006b59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f006c06bfb0 0x7f006c148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:31.411 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.402+0000 7f006b59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f006c06c500 0x7f006c148d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:31.411 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.402+0000 7f006b59e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c1553e0 con 0x7f006c06c500 2022-04-03T19:50:31.411 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.402+0000 7f006b59e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c0bf960 con 0x7f006c06bbe0 2022-04-03T19:50:31.411 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.402+0000 7f006b59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f006c154440 con 0x7f006c06bfb0 2022-04-03T19:50:31.412 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.402+0000 7f006a59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f006c06bbe0 0x7f006c0bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:31.412 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.402+0000 7f006a59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f006c06bbe0 0x7f006c0bfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38156/0 (socket says 172.21.15.77:38156) 2022-04-03T19:50:31.412 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.402+0000 7f006a59c700 1 -- 172.21.15.77:0/2763568947 learned_addr learned my addr 172.21.15.77:0/2763568947 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:50:31.412 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.402+0000 7f0063fff700 1 --2- 172.21.15.77:0/2763568947 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f006c06bfb0 0x7f006c148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:31.413 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.402+0000 7f006ad9d700 1 --2- 172.21.15.77:0/2763568947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f006c06c500 0x7f006c148d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:31.413 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.403+0000 7f006a59c700 1 -- 172.21.15.77:0/2763568947 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f006c06bfb0 msgr2=0x7f006c148830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:31.413 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.403+0000 7f006a59c700 1 --2- 172.21.15.77:0/2763568947 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f006c06bfb0 0x7f006c148830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:31.413 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.403+0000 7f006a59c700 1 -- 172.21.15.77:0/2763568947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f006c06c500 msgr2=0x7f006c148d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:31.413 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.403+0000 7f006a59c700 1 --2- 172.21.15.77:0/2763568947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f006c06c500 0x7f006c148d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:31.414 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.403+0000 7f006a59c700 1 -- 172.21.15.77:0/2763568947 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005c0092a0 con 0x7f006c06bbe0 2022-04-03T19:50:31.414 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.403+0000 7f006a59c700 1 --2- 172.21.15.77:0/2763568947 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f006c06bbe0 0x7f006c0bfc20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f005c009d50 tx=0x7f005c016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:31.414 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.403+0000 7f00627fc700 1 -- 172.21.15.77:0/2763568947 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c000bc0 con 0x7f006c06bbe0 2022-04-03T19:50:31.415 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.403+0000 7f00627fc700 1 -- 172.21.15.77:0/2763568947 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005c020070 con 0x7f006c06bbe0 2022-04-03T19:50:31.415 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.403+0000 7f006b59e700 1 -- 172.21.15.77:0/2763568947 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f006c154e80 con 0x7f006c06bbe0 2022-04-03T19:50:31.415 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.403+0000 7f00627fc700 1 -- 172.21.15.77:0/2763568947 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c025430 con 0x7f006c06bbe0 2022-04-03T19:50:31.415 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.404+0000 7f006b59e700 1 -- 172.21.15.77:0/2763568947 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f006c153df0 con 0x7f006c06bbe0 2022-04-03T19:50:31.415 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.404+0000 7f006b59e700 1 -- 172.21.15.77:0/2763568947 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f006c04b050 con 0x7f006c06bbe0 2022-04-03T19:50:31.416 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.405+0000 7f00627fc700 1 -- 172.21.15.77:0/2763568947 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f005c02b020 con 0x7f006c06bbe0 2022-04-03T19:50:31.416 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.406+0000 7f00627fc700 1 --2- 172.21.15.77:0/2763568947 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0050063d10 0x7f00500661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:50:31.416 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.406+0000 7f00627fc700 1 -- 172.21.15.77:0/2763568947 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6374+0+0 (secure 0 0 0) 0x7f005c082650 con 0x7f006c06bbe0 2022-04-03T19:50:31.416 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.407+0000 7f0063fff700 1 --2- 172.21.15.77:0/2763568947 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0050063d10 0x7f00500661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:50:31.417 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.407+0000 7f0063fff700 1 --2- 172.21.15.77:0/2763568947 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0050063d10 0x7f00500661c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f006c06d970 tx=0x7f0054009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:50:31.417 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.410+0000 7f00627fc700 1 -- 172.21.15.77:0/2763568947 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f005c051080 con 0x7f006c06bbe0 2022-04-03T19:50:31.645 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.643+0000 7f006b59e700 1 -- 172.21.15.77:0/2763568947 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f006c0c7ec0 con 0x7f006c06bbe0 2022-04-03T19:50:31.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.645+0000 7f00627fc700 1 -- 172.21.15.77:0/2763568947 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6113 (secure 0 0 0) 0x7f005c046090 con 0x7f006c06bbe0 2022-04-03T19:50:31.646 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:50:31.647 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5239,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:50:23.308013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:50:31.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.647+0000 7f006b59e700 1 -- 172.21.15.77:0/2763568947 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0050063d10 msgr2=0x7f00500661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:31.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.647+0000 7f006b59e700 1 --2- 172.21.15.77:0/2763568947 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0050063d10 0x7f00500661c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f006c06d970 tx=0x7f0054009040 comp rx=0 tx=0).stop 2022-04-03T19:50:31.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.647+0000 7f006b59e700 1 -- 172.21.15.77:0/2763568947 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f006c06bbe0 msgr2=0x7f006c0bfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:50:31.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.647+0000 7f006b59e700 1 --2- 172.21.15.77:0/2763568947 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f006c06bbe0 0x7f006c0bfc20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f005c009d50 tx=0x7f005c016000 comp rx=0 tx=0).stop 2022-04-03T19:50:31.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.647+0000 7f006b59e700 1 -- 172.21.15.77:0/2763568947 shutdown_connections 2022-04-03T19:50:31.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.647+0000 7f006b59e700 1 --2- 172.21.15.77:0/2763568947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f006c06c500 0x7f006c148d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:31.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.647+0000 7f006b59e700 1 --2- 172.21.15.77:0/2763568947 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f006c06bbe0 0x7f006c0bfc20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:31.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.647+0000 7f006b59e700 1 --2- 172.21.15.77:0/2763568947 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0050063d10 0x7f00500661c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:31.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.647+0000 7f006b59e700 1 --2- 172.21.15.77:0/2763568947 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f006c06bfb0 0x7f006c148830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:50:31.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.647+0000 7f006b59e700 1 -- 172.21.15.77:0/2763568947 >> 172.21.15.77:0/2763568947 conn(0x7f006c0cb6b0 msgr2=0x7f006c0cba80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:50:31.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.647+0000 7f006b59e700 1 -- 172.21.15.77:0/2763568947 shutdown_connections 2022-04-03T19:50:31.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:50:31.648+0000 7f006b59e700 1 -- 172.21.15.77:0/2763568947 wait complete. 2022-04-03T19:50:31.653 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2022-04-03T19:50:31.673 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2022-04-03T19:50:31.673 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 63 secs before thrashing 2022-04-03T19:50:31.682 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/152/15' 2022-04-03T19:50:31.683 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/152/20' 2022-04-03T19:50:31.683 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/152/12' 2022-04-03T19:50:31.683 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/152/10' 2022-04-03T19:50:31.683 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/152/11' 2022-04-03T19:50:31.684 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/152/16' 2022-04-03T19:50:31.684 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/152/14' 2022-04-03T19:50:31.684 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/152/18' 2022-04-03T19:50:31.684 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/152/13' 2022-04-03T19:50:31.684 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/152/17' 2022-04-03T19:50:31.685 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/152' 2022-04-03T19:50:31.685 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/121/19' 2022-04-03T19:50:31.685 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/121/15' 2022-04-03T19:50:31.685 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/121/20' 2022-04-03T19:50:31.686 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/121/12' 2022-04-03T19:50:31.686 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/121/10' 2022-04-03T19:50:31.686 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/121/11' 2022-04-03T19:50:31.686 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/121/16' 2022-04-03T19:50:31.686 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/121/14' 2022-04-03T19:50:31.687 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/121/18' 2022-04-03T19:50:31.687 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/121/13' 2022-04-03T19:50:31.687 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/121/17' 2022-04-03T19:50:31.687 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/121' 2022-04-03T19:50:31.688 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/118/19' 2022-04-03T19:50:31.688 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/118/15' 2022-04-03T19:50:31.688 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/118/20' 2022-04-03T19:50:31.688 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/118/12' 2022-04-03T19:50:31.688 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/118/10' 2022-04-03T19:50:31.689 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/118/11' 2022-04-03T19:50:31.689 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/118/16' 2022-04-03T19:50:31.689 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/118/14' 2022-04-03T19:50:31.689 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/118/18' 2022-04-03T19:50:31.689 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/118/13' 2022-04-03T19:50:31.690 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/118/17' 2022-04-03T19:50:31.690 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/118' 2022-04-03T19:50:31.690 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/196/19' 2022-04-03T19:50:31.690 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/196/15' 2022-04-03T19:50:31.691 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/196/20' 2022-04-03T19:50:31.691 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/196/12' 2022-04-03T19:50:31.691 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/196/10' 2022-04-03T19:50:31.691 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/196/11' 2022-04-03T19:50:31.691 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/196/16' 2022-04-03T19:50:31.692 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/196/14' 2022-04-03T19:50:31.692 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/196/18' 2022-04-03T19:50:31.692 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/196/13' 2022-04-03T19:50:31.692 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/196/17' 2022-04-03T19:50:31.693 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/196' 2022-04-03T19:50:31.693 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/124/19' 2022-04-03T19:50:31.693 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/124/15' 2022-04-03T19:50:31.693 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/124/20' 2022-04-03T19:50:31.693 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/124/12' 2022-04-03T19:50:31.694 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/124/10' 2022-04-03T19:50:31.694 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/124/11' 2022-04-03T19:50:31.694 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/124/16' 2022-04-03T19:50:31.694 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/124/14' 2022-04-03T19:50:31.695 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/124/18' 2022-04-03T19:50:31.695 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/124/13' 2022-04-03T19:50:31.695 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/124/17' 2022-04-03T19:50:31.695 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/124' 2022-04-03T19:50:31.695 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/188/19' 2022-04-03T19:50:31.696 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/188/15' 2022-04-03T19:50:31.696 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/188/20' 2022-04-03T19:50:31.696 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/188/12' 2022-04-03T19:50:31.696 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/188/10' 2022-04-03T19:50:31.697 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/188/11' 2022-04-03T19:50:31.697 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/188/16' 2022-04-03T19:50:31.697 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/188/14' 2022-04-03T19:50:31.697 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/188/18' 2022-04-03T19:50:31.698 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/188/13' 2022-04-03T19:50:31.698 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/188/17' 2022-04-03T19:50:31.698 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/188' 2022-04-03T19:50:31.698 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/145/19' 2022-04-03T19:50:31.698 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/145/15' 2022-04-03T19:50:31.699 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/145/20' 2022-04-03T19:50:31.699 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/145/12' 2022-04-03T19:50:31.699 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/145/10' 2022-04-03T19:50:31.700 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/145/11' 2022-04-03T19:50:31.700 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/145/16' 2022-04-03T19:50:31.700 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/145/14' 2022-04-03T19:50:31.700 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/145/18' 2022-04-03T19:50:31.701 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/145/13' 2022-04-03T19:50:31.701 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/145/17' 2022-04-03T19:50:31.701 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/145' 2022-04-03T19:50:31.701 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/130/19' 2022-04-03T19:50:31.701 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/130/15' 2022-04-03T19:50:31.702 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/130/20' 2022-04-03T19:50:31.702 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/130/12' 2022-04-03T19:50:31.702 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/130/10' 2022-04-03T19:50:31.702 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/130/11' 2022-04-03T19:50:31.703 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/130/16' 2022-04-03T19:50:31.703 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/130/14' 2022-04-03T19:50:31.703 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/130/18' 2022-04-03T19:50:31.703 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/130/13' 2022-04-03T19:50:31.703 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/130/17' 2022-04-03T19:50:31.704 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/130' 2022-04-03T19:50:31.704 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/193/19' 2022-04-03T19:50:31.704 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/193/15' 2022-04-03T19:50:31.704 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/193/20' 2022-04-03T19:50:31.704 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/193/12' 2022-04-03T19:50:31.705 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/193/10' 2022-04-03T19:50:31.705 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/193/11' 2022-04-03T19:50:31.705 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/193/16' 2022-04-03T19:50:31.705 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/193/14' 2022-04-03T19:50:31.705 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/193/18' 2022-04-03T19:50:31.706 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/193/13' 2022-04-03T19:50:31.706 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/193/17' 2022-04-03T19:50:31.706 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/193' 2022-04-03T19:50:31.706 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/175/19' 2022-04-03T19:50:31.707 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/175/15' 2022-04-03T19:50:34.726 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/175/20' 2022-04-03T19:50:34.727 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/175/12' 2022-04-03T19:50:34.727 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/175/10' 2022-04-03T19:50:34.727 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/175/11' 2022-04-03T19:50:34.727 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/175/16' 2022-04-03T19:50:34.728 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/175/14' 2022-04-03T19:50:34.728 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/175/18' 2022-04-03T19:50:34.728 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/175/13' 2022-04-03T19:50:34.728 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/175/17' 2022-04-03T19:50:34.728 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/175' 2022-04-03T19:50:34.729 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/181/19' 2022-04-03T19:50:34.729 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/181/15' 2022-04-03T19:50:34.729 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/181/20' 2022-04-03T19:50:34.730 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/181/12' 2022-04-03T19:50:34.730 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/181/10' 2022-04-03T19:50:34.730 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/181/11' 2022-04-03T19:50:34.730 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/181/16' 2022-04-03T19:50:34.731 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/181/14' 2022-04-03T19:50:34.731 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/181/18' 2022-04-03T19:50:34.731 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/181/13' 2022-04-03T19:50:34.732 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/181/17' 2022-04-03T19:50:34.732 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/181' 2022-04-03T19:50:34.732 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/127/19' 2022-04-03T19:50:34.732 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/127/15' 2022-04-03T19:50:34.732 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/127/20' 2022-04-03T19:50:34.733 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/127/12' 2022-04-03T19:50:34.733 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/127/10' 2022-04-03T19:50:34.733 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/127/11' 2022-04-03T19:50:34.733 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/127/16' 2022-04-03T19:50:34.734 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/127/14' 2022-04-03T19:50:34.734 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/127/18' 2022-04-03T19:50:34.734 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/127/13' 2022-04-03T19:50:34.734 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/127/17' 2022-04-03T19:50:34.734 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/127' 2022-04-03T19:50:34.735 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/108/19' 2022-04-03T19:50:34.735 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/108/15' 2022-04-03T19:50:34.735 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/108/20' 2022-04-03T19:50:34.735 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/108/12' 2022-04-03T19:50:34.736 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/108/10' 2022-04-03T19:50:34.736 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/108/11' 2022-04-03T19:50:34.736 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/108/16' 2022-04-03T19:50:34.736 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/108/14' 2022-04-03T19:50:34.737 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/108/18' 2022-04-03T19:50:34.737 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/108/13' 2022-04-03T19:50:34.737 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/108/17' 2022-04-03T19:50:34.737 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/108' 2022-04-03T19:50:34.737 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/183/19' 2022-04-03T19:50:34.738 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/183/15' 2022-04-03T19:50:34.738 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/183/20' 2022-04-03T19:50:34.738 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/183/12' 2022-04-03T19:50:34.738 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/183/10' 2022-04-03T19:50:34.739 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/183/11' 2022-04-03T19:50:34.739 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/183/16' 2022-04-03T19:50:34.739 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/183/14' 2022-04-03T19:50:34.739 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/183/18' 2022-04-03T19:50:34.740 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/183/13' 2022-04-03T19:50:34.740 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/183/17' 2022-04-03T19:50:34.740 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/183' 2022-04-03T19:50:34.741 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/177/19' 2022-04-03T19:50:34.741 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/177/15' 2022-04-03T19:50:34.741 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/177/20' 2022-04-03T19:50:34.742 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/177/12' 2022-04-03T19:50:34.742 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/177/10' 2022-04-03T19:50:34.742 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/177/11' 2022-04-03T19:50:34.742 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/177/16' 2022-04-03T19:50:34.743 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/177/14' 2022-04-03T19:50:34.743 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/177/18' 2022-04-03T19:50:34.743 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/177/13' 2022-04-03T19:50:34.743 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/177/17' 2022-04-03T19:50:34.743 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/177' 2022-04-03T19:50:34.744 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/169/19' 2022-04-03T19:50:34.744 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/169/15' 2022-04-03T19:50:34.744 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/169/20' 2022-04-03T19:50:34.745 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/169/12' 2022-04-03T19:50:34.745 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/169/10' 2022-04-03T19:50:34.745 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/169/11' 2022-04-03T19:50:34.745 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/169/16' 2022-04-03T19:50:34.746 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/169/14' 2022-04-03T19:50:34.746 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/169/18' 2022-04-03T19:50:34.746 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/169/13' 2022-04-03T19:50:34.746 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/169/17' 2022-04-03T19:50:34.746 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/169' 2022-04-03T19:50:34.747 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/154/19' 2022-04-03T19:50:34.747 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/154/15' 2022-04-03T19:50:34.747 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/154/20' 2022-04-03T19:50:34.747 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/154/12' 2022-04-03T19:50:34.748 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/154/10' 2022-04-03T19:50:34.748 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/154/11' 2022-04-03T19:50:34.748 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/154/16' 2022-04-03T19:50:34.748 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/154/14' 2022-04-03T19:50:34.748 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/154/18' 2022-04-03T19:50:34.749 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/154/13' 2022-04-03T19:50:34.749 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/154/17' 2022-04-03T19:50:34.749 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/154' 2022-04-03T19:50:34.749 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/171/19' 2022-04-03T19:50:34.749 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/171/15' 2022-04-03T19:50:34.750 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/171/20' 2022-04-03T19:50:34.750 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/171/12' 2022-04-03T19:50:34.750 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/171/10' 2022-04-03T19:50:34.750 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/171/11' 2022-04-03T19:50:34.751 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/171/16' 2022-04-03T19:50:34.751 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/171/14' 2022-04-03T19:50:34.751 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/171/18' 2022-04-03T19:50:34.751 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/171/13' 2022-04-03T19:50:34.752 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/171/17' 2022-04-03T19:50:34.752 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/171' 2022-04-03T19:50:34.752 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/139/19' 2022-04-03T19:50:34.752 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/139/15' 2022-04-03T19:50:34.752 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/139/20' 2022-04-03T19:50:34.837 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/139/12' 2022-04-03T19:50:34.838 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/139/10' 2022-04-03T19:50:34.838 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/139/11' 2022-04-03T19:50:34.838 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/139/16' 2022-04-03T19:50:34.838 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/139/14' 2022-04-03T19:50:34.839 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/139/18' 2022-04-03T19:50:34.839 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/139/13' 2022-04-03T19:50:34.839 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/139/17' 2022-04-03T19:50:34.839 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/139' 2022-04-03T19:50:34.839 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/102/19' 2022-04-03T19:50:34.840 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/102/15' 2022-04-03T19:50:34.840 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/102/20' 2022-04-03T19:50:34.840 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/102/12' 2022-04-03T19:50:34.840 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/102/10' 2022-04-03T19:50:34.841 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/102/11' 2022-04-03T19:50:34.841 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/102/16' 2022-04-03T19:50:34.841 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/102/14' 2022-04-03T19:50:34.841 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/102/18' 2022-04-03T19:50:34.842 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/102/13' 2022-04-03T19:50:34.842 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/102/17' 2022-04-03T19:50:34.842 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/102' 2022-04-03T19:50:34.842 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/129/19' 2022-04-03T19:50:34.842 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/129/15' 2022-04-03T19:50:34.843 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/129/20' 2022-04-03T19:50:34.843 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/129/12' 2022-04-03T19:50:34.843 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/129/10' 2022-04-03T19:50:34.843 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/129/11' 2022-04-03T19:50:34.844 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/129/16' 2022-04-03T19:50:34.844 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/129/14' 2022-04-03T19:50:34.844 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/129/18' 2022-04-03T19:50:34.844 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/129/13' 2022-04-03T19:50:34.845 INFO:tasks.workunit.client.0.smithi077.stdout:removed './copyofsnap1/test1/129/17' 2022-04-03T19:50:34.845 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1/129' 2022-04-03T19:50:34.845 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1/test1' 2022-04-03T19:50:34.845 INFO:tasks.workunit.client.0.smithi077.stdout:removed directory './copyofsnap1' 2022-04-03T19:50:34.847 INFO:tasks.workunit.client.0.smithi077.stdout:OK 2022-04-03T19:50:34.847 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-03T19:50:34.848 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-03T19:50:34.975 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2022-04-03T19:50:34.976 DEBUG:teuthology.orchestra.run.smithi077:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2022-04-03T19:50:35.037 INFO:tasks.workunit.client.0.smithi077.stderr:+ set -e 2022-04-03T19:50:35.037 INFO:tasks.workunit.client.0.smithi077.stderr:+ touch foo 2022-04-03T19:50:35.042 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod +x foo 2022-04-03T19:50:35.044 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir .snap/s 2022-04-03T19:50:35.058 INFO:tasks.workunit.client.0.smithi077.stderr:+ find .snap/s/foo -executable 2022-04-03T19:50:35.059 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep foo 2022-04-03T19:50:35.062 INFO:tasks.workunit.client.0.smithi077.stdout:.snap/s/foo 2022-04-03T19:50:35.063 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir .snap/s 2022-04-03T19:50:35.748 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm foo 2022-04-03T19:50:35.802 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo OK 2022-04-03T19:50:35.803 INFO:tasks.workunit.client.0.smithi077.stdout:OK 2022-04-03T19:50:35.804 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-03T19:50:35.804 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-03T19:50:36.640 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2022-04-03T19:50:36.641 DEBUG:teuthology.orchestra.run.smithi077:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2022-04-03T19:50:36.702 INFO:tasks.workunit.client.0.smithi077.stderr:+ set -e 2022-04-03T19:50:36.703 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir foo 2022-04-03T19:50:36.707 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo x 2022-04-03T19:50:36.708 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:36.956 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir foo/.snap/ss 2022-04-03T19:50:36.966 INFO:tasks.workunit.client.0.smithi077.stderr:+ ln foo/x foo/xx 2022-04-03T19:50:36.986 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat foo/.snap/ss/x 2022-04-03T19:50:36.998 INFO:tasks.workunit.client.0.smithi077.stdout:x 2022-04-03T19:50:36.999 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir foo/.snap/ss 2022-04-03T19:50:37.007 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo a 2022-04-03T19:50:37.009 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo b 2022-04-03T19:50:37.009 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir foo/.snap/s 2022-04-03T19:50:37.021 INFO:tasks.workunit.client.0.smithi077.stderr:++ cat foo/.snap/s/a 2022-04-03T19:50:37.038 INFO:tasks.workunit.client.0.smithi077.stderr:+ r=a 2022-04-03T19:50:37.038 INFO:tasks.workunit.client.0.smithi077.stderr:+ '[' -z a ']' 2022-04-03T19:50:37.039 INFO:tasks.workunit.client.0.smithi077.stderr:+ ln foo/b foo/b2 2022-04-03T19:50:37.042 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat foo/.snap/s/b 2022-04-03T19:50:37.057 INFO:tasks.workunit.client.0.smithi077.stdout:b 2022-04-03T19:50:37.058 INFO:tasks.workunit.client.0.smithi077.stdout:this used to hang: 2022-04-03T19:50:37.058 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo 'this used to hang:' 2022-04-03T19:50:37.059 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo more 2022-04-03T19:50:37.059 INFO:tasks.workunit.client.0.smithi077.stdout:oh, it didn't hang! good job. 2022-04-03T19:50:37.059 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo 'oh, it didn'\''t hang! good job.' 2022-04-03T19:50:37.060 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat foo/b 2022-04-03T19:50:37.060 INFO:tasks.workunit.client.0.smithi077.stdout:b 2022-04-03T19:50:37.061 INFO:tasks.workunit.client.0.smithi077.stdout:more 2022-04-03T19:50:37.061 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir foo/.snap/s 2022-04-03T19:50:37.069 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm -r foo 2022-04-03T19:50:37.088 INFO:tasks.workunit.client.0.smithi077.stdout:OK 2022-04-03T19:50:37.090 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo OK 2022-04-03T19:50:37.090 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-03T19:50:37.091 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-03T19:50:37.168 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2022-04-03T19:50:37.168 DEBUG:teuthology.orchestra.run.smithi077:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2022-04-03T19:50:37.228 INFO:tasks.workunit.client.0.smithi077.stderr:+ set -e 2022-04-03T19:50:37.229 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:37.232 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir .snap/bar 2022-04-03T19:50:37.245 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/foo 2022-04-03T19:50:37.254 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/foo 2022-04-03T19:50:37.261 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:37.265 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/bar/a 2022-04-03T19:50:37.271 INFO:tasks.workunit.client.0.smithi077.stdout: File: .snap/bar/a 2022-04-03T19:50:37.271 INFO:tasks.workunit.client.0.smithi077.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2022-04-03T19:50:37.272 INFO:tasks.workunit.client.0.smithi077.stdout:Device: fdh/253d Inode: 1099511633796 Links: 2 2022-04-03T19:50:37.272 INFO:tasks.workunit.client.0.smithi077.stdout:Access: (0775/drwxrwxr-x) Uid: ( 1000/ ubuntu) Gid: ( 1258/ ubuntu) 2022-04-03T19:50:37.272 INFO:tasks.workunit.client.0.smithi077.stdout:Context: system_u:object_r:cephfs_t:s0 2022-04-03T19:50:37.272 INFO:tasks.workunit.client.0.smithi077.stdout:Access: 2022-04-03 19:50:37.230694488 +0000 2022-04-03T19:50:37.272 INFO:tasks.workunit.client.0.smithi077.stdout:Modify: 2022-04-03 19:50:37.230694488 +0000 2022-04-03T19:50:37.273 INFO:tasks.workunit.client.0.smithi077.stdout:Change: 2022-04-03 19:50:37.230694488 +0000 2022-04-03T19:50:37.273 INFO:tasks.workunit.client.0.smithi077.stdout: Birth: - 2022-04-03T19:50:37.274 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir .snap/bar 2022-04-03T19:50:37.282 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo OK 2022-04-03T19:50:37.282 INFO:tasks.workunit.client.0.smithi077.stdout:OK 2022-04-03T19:50:37.283 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-03T19:50:37.284 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-03T19:50:37.362 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2022-04-03T19:50:37.362 DEBUG:teuthology.orchestra.run.smithi077:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2022-04-03T19:50:37.422 INFO:tasks.workunit.client.0.smithi077.stderr:+ set -e 2022-04-03T19:50:37.423 INFO:tasks.workunit.client.0.smithi077.stderr:++ seq 1 20 2022-04-03T19:50:37.426 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 20` 2022-04-03T19:50:37.427 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:37.430 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/one 2022-04-03T19:50:37.430 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat 2022-04-03T19:50:37.443 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/two 2022-04-03T19:50:37.451 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod 777 a/foo 2022-04-03T19:50:37.453 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:37.548 INFO:tasks.workunit.client.0.smithi077.stderr:+ ps 2022-04-03T19:50:37.583 INFO:tasks.workunit.client.0.smithi077.stdout: PID TTY TIME CMD 2022-04-03T19:50:37.583 INFO:tasks.workunit.client.0.smithi077.stdout: 1130 ? 00:00:00 systemd 2022-04-03T19:50:37.584 INFO:tasks.workunit.client.0.smithi077.stdout: 1134 ? 00:00:00 (sd-pam) 2022-04-03T19:50:37.584 INFO:tasks.workunit.client.0.smithi077.stdout: 7899 ? 00:00:01 sshd 2022-04-03T19:50:37.584 INFO:tasks.workunit.client.0.smithi077.stdout: 34204 ? 00:00:00 timeout 2022-04-03T19:50:37.584 INFO:tasks.workunit.client.0.smithi077.stdout: 34221 ? 00:00:00 snaptest-double 2022-04-03T19:50:37.584 INFO:tasks.workunit.client.0.smithi077.stdout: 34230 ? 00:00:00 ps 2022-04-03T19:50:37.585 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/one a/.snap/two 2022-04-03T19:50:37.596 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm a/foo 2022-04-03T19:50:37.601 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:37.609 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 20` 2022-04-03T19:50:37.609 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:37.616 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/one 2022-04-03T19:50:37.616 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat 2022-04-03T19:50:37.627 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/two 2022-04-03T19:50:37.636 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod 777 a/foo 2022-04-03T19:50:37.638 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:37.715 INFO:tasks.workunit.client.0.smithi077.stderr:+ ps 2022-04-03T19:50:37.734 INFO:tasks.workunit.client.0.smithi077.stdout: PID TTY TIME CMD 2022-04-03T19:50:37.734 INFO:tasks.workunit.client.0.smithi077.stdout: 1130 ? 00:00:00 systemd 2022-04-03T19:50:37.735 INFO:tasks.workunit.client.0.smithi077.stdout: 1134 ? 00:00:00 (sd-pam) 2022-04-03T19:50:37.735 INFO:tasks.workunit.client.0.smithi077.stdout: 7899 ? 00:00:01 sshd 2022-04-03T19:50:37.735 INFO:tasks.workunit.client.0.smithi077.stdout: 34204 ? 00:00:00 timeout 2022-04-03T19:50:37.735 INFO:tasks.workunit.client.0.smithi077.stdout: 34221 ? 00:00:00 snaptest-double 2022-04-03T19:50:37.736 INFO:tasks.workunit.client.0.smithi077.stdout: 34240 ? 00:00:00 ps 2022-04-03T19:50:37.736 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/one a/.snap/two 2022-04-03T19:50:37.748 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm a/foo 2022-04-03T19:50:37.752 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:37.760 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 20` 2022-04-03T19:50:37.760 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:37.764 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/one 2022-04-03T19:50:37.765 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat 2022-04-03T19:50:37.777 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/two 2022-04-03T19:50:37.785 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod 777 a/foo 2022-04-03T19:50:37.787 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:37.840 INFO:tasks.workunit.client.0.smithi077.stderr:+ ps 2022-04-03T19:50:37.859 INFO:tasks.workunit.client.0.smithi077.stdout: PID TTY TIME CMD 2022-04-03T19:50:37.859 INFO:tasks.workunit.client.0.smithi077.stdout: 1130 ? 00:00:00 systemd 2022-04-03T19:50:37.860 INFO:tasks.workunit.client.0.smithi077.stdout: 1134 ? 00:00:00 (sd-pam) 2022-04-03T19:50:37.860 INFO:tasks.workunit.client.0.smithi077.stdout: 7899 ? 00:00:01 sshd 2022-04-03T19:50:37.860 INFO:tasks.workunit.client.0.smithi077.stdout: 34204 ? 00:00:00 timeout 2022-04-03T19:50:37.860 INFO:tasks.workunit.client.0.smithi077.stdout: 34221 ? 00:00:00 snaptest-double 2022-04-03T19:50:37.861 INFO:tasks.workunit.client.0.smithi077.stdout: 34250 ? 00:00:00 ps 2022-04-03T19:50:37.861 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/one a/.snap/two 2022-04-03T19:50:37.874 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm a/foo 2022-04-03T19:50:37.878 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:37.886 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 20` 2022-04-03T19:50:37.886 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:37.890 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/one 2022-04-03T19:50:37.890 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat 2022-04-03T19:50:37.903 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/two 2022-04-03T19:50:37.911 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod 777 a/foo 2022-04-03T19:50:37.913 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:37.990 INFO:tasks.workunit.client.0.smithi077.stderr:+ ps 2022-04-03T19:50:38.009 INFO:tasks.workunit.client.0.smithi077.stdout: PID TTY TIME CMD 2022-04-03T19:50:38.010 INFO:tasks.workunit.client.0.smithi077.stdout: 1130 ? 00:00:00 systemd 2022-04-03T19:50:38.010 INFO:tasks.workunit.client.0.smithi077.stdout: 1134 ? 00:00:00 (sd-pam) 2022-04-03T19:50:38.010 INFO:tasks.workunit.client.0.smithi077.stdout: 7899 ? 00:00:01 sshd 2022-04-03T19:50:38.010 INFO:tasks.workunit.client.0.smithi077.stdout: 34204 ? 00:00:00 timeout 2022-04-03T19:50:38.011 INFO:tasks.workunit.client.0.smithi077.stdout: 34221 ? 00:00:00 snaptest-double 2022-04-03T19:50:38.011 INFO:tasks.workunit.client.0.smithi077.stdout: 34260 ? 00:00:00 ps 2022-04-03T19:50:38.011 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/one a/.snap/two 2022-04-03T19:50:38.023 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm a/foo 2022-04-03T19:50:38.027 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:38.038 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 20` 2022-04-03T19:50:38.038 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:38.040 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/one 2022-04-03T19:50:38.040 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat 2022-04-03T19:50:38.052 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/two 2022-04-03T19:50:38.060 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod 777 a/foo 2022-04-03T19:50:38.062 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:38.106 INFO:tasks.workunit.client.0.smithi077.stderr:+ ps 2022-04-03T19:50:38.126 INFO:tasks.workunit.client.0.smithi077.stdout: PID TTY TIME CMD 2022-04-03T19:50:38.126 INFO:tasks.workunit.client.0.smithi077.stdout: 1130 ? 00:00:00 systemd 2022-04-03T19:50:38.127 INFO:tasks.workunit.client.0.smithi077.stdout: 1134 ? 00:00:00 (sd-pam) 2022-04-03T19:50:38.127 INFO:tasks.workunit.client.0.smithi077.stdout: 7899 ? 00:00:01 sshd 2022-04-03T19:50:38.127 INFO:tasks.workunit.client.0.smithi077.stdout: 34204 ? 00:00:00 timeout 2022-04-03T19:50:38.127 INFO:tasks.workunit.client.0.smithi077.stdout: 34221 ? 00:00:00 snaptest-double 2022-04-03T19:50:38.128 INFO:tasks.workunit.client.0.smithi077.stdout: 34270 ? 00:00:00 ps 2022-04-03T19:50:38.128 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/one a/.snap/two 2022-04-03T19:50:38.995 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm a/foo 2022-04-03T19:50:39.000 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:39.009 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 20` 2022-04-03T19:50:39.010 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:39.015 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/one 2022-04-03T19:50:39.015 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat 2022-04-03T19:50:39.029 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/two 2022-04-03T19:50:39.039 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod 777 a/foo 2022-04-03T19:50:39.041 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:39.098 INFO:tasks.workunit.client.0.smithi077.stderr:+ ps 2022-04-03T19:50:39.118 INFO:tasks.workunit.client.0.smithi077.stdout: PID TTY TIME CMD 2022-04-03T19:50:39.119 INFO:tasks.workunit.client.0.smithi077.stdout: 1130 ? 00:00:00 systemd 2022-04-03T19:50:39.119 INFO:tasks.workunit.client.0.smithi077.stdout: 1134 ? 00:00:00 (sd-pam) 2022-04-03T19:50:39.119 INFO:tasks.workunit.client.0.smithi077.stdout: 7899 ? 00:00:01 sshd 2022-04-03T19:50:39.119 INFO:tasks.workunit.client.0.smithi077.stdout: 34204 ? 00:00:00 timeout 2022-04-03T19:50:39.120 INFO:tasks.workunit.client.0.smithi077.stdout: 34221 ? 00:00:00 snaptest-double 2022-04-03T19:50:39.121 INFO:tasks.workunit.client.0.smithi077.stdout: 34280 ? 00:00:00 ps 2022-04-03T19:50:39.121 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/one a/.snap/two 2022-04-03T19:50:39.134 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm a/foo 2022-04-03T19:50:39.138 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:39.147 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 20` 2022-04-03T19:50:39.148 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:39.152 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/one 2022-04-03T19:50:39.152 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat 2022-04-03T19:50:39.297 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/two 2022-04-03T19:50:39.309 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod 777 a/foo 2022-04-03T19:50:39.313 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:39.473 INFO:tasks.workunit.client.0.smithi077.stderr:+ ps 2022-04-03T19:50:39.493 INFO:tasks.workunit.client.0.smithi077.stdout: PID TTY TIME CMD 2022-04-03T19:50:39.494 INFO:tasks.workunit.client.0.smithi077.stdout: 1130 ? 00:00:00 systemd 2022-04-03T19:50:39.494 INFO:tasks.workunit.client.0.smithi077.stdout: 1134 ? 00:00:00 (sd-pam) 2022-04-03T19:50:39.494 INFO:tasks.workunit.client.0.smithi077.stdout: 7899 ? 00:00:01 sshd 2022-04-03T19:50:39.495 INFO:tasks.workunit.client.0.smithi077.stdout: 34204 ? 00:00:00 timeout 2022-04-03T19:50:39.495 INFO:tasks.workunit.client.0.smithi077.stdout: 34221 ? 00:00:00 snaptest-double 2022-04-03T19:50:39.495 INFO:tasks.workunit.client.0.smithi077.stdout: 34290 ? 00:00:00 ps 2022-04-03T19:50:39.496 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/one a/.snap/two 2022-04-03T19:50:39.509 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm a/foo 2022-04-03T19:50:39.513 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:39.521 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 20` 2022-04-03T19:50:39.522 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:39.526 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/one 2022-04-03T19:50:39.526 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat 2022-04-03T19:50:39.540 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/two 2022-04-03T19:50:39.548 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod 777 a/foo 2022-04-03T19:50:39.551 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:39.598 INFO:tasks.workunit.client.0.smithi077.stderr:+ ps 2022-04-03T19:50:39.619 INFO:tasks.workunit.client.0.smithi077.stdout: PID TTY TIME CMD 2022-04-03T19:50:39.619 INFO:tasks.workunit.client.0.smithi077.stdout: 1130 ? 00:00:00 systemd 2022-04-03T19:50:39.620 INFO:tasks.workunit.client.0.smithi077.stdout: 1134 ? 00:00:00 (sd-pam) 2022-04-03T19:50:39.620 INFO:tasks.workunit.client.0.smithi077.stdout: 7899 ? 00:00:01 sshd 2022-04-03T19:50:39.621 INFO:tasks.workunit.client.0.smithi077.stdout: 34204 ? 00:00:00 timeout 2022-04-03T19:50:39.621 INFO:tasks.workunit.client.0.smithi077.stdout: 34221 ? 00:00:00 snaptest-double 2022-04-03T19:50:39.622 INFO:tasks.workunit.client.0.smithi077.stdout: 34308 ? 00:00:00 ps 2022-04-03T19:50:39.623 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/one a/.snap/two 2022-04-03T19:50:39.633 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm a/foo 2022-04-03T19:50:39.638 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:39.645 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 20` 2022-04-03T19:50:39.646 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:39.650 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/one 2022-04-03T19:50:39.651 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat 2022-04-03T19:50:39.663 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/two 2022-04-03T19:50:39.672 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod 777 a/foo 2022-04-03T19:50:39.674 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:39.716 INFO:tasks.workunit.client.0.smithi077.stderr:+ ps 2022-04-03T19:50:39.736 INFO:tasks.workunit.client.0.smithi077.stdout: PID TTY TIME CMD 2022-04-03T19:50:39.736 INFO:tasks.workunit.client.0.smithi077.stdout: 1130 ? 00:00:00 systemd 2022-04-03T19:50:39.736 INFO:tasks.workunit.client.0.smithi077.stdout: 1134 ? 00:00:00 (sd-pam) 2022-04-03T19:50:39.737 INFO:tasks.workunit.client.0.smithi077.stdout: 7899 ? 00:00:01 sshd 2022-04-03T19:50:39.737 INFO:tasks.workunit.client.0.smithi077.stdout: 34204 ? 00:00:00 timeout 2022-04-03T19:50:39.737 INFO:tasks.workunit.client.0.smithi077.stdout: 34221 ? 00:00:00 snaptest-double 2022-04-03T19:50:39.737 INFO:tasks.workunit.client.0.smithi077.stdout: 34318 ? 00:00:00 ps 2022-04-03T19:50:39.738 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/one a/.snap/two 2022-04-03T19:50:39.751 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm a/foo 2022-04-03T19:50:39.756 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:39.764 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 20` 2022-04-03T19:50:39.764 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:39.769 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/one 2022-04-03T19:50:39.769 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat 2022-04-03T19:50:39.783 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/two 2022-04-03T19:50:39.799 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod 777 a/foo 2022-04-03T19:50:39.804 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:39.873 INFO:tasks.workunit.client.0.smithi077.stderr:+ ps 2022-04-03T19:50:39.893 INFO:tasks.workunit.client.0.smithi077.stdout: PID TTY TIME CMD 2022-04-03T19:50:39.894 INFO:tasks.workunit.client.0.smithi077.stdout: 1130 ? 00:00:00 systemd 2022-04-03T19:50:39.894 INFO:tasks.workunit.client.0.smithi077.stdout: 1134 ? 00:00:00 (sd-pam) 2022-04-03T19:50:39.894 INFO:tasks.workunit.client.0.smithi077.stdout: 7899 ? 00:00:01 sshd 2022-04-03T19:50:39.895 INFO:tasks.workunit.client.0.smithi077.stdout: 34204 ? 00:00:00 timeout 2022-04-03T19:50:39.895 INFO:tasks.workunit.client.0.smithi077.stdout: 34221 ? 00:00:00 snaptest-double 2022-04-03T19:50:39.895 INFO:tasks.workunit.client.0.smithi077.stdout: 34328 ? 00:00:00 ps 2022-04-03T19:50:39.903 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/one a/.snap/two 2022-04-03T19:50:39.919 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm a/foo 2022-04-03T19:50:39.923 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:39.932 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 20` 2022-04-03T19:50:39.932 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:39.937 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/one 2022-04-03T19:50:39.937 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat 2022-04-03T19:50:40.229 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/two 2022-04-03T19:50:40.361 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod 777 a/foo 2022-04-03T19:50:40.365 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:40.492 INFO:tasks.workunit.client.0.smithi077.stderr:+ ps 2022-04-03T19:50:40.512 INFO:tasks.workunit.client.0.smithi077.stdout: PID TTY TIME CMD 2022-04-03T19:50:40.512 INFO:tasks.workunit.client.0.smithi077.stdout: 1130 ? 00:00:00 systemd 2022-04-03T19:50:40.512 INFO:tasks.workunit.client.0.smithi077.stdout: 1134 ? 00:00:00 (sd-pam) 2022-04-03T19:50:40.513 INFO:tasks.workunit.client.0.smithi077.stdout: 7899 ? 00:00:01 sshd 2022-04-03T19:50:40.513 INFO:tasks.workunit.client.0.smithi077.stdout: 34204 ? 00:00:00 timeout 2022-04-03T19:50:40.513 INFO:tasks.workunit.client.0.smithi077.stdout: 34221 ? 00:00:00 snaptest-double 2022-04-03T19:50:40.513 INFO:tasks.workunit.client.0.smithi077.stdout: 34338 ? 00:00:00 ps 2022-04-03T19:50:40.514 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/one a/.snap/two 2022-04-03T19:50:40.527 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm a/foo 2022-04-03T19:50:40.531 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:40.539 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 20` 2022-04-03T19:50:40.539 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:40.544 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/one 2022-04-03T19:50:40.544 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat 2022-04-03T19:50:40.812 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/two 2022-04-03T19:50:40.821 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod 777 a/foo 2022-04-03T19:50:40.823 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:40.874 INFO:tasks.workunit.client.0.smithi077.stderr:+ ps 2022-04-03T19:50:40.894 INFO:tasks.workunit.client.0.smithi077.stdout: PID TTY TIME CMD 2022-04-03T19:50:40.894 INFO:tasks.workunit.client.0.smithi077.stdout: 1130 ? 00:00:00 systemd 2022-04-03T19:50:40.894 INFO:tasks.workunit.client.0.smithi077.stdout: 1134 ? 00:00:00 (sd-pam) 2022-04-03T19:50:40.895 INFO:tasks.workunit.client.0.smithi077.stdout: 7899 ? 00:00:01 sshd 2022-04-03T19:50:40.895 INFO:tasks.workunit.client.0.smithi077.stdout: 34204 ? 00:00:00 timeout 2022-04-03T19:50:40.895 INFO:tasks.workunit.client.0.smithi077.stdout: 34221 ? 00:00:00 snaptest-double 2022-04-03T19:50:40.895 INFO:tasks.workunit.client.0.smithi077.stdout: 34348 ? 00:00:00 ps 2022-04-03T19:50:40.896 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/one a/.snap/two 2022-04-03T19:50:40.909 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm a/foo 2022-04-03T19:50:41.172 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:41.180 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 20` 2022-04-03T19:50:41.180 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:41.445 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/one 2022-04-03T19:50:41.445 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat 2022-04-03T19:50:41.504 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/two 2022-04-03T19:50:41.515 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod 777 a/foo 2022-04-03T19:50:41.517 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:41.676 INFO:tasks.workunit.client.0.smithi077.stderr:+ ps 2022-04-03T19:50:41.696 INFO:tasks.workunit.client.0.smithi077.stdout: PID TTY TIME CMD 2022-04-03T19:50:41.696 INFO:tasks.workunit.client.0.smithi077.stdout: 1130 ? 00:00:00 systemd 2022-04-03T19:50:41.697 INFO:tasks.workunit.client.0.smithi077.stdout: 1134 ? 00:00:00 (sd-pam) 2022-04-03T19:50:41.697 INFO:tasks.workunit.client.0.smithi077.stdout: 7899 ? 00:00:01 sshd 2022-04-03T19:50:41.697 INFO:tasks.workunit.client.0.smithi077.stdout: 34204 ? 00:00:00 timeout 2022-04-03T19:50:41.697 INFO:tasks.workunit.client.0.smithi077.stdout: 34221 ? 00:00:00 snaptest-double 2022-04-03T19:50:41.698 INFO:tasks.workunit.client.0.smithi077.stdout: 34358 ? 00:00:00 ps 2022-04-03T19:50:41.698 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/one a/.snap/two 2022-04-03T19:50:41.713 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm a/foo 2022-04-03T19:50:41.717 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:41.726 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 20` 2022-04-03T19:50:41.726 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:41.731 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/one 2022-04-03T19:50:41.731 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat 2022-04-03T19:50:41.744 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/two 2022-04-03T19:50:41.753 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod 777 a/foo 2022-04-03T19:50:41.756 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:41.808 INFO:tasks.workunit.client.0.smithi077.stderr:+ ps 2022-04-03T19:50:41.828 INFO:tasks.workunit.client.0.smithi077.stdout: PID TTY TIME CMD 2022-04-03T19:50:41.828 INFO:tasks.workunit.client.0.smithi077.stdout: 1130 ? 00:00:00 systemd 2022-04-03T19:50:41.828 INFO:tasks.workunit.client.0.smithi077.stdout: 1134 ? 00:00:00 (sd-pam) 2022-04-03T19:50:41.829 INFO:tasks.workunit.client.0.smithi077.stdout: 7899 ? 00:00:01 sshd 2022-04-03T19:50:41.829 INFO:tasks.workunit.client.0.smithi077.stdout: 34204 ? 00:00:00 timeout 2022-04-03T19:50:41.829 INFO:tasks.workunit.client.0.smithi077.stdout: 34221 ? 00:00:00 snaptest-double 2022-04-03T19:50:41.829 INFO:tasks.workunit.client.0.smithi077.stdout: 34368 ? 00:00:00 ps 2022-04-03T19:50:41.830 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/one a/.snap/two 2022-04-03T19:50:41.842 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm a/foo 2022-04-03T19:50:41.847 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:41.855 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 20` 2022-04-03T19:50:41.855 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:41.860 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/one 2022-04-03T19:50:41.860 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat 2022-04-03T19:50:41.872 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/two 2022-04-03T19:50:41.882 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod 777 a/foo 2022-04-03T19:50:41.884 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:41.924 INFO:tasks.workunit.client.0.smithi077.stderr:+ ps 2022-04-03T19:50:41.944 INFO:tasks.workunit.client.0.smithi077.stdout: PID TTY TIME CMD 2022-04-03T19:50:41.944 INFO:tasks.workunit.client.0.smithi077.stdout: 1130 ? 00:00:00 systemd 2022-04-03T19:50:41.945 INFO:tasks.workunit.client.0.smithi077.stdout: 1134 ? 00:00:00 (sd-pam) 2022-04-03T19:50:41.945 INFO:tasks.workunit.client.0.smithi077.stdout: 7899 ? 00:00:01 sshd 2022-04-03T19:50:41.945 INFO:tasks.workunit.client.0.smithi077.stdout: 34204 ? 00:00:00 timeout 2022-04-03T19:50:41.946 INFO:tasks.workunit.client.0.smithi077.stdout: 34221 ? 00:00:00 snaptest-double 2022-04-03T19:50:41.946 INFO:tasks.workunit.client.0.smithi077.stdout: 34378 ? 00:00:00 ps 2022-04-03T19:50:41.946 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/one a/.snap/two 2022-04-03T19:50:41.959 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm a/foo 2022-04-03T19:50:41.963 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:41.973 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 20` 2022-04-03T19:50:41.974 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:41.979 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/one 2022-04-03T19:50:41.979 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat 2022-04-03T19:50:41.992 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/two 2022-04-03T19:50:42.001 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod 777 a/foo 2022-04-03T19:50:42.003 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:42.083 INFO:tasks.workunit.client.0.smithi077.stderr:+ ps 2022-04-03T19:50:42.102 INFO:tasks.workunit.client.0.smithi077.stdout: PID TTY TIME CMD 2022-04-03T19:50:42.103 INFO:tasks.workunit.client.0.smithi077.stdout: 1130 ? 00:00:00 systemd 2022-04-03T19:50:42.103 INFO:tasks.workunit.client.0.smithi077.stdout: 1134 ? 00:00:00 (sd-pam) 2022-04-03T19:50:42.103 INFO:tasks.workunit.client.0.smithi077.stdout: 7899 ? 00:00:01 sshd 2022-04-03T19:50:42.103 INFO:tasks.workunit.client.0.smithi077.stdout: 34204 ? 00:00:00 timeout 2022-04-03T19:50:42.104 INFO:tasks.workunit.client.0.smithi077.stdout: 34221 ? 00:00:00 snaptest-double 2022-04-03T19:50:42.104 INFO:tasks.workunit.client.0.smithi077.stdout: 34388 ? 00:00:00 ps 2022-04-03T19:50:42.105 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/one a/.snap/two 2022-04-03T19:50:42.119 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm a/foo 2022-04-03T19:50:42.123 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:42.132 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 20` 2022-04-03T19:50:42.132 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:42.137 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/one 2022-04-03T19:50:42.137 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat 2022-04-03T19:50:42.152 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/two 2022-04-03T19:50:42.161 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod 777 a/foo 2022-04-03T19:50:42.163 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:42.215 INFO:tasks.workunit.client.0.smithi077.stderr:+ ps 2022-04-03T19:50:42.241 INFO:tasks.workunit.client.0.smithi077.stdout: PID TTY TIME CMD 2022-04-03T19:50:42.241 INFO:tasks.workunit.client.0.smithi077.stdout: 1130 ? 00:00:00 systemd 2022-04-03T19:50:42.241 INFO:tasks.workunit.client.0.smithi077.stdout: 1134 ? 00:00:00 (sd-pam) 2022-04-03T19:50:42.241 INFO:tasks.workunit.client.0.smithi077.stdout: 7899 ? 00:00:01 sshd 2022-04-03T19:50:42.242 INFO:tasks.workunit.client.0.smithi077.stdout: 34204 ? 00:00:00 timeout 2022-04-03T19:50:42.242 INFO:tasks.workunit.client.0.smithi077.stdout: 34221 ? 00:00:00 snaptest-double 2022-04-03T19:50:42.242 INFO:tasks.workunit.client.0.smithi077.stdout: 34398 ? 00:00:00 ps 2022-04-03T19:50:42.243 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/one a/.snap/two 2022-04-03T19:50:42.275 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm a/foo 2022-04-03T19:50:42.279 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:42.288 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 20` 2022-04-03T19:50:42.288 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:42.293 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/one 2022-04-03T19:50:42.293 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat 2022-04-03T19:50:42.307 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/two 2022-04-03T19:50:42.316 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod 777 a/foo 2022-04-03T19:50:42.320 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:42.454 INFO:tasks.workunit.client.0.smithi077.stderr:+ ps 2022-04-03T19:50:42.473 INFO:tasks.workunit.client.0.smithi077.stdout: PID TTY TIME CMD 2022-04-03T19:50:42.474 INFO:tasks.workunit.client.0.smithi077.stdout: 1130 ? 00:00:00 systemd 2022-04-03T19:50:42.474 INFO:tasks.workunit.client.0.smithi077.stdout: 1134 ? 00:00:00 (sd-pam) 2022-04-03T19:50:42.474 INFO:tasks.workunit.client.0.smithi077.stdout: 7899 ? 00:00:01 sshd 2022-04-03T19:50:42.474 INFO:tasks.workunit.client.0.smithi077.stdout: 34204 ? 00:00:00 timeout 2022-04-03T19:50:42.474 INFO:tasks.workunit.client.0.smithi077.stdout: 34221 ? 00:00:00 snaptest-double 2022-04-03T19:50:42.475 INFO:tasks.workunit.client.0.smithi077.stdout: 34408 ? 00:00:00 ps 2022-04-03T19:50:42.475 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/one a/.snap/two 2022-04-03T19:50:42.490 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm a/foo 2022-04-03T19:50:42.494 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:42.503 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 20` 2022-04-03T19:50:42.503 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:42.508 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/one 2022-04-03T19:50:42.508 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat 2022-04-03T19:50:42.522 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/two 2022-04-03T19:50:42.871 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod 777 a/foo 2022-04-03T19:50:42.874 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:42.925 INFO:tasks.workunit.client.0.smithi077.stderr:+ ps 2022-04-03T19:50:42.945 INFO:tasks.workunit.client.0.smithi077.stdout: PID TTY TIME CMD 2022-04-03T19:50:42.945 INFO:tasks.workunit.client.0.smithi077.stdout: 1130 ? 00:00:00 systemd 2022-04-03T19:50:42.945 INFO:tasks.workunit.client.0.smithi077.stdout: 1134 ? 00:00:00 (sd-pam) 2022-04-03T19:50:42.946 INFO:tasks.workunit.client.0.smithi077.stdout: 7899 ? 00:00:01 sshd 2022-04-03T19:50:42.946 INFO:tasks.workunit.client.0.smithi077.stdout: 34204 ? 00:00:00 timeout 2022-04-03T19:50:42.946 INFO:tasks.workunit.client.0.smithi077.stdout: 34221 ? 00:00:00 snaptest-double 2022-04-03T19:50:42.946 INFO:tasks.workunit.client.0.smithi077.stdout: 34418 ? 00:00:00 ps 2022-04-03T19:50:42.947 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/one a/.snap/two 2022-04-03T19:50:42.963 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm a/foo 2022-04-03T19:50:42.967 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:42.976 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 20` 2022-04-03T19:50:42.977 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a 2022-04-03T19:50:42.981 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/one 2022-04-03T19:50:42.982 INFO:tasks.workunit.client.0.smithi077.stderr:+ cat 2022-04-03T19:50:42.996 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir a/.snap/two 2022-04-03T19:50:43.005 INFO:tasks.workunit.client.0.smithi077.stderr:+ chmod 777 a/foo 2022-04-03T19:50:43.008 INFO:tasks.workunit.client.0.smithi077.stderr:+ sync 2022-04-03T19:50:43.059 INFO:tasks.workunit.client.0.smithi077.stderr:+ ps 2022-04-03T19:50:43.079 INFO:tasks.workunit.client.0.smithi077.stdout: PID TTY TIME CMD 2022-04-03T19:50:43.079 INFO:tasks.workunit.client.0.smithi077.stdout: 1130 ? 00:00:00 systemd 2022-04-03T19:50:43.079 INFO:tasks.workunit.client.0.smithi077.stdout: 1134 ? 00:00:00 (sd-pam) 2022-04-03T19:50:43.080 INFO:tasks.workunit.client.0.smithi077.stdout: 7899 ? 00:00:01 sshd 2022-04-03T19:50:43.080 INFO:tasks.workunit.client.0.smithi077.stdout: 34204 ? 00:00:00 timeout 2022-04-03T19:50:43.080 INFO:tasks.workunit.client.0.smithi077.stdout: 34221 ? 00:00:00 snaptest-double 2022-04-03T19:50:43.080 INFO:tasks.workunit.client.0.smithi077.stdout: 34428 ? 00:00:00 ps 2022-04-03T19:50:43.081 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a/.snap/one a/.snap/two 2022-04-03T19:50:43.095 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm a/foo 2022-04-03T19:50:43.099 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir a 2022-04-03T19:50:43.107 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo OK 2022-04-03T19:50:43.108 INFO:tasks.workunit.client.0.smithi077.stdout:OK 2022-04-03T19:50:43.109 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-03T19:50:43.109 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-03T19:50:43.187 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2022-04-03T19:50:43.187 DEBUG:teuthology.orchestra.run.smithi077:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2022-04-03T19:50:43.273 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir .snap/foo 2022-04-03T19:50:43.333 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo 'We want ENOENT, not ESTALE, here.' 2022-04-03T19:50:43.333 INFO:tasks.workunit.client.0.smithi077.stdout:We want ENOENT, not ESTALE, here. 2022-04-03T19:50:43.334 INFO:tasks.workunit.client.0.smithi077.stderr:++ seq 1 100 2022-04-03T19:50:43.336 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.336 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/1 2022-04-03T19:50:43.337 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.340 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/1': No such file or directory 2022-04-03T19:50:43.340 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.341 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/2 2022-04-03T19:50:43.341 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.343 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/2': No such file or directory 2022-04-03T19:50:43.344 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.345 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/3 2022-04-03T19:50:43.346 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.347 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/3': No such file or directory 2022-04-03T19:50:43.348 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.349 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/4 2022-04-03T19:50:43.349 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.352 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/4': No such file or directory 2022-04-03T19:50:43.352 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.353 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/5 2022-04-03T19:50:43.353 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.356 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/5': No such file or directory 2022-04-03T19:50:43.356 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.357 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/6 2022-04-03T19:50:43.357 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.360 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/6': No such file or directory 2022-04-03T19:50:43.361 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.365 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/7 2022-04-03T19:50:43.365 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.369 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/7': No such file or directory 2022-04-03T19:50:43.370 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.374 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/8 2022-04-03T19:50:43.374 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.376 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/8': No such file or directory 2022-04-03T19:50:43.376 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.377 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/9 2022-04-03T19:50:43.378 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.380 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/9': No such file or directory 2022-04-03T19:50:43.380 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.381 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/10 2022-04-03T19:50:43.381 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.383 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/10': No such file or directory 2022-04-03T19:50:43.384 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.385 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/11 2022-04-03T19:50:43.385 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.387 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/11': No such file or directory 2022-04-03T19:50:43.388 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.389 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/12 2022-04-03T19:50:43.389 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.391 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/12': No such file or directory 2022-04-03T19:50:43.392 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.393 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/13 2022-04-03T19:50:43.393 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.396 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/13': No such file or directory 2022-04-03T19:50:43.396 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.397 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/14 2022-04-03T19:50:43.397 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.399 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/14': No such file or directory 2022-04-03T19:50:43.400 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.401 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/15 2022-04-03T19:50:43.401 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.403 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/15': No such file or directory 2022-04-03T19:50:43.404 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.404 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/16 2022-04-03T19:50:43.405 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.407 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/16': No such file or directory 2022-04-03T19:50:43.408 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.409 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/17 2022-04-03T19:50:43.409 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.411 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/17': No such file or directory 2022-04-03T19:50:43.412 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.413 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/18 2022-04-03T19:50:43.413 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.416 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/18': No such file or directory 2022-04-03T19:50:43.416 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.417 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/19 2022-04-03T19:50:43.418 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.420 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/19': No such file or directory 2022-04-03T19:50:43.420 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.421 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/20 2022-04-03T19:50:43.421 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.423 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/20': No such file or directory 2022-04-03T19:50:43.424 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.425 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/21 2022-04-03T19:50:43.425 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.427 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/21': No such file or directory 2022-04-03T19:50:43.428 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.429 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/22 2022-04-03T19:50:43.429 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.431 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/22': No such file or directory 2022-04-03T19:50:43.432 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.433 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/23 2022-04-03T19:50:43.433 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.435 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/23': No such file or directory 2022-04-03T19:50:43.436 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.437 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/24 2022-04-03T19:50:43.437 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.439 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/24': No such file or directory 2022-04-03T19:50:43.440 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.441 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/25 2022-04-03T19:50:43.441 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.444 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/25': No such file or directory 2022-04-03T19:50:43.444 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.445 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/26 2022-04-03T19:50:43.446 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.448 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/26': No such file or directory 2022-04-03T19:50:43.448 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.449 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/27 2022-04-03T19:50:43.449 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.452 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/27': No such file or directory 2022-04-03T19:50:43.453 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.453 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/28 2022-04-03T19:50:43.454 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.456 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/28': No such file or directory 2022-04-03T19:50:43.456 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.457 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/29 2022-04-03T19:50:43.458 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.460 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/29': No such file or directory 2022-04-03T19:50:43.460 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.461 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/30 2022-04-03T19:50:43.462 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.464 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/30': No such file or directory 2022-04-03T19:50:43.464 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.465 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/31 2022-04-03T19:50:43.465 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.467 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/31': No such file or directory 2022-04-03T19:50:43.468 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.469 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/32 2022-04-03T19:50:43.469 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.471 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/32': No such file or directory 2022-04-03T19:50:43.472 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.473 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/33 2022-04-03T19:50:43.473 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.475 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/33': No such file or directory 2022-04-03T19:50:43.476 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.477 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/34 2022-04-03T19:50:43.477 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.479 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/34': No such file or directory 2022-04-03T19:50:43.480 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.480 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/35 2022-04-03T19:50:43.481 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.483 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/35': No such file or directory 2022-04-03T19:50:43.484 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.485 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/36 2022-04-03T19:50:43.485 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.487 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/36': No such file or directory 2022-04-03T19:50:43.487 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.488 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/37 2022-04-03T19:50:43.489 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.491 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/37': No such file or directory 2022-04-03T19:50:43.491 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.492 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/38 2022-04-03T19:50:43.492 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.494 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/38': No such file or directory 2022-04-03T19:50:43.495 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.496 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/39 2022-04-03T19:50:43.497 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.499 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/39': No such file or directory 2022-04-03T19:50:43.499 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.500 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/40 2022-04-03T19:50:43.500 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.502 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/40': No such file or directory 2022-04-03T19:50:43.503 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.504 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/41 2022-04-03T19:50:43.504 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.507 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/41': No such file or directory 2022-04-03T19:50:43.507 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.508 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/42 2022-04-03T19:50:43.509 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.511 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/42': No such file or directory 2022-04-03T19:50:43.511 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.512 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/43 2022-04-03T19:50:43.512 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.514 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/43': No such file or directory 2022-04-03T19:50:43.515 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.516 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/44 2022-04-03T19:50:43.516 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.518 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/44': No such file or directory 2022-04-03T19:50:43.519 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.520 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/45 2022-04-03T19:50:43.520 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.523 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/45': No such file or directory 2022-04-03T19:50:43.523 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.524 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/46 2022-04-03T19:50:43.524 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.527 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/46': No such file or directory 2022-04-03T19:50:43.527 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.528 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/47 2022-04-03T19:50:43.528 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.530 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/47': No such file or directory 2022-04-03T19:50:43.531 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.532 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/48 2022-04-03T19:50:43.532 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.535 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/48': No such file or directory 2022-04-03T19:50:43.535 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.536 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/49 2022-04-03T19:50:43.537 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.539 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/49': No such file or directory 2022-04-03T19:50:43.539 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.540 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/50 2022-04-03T19:50:43.540 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.542 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/50': No such file or directory 2022-04-03T19:50:43.543 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.544 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/51 2022-04-03T19:50:43.544 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.546 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/51': No such file or directory 2022-04-03T19:50:43.547 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.548 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/52 2022-04-03T19:50:43.548 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.550 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/52': No such file or directory 2022-04-03T19:50:43.551 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.552 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/53 2022-04-03T19:50:43.552 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.554 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/53': No such file or directory 2022-04-03T19:50:43.555 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.555 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/54 2022-04-03T19:50:43.556 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.558 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/54': No such file or directory 2022-04-03T19:50:43.559 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.559 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/55 2022-04-03T19:50:43.560 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.562 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/55': No such file or directory 2022-04-03T19:50:43.563 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.563 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/56 2022-04-03T19:50:43.564 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.566 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/56': No such file or directory 2022-04-03T19:50:43.566 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.567 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/57 2022-04-03T19:50:43.568 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.570 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/57': No such file or directory 2022-04-03T19:50:43.571 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.572 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/58 2022-04-03T19:50:43.572 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.574 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/58': No such file or directory 2022-04-03T19:50:43.574 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.575 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/59 2022-04-03T19:50:43.576 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.578 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/59': No such file or directory 2022-04-03T19:50:43.578 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.579 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/60 2022-04-03T19:50:43.579 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.582 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/60': No such file or directory 2022-04-03T19:50:43.582 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.583 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/61 2022-04-03T19:50:43.583 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.586 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/61': No such file or directory 2022-04-03T19:50:43.587 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.587 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/62 2022-04-03T19:50:43.588 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.590 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/62': No such file or directory 2022-04-03T19:50:43.591 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.591 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/63 2022-04-03T19:50:43.592 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.594 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/63': No such file or directory 2022-04-03T19:50:43.595 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.595 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/64 2022-04-03T19:50:43.596 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.598 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/64': No such file or directory 2022-04-03T19:50:43.599 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.599 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/65 2022-04-03T19:50:43.600 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.602 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/65': No such file or directory 2022-04-03T19:50:43.603 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.605 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/66 2022-04-03T19:50:43.605 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.606 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/66': No such file or directory 2022-04-03T19:50:43.606 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.607 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/67 2022-04-03T19:50:43.608 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.610 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/67': No such file or directory 2022-04-03T19:50:43.611 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.612 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/68 2022-04-03T19:50:43.612 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.613 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/68': No such file or directory 2022-04-03T19:50:43.614 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.615 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/69 2022-04-03T19:50:43.615 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.617 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/69': No such file or directory 2022-04-03T19:50:43.618 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.619 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/70 2022-04-03T19:50:43.619 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.622 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/70': No such file or directory 2022-04-03T19:50:43.622 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.623 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/71 2022-04-03T19:50:43.624 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.625 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/71': No such file or directory 2022-04-03T19:50:43.626 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.627 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/72 2022-04-03T19:50:43.627 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.629 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/72': No such file or directory 2022-04-03T19:50:43.630 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.631 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/73 2022-04-03T19:50:43.631 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.633 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/73': No such file or directory 2022-04-03T19:50:43.634 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.635 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/74 2022-04-03T19:50:43.635 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.637 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/74': No such file or directory 2022-04-03T19:50:43.638 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.639 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/75 2022-04-03T19:50:43.639 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.642 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/75': No such file or directory 2022-04-03T19:50:43.642 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.643 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/76 2022-04-03T19:50:43.643 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.646 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/76': No such file or directory 2022-04-03T19:50:43.646 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.647 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/77 2022-04-03T19:50:43.648 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.650 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/77': No such file or directory 2022-04-03T19:50:43.650 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.651 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/78 2022-04-03T19:50:43.652 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.654 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/78': No such file or directory 2022-04-03T19:50:43.654 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.655 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/79 2022-04-03T19:50:43.655 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.657 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/79': No such file or directory 2022-04-03T19:50:43.658 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.659 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/80 2022-04-03T19:50:43.664 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.664 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/80': No such file or directory 2022-04-03T19:50:43.665 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.666 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/81 2022-04-03T19:50:43.666 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.667 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/81': No such file or directory 2022-04-03T19:50:43.668 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.668 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/82 2022-04-03T19:50:43.669 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.670 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/82': No such file or directory 2022-04-03T19:50:43.670 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.671 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/83 2022-04-03T19:50:43.671 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.674 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/83': No such file or directory 2022-04-03T19:50:43.674 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.675 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/84 2022-04-03T19:50:43.675 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.678 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/84': No such file or directory 2022-04-03T19:50:43.678 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.679 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/85 2022-04-03T19:50:43.679 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.682 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/85': No such file or directory 2022-04-03T19:50:43.683 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.683 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/86 2022-04-03T19:50:43.684 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.686 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/86': No such file or directory 2022-04-03T19:50:43.686 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.687 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/87 2022-04-03T19:50:43.687 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.690 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/87': No such file or directory 2022-04-03T19:50:43.690 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.691 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/88 2022-04-03T19:50:43.691 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.693 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/88': No such file or directory 2022-04-03T19:50:43.694 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.695 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/89 2022-04-03T19:50:43.695 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.697 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/89': No such file or directory 2022-04-03T19:50:43.698 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.699 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/90 2022-04-03T19:50:43.699 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.701 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/90': No such file or directory 2022-04-03T19:50:43.702 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.703 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/91 2022-04-03T19:50:43.703 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.706 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/91': No such file or directory 2022-04-03T19:50:43.706 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.707 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/92 2022-04-03T19:50:43.707 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.710 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/92': No such file or directory 2022-04-03T19:50:43.710 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.711 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/93 2022-04-03T19:50:43.712 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.714 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/93': No such file or directory 2022-04-03T19:50:43.714 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.715 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/94 2022-04-03T19:50:43.715 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.718 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/94': No such file or directory 2022-04-03T19:50:43.718 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.719 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/95 2022-04-03T19:50:43.720 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.722 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/95': No such file or directory 2022-04-03T19:50:43.722 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.723 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/96 2022-04-03T19:50:43.723 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.725 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/96': No such file or directory 2022-04-03T19:50:43.726 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.727 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/97 2022-04-03T19:50:43.727 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.729 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/97': No such file or directory 2022-04-03T19:50:43.730 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.731 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/98 2022-04-03T19:50:43.731 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.733 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/98': No such file or directory 2022-04-03T19:50:43.734 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.735 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/99 2022-04-03T19:50:43.735 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.738 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/99': No such file or directory 2022-04-03T19:50:43.738 INFO:tasks.workunit.client.0.smithi077.stderr:+ for f in `seq 1 100` 2022-04-03T19:50:43.739 INFO:tasks.workunit.client.0.smithi077.stderr:+ stat .snap/foo/100 2022-04-03T19:50:43.739 INFO:tasks.workunit.client.0.smithi077.stderr:+ grep 'No such file' 2022-04-03T19:50:43.741 INFO:tasks.workunit.client.0.smithi077.stdout:stat: cannot stat '.snap/foo/100': No such file or directory 2022-04-03T19:50:43.742 INFO:tasks.workunit.client.0.smithi077.stderr:+ rmdir .snap/foo 2022-04-03T19:50:43.751 INFO:tasks.workunit.client.0.smithi077.stdout:OK 2022-04-03T19:50:43.752 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo OK 2022-04-03T19:50:43.752 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-04-03T19:50:43.753 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-04-03T19:50:43.826 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2022-04-03T19:50:43.826 DEBUG:teuthology.orchestra.run.smithi077:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2022-04-03T19:50:43.885 INFO:tasks.workunit.client.0.smithi077.stderr:+ set -e 2022-04-03T19:50:43.886 INFO:tasks.workunit.client.0.smithi077.stderr:+ retried=false 2022-04-03T19:50:43.886 INFO:tasks.workunit.client.0.smithi077.stderr:+ trap -- retry EXIT 2022-04-03T19:50:43.886 INFO:tasks.workunit.client.0.smithi077.stderr:+ rm -rf ceph 2022-04-03T19:50:43.887 INFO:tasks.workunit.client.0.smithi077.stderr:+ timeout 1800 git clone git://git.ceph.com/ceph.git 2022-04-03T19:50:43.894 INFO:tasks.workunit.client.0.smithi077.stderr:Cloning into 'ceph'... 2022-04-03T19:50:46.141 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:50:46.147 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:51:16.247 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:51:16.252 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:51:34.674 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:51:34.890 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.890+0000 7f0d16f29700 1 Processor -- start 2022-04-03T19:51:34.891 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.890+0000 7f0d16f29700 1 -- start start 2022-04-03T19:51:34.892 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.891+0000 7f0d16f29700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0d100ba3c0 0x7f0d10156b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:34.892 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.891+0000 7f0d16f29700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0d100678b0 0x7f0d1014d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:34.892 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.891+0000 7f0d16f29700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d100c3c50 con 0x7f0d10157060 2022-04-03T19:51:34.892 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.891+0000 7f0d16f29700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d100c8020 con 0x7f0d100ba3c0 2022-04-03T19:51:34.893 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.891+0000 7f0d16f29700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d10156690 con 0x7f0d100678b0 2022-04-03T19:51:34.895 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.892+0000 7f0d16728700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0d100678b0 0x7f0d1014d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:34.895 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.892+0000 7f0d15726700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0d10157060 0x7f0d1014cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38482/0 (socket says 172.21.15.77:38482) 2022-04-03T19:51:34.895 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.892+0000 7f0d15726700 1 -- 172.21.15.77:0/26318987 learned_addr learned my addr 172.21.15.77:0/26318987 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:51:34.895 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.892+0000 7f0d15f27700 1 --2- 172.21.15.77:0/26318987 >> v2:172.21.15.77:3300/0 conn(0x7f0d100ba3c0 0x7f0d10156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:34.896 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.892+0000 7f0d16728700 1 -- 172.21.15.77:0/26318987 >> v2:172.21.15.77:3300/0 conn(0x7f0d100ba3c0 msgr2=0x7f0d10156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:51:34.896 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.892+0000 7f0d16728700 1 --2- 172.21.15.77:0/26318987 >> v2:172.21.15.77:3300/0 conn(0x7f0d100ba3c0 0x7f0d10156b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:34.896 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.892+0000 7f0d16728700 1 -- 172.21.15.77:0/26318987 >> v1:172.21.15.77:6789/0 conn(0x7f0d10157060 legacy=0x7f0d1014cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:34.896 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.892+0000 7f0d16728700 1 -- 172.21.15.77:0/26318987 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d100bf350 con 0x7f0d100678b0 2022-04-03T19:51:34.896 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.893+0000 7f0d16728700 1 --2- 172.21.15.77:0/26318987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d100678b0 0x7f0d1014d210 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0d100c2430 tx=0x7f0d08009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63b345050e2530f5 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:34.897 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.893+0000 7f0d14f25700 1 -- 172.21.15.77:0/26318987 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0800e070 con 0x7f0d100678b0 2022-04-03T19:51:34.897 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.893+0000 7f0d14f25700 1 -- 172.21.15.77:0/26318987 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0800bbf0 con 0x7f0d100678b0 2022-04-03T19:51:34.897 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.893+0000 7f0d14f25700 1 -- 172.21.15.77:0/26318987 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d08012430 con 0x7f0d100678b0 2022-04-03T19:51:34.898 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.894+0000 7f0d16f29700 1 -- 172.21.15.77:0/26318987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d100678b0 msgr2=0x7f0d1014d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:34.898 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.894+0000 7f0d16f29700 1 --2- 172.21.15.77:0/26318987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d100678b0 0x7f0d1014d210 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0d100c2430 tx=0x7f0d08009780 comp rx=0 tx=0).stop 2022-04-03T19:51:34.898 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.894+0000 7f0d16f29700 1 -- 172.21.15.77:0/26318987 shutdown_connections 2022-04-03T19:51:34.898 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.894+0000 7f0d16f29700 1 --2- 172.21.15.77:0/26318987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d100678b0 0x7f0d1014d210 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:34.898 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.894+0000 7f0d16f29700 1 --2- 172.21.15.77:0/26318987 >> v2:172.21.15.77:3300/0 conn(0x7f0d100ba3c0 0x7f0d10156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:34.899 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.894+0000 7f0d16f29700 1 -- 172.21.15.77:0/26318987 >> 172.21.15.77:0/26318987 conn(0x7f0d10066da0 msgr2=0x7f0d100671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:34.899 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.894+0000 7f0d16f29700 1 -- 172.21.15.77:0/26318987 shutdown_connections 2022-04-03T19:51:34.899 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.894+0000 7f0d16f29700 1 -- 172.21.15.77:0/26318987 wait complete. 2022-04-03T19:51:34.902 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.897+0000 7f0d16f29700 1 Processor -- start 2022-04-03T19:51:34.902 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.897+0000 7f0d16f29700 1 -- start start 2022-04-03T19:51:34.902 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.898+0000 7f0d16f29700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d100678b0 0x7f0d10149110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:34.903 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.898+0000 7f0d16f29700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d100ba3c0 0x7f0d1014b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:34.903 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.898+0000 7f0d16f29700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d10157060 0x7f0d10165180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:34.903 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.899+0000 7f0d16f29700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d10169270 con 0x7f0d10157060 2022-04-03T19:51:34.903 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.899+0000 7f0d16f29700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d101683c0 con 0x7f0d100ba3c0 2022-04-03T19:51:34.903 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.899+0000 7f0d16f29700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d10168660 con 0x7f0d100678b0 2022-04-03T19:51:34.904 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.899+0000 7f0d16728700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d10157060 0x7f0d10165180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:34.904 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.899+0000 7f0d16728700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d10157060 0x7f0d10165180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42980/0 (socket says 172.21.15.77:42980) 2022-04-03T19:51:34.904 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.899+0000 7f0d15726700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d100ba3c0 0x7f0d1014b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:34.904 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.899+0000 7f0d16728700 1 -- 172.21.15.77:0/3976138979 learned_addr learned my addr 172.21.15.77:0/3976138979 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:51:34.905 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.899+0000 7f0d15f27700 1 --2- 172.21.15.77:0/3976138979 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d100678b0 0x7f0d10149110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:34.905 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.899+0000 7f0d16728700 1 -- 172.21.15.77:0/3976138979 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d100678b0 msgr2=0x7f0d10149110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:34.905 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.899+0000 7f0d16728700 1 --2- 172.21.15.77:0/3976138979 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d100678b0 0x7f0d10149110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:34.905 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.899+0000 7f0d16728700 1 -- 172.21.15.77:0/3976138979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d100ba3c0 msgr2=0x7f0d1014b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:34.905 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.900+0000 7f0d16728700 1 --2- 172.21.15.77:0/3976138979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d100ba3c0 0x7f0d1014b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:34.906 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.900+0000 7f0d16728700 1 -- 172.21.15.77:0/3976138979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d08009a20 con 0x7f0d10157060 2022-04-03T19:51:34.906 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.900+0000 7f0d15f27700 1 --2- 172.21.15.77:0/3976138979 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d100678b0 0x7f0d10149110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:51:34.906 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.900+0000 7f0d16728700 1 --2- 172.21.15.77:0/3976138979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d10157060 0x7f0d10165180 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0d080097c0 tx=0x7f0d08019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:34.906 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.900+0000 7f0cfdffb700 1 -- 172.21.15.77:0/3976138979 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d080129b0 con 0x7f0d10157060 2022-04-03T19:51:34.907 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.900+0000 7f0cfdffb700 1 -- 172.21.15.77:0/3976138979 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d08012b30 con 0x7f0d10157060 2022-04-03T19:51:34.907 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.900+0000 7f0cfdffb700 1 -- 172.21.15.77:0/3976138979 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d08004040 con 0x7f0d10157060 2022-04-03T19:51:34.907 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.900+0000 7f0d16f29700 1 -- 172.21.15.77:0/3976138979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d10168940 con 0x7f0d10157060 2022-04-03T19:51:34.907 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.901+0000 7f0d16f29700 1 -- 172.21.15.77:0/3976138979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d10168f00 con 0x7f0d10157060 2022-04-03T19:51:34.908 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.902+0000 7f0d16f29700 1 -- 172.21.15.77:0/3976138979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d100c74a0 con 0x7f0d10157060 2022-04-03T19:51:34.909 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.903+0000 7f0cfdffb700 1 -- 172.21.15.77:0/3976138979 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f0d080043b0 con 0x7f0d10157060 2022-04-03T19:51:34.909 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.905+0000 7f0cfdffb700 1 --2- 172.21.15.77:0/3976138979 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0cf0063ce0 0x7f0cf0066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:34.909 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.905+0000 7f0cfdffb700 1 -- 172.21.15.77:0/3976138979 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6402+0+0 (secure 0 0 0) 0x7f0d08083420 con 0x7f0d10157060 2022-04-03T19:51:34.909 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.905+0000 7f0d15f27700 1 --2- 172.21.15.77:0/3976138979 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0cf0063ce0 0x7f0cf0066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:34.909 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.906+0000 7f0d15f27700 1 --2- 172.21.15.77:0/3976138979 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0cf0063ce0 0x7f0cf0066190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c009c00 tx=0x7f0d0c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:34.913 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:34.913+0000 7f0cfdffb700 1 -- 172.21.15.77:0/3976138979 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f0d08051db0 con 0x7f0d10157060 2022-04-03T19:51:35.180 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.179+0000 7f0d16f29700 1 -- 172.21.15.77:0/3976138979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d1004b050 con 0x7f0d10157060 2022-04-03T19:51:35.181 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.180+0000 7f0cfdffb700 1 -- 172.21.15.77:0/3976138979 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6113 (secure 0 0 0) 0x7f0d0802c020 con 0x7f0d10157060 2022-04-03T19:51:35.181 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:51:35.181 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5239,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:50:23.308013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:51:35.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.183+0000 7f0d16f29700 1 -- 172.21.15.77:0/3976138979 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0cf0063ce0 msgr2=0x7f0cf0066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:35.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.183+0000 7f0d16f29700 1 --2- 172.21.15.77:0/3976138979 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0cf0063ce0 0x7f0cf0066190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c009c00 tx=0x7f0d0c008040 comp rx=0 tx=0).stop 2022-04-03T19:51:35.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.183+0000 7f0d16f29700 1 -- 172.21.15.77:0/3976138979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d10157060 msgr2=0x7f0d10165180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:35.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.183+0000 7f0d16f29700 1 --2- 172.21.15.77:0/3976138979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d10157060 0x7f0d10165180 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0d080097c0 tx=0x7f0d08019000 comp rx=0 tx=0).stop 2022-04-03T19:51:35.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.183+0000 7f0d16f29700 1 -- 172.21.15.77:0/3976138979 shutdown_connections 2022-04-03T19:51:35.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.183+0000 7f0d16f29700 1 --2- 172.21.15.77:0/3976138979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d10157060 0x7f0d10165180 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:35.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.183+0000 7f0d16f29700 1 --2- 172.21.15.77:0/3976138979 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0cf0063ce0 0x7f0cf0066190 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:35.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.183+0000 7f0d16f29700 1 --2- 172.21.15.77:0/3976138979 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d100678b0 0x7f0d10149110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:35.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.183+0000 7f0d16f29700 1 --2- 172.21.15.77:0/3976138979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d100ba3c0 0x7f0d1014b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:35.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.183+0000 7f0d16f29700 1 -- 172.21.15.77:0/3976138979 >> 172.21.15.77:0/3976138979 conn(0x7f0d10066da0 msgr2=0x7f0d10147020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:35.187 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.184+0000 7f0d16f29700 1 -- 172.21.15.77:0/3976138979 shutdown_connections 2022-04-03T19:51:35.187 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.184+0000 7f0d16f29700 1 -- 172.21.15.77:0/3976138979 wait complete. 2022-04-03T19:51:35.188 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2022-04-03T19:51:35.212 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2022-04-03T19:51:35.213 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2022-04-03T19:51:35.213 INFO:teuthology.orchestra.run:waiting for 300 2022-04-03T19:51:35.225 INFO:tasks.ceph.mds.d:Stopped 2022-04-03T19:51:35.226 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:51:35.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.433+0000 7ff1d7fff700 1 Processor -- start 2022-04-03T19:51:35.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.433+0000 7ff1d7fff700 1 -- start start 2022-04-03T19:51:35.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.434+0000 7ff1d7fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1d806bfb0 0x7ff1d81584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:35.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.434+0000 7ff1d7fff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff1d806c500 0x7ff1d814eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:35.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.435+0000 7ff1d7fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d806d260 con 0x7ff1d806bbe0 2022-04-03T19:51:35.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.435+0000 7ff1d7fff700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1d8159450 con 0x7ff1d806c500 2022-04-03T19:51:35.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.435+0000 7ff1d7fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d80c6330 con 0x7ff1d806bfb0 2022-04-03T19:51:35.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.435+0000 7ff1d67fc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1d806bfb0 0x7ff1d81584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:35.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.435+0000 7ff1d77fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff1d806c500 0x7ff1d814eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:35.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.435+0000 7ff1d67fc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1d806bfb0 0x7ff1d81584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60262/0 (socket says 172.21.15.77:60262) 2022-04-03T19:51:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.435+0000 7ff1d67fc700 1 -- 172.21.15.77:0/3247658394 learned_addr learned my addr 172.21.15.77:0/3247658394 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:51:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.435+0000 7ff1d77fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff1d806c500 0x7ff1d814eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38168/0 (socket says 172.21.15.77:38168) 2022-04-03T19:51:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.436+0000 7ff1d67fc700 1 -- 172.21.15.77:0/3247658394 >> v2:172.21.15.186:3300/0 conn(0x7ff1d806c500 msgr2=0x7ff1d814eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:51:35.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.436+0000 7ff1d67fc700 1 --2- 172.21.15.77:0/3247658394 >> v2:172.21.15.186:3300/0 conn(0x7ff1d806c500 0x7ff1d814eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:35.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.436+0000 7ff1d67fc700 1 -- 172.21.15.77:0/3247658394 >> v1:172.21.15.77:6789/0 conn(0x7ff1d806bbe0 legacy=0x7ff1d806d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:35.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.436+0000 7ff1d67fc700 1 -- 172.21.15.77:0/3247658394 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d8158bb0 con 0x7ff1d806bfb0 2022-04-03T19:51:35.441 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.436+0000 7ff1d67fc700 1 --2- 172.21.15.77:0/3247658394 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1d806bfb0 0x7ff1d81584b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc006fd0 tx=0x7ff1cc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46440e7123114e56 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:35.441 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.436+0000 7ff1d5ffb700 1 -- 172.21.15.77:0/3247658394 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc004030 con 0x7ff1d806bfb0 2022-04-03T19:51:35.441 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.436+0000 7ff1d5ffb700 1 -- 172.21.15.77:0/3247658394 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1cc004190 con 0x7ff1d806bfb0 2022-04-03T19:51:35.441 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.436+0000 7ff1d5ffb700 1 -- 172.21.15.77:0/3247658394 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc011630 con 0x7ff1d806bfb0 2022-04-03T19:51:35.442 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.437+0000 7ff1d7fff700 1 -- 172.21.15.77:0/3247658394 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1d806bfb0 msgr2=0x7ff1d81584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:35.442 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.437+0000 7ff1d7fff700 1 --2- 172.21.15.77:0/3247658394 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1d806bfb0 0x7ff1d81584b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc006fd0 tx=0x7ff1cc0090f0 comp rx=0 tx=0).stop 2022-04-03T19:51:35.442 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.437+0000 7ff1d7fff700 1 -- 172.21.15.77:0/3247658394 shutdown_connections 2022-04-03T19:51:35.443 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.437+0000 7ff1d7fff700 1 --2- 172.21.15.77:0/3247658394 >> v2:172.21.15.186:3300/0 conn(0x7ff1d806c500 0x7ff1d814eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:35.443 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.437+0000 7ff1d7fff700 1 --2- 172.21.15.77:0/3247658394 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1d806bfb0 0x7ff1d81584b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:35.443 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.437+0000 7ff1d7fff700 1 -- 172.21.15.77:0/3247658394 >> 172.21.15.77:0/3247658394 conn(0x7ff1d80cb6b0 msgr2=0x7ff1d80cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:35.444 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.437+0000 7ff1d7fff700 1 -- 172.21.15.77:0/3247658394 shutdown_connections 2022-04-03T19:51:35.444 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.437+0000 7ff1d7fff700 1 -- 172.21.15.77:0/3247658394 wait complete. 2022-04-03T19:51:35.444 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.440+0000 7ff1d7fff700 1 Processor -- start 2022-04-03T19:51:35.444 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.440+0000 7ff1d7fff700 1 -- start start 2022-04-03T19:51:35.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.442+0000 7ff1d7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1d806bbe0 0x7ff1d80c7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:35.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.442+0000 7ff1d7fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1d806bfb0 0x7ff1d8157460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:35.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.442+0000 7ff1d7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1d806c500 0x7ff1d8165160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:35.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.442+0000 7ff1d7fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d8164d30 con 0x7ff1d806c500 2022-04-03T19:51:35.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.442+0000 7ff1d7fff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d8168400 con 0x7ff1d806bbe0 2022-04-03T19:51:35.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.442+0000 7ff1d7fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d80c7ba0 con 0x7ff1d806bfb0 2022-04-03T19:51:35.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.442+0000 7ff1d67fc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1d806bfb0 0x7ff1d8157460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:35.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.442+0000 7ff1d6ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1d806bbe0 0x7ff1d80c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:35.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.442+0000 7ff1d67fc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1d806bfb0 0x7ff1d8157460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60264/0 (socket says 172.21.15.77:60264) 2022-04-03T19:51:35.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.442+0000 7ff1d67fc700 1 -- 172.21.15.77:0/285468416 learned_addr learned my addr 172.21.15.77:0/285468416 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:51:35.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.442+0000 7ff1d77fe700 1 --2- 172.21.15.77:0/285468416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1d806c500 0x7ff1d8165160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:35.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.443+0000 7ff1d67fc700 1 -- 172.21.15.77:0/285468416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1d806bbe0 msgr2=0x7ff1d80c7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:35.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.443+0000 7ff1d67fc700 1 --2- 172.21.15.77:0/285468416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1d806bbe0 0x7ff1d80c7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:35.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.443+0000 7ff1d67fc700 1 -- 172.21.15.77:0/285468416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1d806c500 msgr2=0x7ff1d8165160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:35.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.443+0000 7ff1d67fc700 1 --2- 172.21.15.77:0/285468416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1d806c500 0x7ff1d8165160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:35.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.443+0000 7ff1d67fc700 1 -- 172.21.15.77:0/285468416 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1cc009370 con 0x7ff1d806bfb0 2022-04-03T19:51:35.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.443+0000 7ff1d67fc700 1 --2- 172.21.15.77:0/285468416 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1d806bfb0 0x7ff1d8157460 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8159590 tx=0x7ff1cc016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:35.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.443+0000 7ff1beffd700 1 -- 172.21.15.77:0/285468416 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc011630 con 0x7ff1d806bfb0 2022-04-03T19:51:35.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.443+0000 7ff1beffd700 1 -- 172.21.15.77:0/285468416 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1cc0036f0 con 0x7ff1d806bfb0 2022-04-03T19:51:35.452 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.443+0000 7ff1d7fff700 1 -- 172.21.15.77:0/285468416 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1d8168680 con 0x7ff1d806bfb0 2022-04-03T19:51:35.452 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.443+0000 7ff1beffd700 1 -- 172.21.15.77:0/285468416 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc003ab0 con 0x7ff1d806bfb0 2022-04-03T19:51:35.452 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.444+0000 7ff1d7fff700 1 -- 172.21.15.77:0/285468416 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1d8168c40 con 0x7ff1d806bfb0 2022-04-03T19:51:35.452 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.445+0000 7ff1d7fff700 1 -- 172.21.15.77:0/285468416 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1d804b050 con 0x7ff1d806bfb0 2022-04-03T19:51:35.453 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.447+0000 7ff1beffd700 1 -- 172.21.15.77:0/285468416 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7ff1cc02b030 con 0x7ff1d806bfb0 2022-04-03T19:51:35.454 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.447+0000 7ff1beffd700 1 --2- 172.21.15.77:0/285468416 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff1b8063ce0 0x7ff1b8066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:35.454 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.448+0000 7ff1beffd700 1 -- 172.21.15.77:0/285468416 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6402+0+0 (secure 0 0 0) 0x7ff1cc0821a0 con 0x7ff1d806bfb0 2022-04-03T19:51:35.455 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.448+0000 7ff1d6ffd700 1 --2- 172.21.15.77:0/285468416 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff1b8063ce0 0x7ff1b8066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:35.455 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.449+0000 7ff1d6ffd700 1 --2- 172.21.15.77:0/285468416 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff1b8063ce0 0x7ff1b8066190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80017a0 tx=0x7ff1c8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:35.455 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.452+0000 7ff1beffd700 1 -- 172.21.15.77:0/285468416 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7ff1cc050c20 con 0x7ff1d806bfb0 2022-04-03T19:51:35.771 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.771+0000 7ff1d7fff700 1 -- 172.21.15.77:0/285468416 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1d80c7ec0 con 0x7ff1d806bfb0 2022-04-03T19:51:35.772 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.772+0000 7ff1beffd700 1 -- 172.21.15.77:0/285468416 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6113 (secure 0 0 0) 0x7ff1cc054240 con 0x7ff1d806bfb0 2022-04-03T19:51:35.772 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:51:35.773 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5239,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:50:23.308013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:51:35.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.775+0000 7ff1d7fff700 1 -- 172.21.15.77:0/285468416 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff1b8063ce0 msgr2=0x7ff1b8066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:35.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.775+0000 7ff1d7fff700 1 --2- 172.21.15.77:0/285468416 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff1b8063ce0 0x7ff1b8066190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80017a0 tx=0x7ff1c8009040 comp rx=0 tx=0).stop 2022-04-03T19:51:35.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.775+0000 7ff1d7fff700 1 -- 172.21.15.77:0/285468416 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1d806bfb0 msgr2=0x7ff1d8157460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:35.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.775+0000 7ff1d7fff700 1 --2- 172.21.15.77:0/285468416 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1d806bfb0 0x7ff1d8157460 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8159590 tx=0x7ff1cc016000 comp rx=0 tx=0).stop 2022-04-03T19:51:35.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.775+0000 7ff1d7fff700 1 -- 172.21.15.77:0/285468416 shutdown_connections 2022-04-03T19:51:35.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.775+0000 7ff1d7fff700 1 --2- 172.21.15.77:0/285468416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1d806c500 0x7ff1d8165160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:35.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.775+0000 7ff1d7fff700 1 --2- 172.21.15.77:0/285468416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1d806bbe0 0x7ff1d80c7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:35.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.775+0000 7ff1d7fff700 1 --2- 172.21.15.77:0/285468416 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff1b8063ce0 0x7ff1b8066190 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:35.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.775+0000 7ff1d7fff700 1 --2- 172.21.15.77:0/285468416 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1d806bfb0 0x7ff1d8157460 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:35.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.775+0000 7ff1d7fff700 1 -- 172.21.15.77:0/285468416 >> 172.21.15.77:0/285468416 conn(0x7ff1d80cb6b0 msgr2=0x7ff1d8149460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:35.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.775+0000 7ff1d7fff700 1 -- 172.21.15.77:0/285468416 shutdown_connections 2022-04-03T19:51:35.779 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:35.775+0000 7ff1d7fff700 1 -- 172.21.15.77:0/285468416 wait complete. 2022-04-03T19:51:35.779 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2022-04-03T19:51:35.813 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:51:37.814 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:51:38.019 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.018+0000 7fae4d1dd700 1 Processor -- start 2022-04-03T19:51:38.019 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.019+0000 7fae4d1dd700 1 -- start start 2022-04-03T19:51:38.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.020+0000 7fae4d1dd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae480c5930 0x7fae48155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:38.022 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.020+0000 7fae4d1dd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fae48156120 0x7fae4806c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:38.023 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.020+0000 7fae4d1dd700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae480c2f80 con 0x7fae4806d3f0 2022-04-03T19:51:38.023 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.020+0000 7fae4d1dd700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae480c3260 con 0x7fae48156120 2022-04-03T19:51:38.023 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.020+0000 7fae4d1dd700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae480c3540 con 0x7fae480c5930 2022-04-03T19:51:38.023 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.020+0000 7fae477fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae480c5930 0x7fae48155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:38.023 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.020+0000 7fae47fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fae4806d3f0 0x7fae4806d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38486/0 (socket says 172.21.15.77:38486) 2022-04-03T19:51:38.024 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.020+0000 7fae47fff700 1 -- 172.21.15.77:0/227251187 learned_addr learned my addr 172.21.15.77:0/227251187 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:51:38.024 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.020+0000 7fae4c9dc700 1 --2- 172.21.15.77:0/227251187 >> v2:172.21.15.77:3300/0 conn(0x7fae48156120 0x7fae4806c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:38.024 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.021+0000 7fae477fe700 1 -- 172.21.15.77:0/227251187 >> v2:172.21.15.77:3300/0 conn(0x7fae48156120 msgr2=0x7fae4806c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:51:38.024 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.021+0000 7fae477fe700 1 --2- 172.21.15.77:0/227251187 >> v2:172.21.15.77:3300/0 conn(0x7fae48156120 0x7fae4806c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:38.025 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.021+0000 7fae477fe700 1 -- 172.21.15.77:0/227251187 >> v1:172.21.15.77:6789/0 conn(0x7fae4806d3f0 legacy=0x7fae4806d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:38.025 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.021+0000 7fae477fe700 1 -- 172.21.15.77:0/227251187 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae480c3820 con 0x7fae480c5930 2022-04-03T19:51:38.025 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.021+0000 7fae4c9dc700 1 --2- 172.21.15.77:0/227251187 >> v2:172.21.15.77:3300/0 conn(0x7fae48156120 0x7fae4806c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:51:38.025 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.021+0000 7fae477fe700 1 --2- 172.21.15.77:0/227251187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae480c5930 0x7fae48155be0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fae3c009790 tx=0x7fae3c009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6eda4053ba030c96 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:38.026 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.021+0000 7fae46ffd700 1 -- 172.21.15.77:0/227251187 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c011630 con 0x7fae480c5930 2022-04-03T19:51:38.026 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.022+0000 7fae46ffd700 1 -- 172.21.15.77:0/227251187 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3c011790 con 0x7fae480c5930 2022-04-03T19:51:38.026 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.022+0000 7fae46ffd700 1 -- 172.21.15.77:0/227251187 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c011b50 con 0x7fae480c5930 2022-04-03T19:51:38.030 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.022+0000 7fae4d1dd700 1 -- 172.21.15.77:0/227251187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae480c5930 msgr2=0x7fae48155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:38.030 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.022+0000 7fae4d1dd700 1 --2- 172.21.15.77:0/227251187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae480c5930 0x7fae48155be0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fae3c009790 tx=0x7fae3c009060 comp rx=0 tx=0).stop 2022-04-03T19:51:38.031 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.022+0000 7fae4d1dd700 1 -- 172.21.15.77:0/227251187 shutdown_connections 2022-04-03T19:51:38.031 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.022+0000 7fae4d1dd700 1 --2- 172.21.15.77:0/227251187 >> v2:172.21.15.77:3300/0 conn(0x7fae48156120 0x7fae4806c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:38.031 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.022+0000 7fae4d1dd700 1 --2- 172.21.15.77:0/227251187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae480c5930 0x7fae48155be0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:38.032 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.022+0000 7fae4d1dd700 1 -- 172.21.15.77:0/227251187 >> 172.21.15.77:0/227251187 conn(0x7fae480cb720 msgr2=0x7fae480cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:38.032 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.023+0000 7fae4d1dd700 1 -- 172.21.15.77:0/227251187 shutdown_connections 2022-04-03T19:51:38.032 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.023+0000 7fae4d1dd700 1 -- 172.21.15.77:0/227251187 wait complete. 2022-04-03T19:51:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.025+0000 7fae4d1dd700 1 Processor -- start 2022-04-03T19:51:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.026+0000 7fae4d1dd700 1 -- start start 2022-04-03T19:51:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.027+0000 7fae4d1dd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae4806d3f0 0x7fae48152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.027+0000 7fae4d1dd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae480c5930 0x7fae48150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:38.034 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.027+0000 7fae4d1dd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae48156120 0x7fae481507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:38.035 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.027+0000 7fae4d1dd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4816a0f0 con 0x7fae4806d3f0 2022-04-03T19:51:38.035 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.027+0000 7fae4d1dd700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae480c7940 con 0x7fae480c5930 2022-04-03T19:51:38.035 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.027+0000 7fae4d1dd700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4814fde0 con 0x7fae48156120 2022-04-03T19:51:38.036 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.027+0000 7fae47fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae4806d3f0 0x7fae48152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:38.036 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.027+0000 7fae477fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae480c5930 0x7fae48150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:38.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.027+0000 7fae4c9dc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae48156120 0x7fae481507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:38.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.027+0000 7fae477fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae480c5930 0x7fae48150270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38172/0 (socket says 172.21.15.77:38172) 2022-04-03T19:51:38.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.027+0000 7fae477fe700 1 -- 172.21.15.77:0/1704810477 learned_addr learned my addr 172.21.15.77:0/1704810477 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:51:38.038 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.028+0000 7fae4c9dc700 1 -- 172.21.15.77:0/1704810477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae480c5930 msgr2=0x7fae48150270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:38.038 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.028+0000 7fae4c9dc700 1 --2- 172.21.15.77:0/1704810477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae480c5930 0x7fae48150270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:38.038 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.028+0000 7fae4c9dc700 1 -- 172.21.15.77:0/1704810477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae4806d3f0 msgr2=0x7fae48152440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.028+0000 7fae4c9dc700 1 --2- 172.21.15.77:0/1704810477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae4806d3f0 0x7fae48152440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.028+0000 7fae4c9dc700 1 -- 172.21.15.77:0/1704810477 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c0092a0 con 0x7fae48156120 2022-04-03T19:51:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.028+0000 7fae47fff700 1 --2- 172.21.15.77:0/1704810477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae4806d3f0 0x7fae48152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:51:38.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.028+0000 7fae4c9dc700 1 --2- 172.21.15.77:0/1704810477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae48156120 0x7fae481507b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fae40007f60 tx=0x7fae400098f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:38.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.029+0000 7fae2ffff700 1 -- 172.21.15.77:0/1704810477 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4000db80 con 0x7fae48156120 2022-04-03T19:51:38.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.029+0000 7fae2ffff700 1 -- 172.21.15.77:0/1704810477 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4000f040 con 0x7fae48156120 2022-04-03T19:51:38.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.029+0000 7fae2ffff700 1 -- 172.21.15.77:0/1704810477 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae40003860 con 0x7fae48156120 2022-04-03T19:51:38.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.029+0000 7fae4d1dd700 1 -- 172.21.15.77:0/1704810477 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae480bfed0 con 0x7fae48156120 2022-04-03T19:51:38.041 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.029+0000 7fae4d1dd700 1 -- 172.21.15.77:0/1704810477 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae4816a2f0 con 0x7fae48156120 2022-04-03T19:51:38.042 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.030+0000 7fae4d1dd700 1 -- 172.21.15.77:0/1704810477 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae4806db10 con 0x7fae48156120 2022-04-03T19:51:38.042 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.032+0000 7fae2ffff700 1 -- 172.21.15.77:0/1704810477 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fae4001c030 con 0x7fae48156120 2022-04-03T19:51:38.042 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.033+0000 7fae2ffff700 1 --2- 172.21.15.77:0/1704810477 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fae28063d10 0x7fae280661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:38.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.033+0000 7fae2ffff700 1 -- 172.21.15.77:0/1704810477 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6402+0+0 (secure 0 0 0) 0x7fae400808d0 con 0x7fae48156120 2022-04-03T19:51:38.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.033+0000 7fae47fff700 1 --2- 172.21.15.77:0/1704810477 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fae28063d10 0x7fae280661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:38.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.034+0000 7fae47fff700 1 --2- 172.21.15.77:0/1704810477 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fae28063d10 0x7fae280661c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fae38007fd0 tx=0x7fae38007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:38.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.037+0000 7fae2ffff700 1 -- 172.21.15.77:0/1704810477 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fae4004f350 con 0x7fae48156120 2022-04-03T19:51:38.352 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.351+0000 7fae4d1dd700 1 -- 172.21.15.77:0/1704810477 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae4804b050 con 0x7fae48156120 2022-04-03T19:51:38.353 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.352+0000 7fae2ffff700 1 -- 172.21.15.77:0/1704810477 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6113 (secure 0 0 0) 0x7fae4001a020 con 0x7fae48156120 2022-04-03T19:51:38.353 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:51:38.354 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5239,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:50:23.308013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:51:38.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.355+0000 7fae4d1dd700 1 -- 172.21.15.77:0/1704810477 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fae28063d10 msgr2=0x7fae280661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:38.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.356+0000 7fae4d1dd700 1 --2- 172.21.15.77:0/1704810477 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fae28063d10 0x7fae280661c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fae38007fd0 tx=0x7fae38007d30 comp rx=0 tx=0).stop 2022-04-03T19:51:38.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.356+0000 7fae4d1dd700 1 -- 172.21.15.77:0/1704810477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae48156120 msgr2=0x7fae481507b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:38.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.356+0000 7fae4d1dd700 1 --2- 172.21.15.77:0/1704810477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae48156120 0x7fae481507b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fae40007f60 tx=0x7fae400098f0 comp rx=0 tx=0).stop 2022-04-03T19:51:38.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.356+0000 7fae4d1dd700 1 -- 172.21.15.77:0/1704810477 shutdown_connections 2022-04-03T19:51:38.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.356+0000 7fae4d1dd700 1 --2- 172.21.15.77:0/1704810477 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fae28063d10 0x7fae280661c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:38.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.356+0000 7fae4d1dd700 1 --2- 172.21.15.77:0/1704810477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae48156120 0x7fae481507b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:38.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.356+0000 7fae4d1dd700 1 --2- 172.21.15.77:0/1704810477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae4806d3f0 0x7fae48152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:38.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.356+0000 7fae4d1dd700 1 --2- 172.21.15.77:0/1704810477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae480c5930 0x7fae48150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:38.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.356+0000 7fae4d1dd700 1 -- 172.21.15.77:0/1704810477 >> 172.21.15.77:0/1704810477 conn(0x7fae480cb720 msgr2=0x7fae480c5dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:38.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.356+0000 7fae4d1dd700 1 -- 172.21.15.77:0/1704810477 shutdown_connections 2022-04-03T19:51:38.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:38.356+0000 7fae4d1dd700 1 -- 172.21.15.77:0/1704810477 wait complete. 2022-04-03T19:51:38.360 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2022-04-03T19:51:38.397 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:51:40.398 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:51:40.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.596+0000 7fd94759e700 1 Processor -- start 2022-04-03T19:51:40.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.597+0000 7fd94759e700 1 -- start start 2022-04-03T19:51:40.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.598+0000 7fd94759e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd94806bbe0 0x7fd9481584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:40.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.598+0000 7fd94759e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd94806c500 0x7fd94814eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:40.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.598+0000 7fd94759e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd94806d260 con 0x7fd94806bfb0 2022-04-03T19:51:40.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.598+0000 7fd94759e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd948159450 con 0x7fd94806c500 2022-04-03T19:51:40.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.598+0000 7fd94759e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9480c6330 con 0x7fd94806bbe0 2022-04-03T19:51:40.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.598+0000 7fd94659c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd94806bbe0 0x7fd9481584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:40.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.598+0000 7fd946d9d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd94806c500 0x7fd94814eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:40.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.598+0000 7fd93ffff700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fd94806bfb0 0x7fd94806d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44270/0 (socket says 172.21.15.77:44270) 2022-04-03T19:51:40.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.598+0000 7fd93ffff700 1 -- 172.21.15.77:0/1091357366 learned_addr learned my addr 172.21.15.77:0/1091357366 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:51:40.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.599+0000 7fd946d9d700 1 -- 172.21.15.77:0/1091357366 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd94806bbe0 msgr2=0x7fd9481584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:51:40.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.599+0000 7fd946d9d700 1 --2- 172.21.15.77:0/1091357366 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd94806bbe0 0x7fd9481584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:40.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.599+0000 7fd946d9d700 1 -- 172.21.15.77:0/1091357366 >> v1:172.21.15.186:6789/0 conn(0x7fd94806bfb0 legacy=0x7fd94806d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:40.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.599+0000 7fd946d9d700 1 -- 172.21.15.77:0/1091357366 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd948158bb0 con 0x7fd94806c500 2022-04-03T19:51:40.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.599+0000 7fd94659c700 1 --2- 172.21.15.77:0/1091357366 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd94806bbe0 0x7fd9481584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:51:40.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.599+0000 7fd945d9b700 1 -- 172.21.15.77:0/1091357366 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473884387 0 0) 0x7fd94806d260 con 0x7fd94806bfb0 2022-04-03T19:51:40.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.599+0000 7fd946d9d700 1 --2- 172.21.15.77:0/1091357366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94806c500 0x7fd94814eea0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd940009e60 tx=0x7fd940009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d34bb023cf06bfa2 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:40.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.599+0000 7fd945d9b700 1 -- 172.21.15.77:0/1091357366 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94000e070 con 0x7fd94806c500 2022-04-03T19:51:40.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.599+0000 7fd945d9b700 1 -- 172.21.15.77:0/1091357366 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94000bb60 con 0x7fd94806c500 2022-04-03T19:51:40.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.599+0000 7fd945d9b700 1 -- 172.21.15.77:0/1091357366 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940012430 con 0x7fd94806c500 2022-04-03T19:51:40.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.600+0000 7fd94759e700 1 -- 172.21.15.77:0/1091357366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94806c500 msgr2=0x7fd94814eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:40.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.600+0000 7fd94759e700 1 --2- 172.21.15.77:0/1091357366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94806c500 0x7fd94814eea0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd940009e60 tx=0x7fd940009780 comp rx=0 tx=0).stop 2022-04-03T19:51:40.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.600+0000 7fd94759e700 1 -- 172.21.15.77:0/1091357366 shutdown_connections 2022-04-03T19:51:40.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.600+0000 7fd94759e700 1 --2- 172.21.15.77:0/1091357366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94806c500 0x7fd94814eea0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:40.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.600+0000 7fd94759e700 1 --2- 172.21.15.77:0/1091357366 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd94806bbe0 0x7fd9481584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:40.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.600+0000 7fd94759e700 1 -- 172.21.15.77:0/1091357366 >> 172.21.15.77:0/1091357366 conn(0x7fd9480cb6b0 msgr2=0x7fd9480cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:40.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.600+0000 7fd94759e700 1 -- 172.21.15.77:0/1091357366 shutdown_connections 2022-04-03T19:51:40.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.600+0000 7fd94759e700 1 -- 172.21.15.77:0/1091357366 wait complete. 2022-04-03T19:51:40.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.602+0000 7fd94759e700 1 Processor -- start 2022-04-03T19:51:40.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.602+0000 7fd94759e700 1 -- start start 2022-04-03T19:51:40.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.603+0000 7fd94759e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd94806bbe0 0x7fd9480c08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:40.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.603+0000 7fd94759e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94806bfb0 0x7fd9480c2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:40.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.603+0000 7fd94759e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd94806c500 0x7fd9480c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:40.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.603+0000 7fd94759e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9481683e0 con 0x7fd94806c500 2022-04-03T19:51:40.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.603+0000 7fd94759e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd948168f60 con 0x7fd94806bfb0 2022-04-03T19:51:40.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.603+0000 7fd94759e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9480c3920 con 0x7fd94806bbe0 2022-04-03T19:51:40.611 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.603+0000 7fd946d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd94806c500 0x7fd9480c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:40.611 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.603+0000 7fd946d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd94806c500 0x7fd9480c32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42988/0 (socket says 172.21.15.77:42988) 2022-04-03T19:51:40.611 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.603+0000 7fd946d9d700 1 -- 172.21.15.77:0/1787292596 learned_addr learned my addr 172.21.15.77:0/1787292596 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:51:40.611 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.603+0000 7fd94659c700 1 --2- 172.21.15.77:0/1787292596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd94806bbe0 0x7fd9480c08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:40.612 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.603+0000 7fd93ffff700 1 --2- 172.21.15.77:0/1787292596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94806bfb0 0x7fd9480c2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:40.612 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.603+0000 7fd946d9d700 1 -- 172.21.15.77:0/1787292596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd94806bbe0 msgr2=0x7fd9480c08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:40.612 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.603+0000 7fd946d9d700 1 --2- 172.21.15.77:0/1787292596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd94806bbe0 0x7fd9480c08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:40.612 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.603+0000 7fd946d9d700 1 -- 172.21.15.77:0/1787292596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94806bfb0 msgr2=0x7fd9480c2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:40.612 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.603+0000 7fd946d9d700 1 --2- 172.21.15.77:0/1787292596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94806bfb0 0x7fd9480c2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:40.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.604+0000 7fd946d9d700 1 -- 172.21.15.77:0/1787292596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9380092a0 con 0x7fd94806c500 2022-04-03T19:51:40.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.604+0000 7fd946d9d700 1 --2- 172.21.15.77:0/1787292596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd94806c500 0x7fd9480c32f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd940003b90 tx=0x7fd940019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:40.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.604+0000 7fd93e7fc700 1 -- 172.21.15.77:0/1787292596 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9400127a0 con 0x7fd94806c500 2022-04-03T19:51:40.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.604+0000 7fd93e7fc700 1 -- 172.21.15.77:0/1787292596 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd940012900 con 0x7fd94806c500 2022-04-03T19:51:40.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.604+0000 7fd94759e700 1 -- 172.21.15.77:0/1787292596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd940009a20 con 0x7fd94806c500 2022-04-03T19:51:40.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.604+0000 7fd93e7fc700 1 -- 172.21.15.77:0/1787292596 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940012ca0 con 0x7fd94806c500 2022-04-03T19:51:40.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.604+0000 7fd94759e700 1 -- 172.21.15.77:0/1787292596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd948168910 con 0x7fd94806c500 2022-04-03T19:51:40.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.605+0000 7fd94759e700 1 -- 172.21.15.77:0/1787292596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9481692f0 con 0x7fd94806c500 2022-04-03T19:51:40.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.606+0000 7fd93e7fc700 1 -- 172.21.15.77:0/1787292596 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fd940004250 con 0x7fd94806c500 2022-04-03T19:51:40.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.607+0000 7fd93e7fc700 1 --2- 172.21.15.77:0/1787292596 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd92c063ce0 0x7fd92c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:40.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.607+0000 7fd93e7fc700 1 -- 172.21.15.77:0/1787292596 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6402+0+0 (secure 0 0 0) 0x7fd9400831f0 con 0x7fd94806c500 2022-04-03T19:51:40.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.607+0000 7fd94659c700 1 --2- 172.21.15.77:0/1787292596 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd92c063ce0 0x7fd92c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:40.617 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.608+0000 7fd94659c700 1 --2- 172.21.15.77:0/1787292596 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd92c063ce0 0x7fd92c066190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd938009d20 tx=0x7fd938009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:40.617 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.611+0000 7fd93e7fc700 1 -- 172.21.15.77:0/1787292596 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fd940051c10 con 0x7fd94806c500 2022-04-03T19:51:40.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.845+0000 7fd94759e700 1 -- 172.21.15.77:0/1787292596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9481690a0 con 0x7fd94806c500 2022-04-03T19:51:40.846 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.846+0000 7fd93e7fc700 1 -- 172.21.15.77:0/1787292596 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6113 (secure 0 0 0) 0x7fd940055230 con 0x7fd94806c500 2022-04-03T19:51:40.846 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:51:40.847 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5239,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:50:23.308013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:51:40.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.848+0000 7fd94759e700 1 -- 172.21.15.77:0/1787292596 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd92c063ce0 msgr2=0x7fd92c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:40.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.848+0000 7fd94759e700 1 --2- 172.21.15.77:0/1787292596 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd92c063ce0 0x7fd92c066190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd938009d20 tx=0x7fd938009060 comp rx=0 tx=0).stop 2022-04-03T19:51:40.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.848+0000 7fd94759e700 1 -- 172.21.15.77:0/1787292596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd94806c500 msgr2=0x7fd9480c32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:40.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.848+0000 7fd94759e700 1 --2- 172.21.15.77:0/1787292596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd94806c500 0x7fd9480c32f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd940003b90 tx=0x7fd940019000 comp rx=0 tx=0).stop 2022-04-03T19:51:40.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.848+0000 7fd94759e700 1 -- 172.21.15.77:0/1787292596 shutdown_connections 2022-04-03T19:51:40.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.848+0000 7fd94759e700 1 --2- 172.21.15.77:0/1787292596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd94806c500 0x7fd9480c32f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:40.851 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.848+0000 7fd94759e700 1 --2- 172.21.15.77:0/1787292596 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd92c063ce0 0x7fd92c066190 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:40.851 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.848+0000 7fd94759e700 1 --2- 172.21.15.77:0/1787292596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd94806bbe0 0x7fd9480c08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:40.851 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.848+0000 7fd94759e700 1 --2- 172.21.15.77:0/1787292596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94806bfb0 0x7fd9480c2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:40.851 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.848+0000 7fd94759e700 1 -- 172.21.15.77:0/1787292596 >> 172.21.15.77:0/1787292596 conn(0x7fd9480cb6b0 msgr2=0x7fd94814d320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:40.851 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.848+0000 7fd94759e700 1 -- 172.21.15.77:0/1787292596 shutdown_connections 2022-04-03T19:51:40.852 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:40.848+0000 7fd94759e700 1 -- 172.21.15.77:0/1787292596 wait complete. 2022-04-03T19:51:40.852 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2022-04-03T19:51:40.873 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:51:42.874 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:51:43.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.080+0000 7f31fe5de700 1 Processor -- start 2022-04-03T19:51:43.081 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.080+0000 7f31fe5de700 1 -- start start 2022-04-03T19:51:43.084 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.081+0000 7f31fe5de700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f31f80c2a90 0x7f31f806c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:43.084 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.081+0000 7f31fe5de700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31f81562f0 con 0x7f31f80c5930 2022-04-03T19:51:43.084 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.081+0000 7f31fe5de700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31f80c2f80 con 0x7f31f806d3f0 2022-04-03T19:51:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.081+0000 7f31fe5de700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f80c3260 con 0x7f31f80c2a90 2022-04-03T19:51:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.082+0000 7f31fd5dc700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f31f806d3f0 0x7f31f806d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44272/0 (socket says 172.21.15.77:44272) 2022-04-03T19:51:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.082+0000 7f31fd5dc700 1 -- 172.21.15.77:0/3973249795 learned_addr learned my addr 172.21.15.77:0/3973249795 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:51:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.082+0000 7f31fdddd700 1 --2- 172.21.15.77:0/3973249795 >> v2:172.21.15.77:3300/0 conn(0x7f31f80c2a90 0x7f31f806c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.082+0000 7f31e7fff700 1 -- 172.21.15.77:0/3973249795 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 725402708 0 0) 0x7f31f80c2f80 con 0x7f31f806d3f0 2022-04-03T19:51:43.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.082+0000 7f31e7fff700 1 -- 172.21.15.77:0/3973249795 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31e00033f0 con 0x7f31f806d3f0 2022-04-03T19:51:43.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.082+0000 7f31fdddd700 1 -- 172.21.15.77:0/3973249795 >> v1:172.21.15.186:6789/0 conn(0x7f31f806d3f0 legacy=0x7f31f806d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:43.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.082+0000 7f31e7fff700 1 -- 172.21.15.77:0/3973249795 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019400451 0 0) 0x7f31f81562f0 con 0x7f31f80c5930 2022-04-03T19:51:43.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.082+0000 7f31fdddd700 1 -- 172.21.15.77:0/3973249795 >> v1:172.21.15.77:6789/0 conn(0x7f31f80c5930 legacy=0x7f31f8155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.082+0000 7f31fdddd700 1 -- 172.21.15.77:0/3973249795 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f80c3540 con 0x7f31f80c2a90 2022-04-03T19:51:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.083+0000 7f31fdddd700 1 --2- 172.21.15.77:0/3973249795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f80c2a90 0x7f31f806c040 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f31f0009e90 tx=0x7f31f0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf9e414c5f2bd75b server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.083+0000 7f31e7fff700 1 -- 172.21.15.77:0/3973249795 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f000e070 con 0x7f31f80c2a90 2022-04-03T19:51:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.083+0000 7f31e7fff700 1 -- 172.21.15.77:0/3973249795 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f000bcd0 con 0x7f31f80c2a90 2022-04-03T19:51:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.083+0000 7f31e7fff700 1 -- 172.21.15.77:0/3973249795 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f0012680 con 0x7f31f80c2a90 2022-04-03T19:51:43.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.083+0000 7f31fe5de700 1 -- 172.21.15.77:0/3973249795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f80c2a90 msgr2=0x7f31f806c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:43.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.083+0000 7f31fe5de700 1 --2- 172.21.15.77:0/3973249795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f80c2a90 0x7f31f806c040 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f31f0009e90 tx=0x7f31f0009780 comp rx=0 tx=0).stop 2022-04-03T19:51:43.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.083+0000 7f31fe5de700 1 -- 172.21.15.77:0/3973249795 shutdown_connections 2022-04-03T19:51:43.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.083+0000 7f31fe5de700 1 --2- 172.21.15.77:0/3973249795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f80c2a90 0x7f31f806c040 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:43.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.083+0000 7f31fe5de700 1 -- 172.21.15.77:0/3973249795 >> 172.21.15.77:0/3973249795 conn(0x7f31f80cb720 msgr2=0x7f31f80cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:43.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.084+0000 7f31fe5de700 1 -- 172.21.15.77:0/3973249795 shutdown_connections 2022-04-03T19:51:43.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.084+0000 7f31fe5de700 1 -- 172.21.15.77:0/3973249795 wait complete. 2022-04-03T19:51:43.092 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.085+0000 7f31fe5de700 1 Processor -- start 2022-04-03T19:51:43.093 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.085+0000 7f31fe5de700 1 -- start start 2022-04-03T19:51:43.093 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.086+0000 7f31fe5de700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31f806d3f0 0x7f31f8152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:43.093 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.086+0000 7f31fe5de700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31f80c2a90 0x7f31f8151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:43.093 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.086+0000 7f31fe5de700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f80c5930 0x7f31f8154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:43.093 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.086+0000 7f31fe5de700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f8158920 con 0x7f31f80c5930 2022-04-03T19:51:43.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.086+0000 7f31fe5de700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f8151010 con 0x7f31f80c2a90 2022-04-03T19:51:43.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.086+0000 7f31fe5de700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f8166030 con 0x7f31f806d3f0 2022-04-03T19:51:43.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.086+0000 7f31fdddd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f80c5930 0x7f31f8154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:43.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.086+0000 7f31fd5dc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31f806d3f0 0x7f31f8152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:43.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.086+0000 7f31fdddd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f80c5930 0x7f31f8154380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42992/0 (socket says 172.21.15.77:42992) 2022-04-03T19:51:43.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.086+0000 7f31fd5dc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31f806d3f0 0x7f31f8152a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60274/0 (socket says 172.21.15.77:60274) 2022-04-03T19:51:43.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.086+0000 7f31fcddb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31f80c2a90 0x7f31f8151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:43.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.086+0000 7f31fdddd700 1 -- 172.21.15.77:0/430927337 learned_addr learned my addr 172.21.15.77:0/430927337 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:51:43.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.087+0000 7f31fdddd700 1 -- 172.21.15.77:0/430927337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31f806d3f0 msgr2=0x7f31f8152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:43.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.087+0000 7f31fdddd700 1 --2- 172.21.15.77:0/430927337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31f806d3f0 0x7f31f8152a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:43.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.087+0000 7f31fdddd700 1 -- 172.21.15.77:0/430927337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31f80c2a90 msgr2=0x7f31f8151450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:43.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.087+0000 7f31fdddd700 1 --2- 172.21.15.77:0/430927337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31f80c2a90 0x7f31f8151450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:43.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.087+0000 7f31fdddd700 1 -- 172.21.15.77:0/430927337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f0009a20 con 0x7f31f80c5930 2022-04-03T19:51:43.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.087+0000 7f31fd5dc700 1 --2- 172.21.15.77:0/430927337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31f806d3f0 0x7f31f8152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:51:43.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.087+0000 7f31fdddd700 1 --2- 172.21.15.77:0/430927337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f80c5930 0x7f31f8154380 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f31f80890b0 tx=0x7f31f0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:43.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.087+0000 7f31e57fa700 1 -- 172.21.15.77:0/430927337 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f000be40 con 0x7f31f80c5930 2022-04-03T19:51:43.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.088+0000 7f31e57fa700 1 -- 172.21.15.77:0/430927337 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f001c610 con 0x7f31f80c5930 2022-04-03T19:51:43.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.088+0000 7f31fe5de700 1 -- 172.21.15.77:0/430927337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f80c2680 con 0x7f31f80c5930 2022-04-03T19:51:43.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.088+0000 7f31e57fa700 1 -- 172.21.15.77:0/430927337 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f001c9d0 con 0x7f31f80c5930 2022-04-03T19:51:43.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.088+0000 7f31fe5de700 1 -- 172.21.15.77:0/430927337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f8151ae0 con 0x7f31f80c5930 2022-04-03T19:51:43.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.089+0000 7f31fe5de700 1 -- 172.21.15.77:0/430927337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f81661c0 con 0x7f31f80c5930 2022-04-03T19:51:43.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.090+0000 7f31e57fa700 1 -- 172.21.15.77:0/430927337 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f31f001cd40 con 0x7f31f80c5930 2022-04-03T19:51:43.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.090+0000 7f31e57fa700 1 --2- 172.21.15.77:0/430927337 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f31d8063d10 0x7f31d80661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:43.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.090+0000 7f31e57fa700 1 -- 172.21.15.77:0/430927337 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6402+0+0 (secure 0 0 0) 0x7f31f00825d0 con 0x7f31f80c5930 2022-04-03T19:51:43.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.091+0000 7f31fd5dc700 1 --2- 172.21.15.77:0/430927337 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f31d8063d10 0x7f31d80661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:43.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.091+0000 7f31fd5dc700 1 --2- 172.21.15.77:0/430927337 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f31d8063d10 0x7f31d80661c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f31f40011e0 tx=0x7f31f4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:43.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.094+0000 7f31e57fa700 1 -- 172.21.15.77:0/430927337 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f31f0051050 con 0x7f31f80c5930 2022-04-03T19:51:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.322+0000 7f31fe5de700 1 -- 172.21.15.77:0/430927337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31f804b050 con 0x7f31f80c5930 2022-04-03T19:51:43.324 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.323+0000 7f31e57fa700 1 -- 172.21.15.77:0/430927337 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6113 (secure 0 0 0) 0x7f31f0054670 con 0x7f31f80c5930 2022-04-03T19:51:43.324 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:51:43.324 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5239,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:50:23.308013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:51:43.326 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.325+0000 7f31fe5de700 1 -- 172.21.15.77:0/430927337 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f31d8063d10 msgr2=0x7f31d80661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:43.326 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.325+0000 7f31fe5de700 1 --2- 172.21.15.77:0/430927337 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f31d8063d10 0x7f31d80661c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f31f40011e0 tx=0x7f31f4009040 comp rx=0 tx=0).stop 2022-04-03T19:51:43.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.325+0000 7f31fe5de700 1 -- 172.21.15.77:0/430927337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f80c5930 msgr2=0x7f31f8154380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:43.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.325+0000 7f31fe5de700 1 --2- 172.21.15.77:0/430927337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f80c5930 0x7f31f8154380 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f31f80890b0 tx=0x7f31f0019000 comp rx=0 tx=0).stop 2022-04-03T19:51:43.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.326+0000 7f31fe5de700 1 -- 172.21.15.77:0/430927337 shutdown_connections 2022-04-03T19:51:43.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.326+0000 7f31fe5de700 1 --2- 172.21.15.77:0/430927337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31f80c5930 0x7f31f8154380 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:43.328 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.326+0000 7f31fe5de700 1 --2- 172.21.15.77:0/430927337 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f31d8063d10 0x7f31d80661c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:43.328 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.326+0000 7f31fe5de700 1 --2- 172.21.15.77:0/430927337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31f806d3f0 0x7f31f8152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:43.328 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.326+0000 7f31fe5de700 1 --2- 172.21.15.77:0/430927337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31f80c2a90 0x7f31f8151450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:43.328 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.326+0000 7f31fe5de700 1 -- 172.21.15.77:0/430927337 >> 172.21.15.77:0/430927337 conn(0x7f31f80cb720 msgr2=0x7f31f80ca810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:43.328 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.326+0000 7f31fe5de700 1 -- 172.21.15.77:0/430927337 shutdown_connections 2022-04-03T19:51:43.329 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:43.326+0000 7f31fe5de700 1 -- 172.21.15.77:0/430927337 wait complete. 2022-04-03T19:51:43.329 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2022-04-03T19:51:43.350 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:51:45.351 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:51:45.556 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.555+0000 7f099ffff700 1 Processor -- start 2022-04-03T19:51:45.556 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.555+0000 7f099ffff700 1 -- start start 2022-04-03T19:51:45.557 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.557+0000 7f099ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f09a00af230 0x7f09a0067360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:45.558 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.557+0000 7f099ffff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f09a00c5760 0x7f09a00678a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:45.558 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.557+0000 7f099ffff700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a0066ed0 con 0x7f09a006ccf0 2022-04-03T19:51:45.558 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.557+0000 7f099ffff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a006d930 con 0x7f09a00af230 2022-04-03T19:51:45.559 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.557+0000 7f099ffff700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a006e380 con 0x7f09a00c5760 2022-04-03T19:51:45.559 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.557+0000 7f099f7fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f09a00c5760 0x7f09a00678a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:45.559 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.557+0000 7f0997fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f09a00af230 0x7f09a0067360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:45.559 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.557+0000 7f099effd700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f09a006ccf0 0x7f09a00aeb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44274/0 (socket says 172.21.15.77:44274) 2022-04-03T19:51:45.560 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.557+0000 7f099effd700 1 -- 172.21.15.77:0/756200618 learned_addr learned my addr 172.21.15.77:0/756200618 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:51:45.563 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.557+0000 7f099f7fe700 1 -- 172.21.15.77:0/756200618 >> v2:172.21.15.77:3300/0 conn(0x7f09a00af230 msgr2=0x7f09a0067360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:51:45.564 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.557+0000 7f099f7fe700 1 --2- 172.21.15.77:0/756200618 >> v2:172.21.15.77:3300/0 conn(0x7f09a00af230 0x7f09a0067360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:45.564 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.558+0000 7f099f7fe700 1 -- 172.21.15.77:0/756200618 >> v1:172.21.15.186:6789/0 conn(0x7f09a006ccf0 legacy=0x7f09a00aeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:45.564 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.558+0000 7f099f7fe700 1 -- 172.21.15.77:0/756200618 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a00c6790 con 0x7f09a00c5760 2022-04-03T19:51:45.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.558+0000 7f099f7fe700 1 --2- 172.21.15.77:0/756200618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09a00c5760 0x7f09a00678a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f09a00c6fa0 tx=0x7f0998009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79b535ece8fba298 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:45.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.558+0000 7f099e7fc700 1 -- 172.21.15.77:0/756200618 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099800e070 con 0x7f09a00c5760 2022-04-03T19:51:45.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.558+0000 7f099e7fc700 1 -- 172.21.15.77:0/756200618 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099800bbf0 con 0x7f09a00c5760 2022-04-03T19:51:45.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.558+0000 7f099e7fc700 1 -- 172.21.15.77:0/756200618 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0998012430 con 0x7f09a00c5760 2022-04-03T19:51:45.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.558+0000 7f099ffff700 1 -- 172.21.15.77:0/756200618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09a00c5760 msgr2=0x7f09a00678a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:45.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.558+0000 7f099ffff700 1 --2- 172.21.15.77:0/756200618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09a00c5760 0x7f09a00678a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f09a00c6fa0 tx=0x7f0998009780 comp rx=0 tx=0).stop 2022-04-03T19:51:45.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.559+0000 7f099ffff700 1 -- 172.21.15.77:0/756200618 shutdown_connections 2022-04-03T19:51:45.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.559+0000 7f099ffff700 1 --2- 172.21.15.77:0/756200618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09a00c5760 0x7f09a00678a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:45.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.559+0000 7f099ffff700 1 --2- 172.21.15.77:0/756200618 >> v2:172.21.15.77:3300/0 conn(0x7f09a00af230 0x7f09a0067360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:45.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.559+0000 7f099ffff700 1 -- 172.21.15.77:0/756200618 >> 172.21.15.77:0/756200618 conn(0x7f09a0069990 msgr2=0x7f09a00626d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:45.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.559+0000 7f099ffff700 1 -- 172.21.15.77:0/756200618 shutdown_connections 2022-04-03T19:51:45.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.559+0000 7f099ffff700 1 -- 172.21.15.77:0/756200618 wait complete. 2022-04-03T19:51:45.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.560+0000 7f099ffff700 1 Processor -- start 2022-04-03T19:51:45.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.560+0000 7f099ffff700 1 -- start start 2022-04-03T19:51:45.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.561+0000 7f099ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09a006ccf0 0x7f09a00aa270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:45.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.561+0000 7f099ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09a00af230 0x7f09a00c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:45.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.561+0000 7f099ffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09a00c5760 0x7f09a01546f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:45.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.561+0000 7f099ffff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a00ab6b0 con 0x7f09a00af230 2022-04-03T19:51:45.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.561+0000 7f099ffff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a00bec40 con 0x7f09a006ccf0 2022-04-03T19:51:45.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.561+0000 7f099ffff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a00be770 con 0x7f09a00c5760 2022-04-03T19:51:45.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.562+0000 7f099effd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09a006ccf0 0x7f09a00aa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:45.570 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.562+0000 7f0997fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09a00af230 0x7f09a00c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:45.570 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.562+0000 7f099effd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09a006ccf0 0x7f09a00aa270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38184/0 (socket says 172.21.15.77:38184) 2022-04-03T19:51:45.570 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.562+0000 7f099effd700 1 -- 172.21.15.77:0/1685998120 learned_addr learned my addr 172.21.15.77:0/1685998120 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:51:45.570 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.562+0000 7f099f7fe700 1 --2- 172.21.15.77:0/1685998120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09a00c5760 0x7f09a01546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:45.571 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.562+0000 7f0997fff700 1 -- 172.21.15.77:0/1685998120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09a00c5760 msgr2=0x7f09a01546f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:45.571 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.562+0000 7f0997fff700 1 --2- 172.21.15.77:0/1685998120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09a00c5760 0x7f09a01546f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:45.571 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.562+0000 7f0997fff700 1 -- 172.21.15.77:0/1685998120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09a006ccf0 msgr2=0x7f09a00aa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:45.571 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.562+0000 7f0997fff700 1 --2- 172.21.15.77:0/1685998120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09a006ccf0 0x7f09a00aa270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:45.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.562+0000 7f0997fff700 1 -- 172.21.15.77:0/1685998120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0998009a20 con 0x7f09a00af230 2022-04-03T19:51:45.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.562+0000 7f099effd700 1 --2- 172.21.15.77:0/1685998120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09a006ccf0 0x7f09a00aa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:51:45.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.563+0000 7f0997fff700 1 --2- 172.21.15.77:0/1685998120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09a00af230 0x7f09a00c1eb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f098800e900 tx=0x7f0988009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:45.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.563+0000 7f0996ffd700 1 -- 172.21.15.77:0/1685998120 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0988013680 con 0x7f09a00af230 2022-04-03T19:51:45.573 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.563+0000 7f0996ffd700 1 -- 172.21.15.77:0/1685998120 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09880137e0 con 0x7f09a00af230 2022-04-03T19:51:45.573 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.563+0000 7f0996ffd700 1 -- 172.21.15.77:0/1685998120 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0988013b80 con 0x7f09a00af230 2022-04-03T19:51:45.573 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.563+0000 7f099ffff700 1 -- 172.21.15.77:0/1685998120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a00be490 con 0x7f09a00af230 2022-04-03T19:51:45.573 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.563+0000 7f099ffff700 1 -- 172.21.15.77:0/1685998120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a00c8e90 con 0x7f09a00af230 2022-04-03T19:51:45.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.564+0000 7f099ffff700 1 -- 172.21.15.77:0/1685998120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a00c90b0 con 0x7f09a00af230 2022-04-03T19:51:45.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.565+0000 7f0996ffd700 1 -- 172.21.15.77:0/1685998120 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f0988019030 con 0x7f09a00af230 2022-04-03T19:51:45.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.566+0000 7f0996ffd700 1 --2- 172.21.15.77:0/1685998120 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0984063ce0 0x7f0984066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:45.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.566+0000 7f0996ffd700 1 -- 172.21.15.77:0/1685998120 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6402+0+0 (secure 0 0 0) 0x7f0988080920 con 0x7f09a00af230 2022-04-03T19:51:45.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.566+0000 7f099effd700 1 --2- 172.21.15.77:0/1685998120 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0984063ce0 0x7f0984066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:45.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.567+0000 7f099effd700 1 --2- 172.21.15.77:0/1685998120 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0984063ce0 0x7f0984066190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0990007fd0 tx=0x7f0990007df0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:45.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.570+0000 7f0996ffd700 1 -- 172.21.15.77:0/1685998120 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f098804f3a0 con 0x7f09a00af230 2022-04-03T19:51:45.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.802+0000 7f099ffff700 1 -- 172.21.15.77:0/1685998120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09a004b050 con 0x7f09a00af230 2022-04-03T19:51:45.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.803+0000 7f0996ffd700 1 -- 172.21.15.77:0/1685998120 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6113 (secure 0 0 0) 0x7f0988017090 con 0x7f09a00af230 2022-04-03T19:51:45.804 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:51:45.804 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5239,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:50:23.308013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:51:45.806 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.806+0000 7f099ffff700 1 -- 172.21.15.77:0/1685998120 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0984063ce0 msgr2=0x7f0984066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:45.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.806+0000 7f099ffff700 1 --2- 172.21.15.77:0/1685998120 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0984063ce0 0x7f0984066190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0990007fd0 tx=0x7f0990007df0 comp rx=0 tx=0).stop 2022-04-03T19:51:45.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.806+0000 7f099ffff700 1 -- 172.21.15.77:0/1685998120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09a00af230 msgr2=0x7f09a00c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:45.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.806+0000 7f099ffff700 1 --2- 172.21.15.77:0/1685998120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09a00af230 0x7f09a00c1eb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f098800e900 tx=0x7f0988009060 comp rx=0 tx=0).stop 2022-04-03T19:51:45.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.806+0000 7f099ffff700 1 -- 172.21.15.77:0/1685998120 shutdown_connections 2022-04-03T19:51:45.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.806+0000 7f099ffff700 1 --2- 172.21.15.77:0/1685998120 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0984063ce0 0x7f0984066190 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:45.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.806+0000 7f099ffff700 1 --2- 172.21.15.77:0/1685998120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09a00c5760 0x7f09a01546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:45.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.806+0000 7f099ffff700 1 --2- 172.21.15.77:0/1685998120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09a006ccf0 0x7f09a00aa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:45.809 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.806+0000 7f099ffff700 1 --2- 172.21.15.77:0/1685998120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09a00af230 0x7f09a00c1eb0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:45.809 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.806+0000 7f099ffff700 1 -- 172.21.15.77:0/1685998120 >> 172.21.15.77:0/1685998120 conn(0x7f09a0069990 msgr2=0x7f09a00c7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:45.809 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.806+0000 7f099ffff700 1 -- 172.21.15.77:0/1685998120 shutdown_connections 2022-04-03T19:51:45.809 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:45.806+0000 7f099ffff700 1 -- 172.21.15.77:0/1685998120 wait complete. 2022-04-03T19:51:45.810 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2022-04-03T19:51:45.830 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:51:46.352 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:51:46.357 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:51:47.831 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:51:48.037 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.036+0000 7f74e9591700 1 Processor -- start 2022-04-03T19:51:48.116 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.037+0000 7f74e9591700 1 -- start start 2022-04-03T19:51:48.122 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.038+0000 7f74e9591700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f74e406bbe0 0x7f74e41584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:48.122 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.038+0000 7f74e9591700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f74e406c500 0x7f74e414eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:48.122 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.038+0000 7f74e9591700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e406d260 con 0x7f74e406bfb0 2022-04-03T19:51:48.123 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.038+0000 7f74e9591700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74e4159450 con 0x7f74e406bbe0 2022-04-03T19:51:48.123 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.038+0000 7f74e9591700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74e40c6330 con 0x7f74e406c500 2022-04-03T19:51:48.123 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.038+0000 7f74e3fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f74e406bbe0 0x7f74e41584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:48.123 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.038+0000 7f74e3fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f74e406bbe0 0x7f74e41584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42998/0 (socket says 172.21.15.77:42998) 2022-04-03T19:51:48.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.038+0000 7f74e3fff700 1 -- 172.21.15.77:0/175858861 learned_addr learned my addr 172.21.15.77:0/175858861 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:51:48.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.038+0000 7f74e8d90700 1 --2- 172.21.15.77:0/175858861 >> v2:172.21.15.186:3300/0 conn(0x7f74e406c500 0x7f74e414eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:48.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.039+0000 7f74e3fff700 1 -- 172.21.15.77:0/175858861 >> v2:172.21.15.186:3300/0 conn(0x7f74e406c500 msgr2=0x7f74e414eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:51:48.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.039+0000 7f74e3fff700 1 --2- 172.21.15.77:0/175858861 >> v2:172.21.15.186:3300/0 conn(0x7f74e406c500 0x7f74e414eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:48.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.039+0000 7f74e3fff700 1 -- 172.21.15.77:0/175858861 >> v1:172.21.15.186:6789/0 conn(0x7f74e406bfb0 legacy=0x7f74e406d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:48.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.039+0000 7f74e3fff700 1 -- 172.21.15.77:0/175858861 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e4158bb0 con 0x7f74e406bbe0 2022-04-03T19:51:48.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.039+0000 7f74e3fff700 1 --2- 172.21.15.77:0/175858861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e406bbe0 0x7f74e41584b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f74e40cc6b0 tx=0x7f74d40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e90a48259feb63a server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:48.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.039+0000 7f74e2ffd700 1 -- 172.21.15.77:0/175858861 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d4004030 con 0x7f74e406bbe0 2022-04-03T19:51:48.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.039+0000 7f74e2ffd700 1 -- 172.21.15.77:0/175858861 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d4004190 con 0x7f74e406bbe0 2022-04-03T19:51:48.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.039+0000 7f74e2ffd700 1 -- 172.21.15.77:0/175858861 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d4011630 con 0x7f74e406bbe0 2022-04-03T19:51:48.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.040+0000 7f74e9591700 1 -- 172.21.15.77:0/175858861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e406bbe0 msgr2=0x7f74e41584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:48.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.040+0000 7f74e9591700 1 --2- 172.21.15.77:0/175858861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e406bbe0 0x7f74e41584b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f74e40cc6b0 tx=0x7f74d40090f0 comp rx=0 tx=0).stop 2022-04-03T19:51:48.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.040+0000 7f74e9591700 1 -- 172.21.15.77:0/175858861 shutdown_connections 2022-04-03T19:51:48.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.040+0000 7f74e9591700 1 --2- 172.21.15.77:0/175858861 >> v2:172.21.15.186:3300/0 conn(0x7f74e406c500 0x7f74e414eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:48.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.040+0000 7f74e9591700 1 --2- 172.21.15.77:0/175858861 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e406bbe0 0x7f74e41584b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:48.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.040+0000 7f74e9591700 1 -- 172.21.15.77:0/175858861 >> 172.21.15.77:0/175858861 conn(0x7f74e40cb6b0 msgr2=0x7f74e40cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:48.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.040+0000 7f74e9591700 1 -- 172.21.15.77:0/175858861 shutdown_connections 2022-04-03T19:51:48.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.040+0000 7f74e9591700 1 -- 172.21.15.77:0/175858861 wait complete. 2022-04-03T19:51:48.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.042+0000 7f74e9591700 1 Processor -- start 2022-04-03T19:51:48.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.042+0000 7f74e9591700 1 -- start start 2022-04-03T19:51:48.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.043+0000 7f74e9591700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e406bbe0 0x7f74e4158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:48.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.043+0000 7f74e9591700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74e406bfb0 0x7f74e4154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:48.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.043+0000 7f74e9591700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74e406c500 0x7f74e4150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:48.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.043+0000 7f74e9591700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e414af90 con 0x7f74e406bbe0 2022-04-03T19:51:48.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.043+0000 7f74e9591700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e41502d0 con 0x7f74e406c500 2022-04-03T19:51:48.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.043+0000 7f74e9591700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e4166310 con 0x7f74e406bfb0 2022-04-03T19:51:48.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.043+0000 7f74e3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e406bbe0 0x7f74e4158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:48.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.043+0000 7f74e37fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74e406bfb0 0x7f74e4154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:48.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.043+0000 7f74e8d90700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74e406c500 0x7f74e4150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:48.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.043+0000 7f74e37fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74e406bfb0 0x7f74e4154ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60278/0 (socket says 172.21.15.77:60278) 2022-04-03T19:51:48.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.043+0000 7f74e37fe700 1 -- 172.21.15.77:0/834165506 learned_addr learned my addr 172.21.15.77:0/834165506 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:51:48.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.044+0000 7f74e3fff700 1 -- 172.21.15.77:0/834165506 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74e406bfb0 msgr2=0x7f74e4154ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:48.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.044+0000 7f74e3fff700 1 --2- 172.21.15.77:0/834165506 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74e406bfb0 0x7f74e4154ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:48.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.044+0000 7f74e3fff700 1 -- 172.21.15.77:0/834165506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74e406c500 msgr2=0x7f74e4150760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:48.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.044+0000 7f74e3fff700 1 --2- 172.21.15.77:0/834165506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74e406c500 0x7f74e4150760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:48.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.044+0000 7f74e3fff700 1 -- 172.21.15.77:0/834165506 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d4009370 con 0x7f74e406bbe0 2022-04-03T19:51:48.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.044+0000 7f74e37fe700 1 --2- 172.21.15.77:0/834165506 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74e406bfb0 0x7f74e4154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:51:48.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.044+0000 7f74e3fff700 1 --2- 172.21.15.77:0/834165506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e406bbe0 0x7f74e4158250 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f74d4009340 tx=0x7f74d4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:48.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.044+0000 7f74cbfff700 1 -- 172.21.15.77:0/834165506 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d40036f0 con 0x7f74e406bbe0 2022-04-03T19:51:48.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.044+0000 7f74cbfff700 1 -- 172.21.15.77:0/834165506 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d4003850 con 0x7f74e406bbe0 2022-04-03T19:51:48.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.044+0000 7f74e9591700 1 -- 172.21.15.77:0/834165506 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e414f9e0 con 0x7f74e406bbe0 2022-04-03T19:51:48.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.044+0000 7f74cbfff700 1 -- 172.21.15.77:0/834165506 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d4003c10 con 0x7f74e406bbe0 2022-04-03T19:51:48.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.044+0000 7f74e9591700 1 -- 172.21.15.77:0/834165506 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e414fc80 con 0x7f74e406bbe0 2022-04-03T19:51:48.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.045+0000 7f74e9591700 1 -- 172.21.15.77:0/834165506 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74e41505b0 con 0x7f74e406bbe0 2022-04-03T19:51:48.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.046+0000 7f74cbfff700 1 -- 172.21.15.77:0/834165506 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f74d4017020 con 0x7f74e406bbe0 2022-04-03T19:51:48.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.047+0000 7f74cbfff700 1 --2- 172.21.15.77:0/834165506 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f74c4063d10 0x7f74c40661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:48.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.047+0000 7f74cbfff700 1 -- 172.21.15.77:0/834165506 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6402+0+0 (secure 0 0 0) 0x7f74d4082180 con 0x7f74e406bbe0 2022-04-03T19:51:48.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.048+0000 7f74e37fe700 1 --2- 172.21.15.77:0/834165506 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f74c4063d10 0x7f74c40661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:48.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.048+0000 7f74e37fe700 1 --2- 172.21.15.77:0/834165506 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f74c4063d10 0x7f74c40661c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f74e414efd0 tx=0x7f74d8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:48.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.051+0000 7f74cbfff700 1 -- 172.21.15.77:0/834165506 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f74d4050b10 con 0x7f74e406bbe0 2022-04-03T19:51:48.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.286+0000 7f74e9591700 1 -- 172.21.15.77:0/834165506 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74e404b050 con 0x7f74e406bbe0 2022-04-03T19:51:48.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.287+0000 7f74cbfff700 1 -- 172.21.15.77:0/834165506 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6113 (secure 0 0 0) 0x7f74d4054130 con 0x7f74e406bbe0 2022-04-03T19:51:48.288 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:51:48.288 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5239,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:50:23.308013+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4850},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4850":{"gid":4850,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":27,"addr":"172.21.15.186:6837/3568325507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":3568325507},{"type":"v1","addr":"172.21.15.186:6837","nonce":3568325507}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:51:48.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.289+0000 7f74e9591700 1 -- 172.21.15.77:0/834165506 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f74c4063d10 msgr2=0x7f74c40661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:48.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.289+0000 7f74e9591700 1 --2- 172.21.15.77:0/834165506 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f74c4063d10 0x7f74c40661c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f74e414efd0 tx=0x7f74d8009040 comp rx=0 tx=0).stop 2022-04-03T19:51:48.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.289+0000 7f74e9591700 1 -- 172.21.15.77:0/834165506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e406bbe0 msgr2=0x7f74e4158250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:48.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.289+0000 7f74e9591700 1 --2- 172.21.15.77:0/834165506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e406bbe0 0x7f74e4158250 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f74d4009340 tx=0x7f74d4016000 comp rx=0 tx=0).stop 2022-04-03T19:51:48.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.289+0000 7f74e9591700 1 -- 172.21.15.77:0/834165506 shutdown_connections 2022-04-03T19:51:48.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.289+0000 7f74e9591700 1 --2- 172.21.15.77:0/834165506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74e406c500 0x7f74e4150760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:48.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.289+0000 7f74e9591700 1 --2- 172.21.15.77:0/834165506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74e406bbe0 0x7f74e4158250 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:48.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.289+0000 7f74e9591700 1 --2- 172.21.15.77:0/834165506 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f74c4063d10 0x7f74c40661c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:48.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.289+0000 7f74e9591700 1 --2- 172.21.15.77:0/834165506 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74e406bfb0 0x7f74e4154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:48.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.289+0000 7f74e9591700 1 -- 172.21.15.77:0/834165506 >> 172.21.15.77:0/834165506 conn(0x7f74e40cb6b0 msgr2=0x7f74e40cba80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:48.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.289+0000 7f74e9591700 1 -- 172.21.15.77:0/834165506 shutdown_connections 2022-04-03T19:51:48.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:48.290+0000 7f74e9591700 1 -- 172.21.15.77:0/834165506 wait complete. 2022-04-03T19:51:48.293 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 26 2022-04-03T19:51:48.314 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:51:50.315 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:51:50.522 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.522+0000 7f9e893a2700 1 Processor -- start 2022-04-03T19:51:50.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.522+0000 7f9e893a2700 1 -- start start 2022-04-03T19:51:50.525 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.523+0000 7f9e893a2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e84155370 0x7f9e84069110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:50.525 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.523+0000 7f9e893a2700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e840c3710 con 0x7f9e840c1170 2022-04-03T19:51:50.525 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.523+0000 7f9e893a2700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e84152330 con 0x7f9e840c3b10 2022-04-03T19:51:50.525 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.523+0000 7f9e893a2700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e840c1540 con 0x7f9e84155370 2022-04-03T19:51:50.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.524+0000 7f9e83fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9e840c1170 0x7f9e841527c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38490/0 (socket says 172.21.15.77:38490) 2022-04-03T19:51:50.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.524+0000 7f9e83fff700 1 -- 172.21.15.77:0/2110991590 learned_addr learned my addr 172.21.15.77:0/2110991590 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:51:50.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.524+0000 7f9e837fe700 1 --2- 172.21.15.77:0/2110991590 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e84155370 0x7f9e84069110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:50.526 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.524+0000 7f9e837fe700 1 -- 172.21.15.77:0/2110991590 >> v1:172.21.15.186:6789/0 conn(0x7f9e840c3b10 legacy=0x7f9e8415c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:50.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.524+0000 7f9e837fe700 1 -- 172.21.15.77:0/2110991590 >> v1:172.21.15.77:6789/0 conn(0x7f9e840c1170 legacy=0x7f9e841527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:50.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.524+0000 7f9e837fe700 1 -- 172.21.15.77:0/2110991590 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e840c4270 con 0x7f9e84155370 2022-04-03T19:51:50.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.524+0000 7f9e82ffd700 1 -- 172.21.15.77:0/2110991590 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437616280 0 0) 0x7f9e840c3710 con 0x7f9e840c1170 2022-04-03T19:51:50.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.525+0000 7f9e837fe700 1 --2- 172.21.15.77:0/2110991590 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e84155370 0x7f9e84069110 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9e78009710 tx=0x7f9e78009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=861f1c5fb0856739 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:50.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.525+0000 7f9e82ffd700 1 -- 172.21.15.77:0/2110991590 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e78004030 con 0x7f9e84155370 2022-04-03T19:51:50.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.525+0000 7f9e82ffd700 1 -- 172.21.15.77:0/2110991590 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e78004190 con 0x7f9e84155370 2022-04-03T19:51:50.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.525+0000 7f9e82ffd700 1 -- 172.21.15.77:0/2110991590 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e78011630 con 0x7f9e84155370 2022-04-03T19:51:50.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.525+0000 7f9e893a2700 1 -- 172.21.15.77:0/2110991590 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e84155370 msgr2=0x7f9e84069110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:50.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.525+0000 7f9e893a2700 1 --2- 172.21.15.77:0/2110991590 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e84155370 0x7f9e84069110 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9e78009710 tx=0x7f9e78009060 comp rx=0 tx=0).stop 2022-04-03T19:51:50.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.525+0000 7f9e893a2700 1 -- 172.21.15.77:0/2110991590 shutdown_connections 2022-04-03T19:51:50.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.525+0000 7f9e893a2700 1 --2- 172.21.15.77:0/2110991590 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e84155370 0x7f9e84069110 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:50.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.526+0000 7f9e893a2700 1 -- 172.21.15.77:0/2110991590 >> 172.21.15.77:0/2110991590 conn(0x7f9e84066da0 msgr2=0x7f9e840671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:50.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.526+0000 7f9e893a2700 1 -- 172.21.15.77:0/2110991590 shutdown_connections 2022-04-03T19:51:50.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.526+0000 7f9e893a2700 1 -- 172.21.15.77:0/2110991590 wait complete. 2022-04-03T19:51:50.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.527+0000 7f9e893a2700 1 Processor -- start 2022-04-03T19:51:50.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.527+0000 7f9e893a2700 1 -- start start 2022-04-03T19:51:50.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.528+0000 7f9e893a2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e840c1170 0x7f9e8414c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:50.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.528+0000 7f9e893a2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e840c3b10 0x7f9e841543d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:50.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.528+0000 7f9e893a2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e84155370 0x7f9e841687b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:50.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.528+0000 7f9e893a2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e84150e00 con 0x7f9e840c1170 2022-04-03T19:51:50.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.528+0000 7f9e893a2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e841539e0 con 0x7f9e840c3b10 2022-04-03T19:51:50.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.528+0000 7f9e893a2700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e84150b20 con 0x7f9e84155370 2022-04-03T19:51:50.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.529+0000 7f9e83fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e840c1170 0x7f9e8414c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:50.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.529+0000 7f9e83fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e840c1170 0x7f9e8414c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43002/0 (socket says 172.21.15.77:43002) 2022-04-03T19:51:50.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.529+0000 7f9e83fff700 1 -- 172.21.15.77:0/2103081310 learned_addr learned my addr 172.21.15.77:0/2103081310 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:51:50.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.529+0000 7f9e88ba1700 1 --2- 172.21.15.77:0/2103081310 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e84155370 0x7f9e841687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:50.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.529+0000 7f9e837fe700 1 --2- 172.21.15.77:0/2103081310 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e840c3b10 0x7f9e841543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:50.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.529+0000 7f9e83fff700 1 -- 172.21.15.77:0/2103081310 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e84155370 msgr2=0x7f9e841687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:50.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.529+0000 7f9e83fff700 1 --2- 172.21.15.77:0/2103081310 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e84155370 0x7f9e841687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:50.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.529+0000 7f9e83fff700 1 -- 172.21.15.77:0/2103081310 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e840c3b10 msgr2=0x7f9e841543d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:50.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.529+0000 7f9e83fff700 1 --2- 172.21.15.77:0/2103081310 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e840c3b10 0x7f9e841543d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:50.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.529+0000 7f9e83fff700 1 -- 172.21.15.77:0/2103081310 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e780092a0 con 0x7f9e840c1170 2022-04-03T19:51:50.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.529+0000 7f9e83fff700 1 --2- 172.21.15.77:0/2103081310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e840c1170 0x7f9e8414c710 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9e7400c540 tx=0x7f9e74007ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:50.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.530+0000 7f9e6bfff700 1 -- 172.21.15.77:0/2103081310 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e74007360 con 0x7f9e840c1170 2022-04-03T19:51:50.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.530+0000 7f9e6bfff700 1 -- 172.21.15.77:0/2103081310 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e740074e0 con 0x7f9e840c1170 2022-04-03T19:51:50.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.530+0000 7f9e893a2700 1 -- 172.21.15.77:0/2103081310 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e841530b0 con 0x7f9e840c1170 2022-04-03T19:51:50.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.530+0000 7f9e6bfff700 1 -- 172.21.15.77:0/2103081310 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e74004900 con 0x7f9e840c1170 2022-04-03T19:51:50.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.530+0000 7f9e893a2700 1 -- 172.21.15.77:0/2103081310 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e84153670 con 0x7f9e840c1170 2022-04-03T19:51:50.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.531+0000 7f9e893a2700 1 -- 172.21.15.77:0/2103081310 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e84153cc0 con 0x7f9e840c1170 2022-04-03T19:51:50.540 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.532+0000 7f9e6bfff700 1 -- 172.21.15.77:0/2103081310 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f9e7401a020 con 0x7f9e840c1170 2022-04-03T19:51:50.540 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.532+0000 7f9e6bfff700 1 --2- 172.21.15.77:0/2103081310 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9e64063ce0 0x7f9e64066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:50.540 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.533+0000 7f9e6bfff700 1 -- 172.21.15.77:0/2103081310 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9e74080680 con 0x7f9e840c1170 2022-04-03T19:51:50.540 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.533+0000 7f9e837fe700 1 --2- 172.21.15.77:0/2103081310 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9e64063ce0 0x7f9e64066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:50.541 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.534+0000 7f9e837fe700 1 --2- 172.21.15.77:0/2103081310 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9e64063ce0 0x7f9e64066190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9e780096e0 tx=0x7f9e78013000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:50.541 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.537+0000 7f9e6bfff700 1 -- 172.21.15.77:0/2103081310 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f9e7404f060 con 0x7f9e840c1170 2022-04-03T19:51:50.769 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.769+0000 7f9e893a2700 1 -- 172.21.15.77:0/2103081310 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e840c2e90 con 0x7f9e840c1170 2022-04-03T19:51:50.770 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.770+0000 7f9e6bfff700 1 -- 172.21.15.77:0/2103081310 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5352 (secure 0 0 0) 0x7f9e74052680 con 0x7f9e840c1170 2022-04-03T19:51:50.771 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:51:50.771 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:51:49.190241+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"e","rank":0,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:51:50.774 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.772+0000 7f9e893a2700 1 -- 172.21.15.77:0/2103081310 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9e64063ce0 msgr2=0x7f9e64066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:50.774 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.773+0000 7f9e893a2700 1 --2- 172.21.15.77:0/2103081310 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9e64063ce0 0x7f9e64066190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9e780096e0 tx=0x7f9e78013000 comp rx=0 tx=0).stop 2022-04-03T19:51:50.774 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.773+0000 7f9e893a2700 1 -- 172.21.15.77:0/2103081310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e840c1170 msgr2=0x7f9e8414c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:50.774 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.773+0000 7f9e893a2700 1 --2- 172.21.15.77:0/2103081310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e840c1170 0x7f9e8414c710 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9e7400c540 tx=0x7f9e74007ca0 comp rx=0 tx=0).stop 2022-04-03T19:51:50.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.773+0000 7f9e893a2700 1 -- 172.21.15.77:0/2103081310 shutdown_connections 2022-04-03T19:51:50.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.773+0000 7f9e893a2700 1 --2- 172.21.15.77:0/2103081310 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9e64063ce0 0x7f9e64066190 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:50.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.773+0000 7f9e893a2700 1 --2- 172.21.15.77:0/2103081310 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e84155370 0x7f9e841687b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:50.775 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.773+0000 7f9e893a2700 1 --2- 172.21.15.77:0/2103081310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e840c1170 0x7f9e8414c710 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:50.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.773+0000 7f9e893a2700 1 --2- 172.21.15.77:0/2103081310 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e840c3b10 0x7f9e841543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:50.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.773+0000 7f9e893a2700 1 -- 172.21.15.77:0/2103081310 >> 172.21.15.77:0/2103081310 conn(0x7f9e84066da0 msgr2=0x7f9e84067170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:50.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.773+0000 7f9e893a2700 1 -- 172.21.15.77:0/2103081310 shutdown_connections 2022-04-03T19:51:50.776 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.773+0000 7f9e893a2700 1 -- 172.21.15.77:0/2103081310 wait complete. 2022-04-03T19:51:50.777 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 27 2022-04-03T19:51:50.797 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2022-04-03T19:51:50.797 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-04-03T19:51:50.798 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:51:50.996 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.996+0000 7fd642080700 1 Processor -- start 2022-04-03T19:51:50.997 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.996+0000 7fd642080700 1 -- start start 2022-04-03T19:51:50.998 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.997+0000 7fd642080700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd63c06bbe0 0x7fd63c1584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:50.998 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.997+0000 7fd642080700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd63c06d260 con 0x7fd63c06c500 2022-04-03T19:51:50.998 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.997+0000 7fd642080700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd63c0c6330 con 0x7fd63c06bfb0 2022-04-03T19:51:50.999 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.997+0000 7fd642080700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd63c159450 con 0x7fd63c06bbe0 2022-04-03T19:51:50.999 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.997+0000 7fd64107e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd63c06bbe0 0x7fd63c1584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:50.999 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.997+0000 7fd64087d700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fd63c06bfb0 0x7fd63c06d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44280/0 (socket says 172.21.15.77:44280) 2022-04-03T19:51:51.000 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.997+0000 7fd64187f700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd63c06c500 0x7fd63c14eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38492/0 (socket says 172.21.15.77:38492) 2022-04-03T19:51:51.000 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.997+0000 7fd64087d700 1 -- 172.21.15.77:0/2877351123 learned_addr learned my addr 172.21.15.77:0/2877351123 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:51:51.004 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.998+0000 7fd64107e700 1 -- 172.21.15.77:0/2877351123 >> v1:172.21.15.186:6789/0 conn(0x7fd63c06bfb0 legacy=0x7fd63c06d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:51.004 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.998+0000 7fd64107e700 1 -- 172.21.15.77:0/2877351123 >> v1:172.21.15.77:6789/0 conn(0x7fd63c06c500 legacy=0x7fd63c14eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:51.005 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.998+0000 7fd62bfff700 1 -- 172.21.15.77:0/2877351123 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575119359 0 0) 0x7fd63c0c6330 con 0x7fd63c06bfb0 2022-04-03T19:51:51.005 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.998+0000 7fd64107e700 1 -- 172.21.15.77:0/2877351123 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd63c158bb0 con 0x7fd63c06bbe0 2022-04-03T19:51:51.005 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.998+0000 7fd64107e700 1 --2- 172.21.15.77:0/2877351123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd63c06bbe0 0x7fd63c1584b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd638009760 tx=0x7fd638009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c5ec3a93185b0d8 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:51.005 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.999+0000 7fd62bfff700 1 -- 172.21.15.77:0/2877351123 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd638004030 con 0x7fd63c06bbe0 2022-04-03T19:51:51.006 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.999+0000 7fd62bfff700 1 -- 172.21.15.77:0/2877351123 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd638004190 con 0x7fd63c06bbe0 2022-04-03T19:51:51.006 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.999+0000 7fd62bfff700 1 -- 172.21.15.77:0/2877351123 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd638011630 con 0x7fd63c06bbe0 2022-04-03T19:51:51.006 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.999+0000 7fd642080700 1 -- 172.21.15.77:0/2877351123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd63c06bbe0 msgr2=0x7fd63c1584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:51.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.999+0000 7fd642080700 1 --2- 172.21.15.77:0/2877351123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd63c06bbe0 0x7fd63c1584b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd638009760 tx=0x7fd638009060 comp rx=0 tx=0).stop 2022-04-03T19:51:51.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.999+0000 7fd642080700 1 -- 172.21.15.77:0/2877351123 shutdown_connections 2022-04-03T19:51:51.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.999+0000 7fd642080700 1 --2- 172.21.15.77:0/2877351123 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd63c06bbe0 0x7fd63c1584b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:51.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.999+0000 7fd642080700 1 -- 172.21.15.77:0/2877351123 >> 172.21.15.77:0/2877351123 conn(0x7fd63c0cb6b0 msgr2=0x7fd63c0cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:51.007 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.999+0000 7fd642080700 1 -- 172.21.15.77:0/2877351123 shutdown_connections 2022-04-03T19:51:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:50.999+0000 7fd642080700 1 -- 172.21.15.77:0/2877351123 wait complete. 2022-04-03T19:51:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.001+0000 7fd642080700 1 Processor -- start 2022-04-03T19:51:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.001+0000 7fd642080700 1 -- start start 2022-04-03T19:51:51.008 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.002+0000 7fd642080700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd63c06bbe0 0x7fd63c154fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:51.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.002+0000 7fd642080700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63c06bfb0 0x7fd63c153fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:51.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.002+0000 7fd642080700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd63c06c500 0x7fd63c1544f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:51.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.002+0000 7fd642080700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c151ac0 con 0x7fd63c06c500 2022-04-03T19:51:51.009 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.002+0000 7fd642080700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c168ae0 con 0x7fd63c06bbe0 2022-04-03T19:51:51.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.002+0000 7fd642080700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c152520 con 0x7fd63c06bfb0 2022-04-03T19:51:51.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.003+0000 7fd64087d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63c06bfb0 0x7fd63c153fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:51.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.003+0000 7fd64107e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd63c06bbe0 0x7fd63c154fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:51.010 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.003+0000 7fd64187f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd63c06c500 0x7fd63c1544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:51.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.003+0000 7fd64087d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63c06bfb0 0x7fd63c153fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60284/0 (socket says 172.21.15.77:60284) 2022-04-03T19:51:51.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.003+0000 7fd64087d700 1 -- 172.21.15.77:0/1671525877 learned_addr learned my addr 172.21.15.77:0/1671525877 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:51:51.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.003+0000 7fd64107e700 1 --2- 172.21.15.77:0/1671525877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd63c06bbe0 0x7fd63c154fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38192/0 (socket says 172.21.15.77:38192) 2022-04-03T19:51:51.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.003+0000 7fd64087d700 1 -- 172.21.15.77:0/1671525877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd63c06bbe0 msgr2=0x7fd63c154fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:51.011 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.003+0000 7fd64087d700 1 --2- 172.21.15.77:0/1671525877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd63c06bbe0 0x7fd63c154fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:51.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.003+0000 7fd64087d700 1 -- 172.21.15.77:0/1671525877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd63c06c500 msgr2=0x7fd63c1544f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:51.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.003+0000 7fd64087d700 1 --2- 172.21.15.77:0/1671525877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd63c06c500 0x7fd63c1544f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:51.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.003+0000 7fd64087d700 1 -- 172.21.15.77:0/1671525877 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6380092a0 con 0x7fd63c06bfb0 2022-04-03T19:51:51.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.003+0000 7fd64087d700 1 --2- 172.21.15.77:0/1671525877 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63c06bfb0 0x7fd63c153fb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd62c00c510 tx=0x7fd62c007c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:51.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.004+0000 7fd6297fa700 1 -- 172.21.15.77:0/1671525877 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c00ce80 con 0x7fd63c06bfb0 2022-04-03T19:51:51.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.004+0000 7fd6297fa700 1 -- 172.21.15.77:0/1671525877 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62c007360 con 0x7fd63c06bfb0 2022-04-03T19:51:51.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.004+0000 7fd642080700 1 -- 172.21.15.77:0/1671525877 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd63c168800 con 0x7fd63c06bfb0 2022-04-03T19:51:51.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.004+0000 7fd6297fa700 1 -- 172.21.15.77:0/1671525877 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c0046b0 con 0x7fd63c06bfb0 2022-04-03T19:51:51.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.004+0000 7fd642080700 1 -- 172.21.15.77:0/1671525877 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63c14fc30 con 0x7fd63c06bfb0 2022-04-03T19:51:51.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.005+0000 7fd642080700 1 -- 172.21.15.77:0/1671525877 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63c0b1720 con 0x7fd63c06bfb0 2022-04-03T19:51:51.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.006+0000 7fd6297fa700 1 -- 172.21.15.77:0/1671525877 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fd62c010030 con 0x7fd63c06bfb0 2022-04-03T19:51:51.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.007+0000 7fd6297fa700 1 --2- 172.21.15.77:0/1671525877 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd61c063d60 0x7fd61c066210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:51.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.007+0000 7fd6297fa700 1 -- 172.21.15.77:0/1671525877 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd62c080520 con 0x7fd63c06bfb0 2022-04-03T19:51:51.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.007+0000 7fd64107e700 1 --2- 172.21.15.77:0/1671525877 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd61c063d60 0x7fd61c066210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:51.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.008+0000 7fd64107e700 1 --2- 172.21.15.77:0/1671525877 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd61c063d60 0x7fd61c066210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd638009760 tx=0x7fd638016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:51.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.011+0000 7fd6297fa700 1 -- 172.21.15.77:0/1671525877 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fd62c04ef00 con 0x7fd63c06bfb0 2022-04-03T19:51:51.245 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.245+0000 7fd642080700 1 -- 172.21.15.77:0/1671525877 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd63c04b050 con 0x7fd63c06bfb0 2022-04-03T19:51:51.246 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.246+0000 7fd6297fa700 1 -- 172.21.15.77:0/1671525877 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5352 (secure 0 0 0) 0x7fd62c052520 con 0x7fd63c06bfb0 2022-04-03T19:51:51.247 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:51:51.247 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:51:49.190241+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"e","rank":0,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:51:51.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.248+0000 7fd642080700 1 -- 172.21.15.77:0/1671525877 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd61c063d60 msgr2=0x7fd61c066210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:51.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.248+0000 7fd642080700 1 --2- 172.21.15.77:0/1671525877 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd61c063d60 0x7fd61c066210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd638009760 tx=0x7fd638016000 comp rx=0 tx=0).stop 2022-04-03T19:51:51.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.248+0000 7fd642080700 1 -- 172.21.15.77:0/1671525877 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63c06bfb0 msgr2=0x7fd63c153fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:51.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.248+0000 7fd642080700 1 --2- 172.21.15.77:0/1671525877 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63c06bfb0 0x7fd63c153fb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd62c00c510 tx=0x7fd62c007c70 comp rx=0 tx=0).stop 2022-04-03T19:51:51.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.248+0000 7fd642080700 1 -- 172.21.15.77:0/1671525877 shutdown_connections 2022-04-03T19:51:51.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.248+0000 7fd642080700 1 --2- 172.21.15.77:0/1671525877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd63c06c500 0x7fd63c1544f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:51.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.248+0000 7fd642080700 1 --2- 172.21.15.77:0/1671525877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd63c06bbe0 0x7fd63c154fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:51.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.248+0000 7fd642080700 1 --2- 172.21.15.77:0/1671525877 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd61c063d60 0x7fd61c066210 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:51.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.248+0000 7fd642080700 1 --2- 172.21.15.77:0/1671525877 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63c06bfb0 0x7fd63c153fb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:51.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.248+0000 7fd642080700 1 -- 172.21.15.77:0/1671525877 >> 172.21.15.77:0/1671525877 conn(0x7fd63c0cb6b0 msgr2=0x7fd63c0cba80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:51.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.249+0000 7fd642080700 1 -- 172.21.15.77:0/1671525877 shutdown_connections 2022-04-03T19:51:51.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:51.249+0000 7fd642080700 1 -- 172.21.15.77:0/1671525877 wait complete. 2022-04-03T19:51:51.252 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 27 2022-04-03T19:51:51.275 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 27, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-04-03T19:44:21.275716+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 27, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5239": { "addr": "172.21.15.77:6837/3264384876", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6836", "nonce": 3264384876, "type": "v2" }, { "addr": "172.21.15.77:6837", "nonce": 3264384876, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 5239, "incarnation": 27, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 50, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-04-03T19:51:49.190241+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5239 } } } ], "standbys": [ { "addr": "172.21.15.77:6834/2116223509", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6832", "nonce": 2116223509, "type": "v2" }, { "addr": "172.21.15.77:6834", "nonce": 2116223509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4245, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6836/2374276747", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2374276747, "type": "v2" }, { "addr": "172.21.15.186:6836", "nonce": 2374276747, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4259, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/610990853", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6838", "nonce": 610990853, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 610990853, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4265, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.77:6835/170053412", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6833", "nonce": 170053412, "type": "v2" }, { "addr": "172.21.15.77:6835", "nonce": 170053412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4271, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-04-03T19:51:53.276 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:51:53.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.487+0000 7fe927fff700 1 Processor -- start 2022-04-03T19:51:53.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.488+0000 7fe927fff700 1 -- start start 2022-04-03T19:51:53.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.489+0000 7fe927fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe92806d3f0 0x7fe92806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:53.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.489+0000 7fe927fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9280c2a90 0x7fe92806c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:53.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.489+0000 7fe927fff700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9281562f0 con 0x7fe9280c5930 2022-04-03T19:51:53.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.489+0000 7fe927fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9280c2f80 con 0x7fe92806d3f0 2022-04-03T19:51:53.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.489+0000 7fe927fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9280c3260 con 0x7fe9280c2a90 2022-04-03T19:51:53.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.489+0000 7fe91ffff700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fe9280c5930 0x7fe928155be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44284/0 (socket says 172.21.15.77:44284) 2022-04-03T19:51:53.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.489+0000 7fe9277fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9280c2a90 0x7fe92806c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:53.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.489+0000 7fe91ffff700 1 -- 172.21.15.77:0/3221691609 learned_addr learned my addr 172.21.15.77:0/3221691609 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:51:53.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.489+0000 7fe926ffd700 1 --2- 172.21.15.77:0/3221691609 >> v2:172.21.15.77:3300/0 conn(0x7fe92806d3f0 0x7fe92806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:53.492 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.489+0000 7fe9267fc700 1 -- 172.21.15.77:0/3221691609 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406233022 0 0) 0x7fe9281562f0 con 0x7fe9280c5930 2022-04-03T19:51:53.492 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.489+0000 7fe9267fc700 1 -- 172.21.15.77:0/3221691609 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9080033f0 con 0x7fe9280c5930 2022-04-03T19:51:53.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.490+0000 7fe9277fe700 1 -- 172.21.15.77:0/3221691609 >> v2:172.21.15.77:3300/0 conn(0x7fe92806d3f0 msgr2=0x7fe92806d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:51:53.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.490+0000 7fe9277fe700 1 --2- 172.21.15.77:0/3221691609 >> v2:172.21.15.77:3300/0 conn(0x7fe92806d3f0 0x7fe92806d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:53.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.490+0000 7fe9277fe700 1 -- 172.21.15.77:0/3221691609 >> v1:172.21.15.186:6789/0 conn(0x7fe9280c5930 legacy=0x7fe928155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:53.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.490+0000 7fe9277fe700 1 -- 172.21.15.77:0/3221691609 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9280c3540 con 0x7fe9280c2a90 2022-04-03T19:51:53.497 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.490+0000 7fe926ffd700 1 --2- 172.21.15.77:0/3221691609 >> v2:172.21.15.77:3300/0 conn(0x7fe92806d3f0 0x7fe92806d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:51:53.497 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.490+0000 7fe9277fe700 1 --2- 172.21.15.77:0/3221691609 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9280c2a90 0x7fe92806c040 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe920009e90 tx=0x7fe920009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fcdb33fb430553f server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:53.497 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.490+0000 7fe9267fc700 1 -- 172.21.15.77:0/3221691609 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92000e070 con 0x7fe9280c2a90 2022-04-03T19:51:53.497 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.490+0000 7fe9267fc700 1 -- 172.21.15.77:0/3221691609 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92000bbd0 con 0x7fe9280c2a90 2022-04-03T19:51:53.498 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.490+0000 7fe9267fc700 1 -- 172.21.15.77:0/3221691609 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920012430 con 0x7fe9280c2a90 2022-04-03T19:51:53.498 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.491+0000 7fe927fff700 1 -- 172.21.15.77:0/3221691609 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9280c2a90 msgr2=0x7fe92806c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:53.498 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.491+0000 7fe927fff700 1 --2- 172.21.15.77:0/3221691609 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9280c2a90 0x7fe92806c040 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe920009e90 tx=0x7fe920009780 comp rx=0 tx=0).stop 2022-04-03T19:51:53.498 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.491+0000 7fe927fff700 1 -- 172.21.15.77:0/3221691609 shutdown_connections 2022-04-03T19:51:53.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.491+0000 7fe927fff700 1 --2- 172.21.15.77:0/3221691609 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9280c2a90 0x7fe92806c040 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:53.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.491+0000 7fe927fff700 1 --2- 172.21.15.77:0/3221691609 >> v2:172.21.15.77:3300/0 conn(0x7fe92806d3f0 0x7fe92806d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:53.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.491+0000 7fe927fff700 1 -- 172.21.15.77:0/3221691609 >> 172.21.15.77:0/3221691609 conn(0x7fe9280cb720 msgr2=0x7fe9280cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:53.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.491+0000 7fe927fff700 1 -- 172.21.15.77:0/3221691609 shutdown_connections 2022-04-03T19:51:53.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.491+0000 7fe927fff700 1 -- 172.21.15.77:0/3221691609 wait complete. 2022-04-03T19:51:53.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.493+0000 7fe927fff700 1 Processor -- start 2022-04-03T19:51:53.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.493+0000 7fe927fff700 1 -- start start 2022-04-03T19:51:53.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.493+0000 7fe927fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92806d3f0 0x7fe9281692a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:53.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.494+0000 7fe927fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9280c2a90 0x7fe9280c0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:53.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.494+0000 7fe927fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9280c5930 0x7fe9280c0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:53.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.494+0000 7fe927fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe928152e10 con 0x7fe92806d3f0 2022-04-03T19:51:53.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.494+0000 7fe927fff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9280c0cc0 con 0x7fe9280c2a90 2022-04-03T19:51:53.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.494+0000 7fe927fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe928165340 con 0x7fe9280c5930 2022-04-03T19:51:53.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.494+0000 7fe926ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92806d3f0 0x7fe9281692a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:53.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.494+0000 7fe926ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92806d3f0 0x7fe9281692a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43012/0 (socket says 172.21.15.77:43012) 2022-04-03T19:51:53.502 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.494+0000 7fe926ffd700 1 -- 172.21.15.77:0/4122785205 learned_addr learned my addr 172.21.15.77:0/4122785205 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:51:53.502 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.494+0000 7fe91ffff700 1 --2- 172.21.15.77:0/4122785205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9280c2a90 0x7fe9280c0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:53.502 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.494+0000 7fe9277fe700 1 --2- 172.21.15.77:0/4122785205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9280c5930 0x7fe9280c0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:53.502 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.494+0000 7fe926ffd700 1 -- 172.21.15.77:0/4122785205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9280c5930 msgr2=0x7fe9280c0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:53.502 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.494+0000 7fe926ffd700 1 --2- 172.21.15.77:0/4122785205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9280c5930 0x7fe9280c0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:53.503 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.494+0000 7fe926ffd700 1 -- 172.21.15.77:0/4122785205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9280c2a90 msgr2=0x7fe9280c0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:53.503 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.494+0000 7fe926ffd700 1 --2- 172.21.15.77:0/4122785205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9280c2a90 0x7fe9280c0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:53.503 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.494+0000 7fe926ffd700 1 -- 172.21.15.77:0/4122785205 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9180092a0 con 0x7fe92806d3f0 2022-04-03T19:51:53.503 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.495+0000 7fe926ffd700 1 --2- 172.21.15.77:0/4122785205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92806d3f0 0x7fe9281692a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe91800ea20 tx=0x7fe918009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:53.503 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.495+0000 7fe91effd700 1 -- 172.21.15.77:0/4122785205 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91800ee60 con 0x7fe92806d3f0 2022-04-03T19:51:53.504 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.495+0000 7fe91effd700 1 -- 172.21.15.77:0/4122785205 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91800c8e0 con 0x7fe92806d3f0 2022-04-03T19:51:53.504 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.495+0000 7fe91effd700 1 -- 172.21.15.77:0/4122785205 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91800cc60 con 0x7fe92806d3f0 2022-04-03T19:51:53.504 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.495+0000 7fe927fff700 1 -- 172.21.15.77:0/4122785205 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe920009a20 con 0x7fe92806d3f0 2022-04-03T19:51:53.504 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.495+0000 7fe927fff700 1 -- 172.21.15.77:0/4122785205 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe928165ba0 con 0x7fe92806d3f0 2022-04-03T19:51:53.505 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.496+0000 7fe927fff700 1 -- 172.21.15.77:0/4122785205 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe92804b050 con 0x7fe92806d3f0 2022-04-03T19:51:53.506 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.497+0000 7fe91effd700 1 -- 172.21.15.77:0/4122785205 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fe918019030 con 0x7fe92806d3f0 2022-04-03T19:51:53.506 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.498+0000 7fe91effd700 1 --2- 172.21.15.77:0/4122785205 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe90c063ce0 0x7fe90c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:53.506 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.498+0000 7fe91effd700 1 -- 172.21.15.77:0/4122785205 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe918080b60 con 0x7fe92806d3f0 2022-04-03T19:51:53.506 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.498+0000 7fe91ffff700 1 --2- 172.21.15.77:0/4122785205 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe90c063ce0 0x7fe90c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:53.507 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.499+0000 7fe91ffff700 1 --2- 172.21.15.77:0/4122785205 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe90c063ce0 0x7fe90c066190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe928152610 tx=0x7fe910009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:53.507 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.502+0000 7fe91effd700 1 -- 172.21.15.77:0/4122785205 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fe91804f4c0 con 0x7fe92806d3f0 2022-04-03T19:51:53.733 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.733+0000 7fe927fff700 1 -- 172.21.15.77:0/4122785205 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe92806d010 con 0x7fe92806d3f0 2022-04-03T19:51:53.734 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.733+0000 7fe91effd700 1 -- 172.21.15.77:0/4122785205 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5352 (secure 0 0 0) 0x7fe918021090 con 0x7fe92806d3f0 2022-04-03T19:51:53.734 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:51:53.735 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:51:49.190241+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"e","rank":0,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:51:53.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.736+0000 7fe927fff700 1 -- 172.21.15.77:0/4122785205 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe90c063ce0 msgr2=0x7fe90c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:53.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.736+0000 7fe927fff700 1 --2- 172.21.15.77:0/4122785205 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe90c063ce0 0x7fe90c066190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe928152610 tx=0x7fe910009040 comp rx=0 tx=0).stop 2022-04-03T19:51:53.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.736+0000 7fe927fff700 1 -- 172.21.15.77:0/4122785205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92806d3f0 msgr2=0x7fe9281692a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:53.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.736+0000 7fe927fff700 1 --2- 172.21.15.77:0/4122785205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92806d3f0 0x7fe9281692a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe91800ea20 tx=0x7fe918009060 comp rx=0 tx=0).stop 2022-04-03T19:51:53.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.736+0000 7fe927fff700 1 -- 172.21.15.77:0/4122785205 shutdown_connections 2022-04-03T19:51:53.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.736+0000 7fe927fff700 1 --2- 172.21.15.77:0/4122785205 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe90c063ce0 0x7fe90c066190 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:53.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.736+0000 7fe927fff700 1 --2- 172.21.15.77:0/4122785205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9280c5930 0x7fe9280c0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:53.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.736+0000 7fe927fff700 1 --2- 172.21.15.77:0/4122785205 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92806d3f0 0x7fe9281692a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:53.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.736+0000 7fe927fff700 1 --2- 172.21.15.77:0/4122785205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9280c2a90 0x7fe9280c0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:53.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.736+0000 7fe927fff700 1 -- 172.21.15.77:0/4122785205 >> 172.21.15.77:0/4122785205 conn(0x7fe9280cb720 msgr2=0x7fe928155b30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:53.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.736+0000 7fe927fff700 1 -- 172.21.15.77:0/4122785205 shutdown_connections 2022-04-03T19:51:53.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:53.736+0000 7fe927fff700 1 -- 172.21.15.77:0/4122785205 wait complete. 2022-04-03T19:51:53.740 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 27 2022-04-03T19:51:53.760 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:51:55.762 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:51:55.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.974+0000 7fbc6a4b7700 1 Processor -- start 2022-04-03T19:51:55.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.974+0000 7fbc6a4b7700 1 -- start start 2022-04-03T19:51:55.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.975+0000 7fbc6a4b7700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbc6406bbe0 0x7fbc641584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:55.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.975+0000 7fbc6a4b7700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc6406d260 con 0x7fbc6406bfb0 2022-04-03T19:51:55.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.975+0000 7fbc6a4b7700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc640c6330 con 0x7fbc6406c500 2022-04-03T19:51:55.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.975+0000 7fbc6a4b7700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc64159450 con 0x7fbc6406bbe0 2022-04-03T19:51:55.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.976+0000 7fbc68cb4700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fbc6406bfb0 0x7fbc6406d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38494/0 (socket says 172.21.15.77:38494) 2022-04-03T19:51:55.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.976+0000 7fbc68cb4700 1 -- 172.21.15.77:0/1642994371 learned_addr learned my addr 172.21.15.77:0/1642994371 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:51:55.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.976+0000 7fbc694b5700 1 --2- 172.21.15.77:0/1642994371 >> v2:172.21.15.186:3300/0 conn(0x7fbc6406bbe0 0x7fbc641584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:55.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.976+0000 7fbc53fff700 1 -- 172.21.15.77:0/1642994371 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528985822 0 0) 0x7fbc6406d260 con 0x7fbc6406bfb0 2022-04-03T19:51:55.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.976+0000 7fbc53fff700 1 -- 172.21.15.77:0/1642994371 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc4c0033f0 con 0x7fbc6406bfb0 2022-04-03T19:51:55.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.976+0000 7fbc53fff700 1 -- 172.21.15.77:0/1642994371 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256244026 0 0) 0x7fbc640c6330 con 0x7fbc6406c500 2022-04-03T19:51:55.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.976+0000 7fbc53fff700 1 -- 172.21.15.77:0/1642994371 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc6406d260 con 0x7fbc6406c500 2022-04-03T19:51:55.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.977+0000 7fbc694b5700 1 -- 172.21.15.77:0/1642994371 >> v1:172.21.15.186:6789/0 conn(0x7fbc6406c500 legacy=0x7fbc6414eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:55.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.977+0000 7fbc694b5700 1 -- 172.21.15.77:0/1642994371 >> v1:172.21.15.77:6789/0 conn(0x7fbc6406bfb0 legacy=0x7fbc6406d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:55.980 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.977+0000 7fbc694b5700 1 -- 172.21.15.77:0/1642994371 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc64158bb0 con 0x7fbc6406bbe0 2022-04-03T19:51:55.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.977+0000 7fbc694b5700 1 --2- 172.21.15.77:0/1642994371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc6406bbe0 0x7fbc641584b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbc60009760 tx=0x7fbc60009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7da6e9a0bbceb4ac server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:55.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.977+0000 7fbc53fff700 1 -- 172.21.15.77:0/1642994371 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc600105b0 con 0x7fbc6406bbe0 2022-04-03T19:51:55.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.977+0000 7fbc53fff700 1 -- 172.21.15.77:0/1642994371 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc60010710 con 0x7fbc6406bbe0 2022-04-03T19:51:55.981 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.977+0000 7fbc53fff700 1 -- 172.21.15.77:0/1642994371 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc60010a90 con 0x7fbc6406bbe0 2022-04-03T19:51:55.984 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.978+0000 7fbc6a4b7700 1 -- 172.21.15.77:0/1642994371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc6406bbe0 msgr2=0x7fbc641584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:55.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.978+0000 7fbc6a4b7700 1 --2- 172.21.15.77:0/1642994371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc6406bbe0 0x7fbc641584b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbc60009760 tx=0x7fbc60009060 comp rx=0 tx=0).stop 2022-04-03T19:51:55.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.978+0000 7fbc6a4b7700 1 -- 172.21.15.77:0/1642994371 shutdown_connections 2022-04-03T19:51:55.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.978+0000 7fbc6a4b7700 1 --2- 172.21.15.77:0/1642994371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc6406bbe0 0x7fbc641584b0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:55.986 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.978+0000 7fbc6a4b7700 1 -- 172.21.15.77:0/1642994371 >> 172.21.15.77:0/1642994371 conn(0x7fbc640cb6b0 msgr2=0x7fbc640cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:55.986 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.978+0000 7fbc6a4b7700 1 -- 172.21.15.77:0/1642994371 shutdown_connections 2022-04-03T19:51:55.986 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.978+0000 7fbc6a4b7700 1 -- 172.21.15.77:0/1642994371 wait complete. 2022-04-03T19:51:55.986 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.979+0000 7fbc6a4b7700 1 Processor -- start 2022-04-03T19:51:55.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.980+0000 7fbc6a4b7700 1 -- start start 2022-04-03T19:51:55.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.980+0000 7fbc6a4b7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc6406bbe0 0x7fbc64157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:55.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.980+0000 7fbc6a4b7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc6406bfb0 0x7fbc640c2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:55.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.980+0000 7fbc6a4b7700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbc6406c500 0x7fbc640c3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:55.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.980+0000 7fbc6a4b7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc641553f0 con 0x7fbc6406bfb0 2022-04-03T19:51:55.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.980+0000 7fbc6a4b7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc64155690 con 0x7fbc6406bbe0 2022-04-03T19:51:55.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.980+0000 7fbc6a4b7700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc64165210 con 0x7fbc6406c500 2022-04-03T19:51:55.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.981+0000 7fbc694b5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc6406bbe0 0x7fbc64157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:55.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.981+0000 7fbc68cb4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc6406bfb0 0x7fbc640c2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:55.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.981+0000 7fbc694b5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc6406bbe0 0x7fbc64157e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38198/0 (socket says 172.21.15.77:38198) 2022-04-03T19:51:55.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.981+0000 7fbc694b5700 1 -- 172.21.15.77:0/3190315555 learned_addr learned my addr 172.21.15.77:0/3190315555 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:51:55.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.981+0000 7fbc69cb6700 1 --2- 172.21.15.77:0/3190315555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbc6406c500 0x7fbc640c3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:55.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.981+0000 7fbc694b5700 1 -- 172.21.15.77:0/3190315555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbc6406c500 msgr2=0x7fbc640c3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:55.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.981+0000 7fbc694b5700 1 --2- 172.21.15.77:0/3190315555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbc6406c500 0x7fbc640c3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:55.990 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.981+0000 7fbc694b5700 1 -- 172.21.15.77:0/3190315555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc6406bfb0 msgr2=0x7fbc640c2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:55.990 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.981+0000 7fbc694b5700 1 --2- 172.21.15.77:0/3190315555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc6406bfb0 0x7fbc640c2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:55.990 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.981+0000 7fbc694b5700 1 -- 172.21.15.77:0/3190315555 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc600092a0 con 0x7fbc6406bbe0 2022-04-03T19:51:55.990 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.982+0000 7fbc694b5700 1 --2- 172.21.15.77:0/3190315555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc6406bbe0 0x7fbc64157e20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbc60009d50 tx=0x7fbc60016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:55.991 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.982+0000 7fbc517fa700 1 -- 172.21.15.77:0/3190315555 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc60010880 con 0x7fbc6406bbe0 2022-04-03T19:51:55.991 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.982+0000 7fbc517fa700 1 -- 172.21.15.77:0/3190315555 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc60010a00 con 0x7fbc6406bbe0 2022-04-03T19:51:55.991 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.982+0000 7fbc517fa700 1 -- 172.21.15.77:0/3190315555 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc600042a0 con 0x7fbc6406bbe0 2022-04-03T19:51:55.991 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.982+0000 7fbc6a4b7700 1 -- 172.21.15.77:0/3190315555 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc641654f0 con 0x7fbc6406bbe0 2022-04-03T19:51:55.992 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.982+0000 7fbc6a4b7700 1 -- 172.21.15.77:0/3190315555 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc64165ab0 con 0x7fbc6406bbe0 2022-04-03T19:51:55.992 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.983+0000 7fbc6a4b7700 1 -- 172.21.15.77:0/3190315555 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc640c4340 con 0x7fbc6406bbe0 2022-04-03T19:51:55.992 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.984+0000 7fbc517fa700 1 -- 172.21.15.77:0/3190315555 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fbc6000f070 con 0x7fbc6406bbe0 2022-04-03T19:51:55.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.985+0000 7fbc517fa700 1 --2- 172.21.15.77:0/3190315555 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbc44063ce0 0x7fbc44066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:55.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.985+0000 7fbc517fa700 1 -- 172.21.15.77:0/3190315555 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fbc600825a0 con 0x7fbc6406bbe0 2022-04-03T19:51:55.994 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.985+0000 7fbc68cb4700 1 --2- 172.21.15.77:0/3190315555 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbc44063ce0 0x7fbc44066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:55.994 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.986+0000 7fbc68cb4700 1 --2- 172.21.15.77:0/3190315555 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbc44063ce0 0x7fbc44066190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbc6414cc40 tx=0x7fbc54009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:55.994 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:55.989+0000 7fbc517fa700 1 -- 172.21.15.77:0/3190315555 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fbc60050ff0 con 0x7fbc6406bbe0 2022-04-03T19:51:56.244 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:56.243+0000 7fbc6a4b7700 1 -- 172.21.15.77:0/3190315555 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc64165c40 con 0x7fbc6406bbe0 2022-04-03T19:51:56.245 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:56.244+0000 7fbc517fa700 1 -- 172.21.15.77:0/3190315555 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5352 (secure 0 0 0) 0x7fbc6002a070 con 0x7fbc6406bbe0 2022-04-03T19:51:56.245 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:51:56.245 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:51:49.190241+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"e","rank":0,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:51:56.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:56.248+0000 7fbc6a4b7700 1 -- 172.21.15.77:0/3190315555 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbc44063ce0 msgr2=0x7fbc44066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:56.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:56.248+0000 7fbc6a4b7700 1 --2- 172.21.15.77:0/3190315555 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbc44063ce0 0x7fbc44066190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbc6414cc40 tx=0x7fbc54009040 comp rx=0 tx=0).stop 2022-04-03T19:51:56.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:56.248+0000 7fbc6a4b7700 1 -- 172.21.15.77:0/3190315555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc6406bbe0 msgr2=0x7fbc64157e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:56.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:56.248+0000 7fbc6a4b7700 1 --2- 172.21.15.77:0/3190315555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc6406bbe0 0x7fbc64157e20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbc60009d50 tx=0x7fbc60016000 comp rx=0 tx=0).stop 2022-04-03T19:51:56.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:56.248+0000 7fbc6a4b7700 1 -- 172.21.15.77:0/3190315555 shutdown_connections 2022-04-03T19:51:56.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:56.248+0000 7fbc6a4b7700 1 --2- 172.21.15.77:0/3190315555 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbc44063ce0 0x7fbc44066190 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:56.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:56.248+0000 7fbc6a4b7700 1 --2- 172.21.15.77:0/3190315555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbc6406c500 0x7fbc640c3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:56.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:56.248+0000 7fbc6a4b7700 1 --2- 172.21.15.77:0/3190315555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc6406bbe0 0x7fbc64157e20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:56.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:56.248+0000 7fbc6a4b7700 1 --2- 172.21.15.77:0/3190315555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbc6406bfb0 0x7fbc640c2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:56.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:56.248+0000 7fbc6a4b7700 1 -- 172.21.15.77:0/3190315555 >> 172.21.15.77:0/3190315555 conn(0x7fbc640cb6b0 msgr2=0x7fbc640c50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:56.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:56.248+0000 7fbc6a4b7700 1 -- 172.21.15.77:0/3190315555 shutdown_connections 2022-04-03T19:51:56.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:56.248+0000 7fbc6a4b7700 1 -- 172.21.15.77:0/3190315555 wait complete. 2022-04-03T19:51:56.252 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 27 2022-04-03T19:51:56.274 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:51:58.275 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:51:58.475 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.473+0000 7fdb258d2700 1 Processor -- start 2022-04-03T19:51:58.475 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.473+0000 7fdb258d2700 1 -- start start 2022-04-03T19:51:58.477 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.474+0000 7fdb258d2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb2006bfb0 0x7fdb201584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:58.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.475+0000 7fdb258d2700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdb2006c500 0x7fdb2014eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:58.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.475+0000 7fdb258d2700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb2006d260 con 0x7fdb2006bbe0 2022-04-03T19:51:58.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.475+0000 7fdb258d2700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb20159450 con 0x7fdb2006c500 2022-04-03T19:51:58.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.475+0000 7fdb258d2700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb200c6330 con 0x7fdb2006bfb0 2022-04-03T19:51:58.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.475+0000 7fdb1ffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb2006bfb0 0x7fdb201584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:58.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.475+0000 7fdb250d1700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdb2006c500 0x7fdb2014eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:58.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.475+0000 7fdb248d0700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fdb2006bbe0 0x7fdb2006d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44288/0 (socket says 172.21.15.77:44288) 2022-04-03T19:51:58.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.475+0000 7fdb248d0700 1 -- 172.21.15.77:0/1577466793 learned_addr learned my addr 172.21.15.77:0/1577466793 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:51:58.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.476+0000 7fdb1ffff700 1 -- 172.21.15.77:0/1577466793 >> v2:172.21.15.186:3300/0 conn(0x7fdb2006c500 msgr2=0x7fdb2014eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:51:58.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.476+0000 7fdb1ffff700 1 --2- 172.21.15.77:0/1577466793 >> v2:172.21.15.186:3300/0 conn(0x7fdb2006c500 0x7fdb2014eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:58.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.476+0000 7fdb1f7fe700 1 -- 172.21.15.77:0/1577466793 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255780994 0 0) 0x7fdb2006d260 con 0x7fdb2006bbe0 2022-04-03T19:51:58.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.476+0000 7fdb1ffff700 1 -- 172.21.15.77:0/1577466793 >> v1:172.21.15.186:6789/0 conn(0x7fdb2006bbe0 legacy=0x7fdb2006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:58.481 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.476+0000 7fdb1ffff700 1 -- 172.21.15.77:0/1577466793 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb20158bb0 con 0x7fdb2006bfb0 2022-04-03T19:51:58.481 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.476+0000 7fdb1ffff700 1 --2- 172.21.15.77:0/1577466793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb2006bfb0 0x7fdb201584b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c009760 tx=0x7fdb0c009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec8ea0cd5d8768b0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:58.481 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.476+0000 7fdb1f7fe700 1 -- 172.21.15.77:0/1577466793 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c011630 con 0x7fdb2006bfb0 2022-04-03T19:51:58.481 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.476+0000 7fdb1f7fe700 1 -- 172.21.15.77:0/1577466793 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb0c011790 con 0x7fdb2006bfb0 2022-04-03T19:51:58.482 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.476+0000 7fdb1f7fe700 1 -- 172.21.15.77:0/1577466793 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c011b50 con 0x7fdb2006bfb0 2022-04-03T19:51:58.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.477+0000 7fdb258d2700 1 -- 172.21.15.77:0/1577466793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb2006bfb0 msgr2=0x7fdb201584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:58.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.477+0000 7fdb258d2700 1 --2- 172.21.15.77:0/1577466793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb2006bfb0 0x7fdb201584b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c009760 tx=0x7fdb0c009060 comp rx=0 tx=0).stop 2022-04-03T19:51:58.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.477+0000 7fdb258d2700 1 -- 172.21.15.77:0/1577466793 shutdown_connections 2022-04-03T19:51:58.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.477+0000 7fdb258d2700 1 --2- 172.21.15.77:0/1577466793 >> v2:172.21.15.186:3300/0 conn(0x7fdb2006c500 0x7fdb2014eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:58.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.477+0000 7fdb258d2700 1 --2- 172.21.15.77:0/1577466793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb2006bfb0 0x7fdb201584b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:58.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.477+0000 7fdb258d2700 1 -- 172.21.15.77:0/1577466793 >> 172.21.15.77:0/1577466793 conn(0x7fdb200cb6b0 msgr2=0x7fdb200cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:58.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.477+0000 7fdb258d2700 1 -- 172.21.15.77:0/1577466793 shutdown_connections 2022-04-03T19:51:58.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.477+0000 7fdb258d2700 1 -- 172.21.15.77:0/1577466793 wait complete. 2022-04-03T19:51:58.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.478+0000 7fdb258d2700 1 Processor -- start 2022-04-03T19:51:58.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.479+0000 7fdb258d2700 1 -- start start 2022-04-03T19:51:58.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.479+0000 7fdb258d2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb2006bbe0 0x7fdb20167bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:58.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.480+0000 7fdb258d2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb2006bfb0 0x7fdb20151b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:58.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.480+0000 7fdb258d2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb2006c500 0x7fdb20168460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:58.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.480+0000 7fdb258d2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb20166830 con 0x7fdb2006bfb0 2022-04-03T19:51:58.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.480+0000 7fdb258d2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb20166b10 con 0x7fdb2006c500 2022-04-03T19:51:58.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.480+0000 7fdb258d2700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb20166df0 con 0x7fdb2006bbe0 2022-04-03T19:51:58.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.480+0000 7fdb1ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb2006bfb0 0x7fdb20151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:58.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.480+0000 7fdb248d0700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb2006bbe0 0x7fdb20167bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:58.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.480+0000 7fdb250d1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb2006c500 0x7fdb20168460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:58.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.480+0000 7fdb1ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb2006bfb0 0x7fdb20151b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43016/0 (socket says 172.21.15.77:43016) 2022-04-03T19:51:58.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.480+0000 7fdb1ffff700 1 -- 172.21.15.77:0/3861725958 learned_addr learned my addr 172.21.15.77:0/3861725958 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:51:58.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.480+0000 7fdb1ffff700 1 -- 172.21.15.77:0/3861725958 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb2006bbe0 msgr2=0x7fdb20167bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:58.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.480+0000 7fdb1ffff700 1 --2- 172.21.15.77:0/3861725958 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb2006bbe0 0x7fdb20167bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:58.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.480+0000 7fdb1ffff700 1 -- 172.21.15.77:0/3861725958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb2006c500 msgr2=0x7fdb20168460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:58.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.480+0000 7fdb1ffff700 1 --2- 172.21.15.77:0/3861725958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb2006c500 0x7fdb20168460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:58.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.480+0000 7fdb1ffff700 1 -- 172.21.15.77:0/3861725958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0c0092a0 con 0x7fdb2006bfb0 2022-04-03T19:51:58.492 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.481+0000 7fdb250d1700 1 --2- 172.21.15.77:0/3861725958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb2006c500 0x7fdb20168460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:51:58.492 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.481+0000 7fdb1ffff700 1 --2- 172.21.15.77:0/3861725958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb2006bfb0 0x7fdb20151b90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c009e00 tx=0x7fdb0c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:58.492 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.481+0000 7fdb1cff9700 1 -- 172.21.15.77:0/3861725958 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c000bc0 con 0x7fdb2006bfb0 2022-04-03T19:51:58.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.481+0000 7fdb1cff9700 1 -- 172.21.15.77:0/3861725958 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb0c011900 con 0x7fdb2006bfb0 2022-04-03T19:51:58.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.481+0000 7fdb1cff9700 1 -- 172.21.15.77:0/3861725958 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c025420 con 0x7fdb2006bfb0 2022-04-03T19:51:58.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.481+0000 7fdb258d2700 1 -- 172.21.15.77:0/3861725958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb201670d0 con 0x7fdb2006bfb0 2022-04-03T19:51:58.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.481+0000 7fdb258d2700 1 -- 172.21.15.77:0/3861725958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb20167690 con 0x7fdb2006bfb0 2022-04-03T19:51:58.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.482+0000 7fdb258d2700 1 -- 172.21.15.77:0/3861725958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb2004b050 con 0x7fdb2006bfb0 2022-04-03T19:51:58.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.483+0000 7fdb1cff9700 1 -- 172.21.15.77:0/3861725958 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fdb0c02c030 con 0x7fdb2006bfb0 2022-04-03T19:51:58.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.484+0000 7fdb1cff9700 1 --2- 172.21.15.77:0/3861725958 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdb08063ce0 0x7fdb08066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:51:58.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.484+0000 7fdb1cff9700 1 -- 172.21.15.77:0/3861725958 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fdb0c081e20 con 0x7fdb2006bfb0 2022-04-03T19:51:58.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.484+0000 7fdb248d0700 1 --2- 172.21.15.77:0/3861725958 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdb08063ce0 0x7fdb08066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:51:58.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.485+0000 7fdb248d0700 1 --2- 172.21.15.77:0/3861725958 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdb08063ce0 0x7fdb08066190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdb14004780 tx=0x7fdb14007cd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:51:58.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.488+0000 7fdb1cff9700 1 -- 172.21.15.77:0/3861725958 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fdb0c050870 con 0x7fdb2006bfb0 2022-04-03T19:51:58.721 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.720+0000 7fdb258d2700 1 -- 172.21.15.77:0/3861725958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb20063d10 con 0x7fdb2006bfb0 2022-04-03T19:51:58.722 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.721+0000 7fdb1cff9700 1 -- 172.21.15.77:0/3861725958 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5356 (secure 0 0 0) 0x7fdb0c02a020 con 0x7fdb2006bfb0 2022-04-03T19:51:58.723 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:51:58.723 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:51:57.273217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"e","rank":0,"incarnation":27,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:51:58.725 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.723+0000 7fdb258d2700 1 -- 172.21.15.77:0/3861725958 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdb08063ce0 msgr2=0x7fdb08066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:58.725 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.723+0000 7fdb258d2700 1 --2- 172.21.15.77:0/3861725958 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdb08063ce0 0x7fdb08066190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdb14004780 tx=0x7fdb14007cd0 comp rx=0 tx=0).stop 2022-04-03T19:51:58.726 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.723+0000 7fdb258d2700 1 -- 172.21.15.77:0/3861725958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb2006bfb0 msgr2=0x7fdb20151b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:51:58.726 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.723+0000 7fdb258d2700 1 --2- 172.21.15.77:0/3861725958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb2006bfb0 0x7fdb20151b90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c009e00 tx=0x7fdb0c017000 comp rx=0 tx=0).stop 2022-04-03T19:51:58.726 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.723+0000 7fdb258d2700 1 -- 172.21.15.77:0/3861725958 shutdown_connections 2022-04-03T19:51:58.726 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.723+0000 7fdb258d2700 1 --2- 172.21.15.77:0/3861725958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb2006c500 0x7fdb20168460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:58.726 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.723+0000 7fdb258d2700 1 --2- 172.21.15.77:0/3861725958 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdb08063ce0 0x7fdb08066190 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:58.727 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.723+0000 7fdb258d2700 1 --2- 172.21.15.77:0/3861725958 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb2006bbe0 0x7fdb20167bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:58.727 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.723+0000 7fdb258d2700 1 --2- 172.21.15.77:0/3861725958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb2006bfb0 0x7fdb20151b90 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:51:58.727 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.723+0000 7fdb258d2700 1 -- 172.21.15.77:0/3861725958 >> 172.21.15.77:0/3861725958 conn(0x7fdb200cb6b0 msgr2=0x7fdb20149da0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:51:58.727 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.723+0000 7fdb258d2700 1 -- 172.21.15.77:0/3861725958 shutdown_connections 2022-04-03T19:51:58.728 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:51:58.724+0000 7fdb258d2700 1 -- 172.21.15.77:0/3861725958 wait complete. 2022-04-03T19:51:58.728 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 28 2022-04-03T19:51:58.749 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:52:00.750 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:52:00.957 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.955+0000 7f953890d700 1 Processor -- start 2022-04-03T19:52:00.957 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.955+0000 7f953890d700 1 -- start start 2022-04-03T19:52:00.960 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.956+0000 7f953890d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f953406ccf0 0x7f95340aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:00.960 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.956+0000 7f953890d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95340c5760 0x7f95340af060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:00.961 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.956+0000 7f953890d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9534158810 0x7f9534067360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:00.961 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.956+0000 7f953890d700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f953406d930 con 0x7f953406ccf0 2022-04-03T19:52:00.961 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.956+0000 7f953890d700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9534066ed0 con 0x7f9534158810 2022-04-03T19:52:00.961 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.956+0000 7f953890d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f953406e380 con 0x7f95340c5760 2022-04-03T19:52:00.962 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.957+0000 7f95337fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f953406ccf0 0x7f95340aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:00.962 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.957+0000 7f95337fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f953406ccf0 0x7f95340aeb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43018/0 (socket says 172.21.15.77:43018) 2022-04-03T19:52:00.962 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.957+0000 7f95337fe700 1 -- 172.21.15.77:0/294060444 learned_addr learned my addr 172.21.15.77:0/294060444 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:52:00.963 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.957+0000 7f9533fff700 1 --2- 172.21.15.77:0/294060444 >> v2:172.21.15.186:3300/0 conn(0x7f9534158810 0x7f9534067360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:00.963 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.957+0000 7f9532ffd700 1 --2- 172.21.15.77:0/294060444 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95340c5760 0x7f95340af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:00.963 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.957+0000 7f95337fe700 1 -- 172.21.15.77:0/294060444 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95340c5760 msgr2=0x7f95340af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:52:00.963 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.957+0000 7f95337fe700 1 --2- 172.21.15.77:0/294060444 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95340c5760 0x7f95340af060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:00.964 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.957+0000 7f95337fe700 1 -- 172.21.15.77:0/294060444 >> v2:172.21.15.186:3300/0 conn(0x7f9534158810 msgr2=0x7f9534067360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:52:00.964 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.957+0000 7f95337fe700 1 --2- 172.21.15.77:0/294060444 >> v2:172.21.15.186:3300/0 conn(0x7f9534158810 0x7f9534067360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:00.964 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.957+0000 7f95337fe700 1 -- 172.21.15.77:0/294060444 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95340c6790 con 0x7f953406ccf0 2022-04-03T19:52:00.964 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.957+0000 7f95337fe700 1 --2- 172.21.15.77:0/294060444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f953406ccf0 0x7f95340aeb20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f953406ca90 tx=0x7f95240090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49932b6613c10f8d server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:52:00.964 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.958+0000 7f95327fc700 1 -- 172.21.15.77:0/294060444 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9524004030 con 0x7f953406ccf0 2022-04-03T19:52:00.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.958+0000 7f95327fc700 1 -- 172.21.15.77:0/294060444 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9524004190 con 0x7f953406ccf0 2022-04-03T19:52:00.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.958+0000 7f95327fc700 1 -- 172.21.15.77:0/294060444 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9524011630 con 0x7f953406ccf0 2022-04-03T19:52:00.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.958+0000 7f953890d700 1 -- 172.21.15.77:0/294060444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f953406ccf0 msgr2=0x7f95340aeb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:00.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.958+0000 7f953890d700 1 --2- 172.21.15.77:0/294060444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f953406ccf0 0x7f95340aeb20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f953406ca90 tx=0x7f95240090f0 comp rx=0 tx=0).stop 2022-04-03T19:52:00.966 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.958+0000 7f953890d700 1 -- 172.21.15.77:0/294060444 shutdown_connections 2022-04-03T19:52:00.966 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.958+0000 7f953890d700 1 --2- 172.21.15.77:0/294060444 >> v2:172.21.15.186:3300/0 conn(0x7f9534158810 0x7f9534067360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:00.966 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.958+0000 7f953890d700 1 --2- 172.21.15.77:0/294060444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f953406ccf0 0x7f95340aeb20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:00.966 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.958+0000 7f953890d700 1 --2- 172.21.15.77:0/294060444 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95340c5760 0x7f95340af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:00.967 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.958+0000 7f953890d700 1 -- 172.21.15.77:0/294060444 >> 172.21.15.77:0/294060444 conn(0x7f9534069990 msgr2=0x7f95340626d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:52:00.967 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.958+0000 7f953890d700 1 -- 172.21.15.77:0/294060444 shutdown_connections 2022-04-03T19:52:00.967 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.958+0000 7f953890d700 1 -- 172.21.15.77:0/294060444 wait complete. 2022-04-03T19:52:00.970 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.960+0000 7f953890d700 1 Processor -- start 2022-04-03T19:52:00.970 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.960+0000 7f953890d700 1 -- start start 2022-04-03T19:52:00.970 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.961+0000 7f953890d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f953406ccf0 0x7f95340c29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:00.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.961+0000 7f953890d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95340c5760 0x7f95340c1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:00.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.961+0000 7f953890d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9534158810 0x7f95340a9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:00.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.961+0000 7f953890d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95340ab9f0 con 0x7f95340c5760 2022-04-03T19:52:00.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.961+0000 7f953890d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95341660c0 con 0x7f9534158810 2022-04-03T19:52:00.971 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.961+0000 7f953890d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95340bfb00 con 0x7f953406ccf0 2022-04-03T19:52:00.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.961+0000 7f9532ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95340c5760 0x7f95340c1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:00.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.961+0000 7f9533fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9534158810 0x7f95340a9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:00.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.961+0000 7f95337fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f953406ccf0 0x7f95340c29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:00.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.961+0000 7f9532ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95340c5760 0x7f95340c1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43020/0 (socket says 172.21.15.77:43020) 2022-04-03T19:52:00.973 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.961+0000 7f9532ffd700 1 -- 172.21.15.77:0/3673411103 learned_addr learned my addr 172.21.15.77:0/3673411103 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:52:00.973 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.962+0000 7f9532ffd700 1 -- 172.21.15.77:0/3673411103 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f953406ccf0 msgr2=0x7f95340c29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:00.973 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.962+0000 7f9532ffd700 1 --2- 172.21.15.77:0/3673411103 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f953406ccf0 0x7f95340c29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:00.973 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.962+0000 7f9532ffd700 1 -- 172.21.15.77:0/3673411103 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9534158810 msgr2=0x7f95340a9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:00.974 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.962+0000 7f9532ffd700 1 --2- 172.21.15.77:0/3673411103 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9534158810 0x7f95340a9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:00.974 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.962+0000 7f9532ffd700 1 -- 172.21.15.77:0/3673411103 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9524009370 con 0x7f95340c5760 2022-04-03T19:52:00.974 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.962+0000 7f9532ffd700 1 --2- 172.21.15.77:0/3673411103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95340c5760 0x7f95340c1b50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f952800d8c0 tx=0x7f9528008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:52:00.974 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.962+0000 7f951b7fe700 1 -- 172.21.15.77:0/3673411103 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95280125f0 con 0x7f95340c5760 2022-04-03T19:52:00.974 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.962+0000 7f951b7fe700 1 -- 172.21.15.77:0/3673411103 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9528012750 con 0x7f95340c5760 2022-04-03T19:52:00.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.963+0000 7f953890d700 1 -- 172.21.15.77:0/3673411103 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9534165de0 con 0x7f95340c5760 2022-04-03T19:52:00.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.963+0000 7f951b7fe700 1 -- 172.21.15.77:0/3673411103 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9528012af0 con 0x7f95340c5760 2022-04-03T19:52:00.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.963+0000 7f953890d700 1 -- 172.21.15.77:0/3673411103 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95340bf820 con 0x7f95340c5760 2022-04-03T19:52:00.975 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.964+0000 7f953890d700 1 -- 172.21.15.77:0/3673411103 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9534069f50 con 0x7f95340c5760 2022-04-03T19:52:00.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.964+0000 7f951b7fe700 1 -- 172.21.15.77:0/3673411103 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f952801e460 con 0x7f95340c5760 2022-04-03T19:52:00.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.965+0000 7f951b7fe700 1 --2- 172.21.15.77:0/3673411103 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9514063ce0 0x7f9514066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:00.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.965+0000 7f951b7fe700 1 -- 172.21.15.77:0/3673411103 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9528080ec0 con 0x7f95340c5760 2022-04-03T19:52:00.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.966+0000 7f95337fe700 1 --2- 172.21.15.77:0/3673411103 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9514063ce0 0x7f9514066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:00.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.966+0000 7f95337fe700 1 --2- 172.21.15.77:0/3673411103 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9514063ce0 0x7f9514066190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f953406ca90 tx=0x7f9524016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:52:00.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:00.969+0000 7f951b7fe700 1 -- 172.21.15.77:0/3673411103 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f952804f8b0 con 0x7f95340c5760 2022-04-03T19:52:01.198 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:01.197+0000 7f953890d700 1 -- 172.21.15.77:0/3673411103 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f953404b050 con 0x7f95340c5760 2022-04-03T19:52:01.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:01.197+0000 7f951b7fe700 1 -- 172.21.15.77:0/3673411103 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5356 (secure 0 0 0) 0x7f9528052ed0 con 0x7f95340c5760 2022-04-03T19:52:01.199 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:52:01.200 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:51:57.273217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"e","rank":0,"incarnation":27,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:52:01.202 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:01.200+0000 7f953890d700 1 -- 172.21.15.77:0/3673411103 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9514063ce0 msgr2=0x7f9514066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:01.202 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:01.200+0000 7f953890d700 1 --2- 172.21.15.77:0/3673411103 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9514063ce0 0x7f9514066190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f953406ca90 tx=0x7f9524016000 comp rx=0 tx=0).stop 2022-04-03T19:52:01.202 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:01.200+0000 7f953890d700 1 -- 172.21.15.77:0/3673411103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95340c5760 msgr2=0x7f95340c1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:01.202 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:01.200+0000 7f953890d700 1 --2- 172.21.15.77:0/3673411103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95340c5760 0x7f95340c1b50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f952800d8c0 tx=0x7f9528008040 comp rx=0 tx=0).stop 2022-04-03T19:52:01.203 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:01.200+0000 7f953890d700 1 -- 172.21.15.77:0/3673411103 shutdown_connections 2022-04-03T19:52:01.203 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:01.200+0000 7f953890d700 1 --2- 172.21.15.77:0/3673411103 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9534158810 0x7f95340a9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:01.203 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:01.200+0000 7f953890d700 1 --2- 172.21.15.77:0/3673411103 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9514063ce0 0x7f9514066190 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:01.203 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:01.200+0000 7f953890d700 1 --2- 172.21.15.77:0/3673411103 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f953406ccf0 0x7f95340c29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:01.204 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:01.200+0000 7f953890d700 1 --2- 172.21.15.77:0/3673411103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95340c5760 0x7f95340c1b50 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:01.204 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:01.200+0000 7f953890d700 1 -- 172.21.15.77:0/3673411103 >> 172.21.15.77:0/3673411103 conn(0x7f9534069990 msgr2=0x7f9534155e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:52:01.204 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:01.200+0000 7f953890d700 1 -- 172.21.15.77:0/3673411103 shutdown_connections 2022-04-03T19:52:01.204 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:01.200+0000 7f953890d700 1 -- 172.21.15.77:0/3673411103 wait complete. 2022-04-03T19:52:01.205 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 28 2022-04-03T19:52:01.225 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:52:03.226 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:52:03.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.431+0000 7fbf90a39700 1 Processor -- start 2022-04-03T19:52:03.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.431+0000 7fbf90a39700 1 -- start start 2022-04-03T19:52:03.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.432+0000 7fbf90a39700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf8c06ccf0 0x7fbf8c0aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:03.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.432+0000 7fbf90a39700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbf8c0c5760 0x7fbf8c0af060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:03.436 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.432+0000 7fbf90a39700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf8c066ed0 con 0x7fbf8c158810 2022-04-03T19:52:03.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.432+0000 7fbf90a39700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf8c06d930 con 0x7fbf8c0c5760 2022-04-03T19:52:03.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.432+0000 7fbf90a39700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c06e380 con 0x7fbf8c06ccf0 2022-04-03T19:52:03.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.432+0000 7fbf8ad9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf8c06ccf0 0x7fbf8c0aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:03.437 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.432+0000 7fbf83fff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbf8c0c5760 0x7fbf8c0af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:03.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.432+0000 7fbf8b59e700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fbf8c158810 0x7fbf8c067360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44290/0 (socket says 172.21.15.77:44290) 2022-04-03T19:52:03.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.432+0000 7fbf8b59e700 1 -- 172.21.15.77:0/1479222965 learned_addr learned my addr 172.21.15.77:0/1479222965 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:52:03.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.433+0000 7fbf8a59c700 1 -- 172.21.15.77:0/1479222965 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881643437 0 0) 0x7fbf8c066ed0 con 0x7fbf8c158810 2022-04-03T19:52:03.438 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.433+0000 7fbf8a59c700 1 -- 172.21.15.77:0/1479222965 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf6c0033f0 con 0x7fbf8c158810 2022-04-03T19:52:03.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.433+0000 7fbf8ad9d700 1 -- 172.21.15.77:0/1479222965 >> v2:172.21.15.186:3300/0 conn(0x7fbf8c0c5760 msgr2=0x7fbf8c0af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:52:03.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.433+0000 7fbf8ad9d700 1 --2- 172.21.15.77:0/1479222965 >> v2:172.21.15.186:3300/0 conn(0x7fbf8c0c5760 0x7fbf8c0af060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:03.439 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.433+0000 7fbf8ad9d700 1 -- 172.21.15.77:0/1479222965 >> v1:172.21.15.186:6789/0 conn(0x7fbf8c158810 legacy=0x7fbf8c067360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:03.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.433+0000 7fbf8ad9d700 1 -- 172.21.15.77:0/1479222965 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8c0c6790 con 0x7fbf8c06ccf0 2022-04-03T19:52:03.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.433+0000 7fbf83fff700 1 --2- 172.21.15.77:0/1479222965 >> v2:172.21.15.186:3300/0 conn(0x7fbf8c0c5760 0x7fbf8c0af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:52:03.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.433+0000 7fbf8ad9d700 1 --2- 172.21.15.77:0/1479222965 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf8c06ccf0 0x7fbf8c0aeb20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c009760 tx=0x7fbf7c009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8faf26a9296c06c7 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:52:03.440 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.433+0000 7fbf8a59c700 1 -- 172.21.15.77:0/1479222965 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c011630 con 0x7fbf8c06ccf0 2022-04-03T19:52:03.441 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.434+0000 7fbf8a59c700 1 -- 172.21.15.77:0/1479222965 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf7c011790 con 0x7fbf8c06ccf0 2022-04-03T19:52:03.441 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.434+0000 7fbf8a59c700 1 -- 172.21.15.77:0/1479222965 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c011b30 con 0x7fbf8c06ccf0 2022-04-03T19:52:03.441 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.434+0000 7fbf90a39700 1 -- 172.21.15.77:0/1479222965 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf8c06ccf0 msgr2=0x7fbf8c0aeb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:03.441 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.434+0000 7fbf90a39700 1 --2- 172.21.15.77:0/1479222965 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf8c06ccf0 0x7fbf8c0aeb20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c009760 tx=0x7fbf7c009060 comp rx=0 tx=0).stop 2022-04-03T19:52:03.442 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.434+0000 7fbf90a39700 1 -- 172.21.15.77:0/1479222965 shutdown_connections 2022-04-03T19:52:03.442 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.434+0000 7fbf90a39700 1 --2- 172.21.15.77:0/1479222965 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf8c06ccf0 0x7fbf8c0aeb20 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:03.442 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.434+0000 7fbf90a39700 1 --2- 172.21.15.77:0/1479222965 >> v2:172.21.15.186:3300/0 conn(0x7fbf8c0c5760 0x7fbf8c0af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:03.442 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.434+0000 7fbf90a39700 1 -- 172.21.15.77:0/1479222965 >> 172.21.15.77:0/1479222965 conn(0x7fbf8c069990 msgr2=0x7fbf8c0626d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:52:03.443 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.434+0000 7fbf90a39700 1 -- 172.21.15.77:0/1479222965 shutdown_connections 2022-04-03T19:52:03.443 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.434+0000 7fbf90a39700 1 -- 172.21.15.77:0/1479222965 wait complete. 2022-04-03T19:52:03.446 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.436+0000 7fbf90a39700 1 Processor -- start 2022-04-03T19:52:03.446 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.436+0000 7fbf90a39700 1 -- start start 2022-04-03T19:52:03.446 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.437+0000 7fbf90a39700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf8c06ccf0 0x7fbf8c0c2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:03.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.437+0000 7fbf90a39700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf8c0c5760 0x7fbf8c0aa7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:03.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.437+0000 7fbf90a39700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf8c158810 0x7fbf8c0aad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:03.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.437+0000 7fbf90a39700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c169070 con 0x7fbf8c06ccf0 2022-04-03T19:52:03.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.437+0000 7fbf90a39700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c0aa330 con 0x7fbf8c158810 2022-04-03T19:52:03.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.437+0000 7fbf90a39700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c168b00 con 0x7fbf8c0c5760 2022-04-03T19:52:03.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.437+0000 7fbf8ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf8c06ccf0 0x7fbf8c0c2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:03.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.437+0000 7fbf8ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf8c06ccf0 0x7fbf8c0c2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43022/0 (socket says 172.21.15.77:43022) 2022-04-03T19:52:03.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.437+0000 7fbf8ad9d700 1 -- 172.21.15.77:0/3341284227 learned_addr learned my addr 172.21.15.77:0/3341284227 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:52:03.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.437+0000 7fbf83fff700 1 --2- 172.21.15.77:0/3341284227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf8c0c5760 0x7fbf8c0aa7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:03.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.437+0000 7fbf8b59e700 1 --2- 172.21.15.77:0/3341284227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf8c158810 0x7fbf8c0aad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:03.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.438+0000 7fbf8ad9d700 1 -- 172.21.15.77:0/3341284227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf8c0c5760 msgr2=0x7fbf8c0aa7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:03.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.438+0000 7fbf8ad9d700 1 --2- 172.21.15.77:0/3341284227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf8c0c5760 0x7fbf8c0aa7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:03.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.438+0000 7fbf8ad9d700 1 -- 172.21.15.77:0/3341284227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf8c158810 msgr2=0x7fbf8c0aad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:03.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.438+0000 7fbf8ad9d700 1 --2- 172.21.15.77:0/3341284227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf8c158810 0x7fbf8c0aad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:03.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.438+0000 7fbf8ad9d700 1 -- 172.21.15.77:0/3341284227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf740092a0 con 0x7fbf8c06ccf0 2022-04-03T19:52:03.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.438+0000 7fbf8ad9d700 1 --2- 172.21.15.77:0/3341284227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf8c06ccf0 0x7fbf8c0c2450 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c009d50 tx=0x7fbf7c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:52:03.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.438+0000 7fbf82ffd700 1 -- 172.21.15.77:0/3341284227 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c0036f0 con 0x7fbf8c06ccf0 2022-04-03T19:52:03.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.438+0000 7fbf82ffd700 1 -- 172.21.15.77:0/3341284227 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf7c003850 con 0x7fbf8c06ccf0 2022-04-03T19:52:03.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.438+0000 7fbf82ffd700 1 -- 172.21.15.77:0/3341284227 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c003c10 con 0x7fbf8c06ccf0 2022-04-03T19:52:03.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.438+0000 7fbf90a39700 1 -- 172.21.15.77:0/3341284227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf7c0092a0 con 0x7fbf8c06ccf0 2022-04-03T19:52:03.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.439+0000 7fbf90a39700 1 -- 172.21.15.77:0/3341284227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf8c0c5030 con 0x7fbf8c06ccf0 2022-04-03T19:52:03.452 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.439+0000 7fbf90a39700 1 -- 172.21.15.77:0/3341284227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf8c0aa610 con 0x7fbf8c06ccf0 2022-04-03T19:52:03.452 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.440+0000 7fbf82ffd700 1 -- 172.21.15.77:0/3341284227 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fbf7c00f050 con 0x7fbf8c06ccf0 2022-04-03T19:52:03.452 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.441+0000 7fbf82ffd700 1 --2- 172.21.15.77:0/3341284227 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbf70063cb0 0x7fbf70066160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:03.452 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.441+0000 7fbf82ffd700 1 -- 172.21.15.77:0/3341284227 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fbf7c081f30 con 0x7fbf8c06ccf0 2022-04-03T19:52:03.452 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.441+0000 7fbf83fff700 1 --2- 172.21.15.77:0/3341284227 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbf70063cb0 0x7fbf70066160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:03.453 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.442+0000 7fbf83fff700 1 --2- 172.21.15.77:0/3341284227 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbf70063cb0 0x7fbf70066160 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c157a00 tx=0x7fbf74009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:52:03.453 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.445+0000 7fbf82ffd700 1 -- 172.21.15.77:0/3341284227 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fbf7c0508d0 con 0x7fbf8c06ccf0 2022-04-03T19:52:03.676 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.675+0000 7fbf90a39700 1 -- 172.21.15.77:0/3341284227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf8c04b050 con 0x7fbf8c06ccf0 2022-04-03T19:52:03.677 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.675+0000 7fbf82ffd700 1 -- 172.21.15.77:0/3341284227 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5353 (secure 0 0 0) 0x7fbf7c015070 con 0x7fbf8c06ccf0 2022-04-03T19:52:03.677 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:52:03.677 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:52:02.295867+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"e","rank":0,"incarnation":27,"state":"up:rejoin","state_seq":26,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:52:03.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.677+0000 7fbf90a39700 1 -- 172.21.15.77:0/3341284227 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbf70063cb0 msgr2=0x7fbf70066160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:03.680 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.677+0000 7fbf90a39700 1 --2- 172.21.15.77:0/3341284227 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbf70063cb0 0x7fbf70066160 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c157a00 tx=0x7fbf74009060 comp rx=0 tx=0).stop 2022-04-03T19:52:03.680 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.677+0000 7fbf90a39700 1 -- 172.21.15.77:0/3341284227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf8c06ccf0 msgr2=0x7fbf8c0c2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:03.680 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.677+0000 7fbf90a39700 1 --2- 172.21.15.77:0/3341284227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf8c06ccf0 0x7fbf8c0c2450 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c009d50 tx=0x7fbf7c017000 comp rx=0 tx=0).stop 2022-04-03T19:52:03.680 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.678+0000 7fbf90a39700 1 -- 172.21.15.77:0/3341284227 shutdown_connections 2022-04-03T19:52:03.681 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.678+0000 7fbf90a39700 1 --2- 172.21.15.77:0/3341284227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf8c158810 0x7fbf8c0aad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:03.681 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.678+0000 7fbf90a39700 1 --2- 172.21.15.77:0/3341284227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbf8c06ccf0 0x7fbf8c0c2450 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:03.681 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.678+0000 7fbf90a39700 1 --2- 172.21.15.77:0/3341284227 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbf70063cb0 0x7fbf70066160 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:03.681 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.678+0000 7fbf90a39700 1 --2- 172.21.15.77:0/3341284227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf8c0c5760 0x7fbf8c0aa7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:03.681 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.678+0000 7fbf90a39700 1 -- 172.21.15.77:0/3341284227 >> 172.21.15.77:0/3341284227 conn(0x7fbf8c069990 msgr2=0x7fbf8c155d70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:52:03.682 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.678+0000 7fbf90a39700 1 -- 172.21.15.77:0/3341284227 shutdown_connections 2022-04-03T19:52:03.682 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:03.678+0000 7fbf90a39700 1 -- 172.21.15.77:0/3341284227 wait complete. 2022-04-03T19:52:03.682 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 29 2022-04-03T19:52:03.704 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 29, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-04-03T19:44:21.275716+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 29, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5239": { "addr": "172.21.15.77:6837/3264384876", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6836", "nonce": 3264384876, "type": "v2" }, { "addr": "172.21.15.77:6837", "nonce": 3264384876, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 5239, "incarnation": 27, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:rejoin", "state_seq": 26 } }, "last_failure": 0, "last_failure_osd_epoch": 50, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-04-03T19:52:02.295867+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5239 } } } ], "standbys": [ { "addr": "172.21.15.77:6834/2116223509", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6832", "nonce": 2116223509, "type": "v2" }, { "addr": "172.21.15.77:6834", "nonce": 2116223509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4245, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6836/2374276747", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2374276747, "type": "v2" }, { "addr": "172.21.15.186:6836", "nonce": 2374276747, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4259, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/610990853", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6838", "nonce": 610990853, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 610990853, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4265, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.77:6835/170053412", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6833", "nonce": 170053412, "type": "v2" }, { "addr": "172.21.15.77:6835", "nonce": 170053412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4271, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-04-03T19:52:05.705 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:52:05.912 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.911+0000 7fba394d8700 1 Processor -- start 2022-04-03T19:52:05.913 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.911+0000 7fba394d8700 1 -- start start 2022-04-03T19:52:05.915 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.912+0000 7fba394d8700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fba3406c410 0x7fba3406c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:05.915 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.912+0000 7fba394d8700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fba34069a00 0x7fba34069e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:05.915 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.912+0000 7fba394d8700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba34155cf0 0x7fba34156170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:05.915 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.912+0000 7fba394d8700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba3406d600 con 0x7fba3406c410 2022-04-03T19:52:05.915 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.912+0000 7fba394d8700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba3406bd20 con 0x7fba34069a00 2022-04-03T19:52:05.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.912+0000 7fba394d8700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba340c2bf0 con 0x7fba34155cf0 2022-04-03T19:52:05.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.912+0000 7fba33fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fba3406c410 0x7fba3406c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:05.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.912+0000 7fba33fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fba3406c410 0x7fba3406c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43024/0 (socket says 172.21.15.77:43024) 2022-04-03T19:52:05.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.912+0000 7fba33fff700 1 -- 172.21.15.77:0/2009275142 learned_addr learned my addr 172.21.15.77:0/2009275142 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:52:05.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.912+0000 7fba38cd7700 1 --2- 172.21.15.77:0/2009275142 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba34155cf0 0x7fba34156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:05.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.912+0000 7fba337fe700 1 --2- 172.21.15.77:0/2009275142 >> v2:172.21.15.186:3300/0 conn(0x7fba34069a00 0x7fba34069e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:05.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.913+0000 7fba33fff700 1 -- 172.21.15.77:0/2009275142 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba34155cf0 msgr2=0x7fba34156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:52:05.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.913+0000 7fba33fff700 1 --2- 172.21.15.77:0/2009275142 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba34155cf0 0x7fba34156170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:05.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.913+0000 7fba33fff700 1 -- 172.21.15.77:0/2009275142 >> v2:172.21.15.186:3300/0 conn(0x7fba34069a00 msgr2=0x7fba34069e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:52:05.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.913+0000 7fba33fff700 1 --2- 172.21.15.77:0/2009275142 >> v2:172.21.15.186:3300/0 conn(0x7fba34069a00 0x7fba34069e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:05.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.913+0000 7fba33fff700 1 -- 172.21.15.77:0/2009275142 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba340c2760 con 0x7fba3406c410 2022-04-03T19:52:05.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.913+0000 7fba33fff700 1 --2- 172.21.15.77:0/2009275142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba3406c410 0x7fba3406c7e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fba3406e7a0 tx=0x7fba240090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b83c8f9841b898 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:52:05.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.913+0000 7fba32ffd700 1 -- 172.21.15.77:0/2009275142 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba24004030 con 0x7fba3406c410 2022-04-03T19:52:05.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.913+0000 7fba32ffd700 1 -- 172.21.15.77:0/2009275142 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba24004190 con 0x7fba3406c410 2022-04-03T19:52:05.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.913+0000 7fba32ffd700 1 -- 172.21.15.77:0/2009275142 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba24011630 con 0x7fba3406c410 2022-04-03T19:52:05.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.914+0000 7fba394d8700 1 -- 172.21.15.77:0/2009275142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba3406c410 msgr2=0x7fba3406c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:05.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.914+0000 7fba394d8700 1 --2- 172.21.15.77:0/2009275142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba3406c410 0x7fba3406c7e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fba3406e7a0 tx=0x7fba240090f0 comp rx=0 tx=0).stop 2022-04-03T19:52:05.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.914+0000 7fba394d8700 1 -- 172.21.15.77:0/2009275142 shutdown_connections 2022-04-03T19:52:05.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.914+0000 7fba394d8700 1 --2- 172.21.15.77:0/2009275142 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba34155cf0 0x7fba34156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:05.922 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.914+0000 7fba394d8700 1 --2- 172.21.15.77:0/2009275142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba3406c410 0x7fba3406c7e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:05.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.914+0000 7fba394d8700 1 --2- 172.21.15.77:0/2009275142 >> v2:172.21.15.186:3300/0 conn(0x7fba34069a00 0x7fba34069e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:05.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.914+0000 7fba394d8700 1 -- 172.21.15.77:0/2009275142 >> 172.21.15.77:0/2009275142 conn(0x7fba340c7b60 msgr2=0x7fba340c7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:52:05.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.914+0000 7fba394d8700 1 -- 172.21.15.77:0/2009275142 shutdown_connections 2022-04-03T19:52:05.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.914+0000 7fba394d8700 1 -- 172.21.15.77:0/2009275142 wait complete. 2022-04-03T19:52:05.923 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.916+0000 7fba394d8700 1 Processor -- start 2022-04-03T19:52:05.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.916+0000 7fba394d8700 1 -- start start 2022-04-03T19:52:05.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.917+0000 7fba394d8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba34069a00 0x7fba3415a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:05.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.917+0000 7fba394d8700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba3406c410 0x7fba3414ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:05.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.917+0000 7fba394d8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba34155cf0 0x7fba3414d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:05.924 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.917+0000 7fba394d8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba34169a70 con 0x7fba34069a00 2022-04-03T19:52:05.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.917+0000 7fba394d8700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba341579f0 con 0x7fba34155cf0 2022-04-03T19:52:05.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.917+0000 7fba394d8700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba34169c60 con 0x7fba3406c410 2022-04-03T19:52:05.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.917+0000 7fba33fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba34069a00 0x7fba3415a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:05.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.917+0000 7fba33fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba34069a00 0x7fba3415a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43026/0 (socket says 172.21.15.77:43026) 2022-04-03T19:52:05.925 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.917+0000 7fba33fff700 1 -- 172.21.15.77:0/3683581509 learned_addr learned my addr 172.21.15.77:0/3683581509 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:52:05.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.917+0000 7fba337fe700 1 --2- 172.21.15.77:0/3683581509 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba3406c410 0x7fba3414ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:05.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.917+0000 7fba38cd7700 1 --2- 172.21.15.77:0/3683581509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba34155cf0 0x7fba3414d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:05.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.918+0000 7fba337fe700 1 -- 172.21.15.77:0/3683581509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba34155cf0 msgr2=0x7fba3414d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:05.926 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.918+0000 7fba337fe700 1 --2- 172.21.15.77:0/3683581509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba34155cf0 0x7fba3414d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:05.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.918+0000 7fba337fe700 1 -- 172.21.15.77:0/3683581509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba34069a00 msgr2=0x7fba3415a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:05.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.918+0000 7fba337fe700 1 --2- 172.21.15.77:0/3683581509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba34069a00 0x7fba3415a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:05.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.918+0000 7fba337fe700 1 -- 172.21.15.77:0/3683581509 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba24009370 con 0x7fba3406c410 2022-04-03T19:52:05.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.918+0000 7fba38cd7700 1 --2- 172.21.15.77:0/3683581509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba34155cf0 0x7fba3414d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:52:05.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.918+0000 7fba33fff700 1 --2- 172.21.15.77:0/3683581509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba34069a00 0x7fba3415a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:52:05.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.918+0000 7fba337fe700 1 --2- 172.21.15.77:0/3683581509 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba3406c410 0x7fba3414ce40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fba2800e720 tx=0x7fba28009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:52:05.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.918+0000 7fba1bfff700 1 -- 172.21.15.77:0/3683581509 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28013680 con 0x7fba3406c410 2022-04-03T19:52:05.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.918+0000 7fba1bfff700 1 -- 172.21.15.77:0/3683581509 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba280137e0 con 0x7fba3406c410 2022-04-03T19:52:05.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.918+0000 7fba394d8700 1 -- 172.21.15.77:0/3683581509 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba34169f40 con 0x7fba3406c410 2022-04-03T19:52:05.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.918+0000 7fba1bfff700 1 -- 172.21.15.77:0/3683581509 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28013ba0 con 0x7fba3406c410 2022-04-03T19:52:05.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.918+0000 7fba394d8700 1 -- 172.21.15.77:0/3683581509 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba3416a500 con 0x7fba3406c410 2022-04-03T19:52:05.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.919+0000 7fba394d8700 1 -- 172.21.15.77:0/3683581509 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba34157b90 con 0x7fba3406c410 2022-04-03T19:52:05.929 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.920+0000 7fba1bfff700 1 -- 172.21.15.77:0/3683581509 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fba28019030 con 0x7fba3406c410 2022-04-03T19:52:05.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.921+0000 7fba1bfff700 1 --2- 172.21.15.77:0/3683581509 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fba14063ce0 0x7fba14066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:05.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.921+0000 7fba1bfff700 1 -- 172.21.15.77:0/3683581509 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fba28080820 con 0x7fba3406c410 2022-04-03T19:52:05.930 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.921+0000 7fba33fff700 1 --2- 172.21.15.77:0/3683581509 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fba14063ce0 0x7fba14066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:05.931 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.922+0000 7fba33fff700 1 --2- 172.21.15.77:0/3683581509 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fba14063ce0 0x7fba14066190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fba24009310 tx=0x7fba24016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:52:05.931 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:05.926+0000 7fba1bfff700 1 -- 172.21.15.77:0/3683581509 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fba2804f180 con 0x7fba3406c410 2022-04-03T19:52:06.156 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:06.155+0000 7fba394d8700 1 -- 172.21.15.77:0/3683581509 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba340693b0 con 0x7fba3406c410 2022-04-03T19:52:06.157 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:06.156+0000 7fba1bfff700 1 -- 172.21.15.77:0/3683581509 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5353 (secure 0 0 0) 0x7fba280527a0 con 0x7fba3406c410 2022-04-03T19:52:06.157 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:52:06.158 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:52:02.295867+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"e","rank":0,"incarnation":27,"state":"up:rejoin","state_seq":26,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:52:06.160 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:06.158+0000 7fba394d8700 1 -- 172.21.15.77:0/3683581509 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fba14063ce0 msgr2=0x7fba14066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:06.160 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:06.158+0000 7fba394d8700 1 --2- 172.21.15.77:0/3683581509 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fba14063ce0 0x7fba14066190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fba24009310 tx=0x7fba24016000 comp rx=0 tx=0).stop 2022-04-03T19:52:06.161 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:06.158+0000 7fba394d8700 1 -- 172.21.15.77:0/3683581509 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba3406c410 msgr2=0x7fba3414ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:06.161 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:06.158+0000 7fba394d8700 1 --2- 172.21.15.77:0/3683581509 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba3406c410 0x7fba3414ce40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fba2800e720 tx=0x7fba28009060 comp rx=0 tx=0).stop 2022-04-03T19:52:06.161 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:06.158+0000 7fba394d8700 1 -- 172.21.15.77:0/3683581509 shutdown_connections 2022-04-03T19:52:06.161 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:06.158+0000 7fba394d8700 1 --2- 172.21.15.77:0/3683581509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba34155cf0 0x7fba3414d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:06.161 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:06.158+0000 7fba394d8700 1 --2- 172.21.15.77:0/3683581509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba34069a00 0x7fba3415a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:06.162 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:06.158+0000 7fba394d8700 1 --2- 172.21.15.77:0/3683581509 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fba14063ce0 0x7fba14066190 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:06.162 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:06.158+0000 7fba394d8700 1 --2- 172.21.15.77:0/3683581509 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fba3406c410 0x7fba3414ce40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:06.162 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:06.158+0000 7fba394d8700 1 -- 172.21.15.77:0/3683581509 >> 172.21.15.77:0/3683581509 conn(0x7fba340c7b60 msgr2=0x7fba340c3310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:52:06.162 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:06.159+0000 7fba394d8700 1 -- 172.21.15.77:0/3683581509 shutdown_connections 2022-04-03T19:52:06.163 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:06.159+0000 7fba394d8700 1 -- 172.21.15.77:0/3683581509 wait complete. 2022-04-03T19:52:06.163 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 29 2022-04-03T19:52:06.184 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:52:08.185 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:52:08.392 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.392+0000 7f9ea49b9700 1 Processor -- start 2022-04-03T19:52:08.393 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.392+0000 7f9ea49b9700 1 -- start start 2022-04-03T19:52:08.393 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.393+0000 7f9ea49b9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ea006bbe0 0x7f9ea01584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:08.394 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.393+0000 7f9ea49b9700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9ea006bfb0 0x7f9ea006d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:08.394 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.393+0000 7f9ea49b9700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ea006d260 con 0x7f9ea006c500 2022-04-03T19:52:08.394 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.393+0000 7f9ea49b9700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ea0159450 con 0x7f9ea006bfb0 2022-04-03T19:52:08.394 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.393+0000 7f9ea49b9700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea00c6330 con 0x7f9ea006bbe0 2022-04-03T19:52:08.397 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.394+0000 7f9e9e59c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9ea006bfb0 0x7f9ea006d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:08.397 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.394+0000 7f9e9ed9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ea006bbe0 0x7f9ea01584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:08.397 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.394+0000 7f9e9f59e700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f9ea006c500 0x7f9ea014eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44292/0 (socket says 172.21.15.77:44292) 2022-04-03T19:52:08.397 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.394+0000 7f9e9f59e700 1 -- 172.21.15.77:0/234960258 learned_addr learned my addr 172.21.15.77:0/234960258 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:52:08.397 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.394+0000 7f9e9e59c700 1 -- 172.21.15.77:0/234960258 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ea006bbe0 msgr2=0x7f9ea01584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:52:08.398 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.394+0000 7f9e9e59c700 1 --2- 172.21.15.77:0/234960258 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ea006bbe0 0x7f9ea01584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:08.398 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.394+0000 7f9e9e59c700 1 -- 172.21.15.77:0/234960258 >> v1:172.21.15.186:6789/0 conn(0x7f9ea006c500 legacy=0x7f9ea014eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:08.398 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.394+0000 7f9e9dd9b700 1 -- 172.21.15.77:0/234960258 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1613749683 0 0) 0x7f9ea006d260 con 0x7f9ea006c500 2022-04-03T19:52:08.398 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.394+0000 7f9e9e59c700 1 -- 172.21.15.77:0/234960258 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea0158bb0 con 0x7f9ea006bfb0 2022-04-03T19:52:08.399 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.394+0000 7f9e9e59c700 1 --2- 172.21.15.77:0/234960258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ea006bfb0 0x7f9ea006d6f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9e94009760 tx=0x7f9e94009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2466cbe0200b3763 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:52:08.399 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.395+0000 7f9e9dd9b700 1 -- 172.21.15.77:0/234960258 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94011630 con 0x7f9ea006bfb0 2022-04-03T19:52:08.399 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.395+0000 7f9e9dd9b700 1 -- 172.21.15.77:0/234960258 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e94011790 con 0x7f9ea006bfb0 2022-04-03T19:52:08.399 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.395+0000 7f9e9dd9b700 1 -- 172.21.15.77:0/234960258 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94011b50 con 0x7f9ea006bfb0 2022-04-03T19:52:08.400 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.395+0000 7f9ea49b9700 1 -- 172.21.15.77:0/234960258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ea006bfb0 msgr2=0x7f9ea006d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:08.400 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.395+0000 7f9ea49b9700 1 --2- 172.21.15.77:0/234960258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ea006bfb0 0x7f9ea006d6f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9e94009760 tx=0x7f9e94009060 comp rx=0 tx=0).stop 2022-04-03T19:52:08.400 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.395+0000 7f9ea49b9700 1 -- 172.21.15.77:0/234960258 shutdown_connections 2022-04-03T19:52:08.400 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.395+0000 7f9ea49b9700 1 --2- 172.21.15.77:0/234960258 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ea006bbe0 0x7f9ea01584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:08.401 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.395+0000 7f9ea49b9700 1 --2- 172.21.15.77:0/234960258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ea006bfb0 0x7f9ea006d6f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:08.401 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.395+0000 7f9ea49b9700 1 -- 172.21.15.77:0/234960258 >> 172.21.15.77:0/234960258 conn(0x7f9ea00cb6b0 msgr2=0x7f9ea00cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:52:08.401 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.395+0000 7f9ea49b9700 1 -- 172.21.15.77:0/234960258 shutdown_connections 2022-04-03T19:52:08.401 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.396+0000 7f9ea49b9700 1 -- 172.21.15.77:0/234960258 wait complete. 2022-04-03T19:52:08.404 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.397+0000 7f9ea49b9700 1 Processor -- start 2022-04-03T19:52:08.404 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.397+0000 7f9ea49b9700 1 -- start start 2022-04-03T19:52:08.405 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.398+0000 7f9ea49b9700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ea006bbe0 0x7f9ea0151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:08.405 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.398+0000 7f9ea49b9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ea006bfb0 0x7f9ea0166590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:08.405 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.398+0000 7f9ea49b9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ea006c500 0x7f9ea00c3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:08.405 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.398+0000 7f9ea49b9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea0155880 con 0x7f9ea006c500 2022-04-03T19:52:08.406 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.398+0000 7f9ea49b9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea0152420 con 0x7f9ea006bfb0 2022-04-03T19:52:08.406 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.398+0000 7f9ea49b9700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea0150c60 con 0x7f9ea006bbe0 2022-04-03T19:52:08.406 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.398+0000 7f9e9ed9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ea006bbe0 0x7f9ea0151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:08.406 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.399+0000 7f9e9ed9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ea006bbe0 0x7f9ea0151bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60310/0 (socket says 172.21.15.77:60310) 2022-04-03T19:52:08.407 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.399+0000 7f9e9e59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ea006bfb0 0x7f9ea0166590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:08.407 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.399+0000 7f9e9ed9d700 1 -- 172.21.15.77:0/1965991646 learned_addr learned my addr 172.21.15.77:0/1965991646 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:52:08.407 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.399+0000 7f9e9f59e700 1 --2- 172.21.15.77:0/1965991646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ea006c500 0x7f9ea00c3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:08.407 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.399+0000 7f9e9ed9d700 1 -- 172.21.15.77:0/1965991646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ea006bfb0 msgr2=0x7f9ea0166590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:08.408 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.399+0000 7f9e9ed9d700 1 --2- 172.21.15.77:0/1965991646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ea006bfb0 0x7f9ea0166590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:08.408 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.399+0000 7f9e9ed9d700 1 -- 172.21.15.77:0/1965991646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ea006c500 msgr2=0x7f9ea00c3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:08.408 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.399+0000 7f9e9ed9d700 1 --2- 172.21.15.77:0/1965991646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ea006c500 0x7f9ea00c3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:08.408 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.399+0000 7f9e9ed9d700 1 -- 172.21.15.77:0/1965991646 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e940092a0 con 0x7f9ea006bbe0 2022-04-03T19:52:08.408 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.399+0000 7f9e9e59c700 1 --2- 172.21.15.77:0/1965991646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ea006bfb0 0x7f9ea0166590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:52:08.409 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.399+0000 7f9e9ed9d700 1 --2- 172.21.15.77:0/1965991646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ea006bbe0 0x7f9ea0151bd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9e9000e930 tx=0x7f9e90009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:52:08.409 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.400+0000 7f9e86ffd700 1 -- 172.21.15.77:0/1965991646 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e90013680 con 0x7f9ea006bbe0 2022-04-03T19:52:08.409 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.400+0000 7f9e86ffd700 1 -- 172.21.15.77:0/1965991646 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e900137e0 con 0x7f9ea006bbe0 2022-04-03T19:52:08.409 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.400+0000 7f9e86ffd700 1 -- 172.21.15.77:0/1965991646 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e90013b60 con 0x7f9ea006bbe0 2022-04-03T19:52:08.410 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.400+0000 7f9ea49b9700 1 -- 172.21.15.77:0/1965991646 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea0152650 con 0x7f9ea006bbe0 2022-04-03T19:52:08.410 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.400+0000 7f9ea49b9700 1 -- 172.21.15.77:0/1965991646 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea0150610 con 0x7f9ea006bbe0 2022-04-03T19:52:08.410 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.401+0000 7f9ea49b9700 1 -- 172.21.15.77:0/1965991646 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea006ccd0 con 0x7f9ea006bbe0 2022-04-03T19:52:08.410 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.402+0000 7f9e86ffd700 1 -- 172.21.15.77:0/1965991646 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f9e90019030 con 0x7f9ea006bbe0 2022-04-03T19:52:08.411 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.403+0000 7f9e86ffd700 1 --2- 172.21.15.77:0/1965991646 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9e80063d10 0x7f9e800661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:08.411 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.403+0000 7f9e86ffd700 1 -- 172.21.15.77:0/1965991646 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9e90080940 con 0x7f9ea006bbe0 2022-04-03T19:52:08.411 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.403+0000 7f9e9e59c700 1 --2- 172.21.15.77:0/1965991646 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9e80063d10 0x7f9e800661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:08.411 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.404+0000 7f9e9e59c700 1 --2- 172.21.15.77:0/1965991646 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9e80063d10 0x7f9e800661c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9e94009760 tx=0x7f9e94017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:52:08.412 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.407+0000 7f9e86ffd700 1 -- 172.21.15.77:0/1965991646 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f9e9004f2a0 con 0x7f9ea006bbe0 2022-04-03T19:52:08.640 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.639+0000 7f9ea49b9700 1 -- 172.21.15.77:0/1965991646 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ea004b050 con 0x7f9ea006bbe0 2022-04-03T19:52:08.641 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.640+0000 7f9e86ffd700 1 -- 172.21.15.77:0/1965991646 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5353 (secure 0 0 0) 0x7f9e900528c0 con 0x7f9ea006bbe0 2022-04-03T19:52:08.641 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:52:08.641 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:52:07.666403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"e","rank":0,"incarnation":27,"state":"up:active","state_seq":28,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:52:08.644 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.643+0000 7f9ea49b9700 1 -- 172.21.15.77:0/1965991646 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9e80063d10 msgr2=0x7f9e800661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:08.645 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.643+0000 7f9ea49b9700 1 --2- 172.21.15.77:0/1965991646 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9e80063d10 0x7f9e800661c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9e94009760 tx=0x7f9e94017000 comp rx=0 tx=0).stop 2022-04-03T19:52:08.645 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.643+0000 7f9ea49b9700 1 -- 172.21.15.77:0/1965991646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ea006bbe0 msgr2=0x7f9ea0151bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:08.645 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.643+0000 7f9ea49b9700 1 --2- 172.21.15.77:0/1965991646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ea006bbe0 0x7f9ea0151bd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9e9000e930 tx=0x7f9e90009060 comp rx=0 tx=0).stop 2022-04-03T19:52:08.645 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.643+0000 7f9ea49b9700 1 -- 172.21.15.77:0/1965991646 shutdown_connections 2022-04-03T19:52:08.645 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.643+0000 7f9ea49b9700 1 --2- 172.21.15.77:0/1965991646 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ea006c500 0x7f9ea00c3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:08.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.643+0000 7f9ea49b9700 1 --2- 172.21.15.77:0/1965991646 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9e80063d10 0x7f9e800661c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:08.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.643+0000 7f9ea49b9700 1 --2- 172.21.15.77:0/1965991646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ea006bbe0 0x7f9ea0151bd0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:08.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.643+0000 7f9ea49b9700 1 --2- 172.21.15.77:0/1965991646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ea006bfb0 0x7f9ea0166590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:08.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.643+0000 7f9ea49b9700 1 -- 172.21.15.77:0/1965991646 >> 172.21.15.77:0/1965991646 conn(0x7f9ea00cb6b0 msgr2=0x7f9ea0149bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:52:08.647 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.643+0000 7f9ea49b9700 1 -- 172.21.15.77:0/1965991646 shutdown_connections 2022-04-03T19:52:08.647 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.643+0000 7f9ea49b9700 1 -- 172.21.15.77:0/1965991646 wait complete. 2022-04-03T19:52:08.647 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 30 2022-04-03T19:52:08.668 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=4850 2022-04-03T19:52:08.668 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.d 2022-04-03T19:52:08.668 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2022-04-03T19:52:08.669 INFO:tasks.ceph.mds.d:Restarting daemon 2022-04-03T19:52:08.669 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2022-04-03T19:52:08.672 INFO:tasks.ceph.mds.d:Started 2022-04-03T19:52:08.672 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:52:08.752 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2022-04-03T19:52:08.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.868+0000 7f4b9759e700 1 Processor -- start 2022-04-03T19:52:08.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.868+0000 7f4b9759e700 1 -- start start 2022-04-03T19:52:08.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.869+0000 7f4b9759e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4b980ba3c0 0x7f4b98156b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:08.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.869+0000 7f4b9759e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4b98157060 0x7f4b9814cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:08.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.869+0000 7f4b9759e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4b980678b0 0x7f4b9814d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:08.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.869+0000 7f4b9759e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b980c8020 con 0x7f4b98157060 2022-04-03T19:52:08.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.869+0000 7f4b9759e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b980c3c50 con 0x7f4b980ba3c0 2022-04-03T19:52:08.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.869+0000 7f4b9759e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b98156690 con 0x7f4b980678b0 2022-04-03T19:52:08.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.870+0000 7f4b95d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4b98157060 0x7f4b9814cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:08.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.870+0000 7f4b96d9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4b980678b0 0x7f4b9814d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:08.873 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.870+0000 7f4b95d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4b98157060 0x7f4b9814cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43030/0 (socket says 172.21.15.77:43030) 2022-04-03T19:52:08.873 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.870+0000 7f4b95d9b700 1 -- 172.21.15.77:0/1852359656 learned_addr learned my addr 172.21.15.77:0/1852359656 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:52:08.873 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.870+0000 7f4b9659c700 1 --2- 172.21.15.77:0/1852359656 >> v2:172.21.15.186:3300/0 conn(0x7f4b980ba3c0 0x7f4b98156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:08.873 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.870+0000 7f4b96d9d700 1 -- 172.21.15.77:0/1852359656 >> v2:172.21.15.186:3300/0 conn(0x7f4b980ba3c0 msgr2=0x7f4b98156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:52:08.874 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.870+0000 7f4b96d9d700 1 --2- 172.21.15.77:0/1852359656 >> v2:172.21.15.186:3300/0 conn(0x7f4b980ba3c0 0x7f4b98156b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:08.874 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.870+0000 7f4b96d9d700 1 -- 172.21.15.77:0/1852359656 >> v2:172.21.15.77:3300/0 conn(0x7f4b98157060 msgr2=0x7f4b9814cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:52:08.874 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.870+0000 7f4b96d9d700 1 --2- 172.21.15.77:0/1852359656 >> v2:172.21.15.77:3300/0 conn(0x7f4b98157060 0x7f4b9814cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:08.874 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.870+0000 7f4b96d9d700 1 -- 172.21.15.77:0/1852359656 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b980bf350 con 0x7f4b980678b0 2022-04-03T19:52:08.874 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.870+0000 7f4b96d9d700 1 --2- 172.21.15.77:0/1852359656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4b980678b0 0x7f4b9814d040 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4b98068280 tx=0x7f4b90009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e033a7733db2204 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:52:08.875 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.871+0000 7f4b9559a700 1 -- 172.21.15.77:0/1852359656 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9000e070 con 0x7f4b980678b0 2022-04-03T19:52:08.875 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.871+0000 7f4b9559a700 1 -- 172.21.15.77:0/1852359656 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9000bb40 con 0x7f4b980678b0 2022-04-03T19:52:08.875 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.871+0000 7f4b9559a700 1 -- 172.21.15.77:0/1852359656 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b90012430 con 0x7f4b980678b0 2022-04-03T19:52:08.879 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.871+0000 7f4b9759e700 1 -- 172.21.15.77:0/1852359656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4b980678b0 msgr2=0x7f4b9814d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:08.879 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.871+0000 7f4b9759e700 1 --2- 172.21.15.77:0/1852359656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4b980678b0 0x7f4b9814d040 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4b98068280 tx=0x7f4b90009780 comp rx=0 tx=0).stop 2022-04-03T19:52:08.879 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.871+0000 7f4b9759e700 1 -- 172.21.15.77:0/1852359656 shutdown_connections 2022-04-03T19:52:08.879 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.871+0000 7f4b9759e700 1 --2- 172.21.15.77:0/1852359656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4b980678b0 0x7f4b9814d040 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:08.880 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.871+0000 7f4b9759e700 1 --2- 172.21.15.77:0/1852359656 >> v2:172.21.15.186:3300/0 conn(0x7f4b980ba3c0 0x7f4b98156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:08.880 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.871+0000 7f4b9759e700 1 --2- 172.21.15.77:0/1852359656 >> v2:172.21.15.77:3300/0 conn(0x7f4b98157060 0x7f4b9814cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:08.880 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.871+0000 7f4b9759e700 1 -- 172.21.15.77:0/1852359656 >> 172.21.15.77:0/1852359656 conn(0x7f4b98066da0 msgr2=0x7f4b980671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:52:08.880 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.871+0000 7f4b9759e700 1 -- 172.21.15.77:0/1852359656 shutdown_connections 2022-04-03T19:52:08.881 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.872+0000 7f4b9759e700 1 -- 172.21.15.77:0/1852359656 wait complete. 2022-04-03T19:52:08.881 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.873+0000 7f4b9759e700 1 Processor -- start 2022-04-03T19:52:08.881 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.873+0000 7f4b9759e700 1 -- start start 2022-04-03T19:52:08.881 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.874+0000 7f4b9759e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4b980678b0 0x7f4b98167e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:08.881 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.874+0000 7f4b9759e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b980ba3c0 0x7f4b98146e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:08.882 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.874+0000 7f4b9759e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b98157060 0x7f4b980be5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:08.882 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.874+0000 7f4b9759e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9806a560 con 0x7f4b98157060 2022-04-03T19:52:08.882 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.874+0000 7f4b9759e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b98069320 con 0x7f4b980ba3c0 2022-04-03T19:52:08.882 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.874+0000 7f4b9759e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b980b8a10 con 0x7f4b980678b0 2022-04-03T19:52:08.883 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.874+0000 7f4b9659c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4b980678b0 0x7f4b98167e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:08.883 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.874+0000 7f4b95d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b980ba3c0 0x7f4b98146e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:08.883 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.874+0000 7f4b9659c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4b980678b0 0x7f4b98167e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60314/0 (socket says 172.21.15.77:60314) 2022-04-03T19:52:08.883 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.874+0000 7f4b9659c700 1 -- 172.21.15.77:0/2777133882 learned_addr learned my addr 172.21.15.77:0/2777133882 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:52:08.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.875+0000 7f4b96d9d700 1 --2- 172.21.15.77:0/2777133882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b98157060 0x7f4b980be5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:08.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.875+0000 7f4b9659c700 1 -- 172.21.15.77:0/2777133882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b980ba3c0 msgr2=0x7f4b98146e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:08.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.875+0000 7f4b9659c700 1 --2- 172.21.15.77:0/2777133882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b980ba3c0 0x7f4b98146e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:08.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.875+0000 7f4b9659c700 1 -- 172.21.15.77:0/2777133882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b98157060 msgr2=0x7f4b980be5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:08.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.875+0000 7f4b9659c700 1 --2- 172.21.15.77:0/2777133882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b98157060 0x7f4b980be5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:08.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.875+0000 7f4b9659c700 1 -- 172.21.15.77:0/2777133882 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b90009a20 con 0x7f4b980678b0 2022-04-03T19:52:08.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.875+0000 7f4b96d9d700 1 --2- 172.21.15.77:0/2777133882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b98157060 0x7f4b980be5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:52:08.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.875+0000 7f4b9659c700 1 --2- 172.21.15.77:0/2777133882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4b980678b0 0x7f4b98167e20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4b8800e960 tx=0x7f4b88009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:52:08.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.875+0000 7f4b867fc700 1 -- 172.21.15.77:0/2777133882 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b88013680 con 0x7f4b980678b0 2022-04-03T19:52:08.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.875+0000 7f4b867fc700 1 -- 172.21.15.77:0/2777133882 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b880137e0 con 0x7f4b980678b0 2022-04-03T19:52:08.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.875+0000 7f4b867fc700 1 -- 172.21.15.77:0/2777133882 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b88013b60 con 0x7f4b980678b0 2022-04-03T19:52:08.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.875+0000 7f4b9759e700 1 -- 172.21.15.77:0/2777133882 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b980b9da0 con 0x7f4b980678b0 2022-04-03T19:52:08.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.876+0000 7f4b9759e700 1 -- 172.21.15.77:0/2777133882 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b98069850 con 0x7f4b980678b0 2022-04-03T19:52:08.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.876+0000 7f4b9759e700 1 -- 172.21.15.77:0/2777133882 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b980c74a0 con 0x7f4b980678b0 2022-04-03T19:52:08.887 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.877+0000 7f4b867fc700 1 -- 172.21.15.77:0/2777133882 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f4b88019030 con 0x7f4b980678b0 2022-04-03T19:52:08.887 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.878+0000 7f4b867fc700 1 --2- 172.21.15.77:0/2777133882 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4b74063ce0 0x7f4b74066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:08.887 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.878+0000 7f4b867fc700 1 -- 172.21.15.77:0/2777133882 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f4b88080e50 con 0x7f4b980678b0 2022-04-03T19:52:08.887 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.878+0000 7f4b95d9b700 1 --2- 172.21.15.77:0/2777133882 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4b74063ce0 0x7f4b74066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:08.888 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.879+0000 7f4b95d9b700 1 --2- 172.21.15.77:0/2777133882 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4b74063ce0 0x7f4b74066190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4b9806b2f0 tx=0x7f4b8c0090f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:52:08.888 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:08.881+0000 7f4b867fc700 1 -- 172.21.15.77:0/2777133882 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f4b8804f830 con 0x7f4b980678b0 2022-04-03T19:52:09.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:09.112+0000 7f4b9759e700 1 -- 172.21.15.77:0/2777133882 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b980c44c0 con 0x7f4b980678b0 2022-04-03T19:52:09.113 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:09.113+0000 7f4b867fc700 1 -- 172.21.15.77:0/2777133882 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5353 (secure 0 0 0) 0x7f4b88021090 con 0x7f4b980678b0 2022-04-03T19:52:09.114 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:52:09.114 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:52:07.666403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"e","rank":0,"incarnation":27,"state":"up:active","state_seq":28,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:52:09.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:09.116+0000 7f4b9759e700 1 -- 172.21.15.77:0/2777133882 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4b74063ce0 msgr2=0x7f4b74066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:09.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:09.116+0000 7f4b9759e700 1 --2- 172.21.15.77:0/2777133882 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4b74063ce0 0x7f4b74066190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4b9806b2f0 tx=0x7f4b8c0090f0 comp rx=0 tx=0).stop 2022-04-03T19:52:09.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:09.116+0000 7f4b9759e700 1 -- 172.21.15.77:0/2777133882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4b980678b0 msgr2=0x7f4b98167e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:09.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:09.116+0000 7f4b9759e700 1 --2- 172.21.15.77:0/2777133882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4b980678b0 0x7f4b98167e20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4b8800e960 tx=0x7f4b88009060 comp rx=0 tx=0).stop 2022-04-03T19:52:09.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:09.116+0000 7f4b9759e700 1 -- 172.21.15.77:0/2777133882 shutdown_connections 2022-04-03T19:52:09.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:09.116+0000 7f4b9759e700 1 --2- 172.21.15.77:0/2777133882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b98157060 0x7f4b980be5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:09.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:09.116+0000 7f4b9759e700 1 --2- 172.21.15.77:0/2777133882 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4b74063ce0 0x7f4b74066190 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:09.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:09.116+0000 7f4b9759e700 1 --2- 172.21.15.77:0/2777133882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4b980678b0 0x7f4b98167e20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:09.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:09.116+0000 7f4b9759e700 1 --2- 172.21.15.77:0/2777133882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b980ba3c0 0x7f4b98146e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:09.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:09.116+0000 7f4b9759e700 1 -- 172.21.15.77:0/2777133882 >> 172.21.15.77:0/2777133882 conn(0x7f4b98066da0 msgr2=0x7f4b980c07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:52:09.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:09.116+0000 7f4b9759e700 1 -- 172.21.15.77:0/2777133882 shutdown_connections 2022-04-03T19:52:09.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:09.116+0000 7f4b9759e700 1 -- 172.21.15.77:0/2777133882 wait complete. 2022-04-03T19:52:09.120 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 30 2022-04-03T19:52:09.140 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2022-04-03T19:52:11.142 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:52:11.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.348+0000 7f3559fcb700 1 Processor -- start 2022-04-03T19:52:11.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.348+0000 7f3559fcb700 1 -- start start 2022-04-03T19:52:11.351 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.349+0000 7f3559fcb700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3554069d10 0x7f355415c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:11.351 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.349+0000 7f3559fcb700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35540c4b90 con 0x7f35540c3270 2022-04-03T19:52:11.351 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.349+0000 7f3559fcb700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35540c3d10 con 0x7f35540c3640 2022-04-03T19:52:11.352 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.349+0000 7f3559fcb700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35540c3f80 con 0x7f3554069d10 2022-04-03T19:52:11.352 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.350+0000 7f3558fc9700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f35540c3270 0x7f35540c2500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38496/0 (socket says 172.21.15.77:38496) 2022-04-03T19:52:11.352 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.350+0000 7f3558fc9700 1 -- 172.21.15.77:0/1489877917 learned_addr learned my addr 172.21.15.77:0/1489877917 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:52:11.352 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.350+0000 7f35597ca700 1 --2- 172.21.15.77:0/1489877917 >> v2:172.21.15.77:3300/0 conn(0x7f3554069d10 0x7f355415c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:11.353 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.350+0000 7f35597ca700 1 -- 172.21.15.77:0/1489877917 >> v1:172.21.15.186:6789/0 conn(0x7f35540c3640 legacy=0x7f3554069600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:11.353 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.350+0000 7f35597ca700 1 -- 172.21.15.77:0/1489877917 >> v1:172.21.15.77:6789/0 conn(0x7f35540c3270 legacy=0x7f35540c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:11.353 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.350+0000 7f35597ca700 1 -- 172.21.15.77:0/1489877917 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35540c4260 con 0x7f3554069d10 2022-04-03T19:52:11.353 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.351+0000 7f35597ca700 1 --2- 172.21.15.77:0/1489877917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3554069d10 0x7f355415c0a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f354c009e90 tx=0x7f354c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ddf706a191e7596 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:52:11.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.351+0000 7f35537fe700 1 -- 172.21.15.77:0/1489877917 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c00e070 con 0x7f3554069d10 2022-04-03T19:52:11.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.351+0000 7f35537fe700 1 -- 172.21.15.77:0/1489877917 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354c00bb30 con 0x7f3554069d10 2022-04-03T19:52:11.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.351+0000 7f35537fe700 1 -- 172.21.15.77:0/1489877917 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c00bed0 con 0x7f3554069d10 2022-04-03T19:52:11.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.352+0000 7f3559fcb700 1 -- 172.21.15.77:0/1489877917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3554069d10 msgr2=0x7f355415c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:11.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.352+0000 7f3559fcb700 1 --2- 172.21.15.77:0/1489877917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3554069d10 0x7f355415c0a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f354c009e90 tx=0x7f354c009780 comp rx=0 tx=0).stop 2022-04-03T19:52:11.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.352+0000 7f3559fcb700 1 -- 172.21.15.77:0/1489877917 shutdown_connections 2022-04-03T19:52:11.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.352+0000 7f3559fcb700 1 --2- 172.21.15.77:0/1489877917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3554069d10 0x7f355415c0a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:11.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.352+0000 7f3559fcb700 1 -- 172.21.15.77:0/1489877917 >> 172.21.15.77:0/1489877917 conn(0x7f3554066da0 msgr2=0x7f35540671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:52:11.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.352+0000 7f3559fcb700 1 -- 172.21.15.77:0/1489877917 shutdown_connections 2022-04-03T19:52:11.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.352+0000 7f3559fcb700 1 -- 172.21.15.77:0/1489877917 wait complete. 2022-04-03T19:52:11.360 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.353+0000 7f3559fcb700 1 Processor -- start 2022-04-03T19:52:11.360 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.353+0000 7f3559fcb700 1 -- start start 2022-04-03T19:52:11.360 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.354+0000 7f3559fcb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3554069d10 0x7f3554155b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:11.361 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.354+0000 7f3559fcb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35540c3270 0x7f3554154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:11.361 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.354+0000 7f3559fcb700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35540c3640 0x7f355406c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:11.361 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.354+0000 7f3559fcb700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35540bbad0 con 0x7f3554069d10 2022-04-03T19:52:11.361 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.354+0000 7f3559fcb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35540bbf60 con 0x7f35540c3270 2022-04-03T19:52:11.361 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.354+0000 7f3559fcb700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3554155400 con 0x7f35540c3640 2022-04-03T19:52:11.362 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.355+0000 7f3558fc9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3554069d10 0x7f3554155b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:11.362 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.355+0000 7f3558fc9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3554069d10 0x7f3554155b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43036/0 (socket says 172.21.15.77:43036) 2022-04-03T19:52:11.362 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.355+0000 7f3558fc9700 1 -- 172.21.15.77:0/4187586145 learned_addr learned my addr 172.21.15.77:0/4187586145 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:52:11.362 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.355+0000 7f35597ca700 1 --2- 172.21.15.77:0/4187586145 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35540c3640 0x7f355406c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:11.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.355+0000 7f3553fff700 1 --2- 172.21.15.77:0/4187586145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35540c3270 0x7f3554154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:11.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.355+0000 7f3558fc9700 1 -- 172.21.15.77:0/4187586145 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35540c3640 msgr2=0x7f355406c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:11.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.355+0000 7f3558fc9700 1 --2- 172.21.15.77:0/4187586145 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35540c3640 0x7f355406c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:11.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.355+0000 7f3558fc9700 1 -- 172.21.15.77:0/4187586145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35540c3270 msgr2=0x7f3554154ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:11.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.355+0000 7f3558fc9700 1 --2- 172.21.15.77:0/4187586145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35540c3270 0x7f3554154ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:11.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.355+0000 7f3558fc9700 1 -- 172.21.15.77:0/4187586145 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354c009a20 con 0x7f3554069d10 2022-04-03T19:52:11.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.355+0000 7f3558fc9700 1 --2- 172.21.15.77:0/4187586145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3554069d10 0x7f3554155b10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f354800c810 tx=0x7f3548007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:52:11.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.356+0000 7f3550ff9700 1 -- 172.21.15.77:0/4187586145 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3548007360 con 0x7f3554069d10 2022-04-03T19:52:11.364 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.356+0000 7f3550ff9700 1 -- 172.21.15.77:0/4187586145 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35480074e0 con 0x7f3554069d10 2022-04-03T19:52:11.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.356+0000 7f3550ff9700 1 -- 172.21.15.77:0/4187586145 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3548004900 con 0x7f3554069d10 2022-04-03T19:52:11.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.356+0000 7f3559fcb700 1 -- 172.21.15.77:0/4187586145 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3554153f10 con 0x7f3554069d10 2022-04-03T19:52:11.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.356+0000 7f3559fcb700 1 -- 172.21.15.77:0/4187586145 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35540bb1a0 con 0x7f3554069d10 2022-04-03T19:52:11.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.357+0000 7f3559fcb700 1 -- 172.21.15.77:0/4187586145 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35540bbdb0 con 0x7f3554069d10 2022-04-03T19:52:11.366 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.358+0000 7f3550ff9700 1 -- 172.21.15.77:0/4187586145 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f3548010030 con 0x7f3554069d10 2022-04-03T19:52:11.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.359+0000 7f3550ff9700 1 --2- 172.21.15.77:0/4187586145 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f353c063ce0 0x7f353c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:52:11.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.359+0000 7f3550ff9700 1 -- 172.21.15.77:0/4187586145 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3548080550 con 0x7f3554069d10 2022-04-03T19:52:11.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.359+0000 7f3553fff700 1 --2- 172.21.15.77:0/4187586145 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f353c063ce0 0x7f353c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:52:11.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.360+0000 7f3553fff700 1 --2- 172.21.15.77:0/4187586145 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f353c063ce0 0x7f353c066190 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f355406a680 tx=0x7f3540009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:52:11.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.363+0000 7f3550ff9700 1 -- 172.21.15.77:0/4187586145 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f354804efa0 con 0x7f3554069d10 2022-04-03T19:52:11.592 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.592+0000 7f3559fcb700 1 -- 172.21.15.77:0/4187586145 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f355404b050 con 0x7f3554069d10 2022-04-03T19:52:11.593 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.592+0000 7f3550ff9700 1 -- 172.21.15.77:0/4187586145 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f35480525c0 con 0x7f3554069d10 2022-04-03T19:52:11.593 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:52:11.594 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5039,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:52:07.666403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"e","rank":0,"incarnation":27,"state":"up:active","state_seq":28,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:52:11.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.595+0000 7f3559fcb700 1 -- 172.21.15.77:0/4187586145 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f353c063ce0 msgr2=0x7f353c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:11.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.595+0000 7f3559fcb700 1 --2- 172.21.15.77:0/4187586145 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f353c063ce0 0x7f353c066190 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f355406a680 tx=0x7f3540009040 comp rx=0 tx=0).stop 2022-04-03T19:52:11.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.595+0000 7f3559fcb700 1 -- 172.21.15.77:0/4187586145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3554069d10 msgr2=0x7f3554155b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:52:11.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.595+0000 7f3559fcb700 1 --2- 172.21.15.77:0/4187586145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3554069d10 0x7f3554155b10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f354800c810 tx=0x7f3548007db0 comp rx=0 tx=0).stop 2022-04-03T19:52:11.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.595+0000 7f3559fcb700 1 -- 172.21.15.77:0/4187586145 shutdown_connections 2022-04-03T19:52:11.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.595+0000 7f3559fcb700 1 --2- 172.21.15.77:0/4187586145 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f353c063ce0 0x7f353c066190 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:11.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.595+0000 7f3559fcb700 1 --2- 172.21.15.77:0/4187586145 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35540c3640 0x7f355406c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:11.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.595+0000 7f3559fcb700 1 --2- 172.21.15.77:0/4187586145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3554069d10 0x7f3554155b10 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:11.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.595+0000 7f3559fcb700 1 --2- 172.21.15.77:0/4187586145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35540c3270 0x7f3554154ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:52:11.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.595+0000 7f3559fcb700 1 -- 172.21.15.77:0/4187586145 >> 172.21.15.77:0/4187586145 conn(0x7f3554066da0 msgr2=0x7f35540c0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:52:11.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.595+0000 7f3559fcb700 1 -- 172.21.15.77:0/4187586145 shutdown_connections 2022-04-03T19:52:11.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:52:11.595+0000 7f3559fcb700 1 -- 172.21.15.77:0/4187586145 wait complete. 2022-04-03T19:52:11.599 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 31 2022-04-03T19:52:11.620 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2022-04-03T19:52:11.620 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 54 secs before thrashing 2022-04-03T19:52:14.654 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T19:52:14.654+0000 7fcf8a06f700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-04-03T19:52:16.458 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:52:16.464 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:52:46.583 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:52:46.589 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:53:05.621 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:53:05.821 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.820+0000 7f94d89af700 1 Processor -- start 2022-04-03T19:53:05.821 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.820+0000 7f94d89af700 1 -- start start 2022-04-03T19:53:05.824 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.822+0000 7f94d89af700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f94d40c1170 0x7f94d41527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:05.824 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.822+0000 7f94d89af700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94d4155370 0x7f94d4069110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:05.825 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.822+0000 7f94d89af700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d40c3710 con 0x7f94d40c3b10 2022-04-03T19:53:05.825 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.822+0000 7f94d89af700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94d4152330 con 0x7f94d40c1170 2022-04-03T19:53:05.825 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.822+0000 7f94d89af700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d40c1540 con 0x7f94d4155370 2022-04-03T19:53:05.825 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.822+0000 7f94d37fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f94d40c1170 0x7f94d41527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:05.826 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.822+0000 7f94d2ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94d4155370 0x7f94d4069110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:05.826 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.822+0000 7f94d37fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f94d40c1170 0x7f94d41527c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38228/0 (socket says 172.21.15.77:38228) 2022-04-03T19:53:05.826 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.822+0000 7f94d37fe700 1 -- 172.21.15.77:0/191228218 learned_addr learned my addr 172.21.15.77:0/191228218 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:05.826 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.822+0000 7f94d3fff700 1 --1- 172.21.15.77:0/191228218 >> v1:172.21.15.186:6789/0 conn(0x7f94d40c3b10 0x7f94d40c71f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44308/0 (socket says 172.21.15.77:44308) 2022-04-03T19:53:05.826 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.823+0000 7f94d37fe700 1 -- 172.21.15.77:0/191228218 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94d4155370 msgr2=0x7f94d4069110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:53:05.827 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.823+0000 7f94d37fe700 1 --2- 172.21.15.77:0/191228218 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94d4155370 0x7f94d4069110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:05.827 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.823+0000 7f94d37fe700 1 -- 172.21.15.77:0/191228218 >> v1:172.21.15.186:6789/0 conn(0x7f94d40c3b10 legacy=0x7f94d40c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:05.827 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.823+0000 7f94d37fe700 1 -- 172.21.15.77:0/191228218 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d40c4270 con 0x7f94d40c1170 2022-04-03T19:53:05.827 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.823+0000 7f94d37fe700 1 --2- 172.21.15.77:0/191228218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94d40c1170 0x7f94d41527c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f94c4006fd0 tx=0x7f94c40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88b147cbbafe3b14 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:05.828 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.823+0000 7f94d27fc700 1 -- 172.21.15.77:0/191228218 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4004030 con 0x7f94d40c1170 2022-04-03T19:53:05.828 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.823+0000 7f94d27fc700 1 -- 172.21.15.77:0/191228218 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c40041d0 con 0x7f94d40c1170 2022-04-03T19:53:05.828 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.823+0000 7f94d27fc700 1 -- 172.21.15.77:0/191228218 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4011630 con 0x7f94d40c1170 2022-04-03T19:53:05.828 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.824+0000 7f94d89af700 1 -- 172.21.15.77:0/191228218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94d40c1170 msgr2=0x7f94d41527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:05.828 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.824+0000 7f94d89af700 1 --2- 172.21.15.77:0/191228218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94d40c1170 0x7f94d41527c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f94c4006fd0 tx=0x7f94c40090f0 comp rx=0 tx=0).stop 2022-04-03T19:53:05.829 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.824+0000 7f94d89af700 1 -- 172.21.15.77:0/191228218 shutdown_connections 2022-04-03T19:53:05.829 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.824+0000 7f94d89af700 1 --2- 172.21.15.77:0/191228218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94d40c1170 0x7f94d41527c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:05.829 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.824+0000 7f94d89af700 1 --2- 172.21.15.77:0/191228218 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94d4155370 0x7f94d4069110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:05.829 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.824+0000 7f94d89af700 1 -- 172.21.15.77:0/191228218 >> 172.21.15.77:0/191228218 conn(0x7f94d4066da0 msgr2=0x7f94d40671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:05.829 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.824+0000 7f94d89af700 1 -- 172.21.15.77:0/191228218 shutdown_connections 2022-04-03T19:53:05.830 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.824+0000 7f94d89af700 1 -- 172.21.15.77:0/191228218 wait complete. 2022-04-03T19:53:05.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.825+0000 7f94d89af700 1 Processor -- start 2022-04-03T19:53:05.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.825+0000 7f94d89af700 1 -- start start 2022-04-03T19:53:05.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.826+0000 7f94d89af700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94d40c1170 0x7f94d4164d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:05.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.826+0000 7f94d89af700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d40c3b10 0x7f94d4154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:05.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.826+0000 7f94d89af700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94d4155370 0x7f94d414aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:05.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.826+0000 7f94d89af700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d41652a0 con 0x7f94d40c3b10 2022-04-03T19:53:05.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.826+0000 7f94d89af700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d4160f40 con 0x7f94d40c1170 2022-04-03T19:53:05.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.826+0000 7f94d89af700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d41611e0 con 0x7f94d4155370 2022-04-03T19:53:05.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.826+0000 7f94d37fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94d40c1170 0x7f94d4164d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:05.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.827+0000 7f94d3fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94d4155370 0x7f94d414aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:05.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.827+0000 7f94d37fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94d40c1170 0x7f94d4164d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38230/0 (socket says 172.21.15.77:38230) 2022-04-03T19:53:05.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.827+0000 7f94d37fe700 1 -- 172.21.15.77:0/167816124 learned_addr learned my addr 172.21.15.77:0/167816124 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:05.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.827+0000 7f94d2ffd700 1 --2- 172.21.15.77:0/167816124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d40c3b10 0x7f94d4154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:05.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.827+0000 7f94d37fe700 1 -- 172.21.15.77:0/167816124 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94d4155370 msgr2=0x7f94d414aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:05.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.827+0000 7f94d37fe700 1 --2- 172.21.15.77:0/167816124 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94d4155370 0x7f94d414aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:05.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.827+0000 7f94d37fe700 1 -- 172.21.15.77:0/167816124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d40c3b10 msgr2=0x7f94d4154850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:05.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.827+0000 7f94d37fe700 1 --2- 172.21.15.77:0/167816124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d40c3b10 0x7f94d4154850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:05.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.827+0000 7f94d37fe700 1 -- 172.21.15.77:0/167816124 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c4009370 con 0x7f94d40c1170 2022-04-03T19:53:05.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.827+0000 7f94d37fe700 1 --2- 172.21.15.77:0/167816124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94d40c1170 0x7f94d4164d60 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f94c4004000 tx=0x7f94c4016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:05.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.828+0000 7f94bb7fe700 1 -- 172.21.15.77:0/167816124 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c40036f0 con 0x7f94d40c1170 2022-04-03T19:53:05.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.828+0000 7f94bb7fe700 1 -- 172.21.15.77:0/167816124 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c4003870 con 0x7f94d40c1170 2022-04-03T19:53:05.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.828+0000 7f94d89af700 1 -- 172.21.15.77:0/167816124 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d41614c0 con 0x7f94d40c1170 2022-04-03T19:53:05.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.828+0000 7f94bb7fe700 1 -- 172.21.15.77:0/167816124 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4003c30 con 0x7f94d40c1170 2022-04-03T19:53:05.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.828+0000 7f94d89af700 1 -- 172.21.15.77:0/167816124 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d414b4c0 con 0x7f94d40c1170 2022-04-03T19:53:05.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.829+0000 7f94d89af700 1 -- 172.21.15.77:0/167816124 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d4161cc0 con 0x7f94d40c1170 2022-04-03T19:53:05.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.830+0000 7f94bb7fe700 1 -- 172.21.15.77:0/167816124 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f94c4017020 con 0x7f94d40c1170 2022-04-03T19:53:05.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.830+0000 7f94bb7fe700 1 --2- 172.21.15.77:0/167816124 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f94b4063ce0 0x7f94b4066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:05.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.830+0000 7f94bb7fe700 1 -- 172.21.15.77:0/167816124 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f94c40823a0 con 0x7f94d40c1170 2022-04-03T19:53:05.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.831+0000 7f94d2ffd700 1 --2- 172.21.15.77:0/167816124 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f94b4063ce0 0x7f94b4066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:05.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.831+0000 7f94d2ffd700 1 --2- 172.21.15.77:0/167816124 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f94b4063ce0 0x7f94b4066190 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f94d40bab70 tx=0x7f94c8009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:05.840 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:05.834+0000 7f94bb7fe700 1 -- 172.21.15.77:0/167816124 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f94c4050e10 con 0x7f94d40c1170 2022-04-03T19:53:06.067 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.067+0000 7f94d89af700 1 -- 172.21.15.77:0/167816124 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94d404b050 con 0x7f94d40c1170 2022-04-03T19:53:06.068 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.068+0000 7f94bb7fe700 1 -- 172.21.15.77:0/167816124 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f94c4054430 con 0x7f94d40c1170 2022-04-03T19:53:06.068 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:53:06.069 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5039,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:52:07.666403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"e","rank":0,"incarnation":27,"state":"up:active","state_seq":28,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:53:06.071 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.070+0000 7f94d89af700 1 -- 172.21.15.77:0/167816124 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f94b4063ce0 msgr2=0x7f94b4066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:06.071 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.070+0000 7f94d89af700 1 --2- 172.21.15.77:0/167816124 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f94b4063ce0 0x7f94b4066190 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f94d40bab70 tx=0x7f94c8009060 comp rx=0 tx=0).stop 2022-04-03T19:53:06.072 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.070+0000 7f94d89af700 1 -- 172.21.15.77:0/167816124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94d40c1170 msgr2=0x7f94d4164d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:06.072 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.070+0000 7f94d89af700 1 --2- 172.21.15.77:0/167816124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94d40c1170 0x7f94d4164d60 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f94c4004000 tx=0x7f94c4016000 comp rx=0 tx=0).stop 2022-04-03T19:53:06.072 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.070+0000 7f94d89af700 1 -- 172.21.15.77:0/167816124 shutdown_connections 2022-04-03T19:53:06.072 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.070+0000 7f94d89af700 1 --2- 172.21.15.77:0/167816124 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f94b4063ce0 0x7f94b4066190 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:06.072 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.070+0000 7f94d89af700 1 --2- 172.21.15.77:0/167816124 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94d4155370 0x7f94d414aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:06.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.070+0000 7f94d89af700 1 --2- 172.21.15.77:0/167816124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94d40c1170 0x7f94d4164d60 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:06.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.070+0000 7f94d89af700 1 --2- 172.21.15.77:0/167816124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94d40c3b10 0x7f94d4154850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:06.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.070+0000 7f94d89af700 1 -- 172.21.15.77:0/167816124 >> 172.21.15.77:0/167816124 conn(0x7f94d4066da0 msgr2=0x7f94d414c000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:06.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.070+0000 7f94d89af700 1 -- 172.21.15.77:0/167816124 shutdown_connections 2022-04-03T19:53:06.074 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.070+0000 7f94d89af700 1 -- 172.21.15.77:0/167816124 wait complete. 2022-04-03T19:53:06.074 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 31 2022-04-03T19:53:06.095 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2022-04-03T19:53:06.096 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2022-04-03T19:53:06.096 INFO:teuthology.orchestra.run:waiting for 300 2022-04-03T19:53:06.145 INFO:tasks.ceph.mds.e:Stopped 2022-04-03T19:53:06.145 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:53:06.343 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.343+0000 7f02e359e700 1 Processor -- start 2022-04-03T19:53:06.344 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.343+0000 7f02e359e700 1 -- start start 2022-04-03T19:53:06.345 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.344+0000 7f02e359e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f02e414f150 0x7f02e40c5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:06.345 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.344+0000 7f02e359e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f02e414f520 0x7f02e4158e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:06.345 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.344+0000 7f02e359e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e406de50 con 0x7f02e40c5f10 2022-04-03T19:53:06.346 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.344+0000 7f02e359e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02e40c5430 con 0x7f02e414f520 2022-04-03T19:53:06.346 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.344+0000 7f02e359e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02e41593a0 con 0x7f02e414f150 2022-04-03T19:53:06.346 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.344+0000 7f02e1d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f02e414f520 0x7f02e4158e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:06.346 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.345+0000 7f02e1d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f02e414f520 0x7f02e4158e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43040/0 (socket says 172.21.15.77:43040) 2022-04-03T19:53:06.346 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.345+0000 7f02e1d9b700 1 -- 172.21.15.77:0/1905638551 learned_addr learned my addr 172.21.15.77:0/1905638551 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:06.347 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.345+0000 7f02e259c700 1 --2- 172.21.15.77:0/1905638551 >> v2:172.21.15.186:3300/0 conn(0x7f02e414f150 0x7f02e40c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:06.348 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.345+0000 7f02e1d9b700 1 -- 172.21.15.77:0/1905638551 >> v2:172.21.15.186:3300/0 conn(0x7f02e414f150 msgr2=0x7f02e40c5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:53:06.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.345+0000 7f02e1d9b700 1 --2- 172.21.15.77:0/1905638551 >> v2:172.21.15.186:3300/0 conn(0x7f02e414f150 0x7f02e40c5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:06.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.345+0000 7f02e1d9b700 1 -- 172.21.15.77:0/1905638551 >> v1:172.21.15.186:6789/0 conn(0x7f02e40c5f10 legacy=0x7f02e406bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:06.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.345+0000 7f02e1d9b700 1 -- 172.21.15.77:0/1905638551 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e406e2e0 con 0x7f02e414f520 2022-04-03T19:53:06.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.345+0000 7f02e1d9b700 1 --2- 172.21.15.77:0/1905638551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02e414f520 0x7f02e4158e60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f02e4089940 tx=0x7f02d80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef0b78bbf81ce1ae server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:06.350 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.346+0000 7f02e159a700 1 -- 172.21.15.77:0/1905638551 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d8004030 con 0x7f02e414f520 2022-04-03T19:53:06.350 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.346+0000 7f02e159a700 1 -- 172.21.15.77:0/1905638551 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d8004190 con 0x7f02e414f520 2022-04-03T19:53:06.350 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.346+0000 7f02e159a700 1 -- 172.21.15.77:0/1905638551 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d8011630 con 0x7f02e414f520 2022-04-03T19:53:06.350 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.346+0000 7f02e359e700 1 -- 172.21.15.77:0/1905638551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02e414f520 msgr2=0x7f02e4158e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:06.350 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.346+0000 7f02e359e700 1 --2- 172.21.15.77:0/1905638551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02e414f520 0x7f02e4158e60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f02e4089940 tx=0x7f02d80090f0 comp rx=0 tx=0).stop 2022-04-03T19:53:06.351 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.346+0000 7f02e359e700 1 -- 172.21.15.77:0/1905638551 shutdown_connections 2022-04-03T19:53:06.351 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.346+0000 7f02e359e700 1 --2- 172.21.15.77:0/1905638551 >> v2:172.21.15.186:3300/0 conn(0x7f02e414f150 0x7f02e40c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:06.351 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.346+0000 7f02e359e700 1 --2- 172.21.15.77:0/1905638551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02e414f520 0x7f02e4158e60 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:06.351 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.346+0000 7f02e359e700 1 -- 172.21.15.77:0/1905638551 >> 172.21.15.77:0/1905638551 conn(0x7f02e40cb640 msgr2=0x7f02e40cba40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:06.351 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.346+0000 7f02e359e700 1 -- 172.21.15.77:0/1905638551 shutdown_connections 2022-04-03T19:53:06.352 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.346+0000 7f02e359e700 1 -- 172.21.15.77:0/1905638551 wait complete. 2022-04-03T19:53:06.352 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.348+0000 7f02e359e700 1 Processor -- start 2022-04-03T19:53:06.352 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.348+0000 7f02e359e700 1 -- start start 2022-04-03T19:53:06.355 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.349+0000 7f02e359e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02e40c5f10 0x7f02e41479e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:06.355 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.349+0000 7f02e359e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02e414f150 0x7f02e4147f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:06.355 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.349+0000 7f02e359e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02e414c840 0x7f02e4154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:06.355 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.349+0000 7f02e359e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e41531c0 con 0x7f02e414c840 2022-04-03T19:53:06.356 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.349+0000 7f02e359e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e4157b60 con 0x7f02e40c5f10 2022-04-03T19:53:06.356 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.349+0000 7f02e359e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e4157e40 con 0x7f02e414f150 2022-04-03T19:53:06.356 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.349+0000 7f02e259c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02e40c5f10 0x7f02e41479e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:06.356 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.349+0000 7f02e2d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02e414c840 0x7f02e4154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:06.356 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.349+0000 7f02e259c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02e40c5f10 0x7f02e41479e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38234/0 (socket says 172.21.15.77:38234) 2022-04-03T19:53:06.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.349+0000 7f02e259c700 1 -- 172.21.15.77:0/56680704 learned_addr learned my addr 172.21.15.77:0/56680704 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:06.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.349+0000 7f02e1d9b700 1 --2- 172.21.15.77:0/56680704 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02e414f150 0x7f02e4147f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:06.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.350+0000 7f02e259c700 1 -- 172.21.15.77:0/56680704 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02e414f150 msgr2=0x7f02e4147f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:06.357 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.350+0000 7f02e259c700 1 --2- 172.21.15.77:0/56680704 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02e414f150 0x7f02e4147f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:06.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.350+0000 7f02e259c700 1 -- 172.21.15.77:0/56680704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02e414c840 msgr2=0x7f02e4154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:06.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.350+0000 7f02e259c700 1 --2- 172.21.15.77:0/56680704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02e414c840 0x7f02e4154060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:06.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.350+0000 7f02e259c700 1 -- 172.21.15.77:0/56680704 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d8009370 con 0x7f02e40c5f10 2022-04-03T19:53:06.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.350+0000 7f02e259c700 1 --2- 172.21.15.77:0/56680704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02e40c5f10 0x7f02e41479e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f02d400e930 tx=0x7f02d4009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:06.358 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.350+0000 7f02d27fc700 1 -- 172.21.15.77:0/56680704 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d4013680 con 0x7f02e40c5f10 2022-04-03T19:53:06.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.350+0000 7f02d27fc700 1 -- 172.21.15.77:0/56680704 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d40137e0 con 0x7f02e40c5f10 2022-04-03T19:53:06.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.350+0000 7f02e359e700 1 -- 172.21.15.77:0/56680704 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e4158120 con 0x7f02e40c5f10 2022-04-03T19:53:06.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.350+0000 7f02d27fc700 1 -- 172.21.15.77:0/56680704 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d4013b60 con 0x7f02e40c5f10 2022-04-03T19:53:06.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.350+0000 7f02e359e700 1 -- 172.21.15.77:0/56680704 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e41586e0 con 0x7f02e40c5f10 2022-04-03T19:53:06.359 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.351+0000 7f02e359e700 1 -- 172.21.15.77:0/56680704 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e404b050 con 0x7f02e40c5f10 2022-04-03T19:53:06.360 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.352+0000 7f02d27fc700 1 -- 172.21.15.77:0/56680704 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f02d4019030 con 0x7f02e40c5f10 2022-04-03T19:53:06.360 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.353+0000 7f02d27fc700 1 --2- 172.21.15.77:0/56680704 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f02c0063d10 0x7f02c00661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:06.360 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.353+0000 7f02d27fc700 1 -- 172.21.15.77:0/56680704 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f02d40809b0 con 0x7f02e40c5f10 2022-04-03T19:53:06.360 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.353+0000 7f02e1d9b700 1 --2- 172.21.15.77:0/56680704 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f02c0063d10 0x7f02c00661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:06.361 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.354+0000 7f02e1d9b700 1 --2- 172.21.15.77:0/56680704 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f02c0063d10 0x7f02c00661c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f02e4089940 tx=0x7f02d8016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:06.361 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.357+0000 7f02d27fc700 1 -- 172.21.15.77:0/56680704 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f02d404f400 con 0x7f02e40c5f10 2022-04-03T19:53:06.589 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.589+0000 7f02e359e700 1 -- 172.21.15.77:0/56680704 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02e406c810 con 0x7f02e40c5f10 2022-04-03T19:53:06.590 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.590+0000 7f02d27fc700 1 -- 172.21.15.77:0/56680704 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f02d4052a20 con 0x7f02e40c5f10 2022-04-03T19:53:06.591 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:53:06.591 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5039,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:52:07.666403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"e","rank":0,"incarnation":27,"state":"up:active","state_seq":28,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:53:06.593 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.592+0000 7f02e359e700 1 -- 172.21.15.77:0/56680704 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f02c0063d10 msgr2=0x7f02c00661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:06.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.592+0000 7f02e359e700 1 --2- 172.21.15.77:0/56680704 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f02c0063d10 0x7f02c00661c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f02e4089940 tx=0x7f02d8016000 comp rx=0 tx=0).stop 2022-04-03T19:53:06.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.592+0000 7f02e359e700 1 -- 172.21.15.77:0/56680704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02e40c5f10 msgr2=0x7f02e41479e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:06.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.592+0000 7f02e359e700 1 --2- 172.21.15.77:0/56680704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02e40c5f10 0x7f02e41479e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f02d400e930 tx=0x7f02d4009060 comp rx=0 tx=0).stop 2022-04-03T19:53:06.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.593+0000 7f02e359e700 1 -- 172.21.15.77:0/56680704 shutdown_connections 2022-04-03T19:53:06.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.593+0000 7f02e359e700 1 --2- 172.21.15.77:0/56680704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02e414c840 0x7f02e4154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:06.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.593+0000 7f02e359e700 1 --2- 172.21.15.77:0/56680704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02e40c5f10 0x7f02e41479e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:06.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.593+0000 7f02e359e700 1 --2- 172.21.15.77:0/56680704 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f02c0063d10 0x7f02c00661c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:06.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.593+0000 7f02e359e700 1 --2- 172.21.15.77:0/56680704 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02e414f150 0x7f02e4147f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:06.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.593+0000 7f02e359e700 1 -- 172.21.15.77:0/56680704 >> 172.21.15.77:0/56680704 conn(0x7f02e40cb640 msgr2=0x7f02e414de20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:06.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.593+0000 7f02e359e700 1 -- 172.21.15.77:0/56680704 shutdown_connections 2022-04-03T19:53:06.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:06.593+0000 7f02e359e700 1 -- 172.21.15.77:0/56680704 wait complete. 2022-04-03T19:53:06.596 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 31 2022-04-03T19:53:06.617 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:53:08.618 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:53:08.828 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.827+0000 7f57b759e700 1 Processor -- start 2022-04-03T19:53:08.828 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.827+0000 7f57b759e700 1 -- start start 2022-04-03T19:53:08.829 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.829+0000 7f57b759e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57b80c7c10 0x7f57b8151d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:08.830 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.829+0000 7f57b759e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b8069e20 con 0x7f57b80678b0 2022-04-03T19:53:08.830 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.829+0000 7f57b759e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b80ba620 con 0x7f57b81549d0 2022-04-03T19:53:08.830 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.829+0000 7f57b759e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b80ba900 con 0x7f57b80c7c10 2022-04-03T19:53:08.831 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.829+0000 7f57b5d9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f57b80678b0 0x7f57b80c7500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38498/0 (socket says 172.21.15.77:38498) 2022-04-03T19:53:08.831 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.829+0000 7f57b5d9b700 1 -- 172.21.15.77:0/2452215085 learned_addr learned my addr 172.21.15.77:0/2452215085 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:53:08.831 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.829+0000 7f57b6d9d700 1 --2- 172.21.15.77:0/2452215085 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57b80c7c10 0x7f57b8151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:08.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.830+0000 7f57b559a700 1 -- 172.21.15.77:0/2452215085 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267692905 0 0) 0x7f57b80ba620 con 0x7f57b81549d0 2022-04-03T19:53:08.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.830+0000 7f57b559a700 1 -- 172.21.15.77:0/2452215085 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f579c0033f0 con 0x7f57b81549d0 2022-04-03T19:53:08.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.830+0000 7f57b559a700 1 -- 172.21.15.77:0/2452215085 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590149826 0 0) 0x7f57b8069e20 con 0x7f57b80678b0 2022-04-03T19:53:08.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.830+0000 7f57b559a700 1 -- 172.21.15.77:0/2452215085 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57b80ba620 con 0x7f57b80678b0 2022-04-03T19:53:08.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.830+0000 7f57b6d9d700 1 -- 172.21.15.77:0/2452215085 >> v1:172.21.15.186:6789/0 conn(0x7f57b81549d0 legacy=0x7f57b8154da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:08.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.830+0000 7f57b6d9d700 1 -- 172.21.15.77:0/2452215085 >> v1:172.21.15.77:6789/0 conn(0x7f57b80678b0 legacy=0x7f57b80c7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:08.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.830+0000 7f57b6d9d700 1 -- 172.21.15.77:0/2452215085 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b80babe0 con 0x7f57b80c7c10 2022-04-03T19:53:08.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.830+0000 7f57b6d9d700 1 --2- 172.21.15.77:0/2452215085 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57b80c7c10 0x7f57b8151d00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009e90 tx=0x7f57b0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87a74dd5d33316e7 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:08.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.830+0000 7f57b559a700 1 -- 172.21.15.77:0/2452215085 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b000e070 con 0x7f57b80c7c10 2022-04-03T19:53:08.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.830+0000 7f57b559a700 1 -- 172.21.15.77:0/2452215085 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b000bb80 con 0x7f57b80c7c10 2022-04-03T19:53:08.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.830+0000 7f57b559a700 1 -- 172.21.15.77:0/2452215085 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b0012430 con 0x7f57b80c7c10 2022-04-03T19:53:08.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.831+0000 7f57b759e700 1 -- 172.21.15.77:0/2452215085 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57b80c7c10 msgr2=0x7f57b8151d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:08.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.831+0000 7f57b759e700 1 --2- 172.21.15.77:0/2452215085 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57b80c7c10 0x7f57b8151d00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009e90 tx=0x7f57b0009780 comp rx=0 tx=0).stop 2022-04-03T19:53:08.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.831+0000 7f57b759e700 1 -- 172.21.15.77:0/2452215085 shutdown_connections 2022-04-03T19:53:08.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.831+0000 7f57b759e700 1 --2- 172.21.15.77:0/2452215085 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57b80c7c10 0x7f57b8151d00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:08.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.831+0000 7f57b759e700 1 -- 172.21.15.77:0/2452215085 >> 172.21.15.77:0/2452215085 conn(0x7f57b8066da0 msgr2=0x7f57b80671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:08.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.831+0000 7f57b759e700 1 -- 172.21.15.77:0/2452215085 shutdown_connections 2022-04-03T19:53:08.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.831+0000 7f57b759e700 1 -- 172.21.15.77:0/2452215085 wait complete. 2022-04-03T19:53:08.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.833+0000 7f57b759e700 1 Processor -- start 2022-04-03T19:53:08.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.833+0000 7f57b759e700 1 -- start start 2022-04-03T19:53:08.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.834+0000 7f57b759e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57b80678b0 0x7f57b80bb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:08.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.834+0000 7f57b759e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57b80c7c10 0x7f57b8150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:08.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.834+0000 7f57b759e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57b81549d0 0x7f57b8152430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:08.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.834+0000 7f57b759e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b8161190 con 0x7f57b80678b0 2022-04-03T19:53:08.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.834+0000 7f57b759e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b80bb690 con 0x7f57b80c7c10 2022-04-03T19:53:08.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.834+0000 7f57b759e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b8160860 con 0x7f57b81549d0 2022-04-03T19:53:08.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.834+0000 7f57b659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57b80678b0 0x7f57b80bb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:08.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.834+0000 7f57b6d9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57b81549d0 0x7f57b8152430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:08.843 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.834+0000 7f57b5d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57b80c7c10 0x7f57b8150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:08.843 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.834+0000 7f57b659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57b80678b0 0x7f57b80bb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43044/0 (socket says 172.21.15.77:43044) 2022-04-03T19:53:08.843 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.834+0000 7f57b659c700 1 -- 172.21.15.77:0/4117062033 learned_addr learned my addr 172.21.15.77:0/4117062033 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:08.843 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.834+0000 7f57b659c700 1 -- 172.21.15.77:0/4117062033 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57b81549d0 msgr2=0x7f57b8152430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:08.843 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.834+0000 7f57b659c700 1 --2- 172.21.15.77:0/4117062033 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57b81549d0 0x7f57b8152430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:08.844 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.834+0000 7f57b659c700 1 -- 172.21.15.77:0/4117062033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57b80c7c10 msgr2=0x7f57b8150800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:08.844 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.834+0000 7f57b659c700 1 --2- 172.21.15.77:0/4117062033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57b80c7c10 0x7f57b8150800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:08.844 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.835+0000 7f57b659c700 1 -- 172.21.15.77:0/4117062033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b0009a20 con 0x7f57b80678b0 2022-04-03T19:53:08.844 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.835+0000 7f57b659c700 1 --2- 172.21.15.77:0/4117062033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57b80678b0 0x7f57b80bb000 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f57a800c4a0 tx=0x7f57a8007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:08.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.835+0000 7f57a67fc700 1 -- 172.21.15.77:0/4117062033 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a800ce50 con 0x7f57b80678b0 2022-04-03T19:53:08.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.835+0000 7f57a67fc700 1 -- 172.21.15.77:0/4117062033 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a80050d0 con 0x7f57b80678b0 2022-04-03T19:53:08.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.835+0000 7f57a67fc700 1 -- 172.21.15.77:0/4117062033 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a8011590 con 0x7f57b80678b0 2022-04-03T19:53:08.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.835+0000 7f57b759e700 1 -- 172.21.15.77:0/4117062033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b8160b40 con 0x7f57b80678b0 2022-04-03T19:53:08.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.835+0000 7f57b759e700 1 -- 172.21.15.77:0/4117062033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b8160510 con 0x7f57b80678b0 2022-04-03T19:53:08.846 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.836+0000 7f57b759e700 1 -- 172.21.15.77:0/4117062033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b804b050 con 0x7f57b80678b0 2022-04-03T19:53:08.846 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.837+0000 7f57a67fc700 1 -- 172.21.15.77:0/4117062033 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f57a8019030 con 0x7f57b80678b0 2022-04-03T19:53:08.846 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.838+0000 7f57a67fc700 1 --2- 172.21.15.77:0/4117062033 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5794063ce0 0x7f5794066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:08.846 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.838+0000 7f57a67fc700 1 -- 172.21.15.77:0/4117062033 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f57a807fb20 con 0x7f57b80678b0 2022-04-03T19:53:08.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.838+0000 7f57b5d9b700 1 --2- 172.21.15.77:0/4117062033 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5794063ce0 0x7f5794066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:08.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.839+0000 7f57b5d9b700 1 --2- 172.21.15.77:0/4117062033 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5794063ce0 0x7f5794066190 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f57b814b000 tx=0x7f57ac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:08.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:08.842+0000 7f57a67fc700 1 -- 172.21.15.77:0/4117062033 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f57a804e570 con 0x7f57b80678b0 2022-04-03T19:53:09.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:09.081+0000 7f57b759e700 1 -- 172.21.15.77:0/4117062033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57b806d5e0 con 0x7f57b80678b0 2022-04-03T19:53:09.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:09.082+0000 7f57a67fc700 1 -- 172.21.15.77:0/4117062033 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f57a8017020 con 0x7f57b80678b0 2022-04-03T19:53:09.083 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:53:09.083 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5039,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:52:07.666403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"e","rank":0,"incarnation":27,"state":"up:active","state_seq":28,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:53:09.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:09.084+0000 7f57b759e700 1 -- 172.21.15.77:0/4117062033 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5794063ce0 msgr2=0x7f5794066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:09.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:09.084+0000 7f57b759e700 1 --2- 172.21.15.77:0/4117062033 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5794063ce0 0x7f5794066190 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f57b814b000 tx=0x7f57ac009040 comp rx=0 tx=0).stop 2022-04-03T19:53:09.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:09.084+0000 7f57b759e700 1 -- 172.21.15.77:0/4117062033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57b80678b0 msgr2=0x7f57b80bb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:09.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:09.084+0000 7f57b759e700 1 --2- 172.21.15.77:0/4117062033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57b80678b0 0x7f57b80bb000 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f57a800c4a0 tx=0x7f57a8007c50 comp rx=0 tx=0).stop 2022-04-03T19:53:09.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:09.084+0000 7f57b759e700 1 -- 172.21.15.77:0/4117062033 shutdown_connections 2022-04-03T19:53:09.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:09.084+0000 7f57b759e700 1 --2- 172.21.15.77:0/4117062033 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5794063ce0 0x7f5794066190 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:09.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:09.084+0000 7f57b759e700 1 --2- 172.21.15.77:0/4117062033 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57b81549d0 0x7f57b8152430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:09.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:09.084+0000 7f57b759e700 1 --2- 172.21.15.77:0/4117062033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57b80678b0 0x7f57b80bb000 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:09.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:09.084+0000 7f57b759e700 1 --2- 172.21.15.77:0/4117062033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57b80c7c10 0x7f57b8150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:09.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:09.084+0000 7f57b759e700 1 -- 172.21.15.77:0/4117062033 >> 172.21.15.77:0/4117062033 conn(0x7f57b8066da0 msgr2=0x7f57b80c1900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:09.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:09.084+0000 7f57b759e700 1 -- 172.21.15.77:0/4117062033 shutdown_connections 2022-04-03T19:53:09.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:09.084+0000 7f57b759e700 1 -- 172.21.15.77:0/4117062033 wait complete. 2022-04-03T19:53:09.088 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 31 2022-04-03T19:53:09.110 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:53:11.111 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:53:11.319 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.318+0000 7f1a409ec700 1 Processor -- start 2022-04-03T19:53:11.319 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.319+0000 7f1a409ec700 1 -- start start 2022-04-03T19:53:11.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.320+0000 7f1a409ec700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a3c155370 0x7f1a3c069110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:11.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.320+0000 7f1a409ec700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1a3c0c3b10 0x7f1a3c069650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:11.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.320+0000 7f1a409ec700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a3c0c3710 con 0x7f1a3c0c1170 2022-04-03T19:53:11.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.320+0000 7f1a409ec700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a3c152330 con 0x7f1a3c0c3b10 2022-04-03T19:53:11.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.320+0000 7f1a409ec700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c0c1540 con 0x7f1a3c155370 2022-04-03T19:53:11.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.320+0000 7f1a3b7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1a3c0c1170 0x7f1a3c1527c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38500/0 (socket says 172.21.15.77:38500) 2022-04-03T19:53:11.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.320+0000 7f1a3b7fe700 1 -- 172.21.15.77:0/4012489422 learned_addr learned my addr 172.21.15.77:0/4012489422 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:53:11.323 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.320+0000 7f1a3affd700 1 --2- 172.21.15.77:0/4012489422 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a3c155370 0x7f1a3c069110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:11.323 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.320+0000 7f1a3bfff700 1 --2- 172.21.15.77:0/4012489422 >> v2:172.21.15.186:3300/0 conn(0x7f1a3c0c3b10 0x7f1a3c069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:11.323 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.321+0000 7f1a3a7fc700 1 -- 172.21.15.77:0/4012489422 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182235216 0 0) 0x7f1a3c0c3710 con 0x7f1a3c0c1170 2022-04-03T19:53:11.324 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.321+0000 7f1a3a7fc700 1 -- 172.21.15.77:0/4012489422 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a280033f0 con 0x7f1a3c0c1170 2022-04-03T19:53:11.324 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.321+0000 7f1a3affd700 1 -- 172.21.15.77:0/4012489422 >> v2:172.21.15.186:3300/0 conn(0x7f1a3c0c3b10 msgr2=0x7f1a3c069650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:53:11.324 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.321+0000 7f1a3affd700 1 --2- 172.21.15.77:0/4012489422 >> v2:172.21.15.186:3300/0 conn(0x7f1a3c0c3b10 0x7f1a3c069650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:11.324 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.321+0000 7f1a3affd700 1 -- 172.21.15.77:0/4012489422 >> v1:172.21.15.77:6789/0 conn(0x7f1a3c0c1170 legacy=0x7f1a3c1527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:11.325 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.321+0000 7f1a3affd700 1 -- 172.21.15.77:0/4012489422 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3c0c4270 con 0x7f1a3c155370 2022-04-03T19:53:11.328 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.321+0000 7f1a3affd700 1 --2- 172.21.15.77:0/4012489422 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a3c155370 0x7f1a3c069110 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1a30009760 tx=0x7f1a30009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96cf432a274a9006 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:11.329 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.322+0000 7f1a3a7fc700 1 -- 172.21.15.77:0/4012489422 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a300105b0 con 0x7f1a3c155370 2022-04-03T19:53:11.329 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.322+0000 7f1a3a7fc700 1 -- 172.21.15.77:0/4012489422 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a30010710 con 0x7f1a3c155370 2022-04-03T19:53:11.329 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.322+0000 7f1a3a7fc700 1 -- 172.21.15.77:0/4012489422 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a30010ab0 con 0x7f1a3c155370 2022-04-03T19:53:11.329 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.322+0000 7f1a409ec700 1 -- 172.21.15.77:0/4012489422 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a3c155370 msgr2=0x7f1a3c069110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:11.330 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.322+0000 7f1a409ec700 1 --2- 172.21.15.77:0/4012489422 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a3c155370 0x7f1a3c069110 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1a30009760 tx=0x7f1a30009060 comp rx=0 tx=0).stop 2022-04-03T19:53:11.330 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.322+0000 7f1a409ec700 1 -- 172.21.15.77:0/4012489422 shutdown_connections 2022-04-03T19:53:11.330 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.322+0000 7f1a409ec700 1 --2- 172.21.15.77:0/4012489422 >> v2:172.21.15.186:3300/0 conn(0x7f1a3c0c3b10 0x7f1a3c069650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:11.330 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.322+0000 7f1a409ec700 1 --2- 172.21.15.77:0/4012489422 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a3c155370 0x7f1a3c069110 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:11.331 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.322+0000 7f1a409ec700 1 -- 172.21.15.77:0/4012489422 >> 172.21.15.77:0/4012489422 conn(0x7f1a3c066da0 msgr2=0x7f1a3c0671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:11.331 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.322+0000 7f1a409ec700 1 -- 172.21.15.77:0/4012489422 shutdown_connections 2022-04-03T19:53:11.331 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.322+0000 7f1a409ec700 1 -- 172.21.15.77:0/4012489422 wait complete. 2022-04-03T19:53:11.331 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.324+0000 7f1a409ec700 1 Processor -- start 2022-04-03T19:53:11.332 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.324+0000 7f1a409ec700 1 -- start start 2022-04-03T19:53:11.332 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.325+0000 7f1a409ec700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a3c0c1170 0x7f1a3c151490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:11.332 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.325+0000 7f1a409ec700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a3c0c3b10 0x7f1a3c0b9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:11.332 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.325+0000 7f1a409ec700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a3c155370 0x7f1a3c14d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:11.333 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.325+0000 7f1a409ec700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c14d520 con 0x7f1a3c0c3b10 2022-04-03T19:53:11.333 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.325+0000 7f1a409ec700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c14c690 con 0x7f1a3c155370 2022-04-03T19:53:11.333 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.325+0000 7f1a409ec700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c153380 con 0x7f1a3c0c1170 2022-04-03T19:53:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.325+0000 7f1a3affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a3c0c3b10 0x7f1a3c0b9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.325+0000 7f1a3bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a3c155370 0x7f1a3c14d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.325+0000 7f1a3b7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a3c0c1170 0x7f1a3c151490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:11.334 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.325+0000 7f1a3bfff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a3c155370 0x7f1a3c14d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38240/0 (socket says 172.21.15.77:38240) 2022-04-03T19:53:11.335 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.325+0000 7f1a3bfff700 1 -- 172.21.15.77:0/3156006548 learned_addr learned my addr 172.21.15.77:0/3156006548 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:11.335 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.325+0000 7f1a3bfff700 1 -- 172.21.15.77:0/3156006548 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a3c0c1170 msgr2=0x7f1a3c151490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:11.335 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.325+0000 7f1a3bfff700 1 --2- 172.21.15.77:0/3156006548 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a3c0c1170 0x7f1a3c151490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:11.336 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.325+0000 7f1a3bfff700 1 -- 172.21.15.77:0/3156006548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a3c0c3b10 msgr2=0x7f1a3c0b9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:11.336 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.325+0000 7f1a3bfff700 1 --2- 172.21.15.77:0/3156006548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a3c0c3b10 0x7f1a3c0b9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:11.336 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.325+0000 7f1a3bfff700 1 -- 172.21.15.77:0/3156006548 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a300092a0 con 0x7f1a3c155370 2022-04-03T19:53:11.336 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.326+0000 7f1a3bfff700 1 --2- 172.21.15.77:0/3156006548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a3c155370 0x7f1a3c14d940 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009e90 tx=0x7f1a3400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:11.337 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.326+0000 7f1a237fe700 1 -- 172.21.15.77:0/3156006548 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3400cb70 con 0x7f1a3c155370 2022-04-03T19:53:11.337 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.326+0000 7f1a237fe700 1 -- 172.21.15.77:0/3156006548 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a3400f040 con 0x7f1a3c155370 2022-04-03T19:53:11.337 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.326+0000 7f1a237fe700 1 -- 172.21.15.77:0/3156006548 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a34014480 con 0x7f1a3c155370 2022-04-03T19:53:11.337 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.326+0000 7f1a409ec700 1 -- 172.21.15.77:0/3156006548 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a3c14c8a0 con 0x7f1a3c155370 2022-04-03T19:53:11.338 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.326+0000 7f1a409ec700 1 -- 172.21.15.77:0/3156006548 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a3c14ce60 con 0x7f1a3c155370 2022-04-03T19:53:11.338 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.327+0000 7f1a409ec700 1 -- 172.21.15.77:0/3156006548 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a3c14d660 con 0x7f1a3c155370 2022-04-03T19:53:11.338 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.328+0000 7f1a237fe700 1 -- 172.21.15.77:0/3156006548 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f1a340147f0 con 0x7f1a3c155370 2022-04-03T19:53:11.339 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.329+0000 7f1a237fe700 1 --2- 172.21.15.77:0/3156006548 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a1c063ce0 0x7f1a1c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:11.339 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.329+0000 7f1a237fe700 1 -- 172.21.15.77:0/3156006548 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1a34081a00 con 0x7f1a3c155370 2022-04-03T19:53:11.340 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.329+0000 7f1a3b7fe700 1 --2- 172.21.15.77:0/3156006548 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a1c063ce0 0x7f1a1c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:11.340 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.330+0000 7f1a3b7fe700 1 --2- 172.21.15.77:0/3156006548 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a1c063ce0 0x7f1a1c066190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c005750 tx=0x7f1a2c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:11.340 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.333+0000 7f1a237fe700 1 -- 172.21.15.77:0/3156006548 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f1a3c14d660 con 0x7f1a3c155370 2022-04-03T19:53:11.564 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.564+0000 7f1a409ec700 1 -- 172.21.15.77:0/3156006548 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a3c14d660 con 0x7f1a3c155370 2022-04-03T19:53:11.565 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.565+0000 7f1a237fe700 1 -- 172.21.15.77:0/3156006548 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f1a3c14d660 con 0x7f1a3c155370 2022-04-03T19:53:11.566 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:53:11.566 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5039,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:52:07.666403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"e","rank":0,"incarnation":27,"state":"up:active","state_seq":28,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:53:11.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.567+0000 7f1a409ec700 1 -- 172.21.15.77:0/3156006548 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a1c063ce0 msgr2=0x7f1a1c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:11.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.567+0000 7f1a409ec700 1 --2- 172.21.15.77:0/3156006548 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a1c063ce0 0x7f1a1c066190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c005750 tx=0x7f1a2c009040 comp rx=0 tx=0).stop 2022-04-03T19:53:11.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.568+0000 7f1a409ec700 1 -- 172.21.15.77:0/3156006548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a3c155370 msgr2=0x7f1a3c14d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:11.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.568+0000 7f1a409ec700 1 --2- 172.21.15.77:0/3156006548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a3c155370 0x7f1a3c14d940 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009e90 tx=0x7f1a3400b040 comp rx=0 tx=0).stop 2022-04-03T19:53:11.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.568+0000 7f1a409ec700 1 -- 172.21.15.77:0/3156006548 shutdown_connections 2022-04-03T19:53:11.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.568+0000 7f1a409ec700 1 --2- 172.21.15.77:0/3156006548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a3c155370 0x7f1a3c14d940 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:11.570 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.568+0000 7f1a409ec700 1 --2- 172.21.15.77:0/3156006548 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a1c063ce0 0x7f1a1c066190 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:11.570 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.568+0000 7f1a409ec700 1 --2- 172.21.15.77:0/3156006548 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a3c0c1170 0x7f1a3c151490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:11.570 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.568+0000 7f1a409ec700 1 --2- 172.21.15.77:0/3156006548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a3c0c3b10 0x7f1a3c0b9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:11.570 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.568+0000 7f1a409ec700 1 -- 172.21.15.77:0/3156006548 >> 172.21.15.77:0/3156006548 conn(0x7f1a3c066da0 msgr2=0x7f1a3c06c300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:11.571 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.568+0000 7f1a409ec700 1 -- 172.21.15.77:0/3156006548 shutdown_connections 2022-04-03T19:53:11.571 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:11.568+0000 7f1a409ec700 1 -- 172.21.15.77:0/3156006548 wait complete. 2022-04-03T19:53:11.571 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 31 2022-04-03T19:53:11.592 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:53:13.593 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:53:13.799 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.799+0000 7f1a7d396700 1 Processor -- start 2022-04-03T19:53:13.800 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.799+0000 7f1a7d396700 1 -- start start 2022-04-03T19:53:13.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.800+0000 7f1a7d396700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a7806bbe0 0x7f1a781584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:13.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.800+0000 7f1a7d396700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1a7806c500 0x7f1a7814eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:13.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.800+0000 7f1a7d396700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7806d260 con 0x7f1a7806bfb0 2022-04-03T19:53:13.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.800+0000 7f1a7d396700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a78159450 con 0x7f1a7806c500 2022-04-03T19:53:13.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.800+0000 7f1a7d396700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a780c6330 con 0x7f1a7806bbe0 2022-04-03T19:53:13.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.801+0000 7f1a777fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1a7806bfb0 0x7f1a7806d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38502/0 (socket says 172.21.15.77:38502) 2022-04-03T19:53:13.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.801+0000 7f1a77fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a7806bbe0 0x7f1a781584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:13.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.801+0000 7f1a777fe700 1 -- 172.21.15.77:0/1022421483 learned_addr learned my addr 172.21.15.77:0/1022421483 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:53:13.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.801+0000 7f1a7cb95700 1 --2- 172.21.15.77:0/1022421483 >> v2:172.21.15.186:3300/0 conn(0x7f1a7806c500 0x7f1a7814eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:13.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.801+0000 7f1a76ffd700 1 -- 172.21.15.77:0/1022421483 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222519130 0 0) 0x7f1a7806d260 con 0x7f1a7806bfb0 2022-04-03T19:53:13.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.801+0000 7f1a76ffd700 1 -- 172.21.15.77:0/1022421483 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a640033f0 con 0x7f1a7806bfb0 2022-04-03T19:53:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.801+0000 7f1a7cb95700 1 -- 172.21.15.77:0/1022421483 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a7806bbe0 msgr2=0x7f1a781584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:53:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.801+0000 7f1a7cb95700 1 --2- 172.21.15.77:0/1022421483 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a7806bbe0 0x7f1a781584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.801+0000 7f1a7cb95700 1 -- 172.21.15.77:0/1022421483 >> v1:172.21.15.77:6789/0 conn(0x7f1a7806bfb0 legacy=0x7f1a7806d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:13.805 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.801+0000 7f1a7cb95700 1 -- 172.21.15.77:0/1022421483 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a78158bb0 con 0x7f1a7806c500 2022-04-03T19:53:13.806 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.801+0000 7f1a77fff700 1 --2- 172.21.15.77:0/1022421483 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a7806bbe0 0x7f1a781584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:53:13.806 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.802+0000 7f1a7cb95700 1 --2- 172.21.15.77:0/1022421483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a7806c500 0x7f1a7814eea0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1a70009e60 tx=0x7f1a70009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c43ac7af4d2f9f5b server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:13.806 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.802+0000 7f1a76ffd700 1 -- 172.21.15.77:0/1022421483 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7000e070 con 0x7f1a7806c500 2022-04-03T19:53:13.806 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.802+0000 7f1a76ffd700 1 -- 172.21.15.77:0/1022421483 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a7000bab0 con 0x7f1a7806c500 2022-04-03T19:53:13.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.802+0000 7f1a76ffd700 1 -- 172.21.15.77:0/1022421483 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7000be30 con 0x7f1a7806c500 2022-04-03T19:53:13.810 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.802+0000 7f1a7d396700 1 -- 172.21.15.77:0/1022421483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a7806c500 msgr2=0x7f1a7814eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:13.810 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.802+0000 7f1a7d396700 1 --2- 172.21.15.77:0/1022421483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a7806c500 0x7f1a7814eea0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1a70009e60 tx=0x7f1a70009780 comp rx=0 tx=0).stop 2022-04-03T19:53:13.810 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.803+0000 7f1a7d396700 1 -- 172.21.15.77:0/1022421483 shutdown_connections 2022-04-03T19:53:13.811 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.803+0000 7f1a7d396700 1 --2- 172.21.15.77:0/1022421483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a7806c500 0x7f1a7814eea0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:13.811 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.803+0000 7f1a7d396700 1 --2- 172.21.15.77:0/1022421483 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a7806bbe0 0x7f1a781584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:13.811 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.803+0000 7f1a7d396700 1 -- 172.21.15.77:0/1022421483 >> 172.21.15.77:0/1022421483 conn(0x7f1a780cb6b0 msgr2=0x7f1a780cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:13.811 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.803+0000 7f1a7d396700 1 -- 172.21.15.77:0/1022421483 shutdown_connections 2022-04-03T19:53:13.811 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.803+0000 7f1a7d396700 1 -- 172.21.15.77:0/1022421483 wait complete. 2022-04-03T19:53:13.812 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.804+0000 7f1a7d396700 1 Processor -- start 2022-04-03T19:53:13.812 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.804+0000 7f1a7d396700 1 -- start start 2022-04-03T19:53:13.812 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.805+0000 7f1a7d396700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a7806bbe0 0x7f1a78154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:13.812 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.805+0000 7f1a7d396700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a7806bfb0 0x7f1a781682d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:13.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.805+0000 7f1a7d396700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a7806c500 0x7f1a78148bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:13.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.805+0000 7f1a7d396700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a781546d0 con 0x7f1a7806bbe0 2022-04-03T19:53:13.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.805+0000 7f1a7d396700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a78153d00 con 0x7f1a7806bfb0 2022-04-03T19:53:13.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.805+0000 7f1a7d396700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7814a350 con 0x7f1a7806c500 2022-04-03T19:53:13.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.805+0000 7f1a77fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a7806bbe0 0x7f1a78154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:13.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.806+0000 7f1a777fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a7806bfb0 0x7f1a781682d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:13.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.806+0000 7f1a7cb95700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a7806c500 0x7f1a78148bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:13.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.806+0000 7f1a77fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a7806bbe0 0x7f1a78154190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43048/0 (socket says 172.21.15.77:43048) 2022-04-03T19:53:13.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.806+0000 7f1a77fff700 1 -- 172.21.15.77:0/1224340663 learned_addr learned my addr 172.21.15.77:0/1224340663 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:13.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.806+0000 7f1a777fe700 1 -- 172.21.15.77:0/1224340663 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a7806c500 msgr2=0x7f1a78148bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:13.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.806+0000 7f1a777fe700 1 --2- 172.21.15.77:0/1224340663 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a7806c500 0x7f1a78148bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:13.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.806+0000 7f1a777fe700 1 -- 172.21.15.77:0/1224340663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a7806bbe0 msgr2=0x7f1a78154190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:13.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.806+0000 7f1a777fe700 1 --2- 172.21.15.77:0/1224340663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a7806bbe0 0x7f1a78154190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:13.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.806+0000 7f1a777fe700 1 -- 172.21.15.77:0/1224340663 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a680092a0 con 0x7f1a7806bfb0 2022-04-03T19:53:13.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.806+0000 7f1a7cb95700 1 --2- 172.21.15.77:0/1224340663 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a7806c500 0x7f1a78148bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:53:13.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.806+0000 7f1a77fff700 1 --2- 172.21.15.77:0/1224340663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a7806bbe0 0x7f1a78154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:53:13.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.806+0000 7f1a777fe700 1 --2- 172.21.15.77:0/1224340663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a7806bfb0 0x7f1a781682d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c00c410 tx=0x7f1a6c007b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:13.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.807+0000 7f1a5ffff700 1 -- 172.21.15.77:0/1224340663 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c0070d0 con 0x7f1a7806bfb0 2022-04-03T19:53:13.817 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.807+0000 7f1a5ffff700 1 -- 172.21.15.77:0/1224340663 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a6c007230 con 0x7f1a7806bfb0 2022-04-03T19:53:13.817 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.807+0000 7f1a7d396700 1 -- 172.21.15.77:0/1224340663 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a70009a20 con 0x7f1a7806bfb0 2022-04-03T19:53:13.817 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.807+0000 7f1a5ffff700 1 -- 172.21.15.77:0/1224340663 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c0075f0 con 0x7f1a7806bfb0 2022-04-03T19:53:13.817 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.807+0000 7f1a7d396700 1 -- 172.21.15.77:0/1224340663 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a781536f0 con 0x7f1a7806bfb0 2022-04-03T19:53:13.817 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.808+0000 7f1a7d396700 1 -- 172.21.15.77:0/1224340663 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a781584a0 con 0x7f1a7806bfb0 2022-04-03T19:53:13.818 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.809+0000 7f1a5ffff700 1 -- 172.21.15.77:0/1224340663 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f1a6c019030 con 0x7f1a7806bfb0 2022-04-03T19:53:13.818 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.809+0000 7f1a5ffff700 1 --2- 172.21.15.77:0/1224340663 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a58063d10 0x7f1a580661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:13.818 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.810+0000 7f1a5ffff700 1 -- 172.21.15.77:0/1224340663 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1a6c07ff30 con 0x7f1a7806bfb0 2022-04-03T19:53:13.818 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.810+0000 7f1a77fff700 1 --2- 172.21.15.77:0/1224340663 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a58063d10 0x7f1a580661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:13.819 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.810+0000 7f1a77fff700 1 --2- 172.21.15.77:0/1224340663 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a58063d10 0x7f1a580661c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1a68011580 tx=0x7f1a68009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:13.819 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:13.813+0000 7f1a5ffff700 1 -- 172.21.15.77:0/1224340663 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f1a6c0a8ae0 con 0x7f1a7806bfb0 2022-04-03T19:53:14.048 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:14.047+0000 7f1a7d396700 1 -- 172.21.15.77:0/1224340663 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a780c7ec0 con 0x7f1a7806bfb0 2022-04-03T19:53:14.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:14.049+0000 7f1a5ffff700 1 -- 172.21.15.77:0/1224340663 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f1a6c017030 con 0x7f1a7806bfb0 2022-04-03T19:53:14.049 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:53:14.050 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5039,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:52:07.666403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"e","rank":0,"incarnation":27,"state":"up:active","state_seq":28,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:53:14.051 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:14.051+0000 7f1a7d396700 1 -- 172.21.15.77:0/1224340663 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a58063d10 msgr2=0x7f1a580661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:14.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:14.051+0000 7f1a7d396700 1 --2- 172.21.15.77:0/1224340663 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a58063d10 0x7f1a580661c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1a68011580 tx=0x7f1a68009060 comp rx=0 tx=0).stop 2022-04-03T19:53:14.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:14.051+0000 7f1a7d396700 1 -- 172.21.15.77:0/1224340663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a7806bfb0 msgr2=0x7f1a781682d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:14.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:14.051+0000 7f1a7d396700 1 --2- 172.21.15.77:0/1224340663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a7806bfb0 0x7f1a781682d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c00c410 tx=0x7f1a6c007b70 comp rx=0 tx=0).stop 2022-04-03T19:53:14.052 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:14.051+0000 7f1a7d396700 1 -- 172.21.15.77:0/1224340663 shutdown_connections 2022-04-03T19:53:14.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:14.051+0000 7f1a7d396700 1 --2- 172.21.15.77:0/1224340663 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a58063d10 0x7f1a580661c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:14.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:14.051+0000 7f1a7d396700 1 --2- 172.21.15.77:0/1224340663 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a7806c500 0x7f1a78148bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:14.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:14.051+0000 7f1a7d396700 1 --2- 172.21.15.77:0/1224340663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a7806bbe0 0x7f1a78154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:14.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:14.051+0000 7f1a7d396700 1 --2- 172.21.15.77:0/1224340663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a7806bfb0 0x7f1a781682d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:14.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:14.051+0000 7f1a7d396700 1 -- 172.21.15.77:0/1224340663 >> 172.21.15.77:0/1224340663 conn(0x7f1a780cb6b0 msgr2=0x7f1a7814b550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:14.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:14.051+0000 7f1a7d396700 1 -- 172.21.15.77:0/1224340663 shutdown_connections 2022-04-03T19:53:14.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:14.051+0000 7f1a7d396700 1 -- 172.21.15.77:0/1224340663 wait complete. 2022-04-03T19:53:14.055 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 31 2022-04-03T19:53:14.076 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:53:16.078 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:53:16.302 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.301+0000 7f2efb59e700 1 Processor -- start 2022-04-03T19:53:16.303 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.301+0000 7f2efb59e700 1 -- start start 2022-04-03T19:53:16.305 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.303+0000 7f2efb59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2efc0c3270 0x7f2efc0c2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:16.306 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.303+0000 7f2efb59e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2efc069d10 0x7f2efc151760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:16.306 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.303+0000 7f2efb59e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2efc0c4b90 con 0x7f2efc0c3640 2022-04-03T19:53:16.306 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.303+0000 7f2efb59e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2efc0c3d10 con 0x7f2efc069d10 2022-04-03T19:53:16.306 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.303+0000 7f2efb59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc0c2a40 con 0x7f2efc0c3270 2022-04-03T19:53:16.307 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.303+0000 7f2efa59c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2efc0c3270 0x7f2efc0c2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:16.307 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.303+0000 7f2efad9d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2efc069d10 0x7f2efc151760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:16.307 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.303+0000 7f2efa59c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2efc0c3270 0x7f2efc0c2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60336/0 (socket says 172.21.15.77:60336) 2022-04-03T19:53:16.307 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.303+0000 7f2ef9d9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f2efc0c3640 0x7f2efc069600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38504/0 (socket says 172.21.15.77:38504) 2022-04-03T19:53:16.308 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.303+0000 7f2efa59c700 1 -- 172.21.15.77:0/1240252853 learned_addr learned my addr 172.21.15.77:0/1240252853 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:16.308 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.303+0000 7f2efad9d700 1 -- 172.21.15.77:0/1240252853 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2efc0c3270 msgr2=0x7f2efc0c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:53:16.308 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.304+0000 7f2efad9d700 1 --2- 172.21.15.77:0/1240252853 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2efc0c3270 0x7f2efc0c2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:16.308 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.304+0000 7f2efad9d700 1 -- 172.21.15.77:0/1240252853 >> v1:172.21.15.77:6789/0 conn(0x7f2efc0c3640 legacy=0x7f2efc069600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:16.309 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.304+0000 7f2ef959a700 1 -- 172.21.15.77:0/1240252853 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776160206 0 0) 0x7f2efc0c4b90 con 0x7f2efc0c3640 2022-04-03T19:53:16.309 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.304+0000 7f2efad9d700 1 -- 172.21.15.77:0/1240252853 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc0c3f80 con 0x7f2efc069d10 2022-04-03T19:53:16.309 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.304+0000 7f2efad9d700 1 --2- 172.21.15.77:0/1240252853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2efc069d10 0x7f2efc151760 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4009e60 tx=0x7f2ef4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21499071442e391b server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:16.309 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.304+0000 7f2ef959a700 1 -- 172.21.15.77:0/1240252853 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef400e070 con 0x7f2efc069d10 2022-04-03T19:53:16.310 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.304+0000 7f2ef959a700 1 -- 172.21.15.77:0/1240252853 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef400bb60 con 0x7f2efc069d10 2022-04-03T19:53:16.310 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.304+0000 7f2ef959a700 1 -- 172.21.15.77:0/1240252853 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4012430 con 0x7f2efc069d10 2022-04-03T19:53:16.310 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.305+0000 7f2efb59e700 1 -- 172.21.15.77:0/1240252853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2efc069d10 msgr2=0x7f2efc151760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:16.310 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.305+0000 7f2efb59e700 1 --2- 172.21.15.77:0/1240252853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2efc069d10 0x7f2efc151760 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4009e60 tx=0x7f2ef4009780 comp rx=0 tx=0).stop 2022-04-03T19:53:16.311 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.305+0000 7f2efb59e700 1 -- 172.21.15.77:0/1240252853 shutdown_connections 2022-04-03T19:53:16.311 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.305+0000 7f2efb59e700 1 --2- 172.21.15.77:0/1240252853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2efc069d10 0x7f2efc151760 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:16.311 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.305+0000 7f2efb59e700 1 --2- 172.21.15.77:0/1240252853 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2efc0c3270 0x7f2efc0c2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:16.311 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.305+0000 7f2efb59e700 1 -- 172.21.15.77:0/1240252853 >> 172.21.15.77:0/1240252853 conn(0x7f2efc066da0 msgr2=0x7f2efc0671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:16.311 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.305+0000 7f2efb59e700 1 -- 172.21.15.77:0/1240252853 shutdown_connections 2022-04-03T19:53:16.312 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.305+0000 7f2efb59e700 1 -- 172.21.15.77:0/1240252853 wait complete. 2022-04-03T19:53:16.315 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.306+0000 7f2efb59e700 1 Processor -- start 2022-04-03T19:53:16.315 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.307+0000 7f2efb59e700 1 -- start start 2022-04-03T19:53:16.315 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.307+0000 7f2efb59e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2efc069d10 0x7f2efc168c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:16.316 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.307+0000 7f2efb59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2efc0c3270 0x7f2efc14e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:16.316 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.308+0000 7f2efb59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2efc0c3640 0x7f2efc0c1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:16.316 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.308+0000 7f2efb59e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc0bef40 con 0x7f2efc0c3640 2022-04-03T19:53:16.316 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.308+0000 7f2efb59e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc0c0b80 con 0x7f2efc069d10 2022-04-03T19:53:16.317 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.308+0000 7f2efb59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc0c05a0 con 0x7f2efc0c3270 2022-04-03T19:53:16.317 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.308+0000 7f2efa59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2efc069d10 0x7f2efc168c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:16.317 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.308+0000 7f2efa59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2efc069d10 0x7f2efc168c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38246/0 (socket says 172.21.15.77:38246) 2022-04-03T19:53:16.317 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.308+0000 7f2efa59c700 1 -- 172.21.15.77:0/82293641 learned_addr learned my addr 172.21.15.77:0/82293641 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:16.318 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.308+0000 7f2ef9d9b700 1 --2- 172.21.15.77:0/82293641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2efc0c3270 0x7f2efc14e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:16.318 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.308+0000 7f2efad9d700 1 --2- 172.21.15.77:0/82293641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2efc0c3640 0x7f2efc0c1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:16.318 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.308+0000 7f2efad9d700 1 -- 172.21.15.77:0/82293641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2efc0c3270 msgr2=0x7f2efc14e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:16.318 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.308+0000 7f2efad9d700 1 --2- 172.21.15.77:0/82293641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2efc0c3270 0x7f2efc14e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:16.318 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.308+0000 7f2efad9d700 1 -- 172.21.15.77:0/82293641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2efc069d10 msgr2=0x7f2efc168c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:16.319 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.308+0000 7f2efad9d700 1 --2- 172.21.15.77:0/82293641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2efc069d10 0x7f2efc168c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:16.319 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.308+0000 7f2efad9d700 1 -- 172.21.15.77:0/82293641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef4009a20 con 0x7f2efc0c3640 2022-04-03T19:53:16.319 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.308+0000 7f2efa59c700 1 --2- 172.21.15.77:0/82293641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2efc069d10 0x7f2efc168c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:53:16.320 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.309+0000 7f2efad9d700 1 --2- 172.21.15.77:0/82293641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2efc0c3640 0x7f2efc0c1010 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4016000 tx=0x7f2ef4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:16.320 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.309+0000 7f2eea7fc700 1 -- 172.21.15.77:0/82293641 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef40127a0 con 0x7f2efc0c3640 2022-04-03T19:53:16.320 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.309+0000 7f2eea7fc700 1 -- 172.21.15.77:0/82293641 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef4012940 con 0x7f2efc0c3640 2022-04-03T19:53:16.320 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.309+0000 7f2eea7fc700 1 -- 172.21.15.77:0/82293641 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4012d00 con 0x7f2efc0c3640 2022-04-03T19:53:16.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.309+0000 7f2efb59e700 1 -- 172.21.15.77:0/82293641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2efc0bf170 con 0x7f2efc0c3640 2022-04-03T19:53:16.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.309+0000 7f2efb59e700 1 -- 172.21.15.77:0/82293641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2efc0c0810 con 0x7f2efc0c3640 2022-04-03T19:53:16.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.310+0000 7f2efb59e700 1 -- 172.21.15.77:0/82293641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2efc0c0d30 con 0x7f2efc0c3640 2022-04-03T19:53:16.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.311+0000 7f2eea7fc700 1 -- 172.21.15.77:0/82293641 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f2ef4004250 con 0x7f2efc0c3640 2022-04-03T19:53:16.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.312+0000 7f2eea7fc700 1 --2- 172.21.15.77:0/82293641 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2ed8063ce0 0x7f2ed8066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:16.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.312+0000 7f2eea7fc700 1 -- 172.21.15.77:0/82293641 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2ef4083120 con 0x7f2efc0c3640 2022-04-03T19:53:16.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.312+0000 7f2efa59c700 1 --2- 172.21.15.77:0/82293641 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2ed8063ce0 0x7f2ed8066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:16.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.313+0000 7f2efa59c700 1 --2- 172.21.15.77:0/82293641 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2ed8063ce0 0x7f2ed8066190 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00bfd0 tx=0x7f2eec00b3b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:16.323 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.317+0000 7f2eea7fc700 1 -- 172.21.15.77:0/82293641 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f2efc0c0d30 con 0x7f2efc0c3640 2022-04-03T19:53:16.554 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.553+0000 7f2efb59e700 1 -- 172.21.15.77:0/82293641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2efc0c0d30 con 0x7f2efc0c3640 2022-04-03T19:53:16.555 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.554+0000 7f2eea7fc700 1 -- 172.21.15.77:0/82293641 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f2efc0c0d30 con 0x7f2efc0c3640 2022-04-03T19:53:16.555 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:53:16.556 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5039,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:52:07.666403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"e","rank":0,"incarnation":27,"state":"up:active","state_seq":28,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:53:16.559 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.556+0000 7f2efb59e700 1 -- 172.21.15.77:0/82293641 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2ed8063ce0 msgr2=0x7f2ed8066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:16.559 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.556+0000 7f2efb59e700 1 --2- 172.21.15.77:0/82293641 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2ed8063ce0 0x7f2ed8066190 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00bfd0 tx=0x7f2eec00b3b0 comp rx=0 tx=0).stop 2022-04-03T19:53:16.559 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.556+0000 7f2efb59e700 1 -- 172.21.15.77:0/82293641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2efc0c3640 msgr2=0x7f2efc0c1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:16.560 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.556+0000 7f2efb59e700 1 --2- 172.21.15.77:0/82293641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2efc0c3640 0x7f2efc0c1010 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4016000 tx=0x7f2ef4019000 comp rx=0 tx=0).stop 2022-04-03T19:53:16.560 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.556+0000 7f2efb59e700 1 -- 172.21.15.77:0/82293641 shutdown_connections 2022-04-03T19:53:16.560 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.556+0000 7f2efb59e700 1 --2- 172.21.15.77:0/82293641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2efc0c3640 0x7f2efc0c1010 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:16.561 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.556+0000 7f2efb59e700 1 --2- 172.21.15.77:0/82293641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2efc069d10 0x7f2efc168c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:16.561 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.556+0000 7f2efb59e700 1 --2- 172.21.15.77:0/82293641 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2ed8063ce0 0x7f2ed8066190 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:16.561 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.556+0000 7f2efb59e700 1 --2- 172.21.15.77:0/82293641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2efc0c3270 0x7f2efc14e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:16.561 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.556+0000 7f2efb59e700 1 -- 172.21.15.77:0/82293641 >> 172.21.15.77:0/82293641 conn(0x7f2efc066da0 msgr2=0x7f2efc0694c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:16.561 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.556+0000 7f2efb59e700 1 -- 172.21.15.77:0/82293641 shutdown_connections 2022-04-03T19:53:16.562 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:16.557+0000 7f2efb59e700 1 -- 172.21.15.77:0/82293641 wait complete. 2022-04-03T19:53:16.562 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 31 2022-04-03T19:53:16.582 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:53:16.690 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:53:16.696 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:53:18.583 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:53:18.789 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.788+0000 7f30e2b84700 1 Processor -- start 2022-04-03T19:53:18.789 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.788+0000 7f30e2b84700 1 -- start start 2022-04-03T19:53:18.790 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.789+0000 7f30e2b84700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f30dc06d3f0 0x7f30dc06d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:18.790 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.790+0000 7f30e2b84700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30dc1562f0 con 0x7f30dc0c5930 2022-04-03T19:53:18.791 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.790+0000 7f30e2b84700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30dc0c2f80 con 0x7f30dc0c2a90 2022-04-03T19:53:18.791 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.790+0000 7f30e2b84700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30dc0c3260 con 0x7f30dc06d3f0 2022-04-03T19:53:18.791 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.790+0000 7f30e1381700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f30dc0c5930 0x7f30dc155be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38506/0 (socket says 172.21.15.77:38506) 2022-04-03T19:53:18.791 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.790+0000 7f30e1381700 1 -- 172.21.15.77:0/2926196164 learned_addr learned my addr 172.21.15.77:0/2926196164 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:53:18.792 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.790+0000 7f30e1b82700 1 --2- 172.21.15.77:0/2926196164 >> v2:172.21.15.186:3300/0 conn(0x7f30dc06d3f0 0x7f30dc06d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:18.794 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.790+0000 7f30e0b80700 1 -- 172.21.15.77:0/2926196164 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998103727 0 0) 0x7f30dc1562f0 con 0x7f30dc0c5930 2022-04-03T19:53:18.794 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.790+0000 7f30e0b80700 1 -- 172.21.15.77:0/2926196164 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30c40033f0 con 0x7f30dc0c5930 2022-04-03T19:53:18.794 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.790+0000 7f30e0b80700 1 -- 172.21.15.77:0/2926196164 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4243030550 0 0) 0x7f30dc0c2f80 con 0x7f30dc0c2a90 2022-04-03T19:53:18.795 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.790+0000 7f30e0b80700 1 -- 172.21.15.77:0/2926196164 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30dc1562f0 con 0x7f30dc0c2a90 2022-04-03T19:53:18.795 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.790+0000 7f30e1b82700 1 -- 172.21.15.77:0/2926196164 >> v1:172.21.15.186:6789/0 conn(0x7f30dc0c2a90 legacy=0x7f30dc06c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:18.795 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.790+0000 7f30e1b82700 1 -- 172.21.15.77:0/2926196164 >> v1:172.21.15.77:6789/0 conn(0x7f30dc0c5930 legacy=0x7f30dc155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:18.795 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.790+0000 7f30e1b82700 1 -- 172.21.15.77:0/2926196164 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30dc0c3540 con 0x7f30dc06d3f0 2022-04-03T19:53:18.796 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.791+0000 7f30e1b82700 1 --2- 172.21.15.77:0/2926196164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30dc06d3f0 0x7f30dc06d7c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f30d8009790 tx=0x7f30d8009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3fa2f80aca3d628 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:18.796 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.791+0000 7f30e0b80700 1 -- 172.21.15.77:0/2926196164 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d80105b0 con 0x7f30dc06d3f0 2022-04-03T19:53:18.796 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.791+0000 7f30e0b80700 1 -- 172.21.15.77:0/2926196164 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d8010710 con 0x7f30dc06d3f0 2022-04-03T19:53:18.796 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.791+0000 7f30e0b80700 1 -- 172.21.15.77:0/2926196164 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d8010ab0 con 0x7f30dc06d3f0 2022-04-03T19:53:18.796 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.792+0000 7f30e2b84700 1 -- 172.21.15.77:0/2926196164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30dc06d3f0 msgr2=0x7f30dc06d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:18.797 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.792+0000 7f30e2b84700 1 --2- 172.21.15.77:0/2926196164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30dc06d3f0 0x7f30dc06d7c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f30d8009790 tx=0x7f30d8009060 comp rx=0 tx=0).stop 2022-04-03T19:53:18.797 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.792+0000 7f30e2b84700 1 -- 172.21.15.77:0/2926196164 shutdown_connections 2022-04-03T19:53:18.797 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.792+0000 7f30e2b84700 1 --2- 172.21.15.77:0/2926196164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30dc06d3f0 0x7f30dc06d7c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:18.797 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.792+0000 7f30e2b84700 1 -- 172.21.15.77:0/2926196164 >> 172.21.15.77:0/2926196164 conn(0x7f30dc0cb720 msgr2=0x7f30dc0cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:18.798 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.792+0000 7f30e2b84700 1 -- 172.21.15.77:0/2926196164 shutdown_connections 2022-04-03T19:53:18.798 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.792+0000 7f30e2b84700 1 -- 172.21.15.77:0/2926196164 wait complete. 2022-04-03T19:53:18.798 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.793+0000 7f30e2b84700 1 Processor -- start 2022-04-03T19:53:18.798 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.794+0000 7f30e2b84700 1 -- start start 2022-04-03T19:53:18.801 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.794+0000 7f30e2b84700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30dc06d3f0 0x7f30dc14aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:18.801 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.794+0000 7f30e2b84700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30dc0c2a90 0x7f30dc14af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:18.801 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.794+0000 7f30e2b84700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30dc0c5930 0x7f30dc1576f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:18.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.795+0000 7f30e2b84700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30dc0c0a50 con 0x7f30dc0c2a90 2022-04-03T19:53:18.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.795+0000 7f30e2b84700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30dc0bfe80 con 0x7f30dc06d3f0 2022-04-03T19:53:18.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.795+0000 7f30e2b84700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30dc0c2300 con 0x7f30dc0c5930 2022-04-03T19:53:18.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.795+0000 7f30e1381700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30dc0c2a90 0x7f30dc14af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:18.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.795+0000 7f30e2383700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30dc0c5930 0x7f30dc1576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:18.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.795+0000 7f30e1b82700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30dc06d3f0 0x7f30dc14aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:18.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.795+0000 7f30e1381700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30dc0c2a90 0x7f30dc14af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43056/0 (socket says 172.21.15.77:43056) 2022-04-03T19:53:18.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.795+0000 7f30e1381700 1 -- 172.21.15.77:0/2706361354 learned_addr learned my addr 172.21.15.77:0/2706361354 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:18.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.795+0000 7f30e1381700 1 -- 172.21.15.77:0/2706361354 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30dc0c5930 msgr2=0x7f30dc1576f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:18.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.795+0000 7f30e1381700 1 --2- 172.21.15.77:0/2706361354 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30dc0c5930 0x7f30dc1576f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:18.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.795+0000 7f30e1381700 1 -- 172.21.15.77:0/2706361354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30dc06d3f0 msgr2=0x7f30dc14aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:18.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.795+0000 7f30e1381700 1 --2- 172.21.15.77:0/2706361354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30dc06d3f0 0x7f30dc14aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:18.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.795+0000 7f30e1381700 1 -- 172.21.15.77:0/2706361354 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d80092a0 con 0x7f30dc0c2a90 2022-04-03T19:53:18.805 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.796+0000 7f30e1381700 1 --2- 172.21.15.77:0/2706361354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30dc0c2a90 0x7f30dc14af50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f30cc00c420 tx=0x7f30cc007ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:18.805 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.796+0000 7f30c9ffb700 1 -- 172.21.15.77:0/2706361354 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30cc0070d0 con 0x7f30dc0c2a90 2022-04-03T19:53:18.805 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.796+0000 7f30c9ffb700 1 -- 172.21.15.77:0/2706361354 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30cc007230 con 0x7f30dc0c2a90 2022-04-03T19:53:18.805 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.796+0000 7f30c9ffb700 1 -- 172.21.15.77:0/2706361354 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30cc0075b0 con 0x7f30dc0c2a90 2022-04-03T19:53:18.805 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.796+0000 7f30e2b84700 1 -- 172.21.15.77:0/2706361354 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30dc0c0120 con 0x7f30dc0c2a90 2022-04-03T19:53:18.806 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.796+0000 7f30e2b84700 1 -- 172.21.15.77:0/2706361354 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30dc0c06e0 con 0x7f30dc0c2a90 2022-04-03T19:53:18.806 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.797+0000 7f30e2b84700 1 -- 172.21.15.77:0/2706361354 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30dc04b050 con 0x7f30dc0c2a90 2022-04-03T19:53:18.806 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.798+0000 7f30c9ffb700 1 -- 172.21.15.77:0/2706361354 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f30cc019030 con 0x7f30dc0c2a90 2022-04-03T19:53:18.806 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.799+0000 7f30c9ffb700 1 --2- 172.21.15.77:0/2706361354 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f30bc063ce0 0x7f30bc066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:18.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.799+0000 7f30c9ffb700 1 -- 172.21.15.77:0/2706361354 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f30cc07fb90 con 0x7f30dc0c2a90 2022-04-03T19:53:18.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.799+0000 7f30e1b82700 1 --2- 172.21.15.77:0/2706361354 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f30bc063ce0 0x7f30bc066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:18.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.800+0000 7f30e1b82700 1 --2- 172.21.15.77:0/2706361354 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f30bc063ce0 0x7f30bc066190 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f30d8009fd0 tx=0x7f30d8016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:18.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:18.803+0000 7f30c9ffb700 1 -- 172.21.15.77:0/2706361354 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f30cc04e5a0 con 0x7f30dc0c2a90 2022-04-03T19:53:19.039 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:19.039+0000 7f30e2b84700 1 -- 172.21.15.77:0/2706361354 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30dc06d010 con 0x7f30dc0c2a90 2022-04-03T19:53:19.040 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:19.040+0000 7f30c9ffb700 1 -- 172.21.15.77:0/2706361354 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f30cc017070 con 0x7f30dc0c2a90 2022-04-03T19:53:19.041 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:53:19.041 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5039,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:52:07.666403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5239},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5239":{"gid":5239,"name":"e","rank":0,"incarnation":27,"state":"up:active","state_seq":28,"addr":"172.21.15.77:6837/3264384876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":3264384876},{"type":"v1","addr":"172.21.15.77:6837","nonce":3264384876}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:53:19.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:19.042+0000 7f30e2b84700 1 -- 172.21.15.77:0/2706361354 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f30bc063ce0 msgr2=0x7f30bc066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:19.043 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:19.042+0000 7f30e2b84700 1 --2- 172.21.15.77:0/2706361354 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f30bc063ce0 0x7f30bc066190 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f30d8009fd0 tx=0x7f30d8016000 comp rx=0 tx=0).stop 2022-04-03T19:53:19.044 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:19.042+0000 7f30e2b84700 1 -- 172.21.15.77:0/2706361354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30dc0c2a90 msgr2=0x7f30dc14af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:19.044 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:19.042+0000 7f30e2b84700 1 --2- 172.21.15.77:0/2706361354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30dc0c2a90 0x7f30dc14af50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f30cc00c420 tx=0x7f30cc007ca0 comp rx=0 tx=0).stop 2022-04-03T19:53:19.044 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:19.042+0000 7f30e2b84700 1 -- 172.21.15.77:0/2706361354 shutdown_connections 2022-04-03T19:53:19.044 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:19.042+0000 7f30e2b84700 1 --2- 172.21.15.77:0/2706361354 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f30bc063ce0 0x7f30bc066190 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:19.045 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:19.042+0000 7f30e2b84700 1 --2- 172.21.15.77:0/2706361354 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30dc0c5930 0x7f30dc1576f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:19.045 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:19.042+0000 7f30e2b84700 1 --2- 172.21.15.77:0/2706361354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30dc06d3f0 0x7f30dc14aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:19.045 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:19.042+0000 7f30e2b84700 1 --2- 172.21.15.77:0/2706361354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30dc0c2a90 0x7f30dc14af50 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:19.045 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:19.042+0000 7f30e2b84700 1 -- 172.21.15.77:0/2706361354 >> 172.21.15.77:0/2706361354 conn(0x7f30dc0cb720 msgr2=0x7f30dc06fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:19.045 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:19.043+0000 7f30e2b84700 1 -- 172.21.15.77:0/2706361354 shutdown_connections 2022-04-03T19:53:19.046 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:19.043+0000 7f30e2b84700 1 -- 172.21.15.77:0/2706361354 wait complete. 2022-04-03T19:53:19.046 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 31 2022-04-03T19:53:19.067 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:53:21.068 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:53:21.277 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.276+0000 7f8eec8dc700 1 Processor -- start 2022-04-03T19:53:21.277 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.277+0000 7f8eec8dc700 1 -- start start 2022-04-03T19:53:21.279 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.278+0000 7f8eec8dc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8ee806a280 0x7f8ee8159050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:21.280 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.278+0000 7f8eec8dc700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee8155480 con 0x7f8ee80c2010 2022-04-03T19:53:21.280 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.278+0000 7f8eec8dc700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee8156020 con 0x7f8ee80c4dc0 2022-04-03T19:53:21.280 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.278+0000 7f8eec8dc700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ee80c4200 con 0x7f8ee806a280 2022-04-03T19:53:21.280 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.278+0000 7f8ee6d9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8ee80c2010 0x7f8ee8155910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38508/0 (socket says 172.21.15.77:38508) 2022-04-03T19:53:21.281 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.278+0000 7f8ee6d9d700 1 -- 172.21.15.77:0/4136863782 learned_addr learned my addr 172.21.15.77:0/4136863782 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:53:21.281 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.278+0000 7f8ee659c700 1 --2- 172.21.15.77:0/4136863782 >> v2:172.21.15.77:3300/0 conn(0x7f8ee806a280 0x7f8ee8159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:21.281 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.279+0000 7f8ee5d9b700 1 -- 172.21.15.77:0/4136863782 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690534972 0 0) 0x7f8ee8155480 con 0x7f8ee80c2010 2022-04-03T19:53:21.281 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.279+0000 7f8ee5d9b700 1 -- 172.21.15.77:0/4136863782 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ed40033f0 con 0x7f8ee80c2010 2022-04-03T19:53:21.282 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.279+0000 7f8ee5d9b700 1 -- 172.21.15.77:0/4136863782 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748017139 0 0) 0x7f8ee8156020 con 0x7f8ee80c4dc0 2022-04-03T19:53:21.282 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.279+0000 7f8ee5d9b700 1 -- 172.21.15.77:0/4136863782 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee8155480 con 0x7f8ee80c4dc0 2022-04-03T19:53:21.282 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.279+0000 7f8ee659c700 1 -- 172.21.15.77:0/4136863782 >> v1:172.21.15.186:6789/0 conn(0x7f8ee80c4dc0 legacy=0x7f8ee815f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:21.282 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.279+0000 7f8ee659c700 1 -- 172.21.15.77:0/4136863782 >> v1:172.21.15.77:6789/0 conn(0x7f8ee80c2010 legacy=0x7f8ee8155910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:21.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.279+0000 7f8ee659c700 1 -- 172.21.15.77:0/4136863782 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee80c2d70 con 0x7f8ee806a280 2022-04-03T19:53:21.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.279+0000 7f8ee659c700 1 --2- 172.21.15.77:0/4136863782 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ee806a280 0x7f8ee8159050 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8edc009760 tx=0x7f8edc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229b81319410539b server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:21.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.279+0000 7f8ee5d9b700 1 -- 172.21.15.77:0/4136863782 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc0105b0 con 0x7f8ee806a280 2022-04-03T19:53:21.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.279+0000 7f8ee5d9b700 1 -- 172.21.15.77:0/4136863782 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8edc010710 con 0x7f8ee806a280 2022-04-03T19:53:21.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.279+0000 7f8ee5d9b700 1 -- 172.21.15.77:0/4136863782 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc010a90 con 0x7f8ee806a280 2022-04-03T19:53:21.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.280+0000 7f8eec8dc700 1 -- 172.21.15.77:0/4136863782 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ee806a280 msgr2=0x7f8ee8159050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:21.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.280+0000 7f8eec8dc700 1 --2- 172.21.15.77:0/4136863782 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ee806a280 0x7f8ee8159050 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8edc009760 tx=0x7f8edc009060 comp rx=0 tx=0).stop 2022-04-03T19:53:21.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.280+0000 7f8eec8dc700 1 -- 172.21.15.77:0/4136863782 shutdown_connections 2022-04-03T19:53:21.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.280+0000 7f8eec8dc700 1 --2- 172.21.15.77:0/4136863782 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ee806a280 0x7f8ee8159050 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:21.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.280+0000 7f8eec8dc700 1 -- 172.21.15.77:0/4136863782 >> 172.21.15.77:0/4136863782 conn(0x7f8ee806cb30 msgr2=0x7f8ee806cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:21.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.280+0000 7f8eec8dc700 1 -- 172.21.15.77:0/4136863782 shutdown_connections 2022-04-03T19:53:21.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.280+0000 7f8eec8dc700 1 -- 172.21.15.77:0/4136863782 wait complete. 2022-04-03T19:53:21.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.281+0000 7f8eec8dc700 1 Processor -- start 2022-04-03T19:53:21.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.282+0000 7f8eec8dc700 1 -- start start 2022-04-03T19:53:21.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.282+0000 7f8eec8dc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ee806a280 0x7f8ee814ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:21.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.282+0000 7f8eec8dc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ee80c2010 0x7f8ee80c7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:21.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.282+0000 7f8eec8dc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ee80c4dc0 0x7f8ee80c06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:21.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.283+0000 7f8eec8dc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee8169210 con 0x7f8ee806a280 2022-04-03T19:53:21.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.283+0000 7f8eec8dc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee80bb9a0 con 0x7f8ee80c2010 2022-04-03T19:53:21.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.283+0000 7f8eec8dc700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee80bc2b0 con 0x7f8ee80c4dc0 2022-04-03T19:53:21.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.283+0000 7f8ee6d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ee806a280 0x7f8ee814ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:21.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.283+0000 7f8ee659c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ee80c2010 0x7f8ee80c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:21.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.283+0000 7f8ee6d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ee806a280 0x7f8ee814ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43060/0 (socket says 172.21.15.77:43060) 2022-04-03T19:53:21.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.283+0000 7f8ee6d9d700 1 -- 172.21.15.77:0/3882242058 learned_addr learned my addr 172.21.15.77:0/3882242058 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:21.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.283+0000 7f8ee759e700 1 --2- 172.21.15.77:0/3882242058 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ee80c4dc0 0x7f8ee80c06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:21.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.283+0000 7f8ee659c700 1 -- 172.21.15.77:0/3882242058 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ee80c4dc0 msgr2=0x7f8ee80c06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:21.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.283+0000 7f8ee659c700 1 --2- 172.21.15.77:0/3882242058 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ee80c4dc0 0x7f8ee80c06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:21.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.283+0000 7f8ee659c700 1 -- 172.21.15.77:0/3882242058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ee806a280 msgr2=0x7f8ee814ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:21.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.283+0000 7f8ee659c700 1 --2- 172.21.15.77:0/3882242058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ee806a280 0x7f8ee814ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:21.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.283+0000 7f8ee659c700 1 -- 172.21.15.77:0/3882242058 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8edc0092a0 con 0x7f8ee80c2010 2022-04-03T19:53:21.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.283+0000 7f8ee6d9d700 1 --2- 172.21.15.77:0/3882242058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ee806a280 0x7f8ee814ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:53:21.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.284+0000 7f8ee659c700 1 --2- 172.21.15.77:0/3882242058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ee80c2010 0x7f8ee80c7910 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8edc015000 tx=0x7f8edc016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:21.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.284+0000 7f8eceffd700 1 -- 172.21.15.77:0/3882242058 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc010880 con 0x7f8ee80c2010 2022-04-03T19:53:21.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.284+0000 7f8eceffd700 1 -- 172.21.15.77:0/3882242058 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8edc0109e0 con 0x7f8ee80c2010 2022-04-03T19:53:21.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.284+0000 7f8eceffd700 1 -- 172.21.15.77:0/3882242058 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc025640 con 0x7f8ee80c2010 2022-04-03T19:53:21.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.284+0000 7f8eec8dc700 1 -- 172.21.15.77:0/3882242058 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ee8168bc0 con 0x7f8ee80c2010 2022-04-03T19:53:21.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.284+0000 7f8eec8dc700 1 -- 172.21.15.77:0/3882242058 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ee81504c0 con 0x7f8ee80c2010 2022-04-03T19:53:21.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.285+0000 7f8eec8dc700 1 -- 172.21.15.77:0/3882242058 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ee81693c0 con 0x7f8ee80c2010 2022-04-03T19:53:21.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.286+0000 7f8eceffd700 1 -- 172.21.15.77:0/3882242058 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f8edc00f070 con 0x7f8ee80c2010 2022-04-03T19:53:21.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.287+0000 7f8eceffd700 1 --2- 172.21.15.77:0/3882242058 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8ec8063db0 0x7f8ec8066260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:21.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.287+0000 7f8eceffd700 1 -- 172.21.15.77:0/3882242058 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f8edc081a00 con 0x7f8ee80c2010 2022-04-03T19:53:21.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.287+0000 7f8ee6d9d700 1 --2- 172.21.15.77:0/3882242058 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8ec8063db0 0x7f8ec8066260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:21.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.288+0000 7f8ee6d9d700 1 --2- 172.21.15.77:0/3882242058 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8ec8063db0 0x7f8ec8066260 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8001d50 tx=0x7f8ed8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:21.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.291+0000 7f8eceffd700 1 -- 172.21.15.77:0/3882242058 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f8ee81693c0 con 0x7f8ee80c2010 2022-04-03T19:53:21.523 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.523+0000 7f8eec8dc700 1 -- 172.21.15.77:0/3882242058 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ee81693c0 con 0x7f8ee80c2010 2022-04-03T19:53:21.524 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.524+0000 7f8eceffd700 1 -- 172.21.15.77:0/3882242058 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5352 (secure 0 0 0) 0x7f8ee81693c0 con 0x7f8ee80c2010 2022-04-03T19:53:21.524 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:53:21.525 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:53:19.217636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5039":{"gid":5039,"name":"d","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:53:21.527 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.526+0000 7f8eec8dc700 1 -- 172.21.15.77:0/3882242058 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8ec8063db0 msgr2=0x7f8ec8066260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:21.528 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.526+0000 7f8eec8dc700 1 --2- 172.21.15.77:0/3882242058 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8ec8063db0 0x7f8ec8066260 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8001d50 tx=0x7f8ed8009040 comp rx=0 tx=0).stop 2022-04-03T19:53:21.528 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.526+0000 7f8eec8dc700 1 -- 172.21.15.77:0/3882242058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ee80c2010 msgr2=0x7f8ee80c7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:21.528 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.526+0000 7f8eec8dc700 1 --2- 172.21.15.77:0/3882242058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ee80c2010 0x7f8ee80c7910 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8edc015000 tx=0x7f8edc016000 comp rx=0 tx=0).stop 2022-04-03T19:53:21.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.526+0000 7f8eec8dc700 1 -- 172.21.15.77:0/3882242058 shutdown_connections 2022-04-03T19:53:21.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.526+0000 7f8eec8dc700 1 --2- 172.21.15.77:0/3882242058 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8ec8063db0 0x7f8ec8066260 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:21.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.526+0000 7f8eec8dc700 1 --2- 172.21.15.77:0/3882242058 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ee80c4dc0 0x7f8ee80c06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:21.529 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.526+0000 7f8eec8dc700 1 --2- 172.21.15.77:0/3882242058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8ee806a280 0x7f8ee814ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:21.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.526+0000 7f8eec8dc700 1 --2- 172.21.15.77:0/3882242058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ee80c2010 0x7f8ee80c7910 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:21.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.526+0000 7f8eec8dc700 1 -- 172.21.15.77:0/3882242058 >> 172.21.15.77:0/3882242058 conn(0x7f8ee806cb30 msgr2=0x7f8ee814ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:21.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.527+0000 7f8eec8dc700 1 -- 172.21.15.77:0/3882242058 shutdown_connections 2022-04-03T19:53:21.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.527+0000 7f8eec8dc700 1 -- 172.21.15.77:0/3882242058 wait complete. 2022-04-03T19:53:21.531 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 32 2022-04-03T19:53:21.552 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2022-04-03T19:53:21.552 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-04-03T19:53:21.553 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:53:21.751 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.750+0000 7f120e104700 1 Processor -- start 2022-04-03T19:53:21.751 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.751+0000 7f120e104700 1 -- start start 2022-04-03T19:53:21.752 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.752+0000 7f120e104700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f120806bbe0 0x7f12081584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:21.753 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.752+0000 7f120e104700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f120806c500 0x7f120814eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:21.753 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.752+0000 7f120e104700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120806d260 con 0x7f120806bfb0 2022-04-03T19:53:21.753 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.752+0000 7f120e104700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1208159450 con 0x7f120806bbe0 2022-04-03T19:53:21.753 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.752+0000 7f120e104700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12080c6330 con 0x7f120806c500 2022-04-03T19:53:21.754 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.752+0000 7f120d102700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f120806bbe0 0x7f12081584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:21.754 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.752+0000 7f120d903700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f120806c500 0x7f120814eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:21.754 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.752+0000 7f120d102700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f120806bbe0 0x7f12081584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38256/0 (socket says 172.21.15.77:38256) 2022-04-03T19:53:21.754 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.752+0000 7f120d102700 1 -- 172.21.15.77:0/1105081898 learned_addr learned my addr 172.21.15.77:0/1105081898 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:21.757 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.752+0000 7f120d102700 1 -- 172.21.15.77:0/1105081898 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f120806c500 msgr2=0x7f120814eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:53:21.757 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.752+0000 7f120d102700 1 --2- 172.21.15.77:0/1105081898 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f120806c500 0x7f120814eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:21.757 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.753+0000 7f120d102700 1 -- 172.21.15.77:0/1105081898 >> v1:172.21.15.77:6789/0 conn(0x7f120806bfb0 legacy=0x7f120806d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:21.757 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.753+0000 7f120d102700 1 -- 172.21.15.77:0/1105081898 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1208158bb0 con 0x7f120806bbe0 2022-04-03T19:53:21.758 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.753+0000 7f120d102700 1 --2- 172.21.15.77:0/1105081898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f120806bbe0 0x7f12081584b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1204006fd0 tx=0x7f12040090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fade6934d4245bc9 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:21.758 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.753+0000 7f11f7fff700 1 -- 172.21.15.77:0/1105081898 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1204004030 con 0x7f120806bbe0 2022-04-03T19:53:21.758 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.753+0000 7f11f7fff700 1 -- 172.21.15.77:0/1105081898 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12040041d0 con 0x7f120806bbe0 2022-04-03T19:53:21.758 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.753+0000 7f11f7fff700 1 -- 172.21.15.77:0/1105081898 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1204011630 con 0x7f120806bbe0 2022-04-03T19:53:21.759 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.753+0000 7f120e104700 1 -- 172.21.15.77:0/1105081898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f120806bbe0 msgr2=0x7f12081584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:21.759 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.753+0000 7f120e104700 1 --2- 172.21.15.77:0/1105081898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f120806bbe0 0x7f12081584b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1204006fd0 tx=0x7f12040090f0 comp rx=0 tx=0).stop 2022-04-03T19:53:21.759 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.754+0000 7f120e104700 1 -- 172.21.15.77:0/1105081898 shutdown_connections 2022-04-03T19:53:21.759 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.754+0000 7f120e104700 1 --2- 172.21.15.77:0/1105081898 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f120806c500 0x7f120814eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:21.759 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.754+0000 7f120e104700 1 --2- 172.21.15.77:0/1105081898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f120806bbe0 0x7f12081584b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:21.760 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.754+0000 7f120e104700 1 -- 172.21.15.77:0/1105081898 >> 172.21.15.77:0/1105081898 conn(0x7f12080cb6b0 msgr2=0x7f12080cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:21.760 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.754+0000 7f120e104700 1 -- 172.21.15.77:0/1105081898 shutdown_connections 2022-04-03T19:53:21.760 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.754+0000 7f120e104700 1 -- 172.21.15.77:0/1105081898 wait complete. 2022-04-03T19:53:21.760 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.755+0000 7f120e104700 1 Processor -- start 2022-04-03T19:53:21.761 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.755+0000 7f120e104700 1 -- start start 2022-04-03T19:53:21.761 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.756+0000 7f120e104700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f120806bbe0 0x7f12080c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:21.761 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.756+0000 7f120e104700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f120806bfb0 0x7f1208157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:21.761 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.756+0000 7f120e104700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f120806c500 0x7f1208158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:21.761 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.756+0000 7f120e104700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1208149830 con 0x7f120806c500 2022-04-03T19:53:21.762 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.756+0000 7f120e104700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12080c0060 con 0x7f120806bfb0 2022-04-03T19:53:21.762 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.756+0000 7f120e104700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1208169910 con 0x7f120806bbe0 2022-04-03T19:53:21.764 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.757+0000 7f120d903700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f120806c500 0x7f1208158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:21.765 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.757+0000 7f120d903700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f120806c500 0x7f1208158130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43062/0 (socket says 172.21.15.77:43062) 2022-04-03T19:53:21.765 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.757+0000 7f120d903700 1 -- 172.21.15.77:0/3795290076 learned_addr learned my addr 172.21.15.77:0/3795290076 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:21.765 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.757+0000 7f120d102700 1 --2- 172.21.15.77:0/3795290076 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f120806bbe0 0x7f12080c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:21.765 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.757+0000 7f120c901700 1 --2- 172.21.15.77:0/3795290076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f120806bfb0 0x7f1208157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:21.765 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.757+0000 7f120d903700 1 -- 172.21.15.77:0/3795290076 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f120806bbe0 msgr2=0x7f12080c2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:21.766 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.757+0000 7f120d903700 1 --2- 172.21.15.77:0/3795290076 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f120806bbe0 0x7f12080c2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:21.766 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.757+0000 7f120d903700 1 -- 172.21.15.77:0/3795290076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f120806bfb0 msgr2=0x7f1208157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:21.766 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.757+0000 7f120d903700 1 --2- 172.21.15.77:0/3795290076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f120806bfb0 0x7f1208157bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:21.766 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.757+0000 7f120d903700 1 -- 172.21.15.77:0/3795290076 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1204009370 con 0x7f120806c500 2022-04-03T19:53:21.767 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.758+0000 7f120d903700 1 --2- 172.21.15.77:0/3795290076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f120806c500 0x7f1208158130 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1200009ec0 tx=0x7f120000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:21.767 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.758+0000 7f11f57fa700 1 -- 172.21.15.77:0/3795290076 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120000cb70 con 0x7f120806c500 2022-04-03T19:53:21.767 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.758+0000 7f11f57fa700 1 -- 172.21.15.77:0/3795290076 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120000f040 con 0x7f120806c500 2022-04-03T19:53:21.767 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.758+0000 7f120e104700 1 -- 172.21.15.77:0/3795290076 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1208166b20 con 0x7f120806c500 2022-04-03T19:53:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.758+0000 7f11f57fa700 1 -- 172.21.15.77:0/3795290076 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1200014480 con 0x7f120806c500 2022-04-03T19:53:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.758+0000 7f120e104700 1 -- 172.21.15.77:0/3795290076 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1208166dc0 con 0x7f120806c500 2022-04-03T19:53:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.759+0000 7f120e104700 1 -- 172.21.15.77:0/3795290076 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12080b1300 con 0x7f120806c500 2022-04-03T19:53:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.760+0000 7f11f57fa700 1 -- 172.21.15.77:0/3795290076 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f1200014830 con 0x7f120806c500 2022-04-03T19:53:21.768 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.761+0000 7f11f57fa700 1 --2- 172.21.15.77:0/3795290076 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f11e8063ce0 0x7f11e8066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:21.769 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.761+0000 7f11f57fa700 1 -- 172.21.15.77:0/3795290076 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f1200081610 con 0x7f120806c500 2022-04-03T19:53:21.769 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.761+0000 7f120d102700 1 --2- 172.21.15.77:0/3795290076 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f11e8063ce0 0x7f11e8066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:21.769 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.762+0000 7f120d102700 1 --2- 172.21.15.77:0/3795290076 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f11e8063ce0 0x7f11e8066190 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1204009340 tx=0x7f1204016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:21.770 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.765+0000 7f11f57fa700 1 -- 172.21.15.77:0/3795290076 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f1200050000 con 0x7f120806c500 2022-04-03T19:53:21.996 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.996+0000 7f120e104700 1 -- 172.21.15.77:0/3795290076 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12080c7ec0 con 0x7f120806c500 2022-04-03T19:53:21.997 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.997+0000 7f11f57fa700 1 -- 172.21.15.77:0/3795290076 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5352 (secure 0 0 0) 0x7f1200018020 con 0x7f120806c500 2022-04-03T19:53:21.997 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:53:21.998 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:53:19.217636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5039":{"gid":5039,"name":"d","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:53:22.000 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.999+0000 7f120e104700 1 -- 172.21.15.77:0/3795290076 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f11e8063ce0 msgr2=0x7f11e8066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:22.000 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.999+0000 7f120e104700 1 --2- 172.21.15.77:0/3795290076 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f11e8063ce0 0x7f11e8066190 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1204009340 tx=0x7f1204016000 comp rx=0 tx=0).stop 2022-04-03T19:53:22.001 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.999+0000 7f120e104700 1 -- 172.21.15.77:0/3795290076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f120806c500 msgr2=0x7f1208158130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:22.001 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.999+0000 7f120e104700 1 --2- 172.21.15.77:0/3795290076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f120806c500 0x7f1208158130 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1200009ec0 tx=0x7f120000b040 comp rx=0 tx=0).stop 2022-04-03T19:53:22.001 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.999+0000 7f120e104700 1 -- 172.21.15.77:0/3795290076 shutdown_connections 2022-04-03T19:53:22.001 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.999+0000 7f120e104700 1 --2- 172.21.15.77:0/3795290076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f120806c500 0x7f1208158130 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:22.001 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.999+0000 7f120e104700 1 --2- 172.21.15.77:0/3795290076 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f11e8063ce0 0x7f11e8066190 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:22.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.999+0000 7f120e104700 1 --2- 172.21.15.77:0/3795290076 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f120806bbe0 0x7f12080c2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:22.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.999+0000 7f120e104700 1 --2- 172.21.15.77:0/3795290076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f120806bfb0 0x7f1208157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:22.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.999+0000 7f120e104700 1 -- 172.21.15.77:0/3795290076 >> 172.21.15.77:0/3795290076 conn(0x7f12080cb6b0 msgr2=0x7f120814dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:22.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.999+0000 7f120e104700 1 -- 172.21.15.77:0/3795290076 shutdown_connections 2022-04-03T19:53:22.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:21.999+0000 7f120e104700 1 -- 172.21.15.77:0/3795290076 wait complete. 2022-04-03T19:53:22.003 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 32 2022-04-03T19:53:22.025 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 32, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-04-03T19:44:21.275716+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5039": { "addr": "172.21.15.186:6837/480451242", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 480451242, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 480451242, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 5039, "incarnation": 32, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-04-03T19:53:19.217636+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5039 } } } ], "standbys": [ { "addr": "172.21.15.77:6834/2116223509", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6832", "nonce": 2116223509, "type": "v2" }, { "addr": "172.21.15.77:6834", "nonce": 2116223509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4245, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6836/2374276747", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2374276747, "type": "v2" }, { "addr": "172.21.15.186:6836", "nonce": 2374276747, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4259, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/610990853", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6838", "nonce": 610990853, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 610990853, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4265, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.77:6835/170053412", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6833", "nonce": 170053412, "type": "v2" }, { "addr": "172.21.15.77:6835", "nonce": 170053412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4271, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-04-03T19:53:24.026 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:53:24.236 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.234+0000 7f36af59e700 1 Processor -- start 2022-04-03T19:53:24.236 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.235+0000 7f36af59e700 1 -- start start 2022-04-03T19:53:24.237 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.237+0000 7f36af59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36b015c0d0 0x7f36b00c7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:24.238 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.237+0000 7f36af59e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b00c1f50 con 0x7f36b0162940 2022-04-03T19:53:24.238 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.237+0000 7f36af59e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b00691b0 con 0x7f36b0087a30 2022-04-03T19:53:24.238 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.237+0000 7f36af59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b0069490 con 0x7f36b015c0d0 2022-04-03T19:53:24.239 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.237+0000 7f36ae59c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36b015c0d0 0x7f36b00c7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:24.239 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.237+0000 7f36ae59c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36b015c0d0 0x7f36b00c7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60348/0 (socket says 172.21.15.77:60348) 2022-04-03T19:53:24.239 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.237+0000 7f36ae59c700 1 -- 172.21.15.77:0/3417895675 learned_addr learned my addr 172.21.15.77:0/3417895675 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:24.242 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.238+0000 7f36ae59c700 1 -- 172.21.15.77:0/3417895675 >> v1:172.21.15.186:6789/0 conn(0x7f36b0087a30 legacy=0x7f36b0162230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:24.242 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.238+0000 7f36ae59c700 1 -- 172.21.15.77:0/3417895675 >> v1:172.21.15.77:6789/0 conn(0x7f36b0162940 legacy=0x7f36b016c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:24.243 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.238+0000 7f36ae59c700 1 -- 172.21.15.77:0/3417895675 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b0069770 con 0x7f36b015c0d0 2022-04-03T19:53:24.243 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.239+0000 7f36ae59c700 1 --2- 172.21.15.77:0/3417895675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36b015c0d0 0x7f36b00c7200 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f36a0008830 tx=0x7f36a0008190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc16b2439bcb73d8 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:24.243 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.239+0000 7f36ad59a700 1 -- 172.21.15.77:0/3417895675 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a0010070 con 0x7f36b015c0d0 2022-04-03T19:53:24.243 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.239+0000 7f36ad59a700 1 -- 172.21.15.77:0/3417895675 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a0004030 con 0x7f36b015c0d0 2022-04-03T19:53:24.244 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.239+0000 7f36ad59a700 1 -- 172.21.15.77:0/3417895675 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a00043f0 con 0x7f36b015c0d0 2022-04-03T19:53:24.244 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.240+0000 7f36af59e700 1 -- 172.21.15.77:0/3417895675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36b015c0d0 msgr2=0x7f36b00c7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:24.244 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.240+0000 7f36af59e700 1 --2- 172.21.15.77:0/3417895675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36b015c0d0 0x7f36b00c7200 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f36a0008830 tx=0x7f36a0008190 comp rx=0 tx=0).stop 2022-04-03T19:53:24.244 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.241+0000 7f36af59e700 1 -- 172.21.15.77:0/3417895675 shutdown_connections 2022-04-03T19:53:24.245 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.241+0000 7f36af59e700 1 --2- 172.21.15.77:0/3417895675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36b015c0d0 0x7f36b00c7200 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:24.245 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.241+0000 7f36af59e700 1 -- 172.21.15.77:0/3417895675 >> 172.21.15.77:0/3417895675 conn(0x7f36b0066da0 msgr2=0x7f36b00671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:24.245 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.241+0000 7f36af59e700 1 -- 172.21.15.77:0/3417895675 shutdown_connections 2022-04-03T19:53:24.245 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.241+0000 7f36af59e700 1 -- 172.21.15.77:0/3417895675 wait complete. 2022-04-03T19:53:24.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.243+0000 7f36af59e700 1 Processor -- start 2022-04-03T19:53:24.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.243+0000 7f36af59e700 1 -- start start 2022-04-03T19:53:24.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.245+0000 7f36af59e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36b0087a30 0x7f36b00c6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:24.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.245+0000 7f36af59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36b015c0d0 0x7f36b00b8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:24.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.245+0000 7f36af59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36b0162940 0x7f36b00c3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:24.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.245+0000 7f36af59e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b00b8300 con 0x7f36b015c0d0 2022-04-03T19:53:24.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.245+0000 7f36af59e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b00b88c0 con 0x7f36b0087a30 2022-04-03T19:53:24.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.245+0000 7f36af59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b00ba2b0 con 0x7f36b0162940 2022-04-03T19:53:24.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.245+0000 7f36ae59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36b0087a30 0x7f36b00c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:24.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.245+0000 7f36ae59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36b0087a30 0x7f36b00c6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38262/0 (socket says 172.21.15.77:38262) 2022-04-03T19:53:24.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.245+0000 7f36ae59c700 1 -- 172.21.15.77:0/3109874847 learned_addr learned my addr 172.21.15.77:0/3109874847 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:24.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.246+0000 7f36add9b700 1 --2- 172.21.15.77:0/3109874847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36b015c0d0 0x7f36b00b8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:24.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.246+0000 7f36ae59c700 1 -- 172.21.15.77:0/3109874847 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36b0162940 msgr2=0x7f36b00c3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:24.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.246+0000 7f36ae59c700 1 --2- 172.21.15.77:0/3109874847 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36b0162940 0x7f36b00c3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:24.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.246+0000 7f36ae59c700 1 -- 172.21.15.77:0/3109874847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36b015c0d0 msgr2=0x7f36b00b8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:24.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.247+0000 7f36ae59c700 1 --2- 172.21.15.77:0/3109874847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36b015c0d0 0x7f36b00b8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:24.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.247+0000 7f36ae59c700 1 -- 172.21.15.77:0/3109874847 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a00083f0 con 0x7f36b0087a30 2022-04-03T19:53:24.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.247+0000 7f36add9b700 1 --2- 172.21.15.77:0/3109874847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36b015c0d0 0x7f36b00b8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:53:24.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.248+0000 7f36ae59c700 1 --2- 172.21.15.77:0/3109874847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36b0087a30 0x7f36b00c6830 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f36a00081d0 tx=0x7f36a001b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:24.257 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.248+0000 7f369e7fc700 1 -- 172.21.15.77:0/3109874847 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a00034e0 con 0x7f36b0087a30 2022-04-03T19:53:24.257 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.248+0000 7f369e7fc700 1 -- 172.21.15.77:0/3109874847 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a0003640 con 0x7f36b0087a30 2022-04-03T19:53:24.257 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.248+0000 7f36af59e700 1 -- 172.21.15.77:0/3109874847 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b00ba590 con 0x7f36b0087a30 2022-04-03T19:53:24.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.249+0000 7f369e7fc700 1 -- 172.21.15.77:0/3109874847 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a0003a00 con 0x7f36b0087a30 2022-04-03T19:53:24.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.249+0000 7f36af59e700 1 -- 172.21.15.77:0/3109874847 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b00c3c60 con 0x7f36b0087a30 2022-04-03T19:53:24.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.250+0000 7f36af59e700 1 -- 172.21.15.77:0/3109874847 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b00b8710 con 0x7f36b0087a30 2022-04-03T19:53:24.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.253+0000 7f369e7fc700 1 -- 172.21.15.77:0/3109874847 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f36a0014070 con 0x7f36b0087a30 2022-04-03T19:53:24.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.254+0000 7f369e7fc700 1 --2- 172.21.15.77:0/3109874847 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f368c063ce0 0x7f368c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:24.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.254+0000 7f369e7fc700 1 -- 172.21.15.77:0/3109874847 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f36a00823e0 con 0x7f36b0087a30 2022-04-03T19:53:24.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.255+0000 7f36add9b700 1 --2- 172.21.15.77:0/3109874847 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f368c063ce0 0x7f368c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:24.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.257+0000 7f369e7fc700 1 -- 172.21.15.77:0/3109874847 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f36a0050e90 con 0x7f36b0087a30 2022-04-03T19:53:24.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.257+0000 7f36add9b700 1 --2- 172.21.15.77:0/3109874847 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f368c063ce0 0x7f368c066190 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f36b00c6730 tx=0x7f36a4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:24.506 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.506+0000 7f36af59e700 1 -- 172.21.15.77:0/3109874847 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36b00c3e50 con 0x7f36b0087a30 2022-04-03T19:53:24.507 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.507+0000 7f369e7fc700 1 -- 172.21.15.77:0/3109874847 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5352 (secure 0 0 0) 0x7f36a0054470 con 0x7f36b0087a30 2022-04-03T19:53:24.508 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:53:24.508 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:53:19.217636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5039":{"gid":5039,"name":"d","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:53:24.510 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.509+0000 7f36af59e700 1 -- 172.21.15.77:0/3109874847 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f368c063ce0 msgr2=0x7f368c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:24.510 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.509+0000 7f36af59e700 1 --2- 172.21.15.77:0/3109874847 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f368c063ce0 0x7f368c066190 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f36b00c6730 tx=0x7f36a4009040 comp rx=0 tx=0).stop 2022-04-03T19:53:24.510 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.509+0000 7f36af59e700 1 -- 172.21.15.77:0/3109874847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36b0087a30 msgr2=0x7f36b00c6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:24.511 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.509+0000 7f36af59e700 1 --2- 172.21.15.77:0/3109874847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36b0087a30 0x7f36b00c6830 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f36a00081d0 tx=0x7f36a001b000 comp rx=0 tx=0).stop 2022-04-03T19:53:24.511 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.509+0000 7f36af59e700 1 -- 172.21.15.77:0/3109874847 shutdown_connections 2022-04-03T19:53:24.511 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.509+0000 7f36af59e700 1 --2- 172.21.15.77:0/3109874847 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f368c063ce0 0x7f368c066190 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:24.511 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.509+0000 7f36af59e700 1 --2- 172.21.15.77:0/3109874847 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36b0162940 0x7f36b00c3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:24.512 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.509+0000 7f36af59e700 1 --2- 172.21.15.77:0/3109874847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36b0087a30 0x7f36b00c6830 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:24.512 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.509+0000 7f36af59e700 1 --2- 172.21.15.77:0/3109874847 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f36b015c0d0 0x7f36b00b8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:24.512 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.509+0000 7f36af59e700 1 -- 172.21.15.77:0/3109874847 >> 172.21.15.77:0/3109874847 conn(0x7f36b0066da0 msgr2=0x7f36b0163e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:24.512 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.509+0000 7f36af59e700 1 -- 172.21.15.77:0/3109874847 shutdown_connections 2022-04-03T19:53:24.513 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:24.509+0000 7f36af59e700 1 -- 172.21.15.77:0/3109874847 wait complete. 2022-04-03T19:53:24.513 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 32 2022-04-03T19:53:24.533 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:53:26.534 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:53:26.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.741+0000 7fced4e50700 1 Processor -- start 2022-04-03T19:53:26.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.742+0000 7fced4e50700 1 -- start start 2022-04-03T19:53:26.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.743+0000 7fced4e50700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced006bfb0 0x7fced01584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:26.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.743+0000 7fced4e50700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fced006c500 0x7fced014eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:26.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.743+0000 7fced4e50700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced006d260 con 0x7fced006bbe0 2022-04-03T19:53:26.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.743+0000 7fced4e50700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fced0159450 con 0x7fced006c500 2022-04-03T19:53:26.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.743+0000 7fced4e50700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced00c6330 con 0x7fced006bfb0 2022-04-03T19:53:26.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.743+0000 7fcecf7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fced006bbe0 0x7fced006d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38518/0 (socket says 172.21.15.77:38518) 2022-04-03T19:53:26.747 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.743+0000 7fcecf7fe700 1 -- 172.21.15.77:0/382402511 learned_addr learned my addr 172.21.15.77:0/382402511 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:53:26.747 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.743+0000 7fceceffd700 1 --2- 172.21.15.77:0/382402511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced006bfb0 0x7fced01584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:26.747 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.743+0000 7fcecffff700 1 --2- 172.21.15.77:0/382402511 >> v2:172.21.15.77:3300/0 conn(0x7fced006c500 0x7fced014eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:26.747 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.744+0000 7fceceffd700 1 -- 172.21.15.77:0/382402511 >> v2:172.21.15.77:3300/0 conn(0x7fced006c500 msgr2=0x7fced014eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:53:26.748 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.744+0000 7fceceffd700 1 --2- 172.21.15.77:0/382402511 >> v2:172.21.15.77:3300/0 conn(0x7fced006c500 0x7fced014eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:26.748 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.744+0000 7fceceffd700 1 -- 172.21.15.77:0/382402511 >> v1:172.21.15.77:6789/0 conn(0x7fced006bbe0 legacy=0x7fced006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:26.748 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.744+0000 7fcece7fc700 1 -- 172.21.15.77:0/382402511 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777641905 0 0) 0x7fced006d260 con 0x7fced006bbe0 2022-04-03T19:53:26.748 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.744+0000 7fceceffd700 1 -- 172.21.15.77:0/382402511 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced0158bb0 con 0x7fced006bfb0 2022-04-03T19:53:26.748 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.744+0000 7fcecffff700 1 --2- 172.21.15.77:0/382402511 >> v2:172.21.15.77:3300/0 conn(0x7fced006c500 0x7fced014eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:53:26.749 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.744+0000 7fceceffd700 1 --2- 172.21.15.77:0/382402511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced006bfb0 0x7fced01584b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcec4009760 tx=0x7fcec4009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5db91b39d071f380 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:26.749 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.744+0000 7fcece7fc700 1 -- 172.21.15.77:0/382402511 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec4011630 con 0x7fced006bfb0 2022-04-03T19:53:26.749 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.744+0000 7fcece7fc700 1 -- 172.21.15.77:0/382402511 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec4011790 con 0x7fced006bfb0 2022-04-03T19:53:26.749 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.744+0000 7fcece7fc700 1 -- 172.21.15.77:0/382402511 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec4011b30 con 0x7fced006bfb0 2022-04-03T19:53:26.750 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.745+0000 7fced4e50700 1 -- 172.21.15.77:0/382402511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced006bfb0 msgr2=0x7fced01584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:26.750 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.745+0000 7fced4e50700 1 --2- 172.21.15.77:0/382402511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced006bfb0 0x7fced01584b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcec4009760 tx=0x7fcec4009060 comp rx=0 tx=0).stop 2022-04-03T19:53:26.750 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.745+0000 7fced4e50700 1 -- 172.21.15.77:0/382402511 shutdown_connections 2022-04-03T19:53:26.750 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.745+0000 7fced4e50700 1 --2- 172.21.15.77:0/382402511 >> v2:172.21.15.77:3300/0 conn(0x7fced006c500 0x7fced014eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:26.750 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.745+0000 7fced4e50700 1 --2- 172.21.15.77:0/382402511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced006bfb0 0x7fced01584b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:26.751 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.745+0000 7fced4e50700 1 -- 172.21.15.77:0/382402511 >> 172.21.15.77:0/382402511 conn(0x7fced00cb6b0 msgr2=0x7fced00cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:26.751 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.745+0000 7fced4e50700 1 -- 172.21.15.77:0/382402511 shutdown_connections 2022-04-03T19:53:26.751 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.745+0000 7fced4e50700 1 -- 172.21.15.77:0/382402511 wait complete. 2022-04-03T19:53:26.754 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.747+0000 7fced4e50700 1 Processor -- start 2022-04-03T19:53:26.754 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.747+0000 7fced4e50700 1 -- start start 2022-04-03T19:53:26.754 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.748+0000 7fced4e50700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced006bbe0 0x7fced0165720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:26.755 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.748+0000 7fced4e50700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fced006bfb0 0x7fced0154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:26.755 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.748+0000 7fced4e50700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced006c500 0x7fced0154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:26.755 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.748+0000 7fced4e50700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced0155590 con 0x7fced006bbe0 2022-04-03T19:53:26.755 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.748+0000 7fced4e50700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced0151270 con 0x7fced006bfb0 2022-04-03T19:53:26.756 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.748+0000 7fced4e50700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced00bf750 con 0x7fced006c500 2022-04-03T19:53:26.756 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.748+0000 7fcecf7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced006bbe0 0x7fced0165720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:26.756 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.748+0000 7fcecffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced006c500 0x7fced0154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:26.756 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.748+0000 7fceceffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fced006bfb0 0x7fced0154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:26.756 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.748+0000 7fcecf7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced006bbe0 0x7fced0165720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43068/0 (socket says 172.21.15.77:43068) 2022-04-03T19:53:26.757 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.748+0000 7fcecffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced006c500 0x7fced0154ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60354/0 (socket says 172.21.15.77:60354) 2022-04-03T19:53:26.757 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.748+0000 7fcecf7fe700 1 -- 172.21.15.77:0/692362036 learned_addr learned my addr 172.21.15.77:0/692362036 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:26.757 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.748+0000 7fceceffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fced006bfb0 0x7fced0154760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38264/0 (socket says 172.21.15.77:38264) 2022-04-03T19:53:26.757 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.748+0000 7fcecffff700 1 -- 172.21.15.77:0/692362036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fced006bfb0 msgr2=0x7fced0154760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:26.758 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.748+0000 7fcecffff700 1 --2- 172.21.15.77:0/692362036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fced006bfb0 0x7fced0154760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:26.758 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.748+0000 7fcecffff700 1 -- 172.21.15.77:0/692362036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced006bbe0 msgr2=0x7fced0165720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:26.758 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.748+0000 7fcecffff700 1 --2- 172.21.15.77:0/692362036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced006bbe0 0x7fced0165720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:26.758 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.748+0000 7fcecffff700 1 -- 172.21.15.77:0/692362036 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec40092a0 con 0x7fced006c500 2022-04-03T19:53:26.758 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.748+0000 7fcecf7fe700 1 --2- 172.21.15.77:0/692362036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced006bbe0 0x7fced0165720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:53:26.759 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.749+0000 7fcecffff700 1 --2- 172.21.15.77:0/692362036 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced006c500 0x7fced0154ca0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcec8007f60 tx=0x7fcec80098f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:26.759 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.749+0000 7fceb77fe700 1 -- 172.21.15.77:0/692362036 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec800db80 con 0x7fced006c500 2022-04-03T19:53:26.759 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.749+0000 7fceb77fe700 1 -- 172.21.15.77:0/692362036 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec800f040 con 0x7fced006c500 2022-04-03T19:53:26.759 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.749+0000 7fceb77fe700 1 -- 172.21.15.77:0/692362036 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec8003860 con 0x7fced006c500 2022-04-03T19:53:26.760 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.749+0000 7fced4e50700 1 -- 172.21.15.77:0/692362036 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced0152d50 con 0x7fced006c500 2022-04-03T19:53:26.760 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.749+0000 7fced4e50700 1 -- 172.21.15.77:0/692362036 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced0153310 con 0x7fced006c500 2022-04-03T19:53:26.760 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.750+0000 7fced4e50700 1 -- 172.21.15.77:0/692362036 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced0154480 con 0x7fced006c500 2022-04-03T19:53:26.760 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.751+0000 7fceb77fe700 1 -- 172.21.15.77:0/692362036 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fcec801c030 con 0x7fced006c500 2022-04-03T19:53:26.760 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.752+0000 7fceb77fe700 1 --2- 172.21.15.77:0/692362036 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fceb0063d10 0x7fceb00661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:26.761 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.752+0000 7fceb77fe700 1 -- 172.21.15.77:0/692362036 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7fcec8080a20 con 0x7fced006c500 2022-04-03T19:53:26.761 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.752+0000 7fcecf7fe700 1 --2- 172.21.15.77:0/692362036 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fceb0063d10 0x7fceb00661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:26.761 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.753+0000 7fcecf7fe700 1 --2- 172.21.15.77:0/692362036 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fceb0063d10 0x7fceb00661c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcec00017a0 tx=0x7fcec0007c70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:26.762 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.760+0000 7fceb77fe700 1 -- 172.21.15.77:0/692362036 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fced0154480 con 0x7fced006c500 2022-04-03T19:53:26.996 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.995+0000 7fced4e50700 1 -- 172.21.15.77:0/692362036 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fced00b1720 con 0x7fced006c500 2022-04-03T19:53:26.997 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.996+0000 7fceb77fe700 1 -- 172.21.15.77:0/692362036 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5352 (secure 0 0 0) 0x7fcec801a020 con 0x7fced006c500 2022-04-03T19:53:26.998 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:53:26.998 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:53:19.217636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5039":{"gid":5039,"name":"d","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:53:27.001 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.998+0000 7fced4e50700 1 -- 172.21.15.77:0/692362036 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fceb0063d10 msgr2=0x7fceb00661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:27.001 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.998+0000 7fced4e50700 1 --2- 172.21.15.77:0/692362036 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fceb0063d10 0x7fceb00661c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcec00017a0 tx=0x7fcec0007c70 comp rx=0 tx=0).stop 2022-04-03T19:53:27.002 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.998+0000 7fced4e50700 1 -- 172.21.15.77:0/692362036 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced006c500 msgr2=0x7fced0154ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:27.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.998+0000 7fced4e50700 1 --2- 172.21.15.77:0/692362036 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced006c500 0x7fced0154ca0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcec8007f60 tx=0x7fcec80098f0 comp rx=0 tx=0).stop 2022-04-03T19:53:27.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.999+0000 7fced4e50700 1 -- 172.21.15.77:0/692362036 shutdown_connections 2022-04-03T19:53:27.003 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.999+0000 7fced4e50700 1 --2- 172.21.15.77:0/692362036 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fceb0063d10 0x7fceb00661c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:27.004 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.999+0000 7fced4e50700 1 --2- 172.21.15.77:0/692362036 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced006c500 0x7fced0154ca0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:27.004 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.999+0000 7fced4e50700 1 --2- 172.21.15.77:0/692362036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fced006bbe0 0x7fced0165720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:27.005 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.999+0000 7fced4e50700 1 --2- 172.21.15.77:0/692362036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fced006bfb0 0x7fced0154760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:27.005 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.999+0000 7fced4e50700 1 -- 172.21.15.77:0/692362036 >> 172.21.15.77:0/692362036 conn(0x7fced00cb6b0 msgr2=0x7fced00cba80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:27.006 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.999+0000 7fced4e50700 1 -- 172.21.15.77:0/692362036 shutdown_connections 2022-04-03T19:53:27.006 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:26.999+0000 7fced4e50700 1 -- 172.21.15.77:0/692362036 wait complete. 2022-04-03T19:53:27.006 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 32 2022-04-03T19:53:27.024 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:53:29.025 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:53:29.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.228+0000 7f0f763fa700 1 Processor -- start 2022-04-03T19:53:29.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.228+0000 7f0f763fa700 1 -- start start 2022-04-03T19:53:29.233 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.229+0000 7f0f763fa700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f700c3b10 0x7f0f7015d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:29.234 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.229+0000 7f0f763fa700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f700c3710 con 0x7f0f700c1170 2022-04-03T19:53:29.234 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.230+0000 7f0f763fa700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f70152330 con 0x7f0f70155370 2022-04-03T19:53:29.234 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.230+0000 7f0f763fa700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f700c1540 con 0x7f0f700c3b10 2022-04-03T19:53:29.234 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.230+0000 7f0f753f8700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0f700c1170 0x7f0f701527c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38520/0 (socket says 172.21.15.77:38520) 2022-04-03T19:53:29.235 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.230+0000 7f0f753f8700 1 -- 172.21.15.77:0/746656387 learned_addr learned my addr 172.21.15.77:0/746656387 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:53:29.235 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.230+0000 7f0f75bf9700 1 --2- 172.21.15.77:0/746656387 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f700c3b10 0x7f0f7015d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:29.235 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.230+0000 7f0f5ffff700 1 -- 172.21.15.77:0/746656387 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449275216 0 0) 0x7f0f700c3710 con 0x7f0f700c1170 2022-04-03T19:53:29.235 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.230+0000 7f0f5ffff700 1 -- 172.21.15.77:0/746656387 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f580033f0 con 0x7f0f700c1170 2022-04-03T19:53:29.236 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.230+0000 7f0f5ffff700 1 -- 172.21.15.77:0/746656387 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123786649 0 0) 0x7f0f70152330 con 0x7f0f70155370 2022-04-03T19:53:29.236 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.230+0000 7f0f5ffff700 1 -- 172.21.15.77:0/746656387 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f700c3710 con 0x7f0f70155370 2022-04-03T19:53:29.236 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.231+0000 7f0f5ffff700 1 -- 172.21.15.77:0/746656387 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3373262626 0 0) 0x7f0f580033f0 con 0x7f0f700c1170 2022-04-03T19:53:29.236 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.231+0000 7f0f5ffff700 1 -- 172.21.15.77:0/746656387 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f700c3b10 msgr2=0x7f0f7015d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:53:29.236 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.231+0000 7f0f5ffff700 1 --2- 172.21.15.77:0/746656387 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f700c3b10 0x7f0f7015d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:29.237 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.231+0000 7f0f5ffff700 1 -- 172.21.15.77:0/746656387 >> v1:172.21.15.186:6789/0 conn(0x7f0f70155370 legacy=0x7f0f70069110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:29.237 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.231+0000 7f0f5ffff700 1 -- 172.21.15.77:0/746656387 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f700c4270 con 0x7f0f700c1170 2022-04-03T19:53:29.237 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.231+0000 7f0f5ffff700 1 -- 172.21.15.77:0/746656387 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4219302955 0 0) 0x7f0f6c001470 con 0x7f0f700c1170 2022-04-03T19:53:29.237 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.231+0000 7f0f5ffff700 1 -- 172.21.15.77:0/746656387 >> v1:172.21.15.77:6789/0 conn(0x7f0f700c1170 legacy=0x7f0f701527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:29.237 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.231+0000 7f0f5ffff700 1 --2- 172.21.15.77:0/746656387 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f700c3b10 0x7f0f58003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:29.238 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.231+0000 7f0f5ffff700 1 --2- 172.21.15.77:0/746656387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f58004050 0x7f0f58004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:29.238 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.231+0000 7f0f5ffff700 1 --2- 172.21.15.77:0/746656387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f58004bd0 0x7f0f58005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:29.238 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.231+0000 7f0f5ffff700 1 -- 172.21.15.77:0/746656387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f700c1540 con 0x7f0f58004bd0 2022-04-03T19:53:29.238 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.231+0000 7f0f5ffff700 1 -- 172.21.15.77:0/746656387 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5800ba70 con 0x7f0f58004050 2022-04-03T19:53:29.239 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.231+0000 7f0f5ffff700 1 -- 172.21.15.77:0/746656387 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5800bc30 con 0x7f0f700c3b10 2022-04-03T19:53:29.239 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.231+0000 7f0f753f8700 1 --2- 172.21.15.77:0/746656387 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f700c3b10 0x7f0f58003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:29.239 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.231+0000 7f0f74bf7700 1 --2- 172.21.15.77:0/746656387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f58004050 0x7f0f58004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:29.239 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.231+0000 7f0f75bf9700 1 --2- 172.21.15.77:0/746656387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f58004bd0 0x7f0f58005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:29.239 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.231+0000 7f0f753f8700 1 -- 172.21.15.77:0/746656387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f58004050 msgr2=0x7f0f58004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:53:29.240 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.232+0000 7f0f753f8700 1 --2- 172.21.15.77:0/746656387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f58004050 0x7f0f58004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:29.240 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.232+0000 7f0f753f8700 1 -- 172.21.15.77:0/746656387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f58004bd0 msgr2=0x7f0f58005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:53:29.240 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.232+0000 7f0f753f8700 1 --2- 172.21.15.77:0/746656387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f58004bd0 0x7f0f58005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:29.240 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.232+0000 7f0f753f8700 1 -- 172.21.15.77:0/746656387 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f580033f0 con 0x7f0f700c3b10 2022-04-03T19:53:29.241 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.232+0000 7f0f753f8700 1 --2- 172.21.15.77:0/746656387 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f700c3b10 0x7f0f58003aa0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c007f90 tx=0x7f0f700687c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a592e8f089ad1e0c server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:29.241 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.232+0000 7f0f5ffff700 1 -- 172.21.15.77:0/746656387 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c00c9a0 con 0x7f0f700c3b10 2022-04-03T19:53:29.241 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.232+0000 7f0f5ffff700 1 -- 172.21.15.77:0/746656387 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f6c00cb00 con 0x7f0f700c3b10 2022-04-03T19:53:29.241 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.232+0000 7f0f5ffff700 1 -- 172.21.15.77:0/746656387 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c0070d0 con 0x7f0f700c3b10 2022-04-03T19:53:29.241 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.233+0000 7f0f763fa700 1 -- 172.21.15.77:0/746656387 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f700c3b10 msgr2=0x7f0f58003aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:29.242 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.233+0000 7f0f763fa700 1 --2- 172.21.15.77:0/746656387 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f700c3b10 0x7f0f58003aa0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c007f90 tx=0x7f0f700687c0 comp rx=0 tx=0).stop 2022-04-03T19:53:29.243 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.233+0000 7f0f75bf9700 1 -- 172.21.15.77:0/746656387 reap_dead start 2022-04-03T19:53:29.243 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.233+0000 7f0f763fa700 1 -- 172.21.15.77:0/746656387 shutdown_connections 2022-04-03T19:53:29.244 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.233+0000 7f0f763fa700 1 -- 172.21.15.77:0/746656387 >> 172.21.15.77:0/746656387 conn(0x7f0f70066da0 msgr2=0x7f0f700671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:29.244 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.233+0000 7f0f763fa700 1 -- 172.21.15.77:0/746656387 shutdown_connections 2022-04-03T19:53:29.244 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.233+0000 7f0f763fa700 1 -- 172.21.15.77:0/746656387 wait complete. 2022-04-03T19:53:29.247 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.234+0000 7f0f763fa700 1 Processor -- start 2022-04-03T19:53:29.247 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.234+0000 7f0f763fa700 1 -- start start 2022-04-03T19:53:29.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.235+0000 7f0f763fa700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f700c3b10 0x7f0f70146fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:29.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.235+0000 7f0f763fa700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f70152f10 0x7f0f70141ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:29.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.235+0000 7f0f763fa700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f70142430 0x7f0f70166ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:29.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.235+0000 7f0f763fa700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f700c62d0 con 0x7f0f70152f10 2022-04-03T19:53:29.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.235+0000 7f0f763fa700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f701428a0 con 0x7f0f70142430 2022-04-03T19:53:29.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.235+0000 7f0f763fa700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7014ff00 con 0x7f0f700c3b10 2022-04-03T19:53:29.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.236+0000 7f0f753f8700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f700c3b10 0x7f0f70146fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:29.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.236+0000 7f0f74bf7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f70152f10 0x7f0f70141ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:29.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.236+0000 7f0f753f8700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f700c3b10 0x7f0f70146fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60360/0 (socket says 172.21.15.77:60360) 2022-04-03T19:53:29.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.236+0000 7f0f753f8700 1 -- 172.21.15.77:0/621996971 learned_addr learned my addr 172.21.15.77:0/621996971 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:29.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.236+0000 7f0f75bf9700 1 --2- 172.21.15.77:0/621996971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f70142430 0x7f0f70166ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:29.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.236+0000 7f0f753f8700 1 -- 172.21.15.77:0/621996971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f70142430 msgr2=0x7f0f70166ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:29.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.236+0000 7f0f753f8700 1 --2- 172.21.15.77:0/621996971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f70142430 0x7f0f70166ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:29.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.236+0000 7f0f753f8700 1 -- 172.21.15.77:0/621996971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f70152f10 msgr2=0x7f0f70141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:29.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.236+0000 7f0f753f8700 1 --2- 172.21.15.77:0/621996971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f70152f10 0x7f0f70141ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:29.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.236+0000 7f0f753f8700 1 -- 172.21.15.77:0/621996971 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f7014bf30 con 0x7f0f700c3b10 2022-04-03T19:53:29.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.236+0000 7f0f74bf7700 1 --2- 172.21.15.77:0/621996971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f70152f10 0x7f0f70141ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:53:29.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.236+0000 7f0f753f8700 1 --2- 172.21.15.77:0/621996971 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f700c3b10 0x7f0f70146fd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c0079e0 tx=0x7f0f6c012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:29.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.237+0000 7f0f5d7fa700 1 -- 172.21.15.77:0/621996971 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c0075c0 con 0x7f0f700c3b10 2022-04-03T19:53:29.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.237+0000 7f0f5d7fa700 1 -- 172.21.15.77:0/621996971 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f6c001580 con 0x7f0f700c3b10 2022-04-03T19:53:29.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.237+0000 7f0f5d7fa700 1 -- 172.21.15.77:0/621996971 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c004900 con 0x7f0f700c3b10 2022-04-03T19:53:29.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.237+0000 7f0f763fa700 1 -- 172.21.15.77:0/621996971 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f7014fc20 con 0x7f0f700c3b10 2022-04-03T19:53:29.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.237+0000 7f0f763fa700 1 -- 172.21.15.77:0/621996971 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f700c5480 con 0x7f0f700c3b10 2022-04-03T19:53:29.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.238+0000 7f0f763fa700 1 -- 172.21.15.77:0/621996971 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f700c2e90 con 0x7f0f700c3b10 2022-04-03T19:53:29.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.239+0000 7f0f5d7fa700 1 -- 172.21.15.77:0/621996971 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f0f6c010070 con 0x7f0f700c3b10 2022-04-03T19:53:29.254 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.239+0000 7f0f5d7fa700 1 --2- 172.21.15.77:0/621996971 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0f50063d10 0x7f0f500661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:29.254 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.240+0000 7f0f5d7fa700 1 -- 172.21.15.77:0/621996971 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f0f6c081bb0 con 0x7f0f700c3b10 2022-04-03T19:53:29.254 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.240+0000 7f0f74bf7700 1 --2- 172.21.15.77:0/621996971 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0f50063d10 0x7f0f500661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:29.255 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.240+0000 7f0f74bf7700 1 --2- 172.21.15.77:0/621996971 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0f50063d10 0x7f0f500661c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0f70147e60 tx=0x7f0f60009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:29.255 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.243+0000 7f0f5d7fa700 1 -- 172.21.15.77:0/621996971 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f0f6c0504b0 con 0x7f0f700c3b10 2022-04-03T19:53:29.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.488+0000 7f0f763fa700 1 -- 172.21.15.77:0/621996971 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f70150560 con 0x7f0f700c3b10 2022-04-03T19:53:29.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.489+0000 7f0f5d7fa700 1 -- 172.21.15.77:0/621996971 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5356 (secure 0 0 0) 0x7f0f6c017030 con 0x7f0f700c3b10 2022-04-03T19:53:29.491 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:53:29.491 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:53:28.279103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5039":{"gid":5039,"name":"d","rank":0,"incarnation":32,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:53:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.492+0000 7f0f763fa700 1 -- 172.21.15.77:0/621996971 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0f50063d10 msgr2=0x7f0f500661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.492+0000 7f0f763fa700 1 --2- 172.21.15.77:0/621996971 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0f50063d10 0x7f0f500661c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0f70147e60 tx=0x7f0f60009040 comp rx=0 tx=0).stop 2022-04-03T19:53:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.492+0000 7f0f763fa700 1 -- 172.21.15.77:0/621996971 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f700c3b10 msgr2=0x7f0f70146fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:29.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.492+0000 7f0f763fa700 1 --2- 172.21.15.77:0/621996971 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f700c3b10 0x7f0f70146fd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c0079e0 tx=0x7f0f6c012000 comp rx=0 tx=0).stop 2022-04-03T19:53:29.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.492+0000 7f0f763fa700 1 -- 172.21.15.77:0/621996971 shutdown_connections 2022-04-03T19:53:29.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.492+0000 7f0f763fa700 1 --2- 172.21.15.77:0/621996971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f70142430 0x7f0f70166ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:29.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.492+0000 7f0f763fa700 1 --2- 172.21.15.77:0/621996971 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0f50063d10 0x7f0f500661c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:29.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.492+0000 7f0f763fa700 1 --2- 172.21.15.77:0/621996971 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f700c3b10 0x7f0f70146fd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:29.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.492+0000 7f0f763fa700 1 --2- 172.21.15.77:0/621996971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f70152f10 0x7f0f70141ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:29.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.492+0000 7f0f763fa700 1 -- 172.21.15.77:0/621996971 >> 172.21.15.77:0/621996971 conn(0x7f0f70066da0 msgr2=0x7f0f70155780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:29.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.492+0000 7f0f763fa700 1 -- 172.21.15.77:0/621996971 shutdown_connections 2022-04-03T19:53:29.497 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:29.492+0000 7f0f763fa700 1 -- 172.21.15.77:0/621996971 wait complete. 2022-04-03T19:53:29.498 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 33 2022-04-03T19:53:29.518 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:53:31.518 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:53:31.727 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.725+0000 7f63df59e700 1 Processor -- start 2022-04-03T19:53:31.727 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.725+0000 7f63df59e700 1 -- start start 2022-04-03T19:53:31.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.726+0000 7f63df59e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f63e006bfb0 0x7f63e01584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:31.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.726+0000 7f63df59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63e006c500 0x7f63e014eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:31.730 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.726+0000 7f63df59e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e006d260 con 0x7f63e006bbe0 2022-04-03T19:53:31.731 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.726+0000 7f63df59e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63e0159450 con 0x7f63e006bfb0 2022-04-03T19:53:31.731 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.726+0000 7f63df59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00c6330 con 0x7f63e006c500 2022-04-03T19:53:31.731 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.726+0000 7f63de59c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f63e006bbe0 0x7f63e006d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38522/0 (socket says 172.21.15.77:38522) 2022-04-03T19:53:31.731 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.726+0000 7f63de59c700 1 -- 172.21.15.77:0/610278851 learned_addr learned my addr 172.21.15.77:0/610278851 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:53:31.732 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.727+0000 7f63ddd9b700 1 --2- 172.21.15.77:0/610278851 >> v2:172.21.15.186:3300/0 conn(0x7f63e006bfb0 0x7f63e01584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:31.732 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.727+0000 7f63ded9d700 1 --2- 172.21.15.77:0/610278851 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63e006c500 0x7f63e014eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:31.732 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.727+0000 7f63dd59a700 1 -- 172.21.15.77:0/610278851 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934253866 0 0) 0x7f63e006d260 con 0x7f63e006bbe0 2022-04-03T19:53:31.732 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.727+0000 7f63dd59a700 1 -- 172.21.15.77:0/610278851 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c40033f0 con 0x7f63e006bbe0 2022-04-03T19:53:31.733 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.727+0000 7f63ddd9b700 1 -- 172.21.15.77:0/610278851 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63e006c500 msgr2=0x7f63e014eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:53:31.733 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.727+0000 7f63ddd9b700 1 --2- 172.21.15.77:0/610278851 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63e006c500 0x7f63e014eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:31.733 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.727+0000 7f63ddd9b700 1 -- 172.21.15.77:0/610278851 >> v1:172.21.15.77:6789/0 conn(0x7f63e006bbe0 legacy=0x7f63e006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:31.733 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.727+0000 7f63ddd9b700 1 -- 172.21.15.77:0/610278851 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e0158bb0 con 0x7f63e006bfb0 2022-04-03T19:53:31.733 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.727+0000 7f63ded9d700 1 --2- 172.21.15.77:0/610278851 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63e006c500 0x7f63e014eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:53:31.734 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.727+0000 7f63ddd9b700 1 --2- 172.21.15.77:0/610278851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63e006bfb0 0x7f63e01584b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f63d4009760 tx=0x7f63d4009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a79503ad31f75018 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:31.734 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.728+0000 7f63dd59a700 1 -- 172.21.15.77:0/610278851 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d40105b0 con 0x7f63e006bfb0 2022-04-03T19:53:31.734 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.728+0000 7f63dd59a700 1 -- 172.21.15.77:0/610278851 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d4010710 con 0x7f63e006bfb0 2022-04-03T19:53:31.734 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.728+0000 7f63dd59a700 1 -- 172.21.15.77:0/610278851 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d4010ab0 con 0x7f63e006bfb0 2022-04-03T19:53:31.735 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.728+0000 7f63df59e700 1 -- 172.21.15.77:0/610278851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63e006bfb0 msgr2=0x7f63e01584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:31.735 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.728+0000 7f63df59e700 1 --2- 172.21.15.77:0/610278851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63e006bfb0 0x7f63e01584b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f63d4009760 tx=0x7f63d4009060 comp rx=0 tx=0).stop 2022-04-03T19:53:31.735 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.728+0000 7f63df59e700 1 -- 172.21.15.77:0/610278851 shutdown_connections 2022-04-03T19:53:31.735 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.728+0000 7f63df59e700 1 --2- 172.21.15.77:0/610278851 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63e006c500 0x7f63e014eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:31.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.728+0000 7f63df59e700 1 --2- 172.21.15.77:0/610278851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63e006bfb0 0x7f63e01584b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:31.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.728+0000 7f63df59e700 1 -- 172.21.15.77:0/610278851 >> 172.21.15.77:0/610278851 conn(0x7f63e00cb6b0 msgr2=0x7f63e00cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:31.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.728+0000 7f63df59e700 1 -- 172.21.15.77:0/610278851 shutdown_connections 2022-04-03T19:53:31.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.728+0000 7f63df59e700 1 -- 172.21.15.77:0/610278851 wait complete. 2022-04-03T19:53:31.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.730+0000 7f63df59e700 1 Processor -- start 2022-04-03T19:53:31.740 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.730+0000 7f63df59e700 1 -- start start 2022-04-03T19:53:31.740 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.731+0000 7f63df59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63e006bbe0 0x7f63e00c07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:31.740 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.731+0000 7f63df59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63e006bfb0 0x7f63e0152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:31.740 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.731+0000 7f63df59e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63e006c500 0x7f63e00bf420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:31.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.731+0000 7f63df59e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0153a50 con 0x7f63e006bbe0 2022-04-03T19:53:31.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.731+0000 7f63df59e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00c30c0 con 0x7f63e006c500 2022-04-03T19:53:31.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.731+0000 7f63df59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0168550 con 0x7f63e006bfb0 2022-04-03T19:53:31.741 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.731+0000 7f63de59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63e006bbe0 0x7f63e00c07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:31.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.731+0000 7f63ddd9b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63e006bfb0 0x7f63e0152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:31.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.731+0000 7f63ded9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63e006c500 0x7f63e00bf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:31.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.731+0000 7f63ddd9b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63e006bfb0 0x7f63e0152e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60364/0 (socket says 172.21.15.77:60364) 2022-04-03T19:53:31.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.731+0000 7f63ddd9b700 1 -- 172.21.15.77:0/33931293 learned_addr learned my addr 172.21.15.77:0/33931293 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:31.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.732+0000 7f63ded9d700 1 -- 172.21.15.77:0/33931293 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63e006bfb0 msgr2=0x7f63e0152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:31.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.732+0000 7f63ded9d700 1 --2- 172.21.15.77:0/33931293 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63e006bfb0 0x7f63e0152e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:31.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.732+0000 7f63ded9d700 1 -- 172.21.15.77:0/33931293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63e006bbe0 msgr2=0x7f63e00c07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:31.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.732+0000 7f63ded9d700 1 --2- 172.21.15.77:0/33931293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63e006bbe0 0x7f63e00c07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:31.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.732+0000 7f63ded9d700 1 -- 172.21.15.77:0/33931293 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d8009a20 con 0x7f63e006c500 2022-04-03T19:53:31.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.732+0000 7f63ddd9b700 1 --2- 172.21.15.77:0/33931293 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63e006bfb0 0x7f63e0152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:53:31.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.732+0000 7f63de59c700 1 --2- 172.21.15.77:0/33931293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63e006bbe0 0x7f63e00c07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:53:31.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.732+0000 7f63ded9d700 1 --2- 172.21.15.77:0/33931293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63e006c500 0x7f63e00bf420 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f63d8009f10 tx=0x7f63d800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:31.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.732+0000 7f63ce7fc700 1 -- 172.21.15.77:0/33931293 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d800cc00 con 0x7f63e006c500 2022-04-03T19:53:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.732+0000 7f63ce7fc700 1 -- 172.21.15.77:0/33931293 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d800f040 con 0x7f63e006c500 2022-04-03T19:53:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.732+0000 7f63df59e700 1 -- 172.21.15.77:0/33931293 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d40092a0 con 0x7f63e006c500 2022-04-03T19:53:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.733+0000 7f63ce7fc700 1 -- 172.21.15.77:0/33931293 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d8003860 con 0x7f63e006c500 2022-04-03T19:53:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.733+0000 7f63df59e700 1 -- 172.21.15.77:0/33931293 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e0168b10 con 0x7f63e006c500 2022-04-03T19:53:31.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.734+0000 7f63df59e700 1 -- 172.21.15.77:0/33931293 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e004b050 con 0x7f63e006c500 2022-04-03T19:53:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.735+0000 7f63ce7fc700 1 -- 172.21.15.77:0/33931293 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f63d8014480 con 0x7f63e006c500 2022-04-03T19:53:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.735+0000 7f63ce7fc700 1 --2- 172.21.15.77:0/33931293 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f63bc063ce0 0x7f63bc066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.736+0000 7f63ce7fc700 1 -- 172.21.15.77:0/33931293 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f63d8082170 con 0x7f63e006c500 2022-04-03T19:53:31.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.736+0000 7f63de59c700 1 --2- 172.21.15.77:0/33931293 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f63bc063ce0 0x7f63bc066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.736+0000 7f63de59c700 1 --2- 172.21.15.77:0/33931293 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f63bc063ce0 0x7f63bc066190 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f63d0007fd0 tx=0x7f63d0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:31.747 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.739+0000 7f63ce7fc700 1 -- 172.21.15.77:0/33931293 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f63d8050a90 con 0x7f63e006c500 2022-04-03T19:53:31.972 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.971+0000 7f63df59e700 1 -- 172.21.15.77:0/33931293 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63e00c7ec0 con 0x7f63e006c500 2022-04-03T19:53:31.973 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.971+0000 7f63ce7fc700 1 -- 172.21.15.77:0/33931293 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5356 (secure 0 0 0) 0x7f63d80540b0 con 0x7f63e006c500 2022-04-03T19:53:31.973 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:53:31.973 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:53:28.279103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5039":{"gid":5039,"name":"d","rank":0,"incarnation":32,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:53:31.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.974+0000 7f63df59e700 1 -- 172.21.15.77:0/33931293 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f63bc063ce0 msgr2=0x7f63bc066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:31.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.974+0000 7f63df59e700 1 --2- 172.21.15.77:0/33931293 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f63bc063ce0 0x7f63bc066190 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f63d0007fd0 tx=0x7f63d0009040 comp rx=0 tx=0).stop 2022-04-03T19:53:31.976 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.974+0000 7f63df59e700 1 -- 172.21.15.77:0/33931293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63e006c500 msgr2=0x7f63e00bf420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:31.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.974+0000 7f63df59e700 1 --2- 172.21.15.77:0/33931293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63e006c500 0x7f63e00bf420 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f63d8009f10 tx=0x7f63d800b040 comp rx=0 tx=0).stop 2022-04-03T19:53:31.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.974+0000 7f63df59e700 1 -- 172.21.15.77:0/33931293 shutdown_connections 2022-04-03T19:53:31.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.974+0000 7f63df59e700 1 --2- 172.21.15.77:0/33931293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63e006c500 0x7f63e00bf420 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:31.977 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.974+0000 7f63df59e700 1 --2- 172.21.15.77:0/33931293 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63e006bbe0 0x7f63e00c07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:31.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.974+0000 7f63df59e700 1 --2- 172.21.15.77:0/33931293 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f63bc063ce0 0x7f63bc066190 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:31.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.974+0000 7f63df59e700 1 --2- 172.21.15.77:0/33931293 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63e006bfb0 0x7f63e0152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:31.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.974+0000 7f63df59e700 1 -- 172.21.15.77:0/33931293 >> 172.21.15.77:0/33931293 conn(0x7f63e00cb6b0 msgr2=0x7f63e00c70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:31.978 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.974+0000 7f63df59e700 1 -- 172.21.15.77:0/33931293 shutdown_connections 2022-04-03T19:53:31.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:31.974+0000 7f63df59e700 1 -- 172.21.15.77:0/33931293 wait complete. 2022-04-03T19:53:31.979 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 33 2022-04-03T19:53:32.001 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:53:34.002 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:53:34.209 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.207+0000 7fe187fff700 1 Processor -- start 2022-04-03T19:53:34.209 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.207+0000 7fe187fff700 1 -- start start 2022-04-03T19:53:34.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.208+0000 7fe187fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe18806ccf0 0x7fe1880aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:34.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.209+0000 7fe187fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe188066ed0 con 0x7fe1880c5760 2022-04-03T19:53:34.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.209+0000 7fe187fff700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe18806e380 con 0x7fe188067a70 2022-04-03T19:53:34.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.209+0000 7fe187fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18806d930 con 0x7fe18806ccf0 2022-04-03T19:53:34.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.209+0000 7fe1867fc700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe1880c5760 0x7fe188067360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38526/0 (socket says 172.21.15.77:38526) 2022-04-03T19:53:34.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.209+0000 7fe1867fc700 1 -- 172.21.15.77:0/1399947643 learned_addr learned my addr 172.21.15.77:0/1399947643 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:53:34.212 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.209+0000 7fe186ffd700 1 --2- 172.21.15.77:0/1399947643 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe18806ccf0 0x7fe1880aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:34.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.209+0000 7fe185ffb700 1 -- 172.21.15.77:0/1399947643 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797655324 0 0) 0x7fe188066ed0 con 0x7fe1880c5760 2022-04-03T19:53:34.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.209+0000 7fe185ffb700 1 -- 172.21.15.77:0/1399947643 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1740033f0 con 0x7fe1880c5760 2022-04-03T19:53:34.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.209+0000 7fe185ffb700 1 -- 172.21.15.77:0/1399947643 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568868809 0 0) 0x7fe18806e380 con 0x7fe188067a70 2022-04-03T19:53:34.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.209+0000 7fe185ffb700 1 -- 172.21.15.77:0/1399947643 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe188066ed0 con 0x7fe188067a70 2022-04-03T19:53:34.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.210+0000 7fe186ffd700 1 -- 172.21.15.77:0/1399947643 >> v1:172.21.15.186:6789/0 conn(0x7fe188067a70 legacy=0x7fe18815fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:34.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.210+0000 7fe186ffd700 1 -- 172.21.15.77:0/1399947643 >> v1:172.21.15.77:6789/0 conn(0x7fe1880c5760 legacy=0x7fe188067360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:34.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.210+0000 7fe186ffd700 1 -- 172.21.15.77:0/1399947643 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1880c6790 con 0x7fe18806ccf0 2022-04-03T19:53:34.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.210+0000 7fe186ffd700 1 --2- 172.21.15.77:0/1399947643 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe18806ccf0 0x7fe1880aeb20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe178009790 tx=0x7fe178009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2da4ed8e977a0f8 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:34.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.210+0000 7fe185ffb700 1 -- 172.21.15.77:0/1399947643 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178011630 con 0x7fe18806ccf0 2022-04-03T19:53:34.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.210+0000 7fe185ffb700 1 -- 172.21.15.77:0/1399947643 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe178011790 con 0x7fe18806ccf0 2022-04-03T19:53:34.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.210+0000 7fe185ffb700 1 -- 172.21.15.77:0/1399947643 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178011b50 con 0x7fe18806ccf0 2022-04-03T19:53:34.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.211+0000 7fe187fff700 1 -- 172.21.15.77:0/1399947643 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe18806ccf0 msgr2=0x7fe1880aeb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:34.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.211+0000 7fe187fff700 1 --2- 172.21.15.77:0/1399947643 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe18806ccf0 0x7fe1880aeb20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe178009790 tx=0x7fe178009060 comp rx=0 tx=0).stop 2022-04-03T19:53:34.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.211+0000 7fe187fff700 1 -- 172.21.15.77:0/1399947643 shutdown_connections 2022-04-03T19:53:34.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.211+0000 7fe187fff700 1 --2- 172.21.15.77:0/1399947643 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe18806ccf0 0x7fe1880aeb20 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:34.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.211+0000 7fe187fff700 1 -- 172.21.15.77:0/1399947643 >> 172.21.15.77:0/1399947643 conn(0x7fe188069990 msgr2=0x7fe1880626d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:34.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.211+0000 7fe187fff700 1 -- 172.21.15.77:0/1399947643 shutdown_connections 2022-04-03T19:53:34.218 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.211+0000 7fe187fff700 1 -- 172.21.15.77:0/1399947643 wait complete. 2022-04-03T19:53:34.218 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.212+0000 7fe187fff700 1 Processor -- start 2022-04-03T19:53:34.218 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.213+0000 7fe187fff700 1 -- start start 2022-04-03T19:53:34.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.213+0000 7fe187fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe188067a70 0x7fe188151400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:34.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.213+0000 7fe187fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe18806ccf0 0x7fe188155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:34.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.213+0000 7fe187fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1880c5760 0x7fe1881688a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:34.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.213+0000 7fe187fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe188150f20 con 0x7fe18806ccf0 2022-04-03T19:53:34.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.213+0000 7fe187fff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18814f6a0 con 0x7fe188067a70 2022-04-03T19:53:34.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.213+0000 7fe187fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe188168410 con 0x7fe1880c5760 2022-04-03T19:53:34.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.214+0000 7fe1867fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe18806ccf0 0x7fe188155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:34.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.214+0000 7fe186ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe188067a70 0x7fe188151400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:34.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.214+0000 7fe1877fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1880c5760 0x7fe1881688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:34.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.214+0000 7fe186ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe188067a70 0x7fe188151400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38274/0 (socket says 172.21.15.77:38274) 2022-04-03T19:53:34.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.214+0000 7fe186ffd700 1 -- 172.21.15.77:0/620921557 learned_addr learned my addr 172.21.15.77:0/620921557 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:34.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.214+0000 7fe186ffd700 1 -- 172.21.15.77:0/620921557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1880c5760 msgr2=0x7fe1881688a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:34.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.214+0000 7fe186ffd700 1 --2- 172.21.15.77:0/620921557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1880c5760 0x7fe1881688a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:34.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.214+0000 7fe186ffd700 1 -- 172.21.15.77:0/620921557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe18806ccf0 msgr2=0x7fe188155ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:34.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.214+0000 7fe186ffd700 1 --2- 172.21.15.77:0/620921557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe18806ccf0 0x7fe188155ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:34.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.214+0000 7fe186ffd700 1 -- 172.21.15.77:0/620921557 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1780092a0 con 0x7fe188067a70 2022-04-03T19:53:34.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.214+0000 7fe186ffd700 1 --2- 172.21.15.77:0/620921557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe188067a70 0x7fe188151400 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe178009160 tx=0x7fe178017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:34.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.215+0000 7fe16effd700 1 -- 172.21.15.77:0/620921557 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178011900 con 0x7fe188067a70 2022-04-03T19:53:34.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.215+0000 7fe16effd700 1 -- 172.21.15.77:0/620921557 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe178011a60 con 0x7fe188067a70 2022-04-03T19:53:34.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.215+0000 7fe16effd700 1 -- 172.21.15.77:0/620921557 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178023630 con 0x7fe188067a70 2022-04-03T19:53:34.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.215+0000 7fe187fff700 1 -- 172.21.15.77:0/620921557 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe188151940 con 0x7fe188067a70 2022-04-03T19:53:34.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.215+0000 7fe187fff700 1 -- 172.21.15.77:0/620921557 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe188150680 con 0x7fe188067a70 2022-04-03T19:53:34.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.216+0000 7fe187fff700 1 -- 172.21.15.77:0/620921557 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe188069f50 con 0x7fe188067a70 2022-04-03T19:53:34.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.217+0000 7fe16effd700 1 -- 172.21.15.77:0/620921557 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fe17800f050 con 0x7fe188067a70 2022-04-03T19:53:34.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.218+0000 7fe16effd700 1 --2- 172.21.15.77:0/620921557 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe168063ce0 0x7fe168066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:34.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.218+0000 7fe16effd700 1 -- 172.21.15.77:0/620921557 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7fe178081c50 con 0x7fe188067a70 2022-04-03T19:53:34.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.218+0000 7fe1867fc700 1 --2- 172.21.15.77:0/620921557 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe168063ce0 0x7fe168066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:34.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.219+0000 7fe1867fc700 1 --2- 172.21.15.77:0/620921557 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe168063ce0 0x7fe168066190 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe17c005750 tx=0x7fe17c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:34.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.222+0000 7fe16effd700 1 -- 172.21.15.77:0/620921557 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fe178050550 con 0x7fe188067a70 2022-04-03T19:53:34.461 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.459+0000 7fe187fff700 1 -- 172.21.15.77:0/620921557 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1881686f0 con 0x7fe188067a70 2022-04-03T19:53:34.462 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.460+0000 7fe16effd700 1 -- 172.21.15.77:0/620921557 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5353 (secure 0 0 0) 0x7fe178053b70 con 0x7fe188067a70 2022-04-03T19:53:34.462 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:53:34.462 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:53:33.353643+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5039":{"gid":5039,"name":"d","rank":0,"incarnation":32,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:53:34.465 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.463+0000 7fe187fff700 1 -- 172.21.15.77:0/620921557 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe168063ce0 msgr2=0x7fe168066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:34.465 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.463+0000 7fe187fff700 1 --2- 172.21.15.77:0/620921557 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe168063ce0 0x7fe168066190 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe17c005750 tx=0x7fe17c009040 comp rx=0 tx=0).stop 2022-04-03T19:53:34.465 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.463+0000 7fe187fff700 1 -- 172.21.15.77:0/620921557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe188067a70 msgr2=0x7fe188151400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:34.465 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.463+0000 7fe187fff700 1 --2- 172.21.15.77:0/620921557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe188067a70 0x7fe188151400 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe178009160 tx=0x7fe178017000 comp rx=0 tx=0).stop 2022-04-03T19:53:34.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.463+0000 7fe187fff700 1 -- 172.21.15.77:0/620921557 shutdown_connections 2022-04-03T19:53:34.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.463+0000 7fe187fff700 1 --2- 172.21.15.77:0/620921557 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe168063ce0 0x7fe168066190 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:34.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.463+0000 7fe187fff700 1 --2- 172.21.15.77:0/620921557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1880c5760 0x7fe1881688a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:34.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.463+0000 7fe187fff700 1 --2- 172.21.15.77:0/620921557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe188067a70 0x7fe188151400 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:34.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.463+0000 7fe187fff700 1 --2- 172.21.15.77:0/620921557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe18806ccf0 0x7fe188155ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:34.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.463+0000 7fe187fff700 1 -- 172.21.15.77:0/620921557 >> 172.21.15.77:0/620921557 conn(0x7fe188069990 msgr2=0x7fe188156d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:34.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.463+0000 7fe187fff700 1 -- 172.21.15.77:0/620921557 shutdown_connections 2022-04-03T19:53:34.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:34.463+0000 7fe187fff700 1 -- 172.21.15.77:0/620921557 wait complete. 2022-04-03T19:53:34.468 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 34 2022-04-03T19:53:34.489 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 34, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-04-03T19:44:21.275716+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 34, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5039": { "addr": "172.21.15.186:6837/480451242", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 480451242, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 480451242, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 5039, "incarnation": 32, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-04-03T19:53:33.353643+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5039 } } } ], "standbys": [ { "addr": "172.21.15.77:6834/2116223509", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6832", "nonce": 2116223509, "type": "v2" }, { "addr": "172.21.15.77:6834", "nonce": 2116223509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4245, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6836/2374276747", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2374276747, "type": "v2" }, { "addr": "172.21.15.186:6836", "nonce": 2374276747, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4259, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/610990853", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6838", "nonce": 610990853, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 610990853, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4265, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.77:6835/170053412", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6833", "nonce": 170053412, "type": "v2" }, { "addr": "172.21.15.77:6835", "nonce": 170053412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4271, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-04-03T19:53:36.491 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:53:36.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.693+0000 7f6793fff700 1 Processor -- start 2022-04-03T19:53:36.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.693+0000 7f6793fff700 1 -- start start 2022-04-03T19:53:36.696 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.694+0000 7f6793fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f67940c1170 0x7f67941527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:36.696 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.694+0000 7f6793fff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f67940c3b10 0x7f679415c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:36.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.695+0000 7f6793fff700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67940c3710 con 0x7f6794155370 2022-04-03T19:53:36.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.695+0000 7f6793fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6794152330 con 0x7f67940c1170 2022-04-03T19:53:36.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.695+0000 7f6793fff700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67940c1540 con 0x7f67940c3b10 2022-04-03T19:53:36.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.695+0000 7f6792ffd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f67940c1170 0x7f67941527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:36.698 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.695+0000 7f67927fc700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f6794155370 0x7f6794069110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44324/0 (socket says 172.21.15.77:44324) 2022-04-03T19:53:36.698 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.695+0000 7f67927fc700 1 -- 172.21.15.77:0/814646564 learned_addr learned my addr 172.21.15.77:0/814646564 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:53:36.698 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.695+0000 7f67937fe700 1 --2- 172.21.15.77:0/814646564 >> v2:172.21.15.186:3300/0 conn(0x7f67940c3b10 0x7f679415c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:36.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.695+0000 7f6792ffd700 1 -- 172.21.15.77:0/814646564 >> v2:172.21.15.186:3300/0 conn(0x7f67940c3b10 msgr2=0x7f679415c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:53:36.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.695+0000 7f6792ffd700 1 --2- 172.21.15.77:0/814646564 >> v2:172.21.15.186:3300/0 conn(0x7f67940c3b10 0x7f679415c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:36.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.695+0000 7f6792ffd700 1 -- 172.21.15.77:0/814646564 >> v1:172.21.15.186:6789/0 conn(0x7f6794155370 legacy=0x7f6794069110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:36.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.695+0000 7f6791ffb700 1 -- 172.21.15.77:0/814646564 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323569050 0 0) 0x7f67940c3710 con 0x7f6794155370 2022-04-03T19:53:36.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.695+0000 7f6792ffd700 1 -- 172.21.15.77:0/814646564 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67940c4270 con 0x7f67940c1170 2022-04-03T19:53:36.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.696+0000 7f6792ffd700 1 --2- 172.21.15.77:0/814646564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67940c1170 0x7f67941527c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f67940690a0 tx=0x7f6784009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf93566cc5d39435 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:36.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.696+0000 7f6791ffb700 1 -- 172.21.15.77:0/814646564 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784011630 con 0x7f67940c1170 2022-04-03T19:53:36.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.696+0000 7f6791ffb700 1 -- 172.21.15.77:0/814646564 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6784011790 con 0x7f67940c1170 2022-04-03T19:53:36.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.696+0000 7f6791ffb700 1 -- 172.21.15.77:0/814646564 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784011b30 con 0x7f67940c1170 2022-04-03T19:53:36.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.696+0000 7f6793fff700 1 -- 172.21.15.77:0/814646564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67940c1170 msgr2=0x7f67941527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:36.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.696+0000 7f6793fff700 1 --2- 172.21.15.77:0/814646564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67940c1170 0x7f67941527c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f67940690a0 tx=0x7f6784009060 comp rx=0 tx=0).stop 2022-04-03T19:53:36.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.697+0000 7f6793fff700 1 -- 172.21.15.77:0/814646564 shutdown_connections 2022-04-03T19:53:36.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.697+0000 7f6793fff700 1 --2- 172.21.15.77:0/814646564 >> v2:172.21.15.186:3300/0 conn(0x7f67940c3b10 0x7f679415c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:36.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.697+0000 7f6793fff700 1 --2- 172.21.15.77:0/814646564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67940c1170 0x7f67941527c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:36.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.697+0000 7f6793fff700 1 -- 172.21.15.77:0/814646564 >> 172.21.15.77:0/814646564 conn(0x7f6794066da0 msgr2=0x7f67940671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:36.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.697+0000 7f6793fff700 1 -- 172.21.15.77:0/814646564 shutdown_connections 2022-04-03T19:53:36.704 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.697+0000 7f6793fff700 1 -- 172.21.15.77:0/814646564 wait complete. 2022-04-03T19:53:36.704 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.698+0000 7f6793fff700 1 Processor -- start 2022-04-03T19:53:36.704 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.698+0000 7f6793fff700 1 -- start start 2022-04-03T19:53:36.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.699+0000 7f6793fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67940c1170 0x7f67940b9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:36.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.699+0000 7f6793fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67940c3b10 0x7f6794168f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:36.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.699+0000 7f6793fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6794155370 0x7f67941694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:36.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.699+0000 7f6793fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6794154fc0 con 0x7f67940c3b10 2022-04-03T19:53:36.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.699+0000 7f6793fff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6794153fb0 con 0x7f67940c1170 2022-04-03T19:53:36.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.699+0000 7f6793fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67940c7c10 con 0x7f6794155370 2022-04-03T19:53:36.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.700+0000 7f67927fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67940c3b10 0x7f6794168f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:36.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.700+0000 7f67937fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6794155370 0x7f67941694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:36.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.700+0000 7f67927fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67940c3b10 0x7f6794168f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43084/0 (socket says 172.21.15.77:43084) 2022-04-03T19:53:36.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.700+0000 7f6792ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67940c1170 0x7f67940b9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:36.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.700+0000 7f67927fc700 1 -- 172.21.15.77:0/2479566690 learned_addr learned my addr 172.21.15.77:0/2479566690 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:36.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.700+0000 7f67927fc700 1 -- 172.21.15.77:0/2479566690 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6794155370 msgr2=0x7f67941694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:36.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.700+0000 7f67927fc700 1 --2- 172.21.15.77:0/2479566690 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6794155370 0x7f67941694a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:36.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.700+0000 7f67927fc700 1 -- 172.21.15.77:0/2479566690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67940c1170 msgr2=0x7f67940b9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:36.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.700+0000 7f67927fc700 1 --2- 172.21.15.77:0/2479566690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67940c1170 0x7f67940b9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:36.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.700+0000 7f67927fc700 1 -- 172.21.15.77:0/2479566690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67840092a0 con 0x7f67940c3b10 2022-04-03T19:53:36.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.701+0000 7f67927fc700 1 --2- 172.21.15.77:0/2479566690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67940c3b10 0x7f6794168f60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f678800c440 tx=0x7f6788007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:36.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.701+0000 7f677affd700 1 -- 172.21.15.77:0/2479566690 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6788007360 con 0x7f67940c3b10 2022-04-03T19:53:36.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.701+0000 7f677affd700 1 -- 172.21.15.77:0/2479566690 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67880074c0 con 0x7f67940c3b10 2022-04-03T19:53:36.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.701+0000 7f6793fff700 1 -- 172.21.15.77:0/2479566690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67940c7eb0 con 0x7f67940c3b10 2022-04-03T19:53:36.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.701+0000 7f677affd700 1 -- 172.21.15.77:0/2479566690 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6788004900 con 0x7f67940c3b10 2022-04-03T19:53:36.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.701+0000 7f6793fff700 1 -- 172.21.15.77:0/2479566690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67941544e0 con 0x7f67940c3b10 2022-04-03T19:53:36.712 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.702+0000 7f6793fff700 1 -- 172.21.15.77:0/2479566690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67940c8040 con 0x7f67940c3b10 2022-04-03T19:53:36.712 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.703+0000 7f677affd700 1 -- 172.21.15.77:0/2479566690 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f6788010030 con 0x7f67940c3b10 2022-04-03T19:53:36.712 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.704+0000 7f677affd700 1 --2- 172.21.15.77:0/2479566690 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6774063d10 0x7f67740661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:36.712 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.704+0000 7f677affd700 1 -- 172.21.15.77:0/2479566690 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f67880803f0 con 0x7f67940c3b10 2022-04-03T19:53:36.712 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.704+0000 7f6792ffd700 1 --2- 172.21.15.77:0/2479566690 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6774063d10 0x7f67740661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:36.713 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.705+0000 7f6792ffd700 1 --2- 172.21.15.77:0/2479566690 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6774063d10 0x7f67740661c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6784011fd0 tx=0x7f6784017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:36.713 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.709+0000 7f677affd700 1 -- 172.21.15.77:0/2479566690 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f678804ee60 con 0x7f67940c3b10 2022-04-03T19:53:36.944 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.942+0000 7f6793fff700 1 -- 172.21.15.77:0/2479566690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67941546d0 con 0x7f67940c3b10 2022-04-03T19:53:36.945 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.943+0000 7f677affd700 1 -- 172.21.15.77:0/2479566690 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5353 (secure 0 0 0) 0x7f6788052480 con 0x7f67940c3b10 2022-04-03T19:53:36.945 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:53:36.945 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:53:33.353643+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5039":{"gid":5039,"name":"d","rank":0,"incarnation":32,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:53:36.947 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.945+0000 7f6793fff700 1 -- 172.21.15.77:0/2479566690 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6774063d10 msgr2=0x7f67740661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:36.947 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.945+0000 7f6793fff700 1 --2- 172.21.15.77:0/2479566690 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6774063d10 0x7f67740661c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6784011fd0 tx=0x7f6784017000 comp rx=0 tx=0).stop 2022-04-03T19:53:36.948 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.945+0000 7f6793fff700 1 -- 172.21.15.77:0/2479566690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67940c3b10 msgr2=0x7f6794168f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:36.948 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.945+0000 7f6793fff700 1 --2- 172.21.15.77:0/2479566690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67940c3b10 0x7f6794168f60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f678800c440 tx=0x7f6788007d90 comp rx=0 tx=0).stop 2022-04-03T19:53:36.948 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.945+0000 7f6793fff700 1 -- 172.21.15.77:0/2479566690 shutdown_connections 2022-04-03T19:53:36.948 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.945+0000 7f6793fff700 1 --2- 172.21.15.77:0/2479566690 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f6774063d10 0x7f67740661c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:36.949 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.946+0000 7f6793fff700 1 --2- 172.21.15.77:0/2479566690 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6794155370 0x7f67941694a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:36.949 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.946+0000 7f6793fff700 1 --2- 172.21.15.77:0/2479566690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67940c1170 0x7f67940b9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:36.949 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.946+0000 7f6793fff700 1 --2- 172.21.15.77:0/2479566690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67940c3b10 0x7f6794168f60 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:36.949 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.946+0000 7f6793fff700 1 -- 172.21.15.77:0/2479566690 >> 172.21.15.77:0/2479566690 conn(0x7f6794066da0 msgr2=0x7f679406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:36.949 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.946+0000 7f6793fff700 1 -- 172.21.15.77:0/2479566690 shutdown_connections 2022-04-03T19:53:36.950 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:36.946+0000 7f6793fff700 1 -- 172.21.15.77:0/2479566690 wait complete. 2022-04-03T19:53:36.950 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 34 2022-04-03T19:53:36.972 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:53:38.973 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:53:39.179 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.179+0000 7fe01cbcb700 1 Processor -- start 2022-04-03T19:53:39.180 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.179+0000 7fe01cbcb700 1 -- start start 2022-04-03T19:53:39.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.180+0000 7fe01cbcb700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe01806bbe0 0x7fe0181584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:39.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.180+0000 7fe01cbcb700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe01806c500 0x7fe01814eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:39.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.180+0000 7fe01cbcb700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01806d260 con 0x7fe01806bfb0 2022-04-03T19:53:39.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.180+0000 7fe01cbcb700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe018159450 con 0x7fe01806bbe0 2022-04-03T19:53:39.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.180+0000 7fe01cbcb700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0180c6330 con 0x7fe01806c500 2022-04-03T19:53:39.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.180+0000 7fe016ffd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe01806bfb0 0x7fe01806d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38528/0 (socket says 172.21.15.77:38528) 2022-04-03T19:53:39.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.180+0000 7fe016ffd700 1 -- 172.21.15.77:0/501096539 learned_addr learned my addr 172.21.15.77:0/501096539 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:53:39.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.180+0000 7fe0177fe700 1 --2- 172.21.15.77:0/501096539 >> v2:172.21.15.186:3300/0 conn(0x7fe01806bbe0 0x7fe0181584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:39.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.180+0000 7fe017fff700 1 --2- 172.21.15.77:0/501096539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe01806c500 0x7fe01814eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:39.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.180+0000 7fe0167fc700 1 -- 172.21.15.77:0/501096539 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597426952 0 0) 0x7fe01806d260 con 0x7fe01806bfb0 2022-04-03T19:53:39.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.180+0000 7fe0167fc700 1 -- 172.21.15.77:0/501096539 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0040033f0 con 0x7fe01806bfb0 2022-04-03T19:53:39.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.180+0000 7fe0177fe700 1 -- 172.21.15.77:0/501096539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe01806c500 msgr2=0x7fe01814eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:53:39.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.180+0000 7fe0177fe700 1 --2- 172.21.15.77:0/501096539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe01806c500 0x7fe01814eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:39.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.180+0000 7fe0177fe700 1 -- 172.21.15.77:0/501096539 >> v1:172.21.15.77:6789/0 conn(0x7fe01806bfb0 legacy=0x7fe01806d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:39.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.180+0000 7fe0177fe700 1 -- 172.21.15.77:0/501096539 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe018158bb0 con 0x7fe01806bbe0 2022-04-03T19:53:39.187 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.180+0000 7fe017fff700 1 --2- 172.21.15.77:0/501096539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe01806c500 0x7fe01814eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:53:39.187 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.180+0000 7fe0177fe700 1 --2- 172.21.15.77:0/501096539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe01806bbe0 0x7fe0181584b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe008009760 tx=0x7fe008009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6161b18aa418ddab server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:39.187 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.181+0000 7fe0167fc700 1 -- 172.21.15.77:0/501096539 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0080105b0 con 0x7fe01806bbe0 2022-04-03T19:53:39.187 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.181+0000 7fe0167fc700 1 -- 172.21.15.77:0/501096539 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe008010710 con 0x7fe01806bbe0 2022-04-03T19:53:39.188 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.181+0000 7fe0167fc700 1 -- 172.21.15.77:0/501096539 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe008010ab0 con 0x7fe01806bbe0 2022-04-03T19:53:39.188 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.181+0000 7fe01cbcb700 1 -- 172.21.15.77:0/501096539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe01806bbe0 msgr2=0x7fe0181584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:39.188 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.181+0000 7fe01cbcb700 1 --2- 172.21.15.77:0/501096539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe01806bbe0 0x7fe0181584b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe008009760 tx=0x7fe008009060 comp rx=0 tx=0).stop 2022-04-03T19:53:39.188 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.181+0000 7fe01cbcb700 1 -- 172.21.15.77:0/501096539 shutdown_connections 2022-04-03T19:53:39.188 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.181+0000 7fe01cbcb700 1 --2- 172.21.15.77:0/501096539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe01806c500 0x7fe01814eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:39.189 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.181+0000 7fe01cbcb700 1 --2- 172.21.15.77:0/501096539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe01806bbe0 0x7fe0181584b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:39.189 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.181+0000 7fe01cbcb700 1 -- 172.21.15.77:0/501096539 >> 172.21.15.77:0/501096539 conn(0x7fe0180cb6b0 msgr2=0x7fe0180cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:39.189 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.181+0000 7fe01cbcb700 1 -- 172.21.15.77:0/501096539 shutdown_connections 2022-04-03T19:53:39.189 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.181+0000 7fe01cbcb700 1 -- 172.21.15.77:0/501096539 wait complete. 2022-04-03T19:53:39.193 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.184+0000 7fe01cbcb700 1 Processor -- start 2022-04-03T19:53:39.193 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.184+0000 7fe01cbcb700 1 -- start start 2022-04-03T19:53:39.193 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fe01cbcb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe01806bbe0 0x7fe0181657e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:39.194 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fe01cbcb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe01806bfb0 0x7fe0181520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:39.194 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fe01cbcb700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe01806c500 0x7fe018167a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:39.194 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fe01cbcb700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0181539d0 con 0x7fe01806bfb0 2022-04-03T19:53:39.194 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fe01cbcb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe018152600 con 0x7fe01806bbe0 2022-04-03T19:53:39.195 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fe01cbcb700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe018150d40 con 0x7fe01806c500 2022-04-03T19:53:39.195 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fe016ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe01806bfb0 0x7fe0181520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:39.195 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fe016ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe01806bfb0 0x7fe0181520c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43086/0 (socket says 172.21.15.77:43086) 2022-04-03T19:53:39.195 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fe016ffd700 1 -- 172.21.15.77:0/2627309928 learned_addr learned my addr 172.21.15.77:0/2627309928 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:39.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fe017fff700 1 --2- 172.21.15.77:0/2627309928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe01806c500 0x7fe018167a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:39.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fe0177fe700 1 --2- 172.21.15.77:0/2627309928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe01806bbe0 0x7fe0181657e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:39.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fe016ffd700 1 -- 172.21.15.77:0/2627309928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe01806c500 msgr2=0x7fe018167a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:39.197 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fe016ffd700 1 --2- 172.21.15.77:0/2627309928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe01806c500 0x7fe018167a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:39.197 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fe016ffd700 1 -- 172.21.15.77:0/2627309928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe01806bbe0 msgr2=0x7fe0181657e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:39.197 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fe016ffd700 1 --2- 172.21.15.77:0/2627309928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe01806bbe0 0x7fe0181657e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:39.197 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fe016ffd700 1 -- 172.21.15.77:0/2627309928 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe010009a20 con 0x7fe01806bfb0 2022-04-03T19:53:39.197 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fe016ffd700 1 --2- 172.21.15.77:0/2627309928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe01806bfb0 0x7fe0181520c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe00c00c490 tx=0x7fe00c007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:39.198 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fdfff7fe700 1 -- 172.21.15.77:0/2627309928 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c0072e0 con 0x7fe01806bfb0 2022-04-03T19:53:39.198 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fdfff7fe700 1 -- 172.21.15.77:0/2627309928 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00c007460 con 0x7fe01806bfb0 2022-04-03T19:53:39.198 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fe01cbcb700 1 -- 172.21.15.77:0/2627309928 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0080092a0 con 0x7fe01806bfb0 2022-04-03T19:53:39.198 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fdfff7fe700 1 -- 172.21.15.77:0/2627309928 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00c0059d0 con 0x7fe01806bfb0 2022-04-03T19:53:39.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.185+0000 7fe01cbcb700 1 -- 172.21.15.77:0/2627309928 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe018151300 con 0x7fe01806bfb0 2022-04-03T19:53:39.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.186+0000 7fe01cbcb700 1 -- 172.21.15.77:0/2627309928 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01804b050 con 0x7fe01806bfb0 2022-04-03T19:53:39.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.187+0000 7fdfff7fe700 1 -- 172.21.15.77:0/2627309928 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fe00c019030 con 0x7fe01806bfb0 2022-04-03T19:53:39.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.188+0000 7fdfff7fe700 1 --2- 172.21.15.77:0/2627309928 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdff8063ce0 0x7fdff8066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:39.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.189+0000 7fdfff7fe700 1 -- 172.21.15.77:0/2627309928 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7fe00c07fd60 con 0x7fe01806bfb0 2022-04-03T19:53:39.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.189+0000 7fe0177fe700 1 --2- 172.21.15.77:0/2627309928 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdff8063ce0 0x7fdff8066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:39.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.189+0000 7fe0177fe700 1 --2- 172.21.15.77:0/2627309928 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdff8063ce0 0x7fdff8066190 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe008010fd0 tx=0x7fe008016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:39.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.193+0000 7fdfff7fe700 1 -- 172.21.15.77:0/2627309928 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fe00c04e720 con 0x7fe01806bfb0 2022-04-03T19:53:39.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.427+0000 7fe01cbcb700 1 -- 172.21.15.77:0/2627309928 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0180c7ec0 con 0x7fe01806bfb0 2022-04-03T19:53:39.429 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.428+0000 7fdfff7fe700 1 -- 172.21.15.77:0/2627309928 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5359 (secure 0 0 0) 0x7fe00c017020 con 0x7fe01806bfb0 2022-04-03T19:53:39.430 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:53:39.430 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:53:39.378580+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5039":{"gid":5039,"name":"d","rank":0,"incarnation":32,"state":"up:clientreplay","state_seq":25,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:53:39.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.430+0000 7fe01cbcb700 1 -- 172.21.15.77:0/2627309928 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdff8063ce0 msgr2=0x7fdff8066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:39.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.430+0000 7fe01cbcb700 1 --2- 172.21.15.77:0/2627309928 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdff8063ce0 0x7fdff8066190 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe008010fd0 tx=0x7fe008016000 comp rx=0 tx=0).stop 2022-04-03T19:53:39.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.430+0000 7fe01cbcb700 1 -- 172.21.15.77:0/2627309928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe01806bfb0 msgr2=0x7fe0181520c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:39.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.430+0000 7fe01cbcb700 1 --2- 172.21.15.77:0/2627309928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe01806bfb0 0x7fe0181520c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe00c00c490 tx=0x7fe00c007c70 comp rx=0 tx=0).stop 2022-04-03T19:53:39.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.430+0000 7fe01cbcb700 1 -- 172.21.15.77:0/2627309928 shutdown_connections 2022-04-03T19:53:39.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.430+0000 7fe01cbcb700 1 --2- 172.21.15.77:0/2627309928 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdff8063ce0 0x7fdff8066190 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:39.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.430+0000 7fe01cbcb700 1 --2- 172.21.15.77:0/2627309928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe01806c500 0x7fe018167a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:39.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.430+0000 7fe01cbcb700 1 --2- 172.21.15.77:0/2627309928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe01806bbe0 0x7fe0181657e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:39.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.430+0000 7fe01cbcb700 1 --2- 172.21.15.77:0/2627309928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe01806bfb0 0x7fe0181520c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:39.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.430+0000 7fe01cbcb700 1 -- 172.21.15.77:0/2627309928 >> 172.21.15.77:0/2627309928 conn(0x7fe0180cb6b0 msgr2=0x7fe01814a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:39.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.430+0000 7fe01cbcb700 1 -- 172.21.15.77:0/2627309928 shutdown_connections 2022-04-03T19:53:39.435 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:39.431+0000 7fe01cbcb700 1 -- 172.21.15.77:0/2627309928 wait complete. 2022-04-03T19:53:39.435 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 35 2022-04-03T19:53:39.456 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:53:41.457 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:53:41.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.668+0000 7f831e1e2700 1 Processor -- start 2022-04-03T19:53:41.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.668+0000 7f831e1e2700 1 -- start start 2022-04-03T19:53:41.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.669+0000 7f831e1e2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f831806d3f0 0x7f831806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:41.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.669+0000 7f831e1e2700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83181562f0 con 0x7f83180c2a90 2022-04-03T19:53:41.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.669+0000 7f831e1e2700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83180c2f80 con 0x7f83180c5930 2022-04-03T19:53:41.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.669+0000 7f831e1e2700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83180c3260 con 0x7f831806d3f0 2022-04-03T19:53:41.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.670+0000 7f831d9e1700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f83180c2a90 0x7f831806c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38530/0 (socket says 172.21.15.77:38530) 2022-04-03T19:53:41.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.670+0000 7f831d9e1700 1 -- 172.21.15.77:0/3483176 learned_addr learned my addr 172.21.15.77:0/3483176 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:53:41.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.670+0000 7f831d1e0700 1 --2- 172.21.15.77:0/3483176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f831806d3f0 0x7f831806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:41.673 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.670+0000 7f8307fff700 1 -- 172.21.15.77:0/3483176 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740551634 0 0) 0x7f83181562f0 con 0x7f83180c2a90 2022-04-03T19:53:41.673 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.670+0000 7f8307fff700 1 -- 172.21.15.77:0/3483176 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83000033f0 con 0x7f83180c2a90 2022-04-03T19:53:41.673 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.670+0000 7f8307fff700 1 -- 172.21.15.77:0/3483176 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192249797 0 0) 0x7f83180c2f80 con 0x7f83180c5930 2022-04-03T19:53:41.673 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.670+0000 7f8307fff700 1 -- 172.21.15.77:0/3483176 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83181562f0 con 0x7f83180c5930 2022-04-03T19:53:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.670+0000 7f8307fff700 1 -- 172.21.15.77:0/3483176 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2060790230 0 0) 0x7f83181562f0 con 0x7f83180c5930 2022-04-03T19:53:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.671+0000 7f831d1e0700 1 -- 172.21.15.77:0/3483176 >> v1:172.21.15.186:6789/0 conn(0x7f83180c5930 legacy=0x7f8318155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.671+0000 7f831d1e0700 1 -- 172.21.15.77:0/3483176 >> v1:172.21.15.77:6789/0 conn(0x7f83180c2a90 legacy=0x7f831806c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.671+0000 7f831d1e0700 1 -- 172.21.15.77:0/3483176 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83180c3540 con 0x7f831806d3f0 2022-04-03T19:53:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.671+0000 7f831d1e0700 1 --2- 172.21.15.77:0/3483176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f831806d3f0 0x7f831806d7c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8314009790 tx=0x7f8314009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2697dacedcdd9613 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:41.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.671+0000 7f8307fff700 1 -- 172.21.15.77:0/3483176 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8310001770 con 0x7f831806d3f0 2022-04-03T19:53:41.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.671+0000 7f8307fff700 1 -- 172.21.15.77:0/3483176 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8314004030 con 0x7f831806d3f0 2022-04-03T19:53:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.671+0000 7f8307fff700 1 -- 172.21.15.77:0/3483176 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8308001470 con 0x7f831806d3f0 2022-04-03T19:53:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.672+0000 7f831e1e2700 1 -- 172.21.15.77:0/3483176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f831806d3f0 msgr2=0x7f831806d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.672+0000 7f831e1e2700 1 --2- 172.21.15.77:0/3483176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f831806d3f0 0x7f831806d7c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8314009790 tx=0x7f8314009060 comp rx=0 tx=0).stop 2022-04-03T19:53:41.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.672+0000 7f831e1e2700 1 -- 172.21.15.77:0/3483176 shutdown_connections 2022-04-03T19:53:41.680 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.672+0000 7f831e1e2700 1 --2- 172.21.15.77:0/3483176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f831806d3f0 0x7f831806d7c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:41.680 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.672+0000 7f831e1e2700 1 -- 172.21.15.77:0/3483176 >> 172.21.15.77:0/3483176 conn(0x7f83180cb720 msgr2=0x7f83180cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:41.680 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.672+0000 7f831e1e2700 1 -- 172.21.15.77:0/3483176 shutdown_connections 2022-04-03T19:53:41.680 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.672+0000 7f831e1e2700 1 -- 172.21.15.77:0/3483176 wait complete. 2022-04-03T19:53:41.680 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.673+0000 7f831e1e2700 1 Processor -- start 2022-04-03T19:53:41.681 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.673+0000 7f831e1e2700 1 -- start start 2022-04-03T19:53:41.681 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.674+0000 7f831e1e2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f831806d3f0 0x7f8318165a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:41.681 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.674+0000 7f831e1e2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83180c2a90 0x7f83180c0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:41.681 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.674+0000 7f831e1e2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83180c5930 0x7f8318158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:41.682 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.674+0000 7f831e1e2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83180bf6d0 con 0x7f83180c2a90 2022-04-03T19:53:41.682 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.674+0000 7f831e1e2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83180c1090 con 0x7f831806d3f0 2022-04-03T19:53:41.682 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.674+0000 7f831e1e2700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83180c1370 con 0x7f83180c5930 2022-04-03T19:53:41.682 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.675+0000 7f831c9df700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83180c2a90 0x7f83180c0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:41.683 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.675+0000 7f831d1e0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f831806d3f0 0x7f8318165a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:41.683 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.675+0000 7f831d1e0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f831806d3f0 0x7f8318165a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38284/0 (socket says 172.21.15.77:38284) 2022-04-03T19:53:41.683 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.675+0000 7f831d1e0700 1 -- 172.21.15.77:0/1648599114 learned_addr learned my addr 172.21.15.77:0/1648599114 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:41.683 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.675+0000 7f831d9e1700 1 --2- 172.21.15.77:0/1648599114 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83180c5930 0x7f8318158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:41.684 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.675+0000 7f831c9df700 1 -- 172.21.15.77:0/1648599114 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83180c5930 msgr2=0x7f8318158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:41.684 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.675+0000 7f831c9df700 1 --2- 172.21.15.77:0/1648599114 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83180c5930 0x7f8318158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:41.684 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.675+0000 7f831c9df700 1 -- 172.21.15.77:0/1648599114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f831806d3f0 msgr2=0x7f8318165a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:41.684 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.675+0000 7f831c9df700 1 --2- 172.21.15.77:0/1648599114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f831806d3f0 0x7f8318165a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:41.684 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.675+0000 7f831c9df700 1 -- 172.21.15.77:0/1648599114 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83140092a0 con 0x7f83180c2a90 2022-04-03T19:53:41.685 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.675+0000 7f831c9df700 1 --2- 172.21.15.77:0/1648599114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83180c2a90 0x7f83180c0a70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f830800c460 tx=0x7f83080014e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:41.685 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.676+0000 7f83057fa700 1 -- 172.21.15.77:0/1648599114 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f830800ce40 con 0x7f83180c2a90 2022-04-03T19:53:41.685 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.676+0000 7f83057fa700 1 -- 172.21.15.77:0/1648599114 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83080052e0 con 0x7f83180c2a90 2022-04-03T19:53:41.686 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.676+0000 7f83057fa700 1 -- 172.21.15.77:0/1648599114 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8308003710 con 0x7f83180c2a90 2022-04-03T19:53:41.686 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.676+0000 7f831e1e2700 1 -- 172.21.15.77:0/1648599114 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83180c1650 con 0x7f83180c2a90 2022-04-03T19:53:41.687 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.676+0000 7f831e1e2700 1 -- 172.21.15.77:0/1648599114 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83180bf920 con 0x7f83180c2a90 2022-04-03T19:53:41.687 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.677+0000 7f831e1e2700 1 -- 172.21.15.77:0/1648599114 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83180bfb40 con 0x7f83180c2a90 2022-04-03T19:53:41.687 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.678+0000 7f83057fa700 1 -- 172.21.15.77:0/1648599114 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f8308019030 con 0x7f83180c2a90 2022-04-03T19:53:41.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.679+0000 7f83057fa700 1 --2- 172.21.15.77:0/1648599114 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f82f8063ce0 0x7f82f8066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:41.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.679+0000 7f83057fa700 1 -- 172.21.15.77:0/1648599114 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f830807fb70 con 0x7f83180c2a90 2022-04-03T19:53:41.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.679+0000 7f831d1e0700 1 --2- 172.21.15.77:0/1648599114 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f82f8063ce0 0x7f82f8066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:41.689 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.680+0000 7f831d1e0700 1 --2- 172.21.15.77:0/1648599114 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f82f8063ce0 0x7f82f8066190 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f83140096e0 tx=0x7f8314016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:41.689 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.683+0000 7f83057fa700 1 -- 172.21.15.77:0/1648599114 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f8308048ae0 con 0x7f83180c2a90 2022-04-03T19:53:41.915 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.914+0000 7f831e1e2700 1 -- 172.21.15.77:0/1648599114 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f831804b050 con 0x7f83180c2a90 2022-04-03T19:53:41.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.915+0000 7f83057fa700 1 -- 172.21.15.77:0/1648599114 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5359 (secure 0 0 0) 0x7f8308017030 con 0x7f83180c2a90 2022-04-03T19:53:41.916 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:53:41.916 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:53:39.378580+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5039":{"gid":5039,"name":"d","rank":0,"incarnation":32,"state":"up:clientreplay","state_seq":25,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:53:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.917+0000 7f831e1e2700 1 -- 172.21.15.77:0/1648599114 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f82f8063ce0 msgr2=0x7f82f8066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.917+0000 7f831e1e2700 1 --2- 172.21.15.77:0/1648599114 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f82f8063ce0 0x7f82f8066190 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f83140096e0 tx=0x7f8314016000 comp rx=0 tx=0).stop 2022-04-03T19:53:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.917+0000 7f831e1e2700 1 -- 172.21.15.77:0/1648599114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83180c2a90 msgr2=0x7f83180c0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.917+0000 7f831e1e2700 1 --2- 172.21.15.77:0/1648599114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83180c2a90 0x7f83180c0a70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f830800c460 tx=0x7f83080014e0 comp rx=0 tx=0).stop 2022-04-03T19:53:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.917+0000 7f831e1e2700 1 -- 172.21.15.77:0/1648599114 shutdown_connections 2022-04-03T19:53:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.917+0000 7f831e1e2700 1 --2- 172.21.15.77:0/1648599114 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f82f8063ce0 0x7f82f8066190 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.917+0000 7f831e1e2700 1 --2- 172.21.15.77:0/1648599114 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83180c5930 0x7f8318158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.917+0000 7f831e1e2700 1 --2- 172.21.15.77:0/1648599114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f831806d3f0 0x7f8318165a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.917+0000 7f831e1e2700 1 --2- 172.21.15.77:0/1648599114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f83180c2a90 0x7f83180c0a70 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.917+0000 7f831e1e2700 1 -- 172.21.15.77:0/1648599114 >> 172.21.15.77:0/1648599114 conn(0x7f83180cb720 msgr2=0x7f83181531f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.917+0000 7f831e1e2700 1 -- 172.21.15.77:0/1648599114 shutdown_connections 2022-04-03T19:53:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:41.917+0000 7f831e1e2700 1 -- 172.21.15.77:0/1648599114 wait complete. 2022-04-03T19:53:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 35 2022-04-03T19:53:41.942 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:53:43.943 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:53:44.149 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.149+0000 7f1694a31700 1 Processor -- start 2022-04-03T19:53:44.150 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.149+0000 7f1694a31700 1 -- start start 2022-04-03T19:53:44.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.150+0000 7f1694a31700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f16900c5930 0x7f1690155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:44.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.150+0000 7f1694a31700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1690156120 0x7f169006c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:44.151 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.150+0000 7f1694a31700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16900c2f80 con 0x7f169006d3f0 2022-04-03T19:53:44.152 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.150+0000 7f1694a31700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16900c3260 con 0x7f16900c5930 2022-04-03T19:53:44.152 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.150+0000 7f1694a31700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16900c3540 con 0x7f1690156120 2022-04-03T19:53:44.152 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.150+0000 7f168ed9d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f169006d3f0 0x7f169006d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38532/0 (socket says 172.21.15.77:38532) 2022-04-03T19:53:44.152 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.150+0000 7f168ed9d700 1 -- 172.21.15.77:0/2816854862 learned_addr learned my addr 172.21.15.77:0/2816854862 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:53:44.152 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.150+0000 7f168e59c700 1 --2- 172.21.15.77:0/2816854862 >> v2:172.21.15.77:3300/0 conn(0x7f16900c5930 0x7f1690155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:44.153 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.151+0000 7f168f59e700 1 --2- 172.21.15.77:0/2816854862 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1690156120 0x7f169006c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:44.157 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.151+0000 7f168e59c700 1 -- 172.21.15.77:0/2816854862 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1690156120 msgr2=0x7f169006c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:53:44.157 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.151+0000 7f168e59c700 1 --2- 172.21.15.77:0/2816854862 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1690156120 0x7f169006c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:44.157 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.151+0000 7f168e59c700 1 -- 172.21.15.77:0/2816854862 >> v1:172.21.15.77:6789/0 conn(0x7f169006d3f0 legacy=0x7f169006d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:44.158 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.151+0000 7f168e59c700 1 -- 172.21.15.77:0/2816854862 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16900c3820 con 0x7f16900c5930 2022-04-03T19:53:44.158 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.151+0000 7f168dd9b700 1 -- 172.21.15.77:0/2816854862 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794970205 0 0) 0x7f16900c2f80 con 0x7f169006d3f0 2022-04-03T19:53:44.158 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.151+0000 7f168e59c700 1 --2- 172.21.15.77:0/2816854862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16900c5930 0x7f1690155be0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1684009790 tx=0x7f1684009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=478a0d77d2cd5b05 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:44.158 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.152+0000 7f168dd9b700 1 -- 172.21.15.77:0/2816854862 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1684011630 con 0x7f16900c5930 2022-04-03T19:53:44.158 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.152+0000 7f168dd9b700 1 -- 172.21.15.77:0/2816854862 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1684011790 con 0x7f16900c5930 2022-04-03T19:53:44.159 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.152+0000 7f168dd9b700 1 -- 172.21.15.77:0/2816854862 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1684011b50 con 0x7f16900c5930 2022-04-03T19:53:44.159 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.152+0000 7f1694a31700 1 -- 172.21.15.77:0/2816854862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16900c5930 msgr2=0x7f1690155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:44.159 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.152+0000 7f1694a31700 1 --2- 172.21.15.77:0/2816854862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16900c5930 0x7f1690155be0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1684009790 tx=0x7f1684009060 comp rx=0 tx=0).stop 2022-04-03T19:53:44.159 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.152+0000 7f1694a31700 1 -- 172.21.15.77:0/2816854862 shutdown_connections 2022-04-03T19:53:44.160 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.152+0000 7f1694a31700 1 --2- 172.21.15.77:0/2816854862 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1690156120 0x7f169006c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:44.160 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.152+0000 7f1694a31700 1 --2- 172.21.15.77:0/2816854862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16900c5930 0x7f1690155be0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:44.160 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.152+0000 7f1694a31700 1 -- 172.21.15.77:0/2816854862 >> 172.21.15.77:0/2816854862 conn(0x7f16900cb720 msgr2=0x7f16900cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:44.160 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.152+0000 7f1694a31700 1 -- 172.21.15.77:0/2816854862 shutdown_connections 2022-04-03T19:53:44.160 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.152+0000 7f1694a31700 1 -- 172.21.15.77:0/2816854862 wait complete. 2022-04-03T19:53:44.161 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.154+0000 7f1694a31700 1 Processor -- start 2022-04-03T19:53:44.161 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.154+0000 7f1694a31700 1 -- start start 2022-04-03T19:53:44.161 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.155+0000 7f1694a31700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f169006d3f0 0x7f1690169390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:44.161 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.155+0000 7f1694a31700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16900c5930 0x7f16900c1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:44.162 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.155+0000 7f1694a31700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1690156120 0x7f169014dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:44.162 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.155+0000 7f1694a31700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16900c2400 con 0x7f16900c5930 2022-04-03T19:53:44.162 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.155+0000 7f1694a31700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16900c2200 con 0x7f1690156120 2022-04-03T19:53:44.162 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.155+0000 7f1694a31700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16900c07e0 con 0x7f169006d3f0 2022-04-03T19:53:44.163 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.155+0000 7f168e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16900c5930 0x7f16900c1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:44.163 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.155+0000 7f168e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16900c5930 0x7f16900c1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43092/0 (socket says 172.21.15.77:43092) 2022-04-03T19:53:44.163 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.155+0000 7f168e59c700 1 -- 172.21.15.77:0/143763765 learned_addr learned my addr 172.21.15.77:0/143763765 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:44.163 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.155+0000 7f168ed9d700 1 --2- 172.21.15.77:0/143763765 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f169006d3f0 0x7f1690169390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:44.163 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.155+0000 7f168f59e700 1 --2- 172.21.15.77:0/143763765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1690156120 0x7f169014dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:44.164 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.156+0000 7f168e59c700 1 -- 172.21.15.77:0/143763765 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f169006d3f0 msgr2=0x7f1690169390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:44.164 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.156+0000 7f168e59c700 1 --2- 172.21.15.77:0/143763765 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f169006d3f0 0x7f1690169390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:44.164 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.156+0000 7f168e59c700 1 -- 172.21.15.77:0/143763765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1690156120 msgr2=0x7f169014dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:44.164 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.156+0000 7f168e59c700 1 --2- 172.21.15.77:0/143763765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1690156120 0x7f169014dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:44.167 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.156+0000 7f168e59c700 1 -- 172.21.15.77:0/143763765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16840092a0 con 0x7f16900c5930 2022-04-03T19:53:44.167 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.156+0000 7f168e59c700 1 --2- 172.21.15.77:0/143763765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16900c5930 0x7f16900c1cc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1684009e30 tx=0x7f1684017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:44.167 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.156+0000 7f1676ffd700 1 -- 172.21.15.77:0/143763765 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1684000c90 con 0x7f16900c5930 2022-04-03T19:53:44.167 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.156+0000 7f1676ffd700 1 -- 172.21.15.77:0/143763765 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1684011900 con 0x7f16900c5930 2022-04-03T19:53:44.168 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.156+0000 7f1694a31700 1 -- 172.21.15.77:0/143763765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16900c0ac0 con 0x7f16900c5930 2022-04-03T19:53:44.168 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.156+0000 7f1676ffd700 1 -- 172.21.15.77:0/143763765 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1684025420 con 0x7f16900c5930 2022-04-03T19:53:44.168 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.157+0000 7f1694a31700 1 -- 172.21.15.77:0/143763765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16900c1080 con 0x7f16900c5930 2022-04-03T19:53:44.169 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.158+0000 7f1694a31700 1 -- 172.21.15.77:0/143763765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169004b050 con 0x7f16900c5930 2022-04-03T19:53:44.170 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.158+0000 7f1676ffd700 1 -- 172.21.15.77:0/143763765 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f168402c030 con 0x7f16900c5930 2022-04-03T19:53:44.170 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.159+0000 7f1676ffd700 1 --2- 172.21.15.77:0/143763765 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1670063d10 0x7f16700661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:44.170 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.159+0000 7f1676ffd700 1 -- 172.21.15.77:0/143763765 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f1684081ec0 con 0x7f16900c5930 2022-04-03T19:53:44.170 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.160+0000 7f168ed9d700 1 --2- 172.21.15.77:0/143763765 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1670063d10 0x7f16700661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:44.171 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.160+0000 7f168ed9d700 1 --2- 172.21.15.77:0/143763765 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1670063d10 0x7f16700661c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f16800047d0 tx=0x7f1680007bb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:44.171 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.164+0000 7f1676ffd700 1 -- 172.21.15.77:0/143763765 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f1684050960 con 0x7f16900c5930 2022-04-03T19:53:44.399 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.399+0000 7f1694a31700 1 -- 172.21.15.77:0/143763765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f169006d010 con 0x7f16900c5930 2022-04-03T19:53:44.400 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.399+0000 7f1676ffd700 1 -- 172.21.15.77:0/143763765 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5359 (secure 0 0 0) 0x7f168402a090 con 0x7f16900c5930 2022-04-03T19:53:44.400 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:53:44.401 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:53:39.378580+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5039":{"gid":5039,"name":"d","rank":0,"incarnation":32,"state":"up:clientreplay","state_seq":25,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:53:44.402 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.402+0000 7f1694a31700 1 -- 172.21.15.77:0/143763765 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1670063d10 msgr2=0x7f16700661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:44.403 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.402+0000 7f1694a31700 1 --2- 172.21.15.77:0/143763765 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1670063d10 0x7f16700661c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f16800047d0 tx=0x7f1680007bb0 comp rx=0 tx=0).stop 2022-04-03T19:53:44.403 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.402+0000 7f1694a31700 1 -- 172.21.15.77:0/143763765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16900c5930 msgr2=0x7f16900c1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:44.403 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.402+0000 7f1694a31700 1 --2- 172.21.15.77:0/143763765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16900c5930 0x7f16900c1cc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1684009e30 tx=0x7f1684017000 comp rx=0 tx=0).stop 2022-04-03T19:53:44.403 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.402+0000 7f1694a31700 1 -- 172.21.15.77:0/143763765 shutdown_connections 2022-04-03T19:53:44.404 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.402+0000 7f1694a31700 1 --2- 172.21.15.77:0/143763765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1690156120 0x7f169014dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:44.404 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.402+0000 7f1694a31700 1 --2- 172.21.15.77:0/143763765 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1670063d10 0x7f16700661c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:44.404 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.402+0000 7f1694a31700 1 --2- 172.21.15.77:0/143763765 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f169006d3f0 0x7f1690169390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:44.404 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.402+0000 7f1694a31700 1 --2- 172.21.15.77:0/143763765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16900c5930 0x7f16900c1cc0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:44.404 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.402+0000 7f1694a31700 1 -- 172.21.15.77:0/143763765 >> 172.21.15.77:0/143763765 conn(0x7f16900cb720 msgr2=0x7f16901544e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:44.405 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.402+0000 7f1694a31700 1 -- 172.21.15.77:0/143763765 shutdown_connections 2022-04-03T19:53:44.405 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:44.402+0000 7f1694a31700 1 -- 172.21.15.77:0/143763765 wait complete. 2022-04-03T19:53:44.405 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 35 2022-04-03T19:53:44.427 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:53:46.428 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:53:46.630 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.629+0000 7f371500e700 1 Processor -- start 2022-04-03T19:53:46.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.629+0000 7f371500e700 1 -- start start 2022-04-03T19:53:46.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.630+0000 7f371500e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f371006bbe0 0x7f37101584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:46.634 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.630+0000 7f371500e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f371006d260 con 0x7f371006c500 2022-04-03T19:53:46.634 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.630+0000 7f371500e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37100c6330 con 0x7f371006bfb0 2022-04-03T19:53:46.634 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.630+0000 7f371500e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3710159450 con 0x7f371006bbe0 2022-04-03T19:53:46.635 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.631+0000 7f371480d700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f371006c500 0x7f371014eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38534/0 (socket says 172.21.15.77:38534) 2022-04-03T19:53:46.635 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.631+0000 7f371480d700 1 -- 172.21.15.77:0/292996753 learned_addr learned my addr 172.21.15.77:0/292996753 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:53:46.636 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.631+0000 7f370ffff700 1 --2- 172.21.15.77:0/292996753 >> v2:172.21.15.186:3300/0 conn(0x7f371006bbe0 0x7f37101584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:46.636 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.631+0000 7f370ffff700 1 -- 172.21.15.77:0/292996753 >> v1:172.21.15.186:6789/0 conn(0x7f371006bfb0 legacy=0x7f371006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:46.637 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.631+0000 7f370ffff700 1 -- 172.21.15.77:0/292996753 >> v1:172.21.15.77:6789/0 conn(0x7f371006c500 legacy=0x7f371014eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:46.644 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.631+0000 7f370f7fe700 1 -- 172.21.15.77:0/292996753 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723902225 0 0) 0x7f371006d260 con 0x7f371006c500 2022-04-03T19:53:46.644 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.631+0000 7f370ffff700 1 -- 172.21.15.77:0/292996753 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3710158bb0 con 0x7f371006bbe0 2022-04-03T19:53:46.644 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.632+0000 7f370ffff700 1 --2- 172.21.15.77:0/292996753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f371006bbe0 0x7f37101584b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f37000096e0 tx=0x7f3700009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a75f6895b41b7496 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:46.645 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.632+0000 7f370f7fe700 1 -- 172.21.15.77:0/292996753 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3700004030 con 0x7f371006bbe0 2022-04-03T19:53:46.645 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.632+0000 7f370f7fe700 1 -- 172.21.15.77:0/292996753 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3700004190 con 0x7f371006bbe0 2022-04-03T19:53:46.653 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.632+0000 7f370f7fe700 1 -- 172.21.15.77:0/292996753 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3700011630 con 0x7f371006bbe0 2022-04-03T19:53:46.654 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.632+0000 7f371500e700 1 -- 172.21.15.77:0/292996753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f371006bbe0 msgr2=0x7f37101584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:46.654 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.632+0000 7f371500e700 1 --2- 172.21.15.77:0/292996753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f371006bbe0 0x7f37101584b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f37000096e0 tx=0x7f3700009060 comp rx=0 tx=0).stop 2022-04-03T19:53:46.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.632+0000 7f371500e700 1 -- 172.21.15.77:0/292996753 shutdown_connections 2022-04-03T19:53:46.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.632+0000 7f371500e700 1 --2- 172.21.15.77:0/292996753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f371006bbe0 0x7f37101584b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:46.656 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.632+0000 7f371500e700 1 -- 172.21.15.77:0/292996753 >> 172.21.15.77:0/292996753 conn(0x7f37100cb6b0 msgr2=0x7f37100cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:46.656 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.633+0000 7f371500e700 1 -- 172.21.15.77:0/292996753 shutdown_connections 2022-04-03T19:53:46.656 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.633+0000 7f371500e700 1 -- 172.21.15.77:0/292996753 wait complete. 2022-04-03T19:53:46.657 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.634+0000 7f371500e700 1 Processor -- start 2022-04-03T19:53:46.657 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.634+0000 7f371500e700 1 -- start start 2022-04-03T19:53:46.658 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.635+0000 7f371500e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f371006bbe0 0x7f3710167af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:46.658 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.635+0000 7f371500e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f371006bfb0 0x7f37101666d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:46.659 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.635+0000 7f371500e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f371006c500 0x7f3710166c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:46.659 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.635+0000 7f371500e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3710168030 con 0x7f371006bbe0 2022-04-03T19:53:46.659 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.635+0000 7f371500e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f371014f880 con 0x7f371006c500 2022-04-03T19:53:46.660 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.635+0000 7f371500e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f371014fad0 con 0x7f371006bfb0 2022-04-03T19:53:46.660 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.635+0000 7f370ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f371006bbe0 0x7f3710167af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:46.660 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.636+0000 7f370ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f371006bbe0 0x7f3710167af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43094/0 (socket says 172.21.15.77:43094) 2022-04-03T19:53:46.660 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.636+0000 7f370ffff700 1 -- 172.21.15.77:0/4019317518 learned_addr learned my addr 172.21.15.77:0/4019317518 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:46.661 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.636+0000 7f371480d700 1 --2- 172.21.15.77:0/4019317518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f371006c500 0x7f3710166c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:46.661 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.636+0000 7f3707fff700 1 --2- 172.21.15.77:0/4019317518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f371006bfb0 0x7f37101666d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:46.662 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.636+0000 7f370ffff700 1 -- 172.21.15.77:0/4019317518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f371006bfb0 msgr2=0x7f37101666d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:46.662 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.636+0000 7f370ffff700 1 --2- 172.21.15.77:0/4019317518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f371006bfb0 0x7f37101666d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:46.662 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.636+0000 7f370ffff700 1 -- 172.21.15.77:0/4019317518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f371006c500 msgr2=0x7f3710166c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:46.663 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.636+0000 7f370ffff700 1 --2- 172.21.15.77:0/4019317518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f371006c500 0x7f3710166c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:46.663 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.636+0000 7f370ffff700 1 -- 172.21.15.77:0/4019317518 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37000092a0 con 0x7f371006bbe0 2022-04-03T19:53:46.663 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.636+0000 7f370ffff700 1 --2- 172.21.15.77:0/4019317518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f371006bbe0 0x7f3710167af0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3700009e30 tx=0x7f3700016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:46.663 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.637+0000 7f370cff9700 1 -- 172.21.15.77:0/4019317518 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3700003770 con 0x7f371006bbe0 2022-04-03T19:53:46.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.637+0000 7f370cff9700 1 -- 172.21.15.77:0/4019317518 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37000038d0 con 0x7f371006bbe0 2022-04-03T19:53:46.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.637+0000 7f371500e700 1 -- 172.21.15.77:0/4019317518 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3710151000 con 0x7f371006bbe0 2022-04-03T19:53:46.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.637+0000 7f370cff9700 1 -- 172.21.15.77:0/4019317518 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3700003c70 con 0x7f371006bbe0 2022-04-03T19:53:46.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.637+0000 7f371500e700 1 -- 172.21.15.77:0/4019317518 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37101515c0 con 0x7f371006bbe0 2022-04-03T19:53:46.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.638+0000 7f371500e700 1 -- 172.21.15.77:0/4019317518 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f371004b050 con 0x7f371006bbe0 2022-04-03T19:53:46.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.639+0000 7f370cff9700 1 -- 172.21.15.77:0/4019317518 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f370002b030 con 0x7f371006bbe0 2022-04-03T19:53:46.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.639+0000 7f370cff9700 1 --2- 172.21.15.77:0/4019317518 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f36f4063ce0 0x7f36f4066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:46.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.640+0000 7f370cff9700 1 -- 172.21.15.77:0/4019317518 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f3700082380 con 0x7f371006bbe0 2022-04-03T19:53:46.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.640+0000 7f3707fff700 1 --2- 172.21.15.77:0/4019317518 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f36f4063ce0 0x7f36f4066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:46.666 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.640+0000 7f3707fff700 1 --2- 172.21.15.77:0/4019317518 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f36f4063ce0 0x7f36f4066190 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f37100c4ee0 tx=0x7f36f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:46.666 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.644+0000 7f370cff9700 1 -- 172.21.15.77:0/4019317518 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f3700050d70 con 0x7f371006bbe0 2022-04-03T19:53:46.797 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:53:46.822 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:53:46.879 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.878+0000 7f371500e700 1 -- 172.21.15.77:0/4019317518 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37100c7ec0 con 0x7f371006bbe0 2022-04-03T19:53:46.880 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.879+0000 7f370cff9700 1 -- 172.21.15.77:0/4019317518 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5353 (secure 0 0 0) 0x7f3700029090 con 0x7f371006bbe0 2022-04-03T19:53:46.880 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:53:46.881 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:53:46.235063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5039":{"gid":5039,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":27,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:53:46.883 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.882+0000 7f371500e700 1 -- 172.21.15.77:0/4019317518 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f36f4063ce0 msgr2=0x7f36f4066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:46.883 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.882+0000 7f371500e700 1 --2- 172.21.15.77:0/4019317518 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f36f4063ce0 0x7f36f4066190 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f37100c4ee0 tx=0x7f36f8009040 comp rx=0 tx=0).stop 2022-04-03T19:53:46.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.882+0000 7f371500e700 1 -- 172.21.15.77:0/4019317518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f371006bbe0 msgr2=0x7f3710167af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:46.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.882+0000 7f371500e700 1 --2- 172.21.15.77:0/4019317518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f371006bbe0 0x7f3710167af0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3700009e30 tx=0x7f3700016000 comp rx=0 tx=0).stop 2022-04-03T19:53:46.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.882+0000 7f371500e700 1 -- 172.21.15.77:0/4019317518 shutdown_connections 2022-04-03T19:53:46.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.882+0000 7f371500e700 1 --2- 172.21.15.77:0/4019317518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f371006c500 0x7f3710166c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:46.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.882+0000 7f371500e700 1 --2- 172.21.15.77:0/4019317518 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f371006bbe0 0x7f3710167af0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:46.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.882+0000 7f371500e700 1 --2- 172.21.15.77:0/4019317518 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f36f4063ce0 0x7f36f4066190 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:46.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.882+0000 7f371500e700 1 --2- 172.21.15.77:0/4019317518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f371006bfb0 0x7f37101666d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:46.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.882+0000 7f371500e700 1 -- 172.21.15.77:0/4019317518 >> 172.21.15.77:0/4019317518 conn(0x7f37100cb6b0 msgr2=0x7f371014b120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:46.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.882+0000 7f371500e700 1 -- 172.21.15.77:0/4019317518 shutdown_connections 2022-04-03T19:53:46.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:46.882+0000 7f371500e700 1 -- 172.21.15.77:0/4019317518 wait complete. 2022-04-03T19:53:46.886 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 36 2022-04-03T19:53:46.908 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=5239 2022-04-03T19:53:46.908 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.e 2022-04-03T19:53:47.909 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2022-04-03T19:53:47.910 INFO:tasks.ceph.mds.e:Restarting daemon 2022-04-03T19:53:47.910 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2022-04-03T19:53:47.915 INFO:tasks.ceph.mds.e:Started 2022-04-03T19:53:47.916 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:53:48.074 INFO:tasks.ceph.mds.e.smithi077.stdout:starting mds.e at 2022-04-03T19:53:48.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.127+0000 7f9e13fff700 1 Processor -- start 2022-04-03T19:53:48.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.127+0000 7f9e13fff700 1 -- start start 2022-04-03T19:53:48.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.128+0000 7f9e13fff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9e1414d0f0 0x7f9e14152f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:48.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.128+0000 7f9e13fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e1414d4c0 0x7f9e140c0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:48.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.128+0000 7f9e13fff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e14066e40 con 0x7f9e14156490 2022-04-03T19:53:48.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.128+0000 7f9e13fff700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e140c1f80 con 0x7f9e1414d0f0 2022-04-03T19:53:48.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.128+0000 7f9e13fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e141537f0 con 0x7f9e1414d4c0 2022-04-03T19:53:48.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.129+0000 7f9e12ffd700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9e1414d0f0 0x7f9e14152f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:48.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.129+0000 7f9e137fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9e14156490 0x7f9e140678e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38536/0 (socket says 172.21.15.77:38536) 2022-04-03T19:53:48.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.129+0000 7f9e137fe700 1 -- 172.21.15.77:0/2890928668 learned_addr learned my addr 172.21.15.77:0/2890928668 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:53:48.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.129+0000 7f9e0bfff700 1 --2- 172.21.15.77:0/2890928668 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e1414d4c0 0x7f9e140c0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:48.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.129+0000 7f9e12ffd700 1 -- 172.21.15.77:0/2890928668 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e1414d4c0 msgr2=0x7f9e140c0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:53:48.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.129+0000 7f9e12ffd700 1 --2- 172.21.15.77:0/2890928668 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e1414d4c0 0x7f9e140c0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:48.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.129+0000 7f9e12ffd700 1 -- 172.21.15.77:0/2890928668 >> v1:172.21.15.77:6789/0 conn(0x7f9e14156490 legacy=0x7f9e140678e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:48.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.129+0000 7f9e12ffd700 1 -- 172.21.15.77:0/2890928668 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e14153ad0 con 0x7f9e1414d0f0 2022-04-03T19:53:48.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.129+0000 7f9e127fc700 1 -- 172.21.15.77:0/2890928668 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998819439 0 0) 0x7f9e14066e40 con 0x7f9e14156490 2022-04-03T19:53:48.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.130+0000 7f9e12ffd700 1 --2- 172.21.15.77:0/2890928668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e1414d0f0 0x7f9e14152f40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9e04009760 tx=0x7f9e04009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98d8c0f099653f58 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:48.137 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.130+0000 7f9e127fc700 1 -- 172.21.15.77:0/2890928668 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e04011630 con 0x7f9e1414d0f0 2022-04-03T19:53:48.138 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.130+0000 7f9e127fc700 1 -- 172.21.15.77:0/2890928668 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e04011790 con 0x7f9e1414d0f0 2022-04-03T19:53:48.138 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.130+0000 7f9e127fc700 1 -- 172.21.15.77:0/2890928668 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e04011b50 con 0x7f9e1414d0f0 2022-04-03T19:53:48.138 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.130+0000 7f9e13fff700 1 -- 172.21.15.77:0/2890928668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e1414d0f0 msgr2=0x7f9e14152f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:48.138 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.130+0000 7f9e13fff700 1 --2- 172.21.15.77:0/2890928668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e1414d0f0 0x7f9e14152f40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9e04009760 tx=0x7f9e04009060 comp rx=0 tx=0).stop 2022-04-03T19:53:48.139 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.131+0000 7f9e13fff700 1 -- 172.21.15.77:0/2890928668 shutdown_connections 2022-04-03T19:53:48.139 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.131+0000 7f9e13fff700 1 --2- 172.21.15.77:0/2890928668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e1414d0f0 0x7f9e14152f40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:48.139 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.131+0000 7f9e13fff700 1 --2- 172.21.15.77:0/2890928668 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e1414d4c0 0x7f9e140c0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:48.139 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.131+0000 7f9e13fff700 1 -- 172.21.15.77:0/2890928668 >> 172.21.15.77:0/2890928668 conn(0x7f9e14069990 msgr2=0x7f9e140626d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:48.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.131+0000 7f9e13fff700 1 -- 172.21.15.77:0/2890928668 shutdown_connections 2022-04-03T19:53:48.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.131+0000 7f9e13fff700 1 -- 172.21.15.77:0/2890928668 wait complete. 2022-04-03T19:53:48.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.132+0000 7f9e13fff700 1 Processor -- start 2022-04-03T19:53:48.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.132+0000 7f9e13fff700 1 -- start start 2022-04-03T19:53:48.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.133+0000 7f9e13fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1414d0f0 0x7f9e141472f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:48.141 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.133+0000 7f9e13fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e1414d4c0 0x7f9e1414bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:48.141 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.133+0000 7f9e13fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e14156490 0x7f9e14146700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:48.141 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.133+0000 7f9e13fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e14164fc0 con 0x7f9e1414d0f0 2022-04-03T19:53:48.141 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.133+0000 7f9e13fff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e141657e0 con 0x7f9e1414d4c0 2022-04-03T19:53:48.142 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.133+0000 7f9e13fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e141651d0 con 0x7f9e14156490 2022-04-03T19:53:48.142 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.133+0000 7f9e12ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1414d0f0 0x7f9e141472f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:48.142 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.134+0000 7f9e12ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1414d0f0 0x7f9e141472f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43100/0 (socket says 172.21.15.77:43100) 2022-04-03T19:53:48.142 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.134+0000 7f9e12ffd700 1 -- 172.21.15.77:0/2299836928 learned_addr learned my addr 172.21.15.77:0/2299836928 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:53:48.142 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.134+0000 7f9e0bfff700 1 --2- 172.21.15.77:0/2299836928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e1414d4c0 0x7f9e1414bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:48.143 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.134+0000 7f9e137fe700 1 --2- 172.21.15.77:0/2299836928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e14156490 0x7f9e14146700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:48.143 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.134+0000 7f9e12ffd700 1 -- 172.21.15.77:0/2299836928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e14156490 msgr2=0x7f9e14146700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:48.143 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.134+0000 7f9e12ffd700 1 --2- 172.21.15.77:0/2299836928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e14156490 0x7f9e14146700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:48.143 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.134+0000 7f9e12ffd700 1 -- 172.21.15.77:0/2299836928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e1414d4c0 msgr2=0x7f9e1414bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:48.144 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.134+0000 7f9e12ffd700 1 --2- 172.21.15.77:0/2299836928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e1414d4c0 0x7f9e1414bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:48.144 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.134+0000 7f9e12ffd700 1 -- 172.21.15.77:0/2299836928 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e040092a0 con 0x7f9e1414d0f0 2022-04-03T19:53:48.144 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.134+0000 7f9e12ffd700 1 --2- 172.21.15.77:0/2299836928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1414d0f0 0x7f9e141472f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9e04009e00 tx=0x7f9e04017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:48.144 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.135+0000 7f9e0affd700 1 -- 172.21.15.77:0/2299836928 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e04000bc0 con 0x7f9e1414d0f0 2022-04-03T19:53:48.144 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.135+0000 7f9e0affd700 1 -- 172.21.15.77:0/2299836928 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e04011900 con 0x7f9e1414d0f0 2022-04-03T19:53:48.145 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.135+0000 7f9e13fff700 1 -- 172.21.15.77:0/2299836928 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e14151310 con 0x7f9e1414d0f0 2022-04-03T19:53:48.145 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.135+0000 7f9e0affd700 1 -- 172.21.15.77:0/2299836928 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e04025420 con 0x7f9e1414d0f0 2022-04-03T19:53:48.145 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.135+0000 7f9e13fff700 1 -- 172.21.15.77:0/2299836928 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e14151030 con 0x7f9e1414d0f0 2022-04-03T19:53:48.145 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.136+0000 7f9e13fff700 1 -- 172.21.15.77:0/2299836928 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e14069f50 con 0x7f9e1414d0f0 2022-04-03T19:53:48.146 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.137+0000 7f9e0affd700 1 -- 172.21.15.77:0/2299836928 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f9e0402c030 con 0x7f9e1414d0f0 2022-04-03T19:53:48.146 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.137+0000 7f9e0affd700 1 --2- 172.21.15.77:0/2299836928 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9df8063ce0 0x7f9df8066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:53:48.147 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.138+0000 7f9e0affd700 1 -- 172.21.15.77:0/2299836928 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f9e04081f20 con 0x7f9e1414d0f0 2022-04-03T19:53:48.147 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.138+0000 7f9e0bfff700 1 --2- 172.21.15.77:0/2299836928 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9df8063ce0 0x7f9df8066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:53:48.147 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.139+0000 7f9e0bfff700 1 --2- 172.21.15.77:0/2299836928 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9df8063ce0 0x7f9df8066190 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9e140c34e0 tx=0x7f9dfc009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:53:48.148 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.142+0000 7f9e0affd700 1 -- 172.21.15.77:0/2299836928 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f9e04050870 con 0x7f9e1414d0f0 2022-04-03T19:53:48.382 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.381+0000 7f9e13fff700 1 -- 172.21.15.77:0/2299836928 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e140670e0 con 0x7f9e1414d0f0 2022-04-03T19:53:48.383 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.382+0000 7f9e0affd700 1 -- 172.21.15.77:0/2299836928 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f9e0402a020 con 0x7f9e1414d0f0 2022-04-03T19:53:48.383 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:53:48.383 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5463,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:53:46.235063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5039":{"gid":5039,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":27,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:53:48.385 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.384+0000 7f9e13fff700 1 -- 172.21.15.77:0/2299836928 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9df8063ce0 msgr2=0x7f9df8066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:48.386 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.384+0000 7f9e13fff700 1 --2- 172.21.15.77:0/2299836928 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9df8063ce0 0x7f9df8066190 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9e140c34e0 tx=0x7f9dfc009060 comp rx=0 tx=0).stop 2022-04-03T19:53:48.386 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.384+0000 7f9e13fff700 1 -- 172.21.15.77:0/2299836928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1414d0f0 msgr2=0x7f9e141472f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:53:48.386 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.384+0000 7f9e13fff700 1 --2- 172.21.15.77:0/2299836928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1414d0f0 0x7f9e141472f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9e04009e00 tx=0x7f9e04017000 comp rx=0 tx=0).stop 2022-04-03T19:53:48.386 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.384+0000 7f9e13fff700 1 -- 172.21.15.77:0/2299836928 shutdown_connections 2022-04-03T19:53:48.387 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.384+0000 7f9e13fff700 1 --2- 172.21.15.77:0/2299836928 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9df8063ce0 0x7f9df8066190 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:48.387 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.384+0000 7f9e13fff700 1 --2- 172.21.15.77:0/2299836928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e14156490 0x7f9e14146700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:48.387 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.384+0000 7f9e13fff700 1 --2- 172.21.15.77:0/2299836928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e1414d0f0 0x7f9e141472f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:48.387 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.384+0000 7f9e13fff700 1 --2- 172.21.15.77:0/2299836928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e1414d4c0 0x7f9e1414bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:53:48.388 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.385+0000 7f9e13fff700 1 -- 172.21.15.77:0/2299836928 >> 172.21.15.77:0/2299836928 conn(0x7f9e14069990 msgr2=0x7f9e140c60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:53:48.388 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.385+0000 7f9e13fff700 1 -- 172.21.15.77:0/2299836928 shutdown_connections 2022-04-03T19:53:48.388 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:53:48.385+0000 7f9e13fff700 1 -- 172.21.15.77:0/2299836928 wait complete. 2022-04-03T19:53:48.388 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 37 2022-04-03T19:53:48.409 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2022-04-03T19:53:48.410 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 65 secs before thrashing 2022-04-03T19:53:54.681 INFO:tasks.ceph.mds.d.smithi186.stderr:2022-04-03T19:53:54.680+0000 7f8af6031700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-04-03T19:54:16.956 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:54:16.962 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:54:25.762 INFO:tasks.workunit.client.0.smithi077.stderr:Updating files: 1% (133/11299) Updating files: 2% (226/11299) Updating files: 3% (339/11299) Updating files: 3% (340/11299) Updating files: 3% (450/11299) Updating files: 4% (452/11299) Updating files: 4% (498/11299) Updating files: 5% (565/11299) Updating files: 5% (664/11299) Updating files: 6% (678/11299) Updating files: 7% (791/11299) Updating files: 8% (904/11299) Updating files: 9% (1017/11299) Updating files: 9% (1046/11299) Updating files: 9% (1050/11299) Updating files: 10% (1130/11299) Updating files: 11% (1243/11299) Updating files: 11% (1300/11299) Updating files: 11% (1348/11299) Updating files: 12% (1356/11299) Updating files: 13% (1469/11299) Updating files: 14% (1582/11299) Updating files: 14% (1656/11299) Updating files: 15% (1695/11299) Updating files: 15% (1730/11299) Updating files: 16% (1808/11299) Updating files: 16% (1902/11299) Updating files: 17% (1921/11299) Updating files: 18% (2034/11299) Updating files: 18% (2138/11299) Updating files: 19% (2147/11299) Updating files: 20% (2260/11299) Updating files: 20% (2369/11299) Updating files: 21% (2373/11299) Updating files: 21% (2473/11299) Updating files: 22% (2486/11299) Updating files: 22% (2497/11299) Updating files: 23% (2599/11299) Updating files: 23% (2675/11299) Updating files: 24% (2712/11299) Updating files: 25% (2825/11299) Updating files: 26% (2938/11299) Updating files: 26% (3032/11299) Updating files: 27% (3051/11299) Updating files: 27% (3079/11299) Updating files: 27% (3107/11299) Updating files: 28% (3164/11299) Updating files: 28% (3214/11299) Updating files: 29% (3277/11299) Updating files: 29% (3317/11299) Updating files: 29% (3332/11299) Updating files: 29% (3334/11299) Updating files: 29% (3386/11299) Updating files: 30% (3390/11299) Updating files: 31% (3503/11299) Updating files: 31% (3614/11299) Updating files: 32% (3616/11299) Updating files: 32% (3675/11299) Updating files: 32% (3676/11299) Updating files: 33% (3729/11299) Updating files: 33% (3816/11299) Updating files: 34% (3842/11299) Updating files: 34% (3911/11299) Updating files: 34% (3936/11299) Updating files: 35% (3955/11299) Updating files: 35% (4043/11299) Updating files: 36% (4068/11299) Updating files: 37% (4181/11299) Updating files: 37% (4216/11299) Updating files: 37% (4262/11299) Updating files: 38% (4294/11299) Updating files: 38% (4310/11299) Updating files: 38% (4342/11299) Updating files: 39% (4407/11299) Updating files: 40% (4520/11299) Updating files: 40% (4538/11299) Updating files: 41% (4633/11299) Updating files: 42% (4746/11299) Updating files: 42% (4790/11299) Updating files: 43% (4859/11299) Updating files: 43% (4877/11299) Updating files: 44% (4972/11299) Updating files: 44% (5057/11299) Updating files: 45% (5085/11299) Updating files: 45% (5089/11299) Updating files: 46% (5198/11299) Updating files: 46% (5275/11299) Updating files: 47% (5311/11299) Updating files: 47% (5360/11299) Updating files: 48% (5424/11299) Updating files: 48% (5507/11299) Updating files: 49% (5537/11299) Updating files: 49% (5595/11299) Updating files: 49% (5597/11299) Updating files: 49% (5599/11299) Updating files: 50% (5650/11299) Updating files: 50% (5722/11299) Updating files: 51% (5763/11299) Updating files: 52% (5876/11299) Updating files: 52% (5953/11299) Updating files: 53% (5989/11299) Updating files: 54% (6102/11299) Updating files: 54% (6166/11299) Updating files: 55% (6215/11299) Updating files: 56% (6328/11299) Updating files: 56% (6392/11299) Updating files: 57% (6441/11299) Updating files: 58% (6554/11299) Updating files: 59% (6667/11299) Updating files: 60% (6780/11299) Updating files: 60% (6796/11299) Updating files: 61% (6893/11299) Updating files: 62% (7006/11299) Updating files: 62% (7066/11299) Updating files: 63% (7119/11299) Updating files: 63% (7127/11299) Updating files: 64% (7232/11299) Updating files: 65% (7345/11299) Updating files: 66% (7458/11299) Updating files: 66% (7508/11299) Updating files: 67% (7571/11299) Updating files: 67% (7607/11299) Updating files: 68% (7684/11299) Updating files: 69% (7797/11299) Updating files: 70% (7910/11299) Updating files: 70% (8002/11299) Updating files: 71% (8023/11299) Updating files: 71% (8035/11299) Updating files: 72% (8136/11299) Updating files: 72% (8150/11299) Updating files: 73% (8249/11299) Updating files: 74% (8362/11299) Updating files: 74% (8397/11299) Updating files: 75% (8475/11299) Updating files: 76% (8588/11299) Updating files: 76% (8618/11299) Updating files: 77% (8701/11299) Updating files: 77% (8747/11299) Updating files: 78% (8814/11299) Updating files: 78% (8841/11299) Updating files: 79% (8927/11299) Updating files: 79% (9028/11299) Updating files: 80% (9040/11299) Updating files: 81% (9153/11299) Updating files: 81% (9210/11299) Updating files: 82% (9266/11299) Updating files: 83% (9379/11299) Updating files: 84% (9492/11299) Updating files: 84% (9590/11299) Updating files: 85% (9605/11299) Updating files: 85% (9689/11299) Updating files: 86% (9718/11299) Updating files: 86% (9777/11299) Updating files: 87% (9831/11299) Updating files: 87% (9908/11299) Updating files: 88% (9944/11299) Updating files: 89% (10057/11299) Updating files: 90% (10170/11299) Updating files: 90% (10230/11299) Updating files: 90% (10232/11299) Updating files: 91% (10283/11299) Updating files: 92% (10396/11299) Updating files: 92% (10408/11299) Updating files: 93% (10509/11299) Updating files: 93% (10525/11299) Updating files: 93% (10527/11299) Updating files: 93% (10577/11299) Updating files: 94% (10622/11299) Updating files: 94% (10642/11299) Updating files: 95% (10735/11299) Updating files: 95% (10833/11299) Updating files: 96% (10848/11299) Updating files: 96% (10937/11299) Updating files: 97% (10961/11299) Updating files: 98% (11074/11299) Updating files: 98% (11168/11299) Updating files: 99% (11187/11299) Updating files: 100% (11299/11299) Updating files: 100% (11299/11299), done. 2022-04-03T19:54:26.848 INFO:tasks.workunit.client.0.smithi077.stderr:+ trap - EXIT 2022-04-03T19:54:26.848 INFO:tasks.workunit.client.0.smithi077.stderr:+ cd ceph 2022-04-03T19:54:26.848 INFO:tasks.workunit.client.0.smithi077.stderr:++ seq 1 21 2022-04-03T19:54:26.851 INFO:tasks.workunit.client.0.smithi077.stderr:+ versions='1 2022-04-03T19:54:26.851 INFO:tasks.workunit.client.0.smithi077.stderr:2 2022-04-03T19:54:26.851 INFO:tasks.workunit.client.0.smithi077.stderr:3 2022-04-03T19:54:26.851 INFO:tasks.workunit.client.0.smithi077.stderr:4 2022-04-03T19:54:26.852 INFO:tasks.workunit.client.0.smithi077.stderr:5 2022-04-03T19:54:26.852 INFO:tasks.workunit.client.0.smithi077.stderr:6 2022-04-03T19:54:26.852 INFO:tasks.workunit.client.0.smithi077.stderr:7 2022-04-03T19:54:26.852 INFO:tasks.workunit.client.0.smithi077.stderr:8 2022-04-03T19:54:26.852 INFO:tasks.workunit.client.0.smithi077.stderr:9 2022-04-03T19:54:26.853 INFO:tasks.workunit.client.0.smithi077.stderr:10 2022-04-03T19:54:26.853 INFO:tasks.workunit.client.0.smithi077.stderr:11 2022-04-03T19:54:26.853 INFO:tasks.workunit.client.0.smithi077.stderr:12 2022-04-03T19:54:26.853 INFO:tasks.workunit.client.0.smithi077.stderr:13 2022-04-03T19:54:26.853 INFO:tasks.workunit.client.0.smithi077.stderr:14 2022-04-03T19:54:26.854 INFO:tasks.workunit.client.0.smithi077.stderr:15 2022-04-03T19:54:26.854 INFO:tasks.workunit.client.0.smithi077.stderr:16 2022-04-03T19:54:26.854 INFO:tasks.workunit.client.0.smithi077.stderr:17 2022-04-03T19:54:26.854 INFO:tasks.workunit.client.0.smithi077.stderr:18 2022-04-03T19:54:26.855 INFO:tasks.workunit.client.0.smithi077.stderr:19 2022-04-03T19:54:26.855 INFO:tasks.workunit.client.0.smithi077.stderr:20 2022-04-03T19:54:26.855 INFO:tasks.workunit.client.0.smithi077.stderr:21' 2022-04-03T19:54:26.855 INFO:tasks.workunit.client.0.smithi077.stderr:+ for v in $versions 2022-04-03T19:54:26.855 INFO:tasks.workunit.client.0.smithi077.stderr:+ ver=v0.1 2022-04-03T19:54:26.856 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo v0.1 2022-04-03T19:54:26.856 INFO:tasks.workunit.client.0.smithi077.stdout:v0.1 2022-04-03T19:54:26.856 INFO:tasks.workunit.client.0.smithi077.stderr:+ git reset --hard v0.1 2022-04-03T19:54:47.087 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:54:47.093 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:54:53.411 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:54:53.618 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.618+0000 7f4a53fff700 1 Processor -- start 2022-04-03T19:54:53.618 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.618+0000 7f4a53fff700 1 -- start start 2022-04-03T19:54:53.619 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.619+0000 7f4a53fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4a5406a280 0x7f4a54159050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:53.619 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.619+0000 7f4a53fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a540c4dc0 0x7f4a540c5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:53.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.619+0000 7f4a53fff700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a54155480 con 0x7f4a540c2010 2022-04-03T19:54:53.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.619+0000 7f4a53fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a540c4200 con 0x7f4a5406a280 2022-04-03T19:54:53.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.619+0000 7f4a53fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a54159590 con 0x7f4a540c4dc0 2022-04-03T19:54:53.622 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.619+0000 7f4a537fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a540c4dc0 0x7f4a540c5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:53.623 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.619+0000 7f4a52ffd700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f4a540c2010 0x7f4a54155910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44344/0 (socket says 172.21.15.77:44344) 2022-04-03T19:54:53.623 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.619+0000 7f4a527fc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f4a5406a280 0x7f4a54159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:53.623 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.619+0000 7f4a52ffd700 1 -- 172.21.15.77:0/849459989 learned_addr learned my addr 172.21.15.77:0/849459989 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:54:53.623 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.620+0000 7f4a51ffb700 1 -- 172.21.15.77:0/849459989 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023756107 0 0) 0x7f4a54155480 con 0x7f4a540c2010 2022-04-03T19:54:53.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.620+0000 7f4a51ffb700 1 -- 172.21.15.77:0/849459989 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a400033f0 con 0x7f4a540c2010 2022-04-03T19:54:53.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.620+0000 7f4a537fe700 1 -- 172.21.15.77:0/849459989 >> v2:172.21.15.77:3300/0 conn(0x7f4a5406a280 msgr2=0x7f4a54159050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:54:53.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.620+0000 7f4a537fe700 1 --2- 172.21.15.77:0/849459989 >> v2:172.21.15.77:3300/0 conn(0x7f4a5406a280 0x7f4a54159050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:53.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.620+0000 7f4a537fe700 1 -- 172.21.15.77:0/849459989 >> v1:172.21.15.186:6789/0 conn(0x7f4a540c2010 legacy=0x7f4a54155910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:53.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.620+0000 7f4a537fe700 1 -- 172.21.15.77:0/849459989 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a54156020 con 0x7f4a540c4dc0 2022-04-03T19:54:53.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.620+0000 7f4a527fc700 1 --2- 172.21.15.77:0/849459989 >> v2:172.21.15.77:3300/0 conn(0x7f4a5406a280 0x7f4a54159050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:54:53.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.620+0000 7f4a537fe700 1 --2- 172.21.15.77:0/849459989 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a540c4dc0 0x7f4a540c5260 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c009e60 tx=0x7f4a4c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33a58714a1fa8a0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:54:53.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.620+0000 7f4a51ffb700 1 -- 172.21.15.77:0/849459989 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c00e070 con 0x7f4a540c4dc0 2022-04-03T19:54:53.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.620+0000 7f4a51ffb700 1 -- 172.21.15.77:0/849459989 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a4c00bab0 con 0x7f4a540c4dc0 2022-04-03T19:54:53.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.620+0000 7f4a51ffb700 1 -- 172.21.15.77:0/849459989 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c00be70 con 0x7f4a540c4dc0 2022-04-03T19:54:53.626 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.621+0000 7f4a53fff700 1 -- 172.21.15.77:0/849459989 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a540c4dc0 msgr2=0x7f4a540c5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:53.626 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.621+0000 7f4a53fff700 1 --2- 172.21.15.77:0/849459989 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a540c4dc0 0x7f4a540c5260 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c009e60 tx=0x7f4a4c009780 comp rx=0 tx=0).stop 2022-04-03T19:54:53.626 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.621+0000 7f4a53fff700 1 -- 172.21.15.77:0/849459989 shutdown_connections 2022-04-03T19:54:53.626 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.621+0000 7f4a53fff700 1 --2- 172.21.15.77:0/849459989 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a540c4dc0 0x7f4a540c5260 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:53.627 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.621+0000 7f4a53fff700 1 --2- 172.21.15.77:0/849459989 >> v2:172.21.15.77:3300/0 conn(0x7f4a5406a280 0x7f4a54159050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:53.627 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.621+0000 7f4a53fff700 1 -- 172.21.15.77:0/849459989 >> 172.21.15.77:0/849459989 conn(0x7f4a5406cb30 msgr2=0x7f4a5406cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:54:53.627 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.621+0000 7f4a53fff700 1 -- 172.21.15.77:0/849459989 shutdown_connections 2022-04-03T19:54:53.627 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.621+0000 7f4a53fff700 1 -- 172.21.15.77:0/849459989 wait complete. 2022-04-03T19:54:53.630 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.623+0000 7f4a53fff700 1 Processor -- start 2022-04-03T19:54:53.630 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.623+0000 7f4a53fff700 1 -- start start 2022-04-03T19:54:53.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.624+0000 7f4a53fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a5406a280 0x7f4a540bcd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:53.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.624+0000 7f4a53fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a540c2010 0x7f4a540c0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:53.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.624+0000 7f4a53fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a540c4dc0 0x7f4a54158580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:53.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.624+0000 7f4a53fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a540bf160 con 0x7f4a540c4dc0 2022-04-03T19:54:53.631 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.624+0000 7f4a53fff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a541504c0 con 0x7f4a5406a280 2022-04-03T19:54:53.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.624+0000 7f4a53fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a54168e70 con 0x7f4a540c2010 2022-04-03T19:54:53.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.624+0000 7f4a537fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a540c4dc0 0x7f4a54158580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:53.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.624+0000 7f4a527fc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a540c2010 0x7f4a540c0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:53.632 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.624+0000 7f4a52ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a5406a280 0x7f4a540bcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:53.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.624+0000 7f4a527fc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a540c2010 0x7f4a540c0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60396/0 (socket says 172.21.15.77:60396) 2022-04-03T19:54:53.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.624+0000 7f4a527fc700 1 -- 172.21.15.77:0/3755229629 learned_addr learned my addr 172.21.15.77:0/3755229629 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:54:53.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.624+0000 7f4a527fc700 1 -- 172.21.15.77:0/3755229629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a5406a280 msgr2=0x7f4a540bcd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:53.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.624+0000 7f4a527fc700 1 --2- 172.21.15.77:0/3755229629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a5406a280 0x7f4a540bcd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:53.633 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.624+0000 7f4a527fc700 1 -- 172.21.15.77:0/3755229629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a540c4dc0 msgr2=0x7f4a54158580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:53.634 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.624+0000 7f4a527fc700 1 --2- 172.21.15.77:0/3755229629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a540c4dc0 0x7f4a54158580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:53.634 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.624+0000 7f4a527fc700 1 -- 172.21.15.77:0/3755229629 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a480092a0 con 0x7f4a540c2010 2022-04-03T19:54:53.634 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.625+0000 7f4a527fc700 1 --2- 172.21.15.77:0/3755229629 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a540c2010 0x7f4a540c0d80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4a4800ea50 tx=0x7f4a48009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:54:53.634 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.625+0000 7f4a3affd700 1 -- 172.21.15.77:0/3755229629 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4800ee90 con 0x7f4a540c2010 2022-04-03T19:54:53.634 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.625+0000 7f4a3affd700 1 -- 172.21.15.77:0/3755229629 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a4800c850 con 0x7f4a540c2010 2022-04-03T19:54:53.635 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.625+0000 7f4a53fff700 1 -- 172.21.15.77:0/3755229629 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a4c009a20 con 0x7f4a540c2010 2022-04-03T19:54:53.635 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.625+0000 7f4a3affd700 1 -- 172.21.15.77:0/3755229629 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4800cbd0 con 0x7f4a540c2010 2022-04-03T19:54:53.635 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.625+0000 7f4a53fff700 1 -- 172.21.15.77:0/3755229629 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a5414fb90 con 0x7f4a540c2010 2022-04-03T19:54:53.635 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.626+0000 7f4a53fff700 1 -- 172.21.15.77:0/3755229629 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a54155130 con 0x7f4a540c2010 2022-04-03T19:54:53.635 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.627+0000 7f4a3affd700 1 -- 172.21.15.77:0/3755229629 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f4a48019030 con 0x7f4a540c2010 2022-04-03T19:54:53.636 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.628+0000 7f4a3affd700 1 --2- 172.21.15.77:0/3755229629 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4a34063ce0 0x7f4a34066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:53.636 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.628+0000 7f4a3affd700 1 -- 172.21.15.77:0/3755229629 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f4a48080a70 con 0x7f4a540c2010 2022-04-03T19:54:53.636 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.628+0000 7f4a52ffd700 1 --2- 172.21.15.77:0/3755229629 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4a34063ce0 0x7f4a34066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:53.636 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.629+0000 7f4a52ffd700 1 --2- 172.21.15.77:0/3755229629 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4a34063ce0 0x7f4a34066190 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4a44005750 tx=0x7f4a44009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:54:53.637 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.632+0000 7f4a3affd700 1 -- 172.21.15.77:0/3755229629 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f4a4804f460 con 0x7f4a540c2010 2022-04-03T19:54:53.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.885+0000 7f4a53fff700 1 -- 172.21.15.77:0/3755229629 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a5404b050 con 0x7f4a540c2010 2022-04-03T19:54:53.888 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.886+0000 7f4a3affd700 1 -- 172.21.15.77:0/3755229629 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f4a48021090 con 0x7f4a540c2010 2022-04-03T19:54:53.895 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:54:53.895 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5463,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:53:46.235063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5039":{"gid":5039,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":27,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:54:53.903 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.902+0000 7f4a53fff700 1 -- 172.21.15.77:0/3755229629 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4a34063ce0 msgr2=0x7f4a34066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:53.903 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.902+0000 7f4a53fff700 1 --2- 172.21.15.77:0/3755229629 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4a34063ce0 0x7f4a34066190 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4a44005750 tx=0x7f4a44009040 comp rx=0 tx=0).stop 2022-04-03T19:54:53.903 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.902+0000 7f4a53fff700 1 -- 172.21.15.77:0/3755229629 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a540c2010 msgr2=0x7f4a540c0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:53.903 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.902+0000 7f4a53fff700 1 --2- 172.21.15.77:0/3755229629 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a540c2010 0x7f4a540c0d80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4a4800ea50 tx=0x7f4a48009060 comp rx=0 tx=0).stop 2022-04-03T19:54:53.904 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.902+0000 7f4a53fff700 1 -- 172.21.15.77:0/3755229629 shutdown_connections 2022-04-03T19:54:53.904 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.902+0000 7f4a53fff700 1 --2- 172.21.15.77:0/3755229629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a540c4dc0 0x7f4a54158580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:53.904 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.902+0000 7f4a53fff700 1 --2- 172.21.15.77:0/3755229629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a5406a280 0x7f4a540bcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:53.904 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.902+0000 7f4a53fff700 1 --2- 172.21.15.77:0/3755229629 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4a34063ce0 0x7f4a34066190 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:53.905 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.902+0000 7f4a53fff700 1 --2- 172.21.15.77:0/3755229629 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a540c2010 0x7f4a540c0d80 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:53.905 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.903+0000 7f4a53fff700 1 -- 172.21.15.77:0/3755229629 >> 172.21.15.77:0/3755229629 conn(0x7f4a5406cb30 msgr2=0x7f4a540c8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:54:53.911 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.912+0000 7f4a53fff700 1 -- 172.21.15.77:0/3755229629 shutdown_connections 2022-04-03T19:54:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:53.919+0000 7f4a53fff700 1 -- 172.21.15.77:0/3755229629 wait complete. 2022-04-03T19:54:53.933 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 37 2022-04-03T19:54:53.980 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2022-04-03T19:54:53.980 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2022-04-03T19:54:53.980 INFO:teuthology.orchestra.run:waiting for 300 2022-04-03T19:54:54.003 INFO:tasks.ceph.mds.d:Stopped 2022-04-03T19:54:54.003 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:54:54.212 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.212+0000 7f1d0f59e700 1 Processor -- start 2022-04-03T19:54:54.213 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.212+0000 7f1d0f59e700 1 -- start start 2022-04-03T19:54:54.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.213+0000 7f1d0f59e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1d1006bbe0 0x7f1d101584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:54.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.213+0000 7f1d0f59e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1d1006c500 0x7f1d1014eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:54.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.213+0000 7f1d0f59e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d1006d260 con 0x7f1d1006bfb0 2022-04-03T19:54:54.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.213+0000 7f1d0f59e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d10159450 con 0x7f1d1006bbe0 2022-04-03T19:54:54.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.213+0000 7f1d0f59e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d100c6330 con 0x7f1d1006c500 2022-04-03T19:54:54.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.214+0000 7f1d0e59c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1d1006bbe0 0x7f1d101584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:54.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.214+0000 7f1d0ed9d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1d1006c500 0x7f1d1014eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:54.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.214+0000 7f1d0dd9b700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f1d1006bfb0 0x7f1d1006d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44348/0 (socket says 172.21.15.77:44348) 2022-04-03T19:54:54.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.214+0000 7f1d0dd9b700 1 -- 172.21.15.77:0/2192985918 learned_addr learned my addr 172.21.15.77:0/2192985918 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:54:54.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.214+0000 7f1d0ed9d700 1 -- 172.21.15.77:0/2192985918 >> v2:172.21.15.77:3300/0 conn(0x7f1d1006bbe0 msgr2=0x7f1d101584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:54:54.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.214+0000 7f1d0ed9d700 1 --2- 172.21.15.77:0/2192985918 >> v2:172.21.15.77:3300/0 conn(0x7f1d1006bbe0 0x7f1d101584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:54.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.214+0000 7f1d0ed9d700 1 -- 172.21.15.77:0/2192985918 >> v1:172.21.15.186:6789/0 conn(0x7f1d1006bfb0 legacy=0x7f1d1006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:54.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.214+0000 7f1d0ed9d700 1 -- 172.21.15.77:0/2192985918 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d10158bb0 con 0x7f1d1006c500 2022-04-03T19:54:54.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.214+0000 7f1d0d59a700 1 -- 172.21.15.77:0/2192985918 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263322328 0 0) 0x7f1d1006d260 con 0x7f1d1006bfb0 2022-04-03T19:54:54.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.214+0000 7f1d0ed9d700 1 --2- 172.21.15.77:0/2192985918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d1006c500 0x7f1d1014eea0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1d100cc6b0 tx=0x7f1d08009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cc81b3977c8135c server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:54:54.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.215+0000 7f1d0d59a700 1 -- 172.21.15.77:0/2192985918 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0800e070 con 0x7f1d1006c500 2022-04-03T19:54:54.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.215+0000 7f1d0d59a700 1 -- 172.21.15.77:0/2192985918 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0800bb60 con 0x7f1d1006c500 2022-04-03T19:54:54.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.215+0000 7f1d0d59a700 1 -- 172.21.15.77:0/2192985918 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d08012430 con 0x7f1d1006c500 2022-04-03T19:54:54.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.215+0000 7f1d0f59e700 1 -- 172.21.15.77:0/2192985918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d1006c500 msgr2=0x7f1d1014eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:54.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.215+0000 7f1d0f59e700 1 --2- 172.21.15.77:0/2192985918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d1006c500 0x7f1d1014eea0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1d100cc6b0 tx=0x7f1d08009780 comp rx=0 tx=0).stop 2022-04-03T19:54:54.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.215+0000 7f1d0f59e700 1 -- 172.21.15.77:0/2192985918 shutdown_connections 2022-04-03T19:54:54.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.215+0000 7f1d0f59e700 1 --2- 172.21.15.77:0/2192985918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d1006c500 0x7f1d1014eea0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:54.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.215+0000 7f1d0f59e700 1 --2- 172.21.15.77:0/2192985918 >> v2:172.21.15.77:3300/0 conn(0x7f1d1006bbe0 0x7f1d101584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:54.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.215+0000 7f1d0f59e700 1 -- 172.21.15.77:0/2192985918 >> 172.21.15.77:0/2192985918 conn(0x7f1d100cb6b0 msgr2=0x7f1d100cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:54:54.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.215+0000 7f1d0f59e700 1 -- 172.21.15.77:0/2192985918 shutdown_connections 2022-04-03T19:54:54.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.215+0000 7f1d0f59e700 1 -- 172.21.15.77:0/2192985918 wait complete. 2022-04-03T19:54:54.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.217+0000 7f1d0f59e700 1 Processor -- start 2022-04-03T19:54:54.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.217+0000 7f1d0f59e700 1 -- start start 2022-04-03T19:54:54.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.218+0000 7f1d0f59e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d1006bbe0 0x7f1d101526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:54.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.218+0000 7f1d0f59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d1006bfb0 0x7f1d10148e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:54.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.218+0000 7f1d0f59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d1006c500 0x7f1d10150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:54.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.218+0000 7f1d0f59e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d10158240 con 0x7f1d1006bfb0 2022-04-03T19:54:54.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.218+0000 7f1d0f59e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1014f5f0 con 0x7f1d1006bbe0 2022-04-03T19:54:54.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.218+0000 7f1d0f59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d10164a60 con 0x7f1d1006c500 2022-04-03T19:54:54.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.218+0000 7f1d0dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d1006bfb0 0x7f1d10148e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:54.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.218+0000 7f1d0ed9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d1006c500 0x7f1d10150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:54.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.218+0000 7f1d0ed9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d1006c500 0x7f1d10150110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60398/0 (socket says 172.21.15.77:60398) 2022-04-03T19:54:54.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.218+0000 7f1d0e59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d1006bbe0 0x7f1d101526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:54.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.218+0000 7f1d0ed9d700 1 -- 172.21.15.77:0/3973539378 learned_addr learned my addr 172.21.15.77:0/3973539378 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:54:54.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.218+0000 7f1d0dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d1006bfb0 0x7f1d10148e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43108/0 (socket says 172.21.15.77:43108) 2022-04-03T19:54:54.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.219+0000 7f1d0e59c700 1 -- 172.21.15.77:0/3973539378 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d1006c500 msgr2=0x7f1d10150110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:54.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.219+0000 7f1d0e59c700 1 --2- 172.21.15.77:0/3973539378 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d1006c500 0x7f1d10150110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:54.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.219+0000 7f1d0e59c700 1 -- 172.21.15.77:0/3973539378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d1006bfb0 msgr2=0x7f1d10148e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:54.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.219+0000 7f1d0e59c700 1 --2- 172.21.15.77:0/3973539378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d1006bfb0 0x7f1d10148e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:54.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.219+0000 7f1d0e59c700 1 -- 172.21.15.77:0/3973539378 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d08009a20 con 0x7f1d1006bbe0 2022-04-03T19:54:54.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.219+0000 7f1d0ed9d700 1 --2- 172.21.15.77:0/3973539378 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d1006c500 0x7f1d10150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:54:54.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.219+0000 7f1d0e59c700 1 --2- 172.21.15.77:0/3973539378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d1006bbe0 0x7f1d101526c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000e6f0 tx=0x7f1d00009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:54:54.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.219+0000 7f1cfe7fc700 1 -- 172.21.15.77:0/3973539378 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d00013680 con 0x7f1d1006bbe0 2022-04-03T19:54:54.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.219+0000 7f1cfe7fc700 1 -- 172.21.15.77:0/3973539378 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d000137e0 con 0x7f1d1006bbe0 2022-04-03T19:54:54.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.219+0000 7f1cfe7fc700 1 -- 172.21.15.77:0/3973539378 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d00013b80 con 0x7f1d1006bbe0 2022-04-03T19:54:54.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.219+0000 7f1d0f59e700 1 -- 172.21.15.77:0/3973539378 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d1014f840 con 0x7f1d1006bbe0 2022-04-03T19:54:54.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.220+0000 7f1d0f59e700 1 -- 172.21.15.77:0/3973539378 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d10165000 con 0x7f1d1006bbe0 2022-04-03T19:54:54.230 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.220+0000 7f1d0f59e700 1 -- 172.21.15.77:0/3973539378 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d10165650 con 0x7f1d1006bbe0 2022-04-03T19:54:54.230 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.221+0000 7f1cfe7fc700 1 -- 172.21.15.77:0/3973539378 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f1d00019030 con 0x7f1d1006bbe0 2022-04-03T19:54:54.230 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.222+0000 7f1cfe7fc700 1 --2- 172.21.15.77:0/3973539378 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1cec063ce0 0x7f1cec066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:54.231 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.222+0000 7f1cfe7fc700 1 -- 172.21.15.77:0/3973539378 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f1d00080740 con 0x7f1d1006bbe0 2022-04-03T19:54:54.231 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.222+0000 7f1d0dd9b700 1 --2- 172.21.15.77:0/3973539378 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1cec063ce0 0x7f1cec066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:54.231 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.223+0000 7f1d0dd9b700 1 --2- 172.21.15.77:0/3973539378 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1cec063ce0 0x7f1cec066190 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1d1014e7f0 tx=0x7f1d04007bb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:54:54.231 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.226+0000 7f1cfe7fc700 1 -- 172.21.15.77:0/3973539378 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f1d0004f130 con 0x7f1d1006bbe0 2022-04-03T19:54:54.463 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.463+0000 7f1d0f59e700 1 -- 172.21.15.77:0/3973539378 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d100c7ec0 con 0x7f1d1006bbe0 2022-04-03T19:54:54.464 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.464+0000 7f1cfe7fc700 1 -- 172.21.15.77:0/3973539378 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f1d00052750 con 0x7f1d1006bbe0 2022-04-03T19:54:54.464 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:54:54.465 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5463,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:53:46.235063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5039":{"gid":5039,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":27,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:54:54.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.466+0000 7f1d0f59e700 1 -- 172.21.15.77:0/3973539378 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1cec063ce0 msgr2=0x7f1cec066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:54.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.466+0000 7f1d0f59e700 1 --2- 172.21.15.77:0/3973539378 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1cec063ce0 0x7f1cec066190 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1d1014e7f0 tx=0x7f1d04007bb0 comp rx=0 tx=0).stop 2022-04-03T19:54:54.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.466+0000 7f1d0f59e700 1 -- 172.21.15.77:0/3973539378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d1006bbe0 msgr2=0x7f1d101526c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:54.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.466+0000 7f1d0f59e700 1 --2- 172.21.15.77:0/3973539378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d1006bbe0 0x7f1d101526c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000e6f0 tx=0x7f1d00009060 comp rx=0 tx=0).stop 2022-04-03T19:54:54.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.466+0000 7f1d0f59e700 1 -- 172.21.15.77:0/3973539378 shutdown_connections 2022-04-03T19:54:54.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.466+0000 7f1d0f59e700 1 --2- 172.21.15.77:0/3973539378 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1cec063ce0 0x7f1cec066190 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:54.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.466+0000 7f1d0f59e700 1 --2- 172.21.15.77:0/3973539378 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d1006c500 0x7f1d10150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:54.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.466+0000 7f1d0f59e700 1 --2- 172.21.15.77:0/3973539378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d1006bbe0 0x7f1d101526c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:54.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.466+0000 7f1d0f59e700 1 --2- 172.21.15.77:0/3973539378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d1006bfb0 0x7f1d10148e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:54.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.466+0000 7f1d0f59e700 1 -- 172.21.15.77:0/3973539378 >> 172.21.15.77:0/3973539378 conn(0x7f1d100cb6b0 msgr2=0x7f1d1014b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:54:54.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.466+0000 7f1d0f59e700 1 -- 172.21.15.77:0/3973539378 shutdown_connections 2022-04-03T19:54:54.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:54.466+0000 7f1d0f59e700 1 -- 172.21.15.77:0/3973539378 wait complete. 2022-04-03T19:54:54.470 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 37 2022-04-03T19:54:54.491 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:54:56.492 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:54:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.697+0000 7fafb7fff700 1 Processor -- start 2022-04-03T19:54:56.698 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.698+0000 7fafb7fff700 1 -- start start 2022-04-03T19:54:56.699 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.699+0000 7fafb7fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafb806bbe0 0x7fafb81584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:56.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.699+0000 7fafb7fff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fafb806bfb0 0x7fafb806d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:56.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.699+0000 7fafb7fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fafb806c500 0x7fafb814eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:56.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.699+0000 7fafb7fff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb8159450 con 0x7fafb806c500 2022-04-03T19:54:56.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.699+0000 7fafb7fff700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb806d260 con 0x7fafb806bfb0 2022-04-03T19:54:56.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.699+0000 7fafb7fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb80c6330 con 0x7fafb806bbe0 2022-04-03T19:54:56.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.699+0000 7fafb77fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fafb806c500 0x7fafb814eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:56.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.699+0000 7fafb77fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fafb806c500 0x7fafb814eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43110/0 (socket says 172.21.15.77:43110) 2022-04-03T19:54:56.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.699+0000 7fafb77fe700 1 -- 172.21.15.77:0/832286954 learned_addr learned my addr 172.21.15.77:0/832286954 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:54:56.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.699+0000 7fafb6ffd700 1 --2- 172.21.15.77:0/832286954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafb806bbe0 0x7fafb81584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:56.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.699+0000 7fafb67fc700 1 --2- 172.21.15.77:0/832286954 >> v2:172.21.15.186:3300/0 conn(0x7fafb806bfb0 0x7fafb806d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:56.704 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.700+0000 7fafb77fe700 1 -- 172.21.15.77:0/832286954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafb806bbe0 msgr2=0x7fafb81584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:54:56.704 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.700+0000 7fafb77fe700 1 --2- 172.21.15.77:0/832286954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafb806bbe0 0x7fafb81584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:56.705 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.700+0000 7fafb77fe700 1 -- 172.21.15.77:0/832286954 >> v2:172.21.15.186:3300/0 conn(0x7fafb806bfb0 msgr2=0x7fafb806d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:54:56.705 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.700+0000 7fafb77fe700 1 --2- 172.21.15.77:0/832286954 >> v2:172.21.15.186:3300/0 conn(0x7fafb806bfb0 0x7fafb806d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:56.705 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.700+0000 7fafb77fe700 1 -- 172.21.15.77:0/832286954 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb8158bb0 con 0x7fafb806c500 2022-04-03T19:54:56.705 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.700+0000 7fafb77fe700 1 --2- 172.21.15.77:0/832286954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb806c500 0x7fafb814eea0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fafb8088d10 tx=0x7fafb0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aba6dcfe015edaa server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:54:56.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.700+0000 7fafb5ffb700 1 -- 172.21.15.77:0/832286954 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb000e070 con 0x7fafb806c500 2022-04-03T19:54:56.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.700+0000 7fafb5ffb700 1 -- 172.21.15.77:0/832286954 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb000b970 con 0x7fafb806c500 2022-04-03T19:54:56.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.700+0000 7fafb5ffb700 1 -- 172.21.15.77:0/832286954 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb00092e0 con 0x7fafb806c500 2022-04-03T19:54:56.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.701+0000 7fafb7fff700 1 -- 172.21.15.77:0/832286954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb806c500 msgr2=0x7fafb814eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:56.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.701+0000 7fafb7fff700 1 --2- 172.21.15.77:0/832286954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb806c500 0x7fafb814eea0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fafb8088d10 tx=0x7fafb0009840 comp rx=0 tx=0).stop 2022-04-03T19:54:56.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.701+0000 7fafb7fff700 1 -- 172.21.15.77:0/832286954 shutdown_connections 2022-04-03T19:54:56.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.701+0000 7fafb7fff700 1 --2- 172.21.15.77:0/832286954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb806c500 0x7fafb814eea0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:56.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.701+0000 7fafb7fff700 1 --2- 172.21.15.77:0/832286954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafb806bbe0 0x7fafb81584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:56.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.701+0000 7fafb7fff700 1 --2- 172.21.15.77:0/832286954 >> v2:172.21.15.186:3300/0 conn(0x7fafb806bfb0 0x7fafb806d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:56.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.701+0000 7fafb7fff700 1 -- 172.21.15.77:0/832286954 >> 172.21.15.77:0/832286954 conn(0x7fafb80cb6b0 msgr2=0x7fafb80cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:54:56.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.701+0000 7fafb7fff700 1 -- 172.21.15.77:0/832286954 shutdown_connections 2022-04-03T19:54:56.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.701+0000 7fafb7fff700 1 -- 172.21.15.77:0/832286954 wait complete. 2022-04-03T19:54:56.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.702+0000 7fafb7fff700 1 Processor -- start 2022-04-03T19:54:56.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.703+0000 7fafb7fff700 1 -- start start 2022-04-03T19:54:56.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.703+0000 7fafb7fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafb806bbe0 0x7fafb80c0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:56.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.703+0000 7fafb7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafb806bfb0 0x7fafb80bfb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:56.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.703+0000 7fafb7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb806c500 0x7fafb81520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:56.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.704+0000 7fafb7fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb80c17f0 con 0x7fafb806c500 2022-04-03T19:54:56.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.704+0000 7fafb7fff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb80bf420 con 0x7fafb806bfb0 2022-04-03T19:54:56.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.704+0000 7fafb7fff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb8165260 con 0x7fafb806bbe0 2022-04-03T19:54:56.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.704+0000 7fafb77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb806c500 0x7fafb81520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:56.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.704+0000 7fafb67fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafb806bfb0 0x7fafb80bfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:56.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.704+0000 7fafb6ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafb806bbe0 0x7fafb80c0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:56.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.704+0000 7fafb67fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafb806bfb0 0x7fafb80bfb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38306/0 (socket says 172.21.15.77:38306) 2022-04-03T19:54:56.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.704+0000 7fafb67fc700 1 -- 172.21.15.77:0/1603430576 learned_addr learned my addr 172.21.15.77:0/1603430576 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:54:56.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.704+0000 7fafb77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb806c500 0x7fafb81520c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43112/0 (socket says 172.21.15.77:43112) 2022-04-03T19:54:56.713 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.704+0000 7fafb67fc700 1 -- 172.21.15.77:0/1603430576 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafb806bbe0 msgr2=0x7fafb80c0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:56.713 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.704+0000 7fafb67fc700 1 --2- 172.21.15.77:0/1603430576 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafb806bbe0 0x7fafb80c0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:56.713 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.704+0000 7fafb67fc700 1 -- 172.21.15.77:0/1603430576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb806c500 msgr2=0x7fafb81520c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:56.714 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.704+0000 7fafb67fc700 1 --2- 172.21.15.77:0/1603430576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb806c500 0x7fafb81520c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:56.714 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.704+0000 7fafb67fc700 1 -- 172.21.15.77:0/1603430576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb0009ae0 con 0x7fafb806bfb0 2022-04-03T19:54:56.714 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.704+0000 7fafb77fe700 1 --2- 172.21.15.77:0/1603430576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb806c500 0x7fafb81520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:54:56.714 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.704+0000 7fafb6ffd700 1 --2- 172.21.15.77:0/1603430576 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafb806bbe0 0x7fafb80c0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:54:56.714 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.705+0000 7fafb67fc700 1 --2- 172.21.15.77:0/1603430576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafb806bfb0 0x7fafb80bfb90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fafac00e900 tx=0x7fafac009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:54:56.715 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.705+0000 7faf9effd700 1 -- 172.21.15.77:0/1603430576 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac013680 con 0x7fafb806bfb0 2022-04-03T19:54:56.715 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.705+0000 7faf9effd700 1 -- 172.21.15.77:0/1603430576 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafac0137e0 con 0x7fafb806bfb0 2022-04-03T19:54:56.715 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.705+0000 7fafb7fff700 1 -- 172.21.15.77:0/1603430576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb8165540 con 0x7fafb806bfb0 2022-04-03T19:54:56.715 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.705+0000 7faf9effd700 1 -- 172.21.15.77:0/1603430576 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac013b60 con 0x7fafb806bfb0 2022-04-03T19:54:56.716 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.705+0000 7fafb7fff700 1 -- 172.21.15.77:0/1603430576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb8165b00 con 0x7fafb806bfb0 2022-04-03T19:54:56.716 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.706+0000 7fafb7fff700 1 -- 172.21.15.77:0/1603430576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb80b1720 con 0x7fafb806bfb0 2022-04-03T19:54:56.716 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.707+0000 7faf9effd700 1 -- 172.21.15.77:0/1603430576 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fafac019030 con 0x7fafb806bfb0 2022-04-03T19:54:56.716 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.708+0000 7faf9effd700 1 --2- 172.21.15.77:0/1603430576 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7faf98063d90 0x7faf98066240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:56.716 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.708+0000 7faf9effd700 1 -- 172.21.15.77:0/1603430576 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7fafac080df0 con 0x7fafb806bfb0 2022-04-03T19:54:56.717 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.708+0000 7fafb6ffd700 1 --2- 172.21.15.77:0/1603430576 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7faf98063d90 0x7faf98066240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:56.717 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.709+0000 7fafb6ffd700 1 --2- 172.21.15.77:0/1603430576 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7faf98063d90 0x7faf98066240 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fafa8011310 tx=0x7fafa8009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:54:56.717 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.711+0000 7faf9effd700 1 -- 172.21.15.77:0/1603430576 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fafac04f770 con 0x7fafb806bfb0 2022-04-03T19:54:56.946 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.946+0000 7fafb7fff700 1 -- 172.21.15.77:0/1603430576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafb80c7ec0 con 0x7fafb806bfb0 2022-04-03T19:54:56.947 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.947+0000 7faf9effd700 1 -- 172.21.15.77:0/1603430576 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6110 (secure 0 0 0) 0x7fafac021090 con 0x7fafb806bfb0 2022-04-03T19:54:56.947 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:54:56.948 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5463,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:53:46.235063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5039":{"gid":5039,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":27,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:54:56.950 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.949+0000 7fafb7fff700 1 -- 172.21.15.77:0/1603430576 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7faf98063d90 msgr2=0x7faf98066240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:56.951 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.949+0000 7fafb7fff700 1 --2- 172.21.15.77:0/1603430576 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7faf98063d90 0x7faf98066240 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fafa8011310 tx=0x7fafa8009060 comp rx=0 tx=0).stop 2022-04-03T19:54:56.951 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.949+0000 7fafb7fff700 1 -- 172.21.15.77:0/1603430576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafb806bfb0 msgr2=0x7fafb80bfb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:56.951 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.949+0000 7fafb7fff700 1 --2- 172.21.15.77:0/1603430576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafb806bfb0 0x7fafb80bfb90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fafac00e900 tx=0x7fafac009060 comp rx=0 tx=0).stop 2022-04-03T19:54:56.951 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.949+0000 7fafb7fff700 1 -- 172.21.15.77:0/1603430576 shutdown_connections 2022-04-03T19:54:56.952 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.949+0000 7fafb7fff700 1 --2- 172.21.15.77:0/1603430576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafb806c500 0x7fafb81520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:56.952 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.949+0000 7fafb7fff700 1 --2- 172.21.15.77:0/1603430576 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7faf98063d90 0x7faf98066240 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:56.952 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.949+0000 7fafb7fff700 1 --2- 172.21.15.77:0/1603430576 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafb806bbe0 0x7fafb80c0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:56.952 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.949+0000 7fafb7fff700 1 --2- 172.21.15.77:0/1603430576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafb806bfb0 0x7fafb80bfb90 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:56.952 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.949+0000 7fafb7fff700 1 -- 172.21.15.77:0/1603430576 >> 172.21.15.77:0/1603430576 conn(0x7fafb80cb6b0 msgr2=0x7fafb814b410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:54:56.953 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.950+0000 7fafb7fff700 1 -- 172.21.15.77:0/1603430576 shutdown_connections 2022-04-03T19:54:56.953 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:56.950+0000 7fafb7fff700 1 -- 172.21.15.77:0/1603430576 wait complete. 2022-04-03T19:54:56.953 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 37 2022-04-03T19:54:56.974 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:54:58.975 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:54:59.182 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.182+0000 7f0ac5cec700 1 Processor -- start 2022-04-03T19:54:59.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.182+0000 7f0ac5cec700 1 -- start start 2022-04-03T19:54:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.183+0000 7f0ac5cec700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ac00ba3c0 0x7f0ac0156b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.183+0000 7f0ac5cec700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0ac0157060 0x7f0ac014cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:59.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.183+0000 7f0ac5cec700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac00c3c50 con 0x7f0ac00678b0 2022-04-03T19:54:59.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.183+0000 7f0ac5cec700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ac00c8020 con 0x7f0ac0157060 2022-04-03T19:54:59.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.183+0000 7f0ac5cec700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac0156690 con 0x7f0ac00ba3c0 2022-04-03T19:54:59.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.183+0000 7f0ac4cea700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ac00ba3c0 0x7f0ac0156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:59.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.184+0000 7f0abffff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0ac0157060 0x7f0ac014cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.184+0000 7f0ac54eb700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0ac00678b0 0x7f0ac014d040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38546/0 (socket says 172.21.15.77:38546) 2022-04-03T19:54:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.184+0000 7f0ac4cea700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ac00ba3c0 0x7f0ac0156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60406/0 (socket says 172.21.15.77:60406) 2022-04-03T19:54:59.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.184+0000 7f0ac4cea700 1 -- 172.21.15.77:0/1279254051 learned_addr learned my addr 172.21.15.77:0/1279254051 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:54:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.184+0000 7f0ac4cea700 1 -- 172.21.15.77:0/1279254051 >> v2:172.21.15.186:3300/0 conn(0x7f0ac0157060 msgr2=0x7f0ac014cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:54:59.189 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.184+0000 7f0ac4cea700 1 --2- 172.21.15.77:0/1279254051 >> v2:172.21.15.186:3300/0 conn(0x7f0ac0157060 0x7f0ac014cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:59.190 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.184+0000 7f0ac4cea700 1 -- 172.21.15.77:0/1279254051 >> v1:172.21.15.77:6789/0 conn(0x7f0ac00678b0 legacy=0x7f0ac014d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:59.190 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.184+0000 7f0ac4cea700 1 -- 172.21.15.77:0/1279254051 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac00bf350 con 0x7f0ac00ba3c0 2022-04-03T19:54:59.190 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.184+0000 7f0ac4cea700 1 --2- 172.21.15.77:0/1279254051 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ac00ba3c0 0x7f0ac0156b20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4006fd0 tx=0x7f0ab40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b58b7608d7d2a29 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:54:59.190 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.185+0000 7f0abf7fe700 1 -- 172.21.15.77:0/1279254051 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab4004030 con 0x7f0ac00ba3c0 2022-04-03T19:54:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.185+0000 7f0abf7fe700 1 -- 172.21.15.77:0/1279254051 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab40041d0 con 0x7f0ac00ba3c0 2022-04-03T19:54:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.185+0000 7f0abf7fe700 1 -- 172.21.15.77:0/1279254051 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab4011630 con 0x7f0ac00ba3c0 2022-04-03T19:54:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.185+0000 7f0ac5cec700 1 -- 172.21.15.77:0/1279254051 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ac00ba3c0 msgr2=0x7f0ac0156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.185+0000 7f0ac5cec700 1 --2- 172.21.15.77:0/1279254051 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ac00ba3c0 0x7f0ac0156b20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4006fd0 tx=0x7f0ab40090f0 comp rx=0 tx=0).stop 2022-04-03T19:54:59.191 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.185+0000 7f0ac5cec700 1 -- 172.21.15.77:0/1279254051 shutdown_connections 2022-04-03T19:54:59.192 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.185+0000 7f0ac5cec700 1 --2- 172.21.15.77:0/1279254051 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ac00ba3c0 0x7f0ac0156b20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:59.192 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.185+0000 7f0ac5cec700 1 --2- 172.21.15.77:0/1279254051 >> v2:172.21.15.186:3300/0 conn(0x7f0ac0157060 0x7f0ac014cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:59.192 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.185+0000 7f0ac5cec700 1 -- 172.21.15.77:0/1279254051 >> 172.21.15.77:0/1279254051 conn(0x7f0ac0066da0 msgr2=0x7f0ac00671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:54:59.192 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.185+0000 7f0ac5cec700 1 -- 172.21.15.77:0/1279254051 shutdown_connections 2022-04-03T19:54:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.185+0000 7f0ac5cec700 1 -- 172.21.15.77:0/1279254051 wait complete. 2022-04-03T19:54:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.187+0000 7f0ac5cec700 1 Processor -- start 2022-04-03T19:54:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.187+0000 7f0ac5cec700 1 -- start start 2022-04-03T19:54:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.188+0000 7f0ac5cec700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ac00678b0 0x7f0ac014afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:59.193 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.188+0000 7f0ac5cec700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ac00ba3c0 0x7f0ac0168df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:59.194 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.188+0000 7f0ac5cec700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ac0157060 0x7f0ac0169330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:59.194 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.188+0000 7f0ac5cec700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac014ee00 con 0x7f0ac0157060 2022-04-03T19:54:59.194 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.188+0000 7f0ac5cec700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac0165320 con 0x7f0ac00ba3c0 2022-04-03T19:54:59.194 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.188+0000 7f0ac5cec700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac01689a0 con 0x7f0ac00678b0 2022-04-03T19:54:59.195 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.188+0000 7f0ac4cea700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ac00678b0 0x7f0ac014afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:59.195 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.188+0000 7f0abffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ac00ba3c0 0x7f0ac0168df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:59.195 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.188+0000 7f0abffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ac00ba3c0 0x7f0ac0168df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38310/0 (socket says 172.21.15.77:38310) 2022-04-03T19:54:59.195 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.188+0000 7f0abffff700 1 -- 172.21.15.77:0/3906494485 learned_addr learned my addr 172.21.15.77:0/3906494485 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:54:59.195 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.188+0000 7f0ac54eb700 1 --2- 172.21.15.77:0/3906494485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ac0157060 0x7f0ac0169330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:59.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.189+0000 7f0ac4cea700 1 -- 172.21.15.77:0/3906494485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ac00ba3c0 msgr2=0x7f0ac0168df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:59.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.189+0000 7f0ac4cea700 1 --2- 172.21.15.77:0/3906494485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ac00ba3c0 0x7f0ac0168df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:59.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.189+0000 7f0ac4cea700 1 -- 172.21.15.77:0/3906494485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ac0157060 msgr2=0x7f0ac0169330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:59.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.189+0000 7f0ac4cea700 1 --2- 172.21.15.77:0/3906494485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ac0157060 0x7f0ac0169330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:59.197 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.189+0000 7f0ac4cea700 1 -- 172.21.15.77:0/3906494485 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab4009370 con 0x7f0ac00678b0 2022-04-03T19:54:59.197 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.189+0000 7f0ac4cea700 1 --2- 172.21.15.77:0/3906494485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ac00678b0 0x7f0ac014afd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4009fd0 tx=0x7f0ab4016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:54:59.198 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.189+0000 7f0abcff9700 1 -- 172.21.15.77:0/3906494485 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab40036f0 con 0x7f0ac00678b0 2022-04-03T19:54:59.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.190+0000 7f0abcff9700 1 -- 172.21.15.77:0/3906494485 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab4003870 con 0x7f0ac00678b0 2022-04-03T19:54:59.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.190+0000 7f0ac5cec700 1 -- 172.21.15.77:0/3906494485 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac00c2b00 con 0x7f0ac00678b0 2022-04-03T19:54:59.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.190+0000 7f0abcff9700 1 -- 172.21.15.77:0/3906494485 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab4003c30 con 0x7f0ac00678b0 2022-04-03T19:54:59.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.190+0000 7f0ac5cec700 1 -- 172.21.15.77:0/3906494485 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac01507a0 con 0x7f0ac00678b0 2022-04-03T19:54:59.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.191+0000 7f0ac5cec700 1 -- 172.21.15.77:0/3906494485 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac00c74a0 con 0x7f0ac00678b0 2022-04-03T19:54:59.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.192+0000 7f0abcff9700 1 -- 172.21.15.77:0/3906494485 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f0ab402b030 con 0x7f0ac00678b0 2022-04-03T19:54:59.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.193+0000 7f0abcff9700 1 --2- 172.21.15.77:0/3906494485 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0aa8063ce0 0x7f0aa8066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:54:59.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.193+0000 7f0abcff9700 1 -- 172.21.15.77:0/3906494485 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f0ab40828d0 con 0x7f0ac00678b0 2022-04-03T19:54:59.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.193+0000 7f0abffff700 1 --2- 172.21.15.77:0/3906494485 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0aa8063ce0 0x7f0aa8066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:54:59.201 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.194+0000 7f0abffff700 1 --2- 172.21.15.77:0/3906494485 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0aa8063ce0 0x7f0aa8066190 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0ac014d580 tx=0x7f0aac009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:54:59.201 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.196+0000 7f0abcff9700 1 -- 172.21.15.77:0/3906494485 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f0ab40512e0 con 0x7f0ac00678b0 2022-04-03T19:54:59.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.427+0000 7f0ac5cec700 1 -- 172.21.15.77:0/3906494485 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ac01507a0 con 0x7f0ac00678b0 2022-04-03T19:54:59.428 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.428+0000 7f0abcff9700 1 -- 172.21.15.77:0/3906494485 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f0ac01507a0 con 0x7f0ac00678b0 2022-04-03T19:54:59.429 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:54:59.429 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5463,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:53:46.235063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5039":{"gid":5039,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":27,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:54:59.431 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.430+0000 7f0ac5cec700 1 -- 172.21.15.77:0/3906494485 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0aa8063ce0 msgr2=0x7f0aa8066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:59.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.431+0000 7f0ac5cec700 1 --2- 172.21.15.77:0/3906494485 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0aa8063ce0 0x7f0aa8066190 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0ac014d580 tx=0x7f0aac009060 comp rx=0 tx=0).stop 2022-04-03T19:54:59.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.431+0000 7f0ac5cec700 1 -- 172.21.15.77:0/3906494485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ac00678b0 msgr2=0x7f0ac014afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:54:59.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.431+0000 7f0ac5cec700 1 --2- 172.21.15.77:0/3906494485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ac00678b0 0x7f0ac014afd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4009fd0 tx=0x7f0ab4016000 comp rx=0 tx=0).stop 2022-04-03T19:54:59.432 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.431+0000 7f0ac5cec700 1 -- 172.21.15.77:0/3906494485 shutdown_connections 2022-04-03T19:54:59.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.431+0000 7f0ac5cec700 1 --2- 172.21.15.77:0/3906494485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ac0157060 0x7f0ac0169330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:59.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.431+0000 7f0ac5cec700 1 --2- 172.21.15.77:0/3906494485 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0aa8063ce0 0x7f0aa8066190 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:59.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.431+0000 7f0ac5cec700 1 --2- 172.21.15.77:0/3906494485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ac00678b0 0x7f0ac014afd0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:59.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.431+0000 7f0ac5cec700 1 --2- 172.21.15.77:0/3906494485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ac00ba3c0 0x7f0ac0168df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:54:59.433 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.431+0000 7f0ac5cec700 1 -- 172.21.15.77:0/3906494485 >> 172.21.15.77:0/3906494485 conn(0x7f0ac0066da0 msgr2=0x7f0ac014bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:54:59.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.431+0000 7f0ac5cec700 1 -- 172.21.15.77:0/3906494485 shutdown_connections 2022-04-03T19:54:59.434 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:54:59.431+0000 7f0ac5cec700 1 -- 172.21.15.77:0/3906494485 wait complete. 2022-04-03T19:54:59.434 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 37 2022-04-03T19:54:59.455 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:55:01.456 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:55:01.663 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.663+0000 7f4e7af2b700 1 Processor -- start 2022-04-03T19:55:01.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.663+0000 7f4e7af2b700 1 -- start start 2022-04-03T19:55:01.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.664+0000 7f4e7af2b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7406bbe0 0x7f4e741584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:01.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.664+0000 7f4e7af2b700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4e7406bfb0 0x7f4e7406d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:01.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.664+0000 7f4e7af2b700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e7406d260 con 0x7f4e7406c500 2022-04-03T19:55:01.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.664+0000 7f4e7af2b700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e74159450 con 0x7f4e7406bfb0 2022-04-03T19:55:01.666 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.664+0000 7f4e7af2b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e740c6330 con 0x7f4e7406bbe0 2022-04-03T19:55:01.666 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.665+0000 7f4e79f29700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7406bbe0 0x7f4e741584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:01.666 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.665+0000 7f4e79f29700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7406bbe0 0x7f4e741584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60410/0 (socket says 172.21.15.77:60410) 2022-04-03T19:55:01.666 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.665+0000 7f4e79f29700 1 -- 172.21.15.77:0/2753482300 learned_addr learned my addr 172.21.15.77:0/2753482300 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:01.667 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.665+0000 7f4e79728700 1 --2- 172.21.15.77:0/2753482300 >> v2:172.21.15.186:3300/0 conn(0x7f4e7406bfb0 0x7f4e7406d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:01.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.665+0000 7f4e79f29700 1 -- 172.21.15.77:0/2753482300 >> v2:172.21.15.186:3300/0 conn(0x7f4e7406bfb0 msgr2=0x7f4e7406d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:01.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.665+0000 7f4e79f29700 1 --2- 172.21.15.77:0/2753482300 >> v2:172.21.15.186:3300/0 conn(0x7f4e7406bfb0 0x7f4e7406d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:01.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.665+0000 7f4e79f29700 1 -- 172.21.15.77:0/2753482300 >> v1:172.21.15.186:6789/0 conn(0x7f4e7406c500 legacy=0x7f4e7414eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:01.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.665+0000 7f4e79f29700 1 -- 172.21.15.77:0/2753482300 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e74158bb0 con 0x7f4e7406bbe0 2022-04-03T19:55:01.670 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.665+0000 7f4e79f29700 1 --2- 172.21.15.77:0/2753482300 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7406bbe0 0x7f4e741584b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4e70006fd0 tx=0x7f4e700090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=836d2add5a7fcd3c server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:01.670 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.666+0000 7f4e78f27700 1 -- 172.21.15.77:0/2753482300 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e70004030 con 0x7f4e7406bbe0 2022-04-03T19:55:01.670 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.666+0000 7f4e78f27700 1 -- 172.21.15.77:0/2753482300 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e700041d0 con 0x7f4e7406bbe0 2022-04-03T19:55:01.670 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.666+0000 7f4e78f27700 1 -- 172.21.15.77:0/2753482300 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e70011630 con 0x7f4e7406bbe0 2022-04-03T19:55:01.670 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.666+0000 7f4e7af2b700 1 -- 172.21.15.77:0/2753482300 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7406bbe0 msgr2=0x7f4e741584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:01.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.666+0000 7f4e7af2b700 1 --2- 172.21.15.77:0/2753482300 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7406bbe0 0x7f4e741584b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4e70006fd0 tx=0x7f4e700090f0 comp rx=0 tx=0).stop 2022-04-03T19:55:01.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.666+0000 7f4e7af2b700 1 -- 172.21.15.77:0/2753482300 shutdown_connections 2022-04-03T19:55:01.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.666+0000 7f4e7af2b700 1 --2- 172.21.15.77:0/2753482300 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7406bbe0 0x7f4e741584b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:01.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.666+0000 7f4e7af2b700 1 --2- 172.21.15.77:0/2753482300 >> v2:172.21.15.186:3300/0 conn(0x7f4e7406bfb0 0x7f4e7406d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:01.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.666+0000 7f4e7af2b700 1 -- 172.21.15.77:0/2753482300 >> 172.21.15.77:0/2753482300 conn(0x7f4e740cb6b0 msgr2=0x7f4e740cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:01.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.666+0000 7f4e7af2b700 1 -- 172.21.15.77:0/2753482300 shutdown_connections 2022-04-03T19:55:01.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.666+0000 7f4e7af2b700 1 -- 172.21.15.77:0/2753482300 wait complete. 2022-04-03T19:55:01.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.668+0000 7f4e7af2b700 1 Processor -- start 2022-04-03T19:55:01.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.668+0000 7f4e7af2b700 1 -- start start 2022-04-03T19:55:01.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.669+0000 7f4e7af2b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e7406bbe0 0x7f4e74151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:01.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.669+0000 7f4e7af2b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e7406bfb0 0x7f4e74157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:01.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.669+0000 7f4e7af2b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7406c500 0x7f4e740ca490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:01.676 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.669+0000 7f4e7af2b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e741659a0 con 0x7f4e7406bfb0 2022-04-03T19:55:01.676 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.669+0000 7f4e7af2b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e741586a0 con 0x7f4e7406bbe0 2022-04-03T19:55:01.676 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.669+0000 7f4e7af2b700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e74164ad0 con 0x7f4e7406c500 2022-04-03T19:55:01.676 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.669+0000 7f4e79f29700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e7406bbe0 0x7f4e74151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:01.677 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.670+0000 7f4e79728700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e7406bfb0 0x7f4e74157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:01.677 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.670+0000 7f4e79f29700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e7406bbe0 0x7f4e74151f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38314/0 (socket says 172.21.15.77:38314) 2022-04-03T19:55:01.677 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.670+0000 7f4e79f29700 1 -- 172.21.15.77:0/2492334917 learned_addr learned my addr 172.21.15.77:0/2492334917 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:01.677 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.670+0000 7f4e7a72a700 1 --2- 172.21.15.77:0/2492334917 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7406c500 0x7f4e740ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:01.677 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.670+0000 7f4e79f29700 1 -- 172.21.15.77:0/2492334917 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7406c500 msgr2=0x7f4e740ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:01.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.670+0000 7f4e79f29700 1 --2- 172.21.15.77:0/2492334917 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7406c500 0x7f4e740ca490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:01.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.670+0000 7f4e79f29700 1 -- 172.21.15.77:0/2492334917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e7406bfb0 msgr2=0x7f4e74157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:01.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.670+0000 7f4e79f29700 1 --2- 172.21.15.77:0/2492334917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e7406bfb0 0x7f4e74157dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:01.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.670+0000 7f4e79f29700 1 -- 172.21.15.77:0/2492334917 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e70009370 con 0x7f4e7406bbe0 2022-04-03T19:55:01.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.670+0000 7f4e79f29700 1 --2- 172.21.15.77:0/2492334917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e7406bbe0 0x7f4e74151f90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4e70008000 tx=0x7f4e70016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:01.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.670+0000 7f4e61ffb700 1 -- 172.21.15.77:0/2492334917 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e700036f0 con 0x7f4e7406bbe0 2022-04-03T19:55:01.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.671+0000 7f4e7af2b700 1 -- 172.21.15.77:0/2492334917 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e74164d90 con 0x7f4e7406bbe0 2022-04-03T19:55:01.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.671+0000 7f4e61ffb700 1 -- 172.21.15.77:0/2492334917 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e70003850 con 0x7f4e7406bbe0 2022-04-03T19:55:01.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.671+0000 7f4e61ffb700 1 -- 172.21.15.77:0/2492334917 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e70003c10 con 0x7f4e7406bbe0 2022-04-03T19:55:01.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.671+0000 7f4e7af2b700 1 -- 172.21.15.77:0/2492334917 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e74165350 con 0x7f4e7406bbe0 2022-04-03T19:55:01.680 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.672+0000 7f4e7af2b700 1 -- 172.21.15.77:0/2492334917 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e74157bb0 con 0x7f4e7406bbe0 2022-04-03T19:55:01.680 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.673+0000 7f4e61ffb700 1 -- 172.21.15.77:0/2492334917 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f4e7002b030 con 0x7f4e7406bbe0 2022-04-03T19:55:01.680 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.674+0000 7f4e61ffb700 1 --2- 172.21.15.77:0/2492334917 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4e54063d10 0x7f4e540661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:01.680 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.674+0000 7f4e61ffb700 1 -- 172.21.15.77:0/2492334917 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f4e700820d0 con 0x7f4e7406bbe0 2022-04-03T19:55:01.681 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.674+0000 7f4e79728700 1 --2- 172.21.15.77:0/2492334917 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4e54063d10 0x7f4e540661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:01.681 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.675+0000 7f4e79728700 1 --2- 172.21.15.77:0/2492334917 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4e54063d10 0x7f4e540661c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4e740b18d0 tx=0x7f4e64009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:01.681 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.677+0000 7f4e61ffb700 1 -- 172.21.15.77:0/2492334917 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f4e70050b40 con 0x7f4e7406bbe0 2022-04-03T19:55:01.912 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.911+0000 7f4e7af2b700 1 -- 172.21.15.77:0/2492334917 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e740c7ec0 con 0x7f4e7406bbe0 2022-04-03T19:55:01.913 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.912+0000 7f4e61ffb700 1 -- 172.21.15.77:0/2492334917 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f4e70054160 con 0x7f4e7406bbe0 2022-04-03T19:55:01.914 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:55:01.914 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5463,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:53:46.235063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5039":{"gid":5039,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":27,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:55:01.916 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.914+0000 7f4e7af2b700 1 -- 172.21.15.77:0/2492334917 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4e54063d10 msgr2=0x7f4e540661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:01.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.914+0000 7f4e7af2b700 1 --2- 172.21.15.77:0/2492334917 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4e54063d10 0x7f4e540661c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4e740b18d0 tx=0x7f4e64009060 comp rx=0 tx=0).stop 2022-04-03T19:55:01.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.914+0000 7f4e7af2b700 1 -- 172.21.15.77:0/2492334917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e7406bbe0 msgr2=0x7f4e74151f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:01.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.914+0000 7f4e7af2b700 1 --2- 172.21.15.77:0/2492334917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e7406bbe0 0x7f4e74151f90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4e70008000 tx=0x7f4e70016000 comp rx=0 tx=0).stop 2022-04-03T19:55:01.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.914+0000 7f4e7af2b700 1 -- 172.21.15.77:0/2492334917 shutdown_connections 2022-04-03T19:55:01.917 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.914+0000 7f4e7af2b700 1 --2- 172.21.15.77:0/2492334917 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4e54063d10 0x7f4e540661c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:01.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.914+0000 7f4e7af2b700 1 --2- 172.21.15.77:0/2492334917 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7406c500 0x7f4e740ca490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:01.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.914+0000 7f4e7af2b700 1 --2- 172.21.15.77:0/2492334917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e7406bbe0 0x7f4e74151f90 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:01.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.914+0000 7f4e7af2b700 1 --2- 172.21.15.77:0/2492334917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4e7406bfb0 0x7f4e74157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:01.918 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.915+0000 7f4e7af2b700 1 -- 172.21.15.77:0/2492334917 >> 172.21.15.77:0/2492334917 conn(0x7f4e740cb6b0 msgr2=0x7f4e7414a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:01.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.915+0000 7f4e7af2b700 1 -- 172.21.15.77:0/2492334917 shutdown_connections 2022-04-03T19:55:01.919 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:01.915+0000 7f4e7af2b700 1 -- 172.21.15.77:0/2492334917 wait complete. 2022-04-03T19:55:01.919 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 37 2022-04-03T19:55:01.940 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:55:03.941 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:55:04.152 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.150+0000 7f594b59e700 1 Processor -- start 2022-04-03T19:55:04.152 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.150+0000 7f594b59e700 1 -- start start 2022-04-03T19:55:04.154 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.151+0000 7f594b59e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f594c14f520 0x7f594c0c5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:04.154 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.151+0000 7f594b59e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f594c158e60 0x7f594c06bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:04.154 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.151+0000 7f594b59e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594c06de50 con 0x7f594c14f150 2022-04-03T19:55:04.154 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.151+0000 7f594b59e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f594c0c5430 con 0x7f594c14f520 2022-04-03T19:55:04.155 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.151+0000 7f594b59e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f594c06e2e0 con 0x7f594c158e60 2022-04-03T19:55:04.155 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.152+0000 7f5949d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f594c14f520 0x7f594c0c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:04.155 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.152+0000 7f594ad9d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f594c158e60 0x7f594c06bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:04.155 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.152+0000 7f5949d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f594c14f520 0x7f594c0c5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43120/0 (socket says 172.21.15.77:43120) 2022-04-03T19:55:04.156 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.152+0000 7f5949d9b700 1 -- 172.21.15.77:0/3461137595 learned_addr learned my addr 172.21.15.77:0/3461137595 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:04.156 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.152+0000 7f594a59c700 1 --1- 172.21.15.77:0/3461137595 >> v1:172.21.15.77:6789/0 conn(0x7f594c14f150 0x7f594c0c5f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38548/0 (socket says 172.21.15.77:38548) 2022-04-03T19:55:04.159 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.152+0000 7f5949d9b700 1 -- 172.21.15.77:0/3461137595 >> v2:172.21.15.186:3300/0 conn(0x7f594c158e60 msgr2=0x7f594c06bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:04.159 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.152+0000 7f5949d9b700 1 --2- 172.21.15.77:0/3461137595 >> v2:172.21.15.186:3300/0 conn(0x7f594c158e60 0x7f594c06bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:04.159 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.152+0000 7f5949d9b700 1 -- 172.21.15.77:0/3461137595 >> v1:172.21.15.77:6789/0 conn(0x7f594c14f150 legacy=0x7f594c0c5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:04.160 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.152+0000 7f5949d9b700 1 -- 172.21.15.77:0/3461137595 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594c06cf60 con 0x7f594c14f520 2022-04-03T19:55:04.160 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.152+0000 7f594959a700 1 -- 172.21.15.77:0/3461137595 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321470185 0 0) 0x7f594c06de50 con 0x7f594c14f150 2022-04-03T19:55:04.160 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.153+0000 7f5949d9b700 1 --2- 172.21.15.77:0/3461137595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f594c14f520 0x7f594c0c5660 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f594c152b00 tx=0x7f59400090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a135d83727be3b server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:04.160 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.153+0000 7f594959a700 1 -- 172.21.15.77:0/3461137595 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5940004030 con 0x7f594c14f520 2022-04-03T19:55:04.161 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.153+0000 7f594959a700 1 -- 172.21.15.77:0/3461137595 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5940004190 con 0x7f594c14f520 2022-04-03T19:55:04.161 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.153+0000 7f594959a700 1 -- 172.21.15.77:0/3461137595 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5940011630 con 0x7f594c14f520 2022-04-03T19:55:04.161 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.153+0000 7f594b59e700 1 -- 172.21.15.77:0/3461137595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f594c14f520 msgr2=0x7f594c0c5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:04.161 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.153+0000 7f594b59e700 1 --2- 172.21.15.77:0/3461137595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f594c14f520 0x7f594c0c5660 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f594c152b00 tx=0x7f59400090f0 comp rx=0 tx=0).stop 2022-04-03T19:55:04.162 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.153+0000 7f594b59e700 1 -- 172.21.15.77:0/3461137595 shutdown_connections 2022-04-03T19:55:04.162 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.153+0000 7f594b59e700 1 --2- 172.21.15.77:0/3461137595 >> v2:172.21.15.186:3300/0 conn(0x7f594c158e60 0x7f594c06bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:04.162 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.153+0000 7f594b59e700 1 --2- 172.21.15.77:0/3461137595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f594c14f520 0x7f594c0c5660 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:04.162 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.153+0000 7f594b59e700 1 -- 172.21.15.77:0/3461137595 >> 172.21.15.77:0/3461137595 conn(0x7f594c0cb640 msgr2=0x7f594c0cba40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:04.162 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.154+0000 7f594b59e700 1 -- 172.21.15.77:0/3461137595 shutdown_connections 2022-04-03T19:55:04.163 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.154+0000 7f594b59e700 1 -- 172.21.15.77:0/3461137595 wait complete. 2022-04-03T19:55:04.163 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.155+0000 7f594b59e700 1 Processor -- start 2022-04-03T19:55:04.163 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.155+0000 7f594b59e700 1 -- start start 2022-04-03T19:55:04.163 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.156+0000 7f594b59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f594c14f150 0x7f594c156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:04.163 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.156+0000 7f594b59e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f594c158e60 0x7f594c157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:04.164 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.156+0000 7f594b59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f594c146820 0x7f594c146c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:04.164 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.156+0000 7f594b59e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c151270 con 0x7f594c146820 2022-04-03T19:55:04.164 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.156+0000 7f594b59e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c152cb0 con 0x7f594c158e60 2022-04-03T19:55:04.164 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.156+0000 7f594b59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594c152390 con 0x7f594c14f150 2022-04-03T19:55:04.165 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.156+0000 7f594ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f594c146820 0x7f594c146c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:04.165 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.156+0000 7f594ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f594c146820 0x7f594c146c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43122/0 (socket says 172.21.15.77:43122) 2022-04-03T19:55:04.165 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.156+0000 7f594ad9d700 1 -- 172.21.15.77:0/3143651976 learned_addr learned my addr 172.21.15.77:0/3143651976 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:04.165 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.156+0000 7f5949d9b700 1 --2- 172.21.15.77:0/3143651976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f594c158e60 0x7f594c157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:04.165 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.156+0000 7f594a59c700 1 --2- 172.21.15.77:0/3143651976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f594c14f150 0x7f594c156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:04.166 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.157+0000 7f594ad9d700 1 -- 172.21.15.77:0/3143651976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f594c14f150 msgr2=0x7f594c156ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:04.166 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.157+0000 7f594ad9d700 1 --2- 172.21.15.77:0/3143651976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f594c14f150 0x7f594c156ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:04.166 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.157+0000 7f594ad9d700 1 -- 172.21.15.77:0/3143651976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f594c158e60 msgr2=0x7f594c157400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:04.166 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.157+0000 7f594ad9d700 1 --2- 172.21.15.77:0/3143651976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f594c158e60 0x7f594c157400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:04.166 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.157+0000 7f594ad9d700 1 -- 172.21.15.77:0/3143651976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5940009370 con 0x7f594c146820 2022-04-03T19:55:04.167 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.157+0000 7f594ad9d700 1 --2- 172.21.15.77:0/3143651976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f594c146820 0x7f594c146c90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5944009df0 tx=0x7f594400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:04.167 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.157+0000 7f593a7fc700 1 -- 172.21.15.77:0/3143651976 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594400cb30 con 0x7f594c146820 2022-04-03T19:55:04.167 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.157+0000 7f593a7fc700 1 -- 172.21.15.77:0/3143651976 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594400f040 con 0x7f594c146820 2022-04-03T19:55:04.167 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.157+0000 7f593a7fc700 1 -- 172.21.15.77:0/3143651976 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5944014480 con 0x7f594c146820 2022-04-03T19:55:04.168 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.157+0000 7f594b59e700 1 -- 172.21.15.77:0/3143651976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f594c152650 con 0x7f594c146820 2022-04-03T19:55:04.168 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.158+0000 7f594b59e700 1 -- 172.21.15.77:0/3143651976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f594c150790 con 0x7f594c146820 2022-04-03T19:55:04.169 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.158+0000 7f594b59e700 1 -- 172.21.15.77:0/3143651976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f594c04b050 con 0x7f594c146820 2022-04-03T19:55:04.169 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.159+0000 7f593a7fc700 1 -- 172.21.15.77:0/3143651976 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f5944014830 con 0x7f594c146820 2022-04-03T19:55:04.170 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.160+0000 7f593a7fc700 1 --2- 172.21.15.77:0/3143651976 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5928063ce0 0x7f5928066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:04.170 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.160+0000 7f593a7fc700 1 -- 172.21.15.77:0/3143651976 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f5944081710 con 0x7f594c146820 2022-04-03T19:55:04.170 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.161+0000 7f594a59c700 1 --2- 172.21.15.77:0/3143651976 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5928063ce0 0x7f5928066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:04.170 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.161+0000 7f594a59c700 1 --2- 172.21.15.77:0/3143651976 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5928063ce0 0x7f5928066190 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f593c007690 tx=0x7f593c007bb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:04.171 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.165+0000 7f593a7fc700 1 -- 172.21.15.77:0/3143651976 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f5944050010 con 0x7f594c146820 2022-04-03T19:55:04.400 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.398+0000 7f594b59e700 1 -- 172.21.15.77:0/3143651976 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f594c06c810 con 0x7f594c146820 2022-04-03T19:55:04.401 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.399+0000 7f593a7fc700 1 -- 172.21.15.77:0/3143651976 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f5944018090 con 0x7f594c146820 2022-04-03T19:55:04.401 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:55:04.402 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5463,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:53:46.235063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5039":{"gid":5039,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":27,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:55:04.403 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.401+0000 7f594b59e700 1 -- 172.21.15.77:0/3143651976 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5928063ce0 msgr2=0x7f5928066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:04.404 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.401+0000 7f594b59e700 1 --2- 172.21.15.77:0/3143651976 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5928063ce0 0x7f5928066190 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f593c007690 tx=0x7f593c007bb0 comp rx=0 tx=0).stop 2022-04-03T19:55:04.404 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.401+0000 7f594b59e700 1 -- 172.21.15.77:0/3143651976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f594c146820 msgr2=0x7f594c146c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:04.404 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.401+0000 7f594b59e700 1 --2- 172.21.15.77:0/3143651976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f594c146820 0x7f594c146c90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5944009df0 tx=0x7f594400b040 comp rx=0 tx=0).stop 2022-04-03T19:55:04.405 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.401+0000 7f594b59e700 1 -- 172.21.15.77:0/3143651976 shutdown_connections 2022-04-03T19:55:04.405 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.401+0000 7f594b59e700 1 --2- 172.21.15.77:0/3143651976 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f594c146820 0x7f594c146c90 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:04.405 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.401+0000 7f594b59e700 1 --2- 172.21.15.77:0/3143651976 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5928063ce0 0x7f5928066190 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:04.406 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.401+0000 7f594b59e700 1 --2- 172.21.15.77:0/3143651976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f594c14f150 0x7f594c156ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:04.406 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.401+0000 7f594b59e700 1 --2- 172.21.15.77:0/3143651976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f594c158e60 0x7f594c157400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:04.406 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.401+0000 7f594b59e700 1 -- 172.21.15.77:0/3143651976 >> 172.21.15.77:0/3143651976 conn(0x7f594c0cb640 msgr2=0x7f594c14d180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:04.407 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.401+0000 7f594b59e700 1 -- 172.21.15.77:0/3143651976 shutdown_connections 2022-04-03T19:55:04.407 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:04.401+0000 7f594b59e700 1 -- 172.21.15.77:0/3143651976 wait complete. 2022-04-03T19:55:04.408 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 37 2022-04-03T19:55:04.428 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:55:06.429 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:55:06.636 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.635+0000 7fa62298a700 1 Processor -- start 2022-04-03T19:55:06.637 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.635+0000 7fa62298a700 1 -- start start 2022-04-03T19:55:06.639 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.636+0000 7fa62298a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa61c0c1170 0x7fa61c1527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:06.639 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.636+0000 7fa62298a700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa61c0c3b10 0x7fa61c15c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:06.639 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.636+0000 7fa62298a700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa61c0c3710 con 0x7fa61c155370 2022-04-03T19:55:06.639 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.636+0000 7fa62298a700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa61c152330 con 0x7fa61c0c1170 2022-04-03T19:55:06.640 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.636+0000 7fa62298a700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa61c0c1540 con 0x7fa61c0c3b10 2022-04-03T19:55:06.640 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.636+0000 7fa621988700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa61c0c1170 0x7fa61c1527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:06.640 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.636+0000 7fa621988700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa61c0c1170 0x7fa61c1527c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43124/0 (socket says 172.21.15.77:43124) 2022-04-03T19:55:06.640 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.636+0000 7fa621988700 1 -- 172.21.15.77:0/1205690528 learned_addr learned my addr 172.21.15.77:0/1205690528 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:06.640 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.636+0000 7fa622189700 1 --2- 172.21.15.77:0/1205690528 >> v2:172.21.15.186:3300/0 conn(0x7fa61c0c3b10 0x7fa61c15c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:06.641 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.637+0000 7fa621988700 1 -- 172.21.15.77:0/1205690528 >> v2:172.21.15.186:3300/0 conn(0x7fa61c0c3b10 msgr2=0x7fa61c15c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:06.641 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.637+0000 7fa621988700 1 --2- 172.21.15.77:0/1205690528 >> v2:172.21.15.186:3300/0 conn(0x7fa61c0c3b10 0x7fa61c15c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:06.641 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.637+0000 7fa621988700 1 -- 172.21.15.77:0/1205690528 >> v1:172.21.15.186:6789/0 conn(0x7fa61c155370 legacy=0x7fa61c069110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:06.641 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.637+0000 7fa621988700 1 -- 172.21.15.77:0/1205690528 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61c0c4270 con 0x7fa61c0c1170 2022-04-03T19:55:06.642 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.637+0000 7fa621988700 1 --2- 172.21.15.77:0/1205690528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa61c0c1170 0x7fa61c1527c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa61c0690a0 tx=0x7fa6180090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38eb9cad8548d056 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:06.642 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.637+0000 7fa620986700 1 -- 172.21.15.77:0/1205690528 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618004030 con 0x7fa61c0c1170 2022-04-03T19:55:06.642 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.637+0000 7fa620986700 1 -- 172.21.15.77:0/1205690528 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa618004190 con 0x7fa61c0c1170 2022-04-03T19:55:06.642 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.637+0000 7fa620986700 1 -- 172.21.15.77:0/1205690528 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618011630 con 0x7fa61c0c1170 2022-04-03T19:55:06.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.638+0000 7fa62298a700 1 -- 172.21.15.77:0/1205690528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa61c0c1170 msgr2=0x7fa61c1527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:06.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.638+0000 7fa62298a700 1 --2- 172.21.15.77:0/1205690528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa61c0c1170 0x7fa61c1527c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa61c0690a0 tx=0x7fa6180090f0 comp rx=0 tx=0).stop 2022-04-03T19:55:06.646 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.638+0000 7fa62298a700 1 -- 172.21.15.77:0/1205690528 shutdown_connections 2022-04-03T19:55:06.647 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.638+0000 7fa62298a700 1 --2- 172.21.15.77:0/1205690528 >> v2:172.21.15.186:3300/0 conn(0x7fa61c0c3b10 0x7fa61c15c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:06.647 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.638+0000 7fa62298a700 1 --2- 172.21.15.77:0/1205690528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa61c0c1170 0x7fa61c1527c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:06.647 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.638+0000 7fa62298a700 1 -- 172.21.15.77:0/1205690528 >> 172.21.15.77:0/1205690528 conn(0x7fa61c066da0 msgr2=0x7fa61c0671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:06.647 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.638+0000 7fa62298a700 1 -- 172.21.15.77:0/1205690528 shutdown_connections 2022-04-03T19:55:06.648 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.638+0000 7fa62298a700 1 -- 172.21.15.77:0/1205690528 wait complete. 2022-04-03T19:55:06.648 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.640+0000 7fa62298a700 1 Processor -- start 2022-04-03T19:55:06.648 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.640+0000 7fa62298a700 1 -- start start 2022-04-03T19:55:06.648 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.640+0000 7fa62298a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa61c0c1170 0x7fa61c150ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:06.648 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.641+0000 7fa62298a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa61c0c3b10 0x7fa61c1460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:06.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.641+0000 7fa62298a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa61c155370 0x7fa61c143e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:06.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.641+0000 7fa62298a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c144f60 con 0x7fa61c0c1170 2022-04-03T19:55:06.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.641+0000 7fa62298a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c0c70a0 con 0x7fa61c155370 2022-04-03T19:55:06.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.641+0000 7fa62298a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c0c6c10 con 0x7fa61c0c3b10 2022-04-03T19:55:06.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.641+0000 7fa621988700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa61c0c1170 0x7fa61c150ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:06.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.641+0000 7fa621988700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa61c0c1170 0x7fa61c150ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43128/0 (socket says 172.21.15.77:43128) 2022-04-03T19:55:06.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.641+0000 7fa621988700 1 -- 172.21.15.77:0/3361415758 learned_addr learned my addr 172.21.15.77:0/3361415758 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:06.650 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.641+0000 7fa622189700 1 --2- 172.21.15.77:0/3361415758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa61c155370 0x7fa61c143e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:06.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.641+0000 7fa621187700 1 --2- 172.21.15.77:0/3361415758 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa61c0c3b10 0x7fa61c1460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:06.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.641+0000 7fa621988700 1 -- 172.21.15.77:0/3361415758 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa61c0c3b10 msgr2=0x7fa61c1460e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:06.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.641+0000 7fa621988700 1 --2- 172.21.15.77:0/3361415758 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa61c0c3b10 0x7fa61c1460e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:06.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.641+0000 7fa621988700 1 -- 172.21.15.77:0/3361415758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa61c155370 msgr2=0x7fa61c143e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:06.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.641+0000 7fa621988700 1 --2- 172.21.15.77:0/3361415758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa61c155370 0x7fa61c143e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:06.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.641+0000 7fa621988700 1 -- 172.21.15.77:0/3361415758 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa618009370 con 0x7fa61c0c1170 2022-04-03T19:55:06.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.641+0000 7fa621187700 1 --2- 172.21.15.77:0/3361415758 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa61c0c3b10 0x7fa61c1460e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:55:06.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.642+0000 7fa621988700 1 --2- 172.21.15.77:0/3361415758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa61c0c1170 0x7fa61c150ef0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa61c071610 tx=0x7fa618016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:06.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.642+0000 7fa609ffb700 1 -- 172.21.15.77:0/3361415758 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6180036f0 con 0x7fa61c0c1170 2022-04-03T19:55:06.653 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.642+0000 7fa609ffb700 1 -- 172.21.15.77:0/3361415758 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa618003850 con 0x7fa61c0c1170 2022-04-03T19:55:06.653 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.642+0000 7fa62298a700 1 -- 172.21.15.77:0/3361415758 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa61c0c6040 con 0x7fa61c0c1170 2022-04-03T19:55:06.653 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.642+0000 7fa609ffb700 1 -- 172.21.15.77:0/3361415758 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618003bf0 con 0x7fa61c0c1170 2022-04-03T19:55:06.653 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.642+0000 7fa62298a700 1 -- 172.21.15.77:0/3361415758 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa61c0c62e0 con 0x7fa61c0c1170 2022-04-03T19:55:06.653 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.643+0000 7fa62298a700 1 -- 172.21.15.77:0/3361415758 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa61c155fa0 con 0x7fa61c0c1170 2022-04-03T19:55:06.654 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.644+0000 7fa609ffb700 1 -- 172.21.15.77:0/3361415758 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fa618017020 con 0x7fa61c0c1170 2022-04-03T19:55:06.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.645+0000 7fa609ffb700 1 --2- 172.21.15.77:0/3361415758 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa5fc063d10 0x7fa5fc0661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:06.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.645+0000 7fa609ffb700 1 -- 172.21.15.77:0/3361415758 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7fa618082350 con 0x7fa61c0c1170 2022-04-03T19:55:06.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.645+0000 7fa621187700 1 --2- 172.21.15.77:0/3361415758 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa5fc063d10 0x7fa5fc0661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:06.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.646+0000 7fa621187700 1 --2- 172.21.15.77:0/3361415758 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa5fc063d10 0x7fa5fc0661c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa61c0be2f0 tx=0x7fa60c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:06.656 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.650+0000 7fa609ffb700 1 -- 172.21.15.77:0/3361415758 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fa618050d40 con 0x7fa61c0c1170 2022-04-03T19:55:06.880 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.879+0000 7fa62298a700 1 -- 172.21.15.77:0/3361415758 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa61c0c6ef0 con 0x7fa61c0c1170 2022-04-03T19:55:06.881 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.879+0000 7fa609ffb700 1 -- 172.21.15.77:0/3361415758 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6110 (secure 0 0 0) 0x7fa618054360 con 0x7fa61c0c1170 2022-04-03T19:55:06.881 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:55:06.882 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5463,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:53:46.235063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5039},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5039":{"gid":5039,"name":"d","rank":0,"incarnation":32,"state":"up:active","state_seq":27,"addr":"172.21.15.186:6837/480451242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":480451242},{"type":"v1","addr":"172.21.15.186:6837","nonce":480451242}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:55:06.883 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.882+0000 7fa62298a700 1 -- 172.21.15.77:0/3361415758 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa5fc063d10 msgr2=0x7fa5fc0661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:06.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.882+0000 7fa62298a700 1 --2- 172.21.15.77:0/3361415758 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa5fc063d10 0x7fa5fc0661c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa61c0be2f0 tx=0x7fa60c009040 comp rx=0 tx=0).stop 2022-04-03T19:55:06.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.882+0000 7fa62298a700 1 -- 172.21.15.77:0/3361415758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa61c0c1170 msgr2=0x7fa61c150ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:06.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.882+0000 7fa62298a700 1 --2- 172.21.15.77:0/3361415758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa61c0c1170 0x7fa61c150ef0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa61c071610 tx=0x7fa618016000 comp rx=0 tx=0).stop 2022-04-03T19:55:06.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.882+0000 7fa62298a700 1 -- 172.21.15.77:0/3361415758 shutdown_connections 2022-04-03T19:55:06.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.882+0000 7fa62298a700 1 --2- 172.21.15.77:0/3361415758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa61c155370 0x7fa61c143e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:06.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.882+0000 7fa62298a700 1 --2- 172.21.15.77:0/3361415758 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa61c0c1170 0x7fa61c150ef0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:06.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.882+0000 7fa62298a700 1 --2- 172.21.15.77:0/3361415758 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa5fc063d10 0x7fa5fc0661c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:06.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.882+0000 7fa62298a700 1 --2- 172.21.15.77:0/3361415758 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa61c0c3b10 0x7fa61c1460e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:06.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.882+0000 7fa62298a700 1 -- 172.21.15.77:0/3361415758 >> 172.21.15.77:0/3361415758 conn(0x7fa61c066da0 msgr2=0x7fa61c0690f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:06.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.882+0000 7fa62298a700 1 -- 172.21.15.77:0/3361415758 shutdown_connections 2022-04-03T19:55:06.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:06.882+0000 7fa62298a700 1 -- 172.21.15.77:0/3361415758 wait complete. 2022-04-03T19:55:06.887 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 37 2022-04-03T19:55:06.908 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:55:08.909 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:55:09.114 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.112+0000 7f9090ebc700 1 Processor -- start 2022-04-03T19:55:09.114 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.112+0000 7f9090ebc700 1 -- start start 2022-04-03T19:55:09.116 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.114+0000 7f9090ebc700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f908c0c1170 0x7f908c1527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:09.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.114+0000 7f9090ebc700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f908c155370 0x7f908c069110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:09.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.114+0000 7f9090ebc700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908c0c3710 con 0x7f908c0c3b10 2022-04-03T19:55:09.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.114+0000 7f9090ebc700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f908c152330 con 0x7f908c155370 2022-04-03T19:55:09.117 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.114+0000 7f9090ebc700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f908c0c1540 con 0x7f908c0c1170 2022-04-03T19:55:09.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.114+0000 7f908affd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f908c155370 0x7f908c069110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:09.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.114+0000 7f908b7fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f908c0c1170 0x7f908c1527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:09.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.114+0000 7f908b7fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f908c0c1170 0x7f908c1527c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38328/0 (socket says 172.21.15.77:38328) 2022-04-03T19:55:09.118 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.114+0000 7f908affd700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f908c155370 0x7f908c069110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43130/0 (socket says 172.21.15.77:43130) 2022-04-03T19:55:09.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.114+0000 7f908b7fe700 1 -- 172.21.15.77:0/1968951155 learned_addr learned my addr 172.21.15.77:0/1968951155 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:09.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.114+0000 7f908bfff700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f908c0c3b10 0x7f908c0c71f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44354/0 (socket says 172.21.15.77:44354) 2022-04-03T19:55:09.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.115+0000 7f908b7fe700 1 -- 172.21.15.77:0/1968951155 >> v2:172.21.15.77:3300/0 conn(0x7f908c155370 msgr2=0x7f908c069110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:09.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.115+0000 7f908b7fe700 1 --2- 172.21.15.77:0/1968951155 >> v2:172.21.15.77:3300/0 conn(0x7f908c155370 0x7f908c069110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:09.119 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.115+0000 7f908b7fe700 1 -- 172.21.15.77:0/1968951155 >> v1:172.21.15.186:6789/0 conn(0x7f908c0c3b10 legacy=0x7f908c0c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:09.120 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.115+0000 7f908b7fe700 1 -- 172.21.15.77:0/1968951155 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908c0c4270 con 0x7f908c0c1170 2022-04-03T19:55:09.120 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.115+0000 7f908b7fe700 1 --2- 172.21.15.77:0/1968951155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f908c0c1170 0x7f908c1527c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f908c063790 tx=0x7f907c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38eea39312b03f30 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:09.120 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.115+0000 7f908a7fc700 1 -- 172.21.15.77:0/1968951155 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c004030 con 0x7f908c0c1170 2022-04-03T19:55:09.120 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.115+0000 7f908a7fc700 1 -- 172.21.15.77:0/1968951155 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907c004190 con 0x7f908c0c1170 2022-04-03T19:55:09.121 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.115+0000 7f908a7fc700 1 -- 172.21.15.77:0/1968951155 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c00f6b0 con 0x7f908c0c1170 2022-04-03T19:55:09.123 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.116+0000 7f9090ebc700 1 -- 172.21.15.77:0/1968951155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f908c0c1170 msgr2=0x7f908c1527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:09.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.116+0000 7f9090ebc700 1 --2- 172.21.15.77:0/1968951155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f908c0c1170 0x7f908c1527c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f908c063790 tx=0x7f907c0090f0 comp rx=0 tx=0).stop 2022-04-03T19:55:09.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.116+0000 7f9090ebc700 1 -- 172.21.15.77:0/1968951155 shutdown_connections 2022-04-03T19:55:09.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.116+0000 7f9090ebc700 1 --2- 172.21.15.77:0/1968951155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f908c0c1170 0x7f908c1527c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:09.124 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.116+0000 7f9090ebc700 1 --2- 172.21.15.77:0/1968951155 >> v2:172.21.15.77:3300/0 conn(0x7f908c155370 0x7f908c069110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:09.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.116+0000 7f9090ebc700 1 -- 172.21.15.77:0/1968951155 >> 172.21.15.77:0/1968951155 conn(0x7f908c066da0 msgr2=0x7f908c0671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:09.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.116+0000 7f9090ebc700 1 -- 172.21.15.77:0/1968951155 shutdown_connections 2022-04-03T19:55:09.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.116+0000 7f9090ebc700 1 -- 172.21.15.77:0/1968951155 wait complete. 2022-04-03T19:55:09.125 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.117+0000 7f9090ebc700 1 Processor -- start 2022-04-03T19:55:09.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.117+0000 7f9090ebc700 1 -- start start 2022-04-03T19:55:09.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.118+0000 7f9090ebc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f908c0c1170 0x7f908c166230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:09.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.118+0000 7f9090ebc700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f908c0c3b10 0x7f908c166770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:09.126 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.118+0000 7f9090ebc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f908c155370 0x7f908c13fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:09.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.118+0000 7f9090ebc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c1486b0 con 0x7f908c0c1170 2022-04-03T19:55:09.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.118+0000 7f9090ebc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c165da0 con 0x7f908c155370 2022-04-03T19:55:09.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.118+0000 7f9090ebc700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f908c1480e0 con 0x7f908c0c3b10 2022-04-03T19:55:09.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.119+0000 7f908b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f908c0c1170 0x7f908c166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:09.127 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.119+0000 7f908b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f908c0c1170 0x7f908c166230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43132/0 (socket says 172.21.15.77:43132) 2022-04-03T19:55:09.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.119+0000 7f908b7fe700 1 -- 172.21.15.77:0/2896627237 learned_addr learned my addr 172.21.15.77:0/2896627237 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:09.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.119+0000 7f908bfff700 1 --2- 172.21.15.77:0/2896627237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f908c155370 0x7f908c13fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:09.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.119+0000 7f908affd700 1 --2- 172.21.15.77:0/2896627237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f908c0c3b10 0x7f908c166770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:09.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.119+0000 7f908b7fe700 1 -- 172.21.15.77:0/2896627237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f908c0c3b10 msgr2=0x7f908c166770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:09.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.119+0000 7f908b7fe700 1 --2- 172.21.15.77:0/2896627237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f908c0c3b10 0x7f908c166770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:09.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.119+0000 7f908b7fe700 1 -- 172.21.15.77:0/2896627237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f908c155370 msgr2=0x7f908c13fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:09.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.119+0000 7f908b7fe700 1 --2- 172.21.15.77:0/2896627237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f908c155370 0x7f908c13fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:09.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.119+0000 7f908b7fe700 1 -- 172.21.15.77:0/2896627237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907c009370 con 0x7f908c0c1170 2022-04-03T19:55:09.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.120+0000 7f908b7fe700 1 --2- 172.21.15.77:0/2896627237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f908c0c1170 0x7f908c166230 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f908c0c1680 tx=0x7f907c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:09.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.120+0000 7f90737fe700 1 -- 172.21.15.77:0/2896627237 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c00f6b0 con 0x7f908c0c1170 2022-04-03T19:55:09.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.120+0000 7f90737fe700 1 -- 172.21.15.77:0/2896627237 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907c0036f0 con 0x7f908c0c1170 2022-04-03T19:55:09.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.120+0000 7f9090ebc700 1 -- 172.21.15.77:0/2896627237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f908c148340 con 0x7f908c0c1170 2022-04-03T19:55:09.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.120+0000 7f90737fe700 1 -- 172.21.15.77:0/2896627237 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c003a70 con 0x7f908c0c1170 2022-04-03T19:55:09.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.120+0000 7f9090ebc700 1 -- 172.21.15.77:0/2896627237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908c148be0 con 0x7f908c0c1170 2022-04-03T19:55:09.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.121+0000 7f9090ebc700 1 -- 172.21.15.77:0/2896627237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f908c06c880 con 0x7f908c0c1170 2022-04-03T19:55:09.131 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.122+0000 7f90737fe700 1 -- 172.21.15.77:0/2896627237 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f907c018420 con 0x7f908c0c1170 2022-04-03T19:55:09.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.123+0000 7f90737fe700 1 --2- 172.21.15.77:0/2896627237 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f906c063ce0 0x7f906c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:09.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.123+0000 7f90737fe700 1 -- 172.21.15.77:0/2896627237 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6546+0+0 (secure 0 0 0) 0x7f907c014070 con 0x7f908c0c1170 2022-04-03T19:55:09.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.123+0000 7f908affd700 1 --2- 172.21.15.77:0/2896627237 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f906c063ce0 0x7f906c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:09.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.124+0000 7f908affd700 1 --2- 172.21.15.77:0/2896627237 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f906c063ce0 0x7f906c066190 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f908c147760 tx=0x7f9080009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:09.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.127+0000 7f90737fe700 1 -- 172.21.15.77:0/2896627237 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f907c050120 con 0x7f908c0c1170 2022-04-03T19:55:09.372 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.371+0000 7f9090ebc700 1 -- 172.21.15.77:0/2896627237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f908c1550e0 con 0x7f908c0c1170 2022-04-03T19:55:09.373 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.371+0000 7f90737fe700 1 -- 172.21.15.77:0/2896627237 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5352 (secure 0 0 0) 0x7f907c053740 con 0x7f908c0c1170 2022-04-03T19:55:09.373 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:55:09.374 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:09.252501+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:55:09.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.374+0000 7f9090ebc700 1 -- 172.21.15.77:0/2896627237 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f906c063ce0 msgr2=0x7f906c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:09.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.374+0000 7f9090ebc700 1 --2- 172.21.15.77:0/2896627237 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f906c063ce0 0x7f906c066190 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f908c147760 tx=0x7f9080009060 comp rx=0 tx=0).stop 2022-04-03T19:55:09.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.374+0000 7f9090ebc700 1 -- 172.21.15.77:0/2896627237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f908c0c1170 msgr2=0x7f908c166230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:09.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.374+0000 7f9090ebc700 1 --2- 172.21.15.77:0/2896627237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f908c0c1170 0x7f908c166230 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f908c0c1680 tx=0x7f907c018000 comp rx=0 tx=0).stop 2022-04-03T19:55:09.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.374+0000 7f9090ebc700 1 -- 172.21.15.77:0/2896627237 shutdown_connections 2022-04-03T19:55:09.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.374+0000 7f9090ebc700 1 --2- 172.21.15.77:0/2896627237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f908c155370 0x7f908c13fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:09.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.374+0000 7f9090ebc700 1 --2- 172.21.15.77:0/2896627237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f908c0c1170 0x7f908c166230 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:09.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.374+0000 7f9090ebc700 1 --2- 172.21.15.77:0/2896627237 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f906c063ce0 0x7f906c066190 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:09.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.374+0000 7f9090ebc700 1 --2- 172.21.15.77:0/2896627237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f908c0c3b10 0x7f908c166770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:09.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.374+0000 7f9090ebc700 1 -- 172.21.15.77:0/2896627237 >> 172.21.15.77:0/2896627237 conn(0x7f908c066da0 msgr2=0x7f908c1557c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:09.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.374+0000 7f9090ebc700 1 -- 172.21.15.77:0/2896627237 shutdown_connections 2022-04-03T19:55:09.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.374+0000 7f9090ebc700 1 -- 172.21.15.77:0/2896627237 wait complete. 2022-04-03T19:55:09.379 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 38 2022-04-03T19:55:09.400 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2022-04-03T19:55:09.401 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-04-03T19:55:09.401 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:55:09.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.598+0000 7f35fb59e700 1 Processor -- start 2022-04-03T19:55:09.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.598+0000 7f35fb59e700 1 -- start start 2022-04-03T19:55:09.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.599+0000 7f35fb59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35fc06bbe0 0x7f35fc1584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:09.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.599+0000 7f35fb59e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f35fc06bfb0 0x7f35fc06d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:09.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.599+0000 7f35fb59e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35fc06d260 con 0x7f35fc06c500 2022-04-03T19:55:09.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.599+0000 7f35fb59e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35fc159450 con 0x7f35fc06bfb0 2022-04-03T19:55:09.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.599+0000 7f35fb59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc0c6330 con 0x7f35fc06bbe0 2022-04-03T19:55:09.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.600+0000 7f35f9d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f35fc06bfb0 0x7f35fc06d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:09.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.600+0000 7f35fa59c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35fc06bbe0 0x7f35fc1584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:09.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.600+0000 7f35f9d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f35fc06bfb0 0x7f35fc06d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43134/0 (socket says 172.21.15.77:43134) 2022-04-03T19:55:09.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.600+0000 7f35f9d9b700 1 -- 172.21.15.77:0/3296460806 learned_addr learned my addr 172.21.15.77:0/3296460806 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:09.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.600+0000 7f35fad9d700 1 --1- 172.21.15.77:0/3296460806 >> v1:172.21.15.77:6789/0 conn(0x7f35fc06c500 0x7f35fc14eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38552/0 (socket says 172.21.15.77:38552) 2022-04-03T19:55:09.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.600+0000 7f35f9d9b700 1 -- 172.21.15.77:0/3296460806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35fc06bbe0 msgr2=0x7f35fc1584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:09.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.600+0000 7f35f9d9b700 1 --2- 172.21.15.77:0/3296460806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35fc06bbe0 0x7f35fc1584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:09.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.600+0000 7f35f9d9b700 1 -- 172.21.15.77:0/3296460806 >> v1:172.21.15.77:6789/0 conn(0x7f35fc06c500 legacy=0x7f35fc14eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:09.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.600+0000 7f35f9d9b700 1 -- 172.21.15.77:0/3296460806 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35fc158bb0 con 0x7f35fc06bfb0 2022-04-03T19:55:09.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.600+0000 7f35f9d9b700 1 --2- 172.21.15.77:0/3296460806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35fc06bfb0 0x7f35fc06d6f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f35f0006fd0 tx=0x7f35f00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f9a5179da4ec666 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:09.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.601+0000 7f35f959a700 1 -- 172.21.15.77:0/3296460806 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f0004030 con 0x7f35fc06bfb0 2022-04-03T19:55:09.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.601+0000 7f35f959a700 1 -- 172.21.15.77:0/3296460806 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f00041d0 con 0x7f35fc06bfb0 2022-04-03T19:55:09.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.601+0000 7f35f959a700 1 -- 172.21.15.77:0/3296460806 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f0011630 con 0x7f35fc06bfb0 2022-04-03T19:55:09.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.601+0000 7f35fb59e700 1 -- 172.21.15.77:0/3296460806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35fc06bfb0 msgr2=0x7f35fc06d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:09.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.601+0000 7f35fb59e700 1 --2- 172.21.15.77:0/3296460806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35fc06bfb0 0x7f35fc06d6f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f35f0006fd0 tx=0x7f35f00090f0 comp rx=0 tx=0).stop 2022-04-03T19:55:09.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.601+0000 7f35fb59e700 1 -- 172.21.15.77:0/3296460806 shutdown_connections 2022-04-03T19:55:09.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.601+0000 7f35fb59e700 1 --2- 172.21.15.77:0/3296460806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35fc06bbe0 0x7f35fc1584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:09.608 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.601+0000 7f35fb59e700 1 --2- 172.21.15.77:0/3296460806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35fc06bfb0 0x7f35fc06d6f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:09.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.601+0000 7f35fb59e700 1 -- 172.21.15.77:0/3296460806 >> 172.21.15.77:0/3296460806 conn(0x7f35fc0cb6b0 msgr2=0x7f35fc0cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:09.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.601+0000 7f35fb59e700 1 -- 172.21.15.77:0/3296460806 shutdown_connections 2022-04-03T19:55:09.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.601+0000 7f35fb59e700 1 -- 172.21.15.77:0/3296460806 wait complete. 2022-04-03T19:55:09.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.603+0000 7f35fb59e700 1 Processor -- start 2022-04-03T19:55:09.609 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.603+0000 7f35fb59e700 1 -- start start 2022-04-03T19:55:09.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.604+0000 7f35fb59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35fc06bbe0 0x7f35fc151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:09.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.604+0000 7f35fb59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35fc06bfb0 0x7f35fc152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:09.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.604+0000 7f35fb59e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35fc06c500 0x7f35fc0bf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:09.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.604+0000 7f35fb59e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc165b60 con 0x7f35fc06bbe0 2022-04-03T19:55:09.611 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.604+0000 7f35fb59e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc0bf550 con 0x7f35fc06c500 2022-04-03T19:55:09.611 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.604+0000 7f35fb59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc164fe0 con 0x7f35fc06bfb0 2022-04-03T19:55:09.611 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.604+0000 7f35fa59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35fc06bbe0 0x7f35fc151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:09.611 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.604+0000 7f35fa59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35fc06bbe0 0x7f35fc151bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43136/0 (socket says 172.21.15.77:43136) 2022-04-03T19:55:09.611 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.604+0000 7f35fa59c700 1 -- 172.21.15.77:0/2430289353 learned_addr learned my addr 172.21.15.77:0/2430289353 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:09.612 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.604+0000 7f35fad9d700 1 --2- 172.21.15.77:0/2430289353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35fc06c500 0x7f35fc0bf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:09.612 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.604+0000 7f35f9d9b700 1 --2- 172.21.15.77:0/2430289353 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35fc06bfb0 0x7f35fc152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:09.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.605+0000 7f35fa59c700 1 -- 172.21.15.77:0/2430289353 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35fc06bfb0 msgr2=0x7f35fc152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:09.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.605+0000 7f35fa59c700 1 --2- 172.21.15.77:0/2430289353 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35fc06bfb0 0x7f35fc152a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:09.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.605+0000 7f35fa59c700 1 -- 172.21.15.77:0/2430289353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35fc06c500 msgr2=0x7f35fc0bf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:09.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.605+0000 7f35fa59c700 1 --2- 172.21.15.77:0/2430289353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35fc06c500 0x7f35fc0bf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:09.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.605+0000 7f35fa59c700 1 -- 172.21.15.77:0/2430289353 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f0009370 con 0x7f35fc06bbe0 2022-04-03T19:55:09.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.605+0000 7f35fa59c700 1 --2- 172.21.15.77:0/2430289353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35fc06bbe0 0x7f35fc151bb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f35ec00e930 tx=0x7f35ec009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:09.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.605+0000 7f35ea7fc700 1 -- 172.21.15.77:0/2430289353 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ec013680 con 0x7f35fc06bbe0 2022-04-03T19:55:09.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.605+0000 7f35ea7fc700 1 -- 172.21.15.77:0/2430289353 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35ec0137e0 con 0x7f35fc06bbe0 2022-04-03T19:55:09.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.605+0000 7f35fb59e700 1 -- 172.21.15.77:0/2430289353 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35fc1652c0 con 0x7f35fc06bbe0 2022-04-03T19:55:09.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.605+0000 7f35ea7fc700 1 -- 172.21.15.77:0/2430289353 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ec013b60 con 0x7f35fc06bbe0 2022-04-03T19:55:09.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.606+0000 7f35fb59e700 1 -- 172.21.15.77:0/2430289353 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35fc165880 con 0x7f35fc06bbe0 2022-04-03T19:55:09.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.607+0000 7f35fb59e700 1 -- 172.21.15.77:0/2430289353 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35fc0bf830 con 0x7f35fc06bbe0 2022-04-03T19:55:09.617 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.607+0000 7f35ea7fc700 1 -- 172.21.15.77:0/2430289353 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f35ec019030 con 0x7f35fc06bbe0 2022-04-03T19:55:09.617 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.608+0000 7f35ea7fc700 1 --2- 172.21.15.77:0/2430289353 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f35d8063ce0 0x7f35d8066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:09.617 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.608+0000 7f35ea7fc700 1 -- 172.21.15.77:0/2430289353 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f35ec0809d0 con 0x7f35fc06bbe0 2022-04-03T19:55:09.617 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.609+0000 7f35f9d9b700 1 --2- 172.21.15.77:0/2430289353 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f35d8063ce0 0x7f35d8066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:09.617 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.609+0000 7f35f9d9b700 1 --2- 172.21.15.77:0/2430289353 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f35d8063ce0 0x7f35d8066190 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f35f0009340 tx=0x7f35f0016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:09.618 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.612+0000 7f35ea7fc700 1 -- 172.21.15.77:0/2430289353 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f35ec04f370 con 0x7f35fc06bbe0 2022-04-03T19:55:09.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.846+0000 7f35fb59e700 1 -- 172.21.15.77:0/2430289353 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35fc04b050 con 0x7f35fc06bbe0 2022-04-03T19:55:09.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.847+0000 7f35ea7fc700 1 -- 172.21.15.77:0/2430289353 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5352 (secure 0 0 0) 0x7f35ec017090 con 0x7f35fc06bbe0 2022-04-03T19:55:09.849 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:55:09.849 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:09.252501+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:55:09.851 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.849+0000 7f35fb59e700 1 -- 172.21.15.77:0/2430289353 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f35d8063ce0 msgr2=0x7f35d8066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:09.851 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.849+0000 7f35fb59e700 1 --2- 172.21.15.77:0/2430289353 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f35d8063ce0 0x7f35d8066190 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f35f0009340 tx=0x7f35f0016000 comp rx=0 tx=0).stop 2022-04-03T19:55:09.851 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.849+0000 7f35fb59e700 1 -- 172.21.15.77:0/2430289353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35fc06bbe0 msgr2=0x7f35fc151bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:09.852 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.849+0000 7f35fb59e700 1 --2- 172.21.15.77:0/2430289353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35fc06bbe0 0x7f35fc151bb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f35ec00e930 tx=0x7f35ec009060 comp rx=0 tx=0).stop 2022-04-03T19:55:09.852 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.849+0000 7f35fb59e700 1 -- 172.21.15.77:0/2430289353 shutdown_connections 2022-04-03T19:55:09.852 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.849+0000 7f35fb59e700 1 --2- 172.21.15.77:0/2430289353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35fc06c500 0x7f35fc0bf9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:09.852 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.849+0000 7f35fb59e700 1 --2- 172.21.15.77:0/2430289353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35fc06bbe0 0x7f35fc151bb0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:09.853 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.849+0000 7f35fb59e700 1 --2- 172.21.15.77:0/2430289353 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f35d8063ce0 0x7f35d8066190 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:09.853 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.849+0000 7f35fb59e700 1 --2- 172.21.15.77:0/2430289353 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35fc06bfb0 0x7f35fc152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:09.853 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.849+0000 7f35fb59e700 1 -- 172.21.15.77:0/2430289353 >> 172.21.15.77:0/2430289353 conn(0x7f35fc0cb6b0 msgr2=0x7f35fc0cbcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:09.853 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.849+0000 7f35fb59e700 1 -- 172.21.15.77:0/2430289353 shutdown_connections 2022-04-03T19:55:09.853 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:09.849+0000 7f35fb59e700 1 -- 172.21.15.77:0/2430289353 wait complete. 2022-04-03T19:55:09.854 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 38 2022-04-03T19:55:09.875 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 38, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-04-03T19:44:21.275716+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 38, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5463": { "addr": "172.21.15.77:6837/2777516376", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6836", "nonce": 2777516376, "type": "v2" }, { "addr": "172.21.15.77:6837", "nonce": 2777516376, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 5463, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-04-03T19:55:09.252501+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5463 } } } ], "standbys": [ { "addr": "172.21.15.77:6834/2116223509", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6832", "nonce": 2116223509, "type": "v2" }, { "addr": "172.21.15.77:6834", "nonce": 2116223509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4245, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6836/2374276747", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2374276747, "type": "v2" }, { "addr": "172.21.15.186:6836", "nonce": 2374276747, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4259, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/610990853", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6838", "nonce": 610990853, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 610990853, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4265, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.77:6835/170053412", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6833", "nonce": 170053412, "type": "v2" }, { "addr": "172.21.15.77:6835", "nonce": 170053412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4271, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-04-03T19:55:11.877 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:55:12.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.080+0000 7f0535b79700 1 Processor -- start 2022-04-03T19:55:12.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.080+0000 7f0535b79700 1 -- start start 2022-04-03T19:55:12.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.082+0000 7f0535b79700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05300c1170 0x7f05301527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:12.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.082+0000 7f0535b79700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0530155370 0x7f0530069110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:12.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.082+0000 7f0535b79700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f05300c3b10 0x7f0530069650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:12.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.082+0000 7f0535b79700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05300c3710 con 0x7f05300c3b10 2022-04-03T19:55:12.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.082+0000 7f0535b79700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0530152330 con 0x7f0530155370 2022-04-03T19:55:12.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.082+0000 7f0535b79700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05300c1540 con 0x7f05300c1170 2022-04-03T19:55:12.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.082+0000 7f0535378700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f05300c3b10 0x7f0530069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:12.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.082+0000 7f052ffff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0530155370 0x7f0530069110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:12.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.082+0000 7f0535378700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f05300c3b10 0x7f0530069650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43140/0 (socket says 172.21.15.77:43140) 2022-04-03T19:55:12.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.082+0000 7f0535378700 1 -- 172.21.15.77:0/1390464087 learned_addr learned my addr 172.21.15.77:0/1390464087 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:12.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.082+0000 7f0534b77700 1 --2- 172.21.15.77:0/1390464087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05300c1170 0x7f05301527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:12.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.083+0000 7f0535378700 1 -- 172.21.15.77:0/1390464087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05300c1170 msgr2=0x7f05301527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:12.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.083+0000 7f0535378700 1 --2- 172.21.15.77:0/1390464087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05300c1170 0x7f05301527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:12.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.083+0000 7f0535378700 1 -- 172.21.15.77:0/1390464087 >> v2:172.21.15.186:3300/0 conn(0x7f0530155370 msgr2=0x7f0530069110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:12.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.083+0000 7f0535378700 1 --2- 172.21.15.77:0/1390464087 >> v2:172.21.15.186:3300/0 conn(0x7f0530155370 0x7f0530069110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:12.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.083+0000 7f0535378700 1 -- 172.21.15.77:0/1390464087 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05300c4270 con 0x7f05300c3b10 2022-04-03T19:55:12.090 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.083+0000 7f0535378700 1 --2- 172.21.15.77:0/1390464087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05300c3b10 0x7f0530069650 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0530065600 tx=0x7f052800ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=418123aa049596ee server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:12.090 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.083+0000 7f052f7fe700 1 -- 172.21.15.77:0/1390464087 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0528010070 con 0x7f05300c3b10 2022-04-03T19:55:12.090 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.083+0000 7f052f7fe700 1 -- 172.21.15.77:0/1390464087 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052800ddc0 con 0x7f05300c3b10 2022-04-03T19:55:12.090 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.083+0000 7f052f7fe700 1 -- 172.21.15.77:0/1390464087 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052800b320 con 0x7f05300c3b10 2022-04-03T19:55:12.090 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.084+0000 7f0535b79700 1 -- 172.21.15.77:0/1390464087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05300c3b10 msgr2=0x7f0530069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:12.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.084+0000 7f0535b79700 1 --2- 172.21.15.77:0/1390464087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05300c3b10 0x7f0530069650 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0530065600 tx=0x7f052800ba50 comp rx=0 tx=0).stop 2022-04-03T19:55:12.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.084+0000 7f0535b79700 1 -- 172.21.15.77:0/1390464087 shutdown_connections 2022-04-03T19:55:12.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.084+0000 7f0535b79700 1 --2- 172.21.15.77:0/1390464087 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05300c3b10 0x7f0530069650 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:12.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.084+0000 7f0535b79700 1 --2- 172.21.15.77:0/1390464087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05300c1170 0x7f05301527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:12.092 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.084+0000 7f0535b79700 1 --2- 172.21.15.77:0/1390464087 >> v2:172.21.15.186:3300/0 conn(0x7f0530155370 0x7f0530069110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:12.092 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.084+0000 7f0535b79700 1 -- 172.21.15.77:0/1390464087 >> 172.21.15.77:0/1390464087 conn(0x7f0530066da0 msgr2=0x7f05300671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:12.092 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.084+0000 7f0535b79700 1 -- 172.21.15.77:0/1390464087 shutdown_connections 2022-04-03T19:55:12.092 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.084+0000 7f0535b79700 1 -- 172.21.15.77:0/1390464087 wait complete. 2022-04-03T19:55:12.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.085+0000 7f0535b79700 1 Processor -- start 2022-04-03T19:55:12.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.086+0000 7f0535b79700 1 -- start start 2022-04-03T19:55:12.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.086+0000 7f0535b79700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05300c1170 0x7f0530147e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:12.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.087+0000 7f0535b79700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05300c3b10 0x7f05300c5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:12.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.087+0000 7f0535b79700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0530155370 0x7f053014d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:12.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.087+0000 7f0535b79700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053014e780 con 0x7f05300c1170 2022-04-03T19:55:12.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.087+0000 7f0535b79700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05300c7ff0 con 0x7f05300c3b10 2022-04-03T19:55:12.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.087+0000 7f0535b79700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053014f070 con 0x7f0530155370 2022-04-03T19:55:12.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.087+0000 7f0534b77700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05300c1170 0x7f0530147e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:12.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.087+0000 7f0534b77700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05300c1170 0x7f0530147e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43142/0 (socket says 172.21.15.77:43142) 2022-04-03T19:55:12.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.087+0000 7f0534b77700 1 -- 172.21.15.77:0/2649644220 learned_addr learned my addr 172.21.15.77:0/2649644220 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:12.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.087+0000 7f0535378700 1 --2- 172.21.15.77:0/2649644220 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0530155370 0x7f053014d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:12.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.087+0000 7f052ffff700 1 --2- 172.21.15.77:0/2649644220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05300c3b10 0x7f05300c5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:12.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.087+0000 7f0534b77700 1 -- 172.21.15.77:0/2649644220 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0530155370 msgr2=0x7f053014d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:12.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.087+0000 7f0534b77700 1 --2- 172.21.15.77:0/2649644220 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0530155370 0x7f053014d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:12.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.087+0000 7f0534b77700 1 -- 172.21.15.77:0/2649644220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05300c3b10 msgr2=0x7f05300c5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:12.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.087+0000 7f0534b77700 1 --2- 172.21.15.77:0/2649644220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05300c3b10 0x7f05300c5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:12.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.087+0000 7f0534b77700 1 -- 172.21.15.77:0/2649644220 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052800bcf0 con 0x7f05300c1170 2022-04-03T19:55:12.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.088+0000 7f0534b77700 1 --2- 172.21.15.77:0/2649644220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05300c1170 0x7f0530147e80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f052400e930 tx=0x7f0524009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:12.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.088+0000 7f052cff9700 1 -- 172.21.15.77:0/2649644220 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0524013680 con 0x7f05300c1170 2022-04-03T19:55:12.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.088+0000 7f052cff9700 1 -- 172.21.15.77:0/2649644220 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05240137e0 con 0x7f05300c1170 2022-04-03T19:55:12.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.088+0000 7f0535b79700 1 -- 172.21.15.77:0/2649644220 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053014f350 con 0x7f05300c1170 2022-04-03T19:55:12.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.088+0000 7f052cff9700 1 -- 172.21.15.77:0/2649644220 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0524013b60 con 0x7f05300c1170 2022-04-03T19:55:12.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.088+0000 7f0535b79700 1 -- 172.21.15.77:0/2649644220 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f053014e130 con 0x7f05300c1170 2022-04-03T19:55:12.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.089+0000 7f0535b79700 1 -- 172.21.15.77:0/2649644220 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053014df10 con 0x7f05300c1170 2022-04-03T19:55:12.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.090+0000 7f052cff9700 1 -- 172.21.15.77:0/2649644220 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f0524023070 con 0x7f05300c1170 2022-04-03T19:55:12.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.091+0000 7f052cff9700 1 --2- 172.21.15.77:0/2649644220 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0518063d10 0x7f05180661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:12.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.091+0000 7f052cff9700 1 -- 172.21.15.77:0/2649644220 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f0524080df0 con 0x7f05300c1170 2022-04-03T19:55:12.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.091+0000 7f052ffff700 1 --2- 172.21.15.77:0/2649644220 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0518063d10 0x7f05180661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:12.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.092+0000 7f052ffff700 1 --2- 172.21.15.77:0/2649644220 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0518063d10 0x7f05180661c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0530156190 tx=0x7f051c009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:12.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.095+0000 7f052cff9700 1 -- 172.21.15.77:0/2649644220 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f052404f6a0 con 0x7f05300c1170 2022-04-03T19:55:12.335 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.334+0000 7f0535b79700 1 -- 172.21.15.77:0/2649644220 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05300bfef0 con 0x7f05300c1170 2022-04-03T19:55:12.336 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.335+0000 7f052cff9700 1 -- 172.21.15.77:0/2649644220 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5352 (secure 0 0 0) 0x7f0524052cc0 con 0x7f05300c1170 2022-04-03T19:55:12.337 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:55:12.337 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:09.252501+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:55:12.339 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.338+0000 7f0535b79700 1 -- 172.21.15.77:0/2649644220 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0518063d10 msgr2=0x7f05180661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:12.339 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.338+0000 7f0535b79700 1 --2- 172.21.15.77:0/2649644220 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0518063d10 0x7f05180661c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0530156190 tx=0x7f051c009060 comp rx=0 tx=0).stop 2022-04-03T19:55:12.339 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.338+0000 7f0535b79700 1 -- 172.21.15.77:0/2649644220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05300c1170 msgr2=0x7f0530147e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:12.340 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.338+0000 7f0535b79700 1 --2- 172.21.15.77:0/2649644220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05300c1170 0x7f0530147e80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f052400e930 tx=0x7f0524009060 comp rx=0 tx=0).stop 2022-04-03T19:55:12.340 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.338+0000 7f0535b79700 1 -- 172.21.15.77:0/2649644220 shutdown_connections 2022-04-03T19:55:12.340 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.338+0000 7f0535b79700 1 --2- 172.21.15.77:0/2649644220 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0518063d10 0x7f05180661c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:12.340 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.338+0000 7f0535b79700 1 --2- 172.21.15.77:0/2649644220 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0530155370 0x7f053014d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:12.340 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.338+0000 7f0535b79700 1 --2- 172.21.15.77:0/2649644220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f05300c1170 0x7f0530147e80 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:12.341 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.338+0000 7f0535b79700 1 --2- 172.21.15.77:0/2649644220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05300c3b10 0x7f05300c5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:12.341 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.338+0000 7f0535b79700 1 -- 172.21.15.77:0/2649644220 >> 172.21.15.77:0/2649644220 conn(0x7f0530066da0 msgr2=0x7f0530155880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:12.341 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.338+0000 7f0535b79700 1 -- 172.21.15.77:0/2649644220 shutdown_connections 2022-04-03T19:55:12.341 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:12.338+0000 7f0535b79700 1 -- 172.21.15.77:0/2649644220 wait complete. 2022-04-03T19:55:12.342 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 38 2022-04-03T19:55:12.362 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:55:14.363 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:55:14.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.568+0000 7f7d79ffa700 1 Processor -- start 2022-04-03T19:55:14.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.568+0000 7f7d79ffa700 1 -- start start 2022-04-03T19:55:14.579 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.569+0000 7f7d79ffa700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7d740c2010 0x7f7d74159050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:14.580 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.569+0000 7f7d79ffa700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d7406a280 0x7f7d74155910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:14.580 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.569+0000 7f7d79ffa700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7d740c4dc0 0x7f7d740c5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:14.581 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.569+0000 7f7d79ffa700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d740c4200 con 0x7f7d740c2010 2022-04-03T19:55:14.581 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.569+0000 7f7d79ffa700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d74159590 con 0x7f7d740c4dc0 2022-04-03T19:55:14.582 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.569+0000 7f7d79ffa700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d74155480 con 0x7f7d7406a280 2022-04-03T19:55:14.582 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.570+0000 7f7d78ff8700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7d740c2010 0x7f7d74159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:14.582 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.570+0000 7f7d797f9700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7d740c4dc0 0x7f7d740c5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:14.583 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.570+0000 7f7d797f9700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7d740c4dc0 0x7f7d740c5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38338/0 (socket says 172.21.15.77:38338) 2022-04-03T19:55:14.583 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.570+0000 7f7d797f9700 1 -- 172.21.15.77:0/1754958037 learned_addr learned my addr 172.21.15.77:0/1754958037 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:14.584 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.570+0000 7f7d78ff8700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7d740c2010 0x7f7d74159050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43144/0 (socket says 172.21.15.77:43144) 2022-04-03T19:55:14.584 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.570+0000 7f7d73fff700 1 --2- 172.21.15.77:0/1754958037 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d7406a280 0x7f7d74155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:14.584 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.570+0000 7f7d797f9700 1 -- 172.21.15.77:0/1754958037 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d7406a280 msgr2=0x7f7d74155910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:14.585 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.570+0000 7f7d797f9700 1 --2- 172.21.15.77:0/1754958037 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d7406a280 0x7f7d74155910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:14.585 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.570+0000 7f7d797f9700 1 -- 172.21.15.77:0/1754958037 >> v2:172.21.15.77:3300/0 conn(0x7f7d740c2010 msgr2=0x7f7d74159050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:14.586 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.570+0000 7f7d797f9700 1 --2- 172.21.15.77:0/1754958037 >> v2:172.21.15.77:3300/0 conn(0x7f7d740c2010 0x7f7d74159050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:14.586 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.570+0000 7f7d797f9700 1 -- 172.21.15.77:0/1754958037 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d740c2d70 con 0x7f7d740c4dc0 2022-04-03T19:55:14.587 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.570+0000 7f7d73fff700 1 --2- 172.21.15.77:0/1754958037 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d7406a280 0x7f7d74155910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:55:14.587 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.570+0000 7f7d78ff8700 1 --2- 172.21.15.77:0/1754958037 >> v2:172.21.15.77:3300/0 conn(0x7f7d740c2010 0x7f7d74159050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:55:14.587 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.571+0000 7f7d797f9700 1 --2- 172.21.15.77:0/1754958037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d740c4dc0 0x7f7d740c5260 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7d7406b1b0 tx=0x7f7d6c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33cd3634fc0f82c1 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:14.588 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.571+0000 7f7d737fe700 1 -- 172.21.15.77:0/1754958037 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c00e070 con 0x7f7d740c4dc0 2022-04-03T19:55:14.588 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.571+0000 7f7d737fe700 1 -- 172.21.15.77:0/1754958037 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d6c00ba20 con 0x7f7d740c4dc0 2022-04-03T19:55:14.589 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.571+0000 7f7d737fe700 1 -- 172.21.15.77:0/1754958037 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c0092e0 con 0x7f7d740c4dc0 2022-04-03T19:55:14.589 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.571+0000 7f7d79ffa700 1 -- 172.21.15.77:0/1754958037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d740c4dc0 msgr2=0x7f7d740c5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:14.589 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.571+0000 7f7d79ffa700 1 --2- 172.21.15.77:0/1754958037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d740c4dc0 0x7f7d740c5260 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7d7406b1b0 tx=0x7f7d6c009840 comp rx=0 tx=0).stop 2022-04-03T19:55:14.590 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.571+0000 7f7d79ffa700 1 -- 172.21.15.77:0/1754958037 shutdown_connections 2022-04-03T19:55:14.590 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.571+0000 7f7d79ffa700 1 --2- 172.21.15.77:0/1754958037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d740c4dc0 0x7f7d740c5260 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:14.591 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.571+0000 7f7d79ffa700 1 --2- 172.21.15.77:0/1754958037 >> v2:172.21.15.77:3300/0 conn(0x7f7d740c2010 0x7f7d74159050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:14.591 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.571+0000 7f7d79ffa700 1 --2- 172.21.15.77:0/1754958037 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d7406a280 0x7f7d74155910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:14.592 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.571+0000 7f7d79ffa700 1 -- 172.21.15.77:0/1754958037 >> 172.21.15.77:0/1754958037 conn(0x7f7d7406cb30 msgr2=0x7f7d7406cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:14.592 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.572+0000 7f7d79ffa700 1 -- 172.21.15.77:0/1754958037 shutdown_connections 2022-04-03T19:55:14.592 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.572+0000 7f7d79ffa700 1 -- 172.21.15.77:0/1754958037 wait complete. 2022-04-03T19:55:14.593 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.573+0000 7f7d79ffa700 1 Processor -- start 2022-04-03T19:55:14.593 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.573+0000 7f7d79ffa700 1 -- start start 2022-04-03T19:55:14.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.574+0000 7f7d79ffa700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d7406a280 0x7f7d7414d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:14.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.574+0000 7f7d79ffa700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d740c2010 0x7f7d7415a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:14.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.574+0000 7f7d79ffa700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d740c4dc0 0x7f7d74150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:14.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.574+0000 7f7d79ffa700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7414dc80 con 0x7f7d740c2010 2022-04-03T19:55:14.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.574+0000 7f7d79ffa700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7414d2b0 con 0x7f7d7406a280 2022-04-03T19:55:14.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.574+0000 7f7d79ffa700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7414ccc0 con 0x7f7d740c4dc0 2022-04-03T19:55:14.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.574+0000 7f7d73fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d740c2010 0x7f7d7415a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:14.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.574+0000 7f7d78ff8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d7406a280 0x7f7d7414d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:14.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.574+0000 7f7d78ff8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d7406a280 0x7f7d7414d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38340/0 (socket says 172.21.15.77:38340) 2022-04-03T19:55:14.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.574+0000 7f7d78ff8700 1 -- 172.21.15.77:0/3906489828 learned_addr learned my addr 172.21.15.77:0/3906489828 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:14.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.574+0000 7f7d797f9700 1 --2- 172.21.15.77:0/3906489828 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d740c4dc0 0x7f7d74150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:14.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.575+0000 7f7d78ff8700 1 -- 172.21.15.77:0/3906489828 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d740c4dc0 msgr2=0x7f7d74150330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.575+0000 7f7d78ff8700 1 --2- 172.21.15.77:0/3906489828 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d740c4dc0 0x7f7d74150330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.575+0000 7f7d78ff8700 1 -- 172.21.15.77:0/3906489828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d740c2010 msgr2=0x7f7d7415a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.575+0000 7f7d78ff8700 1 --2- 172.21.15.77:0/3906489828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d740c2010 0x7f7d7415a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:14.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.575+0000 7f7d78ff8700 1 -- 172.21.15.77:0/3906489828 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d680092a0 con 0x7f7d7406a280 2022-04-03T19:55:14.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.575+0000 7f7d78ff8700 1 --2- 172.21.15.77:0/3906489828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d7406a280 0x7f7d7414d740 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800ea20 tx=0x7f7d68009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:14.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.575+0000 7f7d70ff9700 1 -- 172.21.15.77:0/3906489828 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6800ee60 con 0x7f7d7406a280 2022-04-03T19:55:14.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.576+0000 7f7d70ff9700 1 -- 172.21.15.77:0/3906489828 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d6800c850 con 0x7f7d7406a280 2022-04-03T19:55:14.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.576+0000 7f7d79ffa700 1 -- 172.21.15.77:0/3906489828 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d6c009ae0 con 0x7f7d7406a280 2022-04-03T19:55:14.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.576+0000 7f7d70ff9700 1 -- 172.21.15.77:0/3906489828 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6800cbd0 con 0x7f7d7406a280 2022-04-03T19:55:14.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.576+0000 7f7d79ffa700 1 -- 172.21.15.77:0/3906489828 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d74152790 con 0x7f7d7406a280 2022-04-03T19:55:14.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.577+0000 7f7d79ffa700 1 -- 172.21.15.77:0/3906489828 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d7406b3f0 con 0x7f7d7406a280 2022-04-03T19:55:14.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.578+0000 7f7d70ff9700 1 -- 172.21.15.77:0/3906489828 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f7d68019030 con 0x7f7d7406a280 2022-04-03T19:55:14.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.578+0000 7f7d70ff9700 1 --2- 172.21.15.77:0/3906489828 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7d5c063ce0 0x7f7d5c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:14.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.579+0000 7f7d70ff9700 1 -- 172.21.15.77:0/3906489828 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f7d68080aa0 con 0x7f7d7406a280 2022-04-03T19:55:14.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.579+0000 7f7d73fff700 1 --2- 172.21.15.77:0/3906489828 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7d5c063ce0 0x7f7d5c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:14.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.580+0000 7f7d73fff700 1 --2- 172.21.15.77:0/3906489828 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7d5c063ce0 0x7f7d5c066190 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7d7415a280 tx=0x7f7d60008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:14.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.583+0000 7f7d70ff9700 1 -- 172.21.15.77:0/3906489828 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f7d6804f440 con 0x7f7d7406a280 2022-04-03T19:55:14.832 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.831+0000 7f7d79ffa700 1 -- 172.21.15.77:0/3906489828 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d7404b050 con 0x7f7d7406a280 2022-04-03T19:55:14.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.832+0000 7f7d70ff9700 1 -- 172.21.15.77:0/3906489828 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5352 (secure 0 0 0) 0x7f7d68017090 con 0x7f7d7406a280 2022-04-03T19:55:14.833 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:55:14.833 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:09.252501+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:55:14.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.835+0000 7f7d79ffa700 1 -- 172.21.15.77:0/3906489828 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7d5c063ce0 msgr2=0x7f7d5c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:14.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.835+0000 7f7d79ffa700 1 --2- 172.21.15.77:0/3906489828 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7d5c063ce0 0x7f7d5c066190 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7d7415a280 tx=0x7f7d60008040 comp rx=0 tx=0).stop 2022-04-03T19:55:14.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.835+0000 7f7d79ffa700 1 -- 172.21.15.77:0/3906489828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d7406a280 msgr2=0x7f7d7414d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:14.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.835+0000 7f7d79ffa700 1 --2- 172.21.15.77:0/3906489828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d7406a280 0x7f7d7414d740 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800ea20 tx=0x7f7d68009060 comp rx=0 tx=0).stop 2022-04-03T19:55:14.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.835+0000 7f7d79ffa700 1 -- 172.21.15.77:0/3906489828 shutdown_connections 2022-04-03T19:55:14.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.835+0000 7f7d79ffa700 1 --2- 172.21.15.77:0/3906489828 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7d5c063ce0 0x7f7d5c066190 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:14.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.835+0000 7f7d79ffa700 1 --2- 172.21.15.77:0/3906489828 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d740c4dc0 0x7f7d74150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:14.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.835+0000 7f7d79ffa700 1 --2- 172.21.15.77:0/3906489828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d7406a280 0x7f7d7414d740 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:14.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.835+0000 7f7d79ffa700 1 --2- 172.21.15.77:0/3906489828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7d740c2010 0x7f7d7415a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:14.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.835+0000 7f7d79ffa700 1 -- 172.21.15.77:0/3906489828 >> 172.21.15.77:0/3906489828 conn(0x7f7d7406cb30 msgr2=0x7f7d74159700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:14.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.835+0000 7f7d79ffa700 1 -- 172.21.15.77:0/3906489828 shutdown_connections 2022-04-03T19:55:14.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:14.835+0000 7f7d79ffa700 1 -- 172.21.15.77:0/3906489828 wait complete. 2022-04-03T19:55:14.838 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 38 2022-04-03T19:55:14.860 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:55:16.862 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:55:17.074 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.073+0000 7fdf79b6f700 1 Processor -- start 2022-04-03T19:55:17.074 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.073+0000 7fdf79b6f700 1 -- start start 2022-04-03T19:55:17.084 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.075+0000 7fdf79b6f700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdf7406d3f0 0x7fdf7406d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:17.084 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.075+0000 7fdf79b6f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf740c5930 0x7fdf74155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:17.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.075+0000 7fdf79b6f700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf740c2f80 con 0x7fdf74156120 2022-04-03T19:55:17.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.075+0000 7fdf79b6f700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf740c3260 con 0x7fdf7406d3f0 2022-04-03T19:55:17.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.075+0000 7fdf79b6f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf740c3540 con 0x7fdf740c5930 2022-04-03T19:55:17.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.075+0000 7fdf73fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf740c5930 0x7fdf74155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:17.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.075+0000 7fdf78b6d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdf7406d3f0 0x7fdf7406d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:17.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.075+0000 7fdf78b6d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdf7406d3f0 0x7fdf7406d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38342/0 (socket says 172.21.15.77:38342) 2022-04-03T19:55:17.087 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.075+0000 7fdf78b6d700 1 -- 172.21.15.77:0/3836914807 learned_addr learned my addr 172.21.15.77:0/3836914807 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:17.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.075+0000 7fdf7936e700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fdf74156120 0x7fdf7406c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38554/0 (socket says 172.21.15.77:38554) 2022-04-03T19:55:17.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.075+0000 7fdf73fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf740c5930 0x7fdf74155be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60434/0 (socket says 172.21.15.77:60434) 2022-04-03T19:55:17.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.076+0000 7fdf78b6d700 1 -- 172.21.15.77:0/3836914807 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf740c5930 msgr2=0x7fdf74155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:17.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.076+0000 7fdf78b6d700 1 --2- 172.21.15.77:0/3836914807 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf740c5930 0x7fdf74155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:17.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.076+0000 7fdf78b6d700 1 -- 172.21.15.77:0/3836914807 >> v1:172.21.15.77:6789/0 conn(0x7fdf74156120 legacy=0x7fdf7406c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:17.090 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.076+0000 7fdf78b6d700 1 -- 172.21.15.77:0/3836914807 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf740c3820 con 0x7fdf7406d3f0 2022-04-03T19:55:17.090 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.076+0000 7fdf78b6d700 1 --2- 172.21.15.77:0/3836914807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf7406d3f0 0x7fdf7406d7c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdf68009830 tx=0x7fdf680090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a066ea5c87afc5f6 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:17.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.076+0000 7fdf737fe700 1 -- 172.21.15.77:0/3836914807 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf68004030 con 0x7fdf7406d3f0 2022-04-03T19:55:17.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.076+0000 7fdf737fe700 1 -- 172.21.15.77:0/3836914807 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf68004190 con 0x7fdf7406d3f0 2022-04-03T19:55:17.092 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.076+0000 7fdf737fe700 1 -- 172.21.15.77:0/3836914807 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf68011630 con 0x7fdf7406d3f0 2022-04-03T19:55:17.092 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.077+0000 7fdf79b6f700 1 -- 172.21.15.77:0/3836914807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf7406d3f0 msgr2=0x7fdf7406d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:17.092 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.077+0000 7fdf79b6f700 1 --2- 172.21.15.77:0/3836914807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf7406d3f0 0x7fdf7406d7c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdf68009830 tx=0x7fdf680090f0 comp rx=0 tx=0).stop 2022-04-03T19:55:17.093 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.077+0000 7fdf79b6f700 1 -- 172.21.15.77:0/3836914807 shutdown_connections 2022-04-03T19:55:17.093 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.077+0000 7fdf79b6f700 1 --2- 172.21.15.77:0/3836914807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf7406d3f0 0x7fdf7406d7c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:17.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.077+0000 7fdf79b6f700 1 --2- 172.21.15.77:0/3836914807 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf740c5930 0x7fdf74155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:17.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.077+0000 7fdf79b6f700 1 -- 172.21.15.77:0/3836914807 >> 172.21.15.77:0/3836914807 conn(0x7fdf740cb720 msgr2=0x7fdf740cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:17.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.077+0000 7fdf79b6f700 1 -- 172.21.15.77:0/3836914807 shutdown_connections 2022-04-03T19:55:17.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.077+0000 7fdf79b6f700 1 -- 172.21.15.77:0/3836914807 wait complete. 2022-04-03T19:55:17.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.079+0000 7fdf79b6f700 1 Processor -- start 2022-04-03T19:55:17.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.079+0000 7fdf79b6f700 1 -- start start 2022-04-03T19:55:17.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.080+0000 7fdf79b6f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf7406d3f0 0x7fdf74157a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:17.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.080+0000 7fdf79b6f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf740c5930 0x7fdf74157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:17.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.080+0000 7fdf79b6f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf74156120 0x7fdf7414a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:17.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.080+0000 7fdf79b6f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf74150d90 con 0x7fdf740c5930 2022-04-03T19:55:17.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.080+0000 7fdf79b6f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf74166d10 con 0x7fdf74156120 2022-04-03T19:55:17.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.080+0000 7fdf79b6f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf74156670 con 0x7fdf7406d3f0 2022-04-03T19:55:17.098 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.080+0000 7fdf78b6d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf7406d3f0 0x7fdf74157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:17.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.080+0000 7fdf78b6d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf7406d3f0 0x7fdf74157a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60436/0 (socket says 172.21.15.77:60436) 2022-04-03T19:55:17.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.080+0000 7fdf78b6d700 1 -- 172.21.15.77:0/4199849767 learned_addr learned my addr 172.21.15.77:0/4199849767 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:17.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.080+0000 7fdf7936e700 1 --2- 172.21.15.77:0/4199849767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf74156120 0x7fdf7414a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:17.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.080+0000 7fdf73fff700 1 --2- 172.21.15.77:0/4199849767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf740c5930 0x7fdf74157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:17.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.080+0000 7fdf78b6d700 1 -- 172.21.15.77:0/4199849767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf74156120 msgr2=0x7fdf7414a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:17.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.080+0000 7fdf78b6d700 1 --2- 172.21.15.77:0/4199849767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf74156120 0x7fdf7414a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:17.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.080+0000 7fdf78b6d700 1 -- 172.21.15.77:0/4199849767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf740c5930 msgr2=0x7fdf74157fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:17.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.080+0000 7fdf78b6d700 1 --2- 172.21.15.77:0/4199849767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf740c5930 0x7fdf74157fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:17.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.080+0000 7fdf78b6d700 1 -- 172.21.15.77:0/4199849767 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf68009370 con 0x7fdf7406d3f0 2022-04-03T19:55:17.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.081+0000 7fdf78b6d700 1 --2- 172.21.15.77:0/4199849767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf7406d3f0 0x7fdf74157a60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdf740890b0 tx=0x7fdf68016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:17.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.081+0000 7fdf70ff9700 1 -- 172.21.15.77:0/4199849767 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf680036f0 con 0x7fdf7406d3f0 2022-04-03T19:55:17.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.081+0000 7fdf70ff9700 1 -- 172.21.15.77:0/4199849767 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf68003850 con 0x7fdf7406d3f0 2022-04-03T19:55:17.104 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.081+0000 7fdf79b6f700 1 -- 172.21.15.77:0/4199849767 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf74166a30 con 0x7fdf7406d3f0 2022-04-03T19:55:17.104 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.081+0000 7fdf70ff9700 1 -- 172.21.15.77:0/4199849767 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf68003bf0 con 0x7fdf7406d3f0 2022-04-03T19:55:17.105 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.081+0000 7fdf79b6f700 1 -- 172.21.15.77:0/4199849767 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf74156bf0 con 0x7fdf7406d3f0 2022-04-03T19:55:17.105 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.083+0000 7fdf79b6f700 1 -- 172.21.15.77:0/4199849767 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf74159800 con 0x7fdf7406d3f0 2022-04-03T19:55:17.107 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.085+0000 7fdf70ff9700 1 -- 172.21.15.77:0/4199849767 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fdf68017020 con 0x7fdf7406d3f0 2022-04-03T19:55:17.107 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.086+0000 7fdf70ff9700 1 --2- 172.21.15.77:0/4199849767 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdf5c063d10 0x7fdf5c0661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:17.108 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.086+0000 7fdf70ff9700 1 -- 172.21.15.77:0/4199849767 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7fdf68082720 con 0x7fdf7406d3f0 2022-04-03T19:55:17.108 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.086+0000 7fdf73fff700 1 --2- 172.21.15.77:0/4199849767 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdf5c063d10 0x7fdf5c0661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:17.109 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.087+0000 7fdf73fff700 1 --2- 172.21.15.77:0/4199849767 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdf5c063d10 0x7fdf5c0661c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdf7406d900 tx=0x7fdf60009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:17.109 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.089+0000 7fdf70ff9700 1 -- 172.21.15.77:0/4199849767 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fdf680510c0 con 0x7fdf7406d3f0 2022-04-03T19:55:17.194 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:55:17.201 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:55:17.346 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.346+0000 7fdf79b6f700 1 -- 172.21.15.77:0/4199849767 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf741586f0 con 0x7fdf7406d3f0 2022-04-03T19:55:17.347 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.347+0000 7fdf70ff9700 1 -- 172.21.15.77:0/4199849767 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5352 (secure 0 0 0) 0x7fdf680546e0 con 0x7fdf7406d3f0 2022-04-03T19:55:17.348 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:55:17.348 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:09.252501+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:55:17.352 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.351+0000 7fdf79b6f700 1 -- 172.21.15.77:0/4199849767 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdf5c063d10 msgr2=0x7fdf5c0661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:17.352 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.351+0000 7fdf79b6f700 1 --2- 172.21.15.77:0/4199849767 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdf5c063d10 0x7fdf5c0661c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdf7406d900 tx=0x7fdf60009060 comp rx=0 tx=0).stop 2022-04-03T19:55:17.352 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.351+0000 7fdf79b6f700 1 -- 172.21.15.77:0/4199849767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf7406d3f0 msgr2=0x7fdf74157a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:17.352 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.351+0000 7fdf79b6f700 1 --2- 172.21.15.77:0/4199849767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf7406d3f0 0x7fdf74157a60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdf740890b0 tx=0x7fdf68016000 comp rx=0 tx=0).stop 2022-04-03T19:55:17.353 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.351+0000 7fdf79b6f700 1 -- 172.21.15.77:0/4199849767 shutdown_connections 2022-04-03T19:55:17.353 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.351+0000 7fdf79b6f700 1 --2- 172.21.15.77:0/4199849767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf74156120 0x7fdf7414a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:17.353 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.351+0000 7fdf79b6f700 1 --2- 172.21.15.77:0/4199849767 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fdf5c063d10 0x7fdf5c0661c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:17.353 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.351+0000 7fdf79b6f700 1 --2- 172.21.15.77:0/4199849767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf7406d3f0 0x7fdf74157a60 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:17.354 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.351+0000 7fdf79b6f700 1 --2- 172.21.15.77:0/4199849767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf740c5930 0x7fdf74157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:17.354 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.351+0000 7fdf79b6f700 1 -- 172.21.15.77:0/4199849767 >> 172.21.15.77:0/4199849767 conn(0x7fdf740cb720 msgr2=0x7fdf74154c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:17.354 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.351+0000 7fdf79b6f700 1 -- 172.21.15.77:0/4199849767 shutdown_connections 2022-04-03T19:55:17.354 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:17.351+0000 7fdf79b6f700 1 -- 172.21.15.77:0/4199849767 wait complete. 2022-04-03T19:55:17.355 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 38 2022-04-03T19:55:17.390 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:55:19.391 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:55:19.829 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.829+0000 7fb92ce5f700 1 Processor -- start 2022-04-03T19:55:19.830 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.829+0000 7fb92ce5f700 1 -- start start 2022-04-03T19:55:19.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.830+0000 7fb92ce5f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb92806d3f0 0x7fb92806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:19.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.830+0000 7fb92ce5f700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb9280c5930 0x7fb928155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:19.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.830+0000 7fb92ce5f700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb928156120 0x7fb92806c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:19.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.830+0000 7fb92ce5f700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9280c2f80 con 0x7fb928156120 2022-04-03T19:55:19.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.830+0000 7fb92ce5f700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9280c3260 con 0x7fb9280c5930 2022-04-03T19:55:19.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.830+0000 7fb92ce5f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280c3540 con 0x7fb92806d3f0 2022-04-03T19:55:19.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.831+0000 7fb927fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb928156120 0x7fb92806c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:19.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.831+0000 7fb9277fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb92806d3f0 0x7fb92806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:19.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.831+0000 7fb926ffd700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb9280c5930 0x7fb928155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:19.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.831+0000 7fb927fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb928156120 0x7fb92806c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43150/0 (socket says 172.21.15.77:43150) 2022-04-03T19:55:19.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.831+0000 7fb927fff700 1 -- 172.21.15.77:0/148436892 learned_addr learned my addr 172.21.15.77:0/148436892 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:19.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.831+0000 7fb927fff700 1 -- 172.21.15.77:0/148436892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb92806d3f0 msgr2=0x7fb92806d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:19.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.831+0000 7fb927fff700 1 --2- 172.21.15.77:0/148436892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb92806d3f0 0x7fb92806d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:19.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.831+0000 7fb927fff700 1 -- 172.21.15.77:0/148436892 >> v2:172.21.15.186:3300/0 conn(0x7fb9280c5930 msgr2=0x7fb928155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:19.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.831+0000 7fb927fff700 1 --2- 172.21.15.77:0/148436892 >> v2:172.21.15.186:3300/0 conn(0x7fb9280c5930 0x7fb928155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:19.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.831+0000 7fb927fff700 1 -- 172.21.15.77:0/148436892 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9280c3820 con 0x7fb928156120 2022-04-03T19:55:19.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.831+0000 7fb927fff700 1 --2- 172.21.15.77:0/148436892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb928156120 0x7fb92806c040 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb920009f20 tx=0x7fb920009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a1e425968b44cff server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:19.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.832+0000 7fb9267fc700 1 -- 172.21.15.77:0/148436892 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92000e070 con 0x7fb928156120 2022-04-03T19:55:19.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.832+0000 7fb9267fc700 1 -- 172.21.15.77:0/148436892 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92000b9c0 con 0x7fb928156120 2022-04-03T19:55:19.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.832+0000 7fb9267fc700 1 -- 172.21.15.77:0/148436892 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9200092e0 con 0x7fb928156120 2022-04-03T19:55:19.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.832+0000 7fb92ce5f700 1 -- 172.21.15.77:0/148436892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb928156120 msgr2=0x7fb92806c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:19.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.832+0000 7fb92ce5f700 1 --2- 172.21.15.77:0/148436892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb928156120 0x7fb92806c040 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb920009f20 tx=0x7fb920009840 comp rx=0 tx=0).stop 2022-04-03T19:55:19.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.832+0000 7fb92ce5f700 1 -- 172.21.15.77:0/148436892 shutdown_connections 2022-04-03T19:55:19.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.832+0000 7fb92ce5f700 1 --2- 172.21.15.77:0/148436892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb928156120 0x7fb92806c040 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:19.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.832+0000 7fb92ce5f700 1 --2- 172.21.15.77:0/148436892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb92806d3f0 0x7fb92806d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:19.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.832+0000 7fb92ce5f700 1 --2- 172.21.15.77:0/148436892 >> v2:172.21.15.186:3300/0 conn(0x7fb9280c5930 0x7fb928155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:19.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.832+0000 7fb92ce5f700 1 -- 172.21.15.77:0/148436892 >> 172.21.15.77:0/148436892 conn(0x7fb9280cb720 msgr2=0x7fb9280cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:19.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.832+0000 7fb92ce5f700 1 -- 172.21.15.77:0/148436892 shutdown_connections 2022-04-03T19:55:19.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.832+0000 7fb92ce5f700 1 -- 172.21.15.77:0/148436892 wait complete. 2022-04-03T19:55:19.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.834+0000 7fb92ce5f700 1 Processor -- start 2022-04-03T19:55:19.843 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.834+0000 7fb92ce5f700 1 -- start start 2022-04-03T19:55:19.843 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.835+0000 7fb92ce5f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb92806d3f0 0x7fb9280bf680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:19.843 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.835+0000 7fb92ce5f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9280c5930 0x7fb9280c0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:19.843 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.835+0000 7fb92ce5f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb928156120 0x7fb9280c0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:19.844 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.835+0000 7fb92ce5f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280c14a0 con 0x7fb928156120 2022-04-03T19:55:19.844 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.835+0000 7fb92ce5f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280c0080 con 0x7fb92806d3f0 2022-04-03T19:55:19.844 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.835+0000 7fb92ce5f700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb928152c40 con 0x7fb9280c5930 2022-04-03T19:55:19.844 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.835+0000 7fb927fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb928156120 0x7fb9280c0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:19.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.835+0000 7fb926ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9280c5930 0x7fb9280c0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:19.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.835+0000 7fb9277fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb92806d3f0 0x7fb9280bf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:19.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.835+0000 7fb926ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9280c5930 0x7fb9280c0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60440/0 (socket says 172.21.15.77:60440) 2022-04-03T19:55:19.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.835+0000 7fb926ffd700 1 -- 172.21.15.77:0/2253217437 learned_addr learned my addr 172.21.15.77:0/2253217437 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:19.846 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.836+0000 7fb926ffd700 1 -- 172.21.15.77:0/2253217437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb92806d3f0 msgr2=0x7fb9280bf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:19.846 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.836+0000 7fb926ffd700 1 --2- 172.21.15.77:0/2253217437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb92806d3f0 0x7fb9280bf680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:19.846 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.836+0000 7fb926ffd700 1 -- 172.21.15.77:0/2253217437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb928156120 msgr2=0x7fb9280c0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:19.846 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.836+0000 7fb926ffd700 1 --2- 172.21.15.77:0/2253217437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb928156120 0x7fb9280c0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:19.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.836+0000 7fb926ffd700 1 -- 172.21.15.77:0/2253217437 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb920009ae0 con 0x7fb9280c5930 2022-04-03T19:55:19.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.836+0000 7fb927fff700 1 --2- 172.21.15.77:0/2253217437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb928156120 0x7fb9280c0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:55:19.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.836+0000 7fb9277fe700 1 --2- 172.21.15.77:0/2253217437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb92806d3f0 0x7fb9280bf680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:55:19.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.836+0000 7fb926ffd700 1 --2- 172.21.15.77:0/2253217437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9280c5930 0x7fb9280c0510 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb91c00e900 tx=0x7fb91c009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:19.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.836+0000 7fb90f7fe700 1 -- 172.21.15.77:0/2253217437 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91c013680 con 0x7fb9280c5930 2022-04-03T19:55:19.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.836+0000 7fb90f7fe700 1 -- 172.21.15.77:0/2253217437 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91c0137e0 con 0x7fb9280c5930 2022-04-03T19:55:19.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.836+0000 7fb90f7fe700 1 -- 172.21.15.77:0/2253217437 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91c013b80 con 0x7fb9280c5930 2022-04-03T19:55:19.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.836+0000 7fb92ce5f700 1 -- 172.21.15.77:0/2253217437 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb928151c20 con 0x7fb9280c5930 2022-04-03T19:55:19.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.837+0000 7fb92ce5f700 1 -- 172.21.15.77:0/2253217437 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9281661d0 con 0x7fb9280c5930 2022-04-03T19:55:19.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.837+0000 7fb92ce5f700 1 -- 172.21.15.77:0/2253217437 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9280bef80 con 0x7fb9280c5930 2022-04-03T19:55:19.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.839+0000 7fb90f7fe700 1 -- 172.21.15.77:0/2253217437 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fb91c023070 con 0x7fb9280c5930 2022-04-03T19:55:19.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.839+0000 7fb90f7fe700 1 --2- 172.21.15.77:0/2253217437 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb908063ce0 0x7fb908066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:19.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.839+0000 7fb90f7fe700 1 -- 172.21.15.77:0/2253217437 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7fb91c080c60 con 0x7fb9280c5930 2022-04-03T19:55:19.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.840+0000 7fb9277fe700 1 --2- 172.21.15.77:0/2253217437 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb908063ce0 0x7fb908066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:19.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.840+0000 7fb9277fe700 1 --2- 172.21.15.77:0/2253217437 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb908063ce0 0x7fb908066190 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb918009f90 tx=0x7fb918009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:19.854 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:19.843+0000 7fb90f7fe700 1 -- 172.21.15.77:0/2253217437 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fb91c04f620 con 0x7fb9280c5930 2022-04-03T19:55:20.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:20.076+0000 7fb92ce5f700 1 -- 172.21.15.77:0/2253217437 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9280c01c0 con 0x7fb9280c5930 2022-04-03T19:55:20.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:20.078+0000 7fb90f7fe700 1 -- 172.21.15.77:0/2253217437 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5356 (secure 0 0 0) 0x7fb91c017090 con 0x7fb9280c5930 2022-04-03T19:55:20.078 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:55:20.079 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:19.362019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:55:20.081 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:20.080+0000 7fb92ce5f700 1 -- 172.21.15.77:0/2253217437 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb908063ce0 msgr2=0x7fb908066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:20.081 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:20.080+0000 7fb92ce5f700 1 --2- 172.21.15.77:0/2253217437 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb908063ce0 0x7fb908066190 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb918009f90 tx=0x7fb918009060 comp rx=0 tx=0).stop 2022-04-03T19:55:20.081 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:20.080+0000 7fb92ce5f700 1 -- 172.21.15.77:0/2253217437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9280c5930 msgr2=0x7fb9280c0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:20.081 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:20.080+0000 7fb92ce5f700 1 --2- 172.21.15.77:0/2253217437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9280c5930 0x7fb9280c0510 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb91c00e900 tx=0x7fb91c009060 comp rx=0 tx=0).stop 2022-04-03T19:55:20.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:20.080+0000 7fb92ce5f700 1 -- 172.21.15.77:0/2253217437 shutdown_connections 2022-04-03T19:55:20.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:20.080+0000 7fb92ce5f700 1 --2- 172.21.15.77:0/2253217437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb928156120 0x7fb9280c0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:20.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:20.080+0000 7fb92ce5f700 1 --2- 172.21.15.77:0/2253217437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb92806d3f0 0x7fb9280bf680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:20.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:20.080+0000 7fb92ce5f700 1 --2- 172.21.15.77:0/2253217437 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb908063ce0 0x7fb908066190 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:20.083 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:20.080+0000 7fb92ce5f700 1 --2- 172.21.15.77:0/2253217437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9280c5930 0x7fb9280c0510 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:20.083 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:20.080+0000 7fb92ce5f700 1 -- 172.21.15.77:0/2253217437 >> 172.21.15.77:0/2253217437 conn(0x7fb9280cb720 msgr2=0x7fb9281534d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:20.083 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:20.080+0000 7fb92ce5f700 1 -- 172.21.15.77:0/2253217437 shutdown_connections 2022-04-03T19:55:20.083 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:20.080+0000 7fb92ce5f700 1 -- 172.21.15.77:0/2253217437 wait complete. 2022-04-03T19:55:20.084 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 39 2022-04-03T19:55:20.105 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:55:22.106 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:55:22.313 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.312+0000 7f38b1d94700 1 Processor -- start 2022-04-03T19:55:22.313 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.312+0000 7f38b1d94700 1 -- start start 2022-04-03T19:55:22.316 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.313+0000 7f38b1d94700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38ac06d3f0 0x7f38ac06d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:22.317 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.313+0000 7f38b1d94700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f38ac0c2a90 0x7f38ac06c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:22.317 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.313+0000 7f38b1d94700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38ac1562f0 con 0x7f38ac0c5930 2022-04-03T19:55:22.317 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.313+0000 7f38b1d94700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38ac0c2f80 con 0x7f38ac0c2a90 2022-04-03T19:55:22.317 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.313+0000 7f38b1d94700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac0c3260 con 0x7f38ac06d3f0 2022-04-03T19:55:22.317 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.314+0000 7f38b1593700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f38ac0c2a90 0x7f38ac06c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:22.318 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.314+0000 7f38b0d92700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38ac06d3f0 0x7f38ac06d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:22.318 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.314+0000 7f38abfff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f38ac0c5930 0x7f38ac155be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38556/0 (socket says 172.21.15.77:38556) 2022-04-03T19:55:22.318 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.314+0000 7f38abfff700 1 -- 172.21.15.77:0/3147591930 learned_addr learned my addr 172.21.15.77:0/3147591930 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:55:22.318 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.314+0000 7f38b1593700 1 -- 172.21.15.77:0/3147591930 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38ac06d3f0 msgr2=0x7f38ac06d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:22.319 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.314+0000 7f38b1593700 1 --2- 172.21.15.77:0/3147591930 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38ac06d3f0 0x7f38ac06d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:22.319 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.314+0000 7f38b1593700 1 -- 172.21.15.77:0/3147591930 >> v1:172.21.15.77:6789/0 conn(0x7f38ac0c5930 legacy=0x7f38ac155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:22.319 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.314+0000 7f38ab7fe700 1 -- 172.21.15.77:0/3147591930 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412933231 0 0) 0x7f38ac1562f0 con 0x7f38ac0c5930 2022-04-03T19:55:22.319 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.314+0000 7f38b1593700 1 -- 172.21.15.77:0/3147591930 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ac0c3540 con 0x7f38ac0c2a90 2022-04-03T19:55:22.320 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.314+0000 7f38b0d92700 1 --2- 172.21.15.77:0/3147591930 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38ac06d3f0 0x7f38ac06d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:55:22.320 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.315+0000 7f38b1593700 1 --2- 172.21.15.77:0/3147591930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38ac0c2a90 0x7f38ac06c040 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f38a4009e60 tx=0x7f38a4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7bcebc78946abf3 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:22.320 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.315+0000 7f38ab7fe700 1 -- 172.21.15.77:0/3147591930 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a400e070 con 0x7f38ac0c2a90 2022-04-03T19:55:22.320 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.315+0000 7f38ab7fe700 1 -- 172.21.15.77:0/3147591930 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a400bbf0 con 0x7f38ac0c2a90 2022-04-03T19:55:22.320 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.315+0000 7f38ab7fe700 1 -- 172.21.15.77:0/3147591930 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a4012430 con 0x7f38ac0c2a90 2022-04-03T19:55:22.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.315+0000 7f38b1d94700 1 -- 172.21.15.77:0/3147591930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38ac0c2a90 msgr2=0x7f38ac06c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:22.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.315+0000 7f38b1d94700 1 --2- 172.21.15.77:0/3147591930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38ac0c2a90 0x7f38ac06c040 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f38a4009e60 tx=0x7f38a4009780 comp rx=0 tx=0).stop 2022-04-03T19:55:22.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.315+0000 7f38b1d94700 1 -- 172.21.15.77:0/3147591930 shutdown_connections 2022-04-03T19:55:22.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.315+0000 7f38b1d94700 1 --2- 172.21.15.77:0/3147591930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38ac0c2a90 0x7f38ac06c040 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:22.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.315+0000 7f38b1d94700 1 --2- 172.21.15.77:0/3147591930 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38ac06d3f0 0x7f38ac06d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:22.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.315+0000 7f38b1d94700 1 -- 172.21.15.77:0/3147591930 >> 172.21.15.77:0/3147591930 conn(0x7f38ac0cb720 msgr2=0x7f38ac0cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:22.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.316+0000 7f38b1d94700 1 -- 172.21.15.77:0/3147591930 shutdown_connections 2022-04-03T19:55:22.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.316+0000 7f38b1d94700 1 -- 172.21.15.77:0/3147591930 wait complete. 2022-04-03T19:55:22.325 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.317+0000 7f38b1d94700 1 Processor -- start 2022-04-03T19:55:22.326 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.317+0000 7f38b1d94700 1 -- start start 2022-04-03T19:55:22.326 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.318+0000 7f38b1d94700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38ac06d3f0 0x7f38ac1575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:22.326 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.318+0000 7f38b1d94700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38ac0c2a90 0x7f38ac0c6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:22.326 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.318+0000 7f38b1d94700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38ac0c5930 0x7f38ac168770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:22.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.318+0000 7f38b1d94700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac0c0b40 con 0x7f38ac0c2a90 2022-04-03T19:55:22.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.318+0000 7f38b1d94700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac156540 con 0x7f38ac0c5930 2022-04-03T19:55:22.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.318+0000 7f38b1d94700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac1567e0 con 0x7f38ac06d3f0 2022-04-03T19:55:22.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.318+0000 7f38abfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38ac0c2a90 0x7f38ac0c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:22.328 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.318+0000 7f38b0d92700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38ac06d3f0 0x7f38ac1575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:22.328 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.318+0000 7f38abfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38ac0c2a90 0x7f38ac0c6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43156/0 (socket says 172.21.15.77:43156) 2022-04-03T19:55:22.328 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.318+0000 7f38abfff700 1 -- 172.21.15.77:0/2350373454 learned_addr learned my addr 172.21.15.77:0/2350373454 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:22.328 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.318+0000 7f38b1593700 1 --2- 172.21.15.77:0/2350373454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38ac0c5930 0x7f38ac168770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:22.328 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.319+0000 7f38abfff700 1 -- 172.21.15.77:0/2350373454 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38ac06d3f0 msgr2=0x7f38ac1575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:22.329 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.319+0000 7f38abfff700 1 --2- 172.21.15.77:0/2350373454 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38ac06d3f0 0x7f38ac1575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:22.329 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.319+0000 7f38abfff700 1 -- 172.21.15.77:0/2350373454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38ac0c5930 msgr2=0x7f38ac168770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:22.329 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.319+0000 7f38abfff700 1 --2- 172.21.15.77:0/2350373454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38ac0c5930 0x7f38ac168770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:22.329 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.319+0000 7f38abfff700 1 -- 172.21.15.77:0/2350373454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a00092a0 con 0x7f38ac0c2a90 2022-04-03T19:55:22.330 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.319+0000 7f38b0d92700 1 --2- 172.21.15.77:0/2350373454 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38ac06d3f0 0x7f38ac1575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:55:22.330 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.319+0000 7f38abfff700 1 --2- 172.21.15.77:0/2350373454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38ac0c2a90 0x7f38ac0c6890 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f389800c490 tx=0x7f3898007d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:22.330 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.319+0000 7f38a8ff9700 1 -- 172.21.15.77:0/2350373454 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389800ce40 con 0x7f38ac0c2a90 2022-04-03T19:55:22.330 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.319+0000 7f38a8ff9700 1 -- 172.21.15.77:0/2350373454 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3898007360 con 0x7f38ac0c2a90 2022-04-03T19:55:22.331 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.319+0000 7f38b1d94700 1 -- 172.21.15.77:0/2350373454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a4009a20 con 0x7f38ac0c2a90 2022-04-03T19:55:22.331 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.319+0000 7f38a8ff9700 1 -- 172.21.15.77:0/2350373454 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38980046b0 con 0x7f38ac0c2a90 2022-04-03T19:55:22.331 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.320+0000 7f38b1d94700 1 -- 172.21.15.77:0/2350373454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38ac156da0 con 0x7f38ac0c2a90 2022-04-03T19:55:22.331 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.321+0000 7f38b1d94700 1 -- 172.21.15.77:0/2350373454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38ac159780 con 0x7f38ac0c2a90 2022-04-03T19:55:22.331 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.322+0000 7f38a8ff9700 1 -- 172.21.15.77:0/2350373454 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f3898010030 con 0x7f38ac0c2a90 2022-04-03T19:55:22.332 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.323+0000 7f38a8ff9700 1 --2- 172.21.15.77:0/2350373454 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3894063ce0 0x7f3894066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:22.332 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.323+0000 7f38a8ff9700 1 -- 172.21.15.77:0/2350373454 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f38980804a0 con 0x7f38ac0c2a90 2022-04-03T19:55:22.332 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.323+0000 7f38b0d92700 1 --2- 172.21.15.77:0/2350373454 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3894063ce0 0x7f3894066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:22.332 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.324+0000 7f38b0d92700 1 --2- 172.21.15.77:0/2350373454 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3894063ce0 0x7f3894066190 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f38a00114c0 tx=0x7f38a0009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:22.333 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.327+0000 7f38a8ff9700 1 -- 172.21.15.77:0/2350373454 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f389804eec0 con 0x7f38ac0c2a90 2022-04-03T19:55:22.562 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.562+0000 7f38b1d94700 1 -- 172.21.15.77:0/2350373454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38ac04b050 con 0x7f38ac0c2a90 2022-04-03T19:55:22.563 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.562+0000 7f38a8ff9700 1 -- 172.21.15.77:0/2350373454 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5356 (secure 0 0 0) 0x7f38980524e0 con 0x7f38ac0c2a90 2022-04-03T19:55:22.563 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:55:22.563 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:19.362019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:55:22.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.564+0000 7f38b1d94700 1 -- 172.21.15.77:0/2350373454 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3894063ce0 msgr2=0x7f3894066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:22.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.564+0000 7f38b1d94700 1 --2- 172.21.15.77:0/2350373454 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3894063ce0 0x7f3894066190 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f38a00114c0 tx=0x7f38a0009060 comp rx=0 tx=0).stop 2022-04-03T19:55:22.566 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.565+0000 7f38b1d94700 1 -- 172.21.15.77:0/2350373454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38ac0c2a90 msgr2=0x7f38ac0c6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:22.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.565+0000 7f38b1d94700 1 --2- 172.21.15.77:0/2350373454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38ac0c2a90 0x7f38ac0c6890 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f389800c490 tx=0x7f3898007d60 comp rx=0 tx=0).stop 2022-04-03T19:55:22.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.565+0000 7f38b1d94700 1 -- 172.21.15.77:0/2350373454 shutdown_connections 2022-04-03T19:55:22.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.565+0000 7f38b1d94700 1 --2- 172.21.15.77:0/2350373454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38ac0c5930 0x7f38ac168770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:22.567 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.565+0000 7f38b1d94700 1 --2- 172.21.15.77:0/2350373454 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3894063ce0 0x7f3894066190 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:22.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.565+0000 7f38b1d94700 1 --2- 172.21.15.77:0/2350373454 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38ac06d3f0 0x7f38ac1575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:22.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.565+0000 7f38b1d94700 1 --2- 172.21.15.77:0/2350373454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f38ac0c2a90 0x7f38ac0c6890 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:22.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.565+0000 7f38b1d94700 1 -- 172.21.15.77:0/2350373454 >> 172.21.15.77:0/2350373454 conn(0x7f38ac0cb720 msgr2=0x7f38ac14dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:22.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.565+0000 7f38b1d94700 1 -- 172.21.15.77:0/2350373454 shutdown_connections 2022-04-03T19:55:22.568 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:22.565+0000 7f38b1d94700 1 -- 172.21.15.77:0/2350373454 wait complete. 2022-04-03T19:55:22.570 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 39 2022-04-03T19:55:22.590 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 39, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-04-03T19:44:21.275716+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 39, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5463": { "addr": "172.21.15.77:6837/2777516376", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6836", "nonce": 2777516376, "type": "v2" }, { "addr": "172.21.15.77:6837", "nonce": 2777516376, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 5463, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:reconnect", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-04-03T19:55:19.362019+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5463 } } } ], "standbys": [ { "addr": "172.21.15.77:6834/2116223509", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6832", "nonce": 2116223509, "type": "v2" }, { "addr": "172.21.15.77:6834", "nonce": 2116223509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4245, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6836/2374276747", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2374276747, "type": "v2" }, { "addr": "172.21.15.186:6836", "nonce": 2374276747, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4259, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/610990853", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6838", "nonce": 610990853, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 610990853, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4265, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.77:6835/170053412", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6833", "nonce": 170053412, "type": "v2" }, { "addr": "172.21.15.77:6835", "nonce": 170053412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4271, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-04-03T19:55:24.592 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:55:24.799 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.798+0000 7f0f1a9db700 1 Processor -- start 2022-04-03T19:55:24.799 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.798+0000 7f0f1a9db700 1 -- start start 2022-04-03T19:55:24.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.799+0000 7f0f1a9db700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0f140c3270 0x7f0f140c2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:24.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.799+0000 7f0f1a9db700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f140c3640 0x7f0f14069600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:24.802 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.800+0000 7f0f1a9db700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0f140c7af0 0x7f0f14069b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:24.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.800+0000 7f0f1a9db700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f140c4b90 con 0x7f0f140c3270 2022-04-03T19:55:24.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.800+0000 7f0f1a9db700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f140c3d10 con 0x7f0f140c7af0 2022-04-03T19:55:24.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.800+0000 7f0f1a9db700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f140c2a40 con 0x7f0f140c3640 2022-04-03T19:55:24.803 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.800+0000 7f0f199d9700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0f140c3270 0x7f0f140c2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:24.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.800+0000 7f0f199d9700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0f140c3270 0x7f0f140c2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43158/0 (socket says 172.21.15.77:43158) 2022-04-03T19:55:24.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.800+0000 7f0f199d9700 1 -- 172.21.15.77:0/2203889433 learned_addr learned my addr 172.21.15.77:0/2203889433 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:24.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.800+0000 7f0f1a1da700 1 --2- 172.21.15.77:0/2203889433 >> v2:172.21.15.186:3300/0 conn(0x7f0f140c7af0 0x7f0f14069b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:24.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.800+0000 7f0f191d8700 1 --2- 172.21.15.77:0/2203889433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f140c3640 0x7f0f14069600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:24.805 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.800+0000 7f0f199d9700 1 -- 172.21.15.77:0/2203889433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f140c3640 msgr2=0x7f0f14069600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:24.805 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.800+0000 7f0f199d9700 1 --2- 172.21.15.77:0/2203889433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f140c3640 0x7f0f14069600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:24.805 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.800+0000 7f0f199d9700 1 -- 172.21.15.77:0/2203889433 >> v2:172.21.15.186:3300/0 conn(0x7f0f140c7af0 msgr2=0x7f0f14069b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:24.805 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.800+0000 7f0f199d9700 1 --2- 172.21.15.77:0/2203889433 >> v2:172.21.15.186:3300/0 conn(0x7f0f140c7af0 0x7f0f14069b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:24.805 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.800+0000 7f0f199d9700 1 -- 172.21.15.77:0/2203889433 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f140c3f80 con 0x7f0f140c3270 2022-04-03T19:55:24.806 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.801+0000 7f0f199d9700 1 --2- 172.21.15.77:0/2203889433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f140c3270 0x7f0f140c2500 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0f140c7230 tx=0x7f0f100090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44601c55f2497f98 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:24.806 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.801+0000 7f0f189d7700 1 -- 172.21.15.77:0/2203889433 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f10004030 con 0x7f0f140c3270 2022-04-03T19:55:24.806 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.801+0000 7f0f189d7700 1 -- 172.21.15.77:0/2203889433 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f10004190 con 0x7f0f140c3270 2022-04-03T19:55:24.806 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.801+0000 7f0f189d7700 1 -- 172.21.15.77:0/2203889433 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1000f6b0 con 0x7f0f140c3270 2022-04-03T19:55:24.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.801+0000 7f0f1a9db700 1 -- 172.21.15.77:0/2203889433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f140c3270 msgr2=0x7f0f140c2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:24.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.801+0000 7f0f1a9db700 1 --2- 172.21.15.77:0/2203889433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f140c3270 0x7f0f140c2500 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0f140c7230 tx=0x7f0f100090f0 comp rx=0 tx=0).stop 2022-04-03T19:55:24.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.802+0000 7f0f1a9db700 1 -- 172.21.15.77:0/2203889433 shutdown_connections 2022-04-03T19:55:24.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.802+0000 7f0f1a9db700 1 --2- 172.21.15.77:0/2203889433 >> v2:172.21.15.186:3300/0 conn(0x7f0f140c7af0 0x7f0f14069b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:24.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.802+0000 7f0f1a9db700 1 --2- 172.21.15.77:0/2203889433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f140c3270 0x7f0f140c2500 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:24.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.802+0000 7f0f1a9db700 1 --2- 172.21.15.77:0/2203889433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f140c3640 0x7f0f14069600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:24.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.802+0000 7f0f1a9db700 1 -- 172.21.15.77:0/2203889433 >> 172.21.15.77:0/2203889433 conn(0x7f0f14066da0 msgr2=0x7f0f140671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:24.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.802+0000 7f0f1a9db700 1 -- 172.21.15.77:0/2203889433 shutdown_connections 2022-04-03T19:55:24.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.802+0000 7f0f1a9db700 1 -- 172.21.15.77:0/2203889433 wait complete. 2022-04-03T19:55:24.812 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.803+0000 7f0f1a9db700 1 Processor -- start 2022-04-03T19:55:24.812 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.803+0000 7f0f1a9db700 1 -- start start 2022-04-03T19:55:24.812 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.805+0000 7f0f1a9db700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f140c3270 0x7f0f140bd070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:24.812 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.805+0000 7f0f1a9db700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f140c3640 0x7f0f14151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:24.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.805+0000 7f0f1a9db700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f140c7af0 0x7f0f14154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:24.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.805+0000 7f0f1a9db700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f140bcbe0 con 0x7f0f140c3270 2022-04-03T19:55:24.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.805+0000 7f0f1a9db700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f140b6d30 con 0x7f0f140c7af0 2022-04-03T19:55:24.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.805+0000 7f0f1a9db700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f14169fa0 con 0x7f0f140c3640 2022-04-03T19:55:24.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.805+0000 7f0f199d9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f140c3270 0x7f0f140bd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:24.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.806+0000 7f0f1a1da700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f140c7af0 0x7f0f14154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:24.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.806+0000 7f0f199d9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f140c3270 0x7f0f140bd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43160/0 (socket says 172.21.15.77:43160) 2022-04-03T19:55:24.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.806+0000 7f0f199d9700 1 -- 172.21.15.77:0/2540430848 learned_addr learned my addr 172.21.15.77:0/2540430848 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:24.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.806+0000 7f0f191d8700 1 --2- 172.21.15.77:0/2540430848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f140c3640 0x7f0f14151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:24.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.806+0000 7f0f199d9700 1 -- 172.21.15.77:0/2540430848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f140c3640 msgr2=0x7f0f14151ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:24.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.806+0000 7f0f199d9700 1 --2- 172.21.15.77:0/2540430848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f140c3640 0x7f0f14151ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:24.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.806+0000 7f0f199d9700 1 -- 172.21.15.77:0/2540430848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f140c7af0 msgr2=0x7f0f14154cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:24.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.806+0000 7f0f199d9700 1 --2- 172.21.15.77:0/2540430848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f140c7af0 0x7f0f14154cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:24.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.806+0000 7f0f199d9700 1 -- 172.21.15.77:0/2540430848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f10009370 con 0x7f0f140c3270 2022-04-03T19:55:24.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.806+0000 7f0f1a1da700 1 --2- 172.21.15.77:0/2540430848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f140c7af0 0x7f0f14154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:55:24.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.806+0000 7f0f199d9700 1 --2- 172.21.15.77:0/2540430848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f140c3270 0x7f0f140bd070 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0f10009130 tx=0x7f0f10018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:24.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.807+0000 7f0f01ffb700 1 -- 172.21.15.77:0/2540430848 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1000f6b0 con 0x7f0f140c3270 2022-04-03T19:55:24.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.807+0000 7f0f01ffb700 1 -- 172.21.15.77:0/2540430848 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f100036f0 con 0x7f0f140c3270 2022-04-03T19:55:24.817 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.807+0000 7f0f01ffb700 1 -- 172.21.15.77:0/2540430848 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f10003ab0 con 0x7f0f140c3270 2022-04-03T19:55:24.817 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.807+0000 7f0f1a9db700 1 -- 172.21.15.77:0/2540430848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f140bbfd0 con 0x7f0f140c3270 2022-04-03T19:55:24.817 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.807+0000 7f0f1a9db700 1 -- 172.21.15.77:0/2540430848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f140bc590 con 0x7f0f140c3270 2022-04-03T19:55:24.817 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.808+0000 7f0f1a9db700 1 -- 172.21.15.77:0/2540430848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f140bcec0 con 0x7f0f140c3270 2022-04-03T19:55:24.818 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.808+0000 7f0f01ffb700 1 -- 172.21.15.77:0/2540430848 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f0f10018420 con 0x7f0f140c3270 2022-04-03T19:55:24.818 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.809+0000 7f0f01ffb700 1 --2- 172.21.15.77:0/2540430848 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0ef4063ce0 0x7f0ef4066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:24.818 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.809+0000 7f0f01ffb700 1 -- 172.21.15.77:0/2540430848 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f0f10014070 con 0x7f0f140c3270 2022-04-03T19:55:24.818 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.810+0000 7f0f191d8700 1 --2- 172.21.15.77:0/2540430848 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0ef4063ce0 0x7f0ef4066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:24.819 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.810+0000 7f0f191d8700 1 --2- 172.21.15.77:0/2540430848 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0ef4063ce0 0x7f0ef4066190 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0f14152aa0 tx=0x7f0f04009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:24.819 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:24.814+0000 7f0f01ffb700 1 -- 172.21.15.77:0/2540430848 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f0f10050890 con 0x7f0f140c3270 2022-04-03T19:55:25.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:25.049+0000 7f0f1a9db700 1 -- 172.21.15.77:0/2540430848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f1404b050 con 0x7f0f140c3270 2022-04-03T19:55:25.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:25.050+0000 7f0f01ffb700 1 -- 172.21.15.77:0/2540430848 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5356 (secure 0 0 0) 0x7f0f10053eb0 con 0x7f0f140c3270 2022-04-03T19:55:25.051 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:55:25.051 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:19.362019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:55:25.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:25.052+0000 7f0f1a9db700 1 -- 172.21.15.77:0/2540430848 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0ef4063ce0 msgr2=0x7f0ef4066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:25.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:25.052+0000 7f0f1a9db700 1 --2- 172.21.15.77:0/2540430848 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0ef4063ce0 0x7f0ef4066190 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0f14152aa0 tx=0x7f0f04009060 comp rx=0 tx=0).stop 2022-04-03T19:55:25.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:25.052+0000 7f0f1a9db700 1 -- 172.21.15.77:0/2540430848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f140c3270 msgr2=0x7f0f140bd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:25.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:25.052+0000 7f0f1a9db700 1 --2- 172.21.15.77:0/2540430848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f140c3270 0x7f0f140bd070 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0f10009130 tx=0x7f0f10018000 comp rx=0 tx=0).stop 2022-04-03T19:55:25.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:25.052+0000 7f0f1a9db700 1 -- 172.21.15.77:0/2540430848 shutdown_connections 2022-04-03T19:55:25.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:25.052+0000 7f0f1a9db700 1 --2- 172.21.15.77:0/2540430848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f140c7af0 0x7f0f14154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:25.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:25.052+0000 7f0f1a9db700 1 --2- 172.21.15.77:0/2540430848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f140c3270 0x7f0f140bd070 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:25.055 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:25.052+0000 7f0f1a9db700 1 --2- 172.21.15.77:0/2540430848 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f0ef4063ce0 0x7f0ef4066190 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:25.055 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:25.052+0000 7f0f1a9db700 1 --2- 172.21.15.77:0/2540430848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f140c3640 0x7f0f14151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:25.055 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:25.052+0000 7f0f1a9db700 1 -- 172.21.15.77:0/2540430848 >> 172.21.15.77:0/2540430848 conn(0x7f0f14066da0 msgr2=0x7f0f14153350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:25.055 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:25.052+0000 7f0f1a9db700 1 -- 172.21.15.77:0/2540430848 shutdown_connections 2022-04-03T19:55:25.055 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:25.052+0000 7f0f1a9db700 1 -- 172.21.15.77:0/2540430848 wait complete. 2022-04-03T19:55:25.056 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 39 2022-04-03T19:55:25.077 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:55:27.078 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:55:27.286 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.286+0000 7fe4cd282700 1 Processor -- start 2022-04-03T19:55:27.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.286+0000 7fe4cd282700 1 -- start start 2022-04-03T19:55:27.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.287+0000 7fe4cd282700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe4c806bbe0 0x7fe4c81584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:27.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.287+0000 7fe4cd282700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe4c806bfb0 0x7fe4c806d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:27.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.287+0000 7fe4cd282700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4c806c500 0x7fe4c814eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:27.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.287+0000 7fe4cd282700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4c8159450 con 0x7fe4c806bfb0 2022-04-03T19:55:27.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.287+0000 7fe4cd282700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4c806d260 con 0x7fe4c806bbe0 2022-04-03T19:55:27.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.287+0000 7fe4cd282700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c80c6330 con 0x7fe4c806c500 2022-04-03T19:55:27.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.287+0000 7fe4c77fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe4c806bfb0 0x7fe4c806d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:27.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.287+0000 7fe4c77fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe4c806bfb0 0x7fe4c806d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43162/0 (socket says 172.21.15.77:43162) 2022-04-03T19:55:27.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.287+0000 7fe4c77fe700 1 -- 172.21.15.77:0/1378940220 learned_addr learned my addr 172.21.15.77:0/1378940220 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:27.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.287+0000 7fe4c7fff700 1 --2- 172.21.15.77:0/1378940220 >> v2:172.21.15.186:3300/0 conn(0x7fe4c806bbe0 0x7fe4c81584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:27.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.287+0000 7fe4cca81700 1 --2- 172.21.15.77:0/1378940220 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4c806c500 0x7fe4c814eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:27.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.288+0000 7fe4c77fe700 1 -- 172.21.15.77:0/1378940220 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4c806c500 msgr2=0x7fe4c814eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:27.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.288+0000 7fe4c77fe700 1 --2- 172.21.15.77:0/1378940220 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4c806c500 0x7fe4c814eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:27.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.288+0000 7fe4c77fe700 1 -- 172.21.15.77:0/1378940220 >> v2:172.21.15.186:3300/0 conn(0x7fe4c806bbe0 msgr2=0x7fe4c81584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:27.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.288+0000 7fe4c77fe700 1 --2- 172.21.15.77:0/1378940220 >> v2:172.21.15.186:3300/0 conn(0x7fe4c806bbe0 0x7fe4c81584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:27.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.288+0000 7fe4c77fe700 1 -- 172.21.15.77:0/1378940220 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c8158bb0 con 0x7fe4c806bfb0 2022-04-03T19:55:27.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.288+0000 7fe4c77fe700 1 --2- 172.21.15.77:0/1378940220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4c806bfb0 0x7fe4c806d6f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8088d40 tx=0x7fe4bc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a64f4cdca748a46 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:27.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.288+0000 7fe4c6ffd700 1 -- 172.21.15.77:0/1378940220 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc004030 con 0x7fe4c806bfb0 2022-04-03T19:55:27.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.289+0000 7fe4c6ffd700 1 -- 172.21.15.77:0/1378940220 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4bc004190 con 0x7fe4c806bfb0 2022-04-03T19:55:27.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.289+0000 7fe4c6ffd700 1 -- 172.21.15.77:0/1378940220 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc011630 con 0x7fe4c806bfb0 2022-04-03T19:55:27.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.289+0000 7fe4cd282700 1 -- 172.21.15.77:0/1378940220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4c806bfb0 msgr2=0x7fe4c806d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:27.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.289+0000 7fe4cd282700 1 --2- 172.21.15.77:0/1378940220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4c806bfb0 0x7fe4c806d6f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8088d40 tx=0x7fe4bc0090f0 comp rx=0 tx=0).stop 2022-04-03T19:55:27.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.289+0000 7fe4cd282700 1 -- 172.21.15.77:0/1378940220 shutdown_connections 2022-04-03T19:55:27.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.289+0000 7fe4cd282700 1 --2- 172.21.15.77:0/1378940220 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4c806c500 0x7fe4c814eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:27.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.289+0000 7fe4cd282700 1 --2- 172.21.15.77:0/1378940220 >> v2:172.21.15.186:3300/0 conn(0x7fe4c806bbe0 0x7fe4c81584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:27.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.289+0000 7fe4cd282700 1 --2- 172.21.15.77:0/1378940220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4c806bfb0 0x7fe4c806d6f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:27.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.289+0000 7fe4cd282700 1 -- 172.21.15.77:0/1378940220 >> 172.21.15.77:0/1378940220 conn(0x7fe4c80cb6b0 msgr2=0x7fe4c80cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:27.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.289+0000 7fe4cd282700 1 -- 172.21.15.77:0/1378940220 shutdown_connections 2022-04-03T19:55:27.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.289+0000 7fe4cd282700 1 -- 172.21.15.77:0/1378940220 wait complete. 2022-04-03T19:55:27.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.291+0000 7fe4cd282700 1 Processor -- start 2022-04-03T19:55:27.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.291+0000 7fe4cd282700 1 -- start start 2022-04-03T19:55:27.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.292+0000 7fe4cd282700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4c806bbe0 0x7fe4c814a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:27.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.292+0000 7fe4cd282700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4c806bfb0 0x7fe4c81666d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:27.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.292+0000 7fe4cd282700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4c806c500 0x7fe4c8166c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:27.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.292+0000 7fe4cd282700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c8151690 con 0x7fe4c806bbe0 2022-04-03T19:55:27.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.292+0000 7fe4cd282700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c80c3600 con 0x7fe4c806bfb0 2022-04-03T19:55:27.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.292+0000 7fe4cd282700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c81507a0 con 0x7fe4c806c500 2022-04-03T19:55:27.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.292+0000 7fe4c7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4c806bbe0 0x7fe4c814a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:27.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.292+0000 7fe4c7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4c806bbe0 0x7fe4c814a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43164/0 (socket says 172.21.15.77:43164) 2022-04-03T19:55:27.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.292+0000 7fe4c7fff700 1 -- 172.21.15.77:0/1475660727 learned_addr learned my addr 172.21.15.77:0/1475660727 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:27.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.292+0000 7fe4c77fe700 1 --2- 172.21.15.77:0/1475660727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4c806bfb0 0x7fe4c81666d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:27.302 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.292+0000 7fe4cca81700 1 --2- 172.21.15.77:0/1475660727 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4c806c500 0x7fe4c8166c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:27.302 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.293+0000 7fe4c7fff700 1 -- 172.21.15.77:0/1475660727 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4c806c500 msgr2=0x7fe4c8166c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:27.302 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.293+0000 7fe4c7fff700 1 --2- 172.21.15.77:0/1475660727 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4c806c500 0x7fe4c8166c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:27.302 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.293+0000 7fe4c7fff700 1 -- 172.21.15.77:0/1475660727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4c806bfb0 msgr2=0x7fe4c81666d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:27.303 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.293+0000 7fe4c7fff700 1 --2- 172.21.15.77:0/1475660727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4c806bfb0 0x7fe4c81666d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:27.303 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.293+0000 7fe4c7fff700 1 -- 172.21.15.77:0/1475660727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4bc009370 con 0x7fe4c806bbe0 2022-04-03T19:55:27.303 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.293+0000 7fe4c77fe700 1 --2- 172.21.15.77:0/1475660727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4c806bfb0 0x7fe4c81666d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:55:27.303 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.293+0000 7fe4c7fff700 1 --2- 172.21.15.77:0/1475660727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4c806bbe0 0x7fe4c814a5d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe4b800e900 tx=0x7fe4b8009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:27.304 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.293+0000 7fe4affff700 1 -- 172.21.15.77:0/1475660727 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b8013680 con 0x7fe4c806bbe0 2022-04-03T19:55:27.304 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.293+0000 7fe4affff700 1 -- 172.21.15.77:0/1475660727 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b80137e0 con 0x7fe4c806bbe0 2022-04-03T19:55:27.304 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.293+0000 7fe4cd282700 1 -- 172.21.15.77:0/1475660727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c8150a80 con 0x7fe4c806bbe0 2022-04-03T19:55:27.304 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.293+0000 7fe4affff700 1 -- 172.21.15.77:0/1475660727 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b8013b80 con 0x7fe4c806bbe0 2022-04-03T19:55:27.305 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.294+0000 7fe4cd282700 1 -- 172.21.15.77:0/1475660727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c8151040 con 0x7fe4c806bbe0 2022-04-03T19:55:27.305 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.294+0000 7fe4cd282700 1 -- 172.21.15.77:0/1475660727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c81662f0 con 0x7fe4c806bbe0 2022-04-03T19:55:27.305 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.295+0000 7fe4affff700 1 -- 172.21.15.77:0/1475660727 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fe4b8019030 con 0x7fe4c806bbe0 2022-04-03T19:55:27.306 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.296+0000 7fe4affff700 1 --2- 172.21.15.77:0/1475660727 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe4a8063d10 0x7fe4a80661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:27.306 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.297+0000 7fe4affff700 1 -- 172.21.15.77:0/1475660727 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7fe4b8080a40 con 0x7fe4c806bbe0 2022-04-03T19:55:27.307 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.297+0000 7fe4c77fe700 1 --2- 172.21.15.77:0/1475660727 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe4a8063d10 0x7fe4a80661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:27.307 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.298+0000 7fe4c77fe700 1 --2- 172.21.15.77:0/1475660727 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe4a8063d10 0x7fe4a80661c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8088d40 tx=0x7fe4bc016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:27.307 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.301+0000 7fe4affff700 1 -- 172.21.15.77:0/1475660727 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fe4b804f290 con 0x7fe4c806bbe0 2022-04-03T19:55:27.532 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.532+0000 7fe4cd282700 1 -- 172.21.15.77:0/1475660727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4c80c7ec0 con 0x7fe4c806bbe0 2022-04-03T19:55:27.533 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.533+0000 7fe4affff700 1 -- 172.21.15.77:0/1475660727 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5353 (secure 0 0 0) 0x7fe4b80528b0 con 0x7fe4c806bbe0 2022-04-03T19:55:27.534 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:55:27.534 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:25.260684+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:rejoin","state_seq":26,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:55:27.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.535+0000 7fe4cd282700 1 -- 172.21.15.77:0/1475660727 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe4a8063d10 msgr2=0x7fe4a80661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:27.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.535+0000 7fe4cd282700 1 --2- 172.21.15.77:0/1475660727 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe4a8063d10 0x7fe4a80661c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8088d40 tx=0x7fe4bc016000 comp rx=0 tx=0).stop 2022-04-03T19:55:27.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.535+0000 7fe4cd282700 1 -- 172.21.15.77:0/1475660727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4c806bbe0 msgr2=0x7fe4c814a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:27.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.535+0000 7fe4cd282700 1 --2- 172.21.15.77:0/1475660727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4c806bbe0 0x7fe4c814a5d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe4b800e900 tx=0x7fe4b8009060 comp rx=0 tx=0).stop 2022-04-03T19:55:27.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.535+0000 7fe4cd282700 1 -- 172.21.15.77:0/1475660727 shutdown_connections 2022-04-03T19:55:27.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.535+0000 7fe4cd282700 1 --2- 172.21.15.77:0/1475660727 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe4a8063d10 0x7fe4a80661c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:27.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.535+0000 7fe4cd282700 1 --2- 172.21.15.77:0/1475660727 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4c806c500 0x7fe4c8166c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:27.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.535+0000 7fe4cd282700 1 --2- 172.21.15.77:0/1475660727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4c806bbe0 0x7fe4c814a5d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:27.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.535+0000 7fe4cd282700 1 --2- 172.21.15.77:0/1475660727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4c806bfb0 0x7fe4c81666d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:27.538 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.535+0000 7fe4cd282700 1 -- 172.21.15.77:0/1475660727 >> 172.21.15.77:0/1475660727 conn(0x7fe4c80cb6b0 msgr2=0x7fe4c80c50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:27.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.536+0000 7fe4cd282700 1 -- 172.21.15.77:0/1475660727 shutdown_connections 2022-04-03T19:55:27.539 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:27.536+0000 7fe4cd282700 1 -- 172.21.15.77:0/1475660727 wait complete. 2022-04-03T19:55:27.539 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 40 2022-04-03T19:55:27.560 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:55:29.561 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:55:29.767 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.766+0000 7f80d8aad700 1 Processor -- start 2022-04-03T19:55:29.767 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.766+0000 7f80d8aad700 1 -- start start 2022-04-03T19:55:29.769 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.767+0000 7f80d8aad700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f80d40c1170 0x7f80d41527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:29.769 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.768+0000 7f80d8aad700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f80d40c3b10 0x7f80d415c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:29.770 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.768+0000 7f80d8aad700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d40c3710 con 0x7f80d4155370 2022-04-03T19:55:29.770 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.768+0000 7f80d8aad700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80d4152330 con 0x7f80d40c3b10 2022-04-03T19:55:29.770 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.768+0000 7f80d8aad700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80d40c1540 con 0x7f80d40c1170 2022-04-03T19:55:29.770 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.768+0000 7f80d37fe700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f80d40c1170 0x7f80d41527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:29.771 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.768+0000 7f80d3fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f80d40c3b10 0x7f80d415c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:29.771 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.768+0000 7f80d2ffd700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f80d4155370 0x7f80d4069110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44356/0 (socket says 172.21.15.77:44356) 2022-04-03T19:55:29.771 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.768+0000 7f80d2ffd700 1 -- 172.21.15.77:0/2339277046 learned_addr learned my addr 172.21.15.77:0/2339277046 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:55:29.771 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.768+0000 7f80d27fc700 1 -- 172.21.15.77:0/2339277046 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685523288 0 0) 0x7f80d40c3710 con 0x7f80d4155370 2022-04-03T19:55:29.772 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.768+0000 7f80d3fff700 1 -- 172.21.15.77:0/2339277046 >> v2:172.21.15.186:3300/0 conn(0x7f80d40c1170 msgr2=0x7f80d41527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:29.772 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.768+0000 7f80d3fff700 1 --2- 172.21.15.77:0/2339277046 >> v2:172.21.15.186:3300/0 conn(0x7f80d40c1170 0x7f80d41527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:29.772 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.768+0000 7f80d3fff700 1 -- 172.21.15.77:0/2339277046 >> v1:172.21.15.186:6789/0 conn(0x7f80d4155370 legacy=0x7f80d4069110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:29.772 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.769+0000 7f80d3fff700 1 -- 172.21.15.77:0/2339277046 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d40c4270 con 0x7f80d40c3b10 2022-04-03T19:55:29.773 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.769+0000 7f80d3fff700 1 --2- 172.21.15.77:0/2339277046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80d40c3b10 0x7f80d415c6f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f80d4063790 tx=0x7f80cc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cada824273eadb7 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:29.773 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.769+0000 7f80d27fc700 1 -- 172.21.15.77:0/2339277046 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80cc00e070 con 0x7f80d40c3b10 2022-04-03T19:55:29.773 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.769+0000 7f80d27fc700 1 -- 172.21.15.77:0/2339277046 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80cc00bbf0 con 0x7f80d40c3b10 2022-04-03T19:55:29.773 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.769+0000 7f80d27fc700 1 -- 172.21.15.77:0/2339277046 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80cc012430 con 0x7f80d40c3b10 2022-04-03T19:55:29.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.769+0000 7f80d8aad700 1 -- 172.21.15.77:0/2339277046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80d40c3b10 msgr2=0x7f80d415c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:29.777 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.769+0000 7f80d8aad700 1 --2- 172.21.15.77:0/2339277046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80d40c3b10 0x7f80d415c6f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f80d4063790 tx=0x7f80cc009780 comp rx=0 tx=0).stop 2022-04-03T19:55:29.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.770+0000 7f80d8aad700 1 -- 172.21.15.77:0/2339277046 shutdown_connections 2022-04-03T19:55:29.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.770+0000 7f80d8aad700 1 --2- 172.21.15.77:0/2339277046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80d40c3b10 0x7f80d415c6f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:29.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.770+0000 7f80d8aad700 1 --2- 172.21.15.77:0/2339277046 >> v2:172.21.15.186:3300/0 conn(0x7f80d40c1170 0x7f80d41527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:29.778 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.770+0000 7f80d8aad700 1 -- 172.21.15.77:0/2339277046 >> 172.21.15.77:0/2339277046 conn(0x7f80d4066da0 msgr2=0x7f80d40671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:29.779 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.770+0000 7f80d8aad700 1 -- 172.21.15.77:0/2339277046 shutdown_connections 2022-04-03T19:55:29.779 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.770+0000 7f80d8aad700 1 -- 172.21.15.77:0/2339277046 wait complete. 2022-04-03T19:55:29.779 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.771+0000 7f80d8aad700 1 Processor -- start 2022-04-03T19:55:29.779 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.771+0000 7f80d8aad700 1 -- start start 2022-04-03T19:55:29.780 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.772+0000 7f80d8aad700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80d40c1170 0x7f80d4165c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:29.780 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.772+0000 7f80d8aad700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80d40c3b10 0x7f80d4151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:29.780 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.772+0000 7f80d8aad700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80d4155370 0x7f80d40bf960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:29.780 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.772+0000 7f80d8aad700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d4153980 con 0x7f80d4155370 2022-04-03T19:55:29.781 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.772+0000 7f80d8aad700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d415add0 con 0x7f80d40c1170 2022-04-03T19:55:29.781 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.772+0000 7f80d8aad700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80d41661b0 con 0x7f80d40c3b10 2022-04-03T19:55:29.781 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.773+0000 7f80d37fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80d40c1170 0x7f80d4165c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:29.781 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.773+0000 7f80d2ffd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80d40c3b10 0x7f80d4151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:29.781 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.773+0000 7f80d37fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80d40c1170 0x7f80d4165c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38362/0 (socket says 172.21.15.77:38362) 2022-04-03T19:55:29.782 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.773+0000 7f80d37fe700 1 -- 172.21.15.77:0/3725061698 learned_addr learned my addr 172.21.15.77:0/3725061698 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:29.782 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.773+0000 7f80d3fff700 1 --2- 172.21.15.77:0/3725061698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80d4155370 0x7f80d40bf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:29.782 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.773+0000 7f80d37fe700 1 -- 172.21.15.77:0/3725061698 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80d40c3b10 msgr2=0x7f80d4151850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:29.782 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.773+0000 7f80d37fe700 1 --2- 172.21.15.77:0/3725061698 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80d40c3b10 0x7f80d4151850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:29.783 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.773+0000 7f80d37fe700 1 -- 172.21.15.77:0/3725061698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80d4155370 msgr2=0x7f80d40bf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:29.783 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.773+0000 7f80d37fe700 1 --2- 172.21.15.77:0/3725061698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80d4155370 0x7f80d40bf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:29.783 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.773+0000 7f80d37fe700 1 -- 172.21.15.77:0/3725061698 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80cc009a20 con 0x7f80d40c1170 2022-04-03T19:55:29.783 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.773+0000 7f80d37fe700 1 --2- 172.21.15.77:0/3725061698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80d40c1170 0x7f80d4165c70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f80c400e930 tx=0x7f80c4009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:29.784 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.774+0000 7f80bb7fe700 1 -- 172.21.15.77:0/3725061698 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c4013680 con 0x7f80d40c1170 2022-04-03T19:55:29.784 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.774+0000 7f80bb7fe700 1 -- 172.21.15.77:0/3725061698 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c40137e0 con 0x7f80d40c1170 2022-04-03T19:55:29.784 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.774+0000 7f80d8aad700 1 -- 172.21.15.77:0/3725061698 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80d4151d90 con 0x7f80d40c1170 2022-04-03T19:55:29.784 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.774+0000 7f80bb7fe700 1 -- 172.21.15.77:0/3725061698 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c4013b60 con 0x7f80d40c1170 2022-04-03T19:55:29.784 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.774+0000 7f80d8aad700 1 -- 172.21.15.77:0/3725061698 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d415b450 con 0x7f80d40c1170 2022-04-03T19:55:29.785 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.775+0000 7f80d8aad700 1 -- 172.21.15.77:0/3725061698 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d404b050 con 0x7f80d40c1170 2022-04-03T19:55:29.785 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.776+0000 7f80bb7fe700 1 -- 172.21.15.77:0/3725061698 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f80c4019030 con 0x7f80d40c1170 2022-04-03T19:55:29.785 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.777+0000 7f80bb7fe700 1 --2- 172.21.15.77:0/3725061698 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f80b4063d90 0x7f80b4066240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:29.785 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.777+0000 7f80bb7fe700 1 -- 172.21.15.77:0/3725061698 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f80c4080b00 con 0x7f80d40c1170 2022-04-03T19:55:29.786 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.777+0000 7f80d2ffd700 1 --2- 172.21.15.77:0/3725061698 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f80b4063d90 0x7f80b4066240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:29.786 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.778+0000 7f80d2ffd700 1 --2- 172.21.15.77:0/3725061698 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f80b4063d90 0x7f80b4066240 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f80d4069420 tx=0x7f80c8007cd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:29.787 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:29.781+0000 7f80bb7fe700 1 -- 172.21.15.77:0/3725061698 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f80c404f3b0 con 0x7f80d40c1170 2022-04-03T19:55:30.012 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:30.011+0000 7f80d8aad700 1 -- 172.21.15.77:0/3725061698 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80d40c38f0 con 0x7f80d40c1170 2022-04-03T19:55:30.013 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:30.013+0000 7f80bb7fe700 1 -- 172.21.15.77:0/3725061698 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5353 (secure 0 0 0) 0x7f80c4021090 con 0x7f80d40c1170 2022-04-03T19:55:30.013 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:55:30.013 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:25.260684+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:rejoin","state_seq":26,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:55:30.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:30.015+0000 7f80d8aad700 1 -- 172.21.15.77:0/3725061698 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f80b4063d90 msgr2=0x7f80b4066240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:30.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:30.015+0000 7f80d8aad700 1 --2- 172.21.15.77:0/3725061698 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f80b4063d90 0x7f80b4066240 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f80d4069420 tx=0x7f80c8007cd0 comp rx=0 tx=0).stop 2022-04-03T19:55:30.016 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:30.015+0000 7f80d8aad700 1 -- 172.21.15.77:0/3725061698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80d40c1170 msgr2=0x7f80d4165c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:30.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:30.015+0000 7f80d8aad700 1 --2- 172.21.15.77:0/3725061698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80d40c1170 0x7f80d4165c70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f80c400e930 tx=0x7f80c4009060 comp rx=0 tx=0).stop 2022-04-03T19:55:30.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:30.015+0000 7f80d8aad700 1 -- 172.21.15.77:0/3725061698 shutdown_connections 2022-04-03T19:55:30.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:30.015+0000 7f80d8aad700 1 --2- 172.21.15.77:0/3725061698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80d4155370 0x7f80d40bf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:30.017 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:30.015+0000 7f80d8aad700 1 --2- 172.21.15.77:0/3725061698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80d40c1170 0x7f80d4165c70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:30.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:30.015+0000 7f80d8aad700 1 --2- 172.21.15.77:0/3725061698 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f80b4063d90 0x7f80b4066240 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:30.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:30.015+0000 7f80d8aad700 1 --2- 172.21.15.77:0/3725061698 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80d40c3b10 0x7f80d4151850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:30.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:30.015+0000 7f80d8aad700 1 -- 172.21.15.77:0/3725061698 >> 172.21.15.77:0/3725061698 conn(0x7f80d4066da0 msgr2=0x7f80d4155780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:30.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:30.015+0000 7f80d8aad700 1 -- 172.21.15.77:0/3725061698 shutdown_connections 2022-04-03T19:55:30.019 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:30.015+0000 7f80d8aad700 1 -- 172.21.15.77:0/3725061698 wait complete. 2022-04-03T19:55:30.019 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 40 2022-04-03T19:55:30.040 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:55:32.042 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:55:32.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.248+0000 7f94553f5700 1 Processor -- start 2022-04-03T19:55:32.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.248+0000 7f94553f5700 1 -- start start 2022-04-03T19:55:32.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.249+0000 7f94553f5700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f94500ba3c0 0x7f9450156b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:32.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.249+0000 7f94553f5700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9450157060 0x7f945014cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:32.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.249+0000 7f94553f5700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94500c3c50 con 0x7f94500678b0 2022-04-03T19:55:32.254 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.249+0000 7f94553f5700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94500c8020 con 0x7f94500ba3c0 2022-04-03T19:55:32.254 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.249+0000 7f94553f5700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9450156690 con 0x7f9450157060 2022-04-03T19:55:32.254 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.249+0000 7f944ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f94500ba3c0 0x7f9450156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:32.255 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.249+0000 7f944ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f94500ba3c0 0x7f9450156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43170/0 (socket says 172.21.15.77:43170) 2022-04-03T19:55:32.255 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.249+0000 7f944f7fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9450157060 0x7f945014cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:32.256 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.250+0000 7f944ffff700 1 -- 172.21.15.77:0/3839172916 learned_addr learned my addr 172.21.15.77:0/3839172916 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:32.256 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.250+0000 7f944ffff700 1 -- 172.21.15.77:0/3839172916 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9450157060 msgr2=0x7f945014cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:32.257 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.250+0000 7f944ffff700 1 --2- 172.21.15.77:0/3839172916 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9450157060 0x7f945014cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:32.257 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.250+0000 7f944ffff700 1 -- 172.21.15.77:0/3839172916 >> v1:172.21.15.186:6789/0 conn(0x7f94500678b0 legacy=0x7f945014d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:32.257 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.250+0000 7f944ffff700 1 -- 172.21.15.77:0/3839172916 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94500bf350 con 0x7f94500ba3c0 2022-04-03T19:55:32.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.250+0000 7f944ffff700 1 --2- 172.21.15.77:0/3839172916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94500ba3c0 0x7f9450156b20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9440006fd0 tx=0x7f94400090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbfd21e3df8645ed server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:32.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.251+0000 7f944effd700 1 -- 172.21.15.77:0/3839172916 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9440004030 con 0x7f94500ba3c0 2022-04-03T19:55:32.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.251+0000 7f944effd700 1 -- 172.21.15.77:0/3839172916 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94400041b0 con 0x7f94500ba3c0 2022-04-03T19:55:32.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.251+0000 7f944effd700 1 -- 172.21.15.77:0/3839172916 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9440011630 con 0x7f94500ba3c0 2022-04-03T19:55:32.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.251+0000 7f94553f5700 1 -- 172.21.15.77:0/3839172916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94500ba3c0 msgr2=0x7f9450156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:32.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.251+0000 7f94553f5700 1 --2- 172.21.15.77:0/3839172916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94500ba3c0 0x7f9450156b20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9440006fd0 tx=0x7f94400090f0 comp rx=0 tx=0).stop 2022-04-03T19:55:32.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.251+0000 7f94553f5700 1 -- 172.21.15.77:0/3839172916 shutdown_connections 2022-04-03T19:55:32.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.251+0000 7f94553f5700 1 --2- 172.21.15.77:0/3839172916 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94500ba3c0 0x7f9450156b20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:32.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.251+0000 7f94553f5700 1 --2- 172.21.15.77:0/3839172916 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9450157060 0x7f945014cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:32.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.251+0000 7f94553f5700 1 -- 172.21.15.77:0/3839172916 >> 172.21.15.77:0/3839172916 conn(0x7f9450066da0 msgr2=0x7f94500671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:32.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.251+0000 7f94553f5700 1 -- 172.21.15.77:0/3839172916 shutdown_connections 2022-04-03T19:55:32.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.251+0000 7f94553f5700 1 -- 172.21.15.77:0/3839172916 wait complete. 2022-04-03T19:55:32.267 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.253+0000 7f94553f5700 1 Processor -- start 2022-04-03T19:55:32.268 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.253+0000 7f94553f5700 1 -- start start 2022-04-03T19:55:32.268 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.254+0000 7f94553f5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94500678b0 0x7f9450168610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:32.269 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.254+0000 7f94553f5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94500ba3c0 0x7f9450159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:32.269 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.254+0000 7f94553f5700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9450157060 0x7f945014f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:32.270 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.254+0000 7f94553f5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9450159350 con 0x7f94500678b0 2022-04-03T19:55:32.270 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.254+0000 7f94553f5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945014abd0 con 0x7f94500ba3c0 2022-04-03T19:55:32.270 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.254+0000 7f94553f5700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9450157590 con 0x7f9450157060 2022-04-03T19:55:32.271 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.254+0000 7f944ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94500678b0 0x7f9450168610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:32.271 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.254+0000 7f944f7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94500ba3c0 0x7f9450159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:32.272 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.254+0000 7f944ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94500678b0 0x7f9450168610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43172/0 (socket says 172.21.15.77:43172) 2022-04-03T19:55:32.272 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.254+0000 7f944ffff700 1 -- 172.21.15.77:0/288200795 learned_addr learned my addr 172.21.15.77:0/288200795 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:32.272 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.254+0000 7f9454bf4700 1 --2- 172.21.15.77:0/288200795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9450157060 0x7f945014f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:32.273 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.255+0000 7f944ffff700 1 -- 172.21.15.77:0/288200795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9450157060 msgr2=0x7f945014f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:32.273 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.255+0000 7f944ffff700 1 --2- 172.21.15.77:0/288200795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9450157060 0x7f945014f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:32.274 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.255+0000 7f944ffff700 1 -- 172.21.15.77:0/288200795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94500ba3c0 msgr2=0x7f9450159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:32.274 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.255+0000 7f944ffff700 1 --2- 172.21.15.77:0/288200795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94500ba3c0 0x7f9450159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:32.275 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.255+0000 7f944ffff700 1 -- 172.21.15.77:0/288200795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9440009370 con 0x7f94500678b0 2022-04-03T19:55:32.275 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.255+0000 7f944ffff700 1 --2- 172.21.15.77:0/288200795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94500678b0 0x7f9450168610 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9450067fa0 tx=0x7f9440016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:32.275 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.255+0000 7f9437fff700 1 -- 172.21.15.77:0/288200795 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9440004030 con 0x7f94500678b0 2022-04-03T19:55:32.276 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.255+0000 7f9437fff700 1 -- 172.21.15.77:0/288200795 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94400036f0 con 0x7f94500678b0 2022-04-03T19:55:32.276 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.255+0000 7f9437fff700 1 -- 172.21.15.77:0/288200795 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9440003a70 con 0x7f94500678b0 2022-04-03T19:55:32.277 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.256+0000 7f94553f5700 1 -- 172.21.15.77:0/288200795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9450159020 con 0x7f94500678b0 2022-04-03T19:55:32.277 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.256+0000 7f94553f5700 1 -- 172.21.15.77:0/288200795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945014b0f0 con 0x7f94500678b0 2022-04-03T19:55:32.278 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.257+0000 7f94553f5700 1 -- 172.21.15.77:0/288200795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94500683a0 con 0x7f94500678b0 2022-04-03T19:55:32.278 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.258+0000 7f9437fff700 1 -- 172.21.15.77:0/288200795 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f9440017020 con 0x7f94500678b0 2022-04-03T19:55:32.278 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.258+0000 7f9437fff700 1 --2- 172.21.15.77:0/288200795 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9430063d10 0x7f94300661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:32.279 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.258+0000 7f9437fff700 1 -- 172.21.15.77:0/288200795 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f9440081c50 con 0x7f94500678b0 2022-04-03T19:55:32.279 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.259+0000 7f944f7fe700 1 --2- 172.21.15.77:0/288200795 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9430063d10 0x7f94300661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:32.280 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.259+0000 7f944f7fe700 1 --2- 172.21.15.77:0/288200795 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9430063d10 0x7f94300661c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f94500be350 tx=0x7f9444009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:32.280 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.263+0000 7f9437fff700 1 -- 172.21.15.77:0/288200795 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f9440050590 con 0x7f94500678b0 2022-04-03T19:55:32.498 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.498+0000 7f94553f5700 1 -- 172.21.15.77:0/288200795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f945014cb50 con 0x7f94500678b0 2022-04-03T19:55:32.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.499+0000 7f9437fff700 1 -- 172.21.15.77:0/288200795 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+5359 (secure 0 0 0) 0x7f9440053bb0 con 0x7f94500678b0 2022-04-03T19:55:32.500 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:55:32.500 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:31.288938+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:clientreplay","state_seq":28,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:55:32.502 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.501+0000 7f94553f5700 1 -- 172.21.15.77:0/288200795 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9430063d10 msgr2=0x7f94300661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:32.502 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.501+0000 7f94553f5700 1 --2- 172.21.15.77:0/288200795 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9430063d10 0x7f94300661c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f94500be350 tx=0x7f9444009060 comp rx=0 tx=0).stop 2022-04-03T19:55:32.502 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.501+0000 7f94553f5700 1 -- 172.21.15.77:0/288200795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94500678b0 msgr2=0x7f9450168610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:32.503 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.501+0000 7f94553f5700 1 --2- 172.21.15.77:0/288200795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94500678b0 0x7f9450168610 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9450067fa0 tx=0x7f9440016000 comp rx=0 tx=0).stop 2022-04-03T19:55:32.503 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.501+0000 7f94553f5700 1 -- 172.21.15.77:0/288200795 shutdown_connections 2022-04-03T19:55:32.503 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.501+0000 7f94553f5700 1 --2- 172.21.15.77:0/288200795 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9430063d10 0x7f94300661c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:32.503 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.501+0000 7f94553f5700 1 --2- 172.21.15.77:0/288200795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9450157060 0x7f945014f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:32.503 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.501+0000 7f94553f5700 1 --2- 172.21.15.77:0/288200795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94500678b0 0x7f9450168610 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:32.504 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.501+0000 7f94553f5700 1 --2- 172.21.15.77:0/288200795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94500ba3c0 0x7f9450159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:32.504 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.501+0000 7f94553f5700 1 -- 172.21.15.77:0/288200795 >> 172.21.15.77:0/288200795 conn(0x7f9450066da0 msgr2=0x7f945014bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:32.504 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.501+0000 7f94553f5700 1 -- 172.21.15.77:0/288200795 shutdown_connections 2022-04-03T19:55:32.504 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:32.501+0000 7f94553f5700 1 -- 172.21.15.77:0/288200795 wait complete. 2022-04-03T19:55:32.505 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 41 2022-04-03T19:55:32.526 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:55:34.527 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:55:34.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.735+0000 7f00d9d2a700 1 Processor -- start 2022-04-03T19:55:34.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.736+0000 7f00d9d2a700 1 -- start start 2022-04-03T19:55:34.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.737+0000 7f00d9d2a700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f00d406ccf0 0x7f00d40aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:34.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.737+0000 7f00d9d2a700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f00d40c5760 0x7f00d40af060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:34.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.737+0000 7f00d9d2a700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d4066ed0 con 0x7f00d4158810 2022-04-03T19:55:34.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.737+0000 7f00d9d2a700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d406d930 con 0x7f00d40c5760 2022-04-03T19:55:34.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.737+0000 7f00d9d2a700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d406e380 con 0x7f00d406ccf0 2022-04-03T19:55:34.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.737+0000 7f00d3fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f00d40c5760 0x7f00d40af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:34.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.737+0000 7f00d3fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f00d40c5760 0x7f00d40af060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43174/0 (socket says 172.21.15.77:43174) 2022-04-03T19:55:34.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.737+0000 7f00d3fff700 1 -- 172.21.15.77:0/979979395 learned_addr learned my addr 172.21.15.77:0/979979395 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:34.739 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.737+0000 7f00d8d28700 1 --2- 172.21.15.77:0/979979395 >> v2:172.21.15.186:3300/0 conn(0x7f00d406ccf0 0x7f00d40aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:34.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.737+0000 7f00d3fff700 1 -- 172.21.15.77:0/979979395 >> v2:172.21.15.186:3300/0 conn(0x7f00d406ccf0 msgr2=0x7f00d40aeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:34.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.737+0000 7f00d3fff700 1 --2- 172.21.15.77:0/979979395 >> v2:172.21.15.186:3300/0 conn(0x7f00d406ccf0 0x7f00d40aeb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:34.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.737+0000 7f00d3fff700 1 -- 172.21.15.77:0/979979395 >> v1:172.21.15.186:6789/0 conn(0x7f00d4158810 legacy=0x7f00d4067360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:34.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.737+0000 7f00d3fff700 1 -- 172.21.15.77:0/979979395 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d40c6790 con 0x7f00d40c5760 2022-04-03T19:55:34.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.738+0000 7f00d3fff700 1 --2- 172.21.15.77:0/979979395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00d40c5760 0x7f00d40af060 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f00d40ca4b0 tx=0x7f00c00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb73591ed5f80e6 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:34.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.738+0000 7f00d37fe700 1 -- 172.21.15.77:0/979979395 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c0004030 con 0x7f00d40c5760 2022-04-03T19:55:34.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.738+0000 7f00d37fe700 1 -- 172.21.15.77:0/979979395 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00c0004190 con 0x7f00d40c5760 2022-04-03T19:55:34.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.738+0000 7f00d37fe700 1 -- 172.21.15.77:0/979979395 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c0011630 con 0x7f00d40c5760 2022-04-03T19:55:34.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.739+0000 7f00d9d2a700 1 -- 172.21.15.77:0/979979395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00d40c5760 msgr2=0x7f00d40af060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:34.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.739+0000 7f00d9d2a700 1 --2- 172.21.15.77:0/979979395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00d40c5760 0x7f00d40af060 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f00d40ca4b0 tx=0x7f00c00090f0 comp rx=0 tx=0).stop 2022-04-03T19:55:34.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.739+0000 7f00d9d2a700 1 -- 172.21.15.77:0/979979395 shutdown_connections 2022-04-03T19:55:34.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.739+0000 7f00d9d2a700 1 --2- 172.21.15.77:0/979979395 >> v2:172.21.15.186:3300/0 conn(0x7f00d406ccf0 0x7f00d40aeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:34.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.739+0000 7f00d9d2a700 1 --2- 172.21.15.77:0/979979395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00d40c5760 0x7f00d40af060 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:34.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.739+0000 7f00d9d2a700 1 -- 172.21.15.77:0/979979395 >> 172.21.15.77:0/979979395 conn(0x7f00d4069990 msgr2=0x7f00d40626d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:34.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.739+0000 7f00d9d2a700 1 -- 172.21.15.77:0/979979395 shutdown_connections 2022-04-03T19:55:34.747 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.739+0000 7f00d9d2a700 1 -- 172.21.15.77:0/979979395 wait complete. 2022-04-03T19:55:34.747 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.740+0000 7f00d9d2a700 1 Processor -- start 2022-04-03T19:55:34.747 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.740+0000 7f00d9d2a700 1 -- start start 2022-04-03T19:55:34.747 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.741+0000 7f00d9d2a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00d406ccf0 0x7f00d40a9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:34.748 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.741+0000 7f00d9d2a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00d40c5760 0x7f00d40a9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:34.748 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.741+0000 7f00d9d2a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00d4158810 0x7f00d40bf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:34.748 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.741+0000 7f00d9d2a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d4164df0 con 0x7f00d4158810 2022-04-03T19:55:34.748 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.741+0000 7f00d9d2a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d4165bd0 con 0x7f00d406ccf0 2022-04-03T19:55:34.749 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.741+0000 7f00d9d2a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d4164fe0 con 0x7f00d40c5760 2022-04-03T19:55:34.749 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.742+0000 7f00d8d28700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00d406ccf0 0x7f00d40a9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:34.749 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.742+0000 7f00d9529700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00d4158810 0x7f00d40bf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:34.749 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.742+0000 7f00d8d28700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00d406ccf0 0x7f00d40a9a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38370/0 (socket says 172.21.15.77:38370) 2022-04-03T19:55:34.750 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.742+0000 7f00d8d28700 1 -- 172.21.15.77:0/1433072942 learned_addr learned my addr 172.21.15.77:0/1433072942 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:34.750 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.742+0000 7f00d3fff700 1 --2- 172.21.15.77:0/1433072942 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00d40c5760 0x7f00d40a9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:34.750 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.742+0000 7f00d8d28700 1 -- 172.21.15.77:0/1433072942 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00d40c5760 msgr2=0x7f00d40a9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:34.750 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.742+0000 7f00d8d28700 1 --2- 172.21.15.77:0/1433072942 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00d40c5760 0x7f00d40a9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:34.750 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.742+0000 7f00d8d28700 1 -- 172.21.15.77:0/1433072942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00d4158810 msgr2=0x7f00d40bf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:34.751 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.742+0000 7f00d8d28700 1 --2- 172.21.15.77:0/1433072942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00d4158810 0x7f00d40bf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:34.751 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.742+0000 7f00d8d28700 1 -- 172.21.15.77:0/1433072942 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c0009370 con 0x7f00d406ccf0 2022-04-03T19:55:34.751 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.742+0000 7f00d8d28700 1 --2- 172.21.15.77:0/1433072942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00d406ccf0 0x7f00d40a9a50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f00c800e930 tx=0x7f00c8009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:34.754 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.743+0000 7f00d0ff9700 1 -- 172.21.15.77:0/1433072942 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c8013680 con 0x7f00d406ccf0 2022-04-03T19:55:34.755 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.743+0000 7f00d0ff9700 1 -- 172.21.15.77:0/1433072942 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00c80137e0 con 0x7f00d406ccf0 2022-04-03T19:55:34.755 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.743+0000 7f00d0ff9700 1 -- 172.21.15.77:0/1433072942 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c8013b60 con 0x7f00d406ccf0 2022-04-03T19:55:34.756 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.743+0000 7f00d9d2a700 1 -- 172.21.15.77:0/1433072942 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d40bcc90 con 0x7f00d406ccf0 2022-04-03T19:55:34.756 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.743+0000 7f00d9d2a700 1 -- 172.21.15.77:0/1433072942 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d4165580 con 0x7f00d406ccf0 2022-04-03T19:55:34.756 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.744+0000 7f00d9d2a700 1 -- 172.21.15.77:0/1433072942 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d4069f50 con 0x7f00d406ccf0 2022-04-03T19:55:34.756 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.745+0000 7f00d0ff9700 1 -- 172.21.15.77:0/1433072942 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f00c8019030 con 0x7f00d406ccf0 2022-04-03T19:55:34.756 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.745+0000 7f00d0ff9700 1 --2- 172.21.15.77:0/1433072942 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f00bc063ce0 0x7f00bc066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:34.757 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.746+0000 7f00d0ff9700 1 -- 172.21.15.77:0/1433072942 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f00c8080a20 con 0x7f00d406ccf0 2022-04-03T19:55:34.757 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.746+0000 7f00d3fff700 1 --2- 172.21.15.77:0/1433072942 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f00bc063ce0 0x7f00bc066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:34.757 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.746+0000 7f00d3fff700 1 --2- 172.21.15.77:0/1433072942 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f00bc063ce0 0x7f00bc066190 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f00d40ca4b0 tx=0x7f00c0016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:34.757 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.750+0000 7f00d0ff9700 1 -- 172.21.15.77:0/1433072942 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f00c804f3c0 con 0x7f00d406ccf0 2022-04-03T19:55:34.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.987+0000 7f00d9d2a700 1 -- 172.21.15.77:0/1433072942 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00d404b050 con 0x7f00d406ccf0 2022-04-03T19:55:34.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.988+0000 7f00d0ff9700 1 -- 172.21.15.77:0/1433072942 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+5359 (secure 0 0 0) 0x7f00c8017090 con 0x7f00d406ccf0 2022-04-03T19:55:34.989 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:55:34.989 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:31.288938+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:clientreplay","state_seq":28,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:55:34.992 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.991+0000 7f00d9d2a700 1 -- 172.21.15.77:0/1433072942 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f00bc063ce0 msgr2=0x7f00bc066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:34.992 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.991+0000 7f00d9d2a700 1 --2- 172.21.15.77:0/1433072942 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f00bc063ce0 0x7f00bc066190 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f00d40ca4b0 tx=0x7f00c0016000 comp rx=0 tx=0).stop 2022-04-03T19:55:34.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.991+0000 7f00d9d2a700 1 -- 172.21.15.77:0/1433072942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00d406ccf0 msgr2=0x7f00d40a9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:34.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.991+0000 7f00d9d2a700 1 --2- 172.21.15.77:0/1433072942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00d406ccf0 0x7f00d40a9a50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f00c800e930 tx=0x7f00c8009060 comp rx=0 tx=0).stop 2022-04-03T19:55:34.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.991+0000 7f00d9d2a700 1 -- 172.21.15.77:0/1433072942 shutdown_connections 2022-04-03T19:55:34.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.991+0000 7f00d9d2a700 1 --2- 172.21.15.77:0/1433072942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00d4158810 0x7f00d40bf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:34.993 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.991+0000 7f00d9d2a700 1 --2- 172.21.15.77:0/1433072942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00d406ccf0 0x7f00d40a9a50 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:34.994 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.991+0000 7f00d9d2a700 1 --2- 172.21.15.77:0/1433072942 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f00bc063ce0 0x7f00bc066190 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:34.994 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.991+0000 7f00d9d2a700 1 --2- 172.21.15.77:0/1433072942 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00d40c5760 0x7f00d40a9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:34.994 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.991+0000 7f00d9d2a700 1 -- 172.21.15.77:0/1433072942 >> 172.21.15.77:0/1433072942 conn(0x7f00d4069990 msgr2=0x7f00d40c7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:34.994 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.991+0000 7f00d9d2a700 1 -- 172.21.15.77:0/1433072942 shutdown_connections 2022-04-03T19:55:34.995 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:34.991+0000 7f00d9d2a700 1 -- 172.21.15.77:0/1433072942 wait complete. 2022-04-03T19:55:34.995 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 41 2022-04-03T19:55:35.017 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 41, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-04-03T19:44:21.275716+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 41, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5463": { "addr": "172.21.15.77:6837/2777516376", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6836", "nonce": 2777516376, "type": "v2" }, { "addr": "172.21.15.77:6837", "nonce": 2777516376, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 5463, "incarnation": 38, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:clientreplay", "state_seq": 28 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-04-03T19:55:31.288938+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5463 } } } ], "standbys": [ { "addr": "172.21.15.77:6834/2116223509", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6832", "nonce": 2116223509, "type": "v2" }, { "addr": "172.21.15.77:6834", "nonce": 2116223509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4245, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6836/2374276747", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2374276747, "type": "v2" }, { "addr": "172.21.15.186:6836", "nonce": 2374276747, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4259, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/610990853", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6838", "nonce": 610990853, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 610990853, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4265, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.77:6835/170053412", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6833", "nonce": 170053412, "type": "v2" }, { "addr": "172.21.15.77:6835", "nonce": 170053412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4271, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-04-03T19:55:37.019 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:55:37.271 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.270+0000 7f959a385700 1 Processor -- start 2022-04-03T19:55:37.271 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.270+0000 7f959a385700 1 -- start start 2022-04-03T19:55:37.276 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.271+0000 7f959a385700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f959406d3f0 0x7f959406d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:37.276 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.271+0000 7f959a385700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95940c5930 0x7f9594155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:37.276 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.271+0000 7f959a385700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9594156120 0x7f959406c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:37.277 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.271+0000 7f959a385700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95940c2f80 con 0x7f959406d3f0 2022-04-03T19:55:37.277 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.271+0000 7f959a385700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95940c3260 con 0x7f9594156120 2022-04-03T19:55:37.277 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.271+0000 7f959a385700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95940c3540 con 0x7f95940c5930 2022-04-03T19:55:37.278 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.272+0000 7f9598b82700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95940c5930 0x7f9594155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:37.278 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.272+0000 7f9599383700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f959406d3f0 0x7f959406d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:37.279 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.272+0000 7f9599b84700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9594156120 0x7f959406c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:37.279 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.272+0000 7f9598b82700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95940c5930 0x7f9594155be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60464/0 (socket says 172.21.15.77:60464) 2022-04-03T19:55:37.279 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.272+0000 7f9598b82700 1 -- 172.21.15.77:0/483558140 learned_addr learned my addr 172.21.15.77:0/483558140 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:37.280 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.272+0000 7f9598b82700 1 -- 172.21.15.77:0/483558140 >> v2:172.21.15.186:3300/0 conn(0x7f9594156120 msgr2=0x7f959406c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:37.280 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.272+0000 7f9598b82700 1 --2- 172.21.15.77:0/483558140 >> v2:172.21.15.186:3300/0 conn(0x7f9594156120 0x7f959406c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:37.281 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.273+0000 7f9598b82700 1 -- 172.21.15.77:0/483558140 >> v2:172.21.15.77:3300/0 conn(0x7f959406d3f0 msgr2=0x7f959406d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:37.281 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.273+0000 7f9598b82700 1 --2- 172.21.15.77:0/483558140 >> v2:172.21.15.77:3300/0 conn(0x7f959406d3f0 0x7f959406d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:37.281 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.273+0000 7f9598b82700 1 -- 172.21.15.77:0/483558140 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95940c3820 con 0x7f95940c5930 2022-04-03T19:55:37.282 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.273+0000 7f9599383700 1 --2- 172.21.15.77:0/483558140 >> v2:172.21.15.77:3300/0 conn(0x7f959406d3f0 0x7f959406d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:55:37.282 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.273+0000 7f9598b82700 1 --2- 172.21.15.77:0/483558140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95940c5930 0x7f9594155be0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9584006fd0 tx=0x7f95840090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9cc1febe95a6e1d server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:37.282 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.274+0000 7f9583fff700 1 -- 172.21.15.77:0/483558140 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9584004030 con 0x7f95940c5930 2022-04-03T19:55:37.282 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.274+0000 7f9583fff700 1 -- 172.21.15.77:0/483558140 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9584004190 con 0x7f95940c5930 2022-04-03T19:55:37.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.274+0000 7f9583fff700 1 -- 172.21.15.77:0/483558140 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9584011630 con 0x7f95940c5930 2022-04-03T19:55:37.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.274+0000 7f959a385700 1 -- 172.21.15.77:0/483558140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95940c5930 msgr2=0x7f9594155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:37.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.274+0000 7f959a385700 1 --2- 172.21.15.77:0/483558140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95940c5930 0x7f9594155be0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9584006fd0 tx=0x7f95840090f0 comp rx=0 tx=0).stop 2022-04-03T19:55:37.283 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.275+0000 7f959a385700 1 -- 172.21.15.77:0/483558140 shutdown_connections 2022-04-03T19:55:37.284 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.275+0000 7f959a385700 1 --2- 172.21.15.77:0/483558140 >> v2:172.21.15.186:3300/0 conn(0x7f9594156120 0x7f959406c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:37.284 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.275+0000 7f959a385700 1 --2- 172.21.15.77:0/483558140 >> v2:172.21.15.77:3300/0 conn(0x7f959406d3f0 0x7f959406d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:37.284 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.275+0000 7f959a385700 1 --2- 172.21.15.77:0/483558140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95940c5930 0x7f9594155be0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:37.284 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.275+0000 7f959a385700 1 -- 172.21.15.77:0/483558140 >> 172.21.15.77:0/483558140 conn(0x7f95940cb720 msgr2=0x7f95940cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:37.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.275+0000 7f959a385700 1 -- 172.21.15.77:0/483558140 shutdown_connections 2022-04-03T19:55:37.285 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.275+0000 7f959a385700 1 -- 172.21.15.77:0/483558140 wait complete. 2022-04-03T19:55:37.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.277+0000 7f959a385700 1 Processor -- start 2022-04-03T19:55:37.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.277+0000 7f959a385700 1 -- start start 2022-04-03T19:55:37.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.278+0000 7f959a385700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f959406d3f0 0x7f95941665b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:37.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.278+0000 7f959a385700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95940c5930 0x7f95940c0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:37.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.278+0000 7f959a385700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9594156120 0x7f95940c10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:37.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.278+0000 7f959a385700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9594157030 con 0x7f95940c5930 2022-04-03T19:55:37.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.278+0000 7f959a385700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9594156bf0 con 0x7f9594156120 2022-04-03T19:55:37.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.278+0000 7f959a385700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95940c06b0 con 0x7f959406d3f0 2022-04-03T19:55:37.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.278+0000 7f9599383700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f959406d3f0 0x7f95941665b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:37.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.278+0000 7f9599383700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f959406d3f0 0x7f95941665b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60468/0 (socket says 172.21.15.77:60468) 2022-04-03T19:55:37.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.278+0000 7f9599383700 1 -- 172.21.15.77:0/2999399810 learned_addr learned my addr 172.21.15.77:0/2999399810 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:37.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.278+0000 7f9598b82700 1 --2- 172.21.15.77:0/2999399810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95940c5930 0x7f95940c0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:37.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.279+0000 7f9599b84700 1 --2- 172.21.15.77:0/2999399810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9594156120 0x7f95940c10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:37.291 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.279+0000 7f9599383700 1 -- 172.21.15.77:0/2999399810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9594156120 msgr2=0x7f95940c10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:37.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.279+0000 7f9599383700 1 --2- 172.21.15.77:0/2999399810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9594156120 0x7f95940c10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:37.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.279+0000 7f9599383700 1 -- 172.21.15.77:0/2999399810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95940c5930 msgr2=0x7f95940c0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:37.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.279+0000 7f9599383700 1 --2- 172.21.15.77:0/2999399810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95940c5930 0x7f95940c0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:37.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.279+0000 7f9599383700 1 -- 172.21.15.77:0/2999399810 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95900092a0 con 0x7f959406d3f0 2022-04-03T19:55:37.292 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.279+0000 7f9598b82700 1 --2- 172.21.15.77:0/2999399810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95940c5930 0x7f95940c0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:55:37.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.280+0000 7f9599383700 1 --2- 172.21.15.77:0/2999399810 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f959406d3f0 0x7f95941665b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f959000ea50 tx=0x7f9590009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:37.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.280+0000 7f95817fa700 1 -- 172.21.15.77:0/2999399810 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959000c850 con 0x7f959406d3f0 2022-04-03T19:55:37.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.280+0000 7f95817fa700 1 -- 172.21.15.77:0/2999399810 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959000c9b0 con 0x7f959406d3f0 2022-04-03T19:55:37.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.280+0000 7f95817fa700 1 -- 172.21.15.77:0/2999399810 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959000cd30 con 0x7f959406d3f0 2022-04-03T19:55:37.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.280+0000 7f959a385700 1 -- 172.21.15.77:0/2999399810 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9584009370 con 0x7f959406d3f0 2022-04-03T19:55:37.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.281+0000 7f959a385700 1 -- 172.21.15.77:0/2999399810 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95940c1fe0 con 0x7f959406d3f0 2022-04-03T19:55:37.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.282+0000 7f959a385700 1 -- 172.21.15.77:0/2999399810 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95940c0870 con 0x7f959406d3f0 2022-04-03T19:55:37.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.283+0000 7f95817fa700 1 -- 172.21.15.77:0/2999399810 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f9590023070 con 0x7f959406d3f0 2022-04-03T19:55:37.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.284+0000 7f95817fa700 1 --2- 172.21.15.77:0/2999399810 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9574063ce0 0x7f9574066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:37.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.284+0000 7f95817fa700 1 -- 172.21.15.77:0/2999399810 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f9590080fe0 con 0x7f959406d3f0 2022-04-03T19:55:37.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.285+0000 7f9598b82700 1 --2- 172.21.15.77:0/2999399810 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9574063ce0 0x7f9574066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:37.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.286+0000 7f9598b82700 1 --2- 172.21.15.77:0/2999399810 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9574063ce0 0x7f9574066190 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f959414fa50 tx=0x7f9584016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:37.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.292+0000 7f95817fa700 1 -- 172.21.15.77:0/2999399810 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f959004f870 con 0x7f959406d3f0 2022-04-03T19:55:37.571 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.571+0000 7f959a385700 1 -- 172.21.15.77:0/2999399810 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f959404b050 con 0x7f959406d3f0 2022-04-03T19:55:37.572 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.571+0000 7f95817fa700 1 -- 172.21.15.77:0/2999399810 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+5359 (secure 0 0 0) 0x7f9590052e90 con 0x7f959406d3f0 2022-04-03T19:55:37.573 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:55:37.573 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:31.288938+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:clientreplay","state_seq":28,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:55:37.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.575+0000 7f959a385700 1 -- 172.21.15.77:0/2999399810 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9574063ce0 msgr2=0x7f9574066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:37.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.575+0000 7f959a385700 1 --2- 172.21.15.77:0/2999399810 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9574063ce0 0x7f9574066190 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f959414fa50 tx=0x7f9584016000 comp rx=0 tx=0).stop 2022-04-03T19:55:37.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.575+0000 7f959a385700 1 -- 172.21.15.77:0/2999399810 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f959406d3f0 msgr2=0x7f95941665b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:37.577 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.575+0000 7f959a385700 1 --2- 172.21.15.77:0/2999399810 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f959406d3f0 0x7f95941665b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f959000ea50 tx=0x7f9590009060 comp rx=0 tx=0).stop 2022-04-03T19:55:37.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.575+0000 7f959a385700 1 -- 172.21.15.77:0/2999399810 shutdown_connections 2022-04-03T19:55:37.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.575+0000 7f959a385700 1 --2- 172.21.15.77:0/2999399810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9594156120 0x7f95940c10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:37.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.575+0000 7f959a385700 1 --2- 172.21.15.77:0/2999399810 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f9574063ce0 0x7f9574066190 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:37.578 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.575+0000 7f959a385700 1 --2- 172.21.15.77:0/2999399810 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f959406d3f0 0x7f95941665b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:37.579 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.575+0000 7f959a385700 1 --2- 172.21.15.77:0/2999399810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95940c5930 0x7f95940c0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:37.579 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.575+0000 7f959a385700 1 -- 172.21.15.77:0/2999399810 >> 172.21.15.77:0/2999399810 conn(0x7f95940cb720 msgr2=0x7f95940c8140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:37.579 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.575+0000 7f959a385700 1 -- 172.21.15.77:0/2999399810 shutdown_connections 2022-04-03T19:55:37.579 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:37.575+0000 7f959a385700 1 -- 172.21.15.77:0/2999399810 wait complete. 2022-04-03T19:55:37.580 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 41 2022-04-03T19:55:37.615 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:55:39.616 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:55:39.824 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.824+0000 7f1a5a3cf700 1 Processor -- start 2022-04-03T19:55:39.825 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.824+0000 7f1a5a3cf700 1 -- start start 2022-04-03T19:55:39.827 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.825+0000 7f1a5a3cf700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a540ba3c0 0x7f1a54156b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:39.827 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.825+0000 7f1a5a3cf700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a540c3c50 con 0x7f1a540678b0 2022-04-03T19:55:39.827 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.825+0000 7f1a5a3cf700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a54156690 con 0x7f1a54157060 2022-04-03T19:55:39.827 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.825+0000 7f1a5a3cf700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a540c8020 con 0x7f1a540ba3c0 2022-04-03T19:55:39.828 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.826+0000 7f1a593cd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a540ba3c0 0x7f1a54156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:39.828 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.826+0000 7f1a59bce700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f1a540678b0 0x7f1a5415d800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38558/0 (socket says 172.21.15.77:38558) 2022-04-03T19:55:39.828 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.826+0000 7f1a59bce700 1 -- 172.21.15.77:0/3874355321 learned_addr learned my addr 172.21.15.77:0/3874355321 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:55:39.828 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.826+0000 7f1a58bcc700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f1a54157060 0x7f1a5414cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44362/0 (socket says 172.21.15.77:44362) 2022-04-03T19:55:39.829 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.826+0000 7f1a593cd700 1 -- 172.21.15.77:0/3874355321 >> v1:172.21.15.186:6789/0 conn(0x7f1a54157060 legacy=0x7f1a5414cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:39.829 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.826+0000 7f1a593cd700 1 -- 172.21.15.77:0/3874355321 >> v1:172.21.15.77:6789/0 conn(0x7f1a540678b0 legacy=0x7f1a5415d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:39.829 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.826+0000 7f1a43fff700 1 -- 172.21.15.77:0/3874355321 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457901080 0 0) 0x7f1a54156690 con 0x7f1a54157060 2022-04-03T19:55:39.829 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.826+0000 7f1a593cd700 1 -- 172.21.15.77:0/3874355321 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a540bf350 con 0x7f1a540ba3c0 2022-04-03T19:55:39.830 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.826+0000 7f1a593cd700 1 --2- 172.21.15.77:0/3874355321 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a540ba3c0 0x7f1a54156b20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1a50009790 tx=0x7f1a50009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab86237cb8f1b49e server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:39.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.827+0000 7f1a43fff700 1 -- 172.21.15.77:0/3874355321 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50011630 con 0x7f1a540ba3c0 2022-04-03T19:55:39.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.827+0000 7f1a43fff700 1 -- 172.21.15.77:0/3874355321 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a50011790 con 0x7f1a540ba3c0 2022-04-03T19:55:39.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.827+0000 7f1a43fff700 1 -- 172.21.15.77:0/3874355321 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50011b30 con 0x7f1a540ba3c0 2022-04-03T19:55:39.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.827+0000 7f1a5a3cf700 1 -- 172.21.15.77:0/3874355321 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a540ba3c0 msgr2=0x7f1a54156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:39.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.827+0000 7f1a5a3cf700 1 --2- 172.21.15.77:0/3874355321 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a540ba3c0 0x7f1a54156b20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1a50009790 tx=0x7f1a50009060 comp rx=0 tx=0).stop 2022-04-03T19:55:39.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.827+0000 7f1a5a3cf700 1 -- 172.21.15.77:0/3874355321 shutdown_connections 2022-04-03T19:55:39.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.827+0000 7f1a5a3cf700 1 --2- 172.21.15.77:0/3874355321 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a540ba3c0 0x7f1a54156b20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:39.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.827+0000 7f1a5a3cf700 1 -- 172.21.15.77:0/3874355321 >> 172.21.15.77:0/3874355321 conn(0x7f1a54066da0 msgr2=0x7f1a540671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:39.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.827+0000 7f1a5a3cf700 1 -- 172.21.15.77:0/3874355321 shutdown_connections 2022-04-03T19:55:39.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.827+0000 7f1a5a3cf700 1 -- 172.21.15.77:0/3874355321 wait complete. 2022-04-03T19:55:39.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.829+0000 7f1a5a3cf700 1 Processor -- start 2022-04-03T19:55:39.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.829+0000 7f1a5a3cf700 1 -- start start 2022-04-03T19:55:39.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.830+0000 7f1a5a3cf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a540678b0 0x7f1a540be770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:39.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.830+0000 7f1a5a3cf700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a540ba3c0 0x7f1a54168780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:39.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.830+0000 7f1a5a3cf700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a54157060 0x7f1a54168cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:39.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.830+0000 7f1a5a3cf700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a54157520 con 0x7f1a540678b0 2022-04-03T19:55:39.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.830+0000 7f1a5a3cf700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a540c5fc0 con 0x7f1a54157060 2022-04-03T19:55:39.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.830+0000 7f1a5a3cf700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a54158860 con 0x7f1a540ba3c0 2022-04-03T19:55:39.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.830+0000 7f1a593cd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a540678b0 0x7f1a540be770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:39.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.830+0000 7f1a593cd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a540678b0 0x7f1a540be770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43184/0 (socket says 172.21.15.77:43184) 2022-04-03T19:55:39.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.830+0000 7f1a593cd700 1 -- 172.21.15.77:0/2814465968 learned_addr learned my addr 172.21.15.77:0/2814465968 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:39.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.830+0000 7f1a58bcc700 1 --2- 172.21.15.77:0/2814465968 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a540ba3c0 0x7f1a54168780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:39.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.830+0000 7f1a59bce700 1 --2- 172.21.15.77:0/2814465968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a54157060 0x7f1a54168cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:39.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.831+0000 7f1a58bcc700 1 -- 172.21.15.77:0/2814465968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a54157060 msgr2=0x7f1a54168cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:39.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.831+0000 7f1a58bcc700 1 --2- 172.21.15.77:0/2814465968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a54157060 0x7f1a54168cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:39.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.831+0000 7f1a58bcc700 1 -- 172.21.15.77:0/2814465968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a540678b0 msgr2=0x7f1a540be770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:39.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.831+0000 7f1a58bcc700 1 --2- 172.21.15.77:0/2814465968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a540678b0 0x7f1a540be770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:39.839 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.831+0000 7f1a58bcc700 1 -- 172.21.15.77:0/2814465968 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a500092a0 con 0x7f1a540ba3c0 2022-04-03T19:55:39.840 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.831+0000 7f1a58bcc700 1 --2- 172.21.15.77:0/2814465968 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a540ba3c0 0x7f1a54168780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1a44000c00 tx=0x7f1a44007d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:39.840 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.831+0000 7f1a417fa700 1 -- 172.21.15.77:0/2814465968 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a44007150 con 0x7f1a540ba3c0 2022-04-03T19:55:39.840 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.831+0000 7f1a417fa700 1 -- 172.21.15.77:0/2814465968 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a440072d0 con 0x7f1a540ba3c0 2022-04-03T19:55:39.840 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.831+0000 7f1a5a3cf700 1 -- 172.21.15.77:0/2814465968 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a54157750 con 0x7f1a540ba3c0 2022-04-03T19:55:39.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.831+0000 7f1a417fa700 1 -- 172.21.15.77:0/2814465968 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a440046b0 con 0x7f1a540ba3c0 2022-04-03T19:55:39.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.831+0000 7f1a5a3cf700 1 -- 172.21.15.77:0/2814465968 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a540c5690 con 0x7f1a540ba3c0 2022-04-03T19:55:39.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.832+0000 7f1a5a3cf700 1 -- 172.21.15.77:0/2814465968 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a540c74a0 con 0x7f1a540ba3c0 2022-04-03T19:55:39.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.833+0000 7f1a417fa700 1 -- 172.21.15.77:0/2814465968 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f1a44010030 con 0x7f1a540ba3c0 2022-04-03T19:55:39.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.834+0000 7f1a417fa700 1 --2- 172.21.15.77:0/2814465968 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a34063d10 0x7f1a340661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:39.843 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.834+0000 7f1a417fa700 1 -- 172.21.15.77:0/2814465968 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f1a44080330 con 0x7f1a540ba3c0 2022-04-03T19:55:39.843 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.835+0000 7f1a593cd700 1 --2- 172.21.15.77:0/2814465968 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a34063d10 0x7f1a340661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:39.843 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.835+0000 7f1a593cd700 1 --2- 172.21.15.77:0/2814465968 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a34063d10 0x7f1a340661c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1a50004660 tx=0x7f1a50014000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:39.843 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:39.839+0000 7f1a417fa700 1 -- 172.21.15.77:0/2814465968 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f1a4404ecf0 con 0x7f1a540ba3c0 2022-04-03T19:55:40.072 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:40.071+0000 7f1a5a3cf700 1 -- 172.21.15.77:0/2814465968 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a54156e40 con 0x7f1a540ba3c0 2022-04-03T19:55:40.073 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:40.073+0000 7f1a417fa700 1 -- 172.21.15.77:0/2814465968 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5353 (secure 0 0 0) 0x7f1a44052310 con 0x7f1a540ba3c0 2022-04-03T19:55:40.074 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:55:40.074 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:38.315236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:active","state_seq":30,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:55:40.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:40.075+0000 7f1a5a3cf700 1 -- 172.21.15.77:0/2814465968 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a34063d10 msgr2=0x7f1a340661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:40.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:40.075+0000 7f1a5a3cf700 1 --2- 172.21.15.77:0/2814465968 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a34063d10 0x7f1a340661c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1a50004660 tx=0x7f1a50014000 comp rx=0 tx=0).stop 2022-04-03T19:55:40.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:40.075+0000 7f1a5a3cf700 1 -- 172.21.15.77:0/2814465968 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a540ba3c0 msgr2=0x7f1a54168780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:40.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:40.075+0000 7f1a5a3cf700 1 --2- 172.21.15.77:0/2814465968 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a540ba3c0 0x7f1a54168780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1a44000c00 tx=0x7f1a44007d10 comp rx=0 tx=0).stop 2022-04-03T19:55:40.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:40.075+0000 7f1a5a3cf700 1 -- 172.21.15.77:0/2814465968 shutdown_connections 2022-04-03T19:55:40.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:40.075+0000 7f1a5a3cf700 1 --2- 172.21.15.77:0/2814465968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a54157060 0x7f1a54168cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:40.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:40.075+0000 7f1a5a3cf700 1 --2- 172.21.15.77:0/2814465968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a540678b0 0x7f1a540be770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:40.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:40.075+0000 7f1a5a3cf700 1 --2- 172.21.15.77:0/2814465968 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1a34063d10 0x7f1a340661c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:40.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:40.075+0000 7f1a5a3cf700 1 --2- 172.21.15.77:0/2814465968 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a540ba3c0 0x7f1a54168780 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:40.081 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:40.075+0000 7f1a5a3cf700 1 -- 172.21.15.77:0/2814465968 >> 172.21.15.77:0/2814465968 conn(0x7f1a54066da0 msgr2=0x7f1a540c11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:40.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:40.075+0000 7f1a5a3cf700 1 -- 172.21.15.77:0/2814465968 shutdown_connections 2022-04-03T19:55:40.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:40.075+0000 7f1a5a3cf700 1 -- 172.21.15.77:0/2814465968 wait complete. 2022-04-03T19:55:40.084 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 42 2022-04-03T19:55:40.100 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=5039 2022-04-03T19:55:40.101 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.d 2022-04-03T19:55:43.102 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2022-04-03T19:55:43.102 INFO:tasks.ceph.mds.d:Restarting daemon 2022-04-03T19:55:43.103 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2022-04-03T19:55:43.105 INFO:tasks.ceph.mds.d:Started 2022-04-03T19:55:43.106 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:55:43.186 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2022-04-03T19:55:43.311 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.310+0000 7f51c359e700 1 Processor -- start 2022-04-03T19:55:43.311 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.310+0000 7f51c359e700 1 -- start start 2022-04-03T19:55:43.312 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.312+0000 7f51c359e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51c406c410 0x7f51c406c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:43.313 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.312+0000 7f51c359e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c406bd20 con 0x7f51c4069a00 2022-04-03T19:55:43.313 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.312+0000 7f51c359e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c40c2bf0 con 0x7f51c4069e10 2022-04-03T19:55:43.313 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.312+0000 7f51c359e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c406d600 con 0x7f51c406c410 2022-04-03T19:55:43.313 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.312+0000 7f51c259c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51c406c410 0x7f51c406c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:43.313 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.312+0000 7f51c1d9b700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f51c4069a00 0x7f51c4155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38560/0 (socket says 172.21.15.77:38560) 2022-04-03T19:55:43.314 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.312+0000 7f51c1d9b700 1 -- 172.21.15.77:0/2674333235 learned_addr learned my addr 172.21.15.77:0/2674333235 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:55:43.315 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.312+0000 7f51c159a700 1 -- 172.21.15.77:0/2674333235 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 23177982 0 0) 0x7f51bc001490 con 0x7f51c4069e10 2022-04-03T19:55:43.315 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.313+0000 7f51c259c700 1 -- 172.21.15.77:0/2674333235 >> v1:172.21.15.186:6789/0 conn(0x7f51c4069e10 legacy=0x7f51c415fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:43.316 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.313+0000 7f51c259c700 1 -- 172.21.15.77:0/2674333235 >> v1:172.21.15.77:6789/0 conn(0x7f51c4069a00 legacy=0x7f51c4155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:43.316 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.313+0000 7f51c259c700 1 -- 172.21.15.77:0/2674333235 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c40c2760 con 0x7f51c406c410 2022-04-03T19:55:43.316 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.313+0000 7f51c259c700 1 --2- 172.21.15.77:0/2674333235 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51c406c410 0x7f51c406c7e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f51b4009790 tx=0x7f51b4009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dccb8281769f70e7 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:43.316 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.313+0000 7f51c159a700 1 -- 172.21.15.77:0/2674333235 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b4004030 con 0x7f51c406c410 2022-04-03T19:55:43.317 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.313+0000 7f51c159a700 1 -- 172.21.15.77:0/2674333235 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b4004190 con 0x7f51c406c410 2022-04-03T19:55:43.317 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.313+0000 7f51c159a700 1 -- 172.21.15.77:0/2674333235 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b4011630 con 0x7f51c406c410 2022-04-03T19:55:43.317 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.314+0000 7f51c359e700 1 -- 172.21.15.77:0/2674333235 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51c406c410 msgr2=0x7f51c406c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:43.317 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.314+0000 7f51c359e700 1 --2- 172.21.15.77:0/2674333235 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51c406c410 0x7f51c406c7e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f51b4009790 tx=0x7f51b4009060 comp rx=0 tx=0).stop 2022-04-03T19:55:43.317 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.314+0000 7f51c359e700 1 -- 172.21.15.77:0/2674333235 shutdown_connections 2022-04-03T19:55:43.318 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.314+0000 7f51c359e700 1 --2- 172.21.15.77:0/2674333235 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51c406c410 0x7f51c406c7e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:43.318 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.314+0000 7f51c359e700 1 -- 172.21.15.77:0/2674333235 >> 172.21.15.77:0/2674333235 conn(0x7f51c40c7b60 msgr2=0x7f51c40c7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:43.318 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.314+0000 7f51c359e700 1 -- 172.21.15.77:0/2674333235 shutdown_connections 2022-04-03T19:55:43.318 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.314+0000 7f51c359e700 1 -- 172.21.15.77:0/2674333235 wait complete. 2022-04-03T19:55:43.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.315+0000 7f51c359e700 1 Processor -- start 2022-04-03T19:55:43.321 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.316+0000 7f51c359e700 1 -- start start 2022-04-03T19:55:43.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.316+0000 7f51c359e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51c4069a00 0x7f51c41638a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:43.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.316+0000 7f51c359e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51c4069e10 0x7f51c4150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:43.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.316+0000 7f51c359e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51c406c410 0x7f51c40c0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:43.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.316+0000 7f51c359e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c4150b60 con 0x7f51c4069e10 2022-04-03T19:55:43.322 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.317+0000 7f51c359e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c4152060 con 0x7f51c406c410 2022-04-03T19:55:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.317+0000 7f51c359e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c41598c0 con 0x7f51c4069a00 2022-04-03T19:55:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.317+0000 7f51c1d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51c4069e10 0x7f51c4150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.317+0000 7f51c1d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51c4069e10 0x7f51c4150ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43186/0 (socket says 172.21.15.77:43186) 2022-04-03T19:55:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.317+0000 7f51c1d9b700 1 -- 172.21.15.77:0/717036457 learned_addr learned my addr 172.21.15.77:0/717036457 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:43.323 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.317+0000 7f51c259c700 1 --2- 172.21.15.77:0/717036457 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51c4069a00 0x7f51c41638a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:43.324 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.317+0000 7f51c2d9d700 1 --2- 172.21.15.77:0/717036457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51c406c410 0x7f51c40c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:43.324 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.317+0000 7f51c1d9b700 1 -- 172.21.15.77:0/717036457 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51c4069a00 msgr2=0x7f51c41638a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:43.324 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.317+0000 7f51c1d9b700 1 --2- 172.21.15.77:0/717036457 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51c4069a00 0x7f51c41638a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:43.324 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.317+0000 7f51c1d9b700 1 -- 172.21.15.77:0/717036457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51c406c410 msgr2=0x7f51c40c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:43.325 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.317+0000 7f51c1d9b700 1 --2- 172.21.15.77:0/717036457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51c406c410 0x7f51c40c0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:43.325 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.317+0000 7f51c1d9b700 1 -- 172.21.15.77:0/717036457 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b40092a0 con 0x7f51c4069e10 2022-04-03T19:55:43.325 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.317+0000 7f51c1d9b700 1 --2- 172.21.15.77:0/717036457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51c4069e10 0x7f51c4150ff0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f51b800c600 tx=0x7f51b8007d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:43.325 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.318+0000 7f51b27fc700 1 -- 172.21.15.77:0/717036457 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b8007430 con 0x7f51c4069e10 2022-04-03T19:55:43.326 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.318+0000 7f51b27fc700 1 -- 172.21.15.77:0/717036457 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b8007590 con 0x7f51c4069e10 2022-04-03T19:55:43.326 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.318+0000 7f51c359e700 1 -- 172.21.15.77:0/717036457 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c415a660 con 0x7f51c4069e10 2022-04-03T19:55:43.326 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.318+0000 7f51b27fc700 1 -- 172.21.15.77:0/717036457 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b80048e0 con 0x7f51c4069e10 2022-04-03T19:55:43.326 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.318+0000 7f51c359e700 1 -- 172.21.15.77:0/717036457 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c4150510 con 0x7f51c4069e10 2022-04-03T19:55:43.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.319+0000 7f51c359e700 1 -- 172.21.15.77:0/717036457 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c4068680 con 0x7f51c4069e10 2022-04-03T19:55:43.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.320+0000 7f51b27fc700 1 -- 172.21.15.77:0/717036457 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f51b8010030 con 0x7f51c4069e10 2022-04-03T19:55:43.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.321+0000 7f51b27fc700 1 --2- 172.21.15.77:0/717036457 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f51a0063d10 0x7f51a00661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:43.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.321+0000 7f51b27fc700 1 -- 172.21.15.77:0/717036457 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f51b8080670 con 0x7f51c4069e10 2022-04-03T19:55:43.327 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.321+0000 7f51c259c700 1 --2- 172.21.15.77:0/717036457 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f51a0063d10 0x7f51a00661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:43.328 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.322+0000 7f51c259c700 1 --2- 172.21.15.77:0/717036457 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f51a0063d10 0x7f51a00661c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f51b40096e0 tx=0x7f51b4013000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:43.328 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.328+0000 7f51b27fc700 1 -- 172.21.15.77:0/717036457 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f51b804f010 con 0x7f51c4069e10 2022-04-03T19:55:43.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.596+0000 7f51c359e700 1 -- 172.21.15.77:0/717036457 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51c4153d60 con 0x7f51c4069e10 2022-04-03T19:55:43.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.597+0000 7f51b27fc700 1 -- 172.21.15.77:0/717036457 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5353 (secure 0 0 0) 0x7f51b8052630 con 0x7f51c4069e10 2022-04-03T19:55:43.598 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:55:43.598 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:38.315236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:active","state_seq":30,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:55:43.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.600+0000 7f51c359e700 1 -- 172.21.15.77:0/717036457 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f51a0063d10 msgr2=0x7f51a00661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:43.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.600+0000 7f51c359e700 1 --2- 172.21.15.77:0/717036457 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f51a0063d10 0x7f51a00661c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f51b40096e0 tx=0x7f51b4013000 comp rx=0 tx=0).stop 2022-04-03T19:55:43.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.600+0000 7f51c359e700 1 -- 172.21.15.77:0/717036457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51c4069e10 msgr2=0x7f51c4150ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.600+0000 7f51c359e700 1 --2- 172.21.15.77:0/717036457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51c4069e10 0x7f51c4150ff0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f51b800c600 tx=0x7f51b8007d80 comp rx=0 tx=0).stop 2022-04-03T19:55:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.600+0000 7f51c359e700 1 -- 172.21.15.77:0/717036457 shutdown_connections 2022-04-03T19:55:43.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.600+0000 7f51c359e700 1 --2- 172.21.15.77:0/717036457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51c406c410 0x7f51c40c0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:43.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.600+0000 7f51c359e700 1 --2- 172.21.15.77:0/717036457 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f51a0063d10 0x7f51a00661c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:43.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.600+0000 7f51c359e700 1 --2- 172.21.15.77:0/717036457 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51c4069a00 0x7f51c41638a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:43.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.600+0000 7f51c359e700 1 --2- 172.21.15.77:0/717036457 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f51c4069e10 0x7f51c4150ff0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:43.603 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.600+0000 7f51c359e700 1 -- 172.21.15.77:0/717036457 >> 172.21.15.77:0/717036457 conn(0x7f51c40c7b60 msgr2=0x7f51c40c4e00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:43.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.600+0000 7f51c359e700 1 -- 172.21.15.77:0/717036457 shutdown_connections 2022-04-03T19:55:43.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:43.600+0000 7f51c359e700 1 -- 172.21.15.77:0/717036457 wait complete. 2022-04-03T19:55:43.604 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 42 2022-04-03T19:55:43.631 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2022-04-03T19:55:45.633 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:55:45.830 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.830+0000 7f9a0b59e700 1 Processor -- start 2022-04-03T19:55:45.831 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.830+0000 7f9a0b59e700 1 -- start start 2022-04-03T19:55:45.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.831+0000 7f9a0b59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a0c0c3640 0x7f9a0c069600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:45.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.831+0000 7f9a0b59e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9a0c0c7af0 0x7f9a0c069b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:45.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.831+0000 7f9a0b59e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0c0c4b90 con 0x7f9a0c0c3270 2022-04-03T19:55:45.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.831+0000 7f9a0b59e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a0c0c3d10 con 0x7f9a0c0c7af0 2022-04-03T19:55:45.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.831+0000 7f9a0b59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c0c3f80 con 0x7f9a0c0c3640 2022-04-03T19:55:45.834 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.832+0000 7f9a0a59c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9a0c0c3270 0x7f9a0c0c2500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38562/0 (socket says 172.21.15.77:38562) 2022-04-03T19:55:45.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.832+0000 7f9a0a59c700 1 -- 172.21.15.77:0/2952649237 learned_addr learned my addr 172.21.15.77:0/2952649237 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:55:45.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.832+0000 7f9a0ad9d700 1 --2- 172.21.15.77:0/2952649237 >> v2:172.21.15.77:3300/0 conn(0x7f9a0c0c7af0 0x7f9a0c069b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:45.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.832+0000 7f9a09d9b700 1 --2- 172.21.15.77:0/2952649237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a0c0c3640 0x7f9a0c069600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:45.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.832+0000 7f9a0959a700 1 -- 172.21.15.77:0/2952649237 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91917939 0 0) 0x7f9a0c0c4b90 con 0x7f9a0c0c3270 2022-04-03T19:55:45.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.832+0000 7f9a0959a700 1 -- 172.21.15.77:0/2952649237 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f00033f0 con 0x7f9a0c0c3270 2022-04-03T19:55:45.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.832+0000 7f9a0ad9d700 1 -- 172.21.15.77:0/2952649237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a0c0c3640 msgr2=0x7f9a0c069600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:55:45.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.832+0000 7f9a0ad9d700 1 --2- 172.21.15.77:0/2952649237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a0c0c3640 0x7f9a0c069600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:45.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.832+0000 7f9a0ad9d700 1 -- 172.21.15.77:0/2952649237 >> v1:172.21.15.77:6789/0 conn(0x7f9a0c0c3270 legacy=0x7f9a0c0c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:45.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.832+0000 7f9a0ad9d700 1 -- 172.21.15.77:0/2952649237 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a0c0c4260 con 0x7f9a0c0c7af0 2022-04-03T19:55:45.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.833+0000 7f9a0ad9d700 1 --2- 172.21.15.77:0/2952649237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a0c0c7af0 0x7f9a0c069b40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9a04009760 tx=0x7f9a04009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e49d570c190a0a1b server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:45.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.833+0000 7f9a0959a700 1 -- 172.21.15.77:0/2952649237 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a040105b0 con 0x7f9a0c0c7af0 2022-04-03T19:55:45.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.833+0000 7f9a0959a700 1 -- 172.21.15.77:0/2952649237 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a04010710 con 0x7f9a0c0c7af0 2022-04-03T19:55:45.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.833+0000 7f9a0959a700 1 -- 172.21.15.77:0/2952649237 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a04010ab0 con 0x7f9a0c0c7af0 2022-04-03T19:55:45.841 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.833+0000 7f9a0b59e700 1 -- 172.21.15.77:0/2952649237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a0c0c7af0 msgr2=0x7f9a0c069b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:45.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.833+0000 7f9a0b59e700 1 --2- 172.21.15.77:0/2952649237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a0c0c7af0 0x7f9a0c069b40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9a04009760 tx=0x7f9a04009060 comp rx=0 tx=0).stop 2022-04-03T19:55:45.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.834+0000 7f9a0b59e700 1 -- 172.21.15.77:0/2952649237 shutdown_connections 2022-04-03T19:55:45.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.834+0000 7f9a0b59e700 1 --2- 172.21.15.77:0/2952649237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a0c0c7af0 0x7f9a0c069b40 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:45.842 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.834+0000 7f9a0b59e700 1 --2- 172.21.15.77:0/2952649237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a0c0c3640 0x7f9a0c069600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:45.843 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.834+0000 7f9a0b59e700 1 -- 172.21.15.77:0/2952649237 >> 172.21.15.77:0/2952649237 conn(0x7f9a0c066da0 msgr2=0x7f9a0c0671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:45.843 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.834+0000 7f9a0b59e700 1 -- 172.21.15.77:0/2952649237 shutdown_connections 2022-04-03T19:55:45.843 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.834+0000 7f9a0b59e700 1 -- 172.21.15.77:0/2952649237 wait complete. 2022-04-03T19:55:45.843 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.835+0000 7f9a0b59e700 1 Processor -- start 2022-04-03T19:55:45.843 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.835+0000 7f9a0b59e700 1 -- start start 2022-04-03T19:55:45.844 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.836+0000 7f9a0b59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a0c0c3270 0x7f9a0c167190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:45.844 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.836+0000 7f9a0b59e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a0c0c3640 0x7f9a0c1676d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:45.844 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.836+0000 7f9a0b59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a0c0c7af0 0x7f9a0c167f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:45.844 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.836+0000 7f9a0b59e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c0c11b0 con 0x7f9a0c0c7af0 2022-04-03T19:55:45.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.836+0000 7f9a0b59e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c166d00 con 0x7f9a0c0c3640 2022-04-03T19:55:45.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.836+0000 7f9a0b59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c067d50 con 0x7f9a0c0c3270 2022-04-03T19:55:45.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.837+0000 7f9a0a59c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a0c0c3270 0x7f9a0c167190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:45.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.837+0000 7f9a0ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a0c0c7af0 0x7f9a0c167f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:45.845 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.837+0000 7f9a0a59c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a0c0c3270 0x7f9a0c167190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60480/0 (socket says 172.21.15.77:60480) 2022-04-03T19:55:45.846 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.837+0000 7f9a0a59c700 1 -- 172.21.15.77:0/64172693 learned_addr learned my addr 172.21.15.77:0/64172693 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:55:45.846 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.837+0000 7f9a09d9b700 1 --2- 172.21.15.77:0/64172693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a0c0c3640 0x7f9a0c1676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:45.846 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.837+0000 7f9a0a59c700 1 -- 172.21.15.77:0/64172693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a0c0c3640 msgr2=0x7f9a0c1676d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:45.846 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.837+0000 7f9a0a59c700 1 --2- 172.21.15.77:0/64172693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a0c0c3640 0x7f9a0c1676d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:45.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.837+0000 7f9a0a59c700 1 -- 172.21.15.77:0/64172693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a0c0c7af0 msgr2=0x7f9a0c167f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:45.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.837+0000 7f9a0a59c700 1 --2- 172.21.15.77:0/64172693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a0c0c7af0 0x7f9a0c167f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:45.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.837+0000 7f9a0a59c700 1 -- 172.21.15.77:0/64172693 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a040092a0 con 0x7f9a0c0c3270 2022-04-03T19:55:45.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.837+0000 7f9a0ad9d700 1 --2- 172.21.15.77:0/64172693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a0c0c7af0 0x7f9a0c167f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:55:45.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.837+0000 7f9a0a59c700 1 --2- 172.21.15.77:0/64172693 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a0c0c3270 0x7f9a0c167190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f99fc00c860 tx=0x7f99fc0015b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:45.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.838+0000 7f99fa7fc700 1 -- 172.21.15.77:0/64172693 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99fc005d30 con 0x7f9a0c0c3270 2022-04-03T19:55:45.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.838+0000 7f99fa7fc700 1 -- 172.21.15.77:0/64172693 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99fc005e90 con 0x7f9a0c0c3270 2022-04-03T19:55:45.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.838+0000 7f99fa7fc700 1 -- 172.21.15.77:0/64172693 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99fc007300 con 0x7f9a0c0c3270 2022-04-03T19:55:45.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.838+0000 7f9a0b59e700 1 -- 172.21.15.77:0/64172693 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a0c168c60 con 0x7f9a0c0c3270 2022-04-03T19:55:45.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.838+0000 7f9a0b59e700 1 -- 172.21.15.77:0/64172693 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a0c15c880 con 0x7f9a0c0c3270 2022-04-03T19:55:45.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.839+0000 7f9a0b59e700 1 -- 172.21.15.77:0/64172693 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a0c166fe0 con 0x7f9a0c0c3270 2022-04-03T19:55:45.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.840+0000 7f99fa7fc700 1 -- 172.21.15.77:0/64172693 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f99fc010070 con 0x7f9a0c0c3270 2022-04-03T19:55:45.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.841+0000 7f99fa7fc700 1 --2- 172.21.15.77:0/64172693 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f99e8063ce0 0x7f99e8066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:55:45.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.841+0000 7f99fa7fc700 1 -- 172.21.15.77:0/64172693 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f99fc080c90 con 0x7f9a0c0c3270 2022-04-03T19:55:45.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.841+0000 7f9a09d9b700 1 --2- 172.21.15.77:0/64172693 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f99e8063ce0 0x7f99e8066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:55:45.851 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.842+0000 7f9a09d9b700 1 --2- 172.21.15.77:0/64172693 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f99e8063ce0 0x7f99e8066190 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c0c7280 tx=0x7f9a0000a200 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:55:45.851 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:45.844+0000 7f99fa7fc700 1 -- 172.21.15.77:0/64172693 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f99fc04f750 con 0x7f9a0c0c3270 2022-04-03T19:55:46.128 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:46.128+0000 7f9a0b59e700 1 -- 172.21.15.77:0/64172693 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a0c04b050 con 0x7f9a0c0c3270 2022-04-03T19:55:46.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:46.129+0000 7f99fa7fc700 1 -- 172.21.15.77:0/64172693 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f99fc052d70 con 0x7f9a0c0c3270 2022-04-03T19:55:46.130 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:55:46.130 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5668,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:38.315236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:active","state_seq":30,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:55:46.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:46.131+0000 7f9a0b59e700 1 -- 172.21.15.77:0/64172693 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f99e8063ce0 msgr2=0x7f99e8066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:46.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:46.131+0000 7f9a0b59e700 1 --2- 172.21.15.77:0/64172693 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f99e8063ce0 0x7f99e8066190 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c0c7280 tx=0x7f9a0000a200 comp rx=0 tx=0).stop 2022-04-03T19:55:46.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:46.131+0000 7f9a0b59e700 1 -- 172.21.15.77:0/64172693 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a0c0c3270 msgr2=0x7f9a0c167190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:55:46.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:46.131+0000 7f9a0b59e700 1 --2- 172.21.15.77:0/64172693 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a0c0c3270 0x7f9a0c167190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f99fc00c860 tx=0x7f99fc0015b0 comp rx=0 tx=0).stop 2022-04-03T19:55:46.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:46.131+0000 7f9a0b59e700 1 -- 172.21.15.77:0/64172693 shutdown_connections 2022-04-03T19:55:46.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:46.132+0000 7f9a0b59e700 1 --2- 172.21.15.77:0/64172693 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9a0c0c7af0 0x7f9a0c167f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:46.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:46.132+0000 7f9a0b59e700 1 --2- 172.21.15.77:0/64172693 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f99e8063ce0 0x7f99e8066190 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:46.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:46.132+0000 7f9a0b59e700 1 --2- 172.21.15.77:0/64172693 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a0c0c3270 0x7f9a0c167190 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:46.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:46.132+0000 7f9a0b59e700 1 --2- 172.21.15.77:0/64172693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a0c0c3640 0x7f9a0c1676d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:55:46.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:46.132+0000 7f9a0b59e700 1 -- 172.21.15.77:0/64172693 >> 172.21.15.77:0/64172693 conn(0x7f9a0c066da0 msgr2=0x7f9a0c0ba070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:55:46.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:46.132+0000 7f9a0b59e700 1 -- 172.21.15.77:0/64172693 shutdown_connections 2022-04-03T19:55:46.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:55:46.132+0000 7f9a0b59e700 1 -- 172.21.15.77:0/64172693 wait complete. 2022-04-03T19:55:46.135 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 43 2022-04-03T19:55:46.158 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2022-04-03T19:55:46.159 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 68 secs before thrashing 2022-04-03T19:55:47.310 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:55:47.316 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:55:48.258 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T19:55:48.256+0000 7fdd3ac12700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-04-03T19:56:17.419 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:56:17.425 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:56:47.517 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:56:47.524 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:56:54.160 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:56:54.362 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.361+0000 7f357f1f6700 1 Processor -- start 2022-04-03T19:56:54.363 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.361+0000 7f357f1f6700 1 -- start start 2022-04-03T19:56:54.366 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.363+0000 7f357f1f6700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f35780678b0 0x7f357814cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:54.366 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.363+0000 7f357f1f6700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f35780c1b30 0x7f357814d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:54.366 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.363+0000 7f357f1f6700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35780c3c50 con 0x7f35780ba3c0 2022-04-03T19:56:54.366 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.363+0000 7f357f1f6700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35780c8020 con 0x7f35780678b0 2022-04-03T19:56:54.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.363+0000 7f357f1f6700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3578156690 con 0x7f35780c1b30 2022-04-03T19:56:54.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.363+0000 7f357e1f4700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f35780ba3c0 0x7f3578156b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38564/0 (socket says 172.21.15.77:38564) 2022-04-03T19:56:54.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.363+0000 7f357e1f4700 1 -- 172.21.15.77:0/1135218516 learned_addr learned my addr 172.21.15.77:0/1135218516 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:56:54.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.363+0000 7f357d9f3700 1 --2- 172.21.15.77:0/1135218516 >> v2:172.21.15.77:3300/0 conn(0x7f35780678b0 0x7f357814cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:54.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.363+0000 7f357e9f5700 1 --2- 172.21.15.77:0/1135218516 >> v2:172.21.15.186:3300/0 conn(0x7f35780c1b30 0x7f357814d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:54.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.364+0000 7f357d9f3700 1 -- 172.21.15.77:0/1135218516 >> v2:172.21.15.186:3300/0 conn(0x7f35780c1b30 msgr2=0x7f357814d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:56:54.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.364+0000 7f357d9f3700 1 --2- 172.21.15.77:0/1135218516 >> v2:172.21.15.186:3300/0 conn(0x7f35780c1b30 0x7f357814d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:54.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.364+0000 7f357d1f2700 1 -- 172.21.15.77:0/1135218516 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2599881122 0 0) 0x7f35780c3c50 con 0x7f35780ba3c0 2022-04-03T19:56:54.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.364+0000 7f357d9f3700 1 -- 172.21.15.77:0/1135218516 >> v1:172.21.15.77:6789/0 conn(0x7f35780ba3c0 legacy=0x7f3578156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:54.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.364+0000 7f357d9f3700 1 -- 172.21.15.77:0/1135218516 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35780bf350 con 0x7f35780678b0 2022-04-03T19:56:54.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.364+0000 7f357d9f3700 1 --2- 172.21.15.77:0/1135218516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35780678b0 0x7f357814cb00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f35780c2430 tx=0x7f3568009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=505a1d6c571c5fe2 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:56:54.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.364+0000 7f357d1f2700 1 -- 172.21.15.77:0/1135218516 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3568011630 con 0x7f35780678b0 2022-04-03T19:56:54.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.364+0000 7f357d1f2700 1 -- 172.21.15.77:0/1135218516 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3568011790 con 0x7f35780678b0 2022-04-03T19:56:54.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.364+0000 7f357d1f2700 1 -- 172.21.15.77:0/1135218516 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3568011b30 con 0x7f35780678b0 2022-04-03T19:56:54.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.365+0000 7f357f1f6700 1 -- 172.21.15.77:0/1135218516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35780678b0 msgr2=0x7f357814cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:54.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.365+0000 7f357f1f6700 1 --2- 172.21.15.77:0/1135218516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35780678b0 0x7f357814cb00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f35780c2430 tx=0x7f3568009060 comp rx=0 tx=0).stop 2022-04-03T19:56:54.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.365+0000 7f357f1f6700 1 -- 172.21.15.77:0/1135218516 shutdown_connections 2022-04-03T19:56:54.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.365+0000 7f357f1f6700 1 --2- 172.21.15.77:0/1135218516 >> v2:172.21.15.186:3300/0 conn(0x7f35780c1b30 0x7f357814d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:54.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.365+0000 7f357f1f6700 1 --2- 172.21.15.77:0/1135218516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35780678b0 0x7f357814cb00 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:54.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.365+0000 7f357f1f6700 1 -- 172.21.15.77:0/1135218516 >> 172.21.15.77:0/1135218516 conn(0x7f3578066da0 msgr2=0x7f35780671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:56:54.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.365+0000 7f357f1f6700 1 -- 172.21.15.77:0/1135218516 shutdown_connections 2022-04-03T19:56:54.372 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.365+0000 7f357f1f6700 1 -- 172.21.15.77:0/1135218516 wait complete. 2022-04-03T19:56:54.374 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.366+0000 7f357f1f6700 1 Processor -- start 2022-04-03T19:56:54.375 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.367+0000 7f357f1f6700 1 -- start start 2022-04-03T19:56:54.375 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.367+0000 7f357f1f6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35780678b0 0x7f357814b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:54.375 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.367+0000 7f357f1f6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35780ba3c0 0x7f3578146c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:54.375 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.367+0000 7f357f1f6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35780c1b30 0x7f3578147180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:54.375 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.367+0000 7f357f1f6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3578165e80 con 0x7f35780678b0 2022-04-03T19:56:54.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.367+0000 7f357f1f6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3578165200 con 0x7f35780ba3c0 2022-04-03T19:56:54.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.367+0000 7f357f1f6700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35781654e0 con 0x7f35780c1b30 2022-04-03T19:56:54.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.368+0000 7f357e1f4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35780678b0 0x7f357814b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:54.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.368+0000 7f357e1f4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35780678b0 0x7f357814b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43194/0 (socket says 172.21.15.77:43194) 2022-04-03T19:56:54.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.368+0000 7f357e1f4700 1 -- 172.21.15.77:0/3744136854 learned_addr learned my addr 172.21.15.77:0/3744136854 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:56:54.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.368+0000 7f357e9f5700 1 --2- 172.21.15.77:0/3744136854 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35780c1b30 0x7f3578147180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:54.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.368+0000 7f357d9f3700 1 --2- 172.21.15.77:0/3744136854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35780ba3c0 0x7f3578146c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:54.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.368+0000 7f357e1f4700 1 -- 172.21.15.77:0/3744136854 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35780c1b30 msgr2=0x7f3578147180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:54.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.368+0000 7f357e1f4700 1 --2- 172.21.15.77:0/3744136854 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35780c1b30 0x7f3578147180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:54.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.368+0000 7f357e1f4700 1 -- 172.21.15.77:0/3744136854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35780ba3c0 msgr2=0x7f3578146c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:54.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.368+0000 7f357e1f4700 1 --2- 172.21.15.77:0/3744136854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35780ba3c0 0x7f3578146c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:54.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.368+0000 7f357e1f4700 1 -- 172.21.15.77:0/3744136854 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35680092a0 con 0x7f35780678b0 2022-04-03T19:56:54.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.368+0000 7f357d9f3700 1 --2- 172.21.15.77:0/3744136854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35780ba3c0 0x7f3578146c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:56:54.379 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.368+0000 7f357e1f4700 1 --2- 172.21.15.77:0/3744136854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35780678b0 0x7f357814b470 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f357400c4e0 tx=0x7f3574007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:56:54.379 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.369+0000 7f35667fc700 1 -- 172.21.15.77:0/3744136854 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3574007360 con 0x7f35780678b0 2022-04-03T19:56:54.379 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.369+0000 7f35667fc700 1 -- 172.21.15.77:0/3744136854 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3574007500 con 0x7f35780678b0 2022-04-03T19:56:54.379 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.369+0000 7f35667fc700 1 -- 172.21.15.77:0/3744136854 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3574004900 con 0x7f35780678b0 2022-04-03T19:56:54.380 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.369+0000 7f357f1f6700 1 -- 172.21.15.77:0/3744136854 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35781657c0 con 0x7f35780678b0 2022-04-03T19:56:54.380 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.369+0000 7f357f1f6700 1 -- 172.21.15.77:0/3744136854 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35780bef70 con 0x7f35780678b0 2022-04-03T19:56:54.380 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.370+0000 7f357f1f6700 1 -- 172.21.15.77:0/3744136854 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35780c74a0 con 0x7f35780678b0 2022-04-03T19:56:54.380 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.371+0000 7f35667fc700 1 -- 172.21.15.77:0/3744136854 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f3574010030 con 0x7f35780678b0 2022-04-03T19:56:54.380 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.371+0000 7f35667fc700 1 --2- 172.21.15.77:0/3744136854 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3558063ce0 0x7f3558066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:54.381 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.372+0000 7f35667fc700 1 -- 172.21.15.77:0/3744136854 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f3574080790 con 0x7f35780678b0 2022-04-03T19:56:54.381 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.372+0000 7f357d9f3700 1 --2- 172.21.15.77:0/3744136854 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3558063ce0 0x7f3558066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:54.381 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.373+0000 7f357d9f3700 1 --2- 172.21.15.77:0/3744136854 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3558063ce0 0x7f3558066190 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3578067cc0 tx=0x7f3568017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:56:54.382 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.376+0000 7f35667fc700 1 -- 172.21.15.77:0/3744136854 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f357404f0c0 con 0x7f35780678b0 2022-04-03T19:56:54.611 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.607+0000 7f357f1f6700 1 -- 172.21.15.77:0/3744136854 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f357814adc0 con 0x7f35780678b0 2022-04-03T19:56:54.612 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.608+0000 7f35667fc700 1 -- 172.21.15.77:0/3744136854 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f35740526e0 con 0x7f35780678b0 2022-04-03T19:56:54.613 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:56:54.613 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5668,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:38.315236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:active","state_seq":30,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:56:54.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.610+0000 7f357f1f6700 1 -- 172.21.15.77:0/3744136854 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3558063ce0 msgr2=0x7f3558066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:54.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.610+0000 7f357f1f6700 1 --2- 172.21.15.77:0/3744136854 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3558063ce0 0x7f3558066190 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3578067cc0 tx=0x7f3568017000 comp rx=0 tx=0).stop 2022-04-03T19:56:54.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.610+0000 7f357f1f6700 1 -- 172.21.15.77:0/3744136854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35780678b0 msgr2=0x7f357814b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:54.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.610+0000 7f357f1f6700 1 --2- 172.21.15.77:0/3744136854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35780678b0 0x7f357814b470 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f357400c4e0 tx=0x7f3574007c70 comp rx=0 tx=0).stop 2022-04-03T19:56:54.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.610+0000 7f357f1f6700 1 -- 172.21.15.77:0/3744136854 shutdown_connections 2022-04-03T19:56:54.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.610+0000 7f357f1f6700 1 --2- 172.21.15.77:0/3744136854 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3558063ce0 0x7f3558066190 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:54.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.610+0000 7f357f1f6700 1 --2- 172.21.15.77:0/3744136854 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35780c1b30 0x7f3578147180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:54.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.610+0000 7f357f1f6700 1 --2- 172.21.15.77:0/3744136854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f35780678b0 0x7f357814b470 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:54.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.610+0000 7f357f1f6700 1 --2- 172.21.15.77:0/3744136854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35780ba3c0 0x7f3578146c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:54.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.610+0000 7f357f1f6700 1 -- 172.21.15.77:0/3744136854 >> 172.21.15.77:0/3744136854 conn(0x7f3578066da0 msgr2=0x7f35780b9910 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:56:54.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.610+0000 7f357f1f6700 1 -- 172.21.15.77:0/3744136854 shutdown_connections 2022-04-03T19:56:54.617 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.610+0000 7f357f1f6700 1 -- 172.21.15.77:0/3744136854 wait complete. 2022-04-03T19:56:54.617 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 43 2022-04-03T19:56:54.634 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2022-04-03T19:56:54.635 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2022-04-03T19:56:54.635 INFO:teuthology.orchestra.run:waiting for 300 2022-04-03T19:56:54.684 INFO:tasks.ceph.mds.e:Stopped 2022-04-03T19:56:54.685 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:56:54.883 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.882+0000 7fe123211700 1 Processor -- start 2022-04-03T19:56:54.884 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.883+0000 7fe123211700 1 -- start start 2022-04-03T19:56:54.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.884+0000 7fe123211700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fe11c1584b0 0x7fe11c14eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:54.885 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.884+0000 7fe123211700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11c06d260 con 0x7fe11c06bfb0 2022-04-03T19:56:54.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.884+0000 7fe123211700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11c0c6330 con 0x7fe11c06bbe0 2022-04-03T19:56:54.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.884+0000 7fe123211700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe11c159450 con 0x7fe11c1584b0 2022-04-03T19:56:54.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.884+0000 7fe121a0e700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fe11c06bfb0 0x7fe11c06c500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38566/0 (socket says 172.21.15.77:38566) 2022-04-03T19:56:54.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.884+0000 7fe121a0e700 1 -- 172.21.15.77:0/3293935371 learned_addr learned my addr 172.21.15.77:0/3293935371 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:56:54.886 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.884+0000 7fe122a10700 1 --2- 172.21.15.77:0/3293935371 >> v2:172.21.15.77:3300/0 conn(0x7fe11c1584b0 0x7fe11c14eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:54.887 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.885+0000 7fe12120d700 1 -- 172.21.15.77:0/3293935371 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023925318 0 0) 0x7fe11c0c6330 con 0x7fe11c06bbe0 2022-04-03T19:56:54.887 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.885+0000 7fe122a10700 1 -- 172.21.15.77:0/3293935371 >> v1:172.21.15.186:6789/0 conn(0x7fe11c06bbe0 legacy=0x7fe11c06d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:54.887 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.885+0000 7fe122a10700 1 -- 172.21.15.77:0/3293935371 >> v1:172.21.15.77:6789/0 conn(0x7fe11c06bfb0 legacy=0x7fe11c06c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:54.887 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.885+0000 7fe122a10700 1 -- 172.21.15.77:0/3293935371 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11c158910 con 0x7fe11c1584b0 2022-04-03T19:56:54.890 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.885+0000 7fe122a10700 1 --2- 172.21.15.77:0/3293935371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c1584b0 0x7fe11c14eea0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe114009e60 tx=0x7fe114009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb953e2f17fa1c6e server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:56:54.891 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.885+0000 7fe12120d700 1 -- 172.21.15.77:0/3293935371 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11400e070 con 0x7fe11c1584b0 2022-04-03T19:56:54.891 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.885+0000 7fe12120d700 1 -- 172.21.15.77:0/3293935371 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11400bb30 con 0x7fe11c1584b0 2022-04-03T19:56:54.891 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.885+0000 7fe12120d700 1 -- 172.21.15.77:0/3293935371 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe114003860 con 0x7fe11c1584b0 2022-04-03T19:56:54.891 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.886+0000 7fe123211700 1 -- 172.21.15.77:0/3293935371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c1584b0 msgr2=0x7fe11c14eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:54.892 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.886+0000 7fe123211700 1 --2- 172.21.15.77:0/3293935371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c1584b0 0x7fe11c14eea0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe114009e60 tx=0x7fe114009780 comp rx=0 tx=0).stop 2022-04-03T19:56:54.892 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.886+0000 7fe123211700 1 -- 172.21.15.77:0/3293935371 shutdown_connections 2022-04-03T19:56:54.893 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.886+0000 7fe123211700 1 --2- 172.21.15.77:0/3293935371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c1584b0 0x7fe11c14eea0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:54.893 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.886+0000 7fe123211700 1 -- 172.21.15.77:0/3293935371 >> 172.21.15.77:0/3293935371 conn(0x7fe11c0cb6b0 msgr2=0x7fe11c0cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:56:54.894 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.886+0000 7fe123211700 1 -- 172.21.15.77:0/3293935371 shutdown_connections 2022-04-03T19:56:54.894 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.886+0000 7fe123211700 1 -- 172.21.15.77:0/3293935371 wait complete. 2022-04-03T19:56:54.894 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.888+0000 7fe123211700 1 Processor -- start 2022-04-03T19:56:54.895 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.888+0000 7fe123211700 1 -- start start 2022-04-03T19:56:54.895 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.889+0000 7fe123211700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe11c06bbe0 0x7fe11c156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:54.896 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.889+0000 7fe123211700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c06bfb0 0x7fe11c14f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:54.896 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.889+0000 7fe123211700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe11c1584b0 0x7fe11c157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:54.897 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.889+0000 7fe123211700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c1568a0 con 0x7fe11c06bfb0 2022-04-03T19:56:54.897 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.889+0000 7fe123211700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c14f6a0 con 0x7fe11c06bbe0 2022-04-03T19:56:54.897 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.889+0000 7fe123211700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c14ec70 con 0x7fe11c1584b0 2022-04-03T19:56:54.898 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.889+0000 7fe121a0e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c06bfb0 0x7fe11c14f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:54.898 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.889+0000 7fe122a10700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe11c1584b0 0x7fe11c157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:54.899 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.889+0000 7fe12220f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe11c06bbe0 0x7fe11c156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:54.899 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.889+0000 7fe122a10700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe11c1584b0 0x7fe11c157cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60484/0 (socket says 172.21.15.77:60484) 2022-04-03T19:56:54.900 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.889+0000 7fe122a10700 1 -- 172.21.15.77:0/2157864697 learned_addr learned my addr 172.21.15.77:0/2157864697 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:56:54.900 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.890+0000 7fe122a10700 1 -- 172.21.15.77:0/2157864697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe11c06bbe0 msgr2=0x7fe11c156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:54.900 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.890+0000 7fe122a10700 1 --2- 172.21.15.77:0/2157864697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe11c06bbe0 0x7fe11c156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:54.901 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.890+0000 7fe122a10700 1 -- 172.21.15.77:0/2157864697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c06bfb0 msgr2=0x7fe11c14f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:54.901 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.890+0000 7fe122a10700 1 --2- 172.21.15.77:0/2157864697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c06bfb0 0x7fe11c14f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:54.902 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.890+0000 7fe122a10700 1 -- 172.21.15.77:0/2157864697 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe114009a20 con 0x7fe11c1584b0 2022-04-03T19:56:54.902 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.890+0000 7fe12220f700 1 --2- 172.21.15.77:0/2157864697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe11c06bbe0 0x7fe11c156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:56:54.902 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.890+0000 7fe121a0e700 1 --2- 172.21.15.77:0/2157864697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c06bfb0 0x7fe11c14f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:56:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.890+0000 7fe122a10700 1 --2- 172.21.15.77:0/2157864697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe11c1584b0 0x7fe11c157cf0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe11400b6c0 tx=0x7fe114018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:56:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.890+0000 7fe10a7fc700 1 -- 172.21.15.77:0/2157864697 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe114004490 con 0x7fe11c1584b0 2022-04-03T19:56:54.904 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.890+0000 7fe10a7fc700 1 -- 172.21.15.77:0/2157864697 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe114004610 con 0x7fe11c1584b0 2022-04-03T19:56:54.904 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.890+0000 7fe123211700 1 -- 172.21.15.77:0/2157864697 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11c14f120 con 0x7fe11c1584b0 2022-04-03T19:56:54.904 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.890+0000 7fe10a7fc700 1 -- 172.21.15.77:0/2157864697 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11401e430 con 0x7fe11c1584b0 2022-04-03T19:56:54.905 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.890+0000 7fe123211700 1 -- 172.21.15.77:0/2157864697 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11c156250 con 0x7fe11c1584b0 2022-04-03T19:56:54.907 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.891+0000 7fe123211700 1 -- 172.21.15.77:0/2157864697 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe11c156a50 con 0x7fe11c1584b0 2022-04-03T19:56:54.907 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.893+0000 7fe10a7fc700 1 -- 172.21.15.77:0/2157864697 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fe11401e7e0 con 0x7fe11c1584b0 2022-04-03T19:56:54.908 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.893+0000 7fe10a7fc700 1 --2- 172.21.15.77:0/2157864697 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe0fc063d10 0x7fe0fc0661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:54.908 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.893+0000 7fe10a7fc700 1 -- 172.21.15.77:0/2157864697 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7fe114083370 con 0x7fe11c1584b0 2022-04-03T19:56:54.908 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.894+0000 7fe12220f700 1 --2- 172.21.15.77:0/2157864697 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe0fc063d10 0x7fe0fc0661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:54.908 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.894+0000 7fe12220f700 1 --2- 172.21.15.77:0/2157864697 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe0fc063d10 0x7fe0fc0661c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe118007940 tx=0x7fe118009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:56:54.909 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:54.897+0000 7fe10a7fc700 1 -- 172.21.15.77:0/2157864697 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fe114051d30 con 0x7fe11c1584b0 2022-04-03T19:56:55.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:55.129+0000 7fe123211700 1 -- 172.21.15.77:0/2157864697 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe11c156a50 con 0x7fe11c1584b0 2022-04-03T19:56:55.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:55.130+0000 7fe10a7fc700 1 -- 172.21.15.77:0/2157864697 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6110 (secure 0 0 0) 0x7fe11c156a50 con 0x7fe11c1584b0 2022-04-03T19:56:55.131 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:56:55.131 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5668,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:38.315236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:active","state_seq":30,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:56:55.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:55.132+0000 7fe123211700 1 -- 172.21.15.77:0/2157864697 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe0fc063d10 msgr2=0x7fe0fc0661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:55.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:55.132+0000 7fe123211700 1 --2- 172.21.15.77:0/2157864697 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe0fc063d10 0x7fe0fc0661c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe118007940 tx=0x7fe118009040 comp rx=0 tx=0).stop 2022-04-03T19:56:55.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:55.132+0000 7fe123211700 1 -- 172.21.15.77:0/2157864697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe11c1584b0 msgr2=0x7fe11c157cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:55.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:55.132+0000 7fe123211700 1 --2- 172.21.15.77:0/2157864697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe11c1584b0 0x7fe11c157cf0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe11400b6c0 tx=0x7fe114018000 comp rx=0 tx=0).stop 2022-04-03T19:56:55.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:55.133+0000 7fe123211700 1 -- 172.21.15.77:0/2157864697 shutdown_connections 2022-04-03T19:56:55.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:55.133+0000 7fe123211700 1 --2- 172.21.15.77:0/2157864697 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fe0fc063d10 0x7fe0fc0661c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:55.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:55.133+0000 7fe123211700 1 --2- 172.21.15.77:0/2157864697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe11c1584b0 0x7fe11c157cf0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:55.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:55.133+0000 7fe123211700 1 --2- 172.21.15.77:0/2157864697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe11c06bbe0 0x7fe11c156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:55.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:55.133+0000 7fe123211700 1 --2- 172.21.15.77:0/2157864697 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe11c06bfb0 0x7fe11c14f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:55.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:55.133+0000 7fe123211700 1 -- 172.21.15.77:0/2157864697 >> 172.21.15.77:0/2157864697 conn(0x7fe11c0cb6b0 msgr2=0x7fe11c0c7640 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:56:55.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:55.133+0000 7fe123211700 1 -- 172.21.15.77:0/2157864697 shutdown_connections 2022-04-03T19:56:55.136 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:55.133+0000 7fe123211700 1 -- 172.21.15.77:0/2157864697 wait complete. 2022-04-03T19:56:55.137 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 43 2022-04-03T19:56:55.158 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:56:57.159 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:56:57.365 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.365+0000 7f49d8f7a700 1 Processor -- start 2022-04-03T19:56:57.366 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.365+0000 7f49d8f7a700 1 -- start start 2022-04-03T19:56:57.367 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.366+0000 7f49d8f7a700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f49d41584b0 0x7f49d414eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:57.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.366+0000 7f49d8f7a700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d406d260 con 0x7f49d406bfb0 2022-04-03T19:56:57.368 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.366+0000 7f49d8f7a700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d40c6330 con 0x7f49d406bbe0 2022-04-03T19:56:57.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.366+0000 7f49d8f7a700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d4159450 con 0x7f49d41584b0 2022-04-03T19:56:57.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.367+0000 7f49d3fff700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f49d41584b0 0x7f49d414eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:57.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.367+0000 7f49d37fe700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f49d406bbe0 0x7f49d406d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44370/0 (socket says 172.21.15.77:44370) 2022-04-03T19:56:57.369 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.367+0000 7f49d37fe700 1 -- 172.21.15.77:0/3255312269 learned_addr learned my addr 172.21.15.77:0/3255312269 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:56:57.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.367+0000 7f49d3fff700 1 -- 172.21.15.77:0/3255312269 >> v1:172.21.15.186:6789/0 conn(0x7f49d406bbe0 legacy=0x7f49d406d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:57.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.367+0000 7f49d3fff700 1 -- 172.21.15.77:0/3255312269 >> v1:172.21.15.77:6789/0 conn(0x7f49d406bfb0 legacy=0x7f49d406c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:57.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.367+0000 7f49d27fc700 1 -- 172.21.15.77:0/3255312269 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902960680 0 0) 0x7f49d40c6330 con 0x7f49d406bbe0 2022-04-03T19:56:57.370 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.367+0000 7f49d3fff700 1 -- 172.21.15.77:0/3255312269 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d4158910 con 0x7f49d41584b0 2022-04-03T19:56:57.371 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.367+0000 7f49d3fff700 1 --2- 172.21.15.77:0/3255312269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49d41584b0 0x7f49d414eea0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f49cc009e60 tx=0x7f49cc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17a5e0b2b7eee6a7 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:56:57.374 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.368+0000 7f49d27fc700 1 -- 172.21.15.77:0/3255312269 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49cc00e070 con 0x7f49d41584b0 2022-04-03T19:56:57.375 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.368+0000 7f49d27fc700 1 -- 172.21.15.77:0/3255312269 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49cc00bc30 con 0x7f49d41584b0 2022-04-03T19:56:57.375 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.368+0000 7f49d27fc700 1 -- 172.21.15.77:0/3255312269 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49cc012430 con 0x7f49d41584b0 2022-04-03T19:56:57.375 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.368+0000 7f49d8f7a700 1 -- 172.21.15.77:0/3255312269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49d41584b0 msgr2=0x7f49d414eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:57.375 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.368+0000 7f49d8f7a700 1 --2- 172.21.15.77:0/3255312269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49d41584b0 0x7f49d414eea0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f49cc009e60 tx=0x7f49cc009780 comp rx=0 tx=0).stop 2022-04-03T19:56:57.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.368+0000 7f49d8f7a700 1 -- 172.21.15.77:0/3255312269 shutdown_connections 2022-04-03T19:56:57.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.368+0000 7f49d8f7a700 1 --2- 172.21.15.77:0/3255312269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49d41584b0 0x7f49d414eea0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:57.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.368+0000 7f49d8f7a700 1 -- 172.21.15.77:0/3255312269 >> 172.21.15.77:0/3255312269 conn(0x7f49d40cb6b0 msgr2=0x7f49d40cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:56:57.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.368+0000 7f49d8f7a700 1 -- 172.21.15.77:0/3255312269 shutdown_connections 2022-04-03T19:56:57.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.368+0000 7f49d8f7a700 1 -- 172.21.15.77:0/3255312269 wait complete. 2022-04-03T19:56:57.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.370+0000 7f49d8f7a700 1 Processor -- start 2022-04-03T19:56:57.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.370+0000 7f49d8f7a700 1 -- start start 2022-04-03T19:56:57.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.371+0000 7f49d8f7a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49d406bbe0 0x7f49d4157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:57.377 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.371+0000 7f49d8f7a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49d406bfb0 0x7f49d4156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:57.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.371+0000 7f49d8f7a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49d41584b0 0x7f49d4156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:57.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.371+0000 7f49d8f7a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d4158310 con 0x7f49d406bbe0 2022-04-03T19:56:57.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.371+0000 7f49d8f7a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d4152270 con 0x7f49d406bfb0 2022-04-03T19:56:57.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.371+0000 7f49d8f7a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d41513e0 con 0x7f49d41584b0 2022-04-03T19:56:57.378 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.371+0000 7f49d37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49d406bbe0 0x7f49d4157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:57.379 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.371+0000 7f49d37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49d406bbe0 0x7f49d4157dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43200/0 (socket says 172.21.15.77:43200) 2022-04-03T19:56:57.379 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.371+0000 7f49d37fe700 1 -- 172.21.15.77:0/1184404100 learned_addr learned my addr 172.21.15.77:0/1184404100 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:56:57.379 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.371+0000 7f49d3fff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49d41584b0 0x7f49d4156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:57.379 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.371+0000 7f49d2ffd700 1 --2- 172.21.15.77:0/1184404100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49d406bfb0 0x7f49d4156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:57.380 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.372+0000 7f49d37fe700 1 -- 172.21.15.77:0/1184404100 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49d41584b0 msgr2=0x7f49d4156940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:57.380 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.372+0000 7f49d37fe700 1 --2- 172.21.15.77:0/1184404100 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49d41584b0 0x7f49d4156940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:57.380 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.372+0000 7f49d37fe700 1 -- 172.21.15.77:0/1184404100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49d406bfb0 msgr2=0x7f49d4156400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:57.380 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.372+0000 7f49d37fe700 1 --2- 172.21.15.77:0/1184404100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49d406bfb0 0x7f49d4156400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:57.381 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.372+0000 7f49d37fe700 1 -- 172.21.15.77:0/1184404100 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49cc009a20 con 0x7f49d406bbe0 2022-04-03T19:56:57.381 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.372+0000 7f49d37fe700 1 --2- 172.21.15.77:0/1184404100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49d406bbe0 0x7f49d4157dd0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f49c400c490 tx=0x7f49c4007c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:56:57.381 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.372+0000 7f49bb7fe700 1 -- 172.21.15.77:0/1184404100 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c400ce00 con 0x7f49d406bbe0 2022-04-03T19:56:57.381 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.372+0000 7f49bb7fe700 1 -- 172.21.15.77:0/1184404100 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c4007360 con 0x7f49d406bbe0 2022-04-03T19:56:57.381 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.372+0000 7f49bb7fe700 1 -- 172.21.15.77:0/1184404100 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c40046b0 con 0x7f49d406bbe0 2022-04-03T19:56:57.382 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.372+0000 7f49d8f7a700 1 -- 172.21.15.77:0/1184404100 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d4151660 con 0x7f49d406bbe0 2022-04-03T19:56:57.382 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.372+0000 7f49d8f7a700 1 -- 172.21.15.77:0/1184404100 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d4151c20 con 0x7f49d406bbe0 2022-04-03T19:56:57.382 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.373+0000 7f49d8f7a700 1 -- 172.21.15.77:0/1184404100 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d4156250 con 0x7f49d406bbe0 2022-04-03T19:56:57.382 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.374+0000 7f49bb7fe700 1 -- 172.21.15.77:0/1184404100 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f49c4010030 con 0x7f49d406bbe0 2022-04-03T19:56:57.383 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.375+0000 7f49bb7fe700 1 --2- 172.21.15.77:0/1184404100 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f49b4063ce0 0x7f49b4066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:57.384 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.375+0000 7f49bb7fe700 1 -- 172.21.15.77:0/1184404100 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f49c40804d0 con 0x7f49d406bbe0 2022-04-03T19:56:57.384 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.375+0000 7f49d2ffd700 1 --2- 172.21.15.77:0/1184404100 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f49b4063ce0 0x7f49b4066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:57.384 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.376+0000 7f49d2ffd700 1 --2- 172.21.15.77:0/1184404100 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f49b4063ce0 0x7f49b4066190 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f49c8001840 tx=0x7f49c8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:56:57.384 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.379+0000 7f49bb7fe700 1 -- 172.21.15.77:0/1184404100 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f49c404ee00 con 0x7f49d406bbe0 2022-04-03T19:56:57.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.609+0000 7f49d8f7a700 1 -- 172.21.15.77:0/1184404100 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49d40cba80 con 0x7f49d406bbe0 2022-04-03T19:56:57.611 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.610+0000 7f49bb7fe700 1 -- 172.21.15.77:0/1184404100 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f49c4052420 con 0x7f49d406bbe0 2022-04-03T19:56:57.611 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:56:57.611 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5668,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:38.315236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:active","state_seq":30,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:56:57.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.612+0000 7f49d8f7a700 1 -- 172.21.15.77:0/1184404100 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f49b4063ce0 msgr2=0x7f49b4066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:57.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.612+0000 7f49d8f7a700 1 --2- 172.21.15.77:0/1184404100 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f49b4063ce0 0x7f49b4066190 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f49c8001840 tx=0x7f49c8009040 comp rx=0 tx=0).stop 2022-04-03T19:56:57.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.612+0000 7f49d8f7a700 1 -- 172.21.15.77:0/1184404100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49d406bbe0 msgr2=0x7f49d4157dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:57.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.612+0000 7f49d8f7a700 1 --2- 172.21.15.77:0/1184404100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49d406bbe0 0x7f49d4157dd0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f49c400c490 tx=0x7f49c4007c40 comp rx=0 tx=0).stop 2022-04-03T19:56:57.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.613+0000 7f49d8f7a700 1 -- 172.21.15.77:0/1184404100 shutdown_connections 2022-04-03T19:56:57.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.613+0000 7f49d8f7a700 1 --2- 172.21.15.77:0/1184404100 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f49b4063ce0 0x7f49b4066190 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:57.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.613+0000 7f49d8f7a700 1 --2- 172.21.15.77:0/1184404100 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49d41584b0 0x7f49d4156940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:57.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.613+0000 7f49d8f7a700 1 --2- 172.21.15.77:0/1184404100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f49d406bbe0 0x7f49d4157dd0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:57.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.613+0000 7f49d8f7a700 1 --2- 172.21.15.77:0/1184404100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49d406bfb0 0x7f49d4156400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:57.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.613+0000 7f49d8f7a700 1 -- 172.21.15.77:0/1184404100 >> 172.21.15.77:0/1184404100 conn(0x7f49d40cb6b0 msgr2=0x7f49d40c6e70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:56:57.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.613+0000 7f49d8f7a700 1 -- 172.21.15.77:0/1184404100 shutdown_connections 2022-04-03T19:56:57.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:57.613+0000 7f49d8f7a700 1 -- 172.21.15.77:0/1184404100 wait complete. 2022-04-03T19:56:57.616 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 43 2022-04-03T19:56:57.638 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:56:59.639 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:56:59.844 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.843+0000 7f00ec8e6700 1 Processor -- start 2022-04-03T19:56:59.844 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.843+0000 7f00ec8e6700 1 -- start start 2022-04-03T19:56:59.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.844+0000 7f00ec8e6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00e80c1170 0x7f00e81527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:59.847 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.845+0000 7f00ec8e6700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00e80c3710 con 0x7f00e80c3b10 2022-04-03T19:56:59.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.845+0000 7f00ec8e6700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00e8152330 con 0x7f00e8155370 2022-04-03T19:56:59.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.845+0000 7f00ec8e6700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00e80c1540 con 0x7f00e80c1170 2022-04-03T19:56:59.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.845+0000 7f00e7fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f00e80c3b10 0x7f00e815c6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38570/0 (socket says 172.21.15.77:38570) 2022-04-03T19:56:59.848 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.845+0000 7f00e7fff700 1 -- 172.21.15.77:0/4140596977 learned_addr learned my addr 172.21.15.77:0/4140596977 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:56:59.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.845+0000 7f00e77fe700 1 --2- 172.21.15.77:0/4140596977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00e80c1170 0x7f00e81527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:59.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.845+0000 7f00e77fe700 1 -- 172.21.15.77:0/4140596977 >> v1:172.21.15.186:6789/0 conn(0x7f00e8155370 legacy=0x7f00e8069110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:59.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.845+0000 7f00e67fc700 1 -- 172.21.15.77:0/4140596977 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3449749134 0 0) 0x7f00e8152330 con 0x7f00e8155370 2022-04-03T19:56:59.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.845+0000 7f00e77fe700 1 -- 172.21.15.77:0/4140596977 >> v1:172.21.15.77:6789/0 conn(0x7f00e80c3b10 legacy=0x7f00e815c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:59.849 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.846+0000 7f00e77fe700 1 -- 172.21.15.77:0/4140596977 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e80c4270 con 0x7f00e80c1170 2022-04-03T19:56:59.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.846+0000 7f00e77fe700 1 --2- 172.21.15.77:0/4140596977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00e80c1170 0x7f00e81527c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f00d8009790 tx=0x7f00d8009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=830bd4e46f5e87de server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:56:59.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.846+0000 7f00e67fc700 1 -- 172.21.15.77:0/4140596977 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d800f6b0 con 0x7f00e80c1170 2022-04-03T19:56:59.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.846+0000 7f00e67fc700 1 -- 172.21.15.77:0/4140596977 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d800f810 con 0x7f00e80c1170 2022-04-03T19:56:59.850 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.846+0000 7f00e67fc700 1 -- 172.21.15.77:0/4140596977 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d800fbb0 con 0x7f00e80c1170 2022-04-03T19:56:59.851 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.847+0000 7f00ec8e6700 1 -- 172.21.15.77:0/4140596977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00e80c1170 msgr2=0x7f00e81527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:59.851 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.847+0000 7f00ec8e6700 1 --2- 172.21.15.77:0/4140596977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00e80c1170 0x7f00e81527c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f00d8009790 tx=0x7f00d8009060 comp rx=0 tx=0).stop 2022-04-03T19:56:59.851 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.847+0000 7f00ec8e6700 1 -- 172.21.15.77:0/4140596977 shutdown_connections 2022-04-03T19:56:59.851 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.847+0000 7f00ec8e6700 1 --2- 172.21.15.77:0/4140596977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00e80c1170 0x7f00e81527c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:59.852 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.847+0000 7f00ec8e6700 1 -- 172.21.15.77:0/4140596977 >> 172.21.15.77:0/4140596977 conn(0x7f00e8066da0 msgr2=0x7f00e80671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:56:59.852 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.847+0000 7f00ec8e6700 1 -- 172.21.15.77:0/4140596977 shutdown_connections 2022-04-03T19:56:59.852 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.847+0000 7f00ec8e6700 1 -- 172.21.15.77:0/4140596977 wait complete. 2022-04-03T19:56:59.855 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.848+0000 7f00ec8e6700 1 Processor -- start 2022-04-03T19:56:59.855 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.848+0000 7f00ec8e6700 1 -- start start 2022-04-03T19:56:59.855 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.849+0000 7f00ec8e6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e80c1170 0x7f00e8149d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:59.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.849+0000 7f00ec8e6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00e80c3b10 0x7f00e80bb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:59.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.849+0000 7f00ec8e6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00e8155370 0x7f00e8147f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:59.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.849+0000 7f00ec8e6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e80bb330 con 0x7f00e8155370 2022-04-03T19:56:59.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.849+0000 7f00ec8e6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e8168c90 con 0x7f00e80c1170 2022-04-03T19:56:59.856 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.849+0000 7f00ec8e6700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00e80bb000 con 0x7f00e80c3b10 2022-04-03T19:56:59.857 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.850+0000 7f00e7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00e8155370 0x7f00e8147f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:59.857 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.850+0000 7f00e7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00e8155370 0x7f00e8147f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43202/0 (socket says 172.21.15.77:43202) 2022-04-03T19:56:59.857 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.850+0000 7f00e7fff700 1 -- 172.21.15.77:0/1233100245 learned_addr learned my addr 172.21.15.77:0/1233100245 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:56:59.857 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.850+0000 7f00e6ffd700 1 --2- 172.21.15.77:0/1233100245 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00e80c3b10 0x7f00e80bb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:59.858 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.850+0000 7f00e77fe700 1 --2- 172.21.15.77:0/1233100245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e80c1170 0x7f00e8149d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:59.858 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.850+0000 7f00e7fff700 1 -- 172.21.15.77:0/1233100245 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00e80c3b10 msgr2=0x7f00e80bb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:59.858 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.850+0000 7f00e7fff700 1 --2- 172.21.15.77:0/1233100245 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00e80c3b10 0x7f00e80bb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:59.858 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.850+0000 7f00e7fff700 1 -- 172.21.15.77:0/1233100245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e80c1170 msgr2=0x7f00e8149d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:56:59.859 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.850+0000 7f00e7fff700 1 --2- 172.21.15.77:0/1233100245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e80c1170 0x7f00e8149d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:56:59.859 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.850+0000 7f00e7fff700 1 -- 172.21.15.77:0/1233100245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d80092a0 con 0x7f00e8155370 2022-04-03T19:56:59.859 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.850+0000 7f00e7fff700 1 --2- 172.21.15.77:0/1233100245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00e8155370 0x7f00e8147f50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f00e000c790 tx=0x7f00e00015e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:56:59.859 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.851+0000 7f00cf7fe700 1 -- 172.21.15.77:0/1233100245 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e0005d30 con 0x7f00e8155370 2022-04-03T19:56:59.859 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.851+0000 7f00cf7fe700 1 -- 172.21.15.77:0/1233100245 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e0005e90 con 0x7f00e8155370 2022-04-03T19:56:59.860 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.851+0000 7f00ec8e6700 1 -- 172.21.15.77:0/1233100245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00e80c5160 con 0x7f00e8155370 2022-04-03T19:56:59.860 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.851+0000 7f00cf7fe700 1 -- 172.21.15.77:0/1233100245 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e0007300 con 0x7f00e8155370 2022-04-03T19:56:59.860 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.851+0000 7f00ec8e6700 1 -- 172.21.15.77:0/1233100245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00e8169210 con 0x7f00e8155370 2022-04-03T19:56:59.860 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.852+0000 7f00ec8e6700 1 -- 172.21.15.77:0/1233100245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00e806c290 con 0x7f00e8155370 2022-04-03T19:56:59.860 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.853+0000 7f00cf7fe700 1 -- 172.21.15.77:0/1233100245 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f00e0010070 con 0x7f00e8155370 2022-04-03T19:56:59.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.853+0000 7f00cf7fe700 1 --2- 172.21.15.77:0/1233100245 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f00c8063d10 0x7f00c80661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:56:59.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.854+0000 7f00cf7fe700 1 -- 172.21.15.77:0/1233100245 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f00e0080570 con 0x7f00e8155370 2022-04-03T19:56:59.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.854+0000 7f00e77fe700 1 --2- 172.21.15.77:0/1233100245 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f00c8063d10 0x7f00c80661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:56:59.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.854+0000 7f00e77fe700 1 --2- 172.21.15.77:0/1233100245 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f00c8063d10 0x7f00c80661c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f00d800ffd0 tx=0x7f00d8016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:56:59.862 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:56:59.858+0000 7f00cf7fe700 1 -- 172.21.15.77:0/1233100245 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f00e004ef30 con 0x7f00e8155370 2022-04-03T19:57:00.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:00.088+0000 7f00ec8e6700 1 -- 172.21.15.77:0/1233100245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00e80bb470 con 0x7f00e8155370 2022-04-03T19:57:00.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:00.089+0000 7f00cf7fe700 1 -- 172.21.15.77:0/1233100245 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f00e0052550 con 0x7f00e8155370 2022-04-03T19:57:00.090 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:00.090 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5668,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:38.315236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:active","state_seq":30,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:00.092 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:00.091+0000 7f00ec8e6700 1 -- 172.21.15.77:0/1233100245 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f00c8063d10 msgr2=0x7f00c80661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:00.093 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:00.091+0000 7f00ec8e6700 1 --2- 172.21.15.77:0/1233100245 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f00c8063d10 0x7f00c80661c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f00d800ffd0 tx=0x7f00d8016000 comp rx=0 tx=0).stop 2022-04-03T19:57:00.093 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:00.091+0000 7f00ec8e6700 1 -- 172.21.15.77:0/1233100245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00e8155370 msgr2=0x7f00e8147f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:00.093 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:00.091+0000 7f00ec8e6700 1 --2- 172.21.15.77:0/1233100245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00e8155370 0x7f00e8147f50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f00e000c790 tx=0x7f00e00015e0 comp rx=0 tx=0).stop 2022-04-03T19:57:00.093 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:00.091+0000 7f00ec8e6700 1 -- 172.21.15.77:0/1233100245 shutdown_connections 2022-04-03T19:57:00.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:00.091+0000 7f00ec8e6700 1 --2- 172.21.15.77:0/1233100245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00e8155370 0x7f00e8147f50 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:00.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:00.091+0000 7f00ec8e6700 1 --2- 172.21.15.77:0/1233100245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e80c1170 0x7f00e8149d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:00.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:00.091+0000 7f00ec8e6700 1 --2- 172.21.15.77:0/1233100245 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f00c8063d10 0x7f00c80661c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:00.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:00.091+0000 7f00ec8e6700 1 --2- 172.21.15.77:0/1233100245 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00e80c3b10 0x7f00e80bb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:00.094 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:00.091+0000 7f00ec8e6700 1 -- 172.21.15.77:0/1233100245 >> 172.21.15.77:0/1233100245 conn(0x7f00e8066da0 msgr2=0x7f00e8155780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:00.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:00.092+0000 7f00ec8e6700 1 -- 172.21.15.77:0/1233100245 shutdown_connections 2022-04-03T19:57:00.095 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:00.092+0000 7f00ec8e6700 1 -- 172.21.15.77:0/1233100245 wait complete. 2022-04-03T19:57:00.095 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 43 2022-04-03T19:57:00.116 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:57:02.118 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:02.325 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.324+0000 7f1264991700 1 Processor -- start 2022-04-03T19:57:02.325 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.324+0000 7f1264991700 1 -- start start 2022-04-03T19:57:02.328 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.325+0000 7f1264991700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f126006d3f0 0x7f126006d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:02.329 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.325+0000 7f1264991700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12600c2a90 0x7f126006c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:02.329 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.326+0000 7f1264991700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12601562f0 con 0x7f12600c5930 2022-04-03T19:57:02.329 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.326+0000 7f1264991700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12600c2f80 con 0x7f126006d3f0 2022-04-03T19:57:02.329 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.326+0000 7f1264991700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12600c3260 con 0x7f12600c2a90 2022-04-03T19:57:02.330 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.326+0000 7f125effd700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f12600c5930 0x7f1260155be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38574/0 (socket says 172.21.15.77:38574) 2022-04-03T19:57:02.330 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.326+0000 7f125effd700 1 -- 172.21.15.77:0/2545914834 learned_addr learned my addr 172.21.15.77:0/2545914834 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:57:02.330 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.326+0000 7f125ffff700 1 --2- 172.21.15.77:0/2545914834 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12600c2a90 0x7f126006c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:02.330 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.326+0000 7f125f7fe700 1 --2- 172.21.15.77:0/2545914834 >> v2:172.21.15.77:3300/0 conn(0x7f126006d3f0 0x7f126006d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:02.331 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.326+0000 7f125e7fc700 1 -- 172.21.15.77:0/2545914834 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1289303792 0 0) 0x7f12601562f0 con 0x7f12600c5930 2022-04-03T19:57:02.331 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.326+0000 7f125ffff700 1 -- 172.21.15.77:0/2545914834 >> v2:172.21.15.77:3300/0 conn(0x7f126006d3f0 msgr2=0x7f126006d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:02.331 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.326+0000 7f125ffff700 1 --2- 172.21.15.77:0/2545914834 >> v2:172.21.15.77:3300/0 conn(0x7f126006d3f0 0x7f126006d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:02.331 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.326+0000 7f125ffff700 1 -- 172.21.15.77:0/2545914834 >> v1:172.21.15.77:6789/0 conn(0x7f12600c5930 legacy=0x7f1260155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:02.331 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.326+0000 7f125ffff700 1 -- 172.21.15.77:0/2545914834 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12600c3540 con 0x7f12600c2a90 2022-04-03T19:57:02.332 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.327+0000 7f125ffff700 1 --2- 172.21.15.77:0/2545914834 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12600c2a90 0x7f126006c040 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1258009e90 tx=0x7f1258009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b99cefcaf63b749 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:02.332 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.327+0000 7f125e7fc700 1 -- 172.21.15.77:0/2545914834 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125800e070 con 0x7f12600c2a90 2022-04-03T19:57:02.332 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.327+0000 7f125e7fc700 1 -- 172.21.15.77:0/2545914834 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125800bc80 con 0x7f12600c2a90 2022-04-03T19:57:02.332 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.327+0000 7f125e7fc700 1 -- 172.21.15.77:0/2545914834 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1258012430 con 0x7f12600c2a90 2022-04-03T19:57:02.333 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.328+0000 7f1264991700 1 -- 172.21.15.77:0/2545914834 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12600c2a90 msgr2=0x7f126006c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:02.333 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.328+0000 7f1264991700 1 --2- 172.21.15.77:0/2545914834 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12600c2a90 0x7f126006c040 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1258009e90 tx=0x7f1258009780 comp rx=0 tx=0).stop 2022-04-03T19:57:02.333 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.328+0000 7f1264991700 1 -- 172.21.15.77:0/2545914834 shutdown_connections 2022-04-03T19:57:02.333 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.328+0000 7f1264991700 1 --2- 172.21.15.77:0/2545914834 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12600c2a90 0x7f126006c040 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:02.334 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.328+0000 7f1264991700 1 --2- 172.21.15.77:0/2545914834 >> v2:172.21.15.77:3300/0 conn(0x7f126006d3f0 0x7f126006d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:02.334 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.328+0000 7f1264991700 1 -- 172.21.15.77:0/2545914834 >> 172.21.15.77:0/2545914834 conn(0x7f12600cb720 msgr2=0x7f12600cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:02.334 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.328+0000 7f1264991700 1 -- 172.21.15.77:0/2545914834 shutdown_connections 2022-04-03T19:57:02.334 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.328+0000 7f1264991700 1 -- 172.21.15.77:0/2545914834 wait complete. 2022-04-03T19:57:02.337 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.329+0000 7f1264991700 1 Processor -- start 2022-04-03T19:57:02.338 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.330+0000 7f1264991700 1 -- start start 2022-04-03T19:57:02.338 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.330+0000 7f1264991700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f126006d3f0 0x7f126014e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:02.338 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.330+0000 7f1264991700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12600c2a90 0x7f126014b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:02.338 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.331+0000 7f1264991700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12600c5930 0x7f12601417b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:02.339 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.331+0000 7f1264991700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12600c6570 con 0x7f126006d3f0 2022-04-03T19:57:02.339 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.331+0000 7f1264991700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126014cb20 con 0x7f12600c5930 2022-04-03T19:57:02.339 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.331+0000 7f1264991700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12601662f0 con 0x7f12600c2a90 2022-04-03T19:57:02.339 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.331+0000 7f125f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f126006d3f0 0x7f126014e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:02.339 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.331+0000 7f125effd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12600c2a90 0x7f126014b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:02.340 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.331+0000 7f125effd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12600c2a90 0x7f126014b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60496/0 (socket says 172.21.15.77:60496) 2022-04-03T19:57:02.340 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.331+0000 7f125effd700 1 -- 172.21.15.77:0/2873548027 learned_addr learned my addr 172.21.15.77:0/2873548027 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:02.340 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.331+0000 7f125ffff700 1 --2- 172.21.15.77:0/2873548027 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12600c5930 0x7f12601417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:02.340 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.331+0000 7f125effd700 1 -- 172.21.15.77:0/2873548027 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12600c5930 msgr2=0x7f12601417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:02.341 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.331+0000 7f125effd700 1 --2- 172.21.15.77:0/2873548027 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12600c5930 0x7f12601417b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:02.341 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.331+0000 7f125effd700 1 -- 172.21.15.77:0/2873548027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f126006d3f0 msgr2=0x7f126014e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:02.341 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.331+0000 7f125effd700 1 --2- 172.21.15.77:0/2873548027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f126006d3f0 0x7f126014e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:02.341 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.331+0000 7f125effd700 1 -- 172.21.15.77:0/2873548027 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1258009a20 con 0x7f12600c2a90 2022-04-03T19:57:02.342 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.331+0000 7f125f7fe700 1 --2- 172.21.15.77:0/2873548027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f126006d3f0 0x7f126014e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:57:02.342 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.332+0000 7f125effd700 1 --2- 172.21.15.77:0/2873548027 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12600c2a90 0x7f126014b100 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f125400c470 tx=0x7f1254007d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:02.342 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.332+0000 7f12477fe700 1 -- 172.21.15.77:0/2873548027 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1254007360 con 0x7f12600c2a90 2022-04-03T19:57:02.342 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.332+0000 7f12477fe700 1 -- 172.21.15.77:0/2873548027 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12540074c0 con 0x7f12600c2a90 2022-04-03T19:57:02.342 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.332+0000 7f1264991700 1 -- 172.21.15.77:0/2873548027 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1260166780 con 0x7f12600c2a90 2022-04-03T19:57:02.343 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.332+0000 7f12477fe700 1 -- 172.21.15.77:0/2873548027 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1254004900 con 0x7f12600c2a90 2022-04-03T19:57:02.343 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.332+0000 7f1264991700 1 -- 172.21.15.77:0/2873548027 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f126014cd50 con 0x7f12600c2a90 2022-04-03T19:57:02.343 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.333+0000 7f1264991700 1 -- 172.21.15.77:0/2873548027 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126004b050 con 0x7f12600c2a90 2022-04-03T19:57:02.343 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.334+0000 7f12477fe700 1 -- 172.21.15.77:0/2873548027 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f1254010030 con 0x7f12600c2a90 2022-04-03T19:57:02.344 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.335+0000 7f12477fe700 1 --2- 172.21.15.77:0/2873548027 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1240063ce0 0x7f1240066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:02.344 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.335+0000 7f12477fe700 1 -- 172.21.15.77:0/2873548027 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f12540804c0 con 0x7f12600c2a90 2022-04-03T19:57:02.344 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.335+0000 7f125f7fe700 1 --2- 172.21.15.77:0/2873548027 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1240063ce0 0x7f1240066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:02.344 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.336+0000 7f125f7fe700 1 --2- 172.21.15.77:0/2873548027 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1240063ce0 0x7f1240066190 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1250011310 tx=0x7f1250009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:02.345 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.339+0000 7f12477fe700 1 -- 172.21.15.77:0/2873548027 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f125404eee0 con 0x7f12600c2a90 2022-04-03T19:57:02.569 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.569+0000 7f1264991700 1 -- 172.21.15.77:0/2873548027 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f126006d010 con 0x7f12600c2a90 2022-04-03T19:57:02.570 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.570+0000 7f12477fe700 1 -- 172.21.15.77:0/2873548027 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f1254052500 con 0x7f12600c2a90 2022-04-03T19:57:02.570 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:02.571 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5668,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:38.315236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:active","state_seq":30,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:02.573 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.572+0000 7f1264991700 1 -- 172.21.15.77:0/2873548027 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1240063ce0 msgr2=0x7f1240066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:02.574 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.572+0000 7f1264991700 1 --2- 172.21.15.77:0/2873548027 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1240063ce0 0x7f1240066190 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1250011310 tx=0x7f1250009060 comp rx=0 tx=0).stop 2022-04-03T19:57:02.574 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.572+0000 7f1264991700 1 -- 172.21.15.77:0/2873548027 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12600c2a90 msgr2=0x7f126014b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:02.574 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.572+0000 7f1264991700 1 --2- 172.21.15.77:0/2873548027 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12600c2a90 0x7f126014b100 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f125400c470 tx=0x7f1254007d90 comp rx=0 tx=0).stop 2022-04-03T19:57:02.574 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.572+0000 7f1264991700 1 -- 172.21.15.77:0/2873548027 shutdown_connections 2022-04-03T19:57:02.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.572+0000 7f1264991700 1 --2- 172.21.15.77:0/2873548027 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12600c5930 0x7f12601417b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:02.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.572+0000 7f1264991700 1 --2- 172.21.15.77:0/2873548027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f126006d3f0 0x7f126014e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:02.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.572+0000 7f1264991700 1 --2- 172.21.15.77:0/2873548027 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1240063ce0 0x7f1240066190 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:02.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.572+0000 7f1264991700 1 --2- 172.21.15.77:0/2873548027 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12600c2a90 0x7f126014b100 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:02.575 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.572+0000 7f1264991700 1 -- 172.21.15.77:0/2873548027 >> 172.21.15.77:0/2873548027 conn(0x7f12600cb720 msgr2=0x7f1260147130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:02.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.573+0000 7f1264991700 1 -- 172.21.15.77:0/2873548027 shutdown_connections 2022-04-03T19:57:02.576 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:02.573+0000 7f1264991700 1 -- 172.21.15.77:0/2873548027 wait complete. 2022-04-03T19:57:02.576 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 43 2022-04-03T19:57:02.597 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:57:04.598 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:04.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.803+0000 7f552ffff700 1 Processor -- start 2022-04-03T19:57:04.804 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.803+0000 7f552ffff700 1 -- start start 2022-04-03T19:57:04.806 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.804+0000 7f552ffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f553014ef90 0x7f55300c2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:04.806 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.804+0000 7f552ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f553014f360 0x7f55300c7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:04.806 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.805+0000 7f552ffff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55300c5050 con 0x7f55300c7b70 2022-04-03T19:57:04.806 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.805+0000 7f552ffff700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55300c4330 con 0x7f553014f360 2022-04-03T19:57:04.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.805+0000 7f552ffff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5530155e40 con 0x7f553014ef90 2022-04-03T19:57:04.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.805+0000 7f5527fff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f553014f360 0x7f55300c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:04.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.805+0000 7f552f7fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f55300c7b70 0x7f55300626d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38576/0 (socket says 172.21.15.77:38576) 2022-04-03T19:57:04.807 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.805+0000 7f552f7fe700 1 -- 172.21.15.77:0/2379357675 learned_addr learned my addr 172.21.15.77:0/2379357675 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:57:04.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.805+0000 7f552effd700 1 --2- 172.21.15.77:0/2379357675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f553014ef90 0x7f55300c2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:04.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.805+0000 7f5527fff700 1 -- 172.21.15.77:0/2379357675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f553014ef90 msgr2=0x7f55300c2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:04.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.805+0000 7f5527fff700 1 --2- 172.21.15.77:0/2379357675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f553014ef90 0x7f55300c2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:04.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.805+0000 7f5527fff700 1 -- 172.21.15.77:0/2379357675 >> v1:172.21.15.77:6789/0 conn(0x7f55300c7b70 legacy=0x7f55300626d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:04.808 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.806+0000 7f5527fff700 1 -- 172.21.15.77:0/2379357675 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55301559b0 con 0x7f553014f360 2022-04-03T19:57:04.812 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.806+0000 7f5527fff700 1 --2- 172.21.15.77:0/2379357675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f553014f360 0x7f55300c7630 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5518009760 tx=0x7f5518009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93e9d95c6bd58d81 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:04.812 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.806+0000 7f552e7fc700 1 -- 172.21.15.77:0/2379357675 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5518004030 con 0x7f553014f360 2022-04-03T19:57:04.812 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.806+0000 7f552e7fc700 1 -- 172.21.15.77:0/2379357675 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5518004190 con 0x7f553014f360 2022-04-03T19:57:04.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.806+0000 7f552e7fc700 1 -- 172.21.15.77:0/2379357675 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55180105b0 con 0x7f553014f360 2022-04-03T19:57:04.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.806+0000 7f552ffff700 1 -- 172.21.15.77:0/2379357675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f553014f360 msgr2=0x7f55300c7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:04.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.806+0000 7f552ffff700 1 --2- 172.21.15.77:0/2379357675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f553014f360 0x7f55300c7630 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5518009760 tx=0x7f5518009060 comp rx=0 tx=0).stop 2022-04-03T19:57:04.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.807+0000 7f552ffff700 1 -- 172.21.15.77:0/2379357675 shutdown_connections 2022-04-03T19:57:04.813 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.807+0000 7f552ffff700 1 --2- 172.21.15.77:0/2379357675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f553014ef90 0x7f55300c2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:04.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.807+0000 7f552ffff700 1 --2- 172.21.15.77:0/2379357675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f553014f360 0x7f55300c7630 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:04.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.807+0000 7f552ffff700 1 -- 172.21.15.77:0/2379357675 >> 172.21.15.77:0/2379357675 conn(0x7f553006e340 msgr2=0x7f55300677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:04.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.807+0000 7f552ffff700 1 -- 172.21.15.77:0/2379357675 shutdown_connections 2022-04-03T19:57:04.814 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.807+0000 7f552ffff700 1 -- 172.21.15.77:0/2379357675 wait complete. 2022-04-03T19:57:04.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.808+0000 7f552ffff700 1 Processor -- start 2022-04-03T19:57:04.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.808+0000 7f552ffff700 1 -- start start 2022-04-03T19:57:04.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.809+0000 7f552ffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55300c7b70 0x7f5530152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:04.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.809+0000 7f552ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f553014ef90 0x7f5530162c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:04.815 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.809+0000 7f552ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f553014f360 0x7f5530164c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:04.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.809+0000 7f552ffff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5530151f50 con 0x7f553014f360 2022-04-03T19:57:04.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.809+0000 7f552ffff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55300bd9b0 con 0x7f553014ef90 2022-04-03T19:57:04.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.809+0000 7f552ffff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55300bcac0 con 0x7f55300c7b70 2022-04-03T19:57:04.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.810+0000 7f552f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f553014f360 0x7f5530164c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:04.816 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.810+0000 7f552effd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55300c7b70 0x7f5530152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:04.817 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.810+0000 7f552effd700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55300c7b70 0x7f5530152810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60500/0 (socket says 172.21.15.77:60500) 2022-04-03T19:57:04.817 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.810+0000 7f552effd700 1 -- 172.21.15.77:0/2794993612 learned_addr learned my addr 172.21.15.77:0/2794993612 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:04.817 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.810+0000 7f5527fff700 1 --2- 172.21.15.77:0/2794993612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f553014ef90 0x7f5530162c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:04.817 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.810+0000 7f552f7fe700 1 -- 172.21.15.77:0/2794993612 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55300c7b70 msgr2=0x7f5530152810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:04.818 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.810+0000 7f552f7fe700 1 --2- 172.21.15.77:0/2794993612 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55300c7b70 0x7f5530152810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:04.818 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.810+0000 7f552f7fe700 1 -- 172.21.15.77:0/2794993612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f553014ef90 msgr2=0x7f5530162c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:04.818 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.810+0000 7f552f7fe700 1 --2- 172.21.15.77:0/2794993612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f553014ef90 0x7f5530162c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:04.818 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.810+0000 7f552f7fe700 1 -- 172.21.15.77:0/2794993612 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55180092a0 con 0x7f553014f360 2022-04-03T19:57:04.818 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.810+0000 7f552f7fe700 1 --2- 172.21.15.77:0/2794993612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f553014f360 0x7f5530164c40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f552800c8c0 tx=0x7f5528001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:04.819 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.811+0000 7f5526ffd700 1 -- 172.21.15.77:0/2794993612 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5528004c40 con 0x7f553014f360 2022-04-03T19:57:04.819 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.811+0000 7f5526ffd700 1 -- 172.21.15.77:0/2794993612 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5528004da0 con 0x7f553014f360 2022-04-03T19:57:04.819 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.811+0000 7f552ffff700 1 -- 172.21.15.77:0/2794993612 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55300bcda0 con 0x7f553014f360 2022-04-03T19:57:04.819 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.811+0000 7f5526ffd700 1 -- 172.21.15.77:0/2794993612 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55280073a0 con 0x7f553014f360 2022-04-03T19:57:04.820 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.811+0000 7f552ffff700 1 -- 172.21.15.77:0/2794993612 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55300bd360 con 0x7f553014f360 2022-04-03T19:57:04.821 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.812+0000 7f552ffff700 1 -- 172.21.15.77:0/2794993612 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5530068ab0 con 0x7f553014f360 2022-04-03T19:57:04.821 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.813+0000 7f5526ffd700 1 -- 172.21.15.77:0/2794993612 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f552801a030 con 0x7f553014f360 2022-04-03T19:57:04.821 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.814+0000 7f5526ffd700 1 --2- 172.21.15.77:0/2794993612 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5514063ce0 0x7f5514066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:04.821 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.814+0000 7f5526ffd700 1 -- 172.21.15.77:0/2794993612 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f552807ff20 con 0x7f553014f360 2022-04-03T19:57:04.822 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.814+0000 7f552effd700 1 --2- 172.21.15.77:0/2794993612 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5514063ce0 0x7f5514066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:04.822 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.815+0000 7f552effd700 1 --2- 172.21.15.77:0/2794993612 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5514063ce0 0x7f5514066190 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5520009cf0 tx=0x7f5520009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:04.822 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:04.818+0000 7f5526ffd700 1 -- 172.21.15.77:0/2794993612 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f552804e8e0 con 0x7f553014f360 2022-04-03T19:57:05.049 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:05.049+0000 7f552ffff700 1 -- 172.21.15.77:0/2794993612 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f553004b050 con 0x7f553014f360 2022-04-03T19:57:05.050 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:05.049+0000 7f5526ffd700 1 -- 172.21.15.77:0/2794993612 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f5528018020 con 0x7f553014f360 2022-04-03T19:57:05.050 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:05.051 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5668,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:38.315236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:active","state_seq":30,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:05.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:05.052+0000 7f552ffff700 1 -- 172.21.15.77:0/2794993612 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5514063ce0 msgr2=0x7f5514066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:05.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:05.052+0000 7f552ffff700 1 --2- 172.21.15.77:0/2794993612 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5514063ce0 0x7f5514066190 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5520009cf0 tx=0x7f5520009060 comp rx=0 tx=0).stop 2022-04-03T19:57:05.053 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:05.052+0000 7f552ffff700 1 -- 172.21.15.77:0/2794993612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f553014f360 msgr2=0x7f5530164c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:05.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:05.052+0000 7f552ffff700 1 --2- 172.21.15.77:0/2794993612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f553014f360 0x7f5530164c40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f552800c8c0 tx=0x7f5528001630 comp rx=0 tx=0).stop 2022-04-03T19:57:05.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:05.052+0000 7f552ffff700 1 -- 172.21.15.77:0/2794993612 shutdown_connections 2022-04-03T19:57:05.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:05.052+0000 7f552ffff700 1 --2- 172.21.15.77:0/2794993612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f553014f360 0x7f5530164c40 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:05.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:05.052+0000 7f552ffff700 1 --2- 172.21.15.77:0/2794993612 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5514063ce0 0x7f5514066190 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:05.054 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:05.052+0000 7f552ffff700 1 --2- 172.21.15.77:0/2794993612 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55300c7b70 0x7f5530152810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:05.055 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:05.052+0000 7f552ffff700 1 --2- 172.21.15.77:0/2794993612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f553014ef90 0x7f5530162c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:05.055 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:05.052+0000 7f552ffff700 1 -- 172.21.15.77:0/2794993612 >> 172.21.15.77:0/2794993612 conn(0x7f553006e340 msgr2=0x7f55301546d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:05.055 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:05.052+0000 7f552ffff700 1 -- 172.21.15.77:0/2794993612 shutdown_connections 2022-04-03T19:57:05.055 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:05.052+0000 7f552ffff700 1 -- 172.21.15.77:0/2794993612 wait complete. 2022-04-03T19:57:05.056 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 43 2022-04-03T19:57:05.077 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:57:07.078 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:07.284 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.283+0000 7fa38759e700 1 Processor -- start 2022-04-03T19:57:07.284 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.283+0000 7fa38759e700 1 -- start start 2022-04-03T19:57:07.286 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.285+0000 7fa38759e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa388155370 0x7fa388069110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:07.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.285+0000 7fa38759e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3880c3b10 0x7fa388069650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:07.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.285+0000 7fa38759e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3880c3710 con 0x7fa3880c1170 2022-04-03T19:57:07.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.285+0000 7fa38759e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa388152330 con 0x7fa388155370 2022-04-03T19:57:07.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.285+0000 7fa38759e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3880c1540 con 0x7fa3880c3b10 2022-04-03T19:57:07.287 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.285+0000 7fa37ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa388155370 0x7fa388069110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:07.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.285+0000 7fa386d9d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3880c3b10 0x7fa388069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:07.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.285+0000 7fa37ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa388155370 0x7fa388069110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43214/0 (socket says 172.21.15.77:43214) 2022-04-03T19:57:07.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.285+0000 7fa37ffff700 1 -- 172.21.15.77:0/624050988 learned_addr learned my addr 172.21.15.77:0/624050988 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:07.288 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.285+0000 7fa38659c700 1 --1- 172.21.15.77:0/624050988 >> v1:172.21.15.186:6789/0 conn(0x7fa3880c1170 0x7fa3881527c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44374/0 (socket says 172.21.15.77:44374) 2022-04-03T19:57:07.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.285+0000 7fa37ffff700 1 -- 172.21.15.77:0/624050988 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3880c3b10 msgr2=0x7fa388069650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:07.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.285+0000 7fa37ffff700 1 --2- 172.21.15.77:0/624050988 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3880c3b10 0x7fa388069650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:07.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.285+0000 7fa37ffff700 1 -- 172.21.15.77:0/624050988 >> v1:172.21.15.186:6789/0 conn(0x7fa3880c1170 legacy=0x7fa3881527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:07.289 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.285+0000 7fa37ffff700 1 -- 172.21.15.77:0/624050988 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3880c4270 con 0x7fa388155370 2022-04-03T19:57:07.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.286+0000 7fa37ffff700 1 --2- 172.21.15.77:0/624050988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa388155370 0x7fa388069110 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa370006fd0 tx=0x7fa3700090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59bc74cd60404e8c server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:07.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.286+0000 7fa385d9b700 1 -- 172.21.15.77:0/624050988 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa370004030 con 0x7fa388155370 2022-04-03T19:57:07.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.286+0000 7fa385d9b700 1 -- 172.21.15.77:0/624050988 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3700041d0 con 0x7fa388155370 2022-04-03T19:57:07.290 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.286+0000 7fa385d9b700 1 -- 172.21.15.77:0/624050988 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa370011630 con 0x7fa388155370 2022-04-03T19:57:07.293 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.286+0000 7fa38759e700 1 -- 172.21.15.77:0/624050988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa388155370 msgr2=0x7fa388069110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:07.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.286+0000 7fa38759e700 1 --2- 172.21.15.77:0/624050988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa388155370 0x7fa388069110 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa370006fd0 tx=0x7fa3700090f0 comp rx=0 tx=0).stop 2022-04-03T19:57:07.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.287+0000 7fa38759e700 1 -- 172.21.15.77:0/624050988 shutdown_connections 2022-04-03T19:57:07.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.287+0000 7fa38759e700 1 --2- 172.21.15.77:0/624050988 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3880c3b10 0x7fa388069650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:07.294 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.287+0000 7fa38759e700 1 --2- 172.21.15.77:0/624050988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa388155370 0x7fa388069110 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:07.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.287+0000 7fa38759e700 1 -- 172.21.15.77:0/624050988 >> 172.21.15.77:0/624050988 conn(0x7fa388066da0 msgr2=0x7fa3880671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:07.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.287+0000 7fa38759e700 1 -- 172.21.15.77:0/624050988 shutdown_connections 2022-04-03T19:57:07.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.287+0000 7fa38759e700 1 -- 172.21.15.77:0/624050988 wait complete. 2022-04-03T19:57:07.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.288+0000 7fa38759e700 1 Processor -- start 2022-04-03T19:57:07.295 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.288+0000 7fa38759e700 1 -- start start 2022-04-03T19:57:07.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.289+0000 7fa38759e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3880c1170 0x7fa3880bd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:07.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.289+0000 7fa38759e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3880c3b10 0x7fa388144f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:07.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.289+0000 7fa38759e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa388155370 0x7fa38814e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:07.296 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.289+0000 7fa38759e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3880bc5a0 con 0x7fa3880c3b10 2022-04-03T19:57:07.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.289+0000 7fa38759e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3880c1f20 con 0x7fa388155370 2022-04-03T19:57:07.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.289+0000 7fa38759e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3880c2110 con 0x7fa3880c1170 2022-04-03T19:57:07.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.289+0000 7fa38659c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3880c1170 0x7fa3880bd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:07.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.289+0000 7fa386d9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa388155370 0x7fa38814e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:07.297 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.289+0000 7fa38659c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3880c1170 0x7fa3880bd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60504/0 (socket says 172.21.15.77:60504) 2022-04-03T19:57:07.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.289+0000 7fa38659c700 1 -- 172.21.15.77:0/2358502892 learned_addr learned my addr 172.21.15.77:0/2358502892 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:07.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.289+0000 7fa37ffff700 1 --2- 172.21.15.77:0/2358502892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3880c3b10 0x7fa388144f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:07.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.290+0000 7fa38659c700 1 -- 172.21.15.77:0/2358502892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa388155370 msgr2=0x7fa38814e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:07.298 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.290+0000 7fa38659c700 1 --2- 172.21.15.77:0/2358502892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa388155370 0x7fa38814e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:07.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.290+0000 7fa38659c700 1 -- 172.21.15.77:0/2358502892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3880c3b10 msgr2=0x7fa388144f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:07.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.290+0000 7fa38659c700 1 --2- 172.21.15.77:0/2358502892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3880c3b10 0x7fa388144f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:07.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.290+0000 7fa38659c700 1 -- 172.21.15.77:0/2358502892 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa370009370 con 0x7fa3880c1170 2022-04-03T19:57:07.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.290+0000 7fa37ffff700 1 --2- 172.21.15.77:0/2358502892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3880c3b10 0x7fa388144f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:57:07.299 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.290+0000 7fa386d9d700 1 --2- 172.21.15.77:0/2358502892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa388155370 0x7fa38814e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:57:07.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.290+0000 7fa38659c700 1 --2- 172.21.15.77:0/2358502892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3880c1170 0x7fa3880bd740 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa37800c490 tx=0x7fa378007df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:07.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.290+0000 7fa37e7fc700 1 -- 172.21.15.77:0/2358502892 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37800ce00 con 0x7fa3880c1170 2022-04-03T19:57:07.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.291+0000 7fa37e7fc700 1 -- 172.21.15.77:0/2358502892 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3780072e0 con 0x7fa3880c1170 2022-04-03T19:57:07.300 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.291+0000 7fa38759e700 1 -- 172.21.15.77:0/2358502892 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3880bb0a0 con 0x7fa3880c1170 2022-04-03T19:57:07.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.291+0000 7fa37e7fc700 1 -- 172.21.15.77:0/2358502892 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3780046b0 con 0x7fa3880c1170 2022-04-03T19:57:07.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.291+0000 7fa38759e700 1 -- 172.21.15.77:0/2358502892 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3880c7a30 con 0x7fa3880c1170 2022-04-03T19:57:07.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.292+0000 7fa38759e700 1 -- 172.21.15.77:0/2358502892 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3880712c0 con 0x7fa3880c1170 2022-04-03T19:57:07.301 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.293+0000 7fa37e7fc700 1 -- 172.21.15.77:0/2358502892 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fa378010070 con 0x7fa3880c1170 2022-04-03T19:57:07.302 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.294+0000 7fa37e7fc700 1 --2- 172.21.15.77:0/2358502892 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa36c063ce0 0x7fa36c066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:07.303 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.294+0000 7fa37e7fc700 1 -- 172.21.15.77:0/2358502892 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7fa37807fb00 con 0x7fa3880c1170 2022-04-03T19:57:07.303 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.294+0000 7fa37ffff700 1 --2- 172.21.15.77:0/2358502892 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa36c063ce0 0x7fa36c066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:07.303 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.294+0000 7fa37ffff700 1 --2- 172.21.15.77:0/2358502892 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa36c063ce0 0x7fa36c066190 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa370009340 tx=0x7fa370016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:07.303 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.298+0000 7fa37e7fc700 1 -- 172.21.15.77:0/2358502892 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fa378048ae0 con 0x7fa3880c1170 2022-04-03T19:57:07.530 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.530+0000 7fa38759e700 1 -- 172.21.15.77:0/2358502892 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa38804b050 con 0x7fa3880c1170 2022-04-03T19:57:07.531 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.531+0000 7fa37e7fc700 1 -- 172.21.15.77:0/2358502892 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6110 (secure 0 0 0) 0x7fa37804e4a0 con 0x7fa3880c1170 2022-04-03T19:57:07.532 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:07.532 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5668,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:38.315236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:active","state_seq":30,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:07.534 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.534+0000 7fa38759e700 1 -- 172.21.15.77:0/2358502892 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa36c063ce0 msgr2=0x7fa36c066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:07.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.534+0000 7fa38759e700 1 --2- 172.21.15.77:0/2358502892 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa36c063ce0 0x7fa36c066190 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa370009340 tx=0x7fa370016000 comp rx=0 tx=0).stop 2022-04-03T19:57:07.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.534+0000 7fa38759e700 1 -- 172.21.15.77:0/2358502892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3880c1170 msgr2=0x7fa3880bd740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:07.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.534+0000 7fa38759e700 1 --2- 172.21.15.77:0/2358502892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3880c1170 0x7fa3880bd740 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa37800c490 tx=0x7fa378007df0 comp rx=0 tx=0).stop 2022-04-03T19:57:07.535 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.534+0000 7fa38759e700 1 -- 172.21.15.77:0/2358502892 shutdown_connections 2022-04-03T19:57:07.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.534+0000 7fa38759e700 1 --2- 172.21.15.77:0/2358502892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa388155370 0x7fa38814e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:07.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.534+0000 7fa38759e700 1 --2- 172.21.15.77:0/2358502892 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fa36c063ce0 0x7fa36c066190 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:07.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.534+0000 7fa38759e700 1 --2- 172.21.15.77:0/2358502892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3880c1170 0x7fa3880bd740 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:07.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.534+0000 7fa38759e700 1 --2- 172.21.15.77:0/2358502892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3880c3b10 0x7fa388144f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:07.536 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.534+0000 7fa38759e700 1 -- 172.21.15.77:0/2358502892 >> 172.21.15.77:0/2358502892 conn(0x7fa388066da0 msgr2=0x7fa388067170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:07.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.534+0000 7fa38759e700 1 -- 172.21.15.77:0/2358502892 shutdown_connections 2022-04-03T19:57:07.537 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:07.534+0000 7fa38759e700 1 -- 172.21.15.77:0/2358502892 wait complete. 2022-04-03T19:57:07.537 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 43 2022-04-03T19:57:07.559 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:57:09.561 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:09.768 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.767+0000 7f72c641d700 1 Processor -- start 2022-04-03T19:57:09.769 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.767+0000 7f72c641d700 1 -- start start 2022-04-03T19:57:09.779 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.768+0000 7f72c641d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72c00c7610 0x7f72c01559d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:09.780 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.768+0000 7f72c641d700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f72c006cfe0 0x7f72c00c2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:09.780 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.768+0000 7f72c641d700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72c01584e0 con 0x7f72c00c29b0 2022-04-03T19:57:09.780 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.768+0000 7f72c641d700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72c014f820 con 0x7f72c006cfe0 2022-04-03T19:57:09.781 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.768+0000 7f72c641d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c0155f10 con 0x7f72c00c7610 2022-04-03T19:57:09.781 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.769+0000 7f72c541b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72c00c7610 0x7f72c01559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:09.781 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.769+0000 7f72c5c1c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f72c006cfe0 0x7f72c00c2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:09.781 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.769+0000 7f72c5c1c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f72c006cfe0 0x7f72c00c2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38400/0 (socket says 172.21.15.77:38400) 2022-04-03T19:57:09.781 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.769+0000 7f72c541b700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72c00c7610 0x7f72c01559d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60506/0 (socket says 172.21.15.77:60506) 2022-04-03T19:57:09.782 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.769+0000 7f72c5c1c700 1 -- 172.21.15.77:0/2175738522 learned_addr learned my addr 172.21.15.77:0/2175738522 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:09.782 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.769+0000 7f72c4c1a700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f72c00c29b0 0x7f72c006c8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44376/0 (socket says 172.21.15.77:44376) 2022-04-03T19:57:09.782 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.769+0000 7f72c5c1c700 1 -- 172.21.15.77:0/2175738522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72c00c7610 msgr2=0x7f72c01559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:09.782 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.769+0000 7f72c5c1c700 1 --2- 172.21.15.77:0/2175738522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72c00c7610 0x7f72c01559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:09.783 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.769+0000 7f72c5c1c700 1 -- 172.21.15.77:0/2175738522 >> v1:172.21.15.186:6789/0 conn(0x7f72c00c29b0 legacy=0x7f72c006c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:09.783 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.770+0000 7f72c5c1c700 1 -- 172.21.15.77:0/2175738522 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c0159340 con 0x7f72c006cfe0 2022-04-03T19:57:09.783 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.770+0000 7f72c541b700 1 --2- 172.21.15.77:0/2175738522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72c00c7610 0x7f72c01559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:57:09.783 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.769+0000 7f72affff700 1 -- 172.21.15.77:0/2175738522 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584612337 0 0) 0x7f72c01584e0 con 0x7f72c00c29b0 2022-04-03T19:57:09.784 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.770+0000 7f72c5c1c700 1 --2- 172.21.15.77:0/2175738522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72c006cfe0 0x7f72c00c2e20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f72b8009f20 tx=0x7f72b8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ec7cd97e3f8003a server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:09.784 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.770+0000 7f72affff700 1 -- 172.21.15.77:0/2175738522 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b800e070 con 0x7f72c006cfe0 2022-04-03T19:57:09.784 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.770+0000 7f72affff700 1 -- 172.21.15.77:0/2175738522 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b800ba60 con 0x7f72c006cfe0 2022-04-03T19:57:09.784 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.770+0000 7f72affff700 1 -- 172.21.15.77:0/2175738522 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b80092e0 con 0x7f72c006cfe0 2022-04-03T19:57:09.784 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.771+0000 7f72c641d700 1 -- 172.21.15.77:0/2175738522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72c006cfe0 msgr2=0x7f72c00c2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:09.785 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.771+0000 7f72c641d700 1 --2- 172.21.15.77:0/2175738522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72c006cfe0 0x7f72c00c2e20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f72b8009f20 tx=0x7f72b8009840 comp rx=0 tx=0).stop 2022-04-03T19:57:09.785 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.771+0000 7f72c641d700 1 -- 172.21.15.77:0/2175738522 shutdown_connections 2022-04-03T19:57:09.785 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.771+0000 7f72c641d700 1 --2- 172.21.15.77:0/2175738522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72c006cfe0 0x7f72c00c2e20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:09.785 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.771+0000 7f72c641d700 1 --2- 172.21.15.77:0/2175738522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72c00c7610 0x7f72c01559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:09.786 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.771+0000 7f72c641d700 1 -- 172.21.15.77:0/2175738522 >> 172.21.15.77:0/2175738522 conn(0x7f72c00cb640 msgr2=0x7f72c00cba40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:09.786 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.771+0000 7f72c641d700 1 -- 172.21.15.77:0/2175738522 shutdown_connections 2022-04-03T19:57:09.786 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.771+0000 7f72c641d700 1 -- 172.21.15.77:0/2175738522 wait complete. 2022-04-03T19:57:09.786 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.772+0000 7f72c641d700 1 Processor -- start 2022-04-03T19:57:09.786 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.773+0000 7f72c641d700 1 -- start start 2022-04-03T19:57:09.787 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.773+0000 7f72c641d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72c006cfe0 0x7f72c00cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:09.787 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.773+0000 7f72c641d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72c00c29b0 0x7f72c0154000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:09.787 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.773+0000 7f72c641d700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72c00c7610 0x7f72c0151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:09.787 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.773+0000 7f72c641d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c01537d0 con 0x7f72c006cfe0 2022-04-03T19:57:09.788 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.773+0000 7f72c641d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c0165840 con 0x7f72c00c29b0 2022-04-03T19:57:09.788 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.773+0000 7f72c641d700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c0165cd0 con 0x7f72c00c7610 2022-04-03T19:57:09.788 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.774+0000 7f72c541b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72c006cfe0 0x7f72c00cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:09.788 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.774+0000 7f72c5c1c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72c00c7610 0x7f72c0151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:09.788 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.774+0000 7f72c541b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72c006cfe0 0x7f72c00cbbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43218/0 (socket says 172.21.15.77:43218) 2022-04-03T19:57:09.789 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.774+0000 7f72c541b700 1 -- 172.21.15.77:0/1323487399 learned_addr learned my addr 172.21.15.77:0/1323487399 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:09.789 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.774+0000 7f72c4c1a700 1 --2- 172.21.15.77:0/1323487399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72c00c29b0 0x7f72c0154000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:09.789 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.774+0000 7f72c541b700 1 -- 172.21.15.77:0/1323487399 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72c00c7610 msgr2=0x7f72c0151da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:09.789 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.774+0000 7f72c541b700 1 --2- 172.21.15.77:0/1323487399 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72c00c7610 0x7f72c0151da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:09.790 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.774+0000 7f72c541b700 1 -- 172.21.15.77:0/1323487399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72c00c29b0 msgr2=0x7f72c0154000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:09.790 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.774+0000 7f72c541b700 1 --2- 172.21.15.77:0/1323487399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72c00c29b0 0x7f72c0154000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:09.790 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.774+0000 7f72c541b700 1 -- 172.21.15.77:0/1323487399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72bc0092a0 con 0x7f72c006cfe0 2022-04-03T19:57:09.790 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.774+0000 7f72c541b700 1 --2- 172.21.15.77:0/1323487399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72c006cfe0 0x7f72c00cbbd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f72bc00e980 tx=0x7f72bc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:09.790 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.775+0000 7f72ad7fa700 1 -- 172.21.15.77:0/1323487399 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72bc013680 con 0x7f72c006cfe0 2022-04-03T19:57:09.791 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.775+0000 7f72ad7fa700 1 -- 172.21.15.77:0/1323487399 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72bc0137e0 con 0x7f72c006cfe0 2022-04-03T19:57:09.791 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.775+0000 7f72c641d700 1 -- 172.21.15.77:0/1323487399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b8009ae0 con 0x7f72c006cfe0 2022-04-03T19:57:09.791 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.775+0000 7f72ad7fa700 1 -- 172.21.15.77:0/1323487399 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72bc013b60 con 0x7f72c006cfe0 2022-04-03T19:57:09.791 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.775+0000 7f72c641d700 1 -- 172.21.15.77:0/1323487399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c0164f10 con 0x7f72c006cfe0 2022-04-03T19:57:09.792 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.776+0000 7f72c641d700 1 -- 172.21.15.77:0/1323487399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72c0088830 con 0x7f72c006cfe0 2022-04-03T19:57:09.792 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.777+0000 7f72ad7fa700 1 -- 172.21.15.77:0/1323487399 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f72bc019030 con 0x7f72c006cfe0 2022-04-03T19:57:09.792 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.777+0000 7f72ad7fa700 1 --2- 172.21.15.77:0/1323487399 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f72a0063ce0 0x7f72a0066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:09.792 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.777+0000 7f72ad7fa700 1 -- 172.21.15.77:0/1323487399 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f72bc080ae0 con 0x7f72c006cfe0 2022-04-03T19:57:09.792 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.778+0000 7f72c4c1a700 1 --2- 172.21.15.77:0/1323487399 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f72a0063ce0 0x7f72a0066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:09.793 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.778+0000 7f72c4c1a700 1 --2- 172.21.15.77:0/1323487399 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f72a0063ce0 0x7f72a0066190 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f72c00c28d0 tx=0x7f72b0007bb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:09.793 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:09.782+0000 7f72ad7fa700 1 -- 172.21.15.77:0/1323487399 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f72bc04f390 con 0x7f72c006cfe0 2022-04-03T19:57:10.014 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:10.014+0000 7f72c641d700 1 -- 172.21.15.77:0/1323487399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72c0165b20 con 0x7f72c006cfe0 2022-04-03T19:57:10.015 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:10.014+0000 7f72ad7fa700 1 -- 172.21.15.77:0/1323487399 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f72bc0529b0 con 0x7f72c006cfe0 2022-04-03T19:57:10.015 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:10.016 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5668,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:38.315236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:active","state_seq":30,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:10.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:10.017+0000 7f72c641d700 1 -- 172.21.15.77:0/1323487399 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f72a0063ce0 msgr2=0x7f72a0066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:10.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:10.017+0000 7f72c641d700 1 --2- 172.21.15.77:0/1323487399 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f72a0063ce0 0x7f72a0066190 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f72c00c28d0 tx=0x7f72b0007bb0 comp rx=0 tx=0).stop 2022-04-03T19:57:10.018 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:10.017+0000 7f72c641d700 1 -- 172.21.15.77:0/1323487399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72c006cfe0 msgr2=0x7f72c00cbbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:10.019 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:10.017+0000 7f72c641d700 1 --2- 172.21.15.77:0/1323487399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72c006cfe0 0x7f72c00cbbd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f72bc00e980 tx=0x7f72bc009060 comp rx=0 tx=0).stop 2022-04-03T19:57:10.019 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:10.017+0000 7f72c641d700 1 -- 172.21.15.77:0/1323487399 shutdown_connections 2022-04-03T19:57:10.019 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:10.017+0000 7f72c641d700 1 --2- 172.21.15.77:0/1323487399 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f72a0063ce0 0x7f72a0066190 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:10.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:10.017+0000 7f72c641d700 1 --2- 172.21.15.77:0/1323487399 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72c00c7610 0x7f72c0151da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:10.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:10.017+0000 7f72c641d700 1 --2- 172.21.15.77:0/1323487399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72c006cfe0 0x7f72c00cbbd0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:10.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:10.017+0000 7f72c641d700 1 --2- 172.21.15.77:0/1323487399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72c00c29b0 0x7f72c0154000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:10.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:10.017+0000 7f72c641d700 1 -- 172.21.15.77:0/1323487399 >> 172.21.15.77:0/1323487399 conn(0x7f72c00cb640 msgr2=0x7f72c00c9e70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:10.020 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:10.017+0000 7f72c641d700 1 -- 172.21.15.77:0/1323487399 shutdown_connections 2022-04-03T19:57:10.021 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:10.017+0000 7f72c641d700 1 -- 172.21.15.77:0/1323487399 wait complete. 2022-04-03T19:57:10.021 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 43 2022-04-03T19:57:10.042 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:57:12.043 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:12.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.247+0000 7f5f2e303700 1 Processor -- start 2022-04-03T19:57:12.248 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.247+0000 7f5f2e303700 1 -- start start 2022-04-03T19:57:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.248+0000 7f5f2e303700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f2806bfb0 0x7f5f281584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:12.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.248+0000 7f5f2e303700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5f2806c500 0x7f5f2814eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:12.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.248+0000 7f5f2e303700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f2806d260 con 0x7f5f2806bbe0 2022-04-03T19:57:12.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.248+0000 7f5f2e303700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f28159450 con 0x7f5f2806bfb0 2022-04-03T19:57:12.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.248+0000 7f5f2e303700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f280c6330 con 0x7f5f2806c500 2022-04-03T19:57:12.251 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.249+0000 7f5f2cb00700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f2806bfb0 0x7f5f281584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:12.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.249+0000 7f5f2cb00700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5f2806bfb0 0x7f5f281584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43220/0 (socket says 172.21.15.77:43220) 2022-04-03T19:57:12.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.249+0000 7f5f2cb00700 1 -- 172.21.15.77:0/1699735180 learned_addr learned my addr 172.21.15.77:0/1699735180 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:12.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.249+0000 7f5f2db02700 1 --2- 172.21.15.77:0/1699735180 >> v2:172.21.15.186:3300/0 conn(0x7f5f2806c500 0x7f5f2814eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:12.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.249+0000 7f5f2cb00700 1 -- 172.21.15.77:0/1699735180 >> v2:172.21.15.186:3300/0 conn(0x7f5f2806c500 msgr2=0x7f5f2814eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:12.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.249+0000 7f5f2cb00700 1 --2- 172.21.15.77:0/1699735180 >> v2:172.21.15.186:3300/0 conn(0x7f5f2806c500 0x7f5f2814eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:12.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.249+0000 7f5f2cb00700 1 -- 172.21.15.77:0/1699735180 >> v1:172.21.15.186:6789/0 conn(0x7f5f2806bbe0 legacy=0x7f5f2806d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:12.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.249+0000 7f5f2cb00700 1 -- 172.21.15.77:0/1699735180 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f28158bb0 con 0x7f5f2806bfb0 2022-04-03T19:57:12.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.249+0000 7f5f2cb00700 1 --2- 172.21.15.77:0/1699735180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f2806bfb0 0x7f5f281584b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5f280cc6b0 tx=0x7f5f180090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94501f015f091e9c server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:12.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.250+0000 7f5f17fff700 1 -- 172.21.15.77:0/1699735180 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f18004030 con 0x7f5f2806bfb0 2022-04-03T19:57:12.254 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.250+0000 7f5f17fff700 1 -- 172.21.15.77:0/1699735180 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f18004190 con 0x7f5f2806bfb0 2022-04-03T19:57:12.254 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.250+0000 7f5f17fff700 1 -- 172.21.15.77:0/1699735180 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f18011630 con 0x7f5f2806bfb0 2022-04-03T19:57:12.257 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.250+0000 7f5f2e303700 1 -- 172.21.15.77:0/1699735180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f2806bfb0 msgr2=0x7f5f281584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:12.257 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.250+0000 7f5f2e303700 1 --2- 172.21.15.77:0/1699735180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f2806bfb0 0x7f5f281584b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5f280cc6b0 tx=0x7f5f180090f0 comp rx=0 tx=0).stop 2022-04-03T19:57:12.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.250+0000 7f5f2e303700 1 -- 172.21.15.77:0/1699735180 shutdown_connections 2022-04-03T19:57:12.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.250+0000 7f5f2e303700 1 --2- 172.21.15.77:0/1699735180 >> v2:172.21.15.186:3300/0 conn(0x7f5f2806c500 0x7f5f2814eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:12.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.250+0000 7f5f2e303700 1 --2- 172.21.15.77:0/1699735180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f2806bfb0 0x7f5f281584b0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:12.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.250+0000 7f5f2e303700 1 -- 172.21.15.77:0/1699735180 >> 172.21.15.77:0/1699735180 conn(0x7f5f280cb6b0 msgr2=0x7f5f280cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:12.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.250+0000 7f5f2e303700 1 -- 172.21.15.77:0/1699735180 shutdown_connections 2022-04-03T19:57:12.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.250+0000 7f5f2e303700 1 -- 172.21.15.77:0/1699735180 wait complete. 2022-04-03T19:57:12.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.252+0000 7f5f2e303700 1 Processor -- start 2022-04-03T19:57:12.259 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.252+0000 7f5f2e303700 1 -- start start 2022-04-03T19:57:12.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.253+0000 7f5f2e303700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f2806bbe0 0x7f5f280ca650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:12.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.253+0000 7f5f2e303700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f2806bfb0 0x7f5f28153a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:12.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.253+0000 7f5f2e303700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f2806c500 0x7f5f28153f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:12.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.253+0000 7f5f2e303700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f28164e40 con 0x7f5f2806bbe0 2022-04-03T19:57:12.260 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.253+0000 7f5f2e303700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f28165050 con 0x7f5f2806bfb0 2022-04-03T19:57:12.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.253+0000 7f5f2e303700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f28165330 con 0x7f5f2806c500 2022-04-03T19:57:12.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.253+0000 7f5f2d301700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f2806bbe0 0x7f5f280ca650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:12.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.253+0000 7f5f2d301700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f2806bbe0 0x7f5f280ca650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43222/0 (socket says 172.21.15.77:43222) 2022-04-03T19:57:12.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.253+0000 7f5f2d301700 1 -- 172.21.15.77:0/247694445 learned_addr learned my addr 172.21.15.77:0/247694445 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:12.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.253+0000 7f5f2cb00700 1 --2- 172.21.15.77:0/247694445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f2806bfb0 0x7f5f28153a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:12.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.253+0000 7f5f2db02700 1 --2- 172.21.15.77:0/247694445 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f2806c500 0x7f5f28153f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:12.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.254+0000 7f5f2d301700 1 -- 172.21.15.77:0/247694445 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f2806c500 msgr2=0x7f5f28153f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:12.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.254+0000 7f5f2d301700 1 --2- 172.21.15.77:0/247694445 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f2806c500 0x7f5f28153f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:12.263 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.254+0000 7f5f2d301700 1 -- 172.21.15.77:0/247694445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f2806bfb0 msgr2=0x7f5f28153a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:12.263 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.254+0000 7f5f2d301700 1 --2- 172.21.15.77:0/247694445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f2806bfb0 0x7f5f28153a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:12.263 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.254+0000 7f5f2d301700 1 -- 172.21.15.77:0/247694445 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f18009370 con 0x7f5f2806bbe0 2022-04-03T19:57:12.263 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.254+0000 7f5f2cb00700 1 --2- 172.21.15.77:0/247694445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f2806bfb0 0x7f5f28153a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:57:12.263 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.254+0000 7f5f2d301700 1 --2- 172.21.15.77:0/247694445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f2806bbe0 0x7f5f280ca650 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5f2400c440 tx=0x7f5f24007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:12.264 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.254+0000 7f5f157fa700 1 -- 172.21.15.77:0/247694445 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f240050d0 con 0x7f5f2806bbe0 2022-04-03T19:57:12.264 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.254+0000 7f5f157fa700 1 -- 172.21.15.77:0/247694445 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f24005230 con 0x7f5f2806bbe0 2022-04-03T19:57:12.264 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.254+0000 7f5f157fa700 1 -- 172.21.15.77:0/247694445 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f240117e0 con 0x7f5f2806bbe0 2022-04-03T19:57:12.264 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.254+0000 7f5f2e303700 1 -- 172.21.15.77:0/247694445 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f28165610 con 0x7f5f2806bbe0 2022-04-03T19:57:12.265 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.254+0000 7f5f2e303700 1 -- 172.21.15.77:0/247694445 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f281554b0 con 0x7f5f2806bbe0 2022-04-03T19:57:12.265 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.255+0000 7f5f2e303700 1 -- 172.21.15.77:0/247694445 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f28153860 con 0x7f5f2806bbe0 2022-04-03T19:57:12.265 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.256+0000 7f5f157fa700 1 -- 172.21.15.77:0/247694445 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f5f24010050 con 0x7f5f2806bbe0 2022-04-03T19:57:12.266 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.257+0000 7f5f157fa700 1 --2- 172.21.15.77:0/247694445 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5f08063ce0 0x7f5f08066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:12.266 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.258+0000 7f5f157fa700 1 -- 172.21.15.77:0/247694445 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6632+0+0 (secure 0 0 0) 0x7f5f2407f740 con 0x7f5f2806bbe0 2022-04-03T19:57:12.267 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.258+0000 7f5f2cb00700 1 --2- 172.21.15.77:0/247694445 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5f08063ce0 0x7f5f08066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:12.267 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.259+0000 7f5f2cb00700 1 --2- 172.21.15.77:0/247694445 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5f08063ce0 0x7f5f08066190 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5f280cc6b0 tx=0x7f5f18016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:12.267 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.262+0000 7f5f157fa700 1 -- 172.21.15.77:0/247694445 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f5f2404e080 con 0x7f5f2806bbe0 2022-04-03T19:57:12.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.495+0000 7f5f2e303700 1 -- 172.21.15.77:0/247694445 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f280c7ec0 con 0x7f5f2806bbe0 2022-04-03T19:57:12.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.496+0000 7f5f157fa700 1 -- 172.21.15.77:0/247694445 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+6110 (secure 0 0 0) 0x7f5f240516a0 con 0x7f5f2806bbe0 2022-04-03T19:57:12.497 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:12.497 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5668,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:55:38.315236+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5463":{"gid":5463,"name":"e","rank":0,"incarnation":38,"state":"up:active","state_seq":30,"addr":"172.21.15.77:6837/2777516376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":2777516376},{"type":"v1","addr":"172.21.15.77:6837","nonce":2777516376}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:12.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.498+0000 7f5f2e303700 1 -- 172.21.15.77:0/247694445 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5f08063ce0 msgr2=0x7f5f08066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:12.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.498+0000 7f5f2e303700 1 --2- 172.21.15.77:0/247694445 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5f08063ce0 0x7f5f08066190 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5f280cc6b0 tx=0x7f5f18016000 comp rx=0 tx=0).stop 2022-04-03T19:57:12.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.498+0000 7f5f2e303700 1 -- 172.21.15.77:0/247694445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f2806bbe0 msgr2=0x7f5f280ca650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:12.499 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.498+0000 7f5f2e303700 1 --2- 172.21.15.77:0/247694445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f2806bbe0 0x7f5f280ca650 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5f2400c440 tx=0x7f5f24007e00 comp rx=0 tx=0).stop 2022-04-03T19:57:12.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.498+0000 7f5f2e303700 1 -- 172.21.15.77:0/247694445 shutdown_connections 2022-04-03T19:57:12.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.498+0000 7f5f2e303700 1 --2- 172.21.15.77:0/247694445 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f5f08063ce0 0x7f5f08066190 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:12.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.498+0000 7f5f2e303700 1 --2- 172.21.15.77:0/247694445 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f2806c500 0x7f5f28153f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:12.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.498+0000 7f5f2e303700 1 --2- 172.21.15.77:0/247694445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5f2806bbe0 0x7f5f280ca650 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:12.500 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.498+0000 7f5f2e303700 1 --2- 172.21.15.77:0/247694445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f2806bfb0 0x7f5f28153a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:12.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.498+0000 7f5f2e303700 1 -- 172.21.15.77:0/247694445 >> 172.21.15.77:0/247694445 conn(0x7f5f280cb6b0 msgr2=0x7f5f28148e20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:12.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.498+0000 7f5f2e303700 1 -- 172.21.15.77:0/247694445 shutdown_connections 2022-04-03T19:57:12.501 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:12.498+0000 7f5f2e303700 1 -- 172.21.15.77:0/247694445 wait complete. 2022-04-03T19:57:12.502 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 43 2022-04-03T19:57:12.522 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2022-04-03T19:57:14.524 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:14.731 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.730+0000 7f2c25236700 1 Processor -- start 2022-04-03T19:57:14.731 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.731+0000 7f2c25236700 1 -- start start 2022-04-03T19:57:14.733 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.732+0000 7f2c25236700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c2006bbe0 0x7f2c201584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:14.734 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.732+0000 7f2c25236700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2c2006bfb0 0x7f2c2006d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:14.734 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.732+0000 7f2c25236700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c2006c500 0x7f2c2014eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:14.734 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.732+0000 7f2c25236700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c20159450 con 0x7f2c2006c500 2022-04-03T19:57:14.735 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.732+0000 7f2c25236700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c2006d260 con 0x7f2c2006bfb0 2022-04-03T19:57:14.735 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.732+0000 7f2c25236700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c200c6330 con 0x7f2c2006bbe0 2022-04-03T19:57:14.735 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.732+0000 7f2c24a35700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c2006c500 0x7f2c2014eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:14.735 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.732+0000 7f2c1ffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c2006bbe0 0x7f2c201584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:14.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.732+0000 7f2c24a35700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f2c2006c500 0x7f2c2014eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43224/0 (socket says 172.21.15.77:43224) 2022-04-03T19:57:14.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.732+0000 7f2c24a35700 1 -- 172.21.15.77:0/793828401 learned_addr learned my addr 172.21.15.77:0/793828401 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:14.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.732+0000 7f2c1f7fe700 1 --2- 172.21.15.77:0/793828401 >> v2:172.21.15.186:3300/0 conn(0x7f2c2006bfb0 0x7f2c2006d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:14.736 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.733+0000 7f2c24a35700 1 -- 172.21.15.77:0/793828401 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c2006bbe0 msgr2=0x7f2c201584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:14.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.733+0000 7f2c24a35700 1 --2- 172.21.15.77:0/793828401 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c2006bbe0 0x7f2c201584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:14.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.733+0000 7f2c24a35700 1 -- 172.21.15.77:0/793828401 >> v2:172.21.15.186:3300/0 conn(0x7f2c2006bfb0 msgr2=0x7f2c2006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:14.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.733+0000 7f2c24a35700 1 --2- 172.21.15.77:0/793828401 >> v2:172.21.15.186:3300/0 conn(0x7f2c2006bfb0 0x7f2c2006d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:14.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.733+0000 7f2c24a35700 1 -- 172.21.15.77:0/793828401 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c20158bb0 con 0x7f2c2006c500 2022-04-03T19:57:14.737 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.733+0000 7f2c24a35700 1 --2- 172.21.15.77:0/793828401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c2006c500 0x7f2c2014eea0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f2c20088d10 tx=0x7f2c18009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30daeed2277f341d server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:14.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.733+0000 7f2c1effd700 1 -- 172.21.15.77:0/793828401 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1800e070 con 0x7f2c2006c500 2022-04-03T19:57:14.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.733+0000 7f2c1effd700 1 -- 172.21.15.77:0/793828401 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c1800b970 con 0x7f2c2006c500 2022-04-03T19:57:14.738 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.733+0000 7f2c1effd700 1 -- 172.21.15.77:0/793828401 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c180092e0 con 0x7f2c2006c500 2022-04-03T19:57:14.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.734+0000 7f2c25236700 1 -- 172.21.15.77:0/793828401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c2006c500 msgr2=0x7f2c2014eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:14.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.734+0000 7f2c25236700 1 --2- 172.21.15.77:0/793828401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c2006c500 0x7f2c2014eea0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f2c20088d10 tx=0x7f2c18009840 comp rx=0 tx=0).stop 2022-04-03T19:57:14.742 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.734+0000 7f2c25236700 1 -- 172.21.15.77:0/793828401 shutdown_connections 2022-04-03T19:57:14.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.734+0000 7f2c25236700 1 --2- 172.21.15.77:0/793828401 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c2006c500 0x7f2c2014eea0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:14.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.734+0000 7f2c25236700 1 --2- 172.21.15.77:0/793828401 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c2006bbe0 0x7f2c201584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:14.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.734+0000 7f2c25236700 1 --2- 172.21.15.77:0/793828401 >> v2:172.21.15.186:3300/0 conn(0x7f2c2006bfb0 0x7f2c2006d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:14.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.734+0000 7f2c25236700 1 -- 172.21.15.77:0/793828401 >> 172.21.15.77:0/793828401 conn(0x7f2c200cb6b0 msgr2=0x7f2c200cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:14.743 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.734+0000 7f2c25236700 1 -- 172.21.15.77:0/793828401 shutdown_connections 2022-04-03T19:57:14.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.734+0000 7f2c25236700 1 -- 172.21.15.77:0/793828401 wait complete. 2022-04-03T19:57:14.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.736+0000 7f2c25236700 1 Processor -- start 2022-04-03T19:57:14.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.736+0000 7f2c25236700 1 -- start start 2022-04-03T19:57:14.744 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.737+0000 7f2c25236700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c2006bbe0 0x7f2c20150b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:14.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.737+0000 7f2c25236700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c2006bfb0 0x7f2c20164a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:14.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.737+0000 7f2c25236700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c2006c500 0x7f2c20163b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:14.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.737+0000 7f2c25236700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c200c3190 con 0x7f2c2006c500 2022-04-03T19:57:14.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.737+0000 7f2c25236700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2014f600 con 0x7f2c2006bfb0 2022-04-03T19:57:14.745 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.737+0000 7f2c25236700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2014faf0 con 0x7f2c2006bbe0 2022-04-03T19:57:14.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.737+0000 7f2c24a35700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c2006c500 0x7f2c20163b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:14.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.737+0000 7f2c1ffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c2006bbe0 0x7f2c20150b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:14.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.737+0000 7f2c1f7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c2006bfb0 0x7f2c20164a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:14.746 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.737+0000 7f2c1ffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c2006bbe0 0x7f2c20150b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60514/0 (socket says 172.21.15.77:60514) 2022-04-03T19:57:14.747 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.737+0000 7f2c1f7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c2006bfb0 0x7f2c20164a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38410/0 (socket says 172.21.15.77:38410) 2022-04-03T19:57:14.747 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.737+0000 7f2c1ffff700 1 -- 172.21.15.77:0/1549083780 learned_addr learned my addr 172.21.15.77:0/1549083780 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:14.747 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.737+0000 7f2c1ffff700 1 -- 172.21.15.77:0/1549083780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c2006bfb0 msgr2=0x7f2c20164a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:14.747 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.737+0000 7f2c1ffff700 1 --2- 172.21.15.77:0/1549083780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c2006bfb0 0x7f2c20164a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:14.748 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.737+0000 7f2c1ffff700 1 -- 172.21.15.77:0/1549083780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c2006c500 msgr2=0x7f2c20163b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:14.748 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.737+0000 7f2c1ffff700 1 --2- 172.21.15.77:0/1549083780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c2006c500 0x7f2c20163b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:14.748 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.737+0000 7f2c1ffff700 1 -- 172.21.15.77:0/1549083780 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c18009ae0 con 0x7f2c2006bbe0 2022-04-03T19:57:14.748 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.738+0000 7f2c1f7fe700 1 --2- 172.21.15.77:0/1549083780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c2006bfb0 0x7f2c20164a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:57:14.749 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.738+0000 7f2c1ffff700 1 --2- 172.21.15.77:0/1549083780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c2006bbe0 0x7f2c20150b80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000e960 tx=0x7f2c10009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:14.749 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.738+0000 7f2c07fff700 1 -- 172.21.15.77:0/1549083780 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c10013680 con 0x7f2c2006bbe0 2022-04-03T19:57:14.749 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.738+0000 7f2c07fff700 1 -- 172.21.15.77:0/1549083780 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c100137e0 con 0x7f2c2006bbe0 2022-04-03T19:57:14.749 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.738+0000 7f2c07fff700 1 -- 172.21.15.77:0/1549083780 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c10013b60 con 0x7f2c2006bbe0 2022-04-03T19:57:14.750 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.738+0000 7f2c25236700 1 -- 172.21.15.77:0/1549083780 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c201627c0 con 0x7f2c2006bbe0 2022-04-03T19:57:14.750 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.738+0000 7f2c25236700 1 -- 172.21.15.77:0/1549083780 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c2014f810 con 0x7f2c2006bbe0 2022-04-03T19:57:14.750 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.739+0000 7f2c25236700 1 -- 172.21.15.77:0/1549083780 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c200c2ef0 con 0x7f2c2006bbe0 2022-04-03T19:57:14.750 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.740+0000 7f2c07fff700 1 -- 172.21.15.77:0/1549083780 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f2c10019030 con 0x7f2c2006bbe0 2022-04-03T19:57:14.751 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.741+0000 7f2c07fff700 1 --2- 172.21.15.77:0/1549083780 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2c00063ce0 0x7f2c00066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:14.751 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.741+0000 7f2c07fff700 1 -- 172.21.15.77:0/1549083780 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f2c10080be0 con 0x7f2c2006bbe0 2022-04-03T19:57:14.751 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.741+0000 7f2c1f7fe700 1 --2- 172.21.15.77:0/1549083780 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2c00063ce0 0x7f2c00066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:14.752 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.742+0000 7f2c1f7fe700 1 --2- 172.21.15.77:0/1549083780 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2c00063ce0 0x7f2c00066190 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2c200b1350 tx=0x7f2c14009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:14.752 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.745+0000 7f2c07fff700 1 -- 172.21.15.77:0/1549083780 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f2c200c2ef0 con 0x7f2c2006bbe0 2022-04-03T19:57:14.983 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.983+0000 7f2c25236700 1 -- 172.21.15.77:0/1549083780 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c200c3400 con 0x7f2c2006bbe0 2022-04-03T19:57:14.984 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.984+0000 7f2c07fff700 1 -- 172.21.15.77:0/1549083780 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+5352 (secure 0 0 0) 0x7f2c10021090 con 0x7f2c2006bbe0 2022-04-03T19:57:14.985 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:14.985 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:57:14.289208+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5668":{"gid":5668,"name":"d","rank":0,"incarnation":44,"state":"up:replay","state_seq":1,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:14.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.986+0000 7f2c25236700 1 -- 172.21.15.77:0/1549083780 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2c00063ce0 msgr2=0x7f2c00066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:14.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.986+0000 7f2c25236700 1 --2- 172.21.15.77:0/1549083780 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2c00063ce0 0x7f2c00066190 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2c200b1350 tx=0x7f2c14009060 comp rx=0 tx=0).stop 2022-04-03T19:57:14.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.986+0000 7f2c25236700 1 -- 172.21.15.77:0/1549083780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c2006bbe0 msgr2=0x7f2c20150b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:14.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.986+0000 7f2c25236700 1 --2- 172.21.15.77:0/1549083780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c2006bbe0 0x7f2c20150b80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000e960 tx=0x7f2c10009060 comp rx=0 tx=0).stop 2022-04-03T19:57:14.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.986+0000 7f2c25236700 1 -- 172.21.15.77:0/1549083780 shutdown_connections 2022-04-03T19:57:14.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.986+0000 7f2c25236700 1 --2- 172.21.15.77:0/1549083780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c2006c500 0x7f2c20163b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:14.988 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.986+0000 7f2c25236700 1 --2- 172.21.15.77:0/1549083780 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f2c00063ce0 0x7f2c00066190 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:14.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.986+0000 7f2c25236700 1 --2- 172.21.15.77:0/1549083780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2c2006bbe0 0x7f2c20150b80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:14.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.986+0000 7f2c25236700 1 --2- 172.21.15.77:0/1549083780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c2006bfb0 0x7f2c20164a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:14.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.986+0000 7f2c25236700 1 -- 172.21.15.77:0/1549083780 >> 172.21.15.77:0/1549083780 conn(0x7f2c200cb6b0 msgr2=0x7f2c2014a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:14.989 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.986+0000 7f2c25236700 1 -- 172.21.15.77:0/1549083780 shutdown_connections 2022-04-03T19:57:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:14.986+0000 7f2c25236700 1 -- 172.21.15.77:0/1549083780 wait complete. 2022-04-03T19:57:14.990 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 44 2022-04-03T19:57:15.012 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2022-04-03T19:57:15.012 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2022-04-03T19:57:15.012 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:15.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.210+0000 7f7a26581700 1 Processor -- start 2022-04-03T19:57:15.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.210+0000 7f7a26581700 1 -- start start 2022-04-03T19:57:15.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.211+0000 7f7a26581700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7a200c3640 0x7f7a20069600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:15.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.211+0000 7f7a26581700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a200c7af0 0x7f7a20069b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:15.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.211+0000 7f7a26581700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a200c4b90 con 0x7f7a200c3270 2022-04-03T19:57:15.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.211+0000 7f7a26581700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a200c3d10 con 0x7f7a200c3640 2022-04-03T19:57:15.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.211+0000 7f7a26581700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a200c3f80 con 0x7f7a200c7af0 2022-04-03T19:57:15.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.212+0000 7f7a25d80700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a200c7af0 0x7f7a20069b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:15.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.212+0000 7f7a24d7e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7a200c3640 0x7f7a20069600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:15.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.212+0000 7f7a25d80700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a200c7af0 0x7f7a20069b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60516/0 (socket says 172.21.15.77:60516) 2022-04-03T19:57:15.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.212+0000 7f7a25d80700 1 -- 172.21.15.77:0/3357399793 learned_addr learned my addr 172.21.15.77:0/3357399793 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:15.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.212+0000 7f7a24d7e700 1 --2- 172.21.15.77:0/3357399793 >> v2:172.21.15.186:3300/0 conn(0x7f7a200c3640 0x7f7a20069600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38412/0 (socket says 172.21.15.77:38412) 2022-04-03T19:57:15.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.212+0000 7f7a2557f700 1 --1- 172.21.15.77:0/3357399793 >> v1:172.21.15.186:6789/0 conn(0x7f7a200c3270 0x7f7a200c2500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44382/0 (socket says 172.21.15.77:44382) 2022-04-03T19:57:15.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.212+0000 7f7a25d80700 1 -- 172.21.15.77:0/3357399793 >> v2:172.21.15.186:3300/0 conn(0x7f7a200c3640 msgr2=0x7f7a20069600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:15.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.212+0000 7f7a25d80700 1 --2- 172.21.15.77:0/3357399793 >> v2:172.21.15.186:3300/0 conn(0x7f7a200c3640 0x7f7a20069600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:15.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.212+0000 7f7a25d80700 1 -- 172.21.15.77:0/3357399793 >> v1:172.21.15.186:6789/0 conn(0x7f7a200c3270 legacy=0x7f7a200c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:15.218 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.212+0000 7f7a25d80700 1 -- 172.21.15.77:0/3357399793 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a200c4260 con 0x7f7a200c7af0 2022-04-03T19:57:15.218 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.212+0000 7f7a25d80700 1 --2- 172.21.15.77:0/3357399793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a200c7af0 0x7f7a20069b40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009f20 tx=0x7f7a18009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9a6846f3fe9d04b server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:15.218 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.213+0000 7f7a0ffff700 1 -- 172.21.15.77:0/3357399793 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1800e070 con 0x7f7a200c7af0 2022-04-03T19:57:15.218 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.213+0000 7f7a0ffff700 1 -- 172.21.15.77:0/3357399793 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1800b9d0 con 0x7f7a200c7af0 2022-04-03T19:57:15.219 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.213+0000 7f7a0ffff700 1 -- 172.21.15.77:0/3357399793 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a180092e0 con 0x7f7a200c7af0 2022-04-03T19:57:15.219 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.213+0000 7f7a26581700 1 -- 172.21.15.77:0/3357399793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a200c7af0 msgr2=0x7f7a20069b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:15.219 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.213+0000 7f7a26581700 1 --2- 172.21.15.77:0/3357399793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a200c7af0 0x7f7a20069b40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009f20 tx=0x7f7a18009840 comp rx=0 tx=0).stop 2022-04-03T19:57:15.219 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.213+0000 7f7a26581700 1 -- 172.21.15.77:0/3357399793 shutdown_connections 2022-04-03T19:57:15.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.213+0000 7f7a26581700 1 --2- 172.21.15.77:0/3357399793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a200c7af0 0x7f7a20069b40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:15.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.213+0000 7f7a26581700 1 --2- 172.21.15.77:0/3357399793 >> v2:172.21.15.186:3300/0 conn(0x7f7a200c3640 0x7f7a20069600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:15.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.213+0000 7f7a26581700 1 -- 172.21.15.77:0/3357399793 >> 172.21.15.77:0/3357399793 conn(0x7f7a20066da0 msgr2=0x7f7a200671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:15.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.213+0000 7f7a26581700 1 -- 172.21.15.77:0/3357399793 shutdown_connections 2022-04-03T19:57:15.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.213+0000 7f7a26581700 1 -- 172.21.15.77:0/3357399793 wait complete. 2022-04-03T19:57:15.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.215+0000 7f7a26581700 1 Processor -- start 2022-04-03T19:57:15.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.215+0000 7f7a26581700 1 -- start start 2022-04-03T19:57:15.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.216+0000 7f7a26581700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a200c3270 0x7f7a200bb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:15.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.216+0000 7f7a26581700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a200c3640 0x7f7a200c58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:15.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.216+0000 7f7a26581700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a200c7af0 0x7f7a200bd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:15.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.216+0000 7f7a26581700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a20155810 con 0x7f7a200c3270 2022-04-03T19:57:15.225 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.216+0000 7f7a26581700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a20152910 con 0x7f7a200c7af0 2022-04-03T19:57:15.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.216+0000 7f7a26581700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a200bb620 con 0x7f7a200c3640 2022-04-03T19:57:15.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.216+0000 7f7a2557f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a200c3270 0x7f7a200bb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:15.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.216+0000 7f7a25d80700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a200c7af0 0x7f7a200bd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:15.226 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.216+0000 7f7a24d7e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a200c3640 0x7f7a200c58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:15.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.216+0000 7f7a24d7e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a200c3640 0x7f7a200c58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60520/0 (socket says 172.21.15.77:60520) 2022-04-03T19:57:15.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.216+0000 7f7a24d7e700 1 -- 172.21.15.77:0/33595932 learned_addr learned my addr 172.21.15.77:0/33595932 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:15.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.217+0000 7f7a24d7e700 1 -- 172.21.15.77:0/33595932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a200c7af0 msgr2=0x7f7a200bd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:15.227 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.217+0000 7f7a24d7e700 1 --2- 172.21.15.77:0/33595932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a200c7af0 0x7f7a200bd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:15.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.217+0000 7f7a24d7e700 1 -- 172.21.15.77:0/33595932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a200c3270 msgr2=0x7f7a200bb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:15.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.217+0000 7f7a24d7e700 1 --2- 172.21.15.77:0/33595932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a200c3270 0x7f7a200bb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:15.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.217+0000 7f7a24d7e700 1 -- 172.21.15.77:0/33595932 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a18009ae0 con 0x7f7a200c3640 2022-04-03T19:57:15.228 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.217+0000 7f7a25d80700 1 --2- 172.21.15.77:0/33595932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a200c7af0 0x7f7a200bd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:57:15.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.217+0000 7f7a2557f700 1 --2- 172.21.15.77:0/33595932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a200c3270 0x7f7a200bb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:57:15.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.217+0000 7f7a24d7e700 1 --2- 172.21.15.77:0/33595932 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a200c3640 0x7f7a200c58f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7a1000e930 tx=0x7f7a10009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:15.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.217+0000 7f7a0d7fa700 1 -- 172.21.15.77:0/33595932 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a10013680 con 0x7f7a200c3640 2022-04-03T19:57:15.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.218+0000 7f7a0d7fa700 1 -- 172.21.15.77:0/33595932 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a100137e0 con 0x7f7a200c3640 2022-04-03T19:57:15.229 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.218+0000 7f7a26581700 1 -- 172.21.15.77:0/33595932 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a201654d0 con 0x7f7a200c3640 2022-04-03T19:57:15.230 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.218+0000 7f7a0d7fa700 1 -- 172.21.15.77:0/33595932 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a10013b60 con 0x7f7a200c3640 2022-04-03T19:57:15.230 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.218+0000 7f7a26581700 1 -- 172.21.15.77:0/33595932 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a201648c0 con 0x7f7a200c3640 2022-04-03T19:57:15.230 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.219+0000 7f7a26581700 1 -- 172.21.15.77:0/33595932 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a20165660 con 0x7f7a200c3640 2022-04-03T19:57:15.230 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.220+0000 7f7a0d7fa700 1 -- 172.21.15.77:0/33595932 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f7a10019030 con 0x7f7a200c3640 2022-04-03T19:57:15.231 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.221+0000 7f7a0d7fa700 1 --2- 172.21.15.77:0/33595932 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7a00063ce0 0x7f7a00066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:15.231 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.221+0000 7f7a0d7fa700 1 -- 172.21.15.77:0/33595932 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f7a10080d90 con 0x7f7a200c3640 2022-04-03T19:57:15.231 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.221+0000 7f7a2557f700 1 --2- 172.21.15.77:0/33595932 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7a00063ce0 0x7f7a00066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:15.231 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.222+0000 7f7a2557f700 1 --2- 172.21.15.77:0/33595932 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7a00063ce0 0x7f7a00066190 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c007fd0 tx=0x7f7a1c007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:15.232 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.224+0000 7f7a0d7fa700 1 -- 172.21.15.77:0/33595932 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f7a1004f760 con 0x7f7a200c3640 2022-04-03T19:57:15.463 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.462+0000 7f7a26581700 1 -- 172.21.15.77:0/33595932 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a2004b050 con 0x7f7a200c3640 2022-04-03T19:57:15.464 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.464+0000 7f7a0d7fa700 1 -- 172.21.15.77:0/33595932 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+5352 (secure 0 0 0) 0x7f7a10021090 con 0x7f7a200c3640 2022-04-03T19:57:15.464 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:15.464 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:57:14.289208+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5668":{"gid":5668,"name":"d","rank":0,"incarnation":44,"state":"up:replay","state_seq":1,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:15.466 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.466+0000 7f7a26581700 1 -- 172.21.15.77:0/33595932 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7a00063ce0 msgr2=0x7f7a00066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:15.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.466+0000 7f7a26581700 1 --2- 172.21.15.77:0/33595932 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7a00063ce0 0x7f7a00066190 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c007fd0 tx=0x7f7a1c007d30 comp rx=0 tx=0).stop 2022-04-03T19:57:15.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.466+0000 7f7a26581700 1 -- 172.21.15.77:0/33595932 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a200c3640 msgr2=0x7f7a200c58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:15.467 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.466+0000 7f7a26581700 1 --2- 172.21.15.77:0/33595932 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a200c3640 0x7f7a200c58f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7a1000e930 tx=0x7f7a10009060 comp rx=0 tx=0).stop 2022-04-03T19:57:15.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.466+0000 7f7a26581700 1 -- 172.21.15.77:0/33595932 shutdown_connections 2022-04-03T19:57:15.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.466+0000 7f7a26581700 1 --2- 172.21.15.77:0/33595932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a200c7af0 0x7f7a200bd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:15.468 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.466+0000 7f7a26581700 1 --2- 172.21.15.77:0/33595932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a200c3270 0x7f7a200bb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:15.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.466+0000 7f7a26581700 1 --2- 172.21.15.77:0/33595932 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f7a00063ce0 0x7f7a00066190 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:15.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.466+0000 7f7a26581700 1 --2- 172.21.15.77:0/33595932 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a200c3640 0x7f7a200c58f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:15.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.466+0000 7f7a26581700 1 -- 172.21.15.77:0/33595932 >> 172.21.15.77:0/33595932 conn(0x7f7a20066da0 msgr2=0x7f7a20153ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:15.469 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.466+0000 7f7a26581700 1 -- 172.21.15.77:0/33595932 shutdown_connections 2022-04-03T19:57:15.470 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:15.466+0000 7f7a26581700 1 -- 172.21.15.77:0/33595932 wait complete. 2022-04-03T19:57:15.470 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 44 2022-04-03T19:57:15.492 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 44, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-04-03T19:44:21.275716+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 44, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5668": { "addr": "172.21.15.186:6837/113043607", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 113043607, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 113043607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 5668, "incarnation": 44, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 53, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-04-03T19:57:14.289208+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5668 } } } ], "standbys": [ { "addr": "172.21.15.77:6834/2116223509", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6832", "nonce": 2116223509, "type": "v2" }, { "addr": "172.21.15.77:6834", "nonce": 2116223509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4245, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6836/2374276747", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2374276747, "type": "v2" }, { "addr": "172.21.15.186:6836", "nonce": 2374276747, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4259, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/610990853", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6838", "nonce": 610990853, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 610990853, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4265, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.77:6835/170053412", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6833", "nonce": 170053412, "type": "v2" }, { "addr": "172.21.15.77:6835", "nonce": 170053412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4271, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-04-03T19:57:17.494 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:17.657 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:57:17.662 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:57:17.704 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.703+0000 7f290dd4a700 1 Processor -- start 2022-04-03T19:57:17.704 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.703+0000 7f290dd4a700 1 -- start start 2022-04-03T19:57:17.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.704+0000 7f290dd4a700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2908155370 0x7f2908069110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:17.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.704+0000 7f290dd4a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29080c3b10 0x7f2908069650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:17.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.704+0000 7f290dd4a700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29080c3710 con 0x7f29080c1170 2022-04-03T19:57:17.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.704+0000 7f290dd4a700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2908152330 con 0x7f2908155370 2022-04-03T19:57:17.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.704+0000 7f290dd4a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29080c1540 con 0x7f29080c3b10 2022-04-03T19:57:17.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.704+0000 7f290cd48700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f29080c1170 0x7f29081527c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38578/0 (socket says 172.21.15.77:38578) 2022-04-03T19:57:17.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.704+0000 7f290cd48700 1 -- 172.21.15.77:0/910832419 learned_addr learned my addr 172.21.15.77:0/910832419 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:57:17.709 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.704+0000 7f2907fff700 1 --2- 172.21.15.77:0/910832419 >> v2:172.21.15.186:3300/0 conn(0x7f2908155370 0x7f2908069110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:17.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.704+0000 7f290d549700 1 --2- 172.21.15.77:0/910832419 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29080c3b10 0x7f2908069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:17.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.705+0000 7f29077fe700 1 -- 172.21.15.77:0/910832419 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000836215 0 0) 0x7f29080c3710 con 0x7f29080c1170 2022-04-03T19:57:17.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.705+0000 7f29077fe700 1 -- 172.21.15.77:0/910832419 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28ec0033f0 con 0x7f29080c1170 2022-04-03T19:57:17.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.705+0000 7f2907fff700 1 -- 172.21.15.77:0/910832419 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29080c3b10 msgr2=0x7f2908069650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:17.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.705+0000 7f2907fff700 1 --2- 172.21.15.77:0/910832419 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29080c3b10 0x7f2908069650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:17.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.705+0000 7f2907fff700 1 -- 172.21.15.77:0/910832419 >> v1:172.21.15.77:6789/0 conn(0x7f29080c1170 legacy=0x7f29081527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:17.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.705+0000 7f2907fff700 1 -- 172.21.15.77:0/910832419 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29080c4270 con 0x7f2908155370 2022-04-03T19:57:17.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.705+0000 7f290d549700 1 --2- 172.21.15.77:0/910832419 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29080c3b10 0x7f2908069650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:57:17.712 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.705+0000 7f2907fff700 1 --2- 172.21.15.77:0/910832419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2908155370 0x7f2908069110 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f28f4009760 tx=0x7f28f4009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=260c93eb6f56ba8 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:17.712 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.706+0000 7f29077fe700 1 -- 172.21.15.77:0/910832419 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f400f6b0 con 0x7f2908155370 2022-04-03T19:57:17.712 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.706+0000 7f29077fe700 1 -- 172.21.15.77:0/910832419 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f400f810 con 0x7f2908155370 2022-04-03T19:57:17.712 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.706+0000 7f29077fe700 1 -- 172.21.15.77:0/910832419 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f400fbb0 con 0x7f2908155370 2022-04-03T19:57:17.713 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.706+0000 7f290dd4a700 1 -- 172.21.15.77:0/910832419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2908155370 msgr2=0x7f2908069110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:17.713 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.706+0000 7f290dd4a700 1 --2- 172.21.15.77:0/910832419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2908155370 0x7f2908069110 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f28f4009760 tx=0x7f28f4009060 comp rx=0 tx=0).stop 2022-04-03T19:57:17.713 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.706+0000 7f290dd4a700 1 -- 172.21.15.77:0/910832419 shutdown_connections 2022-04-03T19:57:17.713 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.706+0000 7f290dd4a700 1 --2- 172.21.15.77:0/910832419 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29080c3b10 0x7f2908069650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:17.714 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.706+0000 7f290dd4a700 1 --2- 172.21.15.77:0/910832419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2908155370 0x7f2908069110 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:17.714 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.706+0000 7f290dd4a700 1 -- 172.21.15.77:0/910832419 >> 172.21.15.77:0/910832419 conn(0x7f2908066da0 msgr2=0x7f29080671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:17.714 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.706+0000 7f290dd4a700 1 -- 172.21.15.77:0/910832419 shutdown_connections 2022-04-03T19:57:17.714 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.706+0000 7f290dd4a700 1 -- 172.21.15.77:0/910832419 wait complete. 2022-04-03T19:57:17.718 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.708+0000 7f290dd4a700 1 Processor -- start 2022-04-03T19:57:17.718 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.708+0000 7f290dd4a700 1 -- start start 2022-04-03T19:57:17.719 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.709+0000 7f290dd4a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29080c1170 0x7f2908144df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:17.719 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.709+0000 7f290dd4a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29080c3b10 0x7f2908145330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:17.719 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.709+0000 7f290dd4a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2908155370 0x7f29080c7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:17.719 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.709+0000 7f290dd4a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29080c7bf0 con 0x7f29080c1170 2022-04-03T19:57:17.720 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.709+0000 7f290dd4a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29080c67c0 con 0x7f29080c3b10 2022-04-03T19:57:17.720 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.709+0000 7f290dd4a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29080c6aa0 con 0x7f2908155370 2022-04-03T19:57:17.720 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.709+0000 7f290cd48700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29080c1170 0x7f2908144df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:17.720 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.709+0000 7f2907fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29080c3b10 0x7f2908145330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:17.721 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.709+0000 7f290d549700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2908155370 0x7f29080c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:17.721 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.709+0000 7f2907fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29080c3b10 0x7f2908145330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38418/0 (socket says 172.21.15.77:38418) 2022-04-03T19:57:17.721 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.709+0000 7f290cd48700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29080c1170 0x7f2908144df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43232/0 (socket says 172.21.15.77:43232) 2022-04-03T19:57:17.721 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.709+0000 7f2907fff700 1 -- 172.21.15.77:0/3535079979 learned_addr learned my addr 172.21.15.77:0/3535079979 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:17.722 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.710+0000 7f2907fff700 1 -- 172.21.15.77:0/3535079979 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2908155370 msgr2=0x7f29080c7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:17.722 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.710+0000 7f2907fff700 1 --2- 172.21.15.77:0/3535079979 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2908155370 0x7f29080c7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:17.722 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.710+0000 7f2907fff700 1 -- 172.21.15.77:0/3535079979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29080c1170 msgr2=0x7f2908144df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:17.722 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.710+0000 7f2907fff700 1 --2- 172.21.15.77:0/3535079979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29080c1170 0x7f2908144df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:17.722 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.710+0000 7f2907fff700 1 -- 172.21.15.77:0/3535079979 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2900009a20 con 0x7f29080c3b10 2022-04-03T19:57:17.723 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.710+0000 7f2907fff700 1 --2- 172.21.15.77:0/3535079979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29080c3b10 0x7f2908145330 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2908071610 tx=0x7f28f4018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:17.723 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.710+0000 7f2904ff9700 1 -- 172.21.15.77:0/3535079979 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f40036f0 con 0x7f29080c3b10 2022-04-03T19:57:17.723 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.710+0000 7f2904ff9700 1 -- 172.21.15.77:0/3535079979 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f401d040 con 0x7f29080c3b10 2022-04-03T19:57:17.723 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.710+0000 7f290dd4a700 1 -- 172.21.15.77:0/3535079979 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f40092a0 con 0x7f29080c3b10 2022-04-03T19:57:17.724 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.710+0000 7f2904ff9700 1 -- 172.21.15.77:0/3535079979 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f4003aa0 con 0x7f29080c3b10 2022-04-03T19:57:17.724 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.711+0000 7f290dd4a700 1 -- 172.21.15.77:0/3535079979 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f290806c620 con 0x7f29080c3b10 2022-04-03T19:57:17.724 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.712+0000 7f290dd4a700 1 -- 172.21.15.77:0/3535079979 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29080c6c60 con 0x7f29080c3b10 2022-04-03T19:57:17.724 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.713+0000 7f2904ff9700 1 -- 172.21.15.77:0/3535079979 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f28f4028030 con 0x7f29080c3b10 2022-04-03T19:57:17.724 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.713+0000 7f2904ff9700 1 --2- 172.21.15.77:0/3535079979 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f28f0063d10 0x7f28f00661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:17.725 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.713+0000 7f2904ff9700 1 -- 172.21.15.77:0/3535079979 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f28f4081ab0 con 0x7f29080c3b10 2022-04-03T19:57:17.725 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.714+0000 7f290cd48700 1 --2- 172.21.15.77:0/3535079979 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f28f0063d10 0x7f28f00661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:17.725 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.714+0000 7f290cd48700 1 --2- 172.21.15.77:0/3535079979 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f28f0063d10 0x7f28f00661c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f28fc005750 tx=0x7f28fc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:17.726 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.718+0000 7f2904ff9700 1 -- 172.21.15.77:0/3535079979 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f28f4050460 con 0x7f29080c3b10 2022-04-03T19:57:17.959 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.958+0000 7f290dd4a700 1 -- 172.21.15.77:0/3535079979 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29080c2e90 con 0x7f29080c3b10 2022-04-03T19:57:17.960 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.959+0000 7f2904ff9700 1 -- 172.21.15.77:0/3535079979 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+5352 (secure 0 0 0) 0x7f28f4026020 con 0x7f29080c3b10 2022-04-03T19:57:17.960 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:17.960 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:57:14.289208+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5668":{"gid":5668,"name":"d","rank":0,"incarnation":44,"state":"up:replay","state_seq":1,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:17.963 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.961+0000 7f290dd4a700 1 -- 172.21.15.77:0/3535079979 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f28f0063d10 msgr2=0x7f28f00661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:17.963 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.961+0000 7f290dd4a700 1 --2- 172.21.15.77:0/3535079979 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f28f0063d10 0x7f28f00661c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f28fc005750 tx=0x7f28fc009040 comp rx=0 tx=0).stop 2022-04-03T19:57:17.963 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.961+0000 7f290dd4a700 1 -- 172.21.15.77:0/3535079979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29080c3b10 msgr2=0x7f2908145330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:17.963 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.961+0000 7f290dd4a700 1 --2- 172.21.15.77:0/3535079979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29080c3b10 0x7f2908145330 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2908071610 tx=0x7f28f4018000 comp rx=0 tx=0).stop 2022-04-03T19:57:17.964 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.962+0000 7f290dd4a700 1 -- 172.21.15.77:0/3535079979 shutdown_connections 2022-04-03T19:57:17.964 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.962+0000 7f290dd4a700 1 --2- 172.21.15.77:0/3535079979 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f28f0063d10 0x7f28f00661c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:17.964 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.962+0000 7f290dd4a700 1 --2- 172.21.15.77:0/3535079979 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2908155370 0x7f29080c7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:17.964 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.962+0000 7f290dd4a700 1 --2- 172.21.15.77:0/3535079979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f29080c1170 0x7f2908144df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:17.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.962+0000 7f290dd4a700 1 --2- 172.21.15.77:0/3535079979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29080c3b10 0x7f2908145330 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:17.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.962+0000 7f290dd4a700 1 -- 172.21.15.77:0/3535079979 >> 172.21.15.77:0/3535079979 conn(0x7f2908066da0 msgr2=0x7f29080ba480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:17.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.962+0000 7f290dd4a700 1 -- 172.21.15.77:0/3535079979 shutdown_connections 2022-04-03T19:57:17.965 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:17.962+0000 7f290dd4a700 1 -- 172.21.15.77:0/3535079979 wait complete. 2022-04-03T19:57:17.966 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 44 2022-04-03T19:57:17.986 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:57:19.987 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:20.197 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.196+0000 7ff6023c1700 1 Processor -- start 2022-04-03T19:57:20.198 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.196+0000 7ff6023c1700 1 -- start start 2022-04-03T19:57:20.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.197+0000 7ff6023c1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5fc06bfb0 0x7ff5fc1584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:20.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.197+0000 7ff6023c1700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5fc06d260 con 0x7ff5fc06c500 2022-04-03T19:57:20.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.197+0000 7ff6023c1700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5fc0c6330 con 0x7ff5fc06bbe0 2022-04-03T19:57:20.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.197+0000 7ff6023c1700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc159450 con 0x7ff5fc06bfb0 2022-04-03T19:57:20.201 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.197+0000 7ff6013bf700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ff5fc06bbe0 0x7ff5fc06d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44384/0 (socket says 172.21.15.77:44384) 2022-04-03T19:57:20.201 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.197+0000 7ff6013bf700 1 -- 172.21.15.77:0/3931534773 learned_addr learned my addr 172.21.15.77:0/3931534773 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:57:20.201 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.197+0000 7ff600bbe700 1 --2- 172.21.15.77:0/3931534773 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5fc06bfb0 0x7ff5fc1584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:20.201 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.198+0000 7ff5ebfff700 1 -- 172.21.15.77:0/3931534773 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715108785 0 0) 0x7ff5fc0c6330 con 0x7ff5fc06bbe0 2022-04-03T19:57:20.202 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.198+0000 7ff5ebfff700 1 -- 172.21.15.77:0/3931534773 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5e40033f0 con 0x7ff5fc06bbe0 2022-04-03T19:57:20.202 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.198+0000 7ff5ebfff700 1 -- 172.21.15.77:0/3931534773 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537818070 0 0) 0x7ff5fc06d260 con 0x7ff5fc06c500 2022-04-03T19:57:20.202 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.198+0000 7ff5ebfff700 1 -- 172.21.15.77:0/3931534773 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5fc0c6330 con 0x7ff5fc06c500 2022-04-03T19:57:20.203 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.198+0000 7ff600bbe700 1 -- 172.21.15.77:0/3931534773 >> v1:172.21.15.186:6789/0 conn(0x7ff5fc06bbe0 legacy=0x7ff5fc06d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:20.209 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.198+0000 7ff600bbe700 1 -- 172.21.15.77:0/3931534773 >> v1:172.21.15.77:6789/0 conn(0x7ff5fc06c500 legacy=0x7ff5fc14eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:20.209 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.198+0000 7ff600bbe700 1 -- 172.21.15.77:0/3931534773 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5fc158bb0 con 0x7ff5fc06bfb0 2022-04-03T19:57:20.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.198+0000 7ff600bbe700 1 --2- 172.21.15.77:0/3931534773 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5fc06bfb0 0x7ff5fc1584b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec009790 tx=0x7ff5ec009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dba555df18e3481 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:20.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.199+0000 7ff5ebfff700 1 -- 172.21.15.77:0/3931534773 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec011630 con 0x7ff5fc06bfb0 2022-04-03T19:57:20.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.199+0000 7ff5ebfff700 1 -- 172.21.15.77:0/3931534773 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ec011790 con 0x7ff5fc06bfb0 2022-04-03T19:57:20.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.199+0000 7ff5ebfff700 1 -- 172.21.15.77:0/3931534773 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec011b50 con 0x7ff5fc06bfb0 2022-04-03T19:57:20.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.199+0000 7ff6023c1700 1 -- 172.21.15.77:0/3931534773 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5fc06bfb0 msgr2=0x7ff5fc1584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:20.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.199+0000 7ff6023c1700 1 --2- 172.21.15.77:0/3931534773 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5fc06bfb0 0x7ff5fc1584b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec009790 tx=0x7ff5ec009060 comp rx=0 tx=0).stop 2022-04-03T19:57:20.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.199+0000 7ff6023c1700 1 -- 172.21.15.77:0/3931534773 shutdown_connections 2022-04-03T19:57:20.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.199+0000 7ff6023c1700 1 --2- 172.21.15.77:0/3931534773 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5fc06bfb0 0x7ff5fc1584b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:20.215 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.199+0000 7ff6023c1700 1 -- 172.21.15.77:0/3931534773 >> 172.21.15.77:0/3931534773 conn(0x7ff5fc0cb6b0 msgr2=0x7ff5fc0cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:20.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.199+0000 7ff6023c1700 1 -- 172.21.15.77:0/3931534773 shutdown_connections 2022-04-03T19:57:20.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.199+0000 7ff6023c1700 1 -- 172.21.15.77:0/3931534773 wait complete. 2022-04-03T19:57:20.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.201+0000 7ff6023c1700 1 Processor -- start 2022-04-03T19:57:20.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.201+0000 7ff6023c1700 1 -- start start 2022-04-03T19:57:20.216 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.202+0000 7ff6023c1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc06bbe0 0x7ff5fc155650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:20.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.202+0000 7ff6023c1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5fc06bfb0 0x7ff5fc14d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:20.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.202+0000 7ff6023c1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5fc06c500 0x7ff5fc1529d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:20.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.202+0000 7ff6023c1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc155d20 con 0x7ff5fc06bbe0 2022-04-03T19:57:20.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.202+0000 7ff6023c1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc1519b0 con 0x7ff5fc06bfb0 2022-04-03T19:57:20.217 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.202+0000 7ff6023c1700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5fc150da0 con 0x7ff5fc06c500 2022-04-03T19:57:20.218 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.202+0000 7ff600bbe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5fc06bfb0 0x7ff5fc14d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:20.218 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.202+0000 7ff6013bf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc06bbe0 0x7ff5fc155650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:20.218 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.202+0000 7ff601bc0700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5fc06c500 0x7ff5fc1529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:20.218 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.202+0000 7ff600bbe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5fc06bfb0 0x7ff5fc14d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38420/0 (socket says 172.21.15.77:38420) 2022-04-03T19:57:20.219 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.202+0000 7ff600bbe700 1 -- 172.21.15.77:0/3831017773 learned_addr learned my addr 172.21.15.77:0/3831017773 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:20.219 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.202+0000 7ff600bbe700 1 -- 172.21.15.77:0/3831017773 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5fc06c500 msgr2=0x7ff5fc1529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:20.219 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.202+0000 7ff600bbe700 1 --2- 172.21.15.77:0/3831017773 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5fc06c500 0x7ff5fc1529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:20.219 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.202+0000 7ff600bbe700 1 -- 172.21.15.77:0/3831017773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc06bbe0 msgr2=0x7ff5fc155650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:20.219 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.202+0000 7ff600bbe700 1 --2- 172.21.15.77:0/3831017773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc06bbe0 0x7ff5fc155650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:20.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.202+0000 7ff600bbe700 1 -- 172.21.15.77:0/3831017773 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ec0092a0 con 0x7ff5fc06bfb0 2022-04-03T19:57:20.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.203+0000 7ff600bbe700 1 --2- 172.21.15.77:0/3831017773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5fc06bfb0 0x7ff5fc14d760 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec009d80 tx=0x7ff5ec017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:20.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.203+0000 7ff5e97fa700 1 -- 172.21.15.77:0/3831017773 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec004550 con 0x7ff5fc06bfb0 2022-04-03T19:57:20.220 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.203+0000 7ff5e97fa700 1 -- 172.21.15.77:0/3831017773 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ec011900 con 0x7ff5fc06bfb0 2022-04-03T19:57:20.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.203+0000 7ff6023c1700 1 -- 172.21.15.77:0/3831017773 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5fc151080 con 0x7ff5fc06bfb0 2022-04-03T19:57:20.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.203+0000 7ff5e97fa700 1 -- 172.21.15.77:0/3831017773 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec0253f0 con 0x7ff5fc06bfb0 2022-04-03T19:57:20.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.203+0000 7ff6023c1700 1 -- 172.21.15.77:0/3831017773 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5fc151640 con 0x7ff5fc06bfb0 2022-04-03T19:57:20.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.204+0000 7ff6023c1700 1 -- 172.21.15.77:0/3831017773 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5fc1517d0 con 0x7ff5fc06bfb0 2022-04-03T19:57:20.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.205+0000 7ff5e97fa700 1 -- 172.21.15.77:0/3831017773 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7ff5ec00f050 con 0x7ff5fc06bfb0 2022-04-03T19:57:20.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.206+0000 7ff5e97fa700 1 --2- 172.21.15.77:0/3831017773 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff5dc063d10 0x7ff5dc0661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:20.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.206+0000 7ff5e97fa700 1 -- 172.21.15.77:0/3831017773 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6718+0+0 (secure 0 0 0) 0x7ff5ec082220 con 0x7ff5fc06bfb0 2022-04-03T19:57:20.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.206+0000 7ff6013bf700 1 --2- 172.21.15.77:0/3831017773 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff5dc063d10 0x7ff5dc0661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:20.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.207+0000 7ff6013bf700 1 --2- 172.21.15.77:0/3831017773 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff5dc063d10 0x7ff5dc0661c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8005750 tx=0x7ff5f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:20.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.210+0000 7ff5e97fa700 1 -- 172.21.15.77:0/3831017773 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7ff5ec050a80 con 0x7ff5fc06bfb0 2022-04-03T19:57:20.445 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.444+0000 7ff6023c1700 1 -- 172.21.15.77:0/3831017773 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5fc04b050 con 0x7ff5fc06bfb0 2022-04-03T19:57:20.446 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.446+0000 7ff5e97fa700 1 -- 172.21.15.77:0/3831017773 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+5352 (secure 0 0 0) 0x7ff5ec015020 con 0x7ff5fc06bfb0 2022-04-03T19:57:20.447 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:20.447 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:57:14.289208+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5668":{"gid":5668,"name":"d","rank":0,"incarnation":44,"state":"up:replay","state_seq":1,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:20.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.447+0000 7ff6023c1700 1 -- 172.21.15.77:0/3831017773 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff5dc063d10 msgr2=0x7ff5dc0661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:20.449 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.447+0000 7ff6023c1700 1 --2- 172.21.15.77:0/3831017773 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff5dc063d10 0x7ff5dc0661c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8005750 tx=0x7ff5f8009040 comp rx=0 tx=0).stop 2022-04-03T19:57:20.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.447+0000 7ff6023c1700 1 -- 172.21.15.77:0/3831017773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5fc06bfb0 msgr2=0x7ff5fc14d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:20.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.447+0000 7ff6023c1700 1 --2- 172.21.15.77:0/3831017773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5fc06bfb0 0x7ff5fc14d760 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec009d80 tx=0x7ff5ec017000 comp rx=0 tx=0).stop 2022-04-03T19:57:20.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.447+0000 7ff6023c1700 1 -- 172.21.15.77:0/3831017773 shutdown_connections 2022-04-03T19:57:20.450 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.447+0000 7ff6023c1700 1 --2- 172.21.15.77:0/3831017773 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff5dc063d10 0x7ff5dc0661c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:20.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.447+0000 7ff6023c1700 1 --2- 172.21.15.77:0/3831017773 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5fc06c500 0x7ff5fc1529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:20.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.447+0000 7ff6023c1700 1 --2- 172.21.15.77:0/3831017773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5fc06bbe0 0x7ff5fc155650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:20.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.447+0000 7ff6023c1700 1 --2- 172.21.15.77:0/3831017773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5fc06bfb0 0x7ff5fc14d760 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:20.451 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.447+0000 7ff6023c1700 1 -- 172.21.15.77:0/3831017773 >> 172.21.15.77:0/3831017773 conn(0x7ff5fc0cb6b0 msgr2=0x7ff5fc14a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:20.452 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.447+0000 7ff6023c1700 1 -- 172.21.15.77:0/3831017773 shutdown_connections 2022-04-03T19:57:20.452 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:20.448+0000 7ff6023c1700 1 -- 172.21.15.77:0/3831017773 wait complete. 2022-04-03T19:57:20.452 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 44 2022-04-03T19:57:20.473 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:57:22.474 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:22.682 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.680+0000 7fbd7ebe5700 1 Processor -- start 2022-04-03T19:57:22.683 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.680+0000 7fbd7ebe5700 1 -- start start 2022-04-03T19:57:22.691 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.681+0000 7fbd7ebe5700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbd7806bbe0 0x7fbd781584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:22.691 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.681+0000 7fbd7ebe5700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbd7806bfb0 0x7fbd7806d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:22.692 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.681+0000 7fbd7ebe5700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd7806d260 con 0x7fbd7806c500 2022-04-03T19:57:22.692 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.681+0000 7fbd7ebe5700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd78159450 con 0x7fbd7806bbe0 2022-04-03T19:57:22.692 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.682+0000 7fbd7ebe5700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd780c6330 con 0x7fbd7806bfb0 2022-04-03T19:57:22.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.682+0000 7fbd7d3e2700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbd7806bfb0 0x7fbd7806d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:22.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.682+0000 7fbd7dbe3700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbd7806bbe0 0x7fbd781584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:22.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.682+0000 7fbd7e3e4700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fbd7806c500 0x7fbd7814eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44386/0 (socket says 172.21.15.77:44386) 2022-04-03T19:57:22.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.682+0000 7fbd7e3e4700 1 -- 172.21.15.77:0/1946970632 learned_addr learned my addr 172.21.15.77:0/1946970632 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:57:22.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.682+0000 7fbd7dbe3700 1 -- 172.21.15.77:0/1946970632 >> v2:172.21.15.186:3300/0 conn(0x7fbd7806bfb0 msgr2=0x7fbd7806d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:22.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.682+0000 7fbd7dbe3700 1 --2- 172.21.15.77:0/1946970632 >> v2:172.21.15.186:3300/0 conn(0x7fbd7806bfb0 0x7fbd7806d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:22.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.682+0000 7fbd7dbe3700 1 -- 172.21.15.77:0/1946970632 >> v1:172.21.15.186:6789/0 conn(0x7fbd7806c500 legacy=0x7fbd7814eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:22.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.682+0000 7fbd7dbe3700 1 -- 172.21.15.77:0/1946970632 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd78158bb0 con 0x7fbd7806bbe0 2022-04-03T19:57:22.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.682+0000 7fbd7cbe1700 1 -- 172.21.15.77:0/1946970632 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548774676 0 0) 0x7fbd7806d260 con 0x7fbd7806c500 2022-04-03T19:57:22.696 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.682+0000 7fbd7d3e2700 1 --2- 172.21.15.77:0/1946970632 >> v2:172.21.15.186:3300/0 conn(0x7fbd7806bfb0 0x7fbd7806d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:57:22.696 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.683+0000 7fbd7dbe3700 1 --2- 172.21.15.77:0/1946970632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd7806bbe0 0x7fbd781584b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fbd780898c0 tx=0x7fbd74009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e69b66532e4bcbac server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:22.696 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.683+0000 7fbd7cbe1700 1 -- 172.21.15.77:0/1946970632 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd74011630 con 0x7fbd7806bbe0 2022-04-03T19:57:22.696 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.683+0000 7fbd7cbe1700 1 -- 172.21.15.77:0/1946970632 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd74011790 con 0x7fbd7806bbe0 2022-04-03T19:57:22.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.683+0000 7fbd7cbe1700 1 -- 172.21.15.77:0/1946970632 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd74011b30 con 0x7fbd7806bbe0 2022-04-03T19:57:22.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.683+0000 7fbd7ebe5700 1 -- 172.21.15.77:0/1946970632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd7806bbe0 msgr2=0x7fbd781584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:22.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.683+0000 7fbd7ebe5700 1 --2- 172.21.15.77:0/1946970632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd7806bbe0 0x7fbd781584b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fbd780898c0 tx=0x7fbd74009060 comp rx=0 tx=0).stop 2022-04-03T19:57:22.698 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.684+0000 7fbd7ebe5700 1 -- 172.21.15.77:0/1946970632 shutdown_connections 2022-04-03T19:57:22.698 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.684+0000 7fbd7ebe5700 1 --2- 172.21.15.77:0/1946970632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd7806bbe0 0x7fbd781584b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:22.698 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.684+0000 7fbd7ebe5700 1 --2- 172.21.15.77:0/1946970632 >> v2:172.21.15.186:3300/0 conn(0x7fbd7806bfb0 0x7fbd7806d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:22.699 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.684+0000 7fbd7ebe5700 1 -- 172.21.15.77:0/1946970632 >> 172.21.15.77:0/1946970632 conn(0x7fbd780cb6b0 msgr2=0x7fbd780cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:22.699 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.684+0000 7fbd7ebe5700 1 -- 172.21.15.77:0/1946970632 shutdown_connections 2022-04-03T19:57:22.699 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.684+0000 7fbd7ebe5700 1 -- 172.21.15.77:0/1946970632 wait complete. 2022-04-03T19:57:22.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.685+0000 7fbd7ebe5700 1 Processor -- start 2022-04-03T19:57:22.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.686+0000 7fbd7ebe5700 1 -- start start 2022-04-03T19:57:22.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.686+0000 7fbd7ebe5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd7806bbe0 0x7fbd781583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:22.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.686+0000 7fbd7ebe5700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbd7806bfb0 0x7fbd78155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:22.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.686+0000 7fbd7ebe5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd7806c500 0x7fbd781661f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:22.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.686+0000 7fbd7ebe5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd78155e10 con 0x7fbd7806c500 2022-04-03T19:57:22.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.686+0000 7fbd7ebe5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd78150e10 con 0x7fbd7806bbe0 2022-04-03T19:57:22.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.686+0000 7fbd7ebe5700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd78155b80 con 0x7fbd7806bfb0 2022-04-03T19:57:22.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.687+0000 7fbd7dbe3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd7806bbe0 0x7fbd781583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:22.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.687+0000 7fbd7dbe3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd7806bbe0 0x7fbd781583b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38424/0 (socket says 172.21.15.77:38424) 2022-04-03T19:57:22.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.687+0000 7fbd7dbe3700 1 -- 172.21.15.77:0/398179606 learned_addr learned my addr 172.21.15.77:0/398179606 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:22.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.687+0000 7fbd7d3e2700 1 --2- 172.21.15.77:0/398179606 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbd7806bfb0 0x7fbd78155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:22.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.687+0000 7fbd7e3e4700 1 --2- 172.21.15.77:0/398179606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd7806c500 0x7fbd781661f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:22.704 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.687+0000 7fbd7dbe3700 1 -- 172.21.15.77:0/398179606 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbd7806bfb0 msgr2=0x7fbd78155640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:22.704 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.687+0000 7fbd7dbe3700 1 --2- 172.21.15.77:0/398179606 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbd7806bfb0 0x7fbd78155640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:22.704 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.687+0000 7fbd7dbe3700 1 -- 172.21.15.77:0/398179606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd7806c500 msgr2=0x7fbd781661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:22.705 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.687+0000 7fbd7dbe3700 1 --2- 172.21.15.77:0/398179606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd7806c500 0x7fbd781661f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:22.705 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.687+0000 7fbd7dbe3700 1 -- 172.21.15.77:0/398179606 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd680092a0 con 0x7fbd7806bbe0 2022-04-03T19:57:22.705 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.687+0000 7fbd7e3e4700 1 --2- 172.21.15.77:0/398179606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd7806c500 0x7fbd781661f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:57:22.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.688+0000 7fbd7dbe3700 1 --2- 172.21.15.77:0/398179606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd7806bbe0 0x7fbd781583b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbd74009dd0 tx=0x7fbd74017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:22.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.688+0000 7fbd65ffb700 1 -- 172.21.15.77:0/398179606 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd740036f0 con 0x7fbd7806bbe0 2022-04-03T19:57:22.706 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.688+0000 7fbd65ffb700 1 -- 172.21.15.77:0/398179606 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd74003870 con 0x7fbd7806bbe0 2022-04-03T19:57:22.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.688+0000 7fbd65ffb700 1 -- 172.21.15.77:0/398179606 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd74003c30 con 0x7fbd7806bbe0 2022-04-03T19:57:22.707 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.688+0000 7fbd7ebe5700 1 -- 172.21.15.77:0/398179606 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd740092a0 con 0x7fbd7806bbe0 2022-04-03T19:57:22.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.688+0000 7fbd7ebe5700 1 -- 172.21.15.77:0/398179606 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd78150470 con 0x7fbd7806bbe0 2022-04-03T19:57:22.708 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.689+0000 7fbd7ebe5700 1 -- 172.21.15.77:0/398179606 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd780bf100 con 0x7fbd7806bbe0 2022-04-03T19:57:22.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.690+0000 7fbd65ffb700 1 -- 172.21.15.77:0/398179606 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fbd7400f050 con 0x7fbd7806bbe0 2022-04-03T19:57:22.710 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.691+0000 7fbd65ffb700 1 --2- 172.21.15.77:0/398179606 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbd58063ce0 0x7fbd58066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:22.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.691+0000 7fbd65ffb700 1 -- 172.21.15.77:0/398179606 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6718+0+0 (secure 0 0 0) 0x7fbd74081f80 con 0x7fbd7806bbe0 2022-04-03T19:57:22.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.691+0000 7fbd7d3e2700 1 --2- 172.21.15.77:0/398179606 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbd58063ce0 0x7fbd58066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:22.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.692+0000 7fbd7d3e2700 1 --2- 172.21.15.77:0/398179606 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbd58063ce0 0x7fbd58066190 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fbd780c38f0 tx=0x7fbd68009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:22.711 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.695+0000 7fbd65ffb700 1 -- 172.21.15.77:0/398179606 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fbd740508d0 con 0x7fbd7806bbe0 2022-04-03T19:57:22.933 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.932+0000 7fbd7ebe5700 1 -- 172.21.15.77:0/398179606 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd780c7ec0 con 0x7fbd7806bbe0 2022-04-03T19:57:22.934 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.933+0000 7fbd65ffb700 1 -- 172.21.15.77:0/398179606 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+5352 (secure 0 0 0) 0x7fbd74015070 con 0x7fbd7806bbe0 2022-04-03T19:57:22.935 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:22.935 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:57:14.289208+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5668":{"gid":5668,"name":"d","rank":0,"incarnation":44,"state":"up:replay","state_seq":1,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:22.937 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.935+0000 7fbd7ebe5700 1 -- 172.21.15.77:0/398179606 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbd58063ce0 msgr2=0x7fbd58066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:22.937 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.935+0000 7fbd7ebe5700 1 --2- 172.21.15.77:0/398179606 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbd58063ce0 0x7fbd58066190 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fbd780c38f0 tx=0x7fbd68009060 comp rx=0 tx=0).stop 2022-04-03T19:57:22.937 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.935+0000 7fbd7ebe5700 1 -- 172.21.15.77:0/398179606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd7806bbe0 msgr2=0x7fbd781583b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:22.937 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.935+0000 7fbd7ebe5700 1 --2- 172.21.15.77:0/398179606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd7806bbe0 0x7fbd781583b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbd74009dd0 tx=0x7fbd74017000 comp rx=0 tx=0).stop 2022-04-03T19:57:22.938 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.935+0000 7fbd7ebe5700 1 -- 172.21.15.77:0/398179606 shutdown_connections 2022-04-03T19:57:22.938 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.935+0000 7fbd7ebe5700 1 --2- 172.21.15.77:0/398179606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd7806c500 0x7fbd781661f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:22.938 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.935+0000 7fbd7ebe5700 1 --2- 172.21.15.77:0/398179606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd7806bbe0 0x7fbd781583b0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:22.938 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.935+0000 7fbd7ebe5700 1 --2- 172.21.15.77:0/398179606 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbd58063ce0 0x7fbd58066190 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:22.939 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.935+0000 7fbd7ebe5700 1 --2- 172.21.15.77:0/398179606 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbd7806bfb0 0x7fbd78155640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:22.939 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.935+0000 7fbd7ebe5700 1 -- 172.21.15.77:0/398179606 >> 172.21.15.77:0/398179606 conn(0x7fbd780cb6b0 msgr2=0x7fbd78149da0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:22.939 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.935+0000 7fbd7ebe5700 1 -- 172.21.15.77:0/398179606 shutdown_connections 2022-04-03T19:57:22.939 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:22.935+0000 7fbd7ebe5700 1 -- 172.21.15.77:0/398179606 wait complete. 2022-04-03T19:57:22.940 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 44 2022-04-03T19:57:22.961 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:57:24.962 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:25.170 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.169+0000 7f45bd4ad700 1 Processor -- start 2022-04-03T19:57:25.171 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.169+0000 7f45bd4ad700 1 -- start start 2022-04-03T19:57:25.172 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.170+0000 7f45bd4ad700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b80c3640 0x7f45b8069600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:25.173 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.170+0000 7f45bd4ad700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f45b80c7af0 0x7f45b8069b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:25.173 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.170+0000 7f45bd4ad700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b80c4b90 con 0x7f45b80c3270 2022-04-03T19:57:25.173 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.170+0000 7f45bd4ad700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45b80c3d10 con 0x7f45b80c7af0 2022-04-03T19:57:25.173 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.170+0000 7f45bd4ad700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b80c3f80 con 0x7f45b80c3640 2022-04-03T19:57:25.174 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.170+0000 7f45b77fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b80c3640 0x7f45b8069600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:25.174 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.171+0000 7f45b7fff700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f45b80c3270 0x7f45b80c2500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44388/0 (socket says 172.21.15.77:44388) 2022-04-03T19:57:25.174 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.171+0000 7f45b7fff700 1 -- 172.21.15.77:0/3706095814 learned_addr learned my addr 172.21.15.77:0/3706095814 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:57:25.174 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.171+0000 7f45bccac700 1 --2- 172.21.15.77:0/3706095814 >> v2:172.21.15.77:3300/0 conn(0x7f45b80c7af0 0x7f45b8069b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.171+0000 7f45b77fe700 1 -- 172.21.15.77:0/3706095814 >> v2:172.21.15.77:3300/0 conn(0x7f45b80c7af0 msgr2=0x7f45b8069b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.171+0000 7f45b77fe700 1 --2- 172.21.15.77:0/3706095814 >> v2:172.21.15.77:3300/0 conn(0x7f45b80c7af0 0x7f45b8069b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.171+0000 7f45b77fe700 1 -- 172.21.15.77:0/3706095814 >> v1:172.21.15.186:6789/0 conn(0x7f45b80c3270 legacy=0x7f45b80c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.171+0000 7f45b6ffd700 1 -- 172.21.15.77:0/3706095814 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088164465 0 0) 0x7f45b80c4b90 con 0x7f45b80c3270 2022-04-03T19:57:25.175 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.171+0000 7f45b77fe700 1 -- 172.21.15.77:0/3706095814 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b80c4260 con 0x7f45b80c3640 2022-04-03T19:57:25.176 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.171+0000 7f45b77fe700 1 --2- 172.21.15.77:0/3706095814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b80c3640 0x7f45b8069600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f45ac009760 tx=0x7f45ac009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73ddde888703ec0d server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:25.179 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.172+0000 7f45b6ffd700 1 -- 172.21.15.77:0/3706095814 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac011630 con 0x7f45b80c3640 2022-04-03T19:57:25.179 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.172+0000 7f45b6ffd700 1 -- 172.21.15.77:0/3706095814 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ac011790 con 0x7f45b80c3640 2022-04-03T19:57:25.179 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.172+0000 7f45b6ffd700 1 -- 172.21.15.77:0/3706095814 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac011b50 con 0x7f45b80c3640 2022-04-03T19:57:25.180 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.172+0000 7f45bd4ad700 1 -- 172.21.15.77:0/3706095814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b80c3640 msgr2=0x7f45b8069600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:25.180 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.172+0000 7f45bd4ad700 1 --2- 172.21.15.77:0/3706095814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b80c3640 0x7f45b8069600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f45ac009760 tx=0x7f45ac009060 comp rx=0 tx=0).stop 2022-04-03T19:57:25.180 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.172+0000 7f45bd4ad700 1 -- 172.21.15.77:0/3706095814 shutdown_connections 2022-04-03T19:57:25.180 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.172+0000 7f45bd4ad700 1 --2- 172.21.15.77:0/3706095814 >> v2:172.21.15.77:3300/0 conn(0x7f45b80c7af0 0x7f45b8069b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:25.181 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.172+0000 7f45bd4ad700 1 --2- 172.21.15.77:0/3706095814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b80c3640 0x7f45b8069600 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:25.181 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.172+0000 7f45bd4ad700 1 -- 172.21.15.77:0/3706095814 >> 172.21.15.77:0/3706095814 conn(0x7f45b8066da0 msgr2=0x7f45b80671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:25.181 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.172+0000 7f45bd4ad700 1 -- 172.21.15.77:0/3706095814 shutdown_connections 2022-04-03T19:57:25.181 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.172+0000 7f45bd4ad700 1 -- 172.21.15.77:0/3706095814 wait complete. 2022-04-03T19:57:25.181 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.174+0000 7f45bd4ad700 1 Processor -- start 2022-04-03T19:57:25.182 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.174+0000 7f45bd4ad700 1 -- start start 2022-04-03T19:57:25.182 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.175+0000 7f45bd4ad700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45b80c3270 0x7f45b80bf100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:25.182 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.175+0000 7f45bd4ad700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b80c3640 0x7f45b8155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:25.182 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.175+0000 7f45bd4ad700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45b80c7af0 0x7f45b8155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:25.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.175+0000 7f45bd4ad700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b8168d80 con 0x7f45b80c3270 2022-04-03T19:57:25.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.175+0000 7f45bd4ad700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b80c0c80 con 0x7f45b80c7af0 2022-04-03T19:57:25.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.175+0000 7f45bd4ad700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b8168f60 con 0x7f45b80c3640 2022-04-03T19:57:25.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.175+0000 7f45b7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45b80c3270 0x7f45b80bf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:25.183 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.175+0000 7f45b77fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b80c3640 0x7f45b8155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:25.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.175+0000 7f45bccac700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45b80c7af0 0x7f45b8155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:25.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.175+0000 7f45b77fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b80c3640 0x7f45b8155630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60536/0 (socket says 172.21.15.77:60536) 2022-04-03T19:57:25.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.175+0000 7f45b77fe700 1 -- 172.21.15.77:0/2768222611 learned_addr learned my addr 172.21.15.77:0/2768222611 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:25.184 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.175+0000 7f45b77fe700 1 -- 172.21.15.77:0/2768222611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45b80c7af0 msgr2=0x7f45b8155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:25.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.175+0000 7f45b77fe700 1 --2- 172.21.15.77:0/2768222611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45b80c7af0 0x7f45b8155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:25.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.175+0000 7f45b77fe700 1 -- 172.21.15.77:0/2768222611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45b80c3270 msgr2=0x7f45b80bf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:25.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.175+0000 7f45b77fe700 1 --2- 172.21.15.77:0/2768222611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45b80c3270 0x7f45b80bf100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:25.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.176+0000 7f45b77fe700 1 -- 172.21.15.77:0/2768222611 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac0092a0 con 0x7f45b80c3640 2022-04-03T19:57:25.185 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.176+0000 7f45bccac700 1 --2- 172.21.15.77:0/2768222611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45b80c7af0 0x7f45b8155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:57:25.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.176+0000 7f45b77fe700 1 --2- 172.21.15.77:0/2768222611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b80c3640 0x7f45b8155630 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f45ac011fc0 tx=0x7f45ac017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:25.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.176+0000 7f459ffff700 1 -- 172.21.15.77:0/2768222611 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac000bc0 con 0x7f45b80c3640 2022-04-03T19:57:25.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.176+0000 7f459ffff700 1 -- 172.21.15.77:0/2768222611 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ac011900 con 0x7f45b80c3640 2022-04-03T19:57:25.186 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.176+0000 7f45bd4ad700 1 -- 172.21.15.77:0/2768222611 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b8169240 con 0x7f45b80c3640 2022-04-03T19:57:25.187 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.176+0000 7f459ffff700 1 -- 172.21.15.77:0/2768222611 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac025420 con 0x7f45b80c3640 2022-04-03T19:57:25.187 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.176+0000 7f45bd4ad700 1 -- 172.21.15.77:0/2768222611 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b80bdd20 con 0x7f45b80c3640 2022-04-03T19:57:25.187 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.177+0000 7f45bd4ad700 1 -- 172.21.15.77:0/2768222611 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b8069010 con 0x7f45b80c3640 2022-04-03T19:57:25.188 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.178+0000 7f459ffff700 1 -- 172.21.15.77:0/2768222611 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f45ac02c030 con 0x7f45b80c3640 2022-04-03T19:57:25.188 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.179+0000 7f459ffff700 1 --2- 172.21.15.77:0/2768222611 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4598063ce0 0x7f4598066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:25.188 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.179+0000 7f459ffff700 1 -- 172.21.15.77:0/2768222611 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f45ac081f30 con 0x7f45b80c3640 2022-04-03T19:57:25.189 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.179+0000 7f45b7fff700 1 --2- 172.21.15.77:0/2768222611 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4598063ce0 0x7f4598066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:25.189 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.180+0000 7f45b7fff700 1 --2- 172.21.15.77:0/2768222611 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4598063ce0 0x7f4598066190 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f45a8004780 tx=0x7f45a8007cd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:25.189 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.183+0000 7f459ffff700 1 -- 172.21.15.77:0/2768222611 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f45ac0b0ae0 con 0x7f45b80c3640 2022-04-03T19:57:25.421 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.420+0000 7f45bd4ad700 1 -- 172.21.15.77:0/2768222611 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45b8067b40 con 0x7f45b80c3640 2022-04-03T19:57:25.422 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.421+0000 7f459ffff700 1 -- 172.21.15.77:0/2768222611 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+5352 (secure 0 0 0) 0x7f45ac02a020 con 0x7f45b80c3640 2022-04-03T19:57:25.423 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:25.423 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:57:14.289208+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5668":{"gid":5668,"name":"d","rank":0,"incarnation":44,"state":"up:replay","state_seq":1,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:25.425 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.423+0000 7f45bd4ad700 1 -- 172.21.15.77:0/2768222611 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4598063ce0 msgr2=0x7f4598066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:25.425 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.423+0000 7f45bd4ad700 1 --2- 172.21.15.77:0/2768222611 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4598063ce0 0x7f4598066190 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f45a8004780 tx=0x7f45a8007cd0 comp rx=0 tx=0).stop 2022-04-03T19:57:25.426 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.423+0000 7f45bd4ad700 1 -- 172.21.15.77:0/2768222611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b80c3640 msgr2=0x7f45b8155630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:25.426 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.423+0000 7f45bd4ad700 1 --2- 172.21.15.77:0/2768222611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b80c3640 0x7f45b8155630 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f45ac011fc0 tx=0x7f45ac017000 comp rx=0 tx=0).stop 2022-04-03T19:57:25.426 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.423+0000 7f45bd4ad700 1 -- 172.21.15.77:0/2768222611 shutdown_connections 2022-04-03T19:57:25.426 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.423+0000 7f45bd4ad700 1 --2- 172.21.15.77:0/2768222611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45b80c7af0 0x7f45b8155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:25.426 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.423+0000 7f45bd4ad700 1 --2- 172.21.15.77:0/2768222611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45b80c3270 0x7f45b80bf100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:25.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.423+0000 7f45bd4ad700 1 --2- 172.21.15.77:0/2768222611 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f4598063ce0 0x7f4598066190 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:25.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.423+0000 7f45bd4ad700 1 --2- 172.21.15.77:0/2768222611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45b80c3640 0x7f45b8155630 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:25.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.423+0000 7f45bd4ad700 1 -- 172.21.15.77:0/2768222611 >> 172.21.15.77:0/2768222611 conn(0x7f45b8066da0 msgr2=0x7f45b81532f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:25.427 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.423+0000 7f45bd4ad700 1 -- 172.21.15.77:0/2768222611 shutdown_connections 2022-04-03T19:57:25.428 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:25.423+0000 7f45bd4ad700 1 -- 172.21.15.77:0/2768222611 wait complete. 2022-04-03T19:57:25.428 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 44 2022-04-03T19:57:25.450 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:57:27.451 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:27.657 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.656+0000 7f1b4759e700 1 Processor -- start 2022-04-03T19:57:27.658 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.656+0000 7f1b4759e700 1 -- start start 2022-04-03T19:57:27.661 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.657+0000 7f1b4759e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1b480c2010 0x7f1b48159050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:27.661 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.657+0000 7f1b4759e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1b4806a280 0x7f1b48155910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:27.661 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.657+0000 7f1b4759e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b480c4dc0 0x7f1b480c5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:27.661 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.657+0000 7f1b4759e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b480c4200 con 0x7f1b4806a280 2022-04-03T19:57:27.662 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.657+0000 7f1b4759e700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b48159590 con 0x7f1b480c2010 2022-04-03T19:57:27.662 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.657+0000 7f1b4759e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b48155480 con 0x7f1b480c4dc0 2022-04-03T19:57:27.662 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.657+0000 7f1b3ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1b4806a280 0x7f1b48155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:27.662 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.657+0000 7f1b3ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1b4806a280 0x7f1b48155910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43244/0 (socket says 172.21.15.77:43244) 2022-04-03T19:57:27.663 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.657+0000 7f1b3ffff700 1 -- 172.21.15.77:0/1925060343 learned_addr learned my addr 172.21.15.77:0/1925060343 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:27.663 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.657+0000 7f1b4659c700 1 --2- 172.21.15.77:0/1925060343 >> v2:172.21.15.186:3300/0 conn(0x7f1b480c2010 0x7f1b48159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:27.663 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.658+0000 7f1b46d9d700 1 --2- 172.21.15.77:0/1925060343 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b480c4dc0 0x7f1b480c5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:27.663 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.658+0000 7f1b3ffff700 1 -- 172.21.15.77:0/1925060343 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b480c4dc0 msgr2=0x7f1b480c5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:27.663 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.658+0000 7f1b3ffff700 1 --2- 172.21.15.77:0/1925060343 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b480c4dc0 0x7f1b480c5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:27.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.658+0000 7f1b3ffff700 1 -- 172.21.15.77:0/1925060343 >> v2:172.21.15.186:3300/0 conn(0x7f1b480c2010 msgr2=0x7f1b48159050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:27.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.658+0000 7f1b3ffff700 1 --2- 172.21.15.77:0/1925060343 >> v2:172.21.15.186:3300/0 conn(0x7f1b480c2010 0x7f1b48159050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:27.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.658+0000 7f1b3ffff700 1 -- 172.21.15.77:0/1925060343 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b480c2d70 con 0x7f1b4806a280 2022-04-03T19:57:27.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.658+0000 7f1b3ffff700 1 --2- 172.21.15.77:0/1925060343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b4806a280 0x7f1b48155910 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1b4806db80 tx=0x7f1b300090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a726251a5605c028 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:27.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.658+0000 7f1b45d9b700 1 -- 172.21.15.77:0/1925060343 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b30004030 con 0x7f1b4806a280 2022-04-03T19:57:27.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.659+0000 7f1b45d9b700 1 -- 172.21.15.77:0/1925060343 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b300041b0 con 0x7f1b4806a280 2022-04-03T19:57:27.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.659+0000 7f1b45d9b700 1 -- 172.21.15.77:0/1925060343 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b30011630 con 0x7f1b4806a280 2022-04-03T19:57:27.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.659+0000 7f1b4759e700 1 -- 172.21.15.77:0/1925060343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b4806a280 msgr2=0x7f1b48155910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:27.666 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.659+0000 7f1b4759e700 1 --2- 172.21.15.77:0/1925060343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b4806a280 0x7f1b48155910 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1b4806db80 tx=0x7f1b300090f0 comp rx=0 tx=0).stop 2022-04-03T19:57:27.666 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.659+0000 7f1b4759e700 1 -- 172.21.15.77:0/1925060343 shutdown_connections 2022-04-03T19:57:27.666 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.659+0000 7f1b4759e700 1 --2- 172.21.15.77:0/1925060343 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b480c4dc0 0x7f1b480c5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:27.666 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.659+0000 7f1b4759e700 1 --2- 172.21.15.77:0/1925060343 >> v2:172.21.15.186:3300/0 conn(0x7f1b480c2010 0x7f1b48159050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:27.666 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.659+0000 7f1b4759e700 1 --2- 172.21.15.77:0/1925060343 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b4806a280 0x7f1b48155910 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:27.667 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.659+0000 7f1b4759e700 1 -- 172.21.15.77:0/1925060343 >> 172.21.15.77:0/1925060343 conn(0x7f1b4806cb30 msgr2=0x7f1b4806cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:27.667 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.659+0000 7f1b4759e700 1 -- 172.21.15.77:0/1925060343 shutdown_connections 2022-04-03T19:57:27.667 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.659+0000 7f1b4759e700 1 -- 172.21.15.77:0/1925060343 wait complete. 2022-04-03T19:57:27.670 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.661+0000 7f1b4759e700 1 Processor -- start 2022-04-03T19:57:27.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.661+0000 7f1b4759e700 1 -- start start 2022-04-03T19:57:27.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.662+0000 7f1b4759e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b4806a280 0x7f1b48142b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:27.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.662+0000 7f1b4759e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b480c2010 0x7f1b48149a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:27.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.662+0000 7f1b4759e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b480c4dc0 0x7f1b48149fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:27.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.662+0000 7f1b4759e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b481486d0 con 0x7f1b480c2010 2022-04-03T19:57:27.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.662+0000 7f1b4759e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4814d150 con 0x7f1b480c4dc0 2022-04-03T19:57:27.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.662+0000 7f1b4759e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b481490b0 con 0x7f1b4806a280 2022-04-03T19:57:27.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.662+0000 7f1b3ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b480c2010 0x7f1b48149a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:27.673 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.662+0000 7f1b4659c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b4806a280 0x7f1b48142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:27.673 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.662+0000 7f1b3ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b480c2010 0x7f1b48149a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43246/0 (socket says 172.21.15.77:43246) 2022-04-03T19:57:27.673 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.662+0000 7f1b4659c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b4806a280 0x7f1b48142b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60540/0 (socket says 172.21.15.77:60540) 2022-04-03T19:57:27.673 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.662+0000 7f1b3ffff700 1 -- 172.21.15.77:0/871314934 learned_addr learned my addr 172.21.15.77:0/871314934 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.662+0000 7f1b46d9d700 1 --2- 172.21.15.77:0/871314934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b480c4dc0 0x7f1b48149fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.662+0000 7f1b3ffff700 1 -- 172.21.15.77:0/871314934 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b4806a280 msgr2=0x7f1b48142b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.662+0000 7f1b3ffff700 1 --2- 172.21.15.77:0/871314934 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b4806a280 0x7f1b48142b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:27.674 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.662+0000 7f1b3ffff700 1 -- 172.21.15.77:0/871314934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b480c4dc0 msgr2=0x7f1b48149fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.662+0000 7f1b3ffff700 1 --2- 172.21.15.77:0/871314934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b480c4dc0 0x7f1b48149fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.662+0000 7f1b3ffff700 1 -- 172.21.15.77:0/871314934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b30009370 con 0x7f1b480c2010 2022-04-03T19:57:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.663+0000 7f1b3ffff700 1 --2- 172.21.15.77:0/871314934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b480c2010 0x7f1b48149a80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1b48069920 tx=0x7f1b30016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:27.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.663+0000 7f1b3e7fc700 1 -- 172.21.15.77:0/871314934 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b300069b0 con 0x7f1b480c2010 2022-04-03T19:57:27.676 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.663+0000 7f1b3e7fc700 1 -- 172.21.15.77:0/871314934 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b300036f0 con 0x7f1b480c2010 2022-04-03T19:57:27.676 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.663+0000 7f1b4759e700 1 -- 172.21.15.77:0/871314934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b4814cab0 con 0x7f1b480c2010 2022-04-03T19:57:27.676 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.663+0000 7f1b3e7fc700 1 -- 172.21.15.77:0/871314934 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b30003ab0 con 0x7f1b480c2010 2022-04-03T19:57:27.676 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.663+0000 7f1b4759e700 1 -- 172.21.15.77:0/871314934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b48147e40 con 0x7f1b480c2010 2022-04-03T19:57:27.676 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.664+0000 7f1b4759e700 1 -- 172.21.15.77:0/871314934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b4804b050 con 0x7f1b480c2010 2022-04-03T19:57:27.677 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.665+0000 7f1b3e7fc700 1 -- 172.21.15.77:0/871314934 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f1b30017020 con 0x7f1b480c2010 2022-04-03T19:57:27.677 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.666+0000 7f1b3e7fc700 1 --2- 172.21.15.77:0/871314934 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1b2c063d10 0x7f1b2c0661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:27.677 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.666+0000 7f1b3e7fc700 1 -- 172.21.15.77:0/871314934 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f1b30081e80 con 0x7f1b480c2010 2022-04-03T19:57:27.677 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.666+0000 7f1b4659c700 1 --2- 172.21.15.77:0/871314934 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1b2c063d10 0x7f1b2c0661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:27.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.667+0000 7f1b4659c700 1 --2- 172.21.15.77:0/871314934 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1b2c063d10 0x7f1b2c0661c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1b38009c30 tx=0x7f1b38009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:27.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.670+0000 7f1b3e7fc700 1 -- 172.21.15.77:0/871314934 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f1b30050770 con 0x7f1b480c2010 2022-04-03T19:57:27.904 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.901+0000 7f1b4759e700 1 -- 172.21.15.77:0/871314934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b4806ab10 con 0x7f1b480c2010 2022-04-03T19:57:27.979 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.902+0000 7f1b3e7fc700 1 -- 172.21.15.77:0/871314934 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+5356 (secure 0 0 0) 0x7f1b30029020 con 0x7f1b480c2010 2022-04-03T19:57:27.979 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:27.980 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:57:27.301126+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5668":{"gid":5668,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:27.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.904+0000 7f1b4759e700 1 -- 172.21.15.77:0/871314934 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1b2c063d10 msgr2=0x7f1b2c0661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:27.982 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.904+0000 7f1b4759e700 1 --2- 172.21.15.77:0/871314934 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1b2c063d10 0x7f1b2c0661c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1b38009c30 tx=0x7f1b38009060 comp rx=0 tx=0).stop 2022-04-03T19:57:27.983 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.904+0000 7f1b4759e700 1 -- 172.21.15.77:0/871314934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b480c2010 msgr2=0x7f1b48149a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.904+0000 7f1b4759e700 1 --2- 172.21.15.77:0/871314934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b480c2010 0x7f1b48149a80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1b48069920 tx=0x7f1b30016000 comp rx=0 tx=0).stop 2022-04-03T19:57:27.984 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.904+0000 7f1b4759e700 1 -- 172.21.15.77:0/871314934 shutdown_connections 2022-04-03T19:57:27.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.904+0000 7f1b4759e700 1 --2- 172.21.15.77:0/871314934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b480c4dc0 0x7f1b48149fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:27.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.904+0000 7f1b4759e700 1 --2- 172.21.15.77:0/871314934 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1b2c063d10 0x7f1b2c0661c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:27.985 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.904+0000 7f1b4759e700 1 --2- 172.21.15.77:0/871314934 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b4806a280 0x7f1b48142b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:27.986 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.904+0000 7f1b4759e700 1 --2- 172.21.15.77:0/871314934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1b480c2010 0x7f1b48149a80 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:27.986 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.904+0000 7f1b4759e700 1 -- 172.21.15.77:0/871314934 >> 172.21.15.77:0/871314934 conn(0x7f1b4806cb30 msgr2=0x7f1b48152fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:27.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.905+0000 7f1b4759e700 1 -- 172.21.15.77:0/871314934 shutdown_connections 2022-04-03T19:57:27.987 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:27.905+0000 7f1b4759e700 1 -- 172.21.15.77:0/871314934 wait complete. 2022-04-03T19:57:27.988 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 45 2022-04-03T19:57:27.990 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 45, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-04-03T19:44:21.275716+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 45, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5668": { "addr": "172.21.15.186:6837/113043607", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 113043607, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 113043607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 5668, "incarnation": 44, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 27 } }, "last_failure": 0, "last_failure_osd_epoch": 53, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-04-03T19:57:27.301126+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5668 } } } ], "standbys": [ { "addr": "172.21.15.77:6834/2116223509", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6832", "nonce": 2116223509, "type": "v2" }, { "addr": "172.21.15.77:6834", "nonce": 2116223509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4245, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6836/2374276747", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2374276747, "type": "v2" }, { "addr": "172.21.15.186:6836", "nonce": 2374276747, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4259, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/610990853", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6838", "nonce": 610990853, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 610990853, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4265, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.77:6835/170053412", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6833", "nonce": 170053412, "type": "v2" }, { "addr": "172.21.15.77:6835", "nonce": 170053412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4271, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-04-03T19:57:29.991 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:30.195 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.194+0000 7fac5ded6700 1 Processor -- start 2022-04-03T19:57:30.196 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.194+0000 7fac5ded6700 1 -- start start 2022-04-03T19:57:30.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.195+0000 7fac5ded6700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fac5806bbe0 0x7fac581584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:30.199 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.195+0000 7fac5ded6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac5806bfb0 0x7fac5806d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:30.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.195+0000 7fac5ded6700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5806d260 con 0x7fac5806c500 2022-04-03T19:57:30.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.195+0000 7fac5ded6700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac58159450 con 0x7fac5806bbe0 2022-04-03T19:57:30.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.195+0000 7fac5ded6700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac580c6330 con 0x7fac5806bfb0 2022-04-03T19:57:30.200 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.195+0000 7fac5d6d5700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fac5806c500 0x7fac5814eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38582/0 (socket says 172.21.15.77:38582) 2022-04-03T19:57:30.201 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.195+0000 7fac5d6d5700 1 -- 172.21.15.77:0/3372866276 learned_addr learned my addr 172.21.15.77:0/3372866276 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:57:30.201 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.195+0000 7fac5ced4700 1 --2- 172.21.15.77:0/3372866276 >> v2:172.21.15.186:3300/0 conn(0x7fac5806bbe0 0x7fac581584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:30.201 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.195+0000 7fac4ffff700 1 --2- 172.21.15.77:0/3372866276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac5806bfb0 0x7fac5806d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:30.202 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.196+0000 7fac4f7fe700 1 -- 172.21.15.77:0/3372866276 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366687487 0 0) 0x7fac5806d260 con 0x7fac5806c500 2022-04-03T19:57:30.202 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.196+0000 7fac4f7fe700 1 -- 172.21.15.77:0/3372866276 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac440033f0 con 0x7fac5806c500 2022-04-03T19:57:30.202 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.196+0000 7fac4ffff700 1 -- 172.21.15.77:0/3372866276 >> v2:172.21.15.186:3300/0 conn(0x7fac5806bbe0 msgr2=0x7fac581584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:30.202 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.196+0000 7fac4ffff700 1 --2- 172.21.15.77:0/3372866276 >> v2:172.21.15.186:3300/0 conn(0x7fac5806bbe0 0x7fac581584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:30.203 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.196+0000 7fac4ffff700 1 -- 172.21.15.77:0/3372866276 >> v1:172.21.15.77:6789/0 conn(0x7fac5806c500 legacy=0x7fac5814eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:30.203 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.196+0000 7fac4ffff700 1 -- 172.21.15.77:0/3372866276 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac58158bb0 con 0x7fac5806bfb0 2022-04-03T19:57:30.203 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.196+0000 7fac4ffff700 1 --2- 172.21.15.77:0/3372866276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac5806bfb0 0x7fac5806d6f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fac48009760 tx=0x7fac48009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef90469013d3dd7e server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:30.203 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.196+0000 7fac4f7fe700 1 -- 172.21.15.77:0/3372866276 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac480105b0 con 0x7fac5806bfb0 2022-04-03T19:57:30.204 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.197+0000 7fac4f7fe700 1 -- 172.21.15.77:0/3372866276 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac48010710 con 0x7fac5806bfb0 2022-04-03T19:57:30.204 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.197+0000 7fac4f7fe700 1 -- 172.21.15.77:0/3372866276 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48010ab0 con 0x7fac5806bfb0 2022-04-03T19:57:30.204 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.197+0000 7fac5ded6700 1 -- 172.21.15.77:0/3372866276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac5806bfb0 msgr2=0x7fac5806d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:30.204 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.197+0000 7fac5ded6700 1 --2- 172.21.15.77:0/3372866276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac5806bfb0 0x7fac5806d6f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fac48009760 tx=0x7fac48009060 comp rx=0 tx=0).stop 2022-04-03T19:57:30.205 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.197+0000 7fac5ded6700 1 -- 172.21.15.77:0/3372866276 shutdown_connections 2022-04-03T19:57:30.205 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.197+0000 7fac5ded6700 1 --2- 172.21.15.77:0/3372866276 >> v2:172.21.15.186:3300/0 conn(0x7fac5806bbe0 0x7fac581584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:30.205 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.197+0000 7fac5ded6700 1 --2- 172.21.15.77:0/3372866276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac5806bfb0 0x7fac5806d6f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:30.205 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.197+0000 7fac5ded6700 1 -- 172.21.15.77:0/3372866276 >> 172.21.15.77:0/3372866276 conn(0x7fac580cb6b0 msgr2=0x7fac580cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:30.206 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.197+0000 7fac5ded6700 1 -- 172.21.15.77:0/3372866276 shutdown_connections 2022-04-03T19:57:30.206 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.197+0000 7fac5ded6700 1 -- 172.21.15.77:0/3372866276 wait complete. 2022-04-03T19:57:30.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.199+0000 7fac5ded6700 1 Processor -- start 2022-04-03T19:57:30.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.199+0000 7fac5ded6700 1 -- start start 2022-04-03T19:57:30.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.200+0000 7fac5ded6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac5806bbe0 0x7fac580c17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:30.210 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.200+0000 7fac5ded6700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac5806bfb0 0x7fac580c0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:30.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.200+0000 7fac5ded6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac5806c500 0x7fac580c07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:30.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.200+0000 7fac5ded6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac58153060 con 0x7fac5806bbe0 2022-04-03T19:57:30.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.200+0000 7fac5ded6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac58153340 con 0x7fac5806c500 2022-04-03T19:57:30.211 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.200+0000 7fac5ded6700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac580bf4c0 con 0x7fac5806bfb0 2022-04-03T19:57:30.212 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.200+0000 7fac5ced4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac5806bbe0 0x7fac580c17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:30.212 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.200+0000 7fac5ced4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac5806bbe0 0x7fac580c17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43248/0 (socket says 172.21.15.77:43248) 2022-04-03T19:57:30.212 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.200+0000 7fac5ced4700 1 -- 172.21.15.77:0/1950998802 learned_addr learned my addr 172.21.15.77:0/1950998802 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:30.212 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.200+0000 7fac5d6d5700 1 --2- 172.21.15.77:0/1950998802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac5806c500 0x7fac580c07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:30.213 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.200+0000 7fac4ffff700 1 --2- 172.21.15.77:0/1950998802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac5806bfb0 0x7fac580c0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:30.213 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.200+0000 7fac5ced4700 1 -- 172.21.15.77:0/1950998802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac5806bfb0 msgr2=0x7fac580c0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:30.213 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.200+0000 7fac5ced4700 1 --2- 172.21.15.77:0/1950998802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac5806bfb0 0x7fac580c0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:30.213 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.200+0000 7fac5ced4700 1 -- 172.21.15.77:0/1950998802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac5806c500 msgr2=0x7fac580c07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:30.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.200+0000 7fac5ced4700 1 --2- 172.21.15.77:0/1950998802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac5806c500 0x7fac580c07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:30.214 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.200+0000 7fac5ced4700 1 -- 172.21.15.77:0/1950998802 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac480092a0 con 0x7fac5806bbe0 2022-04-03T19:57:30.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.201+0000 7fac5ced4700 1 --2- 172.21.15.77:0/1950998802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac5806bbe0 0x7fac580c17b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fac5400e930 tx=0x7fac54009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:30.221 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.201+0000 7fac4cff9700 1 -- 172.21.15.77:0/1950998802 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac54013680 con 0x7fac5806bbe0 2022-04-03T19:57:30.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.201+0000 7fac4cff9700 1 -- 172.21.15.77:0/1950998802 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac540137e0 con 0x7fac5806bbe0 2022-04-03T19:57:30.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.201+0000 7fac4cff9700 1 -- 172.21.15.77:0/1950998802 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac54013b60 con 0x7fac5806bbe0 2022-04-03T19:57:30.222 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.201+0000 7fac5ded6700 1 -- 172.21.15.77:0/1950998802 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac580bf7a0 con 0x7fac5806bbe0 2022-04-03T19:57:30.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.201+0000 7fac5ded6700 1 -- 172.21.15.77:0/1950998802 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac580bfd60 con 0x7fac5806bbe0 2022-04-03T19:57:30.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.202+0000 7fac5ded6700 1 -- 172.21.15.77:0/1950998802 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5814c270 con 0x7fac5806bbe0 2022-04-03T19:57:30.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.203+0000 7fac4cff9700 1 -- 172.21.15.77:0/1950998802 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fac54019030 con 0x7fac5806bbe0 2022-04-03T19:57:30.223 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.204+0000 7fac4cff9700 1 --2- 172.21.15.77:0/1950998802 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fac38063ce0 0x7fac38066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:30.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.204+0000 7fac4cff9700 1 -- 172.21.15.77:0/1950998802 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6718+0+0 (secure 0 0 0) 0x7fac54080ae0 con 0x7fac5806bbe0 2022-04-03T19:57:30.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.204+0000 7fac4ffff700 1 --2- 172.21.15.77:0/1950998802 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fac38063ce0 0x7fac38066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:30.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.205+0000 7fac4ffff700 1 --2- 172.21.15.77:0/1950998802 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fac38063ce0 0x7fac38066190 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fac581583f0 tx=0x7fac48016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:30.224 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.209+0000 7fac4cff9700 1 -- 172.21.15.77:0/1950998802 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fac5404f340 con 0x7fac5806bbe0 2022-04-03T19:57:30.442 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.441+0000 7fac5ded6700 1 -- 172.21.15.77:0/1950998802 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac580c00d0 con 0x7fac5806bbe0 2022-04-03T19:57:30.443 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.443+0000 7fac4cff9700 1 -- 172.21.15.77:0/1950998802 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+5356 (secure 0 0 0) 0x7fac54052960 con 0x7fac5806bbe0 2022-04-03T19:57:30.444 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:30.444 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:57:27.301126+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5668":{"gid":5668,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:30.445 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.444+0000 7fac5ded6700 1 -- 172.21.15.77:0/1950998802 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fac38063ce0 msgr2=0x7fac38066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:30.446 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.444+0000 7fac5ded6700 1 --2- 172.21.15.77:0/1950998802 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fac38063ce0 0x7fac38066190 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fac581583f0 tx=0x7fac48016000 comp rx=0 tx=0).stop 2022-04-03T19:57:30.446 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.444+0000 7fac5ded6700 1 -- 172.21.15.77:0/1950998802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac5806bbe0 msgr2=0x7fac580c17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:30.446 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.444+0000 7fac5ded6700 1 --2- 172.21.15.77:0/1950998802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac5806bbe0 0x7fac580c17b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fac5400e930 tx=0x7fac54009060 comp rx=0 tx=0).stop 2022-04-03T19:57:30.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.444+0000 7fac5ded6700 1 -- 172.21.15.77:0/1950998802 shutdown_connections 2022-04-03T19:57:30.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.444+0000 7fac5ded6700 1 --2- 172.21.15.77:0/1950998802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac5806c500 0x7fac580c07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:30.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.444+0000 7fac5ded6700 1 --2- 172.21.15.77:0/1950998802 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fac5806bbe0 0x7fac580c17b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:30.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.444+0000 7fac5ded6700 1 --2- 172.21.15.77:0/1950998802 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fac38063ce0 0x7fac38066190 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:30.447 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.444+0000 7fac5ded6700 1 --2- 172.21.15.77:0/1950998802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac5806bfb0 0x7fac580c0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:30.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.444+0000 7fac5ded6700 1 -- 172.21.15.77:0/1950998802 >> 172.21.15.77:0/1950998802 conn(0x7fac580cb6b0 msgr2=0x7fac580c50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:30.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.444+0000 7fac5ded6700 1 -- 172.21.15.77:0/1950998802 shutdown_connections 2022-04-03T19:57:30.448 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:30.444+0000 7fac5ded6700 1 -- 172.21.15.77:0/1950998802 wait complete. 2022-04-03T19:57:30.449 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 45 2022-04-03T19:57:30.470 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:57:32.471 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:32.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.677+0000 7fd4e2612700 1 Processor -- start 2022-04-03T19:57:32.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.677+0000 7fd4e2612700 1 -- start start 2022-04-03T19:57:32.682 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.678+0000 7fd4e2612700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd4dc06bbe0 0x7fd4dc1584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:32.682 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.679+0000 7fd4e2612700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd4dc06c500 0x7fd4dc14eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:32.683 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.679+0000 7fd4e2612700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4dc06d260 con 0x7fd4dc06bfb0 2022-04-03T19:57:32.683 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.679+0000 7fd4e2612700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4dc159450 con 0x7fd4dc06c500 2022-04-03T19:57:32.683 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.679+0000 7fd4e2612700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4dc0c6330 con 0x7fd4dc06bbe0 2022-04-03T19:57:32.684 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.679+0000 7fd4e0e0f700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd4dc06bfb0 0x7fd4dc06d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38584/0 (socket says 172.21.15.77:38584) 2022-04-03T19:57:32.684 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.679+0000 7fd4e0e0f700 1 -- 172.21.15.77:0/125639629 learned_addr learned my addr 172.21.15.77:0/125639629 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:57:32.684 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.679+0000 7fd4e1610700 1 --2- 172.21.15.77:0/125639629 >> v2:172.21.15.186:3300/0 conn(0x7fd4dc06bbe0 0x7fd4dc1584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:32.684 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.679+0000 7fd4e1e11700 1 --2- 172.21.15.77:0/125639629 >> v2:172.21.15.77:3300/0 conn(0x7fd4dc06c500 0x7fd4dc14eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:32.685 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.679+0000 7fd4cbfff700 1 -- 172.21.15.77:0/125639629 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667949139 0 0) 0x7fd4dc06d260 con 0x7fd4dc06bfb0 2022-04-03T19:57:32.685 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.679+0000 7fd4cbfff700 1 -- 172.21.15.77:0/125639629 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4c40033f0 con 0x7fd4dc06bfb0 2022-04-03T19:57:32.685 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.680+0000 7fd4e1610700 1 -- 172.21.15.77:0/125639629 >> v2:172.21.15.77:3300/0 conn(0x7fd4dc06c500 msgr2=0x7fd4dc14eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:32.686 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.680+0000 7fd4e1610700 1 --2- 172.21.15.77:0/125639629 >> v2:172.21.15.77:3300/0 conn(0x7fd4dc06c500 0x7fd4dc14eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:32.686 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.680+0000 7fd4e1610700 1 -- 172.21.15.77:0/125639629 >> v1:172.21.15.77:6789/0 conn(0x7fd4dc06bfb0 legacy=0x7fd4dc06d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:32.686 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.680+0000 7fd4e1610700 1 -- 172.21.15.77:0/125639629 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4dc158bb0 con 0x7fd4dc06bbe0 2022-04-03T19:57:32.686 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.680+0000 7fd4e1610700 1 --2- 172.21.15.77:0/125639629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4dc06bbe0 0x7fd4dc1584b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc088d40 tx=0x7fd4d8009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dba0d0f4c771520 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:32.687 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.680+0000 7fd4cbfff700 1 -- 172.21.15.77:0/125639629 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d80105b0 con 0x7fd4dc06bbe0 2022-04-03T19:57:32.687 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.680+0000 7fd4cbfff700 1 -- 172.21.15.77:0/125639629 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d8010710 con 0x7fd4dc06bbe0 2022-04-03T19:57:32.687 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.680+0000 7fd4cbfff700 1 -- 172.21.15.77:0/125639629 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8010ad0 con 0x7fd4dc06bbe0 2022-04-03T19:57:32.687 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.681+0000 7fd4e2612700 1 -- 172.21.15.77:0/125639629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4dc06bbe0 msgr2=0x7fd4dc1584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:32.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.681+0000 7fd4e2612700 1 --2- 172.21.15.77:0/125639629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4dc06bbe0 0x7fd4dc1584b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc088d40 tx=0x7fd4d8009060 comp rx=0 tx=0).stop 2022-04-03T19:57:32.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.681+0000 7fd4e2612700 1 -- 172.21.15.77:0/125639629 shutdown_connections 2022-04-03T19:57:32.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.681+0000 7fd4e2612700 1 --2- 172.21.15.77:0/125639629 >> v2:172.21.15.77:3300/0 conn(0x7fd4dc06c500 0x7fd4dc14eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:32.688 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.681+0000 7fd4e2612700 1 --2- 172.21.15.77:0/125639629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4dc06bbe0 0x7fd4dc1584b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:32.689 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.681+0000 7fd4e2612700 1 -- 172.21.15.77:0/125639629 >> 172.21.15.77:0/125639629 conn(0x7fd4dc0cb6b0 msgr2=0x7fd4dc0cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:32.689 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.681+0000 7fd4e2612700 1 -- 172.21.15.77:0/125639629 shutdown_connections 2022-04-03T19:57:32.689 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.681+0000 7fd4e2612700 1 -- 172.21.15.77:0/125639629 wait complete. 2022-04-03T19:57:32.692 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.682+0000 7fd4e2612700 1 Processor -- start 2022-04-03T19:57:32.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.683+0000 7fd4e2612700 1 -- start start 2022-04-03T19:57:32.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.683+0000 7fd4e2612700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4dc06bbe0 0x7fd4dc164f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:32.693 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.683+0000 7fd4e2612700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4dc06bfb0 0x7fd4dc152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:32.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.683+0000 7fd4e2612700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4dc06c500 0x7fd4dc157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:32.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.683+0000 7fd4e2612700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc165f00 con 0x7fd4dc06bbe0 2022-04-03T19:57:32.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.683+0000 7fd4e2612700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc158840 con 0x7fd4dc06bfb0 2022-04-03T19:57:32.694 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.683+0000 7fd4e2612700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc154750 con 0x7fd4dc06c500 2022-04-03T19:57:32.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.684+0000 7fd4e0e0f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4dc06bfb0 0x7fd4dc152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:32.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.684+0000 7fd4e0e0f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4dc06bfb0 0x7fd4dc152bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38440/0 (socket says 172.21.15.77:38440) 2022-04-03T19:57:32.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.684+0000 7fd4e1610700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4dc06bbe0 0x7fd4dc164f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:32.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.684+0000 7fd4e0e0f700 1 -- 172.21.15.77:0/3806489790 learned_addr learned my addr 172.21.15.77:0/3806489790 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:32.695 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.684+0000 7fd4e1e11700 1 --2- 172.21.15.77:0/3806489790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4dc06c500 0x7fd4dc157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:32.696 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.684+0000 7fd4e0e0f700 1 -- 172.21.15.77:0/3806489790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4dc06c500 msgr2=0x7fd4dc157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:32.696 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.684+0000 7fd4e0e0f700 1 --2- 172.21.15.77:0/3806489790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4dc06c500 0x7fd4dc157d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:32.696 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.684+0000 7fd4e0e0f700 1 -- 172.21.15.77:0/3806489790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4dc06bbe0 msgr2=0x7fd4dc164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:32.696 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.684+0000 7fd4e0e0f700 1 --2- 172.21.15.77:0/3806489790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4dc06bbe0 0x7fd4dc164f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:32.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.684+0000 7fd4e0e0f700 1 -- 172.21.15.77:0/3806489790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d80092a0 con 0x7fd4dc06bfb0 2022-04-03T19:57:32.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.685+0000 7fd4e0e0f700 1 --2- 172.21.15.77:0/3806489790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4dc06bfb0 0x7fd4dc152bf0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc00c490 tx=0x7fd4cc007c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:32.697 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.685+0000 7fd4c97fa700 1 -- 172.21.15.77:0/3806489790 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc00ce40 con 0x7fd4dc06bfb0 2022-04-03T19:57:32.698 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.685+0000 7fd4c97fa700 1 -- 172.21.15.77:0/3806489790 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4cc0072e0 con 0x7fd4dc06bfb0 2022-04-03T19:57:32.698 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.685+0000 7fd4e2612700 1 -- 172.21.15.77:0/3806489790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4dc153130 con 0x7fd4dc06bfb0 2022-04-03T19:57:32.698 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.685+0000 7fd4c97fa700 1 -- 172.21.15.77:0/3806489790 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc005780 con 0x7fd4dc06bfb0 2022-04-03T19:57:32.698 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.685+0000 7fd4e2612700 1 -- 172.21.15.77:0/3806489790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4dc153660 con 0x7fd4dc06bfb0 2022-04-03T19:57:32.699 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.686+0000 7fd4e2612700 1 -- 172.21.15.77:0/3806489790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4dc04b050 con 0x7fd4dc06bfb0 2022-04-03T19:57:32.699 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.687+0000 7fd4c97fa700 1 -- 172.21.15.77:0/3806489790 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fd4cc019030 con 0x7fd4dc06bfb0 2022-04-03T19:57:32.699 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.688+0000 7fd4c97fa700 1 --2- 172.21.15.77:0/3806489790 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd4bc063ce0 0x7fd4bc066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:32.699 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.688+0000 7fd4c97fa700 1 -- 172.21.15.77:0/3806489790 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6718+0+0 (secure 0 0 0) 0x7fd4cc07fe30 con 0x7fd4dc06bfb0 2022-04-03T19:57:32.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.688+0000 7fd4e1610700 1 --2- 172.21.15.77:0/3806489790 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd4bc063ce0 0x7fd4bc066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:32.700 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.689+0000 7fd4e1610700 1 --2- 172.21.15.77:0/3806489790 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd4bc063ce0 0x7fd4bc066190 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8009fd0 tx=0x7fd4d8016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:32.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.692+0000 7fd4c97fa700 1 -- 172.21.15.77:0/3806489790 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fd4cc04e690 con 0x7fd4dc06bfb0 2022-04-03T19:57:32.927 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.926+0000 7fd4e2612700 1 -- 172.21.15.77:0/3806489790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4dc158980 con 0x7fd4dc06bfb0 2022-04-03T19:57:32.928 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.927+0000 7fd4c97fa700 1 -- 172.21.15.77:0/3806489790 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5353 (secure 0 0 0) 0x7fd4cc017070 con 0x7fd4dc06bfb0 2022-04-03T19:57:32.928 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:32.929 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:57:32.318895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5668":{"gid":5668,"name":"d","rank":0,"incarnation":44,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:32.931 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.930+0000 7fd4e2612700 1 -- 172.21.15.77:0/3806489790 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd4bc063ce0 msgr2=0x7fd4bc066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:32.931 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.930+0000 7fd4e2612700 1 --2- 172.21.15.77:0/3806489790 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd4bc063ce0 0x7fd4bc066190 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8009fd0 tx=0x7fd4d8016000 comp rx=0 tx=0).stop 2022-04-03T19:57:32.931 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.930+0000 7fd4e2612700 1 -- 172.21.15.77:0/3806489790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4dc06bfb0 msgr2=0x7fd4dc152bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:32.931 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.930+0000 7fd4e2612700 1 --2- 172.21.15.77:0/3806489790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4dc06bfb0 0x7fd4dc152bf0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc00c490 tx=0x7fd4cc007c70 comp rx=0 tx=0).stop 2022-04-03T19:57:32.932 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.930+0000 7fd4e2612700 1 -- 172.21.15.77:0/3806489790 shutdown_connections 2022-04-03T19:57:32.932 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.930+0000 7fd4e2612700 1 --2- 172.21.15.77:0/3806489790 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fd4bc063ce0 0x7fd4bc066190 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:32.932 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.930+0000 7fd4e2612700 1 --2- 172.21.15.77:0/3806489790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4dc06c500 0x7fd4dc157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:32.932 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.930+0000 7fd4e2612700 1 --2- 172.21.15.77:0/3806489790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4dc06bbe0 0x7fd4dc164f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:32.933 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.930+0000 7fd4e2612700 1 --2- 172.21.15.77:0/3806489790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4dc06bfb0 0x7fd4dc152bf0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:32.933 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.930+0000 7fd4e2612700 1 -- 172.21.15.77:0/3806489790 >> 172.21.15.77:0/3806489790 conn(0x7fd4dc0cb6b0 msgr2=0x7fd4dc14a500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:32.933 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.930+0000 7fd4e2612700 1 -- 172.21.15.77:0/3806489790 shutdown_connections 2022-04-03T19:57:32.933 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:32.930+0000 7fd4e2612700 1 -- 172.21.15.77:0/3806489790 wait complete. 2022-04-03T19:57:32.934 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 46 2022-04-03T19:57:32.955 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:57:34.956 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:35.162 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.161+0000 7f72e759e700 1 Processor -- start 2022-04-03T19:57:35.162 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.161+0000 7f72e759e700 1 -- start start 2022-04-03T19:57:35.164 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.162+0000 7f72e759e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72e806bbe0 0x7f72e81584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:35.164 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.162+0000 7f72e759e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72e806c500 0x7f72e814eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:35.165 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.162+0000 7f72e759e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e806d260 con 0x7f72e806bfb0 2022-04-03T19:57:35.165 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.162+0000 7f72e759e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e8159450 con 0x7f72e806bbe0 2022-04-03T19:57:35.165 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.162+0000 7f72e759e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e80c6330 con 0x7f72e806c500 2022-04-03T19:57:35.165 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.163+0000 7f72e659c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72e806bbe0 0x7f72e81584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:35.165 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.163+0000 7f72e659c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72e806bbe0 0x7f72e81584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43254/0 (socket says 172.21.15.77:43254) 2022-04-03T19:57:35.166 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.163+0000 7f72e659c700 1 -- 172.21.15.77:0/1058973134 learned_addr learned my addr 172.21.15.77:0/1058973134 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:35.166 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.163+0000 7f72e6d9d700 1 --2- 172.21.15.77:0/1058973134 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72e806c500 0x7f72e814eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:35.166 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.163+0000 7f72e659c700 1 -- 172.21.15.77:0/1058973134 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72e806c500 msgr2=0x7f72e814eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:35.166 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.163+0000 7f72e659c700 1 --2- 172.21.15.77:0/1058973134 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72e806c500 0x7f72e814eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:35.167 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.163+0000 7f72e659c700 1 -- 172.21.15.77:0/1058973134 >> v1:172.21.15.186:6789/0 conn(0x7f72e806bfb0 legacy=0x7f72e806d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:35.167 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.163+0000 7f72e659c700 1 -- 172.21.15.77:0/1058973134 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e8158bb0 con 0x7f72e806bbe0 2022-04-03T19:57:35.167 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.163+0000 7f72e659c700 1 --2- 172.21.15.77:0/1058973134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e806bbe0 0x7f72e81584b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f72d8006fd0 tx=0x7f72d80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f17d2a4cdba557 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:35.167 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.164+0000 7f72e5d9b700 1 -- 172.21.15.77:0/1058973134 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d8004030 con 0x7f72e806bbe0 2022-04-03T19:57:35.167 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.164+0000 7f72e5d9b700 1 -- 172.21.15.77:0/1058973134 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d80041d0 con 0x7f72e806bbe0 2022-04-03T19:57:35.168 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.164+0000 7f72e5d9b700 1 -- 172.21.15.77:0/1058973134 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d8011630 con 0x7f72e806bbe0 2022-04-03T19:57:35.171 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.164+0000 7f72e759e700 1 -- 172.21.15.77:0/1058973134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e806bbe0 msgr2=0x7f72e81584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:35.171 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.164+0000 7f72e759e700 1 --2- 172.21.15.77:0/1058973134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e806bbe0 0x7f72e81584b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f72d8006fd0 tx=0x7f72d80090f0 comp rx=0 tx=0).stop 2022-04-03T19:57:35.171 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.164+0000 7f72e759e700 1 -- 172.21.15.77:0/1058973134 shutdown_connections 2022-04-03T19:57:35.172 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.164+0000 7f72e759e700 1 --2- 172.21.15.77:0/1058973134 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72e806c500 0x7f72e814eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:35.172 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.164+0000 7f72e759e700 1 --2- 172.21.15.77:0/1058973134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e806bbe0 0x7f72e81584b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:35.172 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.164+0000 7f72e759e700 1 -- 172.21.15.77:0/1058973134 >> 172.21.15.77:0/1058973134 conn(0x7f72e80cb6b0 msgr2=0x7f72e80cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:35.172 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.164+0000 7f72e759e700 1 -- 172.21.15.77:0/1058973134 shutdown_connections 2022-04-03T19:57:35.172 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.165+0000 7f72e759e700 1 -- 172.21.15.77:0/1058973134 wait complete. 2022-04-03T19:57:35.173 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.166+0000 7f72e759e700 1 Processor -- start 2022-04-03T19:57:35.173 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.166+0000 7f72e759e700 1 -- start start 2022-04-03T19:57:35.173 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.167+0000 7f72e759e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e806bbe0 0x7f72e8152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:35.173 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.167+0000 7f72e759e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72e806bfb0 0x7f72e80c1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:35.174 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.167+0000 7f72e759e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72e806c500 0x7f72e8168af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:35.174 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.167+0000 7f72e759e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e80c2e40 con 0x7f72e806bbe0 2022-04-03T19:57:35.174 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.167+0000 7f72e759e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e80c0620 con 0x7f72e806c500 2022-04-03T19:57:35.174 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.167+0000 7f72e759e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e8153180 con 0x7f72e806bfb0 2022-04-03T19:57:35.174 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.167+0000 7f72e659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e806bbe0 0x7f72e8152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:35.175 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.167+0000 7f72dffff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72e806bfb0 0x7f72e80c1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:35.175 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.167+0000 7f72e6d9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72e806c500 0x7f72e8168af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:35.175 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.167+0000 7f72e659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e806bbe0 0x7f72e8152c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43256/0 (socket says 172.21.15.77:43256) 2022-04-03T19:57:35.175 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.167+0000 7f72e659c700 1 -- 172.21.15.77:0/1647580632 learned_addr learned my addr 172.21.15.77:0/1647580632 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:35.176 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.168+0000 7f72e659c700 1 -- 172.21.15.77:0/1647580632 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72e806bfb0 msgr2=0x7f72e80c1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:35.176 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.168+0000 7f72e659c700 1 --2- 172.21.15.77:0/1647580632 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72e806bfb0 0x7f72e80c1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:35.176 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.168+0000 7f72e659c700 1 -- 172.21.15.77:0/1647580632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72e806c500 msgr2=0x7f72e8168af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:35.176 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.168+0000 7f72e659c700 1 --2- 172.21.15.77:0/1647580632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72e806c500 0x7f72e8168af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:35.176 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.168+0000 7f72e659c700 1 -- 172.21.15.77:0/1647580632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d8009370 con 0x7f72e806bbe0 2022-04-03T19:57:35.177 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.168+0000 7f72e659c700 1 --2- 172.21.15.77:0/1647580632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e806bbe0 0x7f72e8152c40 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f72d8004000 tx=0x7f72d8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:35.177 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.168+0000 7f72de7fc700 1 -- 172.21.15.77:0/1647580632 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d80036f0 con 0x7f72e806bbe0 2022-04-03T19:57:35.177 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.168+0000 7f72de7fc700 1 -- 172.21.15.77:0/1647580632 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d8003870 con 0x7f72e806bbe0 2022-04-03T19:57:35.177 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.168+0000 7f72e759e700 1 -- 172.21.15.77:0/1647580632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e80c28c0 con 0x7f72e806bbe0 2022-04-03T19:57:35.178 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.168+0000 7f72de7fc700 1 -- 172.21.15.77:0/1647580632 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d8003c30 con 0x7f72e806bbe0 2022-04-03T19:57:35.178 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.169+0000 7f72e759e700 1 -- 172.21.15.77:0/1647580632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e80bffd0 con 0x7f72e806bbe0 2022-04-03T19:57:35.178 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.170+0000 7f72e759e700 1 -- 172.21.15.77:0/1647580632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e80bfe30 con 0x7f72e806bbe0 2022-04-03T19:57:35.178 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.170+0000 7f72de7fc700 1 -- 172.21.15.77:0/1647580632 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f72d8017020 con 0x7f72e806bbe0 2022-04-03T19:57:35.179 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.172+0000 7f72de7fc700 1 --2- 172.21.15.77:0/1647580632 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f72cc063ce0 0x7f72cc066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:35.179 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.172+0000 7f72de7fc700 1 -- 172.21.15.77:0/1647580632 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f72d8082340 con 0x7f72e806bbe0 2022-04-03T19:57:35.180 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.172+0000 7f72dffff700 1 --2- 172.21.15.77:0/1647580632 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f72cc063ce0 0x7f72cc066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:35.180 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.173+0000 7f72dffff700 1 --2- 172.21.15.77:0/1647580632 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f72cc063ce0 0x7f72cc066190 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f72e80c6120 tx=0x7f72d0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:35.180 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.177+0000 7f72de7fc700 1 -- 172.21.15.77:0/1647580632 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f72e80bfe30 con 0x7f72e806bbe0 2022-04-03T19:57:35.411 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.411+0000 7f72e759e700 1 -- 172.21.15.77:0/1647580632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72e80bfe30 con 0x7f72e806bbe0 2022-04-03T19:57:35.412 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.411+0000 7f72de7fc700 1 -- 172.21.15.77:0/1647580632 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5353 (secure 0 0 0) 0x7f72e80bfe30 con 0x7f72e806bbe0 2022-04-03T19:57:35.412 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:35.413 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:57:32.318895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5668":{"gid":5668,"name":"d","rank":0,"incarnation":44,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:35.414 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.414+0000 7f72e759e700 1 -- 172.21.15.77:0/1647580632 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f72cc063ce0 msgr2=0x7f72cc066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:35.415 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.414+0000 7f72e759e700 1 --2- 172.21.15.77:0/1647580632 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f72cc063ce0 0x7f72cc066190 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f72e80c6120 tx=0x7f72d0009040 comp rx=0 tx=0).stop 2022-04-03T19:57:35.415 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.414+0000 7f72e759e700 1 -- 172.21.15.77:0/1647580632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e806bbe0 msgr2=0x7f72e8152c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:35.415 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.414+0000 7f72e759e700 1 --2- 172.21.15.77:0/1647580632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e806bbe0 0x7f72e8152c40 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f72d8004000 tx=0x7f72d8016000 comp rx=0 tx=0).stop 2022-04-03T19:57:35.415 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.414+0000 7f72e759e700 1 -- 172.21.15.77:0/1647580632 shutdown_connections 2022-04-03T19:57:35.416 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.414+0000 7f72e759e700 1 --2- 172.21.15.77:0/1647580632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72e806c500 0x7f72e8168af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:35.416 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.414+0000 7f72e759e700 1 --2- 172.21.15.77:0/1647580632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72e806bbe0 0x7f72e8152c40 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:35.416 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.414+0000 7f72e759e700 1 --2- 172.21.15.77:0/1647580632 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f72cc063ce0 0x7f72cc066190 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:35.416 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.414+0000 7f72e759e700 1 --2- 172.21.15.77:0/1647580632 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72e806bfb0 0x7f72e80c1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:35.417 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.414+0000 7f72e759e700 1 -- 172.21.15.77:0/1647580632 >> 172.21.15.77:0/1647580632 conn(0x7f72e80cb6b0 msgr2=0x7f72e814a420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:35.417 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.414+0000 7f72e759e700 1 -- 172.21.15.77:0/1647580632 shutdown_connections 2022-04-03T19:57:35.417 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:35.414+0000 7f72e759e700 1 -- 172.21.15.77:0/1647580632 wait complete. 2022-04-03T19:57:35.417 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 46 2022-04-03T19:57:35.439 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:57:37.440 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:37.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.648+0000 7f85fd5ff700 1 Processor -- start 2022-04-03T19:57:37.649 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.649+0000 7f85fd5ff700 1 -- start start 2022-04-03T19:57:37.651 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.650+0000 7f85fd5ff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85f81584b0 0x7f85f814eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:37.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.650+0000 7f85fd5ff700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f806d260 con 0x7f85f806bbe0 2022-04-03T19:57:37.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.650+0000 7f85fd5ff700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f80c6330 con 0x7f85f806bfb0 2022-04-03T19:57:37.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.650+0000 7f85fd5ff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f8159450 con 0x7f85f81584b0 2022-04-03T19:57:37.652 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.650+0000 7f85f7fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f85f806bbe0 0x7f85f806d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38588/0 (socket says 172.21.15.77:38588) 2022-04-03T19:57:37.653 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.650+0000 7f85f77fe700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f85f806bfb0 0x7f85f806c500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44392/0 (socket says 172.21.15.77:44392) 2022-04-03T19:57:37.653 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.650+0000 7f85f7fff700 1 -- 172.21.15.77:0/1603623244 learned_addr learned my addr 172.21.15.77:0/1603623244 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:57:37.653 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.650+0000 7f85fcdfe700 1 --2- 172.21.15.77:0/1603623244 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85f81584b0 0x7f85f814eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:37.653 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.650+0000 7f85f6ffd700 1 -- 172.21.15.77:0/1603623244 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265143334 0 0) 0x7f85f80c6330 con 0x7f85f806bfb0 2022-04-03T19:57:37.654 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.651+0000 7f85f6ffd700 1 -- 172.21.15.77:0/1603623244 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85e40033f0 con 0x7f85f806bfb0 2022-04-03T19:57:37.654 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.651+0000 7f85fcdfe700 1 -- 172.21.15.77:0/1603623244 >> v1:172.21.15.186:6789/0 conn(0x7f85f806bfb0 legacy=0x7f85f806c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:37.654 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.651+0000 7f85fcdfe700 1 -- 172.21.15.77:0/1603623244 >> v1:172.21.15.77:6789/0 conn(0x7f85f806bbe0 legacy=0x7f85f806d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:37.654 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.651+0000 7f85fcdfe700 1 -- 172.21.15.77:0/1603623244 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f8158910 con 0x7f85f81584b0 2022-04-03T19:57:37.654 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.651+0000 7f85fcdfe700 1 --2- 172.21.15.77:0/1603623244 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85f81584b0 0x7f85f814eea0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f85f0009e90 tx=0x7f85f0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9176fafa6ed51caa server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:37.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.651+0000 7f85f6ffd700 1 -- 172.21.15.77:0/1603623244 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f000e070 con 0x7f85f81584b0 2022-04-03T19:57:37.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.651+0000 7f85f6ffd700 1 -- 172.21.15.77:0/1603623244 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f000bc30 con 0x7f85f81584b0 2022-04-03T19:57:37.655 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.651+0000 7f85f6ffd700 1 -- 172.21.15.77:0/1603623244 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f0012430 con 0x7f85f81584b0 2022-04-03T19:57:37.663 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.652+0000 7f85fd5ff700 1 -- 172.21.15.77:0/1603623244 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85f81584b0 msgr2=0x7f85f814eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:37.663 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.652+0000 7f85fd5ff700 1 --2- 172.21.15.77:0/1603623244 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85f81584b0 0x7f85f814eea0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f85f0009e90 tx=0x7f85f0009780 comp rx=0 tx=0).stop 2022-04-03T19:57:37.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.652+0000 7f85fd5ff700 1 -- 172.21.15.77:0/1603623244 shutdown_connections 2022-04-03T19:57:37.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.652+0000 7f85fd5ff700 1 --2- 172.21.15.77:0/1603623244 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85f81584b0 0x7f85f814eea0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:37.664 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.652+0000 7f85fd5ff700 1 -- 172.21.15.77:0/1603623244 >> 172.21.15.77:0/1603623244 conn(0x7f85f80cb6b0 msgr2=0x7f85f80cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:37.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.652+0000 7f85fd5ff700 1 -- 172.21.15.77:0/1603623244 shutdown_connections 2022-04-03T19:57:37.665 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.652+0000 7f85fd5ff700 1 -- 172.21.15.77:0/1603623244 wait complete. 2022-04-03T19:57:37.666 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.653+0000 7f85fd5ff700 1 Processor -- start 2022-04-03T19:57:37.667 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.654+0000 7f85fd5ff700 1 -- start start 2022-04-03T19:57:37.667 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.654+0000 7f85fd5ff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85f806bbe0 0x7f85f8165970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:37.667 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.654+0000 7f85fd5ff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85f806bfb0 0x7f85f8155110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:37.668 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.654+0000 7f85fd5ff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85f81584b0 0x7f85f815a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:37.668 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.654+0000 7f85fd5ff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f8151200 con 0x7f85f806bfb0 2022-04-03T19:57:37.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.654+0000 7f85fd5ff700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f8150630 con 0x7f85f806bbe0 2022-04-03T19:57:37.669 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.654+0000 7f85fd5ff700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f81508d0 con 0x7f85f81584b0 2022-04-03T19:57:37.670 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.655+0000 7f85f7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85f806bbe0 0x7f85f8165970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:37.670 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.655+0000 7f85fcdfe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85f81584b0 0x7f85f815a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:37.670 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.655+0000 7f85f7fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85f806bbe0 0x7f85f8165970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38444/0 (socket says 172.21.15.77:38444) 2022-04-03T19:57:37.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.655+0000 7f85f7fff700 1 -- 172.21.15.77:0/3640518265 learned_addr learned my addr 172.21.15.77:0/3640518265 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:37.671 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.655+0000 7f85fcdfe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85f81584b0 0x7f85f815a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60554/0 (socket says 172.21.15.77:60554) 2022-04-03T19:57:37.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.655+0000 7f85f77fe700 1 --2- 172.21.15.77:0/3640518265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85f806bfb0 0x7f85f8155110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:37.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.655+0000 7f85f7fff700 1 -- 172.21.15.77:0/3640518265 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85f81584b0 msgr2=0x7f85f815a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:37.672 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.655+0000 7f85f7fff700 1 --2- 172.21.15.77:0/3640518265 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85f81584b0 0x7f85f815a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:37.673 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.655+0000 7f85f7fff700 1 -- 172.21.15.77:0/3640518265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85f806bfb0 msgr2=0x7f85f8155110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:37.673 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.655+0000 7f85f7fff700 1 --2- 172.21.15.77:0/3640518265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85f806bfb0 0x7f85f8155110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:37.674 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.655+0000 7f85f7fff700 1 -- 172.21.15.77:0/3640518265 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f0009a20 con 0x7f85f806bbe0 2022-04-03T19:57:37.674 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.655+0000 7f85f7fff700 1 --2- 172.21.15.77:0/3640518265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85f806bbe0 0x7f85f8165970 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f85e8007fd0 tx=0x7f85e8007df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:37.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.656+0000 7f85dffff700 1 -- 172.21.15.77:0/3640518265 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e800ce10 con 0x7f85f806bbe0 2022-04-03T19:57:37.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.656+0000 7f85dffff700 1 -- 172.21.15.77:0/3640518265 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e80072e0 con 0x7f85f806bbe0 2022-04-03T19:57:37.675 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.656+0000 7f85fd5ff700 1 -- 172.21.15.77:0/3640518265 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f8150bb0 con 0x7f85f806bbe0 2022-04-03T19:57:37.676 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.656+0000 7f85dffff700 1 -- 172.21.15.77:0/3640518265 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e8007660 con 0x7f85f806bbe0 2022-04-03T19:57:37.676 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.656+0000 7f85fd5ff700 1 -- 172.21.15.77:0/3640518265 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f815af50 con 0x7f85f806bbe0 2022-04-03T19:57:37.676 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.657+0000 7f85fd5ff700 1 -- 172.21.15.77:0/3640518265 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f8158bb0 con 0x7f85f806bbe0 2022-04-03T19:57:37.677 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.658+0000 7f85dffff700 1 -- 172.21.15.77:0/3640518265 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f85e8019030 con 0x7f85f806bbe0 2022-04-03T19:57:37.677 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.659+0000 7f85dffff700 1 --2- 172.21.15.77:0/3640518265 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f85d8063d10 0x7f85d80661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:37.677 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.659+0000 7f85dffff700 1 -- 172.21.15.77:0/3640518265 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f85e807fc30 con 0x7f85f806bbe0 2022-04-03T19:57:37.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.659+0000 7f85f77fe700 1 --2- 172.21.15.77:0/3640518265 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f85d8063d10 0x7f85d80661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:37.678 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.660+0000 7f85f77fe700 1 --2- 172.21.15.77:0/3640518265 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f85d8063d10 0x7f85d80661c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f85ec007990 tx=0x7f85ec007bb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:37.679 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.663+0000 7f85dffff700 1 -- 172.21.15.77:0/3640518265 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f85e8048ae0 con 0x7f85f806bbe0 2022-04-03T19:57:37.895 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.894+0000 7f85fd5ff700 1 -- 172.21.15.77:0/3640518265 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85f806dba0 con 0x7f85f806bbe0 2022-04-03T19:57:37.895 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.895+0000 7f85dffff700 1 -- 172.21.15.77:0/3640518265 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5353 (secure 0 0 0) 0x7f85e8017030 con 0x7f85f806bbe0 2022-04-03T19:57:37.896 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:37.896 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:57:32.318895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5668":{"gid":5668,"name":"d","rank":0,"incarnation":44,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:37.898 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.897+0000 7f85fd5ff700 1 -- 172.21.15.77:0/3640518265 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f85d8063d10 msgr2=0x7f85d80661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:37.899 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.897+0000 7f85fd5ff700 1 --2- 172.21.15.77:0/3640518265 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f85d8063d10 0x7f85d80661c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f85ec007990 tx=0x7f85ec007bb0 comp rx=0 tx=0).stop 2022-04-03T19:57:37.899 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.898+0000 7f85fd5ff700 1 -- 172.21.15.77:0/3640518265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85f806bbe0 msgr2=0x7f85f8165970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:37.899 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.898+0000 7f85fd5ff700 1 --2- 172.21.15.77:0/3640518265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85f806bbe0 0x7f85f8165970 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f85e8007fd0 tx=0x7f85e8007df0 comp rx=0 tx=0).stop 2022-04-03T19:57:37.899 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.898+0000 7f85fd5ff700 1 -- 172.21.15.77:0/3640518265 shutdown_connections 2022-04-03T19:57:37.900 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.898+0000 7f85fd5ff700 1 --2- 172.21.15.77:0/3640518265 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f85d8063d10 0x7f85d80661c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:37.900 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.898+0000 7f85fd5ff700 1 --2- 172.21.15.77:0/3640518265 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85f81584b0 0x7f85f815a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:37.900 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.898+0000 7f85fd5ff700 1 --2- 172.21.15.77:0/3640518265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85f806bbe0 0x7f85f8165970 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:37.900 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.898+0000 7f85fd5ff700 1 --2- 172.21.15.77:0/3640518265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85f806bfb0 0x7f85f8155110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:37.900 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.898+0000 7f85fd5ff700 1 -- 172.21.15.77:0/3640518265 >> 172.21.15.77:0/3640518265 conn(0x7f85f80cb6b0 msgr2=0x7f85f8149b20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:37.901 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.898+0000 7f85fd5ff700 1 -- 172.21.15.77:0/3640518265 shutdown_connections 2022-04-03T19:57:37.901 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:37.898+0000 7f85fd5ff700 1 -- 172.21.15.77:0/3640518265 wait complete. 2022-04-03T19:57:37.901 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 46 2022-04-03T19:57:37.923 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:57:39.924 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:40.129 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.129+0000 7f3c2b59e700 1 Processor -- start 2022-04-03T19:57:40.130 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.129+0000 7f3c2b59e700 1 -- start start 2022-04-03T19:57:40.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.130+0000 7f3c2b59e700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3c2c06bfb0 0x7f3c2c1584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:40.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.130+0000 7f3c2b59e700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c2c06d260 con 0x7f3c2c06bbe0 2022-04-03T19:57:40.132 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.130+0000 7f3c2b59e700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c2c0c6330 con 0x7f3c2c06c500 2022-04-03T19:57:40.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.130+0000 7f3c2b59e700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c2c159450 con 0x7f3c2c06bfb0 2022-04-03T19:57:40.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.130+0000 7f3c29d9b700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3c2c06bfb0 0x7f3c2c1584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:40.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.130+0000 7f3c2a59c700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f3c2c06bbe0 0x7f3c2c06d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38592/0 (socket says 172.21.15.77:38592) 2022-04-03T19:57:40.133 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.130+0000 7f3c2a59c700 1 -- 172.21.15.77:0/3109990080 learned_addr learned my addr 172.21.15.77:0/3109990080 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:57:40.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.131+0000 7f3c29d9b700 1 -- 172.21.15.77:0/3109990080 >> v1:172.21.15.186:6789/0 conn(0x7f3c2c06c500 legacy=0x7f3c2c14eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:40.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.131+0000 7f3c29d9b700 1 -- 172.21.15.77:0/3109990080 >> v1:172.21.15.77:6789/0 conn(0x7f3c2c06bbe0 legacy=0x7f3c2c06d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:40.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.131+0000 7f3c29d9b700 1 -- 172.21.15.77:0/3109990080 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2c158bb0 con 0x7f3c2c06bfb0 2022-04-03T19:57:40.134 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.131+0000 7f3c2959a700 1 -- 172.21.15.77:0/3109990080 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171115365 0 0) 0x7f3c2c06d260 con 0x7f3c2c06bbe0 2022-04-03T19:57:40.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.131+0000 7f3c29d9b700 1 --2- 172.21.15.77:0/3109990080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c2c06bfb0 0x7f3c2c1584b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3c20009760 tx=0x7f3c20009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac253242cb919bf server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:40.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.132+0000 7f3c2959a700 1 -- 172.21.15.77:0/3109990080 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c20011630 con 0x7f3c2c06bfb0 2022-04-03T19:57:40.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.132+0000 7f3c2959a700 1 -- 172.21.15.77:0/3109990080 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c20011790 con 0x7f3c2c06bfb0 2022-04-03T19:57:40.135 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.132+0000 7f3c2959a700 1 -- 172.21.15.77:0/3109990080 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c20011b10 con 0x7f3c2c06bfb0 2022-04-03T19:57:40.139 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.132+0000 7f3c2b59e700 1 -- 172.21.15.77:0/3109990080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c2c06bfb0 msgr2=0x7f3c2c1584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:40.139 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.132+0000 7f3c2b59e700 1 --2- 172.21.15.77:0/3109990080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c2c06bfb0 0x7f3c2c1584b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3c20009760 tx=0x7f3c20009060 comp rx=0 tx=0).stop 2022-04-03T19:57:40.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.132+0000 7f3c2b59e700 1 -- 172.21.15.77:0/3109990080 shutdown_connections 2022-04-03T19:57:40.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.132+0000 7f3c2b59e700 1 --2- 172.21.15.77:0/3109990080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c2c06bfb0 0x7f3c2c1584b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:40.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.132+0000 7f3c2b59e700 1 -- 172.21.15.77:0/3109990080 >> 172.21.15.77:0/3109990080 conn(0x7f3c2c0cb6b0 msgr2=0x7f3c2c0cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:40.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.132+0000 7f3c2b59e700 1 -- 172.21.15.77:0/3109990080 shutdown_connections 2022-04-03T19:57:40.140 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.132+0000 7f3c2b59e700 1 -- 172.21.15.77:0/3109990080 wait complete. 2022-04-03T19:57:40.141 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.134+0000 7f3c2b59e700 1 Processor -- start 2022-04-03T19:57:40.141 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.134+0000 7f3c2b59e700 1 -- start start 2022-04-03T19:57:40.141 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.135+0000 7f3c2b59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c2c06bbe0 0x7f3c2c165a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:40.141 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.135+0000 7f3c2b59e700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c2c06bfb0 0x7f3c2c151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:40.142 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.135+0000 7f3c2b59e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c2c06c500 0x7f3c2c154ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:40.142 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.135+0000 7f3c2b59e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c152790 con 0x7f3c2c06bbe0 2022-04-03T19:57:40.142 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.135+0000 7f3c2b59e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c165f80 con 0x7f3c2c06c500 2022-04-03T19:57:40.142 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.135+0000 7f3c2b59e700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c150900 con 0x7f3c2c06bfb0 2022-04-03T19:57:40.142 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.135+0000 7f3c2a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c2c06bbe0 0x7f3c2c165a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:40.143 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.135+0000 7f3c2a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c2c06bbe0 0x7f3c2c165a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43262/0 (socket says 172.21.15.77:43262) 2022-04-03T19:57:40.143 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.135+0000 7f3c2a59c700 1 -- 172.21.15.77:0/284584338 learned_addr learned my addr 172.21.15.77:0/284584338 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:40.143 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.135+0000 7f3c29d9b700 1 --2- 172.21.15.77:0/284584338 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c2c06bfb0 0x7f3c2c151630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:40.143 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.135+0000 7f3c2ad9d700 1 --2- 172.21.15.77:0/284584338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c2c06c500 0x7f3c2c154ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:40.144 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.136+0000 7f3c2a59c700 1 -- 172.21.15.77:0/284584338 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c2c06bfb0 msgr2=0x7f3c2c151630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:40.144 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.136+0000 7f3c2a59c700 1 --2- 172.21.15.77:0/284584338 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c2c06bfb0 0x7f3c2c151630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:40.144 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.136+0000 7f3c2a59c700 1 -- 172.21.15.77:0/284584338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c2c06c500 msgr2=0x7f3c2c154ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:40.144 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.136+0000 7f3c2a59c700 1 --2- 172.21.15.77:0/284584338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c2c06c500 0x7f3c2c154ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:40.145 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.136+0000 7f3c2a59c700 1 -- 172.21.15.77:0/284584338 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c200092a0 con 0x7f3c2c06bbe0 2022-04-03T19:57:40.145 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.136+0000 7f3c2ad9d700 1 --2- 172.21.15.77:0/284584338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c2c06c500 0x7f3c2c154ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-04-03T19:57:40.145 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.136+0000 7f3c2a59c700 1 --2- 172.21.15.77:0/284584338 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c2c06bbe0 0x7f3c2c165a40 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c00c490 tx=0x7f3c1c007c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:40.145 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.136+0000 7f3c1a7fc700 1 -- 172.21.15.77:0/284584338 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c007360 con 0x7f3c2c06bbe0 2022-04-03T19:57:40.145 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.136+0000 7f3c1a7fc700 1 -- 172.21.15.77:0/284584338 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1c0074c0 con 0x7f3c2c06bbe0 2022-04-03T19:57:40.146 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.136+0000 7f3c1a7fc700 1 -- 172.21.15.77:0/284584338 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c0048e0 con 0x7f3c2c06bbe0 2022-04-03T19:57:40.146 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.136+0000 7f3c2b59e700 1 -- 172.21.15.77:0/284584338 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c2c1511a0 con 0x7f3c2c06bbe0 2022-04-03T19:57:40.146 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.136+0000 7f3c2b59e700 1 -- 172.21.15.77:0/284584338 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c2c150580 con 0x7f3c2c06bbe0 2022-04-03T19:57:40.146 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.137+0000 7f3c2b59e700 1 -- 172.21.15.77:0/284584338 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c2c04b050 con 0x7f3c2c06bbe0 2022-04-03T19:57:40.146 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.138+0000 7f3c1a7fc700 1 -- 172.21.15.77:0/284584338 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f3c1c010030 con 0x7f3c2c06bbe0 2022-04-03T19:57:40.147 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.139+0000 7f3c1a7fc700 1 --2- 172.21.15.77:0/284584338 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3c08063ce0 0x7f3c08066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:40.147 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.139+0000 7f3c1a7fc700 1 -- 172.21.15.77:0/284584338 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f3c1c080690 con 0x7f3c2c06bbe0 2022-04-03T19:57:40.148 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.139+0000 7f3c29d9b700 1 --2- 172.21.15.77:0/284584338 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3c08063ce0 0x7f3c08066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:40.148 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.140+0000 7f3c29d9b700 1 --2- 172.21.15.77:0/284584338 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3c08063ce0 0x7f3c08066190 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c0b18e0 tx=0x7f3c20017000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:40.148 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.143+0000 7f3c1a7fc700 1 -- 172.21.15.77:0/284584338 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f3c1c04ef70 con 0x7f3c2c06bbe0 2022-04-03T19:57:40.375 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.375+0000 7f3c2b59e700 1 -- 172.21.15.77:0/284584338 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c2c063d10 con 0x7f3c2c06bbe0 2022-04-03T19:57:40.376 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.375+0000 7f3c1a7fc700 1 -- 172.21.15.77:0/284584338 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+5359 (secure 0 0 0) 0x7f3c1c052590 con 0x7f3c2c06bbe0 2022-04-03T19:57:40.377 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:40.377 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:57:38.338574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5668":{"gid":5668,"name":"d","rank":0,"incarnation":44,"state":"up:clientreplay","state_seq":31,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:40.379 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.378+0000 7f3c2b59e700 1 -- 172.21.15.77:0/284584338 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3c08063ce0 msgr2=0x7f3c08066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:40.379 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.378+0000 7f3c2b59e700 1 --2- 172.21.15.77:0/284584338 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3c08063ce0 0x7f3c08066190 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c0b18e0 tx=0x7f3c20017000 comp rx=0 tx=0).stop 2022-04-03T19:57:40.379 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.378+0000 7f3c2b59e700 1 -- 172.21.15.77:0/284584338 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c2c06bbe0 msgr2=0x7f3c2c165a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:40.379 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.378+0000 7f3c2b59e700 1 --2- 172.21.15.77:0/284584338 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c2c06bbe0 0x7f3c2c165a40 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c00c490 tx=0x7f3c1c007c40 comp rx=0 tx=0).stop 2022-04-03T19:57:40.380 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.378+0000 7f3c2b59e700 1 -- 172.21.15.77:0/284584338 shutdown_connections 2022-04-03T19:57:40.380 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.378+0000 7f3c2b59e700 1 --2- 172.21.15.77:0/284584338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c2c06c500 0x7f3c2c154ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:40.380 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.378+0000 7f3c2b59e700 1 --2- 172.21.15.77:0/284584338 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c2c06bbe0 0x7f3c2c165a40 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:40.380 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.378+0000 7f3c2b59e700 1 --2- 172.21.15.77:0/284584338 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f3c08063ce0 0x7f3c08066190 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:40.381 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.378+0000 7f3c2b59e700 1 --2- 172.21.15.77:0/284584338 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c2c06bfb0 0x7f3c2c151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:40.381 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.378+0000 7f3c2b59e700 1 -- 172.21.15.77:0/284584338 >> 172.21.15.77:0/284584338 conn(0x7f3c2c0cb6b0 msgr2=0x7f3c2c149df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:40.381 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.378+0000 7f3c2b59e700 1 -- 172.21.15.77:0/284584338 shutdown_connections 2022-04-03T19:57:40.381 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:40.378+0000 7f3c2b59e700 1 -- 172.21.15.77:0/284584338 wait complete. 2022-04-03T19:57:40.382 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 47 2022-04-03T19:57:40.403 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 47, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2022-04-03T19:44:21.275716+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 47, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5668": { "addr": "172.21.15.186:6837/113043607", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6835", "nonce": 113043607, "type": "v2" }, { "addr": "172.21.15.186:6837", "nonce": 113043607, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 5668, "incarnation": 44, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:clientreplay", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 53, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2022-04-03T19:57:38.338574+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5668 } } } ], "standbys": [ { "addr": "172.21.15.77:6834/2116223509", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6832", "nonce": 2116223509, "type": "v2" }, { "addr": "172.21.15.77:6834", "nonce": 2116223509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4245, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6836/2374276747", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6834", "nonce": 2374276747, "type": "v2" }, { "addr": "172.21.15.186:6836", "nonce": 2374276747, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4259, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.186:6839/610990853", "addrs": { "addrvec": [ { "addr": "172.21.15.186:6838", "nonce": 610990853, "type": "v2" }, { "addr": "172.21.15.186:6839", "nonce": 610990853, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4265, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.77:6835/170053412", "addrs": { "addrvec": [ { "addr": "172.21.15.77:6833", "nonce": 170053412, "type": "v2" }, { "addr": "172.21.15.77:6835", "nonce": 170053412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138303579357183, "flags": 0, "gid": 4271, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2022-04-03T19:57:42.405 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:42.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.609+0000 7f93cafa3700 1 Processor -- start 2022-04-03T19:57:42.610 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.609+0000 7f93cafa3700 1 -- start start 2022-04-03T19:57:42.612 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.610+0000 7f93cafa3700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f93c40c5930 0x7f93c4155be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:42.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.610+0000 7f93cafa3700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93c4156120 0x7f93c406c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:42.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.610+0000 7f93cafa3700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93c40c2f80 con 0x7f93c406d3f0 2022-04-03T19:57:42.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.611+0000 7f93cafa3700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c40c3260 con 0x7f93c40c5930 2022-04-03T19:57:42.613 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.611+0000 7f93cafa3700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c40c3540 con 0x7f93c4156120 2022-04-03T19:57:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.611+0000 7f93ca7a2700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93c4156120 0x7f93c406c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.611+0000 7f93c97a0700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f93c40c5930 0x7f93c4155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.611+0000 7f93c97a0700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f93c40c5930 0x7f93c4155be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38448/0 (socket says 172.21.15.77:38448) 2022-04-03T19:57:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.611+0000 7f93c97a0700 1 -- 172.21.15.77:0/203383434 learned_addr learned my addr 172.21.15.77:0/203383434 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.611+0000 7f93c9fa1700 1 --1- 172.21.15.77:0/203383434 >> v1:172.21.15.186:6789/0 conn(0x7f93c406d3f0 0x7f93c406d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44396/0 (socket says 172.21.15.77:44396) 2022-04-03T19:57:42.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.611+0000 7f93c97a0700 1 -- 172.21.15.77:0/203383434 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93c4156120 msgr2=0x7f93c406c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:42.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.611+0000 7f93c97a0700 1 --2- 172.21.15.77:0/203383434 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93c4156120 0x7f93c406c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:42.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.611+0000 7f93c97a0700 1 -- 172.21.15.77:0/203383434 >> v1:172.21.15.186:6789/0 conn(0x7f93c406d3f0 legacy=0x7f93c406d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:42.615 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.611+0000 7f93c97a0700 1 -- 172.21.15.77:0/203383434 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c40c3820 con 0x7f93c40c5930 2022-04-03T19:57:42.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.612+0000 7f93c97a0700 1 --2- 172.21.15.77:0/203383434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93c40c5930 0x7f93c4155be0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f93b4009830 tx=0x7f93b40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b2359856f08952c server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:42.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.612+0000 7f93c8f9f700 1 -- 172.21.15.77:0/203383434 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b4004030 con 0x7f93c40c5930 2022-04-03T19:57:42.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.612+0000 7f93c8f9f700 1 -- 172.21.15.77:0/203383434 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b4004190 con 0x7f93c40c5930 2022-04-03T19:57:42.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.612+0000 7f93c8f9f700 1 -- 172.21.15.77:0/203383434 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b4011630 con 0x7f93c40c5930 2022-04-03T19:57:42.616 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.612+0000 7f93cafa3700 1 -- 172.21.15.77:0/203383434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93c40c5930 msgr2=0x7f93c4155be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:42.617 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.612+0000 7f93cafa3700 1 --2- 172.21.15.77:0/203383434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93c40c5930 0x7f93c4155be0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f93b4009830 tx=0x7f93b40090f0 comp rx=0 tx=0).stop 2022-04-03T19:57:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.613+0000 7f93cafa3700 1 -- 172.21.15.77:0/203383434 shutdown_connections 2022-04-03T19:57:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.613+0000 7f93cafa3700 1 --2- 172.21.15.77:0/203383434 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93c4156120 0x7f93c406c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.613+0000 7f93cafa3700 1 --2- 172.21.15.77:0/203383434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93c40c5930 0x7f93c4155be0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.613+0000 7f93cafa3700 1 -- 172.21.15.77:0/203383434 >> 172.21.15.77:0/203383434 conn(0x7f93c40cb720 msgr2=0x7f93c40cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:42.621 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.613+0000 7f93cafa3700 1 -- 172.21.15.77:0/203383434 shutdown_connections 2022-04-03T19:57:42.621 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.613+0000 7f93cafa3700 1 -- 172.21.15.77:0/203383434 wait complete. 2022-04-03T19:57:42.621 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.614+0000 7f93cafa3700 1 Processor -- start 2022-04-03T19:57:42.621 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.614+0000 7f93cafa3700 1 -- start start 2022-04-03T19:57:42.622 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.615+0000 7f93cafa3700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93c406d3f0 0x7f93c4152780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:42.622 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.615+0000 7f93cafa3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93c4156120 0x7f93c40c1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:42.622 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.615+0000 7f93cafa3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93c4150980 0x7f93c414bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:42.622 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.615+0000 7f93cafa3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c4157590 con 0x7f93c4150980 2022-04-03T19:57:42.623 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.615+0000 7f93cafa3700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c4152cc0 con 0x7f93c4156120 2022-04-03T19:57:42.623 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.615+0000 7f93cafa3700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c4150e70 con 0x7f93c406d3f0 2022-04-03T19:57:42.623 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.615+0000 7f93c97a0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93c4156120 0x7f93c40c1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:42.623 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.615+0000 7f93c9fa1700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93c406d3f0 0x7f93c4152780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:42.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.616+0000 7f93c97a0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93c4156120 0x7f93c40c1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38450/0 (socket says 172.21.15.77:38450) 2022-04-03T19:57:42.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.616+0000 7f93c97a0700 1 -- 172.21.15.77:0/759850777 learned_addr learned my addr 172.21.15.77:0/759850777 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:42.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.616+0000 7f93ca7a2700 1 --2- 172.21.15.77:0/759850777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93c4150980 0x7f93c414bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:42.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.616+0000 7f93c97a0700 1 -- 172.21.15.77:0/759850777 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93c406d3f0 msgr2=0x7f93c4152780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:42.624 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.616+0000 7f93c97a0700 1 --2- 172.21.15.77:0/759850777 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93c406d3f0 0x7f93c4152780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:42.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.616+0000 7f93c97a0700 1 -- 172.21.15.77:0/759850777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93c4150980 msgr2=0x7f93c414bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:42.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.616+0000 7f93c97a0700 1 --2- 172.21.15.77:0/759850777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93c4150980 0x7f93c414bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:42.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.616+0000 7f93c97a0700 1 -- 172.21.15.77:0/759850777 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b4009370 con 0x7f93c4156120 2022-04-03T19:57:42.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.616+0000 7f93c9fa1700 1 --2- 172.21.15.77:0/759850777 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93c406d3f0 0x7f93c4152780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:57:42.625 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.616+0000 7f93c97a0700 1 --2- 172.21.15.77:0/759850777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93c4156120 0x7f93c40c1950 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f93b4008000 tx=0x7f93b4016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:42.626 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.617+0000 7f93b1ffb700 1 -- 172.21.15.77:0/759850777 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b40036f0 con 0x7f93c4156120 2022-04-03T19:57:42.626 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.617+0000 7f93b1ffb700 1 -- 172.21.15.77:0/759850777 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b4003850 con 0x7f93c4156120 2022-04-03T19:57:42.626 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.617+0000 7f93cafa3700 1 -- 172.21.15.77:0/759850777 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c4156980 con 0x7f93c4156120 2022-04-03T19:57:42.626 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.617+0000 7f93b1ffb700 1 -- 172.21.15.77:0/759850777 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b4003c10 con 0x7f93c4156120 2022-04-03T19:57:42.627 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.617+0000 7f93cafa3700 1 -- 172.21.15.77:0/759850777 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93c4156f40 con 0x7f93c4156120 2022-04-03T19:57:42.627 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.618+0000 7f93cafa3700 1 -- 172.21.15.77:0/759850777 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c404b050 con 0x7f93c4156120 2022-04-03T19:57:42.627 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.619+0000 7f93b1ffb700 1 -- 172.21.15.77:0/759850777 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f93b4017020 con 0x7f93c4156120 2022-04-03T19:57:42.627 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.619+0000 7f93b1ffb700 1 --2- 172.21.15.77:0/759850777 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f93a4063d10 0x7f93a40661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:42.627 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.620+0000 7f93b1ffb700 1 -- 172.21.15.77:0/759850777 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f93b4081d50 con 0x7f93c4156120 2022-04-03T19:57:42.628 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.620+0000 7f93c9fa1700 1 --2- 172.21.15.77:0/759850777 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f93a4063d10 0x7f93a40661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:42.628 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.620+0000 7f93c9fa1700 1 --2- 172.21.15.77:0/759850777 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f93a4063d10 0x7f93a40661c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f93c0007fd0 tx=0x7f93c0007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:42.629 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.624+0000 7f93b1ffb700 1 -- 172.21.15.77:0/759850777 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f93b40506a0 con 0x7f93c4156120 2022-04-03T19:57:42.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.860+0000 7f93cafa3700 1 -- 172.21.15.77:0/759850777 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93c406d010 con 0x7f93c4156120 2022-04-03T19:57:42.862 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.861+0000 7f93b1ffb700 1 -- 172.21.15.77:0/759850777 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+5359 (secure 0 0 0) 0x7f93b4006bf0 con 0x7f93c4156120 2022-04-03T19:57:42.862 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:42.863 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:57:38.338574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5668":{"gid":5668,"name":"d","rank":0,"incarnation":44,"state":"up:clientreplay","state_seq":31,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:42.865 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.864+0000 7f93cafa3700 1 -- 172.21.15.77:0/759850777 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f93a4063d10 msgr2=0x7f93a40661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:42.865 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.864+0000 7f93cafa3700 1 --2- 172.21.15.77:0/759850777 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f93a4063d10 0x7f93a40661c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f93c0007fd0 tx=0x7f93c0007d30 comp rx=0 tx=0).stop 2022-04-03T19:57:42.865 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.864+0000 7f93cafa3700 1 -- 172.21.15.77:0/759850777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93c4156120 msgr2=0x7f93c40c1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:42.865 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.864+0000 7f93cafa3700 1 --2- 172.21.15.77:0/759850777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93c4156120 0x7f93c40c1950 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f93b4008000 tx=0x7f93b4016000 comp rx=0 tx=0).stop 2022-04-03T19:57:42.866 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.864+0000 7f93cafa3700 1 -- 172.21.15.77:0/759850777 shutdown_connections 2022-04-03T19:57:42.866 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.864+0000 7f93cafa3700 1 --2- 172.21.15.77:0/759850777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f93c4150980 0x7f93c414bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:42.866 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.864+0000 7f93cafa3700 1 --2- 172.21.15.77:0/759850777 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f93a4063d10 0x7f93a40661c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:42.866 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.864+0000 7f93cafa3700 1 --2- 172.21.15.77:0/759850777 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93c406d3f0 0x7f93c4152780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:42.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.864+0000 7f93cafa3700 1 --2- 172.21.15.77:0/759850777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93c4156120 0x7f93c40c1950 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:42.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.864+0000 7f93cafa3700 1 -- 172.21.15.77:0/759850777 >> 172.21.15.77:0/759850777 conn(0x7f93c40cb720 msgr2=0x7f93c40c5dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:42.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.864+0000 7f93cafa3700 1 -- 172.21.15.77:0/759850777 shutdown_connections 2022-04-03T19:57:42.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:42.864+0000 7f93cafa3700 1 -- 172.21.15.77:0/759850777 wait complete. 2022-04-03T19:57:42.868 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 47 2022-04-03T19:57:42.889 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2022-04-03T19:57:44.890 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:45.096 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.096+0000 7fbdf60af700 1 Processor -- start 2022-04-03T19:57:45.097 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.096+0000 7fbdf60af700 1 -- start start 2022-04-03T19:57:45.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.097+0000 7fbdf60af700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbdf006ccf0 0x7fbdf00aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:45.099 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.097+0000 7fbdf60af700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbdf0067a70 0x7fbdf00af060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:45.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.097+0000 7fbdf60af700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf0066ed0 con 0x7fbdf00c5760 2022-04-03T19:57:45.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.097+0000 7fbdf60af700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf006d930 con 0x7fbdf006ccf0 2022-04-03T19:57:45.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.097+0000 7fbdf60af700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf006e380 con 0x7fbdf0067a70 2022-04-03T19:57:45.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.098+0000 7fbdf50ad700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fbdf006ccf0 0x7fbdf00aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:45.100 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.098+0000 7fbdf58ae700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbdf0067a70 0x7fbdf00af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:45.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.098+0000 7fbdf48ac700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fbdf00c5760 0x7fbdf0067360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44398/0 (socket says 172.21.15.77:44398) 2022-04-03T19:57:45.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.098+0000 7fbdf48ac700 1 -- 172.21.15.77:0/4225273166 learned_addr learned my addr 172.21.15.77:0/4225273166 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:57:45.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.098+0000 7fbdf50ad700 1 -- 172.21.15.77:0/4225273166 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbdf0067a70 msgr2=0x7fbdf00af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:45.101 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.098+0000 7fbdf50ad700 1 --2- 172.21.15.77:0/4225273166 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbdf0067a70 0x7fbdf00af060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:45.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.098+0000 7fbdf50ad700 1 -- 172.21.15.77:0/4225273166 >> v1:172.21.15.186:6789/0 conn(0x7fbdf00c5760 legacy=0x7fbdf0067360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:45.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.098+0000 7fbdf50ad700 1 -- 172.21.15.77:0/4225273166 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf00c6790 con 0x7fbdf006ccf0 2022-04-03T19:57:45.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.098+0000 7fbddffff700 1 -- 172.21.15.77:0/4225273166 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6851991 0 0) 0x7fbdf0066ed0 con 0x7fbdf00c5760 2022-04-03T19:57:45.102 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.099+0000 7fbdf50ad700 1 --2- 172.21.15.77:0/4225273166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdf006ccf0 0x7fbdf00aeb20 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fbdec009760 tx=0x7fbdec009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6cd1471c8d798c server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:45.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.099+0000 7fbddffff700 1 -- 172.21.15.77:0/4225273166 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdec011630 con 0x7fbdf006ccf0 2022-04-03T19:57:45.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.099+0000 7fbddffff700 1 -- 172.21.15.77:0/4225273166 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdec011790 con 0x7fbdf006ccf0 2022-04-03T19:57:45.103 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.099+0000 7fbddffff700 1 -- 172.21.15.77:0/4225273166 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdec011b50 con 0x7fbdf006ccf0 2022-04-03T19:57:45.106 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.099+0000 7fbdf60af700 1 -- 172.21.15.77:0/4225273166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdf006ccf0 msgr2=0x7fbdf00aeb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:45.107 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.099+0000 7fbdf60af700 1 --2- 172.21.15.77:0/4225273166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdf006ccf0 0x7fbdf00aeb20 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fbdec009760 tx=0x7fbdec009060 comp rx=0 tx=0).stop 2022-04-03T19:57:45.107 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.099+0000 7fbdf60af700 1 -- 172.21.15.77:0/4225273166 shutdown_connections 2022-04-03T19:57:45.107 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.099+0000 7fbdf60af700 1 --2- 172.21.15.77:0/4225273166 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbdf0067a70 0x7fbdf00af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:45.107 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.099+0000 7fbdf60af700 1 --2- 172.21.15.77:0/4225273166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdf006ccf0 0x7fbdf00aeb20 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:45.108 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.099+0000 7fbdf60af700 1 -- 172.21.15.77:0/4225273166 >> 172.21.15.77:0/4225273166 conn(0x7fbdf0069990 msgr2=0x7fbdf00626d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:45.108 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.099+0000 7fbdf60af700 1 -- 172.21.15.77:0/4225273166 shutdown_connections 2022-04-03T19:57:45.108 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.099+0000 7fbdf60af700 1 -- 172.21.15.77:0/4225273166 wait complete. 2022-04-03T19:57:45.108 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.101+0000 7fbdf60af700 1 Processor -- start 2022-04-03T19:57:45.108 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.101+0000 7fbdf60af700 1 -- start start 2022-04-03T19:57:45.109 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.102+0000 7fbdf60af700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdf0067a70 0x7fbdf0168460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:45.109 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.102+0000 7fbdf60af700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbdf006ccf0 0x7fbdf01563d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:45.109 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.102+0000 7fbdf60af700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdf00c5760 0x7fbdf0156910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:45.109 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.102+0000 7fbdf60af700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf00c1790 con 0x7fbdf0067a70 2022-04-03T19:57:45.109 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.102+0000 7fbdf60af700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf00c2f10 con 0x7fbdf00c5760 2022-04-03T19:57:45.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.102+0000 7fbdf60af700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf00bd6c0 con 0x7fbdf006ccf0 2022-04-03T19:57:45.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.102+0000 7fbdf50ad700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdf0067a70 0x7fbdf0168460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:45.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.102+0000 7fbdf50ad700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdf0067a70 0x7fbdf0168460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43268/0 (socket says 172.21.15.77:43268) 2022-04-03T19:57:45.110 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.102+0000 7fbdf50ad700 1 -- 172.21.15.77:0/4107310598 learned_addr learned my addr 172.21.15.77:0/4107310598 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:45.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.102+0000 7fbdf58ae700 1 --2- 172.21.15.77:0/4107310598 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdf00c5760 0x7fbdf0156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:45.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.102+0000 7fbdf48ac700 1 --2- 172.21.15.77:0/4107310598 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbdf006ccf0 0x7fbdf01563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:45.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.103+0000 7fbdf50ad700 1 -- 172.21.15.77:0/4107310598 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbdf006ccf0 msgr2=0x7fbdf01563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:45.111 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.103+0000 7fbdf50ad700 1 --2- 172.21.15.77:0/4107310598 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbdf006ccf0 0x7fbdf01563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:45.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.103+0000 7fbdf50ad700 1 -- 172.21.15.77:0/4107310598 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdf00c5760 msgr2=0x7fbdf0156910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:45.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.103+0000 7fbdf50ad700 1 --2- 172.21.15.77:0/4107310598 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdf00c5760 0x7fbdf0156910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:45.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.103+0000 7fbdf50ad700 1 -- 172.21.15.77:0/4107310598 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec0092a0 con 0x7fbdf0067a70 2022-04-03T19:57:45.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.103+0000 7fbdf50ad700 1 --2- 172.21.15.77:0/4107310598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdf0067a70 0x7fbdf0168460 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fbdec011fc0 tx=0x7fbdec017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:45.112 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.103+0000 7fbddd7fa700 1 -- 172.21.15.77:0/4107310598 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdec000bc0 con 0x7fbdf0067a70 2022-04-03T19:57:45.113 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.103+0000 7fbddd7fa700 1 -- 172.21.15.77:0/4107310598 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdec011900 con 0x7fbdf0067a70 2022-04-03T19:57:45.113 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.103+0000 7fbdf60af700 1 -- 172.21.15.77:0/4107310598 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf00c28e0 con 0x7fbdf0067a70 2022-04-03T19:57:45.113 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.103+0000 7fbddd7fa700 1 -- 172.21.15.77:0/4107310598 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdec025420 con 0x7fbdf0067a70 2022-04-03T19:57:45.113 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.104+0000 7fbdf60af700 1 -- 172.21.15.77:0/4107310598 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf00bdc90 con 0x7fbdf0067a70 2022-04-03T19:57:45.114 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.105+0000 7fbdf60af700 1 -- 172.21.15.77:0/4107310598 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf00c8430 con 0x7fbdf0067a70 2022-04-03T19:57:45.114 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.105+0000 7fbddd7fa700 1 -- 172.21.15.77:0/4107310598 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fbdec02c030 con 0x7fbdf0067a70 2022-04-03T19:57:45.115 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.106+0000 7fbddd7fa700 1 --2- 172.21.15.77:0/4107310598 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbdd0063ce0 0x7fbdd0066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:45.115 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.107+0000 7fbddd7fa700 1 -- 172.21.15.77:0/4107310598 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6718+0+0 (secure 0 0 0) 0x7fbdec081ff0 con 0x7fbdf0067a70 2022-04-03T19:57:45.115 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.107+0000 7fbdf48ac700 1 --2- 172.21.15.77:0/4107310598 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbdd0063ce0 0x7fbdd0066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:45.116 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.107+0000 7fbdf48ac700 1 --2- 172.21.15.77:0/4107310598 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbdd0063ce0 0x7fbdd0066190 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0166010 tx=0x7fbde0007cd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:45.116 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.111+0000 7fbddd7fa700 1 -- 172.21.15.77:0/4107310598 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fbdec050920 con 0x7fbdf0067a70 2022-04-03T19:57:45.344 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.344+0000 7fbdf60af700 1 -- 172.21.15.77:0/4107310598 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdf0069f50 con 0x7fbdf0067a70 2022-04-03T19:57:45.345 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.344+0000 7fbddd7fa700 1 -- 172.21.15.77:0/4107310598 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5353 (secure 0 0 0) 0x7fbdec02a020 con 0x7fbdf0067a70 2022-04-03T19:57:45.345 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:45.346 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:57:44.361632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5668":{"gid":5668,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":33,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:45.347 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.347+0000 7fbdf60af700 1 -- 172.21.15.77:0/4107310598 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbdd0063ce0 msgr2=0x7fbdd0066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:45.348 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.347+0000 7fbdf60af700 1 --2- 172.21.15.77:0/4107310598 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbdd0063ce0 0x7fbdd0066190 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0166010 tx=0x7fbde0007cd0 comp rx=0 tx=0).stop 2022-04-03T19:57:45.348 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.347+0000 7fbdf60af700 1 -- 172.21.15.77:0/4107310598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdf0067a70 msgr2=0x7fbdf0168460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:45.348 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.347+0000 7fbdf60af700 1 --2- 172.21.15.77:0/4107310598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdf0067a70 0x7fbdf0168460 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fbdec011fc0 tx=0x7fbdec017000 comp rx=0 tx=0).stop 2022-04-03T19:57:45.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.347+0000 7fbdf60af700 1 -- 172.21.15.77:0/4107310598 shutdown_connections 2022-04-03T19:57:45.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.347+0000 7fbdf60af700 1 --2- 172.21.15.77:0/4107310598 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbdf00c5760 0x7fbdf0156910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:45.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.347+0000 7fbdf60af700 1 --2- 172.21.15.77:0/4107310598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbdf0067a70 0x7fbdf0168460 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:45.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.347+0000 7fbdf60af700 1 --2- 172.21.15.77:0/4107310598 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fbdd0063ce0 0x7fbdd0066190 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:45.349 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.347+0000 7fbdf60af700 1 --2- 172.21.15.77:0/4107310598 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbdf006ccf0 0x7fbdf01563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:45.350 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.347+0000 7fbdf60af700 1 -- 172.21.15.77:0/4107310598 >> 172.21.15.77:0/4107310598 conn(0x7fbdf0069990 msgr2=0x7fbdf00c8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:45.350 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.347+0000 7fbdf60af700 1 -- 172.21.15.77:0/4107310598 shutdown_connections 2022-04-03T19:57:45.350 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.347+0000 7fbdf60af700 1 -- 172.21.15.77:0/4107310598 wait complete. 2022-04-03T19:57:45.351 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 48 2022-04-03T19:57:45.372 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=5463 2022-04-03T19:57:45.372 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.e 2022-04-03T19:57:45.373 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2022-04-03T19:57:45.373 INFO:tasks.ceph.mds.e:Restarting daemon 2022-04-03T19:57:45.373 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2022-04-03T19:57:45.378 INFO:tasks.ceph.mds.e:Started 2022-04-03T19:57:45.378 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:45.530 INFO:tasks.ceph.mds.e.smithi077.stdout:starting mds.e at 2022-04-03T19:57:45.587 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.586+0000 7ff328ad7700 1 Processor -- start 2022-04-03T19:57:45.587 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.587+0000 7ff328ad7700 1 -- start start 2022-04-03T19:57:45.588 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.588+0000 7ff328ad7700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32406bbe0 0x7ff3241584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:45.589 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.588+0000 7ff328ad7700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff32406bfb0 0x7ff32406d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:45.589 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.588+0000 7ff328ad7700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff32406d260 con 0x7ff32406c500 2022-04-03T19:57:45.589 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.588+0000 7ff328ad7700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff324159450 con 0x7ff32406bfb0 2022-04-03T19:57:45.589 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.588+0000 7ff328ad7700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3240c6330 con 0x7ff32406bbe0 2022-04-03T19:57:45.590 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.588+0000 7ff31bfff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff32406bfb0 0x7ff32406d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:45.590 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.588+0000 7ff3237fe700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32406bbe0 0x7ff3241584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:45.590 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.588+0000 7ff323fff700 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ff32406c500 0x7ff32414eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.77:44402/0 (socket says 172.21.15.77:44402) 2022-04-03T19:57:45.590 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.588+0000 7ff323fff700 1 -- 172.21.15.77:0/1896746655 learned_addr learned my addr 172.21.15.77:0/1896746655 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:57:45.593 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.589+0000 7ff322ffd700 1 -- 172.21.15.77:0/1896746655 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713970005 0 0) 0x7ff32406d260 con 0x7ff32406c500 2022-04-03T19:57:45.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.589+0000 7ff3237fe700 1 -- 172.21.15.77:0/1896746655 >> v2:172.21.15.77:3300/0 conn(0x7ff32406bfb0 msgr2=0x7ff32406d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:45.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.589+0000 7ff3237fe700 1 --2- 172.21.15.77:0/1896746655 >> v2:172.21.15.77:3300/0 conn(0x7ff32406bfb0 0x7ff32406d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:45.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.589+0000 7ff3237fe700 1 -- 172.21.15.77:0/1896746655 >> v1:172.21.15.186:6789/0 conn(0x7ff32406c500 legacy=0x7ff32414eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:45.594 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.589+0000 7ff3237fe700 1 -- 172.21.15.77:0/1896746655 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff324158bb0 con 0x7ff32406bbe0 2022-04-03T19:57:45.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.589+0000 7ff3237fe700 1 --2- 172.21.15.77:0/1896746655 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32406bbe0 0x7ff3241584b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff314009760 tx=0x7ff314009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e681d655923973cb server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:45.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.589+0000 7ff322ffd700 1 -- 172.21.15.77:0/1896746655 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff314011630 con 0x7ff32406bbe0 2022-04-03T19:57:45.595 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.589+0000 7ff322ffd700 1 -- 172.21.15.77:0/1896746655 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff314011790 con 0x7ff32406bbe0 2022-04-03T19:57:45.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.589+0000 7ff322ffd700 1 -- 172.21.15.77:0/1896746655 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff314011b50 con 0x7ff32406bbe0 2022-04-03T19:57:45.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.590+0000 7ff328ad7700 1 -- 172.21.15.77:0/1896746655 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32406bbe0 msgr2=0x7ff3241584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:45.596 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.590+0000 7ff328ad7700 1 --2- 172.21.15.77:0/1896746655 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32406bbe0 0x7ff3241584b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff314009760 tx=0x7ff314009060 comp rx=0 tx=0).stop 2022-04-03T19:57:45.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.590+0000 7ff328ad7700 1 -- 172.21.15.77:0/1896746655 shutdown_connections 2022-04-03T19:57:45.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.590+0000 7ff328ad7700 1 --2- 172.21.15.77:0/1896746655 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32406bbe0 0x7ff3241584b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:45.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.590+0000 7ff328ad7700 1 --2- 172.21.15.77:0/1896746655 >> v2:172.21.15.77:3300/0 conn(0x7ff32406bfb0 0x7ff32406d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:45.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.590+0000 7ff328ad7700 1 -- 172.21.15.77:0/1896746655 >> 172.21.15.77:0/1896746655 conn(0x7ff3240cb6b0 msgr2=0x7ff3240cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:45.597 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.590+0000 7ff328ad7700 1 -- 172.21.15.77:0/1896746655 shutdown_connections 2022-04-03T19:57:45.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.590+0000 7ff328ad7700 1 -- 172.21.15.77:0/1896746655 wait complete. 2022-04-03T19:57:45.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.591+0000 7ff328ad7700 1 Processor -- start 2022-04-03T19:57:45.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.591+0000 7ff328ad7700 1 -- start start 2022-04-03T19:57:45.598 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.592+0000 7ff328ad7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff32406bbe0 0x7ff3240c2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:45.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.592+0000 7ff328ad7700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32406bfb0 0x7ff3240c07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:45.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.592+0000 7ff328ad7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff32406c500 0x7ff3240bf420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:45.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.592+0000 7ff328ad7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff324165c90 con 0x7ff32406bbe0 2022-04-03T19:57:45.599 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.592+0000 7ff328ad7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff324165310 con 0x7ff32406c500 2022-04-03T19:57:45.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.592+0000 7ff328ad7700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3240c1000 con 0x7ff32406bfb0 2022-04-03T19:57:45.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.593+0000 7ff3237fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff32406bbe0 0x7ff3240c2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:45.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.593+0000 7ff3237fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff32406bbe0 0x7ff3240c2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43274/0 (socket says 172.21.15.77:43274) 2022-04-03T19:57:45.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.593+0000 7ff31bfff700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32406bfb0 0x7ff3240c07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:45.600 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.593+0000 7ff3237fe700 1 -- 172.21.15.77:0/3330226046 learned_addr learned my addr 172.21.15.77:0/3330226046 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:45.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.593+0000 7ff323fff700 1 --2- 172.21.15.77:0/3330226046 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff32406c500 0x7ff3240bf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:45.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.593+0000 7ff3237fe700 1 -- 172.21.15.77:0/3330226046 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32406bfb0 msgr2=0x7ff3240c07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:45.601 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.593+0000 7ff3237fe700 1 --2- 172.21.15.77:0/3330226046 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32406bfb0 0x7ff3240c07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:45.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.593+0000 7ff3237fe700 1 -- 172.21.15.77:0/3330226046 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff32406c500 msgr2=0x7ff3240bf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:45.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.593+0000 7ff3237fe700 1 --2- 172.21.15.77:0/3330226046 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff32406c500 0x7ff3240bf420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:45.602 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.593+0000 7ff3237fe700 1 -- 172.21.15.77:0/3330226046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3140092a0 con 0x7ff32406bbe0 2022-04-03T19:57:45.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.594+0000 7ff3237fe700 1 --2- 172.21.15.77:0/3330226046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff32406bbe0 0x7ff3240c2f30 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff314009e00 tx=0x7ff314017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:45.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.594+0000 7ff31b7fe700 1 -- 172.21.15.77:0/3330226046 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff314000bc0 con 0x7ff32406bbe0 2022-04-03T19:57:45.604 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.594+0000 7ff31b7fe700 1 -- 172.21.15.77:0/3330226046 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff314011900 con 0x7ff32406bbe0 2022-04-03T19:57:45.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.594+0000 7ff328ad7700 1 -- 172.21.15.77:0/3330226046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3240c12e0 con 0x7ff32406bbe0 2022-04-03T19:57:45.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.594+0000 7ff31b7fe700 1 -- 172.21.15.77:0/3330226046 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff314025420 con 0x7ff32406bbe0 2022-04-03T19:57:45.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.594+0000 7ff328ad7700 1 -- 172.21.15.77:0/3330226046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3241658b0 con 0x7ff32406bbe0 2022-04-03T19:57:45.605 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.595+0000 7ff328ad7700 1 -- 172.21.15.77:0/3330226046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3240b1720 con 0x7ff32406bbe0 2022-04-03T19:57:45.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.596+0000 7ff31b7fe700 1 -- 172.21.15.77:0/3330226046 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7ff31402c030 con 0x7ff32406bbe0 2022-04-03T19:57:45.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.597+0000 7ff31b7fe700 1 --2- 172.21.15.77:0/3330226046 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff308063d10 0x7ff3080661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:45.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.597+0000 7ff31b7fe700 1 -- 172.21.15.77:0/3330226046 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6718+0+0 (secure 0 0 0) 0x7ff314081fe0 con 0x7ff32406bbe0 2022-04-03T19:57:45.606 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.597+0000 7ff31bfff700 1 --2- 172.21.15.77:0/3330226046 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff308063d10 0x7ff3080661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:45.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.598+0000 7ff31bfff700 1 --2- 172.21.15.77:0/3330226046 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff308063d10 0x7ff3080661c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff324149500 tx=0x7ff30c009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:45.607 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.601+0000 7ff31b7fe700 1 -- 172.21.15.77:0/3330226046 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7ff314050840 con 0x7ff32406bbe0 2022-04-03T19:57:45.832 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.831+0000 7ff328ad7700 1 -- 172.21.15.77:0/3330226046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff32404b050 con 0x7ff32406bbe0 2022-04-03T19:57:45.833 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.832+0000 7ff31b7fe700 1 -- 172.21.15.77:0/3330226046 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5353 (secure 0 0 0) 0x7ff31402a020 con 0x7ff32406bbe0 2022-04-03T19:57:45.833 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:45.833 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:57:44.361632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5668":{"gid":5668,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":33,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:45.835 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.834+0000 7ff328ad7700 1 -- 172.21.15.77:0/3330226046 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff308063d10 msgr2=0x7ff3080661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:45.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.834+0000 7ff328ad7700 1 --2- 172.21.15.77:0/3330226046 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff308063d10 0x7ff3080661c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff324149500 tx=0x7ff30c009060 comp rx=0 tx=0).stop 2022-04-03T19:57:45.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.834+0000 7ff328ad7700 1 -- 172.21.15.77:0/3330226046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff32406bbe0 msgr2=0x7ff3240c2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:45.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.834+0000 7ff328ad7700 1 --2- 172.21.15.77:0/3330226046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff32406bbe0 0x7ff3240c2f30 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff314009e00 tx=0x7ff314017000 comp rx=0 tx=0).stop 2022-04-03T19:57:45.836 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.834+0000 7ff328ad7700 1 -- 172.21.15.77:0/3330226046 shutdown_connections 2022-04-03T19:57:45.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.835+0000 7ff328ad7700 1 --2- 172.21.15.77:0/3330226046 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff32406c500 0x7ff3240bf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:45.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.835+0000 7ff328ad7700 1 --2- 172.21.15.77:0/3330226046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff32406bbe0 0x7ff3240c2f30 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:45.837 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.835+0000 7ff328ad7700 1 --2- 172.21.15.77:0/3330226046 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7ff308063d10 0x7ff3080661c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:45.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.835+0000 7ff328ad7700 1 --2- 172.21.15.77:0/3330226046 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32406bfb0 0x7ff3240c07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:45.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.835+0000 7ff328ad7700 1 -- 172.21.15.77:0/3330226046 >> 172.21.15.77:0/3330226046 conn(0x7ff3240cb6b0 msgr2=0x7ff32414a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:45.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.835+0000 7ff328ad7700 1 -- 172.21.15.77:0/3330226046 shutdown_connections 2022-04-03T19:57:45.838 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:45.835+0000 7ff328ad7700 1 -- 172.21.15.77:0/3330226046 wait complete. 2022-04-03T19:57:45.839 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 48 2022-04-03T19:57:45.859 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2022-04-03T19:57:47.764 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:57:47.769 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:57:47.861 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:57:48.059 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.059+0000 7f1056260700 1 Processor -- start 2022-04-03T19:57:48.060 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.059+0000 7f1056260700 1 -- start start 2022-04-03T19:57:48.063 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.060+0000 7f1056260700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f10500c2010 0x7f1050159050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:48.064 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.060+0000 7f1056260700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f105006a280 0x7f1050155910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:48.064 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.060+0000 7f1056260700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10500c4dc0 0x7f10500c5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:48.064 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.060+0000 7f1056260700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10500c4200 con 0x7f105006a280 2022-04-03T19:57:48.064 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.060+0000 7f1056260700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1050159590 con 0x7f10500c2010 2022-04-03T19:57:48.065 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.060+0000 7f1056260700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1050155480 con 0x7f10500c4dc0 2022-04-03T19:57:48.065 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.061+0000 7f1055a5f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10500c4dc0 0x7f10500c5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:48.065 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.061+0000 7f105525e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f10500c2010 0x7f1050159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:48.065 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.061+0000 7f1054a5d700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f105006a280 0x7f1050155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:48.066 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.061+0000 7f1055a5f700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10500c4dc0 0x7f10500c5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.77:60576/0 (socket says 172.21.15.77:60576) 2022-04-03T19:57:48.066 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.061+0000 7f1055a5f700 1 -- 172.21.15.77:0/1908843527 learned_addr learned my addr 172.21.15.77:0/1908843527 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:48.066 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.061+0000 7f105525e700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f10500c2010 0x7f1050159050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.77:38458/0 (socket says 172.21.15.77:38458) 2022-04-03T19:57:48.066 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.061+0000 7f105525e700 1 -- 172.21.15.77:0/1908843527 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10500c4dc0 msgr2=0x7f10500c5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:48.066 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.061+0000 7f105525e700 1 --2- 172.21.15.77:0/1908843527 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10500c4dc0 0x7f10500c5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:48.067 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.061+0000 7f105525e700 1 -- 172.21.15.77:0/1908843527 >> v2:172.21.15.77:3300/0 conn(0x7f105006a280 msgr2=0x7f1050155910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:57:48.067 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.061+0000 7f105525e700 1 --2- 172.21.15.77:0/1908843527 >> v2:172.21.15.77:3300/0 conn(0x7f105006a280 0x7f1050155910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:48.067 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.061+0000 7f105525e700 1 -- 172.21.15.77:0/1908843527 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10500c2d70 con 0x7f10500c2010 2022-04-03T19:57:48.067 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.061+0000 7f1054a5d700 1 --2- 172.21.15.77:0/1908843527 >> v2:172.21.15.77:3300/0 conn(0x7f105006a280 0x7f1050155910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:57:48.068 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.062+0000 7f105525e700 1 --2- 172.21.15.77:0/1908843527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10500c2010 0x7f1050159050 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f105006db80 tx=0x7f104c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b527d7f14107c752 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:48.068 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.062+0000 7f103ffff700 1 -- 172.21.15.77:0/1908843527 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c0105b0 con 0x7f10500c2010 2022-04-03T19:57:48.069 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.062+0000 7f103ffff700 1 -- 172.21.15.77:0/1908843527 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104c010710 con 0x7f10500c2010 2022-04-03T19:57:48.069 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.062+0000 7f103ffff700 1 -- 172.21.15.77:0/1908843527 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c010ad0 con 0x7f10500c2010 2022-04-03T19:57:48.069 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.062+0000 7f1056260700 1 -- 172.21.15.77:0/1908843527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10500c2010 msgr2=0x7f1050159050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:48.069 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.062+0000 7f1056260700 1 --2- 172.21.15.77:0/1908843527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10500c2010 0x7f1050159050 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f105006db80 tx=0x7f104c009060 comp rx=0 tx=0).stop 2022-04-03T19:57:48.070 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.062+0000 7f1056260700 1 -- 172.21.15.77:0/1908843527 shutdown_connections 2022-04-03T19:57:48.070 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.062+0000 7f1056260700 1 --2- 172.21.15.77:0/1908843527 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10500c4dc0 0x7f10500c5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:48.070 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.062+0000 7f1056260700 1 --2- 172.21.15.77:0/1908843527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10500c2010 0x7f1050159050 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:48.070 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.062+0000 7f1056260700 1 --2- 172.21.15.77:0/1908843527 >> v2:172.21.15.77:3300/0 conn(0x7f105006a280 0x7f1050155910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:48.071 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.062+0000 7f1056260700 1 -- 172.21.15.77:0/1908843527 >> 172.21.15.77:0/1908843527 conn(0x7f105006cb30 msgr2=0x7f105006cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:48.071 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.062+0000 7f1056260700 1 -- 172.21.15.77:0/1908843527 shutdown_connections 2022-04-03T19:57:48.071 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.063+0000 7f1056260700 1 -- 172.21.15.77:0/1908843527 wait complete. 2022-04-03T19:57:48.076 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.064+0000 7f1056260700 1 Processor -- start 2022-04-03T19:57:48.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.064+0000 7f1056260700 1 -- start start 2022-04-03T19:57:48.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.065+0000 7f1056260700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f105006a280 0x7f10501525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:48.077 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.065+0000 7f1056260700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10500c2010 0x7f10501519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:48.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.065+0000 7f1056260700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10500c4dc0 0x7f105014d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:48.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.065+0000 7f1056260700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1050150250 con 0x7f10500c4dc0 2022-04-03T19:57:48.078 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.065+0000 7f1056260700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10500bf6e0 con 0x7f105006a280 2022-04-03T19:57:48.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.065+0000 7f1056260700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10500c0420 con 0x7f10500c2010 2022-04-03T19:57:48.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.065+0000 7f1055a5f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10500c4dc0 0x7f105014d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:48.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.065+0000 7f1055a5f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10500c4dc0 0x7f105014d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43280/0 (socket says 172.21.15.77:43280) 2022-04-03T19:57:48.079 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.065+0000 7f1055a5f700 1 -- 172.21.15.77:0/4138372934 learned_addr learned my addr 172.21.15.77:0/4138372934 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:57:48.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.065+0000 7f1054a5d700 1 --2- 172.21.15.77:0/4138372934 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10500c2010 0x7f10501519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:48.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.065+0000 7f105525e700 1 --2- 172.21.15.77:0/4138372934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f105006a280 0x7f10501525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:48.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.066+0000 7f1055a5f700 1 -- 172.21.15.77:0/4138372934 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10500c2010 msgr2=0x7f10501519e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:48.080 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.066+0000 7f1055a5f700 1 --2- 172.21.15.77:0/4138372934 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10500c2010 0x7f10501519e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:48.081 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.066+0000 7f1055a5f700 1 -- 172.21.15.77:0/4138372934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f105006a280 msgr2=0x7f10501525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:48.081 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.066+0000 7f1055a5f700 1 --2- 172.21.15.77:0/4138372934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f105006a280 0x7f10501525d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:48.081 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.066+0000 7f1055a5f700 1 -- 172.21.15.77:0/4138372934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10400092a0 con 0x7f10500c4dc0 2022-04-03T19:57:48.081 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.066+0000 7f1055a5f700 1 --2- 172.21.15.77:0/4138372934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10500c4dc0 0x7f105014d660 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1048009db0 tx=0x7f104800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:48.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.066+0000 7f103d7fa700 1 -- 172.21.15.77:0/4138372934 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104800c890 con 0x7f10500c4dc0 2022-04-03T19:57:48.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.066+0000 7f103d7fa700 1 -- 172.21.15.77:0/4138372934 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104800f040 con 0x7f10500c4dc0 2022-04-03T19:57:48.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.066+0000 7f1056260700 1 -- 172.21.15.77:0/4138372934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f104c0092a0 con 0x7f10500c4dc0 2022-04-03T19:57:48.082 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.066+0000 7f103d7fa700 1 -- 172.21.15.77:0/4138372934 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104800cc40 con 0x7f10500c4dc0 2022-04-03T19:57:48.083 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.067+0000 7f1056260700 1 -- 172.21.15.77:0/4138372934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f105014fb90 con 0x7f10500c4dc0 2022-04-03T19:57:48.083 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.068+0000 7f1056260700 1 -- 172.21.15.77:0/4138372934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1050155080 con 0x7f10500c4dc0 2022-04-03T19:57:48.083 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.068+0000 7f103d7fa700 1 -- 172.21.15.77:0/4138372934 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f1048003860 con 0x7f10500c4dc0 2022-04-03T19:57:48.083 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.069+0000 7f103d7fa700 1 --2- 172.21.15.77:0/4138372934 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1030063d10 0x7f10300661c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:57:48.083 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.069+0000 7f103d7fa700 1 -- 172.21.15.77:0/4138372934 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f1048080f10 con 0x7f10500c4dc0 2022-04-03T19:57:48.084 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.070+0000 7f105525e700 1 --2- 172.21.15.77:0/4138372934 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1030063d10 0x7f10300661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:57:48.084 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.070+0000 7f105525e700 1 --2- 172.21.15.77:0/4138372934 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1030063d10 0x7f10300661c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f104c009fd0 tx=0x7f104c016000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:57:48.084 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.074+0000 7f103d7fa700 1 -- 172.21.15.77:0/4138372934 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f104804f860 con 0x7f10500c4dc0 2022-04-03T19:57:48.304 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.303+0000 7f1056260700 1 -- 172.21.15.77:0/4138372934 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1050152b20 con 0x7f10500c4dc0 2022-04-03T19:57:48.305 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.304+0000 7f103d7fa700 1 -- 172.21.15.77:0/4138372934 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+6107 (secure 0 0 0) 0x7f1048052e80 con 0x7f10500c4dc0 2022-04-03T19:57:48.306 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:57:48.306 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5833,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/493106414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":493106414},{"type":"v1","addr":"172.21.15.77:6837","nonce":493106414}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:57:44.361632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5668":{"gid":5668,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":33,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:57:48.309 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.307+0000 7f1056260700 1 -- 172.21.15.77:0/4138372934 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1030063d10 msgr2=0x7f10300661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:48.309 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.307+0000 7f1056260700 1 --2- 172.21.15.77:0/4138372934 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1030063d10 0x7f10300661c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f104c009fd0 tx=0x7f104c016000 comp rx=0 tx=0).stop 2022-04-03T19:57:48.309 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.307+0000 7f1056260700 1 -- 172.21.15.77:0/4138372934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10500c4dc0 msgr2=0x7f105014d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:57:48.310 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.307+0000 7f1056260700 1 --2- 172.21.15.77:0/4138372934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10500c4dc0 0x7f105014d660 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1048009db0 tx=0x7f104800b040 comp rx=0 tx=0).stop 2022-04-03T19:57:48.310 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.307+0000 7f1056260700 1 -- 172.21.15.77:0/4138372934 shutdown_connections 2022-04-03T19:57:48.310 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.307+0000 7f1056260700 1 --2- 172.21.15.77:0/4138372934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10500c4dc0 0x7f105014d660 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:48.311 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.307+0000 7f1056260700 1 --2- 172.21.15.77:0/4138372934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f105006a280 0x7f10501525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:48.311 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.307+0000 7f1056260700 1 --2- 172.21.15.77:0/4138372934 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f1030063d10 0x7f10300661c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:48.311 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.307+0000 7f1056260700 1 --2- 172.21.15.77:0/4138372934 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10500c2010 0x7f10501519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:57:48.312 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.307+0000 7f1056260700 1 -- 172.21.15.77:0/4138372934 >> 172.21.15.77:0/4138372934 conn(0x7f105006cb30 msgr2=0x7f1050153c60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:57:48.313 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.307+0000 7f1056260700 1 -- 172.21.15.77:0/4138372934 shutdown_connections 2022-04-03T19:57:48.313 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:57:48.307+0000 7f1056260700 1 -- 172.21.15.77:0/4138372934 wait complete. 2022-04-03T19:57:48.313 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 49 2022-04-03T19:57:48.332 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2022-04-03T19:57:48.332 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 87 secs before thrashing 2022-04-03T19:57:53.734 INFO:tasks.ceph.mds.d.smithi186.stderr:2022-04-03T19:57:53.734+0000 7f1f1c2f9700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-04-03T19:58:17.908 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:58:17.915 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:58:48.047 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:58:48.052 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:59:09.164 INFO:tasks.workunit.client.0.smithi077.stderr:Updating files: 2% (286/12089) Updating files: 2% (321/12089) Updating files: 2% (349/12089) Updating files: 3% (363/12089) Updating files: 3% (454/12089) Updating files: 4% (484/12089) Updating files: 4% (494/12089) Updating files: 5% (605/12089) Updating files: 5% (658/12089) Updating files: 6% (726/12089) Updating files: 6% (735/12089) Updating files: 7% (847/12089) Updating files: 7% (862/12089) Updating files: 7% (881/12089) Updating files: 7% (899/12089) Updating files: 7% (961/12089) Updating files: 8% (968/12089) Updating files: 8% (972/12089) Updating files: 8% (1060/12089) Updating files: 9% (1089/12089) Updating files: 9% (1097/12089) Updating files: 9% (1186/12089) Updating files: 10% (1209/12089) Updating files: 10% (1255/12089) Updating files: 10% (1273/12089) Updating files: 11% (1330/12089) Updating files: 11% (1333/12089) Updating files: 11% (1415/12089) Updating files: 11% (1440/12089) Updating files: 12% (1451/12089) Updating files: 12% (1523/12089) Updating files: 13% (1572/12089) Updating files: 13% (1577/12089) Updating files: 13% (1614/12089) Updating files: 13% (1632/12089) Updating files: 13% (1663/12089) Updating files: 14% (1693/12089) Updating files: 14% (1717/12089) Updating files: 14% (1726/12089) Updating files: 14% (1731/12089) Updating files: 15% (1814/12089) Updating files: 15% (1834/12089) Updating files: 15% (1924/12089) Updating files: 16% (1935/12089) Updating files: 16% (2010/12089) Updating files: 17% (2056/12089) Updating files: 17% (2079/12089) Updating files: 17% (2112/12089) Updating files: 17% (2176/12089) Updating files: 18% (2177/12089) Updating files: 18% (2274/12089) Updating files: 19% (2297/12089) Updating files: 19% (2338/12089) Updating files: 19% (2387/12089) Updating files: 19% (2398/12089) Updating files: 20% (2418/12089) Updating files: 20% (2436/12089) Updating files: 20% (2444/12089) Updating files: 20% (2530/12089) Updating files: 21% (2539/12089) Updating files: 21% (2655/12089) Updating files: 22% (2660/12089) Updating files: 22% (2688/12089) Updating files: 23% (2781/12089) Updating files: 23% (2838/12089) Updating files: 23% (2842/12089) Updating files: 24% (2902/12089) Updating files: 24% (2920/12089) Updating files: 24% (2968/12089) Updating files: 25% (3023/12089) Updating files: 25% (3091/12089) Updating files: 26% (3144/12089) Updating files: 26% (3204/12089) Updating files: 27% (3265/12089) Updating files: 27% (3266/12089) Updating files: 27% (3298/12089) Updating files: 27% (3309/12089) Updating files: 28% (3385/12089) Updating files: 28% (3417/12089) Updating files: 28% (3448/12089) Updating files: 28% (3464/12089) Updating files: 29% (3506/12089) Updating files: 29% (3578/12089) Updating files: 29% (3618/12089) Updating files: 30% (3627/12089) Updating files: 30% (3637/12089) Updating files: 30% (3740/12089) Updating files: 31% (3748/12089) Updating files: 31% (3772/12089) Updating files: 31% (3805/12089) Updating files: 31% (3826/12089) Updating files: 32% (3869/12089) Updating files: 32% (3919/12089) Updating files: 32% (3968/12089) Updating files: 33% (3990/12089) Updating files: 34% (4111/12089) Updating files: 34% (4145/12089) Updating files: 34% (4206/12089) Updating files: 35% (4232/12089) Updating files: 35% (4335/12089) Updating files: 36% (4353/12089) Updating files: 36% (4399/12089) Updating files: 37% (4473/12089) Updating files: 37% (4518/12089) Updating files: 37% (4558/12089) Updating files: 38% (4594/12089) Updating files: 38% (4656/12089) Updating files: 38% (4699/12089) Updating files: 39% (4715/12089) Updating files: 39% (4736/12089) Updating files: 39% (4740/12089) Updating files: 39% (4784/12089) Updating files: 39% (4819/12089) Updating files: 40% (4836/12089) Updating files: 40% (4839/12089) Updating files: 41% (4957/12089) Updating files: 41% (5047/12089) Updating files: 42% (5078/12089) Updating files: 42% (5096/12089) Updating files: 42% (5121/12089) Updating files: 42% (5125/12089) Updating files: 43% (5199/12089) Updating files: 43% (5203/12089) Updating files: 43% (5251/12089) Updating files: 44% (5320/12089) Updating files: 44% (5323/12089) Updating files: 44% (5376/12089) Updating files: 44% (5437/12089) Updating files: 45% (5441/12089) Updating files: 45% (5463/12089) Updating files: 45% (5555/12089) Updating files: 46% (5561/12089) Updating files: 46% (5655/12089) Updating files: 47% (5682/12089) Updating files: 47% (5696/12089) Updating files: 48% (5803/12089) Updating files: 48% (5825/12089) Updating files: 48% (5906/12089) Updating files: 49% (5924/12089) Updating files: 49% (6014/12089) Updating files: 50% (6045/12089) Updating files: 50% (6053/12089) Updating files: 50% (6140/12089) Updating files: 51% (6166/12089) Updating files: 51% (6270/12089) Updating files: 52% (6287/12089) Updating files: 52% (6381/12089) Updating files: 53% (6408/12089) Updating files: 53% (6443/12089) Updating files: 53% (6499/12089) Updating files: 54% (6529/12089) Updating files: 54% (6591/12089) Updating files: 55% (6649/12089) Updating files: 55% (6664/12089) Updating files: 56% (6770/12089) Updating files: 56% (6776/12089) Updating files: 56% (6782/12089) Updating files: 56% (6844/12089) Updating files: 57% (6891/12089) Updating files: 57% (6971/12089) Updating files: 57% (6979/12089) Updating files: 58% (7012/12089) Updating files: 58% (7024/12089) Updating files: 59% (7133/12089) Updating files: 59% (7155/12089) Updating files: 60% (7254/12089) Updating files: 60% (7323/12089) Updating files: 61% (7375/12089) Updating files: 61% (7409/12089) Updating files: 61% (7476/12089) Updating files: 62% (7496/12089) Updating files: 62% (7560/12089) Updating files: 62% (7606/12089) Updating files: 63% (7617/12089) Updating files: 63% (7667/12089) Updating files: 63% (7688/12089) Updating files: 63% (7736/12089) Updating files: 64% (7737/12089) Updating files: 64% (7776/12089) Updating files: 65% (7858/12089) Updating files: 65% (7955/12089) Updating files: 66% (7979/12089) Updating files: 66% (8002/12089) Updating files: 66% (8072/12089) Updating files: 66% (8077/12089) Updating files: 67% (8100/12089) Updating files: 67% (8121/12089) Updating files: 67% (8202/12089) Updating files: 68% (8221/12089) Updating files: 68% (8300/12089) Updating files: 69% (8342/12089) Updating files: 70% (8463/12089) Updating files: 70% (8540/12089) Updating files: 70% (8560/12089) Updating files: 71% (8584/12089) Updating files: 71% (8609/12089) Updating files: 72% (8705/12089) Updating files: 72% (8772/12089) Updating files: 72% (8774/12089) Updating files: 72% (8801/12089) Updating files: 73% (8825/12089) Updating files: 73% (8843/12089) Updating files: 73% (8893/12089) Updating files: 73% (8919/12089) Updating files: 74% (8946/12089) Updating files: 74% (8955/12089) Updating files: 74% (8990/12089) Updating files: 74% (9051/12089) Updating files: 75% (9067/12089) Updating files: 75% (9117/12089) Updating files: 76% (9188/12089) Updating files: 76% (9303/12089) Updating files: 77% (9309/12089) Updating files: 78% (9430/12089) Updating files: 78% (9434/12089) Updating files: 78% (9438/12089) Updating files: 78% (9460/12089) Updating files: 78% (9471/12089) Updating files: 78% (9550/12089) Updating files: 79% (9551/12089) Updating files: 79% (9633/12089) Updating files: 79% (9644/12089) Updating files: 79% (9653/12089) Updating files: 80% (9672/12089) Updating files: 80% (9712/12089) Updating files: 81% (9793/12089) Updating files: 81% (9912/12089) Updating files: 82% (9913/12089) Updating files: 82% (9923/12089) Updating files: 82% (9988/12089) Updating files: 82% (10009/12089) Updating files: 83% (10034/12089) Updating files: 83% (10117/12089) Updating files: 84% (10155/12089) Updating files: 84% (10237/12089) Updating files: 85% (10276/12089) Updating files: 85% (10345/12089) Updating files: 85% (10368/12089) Updating files: 86% (10397/12089) Updating files: 86% (10455/12089) Updating files: 86% (10513/12089) Updating files: 86% (10514/12089) Updating files: 87% (10518/12089) Updating files: 87% (10560/12089) Updating files: 87% (10582/12089) Updating files: 88% (10639/12089) Updating files: 88% (10717/12089) Updating files: 89% (10760/12089) Updating files: 89% (10777/12089) Updating files: 89% (10801/12089) Updating files: 90% (10881/12089) Updating files: 91% (11001/12089) Updating files: 91% (11014/12089) Updating files: 91% (11086/12089) Updating files: 92% (11122/12089) Updating files: 92% (11142/12089) Updating files: 92% (11232/12089) Updating files: 93% (11243/12089) Updating files: 94% (11364/12089) Updating files: 94% (11443/12089) Updating files: 95% (11485/12089) Updating files: 96% (11606/12089) Updating files: 96% (11690/12089) Updating files: 96% (11710/12089) Updating files: 97% (11727/12089) Updating files: 97% (11737/12089) Updating files: 97% (11846/12089) Updating files: 98% (11848/12089) Updating files: 98% (11850/12089) Updating files: 98% (11921/12089) Updating files: 99% (11969/12089) Updating files: 99% (12069/12089) Updating files: 100% (12089/12089) Updating files: 100% (12089/12089), done. 2022-04-03T19:59:09.362 INFO:tasks.workunit.client.0.smithi077.stdout:HEAD is now at 204c175eea3 msg: blobhash on entity_name_t causes strange badness... should look into that, but avoid for now 2022-04-03T19:59:09.394 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir .snap/v0.1 2022-04-03T19:59:09.687 INFO:tasks.workunit.client.0.smithi077.stderr:+ for v in $versions 2022-04-03T19:59:09.687 INFO:tasks.workunit.client.0.smithi077.stderr:+ ver=v0.2 2022-04-03T19:59:09.687 INFO:tasks.workunit.client.0.smithi077.stderr:+ echo v0.2 2022-04-03T19:59:09.688 INFO:tasks.workunit.client.0.smithi077.stdout:v0.2 2022-04-03T19:59:09.689 INFO:tasks.workunit.client.0.smithi077.stderr:+ git reset --hard v0.2 2022-04-03T19:59:13.515 INFO:tasks.workunit.client.0.smithi077.stderr:Updating files: 20% (45/222) Updating files: 21% (47/222) Updating files: 22% (49/222) Updating files: 23% (52/222) Updating files: 24% (54/222) Updating files: 25% (56/222) Updating files: 26% (58/222) Updating files: 27% (60/222) Updating files: 28% (63/222) Updating files: 29% (65/222) Updating files: 30% (67/222) Updating files: 31% (69/222) Updating files: 32% (72/222) Updating files: 33% (74/222) Updating files: 34% (76/222) Updating files: 35% (78/222) Updating files: 36% (80/222) Updating files: 37% (83/222) Updating files: 38% (85/222) Updating files: 39% (87/222) Updating files: 40% (89/222) Updating files: 41% (92/222) Updating files: 42% (94/222) Updating files: 43% (96/222) Updating files: 44% (98/222) Updating files: 45% (100/222) Updating files: 46% (103/222) Updating files: 47% (105/222) Updating files: 48% (107/222) Updating files: 49% (109/222) Updating files: 50% (111/222) Updating files: 51% (114/222) Updating files: 52% (116/222) Updating files: 53% (118/222) Updating files: 54% (120/222) Updating files: 55% (123/222) Updating files: 56% (125/222) Updating files: 57% (127/222) Updating files: 58% (129/222) Updating files: 59% (131/222) Updating files: 60% (134/222) Updating files: 61% (136/222) Updating files: 62% (138/222) Updating files: 63% (140/222) Updating files: 64% (143/222) Updating files: 65% (145/222) Updating files: 66% (147/222) Updating files: 67% (149/222) Updating files: 68% (151/222) Updating files: 69% (154/222) Updating files: 70% (156/222) Updating files: 70% (157/222) Updating files: 71% (158/222) Updating files: 72% (160/222) Updating files: 73% (163/222) Updating files: 74% (165/222) Updating files: 75% (167/222) Updating files: 76% (169/222) Updating files: 77% (171/222) Updating files: 78% (174/222) Updating files: 79% (176/222) Updating files: 80% (178/222) Updating files: 81% (180/222) Updating files: 82% (183/222) Updating files: 83% (185/222) Updating files: 84% (187/222) Updating files: 85% (189/222) Updating files: 86% (191/222) Updating files: 87% (194/222) Updating files: 88% (196/222) Updating files: 89% (198/222) Updating files: 90% (200/222) Updating files: 91% (203/222) Updating files: 92% (205/222) Updating files: 93% (207/222) Updating files: 94% (209/222) Updating files: 95% (211/222) Updating files: 96% (214/222) Updating files: 97% (216/222) Updating files: 98% (218/222) Updating files: 99% (220/222) Updating files: 100% (222/222) Updating files: 100% (222/222), done. 2022-04-03T19:59:15.333 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:59:15.476 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.475+0000 7f8d25a2c700 1 Processor -- start 2022-04-03T19:59:15.476 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.475+0000 7f8d25a2c700 1 -- start start 2022-04-03T19:59:15.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.476+0000 7f8d25a2c700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8d2006bfb0 0x7f8d201584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:59:15.483 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.476+0000 7f8d25a2c700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8d2006c500 0x7f8d2014eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:59:15.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.476+0000 7f8d25a2c700 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2006d260 con 0x7f8d2006bbe0 2022-04-03T19:59:15.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.476+0000 7f8d25a2c700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d20159450 con 0x7f8d2006bfb0 2022-04-03T19:59:15.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.476+0000 7f8d25a2c700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d200c6330 con 0x7f8d2006c500 2022-04-03T19:59:15.484 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.476+0000 7f8d1ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8d2006bfb0 0x7f8d201584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:59:15.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.476+0000 7f8d1ffff700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8d2006bfb0 0x7f8d201584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43282/0 (socket says 172.21.15.77:43282) 2022-04-03T19:59:15.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.476+0000 7f8d1ffff700 1 -- 172.21.15.77:0/3142687779 learned_addr learned my addr 172.21.15.77:0/3142687779 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:59:15.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.476+0000 7f8d2522b700 1 --2- 172.21.15.77:0/3142687779 >> v2:172.21.15.186:3300/0 conn(0x7f8d2006c500 0x7f8d2014eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:59:15.485 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.477+0000 7f8d1ffff700 1 -- 172.21.15.77:0/3142687779 >> v2:172.21.15.186:3300/0 conn(0x7f8d2006c500 msgr2=0x7f8d2014eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:59:15.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.477+0000 7f8d1ffff700 1 --2- 172.21.15.77:0/3142687779 >> v2:172.21.15.186:3300/0 conn(0x7f8d2006c500 0x7f8d2014eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:15.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.477+0000 7f8d1ffff700 1 -- 172.21.15.77:0/3142687779 >> v1:172.21.15.186:6789/0 conn(0x7f8d2006bbe0 legacy=0x7f8d2006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:59:15.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.477+0000 7f8d1ffff700 1 -- 172.21.15.77:0/3142687779 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d20158bb0 con 0x7f8d2006bfb0 2022-04-03T19:59:15.486 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.477+0000 7f8d1ffff700 1 --2- 172.21.15.77:0/3142687779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d2006bfb0 0x7f8d201584b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8d200cc6b0 tx=0x7f8d080090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390d7e226572db4b server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:59:15.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.477+0000 7f8d1f7fe700 1 -- 172.21.15.77:0/3142687779 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d08004030 con 0x7f8d2006bfb0 2022-04-03T19:59:15.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.477+0000 7f8d1f7fe700 1 -- 172.21.15.77:0/3142687779 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d08004190 con 0x7f8d2006bfb0 2022-04-03T19:59:15.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.477+0000 7f8d1f7fe700 1 -- 172.21.15.77:0/3142687779 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d08011630 con 0x7f8d2006bfb0 2022-04-03T19:59:15.487 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.477+0000 7f8d25a2c700 1 -- 172.21.15.77:0/3142687779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d2006bfb0 msgr2=0x7f8d201584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:59:15.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.477+0000 7f8d25a2c700 1 --2- 172.21.15.77:0/3142687779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d2006bfb0 0x7f8d201584b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8d200cc6b0 tx=0x7f8d080090f0 comp rx=0 tx=0).stop 2022-04-03T19:59:15.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.477+0000 7f8d25a2c700 1 -- 172.21.15.77:0/3142687779 shutdown_connections 2022-04-03T19:59:15.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.477+0000 7f8d25a2c700 1 --2- 172.21.15.77:0/3142687779 >> v2:172.21.15.186:3300/0 conn(0x7f8d2006c500 0x7f8d2014eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:15.488 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.477+0000 7f8d25a2c700 1 --2- 172.21.15.77:0/3142687779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d2006bfb0 0x7f8d201584b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:15.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.477+0000 7f8d25a2c700 1 -- 172.21.15.77:0/3142687779 >> 172.21.15.77:0/3142687779 conn(0x7f8d200cb6b0 msgr2=0x7f8d200cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:59:15.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.478+0000 7f8d25a2c700 1 -- 172.21.15.77:0/3142687779 shutdown_connections 2022-04-03T19:59:15.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.478+0000 7f8d25a2c700 1 -- 172.21.15.77:0/3142687779 wait complete. 2022-04-03T19:59:15.489 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.479+0000 7f8d25a2c700 1 Processor -- start 2022-04-03T19:59:15.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.479+0000 7f8d25a2c700 1 -- start start 2022-04-03T19:59:15.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.479+0000 7f8d25a2c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d2006bbe0 0x7f8d200ca650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:59:15.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.479+0000 7f8d25a2c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d2006bfb0 0x7f8d20153a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:59:15.490 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.479+0000 7f8d25a2c700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d2006c500 0x7f8d20153f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:59:15.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.479+0000 7f8d25a2c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d20164e40 con 0x7f8d2006bfb0 2022-04-03T19:59:15.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.479+0000 7f8d25a2c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d20165050 con 0x7f8d2006bbe0 2022-04-03T19:59:15.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.479+0000 7f8d25a2c700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d20165330 con 0x7f8d2006c500 2022-04-03T19:59:15.491 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.480+0000 7f8d1ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d2006bfb0 0x7f8d20153a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:59:15.492 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.480+0000 7f8d1ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d2006bfb0 0x7f8d20153a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43284/0 (socket says 172.21.15.77:43284) 2022-04-03T19:59:15.492 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.480+0000 7f8d1ffff700 1 -- 172.21.15.77:0/3216994572 learned_addr learned my addr 172.21.15.77:0/3216994572 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:59:15.492 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.480+0000 7f8d2522b700 1 --2- 172.21.15.77:0/3216994572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d2006c500 0x7f8d20153f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:59:15.492 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.480+0000 7f8d24a2a700 1 --2- 172.21.15.77:0/3216994572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d2006bbe0 0x7f8d200ca650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:59:15.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.480+0000 7f8d1ffff700 1 -- 172.21.15.77:0/3216994572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d2006c500 msgr2=0x7f8d20153f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:59:15.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.480+0000 7f8d1ffff700 1 --2- 172.21.15.77:0/3216994572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d2006c500 0x7f8d20153f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:15.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.480+0000 7f8d1ffff700 1 -- 172.21.15.77:0/3216994572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d2006bbe0 msgr2=0x7f8d200ca650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:59:15.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.480+0000 7f8d1ffff700 1 --2- 172.21.15.77:0/3216994572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d2006bbe0 0x7f8d200ca650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:15.493 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.480+0000 7f8d1ffff700 1 -- 172.21.15.77:0/3216994572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d08009370 con 0x7f8d2006bfb0 2022-04-03T19:59:15.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.480+0000 7f8d1ffff700 1 --2- 172.21.15.77:0/3216994572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d2006bfb0 0x7f8d20153a10 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8d200632c0 tx=0x7f8d08016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:59:15.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.480+0000 7f8d1cff9700 1 -- 172.21.15.77:0/3216994572 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d080036f0 con 0x7f8d2006bfb0 2022-04-03T19:59:15.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.480+0000 7f8d1cff9700 1 -- 172.21.15.77:0/3216994572 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d08003850 con 0x7f8d2006bfb0 2022-04-03T19:59:15.494 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.480+0000 7f8d25a2c700 1 -- 172.21.15.77:0/3216994572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d20165610 con 0x7f8d2006bfb0 2022-04-03T19:59:15.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.480+0000 7f8d1cff9700 1 -- 172.21.15.77:0/3216994572 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d08003c10 con 0x7f8d2006bfb0 2022-04-03T19:59:15.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.480+0000 7f8d25a2c700 1 -- 172.21.15.77:0/3216994572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d201554b0 con 0x7f8d2006bfb0 2022-04-03T19:59:15.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.481+0000 7f8d25a2c700 1 -- 172.21.15.77:0/3216994572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d20153860 con 0x7f8d2006bfb0 2022-04-03T19:59:15.495 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.482+0000 7f8d1cff9700 1 -- 172.21.15.77:0/3216994572 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f8d0802b030 con 0x7f8d2006bfb0 2022-04-03T19:59:15.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.482+0000 7f8d1cff9700 1 --2- 172.21.15.77:0/3216994572 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8d04063ce0 0x7f8d04066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:59:15.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.482+0000 7f8d1cff9700 1 -- 172.21.15.77:0/3216994572 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f8d08082300 con 0x7f8d2006bfb0 2022-04-03T19:59:15.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.483+0000 7f8d24a2a700 1 --2- 172.21.15.77:0/3216994572 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8d04063ce0 0x7f8d04066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:59:15.496 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.483+0000 7f8d24a2a700 1 --2- 172.21.15.77:0/3216994572 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8d04063ce0 0x7f8d04066190 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8d10001ea0 tx=0x7f8d10009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:59:15.497 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.487+0000 7f8d1cff9700 1 -- 172.21.15.77:0/3216994572 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f8d08050c50 con 0x7f8d2006bfb0 2022-04-03T19:59:15.699 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.698+0000 7f8d25a2c700 1 -- 172.21.15.77:0/3216994572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d2004b050 con 0x7f8d2006bfb0 2022-04-03T19:59:15.699 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.698+0000 7f8d1cff9700 1 -- 172.21.15.77:0/3216994572 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+6107 (secure 0 0 0) 0x7f8d08054270 con 0x7f8d2006bfb0 2022-04-03T19:59:15.699 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:59:15.700 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5833,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/493106414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":493106414},{"type":"v1","addr":"172.21.15.77:6837","nonce":493106414}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:57:44.361632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5668":{"gid":5668,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":33,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:59:15.701 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.700+0000 7f8d25a2c700 1 -- 172.21.15.77:0/3216994572 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8d04063ce0 msgr2=0x7f8d04066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:59:15.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.700+0000 7f8d25a2c700 1 --2- 172.21.15.77:0/3216994572 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8d04063ce0 0x7f8d04066190 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8d10001ea0 tx=0x7f8d10009040 comp rx=0 tx=0).stop 2022-04-03T19:59:15.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.700+0000 7f8d25a2c700 1 -- 172.21.15.77:0/3216994572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d2006bfb0 msgr2=0x7f8d20153a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:59:15.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.700+0000 7f8d25a2c700 1 --2- 172.21.15.77:0/3216994572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d2006bfb0 0x7f8d20153a10 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8d200632c0 tx=0x7f8d08016000 comp rx=0 tx=0).stop 2022-04-03T19:59:15.702 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.701+0000 7f8d25a2c700 1 -- 172.21.15.77:0/3216994572 shutdown_connections 2022-04-03T19:59:15.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.701+0000 7f8d25a2c700 1 --2- 172.21.15.77:0/3216994572 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8d04063ce0 0x7f8d04066190 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:15.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.701+0000 7f8d25a2c700 1 --2- 172.21.15.77:0/3216994572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d2006c500 0x7f8d20153f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:15.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.701+0000 7f8d25a2c700 1 --2- 172.21.15.77:0/3216994572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d2006bbe0 0x7f8d200ca650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:15.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.701+0000 7f8d25a2c700 1 --2- 172.21.15.77:0/3216994572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8d2006bfb0 0x7f8d20153a10 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:15.703 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.701+0000 7f8d25a2c700 1 -- 172.21.15.77:0/3216994572 >> 172.21.15.77:0/3216994572 conn(0x7f8d200cb6b0 msgr2=0x7f8d20148e20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:59:15.704 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.701+0000 7f8d25a2c700 1 -- 172.21.15.77:0/3216994572 shutdown_connections 2022-04-03T19:59:15.704 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.701+0000 7f8d25a2c700 1 -- 172.21.15.77:0/3216994572 wait complete. 2022-04-03T19:59:15.705 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 49 2022-04-03T19:59:15.715 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2022-04-03T19:59:15.716 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2022-04-03T19:59:15.716 INFO:teuthology.orchestra.run:waiting for 300 2022-04-03T19:59:15.731 INFO:tasks.ceph.mds.d:Stopped 2022-04-03T19:59:15.731 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:59:15.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.860+0000 7f829e54a700 1 Processor -- start 2022-04-03T19:59:15.861 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.860+0000 7f829e54a700 1 -- start start 2022-04-03T19:59:15.867 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.861+0000 7f829e54a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82980c3440 0x7f82980c3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:59:15.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.861+0000 7f829e54a700 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f82981552d0 0x7f829806c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:59:15.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.861+0000 7f829e54a700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829806bf30 con 0x7f82980c89e0 2022-04-03T19:59:15.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.861+0000 7f829e54a700 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82980c4e70 con 0x7f82981552d0 2022-04-03T19:59:15.868 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.861+0000 7f829e54a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8298156270 con 0x7f82980c3440 2022-04-03T19:59:15.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.861+0000 7f829cae7700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f82980c89e0 0x7f82980c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38598/0 (socket says 172.21.15.77:38598) 2022-04-03T19:59:15.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.861+0000 7f829cae7700 1 -- 172.21.15.77:0/2747513695 learned_addr learned my addr 172.21.15.77:0/2747513695 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:59:15.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.861+0000 7f82977fe700 1 --2- 172.21.15.77:0/2747513695 >> v2:172.21.15.186:3300/0 conn(0x7f82981552d0 0x7f829806c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:59:15.869 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.861+0000 7f8297fff700 1 --2- 172.21.15.77:0/2747513695 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82980c3440 0x7f82980c3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:59:15.870 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.862+0000 7f8296ffd700 1 -- 172.21.15.77:0/2747513695 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2317471381 0 0) 0x7f829806bf30 con 0x7f82980c89e0 2022-04-03T19:59:15.870 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.862+0000 7f8296ffd700 1 -- 172.21.15.77:0/2747513695 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82840033f0 con 0x7f82980c89e0 2022-04-03T19:59:15.870 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.862+0000 7f8297fff700 1 -- 172.21.15.77:0/2747513695 >> v2:172.21.15.186:3300/0 conn(0x7f82981552d0 msgr2=0x7f829806c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:59:15.870 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.862+0000 7f8297fff700 1 --2- 172.21.15.77:0/2747513695 >> v2:172.21.15.186:3300/0 conn(0x7f82981552d0 0x7f829806c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:15.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.862+0000 7f8297fff700 1 -- 172.21.15.77:0/2747513695 >> v1:172.21.15.77:6789/0 conn(0x7f82980c89e0 legacy=0x7f82980c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:59:15.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.862+0000 7f8297fff700 1 -- 172.21.15.77:0/2747513695 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8298155a10 con 0x7f82980c3440 2022-04-03T19:59:15.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.862+0000 7f82977fe700 1 --2- 172.21.15.77:0/2747513695 >> v2:172.21.15.186:3300/0 conn(0x7f82981552d0 0x7f829806c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-04-03T19:59:15.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.862+0000 7f8297fff700 1 --2- 172.21.15.77:0/2747513695 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82980c3440 0x7f82980c3810 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8288009e60 tx=0x7f8288009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9dca42741cac7d2 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:59:15.871 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.862+0000 7f8296ffd700 1 -- 172.21.15.77:0/2747513695 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828800e070 con 0x7f82980c3440 2022-04-03T19:59:15.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.862+0000 7f8296ffd700 1 -- 172.21.15.77:0/2747513695 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828800bab0 con 0x7f82980c3440 2022-04-03T19:59:15.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.862+0000 7f8296ffd700 1 -- 172.21.15.77:0/2747513695 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828800be30 con 0x7f82980c3440 2022-04-03T19:59:15.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.863+0000 7f829e54a700 1 -- 172.21.15.77:0/2747513695 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82980c3440 msgr2=0x7f82980c3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:59:15.872 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.863+0000 7f829e54a700 1 --2- 172.21.15.77:0/2747513695 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82980c3440 0x7f82980c3810 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8288009e60 tx=0x7f8288009780 comp rx=0 tx=0).stop 2022-04-03T19:59:15.873 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.863+0000 7f829e54a700 1 -- 172.21.15.77:0/2747513695 shutdown_connections 2022-04-03T19:59:15.873 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.863+0000 7f829e54a700 1 --2- 172.21.15.77:0/2747513695 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82980c3440 0x7f82980c3810 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:15.873 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.863+0000 7f829e54a700 1 --2- 172.21.15.77:0/2747513695 >> v2:172.21.15.186:3300/0 conn(0x7f82981552d0 0x7f829806c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:15.873 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.863+0000 7f829e54a700 1 -- 172.21.15.77:0/2747513695 >> 172.21.15.77:0/2747513695 conn(0x7f82980cc300 msgr2=0x7f82980c82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:59:15.873 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.863+0000 7f829e54a700 1 -- 172.21.15.77:0/2747513695 shutdown_connections 2022-04-03T19:59:15.874 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.863+0000 7f829e54a700 1 -- 172.21.15.77:0/2747513695 wait complete. 2022-04-03T19:59:15.874 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.864+0000 7f829e54a700 1 Processor -- start 2022-04-03T19:59:15.874 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.864+0000 7f829e54a700 1 -- start start 2022-04-03T19:59:15.874 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.865+0000 7f829e54a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82980c3440 0x7f8298150e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:59:15.874 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.865+0000 7f829e54a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82980c89e0 0x7f82981681b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:59:15.875 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.865+0000 7f829e54a700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82981552d0 0x7f82981686f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:59:15.875 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.865+0000 7f829e54a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8298168d70 con 0x7f82980c89e0 2022-04-03T19:59:15.875 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.865+0000 7f829e54a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82980c0850 con 0x7f82980c3440 2022-04-03T19:59:15.875 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.865+0000 7f829e54a700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82980c0ce0 con 0x7f82981552d0 2022-04-03T19:59:15.876 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.865+0000 7f82977fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82980c89e0 0x7f82981681b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:59:15.876 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.865+0000 7f82977fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82980c89e0 0x7f82981681b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43286/0 (socket says 172.21.15.77:43286) 2022-04-03T19:59:15.876 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.865+0000 7f82977fe700 1 -- 172.21.15.77:0/503650660 learned_addr learned my addr 172.21.15.77:0/503650660 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:59:15.877 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.865+0000 7f829cae7700 1 --2- 172.21.15.77:0/503650660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82981552d0 0x7f82981686f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:59:15.877 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.865+0000 7f8297fff700 1 --2- 172.21.15.77:0/503650660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82980c3440 0x7f8298150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:59:15.877 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.865+0000 7f82977fe700 1 -- 172.21.15.77:0/503650660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82981552d0 msgr2=0x7f82981686f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:59:15.877 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.865+0000 7f82977fe700 1 --2- 172.21.15.77:0/503650660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82981552d0 0x7f82981686f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:15.878 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.865+0000 7f82977fe700 1 -- 172.21.15.77:0/503650660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82980c3440 msgr2=0x7f8298150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:59:15.878 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.865+0000 7f82977fe700 1 --2- 172.21.15.77:0/503650660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82980c3440 0x7f8298150e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:15.878 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.865+0000 7f82977fe700 1 -- 172.21.15.77:0/503650660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f828c009250 con 0x7f82980c89e0 2022-04-03T19:59:15.878 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.866+0000 7f82977fe700 1 --2- 172.21.15.77:0/503650660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82980c89e0 0x7f82981681b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f828c00ecb0 tx=0x7f828c006f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:59:15.879 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.866+0000 7f827ffff700 1 -- 172.21.15.77:0/503650660 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c005880 con 0x7f82980c89e0 2022-04-03T19:59:15.879 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.866+0000 7f827ffff700 1 -- 172.21.15.77:0/503650660 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828c0059e0 con 0x7f82980c89e0 2022-04-03T19:59:15.879 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.866+0000 7f829e54a700 1 -- 172.21.15.77:0/503650660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8288009a20 con 0x7f82980c89e0 2022-04-03T19:59:15.879 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.866+0000 7f827ffff700 1 -- 172.21.15.77:0/503650660 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c005d80 con 0x7f82980c89e0 2022-04-03T19:59:15.879 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.866+0000 7f829e54a700 1 -- 172.21.15.77:0/503650660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82980c04e0 con 0x7f82980c89e0 2022-04-03T19:59:15.880 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.867+0000 7f829e54a700 1 -- 172.21.15.77:0/503650660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f829804b890 con 0x7f82980c89e0 2022-04-03T19:59:15.880 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.867+0000 7f827ffff700 1 -- 172.21.15.77:0/503650660 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7f828c022070 con 0x7f82980c89e0 2022-04-03T19:59:15.881 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.868+0000 7f827ffff700 1 --2- 172.21.15.77:0/503650660 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8278063ce0 0x7f8278066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:59:15.881 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.868+0000 7f827ffff700 1 -- 172.21.15.77:0/503650660 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6718+0+0 (secure 0 0 0) 0x7f828c080ec0 con 0x7f82980c89e0 2022-04-03T19:59:15.881 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.868+0000 7f8297fff700 1 --2- 172.21.15.77:0/503650660 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8278063ce0 0x7f8278066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:59:15.882 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.868+0000 7f8297fff700 1 --2- 172.21.15.77:0/503650660 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8278063ce0 0x7f8278066190 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f82880097c0 tx=0x7f8288019000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:59:15.882 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:15.872+0000 7f827ffff700 1 -- 172.21.15.77:0/503650660 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f828c04f760 con 0x7f82980c89e0 2022-04-03T19:59:16.085 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:16.085+0000 7f829e54a700 1 -- 172.21.15.77:0/503650660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82980c2a00 con 0x7f82980c89e0 2022-04-03T19:59:16.086 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:16.085+0000 7f827ffff700 1 -- 172.21.15.77:0/503650660 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+6107 (secure 0 0 0) 0x7f828c020070 con 0x7f82980c89e0 2022-04-03T19:59:16.086 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:59:16.086 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5833,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/493106414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":493106414},{"type":"v1","addr":"172.21.15.77:6837","nonce":493106414}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:57:44.361632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5668":{"gid":5668,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":33,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:59:16.088 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:16.087+0000 7f829e54a700 1 -- 172.21.15.77:0/503650660 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8278063ce0 msgr2=0x7f8278066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:59:16.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:16.087+0000 7f829e54a700 1 --2- 172.21.15.77:0/503650660 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8278063ce0 0x7f8278066190 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f82880097c0 tx=0x7f8288019000 comp rx=0 tx=0).stop 2022-04-03T19:59:16.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:16.087+0000 7f829e54a700 1 -- 172.21.15.77:0/503650660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82980c89e0 msgr2=0x7f82981681b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:59:16.089 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:16.087+0000 7f829e54a700 1 --2- 172.21.15.77:0/503650660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82980c89e0 0x7f82981681b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f828c00ecb0 tx=0x7f828c006f40 comp rx=0 tx=0).stop 2022-04-03T19:59:16.090 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:16.087+0000 7f829e54a700 1 -- 172.21.15.77:0/503650660 shutdown_connections 2022-04-03T19:59:16.090 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:16.087+0000 7f829e54a700 1 --2- 172.21.15.77:0/503650660 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7f8278063ce0 0x7f8278066190 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:16.090 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:16.087+0000 7f829e54a700 1 --2- 172.21.15.77:0/503650660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82981552d0 0x7f82981686f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:16.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:16.087+0000 7f829e54a700 1 --2- 172.21.15.77:0/503650660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82980c3440 0x7f8298150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:16.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:16.087+0000 7f829e54a700 1 --2- 172.21.15.77:0/503650660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82980c89e0 0x7f82981681b0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:16.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:16.088+0000 7f829e54a700 1 -- 172.21.15.77:0/503650660 >> 172.21.15.77:0/503650660 conn(0x7f82980cc300 msgr2=0x7f82980c82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:59:16.091 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:16.088+0000 7f829e54a700 1 -- 172.21.15.77:0/503650660 shutdown_connections 2022-04-03T19:59:16.092 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:16.088+0000 7f829e54a700 1 -- 172.21.15.77:0/503650660 wait complete. 2022-04-03T19:59:16.092 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 49 2022-04-03T19:59:16.102 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:59:18.104 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:59:18.161 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:59:18.164 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:59:18.249 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.249+0000 7fb4992f0700 1 Processor -- start 2022-04-03T19:59:18.250 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.249+0000 7fb4992f0700 1 -- start start 2022-04-03T19:59:18.252 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.250+0000 7fb4992f0700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb49406bbe0 0x7fb4941584b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:59:18.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.250+0000 7fb4992f0700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb49406c500 0x7fb49414eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:59:18.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.250+0000 7fb4992f0700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49406d260 con 0x7fb49406bfb0 2022-04-03T19:59:18.253 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.250+0000 7fb4992f0700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb494159450 con 0x7fb49406c500 2022-04-03T19:59:18.254 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.250+0000 7fb4992f0700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4940c6330 con 0x7fb49406bbe0 2022-04-03T19:59:18.254 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.250+0000 7fb498aef700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb49406c500 0x7fb49414eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:59:18.254 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.250+0000 7fb4937fe700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb49406bfb0 0x7fb49406d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38600/0 (socket says 172.21.15.77:38600) 2022-04-03T19:59:18.254 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.250+0000 7fb4937fe700 1 -- 172.21.15.77:0/910333546 learned_addr learned my addr 172.21.15.77:0/910333546 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:59:18.255 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.250+0000 7fb493fff700 1 --2- 172.21.15.77:0/910333546 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb49406bbe0 0x7fb4941584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:59:18.255 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.251+0000 7fb498aef700 1 -- 172.21.15.77:0/910333546 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb49406bbe0 msgr2=0x7fb4941584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:59:18.255 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.251+0000 7fb498aef700 1 --2- 172.21.15.77:0/910333546 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb49406bbe0 0x7fb4941584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:18.255 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.251+0000 7fb498aef700 1 -- 172.21.15.77:0/910333546 >> v1:172.21.15.77:6789/0 conn(0x7fb49406bfb0 legacy=0x7fb49406d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:59:18.256 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.251+0000 7fb498aef700 1 -- 172.21.15.77:0/910333546 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb494158bb0 con 0x7fb49406c500 2022-04-03T19:59:18.256 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.251+0000 7fb498aef700 1 --2- 172.21.15.77:0/910333546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb49406c500 0x7fb49414eea0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb48c009760 tx=0x7fb48c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899cf6029944bcec server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:59:18.256 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.251+0000 7fb492ffd700 1 -- 172.21.15.77:0/910333546 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48c011630 con 0x7fb49406c500 2022-04-03T19:59:18.256 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.251+0000 7fb492ffd700 1 -- 172.21.15.77:0/910333546 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48c011790 con 0x7fb49406c500 2022-04-03T19:59:18.256 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.251+0000 7fb492ffd700 1 -- 172.21.15.77:0/910333546 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48c011b30 con 0x7fb49406c500 2022-04-03T19:59:18.257 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.251+0000 7fb4992f0700 1 -- 172.21.15.77:0/910333546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb49406c500 msgr2=0x7fb49414eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:59:18.257 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.251+0000 7fb4992f0700 1 --2- 172.21.15.77:0/910333546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb49406c500 0x7fb49414eea0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb48c009760 tx=0x7fb48c009060 comp rx=0 tx=0).stop 2022-04-03T19:59:18.257 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.251+0000 7fb4992f0700 1 -- 172.21.15.77:0/910333546 shutdown_connections 2022-04-03T19:59:18.257 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.251+0000 7fb4992f0700 1 --2- 172.21.15.77:0/910333546 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb49406c500 0x7fb49414eea0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:18.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.252+0000 7fb4992f0700 1 --2- 172.21.15.77:0/910333546 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb49406bbe0 0x7fb4941584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:18.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.252+0000 7fb4992f0700 1 -- 172.21.15.77:0/910333546 >> 172.21.15.77:0/910333546 conn(0x7fb4940cb6b0 msgr2=0x7fb4940cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:59:18.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.252+0000 7fb4992f0700 1 -- 172.21.15.77:0/910333546 shutdown_connections 2022-04-03T19:59:18.258 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.252+0000 7fb4992f0700 1 -- 172.21.15.77:0/910333546 wait complete. 2022-04-03T19:59:18.261 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.253+0000 7fb4992f0700 1 Processor -- start 2022-04-03T19:59:18.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.253+0000 7fb4992f0700 1 -- start start 2022-04-03T19:59:18.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.253+0000 7fb4992f0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb49406bbe0 0x7fb494151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:59:18.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb4992f0700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb49406bfb0 0x7fb4940c22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:59:18.262 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb4992f0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb49406c500 0x7fb4940c2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:59:18.263 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb4992f0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb494165580 con 0x7fb49406c500 2022-04-03T19:59:18.263 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb4992f0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb494164990 con 0x7fb49406bbe0 2022-04-03T19:59:18.263 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb4992f0700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49414d760 con 0x7fb49406bfb0 2022-04-03T19:59:18.263 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb498aef700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb49406c500 0x7fb4940c2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:59:18.264 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb498aef700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb49406c500 0x7fb4940c2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43290/0 (socket says 172.21.15.77:43290) 2022-04-03T19:59:18.264 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb498aef700 1 -- 172.21.15.77:0/4286801005 learned_addr learned my addr 172.21.15.77:0/4286801005 (peer_addr_for_me v2:172.21.15.77:0/0) 2022-04-03T19:59:18.264 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb4937fe700 1 --2- 172.21.15.77:0/4286801005 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb49406bfb0 0x7fb4940c22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:59:18.264 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb493fff700 1 --2- 172.21.15.77:0/4286801005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb49406bbe0 0x7fb494151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:59:18.264 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb498aef700 1 -- 172.21.15.77:0/4286801005 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb49406bfb0 msgr2=0x7fb4940c22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:59:18.265 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb498aef700 1 --2- 172.21.15.77:0/4286801005 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb49406bfb0 0x7fb4940c22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:18.265 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb498aef700 1 -- 172.21.15.77:0/4286801005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb49406bbe0 msgr2=0x7fb494151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:59:18.265 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb498aef700 1 --2- 172.21.15.77:0/4286801005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb49406bbe0 0x7fb494151bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:18.265 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb498aef700 1 -- 172.21.15.77:0/4286801005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48c0092a0 con 0x7fb49406c500 2022-04-03T19:59:18.266 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb498aef700 1 --2- 172.21.15.77:0/4286801005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb49406c500 0x7fb4940c2800 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb48c009eb0 tx=0x7fb48c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:59:18.266 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb47bfff700 1 -- 172.21.15.77:0/4286801005 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48c003a70 con 0x7fb49406c500 2022-04-03T19:59:18.266 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb47bfff700 1 -- 172.21.15.77:0/4286801005 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48c003bf0 con 0x7fb49406c500 2022-04-03T19:59:18.266 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb47bfff700 1 -- 172.21.15.77:0/4286801005 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48c01a450 con 0x7fb49406c500 2022-04-03T19:59:18.267 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb4992f0700 1 -- 172.21.15.77:0/4286801005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb494164c50 con 0x7fb49406c500 2022-04-03T19:59:18.267 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.254+0000 7fb4992f0700 1 -- 172.21.15.77:0/4286801005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb494165210 con 0x7fb49406c500 2022-04-03T19:59:18.267 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.255+0000 7fb4992f0700 1 -- 172.21.15.77:0/4286801005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb49404b050 con 0x7fb49406c500 2022-04-03T19:59:18.267 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.256+0000 7fb47bfff700 1 -- 172.21.15.77:0/4286801005 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 6) v1 ==== 81922+0+0 (secure 0 0 0) 0x7fb48c00f050 con 0x7fb49406c500 2022-04-03T19:59:18.267 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.257+0000 7fb47bfff700 1 --2- 172.21.15.77:0/4286801005 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb474063ce0 0x7fb474066190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:59:18.268 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.257+0000 7fb47bfff700 1 -- 172.21.15.77:0/4286801005 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6718+0+0 (secure 0 0 0) 0x7fb48c0824b0 con 0x7fb49406c500 2022-04-03T19:59:18.268 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.257+0000 7fb493fff700 1 --2- 172.21.15.77:0/4286801005 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb474063ce0 0x7fb474066190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:59:18.268 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.257+0000 7fb493fff700 1 --2- 172.21.15.77:0/4286801005 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb474063ce0 0x7fb474066190 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb4840043e0 tx=0x7fb4840038f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:59:18.268 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.261+0000 7fb47bfff700 1 -- 172.21.15.77:0/4286801005 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7fb48c050e00 con 0x7fb49406c500 2022-04-03T19:59:18.475 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.474+0000 7fb4992f0700 1 -- 172.21.15.77:0/4286801005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb494063d10 con 0x7fb49406c500 2022-04-03T19:59:18.476 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.475+0000 7fb47bfff700 1 -- 172.21.15.77:0/4286801005 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+6107 (secure 0 0 0) 0x7fb48c02a090 con 0x7fb49406c500 2022-04-03T19:59:18.476 INFO:teuthology.orchestra.run.smithi077.stdout: 2022-04-03T19:59:18.476 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6834/2116223509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":2116223509},{"type":"v1","addr":"172.21.15.77:6834","nonce":2116223509}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4259,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6836/2374276747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2374276747},{"type":"v1","addr":"172.21.15.186:6836","nonce":2374276747}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4265,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/610990853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":610990853},{"type":"v1","addr":"172.21.15.186:6839","nonce":610990853}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4271,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6835/170053412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6833","nonce":170053412},{"type":"v1","addr":"172.21.15.77:6835","nonce":170053412}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5833,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.77:6837/493106414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6836","nonce":493106414},{"type":"v1","addr":"172.21.15.77:6837","nonce":493106414}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-04-03T19:44:21.275716+0000","modified":"2022-04-03T19:57:44.361632+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5668":{"gid":5668,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":33,"addr":"172.21.15.186:6837/113043607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6835","nonce":113043607},{"type":"v1","addr":"172.21.15.186:6837","nonce":113043607}]},"join_fscid":-1,"export_targets":[],"features":4540138303579357183,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-04-03T19:59:18.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.477+0000 7fb4992f0700 1 -- 172.21.15.77:0/4286801005 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb474063ce0 msgr2=0x7fb474066190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:59:18.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.477+0000 7fb4992f0700 1 --2- 172.21.15.77:0/4286801005 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb474063ce0 0x7fb474066190 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb4840043e0 tx=0x7fb4840038f0 comp rx=0 tx=0).stop 2022-04-03T19:59:18.478 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.477+0000 7fb4992f0700 1 -- 172.21.15.77:0/4286801005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb49406c500 msgr2=0x7fb4940c2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:59:18.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.477+0000 7fb4992f0700 1 --2- 172.21.15.77:0/4286801005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb49406c500 0x7fb4940c2800 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb48c009eb0 tx=0x7fb48c017000 comp rx=0 tx=0).stop 2022-04-03T19:59:18.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.477+0000 7fb4992f0700 1 -- 172.21.15.77:0/4286801005 shutdown_connections 2022-04-03T19:59:18.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.477+0000 7fb4992f0700 1 --2- 172.21.15.77:0/4286801005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb49406c500 0x7fb4940c2800 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:18.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.477+0000 7fb4992f0700 1 --2- 172.21.15.77:0/4286801005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb49406bbe0 0x7fb494151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:18.479 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.477+0000 7fb4992f0700 1 --2- 172.21.15.77:0/4286801005 >> [v2:172.21.15.186:6832/24948,v1:172.21.15.186:6833/24948] conn(0x7fb474063ce0 0x7fb474066190 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:18.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.477+0000 7fb4992f0700 1 --2- 172.21.15.77:0/4286801005 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb49406bfb0 0x7fb4940c22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:18.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.477+0000 7fb4992f0700 1 -- 172.21.15.77:0/4286801005 >> 172.21.15.77:0/4286801005 conn(0x7fb4940cb6b0 msgr2=0x7fb494149880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-04-03T19:59:18.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.477+0000 7fb4992f0700 1 -- 172.21.15.77:0/4286801005 shutdown_connections 2022-04-03T19:59:18.480 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:18.477+0000 7fb4992f0700 1 -- 172.21.15.77:0/4286801005 wait complete. 2022-04-03T19:59:18.481 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 49 2022-04-03T19:59:18.492 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2022-04-03T19:59:20.493 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-04-03T19:59:20.637 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:20.636+0000 7f5769aa8700 1 Processor -- start 2022-04-03T19:59:20.637 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:20.637+0000 7f5769aa8700 1 -- start start 2022-04-03T19:59:20.639 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:20.637+0000 7f5769aa8700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f57640c3440 0x7f57640c3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:59:20.639 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:20.637+0000 7f5769aa8700 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57641552d0 0x7f576406c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-04-03T19:59:20.640 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:20.637+0000 7f5769aa8700 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576406bf30 con 0x7f57640c89e0 2022-04-03T19:59:20.640 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:20.637+0000 7f5769aa8700 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57640c4e70 con 0x7f57640c3440 2022-04-03T19:59:20.640 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:20.637+0000 7f5769aa8700 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5764156270 con 0x7f57641552d0 2022-04-03T19:59:20.640 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:20.638+0000 7f57637fe700 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f57640c3440 0x7f57640c3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:59:20.641 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:20.638+0000 7f5763fff700 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f57640c89e0 0x7f57640c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:38602/0 (socket says 172.21.15.77:38602) 2022-04-03T19:59:20.641 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:20.638+0000 7f5763fff700 1 -- 172.21.15.77:0/2055527513 learned_addr learned my addr 172.21.15.77:0/2055527513 (peer_addr_for_me v1:172.21.15.77:0/0) 2022-04-03T19:59:20.641 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:20.638+0000 7f575bfff700 1 --2- 172.21.15.77:0/2055527513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57641552d0 0x7f576406c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-04-03T19:59:20.641 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:20.638+0000 7f57637fe700 1 -- 172.21.15.77:0/2055527513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57641552d0 msgr2=0x7f576406c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-04-03T19:59:20.641 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:20.638+0000 7f57637fe700 1 --2- 172.21.15.77:0/2055527513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57641552d0 0x7f576406c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-04-03T19:59:20.642 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:20.638+0000 7f57637fe700 1 -- 172.21.15.77:0/2055527513 >> v1:172.21.15.77:6789/0 conn(0x7f57640c89e0 legacy=0x7f57640c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-04-03T19:59:20.642 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:20.638+0000 7f57637fe700 1 -- 172.21.15.77:0/2055527513 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5764155a10 con 0x7f57640c3440 2022-04-03T19:59:20.642 INFO:teuthology.orchestra.run.smithi077.stderr:2022-04-03T19:59:20.638+0000 7f57637fe700 1 --2- 172.21.15.77:0/2055527513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57640c3440 0x7f57640c3810 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5754009760 tx=0x7f5754009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b08a5399d0c6bacb server_cookie=0 in_seq=0 out_seq=0 2022-04-03T19:59:48.221 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T19:59:48.225 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:00:14.209 INFO:tasks.ceph.osd.0.smithi077.stderr:2022-04-03T20:00:14.208+0000 7f2861eb2700 -1 bdev(0x55d6a5829800 /var/lib/ceph/osd/ceph-0/block) _aio_thread stalled aio 0x55d6a5995810 since 2022-04-03T19:59:14.182194+0000, timeout is 60s, suicide 2022-04-03T20:00:14.210 INFO:tasks.ceph.osd.0.smithi077.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f2861eb2700 time 2022-04-03T20:00:14.208960+0000 2022-04-03T20:00:14.211 INFO:tasks.ceph.osd.0.smithi077.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: 666: ceph_abort_msg("stalled aio... buggy kernel or bad device?") 2022-04-03T20:00:14.211 INFO:tasks.ceph.osd.0.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:14.212 INFO:tasks.ceph.osd.0.smithi077.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55d6a23dd0eb] 2022-04-03T20:00:14.212 INFO:tasks.ceph.osd.0.smithi077.stderr: 2: (KernelDevice::_aio_thread()+0x1275) [0x55d6a302fd95] 2022-04-03T20:00:14.212 INFO:tasks.ceph.osd.0.smithi077.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55d6a303d2f1] 2022-04-03T20:00:14.213 INFO:tasks.ceph.osd.0.smithi077.stderr: 4: /lib64/libpthread.so.0(+0x814a) [0x7f2872b0514a] 2022-04-03T20:00:14.213 INFO:tasks.ceph.osd.0.smithi077.stderr: 5: clone() 2022-04-03T20:00:14.213 INFO:tasks.ceph.osd.0.smithi077.stderr:*** Caught signal (Aborted) ** 2022-04-03T20:00:14.214 INFO:tasks.ceph.osd.0.smithi077.stderr: in thread 7f2861eb2700 thread_name:bstore_aio 2022-04-03T20:00:14.214 INFO:tasks.ceph.osd.0.smithi077.stderr:2022-04-03T20:00:14.209+0000 7f2861eb2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f2861eb2700 time 2022-04-03T20:00:14.208960+0000 2022-04-03T20:00:14.214 INFO:tasks.ceph.osd.0.smithi077.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: 666: ceph_abort_msg("stalled aio... buggy kernel or bad device?") 2022-04-03T20:00:14.214 INFO:tasks.ceph.osd.0.smithi077.stderr: 2022-04-03T20:00:14.215 INFO:tasks.ceph.osd.0.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:14.215 INFO:tasks.ceph.osd.0.smithi077.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55d6a23dd0eb] 2022-04-03T20:00:14.215 INFO:tasks.ceph.osd.0.smithi077.stderr: 2: (KernelDevice::_aio_thread()+0x1275) [0x55d6a302fd95] 2022-04-03T20:00:14.216 INFO:tasks.ceph.osd.0.smithi077.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55d6a303d2f1] 2022-04-03T20:00:14.216 INFO:tasks.ceph.osd.0.smithi077.stderr: 4: /lib64/libpthread.so.0(+0x814a) [0x7f2872b0514a] 2022-04-03T20:00:14.216 INFO:tasks.ceph.osd.0.smithi077.stderr: 5: clone() 2022-04-03T20:00:14.217 INFO:tasks.ceph.osd.0.smithi077.stderr: 2022-04-03T20:00:14.218 INFO:tasks.ceph.osd.0.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:14.218 INFO:tasks.ceph.osd.0.smithi077.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f2872b0fb20] 2022-04-03T20:00:14.218 INFO:tasks.ceph.osd.0.smithi077.stderr: 2: gsignal() 2022-04-03T20:00:14.219 INFO:tasks.ceph.osd.0.smithi077.stderr: 3: abort() 2022-04-03T20:00:14.219 INFO:tasks.ceph.osd.0.smithi077.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x55d6a23dd1be] 2022-04-03T20:00:14.219 INFO:tasks.ceph.osd.0.smithi077.stderr: 5: (KernelDevice::_aio_thread()+0x1275) [0x55d6a302fd95] 2022-04-03T20:00:14.219 INFO:tasks.ceph.osd.0.smithi077.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55d6a303d2f1] 2022-04-03T20:00:14.220 INFO:tasks.ceph.osd.0.smithi077.stderr: 7: /lib64/libpthread.so.0(+0x814a) [0x7f2872b0514a] 2022-04-03T20:00:14.220 INFO:tasks.ceph.osd.0.smithi077.stderr: 8: clone() 2022-04-03T20:00:14.220 INFO:tasks.ceph.osd.0.smithi077.stderr:2022-04-03T20:00:14.211+0000 7f2861eb2700 -1 *** Caught signal (Aborted) ** 2022-04-03T20:00:14.220 INFO:tasks.ceph.osd.0.smithi077.stderr: in thread 7f2861eb2700 thread_name:bstore_aio 2022-04-03T20:00:14.221 INFO:tasks.ceph.osd.0.smithi077.stderr: 2022-04-03T20:00:14.221 INFO:tasks.ceph.osd.0.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:14.221 INFO:tasks.ceph.osd.0.smithi077.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f2872b0fb20] 2022-04-03T20:00:14.221 INFO:tasks.ceph.osd.0.smithi077.stderr: 2: gsignal() 2022-04-03T20:00:14.222 INFO:tasks.ceph.osd.0.smithi077.stderr: 3: abort() 2022-04-03T20:00:14.222 INFO:tasks.ceph.osd.0.smithi077.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x55d6a23dd1be] 2022-04-03T20:00:14.222 INFO:tasks.ceph.osd.0.smithi077.stderr: 5: (KernelDevice::_aio_thread()+0x1275) [0x55d6a302fd95] 2022-04-03T20:00:14.223 INFO:tasks.ceph.osd.0.smithi077.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55d6a303d2f1] 2022-04-03T20:00:14.223 INFO:tasks.ceph.osd.0.smithi077.stderr: 7: /lib64/libpthread.so.0(+0x814a) [0x7f2872b0514a] 2022-04-03T20:00:14.223 INFO:tasks.ceph.osd.0.smithi077.stderr: 8: clone() 2022-04-03T20:00:14.223 INFO:tasks.ceph.osd.0.smithi077.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-03T20:00:14.223 INFO:tasks.ceph.osd.0.smithi077.stderr: 2022-04-03T20:00:14.229 INFO:tasks.ceph.osd.1.smithi077.stderr:2022-04-03T20:00:14.228+0000 7f420d127700 -1 bdev(0x55f7c7aeb800 /var/lib/ceph/osd/ceph-1/block) _aio_thread stalled aio 0x55f7d0468210 since 2022-04-03T19:59:14.202310+0000, timeout is 60s, suicide 2022-04-03T20:00:14.231 INFO:tasks.ceph.osd.1.smithi077.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f420d127700 time 2022-04-03T20:00:14.229751+0000 2022-04-03T20:00:14.231 INFO:tasks.ceph.osd.1.smithi077.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: 666: ceph_abort_msg("stalled aio... buggy kernel or bad device?") 2022-04-03T20:00:14.231 INFO:tasks.ceph.osd.1.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:14.232 INFO:tasks.ceph.osd.1.smithi077.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55f7c33dd0eb] 2022-04-03T20:00:14.232 INFO:tasks.ceph.osd.1.smithi077.stderr: 2: (KernelDevice::_aio_thread()+0x1275) [0x55f7c402fd95] 2022-04-03T20:00:14.232 INFO:tasks.ceph.osd.1.smithi077.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55f7c403d2f1] 2022-04-03T20:00:14.232 INFO:tasks.ceph.osd.1.smithi077.stderr: 4: /lib64/libpthread.so.0(+0x814a) [0x7f421dd7a14a] 2022-04-03T20:00:14.233 INFO:tasks.ceph.osd.1.smithi077.stderr: 5: clone() 2022-04-03T20:00:14.233 INFO:tasks.ceph.osd.1.smithi077.stderr:2022-04-03T20:00:14.230+0000 7f420d127700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f420d127700 time 2022-04-03T20:00:14.229751+0000 2022-04-03T20:00:14.233 INFO:tasks.ceph.osd.1.smithi077.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: 666: ceph_abort_msg("stalled aio... buggy kernel or bad device?") 2022-04-03T20:00:14.234 INFO:tasks.ceph.osd.1.smithi077.stderr: 2022-04-03T20:00:14.234 INFO:tasks.ceph.osd.1.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:14.234 INFO:tasks.ceph.osd.1.smithi077.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55f7c33dd0eb] 2022-04-03T20:00:14.234 INFO:tasks.ceph.osd.1.smithi077.stderr: 2: (KernelDevice::_aio_thread()+0x1275) [0x55f7c402fd95] 2022-04-03T20:00:14.235 INFO:tasks.ceph.osd.1.smithi077.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55f7c403d2f1] 2022-04-03T20:00:14.235 INFO:tasks.ceph.osd.1.smithi077.stderr: 4: /lib64/libpthread.so.0(+0x814a) [0x7f421dd7a14a] 2022-04-03T20:00:14.235 INFO:tasks.ceph.osd.1.smithi077.stderr: 5: clone() 2022-04-03T20:00:14.235 INFO:tasks.ceph.osd.1.smithi077.stderr: 2022-04-03T20:00:14.236 INFO:tasks.ceph.osd.1.smithi077.stderr:*** Caught signal (Aborted) ** 2022-04-03T20:00:14.236 INFO:tasks.ceph.osd.1.smithi077.stderr: in thread 7f420d127700 thread_name:bstore_aio 2022-04-03T20:00:14.237 INFO:tasks.ceph.osd.0.smithi077.stderr: -2> 2022-04-03T20:00:14.208+0000 7f2861eb2700 -1 bdev(0x55d6a5829800 /var/lib/ceph/osd/ceph-0/block) _aio_thread stalled aio 0x55d6a5995810 since 2022-04-03T19:59:14.182194+0000, timeout is 60s, suicide 2022-04-03T20:00:14.237 INFO:tasks.ceph.osd.0.smithi077.stderr: -1> 2022-04-03T20:00:14.209+0000 7f2861eb2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f2861eb2700 time 2022-04-03T20:00:14.208960+0000 2022-04-03T20:00:14.237 INFO:tasks.ceph.osd.0.smithi077.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: 666: ceph_abort_msg("stalled aio... buggy kernel or bad device?") 2022-04-03T20:00:14.238 INFO:tasks.ceph.osd.0.smithi077.stderr: 2022-04-03T20:00:14.238 INFO:tasks.ceph.osd.0.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:14.238 INFO:tasks.ceph.osd.0.smithi077.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55d6a23dd0eb] 2022-04-03T20:00:14.239 INFO:tasks.ceph.osd.0.smithi077.stderr: 2: (KernelDevice::_aio_thread()+0x1275) [0x55d6a302fd95] 2022-04-03T20:00:14.239 INFO:tasks.ceph.osd.0.smithi077.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55d6a303d2f1] 2022-04-03T20:00:14.239 INFO:tasks.ceph.osd.0.smithi077.stderr: 4: /lib64/libpthread.so.0(+0x814a) [0x7f2872b0514a] 2022-04-03T20:00:14.240 INFO:tasks.ceph.osd.0.smithi077.stderr: 5: clone() 2022-04-03T20:00:14.240 INFO:tasks.ceph.osd.0.smithi077.stderr: 2022-04-03T20:00:14.240 INFO:tasks.ceph.osd.0.smithi077.stderr: 0> 2022-04-03T20:00:14.211+0000 7f2861eb2700 -1 *** Caught signal (Aborted) ** 2022-04-03T20:00:14.240 INFO:tasks.ceph.osd.0.smithi077.stderr: in thread 7f2861eb2700 thread_name:bstore_aio 2022-04-03T20:00:14.241 INFO:tasks.ceph.osd.0.smithi077.stderr: 2022-04-03T20:00:14.241 INFO:tasks.ceph.osd.0.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:14.241 INFO:tasks.ceph.osd.0.smithi077.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f2872b0fb20] 2022-04-03T20:00:14.241 INFO:tasks.ceph.osd.0.smithi077.stderr: 2: gsignal() 2022-04-03T20:00:14.241 INFO:tasks.ceph.osd.0.smithi077.stderr: 3: abort() 2022-04-03T20:00:14.242 INFO:tasks.ceph.osd.0.smithi077.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x55d6a23dd1be] 2022-04-03T20:00:14.242 INFO:tasks.ceph.osd.0.smithi077.stderr: 5: (KernelDevice::_aio_thread()+0x1275) [0x55d6a302fd95] 2022-04-03T20:00:14.242 INFO:tasks.ceph.osd.0.smithi077.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55d6a303d2f1] 2022-04-03T20:00:14.242 INFO:tasks.ceph.osd.0.smithi077.stderr: 7: /lib64/libpthread.so.0(+0x814a) [0x7f2872b0514a] 2022-04-03T20:00:14.243 INFO:tasks.ceph.osd.0.smithi077.stderr: 8: clone() 2022-04-03T20:00:14.243 INFO:tasks.ceph.osd.0.smithi077.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-03T20:00:14.243 INFO:tasks.ceph.osd.0.smithi077.stderr: 2022-04-03T20:00:14.244 INFO:tasks.ceph.osd.1.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:14.244 INFO:tasks.ceph.osd.1.smithi077.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f421dd84b20] 2022-04-03T20:00:14.244 INFO:tasks.ceph.osd.1.smithi077.stderr: 2: gsignal() 2022-04-03T20:00:14.245 INFO:tasks.ceph.osd.1.smithi077.stderr: 3: abort() 2022-04-03T20:00:14.245 INFO:tasks.ceph.osd.1.smithi077.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x55f7c33dd1be] 2022-04-03T20:00:14.245 INFO:tasks.ceph.osd.1.smithi077.stderr: 5: (KernelDevice::_aio_thread()+0x1275) [0x55f7c402fd95] 2022-04-03T20:00:14.245 INFO:tasks.ceph.osd.1.smithi077.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55f7c403d2f1] 2022-04-03T20:00:14.246 INFO:tasks.ceph.osd.1.smithi077.stderr: 7: /lib64/libpthread.so.0(+0x814a) [0x7f421dd7a14a] 2022-04-03T20:00:14.246 INFO:tasks.ceph.osd.1.smithi077.stderr: 8: clone() 2022-04-03T20:00:14.246 INFO:tasks.ceph.osd.1.smithi077.stderr:2022-04-03T20:00:14.232+0000 7f420d127700 -1 *** Caught signal (Aborted) ** 2022-04-03T20:00:14.246 INFO:tasks.ceph.osd.1.smithi077.stderr: in thread 7f420d127700 thread_name:bstore_aio 2022-04-03T20:00:14.247 INFO:tasks.ceph.osd.1.smithi077.stderr: 2022-04-03T20:00:14.247 INFO:tasks.ceph.osd.1.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:14.247 INFO:tasks.ceph.osd.1.smithi077.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f421dd84b20] 2022-04-03T20:00:14.247 INFO:tasks.ceph.osd.1.smithi077.stderr: 2: gsignal() 2022-04-03T20:00:14.248 INFO:tasks.ceph.osd.1.smithi077.stderr: 3: abort() 2022-04-03T20:00:14.248 INFO:tasks.ceph.osd.1.smithi077.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x55f7c33dd1be] 2022-04-03T20:00:14.248 INFO:tasks.ceph.osd.1.smithi077.stderr: 5: (KernelDevice::_aio_thread()+0x1275) [0x55f7c402fd95] 2022-04-03T20:00:14.249 INFO:tasks.ceph.osd.1.smithi077.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55f7c403d2f1] 2022-04-03T20:00:14.249 INFO:tasks.ceph.osd.1.smithi077.stderr: 7: /lib64/libpthread.so.0(+0x814a) [0x7f421dd7a14a] 2022-04-03T20:00:14.249 INFO:tasks.ceph.osd.1.smithi077.stderr: 8: clone() 2022-04-03T20:00:14.250 INFO:tasks.ceph.osd.1.smithi077.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-03T20:00:14.250 INFO:tasks.ceph.osd.1.smithi077.stderr: 2022-04-03T20:00:14.250 INFO:tasks.ceph.osd.0.smithi077.stderr: -3> 2022-04-03T20:00:14.208+0000 7f2861eb2700 -1 bdev(0x55d6a5829800 /var/lib/ceph/osd/ceph-0/block) _aio_thread stalled aio 0x55d6a5995810 since 2022-04-03T19:59:14.182194+0000, timeout is 60s, suicide 2022-04-03T20:00:14.251 INFO:tasks.ceph.osd.0.smithi077.stderr: -2> 2022-04-03T20:00:14.209+0000 7f2861eb2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f2861eb2700 time 2022-04-03T20:00:14.208960+0000 2022-04-03T20:00:14.251 INFO:tasks.ceph.osd.0.smithi077.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: 666: ceph_abort_msg("stalled aio... buggy kernel or bad device?") 2022-04-03T20:00:14.251 INFO:tasks.ceph.osd.0.smithi077.stderr: 2022-04-03T20:00:14.252 INFO:tasks.ceph.osd.0.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:14.252 INFO:tasks.ceph.osd.0.smithi077.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55d6a23dd0eb] 2022-04-03T20:00:14.252 INFO:tasks.ceph.osd.0.smithi077.stderr: 2: (KernelDevice::_aio_thread()+0x1275) [0x55d6a302fd95] 2022-04-03T20:00:14.252 INFO:tasks.ceph.osd.0.smithi077.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55d6a303d2f1] 2022-04-03T20:00:14.252 INFO:tasks.ceph.osd.0.smithi077.stderr: 4: /lib64/libpthread.so.0(+0x814a) [0x7f2872b0514a] 2022-04-03T20:00:14.253 INFO:tasks.ceph.osd.0.smithi077.stderr: 5: clone() 2022-04-03T20:00:14.253 INFO:tasks.ceph.osd.0.smithi077.stderr: 2022-04-03T20:00:14.253 INFO:tasks.ceph.osd.0.smithi077.stderr: -1> 2022-04-03T20:00:14.211+0000 7f2861eb2700 -1 *** Caught signal (Aborted) ** 2022-04-03T20:00:14.253 INFO:tasks.ceph.osd.0.smithi077.stderr: in thread 7f2861eb2700 thread_name:bstore_aio 2022-04-03T20:00:14.254 INFO:tasks.ceph.osd.0.smithi077.stderr: 2022-04-03T20:00:14.254 INFO:tasks.ceph.osd.0.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:14.254 INFO:tasks.ceph.osd.0.smithi077.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f2872b0fb20] 2022-04-03T20:00:14.254 INFO:tasks.ceph.osd.0.smithi077.stderr: 2: gsignal() 2022-04-03T20:00:14.255 INFO:tasks.ceph.osd.0.smithi077.stderr: 3: abort() 2022-04-03T20:00:14.255 INFO:tasks.ceph.osd.0.smithi077.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x55d6a23dd1be] 2022-04-03T20:00:14.255 INFO:tasks.ceph.osd.0.smithi077.stderr: 5: (KernelDevice::_aio_thread()+0x1275) [0x55d6a302fd95] 2022-04-03T20:00:14.255 INFO:tasks.ceph.osd.0.smithi077.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55d6a303d2f1] 2022-04-03T20:00:14.256 INFO:tasks.ceph.osd.0.smithi077.stderr: 7: /lib64/libpthread.so.0(+0x814a) [0x7f2872b0514a] 2022-04-03T20:00:14.256 INFO:tasks.ceph.osd.0.smithi077.stderr: 8: clone() 2022-04-03T20:00:14.256 INFO:tasks.ceph.osd.0.smithi077.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-03T20:00:14.256 INFO:tasks.ceph.osd.0.smithi077.stderr: 2022-04-03T20:00:14.257 INFO:tasks.ceph.osd.1.smithi077.stderr: -2> 2022-04-03T20:00:14.228+0000 7f420d127700 -1 bdev(0x55f7c7aeb800 /var/lib/ceph/osd/ceph-1/block) _aio_thread stalled aio 0x55f7d0468210 since 2022-04-03T19:59:14.202310+0000, timeout is 60s, suicide 2022-04-03T20:00:14.257 INFO:tasks.ceph.osd.1.smithi077.stderr: -1> 2022-04-03T20:00:14.230+0000 7f420d127700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f420d127700 time 2022-04-03T20:00:14.229751+0000 2022-04-03T20:00:14.258 INFO:tasks.ceph.osd.1.smithi077.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: 666: ceph_abort_msg("stalled aio... buggy kernel or bad device?") 2022-04-03T20:00:14.258 INFO:tasks.ceph.osd.1.smithi077.stderr: 2022-04-03T20:00:14.258 INFO:tasks.ceph.osd.1.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:14.259 INFO:tasks.ceph.osd.1.smithi077.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55f7c33dd0eb] 2022-04-03T20:00:14.259 INFO:tasks.ceph.osd.1.smithi077.stderr: 2: (KernelDevice::_aio_thread()+0x1275) [0x55f7c402fd95] 2022-04-03T20:00:14.259 INFO:tasks.ceph.osd.1.smithi077.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55f7c403d2f1] 2022-04-03T20:00:14.260 INFO:tasks.ceph.osd.1.smithi077.stderr: 4: /lib64/libpthread.so.0(+0x814a) [0x7f421dd7a14a] 2022-04-03T20:00:14.260 INFO:tasks.ceph.osd.1.smithi077.stderr: 5: clone() 2022-04-03T20:00:14.260 INFO:tasks.ceph.osd.1.smithi077.stderr: 2022-04-03T20:00:14.260 INFO:tasks.ceph.osd.1.smithi077.stderr: 0> 2022-04-03T20:00:14.232+0000 7f420d127700 -1 *** Caught signal (Aborted) ** 2022-04-03T20:00:14.261 INFO:tasks.ceph.osd.1.smithi077.stderr: in thread 7f420d127700 thread_name:bstore_aio 2022-04-03T20:00:14.261 INFO:tasks.ceph.osd.1.smithi077.stderr: 2022-04-03T20:00:14.261 INFO:tasks.ceph.osd.1.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:14.261 INFO:tasks.ceph.osd.1.smithi077.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f421dd84b20] 2022-04-03T20:00:14.262 INFO:tasks.ceph.osd.1.smithi077.stderr: 2: gsignal() 2022-04-03T20:00:14.262 INFO:tasks.ceph.osd.1.smithi077.stderr: 3: abort() 2022-04-03T20:00:14.262 INFO:tasks.ceph.osd.1.smithi077.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x55f7c33dd1be] 2022-04-03T20:00:14.262 INFO:tasks.ceph.osd.1.smithi077.stderr: 5: (KernelDevice::_aio_thread()+0x1275) [0x55f7c402fd95] 2022-04-03T20:00:14.263 INFO:tasks.ceph.osd.1.smithi077.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55f7c403d2f1] 2022-04-03T20:00:14.263 INFO:tasks.ceph.osd.1.smithi077.stderr: 7: /lib64/libpthread.so.0(+0x814a) [0x7f421dd7a14a] 2022-04-03T20:00:14.263 INFO:tasks.ceph.osd.1.smithi077.stderr: 8: clone() 2022-04-03T20:00:14.263 INFO:tasks.ceph.osd.1.smithi077.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-03T20:00:14.264 INFO:tasks.ceph.osd.1.smithi077.stderr: 2022-04-03T20:00:14.265 INFO:tasks.ceph.osd.1.smithi077.stderr: -7> 2022-04-03T20:00:14.228+0000 7f420d127700 -1 bdev(0x55f7c7aeb800 /var/lib/ceph/osd/ceph-1/block) _aio_thread stalled aio 0x55f7d0468210 since 2022-04-03T19:59:14.202310+0000, timeout is 60s, suicide 2022-04-03T20:00:14.265 INFO:tasks.ceph.osd.1.smithi077.stderr: -6> 2022-04-03T20:00:14.230+0000 7f420d127700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f420d127700 time 2022-04-03T20:00:14.229751+0000 2022-04-03T20:00:14.266 INFO:tasks.ceph.osd.1.smithi077.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: 666: ceph_abort_msg("stalled aio... buggy kernel or bad device?") 2022-04-03T20:00:14.266 INFO:tasks.ceph.osd.1.smithi077.stderr: 2022-04-03T20:00:14.266 INFO:tasks.ceph.osd.1.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:14.266 INFO:tasks.ceph.osd.1.smithi077.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55f7c33dd0eb] 2022-04-03T20:00:14.267 INFO:tasks.ceph.osd.1.smithi077.stderr: 2: (KernelDevice::_aio_thread()+0x1275) [0x55f7c402fd95] 2022-04-03T20:00:14.267 INFO:tasks.ceph.osd.1.smithi077.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55f7c403d2f1] 2022-04-03T20:00:14.267 INFO:tasks.ceph.osd.1.smithi077.stderr: 4: /lib64/libpthread.so.0(+0x814a) [0x7f421dd7a14a] 2022-04-03T20:00:14.267 INFO:tasks.ceph.osd.1.smithi077.stderr: 5: clone() 2022-04-03T20:00:14.268 INFO:tasks.ceph.osd.1.smithi077.stderr: 2022-04-03T20:00:14.268 INFO:tasks.ceph.osd.1.smithi077.stderr: -5> 2022-04-03T20:00:14.232+0000 7f420d127700 -1 *** Caught signal (Aborted) ** 2022-04-03T20:00:14.268 INFO:tasks.ceph.osd.1.smithi077.stderr: in thread 7f420d127700 thread_name:bstore_aio 2022-04-03T20:00:14.268 INFO:tasks.ceph.osd.1.smithi077.stderr: 2022-04-03T20:00:14.269 INFO:tasks.ceph.osd.1.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:14.269 INFO:tasks.ceph.osd.1.smithi077.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f421dd84b20] 2022-04-03T20:00:14.270 INFO:tasks.ceph.osd.1.smithi077.stderr: 2: gsignal() 2022-04-03T20:00:14.270 INFO:tasks.ceph.osd.1.smithi077.stderr: 3: abort() 2022-04-03T20:00:14.270 INFO:tasks.ceph.osd.1.smithi077.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x55f7c33dd1be] 2022-04-03T20:00:14.270 INFO:tasks.ceph.osd.1.smithi077.stderr: 5: (KernelDevice::_aio_thread()+0x1275) [0x55f7c402fd95] 2022-04-03T20:00:14.271 INFO:tasks.ceph.osd.1.smithi077.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55f7c403d2f1] 2022-04-03T20:00:14.271 INFO:tasks.ceph.osd.1.smithi077.stderr: 7: /lib64/libpthread.so.0(+0x814a) [0x7f421dd7a14a] 2022-04-03T20:00:14.271 INFO:tasks.ceph.osd.1.smithi077.stderr: 8: clone() 2022-04-03T20:00:14.271 INFO:tasks.ceph.osd.1.smithi077.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-03T20:00:14.272 INFO:tasks.ceph.osd.1.smithi077.stderr: 2022-04-03T20:00:18.472 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:00:18.477 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:00:20.626 INFO:tasks.ceph.osd.3.smithi077.stderr:2022-04-03T20:00:20.626+0000 7fc4bd62a700 -1 bdev(0x564f8f250000 /var/lib/ceph/osd/ceph-3/block) _aio_thread stalled aio 0x564f9e6fdc10 since 2022-04-03T19:59:20.599879+0000, timeout is 60s, suicide 2022-04-03T20:00:20.628 INFO:tasks.ceph.osd.3.smithi077.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc4bd62a700 time 2022-04-03T20:00:20.626746+0000 2022-04-03T20:00:20.629 INFO:tasks.ceph.osd.3.smithi077.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: 666: ceph_abort_msg("stalled aio... buggy kernel or bad device?") 2022-04-03T20:00:20.629 INFO:tasks.ceph.osd.3.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:20.630 INFO:tasks.ceph.osd.3.smithi077.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x564f8a7dd0eb] 2022-04-03T20:00:20.630 INFO:tasks.ceph.osd.3.smithi077.stderr: 2: (KernelDevice::_aio_thread()+0x1275) [0x564f8b42fd95] 2022-04-03T20:00:20.631 INFO:tasks.ceph.osd.3.smithi077.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x564f8b43d2f1] 2022-04-03T20:00:20.631 INFO:tasks.ceph.osd.3.smithi077.stderr: 4: /lib64/libpthread.so.0(+0x814a) [0x7fc4cd27b14a] 2022-04-03T20:00:20.632 INFO:tasks.ceph.osd.3.smithi077.stderr: 5: clone() 2022-04-03T20:00:20.632 INFO:tasks.ceph.osd.3.smithi077.stderr:*** Caught signal (Aborted) ** 2022-04-03T20:00:20.633 INFO:tasks.ceph.osd.3.smithi077.stderr: in thread 7fc4bd62a700 thread_name:bstore_aio 2022-04-03T20:00:20.633 INFO:tasks.ceph.osd.3.smithi077.stderr:2022-04-03T20:00:20.627+0000 7fc4bd62a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc4bd62a700 time 2022-04-03T20:00:20.626746+0000 2022-04-03T20:00:20.634 INFO:tasks.ceph.osd.3.smithi077.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: 666: ceph_abort_msg("stalled aio... buggy kernel or bad device?") 2022-04-03T20:00:20.634 INFO:tasks.ceph.osd.3.smithi077.stderr: 2022-04-03T20:00:20.635 INFO:tasks.ceph.osd.3.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:20.635 INFO:tasks.ceph.osd.3.smithi077.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x564f8a7dd0eb] 2022-04-03T20:00:20.636 INFO:tasks.ceph.osd.3.smithi077.stderr: 2: (KernelDevice::_aio_thread()+0x1275) [0x564f8b42fd95] 2022-04-03T20:00:20.636 INFO:tasks.ceph.osd.3.smithi077.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x564f8b43d2f1] 2022-04-03T20:00:20.637 INFO:tasks.ceph.osd.3.smithi077.stderr: 4: /lib64/libpthread.so.0(+0x814a) [0x7fc4cd27b14a] 2022-04-03T20:00:20.637 INFO:tasks.ceph.osd.3.smithi077.stderr: 5: clone() 2022-04-03T20:00:20.638 INFO:tasks.ceph.osd.3.smithi077.stderr: 2022-04-03T20:00:20.638 INFO:tasks.ceph.osd.3.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:20.639 INFO:tasks.ceph.osd.3.smithi077.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc4cd285b20] 2022-04-03T20:00:20.639 INFO:tasks.ceph.osd.3.smithi077.stderr: 2: gsignal() 2022-04-03T20:00:20.639 INFO:tasks.ceph.osd.3.smithi077.stderr: 3: abort() 2022-04-03T20:00:20.639 INFO:tasks.ceph.osd.3.smithi077.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x564f8a7dd1be] 2022-04-03T20:00:20.640 INFO:tasks.ceph.osd.3.smithi077.stderr: 5: (KernelDevice::_aio_thread()+0x1275) [0x564f8b42fd95] 2022-04-03T20:00:20.640 INFO:tasks.ceph.osd.3.smithi077.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x564f8b43d2f1] 2022-04-03T20:00:20.640 INFO:tasks.ceph.osd.3.smithi077.stderr: 7: /lib64/libpthread.so.0(+0x814a) [0x7fc4cd27b14a] 2022-04-03T20:00:20.640 INFO:tasks.ceph.osd.3.smithi077.stderr: 8: clone() 2022-04-03T20:00:20.641 INFO:tasks.ceph.osd.3.smithi077.stderr:2022-04-03T20:00:20.629+0000 7fc4bd62a700 -1 *** Caught signal (Aborted) ** 2022-04-03T20:00:20.641 INFO:tasks.ceph.osd.3.smithi077.stderr: in thread 7fc4bd62a700 thread_name:bstore_aio 2022-04-03T20:00:20.641 INFO:tasks.ceph.osd.3.smithi077.stderr: 2022-04-03T20:00:20.641 INFO:tasks.ceph.osd.3.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:20.641 INFO:tasks.ceph.osd.3.smithi077.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc4cd285b20] 2022-04-03T20:00:20.642 INFO:tasks.ceph.osd.3.smithi077.stderr: 2: gsignal() 2022-04-03T20:00:20.642 INFO:tasks.ceph.osd.3.smithi077.stderr: 3: abort() 2022-04-03T20:00:20.642 INFO:tasks.ceph.osd.3.smithi077.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x564f8a7dd1be] 2022-04-03T20:00:20.642 INFO:tasks.ceph.osd.3.smithi077.stderr: 5: (KernelDevice::_aio_thread()+0x1275) [0x564f8b42fd95] 2022-04-03T20:00:20.643 INFO:tasks.ceph.osd.3.smithi077.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x564f8b43d2f1] 2022-04-03T20:00:20.643 INFO:tasks.ceph.osd.3.smithi077.stderr: 7: /lib64/libpthread.so.0(+0x814a) [0x7fc4cd27b14a] 2022-04-03T20:00:20.643 INFO:tasks.ceph.osd.3.smithi077.stderr: 8: clone() 2022-04-03T20:00:20.643 INFO:tasks.ceph.osd.3.smithi077.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-03T20:00:20.643 INFO:tasks.ceph.osd.3.smithi077.stderr: 2022-04-03T20:00:20.648 INFO:tasks.ceph.osd.3.smithi077.stderr: -2> 2022-04-03T20:00:20.626+0000 7fc4bd62a700 -1 bdev(0x564f8f250000 /var/lib/ceph/osd/ceph-3/block) _aio_thread stalled aio 0x564f9e6fdc10 since 2022-04-03T19:59:20.599879+0000, timeout is 60s, suicide 2022-04-03T20:00:20.649 INFO:tasks.ceph.osd.3.smithi077.stderr: -1> 2022-04-03T20:00:20.627+0000 7fc4bd62a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc4bd62a700 time 2022-04-03T20:00:20.626746+0000 2022-04-03T20:00:20.649 INFO:tasks.ceph.osd.3.smithi077.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: 666: ceph_abort_msg("stalled aio... buggy kernel or bad device?") 2022-04-03T20:00:20.649 INFO:tasks.ceph.osd.3.smithi077.stderr: 2022-04-03T20:00:20.650 INFO:tasks.ceph.osd.3.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:20.650 INFO:tasks.ceph.osd.3.smithi077.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x564f8a7dd0eb] 2022-04-03T20:00:20.650 INFO:tasks.ceph.osd.3.smithi077.stderr: 2: (KernelDevice::_aio_thread()+0x1275) [0x564f8b42fd95] 2022-04-03T20:00:20.650 INFO:tasks.ceph.osd.3.smithi077.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x564f8b43d2f1] 2022-04-03T20:00:20.651 INFO:tasks.ceph.osd.3.smithi077.stderr: 4: /lib64/libpthread.so.0(+0x814a) [0x7fc4cd27b14a] 2022-04-03T20:00:20.651 INFO:tasks.ceph.osd.3.smithi077.stderr: 5: clone() 2022-04-03T20:00:20.651 INFO:tasks.ceph.osd.3.smithi077.stderr: 2022-04-03T20:00:20.651 INFO:tasks.ceph.osd.3.smithi077.stderr: 0> 2022-04-03T20:00:20.629+0000 7fc4bd62a700 -1 *** Caught signal (Aborted) ** 2022-04-03T20:00:20.651 INFO:tasks.ceph.osd.3.smithi077.stderr: in thread 7fc4bd62a700 thread_name:bstore_aio 2022-04-03T20:00:20.652 INFO:tasks.ceph.osd.3.smithi077.stderr: 2022-04-03T20:00:20.652 INFO:tasks.ceph.osd.3.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:20.652 INFO:tasks.ceph.osd.3.smithi077.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc4cd285b20] 2022-04-03T20:00:20.652 INFO:tasks.ceph.osd.3.smithi077.stderr: 2: gsignal() 2022-04-03T20:00:20.653 INFO:tasks.ceph.osd.3.smithi077.stderr: 3: abort() 2022-04-03T20:00:20.653 INFO:tasks.ceph.osd.3.smithi077.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x564f8a7dd1be] 2022-04-03T20:00:20.653 INFO:tasks.ceph.osd.3.smithi077.stderr: 5: (KernelDevice::_aio_thread()+0x1275) [0x564f8b42fd95] 2022-04-03T20:00:20.653 INFO:tasks.ceph.osd.3.smithi077.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x564f8b43d2f1] 2022-04-03T20:00:20.653 INFO:tasks.ceph.osd.3.smithi077.stderr: 7: /lib64/libpthread.so.0(+0x814a) [0x7fc4cd27b14a] 2022-04-03T20:00:20.654 INFO:tasks.ceph.osd.3.smithi077.stderr: 8: clone() 2022-04-03T20:00:20.654 INFO:tasks.ceph.osd.3.smithi077.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-03T20:00:20.654 INFO:tasks.ceph.osd.3.smithi077.stderr: 2022-04-03T20:00:20.661 INFO:tasks.ceph.osd.3.smithi077.stderr: -3> 2022-04-03T20:00:20.626+0000 7fc4bd62a700 -1 bdev(0x564f8f250000 /var/lib/ceph/osd/ceph-3/block) _aio_thread stalled aio 0x564f9e6fdc10 since 2022-04-03T19:59:20.599879+0000, timeout is 60s, suicide 2022-04-03T20:00:20.662 INFO:tasks.ceph.osd.3.smithi077.stderr: -2> 2022-04-03T20:00:20.627+0000 7fc4bd62a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc4bd62a700 time 2022-04-03T20:00:20.626746+0000 2022-04-03T20:00:20.662 INFO:tasks.ceph.osd.3.smithi077.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: 666: ceph_abort_msg("stalled aio... buggy kernel or bad device?") 2022-04-03T20:00:20.662 INFO:tasks.ceph.osd.3.smithi077.stderr: 2022-04-03T20:00:20.663 INFO:tasks.ceph.osd.3.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:20.663 INFO:tasks.ceph.osd.3.smithi077.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x564f8a7dd0eb] 2022-04-03T20:00:20.663 INFO:tasks.ceph.osd.3.smithi077.stderr: 2: (KernelDevice::_aio_thread()+0x1275) [0x564f8b42fd95] 2022-04-03T20:00:20.663 INFO:tasks.ceph.osd.3.smithi077.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x564f8b43d2f1] 2022-04-03T20:00:20.664 INFO:tasks.ceph.osd.3.smithi077.stderr: 4: /lib64/libpthread.so.0(+0x814a) [0x7fc4cd27b14a] 2022-04-03T20:00:20.664 INFO:tasks.ceph.osd.3.smithi077.stderr: 5: clone() 2022-04-03T20:00:20.664 INFO:tasks.ceph.osd.3.smithi077.stderr: 2022-04-03T20:00:20.664 INFO:tasks.ceph.osd.3.smithi077.stderr: -1> 2022-04-03T20:00:20.629+0000 7fc4bd62a700 -1 *** Caught signal (Aborted) ** 2022-04-03T20:00:20.665 INFO:tasks.ceph.osd.3.smithi077.stderr: in thread 7fc4bd62a700 thread_name:bstore_aio 2022-04-03T20:00:20.665 INFO:tasks.ceph.osd.3.smithi077.stderr: 2022-04-03T20:00:20.665 INFO:tasks.ceph.osd.3.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:00:20.665 INFO:tasks.ceph.osd.3.smithi077.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc4cd285b20] 2022-04-03T20:00:20.665 INFO:tasks.ceph.osd.3.smithi077.stderr: 2: gsignal() 2022-04-03T20:00:20.666 INFO:tasks.ceph.osd.3.smithi077.stderr: 3: abort() 2022-04-03T20:00:20.666 INFO:tasks.ceph.osd.3.smithi077.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x564f8a7dd1be] 2022-04-03T20:00:20.666 INFO:tasks.ceph.osd.3.smithi077.stderr: 5: (KernelDevice::_aio_thread()+0x1275) [0x564f8b42fd95] 2022-04-03T20:00:20.666 INFO:tasks.ceph.osd.3.smithi077.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x564f8b43d2f1] 2022-04-03T20:00:20.667 INFO:tasks.ceph.osd.3.smithi077.stderr: 7: /lib64/libpthread.so.0(+0x814a) [0x7fc4cd27b14a] 2022-04-03T20:00:20.667 INFO:tasks.ceph.osd.3.smithi077.stderr: 8: clone() 2022-04-03T20:00:20.667 INFO:tasks.ceph.osd.3.smithi077.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-03T20:00:20.667 INFO:tasks.ceph.osd.3.smithi077.stderr: 2022-04-03T20:00:34.789 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:34.788+0000 7f88058fb700 -1 osd.2 53 heartbeat_check: no reply from 172.21.15.77:6828 osd.0 since back 2022-04-03T20:00:13.179511+0000 front 2022-04-03T20:00:13.179542+0000 (oldest deadline 2022-04-03T20:00:34.279548+0000) 2022-04-03T20:00:34.790 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:34.788+0000 7f88058fb700 -1 osd.2 53 heartbeat_check: no reply from 172.21.15.77:6812 osd.1 since back 2022-04-03T20:00:13.179489+0000 front 2022-04-03T20:00:13.179475+0000 (oldest deadline 2022-04-03T20:00:34.279548+0000) 2022-04-03T20:00:35.029 INFO:tasks.ceph.osd.5.smithi186.stderr:2022-04-03T20:00:35.029+0000 7f771fd0c700 -1 osd.5 53 heartbeat_check: no reply from 172.21.15.77:6828 osd.0 since back 2022-04-03T20:00:12.341965+0000 front 2022-04-03T20:00:12.342043+0000 (oldest deadline 2022-04-03T20:00:34.641931+0000) 2022-04-03T20:00:35.030 INFO:tasks.ceph.osd.5.smithi186.stderr:2022-04-03T20:00:35.029+0000 7f771fd0c700 -1 osd.5 53 heartbeat_check: no reply from 172.21.15.77:6812 osd.1 since back 2022-04-03T20:00:12.342004+0000 front 2022-04-03T20:00:12.342090+0000 (oldest deadline 2022-04-03T20:00:34.641931+0000) 2022-04-03T20:00:35.778 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:35.777+0000 7f88058fb700 -1 osd.2 53 heartbeat_check: no reply from 172.21.15.77:6828 osd.0 since back 2022-04-03T20:00:13.179511+0000 front 2022-04-03T20:00:13.179542+0000 (oldest deadline 2022-04-03T20:00:34.279548+0000) 2022-04-03T20:00:35.778 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:35.777+0000 7f88058fb700 -1 osd.2 53 heartbeat_check: no reply from 172.21.15.77:6812 osd.1 since back 2022-04-03T20:00:13.179489+0000 front 2022-04-03T20:00:13.179475+0000 (oldest deadline 2022-04-03T20:00:34.279548+0000) 2022-04-03T20:00:36.771 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:36.770+0000 7f88058fb700 -1 osd.2 53 heartbeat_check: no reply from 172.21.15.77:6828 osd.0 since back 2022-04-03T20:00:13.179511+0000 front 2022-04-03T20:00:13.179542+0000 (oldest deadline 2022-04-03T20:00:34.279548+0000) 2022-04-03T20:00:36.772 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:36.771+0000 7f88058fb700 -1 osd.2 53 heartbeat_check: no reply from 172.21.15.77:6812 osd.1 since back 2022-04-03T20:00:13.179489+0000 front 2022-04-03T20:00:13.179475+0000 (oldest deadline 2022-04-03T20:00:34.279548+0000) 2022-04-03T20:00:40.905 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:40.905+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:00:41.955 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:41.954+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:00:42.039 INFO:tasks.ceph.osd.5.smithi186.stderr:2022-04-03T20:00:42.039+0000 7f771fd0c700 -1 osd.5 56 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:20.543027+0000 front 2022-04-03T20:00:20.543038+0000 (oldest deadline 2022-04-03T20:00:41.043049+0000) 2022-04-03T20:00:42.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:00:42.263+0000 7f88d0ac1700 -1 osd.4 56 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:19.781160+0000 front 2022-04-03T20:00:19.781210+0000 (oldest deadline 2022-04-03T20:00:42.081038+0000) 2022-04-03T20:00:43.004 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:43.003+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:00:43.036 INFO:tasks.ceph.osd.5.smithi186.stderr:2022-04-03T20:00:43.036+0000 7f771fd0c700 -1 osd.5 56 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:20.543027+0000 front 2022-04-03T20:00:20.543038+0000 (oldest deadline 2022-04-03T20:00:41.043049+0000) 2022-04-03T20:00:43.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:00:43.295+0000 7f88d0ac1700 -1 osd.4 56 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:19.781160+0000 front 2022-04-03T20:00:19.781210+0000 (oldest deadline 2022-04-03T20:00:42.081038+0000) 2022-04-03T20:00:43.987 INFO:tasks.ceph.osd.5.smithi186.stderr:2022-04-03T20:00:43.987+0000 7f771fd0c700 -1 osd.5 56 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:20.543027+0000 front 2022-04-03T20:00:20.543038+0000 (oldest deadline 2022-04-03T20:00:41.043049+0000) 2022-04-03T20:00:43.999 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:43.999+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:00:44.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:00:44.320+0000 7f88d0ac1700 -1 osd.4 57 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:19.781160+0000 front 2022-04-03T20:00:19.781210+0000 (oldest deadline 2022-04-03T20:00:42.081038+0000) 2022-04-03T20:00:44.845 INFO:tasks.ceph.osd.7.smithi186.stderr:2022-04-03T20:00:44.845+0000 7fc1bc457700 -1 osd.7 57 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:20.401702+0000 front 2022-04-03T20:00:20.401679+0000 (oldest deadline 2022-04-03T20:00:43.901605+0000) 2022-04-03T20:00:44.965 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:44.965+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:00:44.966 INFO:tasks.ceph.osd.5.smithi186.stderr:2022-04-03T20:00:44.967+0000 7f771fd0c700 -1 osd.5 56 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:20.543027+0000 front 2022-04-03T20:00:20.543038+0000 (oldest deadline 2022-04-03T20:00:41.043049+0000) 2022-04-03T20:00:45.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:00:45.338+0000 7f88d0ac1700 -1 osd.4 57 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:19.781160+0000 front 2022-04-03T20:00:19.781210+0000 (oldest deadline 2022-04-03T20:00:42.081038+0000) 2022-04-03T20:00:45.890 INFO:tasks.ceph.osd.7.smithi186.stderr:2022-04-03T20:00:45.890+0000 7fc1bc457700 -1 osd.7 57 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:20.401702+0000 front 2022-04-03T20:00:20.401679+0000 (oldest deadline 2022-04-03T20:00:43.901605+0000) 2022-04-03T20:00:45.989 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:45.989+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:00:46.005 INFO:tasks.ceph.osd.5.smithi186.stderr:2022-04-03T20:00:46.006+0000 7f771fd0c700 -1 osd.5 57 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:20.543027+0000 front 2022-04-03T20:00:20.543038+0000 (oldest deadline 2022-04-03T20:00:41.043049+0000) 2022-04-03T20:00:46.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:00:46.374+0000 7f88d0ac1700 -1 osd.4 57 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:19.781160+0000 front 2022-04-03T20:00:19.781210+0000 (oldest deadline 2022-04-03T20:00:42.081038+0000) 2022-04-03T20:00:46.985 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:46.985+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:00:48.004 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:48.004+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:00:48.562 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:00:48.567 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:00:48.977 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:48.977+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:00:49.941 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:49.940+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:00:50.913 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:50.913+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:00:51.879 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:51.879+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:00:52.842 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:52.842+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:00:53.881 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:53.881+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:00:54.856 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:54.855+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:00:55.885 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:55.885+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:00:56.869 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:56.869+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:00:57.894 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:57.894+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:00:58.878 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:58.878+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:00:59.844 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:00:59.844+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:00.834 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:00.834+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:01.804 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:01.804+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:02.777 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:02.776+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:03.746 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:03.746+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:04.737 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:04.737+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:05.718 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:05.718+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:06.681 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:06.681+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:07.648 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:07.648+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:08.643 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:08.643+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:09.634 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:09.634+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:10.665 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:10.664+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:11.653 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:11.652+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:12.641 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:12.641+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:13.632 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:13.631+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:14.621 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:14.620+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:15.668 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:15.667+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:16.710 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:16.709+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:17.667 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:17.667+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:18.636 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:01:18.640 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:01:18.657 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:18.655+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:19.614 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:19.614+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:20.539 DEBUG:teuthology.orchestra.run:got remote process result: 124 2022-04-03T20:01:20.540 ERROR:tasks.mds_thrash.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_cfb8f943163b374162da0d7b0240f267dd46e4e1/qa/tasks/mds_thrash.py", line 124, in _run self.do_thrash() File "/home/teuthworker/src/git.ceph.com_ceph-c_cfb8f943163b374162da0d7b0240f267dd46e4e1/qa/tasks/mds_thrash.py", line 288, in do_thrash status = self.fs.status() File "/home/teuthworker/src/git.ceph.com_ceph-c_cfb8f943163b374162da0d7b0240f267dd46e4e1/qa/tasks/cephfs/filesystem.py", line 381, in status return FSStatus(self.mon_manager, epoch) File "/home/teuthworker/src/git.ceph.com_ceph-c_cfb8f943163b374162da0d7b0240f267dd46e4e1/qa/tasks/cephfs/filesystem.py", line 78, in __init__ self.map = json.loads(self.mon.raw_cluster_cmd(*cmd)) File "/home/teuthworker/src/git.ceph.com_ceph-c_cfb8f943163b374162da0d7b0240f267dd46e4e1/qa/tasks/ceph_manager.py", line 1597, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_cfb8f943163b374162da0d7b0240f267dd46e4e1/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi077 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json' 2022-04-03T20:01:20.647 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:20.647+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:21.643 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:21.642+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:22.645 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:22.645+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:23.695 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:23.695+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:24.678 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:24.677+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:25.634 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:25.634+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:25.827 INFO:tasks.daemonwatchdog.daemon_watchdog:thrasher.fs.[cephfs] failed 2022-04-03T20:01:25.827 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2022-04-03T20:01:25.828 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2022-04-03T20:01:25.828 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-04-03T20:01:25.828 DEBUG:teuthology.orchestra.run.smithi077:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2022-04-03T20:01:26.644 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:26.644+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:27.667 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:27.666+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:28.657 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:28.657+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:29.652 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:29.652+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:30.605 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:30.605+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:31.595 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:31.595+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:32.596 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:32.596+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:33.637 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:33.636+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:34.623 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:34.623+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:35.646 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:35.646+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:36.694 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:36.693+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.563 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:37.562+0000 7f88020f4700 -1 bdev(0x5574497cd800 /var/lib/ceph/osd/ceph-2/block) _aio_thread stalled aio 0x557449859610 since 2022-04-03T20:00:37.536584+0000, timeout is 60s, suicide 2022-04-03T20:01:37.564 INFO:tasks.ceph.osd.2.smithi077.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f88020f4700 time 2022-04-03T20:01:37.563316+0000 2022-04-03T20:01:37.565 INFO:tasks.ceph.osd.2.smithi077.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: 666: ceph_abort_msg("stalled aio... buggy kernel or bad device?") 2022-04-03T20:01:37.565 INFO:tasks.ceph.osd.2.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:01:37.565 INFO:tasks.ceph.osd.2.smithi077.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x557444bdd0eb] 2022-04-03T20:01:37.565 INFO:tasks.ceph.osd.2.smithi077.stderr: 2: (KernelDevice::_aio_thread()+0x1275) [0x55744582fd95] 2022-04-03T20:01:37.565 INFO:tasks.ceph.osd.2.smithi077.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55744583d2f1] 2022-04-03T20:01:37.566 INFO:tasks.ceph.osd.2.smithi077.stderr: 4: /lib64/libpthread.so.0(+0x814a) [0x7f8812d4714a] 2022-04-03T20:01:37.566 INFO:tasks.ceph.osd.2.smithi077.stderr: 5: clone() 2022-04-03T20:01:37.566 INFO:tasks.ceph.osd.2.smithi077.stderr:*** Caught signal (Aborted) ** 2022-04-03T20:01:37.566 INFO:tasks.ceph.osd.2.smithi077.stderr: in thread 7f88020f4700 thread_name:bstore_aio 2022-04-03T20:01:37.567 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:37.564+0000 7f88020f4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f88020f4700 time 2022-04-03T20:01:37.563316+0000 2022-04-03T20:01:37.567 INFO:tasks.ceph.osd.2.smithi077.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: 666: ceph_abort_msg("stalled aio... buggy kernel or bad device?") 2022-04-03T20:01:37.567 INFO:tasks.ceph.osd.2.smithi077.stderr: 2022-04-03T20:01:37.567 INFO:tasks.ceph.osd.2.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:01:37.568 INFO:tasks.ceph.osd.2.smithi077.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x557444bdd0eb] 2022-04-03T20:01:37.568 INFO:tasks.ceph.osd.2.smithi077.stderr: 2: (KernelDevice::_aio_thread()+0x1275) [0x55744582fd95] 2022-04-03T20:01:37.568 INFO:tasks.ceph.osd.2.smithi077.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55744583d2f1] 2022-04-03T20:01:37.568 INFO:tasks.ceph.osd.2.smithi077.stderr: 4: /lib64/libpthread.so.0(+0x814a) [0x7f8812d4714a] 2022-04-03T20:01:37.568 INFO:tasks.ceph.osd.2.smithi077.stderr: 5: clone() 2022-04-03T20:01:37.569 INFO:tasks.ceph.osd.2.smithi077.stderr: 2022-04-03T20:01:37.569 INFO:tasks.ceph.osd.2.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:01:37.570 INFO:tasks.ceph.osd.2.smithi077.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f8812d51b20] 2022-04-03T20:01:37.570 INFO:tasks.ceph.osd.2.smithi077.stderr: 2: gsignal() 2022-04-03T20:01:37.570 INFO:tasks.ceph.osd.2.smithi077.stderr: 3: abort() 2022-04-03T20:01:37.570 INFO:tasks.ceph.osd.2.smithi077.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x557444bdd1be] 2022-04-03T20:01:37.571 INFO:tasks.ceph.osd.2.smithi077.stderr: 5: (KernelDevice::_aio_thread()+0x1275) [0x55744582fd95] 2022-04-03T20:01:37.571 INFO:tasks.ceph.osd.2.smithi077.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55744583d2f1] 2022-04-03T20:01:37.571 INFO:tasks.ceph.osd.2.smithi077.stderr: 7: /lib64/libpthread.so.0(+0x814a) [0x7f8812d4714a] 2022-04-03T20:01:37.571 INFO:tasks.ceph.osd.2.smithi077.stderr: 8: clone() 2022-04-03T20:01:37.572 INFO:tasks.ceph.osd.2.smithi077.stderr:2022-04-03T20:01:37.566+0000 7f88020f4700 -1 *** Caught signal (Aborted) ** 2022-04-03T20:01:37.572 INFO:tasks.ceph.osd.2.smithi077.stderr: in thread 7f88020f4700 thread_name:bstore_aio 2022-04-03T20:01:37.572 INFO:tasks.ceph.osd.2.smithi077.stderr: 2022-04-03T20:01:37.572 INFO:tasks.ceph.osd.2.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:01:37.572 INFO:tasks.ceph.osd.2.smithi077.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f8812d51b20] 2022-04-03T20:01:37.573 INFO:tasks.ceph.osd.2.smithi077.stderr: 2: gsignal() 2022-04-03T20:01:37.573 INFO:tasks.ceph.osd.2.smithi077.stderr: 3: abort() 2022-04-03T20:01:37.573 INFO:tasks.ceph.osd.2.smithi077.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x557444bdd1be] 2022-04-03T20:01:37.573 INFO:tasks.ceph.osd.2.smithi077.stderr: 5: (KernelDevice::_aio_thread()+0x1275) [0x55744582fd95] 2022-04-03T20:01:37.574 INFO:tasks.ceph.osd.2.smithi077.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55744583d2f1] 2022-04-03T20:01:37.574 INFO:tasks.ceph.osd.2.smithi077.stderr: 7: /lib64/libpthread.so.0(+0x814a) [0x7f8812d4714a] 2022-04-03T20:01:37.574 INFO:tasks.ceph.osd.2.smithi077.stderr: 8: clone() 2022-04-03T20:01:37.574 INFO:tasks.ceph.osd.2.smithi077.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-03T20:01:37.574 INFO:tasks.ceph.osd.2.smithi077.stderr: 2022-04-03T20:01:37.575 INFO:tasks.ceph.osd.2.smithi077.stderr: -7956> 2022-04-03T20:00:34.788+0000 7f88058fb700 -1 osd.2 53 heartbeat_check: no reply from 172.21.15.77:6828 osd.0 since back 2022-04-03T20:00:13.179511+0000 front 2022-04-03T20:00:13.179542+0000 (oldest deadline 2022-04-03T20:00:34.279548+0000) 2022-04-03T20:01:37.575 INFO:tasks.ceph.osd.2.smithi077.stderr: -7955> 2022-04-03T20:00:34.788+0000 7f88058fb700 -1 osd.2 53 heartbeat_check: no reply from 172.21.15.77:6812 osd.1 since back 2022-04-03T20:00:13.179489+0000 front 2022-04-03T20:00:13.179475+0000 (oldest deadline 2022-04-03T20:00:34.279548+0000) 2022-04-03T20:01:37.576 INFO:tasks.ceph.osd.2.smithi077.stderr: -7886> 2022-04-03T20:00:35.777+0000 7f88058fb700 -1 osd.2 53 heartbeat_check: no reply from 172.21.15.77:6828 osd.0 since back 2022-04-03T20:00:13.179511+0000 front 2022-04-03T20:00:13.179542+0000 (oldest deadline 2022-04-03T20:00:34.279548+0000) 2022-04-03T20:01:37.576 INFO:tasks.ceph.osd.2.smithi077.stderr: -7885> 2022-04-03T20:00:35.777+0000 7f88058fb700 -1 osd.2 53 heartbeat_check: no reply from 172.21.15.77:6812 osd.1 since back 2022-04-03T20:00:13.179489+0000 front 2022-04-03T20:00:13.179475+0000 (oldest deadline 2022-04-03T20:00:34.279548+0000) 2022-04-03T20:01:37.576 INFO:tasks.ceph.osd.2.smithi077.stderr: -7836> 2022-04-03T20:00:36.770+0000 7f88058fb700 -1 osd.2 53 heartbeat_check: no reply from 172.21.15.77:6828 osd.0 since back 2022-04-03T20:00:13.179511+0000 front 2022-04-03T20:00:13.179542+0000 (oldest deadline 2022-04-03T20:00:34.279548+0000) 2022-04-03T20:01:37.576 INFO:tasks.ceph.osd.2.smithi077.stderr: -7835> 2022-04-03T20:00:36.771+0000 7f88058fb700 -1 osd.2 53 heartbeat_check: no reply from 172.21.15.77:6812 osd.1 since back 2022-04-03T20:00:13.179489+0000 front 2022-04-03T20:00:13.179475+0000 (oldest deadline 2022-04-03T20:00:34.279548+0000) 2022-04-03T20:01:37.579 INFO:tasks.ceph.osd.2.smithi077.stderr: -5216> 2022-04-03T20:00:40.905+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.579 INFO:tasks.ceph.osd.2.smithi077.stderr: -5164> 2022-04-03T20:00:41.954+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.579 INFO:tasks.ceph.osd.2.smithi077.stderr: -5123> 2022-04-03T20:00:43.003+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.580 INFO:tasks.ceph.osd.2.smithi077.stderr: -5084> 2022-04-03T20:00:43.999+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.580 INFO:tasks.ceph.osd.2.smithi077.stderr: -5011> 2022-04-03T20:00:44.965+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.580 INFO:tasks.ceph.osd.2.smithi077.stderr: -4859> 2022-04-03T20:00:45.989+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.586 INFO:tasks.ceph.osd.2.smithi077.stderr: -4577> 2022-04-03T20:00:46.985+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.586 INFO:tasks.ceph.osd.2.smithi077.stderr: -4358> 2022-04-03T20:00:48.004+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.587 INFO:tasks.ceph.osd.2.smithi077.stderr: -4237> 2022-04-03T20:00:48.977+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.587 INFO:tasks.ceph.osd.2.smithi077.stderr: -4175> 2022-04-03T20:00:49.940+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.587 INFO:tasks.ceph.osd.2.smithi077.stderr: -4073> 2022-04-03T20:00:50.913+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.587 INFO:tasks.ceph.osd.2.smithi077.stderr: -4022> 2022-04-03T20:00:51.879+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.588 INFO:tasks.ceph.osd.2.smithi077.stderr: -3972> 2022-04-03T20:00:52.842+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.588 INFO:tasks.ceph.osd.2.smithi077.stderr: -3806> 2022-04-03T20:00:53.881+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.588 INFO:tasks.ceph.osd.2.smithi077.stderr: -3685> 2022-04-03T20:00:54.855+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.588 INFO:tasks.ceph.osd.2.smithi077.stderr: -3618> 2022-04-03T20:00:55.885+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.588 INFO:tasks.ceph.osd.2.smithi077.stderr: -3566> 2022-04-03T20:00:56.869+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.589 INFO:tasks.ceph.osd.2.smithi077.stderr: -3519> 2022-04-03T20:00:57.894+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.589 INFO:tasks.ceph.osd.2.smithi077.stderr: -3472> 2022-04-03T20:00:58.878+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.589 INFO:tasks.ceph.osd.2.smithi077.stderr: -3383> 2022-04-03T20:00:59.844+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.589 INFO:tasks.ceph.osd.2.smithi077.stderr: -3268> 2022-04-03T20:01:00.834+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.590 INFO:tasks.ceph.osd.2.smithi077.stderr: -3068> 2022-04-03T20:01:01.804+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.590 INFO:tasks.ceph.osd.2.smithi077.stderr: -3032> 2022-04-03T20:01:02.776+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.590 INFO:tasks.ceph.osd.2.smithi077.stderr: -2963> 2022-04-03T20:01:03.746+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.590 INFO:tasks.ceph.osd.2.smithi077.stderr: -2894> 2022-04-03T20:01:04.737+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.591 INFO:tasks.ceph.osd.2.smithi077.stderr: -2812> 2022-04-03T20:01:05.718+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.591 INFO:tasks.ceph.osd.2.smithi077.stderr: -2761> 2022-04-03T20:01:06.681+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.591 INFO:tasks.ceph.osd.2.smithi077.stderr: -2681> 2022-04-03T20:01:07.648+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.591 INFO:tasks.ceph.osd.2.smithi077.stderr: -2639> 2022-04-03T20:01:08.643+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.591 INFO:tasks.ceph.osd.2.smithi077.stderr: -2413> 2022-04-03T20:01:09.634+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.592 INFO:tasks.ceph.osd.2.smithi077.stderr: -2347> 2022-04-03T20:01:10.664+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.592 INFO:tasks.ceph.osd.2.smithi077.stderr: -2234> 2022-04-03T20:01:11.652+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.592 INFO:tasks.ceph.osd.2.smithi077.stderr: -2155> 2022-04-03T20:01:12.641+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.592 INFO:tasks.ceph.osd.2.smithi077.stderr: -2071> 2022-04-03T20:01:13.631+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.593 INFO:tasks.ceph.osd.2.smithi077.stderr: -1975> 2022-04-03T20:01:14.620+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.593 INFO:tasks.ceph.osd.2.smithi077.stderr: -1925> 2022-04-03T20:01:15.667+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.593 INFO:tasks.ceph.osd.2.smithi077.stderr: -1844> 2022-04-03T20:01:16.709+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.593 INFO:tasks.ceph.osd.2.smithi077.stderr: -1681> 2022-04-03T20:01:17.667+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.593 INFO:tasks.ceph.osd.2.smithi077.stderr: -1641> 2022-04-03T20:01:18.655+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.594 INFO:tasks.ceph.osd.2.smithi077.stderr: -1526> 2022-04-03T20:01:19.614+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.594 INFO:tasks.ceph.osd.2.smithi077.stderr: -1482> 2022-04-03T20:01:20.647+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.594 INFO:tasks.ceph.osd.2.smithi077.stderr: -1425> 2022-04-03T20:01:21.642+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.594 INFO:tasks.ceph.osd.2.smithi077.stderr: -1390> 2022-04-03T20:01:22.645+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.595 INFO:tasks.ceph.osd.2.smithi077.stderr: -1309> 2022-04-03T20:01:23.695+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.595 INFO:tasks.ceph.osd.2.smithi077.stderr: -1217> 2022-04-03T20:01:24.677+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.595 INFO:tasks.ceph.osd.2.smithi077.stderr: -1053> 2022-04-03T20:01:25.634+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.595 INFO:tasks.ceph.osd.2.smithi077.stderr: -972> 2022-04-03T20:01:26.644+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.596 INFO:tasks.ceph.osd.2.smithi077.stderr: -870> 2022-04-03T20:01:27.666+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.596 INFO:tasks.ceph.osd.2.smithi077.stderr: -766> 2022-04-03T20:01:28.657+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.596 INFO:tasks.ceph.osd.2.smithi077.stderr: -702> 2022-04-03T20:01:29.652+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.596 INFO:tasks.ceph.osd.2.smithi077.stderr: -655> 2022-04-03T20:01:30.605+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.597 INFO:tasks.ceph.osd.2.smithi077.stderr: -591> 2022-04-03T20:01:31.595+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.597 INFO:tasks.ceph.osd.2.smithi077.stderr: -479> 2022-04-03T20:01:32.596+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.597 INFO:tasks.ceph.osd.2.smithi077.stderr: -320> 2022-04-03T20:01:33.636+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.597 INFO:tasks.ceph.osd.2.smithi077.stderr: -279> 2022-04-03T20:01:34.623+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.597 INFO:tasks.ceph.osd.2.smithi077.stderr: -153> 2022-04-03T20:01:35.646+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.598 INFO:tasks.ceph.osd.2.smithi077.stderr: -93> 2022-04-03T20:01:36.693+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.598 INFO:tasks.ceph.osd.2.smithi077.stderr: -3> 2022-04-03T20:01:37.562+0000 7f88020f4700 -1 bdev(0x5574497cd800 /var/lib/ceph/osd/ceph-2/block) _aio_thread stalled aio 0x557449859610 since 2022-04-03T20:00:37.536584+0000, timeout is 60s, suicide 2022-04-03T20:01:37.598 INFO:tasks.ceph.osd.2.smithi077.stderr: -2> 2022-04-03T20:01:37.564+0000 7f88020f4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f88020f4700 time 2022-04-03T20:01:37.563316+0000 2022-04-03T20:01:37.598 INFO:tasks.ceph.osd.2.smithi077.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: 666: ceph_abort_msg("stalled aio... buggy kernel or bad device?") 2022-04-03T20:01:37.599 INFO:tasks.ceph.osd.2.smithi077.stderr: 2022-04-03T20:01:37.599 INFO:tasks.ceph.osd.2.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:01:37.599 INFO:tasks.ceph.osd.2.smithi077.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x557444bdd0eb] 2022-04-03T20:01:37.599 INFO:tasks.ceph.osd.2.smithi077.stderr: 2: (KernelDevice::_aio_thread()+0x1275) [0x55744582fd95] 2022-04-03T20:01:37.599 INFO:tasks.ceph.osd.2.smithi077.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55744583d2f1] 2022-04-03T20:01:37.600 INFO:tasks.ceph.osd.2.smithi077.stderr: 4: /lib64/libpthread.so.0(+0x814a) [0x7f8812d4714a] 2022-04-03T20:01:37.600 INFO:tasks.ceph.osd.2.smithi077.stderr: 5: clone() 2022-04-03T20:01:37.600 INFO:tasks.ceph.osd.2.smithi077.stderr: 2022-04-03T20:01:37.600 INFO:tasks.ceph.osd.2.smithi077.stderr: 0> 2022-04-03T20:01:37.566+0000 7f88020f4700 -1 *** Caught signal (Aborted) ** 2022-04-03T20:01:37.600 INFO:tasks.ceph.osd.2.smithi077.stderr: in thread 7f88020f4700 thread_name:bstore_aio 2022-04-03T20:01:37.601 INFO:tasks.ceph.osd.2.smithi077.stderr: 2022-04-03T20:01:37.601 INFO:tasks.ceph.osd.2.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:01:37.601 INFO:tasks.ceph.osd.2.smithi077.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f8812d51b20] 2022-04-03T20:01:37.601 INFO:tasks.ceph.osd.2.smithi077.stderr: 2: gsignal() 2022-04-03T20:01:37.602 INFO:tasks.ceph.osd.2.smithi077.stderr: 3: abort() 2022-04-03T20:01:37.602 INFO:tasks.ceph.osd.2.smithi077.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x557444bdd1be] 2022-04-03T20:01:37.602 INFO:tasks.ceph.osd.2.smithi077.stderr: 5: (KernelDevice::_aio_thread()+0x1275) [0x55744582fd95] 2022-04-03T20:01:37.602 INFO:tasks.ceph.osd.2.smithi077.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55744583d2f1] 2022-04-03T20:01:37.603 INFO:tasks.ceph.osd.2.smithi077.stderr: 7: /lib64/libpthread.so.0(+0x814a) [0x7f8812d4714a] 2022-04-03T20:01:37.603 INFO:tasks.ceph.osd.2.smithi077.stderr: 8: clone() 2022-04-03T20:01:37.603 INFO:tasks.ceph.osd.2.smithi077.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-03T20:01:37.603 INFO:tasks.ceph.osd.2.smithi077.stderr: 2022-04-03T20:01:37.610 INFO:tasks.ceph.osd.2.smithi077.stderr: -7956> 2022-04-03T20:00:34.788+0000 7f88058fb700 -1 osd.2 53 heartbeat_check: no reply from 172.21.15.77:6828 osd.0 since back 2022-04-03T20:00:13.179511+0000 front 2022-04-03T20:00:13.179542+0000 (oldest deadline 2022-04-03T20:00:34.279548+0000) 2022-04-03T20:01:37.610 INFO:tasks.ceph.osd.2.smithi077.stderr: -7955> 2022-04-03T20:00:34.788+0000 7f88058fb700 -1 osd.2 53 heartbeat_check: no reply from 172.21.15.77:6812 osd.1 since back 2022-04-03T20:00:13.179489+0000 front 2022-04-03T20:00:13.179475+0000 (oldest deadline 2022-04-03T20:00:34.279548+0000) 2022-04-03T20:01:37.610 INFO:tasks.ceph.osd.2.smithi077.stderr: -7886> 2022-04-03T20:00:35.777+0000 7f88058fb700 -1 osd.2 53 heartbeat_check: no reply from 172.21.15.77:6828 osd.0 since back 2022-04-03T20:00:13.179511+0000 front 2022-04-03T20:00:13.179542+0000 (oldest deadline 2022-04-03T20:00:34.279548+0000) 2022-04-03T20:01:37.611 INFO:tasks.ceph.osd.2.smithi077.stderr: -7885> 2022-04-03T20:00:35.777+0000 7f88058fb700 -1 osd.2 53 heartbeat_check: no reply from 172.21.15.77:6812 osd.1 since back 2022-04-03T20:00:13.179489+0000 front 2022-04-03T20:00:13.179475+0000 (oldest deadline 2022-04-03T20:00:34.279548+0000) 2022-04-03T20:01:37.611 INFO:tasks.ceph.osd.2.smithi077.stderr: -7836> 2022-04-03T20:00:36.770+0000 7f88058fb700 -1 osd.2 53 heartbeat_check: no reply from 172.21.15.77:6828 osd.0 since back 2022-04-03T20:00:13.179511+0000 front 2022-04-03T20:00:13.179542+0000 (oldest deadline 2022-04-03T20:00:34.279548+0000) 2022-04-03T20:01:37.611 INFO:tasks.ceph.osd.2.smithi077.stderr: -7835> 2022-04-03T20:00:36.771+0000 7f88058fb700 -1 osd.2 53 heartbeat_check: no reply from 172.21.15.77:6812 osd.1 since back 2022-04-03T20:00:13.179489+0000 front 2022-04-03T20:00:13.179475+0000 (oldest deadline 2022-04-03T20:00:34.279548+0000) 2022-04-03T20:01:37.611 INFO:tasks.ceph.osd.2.smithi077.stderr: -5216> 2022-04-03T20:00:40.905+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.611 INFO:tasks.ceph.osd.2.smithi077.stderr: -5164> 2022-04-03T20:00:41.954+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.612 INFO:tasks.ceph.osd.2.smithi077.stderr: -5123> 2022-04-03T20:00:43.003+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.612 INFO:tasks.ceph.osd.2.smithi077.stderr: -5084> 2022-04-03T20:00:43.999+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.612 INFO:tasks.ceph.osd.2.smithi077.stderr: -5011> 2022-04-03T20:00:44.965+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.612 INFO:tasks.ceph.osd.2.smithi077.stderr: -4859> 2022-04-03T20:00:45.989+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.613 INFO:tasks.ceph.osd.2.smithi077.stderr: -4577> 2022-04-03T20:00:46.985+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.613 INFO:tasks.ceph.osd.2.smithi077.stderr: -4358> 2022-04-03T20:00:48.004+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.613 INFO:tasks.ceph.osd.2.smithi077.stderr: -4237> 2022-04-03T20:00:48.977+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.613 INFO:tasks.ceph.osd.2.smithi077.stderr: -4175> 2022-04-03T20:00:49.940+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.613 INFO:tasks.ceph.osd.2.smithi077.stderr: -4073> 2022-04-03T20:00:50.913+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.614 INFO:tasks.ceph.osd.2.smithi077.stderr: -4022> 2022-04-03T20:00:51.879+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.614 INFO:tasks.ceph.osd.2.smithi077.stderr: -3972> 2022-04-03T20:00:52.842+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.614 INFO:tasks.ceph.osd.2.smithi077.stderr: -3806> 2022-04-03T20:00:53.881+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.614 INFO:tasks.ceph.osd.2.smithi077.stderr: -3685> 2022-04-03T20:00:54.855+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.615 INFO:tasks.ceph.osd.2.smithi077.stderr: -3618> 2022-04-03T20:00:55.885+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.615 INFO:tasks.ceph.osd.2.smithi077.stderr: -3566> 2022-04-03T20:00:56.869+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.615 INFO:tasks.ceph.osd.2.smithi077.stderr: -3519> 2022-04-03T20:00:57.894+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.615 INFO:tasks.ceph.osd.2.smithi077.stderr: -3472> 2022-04-03T20:00:58.878+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.616 INFO:tasks.ceph.osd.2.smithi077.stderr: -3383> 2022-04-03T20:00:59.844+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.616 INFO:tasks.ceph.osd.2.smithi077.stderr: -3268> 2022-04-03T20:01:00.834+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.616 INFO:tasks.ceph.osd.2.smithi077.stderr: -3068> 2022-04-03T20:01:01.804+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.616 INFO:tasks.ceph.osd.2.smithi077.stderr: -3032> 2022-04-03T20:01:02.776+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.617 INFO:tasks.ceph.osd.2.smithi077.stderr: -2963> 2022-04-03T20:01:03.746+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.617 INFO:tasks.ceph.osd.2.smithi077.stderr: -2894> 2022-04-03T20:01:04.737+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.617 INFO:tasks.ceph.osd.2.smithi077.stderr: -2812> 2022-04-03T20:01:05.718+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.617 INFO:tasks.ceph.osd.2.smithi077.stderr: -2761> 2022-04-03T20:01:06.681+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.617 INFO:tasks.ceph.osd.2.smithi077.stderr: -2681> 2022-04-03T20:01:07.648+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.618 INFO:tasks.ceph.osd.2.smithi077.stderr: -2639> 2022-04-03T20:01:08.643+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.618 INFO:tasks.ceph.osd.2.smithi077.stderr: -2413> 2022-04-03T20:01:09.634+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.618 INFO:tasks.ceph.osd.2.smithi077.stderr: -2347> 2022-04-03T20:01:10.664+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.618 INFO:tasks.ceph.osd.2.smithi077.stderr: -2234> 2022-04-03T20:01:11.652+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.619 INFO:tasks.ceph.osd.2.smithi077.stderr: -2155> 2022-04-03T20:01:12.641+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.619 INFO:tasks.ceph.osd.2.smithi077.stderr: -2071> 2022-04-03T20:01:13.631+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.619 INFO:tasks.ceph.osd.2.smithi077.stderr: -1975> 2022-04-03T20:01:14.620+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.619 INFO:tasks.ceph.osd.2.smithi077.stderr: -1925> 2022-04-03T20:01:15.667+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.619 INFO:tasks.ceph.osd.2.smithi077.stderr: -1844> 2022-04-03T20:01:16.709+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.620 INFO:tasks.ceph.osd.2.smithi077.stderr: -1681> 2022-04-03T20:01:17.667+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.620 INFO:tasks.ceph.osd.2.smithi077.stderr: -1641> 2022-04-03T20:01:18.655+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.620 INFO:tasks.ceph.osd.2.smithi077.stderr: -1526> 2022-04-03T20:01:19.614+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.620 INFO:tasks.ceph.osd.2.smithi077.stderr: -1482> 2022-04-03T20:01:20.647+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.621 INFO:tasks.ceph.osd.2.smithi077.stderr: -1425> 2022-04-03T20:01:21.642+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.621 INFO:tasks.ceph.osd.2.smithi077.stderr: -1390> 2022-04-03T20:01:22.645+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.621 INFO:tasks.ceph.osd.2.smithi077.stderr: -1309> 2022-04-03T20:01:23.695+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.621 INFO:tasks.ceph.osd.2.smithi077.stderr: -1217> 2022-04-03T20:01:24.677+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.621 INFO:tasks.ceph.osd.2.smithi077.stderr: -1053> 2022-04-03T20:01:25.634+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.622 INFO:tasks.ceph.osd.2.smithi077.stderr: -972> 2022-04-03T20:01:26.644+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.622 INFO:tasks.ceph.osd.2.smithi077.stderr: -870> 2022-04-03T20:01:27.666+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.622 INFO:tasks.ceph.osd.2.smithi077.stderr: -766> 2022-04-03T20:01:28.657+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.622 INFO:tasks.ceph.osd.2.smithi077.stderr: -702> 2022-04-03T20:01:29.652+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.623 INFO:tasks.ceph.osd.2.smithi077.stderr: -655> 2022-04-03T20:01:30.605+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.623 INFO:tasks.ceph.osd.2.smithi077.stderr: -591> 2022-04-03T20:01:31.595+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.623 INFO:tasks.ceph.osd.2.smithi077.stderr: -479> 2022-04-03T20:01:32.596+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.623 INFO:tasks.ceph.osd.2.smithi077.stderr: -320> 2022-04-03T20:01:33.636+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.623 INFO:tasks.ceph.osd.2.smithi077.stderr: -279> 2022-04-03T20:01:34.623+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.624 INFO:tasks.ceph.osd.2.smithi077.stderr: -153> 2022-04-03T20:01:35.646+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.624 INFO:tasks.ceph.osd.2.smithi077.stderr: -93> 2022-04-03T20:01:36.693+0000 7f88058fb700 -1 osd.2 55 heartbeat_check: no reply from 172.21.15.77:6820 osd.3 since back 2022-04-03T20:00:18.380554+0000 front 2022-04-03T20:00:18.380595+0000 (oldest deadline 2022-04-03T20:00:40.680689+0000) 2022-04-03T20:01:37.624 INFO:tasks.ceph.osd.2.smithi077.stderr: -3> 2022-04-03T20:01:37.562+0000 7f88020f4700 -1 bdev(0x5574497cd800 /var/lib/ceph/osd/ceph-2/block) _aio_thread stalled aio 0x557449859610 since 2022-04-03T20:00:37.536584+0000, timeout is 60s, suicide 2022-04-03T20:01:37.624 INFO:tasks.ceph.osd.2.smithi077.stderr: -2> 2022-04-03T20:01:37.564+0000 7f88020f4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f88020f4700 time 2022-04-03T20:01:37.563316+0000 2022-04-03T20:01:37.625 INFO:tasks.ceph.osd.2.smithi077.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11442-gcfb8f943/rpm/el8/BUILD/ceph-17.0.0-11442-gcfb8f943/src/blk/kernel/KernelDevice.cc: 666: ceph_abort_msg("stalled aio... buggy kernel or bad device?") 2022-04-03T20:01:37.625 INFO:tasks.ceph.osd.2.smithi077.stderr: 2022-04-03T20:01:37.625 INFO:tasks.ceph.osd.2.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:01:37.625 INFO:tasks.ceph.osd.2.smithi077.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x557444bdd0eb] 2022-04-03T20:01:37.625 INFO:tasks.ceph.osd.2.smithi077.stderr: 2: (KernelDevice::_aio_thread()+0x1275) [0x55744582fd95] 2022-04-03T20:01:37.626 INFO:tasks.ceph.osd.2.smithi077.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55744583d2f1] 2022-04-03T20:01:37.626 INFO:tasks.ceph.osd.2.smithi077.stderr: 4: /lib64/libpthread.so.0(+0x814a) [0x7f8812d4714a] 2022-04-03T20:01:37.626 INFO:tasks.ceph.osd.2.smithi077.stderr: 5: clone() 2022-04-03T20:01:37.626 INFO:tasks.ceph.osd.2.smithi077.stderr: 2022-04-03T20:01:37.627 INFO:tasks.ceph.osd.2.smithi077.stderr: 0> 2022-04-03T20:01:37.566+0000 7f88020f4700 -1 *** Caught signal (Aborted) ** 2022-04-03T20:01:37.627 INFO:tasks.ceph.osd.2.smithi077.stderr: in thread 7f88020f4700 thread_name:bstore_aio 2022-04-03T20:01:37.627 INFO:tasks.ceph.osd.2.smithi077.stderr: 2022-04-03T20:01:37.627 INFO:tasks.ceph.osd.2.smithi077.stderr: ceph version 17.0.0-11442-gcfb8f943 (cfb8f943163b374162da0d7b0240f267dd46e4e1) quincy (dev) 2022-04-03T20:01:37.627 INFO:tasks.ceph.osd.2.smithi077.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f8812d51b20] 2022-04-03T20:01:37.628 INFO:tasks.ceph.osd.2.smithi077.stderr: 2: gsignal() 2022-04-03T20:01:37.628 INFO:tasks.ceph.osd.2.smithi077.stderr: 3: abort() 2022-04-03T20:01:37.628 INFO:tasks.ceph.osd.2.smithi077.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x557444bdd1be] 2022-04-03T20:01:37.628 INFO:tasks.ceph.osd.2.smithi077.stderr: 5: (KernelDevice::_aio_thread()+0x1275) [0x55744582fd95] 2022-04-03T20:01:37.629 INFO:tasks.ceph.osd.2.smithi077.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55744583d2f1] 2022-04-03T20:01:37.629 INFO:tasks.ceph.osd.2.smithi077.stderr: 7: /lib64/libpthread.so.0(+0x814a) [0x7f8812d4714a] 2022-04-03T20:01:37.629 INFO:tasks.ceph.osd.2.smithi077.stderr: 8: clone() 2022-04-03T20:01:37.629 INFO:tasks.ceph.osd.2.smithi077.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-04-03T20:01:37.629 INFO:tasks.ceph.osd.2.smithi077.stderr: 2022-04-03T20:01:48.708 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:01:48.712 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:01:59.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:01:59.380+0000 7fd3b190b700 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.77:6806 osd.2 since back 2022-04-03T20:01:35.891754+0000 front 2022-04-03T20:01:35.891808+0000 (oldest deadline 2022-04-03T20:01:58.791627+0000) 2022-04-03T20:01:59.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:01:59.560+0000 7f88d0ac1700 -1 osd.4 59 heartbeat_check: no reply from 172.21.15.77:6806 osd.2 since back 2022-04-03T20:01:36.292391+0000 front 2022-04-03T20:01:36.292482+0000 (oldest deadline 2022-04-03T20:01:59.192378+0000) 2022-04-03T20:02:00.316 INFO:tasks.ceph.osd.5.smithi186.stderr:2022-04-03T20:02:00.315+0000 7f771fd0c700 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.77:6806 osd.2 since back 2022-04-03T20:01:34.353431+0000 front 2022-04-03T20:01:34.353393+0000 (oldest deadline 2022-04-03T20:02:00.253329+0000) 2022-04-03T20:02:00.359 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:02:00.358+0000 7fd3b190b700 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.77:6806 osd.2 since back 2022-04-03T20:01:35.891754+0000 front 2022-04-03T20:01:35.891808+0000 (oldest deadline 2022-04-03T20:01:58.791627+0000) 2022-04-03T20:02:00.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:02:00.605+0000 7f88d0ac1700 -1 osd.4 59 heartbeat_check: no reply from 172.21.15.77:6806 osd.2 since back 2022-04-03T20:01:36.292391+0000 front 2022-04-03T20:01:36.292482+0000 (oldest deadline 2022-04-03T20:01:59.192378+0000) 2022-04-03T20:02:01.302 INFO:tasks.ceph.osd.5.smithi186.stderr:2022-04-03T20:02:01.302+0000 7f771fd0c700 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.77:6806 osd.2 since back 2022-04-03T20:01:34.353431+0000 front 2022-04-03T20:01:34.353393+0000 (oldest deadline 2022-04-03T20:02:00.253329+0000) 2022-04-03T20:02:01.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:02:01.357+0000 7fd3b190b700 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.77:6806 osd.2 since back 2022-04-03T20:01:35.891754+0000 front 2022-04-03T20:01:35.891808+0000 (oldest deadline 2022-04-03T20:01:58.791627+0000) 2022-04-03T20:02:01.573 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:02:01.572+0000 7f88d0ac1700 -1 osd.4 59 heartbeat_check: no reply from 172.21.15.77:6806 osd.2 since back 2022-04-03T20:01:36.292391+0000 front 2022-04-03T20:01:36.292482+0000 (oldest deadline 2022-04-03T20:01:59.192378+0000) 2022-04-03T20:02:01.796 INFO:tasks.ceph.osd.7.smithi186.stderr:2022-04-03T20:02:01.795+0000 7fc1bc457700 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.77:6806 osd.2 since back 2022-04-03T20:01:35.540773+0000 front 2022-04-03T20:01:35.540761+0000 (oldest deadline 2022-04-03T20:02:00.840686+0000) 2022-04-03T20:02:02.270 INFO:tasks.ceph.osd.5.smithi186.stderr:2022-04-03T20:02:02.269+0000 7f771fd0c700 -1 osd.5 59 heartbeat_check: no reply from 172.21.15.77:6806 osd.2 since back 2022-04-03T20:01:34.353431+0000 front 2022-04-03T20:01:34.353393+0000 (oldest deadline 2022-04-03T20:02:00.253329+0000) 2022-04-03T20:02:02.370 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:02:02.369+0000 7fd3b190b700 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.77:6806 osd.2 since back 2022-04-03T20:01:35.891754+0000 front 2022-04-03T20:01:35.891808+0000 (oldest deadline 2022-04-03T20:01:58.791627+0000) 2022-04-03T20:02:02.758 INFO:tasks.ceph.osd.7.smithi186.stderr:2022-04-03T20:02:02.757+0000 7fc1bc457700 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.77:6806 osd.2 since back 2022-04-03T20:01:35.540773+0000 front 2022-04-03T20:01:35.540761+0000 (oldest deadline 2022-04-03T20:02:00.840686+0000) 2022-04-03T20:02:03.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:02:03.370+0000 7fd3b190b700 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.77:6806 osd.2 since back 2022-04-03T20:01:35.891754+0000 front 2022-04-03T20:01:35.891808+0000 (oldest deadline 2022-04-03T20:01:58.791627+0000) 2022-04-03T20:02:03.754 INFO:tasks.ceph.osd.7.smithi186.stderr:2022-04-03T20:02:03.753+0000 7fc1bc457700 -1 osd.7 59 heartbeat_check: no reply from 172.21.15.77:6806 osd.2 since back 2022-04-03T20:01:35.540773+0000 front 2022-04-03T20:01:35.540761+0000 (oldest deadline 2022-04-03T20:02:00.840686+0000) 2022-04-03T20:02:04.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:02:04.416+0000 7fd3b190b700 -1 osd.6 59 heartbeat_check: no reply from 172.21.15.77:6806 osd.2 since back 2022-04-03T20:01:35.891754+0000 front 2022-04-03T20:01:35.891808+0000 (oldest deadline 2022-04-03T20:01:58.791627+0000) 2022-04-03T20:02:18.798 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:02:18.802 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:02:48.896 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:02:48.900 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:03:18.970 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:03:18.974 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:03:49.059 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:03:49.063 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:04:19.148 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:04:19.152 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:04:49.223 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:04:49.227 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:05:19.295 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:05:19.299 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:05:49.376 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:05:49.380 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:06:19.456 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:06:19.460 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:06:49.521 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:06:49.525 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:07:19.584 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:07:19.588 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:07:49.694 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:07:49.698 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:08:19.797 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:08:19.801 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:08:49.878 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:08:49.882 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:09:19.967 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:09:19.971 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:09:40.241 INFO:tasks.ceph.mds.c.smithi077.stdout:starting mds.c at 2022-04-03T20:09:50.048 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:09:50.052 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:10:20.118 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:10:20.123 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:10:50.193 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:10:50.197 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:11:20.567 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:11:20.572 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:11:51.192 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:11:51.196 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:12:21.699 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:12:21.703 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:12:51.779 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:12:51.783 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:13:21.852 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:13:21.856 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:13:39.520 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:39.519+0000 7f88d0ac1700 -1 osd.4 111 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:13:40.558 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:40.557+0000 7f88d0ac1700 -1 osd.4 112 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:13:41.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:41.541+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:13:42.540 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:42.539+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:13:43.537 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:43.536+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:13:44.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:44.544+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:13:45.572 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:45.571+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:13:46.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:46.543+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:13:47.526 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:47.525+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:13:48.489 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:48.488+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:13:49.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:49.486+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:13:50.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:13:50.278+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:13:50.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:50.457+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:13:51.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:13:51.309+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:13:51.490 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:51.489+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:13:51.927 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:13:51.931 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:13:52.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:13:52.340+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:13:52.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:52.496+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:13:53.338 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:13:53.337+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:13:53.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:53.533+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:13:54.387 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:13:54.386+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:13:54.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:54.489+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:13:55.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:13:55.348+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:13:55.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:55.481+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:13:56.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:13:56.387+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:13:56.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:56.523+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:13:57.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:13:57.412+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:13:57.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:57.561+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:13:58.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:13:58.416+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:13:58.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:58.600+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:13:59.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:13:59.441+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:13:59.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:13:59.610+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:00.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:00.430+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:00.496 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:14:00.496+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:14:00.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:00.649+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:00.697 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:14:00.697+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:14:01.099 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:14:01.098+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:14:01.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:01.442+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:01.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:01.629+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:01.900 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:14:01.899+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:14:02.452 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:02.451+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:02.664 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:02.663+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:03.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:03.484+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:03.502 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:14:03.502+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:14:03.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:03.629+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:04.506 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:04.505+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:04.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:04.589+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:05.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:05.514+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:05.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:05.547+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:06.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:06.490+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:06.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:06.543+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:06.706 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:14:06.705+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:14:07.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:07.443+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:07.582 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:07.581+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:08.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:08.430+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:08.609 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:08.608+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:09.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:09.404+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:09.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:09.561+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:10.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:10.442+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:10.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:10.588+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:11.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:11.431+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:11.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:11.557+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:12.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:12.464+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:12.522 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:12.522+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:13.113 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:14:13.112+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:14:13.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:13.481+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:13.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:13.481+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:14.484 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:14.483+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:14.508 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:14.508+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:15.448 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:15.447+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:15.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:15.461+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:16.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:16.416+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:16.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:16.487+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:17.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:17.439+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:17.475 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:17.475+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:18.428 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:18.428+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:18.507 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:18.507+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:19.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:19.381+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:19.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:19.534+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:20.342 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:20.342+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:20.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:20.541+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:21.322 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:21.322+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:21.522 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:21.522+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:22.025 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:14:22.029 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:14:22.347 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:22.347+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:22.544 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:22.544+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:23.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:23.390+0000 7fd3b190b700 -1 osd.6 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:23.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:23.574+0000 7f88d0ac1700 -1 osd.4 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:24.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:24.376+0000 7fd3b190b700 -1 osd.6 114 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:24.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:24.591+0000 7f88d0ac1700 -1 osd.4 114 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:25.397 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:25.396+0000 7fd3b190b700 -1 osd.6 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:25.585 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:25.584+0000 7f88d0ac1700 -1 osd.4 115 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:25.917 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:14:25.915+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:14:26.347 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:26.346+0000 7fd3b190b700 -1 osd.6 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:26.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:26.565+0000 7f88d0ac1700 -1 osd.4 115 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:27.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:27.350+0000 7fd3b190b700 -1 osd.6 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:27.539 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:27.538+0000 7f88d0ac1700 -1 osd.4 115 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:28.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:28.308+0000 7fd3b190b700 -1 osd.6 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:28.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:28.575+0000 7f88d0ac1700 -1 osd.4 115 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:29.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:29.339+0000 7fd3b190b700 -1 osd.6 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:29.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:29.587+0000 7f88d0ac1700 -1 osd.4 115 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:30.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:30.290+0000 7fd3b190b700 -1 osd.6 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:30.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:30.549+0000 7f88d0ac1700 -1 osd.4 115 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:31.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:31.303+0000 7fd3b190b700 -1 osd.6 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:31.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:31.523+0000 7f88d0ac1700 -1 osd.4 115 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:32.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:32.284+0000 7fd3b190b700 -1 osd.6 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:32.507 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:32.506+0000 7f88d0ac1700 -1 osd.4 115 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:33.304 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:33.302+0000 7fd3b190b700 -1 osd.6 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:33.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:33.503+0000 7f88d0ac1700 -1 osd.4 115 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:34.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:34.263+0000 7fd3b190b700 -1 osd.6 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:34.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:34.512+0000 7f88d0ac1700 -1 osd.4 115 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:35.220 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:35.219+0000 7fd3b190b700 -1 osd.6 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:35.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:35.482+0000 7f88d0ac1700 -1 osd.4 115 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:36.256 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:36.255+0000 7fd3b190b700 -1 osd.6 116 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:36.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:36.440+0000 7f88d0ac1700 -1 osd.4 116 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:37.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:37.267+0000 7fd3b190b700 -1 osd.6 117 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:37.464 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:37.463+0000 7f88d0ac1700 -1 osd.4 117 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:38.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:38.298+0000 7fd3b190b700 -1 osd.6 117 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:38.503 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:38.501+0000 7f88d0ac1700 -1 osd.4 117 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:39.264 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:39.263+0000 7fd3b190b700 -1 osd.6 117 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:39.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:39.507+0000 7f88d0ac1700 -1 osd.4 117 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:40.306 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:40.305+0000 7fd3b190b700 -1 osd.6 117 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:40.519 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:40.518+0000 7f88d0ac1700 -1 osd.4 117 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:40.917 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:14:40.916+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:14:41.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:41.275+0000 7fd3b190b700 -1 osd.6 117 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:41.502 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:41.501+0000 7f88d0ac1700 -1 osd.4 117 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:42.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:42.256+0000 7fd3b190b700 -1 osd.6 117 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:42.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:42.516+0000 7f88d0ac1700 -1 osd.4 117 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:43.255 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:43.254+0000 7fd3b190b700 -1 osd.6 118 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:43.480 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:43.479+0000 7f88d0ac1700 -1 osd.4 118 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:44.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:44.218+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:44.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:44.466+0000 7f88d0ac1700 -1 osd.4 118 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:45.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:45.181+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:45.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:45.426+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:46.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:46.228+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:46.446 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:46.445+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:47.234 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:47.233+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:47.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:47.486+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:48.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:48.233+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:48.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:48.523+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:49.212 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:49.211+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:49.566 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:49.565+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:50.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:50.174+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:50.604 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:50.603+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:51.144 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:51.142+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:51.606 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:51.605+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:52.157 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:14:52.161 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:14:52.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:52.186+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:52.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:52.576+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:53.155 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:53.154+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:53.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:53.557+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:54.155 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:54.154+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:54.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:54.595+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:55.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:55.170+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:55.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:55.586+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:55.933 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:14:55.932+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:14:56.190 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:56.190+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:56.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:56.542+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:57.225 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:57.224+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:57.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:57.578+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:58.212 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:58.211+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:58.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:58.614+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:14:59.224 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:14:59.224+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:14:59.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:14:59.613+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:00.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:00.196+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:00.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:00.588+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:01.225 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:01.224+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:01.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:01.616+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:02.222 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:02.222+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:02.585 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:02.584+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:03.266 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:03.265+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:03.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:03.537+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:04.236 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:04.235+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:04.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:04.515+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:05.271 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:05.270+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:05.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:05.529+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:06.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:06.259+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:06.506 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:06.506+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:07.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:07.249+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:07.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:07.495+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:08.262 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:08.262+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:08.517 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:08.517+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:09.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:09.248+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:09.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:09.536+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:10.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:10.281+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:10.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:10.567+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:10.934 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:15:10.932+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:15:11.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:11.276+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:11.533 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:11.533+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:12.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:12.299+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:12.501 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:12.501+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:13.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:13.271+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:13.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:13.455+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:14.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:14.315+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:14.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:14.505+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:15.332 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:15.332+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:15.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:15.523+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:16.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:16.305+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:16.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:16.505+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:17.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:17.310+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:17.472 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:17.472+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:18.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:18.328+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:18.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:18.441+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:19.364 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:19.364+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:19.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:19.461+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:20.412 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:20.411+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:20.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:20.457+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:21.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:21.410+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:21.422 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:21.421+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:22.286 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:15:22.290 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:15:22.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:22.373+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:22.448 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:22.447+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:23.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:23.324+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:23.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:23.424+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:24.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:24.342+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:24.420 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:24.418+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:25.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:25.324+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:25.433 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:25.432+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:25.950 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:15:25.949+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:15:26.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:26.362+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:26.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:26.393+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:27.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:27.324+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:27.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:27.417+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:28.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:28.350+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:28.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:28.434+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:29.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:29.372+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:29.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:29.438+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:30.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:30.333+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:30.392 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:30.391+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:31.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:31.299+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:31.392 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:31.391+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:32.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:32.342+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:32.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:32.388+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:33.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:33.303+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:33.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:33.429+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:34.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:34.260+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:34.403 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:34.402+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:35.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:35.295+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:35.356 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:35.355+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:36.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:36.274+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:36.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:36.404+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:37.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:37.264+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:37.448 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:37.447+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:38.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:38.275+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:38.434 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:38.432+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:39.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:39.253+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:39.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:39.439+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:40.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:40.268+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:40.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:40.473+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:40.966 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:15:40.965+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:15:41.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:41.291+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:41.469 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:41.468+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:42.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:42.297+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:42.446 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:42.446+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:43.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:43.266+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:43.441 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:43.440+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:44.105 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T20:15:44.105+0000 7fb3d4e34700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T20:15:44.123 INFO:tasks.ceph.mds.e.smithi077.stderr: -1> 2022-04-03T20:15:44.105+0000 7fb3d4e34700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T20:15:44.150 INFO:tasks.ceph.mds.e.smithi077.stdout:starting mds.e at 2022-04-03T20:15:44.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:44.269+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:44.472 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:44.471+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:45.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:45.238+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:45.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:45.485+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:46.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:46.285+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:46.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:46.531+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:47.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:47.324+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:47.522 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:47.521+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:48.283 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:48.282+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:48.559 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:48.558+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:49.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:49.291+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:49.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:49.534+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:50.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:50.266+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:50.527 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:50.526+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:51.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:51.226+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:51.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:51.542+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:52.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:52.250+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:52.408 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:15:52.412 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:15:52.500 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:52.500+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:53.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:53.289+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:53.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:53.456+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:54.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:54.246+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:54.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:54.439+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:55.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:55.210+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:55.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:55.405+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:55.978 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:15:55.977+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:15:56.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:56.179+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:56.450 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:56.449+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:57.172 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:57.171+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:57.402 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:57.401+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:58.195 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:58.195+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:58.397 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:58.396+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:15:59.170 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:15:59.169+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:15:59.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:15:59.443+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:00.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:00.181+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:00.442 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:00.442+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:01.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:01.210+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:01.453 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:01.453+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:02.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:02.172+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:02.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:02.405+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:03.145 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:03.145+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:03.448 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:03.448+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:04.144 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:04.144+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:04.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:04.412+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:05.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:05.147+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:05.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:05.418+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:06.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:06.147+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:06.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:06.427+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:07.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:07.166+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:07.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:07.429+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:08.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:08.137+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:08.453 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:08.452+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:09.104 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:09.104+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:09.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:09.439+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:10.092 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:10.092+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:10.428 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:10.428+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:10.995 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:16:10.994+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:16:11.054 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:11.054+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:11.395 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:11.394+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:12.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:12.038+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:12.370 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:12.370+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:13.019 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:13.019+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:13.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:13.322+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:14.051 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:14.049+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:14.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:14.320+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:15.083 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:15.081+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:15.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:15.335+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:16.101 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:16.099+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:16.366 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:16.365+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:17.094 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:17.092+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:17.359 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:17.358+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:18.106 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:18.104+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:18.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:18.403+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:19.087 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:19.086+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:19.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:19.425+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:20.108 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:20.107+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:20.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:20.462+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:21.089 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:21.088+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:21.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:21.421+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:22.087 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:22.086+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:22.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:22.460+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:22.524 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:16:22.527 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:16:23.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:23.066+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:23.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:23.494+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:24.087 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:24.084+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:24.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:24.520+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:25.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:25.066+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:25.500 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:25.499+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:25.835 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-04-03T20:16:25.837 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-04-03T20:16:26.010 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:16:26.009+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:16:26.022 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:26.020+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:26.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:26.537+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:26.972 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:26.971+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:27.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:27.514+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:27.993 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:27.991+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:28.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:28.551+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:29.035 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:29.034+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:29.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:29.534+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:30.011 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:30.010+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:30.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:30.532+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:31.014 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:31.013+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:31.556 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:31.554+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:32.054 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:32.053+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:32.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:32.538+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:33.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:33.080+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:33.558 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:33.557+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:34.128 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:34.126+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:34.590 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:34.589+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:35.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:35.155+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:35.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:35.622+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:36.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:36.203+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:36.593 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:36.592+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:37.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:37.245+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:37.634 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:37.633+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:38.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:38.260+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:38.636 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:38.635+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:39.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:39.239+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:39.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:39.624+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:40.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:40.218+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:40.649 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:40.649+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:41.026 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:16:41.025+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:16:41.221 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:41.220+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:41.666 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:41.665+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:42.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:42.176+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:42.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:42.661+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:43.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:43.199+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:43.697 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:43.696+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:44.233 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:44.232+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:44.659 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:44.658+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:45.213 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:45.212+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:45.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:45.661+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:46.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:46.177+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:46.704 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:46.703+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:47.174 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:47.173+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:47.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:47.681+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:48.142 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:48.141+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:48.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:48.668+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:49.141 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:49.139+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:49.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:49.643+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:50.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:50.175+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:50.637 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:50.636+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:51.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:51.177+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:51.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:51.679+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:52.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:52.218+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:52.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:52.657+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:52.662 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:16:52.666 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:16:53.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:53.193+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:53.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:53.623+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:54.183 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:54.182+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:54.609 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:54.609+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:55.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:55.227+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:55.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:55.591+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:56.042 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:16:56.041+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:16:56.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:56.247+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:56.555 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:56.555+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:57.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:57.206+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:57.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:57.542+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:58.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:58.181+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:58.496 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:58.496+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:16:59.136 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:16:59.135+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:16:59.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:16:59.457+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:00.135 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:00.135+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:00.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:00.488+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:01.097 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:01.096+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:01.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:01.447+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:02.140 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:02.139+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:02.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:02.417+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:03.129 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:03.128+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:03.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:03.423+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:04.138 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:04.138+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:04.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:04.376+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:05.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:05.186+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:05.346 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:05.346+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:06.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:06.184+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:06.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:06.302+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:07.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:07.161+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:07.293 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:07.292+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:08.171 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:08.171+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:08.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:08.259+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:09.141 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:09.139+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:09.240 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:09.239+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:10.149 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:10.148+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:10.242 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:10.240+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:11.058 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:17:11.057+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:17:11.130 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:11.129+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:11.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:11.217+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:12.147 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:12.146+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:12.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:12.186+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:13.151 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:13.150+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:13.230 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:13.229+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:14.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:14.194+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:14.252 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:14.250+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:15.169 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:15.168+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:15.233 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:15.231+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:16.158 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:16.156+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:16.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:16.255+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:17.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:17.135+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:17.307 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:17.306+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:18.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:18.155+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:18.271 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:18.270+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:19.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:19.177+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:19.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:19.256+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:20.147 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:20.146+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:20.262 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:20.261+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:21.130 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:21.129+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:21.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:21.285+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:22.084 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:22.083+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:22.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:22.270+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:22.809 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:17:22.813 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:17:23.109 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:23.108+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:23.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:23.257+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:24.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:24.065+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:24.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:24.216+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:25.083 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:25.082+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:25.261 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:25.260+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:26.050 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:26.048+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:26.065 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:17:26.064+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:17:26.289 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:26.288+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:27.051 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:27.050+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:27.287 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:27.286+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:28.036 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:28.035+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:28.287 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:28.286+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:29.008 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:29.006+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:29.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:29.256+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:30.052 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:30.051+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:30.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:30.277+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:31.087 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:31.085+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:31.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:31.309+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:32.059 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:32.058+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:32.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:32.349+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:33.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:33.084+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:33.360 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:33.359+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:34.127 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:34.126+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:34.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:34.329+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:35.116 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:35.114+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:35.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:35.304+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:36.069 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:36.068+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:36.312 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:36.310+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:37.061 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:37.060+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:37.322 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:37.320+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:38.050 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:38.049+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:38.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:38.313+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:39.074 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:39.073+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:39.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:39.305+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:40.033 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:40.032+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:40.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:40.296+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:41.032 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:41.030+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:41.081 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:17:41.080+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:17:41.304 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:41.303+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:41.990 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:41.989+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:42.301 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:42.300+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:42.973 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:42.972+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:43.313 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:43.312+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:43.986 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:43.985+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:44.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:44.351+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:45.000 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:44.999+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:45.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:45.319+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:46.037 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:46.035+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:46.294 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:46.293+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:47.013 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:47.012+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:47.256 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:47.255+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:47.986 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:47.985+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:48.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:48.291+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:48.946 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:48.945+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:49.247 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:49.246+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:49.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:49.948+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:50.228 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:50.228+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:50.929 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:50.929+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:51.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:51.249+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:51.891 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:51.891+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:52.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:52.278+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:52.858 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:52.857+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:52.948 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:17:52.952 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:17:53.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:53.297+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:53.830 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:53.829+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:54.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:54.329+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:54.786 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:54.786+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:55.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:55.377+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:55.804 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:55.804+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:56.090 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:17:56.088+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:17:56.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:56.404+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:56.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:56.842+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:57.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:57.427+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:57.816 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:57.815+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:58.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:58.385+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:58.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:58.791+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:17:59.338 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:17:59.338+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:17:59.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:17:59.784+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:00.312 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:00.312+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:00.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:00.833+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:01.289 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:01.289+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:01.848 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:01.847+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:02.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:02.310+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:02.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:02.829+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:03.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:03.301+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:03.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:03.784+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:04.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:04.283+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:04.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:04.750+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:05.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:05.308+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:05.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:05.739+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:06.270 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:06.269+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:06.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:06.739+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:07.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:07.309+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:07.783 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:07.781+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:08.318 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:08.317+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:08.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:08.800+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:09.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:09.315+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:09.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:09.814+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:10.294 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:10.293+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:10.778 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:10.776+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:11.106 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:18:11.105+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:18:11.287 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:11.286+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:11.804 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:11.802+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:12.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:12.277+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:12.803 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:12.802+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:13.318 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:13.317+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:13.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:13.769+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:14.274 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:14.273+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:14.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:14.739+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:15.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:15.280+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:15.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:15.695+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:16.289 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:16.287+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:16.705 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:16.704+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:17.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:17.258+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:17.670 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:17.669+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:18.253 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:18.252+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:18.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:18.691+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:19.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:19.249+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:19.708 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:19.707+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:20.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:20.295+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:20.717 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:20.715+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:21.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:21.298+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:21.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:21.756+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:22.256 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:22.255+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:22.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:22.724+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:23.113 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:18:23.117 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:18:23.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:23.259+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:23.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:23.682+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:24.261 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:24.260+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:24.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:24.650+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:25.230 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:25.229+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:25.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:25.663+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:26.122 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:18:26.121+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:18:26.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:26.210+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:26.655 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:26.654+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:27.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:27.174+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:27.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:27.643+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:28.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:28.176+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:28.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:28.610+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:29.144 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:29.142+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:29.577 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:29.576+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:30.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:30.116+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:30.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:30.611+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:31.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:31.077+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:31.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:31.614+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:32.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:32.045+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:32.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:32.638+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:33.064 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:33.063+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:33.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:33.597+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:34.108 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:34.108+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:34.629 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:34.628+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:35.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:35.077+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:35.664 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:35.663+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:36.077 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:36.076+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:36.622 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:36.621+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:37.105 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:37.103+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:37.625 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:37.624+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:38.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:38.116+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:38.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:38.668+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:39.069 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:39.068+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:39.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:39.709+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:40.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:40.085+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:40.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:40.737+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:41.099 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:41.099+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:41.138 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:18:41.136+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:18:41.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:41.786+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:42.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:42.096+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:42.784 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:42.783+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:43.129 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:43.128+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:43.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:43.786+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:44.115 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:44.114+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:44.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:44.750+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:45.152 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:45.152+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:45.762 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:45.761+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:46.109 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:46.108+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:46.792 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:46.791+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:47.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:47.084+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:47.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:47.756+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:48.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:48.060+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:48.797 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:48.796+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:49.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:49.041+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:49.798 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:49.797+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:50.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:50.025+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:50.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:50.769+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:51.009 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:51.008+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:51.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:51.751+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:52.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:52.049+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:52.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:52.728+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:53.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:53.065+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:53.268 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:18:53.272 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:18:53.774 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:53.773+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:54.098 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:54.098+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:54.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:54.750+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:55.084 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:55.083+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:55.798 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:55.797+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:56.119 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:56.118+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:56.154 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:18:56.153+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:18:56.781 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:56.781+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:57.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:57.097+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:57.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:57.733+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:58.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:58.078+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:58.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:58.747+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:18:59.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:18:59.078+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:18:59.766 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:18:59.765+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:00.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:00.074+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:00.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:00.729+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:01.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:01.081+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:01.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:01.765+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:02.065 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:02.063+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:02.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:02.805+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:03.057 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:03.055+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:03.761 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:03.760+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:04.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:04.065+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:04.754 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:04.753+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:05.021 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:05.020+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:05.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:05.720+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:06.017 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:06.015+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:06.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:06.681+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:07.022 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:07.021+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:07.659 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:07.658+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:08.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:08.066+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:08.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:08.663+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:09.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:09.110+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:09.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:09.623+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:10.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:10.061+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:10.625 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:10.623+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:11.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:11.040+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:11.170 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:19:11.168+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:19:11.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:11.652+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:12.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:12.014+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:12.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:12.646+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:12.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:12.996+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:13.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:13.684+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:14.002 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:14.000+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:14.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:14.717+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:15.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:15.003+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:15.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:15.708+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:16.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:16.007+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:16.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:16.678+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:16.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:16.991+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:17.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:17.685+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:17.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:17.982+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:18.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:18.733+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:19.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:19.024+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:19.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:19.749+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:20.043 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:20.042+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:20.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:20.753+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:21.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:21.080+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:21.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:21.712+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:22.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:22.060+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:22.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:22.668+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:23.086 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:23.085+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:23.348 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:19:23.352 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:19:23.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:23.706+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:24.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:24.046+0000 7fd3b190b700 -1 osd.6 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:24.125 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-03T20:19:24.124+0000 7f3110986700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T20:19:24.139 INFO:tasks.ceph.mds.f.smithi186.stderr: -1> 2022-04-03T20:19:24.124+0000 7f3110986700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T20:19:24.162 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-03T20:19:24.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:24.700+0000 7f88d0ac1700 -1 osd.4 119 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:25.090 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:25.089+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:25.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:25.651+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:26.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:26.107+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:26.185 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:19:26.183+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:19:26.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:26.612+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:27.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:27.075+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:27.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:27.597+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:28.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:28.037+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:28.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:28.635+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:29.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:29.003+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:29.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:29.681+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:30.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:30.046+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:30.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:30.701+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:31.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:31.006+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:31.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:31.715+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:32.017 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:32.016+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:32.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:32.747+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:33.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:33.025+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:33.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:33.733+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:34.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:34.015+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:34.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:34.713+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:35.011 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:35.009+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:35.691 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:35.691+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:36.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:36.046+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:36.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:36.645+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:37.020 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:37.020+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:37.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:37.666+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:37.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:37.972+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:38.710 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:38.709+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:38.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:38.951+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:39.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:39.710+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:39.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:39.951+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:40.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:40.701+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:40.954 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:40.954+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:41.201 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:19:41.200+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:19:41.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:41.657+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:41.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:41.966+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:42.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:42.683+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:42.972 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:42.972+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:43.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:43.696+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:44.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:44.015+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:44.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:44.735+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:45.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:45.054+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:45.775 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:45.774+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:46.032 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:46.031+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:46.746 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:46.745+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:47.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:47.061+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:47.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:47.749+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:48.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:48.110+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:48.761 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:48.761+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:49.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:49.083+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:49.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:49.719+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:50.118 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:50.118+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:50.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:50.736+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:51.099 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:51.098+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:51.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:51.700+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:52.102 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:52.102+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:52.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:52.714+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:53.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:53.130+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:53.421 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:19:53.425 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:19:53.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:53.665+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:54.098 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:54.097+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:54.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:54.616+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:55.058 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:55.057+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:55.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:55.574+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:56.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:56.040+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:56.217 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:19:56.216+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:19:56.546 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:56.545+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:57.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:57.037+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:57.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:57.531+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:58.056 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:58.054+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:58.519 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:58.518+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:19:59.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:19:59.036+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:19:59.510 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:19:59.509+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:00.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:00.033+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:00.506 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:00.504+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:01.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:01.048+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:01.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:01.530+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:02.092 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:02.091+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:02.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:02.528+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:03.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:03.053+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:03.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:03.546+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:04.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:04.087+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:04.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:04.572+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:05.110 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:05.109+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:05.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:05.587+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:06.092 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:06.090+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:06.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:06.595+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:07.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:07.073+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:07.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:07.562+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:08.119 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:08.118+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:08.523 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:08.522+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:09.140 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:09.139+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:09.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:09.568+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:10.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:10.124+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:10.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:10.553+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:11.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:11.094+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:11.233 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:20:11.232+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:20:11.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:11.520+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:12.078 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:12.077+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:12.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:12.496+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:13.043 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:13.042+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:13.471 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:13.469+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:14.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:14.025+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:14.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:14.511+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:14.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:14.988+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:15.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:15.526+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:16.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:16.024+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:16.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:16.514+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:17.043 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:17.041+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:17.522 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:17.521+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:18.032 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:18.032+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:18.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:18.524+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:19.022 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:19.021+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:19.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:19.531+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:20.064 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:20.062+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:20.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:20.555+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:21.092 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:21.091+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:21.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:21.561+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:22.109 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:22.108+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:22.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:22.524+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:23.102 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:23.101+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:23.486 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:23.485+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:23.561 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:20:23.565 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:20:24.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:24.095+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:24.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:24.511+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:25.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:25.125+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:25.502 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:25.500+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:26.146 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:26.145+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:26.248 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:20:26.247+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:20:26.472 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:26.471+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:27.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:27.153+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:27.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:27.460+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:28.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:28.121+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:28.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:28.508+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:29.108 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:29.106+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:29.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:29.523+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:30.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:30.073+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:30.526 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:30.525+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:31.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:31.038+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:31.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:31.562+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:32.078 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:32.077+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:32.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:32.592+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:33.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:33.034+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:33.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:33.630+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:34.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:34.061+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:34.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:34.630+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:35.060 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:35.059+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:35.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:35.599+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:36.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:36.071+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:36.607 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:36.606+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:37.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:37.116+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:37.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:37.597+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:38.149 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:38.149+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:38.609 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:38.609+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:39.155 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:39.155+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:39.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:39.610+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:40.148 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:40.148+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:40.656 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:40.655+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:41.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:41.198+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:41.264 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:20:41.263+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:20:41.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:41.701+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:42.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:42.228+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:42.661 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:42.661+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:43.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:43.183+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:43.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:43.679+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:44.145 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:44.145+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:44.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:44.630+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:45.186 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:45.186+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:45.602 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:45.602+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:46.221 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:46.220+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:46.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:46.595+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:47.239 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:47.238+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:47.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:47.605+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:48.222 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:48.222+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:48.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:48.595+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:49.258 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:49.258+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:49.573 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:49.572+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:50.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:50.226+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:50.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:50.528+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:51.184 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:51.183+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:51.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:51.514+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:52.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:52.197+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:52.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:52.525+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:53.157 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:53.156+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:53.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:53.493+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:53.708 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:20:53.712 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:20:54.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:54.182+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:54.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:54.526+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:55.162 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:55.161+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:55.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:55.558+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:56.208 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:56.207+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:56.274 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:20:56.273+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:20:56.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:56.541+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:57.200 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:57.199+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:57.508 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:57.507+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:58.162 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:58.161+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:58.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:58.515+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:20:59.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:20:59.140+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:20:59.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:20:59.513+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:00.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:00.130+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:00.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:00.486+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:01.176 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:01.175+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:01.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:01.526+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:02.133 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:02.132+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:02.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:02.480+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:03.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:03.181+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:03.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:03.491+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:04.190 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:04.189+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:04.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:04.526+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:05.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:05.178+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:05.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:05.511+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:06.222 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:06.220+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:06.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:06.487+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:07.209 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:07.208+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:07.486 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:07.484+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:08.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:08.174+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:08.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:08.457+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:09.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:09.196+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:09.475 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:09.474+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:10.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:10.188+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:10.522 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:10.520+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:11.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:11.234+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:11.290 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:21:11.289+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:21:11.540 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:11.538+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:12.256 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:12.254+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:12.540 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:12.539+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:13.255 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:13.254+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:13.558 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:13.557+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:14.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:14.257+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:14.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:14.564+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:15.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:15.283+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:15.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:15.596+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:16.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:16.286+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:16.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:16.638+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:17.267 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:17.265+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:17.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:17.635+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:18.303 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:18.302+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:18.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:18.615+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:19.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:19.319+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:19.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:19.573+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:20.322 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:20.320+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:20.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:20.612+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:21.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:21.315+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:21.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:21.577+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:22.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:22.305+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:22.570 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:22.569+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:23.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:23.295+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:23.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:23.546+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:23.839 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:21:23.843 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:21:24.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:24.264+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:24.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:24.535+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:25.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:25.309+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:25.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:25.553+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:26.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:26.296+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:26.306 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:21:26.305+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:21:26.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:26.561+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:27.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:27.339+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:27.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:27.592+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:28.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:28.298+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:28.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:28.570+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:29.348 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:29.348+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:29.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:29.593+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:30.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:30.390+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:30.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:30.556+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:31.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:31.405+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:31.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:31.533+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:32.432 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:32.432+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:32.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:32.541+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:33.410 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:33.410+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:33.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:33.530+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:34.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:34.421+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:34.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:34.559+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:35.465 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:35.465+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:35.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:35.588+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:36.457 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:36.456+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:36.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:36.598+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:37.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:37.447+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:37.617 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:37.617+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:38.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:38.427+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:38.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:38.636+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:39.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:39.429+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:39.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:39.686+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:40.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:40.447+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:40.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:40.653+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:41.308 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:21:41.306+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:21:41.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:41.413+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:41.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:41.658+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:42.432 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:42.431+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:42.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:42.694+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:43.472 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:43.470+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:43.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:43.721+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:44.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:44.493+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:44.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:44.741+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:45.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:45.458+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:45.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:45.737+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:46.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:46.495+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:46.788 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:46.787+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:47.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:47.490+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:47.804 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:47.803+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:48.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:48.484+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:48.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:48.766+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:49.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:49.472+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:49.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:49.809+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:50.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:50.514+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:50.854 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:50.852+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:51.478 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:51.477+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:51.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:51.871+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:52.452 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:52.451+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:52.885 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:52.884+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:53.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:53.460+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:53.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:53.842+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:53.995 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:21:53.999 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:21:54.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:54.414+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:54.851 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:54.850+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:55.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:55.412+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:55.827 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:55.826+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:56.323 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:21:56.322+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:21:56.449 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:56.448+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:56.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:56.866+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:57.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:57.457+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:57.857 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:57.855+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:58.462 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:58.461+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:58.904 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:58.902+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:21:59.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:21:59.474+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:21:59.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:21:59.895+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:00.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:00.460+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:00.910 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:00.908+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:01.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:01.491+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:01.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:01.944+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:02.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:02.535+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:02.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:02.947+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:03.582 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:03.580+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:03.960 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:03.958+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:04.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:04.570+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:04.921 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:04.920+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:05.524 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:05.523+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:05.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:05.896+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:06.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:06.548+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:06.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:06.877+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:07.507 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:07.506+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:07.917 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:07.916+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:08.499 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:08.498+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:08.907 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:08.906+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:09.508 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:09.506+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:09.861 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:09.860+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:10.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:10.543+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:10.905 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:10.905+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:11.339 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:22:11.338+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:22:11.585 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:11.584+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:11.903 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:11.902+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:12.563 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:12.562+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:12.950 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:12.949+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:13.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:13.517+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:13.921 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:13.919+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:14.559 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:14.557+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:14.950 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:14.949+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:15.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:15.587+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:15.951 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:15.950+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:16.604 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:16.603+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:16.905 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:16.905+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:17.606 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:17.605+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:17.926 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:17.925+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:18.593 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:18.592+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:18.941 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:18.940+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:19.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:19.613+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:19.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:19.944+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:20.615 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:20.614+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:20.934 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:20.933+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:21.626 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:21.626+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:21.940 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:21.939+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:22.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:22.615+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:22.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:22.908+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:23.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:23.572+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:23.954 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:23.954+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:24.117 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:22:24.120 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:22:24.566 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:24.566+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:24.969 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:24.969+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:25.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:25.543+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:25.937 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:25.936+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:26.355 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:22:26.354+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:22:26.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:26.528+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:26.925 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:26.925+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:27.547 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:27.547+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:27.906 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:27.905+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:28.547 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:28.547+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:28.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:28.932+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:29.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:29.542+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:29.911 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:29.911+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:30.522 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:30.522+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:30.880 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:30.880+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:31.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:31.494+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:31.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:31.833+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:32.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:32.459+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:32.859 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:32.858+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:33.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:33.414+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:33.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:33.833+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:34.420 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:34.420+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:34.876 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:34.876+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:35.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:35.405+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:35.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:35.871+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:36.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:36.375+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:36.921 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:36.919+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:37.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:37.384+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:37.970 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:37.968+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:38.397 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:38.396+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:38.959 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:38.958+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:39.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:39.422+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:39.912 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:39.910+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:40.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:40.392+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:40.949 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:40.947+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:41.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:41.361+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:41.371 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:22:41.370+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:22:41.946 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:41.945+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:42.350 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:42.349+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:42.985 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:42.983+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:43.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:43.373+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:44.034 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:44.033+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:44.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:44.376+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:45.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:45.054+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:45.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:45.405+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:46.043 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:46.041+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:46.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:46.411+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:47.010 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:47.008+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:47.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:47.457+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:47.991 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:47.990+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:48.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:48.416+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:48.951 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:48.949+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:49.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:49.437+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:49.973 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:49.972+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:50.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:50.478+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:51.023 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:51.022+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:51.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:51.446+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:52.044 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:52.043+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:52.496 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:52.494+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:53.046 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:53.043+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:53.472 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:53.470+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:54.008 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:54.006+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:54.365 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:22:54.368 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:22:54.465 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:54.463+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:55.035 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:55.033+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:55.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:55.458+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:56.036 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:56.034+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:56.381 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:22:56.378+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:22:56.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:56.462+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:57.022 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:57.020+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:57.446 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:57.444+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:58.053 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:58.050+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:58.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:58.404+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:22:59.101 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:22:59.099+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:22:59.359 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:22:59.357+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:00.100 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:00.098+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:00.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:00.380+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:01.120 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:01.118+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:01.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:01.366+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:02.165 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:02.163+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:02.332 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:02.330+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:03.191 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:03.189+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:03.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:03.307+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:04.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:04.184+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:04.306 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:04.304+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:05.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:05.157+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:05.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:05.326+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:06.191 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:06.189+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:06.300 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:06.298+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:07.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:07.229+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:07.252 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:07.251+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:08.209 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:08.207+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:08.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:08.243+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:09.245 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:09.243+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:09.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:09.255+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:10.230 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:10.228+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:10.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:10.236+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:11.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:11.244+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:11.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:11.255+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:11.396 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:23:11.395+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:23:12.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:12.244+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:12.283 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:12.281+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:13.221 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:13.219+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:13.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:13.255+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:14.210 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:14.208+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:14.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:14.271+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:15.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:15.229+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:15.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:15.284+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:16.239 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:16.237+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:16.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:16.304+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:17.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:17.276+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:17.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:17.314+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:18.290 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:18.288+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:18.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:18.327+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:19.312 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:19.309+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:19.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:19.359+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:20.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:20.322+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:20.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:20.349+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:21.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:21.296+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:21.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:21.366+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:22.329 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:22.326+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:22.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:22.343+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:23.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:23.317+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:23.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:23.331+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:24.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:24.350+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:24.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:24.370+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:24.437 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:23:24.440 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:23:25.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:25.337+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:25.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:25.396+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:26.364 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:26.361+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:26.370 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:26.368+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:26.401 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:23:26.400+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:23:27.376 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:27.373+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:27.387 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:27.385+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:28.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:28.362+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:28.400 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:28.397+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:29.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:29.368+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:29.398 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:29.396+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:30.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:30.350+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:30.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:30.445+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:31.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:31.376+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:31.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:31.428+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:32.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:32.376+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:32.448 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:32.445+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:33.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:33.340+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:33.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:33.406+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:34.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:34.307+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:34.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:34.425+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:35.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:35.351+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:35.436 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:35.434+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:36.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:36.392+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:36.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:36.434+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:37.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:37.345+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:37.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:37.440+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:38.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:38.370+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:38.400 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:38.398+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:39.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:39.358+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:39.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:39.365+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:40.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:40.383+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:40.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:40.395+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:41.397 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:41.395+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:41.413 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:23:41.411+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:23:41.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:41.418+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:42.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:42.351+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:42.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:42.437+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:43.332 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:43.330+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:43.484 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:43.482+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:44.303 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:44.301+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:44.486 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:44.484+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:45.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:45.280+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:45.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:45.449+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:46.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:46.314+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:46.478 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:46.475+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:47.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:47.284+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:47.510 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:47.508+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:48.269 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:48.267+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:48.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:48.479+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:49.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:49.283+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:49.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:49.526+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:50.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:50.303+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:50.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:50.527+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:51.290 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:51.288+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:51.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:51.518+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:52.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:52.249+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:52.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:52.489+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:53.269 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:53.267+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:53.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:53.513+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:54.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:54.217+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:54.551 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:23:54.555 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:23:54.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:54.562+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:55.185 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:55.182+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:55.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:55.537+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:56.190 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:56.188+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:56.430 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:23:56.428+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:23:56.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:56.514+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:57.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:57.152+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:57.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:57.532+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:58.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:58.145+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:58.566 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:58.564+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:23:59.105 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:23:59.103+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:23:59.550 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:23:59.548+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:00.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:00.086+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:00.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:00.586+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:01.130 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:01.128+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:01.585 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:01.583+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:02.100 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:02.098+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:02.552 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:02.550+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:03.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:03.126+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:03.526 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:03.524+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:04.106 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:04.104+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:04.552 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:04.550+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:05.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:05.127+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:05.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:05.568+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:06.146 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:06.144+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:06.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:06.533+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:07.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:07.105+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:07.566 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:07.562+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:08.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:08.120+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:08.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:08.536+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:09.129 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:09.127+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:09.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:09.534+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:10.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:10.145+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:10.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:10.546+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:11.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:11.110+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:11.432 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:24:11.430+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:24:11.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:11.543+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:12.096 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:12.094+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:12.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:12.536+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:13.078 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:13.076+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:13.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:13.553+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:14.051 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:14.049+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:14.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:14.527+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:15.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:15.001+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:15.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:15.495+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:16.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:16.039+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:16.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:16.479+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:16.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:16.995+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:17.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:17.492+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:18.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:18.004+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:18.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:18.521+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:18.963 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:18.961+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:19.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:19.485+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:19.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:19.924+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:20.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:20.446+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:20.932 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:20.929+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:21.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:21.401+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:21.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:21.945+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:22.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:22.371+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:22.929 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:22.926+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:23.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:23.366+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:23.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:23.935+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:24.415 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:24.413+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:24.698 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:24:24.702 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:24:24.976 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:24.974+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:25.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:25.395+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:26.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:26.012+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:26.434 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:26.431+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:26.448 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:24:26.445+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:24:27.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:27.052+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:27.396 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:27.393+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:28.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:28.023+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:28.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:28.350+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:29.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:29.047+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:29.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:29.392+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:30.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:30.023+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:30.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:30.346+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:31.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:31.001+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:31.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:31.318+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:31.961 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:31.958+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:32.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:32.269+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:32.978 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:32.976+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:33.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:33.260+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:33.969 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:33.967+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:34.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:34.298+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:34.944 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:34.941+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:35.283 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:35.281+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:35.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:35.926+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:36.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:36.271+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:36.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:36.891+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:37.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:37.229+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:37.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:37.916+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:38.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:38.182+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:38.880 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:38.878+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:39.226 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:39.223+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:39.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:39.866+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:40.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:40.208+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:40.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:40.875+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:41.165 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:41.163+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:41.464 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:24:41.462+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:24:41.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:41.879+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:42.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:42.177+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:42.895 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:42.893+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:43.138 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:43.135+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:43.919 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:43.917+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:44.102 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:44.100+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:44.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:44.940+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:45.059 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:45.057+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:45.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:45.962+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:46.102 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:46.100+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:46.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:46.986+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:47.071 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:47.069+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:48.009 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:48.007+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:48.072 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:48.069+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:48.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:48.971+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:49.029 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:49.026+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:49.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:49.948+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:50.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:50.010+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:50.945 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:50.943+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:51.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:51.001+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:51.972 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:51.969+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:51.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:51.986+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:53.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:53.014+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:53.031 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:53.029+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:54.042 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:54.040+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:54.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:54.074+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:54.846 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:24:54.850 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:24:55.017 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:55.015+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:55.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:55.061+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:56.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:56.001+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:56.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:56.072+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:56.480 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:24:56.478+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:24:57.042 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:57.040+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:57.113 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:57.111+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:58.051 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:58.049+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:58.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:58.130+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:24:59.065 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:24:59.064+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:24:59.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:24:59.106+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:00.037 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:00.036+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:00.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:00.141+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:01.015 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:01.014+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:01.104 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:01.102+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:02.027 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:02.026+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:02.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:02.120+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:03.044 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:03.043+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:03.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:03.137+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:04.056 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:04.054+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:04.116 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:04.114+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:05.051 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:05.049+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:05.110 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:05.108+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:06.026 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:06.025+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:06.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:06.081+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:07.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:07.001+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:07.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:07.095+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:08.013 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:08.012+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:08.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:08.084+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:09.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:09.036+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:09.044 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:09.042+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:10.036 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:10.034+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:10.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:10.045+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:11.019 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:11.017+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:11.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:11.024+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:11.488 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:25:11.485+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:25:12.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:12.007+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:12.050 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:12.047+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:13.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:13.026+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:13.030 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:13.028+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:14.051 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:14.048+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:14.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:14.051+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:15.006 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:15.003+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:15.039 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:15.036+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:16.043 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:16.041+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:16.062 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:16.059+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:17.025 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:17.022+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:17.072 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:17.070+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:18.021 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:18.019+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:18.044 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:18.041+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:19.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:19.052+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:19.070 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:19.068+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:20.065 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:20.063+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:20.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:20.083+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:21.025 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:21.022+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:21.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:21.052+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:22.021 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:22.019+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:22.101 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:22.099+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:22.986 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:22.983+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:23.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:23.139+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:23.989 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:23.986+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:24.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:24.174+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:24.978 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:24.975+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:25.010 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:25:25.013 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:25:25.177 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:25.175+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:25.971 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:25.968+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:26.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:26.205+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:26.504 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:25:26.502+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:25:26.967 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:26.965+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:27.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:27.157+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:27.936 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:27.933+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:28.209 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:28.207+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:28.914 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:28.911+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:29.247 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:29.244+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:29.865 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:29.862+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:30.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:30.294+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:30.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:30.886+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:31.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:31.332+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:31.865 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:31.863+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:32.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:32.283+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:32.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:32.886+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:33.307 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:33.305+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:33.880 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:33.877+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:34.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:34.349+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:34.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:34.887+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:35.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:35.319+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:35.840 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:35.838+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:36.356 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:36.354+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:36.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:36.880+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:37.338 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:37.336+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:37.871 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:37.869+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:38.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:38.290+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:38.847 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:38.845+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:39.263 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:39.261+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:39.886 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:39.884+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:40.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:40.258+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:40.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:40.894+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:41.280 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:41.278+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:41.520 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:25:41.518+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:25:41.849 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:41.847+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:42.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:42.229+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:42.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:42.800+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:43.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:43.258+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:43.817 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:43.815+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:44.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:44.294+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:44.801 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:44.799+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:45.337 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:45.335+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:45.848 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:45.847+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:46.293 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:46.291+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:46.858 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:46.856+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:47.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:47.322+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:47.831 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:47.829+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:48.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:48.363+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:48.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:48.792+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:49.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:25:49.349+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:25:49.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:49.840+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:50.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:50.876+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:51.902 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:51.900+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:52.935 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:52.934+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:53.887 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:53.885+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:54.898 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:54.896+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:55.149 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:25:55.153 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:25:55.871 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:55.870+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:56.523 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:25:56.519+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:25:56.846 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:56.845+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:57.812 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:57.810+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:58.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:58.775+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:25:59.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:25:59.737+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:00.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:00.714+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:01.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:01.701+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:02.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:02.656+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:03.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:03.633+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:04.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:04.675+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:05.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:05.720+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:06.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:06.740+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:07.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:07.747+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:08.745 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:08.742+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:09.705 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:09.703+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:10.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:10.712+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:11.539 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:26:11.536+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:26:11.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:11.726+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:12.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:12.680+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:13.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:13.715+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:14.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:14.730+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:15.687 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:15.684+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:16.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:16.683+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:17.673 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:17.671+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:18.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:18.636+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:19.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:19.587+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:20.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:20.551+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:21.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:21.556+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:22.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:22.567+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:23.610 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:23.608+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:24.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:24.594+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:25.271 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:26:25.276 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:26:25.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:25.591+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:26.550 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:26.547+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:26.554 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:26:26.552+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:26:27.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:27.573+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:28.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:28.568+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:29.582 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:29.580+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:30.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:30.597+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:31.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:31.555+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:32.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:32.549+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:33.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:33.565+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:34.607 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:34.604+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:35.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:35.560+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:36.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:36.529+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:37.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:37.528+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:38.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:38.480+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:39.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:39.485+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:40.454 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:40.452+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:41.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:41.462+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:41.570 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:26:41.567+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:26:42.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:42.414+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:43.451 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:43.449+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:44.402 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:44.400+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:45.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:45.385+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:46.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:46.393+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:47.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:47.419+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:48.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:48.456+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:49.500 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:49.498+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:50.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:50.475+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:51.437 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:51.435+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:52.426 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:52.424+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:53.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:53.383+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:54.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:54.388+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:55.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:55.396+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:55.413 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:26:55.417 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:26:56.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:56.381+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:56.573 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:26:56.570+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:26:57.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:57.343+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:58.332 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:58.330+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:26:59.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:26:59.361+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:00.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:00.402+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:01.423 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:01.421+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:02.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:02.434+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:03.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:03.446+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:04.464 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:04.461+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:05.457 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:05.454+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:06.450 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:06.447+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:07.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:07.453+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:08.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:08.409+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:09.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:09.394+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:10.422 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:10.419+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:11.466 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:11.464+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:11.588 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:27:11.586+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:27:12.418 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:12.415+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:13.450 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:13.447+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:14.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:14.422+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:15.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:15.415+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:16.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:16.449+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:17.423 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:17.420+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:18.400 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:18.398+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:19.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:19.401+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:20.446 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:20.444+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:21.423 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:21.420+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:22.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:22.466+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:23.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:23.418+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:24.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:24.442+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:25.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:25.480+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:25.787 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:27:25.791 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:27:26.523 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:26.521+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:26.599 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:27:26.596+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:27:27.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:27.528+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:28.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:28.492+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:29.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:29.489+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:30.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:30.486+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:31.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:31.443+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:32.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:32.434+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:33.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:33.453+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:34.500 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:34.498+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:35.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:35.456+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:36.409 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:36.407+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:37.418 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:37.416+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:38.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:38.459+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:39.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:39.449+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:40.472 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:40.470+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:41.432 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:41.430+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:41.615 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:27:41.613+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:27:42.479 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:42.477+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:43.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:43.440+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:44.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:44.418+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:45.453 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:45.452+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:46.502 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:46.498+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:47.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:47.525+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:48.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:48.503+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:49.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:49.530+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:50.577 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:50.575+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:51.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:51.590+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:52.600 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:52.598+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:53.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:53.647+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:54.642 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:54.640+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:55.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:55.683+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:55.918 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:27:55.922 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:27:56.630 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:27:56.627+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:27:56.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:56.670+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:57.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:57.646+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:58.642 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:58.640+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:27:59.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:27:59.635+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:00.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:00.661+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:01.655 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:01.653+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:02.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:02.669+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:03.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:03.627+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:04.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:04.586+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:05.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:05.568+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:06.570 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:06.568+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:07.607 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:07.604+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:08.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:08.595+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:09.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:09.553+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:10.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:10.553+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:11.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:11.573+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:11.645 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:28:11.642+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:28:12.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:12.615+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:13.621 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:13.618+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:14.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:14.651+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:15.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:15.682+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:16.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:16.641+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:17.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:17.660+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:18.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:18.667+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:19.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:19.709+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:20.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:20.748+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:21.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:21.740+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:22.762 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:22.759+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:23.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:23.732+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:24.744 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:24.742+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:25.761 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:25.759+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:26.075 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:28:26.078 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:28:26.661 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:28:26.659+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:28:26.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:26.803+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:27.794 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:27.792+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:28.761 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:28.759+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:29.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:29.745+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:30.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:30.792+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:31.823 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:31.821+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:32.852 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:32.850+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:33.825 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:33.824+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:34.840 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:34.838+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:35.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:35.870+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:36.860 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:36.859+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:37.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:37.814+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:38.850 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:38.849+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:39.805 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:39.804+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:40.803 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:40.801+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:41.662 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:28:41.660+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:28:41.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:41.784+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:42.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:42.746+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:43.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:43.716+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:44.735 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:44.733+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:45.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:45.700+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:46.712 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:46.711+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:47.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:47.749+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:48.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:48.713+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:49.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:49.748+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:50.796 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:50.794+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:51.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:28:51.013+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:28:51.808 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:51.805+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:52.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:28:52.010+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:28:52.821 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:52.818+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:53.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:28:53.001+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:28:53.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:53.861+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:53.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:28:53.992+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:28:54.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:54.875+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:54.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:28:54.990+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:28:55.910 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:55.908+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:56.009 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:28:56.007+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:28:56.138 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:28:56.141 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:28:56.678 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:28:56.675+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:28:56.942 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:56.940+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:56.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:28:56.961+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:28:57.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:28:57.964+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:28:57.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:57.989+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:58.949 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:58.946+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:28:58.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:28:58.946+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:28:59.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:28:59.900+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:28:59.954 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:28:59.952+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:00.891 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:00.888+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:00.942 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:00.939+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:01.892 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:01.890+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:01.972 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:01.970+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:02.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:02.925+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:02.949 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:02.947+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:03.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:03.913+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:03.997 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:03.995+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:04.889 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:04.886+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:04.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:04.945+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:05.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:05.930+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:05.962 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:05.960+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:06.905 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:06.902+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:06.994 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:06.991+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:07.859 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:07.857+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:08.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:08.013+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:08.883 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:08.880+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:09.011 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:09.009+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:09.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:09.837+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:10.000 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:09.996+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:10.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:10.883+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:11.045 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:11.043+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:11.694 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:29:11.692+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:29:11.915 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:11.913+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:12.021 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:12.018+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:12.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:12.876+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:13.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:13.010+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:13.470 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-03T20:29:13.467+0000 7f2fd761c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T20:29:13.470 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-03T20:29:13.467+0000 7f2fd661a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T20:29:13.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:13.880+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:14.008 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:14.005+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:14.883 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:14.881+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:15.028 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:15.026+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:15.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:15.870+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:16.011 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:16.009+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:16.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:16.836+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:16.973 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:16.970+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:17.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:17.837+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:17.946 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:17.943+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:18.826 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:18.823+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:18.934 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:18.932+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:19.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:19.858+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:19.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:19.931+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:20.844 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:20.842+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:20.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:20.977+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:21.867 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:21.864+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:22.025 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:22.022+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:22.895 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:22.892+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:23.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:23.036+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:23.867 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:23.864+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:24.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:24.002+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:24.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:24.875+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:25.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:25.001+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:25.908 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:25.906+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:26.034 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:26.032+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:26.269 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:29:26.273 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:29:26.696 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:29:26.693+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:29:26.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:26.926+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:27.069 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:27.066+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:27.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:27.948+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:28.080 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:28.078+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:28.925 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:28.923+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:29.031 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:29.029+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:29.968 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:29.966+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:30.011 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:30.009+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:30.969 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:30.967+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:31.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:31.008+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:31.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:31.931+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:32.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:32.022+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:32.903 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:32.902+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:33.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:33.038+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:33.927 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:33.925+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:34.034 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:34.033+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:34.906 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:34.904+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:35.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:35.080+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:35.925 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:35.923+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:36.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:36.059+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:36.951 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:36.949+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:37.060 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:37.059+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:37.976 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:37.974+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:38.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:38.080+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:39.000 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:38.999+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:39.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:39.060+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:40.020 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:40.018+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:40.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:40.024+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:40.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:40.987+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:41.001 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:41.000+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:41.711 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:29:41.708+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:29:41.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:41.952+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:42.020 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:42.018+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:42.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:42.984+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:43.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:43.002+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:44.029 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:44.028+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:44.050 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:44.049+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:45.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:45.047+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:45.063 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:45.061+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:46.023 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:46.022+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:46.097 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:46.095+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:46.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:46.982+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:47.074 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:47.072+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:47.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:47.970+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:48.056 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:48.053+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:49.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:49.006+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:49.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:49.014+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:50.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:50.001+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:50.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:50.038+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:51.047 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:51.045+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:51.057 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:51.055+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:52.030 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:52.028+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:52.076 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:52.074+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:53.017 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:53.015+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:53.043 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:53.041+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:54.047 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:54.045+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:54.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:54.058+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:55.044 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:55.041+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:55.058 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:55.056+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:56.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:56.065+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:56.088 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:56.086+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:56.400 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:29:56.404 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:29:56.727 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:29:56.725+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:29:57.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:57.046+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:57.062 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:57.059+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:58.043 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:58.041+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:29:58.052 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:58.050+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:59.045 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:29:59.042+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:29:59.057 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:29:59.055+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:00.058 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:00.055+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:00.064 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:00.062+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:01.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:01.046+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:01.077 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:01.075+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:02.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:02.044+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:02.124 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:02.122+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:03.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:03.085+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:03.136 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:03.134+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:04.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:04.119+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:04.122 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:04.119+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:05.096 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:05.093+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:05.114 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:05.112+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:06.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:06.118+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:06.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:06.161+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:07.098 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:07.095+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:07.143 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:07.141+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:08.137 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:08.135+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:08.191 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:08.189+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:09.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:09.148+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:09.151 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:09.149+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:10.123 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:10.121+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:10.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:10.145+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:11.103 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:11.101+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:11.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:11.153+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:11.733 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:30:11.730+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:30:12.140 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:12.138+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:12.147 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:12.145+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:13.145 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:13.143+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:13.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:13.157+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:14.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:14.129+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:14.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:14.186+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:15.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:15.125+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:15.146 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:15.144+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:16.130 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:16.128+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:16.168 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:16.166+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:17.139 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:17.137+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:17.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:17.177+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:18.145 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:18.143+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:18.160 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:18.158+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:19.127 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:19.125+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:19.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:19.169+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:20.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:20.161+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:20.176 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:20.174+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:21.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:21.121+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:21.139 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:21.137+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:22.152 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:22.150+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:22.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:22.162+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:23.172 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:23.170+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:23.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:23.176+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:24.213 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:24.211+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:24.226 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:24.225+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:25.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:25.177+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:25.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:25.235+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:26.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:26.199+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:26.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:26.242+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:26.481 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:30:26.485 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:30:26.748 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:30:26.745+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:30:27.223 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:27.221+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:27.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:27.268+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:28.233 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:28.232+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:28.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:28.280+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:29.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:29.257+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:29.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:29.278+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:30.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:30.247+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:30.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:30.307+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:31.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:31.295+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:31.301 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:31.300+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:32.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:32.274+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:32.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:32.286+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:33.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:33.251+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:33.264 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:33.263+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:34.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:34.230+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:34.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:34.271+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:35.240 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:35.239+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:35.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:35.278+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:36.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:36.255+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:36.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:36.318+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:37.217 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:37.215+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:37.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:37.295+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:38.184 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:38.182+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:38.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:38.333+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:39.150 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:39.148+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:39.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:39.315+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:40.115 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:40.113+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:40.332 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:40.331+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:41.100 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:41.098+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:41.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:41.377+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:41.764 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:30:41.762+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:30:42.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:42.136+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:42.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:42.380+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:43.150 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:43.148+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:43.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:43.377+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:44.110 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:44.107+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:44.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:44.414+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:45.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:45.145+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:45.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:45.445+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:46.130 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:46.128+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:46.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:46.483+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:47.108 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:47.105+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:47.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:47.445+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:48.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:48.119+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:48.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:48.405+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:49.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:49.169+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:49.375 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:49.372+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:50.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:50.171+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:50.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:50.331+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:51.193 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:51.191+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:51.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:51.341+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:52.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:52.157+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:52.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:52.340+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:53.150 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:53.148+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:53.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:53.372+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:54.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:54.172+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:54.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:54.362+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:55.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:55.213+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:55.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:55.329+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:56.184 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:56.182+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:56.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:56.334+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:56.629 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:30:56.633 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:30:56.780 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:30:56.778+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:30:57.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:57.196+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:57.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:57.348+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:58.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:58.199+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:58.356 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:58.354+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:30:59.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:30:59.186+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:30:59.337 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:30:59.334+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:00.213 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:00.211+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:00.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:00.316+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:01.251 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:01.249+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:01.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:01.334+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:02.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:02.296+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:02.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:02.296+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:03.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:03.311+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:03.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:03.331+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:04.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:04.295+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:04.333 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:04.331+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:05.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:05.292+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:05.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:05.320+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:06.329 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:06.327+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:06.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:06.341+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:07.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:07.281+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:07.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:07.318+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:08.334 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:08.332+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:08.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:08.356+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:09.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:09.328+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:09.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:09.401+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:10.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:10.312+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:10.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:10.382+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:11.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:11.262+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:11.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:11.401+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:11.796 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:31:11.793+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:31:12.216 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:12.214+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:12.396 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:12.394+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:13.255 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:13.252+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:13.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:13.387+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:14.253 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:14.251+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:14.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:14.366+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:15.263 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:15.261+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:15.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:15.392+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:16.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:16.303+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:16.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:16.379+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:17.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:17.332+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:17.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:17.338+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:18.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:18.295+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:18.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:18.375+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:19.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:19.247+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:19.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:19.417+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:20.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:20.201+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:20.408 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:20.405+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:21.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:21.210+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:21.450 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:21.449+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:22.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:22.205+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:22.496 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:22.495+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:23.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:23.239+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:23.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:23.521+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:24.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:24.234+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:24.517 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:24.515+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:25.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:25.256+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:25.498 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:25.497+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:26.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:26.222+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:26.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:26.481+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:26.777 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:31:26.782 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:31:26.811 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:31:26.809+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:31:27.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:27.186+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:27.466 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:27.464+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:28.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:28.163+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:28.450 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:28.448+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:29.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:29.207+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:29.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:29.428+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:30.206 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:30.204+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:30.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:30.443+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:31.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:31.159+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:31.441 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:31.439+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:32.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:32.208+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:32.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:32.429+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:33.206 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:33.204+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:33.469 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:33.467+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:34.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:34.229+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:34.490 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:34.489+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:35.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:35.190+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:35.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:35.452+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:36.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:36.151+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:36.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:36.486+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:37.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:37.145+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:37.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:37.531+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:38.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:38.183+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:38.525 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:38.523+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:39.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:39.197+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:39.570 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:39.567+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:40.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:40.177+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:40.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:40.578+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:41.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:41.206+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:41.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:41.585+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:41.827 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:31:41.825+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:31:42.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:42.212+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:42.593 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:42.590+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:43.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:43.228+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:43.639 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:43.636+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:44.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:44.256+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:44.599 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:44.596+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:45.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:45.248+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:45.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:45.569+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:46.233 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:46.231+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:46.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:46.573+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:47.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:47.273+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:47.527 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:47.524+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:48.235 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:48.232+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:48.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:48.532+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:49.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:49.232+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:49.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:49.529+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:50.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:50.257+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:50.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:50.511+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:51.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:51.228+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:51.526 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:51.523+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:52.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:52.196+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:52.570 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:52.567+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:53.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:53.241+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:53.547 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:53.545+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:54.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:54.279+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:54.563 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:54.561+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:55.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:55.270+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:55.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:55.568+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:56.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:56.289+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:56.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:56.543+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:56.843 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:31:56.840+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:31:56.925 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:31:56.929 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:31:57.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:57.308+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:57.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:57.521+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:58.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:58.350+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:58.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:58.532+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:31:59.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:31:59.315+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:31:59.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:31:59.579+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:00.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:00.302+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:00.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:00.581+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:01.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:01.334+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:01.556 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:01.554+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:02.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:02.325+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:02.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:02.558+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:03.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:03.360+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:03.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:03.600+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:04.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:04.355+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:04.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:04.592+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:05.402 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:05.399+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:05.593 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:05.591+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:06.431 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:06.429+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:06.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:06.549+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:07.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:07.441+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:07.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:07.526+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:08.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:08.474+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:08.575 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:08.573+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:09.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:09.425+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:09.618 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:09.615+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:10.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:10.438+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:10.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:10.585+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:11.453 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:11.451+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:11.617 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:11.615+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:11.845 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:32:11.843+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:32:12.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:12.450+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:12.630 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:12.628+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:13.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:13.412+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:13.610 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:13.608+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:14.421 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:14.419+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:14.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:14.585+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:15.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:15.393+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:15.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:15.619+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:16.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:16.439+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:16.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:16.669+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:17.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:17.415+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:17.624 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:17.623+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:18.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:18.453+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:18.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:18.654+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:19.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:19.438+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:19.666 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:19.665+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:20.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:20.415+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:20.692 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:20.691+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:21.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:21.382+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:21.681 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:21.679+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:22.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:22.388+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:22.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:22.693+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:23.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:23.435+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:23.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:23.656+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:24.402 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:24.400+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:24.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:24.619+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:25.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:25.450+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:25.633 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:25.631+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:26.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:26.408+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:26.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:26.604+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:26.861 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:32:26.859+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:32:27.072 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:32:27.076 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:32:27.431 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:27.430+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:27.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:27.627+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:28.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:28.384+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:28.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:28.601+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:29.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:29.359+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:29.590 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:29.588+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:30.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:30.402+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:30.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:30.544+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:31.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:31.381+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:31.590 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:31.588+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:32.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:32.357+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:32.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:32.542+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:33.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:33.368+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:33.519 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:33.516+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:34.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:34.404+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:34.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:34.511+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:35.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:35.436+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:35.478 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:35.475+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:36.418 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:36.415+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:36.527 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:36.525+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:37.375 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:37.372+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:37.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:37.539+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:38.415 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:38.413+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:38.573 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:38.570+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:39.400 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:39.397+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:39.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:39.614+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:40.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:40.357+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:40.573 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:40.570+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:41.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:41.392+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:41.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:41.535+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:41.877 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:32:41.874+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:32:42.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:42.351+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:42.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:42.489+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:43.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:43.322+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:43.498 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:43.495+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:44.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:44.345+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:44.547 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:44.544+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:45.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:45.369+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:45.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:45.541+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:46.401 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:46.398+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:46.585 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:46.582+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:47.375 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:47.373+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:47.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:47.592+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:48.402 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:48.399+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:48.633 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:48.631+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:49.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:49.365+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:49.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:49.589+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:50.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:50.388+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:50.619 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:50.617+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:51.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:51.368+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:51.596 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:51.594+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:52.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:52.382+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:52.550 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:52.548+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:53.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:53.387+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:53.558 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:53.556+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:54.412 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:54.410+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:54.547 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:54.545+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:55.402 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:55.400+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:55.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:55.532+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:56.382 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:56.379+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:56.555 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:56.552+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:56.892 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:32:56.890+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:32:57.211 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:32:57.214 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:32:57.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:57.380+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:57.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:57.521+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:58.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:58.384+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:58.564 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:58.562+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:32:59.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:32:59.396+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:32:59.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:32:59.519+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:00.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:00.431+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:00.490 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:00.488+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:01.471 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:01.469+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:01.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:01.486+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:02.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:02.486+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:02.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:02.493+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:03.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:03.487+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:03.517 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:03.514+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:04.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:04.458+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:04.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:04.486+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:05.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:05.422+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:05.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:05.465+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:06.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:06.409+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:06.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:06.431+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:07.396 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:07.394+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:07.402 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:07.400+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:08.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:08.397+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:08.407 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:08.405+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:09.379 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:09.377+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:09.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:09.416+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:10.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:10.350+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:10.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:10.370+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:11.343 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:11.341+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:11.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:11.371+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:11.908 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:33:11.906+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:33:12.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:12.307+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:12.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:12.379+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:13.274 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:13.272+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:13.418 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:13.417+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:14.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:14.278+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:14.413 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:14.411+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:15.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:15.294+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:15.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:15.415+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:16.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:16.300+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:16.464 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:16.463+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:17.267 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:17.266+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:17.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:17.454+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:18.301 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:18.300+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:18.413 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:18.412+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:19.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:19.285+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:19.401 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:19.400+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:20.293 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:20.292+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:20.432 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:20.430+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:21.343 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:21.342+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:21.434 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:21.432+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:22.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:22.312+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:22.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:22.406+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:23.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:23.339+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:23.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:23.438+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:24.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:24.360+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:24.453 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:24.451+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:25.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:25.334+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:25.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:25.406+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:26.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:26.375+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:26.396 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:26.393+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:26.924 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:33:26.922+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:33:27.367 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:33:27.371 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:33:27.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:27.375+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:27.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:27.377+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:28.386 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:28.383+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:28.426 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:28.423+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:29.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:29.420+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:29.432 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:29.430+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:30.426 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:30.423+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:30.466 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:30.464+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:31.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:31.390+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:31.448 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:31.445+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:32.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:32.342+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:32.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:32.461+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:33.376 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:33.373+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:33.457 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:33.454+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:34.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:34.347+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:34.480 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:34.477+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:35.376 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:35.374+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:35.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:35.511+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:36.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:36.382+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:36.530 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:36.527+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:37.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:37.349+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:37.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:37.490+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:38.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:38.385+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:38.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:38.489+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:39.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:39.414+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:39.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:39.486+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:40.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:40.445+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:40.509 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:40.506+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:41.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:41.459+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:41.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:41.491+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:41.940 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:33:41.937+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:33:42.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:42.493+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:42.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:42.496+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:43.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:43.489+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:43.506 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:43.504+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:44.463 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:44.461+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:44.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:44.532+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:45.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:45.446+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:45.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:45.520+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:46.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:46.444+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:46.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:46.489+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:47.424 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:47.422+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:47.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:47.511+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:48.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:48.388+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:48.563 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:48.561+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:49.429 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:49.427+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:49.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:49.542+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:50.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:50.395+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:50.564 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:50.562+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:51.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:51.428+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:51.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:51.579+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:52.418 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:52.415+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:52.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:52.610+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:53.437 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:53.435+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:53.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:53.569+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:54.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:54.434+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:54.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:54.544+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:55.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:55.390+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:55.533 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:55.531+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:56.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:56.390+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:56.490 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:56.489+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:56.955 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:33:56.953+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:33:57.424 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:57.422+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:57.515 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:33:57.519 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:33:57.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:57.530+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:58.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:58.415+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:58.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:58.534+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:33:59.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:33:59.417+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:33:59.527 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:33:59.525+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:00.451 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:00.449+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:00.562 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:00.560+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:01.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:01.408+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:01.608 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:01.606+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:02.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:02.372+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:02.604 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:02.602+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:03.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:03.336+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:03.568 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:03.566+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:04.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:04.291+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:04.533 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:04.531+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:05.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:05.328+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:05.547 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:05.545+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:06.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:06.298+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:06.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:06.501+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:07.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:07.313+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:07.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:07.482+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:08.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:08.316+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:08.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:08.519+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:09.284 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:09.282+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:09.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:09.474+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:10.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:10.251+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:10.471 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:10.469+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:11.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:11.298+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:11.473 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:11.471+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:11.971 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:34:11.969+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:34:12.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:12.281+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:12.499 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:12.497+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:13.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:13.288+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:13.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:13.487+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:14.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:14.318+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:14.530 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:14.528+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:15.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:15.289+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:15.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:15.495+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:16.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:16.248+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:16.475 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:16.473+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:17.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:17.253+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:17.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:17.441+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:18.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:18.205+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:18.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:18.434+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:19.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:19.171+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:19.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:19.409+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:20.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:20.182+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:20.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:20.450+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:21.146 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:21.144+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:21.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:21.489+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:22.114 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:22.111+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:22.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:22.483+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:23.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:23.152+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:23.509 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:23.506+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:24.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:24.195+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:24.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:24.540+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:25.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:25.187+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:25.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:25.536+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:26.144 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:26.142+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:26.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:26.510+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:26.987 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:34:26.985+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:34:27.102 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:27.099+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:27.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:27.508+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:27.603 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:34:27.607 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:34:28.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:28.070+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:28.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:28.535+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:29.028 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:29.025+0000 7f88d0ac1700 -1 osd.4 120 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:29.441 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-03T20:34:29.438+0000 7f887e26c700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T20:34:29.456 INFO:tasks.ceph.mds.f.smithi186.stderr: -1> 2022-04-03T20:34:29.438+0000 7f887e26c700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T20:34:29.478 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-03T20:34:29.563 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:29.560+0000 7fd3b190b700 -1 osd.6 120 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:29.993 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:29.991+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:30.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:30.569+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:31.006 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:31.004+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:31.561 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:31.558+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:32.033 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:32.031+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:32.598 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:32.595+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:33.062 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:33.059+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:33.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:33.547+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:34.040 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:34.037+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:34.508 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:34.505+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:35.015 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:35.012+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:35.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:35.489+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:35.980 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:35.977+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:36.473 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:36.471+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:36.988 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:36.985+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:37.424 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:37.422+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:38.029 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:38.027+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:38.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:38.433+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:39.076 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:39.073+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:39.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:39.403+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:40.071 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:40.069+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:40.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:40.388+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:41.111 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:41.108+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:41.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:41.350+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:41.993 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:34:41.991+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:34:42.092 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:42.090+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:42.313 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:42.311+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:43.111 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:43.108+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:43.266 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:43.263+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:44.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:44.147+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:44.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:44.289+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:45.191 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:45.189+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:45.303 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:45.301+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:46.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:46.171+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:46.342 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:46.339+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:47.139 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:47.137+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:47.343 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:47.340+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:48.129 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:48.127+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:48.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:48.365+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:49.171 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:49.168+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:49.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:49.376+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:50.165 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:50.162+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:50.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:50.403+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:51.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:51.162+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:51.365 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:51.362+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:52.144 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:52.142+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:52.346 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:52.344+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:53.152 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:53.150+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:53.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:53.379+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:54.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:54.109+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:54.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:54.416+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:55.145 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:55.143+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:55.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:55.405+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:56.130 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:56.128+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:56.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:56.359+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:57.009 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:34:57.007+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:34:57.149 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:57.147+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:57.398 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:57.396+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:57.751 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:34:57.755 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:34:58.138 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:58.136+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:58.358 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:58.355+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:34:59.100 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:34:59.098+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:34:59.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:34:59.326+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:00.125 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:00.123+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:00.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:00.279+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:01.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:01.161+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:01.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:01.263+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:02.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:02.202+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:02.220 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:02.219+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:03.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:03.166+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:03.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:03.209+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:04.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:04.184+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:04.242 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:04.240+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:05.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:05.214+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:05.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:05.223+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:06.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:06.185+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:06.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:06.229+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:07.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:07.141+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:07.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:07.210+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:08.113 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:08.111+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:08.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:08.246+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:09.091 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:09.090+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:09.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:09.223+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:10.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:10.082+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:10.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:10.272+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:11.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:11.064+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:11.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:11.313+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:12.025 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:35:12.023+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:35:12.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:12.046+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:12.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:12.284+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:13.058 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:13.057+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:13.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:13.262+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:14.032 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:14.030+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:14.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:14.287+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:14.462 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T20:35:14.459+0000 7fd623a1a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T20:35:14.462 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T20:35:14.459+0000 7fd62421b700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T20:35:15.001 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:15.000+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:15.271 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:15.270+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:15.975 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:15.973+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:16.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:16.239+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:16.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:16.974+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:17.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:17.287+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:18.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:18.002+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:18.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:18.323+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:18.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:18.993+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:19.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:19.334+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:19.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:19.964+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:20.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:20.348+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:20.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:20.946+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:21.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:21.363+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:21.980 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:21.977+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:22.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:22.355+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:23.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:23.023+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:23.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:23.355+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:24.042 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:24.039+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:24.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:24.307+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:24.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:24.992+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:25.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:25.331+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:26.018 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:26.015+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:26.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:26.352+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:27.009 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:27.006+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:27.074 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:35:27.039+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:35:27.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:27.396+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:27.890 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:35:27.894 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:35:28.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:28.002+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:28.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:28.390+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:28.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:28.977+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:29.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:29.355+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:29.962 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:29.960+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:30.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:30.340+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:30.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:30.947+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:31.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:31.317+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:31.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:31.939+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:32.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:32.347+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:32.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:32.982+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:33.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:33.378+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:34.014 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:34.012+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:34.401 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:34.399+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:35.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:35.057+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:35.415 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:35.412+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:36.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:36.032+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:36.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:36.431+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:37.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:37.052+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:37.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:37.424+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:38.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:38.046+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:38.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:38.392+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:39.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:39.025+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:39.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:39.377+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:40.069 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:40.067+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:40.337 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:40.335+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:41.108 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:41.105+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:41.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:41.364+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:42.057 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:35:42.054+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:35:42.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:42.078+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:42.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:42.342+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:43.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:43.060+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:43.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:43.312+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:44.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:44.026+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:44.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:44.322+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:45.029 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:45.026+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:45.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:45.332+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:46.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:46.022+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:46.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:46.324+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:46.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:46.987+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:47.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:47.358+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:48.014 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:48.012+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:48.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:48.367+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:48.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:48.993+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:49.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:49.387+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:50.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:50.002+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:50.409 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:50.407+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:50.971 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:50.969+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:51.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:51.402+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:52.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:52.005+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:52.423 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:52.421+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:52.996 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:52.994+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:53.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:53.382+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:54.014 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:54.012+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:54.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:54.375+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:55.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:55.047+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:55.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:55.365+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:56.078 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:56.076+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:56.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:56.317+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:57.072 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:35:57.070+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:35:57.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:57.124+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:57.284 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:57.283+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:58.021 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:35:58.025 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:35:58.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:58.115+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:58.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:58.235+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:35:59.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:35:59.158+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:35:59.212 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:35:59.210+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:00.193 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:00.191+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:00.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:00.214+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:01.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:01.191+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:01.240 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:01.239+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:02.220 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:02.218+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:02.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:02.230+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:03.184 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:03.182+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:03.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:03.192+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:04.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:04.159+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:04.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:04.214+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:05.154 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:05.153+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:05.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:05.170+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:06.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:06.132+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:06.182 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:06.181+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:07.145 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:07.143+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:07.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:07.162+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:08.154 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:08.153+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:08.191 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:08.189+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:09.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:09.167+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:09.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:09.196+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:10.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:10.175+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:10.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:10.187+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:11.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:11.184+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:11.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:11.187+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:12.088 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:36:12.086+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:36:12.144 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:12.141+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:12.150 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:12.148+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:13.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:13.115+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:13.119 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:13.116+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:14.099 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:14.096+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:14.106 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:14.103+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:15.104 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:15.102+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:15.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:15.120+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:16.129 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:16.127+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:16.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:16.141+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:17.172 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:17.170+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:17.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:17.173+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:18.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:18.124+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:18.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:18.208+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:19.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:19.161+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:19.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:19.196+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:20.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:20.153+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:20.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:20.181+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:21.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:21.150+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:21.204 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:21.201+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:22.131 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:22.128+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:22.217 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:22.214+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:23.122 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:23.120+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:23.193 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:23.190+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:24.097 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:24.095+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:24.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:24.198+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:25.099 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:25.096+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:25.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:25.163+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:26.094 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:26.092+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:26.119 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:26.117+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:27.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:27.072+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:27.102 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:27.100+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:27.104 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:36:27.102+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:36:28.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:28.080+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:28.088 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:28.085+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:28.152 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:36:28.155 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:36:29.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:29.081+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:29.128 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:29.126+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:30.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:30.076+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:30.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:30.145+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:31.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:31.126+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:31.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:31.188+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:32.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:32.144+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:32.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:32.154+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:33.137 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:33.135+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:33.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:33.178+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:34.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:34.115+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:34.213 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:34.211+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:35.145 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:35.143+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:35.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:35.177+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:36.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:36.126+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:36.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:36.173+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:37.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:37.156+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:37.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:37.188+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:38.109 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:38.107+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:38.206 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:38.204+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:39.137 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:39.134+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:39.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:39.196+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:40.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:40.154+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:40.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:40.160+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:41.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:41.185+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:41.206 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:41.204+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:42.120 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:36:42.118+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:36:42.206 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:42.204+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:42.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:42.226+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:43.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:43.252+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:43.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:43.272+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:44.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:44.209+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:44.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:44.242+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:45.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:45.198+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:45.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:45.259+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:46.157 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:46.155+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:46.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:46.226+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:47.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:47.172+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:47.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:47.255+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:48.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:48.181+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:48.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:48.236+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:49.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:49.134+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:49.202 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:49.200+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:50.151 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:50.149+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:50.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:50.162+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:51.151 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:51.149+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:51.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:51.182+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:52.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:52.189+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:52.191 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:52.190+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:53.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:53.237+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:53.241 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:53.240+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:54.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:54.192+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:54.243 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:54.242+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:55.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:55.223+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:55.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:55.225+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:56.182 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:56.180+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:56.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:56.181+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:57.136 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:36:57.133+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:36:57.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:57.172+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:57.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:57.195+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:58.180 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:58.178+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:58.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:58.235+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:36:58.274 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:36:58.278 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:36:59.214 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:36:59.213+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:36:59.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:36:59.225+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:00.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:00.186+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:00.205 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:00.204+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:01.139 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:01.137+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:01.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:01.253+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:02.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:02.165+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:02.236 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:02.234+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:03.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:03.202+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:03.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:03.256+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:04.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:04.161+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:04.241 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:04.240+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:05.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:05.197+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:05.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:05.271+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:06.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:06.242+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:06.258 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:06.256+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:07.291 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:07.289+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:07.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:07.289+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:08.300 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:08.298+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:08.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:08.337+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:09.333 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:09.331+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:09.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:09.354+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:10.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:10.309+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:10.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:10.312+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:11.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:11.306+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:11.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:11.348+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:12.151 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:37:12.149+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:37:12.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:12.322+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:12.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:12.370+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:13.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:13.366+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:13.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:13.367+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:14.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:14.350+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:14.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:14.377+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:15.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:15.323+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:15.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:15.369+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:16.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:16.325+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:16.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:16.378+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:17.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:17.289+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:17.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:17.407+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:18.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:18.270+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:18.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:18.365+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:18.561 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-03T20:37:18.558+0000 7f2fd761c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T20:37:18.561 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-03T20:37:18.559+0000 7f2fd661a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T20:37:19.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:19.291+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:19.322 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:19.319+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:20.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:20.260+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:20.333 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:20.331+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:21.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:21.243+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:21.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:21.319+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:22.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:22.271+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:22.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:22.290+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:23.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:23.255+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:23.264 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:23.262+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:24.270 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:24.268+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:24.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:24.284+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:25.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:25.310+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:25.319 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:25.317+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:26.300 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:26.298+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:26.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:26.336+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:27.167 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:37:27.165+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:37:27.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:27.308+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:27.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:27.351+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:28.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:28.334+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:28.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:28.377+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:28.405 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:37:28.408 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:37:29.337 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:29.335+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:29.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:29.350+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:30.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:30.294+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:30.397 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:30.395+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:31.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:31.273+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:31.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:31.402+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:32.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:32.321+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:32.403 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:32.401+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:33.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:33.299+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:33.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:33.423+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:34.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:34.271+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:34.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:34.454+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:35.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:35.306+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:35.502 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:35.500+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:36.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:36.344+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:36.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:36.510+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:37.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:37.326+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:37.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:37.544+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:38.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:38.304+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:38.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:38.519+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:39.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:39.331+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:39.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:39.544+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:40.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:40.313+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:40.561 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:40.559+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:41.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:41.332+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:41.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:41.532+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:42.183 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:37:42.181+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:37:42.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:42.362+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:42.564 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:42.562+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:43.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:43.342+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:43.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:43.514+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:44.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:44.304+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:44.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:44.513+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:45.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:45.316+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:45.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:45.499+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:46.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:46.337+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:46.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:46.526+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:47.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:47.296+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:47.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:47.538+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:48.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:48.304+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:48.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:48.531+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:49.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:49.339+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:49.520 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:49.518+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:50.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:50.315+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:50.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:50.550+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:51.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:51.339+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:51.586 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:51.585+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:52.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:52.301+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:52.566 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:52.564+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:53.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:53.296+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:53.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:53.609+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:54.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:54.318+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:54.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:54.639+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:55.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:55.279+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:55.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:55.675+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:56.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:56.238+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:56.634 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:56.632+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:57.199 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:37:57.196+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:37:57.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:57.201+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:57.598 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:57.596+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:58.213 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:58.212+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:58.481 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:37:58.485 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:37:58.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:58.586+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:37:59.205 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:37:59.203+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:37:59.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:37:59.627+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:00.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:00.197+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:00.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:00.653+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:01.170 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:01.168+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:01.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:01.604+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:02.136 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:02.134+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:02.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:02.590+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:03.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:03.155+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:03.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:03.540+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:04.127 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:04.125+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:04.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:04.534+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:05.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:05.153+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:05.550 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:05.547+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:06.183 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:06.180+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:06.589 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:06.586+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:07.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:07.166+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:07.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:07.610+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:08.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:08.157+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:08.585 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:08.582+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:09.114 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:09.111+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:09.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:09.542+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:10.113 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:10.110+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:10.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:10.509+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:11.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:11.159+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:11.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:11.485+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:12.133 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:12.130+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:12.215 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:38:12.212+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:38:12.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:12.482+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:13.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:13.160+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:13.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:13.443+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:14.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:14.186+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:14.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:14.471+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:15.140 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:15.137+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:15.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:15.487+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:16.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:16.150+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:16.482 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:16.480+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:17.134 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:17.131+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:17.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:17.449+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:18.092 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:18.089+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:18.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:18.459+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:19.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:19.109+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:19.477 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:19.474+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:20.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:20.159+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:20.432 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:20.430+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:21.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:21.198+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:21.442 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:21.440+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:22.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:22.244+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:22.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:22.407+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:23.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:23.288+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:23.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:23.433+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:24.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:24.295+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:24.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:24.386+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:25.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:25.248+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:25.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:25.338+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:26.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:26.285+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:26.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:26.314+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:27.231 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:38:27.228+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:38:27.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:27.255+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:27.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:27.269+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:28.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:28.242+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:28.255 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:28.254+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:28.558 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:38:28.562 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:38:29.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:29.259+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:29.291 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:29.289+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:30.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:30.245+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:30.307 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:30.305+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:31.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:31.213+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:31.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:31.346+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:32.191 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:32.190+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:32.384 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:32.382+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:33.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:33.208+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:33.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:33.399+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:34.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:34.246+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:34.403 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:34.401+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:35.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:35.201+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:35.358 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:35.355+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:36.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:36.153+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:36.342 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:36.341+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:37.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:37.190+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:37.360 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:37.358+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:38.151 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:38.149+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:38.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:38.315+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:39.134 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:39.132+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:39.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:39.309+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:38:40.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:40.155+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:40.358 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:40.356+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:38:41.109 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:41.107+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:41.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:41.403+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:38:42.110 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:42.108+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:42.247 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:38:42.244+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:38:42.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:42.413+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:38:43.088 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:43.086+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:43.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:43.387+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:38:44.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:44.135+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:44.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:44.437+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:38:45.123 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:45.121+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:45.469 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:45.468+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:38:46.130 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:46.128+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:46.478 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:46.476+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:38:47.088 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:47.086+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:47.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:47.441+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:38:48.138 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:48.136+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:48.433 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:48.431+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:38:49.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:49.182+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:49.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:49.455+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:38:50.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:50.185+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:50.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:50.415+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:38:51.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:51.207+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:51.410 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:51.408+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:38:52.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:52.208+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:52.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:52.423+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:38:53.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:53.198+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:53.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:53.426+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:38:54.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:54.207+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:54.408 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:54.407+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:38:55.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:55.177+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:55.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:55.449+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:38:56.145 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:56.142+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:56.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:56.418+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:38:57.151 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:57.148+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:57.262 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:38:57.260+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:38:57.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:57.441+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:38:58.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:58.164+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:58.448 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:58.445+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:38:58.656 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:38:58.659 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:38:59.151 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:38:59.149+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:38:59.452 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:38:59.450+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:00.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:00.184+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:00.471 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:00.468+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:01.169 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:01.167+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:01.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:01.432+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:02.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:02.181+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:02.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:02.390+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:03.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:03.171+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:03.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:03.342+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:04.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:04.146+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:04.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:04.383+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:05.129 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:05.127+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:05.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:05.365+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:06.135 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:06.133+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:06.365 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:06.362+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:07.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:07.154+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:07.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:07.397+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:08.205 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:08.202+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:08.386 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:08.384+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:09.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:09.201+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:09.422 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:09.419+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:10.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:10.240+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:10.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:10.460+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:11.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:11.270+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:11.506 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:11.504+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:12.278 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:39:12.275+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:39:12.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:12.289+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:12.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:12.490+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:13.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:13.331+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:13.453 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:13.451+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:14.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:14.344+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:14.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:14.454+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:15.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:15.329+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:15.433 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:15.431+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:16.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:16.290+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:16.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:16.417+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:17.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:17.249+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:17.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:17.373+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:18.235 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:18.233+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:18.424 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:18.422+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:19.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:19.242+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:19.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:19.423+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:20.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:20.221+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:20.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:20.419+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:21.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:21.194+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:21.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:21.410+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:22.195 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:22.193+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:22.422 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:22.420+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:23.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:23.153+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:23.471 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:23.468+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:24.205 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:24.203+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:24.422 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:24.420+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:25.191 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:25.189+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:25.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:25.406+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:26.145 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:26.142+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:26.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:26.411+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:27.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:27.185+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:27.294 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:39:27.292+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:39:27.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:27.413+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:28.174 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:28.172+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:28.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:28.423+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:28.812 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:39:28.815 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:39:29.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:29.165+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:29.387 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:29.384+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:30.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:30.201+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:30.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:30.339+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:31.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:31.191+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:31.379 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:31.377+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:32.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:32.214+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:32.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:32.413+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:33.195 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:33.193+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:33.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:33.404+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:34.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:34.165+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:34.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:34.449+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:35.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:35.146+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:35.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:35.474+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:36.134 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:36.132+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:36.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:36.489+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:37.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:37.103+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:37.468 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:37.467+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:38.110 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:38.108+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:38.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:38.486+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:39.079 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:39.077+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:39.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:39.437+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:40.093 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:40.091+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:40.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:40.444+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:41.070 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:41.068+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:41.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:41.415+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:42.111 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:42.104+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:42.310 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:39:42.308+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:39:42.372 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:42.370+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:43.116 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:43.114+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:43.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:43.407+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:44.143 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:44.142+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:44.448 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:44.446+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:45.154 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:45.152+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:45.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:45.496+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:46.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:46.147+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:46.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:46.495+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:47.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:47.184+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:47.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:47.465+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:48.235 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:48.233+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:48.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:48.463+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:49.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:49.216+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:49.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:49.429+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:50.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:50.248+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:50.433 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:50.430+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:51.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:51.257+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:51.412 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:51.409+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:52.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:52.270+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:52.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:52.444+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:53.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:53.261+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:53.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:53.495+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:54.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:54.271+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:54.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:54.511+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:55.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:55.309+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:55.472 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:55.469+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:56.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:56.349+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:56.449 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:56.447+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:57.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:57.315+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:57.325 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:39:57.322+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:39:57.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:57.436+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:58.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:58.308+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:58.448 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:58.445+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:39:58.950 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:39:58.954 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:39:59.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:39:59.316+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:39:59.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:39:59.458+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:00.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:00.350+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:00.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:00.462+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:01.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:01.324+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:01.496 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:01.494+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:02.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:02.284+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:02.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:02.536+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:03.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:03.326+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:03.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:03.489+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:04.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:04.335+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:04.490 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:04.488+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:05.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:05.348+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:05.468 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:05.466+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:06.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:06.394+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:06.506 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:06.504+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:07.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:07.350+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:07.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:07.495+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:08.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:08.332+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:08.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:08.471+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:09.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:09.347+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:09.448 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:09.445+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:10.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:10.310+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:10.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:10.443+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:11.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:11.333+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:11.420 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:11.418+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:12.331 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:40:12.328+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:40:12.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:12.331+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:12.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:12.459+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:13.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:13.308+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:13.462 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:13.459+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:14.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:14.296+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:14.434 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:14.432+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:15.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:15.337+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:15.420 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:15.417+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:16.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:16.351+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:16.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:16.422+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:17.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:17.349+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:17.471 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:17.469+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:18.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:18.302+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:18.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:18.512+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:19.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:19.308+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:19.563 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:19.561+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:20.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:20.264+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:20.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:20.577+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:21.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:21.225+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:21.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:21.585+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:22.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:22.207+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:22.589 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:22.587+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:23.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:23.195+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:23.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:23.601+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:24.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:24.152+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:24.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:24.568+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:25.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:25.147+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:25.563 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:25.561+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:26.147 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:26.145+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:26.574 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:26.572+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:27.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:27.110+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:27.346 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:40:27.344+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:40:27.550 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:27.548+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:28.136 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:28.134+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:28.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:28.539+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:29.073 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:40:29.077 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:40:29.162 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:29.161+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:29.540 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:29.537+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:30.119 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:30.117+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:30.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:30.541+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:31.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:31.157+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:31.540 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:31.539+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:32.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:32.179+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:32.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:32.581+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:33.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:33.191+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:33.569 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:33.568+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:34.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:34.173+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:34.554 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:34.553+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:35.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:35.206+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:35.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:35.546+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:36.031 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T20:40:36.028+0000 7f6a019f9700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T20:40:36.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:36.245+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:36.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:36.540+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:37.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:37.249+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:37.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:37.552+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:38.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:38.292+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:38.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:38.581+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:39.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:39.297+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:39.630 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:39.628+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:40.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:40.267+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:40.675 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:40.674+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:41.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:41.310+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:41.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:41.662+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:42.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:42.271+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:42.362 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:40:42.359+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:40:42.703 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:42.702+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:43.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:43.302+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:43.686 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:43.684+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:44.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:44.345+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:44.707 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:44.705+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:45.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:45.337+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:45.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:45.663+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:46.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:46.318+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:46.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:46.645+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:47.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:47.333+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:47.660 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:47.657+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:48.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:48.324+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:48.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:48.645+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:49.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:49.305+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:49.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:49.661+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:50.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:50.259+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:50.649 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:50.646+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:51.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:51.292+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:51.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:51.602+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:52.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:52.263+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:52.557 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:52.554+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:53.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:53.287+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:53.522 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:53.519+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:54.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:54.251+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:54.499 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:54.496+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:55.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:55.225+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:55.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:55.546+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:56.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:56.241+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:56.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:56.582+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:57.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:57.231+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:57.378 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:40:57.376+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:40:57.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:57.574+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:58.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:58.253+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:58.550 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:58.547+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:40:59.204 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:40:59.208 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:40:59.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:40:59.271+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:40:59.590 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:40:59.588+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:00.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:00.253+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:00.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:00.582+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:01.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:01.232+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:01.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:01.589+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:02.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:02.215+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:02.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:02.549+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:03.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:03.175+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:03.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:03.532+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:04.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:04.187+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:04.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:04.542+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:05.142 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:05.139+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:05.586 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:05.584+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:06.165 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:06.162+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:06.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:06.557+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:07.119 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:07.117+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:07.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:07.558+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:08.132 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:08.130+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:08.604 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:08.602+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:09.124 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:09.122+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:09.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:09.574+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:10.077 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:10.074+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:10.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:10.618+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:11.031 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:11.029+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:11.590 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:11.587+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:11.993 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:11.990+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:12.386 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:41:12.384+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:41:12.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:12.633+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:13.010 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:13.008+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:13.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:13.675+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:14.023 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:14.021+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:14.721 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:14.718+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:15.068 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:15.065+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:15.749 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:15.747+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:16.021 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:16.018+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:16.768 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:16.765+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:17.023 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:17.020+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:17.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:17.806+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:18.049 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:18.046+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:18.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:18.786+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:19.096 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:19.093+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:19.814 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:19.811+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:20.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:20.103+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:20.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:20.841+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:21.149 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:21.147+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:21.814 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:21.811+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:22.176 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:22.173+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:22.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:22.825+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:23.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:23.199+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:23.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:23.791+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:24.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:24.191+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:24.752 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:24.750+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:25.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:25.235+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:25.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:25.723+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:26.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:26.264+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:26.771 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:26.769+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:27.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:27.271+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:27.401 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:41:27.398+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:41:27.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:27.816+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:28.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:28.242+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:28.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:28.843+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:29.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:29.222+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:29.351 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:41:29.355 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:41:29.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:29.880+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:30.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:30.196+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:30.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:30.883+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:31.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:31.234+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:31.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:31.856+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:32.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:32.260+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:32.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:32.819+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:33.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:33.273+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:33.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:33.836+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:34.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:34.306+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:34.817 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:34.816+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:35.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:35.300+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:35.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:35.838+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:36.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:36.268+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:36.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:36.858+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:37.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:37.295+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:37.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:37.856+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:38.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:38.248+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:38.874 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:38.873+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:39.212 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:39.210+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:39.905 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:39.902+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:41:40.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:40.200+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:40.891 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:40.888+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:41:41.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:41.193+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:41.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:41.857+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:41:42.149 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:42.146+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:42.417 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:41:42.415+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:41:42.899 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:42.896+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:41:43.116 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:43.114+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:43.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:43.920+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:41:44.128 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:44.125+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:44.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:44.918+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:41:45.141 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:45.139+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:45.963 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:45.960+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:41:46.110 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:46.107+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:46.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:46.974+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:41:47.139 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:47.137+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:47.962 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:47.960+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:41:48.108 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:48.106+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:48.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:48.920+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:41:49.068 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:49.066+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:49.970 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:49.968+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:41:50.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:50.112+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:50.970 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:50.967+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:41:51.154 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:51.151+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:51.944 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:51.941+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:41:52.125 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:52.122+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:52.963 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:52.961+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:41:53.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:53.153+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:54.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:54.000+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:41:54.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:54.178+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:54.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:54.980+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:41:55.176 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:55.174+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:55.968 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:55.965+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:41:56.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:56.185+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:56.969 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:56.966+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:41:57.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:57.229+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:57.433 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:41:57.431+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:41:57.963 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:57.961+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:41:58.217 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:58.214+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:58.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:41:58.985+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:41:59.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:41:59.191+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:41:59.741 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:41:59.745 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:42:00.022 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:00.020+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:00.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:00.225+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:01.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:01.036+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:01.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:01.244+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:02.014 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:02.011+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:02.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:02.257+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:03.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:03.045+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:03.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:03.296+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:04.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:04.065+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:04.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:04.270+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:05.068 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:05.065+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:05.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:05.306+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:06.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:06.101+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:06.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:06.300+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:07.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:07.073+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:07.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:07.292+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:08.045 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:08.043+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:08.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:08.261+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:09.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:09.077+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:09.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:09.253+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:10.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:10.110+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:10.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:10.300+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:11.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:11.157+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:11.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:11.264+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:12.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:12.185+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:12.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:12.313+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:12.449 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:42:12.446+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:42:13.191 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:13.188+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:13.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:13.356+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:14.177 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:14.174+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:14.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:14.333+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:15.157 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:15.155+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:15.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:15.381+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:16.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:16.201+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:16.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:16.396+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:17.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:17.217+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:17.356 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:17.354+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:18.180 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:18.177+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:18.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:18.319+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:19.193 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:19.192+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:19.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:19.271+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:20.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:20.171+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:20.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:20.301+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:21.196 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:21.194+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:21.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:21.317+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:22.194 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:22.192+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:22.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:22.323+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:23.212 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:23.210+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:23.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:23.322+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:24.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:24.224+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:24.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:24.320+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:25.217 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:25.216+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:25.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:25.370+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:26.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:26.168+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:26.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:26.370+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:27.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:27.213+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:27.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:27.385+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:27.465 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:42:27.463+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:42:28.245 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:28.244+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:28.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:28.378+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:29.263 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:29.261+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:29.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:29.379+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:29.896 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:42:29.900 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:42:30.252 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:30.251+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:30.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:30.339+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:31.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:31.294+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:31.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:31.364+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:32.262 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:32.260+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:32.332 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:32.330+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:33.230 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:33.228+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:33.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:33.332+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:34.205 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:34.202+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:34.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:34.289+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:35.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:35.195+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:35.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:35.253+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:36.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:36.156+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:36.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:36.267+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:37.169 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:37.166+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:37.221 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:37.218+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:38.214 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:38.212+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:38.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:38.234+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:39.238 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:39.235+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:39.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:39.262+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:40.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:40.244+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:40.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:40.270+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:41.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:41.283+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:41.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:41.292+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:42.269 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:42.267+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:42.301 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:42.299+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:42.481 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:42:42.478+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:42:43.225 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:43.222+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:43.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:43.306+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:44.233 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:44.230+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:44.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:44.266+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:45.186 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:45.184+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:45.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:45.228+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:46.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:46.172+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:46.235 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:46.232+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:47.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:47.184+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:47.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:47.260+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:48.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:48.172+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:48.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:48.241+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:49.181 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:49.178+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:49.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:49.234+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:50.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:50.161+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:50.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:50.205+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:51.148 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:51.145+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:51.233 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:51.231+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:52.177 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:52.175+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:52.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:52.236+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:53.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:53.198+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:53.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:53.274+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:54.172 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:54.170+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:54.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:54.227+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:55.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:55.198+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:55.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:55.198+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:56.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:56.164+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:56.204 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:56.201+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:57.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:57.193+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:57.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:57.208+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:57.496 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:42:57.493+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:42:58.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:58.209+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:58.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:58.242+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:59.217 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:42:59.215+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:42:59.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:42:59.243+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:42:59.970 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:42:59.974 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:43:00.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:00.213+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:00.252 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:00.250+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:01.193 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:01.191+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:01.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:01.207+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:02.206 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:02.204+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:02.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:02.234+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:03.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:03.196+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:03.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:03.239+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:04.149 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:04.147+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:04.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:04.228+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:05.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:05.159+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:05.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:05.225+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:06.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:06.141+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:06.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:06.207+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:07.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:07.134+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:07.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:07.243+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:08.150 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:08.148+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:08.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:08.244+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:09.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:09.141+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:09.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:09.274+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:10.177 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:10.175+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:10.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:10.259+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:11.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:11.129+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:11.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:11.259+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:12.084 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:12.082+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:12.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:12.290+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:12.513 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:43:12.510+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:43:13.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:13.119+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:13.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:13.327+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:14.148 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:14.146+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:14.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:14.350+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:15.130 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:15.128+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:15.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:15.334+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:16.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:16.157+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:16.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:16.354+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:17.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:17.163+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:17.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:17.370+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:18.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:18.123+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:18.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:18.388+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:19.086 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:19.085+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:19.405 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:19.403+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:20.098 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:20.096+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:20.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:20.364+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:21.144 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:21.142+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:21.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:21.321+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:22.138 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:22.136+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:22.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:22.315+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:23.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:23.101+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:23.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:23.323+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:24.056 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:24.054+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:24.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:24.351+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:25.022 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:25.020+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:25.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:25.311+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:25.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:25.972+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:26.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:26.314+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:27.022 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:27.021+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:27.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:27.278+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:27.522 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:43:27.520+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:43:27.978 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:27.976+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:28.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:28.259+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:28.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:28.955+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:29.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:29.246+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:29.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:29.995+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:30.118 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:43:30.123 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:43:30.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:30.283+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:30.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:30.995+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:31.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:31.297+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:31.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:31.986+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:32.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:32.258+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:33.001 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:32.998+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:33.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:33.237+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:34.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:34.023+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:34.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:34.237+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:35.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:35.060+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:35.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:35.264+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:36.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:36.030+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:36.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:36.287+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:37.043 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:37.040+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:37.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:37.275+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:38.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:38.033+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:38.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:38.278+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:39.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:39.079+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:39.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:39.235+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:40.104 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:40.101+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:40.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:40.226+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:41.060 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:41.057+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:41.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:41.276+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:42.045 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:42.043+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:42.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:42.314+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:42.538 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:43:42.534+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:43:43.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:43.072+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:43.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:43.283+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:44.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:44.100+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:44.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:44.276+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:45.120 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:45.118+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:45.301 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:45.299+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:46.119 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:46.117+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:46.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:46.272+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:47.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:47.139+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:47.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:47.251+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:48.098 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:48.095+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:48.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:48.214+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:49.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:49.060+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:49.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:49.242+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:50.094 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:50.092+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:50.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:50.238+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:51.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:51.132+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:51.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:51.258+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:52.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:52.171+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:52.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:52.267+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:53.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:53.173+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:53.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:53.262+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:54.176 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:54.173+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:54.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:54.300+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:55.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:55.201+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:55.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:55.252+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:56.240 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:56.237+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:56.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:56.293+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:57.212 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:57.210+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:57.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:57.251+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:57.553 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:43:57.551+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:43:58.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:58.176+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:58.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:58.265+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:43:59.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:43:59.205+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:43:59.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:43:59.307+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:00.191 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:00.189+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:00.255 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:44:00.259 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:44:00.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:00.297+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:01.149 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:01.147+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:01.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:01.249+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:02.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:02.121+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:02.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:02.296+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:03.090 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:03.088+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:03.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:03.292+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:04.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:04.125+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:04.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:04.308+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:05.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:05.121+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:05.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:05.318+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:06.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:06.078+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:06.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:06.319+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:07.093 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:07.090+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:07.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:07.298+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:08.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:08.052+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:08.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:08.345+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:09.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:09.057+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:09.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:09.382+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:10.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:10.093+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:10.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:10.390+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:11.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:11.045+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:11.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:11.379+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:12.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:12.002+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:12.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:12.424+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:12.569 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:44:12.567+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:44:12.956 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:12.954+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:13.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:13.376+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:13.987 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:13.985+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:14.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:14.392+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:14.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:14.981+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:15.429 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:15.428+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:15.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:15.982+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:16.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:16.467+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:16.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:16.940+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:17.442 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:17.441+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:17.970 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:17.968+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:18.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:18.447+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:18.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:18.922+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:19.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:19.439+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:19.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:19.883+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:20.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:20.438+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:20.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:20.877+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:21.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:21.418+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:21.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:21.856+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:22.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:22.443+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:22.901 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:22.900+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:23.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:23.440+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:23.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:23.921+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:24.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:24.441+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:24.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:24.873+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:25.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:25.393+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:25.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:25.841+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:26.415 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:26.413+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:26.816 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:26.813+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:27.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:27.381+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:27.585 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:44:27.582+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:44:27.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:27.834+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:28.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:28.407+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:28.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:28.855+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:29.423 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:29.421+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:29.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:29.810+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:30.402 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:44:30.406 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:44:30.464 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:30.462+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:30.856 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:30.854+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:31.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:31.480+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:31.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:31.879+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:32.463 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:32.461+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:32.867 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:32.864+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:33.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:33.411+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:33.915 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:33.913+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:34.388 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:34.386+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:34.939 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:34.937+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:35.405 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:35.403+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:35.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:35.981+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:36.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:36.424+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:36.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:36.956+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:37.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:37.464+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:38.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:38.000+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:38.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:38.506+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:39.045 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:39.042+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:39.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:39.462+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:40.069 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:40.067+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:40.475 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:40.472+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:41.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:41.023+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:41.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:41.446+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:42.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:42.052+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:42.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:42.425+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:42.601 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:44:42.598+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:44:43.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:43.095+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:43.412 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:43.409+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:44.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:44.059+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:44.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:44.386+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:45.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:45.045+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:45.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:45.363+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:46.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:46.009+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:46.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:46.381+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:47.040 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:47.038+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:47.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:47.332+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:48.058 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:48.055+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:48.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:48.294+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:49.023 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:49.021+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:49.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:49.279+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:50.006 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:50.004+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:50.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:50.307+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:50.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:50.993+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:51.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:51.345+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:52.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:52.011+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:52.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:52.300+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:53.009 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:53.006+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:53.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:53.317+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:54.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:54.005+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:54.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:54.275+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:55.034 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:55.032+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:55.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:55.276+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:56.000 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:55.998+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:56.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:56.236+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:56.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:56.955+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:57.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:57.243+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:57.606 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:44:57.604+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:44:57.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:57.987+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:58.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:58.292+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:58.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:58.981+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:44:59.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:44:59.330+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:44:59.946 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:44:59.944+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:00.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:00.340+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:00.558 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:45:00.562 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:45:00.975 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:00.973+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:01.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:01.294+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:01.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:01.950+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:02.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:02.291+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:02.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:02.986+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:03.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:03.264+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:03.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:03.950+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:04.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:04.227+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:04.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:04.956+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:05.202 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:05.200+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:06.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:06.001+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:06.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:06.221+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:07.011 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:07.009+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:07.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:07.265+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:07.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:07.989+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:08.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:08.244+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:08.963 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:08.961+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:09.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:09.287+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:09.945 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:09.944+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:10.283 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:10.281+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:10.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:10.982+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:11.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:11.239+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:11.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:11.989+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:12.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:12.233+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:12.614 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:45:12.610+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:45:12.971 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:12.969+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:13.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:13.208+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:13.956 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:13.954+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:14.169 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:14.167+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:14.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:14.973+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:15.149 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:15.147+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:15.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:15.963+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:16.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:16.151+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:17.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:17.010+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:17.191 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:17.189+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:18.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:18.033+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:18.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:18.192+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:19.050 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:19.047+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:19.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:19.202+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:20.030 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:20.028+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:20.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:20.201+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:21.056 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:21.054+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:21.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:21.221+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:22.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:22.043+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:22.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:22.245+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:23.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:23.035+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:23.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:23.286+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:23.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:23.990+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:24.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:24.273+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:24.975 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:24.972+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:25.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:25.234+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:25.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:25.947+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:26.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:26.206+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:26.916 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:26.914+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:27.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:27.160+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:27.629 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:45:27.627+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:45:27.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:27.962+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:28.205 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:28.202+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:29.002 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:28.998+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:29.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:29.196+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:29.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:29.990+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:30.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:30.201+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:30.698 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:45:30.702 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:45:31.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:31.001+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:31.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:31.242+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:32.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:32.046+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:32.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:32.269+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:33.092 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:33.089+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:33.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:33.296+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:34.068 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:34.066+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:34.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:34.330+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:35.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:35.016+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:35.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:35.344+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:35.987 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:35.985+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:36.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:36.307+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:37.021 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:37.019+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:37.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:37.315+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:37.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:37.995+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:38.332 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:38.329+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:39.006 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:39.003+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:39.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:39.300+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:40.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:40.038+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:40.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:40.302+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:41.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:41.036+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:41.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:41.303+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:42.072 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:42.070+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:42.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:42.288+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:42.645 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:45:42.643+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:45:43.078 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:43.075+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:43.252 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:43.250+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:44.084 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:44.081+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:44.217 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:44.214+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:45.108 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:45.105+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:45.172 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:45.170+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:46.144 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:46.141+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:46.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:46.186+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:47.110 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:47.108+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:47.147 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:47.144+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:48.087 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:48.085+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:48.129 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:48.127+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:49.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:49.080+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:49.124 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:49.122+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:50.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:50.083+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:50.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:50.103+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:51.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:51.046+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:51.132 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:51.130+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:52.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:52.008+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:52.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:52.166+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:52.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:52.988+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:53.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:53.213+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:53.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:53.996+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:54.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:54.235+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:54.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:54.977+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:55.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:55.188+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:55.987 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:55.978+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:56.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:56.161+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:56.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:56.979+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:57.145 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:57.143+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:57.661 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:45:57.658+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:45:57.934 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:57.932+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:58.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:58.110+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:58.936 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:58.934+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:45:59.132 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:45:59.131+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:45:59.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:45:59.940+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:00.170 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:00.168+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:00.837 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:46:00.841 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:46:00.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:00.930+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:01.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:01.161+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:01.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:01.884+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:02.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:02.171+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:02.898 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:02.896+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:03.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:03.189+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:03.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:03.945+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:04.182 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:04.180+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:04.908 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:04.907+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:05.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:05.187+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:05.936 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:05.934+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:06.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:06.154+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:06.926 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:06.924+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:07.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:07.161+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:07.891 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:07.889+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:08.183 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:08.181+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:08.908 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:08.907+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:09.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:09.222+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:09.874 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:09.873+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:10.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:10.253+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:10.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:10.862+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:11.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:11.273+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:11.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:11.876+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:12.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:12.244+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:12.677 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:46:12.674+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:46:12.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:12.842+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:13.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:13.270+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:13.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:13.858+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:14.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:14.317+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:14.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:14.837+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:15.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:15.304+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:15.844 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:15.841+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:16.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:16.299+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:16.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:16.882+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:17.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:17.306+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:17.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:17.840+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:18.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:18.318+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:18.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:18.887+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:19.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:19.344+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:19.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:19.931+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:20.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:20.323+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:20.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:20.945+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:21.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:21.358+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:21.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:21.901+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:22.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:22.367+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:22.859 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:22.856+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:23.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:23.347+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:23.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:23.830+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:24.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:24.316+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:24.856 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:24.853+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:25.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:25.366+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:25.859 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:25.857+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:26.400 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:26.397+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:26.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:26.842+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:27.446 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:27.444+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:27.693 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:46:27.690+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:46:27.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:27.816+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:28.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:28.492+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:28.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:28.779+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:29.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:29.519+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:29.749 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:29.747+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:30.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:30.526+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:30.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:30.698+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:31.126 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:46:31.130 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:46:31.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:31.540+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:31.746 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:31.743+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:32.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:32.495+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:32.703 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:32.701+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:33.498 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:33.496+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:33.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:33.738+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:34.506 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:34.503+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:34.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:34.756+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:35.520 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:35.518+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:35.787 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:35.784+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:36.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:36.552+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:36.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:36.776+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:37.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:37.503+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:37.797 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:37.795+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:38.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:38.507+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:38.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:38.838+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:39.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:39.492+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:39.889 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:39.886+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:40.523 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:40.521+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:40.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:40.891+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:41.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:41.512+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:41.907 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:41.904+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:42.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:42.490+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:42.708 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:46:42.705+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:46:42.871 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:42.868+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:43.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:43.515+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:43.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:43.863+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:44.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:44.540+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:44.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:44.819+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:45.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:45.551+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:45.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:45.779+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:46.564 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:46.562+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:46.795 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:46.793+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:47.566 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:47.563+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:47.749 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:47.746+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:48.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:48.590+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:48.730 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:48.728+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:49.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:49.618+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:49.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:49.686+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:50.648 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:50.646+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:50.697 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:50.694+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:51.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:51.597+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:51.736 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:51.734+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:52.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:52.641+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:52.756 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:52.755+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:53.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:53.610+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:53.730 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:53.728+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:54.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:54.623+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:54.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:54.737+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:55.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:55.650+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:55.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:55.741+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:56.661 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:56.660+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:56.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:56.741+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:57.687 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:57.685+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:57.724 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:46:57.722+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:46:57.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:57.746+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:58.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:58.694+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:58.703 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:58.702+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:46:59.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:46:59.684+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:46:59.752 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:46:59.750+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:00.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:00.721+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:00.790 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:00.788+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:01.190 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:47:01.196 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:47:01.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:01.691+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:01.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:01.773+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:02.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:02.726+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:02.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:02.730+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:03.683 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:03.681+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:03.763 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:03.762+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:04.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:04.712+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:04.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:04.716+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:05.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:05.759+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:05.763 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:05.761+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:06.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:06.740+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:06.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:06.742+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:07.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:07.700+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:07.738 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:07.736+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:08.708 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:08.705+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:08.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:08.744+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:09.703 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:09.700+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:09.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:09.752+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:10.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:10.664+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:10.765 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:10.762+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:11.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:11.652+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:11.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:11.713+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:12.645 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:12.643+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:12.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:12.669+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:12.740 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:47:12.738+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:47:13.642 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:13.639+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:13.650 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:13.647+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:14.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:14.612+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:14.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:14.692+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:15.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:15.647+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:15.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:15.739+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:16.670 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:16.668+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:16.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:16.691+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:17.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:17.654+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:17.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:17.723+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:18.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:18.648+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:18.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:18.691+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:19.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:19.634+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:19.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:19.739+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:20.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:20.642+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:20.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:20.739+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:21.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:21.629+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:21.752 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:21.749+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:22.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:22.602+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:23.043 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:22.765+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:23.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:23.589+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:23.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:23.808+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:24.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:24.559+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:24.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:24.828+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:25.602 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:25.599+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:25.817 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:25.814+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:26.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:26.608+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:26.768 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:26.766+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:27.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:27.576+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:27.747 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:27.745+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:27.750 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:47:27.747+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:47:28.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:28.599+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:28.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:28.781+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:29.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:29.617+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:29.754 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:29.751+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:30.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:30.625+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:30.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:30.741+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:31.318 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:47:31.322 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:47:31.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:31.663+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:31.763 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:31.761+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:32.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:32.705+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:32.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:32.713+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:33.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:33.661+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:33.686 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:33.684+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:34.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:34.643+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:34.646 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:34.643+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:35.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:35.603+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:35.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:35.626+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:36.626 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:36.623+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:36.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:36.635+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:37.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:37.581+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:37.617 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:37.615+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:38.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:38.600+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:38.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:38.637+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:39.570 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:39.567+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:39.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:39.645+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:40.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:40.536+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:40.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:40.654+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:41.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:41.578+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:41.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:41.648+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:42.590 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:42.587+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:42.603 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:42.600+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:42.756 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:47:42.754+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:47:43.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:43.549+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:43.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:43.560+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:44.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:44.519+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:44.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:44.581+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:45.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:45.544+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:45.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:45.626+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:46.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:46.495+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:46.622 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:46.620+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:47.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:47.489+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:47.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:47.645+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:48.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:48.528+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:48.661 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:48.660+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:49.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:49.547+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:49.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:49.701+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:50.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:50.592+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:50.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:50.700+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:51.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:51.543+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:51.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:51.666+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:52.517 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:52.515+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:52.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:52.622+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:53.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:53.489+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:53.629 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:53.627+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:54.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:54.445+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:54.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:54.622+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:55.438 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:55.436+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:55.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:55.573+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:56.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:56.415+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:56.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:56.566+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:57.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:57.373+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:57.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:57.559+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:57.772 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:47:57.770+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:47:58.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:58.338+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:58.606 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:58.605+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:47:59.348 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:47:59.346+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:47:59.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:47:59.634+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:00.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:00.309+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:00.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:00.650+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:01.312 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:01.310+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:01.436 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:48:01.439 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:48:01.625 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:01.623+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:02.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:02.303+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:02.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:02.661+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:03.334 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:03.331+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:03.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:03.689+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:04.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:04.319+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:04.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:04.687+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:05.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:05.349+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:05.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:05.723+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:06.322 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:06.319+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:06.772 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:06.769+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:07.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:07.270+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:07.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:07.734+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:08.261 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:08.258+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:08.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:08.773+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:09.266 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:09.264+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:09.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:09.817+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:10.269 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:10.267+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:10.850 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:10.847+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:11.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:11.295+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:11.895 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:11.893+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:12.344 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:12.342+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:12.788 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:48:12.785+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:48:12.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:12.867+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:13.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:13.370+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:13.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:13.821+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:14.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:14.402+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:14.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:14.776+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:15.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:15.433+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:15.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:15.738+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:16.434 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:16.432+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:16.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:16.728+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:17.392 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:17.390+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:17.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:17.735+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:18.397 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:18.395+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:18.776 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:18.773+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:19.442 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:19.439+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:19.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:19.809+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:20.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:20.440+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:20.848 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:20.846+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:21.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:21.456+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:21.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:21.865+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:22.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:22.441+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:22.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:22.817+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:23.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:23.490+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:23.866 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:23.864+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:24.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:24.453+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:24.900 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:24.898+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:25.484 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:25.481+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:25.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:25.876+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:26.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:26.508+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:26.900 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:26.897+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:27.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:27.543+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:27.804 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:48:27.801+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:48:27.935 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:27.932+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:28.568 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:28.566+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:28.887 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:28.885+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:29.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:29.611+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:29.866 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:29.864+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:30.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:30.590+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:30.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:30.875+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:31.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:31.549+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:31.584 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:48:31.587 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:48:31.859 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:31.857+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:32.593 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:32.590+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:32.850 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:32.848+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:33.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:33.570+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:33.836 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:33.834+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:34.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:34.581+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:34.849 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:34.847+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:35.554 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:35.552+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:35.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:35.862+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:36.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:36.570+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:36.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:36.822+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:37.562 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:37.560+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:37.854 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:37.852+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:38.596 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:38.594+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:38.857 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:38.854+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:39.606 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:39.604+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:39.899 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:39.896+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:40.634 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:40.632+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:40.907 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:40.905+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:41.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:41.656+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:41.875 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:41.873+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:42.696 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:42.694+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:42.819 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:48:42.816+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:48:42.861 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:42.859+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:43.721 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:43.719+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:43.835 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:43.833+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:44.771 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:44.769+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:44.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:44.820+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:45.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:45.727+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:45.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:45.800+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:46.706 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:46.704+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:46.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:46.753+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:47.675 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:47.673+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:47.761 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:47.760+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:48.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:48.663+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:48.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:48.738+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:49.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:49.627+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:49.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:49.719+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:50.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:50.633+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:50.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:50.696+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:51.651 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:51.649+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:51.717 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:51.715+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:52.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:52.691+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:52.706 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:52.704+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:53.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:53.670+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:53.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:53.687+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:54.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:54.655+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:54.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:54.693+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:55.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:55.612+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:55.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:55.683+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:56.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:56.601+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:56.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:56.673+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:57.619 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:57.617+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:57.694 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:57.693+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:57.834 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:48:57.831+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:48:58.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:58.664+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:58.708 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:58.705+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:48:59.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:48:59.679+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:48:59.706 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:48:59.704+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:00.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:00.712+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:00.735 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:00.733+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:01.655 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:49:01.659 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:49:01.674 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:01.672+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:01.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:01.740+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:02.672 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:02.670+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:02.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:02.707+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:03.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:03.679+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:03.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:03.725+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:04.660 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:04.657+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:04.745 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:04.742+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:05.617 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:05.614+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:05.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:05.720+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:06.666 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:06.663+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:06.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:06.698+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:07.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:07.638+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:07.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:07.697+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:08.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:08.673+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:08.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:08.713+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:09.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:09.666+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:09.706 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:09.703+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:10.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:10.644+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:10.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:10.688+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:11.622 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:11.619+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:11.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:11.681+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:12.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:12.629+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:12.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:12.726+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:12.850 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:49:12.848+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:49:13.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:13.623+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:13.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:13.695+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:14.608 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:14.606+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:14.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:14.680+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:15.631 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:15.628+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:15.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:15.697+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:16.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:16.668+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:16.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:16.738+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:17.639 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:17.636+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:17.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:17.776+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:18.634 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:18.632+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:18.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:18.822+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:19.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:19.680+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:19.778 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:19.776+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:20.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:20.666+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:20.812 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:20.809+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:21.706 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:21.704+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:21.817 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:21.815+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:22.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:22.722+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:22.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:22.793+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:23.752 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:23.750+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:23.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:23.822+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:24.746 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:24.744+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:24.853 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:24.851+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:25.720 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:25.718+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:25.883 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:25.874+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:26.719 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:26.717+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:26.900 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:26.897+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:27.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:27.685+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:27.866 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:49:27.863+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:49:27.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:27.875+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:28.707 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:28.704+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:28.905 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:28.903+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:29.696 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:29.694+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:29.916 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:29.914+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:30.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:30.674+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:30.946 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:30.944+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:31.684 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:31.682+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:31.795 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:49:31.798 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:49:31.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:31.992+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:32.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:32.669+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:33.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:33.009+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:33.701 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:33.699+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:34.022 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:34.020+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:34.643 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T20:49:34.640+0000 7f69f81e6700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T20:49:34.654 INFO:tasks.ceph.mds.e.smithi077.stderr: -5758> 2022-04-03T20:40:36.028+0000 7f6a019f9700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T20:49:34.660 INFO:tasks.ceph.mds.e.smithi077.stderr: -1> 2022-04-03T20:49:34.640+0000 7f69f81e6700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T20:49:34.709 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:34.707+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:34.988 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:34.986+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:35.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:35.667+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:35.974 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:35.972+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:36.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:36.618+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:36.959 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:36.957+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:37.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:37.667+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:37.961 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:37.960+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:38.661 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:38.659+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:38.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:38.991+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:39.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:39.693+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:40.030 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:40.028+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:40.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:40.741+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:41.045 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:41.043+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:41.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:41.721+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:42.077 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:42.075+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:42.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:42.688+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:42.877 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:49:42.874+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:49:43.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:43.036+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:43.721 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:43.719+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:44.020 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:44.018+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:44.709 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:44.707+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:45.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:45.053+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:45.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:45.663+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:46.104 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:46.102+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:46.633 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:46.631+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:47.061 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:47.059+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:47.624 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:47.623+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:48.062 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:48.061+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:48.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:48.582+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:49.039 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:49.038+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:49.596 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:49.594+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:50.087 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:50.086+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:50.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:50.623+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:51.059 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:51.057+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:51.639 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:51.637+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:52.070 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:52.068+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:52.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:52.594+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:53.060 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:53.057+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:53.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:53.641+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:54.036 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:54.033+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:54.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:54.617+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:55.053 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:55.050+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:55.653 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:55.650+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:56.009 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:56.007+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:56.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:56.687+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:57.039 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:57.036+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:57.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:57.656+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:57.892 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:49:57.890+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:49:58.059 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:58.056+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:58.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:58.646+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:49:59.046 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:59.044+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:49:59.643 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:49:59.641+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:00.002 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:49:59.999+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:00.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:00.642+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:01.008 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:01.005+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:01.654 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:01.651+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:01.940 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:50:01.943 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:50:01.967 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:01.964+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:02.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:02.686+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:02.956 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:02.953+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:03.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:03.712+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:03.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:03.906+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:04.709 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:04.706+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:04.887 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:04.885+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:05.720 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:05.718+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:05.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:05.875+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:06.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:06.747+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:06.837 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:06.835+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:07.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:07.797+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:07.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:07.826+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:08.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:08.823+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:08.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:08.875+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:09.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:09.809+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:09.912 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:09.910+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:10.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:10.828+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:10.921 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:10.918+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:11.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:11.781+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:11.892 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:11.889+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:12.754 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:12.752+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:12.860 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:12.858+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:12.901 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:50:12.899+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:50:13.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:13.738+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:13.879 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:13.877+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:14.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:14.776+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:14.892 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:14.890+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:15.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:15.776+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:15.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:15.853+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:16.767 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:16.765+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:16.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:16.832+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:17.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:17.733+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:17.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:17.831+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:18.777 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:18.774+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:18.789 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:18.787+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:19.787 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:19.785+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:19.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:19.809+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:20.771 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:20.768+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:20.836 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:20.834+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:21.757 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:21.754+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:21.852 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:21.850+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:22.804 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:22.802+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:22.888 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:22.886+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:23.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:23.794+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:23.937 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:23.934+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:24.764 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:24.761+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:24.917 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:24.915+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:25.787 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:25.785+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:25.902 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:25.900+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:26.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:26.825+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:26.921 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:26.918+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:27.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:27.866+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:27.917 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:50:27.914+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:50:27.923 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:27.921+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:28.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:28.877+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:28.932 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:28.929+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:29.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:29.879+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:29.968 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:29.966+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:30.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:30.875+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:31.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:31.001+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:31.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:31.862+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:32.037 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:32.035+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:32.257 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:50:32.261 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:50:32.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:32.891+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:33.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:33.010+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:33.866 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:33.864+0000 7fd3b190b700 -1 osd.6 121 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:34.060 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:34.058+0000 7f88d0ac1700 -1 osd.4 121 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:34.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:34.876+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:35.097 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:35.095+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:35.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:35.840+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:36.147 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:36.145+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:36.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:36.809+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:37.183 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:37.181+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:37.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:37.833+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:38.214 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:38.212+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:38.883 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:38.881+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:39.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:39.216+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:39.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:39.910+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:40.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:40.186+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:40.894 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:40.892+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:41.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:41.201+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:41.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:41.844+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:42.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:42.239+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:42.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:42.851+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:42.932 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:50:42.930+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:50:43.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:43.234+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:43.866 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:43.864+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:44.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:44.186+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:44.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:44.844+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:45.182 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:45.180+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:45.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:45.888+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:46.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:46.166+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:46.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:46.929+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:47.160 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:47.158+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:47.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:47.971+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:48.182 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:48.179+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:48.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:48.988+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:49.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:49.154+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:50.017 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:50.014+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:50.162 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:50.159+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:51.044 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:51.041+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:51.151 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:51.148+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:52.084 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:52.081+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:52.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:52.193+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:53.043 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:53.041+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:53.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:53.157+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:54.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:54.082+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:54.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:54.184+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:55.069 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:55.066+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:55.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:55.185+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:56.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:56.093+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:56.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:56.196+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:57.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:57.119+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:57.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:57.195+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:57.948 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:50:57.946+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:50:58.280 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:58.109+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:58.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:58.150+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:50:59.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:50:59.129+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:50:59.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:50:59.161+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:00.087 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:00.085+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:00.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:00.195+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:01.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:01.109+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:01.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:01.242+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:02.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:02.077+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:02.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:02.249+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:02.391 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:51:02.395 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:51:03.109 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:03.106+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:03.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:03.222+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:04.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:04.122+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:04.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:04.247+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:05.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:05.123+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:05.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:05.273+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:06.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:06.092+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:06.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:06.303+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:07.106 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:07.104+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:07.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:07.308+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:08.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:08.145+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:08.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:08.324+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:09.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:09.133+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:09.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:09.297+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:10.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:10.137+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:10.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:10.345+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:11.092 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:11.089+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:11.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:11.344+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:12.129 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:12.126+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:12.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:12.388+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:12.964 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:51:12.962+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:51:13.144 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:13.142+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:13.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:13.395+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:14.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:14.132+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:14.421 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:14.418+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:15.162 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:15.159+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:15.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:15.370+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:16.168 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:16.166+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:16.412 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:16.409+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:17.129 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:17.127+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:17.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:17.381+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:18.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:18.121+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:18.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:18.343+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:19.160 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:19.158+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:19.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:19.371+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:20.205 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:20.204+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:20.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:20.395+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:21.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:21.194+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:21.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:21.439+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:22.233 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:22.230+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:22.473 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:22.470+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:23.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:23.185+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:23.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:23.507+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:24.152 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:24.150+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:24.546 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:24.544+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:25.140 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:25.138+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:25.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:25.574+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:26.142 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:26.140+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:26.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:26.590+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:27.104 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:27.102+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:27.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:27.613+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:27.980 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:51:27.977+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:51:28.091 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:28.089+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:28.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:28.649+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:29.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:29.129+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:29.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:29.633+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:30.100 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:30.099+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:30.606 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:30.604+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:31.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:31.093+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:31.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:31.615+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:32.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:32.079+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:32.531 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:51:32.535 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:51:32.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:32.639+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:33.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:33.109+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:33.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:33.625+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:34.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:34.145+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:34.659 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:34.657+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:35.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:35.185+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:35.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:35.678+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:36.152 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:36.150+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:36.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:36.673+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:37.168 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:37.166+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:37.629 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:37.627+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:38.129 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:38.127+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:38.582 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:38.580+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:39.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:39.126+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:39.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:39.540+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:40.120 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:40.118+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:40.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:40.536+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:41.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:41.161+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:41.526 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:41.524+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:42.205 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:42.203+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:42.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:42.539+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:42.996 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:51:42.993+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:51:43.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:43.226+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:43.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:43.545+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:44.239 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:44.236+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:44.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:44.502+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:45.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:45.283+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:45.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:45.510+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:46.245 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:46.242+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:46.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:46.541+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:47.205 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:47.202+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:47.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:47.555+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:48.191 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:48.189+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:48.552 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:48.549+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:49.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:49.176+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:49.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:49.557+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:50.216 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:50.213+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:50.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:50.559+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:51.213 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:51.210+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:51.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:51.546+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:52.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:52.230+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:52.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:52.529+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:53.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:53.256+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:53.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:53.573+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:54.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:54.286+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:54.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:54.579+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:55.287 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:55.285+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:55.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:55.610+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:56.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:56.323+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:56.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:56.562+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:57.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:57.334+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:57.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:57.598+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:58.011 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:51:58.008+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:51:58.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:58.285+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:58.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:58.564+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:51:59.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:51:59.266+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:51:59.539 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:51:59.536+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:00.258 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:00.256+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:00.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:00.510+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:01.301 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:01.299+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:01.484 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:01.481+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:02.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:02.323+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:02.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:02.463+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:02.669 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:52:02.673 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:52:03.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:03.320+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:03.480 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:03.477+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:04.327 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:04.325+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:04.490 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:04.488+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:05.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:05.295+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:05.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:05.510+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:06.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:06.275+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:06.506 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:06.503+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:07.261 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:07.258+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:07.537 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:07.534+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:08.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:08.276+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:08.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:08.568+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:09.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:09.308+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:09.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:09.584+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:10.267 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:10.264+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:10.552 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:10.549+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:11.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:11.269+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:11.519 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:11.516+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:12.269 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:12.266+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:12.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:12.515+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:13.026 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:52:13.024+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:52:13.289 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:13.287+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:13.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:13.528+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:14.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:14.327+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:14.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:14.559+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:15.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:15.289+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:15.550 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:15.548+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:16.334 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:16.332+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:16.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:16.543+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:17.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:17.367+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:17.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:17.502+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:18.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:18.337+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:18.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:18.515+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:19.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:19.313+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:19.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:19.558+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:20.291 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:20.289+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:20.577 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:20.575+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:21.274 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:21.272+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:21.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:21.624+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:22.253 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:22.252+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:22.629 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:22.627+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:23.240 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:23.238+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:23.606 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:23.604+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:24.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:24.282+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:24.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:24.587+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:25.306 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:25.304+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:25.572 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:25.571+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:26.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:26.275+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:26.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:26.530+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:27.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:27.279+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:27.484 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:27.483+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:28.042 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:52:28.040+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:52:28.318 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:28.316+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:28.489 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:28.488+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:29.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:29.297+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:29.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:29.463+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:30.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:30.340+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:30.453 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:30.452+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:31.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:31.351+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:31.424 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:31.423+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:32.360 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:32.359+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:32.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:32.408+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:32.808 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:52:32.812 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:52:33.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:33.381+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:33.398 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:33.396+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:34.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:34.349+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:34.398 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:34.396+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:35.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:35.343+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:35.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:35.416+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:36.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:36.327+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:36.436 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:36.434+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:37.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:37.362+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:37.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:37.461+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:38.409 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:38.406+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:38.472 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:38.470+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:39.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:39.414+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:39.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:39.512+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:40.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:40.455+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:40.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:40.509+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:41.409 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:41.406+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:41.527 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:41.524+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:42.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:42.400+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:42.509 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:42.506+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:43.058 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:52:43.056+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:52:43.421 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:43.419+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:43.469 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:43.466+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:44.464 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:44.461+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:44.498 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:44.495+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:45.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:45.445+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:45.472 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:45.470+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:46.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:46.458+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:46.471 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:46.469+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:47.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:47.471+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:47.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:47.495+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:48.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:48.456+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:48.520 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:48.518+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:49.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:49.411+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:49.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:49.523+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:50.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:50.449+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:50.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:50.560+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:51.480 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:51.478+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:51.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:51.588+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:52.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:52.461+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:52.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:52.544+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:53.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:53.455+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:53.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:53.518+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:54.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:54.436+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:54.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:54.530+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:55.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:55.412+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:55.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:55.510+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:56.366 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:56.364+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:56.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:56.522+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:57.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:57.367+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:57.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:57.491+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:58.074 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:52:58.071+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:52:58.327 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:58.324+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:58.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:58.515+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:52:59.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:52:59.289+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:52:59.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:52:59.525+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:00.333 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:00.331+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:00.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:00.545+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:01.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:01.376+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:01.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:01.571+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:02.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:02.407+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:02.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:02.591+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:02.965 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:53:02.969 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:53:03.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:03.396+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:03.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:03.552+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:04.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:04.409+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:04.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:04.531+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:05.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:05.457+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:05.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:05.486+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:06.436 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:06.434+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:06.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:06.492+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:07.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:07.396+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:07.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:07.455+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:08.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:08.412+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:08.477 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:08.476+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:09.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:09.388+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:09.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:09.489+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:10.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:10.373+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:10.522 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:10.519+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:11.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:11.368+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:11.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:11.479+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:12.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:12.349+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:12.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:12.437+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:13.089 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:53:13.087+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:53:13.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:13.378+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:13.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:13.413+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:14.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:14.423+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:14.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:14.427+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:15.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:15.399+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:15.480 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:15.478+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:16.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:16.433+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:16.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:16.474+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:17.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:17.454+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:17.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:17.457+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:18.472 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:18.471+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:18.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:18.495+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:19.434 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:19.432+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:19.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:19.508+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:20.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:20.396+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:20.480 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:20.478+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:21.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:21.381+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:21.506 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:21.504+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:22.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:22.429+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:22.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:22.478+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:23.480 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:23.479+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:23.498 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:23.497+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:24.506 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:24.504+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:24.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:24.530+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:25.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:25.527+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:25.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:25.533+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:26.546 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:26.544+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:26.550 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:26.548+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:27.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:27.534+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:27.547 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:27.545+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:28.105 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:53:28.103+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:53:28.508 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:28.506+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:28.517 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:28.516+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:29.477 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:29.475+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:29.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:29.514+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:30.446 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:30.444+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:30.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:30.507+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:31.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:31.465+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:31.506 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:31.504+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:32.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:32.426+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:32.517 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:32.515+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:33.096 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:53:33.099 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:53:33.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:33.461+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:33.475 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:33.472+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:34.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:34.440+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:34.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:34.452+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:35.471 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:35.468+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:35.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:35.491+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:36.423 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:36.420+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:36.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:36.534+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:37.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:37.443+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:37.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:37.557+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:38.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:38.479+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:38.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:38.604+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:39.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:39.434+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:39.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:39.576+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:53:40.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:40.400+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:40.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:40.536+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:53:41.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:41.382+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:41.525 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:41.523+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:53:42.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:42.345+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:42.558 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:42.556+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:53:43.121 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:53:43.119+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:53:43.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:43.383+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:43.527 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:43.525+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:53:44.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:44.354+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:44.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:44.486+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:53:45.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:45.371+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:45.520 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:45.517+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:53:46.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:46.338+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:46.506 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:46.503+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:53:47.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:47.347+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:47.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:47.548+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:53:48.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:48.376+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:48.524 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:48.522+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:53:49.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:49.405+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:49.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:49.525+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:53:50.446 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:50.444+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:50.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:50.525+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:53:51.407 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:51.405+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:51.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:51.548+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:53:52.405 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:52.402+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:52.559 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:52.554+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:53:53.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:53.416+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:53.568 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:53.566+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:53:54.437 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:54.435+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:54.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:54.550+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:53:55.464 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:55.462+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:55.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:55.590+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:53:56.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:56.422+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:56.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:56.551+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:53:57.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:57.442+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:57.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:57.600+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:53:58.137 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:53:58.134+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:53:58.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:58.396+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:58.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:58.574+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:53:59.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:53:59.446+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:53:59.573 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:53:59.571+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:00.474 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:00.472+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:00.608 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:00.606+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:01.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:01.447+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:01.593 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:01.591+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:02.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:02.482+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:02.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:02.575+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:03.167 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:54:03.171 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:54:03.460 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:03.458+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:03.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:03.573+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:04.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:04.438+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:04.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:04.617+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:05.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:05.439+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:05.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:05.667+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:06.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:06.425+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:06.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:06.697+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:07.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:07.456+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:07.734 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:07.732+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:08.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:08.478+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:08.777 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:08.775+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:09.477 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:09.474+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:09.766 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:09.764+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:10.468 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:10.465+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:10.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:10.803+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:11.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:11.438+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:11.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:11.782+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:12.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:12.412+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:12.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:12.743+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:13.152 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:54:13.150+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:54:13.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:13.372+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:13.771 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:13.769+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:14.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:14.383+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:14.740 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:14.738+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:15.423 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:15.421+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:15.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:15.760+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:16.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:16.433+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:16.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:16.775+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:17.405 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:17.404+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:17.764 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:17.762+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:18.409 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:18.408+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:18.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:18.783+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:19.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:19.395+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:19.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:19.778+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:20.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:20.367+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:20.819 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:20.817+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:21.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:21.329+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:21.814 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:21.812+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:22.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:22.318+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:22.837 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:22.835+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:23.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:23.317+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:23.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:23.885+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:24.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:24.297+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:24.909 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:24.908+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:25.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:25.256+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:25.944 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:25.943+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:26.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:26.278+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:26.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:26.939+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:27.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:27.256+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:27.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:27.972+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:28.168 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:54:28.166+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:54:28.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:28.298+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:28.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:28.951+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:29.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:29.253+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:29.943 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:29.940+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:30.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:30.247+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:30.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:30.946+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:31.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:31.295+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:31.961 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:31.958+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:32.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:32.277+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:33.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:33.002+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:33.283 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:33.280+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:33.364 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:54:33.368 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:54:34.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:34.002+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:34.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:34.328+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:34.685 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T20:54:34.683+0000 7f2e4a6df900 -1 monclient: get_monmap_and_config failed to get config 2022-04-03T20:54:34.689 INFO:tasks.ceph.mds.e.smithi077.stdout:starting mds.e at 2022-04-03T20:54:34.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:34.996+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:35.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:35.363+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:36.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:36.006+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:36.356 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:36.353+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:37.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:37.011+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:37.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:37.396+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:38.000 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:37.997+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:38.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:38.437+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:39.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:39.021+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:39.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:39.391+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:40.068 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:40.065+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:40.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:40.431+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:41.110 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:41.107+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:41.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:41.462+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:42.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:42.085+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:42.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:42.414+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:43.106 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:43.103+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:43.184 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:54:43.182+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:54:43.407 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:43.404+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:44.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:44.104+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:44.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:44.411+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:45.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:45.057+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:45.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:45.367+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:46.091 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:46.088+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:46.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:46.343+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:47.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:47.120+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:47.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:47.385+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:48.142 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:48.139+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:48.396 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:48.393+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:49.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:49.175+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:49.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:49.389+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:50.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:50.171+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:50.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:50.352+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:51.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:51.133+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:51.375 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:51.373+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:52.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:52.094+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:52.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:52.349+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:53.078 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:53.076+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:53.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:53.346+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:54.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:54.073+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:54.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:54.312+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:55.060 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:55.057+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:55.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:55.270+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:56.089 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:56.086+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:56.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:56.223+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:57.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:57.122+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:57.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:57.266+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:58.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:58.162+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:58.200 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:54:58.198+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:54:58.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:58.264+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:54:59.210 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:54:59.208+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:54:59.284 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:54:59.281+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:00.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:00.213+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:00.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:00.239+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:01.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:01.224+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:01.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:01.283+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:02.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:02.180+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:02.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:02.265+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:03.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:03.186+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:03.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:03.236+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:03.470 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:55:03.475 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:55:04.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:04.202+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:04.210 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:04.208+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:05.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:05.173+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:05.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:05.252+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:06.158 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:06.156+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:06.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:06.284+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:07.128 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:07.126+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:07.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:07.272+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:08.172 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:08.170+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:08.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:08.246+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:09.129 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:09.128+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:09.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:09.230+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:10.132 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:10.130+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:10.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:10.217+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:11.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:11.163+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:11.172 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:11.171+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:12.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:12.166+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:12.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:12.180+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:13.174 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:13.172+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:13.216 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:55:13.214+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:55:13.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:13.228+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:14.135 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:14.133+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:14.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:14.180+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:15.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:15.162+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:15.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:15.199+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:16.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:16.173+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:16.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:16.242+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:17.172 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:17.170+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:17.213 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:17.211+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:18.176 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:18.174+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:18.196 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:18.195+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:19.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:19.185+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:19.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:19.201+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:20.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:20.180+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:20.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:20.185+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:21.149 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:21.147+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:21.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:21.158+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:22.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:22.121+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:22.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:22.160+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:23.092 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:23.090+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:23.147 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:23.144+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:24.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:24.121+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:24.142 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:24.139+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:25.109 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:25.107+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:25.138 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:25.136+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:26.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:26.082+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:26.149 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:26.147+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:27.073 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:27.071+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:27.106 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:27.104+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:28.098 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:28.095+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:28.108 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:28.105+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:28.232 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:55:28.230+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:55:29.054 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:29.051+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:29.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:29.153+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:30.045 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:30.043+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:30.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:30.132+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:31.008 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:31.005+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:31.104 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:31.102+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:32.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:32.009+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:32.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:32.129+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:33.011 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:33.009+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:33.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:33.129+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:33.600 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:55:33.604 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:55:33.999 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:33.996+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:34.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:34.085+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:34.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:34.992+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:35.045 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:35.042+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:35.996 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:35.994+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:36.013 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:36.010+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:36.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:36.951+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:36.975 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:36.973+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:37.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:37.952+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:38.021 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:38.019+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:38.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:38.915+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:39.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:39.013+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:39.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:39.921+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:40.062 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:40.060+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:40.961 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:40.959+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:41.064 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:41.062+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:41.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:41.935+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:42.111 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:42.109+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:42.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:42.950+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:43.116 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:43.114+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:43.247 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:55:43.244+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:55:44.001 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:43.998+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:44.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:44.147+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:45.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:45.014+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:45.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:45.190+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:45.968 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:45.966+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:46.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:46.196+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:47.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:47.006+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:47.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:47.189+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:48.036 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:48.034+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:48.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:48.151+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:49.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:49.032+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:49.162 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:49.160+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:50.000 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:49.997+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:50.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:50.198+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:51.044 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:51.042+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:51.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:51.182+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:52.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:52.046+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:52.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:52.151+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:53.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:53.035+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:53.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:53.185+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:54.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:54.047+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:54.221 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:54.219+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:55.098 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:55.096+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:55.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:55.213+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:56.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:56.095+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:56.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:56.244+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:57.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:57.137+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:57.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:57.249+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:58.109 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:58.106+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:58.262 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:55:58.260+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:55:58.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:58.273+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:55:59.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:55:59.139+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:55:59.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:55:59.261+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:00.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:00.172+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:00.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:00.216+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:01.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:01.126+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:01.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:01.263+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:02.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:02.125+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:02.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:02.223+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:03.108 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:03.107+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:03.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:03.235+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:03.757 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:56:03.761 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:56:04.060 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:04.059+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:04.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:04.222+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:05.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:05.064+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:05.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:05.252+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:06.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:06.045+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:06.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:06.277+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:07.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:07.044+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:07.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:07.316+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:08.030 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:08.028+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:08.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:08.283+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:09.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:09.024+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:09.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:09.236+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:10.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:10.006+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:10.226 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:10.224+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:10.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:10.975+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:11.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:11.237+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:11.960 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:11.958+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:12.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:12.233+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:12.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:12.915+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:13.220 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:13.219+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:13.278 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:56:13.275+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:56:13.909 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:13.907+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:14.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:14.199+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:14.874 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:14.872+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:15.170 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:15.168+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:15.866 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:15.864+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:16.172 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:16.170+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:16.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:16.829+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:17.151 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:17.148+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:17.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:17.803+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:18.169 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:18.166+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:18.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:18.821+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:19.136 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:19.133+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:19.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:19.801+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:20.094 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:20.091+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:20.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:20.816+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:21.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:21.053+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:21.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:21.845+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:22.059 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:22.057+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:22.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:22.874+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:23.035 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:23.032+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:23.871 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:23.868+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:24.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:24.082+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:24.902 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:24.899+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:25.070 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:25.067+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:25.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:25.867+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:26.036 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:26.033+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:26.901 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:26.899+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:26.993 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:26.990+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:27.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:27.919+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:27.958 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:27.956+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:28.294 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:56:28.292+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:56:28.910 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:28.907+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:28.983 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:28.980+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:29.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:29.863+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:30.030 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:30.028+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:30.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:30.857+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:30.991 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:30.988+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:31.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:31.826+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:31.964 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:31.961+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:32.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:32.820+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:32.960 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:32.957+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:33.804 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:33.801+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:33.911 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:56:33.915 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:56:33.936 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:33.933+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:34.814 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:34.811+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:34.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:34.943+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:35.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:35.803+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:35.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:35.943+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:36.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:36.794+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:36.942 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:36.939+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:37.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:37.844+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:37.911 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:37.908+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:38.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:38.882+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:38.888 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:38.886+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:39.899 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:39.897+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T20:56:39.928 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:39.926+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:40.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:40.877+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:56:40.949 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:40.946+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:41.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:41.885+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:56:41.958 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:41.956+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:42.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:42.846+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:56:43.007 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:43.005+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:43.304 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:56:43.301+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:56:43.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:43.860+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:56:44.047 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:44.045+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:44.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:44.885+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:56:45.022 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:45.020+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:45.856 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:45.853+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:56:46.022 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:46.020+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:46.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:46.879+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:56:47.008 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:47.006+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:47.889 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:47.887+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:56:48.042 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:48.040+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:48.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:48.869+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:56:49.033 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:49.030+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:49.894 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:49.891+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:56:50.014 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:50.012+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:50.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:50.894+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:56:51.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:51.001+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:51.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:51.938+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:56:52.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:52.002+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:52.941 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:52.939+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:56:52.957 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:52.954+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:53.971 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:53.969+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:56:54.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:54.000+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:54.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:54.925+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:56:54.985 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:54.983+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:55.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:55.880+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:56:55.987 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:55.986+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:56.852 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:56.850+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:56:56.984 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:56.982+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:57.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:57.868+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:56:57.968 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:57.966+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:58.316 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:56:58.313+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:56:58.889 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:58.887+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:56:59.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:59.014+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:56:59.880 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:56:59.878+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:56:59.968 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:56:59.966+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:00.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:00.830+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:00.947 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:00.945+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:01.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:01.809+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:01.946 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:01.945+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:02.828 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:02.826+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:02.934 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:02.933+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:03.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:03.828+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:03.960 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:03.959+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:04.068 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:57:04.072 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:57:04.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:04.825+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:05.002 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:05.001+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:05.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:05.800+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:06.041 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:06.039+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:06.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:06.833+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:07.061 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:07.059+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:07.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:07.816+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:08.104 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:08.102+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:08.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:08.768+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:09.097 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:09.096+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:09.765 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:09.764+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:10.056 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:10.054+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:10.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:10.780+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:11.092 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:11.089+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:11.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:11.827+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:12.052 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:12.049+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:12.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:12.830+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:13.063 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:13.060+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:13.320 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:57:13.318+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:57:13.880 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:13.877+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:14.025 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:14.023+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:14.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:14.900+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:15.066 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:15.063+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:15.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:15.876+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:16.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:16.112+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:16.926 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:16.923+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:17.117 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:17.115+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:17.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:17.884+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:18.121 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:18.118+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:18.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:18.876+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:19.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:19.070+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:19.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:19.829+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:20.101 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:20.098+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:20.816 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:20.813+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:21.098 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:21.095+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:21.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:21.786+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:22.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:22.110+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:22.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:22.808+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:23.129 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:23.126+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:23.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:23.785+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:24.145 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:24.142+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:24.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:24.777+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:25.101 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:25.099+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:25.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:25.801+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:26.120 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:26.117+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:26.834 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:26.831+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:27.100 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:27.097+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:27.804 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:27.801+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:28.074 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:28.071+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:28.336 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:57:28.334+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:57:28.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:28.828+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:29.079 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:29.077+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:29.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:29.788+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:30.099 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:30.097+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:30.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:30.829+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:31.114 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:31.111+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:31.819 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:31.816+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:32.121 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:32.119+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:32.856 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:32.854+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:33.147 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:33.145+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:33.869 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:33.866+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:34.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:34.175+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:34.239 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:57:34.244 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:57:34.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:34.874+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:35.205 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:35.203+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:35.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:35.898+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:36.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:36.176+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:36.891 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:36.888+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:37.143 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:37.141+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:37.862 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:37.860+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:38.143 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:38.141+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:38.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:38.909+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:39.109 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:39.106+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:39.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:39.953+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:40.079 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:40.077+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:40.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:40.930+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:41.058 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:41.055+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:41.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:41.883+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:42.069 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:42.067+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:42.857 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:42.855+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:43.064 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:43.061+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:43.351 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:57:43.347+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:57:43.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:43.868+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:44.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:44.014+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:44.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:44.841+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:44.972 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:44.970+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:45.798 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:45.795+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:45.987 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:45.985+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:46.816 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:46.813+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:46.982 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:46.980+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:47.826 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:47.824+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:48.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:48.006+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:48.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:48.845+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:48.983 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:48.981+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:49.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:49.835+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:50.025 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:50.023+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:50.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:50.859+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:51.009 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:51.007+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:51.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:51.849+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:52.000 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:51.998+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:52.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:52.819+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:53.008 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:53.006+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:53.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:53.820+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:53.999 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:53.997+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:54.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:54.862+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:55.007 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:55.005+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:55.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:55.883+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:56.056 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:56.054+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:56.728 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T20:57:56.726+0000 7f2e44320700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T20:57:56.729 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T20:57:56.726+0000 7f2e43b1f700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T20:57:56.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:56.864+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:57.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:57.036+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:57.819 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:57.817+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:58.057 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:58.055+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:58.366 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:57:58.364+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:57:58.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:58.841+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:57:59.046 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:57:59.044+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:57:59.866 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:57:59.865+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:00.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:00.014+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:00.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:00.818+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:00.970 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:00.969+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:01.826 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:01.824+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:01.991 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:01.989+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:02.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:02.859+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:03.033 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:03.031+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:03.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:03.901+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:04.076 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:04.074+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:04.313 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:58:04.318 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:58:04.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:04.875+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:05.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:05.072+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:05.901 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:05.898+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:06.102 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:06.099+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:06.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:06.919+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:07.079 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:07.076+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:07.925 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:07.923+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:08.033 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:08.031+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:08.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:08.955+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:09.054 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:09.051+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:09.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:09.996+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:10.048 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:10.046+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:11.022 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:11.019+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:11.045 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:11.043+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:12.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:12.035+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:12.089 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:12.086+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:13.031 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:13.028+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:13.091 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:13.088+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:13.382 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:58:13.380+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:58:14.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:14.009+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:14.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:14.123+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:15.014 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:15.012+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:15.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:15.133+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:15.977 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:15.975+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:16.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:16.153+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:16.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:16.989+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:17.108 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:17.106+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:18.040 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:18.037+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:18.104 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:18.102+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:18.997 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:18.994+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:19.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:19.131+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:20.006 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:20.004+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:20.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:20.162+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:21.011 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:21.009+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:21.185 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:21.183+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:22.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:22.013+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:22.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:22.180+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:22.982 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:22.980+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:23.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:23.136+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:24.024 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:24.021+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:24.102 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:24.100+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:25.027 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:25.025+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:25.064 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:25.061+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:25.991 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:25.988+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:26.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:26.081+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:26.941 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:26.939+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:27.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:27.105+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:27.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:27.950+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:28.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:28.122+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:28.397 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:58:28.394+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:58:28.941 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:28.939+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:29.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:29.124+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:29.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:29.951+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:30.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:30.165+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:30.963 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:30.961+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:31.176 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:31.173+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:31.978 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:31.975+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:32.133 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:32.130+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:32.972 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:32.970+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:33.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:33.177+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:33.974 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:33.971+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:34.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:34.193+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:34.394 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:58:34.398 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:58:34.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:34.977+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:35.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:35.153+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:36.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:36.001+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:36.113 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:36.111+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:36.974 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:36.972+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:37.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:37.086+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:37.929 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:37.926+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:38.087 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:38.085+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:38.879 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:38.877+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:39.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:39.118+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:39.925 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:39.922+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:40.169 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:40.167+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:40.890 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:40.888+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:41.150 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:41.148+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:41.876 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:41.874+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:42.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:42.168+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:42.885 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:42.882+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:43.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:43.120+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:43.413 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:58:43.411+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:58:43.838 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:43.836+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:44.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:44.076+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:44.871 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:44.869+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:45.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:45.082+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:45.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:45.866+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:46.089 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:46.088+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:46.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:46.881+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:47.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:47.133+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:47.927 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:47.925+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:48.091 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:48.089+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:48.964 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:48.962+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:49.100 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:49.098+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:49.932 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:49.930+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:50.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:50.145+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:50.911 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:50.909+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:51.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:51.169+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:51.908 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:51.907+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:52.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:52.213+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:52.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:52.875+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:53.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:53.258+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:53.910 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:53.908+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:54.300 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:54.298+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:54.896 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:54.895+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:55.344 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:55.342+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:55.902 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:55.900+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:56.304 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:56.302+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:56.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:56.853+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:57.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:57.257+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:57.805 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:57.804+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:58.290 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:58.288+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:58.425 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:58:58.422+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:58:58.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:58.809+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:58:59.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:58:59.283+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:58:59.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:58:59.768+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:00.289 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:00.287+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:00.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:00.810+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:01.240 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:01.238+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:01.849 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:01.846+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:02.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:02.232+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:02.830 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:02.827+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:03.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:03.269+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:03.799 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:03.796+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:04.234 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:04.232+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:04.524 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:59:04.528 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:59:04.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:04.797+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:05.217 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:05.215+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:05.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:05.764+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:06.256 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:06.252+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:06.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:06.738+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:07.210 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:07.208+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:07.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:07.696+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:08.186 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:08.184+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:08.678 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:08.675+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:09.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:09.208+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:09.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:09.701+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:10.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:10.168+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:10.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:10.683+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:11.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:11.196+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:11.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:11.712+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:12.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:12.151+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:12.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:12.680+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:13.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:13.119+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:13.440 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:59:13.437+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:59:13.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:13.643+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:14.092 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:14.090+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:14.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:14.627+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:15.045 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:15.043+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:15.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:15.639+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:15.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:15.993+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:16.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:16.678+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:16.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:16.974+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:17.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:17.713+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:17.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:17.926+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:18.708 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:18.706+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:18.967 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:18.965+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:19.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:19.749+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:19.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:19.982+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:20.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:20.734+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:20.961 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:20.959+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:21.764 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:21.761+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:21.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:21.935+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:22.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:22.738+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:22.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:22.953+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:23.764 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:23.761+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:23.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:23.948+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:24.805 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:24.802+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:24.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:24.948+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:25.796 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:25.793+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:25.976 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:25.974+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:26.838 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:26.835+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:27.017 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:27.015+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:27.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:27.874+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:28.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:28.044+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:28.456 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:59:28.454+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:59:28.888 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:28.885+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:29.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:29.061+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:29.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:29.913+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:30.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:30.016+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:30.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:30.870+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:31.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:31.046+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:31.835 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:31.832+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:32.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:32.082+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:32.847 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:32.845+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:33.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:33.033+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:33.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:33.811+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:34.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:34.073+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:34.684 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:59:34.688 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T20:59:34.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:34.821+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:35.078 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:35.076+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:35.859 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:35.857+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:36.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:36.101+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:36.856 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:36.854+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:37.057 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:37.055+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:37.845 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:37.843+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:38.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:38.069+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:38.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:38.822+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:39.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:39.023+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:39.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:39.813+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:40.017 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:40.015+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:40.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:40.771+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:40.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:40.997+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:41.724 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:41.722+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:42.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:42.026+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:42.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:42.733+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:42.987 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:42.985+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:43.459 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:59:43.456+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:59:43.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:43.691+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:43.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:43.940+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:44.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:44.730+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:44.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:44.979+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:45.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:45.695+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:45.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:45.981+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:46.648 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:46.646+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:46.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:46.982+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:47.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:47.629+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:47.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:47.987+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:48.642 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:48.640+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:48.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:48.990+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:49.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:49.667+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:50.000 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:49.998+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:50.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:50.655+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:51.030 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:51.028+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:51.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:51.647+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:52.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:52.065+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:52.606 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:52.605+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:53.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:53.065+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:53.564 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:53.562+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:54.069 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:54.067+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:54.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:54.525+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:55.073 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:55.070+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:55.498 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:55.495+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:56.043 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:56.040+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:56.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:56.454+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:57.072 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:57.069+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:57.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:57.480+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:58.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:58.077+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:58.474 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T20:59:58.471+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T20:59:58.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:58.525+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T20:59:59.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T20:59:59.092+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T20:59:59.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T20:59:59.550+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:00.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:00.080+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:00.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:00.527+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:01.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:01.092+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:01.501 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:01.499+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:02.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:02.114+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:02.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:02.522+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:03.155 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:03.153+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:03.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:03.552+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:04.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:04.172+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:04.550 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:04.548+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:04.806 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:00:04.811 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:00:05.208 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:05.205+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:05.552 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:05.550+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:06.193 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:06.190+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:06.523 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:06.520+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:07.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:07.216+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:07.503 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:07.500+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:08.242 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:08.239+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:08.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:08.488+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:09.230 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:09.227+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:09.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:09.501+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:10.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:10.216+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:10.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:10.453+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:11.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:11.255+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:11.463 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:11.460+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:12.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:12.246+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:12.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:12.432+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:13.225 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:13.222+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:13.402 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:13.399+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:13.477 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:00:13.474+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:00:14.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:14.230+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:14.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:14.396+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:15.251 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:15.249+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:15.437 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:15.434+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:16.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:16.275+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:16.423 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:16.421+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:17.261 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:17.259+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:17.451 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:17.448+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:18.306 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:18.304+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:18.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:18.422+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:19.350 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:19.348+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:19.434 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:19.432+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:20.322 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:20.320+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:20.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:20.446+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:21.308 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:21.305+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:21.468 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:21.466+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:22.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:22.279+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:22.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:22.490+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:23.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:23.248+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:23.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:23.533+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:24.243 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:24.241+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:24.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:24.547+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:25.225 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:25.222+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:25.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:25.512+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:26.200 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:26.198+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:26.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:26.560+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:27.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:27.180+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:27.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:27.609+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:28.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:28.151+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:28.492 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:00:28.490+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:00:28.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:28.643+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:29.166 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:29.164+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:29.666 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:29.663+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:30.180 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:30.178+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:30.705 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:30.702+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:31.162 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:31.160+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:31.724 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:31.722+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:32.155 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:32.152+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:32.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:32.734+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:33.194 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:33.192+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:33.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:33.749+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:34.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:34.216+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:34.739 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:34.737+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:34.967 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:00:34.971 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:00:35.202 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:35.200+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:35.759 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:35.757+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:36.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:36.229+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:36.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:36.780+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:37.237 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:37.235+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:37.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:37.772+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:38.192 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:38.190+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:38.745 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:38.743+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:39.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:39.163+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:39.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:39.712+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:40.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:40.139+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:40.745 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:40.743+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:41.151 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:41.149+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:41.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:41.716+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:42.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:42.126+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:42.744 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:42.742+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:43.091 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:43.089+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:43.508 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:00:43.505+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:00:43.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:43.771+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:44.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:44.087+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:44.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:44.748+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:45.138 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:45.136+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:45.762 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:45.760+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:46.101 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:46.100+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:46.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:46.732+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:47.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:47.051+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:47.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:47.699+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:48.040 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:48.039+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:48.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:48.740+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:49.009 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:49.007+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:49.778 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:49.775+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:49.960 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:49.957+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:50.792 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:50.789+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:50.939 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:50.936+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:51.772 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:51.769+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:51.980 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:51.977+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:52.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:52.735+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:52.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:52.974+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:53.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:53.754+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:53.994 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:53.991+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:54.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:54.735+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:54.980 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:54.977+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:55.712 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:55.709+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:56.021 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:56.018+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:56.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:56.677+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:57.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:57.036+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:57.642 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:57.639+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:58.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:58.080+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:58.524 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:00:58.521+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:00:58.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:58.609+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:00:59.130 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:00:59.127+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:00:59.629 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:00:59.627+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:00.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:00.094+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:00.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:00.642+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:01.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:01.110+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:01.694 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:01.691+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:02.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:02.145+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:02.717 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:02.714+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:03.137 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:03.134+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:03.710 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:03.707+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:04.149 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:04.147+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:04.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:04.736+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:05.131 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:01:05.135 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:01:05.169 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:05.166+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:05.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:05.716+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:06.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:06.179+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:06.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:06.737+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:07.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:07.186+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:07.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:07.704+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:08.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:08.216+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:08.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:08.678+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:09.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:09.171+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:09.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:09.716+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:10.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:10.122+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:10.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:10.716+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:11.078 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:11.076+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:11.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:11.684+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:12.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:12.119+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:12.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:12.716+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:13.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:13.130+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:13.540 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:01:13.537+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:01:13.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:13.709+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:14.151 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:14.149+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:14.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:14.737+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:15.140 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:15.137+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:15.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:15.690+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:16.090 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:16.088+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:16.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:16.723+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:17.052 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:17.050+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:17.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:17.695+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:18.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:18.017+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:18.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:18.699+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:19.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:19.052+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:19.752 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:19.749+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:20.011 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:20.008+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:20.768 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:20.765+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:21.002 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:21.000+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:21.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:21.798+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:21.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:21.981+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:22.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:22.755+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:22.934 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:22.932+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:23.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:23.744+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:23.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:23.926+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:24.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:24.738+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:24.976 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:24.973+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:25.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:25.698+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:26.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:26.013+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:26.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:26.681+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:26.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:26.990+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:27.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:27.695+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:28.006 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:28.003+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:28.555 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:01:28.552+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:01:28.710 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:28.708+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:28.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:28.996+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:29.666 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:29.664+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:29.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:29.955+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:30.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:30.639+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:30.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:30.927+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:31.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:31.639+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:31.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:31.972+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:32.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:32.679+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:32.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:32.936+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:33.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:33.658+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:33.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:33.955+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:34.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:34.643+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:34.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:34.989+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:35.288 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:01:35.292 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:01:35.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:35.642+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:35.944 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:35.942+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:36.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:36.637+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:36.972 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:36.971+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:37.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:37.648+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:37.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:37.936+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:38.633 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:38.631+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:38.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:38.979+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:39.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:39.611+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:39.939 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:39.937+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:40.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:40.578+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:40.913 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:40.911+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:41.607 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:41.605+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:41.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:41.895+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:42.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:42.599+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:42.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:42.927+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:43.568 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:01:43.565+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:01:43.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:43.593+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:43.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:43.956+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:44.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:44.554+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:44.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:44.934+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:45.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:45.596+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:45.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:45.951+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:46.617 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:46.614+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:46.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:46.937+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:47.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:47.565+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:47.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:47.975+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:48.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:48.615+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:48.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:48.979+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:49.633 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:49.631+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:49.935 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:49.932+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:50.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:50.599+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:50.902 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:50.899+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:51.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:51.585+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:51.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:51.858+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:52.539 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:52.537+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:52.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:52.841+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:53.511 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:53.508+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:53.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:53.873+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:54.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:54.533+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:54.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:54.887+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:55.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:55.536+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:55.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:55.878+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:56.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:56.566+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:56.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:56.921+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:57.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:57.545+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:57.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:57.916+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:58.506 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:58.503+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:58.583 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:01:58.581+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:01:58.935 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:58.933+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:01:59.462 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:01:59.460+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:01:59.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:01:59.979+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:00.479 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:00.476+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:01.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:01.002+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:01.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:01.446+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:02.051 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:02.049+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:02.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:02.433+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:03.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:03.009+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:03.462 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:03.460+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:04.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:04.059+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:04.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:04.431+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:05.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:05.072+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:05.426 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:05.423+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:05.493 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:02:05.497 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:02:06.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:06.045+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:06.446 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:06.444+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:07.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:07.002+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:07.460 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:07.458+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:07.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:07.980+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:08.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:08.417+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:09.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:09.025+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:09.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:09.442+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:10.020 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:10.017+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:10.450 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:10.447+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:11.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:11.061+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:11.418 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:11.416+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:12.045 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:12.043+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:12.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:12.405+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:13.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:13.093+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:13.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:13.442+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:13.599 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:02:13.596+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:02:14.115 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:14.112+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:14.480 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:14.478+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:15.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:15.129+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:15.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:15.456+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:16.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:16.132+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:16.463 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:16.461+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:17.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:17.132+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:17.450 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:17.448+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:18.140 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:18.138+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:18.405 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:18.403+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:19.162 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:19.159+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:19.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:19.423+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:20.145 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:20.142+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:20.382 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:20.380+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:21.160 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:21.158+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:21.418 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:21.416+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:22.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:22.152+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:22.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:22.454+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:23.166 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:23.164+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:23.409 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:23.407+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:24.158 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:24.157+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:24.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:24.378+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:25.208 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:25.206+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:25.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:25.415+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:26.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:26.186+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:26.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:26.392+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:27.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:27.163+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:27.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:27.431+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:28.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:28.134+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:28.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:28.382+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:28.607 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:02:28.605+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:02:29.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:29.139+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:29.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:29.417+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:30.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:30.105+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:30.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:30.464+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:31.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:31.122+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:31.480 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:31.478+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:32.096 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:32.094+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:32.431 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:32.429+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:33.065 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:33.064+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:33.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:33.408+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:34.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:34.039+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:34.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:34.404+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:35.031 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:35.029+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:35.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:35.418+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:35.624 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:02:35.627 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:02:36.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:36.031+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:36.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:36.393+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:37.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:37.077+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:37.432 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:37.430+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:38.114 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:38.112+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:38.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:38.393+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:39.068 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:39.065+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:39.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:39.395+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:40.078 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:40.076+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:40.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:40.392+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:41.070 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:41.067+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:41.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:41.424+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:42.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:42.078+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:42.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:42.441+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:43.093 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:43.090+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:43.473 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:43.470+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:43.623 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:02:43.621+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:02:44.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:44.061+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:44.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:44.436+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:45.073 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:45.070+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:45.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:45.479+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:46.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:46.083+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:46.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:46.457+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:47.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:47.129+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:47.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:47.494+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:48.145 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:48.142+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:48.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:48.528+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:49.130 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:49.127+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:49.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:49.513+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:50.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:50.093+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:50.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:50.483+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:51.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:51.079+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:51.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:51.515+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:52.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:52.082+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:52.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:52.528+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:53.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:53.104+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:53.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:53.555+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:54.110 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:54.107+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:54.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:54.571+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:55.157 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:55.155+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:55.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:55.549+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:56.110 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:56.106+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:56.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:56.557+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:57.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:57.129+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:57.540 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:57.537+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:58.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:58.172+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:58.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:58.540+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:02:58.633 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:02:58.630+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:02:59.223 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:02:59.221+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:02:59.550 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:02:59.547+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:00.191 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:00.189+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:00.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:00.575+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:01.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:01.145+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:01.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:01.545+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:02.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:02.163+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:02.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:02.587+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:03.145 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:03.143+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:03.602 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:03.600+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:04.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:04.151+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:04.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:04.557+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:05.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:05.171+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:05.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:05.552+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:05.772 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:03:05.776 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:03:06.152 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:06.150+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:06.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:06.533+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:07.133 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:07.131+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:07.500 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:07.498+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:08.116 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:08.114+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:08.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:08.497+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:09.109 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:09.107+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:09.477 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:09.474+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:10.149 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:10.147+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:10.429 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:10.427+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:11.120 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:11.118+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:11.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:11.413+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:12.157 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:12.155+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:12.388 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:12.385+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:13.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:13.152+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:13.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:13.385+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:13.649 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:03:13.647+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:03:14.151 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:14.149+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:14.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:14.428+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:15.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:15.152+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:15.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:15.404+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:16.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:16.128+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:16.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:16.360+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:17.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:17.101+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:17.364 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:17.362+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:18.094 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:18.092+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:18.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:18.352+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:19.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:19.053+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:19.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:19.304+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:20.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:20.005+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:20.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:20.329+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:20.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:20.963+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:21.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:21.314+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:21.971 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:21.969+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:22.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:22.300+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:22.935 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:22.933+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:23.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:23.267+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:23.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:23.962+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:24.235 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:24.233+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:24.926 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:24.924+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:25.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:25.242+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:25.963 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:25.962+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:26.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:26.244+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:26.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:26.920+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:27.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:27.199+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:27.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:27.872+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:28.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:28.235+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:28.665 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:03:28.662+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:03:28.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:28.856+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:29.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:29.229+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:29.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:29.828+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:30.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:30.195+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:30.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:30.863+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:31.220 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:31.218+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:31.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:31.845+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:32.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:32.222+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:32.797 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:32.795+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:33.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:33.202+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:33.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:33.773+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:34.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:34.239+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:34.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:34.816+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:35.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:35.235+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:35.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:35.774+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:35.920 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:03:35.924 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:03:36.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:36.274+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:36.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:36.821+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:37.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:37.318+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:37.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:37.805+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:38.332 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:38.330+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:38.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:38.841+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:39.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:39.379+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:39.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:39.820+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:40.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:40.379+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:40.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:40.800+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:41.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:41.339+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:41.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:41.833+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:42.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:42.358+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:42.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:42.868+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:43.337 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:43.335+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:43.680 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:03:43.678+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:03:43.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:43.843+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:44.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:44.303+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:44.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:44.888+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:45.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:45.303+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:45.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:45.893+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:46.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:46.279+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:46.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:46.883+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:47.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:47.316+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:47.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:47.844+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:48.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:48.358+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:48.814 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:48.811+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:49.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:49.320+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:49.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:49.770+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:50.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:50.302+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:50.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:50.724+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:51.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:51.319+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:51.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:51.730+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:52.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:52.331+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:52.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:52.779+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:53.284 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:53.282+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:53.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:53.773+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:54.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:54.247+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:54.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:54.791+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:55.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:55.277+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:55.777 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:55.774+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:56.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:56.259+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:56.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:56.804+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:57.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:57.302+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:57.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:57.846+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:58.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:58.294+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:58.683 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:03:58.681+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:03:58.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:58.879+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:03:59.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:03:59.302+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:03:59.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:03:59.891+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:00.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:00.336+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:00.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:00.884+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:01.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:01.308+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:01.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:01.861+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:02.356 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:02.354+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:02.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:02.822+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:03.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:03.334+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:03.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:03.844+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:04.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:04.322+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:04.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:04.840+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:05.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:05.352+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:05.884 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:05.882+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:06.059 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:04:06.062 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:04:06.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:06.344+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:06.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:06.874+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:07.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:07.377+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:07.907 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:07.905+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:08.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:08.414+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:08.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:08.894+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:09.460 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:09.458+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:09.913 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:09.911+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:10.477 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:10.476+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:10.884 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:10.882+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:11.489 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:11.487+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:11.913 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:11.911+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:12.489 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:12.488+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:12.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:12.956+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:13.475 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:13.473+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:13.699 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:04:13.697+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:04:13.987 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:13.985+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:14.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:14.523+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:14.972 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:14.970+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:15.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:15.513+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:16.021 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:16.020+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:16.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:16.540+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:16.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:16.982+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:17.517 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:17.515+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:18.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:18.014+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:18.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:18.495+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:19.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:19.032+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:19.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:19.530+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:19.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:19.983+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:20.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:20.577+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:20.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:20.983+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:21.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:21.627+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:21.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:21.993+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:22.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:22.669+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:22.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:22.980+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:23.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:23.678+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:23.969 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:23.967+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:24.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:24.721+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:24.986 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:24.984+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:25.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:25.713+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:25.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:25.948+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:26.691 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:26.689+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:26.954 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:26.952+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:27.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:27.646+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:27.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:27.929+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:28.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:28.678+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:28.715 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:04:28.713+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:04:28.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:28.949+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:29.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:29.724+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:29.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:29.934+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:30.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:30.724+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:30.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:30.901+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:31.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:31.705+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:31.898 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:31.895+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:32.746 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:32.743+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:32.871 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:32.869+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:33.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:33.716+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:33.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:33.869+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:34.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:34.764+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:34.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:34.859+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:35.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:35.724+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:35.880 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:35.878+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:36.266 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:04:36.269 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:04:36.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:36.721+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:36.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:36.837+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:37.744 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:37.741+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:37.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:37.884+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:38.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:38.718+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:38.899 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:38.897+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:39.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:39.681+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:39.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:39.860+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:40.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:40.656+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:40.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:40.909+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:41.689 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:41.686+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:41.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:41.863+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:42.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:42.690+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:42.817 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:42.815+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:43.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:43.717+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:43.731 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:04:43.728+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:04:43.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:43.835+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:44.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:44.726+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:44.797 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:44.795+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:45.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:45.736+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:45.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:45.759+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:46.717 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:46.714+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:46.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:46.803+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:47.754 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:47.751+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:47.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:47.758+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:48.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:48.730+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:48.752 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:48.748+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:49.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:49.734+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:49.749 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:49.746+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:50.707 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:50.705+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:50.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:50.740+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:51.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:51.693+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:51.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:51.741+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:52.674 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:52.672+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:52.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:52.731+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:53.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:53.682+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:53.692 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:53.690+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:54.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:54.669+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:54.720 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:54.718+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:55.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:55.678+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:55.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:55.699+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:56.678 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:56.676+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:56.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:56.723+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:57.681 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:57.679+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:57.720 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:57.718+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:58.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:58.718+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:04:58.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:58.726+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:58.746 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:04:58.744+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:04:59.734 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:04:59.732+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:04:59.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:04:59.741+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:00.736 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:00.734+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:00.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:00.784+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:01.730 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:01.728+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:01.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:01.775+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:02.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:02.758+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:02.784 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:02.782+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:03.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:03.743+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:03.772 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:03.771+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:04.738 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:04.736+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:04.797 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:04.795+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:05.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:05.693+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:05.763 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:05.761+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:06.395 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:05:06.399 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:05:06.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:06.693+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:06.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:06.730+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:07.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:07.717+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:07.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:07.733+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:08.691 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:08.690+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:08.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:08.773+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:09.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:09.694+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:09.817 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:09.815+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:10.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:10.736+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:10.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:10.782+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:11.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:11.716+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:11.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:11.805+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:12.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:12.730+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:12.790 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:12.788+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:13.724 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:13.722+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:13.762 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:05:13.760+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:05:13.826 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:13.824+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:14.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:14.739+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:14.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:14.794+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:15.766 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:15.765+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:15.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:15.778+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:16.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:16.799+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:16.816 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:16.814+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:17.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:17.834+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:17.856 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:17.854+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:18.830 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:18.828+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:18.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:18.840+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:19.804 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:19.802+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:19.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:19.816+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:20.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:20.768+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:20.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:20.792+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:21.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:21.783+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:21.839 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:21.837+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:22.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:22.811+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:22.823 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:22.822+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:23.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:23.804+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:23.814 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:23.812+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:24.790 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:24.787+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:24.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:24.802+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:25.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:25.777+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:25.807 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:25.805+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:26.767 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:26.765+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:26.841 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:26.839+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:27.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:27.735+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:27.876 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:27.873+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:28.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:28.727+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:28.778 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:05:28.776+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:05:28.841 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:28.838+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:29.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:29.739+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:29.998 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:29.825+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:30.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:30.734+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:30.804 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:30.801+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:31.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:31.742+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:31.840 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:31.837+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:32.704 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:32.701+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:32.797 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:32.795+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:33.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:33.737+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:33.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:33.745+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:34.720 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:34.717+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:34.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:34.783+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:35.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:35.741+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:35.768 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:35.766+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:36.533 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:05:36.537 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:05:36.764 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:36.762+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:36.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:36.779+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:37.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:37.804+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:37.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:37.808+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:38.768 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:38.766+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:38.803 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:38.800+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:39.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:39.754+0000 7f88d0ac1700 -1 osd.4 122 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:39.814 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:39.812+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:39.870 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-03T21:05:39.868+0000 7fcc971ac700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T21:05:39.885 INFO:tasks.ceph.mds.f.smithi186.stderr: -1> 2022-04-03T21:05:39.868+0000 7fcc971ac700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T21:05:39.907 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-03T21:05:39.932 INFO:tasks.ceph.mon.b.smithi186.stderr:2022-04-03T21:05:39.929+0000 7f245d12f700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-04-03T21:05:40.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:40.765+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:40.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:40.819+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:41.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:41.719+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:41.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:41.828+0000 7fd3b190b700 -1 osd.6 122 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:42.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:42.672+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:42.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:42.797+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:43.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:43.643+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:43.794 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:05:43.792+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:05:43.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:43.834+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:44.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:44.676+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 12 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:44.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:44.789+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:45.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:45.720+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:45.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:45.739+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:46.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:46.696+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:46.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:46.714+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:47.675 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:47.673+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:47.724 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:47.721+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:48.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:48.683+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:48.704 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:48.702+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:49.691 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:49.689+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:49.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:49.700+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:50.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:50.715+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:50.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:50.748+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:51.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:51.724+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:51.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:51.754+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:52.708 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:52.706+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:52.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:52.756+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:53.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:53.755+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:53.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:53.789+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:54.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:54.756+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:54.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:54.800+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:55.746 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:55.744+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:55.819 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:55.816+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:56.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:56.716+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:56.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:56.800+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:57.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:57.696+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:57.787 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:57.785+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:58.746 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:58.744+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:58.810 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:05:58.807+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:05:58.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:58.812+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:05:59.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:05:59.741+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:05:59.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:05:59.827+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:00.784 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:00.781+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:00.867 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:00.865+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:01.825 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:01.823+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:01.913 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:01.911+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:02.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:02.789+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:02.889 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:02.887+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:03.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:03.778+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:03.905 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:03.903+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:04.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:04.800+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:04.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:04.868+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:05.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:05.820+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:05.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:05.874+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:06.672 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:06:06.676 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:06:06.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:06.840+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:06.856 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:06.855+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:07.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:07.818+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:07.852 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:07.850+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:08.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:08.800+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:08.844 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:08.842+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:09.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:09.778+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:09.894 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:09.892+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:10.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:10.748+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:10.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:10.858+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:11.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:11.778+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:11.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:11.828+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:12.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:12.818+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:12.828 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:12.826+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:13.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:13.779+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:13.825 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:06:13.823+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:06:13.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:13.841+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:14.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:14.758+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:14.823 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:14.822+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:15.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:15.804+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:15.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:15.814+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:16.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:16.801+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:16.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:16.842+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:17.821 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:17.818+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:17.844 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:17.842+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:18.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:18.820+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:18.870 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:18.867+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:19.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:19.804+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:19.893 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:19.890+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:20.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:20.841+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:20.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:20.907+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:21.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:21.863+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:21.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:21.912+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:22.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:22.846+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:22.946 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:22.944+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:23.880 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:23.877+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:23.927 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:23.924+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:24.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:24.874+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:24.965 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:24.962+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:25.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:25.861+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:25.997 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:25.995+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:26.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:26.876+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:26.988 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:26.986+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:27.834 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:27.832+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:27.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:27.992+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:28.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:28.782+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:28.841 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:06:28.839+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:06:28.963 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:28.960+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:29.817 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:29.814+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:29.942 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:29.940+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:30.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:30.843+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:30.977 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:30.975+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:31.880 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:31.878+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:32.027 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:32.024+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:32.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:32.876+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:33.009 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:33.006+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:33.854 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:33.851+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:34.009 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:34.007+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:34.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:34.837+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:35.046 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:35.044+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:35.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:35.800+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:36.006 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:36.004+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:36.770 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:06:36.774 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:06:36.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:36.810+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:36.982 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:36.980+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:37.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:37.797+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:38.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:38.000+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:38.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:38.833+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:39.041 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:39.039+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:39.871 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:39.868+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:40.022 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:40.020+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:40.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:40.884+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:41.045 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:41.043+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:41.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:41.909+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:42.069 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:42.067+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:42.910 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:42.908+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:43.087 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:43.084+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:43.857 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:06:43.855+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:06:43.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:43.886+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:44.069 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:44.067+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:44.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:44.866+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:45.050 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:45.048+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:45.902 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:45.900+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:46.015 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:46.013+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:46.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:46.921+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:47.001 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:46.999+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:47.914 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:47.911+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:47.966 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:47.963+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:48.943 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:48.941+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:48.988 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:48.986+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:49.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:49.926+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:49.950 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:49.948+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:50.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:50.945+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:50.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:50.994+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:51.905 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:51.903+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:52.008 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:52.006+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:52.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:52.884+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:52.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:52.977+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:53.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:53.915+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:54.017 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:54.015+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:54.925 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:54.923+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:55.061 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:55.059+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:55.934 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:55.932+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:56.019 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:56.017+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:56.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:56.971+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:56.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:56.991+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:57.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:57.935+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:58.011 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:58.009+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:58.873 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:06:58.871+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:06:58.946 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:58.945+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:06:59.047 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:06:59.046+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:06:59.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:06:59.955+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:00.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:00.053+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:00.934 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:00.932+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:01.062 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:01.061+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:01.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:01.956+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:02.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:02.079+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:02.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:02.996+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:03.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:03.113+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:04.027 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:04.025+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:04.071 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:04.069+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:04.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:04.988+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:05.024 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:05.023+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:05.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:05.946+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:06.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:06.053+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:06.912 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:07:06.916 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:07:06.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:06.980+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:07.069 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:07.067+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:07.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:07.978+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:08.078 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:08.077+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:09.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:09.008+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:09.045 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:09.044+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:10.034 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:10.033+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:10.090 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:10.089+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:11.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:11.074+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:11.094 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:11.093+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:12.079 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:12.077+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:12.113 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:12.112+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:13.100 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:13.097+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:13.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:13.110+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:13.889 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:07:13.886+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:07:14.091 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:14.089+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:14.103 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:14.101+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:15.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:15.068+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:15.072 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:15.068+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:16.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:16.046+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:16.106 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:16.104+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:17.044 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:17.041+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:17.066 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:17.063+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:18.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:18.059+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:18.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:18.113+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:19.029 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:19.026+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:19.135 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:19.132+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:20.052 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:20.049+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:20.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:20.175+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:21.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:21.077+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:21.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:21.184+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:22.093 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:22.091+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:22.142 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:22.139+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:23.106 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:23.103+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:23.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:23.109+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:24.118 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:24.115+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:24.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:24.132+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:25.101 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:25.098+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:25.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:25.157+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:26.064 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:26.062+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:26.169 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:26.167+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:27.114 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:27.112+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:27.180 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:27.177+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:28.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:28.154+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:28.186 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:28.184+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:28.905 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:07:28.902+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:07:29.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:29.169+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:29.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:29.175+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:30.132 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:30.130+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:30.200 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:30.198+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:31.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:31.110+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:31.240 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:31.237+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:32.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:32.065+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:32.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:32.205+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:33.048 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:33.045+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:33.237 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:33.234+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:34.097 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:34.095+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:34.280 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:34.278+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:35.113 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:35.111+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:35.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:35.247+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:36.127 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:36.125+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:36.206 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:36.204+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:36.972 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:07:36.976 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:07:37.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:37.113+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:37.252 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:37.250+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:38.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:38.065+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:38.224 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:38.222+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:39.053 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:39.051+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:39.209 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:39.207+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:40.066 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:40.064+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:40.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:40.255+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:41.053 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:41.050+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:41.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:41.230+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:42.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:42.009+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:42.216 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:42.214+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:43.044 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:43.042+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:43.176 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:43.174+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:43.921 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:07:43.918+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:07:44.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:44.001+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:44.168 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:44.166+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:45.028 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:45.027+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:45.209 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:45.207+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:46.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:46.001+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:46.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:46.157+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:47.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:47.003+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:47.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:47.154+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:48.036 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:48.034+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:48.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:48.196+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:49.062 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:49.060+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:49.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:49.213+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:50.095 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:50.092+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:50.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:50.181+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:51.124 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:51.122+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:51.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:51.196+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:52.109 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:52.107+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:52.238 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:52.236+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:53.102 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:53.100+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:53.217 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:53.215+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:54.094 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:54.093+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:54.209 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:54.207+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:55.093 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:55.091+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:55.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:55.159+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:56.129 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:56.127+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:56.169 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:56.168+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:57.106 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:57.104+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:57.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:57.210+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:58.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:58.153+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:58.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:58.177+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:07:58.936 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:07:58.934+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:07:59.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:07:59.111+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:07:59.214 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:07:59.213+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:00.072 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:00.070+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:00.233 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:00.231+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:01.109 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:01.108+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:01.242 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:01.240+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:02.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:02.153+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:02.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:02.242+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:03.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:03.192+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:03.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:03.234+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:04.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:04.217+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:04.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:04.226+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:05.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:05.230+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:05.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:05.242+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:06.221 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:06.219+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:06.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:06.226+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:07.162 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:08:07.166 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:08:07.223 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:07.221+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:07.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:07.243+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:08.212 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:08.210+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:08.261 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:08.259+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:09.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:09.247+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:09.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:09.263+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:10.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:10.236+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:10.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:10.307+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:11.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:11.212+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:11.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:11.333+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:12.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:12.219+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:12.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:12.337+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:13.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:13.172+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:13.354 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:13.351+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:13.952 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:08:13.950+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:08:14.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:14.206+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:14.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:14.338+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:15.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:15.164+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:15.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:15.346+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:16.151 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:16.148+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:16.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:16.307+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:17.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:17.196+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:17.327 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:17.324+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:18.206 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:18.204+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:18.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:18.336+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:19.170 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:19.167+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:19.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:19.381+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:20.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:20.188+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:20.392 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:20.389+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:21.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:21.223+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:21.438 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:21.436+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:22.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:22.197+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:22.392 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:22.390+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:23.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:23.220+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:23.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:23.379+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:23.929 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-03T21:08:23.926+0000 7f2fd761c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T21:08:23.929 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-03T21:08:23.926+0000 7f2fd661a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T21:08:24.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:24.262+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:24.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:24.391+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:25.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:25.285+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:25.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:25.373+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:26.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:26.284+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:26.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:26.323+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:27.289 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:27.287+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:27.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:27.305+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:28.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:28.277+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:28.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:28.300+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:28.968 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:08:28.966+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:08:29.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:29.251+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:29.313 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:29.311+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:30.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:30.244+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:30.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:30.355+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:31.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:31.205+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:31.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:31.375+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:32.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:32.165+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:32.365 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:32.363+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:33.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:33.206+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:33.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:33.411+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:34.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:34.204+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:34.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:34.370+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:35.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:35.181+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:35.386 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:35.384+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:36.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:36.228+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:36.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:36.433+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:37.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:37.183+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:37.232 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:08:37.236 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:08:37.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:37.443+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:38.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:38.185+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:38.480 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:38.478+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:39.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:39.177+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:39.498 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:39.495+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:08:40.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:40.208+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:40.468 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:40.465+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:08:41.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:41.227+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:41.436 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:41.434+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:08:42.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:42.245+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:42.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:42.413+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:08:43.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:43.234+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:43.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:43.404+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:08:43.969 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:08:43.967+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:08:44.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:44.252+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 6 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:44.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:44.361+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:08:45.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:45.251+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:45.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:45.334+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:08:46.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:46.245+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:46.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:46.294+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:08:47.221 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:47.219+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:47.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:47.258+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:08:48.216 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:48.215+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:08:48.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:48.266+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:49.253 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:49.251+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:08:49.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:49.304+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:50.287 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:50.285+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:08:50.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:50.296+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:51.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:51.298+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:51.300 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:51.298+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:08:52.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:52.300+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:52.333 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:52.332+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:08:53.303 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:53.302+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:08:53.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:53.339+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:54.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:54.266+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:08:54.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:54.377+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:55.300 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:55.298+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:08:55.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:55.406+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:56.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:56.280+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:08:56.378 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:56.376+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:57.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:57.286+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:08:57.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:57.372+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:58.307 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:58.306+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:08:58.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:58.418+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:08:58.985 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:08:58.983+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:08:59.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:08:59.338+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:08:59.376 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:08:59.374+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:00.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:00.325+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:00.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:00.384+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:01.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:01.352+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:01.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:01.369+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:02.345 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:02.342+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:02.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:02.372+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:03.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:03.343+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:03.364 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:03.362+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:04.301 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:04.298+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:04.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:04.377+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:05.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:05.318+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:05.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:05.388+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:06.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:06.305+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:06.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:06.375+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:07.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:07.273+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:07.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:07.358+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:07.385 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:09:07.388 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:09:08.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:08.233+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:08.325 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:08.322+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:09.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:09.188+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:09.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:09.282+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:10.162 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:10.160+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:10.301 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:10.299+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:11.191 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:11.188+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:11.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:11.297+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:12.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:12.199+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:12.274 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:12.271+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:13.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:13.228+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:13.280 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:13.277+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:14.001 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:09:13.999+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:09:14.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:14.213+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:14.274 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:14.272+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:15.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:15.220+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:15.319 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:15.316+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:16.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:16.229+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:16.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:16.299+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:17.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:17.233+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:17.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:17.300+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:18.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:18.252+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:18.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:18.280+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:19.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:19.213+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:19.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:19.300+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:20.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:20.185+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:20.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:20.256+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:21.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:21.192+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:21.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:21.303+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:22.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:22.227+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:22.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:22.289+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:23.217 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:23.214+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:23.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:23.266+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:24.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:24.191+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:24.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:24.283+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:25.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:25.220+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:25.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:25.300+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:26.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:26.239+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:26.322 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:26.320+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:27.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:27.275+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:27.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:27.287+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:28.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:28.262+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:28.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:28.307+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:29.017 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:09:29.015+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:09:29.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:29.276+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:29.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:29.303+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:30.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:30.262+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:30.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:30.308+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:31.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:31.296+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:31.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:31.328+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:32.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:32.264+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:32.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:32.283+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:33.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:33.241+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:33.319 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:33.317+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:34.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:34.252+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:34.338 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:34.335+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:35.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:35.221+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:35.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:35.360+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:36.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:36.200+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:36.397 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:36.395+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:37.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:37.200+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:37.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:37.415+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:37.532 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:09:37.536 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:09:38.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:38.247+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:38.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:38.412+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:39.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:39.293+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:39.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:39.435+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:40.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:40.288+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:40.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:40.481+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:41.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:41.293+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:41.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:41.510+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:42.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:42.339+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:42.556 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:42.554+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:43.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:43.308+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:43.568 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:43.567+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:44.032 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:09:44.030+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:09:44.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:44.337+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:44.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:44.519+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:45.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:45.309+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:45.530 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:45.529+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:46.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:46.293+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:46.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:46.558+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:47.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:47.275+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:47.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:47.552+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:48.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:48.235+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:48.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:48.545+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:49.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:49.188+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:49.566 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:49.565+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:50.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:50.161+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:50.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:50.537+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:51.176 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:51.175+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:51.562 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:51.560+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:52.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:52.165+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:52.526 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:52.524+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:53.136 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:53.135+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:53.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:53.493+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:54.118 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:54.116+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:54.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:54.454+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:55.146 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:55.144+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:55.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:55.405+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:56.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:56.110+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:56.424 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:56.422+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:57.154 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:57.152+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:57.473 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:57.470+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:58.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:58.190+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:58.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:58.494+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:09:59.049 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:09:59.046+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:09:59.170 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:09:59.168+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:09:59.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:09:59.483+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:00.140 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:00.137+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:00.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:00.512+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:01.113 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:01.110+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:01.504 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:01.501+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:02.099 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:02.097+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:02.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:02.478+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:03.144 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:03.142+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:03.446 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:03.444+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:04.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:04.156+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:04.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:04.486+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:05.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:05.163+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:05.496 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:05.493+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:06.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:06.184+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:06.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:06.536+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:07.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:07.185+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:07.490 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:07.488+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:07.672 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:10:07.675 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:10:08.146 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:08.144+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:08.525 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:08.523+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:09.106 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:09.104+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:09.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:09.538+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:10.114 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:10.112+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:10.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:10.576+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:11.132 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:11.130+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:11.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:11.546+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:12.152 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:12.150+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:12.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:12.577+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:13.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:13.197+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:13.561 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:13.558+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:14.064 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:10:14.061+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:10:14.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:14.237+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:14.610 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:14.607+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:15.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:15.252+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:15.654 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:15.651+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:16.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:16.270+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:16.609 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:16.606+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:17.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:17.261+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:17.608 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:17.606+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:18.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:18.273+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:18.634 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:18.632+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:19.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:19.319+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:19.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:19.646+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:20.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:20.369+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:20.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:20.693+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:21.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:21.344+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:21.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:21.692+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:22.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:22.301+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:22.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:22.678+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:23.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:23.331+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:23.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:23.660+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:24.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:24.366+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:24.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:24.675+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:25.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:25.344+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:25.718 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:25.716+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:26.301 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:26.299+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:26.722 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:26.720+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:27.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:27.305+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:27.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:27.723+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:28.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:28.334+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:28.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:28.757+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:29.080 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:10:29.078+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:10:29.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:29.297+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:29.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:29.804+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:30.283 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:30.281+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:30.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:30.798+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:31.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:31.267+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:31.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:31.805+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:32.235 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:32.233+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:32.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:32.829+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:33.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:33.230+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:33.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:33.783+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:34.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:34.259+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:34.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:34.780+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:35.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:35.240+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:35.771 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:35.768+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:36.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:36.286+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:36.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:36.768+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:37.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:37.255+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:37.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:37.743+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:37.819 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:10:37.823 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:10:38.221 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:38.219+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:38.746 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:38.744+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:39.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:39.258+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:39.757 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:39.756+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:40.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:40.258+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:40.764 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:40.762+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:41.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:41.271+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:41.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:41.793+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:42.226 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:42.225+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:42.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:42.828+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:43.221 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:43.219+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:43.809 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:43.807+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:44.096 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:10:44.094+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:10:44.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:44.261+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:44.834 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:44.833+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:45.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:45.303+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:45.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:45.783+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:46.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:46.336+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:46.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:46.833+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:47.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:47.340+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:47.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:47.875+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:48.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:48.360+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:48.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:48.828+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:49.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:49.379+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:49.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:49.801+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:50.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:50.379+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:50.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:50.777+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:51.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:51.357+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:51.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:51.777+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:52.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:52.308+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:52.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:52.800+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:53.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:53.353+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:53.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:53.751+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:54.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:54.324+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:54.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:54.770+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:55.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:55.352+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:55.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:55.803+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:56.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:56.365+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:56.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:56.757+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:57.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:57.360+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:57.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:57.776+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:58.401 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:58.398+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:58.792 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:58.790+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:10:59.112 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:10:59.109+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:10:59.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:10:59.390+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:10:59.769 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:10:59.767+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:00.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:00.431+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:00.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:00.723+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:01.466 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:01.463+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:01.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:01.693+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:02.507 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:02.505+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:02.653 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:02.651+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:03.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:03.513+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:03.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:03.641+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:04.558 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:04.556+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:04.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:04.611+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:05.508 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:05.506+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:05.649 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:05.647+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:06.552 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:06.550+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:06.666 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:06.664+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:07.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:07.558+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:07.654 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:07.652+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:07.882 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T21:11:07.879+0000 7f2e43b1f700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T21:11:07.882 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T21:11:07.879+0000 7f2e4331e700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T21:11:07.975 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:11:07.979 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:11:08.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:08.543+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:08.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:08.661+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:09.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:09.559+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:09.696 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:09.694+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:10.595 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:10.593+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:10.730 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:10.728+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:11.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:11.613+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:11.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:11.721+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:12.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:12.590+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:12.683 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:12.680+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:13.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:13.636+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:13.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:13.688+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:14.128 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:11:14.126+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:11:14.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:14.587+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:14.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:14.710+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:15.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:15.635+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:15.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:15.712+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:16.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:16.613+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:16.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:16.756+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:17.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:17.612+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:17.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:17.777+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:18.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:18.624+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:18.756 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:18.753+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:19.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:19.613+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:19.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:19.787+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:20.577 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:20.574+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:20.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:20.780+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:21.606 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:21.604+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:21.751 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:21.749+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:22.583 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:22.581+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:22.798 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:22.796+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:23.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:23.630+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:23.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:23.844+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:24.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:24.625+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:24.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:24.844+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:25.673 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:25.671+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:25.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:25.825+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:26.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:26.690+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:26.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:26.798+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:27.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:27.738+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:27.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:27.776+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:28.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:28.716+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:28.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:28.810+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:29.144 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:11:29.142+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:11:29.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:29.683+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:29.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:29.796+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:30.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:30.711+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:30.809 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:30.807+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:31.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:31.690+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:31.773 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:31.772+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:32.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:32.721+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:32.746 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:32.744+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:33.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:33.698+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:33.734 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:33.733+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:34.708 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:34.707+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:34.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:34.780+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:35.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:35.731+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:35.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:35.749+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:36.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:36.711+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:36.747 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:36.746+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:37.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:37.711+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:37.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:37.760+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:38.100 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:11:38.104 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:11:38.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:38.700+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:38.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:38.728+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:39.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:39.721+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:39.767 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:39.764+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:11:40.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:40.770+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:40.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:40.790+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:11:41.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:41.753+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:11:41.766 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:41.764+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:42.707 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:42.706+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:11:42.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:42.778+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:43.749 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:43.747+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:11:43.754 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:43.753+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:44.159 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:11:44.156+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:11:44.761 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:44.760+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:44.767 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:44.766+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:11:45.722 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:45.720+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:11:45.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:45.748+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:46.697 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:46.695+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:11:46.788 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:46.785+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:47.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:47.690+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:11:47.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:47.776+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:48.696 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:48.694+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:11:48.761 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:48.758+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:49.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:49.653+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:11:49.808 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:49.805+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:50.633 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:50.630+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:11:50.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:50.829+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:51.617 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:51.615+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:11:51.870 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:51.867+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:52.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:52.584+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:11:52.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:52.862+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:53.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:53.546+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:11:53.818 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:53.815+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:54.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:54.569+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:11:54.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:54.819+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:55.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:55.574+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:11:55.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:55.776+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:56.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:56.613+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:11:56.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:56.803+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:57.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:57.626+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:11:57.769 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:57.767+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:58.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:58.653+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:11:58.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:58.741+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:11:59.176 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:11:59.173+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:11:59.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:11:59.638+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:11:59.788 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:11:59.786+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:00.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:00.613+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:00.839 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:00.836+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:01.582 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:01.579+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:01.801 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:01.799+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:02.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:02.609+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:02.851 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:02.849+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:03.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:03.654+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:03.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:03.811+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:04.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:04.641+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:04.857 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:04.855+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:05.653 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:05.651+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:05.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:05.895+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:06.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:06.663+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:06.906 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:06.903+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:07.697 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:07.695+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:07.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:07.875+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:08.229 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:12:08.233 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:12:08.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:08.665+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:08.894 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:08.892+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:09.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:09.630+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:09.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:09.870+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:10.659 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:10.657+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:10.836 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:10.834+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:11.636 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:11.634+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:11.817 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:11.815+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:12.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:12.618+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:12.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:12.861+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:13.606 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:13.604+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:13.901 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:13.899+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:14.191 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:12:14.189+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:12:14.591 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:14.588+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:14.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:14.931+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:15.631 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:15.629+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:15.977 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:15.974+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:16.585 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:16.583+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:17.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:17.001+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:17.612 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:17.609+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:17.967 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:17.964+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:18.660 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:18.658+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:18.928 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:18.926+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:19.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:19.638+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:19.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:19.943+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:20.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:20.620+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:20.927 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:20.925+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:21.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:21.619+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:21.974 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:21.972+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:22.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:22.582+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:22.954 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:22.952+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:23.599 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:23.597+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:23.966 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:23.964+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:24.574 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:24.572+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:24.982 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:24.980+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:25.570 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:25.568+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:25.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:25.932+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:26.582 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:26.581+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:26.938 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:26.936+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:27.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:27.627+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:27.913 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:27.912+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:28.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:28.623+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:28.951 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:28.950+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:29.207 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:12:29.204+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:12:29.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:29.601+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:29.912 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:29.910+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:30.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:30.577+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:30.931 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:30.929+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:31.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:31.537+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:31.937 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:31.936+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:32.564 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:32.562+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:32.917 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:32.916+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:33.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:33.593+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:33.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:33.895+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:34.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:34.575+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:34.859 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:34.857+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:35.585 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:35.583+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:35.859 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:35.857+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:36.564 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:36.562+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:36.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:36.818+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:37.597 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:37.595+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:37.775 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:37.774+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:38.384 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:12:38.388 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:12:38.561 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:38.560+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:38.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:38.776+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:39.522 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:39.520+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:39.752 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:39.751+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:40.556 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:40.555+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:40.801 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:40.799+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:41.565 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:41.562+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:41.790 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:41.787+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:42.533 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:42.531+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:42.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:42.826+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:43.507 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:43.504+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:43.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:43.821+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:44.223 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:12:44.221+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:12:44.519 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:44.515+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:44.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:44.840+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:45.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:45.477+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:45.816 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:45.813+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:46.508 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:46.505+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:46.788 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:46.785+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:47.487 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:47.485+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:47.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:47.746+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:48.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:48.533+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:48.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:48.788+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:49.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:49.488+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:49.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:49.777+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:50.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:50.455+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:50.812 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:50.809+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:51.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:51.499+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:51.764 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:51.761+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:52.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:52.526+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:52.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:52.777+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:53.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:53.490+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:53.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:53.738+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:54.533 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:54.530+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:54.727 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:54.724+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:55.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:55.512+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:55.759 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:55.756+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:56.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:56.473+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:56.752 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:56.750+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:57.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:57.456+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:57.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:57.735+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:58.487 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:58.485+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:58.689 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:58.686+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:12:59.239 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:12:59.236+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:12:59.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:12:59.458+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:12:59.689 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:12:59.686+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:00.433 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:00.431+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:00.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:00.712+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:01.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:01.402+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:01.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:01.687+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:02.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:02.413+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:02.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:02.667+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:03.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:03.451+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:03.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:03.636+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:04.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:04.408+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:04.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:04.625+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:05.457 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:05.454+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:05.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:05.655+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:06.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:06.453+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:06.608 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:06.606+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:07.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:07.481+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:07.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:07.566+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:08.498 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:13:08.501 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:13:08.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:08.511+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:08.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:08.546+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:09.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:09.530+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:09.583 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:09.581+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:10.484 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:10.482+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:10.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:10.563+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:11.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:11.510+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:11.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:11.567+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:12.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:12.501+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:12.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:12.585+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:13.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:13.453+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:13.603 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:13.601+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:14.254 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:13:14.251+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:13:14.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:14.501+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:14.600 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:14.599+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:15.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:15.529+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:15.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:15.645+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:16.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:16.549+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:16.673 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:16.671+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:17.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:17.518+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:17.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:17.684+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:18.569 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:18.567+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:18.655 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:18.653+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:19.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:19.537+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:19.666 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:19.665+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:20.574 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:20.572+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:20.704 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:20.703+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:21.556 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:21.554+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:21.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:21.747+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:22.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:22.532+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:22.705 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:22.703+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:23.561 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:23.559+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:23.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:23.655+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:24.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:24.536+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:24.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:24.639+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:25.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:25.508+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:25.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:25.652+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:26.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:26.479+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:26.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:26.630+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:27.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:27.477+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:27.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:27.622+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:28.507 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:28.505+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:28.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:28.642+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:29.266 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:13:29.264+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:13:29.482 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:29.480+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:29.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:29.623+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:30.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:30.442+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:30.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:30.611+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:31.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:31.417+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:31.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:31.563+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:32.453 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:32.451+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:32.600 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:32.599+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:33.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:33.429+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:33.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:33.560+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:34.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:34.449+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:34.519 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:34.518+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:35.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:35.415+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:35.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:35.490+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:36.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:36.461+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:36.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:36.479+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:37.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:37.441+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:37.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:37.493+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:38.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:38.459+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:38.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:38.528+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:38.604 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:13:38.608 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:13:39.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:39.421+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:39.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:39.511+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:40.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:40.448+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:40.474 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:40.472+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:41.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:41.482+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:41.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:41.511+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:42.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:42.452+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:42.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:42.519+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:43.462 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:43.459+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:43.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:43.493+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:44.277 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:13:44.275+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:13:44.438 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:44.435+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:44.471 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:44.468+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:45.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:45.435+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:45.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:45.481+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:46.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:46.436+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:46.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:46.531+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:47.400 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:47.398+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:47.508 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:47.506+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:48.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:48.361+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:48.528 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:48.525+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:49.410 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:49.408+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:49.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:49.539+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:50.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:50.432+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:50.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:50.559+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:51.436 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:51.434+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:51.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:51.582+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:52.410 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:52.408+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:52.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:52.534+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:53.365 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:53.363+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:53.517 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:53.515+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:54.325 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:54.323+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:54.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:54.511+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:55.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:55.309+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:55.479 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:55.477+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:56.331 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:56.329+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:56.464 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:56.462+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:57.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:57.378+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:57.451 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:57.449+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:58.350 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:58.348+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:13:58.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:58.401+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:59.293 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:13:59.290+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:13:59.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:13:59.355+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:13:59.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:13:59.391+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:00.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:00.371+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:00.408 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:00.406+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:01.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:01.359+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:01.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:01.388+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:02.338 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:02.336+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:02.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:02.434+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:03.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:03.371+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:03.432 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:03.430+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:04.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:04.412+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:04.431 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:04.428+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:05.370 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:05.368+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:05.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:05.423+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:06.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:06.369+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:06.401 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:06.399+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:07.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:07.388+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:07.412 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:07.410+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:08.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:08.365+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:08.401 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:08.399+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:08.760 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:14:08.764 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:14:09.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:09.356+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:09.410 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:09.408+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:10.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:10.344+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:10.428 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:10.426+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:11.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:11.364+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:11.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:11.376+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:12.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:12.342+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:12.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:12.376+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:13.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:13.334+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:13.392 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:13.390+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:14.309 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:14:14.307+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:14:14.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:14.344+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:14.347 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:14.346+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:15.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:15.304+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:15.344 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:15.342+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:16.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:16.292+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:16.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:16.300+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:17.253 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:17.252+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:17.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:17.279+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:18.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:18.241+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:18.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:18.290+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:19.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:19.280+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:19.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:19.326+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:20.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:20.278+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:20.332 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:20.331+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:21.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:21.263+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:21.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:21.322+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:22.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:22.280+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:22.332 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:22.331+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:23.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:23.286+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:23.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:23.370+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:24.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:24.291+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:24.347 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:24.345+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:25.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:25.332+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:25.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:25.360+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:26.342 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:26.341+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:26.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:26.372+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:27.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:27.339+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:27.421 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:27.419+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:28.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:28.335+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:28.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:28.434+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:29.325 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:14:29.323+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:14:29.386 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:29.384+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:29.479 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:29.477+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:30.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:30.372+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:30.478 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:30.475+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:31.343 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:31.340+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:31.468 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:31.466+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:32.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:32.361+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:32.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:32.449+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:33.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:33.319+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:33.477 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:33.474+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:34.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:34.333+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:34.474 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:34.472+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:35.325 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:35.323+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:35.493 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:35.490+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:36.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:36.306+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:36.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:36.496+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:37.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:37.276+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:37.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:37.546+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:38.236 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:38.233+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:38.564 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:38.562+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:38.900 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:14:38.903 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:14:39.240 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:39.238+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:39.520 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:39.518+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:40.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:40.257+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:40.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:40.513+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:41.270 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:41.268+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:41.473 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:41.471+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:42.283 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:42.280+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:42.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:42.449+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:43.298 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:43.296+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:43.438 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:43.436+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:44.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:44.256+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:44.341 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:14:44.338+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:14:44.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:44.465+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:45.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:45.255+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:45.473 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:45.470+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:46.234 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:46.231+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:46.475 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:46.472+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:47.242 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:47.239+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:47.463 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:47.460+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:48.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:48.233+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:48.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:48.430+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:49.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:49.270+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:49.453 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:49.450+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:50.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:50.242+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:50.415 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:50.412+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:51.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:51.266+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:51.442 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:51.440+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:52.287 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:52.285+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:52.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:52.453+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:53.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:53.293+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:53.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:53.404+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:54.298 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:54.296+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:54.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:54.356+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:55.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:55.314+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:55.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:55.395+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:56.354 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:56.352+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:56.401 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:56.398+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:57.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:57.307+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:57.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:57.353+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:58.307 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:58.305+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:58.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:58.371+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:59.306 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:14:59.304+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:14:59.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:14:59.325+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:14:59.357 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:14:59.355+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:15:00.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:00.270+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:00.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:00.294+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:01.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:01.253+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:01.258 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:01.256+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:02.266 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:02.264+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:02.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:02.289+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:03.243 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:03.241+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:03.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:03.272+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:04.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:04.229+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:04.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:04.259+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:05.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:05.208+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:05.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:05.245+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:06.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:06.235+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:06.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:06.266+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:07.269 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:07.267+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:07.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:07.271+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:08.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:08.266+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:08.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:08.313+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:09.037 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:15:09.042 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:15:09.290 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:09.289+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:09.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:09.310+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:10.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:10.247+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:10.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:10.359+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:11.221 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:11.219+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:11.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:11.364+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:12.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:12.181+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:12.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:12.359+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:13.216 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:13.215+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:13.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:13.317+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:14.237 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:14.235+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:14.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:14.360+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:14.372 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:15:14.370+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:15:15.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:15.252+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:15.364 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:15.362+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:16.245 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:16.244+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:16.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:16.324+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:17.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:17.218+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:17.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:17.328+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:18.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:18.181+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:18.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:18.364+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:19.194 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:19.192+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:19.356 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:19.354+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:20.194 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:20.193+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:20.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:20.350+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:21.214 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:21.212+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:21.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:21.343+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:22.243 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:22.241+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:22.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:22.352+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:23.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:23.224+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:23.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:23.364+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:24.255 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:24.253+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:24.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:24.321+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:25.300 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:25.298+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:25.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:25.310+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:26.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:26.261+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:26.293 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:26.290+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:27.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:27.243+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:27.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:27.338+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:28.252 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:28.249+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:28.384 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:28.382+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:29.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:29.272+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:29.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:29.380+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:29.388 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:15:29.385+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:15:30.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:30.223+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:30.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:30.414+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:31.217 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:31.215+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:31.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:31.401+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:32.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:32.206+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:32.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:32.432+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:33.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:33.213+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:33.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:33.453+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:34.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:34.172+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:34.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:34.404+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:35.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:35.207+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:35.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:35.416+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:36.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:36.202+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:36.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:36.371+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:37.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:37.185+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:37.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:37.402+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:38.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:38.221+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:38.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:38.359+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:39.187 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:15:39.191 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:15:39.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:39.236+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:39.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:39.321+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:40.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:40.227+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:40.333 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:40.330+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:41.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:41.237+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:41.356 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:41.354+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:42.283 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:42.281+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:42.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:42.320+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:43.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:43.327+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:43.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:43.351+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:44.313 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:44.310+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:44.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:44.322+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:44.404 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:15:44.402+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:15:45.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:45.319+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:45.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:45.334+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:46.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:46.311+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:46.332 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:46.329+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:47.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:47.289+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:47.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:47.304+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:48.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:48.242+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:48.347 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:48.344+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:49.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:49.261+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:49.386 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:49.384+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:50.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:50.228+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:50.370 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:50.367+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:51.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:51.255+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:51.396 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:51.394+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:52.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:52.270+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:52.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:52.378+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:53.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:53.270+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:53.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:53.402+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:54.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:54.251+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:54.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:54.412+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:55.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:55.217+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:55.386 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:55.383+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:56.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:56.199+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:56.355 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:56.354+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:57.162 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:57.160+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:57.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:57.374+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:58.133 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:58.131+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:58.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:58.396+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:59.101 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:15:59.098+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:15:59.408 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:15:59.406+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:15:59.420 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:15:59.418+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:15:59.922 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:15:59.920+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:15:59.920921+0000) 2022-04-03T21:16:00.151 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:00.149+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:00.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:00.379+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:00.922 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:00.920+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:00.921144+0000) 2022-04-03T21:16:01.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:01.161+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:01.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:01.425+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:01.922 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:01.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:01.921364+0000) 2022-04-03T21:16:02.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:02.186+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:02.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:02.389+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:02.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:02.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:02.921532+0000) 2022-04-03T21:16:03.143 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:03.141+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:03.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:03.438+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:03.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:03.920+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:03.921729+0000) 2022-04-03T21:16:04.151 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:04.150+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:04.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:04.423+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:04.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:04.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:04.921941+0000) 2022-04-03T21:16:05.195 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:05.194+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:05.403 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:05.401+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:05.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:05.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:05.922128+0000) 2022-04-03T21:16:06.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:06.175+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:06.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:06.375+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:06.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:06.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:06.922250+0000) 2022-04-03T21:16:07.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:07.196+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:07.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:07.386+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:07.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:07.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:07.922456+0000) 2022-04-03T21:16:08.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:08.162+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:08.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:08.423+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:08.924 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:08.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:08.922666+0000) 2022-04-03T21:16:09.133 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:09.131+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:09.326 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:16:09.330 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:16:09.426 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:09.424+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:09.924 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:09.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:09.922943+0000) 2022-04-03T21:16:10.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:10.165+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:10.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:10.443+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:10.924 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:10.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:10.923125+0000) 2022-04-03T21:16:11.212 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:11.210+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:11.398 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:11.396+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:11.924 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:11.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:11.923305+0000) 2022-04-03T21:16:12.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:12.255+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:12.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:12.415+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:12.924 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:12.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:12.923509+0000) 2022-04-03T21:16:13.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:13.237+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:13.410 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:13.408+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:13.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:13.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:13.923721+0000) 2022-04-03T21:16:14.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:14.261+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:14.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:14.374+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:14.436 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:16:14.433+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:16:14.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:14.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:14.923928+0000) 2022-04-03T21:16:15.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:15.304+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:15.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:15.327+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:15.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:15.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:15.924125+0000) 2022-04-03T21:16:16.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:16.261+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:16.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:16.361+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:16.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:16.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:16.924279+0000) 2022-04-03T21:16:17.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:17.272+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:17.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:17.405+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:17.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:17.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:17.924426+0000) 2022-04-03T21:16:18.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:18.303+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:18.370 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:18.368+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:18.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:18.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:18.924596+0000) 2022-04-03T21:16:19.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:19.350+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:19.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:19.392+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:19.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:19.924+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:19.924827+0000) 2022-04-03T21:16:20.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:20.319+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:20.347 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:20.345+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:20.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:20.924+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:20.925019+0000) 2022-04-03T21:16:21.306 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:21.303+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:21.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:21.323+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:21.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:21.924+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:21.925187+0000) 2022-04-03T21:16:22.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:22.257+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:22.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:22.294+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:22.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:22.924+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:22.925352+0000) 2022-04-03T21:16:23.217 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:23.214+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:23.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:23.317+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:23.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:23.924+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:23.925562+0000) 2022-04-03T21:16:24.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:24.225+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:24.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:24.322+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:24.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:24.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:24.925796+0000) 2022-04-03T21:16:25.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:25.186+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:25.364 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:25.362+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:25.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:25.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:25.926016+0000) 2022-04-03T21:16:26.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:26.180+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:26.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:26.343+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:26.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:26.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:26.926219+0000) 2022-04-03T21:16:27.181 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:27.178+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:27.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:27.326+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:27.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:27.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:27.926428+0000) 2022-04-03T21:16:28.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:28.181+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:28.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:28.371+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:28.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:28.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:28.926655+0000) 2022-04-03T21:16:29.145 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:29.143+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:29.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:29.391+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:29.452 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:16:29.450+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:16:29.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:29.926+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:29.926868+0000) 2022-04-03T21:16:30.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:30.171+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:30.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:30.358+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:30.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:30.926+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:30.927071+0000) 2022-04-03T21:16:31.160 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:31.158+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:31.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:31.348+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:31.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:31.926+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:31.927216+0000) 2022-04-03T21:16:32.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:32.163+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:32.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:32.383+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:32.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:32.926+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:32.927435+0000) 2022-04-03T21:16:33.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:33.168+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:33.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:33.379+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:33.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:33.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:33.927629+0000) 2022-04-03T21:16:34.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:34.160+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:34.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:34.365+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:34.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:34.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:34.927835+0000) 2022-04-03T21:16:35.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:35.200+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:35.364 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:35.361+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:35.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:35.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:35.927982+0000) 2022-04-03T21:16:36.236 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:36.233+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:36.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:36.406+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:36.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:36.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:36.928123+0000) 2022-04-03T21:16:37.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:37.282+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:37.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:37.364+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:37.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:37.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:37.928331+0000) 2022-04-03T21:16:38.303 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:38.300+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:38.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:38.408+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:38.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:38.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:38.928537+0000) 2022-04-03T21:16:39.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:39.307+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:39.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:39.449+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:39.489 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:16:39.493 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:16:39.930 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:39.928+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:39.928798+0000) 2022-04-03T21:16:40.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:40.262+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:40.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:40.492+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:40.930 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:40.928+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:40.928952+0000) 2022-04-03T21:16:41.269 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:41.266+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:41.493 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:41.491+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:41.930 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:41.928+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:41.929168+0000) 2022-04-03T21:16:42.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:42.283+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:42.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:42.450+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:42.930 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:42.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:42.929307+0000) 2022-04-03T21:16:43.241 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:43.239+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:43.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:43.408+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:43.931 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:43.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:43.929533+0000) 2022-04-03T21:16:44.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:44.270+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:44.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:44.369+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:44.468 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:16:44.465+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:16:44.931 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:44.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:44.929728+0000) 2022-04-03T21:16:45.294 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:45.291+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:45.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:45.401+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:45.931 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:45.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:45.929922+0000) 2022-04-03T21:16:46.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:46.313+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:46.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:46.446+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:46.931 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:46.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:46.930100+0000) 2022-04-03T21:16:47.348 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:47.346+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:47.490 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:47.488+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:47.931 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:47.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:47.930304+0000) 2022-04-03T21:16:48.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:48.309+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:48.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:48.529+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:48.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:48.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:48.930507+0000) 2022-04-03T21:16:49.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:49.307+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:49.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:49.488+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:49.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:49.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:49.930706+0000) 2022-04-03T21:16:50.287 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:50.284+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:50.507 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:50.504+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:50.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:50.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:50.930905+0000) 2022-04-03T21:16:51.251 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:51.249+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:51.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:51.533+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:51.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:51.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:51.931100+0000) 2022-04-03T21:16:52.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:52.293+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:52.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:52.574+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:52.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:52.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:52.931300+0000) 2022-04-03T21:16:53.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:53.327+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:53.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:53.615+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:53.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:53.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:53.931507+0000) 2022-04-03T21:16:54.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:54.290+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:54.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:54.639+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:54.933 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:54.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:54.931689+0000) 2022-04-03T21:16:55.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:55.252+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:55.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:55.688+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:55.933 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:55.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:55.931898+0000) 2022-04-03T21:16:56.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:56.246+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:56.705 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:56.703+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:56.933 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:56.931+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:56.932101+0000) 2022-04-03T21:16:57.275 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:57.273+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:57.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:57.679+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:57.933 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:57.931+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:57.932307+0000) 2022-04-03T21:16:58.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:58.303+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:58.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:58.677+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:58.934 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:58.931+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:58.932529+0000) 2022-04-03T21:16:59.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:16:59.324+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:16:59.477 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:16:59.473+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:16:59.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:16:59.631+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:16:59.934 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:16:59.931+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:16:59.932754+0000) 2022-04-03T21:17:00.364 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:00.363+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:00.678 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:00.677+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:00.934 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:00.932+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:00.932930+0000) 2022-04-03T21:17:01.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:01.398+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:01.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:01.716+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:01.934 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:01.932+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:01.933166+0000) 2022-04-03T21:17:02.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:02.404+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:02.766 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:02.764+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:02.934 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:02.932+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:02.933402+0000) 2022-04-03T21:17:03.392 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:03.391+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:03.810 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:03.808+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:03.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:03.932+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:03.933647+0000) 2022-04-03T21:17:04.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:04.438+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:04.847 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:04.844+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:04.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:04.933+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:04.933851+0000) 2022-04-03T21:17:05.452 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:05.450+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:05.847 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:05.846+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:05.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:05.933+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:05.934020+0000) 2022-04-03T21:17:06.473 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:06.471+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:06.880 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:06.878+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:06.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:06.933+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:06.934179+0000) 2022-04-03T21:17:07.475 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:07.473+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:07.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:07.866+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:07.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:07.933+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:07.934377+0000) 2022-04-03T21:17:08.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:08.489+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:08.826 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:08.824+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:08.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:08.933+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:08.934547+0000) 2022-04-03T21:17:09.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:09.461+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:09.578 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:17:09.582 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:17:09.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:09.832+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:09.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:09.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:09.934819+0000) 2022-04-03T21:17:10.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:10.489+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:10.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:10.818+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:10.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:10.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:10.935016+0000) 2022-04-03T21:17:11.526 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:11.525+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:11.798 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:11.796+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:11.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:11.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:11.935219+0000) 2022-04-03T21:17:12.570 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:12.569+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:12.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:12.821+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:12.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:12.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:12.935423+0000) 2022-04-03T21:17:13.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:13.578+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:13.835 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:13.833+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:13.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:13.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:13.935621+0000) 2022-04-03T21:17:14.491 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:17:14.489+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:17:14.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:14.585+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:14.819 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:14.817+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:14.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:14.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:14.935848+0000) 2022-04-03T21:17:15.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:15.539+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:15.831 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:15.829+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:15.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:15.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:15.935998+0000) 2022-04-03T21:17:16.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:16.582+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:16.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:16.840+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:16.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:16.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:16.936195+0000) 2022-04-03T21:17:17.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:17.611+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:17.892 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:17.889+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:17.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:17.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:17.936364+0000) 2022-04-03T21:17:18.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:18.637+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:18.894 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:18.891+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:18.938 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:18.936+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:18.936557+0000) 2022-04-03T21:17:19.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:19.639+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:19.906 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:19.904+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:19.938 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:19.936+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:19.936723+0000) 2022-04-03T21:17:20.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:20.626+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:20.876 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:20.873+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:20.938 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:20.936+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:20.936944+0000) 2022-04-03T21:17:21.666 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:21.663+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:21.881 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:21.878+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:21.938 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:21.936+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:21.937148+0000) 2022-04-03T21:17:22.643 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:22.641+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:22.886 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:22.884+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:22.938 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:22.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:22.937371+0000) 2022-04-03T21:17:23.649 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:23.646+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:23.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:23.861+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:23.939 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:23.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:23.937571+0000) 2022-04-03T21:17:24.675 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:24.673+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:24.911 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:24.909+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:24.939 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:24.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:24.937792+0000) 2022-04-03T21:17:25.636 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:25.634+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:25.903 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:25.901+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:25.939 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:25.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:25.938009+0000) 2022-04-03T21:17:26.617 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:26.615+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:26.928 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:26.925+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:26.939 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:26.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:26.938208+0000) 2022-04-03T21:17:27.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:27.575+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:27.939 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:27.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:27.938368+0000) 2022-04-03T21:17:27.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:27.951+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:28.590 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:28.587+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:28.940 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:28.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:28.938526+0000) 2022-04-03T21:17:28.978 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:28.974+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:29.507 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:17:29.505+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:17:29.586 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:29.583+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:29.940 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:29.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:29.938731+0000) 2022-04-03T21:17:29.976 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:29.973+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:30.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:30.585+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:30.936 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:30.934+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:30.940 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:30.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:30.938964+0000) 2022-04-03T21:17:31.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:31.569+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:31.940 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:31.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:31.939189+0000) 2022-04-03T21:17:31.975 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:31.972+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:32.569 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:32.567+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:32.940 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:32.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:32.939389+0000) 2022-04-03T21:17:33.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:33.009+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:33.527 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:33.525+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:33.941 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:33.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:33.939575+0000) 2022-04-03T21:17:33.989 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:33.986+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:34.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:34.546+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:34.941 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:34.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:34.939726+0000) 2022-04-03T21:17:34.976 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:34.974+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:35.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:35.516+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:35.941 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:35.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:35.939875+0000) 2022-04-03T21:17:35.949 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:35.947+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:36.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:36.472+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:36.941 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:36.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:36.940080+0000) 2022-04-03T21:17:36.967 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:36.965+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:37.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:37.478+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:37.941 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:37.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:37.940294+0000) 2022-04-03T21:17:37.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:37.946+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:38.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:38.510+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:38.941 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:38.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:38.940480+0000) 2022-04-03T21:17:38.980 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:38.978+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:39.506 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:39.504+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:39.708 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:17:39.712 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:17:39.940 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:39.938+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:39.942 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:39.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:39.940729+0000) 2022-04-03T21:17:40.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:40.526+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:40.942 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:40.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:40.940939+0000) 2022-04-03T21:17:40.977 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:40.974+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:41.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:41.574+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:41.942 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:41.940+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:41.941176+0000) 2022-04-03T21:17:41.964 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:41.961+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:42.585 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:42.583+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:42.942 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:42.940+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:42.941368+0000) 2022-04-03T21:17:43.000 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:42.998+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:43.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:43.534+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:43.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:43.940+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:43.941557+0000) 2022-04-03T21:17:44.032 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:44.030+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:44.523 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:17:44.520+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:17:44.555 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:44.553+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:44.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:44.940+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:44.941721+0000) 2022-04-03T21:17:45.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:45.002+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:45.509 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:45.507+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:45.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:45.941+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:45.941875+0000) 2022-04-03T21:17:45.984 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:45.982+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:46.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:46.529+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:46.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:46.941+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:46.942046+0000) 2022-04-03T21:17:46.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:46.942+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:47.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:47.540+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:47.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:47.941+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:47.942222+0000) 2022-04-03T21:17:47.960 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:47.957+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:48.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:48.551+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:48.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:48.941+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:48.942428+0000) 2022-04-03T21:17:48.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:48.946+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:49.554 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:49.551+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:49.944 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:49.941+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:49.942656+0000) 2022-04-03T21:17:49.998 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:49.996+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:50.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:50.535+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:50.944 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:50.942+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:50.942890+0000) 2022-04-03T21:17:51.008 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:51.006+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:51.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:51.492+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:51.944 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:51.942+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:51.943058+0000) 2022-04-03T21:17:51.990 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:51.988+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:52.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:52.442+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:52.944 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:52.942+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:52.943262+0000) 2022-04-03T21:17:53.007 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:53.005+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:53.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:53.474+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:53.944 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:53.942+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:53.943468+0000) 2022-04-03T21:17:53.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:53.978+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:54.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:54.425+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:54.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:54.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:54.943700+0000) 2022-04-03T21:17:54.960 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:54.958+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:55.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:55.456+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:55.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:55.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:55.943863+0000) 2022-04-03T21:17:55.974 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:55.972+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:56.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:56.479+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:56.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:56.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:56.944072+0000) 2022-04-03T21:17:56.973 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:56.972+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:57.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:57.465+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:57.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:57.943+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:57.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:57.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:57.944307+0000) 2022-04-03T21:17:58.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:58.480+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:58.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:58.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:58.944531+0000) 2022-04-03T21:17:58.993 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:58.991+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:17:59.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:17:59.488+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:17:59.539 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:17:59.537+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:17:59.946 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:17:59.944+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:17:59.944741+0000) 2022-04-03T21:17:59.963 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:17:59.962+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:00.457 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:00.455+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:00.946 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:00.944+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:00.944970+0000) 2022-04-03T21:18:00.964 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:00.962+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:01.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:01.490+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:01.946 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:01.944+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:01.945191+0000) 2022-04-03T21:18:01.971 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:01.969+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:02.472 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:02.470+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:02.946 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:02.944+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:02.945428+0000) 2022-04-03T21:18:02.958 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:02.957+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:03.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:03.489+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:03.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:03.945+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:03.945615+0000) 2022-04-03T21:18:03.947 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:03.946+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:04.442 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:04.440+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:04.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:04.945+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:04.945803+0000) 2022-04-03T21:18:04.986 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:04.985+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:05.466 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:05.464+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:05.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:05.945+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:05.945990+0000) 2022-04-03T21:18:06.018 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:06.017+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:06.436 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:06.434+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:06.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:06.945+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:06.946196+0000) 2022-04-03T21:18:07.033 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:07.031+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:07.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:07.477+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:07.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:07.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:07.946401+0000) 2022-04-03T21:18:08.017 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:08.014+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:08.519 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:08.516+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:08.948 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:08.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:08.946606+0000) 2022-04-03T21:18:08.967 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:08.964+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:09.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:09.508+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:09.880 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:18:09.885 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:18:09.948 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:09.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:09.946870+0000) 2022-04-03T21:18:10.010 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:10.007+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:10.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:10.488+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:10.948 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:10.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:10.947017+0000) 2022-04-03T21:18:11.050 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:11.048+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:11.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:11.484+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:11.948 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:11.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:11.947234+0000) 2022-04-03T21:18:12.009 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:12.006+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:12.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:12.453+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:12.948 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:12.947+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:12.947443+0000) 2022-04-03T21:18:13.033 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:13.031+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:13.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:13.420+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:13.949 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:13.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:13.947653+0000) 2022-04-03T21:18:14.076 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:14.073+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:14.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:14.387+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:14.555 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:18:14.553+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:18:14.949 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:14.947+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:14.947860+0000) 2022-04-03T21:18:15.083 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:15.080+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:15.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:15.213+0000 7fd62421b700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T21:18:15.216 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:15.213+0000 7fd623a1a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T21:18:15.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:15.218+0000 7fd623a1a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:15.219403+0000) 2022-04-03T21:18:15.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:15.350+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:15.949 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:15.947+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:15.948024+0000) 2022-04-03T21:18:16.079 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:16.076+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:16.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:16.312+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:16.949 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:16.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:16.948171+0000) 2022-04-03T21:18:17.046 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:17.043+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:17.266 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:17.263+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:17.949 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:17.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:17.948360+0000) 2022-04-03T21:18:18.068 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:18.065+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:18.298 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:18.295+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:18.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:18.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:18.948508+0000) 2022-04-03T21:18:19.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:19.036+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:19.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:19.258+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:19.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:19.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:19.948688+0000) 2022-04-03T21:18:19.989 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:19.987+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:20.263 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:20.261+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:20.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:20.947+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:20.948879+0000) 2022-04-03T21:18:21.031 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:21.029+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:21.223 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:21.220+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:21.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:21.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:21.949081+0000) 2022-04-03T21:18:22.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:22.079+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:22.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:22.187+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:22.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:22.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:22.949236+0000) 2022-04-03T21:18:23.124 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:23.122+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:23.200 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:23.198+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:23.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:23.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:23.949382+0000) 2022-04-03T21:18:24.100 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:24.098+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:24.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:24.208+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:24.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:24.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:24.949526+0000) 2022-04-03T21:18:25.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:25.053+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:25.237 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:25.234+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:25.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:25.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:25.949680+0000) 2022-04-03T21:18:26.027 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:26.024+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:26.271 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:26.268+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:26.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:26.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:26.949855+0000) 2022-04-03T21:18:27.033 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:27.031+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:27.242 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:27.239+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:27.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:27.949+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:27.950030+0000) 2022-04-03T21:18:28.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:28.070+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:28.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:28.247+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:28.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:28.949+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:28.950161+0000) 2022-04-03T21:18:29.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:29.113+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:29.214 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:29.211+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:29.571 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:18:29.569+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:18:29.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:29.949+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:29.950363+0000) 2022-04-03T21:18:30.114 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:30.112+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:30.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:30.246+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:30.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:30.949+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:30.950543+0000) 2022-04-03T21:18:31.135 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:31.132+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:31.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:31.279+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:31.952 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:31.949+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:31.950720+0000) 2022-04-03T21:18:32.088 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:32.086+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:32.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:32.285+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:32.952 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:32.950+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:32.950891+0000) 2022-04-03T21:18:33.075 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:33.073+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:33.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:33.284+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:33.952 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:33.950+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:33.951055+0000) 2022-04-03T21:18:34.096 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:34.094+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:34.252 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:34.249+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:34.952 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:34.950+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:34.951201+0000) 2022-04-03T21:18:35.090 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:35.087+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:35.223 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:35.220+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:35.952 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:35.950+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:35.951400+0000) 2022-04-03T21:18:36.091 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:36.089+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:36.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:36.186+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:36.952 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:36.950+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:36.951570+0000) 2022-04-03T21:18:37.136 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:37.133+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:37.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:37.160+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:37.953 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:37.951+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:37.951743+0000) 2022-04-03T21:18:38.145 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:38.142+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:38.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:38.146+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:38.953 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:38.951+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:38.951960+0000) 2022-04-03T21:18:39.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:39.162+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:39.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:39.185+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:39.953 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:39.951+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:39.952146+0000) 2022-04-03T21:18:40.046 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:18:40.050 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:18:40.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:40.179+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:40.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:40.222+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:40.953 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:40.951+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:40.952336+0000) 2022-04-03T21:18:41.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:41.173+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:41.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:41.178+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:41.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:41.951+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:41.952533+0000) 2022-04-03T21:18:42.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:42.202+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:42.213 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:42.211+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:42.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:42.952+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:42.952700+0000) 2022-04-03T21:18:43.214 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:43.212+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:43.245 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:43.242+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:43.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:43.952+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:43.952868+0000) 2022-04-03T21:18:44.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:44.240+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:44.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:44.276+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:44.587 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:18:44.585+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:18:44.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:44.952+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:44.953025+0000) 2022-04-03T21:18:45.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:45.234+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:45.327 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:45.325+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:45.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:45.952+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:45.953202+0000) 2022-04-03T21:18:46.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:46.279+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:46.304 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:46.303+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:46.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:46.952+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:46.953395+0000) 2022-04-03T21:18:47.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:47.279+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:47.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:47.280+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:47.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:47.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:47.953539+0000) 2022-04-03T21:18:48.247 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:48.245+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:48.271 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:48.269+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:48.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:48.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:48.953700+0000) 2022-04-03T21:18:49.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:49.229+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:49.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:49.315+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:49.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:49.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:49.953887+0000) 2022-04-03T21:18:50.196 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:50.194+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:50.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:50.318+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:50.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:50.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:50.954033+0000) 2022-04-03T21:18:51.200 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:51.198+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:51.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:51.342+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:51.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:51.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:51.954125+0000) 2022-04-03T21:18:52.168 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:52.167+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:52.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:52.338+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:52.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:52.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:52.954271+0000) 2022-04-03T21:18:53.209 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:53.207+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:53.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:53.357+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:53.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:53.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:53.954423+0000) 2022-04-03T21:18:54.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:54.163+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:54.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:54.340+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:54.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:54.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:54.954597+0000) 2022-04-03T21:18:55.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:55.131+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:55.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:55.312+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:55.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:55.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:55.954754+0000) 2022-04-03T21:18:56.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:56.123+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:56.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:56.281+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:56.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:56.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:56.954895+0000) 2022-04-03T21:18:57.104 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:57.103+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:57.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:57.308+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:57.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:57.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:57.955059+0000) 2022-04-03T21:18:58.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:58.057+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:58.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:58.280+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:58.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:58.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:58.955234+0000) 2022-04-03T21:18:59.045 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:18:59.043+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:18:59.332 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:18:59.330+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:18:59.603 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:18:59.601+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:18:59.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:18:59.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:18:59.955382+0000) 2022-04-03T21:19:00.042 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:00.041+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:00.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:00.308+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:00.957 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:00.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:00.955511+0000) 2022-04-03T21:19:01.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:01.035+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:01.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:01.351+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:01.957 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:01.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:01.955672+0000) 2022-04-03T21:19:02.072 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:02.070+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:02.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:02.386+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:02.957 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:02.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:02.955804+0000) 2022-04-03T21:19:03.043 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:03.041+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:03.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:03.396+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:03.957 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:03.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:03.955971+0000) 2022-04-03T21:19:04.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:04.074+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:04.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:04.440+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:04.957 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:04.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:04.956111+0000) 2022-04-03T21:19:05.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:05.108+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:05.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:05.490+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:05.957 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:05.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:05.956236+0000) 2022-04-03T21:19:06.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:06.080+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:06.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:06.444+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:06.957 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:06.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:06.956411+0000) 2022-04-03T21:19:07.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:07.108+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:07.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:07.481+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:07.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:07.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:07.956587+0000) 2022-04-03T21:19:08.100 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:08.097+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:08.479 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:08.476+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:08.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:08.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:08.956701+0000) 2022-04-03T21:19:09.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:09.108+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:09.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:09.492+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:09.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:09.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:09.956875+0000) 2022-04-03T21:19:10.133 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:10.131+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:10.152 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:19:10.156 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:19:10.517 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:10.514+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:10.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:10.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:10.957011+0000) 2022-04-03T21:19:11.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:11.139+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:11.468 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:11.466+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:11.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:11.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:11.957188+0000) 2022-04-03T21:19:12.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:12.138+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:12.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:12.485+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:12.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:12.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:12.957363+0000) 2022-04-03T21:19:13.113 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:13.110+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:13.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:13.444+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:13.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:13.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:13.957540+0000) 2022-04-03T21:19:14.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:14.109+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:14.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:14.395+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:14.619 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:19:14.616+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:19:14.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:14.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:14.957688+0000) 2022-04-03T21:19:15.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:15.115+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:15.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:15.367+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:15.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:15.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:15.957857+0000) 2022-04-03T21:19:16.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:16.160+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:16.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:16.364+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:16.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:16.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:16.957943+0000) 2022-04-03T21:19:17.204 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:17.201+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:17.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:17.321+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:17.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:17.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:17.958116+0000) 2022-04-03T21:19:18.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:18.186+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:18.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:18.337+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:18.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:18.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:18.958222+0000) 2022-04-03T21:19:19.177 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:19.174+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:19.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:19.338+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:19.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:19.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:19.958413+0000) 2022-04-03T21:19:20.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:20.201+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:20.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:20.290+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:20.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:20.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:20.958544+0000) 2022-04-03T21:19:21.196 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:21.194+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:21.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:21.274+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:21.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:21.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:21.958733+0000) 2022-04-03T21:19:22.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:22.170+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:22.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:22.277+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:22.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:22.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:22.958807+0000) 2022-04-03T21:19:23.222 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:23.219+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:23.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:23.313+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:23.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:23.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:23.958963+0000) 2022-04-03T21:19:24.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:24.213+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:24.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:24.355+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:24.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:24.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:24.959113+0000) 2022-04-03T21:19:25.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:25.208+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:25.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:25.305+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:25.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:25.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:25.959293+0000) 2022-04-03T21:19:26.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:26.223+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:26.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:26.324+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:26.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:26.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:26.959437+0000) 2022-04-03T21:19:27.243 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:27.241+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:27.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:27.292+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:27.961 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:27.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:27.959637+0000) 2022-04-03T21:19:28.237 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:28.234+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:28.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:28.301+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:28.961 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:28.959+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:28.959829+0000) 2022-04-03T21:19:29.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:29.208+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:29.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:29.309+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:29.634 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:19:29.632+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:19:29.961 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:29.959+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:29.959979+0000) 2022-04-03T21:19:30.214 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:30.212+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:30.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:30.339+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:30.961 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:30.959+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:30.960174+0000) 2022-04-03T21:19:31.200 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:31.198+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:31.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:31.388+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:31.961 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:31.959+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:31.960374+0000) 2022-04-03T21:19:32.190 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:32.189+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:32.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:32.344+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:32.961 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:32.960+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:32.960564+0000) 2022-04-03T21:19:33.225 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:33.223+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:33.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:33.337+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:33.962 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:33.960+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:33.960738+0000) 2022-04-03T21:19:34.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:34.178+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:34.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:34.343+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:34.962 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:34.960+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:34.960947+0000) 2022-04-03T21:19:35.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:35.152+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:35.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:35.378+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:35.962 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:35.960+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:35.961093+0000) 2022-04-03T21:19:36.119 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:36.117+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:36.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:36.414+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:36.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:36.960+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:36.961274+0000) 2022-04-03T21:19:37.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:37.126+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:37.457 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:37.455+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:37.962 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:37.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:37.961461+0000) 2022-04-03T21:19:38.148 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:38.146+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:38.462 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:38.460+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:38.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:38.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:38.961610+0000) 2022-04-03T21:19:39.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:39.120+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:39.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:39.412+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:39.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:39.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:39.961837+0000) 2022-04-03T21:19:40.077 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:40.075+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:40.283 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:19:40.286 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:19:40.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:40.453+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:40.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:40.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:40.962023+0000) 2022-04-03T21:19:41.101 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:41.099+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:41.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:41.447+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:41.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:41.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:41.962258+0000) 2022-04-03T21:19:42.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:42.057+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:42.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:42.453+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:42.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:42.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:42.962459+0000) 2022-04-03T21:19:43.092 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:43.090+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:43.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:43.450+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:43.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:43.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:43.962641+0000) 2022-04-03T21:19:44.077 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:44.075+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:44.457 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:44.456+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:44.650 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:19:44.648+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:19:44.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:44.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:44.962829+0000) 2022-04-03T21:19:45.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:45.078+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:45.438 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:45.436+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:45.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:45.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:45.963013+0000) 2022-04-03T21:19:46.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:46.111+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:46.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:46.438+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:46.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:46.963+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:46.963193+0000) 2022-04-03T21:19:47.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:47.127+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:47.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:47.457+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:47.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:47.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:47.963375+0000) 2022-04-03T21:19:48.162 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:48.161+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:48.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:48.451+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:48.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:48.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:48.963557+0000) 2022-04-03T21:19:49.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:49.181+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:49.477 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:49.476+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:49.965 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:49.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:49.963717+0000) 2022-04-03T21:19:50.160 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:50.158+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:50.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:50.429+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:50.965 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:50.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:50.963907+0000) 2022-04-03T21:19:51.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:51.130+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:51.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:51.406+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:51.965 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:51.963+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:51.964070+0000) 2022-04-03T21:19:52.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:52.155+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:52.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:52.384+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:52.965 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:52.963+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:52.964161+0000) 2022-04-03T21:19:53.200 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:53.199+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:53.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:53.388+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:53.965 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:53.963+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:53.964312+0000) 2022-04-03T21:19:54.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:54.191+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:54.429 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:54.427+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:54.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:54.963+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:54.964422+0000) 2022-04-03T21:19:55.192 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:55.190+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:55.426 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:55.424+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:55.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:55.963+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:55.964518+0000) 2022-04-03T21:19:56.210 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:56.207+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:56.407 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:56.404+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:56.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:56.963+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:56.964704+0000) 2022-04-03T21:19:57.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:57.216+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:57.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:57.442+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:57.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:57.964+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:57.964859+0000) 2022-04-03T21:19:58.241 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:58.238+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:58.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:58.467+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:58.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:58.964+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:58.965002+0000) 2022-04-03T21:19:59.212 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:19:59.209+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:19:59.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:19:59.440+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:19:59.666 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:19:59.664+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:19:59.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:19:59.964+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:19:59.965111+0000) 2022-04-03T21:20:00.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:00.185+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:00.460 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:00.458+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:00.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:00.964+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:00.965266+0000) 2022-04-03T21:20:01.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:01.184+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:01.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:01.492+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:01.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:01.964+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:01.965436+0000) 2022-04-03T21:20:02.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:02.195+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:02.526 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:02.523+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:02.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:02.964+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:02.965609+0000) 2022-04-03T21:20:03.166 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:03.164+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:03.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:03.546+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:03.967 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:03.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:03.965759+0000) 2022-04-03T21:20:04.210 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:04.207+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:04.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:04.526+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:04.967 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:04.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:04.965957+0000) 2022-04-03T21:20:05.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:05.164+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:05.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:05.571+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:05.967 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:05.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:05.966134+0000) 2022-04-03T21:20:06.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:06.186+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:06.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:06.587+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:06.967 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:06.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:06.966288+0000) 2022-04-03T21:20:07.193 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:07.190+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:07.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:07.551+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:07.967 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:07.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:07.966467+0000) 2022-04-03T21:20:08.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:08.195+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:08.586 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:08.583+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:08.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:08.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:08.966657+0000) 2022-04-03T21:20:09.193 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:09.190+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:09.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:09.573+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:09.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:09.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:09.966841+0000) 2022-04-03T21:20:10.192 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:10.190+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:10.377 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:20:10.381 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:20:10.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:10.571+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:10.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:10.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:10.967036+0000) 2022-04-03T21:20:11.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:11.205+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:11.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:11.613+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:11.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:11.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:11.967269+0000) 2022-04-03T21:20:12.251 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:12.249+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:12.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:12.632+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:12.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:12.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:12.967452+0000) 2022-04-03T21:20:13.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:13.274+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:13.659 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:13.656+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:13.969 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:13.967+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:13.967632+0000) 2022-04-03T21:20:14.312 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:14.309+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:14.676 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:14.674+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:14.682 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:20:14.680+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:20:14.969 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:14.967+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:14.967799+0000) 2022-04-03T21:20:15.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:15.320+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:15.710 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:15.707+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:15.969 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:15.967+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:15.967981+0000) 2022-04-03T21:20:16.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:16.281+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:16.693 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:16.691+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:16.969 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:16.967+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:16.968138+0000) 2022-04-03T21:20:17.236 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:17.233+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:17.691 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:17.689+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:17.969 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:17.967+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:17.968290+0000) 2022-04-03T21:20:18.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:18.242+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:18.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:18.734+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:18.969 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:18.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:18.968492+0000) 2022-04-03T21:20:19.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:19.279+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:19.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:19.732+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:19.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:19.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:19.968683+0000) 2022-04-03T21:20:20.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:20.313+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:20.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:20.768+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:20.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:20.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:20.968867+0000) 2022-04-03T21:20:21.298 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:21.296+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:21.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:21.732+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:21.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:21.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:21.969026+0000) 2022-04-03T21:20:22.304 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:22.302+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:22.766 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:22.764+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:22.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:22.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:22.969182+0000) 2022-04-03T21:20:23.258 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:23.255+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:23.761 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:23.758+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:23.988 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:23.969+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:23.969333+0000) 2022-04-03T21:20:24.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:24.295+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:24.764 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:24.762+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:24.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:24.969+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:24.969489+0000) 2022-04-03T21:20:25.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:25.326+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:25.786 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:25.783+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:25.971 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:25.969+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:25.969611+0000) 2022-04-03T21:20:26.364 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:26.362+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:26.794 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:26.791+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:26.971 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:26.969+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:26.969759+0000) 2022-04-03T21:20:27.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:27.399+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:27.792 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:27.789+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:27.971 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:27.969+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:27.969936+0000) 2022-04-03T21:20:28.379 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:28.376+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:28.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:28.756+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:28.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:28.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:28.970126+0000) 2022-04-03T21:20:29.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:29.397+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:29.698 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:20:29.696+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:20:29.781 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:29.778+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:29.971 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:29.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:29.970287+0000) 2022-04-03T21:20:30.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:30.366+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:30.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:30.774+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:30.971 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:30.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:30.970381+0000) 2022-04-03T21:20:31.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:31.388+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:31.805 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:31.803+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:31.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:31.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:31.970566+0000) 2022-04-03T21:20:32.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:32.388+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:32.798 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:32.796+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:32.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:32.969+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:32.970786+0000) 2022-04-03T21:20:33.344 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:33.342+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:33.812 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:33.810+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:33.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:33.969+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:33.970936+0000) 2022-04-03T21:20:34.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:34.362+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:34.853 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:34.851+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:34.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:34.969+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:34.971077+0000) 2022-04-03T21:20:35.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:35.367+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:35.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:35.811+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:35.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:35.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:35.971218+0000) 2022-04-03T21:20:36.398 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:36.396+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:36.817 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:36.816+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:36.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:36.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:36.971372+0000) 2022-04-03T21:20:37.354 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:37.352+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:37.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:37.832+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:37.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:37.969+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:37.971513+0000) 2022-04-03T21:20:38.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:38.366+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:38.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:38.863+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:38.973 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:38.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:38.971644+0000) 2022-04-03T21:20:39.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:39.338+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:39.873 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:39.871+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:39.973 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:39.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:39.971847+0000) 2022-04-03T21:20:40.294 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:40.292+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:40.525 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:20:40.528 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:20:40.841 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:40.839+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:40.973 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:40.971+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:40.972049+0000) 2022-04-03T21:20:41.290 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:41.288+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:41.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:41.833+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:41.973 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:41.971+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:41.972258+0000) 2022-04-03T21:20:42.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:42.259+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:42.883 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:42.881+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:42.973 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:42.971+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:42.972380+0000) 2022-04-03T21:20:43.269 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:43.268+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:43.906 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:43.905+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:43.973 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:43.971+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:43.972528+0000) 2022-04-03T21:20:44.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:44.224+0000 7fd3b190b700 -1 osd.6 123 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:44.713 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:20:44.711+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:20:44.940 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-03T21:20:44.938+0000 7f69dc698700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T21:20:44.944 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:44.943+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:44.955 INFO:tasks.ceph.mds.f.smithi186.stderr: -1> 2022-04-03T21:20:44.938+0000 7f69dc698700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T21:20:44.974 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:44.971+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:44.972664+0000) 2022-04-03T21:20:44.977 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-03T21:20:45.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:45.248+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:45.974 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:45.972+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:45.972831+0000) 2022-04-03T21:20:45.978 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:45.976+0000 7f88d0ac1700 -1 osd.4 123 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:46.236 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:46.234+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:46.936 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:46.934+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:46.974 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:46.972+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:46.972931+0000) 2022-04-03T21:20:47.213 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:47.211+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:47.935 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:47.934+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:47.974 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:47.972+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:47.973077+0000) 2022-04-03T21:20:48.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:48.162+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:48.907 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:48.905+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:48.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:48.972+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:48.973187+0000) 2022-04-03T21:20:49.115 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:49.114+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:49.949 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:49.946+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:49.974 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:49.972+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:49.973379+0000) 2022-04-03T21:20:50.137 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:50.134+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:50.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:50.940+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:50.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:50.972+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:50.973555+0000) 2022-04-03T21:20:51.105 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:51.103+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:51.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:51.973+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:51.973740+0000) 2022-04-03T21:20:51.991 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:51.988+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:52.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:52.080+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:52.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:52.973+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:52.973934+0000) 2022-04-03T21:20:53.014 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:53.012+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:53.092 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:53.090+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:53.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:53.973+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:53.974119+0000) 2022-04-03T21:20:53.977 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:53.975+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:54.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:54.126+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:54.934 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:54.932+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:54.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:54.973+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:54.974304+0000) 2022-04-03T21:20:55.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:55.108+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:55.932 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:55.930+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:55.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:55.973+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:55.974486+0000) 2022-04-03T21:20:56.098 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:56.095+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:56.888 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:56.885+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:56.976 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:56.973+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:56.974687+0000) 2022-04-03T21:20:57.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:57.123+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:57.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:57.839+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:57.976 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:57.974+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:57.974868+0000) 2022-04-03T21:20:58.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:58.139+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:58.840 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:58.838+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:58.976 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:58.974+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:58.975085+0000) 2022-04-03T21:20:59.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:20:59.176+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:20:59.729 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:20:59.727+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:20:59.858 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:20:59.856+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:20:59.976 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:20:59.974+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:20:59.975272+0000) 2022-04-03T21:21:00.138 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:00.136+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:00.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:00.867+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:00.976 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:00.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:00.975426+0000) 2022-04-03T21:21:01.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:01.163+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:01.884 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:01.882+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:01.977 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:01.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:01.975576+0000) 2022-04-03T21:21:02.118 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:02.116+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:02.848 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:02.845+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:02.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:02.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:02.975724+0000) 2022-04-03T21:21:03.092 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:03.089+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:03.803 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:03.800+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:03.977 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:03.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:03.975896+0000) 2022-04-03T21:21:04.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:04.101+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:04.792 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:04.790+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:04.977 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:04.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:04.976017+0000) 2022-04-03T21:21:05.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:05.108+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:05.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:05.753+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:05.977 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:05.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:05.976180+0000) 2022-04-03T21:21:06.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:06.110+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:06.712 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:06.710+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:06.977 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:06.976+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:06.976329+0000) 2022-04-03T21:21:07.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:07.076+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:07.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:07.715+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:07.978 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:07.976+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:07.976506+0000) 2022-04-03T21:21:08.058 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:08.056+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:08.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:08.720+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:08.978 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:08.976+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:08.976672+0000) 2022-04-03T21:21:09.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:09.074+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:09.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:09.688+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:09.978 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:09.976+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:09.976845+0000) 2022-04-03T21:21:10.029 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:10.027+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:10.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:10.651+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:10.658 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:21:10.662 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:21:10.978 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:10.976+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:10.976994+0000) 2022-04-03T21:21:11.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:11.008+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:11.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:11.672+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:11.978 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:11.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:11.977184+0000) 2022-04-03T21:21:11.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:11.993+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:12.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:12.664+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:12.978 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:12.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:12.977371+0000) 2022-04-03T21:21:13.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:13.031+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:13.629 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:13.627+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:13.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:13.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:13.977616+0000) 2022-04-03T21:21:13.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:13.993+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:14.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:14.648+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:14.745 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:21:14.743+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:21:14.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:14.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:14.977781+0000) 2022-04-03T21:21:15.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:15.025+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:15.621 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:15.618+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:15.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:15.976+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:15.978024+0000) 2022-04-03T21:21:16.078 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:16.076+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:16.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:16.641+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:16.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:16.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:16.978250+0000) 2022-04-03T21:21:17.077 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:17.075+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:17.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:17.658+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:17.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:17.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:17.978461+0000) 2022-04-03T21:21:18.073 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:18.071+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:18.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:18.686+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:18.980 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:18.976+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:18.978660+0000) 2022-04-03T21:21:19.042 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:19.039+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:19.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:19.693+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:19.980 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:19.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:19.978883+0000) 2022-04-03T21:21:19.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:19.995+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:20.694 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:20.692+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:20.980 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:20.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:20.979029+0000) 2022-04-03T21:21:21.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:21.001+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:21.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:21.693+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:21.980 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:21.978+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:21.979230+0000) 2022-04-03T21:21:22.000 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:21.998+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:22.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:22.661+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:22.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:22.977+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:22.980 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:22.978+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:22.979452+0000) 2022-04-03T21:21:23.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:23.660+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:23.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:23.978+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:23.979692+0000) 2022-04-03T21:21:24.014 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:24.012+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:24.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:24.651+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:24.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:24.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:24.979927+0000) 2022-04-03T21:21:25.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:25.053+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:25.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:25.677+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:25.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:25.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:25.980172+0000) 2022-04-03T21:21:26.077 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:26.074+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:26.648 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:26.646+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:26.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:26.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:26.980324+0000) 2022-04-03T21:21:27.110 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:27.107+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:27.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:27.614+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:27.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:27.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:27.980547+0000) 2022-04-03T21:21:28.093 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:28.092+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:28.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:28.610+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:28.982 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:28.980+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:28.980756+0000) 2022-04-03T21:21:29.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:29.130+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:29.566 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:29.564+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:29.750 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:21:29.748+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:21:29.982 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:29.980+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:29.980977+0000) 2022-04-03T21:21:30.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:30.169+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:30.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:30.586+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:30.982 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:30.980+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:30.981172+0000) 2022-04-03T21:21:31.199 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:31.198+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:31.603 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:31.601+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:31.982 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:31.980+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:31.981379+0000) 2022-04-03T21:21:32.193 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:32.192+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:32.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:32.610+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:32.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:32.980+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:32.981584+0000) 2022-04-03T21:21:33.239 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:33.237+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:33.609 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:33.608+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:33.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:33.981+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:33.981822+0000) 2022-04-03T21:21:34.202 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:34.200+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:34.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:34.618+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:34.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:34.981+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:34.982005+0000) 2022-04-03T21:21:35.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:35.197+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:35.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:35.629+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:35.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:35.981+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:35.982154+0000) 2022-04-03T21:21:36.242 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:36.240+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:36.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:36.669+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:36.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:36.981+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:36.982365+0000) 2022-04-03T21:21:37.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:37.201+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:37.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:37.624+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:37.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:37.981+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:37.982538+0000) 2022-04-03T21:21:38.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:38.169+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:38.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:38.643+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:38.984 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:38.982+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:38.982807+0000) 2022-04-03T21:21:39.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:39.125+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:39.623 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:39.622+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:39.984 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:39.982+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:39.982974+0000) 2022-04-03T21:21:40.089 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:40.087+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:40.623 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:40.622+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:40.727 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:21:40.730 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:21:40.984 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:40.982+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:40.983217+0000) 2022-04-03T21:21:41.068 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:41.066+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:41.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:41.589+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:41.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:41.982+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:41.983477+0000) 2022-04-03T21:21:42.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:42.037+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:42.586 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:42.585+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:42.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:42.983+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:42.983687+0000) 2022-04-03T21:21:43.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:43.044+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:43.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:43.550+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:43.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:43.983+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:43.983904+0000) 2022-04-03T21:21:43.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:43.996+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:44.602 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:44.599+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:44.766 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:21:44.763+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:21:44.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:44.983+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:44.984109+0000) 2022-04-03T21:21:44.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:44.987+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:45.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:45.611+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:45.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:45.979+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:45.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:45.983+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:45.984317+0000) 2022-04-03T21:21:46.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:46.650+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:46.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:46.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:46.984518+0000) 2022-04-03T21:21:47.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:47.026+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:47.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:47.633+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:47.986 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:47.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:47.984724+0000) 2022-04-03T21:21:47.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:47.985+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:48.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:48.645+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:48.986 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:48.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:48.984929+0000) 2022-04-03T21:21:49.020 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:49.017+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:49.659 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:49.656+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:49.986 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:49.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:49.985155+0000) 2022-04-03T21:21:50.044 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:50.042+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:50.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:50.680+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:50.986 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:50.985+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:50.985359+0000) 2022-04-03T21:21:51.057 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:51.054+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:51.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:51.678+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:51.986 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:51.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:51.985528+0000) 2022-04-03T21:21:52.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:52.034+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:52.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:52.655+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:52.987 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:52.985+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:52.985747+0000) 2022-04-03T21:21:53.027 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:53.025+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:53.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:53.681+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:53.987 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:53.985+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:53.985965+0000) 2022-04-03T21:21:54.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:54.043+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:54.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:54.698+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:54.987 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:54.985+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:54.986141+0000) 2022-04-03T21:21:55.005 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:55.002+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:55.674 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:55.672+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:55.987 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:55.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:55.986300+0000) 2022-04-03T21:21:56.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:56.016+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:56.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:56.698+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:56.987 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:56.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:56.986487+0000) 2022-04-03T21:21:57.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:57.032+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:57.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:57.740+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:57.988 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:57.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:57.986699+0000) 2022-04-03T21:21:58.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:58.036+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:58.753 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:58.750+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:58.988 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:58.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:58.986921+0000) 2022-04-03T21:21:59.073 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:21:59.071+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:21:59.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:21:59.749+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:21:59.781 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:21:59.779+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:21:59.988 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:21:59.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:21:59.987075+0000) 2022-04-03T21:22:00.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:00.031+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:00.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:00.791+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:00.988 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:00.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:00.987310+0000) 2022-04-03T21:22:01.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:01.006+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:01.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:01.755+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:01.988 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:01.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:01.987519+0000) 2022-04-03T21:22:02.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:02.010+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:02.804 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:02.802+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:02.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:02.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:02.987739+0000) 2022-04-03T21:22:02.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:02.989+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:03.766 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:03.764+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:03.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:03.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:03.987963+0000) 2022-04-03T21:22:04.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:04.008+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:04.739 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:04.737+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:04.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:04.975+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:04.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:04.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:04.988169+0000) 2022-04-03T21:22:05.781 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:05.779+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:05.959 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:05.956+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:05.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:05.987+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:05.988375+0000) 2022-04-03T21:22:06.763 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:06.761+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:06.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:06.987+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:06.988601+0000) 2022-04-03T21:22:07.006 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:07.004+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:07.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:07.748+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:07.980 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:07.978+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:07.990 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:07.987+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:07.988838+0000) 2022-04-03T21:22:08.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:08.768+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:08.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:08.956+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:08.990 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:08.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:08.989007+0000) 2022-04-03T21:22:09.745 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:09.743+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:09.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:09.950+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:09.990 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:09.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:09.989152+0000) 2022-04-03T21:22:10.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:10.720+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:10.869 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:22:10.873 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:22:10.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:10.949+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:10.990 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:10.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:10.989355+0000) 2022-04-03T21:22:11.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:11.700+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:11.990 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:11.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:11.989583+0000) 2022-04-03T21:22:11.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:11.997+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:12.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:12.700+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:12.962 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:12.960+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:12.991 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:12.989+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:12.989835+0000) 2022-04-03T21:22:13.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:13.666+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:13.945 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:13.944+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:13.991 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:13.989+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:13.989993+0000) 2022-04-03T21:22:14.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:14.710+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:14.797 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:22:14.795+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:22:14.954 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:14.951+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:14.991 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:14.989+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:14.990205+0000) 2022-04-03T21:22:15.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:15.694+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:15.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:15.926+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:15.991 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:15.989+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:15.990435+0000) 2022-04-03T21:22:16.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:16.711+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:16.907 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:16.905+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:16.992 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:16.989+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:16.990643+0000) 2022-04-03T21:22:17.720 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:17.718+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:17.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:17.856+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:17.992 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:17.990+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:17.990864+0000) 2022-04-03T21:22:18.744 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:18.742+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:18.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:18.836+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:18.992 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:18.990+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:18.991030+0000) 2022-04-03T21:22:19.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:19.725+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:19.833 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:19.831+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:19.992 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:19.990+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:19.991239+0000) 2022-04-03T21:22:20.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:20.707+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:20.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:20.858+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:20.992 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:20.990+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:20.991440+0000) 2022-04-03T21:22:21.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:21.694+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:21.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:21.819+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:21.993 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:21.991+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:21.991625+0000) 2022-04-03T21:22:22.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:22.697+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:22.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:22.811+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:22.993 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:22.991+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:22.991855+0000) 2022-04-03T21:22:23.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:23.669+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:23.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:23.816+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:23.993 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:23.991+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:23.992042+0000) 2022-04-03T21:22:24.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:24.665+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:24.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:24.837+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:24.993 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:24.991+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:24.992204+0000) 2022-04-03T21:22:25.621 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:25.619+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:25.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:25.797+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:25.993 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:25.991+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:25.992395+0000) 2022-04-03T21:22:26.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:26.656+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:26.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:26.777+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:26.994 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:26.992+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:26.992592+0000) 2022-04-03T21:22:27.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:27.700+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:27.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:27.759+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:27.994 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:27.992+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:27.992766+0000) 2022-04-03T21:22:28.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:28.665+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:28.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:28.773+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:28.994 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:28.992+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:28.992992+0000) 2022-04-03T21:22:29.625 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:29.623+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:29.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:29.756+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:29.813 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:22:29.811+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:22:29.994 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:29.992+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:29.993233+0000) 2022-04-03T21:22:30.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:30.634+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:30.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:30.728+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:30.994 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:30.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:30.993410+0000) 2022-04-03T21:22:31.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:31.649+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:31.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:31.692+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:31.994 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:31.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:31.993557+0000) 2022-04-03T21:22:32.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:32.651+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:32.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:32.701+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:32.995 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:32.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:32.993748+0000) 2022-04-03T21:22:33.683 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:33.682+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:33.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:33.690+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:33.995 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:33.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:33.993958+0000) 2022-04-03T21:22:34.683 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:34.681+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:34.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:34.736+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:34.995 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:34.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:34.994152+0000) 2022-04-03T21:22:35.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:35.692+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:35.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:35.770+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:35.995 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:35.994+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:35.994341+0000) 2022-04-03T21:22:36.705 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:36.703+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:36.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:36.724+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:36.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:36.994+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:36.994556+0000) 2022-04-03T21:22:37.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:37.685+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:37.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:37.697+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:37.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:37.994+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:37.994744+0000) 2022-04-03T21:22:38.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:38.696+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:38.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:38.696+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:38.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:38.994+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:38.994904+0000) 2022-04-03T21:22:39.717 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:39.714+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:39.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:39.725+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:39.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:39.994+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:39.995132+0000) 2022-04-03T21:22:40.719 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:40.716+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:40.744 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:40.742+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:40.934 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:22:40.938 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:22:40.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:40.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:40.995371+0000) 2022-04-03T21:22:41.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:41.712+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:41.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:41.755+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:41.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:41.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:41.995604+0000) 2022-04-03T21:22:42.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:42.714+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:42.775 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:42.773+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:42.997 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:42.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:42.995745+0000) 2022-04-03T21:22:43.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:43.709+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:43.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:43.736+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:43.997 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:43.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:43.995934+0000) 2022-04-03T21:22:44.751 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:44.748+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:44.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:44.753+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:44.820 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:22:44.817+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:22:44.997 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:44.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:44.996127+0000) 2022-04-03T21:22:45.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:45.720+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:45.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:45.741+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:45.997 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:45.994+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:45.996347+0000) 2022-04-03T21:22:46.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:46.768+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:46.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:46.790+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:46.997 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:46.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:46.996543+0000) 2022-04-03T21:22:47.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:47.733+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:47.774 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:47.772+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:47.998 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:47.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:47.996698+0000) 2022-04-03T21:22:48.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:48.714+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:48.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:48.741+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:48.998 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:48.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:48.996893+0000) 2022-04-03T21:22:49.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:49.691+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:49.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:49.729+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:49.998 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:49.996+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:49.997059+0000) 2022-04-03T21:22:50.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:50.665+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:50.727 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:50.724+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:50.998 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:50.996+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:50.997254+0000) 2022-04-03T21:22:51.707 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:51.704+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:51.764 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:51.761+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:51.998 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:51.996+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:51.997472+0000) 2022-04-03T21:22:52.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:52.729+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:52.809 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:52.806+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:52.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:52.996+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:52.997669+0000) 2022-04-03T21:22:53.718 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:53.716+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:53.778 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:53.775+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:53.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:53.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:53.997831+0000) 2022-04-03T21:22:54.757 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:54.754+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:54.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:54.769+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:54.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:54.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:54.998025+0000) 2022-04-03T21:22:55.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:55.731+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:55.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:55.769+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:55.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:55.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:55.998222+0000) 2022-04-03T21:22:56.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:56.758+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:56.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:56.805+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:56.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:56.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:56.998413+0000) 2022-04-03T21:22:57.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:57.799+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:57.810 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:57.807+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:57.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:57.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:57.998568+0000) 2022-04-03T21:22:58.825 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:58.822+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:58.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:58.837+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:59.000 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:58.998+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:58.998751+0000) 2022-04-03T21:22:59.810 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:22:59.808+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:22:59.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:22:59.822+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:22:59.836 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:22:59.834+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:23:00.000 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:22:59.998+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:22:59.998978+0000) 2022-04-03T21:23:00.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:00.787+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:00.840 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:00.837+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:01.000 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:00.998+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:00.999168+0000) 2022-04-03T21:23:01.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:01.778+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:01.839 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:01.837+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:02.000 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:01.998+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:01.999370+0000) 2022-04-03T21:23:02.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:02.779+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:02.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:02.827+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:03.001 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:02.998+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:02.999567+0000) 2022-04-03T21:23:03.765 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:03.763+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:03.792 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:03.790+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:04.001 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:03.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:03.999810+0000) 2022-04-03T21:23:04.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:04.755+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:04.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:04.782+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:05.001 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:04.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:04.999994+0000) 2022-04-03T21:23:05.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:05.740+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:05.797 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:05.795+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:06.001 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:05.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:06.000202+0000) 2022-04-03T21:23:06.736 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:06.733+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:06.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:06.797+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:07.002 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:06.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:07.000402+0000) 2022-04-03T21:23:07.692 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:07.690+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:07.846 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:07.843+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:08.002 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:08.000+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:08.000603+0000) 2022-04-03T21:23:08.704 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:08.702+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:08.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:08.866+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:09.002 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:09.000+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:09.000843+0000) 2022-04-03T21:23:09.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:09.673+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:09.890 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:09.887+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:10.002 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:10.000+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:10.001060+0000) 2022-04-03T21:23:10.684 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:10.682+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:10.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:10.908+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:11.002 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:11.000+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:11.001217+0000) 2022-04-03T21:23:11.090 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:23:11.094 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:23:11.659 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:11.657+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:11.931 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:11.929+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:12.002 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:12.000+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:12.001399+0000) 2022-04-03T21:23:12.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:12.642+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:12.936 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:12.933+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:13.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:13.001+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:13.001607+0000) 2022-04-03T21:23:13.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:13.638+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:13.956 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:13.953+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:14.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:14.001+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:14.001849+0000) 2022-04-03T21:23:14.675 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:14.673+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:14.852 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:23:14.850+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:23:14.998 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:14.995+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:15.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:15.001+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:15.001998+0000) 2022-04-03T21:23:15.633 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:15.631+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:15.962 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:15.960+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:16.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:16.001+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:16.002223+0000) 2022-04-03T21:23:16.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:16.612+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:16.934 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:16.932+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:17.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:17.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:17.002388+0000) 2022-04-03T21:23:17.649 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:17.646+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:17.911 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:17.910+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:18.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:18.001+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:18.002602+0000) 2022-04-03T21:23:18.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:18.640+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:18.917 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:18.915+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:19.004 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:19.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:19.002854+0000) 2022-04-03T21:23:19.608 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:19.606+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:19.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:19.888+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:20.004 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:20.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:20.003036+0000) 2022-04-03T21:23:20.566 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:20.564+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:20.917 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:20.915+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:21.004 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:21.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:21.003244+0000) 2022-04-03T21:23:21.564 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:21.562+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:21.888 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:21.886+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:22.005 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:22.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:22.003394+0000) 2022-04-03T21:23:22.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:22.590+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:22.922 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:22.921+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:23.004 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:23.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:23.003606+0000) 2022-04-03T21:23:23.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:23.611+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:23.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:23.881+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:24.005 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:24.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:24.003856+0000) 2022-04-03T21:23:24.653 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:24.651+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:24.874 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:24.873+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:25.005 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:25.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:25.004005+0000) 2022-04-03T21:23:25.637 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:25.635+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:25.880 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:25.879+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:26.005 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:26.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:26.004207+0000) 2022-04-03T21:23:26.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:26.628+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:26.854 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:26.852+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:27.005 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:27.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:27.004407+0000) 2022-04-03T21:23:27.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:27.614+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:27.854 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:27.852+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:28.006 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:28.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:28.004604+0000) 2022-04-03T21:23:28.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:28.663+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:28.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:28.804+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:29.006 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:29.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:29.004848+0000) 2022-04-03T21:23:29.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:29.639+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:29.852 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:29.850+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:29.868 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:23:29.865+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:23:30.006 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:30.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:30.005044+0000) 2022-04-03T21:23:30.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:30.606+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:30.896 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:30.894+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:31.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:31.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:31.005228+0000) 2022-04-03T21:23:31.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:31.565+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:31.859 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:31.857+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:32.006 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:32.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:32.005395+0000) 2022-04-03T21:23:32.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:32.586+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:32.899 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:32.896+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:33.006 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:33.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:33.005567+0000) 2022-04-03T21:23:33.610 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:33.607+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:33.918 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:33.916+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:34.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:34.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:34.005799+0000) 2022-04-03T21:23:34.569 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:34.566+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:34.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:34.951+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:35.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:35.005+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:35.006003+0000) 2022-04-03T21:23:35.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:35.539+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:35.919 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:35.916+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:36.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:36.005+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:36.006223+0000) 2022-04-03T21:23:36.554 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:36.551+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:36.920 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:36.918+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:37.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:37.005+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:37.006427+0000) 2022-04-03T21:23:37.509 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:37.506+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:37.961 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:37.959+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:38.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:38.005+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:38.006633+0000) 2022-04-03T21:23:38.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:38.492+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:38.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:38.989+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:39.008 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:39.006+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:39.006844+0000) 2022-04-03T21:23:39.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:39.539+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:23:39.989 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:39.986+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:40.008 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:40.006+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:40.007080+0000) 2022-04-03T21:23:40.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:40.574+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:23:40.964 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:40.962+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:41.008 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:41.006+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:41.007265+0000) 2022-04-03T21:23:41.215 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:23:41.219 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:23:41.554 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:41.552+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:23:41.972 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:41.969+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:42.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:42.006+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:42.007501+0000) 2022-04-03T21:23:42.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:42.508+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:23:42.966 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:42.964+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:43.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:43.006+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:43.007673+0000) 2022-04-03T21:23:43.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:43.460+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:23:44.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:44.007+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:44.007881+0000) 2022-04-03T21:23:44.009 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:44.007+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:44.449 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:44.446+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:23:44.884 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:23:44.882+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:23:45.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:45.007+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:45.008091+0000) 2022-04-03T21:23:45.020 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:45.017+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:45.482 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:45.479+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:23:46.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:46.007+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:46.008298+0000) 2022-04-03T21:23:46.050 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:46.048+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:46.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:46.437+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:23:47.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:47.007+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:47.008502+0000) 2022-04-03T21:23:47.019 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:47.017+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:47.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:47.408+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:23:48.010 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:48.007+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:48.008723+0000) 2022-04-03T21:23:48.027 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:48.025+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:48.365 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:48.362+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:23:49.000 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:48.997+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:49.010 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:49.008+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:49.008947+0000) 2022-04-03T21:23:49.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:49.405+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:23:50.000 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:49.997+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 10 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:50.010 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:50.008+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:50.009104+0000) 2022-04-03T21:23:50.426 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:50.424+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:23:51.010 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:51.008+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:51.009314+0000) 2022-04-03T21:23:51.045 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:51.042+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:51.434 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:51.432+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:23:52.008 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:52.006+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:52.010 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:52.008+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:52.009485+0000) 2022-04-03T21:23:52.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:52.383+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:23:52.962 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:52.960+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:53.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:53.009+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:53.009637+0000) 2022-04-03T21:23:53.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:53.379+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:23:53.951 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:53.949+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:54.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:54.009+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:54.009872+0000) 2022-04-03T21:23:54.347 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:54.345+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:23:54.971 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:54.969+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:55.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:55.009+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:55.010077+0000) 2022-04-03T21:23:55.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:55.319+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:23:55.961 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:55.958+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:56.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:56.009+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:56.010287+0000) 2022-04-03T21:23:56.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:56.270+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:23:56.929 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:56.927+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:57.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:57.010+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:57.010438+0000) 2022-04-03T21:23:57.256 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:57.254+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:23:57.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:57.941+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:58.012 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:58.010+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:58.010647+0000) 2022-04-03T21:23:58.234 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:58.232+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:23:58.969 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:58.967+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:23:59.012 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:23:59.010+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:23:59.010852+0000) 2022-04-03T21:23:59.283 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:23:59.281+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:23:59.892 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:23:59.890+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:23:59.931 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:23:59.928+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:00.012 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:00.010+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:00.011074+0000) 2022-04-03T21:24:00.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:00.286+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:00.884 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:00.881+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:01.012 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:01.010+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:01.011226+0000) 2022-04-03T21:24:01.270 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:01.268+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:01.910 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:01.907+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:02.012 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:02.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:02.011435+0000) 2022-04-03T21:24:02.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:02.282+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:02.903 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:02.900+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:03.012 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:03.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:03.011607+0000) 2022-04-03T21:24:03.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:03.274+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:03.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:03.906+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:04.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:04.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:04.011824+0000) 2022-04-03T21:24:04.319 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:04.317+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:04.939 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:04.937+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:05.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:05.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:05.012000+0000) 2022-04-03T21:24:05.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:05.359+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:05.981 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:05.979+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:06.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:06.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:06.012208+0000) 2022-04-03T21:24:06.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:06.354+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:06.976 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:06.974+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:07.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:07.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:07.012414+0000) 2022-04-03T21:24:07.372 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:07.370+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:08.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:08.010+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:08.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:08.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:08.012623+0000) 2022-04-03T21:24:08.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:08.369+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:09.005 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:09.003+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:09.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:09.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:09.012833+0000) 2022-04-03T21:24:09.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:09.379+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:10.014 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:10.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:10.013028+0000) 2022-04-03T21:24:10.049 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:10.048+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:10.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:10.371+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:11.014 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:11.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:11.013203+0000) 2022-04-03T21:24:11.044 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:11.043+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:11.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:11.417+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:11.512 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:24:11.516 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:24:12.014 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:12.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:12.013445+0000) 2022-04-03T21:24:12.028 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:12.026+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:12.466 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:12.464+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:13.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:13.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:13.013633+0000) 2022-04-03T21:24:13.077 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:13.075+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:13.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:13.474+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:14.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:14.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:14.013812+0000) 2022-04-03T21:24:14.086 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:14.084+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:14.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:14.481+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:14.908 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:24:14.905+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:24:15.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:15.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:15.014030+0000) 2022-04-03T21:24:15.057 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:15.055+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:15.471 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:15.470+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:16.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:16.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:16.014193+0000) 2022-04-03T21:24:16.097 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:16.095+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:16.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:16.493+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:17.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:17.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:17.014394+0000) 2022-04-03T21:24:17.139 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:17.137+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:17.468 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:17.466+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:18.016 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:18.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:18.014622+0000) 2022-04-03T21:24:18.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:18.175+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:18.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:18.459+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:19.016 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:19.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:19.014862+0000) 2022-04-03T21:24:19.133 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:19.131+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:19.420 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:19.419+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:20.016 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:20.014+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:20.015064+0000) 2022-04-03T21:24:20.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:20.113+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:20.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:20.376+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:21.016 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:21.014+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:21.015290+0000) 2022-04-03T21:24:21.086 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:21.085+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:21.398 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:21.396+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:22.016 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:22.014+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:22.015490+0000) 2022-04-03T21:24:22.057 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:22.056+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:22.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:22.359+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:23.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:23.014+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:23.015694+0000) 2022-04-03T21:24:23.099 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:23.097+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:23.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:23.380+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:24.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:24.015+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:24.015918+0000) 2022-04-03T21:24:24.138 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:24.136+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:24.397 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:24.395+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:25.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:25.015+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:25.016099+0000) 2022-04-03T21:24:25.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:25.159+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:25.412 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:25.410+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:26.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:26.015+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:26.016320+0000) 2022-04-03T21:24:26.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:26.171+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:26.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:26.428+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:27.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:27.015+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:27.016546+0000) 2022-04-03T21:24:27.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:27.182+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:27.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:27.460+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:28.018 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:28.015+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:28.016744+0000) 2022-04-03T21:24:28.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:28.183+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:28.462 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:28.459+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:29.018 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:29.016+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:29.016932+0000) 2022-04-03T21:24:29.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:29.178+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:29.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:29.464+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:29.924 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:24:29.922+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:24:30.018 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:30.016+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:30.017142+0000) 2022-04-03T21:24:30.133 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:30.130+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:30.420 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:30.417+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:31.018 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:31.016+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:31.017319+0000) 2022-04-03T21:24:31.118 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:31.114+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:31.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:31.442+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:32.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:32.016+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:32.017535+0000) 2022-04-03T21:24:32.069 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:32.066+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:32.473 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:32.470+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:33.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:33.017+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:33.017754+0000) 2022-04-03T21:24:33.049 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:33.046+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:33.471 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:33.468+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:34.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:34.017+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:34.017937+0000) 2022-04-03T21:24:34.025 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:34.023+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:34.453 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:34.450+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:35.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:35.017+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:35.018138+0000) 2022-04-03T21:24:35.058 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:35.056+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:35.408 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:35.406+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:36.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:36.017+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:36.018355+0000) 2022-04-03T21:24:36.069 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:36.067+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:36.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:36.380+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:37.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:37.017+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:37.018547+0000) 2022-04-03T21:24:37.042 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:37.039+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:37.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:37.422+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:38.020 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:38.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:38.018743+0000) 2022-04-03T21:24:38.027 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:38.025+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:38.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:38.390+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:39.020 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:39.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:39.019003+0000) 2022-04-03T21:24:39.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:39.064+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:39.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:39.402+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:40.020 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:40.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:40.019213+0000) 2022-04-03T21:24:40.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:40.079+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:40.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:40.375+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:41.020 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:41.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:41.019417+0000) 2022-04-03T21:24:41.098 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:41.096+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:41.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:41.339+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:41.657 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:24:41.661 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:24:42.021 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:42.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:42.019650+0000) 2022-04-03T21:24:42.060 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:42.057+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:42.308 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:42.305+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:43.021 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:43.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:43.019851+0000) 2022-04-03T21:24:43.097 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:43.094+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:43.298 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:43.296+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:44.021 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:44.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:44.020085+0000) 2022-04-03T21:24:44.098 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:44.096+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:44.312 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:44.310+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:44.930 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:24:44.928+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:24:45.021 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:45.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:45.020308+0000) 2022-04-03T21:24:45.074 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:45.072+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:45.338 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:45.336+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:46.021 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:46.020+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:46.020504+0000) 2022-04-03T21:24:46.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:46.083+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:46.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:46.317+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:47.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:47.020+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:47.020646+0000) 2022-04-03T21:24:47.066 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:47.064+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:47.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:47.339+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:48.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:48.020+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:48.020878+0000) 2022-04-03T21:24:48.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:48.103+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:48.308 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:48.305+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:49.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:49.020+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:49.021073+0000) 2022-04-03T21:24:49.151 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:49.149+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:49.331 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:49.329+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:50.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:50.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:50.021281+0000) 2022-04-03T21:24:50.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:50.178+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:50.338 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:50.335+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:51.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:51.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:51.021460+0000) 2022-04-03T21:24:51.212 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:51.210+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:51.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:51.374+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:52.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:52.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:52.021632+0000) 2022-04-03T21:24:52.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:52.242+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:52.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:52.411+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:53.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:53.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:53.021859+0000) 2022-04-03T21:24:53.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:53.247+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:53.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:53.372+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:54.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:54.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:54.022016+0000) 2022-04-03T21:24:54.226 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:54.223+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:54.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:54.372+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:55.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:55.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:55.022207+0000) 2022-04-03T21:24:55.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:55.260+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:55.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:55.326+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:56.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:56.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:56.022369+0000) 2022-04-03T21:24:56.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:56.264+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:56.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:56.280+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:57.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:57.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:57.022554+0000) 2022-04-03T21:24:57.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:57.273+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:57.327 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:57.325+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:58.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:58.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:58.022756+0000) 2022-04-03T21:24:58.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:58.271+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:58.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:58.354+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:59.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:24:59.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:24:59.022973+0000) 2022-04-03T21:24:59.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:24:59.272+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:24:59.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:24:59.404+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:24:59.945 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:24:59.942+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:25:00.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:00.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:00.023179+0000) 2022-04-03T21:25:00.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:00.234+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:00.386 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:00.385+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:01.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:01.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:01.023406+0000) 2022-04-03T21:25:01.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:01.194+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:01.365 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:01.362+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:02.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:02.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:02.023615+0000) 2022-04-03T21:25:02.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:02.200+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:02.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:02.314+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:03.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:03.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:03.023848+0000) 2022-04-03T21:25:03.183 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:03.181+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:03.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:03.313+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:04.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:04.023+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:04.024028+0000) 2022-04-03T21:25:04.214 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:04.212+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:04.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:04.361+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:05.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:05.023+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:05.024224+0000) 2022-04-03T21:25:05.205 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:05.203+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:05.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:05.387+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:06.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:06.023+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:06.024439+0000) 2022-04-03T21:25:06.195 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:06.193+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:06.398 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:06.397+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:07.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:07.023+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:07.024610+0000) 2022-04-03T21:25:07.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:07.171+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:07.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:07.382+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:08.026 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:08.024+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:08.024753+0000) 2022-04-03T21:25:08.125 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:08.123+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:08.424 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:08.422+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:09.026 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:09.024+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:09.024985+0000) 2022-04-03T21:25:09.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:09.083+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:09.465 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:09.464+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:10.026 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:10.024+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:10.025190+0000) 2022-04-03T21:25:10.066 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:10.064+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:10.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:10.449+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:11.026 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:11.024+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:11.025377+0000) 2022-04-03T21:25:11.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:11.084+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:11.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:11.481+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:11.813 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:25:11.817 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:25:12.026 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:12.024+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:12.025609+0000) 2022-04-03T21:25:12.089 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:12.088+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:12.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:12.512+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:13.027 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:13.025+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:13.025851+0000) 2022-04-03T21:25:13.076 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:13.075+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:13.496 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:13.494+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:14.027 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:14.025+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:14.026080+0000) 2022-04-03T21:25:14.070 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:14.068+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:14.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:14.459+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:14.961 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:25:14.959+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:25:15.025 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:15.023+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:15.027 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:15.025+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:15.026285+0000) 2022-04-03T21:25:15.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:15.415+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:16.027 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:16.025+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:16.026486+0000) 2022-04-03T21:25:16.070 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:16.068+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:16.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:16.433+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:17.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:17.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:17.026655+0000) 2022-04-03T21:25:17.037 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:17.035+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:17.433 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:17.432+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:18.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:18.002+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:18.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:18.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:18.026911+0000) 2022-04-03T21:25:18.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:18.417+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:18.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:18.993+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:19.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:19.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:19.027121+0000) 2022-04-03T21:25:19.372 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:19.370+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:19.954 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:19.952+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:20.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:20.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:20.027326+0000) 2022-04-03T21:25:20.350 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:20.348+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:20.917 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:20.915+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:21.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:21.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:21.027495+0000) 2022-04-03T21:25:21.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:21.380+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:21.955 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:21.953+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:22.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:22.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:22.027714+0000) 2022-04-03T21:25:22.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:22.336+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:22.975 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:22.972+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:23.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:23.027+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:23.027890+0000) 2022-04-03T21:25:23.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:23.355+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:23.994 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:23.992+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:24.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:24.027+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:24.028068+0000) 2022-04-03T21:25:24.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:24.366+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:24.984 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:24.982+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:25.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:25.027+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:25.028263+0000) 2022-04-03T21:25:25.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:25.334+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:25.998 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:25.995+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:26.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:26.028+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:26.028462+0000) 2022-04-03T21:25:26.366 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:26.363+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:27.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:27.028+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:27.028612+0000) 2022-04-03T21:25:27.046 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:27.043+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:27.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:27.323+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:28.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:28.002+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:28.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:28.028+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:28.028761+0000) 2022-04-03T21:25:28.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:28.284+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:28.989 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:28.987+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:29.030 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:29.028+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:29.028945+0000) 2022-04-03T21:25:29.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:29.293+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:29.977 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:25:29.974+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:25:29.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:29.977+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:30.030 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:30.028+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:30.029161+0000) 2022-04-03T21:25:30.334 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:30.332+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:30.968 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:30.966+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:31.030 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:31.028+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:31.029323+0000) 2022-04-03T21:25:31.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:31.302+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:31.991 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:31.988+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:32.030 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:32.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:32.029468+0000) 2022-04-03T21:25:32.269 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:32.266+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:32.961 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:32.959+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:33.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:33.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:33.029669+0000) 2022-04-03T21:25:33.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:33.292+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:33.974 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:33.972+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:34.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:34.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:34.029830+0000) 2022-04-03T21:25:34.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:34.265+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:35.018 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:35.015+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:35.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:35.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:35.030029+0000) 2022-04-03T21:25:35.264 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:35.262+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:35.989 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:35.987+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:36.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:36.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:36.030226+0000) 2022-04-03T21:25:36.224 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:36.221+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:36.999 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:36.997+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:37.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:37.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:37.030459+0000) 2022-04-03T21:25:37.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:37.227+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:38.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:38.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:38.030699+0000) 2022-04-03T21:25:38.043 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:38.041+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:38.251 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:38.249+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:39.011 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:39.009+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:39.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:39.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:39.030909+0000) 2022-04-03T21:25:39.252 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:39.250+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:40.010 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:40.008+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:40.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:40.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:40.031117+0000) 2022-04-03T21:25:40.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:40.228+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:41.014 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:41.011+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:41.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:41.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:41.031335+0000) 2022-04-03T21:25:41.237 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:41.234+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:41.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:41.977+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:42.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:42.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:42.031507+0000) 2022-04-03T21:25:42.170 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:25:42.174 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:25:42.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:42.282+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:42.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:42.994+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:43.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:43.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:43.031701+0000) 2022-04-03T21:25:43.294 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:43.291+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:44.033 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:44.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:44.031885+0000) 2022-04-03T21:25:44.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:44.036+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:44.298 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:44.295+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:44.993 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:25:44.989+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:25:45.018 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:45.016+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:45.033 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:45.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:45.032091+0000) 2022-04-03T21:25:45.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:45.323+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:46.033 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:46.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:46.032291+0000) 2022-04-03T21:25:46.044 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:46.042+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:46.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:46.319+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:47.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:47.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:47.032519+0000) 2022-04-03T21:25:47.041 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:47.039+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:47.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:47.277+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:48.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:48.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:48.032694+0000) 2022-04-03T21:25:48.086 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:48.084+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:48.243 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:48.241+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:49.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:49.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:49.032812+0000) 2022-04-03T21:25:49.080 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:49.077+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:49.241 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:49.239+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:50.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:50.032+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:50.033069+0000) 2022-04-03T21:25:50.083 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:50.081+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:50.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:50.279+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:51.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:51.032+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:51.033254+0000) 2022-04-03T21:25:51.097 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:51.095+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:51.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:51.327+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:52.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:52.032+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:52.033411+0000) 2022-04-03T21:25:52.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:52.135+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:52.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:52.365+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:53.035 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:53.032+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:53.033659+0000) 2022-04-03T21:25:53.131 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:53.129+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:53.392 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:53.390+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:54.035 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:54.033+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:54.033824+0000) 2022-04-03T21:25:54.132 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:54.130+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:54.403 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:54.400+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:55.035 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:55.033+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:55.033992+0000) 2022-04-03T21:25:55.135 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:55.133+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:55.424 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:55.422+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:56.035 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:56.033+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:56.034193+0000) 2022-04-03T21:25:56.131 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:56.129+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:56.469 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:56.467+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:57.035 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:57.033+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:57.034396+0000) 2022-04-03T21:25:57.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:57.147+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:57.466 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:57.464+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:58.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:58.033+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:58.034598+0000) 2022-04-03T21:25:58.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:58.146+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:58.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:58.461+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:25:59.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:25:59.034+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:25:59.034848+0000) 2022-04-03T21:25:59.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:25:59.178+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:25:59.457 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:25:59.454+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:00.008 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:26:00.006+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:26:00.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:00.034+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:00.035070+0000) 2022-04-03T21:26:00.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:00.227+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:00.457 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:00.455+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:01.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:01.034+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:01.035274+0000) 2022-04-03T21:26:01.214 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:01.212+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:01.484 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:01.482+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:02.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:02.034+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:02.035483+0000) 2022-04-03T21:26:02.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:02.241+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:02.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:02.480+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:03.037 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:03.034+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:03.035699+0000) 2022-04-03T21:26:03.206 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:03.204+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:03.527 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:03.526+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:04.037 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:04.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:04.035864+0000) 2022-04-03T21:26:04.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:04.218+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:04.554 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:04.552+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:05.037 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:05.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:05.036073+0000) 2022-04-03T21:26:05.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:05.186+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:05.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:05.513+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:06.037 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:06.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:06.036279+0000) 2022-04-03T21:26:06.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:06.192+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:06.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:06.508+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:07.037 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:07.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:07.036458+0000) 2022-04-03T21:26:07.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:07.220+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:07.462 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:07.461+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:08.038 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:08.036+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:08.036674+0000) 2022-04-03T21:26:08.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:08.262+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:08.462 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:08.461+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:09.038 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:09.036+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:09.036883+0000) 2022-04-03T21:26:09.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:09.297+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:09.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:09.425+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:10.038 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:10.036+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:10.037105+0000) 2022-04-03T21:26:10.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:10.303+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:10.408 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:10.406+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:11.038 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:11.036+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:11.037286+0000) 2022-04-03T21:26:11.271 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:11.270+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:11.432 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:11.430+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:12.038 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:12.037+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:12.037503+0000) 2022-04-03T21:26:12.226 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:12.225+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:12.234 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:26:12.238 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:26:12.475 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:12.474+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:13.039 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:13.037+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:13.037794+0000) 2022-04-03T21:26:13.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:13.187+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:13.499 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:13.497+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:14.039 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:14.037+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:14.038006+0000) 2022-04-03T21:26:14.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:14.221+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:14.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:14.452+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:15.024 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:26:15.022+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:26:15.039 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:15.037+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:15.038218+0000) 2022-04-03T21:26:15.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:15.191+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:15.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:15.435+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:16.039 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:16.038+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:16.038423+0000) 2022-04-03T21:26:16.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:16.162+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:16.473 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:16.471+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:17.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:17.038+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:17.038616+0000) 2022-04-03T21:26:17.130 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:17.128+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:17.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:17.458+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:18.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:18.038+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:18.038846+0000) 2022-04-03T21:26:18.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:18.164+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:18.487 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:18.485+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:19.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:19.038+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:19.039081+0000) 2022-04-03T21:26:19.151 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:19.149+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:19.484 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:19.481+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:20.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:20.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:20.039300+0000) 2022-04-03T21:26:20.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:20.151+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:20.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:20.442+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:21.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:21.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:21.039500+0000) 2022-04-03T21:26:21.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:21.197+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:21.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:21.435+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:22.041 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:22.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:22.039700+0000) 2022-04-03T21:26:22.235 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:22.232+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:22.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:22.418+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:23.041 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:23.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:23.039888+0000) 2022-04-03T21:26:23.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:23.221+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:23.392 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:23.389+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:24.041 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:24.040+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:24.040127+0000) 2022-04-03T21:26:24.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:24.241+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:24.379 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:24.376+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:25.041 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:25.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:25.040346+0000) 2022-04-03T21:26:25.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:25.273+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:25.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:25.375+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:26.041 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:26.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:26.040548+0000) 2022-04-03T21:26:26.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:26.225+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:26.395 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:26.393+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:27.042 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:27.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:27.040749+0000) 2022-04-03T21:26:27.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:27.225+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:27.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:27.425+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:28.042 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:28.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:28.041006+0000) 2022-04-03T21:26:28.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:28.221+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:28.457 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:28.454+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:29.042 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:29.040+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:29.041174+0000) 2022-04-03T21:26:29.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:29.194+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:29.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:29.454+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:30.039 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:26:30.036+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:26:30.042 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:30.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:30.041409+0000) 2022-04-03T21:26:30.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:30.207+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:30.448 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:30.446+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:31.043 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:31.040+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:31.041612+0000) 2022-04-03T21:26:31.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:31.228+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:31.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:31.437+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:32.043 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:32.040+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:32.041860+0000) 2022-04-03T21:26:32.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:32.273+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:32.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:32.404+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:33.043 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:33.041+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:33.042044+0000) 2022-04-03T21:26:33.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:33.317+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:33.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:33.387+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:34.043 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:34.041+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:34.042244+0000) 2022-04-03T21:26:34.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:34.320+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:34.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:34.350+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:35.043 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:35.041+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:35.042460+0000) 2022-04-03T21:26:35.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:35.272+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:35.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:35.375+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:36.044 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:36.041+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:36.042683+0000) 2022-04-03T21:26:36.252 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:36.250+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:36.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:36.402+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:37.044 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:37.042+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:37.042900+0000) 2022-04-03T21:26:37.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:37.268+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:37.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:37.413+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:38.044 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:38.042+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:38.043063+0000) 2022-04-03T21:26:38.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:38.291+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:38.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:38.375+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:39.044 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:39.042+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:39.043269+0000) 2022-04-03T21:26:39.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:39.265+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:39.400 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:39.397+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:26:40.044 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:40.042+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:40.043500+0000) 2022-04-03T21:26:40.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:40.216+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:40.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:40.387+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:26:41.045 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:41.042+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:41.043721+0000) 2022-04-03T21:26:41.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:41.234+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:41.433 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:41.431+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:26:42.045 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:42.043+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:42.043952+0000) 2022-04-03T21:26:42.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:42.230+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:42.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:42.419+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:26:42.448 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:26:42.452 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:26:43.045 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:43.043+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:43.044180+0000) 2022-04-03T21:26:43.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:43.265+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:43.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:43.449+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:26:44.045 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:44.043+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:44.044368+0000) 2022-04-03T21:26:44.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:44.308+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:44.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:44.441+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:26:45.045 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:45.043+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:45.044526+0000) 2022-04-03T21:26:45.055 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:26:45.053+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:26:45.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:45.304+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:45.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:45.403+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:26:46.046 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:46.044+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:46.044749+0000) 2022-04-03T21:26:46.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:46.312+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:46.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:46.404+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:26:47.046 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:47.044+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:47.045014+0000) 2022-04-03T21:26:47.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:47.341+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:47.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:47.369+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:26:48.046 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:48.044+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:48.045255+0000) 2022-04-03T21:26:48.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:48.343+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:48.400 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:48.398+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:26:49.046 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:49.044+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:49.045470+0000) 2022-04-03T21:26:49.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:49.302+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:49.366 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:49.364+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:26:50.047 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:50.045+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:50.045690+0000) 2022-04-03T21:26:50.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:50.303+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:50.325 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:50.323+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:26:51.047 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:51.045+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:51.045863+0000) 2022-04-03T21:26:51.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:51.327+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:51.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:51.337+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:26:52.047 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:52.045+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:52.046069+0000) 2022-04-03T21:26:52.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:52.301+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:52.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:52.313+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:26:53.047 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:53.045+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:53.046205+0000) 2022-04-03T21:26:53.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:53.259+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:53.270 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:53.268+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:26:54.047 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:54.046+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:54.046404+0000) 2022-04-03T21:26:54.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:54.275+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:26:54.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:54.290+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:55.048 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:55.046+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:55.046583+0000) 2022-04-03T21:26:55.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:55.278+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:55.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:55.309+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:26:56.048 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:56.046+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:56.046858+0000) 2022-04-03T21:26:56.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:56.259+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:56.270 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:56.268+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:26:57.048 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:57.046+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:57.047063+0000) 2022-04-03T21:26:57.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:57.283+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:57.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:57.309+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:26:58.048 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:58.046+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:58.047271+0000) 2022-04-03T21:26:58.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:58.288+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:58.347 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:58.345+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:26:59.049 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:26:59.047+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:26:59.047490+0000) 2022-04-03T21:26:59.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:26:59.271+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:26:59.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:26:59.388+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:00.049 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:00.046+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:00.047670+0000) 2022-04-03T21:27:00.071 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:27:00.069+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:27:00.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:00.296+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:00.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:00.355+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:01.049 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:01.047+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:01.047881+0000) 2022-04-03T21:27:01.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:01.284+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:01.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:01.392+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:02.049 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:02.047+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:02.048104+0000) 2022-04-03T21:27:02.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:02.304+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:02.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:02.416+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:03.049 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:03.047+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:03.048265+0000) 2022-04-03T21:27:03.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:03.269+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:03.462 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:03.460+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:04.049 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:04.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:04.048446+0000) 2022-04-03T21:27:04.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:04.221+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:04.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:04.490+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:05.049 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:05.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:05.048627+0000) 2022-04-03T21:27:05.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:05.214+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:05.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:05.487+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:06.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:06.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:06.048856+0000) 2022-04-03T21:27:06.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:06.179+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:06.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:06.457+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:07.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:07.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:07.049059+0000) 2022-04-03T21:27:07.226 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:07.224+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:07.482 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:07.481+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:08.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:08.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:08.049259+0000) 2022-04-03T21:27:08.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:08.190+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:08.508 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:08.506+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:09.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:09.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:09.049475+0000) 2022-04-03T21:27:09.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:09.240+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:09.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:09.547+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:10.051 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:10.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:10.049699+0000) 2022-04-03T21:27:10.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:10.190+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:10.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:10.512+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:11.051 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:11.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:11.049889+0000) 2022-04-03T21:27:11.144 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:11.142+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:11.469 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:11.466+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:12.051 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:12.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:12.050092+0000) 2022-04-03T21:27:12.120 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:12.117+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:12.507 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:12.504+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:12.527 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:27:12.531 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:27:13.051 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:13.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:13.050348+0000) 2022-04-03T21:27:13.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:13.161+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:13.524 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:13.521+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:14.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:14.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:14.050558+0000) 2022-04-03T21:27:14.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:14.166+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:14.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:14.484+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:15.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:15.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:15.050761+0000) 2022-04-03T21:27:15.087 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:27:15.084+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:27:15.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:15.148+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:15.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:15.472+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:16.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:16.050+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:16.050951+0000) 2022-04-03T21:27:16.183 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:16.181+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:16.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:16.484+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:17.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:17.050+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:17.051157+0000) 2022-04-03T21:27:17.143 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:17.140+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:17.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:17.529+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:18.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:18.050+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:18.051365+0000) 2022-04-03T21:27:18.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:18.164+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:18.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:18.508+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:19.053 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:19.050+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:19.051562+0000) 2022-04-03T21:27:19.131 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:19.129+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:19.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:19.528+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:20.053 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:20.051+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:20.051791+0000) 2022-04-03T21:27:20.083 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:20.080+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:20.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:20.530+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:21.053 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:21.051+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:21.051982+0000) 2022-04-03T21:27:21.077 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:21.075+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:21.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:21.546+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:22.053 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:22.051+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:22.052204+0000) 2022-04-03T21:27:22.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:22.064+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:22.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:22.520+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:23.053 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:23.051+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:23.052396+0000) 2022-04-03T21:27:23.079 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:23.076+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:23.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:23.471+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:24.054 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:24.051+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:24.052626+0000) 2022-04-03T21:27:24.110 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:24.108+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:24.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:24.428+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:25.054 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:25.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:25.052851+0000) 2022-04-03T21:27:25.084 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:25.082+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:25.457 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:25.454+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:26.054 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:26.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:26.053030+0000) 2022-04-03T21:27:26.103 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:26.100+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:26.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:26.456+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:27.054 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:27.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:27.053217+0000) 2022-04-03T21:27:27.119 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:27.117+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:27.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:27.462+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:28.054 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:28.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:28.053370+0000) 2022-04-03T21:27:28.127 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:28.125+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:28.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:28.441+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:29.054 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:29.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:29.053554+0000) 2022-04-03T21:27:29.102 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:29.100+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:29.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:29.421+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:30.055 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:30.053+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:30.053765+0000) 2022-04-03T21:27:30.102 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:27:30.101+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:27:30.128 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:30.126+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:30.462 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:30.460+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:31.055 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:31.053+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:31.053994+0000) 2022-04-03T21:27:31.113 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:31.111+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:31.433 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:31.431+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:32.055 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:32.053+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:32.054221+0000) 2022-04-03T21:27:32.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:32.151+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:32.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:32.423+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:33.055 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:33.053+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:33.054416+0000) 2022-04-03T21:27:33.113 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:33.110+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:33.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:33.436+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:34.056 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:34.054+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:34.054626+0000) 2022-04-03T21:27:34.089 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:34.087+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:34.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:34.397+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:35.056 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:35.054+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:35.054864+0000) 2022-04-03T21:27:35.065 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:35.062+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:35.442 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:35.439+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:36.056 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:36.054+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:36.055073+0000) 2022-04-03T21:27:36.111 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:36.109+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:36.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:36.413+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:37.056 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:37.054+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:37.055280+0000) 2022-04-03T21:27:37.151 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:37.149+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:37.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:37.380+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:38.056 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:38.055+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:38.055493+0000) 2022-04-03T21:27:38.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:38.173+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:38.333 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:38.331+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:39.057 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:39.055+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:39.055713+0000) 2022-04-03T21:27:39.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:39.185+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:39.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:39.327+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:40.057 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:40.055+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:40.055935+0000) 2022-04-03T21:27:40.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:40.160+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:40.348 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:40.346+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:41.057 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:41.055+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:41.056157+0000) 2022-04-03T21:27:41.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:41.135+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:41.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:41.364+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:42.057 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:42.056+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:42.056390+0000) 2022-04-03T21:27:42.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:42.113+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:42.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:42.415+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:42.678 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:27:42.681 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:27:43.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:43.056+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:43.056624+0000) 2022-04-03T21:27:43.089 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:43.087+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:43.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:43.407+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:44.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:44.056+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:44.056862+0000) 2022-04-03T21:27:44.091 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:44.089+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:44.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:44.388+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:45.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:45.056+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:45.057040+0000) 2022-04-03T21:27:45.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:45.079+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:45.118 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:27:45.116+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:27:45.348 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:45.345+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:46.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:46.056+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:46.057202+0000) 2022-04-03T21:27:46.090 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:46.087+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:46.337 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:46.335+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:47.051 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:47.049+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:47.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:47.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:47.057411+0000) 2022-04-03T21:27:47.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:47.380+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:48.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:48.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:48.057624+0000) 2022-04-03T21:27:48.091 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:48.089+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:48.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:48.371+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:49.059 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:49.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:49.057852+0000) 2022-04-03T21:27:49.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:49.079+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:49.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:49.417+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:50.059 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:50.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:50.057980+0000) 2022-04-03T21:27:50.117 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:50.115+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:50.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:50.371+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:51.059 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:51.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:51.058200+0000) 2022-04-03T21:27:51.072 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:51.070+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:51.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:51.324+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:52.059 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:52.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:52.058382+0000) 2022-04-03T21:27:52.091 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:52.089+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:52.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:52.361+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:53.060 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:53.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:53.058584+0000) 2022-04-03T21:27:53.100 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:53.099+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:53.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:53.406+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:54.051 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:54.049+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:54.060 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:54.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:54.058805+0000) 2022-04-03T21:27:54.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:54.442+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:55.060 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:55.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:55.058997+0000) 2022-04-03T21:27:55.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:55.084+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:55.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:55.441+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:56.060 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:56.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:56.059199+0000) 2022-04-03T21:27:56.097 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:56.095+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:56.396 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:56.394+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:57.060 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:57.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:57.059437+0000) 2022-04-03T21:27:57.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:57.111+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:57.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:57.410+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:58.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:58.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:58.059657+0000) 2022-04-03T21:27:58.142 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:58.141+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:58.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:58.443+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:27:59.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:27:59.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:27:59.059842+0000) 2022-04-03T21:27:59.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:27:59.187+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:27:59.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:27:59.400+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:00.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:00.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:00.060038+0000) 2022-04-03T21:28:00.134 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:28:00.131+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:28:00.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:00.221+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:00.364 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:00.362+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:01.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:01.059+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:01.060230+0000) 2022-04-03T21:28:01.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:01.268+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:01.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:01.320+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:02.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:02.059+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:02.060442+0000) 2022-04-03T21:28:02.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:02.307+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:02.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:02.313+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:03.062 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:03.059+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:03.060657+0000) 2022-04-03T21:28:03.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:03.286+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:03.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:03.287+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:04.062 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:04.060+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:04.060869+0000) 2022-04-03T21:28:04.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:04.318+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:04.337 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:04.335+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:05.062 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:05.060+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:05.061085+0000) 2022-04-03T21:28:05.327 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:05.326+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:05.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:05.349+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:06.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:06.060+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:06.061290+0000) 2022-04-03T21:28:06.303 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:06.300+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:06.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:06.349+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:07.062 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:07.060+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:07.061513+0000) 2022-04-03T21:28:07.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:07.308+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:07.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:07.368+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:08.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:08.060+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:08.061741+0000) 2022-04-03T21:28:08.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:08.336+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:08.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:08.384+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:09.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:09.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:09.061966+0000) 2022-04-03T21:28:09.318 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:09.315+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:09.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:09.374+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:10.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:10.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:10.062131+0000) 2022-04-03T21:28:10.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:10.323+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:10.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:10.350+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 18 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:11.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:11.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:11.062290+0000) 2022-04-03T21:28:11.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:11.347+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:11.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:11.358+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:12.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:12.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:12.062458+0000) 2022-04-03T21:28:12.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:12.360+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:12.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:12.360+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:12.834 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:28:12.837 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:28:13.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:13.062+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:13.062687+0000) 2022-04-03T21:28:13.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:13.344+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:13.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:13.386+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:14.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:14.062+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:14.062886+0000) 2022-04-03T21:28:14.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:14.338+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:14.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:14.411+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:15.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:15.062+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:15.063098+0000) 2022-04-03T21:28:15.150 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:28:15.148+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:28:15.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:15.324+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:15.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:15.405+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:16.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:16.062+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:16.063312+0000) 2022-04-03T21:28:16.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:16.301+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:16.370 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:16.368+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:17.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:17.062+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:17.063557+0000) 2022-04-03T21:28:17.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:17.341+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:17.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:17.374+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:18.065 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:18.063+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:18.063755+0000) 2022-04-03T21:28:18.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:18.315+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:18.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:18.327+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:19.065 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:19.063+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:19.063947+0000) 2022-04-03T21:28:19.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:19.270+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:19.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:19.355+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:20.065 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:20.063+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:20.064141+0000) 2022-04-03T21:28:20.271 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:20.269+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:20.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:20.374+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:21.065 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:21.063+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:21.064339+0000) 2022-04-03T21:28:21.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:21.292+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:21.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:21.328+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:22.065 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:22.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:22.064559+0000) 2022-04-03T21:28:22.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:22.272+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:22.372 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:22.369+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:23.066 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:23.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:23.064744+0000) 2022-04-03T21:28:23.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:23.300+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:23.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:23.339+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:24.066 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:24.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:24.064951+0000) 2022-04-03T21:28:24.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:24.274+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:24.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:24.293+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:25.066 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:25.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:25.065064+0000) 2022-04-03T21:28:25.262 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:25.259+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:25.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:25.316+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:26.066 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:26.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:26.065188+0000) 2022-04-03T21:28:26.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:26.275+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:26.291 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:26.289+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:27.066 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:27.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:27.065414+0000) 2022-04-03T21:28:27.266 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:27.264+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:27.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:27.285+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:28.066 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:28.065+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:28.065557+0000) 2022-04-03T21:28:28.241 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:28.238+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:28.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:28.260+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:29.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:29.065+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:29.065728+0000) 2022-04-03T21:28:29.234 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:29.232+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:29.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:29.284+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:30.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:30.065+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:30.065930+0000) 2022-04-03T21:28:30.166 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:28:30.164+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:28:30.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:30.245+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:30.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:30.317+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:31.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:31.065+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:31.066081+0000) 2022-04-03T21:28:31.275 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:31.273+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:31.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:31.274+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:32.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:32.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:32.066302+0000) 2022-04-03T21:28:32.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:32.246+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:32.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:32.319+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:33.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:33.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:33.066444+0000) 2022-04-03T21:28:33.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:33.213+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:33.344 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:33.342+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:34.068 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:34.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:34.066614+0000) 2022-04-03T21:28:34.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:34.185+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:34.325 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:34.323+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:35.068 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:35.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:35.066804+0000) 2022-04-03T21:28:35.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:35.155+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:35.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:35.365+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:36.068 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:36.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:36.066975+0000) 2022-04-03T21:28:36.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:36.184+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:36.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:36.380+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:37.068 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:37.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:37.067131+0000) 2022-04-03T21:28:37.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:37.187+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:37.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:37.349+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:38.068 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:38.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:38.067279+0000) 2022-04-03T21:28:38.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:38.229+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:38.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:38.327+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:39.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:39.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:39.067450+0000) 2022-04-03T21:28:39.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:39.207+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:39.372 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:39.370+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:40.068 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:40.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:40.067584+0000) 2022-04-03T21:28:40.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:40.226+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:40.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:40.350+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:41.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:41.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:41.067726+0000) 2022-04-03T21:28:41.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:41.220+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:41.304 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:41.301+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:42.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:42.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:42.067900+0000) 2022-04-03T21:28:42.213 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:42.211+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:42.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:42.276+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:42.964 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:28:42.968 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:28:43.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:43.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:43.068138+0000) 2022-04-03T21:28:43.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:43.188+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:43.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:43.308+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:44.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:44.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:44.068249+0000) 2022-04-03T21:28:44.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:44.199+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:44.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:44.314+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:45.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:45.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:45.068374+0000) 2022-04-03T21:28:45.182 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:28:45.180+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:28:45.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:45.192+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:45.291 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:45.289+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:46.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:46.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:46.068514+0000) 2022-04-03T21:28:46.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:46.197+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:46.264 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:46.262+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:47.070 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:47.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:47.068685+0000) 2022-04-03T21:28:47.154 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:47.152+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:47.239 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:47.237+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:48.070 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:48.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:48.068824+0000) 2022-04-03T21:28:48.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:48.148+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:48.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:48.194+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:49.070 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:49.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:49.069012+0000) 2022-04-03T21:28:49.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:49.187+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:49.241 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:49.239+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:50.070 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:50.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:50.069156+0000) 2022-04-03T21:28:50.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:50.158+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:50.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:50.266+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:51.070 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:51.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:51.069327+0000) 2022-04-03T21:28:51.182 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:51.181+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:51.298 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:51.296+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:52.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:52.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:52.069489+0000) 2022-04-03T21:28:52.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:52.183+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:52.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:52.263+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:53.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:53.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:53.069694+0000) 2022-04-03T21:28:53.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:53.160+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:53.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:53.271+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:54.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:54.069+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:54.069924+0000) 2022-04-03T21:28:54.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:54.196+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:54.293 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:54.291+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:55.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:55.069+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:55.070126+0000) 2022-04-03T21:28:55.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:55.229+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:55.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:55.327+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:56.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:56.069+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:56.070345+0000) 2022-04-03T21:28:56.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:56.238+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:56.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:56.335+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:57.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:57.069+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:57.070545+0000) 2022-04-03T21:28:57.217 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:57.216+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:57.308 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:57.306+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:58.072 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:58.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:58.070795+0000) 2022-04-03T21:28:58.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:58.175+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:58.331 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:58.329+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:28:59.072 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:28:59.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:28:59.070957+0000) 2022-04-03T21:28:59.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:28:59.216+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:28:59.350 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:28:59.349+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:00.072 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:00.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:00.071164+0000) 2022-04-03T21:29:00.198 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:29:00.196+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:29:00.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:00.206+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:00.334 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:00.332+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:01.072 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:01.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:01.071318+0000) 2022-04-03T21:29:01.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:01.244+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:01.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:01.379+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:02.072 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:02.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:02.071479+0000) 2022-04-03T21:29:02.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:02.294+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:02.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:02.339+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:03.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:03.071+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:03.071690+0000) 2022-04-03T21:29:03.284 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:03.281+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:03.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:03.328+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:04.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:04.071+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:04.071903+0000) 2022-04-03T21:29:04.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:04.311+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:04.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:04.327+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:05.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:05.071+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:05.072078+0000) 2022-04-03T21:29:05.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:05.290+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:05.350 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:05.347+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:06.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:06.071+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:06.072261+0000) 2022-04-03T21:29:06.301 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:06.299+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:06.332 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:06.329+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:07.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:07.072+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:07.072463+0000) 2022-04-03T21:29:07.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:07.268+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:07.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:07.292+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:08.074 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:08.072+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:08.072668+0000) 2022-04-03T21:29:08.313 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:08.310+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:08.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:08.316+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:09.074 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:09.072+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:09.072848+0000) 2022-04-03T21:29:09.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:09.276+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:09.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:09.310+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:10.074 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:10.072+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:10.073056+0000) 2022-04-03T21:29:10.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:10.323+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:10.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:10.346+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:11.074 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:11.072+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:11.073263+0000) 2022-04-03T21:29:11.327 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:11.324+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:11.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:11.338+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:12.074 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:12.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:12.073471+0000) 2022-04-03T21:29:12.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:12.319+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:12.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:12.361+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:13.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:13.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:13.073702+0000) 2022-04-03T21:29:13.086 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:29:13.090 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:29:13.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:13.330+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:13.402 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:13.400+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:14.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:14.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:14.073886+0000) 2022-04-03T21:29:14.370 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:14.367+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:14.376 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:14.373+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:15.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:15.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:15.074013+0000) 2022-04-03T21:29:15.214 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:29:15.211+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:29:15.365 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:15.362+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:15.396 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:15.394+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:16.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:16.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:16.074217+0000) 2022-04-03T21:29:16.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:16.351+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:16.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:16.371+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:17.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:17.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:17.074388+0000) 2022-04-03T21:29:17.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:17.354+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:17.400 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:17.397+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:18.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:18.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:18.074558+0000) 2022-04-03T21:29:18.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:18.341+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:18.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:18.414+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:19.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:19.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:19.074682+0000) 2022-04-03T21:29:19.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:19.298+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:19.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:19.427+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:20.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:20.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:20.074861+0000) 2022-04-03T21:29:20.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:20.323+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:20.400 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:20.398+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:21.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:21.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:21.075021+0000) 2022-04-03T21:29:21.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:21.307+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:21.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:21.388+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:22.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:22.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:22.075154+0000) 2022-04-03T21:29:22.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:22.327+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:22.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:22.417+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:23.077 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:23.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:23.075296+0000) 2022-04-03T21:29:23.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:23.289+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:23.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:23.458+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:24.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:24.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:24.075507+0000) 2022-04-03T21:29:24.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:24.323+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:24.505 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:24.503+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:25.077 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:25.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:25.075693+0000) 2022-04-03T21:29:25.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:25.306+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:25.554 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:25.552+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:26.077 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:26.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:26.075910+0000) 2022-04-03T21:29:26.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:26.286+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:26.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:26.518+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:27.077 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:27.075+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:27.076127+0000) 2022-04-03T21:29:27.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:27.328+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:27.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:27.527+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:28.077 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:28.075+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:28.076299+0000) 2022-04-03T21:29:28.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:28.358+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:28.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:28.535+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:29.077 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:29.075+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:29.076503+0000) 2022-04-03T21:29:29.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:29.389+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:29.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:29.507+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:30.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:30.075+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:30.076711+0000) 2022-04-03T21:29:30.230 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:29:30.226+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:29:30.388 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:30.386+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:30.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:30.508+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:31.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:31.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:31.076914+0000) 2022-04-03T21:29:31.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:31.342+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:31.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:31.510+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:32.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:32.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:32.077135+0000) 2022-04-03T21:29:32.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:32.315+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:32.472 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:32.469+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:33.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:33.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:33.077323+0000) 2022-04-03T21:29:33.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:33.342+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:33.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:33.462+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:34.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:34.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:34.077527+0000) 2022-04-03T21:29:34.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:34.336+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:34.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:34.457+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:35.079 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:35.077+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:35.077755+0000) 2022-04-03T21:29:35.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:35.326+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:35.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:35.485+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:36.079 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:36.077+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:36.077973+0000) 2022-04-03T21:29:36.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:36.288+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:36.442 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:36.440+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:37.079 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:37.077+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:37.078189+0000) 2022-04-03T21:29:37.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:37.254+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:37.412 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:37.410+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:38.079 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:38.077+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:38.078395+0000) 2022-04-03T21:29:38.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:38.247+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:38.457 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:38.455+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:39.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:39.077+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:39.078618+0000) 2022-04-03T21:29:39.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:39.294+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:39.504 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:39.502+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:40.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:40.078+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:40.078831+0000) 2022-04-03T21:29:40.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:40.257+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:40.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:40.501+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:41.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:41.078+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:41.079002+0000) 2022-04-03T21:29:41.301 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:41.300+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:41.496 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:41.494+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:42.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:42.078+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:42.079230+0000) 2022-04-03T21:29:42.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:42.261+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:42.472 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:42.471+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:43.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:43.078+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:43.079382+0000) 2022-04-03T21:29:43.201 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:29:43.204 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:29:43.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:43.309+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:43.499 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:43.497+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:44.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:44.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:44.079621+0000) 2022-04-03T21:29:44.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:44.338+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:44.490 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:44.488+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:45.081 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:45.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:45.079813+0000) 2022-04-03T21:29:45.245 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:29:45.243+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:29:45.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:45.318+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:45.527 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:45.526+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:46.081 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:46.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:46.079964+0000) 2022-04-03T21:29:46.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:46.296+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:46.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:46.530+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:47.081 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:47.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:47.080212+0000) 2022-04-03T21:29:47.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:47.322+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:47.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:47.487+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:48.081 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:48.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:48.080376+0000) 2022-04-03T21:29:48.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:48.355+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:48.450 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:48.448+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:49.081 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:49.080+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:49.080528+0000) 2022-04-03T21:29:49.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:49.328+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:49.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:49.456+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:50.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:50.080+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:50.080688+0000) 2022-04-03T21:29:50.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:50.377+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:50.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:50.429+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:51.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:51.080+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:51.080879+0000) 2022-04-03T21:29:51.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:51.372+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:51.475 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:51.474+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:52.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:52.080+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:52.081093+0000) 2022-04-03T21:29:52.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:52.370+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:52.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:52.446+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:53.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:53.080+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:53.081292+0000) 2022-04-03T21:29:53.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:53.384+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:53.480 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:53.478+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:54.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:54.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:54.081489+0000) 2022-04-03T21:29:54.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:54.347+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:54.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:54.478+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:55.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:55.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:55.081697+0000) 2022-04-03T21:29:55.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:55.306+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:55.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:55.429+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:56.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:56.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:56.081862+0000) 2022-04-03T21:29:56.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:56.320+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:56.475 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:56.473+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:57.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:57.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:57.082043+0000) 2022-04-03T21:29:57.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:57.371+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:57.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:57.521+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:58.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:58.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:58.082236+0000) 2022-04-03T21:29:58.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:58.347+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:58.530 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:58.528+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:29:59.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:29:59.082+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:29:59.082435+0000) 2022-04-03T21:29:59.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:29:59.339+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:29:59.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:29:59.490+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:00.084 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:00.082+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:00.082615+0000) 2022-04-03T21:30:00.261 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:30:00.259+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:30:00.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:00.319+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:00.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:00.485+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:01.084 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:01.082+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:01.082839+0000) 2022-04-03T21:30:01.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:01.370+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:01.530 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:01.527+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:02.084 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:02.082+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:02.082968+0000) 2022-04-03T21:30:02.401 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:02.399+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:02.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:02.568+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:03.084 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:03.083+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:03.083117+0000) 2022-04-03T21:30:03.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:03.360+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:03.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:03.564+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:04.084 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:04.083+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:04.083304+0000) 2022-04-03T21:30:04.382 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:04.380+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:04.589 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:04.586+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:05.085 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:05.083+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:05.083510+0000) 2022-04-03T21:30:05.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:05.356+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:05.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:05.592+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:06.085 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:06.082+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:06.083759+0000) 2022-04-03T21:30:06.332 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:06.329+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:06.597 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:06.594+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:07.085 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:07.082+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:07.083976+0000) 2022-04-03T21:30:07.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:07.283+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:07.634 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:07.631+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:08.085 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:08.083+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:08.084176+0000) 2022-04-03T21:30:08.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:08.266+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:08.643 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:08.640+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:09.085 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:09.083+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:09.084373+0000) 2022-04-03T21:30:09.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:09.220+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:09.674 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:09.672+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:10.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:10.083+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:10.084593+0000) 2022-04-03T21:30:10.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:10.228+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:10.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:10.689+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:11.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:11.083+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:11.084815+0000) 2022-04-03T21:30:11.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:11.225+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:11.666 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:11.663+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:12.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:12.084+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:12.084964+0000) 2022-04-03T21:30:12.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:12.201+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:12.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:12.639+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:13.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:13.084+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:13.085152+0000) 2022-04-03T21:30:13.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:13.225+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:13.557 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:30:13.561 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:30:13.596 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:13.593+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:14.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:14.084+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:14.085360+0000) 2022-04-03T21:30:14.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:14.257+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:14.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:14.614+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:15.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:15.084+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:15.085518+0000) 2022-04-03T21:30:15.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:15.237+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:15.277 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:30:15.275+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:30:15.636 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:15.633+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:16.087 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:16.084+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:16.085713+0000) 2022-04-03T21:30:16.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:16.260+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:16.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:16.623+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:17.087 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:17.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:17.085890+0000) 2022-04-03T21:30:17.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:17.279+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:17.675 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:17.672+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:18.087 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:18.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:18.086095+0000) 2022-04-03T21:30:18.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:18.315+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:18.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:18.691+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:19.087 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:19.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:19.086257+0000) 2022-04-03T21:30:19.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:19.317+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:19.721 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:19.718+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:20.087 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:20.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:20.086446+0000) 2022-04-03T21:30:20.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:20.300+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:20.757 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:20.754+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:21.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:21.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:21.086673+0000) 2022-04-03T21:30:21.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:21.261+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:21.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:21.735+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:22.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:22.086+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:22.086870+0000) 2022-04-03T21:30:22.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:22.267+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:22.786 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:22.784+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:23.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:23.086+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:23.087093+0000) 2022-04-03T21:30:23.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:23.311+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:23.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:23.789+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:24.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:24.086+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:24.087299+0000) 2022-04-03T21:30:24.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:24.316+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:24.795 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:24.792+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:25.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:25.086+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:25.087471+0000) 2022-04-03T21:30:25.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:25.324+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:25.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:25.783+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:26.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:26.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:26.087662+0000) 2022-04-03T21:30:26.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:26.352+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:26.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:26.833+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:27.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:27.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:27.087859+0000) 2022-04-03T21:30:27.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:27.320+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:27.787 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:27.785+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:28.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:28.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:28.088013+0000) 2022-04-03T21:30:28.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:28.360+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:28.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:28.750+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:29.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:29.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:29.088190+0000) 2022-04-03T21:30:29.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:29.348+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:29.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:29.737+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:30.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:30.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:30.088416+0000) 2022-04-03T21:30:30.293 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:30:30.291+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:30:30.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:30.328+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:30.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:30.712+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:31.090 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:31.088+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:31.088637+0000) 2022-04-03T21:30:31.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:31.309+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:31.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:31.723+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:32.090 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:32.088+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:32.088820+0000) 2022-04-03T21:30:32.337 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:32.335+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:32.703 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:32.701+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:33.090 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:33.088+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:33.088976+0000) 2022-04-03T21:30:33.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:33.325+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:33.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:33.708+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:34.090 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:34.088+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:34.089174+0000) 2022-04-03T21:30:34.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:34.330+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:34.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:34.730+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:35.090 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:35.088+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:35.089377+0000) 2022-04-03T21:30:35.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:35.309+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:35.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:35.776+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:36.090 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:36.089+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:36.089566+0000) 2022-04-03T21:30:36.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:36.261+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:36.756 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:36.754+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:37.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:37.089+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:37.089765+0000) 2022-04-03T21:30:37.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:37.240+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:37.804 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:37.802+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:38.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:38.089+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:38.089984+0000) 2022-04-03T21:30:38.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:38.262+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:38.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:38.787+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:39.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:39.089+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:39.090202+0000) 2022-04-03T21:30:39.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:39.284+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:39.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:39.741+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:40.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:40.090+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:40.090436+0000) 2022-04-03T21:30:40.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:40.262+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:40.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:40.775+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:41.092 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:41.090+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:41.090660+0000) 2022-04-03T21:30:41.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:41.275+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:41.817 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:41.815+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:42.092 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:42.090+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:42.090887+0000) 2022-04-03T21:30:42.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:42.273+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:42.854 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:42.852+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:43.092 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:43.090+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:43.091104+0000) 2022-04-03T21:30:43.233 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:43.232+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:43.754 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:30:43.758 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:30:43.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:43.895+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:44.092 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:44.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:44.091317+0000) 2022-04-03T21:30:44.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:44.190+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:44.930 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:44.929+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:45.093 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:45.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:45.091533+0000) 2022-04-03T21:30:45.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:45.162+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:45.309 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:30:45.307+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:30:45.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:45.922+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:46.093 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:46.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:46.091740+0000) 2022-04-03T21:30:46.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:46.113+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:46.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:46.875+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:47.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:47.065+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:47.093 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:47.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:47.091941+0000) 2022-04-03T21:30:47.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:47.899+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:48.034 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:48.033+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:48.093 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:48.092+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:48.092143+0000) 2022-04-03T21:30:48.866 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:48.865+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:49.033 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:49.032+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:49.093 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:49.092+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:49.092296+0000) 2022-04-03T21:30:49.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:49.858+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:50.027 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:50.024+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:50.093 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:50.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:50.092513+0000) 2022-04-03T21:30:50.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:50.852+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:50.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:50.976+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:51.094 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:51.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:51.092727+0000) 2022-04-03T21:30:51.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:51.829+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:51.950 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:51.947+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:52.094 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:52.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:52.092964+0000) 2022-04-03T21:30:52.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:52.842+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:52.951 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:52.948+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:53.094 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:53.092+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:53.093178+0000) 2022-04-03T21:30:53.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:53.808+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:53.955 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:53.953+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:54.094 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:54.092+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:54.093364+0000) 2022-04-03T21:30:54.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:54.780+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:55.002 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:54.999+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:55.095 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:55.092+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:55.093563+0000) 2022-04-03T21:30:55.798 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:55.795+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:56.017 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:56.015+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:56.095 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:56.092+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:56.093801+0000) 2022-04-03T21:30:56.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:56.800+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:57.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:57.064+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:57.095 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:57.093+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:57.093992+0000) 2022-04-03T21:30:57.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:57.822+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:58.049 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:58.047+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:58.095 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:58.093+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:58.094171+0000) 2022-04-03T21:30:58.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:58.818+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:30:59.021 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:30:59.019+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:30:59.095 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:30:59.093+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:30:59.094373+0000) 2022-04-03T21:30:59.777 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:30:59.774+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:00.036 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:00.034+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:00.096 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:00.093+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:00.094610+0000) 2022-04-03T21:31:00.324 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:31:00.322+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:31:00.795 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:00.792+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:01.029 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:01.026+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:01.097 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:01.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:01.094840+0000) 2022-04-03T21:31:01.773 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:01.771+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:02.023 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:02.021+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:02.096 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:02.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:02.095017+0000) 2022-04-03T21:31:02.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:02.779+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:03.024 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:03.021+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:03.096 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:03.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:03.095165+0000) 2022-04-03T21:31:03.738 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:03.736+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:04.063 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:04.060+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:04.101 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:04.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:04.095371+0000) 2022-04-03T21:31:04.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:04.777+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:05.080 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:05.077+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:05.096 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:05.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:05.095598+0000) 2022-04-03T21:31:05.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:05.777+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:06.082 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:06.080+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:06.097 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:06.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:06.095842+0000) 2022-04-03T21:31:06.773 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:06.770+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:07.034 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:07.032+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:07.097 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:07.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:07.096028+0000) 2022-04-03T21:31:07.736 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:07.733+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:08.054 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:08.052+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:08.097 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:08.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:08.096157+0000) 2022-04-03T21:31:08.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:08.715+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:09.051 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:09.048+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:09.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:09.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:09.096362+0000) 2022-04-03T21:31:09.686 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:09.683+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:10.070 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:10.068+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 14 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:10.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:10.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:10.096573+0000) 2022-04-03T21:31:10.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:10.686+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:11.024 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:11.022+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:11.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:11.096+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:11.096800+0000) 2022-04-03T21:31:11.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:11.733+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:12.018 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:12.016+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:12.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:12.096+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:12.096994+0000) 2022-04-03T21:31:12.747 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:12.744+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:13.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:13.065+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:13.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:13.096+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:13.097207+0000) 2022-04-03T21:31:13.706 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:13.704+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:13.901 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:31:13.905 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:31:14.077 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:14.075+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:14.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:14.096+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:14.097416+0000) 2022-04-03T21:31:14.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:14.734+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:15.091 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:15.089+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:15.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:15.097+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:15.097616+0000) 2022-04-03T21:31:15.340 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:31:15.338+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:31:15.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:15.757+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:16.053 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:16.051+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:16.099 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:16.097+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:16.097846+0000) 2022-04-03T21:31:16.768 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:16.765+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:17.018 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:17.016+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:17.099 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:17.097+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:17.098064+0000) 2022-04-03T21:31:17.730 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:17.728+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:17.971 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:17.969+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:18.099 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:18.097+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:18.098274+0000) 2022-04-03T21:31:18.696 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:18.694+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:18.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:18.941+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:19.099 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:19.098+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:19.098505+0000) 2022-04-03T21:31:19.650 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:19.648+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:19.926 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:19.924+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:20.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:20.098+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:20.098732+0000) 2022-04-03T21:31:20.639 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:20.637+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:20.947 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:20.945+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:21.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:21.098+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:21.098903+0000) 2022-04-03T21:31:21.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:21.680+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:21.941 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:21.939+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:22.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:22.098+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:22.099069+0000) 2022-04-03T21:31:22.660 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:22.658+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:22.967 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:22.965+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:23.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:23.098+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:23.099267+0000) 2022-04-03T21:31:23.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:23.688+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:23.926 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:23.924+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:24.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:24.099+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:24.099453+0000) 2022-04-03T21:31:24.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:24.669+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:24.890 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:24.887+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:25.101 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:25.098+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:25.099672+0000) 2022-04-03T21:31:25.652 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:25.649+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:25.886 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:25.884+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:26.101 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:26.099+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:26.099873+0000) 2022-04-03T21:31:26.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:26.642+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:26.849 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:26.847+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:27.101 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:27.099+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:27.100042+0000) 2022-04-03T21:31:27.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:27.667+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:27.831 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:27.829+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:28.101 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:28.100+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:28.100240+0000) 2022-04-03T21:31:28.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:28.619+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:28.810 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:28.808+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:29.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:29.100+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:29.100454+0000) 2022-04-03T21:31:29.590 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:29.588+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:29.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:29.830+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:30.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:30.100+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:30.100689+0000) 2022-04-03T21:31:30.356 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:31:30.354+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:31:30.559 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:30.557+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:30.880 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:30.878+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:31.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:31.100+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:31.100886+0000) 2022-04-03T21:31:31.599 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:31.598+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:31.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:31.832+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:32.103 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:32.100+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:32.101066+0000) 2022-04-03T21:31:32.566 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:32.564+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:32.835 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:32.833+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:33.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:33.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:33.101249+0000) 2022-04-03T21:31:33.590 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:33.588+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:33.851 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:33.850+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:34.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:34.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:34.101450+0000) 2022-04-03T21:31:34.601 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:34.599+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:34.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:34.840+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:35.103 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:35.099+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:35.101686+0000) 2022-04-03T21:31:35.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:35.606+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:35.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:35.870+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:36.103 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:36.100+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:36.101915+0000) 2022-04-03T21:31:36.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:36.587+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:36.913 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:36.911+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:37.103 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:37.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:37.102079+0000) 2022-04-03T21:31:37.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:37.543+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:37.907 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:37.905+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:38.103 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:38.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:38.102286+0000) 2022-04-03T21:31:38.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:38.493+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:38.898 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:38.896+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:39.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:39.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:39.102522+0000) 2022-04-03T21:31:39.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:39.491+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:39.881 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:39.879+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:40.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:40.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:40.102803+0000) 2022-04-03T21:31:40.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:40.526+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:40.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:40.831+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:41.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:41.102+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:41.102990+0000) 2022-04-03T21:31:41.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:41.490+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:41.866 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:41.864+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:42.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:42.102+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:42.103238+0000) 2022-04-03T21:31:42.449 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:42.447+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:42.848 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:42.846+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:43.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:43.102+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:43.103470+0000) 2022-04-03T21:31:43.450 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:43.449+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:43.799 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:43.797+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:44.041 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:31:44.044 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:31:44.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:44.102+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:44.103672+0000) 2022-04-03T21:31:44.500 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:44.499+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:44.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:44.775+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:45.105 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:45.103+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:45.103887+0000) 2022-04-03T21:31:45.372 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:31:45.370+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:31:45.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:45.519+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:45.807 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:45.804+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:46.105 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:46.103+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:46.104039+0000) 2022-04-03T21:31:46.568 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:46.565+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:46.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:46.812+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:47.105 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:47.103+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:47.104279+0000) 2022-04-03T21:31:47.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:47.541+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:47.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:47.860+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:48.105 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:48.103+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:48.104457+0000) 2022-04-03T21:31:48.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:48.515+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:48.828 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:48.825+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:49.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:49.103+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:49.104679+0000) 2022-04-03T21:31:49.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:49.477+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:49.818 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:49.815+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:50.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:50.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:50.104902+0000) 2022-04-03T21:31:50.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:50.488+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:50.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:50.790+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:51.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:51.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:51.105119+0000) 2022-04-03T21:31:51.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:51.461+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:51.788 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:51.786+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:52.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:52.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:52.105310+0000) 2022-04-03T21:31:52.502 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:52.499+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:52.814 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:52.811+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:53.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:53.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:53.105514+0000) 2022-04-03T21:31:53.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:53.508+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:53.786 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:53.784+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:54.107 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:54.105+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:54.105745+0000) 2022-04-03T21:31:54.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:54.535+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:54.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:54.792+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:55.107 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:55.105+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:55.105936+0000) 2022-04-03T21:31:55.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:55.579+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:55.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:55.754+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:56.107 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:56.105+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:56.106161+0000) 2022-04-03T21:31:56.555 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:56.552+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:56.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:56.712+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:57.107 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:57.105+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:57.106387+0000) 2022-04-03T21:31:57.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:57.532+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:57.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:57.744+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:58.108 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:58.106+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:58.106575+0000) 2022-04-03T21:31:58.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:58.569+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:58.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:58.714+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:31:59.108 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:31:59.106+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:31:59.106737+0000) 2022-04-03T21:31:59.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:31:59.538+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:31:59.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:31:59.683+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:00.108 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:00.106+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:00.106910+0000) 2022-04-03T21:32:00.388 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:32:00.386+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:32:00.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:00.491+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:00.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:00.650+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:01.108 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:01.106+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:01.107089+0000) 2022-04-03T21:32:01.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:01.536+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:01.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:01.602+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:02.108 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:02.106+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:02.107261+0000) 2022-04-03T21:32:02.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:02.489+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:02.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:02.585+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:03.108 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:03.107+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:03.107454+0000) 2022-04-03T21:32:03.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:03.461+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:03.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:03.590+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:04.109 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:04.107+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:04.107675+0000) 2022-04-03T21:32:04.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:04.481+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:04.608 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:04.605+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:05.109 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:05.107+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:05.107893+0000) 2022-04-03T21:32:05.530 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:05.528+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:05.610 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:05.608+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:06.109 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:06.107+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:06.108107+0000) 2022-04-03T21:32:06.504 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:06.502+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:06.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:06.559+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:07.109 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:07.108+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:07.108315+0000) 2022-04-03T21:32:07.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:07.520+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:07.607 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:07.604+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:08.109 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:08.108+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:08.108553+0000) 2022-04-03T21:32:08.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:08.511+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:08.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:08.625+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:09.110 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:09.108+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:09.108793+0000) 2022-04-03T21:32:09.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:09.489+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:09.604 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:09.602+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:10.110 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:10.108+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:10.109003+0000) 2022-04-03T21:32:10.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:10.495+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:10.558 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:10.555+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:11.110 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:11.108+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:11.109205+0000) 2022-04-03T21:32:11.468 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:11.466+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:11.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:11.519+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:12.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:12.109+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:12.109420+0000) 2022-04-03T21:32:12.452 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:12.450+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:12.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:12.507+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:13.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:13.109+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:13.109638+0000) 2022-04-03T21:32:13.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:13.404+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:13.517 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:13.515+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:14.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:14.109+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:14.109857+0000) 2022-04-03T21:32:14.188 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:32:14.192 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:32:14.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:14.418+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:14.528 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:14.526+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:15.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:15.109+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:15.110093+0000) 2022-04-03T21:32:15.404 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:32:15.401+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:32:15.436 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:15.434+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:15.572 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:15.570+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:16.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:16.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:16.110295+0000) 2022-04-03T21:32:16.441 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:16.438+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:16.572 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:16.569+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:17.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:17.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:17.110453+0000) 2022-04-03T21:32:17.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:17.454+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:17.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:17.572+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:18.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:18.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:18.110664+0000) 2022-04-03T21:32:18.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:18.445+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:18.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:18.569+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:19.112 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:19.109+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:19.110874+0000) 2022-04-03T21:32:19.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:19.483+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:19.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:19.561+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:20.112 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:20.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:20.111073+0000) 2022-04-03T21:32:20.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:20.459+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:20.583 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:20.582+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:21.112 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:21.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:21.111284+0000) 2022-04-03T21:32:21.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:21.486+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:21.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:21.574+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:22.112 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:22.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:22.111493+0000) 2022-04-03T21:32:22.449 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:22.446+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:22.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:22.577+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:23.113 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:23.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:23.111658+0000) 2022-04-03T21:32:23.469 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:23.467+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:23.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:23.554+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:24.113 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:24.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:24.111839+0000) 2022-04-03T21:32:24.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:24.491+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:24.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:24.542+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:25.113 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:25.111+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:25.112051+0000) 2022-04-03T21:32:25.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:25.511+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:25.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:25.537+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:26.113 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:26.111+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:26.112219+0000) 2022-04-03T21:32:26.498 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:26.496+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:26.500 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:26.499+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:27.113 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:27.111+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:27.112423+0000) 2022-04-03T21:32:27.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:27.480+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:27.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:27.548+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:28.114 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:28.111+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:28.112649+0000) 2022-04-03T21:32:28.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:28.504+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:28.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:28.542+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:29.114 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:29.112+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:29.112867+0000) 2022-04-03T21:32:29.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:29.511+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:29.520 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:29.518+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:30.114 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:30.112+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:30.113105+0000) 2022-04-03T21:32:30.419 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:32:30.417+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:32:30.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:30.511+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:30.519 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:30.518+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:31.114 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:31.112+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:31.113276+0000) 2022-04-03T21:32:31.472 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:31.470+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:31.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:31.512+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:32.114 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:32.112+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:32.113483+0000) 2022-04-03T21:32:32.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:32.480+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:32.500 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:32.499+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:33.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:33.112+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:33.113642+0000) 2022-04-03T21:32:33.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:33.472+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:33.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:33.495+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:34.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:34.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:34.113850+0000) 2022-04-03T21:32:34.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:34.454+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:34.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:34.474+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:35.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:35.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:35.114073+0000) 2022-04-03T21:32:35.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:35.427+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:35.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:35.485+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:36.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:36.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:36.114311+0000) 2022-04-03T21:32:36.420 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:36.418+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:36.477 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:36.476+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:37.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:37.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:37.114505+0000) 2022-04-03T21:32:37.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:37.436+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:37.462 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:37.461+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:38.116 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:38.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:38.114679+0000) 2022-04-03T21:32:38.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:38.419+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:38.479 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:38.477+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:39.116 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:39.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:39.114866+0000) 2022-04-03T21:32:39.432 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:39.430+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:39.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:39.501+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:40.116 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:40.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:40.115081+0000) 2022-04-03T21:32:40.432 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:40.430+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:40.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:40.493+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:41.117 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:41.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:41.115344+0000) 2022-04-03T21:32:41.400 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:41.397+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:41.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:41.511+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:42.117 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:42.115+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:42.115591+0000) 2022-04-03T21:32:42.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:42.440+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:42.470 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:42.467+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:43.117 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:43.115+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:43.115750+0000) 2022-04-03T21:32:43.429 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:43.426+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:43.477 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:43.475+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:44.117 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:44.115+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:44.115960+0000) 2022-04-03T21:32:44.323 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:32:44.327 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:32:44.402 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:44.400+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:44.498 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:44.495+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:45.118 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:45.115+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:45.116154+0000) 2022-04-03T21:32:45.378 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:45.375+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:45.436 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:32:45.433+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:32:45.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:45.454+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:46.117 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:46.115+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:46.116360+0000) 2022-04-03T21:32:46.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:46.400+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:46.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:46.428+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:47.118 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:47.116+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:47.116586+0000) 2022-04-03T21:32:47.382 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:47.379+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:47.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:47.478+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:48.118 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:48.116+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:48.116825+0000) 2022-04-03T21:32:48.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:48.371+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:48.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:48.516+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:49.118 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:49.116+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:49.117049+0000) 2022-04-03T21:32:49.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:49.368+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:49.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:49.550+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:50.118 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:50.116+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:50.117239+0000) 2022-04-03T21:32:50.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:50.403+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:50.504 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:50.501+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:51.118 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:51.117+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:51.117476+0000) 2022-04-03T21:32:51.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:51.355+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:51.490 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:51.487+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:52.119 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:52.117+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:52.117688+0000) 2022-04-03T21:32:52.405 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:52.403+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:52.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:52.472+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:53.119 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:53.117+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:53.117908+0000) 2022-04-03T21:32:53.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:53.381+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:53.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:53.472+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:54.119 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:54.117+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:54.118128+0000) 2022-04-03T21:32:54.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:54.332+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:54.482 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:54.479+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:55.119 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:55.118+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:55.118348+0000) 2022-04-03T21:32:55.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:55.292+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:55.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:55.476+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:56.119 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:56.117+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:56.118595+0000) 2022-04-03T21:32:56.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:56.339+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:56.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:56.474+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:57.120 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:57.118+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:57.118797+0000) 2022-04-03T21:32:57.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:57.323+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:57.502 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:57.500+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:58.120 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:58.118+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:58.118963+0000) 2022-04-03T21:32:58.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:58.342+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:58.498 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:58.495+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:32:59.120 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:32:59.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:32:59.119177+0000) 2022-04-03T21:32:59.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:32:59.368+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:32:59.540 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:32:59.537+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:00.120 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:00.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:00.119402+0000) 2022-04-03T21:33:00.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:00.357+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:00.451 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:33:00.449+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:33:00.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:00.508+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:01.121 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:01.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:01.119620+0000) 2022-04-03T21:33:01.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:01.377+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:01.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:01.484+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:02.121 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:02.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:02.119857+0000) 2022-04-03T21:33:02.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:02.416+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:02.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:02.453+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:03.121 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:03.118+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:03.120066+0000) 2022-04-03T21:33:03.409 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:03.407+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:03.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:03.414+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:04.121 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:04.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:04.120251+0000) 2022-04-03T21:33:04.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:04.427+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:04.431 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:04.429+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:05.121 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:05.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:05.120426+0000) 2022-04-03T21:33:05.434 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:05.432+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:05.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:05.440+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:06.122 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:06.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:06.120640+0000) 2022-04-03T21:33:06.452 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:06.450+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:06.473 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:06.470+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:07.122 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:07.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:07.120834+0000) 2022-04-03T21:33:07.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:07.414+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:07.489 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:07.486+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:08.122 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:08.120+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:08.121025+0000) 2022-04-03T21:33:08.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:08.379+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:08.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:08.445+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:09.122 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:09.120+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:09.121224+0000) 2022-04-03T21:33:09.420 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:09.418+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:09.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:09.441+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:10.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:10.120+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:10.121449+0000) 2022-04-03T21:33:10.412 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:10.410+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:10.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:10.410+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:11.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:11.120+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:11.121679+0000) 2022-04-03T21:33:11.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:11.435+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:11.457 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:11.455+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:12.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:12.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:12.121880+0000) 2022-04-03T21:33:12.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:12.438+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:12.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:12.462+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:13.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:13.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:13.122048+0000) 2022-04-03T21:33:13.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:13.434+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:13.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:13.474+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:14.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:14.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:14.122223+0000) 2022-04-03T21:33:14.449 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:33:14.453 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:33:14.479 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:14.477+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:14.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:14.513+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:15.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:15.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:15.122410+0000) 2022-04-03T21:33:15.467 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:33:15.465+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:33:15.472 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:15.469+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:15.475 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:15.473+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:16.124 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:16.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:16.122649+0000) 2022-04-03T21:33:16.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:16.481+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:16.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:16.516+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:17.124 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:17.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:17.122864+0000) 2022-04-03T21:33:17.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:17.481+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:17.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:17.487+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:18.124 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:18.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:18.123041+0000) 2022-04-03T21:33:18.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:18.481+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:18.523 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:18.521+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:19.124 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:19.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:19.123196+0000) 2022-04-03T21:33:19.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:19.491+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:19.503 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:19.502+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:20.124 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:20.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:20.123429+0000) 2022-04-03T21:33:20.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:20.487+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:20.510 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:20.509+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:21.125 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:21.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:21.123648+0000) 2022-04-03T21:33:21.477 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:21.476+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:21.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:21.548+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:22.125 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:22.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:22.123872+0000) 2022-04-03T21:33:22.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:22.509+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:22.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:22.572+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:23.125 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:23.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:23.124079+0000) 2022-04-03T21:33:23.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:23.547+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:23.607 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:23.605+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:24.125 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:24.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:24.124303+0000) 2022-04-03T21:33:24.547 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:24.545+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:24.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:24.624+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:25.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:25.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:25.124543+0000) 2022-04-03T21:33:25.591 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:25.589+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:25.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:25.663+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:26.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:26.124+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:26.124760+0000) 2022-04-03T21:33:26.558 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:26.556+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:26.656 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:26.654+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:27.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:27.124+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:27.124998+0000) 2022-04-03T21:33:27.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:27.546+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:27.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:27.666+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:28.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:28.124+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:28.125151+0000) 2022-04-03T21:33:28.225 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-03T21:33:28.223+0000 7f2fd661a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T21:33:28.225 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-03T21:33:28.223+0000 7f2fd761c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T21:33:28.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:28.570+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:28.704 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:28.702+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:29.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:29.124+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:29.125350+0000) 2022-04-03T21:33:29.556 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:29.555+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:29.691 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:29.689+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:30.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:30.125+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:30.125563+0000) 2022-04-03T21:33:30.483 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:33:30.480+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:33:30.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:30.603+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:30.720 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:30.718+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:31.127 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:31.125+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:31.125724+0000) 2022-04-03T21:33:31.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:31.592+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:31.765 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:31.764+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:32.127 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:32.125+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:32.125900+0000) 2022-04-03T21:33:32.589 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:32.587+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:32.754 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:32.752+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:33.127 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:33.125+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:33.126112+0000) 2022-04-03T21:33:33.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:33.574+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:33.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:33.753+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:34.127 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:34.125+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:34.126316+0000) 2022-04-03T21:33:34.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:34.552+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:34.727 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:34.724+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:35.128 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:35.126+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:35.126470+0000) 2022-04-03T21:33:35.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:35.534+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:35.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:35.768+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:36.128 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:36.126+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:36.126682+0000) 2022-04-03T21:33:36.500 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:36.497+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:36.768 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:36.766+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:37.128 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:37.126+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:37.126851+0000) 2022-04-03T21:33:37.517 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:37.514+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:37.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:37.734+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:38.128 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:38.126+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:38.127061+0000) 2022-04-03T21:33:38.547 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:38.544+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:38.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:38.769+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:39.128 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:39.127+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:39.127283+0000) 2022-04-03T21:33:39.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:39.564+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:39.727 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:39.725+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:40.129 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:40.127+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:40.127503+0000) 2022-04-03T21:33:40.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:40.599+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:40.689 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:40.686+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:41.129 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:41.127+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:41.127722+0000) 2022-04-03T21:33:41.615 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:41.613+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:41.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:41.696+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:42.129 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:42.127+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:42.127934+0000) 2022-04-03T21:33:42.597 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:42.594+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:42.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:42.730+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:43.129 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:43.127+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:43.128149+0000) 2022-04-03T21:33:43.559 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:43.556+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:43.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:43.775+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:44.129 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:44.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:44.128356+0000) 2022-04-03T21:33:44.582 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:44.579+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:44.588 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:33:44.593 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:33:44.759 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:44.757+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:45.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:45.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:45.128560+0000) 2022-04-03T21:33:45.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:45.128+0000 7fd623a1a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T21:33:45.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:45.128+0000 7fd623a1a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T21:33:45.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:45.131+0000 7fd62421b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:45.131957+0000) 2022-04-03T21:33:45.499 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:33:45.496+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:33:45.573 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:45.571+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:45.763 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:45.760+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:46.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:46.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:46.128713+0000) 2022-04-03T21:33:46.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:46.547+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:46.772 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:46.770+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:47.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:47.127+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:47.128823+0000) 2022-04-03T21:33:47.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:47.590+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:47.816 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:47.814+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:48.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:48.127+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:48.128965+0000) 2022-04-03T21:33:48.586 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:48.584+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:48.849 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:48.842+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:49.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:49.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:49.129135+0000) 2022-04-03T21:33:49.634 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:49.632+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:49.854 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:49.852+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:50.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:50.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:50.129285+0000) 2022-04-03T21:33:50.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:50.591+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:50.850 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:50.848+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:51.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:51.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:51.129404+0000) 2022-04-03T21:33:51.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:51.610+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:51.875 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:51.872+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:52.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:52.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:52.129554+0000) 2022-04-03T21:33:52.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:52.569+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:52.838 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:52.836+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:53.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:53.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:53.129730+0000) 2022-04-03T21:33:53.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:53.575+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:53.867 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:53.864+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:54.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:54.129+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:54.129927+0000) 2022-04-03T21:33:54.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:54.614+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:54.857 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:54.855+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:55.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:55.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:55.130090+0000) 2022-04-03T21:33:55.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:55.601+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:55.848 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:55.846+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:56.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:56.129+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:56.130262+0000) 2022-04-03T21:33:56.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:56.603+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:56.898 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:56.896+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:57.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:57.129+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:57.130423+0000) 2022-04-03T21:33:57.637 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:57.634+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:57.941 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:57.938+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:58.132 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:58.129+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:58.130601+0000) 2022-04-03T21:33:58.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:58.625+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:58.985 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:58.983+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:33:59.132 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:33:59.130+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:33:59.130789+0000) 2022-04-03T21:33:59.652 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:33:59.650+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:33:59.938 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:33:59.936+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:00.132 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:00.130+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:00.130974+0000) 2022-04-03T21:34:00.514 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:34:00.512+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:34:00.678 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:00.676+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:00.961 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:00.958+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:01.132 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:01.130+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:01.131152+0000) 2022-04-03T21:34:01.672 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:01.669+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:02.000 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:01.997+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:02.132 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:02.130+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:02.131330+0000) 2022-04-03T21:34:02.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:02.653+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:03.005 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:03.003+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:03.132 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:03.130+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:03.131522+0000) 2022-04-03T21:34:03.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:03.669+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:04.042 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:04.040+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:04.132 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:04.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:04.131689+0000) 2022-04-03T21:34:04.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:04.655+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:05.029 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:05.027+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:05.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:05.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:05.131806+0000) 2022-04-03T21:34:05.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:05.645+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:05.988 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:05.985+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:06.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:06.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:06.131982+0000) 2022-04-03T21:34:06.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:06.639+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:07.007 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:07.004+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:07.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:07.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:07.132159+0000) 2022-04-03T21:34:07.637 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:07.634+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:07.970 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:07.968+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:08.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:08.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:08.132304+0000) 2022-04-03T21:34:08.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:08.638+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:09.011 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:09.008+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:09.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:09.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:09.132473+0000) 2022-04-03T21:34:09.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:09.655+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:10.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:10.035+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:10.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:10.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:10.132615+0000) 2022-04-03T21:34:10.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:10.661+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:11.071 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:11.069+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:11.134 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:11.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:11.132758+0000) 2022-04-03T21:34:11.709 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:11.707+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:12.113 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:12.111+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:12.134 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:12.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:12.132938+0000) 2022-04-03T21:34:12.736 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:12.734+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:13.107 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:13.106+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:13.134 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:13.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:13.133117+0000) 2022-04-03T21:34:13.720 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:13.719+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:14.102 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:14.100+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:14.134 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:14.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:14.133286+0000) 2022-04-03T21:34:14.711 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:34:14.715 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:34:14.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:14.728+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:15.072 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:15.070+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:15.134 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:15.133+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:15.133445+0000) 2022-04-03T21:34:15.530 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:34:15.528+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:34:15.706 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:15.704+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:16.063 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:16.061+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:16.134 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:16.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:16.133607+0000) 2022-04-03T21:34:16.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:16.722+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:17.096 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:17.094+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:17.135 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:17.133+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:17.133809+0000) 2022-04-03T21:34:17.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:17.729+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:18.093 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:18.092+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:18.135 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:18.133+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:18.133984+0000) 2022-04-03T21:34:18.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:18.758+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:19.135 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:19.133+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:19.134197+0000) 2022-04-03T21:34:19.142 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:19.140+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:19.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:19.711+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:20.126 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:20.125+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:20.135 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:20.134+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:20.134382+0000) 2022-04-03T21:34:20.686 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:20.684+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:21.135 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:21.134+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:21.134557+0000) 2022-04-03T21:34:21.147 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:21.146+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:21.674 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:21.672+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:22.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:22.134+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:22.134699+0000) 2022-04-03T21:34:22.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:22.164+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:22.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:22.678+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:23.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:23.134+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:23.134879+0000) 2022-04-03T21:34:23.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:23.160+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:23.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:23.689+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:24.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:24.134+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:24.135037+0000) 2022-04-03T21:34:24.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:24.205+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:24.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:24.709+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:25.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:25.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:25.135177+0000) 2022-04-03T21:34:25.176 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:25.174+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:25.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:25.721+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:26.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:26.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:26.135328+0000) 2022-04-03T21:34:26.176 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:26.174+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:26.707 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:26.705+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:27.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:27.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:27.135506+0000) 2022-04-03T21:34:27.195 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:27.193+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:27.718 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:27.716+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:28.137 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:28.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:28.135684+0000) 2022-04-03T21:34:28.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:28.232+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:28.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:28.713+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:29.137 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:29.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:29.135860+0000) 2022-04-03T21:34:29.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:29.266+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:29.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:29.687+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:30.137 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:30.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:30.136033+0000) 2022-04-03T21:34:30.226 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:30.223+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:30.546 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:34:30.544+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:34:30.650 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:30.647+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:31.137 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:31.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:31.136206+0000) 2022-04-03T21:34:31.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:31.178+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:31.683 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:31.681+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:32.137 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:32.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:32.136356+0000) 2022-04-03T21:34:32.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:32.154+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:32.654 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:32.651+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:33.138 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:33.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:33.136543+0000) 2022-04-03T21:34:33.145 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:33.143+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:33.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:33.677+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:34.138 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:34.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:34.136724+0000) 2022-04-03T21:34:34.172 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:34.170+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:34.689 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:34.686+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:35.135 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:35.133+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:35.138 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:35.136+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:35.136945+0000) 2022-04-03T21:34:35.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:35.714+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:36.108 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:36.105+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:36.138 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:36.136+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:36.137131+0000) 2022-04-03T21:34:36.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:36.742+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:37.097 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:37.094+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:37.138 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:37.136+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:37.137317+0000) 2022-04-03T21:34:37.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:37.737+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:38.100 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:38.098+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:38.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:38.136+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:38.137499+0000) 2022-04-03T21:34:38.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:38.720+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:39.111 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:39.109+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:39.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:39.136+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:39.137656+0000) 2022-04-03T21:34:39.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:39.748+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:40.100 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:40.097+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:40.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:40.137+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:40.137828+0000) 2022-04-03T21:34:40.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:40.728+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:41.133 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:41.131+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:41.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:41.137+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:41.137982+0000) 2022-04-03T21:34:41.697 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:41.694+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:42.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:42.135+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:42.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:42.137+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:42.138180+0000) 2022-04-03T21:34:42.721 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:42.718+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:43.108 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:43.106+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:43.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:43.137+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:43.138374+0000) 2022-04-03T21:34:43.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:43.691+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:44.083 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:44.080+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:44.140 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:44.137+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:44.138561+0000) 2022-04-03T21:34:44.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:44.711+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:44.909 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:34:44.913 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:34:45.093 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:45.091+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:45.140 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:45.137+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:45.138695+0000) 2022-04-03T21:34:45.562 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:34:45.559+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:34:45.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:45.740+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:46.094 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:46.091+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:46.140 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:46.138+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:46.138877+0000) 2022-04-03T21:34:46.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:46.711+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:47.064 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:47.062+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:47.140 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:47.138+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:47.139059+0000) 2022-04-03T21:34:47.666 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:47.664+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:48.037 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:48.035+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:48.140 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:48.138+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:48.139232+0000) 2022-04-03T21:34:48.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:48.654+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:49.048 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:49.046+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:49.140 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:49.138+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:49.139410+0000) 2022-04-03T21:34:49.639 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:49.636+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:50.027 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:50.025+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:50.141 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:50.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:50.139610+0000) 2022-04-03T21:34:50.604 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:50.602+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:50.982 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:50.979+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:51.141 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:51.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:51.139795+0000) 2022-04-03T21:34:51.651 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:51.648+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:51.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:51.931+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:52.141 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:52.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:52.139990+0000) 2022-04-03T21:34:52.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:52.645+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:52.966 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:52.964+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:53.141 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:53.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:53.140128+0000) 2022-04-03T21:34:53.686 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:53.684+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:53.962 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:53.959+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:54.141 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:54.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:54.140277+0000) 2022-04-03T21:34:54.637 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:54.635+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:55.007 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:55.005+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:55.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:55.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:55.140453+0000) 2022-04-03T21:34:55.599 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:55.596+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:56.048 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:56.046+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:56.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:56.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:56.140628+0000) 2022-04-03T21:34:56.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:56.574+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:57.035 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:57.033+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:57.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:57.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:57.140821+0000) 2022-04-03T21:34:57.527 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:57.525+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:58.026 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:58.024+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:58.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:58.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:58.140961+0000) 2022-04-03T21:34:58.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:58.573+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:34:59.024 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:34:59.022+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:34:59.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:34:59.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:34:59.141113+0000) 2022-04-03T21:34:59.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:34:59.582+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:00.025 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:00.023+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:00.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:00.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:00.141288+0000) 2022-04-03T21:35:00.577 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:35:00.575+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:35:00.590 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:00.588+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:01.033 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:01.031+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:01.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:01.141+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:01.141472+0000) 2022-04-03T21:35:01.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:01.540+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:02.057 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:02.055+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:02.143 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:02.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:02.141669+0000) 2022-04-03T21:35:02.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:02.569+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:03.095 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:03.093+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:03.143 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:03.141+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:03.141854+0000) 2022-04-03T21:35:03.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:03.600+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:04.130 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:04.128+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:04.143 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:04.141+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:04.142036+0000) 2022-04-03T21:35:04.565 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:04.563+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:05.130 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:05.128+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:05.143 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:05.141+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:05.142234+0000) 2022-04-03T21:35:05.556 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:05.554+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:06.113 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:06.111+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:06.143 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:06.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:06.142413+0000) 2022-04-03T21:35:06.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:06.551+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:07.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:07.104+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:07.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:07.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:07.142602+0000) 2022-04-03T21:35:07.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:07.543+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:08.070 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:08.068+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:08.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:08.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:08.142806+0000) 2022-04-03T21:35:08.578 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:08.577+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:09.039 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:09.037+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:09.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:09.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:09.142986+0000) 2022-04-03T21:35:09.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:09.550+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:10.057 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:10.055+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:10.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:10.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:10.143160+0000) 2022-04-03T21:35:10.527 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:10.525+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:11.098 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:11.096+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:11.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:11.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:11.143312+0000) 2022-04-03T21:35:11.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:11.477+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:12.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:12.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:12.143448+0000) 2022-04-03T21:35:12.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:12.146+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:12.519 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:12.518+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:13.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:13.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:13.143592+0000) 2022-04-03T21:35:13.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:13.163+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:13.558 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:13.556+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:14.143 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:14.142+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:14.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:14.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:14.143748+0000) 2022-04-03T21:35:14.575 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:14.573+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:14.999 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:35:15.003 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:35:15.135 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:15.133+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:15.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:15.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:15.143937+0000) 2022-04-03T21:35:15.593 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:35:15.590+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:35:15.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:15.592+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:16.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:16.110+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:16.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:16.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:16.144106+0000) 2022-04-03T21:35:16.631 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:16.629+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:17.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:17.072+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:17.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:17.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:17.144265+0000) 2022-04-03T21:35:17.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:17.655+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:18.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:18.104+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:18.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:18.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:18.144457+0000) 2022-04-03T21:35:18.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:18.696+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:19.094 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:19.092+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:19.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:19.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:19.144651+0000) 2022-04-03T21:35:19.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:19.711+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:20.065 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:20.063+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:20.146 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:20.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:20.144842+0000) 2022-04-03T21:35:20.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:20.711+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:21.062 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:21.060+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:21.146 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:21.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:21.145023+0000) 2022-04-03T21:35:21.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:21.730+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:22.074 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:22.073+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:22.146 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:22.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:22.145208+0000) 2022-04-03T21:35:22.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:22.776+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:23.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:23.083+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:23.146 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:23.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:23.145389+0000) 2022-04-03T21:35:23.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:23.782+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:24.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:24.113+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:24.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:24.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:24.145599+0000) 2022-04-03T21:35:24.797 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:24.795+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:25.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:25.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:25.145783+0000) 2022-04-03T21:35:25.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:25.162+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:25.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:25.833+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:26.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:26.145+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:26.145982+0000) 2022-04-03T21:35:26.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:26.184+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:26.844 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:26.841+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:27.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:27.145+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:27.146155+0000) 2022-04-03T21:35:27.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:27.153+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:27.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:27.849+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:28.143 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:28.140+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:28.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:28.145+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:28.146260+0000) 2022-04-03T21:35:28.841 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:28.839+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:29.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:29.145+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:29.146397+0000) 2022-04-03T21:35:29.151 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:29.148+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:29.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:29.829+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:30.102 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:30.099+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:30.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:30.145+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:30.146544+0000) 2022-04-03T21:35:30.599 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:35:30.597+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:35:30.809 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:30.807+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:31.132 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:31.130+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:31.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:31.146+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:31.146696+0000) 2022-04-03T21:35:31.765 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:31.762+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:32.124 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:32.121+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:32.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:32.146+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:32.146854+0000) 2022-04-03T21:35:32.719 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:32.716+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:33.140 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:33.137+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:33.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:33.146+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:33.147025+0000) 2022-04-03T21:35:33.684 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:33.682+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:34.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:34.146+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:34.147182+0000) 2022-04-03T21:35:34.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:34.178+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:34.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:34.666+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:35.142 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:35.140+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:35.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:35.146+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:35.147357+0000) 2022-04-03T21:35:35.651 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:35.648+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:36.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:36.146+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:36.147515+0000) 2022-04-03T21:35:36.172 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:36.170+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:36.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:36.668+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:37.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:37.146+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:37.147649+0000) 2022-04-03T21:35:37.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:37.200+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:37.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:37.715+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:38.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:38.147+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:38.147846+0000) 2022-04-03T21:35:38.171 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:38.168+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:38.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:38.738+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:39.131 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:39.128+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:39.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:39.147+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:39.147992+0000) 2022-04-03T21:35:39.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:39.693+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:40.135 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:40.132+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:40.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:40.147+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:40.148144+0000) 2022-04-03T21:35:40.736 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:40.734+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:41.135 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:41.132+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:41.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:41.147+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:41.148327+0000) 2022-04-03T21:35:41.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:41.729+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:42.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:42.148+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:42.148462+0000) 2022-04-03T21:35:42.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:42.159+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:42.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:42.710+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:43.144 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:43.142+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:43.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:43.148+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:43.148659+0000) 2022-04-03T21:35:43.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:43.740+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:44.117 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:44.115+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:44.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:44.148+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:44.148815+0000) 2022-04-03T21:35:44.721 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:44.719+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:45.097 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:35:45.101 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:35:45.130 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:45.128+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:45.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:45.148+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:45.148950+0000) 2022-04-03T21:35:45.615 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:35:45.613+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:35:45.736 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:45.734+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:46.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:46.148+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:46.149114+0000) 2022-04-03T21:35:46.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:46.161+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:46.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:46.770+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:47.127 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:47.124+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:47.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:47.148+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:47.149259+0000) 2022-04-03T21:35:47.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:47.819+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:48.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:48.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:48.149374+0000) 2022-04-03T21:35:48.152 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:48.149+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:48.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:48.787+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:49.128 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:49.126+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:49.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:49.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:49.149497+0000) 2022-04-03T21:35:49.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:49.741+0000 7fd3b190b700 -1 osd.6 124 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:50.122 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:50.119+0000 7f88d0ac1700 -1 osd.4 124 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:50.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:50.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:50.149667+0000) 2022-04-03T21:35:50.279 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-03T21:35:50.277+0000 7f0f79ad2700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T21:35:50.295 INFO:tasks.ceph.mds.f.smithi186.stderr: -1> 2022-04-03T21:35:50.277+0000 7f0f79ad2700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T21:35:50.316 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-03T21:35:50.736 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:50.733+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:51.079 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:51.077+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:51.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:51.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:51.149800+0000) 2022-04-03T21:35:51.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:51.746+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:52.111 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:52.108+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:52.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:52.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:52.149953+0000) 2022-04-03T21:35:52.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:52.740+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:53.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:53.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:53.150111+0000) 2022-04-03T21:35:53.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:53.155+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:53.724 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:53.722+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:54.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:54.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:54.150227+0000) 2022-04-03T21:35:54.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:54.173+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:54.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:54.710+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:55.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:55.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:55.150360+0000) 2022-04-03T21:35:55.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:55.174+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 20 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:55.696 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:55.694+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:56.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:56.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:56.150536+0000) 2022-04-03T21:35:56.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:56.161+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:56.708 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:56.706+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:57.152 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:57.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:57.150709+0000) 2022-04-03T21:35:57.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:57.191+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:57.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:57.738+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:58.152 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:58.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:58.150835+0000) 2022-04-03T21:35:58.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:58.235+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:58.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:58.727+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:35:59.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:35:59.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:35:59.150989+0000) 2022-04-03T21:35:59.233 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:35:59.230+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:35:59.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:35:59.726+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:00.152 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:00.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:00.151149+0000) 2022-04-03T21:36:00.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:00.254+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:00.631 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:36:00.627+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:36:00.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:00.711+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:01.152 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:01.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:01.151308+0000) 2022-04-03T21:36:01.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:01.270+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:01.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:01.759+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:02.152 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:02.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:02.151475+0000) 2022-04-03T21:36:02.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:02.270+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:02.787 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:02.785+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:03.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:03.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:03.151602+0000) 2022-04-03T21:36:03.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:03.309+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:03.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:03.830+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:04.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:04.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:04.151787+0000) 2022-04-03T21:36:04.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:04.295+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:04.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:04.835+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:05.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:05.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:05.151976+0000) 2022-04-03T21:36:05.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:05.328+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:05.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:05.828+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:06.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:06.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:06.152134+0000) 2022-04-03T21:36:06.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:06.351+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:06.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:06.848+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:07.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:07.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:07.152252+0000) 2022-04-03T21:36:07.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:07.313+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:07.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:07.839+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:08.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:08.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:08.152371+0000) 2022-04-03T21:36:08.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:08.331+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:08.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:08.830+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:09.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:09.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:09.152504+0000) 2022-04-03T21:36:09.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:09.342+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:09.871 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:09.870+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:10.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:10.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:10.152700+0000) 2022-04-03T21:36:10.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:10.320+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:10.866 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:10.865+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:11.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:11.152+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:11.152870+0000) 2022-04-03T21:36:11.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:11.289+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:11.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:11.859+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:12.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:12.152+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:12.153029+0000) 2022-04-03T21:36:12.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:12.337+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:12.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:12.866+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:13.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:13.152+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:13.153185+0000) 2022-04-03T21:36:13.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:13.320+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:13.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:13.822+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:14.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:14.152+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:14.153382+0000) 2022-04-03T21:36:14.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:14.287+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:14.804 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:14.803+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:15.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:15.152+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:15.153539+0000) 2022-04-03T21:36:15.228 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:36:15.232 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:36:15.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:15.244+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:15.646 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:36:15.644+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:36:15.854 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:15.852+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:16.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:16.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:16.153718+0000) 2022-04-03T21:36:16.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:16.217+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:16.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:16.819+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:17.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:17.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:17.153859+0000) 2022-04-03T21:36:17.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:17.184+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:17.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:17.820+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:18.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:18.134+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:18.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:18.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:18.154016+0000) 2022-04-03T21:36:18.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:18.841+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:19.099 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:19.097+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:19.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:19.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:19.154145+0000) 2022-04-03T21:36:19.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:19.794+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:20.071 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:20.068+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:20.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:20.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:20.154326+0000) 2022-04-03T21:36:20.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:20.804+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:21.087 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:21.084+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:21.156 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:21.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:21.154483+0000) 2022-04-03T21:36:21.768 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:21.765+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:22.102 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:22.099+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:22.156 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:22.154+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:22.154660+0000) 2022-04-03T21:36:22.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:22.790+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:23.098 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:23.095+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:23.156 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:23.154+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:23.154826+0000) 2022-04-03T21:36:23.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:23.741+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:24.099 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:24.096+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:24.156 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:24.154+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:24.154989+0000) 2022-04-03T21:36:24.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:24.720+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:25.140 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:25.138+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:25.156 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:25.154+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:25.155155+0000) 2022-04-03T21:36:25.696 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:25.693+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:26.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:26.154+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:26.155369+0000) 2022-04-03T21:36:26.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:26.158+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:26.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:26.660+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:27.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:27.153+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:27.156 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:27.154+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:27.155574+0000) 2022-04-03T21:36:27.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:27.636+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:28.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:28.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:28.155752+0000) 2022-04-03T21:36:28.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:28.178+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:28.606 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:28.604+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:29.147 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:29.145+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:29.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:29.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:29.155957+0000) 2022-04-03T21:36:29.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:29.600+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:30.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:30.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:30.156133+0000) 2022-04-03T21:36:30.160 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:30.157+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:30.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:30.579+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:30.652 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:36:30.650+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:36:31.127 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:31.124+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:31.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:31.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:31.156307+0000) 2022-04-03T21:36:31.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:31.544+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:32.100 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:32.097+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:32.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:32.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:32.156467+0000) 2022-04-03T21:36:32.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:32.498+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:33.051 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:33.049+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:33.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:33.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:33.156633+0000) 2022-04-03T21:36:33.533 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:33.531+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:34.042 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:34.039+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:34.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:34.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:34.156806+0000) 2022-04-03T21:36:34.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:34.486+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:35.076 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:35.073+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:35.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:35.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:35.156969+0000) 2022-04-03T21:36:35.496 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:35.493+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:36.041 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:36.039+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:36.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:36.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:36.157132+0000) 2022-04-03T21:36:36.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:36.529+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:37.057 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:37.054+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:37.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:37.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:37.157294+0000) 2022-04-03T21:36:37.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:37.551+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:38.095 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:38.092+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:38.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:38.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:38.157480+0000) 2022-04-03T21:36:38.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:38.530+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:39.052 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:39.050+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:39.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:39.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:39.157671+0000) 2022-04-03T21:36:39.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:39.544+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:40.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:40.065+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:40.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:40.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:40.157847+0000) 2022-04-03T21:36:40.517 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:40.515+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:41.114 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:41.112+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:41.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:41.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:41.158031+0000) 2022-04-03T21:36:41.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:41.551+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:42.098 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:42.096+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:42.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:42.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:42.158167+0000) 2022-04-03T21:36:42.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:42.509+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:43.116 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:43.113+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:43.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:43.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:43.158328+0000) 2022-04-03T21:36:43.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:43.507+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:44.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:44.145+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:44.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:44.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:44.158470+0000) 2022-04-03T21:36:44.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:44.551+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:45.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:45.110+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:45.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:45.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:45.158653+0000) 2022-04-03T21:36:45.400 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:36:45.404 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:36:45.520 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:45.518+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:45.668 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:36:45.664+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:36:46.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:46.079+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:46.160 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:46.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:46.158876+0000) 2022-04-03T21:36:46.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:46.474+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:47.109 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:47.106+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:47.160 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:47.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:47.159062+0000) 2022-04-03T21:36:47.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:47.486+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:48.133 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:48.131+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:48.160 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:48.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:48.159279+0000) 2022-04-03T21:36:48.496 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:48.494+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:49.151 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:49.149+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:49.160 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:49.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:49.159491+0000) 2022-04-03T21:36:49.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:49.540+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:50.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:50.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:50.159690+0000) 2022-04-03T21:36:50.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:50.178+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:50.558 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:50.555+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:51.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:51.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:51.159905+0000) 2022-04-03T21:36:51.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:51.165+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:51.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:51.603+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:52.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:52.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:52.160116+0000) 2022-04-03T21:36:52.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:52.198+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:52.596 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:52.593+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:53.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:53.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:53.160324+0000) 2022-04-03T21:36:53.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:53.161+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:53.574 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:53.571+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:54.129 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:54.127+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:54.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:54.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:54.160488+0000) 2022-04-03T21:36:54.555 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:54.553+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:55.162 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:55.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:55.160693+0000) 2022-04-03T21:36:55.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:55.171+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:55.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:55.590+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:56.140 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:56.138+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:56.162 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:56.160+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:56.160912+0000) 2022-04-03T21:36:56.634 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:56.633+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:57.162 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:57.160+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:57.161113+0000) 2022-04-03T21:36:57.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:57.187+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:57.630 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:57.629+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:58.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:58.151+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:58.162 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:58.160+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:58.161283+0000) 2022-04-03T21:36:58.649 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:58.647+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:36:59.162 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:36:59.160+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:36:59.161486+0000) 2022-04-03T21:36:59.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:36:59.196+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:36:59.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:36:59.643+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:00.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:00.160+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:00.161686+0000) 2022-04-03T21:37:00.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:00.177+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:00.617 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:00.615+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:00.683 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:37:00.681+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:37:01.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:01.161+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:01.161842+0000) 2022-04-03T21:37:01.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:01.167+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:01.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:01.604+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:02.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:02.161+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:02.161961+0000) 2022-04-03T21:37:02.202 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:02.201+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:02.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:02.618+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:03.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:03.161+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:03.162153+0000) 2022-04-03T21:37:03.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:03.240+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:03.608 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:03.606+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:04.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:04.161+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:04.162340+0000) 2022-04-03T21:37:04.220 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:04.219+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:04.597 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:04.595+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:05.164 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:05.161+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:05.162541+0000) 2022-04-03T21:37:05.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:05.260+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:05.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:05.621+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:06.164 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:06.162+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:06.162734+0000) 2022-04-03T21:37:06.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:06.251+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:06.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:06.640+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:07.164 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:07.162+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:07.162936+0000) 2022-04-03T21:37:07.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:07.209+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:07.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:07.678+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:08.164 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:08.162+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:08.163135+0000) 2022-04-03T21:37:08.195 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:08.193+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:08.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:08.693+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:09.164 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:09.162+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:09.163293+0000) 2022-04-03T21:37:09.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:09.228+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:09.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:09.693+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:10.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:10.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:10.163506+0000) 2022-04-03T21:37:10.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:10.252+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:10.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:10.654+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:11.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:11.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:11.163702+0000) 2022-04-03T21:37:11.235 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:11.233+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:11.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:11.670+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:12.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:12.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:12.163903+0000) 2022-04-03T21:37:12.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:12.278+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:12.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:12.697+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:13.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:13.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:13.164054+0000) 2022-04-03T21:37:13.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:13.246+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:13.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:13.689+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:14.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:14.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:14.164219+0000) 2022-04-03T21:37:14.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:14.206+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:14.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:14.707+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:15.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:15.164+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:15.164405+0000) 2022-04-03T21:37:15.233 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:15.230+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:15.556 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:37:15.560 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:37:15.699 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:37:15.697+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:37:15.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:15.697+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:16.166 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:16.164+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:16.164641+0000) 2022-04-03T21:37:16.226 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:16.223+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:16.719 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:16.716+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:17.166 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:17.164+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:17.164865+0000) 2022-04-03T21:37:17.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:17.266+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:17.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:17.707+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:18.166 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:18.164+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:18.165077+0000) 2022-04-03T21:37:18.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:18.248+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:18.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:18.692+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:19.166 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:19.164+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:19.165273+0000) 2022-04-03T21:37:19.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:19.279+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:19.738 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:19.736+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:20.167 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:20.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:20.165496+0000) 2022-04-03T21:37:20.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:20.289+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:20.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:20.778+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:21.167 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:21.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:21.165698+0000) 2022-04-03T21:37:21.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:21.331+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:21.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:21.779+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:22.167 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:22.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:22.165901+0000) 2022-04-03T21:37:22.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:22.327+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:22.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:22.750+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:23.167 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:23.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:23.166100+0000) 2022-04-03T21:37:23.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:23.318+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:23.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:23.785+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:24.167 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:24.166+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:24.166282+0000) 2022-04-03T21:37:24.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:24.273+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:24.740 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:24.738+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:25.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:25.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:25.166502+0000) 2022-04-03T21:37:25.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:25.292+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:25.704 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:25.701+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:26.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:26.166+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:26.166705+0000) 2022-04-03T21:37:26.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:26.315+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:26.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:26.707+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:27.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:27.166+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:27.166907+0000) 2022-04-03T21:37:27.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:27.358+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:27.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:27.693+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:28.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:28.166+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:28.167124+0000) 2022-04-03T21:37:28.356 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:28.353+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:28.685 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:28.683+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:29.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:29.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:29.167306+0000) 2022-04-03T21:37:29.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:29.323+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:29.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:29.638+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:30.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:30.166+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:30.167473+0000) 2022-04-03T21:37:30.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:30.308+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:30.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:30.612+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:30.715 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:37:30.712+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:37:31.169 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:31.166+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:31.167691+0000) 2022-04-03T21:37:31.378 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:31.299+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:31.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:31.609+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:32.169 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:32.166+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:32.167909+0000) 2022-04-03T21:37:32.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:32.255+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:32.599 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:32.596+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:33.169 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:33.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:33.168142+0000) 2022-04-03T21:37:33.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:33.237+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:33.601 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:33.599+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:34.169 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:34.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:34.168355+0000) 2022-04-03T21:37:34.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:34.206+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:34.631 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:34.628+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:35.170 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:35.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:35.168555+0000) 2022-04-03T21:37:35.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:35.245+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:35.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:35.661+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:36.170 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:36.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:36.168801+0000) 2022-04-03T21:37:36.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:36.256+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:36.659 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:36.651+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:37.170 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:37.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:37.169008+0000) 2022-04-03T21:37:37.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:37.273+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:37.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:37.612+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:38.170 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:38.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:38.169206+0000) 2022-04-03T21:37:38.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:38.262+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:38.618 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:38.616+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:39.170 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:39.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:39.169372+0000) 2022-04-03T21:37:39.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:39.241+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:39.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:39.590+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:40.171 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:40.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:40.169587+0000) 2022-04-03T21:37:40.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:40.248+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:40.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:40.621+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:41.171 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:41.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:41.169815+0000) 2022-04-03T21:37:41.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:41.245+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:41.597 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:41.594+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:42.171 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:42.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:42.170013+0000) 2022-04-03T21:37:42.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:42.283+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:42.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:42.636+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:43.171 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:43.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:43.170260+0000) 2022-04-03T21:37:43.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:43.254+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:43.645 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:43.643+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:44.172 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:44.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:44.170467+0000) 2022-04-03T21:37:44.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:44.274+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:44.624 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:44.622+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:45.172 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:45.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:45.170619+0000) 2022-04-03T21:37:45.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:45.309+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:45.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:45.653+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:45.729 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:37:45.731 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:37:45.729+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:37:45.732 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:37:46.172 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:46.170+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:46.170893+0000) 2022-04-03T21:37:46.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:46.283+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:46.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:46.696+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:47.172 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:47.170+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:47.171056+0000) 2022-04-03T21:37:47.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:47.273+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:47.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:47.696+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:48.172 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:48.170+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:48.171257+0000) 2022-04-03T21:37:48.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:48.256+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:48.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:48.740+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:49.172 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:49.170+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:49.171452+0000) 2022-04-03T21:37:49.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:49.244+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:49.786 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:49.784+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:50.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:50.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:50.171661+0000) 2022-04-03T21:37:50.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:50.240+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:50.736 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:50.734+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:51.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:51.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:51.171857+0000) 2022-04-03T21:37:51.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:51.254+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:51.746 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:51.744+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:52.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:52.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:52.171999+0000) 2022-04-03T21:37:52.233 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:52.232+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:52.777 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:52.775+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:53.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:53.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:53.172236+0000) 2022-04-03T21:37:53.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:53.279+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:53.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:53.743+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:54.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:54.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:54.172430+0000) 2022-04-03T21:37:54.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:54.244+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:54.703 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:54.701+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:55.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:55.172+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:55.172629+0000) 2022-04-03T21:37:55.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:55.208+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:55.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:55.663+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:56.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:56.172+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:56.172864+0000) 2022-04-03T21:37:56.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:56.222+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:56.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:56.653+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:57.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:57.172+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:57.173064+0000) 2022-04-03T21:37:57.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:57.256+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:57.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:57.626+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:58.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:58.172+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:58.173231+0000) 2022-04-03T21:37:58.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:58.256+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:58.675 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:58.674+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:37:59.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:37:59.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:37:59.173400+0000) 2022-04-03T21:37:59.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:37:59.280+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:37:59.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:37:59.630+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:00.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:00.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:00.173605+0000) 2022-04-03T21:38:00.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:00.285+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:00.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:00.645+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:00.747 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:38:00.744+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:38:01.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:01.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:01.173832+0000) 2022-04-03T21:38:01.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:01.307+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:01.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:01.667+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:02.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:02.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:02.173960+0000) 2022-04-03T21:38:02.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:02.338+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:02.701 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:02.699+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:03.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:03.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:03.174184+0000) 2022-04-03T21:38:03.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:03.348+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:03.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:03.710+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:04.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:04.174+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:04.174402+0000) 2022-04-03T21:38:04.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:04.387+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:04.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:04.759+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:05.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:05.174+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:05.174619+0000) 2022-04-03T21:38:05.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:05.417+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:05.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:05.759+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:06.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:06.174+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:06.174815+0000) 2022-04-03T21:38:06.468 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:06.467+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:06.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:06.780+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:07.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:07.174+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:07.174992+0000) 2022-04-03T21:38:07.437 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:07.434+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:07.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:07.828+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:08.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:08.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:08.175203+0000) 2022-04-03T21:38:08.418 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:08.415+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:08.874 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:08.872+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:09.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:09.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:09.175408+0000) 2022-04-03T21:38:09.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:09.393+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:09.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:09.861+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:10.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:10.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:10.175584+0000) 2022-04-03T21:38:10.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:10.378+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:10.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:10.871+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:11.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:11.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:11.175803+0000) 2022-04-03T21:38:11.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:11.400+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:11.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:11.897+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:12.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:12.174+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:12.176022+0000) 2022-04-03T21:38:12.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:12.447+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:12.907 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:12.905+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:13.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:13.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:13.176221+0000) 2022-04-03T21:38:13.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:13.424+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:13.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:13.914+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:14.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:14.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:14.176430+0000) 2022-04-03T21:38:14.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:14.392+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:14.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:14.865+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:15.178 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:15.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:15.176638+0000) 2022-04-03T21:38:15.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:15.405+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:15.762 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:38:15.759+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:38:15.843 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:38:15.847 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:38:15.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:15.879+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:16.178 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:16.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:16.176896+0000) 2022-04-03T21:38:16.413 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:16.410+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:16.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:16.885+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:17.178 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:17.176+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:17.177143+0000) 2022-04-03T21:38:17.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:17.400+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:17.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:17.875+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:18.178 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:18.176+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:18.177346+0000) 2022-04-03T21:38:18.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:18.370+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:18.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:18.900+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:19.179 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:19.176+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:19.177532+0000) 2022-04-03T21:38:19.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:19.355+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:19.869 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:19.866+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:20.179 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:20.176+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:20.177746+0000) 2022-04-03T21:38:20.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:20.318+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:20.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:20.894+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:21.179 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:21.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:21.177917+0000) 2022-04-03T21:38:21.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:21.344+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:21.850 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:21.848+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:22.179 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:22.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:22.178123+0000) 2022-04-03T21:38:22.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:22.389+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:22.871 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:22.869+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:23.179 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:23.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:23.178326+0000) 2022-04-03T21:38:23.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:23.395+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:23.841 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:23.839+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:24.179 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:24.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:24.178541+0000) 2022-04-03T21:38:24.423 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:24.421+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:24.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:24.887+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:25.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:25.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:25.178751+0000) 2022-04-03T21:38:25.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:25.433+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:25.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:25.850+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:26.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:26.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:26.178949+0000) 2022-04-03T21:38:26.454 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:26.452+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:26.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:26.885+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:27.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:27.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:27.179123+0000) 2022-04-03T21:38:27.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:27.447+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:27.856 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:27.853+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:28.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:28.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:28.179274+0000) 2022-04-03T21:38:28.471 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:28.468+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:28.844 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:28.842+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:29.181 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:29.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:29.179443+0000) 2022-04-03T21:38:29.453 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:29.451+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:29.844 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:29.842+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:30.181 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:30.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:30.179662+0000) 2022-04-03T21:38:30.472 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:30.470+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:30.766 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:38:30.764+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:38:30.871 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:30.868+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:31.181 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:31.179+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:31.179867+0000) 2022-04-03T21:38:31.517 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:31.514+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:31.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:31.876+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:32.181 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:32.179+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:32.180083+0000) 2022-04-03T21:38:32.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:32.551+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:32.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:32.865+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:33.181 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:33.179+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:33.180301+0000) 2022-04-03T21:38:33.602 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:33.600+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:33.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:33.820+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:34.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:34.179+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:34.180507+0000) 2022-04-03T21:38:34.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:34.638+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:34.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:34.785+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:35.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:35.180+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:35.180714+0000) 2022-04-03T21:38:35.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:35.654+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:35.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:35.794+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:36.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:36.180+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:36.180887+0000) 2022-04-03T21:38:36.689 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:36.687+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:36.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:36.836+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:37.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:37.180+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:37.181098+0000) 2022-04-03T21:38:37.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:37.680+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:37.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:37.840+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:38.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:38.180+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:38.181324+0000) 2022-04-03T21:38:38.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:38.714+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:38.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:38.812+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:39.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:39.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:39.181531+0000) 2022-04-03T21:38:39.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:39.730+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:39.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:39.782+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:38:40.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:40.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:40.181760+0000) 2022-04-03T21:38:40.735 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:40.733+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:40.771 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:40.769+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:38:41.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:41.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:41.181988+0000) 2022-04-03T21:38:41.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:41.697+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:41.768 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:41.765+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:38:42.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:42.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:42.182159+0000) 2022-04-03T21:38:42.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:42.735+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:42.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:42.808+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:38:43.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:43.182+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:43.182333+0000) 2022-04-03T21:38:43.763 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:43.761+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:38:43.774 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:43.772+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:44.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:44.182+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:44.182511+0000) 2022-04-03T21:38:44.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:44.715+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:38:44.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:44.732+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:45.184 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:45.182+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:45.182722+0000) 2022-04-03T21:38:45.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:45.738+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:38:45.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:45.747+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:45.781 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:38:45.779+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:38:46.008 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:38:46.011 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:38:46.184 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:46.182+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:46.182939+0000) 2022-04-03T21:38:46.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:46.713+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:46.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:46.774+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:38:47.184 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:47.182+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:47.183178+0000) 2022-04-03T21:38:47.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:47.682+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:47.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:47.753+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:38:48.184 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:48.183+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:48.183388+0000) 2022-04-03T21:38:48.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:48.711+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:48.769 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:48.767+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:38:49.185 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:49.183+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:49.183584+0000) 2022-04-03T21:38:49.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:49.724+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:49.777 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:49.775+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:38:50.185 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:50.183+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:50.183823+0000) 2022-04-03T21:38:50.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:50.726+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:38:50.772 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:50.770+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:51.185 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:51.183+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:51.184025+0000) 2022-04-03T21:38:51.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:51.711+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:38:51.786 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:51.785+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:52.185 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:52.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:52.184220+0000) 2022-04-03T21:38:52.722 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:52.720+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:38:52.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:52.748+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:53.185 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:53.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:53.184381+0000) 2022-04-03T21:38:53.705 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:53.703+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:38:53.783 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:53.782+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:54.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:54.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:54.184569+0000) 2022-04-03T21:38:54.746 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:54.744+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:54.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:54.752+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:38:55.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:55.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:55.184759+0000) 2022-04-03T21:38:55.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:55.707+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:55.773 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:55.771+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:38:56.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:56.183+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:56.184941+0000) 2022-04-03T21:38:56.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:56.701+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:56.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:56.743+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:38:57.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:57.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:57.185131+0000) 2022-04-03T21:38:57.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:57.717+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:57.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:57.758+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:38:58.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:58.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:58.185331+0000) 2022-04-03T21:38:58.693 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:58.692+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:58.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:58.774+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:38:59.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:38:59.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:38:59.185482+0000) 2022-04-03T21:38:59.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:38:59.676+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:38:59.792 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:38:59.791+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:00.187 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:00.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:00.185697+0000) 2022-04-03T21:39:00.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:00.657+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:00.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:00.777+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:00.797 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:39:00.794+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:39:01.187 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:01.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:01.185896+0000) 2022-04-03T21:39:01.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:01.660+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:01.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:01.797+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:02.187 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:02.185+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:02.186090+0000) 2022-04-03T21:39:02.705 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:02.703+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:02.774 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:02.772+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:03.187 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:03.185+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:03.186300+0000) 2022-04-03T21:39:03.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:03.683+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:03.740 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:03.738+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:04.188 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:04.185+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:04.186489+0000) 2022-04-03T21:39:04.664 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:04.661+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:04.763 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:04.760+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:05.188 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:05.185+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:05.186675+0000) 2022-04-03T21:39:05.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:05.683+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:05.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:05.779+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:06.188 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:06.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:06.186865+0000) 2022-04-03T21:39:06.712 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:06.709+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:06.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:06.803+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:07.188 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:07.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:07.187019+0000) 2022-04-03T21:39:07.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:07.693+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:07.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:07.818+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:08.188 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:08.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:08.187211+0000) 2022-04-03T21:39:08.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:08.676+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:08.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:08.804+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:09.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:09.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:09.187401+0000) 2022-04-03T21:39:09.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:09.664+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:09.817 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:09.814+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:10.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:10.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:10.187620+0000) 2022-04-03T21:39:10.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:10.679+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:10.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:10.846+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:11.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:11.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:11.187826+0000) 2022-04-03T21:39:11.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:11.704+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:11.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:11.871+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:12.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:12.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:12.187966+0000) 2022-04-03T21:39:12.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:12.666+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:12.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:12.875+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:13.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:13.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:13.188167+0000) 2022-04-03T21:39:13.705 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:13.702+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:13.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:13.868+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:14.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:14.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:14.188320+0000) 2022-04-03T21:39:14.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:14.744+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:14.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:14.871+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:15.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:15.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:15.188460+0000) 2022-04-03T21:39:15.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:15.704+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:15.813 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:39:15.811+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:39:15.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:15.874+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:16.190 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:16.188+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:16.188616+0000) 2022-04-03T21:39:16.238 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:39:16.242 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:39:16.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:16.741+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:16.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:16.870+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:17.190 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:17.188+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:17.188868+0000) 2022-04-03T21:39:17.717 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:17.714+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:17.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:17.836+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:18.190 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:18.188+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:18.189112+0000) 2022-04-03T21:39:18.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:18.746+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:18.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:18.812+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:19.190 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:19.188+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:19.189313+0000) 2022-04-03T21:39:19.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:19.699+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:19.834 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:19.832+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:20.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:20.188+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:20.189491+0000) 2022-04-03T21:39:20.724 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:20.722+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:20.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:20.862+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:21.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:21.189+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:21.189688+0000) 2022-04-03T21:39:21.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:21.683+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:21.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:21.822+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:22.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:22.189+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:22.189895+0000) 2022-04-03T21:39:22.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:22.663+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:22.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:22.776+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:23.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:23.189+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:23.190086+0000) 2022-04-03T21:39:23.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:23.630+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:23.763 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:23.761+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:24.192 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:24.189+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:24.190275+0000) 2022-04-03T21:39:24.674 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:24.671+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:24.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:24.780+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:25.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:25.190+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:25.190465+0000) 2022-04-03T21:39:25.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:25.664+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:25.746 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:25.744+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:26.192 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:26.190+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:26.190672+0000) 2022-04-03T21:39:26.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:26.636+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:26.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:26.709+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:27.192 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:27.190+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:27.190877+0000) 2022-04-03T21:39:27.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:27.637+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:27.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:27.721+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:28.192 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:28.190+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:28.191065+0000) 2022-04-03T21:39:28.634 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:28.632+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:28.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:28.756+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:29.192 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:29.190+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:29.191244+0000) 2022-04-03T21:39:29.659 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:29.657+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:29.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:29.805+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:30.193 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:30.191+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:30.191458+0000) 2022-04-03T21:39:30.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:30.618+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:30.815 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:39:30.812+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:39:30.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:30.816+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:31.193 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:31.191+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:31.191646+0000) 2022-04-03T21:39:31.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:31.590+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:31.867 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:31.865+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:32.193 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:32.191+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:32.191865+0000) 2022-04-03T21:39:32.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:32.566+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:32.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:32.868+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:33.193 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:33.191+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:33.191999+0000) 2022-04-03T21:39:33.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:33.598+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:33.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:33.823+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:34.193 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:34.191+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:34.192214+0000) 2022-04-03T21:39:34.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:34.613+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:34.867 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:34.864+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:35.193 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:35.192+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:35.192428+0000) 2022-04-03T21:39:35.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:35.603+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:35.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:35.900+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:36.194 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:36.192+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:36.192656+0000) 2022-04-03T21:39:36.617 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:36.615+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:36.894 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:36.892+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:37.194 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:37.192+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:37.192875+0000) 2022-04-03T21:39:37.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:37.587+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:37.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:37.862+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:38.194 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:38.192+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:38.193078+0000) 2022-04-03T21:39:38.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:38.612+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:38.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:38.863+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:39.194 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:39.193+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:39.193264+0000) 2022-04-03T21:39:39.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:39:39.563+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 22 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:39:39.906 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:39.904+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:40.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:40.192+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:40.193465+0000) 2022-04-03T21:39:40.946 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:40.944+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:41.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:41.191+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:41.193671+0000) 2022-04-03T21:39:41.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:41.947+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:42.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:42.192+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:42.193879+0000) 2022-04-03T21:39:42.994 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:42.993+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:43.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:43.192+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:43.194046+0000) 2022-04-03T21:39:44.023 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:44.022+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:44.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:44.193+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:44.194264+0000) 2022-04-03T21:39:45.051 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:45.050+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:45.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:45.193+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:45.194439+0000) 2022-04-03T21:39:45.830 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:39:45.827+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:39:46.014 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:46.012+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:46.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:46.193+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:46.194610+0000) 2022-04-03T21:39:46.382 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:39:46.385 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:39:47.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:47.022+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:47.196 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:47.193+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:47.194869+0000) 2022-04-03T21:39:48.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:48.035+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:48.196 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:48.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:48.195052+0000) 2022-04-03T21:39:49.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:49.044+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:49.196 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:49.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:49.195178+0000) 2022-04-03T21:39:50.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:50.011+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:50.196 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:50.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:50.195373+0000) 2022-04-03T21:39:51.034 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:51.032+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:51.197 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:51.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:51.195560+0000) 2022-04-03T21:39:52.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:52.006+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:52.197 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:52.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:52.195747+0000) 2022-04-03T21:39:53.011 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:53.010+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:53.197 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:53.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:53.196011+0000) 2022-04-03T21:39:53.968 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:53.967+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:54.197 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:54.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:54.196217+0000) 2022-04-03T21:39:54.967 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:54.965+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:55.197 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:55.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:55.196427+0000) 2022-04-03T21:39:55.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:55.991+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:56.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:56.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:56.196630+0000) 2022-04-03T21:39:57.036 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:57.034+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:57.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:57.196+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:57.196851+0000) 2022-04-03T21:39:58.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:58.010+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:58.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:58.196+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:58.197077+0000) 2022-04-03T21:39:59.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:39:59.007+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:39:59.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:39:59.196+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:39:59.197263+0000) 2022-04-03T21:40:00.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:00.044+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:00.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:00.196+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:00.197458+0000) 2022-04-03T21:40:00.847 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:40:00.844+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:40:01.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:01.043+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:01.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:01.197+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:01.197645+0000) 2022-04-03T21:40:02.087 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:02.084+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:02.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:02.197+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:02.197867+0000) 2022-04-03T21:40:03.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:03.073+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:03.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:03.197+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:03.197970+0000) 2022-04-03T21:40:04.091 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:04.088+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:04.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:04.197+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:04.198105+0000) 2022-04-03T21:40:05.086 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:05.080+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:05.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:05.197+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:05.198295+0000) 2022-04-03T21:40:06.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:06.043+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:06.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:06.198+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:06.198505+0000) 2022-04-03T21:40:07.034 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:07.032+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:07.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:07.198+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:07.198689+0000) 2022-04-03T21:40:08.006 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:08.004+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:08.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:08.198+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:08.198883+0000) 2022-04-03T21:40:08.975 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:08.973+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:09.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:09.198+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:09.199063+0000) 2022-04-03T21:40:10.017 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:10.014+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:10.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:10.198+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:10.199257+0000) 2022-04-03T21:40:11.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:11.005+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:11.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:11.198+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:11.199381+0000) 2022-04-03T21:40:11.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:11.983+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:12.201 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:12.198+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:12.199541+0000) 2022-04-03T21:40:12.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:12.936+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:13.201 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:13.199+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:13.199764+0000) 2022-04-03T21:40:13.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:13.947+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:14.201 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:14.199+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:14.200033+0000) 2022-04-03T21:40:14.899 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:14.897+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:15.201 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:15.199+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:15.200225+0000) 2022-04-03T21:40:15.847 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:40:15.845+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:40:15.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:15.848+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:16.201 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:16.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:16.200413+0000) 2022-04-03T21:40:16.504 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:40:16.508 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:40:16.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:16.840+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:17.202 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:17.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:17.200672+0000) 2022-04-03T21:40:17.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:17.807+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:18.202 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:18.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:18.200836+0000) 2022-04-03T21:40:18.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:18.822+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:19.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:19.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:19.201032+0000) 2022-04-03T21:40:19.795 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:19.792+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:20.202 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:20.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:20.201240+0000) 2022-04-03T21:40:20.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:20.829+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:21.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:21.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:21.201414+0000) 2022-04-03T21:40:21.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:21.813+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:22.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:22.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:22.201617+0000) 2022-04-03T21:40:22.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:22.812+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:23.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:23.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:23.201849+0000) 2022-04-03T21:40:23.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:23.819+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:24.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:24.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:24.202025+0000) 2022-04-03T21:40:24.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:24.808+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:25.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:25.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:25.202212+0000) 2022-04-03T21:40:25.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:25.844+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:26.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:26.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:26.202420+0000) 2022-04-03T21:40:26.826 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:26.825+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:27.204 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:27.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:27.202624+0000) 2022-04-03T21:40:27.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:27.841+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:28.204 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:28.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:28.202811+0000) 2022-04-03T21:40:28.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:28.868+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:29.204 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:29.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:29.202991+0000) 2022-04-03T21:40:29.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:29.830+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:30.204 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:30.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:30.203216+0000) 2022-04-03T21:40:30.804 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:30.802+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:30.863 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:40:30.860+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:40:31.204 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:31.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:31.203420+0000) 2022-04-03T21:40:31.754 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:31.752+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:32.205 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:32.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:32.203622+0000) 2022-04-03T21:40:32.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:32.797+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:33.205 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:33.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:33.203857+0000) 2022-04-03T21:40:33.752 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:33.749+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:34.205 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:34.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:34.204048+0000) 2022-04-03T21:40:34.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:34.723+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:35.205 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:35.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:35.204244+0000) 2022-04-03T21:40:35.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:35.711+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:36.205 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:36.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:36.204464+0000) 2022-04-03T21:40:36.740 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:36.739+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:37.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:37.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:37.204651+0000) 2022-04-03T21:40:37.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:37.789+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:38.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:38.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:38.204860+0000) 2022-04-03T21:40:38.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:38.782+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:39.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:39.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:39.205039+0000) 2022-04-03T21:40:39.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:39.739+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:40.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:40.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:40.205209+0000) 2022-04-03T21:40:40.764 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:40.762+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:41.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:41.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:41.205401+0000) 2022-04-03T21:40:41.756 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:41.755+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:42.207 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:42.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:42.205630+0000) 2022-04-03T21:40:42.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:42.723+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:43.207 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:43.205+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:43.205849+0000) 2022-04-03T21:40:43.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:43.681+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:44.207 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:44.205+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:44.206019+0000) 2022-04-03T21:40:44.666 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:44.665+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:45.207 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:45.205+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:45.206217+0000) 2022-04-03T21:40:45.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:45.712+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:45.873 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:40:45.871+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:40:46.207 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:46.205+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:46.206406+0000) 2022-04-03T21:40:46.674 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:40:46.678 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:40:46.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:46.693+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:47.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:47.206+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:47.206650+0000) 2022-04-03T21:40:47.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:47.661+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:48.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:48.206+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:48.206857+0000) 2022-04-03T21:40:48.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:48.710+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:49.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:49.206+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:49.206993+0000) 2022-04-03T21:40:49.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:49.758+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:50.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:50.206+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:50.207192+0000) 2022-04-03T21:40:50.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:50.739+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:51.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:51.206+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:51.207379+0000) 2022-04-03T21:40:51.711 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:51.709+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:52.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:52.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:52.207587+0000) 2022-04-03T21:40:52.696 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:52.693+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:53.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:53.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:53.207801+0000) 2022-04-03T21:40:53.652 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:53.649+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:54.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:54.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:54.207993+0000) 2022-04-03T21:40:54.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:54.697+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:55.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:55.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:55.208158+0000) 2022-04-03T21:40:55.678 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:55.676+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:56.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:56.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:56.208354+0000) 2022-04-03T21:40:56.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:56.674+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:57.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:57.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:57.208570+0000) 2022-04-03T21:40:57.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:57.708+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:58.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:58.208+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:58.208755+0000) 2022-04-03T21:40:58.708 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:58.706+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:40:59.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:40:59.208+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:40:59.208964+0000) 2022-04-03T21:40:59.704 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:40:59.701+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:00.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:00.208+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:00.209100+0000) 2022-04-03T21:41:00.724 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:00.721+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:00.889 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:41:00.887+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:41:01.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:01.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:01.209311+0000) 2022-04-03T21:41:01.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:01.747+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:02.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:02.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:02.209443+0000) 2022-04-03T21:41:02.740 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:02.738+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:03.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:03.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:03.209639+0000) 2022-04-03T21:41:03.786 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:03.784+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:04.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:04.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:04.209885+0000) 2022-04-03T21:41:04.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:04.742+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:05.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:05.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:05.210075+0000) 2022-04-03T21:41:05.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:05.755+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:06.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:06.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:06.210313+0000) 2022-04-03T21:41:06.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:06.773+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:07.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:07.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:07.210501+0000) 2022-04-03T21:41:07.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:07.734+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:08.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:08.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:08.210729+0000) 2022-04-03T21:41:08.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:08.748+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:09.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:09.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:09.210914+0000) 2022-04-03T21:41:09.754 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:09.752+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:10.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:10.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:10.211066+0000) 2022-04-03T21:41:10.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:10.715+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:11.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:11.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:11.211276+0000) 2022-04-03T21:41:11.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:11.724+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:12.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:12.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:12.211461+0000) 2022-04-03T21:41:12.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:12.750+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:13.213 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:13.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:13.211674+0000) 2022-04-03T21:41:13.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:13.726+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:14.213 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:14.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:14.211861+0000) 2022-04-03T21:41:14.766 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:14.764+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:15.213 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:15.211+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:15.212029+0000) 2022-04-03T21:41:15.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:15.779+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:15.905 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:41:15.902+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:41:16.213 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:16.211+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:16.212227+0000) 2022-04-03T21:41:16.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:16.781+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:16.829 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:41:16.833 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:41:17.213 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:17.211+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:17.212457+0000) 2022-04-03T21:41:17.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:17.740+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:18.214 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:18.211+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:18.212647+0000) 2022-04-03T21:41:18.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:18.712+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:19.214 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:19.212+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:19.212879+0000) 2022-04-03T21:41:19.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:19.689+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:20.214 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:20.212+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:20.213065+0000) 2022-04-03T21:41:20.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:20.646+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:21.214 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:21.212+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:21.213241+0000) 2022-04-03T21:41:21.649 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:21.647+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:22.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:22.212+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:22.213451+0000) 2022-04-03T21:41:22.601 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:22.598+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:23.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:23.212+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:23.213646+0000) 2022-04-03T21:41:23.630 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:23.628+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:24.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:24.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:24.213856+0000) 2022-04-03T21:41:24.626 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:24.624+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:25.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:25.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:25.214043+0000) 2022-04-03T21:41:25.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:25.592+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:26.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:26.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:26.214235+0000) 2022-04-03T21:41:26.619 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:26.618+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:27.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:27.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:27.214441+0000) 2022-04-03T21:41:27.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:27.602+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:28.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:28.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:28.214626+0000) 2022-04-03T21:41:28.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:28.577+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:29.216 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:29.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:29.214857+0000) 2022-04-03T21:41:29.562 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:29.561+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:30.216 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:30.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:30.215019+0000) 2022-04-03T21:41:30.570 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:30.568+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:30.921 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:41:30.918+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:41:31.216 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:31.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:31.215230+0000) 2022-04-03T21:41:31.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:31.543+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:32.216 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:32.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:32.215424+0000) 2022-04-03T21:41:32.578 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:32.576+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:33.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:33.215+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:33.215585+0000) 2022-04-03T21:41:33.612 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:33.610+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:34.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:34.215+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:34.215841+0000) 2022-04-03T21:41:34.585 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:34.583+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:35.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:35.215+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:35.216063+0000) 2022-04-03T21:41:35.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:35.612+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:36.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:36.215+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:36.216218+0000) 2022-04-03T21:41:36.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:36.603+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:37.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:37.215+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:37.216396+0000) 2022-04-03T21:41:37.556 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:37.554+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:38.218 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:38.216+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:38.216603+0000) 2022-04-03T21:41:38.578 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:38.573+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:39.218 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:39.216+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:39.216812+0000) 2022-04-03T21:41:39.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:39.544+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:41:40.218 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:40.216+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:40.216979+0000) 2022-04-03T21:41:40.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:40.542+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:41:41.218 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:41.216+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:41.217184+0000) 2022-04-03T21:41:41.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:41.496+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:41:42.218 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:42.217+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:42.217366+0000) 2022-04-03T21:41:42.500 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:42.498+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:41:43.219 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:43.217+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:43.217574+0000) 2022-04-03T21:41:43.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:43.459+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:41:44.219 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:44.217+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:44.217807+0000) 2022-04-03T21:41:44.475 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:44.472+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:41:45.219 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:45.217+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:45.218007+0000) 2022-04-03T21:41:45.498 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:45.495+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:41:45.936 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:41:45.934+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:41:46.219 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:46.217+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:46.218202+0000) 2022-04-03T21:41:46.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:46.534+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:41:46.960 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:41:46.964 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:41:47.219 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:47.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:47.218394+0000) 2022-04-03T21:41:47.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:47.569+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:41:48.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:48.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:48.218555+0000) 2022-04-03T21:41:48.550 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:48.548+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:41:49.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:49.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:49.218727+0000) 2022-04-03T21:41:49.573 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:49.570+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:41:50.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:50.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:50.218938+0000) 2022-04-03T21:41:50.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:50.521+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:41:51.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:51.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:51.219143+0000) 2022-04-03T21:41:51.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:51.564+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:41:52.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:52.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:52.219298+0000) 2022-04-03T21:41:52.593 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:52.591+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:41:53.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:53.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:53.219487+0000) 2022-04-03T21:41:53.591 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:53.589+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:41:54.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:54.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:54.219682+0000) 2022-04-03T21:41:54.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:54.550+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:41:55.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:55.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:55.219864+0000) 2022-04-03T21:41:55.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:55.508+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:41:56.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:56.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:56.220051+0000) 2022-04-03T21:41:56.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:56.512+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:41:57.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:57.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:57.220238+0000) 2022-04-03T21:41:57.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:57.526+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:41:58.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:58.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:58.220403+0000) 2022-04-03T21:41:58.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:58.541+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:41:59.222 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:41:59.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:41:59.220603+0000) 2022-04-03T21:41:59.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:41:59.529+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:00.222 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:00.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:00.220854+0000) 2022-04-03T21:42:00.484 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:00.482+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:00.952 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:42:00.949+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:42:01.222 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:01.220+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:01.221056+0000) 2022-04-03T21:42:01.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:01.531+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:02.222 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:02.220+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:02.221235+0000) 2022-04-03T21:42:02.525 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:02.522+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:03.223 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:03.220+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:03.221386+0000) 2022-04-03T21:42:03.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:03.565+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:04.223 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:04.220+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:04.221570+0000) 2022-04-03T21:42:04.540 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:04.538+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:05.223 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:05.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:05.221805+0000) 2022-04-03T21:42:05.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:05.573+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:06.223 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:06.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:06.222014+0000) 2022-04-03T21:42:06.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:06.532+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:07.223 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:07.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:07.222208+0000) 2022-04-03T21:42:07.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:07.489+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:08.223 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:08.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:08.222343+0000) 2022-04-03T21:42:08.470 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:08.468+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:09.224 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:09.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:09.222544+0000) 2022-04-03T21:42:09.450 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:09.447+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:10.224 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:10.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:10.222806+0000) 2022-04-03T21:42:10.465 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:10.463+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:11.224 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:11.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:11.222961+0000) 2022-04-03T21:42:11.482 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:11.480+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:12.224 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:12.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:12.223170+0000) 2022-04-03T21:42:12.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:12.495+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:13.224 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:13.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:13.223357+0000) 2022-04-03T21:42:13.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:13.510+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:14.224 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:14.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:14.223539+0000) 2022-04-03T21:42:14.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:14.532+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:15.225 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:15.223+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:15.223733+0000) 2022-04-03T21:42:15.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:15.492+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:15.968 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:42:15.966+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:42:16.225 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:16.223+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:16.223949+0000) 2022-04-03T21:42:16.519 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:16.517+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:17.101 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:42:17.105 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:42:17.225 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:17.223+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:17.224163+0000) 2022-04-03T21:42:17.480 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:17.478+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:18.226 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:18.223+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:18.224372+0000) 2022-04-03T21:42:18.527 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:18.525+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:19.226 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:19.224+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:19.224562+0000) 2022-04-03T21:42:19.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:19.508+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:20.226 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:20.224+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:20.224822+0000) 2022-04-03T21:42:20.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:20.510+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:21.226 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:21.224+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:21.225042+0000) 2022-04-03T21:42:21.470 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:21.468+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:22.226 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:22.224+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:22.225226+0000) 2022-04-03T21:42:22.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:22.499+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:23.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:23.225+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:23.225440+0000) 2022-04-03T21:42:23.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:23.491+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:24.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:24.225+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:24.225638+0000) 2022-04-03T21:42:24.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:24.490+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:25.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:25.225+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:25.225845+0000) 2022-04-03T21:42:25.496 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:25.494+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:26.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:26.225+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:26.226045+0000) 2022-04-03T21:42:26.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:26.529+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:27.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:27.225+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:27.226244+0000) 2022-04-03T21:42:27.516 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:27.515+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:28.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:28.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:28.226405+0000) 2022-04-03T21:42:28.516 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:28.515+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:29.228 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:29.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:29.226595+0000) 2022-04-03T21:42:29.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:29.537+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:30.228 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:30.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:30.226849+0000) 2022-04-03T21:42:30.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:30.544+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:30.984 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:42:30.981+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:42:31.228 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:31.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:31.227043+0000) 2022-04-03T21:42:31.520 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:31.518+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:32.228 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:32.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:32.227248+0000) 2022-04-03T21:42:32.526 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:32.524+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:33.229 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:33.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:33.227492+0000) 2022-04-03T21:42:33.526 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:33.524+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:34.229 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:34.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:34.227712+0000) 2022-04-03T21:42:34.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:34.499+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:35.229 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:35.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:35.227881+0000) 2022-04-03T21:42:35.462 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:35.460+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:36.229 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:36.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:36.228035+0000) 2022-04-03T21:42:36.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:36.486+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:37.229 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:37.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:37.228205+0000) 2022-04-03T21:42:37.490 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:37.488+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:38.229 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:38.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:38.228407+0000) 2022-04-03T21:42:38.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:38.537+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:39.230 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:39.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:39.228551+0000) 2022-04-03T21:42:39.564 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:39.562+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:40.230 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:40.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:40.228796+0000) 2022-04-03T21:42:40.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:40.565+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:41.117 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:42:41.114+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:42:41.230 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:41.228+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:41.229000+0000) 2022-04-03T21:42:41.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:41.519+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:42.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:42:42.161+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:42:42.230 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:42.228+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:42.229195+0000) 2022-04-03T21:42:42.502 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:42.500+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:43.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:42:43.164+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:42:43.230 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:43.228+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:43.229425+0000) 2022-04-03T21:42:43.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:43.472+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:44.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:42:44.173+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:42:44.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:44.228+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:44.229649+0000) 2022-04-03T21:42:44.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:44.515+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:45.191 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:42:45.188+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:42:45.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:45.228+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:45.229845+0000) 2022-04-03T21:42:45.563 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:45.560+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:45.992 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:42:45.990+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:42:46.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:42:46.154+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:42:46.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:46.229+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:46.230046+0000) 2022-04-03T21:42:46.569 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:46.566+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:47.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:42:47.195+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:42:47.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:47.229+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:47.230224+0000) 2022-04-03T21:42:47.238 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:42:47.241 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:42:47.575 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:47.573+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:48.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:42:48.191+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:42:48.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:48.229+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:48.230414+0000) 2022-04-03T21:42:48.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:48.544+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:49.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:42:49.195+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:42:49.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:49.229+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:49.230597+0000) 2022-04-03T21:42:49.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:49.509+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:50.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:42:50.216+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:42:50.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:50.230+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:50.230737+0000) 2022-04-03T21:42:50.559 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:50.557+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:51.232 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:51.230+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:51.230932+0000) 2022-04-03T21:42:51.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:42:51.258+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:42:51.547 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:51.544+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:52.232 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:52.230+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:52.231129+0000) 2022-04-03T21:42:52.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:42:52.238+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:42:52.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:52.590+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:53.232 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:53.230+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:53.231297+0000) 2022-04-03T21:42:53.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:42:53.273+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:42:53.565 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:53.563+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:54.232 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:54.230+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:54.231503+0000) 2022-04-03T21:42:54.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:42:54.264+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:42:54.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:54.599+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:55.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:55.230+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:55.231727+0000) 2022-04-03T21:42:55.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:42:55.232+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:42:55.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:55.633+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:56.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:42:56.220+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:42:56.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:56.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:56.231941+0000) 2022-04-03T21:42:56.596 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:56.594+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:57.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:42:57.223+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:42:57.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:57.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:57.232156+0000) 2022-04-03T21:42:57.554 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:57.552+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:58.212 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:42:58.210+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:42:58.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:58.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:58.232339+0000) 2022-04-03T21:42:58.519 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:58.516+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:42:59.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:42:59.183+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:42:59.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:42:59.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:42:59.232534+0000) 2022-04-03T21:42:59.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:42:59.489+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:00.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:00.228+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:00.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:00.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:00.232728+0000) 2022-04-03T21:43:00.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:00.495+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:01.008 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:43:01.006+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:43:01.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:01.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:01.232893+0000) 2022-04-03T21:43:01.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:01.238+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:01.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:01.543+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:02.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:02.226+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:02.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:02.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:02.233092+0000) 2022-04-03T21:43:02.561 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:02.558+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:03.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:03.190+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:03.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:03.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:03.233290+0000) 2022-04-03T21:43:03.530 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:03.527+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:04.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:04.205+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:04.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:04.233+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:04.233491+0000) 2022-04-03T21:43:04.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:04.488+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:05.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:05.157+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:05.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:05.233+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:05.233711+0000) 2022-04-03T21:43:05.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:05.508+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:06.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:06.177+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:06.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:06.233+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:06.233896+0000) 2022-04-03T21:43:06.462 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:06.459+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:07.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:07.176+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:07.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:07.233+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:07.234027+0000) 2022-04-03T21:43:07.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:07.461+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:08.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:08.207+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:08.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:08.233+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:08.234216+0000) 2022-04-03T21:43:08.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:08.486+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:09.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:09.222+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:09.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:09.234+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:09.234416+0000) 2022-04-03T21:43:09.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:09.438+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:10.236 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:10.234+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:10.234588+0000) 2022-04-03T21:43:10.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:10.253+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:10.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:10.465+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:11.236 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:11.234+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:11.234836+0000) 2022-04-03T21:43:11.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:11.291+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:11.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:11.458+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:12.236 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:12.234+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:12.234985+0000) 2022-04-03T21:43:12.284 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:12.281+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:12.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:12.474+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:13.236 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:13.234+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:13.235179+0000) 2022-04-03T21:43:13.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:13.297+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:13.509 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:13.507+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:14.236 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:14.235+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:14.235400+0000) 2022-04-03T21:43:14.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:14.304+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:14.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:14.535+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:15.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:15.235+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:15.235608+0000) 2022-04-03T21:43:15.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:15.333+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:15.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:15.501+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:16.024 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:43:16.022+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:43:16.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:16.235+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:16.235854+0000) 2022-04-03T21:43:16.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:16.325+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:16.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:16.549+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:17.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:17.235+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:17.236037+0000) 2022-04-03T21:43:17.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:17.313+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:17.361 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:43:17.365 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:43:17.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:17.593+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:18.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:18.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:18.236222+0000) 2022-04-03T21:43:18.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:18.285+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:18.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:18.605+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:19.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:19.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:19.236416+0000) 2022-04-03T21:43:19.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:19.256+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:19.651 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:19.649+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:20.238 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:20.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:20.236580+0000) 2022-04-03T21:43:20.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:20.243+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:20.639 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:20.637+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:21.238 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:21.235+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:21.236799+0000) 2022-04-03T21:43:21.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:21.263+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:21.593 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:21.591+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:22.238 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:22.235+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:22.236977+0000) 2022-04-03T21:43:22.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:22.297+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:22.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:22.639+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:23.238 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:23.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:23.237189+0000) 2022-04-03T21:43:23.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:23.341+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:23.689 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:23.688+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:24.238 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:24.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:24.237370+0000) 2022-04-03T21:43:24.376 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:24.374+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:24.704 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:24.703+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:25.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:25.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:25.237541+0000) 2022-04-03T21:43:25.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:25.377+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 4 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:25.701 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:25.699+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:26.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:26.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:26.237751+0000) 2022-04-03T21:43:26.421 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:26.419+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:26.684 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:26.683+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:27.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:27.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:27.237977+0000) 2022-04-03T21:43:27.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:27.435+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:27.719 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:27.718+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:28.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:28.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:28.238173+0000) 2022-04-03T21:43:28.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:28.437+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:28.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:28.757+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:29.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:29.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:29.238368+0000) 2022-04-03T21:43:29.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:29.423+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:29.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:29.727+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:30.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:30.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:30.238544+0000) 2022-04-03T21:43:30.442 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:30.440+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:30.746 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:30.745+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:31.040 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:43:31.038+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:43:31.240 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:31.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:31.238732+0000) 2022-04-03T21:43:31.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:31.438+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:31.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:31.778+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:32.240 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:32.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:32.238935+0000) 2022-04-03T21:43:32.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:32.390+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:32.786 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:32.784+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:33.240 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:33.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:33.239131+0000) 2022-04-03T21:43:33.438 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:33.436+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:33.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:33.816+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:34.240 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:34.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:34.239353+0000) 2022-04-03T21:43:34.457 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:34.455+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:34.852 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:34.849+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:35.241 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:35.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:35.239602+0000) 2022-04-03T21:43:35.434 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:35.431+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:35.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:35.897+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:36.241 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:36.239+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:36.239834+0000) 2022-04-03T21:43:36.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:36.442+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:36.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:36.894+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:37.241 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:37.239+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:37.240042+0000) 2022-04-03T21:43:37.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:37.458+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:37.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:37.846+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:38.241 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:38.239+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:38.240239+0000) 2022-04-03T21:43:38.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:38.473+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:38.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:38.882+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:39.241 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:39.239+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:39.240434+0000) 2022-04-03T21:43:39.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:39.433+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:39.919 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:39.916+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:40.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:40.239+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:40.240605+0000) 2022-04-03T21:43:40.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:40.445+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:40.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:40.867+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:41.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:41.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:41.240738+0000) 2022-04-03T21:43:41.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:41.403+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:41.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:41.882+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:42.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:42.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:42.240944+0000) 2022-04-03T21:43:42.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:42.403+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:42.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:42.898+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:43.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:43.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:43.241148+0000) 2022-04-03T21:43:43.442 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:43.439+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:43.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:43.856+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:44.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:44.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:44.241318+0000) 2022-04-03T21:43:44.434 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:44.432+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:44.867 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:44.864+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:45.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:45.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:45.241523+0000) 2022-04-03T21:43:45.454 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:45.452+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:45.895 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:45.893+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:46.056 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:43:46.054+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:43:46.243 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:46.241+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:46.241737+0000) 2022-04-03T21:43:46.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:46.488+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:46.884 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:46.882+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:47.243 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:47.241+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:47.241961+0000) 2022-04-03T21:43:47.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:47.440+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:47.610 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:43:47.614 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:43:47.884 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:47.882+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:48.243 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:48.241+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:48.242188+0000) 2022-04-03T21:43:48.400 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:48.397+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:48.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:48.885+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:49.243 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:49.241+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:49.242358+0000) 2022-04-03T21:43:49.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:49.396+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:49.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:49.879+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:50.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:50.242+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:50.242551+0000) 2022-04-03T21:43:50.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:50.417+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:50.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:50.920+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:51.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:51.242+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:51.242742+0000) 2022-04-03T21:43:51.437 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:51.435+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:51.916 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:51.914+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:52.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:52.242+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:52.242895+0000) 2022-04-03T21:43:52.424 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:52.422+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:52.946 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:52.944+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:53.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:53.242+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:53.243088+0000) 2022-04-03T21:43:53.429 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:53.427+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:53.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:53.962+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:54.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:54.242+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:54.243296+0000) 2022-04-03T21:43:54.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:54.417+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:54.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:54.963+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:55.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:55.243+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:55.243505+0000) 2022-04-03T21:43:55.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:55.443+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:55.987 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:55.985+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:56.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:56.243+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:56.243714+0000) 2022-04-03T21:43:56.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:56.479+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:56.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:56.980+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:57.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:57.243+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:57.243891+0000) 2022-04-03T21:43:57.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:57.437+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:57.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:57.970+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:58.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:58.243+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:58.244067+0000) 2022-04-03T21:43:58.468 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:58.466+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:43:58.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:43:58.963+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:43:59.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:43:59.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:43:59.244224+0000) 2022-04-03T21:43:59.431 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:43:59.428+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:00.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:00.001+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:00.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:00.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:00.244439+0000) 2022-04-03T21:44:00.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:00.409+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:01.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:01.046+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:01.066 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:44:01.064+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:44:01.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:01.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:01.244601+0000) 2022-04-03T21:44:01.438 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:01.435+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:02.070 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:02.068+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:02.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:02.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:02.244807+0000) 2022-04-03T21:44:02.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:02.434+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:03.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:03.061+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:03.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:03.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:03.244979+0000) 2022-04-03T21:44:03.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:03.425+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:04.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:04.064+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:04.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:04.245+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:04.245166+0000) 2022-04-03T21:44:04.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:04.393+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:05.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:05.064+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:05.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:05.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:05.245373+0000) 2022-04-03T21:44:05.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:05.414+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:06.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:06.072+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:06.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:06.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:06.245522+0000) 2022-04-03T21:44:06.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:06.423+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:07.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:07.110+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:07.247 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:07.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:07.245731+0000) 2022-04-03T21:44:07.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:07.391+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:08.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:08.077+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:08.247 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:08.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:08.245882+0000) 2022-04-03T21:44:08.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:08.370+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:09.120 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:09.118+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:09.247 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:09.245+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:09.246066+0000) 2022-04-03T21:44:09.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:09.390+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:10.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:10.132+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:10.247 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:10.245+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:10.246279+0000) 2022-04-03T21:44:10.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:10.383+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:11.106 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:11.104+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:11.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:11.245+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:11.246515+0000) 2022-04-03T21:44:11.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:11.364+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:12.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:12.133+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:12.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:12.245+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:12.246735+0000) 2022-04-03T21:44:12.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:12.349+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:13.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:13.180+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:13.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:13.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:13.246941+0000) 2022-04-03T21:44:13.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:13.315+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:14.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:14.134+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:14.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:14.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:14.247140+0000) 2022-04-03T21:44:14.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:14.359+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:15.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:15.176+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:15.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:15.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:15.247309+0000) 2022-04-03T21:44:15.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:15.333+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:16.082 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:44:16.080+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:44:16.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:16.169+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:16.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:16.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:16.247487+0000) 2022-04-03T21:44:16.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:16.310+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:17.213 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:17.211+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:17.249 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:17.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:17.247686+0000) 2022-04-03T21:44:17.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:17.295+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:17.750 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:44:17.754 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:44:18.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:18.162+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:18.249 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:18.247+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:18.247937+0000) 2022-04-03T21:44:18.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:18.320+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:19.140 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:19.138+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:19.249 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:19.247+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:19.248145+0000) 2022-04-03T21:44:19.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:19.352+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:20.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:20.129+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:20.249 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:20.247+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:20.248326+0000) 2022-04-03T21:44:20.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:20.367+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:21.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:21.163+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:21.250 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:21.247+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:21.248514+0000) 2022-04-03T21:44:21.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:21.340+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:22.214 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:22.213+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:22.250 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:22.248+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:22.248699+0000) 2022-04-03T21:44:22.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:22.340+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:23.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:23.205+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:23.250 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:23.248+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:23.248887+0000) 2022-04-03T21:44:23.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:23.369+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:24.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:24.217+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:24.250 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:24.248+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:24.249031+0000) 2022-04-03T21:44:24.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:24.387+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:25.193 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:25.191+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:25.250 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:25.248+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:25.249184+0000) 2022-04-03T21:44:25.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:25.348+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:26.186 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:26.184+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:26.250 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:26.248+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:26.249386+0000) 2022-04-03T21:44:26.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:26.338+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:27.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:27.202+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:27.251 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:27.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:27.249601+0000) 2022-04-03T21:44:27.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:27.380+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:28.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:28.193+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:28.251 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:28.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:28.249855+0000) 2022-04-03T21:44:28.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:28.348+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:29.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:29.224+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:29.251 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:29.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:29.250023+0000) 2022-04-03T21:44:29.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:29.318+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:30.251 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:30.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:30.250170+0000) 2022-04-03T21:44:30.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:30.257+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:30.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:30.306+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:31.084 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:44:31.082+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:44:31.252 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:31.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:31.250361+0000) 2022-04-03T21:44:31.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:31.292+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:31.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:31.329+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:32.252 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:32.250+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:32.250556+0000) 2022-04-03T21:44:32.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:32.287+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:32.291 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:32.289+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:33.252 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:33.250+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:33.250750+0000) 2022-04-03T21:44:33.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:33.255+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:33.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:33.333+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:34.230 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:34.227+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:34.252 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:34.250+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:34.250982+0000) 2022-04-03T21:44:34.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:34.369+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:35.209 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:35.206+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:35.252 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:35.250+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:35.251182+0000) 2022-04-03T21:44:35.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:35.354+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:36.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:36.251+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:36.251380+0000) 2022-04-03T21:44:36.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:36.254+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:36.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:36.314+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:37.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:37.251+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:37.251592+0000) 2022-04-03T21:44:37.256 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:37.253+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:37.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:37.318+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:38.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:38.251+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:38.251833+0000) 2022-04-03T21:44:38.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:38.276+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:38.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:38.313+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:39.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:39.245+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:39.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:39.251+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:39.251986+0000) 2022-04-03T21:44:39.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:39.339+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:40.228 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:40.225+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:40.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:40.251+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:40.252196+0000) 2022-04-03T21:44:40.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:40.308+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:41.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:41.246+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:41.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:41.252+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:41.252401+0000) 2022-04-03T21:44:41.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:41.318+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:42.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:42.243+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:42.254 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:42.252+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:42.252630+0000) 2022-04-03T21:44:42.337 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:42.334+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:43.254 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:43.252+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:43.252833+0000) 2022-04-03T21:44:43.287 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:43.285+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:43.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:43.305+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:44.241 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:44.239+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:44.254 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:44.252+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:44.253030+0000) 2022-04-03T21:44:44.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:44.293+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:45.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:45.209+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:45.254 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:45.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:45.253233+0000) 2022-04-03T21:44:45.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:45.253+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:46.090 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:44:46.088+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:44:46.205 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:46.203+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:46.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:46.239+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:46.254 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:46.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:46.253421+0000) 2022-04-03T21:44:47.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:47.172+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:47.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:47.229+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:47.254 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:47.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:47.253617+0000) 2022-04-03T21:44:47.920 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:44:47.924 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:44:48.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:48.199+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:48.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:48.205+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:48.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:48.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:48.253864+0000) 2022-04-03T21:44:49.217 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:49.215+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:49.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:49.252+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:49.254031+0000) 2022-04-03T21:44:49.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:49.255+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:50.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:50.168+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:50.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:50.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:50.254250+0000) 2022-04-03T21:44:50.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:50.304+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:51.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:51.199+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:51.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:51.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:51.254405+0000) 2022-04-03T21:44:51.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:51.259+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:52.186 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:52.184+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:52.256 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:52.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:52.254540+0000) 2022-04-03T21:44:52.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:52.276+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:53.148 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:53.146+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:53.256 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:53.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:53.254735+0000) 2022-04-03T21:44:53.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:53.275+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:54.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:54.118+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:54.256 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:54.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:54.254964+0000) 2022-04-03T21:44:54.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:54.282+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:55.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:55.130+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:55.256 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:55.254+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:55.255170+0000) 2022-04-03T21:44:55.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:55.316+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:56.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:56.122+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:56.256 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:56.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:56.255332+0000) 2022-04-03T21:44:56.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:56.325+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:57.105 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:57.103+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:57.257 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:57.254+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:57.255522+0000) 2022-04-03T21:44:57.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:57.283+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:58.109 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:58.107+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:58.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:58.251+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:58.257 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:58.254+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:58.255725+0000) 2022-04-03T21:44:59.064 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:44:59.062+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:44:59.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:44:59.235+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:44:59.257 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:44:59.255+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:44:59.255966+0000) 2022-04-03T21:45:00.020 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:00.017+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:00.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:00.199+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:00.257 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:00.255+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:00.256183+0000) 2022-04-03T21:45:00.996 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:00.994+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:01.106 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:45:01.104+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:45:01.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:01.225+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:01.257 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:01.255+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:01.256384+0000) 2022-04-03T21:45:01.972 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:01.970+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:02.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:02.182+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:02.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:02.255+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:02.256589+0000) 2022-04-03T21:45:02.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:02.951+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:03.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:03.178+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:03.258 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:03.255+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:03.256695+0000) 2022-04-03T21:45:03.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:03.931+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:04.140 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:04.137+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:04.258 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:04.256+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:04.256893+0000) 2022-04-03T21:45:04.920 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:04.917+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:05.113 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:05.111+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:05.258 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:05.256+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:05.257054+0000) 2022-04-03T21:45:05.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:05.935+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:06.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:06.158+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:06.258 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:06.256+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:06.257251+0000) 2022-04-03T21:45:06.930 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:06.928+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:07.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:07.173+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:07.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:07.256+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:07.257462+0000) 2022-04-03T21:45:07.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:07.910+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:08.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:08.187+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:08.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:08.257+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:08.257614+0000) 2022-04-03T21:45:08.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:08.926+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:09.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:09.177+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:09.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:09.257+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:09.257834+0000) 2022-04-03T21:45:09.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:09.947+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:10.176 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:10.174+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:10.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:10.257+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:10.258037+0000) 2022-04-03T21:45:10.976 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:10.975+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:11.127 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:11.125+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:11.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:11.257+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:11.258256+0000) 2022-04-03T21:45:11.960 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:11.959+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:12.093 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:12.091+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:12.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:12.257+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:12.258427+0000) 2022-04-03T21:45:12.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:12.983+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:13.140 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:13.138+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:13.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:13.258+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:13.258633+0000) 2022-04-03T21:45:13.956 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:13.954+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:14.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:14.157+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:14.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:14.258+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:14.258858+0000) 2022-04-03T21:45:14.920 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:14.919+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:15.149 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:15.147+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:15.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:15.258+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:15.259071+0000) 2022-04-03T21:45:15.920 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:15.919+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:16.122 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:45:16.120+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:45:16.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:16.185+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:16.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:16.258+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:16.259283+0000) 2022-04-03T21:45:16.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:16.916+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:17.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:17.216+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:17.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:17.259+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:17.259490+0000) 2022-04-03T21:45:17.889 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:17.887+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:18.062 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:45:18.066 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:45:18.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:18.210+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:18.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:18.259+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:18.259744+0000) 2022-04-03T21:45:18.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:18.869+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:19.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:19.214+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:19.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:19.259+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:19.259957+0000) 2022-04-03T21:45:19.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:19.884+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:20.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:20.259+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:20.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:20.259+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:20.260167+0000) 2022-04-03T21:45:20.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:20.885+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:21.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:21.260+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:21.260350+0000) 2022-04-03T21:45:21.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:21.276+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:21.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:21.884+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:22.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:22.256+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:22.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:22.260+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:22.260565+0000) 2022-04-03T21:45:22.909 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:22.907+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:23.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:23.254+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:23.262 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:23.260+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:23.260802+0000) 2022-04-03T21:45:23.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:23.914+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:24.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:24.229+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:24.262 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:24.260+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:24.261005+0000) 2022-04-03T21:45:24.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:24.893+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:25.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:25.224+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:25.262 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:25.260+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:25.261219+0000) 2022-04-03T21:45:25.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:25.925+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:26.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:26.205+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:26.262 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:26.261+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:26.261430+0000) 2022-04-03T21:45:26.880 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:26.878+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:27.176 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:27.173+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:27.263 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:27.261+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:27.261646+0000) 2022-04-03T21:45:27.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:27.879+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:28.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:28.186+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:28.264 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:28.261+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:28.261815+0000) 2022-04-03T21:45:28.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:28.869+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:29.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:29.175+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:29.263 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:29.261+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:29.262015+0000) 2022-04-03T21:45:29.909 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:29.907+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:30.214 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:30.211+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:30.263 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:30.262+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:30.262170+0000) 2022-04-03T21:45:30.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:30.946+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:31.138 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:45:31.136+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:45:31.172 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:31.170+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:31.263 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:31.262+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:31.262368+0000) 2022-04-03T21:45:31.935 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:31.932+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:32.169 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:32.167+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:32.264 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:32.261+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:32.262603+0000) 2022-04-03T21:45:32.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:32.891+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:33.147 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:33.144+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:33.264 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:33.261+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:33.262841+0000) 2022-04-03T21:45:33.880 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:33.877+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:34.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:34.194+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:34.264 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:34.261+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:34.263037+0000) 2022-04-03T21:45:34.874 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:34.871+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:35.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:35.229+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:35.264 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:35.262+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:35.263226+0000) 2022-04-03T21:45:35.913 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:35.911+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:36.264 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:36.262+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:36.263429+0000) 2022-04-03T21:45:36.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:36.265+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:36.943 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:36.940+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:37.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:37.246+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:37.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:37.262+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:37.263593+0000) 2022-04-03T21:45:37.946 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:37.944+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:38.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:38.201+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:38.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:38.262+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:38.263764+0000) 2022-04-03T21:45:38.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:38.929+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:39.235 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:39.233+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:39.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:39.263+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:39.263948+0000) 2022-04-03T21:45:39.905 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:39.903+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:40.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:40.229+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:40.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:40.263+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:40.264161+0000) 2022-04-03T21:45:40.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:40.909+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:41.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:41.207+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:41.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:41.263+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:41.264358+0000) 2022-04-03T21:45:41.945 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:41.942+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:42.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:42.194+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:42.266 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:42.263+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:42.264598+0000) 2022-04-03T21:45:42.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:42.986+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:43.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:43.155+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:43.266 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:43.263+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:43.264798+0000) 2022-04-03T21:45:43.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:43.936+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:44.142 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:44.139+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:44.266 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:44.264+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:44.264991+0000) 2022-04-03T21:45:44.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:44.938+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:45.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:45.185+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:45.266 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:45.264+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:45.265197+0000) 2022-04-03T21:45:45.946 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:45.944+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:46.153 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:45:46.151+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:45:46.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:46.190+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:46.266 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:46.264+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:46.265356+0000) 2022-04-03T21:45:46.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:46.931+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:47.214 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:47.211+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:47.266 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:47.264+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:47.265528+0000) 2022-04-03T21:45:47.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:47.919+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:48.210 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:45:48.213 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:45:48.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:48.235+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:48.267 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:48.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:48.265803+0000) 2022-04-03T21:45:48.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:48.870+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:49.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:49.261+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:49.267 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:49.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:49.265978+0000) 2022-04-03T21:45:49.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:49.919+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:50.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:50.260+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:50.267 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:50.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:50.266181+0000) 2022-04-03T21:45:50.934 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:50.931+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:51.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:51.240+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:51.267 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:51.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:51.266372+0000) 2022-04-03T21:45:51.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:51.909+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:52.268 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:52.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:52.266625+0000) 2022-04-03T21:45:52.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:52.267+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:52.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:52.919+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:53.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:53.236+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:53.268 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:53.266+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:53.266808+0000) 2022-04-03T21:45:53.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:53.962+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:54.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:54.248+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:54.268 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:54.266+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:54.267001+0000) 2022-04-03T21:45:54.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:54.921+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:55.268 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:55.266+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:55.267187+0000) 2022-04-03T21:45:55.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:55.279+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:55.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:55.875+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:56.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:56.239+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:56.268 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:56.266+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:56.267407+0000) 2022-04-03T21:45:56.895 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:56.893+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:57.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:57.228+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:57.269 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:57.266+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:57.267586+0000) 2022-04-03T21:45:57.892 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:57.890+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:58.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:58.228+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:58.269 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:58.267+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:58.267842+0000) 2022-04-03T21:45:58.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:58.844+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:45:59.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:45:59.235+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:45:59.269 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:45:59.267+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:45:59.268058+0000) 2022-04-03T21:45:59.884 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:45:59.882+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:00.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:00.220+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:00.269 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:00.267+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:00.268250+0000) 2022-04-03T21:46:00.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:00.849+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:01.169 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:46:01.167+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:46:01.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:01.187+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:01.269 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:01.268+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:01.268477+0000) 2022-04-03T21:46:01.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:01.834+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:02.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:02.195+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:02.270 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:02.268+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:02.268683+0000) 2022-04-03T21:46:02.883 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:02.881+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:03.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:03.222+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:03.270 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:03.268+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:03.268901+0000) 2022-04-03T21:46:03.869 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:03.867+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:04.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:04.195+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:04.270 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:04.268+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:04.269097+0000) 2022-04-03T21:46:04.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:04.828+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:05.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:05.161+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:05.270 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:05.268+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:05.269320+0000) 2022-04-03T21:46:05.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:05.841+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:06.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:06.195+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:06.271 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:06.269+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:06.269448+0000) 2022-04-03T21:46:06.828 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:06.826+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:07.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:07.208+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:07.271 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:07.269+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:07.269646+0000) 2022-04-03T21:46:07.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:07.791+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:08.206 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:08.205+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:08.271 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:08.269+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:08.269855+0000) 2022-04-03T21:46:08.766 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:08.764+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:09.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:09.183+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:09.271 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:09.269+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:09.270044+0000) 2022-04-03T21:46:09.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:09.735+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:10.147 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:10.146+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:10.271 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:10.270+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:10.270222+0000) 2022-04-03T21:46:10.766 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:10.764+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:11.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:11.188+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:11.271 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:11.269+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:11.270375+0000) 2022-04-03T21:46:11.790 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:11.788+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:12.145 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:12.143+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:12.271 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:12.270+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:12.270555+0000) 2022-04-03T21:46:12.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:12.759+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:13.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:13.148+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:13.272 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:13.270+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:13.270752+0000) 2022-04-03T21:46:13.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:13.731+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:14.108 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:14.106+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:14.272 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:14.270+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:14.270972+0000) 2022-04-03T21:46:14.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:14.730+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:15.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:15.154+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:15.272 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:15.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:15.271176+0000) 2022-04-03T21:46:15.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:15.689+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:16.185 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:46:16.183+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:46:16.206 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:16.204+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:16.272 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:16.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:16.271396+0000) 2022-04-03T21:46:16.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:16.727+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:17.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:17.247+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:17.272 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:17.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:17.271584+0000) 2022-04-03T21:46:17.696 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:17.693+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:18.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:18.241+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:18.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:18.270+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:18.271824+0000) 2022-04-03T21:46:18.357 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:46:18.361 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:46:18.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:18.666+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:19.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:19.260+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:19.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:19.270+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:19.272060+0000) 2022-04-03T21:46:19.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:19.708+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:20.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:20.222+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:20.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:20.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:20.272242+0000) 2022-04-03T21:46:20.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:20.725+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:21.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:21.176+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:21.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:21.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:21.272408+0000) 2022-04-03T21:46:21.751 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:21.749+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:22.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:22.153+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:22.274 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:22.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:22.272597+0000) 2022-04-03T21:46:22.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:22.783+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:23.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:23.184+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:23.274 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:23.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:23.272826+0000) 2022-04-03T21:46:23.736 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:23.733+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:24.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:24.201+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:24.274 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:24.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:24.273039+0000) 2022-04-03T21:46:24.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:24.759+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:25.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:25.154+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:25.274 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:25.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:25.273225+0000) 2022-04-03T21:46:25.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:25.729+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:26.158 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:26.155+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:26.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:26.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:26.273432+0000) 2022-04-03T21:46:26.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:26.745+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:27.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:27.176+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:27.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:27.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:27.273647+0000) 2022-04-03T21:46:27.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:27.755+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:28.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:28.175+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:28.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:28.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:28.273872+0000) 2022-04-03T21:46:28.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:28.759+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:29.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:29.166+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:29.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:29.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:29.274062+0000) 2022-04-03T21:46:29.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:29.779+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:30.122 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:30.119+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:30.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:30.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:30.274248+0000) 2022-04-03T21:46:30.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:30.733+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:31.169 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:31.166+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:31.201 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:46:31.198+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:46:31.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:31.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:31.274450+0000) 2022-04-03T21:46:31.751 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:31.749+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:32.129 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:32.127+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:32.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:32.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:32.274673+0000) 2022-04-03T21:46:32.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:32.746+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:33.158 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:33.155+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:33.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:33.274+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:33.274859+0000) 2022-04-03T21:46:33.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:33.791+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:34.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:34.150+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:34.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:34.274+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:34.275073+0000) 2022-04-03T21:46:34.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:34.760+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:35.183 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:35.181+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:35.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:35.274+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:35.275298+0000) 2022-04-03T21:46:35.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:35.739+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:36.202 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:36.199+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:36.277 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:36.274+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:36.275540+0000) 2022-04-03T21:46:36.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:36.776+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:37.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:37.175+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:37.277 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:37.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:37.275742+0000) 2022-04-03T21:46:37.826 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:37.823+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:38.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:38.154+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:38.277 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:38.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:38.275954+0000) 2022-04-03T21:46:38.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:38.779+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:39.171 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:39.168+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:39.277 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:39.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:39.276144+0000) 2022-04-03T21:46:39.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:39.739+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:40.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:40.176+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:40.277 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:40.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:40.276310+0000) 2022-04-03T21:46:40.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:40.777+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:41.132 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:41.129+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:41.277 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:41.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:41.276475+0000) 2022-04-03T21:46:41.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:41.772+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:42.151 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:42.149+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:42.278 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:42.276+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:42.276668+0000) 2022-04-03T21:46:42.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:42.770+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:43.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:43.186+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:43.278 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:43.276+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:43.276878+0000) 2022-04-03T21:46:43.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:43.753+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:44.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:44.194+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:44.278 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:44.276+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:44.277088+0000) 2022-04-03T21:46:44.768 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:44.766+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:45.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:45.229+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:45.278 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:45.276+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:45.277289+0000) 2022-04-03T21:46:45.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:45.773+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:46.217 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:46:46.214+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:46:46.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:46.257+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:46.278 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:46.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:46.277513+0000) 2022-04-03T21:46:46.730 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:46.728+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:47.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:47.247+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:47.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:47.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:47.277728+0000) 2022-04-03T21:46:47.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:47.723+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:48.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:48.244+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:48.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:48.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:48.277881+0000) 2022-04-03T21:46:48.492 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:46:48.496 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:46:48.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:48.676+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:49.212 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:49.209+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:49.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:49.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:49.278086+0000) 2022-04-03T21:46:49.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:49.692+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:50.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:50.240+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:50.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:50.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:50.278288+0000) 2022-04-03T21:46:50.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:50.660+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:51.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:51.198+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:51.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:51.278+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:51.278441+0000) 2022-04-03T21:46:51.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:51.668+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:52.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:52.215+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:52.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:52.278+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:52.278644+0000) 2022-04-03T21:46:52.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:52.715+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:53.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:53.186+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:53.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:53.278+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:53.278853+0000) 2022-04-03T21:46:53.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:53.674+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:54.214 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:54.212+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:54.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:54.278+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:54.278998+0000) 2022-04-03T21:46:54.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:54.724+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:55.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:55.248+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:55.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:55.278+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:55.279198+0000) 2022-04-03T21:46:55.769 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:55.767+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:56.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:56.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:56.279384+0000) 2022-04-03T21:46:56.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:56.283+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:56.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:56.805+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:57.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:57.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:57.279542+0000) 2022-04-03T21:46:57.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:57.328+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:57.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:57.819+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:58.281 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:58.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:58.279751+0000) 2022-04-03T21:46:58.376 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:58.375+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:58.797 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:58.796+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:46:59.281 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:46:59.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:46:59.279977+0000) 2022-04-03T21:46:59.396 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:46:59.395+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:46:59.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:46:59.783+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:00.281 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:00.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:00.280185+0000) 2022-04-03T21:47:00.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:00.412+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:00.773 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:00.771+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:01.233 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:47:01.230+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:47:01.281 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:01.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:01.280387+0000) 2022-04-03T21:47:01.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:01.388+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:01.768 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:01.766+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:02.281 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:02.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:02.280536+0000) 2022-04-03T21:47:02.428 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:02.426+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:02.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:02.803+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:03.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:03.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:03.280709+0000) 2022-04-03T21:47:03.460 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:03.458+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:03.833 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:03.832+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:04.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:04.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:04.280900+0000) 2022-04-03T21:47:04.479 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:04.478+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:04.844 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:04.842+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:05.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:05.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:05.281082+0000) 2022-04-03T21:47:05.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:05.457+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:05.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:05.829+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:06.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:06.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:06.281297+0000) 2022-04-03T21:47:06.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:06.460+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:06.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:06.860+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:07.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:07.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:07.281470+0000) 2022-04-03T21:47:07.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:07.465+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:07.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:07.836+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:08.283 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:08.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:08.281676+0000) 2022-04-03T21:47:08.442 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:08.441+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:08.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:08.856+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:09.283 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:09.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:09.281837+0000) 2022-04-03T21:47:09.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:09.446+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:09.837 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:09.835+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:10.283 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:10.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:10.282046+0000) 2022-04-03T21:47:10.412 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:10.410+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:10.792 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:10.790+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:11.283 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:11.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:11.282255+0000) 2022-04-03T21:47:11.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:11.386+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:11.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:11.830+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:12.283 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:12.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:12.282468+0000) 2022-04-03T21:47:12.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:12.345+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:12.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:12.848+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:13.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:13.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:13.282668+0000) 2022-04-03T21:47:13.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:13.324+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:13.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:13.799+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:14.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:14.282+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:14.282846+0000) 2022-04-03T21:47:14.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:14.346+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:14.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:14.778+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:15.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:15.282+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:15.282998+0000) 2022-04-03T21:47:15.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:15.343+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:15.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:15.743+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:16.249 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:47:16.246+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:47:16.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:16.282+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:16.283130+0000) 2022-04-03T21:47:16.375 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:16.372+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:16.738 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:16.735+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:17.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:17.282+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:17.283333+0000) 2022-04-03T21:47:17.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:17.346+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:17.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:17.755+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:18.285 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:18.282+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:18.283521+0000) 2022-04-03T21:47:18.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:18.349+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:18.627 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:47:18.631 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:47:18.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:18.730+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:19.285 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:19.283+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:19.283787+0000) 2022-04-03T21:47:19.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:19.305+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:19.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:19.732+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:20.285 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:20.283+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:20.283998+0000) 2022-04-03T21:47:20.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:20.342+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:20.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:20.691+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:21.285 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:21.283+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:21.284141+0000) 2022-04-03T21:47:21.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:21.311+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:21.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:21.721+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:22.285 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:22.283+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:22.284327+0000) 2022-04-03T21:47:22.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:22.317+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:22.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:22.760+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:23.285 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:23.283+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:23.284519+0000) 2022-04-03T21:47:23.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:23.287+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:23.764 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:23.761+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:24.286 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:24.284+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:24.284718+0000) 2022-04-03T21:47:24.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:24.322+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:24.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:24.800+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:25.286 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:25.284+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:25.284903+0000) 2022-04-03T21:47:25.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:25.295+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:25.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:25.798+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:47:26.286 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:26.284+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:26.285101+0000) 2022-04-03T21:47:26.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:26.289+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:26.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:26.838+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:27.286 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:27.284+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:27.285294+0000) 2022-04-03T21:47:27.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:27.299+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:27.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:27.810+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:28.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:28.258+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:28.286 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:28.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:28.285510+0000) 2022-04-03T21:47:28.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:28.824+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:29.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:29.236+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:29.287 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:29.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:29.285711+0000) 2022-04-03T21:47:29.854 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:29.851+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:30.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:30.284+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:30.287 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:30.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:30.285898+0000) 2022-04-03T21:47:30.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:30.804+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:31.264 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:47:31.262+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:47:31.287 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:31.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:31.286093+0000) 2022-04-03T21:47:31.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:31.316+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:31.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:31.851+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:32.287 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:32.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:32.286313+0000) 2022-04-03T21:47:32.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:32.361+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:32.901 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:32.899+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:33.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:33.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:33.286525+0000) 2022-04-03T21:47:33.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:33.333+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:33.939 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:33.937+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:34.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:34.286+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:34.286741+0000) 2022-04-03T21:47:34.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:34.331+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:34.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:34.955+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:35.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:35.286+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:35.286969+0000) 2022-04-03T21:47:35.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:35.330+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:35.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:35.945+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:36.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:36.286+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:36.287178+0000) 2022-04-03T21:47:36.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:36.306+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:36.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:36.898+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:37.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:37.276+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:37.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:37.287+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:37.287388+0000) 2022-04-03T21:47:37.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:37.870+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:38.289 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:38.287+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:38.287598+0000) 2022-04-03T21:47:38.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:38.294+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:38.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:38.833+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:39.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:39.254+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:39.289 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:39.287+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:39.287834+0000) 2022-04-03T21:47:39.874 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:39.872+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:40.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:40.220+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:40.289 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:40.287+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:40.288041+0000) 2022-04-03T21:47:40.862 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:40.860+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:41.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:41.243+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:41.289 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:41.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:41.288236+0000) 2022-04-03T21:47:41.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:41.893+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:42.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:42.265+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:42.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:42.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:42.288469+0000) 2022-04-03T21:47:42.880 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:42.878+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:43.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:43.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:43.288668+0000) 2022-04-03T21:47:43.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:43.310+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:43.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:43.919+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:44.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:44.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:44.288888+0000) 2022-04-03T21:47:44.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:44.313+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:44.926 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:44.924+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:45.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:45.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:45.289097+0000) 2022-04-03T21:47:45.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:45.359+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:45.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:45.874+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:46.280 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:47:46.277+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:47:46.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:46.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:46.289306+0000) 2022-04-03T21:47:46.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:46.408+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:46.869 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:46.868+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:47.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:47.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:47.289516+0000) 2022-04-03T21:47:47.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:47.387+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:47.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:47.883+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:48.291 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:48.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:48.289750+0000) 2022-04-03T21:47:48.428 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:48.426+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:48.786 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:47:48.789 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:47:48.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:48.846+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:49.291 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:49.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:49.290048+0000) 2022-04-03T21:47:49.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:49.438+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:49.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:49.833+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:50.291 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:50.289+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:50.290208+0000) 2022-04-03T21:47:50.471 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:50.469+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:50.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:50.874+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:51.291 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:51.289+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:51.290413+0000) 2022-04-03T21:47:51.424 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:51.422+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:51.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:51.836+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:52.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:52.289+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:52.290633+0000) 2022-04-03T21:47:52.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:52.385+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:52.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:52.847+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:53.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:53.289+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:53.290823+0000) 2022-04-03T21:47:53.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:53.412+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:53.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:53.837+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:54.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:54.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:54.291005+0000) 2022-04-03T21:47:54.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:54.369+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:54.798 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:54.796+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:55.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:55.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:55.291218+0000) 2022-04-03T21:47:55.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:55.321+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:55.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:55.775+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:56.283 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:56.282+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:56.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:56.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:56.291442+0000) 2022-04-03T21:47:56.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:56.773+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:57.293 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:57.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:57.291653+0000) 2022-04-03T21:47:57.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:57.321+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:57.795 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:57.794+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:58.293 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:58.291+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:58.291843+0000) 2022-04-03T21:47:58.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:58.333+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:58.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:58.819+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:47:59.293 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:47:59.291+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:47:59.292052+0000) 2022-04-03T21:47:59.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:47:59.341+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:47:59.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:47:59.862+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:00.293 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:00.291+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:00.292235+0000) 2022-04-03T21:48:00.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:00.296+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:00.884 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:00.882+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:01.252 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:01.250+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:01.293 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:01.291+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:01.292437+0000) 2022-04-03T21:48:01.296 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:48:01.294+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:48:01.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:01.926+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:02.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:02.271+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:02.293 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:02.291+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:02.292619+0000) 2022-04-03T21:48:02.883 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:02.881+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:03.294 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:03.292+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:03.292826+0000) 2022-04-03T21:48:03.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:03.298+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:03.901 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:03.900+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:04.294 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:04.292+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:04.293023+0000) 2022-04-03T21:48:04.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:04.310+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:04.929 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:04.927+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:05.294 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:05.292+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:05.293257+0000) 2022-04-03T21:48:05.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:05.313+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:05.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:05.972+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:06.295 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:06.292+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:06.293499+0000) 2022-04-03T21:48:06.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:06.308+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:06.967 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:06.966+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:07.295 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:07.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:07.293702+0000) 2022-04-03T21:48:07.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:07.304+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:07.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:07.971+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:08.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:08.290+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:08.295 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:08.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:08.293921+0000) 2022-04-03T21:48:08.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:08.986+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:09.295 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:09.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:09.294131+0000) 2022-04-03T21:48:09.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:09.310+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:09.959 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:09.957+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:10.295 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:10.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:10.294344+0000) 2022-04-03T21:48:10.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:10.340+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:10.936 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:10.934+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:11.296 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:11.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:11.294581+0000) 2022-04-03T21:48:11.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:11.307+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:11.945 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:11.942+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:12.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:12.272+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:12.296 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:12.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:12.294832+0000) 2022-04-03T21:48:12.914 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:12.911+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:13.296 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:13.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:13.295078+0000) 2022-04-03T21:48:13.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:13.310+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:13.930 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:13.927+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:14.296 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:14.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:14.295283+0000) 2022-04-03T21:48:14.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:14.339+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:14.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:14.949+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:15.296 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:15.295+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:15.295444+0000) 2022-04-03T21:48:15.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:15.367+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:15.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:15.930+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:16.297 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:16.295+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:16.295672+0000) 2022-04-03T21:48:16.312 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:48:16.310+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:48:16.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:16.343+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:16.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:16.947+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:17.297 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:17.295+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:17.295877+0000) 2022-04-03T21:48:17.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:17.376+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:17.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:17.977+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:18.297 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:18.295+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:18.296099+0000) 2022-04-03T21:48:18.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:18.378+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:18.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:18.951+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:19.013 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:48:19.017 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:48:19.297 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:19.296+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:19.296366+0000) 2022-04-03T21:48:19.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:19.349+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:19.909 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:19.906+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:20.297 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:20.296+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:20.296539+0000) 2022-04-03T21:48:20.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:20.388+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:20.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:20.947+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:21.298 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:21.296+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:21.296749+0000) 2022-04-03T21:48:21.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:21.414+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:21.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:21.922+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:22.298 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:22.296+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:22.297008+0000) 2022-04-03T21:48:22.438 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:22.436+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:22.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:22.910+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:23.298 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:23.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:23.297239+0000) 2022-04-03T21:48:23.466 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:23.463+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:23.943 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:23.940+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:24.298 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:24.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:24.297414+0000) 2022-04-03T21:48:24.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:24.503+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:24.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:24.945+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:25.299 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:25.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:25.297605+0000) 2022-04-03T21:48:25.500 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:25.495+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:25.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:25.946+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:26.299 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:26.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:26.297833+0000) 2022-04-03T21:48:26.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:26.489+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:26.970 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:26.967+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:27.299 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:27.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:27.298006+0000) 2022-04-03T21:48:27.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:27.528+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:27.929 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:27.926+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:28.299 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:28.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:28.298177+0000) 2022-04-03T21:48:28.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:28.491+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:28.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:28.948+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:29.299 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:29.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:29.298346+0000) 2022-04-03T21:48:29.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:29.467+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:29.935 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:29.932+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:30.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:30.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:30.298540+0000) 2022-04-03T21:48:30.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:30.454+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:30.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:30.956+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:31.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:31.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:31.298670+0000) 2022-04-03T21:48:31.328 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:48:31.325+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:48:31.451 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:31.449+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:31.919 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:31.917+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:32.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:32.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:32.298816+0000) 2022-04-03T21:48:32.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:32.431+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:32.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:32.919+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:33.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:33.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:33.298985+0000) 2022-04-03T21:48:33.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:33.446+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:33.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:33.876+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:34.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:34.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:34.299171+0000) 2022-04-03T21:48:34.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:34.428+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:34.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:34.844+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:35.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:35.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:35.299317+0000) 2022-04-03T21:48:35.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:35.401+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:35.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:35.840+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:36.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:36.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:36.299498+0000) 2022-04-03T21:48:36.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:36.363+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:36.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:36.886+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:37.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:37.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:37.299691+0000) 2022-04-03T21:48:37.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:37.345+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:37.919 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:37.916+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:38.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:38.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:38.299860+0000) 2022-04-03T21:48:38.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:38.380+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:38.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:38.919+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:39.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:39.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:39.300004+0000) 2022-04-03T21:48:39.356 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:39.355+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:39.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:39.953+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:40.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:40.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:40.300191+0000) 2022-04-03T21:48:40.378 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:40.376+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:40.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:40.978+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:41.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:41.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:41.300375+0000) 2022-04-03T21:48:41.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:41.411+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:41.967 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:41.965+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:42.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:42.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:42.300557+0000) 2022-04-03T21:48:42.418 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:42.416+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:42.939 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:42.937+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:43.302 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:43.300+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:43.300752+0000) 2022-04-03T21:48:43.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:43.379+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:43.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:43.948+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:44.302 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:44.300+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:44.300961+0000) 2022-04-03T21:48:44.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:44.414+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:44.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:44.950+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:45.302 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:45.300+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:45.301104+0000) 2022-04-03T21:48:45.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:45.453+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:45.945 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:45.943+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:46.302 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:46.300+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:46.301287+0000) 2022-04-03T21:48:46.328 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:48:46.326+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:48:46.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:46.448+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:46.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:46.989+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:47.303 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:47.300+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:47.301511+0000) 2022-04-03T21:48:47.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:47.486+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:48.002 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:48.000+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:48.303 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:48.301+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:48.301707+0000) 2022-04-03T21:48:48.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:48.511+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:49.051 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:49.050+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:49.162 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:48:49.166 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:48:49.303 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:49.301+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:49.301910+0000) 2022-04-03T21:48:49.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:49.553+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:50.045 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:50.044+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:50.303 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:50.301+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:50.302100+0000) 2022-04-03T21:48:50.533 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:50.532+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:51.072 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:51.070+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:51.303 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:51.301+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:51.302252+0000) 2022-04-03T21:48:51.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:51.503+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:52.068 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:52.066+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:52.303 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:52.301+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:52.302418+0000) 2022-04-03T21:48:52.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:52.486+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:53.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:53.047+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:53.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:53.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:53.302595+0000) 2022-04-03T21:48:53.463 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:53.461+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:54.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:54.001+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:54.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:54.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:54.302813+0000) 2022-04-03T21:48:54.489 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:54.487+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:54.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:54.964+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:55.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:55.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:55.302996+0000) 2022-04-03T21:48:55.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:55.532+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:55.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:55.951+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:56.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:56.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:56.303161+0000) 2022-04-03T21:48:56.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:56.493+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:56.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:56.925+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:57.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:57.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:57.303367+0000) 2022-04-03T21:48:57.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:57.513+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:57.883 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:57.881+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:58.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:58.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:58.303561+0000) 2022-04-03T21:48:58.511 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:58.510+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:58.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:58.849+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:48:59.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:48:59.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:48:59.303812+0000) 2022-04-03T21:48:59.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:48:59.513+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:48:59.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:48:59.808+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:00.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:00.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:00.303986+0000) 2022-04-03T21:49:00.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:00.548+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:00.841 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:00.839+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:01.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:01.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:01.304195+0000) 2022-04-03T21:49:01.333 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:49:01.330+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:49:01.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:01.516+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:01.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:01.870+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:02.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:02.304+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:02.304403+0000) 2022-04-03T21:49:02.503 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:02.500+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:02.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:02.873+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:03.306 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:03.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:03.304630+0000) 2022-04-03T21:49:03.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:03.535+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:03.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:03.876+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:04.306 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:04.304+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:04.304755+0000) 2022-04-03T21:49:04.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:04.494+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:04.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:04.898+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:05.306 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:05.304+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:05.304976+0000) 2022-04-03T21:49:05.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:05.485+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:05.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:05.858+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:06.306 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:06.304+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:06.305181+0000) 2022-04-03T21:49:06.451 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:06.448+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:06.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:06.887+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:07.306 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:07.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:07.305389+0000) 2022-04-03T21:49:07.471 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:07.468+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:07.908 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:07.906+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:08.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:08.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:08.305535+0000) 2022-04-03T21:49:08.463 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:08.460+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:08.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:08.955+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:09.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:09.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:09.305715+0000) 2022-04-03T21:49:09.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:09.443+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:09.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:09.989+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:10.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:10.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:10.305876+0000) 2022-04-03T21:49:10.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:10.403+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:10.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:10.991+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:11.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:11.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:11.306081+0000) 2022-04-03T21:49:11.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:11.413+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:12.011 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:12.009+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:12.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:12.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:12.306270+0000) 2022-04-03T21:49:12.428 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:12.426+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:12.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:12.986+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:13.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:13.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:13.306468+0000) 2022-04-03T21:49:13.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:13.381+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:13.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:13.947+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:14.308 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:14.304+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:14.306617+0000) 2022-04-03T21:49:14.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:14.390+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:14.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:14.950+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:15.308 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:15.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:15.306849+0000) 2022-04-03T21:49:15.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:15.386+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:15.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:15.997+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:16.308 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:16.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:16.307012+0000) 2022-04-03T21:49:16.348 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:49:16.345+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:49:16.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:16.411+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:16.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:16.975+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:17.308 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:17.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:17.307152+0000) 2022-04-03T21:49:17.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:17.393+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:17.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:17.978+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:18.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:18.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:18.307348+0000) 2022-04-03T21:49:18.401 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:18.398+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:18.976 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:18.973+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:19.243 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:49:19.246 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:49:19.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:19.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:19.307530+0000) 2022-04-03T21:49:19.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:19.416+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:19.930 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:19.928+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:20.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:20.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:20.307738+0000) 2022-04-03T21:49:20.424 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:20.422+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:20.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:20.883+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:21.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:21.307+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:21.307912+0000) 2022-04-03T21:49:21.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:21.408+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:21.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:21.850+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:22.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:22.307+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:22.308060+0000) 2022-04-03T21:49:22.378 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:22.376+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:22.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:22.891+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:23.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:23.307+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:23.308256+0000) 2022-04-03T21:49:23.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:23.377+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:23.901 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:23.899+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:24.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:24.307+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:24.308404+0000) 2022-04-03T21:49:24.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:24.380+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:24.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:24.948+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:25.310 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:25.307+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:25.308586+0000) 2022-04-03T21:49:25.421 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:25.419+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:25.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:25.989+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:26.310 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:26.308+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:26.308763+0000) 2022-04-03T21:49:26.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:26.414+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:27.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:27.037+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:27.310 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:27.308+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:27.309007+0000) 2022-04-03T21:49:27.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:27.395+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:28.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:28.060+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:28.310 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:28.308+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:28.309182+0000) 2022-04-03T21:49:28.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:28.425+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:29.017 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:29.015+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:29.310 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:29.308+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:29.309370+0000) 2022-04-03T21:49:29.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:29.391+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:30.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:30.034+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:30.311 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:30.308+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:30.309573+0000) 2022-04-03T21:49:30.364 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:30.362+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:31.017 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:31.014+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:31.311 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:31.309+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:31.309800+0000) 2022-04-03T21:49:31.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:31.358+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:31.364 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:49:31.362+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:49:31.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:31.990+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:32.311 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:32.309+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:32.309980+0000) 2022-04-03T21:49:32.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:32.402+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:33.030 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:33.028+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:33.311 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:33.309+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:33.310172+0000) 2022-04-03T21:49:33.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:33.358+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:34.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:34.021+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:34.311 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:34.309+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:34.310384+0000) 2022-04-03T21:49:34.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:34.387+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:35.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:35.057+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:35.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:35.309+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:35.310609+0000) 2022-04-03T21:49:35.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:35.389+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:36.031 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:36.028+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:36.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:36.310+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:36.310813+0000) 2022-04-03T21:49:36.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:36.387+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:37.069 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:37.068+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:37.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:37.310+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:37.310990+0000) 2022-04-03T21:49:37.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:37.436+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:38.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:38.061+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:38.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:38.310+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:38.311127+0000) 2022-04-03T21:49:38.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:38.445+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:39.053 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:39.051+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:39.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:39.310+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:39.311293+0000) 2022-04-03T21:49:39.472 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:39.469+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:40.043 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:40.041+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:40.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:40.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:40.311474+0000) 2022-04-03T21:49:40.502 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:40.499+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:41.036 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:41.034+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:41.313 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:41.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:41.311694+0000) 2022-04-03T21:49:41.546 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:41.544+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:42.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:42.037+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:42.313 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:42.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:42.311906+0000) 2022-04-03T21:49:42.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:42.588+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:42.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:42.991+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:43.313 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:43.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:43.312145+0000) 2022-04-03T21:49:43.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:43.594+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:44.042 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:44.041+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:44.313 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:44.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:44.312323+0000) 2022-04-03T21:49:44.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:44.576+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:45.056 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:45.054+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:45.314 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:45.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:45.312526+0000) 2022-04-03T21:49:45.570 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:45.568+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:46.032 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:46.030+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:46.314 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:46.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:46.312754+0000) 2022-04-03T21:49:46.368 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:49:46.366+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:49:46.586 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:46.584+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:46.996 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:46.994+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:47.314 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:47.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:47.312993+0000) 2022-04-03T21:49:47.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:47.629+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:47.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:47.989+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:48.314 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:48.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:48.313182+0000) 2022-04-03T21:49:48.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:48.643+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:49.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:49.039+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:49.314 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:49.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:49.313350+0000) 2022-04-03T21:49:49.391 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:49:49.395 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:49:49.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:49.617+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:50.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:50.083+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:50.315 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:50.313+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:50.313583+0000) 2022-04-03T21:49:50.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:50.591+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:51.100 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:51.098+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:51.315 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:51.313+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:51.313825+0000) 2022-04-03T21:49:51.622 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:51.620+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:52.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:52.134+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:52.315 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:52.313+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:52.313981+0000) 2022-04-03T21:49:52.600 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:52.599+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:53.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:53.146+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:53.315 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:53.313+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:53.314119+0000) 2022-04-03T21:49:53.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:53.566+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:54.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:54.122+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:54.316 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:54.314+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:54.314221+0000) 2022-04-03T21:49:54.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:54.522+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:55.155 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:55.153+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:55.315 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:55.314+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:55.314399+0000) 2022-04-03T21:49:55.474 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:55.472+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:56.109 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:56.107+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:56.316 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:56.314+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:56.314571+0000) 2022-04-03T21:49:56.451 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:56.449+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:57.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:57.094+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:57.316 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:57.314+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:57.314755+0000) 2022-04-03T21:49:57.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:57.443+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:58.096 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:58.094+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:58.316 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:58.313+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:58.314964+0000) 2022-04-03T21:49:58.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:58.452+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:49:59.058 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:49:59.056+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:49:59.316 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:49:59.313+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:49:59.315110+0000) 2022-04-03T21:49:59.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:49:59.406+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:00.089 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:00.086+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:00.316 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:00.314+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:00.315289+0000) 2022-04-03T21:50:00.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:00.442+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:01.073 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:01.071+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:01.316 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:01.314+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:01.315406+0000) 2022-04-03T21:50:01.384 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:50:01.381+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:50:01.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:01.492+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:02.064 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:02.061+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:02.317 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:02.314+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:02.315577+0000) 2022-04-03T21:50:02.493 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:02.490+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:03.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:03.056+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:03.317 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:03.314+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:03.315797+0000) 2022-04-03T21:50:03.462 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:03.459+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:04.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:04.058+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:04.317 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:04.315+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:04.315959+0000) 2022-04-03T21:50:04.473 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:04.470+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:05.022 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:05.020+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:05.317 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:05.315+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:05.316092+0000) 2022-04-03T21:50:05.468 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:05.465+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:05.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:05.977+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:06.317 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:06.315+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:06.316286+0000) 2022-04-03T21:50:06.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:06.484+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:06.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:06.947+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:07.318 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:07.315+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:07.316497+0000) 2022-04-03T21:50:07.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:07.484+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:07.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:07.961+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:08.318 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:08.315+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:08.316670+0000) 2022-04-03T21:50:08.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:08.447+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:08.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:08.971+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:09.318 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:09.316+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:09.316835+0000) 2022-04-03T21:50:09.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:09.456+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:09.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:09.963+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:10.318 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:10.316+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:10.316999+0000) 2022-04-03T21:50:10.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:10.481+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:10.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:10.945+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:11.318 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:11.316+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:11.317200+0000) 2022-04-03T21:50:11.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:11.518+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:11.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:11.971+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:12.318 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:12.316+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:12.317391+0000) 2022-04-03T21:50:12.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:12.527+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:12.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:12.996+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:13.319 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:13.316+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:13.317562+0000) 2022-04-03T21:50:13.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:13.555+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:14.034 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:14.032+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:14.319 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:14.317+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:14.317796+0000) 2022-04-03T21:50:14.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:14.550+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:15.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:15.024+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:15.319 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:15.317+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:15.318007+0000) 2022-04-03T21:50:15.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:15.588+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:15.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:15.979+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:16.319 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:16.317+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:16.318176+0000) 2022-04-03T21:50:16.400 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:50:16.398+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:50:16.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:16.574+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:16.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:16.992+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:17.319 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:17.317+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:17.318366+0000) 2022-04-03T21:50:17.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:17.573+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:18.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:18.001+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:18.320 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:18.317+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:18.318581+0000) 2022-04-03T21:50:18.528 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:18.526+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:18.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:18.963+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:19.320 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:19.318+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:19.318794+0000) 2022-04-03T21:50:19.555 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:50:19.558 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:50:19.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:19.560+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:19.961 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:19.959+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:20.320 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:20.318+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:20.319023+0000) 2022-04-03T21:50:20.351 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T21:50:20.349+0000 7f2e43b1f700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T21:50:20.352 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T21:50:20.349+0000 7f2e44320700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T21:50:20.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:20.599+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:20.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:20.982+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:21.320 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:21.318+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:21.319190+0000) 2022-04-03T21:50:21.625 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:21.622+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:21.943 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:21.941+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:22.320 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:22.318+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:22.319371+0000) 2022-04-03T21:50:22.629 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:22.626+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:22.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:22.901+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:23.321 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:23.319+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:23.319574+0000) 2022-04-03T21:50:23.607 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:23.604+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:23.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:23.901+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:24.321 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:24.319+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:24.319760+0000) 2022-04-03T21:50:24.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:24.569+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:24.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:24.873+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:25.321 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:25.319+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:25.319958+0000) 2022-04-03T21:50:25.572 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:25.570+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:25.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:25.851+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T21:50:26.321 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:26.319+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:26.320152+0000) 2022-04-03T21:50:26.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:26.577+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:26.869 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:26.867+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:27.321 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:27.319+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:27.320346+0000) 2022-04-03T21:50:27.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:27.553+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:27.874 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:27.872+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:28.321 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:28.319+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:28.320477+0000) 2022-04-03T21:50:28.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:28.590+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:28.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:28.839+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:29.322 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:29.319+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:29.320602+0000) 2022-04-03T21:50:29.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:29.610+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:29.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:29.886+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:30.322 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:30.320+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:30.320740+0000) 2022-04-03T21:50:30.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:30.633+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:30.852 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:30.850+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:31.322 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:31.320+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:31.320953+0000) 2022-04-03T21:50:31.416 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:50:31.413+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:50:31.670 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:31.668+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:31.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:31.894+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:32.322 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:32.320+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:32.321157+0000) 2022-04-03T21:50:32.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:32.705+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:32.920 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:32.918+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:33.322 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:33.320+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:33.321329+0000) 2022-04-03T21:50:33.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:33.726+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:33.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:33.937+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:34.322 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:34.321+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:34.321462+0000) 2022-04-03T21:50:34.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:34.699+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:34.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:34.895+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:35.323 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:35.321+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:35.321677+0000) 2022-04-03T21:50:35.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:35.746+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:35.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:35.894+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:36.323 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:36.321+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:36.321872+0000) 2022-04-03T21:50:36.783 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:36.781+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:36.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:36.866+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:37.323 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:37.321+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:37.322015+0000) 2022-04-03T21:50:37.763 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:37.761+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:37.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:37.836+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:38.323 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:38.321+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:38.322142+0000) 2022-04-03T21:50:38.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:38.749+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:38.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:38.841+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:39.323 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:39.322+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:39.322346+0000) 2022-04-03T21:50:39.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:39.786+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:39.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:39.880+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:40.324 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:40.322+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:40.322551+0000) 2022-04-03T21:50:40.752 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:40.750+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:40.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:40.903+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:41.324 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:41.322+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:41.322757+0000) 2022-04-03T21:50:41.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:41.755+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:41.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:41.892+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:42.324 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:42.321+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:42.322993+0000) 2022-04-03T21:50:42.762 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:42.760+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:42.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:42.886+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:43.324 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:43.322+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:43.323198+0000) 2022-04-03T21:50:43.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:43.723+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:43.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:43.861+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:44.324 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:44.322+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:44.323318+0000) 2022-04-03T21:50:44.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:44.766+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:44.889 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:44.888+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:45.325 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:45.322+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:45.323499+0000) 2022-04-03T21:50:45.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:45.723+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:45.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:45.916+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:46.325 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:46.321+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:46.323652+0000) 2022-04-03T21:50:46.431 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:50:46.428+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:50:46.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:46.678+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:46.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:46.916+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:47.325 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:47.322+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:47.323857+0000) 2022-04-03T21:50:47.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:47.681+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:47.894 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:47.892+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:48.325 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:48.323+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:48.324071+0000) 2022-04-03T21:50:48.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:48.719+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:48.866 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:48.865+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:49.325 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:49.323+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:49.324212+0000) 2022-04-03T21:50:49.686 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:50:49.690 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:50:49.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:49.734+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:49.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:49.833+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:50.325 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:50.322+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:50.324382+0000) 2022-04-03T21:50:50.694 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:50.692+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:50.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:50.863+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:51.325 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:51.323+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:51.324592+0000) 2022-04-03T21:50:51.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:51.694+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:51.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:51.860+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:52.326 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:52.323+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:52.324736+0000) 2022-04-03T21:50:52.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:52.739+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:52.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:52.908+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:53.326 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:53.324+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:53.324946+0000) 2022-04-03T21:50:53.776 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:53.774+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:53.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:53.901+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:54.326 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:54.324+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:54.325132+0000) 2022-04-03T21:50:54.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:54.753+0000 7f88d0ac1700 -1 osd.4 125 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:54.929 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:54.927+0000 7fd3b190b700 -1 osd.6 125 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:55.326 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:55.324+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:55.325343+0000) 2022-04-03T21:50:55.493 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-03T21:50:55.490+0000 7fb88b33f700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T21:50:55.507 INFO:tasks.ceph.mds.f.smithi186.stderr: -10> 2022-04-03T21:50:55.490+0000 7fb88b33f700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T21:50:55.529 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-03T21:50:55.801 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:55.798+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:55.971 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:55.969+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:56.326 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:56.324+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:56.325494+0000) 2022-04-03T21:50:56.776 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:56.773+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:56.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:56.928+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:57.327 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:57.324+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:57.325642+0000) 2022-04-03T21:50:57.772 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:57.769+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:57.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:57.961+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:58.327 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:58.325+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:58.325820+0000) 2022-04-03T21:50:58.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:58.746+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:58.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:58.939+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:50:59.327 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:50:59.325+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:50:59.326045+0000) 2022-04-03T21:50:59.724 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:50:59.722+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:50:59.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:50:59.980+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:00.327 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:00.325+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:00.326219+0000) 2022-04-03T21:51:00.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:00.771+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:01.023 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:01.020+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:01.327 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:01.325+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:01.326313+0000) 2022-04-03T21:51:01.447 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:51:01.445+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:51:01.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:01.820+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:02.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:02.061+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:02.327 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:02.325+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:02.326458+0000) 2022-04-03T21:51:02.814 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:02.811+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:03.032 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:03.030+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:03.328 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:03.326+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:03.326629+0000) 2022-04-03T21:51:03.807 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:03.805+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:04.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:04.058+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:04.328 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:04.326+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:04.326858+0000) 2022-04-03T21:51:04.781 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:04.779+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:05.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:05.010+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:05.328 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:05.326+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:05.327055+0000) 2022-04-03T21:51:05.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:05.755+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:05.986 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:05.983+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:06.328 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:06.326+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:06.327173+0000) 2022-04-03T21:51:06.794 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:06.791+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:07.001 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:06.998+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:07.328 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:07.326+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:07.327412+0000) 2022-04-03T21:51:07.823 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:07.820+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:07.962 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:07.960+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:08.329 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:08.326+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:08.327580+0000) 2022-04-03T21:51:08.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:08.859+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:08.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:08.955+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:09.329 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:09.327+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:09.327699+0000) 2022-04-03T21:51:09.873 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:09.870+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:09.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:09.986+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:10.329 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:10.327+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:10.327913+0000) 2022-04-03T21:51:10.907 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:10.905+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:10.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:10.995+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:11.329 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:11.327+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:11.328131+0000) 2022-04-03T21:51:11.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:11.865+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:12.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:12.039+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:12.329 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:12.327+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:12.328282+0000) 2022-04-03T21:51:12.908 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:12.906+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:13.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:13.080+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:13.329 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:13.328+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:13.328454+0000) 2022-04-03T21:51:13.923 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:13.921+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:14.044 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:14.042+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:14.330 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:14.328+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:14.328623+0000) 2022-04-03T21:51:14.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:14.907+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:15.043 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:15.040+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:15.330 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:15.328+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:15.328847+0000) 2022-04-03T21:51:15.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:15.931+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:16.036 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:16.034+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:16.330 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:16.328+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:16.329016+0000) 2022-04-03T21:51:16.463 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:51:16.461+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:51:16.900 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:16.898+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:17.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:17.024+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:17.330 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:17.328+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:17.329210+0000) 2022-04-03T21:51:17.879 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:17.877+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:17.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:17.993+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:18.330 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:18.329+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:18.329407+0000) 2022-04-03T21:51:18.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:18.852+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:19.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:19.013+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:19.331 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:19.329+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:19.329623+0000) 2022-04-03T21:51:19.816 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:51:19.820 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:51:19.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:19.860+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:19.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:19.989+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:20.331 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:20.329+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:20.329849+0000) 2022-04-03T21:51:20.848 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:20.846+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:20.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:20.945+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:21.331 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:21.329+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:21.329997+0000) 2022-04-03T21:51:21.799 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:21.797+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:21.909 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:21.907+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:22.331 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:22.329+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:22.330129+0000) 2022-04-03T21:51:22.845 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:22.842+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:22.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:22.918+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:23.331 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:23.330+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:23.330353+0000) 2022-04-03T21:51:23.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:23.853+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:23.963 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:23.960+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:24.331 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:24.330+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:24.330535+0000) 2022-04-03T21:51:24.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:24.867+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:25.011 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:25.009+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:25.332 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:25.330+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:25.330670+0000) 2022-04-03T21:51:25.908 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:25.906+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:25.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:25.994+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:26.332 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:26.330+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:26.330876+0000) 2022-04-03T21:51:26.866 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:26.864+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:26.959 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:26.957+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:27.332 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:27.330+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:27.331095+0000) 2022-04-03T21:51:27.875 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:27.873+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:28.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:28.005+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:28.332 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:28.330+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:28.331238+0000) 2022-04-03T21:51:28.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:28.870+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:29.042 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:29.040+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:29.332 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:29.330+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:29.331421+0000) 2022-04-03T21:51:29.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:29.866+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:30.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:30.005+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:30.333 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:30.330+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:30.331649+0000) 2022-04-03T21:51:30.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:30.822+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:30.968 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:30.966+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:31.333 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:31.330+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:31.331882+0000) 2022-04-03T21:51:31.479 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:51:31.476+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:51:31.850 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:31.849+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:31.962 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:31.960+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:32.333 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:32.331+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:32.332048+0000) 2022-04-03T21:51:32.823 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:32.821+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:32.961 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:32.959+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:33.333 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:33.331+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:33.332241+0000) 2022-04-03T21:51:33.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:33.831+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:33.941 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:33.940+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:34.333 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:34.331+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:34.332370+0000) 2022-04-03T21:51:34.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:34.861+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:34.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:34.972+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:35.334 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:35.331+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:35.332577+0000) 2022-04-03T21:51:35.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:35.896+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:35.932 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:35.930+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:36.334 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:36.332+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:36.332762+0000) 2022-04-03T21:51:36.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:36.876+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:36.944 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:36.943+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:37.334 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:37.332+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:37.332920+0000) 2022-04-03T21:51:37.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:37.870+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:37.920 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:37.919+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:38.334 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:38.332+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:38.333083+0000) 2022-04-03T21:51:38.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:38.907+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:38.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:38.954+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:39.334 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:39.332+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:39.333234+0000) 2022-04-03T21:51:39.866 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:39.865+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:39.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:39.931+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:40.334 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:40.332+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:40.333413+0000) 2022-04-03T21:51:40.893 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:40.892+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:40.980 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:40.978+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:41.335 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:41.332+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:41.333568+0000) 2022-04-03T21:51:41.925 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:41.923+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:41.954 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:41.953+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:42.335 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:42.333+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:42.333754+0000) 2022-04-03T21:51:42.932 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:42.930+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:42.968 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:42.966+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:43.335 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:43.333+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:43.333982+0000) 2022-04-03T21:51:43.886 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:43.884+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:43.969 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:43.968+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:44.335 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:44.333+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:44.334122+0000) 2022-04-03T21:51:44.927 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:44.925+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:44.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:44.997+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:45.335 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:45.333+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:45.334265+0000) 2022-04-03T21:51:45.890 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:45.888+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:46.044 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:46.041+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:46.336 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:46.333+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:46.334438+0000) 2022-04-03T21:51:46.482 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:51:46.480+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:51:46.879 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:46.876+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:47.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:47.071+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:47.335 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:47.334+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:47.334605+0000) 2022-04-03T21:51:47.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:47.913+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:48.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:48.082+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:48.336 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:48.334+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:48.334800+0000) 2022-04-03T21:51:48.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:48.886+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:49.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:49.128+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:49.336 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:49.334+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:49.334998+0000) 2022-04-03T21:51:49.919 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:49.917+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:49.962 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:51:49.966 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:51:50.138 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:50.136+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:50.336 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:50.334+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:50.335145+0000) 2022-04-03T21:51:50.880 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:50.877+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:51.166 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:51.163+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:51.336 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:51.334+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:51.335357+0000) 2022-04-03T21:51:51.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:51.906+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:52.176 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:52.174+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:52.336 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:52.334+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:52.335482+0000) 2022-04-03T21:51:52.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:52.913+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:53.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:53.151+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:53.337 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:53.335+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:53.335657+0000) 2022-04-03T21:51:53.936 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:53.934+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:54.116 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:54.114+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:54.337 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:54.335+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:54.335809+0000) 2022-04-03T21:51:54.887 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:54.884+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:55.162 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:55.159+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:55.337 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:55.335+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:55.335981+0000) 2022-04-03T21:51:55.932 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:55.929+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:56.208 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:56.206+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:56.337 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:56.335+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:56.336184+0000) 2022-04-03T21:51:56.928 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:56.926+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:57.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:57.217+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:57.337 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:57.335+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:57.336352+0000) 2022-04-03T21:51:57.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:57.946+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:58.258 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:58.255+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:58.337 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:58.335+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:58.336497+0000) 2022-04-03T21:51:58.982 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:51:58.980+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:51:59.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:51:59.285+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:51:59.338 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:51:59.336+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:51:59.336698+0000) 2022-04-03T21:52:00.011 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:00.008+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:00.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:00.322+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:00.338 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:00.336+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:00.336883+0000) 2022-04-03T21:52:00.967 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:00.965+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:01.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:01.315+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:01.338 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:01.336+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:01.337091+0000) 2022-04-03T21:52:01.498 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:52:01.495+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:52:01.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:01.943+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:02.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:02.297+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:02.338 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:02.336+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:02.337247+0000) 2022-04-03T21:52:02.991 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:02.988+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:03.270 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:03.268+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:03.338 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:03.337+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:03.337393+0000) 2022-04-03T21:52:04.002 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:03.999+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:04.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:04.293+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:04.339 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:04.337+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:04.337534+0000) 2022-04-03T21:52:04.968 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:04.966+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:05.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:05.283+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:05.339 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:05.337+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:05.337710+0000) 2022-04-03T21:52:05.964 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:05.962+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:06.247 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:06.245+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:06.339 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:06.337+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:06.337847+0000) 2022-04-03T21:52:06.959 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:06.957+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:07.266 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:07.264+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:07.339 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:07.337+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:07.338017+0000) 2022-04-03T21:52:07.938 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:07.935+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:08.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:08.215+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:08.339 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:08.337+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:08.338190+0000) 2022-04-03T21:52:08.916 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:08.913+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:09.204 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:09.202+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:09.339 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:09.338+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:09.338360+0000) 2022-04-03T21:52:09.919 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:09.917+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:10.176 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:10.173+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:10.340 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:10.338+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:10.338538+0000) 2022-04-03T21:52:10.952 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:10.949+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:11.196 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:11.194+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:11.340 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:11.337+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:11.338707+0000) 2022-04-03T21:52:11.928 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:11.925+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:12.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:12.225+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:12.340 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:12.337+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:12.338835+0000) 2022-04-03T21:52:12.913 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:12.910+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:13.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:13.247+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:13.340 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:13.337+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:13.338957+0000) 2022-04-03T21:52:13.940 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:13.937+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:14.210 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:14.207+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:14.340 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:14.338+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:14.339125+0000) 2022-04-03T21:52:14.984 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:14.981+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:15.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:15.204+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:15.340 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:15.338+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:15.339294+0000) 2022-04-03T21:52:16.010 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:16.008+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:16.256 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:16.254+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:16.340 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:16.338+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:16.339433+0000) 2022-04-03T21:52:16.513 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:52:16.510+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:52:17.058 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:17.056+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:17.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:17.263+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:17.340 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:17.338+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:17.339601+0000) 2022-04-03T21:52:18.066 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:18.064+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:18.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:18.270+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:18.341 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:18.338+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:18.339807+0000) 2022-04-03T21:52:19.020 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:19.018+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:19.266 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:19.264+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:19.341 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:19.339+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:19.340025+0000) 2022-04-03T21:52:20.044 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:20.042+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:20.110 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:52:20.115 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:52:20.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:20.275+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:20.341 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:20.339+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:20.340277+0000) 2022-04-03T21:52:21.022 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:21.020+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:21.240 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:21.238+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:21.342 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:21.339+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:21.340433+0000) 2022-04-03T21:52:22.051 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:22.044+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:22.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:22.233+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:22.342 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:22.339+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:22.340587+0000) 2022-04-03T21:52:23.017 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:23.014+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:23.274 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:23.272+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:23.342 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:23.340+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:23.340756+0000) 2022-04-03T21:52:24.008 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:24.007+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:24.224 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:24.222+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:24.342 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:24.340+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:24.340948+0000) 2022-04-03T21:52:25.001 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:25.000+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:25.180 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:25.178+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:25.342 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:25.340+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:25.341107+0000) 2022-04-03T21:52:26.014 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:26.012+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:26.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:26.133+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:26.342 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:26.340+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:26.341278+0000) 2022-04-03T21:52:26.993 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:26.991+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:27.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:27.182+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:27.343 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:27.340+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:27.341499+0000) 2022-04-03T21:52:27.972 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:27.971+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:28.150 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:28.149+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:28.343 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:28.340+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:28.341638+0000) 2022-04-03T21:52:28.977 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:28.976+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:29.142 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:29.140+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:29.343 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:29.341+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:29.341857+0000) 2022-04-03T21:52:29.968 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:29.966+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:30.104 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:30.103+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:30.343 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:30.341+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:30.342046+0000) 2022-04-03T21:52:31.005 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:31.003+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:31.098 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:31.096+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:31.343 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:31.341+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:31.342279+0000) 2022-04-03T21:52:31.529 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:52:31.527+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:52:31.981 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:31.980+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:32.089 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:32.088+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:32.344 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:32.341+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:32.342475+0000) 2022-04-03T21:52:32.989 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:32.987+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:33.089 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:33.087+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:33.344 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:33.342+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:33.342665+0000) 2022-04-03T21:52:33.980 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:33.978+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:34.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:34.069+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:34.344 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:34.342+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:34.342831+0000) 2022-04-03T21:52:35.009 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:35.007+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:35.084 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:35.083+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:35.344 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:35.342+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:35.343034+0000) 2022-04-03T21:52:35.972 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:35.970+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:36.096 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:36.094+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:36.344 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:36.342+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:36.343210+0000) 2022-04-03T21:52:36.929 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:36.928+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:37.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:37.135+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:37.344 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:37.342+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:37.343351+0000) 2022-04-03T21:52:37.978 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:37.976+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:38.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:38.162+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:38.344 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:38.343+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:38.343487+0000) 2022-04-03T21:52:38.962 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:38.960+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:39.116 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:39.115+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:39.345 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:39.343+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:39.343639+0000) 2022-04-03T21:52:39.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:39.995+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:40.137 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:40.136+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:40.345 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:40.343+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:40.343852+0000) 2022-04-03T21:52:41.032 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:41.030+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:41.106 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:41.104+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:41.345 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:41.343+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:41.343955+0000) 2022-04-03T21:52:42.069 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:42.066+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:42.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:42.141+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:42.345 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:42.343+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:42.344154+0000) 2022-04-03T21:52:43.028 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:43.025+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:43.105 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:43.103+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:43.345 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:43.344+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:43.344365+0000) 2022-04-03T21:52:44.066 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:44.063+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:44.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:44.077+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:44.346 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:44.344+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:44.344552+0000) 2022-04-03T21:52:45.039 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:45.035+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:45.158 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:45.101+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:45.346 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:45.344+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:45.344732+0000) 2022-04-03T21:52:46.049 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:46.047+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:46.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:46.063+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:46.346 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:46.344+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:46.344927+0000) 2022-04-03T21:52:46.545 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:52:46.543+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:52:47.014 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:47.011+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:47.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:47.077+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:47.346 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:47.344+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:47.345125+0000) 2022-04-03T21:52:48.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:48.013+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:48.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:48.071+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:48.346 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:48.344+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:48.345285+0000) 2022-04-03T21:52:49.036 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:49.033+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:49.065 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:49.062+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:49.346 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:49.345+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:49.345462+0000) 2022-04-03T21:52:50.049 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:50.046+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:50.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:50.081+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:50.215 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:52:50.230 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:52:50.347 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:50.344+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:50.345685+0000) 2022-04-03T21:52:51.084 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:51.081+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:51.086 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:51.084+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:51.347 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:51.345+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:51.345857+0000) 2022-04-03T21:52:52.062 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:52.059+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:52.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:52.080+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:52.347 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:52.345+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:52.346028+0000) 2022-04-03T21:52:53.104 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:53.101+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:53.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:53.105+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:53.347 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:53.345+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:53.346209+0000) 2022-04-03T21:52:54.060 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:54.057+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:54.129 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:54.127+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:54.347 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:54.346+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:54.346358+0000) 2022-04-03T21:52:55.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:55.052+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:55.133 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:55.130+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:55.348 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:55.345+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:55.346564+0000) 2022-04-03T21:52:56.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:56.102+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:56.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:56.180+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:56.348 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:56.345+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:56.346689+0000) 2022-04-03T21:52:57.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:57.071+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:57.180 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:57.178+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:57.348 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:57.345+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:57.346887+0000) 2022-04-03T21:52:58.069 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:58.066+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:58.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:58.137+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:58.348 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:58.346+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:58.347045+0000) 2022-04-03T21:52:59.042 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:59.040+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:52:59.162 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:52:59.160+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:52:59.348 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:52:59.346+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:52:59.347277+0000) 2022-04-03T21:52:59.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:52:59.992+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:00.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:00.140+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:00.348 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:00.346+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:00.347481+0000) 2022-04-03T21:53:01.036 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:01.034+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:01.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:01.165+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:01.349 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:01.346+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:01.347662+0000) 2022-04-03T21:53:01.561 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:53:01.558+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:53:02.046 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:02.044+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:02.145 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:02.143+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:02.349 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:02.346+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:02.347820+0000) 2022-04-03T21:53:03.071 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:03.068+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:03.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:03.193+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:03.349 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:03.347+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:03.348014+0000) 2022-04-03T21:53:04.114 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:04.112+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:04.190 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:04.188+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:04.349 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:04.347+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:04.348177+0000) 2022-04-03T21:53:05.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:05.148+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:05.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:05.184+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:05.349 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:05.347+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:05.348328+0000) 2022-04-03T21:53:06.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:06.181+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:06.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:06.230+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:06.350 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:06.347+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:06.348481+0000) 2022-04-03T21:53:07.144 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:07.142+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:07.208 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:07.206+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:07.350 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:07.347+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:07.348630+0000) 2022-04-03T21:53:08.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:08.171+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:08.208 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:08.206+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:08.350 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:08.348+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:08.348830+0000) 2022-04-03T21:53:09.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:09.175+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:09.233 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:09.231+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:09.350 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:09.348+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:09.348998+0000) 2022-04-03T21:53:10.143 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:10.141+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:10.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:10.266+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:10.350 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:10.348+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:10.349168+0000) 2022-04-03T21:53:11.165 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:11.163+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:11.230 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:11.227+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:11.350 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:11.348+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:11.349364+0000) 2022-04-03T21:53:12.195 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:12.193+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:12.214 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:12.212+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:12.350 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:12.348+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:12.349518+0000) 2022-04-03T21:53:13.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:13.195+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:13.216 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:13.215+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:13.351 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:13.348+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:13.349642+0000) 2022-04-03T21:53:14.233 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:14.230+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:14.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:14.255+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:14.351 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:14.349+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:14.349842+0000) 2022-04-03T21:53:15.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:15.252+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:15.269 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:15.267+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:15.351 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:15.349+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:15.350009+0000) 2022-04-03T21:53:16.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:16.216+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:16.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:16.224+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:16.351 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:16.349+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:16.350188+0000) 2022-04-03T21:53:16.577 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:53:16.574+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:53:17.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:17.194+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:17.253 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:17.250+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:17.351 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:17.349+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:17.350348+0000) 2022-04-03T21:53:18.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:18.186+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:18.214 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:18.212+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:18.352 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:18.349+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:18.350532+0000) 2022-04-03T21:53:19.174 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:19.172+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:19.233 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:19.231+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:19.352 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:19.350+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:19.350677+0000) 2022-04-03T21:53:20.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:20.194+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:20.238 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:20.236+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:20.352 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:20.350+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:20.350854+0000) 2022-04-03T21:53:20.372 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:53:20.375 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:53:21.171 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:21.170+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:21.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:21.228+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:21.352 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:21.350+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:21.351043+0000) 2022-04-03T21:53:22.170 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:22.168+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:22.269 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:22.268+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:22.352 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:22.350+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:22.351144+0000) 2022-04-03T21:53:23.140 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:23.138+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:23.252 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:23.250+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:23.352 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:23.350+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:23.351316+0000) 2022-04-03T21:53:24.110 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:24.109+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:24.253 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:24.251+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:24.353 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:24.350+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:24.351485+0000) 2022-04-03T21:53:25.101 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:25.099+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:25.216 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:25.214+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:25.353 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:25.351+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:25.351630+0000) 2022-04-03T21:53:26.065 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:26.063+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:26.206 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:26.205+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:26.353 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:26.351+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:26.351822+0000) 2022-04-03T21:53:27.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:27.110+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:27.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:27.172+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:27.353 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:27.351+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:27.351971+0000) 2022-04-03T21:53:28.130 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:28.129+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:28.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:28.204+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:28.353 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:28.351+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:28.352076+0000) 2022-04-03T21:53:29.116 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:29.114+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:29.243 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:29.241+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:29.353 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:29.351+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:29.352267+0000) 2022-04-03T21:53:30.088 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:30.086+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:30.212 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:30.210+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:30.353 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:30.352+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:30.352447+0000) 2022-04-03T21:53:31.047 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:31.045+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:31.236 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:31.235+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:31.354 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:31.352+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:31.352663+0000) 2022-04-03T21:53:31.592 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:53:31.590+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:53:32.018 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:32.016+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:32.269 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:32.268+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:32.354 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:32.352+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:32.352833+0000) 2022-04-03T21:53:33.010 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:33.008+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:33.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:33.297+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:33.354 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:33.352+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:33.353006+0000) 2022-04-03T21:53:33.997 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:33.995+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:34.327 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:34.325+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:34.354 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:34.352+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:34.353197+0000) 2022-04-03T21:53:35.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:35.014+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:35.290 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:35.287+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:35.354 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:35.353+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:35.353410+0000) 2022-04-03T21:53:36.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:36.009+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:36.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:36.283+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:36.355 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:36.353+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:36.353564+0000) 2022-04-03T21:53:36.989 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:36.986+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:37.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:37.293+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:37.355 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:37.353+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:37.353698+0000) 2022-04-03T21:53:37.968 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:37.965+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:38.325 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:38.322+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:38.355 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:38.353+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:38.353894+0000) 2022-04-03T21:53:38.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:38.945+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:39.355 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:39.354+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:39.354116+0000) 2022-04-03T21:53:39.366 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:39.364+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:39.997 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:39.995+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:40.355 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:40.353+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:40.354294+0000) 2022-04-03T21:53:40.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:40.397+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:41.009 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:41.006+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:41.356 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:41.352+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:41.354478+0000) 2022-04-03T21:53:41.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:41.428+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:41.962 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:41.959+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:42.356 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:42.353+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:42.354673+0000) 2022-04-03T21:53:42.477 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:42.474+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:42.980 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:42.978+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:43.356 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:43.353+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:43.354877+0000) 2022-04-03T21:53:43.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:43.449+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:44.007 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:44.005+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:44.356 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:44.354+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:44.355023+0000) 2022-04-03T21:53:44.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:44.486+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:45.040 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:45.038+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:45.356 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:45.354+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:45.355218+0000) 2022-04-03T21:53:45.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:45.453+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:45.993 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:45.991+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:46.356 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:46.353+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:46.355343+0000) 2022-04-03T21:53:46.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:46.408+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:46.608 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:53:46.606+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:53:47.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:47.035+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:47.357 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:47.354+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:47.355545+0000) 2022-04-03T21:53:47.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:47.412+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:47.990 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:47.988+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:48.357 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:48.354+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:48.355694+0000) 2022-04-03T21:53:48.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:48.374+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:48.963 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:48.960+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:49.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:49.327+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:49.357 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:49.355+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:49.355875+0000) 2022-04-03T21:53:49.987 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:49.985+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:50.329 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:50.326+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:50.357 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:50.355+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:50.356069+0000) 2022-04-03T21:53:50.493 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:53:50.497 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:53:50.939 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:50.936+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:51.308 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:51.305+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:51.357 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:51.355+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:51.356280+0000) 2022-04-03T21:53:51.961 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:51.959+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:52.334 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:52.332+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:52.357 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:52.355+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:52.356481+0000) 2022-04-03T21:53:52.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:52.912+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:53.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:53.300+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:53.358 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:53.355+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:53.356723+0000) 2022-04-03T21:53:53.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:53.865+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:54.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:54.294+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:54.358 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:54.356+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:54.356945+0000) 2022-04-03T21:53:54.881 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:54.879+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:55.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:55.322+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:55.358 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:55.356+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:55.357143+0000) 2022-04-03T21:53:55.905 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:55.903+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:56.304 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:56.302+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:56.358 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:56.356+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:56.357371+0000) 2022-04-03T21:53:56.912 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:56.910+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:57.287 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:57.284+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:57.359 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:57.356+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:57.357506+0000) 2022-04-03T21:53:57.883 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:57.881+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:58.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:58.296+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:58.358 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:58.356+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:58.357584+0000) 2022-04-03T21:53:58.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:58.866+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:53:59.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:53:59.269+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:53:59.359 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:53:59.357+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:53:59.357735+0000) 2022-04-03T21:53:59.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:53:59.861+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:00.271 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:00.268+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:00.359 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:00.357+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:00.357939+0000) 2022-04-03T21:54:00.902 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:00.900+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:01.298 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:01.295+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:01.359 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:01.357+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:01.358095+0000) 2022-04-03T21:54:01.624 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:54:01.622+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:54:01.886 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:01.884+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:02.267 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:02.265+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:02.359 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:02.357+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:02.358268+0000) 2022-04-03T21:54:02.871 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:02.868+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:03.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:03.294+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:03.359 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:03.357+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:03.358470+0000) 2022-04-03T21:54:03.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:03.820+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:04.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:04.257+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:04.360 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:04.357+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:04.358616+0000) 2022-04-03T21:54:04.857 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:04.854+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:05.258 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:05.256+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:05.360 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:05.358+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:05.358833+0000) 2022-04-03T21:54:05.827 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:05.825+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:06.293 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:06.291+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:06.360 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:06.358+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:06.359000+0000) 2022-04-03T21:54:06.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:06.859+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:07.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:07.307+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:07.360 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:07.358+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:07.359207+0000) 2022-04-03T21:54:07.867 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:07.866+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:08.293 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:08.291+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:08.360 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:08.358+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:08.359390+0000) 2022-04-03T21:54:08.841 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:08.838+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:09.289 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:09.287+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:09.361 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:09.359+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:09.359597+0000) 2022-04-03T21:54:09.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:09.827+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:10.293 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:10.291+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:10.361 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:10.359+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:10.359766+0000) 2022-04-03T21:54:10.844 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:10.841+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:11.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:11.313+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:11.361 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:11.359+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:11.359978+0000) 2022-04-03T21:54:11.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:11.813+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:12.318 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:12.315+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:12.361 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:12.359+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:12.360212+0000) 2022-04-03T21:54:12.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:12.841+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:13.322 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:13.320+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:13.361 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:13.360+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:13.360427+0000) 2022-04-03T21:54:13.891 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:13.890+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:14.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:14.348+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:14.362 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:14.360+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:14.360621+0000) 2022-04-03T21:54:14.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:14.907+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:15.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:15.315+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:15.362 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:15.360+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:15.360840+0000) 2022-04-03T21:54:15.918 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:15.917+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:16.333 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:16.332+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:16.362 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:16.360+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:16.361027+0000) 2022-04-03T21:54:16.640 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:54:16.637+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:54:16.881 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:16.879+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:17.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:17.296+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:17.362 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:17.361+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:17.361245+0000) 2022-04-03T21:54:17.898 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:17.897+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:18.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:18.286+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:18.363 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:18.361+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:18.361479+0000) 2022-04-03T21:54:18.922 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:18.920+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:19.332 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:19.330+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:19.363 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:19.361+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:19.361686+0000) 2022-04-03T21:54:19.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:19.876+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:20.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:20.290+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:20.363 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:20.361+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:20.361860+0000) 2022-04-03T21:54:20.632 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:54:20.635 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:54:20.884 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:20.882+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:21.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:21.255+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:21.363 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:21.361+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:21.362075+0000) 2022-04-03T21:54:21.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:21.907+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:22.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:22.246+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:22.363 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:22.362+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:22.362285+0000) 2022-04-03T21:54:22.934 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:22.932+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:23.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:23.210+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:23.364 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:23.362+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:23.362488+0000) 2022-04-03T21:54:23.944 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:23.943+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:24.169 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:24.167+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:24.364 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:24.361+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:24.362708+0000) 2022-04-03T21:54:24.928 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:24.927+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:25.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:25.177+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:25.364 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:25.361+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:25.362908+0000) 2022-04-03T21:54:25.890 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:25.888+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:26.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:26.199+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:26.364 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:26.362+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:26.363157+0000) 2022-04-03T21:54:26.894 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:26.892+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:27.190 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:27.189+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:27.364 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:27.362+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:27.363357+0000) 2022-04-03T21:54:27.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:27.888+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:28.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:28.218+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:28.365 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:28.362+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:28.363576+0000) 2022-04-03T21:54:28.892 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:28.890+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:29.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:29.258+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:29.365 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:29.362+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:29.363796+0000) 2022-04-03T21:54:29.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:29.887+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:30.224 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:30.221+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:30.365 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:30.363+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:30.363996+0000) 2022-04-03T21:54:30.854 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:30.852+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:31.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:31.271+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:31.365 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:31.363+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:31.364222+0000) 2022-04-03T21:54:31.655 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:54:31.652+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:54:31.903 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:31.900+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:32.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:32.307+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:32.365 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:32.363+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:32.364427+0000) 2022-04-03T21:54:32.923 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:32.920+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:33.356 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:33.353+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:33.365 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:33.363+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:33.364632+0000) 2022-04-03T21:54:33.914 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:33.911+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:34.364 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:34.362+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:34.366 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:34.364+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:34.364820+0000) 2022-04-03T21:54:34.891 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:34.888+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:35.366 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:35.364+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:35.365023+0000) 2022-04-03T21:54:35.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:35.373+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:35.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:35.886+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:36.367 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:36.364+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:36.365242+0000) 2022-04-03T21:54:36.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:36.420+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:36.918 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:36.916+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:37.367 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:37.364+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:37.365452+0000) 2022-04-03T21:54:37.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:37.391+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:37.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:37.942+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:38.360 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:38.357+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:38.367 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:38.364+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:38.365658+0000) 2022-04-03T21:54:38.969 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:38.966+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:39.367 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:39.365+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:39.365846+0000) 2022-04-03T21:54:39.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:39.371+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:39.947 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:39.945+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:40.367 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:40.365+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:40.366079+0000) 2022-04-03T21:54:40.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:40.413+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:40.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:40.945+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:41.367 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:41.365+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:41.366252+0000) 2022-04-03T21:54:41.386 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:41.384+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:41.993 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:41.990+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:42.367 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:42.365+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:42.366442+0000) 2022-04-03T21:54:42.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:42.366+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:42.958 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:42.955+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:43.368 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:43.365+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:43.366666+0000) 2022-04-03T21:54:43.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:43.415+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:43.988 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:43.986+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:44.368 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:44.366+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:44.366897+0000) 2022-04-03T21:54:44.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:44.407+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:44.987 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:44.984+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:45.368 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:45.366+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:45.367122+0000) 2022-04-03T21:54:45.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:45.392+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:46.030 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:46.028+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:46.368 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:46.366+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:46.367338+0000) 2022-04-03T21:54:46.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:46.378+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:46.663 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:54:46.661+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:54:47.037 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:47.035+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:47.369 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:47.366+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:47.367571+0000) 2022-04-03T21:54:47.369 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:47.367+0000 7fd623a1a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T21:54:47.370 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:47.367+0000 7fd623a1a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T21:54:47.373 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:47.371+0000 7fd62421b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:47.371884+0000) 2022-04-03T21:54:47.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:47.392+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:48.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:48.035+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:48.369 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:48.367+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:48.367714+0000) 2022-04-03T21:54:48.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:48.413+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:49.039 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:49.036+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:49.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:49.365+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:49.369 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:49.367+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:49.367854+0000) 2022-04-03T21:54:50.007 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:50.004+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:50.369 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:50.367+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:50.368045+0000) 2022-04-03T21:54:50.374 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:50.372+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:50.774 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:54:50.778 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:54:50.972 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:50.970+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:51.369 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:51.367+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:51.368218+0000) 2022-04-03T21:54:51.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:51.371+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:52.002 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:52.000+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:52.369 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:52.367+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:52.368364+0000) 2022-04-03T21:54:52.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:52.421+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:53.021 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:53.019+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:53.370 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:53.368+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:53.368577+0000) 2022-04-03T21:54:53.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:53.428+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:54.039 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:54.036+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:54.370 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:54.368+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:54.368797+0000) 2022-04-03T21:54:54.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:54.423+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:55.024 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:55.022+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:55.370 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:55.368+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:55.369023+0000) 2022-04-03T21:54:55.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:55.376+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:56.065 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:56.063+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:56.370 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:56.368+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:56.369181+0000) 2022-04-03T21:54:56.395 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:56.392+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:57.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:57.036+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:57.370 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:57.369+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:57.369312+0000) 2022-04-03T21:54:57.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:57.380+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:58.001 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:57.999+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:58.371 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:58.369+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:58.369443+0000) 2022-04-03T21:54:58.374 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:58.372+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:58.980 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:58.978+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:54:59.371 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:54:59.369+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:54:59.369633+0000) 2022-04-03T21:54:59.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:54:59.421+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:54:59.937 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:54:59.935+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:00.371 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:00.369+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:00.369851+0000) 2022-04-03T21:55:00.432 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:00.430+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:00.902 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:00.899+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:01.371 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:01.369+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:01.370026+0000) 2022-04-03T21:55:01.466 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:01.464+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:01.679 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:55:01.677+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:55:01.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:01.866+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:02.371 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:02.370+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:02.370188+0000) 2022-04-03T21:55:02.420 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:02.418+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:02.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:02.820+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:03.371 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:03.370+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:03.370334+0000) 2022-04-03T21:55:03.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:03.374+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:03.830 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:03.828+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:04.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:04.350+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:04.371 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:04.370+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:04.370437+0000) 2022-04-03T21:55:04.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:04.841+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:05.306 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:05.304+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:05.372 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:05.370+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:05.370597+0000) 2022-04-03T21:55:05.871 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:05.868+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:06.319 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:06.318+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:06.372 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:06.370+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:06.370747+0000) 2022-04-03T21:55:06.849 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:06.846+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:07.337 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:07.335+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:07.372 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:07.370+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:07.370911+0000) 2022-04-03T21:55:07.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:07.876+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:08.290 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:08.288+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:08.372 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:08.370+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:08.371083+0000) 2022-04-03T21:55:08.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:08.861+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:09.304 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:09.303+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:09.372 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:09.370+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:09.371262+0000) 2022-04-03T21:55:09.860 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:09.858+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:10.308 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:10.306+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:10.372 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:10.370+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:10.371464+0000) 2022-04-03T21:55:10.892 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:10.891+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:11.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:11.355+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:11.373 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:11.370+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:11.371665+0000) 2022-04-03T21:55:11.848 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:11.847+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:12.355 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:12.353+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:12.373 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:12.370+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:12.371868+0000) 2022-04-03T21:55:12.879 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:12.877+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:13.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:13.350+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:13.373 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:13.371+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:13.372060+0000) 2022-04-03T21:55:13.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:13.862+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:14.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:14.350+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:14.373 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:14.370+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:14.372204+0000) 2022-04-03T21:55:14.847 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:14.846+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:15.373 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:15.371+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:15.372345+0000) 2022-04-03T21:55:15.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:15.392+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:15.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:15.841+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:16.373 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:16.371+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:16.372446+0000) 2022-04-03T21:55:16.408 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:16.407+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:16.695 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:55:16.692+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:55:16.827 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:16.825+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:17.374 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:17.371+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:17.372582+0000) 2022-04-03T21:55:17.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:17.435+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:17.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:17.822+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:18.374 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:18.371+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:18.372726+0000) 2022-04-03T21:55:18.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:18.425+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:18.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:18.832+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:19.374 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:19.372+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:19.372879+0000) 2022-04-03T21:55:19.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:19.391+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:19.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:19.792+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:20.374 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:20.372+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:20.373078+0000) 2022-04-03T21:55:20.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:20.405+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:20.784 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:20.782+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:20.855 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:55:20.859 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:55:21.374 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:21.372+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:21.373284+0000) 2022-04-03T21:55:21.449 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:21.447+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:21.819 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:21.818+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:22.374 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:22.372+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:22.373437+0000) 2022-04-03T21:55:22.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:22.407+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:22.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:22.783+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:23.375 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:23.372+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:23.373593+0000) 2022-04-03T21:55:23.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:23.414+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:23.790 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:23.789+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:24.375 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:24.373+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:24.373765+0000) 2022-04-03T21:55:24.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:24.375+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:24.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:24.752+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:25.375 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:25.373+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:25.373919+0000) 2022-04-03T21:55:25.386 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:25.384+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:25.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:25.745+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:26.374 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:26.371+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:26.375 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:26.373+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:26.374073+0000) 2022-04-03T21:55:26.762 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:26.759+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:27.337 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:27.334+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:27.375 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:27.373+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:27.374246+0000) 2022-04-03T21:55:27.746 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:27.744+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:28.375 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:28.373+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:28.374314+0000) 2022-04-03T21:55:28.384 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:28.381+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:28.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:28.747+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:29.376 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:29.373+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:29.374465+0000) 2022-04-03T21:55:29.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:29.405+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:29.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:29.720+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:30.376 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:30.374+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:30.374621+0000) 2022-04-03T21:55:30.408 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:30.405+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:30.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:30.729+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:31.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:31.368+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:31.376 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:31.374+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:31.374809+0000) 2022-04-03T21:55:31.711 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:55:31.708+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:55:31.717 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:31.714+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:32.376 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:32.374+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:32.374946+0000) 2022-04-03T21:55:32.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:32.378+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:32.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:32.722+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:33.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:33.373+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:33.376 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:33.374+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:33.375093+0000) 2022-04-03T21:55:33.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:33.769+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:34.376 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:34.374+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:34.375265+0000) 2022-04-03T21:55:34.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:34.386+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:34.817 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:34.814+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:35.376 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:35.374+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:35.375421+0000) 2022-04-03T21:55:35.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:35.397+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:35.860 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:35.857+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:36.377 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:36.374+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:36.375594+0000) 2022-04-03T21:55:36.410 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:36.408+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:36.831 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:36.828+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:37.377 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:37.375+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:37.375818+0000) 2022-04-03T21:55:37.386 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:37.383+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:37.805 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:37.802+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:38.377 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:38.375+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:38.376028+0000) 2022-04-03T21:55:38.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:38.406+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:38.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:38.782+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:39.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:39.374+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:39.377 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:39.375+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:39.376199+0000) 2022-04-03T21:55:39.827 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:39.824+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:40.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:40.351+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:40.377 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:40.376+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:40.376401+0000) 2022-04-03T21:55:40.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:40.793+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:41.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:41.307+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:41.378 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:41.376+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:41.376583+0000) 2022-04-03T21:55:41.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:41.784+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:42.269 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:42.266+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:42.378 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:42.376+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:42.376812+0000) 2022-04-03T21:55:42.828 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:42.826+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:43.312 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:43.310+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:43.378 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:43.376+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:43.376995+0000) 2022-04-03T21:55:43.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:43.874+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:44.290 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:44.287+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:44.378 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:44.376+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:44.377171+0000) 2022-04-03T21:55:44.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:44.860+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:45.251 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:45.249+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:45.378 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:45.377+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:45.377344+0000) 2022-04-03T21:55:45.860 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:45.857+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:46.220 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:46.218+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:46.379 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:46.377+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:46.377548+0000) 2022-04-03T21:55:46.726 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:55:46.724+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:55:46.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:46.894+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:47.208 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:47.205+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:47.379 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:47.377+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:47.377716+0000) 2022-04-03T21:55:47.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:47.875+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:48.193 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:48.191+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:48.379 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:48.377+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:48.377866+0000) 2022-04-03T21:55:48.907 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:48.904+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:49.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:49.204+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:49.379 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:49.377+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:49.377967+0000) 2022-04-03T21:55:49.951 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:49.949+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:50.241 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:50.239+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:50.379 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:50.378+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:50.378101+0000) 2022-04-03T21:55:50.927 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:50.925+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:50.981 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:55:50.985 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:55:51.223 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:51.221+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:51.379 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:51.378+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:51.378260+0000) 2022-04-03T21:55:51.927 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:51.925+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:52.251 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:52.249+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:52.380 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:52.377+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:52.378464+0000) 2022-04-03T21:55:52.929 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:52.927+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:53.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:53.279+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:53.380 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:53.377+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:53.378646+0000) 2022-04-03T21:55:53.968 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:53.966+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:54.233 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:54.230+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:54.380 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:54.377+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:54.378791+0000) 2022-04-03T21:55:55.005 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:55.003+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:55.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:55.258+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:55.380 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:55.377+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:55.378949+0000) 2022-04-03T21:55:55.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:55.977+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:56.221 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:56.219+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:56.380 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:56.378+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:56.379138+0000) 2022-04-03T21:55:56.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:56.951+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:57.230 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:57.228+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:57.380 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:57.378+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:57.379291+0000) 2022-04-03T21:55:57.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:57.946+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:58.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:58.215+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:58.380 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:58.378+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:58.379415+0000) 2022-04-03T21:55:58.903 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:58.901+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:55:59.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:55:59.199+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:55:59.381 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:55:59.378+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:55:59.379558+0000) 2022-04-03T21:55:59.913 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:55:59.910+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:00.251 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:00.249+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:00.381 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:00.378+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:00.379695+0000) 2022-04-03T21:56:00.895 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:00.893+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:01.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:01.227+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:01.381 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:01.379+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:01.379888+0000) 2022-04-03T21:56:01.742 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:56:01.739+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:56:01.873 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:01.871+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:02.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:02.196+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:02.381 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:02.379+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:02.380037+0000) 2022-04-03T21:56:02.856 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:02.854+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:03.224 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:03.223+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:03.381 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:03.379+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:03.380181+0000) 2022-04-03T21:56:03.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:03.861+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:04.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:04.253+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:04.381 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:04.379+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:04.380338+0000) 2022-04-03T21:56:04.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:04.881+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:05.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:05.218+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:05.382 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:05.379+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:05.380525+0000) 2022-04-03T21:56:05.858 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:05.856+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:06.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:06.230+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:06.382 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:06.379+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:06.380707+0000) 2022-04-03T21:56:06.881 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:06.879+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:07.220 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:07.219+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:07.382 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:07.380+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:07.380932+0000) 2022-04-03T21:56:07.924 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:07.923+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:08.216 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:08.215+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:08.382 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:08.380+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:08.381062+0000) 2022-04-03T21:56:08.928 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:08.927+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:09.233 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:09.231+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:09.382 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:09.380+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:09.381237+0000) 2022-04-03T21:56:09.914 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:09.912+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:10.240 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:10.238+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:10.383 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:10.380+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:10.381447+0000) 2022-04-03T21:56:10.941 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:10.939+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:11.216 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:11.214+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:11.384 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:11.381+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:11.381641+0000) 2022-04-03T21:56:11.991 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:11.989+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:12.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:12.201+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:12.383 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:12.381+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:12.381832+0000) 2022-04-03T21:56:12.982 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:12.980+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:13.186 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:13.185+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:13.383 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:13.381+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:13.382013+0000) 2022-04-03T21:56:13.944 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:13.943+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:14.144 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:14.142+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:14.383 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:14.381+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:14.382151+0000) 2022-04-03T21:56:14.989 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:14.987+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:15.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:15.154+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:15.383 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:15.381+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:15.382290+0000) 2022-04-03T21:56:16.001 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:16.000+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:16.191 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:16.189+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:16.383 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:16.381+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:16.382394+0000) 2022-04-03T21:56:16.758 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:56:16.756+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:56:17.006 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:17.005+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:17.144 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:17.142+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:17.384 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:17.382+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:17.382545+0000) 2022-04-03T21:56:17.981 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:17.979+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:18.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:18.181+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:18.384 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:18.381+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:18.382736+0000) 2022-04-03T21:56:18.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:18.977+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:19.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:19.193+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:19.384 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:19.382+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:19.382971+0000) 2022-04-03T21:56:19.980 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:19.977+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:20.202 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:20.199+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:20.384 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:20.382+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:20.383164+0000) 2022-04-03T21:56:20.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:20.930+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:21.142 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:56:21.148 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:56:21.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:21.199+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:21.384 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:21.382+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:21.383376+0000) 2022-04-03T21:56:21.962 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:21.959+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:22.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:22.172+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:22.385 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:22.383+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:22.383569+0000) 2022-04-03T21:56:22.970 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:22.967+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:23.137 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:23.134+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:23.385 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:23.383+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:23.383725+0000) 2022-04-03T21:56:24.000 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:23.997+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:24.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:24.150+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:24.385 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:24.383+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:24.383929+0000) 2022-04-03T21:56:25.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:25.035+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:25.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:25.196+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:25.385 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:25.383+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:25.384124+0000) 2022-04-03T21:56:26.042 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:26.040+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:26.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:26.170+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:26.387 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:26.384+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:26.384319+0000) 2022-04-03T21:56:27.071 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:27.069+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:27.152 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:27.149+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:27.386 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:27.384+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:27.384507+0000) 2022-04-03T21:56:28.103 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:28.101+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:28.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:28.160+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:28.386 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:28.383+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:28.384698+0000) 2022-04-03T21:56:29.069 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:29.066+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:29.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:29.129+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:29.387 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:29.384+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:29.384900+0000) 2022-04-03T21:56:30.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:30.082+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:30.090 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:30.087+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:30.386 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:30.384+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:30.385119+0000) 2022-04-03T21:56:31.118 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:31.115+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:31.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:31.118+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:31.386 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:31.385+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:31.385274+0000) 2022-04-03T21:56:31.774 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:56:31.772+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:56:32.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:32.103+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:32.106 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:32.104+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:32.386 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:32.385+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:32.385456+0000) 2022-04-03T21:56:33.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:33.068+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:33.135 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:33.132+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:33.387 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:33.385+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:33.385647+0000) 2022-04-03T21:56:34.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:34.057+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:34.101 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:34.099+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:34.387 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:34.385+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:34.385822+0000) 2022-04-03T21:56:35.098 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:35.095+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:35.114 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:35.112+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:35.387 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:35.385+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:35.385981+0000) 2022-04-03T21:56:36.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:36.109+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:36.144 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:36.142+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:36.387 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:36.386+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:36.386111+0000) 2022-04-03T21:56:37.094 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:37.092+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:37.109 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:37.107+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:37.387 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:37.385+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:37.386308+0000) 2022-04-03T21:56:38.063 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:38.061+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:38.114 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:38.112+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:38.387 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:38.385+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:38.386482+0000) 2022-04-03T21:56:39.068 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:39.066+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:39.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:39.072+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:39.388 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:39.385+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:39.386622+0000) 2022-04-03T21:56:40.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:40.039+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:40.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:40.071+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:40.388 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:40.385+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:40.386810+0000) 2022-04-03T21:56:41.050 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:41.048+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:41.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:41.083+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:41.388 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:41.386+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:41.386986+0000) 2022-04-03T21:56:42.023 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:42.021+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:42.088 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:42.085+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:42.388 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:42.385+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:42.387200+0000) 2022-04-03T21:56:42.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:42.996+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:43.056 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:43.054+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:43.388 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:43.386+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:43.387397+0000) 2022-04-03T21:56:43.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:43.983+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:44.019 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:44.017+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:44.389 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:44.386+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:44.387582+0000) 2022-04-03T21:56:44.959 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:44.957+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:44.988 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:44.985+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:45.389 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:45.386+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:45.387757+0000) 2022-04-03T21:56:45.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:45.991+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:46.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:46.001+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:46.389 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:46.387+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:46.387968+0000) 2022-04-03T21:56:46.789 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:56:46.787+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:56:46.993 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:46.990+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:47.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:47.039+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:47.389 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:47.387+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:47.388148+0000) 2022-04-03T21:56:47.980 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:47.978+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:48.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:48.077+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:48.389 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:48.387+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:48.388320+0000) 2022-04-03T21:56:49.025 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:49.022+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:49.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:49.059+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:49.390 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:49.387+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:49.388524+0000) 2022-04-03T21:56:50.063 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:50.061+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:50.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:50.101+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:50.390 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:50.387+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:50.388711+0000) 2022-04-03T21:56:51.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:51.093+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:51.096 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:51.093+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:51.281 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:56:51.285 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:56:51.390 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:51.388+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:51.388951+0000) 2022-04-03T21:56:52.068 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:52.066+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:52.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:52.086+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:52.390 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:52.388+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:52.389116+0000) 2022-04-03T21:56:53.048 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:53.046+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:53.099 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:53.097+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:53.390 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:53.388+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:53.389280+0000) 2022-04-03T21:56:54.052 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:54.049+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:54.140 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:54.138+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:54.391 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:54.388+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:54.389436+0000) 2022-04-03T21:56:55.013 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:55.010+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:55.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:55.133+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:55.391 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:55.388+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:55.389622+0000) 2022-04-03T21:56:55.978 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:55.976+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:56.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:56.159+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:56.391 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:56.389+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:56.389803+0000) 2022-04-03T21:56:56.997 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:56.995+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:57.146 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:57.143+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:57.391 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:57.389+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:57.389996+0000) 2022-04-03T21:56:57.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:57.993+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:58.158 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:58.156+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:58.391 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:58.389+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:58.390202+0000) 2022-04-03T21:56:59.011 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:56:59.009+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:56:59.140 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:56:59.138+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:56:59.391 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:56:59.389+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:56:59.390389+0000) 2022-04-03T21:57:00.006 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:00.004+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:00.151 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:00.149+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:00.392 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:00.390+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:00.390572+0000) 2022-04-03T21:57:01.019 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:01.018+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:01.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:01.177+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:01.392 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:01.390+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:01.390715+0000) 2022-04-03T21:57:01.805 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:57:01.803+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:57:01.987 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:01.985+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:02.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:02.162+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:02.392 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:02.390+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:02.390844+0000) 2022-04-03T21:57:02.973 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:02.972+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:03.186 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:03.184+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:03.392 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:03.390+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:03.391032+0000) 2022-04-03T21:57:03.982 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:03.981+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:04.152 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:04.151+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:04.392 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:04.390+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:04.391198+0000) 2022-04-03T21:57:04.935 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:04.933+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:05.199 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:05.197+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:05.392 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:05.390+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:05.391346+0000) 2022-04-03T21:57:05.890 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:05.888+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:06.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:06.151+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:06.393 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:06.391+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:06.391529+0000) 2022-04-03T21:57:06.906 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:06.904+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:07.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:07.181+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:07.393 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:07.391+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:07.391692+0000) 2022-04-03T21:57:07.902 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:07.900+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:08.301 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:08.219+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:08.393 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:08.391+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:08.391820+0000) 2022-04-03T21:57:08.887 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:08.885+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:09.240 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:09.238+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:09.393 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:09.391+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:09.391974+0000) 2022-04-03T21:57:09.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:09.841+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:10.241 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:10.239+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:10.393 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:10.391+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:10.392137+0000) 2022-04-03T21:57:10.849 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:10.848+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:11.237 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:11.236+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:11.393 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:11.392+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:11.392314+0000) 2022-04-03T21:57:11.875 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:11.873+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:12.245 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:12.244+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:12.394 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:12.392+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:12.392487+0000) 2022-04-03T21:57:12.831 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:12.830+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:13.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:13.284+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:13.394 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:13.392+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:13.392663+0000) 2022-04-03T21:57:13.830 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:13.827+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:14.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:14.245+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:14.394 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:14.392+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:14.392841+0000) 2022-04-03T21:57:14.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:14.810+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:15.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:15.270+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:15.394 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:15.392+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:15.393008+0000) 2022-04-03T21:57:15.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:15.797+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:16.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:16.279+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:16.394 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:16.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:16.393194+0000) 2022-04-03T21:57:16.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:16.810+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:16.821 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:57:16.819+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:57:17.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:17.243+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:17.394 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:17.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:17.393405+0000) 2022-04-03T21:57:17.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:17.839+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:18.261 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:18.258+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:18.395 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:18.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:18.393553+0000) 2022-04-03T21:57:18.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:18.866+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:19.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:19.299+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:19.395 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:19.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:19.393757+0000) 2022-04-03T21:57:19.890 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:19.888+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:20.258 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:20.256+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:20.395 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:20.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:20.394001+0000) 2022-04-03T21:57:20.921 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:20.918+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:21.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:21.224+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:21.353 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:57:21.356 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:57:21.395 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:21.392+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:21.394190+0000) 2022-04-03T21:57:21.957 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:21.954+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:22.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:22.252+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:22.395 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:22.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:22.394378+0000) 2022-04-03T21:57:22.986 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:22.984+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:23.253 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:23.251+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:23.396 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:23.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:23.394495+0000) 2022-04-03T21:57:24.034 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:24.032+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:24.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:24.226+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:24.396 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:24.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:24.394626+0000) 2022-04-03T21:57:25.082 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:25.080+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:25.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:25.258+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:25.396 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:25.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:25.394796+0000) 2022-04-03T21:57:26.132 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:26.129+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:26.230 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:26.228+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:26.396 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:26.394+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:26.394937+0000) 2022-04-03T21:57:27.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:27.109+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:27.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:27.230+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:27.396 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:27.394+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:27.395114+0000) 2022-04-03T21:57:28.106 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:28.104+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:28.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:28.279+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:28.396 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:28.394+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:28.395262+0000) 2022-04-03T21:57:29.090 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:29.087+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:29.275 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:29.273+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:29.396 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:29.394+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:29.395467+0000) 2022-04-03T21:57:30.129 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:30.127+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:30.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:30.296+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:30.397 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:30.394+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:30.395685+0000) 2022-04-03T21:57:31.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:31.154+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:31.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:31.314+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:31.397 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:31.395+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:31.395884+0000) 2022-04-03T21:57:31.837 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:57:31.835+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:57:32.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:32.185+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:32.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:32.273+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:32.397 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:32.395+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:32.396042+0000) 2022-04-03T21:57:33.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:33.209+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:33.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:33.269+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:33.397 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:33.395+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:33.396245+0000) 2022-04-03T21:57:34.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:34.229+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:34.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:34.245+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:34.398 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:34.395+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:34.396438+0000) 2022-04-03T21:57:35.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:35.212+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:35.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:35.293+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:35.398 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:35.395+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:35.396607+0000) 2022-04-03T21:57:36.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:36.174+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:36.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:36.320+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:36.398 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:36.396+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:36.396730+0000) 2022-04-03T21:57:37.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:37.209+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:37.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:37.317+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:37.398 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:37.396+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:37.396862+0000) 2022-04-03T21:57:38.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:38.248+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:38.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:38.298+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:38.398 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:38.396+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:38.396993+0000) 2022-04-03T21:57:39.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:39.283+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:39.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:39.313+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:39.398 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:39.396+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:39.397156+0000) 2022-04-03T21:57:40.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:40.292+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:40.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:40.317+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:40.398 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:40.396+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:40.397378+0000) 2022-04-03T21:57:41.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:41.291+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:41.343 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:41.341+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:41.399 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:41.396+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:41.397576+0000) 2022-04-03T21:57:42.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:42.259+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:42.308 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:42.306+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:42.399 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:42.397+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:42.397819+0000) 2022-04-03T21:57:43.271 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:43.269+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:43.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:43.275+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:43.399 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:43.397+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:43.398062+0000) 2022-04-03T21:57:44.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:44.227+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:44.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:44.230+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:44.399 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:44.397+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:44.398241+0000) 2022-04-03T21:57:45.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:45.220+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:45.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:45.271+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:45.399 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:45.397+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:45.398418+0000) 2022-04-03T21:57:46.238 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:46.235+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:46.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:46.269+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:46.400 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:46.398+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:46.398598+0000) 2022-04-03T21:57:46.853 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:57:46.850+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:57:47.190 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:47.188+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:47.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:47.276+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:47.400 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:47.398+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:47.398857+0000) 2022-04-03T21:57:48.200 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:48.198+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:48.283 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:48.280+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:48.400 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:48.398+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:48.399073+0000) 2022-04-03T21:57:49.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:49.247+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:49.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:49.320+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:49.400 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:49.398+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:49.399225+0000) 2022-04-03T21:57:50.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:50.275+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:50.283 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:50.281+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:50.400 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:50.398+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:50.399432+0000) 2022-04-03T21:57:51.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:51.279+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:51.287 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:51.285+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:51.400 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:51.399+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:51.399629+0000) 2022-04-03T21:57:51.512 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:57:51.515 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:57:52.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:52.232+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:52.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:52.290+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:52.401 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:52.399+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:52.399911+0000) 2022-04-03T21:57:53.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:53.275+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:53.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:53.318+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:53.401 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:53.399+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:53.400125+0000) 2022-04-03T21:57:54.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:54.267+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:54.271 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:54.269+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:54.401 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:54.399+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:54.400351+0000) 2022-04-03T21:57:55.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:55.248+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:55.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:55.282+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:55.401 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:55.400+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:55.400531+0000) 2022-04-03T21:57:56.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:56.296+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:56.301 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:56.300+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:56.402 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:56.400+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:56.400702+0000) 2022-04-03T21:57:57.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:57.252+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:57.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:57.283+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:57.402 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:57.400+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:57.400903+0000) 2022-04-03T21:57:58.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:58.245+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:58.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:58.262+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:58.402 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:58.400+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:58.401148+0000) 2022-04-03T21:57:59.222 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:57:59.220+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:57:59.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:57:59.239+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:57:59.402 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:57:59.401+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:57:59.401354+0000) 2022-04-03T21:58:00.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:00.264+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:00.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:00.289+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:00.403 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:00.401+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:00.401572+0000) 2022-04-03T21:58:01.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:01.309+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:01.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:01.325+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:01.403 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:01.401+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:01.401873+0000) 2022-04-03T21:58:01.869 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:58:01.866+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:58:02.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:02.276+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:02.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:02.316+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:02.403 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:02.401+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:02.402052+0000) 2022-04-03T21:58:03.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:03.271+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:03.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:03.306+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:03.403 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:03.402+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:03.402282+0000) 2022-04-03T21:58:04.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:04.263+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:04.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:04.342+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:04.403 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:04.402+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:04.402511+0000) 2022-04-03T21:58:05.236 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:05.234+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:05.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:05.353+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:05.404 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:05.401+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:05.402740+0000) 2022-04-03T21:58:06.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:06.242+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:06.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:06.366+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:06.404 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:06.401+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:06.403026+0000) 2022-04-03T21:58:07.228 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:07.227+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:07.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:07.342+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:07.404 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:07.402+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:07.403306+0000) 2022-04-03T21:58:08.240 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:08.238+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:08.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:08.368+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:08.404 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:08.402+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:08.403539+0000) 2022-04-03T21:58:09.266 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:09.263+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:09.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:09.341+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:09.405 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:09.402+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:09.403799+0000) 2022-04-03T21:58:10.234 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:10.231+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:10.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:10.369+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:10.405 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:10.403+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:10.404036+0000) 2022-04-03T21:58:11.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:11.241+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:11.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:11.325+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:11.405 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:11.402+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:11.404138+0000) 2022-04-03T21:58:12.213 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:12.211+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:12.284 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:12.281+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:12.405 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:12.403+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:12.404360+0000) 2022-04-03T21:58:13.180 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:13.177+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:13.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:13.282+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:13.405 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:13.403+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:13.404596+0000) 2022-04-03T21:58:14.181 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:14.179+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:14.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:14.306+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:14.406 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:14.403+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:14.404846+0000) 2022-04-03T21:58:15.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:15.170+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:15.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:15.277+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:15.406 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:15.404+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:15.405032+0000) 2022-04-03T21:58:16.169 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:16.166+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:16.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:16.301+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:16.406 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:16.404+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:16.405237+0000) 2022-04-03T21:58:16.884 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:58:16.882+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:58:17.205 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:17.202+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:17.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:17.319+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:17.406 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:17.404+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:17.405443+0000) 2022-04-03T21:58:18.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:18.157+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:18.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:18.277+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:18.407 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:18.404+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:18.405685+0000) 2022-04-03T21:58:19.152 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:19.149+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:19.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:19.260+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:19.407 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:19.405+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:19.405914+0000) 2022-04-03T21:58:20.151 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:20.149+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:20.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:20.308+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:20.407 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:20.405+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:20.406142+0000) 2022-04-03T21:58:21.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:21.184+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:21.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:21.310+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:21.407 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:21.405+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:21.406308+0000) 2022-04-03T21:58:21.684 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:58:21.689 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:58:22.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:22.168+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:22.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:22.267+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:22.408 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:22.405+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:22.406530+0000) 2022-04-03T21:58:23.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:23.151+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:23.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:23.240+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:23.408 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:23.406+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:23.406798+0000) 2022-04-03T21:58:24.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:24.163+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:24.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:24.260+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:24.408 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:24.406+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:24.407022+0000) 2022-04-03T21:58:25.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:25.120+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:25.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:25.300+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:25.408 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:25.406+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:25.407242+0000) 2022-04-03T21:58:26.146 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:26.144+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:26.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:26.338+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:26.408 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:26.406+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:26.407401+0000) 2022-04-03T21:58:27.118 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:27.115+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:27.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:27.314+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:27.409 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:27.406+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:27.407623+0000) 2022-04-03T21:58:28.099 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:28.097+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:28.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:28.358+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:28.409 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:28.407+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:28.407891+0000) 2022-04-03T21:58:29.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:29.101+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:29.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:29.333+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:29.409 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:29.407+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:29.408065+0000) 2022-04-03T21:58:30.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:30.123+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:30.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:30.345+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:30.409 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:30.407+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:30.408214+0000) 2022-04-03T21:58:31.092 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:31.090+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:31.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:31.392+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:31.409 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:31.407+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:31.408384+0000) 2022-04-03T21:58:31.900 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:58:31.898+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:58:32.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:32.051+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:32.409 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:32.407+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:32.408525+0000) 2022-04-03T21:58:32.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:32.441+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:33.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:33.083+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:33.410 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:33.408+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:33.408664+0000) 2022-04-03T21:58:33.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:33.433+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:34.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:34.044+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:34.410 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:34.408+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:34.408897+0000) 2022-04-03T21:58:34.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:34.437+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:35.058 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:35.056+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:35.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:35.397+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:35.410 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:35.408+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:35.409129+0000) 2022-04-03T21:58:36.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:36.036+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:36.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:36.363+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:36.410 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:36.408+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:36.409334+0000) 2022-04-03T21:58:37.011 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:37.009+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:37.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:37.324+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:37.410 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:37.409+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:37.409529+0000) 2022-04-03T21:58:38.032 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:38.030+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:38.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:38.286+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:38.411 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:38.409+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:38.409752+0000) 2022-04-03T21:58:39.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:39.002+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:39.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:39.259+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:39.411 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:39.409+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:39.409970+0000) 2022-04-03T21:58:40.052 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:40.049+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:40.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:40.305+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:40.411 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:40.409+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:40.410168+0000) 2022-04-03T21:58:41.030 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:41.028+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:41.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:41.262+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:41.411 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:41.410+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:41.410373+0000) 2022-04-03T21:58:42.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:42.064+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:42.301 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:42.298+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:42.411 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:42.410+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:42.410594+0000) 2022-04-03T21:58:43.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:43.039+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:43.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:43.272+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:43.412 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:43.410+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:43.410884+0000) 2022-04-03T21:58:44.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:44.011+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:44.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:44.284+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:44.412 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:44.410+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:44.411063+0000) 2022-04-03T21:58:45.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:45.010+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:45.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:45.254+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:45.412 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:45.411+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:45.411276+0000) 2022-04-03T21:58:45.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:45.972+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:46.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:46.262+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:46.412 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:46.410+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:46.411532+0000) 2022-04-03T21:58:46.916 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:58:46.914+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:58:46.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:46.990+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:47.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:47.254+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:47.413 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:47.411+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:47.411729+0000) 2022-04-03T21:58:48.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:48.023+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:48.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:48.253+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:48.413 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:48.411+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:48.411947+0000) 2022-04-03T21:58:49.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:49.060+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:49.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:49.263+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:49.413 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:49.412+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:49.412126+0000) 2022-04-03T21:58:50.070 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:50.069+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:50.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:50.229+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:50.413 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:50.411+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:50.412346+0000) 2022-04-03T21:58:51.100 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:51.098+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:51.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:51.209+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:51.414 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:51.411+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:51.412580+0000) 2022-04-03T21:58:51.790 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:58:51.794 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:58:52.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:52.065+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:52.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:52.226+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:52.414 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:52.411+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:52.412905+0000) 2022-04-03T21:58:53.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:53.079+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:53.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:53.227+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:53.414 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:53.412+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:53.413141+0000) 2022-04-03T21:58:54.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:54.078+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:54.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:54.207+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:54.414 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:54.412+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:54.413383+0000) 2022-04-03T21:58:55.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:55.083+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:55.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:55.230+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:55.414 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:55.412+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:55.413618+0000) 2022-04-03T21:58:56.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:56.080+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:56.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:56.208+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:56.415 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:56.412+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:56.413871+0000) 2022-04-03T21:58:57.110 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:57.108+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:57.202 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:57.201+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:57.415 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:57.413+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:57.414009+0000) 2022-04-03T21:58:58.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:58.146+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:58.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:58.155+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:58.415 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:58.413+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:58.414202+0000) 2022-04-03T21:58:59.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:58:59.109+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:58:59.129 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:58:59.128+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:58:59.415 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:58:59.413+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:58:59.414432+0000) 2022-04-03T21:59:00.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:00.081+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:00.134 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:00.133+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:00.416 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:00.413+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:00.414659+0000) 2022-04-03T21:59:01.069 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:01.067+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:01.158 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:01.156+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:01.416 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:01.414+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:01.414891+0000) 2022-04-03T21:59:01.932 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:59:01.930+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:59:02.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:02.080+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:02.169 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:02.167+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:02.416 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:02.414+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:02.415161+0000) 2022-04-03T21:59:03.051 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:03.049+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:03.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:03.196+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:03.416 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:03.414+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:03.415393+0000) 2022-04-03T21:59:04.060 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:04.058+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:04.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:04.239+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:04.417 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:04.414+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:04.415576+0000) 2022-04-03T21:59:05.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:05.104+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:05.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:05.250+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:05.417 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:05.415+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:05.415750+0000) 2022-04-03T21:59:06.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:06.123+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:06.213 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:06.210+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:06.417 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:06.415+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:06.415912+0000) 2022-04-03T21:59:07.119 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:07.117+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:07.205 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:07.202+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:07.417 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:07.415+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:07.416070+0000) 2022-04-03T21:59:08.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:08.077+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:08.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:08.229+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:08.417 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:08.415+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:08.416271+0000) 2022-04-03T21:59:09.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:09.044+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:09.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:09.215+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:09.418 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:09.415+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:09.416492+0000) 2022-04-03T21:59:10.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:10.073+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:10.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:10.202+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:10.418 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:10.416+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:10.416721+0000) 2022-04-03T21:59:11.070 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:11.067+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:11.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:11.216+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:11.418 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:11.416+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:11.416974+0000) 2022-04-03T21:59:12.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:12.080+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:12.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:12.262+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:12.418 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:12.416+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:12.417185+0000) 2022-04-03T21:59:13.068 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:13.065+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:13.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:13.291+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:13.418 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:13.416+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:13.417385+0000) 2022-04-03T21:59:14.040 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:14.037+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:14.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:14.275+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:14.419 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:14.417+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:14.417573+0000) 2022-04-03T21:59:14.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:14.988+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:15.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:15.230+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:15.419 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:15.417+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:15.417789+0000) 2022-04-03T21:59:15.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:15.994+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:16.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:16.278+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:16.419 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:16.417+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:16.418003+0000) 2022-04-03T21:59:16.948 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:59:16.946+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:59:17.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:17.036+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:17.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:17.297+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:17.419 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:17.417+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:17.418174+0000) 2022-04-03T21:59:18.001 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:17.998+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:18.337 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:18.335+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:18.419 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:18.417+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:18.418367+0000) 2022-04-03T21:59:19.036 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:19.034+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:19.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:19.366+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:19.420 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:19.418+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:19.418594+0000) 2022-04-03T21:59:20.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:20.021+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:20.401 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:20.399+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:20.420 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:20.418+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:20.418876+0000) 2022-04-03T21:59:21.021 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:21.019+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:21.376 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:21.374+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:21.420 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:21.418+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:21.419102+0000) 2022-04-03T21:59:21.954 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:59:21.958 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:59:21.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:21.972+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:22.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:22.342+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:22.420 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:22.418+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:22.419360+0000) 2022-04-03T21:59:22.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:22.987+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:23.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:23.359+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:23.421 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:23.419+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:23.419578+0000) 2022-04-03T21:59:24.000 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:23.998+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:24.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:24.361+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:24.421 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:24.419+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:24.419782+0000) 2022-04-03T21:59:24.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:24.995+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:25.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:25.314+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:25.421 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:25.419+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:25.420002+0000) 2022-04-03T21:59:26.021 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:26.019+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:26.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:26.297+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:26.421 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:26.419+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:26.420259+0000) 2022-04-03T21:59:27.018 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:27.016+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:27.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:27.295+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:27.421 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:27.420+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:27.420421+0000) 2022-04-03T21:59:28.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:28.022+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:28.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:28.286+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:28.422 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:28.420+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:28.420639+0000) 2022-04-03T21:59:29.020 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:29.018+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:29.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:29.273+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:29.422 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:29.420+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:29.420886+0000) 2022-04-03T21:59:30.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:30.017+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:30.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:30.314+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:30.422 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:30.420+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:30.421125+0000) 2022-04-03T21:59:31.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:31.051+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:31.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:31.283+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:31.422 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:31.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:31.421369+0000) 2022-04-03T21:59:31.964 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:59:31.961+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:59:32.030 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:32.028+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:32.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:32.314+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:32.423 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:32.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:32.421570+0000) 2022-04-03T21:59:32.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:32.995+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:33.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:33.302+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:33.423 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:33.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:33.421754+0000) 2022-04-03T21:59:33.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:33.951+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:34.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:34.279+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:34.423 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:34.420+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:34.421964+0000) 2022-04-03T21:59:35.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:35.001+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:35.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:35.309+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:35.423 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:35.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:35.422162+0000) 2022-04-03T21:59:36.030 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:36.028+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:36.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:36.291+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:36.423 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:36.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:36.422302+0000) 2022-04-03T21:59:36.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:36.989+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:37.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:37.278+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:37.424 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:37.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:37.422510+0000) 2022-04-03T21:59:38.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:38.010+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:38.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:38.318+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:38.424 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:38.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:38.422745+0000) 2022-04-03T21:59:39.060 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:39.058+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:39.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:39.351+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:39.424 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:39.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:39.422901+0000) 2022-04-03T21:59:40.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:40.024+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:40.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:40.323+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:40.424 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:40.422+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:40.423051+0000) 2022-04-03T21:59:40.994 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:40.992+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:41.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:41.351+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:41.424 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:41.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:41.423240+0000) 2022-04-03T21:59:41.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:41.950+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:42.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:42.304+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:42.424 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:42.422+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:42.423477+0000) 2022-04-03T21:59:42.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:42.980+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:43.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:43.331+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:43.425 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:43.422+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:43.423659+0000) 2022-04-03T21:59:44.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:44.011+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:44.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:44.337+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:44.425 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:44.423+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:44.423839+0000) 2022-04-03T21:59:44.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:44.979+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:45.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:45.356+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:45.425 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:45.423+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:45.424056+0000) 2022-04-03T21:59:46.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:46.005+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:46.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:46.308+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:46.425 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:46.423+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:46.424291+0000) 2022-04-03T21:59:46.979 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T21:59:46.976+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T21:59:46.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:46.995+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:47.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:47.312+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:47.425 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:47.423+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:47.424495+0000) 2022-04-03T21:59:48.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:48.017+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:48.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:48.351+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:48.426 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:48.423+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:48.424655+0000) 2022-04-03T21:59:48.975 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:48.973+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:49.378 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:49.376+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:49.426 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:49.424+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:49.424877+0000) 2022-04-03T21:59:49.971 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:49.969+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:50.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:50.350+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:50.426 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:50.424+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:50.425136+0000) 2022-04-03T21:59:50.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:50.980+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:51.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:51.349+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:51.426 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:51.424+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:51.425295+0000) 2022-04-03T21:59:51.427 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:51.425+0000 7fd62421b700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T21:59:51.427 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:51.425+0000 7fd623a1a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T21:59:51.428 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:51.425+0000 7fd623a1a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:51.426688+0000) 2022-04-03T21:59:51.960 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:51.958+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:52.108 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:59:52.112 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T21:59:52.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:52.390+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:52.426 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:52.424+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:52.425423+0000) 2022-04-03T21:59:52.975 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:52.973+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:53.400 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:53.399+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:53.427 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:53.424+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:53.425561+0000) 2022-04-03T21:59:54.018 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:54.016+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:54.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:54.364+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:54.427 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:54.425+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:54.425737+0000) 2022-04-03T21:59:55.001 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:55.000+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:55.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:55.401+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:55.427 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:55.425+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:55.425909+0000) 2022-04-03T21:59:56.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:56.005+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:56.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:56.390+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:56.427 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:56.425+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:56.426099+0000) 2022-04-03T21:59:57.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:57.045+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:57.418 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:57.415+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:57.427 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:57.425+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:57.426286+0000) 2022-04-03T21:59:58.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:58.063+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:58.428 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:58.426+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:58.426492+0000) 2022-04-03T21:59:58.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:58.442+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T21:59:59.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T21:59:59.045+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T21:59:59.428 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T21:59:59.425+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T20:59:59.426690+0000) 2022-04-03T21:59:59.484 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T21:59:59.481+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:00.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:00.007+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:00.428 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:00.426+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:00.426855+0000) 2022-04-03T22:00:00.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:00.516+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:01.022 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:01.019+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:01.428 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:01.426+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:01.426991+0000) 2022-04-03T22:00:01.490 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:01.488+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:01.995 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:00:01.992+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:00:02.031 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:02.029+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:02.428 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:02.426+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:02.427132+0000) 2022-04-03T22:00:02.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:02.529+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:03.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:03.051+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:03.428 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:03.426+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:03.427345+0000) 2022-04-03T22:00:03.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:03.482+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:04.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:04.043+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:04.429 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:04.427+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:04.427518+0000) 2022-04-03T22:00:04.511 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:04.508+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:05.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:05.013+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:05.429 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:05.426+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:05.427686+0000) 2022-04-03T22:00:05.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:05.474+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:06.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:06.012+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:06.429 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:06.427+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:06.427810+0000) 2022-04-03T22:00:06.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:06.480+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:06.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:06.980+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:07.429 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:07.427+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:07.427961+0000) 2022-04-03T22:00:07.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:07.456+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:07.996 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:07.994+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:08.429 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:08.427+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:08.428163+0000) 2022-04-03T22:00:08.506 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:08.503+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:08.967 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:08.965+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:09.429 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:09.428+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:09.428393+0000) 2022-04-03T22:00:09.484 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:09.482+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:09.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:09.997+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:10.430 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:10.428+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:10.428602+0000) 2022-04-03T22:00:10.508 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:10.506+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:11.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:11.036+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:11.430 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:11.428+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:11.428818+0000) 2022-04-03T22:00:11.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:11.533+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:12.084 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:12.082+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:12.430 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:12.428+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:12.428984+0000) 2022-04-03T22:00:12.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:12.496+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:13.051 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:13.048+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:13.430 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:13.429+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:13.429164+0000) 2022-04-03T22:00:13.510 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:13.508+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:14.068 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:14.065+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:14.430 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:14.429+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:14.429357+0000) 2022-04-03T22:00:14.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:14.545+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:15.064 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:15.062+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:15.431 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:15.429+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:15.429556+0000) 2022-04-03T22:00:15.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:15.540+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:16.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:16.083+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:16.431 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:16.429+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:16.429696+0000) 2022-04-03T22:00:16.540 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:16.538+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:17.011 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:00:17.008+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:00:17.065 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:17.063+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:17.431 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:17.429+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:17.429877+0000) 2022-04-03T22:00:17.519 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:17.516+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:18.029 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:18.026+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:18.431 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:18.428+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:18.429995+0000) 2022-04-03T22:00:18.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:18.551+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:19.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:19.023+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:19.431 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:19.429+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:19.430156+0000) 2022-04-03T22:00:19.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:19.558+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:20.031 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:20.029+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:20.431 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:20.429+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:20.430368+0000) 2022-04-03T22:00:20.609 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:20.606+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:21.051 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:21.049+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:21.432 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:21.428+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:21.430504+0000) 2022-04-03T22:00:21.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:21.557+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:22.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:22.044+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:22.256 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:00:22.260 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:00:22.432 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:22.428+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:22.430678+0000) 2022-04-03T22:00:22.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:22.561+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:23.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:23.014+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:23.432 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:23.429+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:23.430842+0000) 2022-04-03T22:00:23.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:23.560+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:23.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:23.991+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:24.432 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:24.430+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:24.430997+0000) 2022-04-03T22:00:24.610 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:24.608+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:24.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:24.987+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:25.432 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:25.430+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:25.431161+0000) 2022-04-03T22:00:25.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:25.586+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:25.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:25.962+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:26.432 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:26.430+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:26.431371+0000) 2022-04-03T22:00:26.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:26.541+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:26.963 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:26.960+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:27.433 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:27.430+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:27.431532+0000) 2022-04-03T22:00:27.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:27.564+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:27.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:27.915+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:28.433 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:28.430+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:28.431737+0000) 2022-04-03T22:00:28.522 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:28.520+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:28.943 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:28.941+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:29.433 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:29.431+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:29.431970+0000) 2022-04-03T22:00:29.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:29.569+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:29.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:29.982+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:30.433 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:30.431+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:30.432183+0000) 2022-04-03T22:00:30.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:30.572+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:31.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:31.009+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:31.433 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:31.431+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:31.432381+0000) 2022-04-03T22:00:31.595 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:31.593+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:31.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:31.962+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:32.017 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:00:32.015+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:00:32.434 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:32.431+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:32.432543+0000) 2022-04-03T22:00:32.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:32.629+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:33.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:33.008+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:33.434 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:33.432+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:33.432743+0000) 2022-04-03T22:00:33.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:33.590+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:34.042 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:34.040+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:34.434 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:34.432+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:34.432999+0000) 2022-04-03T22:00:34.585 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:34.583+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:35.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:35.005+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:35.434 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:35.432+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:35.433150+0000) 2022-04-03T22:00:35.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:35.585+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:36.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:36.023+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:36.434 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:36.432+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:36.433348+0000) 2022-04-03T22:00:36.602 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:36.600+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:36.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:36.977+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:37.435 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:37.432+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:37.433519+0000) 2022-04-03T22:00:37.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:37.648+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:37.968 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:37.967+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:38.435 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:38.433+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:38.433678+0000) 2022-04-03T22:00:38.634 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:38.633+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:38.970 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:38.968+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:39.435 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:39.433+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:39.433879+0000) 2022-04-03T22:00:39.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:39.638+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:39.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:39.975+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:40.436 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:40.433+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:40.434079+0000) 2022-04-03T22:00:40.642 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:40.640+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:40.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:40.978+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:41.435 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:41.433+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:41.434260+0000) 2022-04-03T22:00:41.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:41.629+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:41.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:41.965+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:42.436 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:42.433+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:42.434467+0000) 2022-04-03T22:00:42.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:42.614+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:42.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:42.923+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:43.436 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:43.434+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:43.434645+0000) 2022-04-03T22:00:43.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:43.663+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:43.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:43.909+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:44.436 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:44.434+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:44.434846+0000) 2022-04-03T22:00:44.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:44.614+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:44.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:44.951+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:45.436 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:45.434+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:45.435034+0000) 2022-04-03T22:00:45.617 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:45.615+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:45.939 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:45.937+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:46.436 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:46.434+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:46.435264+0000) 2022-04-03T22:00:46.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:46.633+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:46.898 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:46.897+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:47.020 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:00:47.018+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:00:47.437 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:47.435+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:47.435458+0000) 2022-04-03T22:00:47.629 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:47.627+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:47.944 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:47.943+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:48.437 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:48.435+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:48.435625+0000) 2022-04-03T22:00:48.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:48.625+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:48.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:48.916+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:49.437 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:49.435+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:49.435835+0000) 2022-04-03T22:00:49.625 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:49.624+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:49.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:49.945+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:50.437 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:50.435+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:50.436051+0000) 2022-04-03T22:00:50.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:50.592+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:50.909 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:50.908+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:51.437 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:51.435+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:51.436264+0000) 2022-04-03T22:00:51.625 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:51.623+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:51.941 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:51.938+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:52.370 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:00:52.375 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:00:52.437 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:52.435+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:52.436420+0000) 2022-04-03T22:00:52.622 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:52.620+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:52.920 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:52.918+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:53.438 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:53.435+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:53.436661+0000) 2022-04-03T22:00:53.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:53.623+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:53.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:53.900+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:54.438 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:54.436+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:54.436839+0000) 2022-04-03T22:00:54.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:54.625+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:54.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:54.919+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:55.438 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:55.436+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:55.437010+0000) 2022-04-03T22:00:55.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:55.609+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:55.967 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:55.964+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:56.438 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:56.436+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:56.437178+0000) 2022-04-03T22:00:56.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:56.590+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:56.925 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:56.923+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:57.438 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:57.437+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:57.437349+0000) 2022-04-03T22:00:57.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:57.613+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:57.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:57.952+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:58.439 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:58.437+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:58.437502+0000) 2022-04-03T22:00:58.595 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:58.592+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:00:58.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:00:58.980+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:00:59.439 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:00:59.437+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:00:59.437653+0000) 2022-04-03T22:00:59.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:00:59.612+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:00.018 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:00.015+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:00.439 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:00.437+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:00.437861+0000) 2022-04-03T22:01:00.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:00.611+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:00.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:00.983+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:01.440 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:01.437+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:01.438018+0000) 2022-04-03T22:01:01.595 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:01.592+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:01.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:01.955+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:02.031 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:01:02.029+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:01:02.440 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:02.437+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:02.438648+0000) 2022-04-03T22:01:02.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:02.573+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:02.944 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:02.941+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:03.440 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:03.437+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:03.438829+0000) 2022-04-03T22:01:03.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:03.545+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:03.914 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:03.911+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:04.440 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:04.438+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:04.439017+0000) 2022-04-03T22:01:04.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:04.578+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:04.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:04.946+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:05.440 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:05.438+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:05.439199+0000) 2022-04-03T22:01:05.600 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:05.597+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:05.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:05.935+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:06.440 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:06.438+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:06.439375+0000) 2022-04-03T22:01:06.608 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:06.605+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:06.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:06.935+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:07.441 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:07.438+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:07.439582+0000) 2022-04-03T22:01:07.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:07.587+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:07.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:07.945+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:08.441 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:08.438+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:08.439783+0000) 2022-04-03T22:01:08.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:08.563+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:08.987 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:08.984+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:09.441 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:09.439+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:09.439915+0000) 2022-04-03T22:01:09.546 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:09.543+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:09.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:09.949+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:10.441 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:10.439+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:10.440061+0000) 2022-04-03T22:01:10.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:10.563+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:10.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:10.971+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:11.441 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:11.439+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:11.440256+0000) 2022-04-03T22:01:11.520 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:11.518+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:11.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:11.956+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:12.442 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:12.439+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:12.440459+0000) 2022-04-03T22:01:12.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:12.493+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:12.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:12.915+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:13.442 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:13.439+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:13.440633+0000) 2022-04-03T22:01:13.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:13.485+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:13.909 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:13.900+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:14.442 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:14.440+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:14.440821+0000) 2022-04-03T22:01:14.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:14.441+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:14.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:14.914+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:15.415 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:15.413+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:15.442 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:15.440+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:15.441014+0000) 2022-04-03T22:01:15.895 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:15.893+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:16.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:16.417+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:16.442 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:16.440+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:16.441186+0000) 2022-04-03T22:01:16.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:16.866+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:17.047 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:01:17.045+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:01:17.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:17.417+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:17.442 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:17.440+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:17.441341+0000) 2022-04-03T22:01:17.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:17.849+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:18.426 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:18.424+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:18.442 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:18.440+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:18.441532+0000) 2022-04-03T22:01:18.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:18.809+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:19.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:19.383+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:19.443 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:19.441+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:19.441723+0000) 2022-04-03T22:01:19.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:19.800+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:20.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:20.432+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:20.443 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:20.441+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:20.441872+0000) 2022-04-03T22:01:20.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:20.812+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:21.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:21.391+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:21.443 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:21.441+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:21.442051+0000) 2022-04-03T22:01:21.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:21.773+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:22.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:22.361+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:22.443 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:22.441+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:22.442230+0000) 2022-04-03T22:01:22.526 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:01:22.530 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:01:22.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:22.752+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:23.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:23.378+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:23.443 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:23.441+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:23.442423+0000) 2022-04-03T22:01:23.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:23.755+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:24.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:24.415+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:24.444 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:24.442+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:24.442617+0000) 2022-04-03T22:01:24.767 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:24.765+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:25.444 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:25.442+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:25.442831+0000) 2022-04-03T22:01:25.457 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:25.455+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:25.786 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:25.784+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:26.444 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:26.442+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:26.443020+0000) 2022-04-03T22:01:26.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:26.485+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:26.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:26.797+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:27.444 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:27.442+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:27.443170+0000) 2022-04-03T22:01:27.486 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:27.484+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:27.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:27.797+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:28.444 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:28.442+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:28.443316+0000) 2022-04-03T22:01:28.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:28.519+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:28.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:28.770+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:29.444 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:29.442+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:29.443527+0000) 2022-04-03T22:01:29.477 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:29.475+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:29.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:29.747+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:30.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:30.439+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:30.445 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:30.443+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:30.443683+0000) 2022-04-03T22:01:30.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:30.794+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:31.337 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:31.335+0000 7fd623a1a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:31.336474+0000) 2022-04-03T22:01:31.445 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:31.443+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:31.443876+0000) 2022-04-03T22:01:31.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:31.463+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:31.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:31.760+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:32.063 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:01:32.061+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:01:32.445 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:32.443+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:32.444032+0000) 2022-04-03T22:01:32.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:32.502+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:32.746 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:32.744+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:33.445 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:33.443+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:33.444201+0000) 2022-04-03T22:01:33.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:33.494+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:33.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:33.759+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:34.445 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:34.444+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:34.444357+0000) 2022-04-03T22:01:34.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:34.528+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:34.718 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:34.716+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:35.446 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:35.444+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:35.444576+0000) 2022-04-03T22:01:35.573 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:35.571+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:35.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:35.721+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:36.446 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:36.444+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:36.444724+0000) 2022-04-03T22:01:36.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:36.610+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:36.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:36.674+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:37.446 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:37.444+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:37.444873+0000) 2022-04-03T22:01:37.610 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:37.609+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:37.675 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:37.674+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:38.446 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:38.444+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:38.445022+0000) 2022-04-03T22:01:38.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:38.577+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:38.686 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:38.684+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:39.446 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:39.445+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:39.445222+0000) 2022-04-03T22:01:39.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:39.562+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:39.673 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:39.671+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:40.447 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:40.445+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:40.445441+0000) 2022-04-03T22:01:40.573 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:40.571+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:40.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:40.713+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:41.447 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:41.445+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:41.445634+0000) 2022-04-03T22:01:41.604 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:41.603+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:41.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:41.736+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:42.447 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:42.445+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:42.445846+0000) 2022-04-03T22:01:42.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:42.566+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:42.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:42.780+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:43.447 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:43.445+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:43.446071+0000) 2022-04-03T22:01:43.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:43.576+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:43.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:43.776+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:44.447 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:44.446+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:44.446270+0000) 2022-04-03T22:01:44.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:44.625+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:44.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:44.783+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:45.447 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:45.446+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:45.446479+0000) 2022-04-03T22:01:45.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:45.592+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:45.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:45.789+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:46.448 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:46.446+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:46.446691+0000) 2022-04-03T22:01:46.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:46.624+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:46.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:46.833+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:47.079 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:01:47.076+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:01:47.448 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:47.445+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:47.446891+0000) 2022-04-03T22:01:47.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:47.617+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:47.814 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:47.812+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:48.448 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:48.446+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:48.447068+0000) 2022-04-03T22:01:48.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:48.584+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:48.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:48.790+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:49.448 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:49.446+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:49.447220+0000) 2022-04-03T22:01:49.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:49.635+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:49.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:49.817+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:50.448 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:50.446+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:50.447410+0000) 2022-04-03T22:01:50.633 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:50.631+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:50.826 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:50.823+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:51.449 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:51.446+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:51.447612+0000) 2022-04-03T22:01:51.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:51.595+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:51.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:51.829+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:52.449 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:52.446+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:52.447831+0000) 2022-04-03T22:01:52.583 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:52.580+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:52.665 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:01:52.668 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:01:52.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:52.830+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:53.449 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:53.447+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:53.447999+0000) 2022-04-03T22:01:53.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:53.611+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:53.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:53.877+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:54.449 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:54.447+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:54.448205+0000) 2022-04-03T22:01:54.602 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:54.600+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:54.914 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:54.911+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:55.450 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:55.447+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:55.448393+0000) 2022-04-03T22:01:55.552 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:55.550+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:55.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:55.938+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:56.450 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:56.447+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:56.448554+0000) 2022-04-03T22:01:56.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:56.594+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:56.932 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:56.930+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:57.450 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:57.448+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:57.448750+0000) 2022-04-03T22:01:57.606 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:57.604+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:57.944 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:57.942+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:58.450 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:58.448+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:58.448977+0000) 2022-04-03T22:01:58.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:58.577+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:58.970 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:58.967+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:01:59.450 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:01:59.448+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:01:59.449185+0000) 2022-04-03T22:01:59.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:01:59.612+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:01:59.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:01:59.920+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:00.450 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:00.448+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:00.449398+0000) 2022-04-03T22:02:00.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:00.577+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:00.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:00.885+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:01.451 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:01.448+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:01.449608+0000) 2022-04-03T22:02:01.600 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:01.597+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:01.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:01.858+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:02.095 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:02:02.092+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:02:02.451 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:02.449+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:02.449763+0000) 2022-04-03T22:02:02.583 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:02.580+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:02.867 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:02.865+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:03.451 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:03.449+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:03.449983+0000) 2022-04-03T22:02:03.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:03.532+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:03.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:03.827+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:04.451 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:04.449+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:04.450174+0000) 2022-04-03T22:02:04.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:04.545+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:04.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:04.861+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:05.451 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:05.449+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:05.450374+0000) 2022-04-03T22:02:05.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:05.574+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:05.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:05.830+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:06.451 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:06.449+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:06.450574+0000) 2022-04-03T22:02:06.546 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:06.543+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:06.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:06.802+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:07.452 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:07.450+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:07.450801+0000) 2022-04-03T22:02:07.506 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:07.504+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:07.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:07.842+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:08.452 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:08.450+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:08.451011+0000) 2022-04-03T22:02:08.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:08.545+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:08.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:08.805+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:09.452 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:09.450+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:09.451205+0000) 2022-04-03T22:02:09.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:09.512+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:09.797 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:09.795+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:10.452 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:10.450+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:10.451385+0000) 2022-04-03T22:02:10.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:10.492+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:10.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:10.781+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:11.453 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:11.451+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:11.451589+0000) 2022-04-03T22:02:11.500 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:11.498+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:11.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:11.828+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:12.453 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:12.451+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:12.451806+0000) 2022-04-03T22:02:12.528 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:12.526+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:12.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:12.856+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:13.453 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:13.451+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:13.452014+0000) 2022-04-03T22:02:13.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:13.554+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:13.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:13.825+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:14.453 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:14.451+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:14.452222+0000) 2022-04-03T22:02:14.577 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:14.574+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:14.787 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:14.785+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:15.453 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:15.452+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:15.452404+0000) 2022-04-03T22:02:15.586 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:15.584+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:15.777 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:15.775+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:16.454 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:16.452+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:16.452604+0000) 2022-04-03T22:02:16.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:16.578+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:16.730 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:16.728+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:17.111 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:02:17.108+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:02:17.454 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:17.452+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:17.452809+0000) 2022-04-03T22:02:17.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:17.553+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:17.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:17.778+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:18.454 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:18.452+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:18.453013+0000) 2022-04-03T22:02:18.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:18.578+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:18.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:18.757+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:19.454 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:19.452+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:19.453234+0000) 2022-04-03T22:02:19.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:19.550+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:19.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:19.708+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:20.454 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:20.453+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:20.453440+0000) 2022-04-03T22:02:20.552 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:20.550+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:20.697 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:20.696+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:21.455 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:21.453+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:21.453600+0000) 2022-04-03T22:02:21.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:21.565+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:21.708 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:21.707+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:22.455 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:22.453+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:22.453795+0000) 2022-04-03T22:02:22.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:22.595+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:22.711 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:22.709+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:22.822 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:02:22.825 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:02:23.455 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:23.453+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:23.453952+0000) 2022-04-03T22:02:23.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:23.594+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:23.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:23.715+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:24.455 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:24.453+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:24.454159+0000) 2022-04-03T22:02:24.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:24.555+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:24.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:24.742+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:25.455 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:25.454+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:25.454335+0000) 2022-04-03T22:02:25.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:25.548+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:25.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:25.778+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:02:26.455 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:26.454+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:26.454509+0000) 2022-04-03T22:02:26.546 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:26.545+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:26.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:26.786+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:27.456 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:27.454+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:27.454671+0000) 2022-04-03T22:02:27.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:27.587+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:27.769 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:27.767+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:28.456 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:28.454+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:28.454853+0000) 2022-04-03T22:02:28.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:28.618+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:28.756 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:28.754+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:29.456 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:29.455+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:29.455074+0000) 2022-04-03T22:02:29.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:29.610+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:29.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:29.746+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:30.456 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:30.455+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:30.455277+0000) 2022-04-03T22:02:30.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:30.614+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:30.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:30.723+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:31.456 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:31.454+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:31.455454+0000) 2022-04-03T22:02:31.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:31.597+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:31.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:31.731+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:32.126 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:02:32.123+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:02:32.457 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:32.454+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:32.455650+0000) 2022-04-03T22:02:32.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:32.643+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:32.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:32.754+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:33.457 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:33.454+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:33.455830+0000) 2022-04-03T22:02:33.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:33.628+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:33.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:33.759+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:34.457 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:34.455+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:34.455980+0000) 2022-04-03T22:02:34.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:34.633+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:34.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:34.743+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:35.457 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:35.455+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:35.456133+0000) 2022-04-03T22:02:35.642 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:35.640+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:35.696 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:35.694+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:36.457 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:36.455+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:36.456261+0000) 2022-04-03T22:02:36.595 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:36.593+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:36.685 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:36.683+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:37.457 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:37.455+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:37.456420+0000) 2022-04-03T22:02:37.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:37.595+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:37.706 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:37.704+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:38.458 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:38.455+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:38.456607+0000) 2022-04-03T22:02:38.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:38.589+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:38.708 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:38.706+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:39.458 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:39.455+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:39.456806+0000) 2022-04-03T22:02:39.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:39.631+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:39.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:39.701+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:40.458 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:40.456+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:40.456991+0000) 2022-04-03T22:02:40.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:40.610+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:40.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:40.699+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:41.458 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:41.456+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:41.457136+0000) 2022-04-03T22:02:41.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:41.657+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:41.686 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:41.684+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:42.458 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:42.456+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:42.457329+0000) 2022-04-03T22:02:42.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:42.644+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:42.704 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:42.702+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:43.458 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:43.456+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:43.457492+0000) 2022-04-03T22:02:43.621 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:43.618+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:43.675 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:43.672+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:44.459 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:44.456+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:44.457646+0000) 2022-04-03T22:02:44.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:44.644+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:44.650 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:44.648+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:45.459 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:45.457+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:45.457830+0000) 2022-04-03T22:02:45.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:45.654+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:45.659 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:45.657+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:46.459 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:46.457+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:46.458007+0000) 2022-04-03T22:02:46.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:46.661+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:46.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:46.663+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:47.142 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:02:47.140+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:02:47.459 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:47.457+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:47.458163+0000) 2022-04-03T22:02:47.618 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:47.616+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:47.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:47.636+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:48.459 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:48.457+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:48.458315+0000) 2022-04-03T22:02:48.626 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:48.624+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:48.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:48.633+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:49.459 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:49.457+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:49.458448+0000) 2022-04-03T22:02:49.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:49.611+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:49.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:49.677+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:50.460 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:50.458+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:50.458604+0000) 2022-04-03T22:02:50.646 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:50.643+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:50.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:50.697+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:51.460 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:51.458+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:51.458759+0000) 2022-04-03T22:02:51.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:51.689+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:51.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:51.701+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:52.460 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:52.458+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:52.458983+0000) 2022-04-03T22:02:52.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:52.653+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:52.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:52.741+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:52.961 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:02:52.965 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:02:53.460 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:53.458+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:53.459178+0000) 2022-04-03T22:02:53.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:53.608+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:53.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:53.725+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:54.460 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:54.458+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:54.459359+0000) 2022-04-03T22:02:54.600 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:54.597+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:54.694 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:54.691+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:55.461 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:55.459+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:55.459515+0000) 2022-04-03T22:02:55.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:55.581+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:55.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:55.655+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:56.461 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:56.459+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:56.459685+0000) 2022-04-03T22:02:56.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:56.541+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:56.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:56.667+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:57.461 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:57.459+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:57.459888+0000) 2022-04-03T22:02:57.555 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:57.553+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:57.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:57.655+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:58.461 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:58.459+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:58.460071+0000) 2022-04-03T22:02:58.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:58.592+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:58.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:58.689+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:02:59.461 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:02:59.459+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:02:59.460218+0000) 2022-04-03T22:02:59.575 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:02:59.573+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:02:59.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:02:59.716+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:00.461 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:00.460+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:00.460433+0000) 2022-04-03T22:03:00.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:00.575+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:00.678 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:00.676+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:01.462 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:01.460+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:01.460634+0000) 2022-04-03T22:03:01.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:01.574+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:01.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:01.639+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:02.158 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:03:02.156+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:03:02.462 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:02.460+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:02.460821+0000) 2022-04-03T22:03:02.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:02.581+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:02.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:02.679+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:03.462 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:03.460+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:03.461025+0000) 2022-04-03T22:03:03.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:03.626+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:03.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:03.711+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:04.462 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:04.460+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:04.461226+0000) 2022-04-03T22:03:04.645 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:04.643+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:04.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:04.739+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:05.462 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:05.461+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:05.461426+0000) 2022-04-03T22:03:05.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:05.655+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:05.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:05.740+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:06.463 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:06.461+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:06.461630+0000) 2022-04-03T22:03:06.615 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:06.613+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:06.704 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:06.702+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:07.463 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:07.461+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:07.461860+0000) 2022-04-03T22:03:07.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:07.642+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:07.689 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:07.687+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:08.463 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:08.461+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:08.462082+0000) 2022-04-03T22:03:08.622 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:08.620+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:08.727 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:08.725+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:09.463 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:09.462+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:09.462306+0000) 2022-04-03T22:03:09.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:09.617+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:09.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:09.741+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:10.463 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:10.462+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:10.462517+0000) 2022-04-03T22:03:10.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:10.660+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:10.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:10.694+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:11.464 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:11.462+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:11.462722+0000) 2022-04-03T22:03:11.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:11.635+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:11.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:11.707+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:12.464 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:12.462+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:12.462879+0000) 2022-04-03T22:03:12.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:12.600+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:12.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:12.756+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:13.464 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:13.462+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:13.463084+0000) 2022-04-03T22:03:13.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:13.601+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:13.803 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:13.801+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:14.464 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:14.463+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:14.463294+0000) 2022-04-03T22:03:14.600 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:14.598+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:14.853 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:14.850+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:15.464 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:15.462+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:15.463465+0000) 2022-04-03T22:03:15.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:15.580+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:15.880 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:15.877+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:16.465 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:16.462+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:16.463639+0000) 2022-04-03T22:03:16.622 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:16.620+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:16.874 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:16.872+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:17.174 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:03:17.171+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:03:17.465 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:17.462+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:17.463832+0000) 2022-04-03T22:03:17.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:17.621+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:17.908 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:17.905+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:18.465 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:18.462+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:18.463992+0000) 2022-04-03T22:03:18.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:18.592+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:18.875 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:18.873+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:19.465 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:19.463+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:19.464203+0000) 2022-04-03T22:03:19.547 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:19.546+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:19.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:19.833+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:20.465 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:20.463+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:20.464396+0000) 2022-04-03T22:03:20.565 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:20.564+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:20.845 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:20.843+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:21.466 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:21.463+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:21.464616+0000) 2022-04-03T22:03:21.519 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:21.517+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:21.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:21.809+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:22.466 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:22.463+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:22.464799+0000) 2022-04-03T22:03:22.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:22.493+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:22.859 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:22.857+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:23.105 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:03:23.110 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:03:23.449 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:23.447+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:23.466 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:23.464+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:23.465032+0000) 2022-04-03T22:03:23.896 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:23.894+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:24.465 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:24.463+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:24.466 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:24.464+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:24.465246+0000) 2022-04-03T22:03:24.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:24.866+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:25.428 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:25.427+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:25.466 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:25.464+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:25.465463+0000) 2022-04-03T22:03:25.838 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:25.837+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:26.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:26.399+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:26.467 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:26.464+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:26.465693+0000) 2022-04-03T22:03:26.836 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:26.834+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:27.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:27.439+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:27.467 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:27.465+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:27.465873+0000) 2022-04-03T22:03:27.790 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:27.788+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:28.441 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:28.439+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:28.467 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:28.465+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:28.466029+0000) 2022-04-03T22:03:28.818 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:28.816+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:29.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:29.413+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:29.467 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:29.465+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:29.466250+0000) 2022-04-03T22:03:29.792 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:29.790+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:30.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:30.387+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:30.468 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:30.465+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:30.466442+0000) 2022-04-03T22:03:30.794 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:30.792+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:31.387 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:31.385+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:31.468 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:31.465+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:31.466634+0000) 2022-04-03T22:03:31.826 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:31.824+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:32.190 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:03:32.187+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:03:32.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:32.415+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:32.468 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:32.466+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:32.466871+0000) 2022-04-03T22:03:32.825 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:32.823+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:33.466 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:33.464+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:33.468 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:33.466+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:33.467038+0000) 2022-04-03T22:03:33.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:33.821+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:34.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:34.418+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:34.468 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:34.466+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:34.467269+0000) 2022-04-03T22:03:34.778 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:34.776+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:35.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:35.428+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:35.468 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:35.466+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:35.467498+0000) 2022-04-03T22:03:35.759 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:35.756+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:36.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:36.419+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:36.469 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:36.466+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:36.467710+0000) 2022-04-03T22:03:36.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:36.784+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:37.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:37.370+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:37.469 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:37.467+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:37.467963+0000) 2022-04-03T22:03:37.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:37.755+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:38.334 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:38.332+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:38.470 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:38.467+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:38.468170+0000) 2022-04-03T22:03:38.776 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:38.773+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:39.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:39.315+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:39.469 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:39.467+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:39.468428+0000) 2022-04-03T22:03:39.745 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:39.742+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:40.291 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:40.288+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:40.470 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:40.468+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:40.468618+0000) 2022-04-03T22:03:40.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:40.747+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:41.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:41.263+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:41.470 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:41.467+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:41.468748+0000) 2022-04-03T22:03:41.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:41.735+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:42.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:42.303+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:42.470 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:42.468+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:42.468962+0000) 2022-04-03T22:03:42.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:42.722+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:43.270 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:43.267+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:43.470 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:43.468+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:43.469202+0000) 2022-04-03T22:03:43.735 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:43.733+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:44.269 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:44.267+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:44.471 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:44.468+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:44.469445+0000) 2022-04-03T22:03:44.775 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:44.773+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:45.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:45.314+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:45.471 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:45.469+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:45.469692+0000) 2022-04-03T22:03:45.754 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:45.751+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:46.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:46.303+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:46.471 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:46.469+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:46.469899+0000) 2022-04-03T22:03:46.706 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:46.704+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:47.206 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:03:47.203+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:03:47.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:47.349+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:47.471 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:47.469+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:47.470134+0000) 2022-04-03T22:03:47.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:47.747+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:48.374 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:48.372+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:48.471 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:48.469+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:48.470403+0000) 2022-04-03T22:03:48.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:48.734+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:49.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:49.403+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:49.472 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:49.470+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:49.470653+0000) 2022-04-03T22:03:49.772 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:49.770+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:50.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:50.368+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:50.472 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:50.470+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:50.470825+0000) 2022-04-03T22:03:50.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:50.813+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:51.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:51.404+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:51.472 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:51.470+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:51.471018+0000) 2022-04-03T22:03:51.817 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:51.815+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:52.360 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:52.357+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:52.472 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:52.470+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:52.471188+0000) 2022-04-03T22:03:52.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:52.777+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:53.335 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:03:53.339 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:03:53.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:53.370+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:53.472 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:53.470+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:53.471398+0000) 2022-04-03T22:03:53.761 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:53.758+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:54.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:54.359+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:54.473 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:54.471+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:54.471666+0000) 2022-04-03T22:03:54.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:54.733+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:55.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:55.409+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:55.473 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:55.471+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:55.471938+0000) 2022-04-03T22:03:55.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:55.692+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:56.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:56.427+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:56.473 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:56.471+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:56.472165+0000) 2022-04-03T22:03:56.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:56.647+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:57.438 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:57.436+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:57.473 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:57.472+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:57.472383+0000) 2022-04-03T22:03:57.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:57.599+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:58.473 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:58.470+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:58.474 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:58.472+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:58.472630+0000) 2022-04-03T22:03:58.572 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:58.570+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:03:59.474 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:03:59.472+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:03:59.472884+0000) 2022-04-03T22:03:59.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:03:59.483+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:03:59.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:03:59.562+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:00.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:00.456+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:00.474 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:00.472+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:00.473131+0000) 2022-04-03T22:04:00.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:00.596+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:01.424 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:01.422+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:01.474 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:01.472+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:01.473299+0000) 2022-04-03T22:04:01.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:01.546+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:02.222 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:04:02.218+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:04:02.426 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:02.424+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:02.474 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:02.472+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:02.473514+0000) 2022-04-03T22:04:02.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:02.551+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:03.398 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:03.396+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:03.475 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:03.472+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:03.473722+0000) 2022-04-03T22:04:03.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:03.576+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:04.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:04.412+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:04.475 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:04.472+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:04.473913+0000) 2022-04-03T22:04:04.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:04.553+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:05.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:05.370+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:05.475 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:05.473+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:05.474049+0000) 2022-04-03T22:04:05.522 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:05.520+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:06.343 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:06.341+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:06.475 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:06.472+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:06.474184+0000) 2022-04-03T22:04:06.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:06.546+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:07.386 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:07.383+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:07.475 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:07.473+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:07.474393+0000) 2022-04-03T22:04:07.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:07.571+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:08.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:08.375+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:08.475 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:08.473+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:08.474581+0000) 2022-04-03T22:04:08.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:08.553+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:09.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:09.386+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:09.476 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:09.473+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:09.474810+0000) 2022-04-03T22:04:09.507 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:09.504+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:10.397 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:10.392+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:10.478 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:10.474+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:10.475051+0000) 2022-04-03T22:04:10.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:10.536+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:11.372 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:11.370+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:11.476 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:11.474+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:11.475291+0000) 2022-04-03T22:04:11.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:11.563+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:12.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:12.405+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:12.476 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:12.474+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:12.475458+0000) 2022-04-03T22:04:12.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:12.590+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:13.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:13.428+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:13.476 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:13.474+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:13.475629+0000) 2022-04-03T22:04:13.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:13.565+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:14.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:14.437+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:14.477 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:14.475+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:14.475889+0000) 2022-04-03T22:04:14.609 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:14.607+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:15.477 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:15.475+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:15.476129+0000) 2022-04-03T22:04:15.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:15.487+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:15.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:15.578+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:16.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:16.443+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:16.477 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:16.475+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:16.476372+0000) 2022-04-03T22:04:16.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:16.611+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:17.237 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:04:17.235+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:04:17.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:17.429+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:17.478 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:17.475+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:17.476632+0000) 2022-04-03T22:04:17.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:17.626+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:18.386 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:18.385+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:18.478 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:18.476+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:18.476891+0000) 2022-04-03T22:04:18.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:18.652+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:19.343 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:19.341+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:19.478 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:19.476+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:19.477131+0000) 2022-04-03T22:04:19.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:19.668+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:20.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:20.334+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:20.478 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:20.476+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:20.477383+0000) 2022-04-03T22:04:20.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:20.710+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:21.318 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:21.317+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:21.479 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:21.476+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:21.477622+0000) 2022-04-03T22:04:21.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:21.669+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:22.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:22.326+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:22.479 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:22.477+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:22.477892+0000) 2022-04-03T22:04:22.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:22.651+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:23.329 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:23.327+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:23.408 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:04:23.412 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:04:23.479 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:23.477+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:23.478089+0000) 2022-04-03T22:04:23.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:23.677+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:24.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:24.313+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:24.479 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:24.477+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:24.478277+0000) 2022-04-03T22:04:24.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:24.684+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:25.358 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:25.357+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:25.479 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:25.477+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:25.478503+0000) 2022-04-03T22:04:25.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:25.705+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:26.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:26.327+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:26.480 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:26.478+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:26.478752+0000) 2022-04-03T22:04:26.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:26.727+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:27.294 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:27.293+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:27.480 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:27.478+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:27.479029+0000) 2022-04-03T22:04:27.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:27.726+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:28.345 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:28.343+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:28.480 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:28.478+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:28.479280+0000) 2022-04-03T22:04:28.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:28.714+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:29.332 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:29.329+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:29.480 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:29.479+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:29.479468+0000) 2022-04-03T22:04:29.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:29.721+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:30.358 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:30.355+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:30.480 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:30.479+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:30.479687+0000) 2022-04-03T22:04:30.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:30.682+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:31.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:31.354+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:31.481 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:31.479+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:31.479829+0000) 2022-04-03T22:04:31.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:31.711+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:32.253 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:04:32.251+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:04:32.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:32.307+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:32.481 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:32.479+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:32.479998+0000) 2022-04-03T22:04:32.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:32.679+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:33.289 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:33.286+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:33.481 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:33.479+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:33.480215+0000) 2022-04-03T22:04:33.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:33.648+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:34.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:34.308+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:34.481 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:34.480+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:34.480407+0000) 2022-04-03T22:04:34.676 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:34.673+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:35.262 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:35.260+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:35.482 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:35.480+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:35.480639+0000) 2022-04-03T22:04:35.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:35.688+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:36.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:36.278+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:36.482 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:36.480+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:36.480872+0000) 2022-04-03T22:04:36.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:36.687+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:37.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:37.280+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:37.482 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:37.480+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:37.481089+0000) 2022-04-03T22:04:37.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:37.694+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:38.293 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:38.291+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:38.482 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:38.481+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:38.481338+0000) 2022-04-03T22:04:38.674 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:38.671+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:39.252 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:39.249+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:39.483 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:39.481+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:39.481582+0000) 2022-04-03T22:04:39.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:39.657+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:40.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:40.265+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:40.483 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:40.481+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:40.481879+0000) 2022-04-03T22:04:40.659 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:40.657+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:41.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:41.242+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:41.483 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:41.481+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:41.482111+0000) 2022-04-03T22:04:41.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:41.617+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:42.206 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:42.204+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:42.483 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:42.481+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:42.482372+0000) 2022-04-03T22:04:42.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:42.642+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:43.217 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:43.214+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:43.484 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:43.482+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:43.482626+0000) 2022-04-03T22:04:43.659 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:43.657+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:44.190 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:44.188+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:44.484 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:44.481+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:44.482879+0000) 2022-04-03T22:04:44.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:44.647+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:45.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:45.230+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:45.484 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:45.482+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:45.483122+0000) 2022-04-03T22:04:45.634 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:45.632+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:46.210 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:46.207+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:46.484 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:46.481+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:46.483295+0000) 2022-04-03T22:04:46.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:46.633+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:47.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:47.196+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:47.269 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:04:47.265+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:04:47.484 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:47.482+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:47.483512+0000) 2022-04-03T22:04:47.623 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:47.621+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:48.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:48.186+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:48.485 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:48.482+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:48.483761+0000) 2022-04-03T22:04:48.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:48.646+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:49.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:49.185+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:49.485 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:49.483+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:49.484034+0000) 2022-04-03T22:04:49.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:49.662+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:50.181 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:50.178+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:50.485 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:50.483+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:50.484283+0000) 2022-04-03T22:04:50.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:50.616+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:51.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:51.205+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:51.486 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:51.483+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:51.484501+0000) 2022-04-03T22:04:51.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:51.572+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:52.199 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:52.197+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:52.486 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:52.483+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:52.484751+0000) 2022-04-03T22:04:52.572 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:52.570+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:53.228 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:53.226+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:53.486 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:53.484+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:53.485021+0000) 2022-04-03T22:04:53.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:53.566+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:53.576 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:04:53.580 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:04:54.193 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:54.191+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:54.486 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:54.484+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:54.485247+0000) 2022-04-03T22:04:54.583 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:54.580+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:55.177 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:55.175+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:55.486 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:55.484+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:55.485479+0000) 2022-04-03T22:04:55.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:55.553+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:56.202 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:56.200+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:56.487 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:56.484+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:56.485730+0000) 2022-04-03T22:04:56.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:56.592+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:57.176 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:57.174+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:57.487 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:57.485+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:57.486011+0000) 2022-04-03T22:04:57.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:57.591+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:58.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:58.124+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:58.487 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:58.485+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:58.486207+0000) 2022-04-03T22:04:58.582 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:58.580+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:04:59.114 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:04:59.112+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:04:59.487 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:04:59.485+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:04:59.486443+0000) 2022-04-03T22:04:59.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:04:59.628+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:00.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:00.129+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:00.488 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:00.485+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:00.486679+0000) 2022-04-03T22:05:00.609 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:00.607+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:01.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:01.171+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:01.488 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:01.486+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:01.486847+0000) 2022-04-03T22:05:01.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:01.585+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:02.172 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:02.169+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:02.278 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:05:02.276+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:05:02.488 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:02.486+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:02.486996+0000) 2022-04-03T22:05:02.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:02.621+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:03.191 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:03.188+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:03.488 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:03.486+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:03.487166+0000) 2022-04-03T22:05:03.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:03.579+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:04.240 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:04.238+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:04.488 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:04.486+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:04.487388+0000) 2022-04-03T22:05:04.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:04.567+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:05.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:05.209+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:05.489 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:05.487+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:05.487586+0000) 2022-04-03T22:05:05.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:05.587+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:06.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:06.194+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:06.489 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:06.487+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:06.487827+0000) 2022-04-03T22:05:06.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:06.588+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:07.217 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:07.215+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:07.489 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:07.487+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:07.488076+0000) 2022-04-03T22:05:07.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:07.592+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:08.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:08.171+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:08.489 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:08.487+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:08.488326+0000) 2022-04-03T22:05:08.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:08.568+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:09.166 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:09.164+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:09.489 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:09.488+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:09.488541+0000) 2022-04-03T22:05:09.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:09.528+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:10.192 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:10.190+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:10.490 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:10.488+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:10.488716+0000) 2022-04-03T22:05:10.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:10.540+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:11.224 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:11.222+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:11.490 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:11.488+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:11.488940+0000) 2022-04-03T22:05:11.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:11.533+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:12.191 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:12.189+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:12.490 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:12.488+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:12.489197+0000) 2022-04-03T22:05:12.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:12.559+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:13.157 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:13.156+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:13.491 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:13.488+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:13.489487+0000) 2022-04-03T22:05:13.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:13.555+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:14.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:14.193+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:14.491 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:14.489+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:14.489660+0000) 2022-04-03T22:05:14.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:14.569+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:15.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:15.224+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:15.491 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:15.489+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:15.489860+0000) 2022-04-03T22:05:15.585 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:15.583+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:16.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:16.234+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:16.491 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:16.489+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:16.490077+0000) 2022-04-03T22:05:16.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:16.587+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:17.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:17.217+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:17.294 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:05:17.292+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:05:17.491 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:17.489+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:17.490244+0000) 2022-04-03T22:05:17.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:17.637+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:18.263 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:18.261+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:18.492 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:18.490+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:18.490477+0000) 2022-04-03T22:05:18.634 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:18.632+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:19.233 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:19.231+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:19.492 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:19.490+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:19.490670+0000) 2022-04-03T22:05:19.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:19.617+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:20.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:20.186+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:20.492 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:20.490+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:20.490886+0000) 2022-04-03T22:05:20.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:20.591+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:21.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:21.169+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:21.492 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:21.490+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:21.491107+0000) 2022-04-03T22:05:21.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:21.573+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:22.200 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:22.199+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:22.492 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:22.491+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:22.491345+0000) 2022-04-03T22:05:22.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:22.592+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:23.177 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:23.175+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:23.492 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:23.491+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:23.491547+0000) 2022-04-03T22:05:23.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:23.588+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:23.716 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:05:23.720 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:05:24.213 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:24.211+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:24.493 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:24.491+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:24.491733+0000) 2022-04-03T22:05:24.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:24.590+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:25.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:25.180+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:05:25.493 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:25.491+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:25.491948+0000) 2022-04-03T22:05:25.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:25.611+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:26.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:26.153+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:26.493 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:26.492+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:26.492119+0000) 2022-04-03T22:05:26.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:26.579+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:27.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:27.135+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:27.493 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:27.492+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:27.492291+0000) 2022-04-03T22:05:27.573 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:27.570+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:28.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:28.151+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:28.494 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:28.491+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:28.492501+0000) 2022-04-03T22:05:28.537 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:28.534+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:29.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:29.132+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:29.494 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:29.491+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:29.492696+0000) 2022-04-03T22:05:29.544 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:29.541+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:30.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:30.104+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:30.494 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:30.491+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:30.492880+0000) 2022-04-03T22:05:30.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:30.546+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:31.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:31.061+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:31.494 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:31.492+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:31.493080+0000) 2022-04-03T22:05:31.585 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:31.583+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:32.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:32.080+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:32.310 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:05:32.306+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:05:32.494 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:32.492+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:32.493285+0000) 2022-04-03T22:05:32.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:32.564+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:33.106 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:33.103+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:33.495 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:33.492+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:33.493494+0000) 2022-04-03T22:05:33.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:33.613+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:34.094 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:34.091+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:34.495 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:34.492+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:34.493643+0000) 2022-04-03T22:05:34.661 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:34.658+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:35.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:35.136+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:35.495 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:35.492+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:35.493869+0000) 2022-04-03T22:05:35.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:35.642+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:36.129 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:36.126+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:36.495 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:36.493+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:36.494041+0000) 2022-04-03T22:05:36.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:36.680+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:37.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:37.081+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:37.495 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:37.493+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:37.494251+0000) 2022-04-03T22:05:37.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:37.730+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:38.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:38.046+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:38.495 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:38.493+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:38.494459+0000) 2022-04-03T22:05:38.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:38.767+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:39.098 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:39.095+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:39.496 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:39.493+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:39.494662+0000) 2022-04-03T22:05:39.807 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:39.805+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:40.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:40.125+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:40.496 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:40.494+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:40.494890+0000) 2022-04-03T22:05:40.835 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:40.833+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:41.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:41.168+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:41.496 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:41.494+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:41.495062+0000) 2022-04-03T22:05:41.801 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:41.799+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:42.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:42.152+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:42.496 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:42.494+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:42.495249+0000) 2022-04-03T22:05:42.816 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:42.814+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:43.196 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:43.193+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:43.496 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:43.494+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:43.495400+0000) 2022-04-03T22:05:43.821 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:43.819+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:44.234 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:44.232+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:44.497 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:44.494+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:44.495605+0000) 2022-04-03T22:05:44.774 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:44.772+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:45.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:45.269+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:45.497 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:45.495+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:45.495753+0000) 2022-04-03T22:05:45.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:45.775+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:46.224 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:46.222+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:46.497 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:46.495+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:46.495968+0000) 2022-04-03T22:05:46.798 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:46.796+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:47.258 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:47.256+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:47.325 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:05:47.323+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:05:47.497 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:47.495+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:47.496169+0000) 2022-04-03T22:05:47.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:47.818+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:48.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:48.224+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:48.497 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:48.495+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:48.496328+0000) 2022-04-03T22:05:48.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:48.865+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:49.238 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:49.235+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:49.498 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:49.495+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:49.496533+0000) 2022-04-03T22:05:49.837 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:49.835+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:50.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:50.196+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:50.498 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:50.496+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:50.496745+0000) 2022-04-03T22:05:50.798 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:50.795+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:51.243 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:51.241+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:51.498 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:51.496+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:51.496940+0000) 2022-04-03T22:05:51.765 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:51.762+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:52.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:52.216+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:52.498 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:52.496+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:52.497121+0000) 2022-04-03T22:05:52.784 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:52.782+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:53.220 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:53.218+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:53.498 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:53.496+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:53.497332+0000) 2022-04-03T22:05:53.805 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:53.803+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:53.839 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:05:53.842 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:05:54.255 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:54.253+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:54.499 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:54.497+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:54.497578+0000) 2022-04-03T22:05:54.772 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:54.770+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:55.304 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:55.302+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:55.499 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:55.497+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:55.497731+0000) 2022-04-03T22:05:55.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:55.818+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:56.266 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:56.263+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:56.499 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:56.497+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:56.497950+0000) 2022-04-03T22:05:56.835 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:56.833+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:57.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:57.266+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:57.499 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:57.497+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:57.498161+0000) 2022-04-03T22:05:57.823 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:57.820+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:58.230 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:58.228+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:58.500 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:58.497+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:58.498353+0000) 2022-04-03T22:05:58.825 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:58.823+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:05:59.205 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:05:59.203+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:05:59.499 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:05:59.498+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:05:59.498543+0000) 2022-04-03T22:05:59.814 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:05:59.812+0000 7f88d0ac1700 -1 osd.4 126 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:00.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:00.186+0000 7fd3b190b700 -1 osd.6 126 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:00.500 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:00.498+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:00.498760+0000) 2022-04-03T22:06:00.709 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-03T22:06:00.706+0000 7f147e307700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T22:06:00.724 INFO:tasks.ceph.mds.f.smithi186.stderr: -1> 2022-04-03T22:06:00.706+0000 7f147e307700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T22:06:00.746 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-03T22:06:00.853 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:00.850+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:01.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:01.233+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:01.500 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:01.498+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:01.498957+0000) 2022-04-03T22:06:01.840 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:01.838+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:02.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:02.271+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:02.341 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:06:02.339+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:06:02.500 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:02.498+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:02.499160+0000) 2022-04-03T22:06:02.809 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:02.807+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:03.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:03.282+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:03.500 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:03.499+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:03.499372+0000) 2022-04-03T22:06:03.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:03.804+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:04.293 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:04.292+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:04.501 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:04.499+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:04.499588+0000) 2022-04-03T22:06:04.827 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:04.825+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:05.325 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:05.323+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:05.501 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:05.499+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:05.499813+0000) 2022-04-03T22:06:05.810 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:05.808+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 28 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:06.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:06.297+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:06.501 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:06.499+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:06.499982+0000) 2022-04-03T22:06:06.819 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:06.817+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:07.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:07.255+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:07.501 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:07.499+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:07.500202+0000) 2022-04-03T22:06:07.810 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:07.808+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:08.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:08.226+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:08.501 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:08.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:08.500409+0000) 2022-04-03T22:06:08.856 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:08.854+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:09.181 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:09.179+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:09.502 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:09.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:09.500619+0000) 2022-04-03T22:06:09.826 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:09.824+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:10.230 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:10.229+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:10.502 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:10.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:10.500859+0000) 2022-04-03T22:06:10.845 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:10.843+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:11.230 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:11.228+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:11.502 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:11.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:11.501066+0000) 2022-04-03T22:06:11.881 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:11.879+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:12.196 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:12.195+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:12.502 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:12.501+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:12.501256+0000) 2022-04-03T22:06:12.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:12.895+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:13.180 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:13.178+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:13.503 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:13.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:13.501470+0000) 2022-04-03T22:06:13.935 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:13.933+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:14.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:14.195+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:14.503 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:14.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:14.501677+0000) 2022-04-03T22:06:14.928 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:14.927+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:15.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:15.218+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:15.503 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:15.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:15.501897+0000) 2022-04-03T22:06:15.944 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:15.942+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:16.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:16.206+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:16.503 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:16.501+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:16.502112+0000) 2022-04-03T22:06:16.941 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:16.940+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:17.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:17.180+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:17.357 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:06:17.354+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:06:17.503 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:17.501+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:17.502319+0000) 2022-04-03T22:06:17.927 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:17.924+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:18.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:18.195+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:18.503 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:18.501+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:18.502544+0000) 2022-04-03T22:06:18.917 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:18.914+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:19.162 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:19.159+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:19.504 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:19.501+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:19.502759+0000) 2022-04-03T22:06:19.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:19.867+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:20.200 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:20.197+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:20.504 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:20.501+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:20.502992+0000) 2022-04-03T22:06:20.892 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:20.889+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:21.220 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:21.217+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:21.504 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:21.502+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:21.503190+0000) 2022-04-03T22:06:21.850 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:21.847+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:22.210 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:22.208+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:22.504 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:22.502+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:22.503404+0000) 2022-04-03T22:06:22.898 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:22.896+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:23.228 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:23.225+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:23.505 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:23.502+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:23.503636+0000) 2022-04-03T22:06:23.925 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:23.922+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:23.986 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:06:23.989 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:06:24.267 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:24.265+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:24.505 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:24.503+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:24.503888+0000) 2022-04-03T22:06:24.961 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:24.959+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:25.240 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:25.237+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:25.505 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:25.503+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:25.504085+0000) 2022-04-03T22:06:25.954 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:25.951+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:26.271 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:26.269+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:26.505 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:26.503+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:26.504279+0000) 2022-04-03T22:06:26.947 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:26.944+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:27.266 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:27.263+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:27.505 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:27.503+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:27.504476+0000) 2022-04-03T22:06:27.924 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:27.921+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:28.301 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:28.299+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:28.506 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:28.503+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:28.504669+0000) 2022-04-03T22:06:28.935 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:28.933+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:29.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:29.308+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:29.506 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:29.504+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:29.504869+0000) 2022-04-03T22:06:29.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:29.941+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:30.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:30.271+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:30.506 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:30.504+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:30.505049+0000) 2022-04-03T22:06:30.894 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:30.891+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:31.264 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:31.262+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:31.506 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:31.504+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:31.505244+0000) 2022-04-03T22:06:31.926 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:31.924+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:32.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:32.306+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:32.373 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:06:32.370+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:06:32.507 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:32.504+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:32.505450+0000) 2022-04-03T22:06:32.955 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:32.952+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:33.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:33.257+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:33.507 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:33.505+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:33.505632+0000) 2022-04-03T22:06:33.990 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:33.987+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:34.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:34.233+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:34.507 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:34.505+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:34.505852+0000) 2022-04-03T22:06:34.985 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:34.983+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:35.224 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:35.221+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:35.507 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:35.505+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:35.506045+0000) 2022-04-03T22:06:36.018 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:36.016+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:36.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:36.256+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:36.507 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:36.505+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:36.506246+0000) 2022-04-03T22:06:37.047 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:37.045+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:37.301 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:37.298+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:37.507 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:37.505+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:37.506407+0000) 2022-04-03T22:06:38.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:38.083+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:38.343 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:38.341+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:38.508 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:38.505+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:38.506640+0000) 2022-04-03T22:06:39.061 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:39.059+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:39.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:39.373+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:39.508 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:39.506+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:39.506856+0000) 2022-04-03T22:06:40.037 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:40.035+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:40.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:40.338+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:40.508 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:40.506+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:40.507027+0000) 2022-04-03T22:06:41.033 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:41.030+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:41.354 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:41.351+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:41.508 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:41.506+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:41.507230+0000) 2022-04-03T22:06:42.001 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:41.999+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:42.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:42.361+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:42.514 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:42.507+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:42.507471+0000) 2022-04-03T22:06:43.023 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:43.020+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:43.366 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:43.363+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:43.509 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:43.507+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:43.507671+0000) 2022-04-03T22:06:44.062 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:44.060+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:44.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:44.401+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:44.509 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:44.507+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:44.507869+0000) 2022-04-03T22:06:45.048 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:45.045+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:45.379 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:45.377+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:45.509 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:45.507+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:45.508046+0000) 2022-04-03T22:06:46.050 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:46.048+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:46.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:46.401+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:46.509 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:46.507+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:46.508250+0000) 2022-04-03T22:06:47.070 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:47.067+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:47.388 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:06:47.386+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:06:47.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:47.396+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:47.509 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:47.508+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:47.508447+0000) 2022-04-03T22:06:48.058 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:48.055+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:48.438 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:48.435+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:48.509 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:48.508+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:48.508648+0000) 2022-04-03T22:06:49.054 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:49.051+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:49.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:49.445+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:49.510 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:49.508+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:49.508799+0000) 2022-04-03T22:06:50.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:50.083+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:50.446 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:50.444+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:50.510 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:50.508+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:50.509023+0000) 2022-04-03T22:06:51.130 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:51.127+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:51.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:51.415+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:51.510 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:51.509+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:51.509222+0000) 2022-04-03T22:06:52.165 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:52.163+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:52.428 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:52.426+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:52.510 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:52.509+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:52.509420+0000) 2022-04-03T22:06:53.143 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:53.140+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:53.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:53.419+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:53.511 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:53.509+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:53.509626+0000) 2022-04-03T22:06:54.111 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:54.109+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:54.127 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:06:54.131 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:06:54.469 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:54.467+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:54.511 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:54.509+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:54.509894+0000) 2022-04-03T22:06:55.100 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:55.098+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:55.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:55.429+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:55.511 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:55.509+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:55.510046+0000) 2022-04-03T22:06:56.129 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:56.127+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:56.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:56.443+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:56.511 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:56.510+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:56.510250+0000) 2022-04-03T22:06:57.119 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:57.117+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:57.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:57.477+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:57.511 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:57.509+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:57.510449+0000) 2022-04-03T22:06:58.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:58.151+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:58.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:58.442+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:58.512 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:58.509+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:58.510655+0000) 2022-04-03T22:06:59.109 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:06:59.107+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:06:59.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:06:59.415+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:06:59.512 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:06:59.509+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:06:59.510856+0000) 2022-04-03T22:07:00.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:00.103+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:00.428 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:00.426+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:00.512 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:00.510+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:00.511069+0000) 2022-04-03T22:07:01.103 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:01.101+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:01.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:01.387+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:01.512 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:01.510+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:01.511290+0000) 2022-04-03T22:07:02.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:02.104+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:02.404 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:07:02.401+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:07:02.422 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:02.420+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:02.512 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:02.510+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:02.511490+0000) 2022-04-03T22:07:03.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:03.151+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:03.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:03.457+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:03.513 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:03.510+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:03.511692+0000) 2022-04-03T22:07:04.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:04.167+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:04.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:04.479+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:04.513 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:04.510+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:04.511840+0000) 2022-04-03T22:07:05.134 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:05.132+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:05.473 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:05.471+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:05.513 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:05.511+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:05.512047+0000) 2022-04-03T22:07:06.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:06.157+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:06.457 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:06.455+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:06.513 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:06.511+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:06.512248+0000) 2022-04-03T22:07:07.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:07.176+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:07.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:07.495+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:07.513 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:07.511+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:07.512396+0000) 2022-04-03T22:07:08.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:08.177+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:08.514 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:08.511+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:08.512607+0000) 2022-04-03T22:07:08.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:08.533+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:09.183 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:09.182+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:09.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:09.511+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:09.514 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:09.512+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:09.512853+0000) 2022-04-03T22:07:10.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:10.213+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:10.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:10.489+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:10.514 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:10.512+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:10.513084+0000) 2022-04-03T22:07:11.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:11.221+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:11.514 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:11.512+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:11.513299+0000) 2022-04-03T22:07:11.524 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:11.522+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:12.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:12.190+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:12.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:12.492+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:12.514 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:12.512+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:12.513514+0000) 2022-04-03T22:07:13.205 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:13.203+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:13.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:13.512+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:13.515 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:13.513+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:13.513699+0000) 2022-04-03T22:07:14.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:14.171+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:14.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:14.495+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:14.515 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:14.513+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:14.513855+0000) 2022-04-03T22:07:15.144 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:15.142+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:15.515 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:15.513+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:15.514066+0000) 2022-04-03T22:07:15.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:15.534+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:16.107 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:16.104+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:16.516 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:16.513+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:16.514271+0000) 2022-04-03T22:07:16.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:16.550+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:17.149 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:17.146+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:17.420 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:07:17.418+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:07:17.516 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:17.513+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:17.514497+0000) 2022-04-03T22:07:17.559 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:17.556+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:18.106 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:18.103+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:18.516 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:18.514+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:18.514704+0000) 2022-04-03T22:07:18.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:18.565+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:19.091 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:19.089+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:19.516 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:19.514+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:19.514917+0000) 2022-04-03T22:07:19.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:19.532+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:20.086 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:20.084+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:20.516 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:20.514+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:20.515144+0000) 2022-04-03T22:07:20.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:20.535+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:21.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:21.079+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:21.516 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:21.514+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:21.515384+0000) 2022-04-03T22:07:21.522 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:21.520+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:22.117 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:22.115+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:22.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:22.490+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:22.516 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:22.515+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:22.515587+0000) 2022-04-03T22:07:23.158 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:23.155+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:23.505 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:23.503+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:23.517 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:23.515+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:23.515812+0000) 2022-04-03T22:07:24.162 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:24.160+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:24.256 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:07:24.259 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:07:24.504 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:24.502+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:24.517 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:24.515+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:24.516055+0000) 2022-04-03T22:07:25.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:25.154+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:25.517 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:25.515+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:25.516264+0000) 2022-04-03T22:07:25.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:25.541+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:26.143 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:26.140+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:26.517 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:26.516+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:26.516484+0000) 2022-04-03T22:07:26.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:26.564+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:27.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:27.102+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:27.518 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:27.516+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:27.516692+0000) 2022-04-03T22:07:27.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:27.516+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:28.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:28.079+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:28.518 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:28.516+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:28.516902+0000) 2022-04-03T22:07:28.566 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:28.563+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:29.056 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:29.052+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:29.518 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:29.516+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:29.517114+0000) 2022-04-03T22:07:29.522 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:29.520+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:30.098 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:30.096+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:30.518 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:30.516+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:30.517297+0000) 2022-04-03T22:07:30.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:30.564+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:31.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:31.078+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:31.519 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:31.517+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:31.517506+0000) 2022-04-03T22:07:31.612 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:31.610+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:32.050 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:32.045+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:32.436 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:07:32.433+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:07:32.519 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:32.517+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:32.517689+0000) 2022-04-03T22:07:32.634 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:32.631+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:33.056 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:33.054+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:33.519 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:33.517+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:33.517899+0000) 2022-04-03T22:07:33.639 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:33.636+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:34.050 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:34.048+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:34.519 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:34.517+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:34.518082+0000) 2022-04-03T22:07:34.633 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:34.631+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:35.024 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:35.022+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:35.519 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:35.518+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:35.518293+0000) 2022-04-03T22:07:35.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:35.623+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:36.056 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:36.053+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:36.520 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:36.518+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:36.518506+0000) 2022-04-03T22:07:36.659 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:36.657+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:37.095 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:37.092+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:37.520 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:37.517+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:37.518707+0000) 2022-04-03T22:07:37.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:37.645+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:38.080 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:38.078+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:38.520 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:38.518+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:38.518890+0000) 2022-04-03T22:07:38.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:38.623+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:39.126 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:39.124+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:39.520 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:39.518+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:39.519014+0000) 2022-04-03T22:07:39.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:39.663+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:40.095 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:40.092+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:40.520 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:40.519+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:40.519258+0000) 2022-04-03T22:07:40.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:40.680+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:41.113 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:41.110+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:41.521 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:41.518+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:41.519489+0000) 2022-04-03T22:07:41.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:41.685+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:42.107 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:42.105+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:42.521 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:42.518+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:42.519762+0000) 2022-04-03T22:07:42.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:42.654+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:43.108 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:43.106+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:43.521 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:43.518+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:43.519984+0000) 2022-04-03T22:07:43.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:43.609+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:44.093 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:44.090+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:44.521 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:44.519+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:44.520223+0000) 2022-04-03T22:07:44.639 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:44.637+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:45.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:45.053+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:45.522 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:45.519+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:45.520493+0000) 2022-04-03T22:07:45.615 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:45.613+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:46.046 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:46.044+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:46.522 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:46.519+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:46.520712+0000) 2022-04-03T22:07:46.609 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:46.607+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:47.094 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:47.092+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:47.451 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:07:47.448+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:07:47.522 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:47.519+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:47.520932+0000) 2022-04-03T22:07:47.570 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:47.568+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:48.062 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:48.060+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:48.522 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:48.520+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:48.521157+0000) 2022-04-03T22:07:48.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:48.551+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:49.053 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:49.050+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:49.508 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:49.507+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:49.522 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:49.520+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:49.521398+0000) 2022-04-03T22:07:50.066 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:50.064+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:50.465 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:50.463+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:50.523 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:50.520+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:50.521617+0000) 2022-04-03T22:07:51.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:51.072+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:51.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:51.433+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:51.523 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:51.520+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:51.521781+0000) 2022-04-03T22:07:52.076 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:52.074+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:52.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:52.388+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:52.523 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:52.521+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:52.521976+0000) 2022-04-03T22:07:53.060 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:53.058+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:53.366 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:53.365+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:53.523 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:53.521+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:53.522182+0000) 2022-04-03T22:07:54.017 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:54.016+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:54.397 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:07:54.401 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:07:54.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:54.410+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:54.523 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:54.521+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:54.522428+0000) 2022-04-03T22:07:55.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:55.014+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:55.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:55.443+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:55.524 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:55.521+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:55.522690+0000) 2022-04-03T22:07:55.988 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:55.987+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:56.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:56.491+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:56.524 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:56.522+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:56.522941+0000) 2022-04-03T22:07:56.942 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:56.941+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:57.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:57.508+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:57.524 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:57.522+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:57.523175+0000) 2022-04-03T22:07:57.932 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:57.931+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:58.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:58.482+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:58.525 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:58.522+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:58.523374+0000) 2022-04-03T22:07:58.957 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:07:58.955+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:07:59.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:07:59.458+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:07:59.525 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:07:59.522+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:07:59.523582+0000) 2022-04-03T22:08:00.006 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:00.005+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:00.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:00.456+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:00.525 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:00.523+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:00.523859+0000) 2022-04-03T22:08:01.025 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:01.023+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:01.498 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:01.494+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:01.526 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:01.523+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:01.524046+0000) 2022-04-03T22:08:02.011 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:02.009+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:02.467 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:08:02.465+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:08:02.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:02.488+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:02.525 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:02.523+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:02.524193+0000) 2022-04-03T22:08:02.972 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:02.970+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:03.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:03.488+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:03.525 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:03.523+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:03.524418+0000) 2022-04-03T22:08:04.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:04.011+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:04.526 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:04.524+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:04.526 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:04.524+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:04.524657+0000) 2022-04-03T22:08:04.999 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:04.997+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:05.526 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:05.524+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:05.524906+0000) 2022-04-03T22:08:05.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:05.550+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:05.980 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:05.978+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:06.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:06.519+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:06.526 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:06.524+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:06.525155+0000) 2022-04-03T22:08:06.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:06.992+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:07.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:07.490+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:07.526 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:07.524+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:07.525314+0000) 2022-04-03T22:08:08.036 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:08.033+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:08.527 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:08.525+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:08.525522+0000) 2022-04-03T22:08:08.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:08.528+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:09.080 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:09.078+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:09.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:09.500+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:09.527 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:09.525+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:09.525763+0000) 2022-04-03T22:08:10.062 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:10.059+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:10.482 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:10.479+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:10.527 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:10.525+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:10.526004+0000) 2022-04-03T22:08:11.080 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:11.078+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:11.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:11.437+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:11.527 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:11.525+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:11.526234+0000) 2022-04-03T22:08:12.063 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:12.061+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:12.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:12.410+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:12.527 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:12.526+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:12.526432+0000) 2022-04-03T22:08:13.035 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:13.032+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:13.384 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:13.381+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:13.528 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:13.526+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:13.526571+0000) 2022-04-03T22:08:14.000 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:13.998+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:14.432 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:14.430+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:14.528 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:14.526+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:14.526756+0000) 2022-04-03T22:08:15.030 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:15.027+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:15.428 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:15.425+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:15.528 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:15.526+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:15.527009+0000) 2022-04-03T22:08:16.024 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:16.021+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:16.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:16.433+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:16.528 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:16.526+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:16.527255+0000) 2022-04-03T22:08:17.046 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:17.044+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:17.471 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:17.468+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:17.483 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:08:17.481+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:08:17.529 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:17.527+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:17.527524+0000) 2022-04-03T22:08:18.084 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:18.081+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:18.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:18.473+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:18.529 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:18.527+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:18.527819+0000) 2022-04-03T22:08:19.080 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:19.078+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:19.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:19.491+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:19.529 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:19.527+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:19.528061+0000) 2022-04-03T22:08:20.121 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:20.118+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:20.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:20.476+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:20.529 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:20.528+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:20.528313+0000) 2022-04-03T22:08:21.084 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:21.082+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:21.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:21.483+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:21.529 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:21.528+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:21.528555+0000) 2022-04-03T22:08:22.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:22.078+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:22.442 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:22.439+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:22.530 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:22.528+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:22.528825+0000) 2022-04-03T22:08:23.083 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:23.081+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:23.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:23.412+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:23.530 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:23.528+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:23.529068+0000) 2022-04-03T22:08:24.037 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:24.035+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:24.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:24.434+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:24.530 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:24.529+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:24.529310+0000) 2022-04-03T22:08:24.559 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:08:24.563 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:08:25.019 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:25.017+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:25.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:25.451+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:25.531 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:25.529+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:25.529570+0000) 2022-04-03T22:08:26.030 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:26.028+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:26.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:26.435+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:26.531 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:26.528+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:26.529737+0000) 2022-04-03T22:08:27.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:27.053+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:27.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:27.421+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:27.531 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:27.528+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:27.530001+0000) 2022-04-03T22:08:28.022 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:28.020+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:28.473 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:28.471+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:28.531 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:28.529+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:28.530251+0000) 2022-04-03T22:08:29.070 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:29.068+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:29.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:29.515+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:29.531 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:29.529+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:29.530518+0000) 2022-04-03T22:08:30.050 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:30.048+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:30.532 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:30.529+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:30.530743+0000) 2022-04-03T22:08:30.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:30.535+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:31.002 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:31.000+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:31.532 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:31.530+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:31.531022+0000) 2022-04-03T22:08:31.555 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:31.553+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:31.954 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:31.952+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:32.499 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:08:32.496+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:08:32.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:32.520+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:32.532 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:32.530+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:32.531268+0000) 2022-04-03T22:08:32.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:32.931+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:33.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:33.500+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:33.532 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:33.529+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:33.531513+0000) 2022-04-03T22:08:33.903 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:33.901+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:34.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:34.508+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:34.533 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:34.530+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:34.531729+0000) 2022-04-03T22:08:34.892 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:34.889+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:35.533 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:35.531+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:35.531999+0000) 2022-04-03T22:08:35.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:35.542+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:35.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:35.876+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:36.533 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:36.531+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:36.532228+0000) 2022-04-03T22:08:36.558 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:36.556+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:36.844 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:36.842+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:37.533 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:37.531+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:37.532408+0000) 2022-04-03T22:08:37.555 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:37.553+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:37.876 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:37.874+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:38.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:38.511+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:38.533 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:38.531+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:38.532635+0000) 2022-04-03T22:08:38.839 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:38.836+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:39.502 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:39.500+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:39.534 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:39.532+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:39.532851+0000) 2022-04-03T22:08:39.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:39.859+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:40.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:40.486+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:40.534 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:40.532+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:40.533021+0000) 2022-04-03T22:08:40.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:40.859+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:41.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:41.449+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:41.534 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:41.532+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:41.533251+0000) 2022-04-03T22:08:41.871 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:41.869+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:42.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:42.492+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:42.534 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:42.532+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:42.533508+0000) 2022-04-03T22:08:42.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:42.861+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:43.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:43.470+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:43.535 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:43.533+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:43.533731+0000) 2022-04-03T22:08:43.876 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:43.874+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:44.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:44.427+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:44.535 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:44.533+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:44.534013+0000) 2022-04-03T22:08:44.894 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:44.892+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:45.432 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:45.430+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:45.535 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:45.533+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:45.534260+0000) 2022-04-03T22:08:45.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:45.931+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:46.410 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:46.408+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:46.536 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:46.533+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:46.534511+0000) 2022-04-03T22:08:46.920 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:46.919+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:47.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:47.435+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:47.503 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:08:47.501+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:08:47.536 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:47.534+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:47.534812+0000) 2022-04-03T22:08:47.929 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:47.928+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:48.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:48.409+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:48.536 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:48.534+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:48.535095+0000) 2022-04-03T22:08:48.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:48.888+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:49.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:49.374+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:49.536 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:49.534+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:49.535347+0000) 2022-04-03T22:08:49.922 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:49.921+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:50.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:50.416+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:50.536 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:50.535+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:50.535610+0000) 2022-04-03T22:08:50.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:50.951+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:51.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:51.413+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:51.537 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:51.535+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:51.535875+0000) 2022-04-03T22:08:51.925 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:51.923+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:52.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:52.456+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:52.537 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:52.535+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:52.536068+0000) 2022-04-03T22:08:52.974 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:52.972+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:53.441 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:53.439+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:53.537 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:53.535+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:53.536273+0000) 2022-04-03T22:08:53.964 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:53.962+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:54.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:54.419+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:54.537 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:54.536+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:54.536502+0000) 2022-04-03T22:08:54.691 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:08:54.695 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:08:54.935 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:54.934+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:55.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:55.386+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:55.538 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:55.536+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:55.536798+0000) 2022-04-03T22:08:55.904 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:55.903+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:56.355 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:56.354+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:56.538 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:56.536+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:56.537100+0000) 2022-04-03T22:08:56.954 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:56.952+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:57.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:57.392+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:57.538 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:57.536+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:57.537325+0000) 2022-04-03T22:08:57.973 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:57.972+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:58.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:58.403+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:58.539 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:58.537+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:58.537592+0000) 2022-04-03T22:08:58.925 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:58.923+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:08:59.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:08:59.405+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:08:59.539 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:08:59.537+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:08:59.537873+0000) 2022-04-03T22:08:59.895 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:08:59.894+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:00.387 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:00.386+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:00.539 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:00.537+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:00.538134+0000) 2022-04-03T22:09:00.920 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:00.918+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:01.348 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:01.345+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:01.539 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:01.538+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:01.538331+0000) 2022-04-03T22:09:01.904 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:01.900+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:02.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:02.378+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:02.519 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:09:02.517+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:09:02.539 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:02.538+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:02.538504+0000) 2022-04-03T22:09:02.924 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:02.922+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:03.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:03.428+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:03.540 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:03.538+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:03.538653+0000) 2022-04-03T22:09:03.907 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:03.905+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:04.477 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:04.475+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:04.540 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:04.538+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:04.538897+0000) 2022-04-03T22:09:04.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:04.886+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:05.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:05.448+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:05.540 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:05.538+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:05.539106+0000) 2022-04-03T22:09:05.859 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:05.856+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 13 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:06.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:06.402+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:06.540 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:06.539+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:06.539366+0000) 2022-04-03T22:09:06.871 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:06.868+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:07.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:07.429+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:07.541 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:07.539+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:07.539614+0000) 2022-04-03T22:09:07.901 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:07.898+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:08.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:08.387+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:08.541 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:08.539+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:08.539807+0000) 2022-04-03T22:09:08.870 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:08.867+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:09.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:09.346+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:09.541 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:09.539+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:09.539978+0000) 2022-04-03T22:09:09.857 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:09.855+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:10.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:10.338+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:10.541 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:10.538+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:10.540165+0000) 2022-04-03T22:09:10.861 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:10.859+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:11.364 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:11.362+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:11.541 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:11.539+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:11.540403+0000) 2022-04-03T22:09:11.867 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:11.865+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:12.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:12.337+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:12.542 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:12.539+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:12.540578+0000) 2022-04-03T22:09:12.827 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:12.825+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:13.356 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:13.354+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:13.542 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:13.539+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:13.540797+0000) 2022-04-03T22:09:13.875 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:13.873+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:14.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:14.387+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:14.542 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:14.539+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:14.540992+0000) 2022-04-03T22:09:14.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:14.876+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:15.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:15.420+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:15.542 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:15.540+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:15.541182+0000) 2022-04-03T22:09:15.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:15.860+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:16.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:16.459+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:16.548 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:16.540+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:16.541395+0000) 2022-04-03T22:09:16.837 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:16.835+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:17.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:17.459+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:17.535 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:09:17.532+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:09:17.542 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:17.540+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:17.541567+0000) 2022-04-03T22:09:17.819 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:17.816+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:18.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:18.489+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:18.543 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:18.540+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:18.541709+0000) 2022-04-03T22:09:18.821 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:18.819+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:19.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:19.472+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:19.543 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:19.541+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:19.541926+0000) 2022-04-03T22:09:19.851 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:19.849+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:20.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:20.521+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:20.543 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:20.541+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:20.542156+0000) 2022-04-03T22:09:20.874 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:20.872+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:21.543 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:21.541+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:21.542343+0000) 2022-04-03T22:09:21.562 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:21.560+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:21.837 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:21.834+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:22.543 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:22.541+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:22.542547+0000) 2022-04-03T22:09:22.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:22.551+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:22.876 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:22.874+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:23.544 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:23.541+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:23.542742+0000) 2022-04-03T22:09:23.569 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:23.567+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:23.888 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:23.886+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:24.544 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:24.542+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:24.542956+0000) 2022-04-03T22:09:24.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:24.565+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:24.847 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:09:24.851 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:09:24.852 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:24.848+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:25.544 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:25.542+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:25.543165+0000) 2022-04-03T22:09:25.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:25.565+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:25.810 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:25.802+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:26.544 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:26.542+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:26.543368+0000) 2022-04-03T22:09:26.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:26.555+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:26.768 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:26.766+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:27.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:27.509+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:27.544 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:27.542+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:27.543565+0000) 2022-04-03T22:09:27.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:27.809+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:28.517 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:28.515+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:28.545 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:28.543+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:28.543762+0000) 2022-04-03T22:09:28.783 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:28.781+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:29.516 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:29.513+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:29.545 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:29.543+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:29.543980+0000) 2022-04-03T22:09:29.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:29.754+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:30.499 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:30.497+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:30.545 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:30.543+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:30.544205+0000) 2022-04-03T22:09:30.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:30.765+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:31.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:31.508+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:31.545 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:31.543+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:31.544414+0000) 2022-04-03T22:09:31.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:31.745+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:32.471 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:32.469+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:32.546 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:32.544+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:32.544618+0000) 2022-04-03T22:09:32.550 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:09:32.548+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:09:32.776 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:32.773+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:33.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:33.427+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:33.546 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:33.544+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:33.544846+0000) 2022-04-03T22:09:33.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:33.804+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:34.386 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:34.383+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:34.546 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:34.544+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:34.545051+0000) 2022-04-03T22:09:34.849 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:34.846+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:35.412 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:35.410+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:35.546 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:35.544+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:35.545261+0000) 2022-04-03T22:09:35.867 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:35.864+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:36.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:36.404+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:36.547 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:36.544+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:36.545478+0000) 2022-04-03T22:09:36.892 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:36.890+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:37.442 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:37.439+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:37.547 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:37.545+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:37.545681+0000) 2022-04-03T22:09:37.858 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:37.855+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:38.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:38.462+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:38.547 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:38.545+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:38.545857+0000) 2022-04-03T22:09:38.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:38.860+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:39.462 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:39.460+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:39.547 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:39.545+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:39.546092+0000) 2022-04-03T22:09:39.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:39.861+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:40.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:40.419+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:40.547 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:40.545+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:40.546314+0000) 2022-04-03T22:09:40.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:40.896+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:41.449 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:41.448+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:41.547 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:41.546+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:41.546481+0000) 2022-04-03T22:09:41.886 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:41.884+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:42.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:42.492+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:42.548 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:42.546+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:42.546713+0000) 2022-04-03T22:09:42.896 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:42.895+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:43.457 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:43.456+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:43.548 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:43.546+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:43.546919+0000) 2022-04-03T22:09:43.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:43.941+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:44.505 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:44.503+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:44.548 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:44.546+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:44.547124+0000) 2022-04-03T22:09:44.972 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:44.971+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:45.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:45.509+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:45.548 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:45.547+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:45.547329+0000) 2022-04-03T22:09:46.015 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:46.014+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:46.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:46.539+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:46.548 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:46.547+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:46.547526+0000) 2022-04-03T22:09:46.969 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:46.967+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:47.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:47.533+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:47.548 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:47.547+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:47.547722+0000) 2022-04-03T22:09:47.566 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:09:47.564+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:09:47.958 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:47.956+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:48.549 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:48.547+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:48.547884+0000) 2022-04-03T22:09:48.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:48.578+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:48.957 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:48.955+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:49.549 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:49.547+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:49.548040+0000) 2022-04-03T22:09:49.600 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:49.599+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:49.954 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:49.952+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:50.549 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:50.548+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:50.548233+0000) 2022-04-03T22:09:50.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:50.565+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:50.913 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:50.911+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:51.549 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:51.547+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:51.548425+0000) 2022-04-03T22:09:51.601 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:51.600+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:51.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:51.944+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:52.550 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:52.548+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:52.548628+0000) 2022-04-03T22:09:52.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:52.646+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:52.916 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:52.914+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:53.550 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:53.548+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:53.548872+0000) 2022-04-03T22:09:53.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:53.655+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:53.867 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:53.866+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:54.550 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:54.548+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:54.549074+0000) 2022-04-03T22:09:54.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:54.668+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:54.819 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:54.817+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:55.003 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:09:55.007 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:09:55.550 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:55.548+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:55.549275+0000) 2022-04-03T22:09:55.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:55.639+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:55.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:55.803+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:56.551 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:56.548+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:56.549524+0000) 2022-04-03T22:09:56.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:56.612+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:56.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:56.782+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:57.551 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:57.548+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:57.549742+0000) 2022-04-03T22:09:57.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:57.655+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:57.781 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:57.779+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:58.551 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:58.548+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:58.549961+0000) 2022-04-03T22:09:58.652 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:58.650+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:58.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:58.791+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:09:59.551 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:09:59.549+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:09:59.550173+0000) 2022-04-03T22:09:59.696 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:09:59.694+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:09:59.762 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:09:59.760+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:00.551 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:00.549+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:00.550396+0000) 2022-04-03T22:10:00.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:00.665+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:00.763 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:00.760+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:01.552 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:01.549+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:01.550599+0000) 2022-04-03T22:10:01.716 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:01.713+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:01.807 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:01.804+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:02.552 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:02.549+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:02.550822+0000) 2022-04-03T22:10:02.583 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:10:02.579+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:10:02.716 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:02.714+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:02.814 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:02.811+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:03.552 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:03.550+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:03.551000+0000) 2022-04-03T22:10:03.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:03.741+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:03.847 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:03.845+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:04.552 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:04.550+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:04.551201+0000) 2022-04-03T22:10:04.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:04.741+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:04.866 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:04.864+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:05.552 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:05.550+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:05.551386+0000) 2022-04-03T22:10:05.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:05.740+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:05.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:05.829+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:06.553 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:06.550+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:06.551597+0000) 2022-04-03T22:10:06.747 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:06.744+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:06.798 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:06.796+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:07.553 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:07.551+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:07.551746+0000) 2022-04-03T22:10:07.774 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:07.771+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:07.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:07.773+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:08.553 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:08.551+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:08.551959+0000) 2022-04-03T22:10:08.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:08.780+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:08.792 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:08.790+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:09.553 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:09.551+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:09.552174+0000) 2022-04-03T22:10:09.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:09.778+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:09.833 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:09.798+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:10.553 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:10.551+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:10.552365+0000) 2022-04-03T22:10:10.765 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:10.763+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:10.797 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:10.795+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:11.554 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:11.551+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:11.552582+0000) 2022-04-03T22:10:11.769 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:11.767+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:11.798 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:11.796+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:12.554 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:12.552+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:12.552738+0000) 2022-04-03T22:10:12.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:12.756+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:12.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:12.815+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:13.554 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:13.552+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:13.552951+0000) 2022-04-03T22:10:13.720 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:13.717+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:13.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:13.810+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:14.554 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:14.552+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:14.553163+0000) 2022-04-03T22:10:14.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:14.735+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:14.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:14.825+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:15.554 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:15.552+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:15.553319+0000) 2022-04-03T22:10:15.778 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:15.775+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:15.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:15.862+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:16.555 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:16.552+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:16.553527+0000) 2022-04-03T22:10:16.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:16.753+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:16.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:16.841+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:17.555 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:17.553+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:17.553732+0000) 2022-04-03T22:10:17.599 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:10:17.596+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:10:17.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:17.741+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:17.841 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:17.839+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:18.555 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:18.553+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:18.553889+0000) 2022-04-03T22:10:18.744 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:18.742+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:18.874 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:18.872+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:19.555 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:19.553+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:19.554042+0000) 2022-04-03T22:10:19.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:19.725+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:19.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:19.837+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:20.555 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:20.553+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:20.554255+0000) 2022-04-03T22:10:20.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:20.757+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:20.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:20.884+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:21.556 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:21.553+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:21.554463+0000) 2022-04-03T22:10:21.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:21.803+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:21.901 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:21.898+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:22.556 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:22.554+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:22.554671+0000) 2022-04-03T22:10:22.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:22.791+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:22.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:22.891+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:23.556 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:23.554+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:23.554886+0000) 2022-04-03T22:10:23.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:23.775+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:23.936 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:23.934+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:24.556 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:24.554+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:24.555078+0000) 2022-04-03T22:10:24.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:24.744+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:24.905 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:24.903+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:25.143 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:10:25.147 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:10:25.556 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:25.554+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:25.555287+0000) 2022-04-03T22:10:25.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:25.714+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:25.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:25.894+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:26.556 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:26.555+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:26.555470+0000) 2022-04-03T22:10:26.746 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:26.744+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:26.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:26.880+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:27.557 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:27.555+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:27.555633+0000) 2022-04-03T22:10:27.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:27.771+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:27.869 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:27.867+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:28.557 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:28.555+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:28.555869+0000) 2022-04-03T22:10:28.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:28.734+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:28.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:28.870+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:29.557 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:29.555+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:29.556035+0000) 2022-04-03T22:10:29.754 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:29.751+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:29.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:29.916+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:30.557 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:30.555+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:30.556247+0000) 2022-04-03T22:10:30.717 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:30.715+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:30.920 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:30.918+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:31.557 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:31.555+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:31.556490+0000) 2022-04-03T22:10:31.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:31.668+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:31.934 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:31.932+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:32.558 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:32.556+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:32.556704+0000) 2022-04-03T22:10:32.615 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:10:32.612+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:10:32.656 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:32.653+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:32.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:32.894+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:33.558 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:33.556+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:33.556913+0000) 2022-04-03T22:10:33.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:33.644+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:33.945 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:33.943+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:34.558 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:34.556+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:34.557139+0000) 2022-04-03T22:10:34.604 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:34.602+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:34.907 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:34.905+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:35.558 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:35.557+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:35.557361+0000) 2022-04-03T22:10:35.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:35.625+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:35.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:35.869+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:36.559 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:36.557+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:36.557583+0000) 2022-04-03T22:10:36.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:36.667+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:36.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:36.838+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:37.559 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:37.557+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:37.557853+0000) 2022-04-03T22:10:37.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:37.636+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:37.797 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:37.796+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:38.559 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:38.557+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:38.558003+0000) 2022-04-03T22:10:38.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:38.679+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:38.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:38.795+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:39.559 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:39.557+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:39.558211+0000) 2022-04-03T22:10:39.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:39.713+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:39.768 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:39.767+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:40.559 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:40.556+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:40.558439+0000) 2022-04-03T22:10:40.744 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:40.742+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:40.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:40.795+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:41.560 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:41.557+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:41.558641+0000) 2022-04-03T22:10:41.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:41.789+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:41.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:41.792+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:42.560 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:42.557+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:42.558884+0000) 2022-04-03T22:10:42.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:42.798+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:42.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:42.822+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:43.560 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:43.558+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:43.559049+0000) 2022-04-03T22:10:43.756 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:43.755+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:43.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:43.867+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:44.560 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:44.558+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:44.559220+0000) 2022-04-03T22:10:44.756 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:44.754+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:44.821 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:44.820+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:45.560 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:45.558+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:45.559427+0000) 2022-04-03T22:10:45.804 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:45.803+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:45.840 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:45.839+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:46.561 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:46.558+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:46.559645+0000) 2022-04-03T22:10:46.756 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:46.754+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:46.839 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:46.837+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:47.561 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:47.558+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:47.559861+0000) 2022-04-03T22:10:47.630 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:10:47.627+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:10:47.769 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:47.767+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:47.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:47.823+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:48.561 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:48.559+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:48.560056+0000) 2022-04-03T22:10:48.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:48.727+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:48.808 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:48.806+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:49.561 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:49.559+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:49.560263+0000) 2022-04-03T22:10:49.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:49.761+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:49.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:49.840+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:50.561 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:50.559+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:50.560477+0000) 2022-04-03T22:10:50.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:50.776+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:50.801 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:50.799+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:51.562 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:51.559+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:51.560670+0000) 2022-04-03T22:10:51.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:51.727+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:51.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:51.820+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:52.562 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:52.560+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:52.560858+0000) 2022-04-03T22:10:52.685 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:52.682+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:52.849 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:52.846+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:53.562 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:53.560+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:53.561033+0000) 2022-04-03T22:10:53.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:53.723+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:53.817 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:53.814+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:54.562 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:54.560+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:54.561239+0000) 2022-04-03T22:10:54.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:54.724+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:54.775 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:54.772+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:55.288 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:10:55.292 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:10:55.562 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:55.560+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:55.561468+0000) 2022-04-03T22:10:55.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:55.676+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:55.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:55.800+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:56.563 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:56.561+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:56.561669+0000) 2022-04-03T22:10:56.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:56.638+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:56.825 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:56.823+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:57.563 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:57.561+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:57.561876+0000) 2022-04-03T22:10:57.634 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:57.631+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:57.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:57.867+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:58.563 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:58.561+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:58.562097+0000) 2022-04-03T22:10:58.591 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:58.589+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:58.888 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:58.886+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:10:59.563 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:10:59.561+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:10:59.562253+0000) 2022-04-03T22:10:59.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:10:59.600+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:10:59.856 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:10:59.854+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:00.564 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:00.561+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:00.562474+0000) 2022-04-03T22:11:00.652 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:00.649+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:00.848 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:00.845+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:01.564 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:01.562+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:01.562676+0000) 2022-04-03T22:11:01.618 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:01.616+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:01.836 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:01.833+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:02.564 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:02.562+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:02.562888+0000) 2022-04-03T22:11:02.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:02.580+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:02.646 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:11:02.644+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:11:02.865 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:02.862+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:03.564 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:03.562+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:03.563034+0000) 2022-04-03T22:11:03.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:03.623+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:03.887 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:03.884+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:04.564 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:04.562+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:04.563237+0000) 2022-04-03T22:11:04.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:04.669+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:04.865 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:04.862+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:05.564 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:05.562+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:05.563420+0000) 2022-04-03T22:11:05.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:05.623+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:05.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:05.840+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:06.564 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:06.563+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:06.563609+0000) 2022-04-03T22:11:06.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:06.578+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:06.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:06.790+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:07.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:07.530+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:07.565 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:07.563+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:07.563837+0000) 2022-04-03T22:11:07.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:07.774+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:08.565 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:08.563+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:08.564077+0000) 2022-04-03T22:11:08.568 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:08.566+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:08.776 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:08.773+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:09.565 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:09.563+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:09.564255+0000) 2022-04-03T22:11:09.573 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:09.571+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:09.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:09.804+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:10.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:10.531+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:10.565 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:10.564+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:10.564469+0000) 2022-04-03T22:11:10.790 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:10.788+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:11.556 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:11.554+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:11.565 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:11.564+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:11.564665+0000) 2022-04-03T22:11:11.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:11.778+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:12.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:12.528+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:12.566 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:12.564+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:12.564908+0000) 2022-04-03T22:11:12.789 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:12.786+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:13.566 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:13.564+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:13.565106+0000) 2022-04-03T22:11:13.569 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:13.567+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:13.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:13.813+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:14.566 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:14.565+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:14.565267+0000) 2022-04-03T22:11:14.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:14.590+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:14.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:14.799+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:15.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:15.548+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:15.566 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:15.565+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:15.565469+0000) 2022-04-03T22:11:15.754 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:15.752+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:16.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:16.545+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:16.567 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:16.564+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:16.565707+0000) 2022-04-03T22:11:16.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:16.777+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:17.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:17.533+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:17.567 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:17.565+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:17.565912+0000) 2022-04-03T22:11:17.662 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:11:17.660+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:11:17.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:17.797+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:18.498 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:18.496+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:18.567 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:18.565+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:18.566125+0000) 2022-04-03T22:11:18.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:18.797+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:19.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:19.451+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:19.567 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:19.566+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:19.566336+0000) 2022-04-03T22:11:19.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:19.813+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:20.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:20.411+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:20.567 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:20.566+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:20.566564+0000) 2022-04-03T22:11:20.856 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:20.854+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:21.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:21.409+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:21.568 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:21.566+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:21.566825+0000) 2022-04-03T22:11:21.816 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:21.814+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:22.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:22.403+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:22.568 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:22.566+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:22.566961+0000) 2022-04-03T22:11:22.840 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:22.837+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:23.434 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:23.432+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:23.568 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:23.567+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:23.567162+0000) 2022-04-03T22:11:23.836 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:23.834+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:24.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:24.438+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:24.568 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:24.566+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:24.567376+0000) 2022-04-03T22:11:24.831 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:24.829+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:25.437 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:11:25.440 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:11:25.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:25.474+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:25.569 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:25.566+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:25.567577+0000) 2022-04-03T22:11:25.852 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:25.850+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:26.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:26.452+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:26.569 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:26.566+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:26.567839+0000) 2022-04-03T22:11:26.867 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:26.865+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:27.478 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:27.475+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:27.569 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:27.567+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:27.568032+0000) 2022-04-03T22:11:27.886 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:27.884+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:28.505 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:28.503+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:28.570 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:28.566+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:28.568285+0000) 2022-04-03T22:11:28.923 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:28.921+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:29.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:29.532+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:29.569 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:29.567+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:29.568462+0000) 2022-04-03T22:11:29.942 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:29.940+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:30.516 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:30.514+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:30.570 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:30.566+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:30.568693+0000) 2022-04-03T22:11:30.971 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:30.969+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:31.478 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:31.477+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:31.570 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:31.567+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:31.568897+0000) 2022-04-03T22:11:31.937 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:31.936+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:32.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:32.517+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:32.570 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:32.568+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:32.569129+0000) 2022-04-03T22:11:32.678 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:11:32.676+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:11:32.965 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:32.963+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:33.561 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:33.559+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:33.570 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:33.568+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:33.569341+0000) 2022-04-03T22:11:33.950 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:33.948+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:34.571 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:34.568+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:34.569546+0000) 2022-04-03T22:11:34.600 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:34.599+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:34.983 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:34.982+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:35.571 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:35.569+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:35.569749+0000) 2022-04-03T22:11:35.645 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:35.644+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:36.008 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:36.006+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:36.571 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:36.569+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:36.570000+0000) 2022-04-03T22:11:36.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:36.686+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:37.024 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:37.022+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:37.571 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:37.569+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:37.570206+0000) 2022-04-03T22:11:37.718 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:37.717+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:38.058 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:38.056+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:38.571 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:38.569+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:38.570410+0000) 2022-04-03T22:11:38.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:38.686+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:39.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:39.083+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:39.572 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:39.569+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:39.570619+0000) 2022-04-03T22:11:39.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:39.692+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:40.125 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:40.124+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:40.572 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:40.570+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:40.570865+0000) 2022-04-03T22:11:40.645 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:40.644+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:41.108 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:41.107+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:41.572 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:41.570+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:41.571038+0000) 2022-04-03T22:11:41.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:41.626+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:42.142 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:42.140+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:42.572 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:42.570+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:42.571290+0000) 2022-04-03T22:11:42.653 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:42.651+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:43.094 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:43.092+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:43.572 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:43.570+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:43.571478+0000) 2022-04-03T22:11:43.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:43.626+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:44.069 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:44.068+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:44.573 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:44.571+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:44.571691+0000) 2022-04-03T22:11:44.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:44.582+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:45.110 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:45.107+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:45.573 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:45.571+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:45.571855+0000) 2022-04-03T22:11:45.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:45.600+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:46.091 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:46.088+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:46.573 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:46.571+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:46.572043+0000) 2022-04-03T22:11:46.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:46.576+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:47.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:47.078+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:47.565 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:47.562+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:47.573 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:47.571+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:47.572201+0000) 2022-04-03T22:11:47.694 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:11:47.691+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:11:48.091 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:48.089+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:48.573 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:48.571+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:48.572410+0000) 2022-04-03T22:11:48.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:48.602+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:49.108 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:49.105+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:49.574 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:49.572+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:49.572617+0000) 2022-04-03T22:11:49.596 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:49.593+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:50.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:50.083+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:50.574 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:50.572+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:50.572852+0000) 2022-04-03T22:11:50.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:50.578+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:51.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:51.079+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:51.574 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:51.572+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:51.573064+0000) 2022-04-03T22:11:51.600 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:51.598+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:52.132 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:52.129+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:52.574 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:52.572+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:52.573274+0000) 2022-04-03T22:11:52.589 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:52.587+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:53.182 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:53.179+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:53.574 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:53.573+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:53.573468+0000) 2022-04-03T22:11:53.585 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:53.583+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:54.206 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:54.203+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:54.575 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:54.573+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:54.573683+0000) 2022-04-03T22:11:54.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:54.614+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:55.191 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:55.189+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:55.568 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:55.565+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:55.574 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:55.573+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:55.573825+0000) 2022-04-03T22:11:55.585 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:11:55.589 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:11:56.205 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:56.203+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:56.575 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:56.573+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:56.574024+0000) 2022-04-03T22:11:56.598 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:56.596+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:57.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:57.205+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:57.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:57.549+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:57.575 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:57.573+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:57.574234+0000) 2022-04-03T22:11:58.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:58.248+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:58.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:58.537+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:58.575 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:58.574+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:58.574451+0000) 2022-04-03T22:11:59.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:11:59.284+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:11:59.530 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:11:59.527+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:11:59.575 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:11:59.574+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:11:59.574624+0000) 2022-04-03T22:12:00.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:00.304+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:00.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:00.484+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:00.576 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:00.574+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:00.574840+0000) 2022-04-03T22:12:01.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:01.293+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:01.470 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:01.467+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:01.576 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:01.574+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:01.575047+0000) 2022-04-03T22:12:02.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:02.250+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:02.448 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:02.445+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:02.576 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:02.575+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:02.575248+0000) 2022-04-03T22:12:02.709 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:12:02.707+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:12:03.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:03.295+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:03.490 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:03.488+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:03.576 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:03.575+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:03.575408+0000) 2022-04-03T22:12:04.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:04.302+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:04.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:04.519+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:04.576 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:04.575+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:04.575547+0000) 2022-04-03T22:12:05.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:05.296+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:05.508 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:05.506+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:05.576 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:05.575+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:05.575685+0000) 2022-04-03T22:12:06.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:06.256+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:06.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:06.533+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:06.577 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:06.575+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:06.575882+0000) 2022-04-03T22:12:07.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:07.297+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:07.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:07.549+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:07.633 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:07.576+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:07.576129+0000) 2022-04-03T22:12:08.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:08.331+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:08.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:08.526+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:08.577 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:08.575+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:08.576342+0000) 2022-04-03T22:12:09.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:09.337+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:09.540 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:09.538+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:09.578 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:09.575+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:09.576539+0000) 2022-04-03T22:12:10.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:10.368+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:10.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:10.493+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:10.577 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:10.575+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:10.576844+0000) 2022-04-03T22:12:11.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:11.344+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:11.540 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:11.537+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:11.578 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:11.576+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:11.576990+0000) 2022-04-03T22:12:12.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:12.357+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:12.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:12.508+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:12.578 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:12.576+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:12.577164+0000) 2022-04-03T22:12:13.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:13.334+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:13.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:13.459+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:13.578 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:13.576+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:13.577359+0000) 2022-04-03T22:12:14.332 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:14.330+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:14.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:14.414+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:14.579 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:14.576+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:14.577549+0000) 2022-04-03T22:12:15.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:15.359+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:15.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:15.416+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:15.578 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:15.576+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:15.577709+0000) 2022-04-03T22:12:16.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:16.373+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:16.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:16.391+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:16.579 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:16.576+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:16.577892+0000) 2022-04-03T22:12:17.365 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:17.363+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:17.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:17.371+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:17.579 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:17.577+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:17.578086+0000) 2022-04-03T22:12:17.716 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:12:17.714+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:12:18.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:18.363+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:18.374 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:18.373+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:18.579 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:18.577+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:18.578330+0000) 2022-04-03T22:12:19.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:19.366+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:19.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:19.408+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:19.579 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:19.577+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:19.578574+0000) 2022-04-03T22:12:20.354 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:20.352+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:20.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:20.445+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:20.580 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:20.578+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:20.578845+0000) 2022-04-03T22:12:21.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:21.392+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:21.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:21.494+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:21.580 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:21.578+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:21.579064+0000) 2022-04-03T22:12:22.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:22.413+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:22.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:22.536+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:22.580 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:22.578+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:22.579262+0000) 2022-04-03T22:12:23.387 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:23.385+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:23.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:23.553+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:23.580 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:23.578+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:23.579444+0000) 2022-04-03T22:12:24.355 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:24.353+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:24.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:24.529+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:24.580 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:24.578+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:24.579631+0000) 2022-04-03T22:12:25.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:25.387+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:25.501 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:25.500+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:25.581 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:25.579+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:25.579835+0000) 2022-04-03T22:12:25.699 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:12:25.703 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:12:26.420 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:26.418+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:26.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:26.546+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:26.581 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:26.579+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:26.580047+0000) 2022-04-03T22:12:27.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:27.428+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:27.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:27.574+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:27.581 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:27.579+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:27.580248+0000) 2022-04-03T22:12:28.403 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:28.402+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:28.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:28.526+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:28.581 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:28.579+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:28.580464+0000) 2022-04-03T22:12:29.426 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:29.425+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:29.552 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:29.551+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:29.581 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:29.580+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:29.580696+0000) 2022-04-03T22:12:30.441 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:30.439+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:30.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:30.517+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:30.582 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:30.580+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:30.580906+0000) 2022-04-03T22:12:31.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:31.487+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:31.522 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:31.520+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:31.582 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:31.580+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:31.581082+0000) 2022-04-03T22:12:32.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:32.528+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:32.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:32.529+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:32.582 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:32.580+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:32.581293+0000) 2022-04-03T22:12:32.724 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:12:32.722+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:12:33.526 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:33.524+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:33.558 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:33.557+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:33.582 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:33.580+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:33.581485+0000) 2022-04-03T22:12:34.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:34.551+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:34.582 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:34.580+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:34.581652+0000) 2022-04-03T22:12:34.608 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:34.607+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:35.540 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:35.539+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:35.583 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:35.581+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:35.581858+0000) 2022-04-03T22:12:35.649 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:35.648+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:36.564 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:36.562+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:36.583 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:36.581+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:36.582056+0000) 2022-04-03T22:12:36.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:36.655+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:37.566 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:37.564+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:37.583 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:37.581+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:37.582222+0000) 2022-04-03T22:12:37.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:37.696+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:38.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:38.551+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:38.583 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:38.582+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:38.582422+0000) 2022-04-03T22:12:38.654 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:38.653+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:39.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:39.533+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:39.583 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:39.582+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:39.582587+0000) 2022-04-03T22:12:39.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:39.661+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:40.502 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:40.499+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:40.584 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:40.582+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:40.582815+0000) 2022-04-03T22:12:40.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:40.667+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:41.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:41.526+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:41.584 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:41.582+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:41.582963+0000) 2022-04-03T22:12:41.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:41.663+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:42.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:42.527+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:42.584 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:42.582+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:42.583195+0000) 2022-04-03T22:12:42.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:42.617+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:43.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:43.512+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:43.584 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:43.583+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:43.583395+0000) 2022-04-03T22:12:43.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:43.654+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:44.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:44.493+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:44.585 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:44.583+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:44.583610+0000) 2022-04-03T22:12:44.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:44.665+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:45.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:45.515+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:45.585 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:45.583+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:45.583838+0000) 2022-04-03T22:12:45.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:45.621+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:46.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:46.486+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:46.585 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:46.583+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:46.584034+0000) 2022-04-03T22:12:46.630 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:46.627+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:47.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:47.484+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:47.585 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:47.584+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:47.584233+0000) 2022-04-03T22:12:47.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:47.618+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:47.740 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:12:47.738+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:12:48.474 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:48.471+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:48.585 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:48.584+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:48.584452+0000) 2022-04-03T22:12:48.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:48.645+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:49.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:49.492+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:49.585 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:49.584+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:49.584640+0000) 2022-04-03T22:12:49.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:49.639+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:50.506 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:50.503+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:50.586 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:50.584+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:50.584854+0000) 2022-04-03T22:12:50.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:50.655+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:51.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:51.522+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:51.586 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:51.584+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:51.585056+0000) 2022-04-03T22:12:51.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:51.632+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:52.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:52.529+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:52.586 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:52.585+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:52.585276+0000) 2022-04-03T22:12:52.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:52.618+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:53.540 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:53.538+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:53.586 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:53.584+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:53.585476+0000) 2022-04-03T22:12:53.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:53.621+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:54.585 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:54.583+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:54.586 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:54.584+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:54.585696+0000) 2022-04-03T22:12:54.619 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:54.616+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:55.587 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:55.584+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:55.585864+0000) 2022-04-03T22:12:55.602 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:55.599+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:55.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:55.619+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:56.018 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:12:56.022 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:12:56.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:56.555+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:56.587 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:56.585+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:56.586101+0000) 2022-04-03T22:12:56.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:56.618+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:57.587 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:57.585+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:57.586308+0000) 2022-04-03T22:12:57.604 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:57.601+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:57.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:57.614+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:58.587 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:58.585+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:58.586513+0000) 2022-04-03T22:12:58.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:58.601+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:58.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:58.629+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:12:59.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:12:59.581+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:12:59.587 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:12:59.584+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:12:59.586731+0000) 2022-04-03T22:12:59.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:12:59.648+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:00.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:00.548+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:00.588 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:00.586+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:00.586916+0000) 2022-04-03T22:13:00.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:00.667+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:01.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:01.516+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:01.588 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:01.586+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:01.587139+0000) 2022-04-03T22:13:01.656 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:01.654+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:02.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:02.544+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:02.588 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:02.586+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:02.587358+0000) 2022-04-03T22:13:02.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:02.666+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:02.756 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:13:02.754+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:13:03.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:03.534+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:03.588 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:03.586+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:03.587538+0000) 2022-04-03T22:13:03.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:03.681+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:04.554 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:04.551+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:04.589 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:04.586+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:04.587747+0000) 2022-04-03T22:13:04.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:04.720+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:05.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:05.509+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:05.589 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:05.587+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:05.587987+0000) 2022-04-03T22:13:05.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:05.700+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:06.478 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:06.476+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:06.589 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:06.587+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:06.588119+0000) 2022-04-03T22:13:06.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:06.713+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:07.469 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:07.467+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:07.589 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:07.587+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:07.588326+0000) 2022-04-03T22:13:07.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:07.681+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:08.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:08.456+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:08.590 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:08.587+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:08.588533+0000) 2022-04-03T22:13:08.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:08.669+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:09.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:09.491+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:09.590 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:09.588+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:09.588738+0000) 2022-04-03T22:13:09.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:09.711+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:10.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:10.442+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:10.590 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:10.588+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:10.588931+0000) 2022-04-03T22:13:10.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:10.665+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:11.438 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:11.436+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:11.590 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:11.588+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:11.589135+0000) 2022-04-03T22:13:11.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:11.652+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:12.403 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:12.400+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:12.590 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:12.588+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:12.589352+0000) 2022-04-03T22:13:12.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:12.641+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:13.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:13.367+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:13.590 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:13.588+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:13.589506+0000) 2022-04-03T22:13:13.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:13.629+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:14.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:14.389+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:14.591 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:14.589+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:14.589712+0000) 2022-04-03T22:13:14.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:14.613+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:15.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:15.413+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:15.591 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:15.589+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:15.589932+0000) 2022-04-03T22:13:15.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:15.650+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:16.453 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:16.452+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:16.591 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:16.589+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:16.590127+0000) 2022-04-03T22:13:16.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:16.684+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:17.475 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:17.473+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:17.591 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:17.589+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:17.590342+0000) 2022-04-03T22:13:17.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:17.656+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:17.772 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:13:17.770+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:13:18.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:18.443+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:18.591 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:18.590+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:18.590551+0000) 2022-04-03T22:13:18.662 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:18.660+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:19.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:19.441+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:19.592 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:19.590+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:19.590745+0000) 2022-04-03T22:13:19.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:19.642+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:20.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:20.413+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:20.592 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:20.590+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:20.590918+0000) 2022-04-03T22:13:20.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:20.640+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:21.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:21.429+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:21.592 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:21.590+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:21.591070+0000) 2022-04-03T22:13:21.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:21.687+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:22.442 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:22.441+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:22.592 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:22.590+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:22.591247+0000) 2022-04-03T22:13:22.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:22.716+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:23.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:23.441+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:23.592 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:23.591+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:23.591391+0000) 2022-04-03T22:13:23.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:23.676+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:24.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:24.461+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:24.592 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:24.591+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:24.591584+0000) 2022-04-03T22:13:24.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:24.673+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:25.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:25.421+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:25.593 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:25.591+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:25.591792+0000) 2022-04-03T22:13:25.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:25.715+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:26.162 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:13:26.166 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:13:26.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:26.403+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:26.593 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:26.591+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:26.592033+0000) 2022-04-03T22:13:26.720 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:26.719+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:27.365 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:27.363+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:27.593 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:27.591+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:27.592227+0000) 2022-04-03T22:13:27.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:27.697+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:28.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:28.340+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:28.593 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:28.592+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:28.592434+0000) 2022-04-03T22:13:28.744 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:28.742+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:29.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:29.361+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:29.593 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:29.592+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:29.592647+0000) 2022-04-03T22:13:29.752 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:29.751+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:30.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:30.381+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:30.594 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:30.592+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:30.592865+0000) 2022-04-03T22:13:30.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:30.755+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:31.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:31.348+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:31.594 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:31.592+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:31.593090+0000) 2022-04-03T22:13:31.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:31.741+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:32.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:32.382+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:32.594 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:32.593+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:32.593311+0000) 2022-04-03T22:13:32.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:32.736+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:32.788 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:13:32.785+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:13:33.372 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:33.371+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:33.594 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:33.593+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:33.593488+0000) 2022-04-03T22:13:33.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:33.706+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:34.344 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:34.341+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:34.595 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:34.593+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:34.593650+0000) 2022-04-03T22:13:34.694 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:34.691+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:35.331 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:35.329+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:35.595 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:35.593+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:35.593860+0000) 2022-04-03T22:13:35.689 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:35.687+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:36.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:36.311+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:36.596 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:36.593+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:36.594050+0000) 2022-04-03T22:13:36.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:36.667+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:37.333 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:37.330+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:37.596 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:37.593+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:37.594259+0000) 2022-04-03T22:13:37.712 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:37.709+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:38.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:38.336+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:38.596 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:38.593+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:38.594434+0000) 2022-04-03T22:13:38.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:38.724+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:39.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:39.293+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:39.596 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:39.593+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:39.594628+0000) 2022-04-03T22:13:39.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:39.730+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:40.327 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:40.325+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:40.596 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:40.593+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:40.594822+0000) 2022-04-03T22:13:40.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:40.699+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:41.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:41.338+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:41.596 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:41.594+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:41.595022+0000) 2022-04-03T22:13:41.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:41.684+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:42.354 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:42.351+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:42.596 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:42.594+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:42.595278+0000) 2022-04-03T22:13:42.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:42.643+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:43.403 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:43.401+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:43.596 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:43.594+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:43.595476+0000) 2022-04-03T22:13:43.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:43.616+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:44.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:44.378+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:44.597 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:44.594+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:44.595694+0000) 2022-04-03T22:13:44.603 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:44.600+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:45.337 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:45.334+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:45.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:45.553+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:45.597 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:45.594+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:45.595898+0000) 2022-04-03T22:13:46.350 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:46.348+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:46.552 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:46.550+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:46.597 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:46.595+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:46.596087+0000) 2022-04-03T22:13:47.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:47.307+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:47.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:47.501+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:47.597 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:47.595+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:47.596285+0000) 2022-04-03T22:13:47.803 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:13:47.801+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:13:48.331 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:48.329+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:48.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:48.532+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:48.597 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:48.595+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:48.596447+0000) 2022-04-03T22:13:49.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:49.313+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:49.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:49.522+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:49.598 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:49.595+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:49.596642+0000) 2022-04-03T22:13:50.347 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:50.344+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:50.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:50.532+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:50.598 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:50.596+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:50.596862+0000) 2022-04-03T22:13:51.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:51.370+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:51.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:51.527+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:51.598 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:51.596+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:51.597009+0000) 2022-04-03T22:13:52.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:52.380+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:52.570 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:52.568+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:52.598 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:52.596+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:52.597224+0000) 2022-04-03T22:13:53.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:53.338+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:53.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:53.532+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:53.598 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:53.596+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:53.597421+0000) 2022-04-03T22:13:54.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:54.320+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:54.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:54.539+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:54.598 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:54.596+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:54.597619+0000) 2022-04-03T22:13:55.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:55.290+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:55.583 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:55.581+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:55.599 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:55.597+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:55.597862+0000) 2022-04-03T22:13:56.307 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:56.305+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:56.544 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:13:56.548 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:13:56.586 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:56.584+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:56.599 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:56.597+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:56.598033+0000) 2022-04-03T22:13:57.306 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:57.304+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:57.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:57.593+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:57.599 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:57.597+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:57.598218+0000) 2022-04-03T22:13:58.262 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:58.260+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:58.599 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:58.597+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:58.598366+0000) 2022-04-03T22:13:58.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:58.617+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:13:59.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:13:59.283+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:13:59.599 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:13:59.597+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:13:59.598563+0000) 2022-04-03T22:13:59.625 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:13:59.624+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:00.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:00.312+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:00.600 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:00.598+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:00.598789+0000) 2022-04-03T22:14:00.623 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:00.620+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:01.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:01.308+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:01.600 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:01.598+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:01.598997+0000) 2022-04-03T22:14:01.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:01.636+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:02.355 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:02.353+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:02.600 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:02.598+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:02.599202+0000) 2022-04-03T22:14:02.655 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:02.653+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:02.819 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:14:02.817+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:14:03.306 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:03.304+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:03.600 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:03.598+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:03.599385+0000) 2022-04-03T22:14:03.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:03.628+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:04.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:04.318+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:04.601 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:04.599+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:04.599585+0000) 2022-04-03T22:14:04.621 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:04.619+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:05.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:05.361+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:05.601 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:05.599+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:05.599805+0000) 2022-04-03T22:14:05.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:05.633+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:06.327 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:06.325+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:06.601 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:06.599+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:06.599997+0000) 2022-04-03T22:14:06.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:06.667+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:07.347 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:07.345+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:07.601 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:07.599+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:07.600215+0000) 2022-04-03T22:14:07.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:07.634+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:08.308 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:08.306+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:08.601 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:08.600+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:08.600398+0000) 2022-04-03T22:14:08.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:08.651+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:09.327 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:09.326+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:09.601 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:09.600+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:09.600552+0000) 2022-04-03T22:14:09.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:09.615+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:10.359 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:10.357+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:10.602 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:10.600+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:10.600763+0000) 2022-04-03T22:14:10.666 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:10.664+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:11.364 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:11.362+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:11.602 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:11.600+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:11.600951+0000) 2022-04-03T22:14:11.629 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:11.627+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:12.337 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:12.335+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:12.602 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:12.600+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:12.601154+0000) 2022-04-03T22:14:12.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:12.677+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:13.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:13.294+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:13.602 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:13.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:13.601354+0000) 2022-04-03T22:14:13.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:13.718+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:14.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:14.312+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:14.602 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:14.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:14.601538+0000) 2022-04-03T22:14:14.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:14.695+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:15.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:15.321+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:15.603 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:15.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:15.601707+0000) 2022-04-03T22:14:15.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:15.683+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:16.332 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:16.331+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:16.603 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:16.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:16.601911+0000) 2022-04-03T22:14:16.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:16.651+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:17.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:17.365+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:17.603 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:17.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:17.602130+0000) 2022-04-03T22:14:17.610 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:17.609+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:17.835 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:14:17.833+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:14:18.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:18.348+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:18.603 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:18.602+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:18.602294+0000) 2022-04-03T22:14:18.648 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:18.647+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:19.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:19.351+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:19.604 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:19.602+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:19.602502+0000) 2022-04-03T22:14:19.623 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:19.622+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:20.400 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:20.398+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:20.604 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:20.602+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:20.602728+0000) 2022-04-03T22:14:20.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:20.663+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:21.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:21.375+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:21.604 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:21.602+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:21.602894+0000) 2022-04-03T22:14:21.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:21.643+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:22.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:22.412+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:22.604 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:22.602+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:22.603092+0000) 2022-04-03T22:14:22.655 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:22.653+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:23.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:23.429+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:23.604 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:23.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:23.603212+0000) 2022-04-03T22:14:23.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:23.610+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:24.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:24.412+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:24.604 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:24.602+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:24.603368+0000) 2022-04-03T22:14:24.609 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:24.607+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:25.379 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:25.378+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:25.604 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:25.602+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:25.603570+0000) 2022-04-03T22:14:25.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:25.613+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:26.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:26.352+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:26.605 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:26.602+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:26.603806+0000) 2022-04-03T22:14:26.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:26.643+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:26.691 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:14:26.695 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:14:27.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:27.373+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:27.605 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:27.603+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:27.604004+0000) 2022-04-03T22:14:27.661 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:27.659+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:28.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:28.402+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:28.605 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:28.603+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:28.604230+0000) 2022-04-03T22:14:28.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:28.654+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:29.438 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:29.435+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:29.605 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:29.603+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:29.604435+0000) 2022-04-03T22:14:29.691 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:29.689+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:30.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:30.424+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:30.605 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:30.603+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:30.604638+0000) 2022-04-03T22:14:30.661 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:30.658+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:31.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:31.404+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:31.606 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:31.603+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:31.604848+0000) 2022-04-03T22:14:31.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:31.683+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:32.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:32.424+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:32.606 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:32.604+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:32.605068+0000) 2022-04-03T22:14:32.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:32.635+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:32.851 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:14:32.849+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:14:33.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:33.432+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:33.606 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:33.604+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:33.605310+0000) 2022-04-03T22:14:33.610 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:33.607+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:34.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:34.383+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:34.603 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:34.601+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:34.606 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:34.604+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:34.605515+0000) 2022-04-03T22:14:35.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:35.402+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:35.570 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:35.568+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:35.607 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:35.604+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:35.605703+0000) 2022-04-03T22:14:36.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:36.373+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:36.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:36.577+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:36.607 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:36.605+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:36.605892+0000) 2022-04-03T22:14:37.379 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:37.377+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:37.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:37.544+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:37.607 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:37.605+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:37.606081+0000) 2022-04-03T22:14:38.410 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:38.408+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:38.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:38.576+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:38.607 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:38.605+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:38.606288+0000) 2022-04-03T22:14:39.432 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:39.430+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:39.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:39.548+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:39.607 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:39.605+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:39.606444+0000) 2022-04-03T22:14:40.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:40.465+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:40.586 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:40.583+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:40.607 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:40.605+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:40.606638+0000) 2022-04-03T22:14:41.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:41.427+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:41.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:41.562+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:41.608 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:41.606+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:41.606846+0000) 2022-04-03T22:14:42.472 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:42.470+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:42.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:42.543+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:42.608 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:42.606+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:42.607022+0000) 2022-04-03T22:14:43.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:43.462+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:43.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:43.590+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:43.608 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:43.606+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:43.607225+0000) 2022-04-03T22:14:44.441 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:44.438+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:44.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:44.559+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:44.608 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:44.606+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:44.607437+0000) 2022-04-03T22:14:45.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:45.461+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:45.540 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:45.538+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:45.608 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:45.607+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:45.607611+0000) 2022-04-03T22:14:46.470 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:46.468+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:46.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:46.585+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:46.609 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:46.607+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:46.607795+0000) 2022-04-03T22:14:47.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:47.511+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:47.609 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:47.607+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:47.607969+0000) 2022-04-03T22:14:47.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:47.630+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:47.866 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:14:47.864+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:14:48.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:48.516+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:48.609 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:48.607+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:48.608148+0000) 2022-04-03T22:14:48.610 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:48.608+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:49.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:49.539+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:49.609 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:49.607+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:49.608333+0000) 2022-04-03T22:14:49.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:49.610+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:50.564 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:50.562+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:50.609 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:50.608+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:50.608531+0000) 2022-04-03T22:14:50.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:50.624+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:51.610 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:51.608+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:51.608717+0000) 2022-04-03T22:14:51.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:51.609+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:51.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:51.657+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:52.610 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:52.608+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:52.608885+0000) 2022-04-03T22:14:52.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:52.640+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:52.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:52.697+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:53.610 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:53.608+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:53.609040+0000) 2022-04-03T22:14:53.685 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:53.683+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:53.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:53.731+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:54.610 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:54.608+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:54.609216+0000) 2022-04-03T22:14:54.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:54.668+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:54.772 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:54.770+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:55.610 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:55.609+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:55.609358+0000) 2022-04-03T22:14:55.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:55.700+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:55.774 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:55.771+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:56.610 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:56.609+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:56.609534+0000) 2022-04-03T22:14:56.722 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:56.720+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:56.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:56.754+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:56.790 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:14:56.794 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:14:57.611 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:57.609+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:57.609754+0000) 2022-04-03T22:14:57.738 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:57.736+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:57.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:57.783+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:58.611 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:58.609+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:58.609950+0000) 2022-04-03T22:14:58.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:58.693+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:58.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:58.830+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:14:59.611 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:14:59.609+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:14:59.610107+0000) 2022-04-03T22:14:59.736 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:14:59.734+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:14:59.861 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:14:59.859+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:00.611 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:00.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:00.610291+0000) 2022-04-03T22:15:00.752 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:00.749+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:00.879 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:00.877+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:01.611 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:01.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:01.610441+0000) 2022-04-03T22:15:01.749 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:01.747+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:01.914 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:01.912+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:02.612 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:02.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:02.610619+0000) 2022-04-03T22:15:02.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:02.794+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:02.882 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:15:02.880+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:15:02.904 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:02.901+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:03.612 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:03.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:03.610743+0000) 2022-04-03T22:15:03.754 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:03.751+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:03.881 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:03.878+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:04.612 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:04.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:04.610932+0000) 2022-04-03T22:15:04.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:04.727+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:04.881 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:04.879+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:05.612 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:05.611+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:05.611151+0000) 2022-04-03T22:15:05.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:05.698+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:05.894 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:05.892+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:06.612 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:06.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:06.611317+0000) 2022-04-03T22:15:06.711 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:06.708+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:06.900 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:06.898+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:07.612 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:07.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:07.611464+0000) 2022-04-03T22:15:07.704 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:07.703+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:07.932 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:07.930+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:08.613 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:08.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:08.611641+0000) 2022-04-03T22:15:08.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:08.689+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:08.919 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:08.918+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:09.613 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:09.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:09.611852+0000) 2022-04-03T22:15:09.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:09.698+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:09.903 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:09.901+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:10.613 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:10.611+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:10.612056+0000) 2022-04-03T22:15:10.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:10.660+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:10.866 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:10.864+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:11.613 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:11.611+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:11.612213+0000) 2022-04-03T22:15:11.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:11.669+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:11.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:11.895+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:12.613 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:12.611+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:12.612382+0000) 2022-04-03T22:15:12.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:12.634+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:12.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:12.875+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:13.613 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:13.611+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:13.612510+0000) 2022-04-03T22:15:13.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:13.674+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:13.856 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:13.855+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:14.614 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:14.611+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:14.612712+0000) 2022-04-03T22:15:14.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:14.661+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:14.879 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:14.878+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:15.614 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:15.612+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:15.612904+0000) 2022-04-03T22:15:15.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:15.700+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:15.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:15.830+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:16.614 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:16.612+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:16.613093+0000) 2022-04-03T22:15:16.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:16.669+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:16.794 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:16.792+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:17.614 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:17.612+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:17.613275+0000) 2022-04-03T22:15:17.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:17.621+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:17.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:17.746+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:17.898 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:15:17.896+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:15:18.614 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:18.612+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:18.613482+0000) 2022-04-03T22:15:18.654 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:18.652+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:18.710 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:18.708+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:19.615 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:19.612+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:19.613669+0000) 2022-04-03T22:15:19.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:19.619+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:19.673 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:19.672+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:20.615 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:20.613+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:20.613858+0000) 2022-04-03T22:15:20.622 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:20.620+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:20.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:20.623+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:21.578 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:21.576+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:21.615 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:21.613+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:21.614055+0000) 2022-04-03T22:15:21.621 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:21.619+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:22.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:22.590+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:22.615 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:22.613+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:22.614254+0000) 2022-04-03T22:15:22.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:22.623+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:23.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:23.586+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:23.598 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:23.595+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:23.615 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:23.613+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:23.614491+0000) 2022-04-03T22:15:24.600 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:24.598+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:24.610 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:24.607+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:24.615 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:24.614+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:24.614658+0000) 2022-04-03T22:15:25.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:25.591+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:25.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:25.602+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:25.616 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:25.614+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:25.614841+0000) 2022-04-03T22:15:25.665 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T22:15:25.663+0000 7f2e44320700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T22:15:25.665 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T22:15:25.663+0000 7f2e4331e700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T22:15:26.616 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:26.614+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:26.615021+0000) 2022-04-03T22:15:26.634 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:26.631+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:26.654 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:26.652+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:27.114 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:15:27.118 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:15:27.616 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:27.614+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:27.615180+0000) 2022-04-03T22:15:27.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:27.665+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:27.670 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:27.668+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:28.616 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:28.614+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:28.615331+0000) 2022-04-03T22:15:28.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:28.663+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:28.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:28.674+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:29.616 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:29.614+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:29.615486+0000) 2022-04-03T22:15:29.673 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:29.670+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:29.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:29.696+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:30.617 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:30.615+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:30.615657+0000) 2022-04-03T22:15:30.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:30.622+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:30.674 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:30.672+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:31.617 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:31.615+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:31.615857+0000) 2022-04-03T22:15:31.659 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:31.657+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:31.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:31.678+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:32.617 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:32.615+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:32.615981+0000) 2022-04-03T22:15:32.637 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:32.635+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:32.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:32.666+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:32.914 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:15:32.912+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:15:33.617 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:33.615+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:33.616155+0000) 2022-04-03T22:15:33.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:33.634+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:33.643 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:33.641+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:34.617 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:34.615+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:34.616280+0000) 2022-04-03T22:15:34.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:34.639+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:34.642 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:34.640+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:35.617 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:35.616+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:35.616449+0000) 2022-04-03T22:15:35.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:35.628+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:35.633 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:35.630+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:36.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:36.592+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:36.618 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:36.615+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:36.616576+0000) 2022-04-03T22:15:36.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:36.656+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:37.618 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:37.616+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:37.616719+0000) 2022-04-03T22:15:37.626 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:37.624+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:37.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:37.666+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:38.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:38.605+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:38.617 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:38.616+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:38.616844+0000) 2022-04-03T22:15:38.648 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:38.646+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:39.618 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:39.616+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:39.617011+0000) 2022-04-03T22:15:39.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:39.628+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:39.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:39.652+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:40.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:40.586+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:40.618 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:40.616+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:40.617158+0000) 2022-04-03T22:15:40.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:40.654+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:41.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:41.545+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:41.618 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:41.616+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:41.617303+0000) 2022-04-03T22:15:41.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:41.668+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:42.500 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:42.498+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:42.619 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:42.617+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:42.617501+0000) 2022-04-03T22:15:42.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:42.638+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:43.451 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:43.448+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:43.619 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:43.617+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:43.617653+0000) 2022-04-03T22:15:43.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:43.685+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:44.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:44.490+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:44.619 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:44.617+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:44.617734+0000) 2022-04-03T22:15:44.685 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:44.683+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:45.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:45.483+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:45.619 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:45.617+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:45.617827+0000) 2022-04-03T22:15:45.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:45.666+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:46.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:46.503+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:46.619 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:46.617+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:46.617959+0000) 2022-04-03T22:15:46.666 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:46.664+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:47.501 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:47.499+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:47.619 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:47.618+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:47.618180+0000) 2022-04-03T22:15:47.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:47.639+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:47.930 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:15:47.928+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:15:48.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:48.513+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:48.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:48.590+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:48.619 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:48.618+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:48.618364+0000) 2022-04-03T22:15:49.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:49.541+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:49.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:49.558+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:49.619 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:49.618+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:49.618504+0000) 2022-04-03T22:15:50.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:50.547+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:50.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:50.551+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:50.619 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:50.618+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:50.618708+0000) 2022-04-03T22:15:51.550 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:51.548+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:51.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:51.590+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:51.620 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:51.617+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:51.618896+0000) 2022-04-03T22:15:52.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:52.576+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:52.593 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:52.591+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:52.620 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:52.618+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:52.619128+0000) 2022-04-03T22:15:53.539 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:53.537+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:53.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:53.601+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:53.620 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:53.617+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:53.619321+0000) 2022-04-03T22:15:54.552 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:54.550+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:54.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:54.612+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:54.620 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:54.618+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:54.619394+0000) 2022-04-03T22:15:55.520 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:55.517+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:55.615 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:55.613+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:55.620 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:55.618+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:55.619579+0000) 2022-04-03T22:15:56.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:56.474+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:56.621 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:56.618+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:56.619723+0000) 2022-04-03T22:15:56.631 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:56.630+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:57.179 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:15:57.183 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:15:57.462 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:57.460+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:57.621 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:57.618+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:57.619947+0000) 2022-04-03T22:15:57.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:57.656+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:58.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:58.416+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:58.621 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:58.619+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:58.620080+0000) 2022-04-03T22:15:58.639 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:58.637+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:59.463 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:15:59.461+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:15:59.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:15:59.594+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:15:59.621 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:15:59.619+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:15:59.620228+0000) 2022-04-03T22:16:00.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:00.453+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:00.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:00.549+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:00.621 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:00.619+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:00.620429+0000) 2022-04-03T22:16:01.475 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:01.473+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:01.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:01.541+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:01.622 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:01.619+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:01.620600+0000) 2022-04-03T22:16:02.453 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:02.451+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:02.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:02.527+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:02.622 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:02.619+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:02.620703+0000) 2022-04-03T22:16:02.946 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:16:02.944+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:16:03.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:03.463+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:03.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:03.536+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:03.622 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:03.620+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:03.620858+0000) 2022-04-03T22:16:04.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:04.509+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:04.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:04.512+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:04.622 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:04.620+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:04.621001+0000) 2022-04-03T22:16:05.507 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:05.505+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:05.558 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:05.557+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:05.622 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:05.620+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:05.621138+0000) 2022-04-03T22:16:06.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:06.466+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:06.527 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:06.526+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:06.622 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:06.620+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:06.621351+0000) 2022-04-03T22:16:07.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:07.485+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:07.490 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:07.489+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:07.622 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:07.620+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:07.621519+0000) 2022-04-03T22:16:08.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:08.445+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:08.490 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:08.485+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:08.623 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:08.621+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:08.621746+0000) 2022-04-03T22:16:09.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:09.437+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:09.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:09.465+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:09.623 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:09.621+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:09.621895+0000) 2022-04-03T22:16:10.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:10.484+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:10.498 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:10.497+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:10.623 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:10.621+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:10.622057+0000) 2022-04-03T22:16:11.507 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:11.505+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:11.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:11.511+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:11.623 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:11.621+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:11.622211+0000) 2022-04-03T22:16:12.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:12.495+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:12.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:12.545+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:12.623 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:12.621+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:12.622335+0000) 2022-04-03T22:16:13.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:13.514+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:13.556 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:13.554+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:13.623 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:13.621+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:13.622459+0000) 2022-04-03T22:16:14.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:14.541+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:14.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:14.580+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:14.624 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:14.622+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:14.622614+0000) 2022-04-03T22:16:15.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:15.525+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:15.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:15.534+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:15.624 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:15.622+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:15.622764+0000) 2022-04-03T22:16:16.510 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:16.508+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:16.516 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:16.515+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:16.624 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:16.622+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:16.622968+0000) 2022-04-03T22:16:17.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:17.481+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:17.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:17.523+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:17.624 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:17.622+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:17.623101+0000) 2022-04-03T22:16:17.962 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:16:17.960+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:16:18.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:18.483+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:18.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:18.573+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:18.624 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:18.622+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:18.623281+0000) 2022-04-03T22:16:19.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:19.474+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:19.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:19.571+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:19.624 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:19.622+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:19.623431+0000) 2022-04-03T22:16:20.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:20.428+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:20.539 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:20.537+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:20.624 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:20.623+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:20.623574+0000) 2022-04-03T22:16:21.449 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:21.447+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:21.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:21.538+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:21.625 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:21.623+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:21.623701+0000) 2022-04-03T22:16:22.420 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:22.417+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:22.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:22.503+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:22.625 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:22.623+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:22.623826+0000) 2022-04-03T22:16:23.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:23.418+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:23.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:23.543+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:23.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:23.623+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:23.623982+0000) 2022-04-03T22:16:24.433 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:24.431+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:24.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:24.546+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:24.625 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:24.623+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:24.624095+0000) 2022-04-03T22:16:25.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:25.477+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:25.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:25.592+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:25.625 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:25.623+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:25.624269+0000) 2022-04-03T22:16:26.462 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:26.459+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:26.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:26.585+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:26.625 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:26.624+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:26.624386+0000) 2022-04-03T22:16:27.327 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:16:27.331 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:16:27.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:27.438+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:27.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:27.609+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:27.625 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:27.624+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:27.624604+0000) 2022-04-03T22:16:28.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:28.418+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:28.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:28.581+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:28.625 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:28.624+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:28.624741+0000) 2022-04-03T22:16:29.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:29.433+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:29.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:29.540+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:29.626 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:29.624+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:29.624923+0000) 2022-04-03T22:16:30.468 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:30.465+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:30.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:30.588+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:30.626 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:30.624+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:30.625138+0000) 2022-04-03T22:16:31.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:31.421+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:31.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:31.541+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:31.626 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:31.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:31.625278+0000) 2022-04-03T22:16:32.469 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:32.467+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:32.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:32.572+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:32.626 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:32.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:32.625451+0000) 2022-04-03T22:16:32.977 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:16:32.975+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:16:33.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:33.491+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:33.585 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:33.582+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:33.626 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:33.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:33.625572+0000) 2022-04-03T22:16:34.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:34.487+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:34.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:34.616+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:34.626 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:34.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:34.625755+0000) 2022-04-03T22:16:35.508 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:35.506+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:35.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:35.590+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:35.627 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:35.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:35.625935+0000) 2022-04-03T22:16:36.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:36.477+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:36.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:36.551+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:36.627 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:36.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:36.626116+0000) 2022-04-03T22:16:37.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:37.499+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:37.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:37.551+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:37.627 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:37.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:37.626283+0000) 2022-04-03T22:16:38.523 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:38.520+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:38.524 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:38.522+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:38.627 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:38.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:38.626422+0000) 2022-04-03T22:16:39.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:39.488+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:39.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:39.494+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:39.627 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:39.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:39.626570+0000) 2022-04-03T22:16:40.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:40.519+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:40.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:40.528+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:40.628 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:40.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:40.626714+0000) 2022-04-03T22:16:41.550 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:41.548+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:41.570 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:41.568+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:41.628 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:41.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:41.626885+0000) 2022-04-03T22:16:42.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:42.545+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:42.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:42.569+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:42.628 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:42.626+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:42.627088+0000) 2022-04-03T22:16:43.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:43.521+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:43.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:43.535+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:43.628 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:43.626+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:43.627248+0000) 2022-04-03T22:16:44.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:44.510+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:44.528 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:44.526+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:44.628 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:44.626+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:44.627441+0000) 2022-04-03T22:16:45.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:45.511+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:45.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:45.539+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:45.629 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:45.626+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:45.627627+0000) 2022-04-03T22:16:46.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:46.493+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:46.568 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:46.566+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:46.629 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:46.627+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:46.627828+0000) 2022-04-03T22:16:47.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:47.503+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:47.612 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:47.610+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:47.629 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:47.627+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:47.627947+0000) 2022-04-03T22:16:47.993 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:16:47.991+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:16:48.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:48.526+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:48.622 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:48.620+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:48.629 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:48.627+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:48.628098+0000) 2022-04-03T22:16:49.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:49.490+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:49.629 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:49.627+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:49.628270+0000) 2022-04-03T22:16:49.643 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:49.641+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:50.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:50.497+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:50.629 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:50.627+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:50.628494+0000) 2022-04-03T22:16:50.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:50.663+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:51.498 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:51.496+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:51.629 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:51.627+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:51.628716+0000) 2022-04-03T22:16:51.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:51.696+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:52.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:52.454+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:52.630 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:52.628+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:52.628882+0000) 2022-04-03T22:16:52.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:52.725+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:53.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:53.445+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:53.630 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:53.628+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:53.629006+0000) 2022-04-03T22:16:53.773 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:53.771+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:54.463 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:54.461+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:54.630 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:54.628+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:54.629197+0000) 2022-04-03T22:16:54.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:54.794+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:55.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:55.503+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:55.630 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:55.628+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:55.629380+0000) 2022-04-03T22:16:55.773 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:55.772+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:56.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:56.507+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:56.630 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:56.628+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:56.629540+0000) 2022-04-03T22:16:56.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:56.807+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:57.536 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:16:57.540 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:16:57.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:57.553+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:57.631 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:57.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:57.629792+0000) 2022-04-03T22:16:57.819 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:57.817+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:58.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:58.574+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:58.631 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:58.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:58.629960+0000) 2022-04-03T22:16:58.826 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:58.825+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:16:59.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:16:59.559+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:16:59.631 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:16:59.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:16:59.630108+0000) 2022-04-03T22:16:59.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:16:59.780+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:00.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:00.533+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:00.631 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:00.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:00.630310+0000) 2022-04-03T22:17:00.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:00.734+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:01.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:01.523+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:01.631 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:01.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:01.630495+0000) 2022-04-03T22:17:01.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:01.708+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:02.564 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:02.563+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:02.632 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:02.630+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:02.630718+0000) 2022-04-03T22:17:02.716 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:02.714+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:03.009 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:17:03.007+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:17:03.519 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:03.518+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:03.632 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:03.630+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:03.630912+0000) 2022-04-03T22:17:03.705 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:03.704+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:04.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:04.554+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:04.632 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:04.630+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:04.631118+0000) 2022-04-03T22:17:04.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:04.676+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:05.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:05.553+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:05.632 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:05.630+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:05.631223+0000) 2022-04-03T22:17:05.711 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:05.709+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:06.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:06.532+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:06.632 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:06.631+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:06.631422+0000) 2022-04-03T22:17:06.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:06.689+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:07.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:07.578+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:07.633 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:07.631+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:07.631593+0000) 2022-04-03T22:17:07.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:07.686+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:08.582 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:08.580+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:08.633 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:08.631+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:08.631754+0000) 2022-04-03T22:17:08.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:08.663+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:09.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:09.570+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:09.633 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:09.631+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:09.631889+0000) 2022-04-03T22:17:09.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:09.694+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:10.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:10.531+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:10.633 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:10.631+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:10.632102+0000) 2022-04-03T22:17:10.681 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:10.680+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:11.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:11.511+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:11.633 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:11.631+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:11.632257+0000) 2022-04-03T22:17:11.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:11.633+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:12.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:12.551+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:12.610 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:12.608+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:12.633 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:12.632+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:12.632464+0000) 2022-04-03T22:17:13.519 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:13.513+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:13.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:13.626+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:13.633 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:13.632+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:13.632583+0000) 2022-04-03T22:17:14.506 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:14.503+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:14.585 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:14.582+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:14.634 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:14.632+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:14.632824+0000) 2022-04-03T22:17:15.490 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:15.487+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:15.561 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:15.559+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:15.634 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:15.632+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:15.632988+0000) 2022-04-03T22:17:16.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:16.510+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:16.568 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:16.566+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:16.634 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:16.632+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:16.633170+0000) 2022-04-03T22:17:17.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:17.529+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:17.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:17.539+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:17.634 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:17.633+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:17.633293+0000) 2022-04-03T22:17:18.014 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:17:18.012+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:17:18.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:18.492+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:18.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:18.529+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:18.634 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:18.633+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:18.633462+0000) 2022-04-03T22:17:19.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:19.533+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:19.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:19.560+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:19.635 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:19.633+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:19.633612+0000) 2022-04-03T22:17:20.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:20.545+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:20.574 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:20.572+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:20.635 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:20.633+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:20.633829+0000) 2022-04-03T22:17:21.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:21.542+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:21.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:21.549+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:21.635 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:21.632+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:21.633944+0000) 2022-04-03T22:17:22.557 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:22.555+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:22.595 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:22.592+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:22.635 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:22.633+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:22.634091+0000) 2022-04-03T22:17:23.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:23.550+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:23.604 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:23.600+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:23.635 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:23.633+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:23.634253+0000) 2022-04-03T22:17:24.596 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:24.593+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:24.610 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:24.608+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:24.635 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:24.633+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:24.634395+0000) 2022-04-03T22:17:25.564 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:25.561+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:17:25.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:25.568+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:25.635 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:25.632+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:25.634535+0000) 2022-04-03T22:17:26.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:26.527+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:26.558 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:26.556+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:26.636 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:26.633+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:26.634702+0000) 2022-04-03T22:17:27.506 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:27.503+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:27.547 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:27.544+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:27.636 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:27.633+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:27.634865+0000) 2022-04-03T22:17:27.689 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:17:27.693 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:17:28.486 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:28.484+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:28.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:28.581+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:28.636 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:28.634+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:28.635052+0000) 2022-04-03T22:17:29.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:29.532+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:29.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:29.532+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:29.636 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:29.634+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:29.635187+0000) 2022-04-03T22:17:30.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:30.527+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:30.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:30.534+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:30.636 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:30.634+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:30.635357+0000) 2022-04-03T22:17:31.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:31.538+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:31.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:31.553+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:31.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:31.634+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:31.635471+0000) 2022-04-03T22:17:32.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:32.489+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:32.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:32.540+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:32.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:32.634+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:32.635655+0000) 2022-04-03T22:17:33.030 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:17:33.028+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:17:33.526 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:33.524+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:33.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:33.560+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:33.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:33.635+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:33.635863+0000) 2022-04-03T22:17:34.519 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:34.517+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:34.557 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:34.555+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:34.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:34.635+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:34.635994+0000) 2022-04-03T22:17:35.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:35.534+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:35.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:35.566+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:35.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:35.635+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:35.636148+0000) 2022-04-03T22:17:36.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:36.521+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:36.554 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:36.552+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:36.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:36.635+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:36.636319+0000) 2022-04-03T22:17:37.507 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:37.505+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:37.569 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:37.567+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:37.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:37.635+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:37.636494+0000) 2022-04-03T22:17:38.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:38.492+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:38.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:38.592+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:38.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:38.635+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:38.636672+0000) 2022-04-03T22:17:39.528 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:39.526+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:39.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:39.544+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:39.638 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:39.636+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:39.636840+0000) 2022-04-03T22:17:40.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:40.534+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:40.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:40.592+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:40.638 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:40.636+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:40.636962+0000) 2022-04-03T22:17:41.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:41.531+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:41.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:41.587+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:41.638 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:41.636+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:41.637143+0000) 2022-04-03T22:17:42.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:42.503+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:42.585 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:42.584+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:42.638 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:42.636+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:42.637353+0000) 2022-04-03T22:17:43.503 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:43.501+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:43.604 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:43.603+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:43.638 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:43.636+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:43.637539+0000) 2022-04-03T22:17:44.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:44.547+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:44.627 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:44.625+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:44.638 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:44.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:44.637679+0000) 2022-04-03T22:17:45.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:45.514+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:45.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:45.627+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:45.638 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:45.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:45.637847+0000) 2022-04-03T22:17:46.511 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:46.508+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:46.630 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:46.628+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:46.639 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:46.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:46.638016+0000) 2022-04-03T22:17:47.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:47.467+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:47.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:47.619+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:47.639 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:47.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:47.638184+0000) 2022-04-03T22:17:48.045 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:17:48.043+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:17:48.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:48.454+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:48.639 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:48.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:48.638306+0000) 2022-04-03T22:17:48.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:48.654+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:49.472 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:49.469+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:49.639 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:49.638+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:49.638499+0000) 2022-04-03T22:17:49.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:49.661+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:50.510 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:50.507+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:50.630 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:50.628+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:50.639 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:50.638+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:50.638693+0000) 2022-04-03T22:17:51.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:51.533+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:51.640 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:51.638+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:51.638861+0000) 2022-04-03T22:17:51.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:51.675+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:52.546 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:52.544+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:52.640 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:52.638+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:52.639019+0000) 2022-04-03T22:17:52.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:52.669+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:53.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:53.528+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:53.640 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:53.638+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:53.639169+0000) 2022-04-03T22:17:53.678 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:53.677+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:54.572 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:54.570+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:54.640 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:54.638+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:54.639293+0000) 2022-04-03T22:17:54.721 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:54.719+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:55.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:55.609+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:55.640 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:55.639+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:55.639390+0000) 2022-04-03T22:17:55.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:55.690+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:56.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:56.590+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:56.640 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:56.639+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:56.639588+0000) 2022-04-03T22:17:56.740 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:56.738+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:57.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:57.580+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:57.641 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:57.639+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:57.639725+0000) 2022-04-03T22:17:57.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:57.760+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:57.822 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:17:57.826 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:17:58.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:58.549+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:58.641 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:58.639+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:58.639882+0000) 2022-04-03T22:17:58.734 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:58.732+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:17:59.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:17:59.534+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:17:59.641 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:17:59.639+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:17:59.639980+0000) 2022-04-03T22:17:59.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:17:59.692+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:00.501 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:00.499+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:00.641 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:00.639+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:00.640172+0000) 2022-04-03T22:18:00.651 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:00.649+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:01.510 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:01.508+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:01.641 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:01.640+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:01.640310+0000) 2022-04-03T22:18:01.643 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:01.642+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:02.517 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:02.515+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:02.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:02.621+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:02.641 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:02.640+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:02.640384+0000) 2022-04-03T22:18:03.061 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:18:03.059+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:18:03.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:03.522+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:03.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:03.630+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:03.641 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:03.640+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:03.640594+0000) 2022-04-03T22:18:04.519 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:04.518+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:04.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:04.602+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:04.642 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:04.640+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:04.640830+0000) 2022-04-03T22:18:05.546 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:05.545+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:05.559 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:05.557+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:05.642 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:05.639+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:05.641009+0000) 2022-04-03T22:18:06.558 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:06.557+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:06.562 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:06.560+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:06.642 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:06.640+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:06.641162+0000) 2022-04-03T22:18:07.558 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:07.556+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:07.565 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:07.563+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:07.642 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:07.640+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:07.641314+0000) 2022-04-03T22:18:08.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:08.525+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:08.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:08.603+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:08.642 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:08.640+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:08.641487+0000) 2022-04-03T22:18:09.533 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:09.531+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:09.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:09.616+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:09.642 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:09.640+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:09.641647+0000) 2022-04-03T22:18:10.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:10.510+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:10.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:10.640+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:10.641840+0000) 2022-04-03T22:18:10.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:10.649+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:11.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:11.534+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:11.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:11.639+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:11.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:11.641+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:11.641952+0000) 2022-04-03T22:18:12.559 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:12.556+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:12.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:12.641+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:12.642073+0000) 2022-04-03T22:18:12.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:12.667+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:13.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:13.509+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:13.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:13.641+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:13.642196+0000) 2022-04-03T22:18:13.676 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:13.673+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:14.475 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:14.473+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:14.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:14.641+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:14.642360+0000) 2022-04-03T22:18:14.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:14.699+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:15.487 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:15.485+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:15.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:15.641+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:15.642530+0000) 2022-04-03T22:18:15.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:15.700+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:16.499 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:16.497+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:16.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:16.641+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:16.642666+0000) 2022-04-03T22:18:16.693 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:16.691+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:17.547 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:17.544+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:17.644 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:17.642+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:17.642833+0000) 2022-04-03T22:18:17.717 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:17.714+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:18.077 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:18:18.075+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:18:18.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:18.534+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:18.644 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:18.642+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:18.643022+0000) 2022-04-03T22:18:18.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:18.724+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:19.585 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:19.582+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:19.644 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:19.642+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:19.643120+0000) 2022-04-03T22:18:19.753 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:19.751+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:20.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:20.578+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:20.644 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:20.642+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:20.643299+0000) 2022-04-03T22:18:20.788 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:20.785+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:21.565 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:21.563+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:21.644 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:21.642+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:21.643465+0000) 2022-04-03T22:18:21.814 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:21.812+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:22.591 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:22.589+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:22.644 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:22.642+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:22.643647+0000) 2022-04-03T22:18:22.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:22.813+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:23.630 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:23.627+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:23.645 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:23.643+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:23.643877+0000) 2022-04-03T22:18:23.768 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:23.765+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:24.645 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:24.643+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:24.643992+0000) 2022-04-03T22:18:24.661 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:24.659+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:24.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:24.723+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:25.645 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:25.643+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:25.644086+0000) 2022-04-03T22:18:25.673 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:25.671+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:25.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:25.730+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:26.645 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:26.643+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:26.644223+0000) 2022-04-03T22:18:26.711 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:26.709+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:26.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:26.755+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:27.645 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:27.643+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:27.644421+0000) 2022-04-03T22:18:27.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:27.710+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:27.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:27.778+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:27.951 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:18:27.955 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:18:28.645 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:28.644+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:28.644638+0000) 2022-04-03T22:18:28.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:28.730+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:28.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:28.731+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:29.646 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:29.644+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:29.644839+0000) 2022-04-03T22:18:29.767 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:29.765+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:29.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:29.771+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:30.646 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:30.644+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:30.645024+0000) 2022-04-03T22:18:30.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:30.760+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:30.792 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:30.789+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:31.646 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:31.644+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:31.645174+0000) 2022-04-03T22:18:31.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:31.729+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:31.766 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:31.764+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:32.646 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:32.644+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:32.645322+0000) 2022-04-03T22:18:32.704 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:32.702+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:32.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:32.729+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:33.093 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:18:33.091+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:18:33.646 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:33.645+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:33.645432+0000) 2022-04-03T22:18:33.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:33.713+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:33.727 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:33.725+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:34.646 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:34.645+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:34.645612+0000) 2022-04-03T22:18:34.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:34.698+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:34.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:34.770+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:35.647 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:35.645+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:35.645797+0000) 2022-04-03T22:18:35.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:35.666+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:35.752 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:35.749+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:36.647 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:36.645+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:36.645983+0000) 2022-04-03T22:18:36.683 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:36.681+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:36.706 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:36.704+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:37.647 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:37.645+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:37.646155+0000) 2022-04-03T22:18:37.705 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:37.703+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:37.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:37.716+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:38.647 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:38.645+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:38.646336+0000) 2022-04-03T22:18:38.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:38.676+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:38.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:38.746+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:39.647 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:39.646+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:39.646475+0000) 2022-04-03T22:18:39.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:39.688+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:39.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:39.786+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:40.648 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:40.646+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:40.646648+0000) 2022-04-03T22:18:40.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:40.650+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:40.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:40.796+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:41.648 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:41.646+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:41.646827+0000) 2022-04-03T22:18:41.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:41.655+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:41.751 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:41.749+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:42.648 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:42.646+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:42.647045+0000) 2022-04-03T22:18:42.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:42.696+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:42.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:42.791+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:43.648 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:43.646+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:43.647202+0000) 2022-04-03T22:18:43.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:43.667+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:43.816 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:43.814+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:44.648 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:44.646+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:44.647365+0000) 2022-04-03T22:18:44.664 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:44.662+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:44.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:44.849+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:45.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:45.625+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:45.648 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:45.646+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:45.647593+0000) 2022-04-03T22:18:45.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:45.858+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:46.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:46.647+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:46.647754+0000) 2022-04-03T22:18:46.664 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:46.662+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:46.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:46.901+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:47.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:47.613+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:47.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:47.647+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:47.647927+0000) 2022-04-03T22:18:47.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:47.859+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:48.109 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:18:48.107+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:18:48.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:48.576+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:48.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:48.647+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:48.648064+0000) 2022-04-03T22:18:48.895 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:48.894+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:49.586 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:49.584+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:49.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:49.648+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:49.648245+0000) 2022-04-03T22:18:49.929 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:49.927+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:50.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:50.577+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:50.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:50.647+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:50.648387+0000) 2022-04-03T22:18:50.972 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:50.970+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:51.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:51.552+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:51.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:51.647+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:51.648510+0000) 2022-04-03T22:18:51.971 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:51.969+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:52.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:52.558+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:52.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:52.647+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:52.648636+0000) 2022-04-03T22:18:52.986 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:52.985+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:53.607 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:53.606+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:53.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:53.647+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:53.648795+0000) 2022-04-03T22:18:53.963 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:53.961+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:54.650 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:54.647+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:54.648949+0000) 2022-04-03T22:18:54.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:54.650+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:54.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:54.997+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:55.650 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:55.648+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:55.649096+0000) 2022-04-03T22:18:55.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:55.657+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:56.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:56.045+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:56.650 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:56.648+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:56.649317+0000) 2022-04-03T22:18:56.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:56.684+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:57.084 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:57.083+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:57.650 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:57.648+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:57.649421+0000) 2022-04-03T22:18:57.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:57.699+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:58.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:58.040+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:58.099 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:18:58.103 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:18:58.650 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:58.648+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:58.649643+0000) 2022-04-03T22:18:58.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:58.700+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:18:59.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:18:59.038+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:18:59.651 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:18:59.648+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:18:59.649827+0000) 2022-04-03T22:18:59.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:18:59.741+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:00.014 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:00.012+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:00.651 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:00.649+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:00.650050+0000) 2022-04-03T22:19:00.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:00.725+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:01.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:01.061+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:01.651 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:01.649+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:01.650224+0000) 2022-04-03T22:19:01.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:01.729+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:02.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:02.046+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:02.651 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:02.649+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:02.650376+0000) 2022-04-03T22:19:02.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:02.729+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:03.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:03.045+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:03.124 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:19:03.122+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:19:03.651 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:03.649+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:03.650546+0000) 2022-04-03T22:19:03.753 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:03.751+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:04.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:04.000+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:04.652 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:04.649+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:04.650718+0000) 2022-04-03T22:19:04.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:04.707+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:05.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:05.009+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:05.652 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:05.650+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:05.650937+0000) 2022-04-03T22:19:05.735 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:05.733+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:06.014 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:06.012+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:06.652 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:06.650+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:06.651094+0000) 2022-04-03T22:19:06.752 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:06.749+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:07.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:07.053+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:07.652 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:07.650+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:07.651253+0000) 2022-04-03T22:19:07.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:07.777+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:08.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:08.010+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:08.652 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:08.650+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:08.651405+0000) 2022-04-03T22:19:08.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:08.778+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:09.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:09.013+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:09.652 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:09.650+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:09.651566+0000) 2022-04-03T22:19:09.805 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:09.803+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:10.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:10.033+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:10.653 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:10.651+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:10.651742+0000) 2022-04-03T22:19:10.818 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:10.816+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:11.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:11.016+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:11.653 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:11.651+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:11.651908+0000) 2022-04-03T22:19:11.858 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:11.856+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:12.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:12.010+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:12.653 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:12.651+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:12.652008+0000) 2022-04-03T22:19:12.894 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:12.891+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:13.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:13.006+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:13.653 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:13.651+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:13.652181+0000) 2022-04-03T22:19:13.853 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:13.851+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:14.032 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:14.030+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:14.653 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:14.651+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:14.652403+0000) 2022-04-03T22:19:14.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:14.827+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:15.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:15.046+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:15.653 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:15.652+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:15.652592+0000) 2022-04-03T22:19:15.853 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:15.850+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:16.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:16.059+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:16.654 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:16.652+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:16.652796+0000) 2022-04-03T22:19:16.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:16.840+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:17.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:17.080+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:17.654 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:17.652+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:17.652981+0000) 2022-04-03T22:19:17.828 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:17.826+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:18.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:18.064+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:18.141 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:19:18.138+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:19:18.654 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:18.652+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:18.653125+0000) 2022-04-03T22:19:18.814 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:18.812+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:19.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:19.037+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:19.654 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:19.652+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:19.653307+0000) 2022-04-03T22:19:19.819 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:19.817+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:20.021 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:20.019+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:20.654 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:20.652+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:20.653450+0000) 2022-04-03T22:19:20.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:20.780+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:21.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:21.024+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:21.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:21.653+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:21.653611+0000) 2022-04-03T22:19:21.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:21.789+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:22.058 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:22.055+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:22.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:22.653+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:22.653752+0000) 2022-04-03T22:19:22.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:22.767+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:23.017 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:23.014+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:23.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:23.653+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:23.653968+0000) 2022-04-03T22:19:23.803 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:23.800+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:24.002 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:24.000+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:24.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:24.653+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:24.654148+0000) 2022-04-03T22:19:24.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:24.789+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:24.954 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:24.952+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:25.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:25.654+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:25.654312+0000) 2022-04-03T22:19:25.804 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:25.802+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:25.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:25.979+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:26.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:26.654+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:26.654496+0000) 2022-04-03T22:19:26.817 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:26.815+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:26.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:26.962+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:27.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:27.654+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:27.654671+0000) 2022-04-03T22:19:27.827 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:27.825+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:27.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:27.988+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:28.238 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:19:28.242 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:19:28.656 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:28.654+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:28.654933+0000) 2022-04-03T22:19:28.866 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:28.863+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:29.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:29.017+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:29.656 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:29.654+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:29.655084+0000) 2022-04-03T22:19:29.861 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:29.859+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:30.053 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:30.051+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:30.656 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:30.655+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:30.655291+0000) 2022-04-03T22:19:30.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:30.853+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:31.014 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:31.011+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:31.656 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:31.655+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:31.655484+0000) 2022-04-03T22:19:31.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:31.812+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:32.032 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:32.030+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:32.657 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:32.655+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:32.655657+0000) 2022-04-03T22:19:32.775 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:32.773+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:33.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:33.060+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:33.150 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:19:33.148+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:19:33.656 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:33.655+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:33.655786+0000) 2022-04-03T22:19:33.817 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:33.815+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:34.072 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:34.069+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:34.657 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:34.655+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:34.655936+0000) 2022-04-03T22:19:34.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:34.809+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:35.064 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:35.062+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:35.657 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:35.655+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:35.656111+0000) 2022-04-03T22:19:35.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:35.830+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 30 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:36.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:36.073+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:36.658 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:36.655+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:36.656286+0000) 2022-04-03T22:19:36.801 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:36.799+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:37.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:37.078+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:37.657 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:37.655+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:37.656487+0000) 2022-04-03T22:19:37.797 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:37.795+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:38.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:38.080+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:38.658 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:38.655+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:38.656670+0000) 2022-04-03T22:19:38.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:38.793+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:39.043 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:39.041+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:39.658 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:39.655+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:39.656882+0000) 2022-04-03T22:19:39.841 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:39.839+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:40.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:40.024+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:40.658 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:40.656+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:40.657080+0000) 2022-04-03T22:19:40.883 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:40.881+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:41.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:41.005+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:41.658 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:41.656+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:41.657285+0000) 2022-04-03T22:19:41.913 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:41.911+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:42.011 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:42.009+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:42.659 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:42.656+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:42.657475+0000) 2022-04-03T22:19:42.912 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:42.911+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:43.005 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:43.004+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:43.658 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:43.656+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:43.657609+0000) 2022-04-03T22:19:43.949 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:43.947+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:44.027 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:44.026+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:44.659 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:44.656+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:44.657829+0000) 2022-04-03T22:19:44.931 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:44.930+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:45.034 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:45.032+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:45.659 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:45.657+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:45.658022+0000) 2022-04-03T22:19:45.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:45.951+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:46.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:46.045+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:46.659 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:46.657+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:46.658157+0000) 2022-04-03T22:19:46.932 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:46.930+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:47.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:47.062+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:47.659 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:47.657+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:47.658358+0000) 2022-04-03T22:19:47.973 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:47.971+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:48.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:48.080+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:48.166 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:19:48.164+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:19:48.659 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:48.657+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:48.658527+0000) 2022-04-03T22:19:48.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:48.946+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:49.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:49.119+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:49.659 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:49.657+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:49.658709+0000) 2022-04-03T22:19:49.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:49.947+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:50.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:50.072+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:50.660 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:50.658+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:50.658838+0000) 2022-04-03T22:19:50.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:50.946+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:51.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:51.110+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:51.660 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:51.658+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:51.659000+0000) 2022-04-03T22:19:51.982 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:51.981+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:52.150 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:52.149+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:52.660 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:52.658+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:52.659171+0000) 2022-04-03T22:19:52.985 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:52.984+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:53.106 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:53.104+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:53.660 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:53.658+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:53.659315+0000) 2022-04-03T22:19:53.942 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:53.941+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:54.144 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:54.143+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:54.660 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:54.658+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:54.659470+0000) 2022-04-03T22:19:54.896 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:54.894+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:55.102 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:55.100+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:55.661 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:55.659+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:55.659668+0000) 2022-04-03T22:19:55.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:55.875+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:56.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:56.105+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:56.661 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:56.659+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:56.659840+0000) 2022-04-03T22:19:56.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:56.881+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:57.072 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:57.071+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:57.661 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:57.659+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:57.660009+0000) 2022-04-03T22:19:57.838 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:57.837+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:58.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:58.085+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:58.386 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:19:58.390 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:19:58.661 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:58.659+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:58.660197+0000) 2022-04-03T22:19:58.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:58.820+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:19:59.102 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:19:59.099+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:19:59.661 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:19:59.659+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:19:59.660439+0000) 2022-04-03T22:19:59.871 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:19:59.868+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:00.149 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:00.147+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:00.661 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:00.660+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:00.660617+0000) 2022-04-03T22:20:00.918 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:00.916+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:01.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:01.181+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:01.662 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:01.660+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:01.660839+0000) 2022-04-03T22:20:01.890 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:01.887+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:02.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:02.177+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:02.662 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:02.660+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:02.660959+0000) 2022-04-03T22:20:02.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:02.930+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:03.168 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:03.166+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:03.182 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:20:03.180+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:20:03.662 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:03.660+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:03.661165+0000) 2022-04-03T22:20:03.958 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:03.956+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:04.184 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:04.182+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:04.662 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:04.660+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:04.661303+0000) 2022-04-03T22:20:04.913 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:04.910+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:05.208 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:05.206+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:05.662 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:05.661+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:05.661528+0000) 2022-04-03T22:20:05.921 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:05.919+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:06.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:06.247+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:06.662 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:06.661+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:06.661720+0000) 2022-04-03T22:20:06.961 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:06.959+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:07.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:07.223+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:07.663 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:07.661+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:07.661914+0000) 2022-04-03T22:20:07.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:07.913+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:08.180 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:08.178+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:08.663 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:08.661+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:08.662084+0000) 2022-04-03T22:20:08.905 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:08.902+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:09.152 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:09.149+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:09.663 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:09.661+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:09.662256+0000) 2022-04-03T22:20:09.914 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:09.912+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:10.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:10.154+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:10.663 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:10.662+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:10.662447+0000) 2022-04-03T22:20:10.931 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:10.929+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:11.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:11.187+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:11.663 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:11.662+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:11.662579+0000) 2022-04-03T22:20:11.908 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:11.905+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:12.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:12.159+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:12.664 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:12.662+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:12.662752+0000) 2022-04-03T22:20:12.952 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:12.949+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:13.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:13.121+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:13.664 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:13.662+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:13.662902+0000) 2022-04-03T22:20:13.918 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:13.916+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:14.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:14.145+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:14.664 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:14.662+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:14.663115+0000) 2022-04-03T22:20:14.934 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:14.932+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:15.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:15.110+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:15.664 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:15.663+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:15.663248+0000) 2022-04-03T22:20:15.941 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:15.939+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:16.087 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:16.085+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:16.664 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:16.663+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:16.663419+0000) 2022-04-03T22:20:16.988 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:16.986+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:17.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:17.053+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:17.664 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:17.663+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:17.663603+0000) 2022-04-03T22:20:17.941 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:17.939+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:18.065 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:18.063+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:18.198 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:20:18.196+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:20:18.665 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:18.663+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:18.663792+0000) 2022-04-03T22:20:18.903 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:18.901+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:19.092 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:19.090+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:19.665 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:19.663+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:19.663940+0000) 2022-04-03T22:20:19.905 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:19.903+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:20.072 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:20.070+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:20.665 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:20.663+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:20.664145+0000) 2022-04-03T22:20:20.858 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:20.856+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:21.078 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:21.076+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:21.665 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:21.663+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:21.664316+0000) 2022-04-03T22:20:21.888 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:21.886+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:22.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:22.080+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:22.665 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:22.663+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:22.664488+0000) 2022-04-03T22:20:22.914 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:22.912+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:23.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:23.036+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:23.666 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:23.662+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:23.664680+0000) 2022-04-03T22:20:23.954 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:23.952+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:24.017 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:24.015+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:24.666 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:24.663+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:24.664850+0000) 2022-04-03T22:20:24.905 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:24.902+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:24.978 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:24.977+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:20:25.666 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:25.663+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:25.665032+0000) 2022-04-03T22:20:25.937 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:25.935+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:25.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:25.945+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:26.666 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:26.664+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:26.665214+0000) 2022-04-03T22:20:26.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:26.887+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:26.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:26.910+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:27.666 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:27.664+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:27.665341+0000) 2022-04-03T22:20:27.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:27.895+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:27.905 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:27.904+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:28.524 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:20:28.527 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:20:28.666 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:28.664+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:28.665530+0000) 2022-04-03T22:20:28.858 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:28.856+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:28.926 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:28.925+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:29.667 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:29.664+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:29.665666+0000) 2022-04-03T22:20:29.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:29.876+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:29.910 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:29.908+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:30.667 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:30.665+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:30.665858+0000) 2022-04-03T22:20:30.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:30.861+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:30.871 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:30.870+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:31.667 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:31.665+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:31.666035+0000) 2022-04-03T22:20:31.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:31.827+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:31.869 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:31.867+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:32.667 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:32.665+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:32.666146+0000) 2022-04-03T22:20:32.834 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:32.832+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:32.838 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:32.836+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:33.214 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:20:33.212+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:20:33.667 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:33.665+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:33.666270+0000) 2022-04-03T22:20:33.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:33.789+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:33.841 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:33.838+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:34.667 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:34.665+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:34.666396+0000) 2022-04-03T22:20:34.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:34.798+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:34.828 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:34.826+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:35.667 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:35.665+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:35.666556+0000) 2022-04-03T22:20:35.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:35.782+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:35.807 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:35.806+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:36.668 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:36.665+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:36.666742+0000) 2022-04-03T22:20:36.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:36.823+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:36.833 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:36.831+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:37.668 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:37.666+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:37.666917+0000) 2022-04-03T22:20:37.805 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:37.803+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:37.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:37.873+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:38.668 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:38.666+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:38.667124+0000) 2022-04-03T22:20:38.828 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:38.827+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:38.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:38.874+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:39.668 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:39.666+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:39.667293+0000) 2022-04-03T22:20:39.810 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:39.808+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:39.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:39.827+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:40.668 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:40.666+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:40.667456+0000) 2022-04-03T22:20:40.835 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:40.834+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:40.841 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:40.840+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:41.668 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:41.667+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:41.667612+0000) 2022-04-03T22:20:41.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:41.819+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:41.865 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:41.864+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:42.669 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:42.667+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:42.667841+0000) 2022-04-03T22:20:42.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:42.822+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:42.826 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:42.825+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:43.669 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:43.667+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:43.668003+0000) 2022-04-03T22:20:43.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:43.781+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:43.821 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:43.819+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:44.669 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:44.667+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:44.668180+0000) 2022-04-03T22:20:44.810 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:44.808+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:44.819 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:44.817+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:45.669 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:45.667+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:45.668349+0000) 2022-04-03T22:20:45.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:45.769+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:45.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:45.777+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:46.669 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:46.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:46.668530+0000) 2022-04-03T22:20:46.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:46.749+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:46.761 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:46.760+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:47.669 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:47.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:47.668668+0000) 2022-04-03T22:20:47.740 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:47.739+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:47.776 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:47.775+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:48.230 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:20:48.227+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:20:48.670 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:48.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:48.668863+0000) 2022-04-03T22:20:48.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:48.714+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:48.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:48.728+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:49.670 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:49.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:49.669032+0000) 2022-04-03T22:20:49.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:49.746+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:49.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:49.758+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:50.670 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:50.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:50.669167+0000) 2022-04-03T22:20:50.746 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:50.744+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:50.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:50.746+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:51.670 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:51.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:51.669301+0000) 2022-04-03T22:20:51.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:51.697+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:51.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:51.779+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:52.670 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:52.669+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:52.669462+0000) 2022-04-03T22:20:52.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:52.726+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:52.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:52.746+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:53.671 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:53.669+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:53.669646+0000) 2022-04-03T22:20:53.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:53.728+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:53.774 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:53.772+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:54.671 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:54.669+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:54.669838+0000) 2022-04-03T22:20:54.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:54.713+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:54.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:54.731+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:55.671 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:55.669+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:55.670016+0000) 2022-04-03T22:20:55.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:55.674+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:55.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:55.741+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:56.671 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:56.669+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:56.670211+0000) 2022-04-03T22:20:56.675 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:56.673+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:56.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:56.753+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:57.671 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:57.670+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:57.670371+0000) 2022-04-03T22:20:57.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:57.677+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:57.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:57.740+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:58.604 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:20:58.608 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:20:58.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:58.667+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:58.671 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:58.669+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:58.670628+0000) 2022-04-03T22:20:58.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:58.715+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:20:59.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:20:59.669+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:20:59.672 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:20:59.670+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:20:59.670809+0000) 2022-04-03T22:20:59.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:20:59.710+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:00.672 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:00.670+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:00.670968+0000) 2022-04-03T22:21:00.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:00.683+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:00.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:00.696+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:01.648 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:01.645+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:01.672 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:01.670+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:01.671148+0000) 2022-04-03T22:21:01.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:01.746+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:02.662 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:02.660+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:02.672 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:02.671+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:02.671358+0000) 2022-04-03T22:21:02.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:02.698+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:03.245 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:21:03.243+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:21:03.672 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:03.671+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:03.671531+0000) 2022-04-03T22:21:03.691 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:03.689+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:03.707 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:03.705+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:04.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:04.645+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:04.672 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:04.671+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:04.671628+0000) 2022-04-03T22:21:04.722 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:04.720+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:05.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:05.596+0000 7f88d0ac1700 -1 osd.4 127 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:05.673 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:05.670+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:05.671830+0000) 2022-04-03T22:21:05.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:05.721+0000 7fd3b190b700 -1 osd.6 127 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:05.919 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-03T22:21:05.917+0000 7f70556b6700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T22:21:05.934 INFO:tasks.ceph.mds.f.smithi186.stderr: -1> 2022-04-03T22:21:05.917+0000 7f70556b6700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T22:21:05.956 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-03T22:21:06.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:06.630+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:06.673 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:06.670+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:06.671999+0000) 2022-04-03T22:21:06.684 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:06.682+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:07.645 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:07.643+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:07.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:07.665+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:07.673 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:07.671+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:07.672134+0000) 2022-04-03T22:21:08.642 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:08.640+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:08.673 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:08.671+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:08.672323+0000) 2022-04-03T22:21:08.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:08.674+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:09.673 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:09.671+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:09.672429+0000) 2022-04-03T22:21:09.676 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:09.674+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:09.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:09.689+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:10.673 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:10.671+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:10.672595+0000) 2022-04-03T22:21:10.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:10.686+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:10.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:10.700+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:11.659 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:11.656+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:11.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:11.660+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:11.673 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:11.671+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:11.672697+0000) 2022-04-03T22:21:12.674 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:12.671+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:12.672849+0000) 2022-04-03T22:21:12.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:12.681+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:12.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:12.700+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:13.674 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:13.672+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:13.673033+0000) 2022-04-03T22:21:13.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:13.688+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:13.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:13.727+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:14.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:14.653+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:14.674 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:14.672+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:14.673202+0000) 2022-04-03T22:21:14.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:14.727+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:15.674 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:15.672+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:15.673359+0000) 2022-04-03T22:21:15.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:15.677+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:15.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:15.705+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:16.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:16.646+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:16.674 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:16.672+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:16.673544+0000) 2022-04-03T22:21:16.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:16.673+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:17.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:17.636+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:17.674 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:17.672+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:17.673720+0000) 2022-04-03T22:21:17.704 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:17.702+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:18.261 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:21:18.259+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:21:18.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:18.656+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:18.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:18.658+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:18.675 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:18.673+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:18.673926+0000) 2022-04-03T22:21:19.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:19.638+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:19.675 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:19.673+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:19.674134+0000) 2022-04-03T22:21:19.704 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:19.702+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:20.650 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:20.648+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:20.675 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:20.673+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:20.674341+0000) 2022-04-03T22:21:20.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:20.717+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:21.675 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:21.673+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:21.676 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:21.673+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:21.674500+0000) 2022-04-03T22:21:21.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:21.677+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:22.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:22.630+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:22.675 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:22.673+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:22.674670+0000) 2022-04-03T22:21:22.676 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:22.675+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:23.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:23.666+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:23.931 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:23.674+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:23.674838+0000) 2022-04-03T22:21:23.932 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:23.718+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:24.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:24.626+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:24.676 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:24.674+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:24.674988+0000) 2022-04-03T22:21:24.768 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:24.766+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:25.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:25.665+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:25.676 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:25.674+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:25.675126+0000) 2022-04-03T22:21:25.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:25.735+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:26.676 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:26.674+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:26.675342+0000) 2022-04-03T22:21:26.697 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:26.695+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:26.705 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:26.703+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:27.676 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:27.674+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:27.675509+0000) 2022-04-03T22:21:27.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:27.724+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:27.745 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:27.743+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:28.677 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:28.675+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:28.675684+0000) 2022-04-03T22:21:28.707 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:28.706+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:28.735 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:21:28.739 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:21:28.772 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:28.770+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:29.677 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:29.675+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:29.675887+0000) 2022-04-03T22:21:29.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:29.712+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:29.803 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:29.801+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:30.677 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:30.675+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:30.676138+0000) 2022-04-03T22:21:30.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:30.737+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:30.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:30.841+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:31.677 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:31.675+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:31.676384+0000) 2022-04-03T22:21:31.736 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:31.734+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:31.853 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:31.851+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:32.677 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:32.676+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:32.676540+0000) 2022-04-03T22:21:32.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:32.696+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:32.852 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:32.851+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:33.277 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:21:33.274+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:21:33.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:33.675+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:33.677 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:33.676+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:33.676758+0000) 2022-04-03T22:21:33.883 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:33.881+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:34.678 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:34.676+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:34.677028+0000) 2022-04-03T22:21:34.692 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:34.691+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:34.890 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:34.889+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:35.678 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:35.676+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:35.677277+0000) 2022-04-03T22:21:35.720 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:35.719+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:35.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:35.860+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:36.678 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:36.677+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:36.677491+0000) 2022-04-03T22:21:36.765 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:36.763+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:36.840 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:36.839+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:37.679 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:37.677+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:37.677730+0000) 2022-04-03T22:21:37.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:37.725+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:37.797 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:37.796+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:38.679 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:38.677+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:38.678011+0000) 2022-04-03T22:21:38.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:38.740+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:38.776 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:38.775+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:39.679 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:39.677+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:39.678230+0000) 2022-04-03T22:21:39.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:39.730+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:39.766 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:39.764+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:40.679 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:40.678+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:40.678506+0000) 2022-04-03T22:21:40.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:40.736+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:40.747 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:40.746+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:41.679 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:41.677+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:41.678681+0000) 2022-04-03T22:21:41.705 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:41.704+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:41.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:41.749+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:42.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:42.662+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:42.679 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:42.678+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:42.678872+0000) 2022-04-03T22:21:42.764 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:42.763+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:43.619 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:43.617+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:43.680 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:43.678+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:43.679065+0000) 2022-04-03T22:21:43.744 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:43.743+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:44.590 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:44.588+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:44.680 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:44.679+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:44.679285+0000) 2022-04-03T22:21:44.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:44.759+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:45.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:45.586+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:45.683 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:45.679+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:45.679482+0000) 2022-04-03T22:21:45.753 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:45.752+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:46.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:46.601+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:46.681 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:46.679+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:46.679716+0000) 2022-04-03T22:21:46.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:46.732+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:47.627 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:47.625+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:47.681 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:47.679+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:47.679900+0000) 2022-04-03T22:21:47.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:47.748+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:48.292 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:21:48.290+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:21:48.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:48.591+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:48.681 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:48.679+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:48.680062+0000) 2022-04-03T22:21:48.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:48.771+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:49.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:49.544+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:49.681 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:49.679+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:49.680234+0000) 2022-04-03T22:21:49.746 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:49.744+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:50.559 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:50.557+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:50.681 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:50.679+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:50.680457+0000) 2022-04-03T22:21:50.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:50.746+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:51.555 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:51.553+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:51.682 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:51.679+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:51.680672+0000) 2022-04-03T22:21:51.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:51.757+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:52.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:52.540+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:52.682 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:52.679+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:52.680901+0000) 2022-04-03T22:21:52.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:52.799+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:53.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:53.569+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:53.682 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:53.680+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:53.681109+0000) 2022-04-03T22:21:53.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:53.803+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:54.569 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:54.567+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:54.682 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:54.680+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:54.681360+0000) 2022-04-03T22:21:54.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:54.780+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:55.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:55.532+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:55.682 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:55.680+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:55.681549+0000) 2022-04-03T22:21:55.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:55.758+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:56.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:56.564+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:56.683 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:56.680+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:56.681690+0000) 2022-04-03T22:21:56.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:56.749+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:57.556 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:57.554+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:57.683 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:57.680+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:57.681909+0000) 2022-04-03T22:21:57.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:57.719+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:58.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:58.549+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:58.683 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:58.681+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:58.682102+0000) 2022-04-03T22:21:58.710 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:58.708+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:21:58.873 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:21:58.877 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:21:59.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:21:59.535+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:21:59.683 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:21:59.681+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:21:59.682318+0000) 2022-04-03T22:21:59.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:21:59.713+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:00.490 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:00.488+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:00.683 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:00.681+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:00.682588+0000) 2022-04-03T22:22:00.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:00.681+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:01.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:01.498+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:01.670 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:01.667+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:01.683 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:01.682+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:01.682834+0000) 2022-04-03T22:22:02.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:02.510+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:02.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:02.650+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:02.684 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:02.682+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:02.682983+0000) 2022-04-03T22:22:03.302 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:22:03.300+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:22:03.562 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:03.560+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:03.655 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:03.653+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:03.684 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:03.682+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:03.683247+0000) 2022-04-03T22:22:04.601 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:04.599+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:04.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:04.622+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:04.684 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:04.682+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:04.683446+0000) 2022-04-03T22:22:05.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:05.586+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:05.673 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:05.670+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:05.684 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:05.682+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:05.683591+0000) 2022-04-03T22:22:06.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:06.578+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:06.685 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:06.683+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:06.683842+0000) 2022-04-03T22:22:06.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:06.711+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:07.617 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:07.615+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:07.685 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:07.683+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:07.684037+0000) 2022-04-03T22:22:07.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:07.739+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:08.600 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:08.598+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:08.685 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:08.683+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:08.684178+0000) 2022-04-03T22:22:08.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:08.753+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:09.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:09.578+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:09.686 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:09.683+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:09.684375+0000) 2022-04-03T22:22:09.856 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:09.755+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:10.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:10.537+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:10.685 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:10.683+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:10.684560+0000) 2022-04-03T22:22:10.768 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:10.766+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:11.566 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:11.564+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:11.686 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:11.684+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:11.684792+0000) 2022-04-03T22:22:11.774 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:11.772+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:12.599 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:12.596+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:12.686 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:12.684+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:12.684996+0000) 2022-04-03T22:22:12.766 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:12.764+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:13.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:13.605+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:13.686 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:13.684+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:13.685209+0000) 2022-04-03T22:22:13.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:13.778+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:14.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:14.639+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:14.686 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:14.684+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:14.685421+0000) 2022-04-03T22:22:14.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:14.804+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:15.624 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:15.622+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:15.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:15.685+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:15.685663+0000) 2022-04-03T22:22:15.784 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:15.782+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:16.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:16.644+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:16.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:16.685+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:16.685901+0000) 2022-04-03T22:22:16.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:16.735+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:17.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:17.630+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:17.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:17.685+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:17.686132+0000) 2022-04-03T22:22:17.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:17.710+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:18.318 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:22:18.316+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:22:18.586 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:18.584+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:18.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:18.685+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:18.686266+0000) 2022-04-03T22:22:18.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:18.697+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:19.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:19.549+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:19.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:19.686+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:19.686433+0000) 2022-04-03T22:22:19.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:19.728+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:20.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:20.569+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:20.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:20.686+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:20.686654+0000) 2022-04-03T22:22:20.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:20.728+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:21.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:21.604+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:21.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:21.686+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:21.686834+0000) 2022-04-03T22:22:21.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:21.730+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:22.624 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:22.622+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:22.688 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:22.686+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:22.687031+0000) 2022-04-03T22:22:22.783 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:22.781+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:23.612 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:23.610+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:23.688 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:23.686+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:23.687255+0000) 2022-04-03T22:22:23.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:23.733+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:24.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:24.586+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:24.688 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:24.687+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:24.687469+0000) 2022-04-03T22:22:24.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:24.754+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:25.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:25.548+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:25.688 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:25.687+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:25.687642+0000) 2022-04-03T22:22:25.739 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:25.737+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:26.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:26.542+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:26.689 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:26.687+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:26.687850+0000) 2022-04-03T22:22:26.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:26.694+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:27.519 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:27.517+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:27.689 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:27.687+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:27.688041+0000) 2022-04-03T22:22:27.744 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:27.743+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:28.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:28.549+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:28.689 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:28.688+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:28.688253+0000) 2022-04-03T22:22:28.759 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:28.758+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:28.940 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:22:28.944 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:22:29.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:29.551+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:29.689 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:29.688+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:29.688471+0000) 2022-04-03T22:22:29.763 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:29.762+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:30.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:30.542+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:30.689 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:30.688+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:30.688691+0000) 2022-04-03T22:22:30.746 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:30.745+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:31.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:31.535+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:31.690 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:31.688+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:31.688840+0000) 2022-04-03T22:22:31.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:31.786+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:32.487 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:32.486+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:32.690 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:32.688+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:32.689052+0000) 2022-04-03T22:22:32.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:32.771+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:33.334 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:22:33.332+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:22:33.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:33.483+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:33.690 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:33.689+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:33.689242+0000) 2022-04-03T22:22:33.763 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:33.762+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:34.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:34.457+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:34.690 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:34.688+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:34.689398+0000) 2022-04-03T22:22:34.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:34.722+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:35.462 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:35.460+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:35.690 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:35.688+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:35.689610+0000) 2022-04-03T22:22:35.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:35.720+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:36.469 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:36.468+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:36.691 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:36.688+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:36.689831+0000) 2022-04-03T22:22:36.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:36.712+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:37.502 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:37.501+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:37.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:37.688+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:37.690 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:37.689+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:37.690003+0000) 2022-04-03T22:22:38.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:38.542+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:38.662 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:38.660+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:38.691 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:38.689+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:38.690204+0000) 2022-04-03T22:22:39.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:39.516+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:39.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:39.687+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:39.691 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:39.689+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:39.690388+0000) 2022-04-03T22:22:40.490 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:40.488+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:40.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:40.676+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:40.691 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:40.689+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:40.690585+0000) 2022-04-03T22:22:41.524 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:41.522+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:41.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:41.665+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:41.691 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:41.689+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:41.690830+0000) 2022-04-03T22:22:42.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:42.494+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:42.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:42.629+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:42.692 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:42.690+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:42.690992+0000) 2022-04-03T22:22:43.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:43.457+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:43.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:43.611+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:43.692 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:43.690+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:43.691212+0000) 2022-04-03T22:22:44.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:44.462+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:44.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:44.566+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:44.692 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:44.690+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:44.691423+0000) 2022-04-03T22:22:45.499 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:45.497+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:45.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:45.534+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:45.693 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:45.690+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:45.691568+0000) 2022-04-03T22:22:46.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:46.507+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:46.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:46.542+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:46.693 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:46.691+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:46.691804+0000) 2022-04-03T22:22:47.473 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:47.471+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:47.526 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:47.523+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:47.693 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:47.691+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:47.692001+0000) 2022-04-03T22:22:48.342 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:22:48.340+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:22:48.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:48.432+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:48.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:48.481+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:48.693 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:48.691+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:48.692202+0000) 2022-04-03T22:22:49.479 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:49.476+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:49.516 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:49.514+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:49.693 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:49.691+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:49.692400+0000) 2022-04-03T22:22:50.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:50.510+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:50.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:50.514+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:50.693 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:50.691+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:50.692608+0000) 2022-04-03T22:22:51.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:51.482+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:51.517 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:51.515+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:51.694 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:51.692+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:51.692858+0000) 2022-04-03T22:22:52.504 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:52.502+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:52.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:52.529+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:52.694 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:52.692+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:52.693034+0000) 2022-04-03T22:22:53.475 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:53.472+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:53.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:53.507+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:53.694 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:53.692+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:53.693236+0000) 2022-04-03T22:22:54.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:54.495+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:54.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:54.554+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:54.694 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:54.692+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:54.693453+0000) 2022-04-03T22:22:55.465 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:55.463+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:55.540 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:55.537+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:55.695 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:55.693+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:55.693670+0000) 2022-04-03T22:22:56.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:56.414+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:56.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:56.572+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:56.695 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:56.693+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:56.693866+0000) 2022-04-03T22:22:57.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:57.411+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:57.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:57.551+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:57.695 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:57.693+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:57.694058+0000) 2022-04-03T22:22:58.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:58.397+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:58.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:58.588+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:58.695 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:58.693+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:58.694210+0000) 2022-04-03T22:22:59.013 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:22:59.016 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:22:59.387 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:22:59.385+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:22:59.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:22:59.539+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:22:59.695 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:22:59.693+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:22:59.694353+0000) 2022-04-03T22:23:00.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:00.336+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:00.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:00.511+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:00.695 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:00.693+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:00.694591+0000) 2022-04-03T22:23:01.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:01.349+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:01.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:01.527+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:01.695 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:01.694+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:01.694761+0000) 2022-04-03T22:23:02.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:02.338+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:02.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:02.549+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:02.696 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:02.694+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:02.694931+0000) 2022-04-03T22:23:03.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:03.290+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:03.358 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:23:03.356+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:23:03.600 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:03.598+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:03.696 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:03.694+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:03.695122+0000) 2022-04-03T22:23:04.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:04.306+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:04.582 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:04.580+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:04.696 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:04.694+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:04.695292+0000) 2022-04-03T22:23:05.344 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:05.342+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:05.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:05.586+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:05.696 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:05.695+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:05.695501+0000) 2022-04-03T22:23:06.334 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:06.331+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:06.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:06.557+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:06.697 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:06.694+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:06.695664+0000) 2022-04-03T22:23:07.337 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:07.335+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:07.533 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:07.531+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:07.697 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:07.695+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:07.695885+0000) 2022-04-03T22:23:08.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:08.312+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:08.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:08.528+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:08.697 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:08.695+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:08.696119+0000) 2022-04-03T22:23:09.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:09.282+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:09.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:09.483+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:09.697 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:09.696+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:09.696320+0000) 2022-04-03T22:23:10.267 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:10.265+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:10.502 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:10.499+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:10.697 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:10.696+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:10.696510+0000) 2022-04-03T22:23:11.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:11.284+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:11.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:11.483+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:11.697 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:11.696+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:11.696685+0000) 2022-04-03T22:23:12.306 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:12.304+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:12.460 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:12.458+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:12.697 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:12.696+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:12.696887+0000) 2022-04-03T22:23:13.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:13.295+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:13.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:13.467+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:13.698 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:13.696+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:13.697103+0000) 2022-04-03T22:23:14.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:14.290+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:14.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:14.485+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:14.698 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:14.697+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:14.697320+0000) 2022-04-03T22:23:15.269 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:15.267+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:15.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:15.523+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:15.698 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:15.697+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:15.697543+0000) 2022-04-03T22:23:16.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:16.242+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:16.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:16.536+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:16.698 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:16.697+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:16.697701+0000) 2022-04-03T22:23:17.222 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:17.220+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:17.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:17.549+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:17.698 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:17.697+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:17.697889+0000) 2022-04-03T22:23:18.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:18.247+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:18.374 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:23:18.372+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:23:18.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:18.582+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:18.699 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:18.697+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:18.698060+0000) 2022-04-03T22:23:19.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:19.293+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:19.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:19.594+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:19.699 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:19.697+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:19.698251+0000) 2022-04-03T22:23:20.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:20.279+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:20.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:20.641+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:20.699 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:20.697+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:20.698424+0000) 2022-04-03T22:23:21.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:21.278+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:21.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:21.655+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:21.699 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:21.697+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:21.698637+0000) 2022-04-03T22:23:22.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:22.315+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:22.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:22.697+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:22.700 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:22.697+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:22.698845+0000) 2022-04-03T22:23:23.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:23.277+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:23.700 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:23.698+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:23.699055+0000) 2022-04-03T22:23:23.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:23.740+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:24.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:24.308+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:24.700 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:24.697+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:24.699229+0000) 2022-04-03T22:23:24.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:24.759+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:25.264 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:25.262+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:25.700 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:25.698+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:25.699378+0000) 2022-04-03T22:23:25.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:25.727+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:26.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:26.249+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:26.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:26.690+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:26.700 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:26.698+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:26.699586+0000) 2022-04-03T22:23:27.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:27.233+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:27.701 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:27.698+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:27.699695+0000) 2022-04-03T22:23:27.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:27.733+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:28.255 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:28.254+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:28.701 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:28.699+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:28.699863+0000) 2022-04-03T22:23:28.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:28.721+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:29.148 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:23:29.152 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:23:29.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:29.277+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:29.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:29.680+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:29.701 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:29.699+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:29.700102+0000) 2022-04-03T22:23:30.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:30.252+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:30.674 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:30.672+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:30.701 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:30.699+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:30.700310+0000) 2022-04-03T22:23:31.256 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:31.255+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:31.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:31.688+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:31.774 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:31.699+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:31.700493+0000) 2022-04-03T22:23:32.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:32.287+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:32.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:32.677+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:32.701 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:32.699+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:32.700659+0000) 2022-04-03T22:23:33.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:33.321+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:33.377 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:23:33.375+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:23:33.702 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:33.700+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:33.700851+0000) 2022-04-03T22:23:33.727 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:33.726+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:34.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:34.283+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:34.702 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:34.700+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:34.701020+0000) 2022-04-03T22:23:34.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:34.720+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:35.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:35.312+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:35.702 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:35.700+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:35.701222+0000) 2022-04-03T22:23:35.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:35.717+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:36.345 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:36.344+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:36.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:36.690+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:36.702 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:36.700+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:36.701440+0000) 2022-04-03T22:23:37.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:37.339+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:37.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:37.698+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:37.702 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:37.700+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:37.701654+0000) 2022-04-03T22:23:38.354 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:38.352+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:38.703 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:38.701+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:38.701792+0000) 2022-04-03T22:23:38.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:38.731+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:39.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:39.336+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:39.703 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:39.701+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:39.701987+0000) 2022-04-03T22:23:39.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:39.726+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:40.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:40.318+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:40.703 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:40.701+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:40.702142+0000) 2022-04-03T22:23:40.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:40.754+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:41.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:41.367+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:41.703 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:41.701+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:41.702370+0000) 2022-04-03T22:23:41.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:41.716+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:42.360 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:42.358+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:42.703 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:42.701+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:42.702540+0000) 2022-04-03T22:23:42.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:42.710+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:43.397 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:43.395+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:43.703 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:43.702+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:43.702708+0000) 2022-04-03T22:23:43.727 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:43.725+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:44.422 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:44.420+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:44.704 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:44.702+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:44.702943+0000) 2022-04-03T22:23:44.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:44.738+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:45.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:45.442+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:45.704 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:45.702+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:45.703113+0000) 2022-04-03T22:23:45.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:45.727+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:46.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:46.399+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:46.704 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:46.702+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:46.703316+0000) 2022-04-03T22:23:46.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:46.754+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:47.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:47.381+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:47.704 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:47.702+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:47.703556+0000) 2022-04-03T22:23:47.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:47.728+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:48.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:48.378+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:48.393 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:23:48.391+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:23:48.705 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:48.703+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:48.703762+0000) 2022-04-03T22:23:48.712 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:48.710+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:49.402 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:49.400+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:49.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:49.678+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:49.705 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:49.703+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:49.703977+0000) 2022-04-03T22:23:50.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:50.405+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:50.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:50.629+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:50.705 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:50.703+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:50.704191+0000) 2022-04-03T22:23:51.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:51.405+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:51.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:51.628+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:51.705 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:51.704+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:51.704410+0000) 2022-04-03T22:23:52.424 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:52.422+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:52.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:52.609+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:52.705 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:52.704+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:52.704631+0000) 2022-04-03T22:23:53.449 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:53.447+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:53.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:53.635+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:53.705 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:53.704+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:53.704807+0000) 2022-04-03T22:23:54.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:54.427+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:54.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:54.683+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:54.706 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:54.704+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:54.705016+0000) 2022-04-03T22:23:55.453 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:55.451+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:55.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:55.672+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:55.706 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:55.704+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:55.705187+0000) 2022-04-03T22:23:56.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:56.465+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:56.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:56.686+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:56.706 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:56.704+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:56.705409+0000) 2022-04-03T22:23:57.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:57.486+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:57.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:57.665+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:57.706 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:57.705+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:57.705614+0000) 2022-04-03T22:23:58.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:58.521+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:58.622 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:58.619+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:58.707 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:58.705+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:58.705853+0000) 2022-04-03T22:23:59.279 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:23:59.282 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:23:59.480 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:23:59.478+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:23:59.585 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:23:59.583+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:23:59.707 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:23:59.705+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:23:59.706065+0000) 2022-04-03T22:24:00.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:00.462+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:00.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:00.538+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:00.707 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:00.706+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:00.706280+0000) 2022-04-03T22:24:01.475 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:01.473+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:01.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:01.491+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:01.707 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:01.706+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:01.706482+0000) 2022-04-03T22:24:02.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:02.457+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:02.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:02.511+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:02.707 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:02.705+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:02.706686+0000) 2022-04-03T22:24:03.409 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:24:03.407+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:24:03.431 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:03.428+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:03.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:03.479+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:03.708 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:03.706+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:03.706888+0000) 2022-04-03T22:24:04.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:04.443+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:04.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:04.519+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:04.708 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:04.706+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:04.707102+0000) 2022-04-03T22:24:05.432 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:05.431+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:05.477 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:05.475+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:05.708 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:05.706+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:05.707330+0000) 2022-04-03T22:24:06.454 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:06.452+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:06.502 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:06.500+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:06.708 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:06.706+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:06.707555+0000) 2022-04-03T22:24:07.466 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:07.464+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:07.733 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:07.706+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:07.707742+0000) 2022-04-03T22:24:07.734 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:07.526+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:08.478 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:08.476+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:08.482 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:08.480+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:08.709 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:08.706+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:08.707956+0000) 2022-04-03T22:24:09.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:09.467+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:09.470 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:09.467+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:09.709 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:09.707+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:09.708167+0000) 2022-04-03T22:24:10.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:10.428+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 21 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:10.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:10.474+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:10.709 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:10.707+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:10.708355+0000) 2022-04-03T22:24:11.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:11.390+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:11.480 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:11.478+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:11.709 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:11.707+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:11.708557+0000) 2022-04-03T22:24:12.388 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:12.386+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:12.524 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:12.521+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:12.709 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:12.707+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:12.708795+0000) 2022-04-03T22:24:13.429 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:13.427+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:13.555 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:13.553+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:13.710 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:13.708+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:13.709005+0000) 2022-04-03T22:24:14.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:14.402+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:14.520 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:14.518+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:14.710 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:14.708+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:14.709155+0000) 2022-04-03T22:24:15.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:15.356+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:15.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:15.518+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:15.710 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:15.708+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:15.709350+0000) 2022-04-03T22:24:16.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:16.319+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:16.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:16.557+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:16.710 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:16.708+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:16.709561+0000) 2022-04-03T22:24:17.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:17.316+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:17.566 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:17.564+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:17.710 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:17.708+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:17.709753+0000) 2022-04-03T22:24:18.364 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:18.363+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:18.425 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:24:18.423+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:24:18.517 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:18.516+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:18.711 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:18.709+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:18.709908+0000) 2022-04-03T22:24:19.332 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:19.331+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:19.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:19.491+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:19.711 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:19.709+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:19.710099+0000) 2022-04-03T22:24:20.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:20.283+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:20.449 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:20.448+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:20.711 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:20.709+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:20.710311+0000) 2022-04-03T22:24:21.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:21.245+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:21.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:21.405+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:21.711 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:21.709+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:21.710468+0000) 2022-04-03T22:24:22.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:22.214+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:22.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:22.439+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:22.711 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:22.709+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:22.710552+0000) 2022-04-03T22:24:23.252 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:23.250+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:23.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:23.426+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:23.711 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:23.710+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:23.710728+0000) 2022-04-03T22:24:24.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:24.221+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:24.396 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:24.395+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:24.712 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:24.710+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:24.710899+0000) 2022-04-03T22:24:25.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:25.172+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:25.379 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:25.378+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:25.712 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:25.710+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:25.711104+0000) 2022-04-03T22:24:26.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:26.213+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:26.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:26.360+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:26.712 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:26.710+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:26.711283+0000) 2022-04-03T22:24:27.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:27.185+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:27.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:27.400+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:27.712 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:27.710+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:27.711476+0000) 2022-04-03T22:24:28.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:28.196+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:28.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:28.361+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:28.712 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:28.711+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:28.711687+0000) 2022-04-03T22:24:29.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:29.238+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:29.319 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:29.318+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:29.409 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:24:29.412 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:24:29.713 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:29.711+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:29.711917+0000) 2022-04-03T22:24:30.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:30.288+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:30.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:30.291+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:30.713 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:30.711+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:30.712050+0000) 2022-04-03T22:24:31.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:31.323+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:31.342 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:31.341+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:31.713 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:31.711+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:31.712233+0000) 2022-04-03T22:24:32.342 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:32.341+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:32.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:32.356+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:32.713 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:32.711+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:32.712430+0000) 2022-04-03T22:24:33.343 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:33.340+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:33.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:33.355+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:33.441 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:24:33.438+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:24:33.713 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:33.711+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:33.712658+0000) 2022-04-03T22:24:34.325 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:34.323+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:34.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:34.353+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:34.714 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:34.712+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:34.712830+0000) 2022-04-03T22:24:35.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:35.348+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:35.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:35.388+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:35.714 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:35.712+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:35.713018+0000) 2022-04-03T22:24:36.307 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:36.305+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:36.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:36.362+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:36.714 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:36.712+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:36.713179+0000) 2022-04-03T22:24:37.325 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:37.323+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:37.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:37.361+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:37.714 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:37.713+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:37.713396+0000) 2022-04-03T22:24:38.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:38.293+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:38.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:38.318+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:38.714 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:38.713+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:38.713612+0000) 2022-04-03T22:24:39.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:39.310+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:39.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:39.313+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:39.715 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:39.713+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:39.713840+0000) 2022-04-03T22:24:40.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:40.286+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:40.346 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:40.344+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:40.715 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:40.713+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:40.714052+0000) 2022-04-03T22:24:41.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:41.238+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:41.347 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:41.344+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:41.715 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:41.713+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:41.714215+0000) 2022-04-03T22:24:42.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:42.216+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:42.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:42.333+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:42.715 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:42.714+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:42.714464+0000) 2022-04-03T22:24:43.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:43.223+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:43.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:43.369+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:43.715 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:43.714+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:43.714641+0000) 2022-04-03T22:24:44.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:44.236+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:44.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:44.333+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:44.716 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:44.714+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:44.714868+0000) 2022-04-03T22:24:45.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:45.227+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:45.344 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:45.342+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:45.716 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:45.714+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:45.715063+0000) 2022-04-03T22:24:46.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:46.219+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:46.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:46.309+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:46.716 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:46.715+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:46.715247+0000) 2022-04-03T22:24:47.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:47.183+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:47.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:47.312+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:47.716 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:47.715+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:47.715435+0000) 2022-04-03T22:24:48.214 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:48.211+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:48.327 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:48.325+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:48.456 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:24:48.454+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:24:48.716 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:48.715+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:48.715687+0000) 2022-04-03T22:24:49.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:49.246+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:49.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:49.372+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:49.717 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:49.714+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:49.715848+0000) 2022-04-03T22:24:50.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:50.278+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:50.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:50.350+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:50.717 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:50.715+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:50.716027+0000) 2022-04-03T22:24:51.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:51.305+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:51.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:51.388+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:51.717 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:51.715+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:51.716204+0000) 2022-04-03T22:24:52.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:52.279+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:52.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:52.365+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:52.717 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:52.715+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:52.716373+0000) 2022-04-03T22:24:53.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:53.259+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:53.398 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:53.396+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:53.717 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:53.715+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:53.716573+0000) 2022-04-03T22:24:54.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:54.260+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:54.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:54.392+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:54.717 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:54.715+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:54.716828+0000) 2022-04-03T22:24:55.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:55.246+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:55.441 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:55.438+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:55.718 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:55.716+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:55.717012+0000) 2022-04-03T22:24:56.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:56.217+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:56.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:56.414+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:56.718 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:56.716+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:56.717199+0000) 2022-04-03T22:24:57.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:57.171+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:57.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:57.417+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:57.718 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:57.716+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:57.717404+0000) 2022-04-03T22:24:58.176 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:58.174+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:58.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:58.414+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:58.718 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:58.716+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:58.717591+0000) 2022-04-03T22:24:59.191 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:24:59.189+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:24:59.436 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:24:59.434+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:24:59.498 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:24:59.502 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:24:59.719 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:24:59.717+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:24:59.717874+0000) 2022-04-03T22:25:00.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:00.155+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:00.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:00.386+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:00.719 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:00.717+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:00.718089+0000) 2022-04-03T22:25:01.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:01.196+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:01.395 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:01.393+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:01.719 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:01.717+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:01.718242+0000) 2022-04-03T22:25:02.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:02.148+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:02.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:02.387+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:02.719 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:02.717+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:02.718437+0000) 2022-04-03T22:25:03.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:03.151+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:03.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:03.369+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:03.472 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:25:03.470+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:25:03.720 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:03.717+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:03.718589+0000) 2022-04-03T22:25:04.120 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:04.119+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:04.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:04.414+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:04.720 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:04.718+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:04.718798+0000) 2022-04-03T22:25:05.171 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:05.169+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:05.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:05.434+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:05.720 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:05.718+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:05.718997+0000) 2022-04-03T22:25:06.183 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:06.181+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:06.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:06.388+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:06.720 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:06.718+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:06.719163+0000) 2022-04-03T22:25:07.217 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:07.215+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:07.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:07.391+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:07.720 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:07.718+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:07.719355+0000) 2022-04-03T22:25:08.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:08.237+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:08.402 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:08.401+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:08.720 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:08.718+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:08.719541+0000) 2022-04-03T22:25:09.221 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:09.219+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:09.408 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:09.407+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:09.720 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:09.719+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:09.719733+0000) 2022-04-03T22:25:10.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:10.252+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:10.438 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:10.436+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:10.721 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:10.719+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:10.720055+0000) 2022-04-03T22:25:11.212 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:11.210+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:11.436 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:11.434+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:11.721 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:11.719+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:11.720223+0000) 2022-04-03T22:25:12.233 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:12.232+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:12.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:12.453+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:12.721 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:12.719+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:12.720411+0000) 2022-04-03T22:25:13.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:13.242+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:13.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:13.414+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:13.721 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:13.720+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:13.720628+0000) 2022-04-03T22:25:14.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:14.215+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:14.379 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:14.378+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:14.722 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:14.720+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:14.720890+0000) 2022-04-03T22:25:15.252 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:15.250+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:15.348 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:15.346+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:15.722 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:15.720+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:15.721079+0000) 2022-04-03T22:25:16.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:16.271+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:16.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:16.315+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:16.722 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:16.720+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:16.721293+0000) 2022-04-03T22:25:17.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:17.254+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:17.360 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:17.359+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:17.722 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:17.721+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:17.721485+0000) 2022-04-03T22:25:18.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:18.272+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:18.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:18.340+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:18.488 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:25:18.486+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:25:18.722 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:18.721+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:18.721718+0000) 2022-04-03T22:25:19.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:19.280+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:19.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:19.309+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:19.723 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:19.721+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:19.721930+0000) 2022-04-03T22:25:20.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:20.264+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:20.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:20.350+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:20.723 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:20.721+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:20.722177+0000) 2022-04-03T22:25:21.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:21.241+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:21.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:21.320+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:21.723 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:21.722+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:21.722419+0000) 2022-04-03T22:25:22.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:22.206+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:22.303 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:22.301+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:22.723 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:22.721+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:22.722681+0000) 2022-04-03T22:25:23.202 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:23.200+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:23.262 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:23.260+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:23.724 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:23.722+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:23.722875+0000) 2022-04-03T22:25:24.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:24.225+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:24.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:24.264+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:24.724 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:24.722+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:24.723126+0000) 2022-04-03T22:25:25.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:25.208+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:25.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:25.230+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:25.724 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:25.723+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:25.723375+0000) 2022-04-03T22:25:26.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:26.202+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:26.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:26.207+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:26.724 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:26.723+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:26.723590+0000) 2022-04-03T22:25:27.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:27.188+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:27.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:27.246+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:27.725 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:27.723+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:27.723795+0000) 2022-04-03T22:25:28.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:28.176+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:28.252 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:28.249+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:28.725 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:28.723+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:28.723980+0000) 2022-04-03T22:25:29.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:29.172+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:29.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:29.292+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:29.653 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:25:29.657 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:25:29.725 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:29.723+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:29.724134+0000) 2022-04-03T22:25:30.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:30.172+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:30.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:30.291+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:30.725 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:30.724+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:30.724363+0000) 2022-04-03T22:25:31.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:31.199+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:31.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:31.265+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:31.725 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:31.724+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:31.724579+0000) 2022-04-03T22:25:32.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:32.165+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:32.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:32.254+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:32.726 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:32.724+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:32.724807+0000) 2022-04-03T22:25:33.160 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:33.157+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:33.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:33.227+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:33.504 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:25:33.502+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:25:33.726 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:33.723+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:33.724982+0000) 2022-04-03T22:25:34.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:34.199+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:34.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:34.242+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:34.726 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:34.724+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:34.725189+0000) 2022-04-03T22:25:35.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:35.150+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:35.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:35.251+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:35.726 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:35.724+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:35.725349+0000) 2022-04-03T22:25:36.199 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:36.196+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:36.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:36.265+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:36.726 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:36.723+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:36.725589+0000) 2022-04-03T22:25:37.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:37.163+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:37.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:37.240+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:37.727 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:37.724+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:37.725839+0000) 2022-04-03T22:25:38.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:38.124+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:38.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:38.268+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:38.727 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:38.725+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:38.726060+0000) 2022-04-03T22:25:39.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:39.121+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:39.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:39.225+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:39.727 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:39.725+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:39.726285+0000) 2022-04-03T22:25:40.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:40.119+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:40.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:40.176+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:40.727 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:40.725+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:40.726449+0000) 2022-04-03T22:25:41.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:41.144+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:41.160 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:41.158+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:41.727 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:41.725+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:41.726673+0000) 2022-04-03T22:25:42.108 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:42.105+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:42.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:42.196+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:42.728 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:42.726+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:42.726919+0000) 2022-04-03T22:25:43.145 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:43.143+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:43.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:43.162+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:43.728 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:43.726+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:43.727122+0000) 2022-04-03T22:25:44.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:44.115+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:44.119 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:44.117+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:44.728 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:44.726+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:44.727319+0000) 2022-04-03T22:25:45.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:45.159+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:45.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:45.161+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:45.728 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:45.726+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:45.727548+0000) 2022-04-03T22:25:46.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:46.162+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:46.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:46.187+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:46.729 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:46.726+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:46.727664+0000) 2022-04-03T22:25:47.160 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:47.158+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:47.186 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:47.184+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:47.729 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:47.727+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:47.727875+0000) 2022-04-03T22:25:48.113 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:48.111+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:48.204 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:48.202+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:48.520 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:25:48.518+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:25:48.729 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:48.727+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:48.728057+0000) 2022-04-03T22:25:49.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:49.071+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:49.238 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:49.236+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:49.729 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:49.727+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:49.728208+0000) 2022-04-03T22:25:50.071 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:50.069+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:50.233 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:50.231+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:50.729 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:50.727+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:50.728422+0000) 2022-04-03T22:25:51.042 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:51.040+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:51.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:51.245+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:51.730 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:51.727+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:51.728633+0000) 2022-04-03T22:25:52.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:52.035+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:52.199 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:52.197+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:52.730 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:52.728+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:52.728849+0000) 2022-04-03T22:25:53.027 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:53.024+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:53.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:53.177+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:53.730 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:53.728+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:53.729025+0000) 2022-04-03T22:25:53.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:53.990+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:54.162 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:54.160+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:54.730 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:54.728+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:54.729198+0000) 2022-04-03T22:25:54.969 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:54.967+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:55.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:55.157+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:55.730 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:55.728+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:55.729359+0000) 2022-04-03T22:25:55.929 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:55.926+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:56.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:56.176+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:56.730 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:56.728+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:56.729487+0000) 2022-04-03T22:25:56.881 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:56.879+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:57.228 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:57.226+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:57.731 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:57.729+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:57.729677+0000) 2022-04-03T22:25:57.837 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:57.834+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:58.258 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:58.256+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:58.731 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:58.729+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:58.729812+0000) 2022-04-03T22:25:58.809 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:58.807+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:25:59.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:25:59.243+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:25:59.731 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:25:59.729+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:25:59.729949+0000) 2022-04-03T22:25:59.784 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:25:59.787 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:25:59.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:25:59.793+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:00.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:00.273+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:00.731 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:00.729+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:00.730097+0000) 2022-04-03T22:26:00.775 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:00.773+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:01.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:01.271+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:01.731 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:01.729+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:01.730220+0000) 2022-04-03T22:26:01.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:01.784+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:02.271 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:02.269+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:02.731 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:02.729+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:02.730362+0000) 2022-04-03T22:26:02.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:02.820+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:03.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:03.248+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:03.523 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:26:03.521+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:26:03.731 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:03.730+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:03.730501+0000) 2022-04-03T22:26:03.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:03.801+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:04.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:04.294+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:04.731 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:04.730+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:04.730635+0000) 2022-04-03T22:26:04.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:04.755+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:05.247 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:05.245+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:05.732 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:05.730+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:05.731354+0000) 2022-04-03T22:26:05.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:05.754+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:06.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:06.216+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:06.732 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:06.730+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:06.731554+0000) 2022-04-03T22:26:06.759 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:06.758+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:07.267 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:07.265+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:07.732 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:07.731+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:07.731662+0000) 2022-04-03T22:26:07.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:07.732+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:08.304 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:08.302+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:08.733 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:08.731+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:08.731852+0000) 2022-04-03T22:26:08.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:08.746+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:09.270 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:09.268+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:09.733 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:09.731+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:09.732070+0000) 2022-04-03T22:26:09.764 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:09.763+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:10.221 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:10.220+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:10.733 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:10.732+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:10.732276+0000) 2022-04-03T22:26:10.753 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:10.751+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:11.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:11.230+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:11.733 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:11.732+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:11.732410+0000) 2022-04-03T22:26:11.739 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:11.738+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:12.185 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:12.184+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:12.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:12.700+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:12.733 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:12.731+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:12.732602+0000) 2022-04-03T22:26:13.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:13.164+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:13.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:13.720+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:13.733 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:13.732+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:13.732818+0000) 2022-04-03T22:26:14.152 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:14.150+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:14.734 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:14.732+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:14.732962+0000) 2022-04-03T22:26:14.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:14.769+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:15.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:15.169+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:15.734 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:15.733+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:15.733165+0000) 2022-04-03T22:26:15.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:15.754+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:16.169 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:16.168+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:16.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:16.717+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:16.734 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:16.732+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:16.733336+0000) 2022-04-03T22:26:17.149 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:17.148+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:17.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:17.702+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:17.734 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:17.733+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:17.733496+0000) 2022-04-03T22:26:18.186 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:18.185+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:18.538 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:26:18.535+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:26:18.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:18.698+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:18.734 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:18.732+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:18.733714+0000) 2022-04-03T22:26:19.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:19.182+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:19.734 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:19.732+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:19.733862+0000) 2022-04-03T22:26:19.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:19.747+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:20.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:20.188+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:20.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:20.732+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:20.735 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:20.732+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:20.734047+0000) 2022-04-03T22:26:21.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:21.231+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:21.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:21.708+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:21.735 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:21.733+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:21.734186+0000) 2022-04-03T22:26:22.212 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:22.211+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:22.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:22.710+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:22.735 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:22.733+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:22.734351+0000) 2022-04-03T22:26:23.225 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:23.223+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:23.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:23.665+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:23.735 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:23.733+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:23.734535+0000) 2022-04-03T22:26:24.223 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:24.221+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:24.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:24.634+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:24.735 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:24.733+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:24.734673+0000) 2022-04-03T22:26:25.245 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:25.243+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:25.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:25.629+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:25.735 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:25.733+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:25.734859+0000) 2022-04-03T22:26:26.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:26.204+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:26.648 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:26.645+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:26.736 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:26.734+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:26.734990+0000) 2022-04-03T22:26:27.176 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:27.174+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:27.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:27.667+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:27.736 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:27.734+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:27.735184+0000) 2022-04-03T22:26:28.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:28.162+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:28.710 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:28.707+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:28.736 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:28.734+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:28.735341+0000) 2022-04-03T22:26:29.185 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:29.182+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:29.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:29.666+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:29.736 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:29.734+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:29.735554+0000) 2022-04-03T22:26:29.857 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:26:29.861 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:26:30.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:30.161+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:30.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:30.705+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:30.736 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:30.735+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:30.735783+0000) 2022-04-03T22:26:31.186 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:31.184+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:31.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:31.716+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:31.737 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:31.735+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:31.735924+0000) 2022-04-03T22:26:32.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:32.201+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:32.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:32.696+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:32.737 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:32.735+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:32.736052+0000) 2022-04-03T22:26:33.223 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:33.221+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:33.554 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:26:33.552+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:26:33.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:33.697+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:33.737 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:33.735+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:33.736286+0000) 2022-04-03T22:26:34.242 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:34.240+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:34.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:34.666+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:34.737 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:34.735+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:34.736490+0000) 2022-04-03T22:26:35.205 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:35.203+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:35.706 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:35.704+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:35.737 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:35.735+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:35.736694+0000) 2022-04-03T22:26:36.243 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:36.240+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:36.738 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:36.736+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:36.736875+0000) 2022-04-03T22:26:36.752 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:36.750+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:37.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:37.223+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:37.738 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:37.736+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:37.737107+0000) 2022-04-03T22:26:37.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:37.755+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:38.245 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:38.243+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:38.738 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:38.736+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:38.737291+0000) 2022-04-03T22:26:38.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:38.768+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:39.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:39.229+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:39.735 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:39.733+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:39.738 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:39.736+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:39.737526+0000) 2022-04-03T22:26:40.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:40.201+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:40.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:40.723+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:40.738 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:40.737+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:40.737705+0000) 2022-04-03T22:26:41.208 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:41.206+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:41.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:41.736+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:41.739 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:41.737+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:41.737916+0000) 2022-04-03T22:26:42.253 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:42.251+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:42.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:42.728+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:42.739 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:42.737+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:42.738173+0000) 2022-04-03T22:26:43.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:43.225+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:43.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:43.694+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:43.739 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:43.737+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:43.738406+0000) 2022-04-03T22:26:44.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:44.246+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:44.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:44.651+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:44.739 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:44.738+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:44.738590+0000) 2022-04-03T22:26:45.208 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:45.206+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:45.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:45.613+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:45.740 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:45.738+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:45.738811+0000) 2022-04-03T22:26:46.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:46.165+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:46.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:46.614+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:46.740 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:46.738+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:46.739019+0000) 2022-04-03T22:26:47.129 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:47.128+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:47.595 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:47.593+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:47.740 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:47.738+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:47.739192+0000) 2022-04-03T22:26:48.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:48.095+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:48.570 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:26:48.568+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:26:48.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:48.574+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:48.740 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:48.738+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:48.739390+0000) 2022-04-03T22:26:49.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:49.134+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:49.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:49.526+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:49.740 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:49.739+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:49.739561+0000) 2022-04-03T22:26:50.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:50.154+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:50.544 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:50.542+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:50.740 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:50.739+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:50.739790+0000) 2022-04-03T22:26:51.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:51.119+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:51.564 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:51.562+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:51.741 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:51.739+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:51.739999+0000) 2022-04-03T22:26:52.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:52.086+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:52.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:52.560+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:52.741 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:52.739+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:52.740194+0000) 2022-04-03T22:26:53.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:53.096+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:53.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:53.536+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:53.741 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:53.740+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:53.740378+0000) 2022-04-03T22:26:54.050 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:54.048+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:54.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:54.557+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:54.741 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:54.740+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:54.740579+0000) 2022-04-03T22:26:55.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:55.093+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:55.566 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:55.564+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:55.742 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:55.740+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:55.740749+0000) 2022-04-03T22:26:56.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:56.134+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:56.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:56.596+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:56.742 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:56.740+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:56.740921+0000) 2022-04-03T22:26:57.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:57.176+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:57.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:57.614+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:57.742 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:57.740+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:57.741059+0000) 2022-04-03T22:26:58.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:58.141+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:58.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:58.609+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:58.742 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:58.740+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:58.741211+0000) 2022-04-03T22:26:59.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:26:59.173+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:26:59.608 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:26:59.606+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:26:59.742 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:26:59.741+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:26:59.741417+0000) 2022-04-03T22:27:00.014 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:27:00.019 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:27:00.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:00.171+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:00.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:00.637+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:00.742 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:00.740+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:00.741692+0000) 2022-04-03T22:27:01.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:01.172+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:01.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:01.649+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:01.743 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:01.741+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:01.741933+0000) 2022-04-03T22:27:02.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:02.197+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:02.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:02.626+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:02.743 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:02.742+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:02.742086+0000) 2022-04-03T22:27:03.213 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:03.212+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:03.586 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:27:03.583+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:27:03.607 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:03.606+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:03.743 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:03.741+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:03.742288+0000) 2022-04-03T22:27:04.220 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:04.218+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:04.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:04.633+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:04.743 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:04.741+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:04.742517+0000) 2022-04-03T22:27:05.212 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:05.210+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:05.673 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:05.672+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:05.744 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:05.741+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:05.742724+0000) 2022-04-03T22:27:06.193 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:06.191+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:06.708 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:06.707+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:06.881 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:06.741+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:06.742923+0000) 2022-04-03T22:27:07.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:07.197+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:07.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:07.701+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:07.744 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:07.742+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:07.743166+0000) 2022-04-03T22:27:08.160 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:08.159+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:08.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:08.670+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:08.744 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:08.742+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:08.743442+0000) 2022-04-03T22:27:09.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:09.172+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:09.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:09.648+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:09.745 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:09.742+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:09.743709+0000) 2022-04-03T22:27:10.137 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:10.136+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:10.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:10.630+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:10.745 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:10.742+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:10.743886+0000) 2022-04-03T22:27:11.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:11.138+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:11.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:11.639+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:11.745 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:11.743+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:11.744108+0000) 2022-04-03T22:27:12.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:12.163+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:12.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:12.644+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:12.745 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:12.743+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:12.744353+0000) 2022-04-03T22:27:13.210 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:13.208+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:13.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:13.595+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:13.745 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:13.743+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:13.744572+0000) 2022-04-03T22:27:14.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:14.233+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:14.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:14.559+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:14.745 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:14.743+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:14.744725+0000) 2022-04-03T22:27:15.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:15.214+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:15.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:15.516+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:15.746 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:15.744+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:15.744930+0000) 2022-04-03T22:27:16.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:16.216+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:16.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:16.556+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:16.746 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:16.744+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:16.745161+0000) 2022-04-03T22:27:17.192 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:17.191+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:17.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:17.511+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:17.746 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:17.744+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:17.745388+0000) 2022-04-03T22:27:18.233 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:18.231+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:18.479 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:18.477+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:18.602 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:27:18.600+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:27:18.746 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:18.744+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:18.745595+0000) 2022-04-03T22:27:19.243 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:19.240+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:19.498 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:19.495+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:19.747 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:19.745+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:19.745832+0000) 2022-04-03T22:27:20.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:20.242+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:20.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:20.539+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:20.747 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:20.745+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:20.746035+0000) 2022-04-03T22:27:21.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:21.247+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:21.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:21.575+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:21.747 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:21.745+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:21.746248+0000) 2022-04-03T22:27:22.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:22.216+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:22.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:22.609+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:22.747 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:22.745+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:22.746442+0000) 2022-04-03T22:27:23.202 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:23.200+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:23.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:23.586+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:23.747 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:23.746+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:23.746661+0000) 2022-04-03T22:27:24.223 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:24.221+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:24.582 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:24.579+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:24.748 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:24.746+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:24.746818+0000) 2022-04-03T22:27:25.193 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:25.191+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:25.604 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:25.601+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:25.748 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:25.746+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:25.746978+0000) 2022-04-03T22:27:26.243 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:26.241+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:26.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:26.566+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:26.748 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:26.746+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:26.747182+0000) 2022-04-03T22:27:27.243 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:27.241+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:27.582 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:27.579+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:27.748 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:27.746+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:27.747336+0000) 2022-04-03T22:27:28.262 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:28.260+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:28.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:28.560+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:28.748 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:28.747+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:28.747555+0000) 2022-04-03T22:27:29.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:29.308+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:29.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:29.527+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:29.749 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:29.746+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:29.747710+0000) 2022-04-03T22:27:30.147 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:27:30.151 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:27:30.304 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:30.302+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:30.506 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:30.504+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:30.749 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:30.747+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:30.747933+0000) 2022-04-03T22:27:31.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:31.326+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:31.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:31.540+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:31.749 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:31.747+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:31.748143+0000) 2022-04-03T22:27:32.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:32.348+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:32.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:32.494+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:32.749 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:32.747+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:32.748304+0000) 2022-04-03T22:27:33.355 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:33.352+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:33.526 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:33.524+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:33.618 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:27:33.616+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:27:33.749 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:33.748+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:33.748498+0000) 2022-04-03T22:27:34.308 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:34.305+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:34.564 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:34.562+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:34.749 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:34.748+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:34.748644+0000) 2022-04-03T22:27:35.306 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:35.304+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:35.577 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:35.575+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:35.750 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:35.748+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:35.748861+0000) 2022-04-03T22:27:36.312 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:36.310+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:36.586 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:36.583+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:36.750 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:36.748+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:36.749016+0000) 2022-04-03T22:27:37.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:37.309+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:37.621 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:37.619+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:37.750 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:37.748+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:37.749173+0000) 2022-04-03T22:27:38.356 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:38.353+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:38.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:38.572+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:38.750 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:38.749+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:38.749375+0000) 2022-04-03T22:27:39.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:39.387+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:39.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:39.586+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:39.750 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:39.749+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:39.749573+0000) 2022-04-03T22:27:40.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:40.365+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:40.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:40.629+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:40.750 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:40.749+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:40.749796+0000) 2022-04-03T22:27:41.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:41.333+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:41.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:41.626+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:41.751 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:41.749+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:41.749947+0000) 2022-04-03T22:27:42.338 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:42.336+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:42.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:42.642+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:42.751 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:42.749+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:42.750124+0000) 2022-04-03T22:27:43.294 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:43.292+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:43.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:43.616+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:43.751 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:43.750+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:43.750308+0000) 2022-04-03T22:27:44.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:44.275+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:44.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:44.577+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:44.751 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:44.750+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:44.750526+0000) 2022-04-03T22:27:45.294 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:45.291+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:45.577 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:45.575+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:45.751 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:45.750+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:45.750757+0000) 2022-04-03T22:27:46.318 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:46.316+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:46.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:46.587+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:46.752 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:46.750+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:46.750998+0000) 2022-04-03T22:27:47.343 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:47.341+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:47.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:47.636+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:47.752 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:47.751+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:47.751217+0000) 2022-04-03T22:27:48.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:48.355+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:48.633 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:48.630+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:48.633 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:27:48.630+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:27:48.752 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:48.750+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:48.751426+0000) 2022-04-03T22:27:49.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:49.375+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:49.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:49.666+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:49.752 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:49.750+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:49.751627+0000) 2022-04-03T22:27:50.410 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:50.408+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:50.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:50.639+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:50.752 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:50.750+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:50.751841+0000) 2022-04-03T22:27:51.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:51.437+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:51.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:51.645+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:51.753 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:51.751+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:51.752091+0000) 2022-04-03T22:27:52.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:52.425+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:52.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:52.637+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:52.753 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:52.751+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:52.752305+0000) 2022-04-03T22:27:53.397 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:53.396+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:53.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:53.598+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:53.753 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:53.751+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:53.752516+0000) 2022-04-03T22:27:54.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:54.392+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:54.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:54.580+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:54.753 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:54.750+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:54.752715+0000) 2022-04-03T22:27:55.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:55.442+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:55.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:55.545+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:55.754 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:55.752+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:55.752920+0000) 2022-04-03T22:27:56.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:56.441+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:56.511 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:56.509+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:56.755 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:56.752+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:56.753122+0000) 2022-04-03T22:27:57.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:57.454+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:57.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:57.494+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:57.754 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:57.752+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:57.753322+0000) 2022-04-03T22:27:58.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:58.490+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:58.503 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:58.502+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:58.754 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:58.752+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:58.753533+0000) 2022-04-03T22:27:59.480 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:27:59.478+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:27:59.509 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:27:59.508+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:27:59.754 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:27:59.752+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:27:59.753733+0000) 2022-04-03T22:28:00.302 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:28:00.306 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:28:00.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:00.466+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:00.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:00.475+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:00.755 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:00.753+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:00.753995+0000) 2022-04-03T22:28:01.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:01.418+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:01.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:01.484+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:01.755 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:01.753+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:01.754172+0000) 2022-04-03T22:28:02.437 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:02.435+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:02.487 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:02.486+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:02.755 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:02.753+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:02.754320+0000) 2022-04-03T22:28:03.402 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:03.401+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:03.509 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:03.508+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:03.649 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:28:03.647+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:28:03.755 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:03.753+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:03.754509+0000) 2022-04-03T22:28:04.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:04.405+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:04.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:04.509+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:04.756 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:04.753+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:04.754646+0000) 2022-04-03T22:28:05.432 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:05.431+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:05.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:05.551+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:05.756 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:05.754+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:05.754835+0000) 2022-04-03T22:28:06.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:06.384+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:06.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:06.510+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:06.756 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:06.754+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:06.754971+0000) 2022-04-03T22:28:07.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:07.405+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:07.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:07.530+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:07.756 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:07.754+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:07.755119+0000) 2022-04-03T22:28:08.378 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:08.377+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:08.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:08.499+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:08.756 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:08.754+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:08.755242+0000) 2022-04-03T22:28:09.426 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:09.425+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:09.457 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:09.456+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:09.756 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:09.754+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:09.755370+0000) 2022-04-03T22:28:10.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:10.388+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:10.475 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:10.474+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:10.756 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:10.754+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:10.755577+0000) 2022-04-03T22:28:11.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:11.348+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:11.470 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:11.468+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:11.757 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:11.755+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:11.755828+0000) 2022-04-03T22:28:12.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:12.318+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:12.496 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:12.494+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:12.757 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:12.755+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:12.755991+0000) 2022-04-03T22:28:13.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:13.320+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:13.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:13.510+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:13.757 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:13.755+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:13.756190+0000) 2022-04-03T22:28:14.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:14.310+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:14.504 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:14.502+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:14.757 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:14.755+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:14.756419+0000) 2022-04-03T22:28:15.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:15.264+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:15.554 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:15.551+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:15.757 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:15.755+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:15.756584+0000) 2022-04-03T22:28:16.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:16.260+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:16.550 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:16.548+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:16.757 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:16.756+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:16.756823+0000) 2022-04-03T22:28:17.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:17.267+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:17.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:17.579+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:17.758 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:17.756+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:17.756967+0000) 2022-04-03T22:28:18.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:18.317+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:18.624 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:18.622+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:18.665 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:28:18.663+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:28:18.758 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:18.756+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:18.757151+0000) 2022-04-03T22:28:19.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:19.334+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:19.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:19.609+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:19.758 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:19.756+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:19.757303+0000) 2022-04-03T22:28:20.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:20.378+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:20.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:20.655+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:20.758 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:20.757+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:20.757485+0000) 2022-04-03T22:28:21.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:21.381+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:21.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:21.640+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:21.758 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:21.757+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:21.757691+0000) 2022-04-03T22:28:22.407 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:22.404+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:22.622 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:22.620+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:22.759 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:22.757+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:22.757915+0000) 2022-04-03T22:28:23.454 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:23.452+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:23.619 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:23.617+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:23.759 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:23.757+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:23.758104+0000) 2022-04-03T22:28:24.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:24.497+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:24.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:24.636+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:24.759 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:24.758+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:24.758299+0000) 2022-04-03T22:28:25.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:25.492+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:25.610 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:25.608+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:25.759 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:25.757+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:25.758531+0000) 2022-04-03T22:28:26.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:26.467+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:26.565 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:26.563+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:26.760 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:26.758+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:26.758725+0000) 2022-04-03T22:28:27.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:27.446+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:27.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:27.521+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:27.760 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:27.758+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:27.758899+0000) 2022-04-03T22:28:28.437 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:28.435+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:28.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:28.492+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:28.760 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:28.758+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:28.759094+0000) 2022-04-03T22:28:29.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:29.472+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:29.486 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:29.484+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:29.760 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:29.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:29.759292+0000) 2022-04-03T22:28:30.385 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:28:30.390 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:28:30.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:30.426+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:30.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:30.478+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:30.760 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:30.758+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:30.759505+0000) 2022-04-03T22:28:31.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:31.442+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:31.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:31.524+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:31.761 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:31.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:31.759716+0000) 2022-04-03T22:28:32.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:32.449+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:32.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:32.533+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:32.761 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:32.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:32.759913+0000) 2022-04-03T22:28:33.449 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:33.447+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:33.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:33.563+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:33.681 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:28:33.678+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:28:34.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:33.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:33.760068+0000) 2022-04-03T22:28:34.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:34.490+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:34.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:34.565+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:34.761 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:34.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:34.760263+0000) 2022-04-03T22:28:35.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:35.498+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:35.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:35.565+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:35.761 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:35.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:35.760437+0000) 2022-04-03T22:28:36.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:36.541+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:36.603 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:36.601+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:36.761 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:36.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:36.760625+0000) 2022-04-03T22:28:37.563 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:37.561+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:37.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:37.586+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:37.761 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:37.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:37.760842+0000) 2022-04-03T22:28:38.562 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:38.560+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:38.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:38.563+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:38.762 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:38.760+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:38.760971+0000) 2022-04-03T22:28:39.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:39.601+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:39.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:39.603+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:39.762 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:39.760+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:39.761180+0000) 2022-04-03T22:28:40.570 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:40.568+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:40.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:40.645+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:40.762 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:40.760+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:40.761446+0000) 2022-04-03T22:28:41.610 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:41.608+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:41.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:41.652+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:41.762 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:41.760+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:41.761664+0000) 2022-04-03T22:28:42.578 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:42.577+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:42.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:42.626+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:42.763 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:42.761+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:42.761886+0000) 2022-04-03T22:28:43.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:43.530+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:43.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:43.612+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:43.763 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:43.761+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:43.762105+0000) 2022-04-03T22:28:44.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:44.491+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:44.634 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:44.632+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:44.763 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:44.761+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:44.762319+0000) 2022-04-03T22:28:45.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:45.484+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:45.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:45.588+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:45.763 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:45.761+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:45.762495+0000) 2022-04-03T22:28:46.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:46.499+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:46.573 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:46.571+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:46.763 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:46.761+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:46.762719+0000) 2022-04-03T22:28:47.507 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:47.505+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:47.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:47.612+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:47.764 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:47.762+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:47.762828+0000) 2022-04-03T22:28:48.477 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:48.476+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:48.585 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:48.583+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:48.697 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:28:48.695+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:28:48.764 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:48.762+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:48.762998+0000) 2022-04-03T22:28:49.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:49.465+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:49.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:49.624+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:49.764 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:49.762+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:49.763122+0000) 2022-04-03T22:28:50.465 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:50.463+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:50.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:50.659+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:50.764 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:50.762+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:50.763283+0000) 2022-04-03T22:28:51.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:51.427+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:51.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:51.666+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:51.764 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:51.762+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:51.763478+0000) 2022-04-03T22:28:52.379 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:52.378+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:52.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:52.617+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:52.764 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:52.762+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:52.763572+0000) 2022-04-03T22:28:53.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:53.406+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:53.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:53.573+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:53.765 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:53.763+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:53.763831+0000) 2022-04-03T22:28:54.428 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:54.426+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:54.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:54.561+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:54.765 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:54.763+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:54.764004+0000) 2022-04-03T22:28:55.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:55.414+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:55.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:55.574+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:55.765 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:55.763+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:55.764148+0000) 2022-04-03T22:28:56.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:56.454+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:56.582 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:56.581+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:56.765 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:56.763+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:56.764292+0000) 2022-04-03T22:28:57.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:57.475+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:57.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:57.543+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:57.765 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:57.763+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:57.764449+0000) 2022-04-03T22:28:58.433 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:58.432+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:58.582 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:58.581+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:58.765 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:58.764+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:58.764638+0000) 2022-04-03T22:28:59.471 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:28:59.470+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:28:59.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:28:59.618+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:28:59.765 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:28:59.764+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:28:59.764852+0000) 2022-04-03T22:29:00.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:00.490+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:00.517 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:29:00.521 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:29:00.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:00.666+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:00.766 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:00.764+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:00.765091+0000) 2022-04-03T22:29:01.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:01.484+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:01.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:01.627+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:01.766 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:01.764+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:01.765308+0000) 2022-04-03T22:29:02.465 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:02.464+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:02.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:02.578+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:02.766 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:02.765+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:02.765531+0000) 2022-04-03T22:29:03.468 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:03.466+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:03.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:03.626+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:03.713 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:29:03.711+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:29:03.766 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:03.765+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:03.765750+0000) 2022-04-03T22:29:04.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:04.439+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:04.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:04.675+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:04.767 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:04.765+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:04.765938+0000) 2022-04-03T22:29:05.428 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:05.427+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:05.655 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:05.653+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:05.767 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:05.765+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:05.766087+0000) 2022-04-03T22:29:06.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:06.458+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:06.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:06.610+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:06.767 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:06.765+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:06.766231+0000) 2022-04-03T22:29:07.496 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:07.494+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:07.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:07.656+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:07.767 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:07.766+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:07.766406+0000) 2022-04-03T22:29:08.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:08.477+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:08.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:08.637+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:08.767 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:08.766+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:08.766617+0000) 2022-04-03T22:29:09.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:09.442+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:09.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:09.613+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:09.767 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:09.766+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:09.766820+0000) 2022-04-03T22:29:10.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:10.427+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:10.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:10.639+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:10.768 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:10.766+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:10.766953+0000) 2022-04-03T22:29:11.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:11.461+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:11.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:11.666+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:11.768 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:11.766+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:11.767149+0000) 2022-04-03T22:29:12.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:12.451+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:12.664 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:12.662+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:12.768 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:12.767+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:12.767300+0000) 2022-04-03T22:29:13.412 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:13.410+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:13.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:13.708+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:13.768 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:13.767+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:13.767436+0000) 2022-04-03T22:29:14.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:14.432+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:14.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:14.704+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:14.768 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:14.767+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:14.767573+0000) 2022-04-03T22:29:15.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:15.425+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:15.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:15.694+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:15.769 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:15.767+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:15.767806+0000) 2022-04-03T22:29:16.412 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:16.410+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:16.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:16.683+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:16.769 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:16.767+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:16.767976+0000) 2022-04-03T22:29:17.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:17.452+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:17.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:17.730+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:17.769 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:17.767+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:17.768123+0000) 2022-04-03T22:29:18.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:18.461+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:18.728 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:29:18.726+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:29:18.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:18.755+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:18.769 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:18.767+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:18.768275+0000) 2022-04-03T22:29:19.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:19.491+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:19.744 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:19.742+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:19.769 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:19.767+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:19.768429+0000) 2022-04-03T22:29:20.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:20.443+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:20.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:20.726+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:20.769 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:20.767+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:20.768604+0000) 2022-04-03T22:29:21.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:21.487+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:21.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:21.699+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:21.769 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:21.767+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:21.768826+0000) 2022-04-03T22:29:22.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:22.511+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:22.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:22.706+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:22.770 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:22.768+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:22.769005+0000) 2022-04-03T22:29:23.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:23.511+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:23.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:23.684+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:23.770 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:23.768+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:23.769162+0000) 2022-04-03T22:29:24.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:24.491+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:24.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:24.662+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:24.770 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:24.767+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:24.769389+0000) 2022-04-03T22:29:25.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:25.515+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:25.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:25.637+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:25.770 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:25.768+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:25.769561+0000) 2022-04-03T22:29:26.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:26.551+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:26.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:26.596+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:26.770 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:26.768+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:26.769670+0000) 2022-04-03T22:29:27.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:27.584+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:27.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:27.592+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:27.771 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:27.768+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:27.769814+0000) 2022-04-03T22:29:28.566 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:28.564+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:28.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:28.586+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:28.771 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:28.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:28.769964+0000) 2022-04-03T22:29:29.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:29.587+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:29.633 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:29.630+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:29.771 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:29.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:29.770145+0000) 2022-04-03T22:29:30.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:30.581+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:30.604 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:30.601+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:30.636 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:29:30.640 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:29:30.771 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:30.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:30.770376+0000) 2022-04-03T22:29:31.573 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:31.572+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:31.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:31.636+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:31.771 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:31.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:31.770580+0000) 2022-04-03T22:29:32.530 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:32.528+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:32.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:32.641+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:32.771 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:32.770+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:32.770757+0000) 2022-04-03T22:29:33.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:33.527+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:33.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:33.618+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:33.744 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:29:33.742+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:29:33.772 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:33.770+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:33.770948+0000) 2022-04-03T22:29:34.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:34.577+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:34.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:34.604+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:34.772 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:34.770+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:34.771136+0000) 2022-04-03T22:29:35.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:35.593+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:35.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:35.618+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:35.772 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:35.770+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:35.771341+0000) 2022-04-03T22:29:36.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:36.582+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:36.596 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:36.594+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:36.772 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:36.770+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:36.771462+0000) 2022-04-03T22:29:37.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:37.603+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:37.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:37.624+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:37.772 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:37.771+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:37.771665+0000) 2022-04-03T22:29:38.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:38.612+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:38.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:38.650+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:38.772 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:38.771+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:38.771825+0000) 2022-04-03T22:29:39.568 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:39.566+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:39.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:39.617+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:39.773 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:39.771+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:39.771974+0000) 2022-04-03T22:29:40.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:40.545+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:40.583 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:40.582+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:40.773 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:40.771+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:40.772209+0000) 2022-04-03T22:29:41.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:41.541+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:41.564 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:41.562+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:41.773 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:41.771+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:41.772424+0000) 2022-04-03T22:29:42.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:42.543+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:42.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:42.562+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:42.773 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:42.771+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:42.772592+0000) 2022-04-03T22:29:43.564 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:43.563+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:43.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:43.570+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:43.774 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:43.772+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:43.772798+0000) 2022-04-03T22:29:44.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:44.549+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:44.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:44.575+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:44.774 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:44.772+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:44.772940+0000) 2022-04-03T22:29:45.503 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:45.501+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:45.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:45.619+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:45.774 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:45.772+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:45.773111+0000) 2022-04-03T22:29:46.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:46.480+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:46.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:46.661+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:46.774 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:46.772+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:46.773258+0000) 2022-04-03T22:29:47.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:47.482+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:47.645 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:47.643+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:47.774 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:47.772+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:47.773408+0000) 2022-04-03T22:29:48.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:48.487+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:48.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:48.675+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:48.756 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:29:48.754+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:29:48.774 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:48.772+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:48.773621+0000) 2022-04-03T22:29:49.468 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:49.467+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:49.639 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:49.637+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:49.775 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:49.773+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:49.773809+0000) 2022-04-03T22:29:50.451 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:50.449+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:50.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:50.613+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:50.775 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:50.773+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:50.773982+0000) 2022-04-03T22:29:51.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:51.446+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:51.564 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:51.563+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:51.775 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:51.773+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:51.774143+0000) 2022-04-03T22:29:52.438 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:52.437+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:52.540 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:52.539+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:52.775 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:52.774+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:52.774322+0000) 2022-04-03T22:29:53.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:53.408+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:53.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:53.540+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:53.775 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:53.774+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:53.774446+0000) 2022-04-03T22:29:54.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:54.426+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:54.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:54.545+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:54.775 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:54.774+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:54.774619+0000) 2022-04-03T22:29:55.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:55.388+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:55.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:55.533+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:55.775 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:55.774+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:55.774763+0000) 2022-04-03T22:29:56.434 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:56.433+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:56.508 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:56.507+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:56.776 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:56.774+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:56.774940+0000) 2022-04-03T22:29:57.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:57.391+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:57.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:57.547+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:57.776 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:57.774+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:57.775091+0000) 2022-04-03T22:29:58.396 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:58.395+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:58.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:58.512+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:58.776 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:58.775+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:58.775273+0000) 2022-04-03T22:29:59.413 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:29:59.412+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:29:59.502 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:29:59.501+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:29:59.776 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:29:59.775+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:29:59.775471+0000) 2022-04-03T22:30:00.451 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:00.450+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:00.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:00.458+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:00.776 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:00.774+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:00.775624+0000) 2022-04-03T22:30:00.886 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:30:00.890 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:30:01.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:01.411+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:01.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:01.464+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:01.776 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:01.775+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:01.775855+0000) 2022-04-03T22:30:02.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:02.403+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:02.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:02.424+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:02.777 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:02.774+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:02.776018+0000) 2022-04-03T22:30:03.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:03.383+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:03.464 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:03.462+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:03.772 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:30:03.769+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:30:03.777 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:03.774+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:03.776140+0000) 2022-04-03T22:30:04.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:04.346+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:04.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:04.473+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:04.777 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:04.775+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:04.776263+0000) 2022-04-03T22:30:05.331 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:05.328+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:05.484 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:05.482+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:05.777 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:05.775+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:05.776434+0000) 2022-04-03T22:30:06.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:06.347+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:06.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:06.527+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:06.778 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:06.775+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:06.776604+0000) 2022-04-03T22:30:07.386 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:07.383+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:07.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:07.528+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:07.777 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:07.775+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:07.776716+0000) 2022-04-03T22:30:08.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:08.381+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:08.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:08.572+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:08.778 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:08.775+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:08.776900+0000) 2022-04-03T22:30:09.402 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:09.400+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:09.606 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:09.603+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:09.778 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:09.776+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:09.777074+0000) 2022-04-03T22:30:10.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:10.448+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:10.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:10.621+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:10.778 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:10.776+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:10.777274+0000) 2022-04-03T22:30:11.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:11.465+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:11.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:11.589+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:11.778 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:11.776+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:11.777480+0000) 2022-04-03T22:30:12.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:12.435+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:12.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:12.614+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:12.778 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:12.776+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:12.777671+0000) 2022-04-03T22:30:13.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:13.409+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:13.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:13.654+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:13.779 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:13.777+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:13.777861+0000) 2022-04-03T22:30:14.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:14.452+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:14.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:14.676+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:14.779 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:14.777+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:14.778070+0000) 2022-04-03T22:30:15.457 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:15.454+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:15.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:15.705+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:15.779 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:15.777+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:15.778280+0000) 2022-04-03T22:30:16.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:16.457+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:16.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:16.677+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:16.779 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:16.777+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:16.778499+0000) 2022-04-03T22:30:17.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:17.425+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:17.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:17.639+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:17.779 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:17.777+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:17.778711+0000) 2022-04-03T22:30:18.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:18.456+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:18.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:18.629+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:18.780 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:18.778+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:18.778928+0000) 2022-04-03T22:30:18.787 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:30:18.786+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:30:19.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:19.499+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:19.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:19.670+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:19.780 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:19.778+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:19.779123+0000) 2022-04-03T22:30:20.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:20.540+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:20.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:20.639+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:20.780 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:20.778+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:20.779324+0000) 2022-04-03T22:30:21.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:21.586+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:21.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:21.648+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:21.780 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:21.778+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:21.779536+0000) 2022-04-03T22:30:22.606 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:22.604+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:22.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:22.623+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:22.781 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:22.779+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:22.779744+0000) 2022-04-03T22:30:23.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:23.592+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:23.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:23.618+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:23.781 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:23.779+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:23.779925+0000) 2022-04-03T22:30:24.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:24.622+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:24.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:24.638+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:24.781 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:24.779+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:24.780136+0000) 2022-04-03T22:30:25.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:25.635+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:25.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:25.654+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:25.781 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:25.779+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:25.780360+0000) 2022-04-03T22:30:26.603 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:26.601+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:26.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:26.627+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:26.781 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:26.780+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:26.780564+0000) 2022-04-03T22:30:27.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:27.595+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:27.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:27.667+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:27.781 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:27.780+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:27.780744+0000) 2022-04-03T22:30:28.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:28.574+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:28.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:28.693+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:28.782 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:28.780+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:28.780907+0000) 2022-04-03T22:30:29.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:29.611+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:29.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:29.674+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:29.782 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:29.780+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:29.781104+0000) 2022-04-03T22:30:30.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:30.596+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:30.685 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:30.683+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:30.782 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:30.780+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:30.781323+0000) 2022-04-03T22:30:30.966 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:30:30.970 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:30:31.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:31.642+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:31.659 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:31.657+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:31.782 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:31.780+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:31.781559+0000) 2022-04-03T22:30:32.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:32.668+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:32.674 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:32.673+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:32.782 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:32.781+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:32.781727+0000) 2022-04-03T22:30:33.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:33.668+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:33.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:33.688+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:33.783 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:33.781+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:33.781917+0000) 2022-04-03T22:30:33.803 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:30:33.801+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:30:34.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:34.656+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:34.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:34.682+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:34.783 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:34.781+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:34.782064+0000) 2022-04-03T22:30:35.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:35.626+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:35.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:35.731+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:35.783 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:35.781+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:35.782232+0000) 2022-04-03T22:30:36.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:36.619+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:36.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:36.696+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:36.783 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:36.781+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:36.782457+0000) 2022-04-03T22:30:37.630 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:37.628+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:37.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:37.693+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:37.783 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:37.781+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:37.782674+0000) 2022-04-03T22:30:38.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:38.623+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:38.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:38.726+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:38.783 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:38.782+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:38.782898+0000) 2022-04-03T22:30:39.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:39.655+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:39.705 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:39.704+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:39.784 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:39.782+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:39.783112+0000) 2022-04-03T22:30:40.650 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:40.649+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:40.746 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:40.745+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:40.784 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:40.783+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:40.783324+0000) 2022-04-03T22:30:41.675 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:41.673+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:41.781 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:41.779+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:41.784 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:41.783+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:41.783563+0000) 2022-04-03T22:30:42.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:42.691+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:42.785 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:42.783+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:42.783833+0000) 2022-04-03T22:30:42.808 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:42.807+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:43.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:43.655+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:43.785 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:43.783+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:43.784037+0000) 2022-04-03T22:30:43.827 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:43.825+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:44.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:44.667+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:44.785 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:44.784+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:44.784235+0000) 2022-04-03T22:30:44.821 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:44.820+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:45.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:45.668+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:45.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:45.776+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:45.785 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:45.784+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:45.784435+0000) 2022-04-03T22:30:46.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:46.712+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:46.785 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:46.784+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:46.784636+0000) 2022-04-03T22:30:46.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:46.800+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:47.757 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:47.755+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:47.785 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:47.783+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:47.784815+0000) 2022-04-03T22:30:47.805 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:47.804+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:48.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:48.778+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:48.785 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:48.783+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:48.785020+0000) 2022-04-03T22:30:48.819 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:30:48.816+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:30:48.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:48.833+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:49.786 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:49.784+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:49.785191+0000) 2022-04-03T22:30:49.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:49.812+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:49.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:49.820+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:50.786 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:50.784+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:50.785347+0000) 2022-04-03T22:30:50.837 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:50.836+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:50.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:50.864+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:51.786 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:51.784+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:51.785520+0000) 2022-04-03T22:30:51.876 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:51.875+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:51.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:51.899+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:52.786 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:52.784+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:52.785759+0000) 2022-04-03T22:30:52.853 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:52.852+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:52.854 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:52.853+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:53.787 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:53.785+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:53.785961+0000) 2022-04-03T22:30:53.809 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:53.808+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:53.894 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:53.893+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:54.787 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:54.785+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:54.786173+0000) 2022-04-03T22:30:54.837 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:54.836+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:54.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:54.938+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:55.787 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:55.785+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:55.786399+0000) 2022-04-03T22:30:55.840 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:55.838+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:55.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:55.978+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:56.788 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:56.785+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:56.786635+0000) 2022-04-03T22:30:56.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:56.868+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:56.944 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:56.943+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:57.788 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:57.786+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:57.786858+0000) 2022-04-03T22:30:57.899 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:57.897+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:57.910 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:57.908+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:58.788 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:58.786+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:58.787073+0000) 2022-04-03T22:30:58.902 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:58.900+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:58.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:58.931+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:30:59.788 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:30:59.786+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:30:59.787280+0000) 2022-04-03T22:30:59.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:30:59.894+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:30:59.935 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:30:59.933+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:00.788 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:00.786+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:00.787469+0000) 2022-04-03T22:31:00.911 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:00.909+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:00.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:00.921+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:01.023 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:31:01.026 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:31:01.788 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:01.786+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:01.787731+0000) 2022-04-03T22:31:01.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:01.936+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:01.950 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:01.948+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:02.789 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:02.787+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:02.787961+0000) 2022-04-03T22:31:02.901 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:02.899+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:02.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:02.940+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:03.789 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:03.787+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:03.788181+0000) 2022-04-03T22:31:03.835 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:31:03.833+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:31:03.934 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:03.932+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:03.946 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:03.944+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:04.789 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:04.787+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:04.788383+0000) 2022-04-03T22:31:04.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:04.902+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:04.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:04.989+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:05.789 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:05.787+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:05.788590+0000) 2022-04-03T22:31:05.914 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:05.912+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:05.994 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:05.992+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:06.790 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:06.788+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:06.788817+0000) 2022-04-03T22:31:06.916 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:06.914+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:06.964 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:06.962+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:07.790 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:07.788+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:07.789013+0000) 2022-04-03T22:31:07.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:07.940+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:07.958 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:07.956+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:08.790 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:08.788+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:08.789217+0000) 2022-04-03T22:31:08.955 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:08.952+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:08.980 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:08.978+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:09.790 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:09.788+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:09.789401+0000) 2022-04-03T22:31:09.977 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:09.975+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:10.023 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:10.021+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:10.790 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:10.788+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:10.789629+0000) 2022-04-03T22:31:11.022 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:11.020+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:11.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:11.031+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:11.791 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:11.789+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:11.789868+0000) 2022-04-03T22:31:12.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:12.047+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:12.071 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:12.069+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:12.792 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:12.789+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:12.790070+0000) 2022-04-03T22:31:13.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:13.010+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:13.050 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:13.048+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:13.791 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:13.789+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:13.790275+0000) 2022-04-03T22:31:14.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:14.011+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:14.031 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:14.029+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:14.791 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:14.790+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:14.790475+0000) 2022-04-03T22:31:14.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:14.987+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:15.043 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:15.041+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:15.791 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:15.790+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:15.790656+0000) 2022-04-03T22:31:15.962 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:15.960+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:16.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:16.065+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:16.791 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:16.790+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:16.790868+0000) 2022-04-03T22:31:16.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:16.936+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:17.054 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:17.052+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:17.792 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:17.790+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:17.790988+0000) 2022-04-03T22:31:17.916 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:17.914+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:18.062 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:18.059+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:18.792 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:18.790+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:18.791111+0000) 2022-04-03T22:31:18.851 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:31:18.849+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:31:18.874 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:18.871+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:19.098 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:19.096+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:19.792 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:19.790+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:19.791346+0000) 2022-04-03T22:31:19.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:19.845+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:20.141 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:20.138+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:20.793 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:20.791+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:20.791561+0000) 2022-04-03T22:31:20.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:20.833+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:21.172 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:21.170+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:21.792 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:21.791+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:21.791745+0000) 2022-04-03T22:31:21.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:21.822+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:22.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:22.188+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:22.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:22.781+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:22.792 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:22.791+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:22.791970+0000) 2022-04-03T22:31:23.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:23.213+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:23.793 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:23.791+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:23.792166+0000) 2022-04-03T22:31:23.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:23.818+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:24.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:24.164+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:24.793 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:24.792+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:24.792309+0000) 2022-04-03T22:31:24.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:24.816+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:25.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:25.191+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:25.793 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:25.792+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:25.792462+0000) 2022-04-03T22:31:25.817 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:25.815+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:26.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:26.159+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:26.794 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:26.792+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:26.792658+0000) 2022-04-03T22:31:26.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:26.837+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:27.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:27.174+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:27.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:27.789+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:27.793 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:27.792+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:27.792837+0000) 2022-04-03T22:31:28.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:28.184+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:28.796 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:28.792+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:28.793023+0000) 2022-04-03T22:31:28.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:28.837+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:29.139 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:29.137+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:29.794 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:29.793+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:29.793195+0000) 2022-04-03T22:31:29.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:29.843+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:30.090 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:30.088+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:30.794 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:30.793+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:30.793376+0000) 2022-04-03T22:31:30.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:30.805+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:31.116 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:31.114+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:31.162 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:31:31.170 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:31:31.794 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:31.793+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:31.793608+0000) 2022-04-03T22:31:31.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:31.825+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:32.076 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:32.074+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:32.795 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:32.792+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:32.796 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:32.792+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:32.793874+0000) 2022-04-03T22:31:33.032 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:33.030+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:33.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:33.760+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:33.795 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:33.793+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:33.794076+0000) 2022-04-03T22:31:33.867 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:31:33.864+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:31:33.990 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:33.988+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:34.768 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:34.765+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:34.795 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:34.793+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:34.794193+0000) 2022-04-03T22:31:35.034 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:35.033+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:35.795 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:35.793+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:35.794366+0000) 2022-04-03T22:31:35.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:35.806+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:36.053 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:36.049+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:36.795 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:36.793+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:36.794522+0000) 2022-04-03T22:31:36.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:36.826+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:37.021 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:37.019+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:37.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:37.782+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:37.795 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:37.793+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:37.794806+0000) 2022-04-03T22:31:38.064 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:38.063+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:38.796 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:38.793+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:38.794970+0000) 2022-04-03T22:31:38.828 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:38.827+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:39.110 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:39.109+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:39.796 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:39.793+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:39.795117+0000) 2022-04-03T22:31:39.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:39.845+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:40.123 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:40.122+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:40.796 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:40.794+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:40.795291+0000) 2022-04-03T22:31:40.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:40.799+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:41.140 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:41.139+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:41.796 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:41.794+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:41.795482+0000) 2022-04-03T22:31:41.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:41.826+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:42.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:42.180+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:42.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:42.792+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:42.796 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:42.794+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:42.795712+0000) 2022-04-03T22:31:43.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:43.210+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:43.751 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:43.749+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:43.797 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:43.795+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:43.795919+0000) 2022-04-03T22:31:44.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:44.232+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:44.774 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:44.772+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:44.797 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:44.795+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:44.796126+0000) 2022-04-03T22:31:45.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:45.261+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:45.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:45.727+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:45.797 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:45.795+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:45.796302+0000) 2022-04-03T22:31:46.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:46.258+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:46.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:46.727+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:46.797 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:46.795+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:46.796517+0000) 2022-04-03T22:31:47.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:47.240+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:47.777 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:47.776+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:47.797 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:47.795+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:47.796673+0000) 2022-04-03T22:31:48.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:48.257+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:48.757 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:48.756+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:48.798 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:48.796+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:48.796887+0000) 2022-04-03T22:31:48.883 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:31:48.880+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:31:49.212 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:49.210+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:49.730 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:49.728+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:49.798 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:49.796+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:49.797063+0000) 2022-04-03T22:31:50.220 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:50.219+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:50.766 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:50.764+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:50.798 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:50.796+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:50.797312+0000) 2022-04-03T22:31:51.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:51.214+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:51.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:51.779+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:51.798 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:51.796+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:51.797518+0000) 2022-04-03T22:31:52.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:52.218+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:52.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:52.740+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:52.799 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:52.797+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:52.797744+0000) 2022-04-03T22:31:53.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:53.209+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:53.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:53.715+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:53.799 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:53.797+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:53.797983+0000) 2022-04-03T22:31:54.206 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:54.204+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:54.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:54.751+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:54.799 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:54.797+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:54.798179+0000) 2022-04-03T22:31:55.217 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:55.215+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:55.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:55.724+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:55.799 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:55.797+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:55.798365+0000) 2022-04-03T22:31:56.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:56.227+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:56.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:56.747+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:56.799 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:56.798+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:56.798571+0000) 2022-04-03T22:31:57.182 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:57.180+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:57.798 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:57.795+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:57.799 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:57.798+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:57.798806+0000) 2022-04-03T22:31:58.135 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:58.133+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:58.800 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:58.798+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:58.799001+0000) 2022-04-03T22:31:58.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:58.845+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:31:59.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:31:59.179+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:31:59.800 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:31:59.798+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:31:59.799150+0000) 2022-04-03T22:31:59.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:31:59.800+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:00.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:00.228+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:00.800 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:00.798+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:00.799330+0000) 2022-04-03T22:32:00.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:00.845+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:01.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:01.196+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:01.318 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:32:01.321 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:32:01.800 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:01.799+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:01.799545+0000) 2022-04-03T22:32:01.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:01.833+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:02.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:02.161+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:02.801 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:02.799+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:02.799787+0000) 2022-04-03T22:32:02.834 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:02.832+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:03.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:03.198+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:03.792 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:03.790+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:03.800 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:03.799+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:03.799958+0000) 2022-04-03T22:32:03.898 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:32:03.896+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:32:04.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:04.149+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:04.771 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:04.768+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:04.801 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:04.799+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:04.800170+0000) 2022-04-03T22:32:05.136 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:05.134+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:05.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:05.724+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:05.801 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:05.800+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:05.800387+0000) 2022-04-03T22:32:06.134 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:06.132+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:06.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:06.756+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:06.801 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:06.800+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:06.800610+0000) 2022-04-03T22:32:07.133 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:07.131+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:07.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:07.776+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:07.802 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:07.800+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:07.800839+0000) 2022-04-03T22:32:08.117 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:08.115+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:08.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:08.767+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:08.802 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:08.800+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:08.801059+0000) 2022-04-03T22:32:09.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:09.164+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:09.751 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:09.748+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:09.802 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:09.801+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:09.801285+0000) 2022-04-03T22:32:10.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:10.147+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:10.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:10.777+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:10.802 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:10.801+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:10.801472+0000) 2022-04-03T22:32:11.158 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:11.155+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:11.738 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:11.735+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:11.802 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:11.801+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:11.801658+0000) 2022-04-03T22:32:12.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:12.159+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:12.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:12.778+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:12.803 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:12.801+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:12.801867+0000) 2022-04-03T22:32:13.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:13.161+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:13.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:13.739+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:13.803 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:13.801+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:13.802060+0000) 2022-04-03T22:32:14.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:14.154+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:14.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:14.712+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:14.803 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:14.802+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:14.802278+0000) 2022-04-03T22:32:15.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:15.197+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:15.707 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:15.705+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:15.803 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:15.802+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:15.802422+0000) 2022-04-03T22:32:16.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:16.221+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:16.678 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:16.676+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:16.803 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:16.801+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:16.802632+0000) 2022-04-03T22:32:17.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:17.221+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:17.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:17.660+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:17.804 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:17.801+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:17.802845+0000) 2022-04-03T22:32:18.226 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:18.224+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:18.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:18.630+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:18.804 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:18.802+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:18.803059+0000) 2022-04-03T22:32:18.914 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:32:18.911+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:32:19.214 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:19.212+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:19.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:19.593+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:19.804 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:19.802+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:19.803253+0000) 2022-04-03T22:32:20.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:20.164+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:20.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:20.582+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:20.804 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:20.802+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:20.803481+0000) 2022-04-03T22:32:21.174 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:21.171+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:21.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:21.579+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:21.804 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:21.802+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:21.803700+0000) 2022-04-03T22:32:22.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:22.148+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:22.626 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:22.624+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:22.805 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:22.803+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:22.803908+0000) 2022-04-03T22:32:22.805 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:22.803+0000 7fd623a1a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T22:32:22.806 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:22.803+0000 7fd623a1a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T22:32:22.810 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:22.808+0000 7fd62421b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:22.809963+0000) 2022-04-03T22:32:23.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:23.165+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:23.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:23.581+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:23.805 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:23.803+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:23.804052+0000) 2022-04-03T22:32:24.124 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:24.121+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:24.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:24.570+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:24.805 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:24.803+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:24.804216+0000) 2022-04-03T22:32:25.117 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:25.115+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:25.563 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:25.561+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:32:25.805 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:25.803+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:25.804354+0000) 2022-04-03T22:32:26.089 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:26.087+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:26.525 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:26.524+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:26.805 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:26.803+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:26.804522+0000) 2022-04-03T22:32:27.065 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:27.063+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:27.555 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:27.554+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:27.805 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:27.803+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:27.804703+0000) 2022-04-03T22:32:28.099 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:28.097+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:28.599 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:28.597+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:28.806 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:28.804+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:28.804935+0000) 2022-04-03T22:32:29.106 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:29.104+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:29.591 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:29.589+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:29.806 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:29.804+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:29.805122+0000) 2022-04-03T22:32:30.098 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:30.096+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:30.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:30.575+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:30.806 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:30.804+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:30.805341+0000) 2022-04-03T22:32:31.094 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:31.092+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:31.474 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:32:31.478 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:32:31.562 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:31.560+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:31.806 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:31.804+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:31.805537+0000) 2022-04-03T22:32:32.125 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:32.124+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:32.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:32.583+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:32.807 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:32.805+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:32.805747+0000) 2022-04-03T22:32:33.113 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:33.111+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:33.589 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:33.588+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:33.807 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:33.805+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:33.805986+0000) 2022-04-03T22:32:33.929 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:32:33.927+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:32:34.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:34.162+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:34.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:34.565+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:34.807 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:34.805+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:34.806202+0000) 2022-04-03T22:32:35.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:35.206+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:35.575 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:35.573+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:35.807 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:35.805+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:35.806405+0000) 2022-04-03T22:32:36.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:36.250+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:36.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:36.527+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:36.807 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:36.806+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:36.806604+0000) 2022-04-03T22:32:37.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:37.262+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:37.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:37.509+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:37.808 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:37.806+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:37.806813+0000) 2022-04-03T22:32:38.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:38.301+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:38.554 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:38.553+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:38.808 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:38.806+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:38.806974+0000) 2022-04-03T22:32:39.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:39.254+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:39.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:39.550+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:39.808 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:39.806+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:39.807152+0000) 2022-04-03T22:32:40.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:40.266+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:40.520 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:40.519+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:40.808 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:40.806+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:40.807292+0000) 2022-04-03T22:32:41.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:41.227+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:41.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:41.487+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:41.808 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:41.806+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:41.807427+0000) 2022-04-03T22:32:42.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:42.229+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:42.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:42.443+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:42.808 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:42.807+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:42.807598+0000) 2022-04-03T22:32:43.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:43.265+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:43.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:43.478+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:43.808 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:43.807+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:43.807785+0000) 2022-04-03T22:32:44.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:44.227+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:44.470 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:44.468+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:44.809 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:44.807+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:44.807945+0000) 2022-04-03T22:32:45.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:45.185+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:45.471 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:45.470+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:45.809 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:45.807+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:45.808121+0000) 2022-04-03T22:32:46.205 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:46.204+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:46.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:46.500+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:46.809 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:46.807+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:46.808302+0000) 2022-04-03T22:32:47.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:47.192+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:47.465 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:47.464+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:47.809 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:47.808+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:47.808465+0000) 2022-04-03T22:32:48.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:48.177+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:48.500 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:48.498+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:48.809 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:48.808+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:48.808687+0000) 2022-04-03T22:32:48.945 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:32:48.943+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:32:49.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:49.201+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:49.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:49.449+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:49.810 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:49.808+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:49.808901+0000) 2022-04-03T22:32:50.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:50.241+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:50.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:50.452+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:50.810 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:50.808+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:50.809101+0000) 2022-04-03T22:32:51.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:51.259+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:51.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:51.500+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:51.810 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:51.808+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:51.809219+0000) 2022-04-03T22:32:52.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:52.267+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:52.517 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:52.515+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:52.810 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:52.809+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:52.809371+0000) 2022-04-03T22:32:53.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:53.266+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:53.502 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:53.500+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:53.810 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:53.809+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:53.809507+0000) 2022-04-03T22:32:54.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:54.287+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:54.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:54.534+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:54.810 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:54.809+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:54.809668+0000) 2022-04-03T22:32:55.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:55.286+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:55.524 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:55.522+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:55.811 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:55.809+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:55.809855+0000) 2022-04-03T22:32:56.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:56.324+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:56.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:56.570+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:56.811 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:56.809+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:56.810037+0000) 2022-04-03T22:32:57.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:57.286+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:57.574 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:57.572+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:57.811 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:57.810+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:57.810263+0000) 2022-04-03T22:32:58.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:58.298+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:58.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:58.619+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:58.811 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:58.810+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:58.810469+0000) 2022-04-03T22:32:59.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:32:59.271+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:32:59.653 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:32:59.650+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:32:59.811 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:32:59.810+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:32:59.810644+0000) 2022-04-03T22:33:00.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:00.303+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:00.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:00.685+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:00.811 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:00.810+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:00.810843+0000) 2022-04-03T22:33:01.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:01.322+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:01.613 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:33:01.617 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:33:01.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:01.677+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:01.812 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:01.809+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:01.811035+0000) 2022-04-03T22:33:02.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:02.308+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:02.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:02.726+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:02.812 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:02.810+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:02.811226+0000) 2022-04-03T22:33:03.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:03.353+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:03.749 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:03.747+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:03.812 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:03.810+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:03.811393+0000) 2022-04-03T22:33:03.961 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:33:03.958+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:33:04.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:04.359+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:04.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:04.698+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:04.812 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:04.810+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:04.811555+0000) 2022-04-03T22:33:05.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:05.363+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:05.708 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:05.706+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:05.812 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:05.809+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:05.811694+0000) 2022-04-03T22:33:06.409 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:06.407+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:06.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:06.696+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:06.813 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:06.810+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:06.811864+0000) 2022-04-03T22:33:07.396 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:07.393+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:07.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:07.675+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:07.813 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:07.811+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:07.812042+0000) 2022-04-03T22:33:08.388 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:08.386+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:08.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:08.655+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:08.813 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:08.811+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:08.812228+0000) 2022-04-03T22:33:09.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:09.369+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:09.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:09.642+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:09.813 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:09.811+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:09.812408+0000) 2022-04-03T22:33:10.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:10.401+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:10.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:10.624+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:10.813 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:10.811+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:10.812589+0000) 2022-04-03T22:33:11.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:11.367+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:11.674 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:11.672+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:11.814 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:11.811+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:11.812745+0000) 2022-04-03T22:33:12.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:12.345+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:12.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:12.698+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:12.814 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:12.812+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:12.812957+0000) 2022-04-03T22:33:13.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:13.382+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:13.736 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:13.734+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:13.814 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:13.812+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:13.813080+0000) 2022-04-03T22:33:14.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:14.350+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:14.706 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:14.704+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:14.814 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:14.812+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:14.813227+0000) 2022-04-03T22:33:15.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:15.383+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:15.747 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:15.745+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:15.814 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:15.812+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:15.813392+0000) 2022-04-03T22:33:16.409 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:16.406+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:16.795 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:16.792+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:16.814 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:16.812+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:16.813572+0000) 2022-04-03T22:33:17.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:17.371+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:17.814 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:17.813+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:17.813751+0000) 2022-04-03T22:33:17.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:17.834+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:18.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:18.393+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:18.815 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:18.813+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:18.813975+0000) 2022-04-03T22:33:18.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:18.858+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:18.977 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:33:18.975+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:33:19.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:19.368+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:19.815 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:19.813+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:19.814138+0000) 2022-04-03T22:33:19.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:19.886+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:20.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:20.319+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:20.815 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:20.813+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:20.814302+0000) 2022-04-03T22:33:20.905 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:20.903+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:21.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:21.336+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:21.815 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:21.813+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:21.814417+0000) 2022-04-03T22:33:21.943 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:21.940+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:22.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:22.363+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:22.815 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:22.813+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:22.814563+0000) 2022-04-03T22:33:22.925 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:22.923+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:23.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:23.403+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:23.815 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:23.814+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:23.814690+0000) 2022-04-03T22:33:23.960 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:23.958+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:24.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:24.380+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:24.815 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:24.814+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:24.814805+0000) 2022-04-03T22:33:24.941 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:24.939+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:25.428 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:25.426+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:25.816 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:25.814+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:25.814942+0000) 2022-04-03T22:33:25.946 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:25.945+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:26.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:26.426+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:26.816 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:26.814+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:26.815117+0000) 2022-04-03T22:33:26.994 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:26.992+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:27.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:27.391+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:27.816 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:27.814+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:27.815313+0000) 2022-04-03T22:33:28.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:28.007+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:28.364 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:28.362+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:28.816 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:28.815+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:28.815529+0000) 2022-04-03T22:33:28.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:28.979+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:29.412 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:29.411+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:29.816 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:29.815+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:29.815681+0000) 2022-04-03T22:33:29.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:29.963+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:30.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:30.460+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:30.816 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:30.815+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:30.815843+0000) 2022-04-03T22:33:30.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:30.945+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:31.466 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:31.465+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:31.753 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:33:31.757 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:33:31.817 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:31.815+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:31.815964+0000) 2022-04-03T22:33:31.916 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:31.915+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:32.442 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:32.440+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:32.817 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:32.815+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:32.816178+0000) 2022-04-03T22:33:32.960 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:32.959+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:33.405 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:33.404+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:33.817 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:33.816+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:33.816297+0000) 2022-04-03T22:33:33.968 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:33.967+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:33.993 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:33:33.991+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:33:34.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:34.385+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:34.817 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:34.816+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:34.816474+0000) 2022-04-03T22:33:34.936 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:34.935+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:35.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:35.346+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:35.817 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:35.816+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:35.816636+0000) 2022-04-03T22:33:35.968 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:35.967+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:36.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:36.325+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:36.818 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:36.816+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:36.816816+0000) 2022-04-03T22:33:37.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:37.006+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:37.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:37.338+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:37.818 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:37.816+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:37.816993+0000) 2022-04-03T22:33:37.971 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:37.970+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:38.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:38.311+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:38.818 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:38.816+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:38.817167+0000) 2022-04-03T22:33:39.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:39.015+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:39.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:39.277+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:39.818 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:39.817+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:39.817351+0000) 2022-04-03T22:33:39.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:39.998+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:40.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:40.287+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:40.818 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:40.817+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:40.817551+0000) 2022-04-03T22:33:40.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:40.956+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:41.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:41.321+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:41.819 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:41.817+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:41.817684+0000) 2022-04-03T22:33:41.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:41.965+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:42.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:42.304+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:42.819 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:42.817+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:42.817837+0000) 2022-04-03T22:33:42.944 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:42.943+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:43.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:43.305+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:43.819 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:43.817+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:43.818017+0000) 2022-04-03T22:33:43.916 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:43.913+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:44.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:44.265+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:44.819 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:44.818+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:44.818210+0000) 2022-04-03T22:33:44.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:44.925+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:45.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:45.247+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:45.819 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:45.818+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:45.818371+0000) 2022-04-03T22:33:45.975 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:45.973+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:46.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:46.297+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:46.819 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:46.817+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:46.818556+0000) 2022-04-03T22:33:46.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:46.987+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:47.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:47.336+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:47.819 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:47.817+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:47.818732+0000) 2022-04-03T22:33:47.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:47.972+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:48.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:48.329+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:48.819 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:48.817+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:48.818896+0000) 2022-04-03T22:33:48.945 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:48.943+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:48.995 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:33:48.992+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:33:49.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:49.301+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:49.820 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:49.817+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:49.819037+0000) 2022-04-03T22:33:49.899 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:49.897+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:50.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:50.254+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:50.820 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:50.818+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:50.819248+0000) 2022-04-03T22:33:50.941 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:50.939+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:51.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:51.273+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:51.820 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:51.818+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:51.819386+0000) 2022-04-03T22:33:51.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:51.972+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:52.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:52.280+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:52.820 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:52.818+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:52.819571+0000) 2022-04-03T22:33:52.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:52.948+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:53.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:53.309+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:53.820 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:53.818+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:53.819700+0000) 2022-04-03T22:33:53.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:53.981+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:54.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:54.331+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:54.821 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:54.818+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:54.819847+0000) 2022-04-03T22:33:55.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:55.006+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:55.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:55.344+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:55.821 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:55.819+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:55.820003+0000) 2022-04-03T22:33:56.050 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:56.048+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:56.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:56.358+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:56.821 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:56.819+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:56.820175+0000) 2022-04-03T22:33:57.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:57.061+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:57.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:57.339+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:57.821 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:57.819+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:57.820370+0000) 2022-04-03T22:33:58.094 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:58.092+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:58.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:58.294+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:58.821 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:58.819+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:58.820487+0000) 2022-04-03T22:33:59.113 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:33:59.110+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:33:59.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:33:59.259+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:33:59.821 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:33:59.819+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:33:59.820630+0000) 2022-04-03T22:34:00.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:00.124+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:00.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:00.285+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:00.822 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:00.820+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:00.820832+0000) 2022-04-03T22:34:01.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:01.110+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:01.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:01.302+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:01.822 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:01.820+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:01.820984+0000) 2022-04-03T22:34:01.874 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:34:01.878 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:34:02.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:02.151+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:02.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:02.279+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:02.822 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:02.820+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:02.821146+0000) 2022-04-03T22:34:03.140 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:03.138+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:03.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:03.310+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:03.822 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:03.820+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:03.821330+0000) 2022-04-03T22:34:04.011 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:34:04.009+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:34:04.172 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:04.170+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:04.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:04.360+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:04.822 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:04.820+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:04.821472+0000) 2022-04-03T22:34:05.221 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:05.220+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:05.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:05.365+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:05.823 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:05.820+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:05.821657+0000) 2022-04-03T22:34:06.214 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:06.212+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:06.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:06.382+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:06.823 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:06.821+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:06.821839+0000) 2022-04-03T22:34:07.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:07.230+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:07.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:07.337+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:07.823 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:07.821+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:07.822012+0000) 2022-04-03T22:34:08.192 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:08.190+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:08.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:08.369+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:08.823 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:08.821+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:08.822193+0000) 2022-04-03T22:34:09.157 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:09.155+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:09.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:09.384+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:09.823 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:09.821+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:09.822387+0000) 2022-04-03T22:34:10.114 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:10.112+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:10.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:10.345+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:10.823 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:10.822+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:10.822586+0000) 2022-04-03T22:34:11.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:11.124+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:11.396 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:11.394+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:11.824 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:11.822+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:11.822812+0000) 2022-04-03T22:34:12.108 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:12.107+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:12.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:12.346+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:12.824 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:12.822+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:12.823000+0000) 2022-04-03T22:34:13.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:13.119+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:13.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:13.367+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:13.824 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:13.822+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:13.823152+0000) 2022-04-03T22:34:14.150 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:14.148+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:14.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:14.389+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:14.824 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:14.822+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:14.823343+0000) 2022-04-03T22:34:15.186 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:15.184+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:15.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:15.368+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:15.824 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:15.823+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:15.823476+0000) 2022-04-03T22:34:16.137 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:16.136+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:16.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:16.357+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:16.824 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:16.823+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:16.823653+0000) 2022-04-03T22:34:17.110 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:17.108+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:17.400 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:17.398+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:17.825 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:17.823+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:17.823813+0000) 2022-04-03T22:34:18.064 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:18.062+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:18.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:18.437+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:18.825 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:18.823+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:18.823957+0000) 2022-04-03T22:34:19.027 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:34:19.025+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:34:19.099 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:19.098+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:19.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:19.485+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:19.825 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:19.823+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:19.824106+0000) 2022-04-03T22:34:20.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:20.052+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:20.466 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:20.464+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:20.825 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:20.824+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:20.824263+0000) 2022-04-03T22:34:21.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:21.078+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:21.511 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:21.509+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:21.825 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:21.824+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:21.824411+0000) 2022-04-03T22:34:22.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:22.122+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:22.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:22.510+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:22.825 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:22.824+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:22.824565+0000) 2022-04-03T22:34:23.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:23.170+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:23.462 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:23.460+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:23.825 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:23.824+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:23.824686+0000) 2022-04-03T22:34:24.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:24.153+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:24.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:24.466+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:24.826 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:24.824+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:24.824837+0000) 2022-04-03T22:34:25.114 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:25.113+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:25.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:25.515+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:25.826 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:25.824+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:25.824979+0000) 2022-04-03T22:34:26.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:26.138+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:26.477 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:26.476+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:26.826 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:26.825+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:26.825186+0000) 2022-04-03T22:34:27.148 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:27.146+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:27.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:27.503+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:27.826 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:27.825+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:27.825350+0000) 2022-04-03T22:34:28.166 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:28.165+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:28.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:28.545+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:28.826 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:28.824+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:28.825527+0000) 2022-04-03T22:34:29.120 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:29.118+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:29.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:29.551+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:29.827 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:29.825+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:29.825707+0000) 2022-04-03T22:34:30.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:30.158+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:30.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:30.514+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:30.826 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:30.825+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:30.825912+0000) 2022-04-03T22:34:31.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:31.170+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:31.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:31.514+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:31.827 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:31.825+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:31.826077+0000) 2022-04-03T22:34:31.980 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:34:31.984 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:34:32.169 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:32.168+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:32.546 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:32.545+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:32.827 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:32.824+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:32.826292+0000) 2022-04-03T22:34:33.204 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:33.203+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:33.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:33.550+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:33.827 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:33.825+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:33.826435+0000) 2022-04-03T22:34:34.036 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:34:34.033+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:34:34.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:34.165+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:34.570 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:34.569+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:34.828 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:34.824+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:34.826639+0000) 2022-04-03T22:34:35.129 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:35.128+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:35.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:35.587+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:35.827 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:35.825+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:35.826806+0000) 2022-04-03T22:34:36.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:36.079+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:36.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:36.634+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:36.828 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:36.826+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:36.826985+0000) 2022-04-03T22:34:37.053 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:37.052+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:37.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:37.637+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:37.828 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:37.826+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:37.827134+0000) 2022-04-03T22:34:38.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:38.007+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:38.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:38.642+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:38.828 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:38.826+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:38.827250+0000) 2022-04-03T22:34:38.960 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:38.958+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:39.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:39.657+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:39.828 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:39.826+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:39.827370+0000) 2022-04-03T22:34:39.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:39.936+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:40.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:40.624+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:40.828 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:40.826+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:40.827583+0000) 2022-04-03T22:34:40.925 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:40.923+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:41.662 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:41.660+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:41.828 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:41.826+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:41.827738+0000) 2022-04-03T22:34:41.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:41.883+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:42.694 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:42.692+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:42.828 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:42.827+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:42.827968+0000) 2022-04-03T22:34:42.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:42.850+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:43.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:43.680+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:43.829 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:43.827+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:43.828124+0000) 2022-04-03T22:34:43.901 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:43.899+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:44.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:44.644+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:44.829 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:44.827+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:44.828314+0000) 2022-04-03T22:34:44.943 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:44.941+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:45.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:45.625+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:45.829 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:45.827+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:45.828529+0000) 2022-04-03T22:34:45.935 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:45.933+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:46.662 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:46.659+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:46.829 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:46.828+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:46.828730+0000) 2022-04-03T22:34:46.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:46.953+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:47.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:47.707+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:47.829 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:47.828+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:47.828928+0000) 2022-04-03T22:34:47.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:47.940+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:48.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:48.747+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:48.830 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:48.828+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:48.829111+0000) 2022-04-03T22:34:48.959 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:48.957+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:49.052 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:34:49.050+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:34:49.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:49.755+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:49.830 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:49.828+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:49.829293+0000) 2022-04-03T22:34:49.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:49.921+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:50.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:50.745+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:50.830 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:50.828+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:50.829499+0000) 2022-04-03T22:34:50.892 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:50.890+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:51.712 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:51.710+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:51.830 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:51.829+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:51.829690+0000) 2022-04-03T22:34:51.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:51.851+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:52.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:52.738+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:52.831 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:52.829+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:52.829862+0000) 2022-04-03T22:34:52.852 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:52.850+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:53.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:53.785+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:53.831 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:53.829+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:53.830037+0000) 2022-04-03T22:34:53.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:53.879+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:54.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:54.774+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:54.831 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:54.829+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:54.830188+0000) 2022-04-03T22:34:54.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:54.861+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:55.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:55.747+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:55.831 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:55.829+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:55.830314+0000) 2022-04-03T22:34:55.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:55.859+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:56.710 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:56.708+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:56.831 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:56.829+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:56.830470+0000) 2022-04-03T22:34:56.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:56.898+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:57.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:57.669+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:57.831 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:57.830+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:57.830662+0000) 2022-04-03T22:34:57.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:57.877+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:58.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:58.693+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:58.831 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:58.830+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:58.830818+0000) 2022-04-03T22:34:58.837 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:58.835+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:59.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:34:59.712+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:34:59.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:34:59.828+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:34:59.831 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:34:59.830+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:34:59.830930+0000) 2022-04-03T22:35:00.706 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:00.704+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:00.832 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:00.830+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:00.831069+0000) 2022-04-03T22:35:00.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:00.845+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:01.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:01.673+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:01.832 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:01.830+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:01.831266+0000) 2022-04-03T22:35:01.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:01.888+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:02.078 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:35:02.082 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:35:02.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:02.644+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:02.832 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:02.831+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:02.831448+0000) 2022-04-03T22:35:02.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:02.847+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:03.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:03.680+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:03.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:03.797+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:03.832 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:03.830+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:03.831619+0000) 2022-04-03T22:35:04.068 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:35:04.066+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:35:04.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:04.689+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:04.832 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:04.831+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:04.831754+0000) 2022-04-03T22:35:04.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:04.843+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:05.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:05.739+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:05.798 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:05.796+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:05.832 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:05.831+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:05.831965+0000) 2022-04-03T22:35:06.720 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:06.718+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:06.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:06.811+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:06.833 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:06.831+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:06.832144+0000) 2022-04-03T22:35:07.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:07.735+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:07.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:07.820+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:07.833 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:07.832+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:07.832308+0000) 2022-04-03T22:35:07.959 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-03T22:35:07.957+0000 7f2fd661a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T22:35:08.746 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:08.745+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:08.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:08.831+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:08.833 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:08.831+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:08.832524+0000) 2022-04-03T22:35:09.790 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:09.787+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:09.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:09.822+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:09.833 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:09.831+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:09.832720+0000) 2022-04-03T22:35:10.765 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:10.763+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:10.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:10.802+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:10.833 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:10.832+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:10.832922+0000) 2022-04-03T22:35:11.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:11.746+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:11.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:11.799+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:11.834 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:11.832+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:11.833091+0000) 2022-04-03T22:35:12.754 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:12.753+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:12.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:12.780+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:12.834 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:12.833+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:12.833274+0000) 2022-04-03T22:35:13.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:13.757+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:13.792 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:13.790+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:13.834 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:13.833+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:13.833452+0000) 2022-04-03T22:35:14.752 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:14.750+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:14.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:14.785+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:14.834 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:14.833+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:14.833627+0000) 2022-04-03T22:35:15.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:15.739+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:15.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:15.830+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:15.834 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:15.832+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:15.833763+0000) 2022-04-03T22:35:16.777 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:16.775+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:16.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:16.818+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:16.835 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:16.832+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:16.833971+0000) 2022-04-03T22:35:17.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:17.781+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:17.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:17.802+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:17.835 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:17.833+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:17.834147+0000) 2022-04-03T22:35:18.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:18.750+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:18.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:18.801+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:18.835 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:18.833+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:18.834300+0000) 2022-04-03T22:35:19.083 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:35:19.081+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:35:19.768 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:19.767+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:19.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:19.834+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:19.836 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:19.832+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:19.834503+0000) 2022-04-03T22:35:20.807 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:20.806+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:20.835 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:20.833+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:20.834729+0000) 2022-04-03T22:35:20.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:20.877+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:21.836 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:21.833+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:21.834881+0000) 2022-04-03T22:35:21.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:21.842+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:21.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:21.881+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:22.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:22.821+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:22.836 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:22.834+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:22.835110+0000) 2022-04-03T22:35:22.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:22.911+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:23.836 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:23.834+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:23.835273+0000) 2022-04-03T22:35:23.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:23.842+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:23.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:23.936+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:24.836 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:24.834+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:24.835432+0000) 2022-04-03T22:35:24.879 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:24.877+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:24.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:24.923+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:25.830 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:25.828+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:25.836 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:25.834+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:25.835640+0000) 2022-04-03T22:35:25.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:25.926+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:35:26.836 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:26.835+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:26.835876+0000) 2022-04-03T22:35:26.875 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:26.873+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:26.916 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:26.915+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:27.827 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:27.826+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:27.837 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:27.835+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:27.836052+0000) 2022-04-03T22:35:27.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:27.878+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:28.794 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:28.793+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:28.837 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:28.835+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:28.836259+0000) 2022-04-03T22:35:28.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:28.885+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:29.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:29.747+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:29.837 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:29.835+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:29.836409+0000) 2022-04-03T22:35:29.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:29.860+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:30.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:30.699+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:30.837 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:30.835+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:30.836695+0000) 2022-04-03T22:35:30.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:30.842+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:31.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:31.749+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:31.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:31.808+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:31.837 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:31.836+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:31.836931+0000) 2022-04-03T22:35:32.152 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:35:32.156 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:35:32.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:32.792+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:32.838 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:32.836+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:32.837200+0000) 2022-04-03T22:35:32.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:32.847+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:33.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:33.831+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:33.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:33.835+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:33.838 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:33.836+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:33.837467+0000) 2022-04-03T22:35:34.099 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:35:34.097+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:35:34.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:34.809+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:34.837 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:34.835+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:34.838 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:34.837+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:34.837718+0000) 2022-04-03T22:35:35.816 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:35.813+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:35.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:35.827+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:35.838 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:35.837+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:35.837909+0000) 2022-04-03T22:35:36.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:36.799+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:36.839 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:36.837+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:36.838134+0000) 2022-04-03T22:35:36.841 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:36.839+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:37.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:37.808+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:37.827 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:37.825+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:37.839 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:37.837+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:37.838330+0000) 2022-04-03T22:35:38.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:38.779+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:38.803 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:38.800+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:38.839 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:38.837+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:38.838550+0000) 2022-04-03T22:35:39.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:39.760+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:39.812 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:39.810+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:39.840 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:39.838+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:39.838836+0000) 2022-04-03T22:35:40.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:40.733+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:40.774 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:40.772+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:40.840 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:40.838+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:40.839105+0000) 2022-04-03T22:35:41.763 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:41.761+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:41.781 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:41.779+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:41.840 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:41.838+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:41.839351+0000) 2022-04-03T22:35:42.730 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:42.727+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:42.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:42.734+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:42.840 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:42.839+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:42.839614+0000) 2022-04-03T22:35:43.683 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:43.681+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:43.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:43.709+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:43.840 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:43.839+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:43.839881+0000) 2022-04-03T22:35:44.705 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:44.703+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:44.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:44.720+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:44.841 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:44.839+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:44.840134+0000) 2022-04-03T22:35:45.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:45.654+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:45.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:45.732+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:45.841 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:45.839+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:45.840382+0000) 2022-04-03T22:35:46.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:46.618+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:46.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:46.735+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:46.841 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:46.840+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:46.840629+0000) 2022-04-03T22:35:47.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:47.653+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:47.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:47.727+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:47.841 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:47.840+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:47.840886+0000) 2022-04-03T22:35:48.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:48.668+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:48.694 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:48.692+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:48.842 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:48.840+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:48.841137+0000) 2022-04-03T22:35:49.115 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:35:49.113+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:35:49.634 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:49.631+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:49.735 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:49.733+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:49.842 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:49.841+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:49.841335+0000) 2022-04-03T22:35:50.645 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:50.643+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:50.769 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:50.767+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:50.842 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:50.841+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:50.841559+0000) 2022-04-03T22:35:51.608 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:51.606+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:51.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:51.780+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:51.842 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:51.841+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:51.841727+0000) 2022-04-03T22:35:52.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:52.627+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:52.772 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:52.770+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:52.843 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:52.841+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:52.841949+0000) 2022-04-03T22:35:53.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:53.669+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:53.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:53.740+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:53.843 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:53.841+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:53.842144+0000) 2022-04-03T22:35:54.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:54.662+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:54.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:54.723+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:54.843 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:54.842+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:54.842293+0000) 2022-04-03T22:35:55.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:55.622+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:55.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:55.714+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:55.843 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:55.842+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:55.842502+0000) 2022-04-03T22:35:56.626 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:56.624+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:56.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:56.675+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:56.843 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:56.842+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:56.842748+0000) 2022-04-03T22:35:57.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:57.614+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:57.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:57.717+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:57.844 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:57.842+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:57.842965+0000) 2022-04-03T22:35:58.596 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:58.595+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:58.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:58.719+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:58.844 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:58.843+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:58.843184+0000) 2022-04-03T22:35:59.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:35:59.558+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:35:59.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:35:59.758+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:35:59.844 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:35:59.842+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:35:59.843344+0000) 2022-04-03T22:36:00.563 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:00.561+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:00.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:00.723+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:00.844 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:00.842+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:00.843529+0000) 2022-04-03T22:36:01.550 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:01.548+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:01.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:01.765+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:01.845 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:01.842+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:01.843758+0000) 2022-04-03T22:36:02.277 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:36:02.280 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:36:02.502 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:02.501+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:02.803 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:02.801+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:02.845 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:02.842+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:02.844035+0000) 2022-04-03T22:36:03.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:03.486+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:03.809 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:03.808+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:03.845 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:03.843+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:03.844221+0000) 2022-04-03T22:36:04.131 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:36:04.128+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:36:04.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:04.484+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:04.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:04.777+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:04.845 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:04.843+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:04.844470+0000) 2022-04-03T22:36:05.477 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:05.475+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:05.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:05.730+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:05.845 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:05.843+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:05.844636+0000) 2022-04-03T22:36:06.475 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:06.474+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:06.739 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:06.737+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:06.846 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:06.843+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:06.844878+0000) 2022-04-03T22:36:07.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:07.519+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:07.775 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:07.772+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:07.846 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:07.844+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:07.845129+0000) 2022-04-03T22:36:08.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:08.537+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:08.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:08.756+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:08.846 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:08.844+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:08.845403+0000) 2022-04-03T22:36:09.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:09.549+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:09.775 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:09.774+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:09.846 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:09.844+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:09.845625+0000) 2022-04-03T22:36:10.601 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:10.599+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:10.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:10.741+0000 7f88d0ac1700 -1 osd.4 128 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:10.847 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:10.845+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:10.845885+0000) 2022-04-03T22:36:11.136 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-03T22:36:11.135+0000 7f26ee7a1700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T22:36:11.151 INFO:tasks.ceph.mds.f.smithi186.stderr: -1> 2022-04-03T22:36:11.135+0000 7f26ee7a1700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T22:36:11.173 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-03T22:36:11.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:11.579+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:11.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:11.732+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:11.847 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:11.845+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:11.846127+0000) 2022-04-03T22:36:12.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:12.543+0000 7fd3b190b700 -1 osd.6 128 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:12.765 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:12.763+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:12.847 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:12.845+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:12.846334+0000) 2022-04-03T22:36:13.527 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:13.526+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:13.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:13.727+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:13.847 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:13.845+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:13.846503+0000) 2022-04-03T22:36:14.573 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:14.572+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:14.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:14.766+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:14.848 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:14.845+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:14.846733+0000) 2022-04-03T22:36:15.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:15.552+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:15.768 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:15.767+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:15.848 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:15.846+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:15.846954+0000) 2022-04-03T22:36:16.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:16.551+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:16.801 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:16.800+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:16.848 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:16.846+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:16.847150+0000) 2022-04-03T22:36:17.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:17.574+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:17.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:17.793+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:17.848 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:17.846+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:17.847284+0000) 2022-04-03T22:36:18.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:18.535+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:18.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:18.832+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:18.848 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:18.846+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:18.847512+0000) 2022-04-03T22:36:19.139 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:36:19.137+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:36:19.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:19.576+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:19.792 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:19.791+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:19.848 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:19.847+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:19.847723+0000) 2022-04-03T22:36:20.550 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:20.548+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:20.799 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:20.797+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:20.849 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:20.847+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:20.847871+0000) 2022-04-03T22:36:21.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:21.574+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:21.809 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:21.808+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:21.849 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:21.847+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:21.848018+0000) 2022-04-03T22:36:22.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:22.535+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:22.841 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:22.840+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:22.849 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:22.847+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:22.848226+0000) 2022-04-03T22:36:23.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:23.570+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:23.810 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:23.809+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:23.849 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:23.847+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:23.848372+0000) 2022-04-03T22:36:24.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:24.586+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:24.831 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:24.830+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:24.849 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:24.848+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:24.848539+0000) 2022-04-03T22:36:25.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:25.582+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:25.849 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:25.848+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:25.848684+0000) 2022-04-03T22:36:25.852 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:25.851+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:26.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:26.576+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:26.823 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:26.822+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:26.849 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:26.848+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:26.848883+0000) 2022-04-03T22:36:27.573 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:27.572+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:27.789 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:27.787+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:27.850 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:27.848+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:27.849102+0000) 2022-04-03T22:36:28.612 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:28.609+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:28.796 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:28.794+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:28.850 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:28.848+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:28.849333+0000) 2022-04-03T22:36:29.591 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:29.589+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:29.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:29.749+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:29.850 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:29.849+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:29.849560+0000) 2022-04-03T22:36:30.608 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:30.605+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:30.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:30.768+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:30.850 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:30.849+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:30.849739+0000) 2022-04-03T22:36:31.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:31.654+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:31.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:31.726+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:31.851 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:31.849+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:31.849958+0000) 2022-04-03T22:36:32.424 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:36:32.428 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:36:32.684 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:32.682+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:32.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:32.699+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:32.851 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:32.849+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:32.850227+0000) 2022-04-03T22:36:33.649 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:33.647+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:33.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:33.732+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:33.851 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:33.850+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:33.850485+0000) 2022-04-03T22:36:34.155 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:36:34.153+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:36:34.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:34.645+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:34.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:34.696+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:34.851 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:34.850+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:34.850686+0000) 2022-04-03T22:36:35.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:35.678+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:35.681 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:35.679+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:35.852 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:35.850+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:35.850905+0000) 2022-04-03T22:36:36.662 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:36.660+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:36.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:36.698+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:36.852 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:36.850+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:36.851154+0000) 2022-04-03T22:36:37.674 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:37.672+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:37.678 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:37.676+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:37.852 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:37.850+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:37.851434+0000) 2022-04-03T22:36:38.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:38.689+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:38.708 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:38.706+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:38.852 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:38.851+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:38.851675+0000) 2022-04-03T22:36:39.716 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:39.714+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:39.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:39.731+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:39.852 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:39.851+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:39.851921+0000) 2022-04-03T22:36:40.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:40.701+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:40.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:40.712+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:40.853 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:40.851+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:40.852190+0000) 2022-04-03T22:36:41.687 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:41.685+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:41.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:41.724+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:41.853 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:41.852+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:41.852414+0000) 2022-04-03T22:36:42.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:42.637+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:42.707 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:42.704+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:42.853 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:42.852+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:42.852684+0000) 2022-04-03T22:36:43.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:43.595+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:43.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:43.696+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:43.854 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:43.852+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:43.852938+0000) 2022-04-03T22:36:44.602 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:44.600+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:44.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:44.678+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:44.854 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:44.853+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:44.853134+0000) 2022-04-03T22:36:45.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:45.644+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:45.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:45.687+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:45.854 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:45.852+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:45.853339+0000) 2022-04-03T22:36:46.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:46.609+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:46.661 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:46.659+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:46.854 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:46.852+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:46.853509+0000) 2022-04-03T22:36:47.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:47.619+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:47.634 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:47.633+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:47.854 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:47.852+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:47.853686+0000) 2022-04-03T22:36:48.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:48.626+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:48.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:48.662+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:48.855 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:48.852+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:48.853863+0000) 2022-04-03T22:36:49.171 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:36:49.168+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:36:49.621 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:49.619+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:49.706 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:49.704+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:49.855 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:49.853+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:49.854014+0000) 2022-04-03T22:36:50.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:50.599+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:50.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:50.677+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:50.855 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:50.853+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:50.854171+0000) 2022-04-03T22:36:51.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:51.635+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:51.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:51.652+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:51.855 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:51.853+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:51.854342+0000) 2022-04-03T22:36:52.603 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:52.600+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:52.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:52.662+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:52.855 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:52.853+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:52.854592+0000) 2022-04-03T22:36:53.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:53.572+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:53.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:53.691+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:53.855 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:53.853+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:53.854754+0000) 2022-04-03T22:36:54.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:54.591+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:54.654 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:54.653+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:54.856 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:54.854+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:54.855053+0000) 2022-04-03T22:36:55.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:55.591+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:55.675 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:55.673+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:55.856 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:55.854+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:55.855206+0000) 2022-04-03T22:36:56.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:56.554+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:56.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:56.686+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:56.856 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:56.854+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:56.855435+0000) 2022-04-03T22:36:57.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:57.545+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:57.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:57.657+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:57.856 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:57.854+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:57.855653+0000) 2022-04-03T22:36:58.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:58.539+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:58.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:58.688+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:58.856 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:58.855+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:58.855859+0000) 2022-04-03T22:36:59.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:36:59.494+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:36:59.673 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:36:59.670+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:36:59.857 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:36:59.855+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:36:59.856077+0000) 2022-04-03T22:37:00.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:00.512+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:00.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:00.631+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:00.857 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:00.855+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:00.856286+0000) 2022-04-03T22:37:01.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:01.551+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:01.601 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:01.599+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:01.857 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:01.855+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:01.856494+0000) 2022-04-03T22:37:02.569 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:02.566+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:02.571 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:37:02.573 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:02.571+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:02.575 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:37:02.857 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:02.856+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:02.856758+0000) 2022-04-03T22:37:03.554 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:03.552+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:03.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:03.596+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:03.858 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:03.856+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:03.856890+0000) 2022-04-03T22:37:04.187 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:37:04.185+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:37:04.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:04.558+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:04.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:04.610+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:04.858 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:04.856+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:04.857086+0000) 2022-04-03T22:37:05.516 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:05.514+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:05.655 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:05.654+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:05.858 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:05.856+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:05.857298+0000) 2022-04-03T22:37:06.468 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:06.467+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:06.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:06.643+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:06.858 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:06.856+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:06.857468+0000) 2022-04-03T22:37:07.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:07.422+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:07.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:07.635+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:07.858 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:07.857+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:07.857647+0000) 2022-04-03T22:37:08.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:08.380+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:08.629 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:08.628+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:08.859 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:08.857+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:08.857832+0000) 2022-04-03T22:37:09.366 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:09.365+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:09.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:09.596+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:09.859 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:09.857+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:09.858048+0000) 2022-04-03T22:37:10.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:10.324+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:10.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:10.619+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:10.859 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:10.857+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:10.858262+0000) 2022-04-03T22:37:11.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:11.360+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:11.666 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:11.665+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:11.859 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:11.857+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:11.858466+0000) 2022-04-03T22:37:12.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:12.322+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:12.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:12.625+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:12.859 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:12.858+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:12.858647+0000) 2022-04-03T22:37:13.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:13.325+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:13.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:13.644+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:13.860 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:13.858+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:13.858890+0000) 2022-04-03T22:37:14.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:14.349+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:14.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:14.666+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:14.860 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:14.858+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:14.859082+0000) 2022-04-03T22:37:15.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:15.361+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:15.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:15.617+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:15.860 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:15.858+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:15.859280+0000) 2022-04-03T22:37:16.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:16.368+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:16.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:16.633+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:16.860 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:16.858+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:16.859487+0000) 2022-04-03T22:37:17.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:17.322+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:17.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:17.680+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:17.860 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:17.859+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:17.859688+0000) 2022-04-03T22:37:18.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:18.325+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:18.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:18.712+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:18.860 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:18.859+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:18.859856+0000) 2022-04-03T22:37:19.202 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:37:19.201+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:37:19.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:19.352+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:19.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:19.717+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:19.861 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:19.859+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:19.860071+0000) 2022-04-03T22:37:20.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:20.400+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:20.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:20.690+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:20.861 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:20.859+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:20.860310+0000) 2022-04-03T22:37:21.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:21.375+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:21.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:21.664+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:21.861 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:21.860+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:21.860546+0000) 2022-04-03T22:37:22.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:22.408+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:22.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:22.710+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:22.862 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:22.860+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:22.860753+0000) 2022-04-03T22:37:23.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:23.370+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:23.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:23.693+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:23.862 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:23.860+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:23.860959+0000) 2022-04-03T22:37:24.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:24.421+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:24.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:24.654+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:24.862 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:24.860+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:24.861158+0000) 2022-04-03T22:37:25.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:25.373+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:25.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:25.678+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:25.862 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:25.860+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:25.861332+0000) 2022-04-03T22:37:26.366 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:26.364+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:26.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:26.697+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:26.862 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:26.861+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:26.861540+0000) 2022-04-03T22:37:27.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:27.413+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:27.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:27.678+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:27.862 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:27.860+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:27.861702+0000) 2022-04-03T22:37:28.420 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:28.418+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:28.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:28.666+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:28.863 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:28.861+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:28.861916+0000) 2022-04-03T22:37:29.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:29.406+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:29.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:29.665+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:29.863 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:29.862+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:29.862108+0000) 2022-04-03T22:37:30.420 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:30.418+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:30.705 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:30.702+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:30.863 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:30.861+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:30.862304+0000) 2022-04-03T22:37:31.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:31.452+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:31.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:31.666+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:31.863 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:31.861+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:31.862434+0000) 2022-04-03T22:37:32.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:32.456+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:32.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:32.624+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:32.703 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:37:32.707 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:37:32.863 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:32.860+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:32.862671+0000) 2022-04-03T22:37:33.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:33.429+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:33.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:33.614+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:33.863 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:33.861+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:33.862825+0000) 2022-04-03T22:37:34.218 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:37:34.215+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:37:34.428 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:34.425+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:34.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:34.652+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:34.864 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:34.862+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:34.863053+0000) 2022-04-03T22:37:35.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:35.429+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:35.678 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:35.676+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:35.864 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:35.862+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:35.863302+0000) 2022-04-03T22:37:36.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:36.416+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:36.689 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:36.686+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:36.864 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:36.862+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:36.863546+0000) 2022-04-03T22:37:37.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:37.376+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:37.704 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:37.702+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:37.864 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:37.862+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:37.863742+0000) 2022-04-03T22:37:38.396 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:38.394+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:38.705 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:38.703+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:38.865 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:38.862+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:38.863888+0000) 2022-04-03T22:37:39.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:39.374+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:39.666 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:39.663+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:39.865 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:39.863+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:39.864041+0000) 2022-04-03T22:37:40.374 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:40.372+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:40.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:40.697+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:40.865 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:40.863+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:40.864269+0000) 2022-04-03T22:37:41.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:41.414+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:41.659 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:41.657+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:41.865 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:41.863+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:41.864440+0000) 2022-04-03T22:37:42.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:42.388+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:42.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:42.612+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:42.865 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:42.863+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:42.864614+0000) 2022-04-03T22:37:43.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:43.371+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:43.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:43.617+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:43.865 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:43.864+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:43.864779+0000) 2022-04-03T22:37:44.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:44.391+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:44.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:44.585+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:44.866 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:44.864+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:44.865024+0000) 2022-04-03T22:37:45.384 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:45.382+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:45.586 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:45.584+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:45.866 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:45.864+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:45.865239+0000) 2022-04-03T22:37:46.420 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:46.418+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:46.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:46.563+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:46.866 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:46.864+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:46.865480+0000) 2022-04-03T22:37:47.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:47.465+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:47.550 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:47.548+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:47.866 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:47.865+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:47.865725+0000) 2022-04-03T22:37:48.426 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:48.424+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:48.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:48.589+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:48.867 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:48.865+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:48.865988+0000) 2022-04-03T22:37:49.232 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:37:49.230+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:37:49.442 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:49.440+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:49.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:49.586+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:49.867 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:49.865+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:49.866237+0000) 2022-04-03T22:37:50.395 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:50.394+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:50.558 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:50.556+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:50.867 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:50.865+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:50.866444+0000) 2022-04-03T22:37:51.387 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:51.385+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:51.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:51.557+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:51.867 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:51.866+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:51.866660+0000) 2022-04-03T22:37:52.345 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:52.343+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:52.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:52.516+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:52.868 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:52.866+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:52.866874+0000) 2022-04-03T22:37:53.360 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:53.356+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:53.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:53.494+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:53.868 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:53.866+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:53.867097+0000) 2022-04-03T22:37:54.402 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:54.400+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:54.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:54.485+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:54.868 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:54.866+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:54.867294+0000) 2022-04-03T22:37:55.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:55.433+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:55.457 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:55.456+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:55.868 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:55.866+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:55.867490+0000) 2022-04-03T22:37:56.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:56.430+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:56.438 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:56.436+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:56.868 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:56.867+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:56.867685+0000) 2022-04-03T22:37:57.402 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:57.400+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:57.446 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:57.444+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:57.869 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:57.867+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:57.867907+0000) 2022-04-03T22:37:58.412 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:58.410+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:58.478 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:58.476+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:58.869 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:58.867+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:58.868161+0000) 2022-04-03T22:37:59.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:37:59.453+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:37:59.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:37:59.465+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:37:59.869 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:37:59.867+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:37:59.868370+0000) 2022-04-03T22:38:00.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:00.442+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:00.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:00.457+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:00.869 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:00.868+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:00.868624+0000) 2022-04-03T22:38:01.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:01.428+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:01.453 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:01.451+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:01.869 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:01.868+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:01.868821+0000) 2022-04-03T22:38:02.453 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:02.451+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:02.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:02.479+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:02.835 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:38:02.839 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:38:02.870 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:02.868+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:02.868968+0000) 2022-04-03T22:38:03.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:03.459+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:03.520 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:03.519+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:03.870 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:03.868+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:03.869199+0000) 2022-04-03T22:38:04.248 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:38:04.246+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:38:04.446 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:04.444+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:04.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:04.552+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:04.870 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:04.869+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:04.869445+0000) 2022-04-03T22:38:05.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:05.403+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:05.602 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:05.601+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:05.870 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:05.869+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:05.869620+0000) 2022-04-03T22:38:06.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:06.366+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:06.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:06.618+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:06.870 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:06.869+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:06.869814+0000) 2022-04-03T22:38:07.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:07.405+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:07.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:07.666+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:07.871 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:07.869+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:07.870055+0000) 2022-04-03T22:38:08.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:08.379+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:08.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:08.684+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:08.871 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:08.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:08.870275+0000) 2022-04-03T22:38:09.422 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:09.421+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:09.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:09.730+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:09.871 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:09.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:09.870454+0000) 2022-04-03T22:38:10.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:10.388+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:10.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:10.731+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:10.871 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:10.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:10.870633+0000) 2022-04-03T22:38:11.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:11.347+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:11.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:11.706+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:11.871 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:11.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:11.870798+0000) 2022-04-03T22:38:12.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:12.320+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:12.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:12.699+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:12.872 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:12.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:12.871077+0000) 2022-04-03T22:38:13.344 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:13.342+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:13.691 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:13.690+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:13.872 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:13.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:13.871318+0000) 2022-04-03T22:38:14.354 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:14.353+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:14.708 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:14.706+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:14.872 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:14.871+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:14.871517+0000) 2022-04-03T22:38:15.338 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:15.336+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:15.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:15.680+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:15.873 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:15.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:15.871733+0000) 2022-04-03T22:38:16.365 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:16.364+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:16.724 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:16.722+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:16.873 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:16.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:16.872024+0000) 2022-04-03T22:38:17.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:17.389+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:17.764 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:17.763+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:17.873 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:17.871+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:17.872266+0000) 2022-04-03T22:38:18.408 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:18.405+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:18.803 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:18.800+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:18.873 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:18.871+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:18.872516+0000) 2022-04-03T22:38:19.264 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:38:19.261+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:38:19.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:19.392+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:19.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:19.779+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:19.873 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:19.871+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:19.872790+0000) 2022-04-03T22:38:20.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:20.404+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:20.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:20.822+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:20.874 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:20.872+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:20.873065+0000) 2022-04-03T22:38:21.365 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:21.363+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:21.865 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:21.862+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:21.874 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:21.871+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:21.873252+0000) 2022-04-03T22:38:22.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:22.383+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:22.823 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:22.821+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:22.874 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:22.872+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:22.873504+0000) 2022-04-03T22:38:23.434 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:23.431+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:23.821 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:23.819+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:23.874 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:23.872+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:23.873725+0000) 2022-04-03T22:38:24.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:24.387+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:24.846 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:24.844+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:24.875 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:24.873+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:24.873979+0000) 2022-04-03T22:38:25.397 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:25.395+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:25.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:25.860+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:25.875 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:25.873+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:25.874215+0000) 2022-04-03T22:38:26.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:26.388+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:26.865 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:26.862+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:26.875 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:26.873+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:26.874468+0000) 2022-04-03T22:38:27.408 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:27.406+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:27.876 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:27.873+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:27.874744+0000) 2022-04-03T22:38:27.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:27.895+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:28.438 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:28.436+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:28.876 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:28.874+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:28.875004+0000) 2022-04-03T22:38:28.931 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:28.929+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:29.403 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:29.401+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:29.876 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:29.874+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:29.875249+0000) 2022-04-03T22:38:29.917 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:29.915+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:30.412 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:30.409+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:30.876 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:30.874+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:30.875449+0000) 2022-04-03T22:38:30.929 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:30.927+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:31.446 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:31.443+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:31.876 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:31.874+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:31.875671+0000) 2022-04-03T22:38:31.971 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:31.969+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:32.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:32.449+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:32.877 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:32.875+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:32.875924+0000) 2022-04-03T22:38:32.932 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:32.930+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:32.982 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:38:32.987 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:38:33.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:33.445+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:33.877 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:33.875+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:33.876193+0000) 2022-04-03T22:38:33.955 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:33.953+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:34.267 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:38:34.265+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:38:34.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:34.440+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:34.877 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:34.875+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:34.876455+0000) 2022-04-03T22:38:34.940 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:34.938+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:35.480 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:35.478+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:35.878 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:35.876+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:35.876671+0000) 2022-04-03T22:38:35.957 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:35.955+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:36.450 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:36.447+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:36.877 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:36.876+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:36.876866+0000) 2022-04-03T22:38:36.927 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:36.925+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:37.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:37.490+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:37.878 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:37.876+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:37.877095+0000) 2022-04-03T22:38:37.898 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:37.896+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:38.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:38.511+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:38.878 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:38.876+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:38.877293+0000) 2022-04-03T22:38:38.932 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:38.930+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:39.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:39.547+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:39.878 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:39.876+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:39.877524+0000) 2022-04-03T22:38:39.920 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:39.918+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:40.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:40.574+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:40.878 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:40.877+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:40.877747+0000) 2022-04-03T22:38:40.884 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:40.883+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:41.582 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:41.580+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:41.846 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:41.844+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:41.879 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:41.877+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:41.878016+0000) 2022-04-03T22:38:42.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:42.622+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:42.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:42.853+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:42.879 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:42.877+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:42.878284+0000) 2022-04-03T22:38:43.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:43.662+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:43.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:43.840+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:43.879 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:43.878+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:43.878461+0000) 2022-04-03T22:38:44.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:44.678+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:44.879 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:44.878+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:44.878630+0000) 2022-04-03T22:38:44.890 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:44.888+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:45.649 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:45.647+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:45.879 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:45.878+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:45.878886+0000) 2022-04-03T22:38:45.880 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:45.878+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:46.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:46.640+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:46.838 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:46.836+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:46.880 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:46.878+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:46.879107+0000) 2022-04-03T22:38:47.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:47.689+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:47.873 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:47.871+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:47.880 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:47.878+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:47.879328+0000) 2022-04-03T22:38:48.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:48.691+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:48.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:48.827+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:48.880 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:48.879+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:48.879566+0000) 2022-04-03T22:38:49.283 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:38:49.281+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:38:49.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:49.725+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:49.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:49.781+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:49.881 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:49.879+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:49.879806+0000) 2022-04-03T22:38:50.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:50.711+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:50.798 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:50.796+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:50.881 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:50.879+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:50.880001+0000) 2022-04-03T22:38:51.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:51.721+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:51.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:51.750+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:51.881 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:51.879+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:51.880221+0000) 2022-04-03T22:38:52.697 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:52.694+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:52.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:52.771+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:52.881 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:52.880+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:52.880432+0000) 2022-04-03T22:38:53.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:53.693+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:53.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:53.756+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:53.881 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:53.880+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:53.880617+0000) 2022-04-03T22:38:54.697 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:54.695+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:54.775 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:54.773+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:54.881 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:54.880+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:54.880852+0000) 2022-04-03T22:38:55.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:55.723+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:55.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:55.740+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:55.882 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:55.880+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:55.881100+0000) 2022-04-03T22:38:56.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:56.753+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:56.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:56.783+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:56.882 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:56.881+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:56.881323+0000) 2022-04-03T22:38:57.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:57.786+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:57.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:57.798+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:57.882 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:57.881+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:57.881491+0000) 2022-04-03T22:38:58.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:58.769+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:58.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:58.824+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:58.882 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:58.881+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:58.881702+0000) 2022-04-03T22:38:59.796 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:38:59.795+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:38:59.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:38:59.798+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:38:59.882 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:38:59.880+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:38:59.881855+0000) 2022-04-03T22:39:00.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:00.811+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:00.846 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:00.844+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:00.882 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:00.880+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:00.882083+0000) 2022-04-03T22:39:01.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:01.806+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:01.852 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:01.850+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:01.883 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:01.881+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:01.882261+0000) 2022-04-03T22:39:02.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:02.846+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:02.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:02.868+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:02.883 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:02.881+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:02.882485+0000) 2022-04-03T22:39:03.323 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:39:03.327 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:39:03.816 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:03.815+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:03.883 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:03.881+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:03.882763+0000) 2022-04-03T22:39:03.896 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:03.895+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:04.286 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:39:04.284+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:39:04.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:04.768+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:04.879 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:04.878+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:04.883 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:04.882+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:04.883040+0000) 2022-04-03T22:39:05.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:05.818+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:05.884 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:05.882+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:05.883242+0000) 2022-04-03T22:39:05.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:05.887+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:06.841 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:06.840+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:06.884 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:06.882+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:06.883406+0000) 2022-04-03T22:39:06.926 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:06.925+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:07.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:07.842+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:07.884 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:07.882+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:07.883574+0000) 2022-04-03T22:39:07.902 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:07.901+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:08.865 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:08.864+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:08.884 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:08.882+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:08.883846+0000) 2022-04-03T22:39:08.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:08.885+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:09.838 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:09.837+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:09.885 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:09.883+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:09.884096+0000) 2022-04-03T22:39:09.919 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:09.918+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:10.853 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:10.852+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:10.885 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:10.883+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:10.884355+0000) 2022-04-03T22:39:10.946 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:10.944+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:11.823 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:11.821+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:11.885 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:11.883+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:11.884598+0000) 2022-04-03T22:39:11.969 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:11.968+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:12.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:12.780+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:12.885 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:12.884+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:12.884873+0000) 2022-04-03T22:39:12.919 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:12.918+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:13.762 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:13.760+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:13.886 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:13.884+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:13.885075+0000) 2022-04-03T22:39:13.932 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:13.930+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:14.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:14.729+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:14.886 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:14.884+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:14.885243+0000) 2022-04-03T22:39:14.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:14.909+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:15.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:15.730+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 27 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:15.886 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:15.884+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:15.885421+0000) 2022-04-03T22:39:15.956 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:15.953+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:16.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:16.780+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:16.886 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:16.884+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:16.885639+0000) 2022-04-03T22:39:16.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:16.936+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:17.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:17.731+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:17.887 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:17.885+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:17.885885+0000) 2022-04-03T22:39:17.930 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:17.928+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:18.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:18.709+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:18.887 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:18.885+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:18.886068+0000) 2022-04-03T22:39:18.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:18.910+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:19.302 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:39:19.301+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:39:19.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:19.668+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:19.887 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:19.885+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:19.886254+0000) 2022-04-03T22:39:19.932 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:19.930+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:20.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:20.663+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:20.887 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:20.885+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:20.886485+0000) 2022-04-03T22:39:20.960 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:20.958+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:21.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:21.684+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:21.887 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:21.886+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:21.886667+0000) 2022-04-03T22:39:21.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:21.986+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:22.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:22.704+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:22.888 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:22.886+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:22.886890+0000) 2022-04-03T22:39:23.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:23.016+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:23.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:23.734+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:23.888 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:23.886+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:23.887138+0000) 2022-04-03T22:39:24.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:24.037+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:24.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:24.728+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:24.888 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:24.886+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:24.887388+0000) 2022-04-03T22:39:25.064 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:25.062+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:25.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:25.734+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:25.888 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:25.887+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:25.887656+0000) 2022-04-03T22:39:26.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:26.086+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:26.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:26.709+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:26.888 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:26.887+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:26.887929+0000) 2022-04-03T22:39:27.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:27.037+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:27.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:27.709+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:27.889 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:27.887+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:27.888123+0000) 2022-04-03T22:39:28.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:28.031+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:28.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:28.753+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:28.889 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:28.887+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:28.888360+0000) 2022-04-03T22:39:29.057 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:29.054+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:29.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:29.728+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:29.889 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:29.888+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:29.888562+0000) 2022-04-03T22:39:30.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:30.101+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:30.705 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:30.703+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:30.890 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:30.888+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:30.888833+0000) 2022-04-03T22:39:31.064 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:31.062+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:31.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:31.681+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:31.890 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:31.888+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:31.889084+0000) 2022-04-03T22:39:32.101 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:32.099+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:32.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:32.696+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:32.890 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:32.888+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:32.889333+0000) 2022-04-03T22:39:33.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:33.121+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:33.395 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:39:33.399 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:39:33.666 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:33.664+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:33.890 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:33.889+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:33.889523+0000) 2022-04-03T22:39:34.133 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:34.131+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:34.319 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:39:34.316+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:39:34.676 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:34.674+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:34.890 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:34.889+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:34.889753+0000) 2022-04-03T22:39:35.101 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:35.099+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:35.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:35.686+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:35.891 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:35.889+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:35.889970+0000) 2022-04-03T22:39:36.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:36.074+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:36.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:36.678+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:36.891 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:36.889+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:36.890178+0000) 2022-04-03T22:39:37.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:37.096+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:37.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:37.637+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:37.891 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:37.890+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:37.890409+0000) 2022-04-03T22:39:38.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:38.053+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:38.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:38.639+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:38.891 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:38.890+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:38.890660+0000) 2022-04-03T22:39:39.056 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:39.054+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:39.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:39.612+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:39.891 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:39.890+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:39.890881+0000) 2022-04-03T22:39:40.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:40.037+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:40.603 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:40.601+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:40.892 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:40.890+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:40.891047+0000) 2022-04-03T22:39:40.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:40.988+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:41.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:41.588+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:41.892 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:41.891+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:41.891231+0000) 2022-04-03T22:39:41.962 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:41.960+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:42.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:42.540+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:42.898 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:42.890+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:42.891491+0000) 2022-04-03T22:39:42.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:42.950+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:43.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:43.563+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:43.892 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:43.891+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:43.891688+0000) 2022-04-03T22:39:43.996 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:43.994+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:44.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:44.522+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:44.892 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:44.890+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:44.891912+0000) 2022-04-03T22:39:44.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:44.986+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:45.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:45.558+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:45.893 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:45.891+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:45.892129+0000) 2022-04-03T22:39:46.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:46.023+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:46.572 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:46.571+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:46.893 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:46.890+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:46.892337+0000) 2022-04-03T22:39:46.978 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:46.976+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:47.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:47.594+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:47.893 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:47.891+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:47.892535+0000) 2022-04-03T22:39:47.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:47.976+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:48.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:48.557+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:48.893 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:48.891+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:48.892821+0000) 2022-04-03T22:39:48.970 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:48.968+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:49.334 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:39:49.331+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:39:49.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:49.542+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:49.893 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:49.892+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:49.892990+0000) 2022-04-03T22:39:49.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:49.926+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:50.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:50.536+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:50.894 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:50.892+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:50.893192+0000) 2022-04-03T22:39:50.915 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:50.914+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:51.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:51.492+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:51.894 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:51.892+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:51.893431+0000) 2022-04-03T22:39:51.941 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:51.939+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:52.508 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:52.506+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:52.894 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:52.891+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:52.893672+0000) 2022-04-03T22:39:52.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:52.936+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:53.508 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:53.507+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:53.895 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:53.893+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:53.893885+0000) 2022-04-03T22:39:53.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:53.948+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:54.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:54.483+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:54.895 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:54.893+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:54.894115+0000) 2022-04-03T22:39:54.962 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:54.961+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:55.478 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:55.477+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:55.895 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:55.893+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:55.894292+0000) 2022-04-03T22:39:55.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:55.987+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:56.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:56.434+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:56.895 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:56.893+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:56.894524+0000) 2022-04-03T22:39:56.962 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:56.961+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:57.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:57.397+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:57.895 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:57.893+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:57.894745+0000) 2022-04-03T22:39:57.986 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:57.985+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:58.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:58.404+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:58.896 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:58.894+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:58.894968+0000) 2022-04-03T22:39:59.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:39:59.035+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:39:59.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:39:59.382+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:39:59.896 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:39:59.894+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:39:59.895180+0000) 2022-04-03T22:40:00.018 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:00.016+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:00.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:00.425+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:00.896 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:00.894+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:00.895472+0000) 2022-04-03T22:40:00.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:00.984+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:01.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:01.443+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:01.896 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:01.894+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:01.895660+0000) 2022-04-03T22:40:01.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:01.949+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:02.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:02.482+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:02.896 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:02.895+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:02.895811+0000) 2022-04-03T22:40:02.920 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:02.919+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:03.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:03.468+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:03.477 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:40:03.480 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:40:03.897 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:03.895+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:03.896042+0000) 2022-04-03T22:40:03.919 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:03.918+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:04.350 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:40:04.348+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:40:04.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:04.432+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:04.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:04.892+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:04.897 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:04.895+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:04.896270+0000) 2022-04-03T22:40:05.415 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:05.413+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:05.897 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:05.895+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:05.896512+0000) 2022-04-03T22:40:05.932 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:05.931+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:06.388 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:06.386+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:06.897 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:06.896+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:06.896761+0000) 2022-04-03T22:40:06.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:06.922+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:07.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:07.364+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:07.898 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:07.896+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:07.897034+0000) 2022-04-03T22:40:07.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:07.939+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:08.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:08.384+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:08.898 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:08.896+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:08.897277+0000) 2022-04-03T22:40:08.976 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:08.974+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:09.415 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:09.412+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:09.898 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:09.896+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:09.897525+0000) 2022-04-03T22:40:09.972 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:09.969+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:10.375 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:10.373+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:10.899 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:10.897+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:10.897817+0000) 2022-04-03T22:40:10.986 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:10.983+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:11.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:11.415+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:11.899 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:11.897+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:11.898013+0000) 2022-04-03T22:40:11.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:11.971+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:12.434 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:12.432+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:12.899 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:12.897+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:12.898241+0000) 2022-04-03T22:40:12.987 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:12.984+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:13.423 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:13.421+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:13.899 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:13.897+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:13.898451+0000) 2022-04-03T22:40:14.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:14.026+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:14.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:14.387+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:14.899 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:14.898+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:14.898651+0000) 2022-04-03T22:40:15.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:15.022+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:15.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:15.414+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:15.900 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:15.898+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:15.898846+0000) 2022-04-03T22:40:16.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:16.069+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:16.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:16.384+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:16.900 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:16.898+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:16.899039+0000) 2022-04-03T22:40:17.030 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:17.028+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:17.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:17.357+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:17.900 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:17.898+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:17.899258+0000) 2022-04-03T22:40:18.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:18.007+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:18.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:18.337+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:18.900 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:18.899+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:18.899449+0000) 2022-04-03T22:40:19.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:19.045+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:19.366 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:40:19.364+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:40:19.375 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:19.373+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:19.900 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:19.899+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:19.899666+0000) 2022-04-03T22:40:20.094 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:20.092+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:20.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:20.378+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:20.901 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:20.899+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:20.899952+0000) 2022-04-03T22:40:21.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:21.057+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:21.429 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:21.427+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:21.901 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:21.899+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:21.900123+0000) 2022-04-03T22:40:22.023 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:22.021+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:22.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:22.381+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:22.901 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:22.900+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:22.900344+0000) 2022-04-03T22:40:23.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:23.052+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:23.382 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:23.380+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:23.901 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:23.900+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:23.900560+0000) 2022-04-03T22:40:24.091 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:24.089+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:24.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:24.332+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:24.902 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:24.900+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:24.900853+0000) 2022-04-03T22:40:25.137 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:25.136+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:25.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:25.326+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:25.902 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:25.900+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:25.901103+0000) 2022-04-03T22:40:26.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:26.165+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:26.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:26.357+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:26.902 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:26.901+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:26.901348+0000) 2022-04-03T22:40:27.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:27.134+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:27.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:27.320+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:27.902 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:27.901+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:27.901593+0000) 2022-04-03T22:40:28.104 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:28.103+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:28.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:28.364+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:28.902 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:28.901+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:28.901862+0000) 2022-04-03T22:40:29.118 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:29.116+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:29.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:29.333+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:29.903 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:29.901+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:29.902077+0000) 2022-04-03T22:40:30.101 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:30.099+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:30.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:30.378+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:30.903 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:30.901+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:30.902303+0000) 2022-04-03T22:40:31.018 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-03T22:40:31.015+0000 7f2fd661a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T22:40:31.018 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-03T22:40:31.015+0000 7f2fd761c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T22:40:31.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:31.066+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:31.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:31.352+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:31.903 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:31.901+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:31.902550+0000) 2022-04-03T22:40:32.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:32.023+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:32.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:32.376+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:32.903 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:32.901+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:32.902830+0000) 2022-04-03T22:40:33.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:33.071+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:33.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:33.331+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:33.592 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:40:33.596 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:40:33.904 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:33.902+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:33.903123+0000) 2022-04-03T22:40:34.072 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:34.070+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:34.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:34.294+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:34.381 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:40:34.379+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:40:34.904 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:34.902+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:34.903347+0000) 2022-04-03T22:40:35.045 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:35.043+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:35.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:35.271+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:35.904 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:35.902+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:35.903595+0000) 2022-04-03T22:40:36.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:36.059+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:36.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:36.314+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:36.905 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:36.902+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:36.903882+0000) 2022-04-03T22:40:37.031 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:37.029+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:37.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:37.278+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:37.905 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:37.903+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:37.904127+0000) 2022-04-03T22:40:38.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:38.011+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:38.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:38.313+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:38.905 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:38.903+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:38.904373+0000) 2022-04-03T22:40:39.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:39.059+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:39.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:39.318+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:39.905 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:39.903+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:39.904581+0000) 2022-04-03T22:40:40.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:40.074+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:40.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:40.358+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:40.905 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:40.904+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:40.904828+0000) 2022-04-03T22:40:41.034 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:41.033+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:41.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:41.362+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:41.906 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:41.904+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:41.905029+0000) 2022-04-03T22:40:42.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:42.047+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:42.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:42.401+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:42.906 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:42.904+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:42.905294+0000) 2022-04-03T22:40:43.094 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:43.093+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:43.423 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:43.421+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:43.906 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:43.904+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:43.905555+0000) 2022-04-03T22:40:44.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:44.065+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:44.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:44.418+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:44.906 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:44.905+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:44.905815+0000) 2022-04-03T22:40:45.036 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:45.034+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:45.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:45.446+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:45.907 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:45.905+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:45.906057+0000) 2022-04-03T22:40:46.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:46.036+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:46.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:46.456+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:46.907 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:46.905+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:46.906261+0000) 2022-04-03T22:40:47.069 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:47.068+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:47.466 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:47.464+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:47.907 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:47.905+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:47.906475+0000) 2022-04-03T22:40:48.042 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:48.041+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:48.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:48.479+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:48.907 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:48.906+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:48.906676+0000) 2022-04-03T22:40:49.005 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:49.004+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:49.397 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:40:49.395+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:40:49.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:49.465+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:49.907 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:49.906+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:49.906836+0000) 2022-04-03T22:40:50.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:50.046+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:50.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:50.426+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:50.908 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:50.906+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:50.907053+0000) 2022-04-03T22:40:51.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:51.074+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:51.402 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:51.401+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:51.908 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:51.906+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:51.907283+0000) 2022-04-03T22:40:52.052 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:52.050+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:52.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:52.353+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:52.908 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:52.906+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:52.907532+0000) 2022-04-03T22:40:53.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:53.064+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:53.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:53.392+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:53.908 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:53.906+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:53.907729+0000) 2022-04-03T22:40:54.031 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:54.030+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:54.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:54.372+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:54.908 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:54.907+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:54.907947+0000) 2022-04-03T22:40:55.029 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:55.027+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:55.421 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:55.420+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:55.909 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:55.907+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:55.908148+0000) 2022-04-03T22:40:56.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:56.024+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:56.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:56.402+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:56.909 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:56.907+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:56.908396+0000) 2022-04-03T22:40:56.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:56.991+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:57.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:57.383+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:57.909 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:57.908+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:57.908542+0000) 2022-04-03T22:40:57.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:57.983+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:58.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:58.376+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:58.909 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:58.908+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:58.908678+0000) 2022-04-03T22:40:59.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:40:59.027+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:40:59.418 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:40:59.417+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:40:59.909 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:40:59.908+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:40:59.908838+0000) 2022-04-03T22:41:00.065 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:00.063+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:00.460 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:00.459+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:00.910 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:00.908+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:00.909046+0000) 2022-04-03T22:41:01.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:01.046+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:01.418 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:01.417+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:01.910 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:01.908+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:01.909277+0000) 2022-04-03T22:41:02.084 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:02.083+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:02.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:02.380+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:02.910 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:02.909+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:02.909491+0000) 2022-04-03T22:41:03.043 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:03.042+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:03.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:03.376+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:03.673 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:41:03.677 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:41:03.910 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:03.908+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:03.909747+0000) 2022-04-03T22:41:04.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:04.074+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:04.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:04.394+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:04.413 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:41:04.411+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:41:04.911 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:04.909+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:04.910006+0000) 2022-04-03T22:41:05.065 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:05.063+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:05.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:05.439+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:05.911 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:05.909+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:05.910249+0000) 2022-04-03T22:41:06.060 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:06.058+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:06.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:06.453+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:06.911 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:06.910+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:06.910433+0000) 2022-04-03T22:41:07.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:07.094+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:07.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:07.456+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:07.911 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:07.910+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:07.910637+0000) 2022-04-03T22:41:08.100 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:08.098+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:08.479 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:08.477+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:08.911 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:08.910+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:08.910823+0000) 2022-04-03T22:41:09.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:09.101+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:09.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:09.493+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:09.912 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:09.910+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:09.911032+0000) 2022-04-03T22:41:10.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:10.059+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:10.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:10.480+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:10.912 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:10.911+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:10.911276+0000) 2022-04-03T22:41:11.098 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:11.096+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:11.460 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:11.457+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:11.912 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:11.911+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:11.911506+0000) 2022-04-03T22:41:12.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:12.145+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:12.422 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:12.420+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:12.912 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:12.911+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:12.911735+0000) 2022-04-03T22:41:13.185 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:13.182+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:13.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:13.383+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:13.913 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:13.911+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:13.912024+0000) 2022-04-03T22:41:14.181 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:14.179+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:14.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:14.338+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:14.913 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:14.910+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:14.912249+0000) 2022-04-03T22:41:15.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:15.168+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:15.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:15.317+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:15.913 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:15.911+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:15.912479+0000) 2022-04-03T22:41:16.151 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:16.149+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:16.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:16.320+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:16.913 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:16.911+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:16.912660+0000) 2022-04-03T22:41:17.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:17.134+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:17.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:17.278+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:17.913 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:17.911+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:17.912888+0000) 2022-04-03T22:41:18.115 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:18.113+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:18.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:18.328+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:18.914 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:18.912+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:18.913068+0000) 2022-04-03T22:41:19.114 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:19.112+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:19.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:19.300+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:19.429 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:41:19.426+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:41:19.914 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:19.912+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:19.913274+0000) 2022-04-03T22:41:20.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:20.124+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:20.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:20.253+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:20.914 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:20.912+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:20.913515+0000) 2022-04-03T22:41:21.090 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:21.088+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:21.212 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:21.210+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:21.914 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:21.912+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:21.913732+0000) 2022-04-03T22:41:22.045 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:22.044+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:22.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:22.258+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:22.915 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:22.913+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:22.913943+0000) 2022-04-03T22:41:22.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:22.995+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:23.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:23.260+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:23.915 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:23.913+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:23.914181+0000) 2022-04-03T22:41:23.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:23.983+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:24.221 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:24.219+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:24.915 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:24.913+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:24.914412+0000) 2022-04-03T22:41:24.945 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:24.943+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:25.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:25.222+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:25.915 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:25.913+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:25.914600+0000) 2022-04-03T22:41:25.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:25.935+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:26.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:26.256+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:26.915 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:26.914+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:26.914849+0000) 2022-04-03T22:41:26.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:26.929+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:27.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:27.214+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:27.916 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:27.914+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:27.915069+0000) 2022-04-03T22:41:27.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:27.919+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:28.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:28.207+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:28.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:28.879+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:28.916 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:28.914+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:28.915294+0000) 2022-04-03T22:41:29.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:29.209+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:29.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:29.901+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:29.916 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:29.914+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:29.915541+0000) 2022-04-03T22:41:30.171 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:30.169+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:30.884 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:30.881+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:30.916 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:30.914+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:30.915752+0000) 2022-04-03T22:41:31.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:31.216+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:31.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:31.883+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:31.917 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:31.915+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:31.916015+0000) 2022-04-03T22:41:32.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:32.198+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:32.850 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:32.848+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:32.917 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:32.915+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:32.916247+0000) 2022-04-03T22:41:33.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:33.188+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:33.795 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:41:33.799 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:41:33.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:33.863+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:33.917 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:33.915+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:33.916470+0000) 2022-04-03T22:41:34.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:34.161+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:34.445 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:41:34.443+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:41:34.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:34.909+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:34.917 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:34.916+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:34.916696+0000) 2022-04-03T22:41:35.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:35.173+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:35.918 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:35.916+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:35.916904+0000) 2022-04-03T22:41:35.959 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:35.957+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:36.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:36.172+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:36.918 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:36.916+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:36.917136+0000) 2022-04-03T22:41:36.936 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:36.935+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:37.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:37.150+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:37.918 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:37.916+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:37.917369+0000) 2022-04-03T22:41:37.944 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:37.942+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:38.171 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:38.169+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:38.915 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:38.913+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:38.918 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:38.917+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:38.917583+0000) 2022-04-03T22:41:39.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:39.157+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:39.918 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:39.917+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:39.917849+0000) 2022-04-03T22:41:39.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:39.962+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:40.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:40.184+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:40.919 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:40.917+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:40.918072+0000) 2022-04-03T22:41:40.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:40.956+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:41.147 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:41.145+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:41.919 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:41.917+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:41.918259+0000) 2022-04-03T22:41:41.994 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:41.993+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:42.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:42.194+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:42.919 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:42.917+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:42.918480+0000) 2022-04-03T22:41:42.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:42.948+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:43.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:43.199+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:43.919 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:43.918+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:43.918694+0000) 2022-04-03T22:41:43.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:43.920+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:44.235 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:44.234+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:44.908 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:44.907+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:44.919 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:44.918+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:44.918952+0000) 2022-04-03T22:41:45.205 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:45.204+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:45.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:45.861+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:45.920 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:45.918+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:45.919204+0000) 2022-04-03T22:41:46.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:46.209+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:46.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:46.830+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:46.920 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:46.919+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:46.919367+0000) 2022-04-03T22:41:47.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:47.165+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:47.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:47.842+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:47.920 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:47.919+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:47.919507+0000) 2022-04-03T22:41:48.117 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:48.116+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:48.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:48.830+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:48.920 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:48.918+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:48.919714+0000) 2022-04-03T22:41:49.118 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:49.116+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:49.453 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:41:49.451+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:41:49.798 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:49.797+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:49.921 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:49.919+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:49.919926+0000) 2022-04-03T22:41:50.088 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:50.087+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:50.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:50.806+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:50.920 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:50.919+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:50.920127+0000) 2022-04-03T22:41:51.040 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:51.039+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:51.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:51.801+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:51.921 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:51.920+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:51.920323+0000) 2022-04-03T22:41:52.017 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:52.016+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:52.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:52.819+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:52.921 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:52.920+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:52.920526+0000) 2022-04-03T22:41:53.029 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:53.028+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:53.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:53.828+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:53.921 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:53.920+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:53.920740+0000) 2022-04-03T22:41:54.053 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:54.052+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:54.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:54.782+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:54.922 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:54.920+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:54.920953+0000) 2022-04-03T22:41:55.089 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:55.088+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:55.736 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:55.735+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:55.922 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:55.920+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:55.921147+0000) 2022-04-03T22:41:56.065 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:56.064+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:56.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:56.734+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:56.922 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:56.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:56.921334+0000) 2022-04-03T22:41:57.052 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:57.051+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:57.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:57.688+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:57.922 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:57.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:57.921501+0000) 2022-04-03T22:41:58.008 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:58.006+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:58.738 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:58.735+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:58.922 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:58.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:58.921663+0000) 2022-04-03T22:41:58.997 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:41:58.994+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:41:59.757 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:41:59.754+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:41:59.922 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:41:59.920+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:41:59.921869+0000) 2022-04-03T22:42:00.005 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:00.002+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:00.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:00.798+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:00.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:00.920+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:00.922085+0000) 2022-04-03T22:42:01.002 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:01.000+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:01.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:01.757+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:01.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:01.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:01.922351+0000) 2022-04-03T22:42:01.972 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:01.970+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:02.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:02.802+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:02.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:02.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:02.922590+0000) 2022-04-03T22:42:02.983 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:02.980+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:03.769 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:03.766+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:03.918 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:42:03.922 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:42:03.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:03.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:03.922780+0000) 2022-04-03T22:42:03.963 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:03.960+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:04.469 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:42:04.466+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:42:04.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:04.800+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:04.924 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:04.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:04.923019+0000) 2022-04-03T22:42:05.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:05.001+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:05.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:05.756+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:05.924 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:05.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:05.923221+0000) 2022-04-03T22:42:06.034 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:06.032+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:06.734 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:06.732+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:06.924 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:06.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:06.923386+0000) 2022-04-03T22:42:07.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:07.065+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:07.738 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:07.736+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:07.924 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:07.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:07.923606+0000) 2022-04-03T22:42:08.079 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:08.076+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:08.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:08.777+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:08.924 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:08.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:08.923814+0000) 2022-04-03T22:42:09.070 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:09.068+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:09.765 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:09.763+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:09.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:09.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:09.924056+0000) 2022-04-03T22:42:10.098 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:10.096+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:10.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:10.753+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:10.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:10.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:10.924336+0000) 2022-04-03T22:42:11.110 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:11.108+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:11.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:11.726+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:11.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:11.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:11.924551+0000) 2022-04-03T22:42:12.146 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:12.144+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:12.734 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:12.732+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:12.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:12.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:12.924719+0000) 2022-04-03T22:42:13.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:13.151+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:13.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:13.779+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:13.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:13.924+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:13.924936+0000) 2022-04-03T22:42:14.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:14.154+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:14.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:14.800+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:14.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:14.924+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:14.925127+0000) 2022-04-03T22:42:15.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:15.187+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:15.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:15.820+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:15.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:15.924+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:15.925342+0000) 2022-04-03T22:42:16.226 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:16.224+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:16.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:16.849+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:16.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:16.924+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:16.925587+0000) 2022-04-03T22:42:17.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:17.226+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:17.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:17.887+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:17.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:17.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:17.925790+0000) 2022-04-03T22:42:18.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:18.266+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:18.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:18.844+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:18.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:18.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:18.925981+0000) 2022-04-03T22:42:19.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:19.270+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:19.485 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:42:19.483+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:42:19.884 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:19.882+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:19.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:19.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:19.926172+0000) 2022-04-03T22:42:20.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:20.280+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:20.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:20.920+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:20.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:20.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:20.926344+0000) 2022-04-03T22:42:21.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:21.321+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:21.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:21.875+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:21.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:21.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:21.926564+0000) 2022-04-03T22:42:22.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:22.284+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:22.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:22.866+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:22.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:22.926+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:22.926835+0000) 2022-04-03T22:42:23.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:23.312+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:23.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:23.820+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:23.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:23.926+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:23.927035+0000) 2022-04-03T22:42:24.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:24.338+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:24.814 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:24.812+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:24.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:24.926+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:24.927203+0000) 2022-04-03T22:42:25.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:25.312+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:25.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:25.779+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:25.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:25.926+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:25.927421+0000) 2022-04-03T22:42:26.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:26.270+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:26.817 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:26.815+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:26.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:26.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:26.927656+0000) 2022-04-03T22:42:27.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:27.267+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:27.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:27.787+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:27.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:27.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:27.927876+0000) 2022-04-03T22:42:28.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:28.287+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:28.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:28.798+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:28.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:28.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:28.928114+0000) 2022-04-03T22:42:29.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:29.326+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:29.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:29.820+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:29.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:29.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:29.928313+0000) 2022-04-03T22:42:30.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:30.308+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:30.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:30.809+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:30.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:30.928+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:30.928561+0000) 2022-04-03T22:42:31.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:31.318+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:31.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:31.792+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:31.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:31.928+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:31.928836+0000) 2022-04-03T22:42:32.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:32.297+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:32.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:32.837+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:32.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:32.928+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:32.929087+0000) 2022-04-03T22:42:33.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:33.278+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:33.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:33.873+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:33.930 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:33.928+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:33.929294+0000) 2022-04-03T22:42:34.133 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:42:34.137 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:42:34.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:34.253+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:34.488 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:42:34.486+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:42:34.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:34.920+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:34.930 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:34.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:34.929552+0000) 2022-04-03T22:42:35.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:35.275+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:35.898 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:35.896+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:35.930 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:35.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:35.929710+0000) 2022-04-03T22:42:36.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:36.313+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:36.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:36.920+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:36.930 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:36.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:36.929900+0000) 2022-04-03T22:42:37.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:37.360+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:37.931 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:37.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:37.930110+0000) 2022-04-03T22:42:37.960 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:37.959+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:38.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:38.368+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:38.931 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:38.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:38.930323+0000) 2022-04-03T22:42:38.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:38.984+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:39.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:39.326+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:39.931 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:39.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:39.930527+0000) 2022-04-03T22:42:39.996 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:39.994+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:40.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:40.306+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:40.931 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:40.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:40.930750+0000) 2022-04-03T22:42:41.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:41.027+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:41.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:41.278+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:41.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:41.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:41.930959+0000) 2022-04-03T22:42:42.040 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:42.039+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:42.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:42.295+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:42.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:42.931+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:42.931132+0000) 2022-04-03T22:42:42.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:42.992+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:43.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:43.262+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:43.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:43.931+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:43.931341+0000) 2022-04-03T22:42:44.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:44.011+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:44.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:44.293+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:44.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:44.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:44.931548+0000) 2022-04-03T22:42:44.994 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:44.993+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:45.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:45.341+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:45.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:45.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:45.931759+0000) 2022-04-03T22:42:45.978 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:45.977+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:46.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:46.348+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:46.933 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:46.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:46.931969+0000) 2022-04-03T22:42:47.022 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:47.021+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:47.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:47.338+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:47.933 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:47.931+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:47.932173+0000) 2022-04-03T22:42:47.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:47.994+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:48.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:48.383+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:48.933 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:48.931+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:48.932320+0000) 2022-04-03T22:42:49.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:49.034+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:49.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:49.382+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:49.504 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:42:49.501+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:42:49.933 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:49.931+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:49.932523+0000) 2022-04-03T22:42:50.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:50.006+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:50.426 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:50.425+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:50.933 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:50.931+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:50.932739+0000) 2022-04-03T22:42:51.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:51.010+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:51.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:51.402+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:51.933 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:51.932+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:51.932952+0000) 2022-04-03T22:42:52.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:52.023+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:52.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:52.403+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:52.934 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:52.932+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:52.933147+0000) 2022-04-03T22:42:52.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:52.975+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:53.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:53.398+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:53.934 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:53.932+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:53.933349+0000) 2022-04-03T22:42:53.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:53.964+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:54.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:54.428+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:54.934 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:54.932+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:54.933550+0000) 2022-04-03T22:42:54.946 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:54.944+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:55.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:55.392+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:55.934 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:55.932+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:55.933720+0000) 2022-04-03T22:42:55.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:55.972+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:56.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:56.351+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:56.934 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:56.932+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:56.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:56.933+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:56.933937+0000) 2022-04-03T22:42:57.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:57.341+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:57.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:57.931+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:57.934 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:57.933+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:57.934139+0000) 2022-04-03T22:42:58.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:58.344+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:58.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:58.933+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:58.934338+0000) 2022-04-03T22:42:58.972 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:58.970+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:42:59.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:42:59.331+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:42:59.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:42:59.933+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:42:59.934522+0000) 2022-04-03T22:43:00.000 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:42:59.998+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:00.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:00.350+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:00.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:00.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:00.934734+0000) 2022-04-03T22:43:00.960 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:00.958+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:01.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:01.308+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:01.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:01.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:01.934983+0000) 2022-04-03T22:43:01.936 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:01.934+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:02.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:02.267+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:02.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:02.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:02.935115+0000) 2022-04-03T22:43:02.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:02.948+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:03.283 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:03.281+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:03.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:03.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:03.935312+0000) 2022-04-03T22:43:03.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:03.935+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:04.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:04.232+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:04.272 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:43:04.276 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:43:04.520 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:43:04.518+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:43:04.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:04.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:04.935532+0000) 2022-04-03T22:43:04.939 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:04.937+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:05.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:05.276+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:05.935 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:05.933+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:05.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:05.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:05.935732+0000) 2022-04-03T22:43:06.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:06.244+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:06.906 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:06.904+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:06.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:06.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:06.935882+0000) 2022-04-03T22:43:07.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:07.261+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:07.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:07.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:07.936084+0000) 2022-04-03T22:43:07.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:07.953+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:08.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:08.239+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:08.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:08.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:08.936277+0000) 2022-04-03T22:43:08.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:08.964+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:09.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:09.275+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:09.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:09.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:09.936455+0000) 2022-04-03T22:43:09.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:09.988+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:10.235 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:10.233+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:10.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:10.936+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:10.936661+0000) 2022-04-03T22:43:10.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:10.996+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:11.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:11.198+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:11.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:11.936+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:11.936839+0000) 2022-04-03T22:43:12.043 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:12.040+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:12.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:12.189+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:12.938 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:12.936+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:12.937071+0000) 2022-04-03T22:43:13.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:13.078+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:13.191 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:13.189+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:13.938 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:13.936+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:13.937241+0000) 2022-04-03T22:43:14.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:14.038+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:14.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:14.216+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:14.938 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:14.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:14.937362+0000) 2022-04-03T22:43:15.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:15.065+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:15.191 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:15.189+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:15.938 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:15.936+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:15.937573+0000) 2022-04-03T22:43:16.093 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:16.091+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:16.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:16.191+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:16.938 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:16.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:16.937762+0000) 2022-04-03T22:43:17.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:17.125+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:17.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:17.235+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:17.939 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:17.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:17.937964+0000) 2022-04-03T22:43:18.146 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:18.144+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:18.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:18.227+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:18.939 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:18.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:18.938174+0000) 2022-04-03T22:43:19.196 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:19.194+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:19.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:19.262+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:19.521 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:43:19.519+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:43:19.939 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:19.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:19.938372+0000) 2022-04-03T22:43:20.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:20.169+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:20.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:20.247+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:20.939 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:20.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:20.938581+0000) 2022-04-03T22:43:21.172 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:21.170+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:21.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:21.221+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:21.939 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:21.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:21.938746+0000) 2022-04-03T22:43:22.137 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:22.135+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:22.183 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:22.180+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:22.939 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:22.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:22.938973+0000) 2022-04-03T22:43:23.134 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:23.132+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:23.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:23.177+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:23.940 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:23.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:23.939129+0000) 2022-04-03T22:43:24.129 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:24.128+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:24.134 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:24.132+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:24.940 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:24.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:24.939333+0000) 2022-04-03T22:43:25.168 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:25.156+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:25.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:25.179+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:25.940 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:25.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:25.939540+0000) 2022-04-03T22:43:26.149 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:26.147+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:26.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:26.198+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:26.940 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:26.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:26.939705+0000) 2022-04-03T22:43:27.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:27.197+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:27.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:27.201+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:27.940 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:27.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:27.939854+0000) 2022-04-03T22:43:28.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:28.165+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:28.184 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:28.182+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:28.941 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:28.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:28.940041+0000) 2022-04-03T22:43:29.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:29.157+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:29.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:29.224+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:29.941 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:29.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:29.940235+0000) 2022-04-03T22:43:30.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:30.162+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:30.233 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:30.232+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:30.941 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:30.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:30.940443+0000) 2022-04-03T22:43:31.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:31.174+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:31.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:31.224+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:31.941 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:31.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:31.940686+0000) 2022-04-03T22:43:32.149 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:32.147+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:32.247 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:32.245+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:32.942 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:32.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:32.940898+0000) 2022-04-03T22:43:33.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:33.104+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:33.290 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:33.288+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:33.942 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:33.940+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:33.941118+0000) 2022-04-03T22:43:34.068 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:34.067+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:34.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:34.314+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:34.433 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:43:34.437 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:43:34.522 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:43:34.519+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:43:34.942 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:34.940+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:34.941330+0000) 2022-04-03T22:43:35.025 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:35.024+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:35.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:35.281+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:35.942 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:35.940+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:35.941516+0000) 2022-04-03T22:43:36.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:36.010+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:36.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:36.308+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:36.942 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:36.940+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:36.941672+0000) 2022-04-03T22:43:37.026 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:37.025+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:37.358 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:37.357+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:37.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:37.940+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:37.941858+0000) 2022-04-03T22:43:38.053 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:38.052+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:38.348 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:38.346+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:38.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:38.941+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:38.941991+0000) 2022-04-03T22:43:39.070 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:39.069+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:39.364 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:39.363+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:39.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:39.941+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:39.942191+0000) 2022-04-03T22:43:40.082 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:40.081+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:40.395 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:40.393+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:40.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:40.941+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:40.942377+0000) 2022-04-03T22:43:41.039 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:41.038+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:41.442 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:41.441+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:41.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:41.941+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:41.942556+0000) 2022-04-03T22:43:42.008 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:42.007+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:42.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:42.400+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:42.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:42.942+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:42.942798+0000) 2022-04-03T22:43:42.976 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:42.975+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:43.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:43.373+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:43.944 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:43.942+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:43.944 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:43.942+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:43.943009+0000) 2022-04-03T22:43:44.337 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:44.335+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:44.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:44.931+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:44.944 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:44.942+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:44.943202+0000) 2022-04-03T22:43:45.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:45.352+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:45.941 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:45.940+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:45.944 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:45.942+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:45.943384+0000) 2022-04-03T22:43:46.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:46.388+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:46.944 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:46.942+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:46.943562+0000) 2022-04-03T22:43:46.983 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:46.982+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:47.397 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:47.395+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:47.944 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:47.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:47.943750+0000) 2022-04-03T22:43:48.019 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:48.017+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:48.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:48.424+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:48.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:48.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:48.943979+0000) 2022-04-03T22:43:49.005 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:49.003+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:49.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:49.375+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:49.538 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:43:49.536+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:43:49.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:49.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:49.944145+0000) 2022-04-03T22:43:49.973 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:49.971+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:50.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:50.405+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:50.937 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:50.935+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:50.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:50.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:50.944359+0000) 2022-04-03T22:43:51.452 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:51.449+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:51.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:51.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:51.944575+0000) 2022-04-03T22:43:51.968 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:51.966+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:52.408 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:52.406+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:52.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:52.944+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:52.944794+0000) 2022-04-03T22:43:52.961 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:52.959+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:53.386 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:53.383+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:53.913 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:53.911+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:53.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:53.944+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:53.945003+0000) 2022-04-03T22:43:54.428 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:54.426+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:54.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:54.870+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:54.946 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:54.944+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:54.945164+0000) 2022-04-03T22:43:55.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:55.433+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:55.906 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:55.904+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:55.946 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:55.944+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:55.945362+0000) 2022-04-03T22:43:56.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:56.481+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:56.902 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:56.900+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:56.946 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:56.945+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:56.945565+0000) 2022-04-03T22:43:57.473 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:57.471+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:57.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:57.945+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:57.945758+0000) 2022-04-03T22:43:57.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:57.946+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:58.509 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:58.507+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:58.930 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:58.928+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:58.946 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:58.945+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:58.945993+0000) 2022-04-03T22:43:59.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:43:59.526+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:43:59.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:43:59.913+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:43:59.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:43:59.945+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:43:59.946190+0000) 2022-04-03T22:44:00.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:00.529+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:00.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:00.876+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:00.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:00.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:00.946398+0000) 2022-04-03T22:44:01.522 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:01.520+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:01.831 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:01.829+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:01.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:01.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:01.946556+0000) 2022-04-03T22:44:02.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:02.551+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:02.814 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:02.812+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:02.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:02.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:02.946755+0000) 2022-04-03T22:44:03.533 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:03.531+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:03.774 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:03.772+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:03.948 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:03.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:03.946990+0000) 2022-04-03T22:44:04.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:04.546+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:04.554 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:44:04.552+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:44:04.578 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:44:04.581 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:44:04.759 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:04.757+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:04.948 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:04.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:04.947221+0000) 2022-04-03T22:44:05.566 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:05.564+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:05.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:05.739+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:05.948 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:05.947+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:05.947423+0000) 2022-04-03T22:44:06.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:06.543+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:06.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:06.712+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:06.948 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:06.947+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:06.947629+0000) 2022-04-03T22:44:07.578 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:07.576+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:07.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:07.690+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:07.949 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:07.947+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:07.947864+0000) 2022-04-03T22:44:08.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:08.529+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:08.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:08.694+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:08.949 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:08.947+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:08.948046+0000) 2022-04-03T22:44:09.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:09.544+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:09.717 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:09.714+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:09.949 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:09.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:09.948214+0000) 2022-04-03T22:44:10.499 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:10.497+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:10.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:10.678+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:10.949 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:10.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:10.948424+0000) 2022-04-03T22:44:11.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:11.533+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:11.727 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:11.725+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:11.949 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:11.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:11.948609+0000) 2022-04-03T22:44:12.525 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:12.523+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:12.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:12.756+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:12.949 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:12.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:12.948836+0000) 2022-04-03T22:44:13.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:13.526+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:13.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:13.731+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:13.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:13.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:13.949041+0000) 2022-04-03T22:44:14.517 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:14.515+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:14.689 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:14.687+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:14.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:14.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:14.949254+0000) 2022-04-03T22:44:15.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:15.483+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:15.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:15.652+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:15.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:15.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:15.949434+0000) 2022-04-03T22:44:16.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:16.453+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:16.662 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:16.661+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:16.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:16.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:16.949637+0000) 2022-04-03T22:44:17.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:17.458+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:17.706 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:17.703+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:17.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:17.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:17.949813+0000) 2022-04-03T22:44:18.507 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:18.505+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:18.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:18.720+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:18.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:18.949+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:18.950019+0000) 2022-04-03T22:44:19.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:19.458+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:19.570 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:44:19.567+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:44:19.691 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:19.689+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:19.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:19.949+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:19.950187+0000) 2022-04-03T22:44:20.424 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:20.423+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:20.694 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:20.692+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:20.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:20.949+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:20.950397+0000) 2022-04-03T22:44:21.400 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:21.398+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:21.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:21.698+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:21.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:21.949+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:21.950582+0000) 2022-04-03T22:44:22.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:22.414+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:22.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:22.724+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:22.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:22.949+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:22.950811+0000) 2022-04-03T22:44:23.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:23.374+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:23.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:23.712+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:23.952 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:23.950+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:23.950976+0000) 2022-04-03T22:44:24.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:24.361+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:24.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:24.675+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:24.952 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:24.950+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:24.951135+0000) 2022-04-03T22:44:25.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:25.400+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:25.656 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:25.654+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:25.952 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:25.950+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:25.951341+0000) 2022-04-03T22:44:26.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:26.389+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:26.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:26.635+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:26.952 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:26.950+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:26.951548+0000) 2022-04-03T22:44:27.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:27.352+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:27.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:27.650+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:27.952 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:27.951+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:27.951751+0000) 2022-04-03T22:44:28.325 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:28.323+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:28.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:28.700+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:28.953 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:28.951+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:28.952002+0000) 2022-04-03T22:44:29.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:29.276+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:29.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:29.694+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:29.953 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:29.951+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:29.952226+0000) 2022-04-03T22:44:30.228 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:30.227+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:30.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:30.739+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:30.953 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:30.951+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:30.952454+0000) 2022-04-03T22:44:31.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:31.246+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:31.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:31.732+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:31.953 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:31.951+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:31.952624+0000) 2022-04-03T22:44:32.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:32.228+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:32.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:32.702+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:32.953 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:32.952+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:32.952857+0000) 2022-04-03T22:44:33.264 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:33.263+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:33.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:33.689+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:33.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:33.952+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:33.953065+0000) 2022-04-03T22:44:34.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:34.281+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:34.586 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:44:34.584+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:44:34.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:34.678+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:34.734 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:44:34.737 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:44:34.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:34.952+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:34.953263+0000) 2022-04-03T22:44:35.289 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:35.288+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:35.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:35.695+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:35.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:35.952+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:35.953440+0000) 2022-04-03T22:44:36.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:36.323+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:36.656 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:36.655+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:36.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:36.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:36.953654+0000) 2022-04-03T22:44:37.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:37.314+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:37.676 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:37.675+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:37.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:37.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:37.953881+0000) 2022-04-03T22:44:38.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:38.339+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:38.656 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:38.655+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:38.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:38.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:38.954057+0000) 2022-04-03T22:44:39.307 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:39.306+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:39.706 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:39.705+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:39.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:39.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:39.954251+0000) 2022-04-03T22:44:40.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:40.287+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:40.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:40.683+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:40.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:40.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:40.954479+0000) 2022-04-03T22:44:41.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:41.325+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:41.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:41.635+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:41.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:41.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:41.954675+0000) 2022-04-03T22:44:42.372 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:42.370+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:42.595 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:42.594+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:42.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:42.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:42.954887+0000) 2022-04-03T22:44:43.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:43.377+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:43.583 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:43.582+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:43.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:43.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:43.955088+0000) 2022-04-03T22:44:44.364 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:44.362+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:44.623 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:44.621+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:44.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:44.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:44.955266+0000) 2022-04-03T22:44:45.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:45.404+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:45.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:45.597+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:45.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:45.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:45.955438+0000) 2022-04-03T22:44:46.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:46.419+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:46.595 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:46.593+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:46.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:46.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:46.955619+0000) 2022-04-03T22:44:47.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:47.460+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:47.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:47.552+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:47.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:47.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:47.955872+0000) 2022-04-03T22:44:48.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:48.459+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:48.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:48.564+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:48.957 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:48.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:48.956084+0000) 2022-04-03T22:44:49.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:49.429+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:49.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:49.552+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:49.589 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:44:49.588+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:44:49.957 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:49.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:49.956285+0000) 2022-04-03T22:44:50.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:50.402+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:50.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:50.557+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:50.957 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:50.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:50.956512+0000) 2022-04-03T22:44:51.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:51.365+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:51.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:51.585+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:51.957 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:51.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:51.956725+0000) 2022-04-03T22:44:52.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:52.397+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:52.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:52.578+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:52.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:52.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:52.956954+0000) 2022-04-03T22:44:53.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:53.367+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:53.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:53.588+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:53.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:53.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:53.957157+0000) 2022-04-03T22:44:54.345 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:54.343+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:54.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:54.638+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:54.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:54.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:54.957339+0000) 2022-04-03T22:44:55.379 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:55.377+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:55.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:55.660+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:55.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:55.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:55.957546+0000) 2022-04-03T22:44:56.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:56.339+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:56.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:56.632+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:56.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:56.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:56.957716+0000) 2022-04-03T22:44:57.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:57.309+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:57.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:57.616+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:57.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:57.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:57.957924+0000) 2022-04-03T22:44:58.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:58.349+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:58.610 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:58.608+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:58.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:58.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:58.958070+0000) 2022-04-03T22:44:59.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:44:59.387+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:44:59.634 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:44:59.632+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:44:59.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:44:59.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:44:59.958284+0000) 2022-04-03T22:45:00.379 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:00.377+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:00.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:00.665+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:00.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:00.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:00.958446+0000) 2022-04-03T22:45:01.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:01.386+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:01.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:01.701+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:01.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:01.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:01.958653+0000) 2022-04-03T22:45:02.360 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:02.359+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:02.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:02.683+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:02.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:02.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:02.958834+0000) 2022-04-03T22:45:03.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:03.316+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:03.661 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:03.658+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:03.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:03.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:03.959021+0000) 2022-04-03T22:45:04.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:04.283+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:04.605 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:45:04.602+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:45:04.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:04.633+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:04.881 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:45:04.885 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:45:04.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:04.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:04.959180+0000) 2022-04-03T22:45:05.241 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:05.239+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:05.625 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:05.622+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:05.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:05.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:05.959333+0000) 2022-04-03T22:45:06.256 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:06.254+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:06.673 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:06.671+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:06.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:06.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:06.959513+0000) 2022-04-03T22:45:07.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:07.266+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:07.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:07.684+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:07.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:07.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:07.959686+0000) 2022-04-03T22:45:08.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:08.246+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:08.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:08.670+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:08.961 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:08.959+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:08.959897+0000) 2022-04-03T22:45:09.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:09.293+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:09.673 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:09.671+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:09.961 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:09.959+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:09.960115+0000) 2022-04-03T22:45:10.270 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:10.269+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:10.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:10.645+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:10.961 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:10.959+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:10.960369+0000) 2022-04-03T22:45:11.301 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:11.299+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:11.664 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:11.662+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:11.961 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:11.959+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:11.960545+0000) 2022-04-03T22:45:12.308 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:12.306+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:12.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:12.697+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:12.961 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:12.959+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:12.960711+0000) 2022-04-03T22:45:13.342 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:13.340+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:13.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:13.670+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:13.962 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:13.960+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:13.960910+0000) 2022-04-03T22:45:14.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:14.367+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:14.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:14.638+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:14.962 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:14.960+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:14.961115+0000) 2022-04-03T22:45:15.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:15.333+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:15.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:15.617+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:15.962 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:15.960+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:15.961346+0000) 2022-04-03T22:45:16.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:16.369+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:16.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:16.636+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:16.962 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:16.960+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:16.961544+0000) 2022-04-03T22:45:17.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:17.415+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:17.604 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:17.602+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:17.962 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:17.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:17.961757+0000) 2022-04-03T22:45:18.448 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:18.446+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:18.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:18.592+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:18.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:18.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:18.961993+0000) 2022-04-03T22:45:19.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:19.423+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:19.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:19.579+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:19.621 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:45:19.619+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:45:19.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:19.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:19.962136+0000) 2022-04-03T22:45:20.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:20.423+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:20.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:20.540+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:20.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:20.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:20.962348+0000) 2022-04-03T22:45:21.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:21.461+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:21.564 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:21.562+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:21.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:21.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:21.962503+0000) 2022-04-03T22:45:22.472 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:22.470+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:22.582 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:22.581+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:22.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:22.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:22.962704+0000) 2022-04-03T22:45:23.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:23.453+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:23.573 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:23.572+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:23.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:23.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:23.962908+0000) 2022-04-03T22:45:24.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:24.487+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:24.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:24.553+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:24.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:24.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:24.963095+0000) 2022-04-03T22:45:25.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:25.496+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:25.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:25.573+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:25.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:25.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:25.963298+0000) 2022-04-03T22:45:26.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:26.534+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:26.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:26.590+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:26.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:26.963+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:26.963535+0000) 2022-04-03T22:45:27.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:27.559+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:27.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:27.578+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:27.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:27.963+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:27.963822+0000) 2022-04-03T22:45:28.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:28.538+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:28.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:28.542+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:28.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:28.963+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:28.964006+0000) 2022-04-03T22:45:29.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:29.493+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:29.558 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:29.557+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:29.965 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:29.963+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:29.964181+0000) 2022-04-03T22:45:30.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:30.491+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:30.595 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:30.592+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:30.965 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:30.964+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:30.964335+0000) 2022-04-03T22:45:31.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:31.502+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:31.609 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:31.608+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:31.965 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:31.964+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:31.964556+0000) 2022-04-03T22:45:32.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:32.493+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:32.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:32.566+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:32.965 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:32.964+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:32.964795+0000) 2022-04-03T22:45:33.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:33.457+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:33.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:33.577+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:33.965 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:33.964+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:33.965001+0000) 2022-04-03T22:45:34.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:34.493+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:34.544 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:34.542+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:34.623 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:45:34.621+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:45:34.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:34.964+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:34.965201+0000) 2022-04-03T22:45:34.984 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:45:34.988 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:45:35.500 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:35.499+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:35.539 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:35.538+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:35.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:35.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:35.965388+0000) 2022-04-03T22:45:36.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:36.512+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:36.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:36.573+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:36.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:36.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:36.965548+0000) 2022-04-03T22:45:37.547 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:37.545+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:37.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:37.550+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:37.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:37.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:37.965758+0000) 2022-04-03T22:45:38.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:38.569+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:38.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:38.597+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:38.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:38.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:38.965978+0000) 2022-04-03T22:45:39.522 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:39.520+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:39.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:39.589+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:39.967 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:39.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:39.966216+0000) 2022-04-03T22:45:40.505 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:40.502+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:40.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:40.540+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:40.967 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:40.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:40.966433+0000) 2022-04-03T22:45:41.484 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:41.481+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:41.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:41.508+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:41.967 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:41.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:41.966652+0000) 2022-04-03T22:45:42.506 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:42.504+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:42.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:42.543+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:42.967 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:42.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:42.966866+0000) 2022-04-03T22:45:43.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:43.457+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:43.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:43.527+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:43.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:43.967+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:43.967058+0000) 2022-04-03T22:45:44.468 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:44.465+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:44.502 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:44.500+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:44.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:44.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:44.967210+0000) 2022-04-03T22:45:45.434 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:45.432+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:45.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:45.547+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:45.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:45.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:45.967418+0000) 2022-04-03T22:45:46.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:46.429+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:46.546 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:46.544+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:46.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:46.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:46.967620+0000) 2022-04-03T22:45:47.438 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:47.436+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:47.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:47.543+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:47.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:47.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:47.967854+0000) 2022-04-03T22:45:48.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:48.415+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:48.523 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:48.521+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:48.969 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:48.967+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:48.968090+0000) 2022-04-03T22:45:49.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:49.459+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:49.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:49.546+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:49.639 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:45:49.636+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:45:49.969 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:49.967+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:49.968294+0000) 2022-04-03T22:45:50.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:50.493+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:50.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:50.596+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:50.969 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:50.967+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:50.968464+0000) 2022-04-03T22:45:51.468 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:51.466+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:51.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:51.566+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:51.969 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:51.967+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:51.968644+0000) 2022-04-03T22:45:52.478 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:52.476+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:52.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:52.614+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:52.969 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:52.967+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:52.968825+0000) 2022-04-03T22:45:53.465 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:53.463+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:53.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:53.585+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:53.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:53.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:53.969035+0000) 2022-04-03T22:45:54.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:54.457+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:54.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:54.549+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:54.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:54.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:54.969239+0000) 2022-04-03T22:45:55.426 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:55.424+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:55.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:55.588+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:55.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:55.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:55.969474+0000) 2022-04-03T22:45:56.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:56.437+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:56.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:56.574+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:56.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:56.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:56.969692+0000) 2022-04-03T22:45:57.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:57.442+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:57.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:57.533+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:57.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:57.969+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:57.969906+0000) 2022-04-03T22:45:58.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:58.405+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:58.533 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:58.532+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:58.971 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:58.969+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:58.970132+0000) 2022-04-03T22:45:59.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:45:59.406+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:45:59.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:45:59.566+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:45:59.971 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:45:59.969+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:45:59.970305+0000) 2022-04-03T22:46:00.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:00.382+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:00.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:00.614+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:00.971 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:00.969+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:00.970518+0000) 2022-04-03T22:46:01.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:01.405+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:01.573 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:01.572+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:01.971 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:01.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:01.970820+0000) 2022-04-03T22:46:02.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:02.391+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:02.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:02.562+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:02.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:02.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:02.970995+0000) 2022-04-03T22:46:03.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:03.438+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:03.523 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:03.521+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:03.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:03.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:03.971185+0000) 2022-04-03T22:46:04.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:04.423+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:04.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:04.534+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:04.654 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:46:04.653+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:46:04.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:04.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:04.971376+0000) 2022-04-03T22:46:05.131 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:46:05.135 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:46:05.449 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:05.447+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:05.486 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:05.485+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:05.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:05.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:05.971602+0000) 2022-04-03T22:46:06.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:06.467+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:06.487 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:06.485+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:06.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:06.971+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:06.971835+0000) 2022-04-03T22:46:07.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:07.510+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:07.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:07.526+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:07.973 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:07.971+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:07.972022+0000) 2022-04-03T22:46:08.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:08.557+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:08.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:08.557+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:08.973 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:08.971+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:08.972189+0000) 2022-04-03T22:46:09.564 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:09.562+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:09.591 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:09.590+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:09.973 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:09.971+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:09.972387+0000) 2022-04-03T22:46:10.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:10.513+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:10.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:10.565+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:10.973 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:10.972+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:10.972597+0000) 2022-04-03T22:46:11.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:11.486+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:11.609 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:11.607+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:11.973 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:11.972+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:11.972854+0000) 2022-04-03T22:46:12.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:12.440+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:12.568 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:12.566+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:12.973 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:12.972+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:12.972999+0000) 2022-04-03T22:46:13.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:13.404+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:13.593 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:13.591+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:13.974 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:13.972+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:13.973179+0000) 2022-04-03T22:46:14.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:14.356+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:14.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:14.578+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:14.974 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:14.972+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:14.973335+0000) 2022-04-03T22:46:15.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:15.364+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:15.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:15.604+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:15.974 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:15.973+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:15.973512+0000) 2022-04-03T22:46:16.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:16.319+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:16.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:16.604+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:16.974 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:16.972+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:16.973712+0000) 2022-04-03T22:46:17.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:17.321+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:17.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:17.583+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:17.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:17.973+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:17.973906+0000) 2022-04-03T22:46:18.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:18.368+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:18.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:18.566+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:18.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:18.973+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:18.974092+0000) 2022-04-03T22:46:19.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:19.402+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:19.533 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:19.532+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:19.657 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:46:19.655+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:46:19.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:19.974+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:19.974292+0000) 2022-04-03T22:46:20.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:20.356+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:20.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:20.577+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:20.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:20.974+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:20.974500+0000) 2022-04-03T22:46:21.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:21.380+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:21.563 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:21.562+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:21.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:21.974+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:21.974718+0000) 2022-04-03T22:46:22.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:22.424+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:22.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:22.604+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:22.976 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:22.974+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:22.974926+0000) 2022-04-03T22:46:23.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:23.396+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:23.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:23.645+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:23.976 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:23.974+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:23.975129+0000) 2022-04-03T22:46:24.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:24.389+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:24.619 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:24.618+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:24.976 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:24.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:24.975329+0000) 2022-04-03T22:46:25.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:25.341+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:25.606 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:25.605+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:25.976 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:25.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:25.975510+0000) 2022-04-03T22:46:26.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:26.384+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:26.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:26.602+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:26.976 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:26.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:26.975687+0000) 2022-04-03T22:46:27.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:27.418+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:27.606 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:27.605+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:27.976 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:27.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:27.975875+0000) 2022-04-03T22:46:28.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:28.384+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:28.586 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:28.584+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:28.977 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:28.974+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:28.976071+0000) 2022-04-03T22:46:29.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:29.386+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:29.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:29.612+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:29.977 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:29.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:29.976225+0000) 2022-04-03T22:46:30.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:30.340+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:30.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:30.580+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:30.977 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:30.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:30.976388+0000) 2022-04-03T22:46:31.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:31.349+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:31.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:31.586+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:31.977 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:31.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:31.976602+0000) 2022-04-03T22:46:32.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:32.311+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:32.550 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:32.549+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:32.977 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:32.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:32.976850+0000) 2022-04-03T22:46:33.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:33.281+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:33.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:33.511+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:33.978 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:33.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:33.976993+0000) 2022-04-03T22:46:34.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:34.310+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:34.502 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:34.500+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:34.673 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:46:34.670+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:46:34.978 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:34.976+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:34.977203+0000) 2022-04-03T22:46:35.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:35.262+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:35.277 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:46:35.281 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:46:35.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:35.474+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:35.978 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:35.976+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:35.977391+0000) 2022-04-03T22:46:36.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:36.266+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:36.469 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:36.466+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:36.978 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:36.976+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:36.977599+0000) 2022-04-03T22:46:37.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:37.271+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:37.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:37.480+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:37.978 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:37.976+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:37.977829+0000) 2022-04-03T22:46:38.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:38.264+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:38.470 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:38.468+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:38.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:38.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:38.977999+0000) 2022-04-03T22:46:39.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:39.248+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:39.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:39.456+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:39.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:39.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:39.978147+0000) 2022-04-03T22:46:40.226 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:40.224+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:40.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:40.487+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:40.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:40.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:40.978351+0000) 2022-04-03T22:46:41.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:41.192+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:41.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:41.536+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:41.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:41.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:41.978554+0000) 2022-04-03T22:46:42.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:42.242+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:42.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:42.521+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:42.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:42.978+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:42.978826+0000) 2022-04-03T22:46:43.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:43.251+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:43.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:43.521+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:43.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:43.978+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:43.979009+0000) 2022-04-03T22:46:44.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:44.225+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:44.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:44.509+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:44.980 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:44.978+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:44.979240+0000) 2022-04-03T22:46:45.214 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:45.211+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:45.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:45.472+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:45.980 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:45.978+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:45.979439+0000) 2022-04-03T22:46:46.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:46.241+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:46.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:46.483+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:46.980 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:46.978+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:46.979635+0000) 2022-04-03T22:46:47.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:47.226+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:47.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:47.477+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:47.980 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:47.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:47.979864+0000) 2022-04-03T22:46:48.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:48.237+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:48.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:48.444+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:48.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:48.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:48.980020+0000) 2022-04-03T22:46:49.195 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:49.193+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:49.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:49.486+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:49.689 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:46:49.687+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:46:49.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:49.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:49.980211+0000) 2022-04-03T22:46:50.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:50.179+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:50.522 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:50.520+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:50.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:50.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:50.980414+0000) 2022-04-03T22:46:51.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:51.208+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:51.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:51.484+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:51.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:51.980+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:51.980574+0000) 2022-04-03T22:46:52.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:52.242+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:52.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:52.501+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:52.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:52.980+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:52.980806+0000) 2022-04-03T22:46:53.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:53.292+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:53.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:53.453+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:53.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:53.980+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:53.981011+0000) 2022-04-03T22:46:54.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:54.324+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:54.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:54.490+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:54.982 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:54.980+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:54.981244+0000) 2022-04-03T22:46:55.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:55.289+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:55.452 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:55.450+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:55.982 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:55.980+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:55.981458+0000) 2022-04-03T22:46:56.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:56.272+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:56.498 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:56.496+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:56.982 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:56.981+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:56.981649+0000) 2022-04-03T22:46:57.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:57.320+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:57.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:57.540+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:57.982 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:57.981+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:57.981868+0000) 2022-04-03T22:46:58.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:58.360+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:58.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:58.499+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:58.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:58.981+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:58.982057+0000) 2022-04-03T22:46:59.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:46:59.360+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:46:59.477 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:46:59.475+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:46:59.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:46:59.981+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:46:59.982282+0000) 2022-04-03T22:47:00.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:00.388+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:00.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:00.441+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:00.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:00.982+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:00.982479+0000) 2022-04-03T22:47:01.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:01.368+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:01.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:01.424+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:01.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:01.982+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:01.982629+0000) 2022-04-03T22:47:02.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:02.392+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:02.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:02.445+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:02.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:02.982+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:02.982851+0000) 2022-04-03T22:47:03.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:03.344+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:03.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:03.456+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:03.984 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:03.982+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:03.983062+0000) 2022-04-03T22:47:04.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:04.326+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:04.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:04.414+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:04.704 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:47:04.703+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:47:04.984 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:04.982+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:04.983232+0000) 2022-04-03T22:47:05.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:05.298+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:05.372 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:05.370+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:05.601 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:47:05.605 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:47:05.984 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:05.983+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:05.983428+0000) 2022-04-03T22:47:06.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:06.312+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:06.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:06.367+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:06.984 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:06.983+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:06.983637+0000) 2022-04-03T22:47:07.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:07.289+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:07.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:07.333+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:07.984 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:07.983+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:07.983877+0000) 2022-04-03T22:47:08.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:08.284+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:08.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:08.350+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:08.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:08.983+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:08.984082+0000) 2022-04-03T22:47:09.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:09.316+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:09.347 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:09.345+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:09.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:09.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:09.984294+0000) 2022-04-03T22:47:10.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:10.322+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:10.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:10.369+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:10.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:10.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:10.984539+0000) 2022-04-03T22:47:11.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:11.311+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:11.400 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:11.399+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:11.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:11.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:11.984698+0000) 2022-04-03T22:47:12.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:12.341+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:12.360 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:12.358+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:12.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:12.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:12.984894+0000) 2022-04-03T22:47:13.325 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:13.323+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:13.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:13.383+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:13.986 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:13.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:13.985104+0000) 2022-04-03T22:47:14.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:14.322+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:14.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:14.405+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:14.986 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:14.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:14.985294+0000) 2022-04-03T22:47:15.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:15.277+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:15.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:15.419+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:15.986 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:15.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:15.985487+0000) 2022-04-03T22:47:16.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:16.271+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:16.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:16.413+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:16.986 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:16.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:16.985704+0000) 2022-04-03T22:47:17.243 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:17.241+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:17.462 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:17.461+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:17.986 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:17.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:17.985912+0000) 2022-04-03T22:47:18.266 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:18.264+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:18.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:18.466+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:18.987 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:18.985+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:18.986126+0000) 2022-04-03T22:47:19.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:19.270+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:19.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:19.443+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:19.720 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:47:19.718+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:47:19.987 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:19.985+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:19.986347+0000) 2022-04-03T22:47:20.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:20.233+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:20.422 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:20.421+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:20.987 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:20.985+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:20.986553+0000) 2022-04-03T22:47:21.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:21.246+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:21.450 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:21.449+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:21.987 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:21.985+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:21.986796+0000) 2022-04-03T22:47:22.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:22.244+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:22.428 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:22.427+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:22.987 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:22.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:22.986957+0000) 2022-04-03T22:47:23.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:23.286+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:23.424 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:23.423+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:23.988 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:23.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:23.987129+0000) 2022-04-03T22:47:24.325 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:24.324+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:24.474 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:24.473+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:24.988 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:24.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:24.987328+0000) 2022-04-03T22:47:25.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:25.356+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:47:25.507 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:25.506+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:25.988 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:25.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:25.987551+0000) 2022-04-03T22:47:26.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:26.382+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:26.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:26.554+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:26.988 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:26.987+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:26.987764+0000) 2022-04-03T22:47:27.338 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:27.337+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:27.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:27.578+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:27.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:27.987+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:27.987963+0000) 2022-04-03T22:47:28.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:28.367+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:28.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:28.565+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:28.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:28.987+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:28.988179+0000) 2022-04-03T22:47:29.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:29.339+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:29.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:29.592+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:29.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:29.987+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:29.988388+0000) 2022-04-03T22:47:30.312 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:30.309+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:30.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:30.636+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:30.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:30.987+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:30.988587+0000) 2022-04-03T22:47:31.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:31.282+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:31.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:31.592+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:31.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:31.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:31.988722+0000) 2022-04-03T22:47:32.303 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:32.301+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:32.600 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:32.598+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:32.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:32.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:32.988914+0000) 2022-04-03T22:47:33.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:33.257+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:33.622 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:33.619+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:33.990 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:33.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:33.989107+0000) 2022-04-03T22:47:34.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:34.290+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:34.608 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:34.606+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:34.736 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:47:34.734+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:47:34.990 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:34.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:34.989342+0000) 2022-04-03T22:47:35.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:35.255+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:35.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:35.610+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:35.832 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:47:35.836 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:47:35.990 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:35.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:35.989613+0000) 2022-04-03T22:47:36.267 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:36.265+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:36.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:36.650+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:36.990 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:36.989+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:36.989803+0000) 2022-04-03T22:47:37.275 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:37.273+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:37.678 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:37.676+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:37.991 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:37.989+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:37.989977+0000) 2022-04-03T22:47:38.252 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:38.250+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:38.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:38.726+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:38.991 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:38.989+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:38.990164+0000) 2022-04-03T22:47:39.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:39.233+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:39.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:39.697+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:39.991 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:39.989+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:39.990414+0000) 2022-04-03T22:47:40.190 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:40.187+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:40.656 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:40.654+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:40.991 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:40.990+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:40.990604+0000) 2022-04-03T22:47:41.208 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:41.206+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:41.691 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:41.689+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:41.991 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:41.990+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:41.990850+0000) 2022-04-03T22:47:42.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:42.157+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:42.687 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:42.685+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:42.992 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:42.990+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:42.991070+0000) 2022-04-03T22:47:43.146 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:43.144+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:43.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:43.695+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:43.992 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:43.990+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:43.991272+0000) 2022-04-03T22:47:44.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:44.129+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:44.724 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:44.722+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:44.992 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:44.991+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:44.991492+0000) 2022-04-03T22:47:45.142 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:45.140+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:45.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:45.707+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:45.992 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:45.991+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:45.991706+0000) 2022-04-03T22:47:46.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:46.130+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:46.664 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:46.662+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:46.992 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:46.991+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:46.991920+0000) 2022-04-03T22:47:47.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:47.152+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:47.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:47.682+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:47.993 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:47.991+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:47.992097+0000) 2022-04-03T22:47:48.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:48.181+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:48.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:48.729+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:48.993 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:48.991+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:48.992302+0000) 2022-04-03T22:47:49.166 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:49.164+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:49.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:49.736+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:49.752 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:47:49.750+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:47:49.993 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:49.992+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:49.992475+0000) 2022-04-03T22:47:50.133 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:50.131+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:50.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:50.765+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:50.993 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:50.992+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:50.992683+0000) 2022-04-03T22:47:51.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:51.180+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:51.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:51.746+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:51.993 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:51.992+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:51.992905+0000) 2022-04-03T22:47:52.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:52.132+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:52.705 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:52.702+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:52.994 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:52.992+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:52.993128+0000) 2022-04-03T22:47:53.109 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:53.107+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:53.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:53.707+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:53.994 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:53.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:53.993354+0000) 2022-04-03T22:47:54.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:54.078+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:54.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:54.732+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:54.994 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:54.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:54.993513+0000) 2022-04-03T22:47:55.072 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:55.070+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:55.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:55.722+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:55.994 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:55.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:55.993723+0000) 2022-04-03T22:47:56.057 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:56.055+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:56.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:56.717+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:56.995 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:56.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:56.993967+0000) 2022-04-03T22:47:57.094 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:57.093+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:57.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:57.719+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:57.995 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:57.994+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:57.994186+0000) 2022-04-03T22:47:58.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:58.123+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:58.735 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:58.732+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:58.995 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:58.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:58.994404+0000) 2022-04-03T22:47:59.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:47:59.126+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:47:59.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:47:59.727+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:47:59.995 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:47:59.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:47:59.994603+0000) 2022-04-03T22:48:00.116 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:00.115+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:00.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:00.713+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:00.995 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:00.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:00.994864+0000) 2022-04-03T22:48:01.072 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:01.070+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:01.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:01.683+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:01.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:01.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:01.995051+0000) 2022-04-03T22:48:02.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:02.065+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:02.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:02.661+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:02.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:02.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:02.995298+0000) 2022-04-03T22:48:03.031 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:03.029+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:03.693 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:03.691+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:03.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:03.994+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:03.995515+0000) 2022-04-03T22:48:04.032 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:04.031+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:04.727 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:04.725+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:04.756 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:48:04.753+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:48:04.987 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:04.985+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:04.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:04.994+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:04.995692+0000) 2022-04-03T22:48:05.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:05.739+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:05.947 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:48:05.951 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:48:05.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:05.994+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:05.995865+0000) 2022-04-03T22:48:05.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:05.996+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:06.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:06.717+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:06.961 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:06.960+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:06.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:06.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:06.996065+0000) 2022-04-03T22:48:07.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:07.670+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:07.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:07.932+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:07.997 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:07.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:07.996270+0000) 2022-04-03T22:48:08.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:08.649+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:08.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:08.937+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:08.997 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:08.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:08.996478+0000) 2022-04-03T22:48:09.670 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:09.668+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:09.941 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:09.939+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:09.997 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:09.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:09.996712+0000) 2022-04-03T22:48:10.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:10.643+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:10.908 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:10.906+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:10.997 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:10.996+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:10.996937+0000) 2022-04-03T22:48:11.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:11.623+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:11.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:11.910+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:11.998 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:11.996+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:11.997108+0000) 2022-04-03T22:48:12.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:12.671+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:12.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:12.867+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:12.998 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:12.996+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:12.997310+0000) 2022-04-03T22:48:13.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:13.661+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:13.913 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:13.911+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:13.998 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:13.996+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:13.997474+0000) 2022-04-03T22:48:14.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:14.712+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:14.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:14.886+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:14.998 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:14.996+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:14.997675+0000) 2022-04-03T22:48:15.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:15.699+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:15.899 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:15.898+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:15.998 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:15.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:15.997876+0000) 2022-04-03T22:48:16.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:16.710+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:16.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:16.923+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:16.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:16.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:16.998052+0000) 2022-04-03T22:48:17.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:17.681+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:17.915 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:17.914+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:17.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:17.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:17.998224+0000) 2022-04-03T22:48:18.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:18.681+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:18.915 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:18.914+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:18.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:18.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:18.998364+0000) 2022-04-03T22:48:19.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:19.651+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:19.772 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:48:19.770+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:48:19.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:19.873+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:19.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:19.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:19.998577+0000) 2022-04-03T22:48:20.655 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:20.654+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:20.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:20.854+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:20.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:20.998+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:20.998810+0000) 2022-04-03T22:48:21.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:21.637+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:21.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:21.880+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:22.000 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:21.998+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:21.999032+0000) 2022-04-03T22:48:22.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:22.640+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:22.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:22.903+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:23.000 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:22.998+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:22.999230+0000) 2022-04-03T22:48:23.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:23.633+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:23.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:23.946+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:24.000 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:23.998+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:23.999406+0000) 2022-04-03T22:48:24.664 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:24.661+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:24.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:24.935+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:25.000 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:24.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:24.999603+0000) 2022-04-03T22:48:25.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:25.618+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:25.905 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:25.903+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:26.000 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:25.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:25.999826+0000) 2022-04-03T22:48:26.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:26.662+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:26.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:26.885+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:27.000 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:26.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:26.999980+0000) 2022-04-03T22:48:27.678 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:27.675+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:27.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:27.845+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:28.001 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:27.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:28.000109+0000) 2022-04-03T22:48:28.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:28.700+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:28.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:28.808+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:29.001 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:28.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:29.000310+0000) 2022-04-03T22:48:29.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:29.714+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:29.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:29.849+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:30.001 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:30.000+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:30.000496+0000) 2022-04-03T22:48:30.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:30.669+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:30.883 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:30.881+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:31.001 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:31.000+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:31.000715+0000) 2022-04-03T22:48:31.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:31.630+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:31.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:31.853+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:32.002 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:32.000+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:32.000925+0000) 2022-04-03T22:48:32.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:32.675+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:32.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:32.886+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:33.002 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:33.000+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:33.001121+0000) 2022-04-03T22:48:33.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:33.665+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:33.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:33.888+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:34.002 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:34.001+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:34.001339+0000) 2022-04-03T22:48:34.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:34.649+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:34.787 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:48:34.786+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:48:34.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:34.894+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:35.002 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:35.001+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:35.001556+0000) 2022-04-03T22:48:35.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:35.644+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:35.939 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:35.937+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:36.002 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:36.001+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:36.001799+0000) 2022-04-03T22:48:36.096 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:48:36.099 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:48:36.659 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:36.657+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:36.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:36.950+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:37.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:37.001+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:37.002029+0000) 2022-04-03T22:48:37.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:37.637+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:37.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:37.982+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:38.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:38.001+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:38.002190+0000) 2022-04-03T22:48:38.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:38.639+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:38.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:38.978+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:39.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:39.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:39.002388+0000) 2022-04-03T22:48:39.678 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:39.675+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:40.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:40.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:40.002591+0000) 2022-04-03T22:48:40.014 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:40.012+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:40.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:40.723+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:41.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:41.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:41.002742+0000) 2022-04-03T22:48:41.044 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:41.042+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:41.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:41.768+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:42.004 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:42.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:42.002955+0000) 2022-04-03T22:48:42.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:42.064+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:42.812 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:42.810+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:43.004 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:43.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:43.003163+0000) 2022-04-03T22:48:43.044 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:43.042+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:43.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:43.853+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:44.004 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:44.001+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:44.003314+0000) 2022-04-03T22:48:44.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:44.079+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:44.884 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:44.882+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:45.004 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:45.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:45.003515+0000) 2022-04-03T22:48:45.084 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:45.082+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:45.874 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:45.872+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:46.004 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:46.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:46.003719+0000) 2022-04-03T22:48:46.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:46.085+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:46.920 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:46.918+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:47.004 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:47.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:47.003943+0000) 2022-04-03T22:48:47.086 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:47.084+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:47.952 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:47.950+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:48.005 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:48.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:48.004116+0000) 2022-04-03T22:48:48.090 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:48.088+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:49.001 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:48.998+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:49.005 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:49.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:49.004320+0000) 2022-04-03T22:48:49.106 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:49.104+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:49.803 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:48:49.801+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:48:49.989 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:49.987+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:50.005 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:50.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:50.004540+0000) 2022-04-03T22:48:50.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:50.124+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:51.005 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:51.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:51.004754+0000) 2022-04-03T22:48:51.018 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:51.016+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:51.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:51.162+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:52.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:52.001+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:52.005 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:52.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:52.005018+0000) 2022-04-03T22:48:52.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:52.153+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:52.978 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:52.975+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:53.006 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:53.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:53.005171+0000) 2022-04-03T22:48:53.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:53.137+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:54.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:54.002+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:54.006 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:54.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:54.005344+0000) 2022-04-03T22:48:54.176 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:54.173+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:54.955 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:54.954+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:55.006 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:55.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:55.005545+0000) 2022-04-03T22:48:55.138 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:55.135+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:55.913 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:55.911+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:56.006 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:56.005+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:56.005810+0000) 2022-04-03T22:48:56.099 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:56.097+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:56.865 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:56.863+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:57.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:57.005+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:57.006015+0000) 2022-04-03T22:48:57.065 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:57.063+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:57.847 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:57.846+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:58.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:58.005+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:58.006224+0000) 2022-04-03T22:48:58.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:58.054+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:58.803 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:58.801+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:48:59.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:48:59.005+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:48:59.006396+0000) 2022-04-03T22:48:59.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:48:59.069+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:48:59.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:48:59.778+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:00.008 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:00.005+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:00.006621+0000) 2022-04-03T22:49:00.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:00.073+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:00.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:00.749+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:01.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:01.006+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:01.006861+0000) 2022-04-03T22:49:01.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:01.033+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:01.712 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:01.711+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:02.008 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:02.006+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:02.007151+0000) 2022-04-03T22:49:02.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:02.012+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:02.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:02.695+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:02.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:02.976+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:03.008 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:03.006+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:03.007317+0000) 2022-04-03T22:49:03.687 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:03.686+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:03.945 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:03.944+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:04.008 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:04.006+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:04.007459+0000) 2022-04-03T22:49:04.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:04.644+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:04.819 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:49:04.817+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:49:04.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:04.984+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:05.008 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:05.007+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:05.007679+0000) 2022-04-03T22:49:05.606 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:05.605+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:05.941 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:05.940+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:06.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:06.007+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:06.007893+0000) 2022-04-03T22:49:06.239 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:49:06.243 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:49:06.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:06.623+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:06.906 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:06.905+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:07.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:07.007+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:07.008131+0000) 2022-04-03T22:49:07.582 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:07.581+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:07.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:07.911+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:08.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:08.007+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:08.008290+0000) 2022-04-03T22:49:08.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:08.544+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:08.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:08.957+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:09.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:09.007+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:09.008552+0000) 2022-04-03T22:49:09.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:09.517+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:09.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:09.951+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:10.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:10.008+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:10.008745+0000) 2022-04-03T22:49:10.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:10.475+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:10.971 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:10.969+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:11.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:11.008+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:11.009042+0000) 2022-04-03T22:49:11.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:11.511+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:11.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:11.929+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:12.010 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:12.008+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:12.009178+0000) 2022-04-03T22:49:12.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:12.512+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:12.968 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:12.966+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:13.010 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:13.008+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:13.009348+0000) 2022-04-03T22:49:13.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:13.512+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:13.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:13.950+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:14.010 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:14.008+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:14.009558+0000) 2022-04-03T22:49:14.537 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:14.536+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:14.994 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:14.993+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:15.010 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:15.009+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:15.009757+0000) 2022-04-03T22:49:15.501 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:15.499+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:16.010 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:16.009+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:16.009993+0000) 2022-04-03T22:49:16.027 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:16.026+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:16.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:16.508+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:17.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:17.009+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:17.010182+0000) 2022-04-03T22:49:17.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:17.037+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:17.468 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:17.467+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:18.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:18.009+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:18.010413+0000) 2022-04-03T22:49:18.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:18.074+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:18.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:18.497+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:19.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:19.010+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:19.010562+0000) 2022-04-03T22:49:19.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:19.093+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:19.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:19.457+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:19.835 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:49:19.833+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:49:20.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:20.010+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:20.010752+0000) 2022-04-03T22:49:20.090 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:20.088+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:20.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:20.408+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:21.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:21.010+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:21.010949+0000) 2022-04-03T22:49:21.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:21.122+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:21.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:21.406+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:22.012 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:22.010+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:22.011154+0000) 2022-04-03T22:49:22.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:22.081+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:22.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:22.362+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:23.012 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:23.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:23.011345+0000) 2022-04-03T22:49:23.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:23.122+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:23.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:23.333+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:24.012 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:24.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:24.011462+0000) 2022-04-03T22:49:24.084 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:24.082+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:24.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:24.367+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:25.012 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:25.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:25.011626+0000) 2022-04-03T22:49:25.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:25.114+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:25.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:25.402+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:26.012 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:26.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:26.011840+0000) 2022-04-03T22:49:26.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:26.123+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:26.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:26.401+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:27.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:27.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:27.012067+0000) 2022-04-03T22:49:27.120 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:27.118+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:27.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:27.447+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:28.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:28.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:28.012252+0000) 2022-04-03T22:49:28.115 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:28.113+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:28.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:28.408+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:29.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:29.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:29.012471+0000) 2022-04-03T22:49:29.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:29.110+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:29.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:29.409+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:30.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:30.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:30.012661+0000) 2022-04-03T22:49:30.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:30.086+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:30.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:30.454+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:31.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:31.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:31.012878+0000) 2022-04-03T22:49:31.101 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:31.099+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:31.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:31.456+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:32.014 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:32.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:32.013069+0000) 2022-04-03T22:49:32.137 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:32.135+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:32.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:32.414+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:33.014 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:33.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:33.013325+0000) 2022-04-03T22:49:33.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:33.137+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:33.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:33.454+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:34.014 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:34.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:34.013536+0000) 2022-04-03T22:49:34.110 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:34.108+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:34.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:34.494+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:34.836 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:49:34.834+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:49:35.014 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:35.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:35.013761+0000) 2022-04-03T22:49:35.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:35.072+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:35.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:35.480+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:36.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:36.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:36.014002+0000) 2022-04-03T22:49:36.030 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:36.028+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:36.386 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:49:36.390 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:49:36.473 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:36.471+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:37.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:37.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:37.014256+0000) 2022-04-03T22:49:37.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:37.060+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:37.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:37.489+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:38.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:38.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:38.014423+0000) 2022-04-03T22:49:38.056 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:38.054+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:38.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:38.479+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:39.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:39.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:39.014626+0000) 2022-04-03T22:49:39.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:39.072+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:39.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:39.512+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:40.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:40.014+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:40.014897+0000) 2022-04-03T22:49:40.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:40.095+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:40.472 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:40.470+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:41.016 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:41.014+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:41.015122+0000) 2022-04-03T22:49:41.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:41.134+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:41.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:41.428+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:42.016 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:42.014+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:42.015285+0000) 2022-04-03T22:49:42.166 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:42.164+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:42.470 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:42.468+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:43.016 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:43.014+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:43.015524+0000) 2022-04-03T22:49:43.214 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:43.212+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:43.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:43.489+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:44.016 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:44.014+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:44.015743+0000) 2022-04-03T22:49:44.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:44.247+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:44.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:44.495+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:45.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:45.015+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:45.015995+0000) 2022-04-03T22:49:45.202 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:45.201+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:45.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:45.529+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:46.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:46.015+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:46.016208+0000) 2022-04-03T22:49:46.243 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:46.242+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:46.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:46.551+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:47.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:47.015+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:47.016426+0000) 2022-04-03T22:49:47.233 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:47.232+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:47.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:47.572+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:48.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:48.015+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:48.016586+0000) 2022-04-03T22:49:48.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:48.271+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:48.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:48.565+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:49.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:49.016+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:49.016763+0000) 2022-04-03T22:49:49.318 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:49.316+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:49.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:49.529+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:49.852 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:49:49.850+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:49:50.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:50.016+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:50.016991+0000) 2022-04-03T22:49:50.366 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:50.365+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:50.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:50.495+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:51.018 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:51.016+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:51.017244+0000) 2022-04-03T22:49:51.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:51.349+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:51.470 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:51.468+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:52.018 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:52.016+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:52.017426+0000) 2022-04-03T22:49:52.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:52.334+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:52.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:52.437+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:53.018 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:53.017+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:53.017620+0000) 2022-04-03T22:49:53.359 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:53.358+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:53.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:53.464+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:54.018 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:54.017+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:54.017796+0000) 2022-04-03T22:49:54.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:54.347+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:54.511 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:54.509+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:55.018 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:55.017+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:55.017960+0000) 2022-04-03T22:49:55.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:55.323+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:55.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:55.533+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:56.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:56.017+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:56.018156+0000) 2022-04-03T22:49:56.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:56.356+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:56.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:56.494+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:57.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:57.017+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:57.018396+0000) 2022-04-03T22:49:57.345 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:57.344+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:57.471 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:57.470+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:58.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:58.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:58.018578+0000) 2022-04-03T22:49:58.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:58.352+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:58.500 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:58.499+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:49:59.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:49:59.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:49:59.018804+0000) 2022-04-03T22:49:59.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:49:59.386+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:49:59.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:49:59.517+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:00.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:00.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:00.018923+0000) 2022-04-03T22:50:00.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:00.406+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:00.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:00.544+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:01.020 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:01.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:01.019141+0000) 2022-04-03T22:50:01.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:01.387+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:01.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:01.533+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:02.020 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:02.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:02.019356+0000) 2022-04-03T22:50:02.387 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:02.386+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:02.486 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:02.485+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:03.020 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:03.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:03.019551+0000) 2022-04-03T22:50:03.402 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:03.401+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:03.453 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:03.452+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:04.020 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:04.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:04.019726+0000) 2022-04-03T22:50:04.441 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:04.440+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:04.472 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:04.471+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:04.868 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:50:04.866+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:50:05.020 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:05.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:05.019982+0000) 2022-04-03T22:50:05.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:05.454+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:05.478 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:05.477+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:06.021 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:06.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:06.020190+0000) 2022-04-03T22:50:06.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:06.412+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:06.436 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:06.435+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:06.540 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:50:06.543 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:50:07.021 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:07.020+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:07.020443+0000) 2022-04-03T22:50:07.382 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:07.381+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:07.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:07.473+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:08.021 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:08.020+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:08.020678+0000) 2022-04-03T22:50:08.428 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:08.427+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:08.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:08.438+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:09.021 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:09.020+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:09.020896+0000) 2022-04-03T22:50:09.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:09.406+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:09.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:09.429+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:10.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:10.020+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:10.021112+0000) 2022-04-03T22:50:10.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:10.392+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:10.424 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:10.423+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:11.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:11.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:11.021328+0000) 2022-04-03T22:50:11.358 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:11.356+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:11.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:11.407+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:12.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:12.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:12.021519+0000) 2022-04-03T22:50:12.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:12.396+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:12.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:12.400+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:13.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:13.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:13.021706+0000) 2022-04-03T22:50:13.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:13.380+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:13.441 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:13.439+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:14.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:14.020+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:14.021842+0000) 2022-04-03T22:50:14.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:14.428+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:14.466 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:14.464+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:15.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:15.020+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:15.022031+0000) 2022-04-03T22:50:15.479 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:15.477+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:15.496 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:15.494+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:16.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:16.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:16.022240+0000) 2022-04-03T22:50:16.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:16.522+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:16.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:16.537+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:17.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:17.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:17.022482+0000) 2022-04-03T22:50:17.510 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:17.508+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:17.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:17.511+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:18.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:18.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:18.022651+0000) 2022-04-03T22:50:18.478 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:18.476+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:18.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:18.529+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:19.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:19.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:19.022871+0000) 2022-04-03T22:50:19.508 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:19.505+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:19.561 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:19.559+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:19.884 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:50:19.881+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:50:20.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:20.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:20.023056+0000) 2022-04-03T22:50:20.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:20.459+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:20.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:20.580+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:21.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:21.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:21.023276+0000) 2022-04-03T22:50:21.457 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:21.455+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:21.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:21.619+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:22.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:22.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:22.023481+0000) 2022-04-03T22:50:22.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:22.446+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:22.598 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:22.596+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:23.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:23.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:23.023732+0000) 2022-04-03T22:50:23.428 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:23.426+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:23.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:23.633+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:24.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:24.023+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:24.023975+0000) 2022-04-03T22:50:24.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:24.385+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:24.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:24.662+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:25.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:25.023+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:25.024193+0000) 2022-04-03T22:50:25.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:25.355+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:25.689 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:25.687+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T22:50:26.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:26.023+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:26.024352+0000) 2022-04-03T22:50:26.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:26.346+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:26.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:26.693+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:27.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:27.023+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:27.024575+0000) 2022-04-03T22:50:27.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:27.358+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:27.684 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:27.682+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:28.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:28.024+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:28.024814+0000) 2022-04-03T22:50:28.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:28.358+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:28.659 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:28.657+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:29.026 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:29.024+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:29.025041+0000) 2022-04-03T22:50:29.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:29.346+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:29.624 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:29.622+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:30.026 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:30.024+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:30.025261+0000) 2022-04-03T22:50:30.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:30.332+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:30.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:30.668+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:31.026 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:31.024+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:31.025485+0000) 2022-04-03T22:50:31.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:31.346+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:31.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:31.710+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:32.026 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:32.025+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:32.025716+0000) 2022-04-03T22:50:32.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:32.298+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:32.683 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:32.681+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:33.027 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:33.025+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:33.025950+0000) 2022-04-03T22:50:33.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:33.275+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:33.683 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:33.682+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:34.027 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:34.025+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:34.026176+0000) 2022-04-03T22:50:34.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:34.278+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:34.707 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:34.705+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:34.900 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:50:34.898+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:50:35.027 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:35.025+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:35.026406+0000) 2022-04-03T22:50:35.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:35.245+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:35.746 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:35.744+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:36.027 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:36.025+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:36.026503+0000) 2022-04-03T22:50:36.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:36.275+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:36.697 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:50:36.701 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:50:36.792 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:36.790+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:37.027 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:37.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:37.026696+0000) 2022-04-03T22:50:37.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:37.323+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:37.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:37.768+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:38.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:38.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:38.026888+0000) 2022-04-03T22:50:38.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:38.316+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:38.740 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:38.738+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:39.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:39.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:39.027158+0000) 2022-04-03T22:50:39.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:39.272+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:39.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:39.749+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:40.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:40.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:40.027375+0000) 2022-04-03T22:50:40.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:40.235+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:40.766 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:40.765+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:41.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:41.027+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:41.027553+0000) 2022-04-03T22:50:41.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:41.235+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:41.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:41.757+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:42.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:42.027+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:42.027735+0000) 2022-04-03T22:50:42.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:42.202+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:42.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:42.743+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:43.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:43.027+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:43.027951+0000) 2022-04-03T22:50:43.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:43.182+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:43.765 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:43.763+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:43.885 DEBUG:teuthology.orchestra.run:got remote process result: 124 2022-04-03T22:50:44.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:44.027+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:44.028119+0000) 2022-04-03T22:50:44.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:44.220+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:44.724 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:44.722+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:45.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:45.027+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:45.028266+0000) 2022-04-03T22:50:45.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:45.199+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:45.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:45.727+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:46.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:46.028+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:46.028458+0000) 2022-04-03T22:50:46.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:46.193+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:46.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:46.771+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:47.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:47.028+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:47.028567+0000) 2022-04-03T22:50:47.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:47.175+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:47.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:47.810+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:48.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:48.028+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:48.028685+0000) 2022-04-03T22:50:48.170 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:48.169+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:48.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:48.807+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:49.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:49.028+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:49.028799+0000) 2022-04-03T22:50:49.145 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:49.144+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:49.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:49.833+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:49.915 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:50:49.914+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:50:50.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:50.028+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:50.029007+0000) 2022-04-03T22:50:50.138 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:50.136+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:50.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:50.828+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:51.030 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:51.028+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:51.029219+0000) 2022-04-03T22:50:51.098 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:51.097+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:51.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:51.878+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:52.030 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:52.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:52.029367+0000) 2022-04-03T22:50:52.061 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:52.060+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:52.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:52.881+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:53.030 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:53.028+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:53.029576+0000) 2022-04-03T22:50:53.110 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:53.109+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:53.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:53.920+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:54.030 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:54.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:54.029806+0000) 2022-04-03T22:50:54.083 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:54.082+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:54.961 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:54.959+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:55.030 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:55.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:55.029988+0000) 2022-04-03T22:50:55.132 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:55.131+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:55.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:55.922+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:56.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:56.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:56.030151+0000) 2022-04-03T22:50:56.151 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:56.150+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:56.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:56.920+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:57.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:57.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:57.030329+0000) 2022-04-03T22:50:57.170 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:57.169+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:57.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:57.886+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:58.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:58.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:58.030432+0000) 2022-04-03T22:50:58.133 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:58.132+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:58.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:58.889+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:50:59.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:50:59.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:50:59.030626+0000) 2022-04-03T22:50:59.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:50:59.179+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:50:59.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:50:59.863+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:00.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:00.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:00.030870+0000) 2022-04-03T22:51:00.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:00.224+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:00.901 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:00.900+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:01.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:01.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:01.031092+0000) 2022-04-03T22:51:01.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:01.191+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:01.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:01.880+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:02.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:02.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:02.031272+0000) 2022-04-03T22:51:02.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:02.241+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:02.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:02.926+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:03.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:03.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:03.031492+0000) 2022-04-03T22:51:03.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:03.192+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:03.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:03.973+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:04.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:04.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:04.031700+0000) 2022-04-03T22:51:04.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:04.158+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:04.931 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:51:04.929+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:51:05.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:05.011+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:05.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:05.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:05.031880+0000) 2022-04-03T22:51:05.127 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:05.126+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:06.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:06.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:06.032060+0000) 2022-04-03T22:51:06.058 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:06.057+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:06.083 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:06.081+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:06.795 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:51:06.799 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:51:07.033 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:07.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:07.032338+0000) 2022-04-03T22:51:07.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:07.074+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:07.120 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:07.119+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:08.033 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:08.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:08.032558+0000) 2022-04-03T22:51:08.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:08.066+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:08.082 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:08.081+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:09.033 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:09.032+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:09.032826+0000) 2022-04-03T22:51:09.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:09.079+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:09.086 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:09.084+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:10.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:10.032+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:10.033062+0000) 2022-04-03T22:51:10.077 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:10.074+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:10.119 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:10.117+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:11.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:11.032+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:11.033275+0000) 2022-04-03T22:51:11.043 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:11.041+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:11.106 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:11.104+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:12.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:12.032+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:12.033458+0000) 2022-04-03T22:51:12.066 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:12.064+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:12.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:12.132+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:13.035 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:13.032+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:13.033672+0000) 2022-04-03T22:51:13.069 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:13.066+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:13.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:13.164+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:14.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:14.033+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:14.033836+0000) 2022-04-03T22:51:14.093 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:14.091+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:14.172 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:14.170+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:15.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:15.033+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:15.034005+0000) 2022-04-03T22:51:15.102 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:15.100+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:15.191 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:15.189+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:16.035 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:16.033+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:16.034187+0000) 2022-04-03T22:51:16.083 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:16.081+0000 7f88d0ac1700 -1 osd.4 129 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:16.192 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:16.190+0000 7fd3b190b700 -1 osd.6 129 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:16.355 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-03T22:51:16.353+0000 7fa31956d700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T22:51:16.370 INFO:tasks.ceph.mds.f.smithi186.stderr: -1> 2022-04-03T22:51:16.353+0000 7fa31956d700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T22:51:16.392 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-03T22:51:17.035 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:17.033+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:17.034408+0000) 2022-04-03T22:51:17.128 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:17.126+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:17.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:17.162+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:18.035 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:18.033+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:18.034631+0000) 2022-04-03T22:51:18.087 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:18.085+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:18.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:18.133+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:19.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:19.034+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:19.034838+0000) 2022-04-03T22:51:19.087 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:19.084+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:19.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:19.115+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:19.947 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:51:19.945+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:51:20.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:20.034+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:20.035051+0000) 2022-04-03T22:51:20.070 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:20.068+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:20.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:20.130+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:21.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:21.034+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:21.035277+0000) 2022-04-03T22:51:21.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:21.086+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:21.093 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:51:21.091+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:51:22.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:22.034+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:22.035437+0000) 2022-04-03T22:51:22.096 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:22.094+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:23.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:23.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:23.035654+0000) 2022-04-03T22:51:23.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:23.119+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:24.037 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:24.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:24.035868+0000) 2022-04-03T22:51:24.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:24.072+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:25.037 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:25.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:25.036087+0000) 2022-04-03T22:51:25.044 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:25.042+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:26.006 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:26.003+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:26.037 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:26.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:26.036299+0000) 2022-04-03T22:51:27.037 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:27.036+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:27.036517+0000) 2022-04-03T22:51:27.051 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:27.050+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:28.037 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:28.036+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:28.036694+0000) 2022-04-03T22:51:28.073 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:28.071+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:29.037 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:29.036+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:29.036901+0000) 2022-04-03T22:51:29.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:29.076+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:30.038 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:30.036+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:30.037039+0000) 2022-04-03T22:51:30.064 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:30.059+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:31.029 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:31.027+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:31.038 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:31.036+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:31.037264+0000) 2022-04-03T22:51:32.038 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:32.037+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:32.037428+0000) 2022-04-03T22:51:32.070 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:32.068+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:33.038 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:33.037+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:33.037655+0000) 2022-04-03T22:51:33.089 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:33.087+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:34.039 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:34.037+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:34.037877+0000) 2022-04-03T22:51:34.087 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:34.084+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:34.963 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:51:34.961+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:51:35.039 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:35.037+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:35.038055+0000) 2022-04-03T22:51:35.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:35.123+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:36.039 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:36.037+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:36.038252+0000) 2022-04-03T22:51:36.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:36.092+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:36.943 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:51:36.947 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:51:37.039 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:37.038+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:37.038486+0000) 2022-04-03T22:51:37.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:37.109+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:38.039 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:38.038+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:38.038702+0000) 2022-04-03T22:51:38.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:38.141+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:39.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:39.038+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:39.038944+0000) 2022-04-03T22:51:39.157 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:39.156+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:40.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:40.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:40.039149+0000) 2022-04-03T22:51:40.204 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:40.203+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:41.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:41.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:41.039360+0000) 2022-04-03T22:51:41.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:41.247+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:42.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:42.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:42.039536+0000) 2022-04-03T22:51:42.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:42.217+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:43.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:43.038+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:43.039798+0000) 2022-04-03T22:51:43.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:43.247+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:43.886 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2022-04-03T22:51:44.041 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:44.038+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:44.040044+0000) 2022-04-03T22:51:44.294 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:44.292+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:45.041 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:45.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:45.040307+0000) 2022-04-03T22:51:45.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:45.266+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:46.041 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:46.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:46.040554+0000) 2022-04-03T22:51:46.274 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:46.273+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:47.041 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:47.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:47.040816+0000) 2022-04-03T22:51:47.256 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:47.254+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:48.042 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:48.040+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:48.041073+0000) 2022-04-03T22:51:48.298 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:48.297+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:49.042 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:49.040+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:49.041323+0000) 2022-04-03T22:51:49.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:49.338+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:49.978 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:51:49.976+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:51:50.042 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:50.040+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:50.041517+0000) 2022-04-03T22:51:50.344 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:50.343+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:51.042 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:51.040+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:51.041746+0000) 2022-04-03T22:51:51.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:51.380+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:52.043 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:52.041+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:52.042005+0000) 2022-04-03T22:51:52.420 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:52.419+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:53.043 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:53.041+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:53.042252+0000) 2022-04-03T22:51:53.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:53.375+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:54.043 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:54.041+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:54.042466+0000) 2022-04-03T22:51:54.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:54.379+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:55.043 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:55.041+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:55.042707+0000) 2022-04-03T22:51:55.359 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:55.358+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:56.046 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:56.042+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:56.042886+0000) 2022-04-03T22:51:56.356 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:56.355+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:57.043 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:57.042+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:57.043003+0000) 2022-04-03T22:51:57.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:57.388+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:58.044 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:58.042+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:58.043203+0000) 2022-04-03T22:51:58.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:58.366+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:51:59.044 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:51:59.042+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:51:59.043436+0000) 2022-04-03T22:51:59.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:51:59.334+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:00.044 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:00.042+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:00.043701+0000) 2022-04-03T22:52:00.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:00.318+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:01.044 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:01.043+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:01.043907+0000) 2022-04-03T22:52:01.366 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:01.365+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:02.045 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:02.043+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:02.044115+0000) 2022-04-03T22:52:02.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:02.365+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:03.045 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:03.043+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:03.044347+0000) 2022-04-03T22:52:03.358 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:03.356+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:04.045 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:04.043+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:04.044538+0000) 2022-04-03T22:52:04.333 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:04.331+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:04.994 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:52:04.992+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:52:05.045 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:05.044+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:05.044747+0000) 2022-04-03T22:52:05.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:05.360+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:06.046 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:06.044+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:06.045021+0000) 2022-04-03T22:52:06.343 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:06.341+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:07.046 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:07.044+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:07.045245+0000) 2022-04-03T22:52:07.114 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:52:07.119 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:52:07.370 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:07.368+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:08.046 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:08.044+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:08.045484+0000) 2022-04-03T22:52:08.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:08.369+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:09.046 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:09.044+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:09.045727+0000) 2022-04-03T22:52:09.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:09.404+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:10.047 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:10.045+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:10.045943+0000) 2022-04-03T22:52:10.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:10.435+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:11.047 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:11.045+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:11.046163+0000) 2022-04-03T22:52:11.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:11.397+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:12.047 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:12.045+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:12.046408+0000) 2022-04-03T22:52:12.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:12.410+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:13.047 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:13.046+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:13.046618+0000) 2022-04-03T22:52:13.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:13.387+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:14.047 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:14.046+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:14.046852+0000) 2022-04-03T22:52:14.379 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:14.377+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:15.048 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:15.046+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:15.047090+0000) 2022-04-03T22:52:15.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:15.419+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:16.048 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:16.046+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:16.047316+0000) 2022-04-03T22:52:16.387 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:16.385+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:17.048 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:17.047+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:17.047565+0000) 2022-04-03T22:52:17.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:17.351+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:18.048 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:18.047+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:18.047839+0000) 2022-04-03T22:52:18.322 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:18.320+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:19.049 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:19.047+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:19.048071+0000) 2022-04-03T22:52:19.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:19.337+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:20.010 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:52:20.008+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:52:20.049 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:20.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:20.048303+0000) 2022-04-03T22:52:20.337 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:20.335+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:21.049 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:21.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:21.048550+0000) 2022-04-03T22:52:21.325 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:21.323+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:22.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:22.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:22.048749+0000) 2022-04-03T22:52:22.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:22.283+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:23.049 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:23.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:23.048902+0000) 2022-04-03T22:52:23.258 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:23.256+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:24.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:24.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:24.049152+0000) 2022-04-03T22:52:24.216 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:24.214+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:25.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:25.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:25.049398+0000) 2022-04-03T22:52:25.225 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:25.223+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:26.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:26.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:26.049646+0000) 2022-04-03T22:52:26.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:26.247+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:27.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:27.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:27.049888+0000) 2022-04-03T22:52:27.266 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:27.265+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:28.051 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:28.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:28.050088+0000) 2022-04-03T22:52:28.270 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:28.269+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:29.051 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:29.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:29.050302+0000) 2022-04-03T22:52:29.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:29.285+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:30.051 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:30.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:30.050545+0000) 2022-04-03T22:52:30.308 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:30.306+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:31.051 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:31.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:31.050745+0000) 2022-04-03T22:52:31.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:31.284+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:32.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:32.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:32.050958+0000) 2022-04-03T22:52:32.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:32.271+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:33.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:33.050+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:33.051158+0000) 2022-04-03T22:52:33.245 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:33.242+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:34.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:34.050+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:34.051373+0000) 2022-04-03T22:52:34.243 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:34.240+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:35.025 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:52:35.023+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:52:35.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:35.050+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:35.051615+0000) 2022-04-03T22:52:35.212 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:35.210+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:36.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:36.050+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:36.051827+0000) 2022-04-03T22:52:36.206 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:36.204+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:37.053 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:37.051+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:37.052059+0000) 2022-04-03T22:52:37.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:37.230+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:37.264 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:52:37.268 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:52:38.053 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:38.051+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:38.052334+0000) 2022-04-03T22:52:38.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:38.254+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:39.054 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:39.051+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:39.052578+0000) 2022-04-03T22:52:39.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:39.303+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:40.053 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:40.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:40.052812+0000) 2022-04-03T22:52:40.291 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:40.289+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:41.054 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:41.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:41.053063+0000) 2022-04-03T22:52:41.264 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:41.262+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:42.054 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:42.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:42.053276+0000) 2022-04-03T22:52:42.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:42.227+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:43.054 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:43.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:43.053551+0000) 2022-04-03T22:52:43.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:43.247+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:43.888 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2022-04-03T22:52:43.889 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2022-04-03T22:52:43.889 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2022-04-03T22:52:44.054 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:44.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:44.053729+0000) 2022-04-03T22:52:44.236 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:44.235+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:44.254 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_cfb8f943163b374162da0d7b0240f267dd46e4e1/qa/tasks/workunit.py", line 148, in task cleanup=cleanup) File "/home/teuthworker/src/git.ceph.com_ceph-c_cfb8f943163b374162da0d7b0240f267dd46e4e1/qa/tasks/workunit.py", line 298, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_cfb8f943163b374162da0d7b0240f267dd46e4e1/qa/tasks/workunit.py", line 427, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi077 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2022-04-03T22:52:44.357 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=825e66f6adcb4fcca0c073544b9a8f59 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_cfb8f943163b374162da0d7b0240f267dd46e4e1/qa/tasks/workunit.py", line 148, in task cleanup=cleanup) File "/home/teuthworker/src/git.ceph.com_ceph-c_cfb8f943163b374162da0d7b0240f267dd46e4e1/qa/tasks/workunit.py", line 298, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_cfb8f943163b374162da0d7b0240f267dd46e4e1/qa/tasks/workunit.py", line 427, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi077 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb8f943163b374162da0d7b0240f267dd46e4e1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2022-04-03T22:52:44.361 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2022-04-03T22:52:44.371 INFO:tasks.mds_thrash:joining mds_thrasher 2022-04-03T22:52:44.372 ERROR:teuthology.run_tasks:Manager failed: mds_thrash Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/run_tasks.py", line 176, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_cfb8f943163b374162da0d7b0240f267dd46e4e1/qa/tasks/mds_thrash.py", line 432, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2022-04-03T22:52:44.380 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2022-04-03T22:52:44.425 INFO:tasks.kclient:Unmounting kernel clients... 2022-04-03T22:52:44.426 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2022-04-03T22:52:44.427 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-04-03T22:52:44.427 DEBUG:teuthology.orchestra.run.smithi077:> sudo umount /home/ubuntu/cephtest/mnt.0 2022-04-03T22:52:44.461 INFO:teuthology.orchestra.run.smithi077.stderr:umount: /home/ubuntu/cephtest/mnt.0: target is busy. 2022-04-03T22:52:44.462 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-04-03T22:52:44.463 DEBUG:tasks.cephfs.kernel_mount:Killing processes on client.0... 2022-04-03T22:52:44.463 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-04-03T22:52:44.464 DEBUG:teuthology.orchestra.run.smithi077:> sudo PATH=/usr/sbin:$PATH lsof ; ps auxf 2022-04-03T22:52:44.642 INFO:teuthology.orchestra.run.smithi077.stderr:lsof: WARNING: can't stat() ceph file system /home/ubuntu/cephtest/mnt.0 2022-04-03T22:52:44.642 INFO:teuthology.orchestra.run.smithi077.stderr: Output information may be incomplete. 2022-04-03T22:52:44.853 INFO:teuthology.orchestra.run.smithi077.stdout:COMMAND PID TID TASKCMD USER FD TYPE DEVICE SIZE/OFF NODE NAME 2022-04-03T22:52:44.854 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.854 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.854 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root txt REG 8,1 1588952 10333 /usr/lib/systemd/systemd 2022-04-03T22:52:44.855 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:44.855 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:44.855 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:44.856 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:44.856 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:44.856 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:44.857 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:44.857 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 312232 8919 /usr/lib64/libpcap.so.1.9.1 2022-04-03T22:52:44.857 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:44.858 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:44.858 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:44.858 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 26192 7582 /usr/lib64/libattr.so.1.1.2448 2022-04-03T22:52:44.859 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:44.859 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:44.859 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:44.860 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:44.860 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:44.860 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:44.861 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:44.861 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:44.861 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:44.861 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:44.862 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:44.862 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 33328 8926 /usr/lib64/libip4tc.so.2.0.0 2022-04-03T22:52:44.862 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:44.863 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:44.863 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 54816 7584 /usr/lib64/libacl.so.1.1.2253 2022-04-03T22:52:44.863 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:44.864 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:44.864 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:44.864 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:44.865 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 100208 9562 /usr/lib64/libkmod.so.2.3.3 2022-04-03T22:52:44.865 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:44.865 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:44.866 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:44.866 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:44.866 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 132888 8225 /usr/lib64/libseccomp.so.2.5.1 2022-04-03T22:52:44.866 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:44.867 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 2732744 10328 /usr/lib/systemd/libsystemd-shared-239.so 2022-04-03T22:52:44.867 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:44.867 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 19983 989 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2022-04-03T22:52:44.868 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root mem REG 8,1 573545 3022 /etc/selinux/targeted/contexts/files/file_contexts.bin 2022-04-03T22:52:44.868 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:44.868 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:44.869 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:44.869 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 3w CHR 1,11 0t0 10 /dev/kmsg 2022-04-03T22:52:44.869 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 4u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:44.870 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 5u a_inode 0,13 0 9293 [signalfd] 2022-04-03T22:52:44.870 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 6r DIR 0,26 0 1 /sys/fs/cgroup/systemd 2022-04-03T22:52:44.870 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 7u a_inode 0,13 0 9293 [timerfd] 2022-04-03T22:52:44.871 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 8u netlink 0t0 17162 KOBJECT_UEVENT 2022-04-03T22:52:44.871 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 9u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:44.871 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 10r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:44.872 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 11u netlink 0t0 17167 AUDIT 2022-04-03T22:52:44.872 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 12r REG 0,21 0 14832 /proc/1/mountinfo 2022-04-03T22:52:44.872 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 13r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:44.872 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 14r REG 0,21 0 4026532078 /proc/swaps 2022-04-03T22:52:44.873 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 15u unix 0xffff88817f203c00 0t0 17163 /run/systemd/private type=STREAM 2022-04-03T22:52:44.873 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 16u a_inode 0,13 0 9293 [timerfd] 2022-04-03T22:52:44.873 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 17r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:44.874 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 18r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:44.874 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 19u unix 0xffff888109079680 0t0 191133 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.874 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 24u unix 0xffff88810da1d280 0t0 186807 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.875 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 41r CHR 10,235 0t0 109 /dev/autofs 2022-04-03T22:52:44.875 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 45u unix 0xffff88810eb17800 0t0 9626 type=DGRAM 2022-04-03T22:52:44.875 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 49u a_inode 0,13 0 9293 [timerfd] 2022-04-03T22:52:44.876 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 52u netlink 0t0 17242 SELINUX 2022-04-03T22:52:44.876 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 55u unix 0xffff888178df9e00 0t0 19707 type=STREAM 2022-04-03T22:52:44.876 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 75u unix 0xffff888103f88000 0t0 14834 /run/systemd/notify type=DGRAM 2022-04-03T22:52:44.876 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 76u unix 0xffff888103f88780 0t0 14835 /run/systemd/cgroups-agent type=DGRAM 2022-04-03T22:52:44.877 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 77u unix 0xffff888103f88f00 0t0 14836 type=DGRAM 2022-04-03T22:52:44.877 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 78u unix 0xffff888103f89680 0t0 14837 type=DGRAM 2022-04-03T22:52:44.877 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 79u unix 0xffff88810eb16900 0t0 9604 /run/rpcbind.sock type=STREAM 2022-04-03T22:52:44.878 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 80u IPv4 9606 0t0 TCP *:sunrpc (LISTEN) 2022-04-03T22:52:44.878 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 81u IPv4 9607 0t0 UDP *:sunrpc 2022-04-03T22:52:44.878 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 82u IPv6 9608 0t0 TCP *:sunrpc (LISTEN) 2022-04-03T22:52:44.879 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 83u IPv6 9609 0t0 UDP *:sunrpc 2022-04-03T22:52:44.879 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 84u unix 0xffff888103f8b480 0t0 14858 /run/systemd/journal/dev-log type=DGRAM 2022-04-03T22:52:44.879 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 85u unix 0xffff88810348da00 0t0 14034 /var/run/.heim_org.h5l.kcm-socket type=STREAM 2022-04-03T22:52:44.879 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 86r FIFO 0,12 0t0 9617 pipe 2022-04-03T22:52:44.880 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 87u unix 0xffff888103f8a580 0t0 14844 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.880 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 88u unix 0xffff888103f8ad00 0t0 14846 /run/systemd/journal/socket type=DGRAM 2022-04-03T22:52:44.880 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 89u FIFO 0,24 0t0 409 /run/dmeventd-server 2022-04-03T22:52:44.881 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 90u FIFO 0,24 0t0 410 /run/dmeventd-client 2022-04-03T22:52:44.881 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 91u FIFO 0,24 0t0 388 /run/initctl 2022-04-03T22:52:44.881 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 92u netlink 0t0 9612 KOBJECT_UEVENT 2022-04-03T22:52:44.882 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 93u unix 0xffff88817f204380 0t0 17170 /run/systemd/coredump type=SEQPACKET 2022-04-03T22:52:44.882 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 96u unix 0xffff888106ed5a00 0t0 62275 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.882 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 97u unix 0xffff8881795be180 0t0 24676 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.882 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 98u unix 0xffff88816f70a580 0t0 22966 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.883 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 99u unix 0xffff888178dfd280 0t0 18860 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.883 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 100u unix 0xffff888178dfe900 0t0 22606 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.883 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 101u unix 0xffff888178dfc380 0t0 22605 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.884 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 102u unix 0xffff8881050ee900 0t0 18709 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.884 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 103u unix 0xffff888178df8000 0t0 18707 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.884 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 104u unix 0xffff88810eb15280 0t0 16126 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.885 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 105u unix 0xffff88810eb14380 0t0 16118 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.885 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 106u unix 0xffff88817f137080 0t0 16117 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.885 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 107u unix 0xffff88810348bc00 0t0 9805 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.886 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 108u unix 0xffff88817f205a00 0t0 9796 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.886 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 109u unix 0xffff88817f200f00 0t0 9795 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.886 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 110u unix 0xffff888178dfa580 0t0 9794 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.886 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 111u unix 0xffff88817f202d00 0t0 9793 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.887 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 112u unix 0xffff88817f130f00 0t0 15987 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.887 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 113u unix 0xffff888103488780 0t0 18609 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.887 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 114u unix 0xffff88810eb17080 0t0 9610 /run/lvm/lvmpolld.socket type=STREAM 2022-04-03T22:52:44.888 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 115u unix 0xffff88817f204b00 0t0 17172 /run/udev/control type=SEQPACKET 2022-04-03T22:52:44.888 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 116u unix 0xffff88810348f800 0t0 14038 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:44.888 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 117u unix 0xffff88810348d280 0t0 14036 /run/avahi-daemon/socket type=STREAM 2022-04-03T22:52:44.889 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1 root 118u unix 0xffff88810348f080 0t0 14040 /var/run/cups/cups.sock type=STREAM 2022-04-03T22:52:44.889 INFO:teuthology.orchestra.run.smithi077.stdout:kthreadd 2 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.889 INFO:teuthology.orchestra.run.smithi077.stdout:kthreadd 2 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.889 INFO:teuthology.orchestra.run.smithi077.stdout:kthreadd 2 root txt unknown /proc/2/exe 2022-04-03T22:52:44.890 INFO:teuthology.orchestra.run.smithi077.stdout:rcu_gp 3 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.890 INFO:teuthology.orchestra.run.smithi077.stdout:rcu_gp 3 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.890 INFO:teuthology.orchestra.run.smithi077.stdout:rcu_gp 3 root txt unknown /proc/3/exe 2022-04-03T22:52:44.891 INFO:teuthology.orchestra.run.smithi077.stdout:rcu_par_g 4 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.891 INFO:teuthology.orchestra.run.smithi077.stdout:rcu_par_g 4 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.891 INFO:teuthology.orchestra.run.smithi077.stdout:rcu_par_g 4 root txt unknown /proc/4/exe 2022-04-03T22:52:44.892 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/0 6 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.892 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/0 6 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.892 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/0 6 root txt unknown /proc/6/exe 2022-04-03T22:52:44.892 INFO:teuthology.orchestra.run.smithi077.stdout:mm_percpu 8 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.893 INFO:teuthology.orchestra.run.smithi077.stdout:mm_percpu 8 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.893 INFO:teuthology.orchestra.run.smithi077.stdout:mm_percpu 8 root txt unknown /proc/8/exe 2022-04-03T22:52:44.893 INFO:teuthology.orchestra.run.smithi077.stdout:rcu_tasks 9 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.894 INFO:teuthology.orchestra.run.smithi077.stdout:rcu_tasks 9 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.894 INFO:teuthology.orchestra.run.smithi077.stdout:rcu_tasks 9 root txt unknown /proc/9/exe 2022-04-03T22:52:44.894 INFO:teuthology.orchestra.run.smithi077.stdout:rcu_tasks 10 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.895 INFO:teuthology.orchestra.run.smithi077.stdout:rcu_tasks 10 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.895 INFO:teuthology.orchestra.run.smithi077.stdout:rcu_tasks 10 root txt unknown /proc/10/exe 2022-04-03T22:52:44.895 INFO:teuthology.orchestra.run.smithi077.stdout:rcu_tasks 11 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.896 INFO:teuthology.orchestra.run.smithi077.stdout:rcu_tasks 11 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.896 INFO:teuthology.orchestra.run.smithi077.stdout:rcu_tasks 11 root txt unknown /proc/11/exe 2022-04-03T22:52:44.896 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 13 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.896 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 13 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.897 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 13 root txt unknown /proc/13/exe 2022-04-03T22:52:44.897 INFO:teuthology.orchestra.run.smithi077.stdout:rcu_preem 14 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.897 INFO:teuthology.orchestra.run.smithi077.stdout:rcu_preem 14 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.898 INFO:teuthology.orchestra.run.smithi077.stdout:rcu_preem 14 root txt unknown /proc/14/exe 2022-04-03T22:52:44.898 INFO:teuthology.orchestra.run.smithi077.stdout:migration 15 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.898 INFO:teuthology.orchestra.run.smithi077.stdout:migration 15 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.899 INFO:teuthology.orchestra.run.smithi077.stdout:migration 15 root txt unknown /proc/15/exe 2022-04-03T22:52:44.899 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/0 16 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.899 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/0 16 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.900 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/0 16 root txt unknown /proc/16/exe 2022-04-03T22:52:44.900 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/1 17 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.900 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/1 17 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.900 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/1 17 root txt unknown /proc/17/exe 2022-04-03T22:52:44.901 INFO:teuthology.orchestra.run.smithi077.stdout:migration 18 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.901 INFO:teuthology.orchestra.run.smithi077.stdout:migration 18 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.901 INFO:teuthology.orchestra.run.smithi077.stdout:migration 18 root txt unknown /proc/18/exe 2022-04-03T22:52:44.902 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 19 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.902 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 19 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.902 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 19 root txt unknown /proc/19/exe 2022-04-03T22:52:44.902 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/1 21 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.903 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/1 21 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.903 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/1 21 root txt unknown /proc/21/exe 2022-04-03T22:52:44.903 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/2 22 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.904 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/2 22 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.904 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/2 22 root txt unknown /proc/22/exe 2022-04-03T22:52:44.904 INFO:teuthology.orchestra.run.smithi077.stdout:migration 23 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.905 INFO:teuthology.orchestra.run.smithi077.stdout:migration 23 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.905 INFO:teuthology.orchestra.run.smithi077.stdout:migration 23 root txt unknown /proc/23/exe 2022-04-03T22:52:44.905 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 24 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.906 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 24 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.906 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 24 root txt unknown /proc/24/exe 2022-04-03T22:52:44.906 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 26 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.907 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 26 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.907 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 26 root txt unknown /proc/26/exe 2022-04-03T22:52:44.907 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/3 27 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.907 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/3 27 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.908 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/3 27 root txt unknown /proc/27/exe 2022-04-03T22:52:44.908 INFO:teuthology.orchestra.run.smithi077.stdout:migration 28 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.908 INFO:teuthology.orchestra.run.smithi077.stdout:migration 28 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.909 INFO:teuthology.orchestra.run.smithi077.stdout:migration 28 root txt unknown /proc/28/exe 2022-04-03T22:52:44.909 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 29 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.909 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 29 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.909 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 29 root txt unknown /proc/29/exe 2022-04-03T22:52:44.910 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/3 31 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.910 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/3 31 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.910 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/3 31 root txt unknown /proc/31/exe 2022-04-03T22:52:44.911 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/4 32 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.911 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/4 32 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.911 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/4 32 root txt unknown /proc/32/exe 2022-04-03T22:52:44.912 INFO:teuthology.orchestra.run.smithi077.stdout:migration 33 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.912 INFO:teuthology.orchestra.run.smithi077.stdout:migration 33 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.912 INFO:teuthology.orchestra.run.smithi077.stdout:migration 33 root txt unknown /proc/33/exe 2022-04-03T22:52:44.913 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 34 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.913 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 34 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.913 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 34 root txt unknown /proc/34/exe 2022-04-03T22:52:44.913 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 36 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.914 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 36 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.914 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 36 root txt unknown /proc/36/exe 2022-04-03T22:52:44.914 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/5 37 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.915 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/5 37 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.915 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/5 37 root txt unknown /proc/37/exe 2022-04-03T22:52:44.915 INFO:teuthology.orchestra.run.smithi077.stdout:migration 38 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.916 INFO:teuthology.orchestra.run.smithi077.stdout:migration 38 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.916 INFO:teuthology.orchestra.run.smithi077.stdout:migration 38 root txt unknown /proc/38/exe 2022-04-03T22:52:44.916 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 39 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.916 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 39 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.917 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 39 root txt unknown /proc/39/exe 2022-04-03T22:52:44.917 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/5 41 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.917 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/5 41 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.918 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/5 41 root txt unknown /proc/41/exe 2022-04-03T22:52:44.918 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/6 42 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.918 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/6 42 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.919 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/6 42 root txt unknown /proc/42/exe 2022-04-03T22:52:44.919 INFO:teuthology.orchestra.run.smithi077.stdout:migration 43 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.919 INFO:teuthology.orchestra.run.smithi077.stdout:migration 43 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.920 INFO:teuthology.orchestra.run.smithi077.stdout:migration 43 root txt unknown /proc/43/exe 2022-04-03T22:52:44.920 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 44 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.920 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 44 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.920 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 44 root txt unknown /proc/44/exe 2022-04-03T22:52:44.921 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/6 46 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.921 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/6 46 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.921 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/6 46 root txt unknown /proc/46/exe 2022-04-03T22:52:44.922 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/7 47 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.922 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/7 47 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.922 INFO:teuthology.orchestra.run.smithi077.stdout:cpuhp/7 47 root txt unknown /proc/47/exe 2022-04-03T22:52:44.922 INFO:teuthology.orchestra.run.smithi077.stdout:migration 48 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.923 INFO:teuthology.orchestra.run.smithi077.stdout:migration 48 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.923 INFO:teuthology.orchestra.run.smithi077.stdout:migration 48 root txt unknown /proc/48/exe 2022-04-03T22:52:44.923 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 49 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.924 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 49 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.924 INFO:teuthology.orchestra.run.smithi077.stdout:ksoftirqd 49 root txt unknown /proc/49/exe 2022-04-03T22:52:44.924 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/7 51 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.925 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/7 51 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.925 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/7 51 root txt unknown /proc/51/exe 2022-04-03T22:52:44.925 INFO:teuthology.orchestra.run.smithi077.stdout:kdevtmpfs 52 root cwd DIR 0,5 3400 1 / 2022-04-03T22:52:44.925 INFO:teuthology.orchestra.run.smithi077.stdout:kdevtmpfs 52 root rtd DIR 0,5 3400 1 / 2022-04-03T22:52:44.926 INFO:teuthology.orchestra.run.smithi077.stdout:kdevtmpfs 52 root txt unknown /proc/52/exe 2022-04-03T22:52:44.926 INFO:teuthology.orchestra.run.smithi077.stdout:netns 53 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.926 INFO:teuthology.orchestra.run.smithi077.stdout:netns 53 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.927 INFO:teuthology.orchestra.run.smithi077.stdout:netns 53 root txt unknown /proc/53/exe 2022-04-03T22:52:44.927 INFO:teuthology.orchestra.run.smithi077.stdout:inet_frag 54 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.927 INFO:teuthology.orchestra.run.smithi077.stdout:inet_frag 54 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.927 INFO:teuthology.orchestra.run.smithi077.stdout:inet_frag 54 root txt unknown /proc/54/exe 2022-04-03T22:52:44.928 INFO:teuthology.orchestra.run.smithi077.stdout:kauditd 55 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.928 INFO:teuthology.orchestra.run.smithi077.stdout:kauditd 55 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.928 INFO:teuthology.orchestra.run.smithi077.stdout:kauditd 55 root txt unknown /proc/55/exe 2022-04-03T22:52:44.929 INFO:teuthology.orchestra.run.smithi077.stdout:khungtask 56 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.929 INFO:teuthology.orchestra.run.smithi077.stdout:khungtask 56 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.929 INFO:teuthology.orchestra.run.smithi077.stdout:khungtask 56 root txt unknown /proc/56/exe 2022-04-03T22:52:44.930 INFO:teuthology.orchestra.run.smithi077.stdout:oom_reape 57 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.930 INFO:teuthology.orchestra.run.smithi077.stdout:oom_reape 57 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.930 INFO:teuthology.orchestra.run.smithi077.stdout:oom_reape 57 root txt unknown /proc/57/exe 2022-04-03T22:52:44.930 INFO:teuthology.orchestra.run.smithi077.stdout:writeback 58 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.931 INFO:teuthology.orchestra.run.smithi077.stdout:writeback 58 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.931 INFO:teuthology.orchestra.run.smithi077.stdout:writeback 58 root txt unknown /proc/58/exe 2022-04-03T22:52:44.931 INFO:teuthology.orchestra.run.smithi077.stdout:kcompactd 59 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.932 INFO:teuthology.orchestra.run.smithi077.stdout:kcompactd 59 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.932 INFO:teuthology.orchestra.run.smithi077.stdout:kcompactd 59 root txt unknown /proc/59/exe 2022-04-03T22:52:44.932 INFO:teuthology.orchestra.run.smithi077.stdout:ksmd 60 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.932 INFO:teuthology.orchestra.run.smithi077.stdout:ksmd 60 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.933 INFO:teuthology.orchestra.run.smithi077.stdout:ksmd 60 root txt unknown /proc/60/exe 2022-04-03T22:52:44.933 INFO:teuthology.orchestra.run.smithi077.stdout:khugepage 61 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.933 INFO:teuthology.orchestra.run.smithi077.stdout:khugepage 61 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.934 INFO:teuthology.orchestra.run.smithi077.stdout:khugepage 61 root txt unknown /proc/61/exe 2022-04-03T22:52:44.934 INFO:teuthology.orchestra.run.smithi077.stdout:cryptd 62 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.934 INFO:teuthology.orchestra.run.smithi077.stdout:cryptd 62 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.935 INFO:teuthology.orchestra.run.smithi077.stdout:cryptd 62 root txt unknown /proc/62/exe 2022-04-03T22:52:44.935 INFO:teuthology.orchestra.run.smithi077.stdout:kintegrit 63 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.935 INFO:teuthology.orchestra.run.smithi077.stdout:kintegrit 63 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.936 INFO:teuthology.orchestra.run.smithi077.stdout:kintegrit 63 root txt unknown /proc/63/exe 2022-04-03T22:52:44.936 INFO:teuthology.orchestra.run.smithi077.stdout:kblockd 64 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.936 INFO:teuthology.orchestra.run.smithi077.stdout:kblockd 64 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.936 INFO:teuthology.orchestra.run.smithi077.stdout:kblockd 64 root txt unknown /proc/64/exe 2022-04-03T22:52:44.937 INFO:teuthology.orchestra.run.smithi077.stdout:blkcg_pun 65 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.937 INFO:teuthology.orchestra.run.smithi077.stdout:blkcg_pun 65 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.938 INFO:teuthology.orchestra.run.smithi077.stdout:blkcg_pun 65 root txt unknown /proc/65/exe 2022-04-03T22:52:44.938 INFO:teuthology.orchestra.run.smithi077.stdout:ata_sff 73 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.938 INFO:teuthology.orchestra.run.smithi077.stdout:ata_sff 73 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.938 INFO:teuthology.orchestra.run.smithi077.stdout:ata_sff 73 root txt unknown /proc/73/exe 2022-04-03T22:52:44.939 INFO:teuthology.orchestra.run.smithi077.stdout:md 74 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.939 INFO:teuthology.orchestra.run.smithi077.stdout:md 74 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.939 INFO:teuthology.orchestra.run.smithi077.stdout:md 74 root txt unknown /proc/74/exe 2022-04-03T22:52:44.940 INFO:teuthology.orchestra.run.smithi077.stdout:edac-poll 75 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.940 INFO:teuthology.orchestra.run.smithi077.stdout:edac-poll 75 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.940 INFO:teuthology.orchestra.run.smithi077.stdout:edac-poll 75 root txt unknown /proc/75/exe 2022-04-03T22:52:44.940 INFO:teuthology.orchestra.run.smithi077.stdout:devfreq_w 76 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.941 INFO:teuthology.orchestra.run.smithi077.stdout:devfreq_w 76 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.941 INFO:teuthology.orchestra.run.smithi077.stdout:devfreq_w 76 root txt unknown /proc/76/exe 2022-04-03T22:52:44.941 INFO:teuthology.orchestra.run.smithi077.stdout:watchdogd 77 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.942 INFO:teuthology.orchestra.run.smithi077.stdout:watchdogd 77 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.942 INFO:teuthology.orchestra.run.smithi077.stdout:watchdogd 77 root txt unknown /proc/77/exe 2022-04-03T22:52:44.942 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/0 79 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.943 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/0 79 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.943 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/0 79 root txt unknown /proc/79/exe 2022-04-03T22:52:44.943 INFO:teuthology.orchestra.run.smithi077.stdout:kswapd0 80 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.944 INFO:teuthology.orchestra.run.smithi077.stdout:kswapd0 80 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.944 INFO:teuthology.orchestra.run.smithi077.stdout:kswapd0 80 root txt unknown /proc/80/exe 2022-04-03T22:52:44.944 INFO:teuthology.orchestra.run.smithi077.stdout:kthrotld 81 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.945 INFO:teuthology.orchestra.run.smithi077.stdout:kthrotld 81 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.945 INFO:teuthology.orchestra.run.smithi077.stdout:kthrotld 81 root txt unknown /proc/81/exe 2022-04-03T22:52:44.945 INFO:teuthology.orchestra.run.smithi077.stdout:acpi_ther 84 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.945 INFO:teuthology.orchestra.run.smithi077.stdout:acpi_ther 84 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.946 INFO:teuthology.orchestra.run.smithi077.stdout:acpi_ther 84 root txt unknown /proc/84/exe 2022-04-03T22:52:44.946 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_0 87 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.946 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_0 87 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.947 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_0 87 root txt unknown /proc/87/exe 2022-04-03T22:52:44.947 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 88 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.947 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 88 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.948 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 88 root txt unknown /proc/88/exe 2022-04-03T22:52:44.948 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_1 89 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.948 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_1 89 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.949 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_1 89 root txt unknown /proc/89/exe 2022-04-03T22:52:44.949 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 90 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.949 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 90 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.950 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 90 root txt unknown /proc/90/exe 2022-04-03T22:52:44.950 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_2 91 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.950 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_2 91 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.950 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_2 91 root txt unknown /proc/91/exe 2022-04-03T22:52:44.951 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 92 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.951 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 92 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.951 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 92 root txt unknown /proc/92/exe 2022-04-03T22:52:44.952 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_3 93 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.952 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_3 93 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.952 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_3 93 root txt unknown /proc/93/exe 2022-04-03T22:52:44.953 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 94 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.953 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 94 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.953 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 94 root txt unknown /proc/94/exe 2022-04-03T22:52:44.954 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_4 98 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.954 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_4 98 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.955 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_4 98 root txt unknown /proc/98/exe 2022-04-03T22:52:44.955 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 99 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.955 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 99 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.956 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 99 root txt unknown /proc/99/exe 2022-04-03T22:52:44.956 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_5 100 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.956 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_5 100 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.957 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_5 100 root txt unknown /proc/100/exe 2022-04-03T22:52:44.957 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 101 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.957 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 101 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.957 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 101 root txt unknown /proc/101/exe 2022-04-03T22:52:44.958 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_6 102 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.958 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_6 102 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.959 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_6 102 root txt unknown /proc/102/exe 2022-04-03T22:52:44.959 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 103 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.959 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 103 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.960 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 103 root txt unknown /proc/103/exe 2022-04-03T22:52:44.960 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_7 104 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.960 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_7 104 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.961 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_7 104 root txt unknown /proc/104/exe 2022-04-03T22:52:44.961 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 105 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.961 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 105 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.961 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 105 root txt unknown /proc/105/exe 2022-04-03T22:52:44.962 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_8 106 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.962 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_8 106 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.962 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_8 106 root txt unknown /proc/106/exe 2022-04-03T22:52:44.963 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 107 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.963 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 107 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.963 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 107 root txt unknown /proc/107/exe 2022-04-03T22:52:44.964 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_9 108 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.964 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_9 108 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.964 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_eh_9 108 root txt unknown /proc/108/exe 2022-04-03T22:52:44.965 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 109 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.965 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 109 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.965 INFO:teuthology.orchestra.run.smithi077.stdout:scsi_tmf_ 109 root txt unknown /proc/109/exe 2022-04-03T22:52:44.966 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/1 117 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.966 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/1 117 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.966 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/1 117 root txt unknown /proc/117/exe 2022-04-03T22:52:44.967 INFO:teuthology.orchestra.run.smithi077.stdout:dm_bufio_ 118 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.967 INFO:teuthology.orchestra.run.smithi077.stdout:dm_bufio_ 118 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.967 INFO:teuthology.orchestra.run.smithi077.stdout:dm_bufio_ 118 root txt unknown /proc/118/exe 2022-04-03T22:52:44.968 INFO:teuthology.orchestra.run.smithi077.stdout:mld 119 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.968 INFO:teuthology.orchestra.run.smithi077.stdout:mld 119 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.968 INFO:teuthology.orchestra.run.smithi077.stdout:mld 119 root txt unknown /proc/119/exe 2022-04-03T22:52:44.969 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/3 120 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.969 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/3 120 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.969 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/3 120 root txt unknown /proc/120/exe 2022-04-03T22:52:44.970 INFO:teuthology.orchestra.run.smithi077.stdout:ipv6_addr 121 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.970 INFO:teuthology.orchestra.run.smithi077.stdout:ipv6_addr 121 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.970 INFO:teuthology.orchestra.run.smithi077.stdout:ipv6_addr 121 root txt unknown /proc/121/exe 2022-04-03T22:52:44.971 INFO:teuthology.orchestra.run.smithi077.stdout:zswap-shr 137 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.971 INFO:teuthology.orchestra.run.smithi077.stdout:zswap-shr 137 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.971 INFO:teuthology.orchestra.run.smithi077.stdout:zswap-shr 137 root txt unknown /proc/137/exe 2022-04-03T22:52:44.972 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/u 209 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.972 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/u 209 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.972 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/u 209 root txt unknown /proc/209/exe 2022-04-03T22:52:44.973 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 248 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.973 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 248 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.973 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 248 root txt unknown /proc/248/exe 2022-04-03T22:52:44.974 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/6 253 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.974 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/6 253 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.974 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/6 253 root txt unknown /proc/253/exe 2022-04-03T22:52:44.975 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 265 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.975 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 265 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.975 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 265 root txt unknown /proc/265/exe 2022-04-03T22:52:44.975 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/5 274 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.976 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/5 274 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.976 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/5 274 root txt unknown /proc/274/exe 2022-04-03T22:52:44.976 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/7 323 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.977 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/7 323 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.977 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/7 323 root txt unknown /proc/323/exe 2022-04-03T22:52:44.977 INFO:teuthology.orchestra.run.smithi077.stdout:nvme-wq 421 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.978 INFO:teuthology.orchestra.run.smithi077.stdout:nvme-wq 421 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.978 INFO:teuthology.orchestra.run.smithi077.stdout:nvme-wq 421 root txt unknown /proc/421/exe 2022-04-03T22:52:44.978 INFO:teuthology.orchestra.run.smithi077.stdout:nvme-rese 422 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.979 INFO:teuthology.orchestra.run.smithi077.stdout:nvme-rese 422 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.979 INFO:teuthology.orchestra.run.smithi077.stdout:nvme-rese 422 root txt unknown /proc/422/exe 2022-04-03T22:52:44.979 INFO:teuthology.orchestra.run.smithi077.stdout:nvme-dele 423 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.980 INFO:teuthology.orchestra.run.smithi077.stdout:nvme-dele 423 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.980 INFO:teuthology.orchestra.run.smithi077.stdout:nvme-dele 423 root txt unknown /proc/423/exe 2022-04-03T22:52:44.980 INFO:teuthology.orchestra.run.smithi077.stdout:ixgbe 430 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.981 INFO:teuthology.orchestra.run.smithi077.stdout:ixgbe 430 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.981 INFO:teuthology.orchestra.run.smithi077.stdout:ixgbe 430 root txt unknown /proc/430/exe 2022-04-03T22:52:44.981 INFO:teuthology.orchestra.run.smithi077.stdout:jbd2/sda1 445 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.982 INFO:teuthology.orchestra.run.smithi077.stdout:jbd2/sda1 445 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.982 INFO:teuthology.orchestra.run.smithi077.stdout:jbd2/sda1 445 root txt unknown /proc/445/exe 2022-04-03T22:52:44.982 INFO:teuthology.orchestra.run.smithi077.stdout:ext4-rsv- 446 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.983 INFO:teuthology.orchestra.run.smithi077.stdout:ext4-rsv- 446 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.983 INFO:teuthology.orchestra.run.smithi077.stdout:ext4-rsv- 446 root txt unknown /proc/446/exe 2022-04-03T22:52:44.983 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:44.984 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:44.984 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root txt REG 8,1 153960 10343 /usr/lib/systemd/systemd-journald 2022-04-03T22:52:44.984 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 0,24 16777216 50 /run/log/journal/545941180e054f2c980bf1100aaa9165/system.journal 2022-04-03T22:52:44.985 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:44.985 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:44.985 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:44.986 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:44.986 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:44.986 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:44.987 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:44.987 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:44.987 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 312232 8919 /usr/lib64/libpcap.so.1.9.1 2022-04-03T22:52:44.988 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:44.988 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:44.988 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:44.989 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:44.989 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:44.989 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:44.989 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 26192 7582 /usr/lib64/libattr.so.1.1.2448 2022-04-03T22:52:44.990 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:44.990 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:44.990 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:44.991 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:44.991 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:44.991 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:44.992 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:44.992 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 132888 8225 /usr/lib64/libseccomp.so.2.5.1 2022-04-03T22:52:44.992 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 33328 8926 /usr/lib64/libip4tc.so.2.0.0 2022-04-03T22:52:44.993 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:44.993 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:44.993 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 54816 7584 /usr/lib64/libacl.so.1.1.2253 2022-04-03T22:52:44.994 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:44.994 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:44.994 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:44.995 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:44.995 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:44.995 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:44.996 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 2732744 10328 /usr/lib/systemd/libsystemd-shared-239.so 2022-04-03T22:52:44.996 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:44.996 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root mem REG 0,24 8 47 /run/systemd/journal/kernel-seqnum 2022-04-03T22:52:44.997 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:44.997 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 1w CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:44.997 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 2w CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:44.997 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 3u unix 0xffff888103f8a580 0t0 14844 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:44.998 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 4u unix 0xffff888103f8ad00 0t0 14846 /run/systemd/journal/socket type=DGRAM 2022-04-03T22:52:44.998 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 5u unix 0xffff888103f8b480 0t0 14858 /run/systemd/journal/dev-log type=DGRAM 2022-04-03T22:52:44.998 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 6u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:44.999 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 7u CHR 1,11 0t0 10 /dev/kmsg 2022-04-03T22:52:44.999 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 8r REG 0,21 0 16424 /proc/sys/kernel/hostname 2022-04-03T22:52:44.999 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 9u a_inode 0,13 0 9293 [signalfd] 2022-04-03T22:52:45.000 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 10u a_inode 0,13 0 9293 [signalfd] 2022-04-03T22:52:45.000 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 11u a_inode 0,13 0 9293 [signalfd] 2022-04-03T22:52:45.000 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 12u unix 0xffff88817f135a00 0t0 12655 type=DGRAM 2022-04-03T22:52:45.001 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 13u a_inode 0,13 0 9293 [timerfd] 2022-04-03T22:52:45.001 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 14u REG 0,24 16777216 50 /run/log/journal/545941180e054f2c980bf1100aaa9165/system.journal 2022-04-03T22:52:45.001 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 15u unix 0xffff888103488780 0t0 18609 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:45.001 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 16u unix 0xffff888178dfc380 0t0 22605 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:45.002 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 17u unix 0xffff88817f130f00 0t0 15987 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:45.002 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 18u unix 0xffff88816f70a580 0t0 22966 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:45.002 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 19u unix 0xffff88817f202d00 0t0 9793 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:45.003 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 20u unix 0xffff888178dfa580 0t0 9794 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:45.003 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 21u unix 0xffff88817f200f00 0t0 9795 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:45.003 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 22u unix 0xffff88817f205a00 0t0 9796 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:45.004 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 23u unix 0xffff888178df8000 0t0 18707 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:45.004 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 24u unix 0xffff888106ed5a00 0t0 62275 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:45.004 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 25u unix 0xffff8881050ee900 0t0 18709 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:45.005 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 26u unix 0xffff88810348bc00 0t0 9805 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:45.005 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 27u unix 0xffff888178dfe900 0t0 22606 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:45.005 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 28u unix 0xffff88817f137080 0t0 16117 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:45.005 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 29u unix 0xffff88810eb14380 0t0 16118 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:45.006 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 30u unix 0xffff88810eb15280 0t0 16126 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:45.006 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 31u unix 0xffff888109079680 0t0 191133 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:45.006 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 32u unix 0xffff8881795be180 0t0 24676 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:45.007 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 33u unix 0xffff888178dfd280 0t0 18860 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:45.007 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-j 541 root 34u unix 0xffff88810da1d280 0t0 186807 /run/systemd/journal/stdout type=STREAM 2022-04-03T22:52:45.007 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.008 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.008 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root txt REG 8,1 426496 10534 /usr/lib/systemd/systemd-udevd 2022-04-03T22:52:45.008 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.008 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.009 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:45.009 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.009 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 538481 263018 /usr/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/modules.alias.bin 2022-04-03T22:52:45.010 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 9143418 72865 /etc/udev/hwdb.bin 2022-04-03T22:52:45.010 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:45.010 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:45.010 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:45.011 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.011 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:45.011 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:45.012 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.012 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.012 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 312232 8919 /usr/lib64/libpcap.so.1.9.1 2022-04-03T22:52:45.013 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.013 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.013 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:45.014 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:45.014 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.014 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 26192 7582 /usr/lib64/libattr.so.1.1.2448 2022-04-03T22:52:45.015 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.015 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:45.015 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.016 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.016 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.016 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.017 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.017 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.017 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 132888 8225 /usr/lib64/libseccomp.so.2.5.1 2022-04-03T22:52:45.017 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 33328 8926 /usr/lib64/libip4tc.so.2.0.0 2022-04-03T22:52:45.018 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.018 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:45.018 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.019 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.019 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.019 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.020 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.020 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.020 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 54816 7584 /usr/lib64/libacl.so.1.1.2253 2022-04-03T22:52:45.021 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 100208 9562 /usr/lib64/libkmod.so.2.3.3 2022-04-03T22:52:45.021 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 2732744 10328 /usr/lib/systemd/libsystemd-shared-239.so 2022-04-03T22:52:45.021 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.022 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 271884 263028 /usr/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/modules.symbols.bin 2022-04-03T22:52:45.022 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 177218 263023 /usr/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/modules.dep.bin 2022-04-03T22:52:45.022 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root DEL REG 8,1 1132 /etc/selinux/targeted/contexts/files/file_contexts.bin 2022-04-03T22:52:45.023 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root mem REG 8,1 11040 263029 /usr/lib/modules/5.17.0-rc6-ceph-g26d8230c8f91/modules.builtin.bin 2022-04-03T22:52:45.023 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root DEL REG 8,1 1133 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2022-04-03T22:52:45.023 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.024 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root 1u unix 0xffff888103488f00 0t0 13917 type=STREAM 2022-04-03T22:52:45.024 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root 2u unix 0xffff888103488f00 0t0 13917 type=STREAM 2022-04-03T22:52:45.024 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root 3u netlink 0t0 9612 KOBJECT_UEVENT 2022-04-03T22:52:45.024 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root 4u unix 0xffff88817f204b00 0t0 17172 /run/udev/control type=SEQPACKET 2022-04-03T22:52:45.025 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root 5u unix 0xffff888103488000 0t0 13924 type=DGRAM 2022-04-03T22:52:45.025 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root 6r REG 8,1 9143418 72865 /etc/udev/hwdb.bin 2022-04-03T22:52:45.025 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root 7u unix 0xffff888103489680 0t0 13943 type=DGRAM 2022-04-03T22:52:45.026 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root 8u unix 0xffff888105c21e00 0t0 59278 type=STREAM 2022-04-03T22:52:45.026 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root 10u unix 0xffff88810348ad00 0t0 13944 type=DGRAM 2022-04-03T22:52:45.026 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root 11r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.027 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root 12u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.027 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root 13u a_inode 0,13 0 9293 [signalfd] 2022-04-03T22:52:45.027 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root 14u a_inode 0,13 0 9293 [timerfd] 2022-04-03T22:52:45.028 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root 15u a_inode 0,13 0 9293 [timerfd] 2022-04-03T22:52:45.028 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root 16r REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:45.028 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-u 576 root 17r REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.029 INFO:teuthology.orchestra.run.smithi077.stdout:ipmi-msgh 602 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.029 INFO:teuthology.orchestra.run.smithi077.stdout:ipmi-msgh 602 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.029 INFO:teuthology.orchestra.run.smithi077.stdout:ipmi-msgh 602 root txt unknown /proc/602/exe 2022-04-03T22:52:45.030 INFO:teuthology.orchestra.run.smithi077.stdout:kipmi0 620 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.030 INFO:teuthology.orchestra.run.smithi077.stdout:kipmi0 620 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.030 INFO:teuthology.orchestra.run.smithi077.stdout:kipmi0 620 root txt unknown /proc/620/exe 2022-04-03T22:52:45.031 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.031 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.031 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc txt REG 8,1 63160 15533 /usr/bin/rpcbind 2022-04-03T22:52:45.032 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.032 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.032 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.033 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.033 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.033 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.034 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.034 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.034 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.035 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:45.035 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.035 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:45.036 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.036 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.036 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.037 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.037 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.037 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.038 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.038 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:45.038 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:45.039 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:45.039 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:45.039 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.040 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.040 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.040 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 371032 9256 /usr/lib64/libtirpc.so.3.0.0 2022-04-03T22:52:45.041 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.041 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.041 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc 1u unix 0xffff88817f133480 0t0 12951 type=STREAM 2022-04-03T22:52:45.042 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc 2u unix 0xffff88817f133480 0t0 12951 type=STREAM 2022-04-03T22:52:45.042 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc 3u unix 0xffff88810eb16900 0t0 9604 /run/rpcbind.sock type=STREAM 2022-04-03T22:52:45.042 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc 4u IPv4 9606 0t0 TCP *:sunrpc (LISTEN) 2022-04-03T22:52:45.043 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc 5u IPv4 9607 0t0 UDP *:sunrpc 2022-04-03T22:52:45.043 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc 6u IPv6 9608 0t0 TCP *:sunrpc (LISTEN) 2022-04-03T22:52:45.043 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc 7u IPv6 9609 0t0 UDP *:sunrpc 2022-04-03T22:52:45.044 INFO:teuthology.orchestra.run.smithi077.stdout:rpcbind 693 rpc 8rW REG 0,24 0 958 /run/rpcbind/rpcbind.lock 2022-04-03T22:52:45.044 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.044 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.044 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root txt REG 8,1 151360 11391 /usr/sbin/auditd 2022-04-03T22:52:45.045 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.045 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.045 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.046 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.046 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.046 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.047 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.047 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:45.047 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.048 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:45.048 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:45.048 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:45.049 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.049 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.049 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:45.049 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:45.050 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.050 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.050 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:45.051 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:45.051 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 133096 7630 /usr/lib64/libauparse.so.0.0.0 2022-04-03T22:52:45.051 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.052 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.052 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.052 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.053 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root 3u netlink 0t0 18416 AUDIT 2022-04-03T22:52:45.053 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root 4u unix 0xffff88817054da00 0t0 147075 type=STREAM 2022-04-03T22:52:45.053 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root 6w REG 8,1 2696240 72880 /var/log/audit/audit.log 2022-04-03T22:52:45.054 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root 7u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.054 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root 8u unix 0xffff8881050ed280 0t0 18419 type=DGRAM 2022-04-03T22:52:45.054 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root 9u unix 0xffff8881050ea580 0t0 18418 type=STREAM 2022-04-03T22:52:45.055 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root 10u unix 0xffff8881050ecb00 0t0 18420 type=STREAM 2022-04-03T22:52:45.055 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root 11u unix 0xffff8881050ec380 0t0 18421 type=STREAM 2022-04-03T22:52:45.055 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 root 12u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.056 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.056 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.056 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root txt REG 8,1 151360 11391 /usr/sbin/auditd 2022-04-03T22:52:45.057 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.057 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.057 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.058 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.058 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.058 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.059 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.059 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:45.059 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.060 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:45.060 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:45.060 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:45.061 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.061 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.061 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:45.062 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:45.062 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.062 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.063 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:45.063 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:45.063 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 133096 7630 /usr/lib64/libauparse.so.0.0.0 2022-04-03T22:52:45.064 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.064 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.064 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.065 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.065 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root 3u netlink 0t0 18416 AUDIT 2022-04-03T22:52:45.065 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root 4u unix 0xffff88817054da00 0t0 147075 type=STREAM 2022-04-03T22:52:45.065 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root 6w REG 8,1 2696240 72880 /var/log/audit/audit.log 2022-04-03T22:52:45.066 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root 7u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.066 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root 8u unix 0xffff8881050ed280 0t0 18419 type=DGRAM 2022-04-03T22:52:45.067 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root 9u unix 0xffff8881050ea580 0t0 18418 type=STREAM 2022-04-03T22:52:45.067 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root 10u unix 0xffff8881050ecb00 0t0 18420 type=STREAM 2022-04-03T22:52:45.067 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root 11u unix 0xffff8881050ec380 0t0 18421 type=STREAM 2022-04-03T22:52:45.068 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 704 auditd root 12u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.068 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.068 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.069 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root txt REG 8,1 151360 11391 /usr/sbin/auditd 2022-04-03T22:52:45.069 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.069 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.070 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.070 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.070 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.071 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.071 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.071 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:45.072 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.072 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:45.072 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:45.072 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:45.073 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.073 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.073 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:45.074 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:45.074 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.074 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.075 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:45.075 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:45.075 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 133096 7630 /usr/lib64/libauparse.so.0.0.0 2022-04-03T22:52:45.076 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.076 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.076 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.077 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.077 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root 3u netlink 0t0 18416 AUDIT 2022-04-03T22:52:45.077 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root 4u unix 0xffff88817054da00 0t0 147075 type=STREAM 2022-04-03T22:52:45.078 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root 6w REG 8,1 2696240 72880 /var/log/audit/audit.log 2022-04-03T22:52:45.078 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root 7u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.078 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root 8u unix 0xffff8881050ed280 0t0 18419 type=DGRAM 2022-04-03T22:52:45.079 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root 9u unix 0xffff8881050ea580 0t0 18418 type=STREAM 2022-04-03T22:52:45.079 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root 10u unix 0xffff8881050ecb00 0t0 18420 type=STREAM 2022-04-03T22:52:45.079 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root 11u unix 0xffff8881050ec380 0t0 18421 type=STREAM 2022-04-03T22:52:45.080 INFO:teuthology.orchestra.run.smithi077.stdout:auditd 703 706 auditd root 12u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.080 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.080 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.080 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root txt REG 8,1 21168 12598 /usr/sbin/sedispatch 2022-04-03T22:52:45.081 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.081 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.081 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.082 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.082 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.082 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.083 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.083 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.083 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.084 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.084 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.084 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.085 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:45.085 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.085 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.086 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.086 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.086 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 133096 7630 /usr/lib64/libauparse.so.0.0.0 2022-04-03T22:52:45.087 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 351464 9341 /usr/lib64/libdbus-1.so.3.19.7 2022-04-03T22:52:45.087 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:45.087 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.087 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root 0u unix 0xffff8881050ead00 0t0 18417 type=STREAM 2022-04-03T22:52:45.088 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.088 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.088 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root 3u unix 0xffff888106ed1680 0t0 23432 type=STREAM 2022-04-03T22:52:45.089 INFO:teuthology.orchestra.run.smithi077.stdout:sedispatc 705 root 4u unix 0xffff888106ed3480 0t0 23450 type=DGRAM 2022-04-03T22:52:45.089 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.089 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.090 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root txt REG 8,1 63768 13419 /usr/sbin/irqbalance 2022-04-03T22:52:45.090 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.090 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.091 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.091 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.091 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.092 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.092 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.092 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.093 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.093 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.093 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.094 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.094 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.094 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root mem REG 8,1 208616 7207 /usr/lib64/libtinfo.so.6.1 2022-04-03T22:52:45.095 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root mem REG 8,1 300104 7199 /usr/lib64/libncursesw.so.6.1 2022-04-03T22:52:45.095 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root mem REG 8,1 51656 8256 /usr/lib64/libnuma.so.1.0.0 2022-04-03T22:52:45.095 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.096 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.096 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:45.096 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.097 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.097 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root 1u unix 0xffff88817f202580 0t0 17396 type=STREAM 2022-04-03T22:52:45.097 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root 2u unix 0xffff88817f202580 0t0 17396 type=STREAM 2022-04-03T22:52:45.098 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root 3u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.098 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root 4u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.098 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 root 5u unix 0xffff88817f206180 0t0 21555 @irqbalance732.sock type=STREAM 2022-04-03T22:52:45.099 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.099 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.099 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root txt REG 8,1 63768 13419 /usr/sbin/irqbalance 2022-04-03T22:52:45.099 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.100 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.100 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.100 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.101 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.101 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.101 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.102 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.102 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.102 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.103 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.103 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.103 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.104 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root mem REG 8,1 208616 7207 /usr/lib64/libtinfo.so.6.1 2022-04-03T22:52:45.104 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root mem REG 8,1 300104 7199 /usr/lib64/libncursesw.so.6.1 2022-04-03T22:52:45.104 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root mem REG 8,1 51656 8256 /usr/lib64/libnuma.so.1.0.0 2022-04-03T22:52:45.105 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.105 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.105 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:45.106 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.106 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.106 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root 1u unix 0xffff88817f202580 0t0 17396 type=STREAM 2022-04-03T22:52:45.107 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root 2u unix 0xffff88817f202580 0t0 17396 type=STREAM 2022-04-03T22:52:45.107 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root 3u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.107 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root 4u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.108 INFO:teuthology.orchestra.run.smithi077.stdout:irqbalanc 732 754 gmain root 5u unix 0xffff88817f206180 0t0 21555 @irqbalance732.sock type=STREAM 2022-04-03T22:52:45.108 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi cwd DIR 8,1 4096 1070 /etc/avahi 2022-04-03T22:52:45.108 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi rtd DIR 8,1 4096 1070 /etc/avahi 2022-04-03T22:52:45.108 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi txt REG 8,1 146208 16118 /usr/sbin/avahi-daemon 2022-04-03T22:52:45.109 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:45.109 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi DEL REG 8,1 72876 /var/lib/sss/mc/initgroups 2022-04-03T22:52:45.109 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi DEL REG 8,1 72875 /var/lib/sss/mc/group 2022-04-03T22:52:45.110 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.110 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.110 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.111 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.111 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.111 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.112 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.112 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.112 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.113 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.113 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.113 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.114 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.114 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.114 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.115 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.115 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.115 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 351464 9341 /usr/lib64/libdbus-1.so.3.19.7 2022-04-03T22:52:45.116 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.116 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.116 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:45.117 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 30264 8700 /usr/lib64/libdaemon.so.0.5.0 2022-04-03T22:52:45.117 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 231504 16117 /usr/lib64/libavahi-core.so.7.0.2 2022-04-03T22:52:45.117 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 54208 10814 /usr/lib64/libavahi-common.so.3.5.3 2022-04-03T22:52:45.118 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.118 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.118 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi 1u unix 0xffff888178dfad00 0t0 19683 type=STREAM 2022-04-03T22:52:45.119 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi 2u unix 0xffff888178dfad00 0t0 19683 type=STREAM 2022-04-03T22:52:45.119 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi 3u unix 0xffff88810348d280 0t0 14036 /run/avahi-daemon/socket type=STREAM 2022-04-03T22:52:45.119 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi 4r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.119 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi 5r REG 8,1 6940392 72875 /var/lib/sss/mc/group (deleted) 2022-04-03T22:52:45.120 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi 6u unix 0xffff888103595a00 0t0 18693 type=DGRAM 2022-04-03T22:52:45.120 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi 7r REG 8,1 11567160 72876 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:45.120 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi 8u unix 0xffff888103596900 0t0 18700 type=STREAM 2022-04-03T22:52:45.121 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi 9r FIFO 0,12 0t0 18703 pipe 2022-04-03T22:52:45.121 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi 10w FIFO 0,12 0t0 18703 pipe 2022-04-03T22:52:45.121 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi 11r FIFO 0,12 0t0 18704 pipe 2022-04-03T22:52:45.122 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi 12w FIFO 0,12 0t0 18704 pipe 2022-04-03T22:52:45.122 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi 13u unix 0xffff888103597080 0t0 18705 type=STREAM 2022-04-03T22:52:45.136 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi 14r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.136 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi 15u IPv4 21564 0t0 UDP *:mdns 2022-04-03T22:52:45.137 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi 16u IPv6 21565 0t0 UDP *:mdns 2022-04-03T22:52:45.137 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi 17u IPv4 21566 0t0 UDP *:56581 2022-04-03T22:52:45.137 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi 18u IPv6 21567 0t0 UDP *:38337 2022-04-03T22:52:45.138 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 733 avahi 19u netlink 0t0 21568 ROUTE 2022-04-03T22:52:45.138 INFO:teuthology.orchestra.run.smithi077.stdout:rpciod 734 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.138 INFO:teuthology.orchestra.run.smithi077.stdout:rpciod 734 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.139 INFO:teuthology.orchestra.run.smithi077.stdout:rpciod 734 root txt unknown /proc/734/exe 2022-04-03T22:52:45.139 INFO:teuthology.orchestra.run.smithi077.stdout:xprtiod 735 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.139 INFO:teuthology.orchestra.run.smithi077.stdout:xprtiod 735 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.140 INFO:teuthology.orchestra.run.smithi077.stdout:xprtiod 735 root txt unknown /proc/735/exe 2022-04-03T22:52:45.140 INFO:teuthology.orchestra.run.smithi077.stdout:lsmd 736 libstoragemgmt cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.140 INFO:teuthology.orchestra.run.smithi077.stdout:lsmd 736 libstoragemgmt rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.140 INFO:teuthology.orchestra.run.smithi077.stdout:lsmd 736 libstoragemgmt txt REG 8,1 25280 10963 /usr/bin/lsmd 2022-04-03T22:52:45.141 INFO:teuthology.orchestra.run.smithi077.stdout:lsmd 736 libstoragemgmt mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.141 INFO:teuthology.orchestra.run.smithi077.stdout:lsmd 736 libstoragemgmt DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.141 INFO:teuthology.orchestra.run.smithi077.stdout:lsmd 736 libstoragemgmt mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.142 INFO:teuthology.orchestra.run.smithi077.stdout:lsmd 736 libstoragemgmt mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.142 INFO:teuthology.orchestra.run.smithi077.stdout:lsmd 736 libstoragemgmt mem REG 8,1 54680 8427 /usr/lib64/libconfig.so.9.2.0 2022-04-03T22:52:45.142 INFO:teuthology.orchestra.run.smithi077.stdout:lsmd 736 libstoragemgmt mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.143 INFO:teuthology.orchestra.run.smithi077.stdout:lsmd 736 libstoragemgmt 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.143 INFO:teuthology.orchestra.run.smithi077.stdout:lsmd 736 libstoragemgmt 1u unix 0xffff88817f203480 0t0 21550 type=STREAM 2022-04-03T22:52:45.143 INFO:teuthology.orchestra.run.smithi077.stdout:lsmd 736 libstoragemgmt 2u unix 0xffff88817f205280 0t0 21551 type=STREAM 2022-04-03T22:52:45.144 INFO:teuthology.orchestra.run.smithi077.stdout:lsmd 736 libstoragemgmt 3r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.144 INFO:teuthology.orchestra.run.smithi077.stdout:lsmd 736 libstoragemgmt 4u unix 0xffff888103593c00 0t0 18679 /var/run/lsm/ipc/sim type=STREAM 2022-04-03T22:52:45.144 INFO:teuthology.orchestra.run.smithi077.stdout:lsmd 736 libstoragemgmt 5u unix 0xffff888103595280 0t0 18680 /var/run/lsm/ipc/simc type=STREAM 2022-04-03T22:52:45.145 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.145 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.145 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root txt REG 8,1 718504 13416 /usr/sbin/smartd 2022-04-03T22:52:45.146 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.146 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.146 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.147 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.147 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.147 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.147 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.148 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.148 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.148 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.149 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.149 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.149 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.150 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.150 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.150 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:45.150 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.151 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.151 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:45.151 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.152 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.152 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.152 INFO:teuthology.orchestra.run.smithi077.stdout:smartd 749 root 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.153 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.153 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.153 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root txt REG 8,1 91720 12838 /usr/sbin/sssd 2022-04-03T22:52:45.153 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:45.154 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root DEL REG 8,1 72876 /var/lib/sss/mc/initgroups 2022-04-03T22:52:45.154 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 1609728 2793 /var/lib/sss/db/timestamps_implicit_files.ldb 2022-04-03T22:52:45.154 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 3035136 2792 /var/lib/sss/db/cache_implicit_files.ldb 2022-04-03T22:52:45.155 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:45.155 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.155 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.156 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.156 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 7496 44161 /usr/lib64/ldb/modules/ldb/tdb.so 2022-04-03T22:52:45.156 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 11776 44160 /usr/lib64/ldb/modules/ldb/skel.so 2022-04-03T22:52:45.157 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 15856 44159 /usr/lib64/ldb/modules/ldb/server_sort.so 2022-04-03T22:52:45.157 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 11952 44158 /usr/lib64/ldb/modules/ldb/sample.so 2022-04-03T22:52:45.157 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 15912 44157 /usr/lib64/ldb/modules/ldb/rdn_name.so 2022-04-03T22:52:45.158 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 15968 44156 /usr/lib64/ldb/modules/ldb/paged_searches.so 2022-04-03T22:52:45.158 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 70952 45685 /usr/lib64/ldb/modules/ldb/memberof.so 2022-04-03T22:52:45.158 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 7504 44155 /usr/lib64/ldb/modules/ldb/mdb.so 2022-04-03T22:52:45.159 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 91808 8746 /usr/lib64/liblmdb.so.0.0.0 2022-04-03T22:52:45.159 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 12360 44148 /usr/lib64/ldb/libldb-tdb-err-map.so 2022-04-03T22:52:45.159 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 95440 44146 /usr/lib64/ldb/libldb-key-value.so 2022-04-03T22:52:45.160 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 24704 44147 /usr/lib64/ldb/libldb-mdb-int.so 2022-04-03T22:52:45.160 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 24920 44149 /usr/lib64/ldb/libldb-tdb-int.so 2022-04-03T22:52:45.160 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 7608 44154 /usr/lib64/ldb/modules/ldb/ldb.so 2022-04-03T22:52:45.160 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:45.161 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:45.161 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:45.161 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:45.162 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:45.162 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:45.162 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:45.163 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 125328 8879 /usr/lib64/libsasl2.so.3.0.0 2022-04-03T22:52:45.163 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.163 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 67136 8940 /usr/lib64/liblber-2.4.so.2.10.9 2022-04-03T22:52:45.163 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 325184 8942 /usr/lib64/libldap-2.4.so.2.10.9 2022-04-03T22:52:45.164 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 28808 44153 /usr/lib64/ldb/modules/ldb/ldap.so 2022-04-03T22:52:45.164 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 15888 44152 /usr/lib64/ldb/modules/ldb/asq.so 2022-04-03T22:52:45.164 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.165 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.165 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.165 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.165 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.166 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.166 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.166 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.167 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.167 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.167 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.168 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.168 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.168 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.169 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.169 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.169 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.170 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.170 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.170 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 16160 8221 /usr/lib64/libpath_utils.so.1.0.1 2022-04-03T22:52:45.171 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.171 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.171 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.172 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.172 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.172 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 351464 9341 /usr/lib64/libdbus-1.so.3.19.7 2022-04-03T22:52:45.173 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 87648 7444 /usr/lib64/libtalloc.so.2.3.1 2022-04-03T22:52:45.173 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 83488 7526 /usr/lib64/libtevent.so.0.10.2 2022-04-03T22:52:45.173 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 16216 7635 /usr/lib64/libdhash.so.1.1.0 2022-04-03T22:52:45.174 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 266008 35861 /usr/lib64/sssd/libsss_sbus.so 2022-04-03T22:52:45.174 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 152328 35857 /usr/lib64/sssd/libsss_iface.so 2022-04-03T22:52:45.174 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.174 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 20664 35855 /usr/lib64/sssd/libsss_debug.so 2022-04-03T22:52:45.175 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.175 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 24208 35853 /usr/lib64/sssd/libsss_child.so 2022-04-03T22:52:45.175 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 28832 35854 /usr/lib64/sssd/libsss_crypt.so 2022-04-03T22:52:45.176 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 20368 35852 /usr/lib64/sssd/libsss_cert.so 2022-04-03T22:52:45.176 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 55560 7633 /usr/lib64/libcollection.so.4.1.1 2022-04-03T22:52:45.176 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 12024 7637 /usr/lib64/libref_array.so.1.2.1 2022-04-03T22:52:45.177 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 11848 7624 /usr/lib64/libbasicobjects.so.0.1.0 2022-04-03T22:52:45.177 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 117048 8223 /usr/lib64/libini_config.so.5.2.1 2022-04-03T22:52:45.177 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.178 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 100320 7532 /usr/lib64/libtdb.so.1.4.3 2022-04-03T22:52:45.178 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.178 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 220216 10551 /usr/lib64/libldb.so.2.2.0 2022-04-03T22:52:45.178 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 54688 7381 /usr/lib64/libpopt.so.0.0.1 2022-04-03T22:52:45.179 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.179 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 588576 35865 /usr/lib64/sssd/libsss_util.so 2022-04-03T22:52:45.179 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:45.180 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:45.180 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:45.180 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.180 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:45.181 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:45.181 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:45.181 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:45.182 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:45.182 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:45.182 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:45.183 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:45.183 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:45.183 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:45.183 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 1286144 2754 /var/lib/sss/db/config.ldb 2022-04-03T22:52:45.184 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:45.184 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.184 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root 0r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.185 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root 1u unix 0xffff88810348e180 0t0 14157 type=STREAM 2022-04-03T22:52:45.185 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root 2u unix 0xffff88810348e180 0t0 14157 type=STREAM 2022-04-03T22:52:45.185 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root 3w REG 8,1 60015 2785 /var/log/sssd/sssd.log 2022-04-03T22:52:45.186 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root 4u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.186 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root 5u REG 8,1 1286144 2754 /var/lib/sss/db/config.ldb 2022-04-03T22:52:45.186 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root 6r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.187 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.187 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.187 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root 9u unix 0xffff88810e623480 0t0 16167 type=DGRAM 2022-04-03T22:52:45.187 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root 10u REG 8,1 3035136 2792 /var/lib/sss/db/cache_implicit_files.ldb 2022-04-03T22:52:45.188 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.188 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root 12u REG 8,1 1609728 2793 /var/lib/sss/db/timestamps_implicit_files.ldb 2022-04-03T22:52:45.188 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root 13u unix 0xffff88810e626900 0t0 16168 /var/lib/sss/pipes/private/sbus-monitor type=STREAM 2022-04-03T22:52:45.189 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root 14u unix 0xffff88810e627080 0t0 16169 type=STREAM 2022-04-03T22:52:45.189 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root 15u unix 0xffff88810e623c00 0t0 16170 /var/lib/sss/pipes/private/sbus-monitor type=STREAM 2022-04-03T22:52:45.189 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root 16r REG 8,1 11567160 72876 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:45.189 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root 17u netlink 0t0 16174 ROUTE 2022-04-03T22:52:45.190 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root 18u unix 0xffff888106ed7800 0t0 18838 /var/lib/sss/pipes/private/sbus-monitor type=STREAM 2022-04-03T22:52:45.190 INFO:teuthology.orchestra.run.smithi077.stdout:sssd 750 root 19u unix 0xffff8881795b9e00 0t0 18857 /var/lib/sss/pipes/private/sbus-monitor type=STREAM 2022-04-03T22:52:45.190 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.191 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.191 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd txt REG 8,1 142840 44173 /usr/lib/polkit-1/polkitd 2022-04-03T22:52:45.191 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:45.192 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd DEL REG 8,1 72876 /var/lib/sss/mc/initgroups 2022-04-03T22:52:45.192 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.192 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.193 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.193 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.193 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.193 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.194 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.194 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.194 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.195 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.195 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.195 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.196 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.196 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.196 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.196 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.197 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.197 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.197 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.198 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.198 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.198 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.199 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.199 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.199 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.200 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.200 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.200 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.200 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 16024 10213 /usr/lib64/libgmodule-2.0.so.0.5600.4 2022-04-03T22:52:45.201 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.201 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.201 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.202 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:45.202 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 23727960 8433 /usr/lib64/libmozjs-60.so.0.0.0 2022-04-03T22:52:45.202 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:45.203 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.203 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.203 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 347592 10215 /usr/lib64/libgobject-2.0.so.0.5600.4 2022-04-03T22:52:45.204 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 1770096 10209 /usr/lib64/libgio-2.0.so.0.5600.4 2022-04-03T22:52:45.204 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 127944 10594 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2022-04-03T22:52:45.204 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.205 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.205 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.205 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.206 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.206 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd 3r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.206 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd 4r REG 8,1 11567160 72876 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:45.206 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd 5r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.207 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd 6u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.207 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.207 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd 8u unix 0xffff88817f206900 0t0 21573 type=STREAM 2022-04-03T22:52:45.208 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd 9u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.208 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd 10r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.208 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd 11u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.209 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 polkitd 12u unix 0xffff88810e624b00 0t0 16166 type=DGRAM 2022-04-03T22:52:45.209 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.209 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.209 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd txt REG 8,1 142840 44173 /usr/lib/polkit-1/polkitd 2022-04-03T22:52:45.210 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:45.210 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd DEL REG 8,1 72876 /var/lib/sss/mc/initgroups 2022-04-03T22:52:45.210 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.211 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.211 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.211 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.212 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.212 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.212 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.213 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.213 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.213 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.213 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.214 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.214 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.214 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.215 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.215 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.215 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.216 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.216 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.216 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.217 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.217 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.217 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.218 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.218 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.218 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.218 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.219 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.219 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 16024 10213 /usr/lib64/libgmodule-2.0.so.0.5600.4 2022-04-03T22:52:45.219 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.220 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.220 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.220 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:45.220 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 23727960 8433 /usr/lib64/libmozjs-60.so.0.0.0 2022-04-03T22:52:45.221 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:45.221 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.221 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.222 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 347592 10215 /usr/lib64/libgobject-2.0.so.0.5600.4 2022-04-03T22:52:45.222 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 1770096 10209 /usr/lib64/libgio-2.0.so.0.5600.4 2022-04-03T22:52:45.222 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 127944 10594 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2022-04-03T22:52:45.223 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.223 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.223 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.223 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.224 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.224 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd 3r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.224 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd 4r REG 8,1 11567160 72876 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:45.225 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd 5r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.225 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd 6u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.225 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.225 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd 8u unix 0xffff88817f206900 0t0 21573 type=STREAM 2022-04-03T22:52:45.226 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd 9u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.226 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd 10r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.226 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd 11u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.227 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 808 gmain polkitd 12u unix 0xffff88810e624b00 0t0 16166 type=DGRAM 2022-04-03T22:52:45.227 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.227 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.228 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd txt REG 8,1 142840 44173 /usr/lib/polkit-1/polkitd 2022-04-03T22:52:45.228 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:45.228 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd DEL REG 8,1 72876 /var/lib/sss/mc/initgroups 2022-04-03T22:52:45.229 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.229 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.229 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.230 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.230 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.230 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.231 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.231 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.231 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.232 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.232 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.232 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.232 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.233 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.233 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.233 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.234 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.234 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.234 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.235 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.235 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.235 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.236 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.236 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.236 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.236 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.237 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.237 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.237 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 16024 10213 /usr/lib64/libgmodule-2.0.so.0.5600.4 2022-04-03T22:52:45.238 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.238 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.238 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.239 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:45.239 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 23727960 8433 /usr/lib64/libmozjs-60.so.0.0.0 2022-04-03T22:52:45.239 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:45.240 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.240 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.240 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 347592 10215 /usr/lib64/libgobject-2.0.so.0.5600.4 2022-04-03T22:52:45.241 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 1770096 10209 /usr/lib64/libgio-2.0.so.0.5600.4 2022-04-03T22:52:45.241 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 127944 10594 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2022-04-03T22:52:45.241 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.242 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.242 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.242 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.243 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.243 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd 3r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.243 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd 4r REG 8,1 11567160 72876 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:45.243 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd 5r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.244 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd 6u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.244 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.244 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd 8u unix 0xffff88817f206900 0t0 21573 type=STREAM 2022-04-03T22:52:45.245 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd 9u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.245 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd 10r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.245 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd 11u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.245 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 809 gdbus polkitd 12u unix 0xffff88810e624b00 0t0 16166 type=DGRAM 2022-04-03T22:52:45.246 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.246 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.246 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd txt REG 8,1 142840 44173 /usr/lib/polkit-1/polkitd 2022-04-03T22:52:45.247 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:45.247 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd DEL REG 8,1 72876 /var/lib/sss/mc/initgroups 2022-04-03T22:52:45.247 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.248 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.248 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.248 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.249 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.249 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.249 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.249 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.250 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.250 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.250 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.251 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.251 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.251 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.252 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.252 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.252 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.252 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.253 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.253 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.253 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.254 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.254 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.254 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.255 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.255 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.255 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.256 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.256 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 16024 10213 /usr/lib64/libgmodule-2.0.so.0.5600.4 2022-04-03T22:52:45.256 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.257 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.257 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.257 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:45.258 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 23727960 8433 /usr/lib64/libmozjs-60.so.0.0.0 2022-04-03T22:52:45.258 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:45.258 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.259 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.259 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 347592 10215 /usr/lib64/libgobject-2.0.so.0.5600.4 2022-04-03T22:52:45.259 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 1770096 10209 /usr/lib64/libgio-2.0.so.0.5600.4 2022-04-03T22:52:45.260 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 127944 10594 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2022-04-03T22:52:45.260 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.261 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.261 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.261 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.262 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.262 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd 3r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.262 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd 4r REG 8,1 11567160 72876 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:45.263 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd 5r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.263 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd 6u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.263 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.264 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd 8u unix 0xffff88817f206900 0t0 21573 type=STREAM 2022-04-03T22:52:45.264 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd 9u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.264 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd 10r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.265 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd 11u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.265 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 810 JS\x20Hel polkitd 12u unix 0xffff88810e624b00 0t0 16166 type=DGRAM 2022-04-03T22:52:45.265 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.265 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.266 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd txt REG 8,1 142840 44173 /usr/lib/polkit-1/polkitd 2022-04-03T22:52:45.266 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:45.266 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd DEL REG 8,1 72876 /var/lib/sss/mc/initgroups 2022-04-03T22:52:45.267 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.267 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.267 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.268 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.268 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.268 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.268 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.269 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.269 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.269 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.270 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.270 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.270 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.271 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.271 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.271 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.272 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.272 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.272 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.273 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.273 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.273 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.273 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.274 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.274 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.274 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.275 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.275 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.275 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 16024 10213 /usr/lib64/libgmodule-2.0.so.0.5600.4 2022-04-03T22:52:45.276 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.276 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.276 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.277 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:45.277 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 23727960 8433 /usr/lib64/libmozjs-60.so.0.0.0 2022-04-03T22:52:45.277 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:45.277 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.278 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.278 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 347592 10215 /usr/lib64/libgobject-2.0.so.0.5600.4 2022-04-03T22:52:45.278 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 1770096 10209 /usr/lib64/libgio-2.0.so.0.5600.4 2022-04-03T22:52:45.279 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 127944 10594 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2022-04-03T22:52:45.279 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.279 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.280 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.280 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.280 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.280 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd 3r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.281 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd 4r REG 8,1 11567160 72876 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:45.281 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd 5r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.281 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd 6u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.282 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.282 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd 8u unix 0xffff88817f206900 0t0 21573 type=STREAM 2022-04-03T22:52:45.282 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd 9u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.283 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd 10r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.283 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd 11u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.283 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 811 JS\x20Hel polkitd 12u unix 0xffff88810e624b00 0t0 16166 type=DGRAM 2022-04-03T22:52:45.283 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.284 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.284 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd txt REG 8,1 142840 44173 /usr/lib/polkit-1/polkitd 2022-04-03T22:52:45.284 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:45.285 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd DEL REG 8,1 72876 /var/lib/sss/mc/initgroups 2022-04-03T22:52:45.285 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.285 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.286 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.286 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.286 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.287 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.287 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.287 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.288 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.288 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.288 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.288 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.289 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.289 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.289 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.290 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.290 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.290 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.291 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.291 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.291 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.292 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.292 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.292 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.293 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.293 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.293 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.294 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.294 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 16024 10213 /usr/lib64/libgmodule-2.0.so.0.5600.4 2022-04-03T22:52:45.294 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.294 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.295 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.295 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:45.295 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 23727960 8433 /usr/lib64/libmozjs-60.so.0.0.0 2022-04-03T22:52:45.296 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:45.296 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.296 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.297 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 347592 10215 /usr/lib64/libgobject-2.0.so.0.5600.4 2022-04-03T22:52:45.297 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 1770096 10209 /usr/lib64/libgio-2.0.so.0.5600.4 2022-04-03T22:52:45.297 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 127944 10594 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2022-04-03T22:52:45.298 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.298 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.298 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.299 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.299 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.299 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd 3r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.300 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd 4r REG 8,1 11567160 72876 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:45.300 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd 5r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.300 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd 6u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.301 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.301 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd 8u unix 0xffff88817f206900 0t0 21573 type=STREAM 2022-04-03T22:52:45.301 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd 9u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.301 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd 10r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.302 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd 11u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.302 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 812 JS\x20Hel polkitd 12u unix 0xffff88810e624b00 0t0 16166 type=DGRAM 2022-04-03T22:52:45.303 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.303 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.303 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd txt REG 8,1 142840 44173 /usr/lib/polkit-1/polkitd 2022-04-03T22:52:45.304 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:45.304 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd DEL REG 8,1 72876 /var/lib/sss/mc/initgroups 2022-04-03T22:52:45.304 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.305 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.305 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.305 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.306 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.306 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.306 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.307 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.307 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.307 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.308 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.308 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.308 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.309 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.309 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.309 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.310 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.310 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.310 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.311 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.311 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.311 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.312 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.312 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.312 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.312 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.313 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.313 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.313 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 16024 10213 /usr/lib64/libgmodule-2.0.so.0.5600.4 2022-04-03T22:52:45.314 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.314 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.314 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.315 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:45.315 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 23727960 8433 /usr/lib64/libmozjs-60.so.0.0.0 2022-04-03T22:52:45.315 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:45.316 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.316 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.316 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 347592 10215 /usr/lib64/libgobject-2.0.so.0.5600.4 2022-04-03T22:52:45.316 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 1770096 10209 /usr/lib64/libgio-2.0.so.0.5600.4 2022-04-03T22:52:45.317 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 127944 10594 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2022-04-03T22:52:45.317 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.317 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.318 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.318 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.318 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.319 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd 3r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.319 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd 4r REG 8,1 11567160 72876 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:45.319 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd 5r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.319 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd 6u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.320 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.320 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd 8u unix 0xffff88817f206900 0t0 21573 type=STREAM 2022-04-03T22:52:45.320 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd 9u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.321 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd 10r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.321 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd 11u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.321 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 813 JS\x20Hel polkitd 12u unix 0xffff88810e624b00 0t0 16166 type=DGRAM 2022-04-03T22:52:45.322 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.322 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.322 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd txt REG 8,1 142840 44173 /usr/lib/polkit-1/polkitd 2022-04-03T22:52:45.322 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:45.323 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd DEL REG 8,1 72876 /var/lib/sss/mc/initgroups 2022-04-03T22:52:45.323 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.323 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.324 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.324 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.324 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.325 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.325 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.325 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.326 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.326 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.326 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.326 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.327 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.327 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.327 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.328 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.328 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.328 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.329 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.329 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.329 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.329 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.330 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.330 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.330 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.331 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.331 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.331 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.332 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 16024 10213 /usr/lib64/libgmodule-2.0.so.0.5600.4 2022-04-03T22:52:45.332 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.332 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.332 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.333 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:45.333 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 23727960 8433 /usr/lib64/libmozjs-60.so.0.0.0 2022-04-03T22:52:45.333 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:45.334 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.334 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.334 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 347592 10215 /usr/lib64/libgobject-2.0.so.0.5600.4 2022-04-03T22:52:45.335 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 1770096 10209 /usr/lib64/libgio-2.0.so.0.5600.4 2022-04-03T22:52:45.335 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 127944 10594 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2022-04-03T22:52:45.335 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.335 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.336 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.336 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.336 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.337 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd 3r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.337 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd 4r REG 8,1 11567160 72876 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:45.337 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd 5r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.337 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd 6u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.338 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.338 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd 8u unix 0xffff88817f206900 0t0 21573 type=STREAM 2022-04-03T22:52:45.338 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd 9u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.339 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd 10r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.339 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd 11u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.339 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 814 JS\x20Hel polkitd 12u unix 0xffff88810e624b00 0t0 16166 type=DGRAM 2022-04-03T22:52:45.340 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.340 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.340 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd txt REG 8,1 142840 44173 /usr/lib/polkit-1/polkitd 2022-04-03T22:52:45.341 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:45.341 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd DEL REG 8,1 72876 /var/lib/sss/mc/initgroups 2022-04-03T22:52:45.341 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.341 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.342 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.342 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.342 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.343 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.343 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.343 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.344 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.344 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.344 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.344 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.345 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.345 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.345 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.346 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.346 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.346 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.347 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.347 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.347 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.348 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.348 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.348 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.348 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.349 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.349 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.349 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.350 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 16024 10213 /usr/lib64/libgmodule-2.0.so.0.5600.4 2022-04-03T22:52:45.350 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.350 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.351 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.351 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:45.351 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 23727960 8433 /usr/lib64/libmozjs-60.so.0.0.0 2022-04-03T22:52:45.351 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:45.352 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.352 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.352 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 347592 10215 /usr/lib64/libgobject-2.0.so.0.5600.4 2022-04-03T22:52:45.353 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 1770096 10209 /usr/lib64/libgio-2.0.so.0.5600.4 2022-04-03T22:52:45.353 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 127944 10594 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2022-04-03T22:52:45.353 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.354 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.354 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.354 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.355 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.355 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd 3r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.355 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd 4r REG 8,1 11567160 72876 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:45.355 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd 5r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.356 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd 6u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.356 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.356 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd 8u unix 0xffff88817f206900 0t0 21573 type=STREAM 2022-04-03T22:52:45.357 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd 9u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.357 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd 10r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.357 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd 11u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.357 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 815 JS\x20Hel polkitd 12u unix 0xffff88810e624b00 0t0 16166 type=DGRAM 2022-04-03T22:52:45.358 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.358 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.358 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd txt REG 8,1 142840 44173 /usr/lib/polkit-1/polkitd 2022-04-03T22:52:45.359 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:45.359 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd DEL REG 8,1 72876 /var/lib/sss/mc/initgroups 2022-04-03T22:52:45.359 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.360 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.360 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.360 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.360 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.361 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.361 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.361 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.362 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.362 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.362 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.363 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.363 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.363 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.364 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.364 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.364 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.365 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.365 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.365 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.366 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.366 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.366 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.367 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.367 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.367 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.368 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.368 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.368 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 16024 10213 /usr/lib64/libgmodule-2.0.so.0.5600.4 2022-04-03T22:52:45.369 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.369 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.369 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.369 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:45.370 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 23727960 8433 /usr/lib64/libmozjs-60.so.0.0.0 2022-04-03T22:52:45.370 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:45.370 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.371 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.371 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 347592 10215 /usr/lib64/libgobject-2.0.so.0.5600.4 2022-04-03T22:52:45.371 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 1770096 10209 /usr/lib64/libgio-2.0.so.0.5600.4 2022-04-03T22:52:45.372 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 127944 10594 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2022-04-03T22:52:45.372 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.372 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.373 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.373 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.373 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.374 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd 3r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.374 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd 4r REG 8,1 11567160 72876 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:45.374 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd 5r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.375 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd 6u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.375 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.375 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd 8u unix 0xffff88817f206900 0t0 21573 type=STREAM 2022-04-03T22:52:45.376 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd 9u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.376 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd 10r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.376 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd 11u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.377 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 816 JS\x20Hel polkitd 12u unix 0xffff88810e624b00 0t0 16166 type=DGRAM 2022-04-03T22:52:45.377 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.377 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.377 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd txt REG 8,1 142840 44173 /usr/lib/polkit-1/polkitd 2022-04-03T22:52:45.378 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:45.378 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd DEL REG 8,1 72876 /var/lib/sss/mc/initgroups 2022-04-03T22:52:45.379 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.379 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.380 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.380 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.380 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.381 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.381 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.381 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.382 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.382 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.382 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.383 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.383 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.383 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.384 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.384 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.384 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.385 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.385 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.385 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.386 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.386 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.386 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.387 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.387 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.388 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.388 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.388 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.388 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 16024 10213 /usr/lib64/libgmodule-2.0.so.0.5600.4 2022-04-03T22:52:45.389 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.389 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.390 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.390 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:45.390 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 23727960 8433 /usr/lib64/libmozjs-60.so.0.0.0 2022-04-03T22:52:45.391 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:45.391 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.391 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.392 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 347592 10215 /usr/lib64/libgobject-2.0.so.0.5600.4 2022-04-03T22:52:45.392 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 1770096 10209 /usr/lib64/libgio-2.0.so.0.5600.4 2022-04-03T22:52:45.392 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 127944 10594 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2022-04-03T22:52:45.393 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.393 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.393 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.394 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.394 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.394 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd 3r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.395 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd 4r REG 8,1 11567160 72876 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:45.395 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd 5r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.395 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd 6u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.396 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.396 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd 8u unix 0xffff88817f206900 0t0 21573 type=STREAM 2022-04-03T22:52:45.396 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd 9u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.397 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd 10r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.397 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd 11u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.397 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 817 JS\x20Hel polkitd 12u unix 0xffff88810e624b00 0t0 16166 type=DGRAM 2022-04-03T22:52:45.397 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.398 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.398 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd txt REG 8,1 142840 44173 /usr/lib/polkit-1/polkitd 2022-04-03T22:52:45.398 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:45.399 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd DEL REG 8,1 72876 /var/lib/sss/mc/initgroups 2022-04-03T22:52:45.399 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.399 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.400 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.400 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.400 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.401 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.401 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.401 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.402 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.402 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.402 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.403 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.403 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.403 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.404 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.404 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.404 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.405 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.405 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.405 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.406 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.406 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.406 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.407 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.407 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.407 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.408 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.408 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.408 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 16024 10213 /usr/lib64/libgmodule-2.0.so.0.5600.4 2022-04-03T22:52:45.409 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.409 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.409 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.410 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:45.410 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 23727960 8433 /usr/lib64/libmozjs-60.so.0.0.0 2022-04-03T22:52:45.410 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:45.411 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.411 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.411 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 347592 10215 /usr/lib64/libgobject-2.0.so.0.5600.4 2022-04-03T22:52:45.412 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 1770096 10209 /usr/lib64/libgio-2.0.so.0.5600.4 2022-04-03T22:52:45.412 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 127944 10594 /usr/lib64/libpolkit-gobject-1.so.0.0.0 2022-04-03T22:52:45.412 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.413 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.413 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.413 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.414 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.414 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd 3r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.414 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd 4r REG 8,1 11567160 72876 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:45.415 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd 5r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.415 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd 6u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.415 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.416 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd 8u unix 0xffff88817f206900 0t0 21573 type=STREAM 2022-04-03T22:52:45.416 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd 9u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.416 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd 10r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.417 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd 11u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.417 INFO:teuthology.orchestra.run.smithi077.stdout:polkitd 752 839 polkitd polkitd 12u unix 0xffff88810e624b00 0t0 16166 type=DGRAM 2022-04-03T22:52:45.417 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.418 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.418 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus txt REG 8,1 245648 9587 /usr/bin/dbus-daemon 2022-04-03T22:52:45.418 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:45.419 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus DEL REG 8,1 3924 /var/lib/sss/mc/initgroups 2022-04-03T22:52:45.419 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.419 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus DEL REG 8,1 1109 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.420 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.420 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.420 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.421 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.421 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.421 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.422 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.422 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.422 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.423 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.423 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.423 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.424 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.424 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.424 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.425 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:45.425 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:45.425 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.426 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:45.426 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.426 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 351464 9341 /usr/lib64/libdbus-1.so.3.19.7 2022-04-03T22:52:45.427 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.427 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.428 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 1u unix 0xffff88817f134380 0t0 13264 type=STREAM 2022-04-03T22:52:45.428 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 2u unix 0xffff88817f134380 0t0 13264 type=STREAM 2022-04-03T22:52:45.428 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 3u unix 0xffff88810348f800 0t0 14038 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.429 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 4u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.429 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 5u unix 0xffff888178e4b480 0t0 52047 type=STREAM 2022-04-03T22:52:45.429 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 6u unix 0xffff888106ed3c00 0t0 26643 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.430 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 7u netlink 0t0 21342 AUDIT 2022-04-03T22:52:45.430 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 9u netlink 0t0 21343 SELINUX 2022-04-03T22:52:45.430 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 10r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.431 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 11u unix 0xffff8881050e9e00 0t0 21348 type=STREAM 2022-04-03T22:52:45.431 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 12u unix 0xffff8881050e8f00 0t0 21349 type=STREAM 2022-04-03T22:52:45.431 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 13u unix 0xffff888178df9680 0t0 21350 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.432 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 14u unix 0xffff888103597800 0t0 21354 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.432 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 15u unix 0xffff88817f207080 0t0 21358 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.432 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 16u unix 0xffff888178df8780 0t0 21359 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.433 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 17u unix 0xffff88817f207800 0t0 21576 type=DGRAM 2022-04-03T22:52:45.433 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 18u unix 0xffff888106ed1e00 0t0 180037 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.433 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 19r REG 8,1 9253600 1109 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.434 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 20u unix 0xffff88810eb10780 0t0 19975 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.434 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 21u unix 0xffff8881795ba580 0t0 20388 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.434 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 22u unix 0xffff8881050eda00 0t0 18813 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.435 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 23u unix 0xffff88817f134b00 0t0 22827 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.435 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 24u unix 0xffff88816f709680 0t0 25854 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.435 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 dbus 25r REG 8,1 11567160 3924 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:45.436 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.436 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.436 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus txt REG 8,1 245648 9587 /usr/bin/dbus-daemon 2022-04-03T22:52:45.437 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:45.437 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus DEL REG 8,1 3924 /var/lib/sss/mc/initgroups 2022-04-03T22:52:45.437 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.438 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus DEL REG 8,1 1109 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.438 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.438 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.439 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.439 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.439 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.440 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.440 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.440 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.441 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.441 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.441 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.442 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.442 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.442 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.443 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.443 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:45.443 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:45.444 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.444 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:45.444 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.445 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 351464 9341 /usr/lib64/libdbus-1.so.3.19.7 2022-04-03T22:52:45.445 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.446 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.446 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 1u unix 0xffff88817f134380 0t0 13264 type=STREAM 2022-04-03T22:52:45.446 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 2u unix 0xffff88817f134380 0t0 13264 type=STREAM 2022-04-03T22:52:45.447 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 3u unix 0xffff88810348f800 0t0 14038 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.447 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 4u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.447 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 5u unix 0xffff888178e4b480 0t0 52047 type=STREAM 2022-04-03T22:52:45.448 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 6u unix 0xffff888106ed3c00 0t0 26643 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.448 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 7u netlink 0t0 21342 AUDIT 2022-04-03T22:52:45.448 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 9u netlink 0t0 21343 SELINUX 2022-04-03T22:52:45.449 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 10r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.449 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 11u unix 0xffff8881050e9e00 0t0 21348 type=STREAM 2022-04-03T22:52:45.449 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 12u unix 0xffff8881050e8f00 0t0 21349 type=STREAM 2022-04-03T22:52:45.450 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 13u unix 0xffff888178df9680 0t0 21350 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.450 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 14u unix 0xffff888103597800 0t0 21354 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.450 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 15u unix 0xffff88817f207080 0t0 21358 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.451 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 16u unix 0xffff888178df8780 0t0 21359 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.451 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 17u unix 0xffff88817f207800 0t0 21576 type=DGRAM 2022-04-03T22:52:45.451 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 18u unix 0xffff888106ed1e00 0t0 180037 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.452 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 19r REG 8,1 9253600 1109 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.452 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 20u unix 0xffff88810eb10780 0t0 19975 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.452 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 21u unix 0xffff8881795ba580 0t0 20388 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.453 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 22u unix 0xffff8881050eda00 0t0 18813 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.453 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 23u unix 0xffff88817f134b00 0t0 22827 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.453 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 24u unix 0xffff88816f709680 0t0 25854 /run/dbus/system_bus_socket type=STREAM 2022-04-03T22:52:45.454 INFO:teuthology.orchestra.run.smithi077.stdout:dbus-daem 753 803 dbus-daem dbus 25r REG 8,1 11567160 3924 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:45.454 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.454 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.455 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root txt REG 8,1 3579248 11024 /usr/sbin/NetworkManager (deleted) 2022-04-03T22:52:45.455 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root DEL REG 8,1 47301 /usr/lib64/NetworkManager/1.30.0-7.el8/libnm-settings-plugin-ifcfg-rh.so 2022-04-03T22:52:45.455 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 351464 9341 /usr/lib64/libdbus-1.so.3.19.7 2022-04-03T22:52:45.456 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 65736 7781 /usr/lib64/libjansson.so.4.11.0 2022-04-03T22:52:45.456 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 24416 12815 /usr/lib64/libteamdctl.so.0.1.5 2022-04-03T22:52:45.456 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root DEL REG 8,1 48232 /usr/lib64/NetworkManager/1.30.0-7.el8/libnm-device-plugin-team.so 2022-04-03T22:52:45.457 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.457 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.457 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.458 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:45.458 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 37336 8429 /usr/lib64/libmodman.so.1.0.0 2022-04-03T22:52:45.458 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 144768 8431 /usr/lib64/libproxy.so.1.0.0 2022-04-03T22:52:45.459 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:45.459 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:45.459 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 134872 8681 /usr/lib64/libbrotlicommon.so.1.0.6 2022-04-03T22:52:45.460 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.460 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 125328 8879 /usr/lib64/libsasl2.so.3.0.0 2022-04-03T22:52:45.460 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:45.461 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:45.461 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.461 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.462 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.462 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.462 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.463 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.463 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 53560 8683 /usr/lib64/libbrotlidec.so.1.0.6 2022-04-03T22:52:45.463 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 67136 8940 /usr/lib64/liblber-2.4.so.2.10.9 2022-04-03T22:52:45.464 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 325184 8942 /usr/lib64/libldap-2.4.so.2.10.9 2022-04-03T22:52:45.464 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:45.464 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:45.465 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:45.465 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:45.465 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.466 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:45.466 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 69672 8866 /usr/lib64/libpsl.so.5.3.1 2022-04-03T22:52:45.466 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 465208 8936 /usr/lib64/libssh.so.4.8.5 2022-04-03T22:52:45.467 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 166888 8705 /usr/lib64/libnghttp2.so.14.17.0 2022-04-03T22:52:45.467 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:45.467 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.468 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.468 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.469 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.469 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.469 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.470 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.470 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.470 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.471 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.471 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.471 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.472 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.472 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.472 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.473 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.473 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.473 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.474 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.474 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 16024 10213 /usr/lib64/libgmodule-2.0.so.0.5600.4 2022-04-03T22:52:45.474 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 347592 10215 /usr/lib64/libgobject-2.0.so.0.5600.4 2022-04-03T22:52:45.475 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 1770096 10209 /usr/lib64/libgio-2.0.so.0.5600.4 2022-04-03T22:52:45.475 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 592280 9001 /usr/lib64/libcurl.so.4.5.0 2022-04-03T22:52:45.475 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:45.476 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.476 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 29040 8703 /usr/lib64/libndp.so.0.1.1 2022-04-03T22:52:45.476 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.477 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:45.477 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.477 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.478 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.478 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:45.478 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:45.479 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:45.479 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:45.479 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:45.480 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:45.480 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:45.480 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:45.481 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:45.481 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:45.481 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.482 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:45.482 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.482 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 1u unix 0xffff88810eb13c00 0t0 9865 type=STREAM 2022-04-03T22:52:45.483 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 2u unix 0xffff88810eb13c00 0t0 9865 type=STREAM 2022-04-03T22:52:45.483 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 3u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.483 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 4u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.484 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 5u unix 0xffff888178dfda00 0t0 19743 type=DGRAM 2022-04-03T22:52:45.484 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 6u unix 0xffff888178df8f00 0t0 19744 type=STREAM 2022-04-03T22:52:45.484 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.485 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 8r REG 0,4 0 4026532000 net 2022-04-03T22:52:45.485 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 9r REG 0,4 0 4026532499 mnt 2022-04-03T22:52:45.485 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 10u netlink 0t0 19748 KOBJECT_UEVENT 2022-04-03T22:52:45.485 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 11u netlink 0t0 19816 GENERIC 2022-04-03T22:52:45.486 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 12u netlink 0t0 19817 ROUTE 2022-04-03T22:52:45.486 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 13r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.486 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 14r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.487 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 15r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.487 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 16u netlink 0t0 18717 KOBJECT_UEVENT 2022-04-03T22:52:45.487 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 17u unix 0xffff88810eb10000 0t0 10029 type=STREAM 2022-04-03T22:52:45.488 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.488 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 19u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.488 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 20u a_inode 0,13 0 9293 [timerfd] 2022-04-03T22:52:45.489 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 21u pack 24596 0t0 IP type=SOCK_DGRAM 2022-04-03T22:52:45.489 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 22u IPv4 25871 0t0 UDP smithi077.front.sepia.ceph.com:bootpc->store01.front.sepia.ceph.com:bootps 2022-04-03T22:52:45.490 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 root 23w FIFO 0,24 0t0 1055 /run/systemd/inhibit/1.ref 2022-04-03T22:52:45.490 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.490 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.491 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root txt REG 8,1 3579248 11024 /usr/sbin/NetworkManager (deleted) 2022-04-03T22:52:45.491 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root DEL REG 8,1 47301 /usr/lib64/NetworkManager/1.30.0-7.el8/libnm-settings-plugin-ifcfg-rh.so 2022-04-03T22:52:45.491 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 351464 9341 /usr/lib64/libdbus-1.so.3.19.7 2022-04-03T22:52:45.491 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 65736 7781 /usr/lib64/libjansson.so.4.11.0 2022-04-03T22:52:45.492 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 24416 12815 /usr/lib64/libteamdctl.so.0.1.5 2022-04-03T22:52:45.492 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root DEL REG 8,1 48232 /usr/lib64/NetworkManager/1.30.0-7.el8/libnm-device-plugin-team.so 2022-04-03T22:52:45.492 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.493 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.493 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.493 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:45.494 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 37336 8429 /usr/lib64/libmodman.so.1.0.0 2022-04-03T22:52:45.494 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 144768 8431 /usr/lib64/libproxy.so.1.0.0 2022-04-03T22:52:45.494 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:45.495 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:45.495 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 134872 8681 /usr/lib64/libbrotlicommon.so.1.0.6 2022-04-03T22:52:45.495 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.496 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 125328 8879 /usr/lib64/libsasl2.so.3.0.0 2022-04-03T22:52:45.496 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:45.496 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:45.497 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.497 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.497 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.498 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.498 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.498 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.499 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 53560 8683 /usr/lib64/libbrotlidec.so.1.0.6 2022-04-03T22:52:45.499 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 67136 8940 /usr/lib64/liblber-2.4.so.2.10.9 2022-04-03T22:52:45.499 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 325184 8942 /usr/lib64/libldap-2.4.so.2.10.9 2022-04-03T22:52:45.500 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:45.500 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:45.500 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:45.501 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:45.501 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.501 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:45.502 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 69672 8866 /usr/lib64/libpsl.so.5.3.1 2022-04-03T22:52:45.502 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 465208 8936 /usr/lib64/libssh.so.4.8.5 2022-04-03T22:52:45.502 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 166888 8705 /usr/lib64/libnghttp2.so.14.17.0 2022-04-03T22:52:45.503 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:45.503 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.503 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.504 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.504 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.504 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.505 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.505 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.505 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.506 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.506 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.506 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.507 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.507 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.507 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.508 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.508 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.508 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.509 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.509 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.509 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 16024 10213 /usr/lib64/libgmodule-2.0.so.0.5600.4 2022-04-03T22:52:45.510 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 347592 10215 /usr/lib64/libgobject-2.0.so.0.5600.4 2022-04-03T22:52:45.510 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 1770096 10209 /usr/lib64/libgio-2.0.so.0.5600.4 2022-04-03T22:52:45.510 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 592280 9001 /usr/lib64/libcurl.so.4.5.0 2022-04-03T22:52:45.511 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:45.511 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.511 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 29040 8703 /usr/lib64/libndp.so.0.1.1 2022-04-03T22:52:45.512 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.512 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:45.512 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.513 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.513 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.513 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:45.514 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:45.514 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:45.514 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:45.515 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:45.515 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:45.515 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:45.516 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:45.516 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:45.516 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:45.517 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.517 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:45.517 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.518 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 1u unix 0xffff88810eb13c00 0t0 9865 type=STREAM 2022-04-03T22:52:45.518 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 2u unix 0xffff88810eb13c00 0t0 9865 type=STREAM 2022-04-03T22:52:45.518 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 3u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.519 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 4u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.519 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 5u unix 0xffff888178dfda00 0t0 19743 type=DGRAM 2022-04-03T22:52:45.519 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 6u unix 0xffff888178df8f00 0t0 19744 type=STREAM 2022-04-03T22:52:45.520 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.520 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 8r REG 0,4 0 4026532000 net 2022-04-03T22:52:45.520 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 9r REG 0,4 0 4026532499 mnt 2022-04-03T22:52:45.521 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 10u netlink 0t0 19748 KOBJECT_UEVENT 2022-04-03T22:52:45.521 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 11u netlink 0t0 19816 GENERIC 2022-04-03T22:52:45.521 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 12u netlink 0t0 19817 ROUTE 2022-04-03T22:52:45.522 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 13r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.522 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 14r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.522 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 15r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.523 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 16u netlink 0t0 18717 KOBJECT_UEVENT 2022-04-03T22:52:45.523 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 17u unix 0xffff88810eb10000 0t0 10029 type=STREAM 2022-04-03T22:52:45.523 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.524 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 19u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.524 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 20u a_inode 0,13 0 9293 [timerfd] 2022-04-03T22:52:45.524 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 21u pack 24596 0t0 IP type=SOCK_DGRAM 2022-04-03T22:52:45.525 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 22u IPv4 25871 0t0 UDP smithi077.front.sepia.ceph.com:bootpc->store01.front.sepia.ceph.com:bootps 2022-04-03T22:52:45.525 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 818 gmain root 23w FIFO 0,24 0t0 1055 /run/systemd/inhibit/1.ref 2022-04-03T22:52:45.525 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.526 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.526 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root txt REG 8,1 3579248 11024 /usr/sbin/NetworkManager (deleted) 2022-04-03T22:52:45.526 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root DEL REG 8,1 47301 /usr/lib64/NetworkManager/1.30.0-7.el8/libnm-settings-plugin-ifcfg-rh.so 2022-04-03T22:52:45.526 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 351464 9341 /usr/lib64/libdbus-1.so.3.19.7 2022-04-03T22:52:45.527 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 65736 7781 /usr/lib64/libjansson.so.4.11.0 2022-04-03T22:52:45.527 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 24416 12815 /usr/lib64/libteamdctl.so.0.1.5 2022-04-03T22:52:45.527 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root DEL REG 8,1 48232 /usr/lib64/NetworkManager/1.30.0-7.el8/libnm-device-plugin-team.so 2022-04-03T22:52:45.528 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.528 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.528 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.529 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:45.529 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 37336 8429 /usr/lib64/libmodman.so.1.0.0 2022-04-03T22:52:45.529 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 144768 8431 /usr/lib64/libproxy.so.1.0.0 2022-04-03T22:52:45.529 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:45.530 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:45.530 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 134872 8681 /usr/lib64/libbrotlicommon.so.1.0.6 2022-04-03T22:52:45.531 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.531 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 125328 8879 /usr/lib64/libsasl2.so.3.0.0 2022-04-03T22:52:45.531 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:45.531 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:45.532 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.532 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.532 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.533 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.533 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.533 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.534 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 53560 8683 /usr/lib64/libbrotlidec.so.1.0.6 2022-04-03T22:52:45.534 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 67136 8940 /usr/lib64/liblber-2.4.so.2.10.9 2022-04-03T22:52:45.534 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 325184 8942 /usr/lib64/libldap-2.4.so.2.10.9 2022-04-03T22:52:45.535 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:45.535 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:45.535 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:45.536 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:45.536 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.536 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:45.537 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 69672 8866 /usr/lib64/libpsl.so.5.3.1 2022-04-03T22:52:45.537 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 465208 8936 /usr/lib64/libssh.so.4.8.5 2022-04-03T22:52:45.537 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 166888 8705 /usr/lib64/libnghttp2.so.14.17.0 2022-04-03T22:52:45.538 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:45.538 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.538 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.538 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.539 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.539 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.539 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.540 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.540 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.540 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.541 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.541 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.541 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.542 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.542 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.542 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.543 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.543 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.543 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.544 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.544 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 16024 10213 /usr/lib64/libgmodule-2.0.so.0.5600.4 2022-04-03T22:52:45.544 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 347592 10215 /usr/lib64/libgobject-2.0.so.0.5600.4 2022-04-03T22:52:45.544 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 1770096 10209 /usr/lib64/libgio-2.0.so.0.5600.4 2022-04-03T22:52:45.545 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 592280 9001 /usr/lib64/libcurl.so.4.5.0 2022-04-03T22:52:45.545 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:45.545 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.546 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 29040 8703 /usr/lib64/libndp.so.0.1.1 2022-04-03T22:52:45.546 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.546 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:45.547 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.547 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.547 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.548 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:45.548 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:45.548 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:45.549 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:45.549 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:45.549 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:45.550 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:45.550 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:45.550 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:45.550 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:45.551 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.551 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:45.551 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.552 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 1u unix 0xffff88810eb13c00 0t0 9865 type=STREAM 2022-04-03T22:52:45.552 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 2u unix 0xffff88810eb13c00 0t0 9865 type=STREAM 2022-04-03T22:52:45.553 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 3u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.553 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 4u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.553 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 5u unix 0xffff888178dfda00 0t0 19743 type=DGRAM 2022-04-03T22:52:45.553 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 6u unix 0xffff888178df8f00 0t0 19744 type=STREAM 2022-04-03T22:52:45.554 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.554 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 8r REG 0,4 0 4026532000 net 2022-04-03T22:52:45.554 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 9r REG 0,4 0 4026532499 mnt 2022-04-03T22:52:45.555 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 10u netlink 0t0 19748 KOBJECT_UEVENT 2022-04-03T22:52:45.555 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 11u netlink 0t0 19816 GENERIC 2022-04-03T22:52:45.555 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 12u netlink 0t0 19817 ROUTE 2022-04-03T22:52:45.556 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 13r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.556 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 14r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.556 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 15r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.557 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 16u netlink 0t0 18717 KOBJECT_UEVENT 2022-04-03T22:52:45.557 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 17u unix 0xffff88810eb10000 0t0 10029 type=STREAM 2022-04-03T22:52:45.557 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.558 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 19u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.558 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 20u a_inode 0,13 0 9293 [timerfd] 2022-04-03T22:52:45.558 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 21u pack 24596 0t0 IP type=SOCK_DGRAM 2022-04-03T22:52:45.559 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 22u IPv4 25871 0t0 UDP smithi077.front.sepia.ceph.com:bootpc->store01.front.sepia.ceph.com:bootps 2022-04-03T22:52:45.559 INFO:teuthology.orchestra.run.smithi077.stdout:NetworkMa 755 819 gdbus root 23w FIFO 0,24 0t0 1055 /run/systemd/inhibit/1.ref 2022-04-03T22:52:45.559 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.560 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.560 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root txt REG 8,1 1150736 7265 /usr/bin/bash 2022-04-03T22:52:45.560 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.561 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.561 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.561 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:45.562 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.562 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.562 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root mem REG 8,1 208616 7207 /usr/lib64/libtinfo.so.6.1 2022-04-03T22:52:45.563 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.563 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:45.563 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:45.564 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:45.564 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:45.564 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:45.565 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:45.565 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:45.565 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:45.566 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:45.566 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:45.566 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.567 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:45.567 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.567 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root 1u unix 0xffff88810eb14b00 0t0 9943 type=STREAM 2022-04-03T22:52:45.568 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root 2u unix 0xffff88810eb14b00 0t0 9943 type=STREAM 2022-04-03T22:52:45.568 INFO:teuthology.orchestra.run.smithi077.stdout:ksmtuned 761 root 3r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.568 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.569 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.569 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi txt REG 8,1 146208 16118 /usr/sbin/avahi-daemon 2022-04-03T22:52:45.569 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:45.569 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi DEL REG 8,1 72876 /var/lib/sss/mc/initgroups 2022-04-03T22:52:45.570 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi DEL REG 8,1 72875 /var/lib/sss/mc/group 2022-04-03T22:52:45.570 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.570 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.571 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.571 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.571 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.572 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.572 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.572 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.572 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.573 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.573 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.573 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.574 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.574 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.574 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.575 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.575 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.575 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 351464 9341 /usr/lib64/libdbus-1.so.3.19.7 2022-04-03T22:52:45.576 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.576 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.576 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:45.577 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 30264 8700 /usr/lib64/libdaemon.so.0.5.0 2022-04-03T22:52:45.577 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 231504 16117 /usr/lib64/libavahi-core.so.7.0.2 2022-04-03T22:52:45.577 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 54208 10814 /usr/lib64/libavahi-common.so.3.5.3 2022-04-03T22:52:45.578 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.578 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.578 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi 1u unix 0xffff888178dfad00 0t0 19683 type=STREAM 2022-04-03T22:52:45.579 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi 2u unix 0xffff888178dfad00 0t0 19683 type=STREAM 2022-04-03T22:52:45.579 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi 3u unix 0xffff88810348d280 0t0 14036 /run/avahi-daemon/socket type=STREAM 2022-04-03T22:52:45.579 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi 4r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.579 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi 5r REG 8,1 6940392 72875 /var/lib/sss/mc/group (deleted) 2022-04-03T22:52:45.580 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi 6u unix 0xffff888103595a00 0t0 18693 type=DGRAM 2022-04-03T22:52:45.580 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi 7r REG 8,1 11567160 72876 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:45.580 INFO:teuthology.orchestra.run.smithi077.stdout:avahi-dae 805 avahi 9u unix 0xffff888103596180 0t0 18701 type=STREAM 2022-04-03T22:52:45.581 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.581 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.581 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root txt REG 8,1 11816 8955 /usr/libexec/platform-python3.6 2022-04-03T22:52:45.582 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:45.582 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.582 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.583 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.583 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.583 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 239416 10638 /usr/lib64/libgirepository-1.0.so.1.0.0 2022-04-03T22:52:45.584 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 340928 42138 /usr/lib64/python3.6/site-packages/gi/_gi.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.584 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.584 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.584 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.585 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.585 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.585 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.586 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.586 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.586 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 16024 10213 /usr/lib64/libgmodule-2.0.so.0.5600.4 2022-04-03T22:52:45.587 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.587 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.587 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.588 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 1770096 10209 /usr/lib64/libgio-2.0.so.0.5600.4 2022-04-03T22:52:45.588 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.588 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 347592 10215 /usr/lib64/libgobject-2.0.so.0.5600.4 2022-04-03T22:52:45.589 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 198680 11727 /usr/lib64/libdbus-glib-1.so.2.3.4 2022-04-03T22:52:45.589 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 14264 42274 /usr/lib64/python3.6/site-packages/_dbus_glib_bindings.so 2022-04-03T22:52:45.589 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:45.590 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.590 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.590 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.590 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.591 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.591 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.591 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.592 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.592 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.592 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.593 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.593 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.593 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.594 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 351464 9341 /usr/lib64/libdbus-1.so.3.19.7 2022-04-03T22:52:45.594 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 214536 42273 /usr/lib64/python3.6/site-packages/_dbus_bindings.so 2022-04-03T22:52:45.594 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.594 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.595 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.595 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.595 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:45.596 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.596 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.596 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.597 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.597 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.597 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.598 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:45.598 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.598 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.599 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.599 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.599 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.600 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.600 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.600 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 185712 45020 /usr/lib64/girepository-1.0/GLib-2.0.typelib 2022-04-03T22:52:45.600 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.601 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.601 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.602 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.602 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:45.602 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.603 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.603 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.603 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:45.603 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.604 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.604 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:45.604 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.605 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.605 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:45.605 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:45.606 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:45.606 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:45.606 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:45.607 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:45.607 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:45.607 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:45.608 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:45.608 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:45.608 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.609 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:45.609 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.609 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root 1u unix 0xffff888178dff080 0t0 19815 type=STREAM 2022-04-03T22:52:45.610 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root 2u unix 0xffff888178dff080 0t0 19815 type=STREAM 2022-04-03T22:52:45.610 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root 3r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.610 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root 4w REG 8,1 18760 2799 /var/log/tuned/tuned.log 2022-04-03T22:52:45.611 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root 5u netlink 0t0 19990 KOBJECT_UEVENT 2022-04-03T22:52:45.611 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root 6u unix 0xffff88817f136180 0t0 24605 type=STREAM 2022-04-03T22:52:45.611 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.612 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root 8w CHR 10,126 0t0 148 /dev/cpu_dma_latency 2022-04-03T22:52:45.612 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root 9r FIFO 0,12 0t0 22834 pipe 2022-04-03T22:52:45.612 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root 10w FIFO 0,12 0t0 22834 pipe 2022-04-03T22:52:45.613 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root 11u unix 0xffff888178dfcb00 0t0 20005 type=STREAM 2022-04-03T22:52:45.613 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 root 12u unix 0xffff888178dfb480 0t0 20006 type=STREAM 2022-04-03T22:52:45.613 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.614 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.614 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root txt REG 8,1 11816 8955 /usr/libexec/platform-python3.6 2022-04-03T22:52:45.614 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:45.615 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.615 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.615 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.616 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.616 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 239416 10638 /usr/lib64/libgirepository-1.0.so.1.0.0 2022-04-03T22:52:45.616 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 340928 42138 /usr/lib64/python3.6/site-packages/gi/_gi.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.617 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.617 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.617 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.618 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.618 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.618 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.619 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.619 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.619 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 16024 10213 /usr/lib64/libgmodule-2.0.so.0.5600.4 2022-04-03T22:52:45.619 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.620 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.620 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.620 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 1770096 10209 /usr/lib64/libgio-2.0.so.0.5600.4 2022-04-03T22:52:45.621 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.621 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 347592 10215 /usr/lib64/libgobject-2.0.so.0.5600.4 2022-04-03T22:52:45.621 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 198680 11727 /usr/lib64/libdbus-glib-1.so.2.3.4 2022-04-03T22:52:45.622 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 14264 42274 /usr/lib64/python3.6/site-packages/_dbus_glib_bindings.so 2022-04-03T22:52:45.622 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:45.622 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.623 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.623 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.624 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.624 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.624 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.625 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.625 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.625 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.625 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.626 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.626 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.626 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.627 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 351464 9341 /usr/lib64/libdbus-1.so.3.19.7 2022-04-03T22:52:45.627 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 214536 42273 /usr/lib64/python3.6/site-packages/_dbus_bindings.so 2022-04-03T22:52:45.628 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.628 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.628 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.628 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.629 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:45.629 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.629 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.630 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.630 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.630 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.631 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.631 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:45.631 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.632 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.632 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.632 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.633 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.633 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.633 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.634 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 185712 45020 /usr/lib64/girepository-1.0/GLib-2.0.typelib 2022-04-03T22:52:45.634 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.634 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.635 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.635 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.635 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:45.636 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.636 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.636 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.637 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:45.637 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.637 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.638 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:45.638 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.638 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.639 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:45.639 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:45.639 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:45.640 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:45.640 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:45.640 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:45.641 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:45.641 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:45.641 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:45.642 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:45.642 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.642 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:45.643 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.643 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root 1u unix 0xffff888178dff080 0t0 19815 type=STREAM 2022-04-03T22:52:45.643 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root 2u unix 0xffff888178dff080 0t0 19815 type=STREAM 2022-04-03T22:52:45.644 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root 3r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.644 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root 4w REG 8,1 18760 2799 /var/log/tuned/tuned.log 2022-04-03T22:52:45.644 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root 5u netlink 0t0 19990 KOBJECT_UEVENT 2022-04-03T22:52:45.645 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root 6u unix 0xffff88817f136180 0t0 24605 type=STREAM 2022-04-03T22:52:45.645 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.645 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root 8w CHR 10,126 0t0 148 /dev/cpu_dma_latency 2022-04-03T22:52:45.646 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root 9r FIFO 0,12 0t0 22834 pipe 2022-04-03T22:52:45.646 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root 10w FIFO 0,12 0t0 22834 pipe 2022-04-03T22:52:45.646 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root 11u unix 0xffff888178dfcb00 0t0 20005 type=STREAM 2022-04-03T22:52:45.647 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1029 tuned root 12u unix 0xffff888178dfb480 0t0 20006 type=STREAM 2022-04-03T22:52:45.647 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.647 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.648 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root txt REG 8,1 11816 8955 /usr/libexec/platform-python3.6 2022-04-03T22:52:45.648 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:45.648 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.649 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.649 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.649 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.650 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 239416 10638 /usr/lib64/libgirepository-1.0.so.1.0.0 2022-04-03T22:52:45.650 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 340928 42138 /usr/lib64/python3.6/site-packages/gi/_gi.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.650 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.651 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.651 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.651 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.652 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.652 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.652 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.653 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.653 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 16024 10213 /usr/lib64/libgmodule-2.0.so.0.5600.4 2022-04-03T22:52:45.653 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.654 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.654 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.654 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 1770096 10209 /usr/lib64/libgio-2.0.so.0.5600.4 2022-04-03T22:52:45.655 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.655 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 347592 10215 /usr/lib64/libgobject-2.0.so.0.5600.4 2022-04-03T22:52:45.655 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 198680 11727 /usr/lib64/libdbus-glib-1.so.2.3.4 2022-04-03T22:52:45.656 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 14264 42274 /usr/lib64/python3.6/site-packages/_dbus_glib_bindings.so 2022-04-03T22:52:45.656 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:45.656 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.657 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.657 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.657 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.658 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.658 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.658 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.659 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.659 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.659 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.660 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.660 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.660 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.661 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 351464 9341 /usr/lib64/libdbus-1.so.3.19.7 2022-04-03T22:52:45.661 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 214536 42273 /usr/lib64/python3.6/site-packages/_dbus_bindings.so 2022-04-03T22:52:45.661 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.662 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.662 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.663 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.663 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:45.663 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.664 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.664 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.664 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.665 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.665 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.665 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:45.666 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.666 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.666 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.667 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.667 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.667 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.668 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.668 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 185712 45020 /usr/lib64/girepository-1.0/GLib-2.0.typelib 2022-04-03T22:52:45.668 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.669 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.669 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.669 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.670 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:45.670 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.670 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.671 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.671 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:45.671 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.672 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.672 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:45.672 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.673 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.673 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:45.673 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:45.674 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:45.674 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:45.674 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:45.675 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:45.675 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:45.675 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:45.676 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:45.676 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:45.676 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.677 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:45.677 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.677 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root 1u unix 0xffff888178dff080 0t0 19815 type=STREAM 2022-04-03T22:52:45.678 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root 2u unix 0xffff888178dff080 0t0 19815 type=STREAM 2022-04-03T22:52:45.678 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root 3r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.678 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root 4w REG 8,1 18760 2799 /var/log/tuned/tuned.log 2022-04-03T22:52:45.679 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root 5u netlink 0t0 19990 KOBJECT_UEVENT 2022-04-03T22:52:45.679 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root 6u unix 0xffff88817f136180 0t0 24605 type=STREAM 2022-04-03T22:52:45.679 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.680 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root 8w CHR 10,126 0t0 148 /dev/cpu_dma_latency 2022-04-03T22:52:45.680 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root 9r FIFO 0,12 0t0 22834 pipe 2022-04-03T22:52:45.680 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root 10w FIFO 0,12 0t0 22834 pipe 2022-04-03T22:52:45.681 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root 11u unix 0xffff888178dfcb00 0t0 20005 type=STREAM 2022-04-03T22:52:45.681 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1033 tuned root 12u unix 0xffff888178dfb480 0t0 20006 type=STREAM 2022-04-03T22:52:45.681 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.682 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.682 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root txt REG 8,1 11816 8955 /usr/libexec/platform-python3.6 2022-04-03T22:52:45.682 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:45.683 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.683 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.683 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.684 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.684 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 239416 10638 /usr/lib64/libgirepository-1.0.so.1.0.0 2022-04-03T22:52:45.684 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 340928 42138 /usr/lib64/python3.6/site-packages/gi/_gi.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.685 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.685 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.685 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.686 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.686 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.686 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.687 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.687 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.687 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 16024 10213 /usr/lib64/libgmodule-2.0.so.0.5600.4 2022-04-03T22:52:45.688 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.688 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.688 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.689 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 1770096 10209 /usr/lib64/libgio-2.0.so.0.5600.4 2022-04-03T22:52:45.689 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.689 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 347592 10215 /usr/lib64/libgobject-2.0.so.0.5600.4 2022-04-03T22:52:45.689 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 198680 11727 /usr/lib64/libdbus-glib-1.so.2.3.4 2022-04-03T22:52:45.690 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 14264 42274 /usr/lib64/python3.6/site-packages/_dbus_glib_bindings.so 2022-04-03T22:52:45.690 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:45.690 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.691 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.691 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.691 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.692 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.692 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.692 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.693 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.693 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.693 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.694 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.694 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.694 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.695 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 351464 9341 /usr/lib64/libdbus-1.so.3.19.7 2022-04-03T22:52:45.695 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 214536 42273 /usr/lib64/python3.6/site-packages/_dbus_bindings.so 2022-04-03T22:52:45.695 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.696 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.696 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.696 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.697 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:45.697 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.697 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.698 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.698 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.698 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.699 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.699 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:45.699 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.700 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.700 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.700 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.701 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.701 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.701 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.702 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 185712 45020 /usr/lib64/girepository-1.0/GLib-2.0.typelib 2022-04-03T22:52:45.702 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:45.702 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.703 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.703 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.704 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:45.704 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.704 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.704 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.705 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:45.705 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.705 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.706 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:45.706 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.707 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.707 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:45.707 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:45.708 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:45.708 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:45.708 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:45.709 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:45.709 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:45.709 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:45.710 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:45.710 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:45.710 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.711 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:45.711 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.712 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root 1u unix 0xffff888178dff080 0t0 19815 type=STREAM 2022-04-03T22:52:45.712 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root 2u unix 0xffff888178dff080 0t0 19815 type=STREAM 2022-04-03T22:52:45.712 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root 3r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.713 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root 4w REG 8,1 18760 2799 /var/log/tuned/tuned.log 2022-04-03T22:52:45.713 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root 5u netlink 0t0 19990 KOBJECT_UEVENT 2022-04-03T22:52:45.713 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root 6u unix 0xffff88817f136180 0t0 24605 type=STREAM 2022-04-03T22:52:45.714 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.714 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root 8w CHR 10,126 0t0 148 /dev/cpu_dma_latency 2022-04-03T22:52:45.714 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root 9r FIFO 0,12 0t0 22834 pipe 2022-04-03T22:52:45.715 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root 10w FIFO 0,12 0t0 22834 pipe 2022-04-03T22:52:45.715 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root 11u unix 0xffff888178dfcb00 0t0 20005 type=STREAM 2022-04-03T22:52:45.715 INFO:teuthology.orchestra.run.smithi077.stdout:tuned 821 1050 tuned root 12u unix 0xffff888178dfb480 0t0 20006 type=STREAM 2022-04-03T22:52:45.716 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.716 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.716 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root txt REG 8,1 467744 16909 /usr/sbin/cupsd 2022-04-03T22:52:45.717 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root DEL REG 8,1 72875 /var/lib/sss/mc/group 2022-04-03T22:52:45.717 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.717 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.718 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.718 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.718 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.719 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.719 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.719 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.720 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:45.720 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.720 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.721 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.721 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:45.721 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:45.722 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.722 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.722 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.723 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.723 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.723 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.724 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.724 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 26192 7582 /usr/lib64/libattr.so.1.1.2448 2022-04-03T22:52:45.724 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:45.725 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.725 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.725 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.726 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.726 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.726 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.727 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.727 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.727 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:45.728 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.728 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.728 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:45.729 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:45.729 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:45.729 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:45.730 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 629352 10828 /usr/lib64/libcups.so.2 2022-04-03T22:52:45.730 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 74352 10812 /usr/lib64/libavahi-client.so.3.2.9 2022-04-03T22:52:45.730 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 54208 10814 /usr/lib64/libavahi-common.so.3.5.3 2022-04-03T22:52:45.731 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.731 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 351464 9341 /usr/lib64/libdbus-1.so.3.19.7 2022-04-03T22:52:45.731 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 54816 7584 /usr/lib64/libacl.so.1.1.2253 2022-04-03T22:52:45.732 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.732 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:45.732 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.733 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.733 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 28784 10831 /usr/lib64/libcupsmime.so.1 2022-04-03T22:52:45.733 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.734 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.734 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:45.734 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.735 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root 1u unix 0xffff8881050ee180 0t0 21495 type=STREAM 2022-04-03T22:52:45.735 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root 2u unix 0xffff8881050ee180 0t0 21495 type=STREAM 2022-04-03T22:52:45.735 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root 3u unix 0xffff88810348f080 0t0 14040 /var/run/cups/cups.sock type=STREAM 2022-04-03T22:52:45.736 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root 4u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.736 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root 5r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.736 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root 6r REG 8,1 6940392 72875 /var/lib/sss/mc/group (deleted) 2022-04-03T22:52:45.737 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root 7u unix 0xffff8881050ef080 0t0 23619 type=STREAM 2022-04-03T22:52:45.737 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root 8u IPv6 23620 0t0 TCP localhost:ipp (LISTEN) 2022-04-03T22:52:45.737 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root 9u IPv4 23621 0t0 TCP localhost:ipp (LISTEN) 2022-04-03T22:52:45.738 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root 10r FIFO 0,12 0t0 23622 pipe 2022-04-03T22:52:45.738 INFO:teuthology.orchestra.run.smithi077.stdout:cupsd 823 root 11w FIFO 0,12 0t0 23622 pipe 2022-04-03T22:52:45.738 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.739 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.739 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root txt REG 8,1 877984 13329 /usr/sbin/sshd 2022-04-03T22:52:45.739 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root DEL REG 8,1 72875 /var/lib/sss/mc/group 2022-04-03T22:52:45.740 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.740 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.740 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.741 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.741 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.741 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:45.742 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:45.742 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.742 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.743 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.743 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.743 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.744 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.744 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.744 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.745 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.745 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:45.745 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.746 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:45.746 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:45.746 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:45.747 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:45.747 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.747 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.748 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:45.748 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.749 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:45.749 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.749 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.750 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.750 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:45.750 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:45.750 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.751 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.751 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root 1u unix 0xffff888178dfe180 0t0 19888 type=STREAM 2022-04-03T22:52:45.751 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root 2u unix 0xffff888178dfe180 0t0 19888 type=STREAM 2022-04-03T22:52:45.752 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root 3r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.752 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root 4u IPv4 19969 0t0 TCP *:ssh (LISTEN) 2022-04-03T22:52:45.753 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root 5r REG 8,1 6940392 72875 /var/lib/sss/mc/group (deleted) 2022-04-03T22:52:45.753 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 828 root 6u IPv6 19971 0t0 TCP *:ssh (LISTEN) 2022-04-03T22:52:45.753 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.754 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.754 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root txt REG 8,1 25672 13032 /usr/bin/rhsmcertd 2022-04-03T22:52:45.754 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:45.755 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.755 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.755 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.756 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.756 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.756 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.757 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.757 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.757 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.758 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.758 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.758 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.759 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.759 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.759 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.760 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:45.760 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:45.760 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:45.761 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:45.761 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:45.761 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:45.762 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:45.762 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:45.762 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:45.763 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:45.763 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.763 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:45.764 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.764 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.764 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.765 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root 3wW REG 0,24 0 1024 /run/lock/subsys/rhsmcertd 2022-04-03T22:52:45.765 INFO:teuthology.orchestra.run.smithi077.stdout:rhsmcertd 832 root 4u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.765 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.766 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.766 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root txt REG 8,1 135480 15652 /usr/sbin/gssproxy 2022-04-03T22:52:45.766 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 379136 11112 /usr/lib64/libevent-2.1.so.6.0.2 2022-04-03T22:52:45.767 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 12136 15651 /usr/lib64/libverto-libevent.so.1.0.0 2022-04-03T22:52:45.767 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:45.767 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.768 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.768 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 16160 8221 /usr/lib64/libpath_utils.so.1.0.1 2022-04-03T22:52:45.768 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.769 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.769 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.769 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:45.770 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:45.770 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.771 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.771 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:45.771 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 138832 9239 /usr/lib64/libgssrpc.so.4.2 2022-04-03T22:52:45.772 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.772 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 55560 7633 /usr/lib64/libcollection.so.4.1.1 2022-04-03T22:52:45.772 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 12024 7637 /usr/lib64/libref_array.so.1.2.1 2022-04-03T22:52:45.773 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 11848 7624 /usr/lib64/libbasicobjects.so.0.1.0 2022-04-03T22:52:45.773 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 117048 8223 /usr/lib64/libini_config.so.5.2.1 2022-04-03T22:52:45.773 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 24872 8745 /usr/lib64/libverto.so.1.0.0 2022-04-03T22:52:45.774 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:45.774 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:45.774 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:45.775 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 54688 7381 /usr/lib64/libpopt.so.0.0.1 2022-04-03T22:52:45.775 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.775 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:45.776 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:45.776 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:45.776 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:45.777 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:45.777 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:45.777 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:45.778 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:45.778 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:45.778 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:45.779 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:45.779 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.779 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.780 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root 1u unix 0xffff888178dff800 0t0 19956 type=STREAM 2022-04-03T22:52:45.780 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root 2u unix 0xffff888178dff800 0t0 19956 type=STREAM 2022-04-03T22:52:45.780 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root 3u unix 0xffff88817f201e00 0t0 21579 type=DGRAM 2022-04-03T22:52:45.781 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root 4r FIFO 0,12 0t0 21585 pipe 2022-04-03T22:52:45.781 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.781 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root 6r FIFO 0,12 0t0 21580 pipe 2022-04-03T22:52:45.782 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root 7w FIFO 0,12 0t0 21580 pipe 2022-04-03T22:52:45.782 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root 8u unix 0xffff88817f201680 0t0 21581 /var/lib/gssproxy/default.sock type=STREAM 2022-04-03T22:52:45.782 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root 9u unix 0xffff88817f200000 0t0 21582 /run/gssproxy.sock type=STREAM 2022-04-03T22:52:45.782 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 root 10w FIFO 0,12 0t0 21585 pipe 2022-04-03T22:52:45.783 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.783 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.783 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root txt REG 8,1 135480 15652 /usr/sbin/gssproxy 2022-04-03T22:52:45.784 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 379136 11112 /usr/lib64/libevent-2.1.so.6.0.2 2022-04-03T22:52:45.784 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 12136 15651 /usr/lib64/libverto-libevent.so.1.0.0 2022-04-03T22:52:45.784 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:45.785 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.785 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.785 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 16160 8221 /usr/lib64/libpath_utils.so.1.0.1 2022-04-03T22:52:45.786 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.786 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.786 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.787 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:45.787 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:45.787 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.788 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.788 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:45.788 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 138832 9239 /usr/lib64/libgssrpc.so.4.2 2022-04-03T22:52:45.789 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.789 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 55560 7633 /usr/lib64/libcollection.so.4.1.1 2022-04-03T22:52:45.789 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 12024 7637 /usr/lib64/libref_array.so.1.2.1 2022-04-03T22:52:45.790 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 11848 7624 /usr/lib64/libbasicobjects.so.0.1.0 2022-04-03T22:52:45.790 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 117048 8223 /usr/lib64/libini_config.so.5.2.1 2022-04-03T22:52:45.790 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 24872 8745 /usr/lib64/libverto.so.1.0.0 2022-04-03T22:52:45.791 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:45.791 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:45.791 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:45.792 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 54688 7381 /usr/lib64/libpopt.so.0.0.1 2022-04-03T22:52:45.792 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.792 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:45.793 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:45.793 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:45.793 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:45.794 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:45.794 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:45.794 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:45.795 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:45.795 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:45.795 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:45.796 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:45.796 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.796 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.797 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root 1u unix 0xffff888178dff800 0t0 19956 type=STREAM 2022-04-03T22:52:45.797 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root 2u unix 0xffff888178dff800 0t0 19956 type=STREAM 2022-04-03T22:52:45.797 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root 3u unix 0xffff88817f201e00 0t0 21579 type=DGRAM 2022-04-03T22:52:45.798 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root 4r FIFO 0,12 0t0 21585 pipe 2022-04-03T22:52:45.798 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.798 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root 6r FIFO 0,12 0t0 21580 pipe 2022-04-03T22:52:45.799 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root 7w FIFO 0,12 0t0 21580 pipe 2022-04-03T22:52:45.799 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root 8u unix 0xffff88817f201680 0t0 21581 /var/lib/gssproxy/default.sock type=STREAM 2022-04-03T22:52:45.799 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root 9u unix 0xffff88817f200000 0t0 21582 /run/gssproxy.sock type=STREAM 2022-04-03T22:52:45.800 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 840 gssproxy root 10w FIFO 0,12 0t0 21585 pipe 2022-04-03T22:52:45.800 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.800 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.800 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root txt REG 8,1 135480 15652 /usr/sbin/gssproxy 2022-04-03T22:52:45.801 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 379136 11112 /usr/lib64/libevent-2.1.so.6.0.2 2022-04-03T22:52:45.801 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 12136 15651 /usr/lib64/libverto-libevent.so.1.0.0 2022-04-03T22:52:45.801 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:45.802 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.802 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.802 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 16160 8221 /usr/lib64/libpath_utils.so.1.0.1 2022-04-03T22:52:45.803 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.803 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.803 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.804 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:45.804 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:45.804 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.805 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.805 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:45.805 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 138832 9239 /usr/lib64/libgssrpc.so.4.2 2022-04-03T22:52:45.806 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.806 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 55560 7633 /usr/lib64/libcollection.so.4.1.1 2022-04-03T22:52:45.806 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 12024 7637 /usr/lib64/libref_array.so.1.2.1 2022-04-03T22:52:45.807 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 11848 7624 /usr/lib64/libbasicobjects.so.0.1.0 2022-04-03T22:52:45.807 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 117048 8223 /usr/lib64/libini_config.so.5.2.1 2022-04-03T22:52:45.807 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 24872 8745 /usr/lib64/libverto.so.1.0.0 2022-04-03T22:52:45.808 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:45.808 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:45.808 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:45.809 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 54688 7381 /usr/lib64/libpopt.so.0.0.1 2022-04-03T22:52:45.809 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.809 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:45.810 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:45.810 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:45.810 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:45.811 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:45.811 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:45.811 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:45.812 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:45.812 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:45.812 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:45.813 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:45.813 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.813 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.813 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root 1u unix 0xffff888178dff800 0t0 19956 type=STREAM 2022-04-03T22:52:45.814 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root 2u unix 0xffff888178dff800 0t0 19956 type=STREAM 2022-04-03T22:52:45.814 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root 3u unix 0xffff88817f201e00 0t0 21579 type=DGRAM 2022-04-03T22:52:45.815 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root 4r FIFO 0,12 0t0 21585 pipe 2022-04-03T22:52:45.815 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.815 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root 6r FIFO 0,12 0t0 21580 pipe 2022-04-03T22:52:45.816 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root 7w FIFO 0,12 0t0 21580 pipe 2022-04-03T22:52:45.816 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root 8u unix 0xffff88817f201680 0t0 21581 /var/lib/gssproxy/default.sock type=STREAM 2022-04-03T22:52:45.816 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root 9u unix 0xffff88817f200000 0t0 21582 /run/gssproxy.sock type=STREAM 2022-04-03T22:52:45.817 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 841 gssproxy root 10w FIFO 0,12 0t0 21585 pipe 2022-04-03T22:52:45.817 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.817 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.818 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root txt REG 8,1 135480 15652 /usr/sbin/gssproxy 2022-04-03T22:52:45.818 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 379136 11112 /usr/lib64/libevent-2.1.so.6.0.2 2022-04-03T22:52:45.818 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 12136 15651 /usr/lib64/libverto-libevent.so.1.0.0 2022-04-03T22:52:45.819 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:45.819 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.819 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.820 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 16160 8221 /usr/lib64/libpath_utils.so.1.0.1 2022-04-03T22:52:45.820 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.820 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.821 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.821 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:45.821 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:45.822 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.822 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.822 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:45.823 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 138832 9239 /usr/lib64/libgssrpc.so.4.2 2022-04-03T22:52:45.823 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.823 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 55560 7633 /usr/lib64/libcollection.so.4.1.1 2022-04-03T22:52:45.824 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 12024 7637 /usr/lib64/libref_array.so.1.2.1 2022-04-03T22:52:45.824 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 11848 7624 /usr/lib64/libbasicobjects.so.0.1.0 2022-04-03T22:52:45.824 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 117048 8223 /usr/lib64/libini_config.so.5.2.1 2022-04-03T22:52:45.825 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 24872 8745 /usr/lib64/libverto.so.1.0.0 2022-04-03T22:52:45.825 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:45.825 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:45.826 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:45.826 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 54688 7381 /usr/lib64/libpopt.so.0.0.1 2022-04-03T22:52:45.826 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.827 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:45.827 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:45.827 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:45.828 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:45.828 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:45.828 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:45.829 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:45.829 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:45.829 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:45.830 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:45.830 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:45.830 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.831 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.831 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root 1u unix 0xffff888178dff800 0t0 19956 type=STREAM 2022-04-03T22:52:45.831 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root 2u unix 0xffff888178dff800 0t0 19956 type=STREAM 2022-04-03T22:52:45.832 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root 3u unix 0xffff88817f201e00 0t0 21579 type=DGRAM 2022-04-03T22:52:45.832 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root 4r FIFO 0,12 0t0 21585 pipe 2022-04-03T22:52:45.832 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.833 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root 6r FIFO 0,12 0t0 21580 pipe 2022-04-03T22:52:45.833 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root 7w FIFO 0,12 0t0 21580 pipe 2022-04-03T22:52:45.833 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root 8u unix 0xffff88817f201680 0t0 21581 /var/lib/gssproxy/default.sock type=STREAM 2022-04-03T22:52:45.834 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root 9u unix 0xffff88817f200000 0t0 21582 /run/gssproxy.sock type=STREAM 2022-04-03T22:52:45.834 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 842 gssproxy root 10w FIFO 0,12 0t0 21585 pipe 2022-04-03T22:52:45.834 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.835 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.835 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root txt REG 8,1 135480 15652 /usr/sbin/gssproxy 2022-04-03T22:52:45.835 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 379136 11112 /usr/lib64/libevent-2.1.so.6.0.2 2022-04-03T22:52:45.836 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 12136 15651 /usr/lib64/libverto-libevent.so.1.0.0 2022-04-03T22:52:45.836 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:45.836 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.837 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.837 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 16160 8221 /usr/lib64/libpath_utils.so.1.0.1 2022-04-03T22:52:45.837 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.838 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.838 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.838 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:45.839 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:45.839 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.839 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.840 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:45.840 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 138832 9239 /usr/lib64/libgssrpc.so.4.2 2022-04-03T22:52:45.840 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.841 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 55560 7633 /usr/lib64/libcollection.so.4.1.1 2022-04-03T22:52:45.841 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 12024 7637 /usr/lib64/libref_array.so.1.2.1 2022-04-03T22:52:45.841 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 11848 7624 /usr/lib64/libbasicobjects.so.0.1.0 2022-04-03T22:52:45.842 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 117048 8223 /usr/lib64/libini_config.so.5.2.1 2022-04-03T22:52:45.842 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 24872 8745 /usr/lib64/libverto.so.1.0.0 2022-04-03T22:52:45.842 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:45.843 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:45.843 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:45.843 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 54688 7381 /usr/lib64/libpopt.so.0.0.1 2022-04-03T22:52:45.844 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.844 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:45.844 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:45.845 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:45.845 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:45.845 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:45.846 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:45.846 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:45.846 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:45.847 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:45.847 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:45.847 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:45.848 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.848 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.848 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root 1u unix 0xffff888178dff800 0t0 19956 type=STREAM 2022-04-03T22:52:45.849 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root 2u unix 0xffff888178dff800 0t0 19956 type=STREAM 2022-04-03T22:52:45.849 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root 3u unix 0xffff88817f201e00 0t0 21579 type=DGRAM 2022-04-03T22:52:45.849 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root 4r FIFO 0,12 0t0 21585 pipe 2022-04-03T22:52:45.850 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.850 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root 6r FIFO 0,12 0t0 21580 pipe 2022-04-03T22:52:45.850 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root 7w FIFO 0,12 0t0 21580 pipe 2022-04-03T22:52:45.851 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root 8u unix 0xffff88817f201680 0t0 21581 /var/lib/gssproxy/default.sock type=STREAM 2022-04-03T22:52:45.851 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root 9u unix 0xffff88817f200000 0t0 21582 /run/gssproxy.sock type=STREAM 2022-04-03T22:52:45.851 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 843 gssproxy root 10w FIFO 0,12 0t0 21585 pipe 2022-04-03T22:52:45.852 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.852 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.852 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root txt REG 8,1 135480 15652 /usr/sbin/gssproxy 2022-04-03T22:52:45.853 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 379136 11112 /usr/lib64/libevent-2.1.so.6.0.2 2022-04-03T22:52:45.853 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 12136 15651 /usr/lib64/libverto-libevent.so.1.0.0 2022-04-03T22:52:45.853 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:45.854 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.854 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.854 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 16160 8221 /usr/lib64/libpath_utils.so.1.0.1 2022-04-03T22:52:45.855 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.855 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.855 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.856 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:45.856 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:45.856 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.857 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.857 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:45.857 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 138832 9239 /usr/lib64/libgssrpc.so.4.2 2022-04-03T22:52:45.858 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.858 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 55560 7633 /usr/lib64/libcollection.so.4.1.1 2022-04-03T22:52:45.858 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 12024 7637 /usr/lib64/libref_array.so.1.2.1 2022-04-03T22:52:45.859 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 11848 7624 /usr/lib64/libbasicobjects.so.0.1.0 2022-04-03T22:52:45.859 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 117048 8223 /usr/lib64/libini_config.so.5.2.1 2022-04-03T22:52:45.859 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 24872 8745 /usr/lib64/libverto.so.1.0.0 2022-04-03T22:52:45.860 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:45.860 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:45.860 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:45.861 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 54688 7381 /usr/lib64/libpopt.so.0.0.1 2022-04-03T22:52:45.861 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.861 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:45.862 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:45.862 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:45.862 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:45.863 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:45.863 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:45.863 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:45.864 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:45.864 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:45.864 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:45.865 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:45.865 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.865 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.866 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root 1u unix 0xffff888178dff800 0t0 19956 type=STREAM 2022-04-03T22:52:45.866 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root 2u unix 0xffff888178dff800 0t0 19956 type=STREAM 2022-04-03T22:52:45.866 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root 3u unix 0xffff88817f201e00 0t0 21579 type=DGRAM 2022-04-03T22:52:45.867 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root 4r FIFO 0,12 0t0 21585 pipe 2022-04-03T22:52:45.867 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.867 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root 6r FIFO 0,12 0t0 21580 pipe 2022-04-03T22:52:45.868 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root 7w FIFO 0,12 0t0 21580 pipe 2022-04-03T22:52:45.868 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root 8u unix 0xffff88817f201680 0t0 21581 /var/lib/gssproxy/default.sock type=STREAM 2022-04-03T22:52:45.868 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root 9u unix 0xffff88817f200000 0t0 21582 /run/gssproxy.sock type=STREAM 2022-04-03T22:52:45.869 INFO:teuthology.orchestra.run.smithi077.stdout:gssproxy 838 844 gssproxy root 10w FIFO 0,12 0t0 21585 pipe 2022-04-03T22:52:45.869 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.869 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.870 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root txt REG 8,1 241976 55039 /usr/libexec/sssd/sssd_be 2022-04-03T22:52:45.870 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 68208 10624 /usr/lib64/libsss_certmap.so.0.2.0 2022-04-03T22:52:45.870 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 54008 35856 /usr/lib64/sssd/libsss_files.so 2022-04-03T22:52:45.870 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:45.871 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root DEL REG 8,1 72876 /var/lib/sss/mc/initgroups 2022-04-03T22:52:45.871 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.871 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.872 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.872 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 1609728 2793 /var/lib/sss/db/timestamps_implicit_files.ldb 2022-04-03T22:52:45.872 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 3035136 2792 /var/lib/sss/db/cache_implicit_files.ldb 2022-04-03T22:52:45.873 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 7496 44161 /usr/lib64/ldb/modules/ldb/tdb.so 2022-04-03T22:52:45.873 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 11776 44160 /usr/lib64/ldb/modules/ldb/skel.so 2022-04-03T22:52:45.873 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 15856 44159 /usr/lib64/ldb/modules/ldb/server_sort.so 2022-04-03T22:52:45.874 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 11952 44158 /usr/lib64/ldb/modules/ldb/sample.so 2022-04-03T22:52:45.874 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 15912 44157 /usr/lib64/ldb/modules/ldb/rdn_name.so 2022-04-03T22:52:45.874 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 15968 44156 /usr/lib64/ldb/modules/ldb/paged_searches.so 2022-04-03T22:52:45.875 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 70952 45685 /usr/lib64/ldb/modules/ldb/memberof.so 2022-04-03T22:52:45.875 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 7504 44155 /usr/lib64/ldb/modules/ldb/mdb.so 2022-04-03T22:52:45.875 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 91808 8746 /usr/lib64/liblmdb.so.0.0.0 2022-04-03T22:52:45.876 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 12360 44148 /usr/lib64/ldb/libldb-tdb-err-map.so 2022-04-03T22:52:45.876 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 95440 44146 /usr/lib64/ldb/libldb-key-value.so 2022-04-03T22:52:45.876 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 24704 44147 /usr/lib64/ldb/libldb-mdb-int.so 2022-04-03T22:52:45.877 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 24920 44149 /usr/lib64/ldb/libldb-tdb-int.so 2022-04-03T22:52:45.877 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 7608 44154 /usr/lib64/ldb/modules/ldb/ldb.so 2022-04-03T22:52:45.877 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:45.878 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:45.878 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:45.878 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:45.879 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:45.879 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:45.880 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:45.880 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:45.880 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 125328 8879 /usr/lib64/libsasl2.so.3.0.0 2022-04-03T22:52:45.881 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.881 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 67136 8940 /usr/lib64/liblber-2.4.so.2.10.9 2022-04-03T22:52:45.881 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 325184 8942 /usr/lib64/libldap-2.4.so.2.10.9 2022-04-03T22:52:45.882 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 28808 44153 /usr/lib64/ldb/modules/ldb/ldap.so 2022-04-03T22:52:45.882 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 15888 44152 /usr/lib64/ldb/modules/ldb/asq.so 2022-04-03T22:52:45.882 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:45.883 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.883 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.883 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.884 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.884 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.884 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.885 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.885 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.885 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.886 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.886 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.887 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:45.887 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.887 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.888 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.888 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.888 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.889 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.889 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.889 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.890 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 16160 8221 /usr/lib64/libpath_utils.so.1.0.1 2022-04-03T22:52:45.890 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.890 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.891 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:45.891 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.891 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.892 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 351464 9341 /usr/lib64/libdbus-1.so.3.19.7 2022-04-03T22:52:45.892 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 87648 7444 /usr/lib64/libtalloc.so.2.3.1 2022-04-03T22:52:45.892 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 83488 7526 /usr/lib64/libtevent.so.0.10.2 2022-04-03T22:52:45.893 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 16216 7635 /usr/lib64/libdhash.so.1.1.0 2022-04-03T22:52:45.893 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 266008 35861 /usr/lib64/sssd/libsss_sbus.so 2022-04-03T22:52:45.893 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 152328 35857 /usr/lib64/sssd/libsss_iface.so 2022-04-03T22:52:45.894 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.894 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 20664 35855 /usr/lib64/sssd/libsss_debug.so 2022-04-03T22:52:45.894 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.895 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 24208 35853 /usr/lib64/sssd/libsss_child.so 2022-04-03T22:52:45.895 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 28832 35854 /usr/lib64/sssd/libsss_crypt.so 2022-04-03T22:52:45.895 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 20368 35852 /usr/lib64/sssd/libsss_cert.so 2022-04-03T22:52:45.896 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 55560 7633 /usr/lib64/libcollection.so.4.1.1 2022-04-03T22:52:45.896 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 12024 7637 /usr/lib64/libref_array.so.1.2.1 2022-04-03T22:52:45.896 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 11848 7624 /usr/lib64/libbasicobjects.so.0.1.0 2022-04-03T22:52:45.897 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 117048 8223 /usr/lib64/libini_config.so.5.2.1 2022-04-03T22:52:45.897 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.897 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 100320 7532 /usr/lib64/libtdb.so.1.4.3 2022-04-03T22:52:45.898 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.898 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 220216 10551 /usr/lib64/libldb.so.2.2.0 2022-04-03T22:52:45.898 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 54688 7381 /usr/lib64/libpopt.so.0.0.1 2022-04-03T22:52:45.899 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.899 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 588576 35865 /usr/lib64/sssd/libsss_util.so 2022-04-03T22:52:45.899 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:45.900 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 94472 8687 /usr/lib64/libcares.so.2.2.0 2022-04-03T22:52:45.900 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.900 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.901 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 1286144 2754 /var/lib/sss/db/config.ldb 2022-04-03T22:52:45.901 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:45.901 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:45.902 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:45.902 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:45.902 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:45.903 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:45.903 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:45.903 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:45.904 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:45.904 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:45.904 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.905 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:45.905 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 0u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.905 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 1u unix 0xffff88810348e180 0t0 14157 type=STREAM 2022-04-03T22:52:45.906 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 2u unix 0xffff88810348e180 0t0 14157 type=STREAM 2022-04-03T22:52:45.906 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 3u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.906 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 4u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.907 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 5u REG 8,1 1286144 2754 /var/lib/sss/db/config.ldb 2022-04-03T22:52:45.907 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 6w REG 8,1 2807 2794 /var/log/sssd/sssd_implicit_files.log 2022-04-03T22:52:45.908 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.908 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 8r FIFO 0,12 0t0 23624 pipe 2022-04-03T22:52:45.908 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 9w FIFO 0,12 0t0 23624 pipe 2022-04-03T22:52:45.909 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 10u unix 0xffff8881050e8000 0t0 23625 type=DGRAM 2022-04-03T22:52:45.909 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.909 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 12u REG 8,1 3035136 2792 /var/lib/sss/db/cache_implicit_files.ldb 2022-04-03T22:52:45.910 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 13u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.910 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 14u REG 8,1 1609728 2793 /var/lib/sss/db/timestamps_implicit_files.ldb 2022-04-03T22:52:45.910 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 15u unix 0xffff8881050ebc00 0t0 23626 /var/lib/sss/pipes/private/sbus-dp_implicit_files.847 type=STREAM 2022-04-03T22:52:45.911 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 16u unix 0xffff8881050eb480 0t0 23627 type=STREAM 2022-04-03T22:52:45.911 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 17u unix 0xffff8881050e9680 0t0 23628 /var/lib/sss/pipes/private/sbus-dp_implicit_files.847 type=STREAM 2022-04-03T22:52:45.911 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 18r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.911 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 19r REG 8,1 11567160 72876 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:45.912 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 20r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.912 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 21r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.912 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 22u unix 0xffff888106ed7080 0t0 22817 type=STREAM 2022-04-03T22:52:45.913 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_be 847 root 23u unix 0xffff888103592580 0t0 16195 /var/lib/sss/pipes/private/sbus-dp_implicit_files.847 type=STREAM 2022-04-03T22:52:45.913 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.913 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.914 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root txt REG 8,1 314024 55041 /usr/libexec/sssd/sssd_nss 2022-04-03T22:52:45.914 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem-w REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:45.914 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem-w REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:45.915 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem-w REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.915 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root DEL REG 8,1 72874 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.915 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.916 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.916 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 1609728 2793 /var/lib/sss/db/timestamps_implicit_files.ldb 2022-04-03T22:52:45.916 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 3035136 2792 /var/lib/sss/db/cache_implicit_files.ldb 2022-04-03T22:52:45.917 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 7496 44161 /usr/lib64/ldb/modules/ldb/tdb.so 2022-04-03T22:52:45.917 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 11776 44160 /usr/lib64/ldb/modules/ldb/skel.so 2022-04-03T22:52:45.917 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 15856 44159 /usr/lib64/ldb/modules/ldb/server_sort.so 2022-04-03T22:52:45.918 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 11952 44158 /usr/lib64/ldb/modules/ldb/sample.so 2022-04-03T22:52:45.918 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 15912 44157 /usr/lib64/ldb/modules/ldb/rdn_name.so 2022-04-03T22:52:45.918 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 15968 44156 /usr/lib64/ldb/modules/ldb/paged_searches.so 2022-04-03T22:52:45.919 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 70952 45685 /usr/lib64/ldb/modules/ldb/memberof.so 2022-04-03T22:52:45.919 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 7504 44155 /usr/lib64/ldb/modules/ldb/mdb.so 2022-04-03T22:52:45.919 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 91808 8746 /usr/lib64/liblmdb.so.0.0.0 2022-04-03T22:52:45.920 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 12360 44148 /usr/lib64/ldb/libldb-tdb-err-map.so 2022-04-03T22:52:45.920 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 95440 44146 /usr/lib64/ldb/libldb-key-value.so 2022-04-03T22:52:45.920 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 24704 44147 /usr/lib64/ldb/libldb-mdb-int.so 2022-04-03T22:52:45.921 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 24920 44149 /usr/lib64/ldb/libldb-tdb-int.so 2022-04-03T22:52:45.921 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 7608 44154 /usr/lib64/ldb/modules/ldb/ldb.so 2022-04-03T22:52:45.921 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:45.922 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:45.922 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:45.922 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:45.923 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:45.923 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:45.923 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:45.924 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:45.924 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 125328 8879 /usr/lib64/libsasl2.so.3.0.0 2022-04-03T22:52:45.924 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:45.925 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 67136 8940 /usr/lib64/liblber-2.4.so.2.10.9 2022-04-03T22:52:45.925 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 325184 8942 /usr/lib64/libldap-2.4.so.2.10.9 2022-04-03T22:52:45.925 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 28808 44153 /usr/lib64/ldb/modules/ldb/ldap.so 2022-04-03T22:52:45.926 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 15888 44152 /usr/lib64/ldb/modules/ldb/asq.so 2022-04-03T22:52:45.926 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:45.926 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:45.927 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 1503456 7707 /usr/lib64/libgmp.so.10.3.2 2022-04-03T22:52:45.927 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 197512 7900 /usr/lib64/libhogweed.so.4.5 2022-04-03T22:52:45.927 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:45.928 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 81408 7932 /usr/lib64/libtasn1.so.6.5.5 2022-04-03T22:52:45.928 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.928 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.929 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 1246520 7786 /usr/lib64/libp11-kit.so.0.3.0 2022-04-03T22:52:45.929 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.929 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.930 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.930 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 2052224 10280 /usr/lib64/libgnutls.so.30.28.0 2022-04-03T22:52:45.930 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.931 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.931 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.931 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.932 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.932 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.932 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.933 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 16160 8221 /usr/lib64/libpath_utils.so.1.0.1 2022-04-03T22:52:45.933 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.933 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.934 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.934 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 1168024 10211 /usr/lib64/libglib-2.0.so.0.5600.4 2022-04-03T22:52:45.934 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 351464 9341 /usr/lib64/libdbus-1.so.3.19.7 2022-04-03T22:52:45.935 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 87648 7444 /usr/lib64/libtalloc.so.2.3.1 2022-04-03T22:52:45.935 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 83488 7526 /usr/lib64/libtevent.so.0.10.2 2022-04-03T22:52:45.935 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 16216 7635 /usr/lib64/libdhash.so.1.1.0 2022-04-03T22:52:45.936 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 266008 35861 /usr/lib64/sssd/libsss_sbus.so 2022-04-03T22:52:45.936 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 152328 35857 /usr/lib64/sssd/libsss_iface.so 2022-04-03T22:52:45.936 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:45.937 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 20664 35855 /usr/lib64/sssd/libsss_debug.so 2022-04-03T22:52:45.937 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.937 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 24208 35853 /usr/lib64/sssd/libsss_child.so 2022-04-03T22:52:45.937 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 28832 35854 /usr/lib64/sssd/libsss_crypt.so 2022-04-03T22:52:45.938 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 55560 7633 /usr/lib64/libcollection.so.4.1.1 2022-04-03T22:52:45.938 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 12024 7637 /usr/lib64/libref_array.so.1.2.1 2022-04-03T22:52:45.938 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 11848 7624 /usr/lib64/libbasicobjects.so.0.1.0 2022-04-03T22:52:45.939 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 117048 8223 /usr/lib64/libini_config.so.5.2.1 2022-04-03T22:52:45.939 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 471224 7534 /usr/lib64/libpcre.so.1.2.10 2022-04-03T22:52:45.939 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 100320 7532 /usr/lib64/libtdb.so.1.4.3 2022-04-03T22:52:45.940 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.940 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 220216 10551 /usr/lib64/libldb.so.2.2.0 2022-04-03T22:52:45.940 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 54688 7381 /usr/lib64/libpopt.so.0.0.1 2022-04-03T22:52:45.941 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.941 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 588576 35865 /usr/lib64/sssd/libsss_util.so 2022-04-03T22:52:45.942 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 20368 35852 /usr/lib64/sssd/libsss_cert.so 2022-04-03T22:52:45.942 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 28808 7709 /usr/lib64/libsss_idmap.so.0.5.1 2022-04-03T22:52:45.942 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.943 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.943 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 1286144 2754 /var/lib/sss/db/config.ldb 2022-04-03T22:52:45.943 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:45.944 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:45.944 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:45.944 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:45.945 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:45.945 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:45.945 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:45.946 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:45.946 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:45.946 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:45.947 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.947 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:45.947 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 0u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.948 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 1u unix 0xffff88810348e180 0t0 14157 type=STREAM 2022-04-03T22:52:45.948 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 2u unix 0xffff88810348e180 0t0 14157 type=STREAM 2022-04-03T22:52:45.948 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 3u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.949 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 4u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.949 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 5u REG 8,1 1286144 2754 /var/lib/sss/db/config.ldb 2022-04-03T22:52:45.949 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 6w REG 8,1 1677 2795 /var/log/sssd/sssd_nss.log 2022-04-03T22:52:45.950 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 7u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:45.950 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 8r FIFO 0,12 0t0 18839 pipe 2022-04-03T22:52:45.950 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 9w FIFO 0,12 0t0 18839 pipe 2022-04-03T22:52:45.951 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 10u unix 0xffff888103594380 0t0 18840 type=DGRAM 2022-04-03T22:52:45.951 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 11u unix 0xffff888103592d00 0t0 18841 type=STREAM 2022-04-03T22:52:45.951 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 12u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.952 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 13u REG 8,1 3035136 2792 /var/lib/sss/db/cache_implicit_files.ldb 2022-04-03T22:52:45.952 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 14u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.952 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 15u REG 8,1 1609728 2793 /var/lib/sss/db/timestamps_implicit_files.ldb 2022-04-03T22:52:45.953 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 16u unix 0xffff8881795b8f00 0t0 23658 /var/lib/sss/pipes/nss type=STREAM 2022-04-03T22:52:45.953 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 17r REG 8,1 9253600 72874 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.953 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 18uw REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.954 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 19uw REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:45.954 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 20uw REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:45.954 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 21u unix 0xffff8881795b9680 0t0 23661 type=STREAM 2022-04-03T22:52:45.955 INFO:teuthology.orchestra.run.smithi077.stdout:sssd_nss 868 root 22u unix 0xffff888178e4d280 0t0 199806 /var/lib/sss/pipes/nss type=STREAM 2022-04-03T22:52:45.955 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.955 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.956 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root txt REG 8,1 251008 10345 /usr/lib/systemd/systemd-logind 2022-04-03T22:52:45.956 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.956 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.957 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.957 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:45.957 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:45.958 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:45.958 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:45.958 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:45.959 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:45.959 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:45.959 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:45.960 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.960 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 312232 8919 /usr/lib64/libpcap.so.1.9.1 2022-04-03T22:52:45.960 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.961 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:45.961 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:45.961 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:45.962 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:45.962 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:45.962 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:45.963 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 26192 7582 /usr/lib64/libattr.so.1.1.2448 2022-04-03T22:52:45.963 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:45.963 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:45.964 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:45.964 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:45.964 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:45.965 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.965 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 132888 8225 /usr/lib64/libseccomp.so.2.5.1 2022-04-03T22:52:45.965 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 33328 8926 /usr/lib64/libip4tc.so.2.0.0 2022-04-03T22:52:45.966 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:45.966 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:45.966 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:45.967 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.967 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.967 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.968 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:45.968 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 54816 7584 /usr/lib64/libacl.so.1.1.2253 2022-04-03T22:52:45.968 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 2732744 10328 /usr/lib/systemd/libsystemd-shared-239.so 2022-04-03T22:52:45.969 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.969 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root DEL REG 8,1 1132 /etc/selinux/targeted/contexts/files/file_contexts.bin 2022-04-03T22:52:45.969 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root DEL REG 8,1 1133 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2022-04-03T22:52:45.970 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.970 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 1u unix 0xffff888178dfbc00 0t0 20081 type=STREAM 2022-04-03T22:52:45.970 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 2u unix 0xffff888178dfbc00 0t0 20081 type=STREAM 2022-04-03T22:52:45.971 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 3u unix 0xffff88816f708000 0t0 20082 type=DGRAM 2022-04-03T22:52:45.971 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 4u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:45.971 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 5u a_inode 0,13 0 9293 [signalfd] 2022-04-03T22:52:45.972 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 6u a_inode 0,13 0 9293 [timerfd] 2022-04-03T22:52:45.972 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 7r REG 0,20 4096 26990 /sys/devices/virtual/tty/tty0/active 2022-04-03T22:52:45.972 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 8u netlink 0t0 20088 KOBJECT_UEVENT 2022-04-03T22:52:45.973 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 9u netlink 0t0 20089 KOBJECT_UEVENT 2022-04-03T22:52:45.973 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 10u netlink 0t0 20090 KOBJECT_UEVENT 2022-04-03T22:52:45.973 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 11u netlink 0t0 20091 KOBJECT_UEVENT 2022-04-03T22:52:45.974 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 12u unix 0xffff88816f708f00 0t0 20092 type=STREAM 2022-04-03T22:52:45.974 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 13u a_inode 0,13 0 9293 [timerfd] 2022-04-03T22:52:45.975 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 14u CHR 13,65 0t0 136 /dev/input/event1 2022-04-03T22:52:45.975 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 15u CHR 13,64 0t0 135 /dev/input/event0 2022-04-03T22:52:45.975 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 16u CHR 13,66 0t0 158 /dev/input/event2 2022-04-03T22:52:45.976 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 17u CHR 4,6 0t0 25 /dev/tty6 2022-04-03T22:52:45.976 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 18r FIFO 0,24 0t0 1055 /run/systemd/inhibit/1.ref 2022-04-03T22:52:45.976 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 19u unix 0xffff88810cabcb00 0t0 176687 type=STREAM 2022-04-03T22:52:45.977 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 20r FIFO 0,24 0t0 1554 /run/systemd/sessions/4.ref 2022-04-03T22:52:45.977 INFO:teuthology.orchestra.run.smithi077.stdout:systemd-l 1052 root 21r REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.977 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.978 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.978 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root txt REG 8,1 99584 10631 /usr/sbin/crond 2022-04-03T22:52:45.978 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.979 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.979 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.979 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:45.980 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.980 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:45.980 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.981 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.981 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:45.981 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.982 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:45.982 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.983 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.983 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:45.983 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:45.984 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:45.984 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:45.984 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:45.984 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:45.985 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:45.985 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:45.986 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:45.986 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:45.986 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:45.987 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:45.987 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.987 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root 1u unix 0xffff88816f709e00 0t0 20169 type=STREAM 2022-04-03T22:52:45.987 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root 2u unix 0xffff88816f709e00 0t0 20169 type=STREAM 2022-04-03T22:52:45.988 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root 3u unix 0xffff88816f70ad00 0t0 20170 type=DGRAM 2022-04-03T22:52:45.988 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root 4r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:45.988 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root 5u unix 0xffff88810e620f00 0t0 181350 type=STREAM 2022-04-03T22:52:45.989 INFO:teuthology.orchestra.run.smithi077.stdout:crond 1054 root 6r REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.989 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root cwd DIR 8,1 4096 2483 /var/spool/at 2022-04-03T22:52:45.989 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.990 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root txt REG 8,1 34144 11618 /usr/sbin/atd 2022-04-03T22:52:45.990 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root DEL REG 8,1 72871 /var/lib/sss/mc/group 2022-04-03T22:52:45.990 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:45.991 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root DEL REG 8,1 3014 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.991 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.991 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:45.992 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:45.992 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:45.992 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:45.993 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:45.993 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:45.993 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:45.994 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root mem REG 8,1 16536 9706 /usr/lib64/libpam_misc.so.0.82.1 2022-04-03T22:52:45.994 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:45.994 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:45.995 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:45.995 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.995 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.996 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:45.996 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root 3r REG 8,1 9253600 3014 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:45.996 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root 4u unix 0xffff888106ed2d00 0t0 22977 type=STREAM 2022-04-03T22:52:45.997 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root 5r REG 8,1 6940392 72871 /var/lib/sss/mc/group (deleted) 2022-04-03T22:52:45.997 INFO:teuthology.orchestra.run.smithi077.stdout:atd 1057 root 6uW REG 0,24 5 1073 /run/atd.pid 2022-04-03T22:52:45.997 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe cwd DIR 8,1 4096 2 / 2022-04-03T22:52:45.998 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe rtd DIR 8,1 4096 2 / 2022-04-03T22:52:45.998 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe txt REG 8,1 82448 14866 /usr/sbin/nrpe 2022-04-03T22:52:45.998 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe DEL REG 8,1 72871 /var/lib/sss/mc/group 2022-04-03T22:52:45.999 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe DEL REG 8,1 3014 /var/lib/sss/mc/passwd 2022-04-03T22:52:45.999 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:45.999 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.000 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.000 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.000 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.001 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.001 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:46.001 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.002 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.002 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe 1w CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.002 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe 2w CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.003 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe 3r REG 8,1 9253600 3014 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:46.003 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe 4u unix 0xffff888178e4f080 0t0 21794 type=DGRAM 2022-04-03T22:52:46.004 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe 5r REG 8,1 6940392 72871 /var/lib/sss/mc/group (deleted) 2022-04-03T22:52:46.004 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe 6u IPv4 21802 0t0 TCP *:nrpe (LISTEN) 2022-04-03T22:52:46.004 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 nrpe 7u IPv6 21803 0t0 TCP *:nrpe (LISTEN) 2022-04-03T22:52:46.005 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1114 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.005 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1114 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.006 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1114 root txt REG 8,1 64016 9849 /usr/sbin/agetty 2022-04-03T22:52:46.006 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1114 root DEL REG 8,1 72871 /var/lib/sss/mc/group 2022-04-03T22:52:46.006 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1114 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:46.007 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1114 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.007 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1114 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.007 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1114 root mem REG 8,1 337024 21558 /usr/lib/locale/C.utf8/LC_CTYPE 2022-04-03T22:52:46.008 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1114 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:46.008 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1114 root 0u CHR 4,1 0t0 20 /dev/tty1 2022-04-03T22:52:46.008 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1114 root 1u CHR 4,1 0t0 20 /dev/tty1 2022-04-03T22:52:46.009 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1114 root 2u CHR 4,1 0t0 20 /dev/tty1 2022-04-03T22:52:46.009 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1114 root 3r REG 8,1 6940392 72871 /var/lib/sss/mc/group (deleted) 2022-04-03T22:52:46.009 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1114 root 4u unix 0xffff888103590000 0t0 18992 type=STREAM 2022-04-03T22:52:46.010 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1114 root 6r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:46.010 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.010 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.011 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root txt REG 8,1 64016 9849 /usr/sbin/agetty 2022-04-03T22:52:46.011 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root DEL REG 8,1 72871 /var/lib/sss/mc/group 2022-04-03T22:52:46.011 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:46.012 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:46.012 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.012 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.013 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:46.013 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:46.013 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:46.014 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:46.014 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:46.015 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:46.015 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:46.015 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:46.016 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:46.016 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:46.016 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:46.017 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:46.017 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root 0u CHR 4,65 0t0 112 /dev/ttyS1 2022-04-03T22:52:46.017 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root 1u CHR 4,65 0t0 112 /dev/ttyS1 2022-04-03T22:52:46.018 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root 2u CHR 4,65 0t0 112 /dev/ttyS1 2022-04-03T22:52:46.018 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root 3r REG 8,1 6940392 72871 /var/lib/sss/mc/group (deleted) 2022-04-03T22:52:46.018 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root 4u unix 0xffff88810eb12d00 0t0 25910 type=STREAM 2022-04-03T22:52:46.019 INFO:teuthology.orchestra.run.smithi077.stdout:agetty 1115 root 6r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:46.019 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.019 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.020 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu txt REG 8,1 1588952 10333 /usr/lib/systemd/systemd 2022-04-03T22:52:46.020 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.020 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.021 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.021 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.021 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.022 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:46.022 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:46.022 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 312232 8919 /usr/lib64/libpcap.so.1.9.1 2022-04-03T22:52:46.023 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:46.023 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:46.023 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:46.024 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 26192 7582 /usr/lib64/libattr.so.1.1.2448 2022-04-03T22:52:46.024 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.024 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:46.025 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.025 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:46.025 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.026 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.026 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.027 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.027 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.027 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:46.027 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:46.028 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 33328 8926 /usr/lib64/libip4tc.so.2.0.0 2022-04-03T22:52:46.028 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:46.028 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:46.029 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 54816 7584 /usr/lib64/libacl.so.1.1.2253 2022-04-03T22:52:46.029 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:46.029 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.030 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.030 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.030 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 100208 9562 /usr/lib64/libkmod.so.2.3.3 2022-04-03T22:52:46.031 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:46.031 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:46.031 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.032 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.032 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 132888 8225 /usr/lib64/libseccomp.so.2.5.1 2022-04-03T22:52:46.032 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.033 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 2732744 10328 /usr/lib/systemd/libsystemd-shared-239.so 2022-04-03T22:52:46.033 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.033 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.034 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 1u unix 0xffff8881795bda00 0t0 23886 type=STREAM 2022-04-03T22:52:46.034 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 2u unix 0xffff8881795bda00 0t0 23886 type=STREAM 2022-04-03T22:52:46.034 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 3u unix 0xffff8881795bd280 0t0 23903 type=DGRAM 2022-04-03T22:52:46.035 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 4u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.035 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 5u a_inode 0,13 0 9293 [signalfd] 2022-04-03T22:52:46.035 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 6r DIR 0,26 0 798 /sys/fs/cgroup/systemd/user.slice/user-1000.slice/user@1000.service 2022-04-03T22:52:46.036 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 7u a_inode 0,13 0 9293 [timerfd] 2022-04-03T22:52:46.036 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 8u netlink 0t0 23905 KOBJECT_UEVENT 2022-04-03T22:52:46.036 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 9u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.037 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 10r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:46.037 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 11u unix 0xffff8881795b8780 0t0 23921 /run/user/1000/bus type=STREAM 2022-04-03T22:52:46.037 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 12r REG 0,21 0 23910 /proc/1130/mountinfo 2022-04-03T22:52:46.038 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 13r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:46.038 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 14r REG 0,21 0 4026532078 /proc/swaps 2022-04-03T22:52:46.038 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 15u unix 0xffff8881795bcb00 0t0 23911 /run/user/1000/systemd/notify type=DGRAM 2022-04-03T22:52:46.039 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 16u unix 0xffff8881795bb480 0t0 23912 type=DGRAM 2022-04-03T22:52:46.039 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 17u unix 0xffff8881795bad00 0t0 23913 type=DGRAM 2022-04-03T22:52:46.039 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 18u unix 0xffff8881795bc380 0t0 23914 /run/user/1000/systemd/private type=STREAM 2022-04-03T22:52:46.040 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 19u unix 0xffff8881795bbc00 0t0 23915 type=STREAM 2022-04-03T22:52:46.040 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 20u a_inode 0,13 0 9293 [timerfd] 2022-04-03T22:52:46.040 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 1130 ubuntu 26u netlink 0t0 23923 SELINUX 2022-04-03T22:52:46.041 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.041 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.041 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu txt REG 8,1 1588952 10333 /usr/lib/systemd/systemd 2022-04-03T22:52:46.042 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 7552 23271 /usr/lib64/security/pam_deny.so 2022-04-03T22:52:46.042 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 15928 23305 /usr/lib64/security/pam_succeed_if.so 2022-04-03T22:52:46.042 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 16536 9706 /usr/lib64/libpam_misc.so.0.82.1 2022-04-03T22:52:46.043 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 915232 23524 /usr/lib64/security/pam_systemd.so 2022-04-03T22:52:46.043 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 24048 23285 /usr/lib64/security/pam_limits.so 2022-04-03T22:52:46.043 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 11736 23283 /usr/lib64/security/pam_keyinit.so 2022-04-03T22:52:46.044 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 11760 23288 /usr/lib64/security/pam_loginuid.so 2022-04-03T22:52:46.044 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 24168 23300 /usr/lib64/security/pam_selinux.so 2022-04-03T22:52:46.044 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 7640 23294 /usr/lib64/security/pam_permit.so 2022-04-03T22:52:46.045 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 58032 24580 /usr/lib64/security/pam_sss.so 2022-04-03T22:52:46.045 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 11816 23316 /usr/lib64/security/pam_usertype.so 2022-04-03T22:52:46.045 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 11744 23287 /usr/lib64/security/pam_localuser.so 2022-04-03T22:52:46.046 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.046 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:46.046 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:46.047 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:46.047 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:46.047 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:46.048 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:46.048 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 371032 9256 /usr/lib64/libtirpc.so.3.0.0 2022-04-03T22:52:46.048 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 116408 9258 /usr/lib64/libnsl.so.2.0.0 2022-04-03T22:52:46.049 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:46.049 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 58016 23310 /usr/lib64/security/pam_unix.so 2022-04-03T22:52:46.049 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:46.049 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:46.050 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu DEL REG 8,1 72872 /var/lib/sss/mc/initgroups 2022-04-03T22:52:46.050 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu DEL REG 8,1 3014 /var/lib/sss/mc/passwd 2022-04-03T22:52:46.050 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:46.051 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.051 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.051 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.052 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.052 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.052 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:46.053 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:46.053 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 312232 8919 /usr/lib64/libpcap.so.1.9.1 2022-04-03T22:52:46.053 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:46.054 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:46.054 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:46.054 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 26192 7582 /usr/lib64/libattr.so.1.1.2448 2022-04-03T22:52:46.055 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.055 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:46.055 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.056 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:46.056 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.056 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.057 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.057 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.057 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.058 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:46.058 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:46.059 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 33328 8926 /usr/lib64/libip4tc.so.2.0.0 2022-04-03T22:52:46.059 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:46.059 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:46.060 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 54816 7584 /usr/lib64/libacl.so.1.1.2253 2022-04-03T22:52:46.060 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:46.060 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.060 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.061 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.061 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 100208 9562 /usr/lib64/libkmod.so.2.3.3 2022-04-03T22:52:46.061 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:46.062 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:46.062 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.063 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.063 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 132888 8225 /usr/lib64/libseccomp.so.2.5.1 2022-04-03T22:52:46.063 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.063 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 2732744 10328 /usr/lib/systemd/libsystemd-shared-239.so 2022-04-03T22:52:46.064 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.064 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu DEL REG 8,1 1132 /etc/selinux/targeted/contexts/files/file_contexts.bin 2022-04-03T22:52:46.065 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu DEL REG 8,1 1133 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2022-04-03T22:52:46.065 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.065 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu 1u unix 0xffff8881795bda00 0t0 23886 type=STREAM 2022-04-03T22:52:46.065 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu 2u unix 0xffff8881795bda00 0t0 23886 type=STREAM 2022-04-03T22:52:46.066 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu 3r REG 8,1 9253600 3014 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:46.066 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu 4r REG 8,1 11567160 72872 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:46.066 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu 5u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:46.067 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu 6u a_inode 0,13 0 9293 [eventfd] 2022-04-03T22:52:46.067 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu 8w FIFO 0,12 0t0 23887 pipe 2022-04-03T22:52:46.067 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu 9u unix 0xffff8881795be900 0t0 23899 type=STREAM 2022-04-03T22:52:46.068 INFO:teuthology.orchestra.run.smithi077.stdout:(sd-pam) 1134 ubuntu 10u unix 0xffff8881795bf800 0t0 23900 type=DGRAM 2022-04-03T22:52:46.068 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.068 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.069 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root txt REG 8,1 733240 10809 /usr/sbin/rsyslogd 2022-04-03T22:52:46.069 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 0,24 16777216 50 /run/log/journal/545941180e054f2c980bf1100aaa9165/system.journal 2022-04-03T22:52:46.069 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 34200 10779 /usr/lib64/rsyslog/imjournal.so 2022-04-03T22:52:46.070 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 38784 10786 /usr/lib64/rsyslog/imuxsock.so 2022-04-03T22:52:46.070 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 29304 10787 /usr/lib64/rsyslog/lmnet.so 2022-04-03T22:52:46.070 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.071 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:46.071 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.071 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.072 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:46.072 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.072 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:46.073 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.073 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:46.073 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.074 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.074 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.074 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:46.075 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 48456 8858 /usr/lib64/libfastjson.so.4.2.0 2022-04-03T22:52:46.075 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 16448 8856 /usr/lib64/libestr.so.0.0.0 2022-04-03T22:52:46.075 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.076 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.076 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.076 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.077 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.077 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.077 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root 1w CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.078 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root 2w CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.078 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root 3r CHR 1,9 0t0 9 /dev/urandom 2022-04-03T22:52:46.078 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root 4u unix 0xffff88810907cb00 0t0 26111 type=DGRAM 2022-04-03T22:52:46.079 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root 5w REG 8,1 1670534 524306 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-04-03T22:52:46.079 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root 6r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:46.079 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root 7w REG 8,1 2508 2567 /var/log/cron 2022-04-03T22:52:46.079 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root 8r REG 0,24 16777216 50 /run/log/journal/545941180e054f2c980bf1100aaa9165/system.journal 2022-04-03T22:52:46.080 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root 9w REG 8,1 864839 2469 /var/log/secure 2022-04-03T22:52:46.080 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root 10w REG 8,1 118865 524305 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-04-03T22:52:46.080 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 root 11w REG 8,1 1931767 2468 /var/log/messages 2022-04-03T22:52:46.081 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.081 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.081 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root txt REG 8,1 733240 10809 /usr/sbin/rsyslogd 2022-04-03T22:52:46.082 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 0,24 16777216 50 /run/log/journal/545941180e054f2c980bf1100aaa9165/system.journal 2022-04-03T22:52:46.082 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 34200 10779 /usr/lib64/rsyslog/imjournal.so 2022-04-03T22:52:46.082 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 38784 10786 /usr/lib64/rsyslog/imuxsock.so 2022-04-03T22:52:46.083 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 29304 10787 /usr/lib64/rsyslog/lmnet.so 2022-04-03T22:52:46.083 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.083 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:46.084 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.084 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.084 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:46.085 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.085 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:46.085 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.086 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:46.086 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.086 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.087 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.087 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:46.087 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 48456 8858 /usr/lib64/libfastjson.so.4.2.0 2022-04-03T22:52:46.088 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 16448 8856 /usr/lib64/libestr.so.0.0.0 2022-04-03T22:52:46.088 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.088 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.089 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.089 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.089 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.089 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.090 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root 1w CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.090 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root 2w CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.090 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root 3r CHR 1,9 0t0 9 /dev/urandom 2022-04-03T22:52:46.091 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root 4u unix 0xffff88810907cb00 0t0 26111 type=DGRAM 2022-04-03T22:52:46.091 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root 5w REG 8,1 1670534 524306 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-04-03T22:52:46.091 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root 6r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:46.092 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root 7w REG 8,1 2508 2567 /var/log/cron 2022-04-03T22:52:46.092 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root 8r REG 0,24 16777216 50 /run/log/journal/545941180e054f2c980bf1100aaa9165/system.journal 2022-04-03T22:52:46.092 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root 9w REG 8,1 864839 2469 /var/log/secure 2022-04-03T22:52:46.093 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root 10w REG 8,1 118865 524305 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-04-03T22:52:46.093 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1370 in:imjour root 11w REG 8,1 1931767 2468 /var/log/messages 2022-04-03T22:52:46.093 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.093 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.094 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root txt REG 8,1 733240 10809 /usr/sbin/rsyslogd 2022-04-03T22:52:46.094 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 0,24 16777216 50 /run/log/journal/545941180e054f2c980bf1100aaa9165/system.journal 2022-04-03T22:52:46.094 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 34200 10779 /usr/lib64/rsyslog/imjournal.so 2022-04-03T22:52:46.095 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 38784 10786 /usr/lib64/rsyslog/imuxsock.so 2022-04-03T22:52:46.095 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 29304 10787 /usr/lib64/rsyslog/lmnet.so 2022-04-03T22:52:46.095 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.096 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:46.096 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.096 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.097 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:46.097 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.097 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:46.098 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.098 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:46.098 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.099 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.099 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.099 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:46.100 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 48456 8858 /usr/lib64/libfastjson.so.4.2.0 2022-04-03T22:52:46.100 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 16448 8856 /usr/lib64/libestr.so.0.0.0 2022-04-03T22:52:46.100 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.101 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.101 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.101 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.102 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.102 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.102 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root 1w CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.103 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root 2w CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.103 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root 3r CHR 1,9 0t0 9 /dev/urandom 2022-04-03T22:52:46.103 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root 4u unix 0xffff88810907cb00 0t0 26111 type=DGRAM 2022-04-03T22:52:46.104 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root 5w REG 8,1 1670534 524306 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-04-03T22:52:46.104 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root 6r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:46.104 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root 7w REG 8,1 2508 2567 /var/log/cron 2022-04-03T22:52:46.105 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root 8r REG 0,24 16777216 50 /run/log/journal/545941180e054f2c980bf1100aaa9165/system.journal 2022-04-03T22:52:46.105 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root 9w REG 8,1 864839 2469 /var/log/secure 2022-04-03T22:52:46.105 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root 10w REG 8,1 118865 524305 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-04-03T22:52:46.106 INFO:teuthology.orchestra.run.smithi077.stdout:rsyslogd 1366 1371 rs:main root 11w REG 8,1 1931767 2468 /var/log/messages 2022-04-03T22:52:46.106 INFO:teuthology.orchestra.run.smithi077.stdout:kdmflush/ 6378 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.106 INFO:teuthology.orchestra.run.smithi077.stdout:kdmflush/ 6378 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.107 INFO:teuthology.orchestra.run.smithi077.stdout:kdmflush/ 6378 root txt unknown /proc/6378/exe 2022-04-03T22:52:46.107 INFO:teuthology.orchestra.run.smithi077.stdout:kdmflush/ 6411 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.107 INFO:teuthology.orchestra.run.smithi077.stdout:kdmflush/ 6411 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.108 INFO:teuthology.orchestra.run.smithi077.stdout:kdmflush/ 6411 root txt unknown /proc/6411/exe 2022-04-03T22:52:46.108 INFO:teuthology.orchestra.run.smithi077.stdout:kdmflush/ 6444 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.108 INFO:teuthology.orchestra.run.smithi077.stdout:kdmflush/ 6444 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.109 INFO:teuthology.orchestra.run.smithi077.stdout:kdmflush/ 6444 root txt unknown /proc/6444/exe 2022-04-03T22:52:46.109 INFO:teuthology.orchestra.run.smithi077.stdout:kdmflush/ 6477 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.109 INFO:teuthology.orchestra.run.smithi077.stdout:kdmflush/ 6477 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.110 INFO:teuthology.orchestra.run.smithi077.stdout:kdmflush/ 6477 root txt unknown /proc/6477/exe 2022-04-03T22:52:46.110 INFO:teuthology.orchestra.run.smithi077.stdout:kdmflush/ 6510 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.110 INFO:teuthology.orchestra.run.smithi077.stdout:kdmflush/ 6510 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.111 INFO:teuthology.orchestra.run.smithi077.stdout:kdmflush/ 6510 root txt unknown /proc/6510/exe 2022-04-03T22:52:46.111 INFO:teuthology.orchestra.run.smithi077.stdout:xfsalloc 6690 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.111 INFO:teuthology.orchestra.run.smithi077.stdout:xfsalloc 6690 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.112 INFO:teuthology.orchestra.run.smithi077.stdout:xfsalloc 6690 root txt unknown /proc/6690/exe 2022-04-03T22:52:46.112 INFO:teuthology.orchestra.run.smithi077.stdout:xfs_mru_c 6691 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.112 INFO:teuthology.orchestra.run.smithi077.stdout:xfs_mru_c 6691 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.113 INFO:teuthology.orchestra.run.smithi077.stdout:xfs_mru_c 6691 root txt unknown /proc/6691/exe 2022-04-03T22:52:46.113 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-buf/d 6693 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.113 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-buf/d 6693 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.114 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-buf/d 6693 root txt unknown /proc/6693/exe 2022-04-03T22:52:46.114 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-conv/ 6694 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.114 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-conv/ 6694 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.115 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-conv/ 6694 root txt unknown /proc/6694/exe 2022-04-03T22:52:46.115 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-recla 6695 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.115 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-recla 6695 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.116 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-recla 6695 root txt unknown /proc/6695/exe 2022-04-03T22:52:46.116 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-block 6696 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.116 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-block 6696 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.116 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-block 6696 root txt unknown /proc/6696/exe 2022-04-03T22:52:46.117 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-inode 6697 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.117 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-inode 6697 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.117 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-inode 6697 root txt unknown /proc/6697/exe 2022-04-03T22:52:46.118 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-log/d 6698 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.118 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-log/d 6698 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.118 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-log/d 6698 root txt unknown /proc/6698/exe 2022-04-03T22:52:46.119 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-cil/d 6699 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.119 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-cil/d 6699 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.119 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-cil/d 6699 root txt unknown /proc/6699/exe 2022-04-03T22:52:46.120 INFO:teuthology.orchestra.run.smithi077.stdout:xfsaild/d 6700 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.120 INFO:teuthology.orchestra.run.smithi077.stdout:xfsaild/d 6700 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.120 INFO:teuthology.orchestra.run.smithi077.stdout:xfsaild/d 6700 root txt unknown /proc/6700/exe 2022-04-03T22:52:46.121 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.121 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.121 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root txt REG 8,1 877984 13329 /usr/sbin/sshd 2022-04-03T22:52:46.122 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:46.122 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:46.122 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root DEL REG 8,1 72872 /var/lib/sss/mc/initgroups 2022-04-03T22:52:46.123 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 20128 23284 /usr/lib64/security/pam_lastlog.so 2022-04-03T22:52:46.123 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 11728 23309 /usr/lib64/security/pam_umask.so 2022-04-03T22:52:46.123 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 15928 23305 /usr/lib64/security/pam_succeed_if.so 2022-04-03T22:52:46.124 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 16536 9706 /usr/lib64/libpam_misc.so.0.82.1 2022-04-03T22:52:46.124 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 915232 23524 /usr/lib64/security/pam_systemd.so 2022-04-03T22:52:46.124 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 24048 23285 /usr/lib64/security/pam_limits.so 2022-04-03T22:52:46.125 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 11840 23291 /usr/lib64/security/pam_motd.so 2022-04-03T22:52:46.125 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 11736 23283 /usr/lib64/security/pam_keyinit.so 2022-04-03T22:52:46.125 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 45312 23292 /usr/lib64/security/pam_namespace.so 2022-04-03T22:52:46.126 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 11760 23288 /usr/lib64/security/pam_loginuid.so 2022-04-03T22:52:46.126 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 24168 23300 /usr/lib64/security/pam_selinux.so 2022-04-03T22:52:46.126 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 46232 9162 /usr/lib64/libcrack.so.2.9.0 2022-04-03T22:52:46.127 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 24744 9666 /usr/lib64/libpwquality.so.1.0.2 2022-04-03T22:52:46.127 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 11720 23217 /usr/lib64/security/pam_pwquality.so 2022-04-03T22:52:46.127 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 7640 23294 /usr/lib64/security/pam_permit.so 2022-04-03T22:52:46.128 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 11768 23293 /usr/lib64/security/pam_nologin.so 2022-04-03T22:52:46.128 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 15880 23302 /usr/lib64/security/pam_sepermit.so 2022-04-03T22:52:46.128 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 7552 23271 /usr/lib64/security/pam_deny.so 2022-04-03T22:52:46.129 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 58032 24580 /usr/lib64/security/pam_sss.so 2022-04-03T22:52:46.129 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 371032 9256 /usr/lib64/libtirpc.so.3.0.0 2022-04-03T22:52:46.129 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 116408 9258 /usr/lib64/libnsl.so.2.0.0 2022-04-03T22:52:46.130 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 58016 23310 /usr/lib64/security/pam_unix.so 2022-04-03T22:52:46.130 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 11744 23287 /usr/lib64/security/pam_localuser.so 2022-04-03T22:52:46.130 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 11816 23316 /usr/lib64/security/pam_usertype.so 2022-04-03T22:52:46.131 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 11720 23275 /usr/lib64/security/pam_faildelay.so 2022-04-03T22:52:46.131 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 20160 23273 /usr/lib64/security/pam_env.so 2022-04-03T22:52:46.131 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root DEL REG 8,1 72871 /var/lib/sss/mc/group 2022-04-03T22:52:46.132 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root DEL REG 8,1 3014 /var/lib/sss/mc/passwd 2022-04-03T22:52:46.132 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:46.132 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:46.132 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.133 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.133 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:46.133 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:46.134 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.134 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.134 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:46.135 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.135 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:46.135 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.136 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:46.136 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.136 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.137 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:46.137 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.137 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:46.138 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:46.138 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:46.138 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:46.139 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.139 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.139 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:46.139 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.140 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:46.140 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.140 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.141 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:46.141 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:46.141 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:46.142 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.142 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.142 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.143 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.143 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root 3u unix 0xffff888103f8cb00 0t0 61170 type=DGRAM 2022-04-03T22:52:46.143 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root 4r REG 8,1 9253600 3014 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:46.144 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root 5u IPv4 62511 0t0 TCP smithi077.front.sepia.ceph.com:ssh->teuthology.front.sepia.ceph.com:40226 (ESTABLISHED) 2022-04-03T22:52:46.144 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root 6u unix 0xffff888103f8f080 0t0 61142 type=STREAM 2022-04-03T22:52:46.144 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root 7r REG 8,1 11567160 72872 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:46.145 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root 8r REG 8,1 6940392 72871 /var/lib/sss/mc/group (deleted) 2022-04-03T22:52:46.145 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root 9u unix 0xffff888103f8e900 0t0 61146 type=STREAM 2022-04-03T22:52:46.145 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root 11w FIFO 0,24 0t0 1554 /run/systemd/sessions/4.ref 2022-04-03T22:52:46.145 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7894 root 12u unix 0xffff888103f8c380 0t0 61174 type=STREAM 2022-04-03T22:52:46.146 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.146 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.147 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu txt REG 8,1 877984 13329 /usr/sbin/sshd 2022-04-03T22:52:46.147 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:46.147 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:46.148 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu DEL REG 8,1 72872 /var/lib/sss/mc/initgroups 2022-04-03T22:52:46.148 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 20128 23284 /usr/lib64/security/pam_lastlog.so 2022-04-03T22:52:46.148 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 11728 23309 /usr/lib64/security/pam_umask.so 2022-04-03T22:52:46.149 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 15928 23305 /usr/lib64/security/pam_succeed_if.so 2022-04-03T22:52:46.149 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 16536 9706 /usr/lib64/libpam_misc.so.0.82.1 2022-04-03T22:52:46.149 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 915232 23524 /usr/lib64/security/pam_systemd.so 2022-04-03T22:52:46.149 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 24048 23285 /usr/lib64/security/pam_limits.so 2022-04-03T22:52:46.150 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 11840 23291 /usr/lib64/security/pam_motd.so 2022-04-03T22:52:46.150 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 11736 23283 /usr/lib64/security/pam_keyinit.so 2022-04-03T22:52:46.150 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 45312 23292 /usr/lib64/security/pam_namespace.so 2022-04-03T22:52:46.151 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 11760 23288 /usr/lib64/security/pam_loginuid.so 2022-04-03T22:52:46.151 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 24168 23300 /usr/lib64/security/pam_selinux.so 2022-04-03T22:52:46.151 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 46232 9162 /usr/lib64/libcrack.so.2.9.0 2022-04-03T22:52:46.152 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 24744 9666 /usr/lib64/libpwquality.so.1.0.2 2022-04-03T22:52:46.152 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 11720 23217 /usr/lib64/security/pam_pwquality.so 2022-04-03T22:52:46.152 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 7640 23294 /usr/lib64/security/pam_permit.so 2022-04-03T22:52:46.153 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 11768 23293 /usr/lib64/security/pam_nologin.so 2022-04-03T22:52:46.153 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 15880 23302 /usr/lib64/security/pam_sepermit.so 2022-04-03T22:52:46.153 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 7552 23271 /usr/lib64/security/pam_deny.so 2022-04-03T22:52:46.154 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 58032 24580 /usr/lib64/security/pam_sss.so 2022-04-03T22:52:46.154 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 371032 9256 /usr/lib64/libtirpc.so.3.0.0 2022-04-03T22:52:46.154 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 116408 9258 /usr/lib64/libnsl.so.2.0.0 2022-04-03T22:52:46.155 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 58016 23310 /usr/lib64/security/pam_unix.so 2022-04-03T22:52:46.155 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 11744 23287 /usr/lib64/security/pam_localuser.so 2022-04-03T22:52:46.155 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 11816 23316 /usr/lib64/security/pam_usertype.so 2022-04-03T22:52:46.156 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 11720 23275 /usr/lib64/security/pam_faildelay.so 2022-04-03T22:52:46.156 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 20160 23273 /usr/lib64/security/pam_env.so 2022-04-03T22:52:46.156 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu DEL REG 8,1 72871 /var/lib/sss/mc/group 2022-04-03T22:52:46.157 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu DEL REG 8,1 3014 /var/lib/sss/mc/passwd 2022-04-03T22:52:46.157 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:46.157 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:46.157 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.158 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.158 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:46.158 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:46.159 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.159 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.159 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:46.160 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.160 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:46.160 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.161 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:46.161 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.161 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.162 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:46.162 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.162 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:46.163 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:46.163 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:46.163 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:46.164 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.164 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.164 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:46.165 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.165 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:46.165 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.166 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.166 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 1367600 9458 /usr/lib64/libsystemd.so.0.23.0 2022-04-03T22:52:46.166 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:46.167 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:46.167 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.167 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 0u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.168 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 1u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.168 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.168 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 3u unix 0xffff888103f8cb00 0t0 61170 type=DGRAM 2022-04-03T22:52:46.168 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 4r REG 8,1 9253600 3014 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:46.169 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 5u IPv4 62511 0t0 TCP smithi077.front.sepia.ceph.com:ssh->teuthology.front.sepia.ceph.com:40226 (ESTABLISHED) 2022-04-03T22:52:46.169 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 6u unix 0xffff888103f8f080 0t0 61142 type=STREAM 2022-04-03T22:52:46.169 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 7r REG 8,1 11567160 72872 /var/lib/sss/mc/initgroups (deleted) 2022-04-03T22:52:46.170 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 8r REG 8,1 6940392 72871 /var/lib/sss/mc/group (deleted) 2022-04-03T22:52:46.170 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 9u unix 0xffff888103f8e900 0t0 61146 type=STREAM 2022-04-03T22:52:46.170 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 10u unix 0xffff888103f89e00 0t0 61173 type=STREAM 2022-04-03T22:52:46.171 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 11w FIFO 0,24 0t0 1554 /run/systemd/sessions/4.ref 2022-04-03T22:52:46.171 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 12r FIFO 0,12 0t0 54071 pipe 2022-04-03T22:52:46.171 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 13w FIFO 0,12 0t0 54071 pipe 2022-04-03T22:52:46.172 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 17w FIFO 0,12 0t0 78419 pipe 2022-04-03T22:52:46.172 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 18r FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.172 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 19w FIFO 0,12 0t0 78423 pipe 2022-04-03T22:52:46.173 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 20r FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.173 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 21r FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:46.173 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 22w FIFO 0,12 0t0 80529 pipe 2022-04-03T22:52:46.173 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 23r FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:46.174 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 24r FIFO 0,12 0t0 80530 pipe 2022-04-03T22:52:46.174 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 25w FIFO 0,12 0t0 79452 pipe 2022-04-03T22:52:46.174 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 26r FIFO 0,12 0t0 80531 pipe 2022-04-03T22:52:46.175 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 27r FIFO 0,12 0t0 79453 pipe 2022-04-03T22:52:46.175 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 28w FIFO 0,12 0t0 79458 pipe 2022-04-03T22:52:46.175 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 29r FIFO 0,12 0t0 79454 pipe 2022-04-03T22:52:46.176 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 30r FIFO 0,12 0t0 79459 pipe 2022-04-03T22:52:46.176 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 31w FIFO 0,12 0t0 81689 pipe 2022-04-03T22:52:46.176 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 32r FIFO 0,12 0t0 79460 pipe 2022-04-03T22:52:46.177 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 33r FIFO 0,12 0t0 81690 pipe 2022-04-03T22:52:46.177 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 34w FIFO 0,12 0t0 79823 pipe 2022-04-03T22:52:46.177 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 35r FIFO 0,12 0t0 81691 pipe 2022-04-03T22:52:46.178 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 36r FIFO 0,12 0t0 79824 pipe 2022-04-03T22:52:46.178 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 37w FIFO 0,12 0t0 79827 pipe 2022-04-03T22:52:46.178 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 38r FIFO 0,12 0t0 79825 pipe 2022-04-03T22:52:46.179 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 39r FIFO 0,12 0t0 79828 pipe 2022-04-03T22:52:46.179 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 40w FIFO 0,12 0t0 139391 pipe 2022-04-03T22:52:46.179 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 41r FIFO 0,12 0t0 79829 pipe 2022-04-03T22:52:46.180 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 42r FIFO 0,12 0t0 139392 pipe 2022-04-03T22:52:46.180 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 44r FIFO 0,12 0t0 139393 pipe 2022-04-03T22:52:46.180 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 45r FIFO 0,12 0t0 125987 pipe 2022-04-03T22:52:46.181 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 46r FIFO 0,12 0t0 137832 pipe 2022-04-03T22:52:46.181 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 47r FIFO 0,12 0t0 125988 pipe 2022-04-03T22:52:46.181 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 48r FIFO 0,12 0t0 194926 pipe 2022-04-03T22:52:46.181 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 49r FIFO 0,12 0t0 137833 pipe 2022-04-03T22:52:46.182 INFO:teuthology.orchestra.run.smithi077.stdout:sshd 7899 ubuntu 51r FIFO 0,12 0t0 194927 pipe 2022-04-03T22:52:46.182 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.182 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.183 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony txt REG 8,1 275232 13244 /usr/sbin/chronyd 2022-04-03T22:52:46.183 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.183 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony mem REG 8,1 35472 7243 /usr/lib64/libnss_dns-2.28.so 2022-04-03T22:52:46.184 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:46.184 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony DEL REG 8,1 3014 /var/lib/sss/mc/passwd 2022-04-03T22:52:46.184 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:46.185 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.185 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.185 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony mem REG 8,1 132888 8225 /usr/lib64/libseccomp.so.2.5.1 2022-04-03T22:52:46.185 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:46.186 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:46.186 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.186 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.187 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.187 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony 1w CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.187 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony 2u CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.188 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony 3u unix 0xffff888170003480 0t0 62588 type=DGRAM 2022-04-03T22:52:46.188 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony 5r REG 8,1 9253600 3014 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:46.188 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony 6u unix 0xffff888170001680 0t0 62591 type=STREAM 2022-04-03T22:52:46.189 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony 7u IPv4 62592 0t0 UDP localhost:323 2022-04-03T22:52:46.189 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony 8u IPv6 62593 0t0 UDP localhost:323 2022-04-03T22:52:46.189 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony 9u unix 0xffff888170000000 0t0 62594 /var/run/chrony/chronyd.sock type=DGRAM 2022-04-03T22:52:46.190 INFO:teuthology.orchestra.run.smithi077.stdout:chronyd 7959 chrony 10u sock 0,8 0t0 138492 protocol: UDP 2022-04-03T22:52:46.190 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.190 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.191 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root txt REG 8,1 11816 8955 /usr/libexec/platform-python3.6 2022-04-03T22:52:46.191 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.191 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.192 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.192 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.192 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.193 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.193 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:46.193 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root DEL REG 8,1 1109 /var/lib/sss/mc/passwd 2022-04-03T22:52:46.194 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:46.194 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:46.194 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.195 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.195 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.195 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:46.196 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.196 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.196 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:46.197 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.197 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.197 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:46.198 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:46.198 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:46.198 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:46.199 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:46.199 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:46.199 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:46.200 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:46.200 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:46.200 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:46.201 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:46.201 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:46.201 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.202 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root 1u unix 0xffff888106ed0f00 0t0 64651 type=STREAM 2022-04-03T22:52:46.202 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root 2u unix 0xffff888106ed0f00 0t0 64651 type=STREAM 2022-04-03T22:52:46.202 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root 3r REG 8,1 9253600 1109 /var/lib/sss/mc/passwd (deleted) 2022-04-03T22:52:46.202 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-cras 8270 root 4u unix 0xffff888106ed6180 0t0 64654 type=STREAM 2022-04-03T22:52:46.203 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-buf/d 14356 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.203 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-buf/d 14356 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.203 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-buf/d 14356 root txt unknown /proc/14356/exe 2022-04-03T22:52:46.204 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-conv/ 14357 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.204 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-conv/ 14357 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.204 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-conv/ 14357 root txt unknown /proc/14357/exe 2022-04-03T22:52:46.205 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-recla 14358 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.205 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-recla 14358 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.205 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-recla 14358 root txt unknown /proc/14358/exe 2022-04-03T22:52:46.206 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-block 14359 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.206 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-block 14359 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.206 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-block 14359 root txt unknown /proc/14359/exe 2022-04-03T22:52:46.207 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-inode 14360 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.207 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-inode 14360 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.207 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-inode 14360 root txt unknown /proc/14360/exe 2022-04-03T22:52:46.208 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-log/d 14361 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.208 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-log/d 14361 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.208 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-log/d 14361 root txt unknown /proc/14361/exe 2022-04-03T22:52:46.209 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-cil/d 14362 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.209 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-cil/d 14362 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.209 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-cil/d 14362 root txt unknown /proc/14362/exe 2022-04-03T22:52:46.210 INFO:teuthology.orchestra.run.smithi077.stdout:xfsaild/d 14363 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.210 INFO:teuthology.orchestra.run.smithi077.stdout:xfsaild/d 14363 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.210 INFO:teuthology.orchestra.run.smithi077.stdout:xfsaild/d 14363 root txt unknown /proc/14363/exe 2022-04-03T22:52:46.211 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-buf/d 14436 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.211 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-buf/d 14436 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.211 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-buf/d 14436 root txt unknown /proc/14436/exe 2022-04-03T22:52:46.212 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-conv/ 14437 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.212 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-conv/ 14437 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.212 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-conv/ 14437 root txt unknown /proc/14437/exe 2022-04-03T22:52:46.213 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-recla 14438 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.213 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-recla 14438 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.213 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-recla 14438 root txt unknown /proc/14438/exe 2022-04-03T22:52:46.214 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-block 14439 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.214 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-block 14439 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.214 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-block 14439 root txt unknown /proc/14439/exe 2022-04-03T22:52:46.215 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-inode 14440 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.215 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-inode 14440 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.215 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-inode 14440 root txt unknown /proc/14440/exe 2022-04-03T22:52:46.215 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-log/d 14441 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.216 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-log/d 14441 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.216 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-log/d 14441 root txt unknown /proc/14441/exe 2022-04-03T22:52:46.216 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-cil/d 14442 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.217 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-cil/d 14442 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.217 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-cil/d 14442 root txt unknown /proc/14442/exe 2022-04-03T22:52:46.217 INFO:teuthology.orchestra.run.smithi077.stdout:xfsaild/d 14443 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.218 INFO:teuthology.orchestra.run.smithi077.stdout:xfsaild/d 14443 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.218 INFO:teuthology.orchestra.run.smithi077.stdout:xfsaild/d 14443 root txt unknown /proc/14443/exe 2022-04-03T22:52:46.218 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-buf/d 14516 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.219 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-buf/d 14516 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.219 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-buf/d 14516 root txt unknown /proc/14516/exe 2022-04-03T22:52:46.219 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-conv/ 14517 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.220 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-conv/ 14517 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.220 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-conv/ 14517 root txt unknown /proc/14517/exe 2022-04-03T22:52:46.220 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-recla 14518 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.221 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-recla 14518 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.221 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-recla 14518 root txt unknown /proc/14518/exe 2022-04-03T22:52:46.221 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-block 14519 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.222 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-block 14519 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.222 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-block 14519 root txt unknown /proc/14519/exe 2022-04-03T22:52:46.222 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-inode 14520 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.222 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-inode 14520 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.223 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-inode 14520 root txt unknown /proc/14520/exe 2022-04-03T22:52:46.223 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-log/d 14521 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.223 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-log/d 14521 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.224 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-log/d 14521 root txt unknown /proc/14521/exe 2022-04-03T22:52:46.224 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-cil/d 14522 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.224 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-cil/d 14522 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.225 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-cil/d 14522 root txt unknown /proc/14522/exe 2022-04-03T22:52:46.225 INFO:teuthology.orchestra.run.smithi077.stdout:xfsaild/d 14523 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.225 INFO:teuthology.orchestra.run.smithi077.stdout:xfsaild/d 14523 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.226 INFO:teuthology.orchestra.run.smithi077.stdout:xfsaild/d 14523 root txt unknown /proc/14523/exe 2022-04-03T22:52:46.226 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-buf/d 14596 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.226 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-buf/d 14596 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.227 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-buf/d 14596 root txt unknown /proc/14596/exe 2022-04-03T22:52:46.227 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-conv/ 14597 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.227 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-conv/ 14597 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.228 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-conv/ 14597 root txt unknown /proc/14597/exe 2022-04-03T22:52:46.228 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-recla 14598 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.228 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-recla 14598 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.228 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-recla 14598 root txt unknown /proc/14598/exe 2022-04-03T22:52:46.229 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-block 14599 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.229 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-block 14599 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.229 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-block 14599 root txt unknown /proc/14599/exe 2022-04-03T22:52:46.230 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-inode 14600 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.230 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-inode 14600 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.230 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-inode 14600 root txt unknown /proc/14600/exe 2022-04-03T22:52:46.231 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-log/d 14601 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.231 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-log/d 14601 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.231 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-log/d 14601 root txt unknown /proc/14601/exe 2022-04-03T22:52:46.232 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-cil/d 14602 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.232 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-cil/d 14602 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.232 INFO:teuthology.orchestra.run.smithi077.stdout:xfs-cil/d 14602 root txt unknown /proc/14602/exe 2022-04-03T22:52:46.233 INFO:teuthology.orchestra.run.smithi077.stdout:xfsaild/d 14603 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.233 INFO:teuthology.orchestra.run.smithi077.stdout:xfsaild/d 14603 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.233 INFO:teuthology.orchestra.run.smithi077.stdout:xfsaild/d 14603 root txt unknown /proc/14603/exe 2022-04-03T22:52:46.234 INFO:teuthology.orchestra.run.smithi077.stdout:dio/dm-0 14649 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.234 INFO:teuthology.orchestra.run.smithi077.stdout:dio/dm-0 14649 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.234 INFO:teuthology.orchestra.run.smithi077.stdout:dio/dm-0 14649 root txt unknown /proc/14649/exe 2022-04-03T22:52:46.235 INFO:teuthology.orchestra.run.smithi077.stdout:dio/dm-1 15819 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.235 INFO:teuthology.orchestra.run.smithi077.stdout:dio/dm-1 15819 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.235 INFO:teuthology.orchestra.run.smithi077.stdout:dio/dm-1 15819 root txt unknown /proc/15819/exe 2022-04-03T22:52:46.236 INFO:teuthology.orchestra.run.smithi077.stdout:dio/dm-2 16989 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.236 INFO:teuthology.orchestra.run.smithi077.stdout:dio/dm-2 16989 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.236 INFO:teuthology.orchestra.run.smithi077.stdout:dio/dm-2 16989 root txt unknown /proc/16989/exe 2022-04-03T22:52:46.237 INFO:teuthology.orchestra.run.smithi077.stdout:dio/dm-3 18159 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:46.237 INFO:teuthology.orchestra.run.smithi077.stdout:dio/dm-3 18159 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.237 INFO:teuthology.orchestra.run.smithi077.stdout:dio/dm-3 18159 root txt unknown /proc/18159/exe 2022-04-03T22:52:46.237 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.238 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.238 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root txt REG 8,1 165640 12133 /usr/bin/sudo 2022-04-03T22:52:46.238 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 15928 23305 /usr/lib64/security/pam_succeed_if.so 2022-04-03T22:52:46.239 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 16536 9706 /usr/lib64/libpam_misc.so.0.82.1 2022-04-03T22:52:46.239 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 915232 23524 /usr/lib64/security/pam_systemd.so 2022-04-03T22:52:46.239 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 24048 23285 /usr/lib64/security/pam_limits.so 2022-04-03T22:52:46.240 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 11736 23283 /usr/lib64/security/pam_keyinit.so 2022-04-03T22:52:46.240 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 46232 9162 /usr/lib64/libcrack.so.2.9.0 2022-04-03T22:52:46.240 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 24744 9666 /usr/lib64/libpwquality.so.1.0.2 2022-04-03T22:52:46.241 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 11720 23217 /usr/lib64/security/pam_pwquality.so 2022-04-03T22:52:46.241 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 7640 23294 /usr/lib64/security/pam_permit.so 2022-04-03T22:52:46.241 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 7552 23271 /usr/lib64/security/pam_deny.so 2022-04-03T22:52:46.242 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 58032 24580 /usr/lib64/security/pam_sss.so 2022-04-03T22:52:46.242 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 371032 9256 /usr/lib64/libtirpc.so.3.0.0 2022-04-03T22:52:46.242 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 116408 9258 /usr/lib64/libnsl.so.2.0.0 2022-04-03T22:52:46.243 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 58016 23310 /usr/lib64/security/pam_unix.so 2022-04-03T22:52:46.243 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 11744 23287 /usr/lib64/security/pam_localuser.so 2022-04-03T22:52:46.243 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 11816 23316 /usr/lib64/security/pam_usertype.so 2022-04-03T22:52:46.243 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 11720 23275 /usr/lib64/security/pam_faildelay.so 2022-04-03T22:52:46.244 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 20160 23273 /usr/lib64/security/pam_env.so 2022-04-03T22:52:46.244 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:46.244 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.245 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.245 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.245 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.246 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:46.246 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.246 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:46.247 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:46.247 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:46.247 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:46.248 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:46.248 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:46.248 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:46.248 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:46.249 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:46.249 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:46.249 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.250 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:46.250 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 125328 8879 /usr/lib64/libsasl2.so.3.0.0 2022-04-03T22:52:46.250 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.251 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.251 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 67136 8940 /usr/lib64/liblber-2.4.so.2.10.9 2022-04-03T22:52:46.251 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 325184 8942 /usr/lib64/libldap-2.4.so.2.10.9 2022-04-03T22:52:46.252 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:46.252 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 432872 35707 /usr/libexec/sudo/sudoers.so 2022-04-03T22:52:46.252 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:46.253 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:46.253 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:46.253 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.254 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:46.254 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.254 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.255 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.255 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 104424 35704 /usr/libexec/sudo/libsudo_util.so.0.0.0 2022-04-03T22:52:46.255 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:46.256 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.256 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:46.256 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.257 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:46.257 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:46.257 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:46.258 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:46.258 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:46.258 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:46.259 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:46.259 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:46.259 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:46.260 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:46.260 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:46.260 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:46.261 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root 0r FIFO 0,12 0t0 78419 pipe 2022-04-03T22:52:46.261 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.261 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.262 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root 3r REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:46.262 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root 4u unix 0xffff88810da1f080 0t0 84005 type=STREAM 2022-04-03T22:52:46.262 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root 5u netlink 0t0 78437 AUDIT 2022-04-03T22:52:46.263 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root 6r REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:46.263 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root 7r REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:46.263 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root 9u unix 0xffff88817f131680 0t0 78440 type=DGRAM 2022-04-03T22:52:46.264 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root 10r FIFO 0,12 0t0 84025 pipe 2022-04-03T22:52:46.264 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19922 root 11w FIFO 0,12 0t0 84025 pipe 2022-04-03T22:52:46.264 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.265 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.265 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root txt REG 8,1 165640 12133 /usr/bin/sudo 2022-04-03T22:52:46.265 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 15928 23305 /usr/lib64/security/pam_succeed_if.so 2022-04-03T22:52:46.266 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 16536 9706 /usr/lib64/libpam_misc.so.0.82.1 2022-04-03T22:52:46.266 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 915232 23524 /usr/lib64/security/pam_systemd.so 2022-04-03T22:52:46.266 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 24048 23285 /usr/lib64/security/pam_limits.so 2022-04-03T22:52:46.267 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 11736 23283 /usr/lib64/security/pam_keyinit.so 2022-04-03T22:52:46.267 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 46232 9162 /usr/lib64/libcrack.so.2.9.0 2022-04-03T22:52:46.267 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 24744 9666 /usr/lib64/libpwquality.so.1.0.2 2022-04-03T22:52:46.268 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 11720 23217 /usr/lib64/security/pam_pwquality.so 2022-04-03T22:52:46.268 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 7640 23294 /usr/lib64/security/pam_permit.so 2022-04-03T22:52:46.268 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 7552 23271 /usr/lib64/security/pam_deny.so 2022-04-03T22:52:46.269 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 58032 24580 /usr/lib64/security/pam_sss.so 2022-04-03T22:52:46.269 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 371032 9256 /usr/lib64/libtirpc.so.3.0.0 2022-04-03T22:52:46.269 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 116408 9258 /usr/lib64/libnsl.so.2.0.0 2022-04-03T22:52:46.270 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 58016 23310 /usr/lib64/security/pam_unix.so 2022-04-03T22:52:46.270 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 11744 23287 /usr/lib64/security/pam_localuser.so 2022-04-03T22:52:46.271 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 11816 23316 /usr/lib64/security/pam_usertype.so 2022-04-03T22:52:46.271 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 11720 23275 /usr/lib64/security/pam_faildelay.so 2022-04-03T22:52:46.271 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 20160 23273 /usr/lib64/security/pam_env.so 2022-04-03T22:52:46.272 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:46.272 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.272 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.273 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.273 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.273 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:46.274 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.274 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:46.274 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:46.274 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:46.275 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:46.275 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:46.275 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:46.276 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:46.276 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:46.276 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:46.277 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:46.277 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.277 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:46.278 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 125328 8879 /usr/lib64/libsasl2.so.3.0.0 2022-04-03T22:52:46.278 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.278 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.279 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 67136 8940 /usr/lib64/liblber-2.4.so.2.10.9 2022-04-03T22:52:46.279 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 325184 8942 /usr/lib64/libldap-2.4.so.2.10.9 2022-04-03T22:52:46.279 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:46.280 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 432872 35707 /usr/libexec/sudo/sudoers.so 2022-04-03T22:52:46.280 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:46.280 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:46.281 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:46.281 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.281 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:46.282 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.282 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.282 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.283 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 104424 35704 /usr/libexec/sudo/libsudo_util.so.0.0.0 2022-04-03T22:52:46.283 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:46.283 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.284 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:46.284 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.284 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:46.285 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:46.285 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:46.285 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:46.286 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:46.286 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:46.286 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:46.287 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:46.287 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:46.287 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:46.287 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:46.288 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:46.288 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root 0r FIFO 0,12 0t0 78423 pipe 2022-04-03T22:52:46.288 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:46.289 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:46.289 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root 3r REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:46.289 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root 4u unix 0xffff888105c27800 0t0 83545 type=STREAM 2022-04-03T22:52:46.290 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root 5u netlink 0t0 80191 AUDIT 2022-04-03T22:52:46.290 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root 6r REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:46.291 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root 7r REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:46.291 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root 9u unix 0xffff888105c26900 0t0 83551 type=DGRAM 2022-04-03T22:52:46.291 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root 10r FIFO 0,12 0t0 83554 pipe 2022-04-03T22:52:46.291 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 19929 root 11w FIFO 0,12 0t0 83554 pipe 2022-04-03T22:52:46.292 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.292 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.293 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root txt REG 8,1 11816 8955 /usr/libexec/platform-python3.6 2022-04-03T22:52:46.293 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.293 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.293 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.294 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.294 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.294 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.295 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:46.295 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.295 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.296 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.296 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:46.296 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.297 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.297 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:46.297 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.298 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.298 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:46.298 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:46.299 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:46.299 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:46.299 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:46.300 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:46.300 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:46.300 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:46.301 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:46.301 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:46.301 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:46.302 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:46.302 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root 0r FIFO 0,12 0t0 78419 pipe 2022-04-03T22:52:46.302 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.303 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19968 root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.303 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.303 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.304 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root txt REG 8,1 11816 8955 /usr/libexec/platform-python3.6 2022-04-03T22:52:46.304 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.304 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.305 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.305 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.305 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.306 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.306 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:46.306 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.307 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.307 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.307 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:46.308 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.308 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.308 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:46.309 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.309 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.309 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:46.310 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:46.310 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:46.310 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:46.311 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:46.311 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:46.311 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:46.312 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:46.312 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:46.312 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:46.313 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:46.313 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:46.313 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root 0r FIFO 0,12 0t0 78423 pipe 2022-04-03T22:52:46.314 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:46.314 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 19972 root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:46.314 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.315 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.315 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.315 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.316 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.316 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.316 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.317 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.317 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.317 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.318 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.318 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.318 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.319 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.319 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.319 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.320 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.320 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.320 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.321 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.321 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.321 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.321 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.322 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.322 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.323 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.323 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.323 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.323 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.324 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.324 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.324 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.325 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.325 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.325 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.326 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.326 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.326 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.327 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.327 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.327 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.328 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.328 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.328 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.328 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.329 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.329 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.329 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.330 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.330 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.330 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.331 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.331 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.331 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.332 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.332 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.332 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.332 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.333 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.333 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.333 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.334 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.334 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.334 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.335 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.335 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.335 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.336 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.336 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.336 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.337 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.337 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.337 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.337 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.338 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.338 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.338 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.339 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.339 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.339 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.340 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.340 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.340 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.341 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.341 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.341 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.342 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.342 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.342 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.343 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.343 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.343 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.344 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.344 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.344 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.345 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.345 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.345 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.346 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.346 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.346 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.347 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.347 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.347 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.347 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.348 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.348 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.348 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.349 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.349 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.349 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.350 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.350 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.350 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.351 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.351 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.351 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.352 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.352 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.352 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.353 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.353 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.353 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.354 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.354 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.354 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.355 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.355 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.355 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.356 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.356 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.356 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.357 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.357 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.357 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.358 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.358 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.359 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.359 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.359 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.360 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.360 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.360 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.361 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.361 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.361 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.362 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.362 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.362 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.363 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.363 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.363 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.364 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.364 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19977 log root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.364 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.365 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.365 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.365 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.366 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.366 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.366 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.367 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.367 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.367 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.368 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.368 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.368 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.369 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.369 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.369 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.370 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.370 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.370 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.371 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.371 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.371 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.372 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.372 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.372 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.372 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.373 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.373 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.373 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.374 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.374 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.374 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.375 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.375 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.375 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.376 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.376 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.376 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.377 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.377 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.377 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.378 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.378 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.378 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.379 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.379 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.379 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.380 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.380 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.380 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.381 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.381 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.381 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.381 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.382 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.382 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.382 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.383 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.383 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.383 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.384 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.384 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.384 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.385 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.385 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.385 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.386 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.386 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.386 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.387 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.387 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.387 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.388 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.388 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.388 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.389 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.389 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19978 service root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.389 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.390 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.390 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.390 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.391 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.391 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.391 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.392 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.392 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.392 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.393 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.393 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.393 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.393 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.394 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.394 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.395 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.395 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.395 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.396 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.396 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.396 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.396 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.397 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.397 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.397 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.398 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.398 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.398 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.399 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.399 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.399 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.400 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.400 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.400 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.401 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.401 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.401 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.402 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.402 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.402 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.403 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.403 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.403 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.404 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.404 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.404 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.405 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.405 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.405 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.406 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.406 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.406 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.407 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.407 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.407 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.408 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.408 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.408 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.409 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.409 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.409 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.410 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.410 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.411 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.411 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.411 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.412 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.412 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.412 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.413 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.413 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.413 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.414 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.414 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.414 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.415 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19979 admin_soc root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.415 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.415 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.416 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.416 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.416 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.417 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.417 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.417 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.418 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.418 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.418 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.419 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.419 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.419 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.420 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.420 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.420 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.421 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.421 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.421 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.422 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.422 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.422 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.423 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.423 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.423 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.424 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.424 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.424 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.424 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.425 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.425 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.425 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.426 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.426 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.426 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.427 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.427 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.427 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.428 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.428 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.428 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.429 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.429 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.429 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.430 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.430 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.430 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.431 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.431 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.431 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.432 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.432 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.432 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.433 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.433 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.433 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.434 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.434 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.434 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.435 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.435 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.435 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.436 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.436 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.436 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.437 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.437 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.437 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.438 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.438 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.438 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.439 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.439 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.439 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.440 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.440 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19980 signal_ha root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.440 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.441 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.441 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.441 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.442 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.442 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.442 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.442 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.443 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.443 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.444 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.444 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.444 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.445 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.445 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.447 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.447 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.448 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.448 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.448 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.449 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.449 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.449 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.450 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.450 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.450 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.451 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.451 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.451 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.452 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.452 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.453 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.453 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.453 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.453 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.454 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.454 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.455 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.455 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.455 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.456 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.456 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.456 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.456 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.457 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.457 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.457 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.458 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.458 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.458 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.459 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.459 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.459 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.460 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.460 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.461 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.461 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.461 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.462 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.462 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.462 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.463 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.463 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.463 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.464 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.464 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.464 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.465 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.465 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.465 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.466 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.466 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.466 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.466 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.467 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.467 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.467 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19981 rocksdb:l root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.468 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.468 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.468 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.469 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.469 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.469 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.470 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.470 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.470 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.471 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.471 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.471 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.472 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.472 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.472 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.473 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.473 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.473 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.474 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.474 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.474 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.475 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.475 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.475 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.476 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.476 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.476 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.477 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.477 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.477 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.478 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.478 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.478 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.479 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.479 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.479 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.480 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.480 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.480 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.481 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.481 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.481 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.482 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.482 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.482 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.483 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.483 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.483 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.484 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.484 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.484 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.485 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.485 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.485 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.486 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.486 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.486 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.487 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.487 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.487 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.488 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.488 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.488 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.489 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.489 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.489 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.490 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.490 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.490 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.491 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.491 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.491 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.492 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.492 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.492 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.493 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.493 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19982 rocksdb:h root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.493 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.494 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.494 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.494 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.495 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.495 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.495 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.496 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.496 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.496 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.497 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.497 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.497 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.498 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.498 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.498 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.499 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.499 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.499 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.500 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.500 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.500 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.501 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.501 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.501 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.502 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.502 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.502 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.503 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.503 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.503 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.504 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.504 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.504 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.505 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.505 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.505 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.506 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.506 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.506 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.507 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.507 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.507 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.507 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.508 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.508 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.508 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.509 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.509 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.509 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.510 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.510 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.510 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.511 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.511 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.511 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.512 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.512 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.513 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.513 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.513 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.514 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.514 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.514 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.515 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.515 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.515 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.516 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.516 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.516 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.517 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.517 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.517 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.518 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.518 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.518 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.519 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19998 ceph-mon root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.519 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.519 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.520 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.520 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.520 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.521 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.521 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.521 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.522 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.522 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.522 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.523 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.523 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.523 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.524 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.524 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.524 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.525 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.525 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.525 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.526 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.526 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.526 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.526 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.527 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.527 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.527 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.528 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.528 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.528 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.529 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.529 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.529 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.530 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.530 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.530 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.531 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.531 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.531 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.532 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.532 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.532 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.532 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.533 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.533 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.533 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.534 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.534 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.534 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.535 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.535 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.535 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.536 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.536 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.536 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.537 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.537 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.537 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.538 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.538 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.538 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.538 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.539 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.539 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.539 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.540 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.540 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.540 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.541 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.541 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.541 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.542 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.542 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.542 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.543 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.543 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.543 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 19999 fn_monsto root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.544 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.544 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.544 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.545 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.545 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.545 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.546 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.546 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.546 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.546 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.547 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.547 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.547 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.548 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.548 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.548 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.549 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.549 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.549 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.550 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.550 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.550 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.551 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.551 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.551 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.552 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.552 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.552 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.553 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.553 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.553 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.554 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.554 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.554 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.555 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.555 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.555 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.556 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.556 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.556 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.557 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.557 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.557 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.558 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.558 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.558 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.559 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.559 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.559 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.560 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.560 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.560 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.561 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.561 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.561 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.562 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.562 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.562 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.563 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.563 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.563 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.564 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.564 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.564 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.565 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.565 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.565 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.565 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.566 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.566 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.566 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.567 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.567 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.567 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.568 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.568 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.568 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20000 msgr-work root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.569 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.569 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.569 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.570 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.570 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.570 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.571 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.571 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.571 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.572 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.572 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.572 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.573 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.573 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.573 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.574 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.574 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.574 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.575 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.575 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.575 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.576 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.576 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.576 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.577 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.577 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.577 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.577 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.578 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.578 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.579 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.579 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.579 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.580 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.580 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.580 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.581 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.581 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.581 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.582 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.582 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.582 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.583 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.583 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.583 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.584 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.584 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.584 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.585 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.585 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.585 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.586 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.586 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.586 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.587 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.587 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.587 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.588 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.588 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.588 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.589 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.589 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.589 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.590 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.590 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.590 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.591 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.591 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.591 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.592 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.592 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.592 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.593 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.593 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.593 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.594 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.594 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20001 msgr-work root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.594 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.595 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.595 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.595 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.596 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.596 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.596 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.597 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.597 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.597 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.597 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.598 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.598 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.598 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.599 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.599 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.599 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.600 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.600 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.600 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.601 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.601 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.601 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.601 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.602 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.602 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.603 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.603 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.603 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.603 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.604 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.604 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.604 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.605 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.605 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.605 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.606 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.606 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.606 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.607 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.607 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.607 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.608 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.608 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.608 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.609 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.609 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.609 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.609 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.610 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.610 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.611 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.611 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.611 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.612 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.612 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.612 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.612 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.613 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.613 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.613 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.614 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.614 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.614 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.615 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.615 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.615 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.616 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.616 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.617 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.617 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.617 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.618 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.618 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.618 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.619 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.619 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20002 msgr-work root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.619 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.620 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.620 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.620 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.621 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.621 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.621 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.622 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.622 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.622 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.623 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.623 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.623 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.624 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.624 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.624 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.625 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.625 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.625 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.625 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.626 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.626 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.626 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.627 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.627 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.627 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.628 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.628 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.628 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.629 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.629 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.629 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.630 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.630 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.630 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.631 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.631 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.631 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.632 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.632 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.632 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.633 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.633 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.633 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.633 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.634 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.634 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.634 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.635 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.635 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.635 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.636 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.636 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.636 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.637 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.637 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.637 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.638 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.638 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.638 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.639 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.639 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.639 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.640 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.640 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.640 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.641 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.641 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.641 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.641 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.642 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.642 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.642 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.643 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.643 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.643 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.644 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20003 OpHistory root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.644 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.644 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.645 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.645 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.645 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.646 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.646 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.647 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.647 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.647 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.647 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.648 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.648 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.648 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.649 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.649 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.649 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.650 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.650 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.650 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.651 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.651 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.652 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.652 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.652 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.652 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.653 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.653 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.653 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.654 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.654 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.655 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.655 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.655 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.655 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.656 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.656 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.657 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.657 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.657 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.658 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.658 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.658 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.659 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.659 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.659 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.660 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.660 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.660 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.660 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.661 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.661 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.661 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.662 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.662 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.662 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.663 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.663 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.663 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.664 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.664 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.664 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.665 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.665 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.665 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.666 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.666 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.666 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.667 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.667 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.667 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.667 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.668 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.668 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.668 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.669 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.669 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20006 fin root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.669 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.670 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.670 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.670 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.671 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.671 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.671 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.672 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.672 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.672 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.673 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.673 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.673 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.674 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.674 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.674 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.675 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.675 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.675 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.676 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.676 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.676 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.677 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.677 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.677 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.678 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.678 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.678 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.679 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.679 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.679 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.680 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.680 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.680 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.680 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.681 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.681 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.681 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.682 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.682 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.682 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.683 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.683 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.683 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.684 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.684 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.684 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.685 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.685 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.685 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.686 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.686 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.686 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.687 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.687 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.687 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.688 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.688 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.688 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.689 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.689 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.689 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.690 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.690 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.690 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.691 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.691 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.691 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.692 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.692 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.692 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.693 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.693 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.693 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.694 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.694 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.694 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20007 safe_time root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.695 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.695 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.695 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.696 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.696 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.696 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.697 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.697 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.697 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.698 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.698 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.698 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.699 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.699 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.699 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.700 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.700 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.700 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.701 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.701 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.701 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.701 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.702 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.702 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.702 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.703 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.703 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.703 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.704 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.704 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.704 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.705 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.705 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.705 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.706 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.706 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.706 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.707 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.707 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.707 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.708 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.708 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.708 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.708 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.709 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.709 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.709 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.710 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.710 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.710 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.711 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.711 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.712 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.712 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.712 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.713 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.713 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.713 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.714 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.714 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.714 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.715 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.715 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.715 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.716 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.716 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.716 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.717 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.717 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.717 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.718 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.718 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.718 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.719 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.719 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.719 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.720 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20009 cpu_tp root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.720 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.720 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.721 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.721 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.721 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.721 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.722 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.722 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.722 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.723 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.723 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.723 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.724 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.724 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.724 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.725 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.725 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.725 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.726 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.726 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.726 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.727 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.727 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.727 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.727 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.728 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.728 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.728 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.729 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.729 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.729 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.730 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.730 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.730 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.731 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.731 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.731 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.732 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.732 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.732 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.733 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.733 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.733 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.733 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.734 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.734 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.734 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.735 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.735 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.735 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.736 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.736 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.737 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.737 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.737 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.737 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.738 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.738 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.739 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.739 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.739 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.739 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.740 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.740 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.740 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.741 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.741 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.741 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.742 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.742 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.742 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.743 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.743 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.743 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.744 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.744 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.744 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20011 cpu_tp root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.745 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.745 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.745 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.746 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.746 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.746 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.747 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.747 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.747 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.748 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.748 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.749 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.749 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.749 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.750 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.750 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.750 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.751 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.751 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.752 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.752 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.752 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.753 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.753 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.753 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.754 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.754 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.754 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.755 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.755 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.756 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.756 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.756 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.757 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.757 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.757 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.757 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.758 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.758 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.759 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.759 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.759 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.760 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.760 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.760 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.761 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.761 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.761 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.762 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.762 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.762 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.763 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.763 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.763 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.764 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.764 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.764 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.765 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.765 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.765 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.766 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.766 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.766 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.767 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.767 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.767 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.768 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.768 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.768 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.769 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.769 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.769 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.770 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.770 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.770 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.771 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.771 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20013 cpu_tp root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.771 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.772 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.772 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.772 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.773 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.773 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.773 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.774 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.774 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.774 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.775 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.775 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.775 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.776 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.776 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.776 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.777 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.777 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.777 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.778 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.778 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.778 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.779 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.779 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.779 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.779 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.780 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.780 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.780 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.781 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.781 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.781 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.782 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.782 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.782 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.783 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.783 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.783 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.784 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.784 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.784 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.785 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.785 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.785 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.786 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.786 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.786 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.787 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.787 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.787 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.788 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.788 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.788 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.789 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.789 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.789 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.789 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.790 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.790 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.790 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.791 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.791 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.792 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.792 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.792 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.792 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.793 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.793 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.793 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.794 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.794 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.794 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.795 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.795 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.795 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.796 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.796 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20014 cpu_tp root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.796 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.797 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.797 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.797 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.798 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.798 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.798 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.799 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.799 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.799 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.800 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.800 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.800 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.801 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.801 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.801 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.801 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.802 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.802 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.802 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.803 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.803 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.803 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.804 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.804 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.804 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.805 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.805 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.805 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.806 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.806 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.806 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.807 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.807 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.807 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.808 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.808 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.808 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.809 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.809 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.809 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.810 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.810 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.810 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.811 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.811 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.811 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.812 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.812 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.812 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.813 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.813 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.813 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.814 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.814 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.814 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.815 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.815 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.815 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.816 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.816 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.816 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.817 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.817 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.817 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.818 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.818 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.818 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.819 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.819 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.819 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.820 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.820 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.820 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.821 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.821 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.821 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20016 ms_dispat root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.822 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.822 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.822 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.823 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.823 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.823 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.824 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.824 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.824 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.825 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.825 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.825 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.826 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.826 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.826 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.827 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.827 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.827 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.828 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.828 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.828 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.829 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.829 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.829 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.830 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.830 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.830 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.831 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.831 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.831 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.831 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.832 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.832 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.832 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.833 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.833 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.833 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.834 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.834 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.834 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.835 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.835 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.835 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.836 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.836 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.836 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.837 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.837 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.837 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.837 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.838 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.838 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.838 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.839 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.839 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.839 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.840 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.840 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.840 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.841 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.841 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.841 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.842 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.842 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.842 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.843 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.843 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.843 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.844 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.844 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.844 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.845 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.845 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.845 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.846 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.846 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.846 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20018 ms_local root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.847 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.847 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.847 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.848 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.848 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.848 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.849 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.849 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.849 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.850 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.850 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.850 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.851 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.851 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.851 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.852 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.852 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.852 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.853 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.853 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.853 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.854 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.854 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.854 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.855 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.855 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.855 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.856 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.856 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.856 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.857 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.857 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.857 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.858 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.858 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.858 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.859 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.859 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.859 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.859 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.860 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.860 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.860 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.861 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.861 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.861 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.862 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.862 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.862 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.863 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.863 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.863 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.864 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.864 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.864 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.865 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.865 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.865 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.866 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.866 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.866 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.867 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.867 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.867 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.868 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.868 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.868 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.869 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.869 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.869 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.870 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.870 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.870 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.871 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.871 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.871 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.872 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20019 safe_time root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.872 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.872 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.873 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.873 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.873 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.874 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.874 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.874 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.875 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.875 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.875 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.875 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.876 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.876 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.876 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.877 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.877 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.877 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.878 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.878 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.878 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.879 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.879 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.879 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.880 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.880 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.880 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.880 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.881 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.881 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.881 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.882 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.882 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.882 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.883 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.883 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.883 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.884 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.884 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.884 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.885 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.885 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.885 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.886 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.886 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.886 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.887 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.887 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.887 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.888 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.888 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.888 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.889 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.889 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.889 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.890 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.890 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.890 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.891 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.891 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.891 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.892 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.892 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.892 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.893 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.893 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.893 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.893 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.894 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.894 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.894 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.895 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.895 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.895 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.896 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.896 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.896 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20021 ms_dispat root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.897 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.897 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.897 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.898 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.898 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.898 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.899 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.899 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.899 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.900 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.900 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.900 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.901 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.901 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.901 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.902 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.902 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.902 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.903 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.903 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.903 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.904 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.904 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.904 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.905 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.905 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.905 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.906 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.906 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.906 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.907 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.907 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.907 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.908 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.908 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.908 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.909 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.909 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.909 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.910 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.910 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.910 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.911 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.911 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.911 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.912 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.912 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.912 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.913 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.913 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.913 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.914 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.914 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.914 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.915 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.915 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.915 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.916 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.916 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.916 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.917 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.917 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.917 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.918 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.918 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.918 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.919 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.919 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.919 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.920 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.920 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.920 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.921 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.921 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.921 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.922 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.922 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 20022 ms_local root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.922 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.923 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.923 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root txt REG 8,1 11378752 4089 /usr/bin/ceph-mon 2022-04-03T22:52:46.923 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 2083840 788555 /usr/lib64/ceph/erasure-code/libec_lrc.so 2022-04-03T22:52:46.924 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 414240 788551 /usr/lib64/ceph/erasure-code/libec_jerasure.so 2022-04-03T22:52:46.924 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.924 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.925 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.925 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.925 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.926 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.926 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.926 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.927 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.927 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.927 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.928 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.928 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.928 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.929 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.929 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.929 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.930 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.930 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.930 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.931 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.931 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 269368 8693 /usr/lib64/libfuse.so.2.9.7 2022-04-03T22:52:46.931 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 34208 7884 /usr/lib64/libaio.so.1.0.1 2022-04-03T22:52:46.932 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.932 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.932 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:46.932 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 46256 8459 /usr/lib64/libsnappy.so.1.1.8 2022-04-03T22:52:46.933 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.933 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.933 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.934 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.934 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.934 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.935 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 1w FIFO 0,12 0t0 78420 pipe 2022-04-03T22:52:46.935 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 2w FIFO 0,12 0t0 78421 pipe 2022-04-03T22:52:46.935 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 3r FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.936 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 4w FIFO 0,12 0t0 82597 pipe 2022-04-03T22:52:46.936 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 5w REG 8,1 56573732 790200 /var/log/ceph/ceph-mon.a.log 2022-04-03T22:52:46.936 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 6uW REG 0,24 6 1645 /run/ceph/ceph-mon.a.pid 2022-04-03T22:52:46.937 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 7r FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.937 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 8w FIFO 0,12 0t0 82598 pipe 2022-04-03T22:52:46.937 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 9u unix 0xffff88817054bc00 0t0 82599 /var/run/ceph/ceph-mon.a.asok type=STREAM 2022-04-03T22:52:46.938 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 10r FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.938 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 11w FIFO 0,12 0t0 82600 pipe 2022-04-03T22:52:46.939 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 12r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.939 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 13uW REG 253,4 0 25165959 /var/lib/ceph/mon/ceph-a/store.db/LOCK 2022-04-03T22:52:46.939 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 14w REG 253,4 513029 25165963 /var/lib/ceph/mon/ceph-a/store.db/000022.log 2022-04-03T22:52:46.940 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 15r DIR 253,4 152 25165958 /var/lib/ceph/mon/ceph-a/store.db 2022-04-03T22:52:46.940 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 16u IPv4 183672 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43602 (ESTABLISHED) 2022-04-03T22:52:46.940 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 17w REG 253,4 885 25165964 /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-04-03T22:52:46.941 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 18u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.941 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 19r FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.941 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 20w FIFO 0,12 0t0 83559 pipe 2022-04-03T22:52:46.942 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 21u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.942 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 22r FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.942 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 23w FIFO 0,12 0t0 83560 pipe 2022-04-03T22:52:46.943 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 24u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.943 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 25r FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.943 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 26w FIFO 0,12 0t0 83561 pipe 2022-04-03T22:52:46.944 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 27u IPv4 80211 0t0 TCP smithi077.front.sepia.ceph.com:3300 (LISTEN) 2022-04-03T22:52:46.944 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 28u IPv4 80212 0t0 TCP smithi077.front.sepia.ceph.com:smc-https (LISTEN) 2022-04-03T22:52:46.944 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 29u IPv4 191908 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43622 (ESTABLISHED) 2022-04-03T22:52:46.945 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 30r FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.945 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 31w FIFO 0,12 0t0 79410 pipe 2022-04-03T22:52:46.945 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 32u IPv4 191403 0t0 TCP smithi077.front.sepia.ceph.com:3300->smithi077.front.sepia.ceph.com:43642 (ESTABLISHED) 2022-04-03T22:52:46.946 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 33r FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.946 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 34w FIFO 0,12 0t0 79411 pipe 2022-04-03T22:52:46.946 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 35r FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.947 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 36w FIFO 0,12 0t0 79412 pipe 2022-04-03T22:52:46.947 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 37w REG 8,1 93948 790204 /var/log/ceph/ceph.audit.log 2022-04-03T22:52:46.947 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 38w REG 8,1 113099 790202 /var/log/ceph/ceph.log 2022-04-03T22:52:46.948 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mon 19975 35286 rstore_co root 43r REG 253,4 3849201 25165969 /var/lib/ceph/mon/ceph-a/store.db/000024.sst 2022-04-03T22:52:46.948 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.948 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.949 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:46.949 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.949 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:46.950 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.950 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:46.950 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:46.951 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:46.951 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:46.951 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.951 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.952 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.952 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.952 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.953 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.953 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.953 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:46.954 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:46.954 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.954 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:46.955 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:46.955 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:46.955 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:46.956 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.956 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.956 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:46.957 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.957 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:46.957 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.958 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:46.958 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.958 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.959 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.959 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.959 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.960 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.960 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.960 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.961 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.961 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.961 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.962 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:46.962 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.962 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.963 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.963 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.963 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.964 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:46.964 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.964 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.965 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:46.965 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.965 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:46.966 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.966 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.966 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.967 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:46.967 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.967 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.968 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.968 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.968 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.969 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.969 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.969 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.970 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.970 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.970 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.971 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:46.971 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:46.971 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:46.972 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:46.972 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:46.972 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:46.973 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:46.973 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:46.973 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:46.974 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:46.974 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:46.974 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:46.975 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:46.975 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:46.975 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:46.976 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:46.976 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:46.976 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:46.977 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:46.977 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:46.977 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:46.978 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:46.978 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:46.978 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:46.979 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:46.979 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:46.979 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:46.980 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:46.980 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:46.980 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:46.981 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:46.981 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:46.981 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:46.982 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:46.982 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:46.982 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:46.983 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:46.983 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:46.983 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:46.984 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:46.984 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.984 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:46.984 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:46.985 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.985 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:46.986 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:46.986 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:46.986 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:46.987 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:46.987 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:46.987 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:46.988 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:46.988 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:46.988 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:46.989 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:46.989 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:46.989 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:46.990 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:46.990 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:46.990 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:46.991 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:46.991 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:46.991 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:46.992 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.992 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:46.992 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.993 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:46.993 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:46.993 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:46.994 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:46.994 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.994 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.995 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.995 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.995 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.996 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.996 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.996 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:46.997 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:46.997 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.997 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:46.998 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:46.998 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:46.998 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:46.999 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.999 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:46.999 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:47.000 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.000 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:47.000 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.001 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:47.001 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.001 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.002 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.002 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.002 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.003 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.003 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.003 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.004 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.004 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.004 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.005 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:47.005 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.005 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.006 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.006 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.006 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.007 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.007 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.007 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.008 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:47.008 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.008 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:47.009 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.009 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.009 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.010 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:47.010 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.010 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.011 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.011 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.011 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.012 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.012 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.012 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.013 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.013 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.013 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.014 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:47.014 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:47.014 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:47.015 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:47.015 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.015 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.016 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:47.016 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:47.016 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.017 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.017 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:47.017 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.018 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.018 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:47.018 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.019 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.019 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.019 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:47.020 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.020 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:47.020 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:47.021 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:47.021 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.021 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.022 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.022 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.022 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:47.023 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:47.023 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.023 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.024 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:47.024 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:47.024 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.025 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.025 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.025 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:47.026 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:47.026 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:47.026 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.027 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.027 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.027 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.028 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.028 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.028 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.029 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.029 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.029 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.030 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.030 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:47.030 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:47.031 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.031 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.031 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:47.032 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.032 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.032 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.033 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.033 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:47.033 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20004 log root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:47.034 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.034 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.034 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:47.035 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.035 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:47.035 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.036 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:47.036 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:47.036 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:47.037 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:47.037 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.037 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.038 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.038 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.038 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.039 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.039 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.039 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:47.040 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:47.040 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.040 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:47.041 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:47.041 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:47.041 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:47.042 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.042 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.042 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:47.043 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.043 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:47.043 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.044 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:47.044 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.044 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.045 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.045 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.046 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.046 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.046 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.047 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.047 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.047 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.048 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.048 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:47.048 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.049 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.049 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.049 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.050 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.050 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.050 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.051 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.051 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:47.051 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.052 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:47.052 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.052 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.053 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.053 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:47.053 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.054 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.054 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.054 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.055 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.055 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.055 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.056 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.056 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.056 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.057 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.057 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:47.057 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:47.058 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:47.058 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:47.058 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.059 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.059 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:47.059 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:47.060 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.060 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.060 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:47.061 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.061 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.061 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:47.062 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.062 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.062 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.063 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:47.063 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.063 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:47.064 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:47.064 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:47.064 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.065 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.065 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.065 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.066 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:47.066 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:47.066 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.067 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.067 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:47.067 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:47.068 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.068 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.068 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.069 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:47.069 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:47.069 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:47.070 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.070 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.070 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.071 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.071 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.071 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.072 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.072 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.072 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.073 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.073 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.073 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:47.074 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:47.074 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.074 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.075 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:47.075 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.075 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.076 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.076 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.076 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:47.077 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20008 msgr-work root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:47.077 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.077 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.078 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:47.078 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.078 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:47.079 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.079 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:47.079 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:47.080 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:47.080 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:47.080 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.081 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.081 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.081 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.082 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.082 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.082 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.083 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:47.083 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:47.083 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.084 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:47.084 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:47.084 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:47.085 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:47.085 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.085 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.086 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:47.086 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.086 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:47.087 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.087 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:47.087 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.088 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.088 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.088 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.089 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.089 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.089 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.090 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.090 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.090 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.091 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.091 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:47.091 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.092 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.092 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.092 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.093 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.093 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.093 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.093 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.094 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:47.094 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.094 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:47.095 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.095 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.095 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.096 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:47.096 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.096 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.097 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.097 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.097 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.098 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.098 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.098 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.099 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.099 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.099 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.100 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:47.100 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:47.100 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:47.101 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:47.101 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.101 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.102 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:47.102 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:47.102 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.103 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.103 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:47.103 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.103 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.104 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:47.104 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.104 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.105 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.105 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:47.105 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.106 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:47.106 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:47.106 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:47.107 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.107 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.107 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.108 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.108 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:47.108 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:47.109 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.109 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.109 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:47.110 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:47.110 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.110 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.111 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.111 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:47.111 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:47.112 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:47.112 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.112 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.113 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.113 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.113 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.113 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.114 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.114 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.114 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.115 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.115 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.115 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:47.116 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:47.116 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.116 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.117 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:47.117 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.117 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.118 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.118 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.118 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:47.119 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20010 msgr-work root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:47.119 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.120 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.120 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:47.120 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.121 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:47.121 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.121 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:47.122 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:47.122 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:47.122 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:47.123 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.123 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.123 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.123 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.124 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.124 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.124 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.125 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:47.125 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:47.125 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.126 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:47.126 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:47.126 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:47.127 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:47.127 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.128 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.128 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:47.128 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.129 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:47.129 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.129 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:47.129 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.130 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.130 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.130 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.131 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.131 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.131 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.132 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.132 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.132 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.133 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.133 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:47.133 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.134 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.134 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.134 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.135 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.135 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.135 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.136 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.136 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:47.136 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.137 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:47.137 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.137 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.138 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.138 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:47.138 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.139 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.139 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.139 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.140 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.140 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.140 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.140 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.141 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.141 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.141 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.142 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:47.142 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:47.142 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:47.143 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:47.143 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.143 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.144 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:47.144 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:47.144 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.145 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.145 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:47.145 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.146 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.146 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:47.146 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.147 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.147 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.147 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:47.148 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.148 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:47.148 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:47.148 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:47.149 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.149 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.149 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.150 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.150 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:47.150 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:47.151 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.151 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.151 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:47.152 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:47.152 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.152 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.153 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.153 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:47.153 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:47.154 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:47.154 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.154 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.155 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.155 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.155 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.156 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.156 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.156 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.157 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.157 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.157 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.158 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:47.158 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:47.158 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.159 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.159 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:47.159 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.160 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.160 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.160 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.161 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:47.161 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20012 msgr-work root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:47.161 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.162 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.162 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:47.162 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.163 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:47.163 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.163 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:47.164 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:47.164 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:47.164 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:47.165 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.165 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.165 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.166 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.166 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.166 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.166 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.167 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:47.167 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:47.167 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.168 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:47.168 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:47.168 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:47.169 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:47.169 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.169 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.170 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:47.170 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.170 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:47.171 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.171 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:47.171 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.172 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.172 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.172 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.173 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.173 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.173 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.174 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.174 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.174 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.175 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.175 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:47.175 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.176 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.176 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.176 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.177 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.177 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.177 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.177 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.178 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:47.178 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.178 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:47.179 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.179 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.179 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.180 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:47.180 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.180 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.181 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.181 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.181 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.182 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.182 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.182 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.183 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.183 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.183 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.184 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:47.184 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:47.184 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:47.185 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:47.185 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.185 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.186 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:47.186 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:47.186 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.187 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.187 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:47.187 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.188 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.188 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:47.188 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.189 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.189 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.189 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:47.190 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.190 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:47.190 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:47.191 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:47.191 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.191 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.192 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.192 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.192 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:47.193 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:47.193 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.193 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.193 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:47.194 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:47.194 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.194 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.195 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.195 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:47.195 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:47.196 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:47.196 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.196 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.197 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.197 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.197 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.198 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.198 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.198 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.199 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.199 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.199 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.200 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:47.200 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:47.200 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.201 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.201 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:47.201 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.202 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.202 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.202 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.203 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:47.203 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20044 service root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:47.203 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.203 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.204 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:47.204 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.204 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:47.205 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.205 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:47.205 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:47.206 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:47.206 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:47.206 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.207 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.207 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.207 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.208 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.208 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.208 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.209 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:47.209 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:47.209 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.210 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:47.210 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:47.210 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:47.211 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:47.211 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.211 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.212 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:47.212 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.212 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:47.212 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.213 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:47.213 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.213 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.214 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.214 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.214 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.215 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.215 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.215 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.216 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.216 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.216 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.217 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:47.217 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.217 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.218 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.218 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.219 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.219 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.219 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.220 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.220 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:47.220 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.221 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:47.221 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.221 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.222 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.222 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:47.222 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.223 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.223 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.223 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.224 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.224 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.224 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.225 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.225 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.225 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.226 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.226 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:47.226 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:47.227 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:47.227 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:47.227 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.228 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.228 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:47.228 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:47.229 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.229 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.229 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:47.230 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.230 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.230 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:47.230 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.231 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.231 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.231 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:47.232 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.232 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:47.232 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:47.233 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:47.233 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.233 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.234 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.234 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.234 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:47.235 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:47.235 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.235 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.235 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:47.236 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:47.236 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.236 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.237 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.237 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:47.237 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:47.238 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:47.238 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.238 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.239 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.239 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.239 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.240 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.240 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.241 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.241 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.241 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.242 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.242 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:47.242 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:47.243 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.243 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.243 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:47.244 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.244 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.244 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.245 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.245 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:47.245 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20045 admin_soc root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:47.246 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.246 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.246 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:47.247 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.247 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:47.247 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.247 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:47.248 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:47.248 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:47.248 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:47.249 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.249 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.249 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.250 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.250 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.250 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.251 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.251 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:47.251 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:47.252 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.252 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:47.252 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:47.252 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:47.253 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:47.253 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.253 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.254 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:47.254 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.254 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:47.255 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.255 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:47.255 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.256 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.256 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.256 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.257 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.257 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.257 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.258 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.258 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.258 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:45.053+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:45.053968+0000) 2022-04-03T22:52:47.267 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:45.276+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:47.267 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:46.323+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:47.269 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.269 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:47.269 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.270 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.270 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.270 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.271 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.271 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.271 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.272 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.272 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:47.272 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.273 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:47.273 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.273 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.274 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.274 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:47.274 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.275 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.275 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.275 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.276 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.276 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.276 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.277 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.277 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.277 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.278 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.278 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:47.278 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:47.279 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:47.279 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:47.279 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.280 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.280 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:47.280 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:47.281 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.281 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.281 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:47.282 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.282 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.282 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:47.283 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.283 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.283 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.284 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:47.284 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.284 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:47.285 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:47.285 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:47.285 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.286 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.286 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.286 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.287 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:47.287 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:47.287 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.288 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.288 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:47.288 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:47.289 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.289 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.289 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.289 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:47.290 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:47.290 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:47.290 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.291 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.291 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.291 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.292 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.292 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.292 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.293 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.293 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.293 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.294 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.294 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:47.294 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:47.295 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.295 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.295 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:47.296 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.296 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.296 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.297 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.297 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:47.297 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20046 ceph_time root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:47.297 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.298 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.298 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:47.298 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.299 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:47.299 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.299 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:47.300 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:47.300 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:47.300 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:47.301 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.301 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.301 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.302 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.302 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.302 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.303 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.303 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:47.303 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:47.304 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.304 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:47.304 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:47.304 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:47.305 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:47.305 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.306 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.306 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:47.306 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.307 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:47.307 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.307 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:47.307 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.308 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.308 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.308 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.309 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.309 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.309 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.310 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.310 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.310 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.311 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.311 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:47.311 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.311 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.312 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.312 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.312 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.313 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.313 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.313 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.314 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:47.314 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.314 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:47.315 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.315 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.315 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.316 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:47.316 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.316 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.317 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.317 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.317 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.318 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.318 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.318 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.319 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.319 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.319 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.320 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:47.320 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:47.320 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:47.321 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:47.321 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.321 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.322 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:47.322 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:47.322 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.323 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.323 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:47.323 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.324 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.324 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:47.324 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.325 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.325 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.325 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:47.325 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.326 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:47.326 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:47.326 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:47.327 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.327 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.327 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.328 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.328 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:47.328 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:47.329 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.329 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.329 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:47.330 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:47.330 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.330 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.331 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.331 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:47.331 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:47.331 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:47.332 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.332 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.332 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.333 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.333 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.333 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.334 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.334 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.334 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.334 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.335 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.335 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:47.336 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:47.336 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.336 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.337 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:47.337 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.337 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.338 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.338 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.338 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:47.338 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20047 fn_anonym root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:47.339 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.339 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.339 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:47.340 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.340 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:47.341 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.341 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:47.341 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:47.341 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:47.342 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:47.342 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.343 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.343 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.343 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.343 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.344 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.344 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.344 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:47.345 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:47.345 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.345 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:47.346 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:47.346 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:47.346 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:47.347 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.347 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.347 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:47.348 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.348 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:47.348 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.349 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:47.349 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.349 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.350 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.350 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.350 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.351 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.351 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.351 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.352 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.352 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.352 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.353 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:47.353 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.353 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.354 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.354 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.354 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.355 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.355 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.355 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.356 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:47.356 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.356 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:47.357 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.357 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.357 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.358 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:47.358 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.358 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.359 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.359 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.359 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.360 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.360 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.360 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.361 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.361 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.361 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.362 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:47.362 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:47.362 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:47.363 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:47.363 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.363 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.364 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:47.364 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:47.364 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.365 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.365 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:47.365 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.366 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.366 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:47.366 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.367 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.367 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.367 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:47.368 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.368 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:47.368 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:47.369 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:47.369 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.369 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.370 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.370 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.370 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:47.371 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:47.371 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.371 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.372 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:47.372 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:47.372 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.373 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.373 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.373 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:47.374 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:47.374 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:47.374 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.375 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.375 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.375 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.376 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.376 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.376 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.377 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.377 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.377 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.378 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.378 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:47.378 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:47.379 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.379 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.379 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:47.379 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.380 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.380 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.380 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.381 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:47.381 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20048 signal_ha root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:47.381 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.382 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.382 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:47.382 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.383 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:47.383 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.383 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:47.384 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:47.384 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:47.384 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:47.384 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.385 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.385 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.385 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.386 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.386 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.386 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.387 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:47.387 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:47.387 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.388 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:47.388 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:47.388 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:47.389 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:47.389 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.389 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.390 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:47.390 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.390 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:47.391 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.391 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:47.391 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.392 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.392 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.392 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.393 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.393 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.393 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.394 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.394 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.394 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.395 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.395 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:47.395 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.396 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.396 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.396 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.397 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.397 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.397 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.397 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.398 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:47.398 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.398 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:47.399 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.399 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.399 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.400 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:47.400 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.400 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.400 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.401 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.401 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.401 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.402 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.402 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.402 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.403 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.403 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.403 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:47.404 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:47.404 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:47.404 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:47.405 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.405 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.405 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:47.405 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:47.406 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.406 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.406 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:47.407 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.407 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.407 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:47.408 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.408 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.408 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.409 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:47.409 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.409 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:47.410 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:47.410 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:47.410 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.411 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.411 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.411 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.412 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:47.412 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:47.412 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.413 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.413 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:47.413 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:47.414 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.414 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.414 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.415 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:47.415 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:47.415 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:47.416 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.416 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.416 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.417 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.417 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.417 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.418 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.418 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.418 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.419 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.419 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.419 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:47.420 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:47.420 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.420 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.421 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:47.421 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.421 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.422 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.422 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.422 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:47.423 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20049 mgrsb-fin root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:47.423 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.423 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.424 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:47.424 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.424 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:47.425 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.425 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:47.425 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:47.426 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:47.426 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:47.426 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.427 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.427 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.427 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.428 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.428 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.428 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.429 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:47.429 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:47.429 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.430 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:47.430 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:47.430 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:47.431 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:47.431 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.431 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.432 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:47.432 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.432 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:47.433 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.433 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:47.433 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.434 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.434 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.434 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.435 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.435 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.435 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.436 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.436 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.436 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.436 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.437 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:47.437 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.437 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.438 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.438 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.438 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.439 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.439 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.439 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.440 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:47.440 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.440 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:47.441 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.441 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.441 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.442 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:47.442 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.442 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.443 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.443 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.443 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.444 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.444 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.444 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.445 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.445 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.445 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.446 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:47.446 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:47.446 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:47.447 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:47.447 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.447 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.448 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:47.448 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:47.448 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.449 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.449 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:47.449 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.450 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.450 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:47.450 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.451 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.451 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.451 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:47.452 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.452 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:47.452 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:47.452 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:47.453 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.453 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.453 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.454 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.454 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:47.454 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:47.455 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.455 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.455 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:47.456 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:47.456 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.456 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.457 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.457 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:47.457 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:47.458 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:47.458 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.458 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.458 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.459 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.459 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.459 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.460 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.460 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.460 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.461 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.461 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.461 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:47.462 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:47.462 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.462 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.463 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:47.463 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.463 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.464 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.464 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.464 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:47.465 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20050 ms_dispat root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:47.465 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.465 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.465 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:47.466 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.466 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:47.467 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.467 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:47.467 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:47.467 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:47.468 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:47.468 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.468 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.469 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.469 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.469 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.470 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.470 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.470 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:47.471 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:47.471 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.471 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:47.472 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:47.472 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:47.472 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:47.473 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.473 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.473 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:47.474 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.474 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:47.475 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.475 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:47.475 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.476 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.476 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.476 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.476 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.477 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.477 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.477 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.478 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.478 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.478 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.479 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:47.479 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.480 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.480 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.480 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.480 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.481 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.481 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.481 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.482 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:47.482 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.482 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:47.483 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.483 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.483 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.484 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:47.484 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.484 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.485 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.485 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.485 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.486 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.486 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.486 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.487 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.487 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.487 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.488 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:47.488 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:47.488 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:47.489 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:47.489 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.489 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.490 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:47.490 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:47.490 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.491 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.491 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:47.491 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.492 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.492 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:47.493 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.493 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.493 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.494 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:47.494 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.494 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:47.495 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:47.495 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:47.495 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.496 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.496 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.496 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.497 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:47.497 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:47.497 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.498 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.498 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:47.498 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:47.499 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.499 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.499 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.500 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:47.500 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:47.500 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:47.501 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.501 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.501 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.502 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.502 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.502 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.503 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.503 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.503 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.504 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.504 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.504 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:47.505 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:47.505 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.505 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.506 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:47.506 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.506 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.507 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.507 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.507 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:47.508 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20051 ms_local root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:47.508 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.508 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.509 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:47.509 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.509 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:47.510 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.510 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:47.510 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:47.511 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:47.511 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:47.511 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.512 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.512 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.512 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.513 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.513 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.513 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.513 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:47.514 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:47.514 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.514 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:47.515 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:47.515 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:47.516 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:47.516 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.516 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.517 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:47.517 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.517 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:47.518 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.518 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:47.518 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.519 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.519 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.519 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.520 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.520 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.520 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.521 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.521 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.521 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.522 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.522 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:47.522 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.523 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.523 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.523 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.524 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.524 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.524 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.525 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.525 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:47.525 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.526 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:47.526 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.526 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.527 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.527 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:47.527 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.527 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.528 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.528 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.528 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.529 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.529 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.529 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.530 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.530 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.530 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.531 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:47.531 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:47.531 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:47.532 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:47.532 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.532 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.533 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:47.533 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:47.533 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.534 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.534 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:47.534 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.535 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.535 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:47.535 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.536 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.536 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.536 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:47.537 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.537 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:47.537 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:47.538 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:47.538 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.538 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.539 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.539 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.540 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:47.540 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:47.540 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.541 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.541 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:47.541 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:47.542 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.542 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.542 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.543 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:47.543 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:47.543 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:47.544 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.544 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.544 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.545 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.545 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.545 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.546 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.546 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.546 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.547 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.547 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.547 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:47.548 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:47.548 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.548 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.549 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:47.549 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.549 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.550 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.550 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.550 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:47.551 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20052 io_contex root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:47.551 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.551 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.552 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:47.552 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.552 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:47.552 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.553 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:47.553 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:47.554 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:47.554 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:47.554 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.555 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.555 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.555 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.556 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.556 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.556 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.557 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:47.557 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:47.557 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.557 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:47.558 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:47.558 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:47.559 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:47.559 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.559 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.560 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:47.560 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.560 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:47.560 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.561 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:47.561 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.561 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.562 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.562 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.563 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.563 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.563 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.564 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.564 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.564 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.565 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.565 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:47.565 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.566 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.566 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.567 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.567 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.567 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.568 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.568 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.568 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:47.569 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.569 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:47.569 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.569 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.570 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.570 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:47.571 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.571 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.571 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.572 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.572 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.572 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.572 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.573 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.573 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.573 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.574 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.574 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:47.574 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:47.575 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:47.575 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:47.575 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.576 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.576 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:47.576 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:47.577 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.577 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.577 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:47.578 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.578 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.578 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:47.579 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.579 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.579 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.580 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:47.580 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.580 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:47.581 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:47.581 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:47.581 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.582 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.582 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.582 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.583 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:47.583 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:47.583 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.584 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.584 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:47.584 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:47.585 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.585 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.585 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.586 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:47.586 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:47.586 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:47.587 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.587 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.587 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.588 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.588 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.588 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.589 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.589 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.589 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.590 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.590 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.590 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:47.591 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:47.591 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.591 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.592 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:47.592 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.592 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.593 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.593 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.593 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:47.594 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20053 io_contex root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:47.594 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.594 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.595 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:47.595 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.595 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:47.596 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.596 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:47.596 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:47.597 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:47.597 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:47.597 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.598 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.598 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.598 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.599 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.599 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.599 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.600 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:47.600 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:47.600 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.600 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:47.601 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:47.601 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:47.602 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:47.602 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.602 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.603 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:47.603 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.603 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:47.604 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.604 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:47.604 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.605 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.605 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.605 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.606 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.606 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.607 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.607 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.607 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.608 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.608 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.608 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:47.609 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.609 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.609 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.610 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.610 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.610 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.611 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.611 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.611 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:47.612 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.612 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:47.613 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.613 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.613 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.614 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:47.614 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.614 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.615 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.615 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.615 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.615 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.616 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.616 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.616 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.617 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.617 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.617 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:47.618 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:47.618 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:47.619 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:47.619 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.619 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.620 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:47.620 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:47.620 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.621 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.621 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:47.621 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.622 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.622 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:47.622 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.623 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.623 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.623 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:47.624 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.624 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:47.624 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:47.625 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:47.625 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.625 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.626 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.626 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.626 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:47.627 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:47.627 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.627 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.628 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:47.628 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:47.628 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.629 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.629 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.629 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:47.630 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:47.630 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:47.631 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.631 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.631 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.631 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.632 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.632 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.633 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.633 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.633 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.633 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.634 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.634 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:47.634 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:47.635 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.635 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.635 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:47.636 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.636 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.636 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.637 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.637 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:47.637 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20054 safe_time root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:47.638 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.638 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.638 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:47.639 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.639 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:47.639 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.640 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:47.640 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:47.640 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:47.641 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:47.641 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.641 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.642 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.642 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.643 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.643 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.643 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.644 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:47.644 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:47.644 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.645 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:47.645 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:47.645 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:47.646 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:47.646 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.646 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.647 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:47.647 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.647 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:47.648 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.648 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:47.648 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.649 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.649 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.649 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.650 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.650 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.650 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.651 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.651 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.651 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.652 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.652 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:47.652 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.653 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.653 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.653 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.654 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.654 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.654 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.655 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.655 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:47.655 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.656 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:47.656 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.656 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.657 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.657 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:47.657 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.658 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.658 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.658 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.659 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.659 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.659 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.660 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.660 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.660 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.661 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.661 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:47.661 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:47.662 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:47.662 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:47.662 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.663 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.663 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:47.663 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:47.664 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.664 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.664 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:47.665 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.665 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.665 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:47.666 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.666 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.666 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.667 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:47.667 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.667 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:47.668 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:47.668 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:47.668 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.669 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.669 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.669 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.670 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:47.670 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:47.670 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.671 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.671 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:47.671 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:47.672 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.672 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.672 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.673 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:47.673 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:47.673 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:47.673 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.674 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.674 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.674 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.675 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.675 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.675 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.676 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.676 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.676 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.677 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.677 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:47.677 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:47.678 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.678 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.678 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:47.679 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.679 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.679 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.680 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.680 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:47.680 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20055 safe_time root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:47.681 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.681 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.681 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:47.682 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.682 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:47.682 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.683 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:47.683 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:47.683 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:47.684 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:47.684 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.684 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.685 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.685 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.685 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.686 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.686 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.686 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:47.687 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:47.687 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.687 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:47.688 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:47.688 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:47.688 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:47.689 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.689 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.689 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:47.690 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.690 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:47.690 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.691 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:47.691 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.691 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.692 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.692 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.692 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.693 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.693 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.693 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.694 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.694 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.694 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.695 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:47.695 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.695 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.696 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.696 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.696 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.697 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.697 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.697 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.698 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:47.698 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.698 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:47.699 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.699 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.699 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.700 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:47.700 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.700 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.701 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.701 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.701 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.702 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.702 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.702 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.703 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.703 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.703 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.704 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:47.704 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:47.704 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:47.705 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:47.705 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.705 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.706 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:47.706 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:47.706 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.707 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.707 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:47.707 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.708 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.708 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:47.708 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.709 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.709 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.709 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:47.710 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.710 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:47.710 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:47.711 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:47.711 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.711 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.712 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.712 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.712 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:47.713 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:47.713 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.713 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.714 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:47.714 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:47.714 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.715 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.715 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.715 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:47.716 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:47.716 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:47.716 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.717 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.717 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.717 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.718 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.718 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.718 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.719 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.719 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.719 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.720 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.720 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:47.720 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:47.721 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.721 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.721 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:47.722 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.722 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.722 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.723 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.723 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:47.723 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20056 safe_time root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:47.724 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.724 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.724 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:47.725 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.725 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:47.725 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.726 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:47.726 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:47.726 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:47.727 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:47.727 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.727 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.728 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.728 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.728 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.729 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.729 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.729 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:47.730 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:47.730 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.731 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:47.731 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:47.731 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:47.732 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:47.732 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.732 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.733 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:47.733 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.733 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:47.734 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.734 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:47.734 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.735 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.735 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.735 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.736 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.736 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.736 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.737 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.737 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.737 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.738 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.738 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:47.738 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.739 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.739 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.739 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.740 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.740 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.740 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.741 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.741 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:47.741 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.742 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:47.742 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.742 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.743 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.743 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:47.743 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.744 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.744 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.744 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.745 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.745 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.745 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.746 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.746 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.746 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.747 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.747 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:47.747 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:47.748 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:47.748 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:47.748 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.749 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.749 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:47.749 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:47.750 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.750 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.750 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:47.751 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.751 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.752 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:47.752 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.752 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.753 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.753 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:47.753 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.754 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:47.754 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:47.754 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:47.755 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.755 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.755 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.756 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.756 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:47.756 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:47.757 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.757 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.757 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:47.758 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:47.758 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.758 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.759 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.759 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:47.759 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:47.760 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:47.760 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.760 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.761 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.761 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.761 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.762 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.762 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.762 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.763 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.763 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.763 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.764 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:47.764 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:47.765 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.765 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.765 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:47.765 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.766 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.766 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.767 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.767 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:47.767 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20057 fn_anonym root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:47.768 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.768 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.768 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:47.769 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.769 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:47.769 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.770 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:47.770 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:47.770 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:47.771 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:47.771 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.771 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.772 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.772 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.772 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.773 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.773 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.773 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:47.774 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:47.774 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.774 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:47.775 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:47.775 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:47.775 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:47.776 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.776 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.776 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:47.777 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.777 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:47.777 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.778 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:47.778 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.778 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.779 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.779 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.779 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.780 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.780 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.780 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.781 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.781 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.781 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.782 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:47.782 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.783 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.783 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.783 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.784 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.784 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.784 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.785 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.785 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:47.785 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.786 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:47.786 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.786 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.787 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.787 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:47.787 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.788 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.788 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.788 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.789 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.789 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.789 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.790 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.790 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.790 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.791 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.791 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:47.791 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:47.792 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:47.792 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:47.792 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.793 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.793 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:47.793 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:47.794 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.794 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.794 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:47.795 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.795 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.795 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:47.796 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.796 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.796 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.797 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:47.797 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.797 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:47.798 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:47.798 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:47.798 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.799 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.799 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.799 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.800 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:47.800 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:47.800 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.801 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.801 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:47.801 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:47.802 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.802 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.802 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.803 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:47.803 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:47.803 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:47.804 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.804 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.804 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.805 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.805 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.805 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.806 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.806 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.806 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.807 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.807 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.807 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:47.808 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:47.808 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.808 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.809 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:47.809 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.809 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.810 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.810 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.810 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:47.811 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20058 flusher root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:47.811 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.811 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.812 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root txt REG 8,1 5685256 1107 /usr/bin/ceph-mgr 2022-04-03T22:52:47.812 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 29992 40799 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.812 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 239456 7902 /usr/lib64/libnettle.so.6.5 2022-04-03T22:52:47.813 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 76048 40762 /usr/lib64/python3.6/lib-dynload/_elementtree.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.813 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:47.813 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:47.814 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:47.814 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 12742816 2968 /usr/lib64/librbd.so.1.17.0 2022-04-03T22:52:47.815 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 1321072 3402 /usr/lib64/python3.6/site-packages/rbd.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.815 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 74568 525948 /usr/lib64/python3.6/site-packages/scipy/_lib/_ccallback_c.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.815 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 937648 525292 /usr/lib64/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.816 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 44512 525135 /usr/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.816 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 151544 525212 /usr/lib64/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.816 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 24016 525214 /usr/lib64/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.817 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 1527544 525024 /usr/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.817 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:47.817 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 243984 7529 /usr/lib64/libexpat.so.1.6.7 2022-04-03T22:52:47.818 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 66904 40793 /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.818 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 267496 3270 /usr/lib64/libquadmath.so.0.0.0 2022-04-03T22:52:47.818 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 2622416 3275 /usr/lib64/libgfortran.so.5.0.0 2022-04-03T22:52:47.819 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 29168064 3280 /usr/lib64/libopenblas-r0.3.12.so 2022-04-03T22:52:47.819 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 30683352 3277 /usr/lib64/libopenblasp-r0.3.12.so 2022-04-03T22:52:47.819 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 1901024 524963 /usr/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.820 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 39464 40755 /usr/lib64/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.820 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 133904 8227 /usr/lib64/libyaml-0.so.2.0.5 2022-04-03T22:52:47.820 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 249472 42192 /usr/lib64/python3.6/site-packages/_yaml.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.821 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 824472 656788 /usr/lib64/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so 2022-04-03T22:52:47.821 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 181064 3497 /usr/lib64/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.821 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 43352 657492 /usr/lib64/python3.6/site-packages/bcrypt/_bcrypt.abi3.so 2022-04-03T22:52:47.822 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 55488 40744 /usr/lib64/python3.6/lib-dynload/_asyncio.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.822 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 16888 40771 /usr/lib64/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.822 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 66056 40782 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.823 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 899568 40800 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.823 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 131584 40773 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.823 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 12200 524719 /usr/lib64/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.824 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 315336 40761 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.824 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 17288 40766 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.824 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 120344 40779 /usr/lib64/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.825 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 30840 40784 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.825 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.825 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 2547600 3114 /usr/lib64/libcephfs.so.2.0.0 2022-04-03T22:52:47.826 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 574856 3301 /usr/lib64/python3.6/site-packages/cephfs.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.826 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 20528 40775 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.826 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 13696 40745 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.827 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 110640 40776 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.827 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 129528 40746 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.827 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.828 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 47920 40764 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.828 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 13200 40787 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.828 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:47.829 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 42856 40769 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.829 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 74744 7437 /usr/lib64/libbz2.so.1.0.6 2022-04-03T22:52:47.829 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 23032 40747 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.830 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 34392 40802 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.830 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.830 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 37024 7783 /usr/lib64/libffi.so.6.0.2 2022-04-03T22:52:47.831 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 133456 40756 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.831 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 110024 40778 /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.831 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 92984 40777 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.831 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 115216 40759 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.832 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 1292048 3116 /usr/lib64/python3.6/site-packages/rados.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.832 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.832 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.833 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.833 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 80112 40767 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.833 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 11592 40772 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.834 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.834 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:47.834 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:47.835 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:47.835 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:47.835 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.835 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.836 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:47.836 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:47.836 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.837 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.837 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:47.837 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.838 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.838 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 2152288 2965 /usr/lib64/librados.so.2.0.0 2022-04-03T22:52:47.838 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.839 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.839 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.839 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:47.839 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.840 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:47.840 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:47.840 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:47.841 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.841 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.841 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.841 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.842 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:47.842 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:47.842 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.843 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.843 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 1151640 7517 /usr/lib64/libsqlite3.so.0.8.6 2022-04-03T22:52:47.843 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 536152 3142 /usr/lib64/libcephsqlite.so 2022-04-03T22:52:47.844 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.844 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.844 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.844 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:47.845 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 1w FIFO 0,12 0t0 78424 pipe 2022-04-03T22:52:47.845 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 2w FIFO 0,12 0t0 78425 pipe 2022-04-03T22:52:47.845 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 3r FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.846 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 4w FIFO 0,12 0t0 78451 pipe 2022-04-03T22:52:47.846 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.846 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 6r FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.847 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 7w FIFO 0,12 0t0 78452 pipe 2022-04-03T22:52:47.847 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.847 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 9r FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.848 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 10w FIFO 0,12 0t0 78453 pipe 2022-04-03T22:52:47.848 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:47.848 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 12r FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 13w FIFO 0,12 0t0 78454 pipe 2022-04-03T22:52:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 14w REG 8,1 11151475 790203 /var/log/ceph/ceph-mgr.y.log 2022-04-03T22:52:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 15uW REG 0,24 6 1648 /run/ceph/ceph-mgr.y.pid 2022-04-03T22:52:47.849 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 16r FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.850 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 17w FIFO 0,12 0t0 82614 pipe 2022-04-03T22:52:47.850 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 18u unix 0xffff88817054f800 0t0 82615 /var/run/ceph/ceph-mgr.y.asok type=STREAM 2022-04-03T22:52:47.850 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 19r FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.851 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 20w FIFO 0,12 0t0 82619 pipe 2022-04-03T22:52:47.851 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 21r FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.851 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 22w FIFO 0,12 0t0 82620 pipe 2022-04-03T22:52:47.852 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 23u IPv4 80219 0t0 TCP smithi077.front.sepia.ceph.com:37578->smithi186.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:47.852 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mgr 19976 20059 safe_time root 24u IPv4 194721 0t0 TCP smithi077.front.sepia.ceph.com:60778->smithi186.front.sepia.ceph.com:6832 (ESTABLISHED) 2022-04-03T22:52:47.852 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.853 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.853 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root txt REG 8,1 165640 12133 /usr/bin/sudo 2022-04-03T22:52:47.853 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 15928 23305 /usr/lib64/security/pam_succeed_if.so 2022-04-03T22:52:47.854 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 16536 9706 /usr/lib64/libpam_misc.so.0.82.1 2022-04-03T22:52:47.854 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 915232 23524 /usr/lib64/security/pam_systemd.so 2022-04-03T22:52:47.854 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 24048 23285 /usr/lib64/security/pam_limits.so 2022-04-03T22:52:47.855 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 11736 23283 /usr/lib64/security/pam_keyinit.so 2022-04-03T22:52:47.855 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 46232 9162 /usr/lib64/libcrack.so.2.9.0 2022-04-03T22:52:47.855 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 24744 9666 /usr/lib64/libpwquality.so.1.0.2 2022-04-03T22:52:47.856 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 11720 23217 /usr/lib64/security/pam_pwquality.so 2022-04-03T22:52:47.856 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 7640 23294 /usr/lib64/security/pam_permit.so 2022-04-03T22:52:47.856 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 7552 23271 /usr/lib64/security/pam_deny.so 2022-04-03T22:52:47.857 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 58032 24580 /usr/lib64/security/pam_sss.so 2022-04-03T22:52:47.857 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 371032 9256 /usr/lib64/libtirpc.so.3.0.0 2022-04-03T22:52:47.857 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 116408 9258 /usr/lib64/libnsl.so.2.0.0 2022-04-03T22:52:47.858 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 58016 23310 /usr/lib64/security/pam_unix.so 2022-04-03T22:52:47.858 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 11744 23287 /usr/lib64/security/pam_localuser.so 2022-04-03T22:52:47.858 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 11816 23316 /usr/lib64/security/pam_usertype.so 2022-04-03T22:52:47.859 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 11720 23275 /usr/lib64/security/pam_faildelay.so 2022-04-03T22:52:47.859 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 20160 23273 /usr/lib64/security/pam_env.so 2022-04-03T22:52:47.859 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:47.860 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.860 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.860 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.861 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.861 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:47.861 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.862 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:47.862 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:47.863 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:47.863 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:47.863 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:47.864 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:47.864 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:47.864 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:47.865 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:47.865 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:47.865 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.866 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.866 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 125328 8879 /usr/lib64/libsasl2.so.3.0.0 2022-04-03T22:52:47.866 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.867 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.867 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 67136 8940 /usr/lib64/liblber-2.4.so.2.10.9 2022-04-03T22:52:47.867 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 325184 8942 /usr/lib64/libldap-2.4.so.2.10.9 2022-04-03T22:52:47.868 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:47.868 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 432872 35707 /usr/libexec/sudo/sudoers.so 2022-04-03T22:52:47.868 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:47.869 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:47.869 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:47.869 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.870 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:47.870 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.870 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.871 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.871 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 104424 35704 /usr/libexec/sudo/libsudo_util.so.0.0.0 2022-04-03T22:52:47.871 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.872 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.872 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:47.872 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.873 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.873 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:47.873 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:47.874 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:47.874 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:47.874 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:47.875 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:47.875 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:47.875 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:47.875 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:47.876 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.876 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:47.876 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root 0r FIFO 0,12 0t0 80529 pipe 2022-04-03T22:52:47.877 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root 1w FIFO 0,12 0t0 80530 pipe 2022-04-03T22:52:47.877 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root 2w FIFO 0,12 0t0 80531 pipe 2022-04-03T22:52:47.877 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root 3r REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:47.878 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root 4u unix 0xffff88810da1b480 0t0 84112 type=STREAM 2022-04-03T22:52:47.878 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root 5u netlink 0t0 84136 AUDIT 2022-04-03T22:52:47.878 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root 6r REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:47.879 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root 7r REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:47.879 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root 9u unix 0xffff888106ed0780 0t0 79476 type=DGRAM 2022-04-03T22:52:47.879 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root 10r FIFO 0,12 0t0 79479 pipe 2022-04-03T22:52:47.880 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20158 root 11w FIFO 0,12 0t0 79479 pipe 2022-04-03T22:52:47.880 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.880 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.881 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root txt REG 8,1 165640 12133 /usr/bin/sudo 2022-04-03T22:52:47.881 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 15928 23305 /usr/lib64/security/pam_succeed_if.so 2022-04-03T22:52:47.881 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 16536 9706 /usr/lib64/libpam_misc.so.0.82.1 2022-04-03T22:52:47.882 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 915232 23524 /usr/lib64/security/pam_systemd.so 2022-04-03T22:52:47.882 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 24048 23285 /usr/lib64/security/pam_limits.so 2022-04-03T22:52:47.882 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 11736 23283 /usr/lib64/security/pam_keyinit.so 2022-04-03T22:52:47.883 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 46232 9162 /usr/lib64/libcrack.so.2.9.0 2022-04-03T22:52:47.883 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 24744 9666 /usr/lib64/libpwquality.so.1.0.2 2022-04-03T22:52:47.883 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 11720 23217 /usr/lib64/security/pam_pwquality.so 2022-04-03T22:52:47.883 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 7640 23294 /usr/lib64/security/pam_permit.so 2022-04-03T22:52:47.884 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 7552 23271 /usr/lib64/security/pam_deny.so 2022-04-03T22:52:47.884 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 58032 24580 /usr/lib64/security/pam_sss.so 2022-04-03T22:52:47.884 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 371032 9256 /usr/lib64/libtirpc.so.3.0.0 2022-04-03T22:52:47.885 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 116408 9258 /usr/lib64/libnsl.so.2.0.0 2022-04-03T22:52:47.885 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 58016 23310 /usr/lib64/security/pam_unix.so 2022-04-03T22:52:47.885 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 11744 23287 /usr/lib64/security/pam_localuser.so 2022-04-03T22:52:47.886 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 11816 23316 /usr/lib64/security/pam_usertype.so 2022-04-03T22:52:47.886 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 11720 23275 /usr/lib64/security/pam_faildelay.so 2022-04-03T22:52:47.886 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 20160 23273 /usr/lib64/security/pam_env.so 2022-04-03T22:52:47.886 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:47.887 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.887 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.887 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.888 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.888 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:47.888 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.888 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:47.889 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:47.889 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:47.889 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:47.890 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:47.890 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:47.890 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:47.891 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:47.891 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:47.891 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:47.892 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.892 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.892 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 125328 8879 /usr/lib64/libsasl2.so.3.0.0 2022-04-03T22:52:47.893 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.893 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.893 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 67136 8940 /usr/lib64/liblber-2.4.so.2.10.9 2022-04-03T22:52:47.893 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 325184 8942 /usr/lib64/libldap-2.4.so.2.10.9 2022-04-03T22:52:47.894 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:47.894 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 432872 35707 /usr/libexec/sudo/sudoers.so 2022-04-03T22:52:47.894 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:47.895 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:47.895 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:47.895 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.896 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:47.896 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.896 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.897 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.897 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 104424 35704 /usr/libexec/sudo/libsudo_util.so.0.0.0 2022-04-03T22:52:47.897 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.897 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.898 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:47.898 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.898 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.899 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:47.899 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:47.899 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:47.900 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:47.900 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:47.900 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:47.901 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:47.901 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:47.901 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:47.901 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.902 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:47.902 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root 0r FIFO 0,12 0t0 79452 pipe 2022-04-03T22:52:47.902 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root 1w FIFO 0,12 0t0 79453 pipe 2022-04-03T22:52:47.903 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root 2w FIFO 0,12 0t0 79454 pipe 2022-04-03T22:52:47.903 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root 3r REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:47.903 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root 4u unix 0xffff888178e48780 0t0 77727 type=STREAM 2022-04-03T22:52:47.904 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root 5u netlink 0t0 84156 AUDIT 2022-04-03T22:52:47.904 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root 6r REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:47.904 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root 7r REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:47.905 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root 9u unix 0xffff888170003c00 0t0 81712 type=DGRAM 2022-04-03T22:52:47.905 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root 10r FIFO 0,12 0t0 81715 pipe 2022-04-03T22:52:47.905 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20160 root 11w FIFO 0,12 0t0 81715 pipe 2022-04-03T22:52:47.905 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.906 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.906 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root txt REG 8,1 165640 12133 /usr/bin/sudo 2022-04-03T22:52:47.906 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 15928 23305 /usr/lib64/security/pam_succeed_if.so 2022-04-03T22:52:47.907 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 16536 9706 /usr/lib64/libpam_misc.so.0.82.1 2022-04-03T22:52:47.907 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 915232 23524 /usr/lib64/security/pam_systemd.so 2022-04-03T22:52:47.907 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 24048 23285 /usr/lib64/security/pam_limits.so 2022-04-03T22:52:47.908 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 11736 23283 /usr/lib64/security/pam_keyinit.so 2022-04-03T22:52:47.908 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 46232 9162 /usr/lib64/libcrack.so.2.9.0 2022-04-03T22:52:47.908 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 24744 9666 /usr/lib64/libpwquality.so.1.0.2 2022-04-03T22:52:47.909 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 11720 23217 /usr/lib64/security/pam_pwquality.so 2022-04-03T22:52:47.909 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 7640 23294 /usr/lib64/security/pam_permit.so 2022-04-03T22:52:47.909 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 7552 23271 /usr/lib64/security/pam_deny.so 2022-04-03T22:52:47.910 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 58032 24580 /usr/lib64/security/pam_sss.so 2022-04-03T22:52:47.910 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 371032 9256 /usr/lib64/libtirpc.so.3.0.0 2022-04-03T22:52:47.910 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 116408 9258 /usr/lib64/libnsl.so.2.0.0 2022-04-03T22:52:47.910 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 58016 23310 /usr/lib64/security/pam_unix.so 2022-04-03T22:52:47.911 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 11744 23287 /usr/lib64/security/pam_localuser.so 2022-04-03T22:52:47.911 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 11816 23316 /usr/lib64/security/pam_usertype.so 2022-04-03T22:52:47.911 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 11720 23275 /usr/lib64/security/pam_faildelay.so 2022-04-03T22:52:47.912 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 20160 23273 /usr/lib64/security/pam_env.so 2022-04-03T22:52:47.912 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:47.912 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.913 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.913 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.913 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.914 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:47.914 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.914 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:47.915 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:47.915 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:47.915 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:47.916 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:47.916 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:47.916 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:47.917 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:47.917 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:47.917 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:47.918 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.918 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.918 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 125328 8879 /usr/lib64/libsasl2.so.3.0.0 2022-04-03T22:52:47.919 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.919 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.919 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 67136 8940 /usr/lib64/liblber-2.4.so.2.10.9 2022-04-03T22:52:47.920 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 325184 8942 /usr/lib64/libldap-2.4.so.2.10.9 2022-04-03T22:52:47.920 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:47.920 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 432872 35707 /usr/libexec/sudo/sudoers.so 2022-04-03T22:52:47.921 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:47.921 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:47.921 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:47.921 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.922 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:47.922 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.922 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.923 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.923 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 104424 35704 /usr/libexec/sudo/libsudo_util.so.0.0.0 2022-04-03T22:52:47.923 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.924 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.924 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:47.924 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.925 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.925 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:47.925 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:47.925 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:47.926 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:47.926 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:47.926 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:47.927 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:47.927 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:47.927 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:47.928 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.928 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:47.928 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root 0r FIFO 0,12 0t0 79458 pipe 2022-04-03T22:52:47.929 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root 1w FIFO 0,12 0t0 79459 pipe 2022-04-03T22:52:47.929 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root 2w FIFO 0,12 0t0 79460 pipe 2022-04-03T22:52:47.929 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root 3r REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:47.930 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root 4u unix 0xffff888170002580 0t0 81701 type=STREAM 2022-04-03T22:52:47.930 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root 5u netlink 0t0 81737 AUDIT 2022-04-03T22:52:47.930 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root 6r REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:47.931 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root 7r REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:47.931 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root 9u unix 0xffff88810907e180 0t0 80592 type=DGRAM 2022-04-03T22:52:47.931 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root 10r FIFO 0,12 0t0 80595 pipe 2022-04-03T22:52:47.932 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20165 root 11w FIFO 0,12 0t0 80595 pipe 2022-04-03T22:52:47.932 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.932 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.932 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root txt REG 8,1 165640 12133 /usr/bin/sudo 2022-04-03T22:52:47.933 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 15928 23305 /usr/lib64/security/pam_succeed_if.so 2022-04-03T22:52:47.933 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 16536 9706 /usr/lib64/libpam_misc.so.0.82.1 2022-04-03T22:52:47.933 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 915232 23524 /usr/lib64/security/pam_systemd.so 2022-04-03T22:52:47.934 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 24048 23285 /usr/lib64/security/pam_limits.so 2022-04-03T22:52:47.934 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 11736 23283 /usr/lib64/security/pam_keyinit.so 2022-04-03T22:52:47.934 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 46232 9162 /usr/lib64/libcrack.so.2.9.0 2022-04-03T22:52:47.935 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 24744 9666 /usr/lib64/libpwquality.so.1.0.2 2022-04-03T22:52:47.935 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 11720 23217 /usr/lib64/security/pam_pwquality.so 2022-04-03T22:52:47.935 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 7640 23294 /usr/lib64/security/pam_permit.so 2022-04-03T22:52:47.935 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 7552 23271 /usr/lib64/security/pam_deny.so 2022-04-03T22:52:47.936 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 58032 24580 /usr/lib64/security/pam_sss.so 2022-04-03T22:52:47.936 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 371032 9256 /usr/lib64/libtirpc.so.3.0.0 2022-04-03T22:52:47.936 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 116408 9258 /usr/lib64/libnsl.so.2.0.0 2022-04-03T22:52:47.937 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 58016 23310 /usr/lib64/security/pam_unix.so 2022-04-03T22:52:47.937 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 11744 23287 /usr/lib64/security/pam_localuser.so 2022-04-03T22:52:47.937 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 11816 23316 /usr/lib64/security/pam_usertype.so 2022-04-03T22:52:47.937 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 11720 23275 /usr/lib64/security/pam_faildelay.so 2022-04-03T22:52:47.938 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 20160 23273 /usr/lib64/security/pam_env.so 2022-04-03T22:52:47.938 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:47.938 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:47.939 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:47.939 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:47.939 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:47.940 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:47.940 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:47.940 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:47.940 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:47.941 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:47.941 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:47.941 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:47.942 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:47.942 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:47.942 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:47.943 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:47.943 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:47.943 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.943 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:47.944 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 125328 8879 /usr/lib64/libsasl2.so.3.0.0 2022-04-03T22:52:47.944 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:47.944 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.945 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 67136 8940 /usr/lib64/liblber-2.4.so.2.10.9 2022-04-03T22:52:47.945 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 325184 8942 /usr/lib64/libldap-2.4.so.2.10.9 2022-04-03T22:52:47.945 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:47.946 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 432872 35707 /usr/libexec/sudo/sudoers.so 2022-04-03T22:52:47.946 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:47.946 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:47.947 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:47.947 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:47.947 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:47.948 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.948 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.948 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.949 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 104424 35704 /usr/libexec/sudo/libsudo_util.so.0.0.0 2022-04-03T22:52:47.949 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.949 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:47.949 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:47.950 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.950 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.950 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:47.951 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:47.951 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:47.951 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:47.952 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:47.952 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:47.952 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:47.953 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:47.953 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:47.953 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.953 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:47.954 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root 0r FIFO 0,12 0t0 81689 pipe 2022-04-03T22:52:47.954 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root 1w FIFO 0,12 0t0 81690 pipe 2022-04-03T22:52:47.954 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root 2w FIFO 0,12 0t0 81691 pipe 2022-04-03T22:52:47.955 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root 3r REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:47.955 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root 4u unix 0xffff88810907bc00 0t0 80580 type=STREAM 2022-04-03T22:52:47.955 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root 5u netlink 0t0 84179 AUDIT 2022-04-03T22:52:47.956 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root 6r REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:47.956 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root 7r REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:47.956 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root 9u unix 0xffff88810907b480 0t0 80591 type=DGRAM 2022-04-03T22:52:47.956 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root 10r FIFO 0,12 0t0 79487 pipe 2022-04-03T22:52:47.957 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 20172 root 11w FIFO 0,12 0t0 79487 pipe 2022-04-03T22:52:47.957 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.957 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.958 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root txt REG 8,1 11816 8955 /usr/libexec/platform-python3.6 2022-04-03T22:52:47.958 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.958 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.959 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.959 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.959 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.960 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.960 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:47.960 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.960 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.961 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.961 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.961 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.962 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.962 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.962 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.963 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.963 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.963 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:47.964 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:47.964 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:47.964 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:47.964 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:47.965 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:47.965 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:47.965 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:47.966 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:47.966 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.966 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:47.967 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root 0r FIFO 0,12 0t0 80529 pipe 2022-04-03T22:52:47.967 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root 1w FIFO 0,12 0t0 80530 pipe 2022-04-03T22:52:47.967 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20227 root 2w FIFO 0,12 0t0 80531 pipe 2022-04-03T22:52:47.967 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.968 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.968 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root txt REG 8,1 11816 8955 /usr/libexec/platform-python3.6 2022-04-03T22:52:47.968 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.969 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.969 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.969 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.970 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.970 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.970 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:47.971 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.971 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.971 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.972 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.972 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.972 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.973 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.973 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.973 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.973 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.974 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:47.974 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:47.975 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:47.975 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:47.975 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:47.975 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:47.976 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:47.976 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:47.976 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:47.977 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.977 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:47.977 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root 0r FIFO 0,12 0t0 79452 pipe 2022-04-03T22:52:47.978 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root 1w FIFO 0,12 0t0 79453 pipe 2022-04-03T22:52:47.978 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20231 root 2w FIFO 0,12 0t0 79454 pipe 2022-04-03T22:52:47.978 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.979 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.979 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root txt REG 8,1 11816 8955 /usr/libexec/platform-python3.6 2022-04-03T22:52:47.979 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.980 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.980 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.980 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.981 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.981 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.981 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:47.981 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.982 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.982 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.982 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.983 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.983 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.983 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.984 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.984 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.984 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.985 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:47.985 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:47.985 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:47.986 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:47.986 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:47.986 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:47.987 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:47.987 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:47.987 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:47.988 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.988 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:47.988 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root 0r FIFO 0,12 0t0 79458 pipe 2022-04-03T22:52:47.989 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root 1w FIFO 0,12 0t0 79459 pipe 2022-04-03T22:52:47.989 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20239 root 2w FIFO 0,12 0t0 79460 pipe 2022-04-03T22:52:47.989 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:47.990 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:47.990 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root txt REG 8,1 11816 8955 /usr/libexec/platform-python3.6 2022-04-03T22:52:47.990 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.991 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.991 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.991 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.992 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.992 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:47.992 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:47.993 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:47.993 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:47.993 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:47.994 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:47.994 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:47.994 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:47.995 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:47.995 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:47.995 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:47.996 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:47.996 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:47.996 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:47.996 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:47.997 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:47.997 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:47.997 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:47.998 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:47.998 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:47.998 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:47.999 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:47.999 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:47.999 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root 0r FIFO 0,12 0t0 81689 pipe 2022-04-03T22:52:47.999 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root 1w FIFO 0,12 0t0 81690 pipe 2022-04-03T22:52:48.000 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 20240 root 2w FIFO 0,12 0t0 81691 pipe 2022-04-03T22:52:48.000 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.000 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.001 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root txt unknown /proc/20248/task/23617/exe 2022-04-03T22:52:48.001 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.001 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 1w FIFO 0,12 0t0 79459 pipe 2022-04-03T22:52:48.002 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 2w FIFO 0,12 0t0 79460 pipe 2022-04-03T22:52:48.002 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 3r FIFO 0,12 0t0 77777 pipe 2022-04-03T22:52:48.002 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 4w FIFO 0,12 0t0 77777 pipe 2022-04-03T22:52:48.003 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.003 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 6r FIFO 0,12 0t0 77778 pipe 2022-04-03T22:52:48.003 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 7w FIFO 0,12 0t0 77778 pipe 2022-04-03T22:52:48.004 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.004 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 9r FIFO 0,12 0t0 77779 pipe 2022-04-03T22:52:48.004 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 10w FIFO 0,12 0t0 77779 pipe 2022-04-03T22:52:48.005 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.005 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 12r FIFO 0,12 0t0 77780 pipe 2022-04-03T22:52:48.005 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 13w FIFO 0,12 0t0 77780 pipe 2022-04-03T22:52:48.005 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 14w REG 253,4 2271343 25165970 /var/lib/ceph/crash/2022-04-03T20:01:37.566747Z_e5c9cbee-875a-4526-bedc-b6b60552542b/log 2022-04-03T22:52:48.006 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 15w REG 253,4 0 25165967 /var/lib/ceph/crash/2022-04-03T20:01:37.566747Z_e5c9cbee-875a-4526-bedc-b6b60552542b/done 2022-04-03T22:52:48.006 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 16r FIFO 0,12 0t0 77781 pipe 2022-04-03T22:52:48.006 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 17w FIFO 0,12 0t0 77781 pipe 2022-04-03T22:52:48.007 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 18u unix 0xffff888178e4da00 0t0 77782 /var/run/ceph/ceph-osd.2.asok type=STREAM 2022-04-03T22:52:48.007 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 19u IPv4 81823 0t0 TCP *:6800 (LISTEN) 2022-04-03T22:52:48.007 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 20u IPv4 81825 0t0 TCP *:acnet (LISTEN) 2022-04-03T22:52:48.008 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 21u IPv4 81828 0t0 TCP *:6802 (LISTEN) 2022-04-03T22:52:48.008 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 22u IPv4 81832 0t0 TCP *:6803 (LISTEN) 2022-04-03T22:52:48.008 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 23u IPv4 78697 0t0 TCP *:6806 (LISTEN) 2022-04-03T22:52:48.009 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 24u IPv4 78706 0t0 TCP *:6808 (LISTEN) 2022-04-03T22:52:48.009 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 25u IPv4 78717 0t0 TCP *:6810 (LISTEN) 2022-04-03T22:52:48.009 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 26u IPv4 78731 0t0 TCP *:6813 (LISTEN) 2022-04-03T22:52:48.010 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 27r FIFO 0,12 0t0 84218 pipe 2022-04-03T22:52:48.010 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 28w FIFO 0,12 0t0 84218 pipe 2022-04-03T22:52:48.010 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 29r FIFO 0,12 0t0 84219 pipe 2022-04-03T22:52:48.011 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 30w FIFO 0,12 0t0 84219 pipe 2022-04-03T22:52:48.011 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 31r DIR 253,2 278 64 /var/lib/ceph/osd/ceph-2 2022-04-03T22:52:48.011 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 32uW REG 253,2 37 69 /var/lib/ceph/osd/ceph-2/fsid 2022-04-03T22:52:48.012 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 33u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.012 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 34u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.012 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 35u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.013 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 36u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.013 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 37u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.013 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 38u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.014 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 39u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.014 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 40u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.015 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 41u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.015 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 42u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.015 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 43u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.015 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 44u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.016 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 45u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.016 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 46u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.016 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 47u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.017 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 48u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.017 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 49u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.017 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 50u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.018 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 51u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.018 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 52u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.018 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 53u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.019 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 54u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.019 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 55u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.019 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 56u REG 253,2 96636764160 70 /var/lib/ceph/osd/ceph-2/block 2022-04-03T22:52:48.020 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 57u IPv4 79674 0t0 TCP smithi077.front.sepia.ceph.com:59722->smithi186.front.sepia.ceph.com:3301 (CLOSE_WAIT) 2022-04-03T22:52:48.020 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 58r FIFO 0,12 0t0 86105 pipe 2022-04-03T22:52:48.020 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 59w FIFO 0,12 0t0 86105 pipe 2022-04-03T22:52:48.021 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 60u IPv4 140657 0t0 TCP smithi077.front.sepia.ceph.com:6806->smithi186.front.sepia.ceph.com:52284 (CLOSE_WAIT) 2022-04-03T22:52:48.021 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 61r FIFO 0,12 0t0 86106 pipe 2022-04-03T22:52:48.021 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 62w FIFO 0,12 0t0 86106 pipe 2022-04-03T22:52:48.022 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 63u IPv4 140660 0t0 TCP smithi077.front.sepia.ceph.com:6810->smithi186.front.sepia.ceph.com:52694 (CLOSE_WAIT) 2022-04-03T22:52:48.022 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 64u IPv4 136198 0t0 TCP smithi077.front.sepia.ceph.com:6810->smithi186.front.sepia.ceph.com:52250 (CLOSE_WAIT) 2022-04-03T22:52:48.022 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 65u IPv4 138920 0t0 TCP smithi077.front.sepia.ceph.com:56578->smithi186.front.sepia.ceph.com:6832 (CLOSE_WAIT) 2022-04-03T22:52:48.023 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 66u IPv4 133710 0t0 TCP smithi077.front.sepia.ceph.com:6806->smithi186.front.sepia.ceph.com:51938 (CLOSE_WAIT) 2022-04-03T22:52:48.023 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 67u IPv4 131004 0t0 TCP smithi077.front.sepia.ceph.com:6806->smithi077.front.sepia.ceph.com:47782 (ESTABLISHED) 2022-04-03T22:52:48.023 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 69u IPv4 120118 0t0 TCP smithi077.front.sepia.ceph.com:58832->smithi077.front.sepia.ceph.com:6822 (ESTABLISHED) 2022-04-03T22:52:48.023 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 70u IPv4 118499 0t0 TCP smithi077.front.sepia.ceph.com:47484->smithi077.front.sepia.ceph.com:6820 (ESTABLISHED) 2022-04-03T22:52:48.024 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 71u IPv4 86120 0t0 TCP smithi077.front.sepia.ceph.com:42280->smithi186.front.sepia.ceph.com:6830 (CLOSE_WAIT) 2022-04-03T22:52:48.024 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 72u IPv4 87100 0t0 TCP smithi077.front.sepia.ceph.com:33592->smithi186.front.sepia.ceph.com:6828 (CLOSE_WAIT) 2022-04-03T22:52:48.024 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 73u IPv4 136767 0t0 TCP smithi077.front.sepia.ceph.com:34408->smithi186.front.sepia.ceph.com:6814 (CLOSE_WAIT) 2022-04-03T22:52:48.025 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 74u IPv4 140472 0t0 TCP smithi077.front.sepia.ceph.com:34790->smithi186.front.sepia.ceph.com:6812 (CLOSE_WAIT) 2022-04-03T22:52:48.025 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 75u IPv4 137789 0t0 TCP smithi077.front.sepia.ceph.com:52016->smithi186.front.sepia.ceph.com:6804 (CLOSE_WAIT) 2022-04-03T22:52:48.025 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 76u IPv4 123892 0t0 TCP smithi077.front.sepia.ceph.com:40372->smithi186.front.sepia.ceph.com:6825 (CLOSE_WAIT) 2022-04-03T22:52:48.026 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 77u IPv4 123893 0t0 TCP smithi077.front.sepia.ceph.com:57100->smithi186.front.sepia.ceph.com:6821 (CLOSE_WAIT) 2022-04-03T22:52:48.026 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 78u IPv4 142403 0t0 TCP smithi077.front.sepia.ceph.com:43092->smithi186.front.sepia.ceph.com:6806 (CLOSE_WAIT) 2022-04-03T22:52:48.026 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 81u IPv4 134016 0t0 TCP smithi077.front.sepia.ceph.com:6810->smithi077.front.sepia.ceph.com:45508 (ESTABLISHED) 2022-04-03T22:52:48.027 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 82u IPv4 133713 0t0 TCP smithi077.front.sepia.ceph.com:6810->smithi186.front.sepia.ceph.com:52350 (CLOSE_WAIT) 2022-04-03T22:52:48.027 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 83u IPv4 131001 0t0 TCP smithi077.front.sepia.ceph.com:6810->smithi077.front.sepia.ceph.com:45512 (ESTABLISHED) 2022-04-03T22:52:48.027 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 84u IPv4 134019 0t0 TCP smithi077.front.sepia.ceph.com:6806->smithi077.front.sepia.ceph.com:47778 (ESTABLISHED) 2022-04-03T22:52:48.028 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 85u IPv4 139596 0t0 TCP smithi077.front.sepia.ceph.com:6810->smithi186.front.sepia.ceph.com:52628 (CLOSE_WAIT) 2022-04-03T22:52:48.028 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 86u IPv4 136201 0t0 TCP smithi077.front.sepia.ceph.com:6806->smithi186.front.sepia.ceph.com:51840 (CLOSE_WAIT) 2022-04-03T22:52:48.028 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 87u IPv4 138477 0t0 TCP smithi077.front.sepia.ceph.com:6810->smithi077.front.sepia.ceph.com:45510 (ESTABLISHED) 2022-04-03T22:52:48.029 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 88u IPv4 139599 0t0 TCP smithi077.front.sepia.ceph.com:6806->smithi186.front.sepia.ceph.com:52216 (CLOSE_WAIT) 2022-04-03T22:52:48.029 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 89u IPv4 138480 0t0 TCP smithi077.front.sepia.ceph.com:6806->smithi077.front.sepia.ceph.com:47780 (ESTABLISHED) 2022-04-03T22:52:48.029 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 91u IPv4 136759 0t0 TCP smithi077.front.sepia.ceph.com:6802->smithi186.front.sepia.ceph.com:57390 (CLOSE_WAIT) 2022-04-03T22:52:48.030 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 92u IPv4 131038 0t0 TCP smithi077.front.sepia.ceph.com:6802->smithi077.front.sepia.ceph.com:53516 (ESTABLISHED) 2022-04-03T22:52:48.030 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 93u IPv4 140404 0t0 TCP smithi077.front.sepia.ceph.com:6802->smithi186.front.sepia.ceph.com:57376 (CLOSE_WAIT) 2022-04-03T22:52:48.030 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 94u IPv4 83726 0t0 TCP smithi077.front.sepia.ceph.com:40422->smithi186.front.sepia.ceph.com:6818 (CLOSE_WAIT) 2022-04-03T22:52:48.031 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 96u IPv4 138590 0t0 TCP smithi077.front.sepia.ceph.com:6800->smithi186.front.sepia.ceph.com:58454 (CLOSE_WAIT) 2022-04-03T22:52:48.031 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 98u IPv4 132765 0t0 TCP smithi077.front.sepia.ceph.com:6802->smithi186.front.sepia.ceph.com:57248 (CLOSE_WAIT) 2022-04-03T22:52:48.031 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 99u IPv4 105625 0t0 TCP smithi077.front.sepia.ceph.com:6800->smithi186.front.sepia.ceph.com:57160 (CLOSE_WAIT) 2022-04-03T22:52:48.032 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20248 23617 tp_osd_tp root 101u IPv4 137672 0t0 TCP smithi077.front.sepia.ceph.com:acnet->192.168.0.1:37858 (ESTABLISHED) 2022-04-03T22:52:48.032 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.032 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.033 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root txt unknown /proc/20249/task/20472/exe 2022-04-03T22:52:48.033 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.033 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 1w FIFO 0,12 0t0 79453 pipe 2022-04-03T22:52:48.033 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 2w FIFO 0,12 0t0 79454 pipe 2022-04-03T22:52:48.034 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 3r FIFO 0,12 0t0 81782 pipe 2022-04-03T22:52:48.034 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 4w FIFO 0,12 0t0 81782 pipe 2022-04-03T22:52:48.034 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.035 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 6r FIFO 0,12 0t0 81783 pipe 2022-04-03T22:52:48.035 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 7w FIFO 0,12 0t0 81783 pipe 2022-04-03T22:52:48.035 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.036 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 9r FIFO 0,12 0t0 81784 pipe 2022-04-03T22:52:48.036 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 10w FIFO 0,12 0t0 81784 pipe 2022-04-03T22:52:48.036 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.037 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 12r FIFO 0,12 0t0 81785 pipe 2022-04-03T22:52:48.037 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 13w FIFO 0,12 0t0 81785 pipe 2022-04-03T22:52:48.037 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 14w REG 253,4 2260248 8388746 /var/lib/ceph/crash/2022-04-03T20:00:14.233132Z_54eab8da-9d10-4876-9436-af2af843fa7a/log 2022-04-03T22:52:48.038 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 15w REG 253,4 0 8388745 /var/lib/ceph/crash/2022-04-03T20:00:14.233132Z_54eab8da-9d10-4876-9436-af2af843fa7a/done 2022-04-03T22:52:48.038 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 16r FIFO 0,12 0t0 84209 pipe 2022-04-03T22:52:48.038 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 17w FIFO 0,12 0t0 84209 pipe 2022-04-03T22:52:48.039 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 18u unix 0xffff88810da1e180 0t0 84210 /var/run/ceph/ceph-osd.1.asok type=STREAM 2022-04-03T22:52:48.039 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 19u IPv4 79526 0t0 TCP *:6804 (LISTEN) 2022-04-03T22:52:48.039 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 20u IPv4 79532 0t0 TCP *:6805 (LISTEN) 2022-04-03T22:52:48.040 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 21u IPv4 79540 0t0 TCP *:6807 (LISTEN) 2022-04-03T22:52:48.040 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 22u IPv4 79550 0t0 TCP *:6809 (LISTEN) 2022-04-03T22:52:48.040 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 23u IPv4 81847 0t0 TCP *:6812 (LISTEN) 2022-04-03T22:52:48.040 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 24u IPv4 79555 0t0 TCP *:6815 (LISTEN) 2022-04-03T22:52:48.041 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 25u IPv4 78749 0t0 TCP *:pentbox-sim (LISTEN) 2022-04-03T22:52:48.041 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 26u IPv4 78769 0t0 TCP *:6819 (LISTEN) 2022-04-03T22:52:48.041 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 27r FIFO 0,12 0t0 81859 pipe 2022-04-03T22:52:48.042 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 28w FIFO 0,12 0t0 81859 pipe 2022-04-03T22:52:48.042 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 29r FIFO 0,12 0t0 81860 pipe 2022-04-03T22:52:48.042 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 30w FIFO 0,12 0t0 81860 pipe 2022-04-03T22:52:48.043 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 31r DIR 253,1 278 64 /var/lib/ceph/osd/ceph-1 2022-04-03T22:52:48.043 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 32uW REG 253,1 37 69 /var/lib/ceph/osd/ceph-1/fsid 2022-04-03T22:52:48.043 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 33u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.044 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 34u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.044 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 35u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.044 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 36u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.044 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 37u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.045 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 38u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.045 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 39u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.045 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 40u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.046 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 41u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.046 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 42u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.046 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 43u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.047 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 44u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.047 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 45u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.047 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 46u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.048 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 47u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.048 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 48u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.048 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 49u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.049 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 50u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.049 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 51u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.049 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 52u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.050 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 53u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.050 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 54u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.050 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 55u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.051 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 56u REG 253,1 96636764160 70 /var/lib/ceph/osd/ceph-1/block 2022-04-03T22:52:48.051 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 57r FIFO 0,12 0t0 81913 pipe 2022-04-03T22:52:48.051 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 58w FIFO 0,12 0t0 81913 pipe 2022-04-03T22:52:48.052 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 59u IPv4 83684 0t0 TCP smithi077.front.sepia.ceph.com:59724->smithi186.front.sepia.ceph.com:3301 (CLOSE_WAIT) 2022-04-03T22:52:48.052 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 60u IPv4 130123 0t0 TCP smithi077.front.sepia.ceph.com:pentbox-sim->smithi077.front.sepia.ceph.com:34776 (ESTABLISHED) 2022-04-03T22:52:48.052 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 61r FIFO 0,12 0t0 81914 pipe 2022-04-03T22:52:48.053 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 62w FIFO 0,12 0t0 81914 pipe 2022-04-03T22:52:48.053 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 63u IPv4 136177 0t0 TCP smithi077.front.sepia.ceph.com:53788->smithi186.front.sepia.ceph.com:6824 (CLOSE_WAIT) 2022-04-03T22:52:48.053 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 64u IPv4 127200 0t0 TCP smithi077.front.sepia.ceph.com:6812->smithi186.front.sepia.ceph.com:32864 (CLOSE_WAIT) 2022-04-03T22:52:48.053 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 65u IPv4 127203 0t0 TCP smithi077.front.sepia.ceph.com:pentbox-sim->smithi186.front.sepia.ceph.com:56956 (CLOSE_WAIT) 2022-04-03T22:52:48.054 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 66u IPv4 83711 0t0 TCP smithi077.front.sepia.ceph.com:pentbox-sim->smithi077.front.sepia.ceph.com:34768 (CLOSE_WAIT) 2022-04-03T22:52:48.054 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 67u IPv4 131568 0t0 TCP smithi077.front.sepia.ceph.com:pentbox-sim->smithi077.front.sepia.ceph.com:34778 (ESTABLISHED) 2022-04-03T22:52:48.055 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 68u IPv4 136866 0t0 TCP smithi077.front.sepia.ceph.com:pentbox-sim->smithi186.front.sepia.ceph.com:57746 (CLOSE_WAIT) 2022-04-03T22:52:48.055 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 69u IPv4 83716 0t0 TCP smithi077.front.sepia.ceph.com:6812->smithi077.front.sepia.ceph.com:55462 (CLOSE_WAIT) 2022-04-03T22:52:48.055 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 70u IPv4 136869 0t0 TCP smithi077.front.sepia.ceph.com:6812->smithi186.front.sepia.ceph.com:33654 (CLOSE_WAIT) 2022-04-03T22:52:48.056 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 71u IPv4 131571 0t0 TCP smithi077.front.sepia.ceph.com:6812->smithi077.front.sepia.ceph.com:55472 (ESTABLISHED) 2022-04-03T22:52:48.056 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 72u IPv4 126901 0t0 TCP smithi077.front.sepia.ceph.com:6812->smithi186.front.sepia.ceph.com:33192 (CLOSE_WAIT) 2022-04-03T22:52:48.056 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 73u IPv4 141487 0t0 TCP smithi077.front.sepia.ceph.com:6812->smithi186.front.sepia.ceph.com:33650 (CLOSE_WAIT) 2022-04-03T22:52:48.057 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 74u IPv4 141490 0t0 TCP smithi077.front.sepia.ceph.com:pentbox-sim->smithi186.front.sepia.ceph.com:57742 (CLOSE_WAIT) 2022-04-03T22:52:48.057 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 75u IPv4 126904 0t0 TCP smithi077.front.sepia.ceph.com:pentbox-sim->smithi186.front.sepia.ceph.com:57288 (CLOSE_WAIT) 2022-04-03T22:52:48.057 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 76u IPv4 139695 0t0 TCP smithi077.front.sepia.ceph.com:49882->smithi077.front.sepia.ceph.com:6828 (ESTABLISHED) 2022-04-03T22:52:48.057 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 77u IPv4 130943 0t0 TCP smithi077.front.sepia.ceph.com:45510->smithi077.front.sepia.ceph.com:6810 (ESTABLISHED) 2022-04-03T22:52:48.058 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 78u IPv4 88570 0t0 TCP smithi077.front.sepia.ceph.com:43068->smithi186.front.sepia.ceph.com:6806 (CLOSE_WAIT) 2022-04-03T22:52:48.058 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 79u IPv4 140446 0t0 TCP smithi077.front.sepia.ceph.com:33630->smithi186.front.sepia.ceph.com:6828 (CLOSE_WAIT) 2022-04-03T22:52:48.058 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 80u IPv4 130944 0t0 TCP smithi077.front.sepia.ceph.com:47780->smithi077.front.sepia.ceph.com:6806 (ESTABLISHED) 2022-04-03T22:52:48.059 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 81u IPv4 87674 0t0 TCP smithi077.front.sepia.ceph.com:51964->smithi186.front.sepia.ceph.com:6804 (CLOSE_WAIT) 2022-04-03T22:52:48.059 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 82u IPv4 136106 0t0 TCP smithi077.front.sepia.ceph.com:42316->smithi186.front.sepia.ceph.com:6830 (CLOSE_WAIT) 2022-04-03T22:52:48.059 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 83u IPv4 141403 0t0 TCP smithi077.front.sepia.ceph.com:34792->smithi186.front.sepia.ceph.com:6812 (CLOSE_WAIT) 2022-04-03T22:52:48.060 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 84u IPv4 131704 0t0 TCP smithi077.front.sepia.ceph.com:40374->smithi186.front.sepia.ceph.com:6825 (CLOSE_WAIT) 2022-04-03T22:52:48.060 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 85u IPv4 130595 0t0 TCP smithi077.front.sepia.ceph.com:58850->smithi077.front.sepia.ceph.com:6822 (ESTABLISHED) 2022-04-03T22:52:48.060 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 86u IPv4 136842 0t0 TCP smithi077.front.sepia.ceph.com:50996->smithi077.front.sepia.ceph.com:6830 (ESTABLISHED) 2022-04-03T22:52:48.061 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 87u IPv4 130596 0t0 TCP smithi077.front.sepia.ceph.com:47506->smithi077.front.sepia.ceph.com:6820 (ESTABLISHED) 2022-04-03T22:52:48.061 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 88u IPv4 137663 0t0 TCP smithi077.front.sepia.ceph.com:34410->smithi186.front.sepia.ceph.com:6814 (CLOSE_WAIT) 2022-04-03T22:52:48.061 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 89u IPv4 131052 0t0 TCP smithi077.front.sepia.ceph.com:55706->smithi077.front.sepia.ceph.com:6816 (ESTABLISHED) 2022-04-03T22:52:48.062 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 90u IPv4 82798 0t0 TCP smithi077.front.sepia.ceph.com:56036->smithi186.front.sepia.ceph.com:6832 (CLOSE_WAIT) 2022-04-03T22:52:48.062 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 91u IPv4 142400 0t0 TCP smithi077.front.sepia.ceph.com:36274->smithi186.front.sepia.ceph.com:6810 (CLOSE_WAIT) 2022-04-03T22:52:48.062 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 92u IPv4 125919 0t0 TCP smithi077.front.sepia.ceph.com:6812->smithi077.front.sepia.ceph.com:55470 (ESTABLISHED) 2022-04-03T22:52:48.063 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 93u IPv4 136774 0t0 TCP smithi077.front.sepia.ceph.com:58982->smithi077.front.sepia.ceph.com:6826 (ESTABLISHED) 2022-04-03T22:52:48.063 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 94u IPv4 130668 0t0 TCP smithi077.front.sepia.ceph.com:57102->smithi186.front.sepia.ceph.com:6821 (CLOSE_WAIT) 2022-04-03T22:52:48.063 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 95u IPv4 136131 0t0 TCP smithi077.front.sepia.ceph.com:6804->smithi186.front.sepia.ceph.com:59598 (CLOSE_WAIT) 2022-04-03T22:52:48.064 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 96u IPv4 131043 0t0 TCP smithi077.front.sepia.ceph.com:6807->smithi186.front.sepia.ceph.com:49662 (CLOSE_WAIT) 2022-04-03T22:52:48.064 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 97u IPv4 135610 0t0 TCP smithi077.front.sepia.ceph.com:40534->smithi186.front.sepia.ceph.com:6818 (CLOSE_WAIT) 2022-04-03T22:52:48.064 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 98u IPv4 138670 0t0 TCP smithi077.front.sepia.ceph.com:6807->smithi077.front.sepia.ceph.com:56150 (CLOSE_WAIT) 2022-04-03T22:52:48.065 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20249 20472 ceph-osd root 99u IPv4 140477 0t0 TCP smithi077.front.sepia.ceph.com:6805->192.168.0.1:57288 (ESTABLISHED) 2022-04-03T22:52:48.065 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.065 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.066 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root txt unknown /proc/20250/task/20255/exe 2022-04-03T22:52:48.066 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.066 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 1w FIFO 0,12 0t0 81690 pipe 2022-04-03T22:52:48.067 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 2w FIFO 0,12 0t0 81691 pipe 2022-04-03T22:52:48.067 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 3r FIFO 0,12 0t0 84201 pipe 2022-04-03T22:52:48.067 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 4w FIFO 0,12 0t0 84201 pipe 2022-04-03T22:52:48.068 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.068 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 6r FIFO 0,12 0t0 84202 pipe 2022-04-03T22:52:48.068 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 7w FIFO 0,12 0t0 84202 pipe 2022-04-03T22:52:48.068 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.069 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 9r FIFO 0,12 0t0 84203 pipe 2022-04-03T22:52:48.069 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 10w FIFO 0,12 0t0 84203 pipe 2022-04-03T22:52:48.069 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.070 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 12r FIFO 0,12 0t0 84204 pipe 2022-04-03T22:52:48.070 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 13w FIFO 0,12 0t0 84204 pipe 2022-04-03T22:52:48.071 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 14w REG 253,4 2298184 16797838 /var/lib/ceph/crash/2022-04-03T20:00:20.630169Z_1eb84f7c-91ae-4053-94ab-9c2cc1875894/log 2022-04-03T22:52:48.071 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 15w REG 253,4 0 16797837 /var/lib/ceph/crash/2022-04-03T20:00:20.630169Z_1eb84f7c-91ae-4053-94ab-9c2cc1875894/done 2022-04-03T22:52:48.071 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 16r FIFO 0,12 0t0 79512 pipe 2022-04-03T22:52:48.072 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 17w FIFO 0,12 0t0 79512 pipe 2022-04-03T22:52:48.072 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 18u unix 0xffff888106ed6900 0t0 79513 /var/run/ceph/ceph-osd.3.asok type=STREAM 2022-04-03T22:52:48.072 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 19u IPv4 82694 0t0 TCP *:6811 (LISTEN) 2022-04-03T22:52:48.073 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 20u IPv4 82709 0t0 TCP *:6814 (LISTEN) 2022-04-03T22:52:48.073 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 21u IPv4 80645 0t0 TCP *:6816 (LISTEN) 2022-04-03T22:52:48.073 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 22u IPv4 80664 0t0 TCP *:6818 (LISTEN) 2022-04-03T22:52:48.073 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 23u IPv4 80685 0t0 TCP *:6820 (LISTEN) 2022-04-03T22:52:48.074 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 24u IPv4 80707 0t0 TCP *:6821 (LISTEN) 2022-04-03T22:52:48.074 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 25u IPv4 80730 0t0 TCP *:6822 (LISTEN) 2022-04-03T22:52:48.074 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 26u IPv4 80755 0t0 TCP *:6824 (LISTEN) 2022-04-03T22:52:48.075 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 27r FIFO 0,12 0t0 78770 pipe 2022-04-03T22:52:48.075 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 28w FIFO 0,12 0t0 78770 pipe 2022-04-03T22:52:48.075 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 29r FIFO 0,12 0t0 78771 pipe 2022-04-03T22:52:48.076 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 30w FIFO 0,12 0t0 78771 pipe 2022-04-03T22:52:48.076 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 31r DIR 253,3 278 64 /var/lib/ceph/osd/ceph-3 2022-04-03T22:52:48.076 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 32uW REG 253,3 37 69 /var/lib/ceph/osd/ceph-3/fsid 2022-04-03T22:52:48.077 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 33u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.077 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 34u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.077 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 35u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.078 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 36u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.078 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 37u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.078 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 38u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.079 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 39u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.079 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 40u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.079 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 41u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.080 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 42u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.080 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 43u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.080 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 44u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.081 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 45u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.081 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 46u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.081 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 47u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.082 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 48u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.082 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 49u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.082 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 50u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.083 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 51u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.083 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 52u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.083 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 53u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.084 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 54u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.084 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 55u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.084 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 56u REG 253,3 96636764160 70 /var/lib/ceph/osd/ceph-3/block 2022-04-03T22:52:48.084 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 57r FIFO 0,12 0t0 86099 pipe 2022-04-03T22:52:48.085 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 58w FIFO 0,12 0t0 86099 pipe 2022-04-03T22:52:48.085 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 59u IPv4 81908 0t0 TCP smithi077.front.sepia.ceph.com:37598->smithi186.front.sepia.ceph.com:3300 (CLOSE_WAIT) 2022-04-03T22:52:48.085 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 60u IPv4 138692 0t0 TCP smithi077.front.sepia.ceph.com:6822->smithi186.front.sepia.ceph.com:46854 (CLOSE_WAIT) 2022-04-03T22:52:48.086 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 61r FIFO 0,12 0t0 86100 pipe 2022-04-03T22:52:48.086 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 62w FIFO 0,12 0t0 86100 pipe 2022-04-03T22:52:48.086 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 63u IPv4 138695 0t0 TCP smithi077.front.sepia.ceph.com:6820->smithi186.front.sepia.ceph.com:44728 (CLOSE_WAIT) 2022-04-03T22:52:48.087 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 64u IPv4 130801 0t0 TCP smithi077.front.sepia.ceph.com:45508->smithi077.front.sepia.ceph.com:6810 (ESTABLISHED) 2022-04-03T22:52:48.087 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 65u IPv4 130802 0t0 TCP smithi077.front.sepia.ceph.com:47778->smithi077.front.sepia.ceph.com:6806 (ESTABLISHED) 2022-04-03T22:52:48.087 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 66u IPv4 126900 0t0 TCP smithi077.front.sepia.ceph.com:33624->smithi186.front.sepia.ceph.com:6828 (CLOSE_WAIT) 2022-04-03T22:52:48.088 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 67u IPv4 129948 0t0 TCP smithi077.front.sepia.ceph.com:43088->smithi186.front.sepia.ceph.com:6806 (CLOSE_WAIT) 2022-04-03T22:52:48.088 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 68u IPv4 139561 0t0 TCP smithi077.front.sepia.ceph.com:57106->smithi186.front.sepia.ceph.com:6821 (CLOSE_WAIT) 2022-04-03T22:52:48.088 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 69u IPv4 129944 0t0 TCP smithi077.front.sepia.ceph.com:34778->smithi077.front.sepia.ceph.com:pentbox-sim (ESTABLISHED) 2022-04-03T22:52:48.089 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 70u IPv4 132110 0t0 TCP smithi077.front.sepia.ceph.com:42308->smithi186.front.sepia.ceph.com:6830 (CLOSE_WAIT) 2022-04-03T22:52:48.089 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 71u IPv4 136657 0t0 TCP smithi077.front.sepia.ceph.com:34406->smithi186.front.sepia.ceph.com:6814 (CLOSE_WAIT) 2022-04-03T22:52:48.089 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 72u IPv4 135570 0t0 TCP smithi077.front.sepia.ceph.com:52010->smithi186.front.sepia.ceph.com:6804 (CLOSE_WAIT) 2022-04-03T22:52:48.090 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 73u IPv4 136763 0t0 TCP smithi077.front.sepia.ceph.com:49880->smithi077.front.sepia.ceph.com:6828 (ESTABLISHED) 2022-04-03T22:52:48.090 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 74u IPv4 135562 0t0 TCP smithi077.front.sepia.ceph.com:55472->smithi077.front.sepia.ceph.com:6812 (ESTABLISHED) 2022-04-03T22:52:48.090 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 75u IPv4 117359 0t0 TCP smithi077.front.sepia.ceph.com:6820->smithi077.front.sepia.ceph.com:47484 (ESTABLISHED) 2022-04-03T22:52:48.091 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 76u IPv4 138673 0t0 TCP smithi077.front.sepia.ceph.com:6822->smithi077.front.sepia.ceph.com:58854 (ESTABLISHED) 2022-04-03T22:52:48.091 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 77u IPv4 130755 0t0 TCP smithi077.front.sepia.ceph.com:6822->smithi186.front.sepia.ceph.com:46558 (CLOSE_WAIT) 2022-04-03T22:52:48.091 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 78u IPv4 137653 0t0 TCP smithi077.front.sepia.ceph.com:40378->smithi186.front.sepia.ceph.com:6825 (CLOSE_WAIT) 2022-04-03T22:52:48.092 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 79u IPv4 136101 0t0 TCP smithi077.front.sepia.ceph.com:50994->smithi077.front.sepia.ceph.com:6830 (ESTABLISHED) 2022-04-03T22:52:48.092 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 80u IPv4 138676 0t0 TCP smithi077.front.sepia.ceph.com:6820->smithi077.front.sepia.ceph.com:47512 (ESTABLISHED) 2022-04-03T22:52:48.092 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 81u IPv4 117362 0t0 TCP smithi077.front.sepia.ceph.com:6822->smithi077.front.sepia.ceph.com:58832 (ESTABLISHED) 2022-04-03T22:52:48.093 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 82u IPv4 130752 0t0 TCP smithi077.front.sepia.ceph.com:6820->smithi186.front.sepia.ceph.com:44432 (CLOSE_WAIT) 2022-04-03T22:52:48.093 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 83u IPv4 117442 0t0 TCP smithi077.front.sepia.ceph.com:6820->smithi186.front.sepia.ceph.com:43846 (CLOSE_WAIT) 2022-04-03T22:52:48.093 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 84u IPv4 139732 0t0 TCP smithi077.front.sepia.ceph.com:6820->smithi186.front.sepia.ceph.com:44738 (CLOSE_WAIT) 2022-04-03T22:52:48.094 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 85u IPv4 139735 0t0 TCP smithi077.front.sepia.ceph.com:6822->smithi186.front.sepia.ceph.com:46868 (CLOSE_WAIT) 2022-04-03T22:52:48.094 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 86u IPv4 117445 0t0 TCP smithi077.front.sepia.ceph.com:6822->smithi186.front.sepia.ceph.com:45974 (CLOSE_WAIT) 2022-04-03T22:52:48.094 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 87u IPv4 135913 0t0 TCP smithi077.front.sepia.ceph.com:34788->smithi186.front.sepia.ceph.com:6812 (CLOSE_WAIT) 2022-04-03T22:52:48.095 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 88u IPv4 136447 0t0 TCP smithi077.front.sepia.ceph.com:6822->smithi077.front.sepia.ceph.com:58850 (ESTABLISHED) 2022-04-03T22:52:48.095 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 89u IPv4 136450 0t0 TCP smithi077.front.sepia.ceph.com:6820->smithi077.front.sepia.ceph.com:47506 (ESTABLISHED) 2022-04-03T22:52:48.095 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 90u IPv4 139610 0t0 TCP smithi077.front.sepia.ceph.com:56564->smithi186.front.sepia.ceph.com:6832 (CLOSE_WAIT) 2022-04-03T22:52:48.096 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 91u IPv4 140607 0t0 TCP smithi077.front.sepia.ceph.com:58984->smithi077.front.sepia.ceph.com:6826 (ESTABLISHED) 2022-04-03T22:52:48.096 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 92u IPv4 138699 0t0 TCP smithi077.front.sepia.ceph.com:40548->smithi186.front.sepia.ceph.com:6818 (CLOSE_WAIT) 2022-04-03T22:52:48.096 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 93u IPv4 82892 0t0 TCP smithi077.front.sepia.ceph.com:6811->smithi186.front.sepia.ceph.com:55342 (CLOSE_WAIT) 2022-04-03T22:52:48.097 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 94u IPv4 132729 0t0 TCP smithi077.front.sepia.ceph.com:53754->smithi186.front.sepia.ceph.com:6824 (CLOSE_WAIT) 2022-04-03T22:52:48.097 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 95u IPv4 139341 0t0 TCP smithi077.front.sepia.ceph.com:36260->smithi186.front.sepia.ceph.com:6810 (CLOSE_WAIT) 2022-04-03T22:52:48.097 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 96u IPv4 138555 0t0 TCP smithi077.front.sepia.ceph.com:53516->smithi077.front.sepia.ceph.com:6802 (ESTABLISHED) 2022-04-03T22:52:48.098 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 97u IPv4 140447 0t0 TCP smithi077.front.sepia.ceph.com:43404->smithi186.front.sepia.ceph.com:6802 (CLOSE_WAIT) 2022-04-03T22:52:48.098 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 100u IPv4 139593 0t0 TCP smithi077.front.sepia.ceph.com:6816->smithi077.front.sepia.ceph.com:55706 (ESTABLISHED) 2022-04-03T22:52:48.098 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20250 20255 msgr-work root 101u IPv4 139608 0t0 TCP smithi077.front.sepia.ceph.com:6814->192.168.0.1:60014 (ESTABLISHED) 2022-04-03T22:52:48.099 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.099 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.099 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root txt unknown /proc/20252/task/22756/exe 2022-04-03T22:52:48.100 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.100 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 1w FIFO 0,12 0t0 80530 pipe 2022-04-03T22:52:48.100 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 2w FIFO 0,12 0t0 80531 pipe 2022-04-03T22:52:48.101 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 3r FIFO 0,12 0t0 80620 pipe 2022-04-03T22:52:48.101 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 4w FIFO 0,12 0t0 80620 pipe 2022-04-03T22:52:48.101 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.102 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 6r FIFO 0,12 0t0 80621 pipe 2022-04-03T22:52:48.102 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 7w FIFO 0,12 0t0 80621 pipe 2022-04-03T22:52:48.102 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.103 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 9r FIFO 0,12 0t0 80622 pipe 2022-04-03T22:52:48.103 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 10w FIFO 0,12 0t0 80622 pipe 2022-04-03T22:52:48.103 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.103 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 12r FIFO 0,12 0t0 80623 pipe 2022-04-03T22:52:48.104 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 13w FIFO 0,12 0t0 80623 pipe 2022-04-03T22:52:48.104 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 14w REG 253,4 2218518 141 /var/lib/ceph/crash/posted/2022-04-03T20:00:14.212328Z_4b1adc59-a97f-49e8-ba34-7aa35df50774/log 2022-04-03T22:52:48.104 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 15w REG 253,4 0 140 /var/lib/ceph/crash/posted/2022-04-03T20:00:14.212328Z_4b1adc59-a97f-49e8-ba34-7aa35df50774/done 2022-04-03T22:52:48.105 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 16r FIFO 0,12 0t0 80624 pipe 2022-04-03T22:52:48.105 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 17w FIFO 0,12 0t0 80624 pipe 2022-04-03T22:52:48.105 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 18u unix 0xffff88810907e900 0t0 80625 /var/run/ceph/ceph-osd.0.asok type=STREAM 2022-04-03T22:52:48.106 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 19u IPv4 85019 0t0 TCP *:6823 (LISTEN) 2022-04-03T22:52:48.106 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 20u IPv4 85045 0t0 TCP *:6825 (LISTEN) 2022-04-03T22:52:48.106 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 21u IPv4 85072 0t0 TCP *:6826 (LISTEN) 2022-04-03T22:52:48.107 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 22u IPv4 85100 0t0 TCP *:6827 (LISTEN) 2022-04-03T22:52:48.107 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 23u IPv4 85129 0t0 TCP *:6828 (LISTEN) 2022-04-03T22:52:48.107 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 24u IPv4 85159 0t0 TCP *:6829 (LISTEN) 2022-04-03T22:52:48.108 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 25u IPv4 85190 0t0 TCP *:6830 (LISTEN) 2022-04-03T22:52:48.108 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 26u IPv4 85222 0t0 TCP *:ambit-lm (LISTEN) 2022-04-03T22:52:48.108 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 27r FIFO 0,12 0t0 82712 pipe 2022-04-03T22:52:48.109 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 28w FIFO 0,12 0t0 82712 pipe 2022-04-03T22:52:48.109 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 29r FIFO 0,12 0t0 82713 pipe 2022-04-03T22:52:48.109 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 30w FIFO 0,12 0t0 82713 pipe 2022-04-03T22:52:48.110 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 31r DIR 253,0 278 64 /var/lib/ceph/osd/ceph-0 2022-04-03T22:52:48.110 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 32u REG 253,0 37 69 /var/lib/ceph/osd/ceph-0/fsid 2022-04-03T22:52:48.110 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 33u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.111 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 34u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.111 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 35u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.111 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 36u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.112 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 37u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.112 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 38u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.112 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 39u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.113 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 40u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.113 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 41u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.113 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 42u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.113 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 43u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.114 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 44u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.114 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 45u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.114 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 46u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.115 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 47u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.115 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 48u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.115 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 49u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.116 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 50u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.116 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 51u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.116 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 52u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.117 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 53u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.117 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 54u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.117 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 55u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.118 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 56u REG 253,0 96636764160 70 /var/lib/ceph/osd/ceph-0/block 2022-04-03T22:52:48.118 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 57r FIFO 0,12 0t0 85463 pipe 2022-04-03T22:52:48.118 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 58u IPv4 87055 0t0 TCP smithi077.front.sepia.ceph.com:42418->smithi077.front.sepia.ceph.com:3300 (CLOSE_WAIT) 2022-04-03T22:52:48.119 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 59w FIFO 0,12 0t0 85463 pipe 2022-04-03T22:52:48.119 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 60r FIFO 0,12 0t0 85464 pipe 2022-04-03T22:52:48.119 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 61w FIFO 0,12 0t0 85464 pipe 2022-04-03T22:52:48.120 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 62u IPv4 129600 0t0 TCP smithi077.front.sepia.ceph.com:34776->smithi077.front.sepia.ceph.com:pentbox-sim (ESTABLISHED) 2022-04-03T22:52:48.120 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 63u IPv4 128937 0t0 TCP smithi077.front.sepia.ceph.com:55470->smithi077.front.sepia.ceph.com:6812 (ESTABLISHED) 2022-04-03T22:52:48.120 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 64u IPv4 106178 0t0 TCP smithi077.front.sepia.ceph.com:34394->smithi186.front.sepia.ceph.com:6814 (CLOSE_WAIT) 2022-04-03T22:52:48.120 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 65u IPv4 140445 0t0 TCP smithi077.front.sepia.ceph.com:57108->smithi186.front.sepia.ceph.com:6821 (CLOSE_WAIT) 2022-04-03T22:52:48.121 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 66u IPv4 136749 0t0 TCP smithi077.front.sepia.ceph.com:47782->smithi077.front.sepia.ceph.com:6806 (ESTABLISHED) 2022-04-03T22:52:48.121 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 67u IPv4 138500 0t0 TCP smithi077.front.sepia.ceph.com:45512->smithi077.front.sepia.ceph.com:6810 (ESTABLISHED) 2022-04-03T22:52:48.121 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 68u IPv4 130941 0t0 TCP smithi077.front.sepia.ceph.com:42314->smithi186.front.sepia.ceph.com:6830 (CLOSE_WAIT) 2022-04-03T22:52:48.122 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 69u IPv4 140616 0t0 TCP smithi077.front.sepia.ceph.com:58854->smithi077.front.sepia.ceph.com:6822 (ESTABLISHED) 2022-04-03T22:52:48.122 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 70u IPv4 130942 0t0 TCP smithi077.front.sepia.ceph.com:33628->smithi186.front.sepia.ceph.com:6828 (CLOSE_WAIT) 2022-04-03T22:52:48.122 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 71u IPv4 136753 0t0 TCP smithi077.front.sepia.ceph.com:6828->smithi186.front.sepia.ceph.com:44496 (CLOSE_WAIT) 2022-04-03T22:52:48.123 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 72u IPv4 124607 0t0 TCP smithi077.front.sepia.ceph.com:6828->smithi186.front.sepia.ceph.com:43970 (CLOSE_WAIT) 2022-04-03T22:52:48.123 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 73u IPv4 136756 0t0 TCP smithi077.front.sepia.ceph.com:6830->smithi186.front.sepia.ceph.com:53098 (CLOSE_WAIT) 2022-04-03T22:52:48.123 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 74u IPv4 124610 0t0 TCP smithi077.front.sepia.ceph.com:6830->smithi186.front.sepia.ceph.com:52572 (CLOSE_WAIT) 2022-04-03T22:52:48.124 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 75u IPv4 105041 0t0 TCP smithi077.front.sepia.ceph.com:34776->smithi186.front.sepia.ceph.com:6812 (CLOSE_WAIT) 2022-04-03T22:52:48.124 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 76u IPv4 131015 0t0 TCP smithi077.front.sepia.ceph.com:40380->smithi186.front.sepia.ceph.com:6825 (CLOSE_WAIT) 2022-04-03T22:52:48.124 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 77u IPv4 141486 0t0 TCP smithi077.front.sepia.ceph.com:47512->smithi077.front.sepia.ceph.com:6820 (ESTABLISHED) 2022-04-03T22:52:48.125 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 78u IPv4 130947 0t0 TCP smithi077.front.sepia.ceph.com:52012->smithi186.front.sepia.ceph.com:6804 (CLOSE_WAIT) 2022-04-03T22:52:48.125 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 79u IPv4 131122 0t0 TCP smithi077.front.sepia.ceph.com:6828->smithi077.front.sepia.ceph.com:49874 (CLOSE_WAIT) 2022-04-03T22:52:48.125 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 80u IPv4 88075 0t0 TCP smithi077.front.sepia.ceph.com:6830->smithi186.front.sepia.ceph.com:51702 (CLOSE_WAIT) 2022-04-03T22:52:48.126 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 81u IPv4 138483 0t0 TCP smithi077.front.sepia.ceph.com:43090->smithi186.front.sepia.ceph.com:6806 (CLOSE_WAIT) 2022-04-03T22:52:48.126 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 82u IPv4 85502 0t0 TCP smithi077.front.sepia.ceph.com:6828->smithi186.front.sepia.ceph.com:43112 (CLOSE_WAIT) 2022-04-03T22:52:48.126 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 83u IPv4 131125 0t0 TCP smithi077.front.sepia.ceph.com:6830->smithi077.front.sepia.ceph.com:50986 (CLOSE_WAIT) 2022-04-03T22:52:48.127 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 84u IPv4 136095 0t0 TCP smithi077.front.sepia.ceph.com:6828->smithi077.front.sepia.ceph.com:49880 (ESTABLISHED) 2022-04-03T22:52:48.127 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 85u IPv4 137654 0t0 TCP smithi077.front.sepia.ceph.com:6830->smithi077.front.sepia.ceph.com:50994 (ESTABLISHED) 2022-04-03T22:52:48.127 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 86u IPv4 124244 0t0 TCP smithi077.front.sepia.ceph.com:6830->smithi186.front.sepia.ceph.com:52508 (CLOSE_WAIT) 2022-04-03T22:52:48.128 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 87u IPv4 124247 0t0 TCP smithi077.front.sepia.ceph.com:6828->smithi186.front.sepia.ceph.com:43910 (CLOSE_WAIT) 2022-04-03T22:52:48.128 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 88u IPv4 142394 0t0 TCP smithi077.front.sepia.ceph.com:6828->smithi077.front.sepia.ceph.com:49882 (ESTABLISHED) 2022-04-03T22:52:48.128 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 89u IPv4 142397 0t0 TCP smithi077.front.sepia.ceph.com:6830->smithi077.front.sepia.ceph.com:50996 (ESTABLISHED) 2022-04-03T22:52:48.129 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 90u IPv4 131058 0t0 TCP smithi077.front.sepia.ceph.com:6826->smithi186.front.sepia.ceph.com:38786 (CLOSE_WAIT) 2022-04-03T22:52:48.129 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 91u IPv4 131065 0t0 TCP smithi077.front.sepia.ceph.com:56566->smithi186.front.sepia.ceph.com:6832 (CLOSE_WAIT) 2022-04-03T22:52:48.129 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 92u IPv4 141411 0t0 TCP smithi077.front.sepia.ceph.com:6826->smithi186.front.sepia.ceph.com:38842 (CLOSE_WAIT) 2022-04-03T22:52:48.130 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 93u IPv4 131072 0t0 TCP smithi077.front.sepia.ceph.com:6826->smithi186.front.sepia.ceph.com:38850 (CLOSE_WAIT) 2022-04-03T22:52:48.130 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 95u IPv4 82814 0t0 TCP smithi077.front.sepia.ceph.com:6823->smithi186.front.sepia.ceph.com:54492 (CLOSE_WAIT) 2022-04-03T22:52:48.130 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 96u IPv4 140608 0t0 TCP smithi077.front.sepia.ceph.com:6826->smithi077.front.sepia.ceph.com:58984 (ESTABLISHED) 2022-04-03T22:52:48.131 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 97u IPv4 82811 0t0 TCP smithi077.front.sepia.ceph.com:6826->smithi077.front.sepia.ceph.com:58914 (CLOSE_WAIT) 2022-04-03T22:52:48.131 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 99u IPv4 141406 0t0 TCP smithi077.front.sepia.ceph.com:6825->192.168.0.1:42326 (ESTABLISHED) 2022-04-03T22:52:48.131 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 100u IPv4 138507 0t0 TCP smithi077.front.sepia.ceph.com:43402->smithi186.front.sepia.ceph.com:6802 (CLOSE_WAIT) 2022-04-03T22:52:48.132 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-osd 20252 22756 bstore_ai root 101u IPv4 136775 0t0 TCP smithi077.front.sepia.ceph.com:6826->smithi077.front.sepia.ceph.com:58982 (ESTABLISHED) 2022-04-03T22:52:48.132 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/3 23724 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:48.132 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/3 23724 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.133 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/3 23724 root txt unknown /proc/23724/exe 2022-04-03T22:52:48.133 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 24172 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:48.133 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 24172 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.133 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 24172 root txt unknown /proc/24172/exe 2022-04-03T22:52:48.134 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.134 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.135 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root txt REG 8,1 165640 12133 /usr/bin/sudo 2022-04-03T22:52:48.135 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 15928 23305 /usr/lib64/security/pam_succeed_if.so 2022-04-03T22:52:48.135 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 16536 9706 /usr/lib64/libpam_misc.so.0.82.1 2022-04-03T22:52:48.135 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 915232 23524 /usr/lib64/security/pam_systemd.so 2022-04-03T22:52:48.136 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 24048 23285 /usr/lib64/security/pam_limits.so 2022-04-03T22:52:48.136 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 11736 23283 /usr/lib64/security/pam_keyinit.so 2022-04-03T22:52:48.136 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 46232 9162 /usr/lib64/libcrack.so.2.9.0 2022-04-03T22:52:48.137 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 24744 9666 /usr/lib64/libpwquality.so.1.0.2 2022-04-03T22:52:48.137 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 11720 23217 /usr/lib64/security/pam_pwquality.so 2022-04-03T22:52:48.137 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 7640 23294 /usr/lib64/security/pam_permit.so 2022-04-03T22:52:48.138 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 7552 23271 /usr/lib64/security/pam_deny.so 2022-04-03T22:52:48.138 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 58032 24580 /usr/lib64/security/pam_sss.so 2022-04-03T22:52:48.138 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 371032 9256 /usr/lib64/libtirpc.so.3.0.0 2022-04-03T22:52:48.139 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 116408 9258 /usr/lib64/libnsl.so.2.0.0 2022-04-03T22:52:48.139 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 58016 23310 /usr/lib64/security/pam_unix.so 2022-04-03T22:52:48.139 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 11744 23287 /usr/lib64/security/pam_localuser.so 2022-04-03T22:52:48.139 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 11816 23316 /usr/lib64/security/pam_usertype.so 2022-04-03T22:52:48.140 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 11720 23275 /usr/lib64/security/pam_faildelay.so 2022-04-03T22:52:48.140 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 20160 23273 /usr/lib64/security/pam_env.so 2022-04-03T22:52:48.140 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:48.141 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.141 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.141 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.141 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.142 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:48.142 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.142 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:48.143 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:48.143 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:48.143 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:48.144 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:48.144 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:48.144 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:48.145 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:48.145 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:48.145 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:48.146 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.146 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:48.146 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 125328 8879 /usr/lib64/libsasl2.so.3.0.0 2022-04-03T22:52:48.147 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.147 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.147 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 67136 8940 /usr/lib64/liblber-2.4.so.2.10.9 2022-04-03T22:52:48.148 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 325184 8942 /usr/lib64/libldap-2.4.so.2.10.9 2022-04-03T22:52:48.148 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:48.148 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 432872 35707 /usr/libexec/sudo/sudoers.so 2022-04-03T22:52:48.149 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:48.149 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:48.149 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:48.150 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.150 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:48.150 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.151 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.151 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.152 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 104424 35704 /usr/libexec/sudo/libsudo_util.so.0.0.0 2022-04-03T22:52:48.152 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:48.152 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.152 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:48.153 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.153 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:48.153 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:48.154 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:48.154 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:48.154 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:48.155 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:48.155 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:48.155 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:48.156 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:48.156 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:48.156 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:48.157 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:48.157 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root 0r FIFO 0,12 0t0 79823 pipe 2022-04-03T22:52:48.157 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root 1w FIFO 0,12 0t0 79824 pipe 2022-04-03T22:52:48.158 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root 2w FIFO 0,12 0t0 79825 pipe 2022-04-03T22:52:48.158 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root 3r REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:48.158 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root 4u unix 0xffff888170007800 0t0 88308 type=STREAM 2022-04-03T22:52:48.158 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root 5u netlink 0t0 88316 AUDIT 2022-04-03T22:52:48.159 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root 6r REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:48.159 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root 7r REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:48.159 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root 9u unix 0xffff888170004380 0t0 88319 type=DGRAM 2022-04-03T22:52:48.160 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root 10r FIFO 0,12 0t0 89113 pipe 2022-04-03T22:52:48.160 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25180 root 11w FIFO 0,12 0t0 89113 pipe 2022-04-03T22:52:48.160 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.161 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.161 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root txt REG 8,1 165640 12133 /usr/bin/sudo 2022-04-03T22:52:48.161 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 15928 23305 /usr/lib64/security/pam_succeed_if.so 2022-04-03T22:52:48.162 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 16536 9706 /usr/lib64/libpam_misc.so.0.82.1 2022-04-03T22:52:48.162 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 915232 23524 /usr/lib64/security/pam_systemd.so 2022-04-03T22:52:48.162 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 24048 23285 /usr/lib64/security/pam_limits.so 2022-04-03T22:52:48.162 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 11736 23283 /usr/lib64/security/pam_keyinit.so 2022-04-03T22:52:48.163 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 46232 9162 /usr/lib64/libcrack.so.2.9.0 2022-04-03T22:52:48.163 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 24744 9666 /usr/lib64/libpwquality.so.1.0.2 2022-04-03T22:52:48.163 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 11720 23217 /usr/lib64/security/pam_pwquality.so 2022-04-03T22:52:48.164 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 7640 23294 /usr/lib64/security/pam_permit.so 2022-04-03T22:52:48.164 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 7552 23271 /usr/lib64/security/pam_deny.so 2022-04-03T22:52:48.164 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 58032 24580 /usr/lib64/security/pam_sss.so 2022-04-03T22:52:48.165 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 371032 9256 /usr/lib64/libtirpc.so.3.0.0 2022-04-03T22:52:48.165 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 116408 9258 /usr/lib64/libnsl.so.2.0.0 2022-04-03T22:52:48.165 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 58016 23310 /usr/lib64/security/pam_unix.so 2022-04-03T22:52:48.166 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 11744 23287 /usr/lib64/security/pam_localuser.so 2022-04-03T22:52:48.166 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 11816 23316 /usr/lib64/security/pam_usertype.so 2022-04-03T22:52:48.166 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 11720 23275 /usr/lib64/security/pam_faildelay.so 2022-04-03T22:52:48.167 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 20160 23273 /usr/lib64/security/pam_env.so 2022-04-03T22:52:48.167 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:48.167 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.168 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.168 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.168 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.169 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:48.169 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.169 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:48.170 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:48.170 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:48.170 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:48.171 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:48.171 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:48.171 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:48.172 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:48.172 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:48.172 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:48.173 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.173 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:48.173 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 125328 8879 /usr/lib64/libsasl2.so.3.0.0 2022-04-03T22:52:48.174 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.174 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.174 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 67136 8940 /usr/lib64/liblber-2.4.so.2.10.9 2022-04-03T22:52:48.175 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 325184 8942 /usr/lib64/libldap-2.4.so.2.10.9 2022-04-03T22:52:48.175 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:48.175 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 432872 35707 /usr/libexec/sudo/sudoers.so 2022-04-03T22:52:48.175 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:48.176 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:48.176 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:48.176 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.177 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:48.177 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.177 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.178 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.178 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 104424 35704 /usr/libexec/sudo/libsudo_util.so.0.0.0 2022-04-03T22:52:48.178 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:48.179 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.179 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:48.179 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.180 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:48.180 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:48.180 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:48.181 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:48.181 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:48.181 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:48.181 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:48.182 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:48.182 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:48.183 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:48.183 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:48.183 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:48.183 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root 0r FIFO 0,12 0t0 79827 pipe 2022-04-03T22:52:48.184 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root 1w FIFO 0,12 0t0 79828 pipe 2022-04-03T22:52:48.184 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root 2w FIFO 0,12 0t0 79829 pipe 2022-04-03T22:52:48.184 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root 3r REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:48.185 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root 4u unix 0xffff88817054e180 0t0 89106 type=STREAM 2022-04-03T22:52:48.185 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root 5u netlink 0t0 84432 AUDIT 2022-04-03T22:52:48.185 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root 6r REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:48.186 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root 7r REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:48.186 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root 9u unix 0xffff888178e4f800 0t0 85679 type=DGRAM 2022-04-03T22:52:48.186 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root 10r FIFO 0,12 0t0 87399 pipe 2022-04-03T22:52:48.187 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 25182 root 11w FIFO 0,12 0t0 87399 pipe 2022-04-03T22:52:48.187 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.187 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.188 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root txt REG 8,1 11816 8955 /usr/libexec/platform-python3.6 2022-04-03T22:52:48.188 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:48.188 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:48.189 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:48.189 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:48.189 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:48.190 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:48.190 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:48.190 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.191 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.191 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.191 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:48.192 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.192 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.192 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:48.192 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.193 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.193 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:48.193 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:48.194 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:48.194 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:48.194 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:48.195 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:48.195 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:48.195 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:48.196 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:48.196 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:48.196 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:48.197 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:48.197 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root 0r FIFO 0,12 0t0 79823 pipe 2022-04-03T22:52:48.197 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root 1w FIFO 0,12 0t0 79824 pipe 2022-04-03T22:52:48.197 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25242 root 2w FIFO 0,12 0t0 79825 pipe 2022-04-03T22:52:48.198 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.198 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.198 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root txt REG 8,1 11816 8955 /usr/libexec/platform-python3.6 2022-04-03T22:52:48.199 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:48.199 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:48.199 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:48.200 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:48.200 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:48.200 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:48.201 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:48.201 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.201 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.202 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.202 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:48.202 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.203 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.203 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:48.203 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.204 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.204 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:48.204 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:48.205 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:48.205 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:48.205 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:48.206 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:48.206 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:48.206 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:48.207 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:48.207 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:48.207 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:48.207 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:48.208 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root 0r FIFO 0,12 0t0 79827 pipe 2022-04-03T22:52:48.208 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root 1w FIFO 0,12 0t0 79828 pipe 2022-04-03T22:52:48.208 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 25244 root 2w FIFO 0,12 0t0 79829 pipe 2022-04-03T22:52:48.209 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.209 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.209 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.210 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.210 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.211 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.211 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.211 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.211 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.212 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.212 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.212 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.213 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.213 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.213 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.214 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.214 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.214 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.215 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.215 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.215 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.216 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.216 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.216 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.217 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.217 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.218 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.218 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.219 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.219 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.219 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.220 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.220 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.220 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 1w FIFO 0,12 0t0 79828 pipe 2022-04-03T22:52:48.221 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 2w FIFO 0,12 0t0 79829 pipe 2022-04-03T22:52:48.221 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 3r FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.221 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 4w FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.222 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.222 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 6r FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.222 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 7w FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.223 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.223 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 9r FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.223 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 10w FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.224 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.224 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 12r FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.224 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 13w FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.225 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 14w REG 8,1 7863232 790211 /var/log/ceph/ceph-mds.c.log 2022-04-03T22:52:48.225 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 15uW REG 0,24 6 1664 /run/ceph/ceph-mds.c.pid 2022-04-03T22:52:48.225 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 16r FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.226 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 17w FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.226 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 18u unix 0xffff88816f70f800 0t0 145566 /var/run/ceph/ceph-mds.c.asok type=STREAM 2022-04-03T22:52:48.226 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 19u IPv4 146493 0t0 TCP *:6833 (LISTEN) 2022-04-03T22:52:48.227 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 20u IPv4 146529 0t0 TCP *:6835 (LISTEN) 2022-04-03T22:52:48.227 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 21r FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.227 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 22w FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.228 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 23r FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.228 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 24w FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.228 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 root 27u IPv4 195686 0t0 TCP smithi077.front.sepia.ceph.com:43642->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.229 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.229 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.229 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.230 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.230 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.230 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.231 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.231 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.231 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.232 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.232 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.232 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.233 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.233 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.233 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.234 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.234 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.234 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.235 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.235 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.235 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.235 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.236 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.236 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.236 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.237 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.237 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.237 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.238 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.238 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.238 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.239 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.239 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.239 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 1w FIFO 0,12 0t0 79828 pipe 2022-04-03T22:52:48.240 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 2w FIFO 0,12 0t0 79829 pipe 2022-04-03T22:52:48.240 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 3r FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.240 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 4w FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.241 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.241 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 6r FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.241 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 7w FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.242 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.242 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 9r FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.242 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 10w FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.243 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.243 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 12r FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.243 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 13w FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.244 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 14w REG 8,1 7863232 790211 /var/log/ceph/ceph-mds.c.log 2022-04-03T22:52:48.244 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 15uW REG 0,24 6 1664 /run/ceph/ceph-mds.c.pid 2022-04-03T22:52:48.244 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 16r FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.245 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 17w FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.245 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 18u unix 0xffff88816f70f800 0t0 145566 /var/run/ceph/ceph-mds.c.asok type=STREAM 2022-04-03T22:52:48.245 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 19u IPv4 146493 0t0 TCP *:6833 (LISTEN) 2022-04-03T22:52:48.246 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 20u IPv4 146529 0t0 TCP *:6835 (LISTEN) 2022-04-03T22:52:48.246 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 21r FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.246 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 22w FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.247 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 23r FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.247 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 24w FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.247 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39610 log root 27u IPv4 195686 0t0 TCP smithi077.front.sepia.ceph.com:43642->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.248 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.248 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.248 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.249 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.249 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.249 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.249 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.250 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.250 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.251 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.251 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.251 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.252 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.252 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.252 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.252 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.253 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.253 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.253 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.254 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.254 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.254 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.255 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.255 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.255 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.256 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.256 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.256 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.257 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.257 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.257 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.258 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.258 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.258 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 1w FIFO 0,12 0t0 79828 pipe 2022-04-03T22:52:48.259 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 2w FIFO 0,12 0t0 79829 pipe 2022-04-03T22:52:48.259 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 3r FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.259 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 4w FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.260 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.260 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 6r FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.260 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 7w FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.261 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.261 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 9r FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.261 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 10w FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.262 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.262 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 12r FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.262 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 13w FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.263 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 14w REG 8,1 7863232 790211 /var/log/ceph/ceph-mds.c.log 2022-04-03T22:52:48.263 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 15uW REG 0,24 6 1664 /run/ceph/ceph-mds.c.pid 2022-04-03T22:52:48.263 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 16r FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.264 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 17w FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.264 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 18u unix 0xffff88816f70f800 0t0 145566 /var/run/ceph/ceph-mds.c.asok type=STREAM 2022-04-03T22:52:48.264 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 19u IPv4 146493 0t0 TCP *:6833 (LISTEN) 2022-04-03T22:52:48.265 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 20u IPv4 146529 0t0 TCP *:6835 (LISTEN) 2022-04-03T22:52:48.265 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 21r FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.265 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 22w FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.266 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 23r FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.266 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 24w FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.266 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39612 msgr-work root 27u IPv4 195686 0t0 TCP smithi077.front.sepia.ceph.com:43642->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.267 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.267 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.267 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.268 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.268 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.268 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.269 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.269 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.269 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.270 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.270 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.270 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.271 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.271 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.271 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.272 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.272 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.272 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.273 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.273 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.273 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.274 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.274 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.274 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.275 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.275 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.275 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.276 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.276 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.276 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.277 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.277 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.277 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.277 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 1w FIFO 0,12 0t0 79828 pipe 2022-04-03T22:52:48.278 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 2w FIFO 0,12 0t0 79829 pipe 2022-04-03T22:52:48.278 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 3r FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.279 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 4w FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.279 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.279 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 6r FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.280 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 7w FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.280 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.280 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 9r FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.280 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 10w FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.281 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.281 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 12r FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.281 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 13w FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.282 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 14w REG 8,1 7863232 790211 /var/log/ceph/ceph-mds.c.log 2022-04-03T22:52:48.282 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 15uW REG 0,24 6 1664 /run/ceph/ceph-mds.c.pid 2022-04-03T22:52:48.282 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 16r FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.283 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 17w FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.283 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 18u unix 0xffff88816f70f800 0t0 145566 /var/run/ceph/ceph-mds.c.asok type=STREAM 2022-04-03T22:52:48.283 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 19u IPv4 146493 0t0 TCP *:6833 (LISTEN) 2022-04-03T22:52:48.284 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 20u IPv4 146529 0t0 TCP *:6835 (LISTEN) 2022-04-03T22:52:48.284 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 21r FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.284 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 22w FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.285 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 23r FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.285 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 24w FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.285 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39613 msgr-work root 27u IPv4 195686 0t0 TCP smithi077.front.sepia.ceph.com:43642->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.286 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.286 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.286 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.287 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.287 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.287 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.288 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.288 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.288 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.289 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.289 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.289 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.289 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.290 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.290 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.290 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.291 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.291 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.291 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.292 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.292 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.292 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.293 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.293 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.293 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.293 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.294 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.294 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.294 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.295 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.295 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.295 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.296 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.296 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 1w FIFO 0,12 0t0 79828 pipe 2022-04-03T22:52:48.296 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 2w FIFO 0,12 0t0 79829 pipe 2022-04-03T22:52:48.297 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 3r FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.297 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 4w FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.297 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.298 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 6r FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.298 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 7w FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.298 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.299 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 9r FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.299 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 10w FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.299 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.300 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 12r FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.300 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 13w FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.300 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 14w REG 8,1 7863232 790211 /var/log/ceph/ceph-mds.c.log 2022-04-03T22:52:48.301 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 15uW REG 0,24 6 1664 /run/ceph/ceph-mds.c.pid 2022-04-03T22:52:48.301 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 16r FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.301 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 17w FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.302 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 18u unix 0xffff88816f70f800 0t0 145566 /var/run/ceph/ceph-mds.c.asok type=STREAM 2022-04-03T22:52:48.302 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 19u IPv4 146493 0t0 TCP *:6833 (LISTEN) 2022-04-03T22:52:48.302 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 20u IPv4 146529 0t0 TCP *:6835 (LISTEN) 2022-04-03T22:52:48.302 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 21r FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.303 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 22w FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.303 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 23r FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.303 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 24w FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.304 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39614 msgr-work root 27u IPv4 195686 0t0 TCP smithi077.front.sepia.ceph.com:43642->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.304 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.304 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.305 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.305 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.305 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.306 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.306 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.306 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.307 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.307 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.307 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.308 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.308 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.308 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.309 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.309 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.309 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.310 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.310 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.310 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.311 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.311 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.311 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.312 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.312 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.312 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.312 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.313 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.313 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.313 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.314 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.314 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.314 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.315 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 1w FIFO 0,12 0t0 79828 pipe 2022-04-03T22:52:48.315 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 2w FIFO 0,12 0t0 79829 pipe 2022-04-03T22:52:48.315 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 3r FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.316 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 4w FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.316 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.316 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 6r FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.317 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 7w FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.317 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.317 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 9r FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.318 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 10w FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.318 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.318 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 12r FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.318 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 13w FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.319 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 14w REG 8,1 7863232 790211 /var/log/ceph/ceph-mds.c.log 2022-04-03T22:52:48.319 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 15uW REG 0,24 6 1664 /run/ceph/ceph-mds.c.pid 2022-04-03T22:52:48.319 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 16r FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.320 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 17w FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.320 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 18u unix 0xffff88816f70f800 0t0 145566 /var/run/ceph/ceph-mds.c.asok type=STREAM 2022-04-03T22:52:48.320 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 19u IPv4 146493 0t0 TCP *:6833 (LISTEN) 2022-04-03T22:52:48.321 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 20u IPv4 146529 0t0 TCP *:6835 (LISTEN) 2022-04-03T22:52:48.321 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 21r FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.321 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 22w FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.322 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 23r FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.322 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 24w FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.322 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39618 service root 27u IPv4 195686 0t0 TCP smithi077.front.sepia.ceph.com:43642->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.323 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.323 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.323 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.323 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.324 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.324 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.324 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.325 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.325 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.325 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.326 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.326 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.327 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.327 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.327 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.327 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.328 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.328 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.328 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.329 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.329 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.329 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.330 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.330 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.330 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.331 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.331 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.331 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.331 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.332 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.332 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.332 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.333 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.333 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 1w FIFO 0,12 0t0 79828 pipe 2022-04-03T22:52:48.333 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 2w FIFO 0,12 0t0 79829 pipe 2022-04-03T22:52:48.334 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 3r FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.334 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 4w FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.334 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.335 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 6r FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.335 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 7w FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.335 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.336 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 9r FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.336 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 10w FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.336 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.337 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 12r FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.337 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 13w FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.337 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 14w REG 8,1 7863232 790211 /var/log/ceph/ceph-mds.c.log 2022-04-03T22:52:48.338 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 15uW REG 0,24 6 1664 /run/ceph/ceph-mds.c.pid 2022-04-03T22:52:48.338 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 16r FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.338 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 17w FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.338 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 18u unix 0xffff88816f70f800 0t0 145566 /var/run/ceph/ceph-mds.c.asok type=STREAM 2022-04-03T22:52:48.339 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 19u IPv4 146493 0t0 TCP *:6833 (LISTEN) 2022-04-03T22:52:48.339 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 20u IPv4 146529 0t0 TCP *:6835 (LISTEN) 2022-04-03T22:52:48.339 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 21r FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.340 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 22w FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.340 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 23r FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.340 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 24w FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.340 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39619 admin_soc root 27u IPv4 195686 0t0 TCP smithi077.front.sepia.ceph.com:43642->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.341 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.341 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.341 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.342 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.342 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.342 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.342 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.343 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.343 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.343 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.344 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.344 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.344 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.345 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.345 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.345 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.345 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.346 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.346 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.346 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.347 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.347 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.347 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.348 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.348 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.348 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.349 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.349 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.349 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.349 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.350 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.350 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.350 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.351 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 1w FIFO 0,12 0t0 79828 pipe 2022-04-03T22:52:48.351 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 2w FIFO 0,12 0t0 79829 pipe 2022-04-03T22:52:48.351 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 3r FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.352 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 4w FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.352 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.352 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 6r FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.352 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 7w FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.353 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.353 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 9r FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.353 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 10w FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.354 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.354 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 12r FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.354 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 13w FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.355 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 14w REG 8,1 7863232 790211 /var/log/ceph/ceph-mds.c.log 2022-04-03T22:52:48.355 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 15uW REG 0,24 6 1664 /run/ceph/ceph-mds.c.pid 2022-04-03T22:52:48.355 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 16r FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.356 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 17w FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.356 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 18u unix 0xffff88816f70f800 0t0 145566 /var/run/ceph/ceph-mds.c.asok type=STREAM 2022-04-03T22:52:48.356 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 19u IPv4 146493 0t0 TCP *:6833 (LISTEN) 2022-04-03T22:52:48.357 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 20u IPv4 146529 0t0 TCP *:6835 (LISTEN) 2022-04-03T22:52:48.357 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 21r FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.357 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 22w FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.358 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 23r FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.358 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 24w FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.358 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39620 signal_ha root 27u IPv4 195686 0t0 TCP smithi077.front.sepia.ceph.com:43642->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.358 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.359 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.359 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.359 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.360 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.360 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.360 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.361 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.361 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.361 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.361 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.362 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.362 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.362 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.363 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.363 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.363 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.364 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.364 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.364 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.365 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.365 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.365 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.366 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.366 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.366 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.366 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.367 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.367 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.367 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.368 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.368 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.368 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.368 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 1w FIFO 0,12 0t0 79828 pipe 2022-04-03T22:52:48.369 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 2w FIFO 0,12 0t0 79829 pipe 2022-04-03T22:52:48.369 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 3r FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.369 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 4w FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.370 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.370 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 6r FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.370 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 7w FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.371 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.371 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 9r FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.371 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 10w FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.371 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.372 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 12r FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.372 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 13w FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.372 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 14w REG 8,1 7863232 790211 /var/log/ceph/ceph-mds.c.log 2022-04-03T22:52:48.373 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 15uW REG 0,24 6 1664 /run/ceph/ceph-mds.c.pid 2022-04-03T22:52:48.373 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 16r FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.373 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 17w FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.374 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 18u unix 0xffff88816f70f800 0t0 145566 /var/run/ceph/ceph-mds.c.asok type=STREAM 2022-04-03T22:52:48.374 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 19u IPv4 146493 0t0 TCP *:6833 (LISTEN) 2022-04-03T22:52:48.374 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 20u IPv4 146529 0t0 TCP *:6835 (LISTEN) 2022-04-03T22:52:48.374 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 21r FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.375 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 22w FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.375 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 23r FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.375 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 24w FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.376 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39621 io_contex root 27u IPv4 195686 0t0 TCP smithi077.front.sepia.ceph.com:43642->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.376 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.376 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.376 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.377 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.377 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.378 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.378 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.378 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.378 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.379 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.379 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.380 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.380 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.380 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.381 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.381 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.381 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.382 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.382 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.382 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.383 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.383 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.383 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.384 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.384 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.384 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.385 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.385 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.385 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.386 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.386 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.386 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.387 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.387 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 1w FIFO 0,12 0t0 79828 pipe 2022-04-03T22:52:48.387 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 2w FIFO 0,12 0t0 79829 pipe 2022-04-03T22:52:48.388 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 3r FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.388 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 4w FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.388 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.389 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 6r FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.389 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 7w FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.389 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.389 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 9r FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.390 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 10w FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.390 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.390 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 12r FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.391 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 13w FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.391 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 14w REG 8,1 7863232 790211 /var/log/ceph/ceph-mds.c.log 2022-04-03T22:52:48.391 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 15uW REG 0,24 6 1664 /run/ceph/ceph-mds.c.pid 2022-04-03T22:52:48.392 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 16r FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.392 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 17w FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.392 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 18u unix 0xffff88816f70f800 0t0 145566 /var/run/ceph/ceph-mds.c.asok type=STREAM 2022-04-03T22:52:48.393 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 19u IPv4 146493 0t0 TCP *:6833 (LISTEN) 2022-04-03T22:52:48.393 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 20u IPv4 146529 0t0 TCP *:6835 (LISTEN) 2022-04-03T22:52:48.393 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 21r FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.394 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 22w FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.394 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 23r FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.394 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 24w FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.395 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39622 io_contex root 27u IPv4 195686 0t0 TCP smithi077.front.sepia.ceph.com:43642->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.395 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.395 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.396 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.396 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.396 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.397 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.397 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.397 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.397 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.398 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.398 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.398 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.399 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.399 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.400 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.400 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.400 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.400 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.401 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.401 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.401 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.402 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.402 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.402 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.403 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.403 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.403 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.404 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.404 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.404 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.405 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.405 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.405 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.406 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 1w FIFO 0,12 0t0 79828 pipe 2022-04-03T22:52:48.406 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 2w FIFO 0,12 0t0 79829 pipe 2022-04-03T22:52:48.406 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 3r FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.407 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 4w FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.407 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.407 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 6r FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.408 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 7w FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.408 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.408 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 9r FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.409 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 10w FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.409 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.409 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 12r FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.410 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 13w FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.410 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 14w REG 8,1 7863232 790211 /var/log/ceph/ceph-mds.c.log 2022-04-03T22:52:48.410 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 15uW REG 0,24 6 1664 /run/ceph/ceph-mds.c.pid 2022-04-03T22:52:48.411 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 16r FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.411 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 17w FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.411 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 18u unix 0xffff88816f70f800 0t0 145566 /var/run/ceph/ceph-mds.c.asok type=STREAM 2022-04-03T22:52:48.412 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 19u IPv4 146493 0t0 TCP *:6833 (LISTEN) 2022-04-03T22:52:48.412 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 20u IPv4 146529 0t0 TCP *:6835 (LISTEN) 2022-04-03T22:52:48.412 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 21r FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.413 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 22w FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.413 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 23r FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.413 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 24w FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.414 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39623 ms_dispat root 27u IPv4 195686 0t0 TCP smithi077.front.sepia.ceph.com:43642->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.414 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.415 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.415 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.415 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.416 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.416 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.416 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.417 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.417 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.417 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.418 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.418 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.418 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.419 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.419 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.419 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.420 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.420 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.420 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.421 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.421 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.421 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.421 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.422 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.422 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.423 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.423 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.423 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.424 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.424 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.424 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.425 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.425 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.425 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 1w FIFO 0,12 0t0 79828 pipe 2022-04-03T22:52:48.426 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 2w FIFO 0,12 0t0 79829 pipe 2022-04-03T22:52:48.426 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 3r FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.426 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 4w FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.427 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.427 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 6r FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.427 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 7w FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.428 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.428 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 9r FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.428 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 10w FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.429 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.429 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 12r FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.429 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 13w FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.429 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 14w REG 8,1 7863232 790211 /var/log/ceph/ceph-mds.c.log 2022-04-03T22:52:48.430 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 15uW REG 0,24 6 1664 /run/ceph/ceph-mds.c.pid 2022-04-03T22:52:48.430 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 16r FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.430 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 17w FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.431 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 18u unix 0xffff88816f70f800 0t0 145566 /var/run/ceph/ceph-mds.c.asok type=STREAM 2022-04-03T22:52:48.431 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 19u IPv4 146493 0t0 TCP *:6833 (LISTEN) 2022-04-03T22:52:48.431 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 20u IPv4 146529 0t0 TCP *:6835 (LISTEN) 2022-04-03T22:52:48.432 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 21r FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.432 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 22w FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.432 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 23r FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.433 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 24w FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.433 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39624 ms_local root 27u IPv4 195686 0t0 TCP smithi077.front.sepia.ceph.com:43642->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.433 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.434 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.434 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.434 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.435 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.435 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.435 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.436 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.436 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.436 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.437 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.437 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.437 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.438 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.438 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.438 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.439 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.439 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.439 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.440 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.440 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.440 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.440 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.441 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.441 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.441 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.442 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.442 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.442 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.443 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.443 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.443 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.443 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.444 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 1w FIFO 0,12 0t0 79828 pipe 2022-04-03T22:52:48.444 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 2w FIFO 0,12 0t0 79829 pipe 2022-04-03T22:52:48.444 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 3r FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.445 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 4w FIFO 0,12 0t0 146456 pipe 2022-04-03T22:52:48.445 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.445 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 6r FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.446 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 7w FIFO 0,12 0t0 146457 pipe 2022-04-03T22:52:48.446 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.446 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 9r FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.447 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 10w FIFO 0,12 0t0 146458 pipe 2022-04-03T22:52:48.447 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.447 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 12r FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.447 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 13w FIFO 0,12 0t0 146459 pipe 2022-04-03T22:52:48.448 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 14w REG 8,1 7863232 790211 /var/log/ceph/ceph-mds.c.log 2022-04-03T22:52:48.448 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 15uW REG 0,24 6 1664 /run/ceph/ceph-mds.c.pid 2022-04-03T22:52:48.448 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 16r FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.449 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 17w FIFO 0,12 0t0 145565 pipe 2022-04-03T22:52:48.449 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 18u unix 0xffff88816f70f800 0t0 145566 /var/run/ceph/ceph-mds.c.asok type=STREAM 2022-04-03T22:52:48.449 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 19u IPv4 146493 0t0 TCP *:6833 (LISTEN) 2022-04-03T22:52:48.450 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 20u IPv4 146529 0t0 TCP *:6835 (LISTEN) 2022-04-03T22:52:48.450 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 21r FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.450 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 22w FIFO 0,12 0t0 145568 pipe 2022-04-03T22:52:48.450 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 23r FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.451 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 24w FIFO 0,12 0t0 145569 pipe 2022-04-03T22:52:48.451 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25253 39625 safe_time root 27u IPv4 195686 0t0 TCP smithi077.front.sepia.ceph.com:43642->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.451 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.452 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.452 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.453 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.453 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.453 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.454 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.454 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.454 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.455 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.455 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.455 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.456 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.456 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.456 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.456 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.457 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.457 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.457 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.458 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.458 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.458 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.459 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.459 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.459 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.460 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.460 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.460 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.461 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.461 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.461 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.462 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.462 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.462 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 1w FIFO 0,12 0t0 79824 pipe 2022-04-03T22:52:48.462 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 2w FIFO 0,12 0t0 79825 pipe 2022-04-03T22:52:48.463 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 3r FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.463 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 4w FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.464 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.464 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 6r FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.464 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 7w FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.465 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.465 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 9r FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.465 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 10w FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.466 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.466 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 12r FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.466 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 13w FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.466 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 14w REG 8,1 7938854 790212 /var/log/ceph/ceph-mds.a.log 2022-04-03T22:52:48.467 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 15uW REG 0,24 6 1665 /run/ceph/ceph-mds.a.pid 2022-04-03T22:52:48.467 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 16r FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.467 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 17w FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.468 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 18u unix 0xffff888170000780 0t0 88364 /var/run/ceph/ceph-mds.a.asok type=STREAM 2022-04-03T22:52:48.468 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 19u IPv4 87444 0t0 TCP *:6832 (LISTEN) 2022-04-03T22:52:48.468 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 20u IPv4 87479 0t0 TCP *:6834 (LISTEN) 2022-04-03T22:52:48.469 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 21r FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.469 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 22w FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.469 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 23r FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.470 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 24w FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.470 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 25r FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.470 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 27w FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.471 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 28r FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.471 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 29w FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.471 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 root 30u IPv4 189854 0t0 TCP smithi077.front.sepia.ceph.com:43622->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.472 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.472 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.472 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.473 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.473 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.473 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.474 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.474 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.474 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.475 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.475 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.475 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.476 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.476 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.476 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.477 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.477 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.477 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.478 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.478 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.478 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.479 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.479 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.479 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.480 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.480 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.480 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.480 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.481 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.481 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.481 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.482 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.482 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.482 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 1w FIFO 0,12 0t0 79824 pipe 2022-04-03T22:52:48.483 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 2w FIFO 0,12 0t0 79825 pipe 2022-04-03T22:52:48.483 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 3r FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.483 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 4w FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.484 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.484 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 6r FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.484 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 7w FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.485 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.485 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 9r FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.485 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 10w FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.486 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.486 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 12r FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.486 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 13w FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.487 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 14w REG 8,1 7938854 790212 /var/log/ceph/ceph-mds.a.log 2022-04-03T22:52:48.487 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 15uW REG 0,24 6 1665 /run/ceph/ceph-mds.a.pid 2022-04-03T22:52:48.487 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 16r FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.488 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 17w FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.488 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 18u unix 0xffff888170000780 0t0 88364 /var/run/ceph/ceph-mds.a.asok type=STREAM 2022-04-03T22:52:48.488 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 19u IPv4 87444 0t0 TCP *:6832 (LISTEN) 2022-04-03T22:52:48.489 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 20u IPv4 87479 0t0 TCP *:6834 (LISTEN) 2022-04-03T22:52:48.489 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 21r FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.489 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 22w FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.490 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 23r FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.490 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 24w FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.490 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 25r FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.490 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 27w FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.491 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 28r FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.491 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 29w FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.491 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25257 log root 30u IPv4 189854 0t0 TCP smithi077.front.sepia.ceph.com:43622->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.492 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.492 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.492 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.493 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.493 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.493 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.493 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.494 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.494 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.494 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.495 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.495 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.495 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.496 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.496 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.496 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.497 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.497 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.497 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.497 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.498 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.498 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.498 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.499 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.499 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.499 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.500 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.500 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.500 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.501 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.501 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.501 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.502 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.502 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 1w FIFO 0,12 0t0 79824 pipe 2022-04-03T22:52:48.502 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 2w FIFO 0,12 0t0 79825 pipe 2022-04-03T22:52:48.503 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 3r FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.503 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 4w FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.503 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.504 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 6r FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.504 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 7w FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.504 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.505 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 9r FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.505 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 10w FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.505 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.505 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 12r FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.506 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 13w FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.506 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 14w REG 8,1 7938854 790212 /var/log/ceph/ceph-mds.a.log 2022-04-03T22:52:48.506 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 15uW REG 0,24 6 1665 /run/ceph/ceph-mds.a.pid 2022-04-03T22:52:48.507 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 16r FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.507 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 17w FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.507 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 18u unix 0xffff888170000780 0t0 88364 /var/run/ceph/ceph-mds.a.asok type=STREAM 2022-04-03T22:52:48.508 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 19u IPv4 87444 0t0 TCP *:6832 (LISTEN) 2022-04-03T22:52:48.508 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 20u IPv4 87479 0t0 TCP *:6834 (LISTEN) 2022-04-03T22:52:48.508 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 21r FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.508 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 22w FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.509 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 23r FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.509 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 24w FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.509 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 25r FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.510 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 27w FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.510 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 28r FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.510 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 29w FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.511 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25261 msgr-work root 30u IPv4 189854 0t0 TCP smithi077.front.sepia.ceph.com:43622->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.511 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.511 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.512 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.512 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.512 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.513 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.513 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.513 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.513 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.514 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.514 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.514 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.515 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.515 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.515 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.516 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.516 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.516 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.517 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.517 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.517 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.518 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.518 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.518 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.519 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.519 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.519 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.520 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.520 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.520 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.521 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.521 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.521 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.522 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 1w FIFO 0,12 0t0 79824 pipe 2022-04-03T22:52:48.522 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 2w FIFO 0,12 0t0 79825 pipe 2022-04-03T22:52:48.522 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 3r FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.523 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 4w FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.523 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.523 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 6r FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.523 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 7w FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.524 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.524 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 9r FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.524 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 10w FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.525 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.525 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 12r FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.525 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 13w FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.526 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 14w REG 8,1 7938854 790212 /var/log/ceph/ceph-mds.a.log 2022-04-03T22:52:48.526 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 15uW REG 0,24 6 1665 /run/ceph/ceph-mds.a.pid 2022-04-03T22:52:48.526 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 16r FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.527 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 17w FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.527 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 18u unix 0xffff888170000780 0t0 88364 /var/run/ceph/ceph-mds.a.asok type=STREAM 2022-04-03T22:52:48.527 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 19u IPv4 87444 0t0 TCP *:6832 (LISTEN) 2022-04-03T22:52:48.528 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 20u IPv4 87479 0t0 TCP *:6834 (LISTEN) 2022-04-03T22:52:48.528 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 21r FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.528 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 22w FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.529 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 23r FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.529 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 24w FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.529 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 25r FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.529 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 27w FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.530 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 28r FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.530 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 29w FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.530 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25263 msgr-work root 30u IPv4 189854 0t0 TCP smithi077.front.sepia.ceph.com:43622->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.531 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.531 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.531 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.532 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.532 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.532 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.533 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.533 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.533 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.533 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.534 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.534 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.534 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.535 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.535 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.535 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.536 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.536 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.536 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.537 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.537 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.537 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.537 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.538 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.538 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.538 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.539 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.539 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.539 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.540 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.540 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.540 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.540 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.541 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 1w FIFO 0,12 0t0 79824 pipe 2022-04-03T22:52:48.541 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 2w FIFO 0,12 0t0 79825 pipe 2022-04-03T22:52:48.541 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 3r FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.542 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 4w FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.542 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.542 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 6r FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.542 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 7w FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.543 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.543 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 9r FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.543 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 10w FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.544 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.544 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 12r FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.544 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 13w FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.545 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 14w REG 8,1 7938854 790212 /var/log/ceph/ceph-mds.a.log 2022-04-03T22:52:48.545 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 15uW REG 0,24 6 1665 /run/ceph/ceph-mds.a.pid 2022-04-03T22:52:48.545 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 16r FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.545 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 17w FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.546 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 18u unix 0xffff888170000780 0t0 88364 /var/run/ceph/ceph-mds.a.asok type=STREAM 2022-04-03T22:52:48.546 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 19u IPv4 87444 0t0 TCP *:6832 (LISTEN) 2022-04-03T22:52:48.546 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 20u IPv4 87479 0t0 TCP *:6834 (LISTEN) 2022-04-03T22:52:48.547 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 21r FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.547 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 22w FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.547 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 23r FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.548 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 24w FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.548 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 25r FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.548 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 27w FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.548 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 28r FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.549 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 29w FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.549 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25265 msgr-work root 30u IPv4 189854 0t0 TCP smithi077.front.sepia.ceph.com:43622->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.549 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.550 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.550 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.550 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.551 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.551 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.551 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.551 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.552 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.552 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.552 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.553 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.553 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.553 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.553 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.554 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.554 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.554 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.555 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.555 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.555 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.556 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.556 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.556 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.557 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.557 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.557 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.557 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.558 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.558 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.558 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.559 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.559 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.559 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 1w FIFO 0,12 0t0 79824 pipe 2022-04-03T22:52:48.560 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 2w FIFO 0,12 0t0 79825 pipe 2022-04-03T22:52:48.560 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 3r FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.560 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 4w FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.561 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.561 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 6r FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.561 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 7w FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.562 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.562 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 9r FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.562 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 10w FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.562 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.563 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 12r FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.563 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 13w FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.563 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 14w REG 8,1 7938854 790212 /var/log/ceph/ceph-mds.a.log 2022-04-03T22:52:48.564 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 15uW REG 0,24 6 1665 /run/ceph/ceph-mds.a.pid 2022-04-03T22:52:48.564 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 16r FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.564 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 17w FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.565 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 18u unix 0xffff888170000780 0t0 88364 /var/run/ceph/ceph-mds.a.asok type=STREAM 2022-04-03T22:52:48.565 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 19u IPv4 87444 0t0 TCP *:6832 (LISTEN) 2022-04-03T22:52:48.565 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 20u IPv4 87479 0t0 TCP *:6834 (LISTEN) 2022-04-03T22:52:48.566 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 21r FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.566 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 22w FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.566 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 23r FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.567 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 24w FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.567 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 25r FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.567 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 27w FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.568 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 28r FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.568 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 29w FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.568 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25273 service root 30u IPv4 189854 0t0 TCP smithi077.front.sepia.ceph.com:43622->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.568 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.569 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.569 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.569 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.570 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.570 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.570 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.571 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.571 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.571 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.572 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.572 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.572 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.573 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.573 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.573 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.574 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.574 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.574 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.575 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.575 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.575 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.576 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.576 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.576 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.577 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.577 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.577 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.578 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.578 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.578 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.578 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.579 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.579 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 1w FIFO 0,12 0t0 79824 pipe 2022-04-03T22:52:48.579 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 2w FIFO 0,12 0t0 79825 pipe 2022-04-03T22:52:48.580 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 3r FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.580 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 4w FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.580 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.581 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 6r FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.581 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 7w FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.581 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.582 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 9r FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.582 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 10w FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.583 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.583 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 12r FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.583 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 13w FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.583 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 14w REG 8,1 7938854 790212 /var/log/ceph/ceph-mds.a.log 2022-04-03T22:52:48.584 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 15uW REG 0,24 6 1665 /run/ceph/ceph-mds.a.pid 2022-04-03T22:52:48.584 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 16r FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.584 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 17w FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.585 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 18u unix 0xffff888170000780 0t0 88364 /var/run/ceph/ceph-mds.a.asok type=STREAM 2022-04-03T22:52:48.585 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 19u IPv4 87444 0t0 TCP *:6832 (LISTEN) 2022-04-03T22:52:48.585 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 20u IPv4 87479 0t0 TCP *:6834 (LISTEN) 2022-04-03T22:52:48.586 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 21r FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.586 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 22w FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.586 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 23r FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.587 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 24w FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.587 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 25r FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.587 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 27w FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.588 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 28r FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.588 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 29w FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.588 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25274 admin_soc root 30u IPv4 189854 0t0 TCP smithi077.front.sepia.ceph.com:43622->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.589 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.589 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.589 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.590 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.590 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.590 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.591 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.591 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.591 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.592 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.592 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.592 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.592 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.593 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.593 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.593 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.594 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.594 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.594 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.595 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.595 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.595 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.596 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.596 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.596 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.597 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.597 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.597 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.598 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.598 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.598 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.599 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.599 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.599 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 1w FIFO 0,12 0t0 79824 pipe 2022-04-03T22:52:48.600 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 2w FIFO 0,12 0t0 79825 pipe 2022-04-03T22:52:48.600 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 3r FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.600 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 4w FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.600 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.601 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 6r FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.601 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 7w FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.601 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.602 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 9r FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.602 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 10w FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.602 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.603 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 12r FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.603 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 13w FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.603 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 14w REG 8,1 7938854 790212 /var/log/ceph/ceph-mds.a.log 2022-04-03T22:52:48.603 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 15uW REG 0,24 6 1665 /run/ceph/ceph-mds.a.pid 2022-04-03T22:52:48.604 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 16r FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.604 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 17w FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.604 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 18u unix 0xffff888170000780 0t0 88364 /var/run/ceph/ceph-mds.a.asok type=STREAM 2022-04-03T22:52:48.605 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 19u IPv4 87444 0t0 TCP *:6832 (LISTEN) 2022-04-03T22:52:48.605 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 20u IPv4 87479 0t0 TCP *:6834 (LISTEN) 2022-04-03T22:52:48.605 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 21r FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.606 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 22w FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.606 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 23r FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.606 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 24w FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.607 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 25r FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.607 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 27w FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.607 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 28r FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.608 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 29w FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.608 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25276 signal_ha root 30u IPv4 189854 0t0 TCP smithi077.front.sepia.ceph.com:43622->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.608 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.609 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.609 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.609 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.609 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.610 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.610 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.610 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.611 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.611 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.611 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.612 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.612 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.612 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.613 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.613 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.613 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.614 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.614 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.614 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.615 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.615 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.615 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.616 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.616 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.616 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.617 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.617 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.617 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.618 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.618 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.618 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.619 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.619 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 1w FIFO 0,12 0t0 79824 pipe 2022-04-03T22:52:48.619 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 2w FIFO 0,12 0t0 79825 pipe 2022-04-03T22:52:48.620 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 3r FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.620 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 4w FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.620 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.621 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 6r FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.621 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 7w FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.621 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.621 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 9r FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.622 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 10w FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.622 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.622 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 12r FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.623 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 13w FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.623 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 14w REG 8,1 7938854 790212 /var/log/ceph/ceph-mds.a.log 2022-04-03T22:52:48.623 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 15uW REG 0,24 6 1665 /run/ceph/ceph-mds.a.pid 2022-04-03T22:52:48.624 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 16r FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.624 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 17w FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.624 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 18u unix 0xffff888170000780 0t0 88364 /var/run/ceph/ceph-mds.a.asok type=STREAM 2022-04-03T22:52:48.625 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 19u IPv4 87444 0t0 TCP *:6832 (LISTEN) 2022-04-03T22:52:48.625 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 20u IPv4 87479 0t0 TCP *:6834 (LISTEN) 2022-04-03T22:52:48.625 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 21r FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.626 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 22w FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.626 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 23r FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.626 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 24w FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.627 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 25r FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.627 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 27w FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.627 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 28r FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.628 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 29w FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.628 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25277 io_contex root 30u IPv4 189854 0t0 TCP smithi077.front.sepia.ceph.com:43622->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.628 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.629 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.629 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.629 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.630 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.630 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.630 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.631 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.631 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.631 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.632 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.632 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.632 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.633 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.633 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.633 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.634 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.634 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.634 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.635 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.635 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.635 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.636 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.636 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.636 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.637 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.637 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.637 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.638 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.638 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.638 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.639 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.639 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.639 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 1w FIFO 0,12 0t0 79824 pipe 2022-04-03T22:52:48.640 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 2w FIFO 0,12 0t0 79825 pipe 2022-04-03T22:52:48.640 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 3r FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.640 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 4w FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.641 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.641 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 6r FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.641 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 7w FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.641 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.642 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 9r FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.642 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 10w FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.642 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.643 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 12r FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.643 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 13w FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.643 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 14w REG 8,1 7938854 790212 /var/log/ceph/ceph-mds.a.log 2022-04-03T22:52:48.644 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 15uW REG 0,24 6 1665 /run/ceph/ceph-mds.a.pid 2022-04-03T22:52:48.644 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 16r FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.644 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 17w FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.645 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 18u unix 0xffff888170000780 0t0 88364 /var/run/ceph/ceph-mds.a.asok type=STREAM 2022-04-03T22:52:48.645 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 19u IPv4 87444 0t0 TCP *:6832 (LISTEN) 2022-04-03T22:52:48.645 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 20u IPv4 87479 0t0 TCP *:6834 (LISTEN) 2022-04-03T22:52:48.646 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 21r FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.646 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 22w FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.646 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 23r FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.647 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 24w FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.647 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 25r FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.647 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 27w FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.648 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 28r FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.648 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 29w FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.648 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25279 io_contex root 30u IPv4 189854 0t0 TCP smithi077.front.sepia.ceph.com:43622->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.649 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.649 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.649 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.650 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.650 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.650 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.650 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.651 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.651 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.652 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.652 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.652 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.653 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.653 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.653 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.653 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.654 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.654 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.654 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.655 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.655 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.655 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.656 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.656 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.656 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.657 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.657 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.657 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.658 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.658 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.658 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.659 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.659 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.659 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 1w FIFO 0,12 0t0 79824 pipe 2022-04-03T22:52:48.659 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 2w FIFO 0,12 0t0 79825 pipe 2022-04-03T22:52:48.660 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 3r FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.660 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 4w FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.660 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.661 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 6r FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.661 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 7w FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.661 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.662 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 9r FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.662 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 10w FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.662 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.663 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 12r FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.663 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 13w FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.663 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 14w REG 8,1 7938854 790212 /var/log/ceph/ceph-mds.a.log 2022-04-03T22:52:48.664 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 15uW REG 0,24 6 1665 /run/ceph/ceph-mds.a.pid 2022-04-03T22:52:48.664 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 16r FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.664 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 17w FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.665 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 18u unix 0xffff888170000780 0t0 88364 /var/run/ceph/ceph-mds.a.asok type=STREAM 2022-04-03T22:52:48.665 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 19u IPv4 87444 0t0 TCP *:6832 (LISTEN) 2022-04-03T22:52:48.665 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 20u IPv4 87479 0t0 TCP *:6834 (LISTEN) 2022-04-03T22:52:48.666 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 21r FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.666 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 22w FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.666 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 23r FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.667 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 24w FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.667 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 25r FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.667 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 27w FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.667 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 28r FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.668 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 29w FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.668 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25282 ms_dispat root 30u IPv4 189854 0t0 TCP smithi077.front.sepia.ceph.com:43622->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.668 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.669 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.669 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.669 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.669 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.670 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.670 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.670 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.671 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.671 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.671 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.672 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.672 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.672 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.673 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.673 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.673 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.673 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.674 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.674 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.675 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.675 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.675 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.675 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.676 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.676 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.676 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.677 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.677 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.677 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.677 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.678 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.678 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.678 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 1w FIFO 0,12 0t0 79824 pipe 2022-04-03T22:52:48.679 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 2w FIFO 0,12 0t0 79825 pipe 2022-04-03T22:52:48.679 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 3r FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.680 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 4w FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.680 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.680 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 6r FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.681 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 7w FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.681 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.681 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 9r FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.682 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 10w FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.682 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.682 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 12r FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.683 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 13w FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.683 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 14w REG 8,1 7938854 790212 /var/log/ceph/ceph-mds.a.log 2022-04-03T22:52:48.683 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 15uW REG 0,24 6 1665 /run/ceph/ceph-mds.a.pid 2022-04-03T22:52:48.684 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 16r FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.684 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 17w FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.684 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 18u unix 0xffff888170000780 0t0 88364 /var/run/ceph/ceph-mds.a.asok type=STREAM 2022-04-03T22:52:48.685 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 19u IPv4 87444 0t0 TCP *:6832 (LISTEN) 2022-04-03T22:52:48.685 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 20u IPv4 87479 0t0 TCP *:6834 (LISTEN) 2022-04-03T22:52:48.685 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 21r FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.686 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 22w FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.686 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 23r FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.686 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 24w FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.687 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 25r FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.687 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 27w FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.687 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 28r FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.687 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 29w FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.688 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25284 ms_local root 30u IPv4 189854 0t0 TCP smithi077.front.sepia.ceph.com:43622->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.688 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.688 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.689 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.689 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.689 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.690 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.690 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.690 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.691 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.691 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.691 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.691 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.692 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.692 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.692 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.693 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.693 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.693 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.694 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.694 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.694 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.695 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.695 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.695 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.696 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.696 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.696 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.697 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.697 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.697 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.698 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.698 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.698 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.699 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 1w FIFO 0,12 0t0 79824 pipe 2022-04-03T22:52:48.699 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 2w FIFO 0,12 0t0 79825 pipe 2022-04-03T22:52:48.699 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 3r FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.700 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 4w FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.700 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.700 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 6r FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.701 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 7w FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.701 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.701 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 9r FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.702 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 10w FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.702 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.702 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 12r FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.703 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 13w FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.703 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 14w REG 8,1 7938854 790212 /var/log/ceph/ceph-mds.a.log 2022-04-03T22:52:48.703 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 15uW REG 0,24 6 1665 /run/ceph/ceph-mds.a.pid 2022-04-03T22:52:48.704 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 16r FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.704 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 17w FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.704 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 18u unix 0xffff888170000780 0t0 88364 /var/run/ceph/ceph-mds.a.asok type=STREAM 2022-04-03T22:52:48.705 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 19u IPv4 87444 0t0 TCP *:6832 (LISTEN) 2022-04-03T22:52:48.705 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 20u IPv4 87479 0t0 TCP *:6834 (LISTEN) 2022-04-03T22:52:48.705 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 21r FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.705 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 22w FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.706 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 23r FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.706 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 24w FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.706 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 25r FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.707 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 27w FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.707 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 28r FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.707 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 29w FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.708 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25286 safe_time root 30u IPv4 189854 0t0 TCP smithi077.front.sepia.ceph.com:43622->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.708 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.708 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.708 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.709 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.709 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.709 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.710 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.710 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.710 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.711 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.711 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.711 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.711 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.712 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.712 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.712 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.713 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.713 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.713 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.714 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.714 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.714 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.714 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.715 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.715 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.715 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.716 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.716 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.716 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.717 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.717 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.717 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.718 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.718 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 1w FIFO 0,12 0t0 79824 pipe 2022-04-03T22:52:48.718 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 2w FIFO 0,12 0t0 79825 pipe 2022-04-03T22:52:48.719 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 3r FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.719 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 4w FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.719 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.720 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 6r FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.720 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 7w FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.720 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.721 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 9r FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.721 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 10w FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.721 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.721 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 12r FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.722 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 13w FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.722 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 14w REG 8,1 7938854 790212 /var/log/ceph/ceph-mds.a.log 2022-04-03T22:52:48.722 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 15uW REG 0,24 6 1665 /run/ceph/ceph-mds.a.pid 2022-04-03T22:52:48.723 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 16r FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.723 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 17w FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.723 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 18u unix 0xffff888170000780 0t0 88364 /var/run/ceph/ceph-mds.a.asok type=STREAM 2022-04-03T22:52:48.724 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 19u IPv4 87444 0t0 TCP *:6832 (LISTEN) 2022-04-03T22:52:48.724 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 20u IPv4 87479 0t0 TCP *:6834 (LISTEN) 2022-04-03T22:52:48.724 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 21r FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.724 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 22w FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.725 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 23r FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.725 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 24w FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.725 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 25r FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.726 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 27w FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.726 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 28r FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.726 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 29w FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.727 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25292 safe_time root 30u IPv4 189854 0t0 TCP smithi077.front.sepia.ceph.com:43622->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.727 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.727 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.728 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.728 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.728 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.728 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.729 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.729 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.729 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.730 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.730 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.730 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.730 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.731 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.731 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.731 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.732 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.732 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.732 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.733 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.733 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.733 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.734 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.734 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.734 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.735 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.735 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.735 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.736 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.736 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.736 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.736 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.737 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.737 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 1w FIFO 0,12 0t0 79824 pipe 2022-04-03T22:52:48.737 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 2w FIFO 0,12 0t0 79825 pipe 2022-04-03T22:52:48.738 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 3r FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.738 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 4w FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.738 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.739 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 6r FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.739 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 7w FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.739 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.740 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 9r FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.740 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 10w FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.740 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.741 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 12r FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.741 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 13w FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.741 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 14w REG 8,1 7938854 790212 /var/log/ceph/ceph-mds.a.log 2022-04-03T22:52:48.741 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 15uW REG 0,24 6 1665 /run/ceph/ceph-mds.a.pid 2022-04-03T22:52:48.742 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 16r FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.742 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 17w FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.742 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 18u unix 0xffff888170000780 0t0 88364 /var/run/ceph/ceph-mds.a.asok type=STREAM 2022-04-03T22:52:48.743 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 19u IPv4 87444 0t0 TCP *:6832 (LISTEN) 2022-04-03T22:52:48.743 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 20u IPv4 87479 0t0 TCP *:6834 (LISTEN) 2022-04-03T22:52:48.743 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 21r FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.744 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 22w FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.744 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 23r FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.744 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 24w FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.745 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 25r FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.745 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 27w FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.745 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 28r FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.746 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 29w FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.746 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25294 ceph-mds root 30u IPv4 189854 0t0 TCP smithi077.front.sepia.ceph.com:43622->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.746 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.747 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.747 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.747 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.748 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.748 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.748 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.749 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.749 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.749 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.749 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.750 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.750 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.750 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.751 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.751 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.751 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.752 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.752 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.752 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.753 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.753 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.753 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.753 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.754 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.754 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.754 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.755 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.755 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.755 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.756 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.756 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.756 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.757 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 1w FIFO 0,12 0t0 79824 pipe 2022-04-03T22:52:48.757 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 2w FIFO 0,12 0t0 79825 pipe 2022-04-03T22:52:48.757 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 3r FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.758 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 4w FIFO 0,12 0t0 89125 pipe 2022-04-03T22:52:48.758 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.758 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 6r FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.759 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 7w FIFO 0,12 0t0 89126 pipe 2022-04-03T22:52:48.759 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.759 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 9r FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.760 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 10w FIFO 0,12 0t0 89127 pipe 2022-04-03T22:52:48.760 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.760 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 12r FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.761 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 13w FIFO 0,12 0t0 89128 pipe 2022-04-03T22:52:48.761 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 14w REG 8,1 7938854 790212 /var/log/ceph/ceph-mds.a.log 2022-04-03T22:52:48.761 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 15uW REG 0,24 6 1665 /run/ceph/ceph-mds.a.pid 2022-04-03T22:52:48.762 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 16r FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.762 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 17w FIFO 0,12 0t0 88363 pipe 2022-04-03T22:52:48.762 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 18u unix 0xffff888170000780 0t0 88364 /var/run/ceph/ceph-mds.a.asok type=STREAM 2022-04-03T22:52:48.763 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 19u IPv4 87444 0t0 TCP *:6832 (LISTEN) 2022-04-03T22:52:48.763 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 20u IPv4 87479 0t0 TCP *:6834 (LISTEN) 2022-04-03T22:52:48.763 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 21r FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.763 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 22w FIFO 0,12 0t0 88365 pipe 2022-04-03T22:52:48.764 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 23r FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.764 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 24w FIFO 0,12 0t0 88366 pipe 2022-04-03T22:52:48.764 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 25r FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.765 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 27w FIFO 0,12 0t0 88367 pipe 2022-04-03T22:52:48.765 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 28r FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.765 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 29w FIFO 0,12 0t0 88368 pipe 2022-04-03T22:52:48.766 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 25254 25397 safe_time root 30u IPv4 189854 0t0 TCP smithi077.front.sepia.ceph.com:43622->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.766 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-msgr 28193 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:48.766 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-msgr 28193 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.767 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-msgr 28193 root txt unknown /proc/28193/exe 2022-04-03T22:52:48.767 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-watc 28203 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:48.767 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-watc 28203 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.767 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-watc 28203 root txt unknown /proc/28203/exe 2022-04-03T22:52:48.768 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-comp 28204 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:48.768 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-comp 28204 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.768 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-comp 28204 root txt unknown /proc/28204/exe 2022-04-03T22:52:48.769 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/0 33320 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:48.769 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/0 33320 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.769 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/0 33320 root txt unknown /proc/33320/exe 2022-04-03T22:52:48.770 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/u 34229 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:48.770 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/u 34229 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.770 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/u 34229 root txt unknown /proc/34229/exe 2022-04-03T22:52:48.771 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 35559 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:48.771 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 35559 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.771 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 35559 root txt unknown /proc/35559/exe 2022-04-03T22:52:48.772 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/3 36525 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:48.772 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/3 36525 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.772 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/3 36525 root txt unknown /proc/36525/exe 2022-04-03T22:52:48.773 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/0 37577 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:48.773 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/0 37577 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.773 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/0 37577 root txt unknown /proc/37577/exe 2022-04-03T22:52:48.773 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.774 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.774 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root txt REG 8,1 165640 12133 /usr/bin/sudo 2022-04-03T22:52:48.774 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 15928 23305 /usr/lib64/security/pam_succeed_if.so 2022-04-03T22:52:48.775 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 16536 9706 /usr/lib64/libpam_misc.so.0.82.1 2022-04-03T22:52:48.775 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 915232 23524 /usr/lib64/security/pam_systemd.so 2022-04-03T22:52:48.775 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 24048 23285 /usr/lib64/security/pam_limits.so 2022-04-03T22:52:48.776 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 11736 23283 /usr/lib64/security/pam_keyinit.so 2022-04-03T22:52:48.776 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 46232 9162 /usr/lib64/libcrack.so.2.9.0 2022-04-03T22:52:48.776 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 24744 9666 /usr/lib64/libpwquality.so.1.0.2 2022-04-03T22:52:48.777 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 11720 23217 /usr/lib64/security/pam_pwquality.so 2022-04-03T22:52:48.777 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 7640 23294 /usr/lib64/security/pam_permit.so 2022-04-03T22:52:48.777 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 7552 23271 /usr/lib64/security/pam_deny.so 2022-04-03T22:52:48.778 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 58032 24580 /usr/lib64/security/pam_sss.so 2022-04-03T22:52:48.778 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 371032 9256 /usr/lib64/libtirpc.so.3.0.0 2022-04-03T22:52:48.778 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 116408 9258 /usr/lib64/libnsl.so.2.0.0 2022-04-03T22:52:48.779 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 58016 23310 /usr/lib64/security/pam_unix.so 2022-04-03T22:52:48.779 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 11744 23287 /usr/lib64/security/pam_localuser.so 2022-04-03T22:52:48.779 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 11816 23316 /usr/lib64/security/pam_usertype.so 2022-04-03T22:52:48.780 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 11720 23275 /usr/lib64/security/pam_faildelay.so 2022-04-03T22:52:48.780 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 20160 23273 /usr/lib64/security/pam_env.so 2022-04-03T22:52:48.780 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:48.781 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.781 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.781 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.782 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.782 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:48.782 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.783 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:48.783 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:48.783 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:48.784 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:48.784 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:48.784 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:48.785 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:48.785 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:48.785 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:48.786 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:48.786 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.786 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:48.787 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 125328 8879 /usr/lib64/libsasl2.so.3.0.0 2022-04-03T22:52:48.787 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.787 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.787 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 67136 8940 /usr/lib64/liblber-2.4.so.2.10.9 2022-04-03T22:52:48.788 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 325184 8942 /usr/lib64/libldap-2.4.so.2.10.9 2022-04-03T22:52:48.788 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:48.788 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 432872 35707 /usr/libexec/sudo/sudoers.so 2022-04-03T22:52:48.789 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:48.789 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:48.789 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:48.790 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.790 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:48.790 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.790 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.791 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.791 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 104424 35704 /usr/libexec/sudo/libsudo_util.so.0.0.0 2022-04-03T22:52:48.791 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:48.792 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.792 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:48.792 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.793 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:48.793 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:48.793 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:48.794 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:48.794 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:48.794 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:48.794 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:48.795 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:48.795 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:48.795 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:48.796 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:48.796 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:48.796 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root 0r FIFO 0,12 0t0 139391 pipe 2022-04-03T22:52:48.797 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root 1w FIFO 0,12 0t0 139392 pipe 2022-04-03T22:52:48.797 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root 2w FIFO 0,12 0t0 139393 pipe 2022-04-03T22:52:48.797 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root 3r REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:48.798 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root 4u unix 0xffff88810eb10f00 0t0 135982 type=STREAM 2022-04-03T22:52:48.798 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root 5u netlink 0t0 137556 AUDIT 2022-04-03T22:52:48.798 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root 6r REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:48.799 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root 7r REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:48.799 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root 9u unix 0xffff888105c23c00 0t0 137559 type=DGRAM 2022-04-03T22:52:48.799 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root 10r FIFO 0,12 0t0 137562 pipe 2022-04-03T22:52:48.800 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 38612 root 11w FIFO 0,12 0t0 137562 pipe 2022-04-03T22:52:48.800 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.800 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.801 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root txt REG 8,1 11816 8955 /usr/libexec/platform-python3.6 2022-04-03T22:52:48.801 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 59352 40788 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:48.801 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 20688 40774 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:48.802 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 56784 40780 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:48.802 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 29256 40796 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:48.802 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 19464 40786 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:48.803 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 27128 40765 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so 2022-04-03T22:52:48.803 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:48.803 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.803 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.804 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.804 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:48.804 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.805 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.805 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 3289568 8891 /usr/lib64/libpython3.6m.so.1.0 2022-04-03T22:52:48.805 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.806 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.806 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:48.806 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:48.807 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:48.807 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:48.807 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:48.807 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:48.808 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:48.808 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:48.808 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:48.809 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:48.809 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:48.809 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:48.810 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root 0r FIFO 0,12 0t0 139391 pipe 2022-04-03T22:52:48.810 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root 1w FIFO 0,12 0t0 139392 pipe 2022-04-03T22:52:48.810 INFO:teuthology.orchestra.run.smithi077.stdout:daemon-he 38644 root 2w FIFO 0,12 0t0 139393 pipe 2022-04-03T22:52:48.811 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.811 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.811 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.812 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.812 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.812 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.812 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.813 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.813 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.813 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.814 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.814 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.814 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.815 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.815 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.815 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.816 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.816 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.816 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.817 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.817 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.817 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.818 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.818 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.818 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.819 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.819 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.819 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.819 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.820 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.820 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.820 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.821 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.821 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 1w FIFO 0,12 0t0 139392 pipe 2022-04-03T22:52:48.821 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 2w FIFO 0,12 0t0 139393 pipe 2022-04-03T22:52:48.822 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 3r FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:48.822 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 4w FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:48.822 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.823 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 6r FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:48.823 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 7w FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:48.823 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.824 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 9r FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:48.824 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 10w FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:48.824 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.825 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 12r FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:48.825 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 13w FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:48.825 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 14w REG 8,1 3749642861 790213 /var/log/ceph/ceph-mds.e.log 2022-04-03T22:52:48.826 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 15uW REG 0,24 6 1668 /run/ceph/ceph-mds.e.pid 2022-04-03T22:52:48.826 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 16r FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:48.826 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 17w FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:48.827 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 18u unix 0xffff888109c56180 0t0 160294 /var/run/ceph/ceph-mds.e.asok type=STREAM 2022-04-03T22:52:48.827 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 19u IPv4 156304 0t0 TCP *:6836 (LISTEN) 2022-04-03T22:52:48.827 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 20u IPv4 156342 0t0 TCP *:6837 (LISTEN) 2022-04-03T22:52:48.828 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 21r FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:48.828 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 22w FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:48.828 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 23r FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:48.829 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 24w FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:48.829 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 root 25u IPv4 184482 0t0 TCP smithi077.front.sepia.ceph.com:43602->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.829 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.830 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.830 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.830 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.831 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.831 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.831 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.832 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.832 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.832 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.833 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.833 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.833 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.834 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.834 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.834 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.835 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.835 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.835 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.836 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.836 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.836 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.837 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.837 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.837 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.838 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.838 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.839 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.839 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.839 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.840 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.840 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.840 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.841 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 1w FIFO 0,12 0t0 139392 pipe 2022-04-03T22:52:48.841 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 2w FIFO 0,12 0t0 139393 pipe 2022-04-03T22:52:48.841 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 3r FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:48.842 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 4w FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:48.842 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.842 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 6r FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:48.843 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 7w FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:48.843 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.843 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 9r FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:48.844 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 10w FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:48.844 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.844 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 12r FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:48.845 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 13w FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:48.845 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 14w REG 8,1 3749642861 790213 /var/log/ceph/ceph-mds.e.log 2022-04-03T22:52:48.845 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 15uW REG 0,24 6 1668 /run/ceph/ceph-mds.e.pid 2022-04-03T22:52:48.846 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 16r FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:48.846 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 17w FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:48.846 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 18u unix 0xffff888109c56180 0t0 160294 /var/run/ceph/ceph-mds.e.asok type=STREAM 2022-04-03T22:52:48.847 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 19u IPv4 156304 0t0 TCP *:6836 (LISTEN) 2022-04-03T22:52:48.847 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 20u IPv4 156342 0t0 TCP *:6837 (LISTEN) 2022-04-03T22:52:48.847 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 21r FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:48.848 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 22w FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:48.848 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 23r FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:48.848 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 24w FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:48.849 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42043 log root 25u IPv4 184482 0t0 TCP smithi077.front.sepia.ceph.com:43602->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.849 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.849 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.850 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.850 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.850 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.851 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.851 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.851 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.852 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.852 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.852 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.853 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.853 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.853 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.853 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.854 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.854 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.855 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.855 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.855 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.856 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.856 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.856 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.857 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.857 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.857 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.857 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.858 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.858 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.859 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.859 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.859 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.860 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.860 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 1w FIFO 0,12 0t0 139392 pipe 2022-04-03T22:52:48.860 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 2w FIFO 0,12 0t0 139393 pipe 2022-04-03T22:52:48.861 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 3r FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:48.861 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 4w FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:48.861 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.861 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 6r FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:48.862 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 7w FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:48.862 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.862 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 9r FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:48.863 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 10w FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:48.863 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.863 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 12r FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:48.864 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 13w FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:48.864 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 14w REG 8,1 3749642861 790213 /var/log/ceph/ceph-mds.e.log 2022-04-03T22:52:48.864 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 15uW REG 0,24 6 1668 /run/ceph/ceph-mds.e.pid 2022-04-03T22:52:48.865 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 16r FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:48.865 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 17w FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:48.865 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 18u unix 0xffff888109c56180 0t0 160294 /var/run/ceph/ceph-mds.e.asok type=STREAM 2022-04-03T22:52:48.866 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 19u IPv4 156304 0t0 TCP *:6836 (LISTEN) 2022-04-03T22:52:48.866 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 20u IPv4 156342 0t0 TCP *:6837 (LISTEN) 2022-04-03T22:52:48.866 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 21r FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:48.867 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 22w FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:48.867 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 23r FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:48.867 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 24w FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:48.868 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42045 msgr-work root 25u IPv4 184482 0t0 TCP smithi077.front.sepia.ceph.com:43602->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.868 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.868 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.869 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.869 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.869 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.870 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.870 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.870 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.871 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.871 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.871 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.872 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.872 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.872 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.873 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.873 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.873 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.874 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.874 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.874 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.875 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.875 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.875 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.876 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.876 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.876 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.877 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.877 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.878 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.878 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.878 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.879 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.879 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.879 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 1w FIFO 0,12 0t0 139392 pipe 2022-04-03T22:52:48.880 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 2w FIFO 0,12 0t0 139393 pipe 2022-04-03T22:52:48.880 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 3r FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:48.880 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 4w FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:48.881 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.881 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 6r FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:48.881 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 7w FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:48.881 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.882 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 9r FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:48.882 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 10w FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:48.883 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.883 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 12r FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:48.883 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 13w FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:48.884 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 14w REG 8,1 3749642861 790213 /var/log/ceph/ceph-mds.e.log 2022-04-03T22:52:48.884 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 15uW REG 0,24 6 1668 /run/ceph/ceph-mds.e.pid 2022-04-03T22:52:48.884 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 16r FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:48.885 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 17w FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:48.885 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 18u unix 0xffff888109c56180 0t0 160294 /var/run/ceph/ceph-mds.e.asok type=STREAM 2022-04-03T22:52:48.885 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 19u IPv4 156304 0t0 TCP *:6836 (LISTEN) 2022-04-03T22:52:48.886 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 20u IPv4 156342 0t0 TCP *:6837 (LISTEN) 2022-04-03T22:52:48.886 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 21r FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:48.886 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 22w FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:48.887 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 23r FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:48.887 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 24w FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:48.887 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42046 msgr-work root 25u IPv4 184482 0t0 TCP smithi077.front.sepia.ceph.com:43602->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.888 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.888 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.888 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.889 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.889 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.889 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.889 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.890 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.890 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.890 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.891 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.891 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.891 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.892 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.892 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.892 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.893 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.893 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.893 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.894 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.894 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.894 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.895 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.895 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.895 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.896 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.896 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.896 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.896 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.897 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.897 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.897 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.898 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.898 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 1w FIFO 0,12 0t0 139392 pipe 2022-04-03T22:52:48.898 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 2w FIFO 0,12 0t0 139393 pipe 2022-04-03T22:52:48.899 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 3r FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:48.899 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 4w FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:48.899 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.900 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 6r FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:48.900 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 7w FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:48.900 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.901 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 9r FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:48.901 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 10w FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:48.901 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.902 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 12r FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:48.902 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 13w FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:48.902 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 14w REG 8,1 3749642861 790213 /var/log/ceph/ceph-mds.e.log 2022-04-03T22:52:48.903 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 15uW REG 0,24 6 1668 /run/ceph/ceph-mds.e.pid 2022-04-03T22:52:48.903 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 16r FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:48.903 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 17w FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:48.904 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 18u unix 0xffff888109c56180 0t0 160294 /var/run/ceph/ceph-mds.e.asok type=STREAM 2022-04-03T22:52:48.904 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 19u IPv4 156304 0t0 TCP *:6836 (LISTEN) 2022-04-03T22:52:48.904 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 20u IPv4 156342 0t0 TCP *:6837 (LISTEN) 2022-04-03T22:52:48.905 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 21r FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:48.905 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 22w FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:48.905 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 23r FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:48.906 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 24w FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:48.906 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42047 msgr-work root 25u IPv4 184482 0t0 TCP smithi077.front.sepia.ceph.com:43602->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.906 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.907 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.907 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.907 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.908 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.908 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.908 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.909 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.909 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.909 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.910 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.910 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.910 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.911 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.911 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.911 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.911 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.912 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.912 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.912 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.913 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.913 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.913 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.914 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.914 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.914 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.915 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.915 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.915 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.916 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.916 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.916 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.917 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.917 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 1w FIFO 0,12 0t0 139392 pipe 2022-04-03T22:52:48.917 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 2w FIFO 0,12 0t0 139393 pipe 2022-04-03T22:52:48.918 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 3r FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:48.918 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 4w FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:48.918 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.919 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 6r FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:48.919 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 7w FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:48.919 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.920 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 9r FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:48.920 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 10w FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:48.920 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.921 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 12r FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:48.921 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 13w FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:48.921 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 14w REG 8,1 3749642861 790213 /var/log/ceph/ceph-mds.e.log 2022-04-03T22:52:48.922 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 15uW REG 0,24 6 1668 /run/ceph/ceph-mds.e.pid 2022-04-03T22:52:48.922 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 16r FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:48.922 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 17w FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:48.923 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 18u unix 0xffff888109c56180 0t0 160294 /var/run/ceph/ceph-mds.e.asok type=STREAM 2022-04-03T22:52:48.923 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 19u IPv4 156304 0t0 TCP *:6836 (LISTEN) 2022-04-03T22:52:48.923 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 20u IPv4 156342 0t0 TCP *:6837 (LISTEN) 2022-04-03T22:52:48.924 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 21r FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:48.924 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 22w FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:48.924 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 23r FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:48.925 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 24w FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:48.925 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42362 service root 25u IPv4 184482 0t0 TCP smithi077.front.sepia.ceph.com:43602->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.925 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.926 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.926 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.926 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.927 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.927 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.927 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.927 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.928 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.928 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.928 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.929 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.929 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.929 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.930 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.930 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.930 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.931 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.931 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.931 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.932 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.932 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.932 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.933 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.933 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.933 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.934 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.934 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.934 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.935 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.935 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.935 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.936 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.936 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 1w FIFO 0,12 0t0 139392 pipe 2022-04-03T22:52:48.936 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 2w FIFO 0,12 0t0 139393 pipe 2022-04-03T22:52:48.937 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 3r FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:48.937 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 4w FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:48.937 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.938 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 6r FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:48.938 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 7w FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:48.938 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.939 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 9r FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:48.939 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 10w FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:48.939 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.940 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 12r FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:48.940 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 13w FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:48.940 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 14w REG 8,1 3749642861 790213 /var/log/ceph/ceph-mds.e.log 2022-04-03T22:52:48.941 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 15uW REG 0,24 6 1668 /run/ceph/ceph-mds.e.pid 2022-04-03T22:52:48.941 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 16r FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:48.941 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 17w FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:48.942 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 18u unix 0xffff888109c56180 0t0 160294 /var/run/ceph/ceph-mds.e.asok type=STREAM 2022-04-03T22:52:48.942 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 19u IPv4 156304 0t0 TCP *:6836 (LISTEN) 2022-04-03T22:52:48.942 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 20u IPv4 156342 0t0 TCP *:6837 (LISTEN) 2022-04-03T22:52:48.943 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 21r FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:48.943 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 22w FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:48.943 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 23r FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:48.944 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 24w FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:48.944 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42363 admin_soc root 25u IPv4 184482 0t0 TCP smithi077.front.sepia.ceph.com:43602->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.944 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.945 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.945 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.945 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.946 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.946 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.946 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.947 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.947 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.948 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.948 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.948 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.949 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.949 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.949 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.950 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.950 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.950 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.951 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.951 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.951 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.951 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.952 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.952 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.952 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.953 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.953 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.953 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.954 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.954 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.954 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.955 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.955 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.955 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 1w FIFO 0,12 0t0 139392 pipe 2022-04-03T22:52:48.956 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 2w FIFO 0,12 0t0 139393 pipe 2022-04-03T22:52:48.956 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 3r FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:48.956 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 4w FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:48.957 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.957 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 6r FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:48.957 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 7w FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:48.958 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.958 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 9r FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:48.958 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 10w FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:48.959 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.959 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 12r FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:48.959 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 13w FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:48.960 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 14w REG 8,1 3749642861 790213 /var/log/ceph/ceph-mds.e.log 2022-04-03T22:52:48.960 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 15uW REG 0,24 6 1668 /run/ceph/ceph-mds.e.pid 2022-04-03T22:52:48.960 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 16r FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:48.961 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 17w FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:48.961 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 18u unix 0xffff888109c56180 0t0 160294 /var/run/ceph/ceph-mds.e.asok type=STREAM 2022-04-03T22:52:48.961 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 19u IPv4 156304 0t0 TCP *:6836 (LISTEN) 2022-04-03T22:52:48.962 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 20u IPv4 156342 0t0 TCP *:6837 (LISTEN) 2022-04-03T22:52:48.962 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 21r FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:48.962 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 22w FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:48.963 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 23r FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:48.963 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 24w FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:48.963 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42364 signal_ha root 25u IPv4 184482 0t0 TCP smithi077.front.sepia.ceph.com:43602->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.964 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.964 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.964 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.965 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.965 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.965 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.966 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.966 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.966 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.967 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.967 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.967 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.968 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.968 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.968 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.969 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.969 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.969 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.970 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.970 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.970 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.971 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.971 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.971 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.972 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.972 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.972 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.973 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.973 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.973 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.974 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.974 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.975 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.975 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 1w FIFO 0,12 0t0 139392 pipe 2022-04-03T22:52:48.975 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 2w FIFO 0,12 0t0 139393 pipe 2022-04-03T22:52:48.976 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 3r FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:48.976 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 4w FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:48.976 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.977 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 6r FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:48.977 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 7w FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:48.977 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.978 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 9r FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:48.978 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 10w FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:48.978 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.979 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 12r FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:48.979 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 13w FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:48.979 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 14w REG 8,1 3749642861 790213 /var/log/ceph/ceph-mds.e.log 2022-04-03T22:52:48.980 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 15uW REG 0,24 6 1668 /run/ceph/ceph-mds.e.pid 2022-04-03T22:52:48.980 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 16r FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:48.980 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 17w FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:48.981 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 18u unix 0xffff888109c56180 0t0 160294 /var/run/ceph/ceph-mds.e.asok type=STREAM 2022-04-03T22:52:48.981 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 19u IPv4 156304 0t0 TCP *:6836 (LISTEN) 2022-04-03T22:52:48.981 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 20u IPv4 156342 0t0 TCP *:6837 (LISTEN) 2022-04-03T22:52:48.982 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 21r FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:48.982 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 22w FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:48.982 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 23r FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:48.983 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 24w FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:48.983 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42365 io_contex root 25u IPv4 184482 0t0 TCP smithi077.front.sepia.ceph.com:43602->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:48.983 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:48.984 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:48.984 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:48.984 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:48.985 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:48.985 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:48.986 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:48.986 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:48.986 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:48.987 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:48.987 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:48.987 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:48.988 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:48.988 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:48.988 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:48.989 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:48.989 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:48.989 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:48.990 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:48.990 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:48.990 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:48.991 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:48.991 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:48.991 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:48.992 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:48.992 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:48.992 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:48.993 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:48.993 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:48.993 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:48.994 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:48.994 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:48.994 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:48.995 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 1w FIFO 0,12 0t0 139392 pipe 2022-04-03T22:52:48.995 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 2w FIFO 0,12 0t0 139393 pipe 2022-04-03T22:52:48.995 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 3r FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:48.996 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 4w FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:48.996 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.996 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 6r FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:48.997 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 7w FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:48.997 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.997 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 9r FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:48.998 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 10w FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:48.998 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:48.998 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 12r FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:48.999 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 13w FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:48.999 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 14w REG 8,1 3749642861 790213 /var/log/ceph/ceph-mds.e.log 2022-04-03T22:52:48.999 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 15uW REG 0,24 6 1668 /run/ceph/ceph-mds.e.pid 2022-04-03T22:52:49.000 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 16r FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:49.000 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 17w FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:49.000 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 18u unix 0xffff888109c56180 0t0 160294 /var/run/ceph/ceph-mds.e.asok type=STREAM 2022-04-03T22:52:49.001 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 19u IPv4 156304 0t0 TCP *:6836 (LISTEN) 2022-04-03T22:52:49.001 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 20u IPv4 156342 0t0 TCP *:6837 (LISTEN) 2022-04-03T22:52:49.001 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 21r FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:49.002 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 22w FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:49.002 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 23r FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:49.002 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 24w FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:49.003 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42366 io_contex root 25u IPv4 184482 0t0 TCP smithi077.front.sepia.ceph.com:43602->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:49.003 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:49.003 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.004 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:49.004 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:49.004 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:49.005 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:49.005 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:49.005 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:49.006 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:49.006 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:49.006 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:49.007 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:49.007 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:49.007 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:49.008 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:49.008 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:49.008 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:49.009 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:49.009 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:49.009 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:49.010 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:49.010 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:49.010 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:49.010 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:49.011 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:49.011 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:49.011 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:49.012 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:49.012 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:49.012 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:49.013 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:49.013 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:49.013 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:49.014 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 1w FIFO 0,12 0t0 139392 pipe 2022-04-03T22:52:49.014 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 2w FIFO 0,12 0t0 139393 pipe 2022-04-03T22:52:49.014 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 3r FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:49.015 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 4w FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:49.015 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:49.015 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 6r FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:49.016 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 7w FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:49.016 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:49.016 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 9r FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:49.017 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 10w FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:49.017 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:49.017 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 12r FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:49.018 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 13w FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:49.018 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 14w REG 8,1 3749642861 790213 /var/log/ceph/ceph-mds.e.log 2022-04-03T22:52:49.018 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 15uW REG 0,24 6 1668 /run/ceph/ceph-mds.e.pid 2022-04-03T22:52:49.019 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 16r FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:49.019 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 17w FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:49.019 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 18u unix 0xffff888109c56180 0t0 160294 /var/run/ceph/ceph-mds.e.asok type=STREAM 2022-04-03T22:52:49.020 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 19u IPv4 156304 0t0 TCP *:6836 (LISTEN) 2022-04-03T22:52:49.020 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 20u IPv4 156342 0t0 TCP *:6837 (LISTEN) 2022-04-03T22:52:49.020 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 21r FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:49.021 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 22w FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:49.021 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 23r FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:49.021 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 24w FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:49.022 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42367 ms_dispat root 25u IPv4 184482 0t0 TCP smithi077.front.sepia.ceph.com:43602->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:49.022 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:49.022 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.023 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:49.023 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:49.023 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:49.024 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:49.024 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:49.024 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:49.025 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:49.025 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:49.025 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:49.026 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:49.026 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:49.026 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:49.027 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:49.027 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:49.027 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:49.028 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:49.028 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:49.028 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:49.029 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:49.029 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:49.029 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:49.030 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:49.030 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:49.030 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:49.031 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:49.031 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:49.031 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:49.032 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:49.032 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:49.032 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:49.033 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:49.033 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 1w FIFO 0,12 0t0 139392 pipe 2022-04-03T22:52:49.033 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 2w FIFO 0,12 0t0 139393 pipe 2022-04-03T22:52:49.034 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 3r FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:49.034 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 4w FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:49.034 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:49.035 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 6r FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:49.035 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 7w FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:49.035 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:49.036 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 9r FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:49.036 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 10w FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:49.036 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:49.037 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 12r FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:49.037 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 13w FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:49.037 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 14w REG 8,1 3749642861 790213 /var/log/ceph/ceph-mds.e.log 2022-04-03T22:52:49.038 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 15uW REG 0,24 6 1668 /run/ceph/ceph-mds.e.pid 2022-04-03T22:52:49.038 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 16r FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:49.038 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 17w FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:49.039 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 18u unix 0xffff888109c56180 0t0 160294 /var/run/ceph/ceph-mds.e.asok type=STREAM 2022-04-03T22:52:49.039 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 19u IPv4 156304 0t0 TCP *:6836 (LISTEN) 2022-04-03T22:52:49.039 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 20u IPv4 156342 0t0 TCP *:6837 (LISTEN) 2022-04-03T22:52:49.040 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 21r FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:49.040 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 22w FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:49.040 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 23r FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:49.041 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 24w FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:49.041 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42368 ms_local root 25u IPv4 184482 0t0 TCP smithi077.front.sepia.ceph.com:43602->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:49.041 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:49.042 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.042 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root txt REG 8,1 8663744 4083 /usr/bin/ceph-mds 2022-04-03T22:52:49.042 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 20592 15565 /usr/lib64/liburcu-common.so.6.0.0 2022-04-03T22:52:49.043 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 41776 15563 /usr/lib64/liburcu-cds.so.6.0.0 2022-04-03T22:52:49.043 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 33776 15561 /usr/lib64/liburcu-bp.so.6.0.0 2022-04-03T22:52:49.043 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 63616 3089 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 2022-04-03T22:52:49.044 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:49.044 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:49.044 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:49.045 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:49.045 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:49.045 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:49.046 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:49.046 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 76752 3904 /usr/lib64/libunwind.so.8.0.1 2022-04-03T22:52:49.046 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:49.047 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:49.047 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:49.047 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 1661376 7433 /usr/lib64/libstdc++.so.6.0.25 2022-04-03T22:52:49.048 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:49.048 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 109600 16114 /usr/lib64/librdmacm.so.1.3.32.0 2022-04-03T22:52:49.048 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:49.049 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:49.049 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:49.049 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:49.050 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:49.050 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:49.050 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 309104 7702 /usr/lib64/liblua-5.3.so 2022-04-03T22:52:49.051 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 12568720 524563 /usr/lib64/ceph/libceph-common.so.2 2022-04-03T22:52:49.051 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 334544 3914 /usr/lib64/libtcmalloc.so.4.5.3 2022-04-03T22:52:49.051 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:49.052 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:49.052 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:49.052 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 1w FIFO 0,12 0t0 139392 pipe 2022-04-03T22:52:49.053 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 2w FIFO 0,12 0t0 139393 pipe 2022-04-03T22:52:49.053 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 3r FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:49.053 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 4w FIFO 0,12 0t0 156111 pipe 2022-04-03T22:52:49.053 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 5u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:49.054 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 6r FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:49.054 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 7w FIFO 0,12 0t0 156112 pipe 2022-04-03T22:52:49.054 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 8u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:49.055 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 9r FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:49.055 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 10w FIFO 0,12 0t0 156113 pipe 2022-04-03T22:52:49.055 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 11u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:49.056 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 12r FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:49.056 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 13w FIFO 0,12 0t0 156114 pipe 2022-04-03T22:52:49.056 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 14w REG 8,1 3749642861 790213 /var/log/ceph/ceph-mds.e.log 2022-04-03T22:52:49.057 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 15uW REG 0,24 6 1668 /run/ceph/ceph-mds.e.pid 2022-04-03T22:52:49.057 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 16r FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:49.057 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 17w FIFO 0,12 0t0 160293 pipe 2022-04-03T22:52:49.058 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 18u unix 0xffff888109c56180 0t0 160294 /var/run/ceph/ceph-mds.e.asok type=STREAM 2022-04-03T22:52:49.058 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 19u IPv4 156304 0t0 TCP *:6836 (LISTEN) 2022-04-03T22:52:49.058 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 20u IPv4 156342 0t0 TCP *:6837 (LISTEN) 2022-04-03T22:52:49.059 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 21r FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:49.059 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 22w FIFO 0,12 0t0 160296 pipe 2022-04-03T22:52:49.059 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 23r FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:49.060 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 24w FIFO 0,12 0t0 160297 pipe 2022-04-03T22:52:49.060 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-mds 38649 42369 safe_time root 25u IPv4 184482 0t0 TCP smithi077.front.sepia.ceph.com:43602->smithi077.front.sepia.ceph.com:3300 (ESTABLISHED) 2022-04-03T22:52:49.060 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu cwd unknown /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph (stat: Stale file handle) 2022-04-03T22:52:49.061 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.061 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu txt REG 8,1 3206088 14899 /usr/bin/git 2022-04-03T22:52:49.061 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:49.061 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu mem REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:49.062 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:49.062 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu mem REG 0,42 1099511633855 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph/.git/objects/pack/pack-b93e938bb423df5084fe2562e9af2436726e221d.pack (stat: Input/output error) 2022-04-03T22:52:49.062 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu mem REG 0,42 1099511633263 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph/.git/objects/pack/pack-b93e938bb423df5084fe2562e9af2436726e221d.idx (stat: Input/output error) 2022-04-03T22:52:49.063 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:49.063 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:49.063 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:49.064 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:49.064 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:49.064 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:49.065 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:49.065 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:49.065 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu mem REG 0,42 1099511633878 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph/.git/packed-refs (stat: Input/output error) 2022-04-03T22:52:49.066 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:49.066 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:49.066 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:49.067 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:49.067 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu 0r FIFO 0,12 0t0 125986 pipe 2022-04-03T22:52:49.067 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu 1w FIFO 0,12 0t0 125987 pipe 2022-04-03T22:52:49.068 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu 2w FIFO 0,12 0t0 125988 pipe 2022-04-03T22:52:49.068 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu 3w unknown /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph/.git/logs/HEAD (stat: Stale file handle) 2022-04-03T22:52:49.068 INFO:teuthology.orchestra.run.smithi077.stdout:git 38794 ubuntu 4r REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:49.069 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/6 38795 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.069 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/6 38795 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.069 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/6 38795 root txt unknown /proc/38795/exe 2022-04-03T22:52:49.070 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 38969 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.070 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 38969 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.070 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 38969 root txt unknown /proc/38969/exe 2022-04-03T22:52:49.071 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:49.071 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.071 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root txt REG 8,1 165640 12133 /usr/bin/sudo 2022-04-03T22:52:49.072 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 15928 23305 /usr/lib64/security/pam_succeed_if.so 2022-04-03T22:52:49.072 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 16536 9706 /usr/lib64/libpam_misc.so.0.82.1 2022-04-03T22:52:49.072 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 915232 23524 /usr/lib64/security/pam_systemd.so 2022-04-03T22:52:49.073 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 24048 23285 /usr/lib64/security/pam_limits.so 2022-04-03T22:52:49.073 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 11736 23283 /usr/lib64/security/pam_keyinit.so 2022-04-03T22:52:49.073 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 46232 9162 /usr/lib64/libcrack.so.2.9.0 2022-04-03T22:52:49.074 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 24744 9666 /usr/lib64/libpwquality.so.1.0.2 2022-04-03T22:52:49.074 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 11720 23217 /usr/lib64/security/pam_pwquality.so 2022-04-03T22:52:49.074 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 7640 23294 /usr/lib64/security/pam_permit.so 2022-04-03T22:52:49.075 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 7552 23271 /usr/lib64/security/pam_deny.so 2022-04-03T22:52:49.075 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 58032 24580 /usr/lib64/security/pam_sss.so 2022-04-03T22:52:49.075 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 371032 9256 /usr/lib64/libtirpc.so.3.0.0 2022-04-03T22:52:49.076 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 116408 9258 /usr/lib64/libnsl.so.2.0.0 2022-04-03T22:52:49.076 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 58016 23310 /usr/lib64/security/pam_unix.so 2022-04-03T22:52:49.076 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 11744 23287 /usr/lib64/security/pam_localuser.so 2022-04-03T22:52:49.077 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 11816 23316 /usr/lib64/security/pam_usertype.so 2022-04-03T22:52:49.077 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 11720 23275 /usr/lib64/security/pam_faildelay.so 2022-04-03T22:52:49.077 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 20160 23273 /usr/lib64/security/pam_env.so 2022-04-03T22:52:49.078 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:49.078 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:49.078 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:49.079 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:49.079 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:49.079 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:49.080 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:49.080 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:49.080 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:49.081 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:49.081 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:49.081 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:49.082 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:49.082 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:49.082 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:49.083 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:49.083 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:49.083 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:49.084 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:49.084 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 125328 8879 /usr/lib64/libsasl2.so.3.0.0 2022-04-03T22:52:49.084 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:49.085 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:49.085 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 67136 8940 /usr/lib64/liblber-2.4.so.2.10.9 2022-04-03T22:52:49.085 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 325184 8942 /usr/lib64/libldap-2.4.so.2.10.9 2022-04-03T22:52:49.086 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:49.086 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 432872 35707 /usr/libexec/sudo/sudoers.so 2022-04-03T22:52:49.086 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:49.087 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:49.087 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:49.087 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:49.088 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:49.088 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:49.088 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:49.089 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:49.089 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 104424 35704 /usr/libexec/sudo/libsudo_util.so.0.0.0 2022-04-03T22:52:49.089 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:49.090 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:49.090 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:49.090 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:49.091 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:49.091 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:49.091 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:49.091 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:49.092 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:49.092 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:49.092 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:49.093 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:49.093 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:49.093 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:49.094 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:49.094 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:49.094 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root 0r FIFO 0,12 0t0 137831 pipe 2022-04-03T22:52:49.095 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root 1w FIFO 0,12 0t0 137832 pipe 2022-04-03T22:52:49.095 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root 2w FIFO 0,12 0t0 137833 pipe 2022-04-03T22:52:49.095 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root 3r REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:49.096 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root 4u unix 0xffff88810da1f800 0t0 142439 type=STREAM 2022-04-03T22:52:49.096 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root 5u netlink 0t0 142447 AUDIT 2022-04-03T22:52:49.096 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root 6r REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:49.097 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root 7r REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:49.097 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root 9u unix 0xffff88810da1ad00 0t0 142450 type=DGRAM 2022-04-03T22:52:49.097 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root 10r FIFO 0,12 0t0 142453 pipe 2022-04-03T22:52:49.098 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 39082 root 11w FIFO 0,12 0t0 142453 pipe 2022-04-03T22:52:49.098 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:49.098 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.099 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root txt REG 8,1 33664 9837 /usr/bin/umount 2022-04-03T22:52:49.099 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:49.099 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:49.100 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:49.100 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:49.100 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:49.101 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:49.101 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:49.101 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:49.102 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:49.102 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:49.102 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:49.103 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:49.103 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:49.103 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:49.104 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:49.104 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:49.104 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:49.105 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:49.105 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:49.105 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:49.106 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:49.106 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:49.106 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:49.107 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root 0r FIFO 0,12 0t0 137831 pipe 2022-04-03T22:52:49.107 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root 1w FIFO 0,12 0t0 137832 pipe 2022-04-03T22:52:49.107 INFO:teuthology.orchestra.run.smithi077.stdout:umount 39098 root 2w FIFO 0,12 0t0 137833 pipe 2022-04-03T22:52:49.108 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/0 39124 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.108 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/0 39124 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.108 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/0 39124 root txt unknown /proc/39124/exe 2022-04-03T22:52:49.108 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/6 39896 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.109 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/6 39896 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.109 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/6 39896 root txt unknown /proc/39896/exe 2022-04-03T22:52:49.109 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 40538 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.110 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 40538 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.110 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 40538 root txt unknown /proc/40538/exe 2022-04-03T22:52:49.110 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 40557 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.111 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 40557 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.111 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 40557 root txt unknown /proc/40557/exe 2022-04-03T22:52:49.111 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/7 40610 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.112 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/7 40610 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.112 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/7 40610 root txt unknown /proc/40610/exe 2022-04-03T22:52:49.112 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 42608 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.113 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 42608 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.113 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 42608 root txt unknown /proc/42608/exe 2022-04-03T22:52:49.113 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 43728 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.114 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 43728 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.114 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 43728 root txt unknown /proc/43728/exe 2022-04-03T22:52:49.114 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/5 45842 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.115 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/5 45842 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.115 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/5 45842 root txt unknown /proc/45842/exe 2022-04-03T22:52:49.115 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/7 46357 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.115 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/7 46357 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.116 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/7 46357 root txt unknown /proc/46357/exe 2022-04-03T22:52:49.116 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/u 46520 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.116 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/u 46520 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.117 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/u 46520 root txt unknown /proc/46520/exe 2022-04-03T22:52:49.117 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/1 46907 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.118 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/1 46907 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.118 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/1 46907 root txt unknown /proc/46907/exe 2022-04-03T22:52:49.118 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/u 47073 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.119 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/u 47073 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.119 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/u 47073 root txt unknown /proc/47073/exe 2022-04-03T22:52:49.119 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.120 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.120 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root txt REG 8,1 1588952 10333 /usr/lib/systemd/systemd 2022-04-03T22:52:49.120 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:49.121 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:49.121 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:49.121 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:49.122 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:49.122 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:49.122 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:49.123 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 312232 8919 /usr/lib64/libpcap.so.1.9.1 2022-04-03T22:52:49.123 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:49.123 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:49.124 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:49.124 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 26192 7582 /usr/lib64/libattr.so.1.1.2448 2022-04-03T22:52:49.124 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:49.125 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:49.125 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:49.125 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:49.125 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:49.126 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:49.126 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:49.127 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:49.127 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:49.127 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:49.128 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:49.128 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 33328 8926 /usr/lib64/libip4tc.so.2.0.0 2022-04-03T22:52:49.128 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:49.128 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:49.129 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 54816 7584 /usr/lib64/libacl.so.1.1.2253 2022-04-03T22:52:49.129 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:49.129 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:49.130 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:49.130 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:49.130 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 100208 9562 /usr/lib64/libkmod.so.2.3.3 2022-04-03T22:52:49.131 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:49.131 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:49.132 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:49.132 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:49.132 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 132888 8225 /usr/lib64/libseccomp.so.2.5.1 2022-04-03T22:52:49.133 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:49.133 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 2732744 10328 /usr/lib/systemd/libsystemd-shared-239.so 2022-04-03T22:52:49.133 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:49.134 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:49.134 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 1u unix 0xffff88810da1bc00 0t0 185776 type=STREAM 2022-04-03T22:52:49.134 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 2u unix 0xffff88810da1bc00 0t0 185776 type=STREAM 2022-04-03T22:52:49.135 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 3u unix 0xffff88810348b480 0t0 186813 type=DGRAM 2022-04-03T22:52:49.135 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 4u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:49.135 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 5u a_inode 0,13 0 9293 [signalfd] 2022-04-03T22:52:49.136 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 6r DIR 0,26 0 1583 /sys/fs/cgroup/systemd/user.slice/user-0.slice/user@0.service 2022-04-03T22:52:49.136 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 7u a_inode 0,13 0 9293 [timerfd] 2022-04-03T22:52:49.136 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 9u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:49.137 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 10r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:49.137 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 11u unix 0xffff888106ed0000 0t0 176707 /run/user/0/systemd/private type=STREAM 2022-04-03T22:52:49.137 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 14r REG 0,21 0 4026532078 /proc/swaps 2022-04-03T22:52:49.138 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 15u unix 0xffff88810348e900 0t0 186823 /run/user/0/systemd/notify type=DGRAM 2022-04-03T22:52:49.138 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 16u unix 0xffff888103489e00 0t0 186824 type=DGRAM 2022-04-03T22:52:49.138 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 17u unix 0xffff88810348a580 0t0 186825 type=DGRAM 2022-04-03T22:52:49.139 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 18u unix 0xffff88810348cb00 0t0 186826 /run/user/0/systemd/private type=STREAM 2022-04-03T22:52:49.139 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 19u unix 0xffff88810348c380 0t0 186827 type=STREAM 2022-04-03T22:52:49.139 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 20u a_inode 0,13 0 9293 [timerfd] 2022-04-03T22:52:49.140 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 21r FIFO 0,12 0t0 186850 pipe 2022-04-03T22:52:49.140 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 22w FIFO 0,12 0t0 186850 pipe 2022-04-03T22:52:49.140 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 23r FIFO 0,12 0t0 186851 pipe 2022-04-03T22:52:49.140 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 24w FIFO 0,12 0t0 186851 pipe 2022-04-03T22:52:49.141 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47303 root 26u netlink 0t0 186836 SELINUX 2022-04-03T22:52:49.141 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/7 47310 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.141 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/7 47310 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.142 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/7 47310 root txt unknown /proc/47310/exe 2022-04-03T22:52:49.142 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 47312 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.142 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 47312 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.143 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 47312 root txt unknown /proc/47312/exe 2022-04-03T22:52:49.143 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/6 47324 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.143 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/6 47324 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.144 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/6 47324 root txt unknown /proc/47324/exe 2022-04-03T22:52:49.144 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/u 47361 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.144 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/u 47361 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.145 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/u 47361 root txt unknown /proc/47361/exe 2022-04-03T22:52:49.145 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 47419 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.145 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 47419 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.146 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/4 47419 root txt unknown /proc/47419/exe 2022-04-03T22:52:49.146 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/0 47665 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.146 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/0 47665 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.147 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/0 47665 root txt unknown /proc/47665/exe 2022-04-03T22:52:49.147 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.147 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.148 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root txt REG 8,1 1588952 10333 /usr/lib/systemd/systemd 2022-04-03T22:52:49.148 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 147336 7691 /usr/lib64/libnl-3.so.200.26.0 2022-04-03T22:52:49.148 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 549824 7699 /usr/lib64/libnl-route-3.so.200.26.0 2022-04-03T22:52:49.149 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 131056 8911 /usr/lib64/libibverbs.so.1.11.32.0 2022-04-03T22:52:49.149 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 2191840 7237 /usr/lib64/libm-2.28.so 2022-04-03T22:52:49.149 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 632944 9460 /usr/lib64/libudev.so.1.6.11 2022-04-03T22:52:49.150 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 739152 7375 /usr/lib64/libsepol.so.1 2022-04-03T22:52:49.150 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 1760264 7719 /usr/lib64/libunistring.so.2.1.0 2022-04-03T22:52:49.150 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 312232 8919 /usr/lib64/libpcap.so.1.9.1 2022-04-03T22:52:49.150 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 144392 7447 /usr/lib64/libgpg-error.so.0.24.2 2022-04-03T22:52:49.151 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 66752 7684 /usr/lib64/libjson-c.so.4.0.0 2022-04-03T22:52:49.151 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 371224 9529 /usr/lib64/libdevmapper.so.1.02 2022-04-03T22:52:49.151 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 26192 7582 /usr/lib64/libattr.so.1.1.2448 2022-04-03T22:52:49.152 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:49.152 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:49.152 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:49.153 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:49.153 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:49.153 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:49.154 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:49.154 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:49.154 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 119680 7888 /usr/lib64/liblz4.so.1.8.3 2022-04-03T22:52:49.155 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 192016 7418 /usr/lib64/liblzma.so.5.2.4 2022-04-03T22:52:49.155 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 162224 7721 /usr/lib64/libidn2.so.0.3.6 2022-04-03T22:52:49.155 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 33328 8926 /usr/lib64/libip4tc.so.2.0.0 2022-04-03T22:52:49.156 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 1188080 7705 /usr/lib64/libgcrypt.so.20.2.5 2022-04-03T22:52:49.156 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 510096 9565 /usr/lib64/libcryptsetup.so.12.6.0 2022-04-03T22:52:49.156 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 54816 7584 /usr/lib64/libacl.so.1.1.2253 2022-04-03T22:52:49.157 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:49.157 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:49.157 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:49.158 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:49.158 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 100208 9562 /usr/lib64/libkmod.so.2.3.3 2022-04-03T22:52:49.158 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:49.159 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:49.159 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:49.159 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:49.160 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 132888 8225 /usr/lib64/libseccomp.so.2.5.1 2022-04-03T22:52:49.160 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:49.160 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 2732744 10328 /usr/lib/systemd/libsystemd-shared-239.so 2022-04-03T22:52:49.161 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:49.161 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:49.161 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root 1u unix 0xffff88810907f800 0t0 192057 type=STREAM 2022-04-03T22:52:49.162 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root 2u unix 0xffff88810907f800 0t0 192057 type=STREAM 2022-04-03T22:52:49.162 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root 3u unix 0xffff88810907ad00 0t0 192076 type=DGRAM 2022-04-03T22:52:49.162 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root 4u a_inode 0,13 0 9293 [eventpoll] 2022-04-03T22:52:49.163 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root 5u a_inode 0,13 0 9293 [signalfd] 2022-04-03T22:52:49.163 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root 6r DIR 0,26 0 1583 /sys/fs/cgroup/systemd/user.slice/user-0.slice/user@0.service 2022-04-03T22:52:49.163 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root 7u a_inode 0,13 0 9293 [timerfd] 2022-04-03T22:52:49.164 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root 10r a_inode 0,13 0 9293 inotify 2022-04-03T22:52:49.164 INFO:teuthology.orchestra.run.smithi077.stdout:systemd 47836 root 11u LINK 8,1 29 497 /etc/localtime 2022-04-03T22:52:49.164 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/1 48201 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.165 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/1 48201 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.165 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/1 48201 root txt unknown /proc/48201/exe 2022-04-03T22:52:49.165 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/u 48287 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.166 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/u 48287 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.166 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/u 48287 root txt unknown /proc/48287/exe 2022-04-03T22:52:49.166 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/5 48366 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.167 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/5 48366 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.167 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/5 48366 root txt unknown /proc/48366/exe 2022-04-03T22:52:49.167 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 48771 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.168 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 48771 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.168 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/2 48771 root txt unknown /proc/48771/exe 2022-04-03T22:52:49.168 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/6 48772 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.169 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/6 48772 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.169 INFO:teuthology.orchestra.run.smithi077.stdout:kworker/6 48772 root txt unknown /proc/48772/exe 2022-04-03T22:52:49.169 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root cwd DIR 8,1 4096 2 / 2022-04-03T22:52:49.170 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.170 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root txt REG 8,1 38432 9424 /usr/bin/sleep 2022-04-03T22:52:49.170 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:49.171 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:49.171 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:49.171 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:49.172 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:49.172 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:49.172 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:49.173 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:49.173 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:49.173 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:49.174 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:49.174 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:49.174 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:49.175 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:49.175 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:49.175 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root 0r CHR 1,3 0t0 4 /dev/null 2022-04-03T22:52:49.176 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root 1u unix 0xffff88810eb14b00 0t0 9943 type=STREAM 2022-04-03T22:52:49.176 INFO:teuthology.orchestra.run.smithi077.stdout:sleep 48881 root 2u unix 0xffff88810eb14b00 0t0 9943 type=STREAM 2022-04-03T22:52:49.176 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:49.177 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.177 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu txt REG 8,1 1150736 7265 /usr/bin/bash 2022-04-03T22:52:49.177 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:49.178 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:49.178 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:49.178 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu mem REG 8,1 208616 7207 /usr/lib64/libtinfo.so.6.1 2022-04-03T22:52:49.179 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:49.179 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:49.179 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:49.180 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:49.180 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:49.180 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:49.180 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:49.181 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:49.181 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:49.181 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:49.182 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:49.182 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:49.182 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:49.183 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu 0r FIFO 0,12 0t0 194925 pipe 2022-04-03T22:52:49.183 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu 1w FIFO 0,12 0t0 194926 pipe 2022-04-03T22:52:49.183 INFO:teuthology.orchestra.run.smithi077.stdout:bash 48916 ubuntu 2w FIFO 0,12 0t0 194927 pipe 2022-04-03T22:52:49.184 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:49.184 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.185 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root txt REG 8,1 165640 12133 /usr/bin/sudo 2022-04-03T22:52:49.185 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 15928 23305 /usr/lib64/security/pam_succeed_if.so 2022-04-03T22:52:49.185 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 16536 9706 /usr/lib64/libpam_misc.so.0.82.1 2022-04-03T22:52:49.185 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 915232 23524 /usr/lib64/security/pam_systemd.so 2022-04-03T22:52:49.186 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 24048 23285 /usr/lib64/security/pam_limits.so 2022-04-03T22:52:49.186 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 11736 23283 /usr/lib64/security/pam_keyinit.so 2022-04-03T22:52:49.187 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 46232 9162 /usr/lib64/libcrack.so.2.9.0 2022-04-03T22:52:49.187 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 24744 9666 /usr/lib64/libpwquality.so.1.0.2 2022-04-03T22:52:49.187 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 11720 23217 /usr/lib64/security/pam_pwquality.so 2022-04-03T22:52:49.188 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 7640 23294 /usr/lib64/security/pam_permit.so 2022-04-03T22:52:49.188 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 7552 23271 /usr/lib64/security/pam_deny.so 2022-04-03T22:52:49.188 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 58032 24580 /usr/lib64/security/pam_sss.so 2022-04-03T22:52:49.189 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 371032 9256 /usr/lib64/libtirpc.so.3.0.0 2022-04-03T22:52:49.189 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 116408 9258 /usr/lib64/libnsl.so.2.0.0 2022-04-03T22:52:49.189 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 58016 23310 /usr/lib64/security/pam_unix.so 2022-04-03T22:52:49.189 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 11744 23287 /usr/lib64/security/pam_localuser.so 2022-04-03T22:52:49.190 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 11816 23316 /usr/lib64/security/pam_usertype.so 2022-04-03T22:52:49.190 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 11720 23275 /usr/lib64/security/pam_faildelay.so 2022-04-03T22:52:49.191 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 20160 23273 /usr/lib64/security/pam_env.so 2022-04-03T22:52:49.191 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:49.191 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 33392 7435 /usr/lib64/libuuid.so.1.3.0 2022-04-03T22:52:49.192 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 343544 9528 /usr/lib64/libblkid.so.1.1.0 2022-04-03T22:52:49.192 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 99656 4745 /usr/lib64/libgcc_s-8-20200928.so.1 2022-04-03T22:52:49.192 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 371312 9339 /usr/lib64/libmount.so.1.1.0 2022-04-03T22:52:49.193 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 24976 7426 /usr/lib64/libcap.so.2.26 2022-04-03T22:52:49.193 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 69024 7251 /usr/lib64/librt-2.28.so 2022-04-03T22:52:49.193 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 915176 9456 /usr/lib64/libnss_systemd.so.2 2022-04-03T22:52:49.193 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:49.194 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 83728 7245 /usr/lib64/libnss_files-2.28.so 2022-04-03T22:52:49.194 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:49.194 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:49.195 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:49.195 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:49.195 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:49.196 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:49.196 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 142712 7441 /usr/lib64/libcrypt.so.1.1.0 2022-04-03T22:52:49.196 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:49.197 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 615576 9539 /usr/lib64/libssl.so.1.1.1g 2022-04-03T22:52:49.197 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 125328 8879 /usr/lib64/libsasl2.so.3.0.0 2022-04-03T22:52:49.197 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:49.198 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:49.198 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 67136 8940 /usr/lib64/liblber-2.4.so.2.10.9 2022-04-03T22:52:49.198 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 325184 8942 /usr/lib64/libldap-2.4.so.2.10.9 2022-04-03T22:52:49.199 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 66432 9704 /usr/lib64/libpam.so.0.84.2 2022-04-03T22:52:49.199 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 432872 35707 /usr/libexec/sudo/sudoers.so 2022-04-03T22:52:49.199 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:49.200 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 46272 12131 /usr/lib64/libnss_sss.so.2 2022-04-03T22:52:49.200 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 2586930 21497 /usr/lib/locale/en_US.utf8/LC_COLLATE 2022-04-03T22:52:49.200 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:49.201 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 30256 7626 /usr/lib64/libcap-ng.so.0.0.0 2022-04-03T22:52:49.201 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:49.201 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:49.202 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:49.202 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 104424 35704 /usr/libexec/sudo/libsudo_util.so.0.0.0 2022-04-03T22:52:49.202 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 23464 7255 /usr/lib64/libutil-2.28.so 2022-04-03T22:52:49.203 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:49.203 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 123336 7628 /usr/lib64/libaudit.so.1.0.0 2022-04-03T22:52:49.203 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:49.204 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:49.204 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 54 21502 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2022-04-03T22:52:49.204 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 3316 21455 /usr/lib/locale/en_US.utf8/LC_TIME 2022-04-03T22:52:49.205 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 286 21453 /usr/lib/locale/en_US.utf8/LC_MONETARY 2022-04-03T22:52:49.205 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 57 21500 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2022-04-03T22:52:49.205 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 34 21517 /usr/lib/locale/en_US.utf8/LC_PAPER 2022-04-03T22:52:49.206 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 77 21501 /usr/lib/locale/en_US.utf8/LC_NAME 2022-04-03T22:52:49.206 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 167 21449 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2022-04-03T22:52:49.206 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 59 21454 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2022-04-03T22:52:49.207 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 23 21451 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2022-04-03T22:52:49.207 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:49.207 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root mem REG 8,1 368 21450 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2022-04-03T22:52:49.208 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root 0r FIFO 0,12 0t0 194925 pipe 2022-04-03T22:52:49.208 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root 1w FIFO 0,12 0t0 194926 pipe 2022-04-03T22:52:49.208 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root 2w FIFO 0,12 0t0 194927 pipe 2022-04-03T22:52:49.209 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root 3r REG 8,1 9253600 1329 /var/lib/sss/mc/passwd 2022-04-03T22:52:49.209 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root 4u unix 0xffff888178e4ad00 0t0 194466 type=STREAM 2022-04-03T22:52:49.209 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root 5u netlink 0t0 196040 AUDIT 2022-04-03T22:52:49.210 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root 6r REG 8,1 11567160 4100 /var/lib/sss/mc/initgroups 2022-04-03T22:52:49.210 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root 7r REG 8,1 6940392 1327 /var/lib/sss/mc/group 2022-04-03T22:52:49.210 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root 9u unix 0xffff88816f70f080 0t0 196043 type=DGRAM 2022-04-03T22:52:49.211 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root 10r FIFO 0,12 0t0 196046 pipe 2022-04-03T22:52:49.211 INFO:teuthology.orchestra.run.smithi077.stdout:sudo 48931 root 11w FIFO 0,12 0t0 196046 pipe 2022-04-03T22:52:49.211 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:49.212 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.212 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root txt REG 8,1 179672 13626 /usr/bin/lsof 2022-04-03T22:52:49.212 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:49.213 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:49.213 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:49.213 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:49.214 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:49.214 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:49.214 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:49.215 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:49.215 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:49.215 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:49.216 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:49.216 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:49.216 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:49.217 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root mem REG 8,1 371032 9256 /usr/lib64/libtirpc.so.3.0.0 2022-04-03T22:52:49.217 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:49.217 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:49.218 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:49.218 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:49.218 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root 0r FIFO 0,12 0t0 194925 pipe 2022-04-03T22:52:49.219 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root 1w FIFO 0,12 0t0 194926 pipe 2022-04-03T22:52:49.219 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root 2w FIFO 0,12 0t0 194927 pipe 2022-04-03T22:52:49.219 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root 3r DIR 0,21 0 1 /proc 2022-04-03T22:52:49.220 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root 4r DIR 0,21 0 194488 /proc/48933/fd 2022-04-03T22:52:49.220 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root 5w FIFO 0,12 0t0 194498 pipe 2022-04-03T22:52:49.220 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48933 root 6r FIFO 0,12 0t0 194499 pipe 2022-04-03T22:52:49.221 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root cwd DIR 8,1 4096 2300 /home/ubuntu 2022-04-03T22:52:49.221 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root rtd DIR 8,1 4096 2 / 2022-04-03T22:52:49.221 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root txt REG 8,1 179672 13626 /usr/bin/lsof 2022-04-03T22:52:49.222 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root mem REG 8,1 95416 7377 /usr/lib64/libz.so.1.2.11 2022-04-03T22:52:49.222 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root mem REG 8,1 123520 7249 /usr/lib64/libresolv-2.28.so 2022-04-03T22:52:49.222 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root mem REG 8,1 3071456 9537 /usr/lib64/libcrypto.so.1.1.1g 2022-04-03T22:52:49.223 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root mem REG 8,1 16240 7686 /usr/lib64/libkeyutils.so.1.6 2022-04-03T22:52:49.223 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root mem REG 8,1 71480 9247 /usr/lib64/libkrb5support.so.0.1 2022-04-03T22:52:49.223 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root mem REG 8,1 320704 7247 /usr/lib64/libpthread-2.28.so 2022-04-03T22:52:49.224 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root mem REG 8,1 17336 7379 /usr/lib64/libcom_err.so.2.1 2022-04-03T22:52:49.224 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root mem REG 8,1 96320 9241 /usr/lib64/libk5crypto.so.3.1 2022-04-03T22:52:49.224 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root mem REG 8,1 967368 9246 /usr/lib64/libkrb5.so.3.3 2022-04-03T22:52:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root mem REG 8,1 356000 9238 /usr/lib64/libgssapi_krb5.so.2.2 2022-04-03T22:52:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root mem REG 8,1 28816 7235 /usr/lib64/libdl-2.28.so 2022-04-03T22:52:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root mem REG 8,1 543160 7183 /usr/lib64/libpcre2-8.so.0.7.1 2022-04-03T22:52:49.226 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root mem REG 8,1 3167976 7233 /usr/lib64/libc-2.28.so 2022-04-03T22:52:49.226 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root mem REG 8,1 371032 9256 /usr/lib64/libtirpc.so.3.0.0 2022-04-03T22:52:49.226 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root mem REG 8,1 168536 7187 /usr/lib64/libselinux.so.1 2022-04-03T22:52:49.227 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root mem REG 8,1 278504 7226 /usr/lib64/ld-2.28.so 2022-04-03T22:52:49.227 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root mem REG 8,1 337024 21498 /usr/lib/locale/en_US.utf8/LC_CTYPE 2022-04-03T22:52:49.227 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root mem REG 8,1 26998 18232 /usr/lib64/gconv/gconv-modules.cache 2022-04-03T22:52:49.228 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root 4r FIFO 0,12 0t0 194498 pipe 2022-04-03T22:52:49.228 INFO:teuthology.orchestra.run.smithi077.stdout:lsof 48934 root 7w FIFO 0,12 0t0 194499 pipe 2022-04-03T22:52:49.228 INFO:teuthology.orchestra.run.smithi077.stdout:USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 2022-04-03T22:52:49.229 INFO:teuthology.orchestra.run.smithi077.stdout:root 2 0.0 0.0 0 0 ? S 19:34 0:00 [kthreadd] 2022-04-03T22:52:49.229 INFO:teuthology.orchestra.run.smithi077.stdout:root 3 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [rcu_gp] 2022-04-03T22:52:49.229 INFO:teuthology.orchestra.run.smithi077.stdout:root 4 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [rcu_par_gp] 2022-04-03T22:52:49.230 INFO:teuthology.orchestra.run.smithi077.stdout:root 6 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [kworker/0:0H-events_highpri] 2022-04-03T22:52:49.230 INFO:teuthology.orchestra.run.smithi077.stdout:root 8 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [mm_percpu_wq] 2022-04-03T22:52:49.230 INFO:teuthology.orchestra.run.smithi077.stdout:root 9 0.0 0.0 0 0 ? I 19:34 0:00 \_ [rcu_tasks_kthread] 2022-04-03T22:52:49.231 INFO:teuthology.orchestra.run.smithi077.stdout:root 10 0.0 0.0 0 0 ? I 19:34 0:00 \_ [rcu_tasks_rude_kthread] 2022-04-03T22:52:49.231 INFO:teuthology.orchestra.run.smithi077.stdout:root 11 0.0 0.0 0 0 ? I 19:34 0:00 \_ [rcu_tasks_trace_kthread] 2022-04-03T22:52:49.231 INFO:teuthology.orchestra.run.smithi077.stdout:root 13 0.0 0.0 0 0 ? S 19:34 0:00 \_ [ksoftirqd/0] 2022-04-03T22:52:49.232 INFO:teuthology.orchestra.run.smithi077.stdout:root 14 0.0 0.0 0 0 ? I 19:34 0:03 \_ [rcu_preempt] 2022-04-03T22:52:49.232 INFO:teuthology.orchestra.run.smithi077.stdout:root 15 0.0 0.0 0 0 ? S 19:34 0:00 \_ [migration/0] 2022-04-03T22:52:49.232 INFO:teuthology.orchestra.run.smithi077.stdout:root 16 0.0 0.0 0 0 ? S 19:34 0:00 \_ [cpuhp/0] 2022-04-03T22:52:49.233 INFO:teuthology.orchestra.run.smithi077.stdout:root 17 0.0 0.0 0 0 ? S 19:34 0:00 \_ [cpuhp/1] 2022-04-03T22:52:49.233 INFO:teuthology.orchestra.run.smithi077.stdout:root 18 0.0 0.0 0 0 ? S 19:34 0:00 \_ [migration/1] 2022-04-03T22:52:49.233 INFO:teuthology.orchestra.run.smithi077.stdout:root 19 0.0 0.0 0 0 ? S 19:34 0:00 \_ [ksoftirqd/1] 2022-04-03T22:52:49.234 INFO:teuthology.orchestra.run.smithi077.stdout:root 21 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [kworker/1:0H-kblockd] 2022-04-03T22:52:49.234 INFO:teuthology.orchestra.run.smithi077.stdout:root 22 0.0 0.0 0 0 ? S 19:34 0:00 \_ [cpuhp/2] 2022-04-03T22:52:49.234 INFO:teuthology.orchestra.run.smithi077.stdout:root 23 0.0 0.0 0 0 ? S 19:34 0:00 \_ [migration/2] 2022-04-03T22:52:49.235 INFO:teuthology.orchestra.run.smithi077.stdout:root 24 0.0 0.0 0 0 ? S 19:34 0:00 \_ [ksoftirqd/2] 2022-04-03T22:52:49.235 INFO:teuthology.orchestra.run.smithi077.stdout:root 26 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [kworker/2:0H-events_highpri] 2022-04-03T22:52:49.235 INFO:teuthology.orchestra.run.smithi077.stdout:root 27 0.0 0.0 0 0 ? S 19:34 0:00 \_ [cpuhp/3] 2022-04-03T22:52:49.236 INFO:teuthology.orchestra.run.smithi077.stdout:root 28 0.0 0.0 0 0 ? S 19:34 0:00 \_ [migration/3] 2022-04-03T22:52:49.236 INFO:teuthology.orchestra.run.smithi077.stdout:root 29 0.0 0.0 0 0 ? S 19:34 0:00 \_ [ksoftirqd/3] 2022-04-03T22:52:49.236 INFO:teuthology.orchestra.run.smithi077.stdout:root 31 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [kworker/3:0H-events_highpri] 2022-04-03T22:52:49.237 INFO:teuthology.orchestra.run.smithi077.stdout:root 32 0.0 0.0 0 0 ? S 19:34 0:00 \_ [cpuhp/4] 2022-04-03T22:52:49.237 INFO:teuthology.orchestra.run.smithi077.stdout:root 33 0.0 0.0 0 0 ? S 19:34 0:00 \_ [migration/4] 2022-04-03T22:52:49.237 INFO:teuthology.orchestra.run.smithi077.stdout:root 34 0.0 0.0 0 0 ? S 19:34 0:00 \_ [ksoftirqd/4] 2022-04-03T22:52:49.238 INFO:teuthology.orchestra.run.smithi077.stdout:root 36 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [kworker/4:0H-events_highpri] 2022-04-03T22:52:49.238 INFO:teuthology.orchestra.run.smithi077.stdout:root 37 0.0 0.0 0 0 ? S 19:34 0:00 \_ [cpuhp/5] 2022-04-03T22:52:49.238 INFO:teuthology.orchestra.run.smithi077.stdout:root 38 0.0 0.0 0 0 ? S 19:34 0:00 \_ [migration/5] 2022-04-03T22:52:49.239 INFO:teuthology.orchestra.run.smithi077.stdout:root 39 0.0 0.0 0 0 ? S 19:34 0:00 \_ [ksoftirqd/5] 2022-04-03T22:52:49.239 INFO:teuthology.orchestra.run.smithi077.stdout:root 41 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [kworker/5:0H-events_highpri] 2022-04-03T22:52:49.239 INFO:teuthology.orchestra.run.smithi077.stdout:root 42 0.0 0.0 0 0 ? S 19:34 0:00 \_ [cpuhp/6] 2022-04-03T22:52:49.240 INFO:teuthology.orchestra.run.smithi077.stdout:root 43 0.0 0.0 0 0 ? S 19:34 0:00 \_ [migration/6] 2022-04-03T22:52:49.240 INFO:teuthology.orchestra.run.smithi077.stdout:root 44 0.0 0.0 0 0 ? S 19:34 0:00 \_ [ksoftirqd/6] 2022-04-03T22:52:49.240 INFO:teuthology.orchestra.run.smithi077.stdout:root 46 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [kworker/6:0H-events_highpri] 2022-04-03T22:52:49.241 INFO:teuthology.orchestra.run.smithi077.stdout:root 47 0.0 0.0 0 0 ? S 19:34 0:00 \_ [cpuhp/7] 2022-04-03T22:52:49.241 INFO:teuthology.orchestra.run.smithi077.stdout:root 48 0.0 0.0 0 0 ? S 19:34 0:00 \_ [migration/7] 2022-04-03T22:52:49.241 INFO:teuthology.orchestra.run.smithi077.stdout:root 49 0.0 0.0 0 0 ? S 19:34 0:00 \_ [ksoftirqd/7] 2022-04-03T22:52:49.242 INFO:teuthology.orchestra.run.smithi077.stdout:root 51 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [kworker/7:0H-events_highpri] 2022-04-03T22:52:49.242 INFO:teuthology.orchestra.run.smithi077.stdout:root 52 0.0 0.0 0 0 ? S 19:34 0:00 \_ [kdevtmpfs] 2022-04-03T22:52:49.242 INFO:teuthology.orchestra.run.smithi077.stdout:root 53 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [netns] 2022-04-03T22:52:49.243 INFO:teuthology.orchestra.run.smithi077.stdout:root 54 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [inet_frag_wq] 2022-04-03T22:52:49.243 INFO:teuthology.orchestra.run.smithi077.stdout:root 55 0.0 0.0 0 0 ? S 19:34 0:00 \_ [kauditd] 2022-04-03T22:52:49.243 INFO:teuthology.orchestra.run.smithi077.stdout:root 56 0.0 0.0 0 0 ? S 19:34 0:00 \_ [khungtaskd] 2022-04-03T22:52:49.244 INFO:teuthology.orchestra.run.smithi077.stdout:root 57 0.0 0.0 0 0 ? S 19:34 0:00 \_ [oom_reaper] 2022-04-03T22:52:49.244 INFO:teuthology.orchestra.run.smithi077.stdout:root 58 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [writeback] 2022-04-03T22:52:49.245 INFO:teuthology.orchestra.run.smithi077.stdout:root 59 0.0 0.0 0 0 ? S 19:34 0:00 \_ [kcompactd0] 2022-04-03T22:52:49.245 INFO:teuthology.orchestra.run.smithi077.stdout:root 60 0.0 0.0 0 0 ? SN 19:34 0:00 \_ [ksmd] 2022-04-03T22:52:49.245 INFO:teuthology.orchestra.run.smithi077.stdout:root 61 0.0 0.0 0 0 ? SN 19:34 0:00 \_ [khugepaged] 2022-04-03T22:52:49.246 INFO:teuthology.orchestra.run.smithi077.stdout:root 62 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [cryptd] 2022-04-03T22:52:49.246 INFO:teuthology.orchestra.run.smithi077.stdout:root 63 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [kintegrityd] 2022-04-03T22:52:49.246 INFO:teuthology.orchestra.run.smithi077.stdout:root 64 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [kblockd] 2022-04-03T22:52:49.247 INFO:teuthology.orchestra.run.smithi077.stdout:root 65 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [blkcg_punt_bio] 2022-04-03T22:52:49.247 INFO:teuthology.orchestra.run.smithi077.stdout:root 73 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [ata_sff] 2022-04-03T22:52:49.247 INFO:teuthology.orchestra.run.smithi077.stdout:root 74 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [md] 2022-04-03T22:52:49.248 INFO:teuthology.orchestra.run.smithi077.stdout:root 75 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [edac-poller] 2022-04-03T22:52:49.248 INFO:teuthology.orchestra.run.smithi077.stdout:root 76 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [devfreq_wq] 2022-04-03T22:52:49.248 INFO:teuthology.orchestra.run.smithi077.stdout:root 77 0.0 0.0 0 0 ? S 19:34 0:00 \_ [watchdogd] 2022-04-03T22:52:49.249 INFO:teuthology.orchestra.run.smithi077.stdout:root 79 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [kworker/0:1H-kblockd] 2022-04-03T22:52:49.249 INFO:teuthology.orchestra.run.smithi077.stdout:root 80 0.0 0.0 0 0 ? S 19:34 0:00 \_ [kswapd0] 2022-04-03T22:52:49.249 INFO:teuthology.orchestra.run.smithi077.stdout:root 81 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [kthrotld] 2022-04-03T22:52:49.250 INFO:teuthology.orchestra.run.smithi077.stdout:root 84 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [acpi_thermal_pm] 2022-04-03T22:52:49.250 INFO:teuthology.orchestra.run.smithi077.stdout:root 87 0.0 0.0 0 0 ? S 19:34 0:00 \_ [scsi_eh_0] 2022-04-03T22:52:49.250 INFO:teuthology.orchestra.run.smithi077.stdout:root 88 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [scsi_tmf_0] 2022-04-03T22:52:49.251 INFO:teuthology.orchestra.run.smithi077.stdout:root 89 0.0 0.0 0 0 ? S 19:34 0:00 \_ [scsi_eh_1] 2022-04-03T22:52:49.251 INFO:teuthology.orchestra.run.smithi077.stdout:root 90 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [scsi_tmf_1] 2022-04-03T22:52:49.251 INFO:teuthology.orchestra.run.smithi077.stdout:root 91 0.0 0.0 0 0 ? S 19:34 0:00 \_ [scsi_eh_2] 2022-04-03T22:52:49.252 INFO:teuthology.orchestra.run.smithi077.stdout:root 92 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [scsi_tmf_2] 2022-04-03T22:52:49.252 INFO:teuthology.orchestra.run.smithi077.stdout:root 93 0.0 0.0 0 0 ? S 19:34 0:00 \_ [scsi_eh_3] 2022-04-03T22:52:49.252 INFO:teuthology.orchestra.run.smithi077.stdout:root 94 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [scsi_tmf_3] 2022-04-03T22:52:49.253 INFO:teuthology.orchestra.run.smithi077.stdout:root 98 0.0 0.0 0 0 ? S 19:34 0:00 \_ [scsi_eh_4] 2022-04-03T22:52:49.253 INFO:teuthology.orchestra.run.smithi077.stdout:root 99 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [scsi_tmf_4] 2022-04-03T22:52:49.253 INFO:teuthology.orchestra.run.smithi077.stdout:root 100 0.0 0.0 0 0 ? S 19:34 0:00 \_ [scsi_eh_5] 2022-04-03T22:52:49.254 INFO:teuthology.orchestra.run.smithi077.stdout:root 101 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [scsi_tmf_5] 2022-04-03T22:52:49.254 INFO:teuthology.orchestra.run.smithi077.stdout:root 102 0.0 0.0 0 0 ? S 19:34 0:00 \_ [scsi_eh_6] 2022-04-03T22:52:49.254 INFO:teuthology.orchestra.run.smithi077.stdout:root 103 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [scsi_tmf_6] 2022-04-03T22:52:49.255 INFO:teuthology.orchestra.run.smithi077.stdout:root 104 0.0 0.0 0 0 ? S 19:34 0:00 \_ [scsi_eh_7] 2022-04-03T22:52:49.255 INFO:teuthology.orchestra.run.smithi077.stdout:root 105 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [scsi_tmf_7] 2022-04-03T22:52:49.255 INFO:teuthology.orchestra.run.smithi077.stdout:root 106 0.0 0.0 0 0 ? S 19:34 0:00 \_ [scsi_eh_8] 2022-04-03T22:52:49.256 INFO:teuthology.orchestra.run.smithi077.stdout:root 107 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [scsi_tmf_8] 2022-04-03T22:52:49.256 INFO:teuthology.orchestra.run.smithi077.stdout:root 108 0.0 0.0 0 0 ? S 19:34 0:00 \_ [scsi_eh_9] 2022-04-03T22:52:49.256 INFO:teuthology.orchestra.run.smithi077.stdout:root 109 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [scsi_tmf_9] 2022-04-03T22:52:49.257 INFO:teuthology.orchestra.run.smithi077.stdout:root 117 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [kworker/1:1H-kblockd] 2022-04-03T22:52:49.257 INFO:teuthology.orchestra.run.smithi077.stdout:root 118 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [dm_bufio_cache] 2022-04-03T22:52:49.257 INFO:teuthology.orchestra.run.smithi077.stdout:root 119 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [mld] 2022-04-03T22:52:49.258 INFO:teuthology.orchestra.run.smithi077.stdout:root 120 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [kworker/3:1H-kblockd] 2022-04-03T22:52:49.258 INFO:teuthology.orchestra.run.smithi077.stdout:root 121 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [ipv6_addrconf] 2022-04-03T22:52:49.258 INFO:teuthology.orchestra.run.smithi077.stdout:root 137 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [zswap-shrink] 2022-04-03T22:52:49.259 INFO:teuthology.orchestra.run.smithi077.stdout:root 209 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [kworker/u17:0] 2022-04-03T22:52:49.259 INFO:teuthology.orchestra.run.smithi077.stdout:root 248 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [kworker/4:1H-kblockd] 2022-04-03T22:52:49.259 INFO:teuthology.orchestra.run.smithi077.stdout:root 253 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [kworker/6:1H-kblockd] 2022-04-03T22:52:49.260 INFO:teuthology.orchestra.run.smithi077.stdout:root 265 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [kworker/2:1H-kblockd] 2022-04-03T22:52:49.260 INFO:teuthology.orchestra.run.smithi077.stdout:root 274 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [kworker/5:1H-kblockd] 2022-04-03T22:52:49.260 INFO:teuthology.orchestra.run.smithi077.stdout:root 323 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [kworker/7:1H-kblockd] 2022-04-03T22:52:49.261 INFO:teuthology.orchestra.run.smithi077.stdout:root 421 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [nvme-wq] 2022-04-03T22:52:49.261 INFO:teuthology.orchestra.run.smithi077.stdout:root 422 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [nvme-reset-wq] 2022-04-03T22:52:49.261 INFO:teuthology.orchestra.run.smithi077.stdout:root 423 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [nvme-delete-wq] 2022-04-03T22:52:49.262 INFO:teuthology.orchestra.run.smithi077.stdout:root 430 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [ixgbe] 2022-04-03T22:52:49.262 INFO:teuthology.orchestra.run.smithi077.stdout:root 445 0.0 0.0 0 0 ? S 19:34 0:00 \_ [jbd2/sda1-8] 2022-04-03T22:52:49.262 INFO:teuthology.orchestra.run.smithi077.stdout:root 446 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [ext4-rsv-conver] 2022-04-03T22:52:49.263 INFO:teuthology.orchestra.run.smithi077.stdout:root 602 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [ipmi-msghandler] 2022-04-03T22:52:49.263 INFO:teuthology.orchestra.run.smithi077.stdout:root 620 0.0 0.0 0 0 ? SN 19:34 0:00 \_ [kipmi0] 2022-04-03T22:52:49.263 INFO:teuthology.orchestra.run.smithi077.stdout:root 734 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [rpciod] 2022-04-03T22:52:49.264 INFO:teuthology.orchestra.run.smithi077.stdout:root 735 0.0 0.0 0 0 ? I< 19:34 0:00 \_ [xprtiod] 2022-04-03T22:52:49.264 INFO:teuthology.orchestra.run.smithi077.stdout:root 6378 0.0 0.0 0 0 ? I< 19:40 0:00 \_ [kdmflush/253:0] 2022-04-03T22:52:49.264 INFO:teuthology.orchestra.run.smithi077.stdout:root 6411 0.0 0.0 0 0 ? I< 19:40 0:00 \_ [kdmflush/253:1] 2022-04-03T22:52:49.265 INFO:teuthology.orchestra.run.smithi077.stdout:root 6444 0.0 0.0 0 0 ? I< 19:40 0:00 \_ [kdmflush/253:2] 2022-04-03T22:52:49.265 INFO:teuthology.orchestra.run.smithi077.stdout:root 6477 0.0 0.0 0 0 ? I< 19:40 0:00 \_ [kdmflush/253:3] 2022-04-03T22:52:49.265 INFO:teuthology.orchestra.run.smithi077.stdout:root 6510 0.0 0.0 0 0 ? I< 19:40 0:00 \_ [kdmflush/253:4] 2022-04-03T22:52:49.266 INFO:teuthology.orchestra.run.smithi077.stdout:root 6690 0.0 0.0 0 0 ? I< 19:40 0:00 \_ [xfsalloc] 2022-04-03T22:52:49.266 INFO:teuthology.orchestra.run.smithi077.stdout:root 6691 0.0 0.0 0 0 ? I< 19:40 0:00 \_ [xfs_mru_cache] 2022-04-03T22:52:49.266 INFO:teuthology.orchestra.run.smithi077.stdout:root 6693 0.0 0.0 0 0 ? I< 19:40 0:00 \_ [xfs-buf/dm-4] 2022-04-03T22:52:49.267 INFO:teuthology.orchestra.run.smithi077.stdout:root 6694 0.0 0.0 0 0 ? I< 19:40 0:00 \_ [xfs-conv/dm-4] 2022-04-03T22:52:49.267 INFO:teuthology.orchestra.run.smithi077.stdout:root 6695 0.0 0.0 0 0 ? I< 19:40 0:00 \_ [xfs-reclaim/dm-] 2022-04-03T22:52:49.267 INFO:teuthology.orchestra.run.smithi077.stdout:root 6696 0.0 0.0 0 0 ? I< 19:40 0:00 \_ [xfs-blockgc/dm-] 2022-04-03T22:52:49.268 INFO:teuthology.orchestra.run.smithi077.stdout:root 6697 0.0 0.0 0 0 ? I< 19:40 0:00 \_ [xfs-inodegc/dm-] 2022-04-03T22:52:49.268 INFO:teuthology.orchestra.run.smithi077.stdout:root 6698 0.0 0.0 0 0 ? I< 19:40 0:00 \_ [xfs-log/dm-4] 2022-04-03T22:52:49.268 INFO:teuthology.orchestra.run.smithi077.stdout:root 6699 0.0 0.0 0 0 ? I< 19:40 0:00 \_ [xfs-cil/dm-4] 2022-04-03T22:52:49.269 INFO:teuthology.orchestra.run.smithi077.stdout:root 6700 0.0 0.0 0 0 ? S 19:40 0:00 \_ [xfsaild/dm-4] 2022-04-03T22:52:49.269 INFO:teuthology.orchestra.run.smithi077.stdout:root 14356 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-buf/dm-0] 2022-04-03T22:52:49.269 INFO:teuthology.orchestra.run.smithi077.stdout:root 14357 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-conv/dm-0] 2022-04-03T22:52:49.270 INFO:teuthology.orchestra.run.smithi077.stdout:root 14358 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-reclaim/dm-] 2022-04-03T22:52:49.270 INFO:teuthology.orchestra.run.smithi077.stdout:root 14359 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-blockgc/dm-] 2022-04-03T22:52:49.270 INFO:teuthology.orchestra.run.smithi077.stdout:root 14360 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-inodegc/dm-] 2022-04-03T22:52:49.271 INFO:teuthology.orchestra.run.smithi077.stdout:root 14361 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-log/dm-0] 2022-04-03T22:52:49.271 INFO:teuthology.orchestra.run.smithi077.stdout:root 14362 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-cil/dm-0] 2022-04-03T22:52:49.271 INFO:teuthology.orchestra.run.smithi077.stdout:root 14363 0.0 0.0 0 0 ? S 19:43 0:00 \_ [xfsaild/dm-0] 2022-04-03T22:52:49.272 INFO:teuthology.orchestra.run.smithi077.stdout:root 14436 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-buf/dm-1] 2022-04-03T22:52:49.272 INFO:teuthology.orchestra.run.smithi077.stdout:root 14437 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-conv/dm-1] 2022-04-03T22:52:49.272 INFO:teuthology.orchestra.run.smithi077.stdout:root 14438 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-reclaim/dm-] 2022-04-03T22:52:49.273 INFO:teuthology.orchestra.run.smithi077.stdout:root 14439 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-blockgc/dm-] 2022-04-03T22:52:49.273 INFO:teuthology.orchestra.run.smithi077.stdout:root 14440 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-inodegc/dm-] 2022-04-03T22:52:49.273 INFO:teuthology.orchestra.run.smithi077.stdout:root 14441 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-log/dm-1] 2022-04-03T22:52:49.274 INFO:teuthology.orchestra.run.smithi077.stdout:root 14442 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-cil/dm-1] 2022-04-03T22:52:49.274 INFO:teuthology.orchestra.run.smithi077.stdout:root 14443 0.0 0.0 0 0 ? S 19:43 0:00 \_ [xfsaild/dm-1] 2022-04-03T22:52:49.274 INFO:teuthology.orchestra.run.smithi077.stdout:root 14516 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-buf/dm-2] 2022-04-03T22:52:49.275 INFO:teuthology.orchestra.run.smithi077.stdout:root 14517 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-conv/dm-2] 2022-04-03T22:52:49.275 INFO:teuthology.orchestra.run.smithi077.stdout:root 14518 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-reclaim/dm-] 2022-04-03T22:52:49.275 INFO:teuthology.orchestra.run.smithi077.stdout:root 14519 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-blockgc/dm-] 2022-04-03T22:52:49.276 INFO:teuthology.orchestra.run.smithi077.stdout:root 14520 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-inodegc/dm-] 2022-04-03T22:52:49.276 INFO:teuthology.orchestra.run.smithi077.stdout:root 14521 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-log/dm-2] 2022-04-03T22:52:49.276 INFO:teuthology.orchestra.run.smithi077.stdout:root 14522 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-cil/dm-2] 2022-04-03T22:52:49.277 INFO:teuthology.orchestra.run.smithi077.stdout:root 14523 0.0 0.0 0 0 ? S 19:43 0:03 \_ [xfsaild/dm-2] 2022-04-03T22:52:49.277 INFO:teuthology.orchestra.run.smithi077.stdout:root 14596 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-buf/dm-3] 2022-04-03T22:52:49.277 INFO:teuthology.orchestra.run.smithi077.stdout:root 14597 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-conv/dm-3] 2022-04-03T22:52:49.278 INFO:teuthology.orchestra.run.smithi077.stdout:root 14598 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-reclaim/dm-] 2022-04-03T22:52:49.278 INFO:teuthology.orchestra.run.smithi077.stdout:root 14599 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-blockgc/dm-] 2022-04-03T22:52:49.278 INFO:teuthology.orchestra.run.smithi077.stdout:root 14600 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-inodegc/dm-] 2022-04-03T22:52:49.279 INFO:teuthology.orchestra.run.smithi077.stdout:root 14601 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-log/dm-3] 2022-04-03T22:52:49.279 INFO:teuthology.orchestra.run.smithi077.stdout:root 14602 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [xfs-cil/dm-3] 2022-04-03T22:52:49.279 INFO:teuthology.orchestra.run.smithi077.stdout:root 14603 0.0 0.0 0 0 ? S 19:43 0:03 \_ [xfsaild/dm-3] 2022-04-03T22:52:49.280 INFO:teuthology.orchestra.run.smithi077.stdout:root 14649 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [dio/dm-0] 2022-04-03T22:52:49.280 INFO:teuthology.orchestra.run.smithi077.stdout:root 15819 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [dio/dm-1] 2022-04-03T22:52:49.280 INFO:teuthology.orchestra.run.smithi077.stdout:root 16989 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [dio/dm-2] 2022-04-03T22:52:49.281 INFO:teuthology.orchestra.run.smithi077.stdout:root 18159 0.0 0.0 0 0 ? I< 19:43 0:00 \_ [dio/dm-3] 2022-04-03T22:52:49.281 INFO:teuthology.orchestra.run.smithi077.stdout:root 23724 0.0 0.0 0 0 ? I 19:44 0:00 \_ [kworker/3:5-ceph-msgr] 2022-04-03T22:52:49.281 INFO:teuthology.orchestra.run.smithi077.stdout:root 24172 0.0 0.0 0 0 ? D 19:44 0:03 \_ [kworker/2:75+ceph-msgr] 2022-04-03T22:52:49.282 INFO:teuthology.orchestra.run.smithi077.stdout:root 28193 0.0 0.0 0 0 ? I< 19:44 0:00 \_ [ceph-msgr] 2022-04-03T22:52:49.282 INFO:teuthology.orchestra.run.smithi077.stdout:root 28203 0.0 0.0 0 0 ? I< 19:44 0:00 \_ [ceph-watch-noti] 2022-04-03T22:52:49.282 INFO:teuthology.orchestra.run.smithi077.stdout:root 28204 0.0 0.0 0 0 ? I< 19:44 0:00 \_ [ceph-completion] 2022-04-03T22:52:49.283 INFO:teuthology.orchestra.run.smithi077.stdout:root 33320 0.0 0.0 0 0 ? D 19:49 0:01 \_ [kworker/0:0+ceph-msgr] 2022-04-03T22:52:49.283 INFO:teuthology.orchestra.run.smithi077.stdout:root 34229 0.0 0.0 0 0 ? D 19:50 0:00 \_ [kworker/u16:2+flush-ceph-1] 2022-04-03T22:52:49.283 INFO:teuthology.orchestra.run.smithi077.stdout:root 35559 0.0 0.0 0 0 ? D 19:52 0:01 \_ [kworker/4:0+events] 2022-04-03T22:52:49.284 INFO:teuthology.orchestra.run.smithi077.stdout:root 36525 0.0 0.0 0 0 ? I 19:54 0:00 \_ [kworker/3:0-ceph-msgr] 2022-04-03T22:52:49.284 INFO:teuthology.orchestra.run.smithi077.stdout:root 37577 0.0 0.0 0 0 ? D 19:55 0:00 \_ [kworker/0:1+ceph-msgr] 2022-04-03T22:52:49.284 INFO:teuthology.orchestra.run.smithi077.stdout:root 38795 0.0 0.0 0 0 ? D 19:59 0:00 \_ [kworker/6:1+events] 2022-04-03T22:52:49.285 INFO:teuthology.orchestra.run.smithi077.stdout:root 38969 0.0 0.0 0 0 ? D 19:59 0:00 \_ [kworker/4:1+events] 2022-04-03T22:52:49.285 INFO:teuthology.orchestra.run.smithi077.stdout:root 39124 0.0 0.0 0 0 ? I 20:01 0:04 \_ [kworker/0:3-mm_percpu_wq] 2022-04-03T22:52:49.285 INFO:teuthology.orchestra.run.smithi077.stdout:root 39896 0.0 0.0 0 0 ? D 20:14 0:00 \_ [kworker/6:0+cgroup_destroy] 2022-04-03T22:52:49.286 INFO:teuthology.orchestra.run.smithi077.stdout:root 40538 0.0 0.0 0 0 ? D 20:26 0:00 \_ [kworker/2:0+ceph-msgr] 2022-04-03T22:52:49.286 INFO:teuthology.orchestra.run.smithi077.stdout:root 40557 0.0 0.0 0 0 ? D 20:26 0:00 \_ [kworker/4:3+cgroup_destroy] 2022-04-03T22:52:49.286 INFO:teuthology.orchestra.run.smithi077.stdout:root 40610 0.0 0.0 0 0 ? I 20:26 0:00 \_ [kworker/7:3-cgroup_pidlist_destroy] 2022-04-03T22:52:49.287 INFO:teuthology.orchestra.run.smithi077.stdout:root 42608 0.0 0.0 0 0 ? I 21:00 0:00 \_ [kworker/2:2-mm_percpu_wq] 2022-04-03T22:52:49.287 INFO:teuthology.orchestra.run.smithi077.stdout:root 43728 0.0 0.0 0 0 ? I 21:20 0:00 \_ [kworker/4:2-cgroup_pidlist_destroy] 2022-04-03T22:52:49.287 INFO:teuthology.orchestra.run.smithi077.stdout:root 45842 0.0 0.0 0 0 ? I 22:00 0:02 \_ [kworker/5:1-mm_percpu_wq] 2022-04-03T22:52:49.287 INFO:teuthology.orchestra.run.smithi077.stdout:root 46357 0.0 0.0 0 0 ? I 22:10 0:00 \_ [kworker/7:0-mm_percpu_wq] 2022-04-03T22:52:49.288 INFO:teuthology.orchestra.run.smithi077.stdout:root 46520 0.0 0.0 0 0 ? D 22:12 0:00 \_ [kworker/u16:3+events_unbound] 2022-04-03T22:52:49.288 INFO:teuthology.orchestra.run.smithi077.stdout:root 46907 0.0 0.0 0 0 ? I 22:20 0:00 \_ [kworker/1:1-mm_percpu_wq] 2022-04-03T22:52:49.288 INFO:teuthology.orchestra.run.smithi077.stdout:root 47073 0.0 0.0 0 0 ? I 22:23 0:00 \_ [kworker/u16:0-ixgbe] 2022-04-03T22:52:49.289 INFO:teuthology.orchestra.run.smithi077.stdout:root 47310 0.0 0.0 0 0 ? D 22:26 0:00 \_ [kworker/7:1+cgroup_destroy] 2022-04-03T22:52:49.289 INFO:teuthology.orchestra.run.smithi077.stdout:root 47312 0.0 0.0 0 0 ? I 22:26 0:00 \_ [kworker/2:1-events] 2022-04-03T22:52:49.289 INFO:teuthology.orchestra.run.smithi077.stdout:root 47324 0.0 0.0 0 0 ? I 22:26 0:00 \_ [kworker/6:2-cgroup_pidlist_destroy] 2022-04-03T22:52:49.290 INFO:teuthology.orchestra.run.smithi077.stdout:root 47361 0.0 0.0 0 0 ? I 22:26 0:00 \_ [kworker/u16:4-events_unbound] 2022-04-03T22:52:49.290 INFO:teuthology.orchestra.run.smithi077.stdout:root 47419 0.0 0.0 0 0 ? I 22:26 0:00 \_ [kworker/4:4-mm_percpu_wq] 2022-04-03T22:52:49.290 INFO:teuthology.orchestra.run.smithi077.stdout:root 47665 0.0 0.0 0 0 ? I 22:30 0:00 \_ [kworker/0:4] 2022-04-03T22:52:49.291 INFO:teuthology.orchestra.run.smithi077.stdout:root 48201 0.0 0.0 0 0 ? I 22:39 0:00 \_ [kworker/1:2] 2022-04-03T22:52:49.291 INFO:teuthology.orchestra.run.smithi077.stdout:root 48287 0.0 0.0 0 0 ? I 22:41 0:00 \_ [kworker/u16:1] 2022-04-03T22:52:49.291 INFO:teuthology.orchestra.run.smithi077.stdout:root 48366 0.0 0.0 0 0 ? I 22:43 0:00 \_ [kworker/5:0] 2022-04-03T22:52:49.292 INFO:teuthology.orchestra.run.smithi077.stdout:root 48771 0.0 0.0 0 0 ? I 22:50 0:00 \_ [kworker/2:3] 2022-04-03T22:52:49.292 INFO:teuthology.orchestra.run.smithi077.stdout:root 48772 0.0 0.0 0 0 ? I 22:50 0:00 \_ [kworker/6:3-events] 2022-04-03T22:52:49.292 INFO:teuthology.orchestra.run.smithi077.stdout:root 1 0.0 0.0 254396 16520 ? Ss 19:34 0:07 /usr/lib/systemd/systemd --switched-root --system --deserialize 18 2022-04-03T22:52:49.293 INFO:teuthology.orchestra.run.smithi077.stdout:root 541 0.0 0.0 109248 22004 ? Ss 19:34 0:01 /usr/lib/systemd/systemd-journald 2022-04-03T22:52:49.293 INFO:teuthology.orchestra.run.smithi077.stdout:root 576 0.0 0.0 127160 13728 ? Ss 19:34 0:01 /usr/lib/systemd/systemd-udevd 2022-04-03T22:52:49.293 INFO:teuthology.orchestra.run.smithi077.stdout:rpc 693 0.0 0.0 67176 5444 ? Ss 19:34 0:00 /usr/bin/rpcbind -w -f 2022-04-03T22:52:49.294 INFO:teuthology.orchestra.run.smithi077.stdout:root 703 0.0 0.0 134908 2564 ? S 2022-04-03T22:52:49.303 INFO:teuthology.orchestra.run.smithi077.stdout:root 20160 0.0 0.0 150084 7944 ? Ss 19:43 0:00 \_ sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2022-04-03T22:52:49.304 INFO:teuthology.orchestra.run.smithi077.stdout:root 20231 0.0 0.0 47108 12092 ? S 19:43 0:02 | \_ /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 2022-04-03T22:52:49.304 INFO:teuthology.orchestra.run.smithi077.stdout:root 20249 0.3 0.0 0 0 ? Zsl 19:43 0:41 | \_ [ceph-osd] 2022-04-03T22:52:49.304 INFO:teuthology.orchestra.run.smithi077.stdout:root 20165 0.0 0.0 150084 8124 ? Ss 19:43 0:00 \_ sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2022-04-03T22:52:49.304 INFO:teuthology.orchestra.run.smithi077.stdout:root 20239 0.0 0.0 47108 12092 ? S 19:43 0:02 | \_ /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 2022-04-03T22:52:49.305 INFO:teuthology.orchestra.run.smithi077.stdout:root 20248 0.2 0.0 0 0 ? Zsl 19:43 0:28 | \_ [ceph-osd] 2022-04-03T22:52:49.305 INFO:teuthology.orchestra.run.smithi077.stdout:root 20172 0.0 0.0 150084 7972 ? Ss 19:43 0:00 \_ sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2022-04-03T22:52:49.306 INFO:teuthology.orchestra.run.smithi077.stdout:root 20240 0.0 0.0 47108 12308 ? S 19:43 0:03 | \_ /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 2022-04-03T22:52:49.306 INFO:teuthology.orchestra.run.smithi077.stdout:root 20250 0.3 0.0 0 0 ? Zsl 19:43 0:44 | \_ [ceph-osd] 2022-04-03T22:52:49.306 INFO:teuthology.orchestra.run.smithi077.stdout:root 25180 0.0 0.0 150084 8184 ? Ss 19:44 0:00 \_ sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-04-03T22:52:49.307 INFO:teuthology.orchestra.run.smithi077.stdout:root 25242 0.0 0.0 47108 11992 ? S 19:44 0:02 | \_ /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-04-03T22:52:49.307 INFO:teuthology.orchestra.run.smithi077.stdout:root 25254 0.0 0.1 241380 41756 ? Ssl 19:44 0:05 | \_ ceph-mds -f --cluster ceph -i a 2022-04-03T22:52:49.307 INFO:teuthology.orchestra.run.smithi077.stdout:root 25182 0.0 0.0 150084 8112 ? Ss 19:44 0:00 \_ sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-04-03T22:52:49.308 INFO:teuthology.orchestra.run.smithi077.stdout:root 25244 0.0 0.0 47108 12148 ? S 19:44 0:02 | \_ /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i c 2022-04-03T22:52:49.308 INFO:teuthology.orchestra.run.smithi077.stdout:root 25253 0.0 0.1 214748 38864 ? Ssl 19:44 0:04 | \_ ceph-mds -f --cluster ceph -i c 2022-04-03T22:52:49.308 INFO:teuthology.orchestra.run.smithi077.stdout:root 38612 0.0 0.0 150084 8184 ? Ss 19:57 0:00 \_ sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2022-04-03T22:52:49.309 INFO:teuthology.orchestra.run.smithi077.stdout:root 38644 0.0 0.0 47108 12324 ? S 19:57 0:02 | \_ /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i e 2022-04-03T22:52:49.309 INFO:teuthology.orchestra.run.smithi077.stdout:root 38649 0.0 0.1 214748 38988 ? Ssl 19:57 0:05 | \_ ceph-mds -f --cluster ceph -i e 2022-04-03T22:52:49.309 INFO:teuthology.orchestra.run.smithi077.stdout:root 39082 0.0 0.0 150084 8084 ? Ss 20:01 0:00 \_ sudo umount /home/ubuntu/cephtest/mnt.0 -f 2022-04-03T22:52:49.309 INFO:teuthology.orchestra.run.smithi077.stdout:root 39098 0.0 0.0 25300 1460 ? D 20:01 0:00 | \_ umount /home/ubuntu/cephtest/mnt.0 -f 2022-04-03T22:52:49.310 INFO:teuthology.orchestra.run.smithi077.stdout:ubuntu 48916 0.0 0.0 12724 3244 ? Ss 22:52 0:00 \_ bash -c sudo PATH=/usr/sbin:$PATH lsof ; ps auxf 2022-04-03T22:52:49.310 INFO:teuthology.orchestra.run.smithi077.stdout:ubuntu 48935 0.0 0.0 59164 4776 ? R 22:52 0:00 \_ ps auxf 2022-04-03T22:52:49.311 INFO:teuthology.orchestra.run.smithi077.stdout:root 832 0.0 0.0 41784 536 ? Ss 19:34 0:00 /usr/bin/rhsmcertd 2022-04-03T22:52:49.311 INFO:teuthology.orchestra.run.smithi077.stdout:root 838 0.0 0.0 101760 3724 ? Ssl 19:34 0:00 /usr/sbin/gssproxy -D 2022-04-03T22:52:49.311 INFO:teuthology.orchestra.run.smithi077.stdout:root 1052 0.0 0.0 105812 10228 ? Ss 19:34 0:00 /usr/lib/systemd/systemd-logind 2022-04-03T22:52:49.312 INFO:teuthology.orchestra.run.smithi077.stdout:root 1054 0.0 0.0 36952 3580 ? Ss 19:34 0:00 /usr/sbin/crond -n 2022-04-03T22:52:49.312 INFO:teuthology.orchestra.run.smithi077.stdout:root 1057 0.0 0.0 44004 2480 ? Ss 19:34 0:00 /usr/sbin/atd -f 2022-04-03T22:52:49.312 INFO:teuthology.orchestra.run.smithi077.stdout:nrpe 1105 0.0 0.0 36392 3400 ? Ss 19:34 0:00 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f -n 2022-04-03T22:52:49.313 INFO:teuthology.orchestra.run.smithi077.stdout:root 1114 0.0 0.0 13656 1800 tty1 Ss+ 19:34 0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux 2022-04-03T22:52:49.313 INFO:teuthology.orchestra.run.smithi077.stdout:root 1115 0.0 0.0 16224 1992 ttyS1 Ss+ 19:34 0:00 /sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS1 vt220 2022-04-03T22:52:49.313 INFO:teuthology.orchestra.run.smithi077.stdout:ubuntu 1130 0.0 0.0 100764 10112 ? Ss 19:35 0:00 /usr/lib/systemd/systemd --user 2022-04-03T22:52:49.313 INFO:teuthology.orchestra.run.smithi077.stdout:ubuntu 1134 0.0 0.0 182240 4944 ? S 19:35 0:00 \_ (sd-pam) 2022-04-03T22:52:49.314 INFO:teuthology.orchestra.run.smithi077.stdout:root 1366 0.0 0.0 217660 9920 ? Ssl 19:35 0:01 /usr/sbin/rsyslogd -n 2022-04-03T22:52:49.314 INFO:teuthology.orchestra.run.smithi077.stdout:chrony 7959 0.0 0.0 108124 2476 ? S 19:40 0:00 /usr/sbin/chronyd 2022-04-03T22:52:49.315 INFO:teuthology.orchestra.run.smithi077.stdout:root 8270 0.0 0.0 61672 13760 ? Ss 19:42 0:00 /usr/libexec/platform-python -s /usr/bin/ceph-crash 2022-04-03T22:52:49.315 INFO:teuthology.orchestra.run.smithi077.stdout:ubuntu 38794 0.0 0.3 400212 116664 ? D 19:59 0:00 git reset --hard v0.2 2022-04-03T22:52:49.315 INFO:teuthology.orchestra.run.smithi077.stdout:root 47303 0.0 0.0 100796 10260 ? Ds 22:26 0:00 /usr/lib/systemd/systemd --user 2022-04-03T22:52:49.315 INFO:teuthology.orchestra.run.smithi077.stdout:root 47306 0.0 0.0 0 0 ? Z 22:26 0:00 \_ [(sd-pam)] 2022-04-03T22:52:49.316 INFO:teuthology.orchestra.run.smithi077.stdout:root 47321 0.0 0.0 0 0 ? Zs 22:26 0:00 \_ [systemctl] 2022-04-03T22:52:49.316 INFO:teuthology.orchestra.run.smithi077.stdout:root 47836 0.0 0.0 100436 8296 ? Ds 22:34 0:00 /usr/lib/systemd/systemd --user 2022-04-03T22:52:49.316 INFO:teuthology.orchestra.run.smithi077.stdout:root 47838 0.0 0.0 0 0 ? Z 22:34 0:00 \_ [(sd-pam)] 2022-04-03T22:52:49.317 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2022-04-03T22:52:49.318 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2022-04-03T22:52:49.318 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-04-03T22:52:49.319 DEBUG:teuthology.orchestra.run.smithi077:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2022-04-03T22:52:49.320 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:46.053+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:46.054175+0000) 2022-04-03T22:52:49.321 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:47.053+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:47.054375+0000) 2022-04-03T22:52:49.321 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:48.053+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:48.054592+0000) 2022-04-03T22:52:49.321 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:49.054+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:49.054845+0000) 2022-04-03T22:52:49.322 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:47.356+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:49.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:48.380+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:49.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:49.406+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:50.041 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:52:50.040+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:52:50.055 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:50.054+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:50.055075+0000) 2022-04-03T22:52:50.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:50.376+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:51.056 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:51.054+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:51.055293+0000) 2022-04-03T22:52:51.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:51.377+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:52.056 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:52.054+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:52.055496+0000) 2022-04-03T22:52:52.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:52.403+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:53.056 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:53.055+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:53.055644+0000) 2022-04-03T22:52:53.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:53.360+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:54.056 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:54.055+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:54.055851+0000) 2022-04-03T22:52:54.355 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:54.354+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:55.056 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:55.055+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:55.056020+0000) 2022-04-03T22:52:55.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:55.367+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:56.057 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:56.055+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:56.056226+0000) 2022-04-03T22:52:56.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:56.339+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:57.057 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:57.055+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:57.056422+0000) 2022-04-03T22:52:57.300 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:57.299+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:58.057 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:58.056+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:58.056625+0000) 2022-04-03T22:52:58.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:58.257+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:52:59.057 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:52:59.056+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:52:59.056827+0000) 2022-04-03T22:52:59.290 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:52:59.288+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:00.057 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:00.056+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:00.056966+0000) 2022-04-03T22:53:00.318 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:00.315+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:01.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:01.056+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:01.057125+0000) 2022-04-03T22:53:01.329 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:01.327+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:02.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:02.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:02.057336+0000) 2022-04-03T22:53:02.280 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:02.277+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:03.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:03.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:03.057548+0000) 2022-04-03T22:53:03.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:03.321+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:04.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:04.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:04.057724+0000) 2022-04-03T22:53:04.372 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:04.370+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:05.057 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:53:05.056+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:53:05.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:05.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:05.057931+0000) 2022-04-03T22:53:05.331 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:05.329+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:06.059 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:06.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:06.058139+0000) 2022-04-03T22:53:06.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:06.367+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:07.059 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:07.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:07.058344+0000) 2022-04-03T22:53:07.359 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:07.357+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:07.412 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:53:07.416 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:53:08.059 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:08.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:08.058578+0000) 2022-04-03T22:53:08.325 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:08.323+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:09.059 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:09.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:09.058820+0000) 2022-04-03T22:53:09.334 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:09.331+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:10.059 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:10.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:10.058997+0000) 2022-04-03T22:53:10.342 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:10.339+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:11.060 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:11.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:11.059225+0000) 2022-04-03T22:53:11.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:11.312+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:12.060 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:12.059+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:12.059414+0000) 2022-04-03T22:53:12.289 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:12.287+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:13.060 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:13.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:13.059619+0000) 2022-04-03T22:53:13.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:13.293+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:14.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:14.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:14.059870+0000) 2022-04-03T22:53:14.266 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:14.264+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:15.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:15.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:15.060020+0000) 2022-04-03T22:53:15.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:15.293+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:16.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:16.059+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:16.060217+0000) 2022-04-03T22:53:16.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:16.257+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:17.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:17.059+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:17.060422+0000) 2022-04-03T22:53:17.239 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:17.238+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:18.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:18.059+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:18.060559+0000) 2022-04-03T22:53:18.200 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:18.198+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:19.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:19.059+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:19.060759+0000) 2022-04-03T22:53:19.196 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:19.194+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:20.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:20.060+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:20.060922+0000) 2022-04-03T22:53:20.073 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:53:20.071+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:53:20.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:20.172+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:21.062 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:21.060+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:21.061030+0000) 2022-04-03T22:53:21.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:21.157+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:22.062 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:22.060+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:22.061195+0000) 2022-04-03T22:53:22.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:22.194+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:23.062 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:23.060+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:23.061365+0000) 2022-04-03T22:53:23.172 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:23.170+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:24.062 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:24.060+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:24.061588+0000) 2022-04-03T22:53:24.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:24.157+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:25.062 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:25.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:25.061822+0000) 2022-04-03T22:53:25.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:25.165+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:26.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:26.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:26.062044+0000) 2022-04-03T22:53:26.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:26.185+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:27.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:27.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:27.062287+0000) 2022-04-03T22:53:27.140 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:27.138+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:28.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:28.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:28.062450+0000) 2022-04-03T22:53:28.129 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:28.127+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:29.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:29.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:29.062642+0000) 2022-04-03T22:53:29.155 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:29.154+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:30.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:30.062+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:30.062877+0000) 2022-04-03T22:53:30.142 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:30.141+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:31.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:31.062+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:31.063107+0000) 2022-04-03T22:53:31.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:31.157+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:32.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:32.062+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:32.063326+0000) 2022-04-03T22:53:32.133 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:32.131+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:33.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:33.062+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:33.063531+0000) 2022-04-03T22:53:33.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:33.086+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:34.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:34.063+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:34.063754+0000) 2022-04-03T22:53:34.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:34.123+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:35.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:35.063+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:35.064005+0000) 2022-04-03T22:53:35.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:35.079+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:35.089 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:53:35.087+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:53:36.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:36.060+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:36.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:36.063+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:36.064205+0000) 2022-04-03T22:53:37.065 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:37.063+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:37.064373+0000) 2022-04-03T22:53:37.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:37.081+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:37.598 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:53:37.602 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:53:38.065 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:38.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:38.064591+0000) 2022-04-03T22:53:38.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:38.125+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:39.065 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:39.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:39.064831+0000) 2022-04-03T22:53:39.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:39.087+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:40.065 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:40.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:40.065036+0000) 2022-04-03T22:53:40.100 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:40.099+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:41.066 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:41.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:41.065205+0000) 2022-04-03T22:53:41.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:41.127+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:42.066 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:42.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:42.065409+0000) 2022-04-03T22:53:42.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:42.153+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:43.066 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:43.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:43.065603+0000) 2022-04-03T22:53:43.151 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:43.150+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:44.066 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:44.065+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:44.065752+0000) 2022-04-03T22:53:44.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:44.160+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:45.066 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:45.065+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:45.065970+0000) 2022-04-03T22:53:45.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:45.160+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:46.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:46.065+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:46.066194+0000) 2022-04-03T22:53:46.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:46.202+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:47.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:47.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:47.066404+0000) 2022-04-03T22:53:47.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:47.231+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:48.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:48.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:48.066629+0000) 2022-04-03T22:53:48.236 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:48.235+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:49.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:49.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:49.066859+0000) 2022-04-03T22:53:49.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:49.249+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:50.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:50.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:50.067064+0000) 2022-04-03T22:53:50.105 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:53:50.103+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:53:50.241 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:50.240+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:51.068 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:51.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:51.067235+0000) 2022-04-03T22:53:51.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:51.243+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:52.068 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:52.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:52.067433+0000) 2022-04-03T22:53:52.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:52.234+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:53.068 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:53.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:53.067650+0000) 2022-04-03T22:53:53.262 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:53.260+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:54.068 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:54.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:54.067823+0000) 2022-04-03T22:53:54.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:54.274+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:55.068 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:55.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:55.068009+0000) 2022-04-03T22:53:55.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:55.297+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:56.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:56.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:56.068239+0000) 2022-04-03T22:53:56.313 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:56.311+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:57.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:57.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:57.068410+0000) 2022-04-03T22:53:57.271 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:57.269+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:58.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:58.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:58.068616+0000) 2022-04-03T22:53:58.313 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:58.311+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:53:59.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:53:59.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:53:59.068843+0000) 2022-04-03T22:53:59.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:53:59.334+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:00.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:00.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:00.069047+0000) 2022-04-03T22:54:00.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:00.370+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:01.070 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:01.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:01.069282+0000) 2022-04-03T22:54:01.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:01.373+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:02.070 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:02.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:02.069509+0000) 2022-04-03T22:54:02.343 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:02.341+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:03.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:03.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:03.069728+0000) 2022-04-03T22:54:03.345 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:03.344+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:04.070 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:04.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:04.069933+0000) 2022-04-03T22:54:04.366 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:04.364+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:05.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:05.069+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:05.070089+0000) 2022-04-03T22:54:05.121 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:54:05.119+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:54:05.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:05.337+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:06.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:06.069+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:06.070291+0000) 2022-04-03T22:54:06.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:06.387+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:07.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:07.069+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:07.070483+0000) 2022-04-03T22:54:07.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:07.419+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:07.735 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:54:07.740 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:54:08.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:08.069+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:08.070684+0000) 2022-04-03T22:54:08.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:08.465+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:09.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:09.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:09.070880+0000) 2022-04-03T22:54:09.504 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:09.502+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:10.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:10.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:10.071054+0000) 2022-04-03T22:54:10.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:10.549+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:11.072 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:11.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:11.071213+0000) 2022-04-03T22:54:11.556 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:11.554+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:12.072 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:12.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:12.071360+0000) 2022-04-03T22:54:12.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:12.519+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:13.072 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:13.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:13.071573+0000) 2022-04-03T22:54:13.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:13.551+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:14.072 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:14.071+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:14.071848+0000) 2022-04-03T22:54:14.525 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:14.523+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:15.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:15.071+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:15.071994+0000) 2022-04-03T22:54:15.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:15.547+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:16.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:16.071+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:16.072190+0000) 2022-04-03T22:54:16.557 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:16.555+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:17.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:17.071+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:17.072388+0000) 2022-04-03T22:54:17.563 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:17.561+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:18.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:18.071+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:18.072598+0000) 2022-04-03T22:54:18.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:18.594+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:19.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:19.072+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:19.072836+0000) 2022-04-03T22:54:19.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:19.574+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:20.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:20.072+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:20.073003+0000) 2022-04-03T22:54:20.137 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:54:20.135+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:54:20.566 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:20.564+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:21.074 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:21.072+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:21.073210+0000) 2022-04-03T22:54:21.590 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:21.588+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:22.074 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:22.072+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:22.073397+0000) 2022-04-03T22:54:22.409 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:22.407+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:22.586 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:22.584+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:23.074 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:23.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:23.073594+0000) 2022-04-03T22:54:23.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:23.382+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:23.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:23.551+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:24.074 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:24.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:24.073824+0000) 2022-04-03T22:54:24.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:24.403+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:24.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:24.540+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:25.074 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:25.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:25.074006+0000) 2022-04-03T22:54:25.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:25.370+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:25.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:25.492+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:26.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:26.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:26.074209+0000) 2022-04-03T22:54:26.388 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:26.387+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:26.530 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:26.529+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:27.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:27.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:27.074371+0000) 2022-04-03T22:54:27.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:27.352+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:27.519 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:27.517+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:28.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:28.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:28.074557+0000) 2022-04-03T22:54:28.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:28.328+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:28.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:28.547+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:29.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:29.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:29.074758+0000) 2022-04-03T22:54:29.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:29.297+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:29.533 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:29.531+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:30.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:30.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:30.074940+0000) 2022-04-03T22:54:30.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:30.340+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:30.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:30.519+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:31.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:31.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:31.075106+0000) 2022-04-03T22:54:31.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:31.328+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:31.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:31.472+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:32.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:32.075+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:32.075330+0000) 2022-04-03T22:54:32.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:32.335+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:32.433 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:32.432+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:33.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:33.075+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:33.075524+0000) 2022-04-03T22:54:33.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:33.309+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:33.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:33.454+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:34.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:34.075+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:34.075724+0000) 2022-04-03T22:54:34.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:34.297+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:34.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:34.434+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:35.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:35.075+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:35.075908+0000) 2022-04-03T22:54:35.152 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:54:35.151+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:54:35.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:35.347+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:35.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:35.426+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:36.077 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:36.075+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:36.076120+0000) 2022-04-03T22:54:36.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:36.366+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:36.384 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:36.383+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:37.077 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:37.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:37.076294+0000) 2022-04-03T22:54:37.356 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:37.354+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:37.364 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:37.363+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:37.816 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:54:37.820 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:54:38.077 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:38.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:38.076555+0000) 2022-04-03T22:54:38.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:38.394+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:38.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:38.410+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:39.077 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:39.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:39.076808+0000) 2022-04-03T22:54:39.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:39.372+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:39.387 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:39.385+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:40.077 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:40.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:40.077024+0000) 2022-04-03T22:54:40.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:40.353+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:40.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:40.382+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:41.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:41.077+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:41.077181+0000) 2022-04-03T22:54:41.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:41.391+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:41.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:41.422+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:42.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:42.077+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:42.077339+0000) 2022-04-03T22:54:42.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:42.365+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:42.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:42.392+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:43.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:43.075+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:43.077560+0000) 2022-04-03T22:54:43.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:43.320+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:43.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:43.415+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:44.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:44.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:44.077693+0000) 2022-04-03T22:54:44.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:44.321+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:44.428 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:44.427+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:45.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:45.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:45.077883+0000) 2022-04-03T22:54:45.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:45.361+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:45.466 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:45.464+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:46.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:46.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:46.077980+0000) 2022-04-03T22:54:46.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:46.329+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:46.465 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:46.464+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:47.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:47.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:47.078085+0000) 2022-04-03T22:54:47.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:47.318+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:47.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:47.480+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:48.079 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:48.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:48.078223+0000) 2022-04-03T22:54:48.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:48.349+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:48.517 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:48.515+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:49.079 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:49.077+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:49.078392+0000) 2022-04-03T22:54:49.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:49.337+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:49.498 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:49.496+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:50.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:50.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:50.078562+0000) 2022-04-03T22:54:50.164 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:54:50.161+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:54:50.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:50.321+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:50.487 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:50.485+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:51.079 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:51.077+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:51.078730+0000) 2022-04-03T22:54:51.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:51.308+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:51.519 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:51.517+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:52.079 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:52.077+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:52.078872+0000) 2022-04-03T22:54:52.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:52.267+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:52.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:52.481+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:53.079 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:53.078+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:53.079018+0000) 2022-04-03T22:54:53.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:53.261+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:53.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:53.489+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:54.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:54.078+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:54.079187+0000) 2022-04-03T22:54:54.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:54.217+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:54.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:54.454+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:55.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:55.078+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:55.079339+0000) 2022-04-03T22:54:55.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:55.182+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:55.480 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:55.477+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:56.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:56.078+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:56.079472+0000) 2022-04-03T22:54:56.152 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:56.150+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:56.478 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:56.476+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:57.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:57.078+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:57.079611+0000) 2022-04-03T22:54:57.131 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:57.129+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:57.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:57.526+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:58.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:58.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:58.079747+0000) 2022-04-03T22:54:58.131 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:58.130+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:58.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:58.484+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:54:59.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:54:59.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:54:59.079885+0000) 2022-04-03T22:54:59.099 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:54:59.097+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:54:59.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:54:59.529+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:00.058 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:00.056+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:00.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:00.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:00.079987+0000) 2022-04-03T22:55:00.573 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:00.571+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:01.019 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:01.017+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:01.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:01.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:01.080140+0000) 2022-04-03T22:55:01.540 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:01.538+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:02.058 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:02.056+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:02.081 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:02.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:02.080314+0000) 2022-04-03T22:55:02.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:02.489+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:03.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:03.014+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:03.081 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:03.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:03.080522+0000) 2022-04-03T22:55:03.470 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:03.468+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:04.027 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:04.025+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:04.081 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:04.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:04.080746+0000) 2022-04-03T22:55:04.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:04.516+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:05.061 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:05.059+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:05.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:05.080+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:05.080971+0000) 2022-04-03T22:55:05.180 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:55:05.178+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:55:05.562 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:05.560+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:06.061 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:06.059+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:06.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:06.080+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:06.081158+0000) 2022-04-03T22:55:06.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:06.532+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:07.031 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:07.029+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:07.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:07.080+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:07.081364+0000) 2022-04-03T22:55:07.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:07.533+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:07.996 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:55:08.001 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:55:08.077 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:08.075+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:08.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:08.080+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:08.081640+0000) 2022-04-03T22:55:08.498 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:08.496+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:09.042 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:09.040+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:09.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:09.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:09.081824+0000) 2022-04-03T22:55:09.519 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:09.518+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:10.061 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:10.059+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:10.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:10.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:10.082010+0000) 2022-04-03T22:55:10.520 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:10.518+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:11.013 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:11.012+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:11.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:11.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:11.082215+0000) 2022-04-03T22:55:11.569 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:11.567+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:11.977 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:11.974+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:12.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:12.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:12.082371+0000) 2022-04-03T22:55:12.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:12.558+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:12.944 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:12.942+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:13.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:13.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:13.082546+0000) 2022-04-03T22:55:13.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:13.537+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:13.929 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:13.927+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:14.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:14.082+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:14.082755+0000) 2022-04-03T22:55:14.585 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:14.583+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:14.895 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:14.893+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:15.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:15.082+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:15.082942+0000) 2022-04-03T22:55:15.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:15.579+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:15.865 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:15.863+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:16.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:16.082+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:16.083112+0000) 2022-04-03T22:55:16.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:16.621+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:16.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:16.818+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:17.084 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:17.082+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:17.083314+0000) 2022-04-03T22:55:17.653 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:17.652+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:17.835 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:17.833+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:18.084 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:18.083+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:18.083558+0000) 2022-04-03T22:55:18.634 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:18.632+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:18.825 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:18.823+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:19.084 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:19.083+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:19.083742+0000) 2022-04-03T22:55:19.683 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:19.681+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:19.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:19.841+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:20.084 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:20.083+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:20.083978+0000) 2022-04-03T22:55:20.195 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:55:20.194+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:55:20.708 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:20.706+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:20.853 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:20.852+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:21.085 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:21.083+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:21.084176+0000) 2022-04-03T22:55:21.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:21.724+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:21.828 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:21.827+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:22.085 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:22.083+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:22.084398+0000) 2022-04-03T22:55:22.705 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:22.703+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:22.804 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:22.802+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:23.085 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:23.084+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:23.084589+0000) 2022-04-03T22:55:23.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:23.690+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:23.839 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:23.837+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:24.085 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:24.084+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:24.084825+0000) 2022-04-03T22:55:24.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:24.654+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:24.874 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:24.871+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:25.085 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:25.084+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:25.085027+0000) 2022-04-03T22:55:25.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:25.678+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:25.883 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:25.881+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:26.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:26.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:26.085231+0000) 2022-04-03T22:55:26.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:26.723+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:26.919 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:26.917+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:27.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:27.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:27.085431+0000) 2022-04-03T22:55:27.756 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:27.755+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:27.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:27.888+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:28.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:28.084+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:28.085611+0000) 2022-04-03T22:55:28.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:28.734+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:28.886 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:28.884+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:29.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:29.084+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:29.085834+0000) 2022-04-03T22:55:29.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:29.728+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:29.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:29.867+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:30.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:30.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:30.086038+0000) 2022-04-03T22:55:30.685 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:30.684+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:30.898 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:30.897+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:31.087 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:31.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:31.086265+0000) 2022-04-03T22:55:31.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:31.725+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:31.883 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:31.882+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:32.087 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:32.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:32.086478+0000) 2022-04-03T22:55:32.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:32.743+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:32.852 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:32.851+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:33.087 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:33.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:33.086685+0000) 2022-04-03T22:55:33.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:33.758+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:33.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:33.832+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:34.087 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:34.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:34.086860+0000) 2022-04-03T22:55:34.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:34.776+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:34.788 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:34.787+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:35.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:35.086+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:35.087044+0000) 2022-04-03T22:55:35.211 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:55:35.209+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:55:35.784 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:35.783+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:35.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:35.784+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:36.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:36.086+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:36.087217+0000) 2022-04-03T22:55:36.757 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:36.756+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:36.794 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:36.793+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:37.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:37.086+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:37.087394+0000) 2022-04-03T22:55:37.718 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:37.717+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:37.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:37.801+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:38.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:38.086+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:38.087552+0000) 2022-04-03T22:55:38.143 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:55:38.147 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:55:38.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:38.679+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:38.789 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:38.787+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:39.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:39.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:39.087801+0000) 2022-04-03T22:55:39.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:39.709+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:39.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:39.769+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:40.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:40.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:40.087984+0000) 2022-04-03T22:55:40.681 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:40.680+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:40.768 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:40.767+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:41.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:41.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:41.088182+0000) 2022-04-03T22:55:41.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:41.728+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:41.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:41.812+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:42.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:42.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:42.088395+0000) 2022-04-03T22:55:42.747 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:42.746+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:42.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:42.821+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:43.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:43.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:43.088581+0000) 2022-04-03T22:55:43.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:43.735+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:43.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:43.860+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:44.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:44.088+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:44.088757+0000) 2022-04-03T22:55:44.787 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:44.785+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:44.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:44.820+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:45.090 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:45.088+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:45.088951+0000) 2022-04-03T22:55:45.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:45.781+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:45.819 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:45.817+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:46.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:46.088+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:46.089155+0000) 2022-04-03T22:55:46.816 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:46.814+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:46.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:46.822+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:47.090 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:47.088+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:47.089353+0000) 2022-04-03T22:55:47.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:47.777+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:47.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:47.817+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:48.090 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:48.088+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:48.089538+0000) 2022-04-03T22:55:48.797 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:48.795+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:48.854 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:48.852+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:49.090 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:49.089+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:49.089712+0000) 2022-04-03T22:55:49.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:49.749+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:49.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:49.819+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:50.090 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:50.089+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:50.089911+0000) 2022-04-03T22:55:50.227 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:55:50.225+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:55:50.776 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:50.774+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:50.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:50.799+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:51.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:51.089+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:51.090150+0000) 2022-04-03T22:55:51.739 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:51.738+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:51.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:51.825+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:52.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:52.089+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:52.090308+0000) 2022-04-03T22:55:52.761 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:52.758+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:52.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:52.827+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:53.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:53.089+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:53.090519+0000) 2022-04-03T22:55:53.810 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:53.808+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:53.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:53.861+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:54.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:54.090+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:54.090733+0000) 2022-04-03T22:55:54.817 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:54.815+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:54.908 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:54.906+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:55.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:55.090+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:55.090970+0000) 2022-04-03T22:55:55.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:55.840+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:55.869 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:55.867+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:56.092 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:56.090+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:56.091211+0000) 2022-04-03T22:55:56.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:56.846+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:56.888 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:56.886+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:57.092 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:57.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:57.091462+0000) 2022-04-03T22:55:57.884 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:57.882+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:57.927 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:57.925+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:58.092 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:58.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:58.091702+0000) 2022-04-03T22:55:58.837 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:58.835+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:58.891 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:58.889+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:55:59.092 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:55:59.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:55:59.091947+0000) 2022-04-03T22:55:59.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:55:59.813+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:55:59.902 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:55:59.900+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:00.093 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:00.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:00.092192+0000) 2022-04-03T22:56:00.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:00.819+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:00.952 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:00.950+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:01.093 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:01.092+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:01.092443+0000) 2022-04-03T22:56:01.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:01.860+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:01.985 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:01.983+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:02.093 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:02.092+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:02.092688+0000) 2022-04-03T22:56:02.817 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:02.815+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:03.013 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:03.011+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:03.093 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:03.092+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:03.092927+0000) 2022-04-03T22:56:03.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:03.816+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:04.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:04.002+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:04.094 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:04.092+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:04.093172+0000) 2022-04-03T22:56:04.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:04.767+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:04.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:04.977+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:05.094 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:05.093+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:05.093418+0000) 2022-04-03T22:56:05.243 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:56:05.241+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:56:05.790 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:05.787+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:06.002 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:06.000+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:06.094 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:06.093+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:06.093610+0000) 2022-04-03T22:56:06.774 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:06.772+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:07.022 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:07.020+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:07.094 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:07.093+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:07.093856+0000) 2022-04-03T22:56:07.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:07.816+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:08.064 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:08.060+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:08.094 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:08.093+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:08.094094+0000) 2022-04-03T22:56:08.281 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:56:08.285 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:56:08.809 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:08.807+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:09.031 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:09.030+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:09.095 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:09.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:09.094324+0000) 2022-04-03T22:56:09.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:09.782+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:10.035 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:10.033+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:10.095 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:10.093+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:10.094583+0000) 2022-04-03T22:56:10.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:10.758+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:11.049 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:11.047+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:11.095 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:11.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:11.094862+0000) 2022-04-03T22:56:11.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:11.774+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:12.083 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:12.081+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:12.095 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:12.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:12.095104+0000) 2022-04-03T22:56:12.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:12.726+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:13.096 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:13.093+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:13.095221+0000) 2022-04-03T22:56:13.107 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:13.105+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:13.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:13.758+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:14.080 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:14.078+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:14.096 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:14.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:14.095436+0000) 2022-04-03T22:56:14.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:14.721+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:15.096 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:15.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:15.095683+0000) 2022-04-03T22:56:15.116 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:15.114+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:15.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:15.758+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:16.096 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:16.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:16.095887+0000) 2022-04-03T22:56:16.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:16.151+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:16.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:16.807+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:17.096 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:17.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:17.096073+0000) 2022-04-03T22:56:17.182 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:17.180+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:17.771 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:17.770+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:18.097 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:18.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:18.096285+0000) 2022-04-03T22:56:18.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:18.227+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:18.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:18.721+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:19.097 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:19.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:19.096530+0000) 2022-04-03T22:56:19.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:19.187+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:19.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:19.678+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:20.097 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:20.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:20.096801+0000) 2022-04-03T22:56:20.213 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:20.211+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:20.247 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:56:20.245+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:56:20.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:20.668+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:21.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:21.096+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:21.097058+0000) 2022-04-03T22:56:21.221 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:21.219+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:21.645 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:21.644+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:22.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:22.096+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:22.097263+0000) 2022-04-03T22:56:22.206 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:22.204+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:22.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:22.609+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:23.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:23.096+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:23.097495+0000) 2022-04-03T22:56:23.162 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:23.161+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:23.654 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:23.653+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:24.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:24.096+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:24.097740+0000) 2022-04-03T22:56:24.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:24.210+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:24.684 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:24.683+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:25.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:25.097+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:25.097977+0000) 2022-04-03T22:56:25.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:25.206+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:25.681 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:25.680+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:26.099 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:26.097+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:26.098218+0000) 2022-04-03T22:56:26.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:26.237+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:26.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:26.725+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:27.099 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:27.097+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:27.098461+0000) 2022-04-03T22:56:27.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:27.208+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:27.692 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:27.691+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:28.099 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:28.097+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:28.098706+0000) 2022-04-03T22:56:28.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:28.200+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:28.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:28.678+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:29.099 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:29.098+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:29.098881+0000) 2022-04-03T22:56:29.176 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:29.175+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:29.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:29.666+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:30.099 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:30.098+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:30.099056+0000) 2022-04-03T22:56:30.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:30.147+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:30.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:30.637+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:31.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:31.098+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:31.099245+0000) 2022-04-03T22:56:31.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:31.167+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:31.631 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:31.630+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:32.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:32.098+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:32.099447+0000) 2022-04-03T22:56:32.126 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:32.125+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:32.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:32.669+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:33.077 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:33.076+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:33.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:33.099+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:33.099664+0000) 2022-04-03T22:56:33.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:33.675+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:34.070 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:34.069+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:34.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:34.099+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:34.099815+0000) 2022-04-03T22:56:34.720 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:34.719+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:35.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:35.099+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:35.100008+0000) 2022-04-03T22:56:35.102 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:35.101+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:35.263 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:56:35.261+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:56:35.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:35.686+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:36.077 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:36.076+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:36.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:36.099+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:36.100224+0000) 2022-04-03T22:56:36.649 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:36.648+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:37.097 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:37.096+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:37.101 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:37.099+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:37.100468+0000) 2022-04-03T22:56:37.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:37.634+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:38.087 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:38.085+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:38.101 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:38.100+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:38.100717+0000) 2022-04-03T22:56:38.396 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:56:38.400 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:56:38.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:38.585+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:39.041 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:39.039+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:39.101 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:39.100+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:39.100986+0000) 2022-04-03T22:56:39.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:39.577+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:40.039 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:40.037+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:40.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:40.100+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:40.101240+0000) 2022-04-03T22:56:40.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:40.621+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:41.087 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:41.085+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:41.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:41.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:41.101490+0000) 2022-04-03T22:56:41.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:41.665+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:42.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:42.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:42.101732+0000) 2022-04-03T22:56:42.103 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:42.100+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:42.686 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:42.684+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:43.056 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:43.054+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:43.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:43.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:43.101913+0000) 2022-04-03T22:56:43.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:43.689+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:44.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:44.083+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:44.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:44.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:44.102127+0000) 2022-04-03T22:56:44.651 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:44.649+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:45.058 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:45.056+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:45.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:45.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:45.102331+0000) 2022-04-03T22:56:45.606 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:45.604+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:46.018 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:46.017+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:46.103 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:46.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:46.102456+0000) 2022-04-03T22:56:46.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:46.614+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:46.998 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:46.996+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:47.103 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:47.102+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:47.102695+0000) 2022-04-03T22:56:47.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:47.640+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:48.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:48.036+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:48.103 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:48.102+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:48.102841+0000) 2022-04-03T22:56:48.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:48.630+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:49.079 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:49.077+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:49.103 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:49.102+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:49.103017+0000) 2022-04-03T22:56:49.652 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:49.650+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:50.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:50.071+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:50.103 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:50.102+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:50.103259+0000) 2022-04-03T22:56:50.279 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:56:50.277+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:56:50.633 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:50.631+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:51.051 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:51.049+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:51.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:51.103+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:51.103446+0000) 2022-04-03T22:56:51.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:51.675+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:52.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:52.066+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:52.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:52.103+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:52.103636+0000) 2022-04-03T22:56:52.672 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:52.671+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:53.080 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:53.078+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:53.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:53.103+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:53.103869+0000) 2022-04-03T22:56:53.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:53.688+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:54.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:54.103+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:54.104061+0000) 2022-04-03T22:56:54.120 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:54.118+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:54.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:54.712+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:55.105 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:55.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:55.104310+0000) 2022-04-03T22:56:55.119 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:55.118+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:55.764 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:55.762+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:56.105 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:56.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:56.104515+0000) 2022-04-03T22:56:56.128 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:56.127+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:56.790 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:56.788+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:57.105 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:57.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:57.104681+0000) 2022-04-03T22:56:57.146 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:57.144+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:57.814 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:57.812+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:58.105 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:58.103+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:58.104933+0000) 2022-04-03T22:56:58.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:58.114+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:58.850 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:58.848+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:56:59.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:56:59.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:56:59.105172+0000) 2022-04-03T22:56:59.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:56:59.160+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:56:59.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:56:59.856+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:00.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:00.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:00.105395+0000) 2022-04-03T22:57:00.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:00.184+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:00.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:00.811+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:01.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:01.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:01.105637+0000) 2022-04-03T22:57:01.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:01.206+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:01.777 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:01.775+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:02.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:02.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:02.105828+0000) 2022-04-03T22:57:02.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:02.256+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:02.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:02.773+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:03.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:03.105+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:03.105994+0000) 2022-04-03T22:57:03.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:03.276+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:03.763 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:03.761+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:04.107 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:04.105+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:04.106163+0000) 2022-04-03T22:57:04.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:04.239+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:04.777 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:04.776+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:05.107 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:05.105+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:05.106374+0000) 2022-04-03T22:57:05.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:05.221+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:05.295 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:57:05.292+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:57:05.763 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:05.762+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:06.107 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:06.105+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:06.106597+0000) 2022-04-03T22:57:06.182 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:06.180+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:06.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:06.725+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:07.107 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:07.105+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:07.106865+0000) 2022-04-03T22:57:07.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:07.206+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:07.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:07.743+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:08.107 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:08.106+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:08.107112+0000) 2022-04-03T22:57:08.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:08.216+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:08.528 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:57:08.532 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:57:08.792 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:08.791+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:09.108 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:09.106+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:09.107341+0000) 2022-04-03T22:57:09.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:09.183+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:09.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:09.810+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:10.108 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:10.106+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:10.107507+0000) 2022-04-03T22:57:10.205 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:10.204+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:10.764 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:10.762+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:11.108 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:11.106+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:11.107732+0000) 2022-04-03T22:57:11.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:11.246+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:11.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:11.782+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:12.108 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:12.107+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:12.107971+0000) 2022-04-03T22:57:12.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:12.267+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:12.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:12.789+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:13.109 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:13.107+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:13.108186+0000) 2022-04-03T22:57:13.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:13.273+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:13.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:13.770+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:14.109 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:14.107+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:14.108391+0000) 2022-04-03T22:57:14.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:14.276+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:14.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:14.743+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:15.109 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:15.107+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:15.108589+0000) 2022-04-03T22:57:15.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:15.252+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:15.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:15.741+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:16.109 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:16.108+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:16.108833+0000) 2022-04-03T22:57:16.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:16.294+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:16.721 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:16.720+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:17.109 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:17.108+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:17.108984+0000) 2022-04-03T22:57:17.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:17.246+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:17.740 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:17.739+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:18.110 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:18.108+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:18.109193+0000) 2022-04-03T22:57:18.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:18.271+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:18.719 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:18.718+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:19.110 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:19.108+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:19.109439+0000) 2022-04-03T22:57:19.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:19.264+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:19.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:19.726+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:20.110 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:20.109+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:20.109610+0000) 2022-04-03T22:57:20.212 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-03T22:57:20.210+0000 7f2fd6e1b700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T22:57:20.213 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-03T22:57:20.210+0000 7f2fd661a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T22:57:20.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:20.253+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:20.310 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:57:20.309+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:57:20.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:20.744+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:21.110 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:21.109+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:21.109859+0000) 2022-04-03T22:57:21.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:21.244+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:21.721 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:21.720+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:22.110 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:22.109+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:22.110051+0000) 2022-04-03T22:57:22.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:22.254+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:22.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:22.768+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:23.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:23.109+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:23.110263+0000) 2022-04-03T22:57:23.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:23.280+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:23.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:23.774+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:24.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:24.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:24.110519+0000) 2022-04-03T22:57:24.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:24.316+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:24.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:24.743+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:25.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:25.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:25.110683+0000) 2022-04-03T22:57:25.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:25.356+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:25.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:25.732+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:26.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:26.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:26.110898+0000) 2022-04-03T22:57:26.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:26.379+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:26.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:26.752+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:27.112 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:27.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:27.111096+0000) 2022-04-03T22:57:27.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:27.413+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:27.718 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:27.717+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:28.112 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:28.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:28.111279+0000) 2022-04-03T22:57:28.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:28.448+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:28.721 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:28.720+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:29.112 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:29.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:29.111491+0000) 2022-04-03T22:57:29.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:29.475+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:29.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:29.742+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:30.112 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:30.111+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:30.111633+0000) 2022-04-03T22:57:30.432 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:30.431+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:30.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:30.777+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:31.112 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:31.111+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:31.111869+0000) 2022-04-03T22:57:31.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:31.455+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:31.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:31.802+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:32.112 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:32.111+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:32.112099+0000) 2022-04-03T22:57:32.413 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:32.412+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:32.852 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:32.851+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:33.113 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:33.111+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:33.112313+0000) 2022-04-03T22:57:33.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:33.409+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:33.841 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:33.839+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:34.113 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:34.112+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:34.112559+0000) 2022-04-03T22:57:34.437 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:34.435+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:34.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:34.884+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:35.113 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:35.112+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:35.112837+0000) 2022-04-03T22:57:35.326 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:57:35.324+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:57:35.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:35.409+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:35.932 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:35.930+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:36.114 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:36.112+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:36.113072+0000) 2022-04-03T22:57:36.423 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:36.421+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:36.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:36.898+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:37.114 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:37.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:37.113317+0000) 2022-04-03T22:57:37.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:37.379+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:37.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:37.867+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:38.114 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:38.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:38.113556+0000) 2022-04-03T22:57:38.405 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:38.403+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:38.674 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:57:38.678 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:57:38.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:38.827+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:39.114 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:39.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:39.113791+0000) 2022-04-03T22:57:39.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:39.406+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:39.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:39.818+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:40.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:40.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:40.114069+0000) 2022-04-03T22:57:40.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:40.365+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:40.854 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:40.852+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:41.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:41.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:41.114254+0000) 2022-04-03T22:57:41.337 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:41.335+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:41.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:41.845+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:42.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:42.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:42.114469+0000) 2022-04-03T22:57:42.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:42.340+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:42.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:42.813+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:43.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:43.112+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:43.114650+0000) 2022-04-03T22:57:43.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:43.306+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:43.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:43.858+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:44.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:44.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:44.114873+0000) 2022-04-03T22:57:44.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:44.302+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:44.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:44.866+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:45.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:45.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:45.115111+0000) 2022-04-03T22:57:45.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:45.336+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:45.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:45.872+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:46.116 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:46.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:46.115280+0000) 2022-04-03T22:57:46.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:46.289+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:46.919 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:46.918+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:47.116 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:47.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:47.115496+0000) 2022-04-03T22:57:47.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:47.311+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:47.969 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:47.967+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:48.116 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:48.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:48.115734+0000) 2022-04-03T22:57:48.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:48.314+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:48.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:48.963+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:49.116 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:49.115+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:49.115949+0000) 2022-04-03T22:57:49.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:49.364+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:49.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:49.982+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:50.116 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:50.115+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:50.116174+0000) 2022-04-03T22:57:50.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:50.338+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:50.342 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:57:50.340+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:57:50.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:50.978+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:51.117 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:51.115+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:51.116438+0000) 2022-04-03T22:57:51.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:51.349+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:51.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:51.963+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:52.117 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:52.115+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:52.116681+0000) 2022-04-03T22:57:52.332 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:52.330+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:52.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:52.936+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:53.117 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:53.116+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:53.116920+0000) 2022-04-03T22:57:53.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:53.340+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:53.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:53.940+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:54.118 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:54.116+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:54.117161+0000) 2022-04-03T22:57:54.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:54.294+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:54.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:54.920+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:55.118 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:55.116+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:55.117429+0000) 2022-04-03T22:57:55.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:55.287+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:55.956 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:55.954+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:56.118 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:56.116+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:56.117624+0000) 2022-04-03T22:57:56.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:56.325+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:56.980 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:56.978+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:57.118 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:57.117+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:57.117873+0000) 2022-04-03T22:57:57.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:57.341+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:57.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:57.990+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:58.118 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:58.117+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:58.118108+0000) 2022-04-03T22:57:58.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:58.351+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:57:58.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:57:58.980+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:57:59.119 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:57:59.117+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:57:59.118355+0000) 2022-04-03T22:57:59.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:57:59.344+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:00.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:00.001+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:00.119 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:00.117+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:00.118569+0000) 2022-04-03T22:58:00.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:00.387+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:01.000 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:00.998+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:01.119 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:01.118+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:01.118800+0000) 2022-04-03T22:58:01.437 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:01.436+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:02.040 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:02.039+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:02.119 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:02.118+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:02.118948+0000) 2022-04-03T22:58:02.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:02.432+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:03.018 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:03.016+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:03.119 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:03.118+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:03.119131+0000) 2022-04-03T22:58:03.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:03.389+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:04.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:04.040+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:04.120 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:04.118+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:04.119350+0000) 2022-04-03T22:58:04.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:04.369+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:05.052 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:05.051+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:05.120 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:05.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:05.119614+0000) 2022-04-03T22:58:05.358 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:58:05.356+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:58:05.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:05.357+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:06.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:06.044+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:06.120 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:06.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:06.119867+0000) 2022-04-03T22:58:06.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:06.325+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:07.092 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:07.090+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:07.120 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:07.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:07.120143+0000) 2022-04-03T22:58:07.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:07.281+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:08.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:08.086+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:08.121 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:08.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:08.120409+0000) 2022-04-03T22:58:08.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:08.299+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:08.812 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:58:08.817 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:58:09.086 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:09.085+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:09.121 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:09.120+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:09.120638+0000) 2022-04-03T22:58:09.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:09.324+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:10.052 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:10.050+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:10.121 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:10.120+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:10.120876+0000) 2022-04-03T22:58:10.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:10.306+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:11.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:11.007+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:11.121 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:11.120+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:11.121107+0000) 2022-04-03T22:58:11.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:11.301+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:12.053 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:12.051+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:12.122 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:12.120+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:12.121280+0000) 2022-04-03T22:58:12.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:12.277+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:13.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:13.087+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:13.122 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:13.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:13.121486+0000) 2022-04-03T22:58:13.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:13.250+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:14.122 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:14.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:14.121670+0000) 2022-04-03T22:58:14.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:14.123+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:14.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:14.236+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:15.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:15.079+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:15.122 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:15.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:15.121843+0000) 2022-04-03T22:58:15.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:15.266+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:16.087 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:16.086+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:16.122 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:16.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:16.122023+0000) 2022-04-03T22:58:16.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:16.255+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:17.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:17.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:17.122259+0000) 2022-04-03T22:58:17.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:17.122+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:17.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:17.303+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:18.084 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:18.082+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:18.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:18.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:18.122436+0000) 2022-04-03T22:58:18.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:18.263+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:19.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:19.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:19.122602+0000) 2022-04-03T22:58:19.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:19.126+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:19.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:19.257+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:20.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:20.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:20.122807+0000) 2022-04-03T22:58:20.138 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:20.137+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:20.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:20.224+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:20.374 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:58:20.372+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:58:21.106 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:21.105+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:21.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:21.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:21.123035+0000) 2022-04-03T22:58:21.202 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:21.201+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:22.124 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:22.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:22.123255+0000) 2022-04-03T22:58:22.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:22.134+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:22.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:22.174+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:23.124 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:23.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:23.123504+0000) 2022-04-03T22:58:23.155 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:23.154+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:23.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:23.221+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:24.124 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:24.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:24.123709+0000) 2022-04-03T22:58:24.155 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:24.154+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:24.205 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:24.204+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:25.124 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:25.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:25.123869+0000) 2022-04-03T22:58:25.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:25.132+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:25.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:25.179+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:26.124 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:26.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:26.124081+0000) 2022-04-03T22:58:26.172 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:26.171+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:26.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:26.191+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:27.125 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:27.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:27.124299+0000) 2022-04-03T22:58:27.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:27.197+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:27.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:27.231+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:28.125 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:28.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:28.124487+0000) 2022-04-03T22:58:28.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:28.242+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:28.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:28.246+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:29.125 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:29.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:29.124695+0000) 2022-04-03T22:58:29.200 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:29.198+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:29.212 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:29.210+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:30.125 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:30.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:30.124829+0000) 2022-04-03T22:58:30.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:30.198+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:30.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:30.216+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:31.125 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:31.124+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:31.125082+0000) 2022-04-03T22:58:31.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:31.160+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:31.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:31.169+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:32.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:32.124+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:32.125337+0000) 2022-04-03T22:58:32.127 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:32.124+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:32.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:32.165+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:33.117 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:33.114+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:33.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:33.124+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:33.125548+0000) 2022-04-03T22:58:33.190 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:33.188+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:34.111 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:34.109+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:34.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:34.124+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:34.125825+0000) 2022-04-03T22:58:34.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:34.162+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:35.111 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:35.109+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:35.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:35.125+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:35.126025+0000) 2022-04-03T22:58:35.160 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:35.158+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:35.390 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:58:35.388+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:58:36.127 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:36.125+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:36.126242+0000) 2022-04-03T22:58:36.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:36.145+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:36.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:36.146+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:37.127 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:37.125+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:37.126454+0000) 2022-04-03T22:58:37.137 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:37.136+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:37.142 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:37.140+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:38.106 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:38.104+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:38.127 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:38.125+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:38.126695+0000) 2022-04-03T22:58:38.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:38.159+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:38.950 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:58:38.954 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:58:39.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:39.121+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:39.127 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:39.126+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:39.126960+0000) 2022-04-03T22:58:39.129 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:39.127+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:40.128 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:40.126+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:40.127176+0000) 2022-04-03T22:58:40.134 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:40.132+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:40.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:40.159+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:41.092 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:41.089+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:41.128 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:41.126+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:41.127466+0000) 2022-04-03T22:58:41.148 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:41.147+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:42.089 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:42.087+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:42.128 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:42.126+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:42.127741+0000) 2022-04-03T22:58:42.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:42.168+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:43.128 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:43.127+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:43.127860+0000) 2022-04-03T22:58:43.131 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:43.130+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:43.196 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:43.194+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:44.118 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:44.116+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:44.128 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:44.127+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:44.128043+0000) 2022-04-03T22:58:44.168 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:44.166+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:45.128 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:45.127+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:45.128180+0000) 2022-04-03T22:58:45.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:45.162+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:45.172 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:45.170+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:46.123 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:46.121+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:46.129 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:46.127+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:46.128371+0000) 2022-04-03T22:58:46.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:46.154+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:47.129 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:47.127+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:47.128563+0000) 2022-04-03T22:58:47.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:47.167+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:47.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:47.168+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:48.129 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:48.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:48.128733+0000) 2022-04-03T22:58:48.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:48.165+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:48.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:48.197+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:49.129 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:49.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:49.128908+0000) 2022-04-03T22:58:49.162 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:49.160+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:49.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:49.208+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:50.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:50.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:50.129122+0000) 2022-04-03T22:58:50.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:50.133+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:50.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:50.253+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:50.405 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:58:50.404+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:58:51.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:51.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:51.129384+0000) 2022-04-03T22:58:51.140 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:51.139+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:51.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:51.298+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:52.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:52.129+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:52.129598+0000) 2022-04-03T22:58:52.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:52.134+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:52.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:52.254+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:53.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:53.129+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:53.129824+0000) 2022-04-03T22:58:53.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:53.180+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:53.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:53.296+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:54.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:54.129+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:54.130062+0000) 2022-04-03T22:58:54.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:54.152+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:54.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:54.247+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:55.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:55.129+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:55.130266+0000) 2022-04-03T22:58:55.204 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:55.202+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:55.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:55.240+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:56.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:56.130+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:56.130473+0000) 2022-04-03T22:58:56.199 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:56.197+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:56.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:56.255+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:57.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:57.130+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:57.130663+0000) 2022-04-03T22:58:57.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:57.187+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:57.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:57.229+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:58.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:58.130+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:58.130878+0000) 2022-04-03T22:58:58.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:58.224+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:58.271 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:58.270+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:58:59.132 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:58:59.130+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:58:59.131103+0000) 2022-04-03T22:58:59.237 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:58:59.235+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:58:59.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:58:59.240+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:00.132 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:00.130+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:00.131314+0000) 2022-04-03T22:59:00.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:00.264+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:00.267 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:00.266+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:01.132 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:01.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:01.131485+0000) 2022-04-03T22:59:01.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:01.268+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:01.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:01.283+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:02.132 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:02.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:02.131692+0000) 2022-04-03T22:59:02.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:02.310+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:02.312 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:02.310+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:03.132 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:03.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:03.131896+0000) 2022-04-03T22:59:03.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:03.298+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:03.303 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:03.301+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:04.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:04.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:04.132112+0000) 2022-04-03T22:59:04.255 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:04.253+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:04.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:04.307+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:05.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:05.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:05.132278+0000) 2022-04-03T22:59:05.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:05.226+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:05.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:05.353+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:05.421 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:59:05.419+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:59:06.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:06.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:06.132456+0000) 2022-04-03T22:59:06.209 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:06.208+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:06.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:06.370+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:07.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:07.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:07.132624+0000) 2022-04-03T22:59:07.243 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:07.241+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:07.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:07.343+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:08.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:08.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:08.132836+0000) 2022-04-03T22:59:08.275 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:08.274+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:08.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:08.303+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:09.023 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:59:09.027 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:59:09.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:09.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:09.133101+0000) 2022-04-03T22:59:09.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:09.266+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:09.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:09.280+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:10.134 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:10.133+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:10.133327+0000) 2022-04-03T22:59:10.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:10.228+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:10.294 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:10.293+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:11.134 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:11.133+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:11.133571+0000) 2022-04-03T22:59:11.195 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:11.193+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:11.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:11.245+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:12.134 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:12.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:12.133829+0000) 2022-04-03T22:59:12.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:12.226+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:12.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:12.243+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:13.134 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:13.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:13.134010+0000) 2022-04-03T22:59:13.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:13.206+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:13.263 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:13.262+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:14.135 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:14.133+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:14.134220+0000) 2022-04-03T22:59:14.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:14.167+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:14.247 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:14.246+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:15.135 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:15.133+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:15.134429+0000) 2022-04-03T22:59:15.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:15.214+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:15.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:15.283+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:16.135 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:16.133+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:16.134665+0000) 2022-04-03T22:59:16.182 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:16.181+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:16.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:16.319+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:17.135 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:17.133+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:17.134884+0000) 2022-04-03T22:59:17.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:17.200+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:17.327 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:17.326+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:18.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:18.134+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:18.135088+0000) 2022-04-03T22:59:18.154 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:18.153+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:18.343 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:18.342+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:19.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:19.134+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:19.135310+0000) 2022-04-03T22:59:19.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:19.172+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:19.387 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:19.386+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:20.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:20.134+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:20.135504+0000) 2022-04-03T22:59:20.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:20.195+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:20.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:20.416+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:20.437 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:59:20.434+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:59:21.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:21.134+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:21.135706+0000) 2022-04-03T22:59:21.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:21.174+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:21.402 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:21.401+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:22.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:22.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:22.135908+0000) 2022-04-03T22:59:22.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:22.154+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:22.360 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:22.359+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:23.137 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:23.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:23.136112+0000) 2022-04-03T22:59:23.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:23.185+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:23.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:23.338+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:24.137 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:24.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:24.136300+0000) 2022-04-03T22:59:24.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:24.165+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:24.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:24.366+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:25.137 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:25.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:25.136459+0000) 2022-04-03T22:59:25.169 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:25.167+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:25.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:25.378+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:26.137 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:26.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:26.136689+0000) 2022-04-03T22:59:26.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:26.199+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:26.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:26.373+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:27.137 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:27.136+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:27.136871+0000) 2022-04-03T22:59:27.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:27.171+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:27.422 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:27.420+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:28.138 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:28.136+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:28.137102+0000) 2022-04-03T22:59:28.182 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:28.179+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:28.442 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:28.440+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:29.138 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:29.136+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:29.137337+0000) 2022-04-03T22:59:29.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:29.209+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:29.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:29.429+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:30.138 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:30.136+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:30.137468+0000) 2022-04-03T22:59:30.221 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:30.219+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:30.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:30.404+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:31.138 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:31.136+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:31.137609+0000) 2022-04-03T22:59:31.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:31.185+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:31.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:31.411+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:32.138 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:32.137+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:32.137820+0000) 2022-04-03T22:59:32.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:32.183+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:32.462 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:32.460+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:33.138 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:33.137+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:33.137978+0000) 2022-04-03T22:59:33.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:33.213+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:33.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:33.459+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:34.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:34.137+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:34.138120+0000) 2022-04-03T22:59:34.202 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:34.200+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:34.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:34.417+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:35.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:35.137+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:35.138267+0000) 2022-04-03T22:59:35.170 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:35.168+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:35.453 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:59:35.451+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:59:35.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:35.462+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:36.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:36.137+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:36.138474+0000) 2022-04-03T22:59:36.147 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:36.145+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:36.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:36.435+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:37.120 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:37.118+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:37.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:37.138+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:37.138651+0000) 2022-04-03T22:59:37.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:37.412+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:38.101 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:38.099+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:38.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:38.138+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:38.138842+0000) 2022-04-03T22:59:38.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:38.441+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:39.140 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:39.138+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:39.139024+0000) 2022-04-03T22:59:39.142 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:39.140+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:39.169 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:59:39.173 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T22:59:39.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:39.422+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:40.140 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:40.138+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:40.139256+0000) 2022-04-03T22:59:40.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:40.151+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:40.441 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:40.439+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:41.140 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:41.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:41.139477+0000) 2022-04-03T22:59:41.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:41.166+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:41.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:41.428+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:42.140 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:42.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:42.139687+0000) 2022-04-03T22:59:42.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:42.213+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:42.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:42.474+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:43.140 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:43.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:43.139913+0000) 2022-04-03T22:59:43.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:43.220+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:43.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:43.494+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:44.141 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:44.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:44.140123+0000) 2022-04-03T22:59:44.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:44.213+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:44.508 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:44.506+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:45.141 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:45.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:45.140296+0000) 2022-04-03T22:59:45.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:45.230+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:45.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:45.459+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:46.141 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:46.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:46.140508+0000) 2022-04-03T22:59:46.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:46.188+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:46.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:46.501+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:47.141 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:47.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:47.140665+0000) 2022-04-03T22:59:47.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:47.199+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:47.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:47.516+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:48.141 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:48.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:48.140884+0000) 2022-04-03T22:59:48.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:48.196+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:48.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:48.530+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:49.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:49.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:49.141088+0000) 2022-04-03T22:59:49.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:49.189+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:49.504 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:49.503+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:50.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:50.141+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:50.141275+0000) 2022-04-03T22:59:50.144 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:50.142+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:50.469 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T22:59:50.467+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T22:59:50.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:50.521+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:51.122 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:51.121+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:51.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:51.141+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:51.141485+0000) 2022-04-03T22:59:51.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:51.485+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:52.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:52.141+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:52.141713+0000) 2022-04-03T22:59:52.149 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:52.147+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:52.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:52.437+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:53.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:53.141+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:53.141951+0000) 2022-04-03T22:59:53.183 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:53.182+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:53.441 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:53.439+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:54.143 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:54.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:54.142192+0000) 2022-04-03T22:59:54.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:54.187+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:54.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:54.405+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:55.143 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:55.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:55.142405+0000) 2022-04-03T22:59:55.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:55.184+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:55.392 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:55.390+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:56.143 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:56.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:56.142621+0000) 2022-04-03T22:59:56.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:56.230+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:56.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:56.413+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:57.143 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:57.141+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:57.142854+0000) 2022-04-03T22:59:57.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:57.196+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:57.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:57.403+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:58.143 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:58.141+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:58.143048+0000) 2022-04-03T22:59:58.160 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:58.159+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:58.372 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:58.370+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T22:59:59.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T22:59:59.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T21:59:59.143199+0000) 2022-04-03T22:59:59.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T22:59:59.182+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T22:59:59.402 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T22:59:59.401+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:00.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:00.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:00.143375+0000) 2022-04-03T23:00:00.151 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:00.150+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:00.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:00.356+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:01.119 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:01.117+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:01.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:01.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:01.143541+0000) 2022-04-03T23:00:01.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:01.351+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:02.117 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:02.115+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:02.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:02.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:02.143694+0000) 2022-04-03T23:00:02.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:02.386+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:03.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:03.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:03.143909+0000) 2022-04-03T23:00:03.147 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:03.146+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:03.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:03.351+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:04.119 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:04.117+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:04.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:04.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:04.144116+0000) 2022-04-03T23:00:04.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:04.372+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:05.088 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:05.087+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:05.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:05.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:05.144346+0000) 2022-04-03T23:00:05.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:05.352+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:05.485 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:00:05.482+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:00:06.121 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:06.120+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:06.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:06.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:06.144583+0000) 2022-04-03T23:00:06.365 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:06.364+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:07.075 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:07.073+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:07.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:07.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:07.144834+0000) 2022-04-03T23:00:07.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:07.384+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:08.065 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:08.064+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:08.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:08.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:08.145030+0000) 2022-04-03T23:00:08.350 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:08.349+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:09.086 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:09.084+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:09.146 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:09.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:09.145228+0000) 2022-04-03T23:00:09.275 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:00:09.279 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:00:09.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:09.372+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:10.121 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:10.120+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:10.146 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:10.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:10.145431+0000) 2022-04-03T23:00:10.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:10.348+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:11.145 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:11.143+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:11.146 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:11.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:11.145664+0000) 2022-04-03T23:00:11.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:11.335+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:12.146 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:12.145+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:12.145843+0000) 2022-04-03T23:00:12.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:12.186+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:12.306 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:12.305+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:13.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:13.145+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:13.146025+0000) 2022-04-03T23:00:13.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:13.227+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:13.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:13.296+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:14.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:14.145+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:14.146194+0000) 2022-04-03T23:00:14.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:14.249+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:14.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:14.313+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:15.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:15.145+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:15.146384+0000) 2022-04-03T23:00:15.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:15.257+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:15.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:15.276+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:16.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:16.145+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:16.146545+0000) 2022-04-03T23:00:16.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:16.217+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:16.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:16.316+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:17.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:17.146+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:17.146751+0000) 2022-04-03T23:00:17.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:17.261+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:17.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:17.348+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:18.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:18.146+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:18.146964+0000) 2022-04-03T23:00:18.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:18.226+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:18.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:18.309+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:19.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:19.146+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:19.147122+0000) 2022-04-03T23:00:19.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:19.267+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:19.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:19.314+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:20.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:20.146+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:20.147316+0000) 2022-04-03T23:00:20.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:20.251+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:20.364 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:20.362+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:20.500 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:00:20.499+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:00:21.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:21.146+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:21.147511+0000) 2022-04-03T23:00:21.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:21.234+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:21.364 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:21.362+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:22.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:22.147+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:22.147725+0000) 2022-04-03T23:00:22.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:22.230+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:22.392 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:22.390+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:23.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:23.147+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:23.147941+0000) 2022-04-03T23:00:23.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:23.214+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:23.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:23.387+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:24.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:24.147+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:24.148150+0000) 2022-04-03T23:00:24.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:24.183+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:24.428 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:24.426+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:25.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:25.147+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:25.148375+0000) 2022-04-03T23:00:25.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:25.178+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:25.450 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:25.448+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:26.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:26.148+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:26.148540+0000) 2022-04-03T23:00:26.158 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:26.156+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:26.410 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:26.408+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:27.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:27.148+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:27.148738+0000) 2022-04-03T23:00:27.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:27.184+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:27.412 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:27.410+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:28.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:28.148+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:28.148991+0000) 2022-04-03T23:00:28.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:28.184+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:28.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:28.387+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:29.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:29.148+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:29.149216+0000) 2022-04-03T23:00:29.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:29.194+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:29.358 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:29.356+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:30.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:30.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:30.149456+0000) 2022-04-03T23:00:30.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:30.233+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:30.397 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:30.395+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:31.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:31.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:31.149592+0000) 2022-04-03T23:00:31.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:31.205+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:31.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:31.411+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:32.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:32.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:32.149765+0000) 2022-04-03T23:00:32.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:32.165+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:32.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:32.445+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:33.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:33.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:33.149951+0000) 2022-04-03T23:00:33.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:33.198+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:33.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:33.472+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:34.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:34.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:34.150169+0000) 2022-04-03T23:00:34.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:34.242+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:34.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:34.449+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:35.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:35.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:35.150376+0000) 2022-04-03T23:00:35.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:35.278+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:35.441 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:35.439+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:35.516 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:00:35.515+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:00:36.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:36.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:36.150564+0000) 2022-04-03T23:00:36.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:36.247+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:36.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:36.428+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:37.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:37.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:37.150719+0000) 2022-04-03T23:00:37.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:37.198+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:37.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:37.420+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:38.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:38.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:38.150918+0000) 2022-04-03T23:00:38.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:38.246+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:38.424 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:38.422+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:39.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:39.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:39.151169+0000) 2022-04-03T23:00:39.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:39.275+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:39.406 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:00:39.409 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:00:39.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:39.445+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:40.152 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:40.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:40.151410+0000) 2022-04-03T23:00:40.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:40.294+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:40.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:40.437+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:41.152 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:41.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:41.151584+0000) 2022-04-03T23:00:41.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:41.298+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:41.475 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:41.473+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:42.152 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:42.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:42.151745+0000) 2022-04-03T23:00:42.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:42.291+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:42.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:42.473+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:43.152 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:43.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:43.151926+0000) 2022-04-03T23:00:43.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:43.324+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:43.507 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:43.505+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:44.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:44.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:44.152145+0000) 2022-04-03T23:00:44.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:44.361+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:44.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:44.512+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:45.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:45.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:45.152338+0000) 2022-04-03T23:00:45.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:45.368+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:45.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:45.533+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:46.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:46.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:46.152541+0000) 2022-04-03T23:00:46.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:46.404+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:46.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:46.552+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:47.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:47.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:47.152733+0000) 2022-04-03T23:00:47.378 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:47.376+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:47.574 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:47.573+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:48.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:48.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:48.152939+0000) 2022-04-03T23:00:48.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:48.367+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:48.558 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:48.556+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:49.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:49.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:49.153129+0000) 2022-04-03T23:00:49.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:49.391+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:49.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:49.513+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:50.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:50.152+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:50.153329+0000) 2022-04-03T23:00:50.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:50.344+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:50.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:50.511+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:50.532 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:00:50.530+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:00:51.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:51.152+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:51.153554+0000) 2022-04-03T23:00:51.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:51.320+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:51.506 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:51.504+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:52.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:52.152+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:52.153720+0000) 2022-04-03T23:00:52.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:52.287+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:52.487 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:52.484+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:53.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:53.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:53.153864+0000) 2022-04-03T23:00:53.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:53.272+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:53.496 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:53.494+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:54.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:54.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:54.154038+0000) 2022-04-03T23:00:54.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:54.280+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:54.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:54.512+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:55.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:55.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:55.154254+0000) 2022-04-03T23:00:55.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:55.307+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:55.556 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:55.555+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:56.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:56.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:56.154426+0000) 2022-04-03T23:00:56.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:56.305+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:56.599 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:56.597+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:57.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:57.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:57.154580+0000) 2022-04-03T23:00:57.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:57.272+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:57.622 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:57.621+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:58.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:58.154+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:58.154730+0000) 2022-04-03T23:00:58.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:58.263+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:58.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:58.631+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:00:59.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:00:59.154+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:00:59.154923+0000) 2022-04-03T23:00:59.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:00:59.303+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:00:59.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:00:59.678+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:00.156 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:00.154+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:00.155088+0000) 2022-04-03T23:01:00.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:00.304+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:00.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:00.716+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:01.156 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:01.154+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:01.155236+0000) 2022-04-03T23:01:01.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:01.332+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:01.673 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:01.672+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:02.156 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:02.154+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:02.155488+0000) 2022-04-03T23:01:02.301 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:02.300+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:02.697 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:02.696+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:03.156 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:03.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:03.155707+0000) 2022-04-03T23:01:03.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:03.295+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:03.692 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:03.691+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:04.156 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:04.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:04.155928+0000) 2022-04-03T23:01:04.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:04.305+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:04.653 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:04.652+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:05.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:05.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:05.156115+0000) 2022-04-03T23:01:05.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:05.290+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:05.546 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:01:05.544+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:01:05.619 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:05.619+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:06.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:06.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:06.156339+0000) 2022-04-03T23:01:06.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:06.321+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:06.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:06.601+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:07.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:07.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:07.156574+0000) 2022-04-03T23:01:07.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:07.302+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:07.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:07.624+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:08.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:08.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:08.156715+0000) 2022-04-03T23:01:08.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:08.333+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:08.593 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:08.592+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:09.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:09.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:09.156841+0000) 2022-04-03T23:01:09.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:09.287+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:09.542 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:01:09.546 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:01:09.617 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:09.616+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:10.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:10.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:10.157069+0000) 2022-04-03T23:01:10.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:10.259+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:10.659 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:10.658+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:11.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:11.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:11.157299+0000) 2022-04-03T23:01:11.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:11.248+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:11.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:11.630+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:12.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:12.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:12.157425+0000) 2022-04-03T23:01:12.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:12.295+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:12.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:12.593+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:13.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:13.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:13.157631+0000) 2022-04-03T23:01:13.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:13.315+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:13.566 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:13.564+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:14.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:14.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:14.157860+0000) 2022-04-03T23:01:14.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:14.350+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:14.556 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:14.554+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:15.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:15.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:15.158036+0000) 2022-04-03T23:01:15.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:15.346+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:15.578 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:15.577+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:16.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:16.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:16.158229+0000) 2022-04-03T23:01:16.396 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:16.394+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:16.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:16.609+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:17.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:17.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:17.158395+0000) 2022-04-03T23:01:17.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:17.406+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:17.637 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:17.635+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:18.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:18.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:18.158611+0000) 2022-04-03T23:01:18.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:18.381+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:18.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:18.645+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:19.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:19.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:19.158853+0000) 2022-04-03T23:01:19.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:19.372+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:19.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:19.645+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:20.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:20.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:20.159000+0000) 2022-04-03T23:01:20.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:20.324+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:20.562 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:01:20.561+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:01:20.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:20.668+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:21.160 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:21.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:21.159205+0000) 2022-04-03T23:01:21.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:21.315+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:21.711 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:21.709+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:22.160 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:22.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:22.159423+0000) 2022-04-03T23:01:22.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:22.298+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:22.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:22.746+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:23.160 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:23.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:23.159581+0000) 2022-04-03T23:01:23.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:23.272+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:23.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:23.697+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:24.160 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:24.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:24.159749+0000) 2022-04-03T23:01:24.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:24.321+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:24.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:24.724+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:25.160 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:25.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:25.159925+0000) 2022-04-03T23:01:25.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:25.350+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:25.692 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:25.690+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:26.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:26.160+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:26.160073+0000) 2022-04-03T23:01:26.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:26.301+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:26.645 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:26.643+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:27.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:27.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:27.160266+0000) 2022-04-03T23:01:27.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:27.252+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:27.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:27.690+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:28.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:28.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:28.160477+0000) 2022-04-03T23:01:28.271 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:28.269+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:28.722 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:28.720+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:29.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:29.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:29.160657+0000) 2022-04-03T23:01:29.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:29.306+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:29.752 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:29.750+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:30.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:30.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:30.160855+0000) 2022-04-03T23:01:30.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:30.277+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:30.767 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:30.765+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:31.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:31.160+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:31.161083+0000) 2022-04-03T23:01:31.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:31.229+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:31.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:31.792+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:32.162 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:32.160+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:32.161296+0000) 2022-04-03T23:01:32.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:32.256+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:32.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:32.821+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:33.162 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:33.160+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:33.161515+0000) 2022-04-03T23:01:33.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:33.235+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:33.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:33.868+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:34.162 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:34.160+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:34.161743+0000) 2022-04-03T23:01:34.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:34.223+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:34.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:34.846+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:35.162 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:35.161+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:35.161931+0000) 2022-04-03T23:01:35.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:35.213+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:35.578 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:01:35.576+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:01:35.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:35.838+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:36.162 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:36.161+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:36.162138+0000) 2022-04-03T23:01:36.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:36.222+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:36.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:36.792+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:37.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:37.161+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:37.162370+0000) 2022-04-03T23:01:37.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:37.206+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:37.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:37.777+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:38.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:38.161+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:38.162585+0000) 2022-04-03T23:01:38.170 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:38.168+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:38.730 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:38.729+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:39.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:39.162+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:39.162814+0000) 2022-04-03T23:01:39.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:39.214+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:39.658 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:01:39.663 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:01:39.721 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:39.719+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:40.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:40.162+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:40.163020+0000) 2022-04-03T23:01:40.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:40.187+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:40.675 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:40.673+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:41.164 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:41.162+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:41.163230+0000) 2022-04-03T23:01:41.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:41.220+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:41.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:41.624+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:42.164 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:42.162+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:42.163436+0000) 2022-04-03T23:01:42.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:42.187+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:42.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:42.620+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:43.164 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:43.162+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:43.163614+0000) 2022-04-03T23:01:43.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:43.230+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:43.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:43.663+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:44.164 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:44.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:44.163852+0000) 2022-04-03T23:01:44.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:44.220+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:44.661 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:44.660+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:45.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:45.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:45.164026+0000) 2022-04-03T23:01:45.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:45.187+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:45.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:45.619+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:46.146 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:46.144+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:46.164 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:46.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:46.164208+0000) 2022-04-03T23:01:46.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:46.638+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:47.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:47.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:47.164415+0000) 2022-04-03T23:01:47.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:47.163+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:47.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:47.640+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:48.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:48.164+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:48.164627+0000) 2022-04-03T23:01:48.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:48.207+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:48.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:48.622+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:49.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:49.164+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:49.164860+0000) 2022-04-03T23:01:49.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:49.244+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:49.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:49.670+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:50.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:50.164+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:50.165046+0000) 2022-04-03T23:01:50.226 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:50.225+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:50.594 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:01:50.592+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:01:50.706 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:50.704+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:51.166 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:51.164+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:51.165266+0000) 2022-04-03T23:01:51.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:51.244+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:51.718 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:51.716+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:52.166 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:52.164+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:52.165432+0000) 2022-04-03T23:01:52.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:52.285+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:52.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:52.686+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:53.166 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:53.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:53.165633+0000) 2022-04-03T23:01:53.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:53.334+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:53.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:53.654+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:54.166 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:54.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:54.165830+0000) 2022-04-03T23:01:54.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:54.289+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:54.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:54.689+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:55.166 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:55.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:55.166003+0000) 2022-04-03T23:01:55.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:55.286+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:55.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:55.647+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:56.167 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:56.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:56.166214+0000) 2022-04-03T23:01:56.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:56.252+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:56.626 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:56.625+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:57.167 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:57.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:57.166400+0000) 2022-04-03T23:01:57.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:57.295+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:57.646 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:57.645+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:58.167 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:58.166+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:58.166620+0000) 2022-04-03T23:01:58.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:58.252+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:58.666 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:58.665+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:01:59.167 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:01:59.166+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:01:59.166827+0000) 2022-04-03T23:01:59.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:01:59.225+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:01:59.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:01:59.634+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:00.167 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:00.166+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:00.167042+0000) 2022-04-03T23:02:00.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:00.214+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:00.618 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:00.617+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:01.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:01.166+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:01.167258+0000) 2022-04-03T23:02:01.217 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:01.216+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:01.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:01.613+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:02.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:02.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:02.167409+0000) 2022-04-03T23:02:02.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:02.239+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:02.608 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:02.607+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:03.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:03.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:03.167622+0000) 2022-04-03T23:02:03.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:03.253+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:03.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:03.601+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:04.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:04.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:04.167853+0000) 2022-04-03T23:02:04.233 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:04.232+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:04.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:04.641+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:05.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:05.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:05.167988+0000) 2022-04-03T23:02:05.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:05.272+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:05.610 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:02:05.608+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:02:05.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:05.637+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:06.169 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:06.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:06.168197+0000) 2022-04-03T23:02:06.271 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:06.270+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:06.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:06.679+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:07.169 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:07.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:07.168409+0000) 2022-04-03T23:02:07.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:07.266+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:07.673 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:07.672+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:08.169 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:08.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:08.168613+0000) 2022-04-03T23:02:08.233 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:08.232+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:08.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:08.680+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:09.169 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:09.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:09.168839+0000) 2022-04-03T23:02:09.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:09.236+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:09.666 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:09.664+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:09.797 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:02:09.801 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:02:10.169 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:10.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:10.169025+0000) 2022-04-03T23:02:10.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:10.248+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:10.651 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:10.648+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:11.170 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:11.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:11.169241+0000) 2022-04-03T23:02:11.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:11.273+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:11.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:11.678+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:12.170 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:12.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:12.169410+0000) 2022-04-03T23:02:12.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:12.266+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:12.709 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:12.707+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:13.170 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:13.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:13.169590+0000) 2022-04-03T23:02:13.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:13.228+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:13.661 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:13.659+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:14.170 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:14.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:14.169830+0000) 2022-04-03T23:02:14.213 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:14.211+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:14.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:14.646+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:15.170 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:15.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:15.169974+0000) 2022-04-03T23:02:15.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:15.233+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:15.659 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:15.657+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:16.171 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:16.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:16.170150+0000) 2022-04-03T23:02:16.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:16.268+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:16.686 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:16.684+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:17.171 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:17.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:17.170313+0000) 2022-04-03T23:02:17.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:17.291+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:17.660 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:17.658+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:18.171 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:18.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:18.170489+0000) 2022-04-03T23:02:18.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:18.292+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:18.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:18.656+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:19.171 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:19.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:19.170713+0000) 2022-04-03T23:02:19.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:19.261+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:19.652 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:19.650+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:20.171 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:20.170+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:20.170951+0000) 2022-04-03T23:02:20.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:20.270+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:20.615 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:02:20.613+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:02:20.674 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:20.672+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:21.172 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:21.170+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:21.171178+0000) 2022-04-03T23:02:21.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:21.235+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:21.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:21.654+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:22.172 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:22.170+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:22.171376+0000) 2022-04-03T23:02:22.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:22.205+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:22.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:22.640+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:23.172 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:23.170+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:23.171557+0000) 2022-04-03T23:02:23.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:23.249+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:23.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:23.654+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:24.172 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:24.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:24.171812+0000) 2022-04-03T23:02:24.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:24.213+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:24.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:24.639+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:25.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:25.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:25.172019+0000) 2022-04-03T23:02:25.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:25.234+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:25.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:25.665+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:02:26.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:26.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:26.172226+0000) 2022-04-03T23:02:26.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:26.264+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:26.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:26.685+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:27.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:27.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:27.172420+0000) 2022-04-03T23:02:27.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:27.302+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:27.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:27.654+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:28.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:28.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:28.172640+0000) 2022-04-03T23:02:28.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:28.328+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:28.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:28.661+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:29.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:29.172+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:29.172852+0000) 2022-04-03T23:02:29.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:29.359+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:29.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:29.685+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:30.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:30.172+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:30.173014+0000) 2022-04-03T23:02:30.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:30.372+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:30.683 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:30.682+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:31.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:31.172+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:31.173225+0000) 2022-04-03T23:02:31.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:31.325+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:31.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:31.708+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:32.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:32.172+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:32.173442+0000) 2022-04-03T23:02:32.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:32.339+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:32.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:32.662+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:33.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:33.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:33.173667+0000) 2022-04-03T23:02:33.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:33.303+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:33.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:33.692+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:34.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:34.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:34.173844+0000) 2022-04-03T23:02:34.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:34.344+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:34.703 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:34.702+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:35.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:35.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:35.174052+0000) 2022-04-03T23:02:35.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:35.320+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:35.631 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:02:35.629+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:02:35.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:35.726+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:36.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:36.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:36.174210+0000) 2022-04-03T23:02:36.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:36.311+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:36.769 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:36.768+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:37.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:37.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:37.174365+0000) 2022-04-03T23:02:37.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:37.286+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:37.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:37.794+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:38.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:38.174+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:38.174568+0000) 2022-04-03T23:02:38.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:38.295+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:38.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:38.808+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:39.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:39.174+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:39.174809+0000) 2022-04-03T23:02:39.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:39.253+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:39.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:39.789+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:39.910 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:02:39.914 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:02:40.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:40.174+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:40.175018+0000) 2022-04-03T23:02:40.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:40.253+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:40.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:40.754+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:41.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:41.174+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:41.175263+0000) 2022-04-03T23:02:41.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:41.218+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:41.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:41.712+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:42.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:42.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:42.175416+0000) 2022-04-03T23:02:42.252 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:42.250+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:42.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:42.688+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:43.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:43.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:43.175589+0000) 2022-04-03T23:02:43.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:43.236+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:43.707 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:43.705+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:44.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:44.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:44.175837+0000) 2022-04-03T23:02:44.205 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:44.204+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:44.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:44.740+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:45.160 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:45.158+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:45.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:45.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:45.176004+0000) 2022-04-03T23:02:45.697 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:45.696+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:46.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:46.155+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:46.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:46.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:46.176210+0000) 2022-04-03T23:02:46.704 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:46.702+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:47.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:47.113+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:47.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:47.176+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:47.176425+0000) 2022-04-03T23:02:47.749 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:47.748+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:48.070 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:48.068+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:48.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:48.176+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:48.176643+0000) 2022-04-03T23:02:48.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:48.736+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:49.118 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:49.117+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:49.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:49.176+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:49.176847+0000) 2022-04-03T23:02:49.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:49.783+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:50.079 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:50.078+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:50.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:50.176+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:50.177062+0000) 2022-04-03T23:02:50.647 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:02:50.645+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:02:50.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:50.827+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:51.100 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:51.099+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:51.178 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:51.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:51.177293+0000) 2022-04-03T23:02:51.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:51.862+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:52.076 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:52.075+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:52.178 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:52.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:52.177498+0000) 2022-04-03T23:02:52.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:52.863+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:53.117 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:53.116+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:53.178 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:53.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:53.177714+0000) 2022-04-03T23:02:53.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:53.887+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:54.096 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:54.095+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:54.178 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:54.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:54.177934+0000) 2022-04-03T23:02:54.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:54.895+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:55.127 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:55.126+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:55.179 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:55.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:55.178145+0000) 2022-04-03T23:02:55.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:55.847+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:56.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:56.156+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:56.179 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:56.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:56.178388+0000) 2022-04-03T23:02:56.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:56.871+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:57.179 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:57.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:57.178621+0000) 2022-04-03T23:02:57.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:57.188+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:57.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:57.874+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:58.179 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:58.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:58.178851+0000) 2022-04-03T23:02:58.183 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:58.182+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:58.902 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:58.901+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:02:59.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:02:59.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:02:59.179053+0000) 2022-04-03T23:02:59.212 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:02:59.211+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:02:59.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:02:59.901+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:00.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:00.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:00.179276+0000) 2022-04-03T23:03:00.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:00.215+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:00.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:00.899+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:01.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:01.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:01.179511+0000) 2022-04-03T23:03:01.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:01.226+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:01.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:01.932+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:02.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:02.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:02.179747+0000) 2022-04-03T23:03:02.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:02.276+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:02.916 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:02.915+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:03.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:03.179+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:03.179994+0000) 2022-04-03T23:03:03.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:03.273+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:03.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:03.895+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:04.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:04.179+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:04.180163+0000) 2022-04-03T23:03:04.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:04.262+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:04.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:04.940+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:05.181 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:05.179+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:05.180377+0000) 2022-04-03T23:03:05.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:05.296+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:05.663 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:03:05.660+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:03:05.946 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:05.944+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:06.181 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:06.179+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:06.180586+0000) 2022-04-03T23:03:06.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:06.292+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:06.915 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:06.913+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:07.181 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:07.180+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:07.180809+0000) 2022-04-03T23:03:07.301 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:07.299+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:07.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:07.876+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:08.181 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:08.180+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:08.180942+0000) 2022-04-03T23:03:08.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:08.308+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:08.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:08.868+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:09.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:09.180+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:09.181078+0000) 2022-04-03T23:03:09.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:09.316+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:09.913 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:09.911+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:10.057 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:03:10.062 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:03:10.181 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:10.180+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:10.181203+0000) 2022-04-03T23:03:10.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:10.310+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:10.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:10.871+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:11.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:11.180+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:11.181351+0000) 2022-04-03T23:03:11.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:11.314+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:11.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:11.829+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:12.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:12.180+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:12.181495+0000) 2022-04-03T23:03:12.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:12.301+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:12.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:12.789+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:13.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:13.180+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:13.181627+0000) 2022-04-03T23:03:13.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:13.297+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:13.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:13.759+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:14.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:14.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:14.181798+0000) 2022-04-03T23:03:14.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:14.266+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:14.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:14.756+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:15.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:15.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:15.181952+0000) 2022-04-03T23:03:15.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:15.232+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:15.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:15.715+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:16.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:16.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:16.182110+0000) 2022-04-03T23:03:16.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:16.260+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:16.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:16.725+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:17.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:17.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:17.182262+0000) 2022-04-03T23:03:17.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:17.238+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:17.721 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:17.719+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:18.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:18.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:18.182414+0000) 2022-04-03T23:03:18.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:18.276+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:18.766 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:18.764+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:19.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:19.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:19.182565+0000) 2022-04-03T23:03:19.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:19.237+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:19.724 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:19.722+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:20.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:20.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:20.182703+0000) 2022-04-03T23:03:20.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:20.246+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:20.679 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:03:20.677+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:03:20.705 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:20.703+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:21.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:21.182+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:21.182859+0000) 2022-04-03T23:03:21.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:21.279+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:21.674 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:21.672+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:22.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:22.182+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:22.183093+0000) 2022-04-03T23:03:22.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:22.287+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:22.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:22.722+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:23.184 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:23.182+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:23.183268+0000) 2022-04-03T23:03:23.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:23.278+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:23.719 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:23.718+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:24.184 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:24.183+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:24.183480+0000) 2022-04-03T23:03:24.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:24.291+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:24.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:24.759+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:25.184 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:25.182+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:25.183719+0000) 2022-04-03T23:03:25.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:25.276+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:25.798 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:25.796+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:26.184 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:26.183+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:26.183902+0000) 2022-04-03T23:03:26.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:26.325+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:26.771 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:26.769+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:27.184 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:27.183+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:27.184081+0000) 2022-04-03T23:03:27.301 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:27.299+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:27.749 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:27.748+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:28.185 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:28.183+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:28.184284+0000) 2022-04-03T23:03:28.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:28.288+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:28.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:28.713+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:29.185 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:29.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:29.184493+0000) 2022-04-03T23:03:29.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:29.290+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:29.757 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:29.755+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:30.185 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:30.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:30.184696+0000) 2022-04-03T23:03:30.283 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:30.282+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:30.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:30.715+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:31.185 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:31.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:31.184907+0000) 2022-04-03T23:03:31.235 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:31.234+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:31.746 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:31.744+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:32.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:32.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:32.185126+0000) 2022-04-03T23:03:32.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:32.266+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:32.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:32.727+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:33.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:33.185+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:33.185346+0000) 2022-04-03T23:03:33.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:33.310+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:33.704 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:33.702+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:34.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:34.185+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:34.185570+0000) 2022-04-03T23:03:34.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:34.341+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:34.681 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:34.680+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:35.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:35.185+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:35.185804+0000) 2022-04-03T23:03:35.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:35.372+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:35.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:35.630+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:35.694 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:03:35.693+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:03:36.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:36.185+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:36.186003+0000) 2022-04-03T23:03:36.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:36.366+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:36.610 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:36.608+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:37.187 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:37.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:37.186246+0000) 2022-04-03T23:03:37.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:37.357+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:37.598 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:37.596+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:38.187 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:38.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:38.186448+0000) 2022-04-03T23:03:38.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:38.394+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:38.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:38.612+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:39.187 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:39.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:39.186652+0000) 2022-04-03T23:03:39.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:39.438+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:39.649 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:39.648+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:40.187 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:40.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:40.186870+0000) 2022-04-03T23:03:40.204 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:03:40.208 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:03:40.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:40.443+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:40.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:40.631+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:41.187 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:41.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:41.187126+0000) 2022-04-03T23:03:41.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:41.423+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:41.630 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:41.628+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:42.188 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:42.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:42.187351+0000) 2022-04-03T23:03:42.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:42.390+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:42.666 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:42.665+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:43.188 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:43.185+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:43.187581+0000) 2022-04-03T23:03:43.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:43.419+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:43.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:43.647+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:44.188 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:44.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:44.187800+0000) 2022-04-03T23:03:44.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:44.407+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:44.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:44.697+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:45.188 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:45.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:45.188018+0000) 2022-04-03T23:03:45.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:45.382+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:45.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:45.726+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:46.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:46.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:46.188217+0000) 2022-04-03T23:03:46.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:46.409+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:46.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:46.686+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:47.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:47.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:47.188417+0000) 2022-04-03T23:03:47.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:47.440+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:47.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:47.668+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:48.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:48.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:48.188603+0000) 2022-04-03T23:03:48.405 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:48.404+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:48.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:48.667+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:49.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:49.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:49.188786+0000) 2022-04-03T23:03:49.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:49.398+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:49.701 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:49.700+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:50.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:50.188+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:50.189000+0000) 2022-04-03T23:03:50.437 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:50.436+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:50.710 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:03:50.708+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:03:50.730 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:50.729+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:51.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:51.188+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:51.189192+0000) 2022-04-03T23:03:51.462 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:51.460+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:51.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:51.736+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:52.190 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:52.188+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:52.189394+0000) 2022-04-03T23:03:52.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:52.444+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:52.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:52.779+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:53.190 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:53.188+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:53.189611+0000) 2022-04-03T23:03:53.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:53.403+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:53.773 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:53.771+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:54.190 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:54.189+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:54.189847+0000) 2022-04-03T23:03:54.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:54.398+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:54.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:54.760+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:55.190 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:55.189+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:55.189992+0000) 2022-04-03T23:03:55.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:55.376+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:55.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:55.809+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:56.190 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:56.189+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:56.190192+0000) 2022-04-03T23:03:56.401 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:56.400+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:56.786 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:56.785+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:57.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:57.189+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:57.190338+0000) 2022-04-03T23:03:57.422 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:57.421+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:57.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:57.787+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:58.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:58.189+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:58.190545+0000) 2022-04-03T23:03:58.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:58.431+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:58.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:58.804+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:03:59.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:03:59.190+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:03:59.190756+0000) 2022-04-03T23:03:59.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:03:59.415+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:03:59.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:03:59.813+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:00.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:00.190+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:00.190933+0000) 2022-04-03T23:04:00.450 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:00.448+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:00.774 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:00.772+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:01.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:01.190+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:01.191143+0000) 2022-04-03T23:04:01.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:01.495+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:01.751 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:01.749+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:02.192 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:02.190+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:02.191348+0000) 2022-04-03T23:04:02.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:02.492+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:02.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:02.797+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:03.192 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:03.190+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:03.191514+0000) 2022-04-03T23:04:03.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:03.446+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:03.756 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:03.754+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:04.192 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:04.191+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:04.191701+0000) 2022-04-03T23:04:04.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:04.489+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:04.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:04.741+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:05.192 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:05.191+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:05.191908+0000) 2022-04-03T23:04:05.539 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:05.537+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:05.726 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:04:05.724+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:04:05.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:05.735+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:06.192 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:06.191+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:06.192127+0000) 2022-04-03T23:04:06.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:06.530+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:06.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:06.698+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:07.193 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:07.191+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:07.192372+0000) 2022-04-03T23:04:07.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:07.569+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:07.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:07.711+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:08.193 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:08.192+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:08.192612+0000) 2022-04-03T23:04:08.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:08.557+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:08.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:08.677+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:09.193 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:09.192+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:09.192838+0000) 2022-04-03T23:04:09.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:09.579+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:09.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:09.666+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:10.193 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:10.192+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:10.193008+0000) 2022-04-03T23:04:10.335 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:04:10.339 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:04:10.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:10.603+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:10.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:10.675+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:11.194 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:11.192+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:11.193244+0000) 2022-04-03T23:04:11.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:11.589+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:11.701 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:11.699+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:12.194 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:12.193+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:12.193393+0000) 2022-04-03T23:04:12.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:12.558+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:12.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:12.737+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:13.194 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:13.193+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:13.193547+0000) 2022-04-03T23:04:13.550 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:13.548+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:13.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:13.710+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:14.194 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:14.193+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:14.193718+0000) 2022-04-03T23:04:14.510 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:14.507+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:14.683 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:14.682+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:15.194 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:15.193+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:15.193960+0000) 2022-04-03T23:04:15.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:15.529+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:15.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:15.693+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:16.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:16.193+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:16.194154+0000) 2022-04-03T23:04:16.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:16.528+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:16.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:16.666+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:17.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:17.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:17.194378+0000) 2022-04-03T23:04:17.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:17.576+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:17.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:17.669+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:18.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:18.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:18.194560+0000) 2022-04-03T23:04:18.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:18.585+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:18.692 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:18.690+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:19.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:19.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:19.194787+0000) 2022-04-03T23:04:19.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:19.546+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:19.689 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:19.687+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:20.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:20.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:20.194964+0000) 2022-04-03T23:04:20.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:20.551+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:20.703 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:20.702+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:20.742 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:04:20.740+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:04:21.196 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:21.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:21.195167+0000) 2022-04-03T23:04:21.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:21.566+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:21.686 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:21.684+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:22.196 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:22.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:22.195340+0000) 2022-04-03T23:04:22.520 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:22.519+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:22.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:22.676+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:23.196 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:23.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:23.195552+0000) 2022-04-03T23:04:23.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:23.514+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:23.678 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:23.676+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:24.196 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:24.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:24.195765+0000) 2022-04-03T23:04:24.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:24.483+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:24.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:24.630+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:25.196 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:25.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:25.196007+0000) 2022-04-03T23:04:25.478 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:25.477+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:25.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:25.669+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:26.197 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:26.196+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:26.196216+0000) 2022-04-03T23:04:26.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:26.442+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:26.649 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:26.647+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:27.197 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:27.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:27.196434+0000) 2022-04-03T23:04:27.446 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:27.444+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:27.600 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:27.598+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:28.197 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:28.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:28.196607+0000) 2022-04-03T23:04:28.438 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:28.436+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:28.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:28.646+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:29.197 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:29.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:29.196837+0000) 2022-04-03T23:04:29.434 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:29.433+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:29.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:29.630+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:30.197 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:30.196+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:30.197047+0000) 2022-04-03T23:04:30.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:30.446+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:30.593 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:30.591+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:31.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:31.196+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:31.197233+0000) 2022-04-03T23:04:31.474 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:31.472+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:31.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:31.618+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:32.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:32.196+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:32.197440+0000) 2022-04-03T23:04:32.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:32.425+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:32.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:32.633+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:33.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:33.196+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:33.197662+0000) 2022-04-03T23:04:33.388 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:33.386+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:33.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:33.601+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:34.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:34.196+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:34.197903+0000) 2022-04-03T23:04:34.356 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:34.355+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:34.627 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:34.626+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:35.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:35.196+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:35.198072+0000) 2022-04-03T23:04:35.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:35.393+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:35.589 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:35.588+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:35.758 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:04:35.755+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:04:36.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:36.197+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:36.198276+0000) 2022-04-03T23:04:36.356 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:36.355+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:36.559 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:36.558+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:37.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:37.197+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:37.198488+0000) 2022-04-03T23:04:37.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:37.353+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:37.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:37.513+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:38.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:38.197+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:38.198689+0000) 2022-04-03T23:04:38.356 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:38.355+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:38.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:38.494+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:39.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:39.198+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:39.198909+0000) 2022-04-03T23:04:39.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:39.356+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:39.469 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:39.468+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:40.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:40.198+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:40.199123+0000) 2022-04-03T23:04:40.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:40.351+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:40.471 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:40.470+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:40.488 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:04:40.492 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:04:41.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:41.198+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:41.199378+0000) 2022-04-03T23:04:41.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:41.356+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:41.509 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:41.508+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:42.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:42.198+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:42.199563+0000) 2022-04-03T23:04:42.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:42.380+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:42.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:42.513+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:43.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:43.199+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:43.199805+0000) 2022-04-03T23:04:43.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:43.409+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:43.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:43.541+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:44.201 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:44.199+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:44.200044+0000) 2022-04-03T23:04:44.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:44.458+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:44.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:44.511+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:45.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:45.199+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:45.200220+0000) 2022-04-03T23:04:45.446 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:45.445+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:45.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:45.545+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:46.201 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:46.199+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:46.200434+0000) 2022-04-03T23:04:46.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:46.482+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:46.505 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:46.504+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:47.201 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:47.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:47.200651+0000) 2022-04-03T23:04:47.462 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:47.461+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:47.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:47.548+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:48.201 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:48.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:48.200804+0000) 2022-04-03T23:04:48.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:48.464+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:48.565 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:48.564+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:49.201 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:49.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:49.201009+0000) 2022-04-03T23:04:49.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:49.466+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:49.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:49.528+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:50.201 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:50.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:50.201187+0000) 2022-04-03T23:04:50.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:50.484+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:50.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:50.495+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:50.774 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:04:50.772+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:04:51.202 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:51.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:51.201388+0000) 2022-04-03T23:04:51.452 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:51.451+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:51.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:51.534+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:52.202 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:52.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:52.201598+0000) 2022-04-03T23:04:52.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:52.455+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:52.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:52.530+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:53.202 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:53.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:53.201819+0000) 2022-04-03T23:04:53.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:53.415+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:53.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:53.512+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:54.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:54.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:54.202017+0000) 2022-04-03T23:04:54.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:54.407+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:54.478 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:54.476+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:55.202 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:55.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:55.202205+0000) 2022-04-03T23:04:55.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:55.369+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:55.429 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:55.428+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:56.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:56.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:56.202373+0000) 2022-04-03T23:04:56.387 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:56.385+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:56.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:56.395+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:57.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:57.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:57.202530+0000) 2022-04-03T23:04:57.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:57.348+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:57.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:57.421+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:58.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:58.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:58.202701+0000) 2022-04-03T23:04:58.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:58.353+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:58.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:58.449+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:04:59.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:04:59.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:04:59.202912+0000) 2022-04-03T23:04:59.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:04:59.384+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:04:59.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:04:59.457+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:00.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:00.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:00.203093+0000) 2022-04-03T23:05:00.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:00.358+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:00.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:00.433+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:01.204 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:01.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:01.203306+0000) 2022-04-03T23:05:01.395 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:01.393+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:01.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:01.404+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:02.204 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:02.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:02.203523+0000) 2022-04-03T23:05:02.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:02.372+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:02.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:02.411+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:03.204 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:03.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:03.203696+0000) 2022-04-03T23:05:03.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:03.365+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:03.424 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:03.422+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:04.204 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:04.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:04.203900+0000) 2022-04-03T23:05:04.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:04.358+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:04.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:04.375+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:05.204 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:05.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:05.204079+0000) 2022-04-03T23:05:05.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:05.309+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:05.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:05.337+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:05.789 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:05:05.788+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:05:06.205 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:06.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:06.204293+0000) 2022-04-03T23:05:06.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:06.327+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:06.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:06.329+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:07.205 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:07.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:07.204489+0000) 2022-04-03T23:05:07.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:07.313+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:07.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:07.369+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:08.205 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:08.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:08.204692+0000) 2022-04-03T23:05:08.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:08.323+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:08.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:08.332+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:09.205 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:09.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:09.204898+0000) 2022-04-03T23:05:09.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:09.315+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:09.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:09.342+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:10.205 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:10.205+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:10.205118+0000) 2022-04-03T23:05:10.304 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:10.302+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:10.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:10.314+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:10.561 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:05:10.565 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:05:11.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:11.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:11.205373+0000) 2022-04-03T23:05:11.255 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:11.253+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:11.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:11.336+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:12.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:12.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:12.205528+0000) 2022-04-03T23:05:12.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:12.293+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:12.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:12.325+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:13.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:13.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:13.205744+0000) 2022-04-03T23:05:13.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:13.337+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:13.356 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:13.354+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:14.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:14.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:14.205943+0000) 2022-04-03T23:05:14.304 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:14.302+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:14.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:14.347+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:15.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:15.205+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:15.206133+0000) 2022-04-03T23:05:15.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:15.321+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:15.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:15.364+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:16.207 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:16.205+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:16.206327+0000) 2022-04-03T23:05:16.307 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:16.305+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:16.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:16.324+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:17.207 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:17.205+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:17.206544+0000) 2022-04-03T23:05:17.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:17.280+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:17.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:17.326+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:18.207 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:18.205+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:18.206746+0000) 2022-04-03T23:05:18.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:18.257+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:18.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:18.360+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:19.207 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:19.206+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:19.207008+0000) 2022-04-03T23:05:19.290 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:19.289+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:19.401 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:19.399+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:20.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:20.206+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:20.207247+0000) 2022-04-03T23:05:20.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:20.255+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:20.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:20.355+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:20.805 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:05:20.803+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:05:21.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:21.206+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:21.207426+0000) 2022-04-03T23:05:21.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:21.283+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:21.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:21.308+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:22.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:22.206+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:22.207652+0000) 2022-04-03T23:05:22.300 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:22.299+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:22.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:22.338+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:23.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:23.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:23.207845+0000) 2022-04-03T23:05:23.252 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:23.250+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:23.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:23.385+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:24.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:24.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:24.208046+0000) 2022-04-03T23:05:24.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:24.264+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:24.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:24.417+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:25.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:25.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:25.208230+0000) 2022-04-03T23:05:25.258 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:25.257+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:05:25.412 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:25.411+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:26.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:26.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:26.208448+0000) 2022-04-03T23:05:26.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:26.225+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:26.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:26.440+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:27.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:27.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:27.208678+0000) 2022-04-03T23:05:27.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:27.229+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:27.477 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:27.475+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:28.185 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:28.184+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:28.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:28.208+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:28.208865+0000) 2022-04-03T23:05:28.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:28.442+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:29.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:29.163+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:29.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:29.208+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:29.209074+0000) 2022-04-03T23:05:29.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:29.396+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:30.176 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:30.174+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:30.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:30.208+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:30.209264+0000) 2022-04-03T23:05:30.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:30.418+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:31.155 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:31.154+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:31.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:31.208+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:31.209448+0000) 2022-04-03T23:05:31.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:31.392+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:32.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:32.172+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:32.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:32.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:32.209642+0000) 2022-04-03T23:05:32.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:32.370+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:33.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:33.197+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:33.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:33.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:33.209856+0000) 2022-04-03T23:05:33.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:33.339+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:34.181 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:34.180+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:34.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:34.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:34.210036+0000) 2022-04-03T23:05:34.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:34.305+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:35.185 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:35.184+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:35.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:35.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:35.210213+0000) 2022-04-03T23:05:35.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:35.316+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:35.821 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:05:35.820+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:05:36.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:36.142+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:36.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:36.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:36.210422+0000) 2022-04-03T23:05:36.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:36.295+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:37.186 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:37.185+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:37.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:37.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:37.210621+0000) 2022-04-03T23:05:37.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:37.328+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:38.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:38.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:38.210802+0000) 2022-04-03T23:05:38.216 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:38.215+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:38.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:38.378+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:39.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:39.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:39.210962+0000) 2022-04-03T23:05:39.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:39.217+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:39.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:39.394+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:40.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:40.182+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:40.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:40.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:40.211174+0000) 2022-04-03T23:05:40.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:40.429+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:40.710 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:05:40.714 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:05:41.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:41.140+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:41.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:41.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:41.211418+0000) 2022-04-03T23:05:41.415 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:41.414+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:42.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:42.187+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:42.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:42.211+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:42.211577+0000) 2022-04-03T23:05:42.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:42.446+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:43.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:43.211+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:43.211795+0000) 2022-04-03T23:05:43.237 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:43.236+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:43.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:43.466+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:44.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:44.211+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:44.212009+0000) 2022-04-03T23:05:44.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:44.228+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:44.501 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:44.500+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:45.186 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:45.185+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:45.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:45.211+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:45.212207+0000) 2022-04-03T23:05:45.522 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:45.521+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:46.152 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:46.151+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:46.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:46.212+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:46.212411+0000) 2022-04-03T23:05:46.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:46.513+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:47.169 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:47.168+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:47.213 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:47.212+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:47.212627+0000) 2022-04-03T23:05:47.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:47.514+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:48.193 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:48.192+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:48.213 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:48.212+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:48.212788+0000) 2022-04-03T23:05:48.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:48.540+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:49.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:49.209+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:49.213 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:49.212+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:49.212965+0000) 2022-04-03T23:05:49.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:49.577+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:50.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:50.165+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:50.214 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:50.212+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:50.213181+0000) 2022-04-03T23:05:50.573 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:50.571+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:50.837 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:05:50.835+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:05:51.129 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:51.127+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:51.214 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:51.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:51.213354+0000) 2022-04-03T23:05:51.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:51.559+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:52.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:52.161+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:52.214 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:52.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:52.213558+0000) 2022-04-03T23:05:52.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:52.579+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:53.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:53.159+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:53.214 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:53.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:53.213760+0000) 2022-04-03T23:05:53.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:53.599+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:54.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:54.193+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:54.214 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:54.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:54.213928+0000) 2022-04-03T23:05:54.566 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:54.564+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:55.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:55.193+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:55.214 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:55.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:55.214160+0000) 2022-04-03T23:05:55.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:55.587+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:56.151 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:56.149+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:56.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:56.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:56.214382+0000) 2022-04-03T23:05:56.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:56.628+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:57.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:57.162+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:57.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:57.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:57.214587+0000) 2022-04-03T23:05:57.623 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:57.621+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:58.208 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:58.206+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:58.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:58.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:58.214799+0000) 2022-04-03T23:05:58.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:58.669+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:05:59.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:05:59.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:05:59.214995+0000) 2022-04-03T23:05:59.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:05:59.229+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:05:59.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:05:59.669+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:00.216 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:00.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:00.215196+0000) 2022-04-03T23:06:00.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:00.224+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:00.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:00.641+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:01.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:01.193+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:01.216 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:01.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:01.215378+0000) 2022-04-03T23:06:01.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:01.686+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:02.206 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:02.204+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:02.216 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:02.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:02.215556+0000) 2022-04-03T23:06:02.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:02.673+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:03.200 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:03.198+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:03.216 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:03.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:03.215761+0000) 2022-04-03T23:06:03.664 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:03.662+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:04.216 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:04.215+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:04.216007+0000) 2022-04-03T23:06:04.230 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:04.229+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:04.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:04.617+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:05.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:05.215+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:05.216216+0000) 2022-04-03T23:06:05.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:05.226+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:05.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:05.665+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:05.853 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:06:05.851+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:06:06.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:06.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:06.216435+0000) 2022-04-03T23:06:06.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:06.271+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:06.659 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:06.657+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:07.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:07.215+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:07.216658+0000) 2022-04-03T23:06:07.242 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:07.240+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:07.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:07.690+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:08.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:08.216+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:08.216885+0000) 2022-04-03T23:06:08.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:08.267+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:08.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:08.739+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:09.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:09.216+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:09.217074+0000) 2022-04-03T23:06:09.271 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:09.269+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:09.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:09.730+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:10.218 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:10.216+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:10.217285+0000) 2022-04-03T23:06:10.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:10.307+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:10.691 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:10.689+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:10.832 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:06:10.836 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:06:11.218 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:11.216+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:11.217518+0000) 2022-04-03T23:06:11.312 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:11.310+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:11.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:11.644+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:12.218 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:12.216+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:12.217665+0000) 2022-04-03T23:06:12.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:12.326+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:12.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:12.678+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:13.218 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:13.217+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:13.217863+0000) 2022-04-03T23:06:13.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:13.361+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:13.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:13.702+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:14.218 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:14.217+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:14.218045+0000) 2022-04-03T23:06:14.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:14.333+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:14.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:14.657+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:15.218 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:15.217+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:15.218249+0000) 2022-04-03T23:06:15.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:15.326+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:15.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:15.683+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:16.219 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:16.217+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:16.218473+0000) 2022-04-03T23:06:16.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:16.280+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:16.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:16.726+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:17.219 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:17.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:17.218692+0000) 2022-04-03T23:06:17.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:17.294+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:17.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:17.740+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:18.219 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:18.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:18.218895+0000) 2022-04-03T23:06:18.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:18.308+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:18.776 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:18.775+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:19.219 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:19.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:19.219046+0000) 2022-04-03T23:06:19.287 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:19.285+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:19.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:19.755+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:20.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:20.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:20.219258+0000) 2022-04-03T23:06:20.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:20.315+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:20.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:20.715+0000 7f88d0ac1700 -1 osd.4 130 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:20.869 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:06:20.867+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:06:21.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:21.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:21.219476+0000) 2022-04-03T23:06:21.338 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:21.336+0000 7fd3b190b700 -1 osd.6 130 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:21.571 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-03T23:06:21.569+0000 7fbf00b7c700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T23:06:21.586 INFO:tasks.ceph.mds.f.smithi186.stderr: -1> 2022-04-03T23:06:21.569+0000 7fbf00b7c700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T23:06:21.607 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-03T23:06:21.674 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:21.672+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:22.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:22.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:22.219663+0000) 2022-04-03T23:06:22.293 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:22.291+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:22.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:22.626+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:23.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:23.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:23.219877+0000) 2022-04-03T23:06:23.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:23.322+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:23.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:23.619+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:24.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:24.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:24.220057+0000) 2022-04-03T23:06:24.347 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:24.345+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:24.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:24.635+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:25.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:25.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:25.220259+0000) 2022-04-03T23:06:25.396 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:25.394+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:25.622 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:25.621+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:26.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:26.220+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:26.220463+0000) 2022-04-03T23:06:26.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:26.352+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:26.664 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:26.663+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 38 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:27.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:27.220+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:27.220677+0000) 2022-04-03T23:06:27.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:27.381+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:27.687 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:27.686+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:28.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:28.220+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:28.220894+0000) 2022-04-03T23:06:28.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:28.379+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:28.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:28.668+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:29.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:29.220+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:29.221093+0000) 2022-04-03T23:06:29.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:29.417+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:29.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:29.709+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:30.222 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:30.220+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:30.221311+0000) 2022-04-03T23:06:30.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:30.454+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:30.704 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:30.703+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:31.222 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:31.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:31.221538+0000) 2022-04-03T23:06:31.408 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:31.407+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:31.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:31.656+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:32.222 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:32.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:32.221764+0000) 2022-04-03T23:06:32.387 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:32.386+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:32.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:32.652+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:33.222 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:33.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:33.221959+0000) 2022-04-03T23:06:33.348 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:33.347+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:33.607 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:33.606+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:34.222 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:34.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:34.222132+0000) 2022-04-03T23:06:34.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:34.368+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:34.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:34.564+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:35.226 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:35.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:35.222341+0000) 2022-04-03T23:06:35.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:35.338+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:35.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:35.528+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:35.884 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:06:35.883+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:06:36.223 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:36.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:36.222490+0000) 2022-04-03T23:06:36.346 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:36.345+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:36.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:36.514+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:37.223 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:37.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:37.222695+0000) 2022-04-03T23:06:37.337 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:37.336+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:37.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:37.466+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:38.223 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:38.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:38.222902+0000) 2022-04-03T23:06:38.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:38.301+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:38.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:38.435+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:39.223 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:39.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:39.223124+0000) 2022-04-03T23:06:39.350 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:39.349+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:39.450 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:39.449+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:40.224 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:40.223+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:40.223322+0000) 2022-04-03T23:06:40.318 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:40.317+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:40.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:40.482+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:41.178 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:06:41.182 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:06:41.224 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:41.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:41.223557+0000) 2022-04-03T23:06:41.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:41.352+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:41.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:41.513+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:42.224 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:42.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:42.223743+0000) 2022-04-03T23:06:42.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:42.389+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:42.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:42.475+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:43.224 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:43.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:43.223960+0000) 2022-04-03T23:06:43.432 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:43.431+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:43.433 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:43.433+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:44.225 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:44.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:44.224132+0000) 2022-04-03T23:06:44.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:44.408+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:44.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:44.479+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:45.225 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:45.223+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:45.224313+0000) 2022-04-03T23:06:45.376 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:45.374+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:45.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:45.438+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:46.225 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:46.223+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:46.224523+0000) 2022-04-03T23:06:46.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:46.406+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:46.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:46.443+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:47.225 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:47.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:47.224645+0000) 2022-04-03T23:06:47.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:47.366+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:47.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:47.419+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:48.225 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:48.223+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:48.224875+0000) 2022-04-03T23:06:48.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:48.384+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:48.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:48.454+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:49.225 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:49.224+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:49.225052+0000) 2022-04-03T23:06:49.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:49.342+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:49.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:49.495+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:50.226 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:50.224+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:50.225268+0000) 2022-04-03T23:06:50.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:50.303+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:50.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:50.530+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:50.891 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:06:50.890+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:06:51.226 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:51.224+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:51.225475+0000) 2022-04-03T23:06:51.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:51.313+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:51.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:51.539+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:52.226 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:52.224+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:52.225636+0000) 2022-04-03T23:06:52.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:52.302+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:52.561 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:52.559+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:53.226 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:53.225+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:53.225870+0000) 2022-04-03T23:06:53.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:53.331+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:53.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:53.590+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:54.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:54.225+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:54.226069+0000) 2022-04-03T23:06:54.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:54.367+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:54.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:54.577+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:55.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:55.225+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:55.226240+0000) 2022-04-03T23:06:55.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:55.334+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:55.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:55.575+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:56.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:56.225+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:56.226400+0000) 2022-04-03T23:06:56.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:56.356+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:56.578 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:56.576+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:57.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:57.225+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:57.226574+0000) 2022-04-03T23:06:57.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:57.382+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:57.612 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:57.610+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:58.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:58.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:58.226812+0000) 2022-04-03T23:06:58.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:58.333+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:58.604 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:58.602+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:06:59.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:06:59.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:06:59.226989+0000) 2022-04-03T23:06:59.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:06:59.350+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:06:59.639 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:06:59.637+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:00.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:00.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:00.227199+0000) 2022-04-03T23:07:00.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:00.378+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:00.627 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:00.625+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:01.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:01.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:01.227391+0000) 2022-04-03T23:07:01.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:01.364+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:01.606 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:01.604+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:02.228 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:02.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:02.227612+0000) 2022-04-03T23:07:02.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:02.392+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:02.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:02.579+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:03.228 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:03.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:03.227847+0000) 2022-04-03T23:07:03.407 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:03.405+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:03.626 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:03.624+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:04.228 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:04.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:04.228089+0000) 2022-04-03T23:07:04.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:04.415+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:04.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:04.653+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:05.229 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:05.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:05.228296+0000) 2022-04-03T23:07:05.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:05.458+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:05.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:05.610+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:05.898 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:07:05.897+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:07:06.229 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:06.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:06.228503+0000) 2022-04-03T23:07:06.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:06.482+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:06.575 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:06.573+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:07.229 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:07.228+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:07.228759+0000) 2022-04-03T23:07:07.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:07.479+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:07.622 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:07.620+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:08.229 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:08.228+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:08.229016+0000) 2022-04-03T23:07:08.468 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:08.467+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:08.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:08.578+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:09.229 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:09.228+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:09.229215+0000) 2022-04-03T23:07:09.422 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:09.420+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:09.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:09.621+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:10.230 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:10.228+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:10.229462+0000) 2022-04-03T23:07:10.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:10.404+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:10.597 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:10.595+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:11.230 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:11.229+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:11.229677+0000) 2022-04-03T23:07:11.320 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:07:11.324 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:07:11.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:11.382+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:11.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:11.550+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:12.230 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:12.229+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:12.229897+0000) 2022-04-03T23:07:12.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:12.418+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:12.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:12.502+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:13.230 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:13.229+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:13.230060+0000) 2022-04-03T23:07:13.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:13.454+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:13.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:13.458+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:14.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:14.229+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:14.230267+0000) 2022-04-03T23:07:14.428 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:14.427+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:14.487 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:14.485+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:15.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:15.229+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:15.230509+0000) 2022-04-03T23:07:15.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:15.453+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:15.475 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:15.474+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:16.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:16.230+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:16.230676+0000) 2022-04-03T23:07:16.438 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:16.437+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:16.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:16.446+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:17.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:17.230+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:17.230826+0000) 2022-04-03T23:07:17.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:17.412+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:17.468 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:17.467+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:18.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:18.230+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:18.231012+0000) 2022-04-03T23:07:18.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:18.376+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:18.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:18.442+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:19.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:19.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:19.231208+0000) 2022-04-03T23:07:19.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:19.359+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:19.493 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:19.491+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:20.232 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:20.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:20.231417+0000) 2022-04-03T23:07:20.384 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:20.382+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:20.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:20.494+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:20.914 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:07:20.913+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:07:21.232 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:21.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:21.231629+0000) 2022-04-03T23:07:21.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:21.400+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:21.533 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:21.532+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:22.232 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:22.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:22.231862+0000) 2022-04-03T23:07:22.392 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:22.391+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:22.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:22.527+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:23.232 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:23.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:23.232014+0000) 2022-04-03T23:07:23.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:23.416+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:23.546 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:23.545+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:24.232 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:24.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:24.232224+0000) 2022-04-03T23:07:24.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:24.381+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:24.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:24.528+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:25.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:25.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:25.232440+0000) 2022-04-03T23:07:25.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:25.351+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:25.572 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:25.571+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:26.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:26.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:26.232652+0000) 2022-04-03T23:07:26.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:26.350+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:26.537 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:26.536+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:27.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:27.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:27.232847+0000) 2022-04-03T23:07:27.397 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:27.396+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:27.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:27.579+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:28.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:28.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:28.233034+0000) 2022-04-03T23:07:28.432 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:28.431+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:28.604 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:28.603+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:29.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:29.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:29.233224+0000) 2022-04-03T23:07:29.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:29.429+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:29.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:29.627+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:30.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:30.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:30.233442+0000) 2022-04-03T23:07:30.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:30.446+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:30.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:30.664+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:31.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:31.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:31.233619+0000) 2022-04-03T23:07:31.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:31.430+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:31.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:31.668+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:32.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:32.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:32.233850+0000) 2022-04-03T23:07:32.403 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:32.402+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:32.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:32.715+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:33.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:33.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:33.234011+0000) 2022-04-03T23:07:33.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:33.406+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:33.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:33.694+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:34.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:34.233+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:34.234182+0000) 2022-04-03T23:07:34.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:34.400+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:34.662 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:34.661+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:35.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:35.233+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:35.234351+0000) 2022-04-03T23:07:35.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:35.351+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:35.706 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:35.705+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:35.930 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:07:35.928+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:07:36.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:36.233+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:36.234581+0000) 2022-04-03T23:07:36.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:36.387+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:36.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:36.722+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:37.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:37.234+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:37.234809+0000) 2022-04-03T23:07:37.346 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:37.345+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:37.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:37.742+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:38.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:38.234+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:38.234960+0000) 2022-04-03T23:07:38.332 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:38.331+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:38.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:38.736+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:39.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:39.234+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:39.235145+0000) 2022-04-03T23:07:39.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:39.359+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:39.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:39.732+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:40.236 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:40.234+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:40.235340+0000) 2022-04-03T23:07:40.384 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:40.382+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:40.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:40.732+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:41.236 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:41.234+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:41.235550+0000) 2022-04-03T23:07:41.346 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:41.344+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:41.473 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:07:41.477 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:07:41.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:41.755+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:42.236 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:42.235+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:42.235839+0000) 2022-04-03T23:07:42.322 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:42.320+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:42.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:42.783+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:43.236 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:43.235+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:43.236024+0000) 2022-04-03T23:07:43.274 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:43.272+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:43.783 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:43.781+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:44.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:44.235+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:44.236206+0000) 2022-04-03T23:07:44.261 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:44.259+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:44.776 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:44.774+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:45.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:45.235+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:45.236427+0000) 2022-04-03T23:07:45.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:45.246+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:45.825 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:45.823+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:46.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:46.235+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:46.236576+0000) 2022-04-03T23:07:46.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:46.276+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:46.837 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:46.835+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:47.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:47.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:47.236827+0000) 2022-04-03T23:07:47.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:47.318+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:47.808 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:47.806+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:48.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:48.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:48.237020+0000) 2022-04-03T23:07:48.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:48.324+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:48.769 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:48.767+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:49.238 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:49.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:49.237216+0000) 2022-04-03T23:07:49.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:49.284+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:49.326 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-04-03T23:07:49.338 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-04-03T23:07:49.812 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:49.810+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:50.238 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:50.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:50.237454+0000) 2022-04-03T23:07:50.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:50.277+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:50.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:50.765+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:50.946 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:07:50.944+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:07:51.238 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:51.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:51.237680+0000) 2022-04-03T23:07:51.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:51.319+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:51.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:51.809+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:52.238 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:52.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:52.237830+0000) 2022-04-03T23:07:52.307 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:52.305+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:52.856 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:52.854+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:53.238 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:53.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:53.238055+0000) 2022-04-03T23:07:53.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:53.351+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:53.860 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:53.858+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:54.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:54.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:54.238265+0000) 2022-04-03T23:07:54.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:54.338+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:54.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:54.887+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:55.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:55.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:55.238446+0000) 2022-04-03T23:07:55.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:55.377+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:55.892 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:55.890+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:56.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:56.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:56.238638+0000) 2022-04-03T23:07:56.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:56.370+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:56.856 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:56.854+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:57.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:57.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:57.238859+0000) 2022-04-03T23:07:57.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:57.337+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:57.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:57.841+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:58.240 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:58.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:58.239076+0000) 2022-04-03T23:07:58.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:58.376+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:58.844 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:58.842+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:07:59.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:07:59.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:07:59.239266+0000) 2022-04-03T23:07:59.342 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:07:59.341+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:07:59.887 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:07:59.885+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:00.240 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:00.239+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:00.239472+0000) 2022-04-03T23:08:00.338 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:00.336+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:00.906 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:00.905+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:01.240 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:01.239+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:01.239706+0000) 2022-04-03T23:08:01.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:01.297+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:01.921 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:01.919+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:02.240 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:02.239+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:02.239874+0000) 2022-04-03T23:08:02.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:02.333+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:02.880 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:02.878+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:03.240 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:03.239+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:03.240057+0000) 2022-04-03T23:08:03.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:03.361+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:03.885 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:03.883+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:04.241 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:04.239+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:04.240256+0000) 2022-04-03T23:08:04.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:04.371+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:04.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:04.895+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:05.241 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:05.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:05.240459+0000) 2022-04-03T23:08:05.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:05.359+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:05.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:05.913+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:05.961 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:08:05.960+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:08:06.241 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:06.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:06.240608+0000) 2022-04-03T23:08:06.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:06.362+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:06.905 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:06.903+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:07.241 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:07.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:07.240786+0000) 2022-04-03T23:08:07.356 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:07.354+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:07.887 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:07.885+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:08.241 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:08.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:08.240994+0000) 2022-04-03T23:08:08.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:08.320+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:08.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:08.913+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:09.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:09.241+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:09.241208+0000) 2022-04-03T23:08:09.313 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:09.311+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:09.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:09.877+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:10.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:10.241+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:10.241369+0000) 2022-04-03T23:08:10.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:10.333+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:10.875 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:10.874+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:11.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:11.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:11.241580+0000) 2022-04-03T23:08:11.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:11.361+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:11.625 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:08:11.629 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:08:11.849 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:11.847+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:12.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:12.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:12.241780+0000) 2022-04-03T23:08:12.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:12.347+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:12.898 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:12.896+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:13.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:13.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:13.241998+0000) 2022-04-03T23:08:13.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:13.328+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:13.871 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:13.870+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:14.243 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:14.241+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:14.242187+0000) 2022-04-03T23:08:14.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:14.366+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:14.910 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:14.909+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:15.243 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:15.241+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:15.242396+0000) 2022-04-03T23:08:15.370 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:15.368+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:15.867 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:15.866+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:16.243 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:16.241+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:16.242583+0000) 2022-04-03T23:08:16.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:16.350+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:16.839 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:16.838+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:17.243 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:17.241+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:17.242806+0000) 2022-04-03T23:08:17.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:17.380+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:17.814 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:17.813+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:18.243 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:18.242+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:18.243012+0000) 2022-04-03T23:08:18.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:18.387+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:18.851 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:18.850+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:19.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:19.242+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:19.243255+0000) 2022-04-03T23:08:19.395 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:19.394+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:19.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:19.896+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:20.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:20.242+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:20.243438+0000) 2022-04-03T23:08:20.347 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:20.346+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:20.852 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:20.851+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:20.977 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:08:20.976+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:08:21.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:21.242+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:21.243657+0000) 2022-04-03T23:08:21.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:21.320+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:21.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:21.854+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:22.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:22.243+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:22.243863+0000) 2022-04-03T23:08:22.366 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:22.365+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:22.893 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:22.892+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:23.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:23.243+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:23.244076+0000) 2022-04-03T23:08:23.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:23.320+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:23.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:23.877+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:24.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:24.243+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:24.244278+0000) 2022-04-03T23:08:24.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:24.281+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:24.911 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:24.910+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:25.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:25.243+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:25.244467+0000) 2022-04-03T23:08:25.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:25.267+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:25.886 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:25.885+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:26.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:26.243+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:26.244661+0000) 2022-04-03T23:08:26.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:26.259+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:26.911 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:26.910+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:27.212 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:27.211+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:27.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:27.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:27.244828+0000) 2022-04-03T23:08:27.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:27.952+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:28.204 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:28.204+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:28.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:28.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:28.244987+0000) 2022-04-03T23:08:28.930 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:28.929+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:29.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:29.164+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:29.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:29.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:29.245175+0000) 2022-04-03T23:08:29.904 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:29.902+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:30.150 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:30.149+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:30.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:30.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:30.245361+0000) 2022-04-03T23:08:30.949 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:30.948+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:31.138 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:31.137+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:31.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:31.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:31.245585+0000) 2022-04-03T23:08:31.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:31.914+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:32.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:32.178+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:32.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:32.245+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:32.245737+0000) 2022-04-03T23:08:32.941 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:32.940+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:33.210 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:33.208+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:33.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:33.245+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:33.245952+0000) 2022-04-03T23:08:33.899 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:33.898+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:34.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:34.245+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:34.246139+0000) 2022-04-03T23:08:34.255 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:34.253+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:34.924 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:34.922+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:35.247 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:35.245+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:35.246345+0000) 2022-04-03T23:08:35.270 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:35.268+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:35.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:35.887+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:35.993 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:08:35.991+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:08:36.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:36.225+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:36.247 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:36.245+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:36.246546+0000) 2022-04-03T23:08:36.896 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:36.894+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:37.212 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:37.210+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:37.247 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:37.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:37.246750+0000) 2022-04-03T23:08:37.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:37.867+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:38.230 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:38.228+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:38.247 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:38.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:38.246972+0000) 2022-04-03T23:08:38.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:38.870+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:39.213 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:39.211+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:39.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:39.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:39.247192+0000) 2022-04-03T23:08:39.907 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:39.905+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:40.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:40.188+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:40.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:40.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:40.247392+0000) 2022-04-03T23:08:40.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:40.875+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:41.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:41.172+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:41.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:41.247+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:41.247609+0000) 2022-04-03T23:08:41.817 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:08:41.821 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:08:41.835 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:41.833+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:42.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:42.125+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:42.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:42.247+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:42.247882+0000) 2022-04-03T23:08:42.846 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:42.844+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:43.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:43.158+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:43.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:43.247+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:43.248045+0000) 2022-04-03T23:08:43.876 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:43.874+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:44.185 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:44.184+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:44.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:44.247+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:44.248258+0000) 2022-04-03T23:08:44.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:44.827+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:45.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:45.193+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:45.249 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:45.248+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:45.248436+0000) 2022-04-03T23:08:45.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:45.798+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:46.217 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:46.215+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:46.249 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:46.248+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:46.248636+0000) 2022-04-03T23:08:46.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:46.804+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:47.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:47.225+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:47.249 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:47.248+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:47.248866+0000) 2022-04-03T23:08:47.856 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:47.854+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:48.239 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:48.238+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:48.249 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:48.248+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:48.249069+0000) 2022-04-03T23:08:48.866 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:48.864+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:49.212 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:49.210+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:49.249 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:49.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:49.249275+0000) 2022-04-03T23:08:49.874 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:49.872+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:50.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:50.193+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:50.250 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:50.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:50.249497+0000) 2022-04-03T23:08:50.871 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:50.869+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:51.009 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:08:51.007+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:08:51.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:51.171+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:51.250 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:51.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:51.249672+0000) 2022-04-03T23:08:51.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:51.887+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:52.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:52.159+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:52.250 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:52.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:52.249829+0000) 2022-04-03T23:08:52.913 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:52.911+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:53.191 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:53.190+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:53.250 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:53.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:53.249994+0000) 2022-04-03T23:08:53.901 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:53.900+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:54.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:54.224+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:54.250 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:54.250+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:54.250203+0000) 2022-04-03T23:08:54.927 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:54.925+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:55.251 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:55.250+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:55.250434+0000) 2022-04-03T23:08:55.275 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:55.274+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:55.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:55.942+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:56.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:56.248+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:56.251 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:56.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:56.250589+0000) 2022-04-03T23:08:56.974 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:56.973+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:57.205 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:57.203+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:57.251 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:57.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:57.250737+0000) 2022-04-03T23:08:57.936 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:57.934+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:58.199 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:58.197+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:58.251 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:58.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:58.250894+0000) 2022-04-03T23:08:58.960 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:58.958+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:08:59.217 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:08:59.215+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:08:59.251 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:08:59.250+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:08:59.251030+0000) 2022-04-03T23:08:59.950 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:08:59.949+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:00.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:00.226+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:00.251 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:00.250+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:00.251230+0000) 2022-04-03T23:09:00.973 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:00.972+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:01.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:01.201+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:01.252 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:01.250+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:01.251496+0000) 2022-04-03T23:09:01.982 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:01.980+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:02.236 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:02.234+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:02.252 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:02.250+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:02.251719+0000) 2022-04-03T23:09:02.993 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:02.991+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:03.252 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:03.250+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:03.251913+0000) 2022-04-03T23:09:03.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:03.272+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:03.981 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:03.980+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:04.252 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:04.251+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:04.252120+0000) 2022-04-03T23:09:04.280 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:04.279+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:04.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:04.993+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:05.252 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:05.251+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:05.252308+0000) 2022-04-03T23:09:05.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:05.314+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:06.025 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:09:06.023+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:09:06.039 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:06.038+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:06.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:06.251+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:06.252511+0000) 2022-04-03T23:09:06.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:06.294+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:07.068 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:07.067+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:07.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:07.251+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:07.252718+0000) 2022-04-03T23:09:07.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:07.280+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:08.101 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:08.099+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:08.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:08.252+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:08.252941+0000) 2022-04-03T23:09:08.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:08.279+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:09.104 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:09.102+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:09.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:09.252+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:09.253147+0000) 2022-04-03T23:09:09.255 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:09.253+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:10.110 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:10.109+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:10.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:10.230+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:10.254 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:10.252+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:10.253365+0000) 2022-04-03T23:09:11.070 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:11.069+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:11.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:11.225+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:11.254 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:11.252+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:11.253557+0000) 2022-04-03T23:09:11.887 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:09:11.891 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:09:12.097 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:12.095+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:12.205 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:12.203+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:12.254 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:12.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:12.253809+0000) 2022-04-03T23:09:13.087 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:13.086+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:13.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:13.154+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:13.254 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:13.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:13.254011+0000) 2022-04-03T23:09:14.129 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:14.128+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:14.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:14.169+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:14.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:14.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:14.254228+0000) 2022-04-03T23:09:15.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:15.172+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:15.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:15.202+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:15.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:15.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:15.254440+0000) 2022-04-03T23:09:16.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:16.156+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:16.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:16.158+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:16.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:16.254+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:16.254636+0000) 2022-04-03T23:09:17.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:17.173+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:17.206 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:17.205+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:17.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:17.254+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:17.254883+0000) 2022-04-03T23:09:18.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:18.158+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:18.181 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:18.180+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:18.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:18.254+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:18.255088+0000) 2022-04-03T23:09:19.142 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:19.141+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:19.144 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:19.144+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:19.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:19.254+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:19.255266+0000) 2022-04-03T23:09:20.119 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:20.117+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:20.135 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:20.134+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:20.256 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:20.254+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:20.255471+0000) 2022-04-03T23:09:21.041 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:09:21.039+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:09:21.107 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:21.106+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:21.118 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:21.117+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:21.256 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:21.255+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:21.255653+0000) 2022-04-03T23:09:22.065 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:22.064+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:22.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:22.138+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:22.256 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:22.255+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:22.255862+0000) 2022-04-03T23:09:23.097 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:23.096+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:23.118 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:23.117+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:23.256 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:23.255+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:23.256086+0000) 2022-04-03T23:09:24.131 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:24.130+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:24.149 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:24.148+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:24.257 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:24.255+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:24.256326+0000) 2022-04-03T23:09:25.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:25.102+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:25.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:25.165+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:25.257 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:25.256+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:25.256533+0000) 2022-04-03T23:09:26.078 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:26.077+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:26.205 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:26.205+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 11 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:26.257 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:26.256+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:26.256740+0000) 2022-04-03T23:09:27.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:27.096+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:27.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:27.237+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:27.257 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:27.256+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:27.256982+0000) 2022-04-03T23:09:28.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:28.126+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:28.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:28.235+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:28.257 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:28.256+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:28.257194+0000) 2022-04-03T23:09:29.142 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:29.140+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:29.258 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:29.257+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:29.257395+0000) 2022-04-03T23:09:29.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:29.281+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:30.155 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:30.153+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:30.258 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:30.257+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:30.257614+0000) 2022-04-03T23:09:30.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:30.309+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:31.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:31.123+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:31.258 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:31.257+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:31.257851+0000) 2022-04-03T23:09:31.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:31.301+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:32.144 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:32.142+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:32.258 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:32.257+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:32.258053+0000) 2022-04-03T23:09:32.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:32.275+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:33.144 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:33.142+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:33.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:33.249+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:33.258 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:33.257+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:33.258279+0000) 2022-04-03T23:09:34.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:34.123+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:34.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:34.235+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:34.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:34.257+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:34.258441+0000) 2022-04-03T23:09:35.146 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:35.144+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:35.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:35.188+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:35.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:35.258+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:35.258641+0000) 2022-04-03T23:09:36.056 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:09:36.055+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:09:36.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:36.095+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:36.182 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:36.181+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:36.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:36.258+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:36.258881+0000) 2022-04-03T23:09:37.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:37.065+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:37.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:37.188+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:37.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:37.258+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:37.259076+0000) 2022-04-03T23:09:38.027 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:38.025+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:38.221 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:38.219+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:38.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:38.259+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:38.259293+0000) 2022-04-03T23:09:39.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:39.046+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:39.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:39.202+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:39.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:39.259+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:39.259496+0000) 2022-04-03T23:09:40.068 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:40.066+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:40.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:40.251+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:40.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:40.259+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:40.259657+0000) 2022-04-03T23:09:41.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:41.078+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:41.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:41.258+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:41.259812+0000) 2022-04-03T23:09:41.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:41.286+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:42.026 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:09:42.030 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:09:42.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:42.036+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:42.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:42.258+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:42.260036+0000) 2022-04-03T23:09:42.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:42.287+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:43.052 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:43.050+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:43.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:43.259+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:43.260214+0000) 2022-04-03T23:09:43.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:43.316+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:44.036 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:44.034+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:44.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:44.259+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:44.260433+0000) 2022-04-03T23:09:44.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:44.273+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:44.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:44.991+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:45.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:45.259+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:45.260640+0000) 2022-04-03T23:09:45.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:45.318+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:46.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:46.002+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:46.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:46.259+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:46.260852+0000) 2022-04-03T23:09:46.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:46.327+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:46.961 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:46.960+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:47.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:47.260+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:47.261052+0000) 2022-04-03T23:09:47.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:47.296+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:47.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:47.929+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:48.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:48.248+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:48.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:48.260+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:48.261254+0000) 2022-04-03T23:09:48.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:48.963+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:49.262 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:49.260+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:49.261459+0000) 2022-04-03T23:09:49.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:49.288+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:49.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:49.949+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:50.262 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:50.260+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:50.261671+0000) 2022-04-03T23:09:50.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:50.295+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:50.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:50.951+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:51.072 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:09:51.071+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:09:51.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:51.253+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:51.262 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:51.261+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:51.261897+0000) 2022-04-03T23:09:51.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:51.986+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:52.252 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:52.250+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:52.262 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:52.261+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:52.262072+0000) 2022-04-03T23:09:52.945 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:52.943+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:53.263 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:53.261+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:53.262274+0000) 2022-04-03T23:09:53.283 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:53.281+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:53.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:53.925+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:54.263 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:54.261+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:54.262442+0000) 2022-04-03T23:09:54.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:54.303+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:54.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:54.876+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:55.263 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:55.261+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:55.262641+0000) 2022-04-03T23:09:55.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:55.348+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:55.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:55.898+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:56.263 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:56.262+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:56.262846+0000) 2022-04-03T23:09:56.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:56.363+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:56.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:56.871+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:57.263 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:57.262+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:57.263070+0000) 2022-04-03T23:09:57.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:57.342+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:57.909 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:57.908+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:58.264 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:58.262+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:58.263292+0000) 2022-04-03T23:09:58.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:58.369+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:58.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:58.922+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:09:59.264 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:09:59.262+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:09:59.263526+0000) 2022-04-03T23:09:59.332 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:09:59.330+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:09:59.889 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:09:59.887+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:00.264 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:00.262+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:00.263724+0000) 2022-04-03T23:10:00.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:00.291+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:00.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:00.879+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:01.264 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:01.263+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:01.263994+0000) 2022-04-03T23:10:01.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:01.287+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:01.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:01.852+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:02.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:02.263+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:02.264287+0000) 2022-04-03T23:10:02.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:02.265+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:02.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:02.859+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:03.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:03.236+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:03.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:03.263+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:03.264527+0000) 2022-04-03T23:10:03.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:03.839+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:04.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:04.227+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:04.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:04.264+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:04.264816+0000) 2022-04-03T23:10:04.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:04.877+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:05.202 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:05.201+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:05.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:05.264+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:05.265097+0000) 2022-04-03T23:10:05.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:05.891+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:06.088 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:10:06.087+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:10:06.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:06.192+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:06.266 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:06.264+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:06.265348+0000) 2022-04-03T23:10:06.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:06.932+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:07.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:07.179+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:07.266 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:07.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:07.265594+0000) 2022-04-03T23:10:07.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:07.981+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:08.147 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:08.145+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:08.266 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:08.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:08.265861+0000) 2022-04-03T23:10:08.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:08.990+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:09.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:09.184+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:09.267 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:09.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:09.266105+0000) 2022-04-03T23:10:09.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:09.991+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:10.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:10.180+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:10.266 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:10.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:10.266358+0000) 2022-04-03T23:10:11.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:11.027+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:11.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:11.195+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:11.267 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:11.266+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:11.266611+0000) 2022-04-03T23:10:11.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:11.989+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:12.175 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:10:12.180 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:10:12.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:12.214+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:12.267 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:12.266+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:12.266850+0000) 2022-04-03T23:10:12.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:12.982+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:13.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:13.222+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:13.267 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:13.266+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:13.267001+0000) 2022-04-03T23:10:13.946 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:13.945+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:14.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:14.174+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:14.267 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:14.266+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:14.267145+0000) 2022-04-03T23:10:14.980 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:14.978+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:15.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:15.214+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:15.268 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:15.267+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:15.267362+0000) 2022-04-03T23:10:15.962 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:15.961+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:16.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:16.241+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:16.268 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:16.267+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:16.267578+0000) 2022-04-03T23:10:16.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:16.998+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:17.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:17.193+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:17.268 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:17.267+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:17.267787+0000) 2022-04-03T23:10:18.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:18.002+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:18.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:18.221+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:18.268 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:18.267+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:18.268011+0000) 2022-04-03T23:10:19.040 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:19.040+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:19.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:19.213+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:19.269 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:19.267+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:19.268181+0000) 2022-04-03T23:10:20.090 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:20.089+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:20.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:20.215+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:20.269 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:20.268+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:20.268392+0000) 2022-04-03T23:10:21.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:21.082+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:21.104 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:10:21.102+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:10:21.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:21.237+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:21.269 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:21.268+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:21.268660+0000) 2022-04-03T23:10:22.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:22.074+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:22.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:22.217+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:22.322 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:22.268+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:22.268860+0000) 2022-04-03T23:10:23.078 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:23.077+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:23.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:23.249+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:23.269 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:23.268+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:23.269030+0000) 2022-04-03T23:10:24.040 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:24.039+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:24.270 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:24.269+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:24.269247+0000) 2022-04-03T23:10:24.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:24.286+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:25.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:25.077+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:25.270 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:25.269+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:25.269492+0000) 2022-04-03T23:10:25.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:25.298+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:26.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:26.078+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:26.270 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:26.268+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:26.269674+0000) 2022-04-03T23:10:26.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:26.329+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:27.101 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:27.099+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:27.270 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:27.268+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:27.269889+0000) 2022-04-03T23:10:27.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:27.302+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:28.056 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:28.054+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:28.270 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:28.269+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:28.270136+0000) 2022-04-03T23:10:28.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:28.289+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:29.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:29.072+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:29.271 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:29.269+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:29.270373+0000) 2022-04-03T23:10:29.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:29.300+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:30.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:30.065+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:30.271 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:30.269+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:30.270652+0000) 2022-04-03T23:10:30.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:30.281+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:31.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:31.036+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:31.271 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:31.269+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:31.270904+0000) 2022-04-03T23:10:31.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:31.288+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:31.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:31.990+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:32.271 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:32.270+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:32.271127+0000) 2022-04-03T23:10:32.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:32.273+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:32.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:32.948+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:33.272 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:33.270+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:33.271334+0000) 2022-04-03T23:10:33.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:33.307+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:33.976 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:33.974+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:34.272 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:34.270+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:34.271507+0000) 2022-04-03T23:10:34.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:34.286+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:34.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:34.988+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:35.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:35.268+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:35.272 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:35.270+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:35.271656+0000) 2022-04-03T23:10:36.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:36.007+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:36.120 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:10:36.118+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:10:36.272 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:36.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:36.271876+0000) 2022-04-03T23:10:36.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:36.289+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:36.996 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:36.994+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:37.272 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:37.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:37.272077+0000) 2022-04-03T23:10:37.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:37.311+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:38.022 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:38.020+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:38.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:38.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:38.272312+0000) 2022-04-03T23:10:38.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:38.285+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:39.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:39.052+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:39.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:39.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:39.272459+0000) 2022-04-03T23:10:39.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:39.323+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:40.058 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:40.056+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:40.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:40.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:40.272614+0000) 2022-04-03T23:10:40.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:40.283+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:41.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:41.035+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:41.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:41.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:41.272869+0000) 2022-04-03T23:10:41.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:41.300+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:42.022 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:42.021+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:42.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:42.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:42.273114+0000) 2022-04-03T23:10:42.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:42.322+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:42.339 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:10:42.343 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:10:43.053 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:43.052+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:43.274 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:43.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:43.273288+0000) 2022-04-03T23:10:43.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:43.359+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:44.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:44.014+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:44.274 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:44.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:44.273471+0000) 2022-04-03T23:10:44.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:44.330+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:45.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:45.034+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:45.274 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:45.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:45.273634+0000) 2022-04-03T23:10:45.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:45.312+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:46.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:46.008+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:46.274 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:46.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:46.273847+0000) 2022-04-03T23:10:46.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:46.328+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:46.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:46.965+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:47.274 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:47.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:47.274010+0000) 2022-04-03T23:10:47.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:47.313+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:47.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:47.936+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:48.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:48.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:48.274245+0000) 2022-04-03T23:10:48.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:48.319+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:48.986 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:48.985+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:49.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:49.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:49.274486+0000) 2022-04-03T23:10:49.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:49.291+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:50.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:50.006+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:50.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:50.274+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:50.274703+0000) 2022-04-03T23:10:50.284 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:50.282+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:50.967 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:50.965+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:51.135 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:10:51.134+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:10:51.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:51.274+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:51.274871+0000) 2022-04-03T23:10:51.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:51.307+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:51.978 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:51.976+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:52.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:52.274+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:52.275016+0000) 2022-04-03T23:10:52.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:52.321+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:52.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:52.998+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:53.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:53.274+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:53.275135+0000) 2022-04-03T23:10:53.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:53.289+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:53.963 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:53.960+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:54.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:54.274+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:54.275328+0000) 2022-04-03T23:10:54.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:54.338+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:55.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:55.005+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:55.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:55.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:55.275575+0000) 2022-04-03T23:10:55.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:55.293+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:56.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:56.038+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:56.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:56.262+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:56.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:56.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:56.275840+0000) 2022-04-03T23:10:57.052 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:57.050+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:57.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:57.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:57.276013+0000) 2022-04-03T23:10:57.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:57.279+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:58.078 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:58.076+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:58.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:58.241+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:10:58.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:58.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:58.276131+0000) 2022-04-03T23:10:59.086 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:10:59.085+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:10:59.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:10:59.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:10:59.276273+0000) 2022-04-03T23:10:59.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:10:59.284+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:00.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:00.115+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:00.277 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:00.276+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:00.276471+0000) 2022-04-03T23:11:00.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:00.318+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:01.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:01.140+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:01.277 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:01.276+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:01.276653+0000) 2022-04-03T23:11:01.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:01.348+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:02.166 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:02.165+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:02.277 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:02.276+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:02.276819+0000) 2022-04-03T23:11:02.376 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:02.374+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:03.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:03.133+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:03.277 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:03.276+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:03.277021+0000) 2022-04-03T23:11:03.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:03.410+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:04.087 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:04.086+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:04.278 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:04.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:04.277300+0000) 2022-04-03T23:11:04.364 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:04.363+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:05.133 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:05.132+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:05.278 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:05.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:05.277540+0000) 2022-04-03T23:11:05.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:05.368+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:06.130 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:06.129+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:06.151 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:11:06.150+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:11:06.278 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:06.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:06.277815+0000) 2022-04-03T23:11:06.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:06.394+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:07.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:07.127+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:07.278 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:07.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:07.278017+0000) 2022-04-03T23:11:07.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:07.373+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:08.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:08.125+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:08.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:08.278+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:08.278269+0000) 2022-04-03T23:11:08.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:08.383+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:09.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:09.152+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:09.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:09.278+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:09.278450+0000) 2022-04-03T23:11:09.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:09.362+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:10.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:10.124+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:10.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:10.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:10.278588+0000) 2022-04-03T23:11:10.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:10.398+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:11.162 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:11.160+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:11.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:11.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:11.278794+0000) 2022-04-03T23:11:11.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:11.376+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:12.149 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:12.148+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:12.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:12.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:12.278919+0000) 2022-04-03T23:11:12.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:12.388+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:12.460 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:11:12.465 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:11:13.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:13.152+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:13.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:13.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:13.279025+0000) 2022-04-03T23:11:13.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:13.353+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:14.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:14.182+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:14.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:14.278+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:14.279154+0000) 2022-04-03T23:11:14.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:14.320+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:15.157 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:15.155+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:15.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:15.278+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:15.279385+0000) 2022-04-03T23:11:15.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:15.297+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:16.108 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:16.107+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:16.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:16.278+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:16.279553+0000) 2022-04-03T23:11:16.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:16.324+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:17.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:17.082+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:17.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:17.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:17.279726+0000) 2022-04-03T23:11:17.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:17.296+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:18.110 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:18.109+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:18.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:18.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:18.279888+0000) 2022-04-03T23:11:18.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:18.293+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:19.091 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:19.090+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:19.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:19.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:19.280039+0000) 2022-04-03T23:11:19.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:19.283+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:20.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:20.139+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:20.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:20.273+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:20.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:20.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:20.280248+0000) 2022-04-03T23:11:21.152 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:21.150+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:21.167 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:11:21.166+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:11:21.281 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:21.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:21.280474+0000) 2022-04-03T23:11:21.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:21.314+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:22.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:22.139+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:22.281 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:22.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:22.280646+0000) 2022-04-03T23:11:22.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:22.294+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:23.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:23.173+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:23.281 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:23.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:23.280862+0000) 2022-04-03T23:11:23.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:23.297+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:24.200 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:24.198+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:24.281 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:24.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:24.281063+0000) 2022-04-03T23:11:24.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:24.281+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:25.225 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:25.223+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:25.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:25.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:25.281252+0000) 2022-04-03T23:11:25.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:25.319+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:26.238 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:26.236+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:26.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:26.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:26.281494+0000) 2022-04-03T23:11:26.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:26.347+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:27.251 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:27.249+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:27.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:27.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:27.281705+0000) 2022-04-03T23:11:27.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:27.382+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:28.233 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:28.230+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:28.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:28.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:28.281887+0000) 2022-04-03T23:11:28.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:28.418+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:29.261 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:29.258+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:29.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:29.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:29.282041+0000) 2022-04-03T23:11:29.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:29.437+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:30.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:30.263+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:30.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:30.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:30.282209+0000) 2022-04-03T23:11:30.405 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:30.403+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:31.283 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:31.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:31.282349+0000) 2022-04-03T23:11:31.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:31.313+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:31.401 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:31.389+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:32.283 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:32.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:32.282509+0000) 2022-04-03T23:11:32.313 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:32.312+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:32.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:32.410+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:33.283 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:33.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:33.282680+0000) 2022-04-03T23:11:33.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:33.334+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:33.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:33.438+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:34.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:34.282+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:34.282910+0000) 2022-04-03T23:11:34.318 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:34.316+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:34.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:34.485+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:35.283 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:35.282+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:35.283084+0000) 2022-04-03T23:11:35.298 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:35.296+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:35.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:35.517+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:36.179 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:11:36.177+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:11:36.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:36.282+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:36.283287+0000) 2022-04-03T23:11:36.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:36.318+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:36.522 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:36.520+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:37.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:37.282+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:37.283486+0000) 2022-04-03T23:11:37.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:37.367+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:37.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:37.540+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:38.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:38.283+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:38.283709+0000) 2022-04-03T23:11:38.412 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:38.410+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:38.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:38.552+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:39.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:39.283+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:39.283855+0000) 2022-04-03T23:11:39.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:39.436+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:39.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:39.548+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:40.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:40.283+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:40.283997+0000) 2022-04-03T23:11:40.442 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:40.440+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:40.570 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:40.568+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:41.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:41.283+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:41.284164+0000) 2022-04-03T23:11:41.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:41.483+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:41.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:41.533+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:42.285 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:42.283+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:42.284368+0000) 2022-04-03T23:11:42.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:42.491+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:42.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:42.530+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:42.787 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:11:42.791 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:11:43.285 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:43.284+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:43.284621+0000) 2022-04-03T23:11:43.482 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:43.480+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:43.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:43.555+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:44.285 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:44.284+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:44.284873+0000) 2022-04-03T23:11:44.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:44.502+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:44.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:44.510+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:45.285 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:45.284+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:45.285002+0000) 2022-04-03T23:11:45.520 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:45.519+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:45.530 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:45.528+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:46.286 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:46.284+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:46.285161+0000) 2022-04-03T23:11:46.500 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:46.499+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:46.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:46.510+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:47.286 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:47.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:47.285311+0000) 2022-04-03T23:11:47.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:47.489+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:47.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:47.525+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:48.286 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:48.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:48.285540+0000) 2022-04-03T23:11:48.469 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:48.468+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:48.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:48.497+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:49.286 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:49.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:49.285833+0000) 2022-04-03T23:11:49.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:49.459+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:49.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:49.523+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:50.286 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:50.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:50.286073+0000) 2022-04-03T23:11:50.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:50.491+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:50.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:50.555+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:51.184 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:11:51.182+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:11:51.287 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:51.286+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:51.286265+0000) 2022-04-03T23:11:51.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:51.527+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:51.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:51.540+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:52.287 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:52.286+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:52.286470+0000) 2022-04-03T23:11:52.522 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:52.520+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:52.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:52.530+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:53.287 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:53.286+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:53.286611+0000) 2022-04-03T23:11:53.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:53.483+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:53.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:53.530+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:54.287 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:54.286+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:54.286862+0000) 2022-04-03T23:11:54.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:54.532+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:54.550 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:54.549+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:55.287 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:55.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:55.287043+0000) 2022-04-03T23:11:55.493 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:55.491+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:55.519 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:55.517+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:56.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:56.286+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:56.287230+0000) 2022-04-03T23:11:56.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:56.490+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:56.498 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:56.496+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:57.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:57.286+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:57.287336+0000) 2022-04-03T23:11:57.453 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:57.452+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:57.468 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:57.467+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:58.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:58.286+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:58.287544+0000) 2022-04-03T23:11:58.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:58.429+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:58.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:58.507+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:11:59.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:11:59.286+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:11:59.287739+0000) 2022-04-03T23:11:59.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:11:59.393+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:11:59.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:11:59.517+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:00.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:00.287+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:00.287946+0000) 2022-04-03T23:12:00.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:00.367+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:00.472 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:00.471+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:01.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:01.287+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:01.288135+0000) 2022-04-03T23:12:01.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:01.413+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:01.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:01.435+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:02.289 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:02.287+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:02.288382+0000) 2022-04-03T23:12:02.396 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:02.395+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:02.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:02.436+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:03.289 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:03.287+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:03.288579+0000) 2022-04-03T23:12:03.437 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:03.436+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:03.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:03.466+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:04.289 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:04.287+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:04.288762+0000) 2022-04-03T23:12:04.442 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:04.440+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:04.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:04.484+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:05.289 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:05.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:05.288927+0000) 2022-04-03T23:12:05.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:05.398+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:05.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:05.500+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:06.197 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:12:06.195+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:12:06.289 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:06.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:06.289151+0000) 2022-04-03T23:12:06.376 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:06.375+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:06.505 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:06.504+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:07.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:07.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:07.289311+0000) 2022-04-03T23:12:07.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:07.376+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:07.457 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:07.456+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:08.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:08.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:08.289478+0000) 2022-04-03T23:12:08.421 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:08.420+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:08.496 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:08.495+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:09.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:09.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:09.289655+0000) 2022-04-03T23:12:09.429 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:09.428+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:09.540 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:09.539+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:10.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:10.289+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:10.289897+0000) 2022-04-03T23:12:10.429 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:10.428+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:10.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:10.553+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:11.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:11.289+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:11.290130+0000) 2022-04-03T23:12:11.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:11.418+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:11.565 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:11.564+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:12.291 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:12.289+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:12.290295+0000) 2022-04-03T23:12:12.451 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:12.450+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:12.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:12.542+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:12.927 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:12:12.931 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:12:13.291 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:13.289+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:13.290501+0000) 2022-04-03T23:12:13.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:13.487+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:13.589 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:13.588+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:14.291 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:14.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:14.290688+0000) 2022-04-03T23:12:14.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:14.443+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:14.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:14.611+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:15.291 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:15.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:15.290874+0000) 2022-04-03T23:12:15.412 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:15.410+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:15.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:15.642+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:16.291 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:16.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:16.290998+0000) 2022-04-03T23:12:16.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:16.389+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:16.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:16.633+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:17.291 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:17.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:17.291132+0000) 2022-04-03T23:12:17.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:17.379+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:17.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:17.611+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:18.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:18.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:18.291269+0000) 2022-04-03T23:12:18.364 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:18.362+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:18.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:18.581+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:19.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:19.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:19.291481+0000) 2022-04-03T23:12:19.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:19.329+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:19.570 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:19.567+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:20.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:20.291+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:20.291660+0000) 2022-04-03T23:12:20.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:20.361+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:20.606 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:20.605+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:21.212 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:12:21.211+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:12:21.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:21.291+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:21.291921+0000) 2022-04-03T23:12:21.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:21.323+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:21.600 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:21.598+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:22.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:22.285+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:22.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:22.291+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:22.292105+0000) 2022-04-03T23:12:22.596 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:22.594+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:23.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:23.264+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:23.293 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:23.291+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:23.292255+0000) 2022-04-03T23:12:23.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:23.626+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:24.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:24.275+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:24.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:24.291+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:24.292437+0000) 2022-04-03T23:12:24.608 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:24.606+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:25.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:25.254+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:25.293 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:25.291+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:25.292598+0000) 2022-04-03T23:12:25.590 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:25.588+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:26.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:26.280+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:26.293 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:26.292+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:26.292836+0000) 2022-04-03T23:12:26.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:26.581+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:27.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:27.236+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:27.293 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:27.292+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:27.293057+0000) 2022-04-03T23:12:27.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:27.534+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:28.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:28.228+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:28.294 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:28.292+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:28.293203+0000) 2022-04-03T23:12:28.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:28.519+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:29.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:29.237+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:29.294 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:29.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:29.293376+0000) 2022-04-03T23:12:29.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:29.513+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:30.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:30.197+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:30.294 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:30.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:30.293553+0000) 2022-04-03T23:12:30.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:30.546+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:31.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:31.209+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:31.294 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:31.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:31.293761+0000) 2022-04-03T23:12:31.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:31.578+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:32.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:32.232+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:32.294 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:32.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:32.293925+0000) 2022-04-03T23:12:32.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:32.547+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:33.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:33.254+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:33.294 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:33.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:33.294095+0000) 2022-04-03T23:12:33.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:33.532+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:34.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:34.246+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:34.295 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:34.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:34.294243+0000) 2022-04-03T23:12:34.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:34.558+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:35.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:35.273+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:35.295 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:35.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:35.294433+0000) 2022-04-03T23:12:35.578 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:35.576+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:36.228 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:12:36.227+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:12:36.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:36.272+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:36.295 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:36.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:36.294643+0000) 2022-04-03T23:12:36.599 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:36.597+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:37.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:37.262+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:37.295 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:37.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:37.294815+0000) 2022-04-03T23:12:37.612 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:37.610+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:38.295 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:38.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:38.294994+0000) 2022-04-03T23:12:38.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:38.309+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:38.598 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:38.596+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:39.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:39.280+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:39.295 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:39.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:39.295169+0000) 2022-04-03T23:12:39.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:39.627+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:40.296 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:40.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:40.295349+0000) 2022-04-03T23:12:40.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:40.308+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:40.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:40.593+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:41.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:41.290+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:41.296 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:41.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:41.295493+0000) 2022-04-03T23:12:41.554 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:41.552+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:42.296 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:42.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:42.295675+0000) 2022-04-03T23:12:42.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:42.335+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:42.526 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:42.525+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:43.075 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:12:43.079 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:12:43.296 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:43.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:43.295951+0000) 2022-04-03T23:12:43.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:43.363+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:43.524 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:43.522+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:44.297 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:44.295+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:44.296176+0000) 2022-04-03T23:12:44.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:44.391+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:44.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:44.558+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:45.297 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:45.295+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:45.296356+0000) 2022-04-03T23:12:45.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:45.344+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:45.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:45.604+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:46.297 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:46.295+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:46.296564+0000) 2022-04-03T23:12:46.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:46.366+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:46.636 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:46.635+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:47.297 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:47.295+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:47.296732+0000) 2022-04-03T23:12:47.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:47.350+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:47.636 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:47.634+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:48.297 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:48.295+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:48.296884+0000) 2022-04-03T23:12:48.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:48.301+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:48.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:48.677+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:49.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:49.263+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:49.298 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:49.296+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:49.297079+0000) 2022-04-03T23:12:49.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:49.681+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:50.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:50.227+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:50.298 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:50.296+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:50.297223+0000) 2022-04-03T23:12:50.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:50.697+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:51.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:51.196+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:51.230 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:12:51.228+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:12:51.298 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:51.296+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:51.297343+0000) 2022-04-03T23:12:51.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:51.730+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:52.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:52.222+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:52.298 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:52.296+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:52.297526+0000) 2022-04-03T23:12:52.763 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:52.762+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:53.206 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:53.204+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:53.298 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:53.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:53.297759+0000) 2022-04-03T23:12:53.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:53.731+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:54.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:54.188+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:54.298 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:54.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:54.298020+0000) 2022-04-03T23:12:54.774 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:54.773+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:55.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:55.215+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:55.299 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:55.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:55.298233+0000) 2022-04-03T23:12:55.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:55.800+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:56.174 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:56.173+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:56.299 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:56.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:56.298443+0000) 2022-04-03T23:12:56.754 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:56.753+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:57.191 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:57.189+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:57.299 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:57.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:57.298697+0000) 2022-04-03T23:12:57.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:57.802+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:58.144 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:58.143+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:58.299 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:58.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:58.298883+0000) 2022-04-03T23:12:58.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:58.847+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:12:59.120 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:12:59.119+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:12:59.299 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:12:59.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:12:59.299074+0000) 2022-04-03T23:12:59.850 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:12:59.849+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:00.102 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:00.101+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:00.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:00.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:00.299264+0000) 2022-04-03T23:13:00.852 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:00.851+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:01.110 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:01.109+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:01.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:01.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:01.299467+0000) 2022-04-03T23:13:01.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:01.862+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:02.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:02.084+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:02.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:02.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:02.299683+0000) 2022-04-03T23:13:02.895 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:02.894+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:03.108 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:03.107+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:03.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:03.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:03.299882+0000) 2022-04-03T23:13:03.902 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:03.901+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:04.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:04.104+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:04.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:04.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:04.300089+0000) 2022-04-03T23:13:04.915 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:04.914+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:05.146 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:05.145+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:05.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:05.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:05.300306+0000) 2022-04-03T23:13:05.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:05.927+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:06.130 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:06.129+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:06.246 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:13:06.244+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:13:06.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:06.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:06.300516+0000) 2022-04-03T23:13:06.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:06.948+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:07.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:07.084+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:07.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:07.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:07.300714+0000) 2022-04-03T23:13:07.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:07.946+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:08.098 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:08.097+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:08.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:08.300+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:08.300871+0000) 2022-04-03T23:13:08.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:08.948+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:09.083 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:09.082+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:09.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:09.300+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:09.301027+0000) 2022-04-03T23:13:09.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:09.990+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:10.042 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:10.040+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:10.302 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:10.300+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:10.301201+0000) 2022-04-03T23:13:11.030 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:11.028+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:11.059 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:11.057+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:11.302 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:11.300+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:11.301471+0000) 2022-04-03T23:13:12.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:12.052+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:12.093 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:12.091+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:12.302 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:12.301+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:12.301703+0000) 2022-04-03T23:13:13.005 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:13.003+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:13.092 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:13.090+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:13.173 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:13:13.177 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:13:13.302 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:13.301+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:13.301929+0000) 2022-04-03T23:13:14.011 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:14.009+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:14.053 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:14.051+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:14.303 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:14.301+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:14.302140+0000) 2022-04-03T23:13:15.039 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:15.037+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:15.051 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:15.049+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:15.303 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:15.301+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:15.302406+0000) 2022-04-03T23:13:16.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:16.022+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:16.054 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:16.052+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:16.303 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:16.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:16.302578+0000) 2022-04-03T23:13:17.022 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:17.020+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:17.034 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:17.032+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:17.303 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:17.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:17.302729+0000) 2022-04-03T23:13:18.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:18.017+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:18.066 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:18.064+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:18.303 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:18.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:18.302966+0000) 2022-04-03T23:13:18.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:18.979+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:19.035 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:19.033+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:19.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:19.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:19.303156+0000) 2022-04-03T23:13:19.932 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:19.930+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:20.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:20.036+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:20.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:20.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:20.303322+0000) 2022-04-03T23:13:20.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:20.951+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:21.001 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:20.999+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:21.262 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:13:21.260+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:13:21.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:21.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:21.303427+0000) 2022-04-03T23:13:21.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:21.988+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:22.032 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:22.030+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:22.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:22.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:22.303549+0000) 2022-04-03T23:13:22.954 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:22.952+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:23.071 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:23.069+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:23.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:23.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:23.303754+0000) 2022-04-03T23:13:24.002 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:24.000+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:24.080 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:24.078+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:24.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:24.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:24.303973+0000) 2022-04-03T23:13:24.963 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:24.961+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:25.047 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:25.045+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:25.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:25.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:25.304151+0000) 2022-04-03T23:13:25.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:25.975+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:26.088 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:26.086+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:26.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:26.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:26.304315+0000) 2022-04-03T23:13:26.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:26.986+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:27.086 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:27.083+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:27.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:27.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:27.304532+0000) 2022-04-03T23:13:28.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:28.032+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:28.109 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:28.107+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:28.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:28.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:28.304734+0000) 2022-04-03T23:13:29.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:29.007+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:29.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:29.113+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:29.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:29.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:29.304927+0000) 2022-04-03T23:13:30.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:30.056+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:30.129 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:30.127+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:30.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:30.304+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:30.305047+0000) 2022-04-03T23:13:31.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:31.083+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:31.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:31.166+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:31.306 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:31.304+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:31.305269+0000) 2022-04-03T23:13:32.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:32.073+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:32.136 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:32.134+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:32.306 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:32.304+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:32.305426+0000) 2022-04-03T23:13:33.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:33.096+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:33.113 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:33.112+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:33.306 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:33.304+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:33.305558+0000) 2022-04-03T23:13:34.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:34.139+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:34.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:34.154+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:34.306 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:34.304+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:34.305757+0000) 2022-04-03T23:13:35.116 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:35.114+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:35.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:35.158+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:35.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:35.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:35.306029+0000) 2022-04-03T23:13:36.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:36.115+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:36.124 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:36.123+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:36.276 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:13:36.274+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:13:36.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:36.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:36.306268+0000) 2022-04-03T23:13:37.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:37.122+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:37.125 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:37.123+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:37.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:37.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:37.306445+0000) 2022-04-03T23:13:38.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:38.078+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:38.087 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:38.085+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:38.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:38.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:38.306607+0000) 2022-04-03T23:13:39.058 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:39.056+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:39.073 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:39.072+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:39.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:39.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:39.306869+0000) 2022-04-03T23:13:40.017 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:40.015+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:40.114 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:40.113+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:40.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:40.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:40.307087+0000) 2022-04-03T23:13:41.050 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:41.048+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:41.087 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:41.085+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:41.308 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:41.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:41.307261+0000) 2022-04-03T23:13:42.090 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:42.087+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:42.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:42.127+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:42.308 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:42.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:42.307469+0000) 2022-04-03T23:13:43.115 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:43.114+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:43.117 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:43.116+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:43.263 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:13:43.267 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:13:43.308 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:43.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:43.307680+0000) 2022-04-03T23:13:44.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:44.074+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:44.077 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:44.075+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:44.308 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:44.307+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:44.307888+0000) 2022-04-03T23:13:45.029 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:45.026+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:45.056 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:45.054+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:45.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:45.307+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:45.308121+0000) 2022-04-03T23:13:46.005 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:46.003+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:46.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:46.060+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:46.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:46.307+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:46.308266+0000) 2022-04-03T23:13:46.985 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:46.983+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:47.086 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:47.085+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:47.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:47.307+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:47.308502+0000) 2022-04-03T23:13:47.949 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:47.948+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:48.058 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:48.056+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:48.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:48.308+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:48.308720+0000) 2022-04-03T23:13:48.968 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:48.966+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:49.105 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:49.104+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:49.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:49.308+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:49.308941+0000) 2022-04-03T23:13:49.994 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:49.993+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:50.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:50.065+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:50.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:50.308+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:50.309129+0000) 2022-04-03T23:13:51.019 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:51.018+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:51.072 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:51.072+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:51.291 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:13:51.290+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:13:51.310 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:51.308+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:51.309325+0000) 2022-04-03T23:13:52.026 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:52.025+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:52.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:52.065+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:52.310 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:52.308+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:52.309461+0000) 2022-04-03T23:13:52.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:52.994+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:53.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:53.058+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:53.310 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:53.309+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:53.309604+0000) 2022-04-03T23:13:54.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:54.037+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:54.091 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:54.090+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:54.310 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:54.309+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:54.309819+0000) 2022-04-03T23:13:55.083 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:55.082+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:55.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:55.140+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:55.310 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:55.309+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:55.310042+0000) 2022-04-03T23:13:56.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:56.110+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:56.155 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:56.154+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:56.311 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:56.309+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:56.310250+0000) 2022-04-03T23:13:57.078 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:57.077+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:57.133 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:57.132+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:57.311 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:57.310+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:57.310430+0000) 2022-04-03T23:13:58.075 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:58.074+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:58.176 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:58.176+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:58.311 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:58.310+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:58.310576+0000) 2022-04-03T23:13:59.043 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:13:59.042+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:13:59.185 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:13:59.184+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:13:59.311 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:13:59.310+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:13:59.310838+0000) 2022-04-03T23:14:00.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:00.084+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:00.155 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:00.154+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:00.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:00.310+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:00.311008+0000) 2022-04-03T23:14:01.050 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:01.049+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:01.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:01.120+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:01.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:01.310+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:01.311209+0000) 2022-04-03T23:14:02.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:02.066+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:02.093 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:02.092+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:02.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:02.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:02.311418+0000) 2022-04-03T23:14:03.023 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:03.022+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:03.056 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:03.055+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:03.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:03.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:03.311625+0000) 2022-04-03T23:14:04.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:04.007+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:04.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:04.015+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:04.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:04.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:04.311744+0000) 2022-04-03T23:14:04.959 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:04.957+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:05.047 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:05.045+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:05.313 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:05.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:05.311949+0000) 2022-04-03T23:14:05.996 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:05.994+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:06.009 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:06.007+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:06.305 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:14:06.303+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:14:06.313 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:06.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:06.312194+0000) 2022-04-03T23:14:06.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:06.977+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:07.053 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:07.051+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:07.313 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:07.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:07.312421+0000) 2022-04-03T23:14:07.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:07.988+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:08.057 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:08.055+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:08.313 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:08.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:08.312581+0000) 2022-04-03T23:14:08.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:08.986+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:09.025 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:09.023+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:09.313 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:09.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:09.312727+0000) 2022-04-03T23:14:09.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:09.964+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:10.060 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:10.058+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:10.313 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:10.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:10.312929+0000) 2022-04-03T23:14:10.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:10.921+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:11.021 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:11.019+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:11.313 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:11.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:11.313102+0000) 2022-04-03T23:14:11.883 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:11.881+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:12.009 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:12.007+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:12.314 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:12.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:12.313225+0000) 2022-04-03T23:14:12.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:12.874+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:13.054 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:13.052+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:13.314 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:13.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:13.313422+0000) 2022-04-03T23:14:13.344 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:14:13.348 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:14:13.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:13.902+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:14.045 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:14.043+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:14.314 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:14.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:14.313631+0000) 2022-04-03T23:14:14.880 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:14.878+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:15.064 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:15.062+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:15.315 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:15.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:15.313886+0000) 2022-04-03T23:14:15.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:15.871+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:16.078 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:16.076+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:16.315 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:16.313+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:16.314114+0000) 2022-04-03T23:14:16.910 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:16.908+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:17.031 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:17.029+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:17.315 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:17.313+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:17.314357+0000) 2022-04-03T23:14:17.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:17.880+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:18.040 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:18.037+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:18.315 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:18.313+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:18.314612+0000) 2022-04-03T23:14:18.833 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:18.831+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:19.022 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:19.020+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:19.315 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:19.313+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:19.314798+0000) 2022-04-03T23:14:19.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:19.874+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:19.997 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:19.995+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:20.315 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:20.314+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:20.314984+0000) 2022-04-03T23:14:20.859 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:20.857+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:20.993 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:20.991+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:21.316 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:21.314+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:21.315185+0000) 2022-04-03T23:14:21.319 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:14:21.317+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:14:21.871 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:21.869+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:21.958 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:21.956+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:22.316 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:22.314+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:22.315391+0000) 2022-04-03T23:14:22.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:22.871+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:22.917 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:22.916+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:23.316 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:23.314+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:23.315601+0000) 2022-04-03T23:14:23.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:23.866+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:23.916 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:23.913+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:24.316 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:24.315+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:24.315873+0000) 2022-04-03T23:14:24.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:24.827+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:24.904 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:24.902+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:25.317 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:25.315+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:25.316094+0000) 2022-04-03T23:14:25.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:25.844+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:25.913 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:25.912+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:26.317 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:26.315+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:26.316335+0000) 2022-04-03T23:14:26.880 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:26.878+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:26.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:26.887+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:27.317 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:27.315+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:27.316582+0000) 2022-04-03T23:14:27.851 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:27.849+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:27.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:27.921+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:28.317 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:28.316+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:28.316837+0000) 2022-04-03T23:14:28.805 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:28.803+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:28.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:28.898+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:29.318 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:29.316+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:29.317052+0000) 2022-04-03T23:14:29.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:29.769+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:29.926 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:29.924+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:30.318 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:30.316+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:30.317251+0000) 2022-04-03T23:14:30.724 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:30.722+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:30.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:30.891+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:31.318 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:31.316+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:31.317515+0000) 2022-04-03T23:14:31.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:31.730+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:31.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:31.932+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:32.318 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:32.317+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:32.317712+0000) 2022-04-03T23:14:32.746 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:32.745+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:32.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:32.945+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:33.318 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:33.317+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:33.317934+0000) 2022-04-03T23:14:33.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:33.726+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:33.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:33.921+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:34.319 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:34.317+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:34.318132+0000) 2022-04-03T23:14:34.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:34.719+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:34.898 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:34.896+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:35.319 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:35.317+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:35.318374+0000) 2022-04-03T23:14:35.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:35.680+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:35.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:35.946+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:36.319 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:36.317+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:36.318597+0000) 2022-04-03T23:14:36.335 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:14:36.333+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:14:36.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:36.635+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:36.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:36.979+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:37.319 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:37.318+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:37.318846+0000) 2022-04-03T23:14:37.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:37.670+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:37.968 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:37.967+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:38.320 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:38.318+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:38.319076+0000) 2022-04-03T23:14:38.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:38.669+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:38.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:38.919+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:39.320 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:39.318+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:39.319320+0000) 2022-04-03T23:14:39.694 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:39.693+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:39.941 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:39.939+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:40.320 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:40.319+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:40.319546+0000) 2022-04-03T23:14:40.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:40.723+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:40.980 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:40.978+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:41.320 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:41.319+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:41.319799+0000) 2022-04-03T23:14:41.687 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:41.686+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:41.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:41.979+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:42.321 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:42.319+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:42.320024+0000) 2022-04-03T23:14:42.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:42.682+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:42.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:42.956+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:43.321 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:43.319+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:43.320296+0000) 2022-04-03T23:14:43.483 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:14:43.487 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:14:43.694 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:43.692+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:43.972 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:43.971+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:44.321 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:44.320+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:44.320549+0000) 2022-04-03T23:14:44.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:44.695+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:44.967 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:44.966+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:45.321 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:45.320+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:45.320816+0000) 2022-04-03T23:14:45.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:45.701+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:45.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:45.972+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:46.321 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:46.320+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:46.321020+0000) 2022-04-03T23:14:46.662 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:46.661+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:47.005 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:47.004+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:47.322 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:47.320+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:47.321255+0000) 2022-04-03T23:14:47.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:47.688+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:47.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:47.964+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:48.322 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:48.321+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:48.321487+0000) 2022-04-03T23:14:48.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:48.670+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:48.926 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:48.925+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:49.322 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:49.321+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:49.321725+0000) 2022-04-03T23:14:49.623 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:49.622+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:49.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:49.916+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:50.323 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:50.321+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:50.321953+0000) 2022-04-03T23:14:50.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:50.580+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:50.935 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:50.934+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:51.323 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:51.321+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:51.322180+0000) 2022-04-03T23:14:51.351 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:14:51.349+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:14:51.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:51.580+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:51.956 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:51.955+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:52.323 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:52.322+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:52.322412+0000) 2022-04-03T23:14:52.564 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:52.563+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:53.002 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:53.001+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:53.323 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:53.322+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:53.322683+0000) 2022-04-03T23:14:53.607 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:53.606+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:53.970 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:53.969+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:54.324 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:54.321+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:54.322939+0000) 2022-04-03T23:14:54.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:54.597+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:54.971 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:54.970+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:55.324 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:55.322+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:55.323206+0000) 2022-04-03T23:14:55.602 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:55.601+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:56.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:56.011+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:56.324 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:56.322+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:56.323397+0000) 2022-04-03T23:14:56.604 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:56.603+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:57.040 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:57.039+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:57.324 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:57.322+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:57.323614+0000) 2022-04-03T23:14:57.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:57.614+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:58.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:58.008+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:58.324 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:58.322+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:58.323868+0000) 2022-04-03T23:14:58.656 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:58.655+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:14:59.029 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:14:59.028+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:14:59.325 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:14:59.323+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:14:59.324122+0000) 2022-04-03T23:14:59.634 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:14:59.631+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:00.017 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:00.015+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:00.325 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:00.323+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:00.324353+0000) 2022-04-03T23:15:00.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:00.610+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:01.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:01.031+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:01.325 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:01.323+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:01.324612+0000) 2022-04-03T23:15:01.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:01.576+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:01.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:01.989+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:02.325 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:02.323+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:02.324828+0000) 2022-04-03T23:15:02.586 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:02.583+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:02.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:02.993+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:03.326 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:03.324+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:03.325001+0000) 2022-04-03T23:15:03.602 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:03.599+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:03.980 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:03.978+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:04.326 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:04.324+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:04.325154+0000) 2022-04-03T23:15:04.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:04.636+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:04.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:04.931+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:05.326 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:05.324+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:05.325368+0000) 2022-04-03T23:15:05.604 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:05.602+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:05.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:05.895+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:06.326 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:06.324+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:06.325605+0000) 2022-04-03T23:15:06.367 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:15:06.365+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:15:06.602 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:06.600+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:06.907 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:06.905+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:07.326 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:07.325+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:07.325857+0000) 2022-04-03T23:15:07.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:07.579+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:07.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:07.866+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:08.327 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:08.325+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:08.326100+0000) 2022-04-03T23:15:08.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:08.551+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:08.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:08.836+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:09.327 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:09.325+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:09.326304+0000) 2022-04-03T23:15:09.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:09.598+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:09.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:09.825+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:10.327 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:10.325+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:10.326542+0000) 2022-04-03T23:15:10.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:10.574+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:10.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:10.778+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:11.327 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:11.326+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:11.326838+0000) 2022-04-03T23:15:11.582 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:11.579+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:11.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:11.770+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:12.328 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:12.326+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:12.327024+0000) 2022-04-03T23:15:12.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:12.603+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:12.747 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:12.745+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:13.328 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:13.326+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:13.327219+0000) 2022-04-03T23:15:13.564 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:13.562+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:13.573 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:15:13.577 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:15:13.734 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:13.732+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:14.328 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:14.326+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:14.327457+0000) 2022-04-03T23:15:14.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:14.587+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:14.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:14.689+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:15.328 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:15.327+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:15.327633+0000) 2022-04-03T23:15:15.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:15.586+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:15.666 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:15.664+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:16.328 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:16.327+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:16.327875+0000) 2022-04-03T23:15:16.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:16.632+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:16.661 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:16.659+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:17.329 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:17.327+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:17.328034+0000) 2022-04-03T23:15:17.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:17.648+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:17.704 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:17.702+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:18.329 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:18.327+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:18.328191+0000) 2022-04-03T23:15:18.655 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:18.653+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:18.708 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:18.706+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:19.329 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:19.327+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:19.328404+0000) 2022-04-03T23:15:19.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:19.641+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:19.661 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:19.659+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:20.329 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:20.328+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:20.328636+0000) 2022-04-03T23:15:20.629 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:20.627+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:20.673 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:20.671+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:21.330 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:21.328+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:21.328839+0000) 2022-04-03T23:15:21.383 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:15:21.381+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:15:21.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:21.647+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:21.666 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:21.665+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:22.330 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:22.328+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:22.329045+0000) 2022-04-03T23:15:22.636 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:22.634+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:22.676 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:22.674+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:23.330 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:23.328+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:23.329254+0000) 2022-04-03T23:15:23.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:23.678+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:23.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:23.693+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:24.330 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:24.329+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:24.329460+0000) 2022-04-03T23:15:24.633 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:24.632+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:24.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:24.737+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:25.330 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:25.329+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:25.329672+0000) 2022-04-03T23:15:25.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:25.666+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:25.764 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:25.763+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:26.331 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:26.329+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:26.329909+0000) 2022-04-03T23:15:26.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:26.688+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:26.805 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:26.804+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:27.331 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:27.329+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:27.330172+0000) 2022-04-03T23:15:27.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:27.699+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:27.783 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:27.781+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:28.331 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:28.330+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:28.330359+0000) 2022-04-03T23:15:28.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:28.711+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:28.765 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:28.763+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:29.331 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:29.330+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:29.330567+0000) 2022-04-03T23:15:29.703 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:29.701+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:29.745 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:29.744+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:30.331 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:30.330+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:30.330807+0000) 2022-04-03T23:15:30.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:30.691+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:30.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:30.750+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:31.332 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:31.330+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:31.331051+0000) 2022-04-03T23:15:31.689 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:31.687+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:31.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:31.729+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:32.332 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:32.331+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:32.331294+0000) 2022-04-03T23:15:32.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:32.641+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:32.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:32.739+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:33.332 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:33.331+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:33.331516+0000) 2022-04-03T23:15:33.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:33.633+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:33.788 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:33.786+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:34.332 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:34.331+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:34.331744+0000) 2022-04-03T23:15:34.639 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:34.637+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:34.808 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:34.805+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:35.332 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:35.331+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:35.331949+0000) 2022-04-03T23:15:35.634 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:35.633+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:35.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:35.840+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:36.333 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:36.332+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:36.332132+0000) 2022-04-03T23:15:36.398 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:15:36.397+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:15:36.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:36.628+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:36.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:36.793+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:37.333 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:37.332+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:37.332340+0000) 2022-04-03T23:15:37.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:37.612+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:37.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:37.747+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:38.333 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:38.331+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:38.332597+0000) 2022-04-03T23:15:38.604 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:38.603+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:38.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:38.716+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:39.333 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:39.331+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:39.332780+0000) 2022-04-03T23:15:39.631 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:39.630+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:39.745 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:39.744+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:40.334 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:40.331+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:40.333018+0000) 2022-04-03T23:15:40.661 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:40.660+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:40.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:40.740+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:41.334 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:41.332+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:41.333301+0000) 2022-04-03T23:15:41.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:41.646+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:41.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:41.765+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:42.334 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:42.332+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:42.333479+0000) 2022-04-03T23:15:42.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:42.654+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:42.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:42.800+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:43.334 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:43.332+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:43.333696+0000) 2022-04-03T23:15:43.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:43.665+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:43.718 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:15:43.722 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:15:43.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:43.753+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:44.335 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:44.332+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:44.333972+0000) 2022-04-03T23:15:44.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:44.711+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:44.752 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:44.750+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:45.335 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:45.333+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:45.334190+0000) 2022-04-03T23:15:45.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:45.741+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:45.757 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:45.756+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:46.335 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:46.333+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:46.334402+0000) 2022-04-03T23:15:46.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:46.705+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:46.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:46.782+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:47.335 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:47.333+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:47.334642+0000) 2022-04-03T23:15:47.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:47.689+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:47.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:47.825+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:48.336 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:48.333+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:48.334882+0000) 2022-04-03T23:15:48.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:48.733+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:48.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:48.802+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:49.336 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:49.334+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:49.335125+0000) 2022-04-03T23:15:49.752 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:49.751+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:49.797 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:49.795+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:50.336 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:50.334+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:50.335369+0000) 2022-04-03T23:15:50.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:50.790+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:50.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:50.844+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:51.336 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:51.334+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:51.335631+0000) 2022-04-03T23:15:51.402 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:15:51.401+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:15:51.792 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:51.791+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:51.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:51.822+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:52.337 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:52.335+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:52.335846+0000) 2022-04-03T23:15:52.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:52.781+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:52.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:52.809+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:53.337 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:53.335+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:53.336011+0000) 2022-04-03T23:15:53.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:53.805+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:53.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:53.810+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:54.337 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:54.335+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:54.336207+0000) 2022-04-03T23:15:54.767 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:54.764+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:54.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:54.798+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:55.337 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:55.335+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:55.336423+0000) 2022-04-03T23:15:55.749 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:55.747+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:55.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:55.756+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:56.337 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:56.335+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:56.336662+0000) 2022-04-03T23:15:56.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:56.784+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:56.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:56.792+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:57.338 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:57.336+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:57.336907+0000) 2022-04-03T23:15:57.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:57.748+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:57.771 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:57.769+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:58.338 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:58.336+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:58.337149+0000) 2022-04-03T23:15:58.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:58.728+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:58.754 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:58.752+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:15:59.338 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:15:59.336+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:15:59.337389+0000) 2022-04-03T23:15:59.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:15:59.768+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:15:59.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:15:59.801+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:00.338 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:00.336+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:00.337622+0000) 2022-04-03T23:16:00.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:00.768+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:00.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:00.806+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:01.339 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:01.337+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:01.337879+0000) 2022-04-03T23:16:01.753 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:01.751+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:01.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:01.780+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:02.339 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:02.337+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:02.338101+0000) 2022-04-03T23:16:02.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:02.753+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:02.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:02.825+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:03.339 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:03.337+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:03.338345+0000) 2022-04-03T23:16:03.739 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:03.736+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:03.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:03.804+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:04.339 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:04.338+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:04.338518+0000) 2022-04-03T23:16:04.774 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:04.771+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:04.841 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:04.839+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:05.339 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:05.338+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:05.338753+0000) 2022-04-03T23:16:05.746 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:05.744+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:05.880 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:05.878+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:06.339 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:06.338+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:06.338993+0000) 2022-04-03T23:16:06.418 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:16:06.416+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:16:06.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:06.739+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:06.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:06.901+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:07.340 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:07.338+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:07.339208+0000) 2022-04-03T23:16:07.739 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:07.737+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:07.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:07.859+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:08.340 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:08.338+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:08.339456+0000) 2022-04-03T23:16:08.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:08.699+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:08.898 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:08.896+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:09.340 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:09.338+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:09.339717+0000) 2022-04-03T23:16:09.673 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:09.671+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:09.945 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:09.942+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:10.341 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:10.339+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:10.339966+0000) 2022-04-03T23:16:10.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:10.628+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:10.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:10.919+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:11.341 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:11.339+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:11.340171+0000) 2022-04-03T23:16:11.623 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:11.621+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:11.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:11.910+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:12.341 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:12.340+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:12.340395+0000) 2022-04-03T23:16:12.610 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:12.609+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:12.891 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:12.889+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:13.341 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:13.340+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:13.340640+0000) 2022-04-03T23:16:13.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:13.585+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:13.886 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:16:13.890 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:16:13.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:13.894+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:14.342 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:14.340+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:14.340866+0000) 2022-04-03T23:16:14.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:14.613+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:14.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:14.846+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:15.342 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:15.340+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:15.341089+0000) 2022-04-03T23:16:15.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:15.614+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:15.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:15.820+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:16.342 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:16.341+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:16.341313+0000) 2022-04-03T23:16:16.600 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:16.598+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:16.797 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:16.795+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:17.342 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:17.341+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:17.341532+0000) 2022-04-03T23:16:17.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:17.633+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:17.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:17.820+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:18.342 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:18.341+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:18.341741+0000) 2022-04-03T23:16:18.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:18.644+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:18.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:18.846+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:19.343 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:19.341+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:19.341959+0000) 2022-04-03T23:16:19.670 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:19.668+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:19.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:19.876+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:20.343 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:20.342+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:20.342140+0000) 2022-04-03T23:16:20.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:20.714+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:20.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:20.898+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:21.343 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:21.342+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:21.342364+0000) 2022-04-03T23:16:21.433 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:16:21.431+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:16:21.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:21.699+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:21.908 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:21.906+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:22.343 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:22.342+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:22.342629+0000) 2022-04-03T23:16:22.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:22.690+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:22.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:22.936+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:23.343 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:23.342+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:23.342831+0000) 2022-04-03T23:16:23.724 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:23.722+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:23.929 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:23.927+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:24.344 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:24.341+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:24.343002+0000) 2022-04-03T23:16:24.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:24.699+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:24.908 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:24.905+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:25.344 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:25.341+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:25.343180+0000) 2022-04-03T23:16:25.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:25.702+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:25.867 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:25.865+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:26.344 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:26.342+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:26.343366+0000) 2022-04-03T23:16:26.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:26.696+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:26.898 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:26.896+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:27.344 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:27.341+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:27.343630+0000) 2022-04-03T23:16:27.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:27.684+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:27.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:27.895+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:28.344 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:28.342+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:28.343889+0000) 2022-04-03T23:16:28.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:28.670+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:28.905 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:28.903+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:29.345 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:29.343+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:29.344061+0000) 2022-04-03T23:16:29.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:29.716+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:29.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:29.895+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:30.345 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:30.343+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:30.344244+0000) 2022-04-03T23:16:30.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:30.705+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:30.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:30.863+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:31.345 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:31.343+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:31.344496+0000) 2022-04-03T23:16:31.720 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:31.718+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:31.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:31.849+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:32.345 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:32.343+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:32.344645+0000) 2022-04-03T23:16:32.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:32.679+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:32.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:32.896+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:33.345 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:33.343+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:33.344814+0000) 2022-04-03T23:16:33.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:33.679+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:33.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:33.857+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:34.346 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:34.344+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:34.345038+0000) 2022-04-03T23:16:34.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:34.700+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:34.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:34.824+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:35.346 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:35.344+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:35.345301+0000) 2022-04-03T23:16:35.727 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:35.725+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:35.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:35.807+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:36.346 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:36.344+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:36.345472+0000) 2022-04-03T23:16:36.449 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:16:36.447+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:16:36.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:36.722+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:36.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:36.829+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:37.346 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:37.344+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:37.345622+0000) 2022-04-03T23:16:37.762 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:37.761+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:37.841 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:37.839+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:38.346 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:38.345+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:38.345739+0000) 2022-04-03T23:16:38.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:38.770+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:38.857 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:38.855+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:39.346 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:39.345+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:39.345880+0000) 2022-04-03T23:16:39.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:39.727+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:39.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:39.840+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:40.347 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:40.345+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:40.346002+0000) 2022-04-03T23:16:40.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:40.732+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:40.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:40.872+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:41.347 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:41.345+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:41.346142+0000) 2022-04-03T23:16:41.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:41.766+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:41.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:41.841+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:42.347 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:42.345+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:42.346279+0000) 2022-04-03T23:16:42.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:42.799+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:42.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:42.829+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:43.347 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:43.345+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:43.346490+0000) 2022-04-03T23:16:43.753 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:43.752+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:43.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:43.795+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:44.022 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:16:44.026 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:16:44.347 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:44.346+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:44.346669+0000) 2022-04-03T23:16:44.705 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:44.704+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:44.747 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:44.746+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:45.347 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:45.346+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:45.346884+0000) 2022-04-03T23:16:45.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:45.746+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:45.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:45.795+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:46.348 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:46.346+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:46.347074+0000) 2022-04-03T23:16:46.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:46.755+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:46.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:46.823+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:47.348 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:47.346+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:47.347331+0000) 2022-04-03T23:16:47.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:47.706+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:47.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:47.853+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:48.348 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:48.347+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:48.347571+0000) 2022-04-03T23:16:48.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:48.733+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:48.857 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:48.855+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:49.348 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:49.347+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:49.347739+0000) 2022-04-03T23:16:49.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:49.744+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:49.871 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:49.868+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:50.349 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:50.347+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:50.347935+0000) 2022-04-03T23:16:50.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:50.731+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:50.852 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:50.850+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:51.349 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:51.347+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:51.348108+0000) 2022-04-03T23:16:51.465 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:16:51.463+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:16:51.766 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:51.764+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:51.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:51.849+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:52.349 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:52.347+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:52.348305+0000) 2022-04-03T23:16:52.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:52.719+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:52.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:52.844+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:53.349 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:53.348+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:53.348472+0000) 2022-04-03T23:16:53.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:53.734+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:53.833 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:53.831+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:54.349 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:54.348+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:54.348682+0000) 2022-04-03T23:16:54.759 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:54.757+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:54.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:54.845+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:55.350 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:55.348+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:55.348937+0000) 2022-04-03T23:16:55.809 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:55.807+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:55.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:55.823+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:56.350 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:56.348+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:56.349153+0000) 2022-04-03T23:16:56.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:56.821+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:56.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:56.830+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:57.350 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:57.349+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:57.349337+0000) 2022-04-03T23:16:57.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:57.820+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:57.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:57.869+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:58.350 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:58.349+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:58.349513+0000) 2022-04-03T23:16:58.826 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:58.824+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:58.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:58.874+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:16:59.351 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:16:59.349+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:16:59.349727+0000) 2022-04-03T23:16:59.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:16:59.861+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:16:59.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:16:59.916+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:00.351 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:00.349+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:00.349924+0000) 2022-04-03T23:17:00.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:00.853+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:00.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:00.949+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:01.351 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:01.349+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:01.350119+0000) 2022-04-03T23:17:01.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:01.830+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:01.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:01.950+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:02.351 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:02.350+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:02.350337+0000) 2022-04-03T23:17:02.853 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:02.851+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:02.976 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:02.974+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:03.351 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:03.350+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:03.350587+0000) 2022-04-03T23:17:03.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:03.867+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:04.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:04.004+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:04.351 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:04.350+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:04.350737+0000) 2022-04-03T23:17:04.917 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:04.914+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:05.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:05.013+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:05.352 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:05.350+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:05.350931+0000) 2022-04-03T23:17:05.395 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T23:17:05.393+0000 7f2e44320700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T23:17:05.396 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T23:17:05.393+0000 7f2e4331e700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T23:17:05.912 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:05.910+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:06.065 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:06.062+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:06.352 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:06.350+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:06.351158+0000) 2022-04-03T23:17:06.481 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:17:06.479+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:17:06.911 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:06.909+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:07.086 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:07.084+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:07.352 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:07.351+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:07.351354+0000) 2022-04-03T23:17:07.919 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:07.917+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:08.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:08.120+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:08.352 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:08.351+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:08.351547+0000) 2022-04-03T23:17:08.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:08.951+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:09.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:09.110+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:09.353 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:09.350+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:09.351777+0000) 2022-04-03T23:17:09.929 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:09.927+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:10.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:10.081+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:10.353 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:10.350+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:10.352017+0000) 2022-04-03T23:17:10.946 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:10.944+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:11.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:11.046+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:11.353 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:11.351+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:11.352298+0000) 2022-04-03T23:17:11.982 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:11.980+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:12.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:12.022+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:12.353 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:12.351+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:12.352493+0000) 2022-04-03T23:17:12.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:12.941+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:12.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:12.977+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:13.353 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:13.351+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:13.352714+0000) 2022-04-03T23:17:13.906 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:13.904+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:13.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:13.945+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:14.165 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:17:14.169 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:17:14.354 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:14.351+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:14.352979+0000) 2022-04-03T23:17:14.926 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:14.924+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:14.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:14.949+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:15.354 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:15.352+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:15.353218+0000) 2022-04-03T23:17:15.890 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:15.888+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:15.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:15.987+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:16.354 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:16.352+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:16.353446+0000) 2022-04-03T23:17:16.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:16.842+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:16.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:16.954+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:17.354 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:17.352+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:17.353666+0000) 2022-04-03T23:17:17.845 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:17.843+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:17.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:17.964+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:18.355 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:18.352+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:18.353886+0000) 2022-04-03T23:17:18.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:18.809+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:18.934 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:18.931+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:19.355 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:19.353+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:19.354125+0000) 2022-04-03T23:17:19.850 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:19.847+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:19.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:19.926+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:20.355 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:20.353+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:20.354333+0000) 2022-04-03T23:17:20.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:20.875+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:20.929 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:20.927+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:21.355 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:21.353+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:21.354538+0000) 2022-04-03T23:17:21.496 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:17:21.495+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:17:21.904 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:21.902+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:21.967 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:21.965+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:22.355 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:22.353+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:22.354723+0000) 2022-04-03T23:17:22.952 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:22.950+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:22.996 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:22.993+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:23.356 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:23.354+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:23.354930+0000) 2022-04-03T23:17:23.912 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:23.910+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:23.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:23.984+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:24.356 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:24.354+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:24.355151+0000) 2022-04-03T23:17:24.905 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:24.903+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:24.968 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:24.967+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:17:25.356 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:25.354+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:25.355329+0000) 2022-04-03T23:17:25.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:25.869+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:26.000 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:25.998+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:26.356 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:26.354+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:26.355497+0000) 2022-04-03T23:17:26.885 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:26.883+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:26.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:26.975+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:27.356 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:27.355+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:27.355706+0000) 2022-04-03T23:17:27.935 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:27.934+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:27.967 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:27.966+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:28.357 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:28.355+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:28.355945+0000) 2022-04-03T23:17:28.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:28.908+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:29.005 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:29.003+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:29.357 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:29.355+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:29.356181+0000) 2022-04-03T23:17:29.928 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:29.926+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:29.969 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:29.968+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:30.357 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:30.355+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:30.356358+0000) 2022-04-03T23:17:30.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:30.922+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:30.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:30.952+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:31.357 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:31.355+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:31.356564+0000) 2022-04-03T23:17:31.905 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:31.904+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:31.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:31.956+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:32.357 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:32.356+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:32.356761+0000) 2022-04-03T23:17:32.934 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:32.933+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:32.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:32.950+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:33.358 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:33.356+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:33.357014+0000) 2022-04-03T23:17:33.906 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:33.904+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:33.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:33.987+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:34.358 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:34.356+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:34.357163+0000) 2022-04-03T23:17:34.927 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:34.925+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:34.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:34.987+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:35.358 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:35.356+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:35.357411+0000) 2022-04-03T23:17:35.922 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:35.921+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:35.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:35.946+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:36.358 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:36.357+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:36.357638+0000) 2022-04-03T23:17:36.512 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:17:36.511+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:17:36.899 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:36.897+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:36.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:36.972+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:37.359 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:37.357+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:37.357835+0000) 2022-04-03T23:17:37.873 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:37.872+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:38.018 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:38.017+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:38.359 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:38.357+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:38.358015+0000) 2022-04-03T23:17:38.857 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:38.856+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:39.032 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:39.030+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:39.359 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:39.357+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:39.358224+0000) 2022-04-03T23:17:39.857 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:39.855+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:40.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:40.010+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:40.359 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:40.358+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:40.358386+0000) 2022-04-03T23:17:40.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:40.814+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:40.969 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:40.968+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:41.359 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:41.358+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:41.358588+0000) 2022-04-03T23:17:41.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:41.827+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:42.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:42.013+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:42.360 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:42.358+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:42.358846+0000) 2022-04-03T23:17:42.789 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:42.788+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:43.018 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:43.017+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:43.360 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:43.358+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:43.359039+0000) 2022-04-03T23:17:43.786 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:43.783+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:44.023 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:44.021+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:44.305 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:17:44.309 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:17:44.360 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:44.358+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:44.359220+0000) 2022-04-03T23:17:44.814 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:44.812+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:45.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:45.060+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:45.360 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:45.359+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:45.359449+0000) 2022-04-03T23:17:45.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:45.809+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:46.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:46.100+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:46.360 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:46.359+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:46.359645+0000) 2022-04-03T23:17:46.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:46.792+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:47.149 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:47.147+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:47.360 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:47.359+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:47.359830+0000) 2022-04-03T23:17:47.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:47.782+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:48.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:48.134+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:48.361 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:48.359+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:48.360029+0000) 2022-04-03T23:17:48.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:48.740+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:49.138 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:49.135+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:49.361 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:49.360+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:49.360194+0000) 2022-04-03T23:17:49.784 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:49.782+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:50.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:50.123+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:50.361 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:50.360+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:50.360381+0000) 2022-04-03T23:17:50.819 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:50.816+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:51.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:51.161+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:51.361 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:51.360+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:51.360559+0000) 2022-04-03T23:17:51.528 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:17:51.526+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:17:51.856 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:51.854+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:52.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:52.121+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:52.361 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:52.359+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:52.360731+0000) 2022-04-03T23:17:52.891 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:52.888+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:53.077 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:53.075+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:53.361 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:53.360+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:53.360951+0000) 2022-04-03T23:17:53.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:53.866+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:54.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:54.052+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:54.362 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:54.360+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:54.361204+0000) 2022-04-03T23:17:54.837 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:54.834+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:55.022 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:55.020+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:55.362 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:55.360+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:55.361371+0000) 2022-04-03T23:17:55.818 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:55.816+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:56.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:56.006+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:56.362 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:56.359+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:56.361548+0000) 2022-04-03T23:17:56.812 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:56.809+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:57.040 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:57.038+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:57.362 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:57.360+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:57.361741+0000) 2022-04-03T23:17:57.765 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:57.763+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:58.042 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:58.040+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:58.363 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:58.360+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:58.361873+0000) 2022-04-03T23:17:58.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:58.767+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:59.031 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:59.028+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:17:59.363 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:17:59.361+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:17:59.362087+0000) 2022-04-03T23:17:59.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:17:59.724+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:17:59.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:17:59.997+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:00.363 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:00.361+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:00.362322+0000) 2022-04-03T23:18:00.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:00.711+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:00.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:00.950+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:01.363 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:01.361+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:01.362568+0000) 2022-04-03T23:18:01.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:01.695+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:01.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:01.945+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:02.364 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:02.361+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:02.362829+0000) 2022-04-03T23:18:02.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:02.656+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:02.960 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:02.958+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:03.364 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:03.362+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:03.363038+0000) 2022-04-03T23:18:03.694 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:03.692+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:03.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:03.918+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:04.364 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:04.362+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:04.363198+0000) 2022-04-03T23:18:04.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:04.666+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:04.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:04.885+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:05.364 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:05.362+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:05.363402+0000) 2022-04-03T23:18:05.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:05.681+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:05.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:05.846+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:06.364 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:06.362+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:06.363571+0000) 2022-04-03T23:18:06.544 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:18:06.542+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:18:06.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:06.661+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:06.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:06.821+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:07.364 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:07.363+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:07.363814+0000) 2022-04-03T23:18:07.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:07.641+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:07.774 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:07.772+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:08.365 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:08.363+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:08.364043+0000) 2022-04-03T23:18:08.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:08.670+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:08.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:08.723+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:09.365 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:09.363+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:09.364276+0000) 2022-04-03T23:18:09.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:09.675+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:09.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:09.757+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:10.365 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:10.363+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:10.364442+0000) 2022-04-03T23:18:10.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:10.649+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:10.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:10.789+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:11.365 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:11.363+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:11.364624+0000) 2022-04-03T23:18:11.617 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:11.615+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:11.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:11.790+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:12.366 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:12.364+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:12.364826+0000) 2022-04-03T23:18:12.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:12.568+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:12.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:12.777+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:13.366 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:13.364+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:13.365047+0000) 2022-04-03T23:18:13.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:13.588+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:13.819 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:13.816+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:14.366 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:14.364+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:14.365242+0000) 2022-04-03T23:18:14.445 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:18:14.448 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:18:14.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:14.585+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:14.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:14.794+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:15.366 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:15.364+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:15.365432+0000) 2022-04-03T23:18:15.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:15.558+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:15.798 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:15.796+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:16.366 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:16.365+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:16.365642+0000) 2022-04-03T23:18:16.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:16.596+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:16.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:16.802+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:17.367 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:17.365+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:17.365879+0000) 2022-04-03T23:18:17.642 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:17.640+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:17.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:17.847+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:18.367 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:18.365+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:18.366118+0000) 2022-04-03T23:18:18.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:18.617+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:18.859 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:18.857+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:19.367 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:19.365+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:19.366371+0000) 2022-04-03T23:18:19.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:19.664+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:19.837 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:19.835+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:20.367 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:20.366+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:20.366623+0000) 2022-04-03T23:18:20.687 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:20.685+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:20.837 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:20.836+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:21.368 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:21.366+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:21.366889+0000) 2022-04-03T23:18:21.560 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:18:21.558+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:18:21.724 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:21.723+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:21.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:21.844+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:22.368 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:22.366+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:22.367121+0000) 2022-04-03T23:18:22.694 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:22.692+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:22.883 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:22.881+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:23.368 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:23.366+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:23.367308+0000) 2022-04-03T23:18:23.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:23.648+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:23.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:23.842+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:24.368 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:24.367+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:24.367513+0000) 2022-04-03T23:18:24.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:24.609+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:24.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:24.802+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:25.368 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:25.367+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:25.367701+0000) 2022-04-03T23:18:25.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:25.589+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:25.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:25.822+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:26.369 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:26.367+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:26.367928+0000) 2022-04-03T23:18:26.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:26.638+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:26.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:26.799+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:27.369 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:27.367+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:27.368177+0000) 2022-04-03T23:18:27.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:27.639+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:27.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:27.820+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:28.369 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:28.368+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:28.368419+0000) 2022-04-03T23:18:28.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:28.630+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:28.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:28.788+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:29.369 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:29.368+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:29.368649+0000) 2022-04-03T23:18:29.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:29.678+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:29.797 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:29.796+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:30.370 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:30.368+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:30.368885+0000) 2022-04-03T23:18:30.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:30.696+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:30.814 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:30.812+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:31.370 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:31.368+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:31.369083+0000) 2022-04-03T23:18:31.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:31.661+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:31.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:31.848+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:32.370 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:32.369+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:32.369291+0000) 2022-04-03T23:18:32.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:32.625+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:32.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:32.810+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:33.370 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:33.369+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:33.369529+0000) 2022-04-03T23:18:33.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:33.675+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:33.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:33.856+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:34.371 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:34.369+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:34.369798+0000) 2022-04-03T23:18:34.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:34.644+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:34.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:34.821+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:35.371 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:35.369+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:35.370037+0000) 2022-04-03T23:18:35.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:35.694+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:35.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:35.869+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:36.371 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:36.370+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:36.370281+0000) 2022-04-03T23:18:36.576 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:18:36.574+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:18:36.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:36.717+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:36.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:36.833+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:37.371 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:37.370+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:37.370471+0000) 2022-04-03T23:18:37.708 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:37.706+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:37.792 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:37.791+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:38.371 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:38.369+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:38.370639+0000) 2022-04-03T23:18:38.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:38.752+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:38.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:38.780+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:39.371 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:39.369+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:39.370817+0000) 2022-04-03T23:18:39.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:39.719+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:39.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:39.781+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:40.372 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:40.369+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:40.371061+0000) 2022-04-03T23:18:40.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:40.764+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:40.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:40.822+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:41.372 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:41.369+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:41.371336+0000) 2022-04-03T23:18:41.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:41.752+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:41.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:41.786+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:42.372 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:42.369+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:42.371588+0000) 2022-04-03T23:18:42.708 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:42.706+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:42.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:42.797+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:43.373 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:43.370+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:43.371853+0000) 2022-04-03T23:18:43.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:43.725+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:43.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:43.798+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:44.373 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:44.371+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:44.372108+0000) 2022-04-03T23:18:44.510 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:18:44.514 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:18:44.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:44.726+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:44.752 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:44.750+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:45.373 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:45.371+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:45.372323+0000) 2022-04-03T23:18:45.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:45.731+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:45.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:45.791+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:46.373 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:46.371+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:46.372531+0000) 2022-04-03T23:18:46.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:46.753+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:46.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:46.777+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:47.374 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:47.371+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:47.372744+0000) 2022-04-03T23:18:47.765 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:47.762+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:47.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:47.776+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:48.374 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:48.372+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:48.373017+0000) 2022-04-03T23:18:48.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:48.809+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:48.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:48.812+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:49.374 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:49.372+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:49.373194+0000) 2022-04-03T23:18:49.766 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:49.764+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:49.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:49.798+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:50.374 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:50.372+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:50.373412+0000) 2022-04-03T23:18:50.807 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:50.804+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:50.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:50.832+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:51.374 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:51.372+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:51.373591+0000) 2022-04-03T23:18:51.592 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:18:51.590+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:18:51.790 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:51.787+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:51.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:51.791+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:52.375 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:52.373+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:52.373852+0000) 2022-04-03T23:18:52.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:52.811+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:52.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:52.825+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:53.375 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:53.373+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:53.374067+0000) 2022-04-03T23:18:53.786 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:53.783+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:53.797 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:53.795+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:54.375 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:54.373+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:54.374279+0000) 2022-04-03T23:18:54.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:54.749+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:54.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:54.798+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:55.375 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:55.373+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:55.374520+0000) 2022-04-03T23:18:55.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:55.732+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:55.768 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:55.766+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:56.376 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:56.374+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:56.374762+0000) 2022-04-03T23:18:56.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:56.765+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:56.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:56.809+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:57.376 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:57.374+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:57.375029+0000) 2022-04-03T23:18:57.766 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:57.764+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:57.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:57.811+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:58.376 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:58.374+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:58.375273+0000) 2022-04-03T23:18:58.774 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:58.772+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:58.804 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:58.801+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:18:59.376 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:18:59.374+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:18:59.375535+0000) 2022-04-03T23:18:59.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:18:59.751+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:18:59.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:18:59.754+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:00.377 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:00.375+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:00.375810+0000) 2022-04-03T23:19:00.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:00.741+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:00.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:00.756+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:01.377 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:01.375+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:01.376063+0000) 2022-04-03T23:19:01.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:01.708+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:01.775 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:01.773+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:02.377 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:02.375+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:02.376256+0000) 2022-04-03T23:19:02.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:02.665+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:02.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:02.727+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:03.377 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:03.375+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:03.376480+0000) 2022-04-03T23:19:03.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:03.711+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:03.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:03.765+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:04.377 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:04.376+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:04.376740+0000) 2022-04-03T23:19:04.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:04.692+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:04.765 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:04.763+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:05.378 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:05.376+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:05.376893+0000) 2022-04-03T23:19:05.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:05.668+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:05.735 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:05.733+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:06.378 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:06.376+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:06.377111+0000) 2022-04-03T23:19:06.597 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:19:06.595+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:19:06.643 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:06.640+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:06.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:06.693+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:07.378 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:07.376+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:07.377352+0000) 2022-04-03T23:19:07.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:07.665+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:07.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:07.686+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:08.378 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:08.377+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:08.377600+0000) 2022-04-03T23:19:08.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:08.648+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:08.678 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:08.676+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:09.379 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:09.377+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:09.377876+0000) 2022-04-03T23:19:09.603 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:09.601+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:09.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:09.680+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:10.379 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:10.377+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:10.378085+0000) 2022-04-03T23:19:10.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:10.575+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:10.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:10.638+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:11.379 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:11.377+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:11.378331+0000) 2022-04-03T23:19:11.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:11.569+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:11.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:11.654+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:12.379 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:12.378+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:12.378521+0000) 2022-04-03T23:19:12.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:12.532+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:12.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:12.668+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:13.379 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:13.378+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:13.378695+0000) 2022-04-03T23:19:13.544 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:13.542+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:13.718 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:13.716+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:14.380 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:14.378+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:14.378920+0000) 2022-04-03T23:19:14.500 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:14.498+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:14.608 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:19:14.612 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:19:14.751 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:14.749+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:15.380 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:15.378+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:15.379180+0000) 2022-04-03T23:19:15.484 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:15.482+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:15.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:15.731+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:16.380 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:16.379+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:16.379354+0000) 2022-04-03T23:19:16.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:16.490+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:16.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:16.690+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:17.381 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:17.379+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:17.379575+0000) 2022-04-03T23:19:17.503 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:17.502+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:17.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:17.685+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:18.381 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:18.379+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:18.379836+0000) 2022-04-03T23:19:18.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:18.474+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:18.703 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:18.701+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:19.381 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:19.379+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:19.380072+0000) 2022-04-03T23:19:19.468 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:19.466+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:19.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:19.677+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:20.381 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:20.380+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:20.380317+0000) 2022-04-03T23:19:20.463 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:20.461+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:20.686 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:20.685+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:21.381 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:21.379+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:21.380571+0000) 2022-04-03T23:19:21.431 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:21.430+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:21.613 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:19:21.611+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:19:21.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:21.701+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:22.382 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:22.380+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:22.380804+0000) 2022-04-03T23:19:22.470 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:22.469+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:22.736 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:22.734+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:23.382 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:23.379+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:23.380996+0000) 2022-04-03T23:19:23.451 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:23.450+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:23.771 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:23.770+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:24.382 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:24.380+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:24.381207+0000) 2022-04-03T23:19:24.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:24.416+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:24.816 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:24.814+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:25.382 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:25.379+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:25.381343+0000) 2022-04-03T23:19:25.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:25.454+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:25.767 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:25.766+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:26.382 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:26.380+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:26.381542+0000) 2022-04-03T23:19:26.478 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:26.477+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:26.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:26.779+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:27.382 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:27.380+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:27.381694+0000) 2022-04-03T23:19:27.429 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:27.427+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:27.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:27.741+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:28.383 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:28.380+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:28.381858+0000) 2022-04-03T23:19:28.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:28.453+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:28.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:28.698+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:29.383 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:29.381+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:29.382054+0000) 2022-04-03T23:19:29.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:29.412+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:29.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:29.741+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:30.383 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:30.381+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:30.382211+0000) 2022-04-03T23:19:30.422 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:30.421+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:30.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:30.734+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:31.383 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:31.381+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:31.382421+0000) 2022-04-03T23:19:31.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:31.458+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:31.696 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:31.695+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:32.383 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:32.381+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:32.382606+0000) 2022-04-03T23:19:32.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:32.490+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:32.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:32.667+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:33.384 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:33.381+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:33.382829+0000) 2022-04-03T23:19:33.457 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:33.455+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:33.674 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:33.672+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:34.384 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:34.382+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:34.383039+0000) 2022-04-03T23:19:34.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:34.464+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:34.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:34.656+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:35.384 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:35.382+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:35.383242+0000) 2022-04-03T23:19:35.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:35.474+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:35.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:35.619+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:36.384 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:36.382+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:36.383394+0000) 2022-04-03T23:19:36.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:36.489+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:36.578 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:36.575+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:36.629 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:19:36.627+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:19:37.385 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:37.382+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:37.383586+0000) 2022-04-03T23:19:37.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:37.527+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:37.618 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:37.616+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:38.385 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:38.383+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:38.383800+0000) 2022-04-03T23:19:38.522 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:38.520+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:38.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:38.574+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:39.385 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:39.383+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:39.383987+0000) 2022-04-03T23:19:39.502 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:39.499+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:39.540 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:39.538+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:40.385 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:40.383+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:40.384185+0000) 2022-04-03T23:19:40.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:40.535+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:40.562 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:40.560+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:41.385 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:41.383+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:41.384402+0000) 2022-04-03T23:19:41.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:41.556+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:41.596 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:41.594+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:42.385 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:42.383+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:42.384576+0000) 2022-04-03T23:19:42.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:42.527+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:42.598 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:42.596+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:43.386 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:43.384+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:43.384754+0000) 2022-04-03T23:19:43.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:43.510+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:43.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:43.605+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:44.386 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:44.384+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:44.384975+0000) 2022-04-03T23:19:44.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:44.523+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:44.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:44.582+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:44.753 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:19:44.757 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:19:45.386 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:45.384+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:45.385200+0000) 2022-04-03T23:19:45.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:45.531+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:45.590 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:45.588+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:46.386 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:46.384+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:46.385404+0000) 2022-04-03T23:19:46.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:46.516+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:46.610 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:46.608+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:47.386 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:47.385+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:47.385607+0000) 2022-04-03T23:19:47.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:47.565+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:47.649 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:47.647+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:48.387 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:48.385+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:48.385824+0000) 2022-04-03T23:19:48.609 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:48.606+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:48.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:48.626+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:49.387 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:49.385+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:49.386011+0000) 2022-04-03T23:19:49.599 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:49.597+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:49.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:49.628+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:50.387 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:50.385+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:50.386218+0000) 2022-04-03T23:19:50.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:50.574+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:50.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:50.630+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:51.387 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:51.385+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:51.386400+0000) 2022-04-03T23:19:51.589 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:51.587+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:51.645 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:19:51.643+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:19:51.661 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:51.659+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:52.387 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:52.386+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:52.386552+0000) 2022-04-03T23:19:52.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:52.584+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:52.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:52.635+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:53.388 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:53.385+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:53.386728+0000) 2022-04-03T23:19:53.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:53.600+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:53.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:53.611+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:54.388 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:54.386+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:54.386884+0000) 2022-04-03T23:19:54.636 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:54.634+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:54.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:54.636+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:55.388 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:55.386+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:55.387091+0000) 2022-04-03T23:19:55.653 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:55.651+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:55.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:55.661+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:56.388 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:56.386+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:56.387234+0000) 2022-04-03T23:19:56.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:56.612+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:56.703 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:56.701+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:57.390 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:57.387+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:57.387447+0000) 2022-04-03T23:19:57.603 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:57.601+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:57.672 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:57.670+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:58.388 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:58.387+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:58.387628+0000) 2022-04-03T23:19:58.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:58.586+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:58.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:58.674+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:19:59.389 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:19:59.387+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:19:59.387857+0000) 2022-04-03T23:19:59.622 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:19:59.620+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:19:59.652 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:19:59.650+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:00.389 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:00.387+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:00.388066+0000) 2022-04-03T23:20:00.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:00.612+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:00.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:00.651+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:01.389 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:01.388+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:01.388296+0000) 2022-04-03T23:20:01.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:01.637+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:01.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:01.637+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:02.389 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:02.388+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:02.388535+0000) 2022-04-03T23:20:02.607 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:02.605+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:02.610 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:02.609+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:03.390 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:03.387+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:03.388729+0000) 2022-04-03T23:20:03.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:03.625+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:03.651 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:03.649+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:04.390 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:04.388+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:04.388944+0000) 2022-04-03T23:20:04.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:04.576+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:04.686 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:04.683+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:05.390 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:05.389+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:05.389123+0000) 2022-04-03T23:20:05.550 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:05.548+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:05.697 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:05.695+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:06.390 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:06.388+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:06.389299+0000) 2022-04-03T23:20:06.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:06.588+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:06.655 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:20:06.653+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:20:06.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:06.709+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:07.390 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:07.389+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:07.389480+0000) 2022-04-03T23:20:07.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:07.625+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:07.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:07.675+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:08.391 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:08.388+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:08.389697+0000) 2022-04-03T23:20:08.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:08.673+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:08.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:08.721+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:09.391 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:09.388+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:09.389904+0000) 2022-04-03T23:20:09.666 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:09.664+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:09.738 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:09.737+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:10.391 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:10.389+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:10.390141+0000) 2022-04-03T23:20:10.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:10.678+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:10.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:10.689+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:11.391 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:11.389+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:11.390337+0000) 2022-04-03T23:20:11.670 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:11.668+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:11.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:11.712+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:12.391 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:12.389+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:12.390556+0000) 2022-04-03T23:20:12.685 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:12.683+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:12.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:12.699+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:13.392 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:13.389+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:13.390765+0000) 2022-04-03T23:20:13.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:13.664+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:13.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:13.707+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:14.392 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:14.390+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:14.391541+0000) 2022-04-03T23:20:14.630 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:14.629+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:14.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:14.678+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:14.884 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:20:14.888 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:20:15.392 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:15.390+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:15.391785+0000) 2022-04-03T23:20:15.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:15.656+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:15.691 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:15.689+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:16.393 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:16.391+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:16.392010+0000) 2022-04-03T23:20:16.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:16.624+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:16.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:16.656+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:17.393 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:17.391+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:17.392208+0000) 2022-04-03T23:20:17.617 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:17.616+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:17.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:17.664+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:18.393 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:18.391+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:18.392426+0000) 2022-04-03T23:20:18.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:18.637+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:18.666 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:18.665+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:19.393 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:19.391+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:19.392572+0000) 2022-04-03T23:20:19.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:19.668+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:19.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:19.711+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:20.394 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:20.392+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:20.392829+0000) 2022-04-03T23:20:20.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:20.697+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:20.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:20.715+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:21.394 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:21.392+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:21.393044+0000) 2022-04-03T23:20:21.659 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:21.657+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:21.671 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:20:21.669+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:20:21.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:21.727+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:22.394 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:22.392+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:22.393242+0000) 2022-04-03T23:20:22.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:22.675+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:22.752 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:22.751+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:23.394 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:23.392+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:23.393409+0000) 2022-04-03T23:20:23.704 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:23.702+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:23.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:23.780+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:24.395 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:24.392+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:24.393624+0000) 2022-04-03T23:20:24.738 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:24.736+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:24.745 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:24.744+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:25.395 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:25.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:25.393854+0000) 2022-04-03T23:20:25.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:25.698+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:20:25.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:25.740+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:26.395 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:26.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:26.394074+0000) 2022-04-03T23:20:26.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:26.730+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:26.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:26.778+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:27.395 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:27.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:27.394245+0000) 2022-04-03T23:20:27.686 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:27.684+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:27.826 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:27.825+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:28.395 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:28.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:28.394458+0000) 2022-04-03T23:20:28.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:28.655+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:28.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:28.790+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:29.395 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:29.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:29.394676+0000) 2022-04-03T23:20:29.650 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:29.647+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:29.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:29.829+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:30.396 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:30.394+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:30.394875+0000) 2022-04-03T23:20:30.627 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:30.625+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:30.873 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:30.871+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:31.396 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:31.394+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:31.395040+0000) 2022-04-03T23:20:31.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:31.656+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:31.911 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:31.908+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:32.396 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:32.394+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:32.395247+0000) 2022-04-03T23:20:32.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:32.688+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:32.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:32.941+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:33.396 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:33.394+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:33.395450+0000) 2022-04-03T23:20:33.686 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:33.684+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:33.968 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:33.965+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:34.397 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:34.395+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:34.395692+0000) 2022-04-03T23:20:34.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:34.697+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:34.932 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:34.929+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:35.397 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:35.395+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:35.395860+0000) 2022-04-03T23:20:35.685 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:35.683+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:35.977 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:35.975+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:36.397 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:36.395+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:36.395996+0000) 2022-04-03T23:20:36.687 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:20:36.685+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:20:36.696 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:36.694+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:37.000 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:36.997+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:37.397 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:37.395+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:37.396198+0000) 2022-04-03T23:20:37.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:37.646+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:37.980 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:37.978+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:38.397 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:38.395+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:38.396373+0000) 2022-04-03T23:20:38.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:38.610+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:38.947 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:38.944+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:39.397 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:39.396+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:39.396530+0000) 2022-04-03T23:20:39.637 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:39.635+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:39.982 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:39.979+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:40.398 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:40.396+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:40.396662+0000) 2022-04-03T23:20:40.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:40.614+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:40.997 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:40.994+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:41.398 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:41.396+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:41.396828+0000) 2022-04-03T23:20:41.645 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:41.643+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:41.969 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:41.967+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:42.398 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:42.396+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:42.396990+0000) 2022-04-03T23:20:42.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:42.633+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:42.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:42.930+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:43.398 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:43.396+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:43.397191+0000) 2022-04-03T23:20:43.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:43.591+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:43.955 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:43.952+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:44.398 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:44.397+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:44.397389+0000) 2022-04-03T23:20:44.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:44.612+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:44.990 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:44.987+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:45.026 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:20:45.030 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:20:45.398 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:45.396+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:45.397637+0000) 2022-04-03T23:20:45.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:45.611+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:45.949 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:45.947+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:46.399 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:46.397+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:46.397848+0000) 2022-04-03T23:20:46.660 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:46.658+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:46.935 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:46.933+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:47.399 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:47.397+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:47.398056+0000) 2022-04-03T23:20:47.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:47.633+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:47.969 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:47.966+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:48.399 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:48.398+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:48.398262+0000) 2022-04-03T23:20:48.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:48.593+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:48.971 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:48.969+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:49.399 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:49.398+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:49.398457+0000) 2022-04-03T23:20:49.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:49.573+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:49.969 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:49.967+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:50.399 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:50.397+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:50.398697+0000) 2022-04-03T23:20:50.604 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:50.602+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:50.939 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:50.936+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:51.400 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:51.398+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:51.398912+0000) 2022-04-03T23:20:51.653 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:51.650+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:51.692 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:20:51.690+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:20:51.925 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:51.922+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:52.400 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:52.398+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:52.399048+0000) 2022-04-03T23:20:52.684 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:52.681+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:52.898 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:52.896+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:53.400 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:53.398+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:53.399217+0000) 2022-04-03T23:20:53.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:53.638+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:53.924 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:53.922+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:54.400 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:54.398+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:54.399363+0000) 2022-04-03T23:20:54.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:54.662+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:54.903 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:54.901+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:55.400 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:55.398+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:55.399567+0000) 2022-04-03T23:20:55.617 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:55.615+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:55.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:55.887+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:56.401 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:56.398+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:56.399801+0000) 2022-04-03T23:20:56.582 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:56.579+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:56.898 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:56.896+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:57.401 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:57.399+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:57.400009+0000) 2022-04-03T23:20:57.568 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:57.566+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:57.907 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:57.904+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:58.401 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:58.399+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:58.400176+0000) 2022-04-03T23:20:58.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:58.605+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:58.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:58.941+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:20:59.401 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:20:59.399+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:20:59.400387+0000) 2022-04-03T23:20:59.591 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:20:59.589+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:20:59.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:20:59.946+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:00.402 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:00.398+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:00.400609+0000) 2022-04-03T23:21:00.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:00.592+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:00.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:00.994+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:01.402 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:01.399+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:01.400839+0000) 2022-04-03T23:21:01.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:01.590+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:02.000 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:01.997+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:02.402 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:02.400+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:02.401057+0000) 2022-04-03T23:21:02.569 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:02.566+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:02.975 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:02.973+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:03.402 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:03.400+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:03.401262+0000) 2022-04-03T23:21:03.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:03.574+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:03.938 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:03.936+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:04.402 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:04.400+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:04.401464+0000) 2022-04-03T23:21:04.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:04.611+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:04.969 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:04.967+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:05.402 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:05.400+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:05.401699+0000) 2022-04-03T23:21:05.585 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:05.584+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:06.015 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:06.013+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:06.403 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:06.401+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:06.401899+0000) 2022-04-03T23:21:06.585 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:06.584+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:06.708 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:21:06.706+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:21:07.009 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:07.008+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:07.403 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:07.401+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:07.402076+0000) 2022-04-03T23:21:07.556 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:07.554+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:08.030 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:08.028+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:08.403 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:08.401+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:08.402277+0000) 2022-04-03T23:21:08.593 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:08.592+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:09.028 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:09.026+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:09.403 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:09.401+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:09.402465+0000) 2022-04-03T23:21:09.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:09.552+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:10.036 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:10.035+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:10.404 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:10.401+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:10.402675+0000) 2022-04-03T23:21:10.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:10.527+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:11.079 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:11.078+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:11.404 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:11.402+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:11.402896+0000) 2022-04-03T23:21:11.568 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:11.566+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:12.031 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:12.030+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:12.404 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:12.402+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:12.403134+0000) 2022-04-03T23:21:12.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:12.544+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:13.044 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:13.042+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:13.404 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:13.402+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:13.403334+0000) 2022-04-03T23:21:13.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:13.508+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:14.093 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:14.092+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:14.404 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:14.402+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:14.403519+0000) 2022-04-03T23:21:14.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:14.494+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:15.075 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:15.073+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:15.108 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:21:15.111 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:21:15.404 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:15.403+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:15.403696+0000) 2022-04-03T23:21:15.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:15.466+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:16.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:16.079+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:16.405 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:16.403+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:16.403912+0000) 2022-04-03T23:21:16.473 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:16.472+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:17.046 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:17.045+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:17.405 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:17.403+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:17.404117+0000) 2022-04-03T23:21:17.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:17.473+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:18.035 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:18.034+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:18.405 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:18.403+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:18.404330+0000) 2022-04-03T23:21:18.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:18.457+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:19.041 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:19.039+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:19.405 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:19.404+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:19.404531+0000) 2022-04-03T23:21:19.466 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:19.465+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:20.078 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:20.076+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:20.405 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:20.404+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:20.404691+0000) 2022-04-03T23:21:20.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:20.501+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:21.063 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:21.062+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:21.406 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:21.404+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:21.404921+0000) 2022-04-03T23:21:21.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:21.514+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:21.713 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:21:21.711+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:21:22.017 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:22.016+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:22.406 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:22.404+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:22.405116+0000) 2022-04-03T23:21:22.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:22.487+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:22.976 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:22.975+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:23.406 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:23.404+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:23.405309+0000) 2022-04-03T23:21:23.472 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:23.470+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:23.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:23.943+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:24.406 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:24.405+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:24.405466+0000) 2022-04-03T23:21:24.462 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:24.460+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:24.955 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:24.953+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:25.406 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:25.405+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:25.405581+0000) 2022-04-03T23:21:25.469 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:25.467+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:25.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:25.913+0000 7f88d0ac1700 -1 osd.4 131 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:26.407 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:26.405+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:26.405803+0000) 2022-04-03T23:21:26.442 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:26.440+0000 7fd3b190b700 -1 osd.6 131 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:26.788 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-03T23:21:26.786+0000 7f4178a6f700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T23:21:26.803 INFO:tasks.ceph.mds.f.smithi186.stderr: -1> 2022-04-03T23:21:26.786+0000 7f4178a6f700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T23:21:26.825 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-03T23:21:26.949 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:26.946+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:27.407 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:27.405+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:27.406006+0000) 2022-04-03T23:21:27.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:27.429+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:27.925 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:27.922+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:28.407 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:28.405+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:28.406166+0000) 2022-04-03T23:21:28.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:28.435+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:28.972 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:28.970+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:29.407 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:29.406+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:29.406381+0000) 2022-04-03T23:21:29.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:29.428+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:29.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:29.931+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:30.407 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:30.406+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:30.406553+0000) 2022-04-03T23:21:30.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:30.414+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:30.893 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:30.890+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:31.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:31.380+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:31.407 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:31.406+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:31.406710+0000) 2022-04-03T23:21:31.867 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:31.864+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 40 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:32.408 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:32.406+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:32.406887+0000) 2022-04-03T23:21:32.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:32.413+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:32.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:32.821+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:33.408 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:33.406+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:33.407093+0000) 2022-04-03T23:21:33.448 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:33.446+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:33.853 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:33.850+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:34.408 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:34.407+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:34.407293+0000) 2022-04-03T23:21:34.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:34.443+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:34.839 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:34.837+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:35.408 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:35.407+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:35.407482+0000) 2022-04-03T23:21:35.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:35.417+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:35.854 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:35.852+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:36.409 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:36.407+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:36.407690+0000) 2022-04-03T23:21:36.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:36.435+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:36.724 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:21:36.722+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:21:36.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:36.804+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:37.409 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:37.407+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:37.407861+0000) 2022-04-03T23:21:37.410 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:37.408+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:37.825 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:37.823+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:38.409 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:38.406+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:38.408080+0000) 2022-04-03T23:21:38.420 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:38.418+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:38.821 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:38.819+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:39.409 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:39.407+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:39.408281+0000) 2022-04-03T23:21:39.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:39.453+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:39.865 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:39.863+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:40.409 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:40.407+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:40.408474+0000) 2022-04-03T23:21:40.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:40.428+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:40.854 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:40.852+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:41.409 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:41.406+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:41.408659+0000) 2022-04-03T23:21:41.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:41.427+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:41.835 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:41.833+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:42.410 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:42.407+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:42.408825+0000) 2022-04-03T23:21:42.478 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:42.476+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:42.828 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:42.826+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:43.410 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:43.408+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:43.409025+0000) 2022-04-03T23:21:43.506 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:43.503+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:43.865 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:43.863+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:44.410 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:44.408+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:44.409244+0000) 2022-04-03T23:21:44.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:44.551+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:44.821 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:44.819+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:45.256 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:21:45.260 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:21:45.410 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:45.408+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:45.409465+0000) 2022-04-03T23:21:45.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:45.536+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:45.851 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:45.849+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:46.411 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:46.408+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:46.409664+0000) 2022-04-03T23:21:46.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:46.526+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:46.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:46.887+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:47.411 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:47.409+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:47.409890+0000) 2022-04-03T23:21:47.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:47.528+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:47.901 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:47.898+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:48.411 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:48.409+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:48.410122+0000) 2022-04-03T23:21:48.519 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:48.517+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:48.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:48.876+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:49.411 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:49.409+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:49.410328+0000) 2022-04-03T23:21:49.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:49.532+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:49.853 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:49.851+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:50.411 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:50.409+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:50.410564+0000) 2022-04-03T23:21:50.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:50.499+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:50.845 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:50.843+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:51.412 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:51.410+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:51.410869+0000) 2022-04-03T23:21:51.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:51.539+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:51.740 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:21:51.738+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:21:51.817 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:51.815+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:52.412 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:52.410+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:52.411115+0000) 2022-04-03T23:21:52.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:52.586+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:52.778 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:52.775+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:53.412 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:53.410+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:53.411355+0000) 2022-04-03T23:21:53.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:53.636+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:53.769 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:53.767+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:54.412 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:54.410+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:54.411570+0000) 2022-04-03T23:21:54.637 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:54.635+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:54.792 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:54.790+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:55.413 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:55.411+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:55.411753+0000) 2022-04-03T23:21:55.659 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:55.657+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:55.839 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:55.837+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:56.413 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:56.411+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:56.411960+0000) 2022-04-03T23:21:56.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:56.698+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:56.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:56.862+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:57.413 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:57.411+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:57.412182+0000) 2022-04-03T23:21:57.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:57.737+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:57.911 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:57.909+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:58.413 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:58.411+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:58.412424+0000) 2022-04-03T23:21:58.765 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:58.763+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:58.957 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:58.955+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:21:59.414 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:21:59.412+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:21:59.412654+0000) 2022-04-03T23:21:59.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:21:59.726+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:21:59.965 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:21:59.962+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:00.414 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:00.412+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:00.412889+0000) 2022-04-03T23:22:00.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:00.692+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:00.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:00.950+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:01.414 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:01.412+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:01.413053+0000) 2022-04-03T23:22:01.720 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:01.718+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:01.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:01.977+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:02.414 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:02.412+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:02.413247+0000) 2022-04-03T23:22:02.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:02.674+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:02.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:02.946+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:03.414 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:03.412+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:03.413410+0000) 2022-04-03T23:22:03.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:03.628+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:03.932 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:03.930+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:04.414 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:04.412+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:04.413587+0000) 2022-04-03T23:22:04.636 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:04.634+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:04.944 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:04.942+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:05.415 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:05.413+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:05.413806+0000) 2022-04-03T23:22:05.612 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:05.610+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:05.991 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:05.990+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:06.416 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:06.413+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:06.414036+0000) 2022-04-03T23:22:06.589 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:06.587+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:06.756 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:22:06.754+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:22:06.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:06.977+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:07.415 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:07.413+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:07.414267+0000) 2022-04-03T23:22:07.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:07.577+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:07.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:07.978+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:08.415 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:08.414+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:08.414502+0000) 2022-04-03T23:22:08.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:08.551+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:09.017 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:09.016+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:09.416 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:09.414+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:09.414741+0000) 2022-04-03T23:22:09.520 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:09.518+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:10.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:10.002+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:10.416 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:10.414+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:10.414993+0000) 2022-04-03T23:22:10.566 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:10.565+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:10.991 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:10.990+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:11.416 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:11.414+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:11.415249+0000) 2022-04-03T23:22:11.582 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:11.580+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:12.031 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:12.030+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:12.416 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:12.415+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:12.415446+0000) 2022-04-03T23:22:12.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:12.574+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:13.008 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:13.006+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:13.417 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:13.415+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:13.415642+0000) 2022-04-03T23:22:13.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:13.529+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:13.960 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:13.958+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:14.417 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:14.415+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:14.415836+0000) 2022-04-03T23:22:14.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:14.490+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:15.010 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:15.008+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:15.337 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:22:15.341 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:22:15.417 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:15.415+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:15.416035+0000) 2022-04-03T23:22:15.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:15.454+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:15.974 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:15.973+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:16.417 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:16.416+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:16.416246+0000) 2022-04-03T23:22:16.500 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:16.499+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:17.024 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:17.023+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:17.417 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:17.415+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:17.416461+0000) 2022-04-03T23:22:17.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:17.478+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:18.015 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:18.013+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:18.418 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:18.416+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:18.416679+0000) 2022-04-03T23:22:18.499 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:18.496+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:19.039 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:19.037+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:19.418 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:19.416+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:19.416903+0000) 2022-04-03T23:22:19.519 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:19.516+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:20.024 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:20.022+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:20.418 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:20.416+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:20.417118+0000) 2022-04-03T23:22:20.533 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:20.531+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:21.019 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:21.017+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:21.418 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:21.417+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:21.417363+0000) 2022-04-03T23:22:21.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:21.576+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:21.772 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:22:21.770+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:22:22.013 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:22.010+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:22.419 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:22.416+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:22.417629+0000) 2022-04-03T23:22:22.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:22.613+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:22.994 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:22.992+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:23.419 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:23.416+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:23.417876+0000) 2022-04-03T23:22:23.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:23.574+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:23.950 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:23.947+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:24.419 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:24.417+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:24.418022+0000) 2022-04-03T23:22:24.590 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:24.588+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:24.919 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:24.917+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:25.419 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:25.417+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:25.418206+0000) 2022-04-03T23:22:25.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:25.578+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:25.954 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:25.952+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:26.419 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:26.417+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:26.418420+0000) 2022-04-03T23:22:26.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:26.574+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:26.939 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:26.937+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:27.420 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:27.417+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:27.418644+0000) 2022-04-03T23:22:27.569 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:27.567+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:27.914 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:27.911+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:28.420 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:28.417+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:28.418882+0000) 2022-04-03T23:22:28.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:28.579+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:28.893 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:28.891+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:29.420 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:29.418+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:29.419097+0000) 2022-04-03T23:22:29.456 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T23:22:29.453+0000 7f2e43b1f700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T23:22:29.457 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T23:22:29.453+0000 7f2e44320700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T23:22:29.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:29.605+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:29.932 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:29.930+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:30.420 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:30.417+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:30.419347+0000) 2022-04-03T23:22:30.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:30.654+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:30.901 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:30.899+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:31.421 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:31.418+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:31.419581+0000) 2022-04-03T23:22:31.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:31.693+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:31.931 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:31.928+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:32.421 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:32.418+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:32.419824+0000) 2022-04-03T23:22:32.704 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:32.701+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:32.907 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:32.905+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:33.421 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:33.419+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:33.419969+0000) 2022-04-03T23:22:33.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:33.740+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:33.946 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:33.943+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:34.421 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:34.419+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:34.420107+0000) 2022-04-03T23:22:34.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:34.712+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:34.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:34.950+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:35.421 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:35.419+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:35.420321+0000) 2022-04-03T23:22:35.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:35.715+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:35.931 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:35.929+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:36.421 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:36.419+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:36.420468+0000) 2022-04-03T23:22:36.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:36.759+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:36.788 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:22:36.786+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:22:36.894 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:36.891+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:37.421 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:37.419+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:37.420645+0000) 2022-04-03T23:22:37.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:37.789+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:37.918 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:37.915+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:38.422 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:38.420+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:38.420873+0000) 2022-04-03T23:22:38.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:38.810+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:38.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:38.951+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:39.422 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:39.420+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:39.421040+0000) 2022-04-03T23:22:39.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:39.843+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:39.961 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:39.958+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:40.422 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:40.420+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:40.421245+0000) 2022-04-03T23:22:40.874 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:40.872+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:40.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:40.913+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:41.422 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:41.420+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:41.421491+0000) 2022-04-03T23:22:41.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:41.880+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:41.899 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:41.897+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:42.423 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:42.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:42.421728+0000) 2022-04-03T23:22:42.856 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:42.854+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:42.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:42.888+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:43.423 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:43.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:43.421905+0000) 2022-04-03T23:22:43.823 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:43.820+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:43.891 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:43.889+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:44.423 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:44.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:44.422043+0000) 2022-04-03T23:22:44.830 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:44.828+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:44.874 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:44.872+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:45.403 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:22:45.407 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:22:45.423 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:45.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:45.422211+0000) 2022-04-03T23:22:45.859 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:45.857+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:45.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:45.895+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:46.423 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:46.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:46.422411+0000) 2022-04-03T23:22:46.853 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:46.850+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:46.854 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:46.851+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:47.423 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:47.422+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:47.422565+0000) 2022-04-03T23:22:47.844 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:47.842+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:47.903 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:47.901+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:48.424 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:48.422+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:48.422760+0000) 2022-04-03T23:22:48.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:48.822+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:48.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:48.941+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:49.424 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:49.422+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:49.422938+0000) 2022-04-03T23:22:49.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:49.798+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:49.941 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:49.938+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:50.424 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:50.422+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:50.423128+0000) 2022-04-03T23:22:50.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:50.825+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:50.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:50.894+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:51.424 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:51.422+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:51.423303+0000) 2022-04-03T23:22:51.804 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:22:51.802+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:22:51.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:51.820+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:51.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:51.860+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:52.425 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:52.423+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:52.423480+0000) 2022-04-03T23:22:52.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:52.846+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:52.892 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:52.890+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:53.425 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:53.423+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:53.423694+0000) 2022-04-03T23:22:53.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:53.876+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:53.899 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:53.897+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:54.425 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:54.423+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:54.423899+0000) 2022-04-03T23:22:54.861 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:54.859+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:54.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:54.895+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:55.425 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:55.423+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:55.424108+0000) 2022-04-03T23:22:55.865 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:55.863+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:55.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:55.895+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:56.425 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:56.423+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:56.424321+0000) 2022-04-03T23:22:56.819 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:56.817+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:56.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:56.945+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:57.425 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:57.424+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:57.424547+0000) 2022-04-03T23:22:57.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:57.780+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:57.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:57.987+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:58.426 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:58.424+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:58.424756+0000) 2022-04-03T23:22:58.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:58.820+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:59.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:59.037+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:22:59.426 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:22:59.424+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:22:59.424881+0000) 2022-04-03T23:22:59.826 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:22:59.824+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:22:59.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:22:59.997+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:00.426 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:00.424+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:00.425054+0000) 2022-04-03T23:23:00.845 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:00.844+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:01.034 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:01.033+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:01.426 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:01.425+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:01.425258+0000) 2022-04-03T23:23:01.876 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:01.874+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:02.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:02.033+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:02.426 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:02.425+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:02.425446+0000) 2022-04-03T23:23:02.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:02.876+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:03.044 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:03.042+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:03.427 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:03.425+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:03.425659+0000) 2022-04-03T23:23:03.853 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:03.852+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:04.044 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:04.042+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:04.427 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:04.425+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:04.425882+0000) 2022-04-03T23:23:04.847 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:04.845+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:05.011 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:05.010+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:05.427 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:05.425+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:05.426038+0000) 2022-04-03T23:23:05.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:05.860+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:06.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:06.053+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:06.427 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:06.426+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:06.426256+0000) 2022-04-03T23:23:06.820 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:23:06.817+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:23:06.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:06.854+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:07.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:07.045+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:07.427 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:07.425+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:07.426458+0000) 2022-04-03T23:23:07.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:07.887+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:08.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:08.080+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:08.427 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:08.425+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:08.426625+0000) 2022-04-03T23:23:08.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:08.880+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:09.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:09.054+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:09.428 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:09.425+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:09.426807+0000) 2022-04-03T23:23:09.908 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:09.906+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:10.084 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:10.082+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:10.428 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:10.426+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:10.427028+0000) 2022-04-03T23:23:10.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:10.862+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:11.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:11.059+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:11.428 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:11.426+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:11.427252+0000) 2022-04-03T23:23:11.861 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:11.859+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:12.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:12.062+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:12.428 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:12.426+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:12.427454+0000) 2022-04-03T23:23:12.907 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:12.906+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:13.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:13.013+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:13.429 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:13.425+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:13.427680+0000) 2022-04-03T23:23:13.906 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:13.904+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:14.030 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:14.027+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:14.429 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:14.426+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:14.427921+0000) 2022-04-03T23:23:14.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:14.941+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:15.018 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:15.015+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:15.429 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:15.427+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:15.428130+0000) 2022-04-03T23:23:15.518 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:23:15.522 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:23:15.944 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:15.941+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:15.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:15.990+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:16.429 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:16.427+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:16.428285+0000) 2022-04-03T23:23:16.922 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:16.919+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:16.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:16.990+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:17.429 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:17.427+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:17.428434+0000) 2022-04-03T23:23:17.898 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:17.895+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:17.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:17.987+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:18.429 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:18.427+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:18.428612+0000) 2022-04-03T23:23:18.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:18.859+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:18.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:18.984+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:19.430 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:19.428+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:19.428859+0000) 2022-04-03T23:23:19.821 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:19.818+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:19.946 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:19.943+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:20.430 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:20.428+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:20.429055+0000) 2022-04-03T23:23:20.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:20.852+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:20.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:20.925+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:21.430 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:21.428+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:21.429253+0000) 2022-04-03T23:23:21.835 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:23:21.833+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:23:21.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:21.862+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:21.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:21.949+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:22.430 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:22.428+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:22.429411+0000) 2022-04-03T23:23:22.847 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:22.844+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:22.943 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:22.941+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:23.431 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:23.428+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:23.429576+0000) 2022-04-03T23:23:23.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:23.866+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:23.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:23.956+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:24.431 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:24.429+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:24.429810+0000) 2022-04-03T23:23:24.876 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:24.873+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:24.970 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:24.968+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:25.431 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:25.429+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:25.430026+0000) 2022-04-03T23:23:25.883 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:25.881+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:25.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:25.925+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:26.431 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:26.429+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:26.430215+0000) 2022-04-03T23:23:26.873 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:26.870+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:26.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:26.963+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:27.431 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:27.429+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:27.430417+0000) 2022-04-03T23:23:27.858 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:27.855+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:27.916 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:27.913+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:28.432 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:28.430+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:28.430621+0000) 2022-04-03T23:23:28.853 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:28.851+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:28.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:28.895+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:29.432 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:29.430+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:29.430854+0000) 2022-04-03T23:23:29.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:29.859+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:29.908 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:29.906+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:30.432 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:30.430+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:30.431050+0000) 2022-04-03T23:23:30.817 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:30.814+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:30.934 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:30.931+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:31.432 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:31.430+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:31.431212+0000) 2022-04-03T23:23:31.837 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:31.834+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:31.895 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:31.892+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:32.432 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:32.430+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:32.431405+0000) 2022-04-03T23:23:32.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:32.860+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:32.867 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:32.865+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:33.433 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:33.431+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:33.431637+0000) 2022-04-03T23:23:33.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:33.866+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:33.916 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:33.913+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:34.433 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:34.431+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:34.431861+0000) 2022-04-03T23:23:34.825 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:34.822+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:34.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:34.883+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:35.433 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:35.431+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:35.432063+0000) 2022-04-03T23:23:35.844 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:35.842+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:35.915 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:35.913+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:36.433 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:36.431+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:36.432222+0000) 2022-04-03T23:23:36.809 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:36.807+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:36.851 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:23:36.849+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:23:36.892 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:36.890+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:37.433 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:37.432+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:37.432417+0000) 2022-04-03T23:23:37.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:37.771+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:37.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:37.858+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:38.433 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:38.432+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:38.432619+0000) 2022-04-03T23:23:38.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:38.748+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:38.874 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:38.872+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:39.434 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:39.432+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:39.432864+0000) 2022-04-03T23:23:39.764 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:39.762+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:39.901 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:39.899+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:40.434 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:40.432+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:40.433053+0000) 2022-04-03T23:23:40.762 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:40.760+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:40.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:40.870+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:41.434 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:41.432+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:41.433199+0000) 2022-04-03T23:23:41.768 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:41.766+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:41.909 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:41.906+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:42.434 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:42.432+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:42.433408+0000) 2022-04-03T23:23:42.781 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:42.779+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:42.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:42.909+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:43.434 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:43.433+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:43.433554+0000) 2022-04-03T23:23:43.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:43.734+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:43.945 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:43.942+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:44.435 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:44.433+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:44.433746+0000) 2022-04-03T23:23:44.694 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:44.692+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:44.959 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:44.957+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:45.435 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:45.433+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:45.434020+0000) 2022-04-03T23:23:45.641 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:23:45.645 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:23:45.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:45.705+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:45.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:45.977+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:46.435 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:46.434+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:46.434300+0000) 2022-04-03T23:23:46.661 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:46.659+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:47.009 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:47.007+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:47.436 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:47.433+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:47.434588+0000) 2022-04-03T23:23:47.642 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:47.639+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:48.030 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:48.028+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:48.436 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:48.434+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:48.434765+0000) 2022-04-03T23:23:48.687 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:48.685+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:49.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:49.061+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:49.436 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:49.434+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:49.434950+0000) 2022-04-03T23:23:49.670 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:49.668+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:50.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:50.078+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:50.436 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:50.435+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:50.435166+0000) 2022-04-03T23:23:50.673 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:50.671+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:51.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:51.073+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:51.436 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:51.435+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:51.435344+0000) 2022-04-03T23:23:51.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:51.699+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:51.857 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:23:51.854+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:23:52.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:52.077+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:52.436 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:52.434+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:52.435557+0000) 2022-04-03T23:23:52.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:52.655+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:53.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:53.072+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:53.437 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:53.434+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:53.435823+0000) 2022-04-03T23:23:53.666 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:53.665+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:54.060 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:54.059+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:54.437 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:54.435+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:54.436064+0000) 2022-04-03T23:23:54.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:54.651+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:55.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:55.011+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:55.437 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:55.435+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:55.436305+0000) 2022-04-03T23:23:55.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:55.694+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:55.980 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:55.978+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:56.437 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:56.434+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:56.436524+0000) 2022-04-03T23:23:56.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:56.665+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:56.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:56.979+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:57.438 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:57.435+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:57.436724+0000) 2022-04-03T23:23:57.622 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:57.620+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:57.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:57.993+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:58.438 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:58.435+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:58.436939+0000) 2022-04-03T23:23:58.622 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:58.620+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:23:59.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:23:59.003+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:23:59.438 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:23:59.436+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:23:59.437178+0000) 2022-04-03T23:23:59.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:23:59.636+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:00.052 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:00.050+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:00.438 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:00.436+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:00.437420+0000) 2022-04-03T23:24:00.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:00.609+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:01.023 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:01.021+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:01.439 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:01.436+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:01.437680+0000) 2022-04-03T23:24:01.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:01.577+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:01.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:01.996+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:02.439 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:02.437+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:02.437913+0000) 2022-04-03T23:24:02.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:02.592+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:03.000 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:02.998+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:03.439 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:03.437+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:03.438165+0000) 2022-04-03T23:24:03.602 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:03.600+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:04.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:04.039+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:04.439 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:04.437+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:04.438373+0000) 2022-04-03T23:24:04.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:04.613+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:05.068 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:05.066+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:05.440 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:05.437+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:05.438611+0000) 2022-04-03T23:24:05.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:05.611+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:06.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:06.081+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:06.440 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:06.438+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:06.438850+0000) 2022-04-03T23:24:06.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:06.630+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:06.867 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:24:06.864+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:24:07.100 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:07.098+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:07.440 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:07.438+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:07.439120+0000) 2022-04-03T23:24:07.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:07.663+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:08.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:08.140+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:08.440 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:08.438+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:08.439381+0000) 2022-04-03T23:24:08.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:08.695+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:09.099 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:09.096+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:09.441 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:09.438+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:09.439621+0000) 2022-04-03T23:24:09.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:09.746+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:10.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:10.123+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:10.441 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:10.439+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:10.439878+0000) 2022-04-03T23:24:10.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:10.697+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:11.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:11.079+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:11.441 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:11.439+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:11.440107+0000) 2022-04-03T23:24:11.724 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:11.721+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:12.078 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:12.075+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:12.441 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:12.439+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:12.440284+0000) 2022-04-03T23:24:12.774 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:12.771+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:13.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:13.035+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:13.441 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:13.439+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:13.440456+0000) 2022-04-03T23:24:13.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:13.739+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:14.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:14.061+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:14.442 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:14.440+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:14.440654+0000) 2022-04-03T23:24:14.774 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:14.771+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:15.031 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:15.028+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:15.442 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:15.440+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:15.440884+0000) 2022-04-03T23:24:15.772 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:24:15.776 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:24:15.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:15.817+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:15.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:15.995+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:16.442 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:16.440+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:16.441118+0000) 2022-04-03T23:24:16.794 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:16.791+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:16.968 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:16.965+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:17.442 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:17.440+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:17.441350+0000) 2022-04-03T23:24:17.844 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:17.841+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:17.946 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:17.943+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:18.443 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:18.441+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:18.441565+0000) 2022-04-03T23:24:18.810 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:18.808+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:18.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:18.986+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:19.443 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:19.441+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:19.441837+0000) 2022-04-03T23:24:19.814 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:19.812+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:19.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:19.978+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:20.443 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:20.441+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:20.442033+0000) 2022-04-03T23:24:20.765 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:20.763+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:21.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:21.026+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:21.443 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:21.441+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:21.442255+0000) 2022-04-03T23:24:21.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:21.789+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:21.883 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:24:21.880+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:24:22.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:22.051+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:22.443 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:22.442+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:22.442509+0000) 2022-04-03T23:24:22.836 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:22.834+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:23.011 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:23.009+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:23.444 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:23.442+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:23.442752+0000) 2022-04-03T23:24:23.840 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:23.838+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:24.057 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:24.055+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:24.444 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:24.442+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:24.443034+0000) 2022-04-03T23:24:24.831 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:24.828+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:25.073 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:25.071+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:25.444 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:25.442+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:25.443276+0000) 2022-04-03T23:24:25.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:25.822+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:26.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:26.079+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:26.444 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:26.443+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:26.443487+0000) 2022-04-03T23:24:26.776 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:26.774+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:27.053 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:27.051+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:27.445 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:27.443+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:27.443699+0000) 2022-04-03T23:24:27.792 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:27.789+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:28.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:28.024+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:28.445 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:28.443+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:28.443945+0000) 2022-04-03T23:24:28.840 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:28.837+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:29.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:29.056+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:29.445 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:29.443+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:29.444137+0000) 2022-04-03T23:24:29.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:29.830+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:30.031 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:30.028+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:30.445 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:30.444+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:30.444350+0000) 2022-04-03T23:24:30.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:30.792+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:31.056 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:31.054+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:31.445 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:31.444+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:31.444604+0000) 2022-04-03T23:24:31.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:31.789+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:32.092 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:32.090+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:32.446 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:32.444+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:32.444861+0000) 2022-04-03T23:24:32.821 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:32.818+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:33.099 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:33.097+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:33.446 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:33.444+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:33.445053+0000) 2022-04-03T23:24:33.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:33.788+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:34.109 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:34.106+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:34.446 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:34.445+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:34.445269+0000) 2022-04-03T23:24:34.831 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:34.828+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:35.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:35.060+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:35.446 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:35.445+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:35.445495+0000) 2022-04-03T23:24:35.792 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:35.789+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:36.068 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:36.065+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:36.447 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:36.444+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:36.445726+0000) 2022-04-03T23:24:36.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:36.817+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:36.898 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:24:36.895+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:24:37.070 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:37.067+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:37.447 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:37.444+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:37.445990+0000) 2022-04-03T23:24:37.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:37.861+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:38.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:38.078+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:38.447 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:38.445+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:38.446238+0000) 2022-04-03T23:24:38.907 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:38.905+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:39.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:39.051+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:39.448 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:39.445+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:39.446481+0000) 2022-04-03T23:24:39.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:39.907+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:40.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:40.094+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:40.448 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:40.445+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:40.446745+0000) 2022-04-03T23:24:40.888 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:40.885+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:41.146 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:41.144+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:41.448 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:41.446+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:41.447028+0000) 2022-04-03T23:24:41.886 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:41.883+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:42.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:42.171+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:42.448 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:42.446+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:42.447245+0000) 2022-04-03T23:24:42.867 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:42.865+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:43.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:43.180+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:43.448 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:43.446+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:43.447487+0000) 2022-04-03T23:24:43.865 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:43.863+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:44.166 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:44.163+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:44.449 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:44.446+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:44.447754+0000) 2022-04-03T23:24:44.836 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:44.835+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:45.168 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:45.166+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:45.449 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:45.446+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:45.447985+0000) 2022-04-03T23:24:45.873 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:45.872+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:45.923 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:24:45.927 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:24:46.133 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:46.131+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:46.449 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:46.447+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:46.448213+0000) 2022-04-03T23:24:46.871 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:46.869+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:47.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:47.176+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:47.449 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:47.447+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:47.448439+0000) 2022-04-03T23:24:47.908 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:47.906+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:48.185 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:48.184+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:48.450 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:48.447+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:48.448686+0000) 2022-04-03T23:24:48.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:48.951+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:49.140 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:49.138+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:49.450 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:49.448+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:49.448903+0000) 2022-04-03T23:24:49.925 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:49.924+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:50.113 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:50.111+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:50.450 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:50.448+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:50.449119+0000) 2022-04-03T23:24:50.884 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:50.882+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:51.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:51.160+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:51.450 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:51.448+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:51.449353+0000) 2022-04-03T23:24:51.853 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:51.851+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:51.899 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:24:51.897+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:24:52.209 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:52.207+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:52.450 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:52.448+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:52.449616+0000) 2022-04-03T23:24:52.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:52.811+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:53.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:53.199+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:53.451 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:53.449+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:53.449802+0000) 2022-04-03T23:24:53.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:53.812+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:54.251 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:54.250+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:54.451 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:54.449+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:54.449970+0000) 2022-04-03T23:24:54.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:54.860+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:55.206 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:55.204+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:55.451 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:55.449+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:55.450188+0000) 2022-04-03T23:24:55.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:55.867+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:56.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:56.196+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:56.451 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:56.449+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:56.450431+0000) 2022-04-03T23:24:56.846 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:56.844+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:57.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:57.161+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:57.452 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:57.450+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:57.450685+0000) 2022-04-03T23:24:57.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:57.818+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:58.133 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:58.132+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:58.452 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:58.450+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:58.450931+0000) 2022-04-03T23:24:58.818 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:58.817+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:24:59.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:24:59.130+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:24:59.452 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:24:59.450+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:24:59.451181+0000) 2022-04-03T23:24:59.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:24:59.862+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:00.096 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:00.095+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:00.452 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:00.450+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:00.451455+0000) 2022-04-03T23:25:00.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:00.833+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:01.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:01.080+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:01.453 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:01.451+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:01.451708+0000) 2022-04-03T23:25:01.819 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:01.817+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:02.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:02.073+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:02.453 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:02.451+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:02.451953+0000) 2022-04-03T23:25:02.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:02.777+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:03.094 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:03.091+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:03.453 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:03.451+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:03.452200+0000) 2022-04-03T23:25:03.789 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:03.787+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:04.072 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:04.069+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:04.454 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:04.451+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:04.452440+0000) 2022-04-03T23:25:04.819 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:04.816+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:05.040 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:05.038+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:05.454 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:05.452+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:05.452616+0000) 2022-04-03T23:25:05.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:05.817+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:06.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:06.005+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:06.454 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:06.452+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:06.452853+0000) 2022-04-03T23:25:06.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:06.813+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:06.917 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:25:06.915+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:25:07.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:07.005+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:07.454 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:07.452+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:07.453071+0000) 2022-04-03T23:25:07.768 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:07.765+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:08.020 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:08.017+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:08.454 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:08.452+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:08.453288+0000) 2022-04-03T23:25:08.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:08.798+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:09.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:09.065+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:09.454 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:09.453+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:09.453498+0000) 2022-04-03T23:25:09.819 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:09.816+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:10.043 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:10.040+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:10.455 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:10.453+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:10.453752+0000) 2022-04-03T23:25:10.801 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:10.798+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:11.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:11.007+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:11.455 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:11.453+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:11.454067+0000) 2022-04-03T23:25:11.848 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:11.846+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:11.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:11.994+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:12.455 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:12.453+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:12.454308+0000) 2022-04-03T23:25:12.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:12.798+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:13.042 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:13.039+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:13.456 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:13.454+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:13.454552+0000) 2022-04-03T23:25:13.817 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:13.814+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:14.030 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:14.027+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:14.456 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:14.454+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:14.454797+0000) 2022-04-03T23:25:14.783 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:14.781+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:15.051 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:15.048+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:15.456 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:15.454+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:15.455042+0000) 2022-04-03T23:25:15.825 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:15.823+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:16.004 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:25:16.008 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:25:16.094 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:16.091+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:16.456 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:16.454+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:16.455273+0000) 2022-04-03T23:25:16.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:16.797+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:17.118 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:17.115+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:17.456 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:17.455+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:17.455497+0000) 2022-04-03T23:25:17.798 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:17.795+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:18.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:18.157+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:18.457 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:18.455+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:18.455779+0000) 2022-04-03T23:25:18.766 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:18.763+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:19.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:19.165+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:19.457 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:19.455+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:19.456017+0000) 2022-04-03T23:25:19.764 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:19.761+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:20.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:20.144+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:20.457 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:20.456+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:20.456264+0000) 2022-04-03T23:25:20.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:20.777+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:21.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:21.172+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:21.457 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:21.455+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:21.456521+0000) 2022-04-03T23:25:21.830 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:21.828+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:21.933 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:25:21.930+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:25:22.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:22.140+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:22.457 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:22.455+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:22.456692+0000) 2022-04-03T23:25:22.804 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:22.802+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:23.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:23.177+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:23.458 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:23.455+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:23.456908+0000) 2022-04-03T23:25:23.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:23.765+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:24.157 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:24.155+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:24.458 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:24.455+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:24.457150+0000) 2022-04-03T23:25:24.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:24.793+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:25.114 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:25.111+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:25.458 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:25.456+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:25.457391+0000) 2022-04-03T23:25:25.808 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:25.805+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:26.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:26.085+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:26.459 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:26.456+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:26.457636+0000) 2022-04-03T23:25:26.799 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:26.797+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:27.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:27.058+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:27.459 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:27.456+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:27.457874+0000) 2022-04-03T23:25:27.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:27.771+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:28.017 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:28.015+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:28.459 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:28.457+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:28.458130+0000) 2022-04-03T23:25:28.792 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:28.790+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:29.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:29.013+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:29.459 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:29.457+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:29.458366+0000) 2022-04-03T23:25:29.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:29.744+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:30.057 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:30.054+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:30.460 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:30.457+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:30.458560+0000) 2022-04-03T23:25:30.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:30.719+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:31.042 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:31.040+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:31.460 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:31.457+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:31.458728+0000) 2022-04-03T23:25:31.704 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:31.702+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:32.036 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:32.033+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:32.460 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:32.458+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:32.458949+0000) 2022-04-03T23:25:32.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:32.714+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:33.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:33.069+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:33.460 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:33.458+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:33.459185+0000) 2022-04-03T23:25:33.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:33.754+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:34.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:34.035+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:34.460 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:34.458+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:34.459431+0000) 2022-04-03T23:25:34.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:34.740+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:35.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:35.083+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:35.461 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:35.458+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:35.459674+0000) 2022-04-03T23:25:35.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:35.732+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:36.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:36.109+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:36.461 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:36.459+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:36.459830+0000) 2022-04-03T23:25:36.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:36.701+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:36.949 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:25:36.946+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:25:37.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:37.068+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:37.461 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:37.459+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:37.460049+0000) 2022-04-03T23:25:37.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:37.748+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:38.051 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:38.048+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:38.461 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:38.459+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:38.460210+0000) 2022-04-03T23:25:38.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:38.734+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:39.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:39.002+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:39.461 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:39.459+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:39.460436+0000) 2022-04-03T23:25:39.691 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:39.688+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:39.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:39.982+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:40.462 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:40.459+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:40.460677+0000) 2022-04-03T23:25:40.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:40.658+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:41.023 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:41.021+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:41.462 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:41.460+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:41.460897+0000) 2022-04-03T23:25:41.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:41.649+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:42.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:42.058+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:42.462 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:42.460+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:42.461067+0000) 2022-04-03T23:25:42.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:42.637+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:43.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:43.093+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:43.462 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:43.460+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:43.461300+0000) 2022-04-03T23:25:43.662 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:43.661+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:44.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:44.105+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:44.463 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:44.460+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:44.461502+0000) 2022-04-03T23:25:44.634 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:44.632+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:45.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:45.110+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:45.463 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:45.461+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:45.461719+0000) 2022-04-03T23:25:45.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:45.612+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:46.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:46.074+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:46.141 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:25:46.145 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:25:46.463 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:46.461+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:46.461993+0000) 2022-04-03T23:25:46.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:46.643+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:47.119 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:47.117+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:47.463 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:47.461+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:47.462167+0000) 2022-04-03T23:25:47.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:47.649+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:48.146 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:48.144+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:48.463 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:48.461+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:48.462415+0000) 2022-04-03T23:25:48.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:48.610+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:49.137 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:49.135+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:49.463 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:49.461+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:49.462586+0000) 2022-04-03T23:25:49.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:49.630+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:50.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:50.165+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:50.464 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:50.462+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:50.462753+0000) 2022-04-03T23:25:50.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:50.646+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:51.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:51.161+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:51.464 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:51.462+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:51.463006+0000) 2022-04-03T23:25:51.629 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:51.627+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 42 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:51.964 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:25:51.962+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:25:52.185 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:52.183+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:52.464 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:52.462+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:52.463252+0000) 2022-04-03T23:25:52.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:52.675+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:53.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:53.187+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:53.464 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:53.463+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:53.463454+0000) 2022-04-03T23:25:53.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:53.684+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:54.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:54.142+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:54.465 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:54.462+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:54.463690+0000) 2022-04-03T23:25:54.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:54.642+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:55.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:55.095+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:55.465 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:55.463+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:55.463918+0000) 2022-04-03T23:25:55.659 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:55.657+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:56.108 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:56.106+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:56.465 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:56.463+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:56.464091+0000) 2022-04-03T23:25:56.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:56.700+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:57.113 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:57.111+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:57.465 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:57.463+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:57.464293+0000) 2022-04-03T23:25:57.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:57.729+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:58.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:58.114+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:58.465 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:58.464+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:58.464451+0000) 2022-04-03T23:25:58.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:58.757+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:25:59.108 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:25:59.105+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:25:59.466 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:25:59.464+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:25:59.464608+0000) 2022-04-03T23:25:59.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:25:59.777+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:00.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:00.122+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:00.466 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:00.464+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:00.464832+0000) 2022-04-03T23:26:00.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:00.754+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:01.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:01.136+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:01.466 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:01.464+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:01.465006+0000) 2022-04-03T23:26:01.798 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:01.796+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:02.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:02.167+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:02.466 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:02.465+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:02.465217+0000) 2022-04-03T23:26:02.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:02.817+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:03.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:03.153+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:03.467 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:03.465+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:03.465475+0000) 2022-04-03T23:26:03.870 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:03.867+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:04.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:04.150+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:04.467 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:04.465+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:04.465703+0000) 2022-04-03T23:26:04.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:04.852+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:05.109 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:05.107+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:05.467 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:05.465+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:05.465933+0000) 2022-04-03T23:26:05.893 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:05.891+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:06.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:06.060+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:06.467 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:06.465+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:06.466113+0000) 2022-04-03T23:26:06.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:06.859+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:06.972 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:26:06.968+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:26:07.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:07.039+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:07.467 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:07.465+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:07.466272+0000) 2022-04-03T23:26:07.904 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:07.901+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:08.057 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:08.054+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:08.467 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:08.465+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:08.466473+0000) 2022-04-03T23:26:08.923 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:08.920+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:09.094 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:09.092+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:09.468 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:09.465+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:09.466657+0000) 2022-04-03T23:26:09.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:09.895+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:10.137 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:10.135+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:10.468 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:10.465+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:10.466854+0000) 2022-04-03T23:26:10.848 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:10.845+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:11.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:11.121+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:11.468 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:11.466+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:11.467032+0000) 2022-04-03T23:26:11.858 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:11.856+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:12.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:12.162+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:12.468 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:12.466+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:12.467254+0000) 2022-04-03T23:26:12.848 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:12.845+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:13.158 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:13.155+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:13.468 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:13.466+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:13.467463+0000) 2022-04-03T23:26:13.821 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:13.819+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:14.151 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:14.148+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:14.469 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:14.465+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:14.467751+0000) 2022-04-03T23:26:14.867 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:14.865+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:15.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:15.169+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:15.469 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:15.467+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:15.468051+0000) 2022-04-03T23:26:15.840 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:15.837+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:16.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:16.179+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:16.282 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:26:16.286 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:26:16.469 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:16.467+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:16.468301+0000) 2022-04-03T23:26:16.841 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:16.839+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:17.172 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:17.169+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:17.469 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:17.467+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:17.468548+0000) 2022-04-03T23:26:17.838 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:17.836+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:18.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:18.139+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:18.470 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:18.468+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:18.468814+0000) 2022-04-03T23:26:18.860 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:18.858+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:19.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:19.161+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:19.470 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:19.468+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:19.469058+0000) 2022-04-03T23:26:19.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:19.861+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:20.118 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:20.116+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:20.470 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:20.468+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:20.469262+0000) 2022-04-03T23:26:20.896 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:20.894+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:21.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:21.077+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:21.470 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:21.468+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:21.469470+0000) 2022-04-03T23:26:21.927 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:21.925+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:21.987 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:26:21.985+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:26:22.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:22.100+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:22.471 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:22.468+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:22.469686+0000) 2022-04-03T23:26:22.881 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:22.878+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:23.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:23.061+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:23.471 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:23.469+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:23.469963+0000) 2022-04-03T23:26:23.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:23.860+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:24.014 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:24.012+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:24.471 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:24.469+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:24.470195+0000) 2022-04-03T23:26:24.896 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:24.894+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:24.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:24.995+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:25.471 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:25.469+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:25.470394+0000) 2022-04-03T23:26:25.914 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:25.912+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:26.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:26.037+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:26.472 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:26.469+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:26.470601+0000) 2022-04-03T23:26:26.880 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:26.877+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:27.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:27.046+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:27.472 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:27.470+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:27.470861+0000) 2022-04-03T23:26:27.903 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:27.901+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:28.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:28.064+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:28.472 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:28.470+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:28.471110+0000) 2022-04-03T23:26:28.935 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:28.933+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:29.073 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:29.070+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:29.472 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:29.470+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:29.471351+0000) 2022-04-03T23:26:29.891 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:29.889+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:30.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:30.078+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:30.473 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:30.471+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:30.471600+0000) 2022-04-03T23:26:30.895 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:30.892+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:31.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:31.120+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:31.473 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:31.471+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:31.471874+0000) 2022-04-03T23:26:31.941 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:31.939+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:32.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:32.162+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:32.473 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:32.471+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:32.472082+0000) 2022-04-03T23:26:32.902 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:32.900+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:33.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:33.130+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:33.473 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:33.471+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:33.472300+0000) 2022-04-03T23:26:33.927 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:33.925+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:34.106 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:34.104+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:34.473 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:34.472+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:34.472540+0000) 2022-04-03T23:26:34.932 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:34.929+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:35.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:35.109+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:35.474 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:35.472+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:35.472813+0000) 2022-04-03T23:26:35.975 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:35.973+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:36.077 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:36.075+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:36.474 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:36.472+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:36.473050+0000) 2022-04-03T23:26:36.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:36.931+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:37.003 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:26:37.001+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:26:37.084 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:37.082+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:37.474 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:37.472+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:37.473252+0000) 2022-04-03T23:26:37.950 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:37.948+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:38.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:38.126+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:38.474 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:38.473+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:38.473503+0000) 2022-04-03T23:26:38.937 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:38.935+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:39.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:39.077+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:39.475 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:39.472+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:39.473704+0000) 2022-04-03T23:26:39.971 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:39.969+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:40.100 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:40.099+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:40.475 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:40.473+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:40.473924+0000) 2022-04-03T23:26:40.999 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:40.998+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:41.110 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:41.108+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:41.475 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:41.473+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:41.474078+0000) 2022-04-03T23:26:42.021 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:42.019+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:42.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:42.131+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:42.476 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:42.473+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:42.474298+0000) 2022-04-03T23:26:43.006 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:43.005+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:43.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:43.105+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:43.476 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:43.474+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:43.474518+0000) 2022-04-03T23:26:43.980 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:43.979+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:44.113 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:44.112+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:44.476 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:44.474+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:44.474802+0000) 2022-04-03T23:26:44.951 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:44.949+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:45.151 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:45.149+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:45.476 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:45.474+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:45.475010+0000) 2022-04-03T23:26:45.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:45.941+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:46.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:46.199+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:46.413 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:26:46.417 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:26:46.476 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:46.474+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:46.475214+0000) 2022-04-03T23:26:46.904 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:46.902+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:47.181 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:47.179+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:47.476 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:47.475+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:47.475434+0000) 2022-04-03T23:26:47.857 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:47.855+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:48.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:48.140+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:48.477 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:48.475+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:48.475687+0000) 2022-04-03T23:26:48.871 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:48.869+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:49.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:49.096+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:49.477 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:49.475+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:49.475951+0000) 2022-04-03T23:26:49.838 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:49.836+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:50.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:50.125+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:50.477 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:50.476+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:50.476090+0000) 2022-04-03T23:26:50.881 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:50.879+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:51.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:51.110+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:51.477 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:51.475+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:51.476315+0000) 2022-04-03T23:26:51.839 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:51.837+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:52.019 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:26:52.016+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:26:52.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:52.078+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:52.478 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:52.475+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:52.476521+0000) 2022-04-03T23:26:52.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:52.808+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:53.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:53.125+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:53.478 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:53.475+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:53.476782+0000) 2022-04-03T23:26:53.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:53.841+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:54.142 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:54.139+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:54.478 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:54.476+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:54.477037+0000) 2022-04-03T23:26:54.810 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:54.808+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:55.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:55.124+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:55.478 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:55.476+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:55.477284+0000) 2022-04-03T23:26:55.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:55.783+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:56.078 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:56.075+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:56.479 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:56.476+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:56.477479+0000) 2022-04-03T23:26:56.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:56.755+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:57.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:57.057+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:57.479 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:57.476+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:57.477641+0000) 2022-04-03T23:26:57.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:57.713+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:58.070 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:58.067+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:58.479 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:58.476+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:58.477876+0000) 2022-04-03T23:26:58.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:58.675+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:26:59.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:26:59.109+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:26:59.479 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:26:59.477+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:26:59.478123+0000) 2022-04-03T23:26:59.720 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:26:59.718+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:00.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:00.073+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:00.479 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:00.477+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:00.478293+0000) 2022-04-03T23:27:00.753 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:00.750+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:01.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:01.032+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:01.479 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:01.477+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:01.478509+0000) 2022-04-03T23:27:01.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:01.713+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:02.072 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:02.069+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:02.480 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:02.477+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:02.478682+0000) 2022-04-03T23:27:02.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:02.669+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:03.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:03.076+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:03.480 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:03.478+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:03.478909+0000) 2022-04-03T23:27:03.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:03.628+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:04.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:04.124+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:04.480 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:04.478+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:04.479159+0000) 2022-04-03T23:27:04.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:04.591+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:05.098 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:05.096+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:05.481 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:05.478+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:05.479398+0000) 2022-04-03T23:27:05.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:05.625+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:06.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:06.114+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:06.481 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:06.478+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:06.479576+0000) 2022-04-03T23:27:06.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:06.649+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:07.035 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:27:07.033+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:27:07.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:07.161+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:07.481 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:07.479+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:07.479814+0000) 2022-04-03T23:27:07.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:07.665+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:08.213 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:08.211+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:08.481 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:08.479+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:08.480055+0000) 2022-04-03T23:27:08.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:08.681+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:09.225 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:09.222+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:09.481 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:09.479+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:09.480232+0000) 2022-04-03T23:27:09.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:09.641+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:10.230 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:10.227+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:10.481 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:10.479+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:10.480392+0000) 2022-04-03T23:27:10.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:10.612+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:11.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:11.216+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:11.482 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:11.479+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:11.480617+0000) 2022-04-03T23:27:11.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:11.616+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:12.225 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:12.223+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:12.482 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:12.480+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:12.480814+0000) 2022-04-03T23:27:12.625 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:12.623+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:13.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:13.195+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:13.482 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:13.480+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:13.480965+0000) 2022-04-03T23:27:13.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:13.578+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:14.151 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:14.148+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:14.482 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:14.480+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:14.481154+0000) 2022-04-03T23:27:14.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:14.611+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:15.200 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:15.198+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:15.482 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:15.480+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:15.481325+0000) 2022-04-03T23:27:15.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:15.567+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:16.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:16.229+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:16.483 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:16.481+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:16.481518+0000) 2022-04-03T23:27:16.552 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:27:16.556 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:27:16.583 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:16.580+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:17.217 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:17.214+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:17.483 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:17.481+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:17.481819+0000) 2022-04-03T23:27:17.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:17.595+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:18.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:18.173+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:18.483 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:18.481+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:18.482001+0000) 2022-04-03T23:27:18.617 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:18.615+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:19.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:19.159+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:19.483 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:19.481+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:19.482240+0000) 2022-04-03T23:27:19.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:19.616+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:20.192 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:20.190+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:20.483 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:20.482+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:20.482436+0000) 2022-04-03T23:27:20.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:20.611+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:21.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:21.176+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:21.484 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:21.481+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:21.482656+0000) 2022-04-03T23:27:21.664 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:21.662+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:22.050 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:27:22.048+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:27:22.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:22.165+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:22.484 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:22.482+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:22.482922+0000) 2022-04-03T23:27:22.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:22.633+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:23.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:23.185+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:23.484 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:23.482+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:23.483113+0000) 2022-04-03T23:27:23.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:23.681+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:24.234 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:24.232+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:24.484 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:24.482+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:24.483329+0000) 2022-04-03T23:27:24.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:24.680+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:25.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:25.193+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:25.484 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:25.483+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:25.483555+0000) 2022-04-03T23:27:25.568 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-03T23:27:25.566+0000 7f2fd761c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T23:27:25.568 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-03T23:27:25.566+0000 7f2fd661a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T23:27:25.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:25.694+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:26.166 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:26.164+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:26.485 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:26.483+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:26.483733+0000) 2022-04-03T23:27:26.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:26.651+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:27.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:27.181+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:27.485 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:27.483+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:27.483912+0000) 2022-04-03T23:27:27.659 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:27.657+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:28.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:28.144+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:28.485 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:28.483+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:28.484083+0000) 2022-04-03T23:27:28.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:28.651+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:29.158 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:29.156+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:29.485 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:29.484+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:29.484252+0000) 2022-04-03T23:27:29.634 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:29.632+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:30.190 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:30.188+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:30.486 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:30.484+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:30.484467+0000) 2022-04-03T23:27:30.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:30.643+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:31.191 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:31.189+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:31.486 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:31.484+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:31.484660+0000) 2022-04-03T23:27:31.634 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:31.632+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:32.176 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:32.174+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:32.486 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:32.484+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:32.484856+0000) 2022-04-03T23:27:32.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:32.675+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:33.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:33.146+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:33.486 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:33.484+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:33.485043+0000) 2022-04-03T23:27:33.664 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:33.663+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:34.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:34.120+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:34.486 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:34.485+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:34.485258+0000) 2022-04-03T23:27:34.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:34.630+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:35.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:35.166+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:35.486 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:35.485+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:35.485487+0000) 2022-04-03T23:27:35.676 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:35.675+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:36.138 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:36.137+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:36.487 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:36.484+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:36.485703+0000) 2022-04-03T23:27:36.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:36.681+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:37.066 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:27:37.063+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:27:37.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:37.187+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:37.487 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:37.484+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:37.485880+0000) 2022-04-03T23:27:37.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:37.714+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:38.168 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:38.167+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:38.487 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:38.485+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:38.486099+0000) 2022-04-03T23:27:38.691 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:38.690+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:39.206 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:39.204+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:39.487 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:39.485+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:39.486344+0000) 2022-04-03T23:27:39.676 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:39.674+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:40.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:40.201+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:40.487 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:40.485+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:40.486573+0000) 2022-04-03T23:27:40.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:40.625+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:41.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:41.169+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:41.488 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:41.485+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:41.486809+0000) 2022-04-03T23:27:41.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:41.650+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:42.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:42.217+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:42.488 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:42.486+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:42.487052+0000) 2022-04-03T23:27:42.622 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:42.620+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:43.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:43.233+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:43.488 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:43.486+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:43.487303+0000) 2022-04-03T23:27:43.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:43.645+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:44.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:44.205+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:44.489 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:44.486+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:44.487525+0000) 2022-04-03T23:27:44.603 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:44.601+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:45.222 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:45.220+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:45.489 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:45.486+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:45.487748+0000) 2022-04-03T23:27:45.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:45.629+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:46.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:46.213+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:46.489 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:46.487+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:46.487980+0000) 2022-04-03T23:27:46.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:46.612+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:46.659 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:27:46.663 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:27:47.261 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:47.259+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:47.489 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:47.487+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:47.488227+0000) 2022-04-03T23:27:47.577 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:47.574+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:48.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:48.289+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:48.490 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:48.487+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:48.488474+0000) 2022-04-03T23:27:48.533 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:48.531+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:49.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:49.284+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:49.490 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:49.487+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:49.488719+0000) 2022-04-03T23:27:49.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:49.512+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:50.275 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:50.272+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:50.490 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:50.488+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:50.488984+0000) 2022-04-03T23:27:50.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:50.532+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:51.234 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:51.232+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:51.490 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:51.488+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:51.489190+0000) 2022-04-03T23:27:51.544 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:51.541+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:52.082 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:27:52.080+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:27:52.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:52.244+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:52.490 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:52.488+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:52.489416+0000) 2022-04-03T23:27:52.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:52.546+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:53.262 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:53.260+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:53.490 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:53.488+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:53.489633+0000) 2022-04-03T23:27:53.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:53.551+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:54.261 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:54.259+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:54.491 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:54.489+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:54.489869+0000) 2022-04-03T23:27:54.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:54.512+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:55.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:55.275+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:55.491 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:55.489+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:55.490077+0000) 2022-04-03T23:27:55.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:55.541+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:56.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:56.252+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:56.491 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:56.489+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:56.490336+0000) 2022-04-03T23:27:56.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:56.572+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:57.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:57.209+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:57.491 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:57.489+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:57.490598+0000) 2022-04-03T23:27:57.600 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:57.598+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:58.210 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:58.208+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:58.492 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:58.490+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:58.490887+0000) 2022-04-03T23:27:58.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:58.590+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:27:59.166 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:27:59.163+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:27:59.492 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:27:59.490+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:27:59.491130+0000) 2022-04-03T23:27:59.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:27:59.560+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:00.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:00.168+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:00.492 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:00.490+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:00.491373+0000) 2022-04-03T23:28:00.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:00.561+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:01.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:01.186+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:01.493 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:01.491+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:01.491636+0000) 2022-04-03T23:28:01.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:01.610+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:02.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:02.175+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:02.493 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:02.491+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:02.491878+0000) 2022-04-03T23:28:02.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:02.566+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:03.224 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:03.222+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:03.493 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:03.491+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:03.492099+0000) 2022-04-03T23:28:03.573 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:03.570+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:04.238 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:04.236+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:04.493 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:04.491+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:04.492357+0000) 2022-04-03T23:28:04.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:04.531+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:05.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:05.224+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:05.494 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:05.492+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:05.492576+0000) 2022-04-03T23:28:05.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:05.491+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:06.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:06.257+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:06.494 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:06.492+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:06.492856+0000) 2022-04-03T23:28:06.517 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:06.515+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:07.098 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:28:07.096+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:28:07.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:07.273+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:07.470 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:07.467+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:07.494 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:07.492+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:07.493084+0000) 2022-04-03T23:28:08.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:08.285+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:08.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:08.453+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:08.494 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:08.492+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:08.493317+0000) 2022-04-03T23:28:09.275 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:09.273+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:09.438 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:09.435+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:09.495 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:09.493+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:09.493529+0000) 2022-04-03T23:28:10.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:10.313+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:10.438 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:10.435+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:10.495 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:10.493+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:10.493790+0000) 2022-04-03T23:28:11.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:11.358+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:11.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:11.415+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:11.495 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:11.493+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:11.494001+0000) 2022-04-03T23:28:12.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:12.370+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:12.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:12.402+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:12.495 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:12.493+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:12.494222+0000) 2022-04-03T23:28:13.388 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:13.385+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:13.442 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:13.440+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:13.495 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:13.494+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:13.494436+0000) 2022-04-03T23:28:14.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:14.382+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:14.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:14.413+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:14.496 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:14.494+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:14.494682+0000) 2022-04-03T23:28:15.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:15.354+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:15.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:15.458+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:15.496 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:15.494+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:15.494933+0000) 2022-04-03T23:28:16.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:16.364+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:16.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:16.493+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:16.496 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:16.494+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:16.495135+0000) 2022-04-03T23:28:16.806 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:28:16.810 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:28:17.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:17.321+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:17.496 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:17.495+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:17.495375+0000) 2022-04-03T23:28:17.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:17.512+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:18.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:18.310+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:18.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:18.489+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:18.496 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:18.495+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:18.495576+0000) 2022-04-03T23:28:19.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:19.357+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:19.497 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:19.495+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:19.495716+0000) 2022-04-03T23:28:19.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:19.513+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:20.400 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:20.397+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:20.497 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:20.494+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:20.495915+0000) 2022-04-03T23:28:20.502 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:20.500+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:21.376 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:21.373+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:21.477 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:21.475+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:21.497 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:21.495+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:21.496168+0000) 2022-04-03T23:28:22.114 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:28:22.111+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:28:22.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:22.397+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:22.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:22.453+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:22.497 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:22.495+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:22.496356+0000) 2022-04-03T23:28:23.402 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:23.400+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:23.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:23.486+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:23.497 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:23.495+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:23.496586+0000) 2022-04-03T23:28:24.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:24.447+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:24.498 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:24.495+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:24.496798+0000) 2022-04-03T23:28:24.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:24.526+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:25.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:25.430+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:25.487 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:25.485+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:25.498 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:25.496+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:25.497032+0000) 2022-04-03T23:28:26.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:26.474+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:26.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:26.483+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:26.498 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:26.496+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:26.497278+0000) 2022-04-03T23:28:27.499 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:27.496+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:27.497518+0000) 2022-04-03T23:28:27.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:27.503+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:27.507 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:27.506+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:28.499 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:28.496+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:28.497754+0000) 2022-04-03T23:28:28.507 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:28.505+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:28.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:28.543+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:29.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:29.456+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:29.499 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:29.497+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:29.497975+0000) 2022-04-03T23:28:29.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:29.549+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:30.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:30.459+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:30.499 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:30.497+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:30.498098+0000) 2022-04-03T23:28:30.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:30.502+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:31.489 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:31.488+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:31.499 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:31.497+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:31.498302+0000) 2022-04-03T23:28:31.517 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:31.516+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:32.500 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:32.497+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:32.498520+0000) 2022-04-03T23:28:32.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:32.512+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:32.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:32.550+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:33.500 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:33.497+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:33.498753+0000) 2022-04-03T23:28:33.502 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:33.500+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:33.586 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:33.585+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:34.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:34.459+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:34.500 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:34.498+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:34.498975+0000) 2022-04-03T23:28:34.631 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:34.629+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:35.450 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:35.448+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:35.504 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:35.498+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:35.499198+0000) 2022-04-03T23:28:35.582 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:35.581+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:36.471 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:36.469+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:36.500 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:36.498+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:36.499407+0000) 2022-04-03T23:28:36.598 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:36.597+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:37.117 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:28:37.115+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:28:37.501 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:37.498+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:37.499642+0000) 2022-04-03T23:28:37.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:37.508+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:37.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:37.619+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:38.480 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:38.479+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:38.501 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:38.499+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:38.499885+0000) 2022-04-03T23:28:38.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:38.639+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:39.471 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:39.469+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:39.501 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:39.499+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:39.500163+0000) 2022-04-03T23:28:39.619 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:39.618+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:40.484 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:40.482+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:40.501 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:40.499+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:40.500371+0000) 2022-04-03T23:28:40.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:40.639+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:41.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:41.497+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:41.501 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:41.499+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:41.500592+0000) 2022-04-03T23:28:41.639 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:41.637+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:42.498 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:42.495+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:42.502 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:42.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:42.500828+0000) 2022-04-03T23:28:42.624 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:42.621+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:43.502 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:43.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:43.501026+0000) 2022-04-03T23:28:43.508 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:43.505+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:43.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:43.652+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:44.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:44.489+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:44.502 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:44.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:44.501281+0000) 2022-04-03T23:28:44.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:44.640+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:45.502 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:45.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:45.501485+0000) 2022-04-03T23:28:45.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:45.526+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:45.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:45.662+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:46.503 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:46.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:46.501719+0000) 2022-04-03T23:28:46.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:46.507+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:46.650 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:46.647+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:46.892 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:28:46.896 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:28:47.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:47.484+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:47.503 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:47.501+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:47.501925+0000) 2022-04-03T23:28:47.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:47.696+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:48.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:48.456+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:48.503 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:48.501+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:48.502143+0000) 2022-04-03T23:28:48.673 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:48.670+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:49.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:49.438+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:49.503 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:49.501+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:49.502352+0000) 2022-04-03T23:28:49.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:49.685+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:50.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:50.402+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:50.503 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:50.502+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:50.502501+0000) 2022-04-03T23:28:50.674 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:50.671+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:51.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:51.395+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 23 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:51.504 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:51.502+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:51.502704+0000) 2022-04-03T23:28:51.660 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:51.657+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:52.134 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:28:52.131+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:28:52.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:52.359+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:52.504 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:52.502+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:52.502910+0000) 2022-04-03T23:28:52.708 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:52.706+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:53.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:53.326+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:53.504 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:53.502+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:53.503039+0000) 2022-04-03T23:28:53.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:53.711+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:54.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:54.342+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:54.504 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:54.502+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:54.503184+0000) 2022-04-03T23:28:54.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:54.737+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:55.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:55.308+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:55.504 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:55.503+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:55.503380+0000) 2022-04-03T23:28:55.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:55.726+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:56.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:56.283+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:56.505 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:56.503+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:56.503639+0000) 2022-04-03T23:28:56.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:56.708+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:57.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:57.273+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:57.505 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:57.503+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:57.503871+0000) 2022-04-03T23:28:57.756 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:57.753+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:58.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:58.229+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:58.505 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:58.503+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:58.504103+0000) 2022-04-03T23:28:58.708 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:58.706+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:28:59.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:28:59.228+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:28:59.505 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:28:59.504+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:28:59.504317+0000) 2022-04-03T23:28:59.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:28:59.698+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:00.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:00.239+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:00.506 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:00.504+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:00.504495+0000) 2022-04-03T23:29:00.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:00.727+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:01.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:01.236+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:01.506 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:01.504+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:01.504667+0000) 2022-04-03T23:29:01.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:01.715+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:02.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:02.273+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:02.506 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:02.504+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:02.504904+0000) 2022-04-03T23:29:02.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:02.753+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:03.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:03.258+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:03.506 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:03.505+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:03.505174+0000) 2022-04-03T23:29:03.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:03.789+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:04.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:04.282+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:04.507 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:04.504+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:04.505437+0000) 2022-04-03T23:29:04.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:04.805+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:05.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:05.318+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:05.507 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:05.504+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:05.505681+0000) 2022-04-03T23:29:05.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:05.759+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:06.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:06.332+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:06.507 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:06.504+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:06.505921+0000) 2022-04-03T23:29:06.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:06.729+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:07.149 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:29:07.146+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:29:07.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:07.345+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:07.507 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:07.505+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:07.506191+0000) 2022-04-03T23:29:07.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:07.721+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:08.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:08.329+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:08.507 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:08.505+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:08.506397+0000) 2022-04-03T23:29:08.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:08.726+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:09.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:09.308+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:09.508 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:09.505+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:09.506631+0000) 2022-04-03T23:29:09.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:09.774+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:10.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:10.276+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:10.508 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:10.505+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:10.506817+0000) 2022-04-03T23:29:10.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:10.818+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:11.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:11.271+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:11.508 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:11.506+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:11.507040+0000) 2022-04-03T23:29:11.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:11.836+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:12.284 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:12.281+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:12.508 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:12.506+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:12.507262+0000) 2022-04-03T23:29:12.852 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:12.850+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:13.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:13.274+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:13.509 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:13.505+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:13.507523+0000) 2022-04-03T23:29:13.833 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:13.830+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:14.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:14.271+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:14.509 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:14.506+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:14.507757+0000) 2022-04-03T23:29:14.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:14.791+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:15.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:15.317+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:15.509 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:15.507+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:15.507989+0000) 2022-04-03T23:29:15.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:15.840+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:16.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:16.315+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:16.509 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:16.507+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:16.508259+0000) 2022-04-03T23:29:16.833 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:16.831+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:17.040 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:29:17.044 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:29:17.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:17.333+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:17.509 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:17.507+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:17.508503+0000) 2022-04-03T23:29:17.866 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:17.864+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:18.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:18.294+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:18.510 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:18.507+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:18.508716+0000) 2022-04-03T23:29:18.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:18.820+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:19.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:19.262+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:19.510 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:19.508+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:19.508879+0000) 2022-04-03T23:29:19.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:19.867+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:20.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:20.308+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:20.510 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:20.508+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:20.509041+0000) 2022-04-03T23:29:20.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:20.828+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:21.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:21.347+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:21.510 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:21.508+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:21.509200+0000) 2022-04-03T23:29:21.787 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:21.786+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:22.165 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:29:22.163+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:29:22.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:22.368+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:22.510 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:22.508+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:22.509411+0000) 2022-04-03T23:29:22.752 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:22.751+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:23.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:23.346+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:23.511 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:23.508+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:23.509645+0000) 2022-04-03T23:29:23.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:23.738+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:24.378 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:24.377+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:24.511 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:24.509+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:24.509840+0000) 2022-04-03T23:29:24.718 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:24.717+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:25.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:25.392+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:25.511 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:25.509+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:25.510068+0000) 2022-04-03T23:29:25.757 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:25.755+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:26.412 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:26.410+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:26.511 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:26.509+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:26.510262+0000) 2022-04-03T23:29:26.747 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:26.745+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:27.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:27.447+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:27.511 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:27.509+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:27.510427+0000) 2022-04-03T23:29:27.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:27.777+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:28.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:28.431+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:28.512 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:28.510+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:28.510644+0000) 2022-04-03T23:29:28.738 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:28.736+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:29.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:29.463+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:29.512 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:29.510+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:29.510883+0000) 2022-04-03T23:29:29.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:29.723+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:30.510 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:30.508+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:30.512 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:30.510+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:30.511129+0000) 2022-04-03T23:29:30.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:30.686+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:31.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:31.493+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:31.512 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:31.510+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:31.511389+0000) 2022-04-03T23:29:31.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:31.637+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:32.513 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:32.511+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:32.511606+0000) 2022-04-03T23:29:32.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:32.536+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:32.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:32.681+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:33.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:33.502+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:33.513 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:33.511+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:33.511815+0000) 2022-04-03T23:29:33.689 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:33.688+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:34.513 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:34.511+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:34.512079+0000) 2022-04-03T23:29:34.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:34.513+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:34.661 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:34.660+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:35.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:35.507+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:35.513 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:35.511+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:35.512317+0000) 2022-04-03T23:29:35.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:35.656+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:36.513 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:36.512+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:36.512511+0000) 2022-04-03T23:29:36.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:36.541+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:36.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:36.692+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:37.181 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:29:37.179+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:29:37.514 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:37.512+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:37.512731+0000) 2022-04-03T23:29:37.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:37.521+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:37.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:37.728+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:38.514 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:38.512+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:38.512946+0000) 2022-04-03T23:29:38.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:38.545+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:38.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:38.759+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:39.501 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:39.498+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:39.514 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:39.512+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:39.513155+0000) 2022-04-03T23:29:39.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:39.793+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:40.473 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:40.470+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:40.514 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:40.513+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:40.513369+0000) 2022-04-03T23:29:40.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:40.745+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:41.434 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:41.431+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:41.514 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:41.513+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:41.513624+0000) 2022-04-03T23:29:41.701 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:41.698+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:42.462 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:42.459+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:42.515 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:42.513+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:42.513876+0000) 2022-04-03T23:29:42.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:42.722+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:43.426 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:43.423+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:43.515 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:43.513+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:43.514043+0000) 2022-04-03T23:29:43.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:43.711+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:44.424 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:44.421+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:44.515 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:44.514+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:44.514277+0000) 2022-04-03T23:29:44.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:44.731+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:45.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:45.388+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:45.516 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:45.514+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:45.514524+0000) 2022-04-03T23:29:45.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:45.714+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:46.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:46.411+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:46.516 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:46.514+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:46.514790+0000) 2022-04-03T23:29:46.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:46.758+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:47.166 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:29:47.170 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:29:47.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:47.383+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:47.516 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:47.514+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:47.515015+0000) 2022-04-03T23:29:47.768 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:47.765+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:48.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:48.364+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:48.516 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:48.515+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:48.515228+0000) 2022-04-03T23:29:48.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:48.731+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:49.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:49.405+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:49.516 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:49.515+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:49.515478+0000) 2022-04-03T23:29:49.722 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:49.720+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:50.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:50.368+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:50.517 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:50.514+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:50.515725+0000) 2022-04-03T23:29:50.696 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:50.693+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:51.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:51.337+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:51.517 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:51.514+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:51.516008+0000) 2022-04-03T23:29:51.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:51.726+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:52.197 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:29:52.194+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:29:52.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:52.331+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:52.517 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:52.515+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:52.516250+0000) 2022-04-03T23:29:52.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:52.739+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:53.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:53.364+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:53.517 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:53.515+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:53.516459+0000) 2022-04-03T23:29:53.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:53.728+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:54.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:54.332+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:54.518 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:54.515+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:54.516680+0000) 2022-04-03T23:29:54.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:54.735+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:55.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:55.302+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:55.518 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:55.515+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:55.516915+0000) 2022-04-03T23:29:55.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:55.731+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:56.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:56.339+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:56.518 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:56.516+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:56.517125+0000) 2022-04-03T23:29:56.763 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:56.760+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:57.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:57.356+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:57.518 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:57.516+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:57.517287+0000) 2022-04-03T23:29:57.740 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:57.738+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:58.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:58.338+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:58.519 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:58.516+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:58.517499+0000) 2022-04-03T23:29:58.769 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:58.766+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:29:59.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:29:59.305+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:29:59.519 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:29:59.516+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:29:59.517697+0000) 2022-04-03T23:29:59.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:29:59.759+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:00.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:00.294+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:00.519 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:00.517+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:00.517915+0000) 2022-04-03T23:30:00.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:00.809+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:01.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:01.320+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:01.519 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:01.517+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:01.518149+0000) 2022-04-03T23:30:01.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:01.780+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:02.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:02.339+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:02.519 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:02.517+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:02.518363+0000) 2022-04-03T23:30:02.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:02.779+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:03.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:03.292+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:03.520 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:03.517+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:03.518582+0000) 2022-04-03T23:30:03.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:03.751+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:04.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:04.259+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:04.520 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:04.518+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:04.518786+0000) 2022-04-03T23:30:04.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:04.773+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:05.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:05.291+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:05.520 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:05.518+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:05.518955+0000) 2022-04-03T23:30:05.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:05.793+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:06.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:06.297+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:06.520 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:06.518+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:06.519117+0000) 2022-04-03T23:30:06.833 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:06.830+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:07.213 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:30:07.210+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:30:07.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:07.284+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:07.520 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:07.518+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:07.519372+0000) 2022-04-03T23:30:07.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:07.856+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:08.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:08.318+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:08.521 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:08.518+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:08.519632+0000) 2022-04-03T23:30:08.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:08.875+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:09.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:09.288+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:09.521 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:09.519+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:09.519811+0000) 2022-04-03T23:30:09.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:09.861+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:10.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:10.306+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:10.521 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:10.519+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:10.520047+0000) 2022-04-03T23:30:10.841 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:10.838+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:11.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:11.260+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:11.521 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:11.519+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:11.520226+0000) 2022-04-03T23:30:11.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:11.861+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:12.233 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:12.231+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:12.522 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:12.519+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:12.520440+0000) 2022-04-03T23:30:12.892 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:12.890+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:13.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:13.213+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:13.522 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:13.520+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:13.520680+0000) 2022-04-03T23:30:13.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:13.910+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:14.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:14.235+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:14.522 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:14.520+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:14.520921+0000) 2022-04-03T23:30:14.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:14.898+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:15.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:15.278+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:15.522 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:15.520+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:15.521119+0000) 2022-04-03T23:30:15.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:15.858+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:16.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:16.230+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:16.522 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:16.520+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:16.521286+0000) 2022-04-03T23:30:16.859 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:16.857+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:17.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:17.279+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:17.300 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:30:17.304 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:30:17.523 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:17.520+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:17.521525+0000) 2022-04-03T23:30:17.826 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:17.824+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:18.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:18.238+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:18.523 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:18.521+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:18.521759+0000) 2022-04-03T23:30:18.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:18.809+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:19.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:19.208+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:19.523 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:19.521+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:19.522031+0000) 2022-04-03T23:30:19.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:19.778+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:20.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:20.176+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:20.523 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:20.521+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:20.522221+0000) 2022-04-03T23:30:20.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:20.780+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:21.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:21.222+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:21.523 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:21.522+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:21.522488+0000) 2022-04-03T23:30:21.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:21.789+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:22.228 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:30:22.226+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:30:22.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:22.244+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:22.524 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:22.522+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:22.522675+0000) 2022-04-03T23:30:22.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:22.813+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:23.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:23.272+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:23.524 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:23.522+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:23.522890+0000) 2022-04-03T23:30:23.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:23.794+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:24.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:24.287+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:24.524 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:24.522+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:24.523002+0000) 2022-04-03T23:30:24.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:24.810+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:25.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:25.331+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:25.524 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:25.522+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:25.523212+0000) 2022-04-03T23:30:25.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:25.822+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:26.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:26.297+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:26.525 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:26.522+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:26.523477+0000) 2022-04-03T23:30:26.834 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:26.832+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:27.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:27.316+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:27.525 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:27.522+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:27.523672+0000) 2022-04-03T23:30:27.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:27.810+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:28.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:28.307+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:28.525 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:28.523+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:28.523883+0000) 2022-04-03T23:30:28.837 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:28.836+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:29.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:29.318+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:29.525 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:29.523+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:29.524131+0000) 2022-04-03T23:30:29.795 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:29.793+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:30.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:30.337+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:30.525 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:30.524+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:30.524362+0000) 2022-04-03T23:30:30.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:30.801+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:31.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:31.357+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:31.526 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:31.524+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:31.524532+0000) 2022-04-03T23:30:31.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:31.780+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:32.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:32.335+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:32.526 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:32.524+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:32.524788+0000) 2022-04-03T23:30:32.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:32.819+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:33.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:33.307+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:33.526 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:33.524+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:33.525047+0000) 2022-04-03T23:30:33.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:33.812+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:34.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:34.318+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:34.526 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:34.525+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:34.525290+0000) 2022-04-03T23:30:34.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:34.838+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:35.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:35.306+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:35.527 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:35.524+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:35.525534+0000) 2022-04-03T23:30:35.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:35.888+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:36.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:36.323+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:36.527 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:36.524+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:36.525798+0000) 2022-04-03T23:30:36.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:36.901+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:37.245 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:30:37.242+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:30:37.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:37.275+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:37.527 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:37.524+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:37.526047+0000) 2022-04-03T23:30:37.857 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:37.854+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:38.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:38.286+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:38.527 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:38.525+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:38.526251+0000) 2022-04-03T23:30:38.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:38.823+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:39.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:39.256+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:39.528 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:39.525+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:39.526448+0000) 2022-04-03T23:30:39.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:39.798+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:40.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:40.241+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:40.528 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:40.525+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:40.526686+0000) 2022-04-03T23:30:40.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:40.819+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:41.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:41.282+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:41.528 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:41.525+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:41.526946+0000) 2022-04-03T23:30:41.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:41.851+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:42.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:42.295+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:42.528 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:42.526+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:42.527163+0000) 2022-04-03T23:30:42.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:42.819+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:43.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:43.250+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:43.528 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:43.526+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:43.527398+0000) 2022-04-03T23:30:43.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:43.869+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:44.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:44.283+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:44.531 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:44.526+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:44.527650+0000) 2022-04-03T23:30:44.906 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:44.904+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:45.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:45.235+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:45.529 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:45.527+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:45.527828+0000) 2022-04-03T23:30:45.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:45.858+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:46.205 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:46.202+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:46.529 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:46.527+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:46.528065+0000) 2022-04-03T23:30:46.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:46.862+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:47.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:47.240+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:47.529 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:47.527+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:47.528262+0000) 2022-04-03T23:30:47.539 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:30:47.543 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:30:47.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:47.874+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:48.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:48.245+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:48.529 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:48.527+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:48.528418+0000) 2022-04-03T23:30:48.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:48.846+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:49.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:49.197+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:49.530 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:49.527+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:49.528634+0000) 2022-04-03T23:30:49.844 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:49.841+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:50.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:50.239+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:50.530 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:50.528+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:50.528854+0000) 2022-04-03T23:30:50.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:50.826+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:51.217 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:51.214+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:51.530 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:51.528+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:51.529080+0000) 2022-04-03T23:30:51.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:51.836+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:52.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:52.225+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:52.250 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:30:52.248+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:30:52.530 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:52.528+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:52.529296+0000) 2022-04-03T23:30:52.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:52.871+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:53.195 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:53.193+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:53.531 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:53.528+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:53.529521+0000) 2022-04-03T23:30:53.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:53.882+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:54.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:54.171+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:54.531 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:54.529+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:54.529698+0000) 2022-04-03T23:30:54.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:54.883+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:55.174 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:55.172+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:55.531 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:55.529+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:55.529893+0000) 2022-04-03T23:30:55.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:55.856+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:56.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:56.221+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:56.531 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:56.529+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:56.530109+0000) 2022-04-03T23:30:56.892 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:56.890+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:57.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:57.262+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:57.531 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:57.529+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:57.530304+0000) 2022-04-03T23:30:57.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:57.920+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:58.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:58.306+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:58.531 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:58.529+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:58.530475+0000) 2022-04-03T23:30:58.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:58.962+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:30:59.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:30:59.286+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:30:59.532 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:30:59.530+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:30:59.530677+0000) 2022-04-03T23:30:59.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:30:59.981+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:00.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:00.300+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:00.532 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:00.530+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:00.530886+0000) 2022-04-03T23:31:01.023 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:01.021+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:01.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:01.277+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:01.532 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:01.530+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:01.531085+0000) 2022-04-03T23:31:02.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:02.039+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:02.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:02.289+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:02.532 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:02.530+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:02.531310+0000) 2022-04-03T23:31:03.084 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:03.082+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:03.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:03.322+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:03.533 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:03.531+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:03.531531+0000) 2022-04-03T23:31:04.101 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:04.099+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:04.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:04.327+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:04.533 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:04.531+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:04.531707+0000) 2022-04-03T23:31:05.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:05.140+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:05.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:05.292+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:05.533 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:05.531+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:05.531912+0000) 2022-04-03T23:31:06.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:06.132+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:06.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:06.268+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:06.533 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:06.531+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:06.532135+0000) 2022-04-03T23:31:07.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:07.108+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:07.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:07.236+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:07.265 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:31:07.263+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:31:07.533 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:07.531+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:07.532358+0000) 2022-04-03T23:31:08.069 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:08.067+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:08.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:08.270+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:08.534 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:08.532+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:08.532600+0000) 2022-04-03T23:31:09.116 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:09.114+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:09.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:09.313+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:09.534 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:09.532+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:09.532802+0000) 2022-04-03T23:31:10.093 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:10.091+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:10.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:10.348+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:10.534 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:10.532+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:10.533047+0000) 2022-04-03T23:31:11.113 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:11.111+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:11.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:11.359+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:11.534 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:11.532+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:11.533221+0000) 2022-04-03T23:31:12.102 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:12.101+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:12.375 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:12.373+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:12.534 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:12.533+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:12.533433+0000) 2022-04-03T23:31:13.106 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:13.105+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:13.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:13.338+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:13.535 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:13.533+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:13.533680+0000) 2022-04-03T23:31:14.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:14.146+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:14.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:14.321+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:14.535 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:14.533+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:14.533875+0000) 2022-04-03T23:31:15.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:15.133+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:15.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:15.321+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:15.535 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:15.533+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:15.534107+0000) 2022-04-03T23:31:16.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:16.130+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:16.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:16.296+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:16.535 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:16.534+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:16.534364+0000) 2022-04-03T23:31:17.099 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:17.097+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:17.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:17.293+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:17.536 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:17.534+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:17.534585+0000) 2022-04-03T23:31:17.653 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:31:17.657 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:31:18.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:18.105+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:18.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:18.326+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:18.536 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:18.534+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:18.534792+0000) 2022-04-03T23:31:19.104 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:19.103+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:19.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:19.324+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:19.536 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:19.534+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:19.534945+0000) 2022-04-03T23:31:20.116 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:20.115+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:20.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:20.317+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:20.536 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:20.534+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:20.535121+0000) 2022-04-03T23:31:21.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:21.151+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:21.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:21.346+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:21.536 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:21.534+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:21.535345+0000) 2022-04-03T23:31:22.191 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:22.190+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:22.281 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:31:22.278+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:31:22.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:22.367+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:22.536 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:22.534+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:22.535476+0000) 2022-04-03T23:31:23.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:23.226+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:23.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:23.337+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:23.536 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:23.534+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:23.535629+0000) 2022-04-03T23:31:24.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:24.227+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:24.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:24.317+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:24.537 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:24.534+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:24.535818+0000) 2022-04-03T23:31:25.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:25.187+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:25.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:25.315+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:25.537 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:25.534+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:25.535960+0000) 2022-04-03T23:31:26.152 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:26.150+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:26.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:26.308+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:26.537 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:26.535+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:26.536203+0000) 2022-04-03T23:31:27.104 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:27.102+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:27.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:27.325+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:27.538 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:27.535+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:27.536436+0000) 2022-04-03T23:31:28.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:28.060+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:28.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:28.358+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:28.538 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:28.535+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:28.536665+0000) 2022-04-03T23:31:29.102 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:29.099+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:29.405 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:29.402+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:29.538 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:29.536+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:29.536893+0000) 2022-04-03T23:31:30.087 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:30.084+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:30.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:30.374+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:30.538 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:30.536+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:30.537144+0000) 2022-04-03T23:31:31.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:31.129+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:31.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:31.411+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:31.538 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:31.536+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:31.537336+0000) 2022-04-03T23:31:32.087 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:32.085+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:32.396 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:32.394+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:32.538 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:32.536+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:32.537507+0000) 2022-04-03T23:31:33.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:33.068+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:33.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:33.391+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:33.539 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:33.536+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:33.537663+0000) 2022-04-03T23:31:34.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:34.022+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:34.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:34.402+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:34.539 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:34.537+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:34.537821+0000) 2022-04-03T23:31:35.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:35.059+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:35.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:35.394+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:35.539 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:35.537+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:35.538022+0000) 2022-04-03T23:31:36.020 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:36.017+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:36.438 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:36.435+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:36.539 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:36.537+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:36.538260+0000) 2022-04-03T23:31:36.971 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:36.968+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:37.297 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:31:37.295+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:31:37.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:37.389+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:37.539 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:37.537+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:37.538512+0000) 2022-04-03T23:31:37.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:37.986+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:38.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:38.342+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:38.540 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:38.538+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:38.538729+0000) 2022-04-03T23:31:38.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:38.986+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:39.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:39.347+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:39.540 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:39.538+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:39.538864+0000) 2022-04-03T23:31:39.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:39.955+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:40.337 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:40.334+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:40.540 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:40.538+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:40.539000+0000) 2022-04-03T23:31:40.969 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:40.966+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:41.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:41.351+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:41.540 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:41.538+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:41.539212+0000) 2022-04-03T23:31:41.939 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:41.936+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:42.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:42.326+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:42.540 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:42.538+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:42.539442+0000) 2022-04-03T23:31:42.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:42.956+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:43.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:43.376+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:43.541 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:43.539+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:43.539695+0000) 2022-04-03T23:31:43.920 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:43.917+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:44.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:44.355+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:44.541 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:44.539+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:44.539923+0000) 2022-04-03T23:31:44.874 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:44.872+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:45.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:45.359+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:45.541 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:45.539+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:45.540163+0000) 2022-04-03T23:31:45.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:45.839+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:46.409 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:46.406+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:46.541 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:46.539+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:46.540331+0000) 2022-04-03T23:31:46.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:46.846+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:47.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:47.374+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:47.542 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:47.540+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:47.540573+0000) 2022-04-03T23:31:47.777 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:31:47.781 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:31:47.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:47.811+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:48.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:48.413+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:48.542 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:48.540+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:48.540863+0000) 2022-04-03T23:31:48.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:48.845+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:49.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:49.441+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:49.542 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:49.540+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:49.541072+0000) 2022-04-03T23:31:49.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:49.804+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:50.472 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:50.469+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:50.542 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:50.540+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:50.541285+0000) 2022-04-03T23:31:50.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:50.835+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:51.500 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:51.497+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:51.543 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:51.541+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:51.541522+0000) 2022-04-03T23:31:51.850 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:51.848+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:52.313 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:31:52.311+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:31:52.477 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:52.474+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:52.543 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:52.541+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:52.541748+0000) 2022-04-03T23:31:52.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:52.845+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:53.428 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:53.426+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:53.543 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:53.541+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:53.542014+0000) 2022-04-03T23:31:53.850 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:53.848+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:54.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:54.473+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:54.543 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:54.541+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:54.542258+0000) 2022-04-03T23:31:54.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:54.829+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:55.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:55.516+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:55.543 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:55.542+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:55.542499+0000) 2022-04-03T23:31:55.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:55.872+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:56.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:56.529+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:56.544 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:56.542+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:56.542804+0000) 2022-04-03T23:31:56.915 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:56.913+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:57.507 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:57.505+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:57.544 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:57.542+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:57.543050+0000) 2022-04-03T23:31:57.906 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:57.904+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:58.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:58.481+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:58.544 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:58.543+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:58.543287+0000) 2022-04-03T23:31:58.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:58.919+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:31:59.502 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:31:59.500+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:31:59.544 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:31:59.542+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:31:59.543557+0000) 2022-04-03T23:31:59.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:31:59.921+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:00.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:00.482+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:00.545 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:00.543+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:00.543756+0000) 2022-04-03T23:32:00.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:00.934+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:01.533 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:01.531+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:01.545 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:01.543+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:01.543958+0000) 2022-04-03T23:32:01.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:01.919+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:02.545 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:02.544+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:02.544130+0000) 2022-04-03T23:32:02.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:02.554+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:02.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:02.949+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:03.545 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:03.544+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:03.544310+0000) 2022-04-03T23:32:03.606 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:03.603+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:03.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:03.986+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:04.545 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:04.543+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:04.544452+0000) 2022-04-03T23:32:04.606 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:04.604+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:04.945 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:04.943+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:05.546 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:05.543+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:05.544669+0000) 2022-04-03T23:32:05.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:05.616+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:05.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:05.979+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:06.546 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:06.543+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:06.544908+0000) 2022-04-03T23:32:06.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:06.650+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:06.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:06.936+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:07.329 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:32:07.326+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:32:07.546 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:07.544+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:07.545152+0000) 2022-04-03T23:32:07.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:07.619+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:07.934 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:07.932+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:08.547 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:08.544+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:08.545389+0000) 2022-04-03T23:32:08.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:08.619+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:08.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:08.920+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:09.547 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:09.544+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:09.545630+0000) 2022-04-03T23:32:09.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:09.613+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:09.892 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:09.890+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:10.547 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:10.544+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:10.545883+0000) 2022-04-03T23:32:10.586 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:10.584+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:10.883 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:10.881+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:11.537 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:11.536+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:11.547 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:11.544+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:11.546087+0000) 2022-04-03T23:32:11.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:11.883+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:12.547 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:12.545+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:12.546306+0000) 2022-04-03T23:32:12.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:12.561+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:12.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:12.895+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:13.548 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:13.545+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:13.546555+0000) 2022-04-03T23:32:13.572 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:13.571+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:13.905 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:13.903+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:14.548 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:14.545+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:14.546824+0000) 2022-04-03T23:32:14.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:14.590+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:14.892 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:14.890+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:15.546 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:15.544+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:15.548 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:15.546+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:15.547063+0000) 2022-04-03T23:32:15.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:15.880+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:16.548 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:16.546+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:16.547300+0000) 2022-04-03T23:32:16.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:16.579+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:16.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:16.879+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:17.549 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:17.546+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:17.547515+0000) 2022-04-03T23:32:17.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:17.555+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:17.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:17.849+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:17.990 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:32:17.994 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:32:18.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:18.523+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:18.549 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:18.547+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:18.547811+0000) 2022-04-03T23:32:18.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:18.850+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:19.549 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:19.547+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:19.548069+0000) 2022-04-03T23:32:19.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:19.559+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:19.816 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:19.814+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:20.549 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:20.547+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:20.548319+0000) 2022-04-03T23:32:20.585 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:20.583+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:20.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:20.859+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:21.550 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:21.547+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:21.548525+0000) 2022-04-03T23:32:21.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:21.574+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:21.909 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:21.906+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:22.345 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:32:22.343+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:32:22.550 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:22.548+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:22.548758+0000) 2022-04-03T23:32:22.609 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:22.607+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:22.895 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:22.892+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:23.550 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:23.548+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:23.549030+0000) 2022-04-03T23:32:23.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:23.634+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:23.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:23.853+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:24.550 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:24.548+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:24.549202+0000) 2022-04-03T23:32:24.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:24.679+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:24.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:24.845+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:25.550 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:25.548+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:25.549382+0000) 2022-04-03T23:32:25.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:25.720+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:25.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:25.876+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:32:26.551 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:26.549+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:26.549623+0000) 2022-04-03T23:32:26.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:26.683+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:26.859 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:26.857+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:27.551 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:27.549+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:27.549897+0000) 2022-04-03T23:32:27.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:27.698+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:27.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:27.861+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:28.551 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:28.549+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:28.550105+0000) 2022-04-03T23:32:28.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:28.682+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:28.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:28.862+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:29.551 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:29.549+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:29.550318+0000) 2022-04-03T23:32:29.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:29.660+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:29.816 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:29.814+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:30.552 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:30.550+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:30.550514+0000) 2022-04-03T23:32:30.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:30.623+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:30.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:30.825+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:31.552 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:31.549+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:31.550751+0000) 2022-04-03T23:32:31.603 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:31.600+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:31.862 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:31.859+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:32.552 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:32.550+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:32.551025+0000) 2022-04-03T23:32:32.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:32.629+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:32.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:32.824+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:33.552 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:33.550+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:33.551238+0000) 2022-04-03T23:32:33.608 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:33.606+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:33.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:33.824+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:34.552 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:34.551+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:34.551414+0000) 2022-04-03T23:32:34.573 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:34.570+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:34.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:34.824+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:35.553 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:35.551+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:35.551588+0000) 2022-04-03T23:32:35.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:35.578+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:35.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:35.805+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:36.553 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:36.551+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:36.551802+0000) 2022-04-03T23:32:36.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:36.560+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:36.828 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:36.825+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:37.361 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:32:37.359+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:32:37.553 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:37.551+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:37.552039+0000) 2022-04-03T23:32:37.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:37.568+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:37.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:37.809+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:38.553 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:38.551+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:38.552289+0000) 2022-04-03T23:32:38.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:38.591+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:38.857 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:38.854+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:39.553 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:39.552+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:39.552531+0000) 2022-04-03T23:32:39.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:39.632+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:39.869 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:39.866+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:40.554 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:40.552+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:40.552724+0000) 2022-04-03T23:32:40.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:40.609+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:40.913 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:40.911+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:41.554 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:41.552+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:41.552903+0000) 2022-04-03T23:32:41.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:41.643+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:41.910 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:41.907+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:42.554 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:42.552+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:42.553071+0000) 2022-04-03T23:32:42.608 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:42.606+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:42.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:42.922+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:43.554 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:43.553+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:43.553289+0000) 2022-04-03T23:32:43.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:43.645+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:43.929 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:43.927+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:44.554 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:44.553+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:44.553510+0000) 2022-04-03T23:32:44.694 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:44.691+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:44.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:44.962+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:45.555 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:45.553+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:45.553729+0000) 2022-04-03T23:32:45.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:45.727+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:45.969 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:45.966+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:46.555 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:46.553+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:46.553951+0000) 2022-04-03T23:32:46.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:46.712+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:46.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:46.955+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:47.556 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:47.554+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:47.554203+0000) 2022-04-03T23:32:47.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:47.741+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:47.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:47.990+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:48.111 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:32:48.115 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:32:48.555 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:48.554+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:48.554409+0000) 2022-04-03T23:32:48.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:48.777+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:48.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:48.977+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:49.556 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:49.553+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:49.554630+0000) 2022-04-03T23:32:49.788 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:49.785+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:49.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:49.948+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:50.556 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:50.553+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:50.554887+0000) 2022-04-03T23:32:50.799 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:50.797+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:50.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:50.987+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:51.556 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:51.554+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:51.555145+0000) 2022-04-03T23:32:51.764 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:51.761+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:51.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:51.950+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:52.377 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:32:52.373+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:32:52.557 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:52.554+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:52.555366+0000) 2022-04-03T23:32:52.762 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:52.759+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:52.906 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:52.903+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:53.557 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:53.553+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:53.555609+0000) 2022-04-03T23:32:53.804 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:53.802+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:53.939 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:53.937+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:54.557 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:54.554+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:54.555865+0000) 2022-04-03T23:32:54.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:54.783+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:54.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:54.945+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:55.557 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:55.555+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:55.556090+0000) 2022-04-03T23:32:55.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:55.811+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:55.929 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:55.926+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:56.557 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:56.555+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:56.556317+0000) 2022-04-03T23:32:56.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:56.831+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:56.902 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:56.900+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:57.557 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:57.555+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:57.556502+0000) 2022-04-03T23:32:57.786 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:57.783+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:57.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:57.853+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:58.558 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:58.555+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:58.556707+0000) 2022-04-03T23:32:58.753 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:58.751+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:58.828 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:58.826+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:32:59.558 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:32:59.556+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:32:59.556887+0000) 2022-04-03T23:32:59.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:32:59.758+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:32:59.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:32:59.836+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:00.558 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:00.556+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:00.557038+0000) 2022-04-03T23:33:00.801 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:00.798+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:00.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:00.811+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:01.558 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:01.556+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:01.557219+0000) 2022-04-03T23:33:01.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:01.777+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:01.792 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:01.791+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:02.558 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:02.556+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:02.557391+0000) 2022-04-03T23:33:02.816 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:02.814+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:02.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:02.818+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:03.559 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:03.556+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:03.557596+0000) 2022-04-03T23:33:03.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:03.785+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:03.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:03.851+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:04.559 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:04.557+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:04.557800+0000) 2022-04-03T23:33:04.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:04.775+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:04.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:04.840+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:05.559 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:05.557+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:05.558037+0000) 2022-04-03T23:33:05.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:05.785+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:05.892 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:05.890+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:06.559 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:06.557+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:06.558233+0000) 2022-04-03T23:33:06.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:06.791+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:06.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:06.902+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:07.393 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:33:07.390+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:33:07.559 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:07.557+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:07.558410+0000) 2022-04-03T23:33:07.807 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:07.805+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:07.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:07.876+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:08.560 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:08.557+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:08.558626+0000) 2022-04-03T23:33:08.790 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:08.788+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:08.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:08.883+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:09.560 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:09.558+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:09.558865+0000) 2022-04-03T23:33:09.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:09.783+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:09.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:09.868+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:10.560 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:10.558+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:10.559106+0000) 2022-04-03T23:33:10.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:10.765+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:10.852 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:10.850+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:11.560 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:11.558+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:11.559360+0000) 2022-04-03T23:33:11.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:11.728+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:11.857 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:11.855+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:12.561 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:12.559+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:12.559619+0000) 2022-04-03T23:33:12.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:12.716+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:12.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:12.817+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:13.561 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:13.559+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:13.559862+0000) 2022-04-03T23:33:13.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:13.724+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:13.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:13.774+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:14.561 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:14.559+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:14.560060+0000) 2022-04-03T23:33:14.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:14.696+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:14.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:14.733+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:15.561 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:15.559+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:15.560272+0000) 2022-04-03T23:33:15.705 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:15.703+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:15.710 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:15.709+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:16.562 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:16.559+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:16.560482+0000) 2022-04-03T23:33:16.678 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:16.676+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:16.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:16.680+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:17.562 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:17.559+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:17.560663+0000) 2022-04-03T23:33:17.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:17.655+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:17.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:17.665+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:18.258 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:33:18.263 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:33:18.562 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:18.560+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:18.560894+0000) 2022-04-03T23:33:18.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:18.623+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:18.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:18.696+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:19.562 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:19.560+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:19.561150+0000) 2022-04-03T23:33:19.666 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:19.663+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:19.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:19.728+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:20.563 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:20.560+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:20.561399+0000) 2022-04-03T23:33:20.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:20.681+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:20.709 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:20.706+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:21.563 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:21.561+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:21.561556+0000) 2022-04-03T23:33:21.655 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:21.652+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:21.660 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:21.658+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:22.398 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:33:22.396+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:33:22.563 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:22.561+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:22.561746+0000) 2022-04-03T23:33:22.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:22.666+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:22.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:22.687+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:23.563 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:23.561+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:23.561958+0000) 2022-04-03T23:33:23.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:23.685+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:23.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:23.724+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:24.563 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:24.561+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:24.562124+0000) 2022-04-03T23:33:24.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:24.684+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:24.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:24.706+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:25.563 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:25.562+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:25.562280+0000) 2022-04-03T23:33:25.674 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:25.671+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:25.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:25.697+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:26.564 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:26.562+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:26.562471+0000) 2022-04-03T23:33:26.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:26.690+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:26.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:26.725+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:27.564 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:27.562+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:27.562686+0000) 2022-04-03T23:33:27.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:27.678+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:27.764 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:27.761+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:28.564 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:28.562+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:28.562910+0000) 2022-04-03T23:33:28.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:28.660+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:28.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:28.725+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:29.564 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:29.562+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:29.563160+0000) 2022-04-03T23:33:29.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:29.623+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:29.759 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:29.756+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:30.565 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:30.563+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:30.563402+0000) 2022-04-03T23:33:30.586 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:30.584+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:30.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:30.740+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:31.564 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:31.561+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:31.565 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:31.563+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:31.563628+0000) 2022-04-03T23:33:31.788 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:31.785+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:32.564 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:32.561+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:32.588 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:32.563+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:32.563864+0000) 2022-04-03T23:33:32.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:32.748+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:33.520 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:33.517+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:33.565 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:33.563+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:33.564043+0000) 2022-04-03T23:33:33.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:33.778+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:34.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:34.537+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:34.565 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:34.563+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:34.564171+0000) 2022-04-03T23:33:34.823 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:34.821+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:35.565 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:35.563+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:35.564337+0000) 2022-04-03T23:33:35.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:35.578+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:35.823 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:35.820+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:36.566 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:36.563+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:36.564523+0000) 2022-04-03T23:33:36.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:36.626+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:36.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:36.792+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:37.414 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:33:37.411+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:33:37.566 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:37.563+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:37.564686+0000) 2022-04-03T23:33:37.591 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:37.589+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:37.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:37.822+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:38.566 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:38.563+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:38.564836+0000) 2022-04-03T23:33:38.636 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:38.633+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:38.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:38.821+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:39.566 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:39.564+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:39.564966+0000) 2022-04-03T23:33:39.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:39.601+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:39.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:39.869+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:40.566 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:40.564+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:40.565124+0000) 2022-04-03T23:33:40.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:40.605+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:40.875 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:40.872+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:41.566 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:41.564+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:41.565318+0000) 2022-04-03T23:33:41.610 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:41.607+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:41.907 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:41.904+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 44 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:42.567 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:42.564+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:42.565523+0000) 2022-04-03T23:33:42.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:42.630+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:42.930 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:42.927+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:43.567 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:43.564+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:43.565702+0000) 2022-04-03T23:33:43.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:43.627+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:43.922 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:43.919+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:44.567 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:44.565+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:44.565907+0000) 2022-04-03T23:33:44.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:44.673+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:44.929 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:44.927+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:45.567 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:45.565+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:45.566092+0000) 2022-04-03T23:33:45.701 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:45.699+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:45.972 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:45.970+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:46.567 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:46.565+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:46.566256+0000) 2022-04-03T23:33:46.738 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:46.736+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:47.020 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:47.018+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:47.567 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:47.565+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:47.566424+0000) 2022-04-03T23:33:47.721 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:47.719+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:47.988 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:47.986+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:48.388 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:33:48.392 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:33:48.568 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:48.565+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:48.566594+0000) 2022-04-03T23:33:48.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:48.697+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:48.971 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:48.969+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:49.568 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:49.566+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:49.566795+0000) 2022-04-03T23:33:49.675 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:49.673+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:50.018 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:50.016+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:50.568 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:50.566+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:50.566946+0000) 2022-04-03T23:33:50.704 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:50.701+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:51.057 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:51.054+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:51.568 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:51.566+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:51.567158+0000) 2022-04-03T23:33:51.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:51.666+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:52.052 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:52.050+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:52.430 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:33:52.427+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:33:52.568 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:52.566+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:52.567308+0000) 2022-04-03T23:33:52.661 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:52.659+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:53.014 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:53.012+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:53.569 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:53.566+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:53.567497+0000) 2022-04-03T23:33:53.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:53.642+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:54.059 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:54.056+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:54.569 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:54.567+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:54.567690+0000) 2022-04-03T23:33:54.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:54.666+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:55.045 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:55.042+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:55.569 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:55.567+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:55.567897+0000) 2022-04-03T23:33:55.704 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:55.702+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:56.007 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:56.005+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:56.569 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:56.567+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:56.568093+0000) 2022-04-03T23:33:56.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:56.742+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:57.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:57.002+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:57.569 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:57.567+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:57.568311+0000) 2022-04-03T23:33:57.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:57.780+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:58.041 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:58.039+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:58.570 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:58.567+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:58.568520+0000) 2022-04-03T23:33:58.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:58.797+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:33:59.089 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:33:59.087+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:33:59.570 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:33:59.568+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:33:59.568741+0000) 2022-04-03T23:33:59.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:33:59.782+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:00.071 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:00.069+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:00.570 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:00.568+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:00.568982+0000) 2022-04-03T23:34:00.774 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:00.772+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:01.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:01.071+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:01.570 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:01.568+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:01.569193+0000) 2022-04-03T23:34:01.777 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:01.775+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:02.041 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:02.039+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:02.570 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:02.568+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:02.569411+0000) 2022-04-03T23:34:02.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:02.740+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:02.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:02.993+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:03.571 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:03.569+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:03.569569+0000) 2022-04-03T23:34:03.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:03.756+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:03.957 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:03.956+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:04.571 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:04.569+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:04.569792+0000) 2022-04-03T23:34:04.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:04.722+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:04.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:04.977+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:05.571 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:05.569+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:05.569990+0000) 2022-04-03T23:34:05.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:05.677+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:05.938 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:05.936+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:06.571 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:06.569+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:06.570139+0000) 2022-04-03T23:34:06.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:06.660+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:06.957 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:06.956+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:07.446 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:34:07.443+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:34:07.572 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:07.570+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:07.570363+0000) 2022-04-03T23:34:07.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:07.691+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:07.955 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:07.953+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:08.572 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:08.569+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:08.570581+0000) 2022-04-03T23:34:08.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:08.661+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:08.920 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:08.918+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:09.572 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:09.570+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:09.570799+0000) 2022-04-03T23:34:09.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:09.661+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:09.951 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:09.950+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:10.572 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:10.570+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:10.571001+0000) 2022-04-03T23:34:10.615 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:10.613+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:10.934 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:10.932+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:11.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:11.566+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:11.572 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:11.570+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:11.571232+0000) 2022-04-03T23:34:11.900 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:11.897+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:12.564 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:12.561+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:12.572 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:12.571+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:12.571472+0000) 2022-04-03T23:34:12.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:12.906+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:13.573 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:13.571+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:13.571699+0000) 2022-04-03T23:34:13.589 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:13.586+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:13.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:13.943+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:14.562 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:14.559+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:14.573 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:14.571+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:14.571893+0000) 2022-04-03T23:34:14.991 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:14.989+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:15.516 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:15.513+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:15.573 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:15.571+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:15.572053+0000) 2022-04-03T23:34:15.974 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:15.971+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:16.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:16.512+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:16.573 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:16.572+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:16.572261+0000) 2022-04-03T23:34:17.002 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:16.999+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:17.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:17.532+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:17.574 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:17.572+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:17.572501+0000) 2022-04-03T23:34:17.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:17.989+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:18.528 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:34:18.532 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:34:18.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:18.546+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:18.574 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:18.571+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:18.572735+0000) 2022-04-03T23:34:18.986 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:18.983+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:19.574 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:19.571+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:19.572952+0000) 2022-04-03T23:34:19.586 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:19.584+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:20.030 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:20.027+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:20.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:20.568+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:20.574 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:20.572+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:20.573167+0000) 2022-04-03T23:34:21.035 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:21.032+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:21.574 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:21.572+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:21.573387+0000) 2022-04-03T23:34:21.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:21.610+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:22.066 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:22.063+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:22.461 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:34:22.458+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:34:22.575 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:22.572+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:22.573596+0000) 2022-04-03T23:34:22.596 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:22.594+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:23.098 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:23.095+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:23.575 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:23.572+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:23.573862+0000) 2022-04-03T23:34:23.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:23.579+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:24.086 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:24.083+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:24.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:24.543+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:24.575 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:24.572+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:24.574106+0000) 2022-04-03T23:34:25.131 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:25.128+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:25.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:25.536+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:25.575 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:25.573+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:25.574356+0000) 2022-04-03T23:34:26.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:26.177+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:26.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:26.546+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:26.576 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:26.573+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:26.574588+0000) 2022-04-03T23:34:27.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:27.153+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:27.576 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:27.573+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:27.574816+0000) 2022-04-03T23:34:27.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:27.591+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:28.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:28.162+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:28.576 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:28.574+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:28.575098+0000) 2022-04-03T23:34:28.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:28.613+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:29.117 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:29.115+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:29.576 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:29.574+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:29.575333+0000) 2022-04-03T23:34:29.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:29.639+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:30.117 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:30.114+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:30.577 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:30.574+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:30.575571+0000) 2022-04-03T23:34:30.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:30.600+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:31.089 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:31.086+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:31.577 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:31.575+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:31.575873+0000) 2022-04-03T23:34:31.597 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:31.595+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:32.132 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:32.129+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:32.577 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:32.575+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:32.576108+0000) 2022-04-03T23:34:32.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:32.609+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:33.093 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:33.091+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:33.577 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:33.575+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:33.576352+0000) 2022-04-03T23:34:33.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:33.659+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:34.141 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:34.138+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:34.578 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:34.575+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:34.576571+0000) 2022-04-03T23:34:34.637 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:34.634+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:35.171 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:35.168+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:35.578 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:35.576+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:35.576798+0000) 2022-04-03T23:34:35.686 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:35.683+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:36.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:36.185+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:36.578 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:36.576+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:36.577025+0000) 2022-04-03T23:34:36.685 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:36.683+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:37.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:37.201+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:37.477 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:34:37.475+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:34:37.578 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:37.576+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:37.577261+0000) 2022-04-03T23:34:37.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:37.724+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:38.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:38.182+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:38.579 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:38.576+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:38.577494+0000) 2022-04-03T23:34:38.764 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:38.761+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:39.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:39.201+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:39.579 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:39.577+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:39.577749+0000) 2022-04-03T23:34:39.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:39.797+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:40.217 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:40.214+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:40.579 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:40.577+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:40.577982+0000) 2022-04-03T23:34:40.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:40.778+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:41.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:41.201+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:41.579 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:41.577+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:41.578152+0000) 2022-04-03T23:34:41.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:41.805+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:42.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:42.225+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:42.579 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:42.577+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:42.578370+0000) 2022-04-03T23:34:42.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:42.797+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:43.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:43.239+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:43.580 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:43.578+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:43.578610+0000) 2022-04-03T23:34:43.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:43.837+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:44.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:44.275+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:44.580 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:44.578+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:44.578908+0000) 2022-04-03T23:34:44.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:44.840+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:45.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:45.287+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:45.580 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:45.578+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:45.579085+0000) 2022-04-03T23:34:45.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:45.825+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:46.271 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:46.268+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:46.580 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:46.578+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:46.579292+0000) 2022-04-03T23:34:46.797 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:46.795+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:47.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:47.313+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:47.581 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:47.579+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:47.579484+0000) 2022-04-03T23:34:47.828 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:47.826+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:48.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:48.267+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:48.581 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:48.579+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:48.579692+0000) 2022-04-03T23:34:48.658 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:34:48.662 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:34:48.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:48.813+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:49.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:49.254+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:49.581 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:49.579+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:49.579922+0000) 2022-04-03T23:34:49.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:49.842+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:50.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:50.290+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:50.581 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:50.579+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:50.580085+0000) 2022-04-03T23:34:50.816 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:50.814+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:51.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:51.319+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:51.581 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:51.579+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:51.580307+0000) 2022-04-03T23:34:51.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:51.797+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:52.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:52.294+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:52.493 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:34:52.491+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:34:52.582 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:52.580+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:52.580499+0000) 2022-04-03T23:34:52.814 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:52.812+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:53.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:53.267+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:53.582 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:53.580+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:53.580707+0000) 2022-04-03T23:34:53.768 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:53.766+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:54.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:54.241+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:54.582 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:54.580+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:54.580931+0000) 2022-04-03T23:34:54.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:54.806+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:55.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:55.267+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:55.582 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:55.580+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:55.581143+0000) 2022-04-03T23:34:55.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:55.779+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:56.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:56.257+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:56.582 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:56.581+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:56.581345+0000) 2022-04-03T23:34:56.749 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:56.747+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:57.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:57.256+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:57.583 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:57.581+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:57.581555+0000) 2022-04-03T23:34:57.795 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:57.793+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:58.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:58.283+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:58.583 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:58.581+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:58.581705+0000) 2022-04-03T23:34:58.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:58.813+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:34:59.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:34:59.253+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:34:59.583 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:34:59.581+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:34:59.581939+0000) 2022-04-03T23:34:59.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:34:59.859+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:00.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:00.244+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:00.583 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:00.581+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:00.582149+0000) 2022-04-03T23:35:00.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:00.885+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:01.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:01.252+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:01.583 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:01.582+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:01.582358+0000) 2022-04-03T23:35:01.866 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:01.864+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:02.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:02.268+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:02.584 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:02.582+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:02.582572+0000) 2022-04-03T23:35:02.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:02.884+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:03.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:03.247+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:03.584 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:03.582+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:03.582844+0000) 2022-04-03T23:35:03.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:03.862+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:04.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:04.267+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:04.584 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:04.581+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:04.583085+0000) 2022-04-03T23:35:04.905 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:04.904+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:05.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:05.295+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:05.585 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:05.582+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:05.583331+0000) 2022-04-03T23:35:05.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:05.885+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:06.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:06.286+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:06.584 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:06.582+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:06.583528+0000) 2022-04-03T23:35:06.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:06.848+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:07.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:07.314+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:07.509 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:35:07.506+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:35:07.585 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:07.582+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:07.583740+0000) 2022-04-03T23:35:07.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:07.817+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:08.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:08.363+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:08.585 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:08.582+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:08.584008+0000) 2022-04-03T23:35:08.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:08.827+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:09.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:09.351+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:09.585 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:09.583+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:09.584218+0000) 2022-04-03T23:35:09.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:09.858+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:10.378 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:10.375+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:10.585 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:10.583+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:10.584430+0000) 2022-04-03T23:35:10.892 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:10.890+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:11.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:11.349+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:11.586 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:11.583+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:11.584643+0000) 2022-04-03T23:35:11.894 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:11.892+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:12.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:12.344+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:12.586 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:12.583+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:12.584883+0000) 2022-04-03T23:35:12.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:12.930+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:13.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:13.385+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:13.586 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:13.584+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:13.585067+0000) 2022-04-03T23:35:13.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:13.972+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:14.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:14.416+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:14.586 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:14.584+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:14.585305+0000) 2022-04-03T23:35:14.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:14.950+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:15.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:15.385+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:15.587 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:15.584+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:15.585510+0000) 2022-04-03T23:35:15.962 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:15.959+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:16.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:16.381+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:16.587 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:16.584+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:16.585655+0000) 2022-04-03T23:35:16.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:16.986+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:17.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:17.360+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:17.587 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:17.585+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:17.585889+0000) 2022-04-03T23:35:17.986 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:17.983+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:18.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:18.338+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:18.587 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:18.585+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:18.586118+0000) 2022-04-03T23:35:18.930 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:35:18.934 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:35:19.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:19.021+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:19.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:19.313+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:19.588 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:19.585+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:19.586364+0000) 2022-04-03T23:35:20.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:20.059+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:20.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:20.340+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:20.588 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:20.585+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:20.586578+0000) 2022-04-03T23:35:21.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:21.059+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:21.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:21.381+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:21.588 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:21.586+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:21.586826+0000) 2022-04-03T23:35:22.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:22.060+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:22.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:22.383+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:22.511 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:35:22.509+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:35:22.588 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:22.586+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:22.587106+0000) 2022-04-03T23:35:23.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:23.039+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:23.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:23.432+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:23.588 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:23.586+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:23.587301+0000) 2022-04-03T23:35:24.073 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:24.071+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:24.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:24.404+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:24.588 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:24.586+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:24.587451+0000) 2022-04-03T23:35:25.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:25.114+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:35:25.423 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:25.420+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:25.589 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:25.586+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:25.587687+0000) 2022-04-03T23:35:26.090 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:26.088+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:26.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:26.408+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:26.589 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:26.587+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:26.587896+0000) 2022-04-03T23:35:27.092 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:27.089+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:27.400 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:27.397+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:27.589 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:27.587+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:27.588048+0000) 2022-04-03T23:35:28.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:28.123+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:28.438 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:28.436+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:28.589 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:28.587+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:28.588224+0000) 2022-04-03T23:35:29.091 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:29.088+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:29.442 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:29.439+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:29.668 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:29.587+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:29.588447+0000) 2022-04-03T23:35:30.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:30.137+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:30.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:30.392+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:30.590 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:30.588+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:30.588687+0000) 2022-04-03T23:35:31.160 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:31.158+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:31.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:31.375+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:31.590 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:31.588+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:31.588873+0000) 2022-04-03T23:35:32.118 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:32.115+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:32.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:32.369+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:32.590 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:32.588+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:32.589091+0000) 2022-04-03T23:35:33.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:33.110+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:33.421 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:33.418+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:33.590 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:33.588+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:33.589357+0000) 2022-04-03T23:35:34.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:34.061+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:34.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:34.443+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:34.591 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:34.589+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:34.589609+0000) 2022-04-03T23:35:35.094 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:35.091+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:35.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:35.407+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:35.591 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:35.589+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:35.589820+0000) 2022-04-03T23:35:36.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:36.121+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:36.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:36.415+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:36.591 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:36.589+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:36.590082+0000) 2022-04-03T23:35:37.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:37.140+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:37.426 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:37.423+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:37.527 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:35:37.525+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:35:37.591 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:37.589+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:37.590252+0000) 2022-04-03T23:35:38.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:38.145+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:38.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:38.442+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:38.592 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:38.589+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:38.590489+0000) 2022-04-03T23:35:39.133 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:39.130+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:39.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:39.402+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:39.592 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:39.590+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:39.590703+0000) 2022-04-03T23:35:40.113 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:40.111+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:40.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:40.428+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:40.592 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:40.590+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:40.590891+0000) 2022-04-03T23:35:41.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:41.158+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:41.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:41.422+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:41.592 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:41.590+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:41.591076+0000) 2022-04-03T23:35:42.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:42.128+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:42.429 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:42.427+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:42.592 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:42.590+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:42.591252+0000) 2022-04-03T23:35:43.110 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:43.108+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:43.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:43.443+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:43.593 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:43.591+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:43.591464+0000) 2022-04-03T23:35:44.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:44.126+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:44.478 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:44.476+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:44.593 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:44.591+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:44.591650+0000) 2022-04-03T23:35:45.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:45.158+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:45.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:45.457+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:45.593 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:45.591+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:45.591882+0000) 2022-04-03T23:35:46.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:46.121+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:46.486 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:46.484+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:46.593 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:46.591+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:46.592055+0000) 2022-04-03T23:35:47.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:47.123+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:47.466 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:47.464+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:47.593 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:47.592+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:47.592260+0000) 2022-04-03T23:35:48.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:48.162+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:48.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:48.438+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:48.594 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:48.592+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:48.592492+0000) 2022-04-03T23:35:49.077 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:35:49.081 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:35:49.152 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:49.150+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:49.466 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:49.464+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:49.593 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:49.591+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:49.592666+0000) 2022-04-03T23:35:50.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:50.110+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:50.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:50.456+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:50.594 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:50.591+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:50.592877+0000) 2022-04-03T23:35:51.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:51.061+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:51.464 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:51.462+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:51.594 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:51.591+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:51.593052+0000) 2022-04-03T23:35:52.027 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:52.025+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:52.457 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:52.456+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:52.543 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:35:52.540+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:35:52.594 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:52.592+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:52.593280+0000) 2022-04-03T23:35:53.018 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:53.016+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:53.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:53.489+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:53.595 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:53.592+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:53.593504+0000) 2022-04-03T23:35:53.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:53.996+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:54.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:54.485+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:54.595 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:54.592+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:54.593733+0000) 2022-04-03T23:35:55.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:55.031+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:55.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:55.495+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:55.595 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:55.593+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:55.594007+0000) 2022-04-03T23:35:56.053 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:56.052+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:56.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:56.464+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:56.595 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:56.593+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:56.594193+0000) 2022-04-03T23:35:57.065 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:57.063+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:57.493 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:57.491+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:57.595 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:57.593+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:57.594410+0000) 2022-04-03T23:35:58.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:58.031+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:58.477 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:58.475+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:58.595 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:58.593+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:58.594622+0000) 2022-04-03T23:35:59.018 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:35:59.017+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:35:59.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:35:59.507+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:35:59.596 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:35:59.593+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:35:59.594813+0000) 2022-04-03T23:36:00.050 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:00.048+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:00.460 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:00.458+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:00.596 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:00.594+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:00.595017+0000) 2022-04-03T23:36:01.044 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:01.042+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:01.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:01.466+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:01.596 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:01.594+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:01.595194+0000) 2022-04-03T23:36:02.020 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:02.017+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:02.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:02.436+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:02.597 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:02.594+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:02.595438+0000) 2022-04-03T23:36:03.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:03.023+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:03.402 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:03.399+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:03.597 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:03.594+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:03.595653+0000) 2022-04-03T23:36:04.022 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:04.019+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:04.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:04.358+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:04.597 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:04.595+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:04.595876+0000) 2022-04-03T23:36:05.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:05.056+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:05.364 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:05.361+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:05.597 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:05.595+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:05.596138+0000) 2022-04-03T23:36:06.014 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:06.011+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:06.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:06.338+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:06.598 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:06.595+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:06.596385+0000) 2022-04-03T23:36:07.045 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:07.042+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:07.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:07.354+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:07.559 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:36:07.556+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:36:07.598 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:07.595+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:07.596621+0000) 2022-04-03T23:36:08.017 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:08.014+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:08.400 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:08.397+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:08.598 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:08.596+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:08.596810+0000) 2022-04-03T23:36:08.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:08.994+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:09.432 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:09.429+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:09.598 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:09.596+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:09.597020+0000) 2022-04-03T23:36:09.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:09.978+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:10.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:10.423+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:10.598 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:10.596+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:10.597237+0000) 2022-04-03T23:36:11.018 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:11.015+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:11.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:11.391+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:11.599 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:11.596+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:11.597461+0000) 2022-04-03T23:36:11.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:11.985+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:12.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:12.346+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:12.599 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:12.597+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:12.597628+0000) 2022-04-03T23:36:12.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:12.990+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:13.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:13.302+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:13.599 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:13.597+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:13.597765+0000) 2022-04-03T23:36:14.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:14.007+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:14.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:14.339+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:14.599 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:14.597+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:14.597993+0000) 2022-04-03T23:36:15.057 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:15.054+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:15.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:15.351+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:15.599 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:15.597+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:15.598221+0000) 2022-04-03T23:36:16.089 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:16.086+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:16.388 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:16.385+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:16.599 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:16.597+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:16.598423+0000) 2022-04-03T23:36:17.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:17.046+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:17.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:17.382+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:17.600 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:17.598+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:17.598565+0000) 2022-04-03T23:36:18.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:18.005+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:18.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:18.423+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:18.600 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:18.598+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:18.598797+0000) 2022-04-03T23:36:19.044 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:19.042+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:19.158 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:36:19.161 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:36:19.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:19.382+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:19.600 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:19.598+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:19.599047+0000) 2022-04-03T23:36:20.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:20.083+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:20.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:20.389+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:20.600 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:20.598+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:20.599286+0000) 2022-04-03T23:36:21.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:21.077+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:21.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:21.370+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:21.600 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:21.599+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:21.599489+0000) 2022-04-03T23:36:22.057 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:22.054+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:22.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:22.360+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:22.575 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:36:22.572+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:36:22.601 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:22.599+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:22.599730+0000) 2022-04-03T23:36:23.022 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:23.019+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:23.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:23.326+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:23.601 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:23.599+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:23.599985+0000) 2022-04-03T23:36:23.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:23.993+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:24.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:24.318+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:24.601 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:24.599+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:24.600206+0000) 2022-04-03T23:36:24.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:24.972+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:25.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:25.326+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:25.602 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:25.600+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:25.600434+0000) 2022-04-03T23:36:25.932 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:25.930+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:26.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:26.356+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:26.601 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:26.600+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:26.600618+0000) 2022-04-03T23:36:26.975 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:26.972+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:27.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:27.331+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:27.602 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:27.600+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:27.600855+0000) 2022-04-03T23:36:28.021 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:28.018+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:28.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:28.358+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:28.602 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:28.600+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:28.601090+0000) 2022-04-03T23:36:29.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:29.036+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:29.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:29.407+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:29.602 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:29.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:29.601310+0000) 2022-04-03T23:36:29.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:29.988+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:30.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:30.378+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:30.603 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:30.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:30.601539+0000) 2022-04-03T23:36:30.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:30.951+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:31.429 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:31.426+0000 7f88d0ac1700 -1 osd.4 132 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:31.603 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:31.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:31.601799+0000) 2022-04-03T23:36:31.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:31.971+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:32.009 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-03T23:36:32.007+0000 7ff79f425700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T23:36:32.024 INFO:tasks.ceph.mds.f.smithi186.stderr: -1> 2022-04-03T23:36:32.007+0000 7ff79f425700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T23:36:32.046 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-03T23:36:32.421 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:32.419+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:32.603 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:32.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:32.602025+0000) 2022-04-03T23:36:32.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:32.964+0000 7fd3b190b700 -1 osd.6 132 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:33.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:33.454+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:33.603 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:33.602+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:33.602218+0000) 2022-04-03T23:36:33.971 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:33.968+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:34.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:34.406+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:34.603 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:34.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:34.602426+0000) 2022-04-03T23:36:34.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:34.956+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:35.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:35.370+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:35.604 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:35.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:35.602650+0000) 2022-04-03T23:36:36.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:36.002+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:36.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:36.344+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:36.604 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:36.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:36.602881+0000) 2022-04-03T23:36:36.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:36.956+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:37.301 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:37.300+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:37.590 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:36:37.587+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:36:37.604 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:37.602+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:37.603126+0000) 2022-04-03T23:36:37.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:37.992+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:38.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:38.345+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:38.604 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:38.602+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:38.603356+0000) 2022-04-03T23:36:39.034 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:39.032+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:39.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:39.310+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:39.605 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:39.602+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:39.603590+0000) 2022-04-03T23:36:40.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:40.023+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:40.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:40.294+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:40.605 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:40.602+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:40.603718+0000) 2022-04-03T23:36:40.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:40.980+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:41.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:41.316+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 29 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:41.605 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:41.602+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:41.603922+0000) 2022-04-03T23:36:41.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:41.955+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:42.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:42.279+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:42.605 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:42.603+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:42.604146+0000) 2022-04-03T23:36:42.980 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:42.978+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:43.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:43.273+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:43.606 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:43.603+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:43.604365+0000) 2022-04-03T23:36:43.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:43.991+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:44.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:44.274+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:44.606 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:44.603+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:44.604538+0000) 2022-04-03T23:36:45.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:45.017+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:45.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:45.284+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:45.606 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:45.604+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:45.604761+0000) 2022-04-03T23:36:46.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:46.005+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:46.284 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:46.282+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:46.606 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:46.604+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:46.604925+0000) 2022-04-03T23:36:47.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:47.053+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:47.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:47.325+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:47.606 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:47.604+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:47.605091+0000) 2022-04-03T23:36:48.058 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:48.056+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:48.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:48.276+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:48.606 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:48.604+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:48.605281+0000) 2022-04-03T23:36:49.052 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:49.051+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:49.304 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:36:49.308 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:36:49.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:49.307+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:49.607 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:49.604+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:49.605521+0000) 2022-04-03T23:36:50.056 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:50.054+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:50.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:50.290+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:50.607 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:50.604+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:50.605730+0000) 2022-04-03T23:36:51.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:51.039+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:51.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:51.244+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:51.607 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:51.605+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:51.605963+0000) 2022-04-03T23:36:52.034 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:52.032+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:52.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:52.217+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:52.606 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:36:52.604+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:36:52.607 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:52.605+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:52.606123+0000) 2022-04-03T23:36:53.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:53.060+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:53.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:53.191+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:53.607 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:53.605+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:53.606349+0000) 2022-04-03T23:36:54.020 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:54.018+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:54.147 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:54.146+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:54.608 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:54.605+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:54.606579+0000) 2022-04-03T23:36:55.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:55.061+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:55.139 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:55.137+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:55.608 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:55.606+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:55.606823+0000) 2022-04-03T23:36:56.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:56.078+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:56.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:56.113+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:56.608 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:56.606+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:56.607061+0000) 2022-04-03T23:36:57.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:57.124+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:57.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:57.146+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:57.608 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:57.606+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:57.607289+0000) 2022-04-03T23:36:58.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:58.158+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:58.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:58.163+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:58.609 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:58.606+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:58.607517+0000) 2022-04-03T23:36:59.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:36:59.154+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:36:59.158 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:36:59.156+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:36:59.609 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:36:59.607+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:36:59.607683+0000) 2022-04-03T23:37:00.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:00.157+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:00.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:00.177+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:00.609 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:00.607+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:00.607857+0000) 2022-04-03T23:37:01.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:01.193+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:01.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:01.198+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:01.609 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:01.607+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:01.608047+0000) 2022-04-03T23:37:02.194 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:02.192+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:02.220 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:02.217+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:02.609 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:02.607+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:02.608248+0000) 2022-04-03T23:37:03.146 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:03.143+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:03.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:03.237+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:03.610 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:03.608+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:03.608494+0000) 2022-04-03T23:37:04.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:04.132+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:04.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:04.240+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:04.610 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:04.608+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:04.608673+0000) 2022-04-03T23:37:05.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:05.171+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:05.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:05.259+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:05.610 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:05.608+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:05.608876+0000) 2022-04-03T23:37:06.214 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:06.212+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:06.233 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:06.230+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:06.610 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:06.608+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:06.609111+0000) 2022-04-03T23:37:07.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:07.216+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:07.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:07.239+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:07.610 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:07.608+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:07.609333+0000) 2022-04-03T23:37:07.622 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:37:07.620+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:37:08.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:08.222+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:08.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:08.255+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:08.611 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:08.609+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:08.609565+0000) 2022-04-03T23:37:09.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:09.186+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:09.274 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:09.272+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:09.611 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:09.609+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:09.609818+0000) 2022-04-03T23:37:10.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:10.204+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:10.233 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:10.230+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:10.611 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:10.609+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:10.610048+0000) 2022-04-03T23:37:11.169 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:11.166+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:11.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:11.254+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:11.611 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:11.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:11.610288+0000) 2022-04-03T23:37:12.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:12.190+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:12.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:12.257+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:12.611 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:12.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:12.610481+0000) 2022-04-03T23:37:13.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:13.217+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:13.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:13.226+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:13.612 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:13.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:13.610665+0000) 2022-04-03T23:37:14.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:14.198+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:14.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:14.212+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:14.612 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:14.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:14.610893+0000) 2022-04-03T23:37:15.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:15.160+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:15.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:15.246+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:15.612 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:15.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:15.611123+0000) 2022-04-03T23:37:16.209 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:16.206+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:16.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:16.222+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:16.613 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:16.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:16.611387+0000) 2022-04-03T23:37:17.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:17.164+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:17.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:17.184+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:17.613 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:17.611+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:17.611551+0000) 2022-04-03T23:37:18.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:18.123+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:18.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:18.225+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:18.613 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:18.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:18.611734+0000) 2022-04-03T23:37:19.101 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:19.099+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:19.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:19.241+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:19.372 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:37:19.375 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:37:19.613 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:19.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:19.611934+0000) 2022-04-03T23:37:20.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:20.079+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:20.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:20.227+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:20.613 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:20.611+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:20.612204+0000) 2022-04-03T23:37:21.118 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:21.115+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:21.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:21.184+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:21.613 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:21.611+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:21.612454+0000) 2022-04-03T23:37:22.160 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:22.158+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:22.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:22.186+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:22.614 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:22.611+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:22.612617+0000) 2022-04-03T23:37:22.630 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:37:22.627+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:37:23.133 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:23.130+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:23.142 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:23.139+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:23.614 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:23.611+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:23.612850+0000) 2022-04-03T23:37:24.089 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:24.086+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:24.131 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:24.128+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:24.614 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:24.612+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:24.613088+0000) 2022-04-03T23:37:25.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:25.114+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:25.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:25.176+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:25.614 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:25.612+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:25.613275+0000) 2022-04-03T23:37:26.150 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:26.148+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:26.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:26.172+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:26.615 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:26.612+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:26.613434+0000) 2022-04-03T23:37:27.186 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:27.183+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:27.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:27.222+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:27.615 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:27.612+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:27.613591+0000) 2022-04-03T23:37:28.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:28.160+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:28.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:28.257+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:28.615 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:28.612+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:28.613763+0000) 2022-04-03T23:37:29.199 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:29.197+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:29.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:29.259+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:29.615 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:29.613+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:29.614000+0000) 2022-04-03T23:37:30.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:30.217+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:30.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:30.263+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:30.615 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:30.613+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:30.614229+0000) 2022-04-03T23:37:31.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:31.228+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:31.284 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:31.282+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:31.616 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:31.613+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:31.614470+0000) 2022-04-03T23:37:32.228 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:32.226+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:32.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:32.321+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:32.616 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:32.613+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:32.614729+0000) 2022-04-03T23:37:33.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:33.213+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:33.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:33.331+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:33.616 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:33.614+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:33.615015+0000) 2022-04-03T23:37:34.208 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:34.206+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:34.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:34.311+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:34.616 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:34.614+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:34.615166+0000) 2022-04-03T23:37:35.168 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:35.166+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:35.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:35.280+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:35.616 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:35.614+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:35.615401+0000) 2022-04-03T23:37:35.676 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-03T23:37:35.674+0000 7f2fd6e1b700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T23:37:35.677 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-03T23:37:35.674+0000 7f2fd761c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T23:37:36.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:36.133+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:36.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:36.317+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:36.617 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:36.614+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:36.615612+0000) 2022-04-03T23:37:37.144 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:37.137+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:37.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:37.342+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:37.617 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:37.615+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:37.615866+0000) 2022-04-03T23:37:37.646 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:37:37.643+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:37:38.151 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:38.149+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:38.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:38.311+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:38.617 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:38.615+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:38.616107+0000) 2022-04-03T23:37:39.102 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:39.100+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:39.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:39.312+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:39.617 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:39.615+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:39.616301+0000) 2022-04-03T23:37:40.116 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:40.114+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:40.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:40.327+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:40.618 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:40.615+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:40.616512+0000) 2022-04-03T23:37:41.142 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:41.140+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:41.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:41.340+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:41.618 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:41.616+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:41.616761+0000) 2022-04-03T23:37:42.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:42.110+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:42.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:42.328+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:42.618 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:42.616+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:42.616950+0000) 2022-04-03T23:37:43.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:43.151+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:43.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:43.286+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:43.618 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:43.616+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:43.617168+0000) 2022-04-03T23:37:44.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:44.134+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:44.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:44.275+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:44.618 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:44.616+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:44.617403+0000) 2022-04-03T23:37:45.148 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:45.146+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:45.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:45.225+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:45.619 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:45.617+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:45.617571+0000) 2022-04-03T23:37:46.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:46.106+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:46.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:46.178+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:46.619 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:46.617+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:46.617759+0000) 2022-04-03T23:37:47.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:47.145+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:47.214 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:47.212+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:47.619 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:47.617+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:47.618004+0000) 2022-04-03T23:37:48.150 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:48.148+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:48.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:48.186+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:48.619 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:48.617+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:48.618210+0000) 2022-04-03T23:37:49.160 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:49.158+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:49.226 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:49.225+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:49.516 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:37:49.519 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:37:49.620 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:49.618+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:49.618491+0000) 2022-04-03T23:37:50.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:50.132+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:50.217 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:50.215+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:50.620 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:50.618+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:50.618731+0000) 2022-04-03T23:37:51.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:51.122+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:51.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:51.244+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:51.620 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:51.618+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:51.619008+0000) 2022-04-03T23:37:52.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:52.128+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:52.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:52.200+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:52.620 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:52.618+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:52.619200+0000) 2022-04-03T23:37:52.661 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:37:52.659+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:37:53.120 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:53.117+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:53.252 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:53.248+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:53.621 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:53.619+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:53.619392+0000) 2022-04-03T23:37:54.115 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:54.112+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:54.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:54.273+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:54.621 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:54.619+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:54.619581+0000) 2022-04-03T23:37:55.096 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:55.094+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:55.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:55.316+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:55.621 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:55.619+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:55.619732+0000) 2022-04-03T23:37:56.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:56.051+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:56.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:56.297+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:56.621 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:56.619+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:56.619858+0000) 2022-04-03T23:37:57.096 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:57.093+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:57.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:57.275+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:57.621 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:57.619+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:57.620076+0000) 2022-04-03T23:37:58.072 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:58.069+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:58.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:58.320+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:58.621 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:58.620+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:58.620320+0000) 2022-04-03T23:37:59.072 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:37:59.069+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:37:59.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:37:59.336+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:37:59.622 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:37:59.620+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:37:59.620568+0000) 2022-04-03T23:38:00.022 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:00.019+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:00.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:00.343+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:00.622 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:00.620+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:00.620840+0000) 2022-04-03T23:38:01.045 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:01.042+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:01.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:01.317+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:01.622 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:01.620+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:01.621082+0000) 2022-04-03T23:38:02.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:02.035+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:02.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:02.273+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:02.622 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:02.621+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:02.621277+0000) 2022-04-03T23:38:02.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:02.986+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:03.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:03.269+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:03.622 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:03.621+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:03.621419+0000) 2022-04-03T23:38:04.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:04.036+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:04.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:04.304+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:04.623 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:04.620+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:04.621588+0000) 2022-04-03T23:38:05.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:05.082+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:05.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:05.260+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:05.623 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:05.620+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:05.621821+0000) 2022-04-03T23:38:06.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:06.073+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:06.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:06.248+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:06.623 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:06.620+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:06.622064+0000) 2022-04-03T23:38:07.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:07.060+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:07.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:07.276+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:07.623 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:07.621+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:07.622312+0000) 2022-04-03T23:38:07.666 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:38:07.663+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:38:08.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:08.016+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:08.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:08.326+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:08.623 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:08.621+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:08.622528+0000) 2022-04-03T23:38:08.976 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:08.974+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:09.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:09.320+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:09.624 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:09.621+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:09.622753+0000) 2022-04-03T23:38:09.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:09.937+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:10.283 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:10.281+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:10.624 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:10.622+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:10.623009+0000) 2022-04-03T23:38:10.963 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:10.961+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:11.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:11.301+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:11.624 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:11.622+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:11.623277+0000) 2022-04-03T23:38:11.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:11.976+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:12.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:12.316+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:12.625 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:12.622+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:12.623502+0000) 2022-04-03T23:38:12.994 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:12.991+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:13.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:13.308+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:13.625 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:13.622+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:13.623687+0000) 2022-04-03T23:38:14.022 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:14.019+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:14.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:14.318+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:14.625 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:14.623+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:14.623932+0000) 2022-04-03T23:38:15.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:15.044+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:15.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:15.271+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:15.625 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:15.623+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:15.624148+0000) 2022-04-03T23:38:16.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:16.008+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:16.235 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:16.233+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:16.625 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:16.623+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:16.624328+0000) 2022-04-03T23:38:16.972 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:16.970+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:17.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:17.265+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:17.626 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:17.623+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:17.624547+0000) 2022-04-03T23:38:17.941 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:17.939+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:18.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:18.231+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:18.626 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:18.624+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:18.624826+0000) 2022-04-03T23:38:18.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:18.921+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:19.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:19.255+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:19.626 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:19.624+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:19.625069+0000) 2022-04-03T23:38:19.645 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:38:19.649 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:38:19.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:19.885+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:20.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:20.229+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:20.626 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:20.624+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:20.625332+0000) 2022-04-03T23:38:20.627 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:20.625+0000 7fd62421b700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T23:38:20.628 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:20.625+0000 7fd623a1a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T23:38:20.639 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:20.637+0000 7fd623a1a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:20.638407+0000) 2022-04-03T23:38:20.910 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:20.907+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:21.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:21.253+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:21.627 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:21.624+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:21.625543+0000) 2022-04-03T23:38:21.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:21.939+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:22.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:22.206+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:22.627 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:22.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:22.625696+0000) 2022-04-03T23:38:22.682 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:38:22.680+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:38:22.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:22.987+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:23.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:23.185+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:23.627 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:23.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:23.625880+0000) 2022-04-03T23:38:23.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:23.954+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:24.182 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:24.180+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:24.627 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:24.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:24.626060+0000) 2022-04-03T23:38:24.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:24.975+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:25.202 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:25.200+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:25.627 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:25.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:25.626216+0000) 2022-04-03T23:38:25.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:25.987+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:26.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:26.160+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:26.627 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:26.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:26.626396+0000) 2022-04-03T23:38:27.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:27.012+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:27.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:27.164+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:27.628 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:27.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:27.626572+0000) 2022-04-03T23:38:27.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:27.990+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:28.145 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:28.143+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:28.628 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:28.626+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:28.626735+0000) 2022-04-03T23:38:28.980 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:28.978+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:29.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:29.162+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:29.628 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:29.626+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:29.626934+0000) 2022-04-03T23:38:29.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:29.964+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:30.202 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:30.200+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:30.628 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:30.626+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:30.627103+0000) 2022-04-03T23:38:30.944 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:30.942+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:31.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:31.220+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:31.628 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:31.626+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:31.627283+0000) 2022-04-03T23:38:31.908 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:31.907+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:32.191 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:32.189+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:32.629 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:32.627+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:32.627433+0000) 2022-04-03T23:38:32.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:32.909+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:33.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:33.209+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:33.629 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:33.627+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:33.627584+0000) 2022-04-03T23:38:33.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:33.953+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:34.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:34.257+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:34.629 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:34.627+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:34.627734+0000) 2022-04-03T23:38:34.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:34.988+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:35.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:35.228+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:35.629 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:35.627+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:35.627899+0000) 2022-04-03T23:38:35.961 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:35.959+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:36.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:36.208+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:36.629 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:36.627+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:36.628051+0000) 2022-04-03T23:38:36.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:36.964+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:37.172 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:37.171+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:37.629 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:37.627+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:37.628238+0000) 2022-04-03T23:38:37.698 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:38:37.695+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:38:37.996 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:37.994+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:38.138 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:38.136+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:38.630 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:38.628+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:38.628445+0000) 2022-04-03T23:38:39.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:39.014+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:39.136 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:39.134+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:39.630 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:39.627+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:39.628614+0000) 2022-04-03T23:38:40.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:40.006+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:40.170 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:40.168+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:40.630 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:40.628+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:40.628824+0000) 2022-04-03T23:38:41.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:41.013+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:41.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:41.176+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:41.630 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:41.628+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:41.629016+0000) 2022-04-03T23:38:42.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:42.060+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:42.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:42.226+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:42.630 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:42.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:42.629201+0000) 2022-04-03T23:38:43.093 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:43.091+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:43.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:43.248+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:43.631 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:43.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:43.629384+0000) 2022-04-03T23:38:44.058 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:44.056+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:44.283 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:44.281+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:44.631 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:44.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:44.629585+0000) 2022-04-03T23:38:45.029 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:45.027+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:45.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:45.276+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:45.631 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:45.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:45.629808+0000) 2022-04-03T23:38:46.034 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:46.032+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:46.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:46.235+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:46.631 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:46.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:46.629998+0000) 2022-04-03T23:38:47.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:47.022+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:47.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:47.198+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:47.631 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:47.630+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:47.630182+0000) 2022-04-03T23:38:48.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:48.032+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:48.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:48.186+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:48.631 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:48.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:48.630391+0000) 2022-04-03T23:38:49.030 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:49.027+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:49.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:49.181+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:49.632 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:49.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:49.630566+0000) 2022-04-03T23:38:49.776 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:38:49.780 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:38:50.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:50.012+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:50.154 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:50.151+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:50.632 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:50.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:50.630725+0000) 2022-04-03T23:38:51.042 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:51.039+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:51.170 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:51.167+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:51.632 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:51.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:51.630956+0000) 2022-04-03T23:38:52.029 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:52.027+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:52.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:52.170+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:52.632 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:52.630+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:52.631134+0000) 2022-04-03T23:38:52.714 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:38:52.710+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:38:53.021 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:53.018+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:53.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:53.146+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:53.633 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:53.630+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:53.631288+0000) 2022-04-03T23:38:54.017 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:54.015+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:54.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:54.175+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:54.633 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:54.630+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:54.631448+0000) 2022-04-03T23:38:55.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:55.046+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:55.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:55.215+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:55.633 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:55.630+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:55.631603+0000) 2022-04-03T23:38:56.084 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:56.081+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:56.174 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:56.171+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:56.633 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:56.630+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:56.631804+0000) 2022-04-03T23:38:57.118 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:57.115+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:57.160 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:57.157+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:57.633 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:57.630+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:57.632004+0000) 2022-04-03T23:38:58.119 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:58.116+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:58.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:58.138+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:58.633 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:58.631+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:58.632167+0000) 2022-04-03T23:38:59.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:38:59.162+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:38:59.169 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:38:59.166+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:38:59.634 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:38:59.631+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:38:59.632321+0000) 2022-04-03T23:39:00.192 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:00.190+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:00.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:00.202+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:00.634 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:00.631+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:00.632484+0000) 2022-04-03T23:39:01.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:01.153+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:01.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:01.157+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:01.634 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:01.631+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:01.632665+0000) 2022-04-03T23:39:02.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:02.121+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:02.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:02.186+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:02.634 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:02.632+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:02.632834+0000) 2022-04-03T23:39:03.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:03.078+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:03.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:03.208+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:03.634 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:03.632+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:03.632987+0000) 2022-04-03T23:39:04.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:04.104+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:04.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:04.187+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:04.634 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:04.632+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:04.633137+0000) 2022-04-03T23:39:05.065 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:05.062+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:05.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:05.147+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:05.634 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:05.632+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:05.633265+0000) 2022-04-03T23:39:06.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:06.031+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:06.119 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:06.116+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:06.634 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:06.632+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:06.633390+0000) 2022-04-03T23:39:07.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:07.070+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:07.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:07.070+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:07.635 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:07.632+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:07.633550+0000) 2022-04-03T23:39:07.730 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:39:07.727+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:39:08.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:08.026+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:08.088 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:08.085+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:08.635 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:08.633+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:08.633715+0000) 2022-04-03T23:39:09.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:09.064+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:09.133 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:09.131+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:09.635 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:09.633+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:09.633883+0000) 2022-04-03T23:39:10.102 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:10.100+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:10.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:10.150+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:10.636 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:10.633+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:10.634052+0000) 2022-04-03T23:39:11.073 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:11.071+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:11.170 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:11.167+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:11.635 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:11.633+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:11.634225+0000) 2022-04-03T23:39:12.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:12.065+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:12.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:12.213+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:12.635 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:12.633+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:12.634370+0000) 2022-04-03T23:39:13.058 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:13.056+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:13.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:13.258+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:13.636 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:13.634+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:13.634526+0000) 2022-04-03T23:39:14.093 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:14.090+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:14.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:14.222+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:14.636 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:14.634+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:14.634697+0000) 2022-04-03T23:39:15.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:15.121+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:15.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:15.222+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:15.636 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:15.634+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:15.634863+0000) 2022-04-03T23:39:16.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:16.131+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:16.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:16.258+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:16.636 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:16.634+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:16.635049+0000) 2022-04-03T23:39:17.099 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:17.096+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:17.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:17.265+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:17.636 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:17.634+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:17.635210+0000) 2022-04-03T23:39:18.058 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:18.055+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:18.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:18.293+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:18.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:18.634+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:18.635366+0000) 2022-04-03T23:39:19.057 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:19.054+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:19.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:19.283+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:19.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:19.635+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:19.635497+0000) 2022-04-03T23:39:19.854 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:39:19.858 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:39:20.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:20.013+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:20.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:20.332+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:20.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:20.635+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:20.635657+0000) 2022-04-03T23:39:21.056 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:21.054+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:21.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:21.298+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:21.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:21.635+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:21.635867+0000) 2022-04-03T23:39:22.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:22.074+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:22.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:22.325+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:22.638 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:22.635+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:22.635990+0000) 2022-04-03T23:39:22.745 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:39:22.743+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:39:23.089 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:23.087+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:23.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:23.277+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:23.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:23.635+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:23.636151+0000) 2022-04-03T23:39:24.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:24.100+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:24.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:24.242+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:24.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:24.636+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:24.636294+0000) 2022-04-03T23:39:25.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:25.119+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:25.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:25.253+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:25.638 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:25.636+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:25.636454+0000) 2022-04-03T23:39:26.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:26.163+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:26.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:26.251+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:26.638 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:26.636+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:26.636616+0000) 2022-04-03T23:39:26.655 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T23:39:26.653+0000 7f2e44320700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T23:39:26.656 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-03T23:39:26.653+0000 7f2e4331e700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T23:39:27.140 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:27.138+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:27.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:27.221+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:27.638 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:27.636+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:27.636786+0000) 2022-04-03T23:39:28.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:28.180+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:28.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:28.256+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:28.638 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:28.636+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:28.636947+0000) 2022-04-03T23:39:29.184 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:29.182+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:29.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:29.258+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:29.638 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:29.636+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:29.637106+0000) 2022-04-03T23:39:30.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:30.145+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:30.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:30.281+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:30.638 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:30.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:30.637272+0000) 2022-04-03T23:39:31.115 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:31.114+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:31.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:31.278+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:31.639 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:31.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:31.637443+0000) 2022-04-03T23:39:32.113 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:32.111+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:32.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:32.242+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:32.639 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:32.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:32.637603+0000) 2022-04-03T23:39:33.148 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:33.146+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:33.283 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:33.281+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:33.639 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:33.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:33.637733+0000) 2022-04-03T23:39:34.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:34.120+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:34.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:34.297+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:34.639 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:34.636+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:34.637931+0000) 2022-04-03T23:39:35.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:35.162+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:35.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:35.329+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:35.639 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:35.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:35.638116+0000) 2022-04-03T23:39:36.212 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:36.211+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:36.283 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:36.281+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 34 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:36.639 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:36.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:36.638284+0000) 2022-04-03T23:39:37.205 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:37.204+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:37.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:37.303+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:37.640 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:37.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:37.638446+0000) 2022-04-03T23:39:37.761 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:39:37.758+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:39:38.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:38.195+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:38.337 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:38.335+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:38.640 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:38.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:38.638578+0000) 2022-04-03T23:39:39.220 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:39.218+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:39.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:39.306+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:39.640 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:39.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:39.638739+0000) 2022-04-03T23:39:40.205 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:40.204+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:40.284 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:40.282+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:40.640 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:40.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:40.638893+0000) 2022-04-03T23:39:41.247 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:41.245+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:41.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:41.257+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:41.640 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:41.638+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:41.639076+0000) 2022-04-03T23:39:42.214 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:42.211+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:42.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:42.273+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:42.640 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:42.638+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:42.639216+0000) 2022-04-03T23:39:43.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:43.182+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:43.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:43.262+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:43.640 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:43.638+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:43.639350+0000) 2022-04-03T23:39:44.149 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:44.146+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:44.238 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:44.235+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:44.641 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:44.638+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:44.639527+0000) 2022-04-03T23:39:45.144 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:45.141+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:45.275 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:45.273+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:45.641 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:45.638+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:45.639698+0000) 2022-04-03T23:39:46.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:46.162+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:46.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:46.307+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:46.641 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:46.639+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:46.639861+0000) 2022-04-03T23:39:47.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:47.157+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:47.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:47.308+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:47.641 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:47.639+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:47.640026+0000) 2022-04-03T23:39:48.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:48.189+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:48.342 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:48.340+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:48.642 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:48.639+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:48.640200+0000) 2022-04-03T23:39:49.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:49.226+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:49.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:49.338+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:49.642 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:49.639+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:49.640396+0000) 2022-04-03T23:39:49.930 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:39:49.934 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:39:50.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:50.193+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:50.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:50.332+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:50.642 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:50.639+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:50.640569+0000) 2022-04-03T23:39:51.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:51.147+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:51.293 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:51.290+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:51.642 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:51.640+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:51.640755+0000) 2022-04-03T23:39:52.140 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:52.137+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:52.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:52.314+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:52.642 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:52.640+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:52.640940+0000) 2022-04-03T23:39:52.777 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:39:52.774+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:39:53.172 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:53.170+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:53.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:53.318+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:53.642 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:53.640+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:53.641104+0000) 2022-04-03T23:39:54.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:54.194+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:54.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:54.312+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:54.642 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:54.640+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:54.641277+0000) 2022-04-03T23:39:55.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:55.237+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:55.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:55.275+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:55.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:55.640+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:55.641425+0000) 2022-04-03T23:39:56.228 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:56.226+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:56.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:56.237+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:56.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:56.641+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:56.641596+0000) 2022-04-03T23:39:57.217 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:57.215+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:57.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:57.251+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:57.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:57.641+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:57.641763+0000) 2022-04-03T23:39:58.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:58.193+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:58.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:58.260+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:58.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:58.641+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:58.641893+0000) 2022-04-03T23:39:59.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:39:59.220+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:39:59.237 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:39:59.235+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:39:59.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:39:59.641+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:39:59.642045+0000) 2022-04-03T23:40:00.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:00.175+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:00.210 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:00.207+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:00.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:00.641+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:00.642210+0000) 2022-04-03T23:40:01.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:01.167+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:01.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:01.183+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:01.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:01.641+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:01.642363+0000) 2022-04-03T23:40:02.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:02.145+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:02.160 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:02.157+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:02.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:02.642+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:02.642511+0000) 2022-04-03T23:40:03.160 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:03.157+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:03.171 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:03.168+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:03.644 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:03.642+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:03.642662+0000) 2022-04-03T23:40:04.138 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:04.135+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:04.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:04.217+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:04.644 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:04.642+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:04.642864+0000) 2022-04-03T23:40:05.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:05.126+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:05.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:05.256+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:05.644 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:05.642+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:05.643055+0000) 2022-04-03T23:40:06.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:06.123+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:06.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:06.264+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:06.644 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:06.642+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:06.643217+0000) 2022-04-03T23:40:07.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:07.079+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:07.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:07.220+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:07.645 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:07.643+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:07.643366+0000) 2022-04-03T23:40:07.793 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:40:07.790+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:40:08.077 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:08.075+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:08.214 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:08.212+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:08.645 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:08.642+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:08.643564+0000) 2022-04-03T23:40:09.118 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:09.115+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:09.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:09.259+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:09.645 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:09.643+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:09.643738+0000) 2022-04-03T23:40:10.166 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:10.163+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:10.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:10.225+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:10.645 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:10.643+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:10.643878+0000) 2022-04-03T23:40:11.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:11.140+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:11.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:11.276+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:11.645 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:11.643+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:11.644074+0000) 2022-04-03T23:40:12.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:12.126+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:12.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:12.257+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:12.645 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:12.644+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:12.644236+0000) 2022-04-03T23:40:13.091 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:13.089+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:13.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:13.267+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:13.646 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:13.644+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:13.644376+0000) 2022-04-03T23:40:14.104 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:14.101+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:14.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:14.289+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:14.646 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:14.643+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:14.644586+0000) 2022-04-03T23:40:15.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:15.132+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:15.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:15.291+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:15.646 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:15.644+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:15.644777+0000) 2022-04-03T23:40:16.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:16.172+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:16.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:16.303+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:16.646 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:16.644+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:16.644911+0000) 2022-04-03T23:40:17.213 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:17.211+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:17.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:17.348+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:17.646 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:17.644+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:17.645059+0000) 2022-04-03T23:40:18.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:18.172+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:18.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:18.376+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:18.646 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:18.645+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:18.645238+0000) 2022-04-03T23:40:19.194 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:19.192+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:19.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:19.407+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:19.647 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:19.644+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:19.645389+0000) 2022-04-03T23:40:20.016 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:40:20.020 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:40:20.223 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:20.221+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:20.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:20.412+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:20.647 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:20.644+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:20.645552+0000) 2022-04-03T23:40:21.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:21.181+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:21.429 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:21.426+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:21.647 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:21.644+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:21.645747+0000) 2022-04-03T23:40:22.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:22.205+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:22.409 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:22.408+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:22.647 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:22.644+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:22.645941+0000) 2022-04-03T23:40:22.796 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:40:22.793+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:40:23.221 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:23.219+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:23.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:23.410+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:23.647 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:23.645+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:23.646126+0000) 2022-04-03T23:40:24.267 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:24.265+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:24.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:24.370+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:24.647 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:24.645+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:24.646306+0000) 2022-04-03T23:40:25.287 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:25.285+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:25.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:25.321+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:25.648 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:25.645+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:25.646472+0000) 2022-04-03T23:40:26.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:26.295+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:26.313 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:26.311+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:26.648 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:26.645+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:26.646638+0000) 2022-04-03T23:40:27.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:27.270+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:27.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:27.285+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:27.648 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:27.645+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:27.646799+0000) 2022-04-03T23:40:28.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:28.279+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:28.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:28.290+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:28.648 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:28.646+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:28.646995+0000) 2022-04-03T23:40:29.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:29.327+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:29.334 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:29.332+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:29.648 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:29.646+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:29.647200+0000) 2022-04-03T23:40:30.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:30.290+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:30.356 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:30.354+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:30.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:30.646+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:30.647359+0000) 2022-04-03T23:40:31.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:31.303+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:31.359 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:31.357+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:31.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:31.646+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:31.647547+0000) 2022-04-03T23:40:32.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:32.274+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:32.312 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:32.310+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:32.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:32.646+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:32.647707+0000) 2022-04-03T23:40:33.252 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:33.250+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:33.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:33.294+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:33.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:33.647+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:33.647821+0000) 2022-04-03T23:40:34.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:34.251+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:34.331 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:34.329+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:34.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:34.647+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:34.647940+0000) 2022-04-03T23:40:35.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:35.223+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:35.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:35.350+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:35.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:35.647+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:35.648116+0000) 2022-04-03T23:40:36.221 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:36.219+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:36.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:36.376+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:36.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:36.647+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:36.648234+0000) 2022-04-03T23:40:37.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:37.197+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:37.400 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:37.397+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:37.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:37.647+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:37.648352+0000) 2022-04-03T23:40:37.812 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:40:37.810+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:40:38.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:38.188+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:38.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:38.422+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:38.650 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:38.647+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:38.648517+0000) 2022-04-03T23:40:39.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:39.182+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:39.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:39.381+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:39.650 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:39.648+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:39.648703+0000) 2022-04-03T23:40:40.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:40.197+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:40.408 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:40.406+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:40.650 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:40.648+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:40.648827+0000) 2022-04-03T23:40:41.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:41.200+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:41.449 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:41.447+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:41.650 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:41.648+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:41.648990+0000) 2022-04-03T23:40:42.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:42.193+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:42.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:42.428+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:42.650 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:42.648+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:42.649163+0000) 2022-04-03T23:40:43.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:43.147+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:43.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:43.453+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:43.650 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:43.648+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:43.649320+0000) 2022-04-03T23:40:44.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:44.176+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:44.471 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:44.468+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:44.651 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:44.648+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:44.649468+0000) 2022-04-03T23:40:45.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:45.188+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:45.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:45.422+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:45.651 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:45.649+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:45.649648+0000) 2022-04-03T23:40:46.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:46.201+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:46.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:46.402+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:46.651 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:46.649+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:46.649833+0000) 2022-04-03T23:40:47.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:47.206+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:47.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:47.420+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:47.651 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:47.649+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:47.650030+0000) 2022-04-03T23:40:48.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:48.174+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:48.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:48.420+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:48.651 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:48.649+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:48.650206+0000) 2022-04-03T23:40:49.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:49.152+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:49.395 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:49.392+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:49.651 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:49.649+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:49.650271+0000) 2022-04-03T23:40:50.158 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:40:50.162 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:40:50.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:50.197+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:50.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:50.436+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:50.651 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:50.649+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:50.650452+0000) 2022-04-03T23:40:51.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:51.195+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:51.434 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:51.431+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:51.652 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:51.650+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:51.650682+0000) 2022-04-03T23:40:52.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:52.226+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:52.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:52.422+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:52.652 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:52.650+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:52.650854+0000) 2022-04-03T23:40:52.828 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:40:52.826+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:40:53.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:53.259+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:53.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:53.428+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:53.652 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:53.650+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:53.650975+0000) 2022-04-03T23:40:54.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:54.303+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:54.395 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:54.392+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:54.652 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:54.650+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:54.651150+0000) 2022-04-03T23:40:55.301 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:55.298+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:55.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:55.397+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:55.652 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:55.651+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:55.651326+0000) 2022-04-03T23:40:56.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:56.300+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:56.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:56.380+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:56.653 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:56.651+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:56.651500+0000) 2022-04-03T23:40:56.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:56.953+0000 7fd3baf7d700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-03T23:40:57.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:57.334+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:57.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:57.413+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:57.653 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:57.651+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:57.651656+0000) 2022-04-03T23:40:58.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:58.287+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:58.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:58.396+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:58.653 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:58.651+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:58.651833+0000) 2022-04-03T23:40:59.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:40:59.318+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:40:59.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:40:59.416+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:40:59.653 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:40:59.651+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:40:59.651990+0000) 2022-04-03T23:41:00.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:00.315+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:00.402 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:00.399+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:00.653 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:00.652+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:00.652144+0000) 2022-04-03T23:41:01.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:01.340+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:01.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:01.380+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:01.653 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:01.652+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:01.652301+0000) 2022-04-03T23:41:02.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:02.313+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:02.356 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:02.354+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:02.653 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:02.651+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:02.652468+0000) 2022-04-03T23:41:03.364 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:03.361+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:03.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:03.403+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:03.654 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:03.652+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:03.652588+0000) 2022-04-03T23:41:04.366 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:04.364+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:04.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:04.396+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:04.654 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:04.651+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:04.652750+0000) 2022-04-03T23:41:05.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:05.332+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:05.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:05.369+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:05.654 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:05.651+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:05.652932+0000) 2022-04-03T23:41:06.359 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:06.357+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:06.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:06.412+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:06.654 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:06.652+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:06.653077+0000) 2022-04-03T23:41:07.334 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:07.331+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:07.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:07.384+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:07.654 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:07.652+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:07.653242+0000) 2022-04-03T23:41:07.833 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:41:07.830+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:41:08.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:08.354+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:08.388 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:08.386+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:08.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:08.651+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:08.653353+0000) 2022-04-03T23:41:09.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:09.356+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:09.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:09.361+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:09.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:09.652+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:09.653525+0000) 2022-04-03T23:41:10.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:10.313+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:10.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:10.318+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:10.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:10.652+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:10.653632+0000) 2022-04-03T23:41:11.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:11.299+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:11.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:11.333+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:11.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:11.652+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:11.653819+0000) 2022-04-03T23:41:12.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:12.332+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:12.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:12.343+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:12.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:12.653+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:12.653960+0000) 2022-04-03T23:41:13.376 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:13.373+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:13.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:13.379+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:13.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:13.652+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:13.654083+0000) 2022-04-03T23:41:14.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:14.338+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:14.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:14.381+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:14.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:14.653+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:14.654158+0000) 2022-04-03T23:41:15.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:15.328+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:15.376 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:15.374+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:15.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:15.653+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:15.654307+0000) 2022-04-03T23:41:16.347 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:16.345+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:16.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:16.367+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:16.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:16.653+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:16.654487+0000) 2022-04-03T23:41:17.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:17.334+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:17.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:17.388+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:17.656 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:17.653+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:17.654586+0000) 2022-04-03T23:41:18.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:18.359+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:18.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:18.383+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:18.656 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:18.654+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:18.654735+0000) 2022-04-03T23:41:19.344 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:19.342+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:19.434 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:19.432+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:19.656 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:19.654+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:19.654913+0000) 2022-04-03T23:41:20.229 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:41:20.233 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:41:20.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:20.376+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:20.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:20.395+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:20.656 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:20.654+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:20.655088+0000) 2022-04-03T23:41:21.350 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:21.348+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:21.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:21.445+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:21.656 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:21.654+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:21.655274+0000) 2022-04-03T23:41:22.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:22.383+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:22.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:22.402+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:22.657 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:22.654+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:22.655480+0000) 2022-04-03T23:41:22.849 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:41:22.847+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:41:23.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:23.361+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:23.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:23.409+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:23.657 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:23.655+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:23.655650+0000) 2022-04-03T23:41:24.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:24.339+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:24.392 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:24.390+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:24.657 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:24.655+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:24.655829+0000) 2022-04-03T23:41:25.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:25.320+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:25.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:25.419+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:25.657 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:25.655+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:25.656006+0000) 2022-04-03T23:41:26.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:26.311+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:26.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:26.445+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:26.657 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:26.655+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:26.656141+0000) 2022-04-03T23:41:27.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:27.323+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:27.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:27.404+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:27.657 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:27.655+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:27.656316+0000) 2022-04-03T23:41:28.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:28.331+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:28.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:28.419+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:28.657 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:28.655+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:28.656490+0000) 2022-04-03T23:41:29.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:29.345+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:29.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:29.449+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:29.658 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:29.656+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:29.656629+0000) 2022-04-03T23:41:30.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:30.390+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:30.412 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:30.410+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:30.658 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:30.656+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:30.656858+0000) 2022-04-03T23:41:31.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:31.370+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:31.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:31.375+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:31.658 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:31.656+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:31.657101+0000) 2022-04-03T23:41:32.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:32.368+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:32.403 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:32.401+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:32.658 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:32.656+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:32.657268+0000) 2022-04-03T23:41:33.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:33.394+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:33.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:33.415+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:33.659 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:33.656+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:33.657465+0000) 2022-04-03T23:41:34.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:34.381+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:34.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:34.387+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:34.659 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:34.657+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:34.657637+0000) 2022-04-03T23:41:35.345 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:35.342+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:35.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:35.387+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:35.659 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:35.657+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:35.657865+0000) 2022-04-03T23:41:36.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:36.382+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:36.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:36.427+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:36.659 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:36.657+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:36.658044+0000) 2022-04-03T23:41:37.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:37.398+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:37.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:37.444+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:37.659 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:37.657+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:37.658167+0000) 2022-04-03T23:41:37.865 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:41:37.862+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:41:38.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:38.436+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:38.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:38.441+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:38.659 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:38.657+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:38.658290+0000) 2022-04-03T23:41:39.400 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:39.397+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:39.426 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:39.424+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:39.659 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:39.658+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:39.658467+0000) 2022-04-03T23:41:40.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:40.356+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:40.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:40.398+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:40.660 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:40.658+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:40.658631+0000) 2022-04-03T23:41:41.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:41.371+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:41.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:41.411+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:41.660 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:41.658+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:41.658866+0000) 2022-04-03T23:41:42.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:42.395+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:42.428 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:42.426+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:42.660 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:42.658+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:42.658980+0000) 2022-04-03T23:41:43.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:43.352+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:43.408 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:43.406+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:43.660 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:43.658+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:43.659158+0000) 2022-04-03T23:41:44.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:44.369+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:44.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:44.424+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:44.660 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:44.658+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:44.659267+0000) 2022-04-03T23:41:45.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:45.386+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:45.400 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:45.398+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:45.660 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:45.659+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:45.659444+0000) 2022-04-03T23:41:46.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:46.430+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:46.434 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:46.431+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:46.661 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:46.658+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:46.659542+0000) 2022-04-03T23:41:47.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:47.392+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:47.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:47.466+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:47.661 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:47.659+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:47.659749+0000) 2022-04-03T23:41:48.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:48.385+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:48.434 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:48.431+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:48.661 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:48.659+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:48.659958+0000) 2022-04-03T23:41:49.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:49.424+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:49.432 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:49.430+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:49.661 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:49.659+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:49.660138+0000) 2022-04-03T23:41:50.360 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:41:50.364 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:41:50.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:50.389+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:50.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:50.412+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:50.661 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:50.658+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:50.660394+0000) 2022-04-03T23:41:51.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:51.394+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:51.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:51.453+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:51.662 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:51.658+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:51.660602+0000) 2022-04-03T23:41:52.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:52.417+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:52.469 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:52.467+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:52.662 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:52.659+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:52.660816+0000) 2022-04-03T23:41:52.868 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:41:52.865+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:41:53.422 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:53.419+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:53.478 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:53.475+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:53.662 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:53.659+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:53.660987+0000) 2022-04-03T23:41:54.428 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:54.425+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:54.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:54.492+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:54.662 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:54.660+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:54.661174+0000) 2022-04-03T23:41:55.438 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:55.435+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:55.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:55.539+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:55.662 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:55.660+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:55.661378+0000) 2022-04-03T23:41:56.407 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:56.405+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:56.554 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:56.551+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:56.663 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:56.660+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:56.661569+0000) 2022-04-03T23:41:57.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:57.405+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:57.578 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:57.576+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:57.663 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:57.660+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:57.661725+0000) 2022-04-03T23:41:58.415 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:58.413+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:58.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:58.533+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:58.663 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:58.661+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:58.661919+0000) 2022-04-03T23:41:59.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:41:59.390+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:41:59.525 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:41:59.522+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:41:59.663 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:41:59.661+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:41:59.662162+0000) 2022-04-03T23:42:00.442 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:00.439+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:00.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:00.546+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:00.663 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:00.661+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:00.662346+0000) 2022-04-03T23:42:01.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:01.480+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:01.562 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:01.560+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:01.664 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:01.661+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:01.662548+0000) 2022-04-03T23:42:02.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:02.478+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:02.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:02.529+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:02.664 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:02.661+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:02.662701+0000) 2022-04-03T23:42:03.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:03.494+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:03.504 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:03.502+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:03.664 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:03.662+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:03.662915+0000) 2022-04-03T23:42:04.505 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:04.503+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:04.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:04.514+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:04.664 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:04.662+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:04.663119+0000) 2022-04-03T23:42:05.520 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:05.518+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:05.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:05.554+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:05.664 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:05.662+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:05.663274+0000) 2022-04-03T23:42:06.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:06.550+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:06.554 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:06.552+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:06.664 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:06.662+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:06.663407+0000) 2022-04-03T23:42:07.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:07.540+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:07.566 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:07.564+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:07.665 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:07.662+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:07.663614+0000) 2022-04-03T23:42:07.884 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:42:07.881+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:42:08.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:08.513+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:08.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:08.514+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:08.665 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:08.663+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:08.663827+0000) 2022-04-03T23:42:09.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:09.502+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:09.554 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:09.552+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:09.665 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:09.663+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:09.664014+0000) 2022-04-03T23:42:10.520 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:10.518+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:10.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:10.539+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:10.665 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:10.663+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:10.664229+0000) 2022-04-03T23:42:11.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:11.486+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:11.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:11.539+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:11.665 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:11.663+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:11.664474+0000) 2022-04-03T23:42:12.474 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:12.472+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:12.533 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:12.532+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:12.666 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:12.664+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:12.664659+0000) 2022-04-03T23:42:13.462 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:13.461+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:13.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:13.546+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:13.666 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:13.664+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:13.664852+0000) 2022-04-03T23:42:14.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:14.490+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:14.500 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:14.499+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:14.666 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:14.664+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:14.665049+0000) 2022-04-03T23:42:15.506 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:15.505+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:15.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:15.535+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:15.666 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:15.664+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:15.665223+0000) 2022-04-03T23:42:16.471 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:16.469+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:16.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:16.488+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:16.666 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:16.664+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:16.665370+0000) 2022-04-03T23:42:17.421 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:17.419+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:17.499 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:17.498+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:17.667 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:17.665+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:17.665563+0000) 2022-04-03T23:42:18.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:18.388+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:18.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:18.532+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:18.667 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:18.665+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:18.665800+0000) 2022-04-03T23:42:19.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:19.379+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:19.555 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:19.554+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:19.667 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:19.665+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:19.666001+0000) 2022-04-03T23:42:20.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:20.395+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:20.423 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:42:20.426 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:42:20.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:20.530+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:20.667 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:20.665+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:20.666163+0000) 2022-04-03T23:42:21.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:21.410+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:21.517 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:21.515+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:21.667 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:21.666+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:21.666387+0000) 2022-04-03T23:42:22.413 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:22.411+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:22.478 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:22.476+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:22.667 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:22.666+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:22.666548+0000) 2022-04-03T23:42:22.900 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:42:22.897+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:42:23.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:23.372+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:23.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:23.487+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:23.668 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:23.666+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:23.666749+0000) 2022-04-03T23:42:24.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:24.338+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:24.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:24.479+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:24.668 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:24.666+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:24.666960+0000) 2022-04-03T23:42:25.378 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:25.377+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:25.517 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:25.515+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:25.668 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:25.666+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:25.667149+0000) 2022-04-03T23:42:26.388 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:26.386+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:26.473 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:26.471+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:26.668 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:26.667+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:26.667349+0000) 2022-04-03T23:42:27.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:27.363+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:27.499 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:27.496+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:27.668 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:27.667+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:27.667529+0000) 2022-04-03T23:42:28.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:28.357+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:28.490 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:28.487+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:28.669 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:28.667+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:28.667722+0000) 2022-04-03T23:42:29.409 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:29.407+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:29.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:29.482+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:29.669 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:29.667+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:29.667902+0000) 2022-04-03T23:42:30.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:30.441+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:30.457 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:30.455+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:30.669 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:30.667+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:30.668092+0000) 2022-04-03T23:42:31.452 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:31.449+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:31.486 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:31.483+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:31.669 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:31.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:31.668299+0000) 2022-04-03T23:42:32.490 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:32.488+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:32.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:32.518+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:32.670 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:32.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:32.668501+0000) 2022-04-03T23:42:33.468 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:33.465+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:33.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:33.527+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:33.670 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:33.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:33.668710+0000) 2022-04-03T23:42:34.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:34.438+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:34.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:34.572+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:34.670 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:34.667+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:34.668907+0000) 2022-04-03T23:42:35.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:35.487+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:35.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:35.526+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:35.670 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:35.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:35.669112+0000) 2022-04-03T23:42:36.469 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:36.466+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:36.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:36.532+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:36.670 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:36.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:36.669249+0000) 2022-04-03T23:42:37.486 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:37.483+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:37.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:37.498+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:37.671 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:37.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:37.669425+0000) 2022-04-03T23:42:37.916 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:42:37.912+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:42:38.480 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:38.477+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:38.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:38.503+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:38.670 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:38.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:38.669620+0000) 2022-04-03T23:42:39.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:39.461+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:39.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:39.502+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:39.671 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:39.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:39.669857+0000) 2022-04-03T23:42:40.468 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:40.466+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:40.544 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:40.541+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:40.671 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:40.669+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:40.670064+0000) 2022-04-03T23:42:41.477 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:41.474+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:41.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:41.509+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:41.671 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:41.669+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:41.670266+0000) 2022-04-03T23:42:42.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:42.509+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:42.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:42.551+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:42.671 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:42.669+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:42.670408+0000) 2022-04-03T23:42:43.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:43.520+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:43.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:43.528+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:43.672 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:43.669+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:43.670562+0000) 2022-04-03T23:42:44.540 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:44.537+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:44.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:44.539+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:44.672 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:44.670+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:44.670757+0000) 2022-04-03T23:42:45.503 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:45.501+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:45.505 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:45.502+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:45.672 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:45.670+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:45.670966+0000) 2022-04-03T23:42:46.520 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:46.517+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:46.544 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:46.542+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:46.672 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:46.670+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:46.671164+0000) 2022-04-03T23:42:47.558 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:47.555+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:47.566 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:47.563+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:47.672 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:47.670+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:47.671390+0000) 2022-04-03T23:42:48.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:48.544+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:48.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:48.602+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:48.673 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:48.670+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:48.671586+0000) 2022-04-03T23:42:49.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:49.565+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:49.573 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:49.570+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:49.673 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:49.671+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:49.671737+0000) 2022-04-03T23:42:50.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:50.534+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:50.562 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:42:50.566 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:42:50.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:50.585+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:50.673 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:50.671+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:50.672025+0000) 2022-04-03T23:42:51.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:51.494+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:51.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:51.557+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:51.673 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:51.671+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:51.672242+0000) 2022-04-03T23:42:52.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:52.543+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:52.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:52.597+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:52.673 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:52.671+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:52.672413+0000) 2022-04-03T23:42:52.931 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:42:52.929+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:42:53.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:53.541+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:53.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:53.617+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:53.674 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:53.671+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:53.672624+0000) 2022-04-03T23:42:54.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:54.546+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:54.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:54.649+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:54.674 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:54.672+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:54.672845+0000) 2022-04-03T23:42:55.597 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:55.595+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:55.674 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:55.672+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:55.672990+0000) 2022-04-03T23:42:55.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:55.688+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:56.617 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:56.615+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:56.674 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:56.672+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:56.673164+0000) 2022-04-03T23:42:56.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:56.677+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:57.659 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:57.657+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:57.674 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:57.672+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:57.673367+0000) 2022-04-03T23:42:57.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:57.720+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:58.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:58.633+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:58.675 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:58.673+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:58.673562+0000) 2022-04-03T23:42:58.745 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:58.743+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:42:59.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:42:59.619+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:42:59.675 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:42:59.673+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:42:59.673762+0000) 2022-04-03T23:42:59.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:42:59.788+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:00.593 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:00.591+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:00.675 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:00.673+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:00.673981+0000) 2022-04-03T23:43:00.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:00.769+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:01.627 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:01.625+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:01.675 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:01.673+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:01.674183+0000) 2022-04-03T23:43:01.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:01.788+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:02.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:02.654+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:02.675 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:02.673+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:02.674369+0000) 2022-04-03T23:43:02.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:02.778+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:03.633 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:03.631+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:03.676 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:03.674+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:03.674575+0000) 2022-04-03T23:43:03.830 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:03.827+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:04.672 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:04.669+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:04.676 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:04.674+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:04.674791+0000) 2022-04-03T23:43:04.788 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:04.785+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:05.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:05.654+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:05.676 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:05.674+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:05.674993+0000) 2022-04-03T23:43:05.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:05.798+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:06.646 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:06.643+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:06.676 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:06.674+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:06.675189+0000) 2022-04-03T23:43:06.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:06.841+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:07.597 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:07.595+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:07.676 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:07.675+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:07.675372+0000) 2022-04-03T23:43:07.857 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:07.855+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:07.947 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:43:07.945+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:43:08.589 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:08.587+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:08.677 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:08.675+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:08.675583+0000) 2022-04-03T23:43:08.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:08.853+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:09.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:09.627+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:09.677 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:09.675+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:09.675798+0000) 2022-04-03T23:43:09.861 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:09.859+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:10.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:10.586+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:10.677 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:10.675+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:10.675998+0000) 2022-04-03T23:43:10.853 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:10.851+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:11.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:11.536+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:11.677 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:11.675+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:11.676174+0000) 2022-04-03T23:43:11.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:11.880+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:12.540 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:12.538+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:12.677 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:12.676+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:12.676382+0000) 2022-04-03T23:43:12.920 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:12.918+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:13.555 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:13.553+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:13.678 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:13.676+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:13.676586+0000) 2022-04-03T23:43:13.911 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:13.910+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:14.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:14.532+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:14.678 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:14.676+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:14.676806+0000) 2022-04-03T23:43:14.898 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:14.896+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:15.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:15.513+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:15.678 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:15.676+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:15.676986+0000) 2022-04-03T23:43:15.856 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:15.854+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:16.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:16.547+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:16.678 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:16.677+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:16.677193+0000) 2022-04-03T23:43:16.874 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:16.872+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:17.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:17.527+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:17.678 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:17.677+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:17.677407+0000) 2022-04-03T23:43:17.844 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:17.842+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:18.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:18.536+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:18.679 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:18.677+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:18.677574+0000) 2022-04-03T23:43:18.799 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:18.797+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:19.585 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:19.583+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:19.679 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:19.676+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:19.677809+0000) 2022-04-03T23:43:19.784 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:19.782+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:20.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:20.609+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:20.679 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:20.676+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:20.678013+0000) 2022-04-03T23:43:20.702 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:43:20.706 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:43:20.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:20.778+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:21.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:21.630+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:21.679 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:21.677+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:21.678191+0000) 2022-04-03T23:43:21.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:21.741+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:22.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:22.677+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:22.680 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:22.677+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:22.678394+0000) 2022-04-03T23:43:22.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:22.733+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:22.963 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:43:22.960+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:43:23.680 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:23.677+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:23.678616+0000) 2022-04-03T23:43:23.686 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:23.683+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:23.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:23.757+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:24.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:24.659+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:24.680 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:24.677+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:24.678861+0000) 2022-04-03T23:43:24.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:24.770+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:25.680 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:25.678+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:25.679063+0000) 2022-04-03T23:43:25.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:25.692+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:25.794 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:25.791+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:26.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:26.653+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:26.680 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:26.678+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:26.679264+0000) 2022-04-03T23:43:26.768 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:26.765+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:27.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:27.620+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:27.680 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:27.678+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:27.679431+0000) 2022-04-03T23:43:27.724 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:27.722+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:28.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:28.592+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:28.678 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:28.675+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:28.680 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:28.678+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:28.679604+0000) 2022-04-03T23:43:29.600 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:29.597+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:29.678 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:29.675+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:29.681 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:29.678+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:29.679835+0000) 2022-04-03T23:43:30.634 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:30.631+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:30.643 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:30.640+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:30.681 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:30.679+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:30.680039+0000) 2022-04-03T23:43:31.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:31.591+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:31.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:31.625+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:31.681 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:31.679+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:31.680251+0000) 2022-04-03T23:43:32.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:32.581+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:32.590 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:32.588+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:32.681 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:32.679+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:32.680461+0000) 2022-04-03T23:43:33.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:33.594+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:33.612 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:33.610+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:33.682 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:33.679+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:33.680696+0000) 2022-04-03T23:43:34.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:34.548+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:34.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:34.625+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:34.682 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:34.680+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:34.680919+0000) 2022-04-03T23:43:35.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:35.584+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:35.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:35.594+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:35.682 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:35.680+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:35.681142+0000) 2022-04-03T23:43:36.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:36.581+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:36.602 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:36.599+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:36.682 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:36.680+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:36.681351+0000) 2022-04-03T23:43:37.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:37.577+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:37.598 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:37.595+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:37.683 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:37.680+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:37.681506+0000) 2022-04-03T23:43:37.979 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:43:37.977+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:43:38.610 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:38.607+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:38.615 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:38.612+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:38.683 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:38.681+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:38.681714+0000) 2022-04-03T23:43:39.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:39.613+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:39.646 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:39.643+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:39.683 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:39.681+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:39.681900+0000) 2022-04-03T23:43:40.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:40.628+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:40.650 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:40.647+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:40.688 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:40.681+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:40.682074+0000) 2022-04-03T23:43:41.608 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:41.606+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:41.659 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:41.656+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:41.683 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:41.681+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:41.682271+0000) 2022-04-03T23:43:42.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:42.622+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:42.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:42.647+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:42.684 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:42.681+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:42.682452+0000) 2022-04-03T23:43:43.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:43.628+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:43.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:43.644+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:43.684 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:43.682+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:43.682661+0000) 2022-04-03T23:43:44.624 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:44.621+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:44.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:44.638+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:44.684 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:44.682+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:44.682855+0000) 2022-04-03T23:43:45.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:45.608+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:45.674 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:45.671+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:45.684 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:45.682+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:45.683012+0000) 2022-04-03T23:43:46.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:46.582+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:46.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:46.630+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:46.684 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:46.682+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:46.683229+0000) 2022-04-03T23:43:47.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:47.585+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:47.589 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:47.586+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:47.684 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:47.682+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:47.683434+0000) 2022-04-03T23:43:48.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:48.570+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:48.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:48.635+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:48.685 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:48.683+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:48.683646+0000) 2022-04-03T23:43:49.608 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:49.606+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:49.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:49.637+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:49.685 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:49.683+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:49.683818+0000) 2022-04-03T23:43:50.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:50.656+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:50.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:50.681+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:50.685 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:50.683+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:50.684030+0000) 2022-04-03T23:43:50.840 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:43:50.844 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:43:51.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:51.649+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:51.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:51.654+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:51.685 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:51.683+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:51.684277+0000) 2022-04-03T23:43:52.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:52.602+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:52.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:52.666+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:52.685 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:52.683+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:52.684492+0000) 2022-04-03T23:43:52.995 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:43:52.993+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:43:53.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:53.553+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:53.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:53.666+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:53.686 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:53.684+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:53.684699+0000) 2022-04-03T23:43:54.522 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:54.519+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:54.686 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:54.684+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:54.684872+0000) 2022-04-03T23:43:54.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:54.696+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:55.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:55.547+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:55.686 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:55.684+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:55.685082+0000) 2022-04-03T23:43:55.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:55.742+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:56.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:56.533+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:56.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:56.685+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:56.685290+0000) 2022-04-03T23:43:56.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:56.750+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:57.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:57.484+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:57.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:57.685+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:57.685501+0000) 2022-04-03T23:43:57.798 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:57.795+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:58.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:58.521+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:58.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:58.685+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:58.685711+0000) 2022-04-03T23:43:58.766 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:58.764+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:43:59.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:43:59.541+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:43:59.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:43:59.685+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:43:59.685942+0000) 2022-04-03T23:43:59.771 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:43:59.769+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:00.558 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:00.555+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:00.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:00.685+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:00.686101+0000) 2022-04-03T23:44:00.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:00.757+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:01.595 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:01.593+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:01.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:01.686+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:01.686317+0000) 2022-04-03T23:44:01.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:01.759+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:02.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:02.572+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:02.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:02.686+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:02.686464+0000) 2022-04-03T23:44:02.730 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:02.728+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:03.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:03.523+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:03.688 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:03.685+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:03.686689+0000) 2022-04-03T23:44:03.704 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:03.702+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:04.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:04.558+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:04.688 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:04.685+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:04.686876+0000) 2022-04-03T23:44:04.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:04.696+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:05.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:05.573+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:05.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:05.679+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:05.688 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:05.686+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:05.687067+0000) 2022-04-03T23:44:06.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:06.534+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:06.643 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:06.642+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:06.688 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:06.686+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:06.687283+0000) 2022-04-03T23:44:07.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:07.554+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:07.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:07.612+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:07.688 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:07.686+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:07.687520+0000) 2022-04-03T23:44:08.011 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:44:08.008+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:44:08.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:08.579+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:08.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:08.583+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:08.689 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:08.686+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:08.687735+0000) 2022-04-03T23:44:09.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:09.533+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:09.574 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:09.572+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:09.689 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:09.686+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:09.687939+0000) 2022-04-03T23:44:10.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:10.541+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:10.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:10.612+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:10.689 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:10.687+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:10.688141+0000) 2022-04-03T23:44:11.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:11.523+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:11.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:11.570+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:11.689 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:11.687+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:11.688353+0000) 2022-04-03T23:44:12.490 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:12.489+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:12.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:12.593+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:12.689 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:12.687+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:12.688525+0000) 2022-04-03T23:44:13.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:13.513+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:13.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:13.633+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:13.690 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:13.687+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:13.688733+0000) 2022-04-03T23:44:14.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:14.480+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:14.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:14.663+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:14.690 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:14.688+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:14.688939+0000) 2022-04-03T23:44:15.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:15.485+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:15.631 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:15.629+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:15.690 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:15.688+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:15.689114+0000) 2022-04-03T23:44:16.474 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:16.472+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:16.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:16.667+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:16.690 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:16.688+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:16.689344+0000) 2022-04-03T23:44:17.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:17.456+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:17.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:17.641+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:17.691 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:17.688+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:17.689557+0000) 2022-04-03T23:44:18.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:18.427+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:18.627 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:18.624+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:18.691 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:18.689+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:18.689764+0000) 2022-04-03T23:44:19.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:19.414+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:19.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:19.652+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:19.691 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:19.689+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:19.689995+0000) 2022-04-03T23:44:20.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:20.403+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:20.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:20.610+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:20.691 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:20.689+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:20.690208+0000) 2022-04-03T23:44:20.990 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:44:20.995 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:44:21.437 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:21.434+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:21.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:21.632+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:21.691 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:21.689+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:21.690440+0000) 2022-04-03T23:44:22.470 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:22.467+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:22.589 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:22.587+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:22.692 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:22.689+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:22.690653+0000) 2022-04-03T23:44:23.027 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:44:23.024+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:44:23.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:23.441+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:23.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:23.548+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:23.692 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:23.690+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:23.690865+0000) 2022-04-03T23:44:24.412 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:24.409+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:24.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:24.532+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:24.692 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:24.690+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:24.691073+0000) 2022-04-03T23:44:25.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:25.433+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:25.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:25.490+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:25.692 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:25.690+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:25.691288+0000) 2022-04-03T23:44:26.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:26.473+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:26.477 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:26.474+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:26.693 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:26.690+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:26.691522+0000) 2022-04-03T23:44:27.472 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:27.469+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:27.519 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:27.517+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:27.693 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:27.691+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:27.691728+0000) 2022-04-03T23:44:28.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:28.511+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:28.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:28.542+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:28.693 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:28.691+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:28.691886+0000) 2022-04-03T23:44:29.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:29.509+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:29.537 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:29.534+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:29.693 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:29.691+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:29.692080+0000) 2022-04-03T23:44:30.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:30.518+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:30.586 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:30.583+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:30.693 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:30.691+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:30.692261+0000) 2022-04-03T23:44:31.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:31.489+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:31.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:31.603+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:31.693 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:31.691+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:31.692444+0000) 2022-04-03T23:44:32.477 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:32.474+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:32.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:32.595+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:32.694 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:32.692+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:32.692628+0000) 2022-04-03T23:44:33.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:33.508+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:33.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:33.589+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:33.694 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:33.692+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:33.692834+0000) 2022-04-03T23:44:34.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:34.540+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:34.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:34.595+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:34.694 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:34.692+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:34.693065+0000) 2022-04-03T23:44:35.557 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:35.554+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:35.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:35.622+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:35.694 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:35.692+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:35.693282+0000) 2022-04-03T23:44:36.524 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:36.521+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:36.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:36.582+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:36.694 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:36.693+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:36.693504+0000) 2022-04-03T23:44:37.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:37.540+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:37.583 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:37.581+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:37.695 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:37.693+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:37.693684+0000) 2022-04-03T23:44:38.042 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:44:38.040+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:44:38.537 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:38.535+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:38.569 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:38.567+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:38.695 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:38.693+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:38.693855+0000) 2022-04-03T23:44:39.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:39.519+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:39.570 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:39.567+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:39.695 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:39.693+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:39.694089+0000) 2022-04-03T23:44:40.530 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:40.528+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:40.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:40.596+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:40.695 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:40.694+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:40.694290+0000) 2022-04-03T23:44:41.558 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:41.556+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:41.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:41.588+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:41.695 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:41.694+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:41.694513+0000) 2022-04-03T23:44:42.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:42.541+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:42.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:42.616+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:42.696 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:42.694+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:42.694721+0000) 2022-04-03T23:44:43.525 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:43.522+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:43.666 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:43.664+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:43.696 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:43.694+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:43.694921+0000) 2022-04-03T23:44:44.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:44.536+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:44.678 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:44.675+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:44.696 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:44.694+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:44.695142+0000) 2022-04-03T23:44:45.582 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:45.580+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:45.696 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:45.695+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:45.695281+0000) 2022-04-03T23:44:45.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:45.720+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:46.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:46.605+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:46.696 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:46.695+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:46.695487+0000) 2022-04-03T23:44:46.739 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:46.737+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:47.566 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:47.564+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:47.697 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:47.695+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:47.695685+0000) 2022-04-03T23:44:47.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:47.735+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:48.593 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:48.590+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:48.714 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:48.695+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:48.695868+0000) 2022-04-03T23:44:48.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:48.731+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:49.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:49.569+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:49.697 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:49.694+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:49.696058+0000) 2022-04-03T23:44:49.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:49.729+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:50.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:50.535+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:50.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:50.680+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:50.697 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:50.695+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:50.696250+0000) 2022-04-03T23:44:51.125 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:44:51.129 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:44:51.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:51.492+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:51.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:51.648+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:51.697 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:51.695+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:51.696442+0000) 2022-04-03T23:44:52.520 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:52.518+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:52.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:52.683+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:52.698 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:52.695+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:52.696631+0000) 2022-04-03T23:44:53.058 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:44:53.055+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:44:53.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:53.533+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:53.698 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:53.695+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:53.696839+0000) 2022-04-03T23:44:53.717 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:53.715+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:54.490 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:54.488+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:54.698 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:54.696+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:54.697022+0000) 2022-04-03T23:44:54.752 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:54.750+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:55.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:55.533+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:55.698 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:55.696+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:55.697223+0000) 2022-04-03T23:44:55.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:55.712+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:56.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:56.533+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:56.698 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:56.696+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:56.697408+0000) 2022-04-03T23:44:56.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:56.738+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:57.522 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:57.520+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:57.699 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:57.696+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:57.697599+0000) 2022-04-03T23:44:57.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:57.733+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:58.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:58.484+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:58.699 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:58.696+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:58.697830+0000) 2022-04-03T23:44:58.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:58.749+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:44:59.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:44:59.456+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:44:59.699 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:44:59.697+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:44:59.698002+0000) 2022-04-03T23:44:59.759 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:44:59.757+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:00.502 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:00.500+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:00.699 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:00.697+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:00.698185+0000) 2022-04-03T23:45:00.720 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:00.719+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:01.477 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:01.475+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:01.700 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:01.697+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:01.698417+0000) 2022-04-03T23:45:01.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:01.746+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:02.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:02.457+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:02.700 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:02.697+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:02.698609+0000) 2022-04-03T23:45:02.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:02.699+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:03.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:03.493+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:03.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:03.665+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:03.700 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:03.698+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:03.698873+0000) 2022-04-03T23:45:04.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:04.456+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:04.700 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:04.698+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:04.699089+0000) 2022-04-03T23:45:04.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:04.700+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:05.498 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:05.497+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:05.700 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:05.698+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:05.699241+0000) 2022-04-03T23:45:05.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:05.728+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:06.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:06.465+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:06.701 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:06.698+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:06.699446+0000) 2022-04-03T23:45:06.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:06.738+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:07.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:07.458+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:07.701 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:07.698+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:07.699639+0000) 2022-04-03T23:45:07.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:07.730+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:08.074 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:45:08.072+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:45:08.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:08.418+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:08.701 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:08.699+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:08.699879+0000) 2022-04-03T23:45:08.764 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:08.762+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:09.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:09.384+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:09.701 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:09.699+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:09.700087+0000) 2022-04-03T23:45:09.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:09.740+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:10.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:10.402+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:10.701 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:10.699+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:10.700269+0000) 2022-04-03T23:45:10.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:10.728+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:11.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:11.429+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:11.701 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:11.699+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:11.700526+0000) 2022-04-03T23:45:11.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:11.771+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:12.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:12.425+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:12.702 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:12.700+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:12.700742+0000) 2022-04-03T23:45:12.769 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:12.766+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:13.398 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:13.395+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:13.702 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:13.700+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:13.700952+0000) 2022-04-03T23:45:13.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:13.753+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:14.358 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:14.355+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:14.702 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:14.700+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:14.701174+0000) 2022-04-03T23:45:14.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:14.793+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:15.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:15.308+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:15.703 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:15.700+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:15.701432+0000) 2022-04-03T23:45:15.826 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:15.823+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:16.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:16.299+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:16.703 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:16.701+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:16.701671+0000) 2022-04-03T23:45:16.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:16.782+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:17.271 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:17.268+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:17.703 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:17.701+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:17.701928+0000) 2022-04-03T23:45:17.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:17.769+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:18.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:18.232+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:18.703 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:18.701+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:18.702148+0000) 2022-04-03T23:45:18.769 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:18.767+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:19.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:19.233+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:19.703 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:19.701+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:19.702399+0000) 2022-04-03T23:45:19.817 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:19.814+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:20.217 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:20.214+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:20.704 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:20.702+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:20.702623+0000) 2022-04-03T23:45:20.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:20.771+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:21.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:21.184+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:21.272 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:45:21.276 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:45:21.704 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:21.702+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:21.702874+0000) 2022-04-03T23:45:21.724 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:21.721+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:22.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:22.167+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:22.704 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:22.702+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:22.703084+0000) 2022-04-03T23:45:22.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:22.726+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:23.082 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:45:23.080+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:45:23.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:23.186+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:23.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:23.694+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:23.704 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:23.702+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:23.703324+0000) 2022-04-03T23:45:24.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:24.175+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:24.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:24.665+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:24.704 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:24.703+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:24.703550+0000) 2022-04-03T23:45:25.177 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:25.174+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:25.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:25.648+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:25.705 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:25.703+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:25.703809+0000) 2022-04-03T23:45:26.157 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:26.154+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:26.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:26.684+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:26.705 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:26.703+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:26.703929+0000) 2022-04-03T23:45:27.181 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:27.179+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:27.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:27.690+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:27.705 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:27.703+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:27.704092+0000) 2022-04-03T23:45:28.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:28.209+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:28.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:28.697+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:28.705 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:28.704+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:28.704293+0000) 2022-04-03T23:45:29.210 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:29.208+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:29.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:29.678+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:29.705 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:29.704+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:29.704546+0000) 2022-04-03T23:45:30.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:30.216+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:30.706 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:30.704+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:30.704802+0000) 2022-04-03T23:45:30.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:30.718+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:31.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:31.251+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:31.706 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:31.704+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:31.705005+0000) 2022-04-03T23:45:31.763 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:31.761+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:32.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:32.262+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:32.706 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:32.705+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:32.705214+0000) 2022-04-03T23:45:32.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:32.775+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:33.269 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:33.267+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:33.706 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:33.705+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:33.705417+0000) 2022-04-03T23:45:33.790 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:33.787+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:34.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:34.245+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:34.707 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:34.704+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:34.705649+0000) 2022-04-03T23:45:34.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:34.768+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:35.212 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:35.210+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:35.707 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:35.704+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:35.705859+0000) 2022-04-03T23:45:35.796 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:35.793+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:36.196 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:36.194+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:36.707 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:36.704+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:36.706062+0000) 2022-04-03T23:45:36.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:36.821+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:37.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:37.159+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:37.707 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:37.705+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:37.706301+0000) 2022-04-03T23:45:37.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:37.791+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:38.098 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:45:38.095+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:45:38.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:38.195+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:38.708 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:38.705+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:38.706487+0000) 2022-04-03T23:45:38.769 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:38.766+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:39.237 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:39.234+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:39.708 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:39.705+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:39.706697+0000) 2022-04-03T23:45:39.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:39.728+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:40.233 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:40.230+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:40.708 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:40.705+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:40.709 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:40.705+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:40.706878+0000) 2022-04-03T23:45:41.252 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:41.249+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:41.708 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:41.706+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:41.707106+0000) 2022-04-03T23:45:41.744 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:41.742+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:42.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:42.245+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:42.709 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:42.706+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:42.707329+0000) 2022-04-03T23:45:42.762 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:42.760+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:43.214 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:43.212+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:43.709 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:43.706+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:43.707579+0000) 2022-04-03T23:45:43.775 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:43.772+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:44.172 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:44.170+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:44.709 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:44.706+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:44.707812+0000) 2022-04-03T23:45:44.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:44.813+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:45.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:45.187+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:45.709 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:45.707+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:45.708008+0000) 2022-04-03T23:45:45.823 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:45.821+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:46.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:46.193+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:46.709 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:46.707+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:46.708175+0000) 2022-04-03T23:45:46.850 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:46.847+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:47.245 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:47.243+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:47.710 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:47.707+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:47.708395+0000) 2022-04-03T23:45:47.854 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:47.852+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:48.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:48.209+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:48.710 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:48.707+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:48.708595+0000) 2022-04-03T23:45:48.870 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:48.868+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:49.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:49.252+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:49.710 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:49.708+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:49.708828+0000) 2022-04-03T23:45:49.911 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:49.908+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:50.234 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:50.232+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:50.710 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:50.708+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:50.708993+0000) 2022-04-03T23:45:50.912 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:50.909+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:51.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:51.184+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:51.424 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:45:51.428 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:45:51.710 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:51.708+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:51.709237+0000) 2022-04-03T23:45:51.946 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:51.944+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:52.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:52.201+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:52.710 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:52.708+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:52.709406+0000) 2022-04-03T23:45:52.903 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:52.901+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:53.114 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:45:53.111+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:45:53.155 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:53.154+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:53.711 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:53.708+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:53.709647+0000) 2022-04-03T23:45:53.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:53.862+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:54.144 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:54.142+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:54.711 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:54.709+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:54.709880+0000) 2022-04-03T23:45:54.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:54.880+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:55.162 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:55.160+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:55.714 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:55.709+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:55.710137+0000) 2022-04-03T23:45:55.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:55.867+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:56.119 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:56.117+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:56.711 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:56.709+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:56.710369+0000) 2022-04-03T23:45:56.904 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:56.902+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:57.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:57.133+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:57.712 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:57.709+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:57.710595+0000) 2022-04-03T23:45:57.951 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:57.949+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:58.150 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:58.148+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:58.712 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:58.710+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:58.710824+0000) 2022-04-03T23:45:58.985 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:58.983+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:45:59.180 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:45:59.178+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:45:59.712 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:45:59.710+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:45:59.711070+0000) 2022-04-03T23:45:59.949 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:45:59.947+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:00.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:00.130+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:00.713 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:00.710+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:00.711307+0000) 2022-04-03T23:46:00.939 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:00.937+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:01.093 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:01.091+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:01.713 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:01.711+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:01.711547+0000) 2022-04-03T23:46:01.895 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:01.893+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:02.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:02.109+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:02.713 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:02.711+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:02.711759+0000) 2022-04-03T23:46:02.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:02.895+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:03.099 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:03.097+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:03.713 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:03.711+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:03.712048+0000) 2022-04-03T23:46:03.921 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:03.919+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:04.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:04.135+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:04.713 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:04.711+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:04.712263+0000) 2022-04-03T23:46:04.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:04.907+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:05.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:05.135+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:05.713 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:05.712+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:05.712446+0000) 2022-04-03T23:46:05.936 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:05.934+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:06.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:06.137+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:06.714 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:06.712+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:06.712657+0000) 2022-04-03T23:46:06.960 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:06.957+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:07.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:07.165+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:07.714 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:07.712+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:07.712913+0000) 2022-04-03T23:46:07.916 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:07.914+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:08.130 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:46:08.127+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:46:08.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:08.136+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:08.714 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:08.712+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:08.713116+0000) 2022-04-03T23:46:08.888 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:08.885+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:09.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:09.110+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:09.714 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:09.712+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:09.713271+0000) 2022-04-03T23:46:09.896 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:09.893+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:10.100 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:10.097+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:10.715 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:10.713+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:10.713504+0000) 2022-04-03T23:46:10.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:10.875+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:11.093 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:11.090+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:11.715 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:11.713+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:11.713765+0000) 2022-04-03T23:46:11.906 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:11.903+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:12.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:12.092+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:12.715 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:12.713+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:12.714025+0000) 2022-04-03T23:46:12.910 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:12.907+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:13.068 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:13.066+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:13.715 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:13.713+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:13.714172+0000) 2022-04-03T23:46:13.946 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:13.943+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:14.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:14.073+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:14.715 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:14.714+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:14.714366+0000) 2022-04-03T23:46:14.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:14.945+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:15.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:15.034+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:15.716 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:15.714+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:15.714576+0000) 2022-04-03T23:46:15.926 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:15.923+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:16.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:16.002+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:16.716 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:16.714+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:16.714817+0000) 2022-04-03T23:46:16.966 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:16.963+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:17.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:17.035+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:17.716 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:17.714+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:17.714995+0000) 2022-04-03T23:46:17.954 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:17.951+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:18.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:18.072+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:18.716 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:18.715+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:18.715173+0000) 2022-04-03T23:46:18.921 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:18.918+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:19.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:19.101+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:19.716 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:19.715+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:19.715369+0000) 2022-04-03T23:46:19.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:19.940+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:20.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:20.130+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:20.717 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:20.714+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:20.715513+0000) 2022-04-03T23:46:20.900 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:20.897+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:21.093 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:21.091+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:21.567 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:46:21.571 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:46:21.717 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:21.713+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:21.715753+0000) 2022-04-03T23:46:21.904 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:21.901+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:22.057 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:22.054+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:22.717 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:22.714+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:22.715974+0000) 2022-04-03T23:46:22.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:22.945+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:23.091 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:23.088+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:23.140 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:46:23.137+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:46:23.717 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:23.715+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:23.716212+0000) 2022-04-03T23:46:23.975 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:23.973+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:24.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:24.110+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:24.717 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:24.715+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:24.716412+0000) 2022-04-03T23:46:24.989 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:24.986+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:25.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:25.125+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:25.718 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:25.715+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:25.716590+0000) 2022-04-03T23:46:25.998 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:25.995+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:26.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:26.092+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:26.718 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:26.715+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:26.716710+0000) 2022-04-03T23:46:26.997 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:26.995+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:27.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:27.043+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:27.718 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:27.715+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:27.716928+0000) 2022-04-03T23:46:27.980 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:27.977+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:28.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:28.063+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:28.718 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:28.716+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:28.717074+0000) 2022-04-03T23:46:28.994 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:28.992+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:29.023 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:29.021+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:29.718 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:29.716+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:29.717262+0000) 2022-04-03T23:46:29.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:29.987+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:30.036 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:30.034+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:30.719 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:30.716+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:30.717503+0000) 2022-04-03T23:46:30.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:30.952+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:31.075 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:31.073+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:31.719 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:31.716+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:31.717729+0000) 2022-04-03T23:46:31.910 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:31.908+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:32.080 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:32.077+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:32.719 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:32.717+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:32.717976+0000) 2022-04-03T23:46:32.871 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:32.868+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:33.098 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:33.095+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:33.719 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:33.717+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:33.718211+0000) 2022-04-03T23:46:33.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:33.827+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:34.049 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:34.046+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:34.720 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:34.717+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:34.718357+0000) 2022-04-03T23:46:34.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:34.779+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:35.074 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:35.071+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:35.720 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:35.717+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:35.718492+0000) 2022-04-03T23:46:35.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:35.785+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:36.075 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:36.072+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:36.720 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:36.717+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:36.718699+0000) 2022-04-03T23:46:36.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:36.781+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:37.099 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:37.096+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:37.720 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:37.718+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:37.718882+0000) 2022-04-03T23:46:37.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:37.753+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:38.146 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:38.143+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:38.156 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:46:38.154+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:46:38.720 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:38.718+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:38.719104+0000) 2022-04-03T23:46:38.769 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:38.766+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:39.149 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:39.146+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:39.720 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:39.718+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:39.719346+0000) 2022-04-03T23:46:39.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:39.808+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:40.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:40.177+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:40.720 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:40.718+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:40.719517+0000) 2022-04-03T23:46:40.844 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:40.842+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:41.143 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:41.141+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:41.721 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:41.719+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:41.719742+0000) 2022-04-03T23:46:41.809 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:41.807+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:42.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:42.135+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:42.721 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:42.719+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:42.719954+0000) 2022-04-03T23:46:42.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:42.821+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:43.172 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:43.170+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:43.721 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:43.719+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:43.720163+0000) 2022-04-03T23:46:43.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:43.870+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:44.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:44.171+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:44.721 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:44.719+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:44.720400+0000) 2022-04-03T23:46:44.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:44.879+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:45.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:45.161+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:45.722 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:45.719+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:45.720586+0000) 2022-04-03T23:46:45.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:45.860+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:46.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:46.200+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:46.722 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:46.720+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:46.720799+0000) 2022-04-03T23:46:46.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:46.834+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:47.182 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:47.180+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:47.722 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:47.720+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:47.721033+0000) 2022-04-03T23:46:47.826 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:47.824+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:48.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:48.187+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:48.722 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:48.720+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:48.721253+0000) 2022-04-03T23:46:48.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:48.782+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:49.144 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:49.142+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:49.722 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:49.720+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:49.721398+0000) 2022-04-03T23:46:49.747 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:49.745+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:50.152 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:50.150+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:50.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:50.696+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:50.723 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:50.721+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:50.721576+0000) 2022-04-03T23:46:51.183 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:51.181+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:51.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:51.698+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:51.711 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:46:51.715 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:46:51.723 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:51.720+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:51.721733+0000) 2022-04-03T23:46:52.143 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:52.141+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:52.723 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:52.721+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:52.721895+0000) 2022-04-03T23:46:52.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:52.740+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:53.141 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:53.139+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:53.172 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:46:53.170+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:46:53.723 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:53.721+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:53.722104+0000) 2022-04-03T23:46:53.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:53.759+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:54.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:54.154+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:54.723 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:54.721+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:54.722347+0000) 2022-04-03T23:46:54.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:54.757+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:55.119 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:55.117+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:55.724 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:55.721+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:55.722552+0000) 2022-04-03T23:46:55.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:55.792+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:56.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:56.135+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:56.724 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:56.722+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:56.722793+0000) 2022-04-03T23:46:56.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:56.758+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:57.102 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:57.100+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:57.724 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:57.722+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:57.723013+0000) 2022-04-03T23:46:57.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:57.786+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:58.135 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:58.133+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:58.724 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:58.723+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:58.723265+0000) 2022-04-03T23:46:58.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:58.822+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:46:59.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:46:59.110+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:46:59.724 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:46:59.723+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:46:59.723446+0000) 2022-04-03T23:46:59.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:46:59.838+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:00.068 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:00.065+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:00.725 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:00.723+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:00.723618+0000) 2022-04-03T23:47:00.804 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:00.802+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:01.079 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:01.077+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:01.725 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:01.723+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:01.723869+0000) 2022-04-03T23:47:01.852 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:01.759+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:02.129 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:02.126+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:02.725 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:02.723+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:02.724065+0000) 2022-04-03T23:47:02.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:02.790+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:03.141 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:03.138+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:03.726 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:03.724+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:03.724271+0000) 2022-04-03T23:47:03.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:03.812+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:04.191 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:04.188+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:04.726 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:04.724+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:04.724486+0000) 2022-04-03T23:47:04.774 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:04.771+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:05.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:05.232+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:05.726 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:05.723+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:05.724702+0000) 2022-04-03T23:47:05.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:05.807+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:06.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:06.198+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:06.726 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:06.723+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:06.724920+0000) 2022-04-03T23:47:06.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:06.759+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:07.217 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:07.215+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:07.726 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:07.724+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:07.725112+0000) 2022-04-03T23:47:07.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:07.776+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:08.188 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:47:08.185+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:47:08.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:08.227+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:08.726 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:08.724+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:08.725231+0000) 2022-04-03T23:47:08.734 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:08.731+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:09.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:09.207+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:09.718 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:09.715+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:09.726 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:09.724+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:09.725382+0000) 2022-04-03T23:47:10.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:10.191+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:10.727 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:10.723+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:10.725588+0000) 2022-04-03T23:47:10.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:10.756+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:11.171 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:11.168+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:11.727 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:11.724+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:11.725860+0000) 2022-04-03T23:47:11.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:11.729+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:12.125 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:12.122+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:12.727 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:12.725+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:12.726088+0000) 2022-04-03T23:47:12.766 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:12.763+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:13.162 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:13.159+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:13.722 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:13.720+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:13.727 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:13.725+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:13.726278+0000) 2022-04-03T23:47:14.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:14.208+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:14.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:14.691+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:14.728 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:14.725+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:14.726479+0000) 2022-04-03T23:47:15.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:15.229+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:15.697 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:15.694+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:15.728 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:15.725+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:15.726706+0000) 2022-04-03T23:47:16.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:16.216+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:16.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:16.692+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:16.728 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:16.726+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:16.726947+0000) 2022-04-03T23:47:17.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:17.255+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:17.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:17.677+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:17.728 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:17.726+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:17.727184+0000) 2022-04-03T23:47:18.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:18.225+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:18.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:18.724+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:18.728 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:18.726+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:18.727383+0000) 2022-04-03T23:47:19.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:19.263+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:19.729 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:19.726+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:19.727547+0000) 2022-04-03T23:47:19.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:19.748+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:20.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:20.285+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:20.716 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:20.714+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:20.729 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:20.726+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:20.727695+0000) 2022-04-03T23:47:21.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:21.335+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:21.729 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:21.727+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:21.727916+0000) 2022-04-03T23:47:21.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:21.735+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:21.835 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:47:21.839 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:47:22.332 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:22.330+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:22.729 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:22.727+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:22.728077+0000) 2022-04-03T23:47:22.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:22.732+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:23.191 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:47:23.189+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:47:23.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:23.301+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:23.709 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:23.707+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:23.729 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:23.727+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:23.728281+0000) 2022-04-03T23:47:24.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:24.284+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:24.696 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:24.694+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:24.730 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:24.727+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:24.728500+0000) 2022-04-03T23:47:25.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:25.304+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:25.706 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:25.703+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:47:25.730 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:25.728+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:25.728719+0000) 2022-04-03T23:47:26.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:26.328+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:26.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:26.685+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:26.730 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:26.728+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:26.728897+0000) 2022-04-03T23:47:27.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:27.332+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:27.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:27.669+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:27.730 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:27.728+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:27.729026+0000) 2022-04-03T23:47:28.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:28.324+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:28.674 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:28.671+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:28.730 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:28.728+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:28.729260+0000) 2022-04-03T23:47:29.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:29.277+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:29.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:29.675+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:29.731 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:29.728+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:29.729513+0000) 2022-04-03T23:47:30.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:30.235+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:30.651 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:30.648+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:30.731 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:30.729+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:30.729725+0000) 2022-04-03T23:47:31.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:31.267+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:31.650 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:31.647+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:31.731 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:31.729+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:31.729957+0000) 2022-04-03T23:47:32.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:32.299+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:32.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:32.653+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:32.731 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:32.729+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:32.730121+0000) 2022-04-03T23:47:33.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:33.348+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:33.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:33.673+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:33.731 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:33.729+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:33.730320+0000) 2022-04-03T23:47:34.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:34.349+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:34.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:34.645+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:34.732 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:34.729+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:34.730560+0000) 2022-04-03T23:47:35.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:35.370+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:35.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:35.621+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:35.732 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:35.730+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:35.730817+0000) 2022-04-03T23:47:36.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:36.411+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:36.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:36.665+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:36.732 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:36.730+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:36.731043+0000) 2022-04-03T23:47:37.400 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:37.397+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:37.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:37.638+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:37.732 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:37.730+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:37.731237+0000) 2022-04-03T23:47:38.207 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:47:38.204+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:47:38.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:38.401+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:38.597 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:38.595+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:38.733 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:38.731+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:38.731430+0000) 2022-04-03T23:47:39.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:39.414+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:39.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:39.600+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:39.733 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:39.730+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:39.731693+0000) 2022-04-03T23:47:40.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:40.371+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:40.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:40.640+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:40.733 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:40.731+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:40.731869+0000) 2022-04-03T23:47:41.415 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:41.412+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:41.634 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:41.632+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:41.733 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:41.731+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:41.732090+0000) 2022-04-03T23:47:42.375 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:42.373+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:42.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:42.605+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:42.733 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:42.732+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:42.732283+0000) 2022-04-03T23:47:43.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:43.326+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:43.626 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:43.624+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:43.734 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:43.732+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:43.732487+0000) 2022-04-03T23:47:44.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:44.372+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:44.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:44.668+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:44.734 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:44.732+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:44.732731+0000) 2022-04-03T23:47:45.378 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:45.377+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:45.708 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:45.706+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:45.734 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:45.732+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:45.732950+0000) 2022-04-03T23:47:46.429 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:46.427+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:46.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:46.700+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:46.735 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:46.733+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:46.733193+0000) 2022-04-03T23:47:47.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:47.401+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:47.735 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:47.732+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:47.733440+0000) 2022-04-03T23:47:47.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:47.747+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:48.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:48.433+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:48.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:48.711+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:48.735 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:48.733+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:48.733662+0000) 2022-04-03T23:47:49.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:49.442+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:49.735 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:49.733+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:49.733893+0000) 2022-04-03T23:47:49.746 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:49.744+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:50.426 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:50.424+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:50.735 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:50.733+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:50.734107+0000) 2022-04-03T23:47:50.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:50.742+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:51.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:51.404+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:51.735 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:51.733+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:51.734326+0000) 2022-04-03T23:47:51.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:51.756+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:52.058 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:47:52.062 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:47:52.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:52.402+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:52.736 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:52.733+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:52.734519+0000) 2022-04-03T23:47:52.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:52.792+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:53.223 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:47:53.219+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:47:53.446 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:53.444+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:53.736 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:53.733+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:53.734675+0000) 2022-04-03T23:47:53.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:53.758+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:54.418 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:54.416+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:54.736 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:54.733+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:54.734833+0000) 2022-04-03T23:47:54.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:54.734+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:55.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:55.377+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:55.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:55.697+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:55.736 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:55.734+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:55.734985+0000) 2022-04-03T23:47:56.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:56.389+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:56.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:56.731+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:56.736 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:56.734+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:56.735218+0000) 2022-04-03T23:47:57.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:57.352+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:57.736 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:57.734+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:57.735396+0000) 2022-04-03T23:47:57.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:57.755+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:58.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:58.346+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:58.738 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:58.734+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:58.735603+0000) 2022-04-03T23:47:58.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:58.798+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:47:59.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:47:59.342+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:47:59.737 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:47:59.734+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:47:59.735846+0000) 2022-04-03T23:47:59.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:47:59.836+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:00.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:00.333+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:00.737 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:00.735+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:00.736034+0000) 2022-04-03T23:48:00.856 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:00.853+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:01.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:01.345+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:01.737 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:01.735+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:01.736252+0000) 2022-04-03T23:48:01.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:01.830+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:02.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:02.323+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:02.737 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:02.735+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:02.736474+0000) 2022-04-03T23:48:02.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:02.804+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:03.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:03.288+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:03.738 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:03.735+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:03.736714+0000) 2022-04-03T23:48:03.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:03.773+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:04.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:04.318+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:04.738 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:04.736+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:04.736971+0000) 2022-04-03T23:48:04.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:04.808+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:05.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:05.341+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:05.738 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:05.736+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:05.737158+0000) 2022-04-03T23:48:05.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:05.790+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:06.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:06.295+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:06.738 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:06.736+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:06.737326+0000) 2022-04-03T23:48:06.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:06.822+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:07.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:07.250+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:07.739 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:07.736+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:07.737495+0000) 2022-04-03T23:48:07.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:07.804+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:08.239 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:48:08.236+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:48:08.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:08.288+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:08.739 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:08.736+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:08.737704+0000) 2022-04-03T23:48:08.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:08.835+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:09.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:09.279+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:09.739 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:09.737+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:09.737968+0000) 2022-04-03T23:48:09.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:09.858+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:10.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:10.311+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:10.739 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:10.737+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:10.738160+0000) 2022-04-03T23:48:10.819 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:10.816+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:11.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:11.300+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:11.739 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:11.737+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:11.738380+0000) 2022-04-03T23:48:11.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:11.779+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:12.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:12.278+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:12.740 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:12.737+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:12.738605+0000) 2022-04-03T23:48:12.754 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:12.752+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:13.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:13.251+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:13.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:13.736+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:13.740 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:13.738+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:13.738738+0000) 2022-04-03T23:48:14.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:14.220+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:14.740 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:14.738+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:14.738934+0000) 2022-04-03T23:48:14.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:14.768+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:15.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:15.233+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:15.740 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:15.738+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:15.739110+0000) 2022-04-03T23:48:15.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:15.807+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:16.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:16.244+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:16.740 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:16.738+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:16.739320+0000) 2022-04-03T23:48:16.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:16.832+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:17.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:17.286+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:17.741 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:17.738+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:17.739503+0000) 2022-04-03T23:48:17.814 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:17.812+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:18.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:18.292+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:18.741 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:18.739+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:18.739694+0000) 2022-04-03T23:48:18.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:18.805+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:19.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:19.273+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:19.741 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:19.739+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:19.739899+0000) 2022-04-03T23:48:19.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:19.788+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:20.233 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:20.230+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:20.741 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:20.739+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:20.740103+0000) 2022-04-03T23:48:20.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:20.770+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:21.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:21.253+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:21.741 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:21.739+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:21.740321+0000) 2022-04-03T23:48:21.809 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:21.807+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:22.198 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:48:22.202 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:48:22.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:22.228+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:22.742 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:22.739+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:22.740528+0000) 2022-04-03T23:48:22.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:22.807+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:23.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:23.213+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:23.254 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:48:23.252+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:48:23.742 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:23.740+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:23.740753+0000) 2022-04-03T23:48:23.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:23.805+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:24.172 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:24.169+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:24.742 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:24.740+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:24.740978+0000) 2022-04-03T23:48:24.809 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:24.806+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:25.162 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:25.159+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:25.742 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:25.740+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:25.741196+0000) 2022-04-03T23:48:25.790 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:25.787+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:26.195 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:26.193+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:26.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:26.739+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:26.743 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:26.741+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:26.741385+0000) 2022-04-03T23:48:27.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:27.165+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:27.743 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:27.741+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:27.741591+0000) 2022-04-03T23:48:27.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:27.753+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:28.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:28.155+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:28.743 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:28.741+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:28.741810+0000) 2022-04-03T23:48:28.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:28.797+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:29.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:29.183+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:29.743 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:29.741+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:29.742007+0000) 2022-04-03T23:48:29.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:29.790+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:30.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:30.177+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:30.743 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:30.741+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:30.742132+0000) 2022-04-03T23:48:30.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:30.791+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:31.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:31.213+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:31.743 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:31.742+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:31.742290+0000) 2022-04-03T23:48:31.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:31.778+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:32.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:32.191+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:32.744 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:32.742+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:32.742487+0000) 2022-04-03T23:48:32.790 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:32.788+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:33.176 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:33.174+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:33.744 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:33.742+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:33.742694+0000) 2022-04-03T23:48:33.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:33.780+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:34.146 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:34.144+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:34.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:34.733+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:34.744 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:34.742+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:34.742890+0000) 2022-04-03T23:48:35.131 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:35.129+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:35.744 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:35.742+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:35.743082+0000) 2022-04-03T23:48:35.769 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:35.767+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:36.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:36.159+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:36.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:36.740+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:36.744 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:36.742+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:36.743284+0000) 2022-04-03T23:48:37.123 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:37.121+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:37.745 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:37.742+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:37.743482+0000) 2022-04-03T23:48:37.763 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:37.761+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:38.141 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:38.139+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:38.270 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:48:38.267+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:48:38.745 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:38.742+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:38.743694+0000) 2022-04-03T23:48:38.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:38.789+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:39.094 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:39.092+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:39.745 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:39.742+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:39.743911+0000) 2022-04-03T23:48:39.756 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:39.755+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:40.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:40.079+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:40.745 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:40.743+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:40.744126+0000) 2022-04-03T23:48:40.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:40.780+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:41.033 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:41.031+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:41.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:41.735+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:41.745 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:41.743+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:41.744333+0000) 2022-04-03T23:48:42.017 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:42.015+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:42.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:42.726+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:42.745 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:42.743+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:42.744530+0000) 2022-04-03T23:48:42.972 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:42.971+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:43.746 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:43.743+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:43.744747+0000) 2022-04-03T23:48:43.765 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:43.763+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:43.966 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:43.964+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:44.746 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:44.744+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:44.745007+0000) 2022-04-03T23:48:44.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:44.777+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:44.976 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:44.974+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:45.746 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:45.744+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:45.745207+0000) 2022-04-03T23:48:45.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:45.823+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:46.018 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:46.016+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:46.747 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:46.744+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:46.745406+0000) 2022-04-03T23:48:46.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:46.781+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:47.066 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:47.064+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:47.738 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:47.736+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:47.746 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:47.744+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:47.745576+0000) 2022-04-03T23:48:48.020 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:48.018+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:48.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:48.708+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:48.747 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:48.744+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:48.745739+0000) 2022-04-03T23:48:49.052 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:49.050+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:49.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:49.723+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:49.747 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:49.745+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:49.745918+0000) 2022-04-03T23:48:50.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:50.002+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:50.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:50.713+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:50.747 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:50.745+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:50.746116+0000) 2022-04-03T23:48:51.045 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:51.043+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:51.701 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:51.699+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:51.747 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:51.745+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:51.746317+0000) 2022-04-03T23:48:52.082 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:52.079+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:52.262 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:48:52.266 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:48:52.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:52.691+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:52.748 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:52.745+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:52.746488+0000) 2022-04-03T23:48:53.089 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:53.085+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:53.286 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:48:53.284+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:48:53.721 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:53.718+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:53.747 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:53.746+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:53.746634+0000) 2022-04-03T23:48:54.128 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:54.125+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:54.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:54.729+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:54.748 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:54.746+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:54.746864+0000) 2022-04-03T23:48:55.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:55.078+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:55.709 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:55.706+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:55.748 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:55.746+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:55.747031+0000) 2022-04-03T23:48:56.062 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:56.059+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:56.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:56.708+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:56.748 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:56.746+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:56.747188+0000) 2022-04-03T23:48:57.093 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:57.090+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:57.707 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:57.704+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:57.748 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:57.746+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:57.747305+0000) 2022-04-03T23:48:58.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:58.070+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:58.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:58.673+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:58.749 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:58.746+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:58.747506+0000) 2022-04-03T23:48:59.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:48:59.112+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:48:59.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:48:59.679+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:48:59.749 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:48:59.747+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:48:59.747733+0000) 2022-04-03T23:49:00.100 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:00.097+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:00.681 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:00.678+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:00.749 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:00.747+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:00.747919+0000) 2022-04-03T23:49:01.078 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:01.075+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:01.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:01.635+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:01.749 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:01.747+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:01.748126+0000) 2022-04-03T23:49:02.075 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:02.072+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:02.643 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:02.640+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:02.749 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:02.747+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:02.748288+0000) 2022-04-03T23:49:03.039 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:03.036+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:03.598 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:03.596+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:03.750 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:03.748+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:03.748490+0000) 2022-04-03T23:49:03.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:03.992+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:04.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:04.626+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:04.750 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:04.748+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:04.748692+0000) 2022-04-03T23:49:05.032 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:05.029+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:05.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:05.608+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:05.750 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:05.748+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:05.748874+0000) 2022-04-03T23:49:06.001 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:05.998+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:06.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:06.654+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:06.750 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:06.748+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:06.749078+0000) 2022-04-03T23:49:06.985 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:06.982+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:07.672 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:07.669+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:07.751 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:07.748+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:07.749295+0000) 2022-04-03T23:49:07.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:07.977+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:08.302 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:49:08.300+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:49:08.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:08.695+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:08.750 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:08.749+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:08.749485+0000) 2022-04-03T23:49:08.976 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:08.973+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:09.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:09.674+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:09.751 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:09.749+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:09.749658+0000) 2022-04-03T23:49:10.019 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:10.016+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:10.711 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:10.708+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:10.751 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:10.749+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:10.749862+0000) 2022-04-03T23:49:11.001 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:10.998+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:11.751 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:11.749+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:11.750044+0000) 2022-04-03T23:49:11.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:11.752+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:11.999 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:11.996+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:12.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:12.721+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:12.751 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:12.749+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:12.750253+0000) 2022-04-03T23:49:13.034 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:13.032+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:13.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:13.674+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:13.752 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:13.750+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:13.750458+0000) 2022-04-03T23:49:14.046 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:14.043+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:14.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:14.696+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:14.752 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:14.750+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:14.750667+0000) 2022-04-03T23:49:15.096 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:15.093+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:15.689 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:15.686+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:15.752 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:15.750+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:15.750827+0000) 2022-04-03T23:49:16.106 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:16.104+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:16.703 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:16.701+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:16.752 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:16.750+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:16.750983+0000) 2022-04-03T23:49:17.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:17.151+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:17.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:17.652+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:17.752 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:17.751+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:17.751207+0000) 2022-04-03T23:49:18.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:18.187+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:18.606 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:18.603+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:18.753 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:18.751+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:18.751442+0000) 2022-04-03T23:49:19.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:19.215+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:19.615 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:19.613+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:19.753 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:19.751+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:19.751645+0000) 2022-04-03T23:49:20.172 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:20.169+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:20.619 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:20.617+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:20.753 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:20.750+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:20.751883+0000) 2022-04-03T23:49:21.136 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:21.133+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:21.573 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:21.570+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:21.753 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:21.751+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:21.752110+0000) 2022-04-03T23:49:22.140 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:22.138+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:22.394 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:49:22.398 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:49:22.562 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:22.560+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:22.753 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:22.751+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:22.752310+0000) 2022-04-03T23:49:23.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:23.113+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:23.312 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:49:23.308+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:49:23.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:23.557+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:23.754 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:23.751+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:23.752523+0000) 2022-04-03T23:49:24.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:24.078+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:24.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:24.543+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:24.754 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:24.751+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:24.752743+0000) 2022-04-03T23:49:25.039 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:25.036+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:25.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:25.532+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:25.754 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:25.752+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:25.752946+0000) 2022-04-03T23:49:26.045 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:26.043+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:26.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:26.539+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:26.754 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:26.752+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:26.753124+0000) 2022-04-03T23:49:27.011 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:27.009+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:27.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:27.500+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:27.754 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:27.752+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:27.753366+0000) 2022-04-03T23:49:27.999 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:27.997+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:28.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:28.550+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:28.755 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:28.752+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:28.753584+0000) 2022-04-03T23:49:29.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:29.002+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:29.597 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:29.595+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:29.755 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:29.752+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:29.753722+0000) 2022-04-03T23:49:29.987 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:29.985+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:30.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:30.638+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:30.755 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:30.753+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:30.753890+0000) 2022-04-03T23:49:31.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:31.001+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:31.650 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:31.648+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:31.755 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:31.753+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:31.754082+0000) 2022-04-03T23:49:31.960 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:31.958+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:32.626 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:32.624+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:32.755 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:32.753+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:32.754295+0000) 2022-04-03T23:49:32.951 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:32.949+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:33.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:33.609+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:33.756 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:33.753+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:33.754507+0000) 2022-04-03T23:49:33.964 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:33.962+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:34.615 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:34.613+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:34.756 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:34.753+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:34.754708+0000) 2022-04-03T23:49:34.950 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:34.948+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:35.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:35.609+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:35.756 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:35.754+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:35.754892+0000) 2022-04-03T23:49:35.968 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:35.966+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:36.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:36.578+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:36.756 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:36.754+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:36.755072+0000) 2022-04-03T23:49:37.018 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:37.016+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:37.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:37.612+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:37.756 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:37.754+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:37.755311+0000) 2022-04-03T23:49:38.011 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:38.010+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:38.327 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:49:38.325+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:49:38.639 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:38.636+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:38.757 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:38.754+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:38.755552+0000) 2022-04-03T23:49:39.058 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:39.057+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:39.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:39.638+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:39.757 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:39.754+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:39.755730+0000) 2022-04-03T23:49:40.092 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:40.090+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:40.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:40.614+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:40.759 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:40.755+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:40.755858+0000) 2022-04-03T23:49:41.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:41.065+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:41.600 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:41.598+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:41.757 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:41.755+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:41.756064+0000) 2022-04-03T23:49:42.109 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:42.107+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:42.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:42.611+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:42.757 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:42.755+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:42.756269+0000) 2022-04-03T23:49:43.068 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:43.066+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:43.615 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:43.613+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:43.758 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:43.755+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:43.756464+0000) 2022-04-03T23:49:44.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:44.114+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:44.570 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:44.568+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:44.758 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:44.756+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:44.756687+0000) 2022-04-03T23:49:45.097 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:45.095+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:45.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:45.540+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:45.758 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:45.756+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:45.756847+0000) 2022-04-03T23:49:46.099 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:46.097+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:46.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:46.519+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:46.758 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:46.756+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:46.757029+0000) 2022-04-03T23:49:47.104 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:47.101+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:47.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:47.508+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:47.758 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:47.756+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:47.757210+0000) 2022-04-03T23:49:48.126 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:48.123+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:48.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:48.491+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:48.758 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:48.756+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:48.757366+0000) 2022-04-03T23:49:49.171 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:49.168+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:49.482 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:49.479+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:49.759 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:49.757+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:49.757538+0000) 2022-04-03T23:49:50.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:50.212+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:50.527 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:50.524+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:50.759 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:50.757+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:50.757714+0000) 2022-04-03T23:49:51.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:51.226+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:51.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:51.509+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:51.759 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:51.757+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:51.757897+0000) 2022-04-03T23:49:52.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:52.264+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:52.500 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:52.497+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:52.521 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:49:52.525 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:49:52.759 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:52.757+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:52.758120+0000) 2022-04-03T23:49:53.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:53.304+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:53.343 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:49:53.341+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:49:53.465 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:53.462+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:53.759 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:53.758+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:53.758353+0000) 2022-04-03T23:49:54.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:54.284+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:54.434 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:54.431+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:54.760 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:54.758+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:54.758589+0000) 2022-04-03T23:49:55.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:55.264+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:55.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:55.462+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:55.760 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:55.758+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:55.758844+0000) 2022-04-03T23:49:56.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:56.247+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:56.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:56.455+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:56.760 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:56.758+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:56.758997+0000) 2022-04-03T23:49:57.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:57.277+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:57.426 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:57.423+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:57.760 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:57.758+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:57.759203+0000) 2022-04-03T23:49:58.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:58.293+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:58.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:58.473+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:58.761 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:58.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:58.759397+0000) 2022-04-03T23:49:59.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:49:59.335+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:49:59.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:49:59.462+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:49:59.761 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:49:59.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:49:59.759601+0000) 2022-04-03T23:50:00.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:00.381+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:00.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:00.438+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:00.761 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:00.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:00.759765+0000) 2022-04-03T23:50:01.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:01.367+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:01.438 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:01.436+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:01.761 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:01.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:01.759938+0000) 2022-04-03T23:50:02.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:02.351+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:02.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:02.432+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:02.761 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:02.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:02.760111+0000) 2022-04-03T23:50:03.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:03.356+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:03.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:03.453+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:03.761 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:03.760+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:03.760325+0000) 2022-04-03T23:50:04.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:04.382+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:04.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:04.445+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:04.762 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:04.760+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:04.760550+0000) 2022-04-03T23:50:05.415 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:05.412+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:05.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:05.454+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:05.762 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:05.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:05.760792+0000) 2022-04-03T23:50:06.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:06.381+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:06.457 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:06.454+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:06.762 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:06.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:06.761010+0000) 2022-04-03T23:50:07.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:07.365+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:07.452 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:07.450+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:07.763 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:07.760+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:07.761229+0000) 2022-04-03T23:50:08.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:08.355+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:08.359 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:50:08.356+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:50:08.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:08.427+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:08.763 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:08.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:08.761471+0000) 2022-04-03T23:50:09.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:09.311+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:09.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:09.461+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:09.763 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:09.760+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:09.761677+0000) 2022-04-03T23:50:10.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:10.291+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:10.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:10.488+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:10.763 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:10.760+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:10.761827+0000) 2022-04-03T23:50:11.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:11.249+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:11.499 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:11.497+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:11.763 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:11.761+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:11.762035+0000) 2022-04-03T23:50:12.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:12.291+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:12.462 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:12.460+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:12.763 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:12.761+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:12.762152+0000) 2022-04-03T23:50:13.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:13.340+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:13.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:13.414+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:13.763 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:13.761+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:13.762314+0000) 2022-04-03T23:50:14.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:14.364+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:14.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:14.371+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:14.764 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:14.761+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:14.762541+0000) 2022-04-03T23:50:15.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:15.325+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:15.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:15.387+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:15.764 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:15.762+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:15.762749+0000) 2022-04-03T23:50:16.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:16.370+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:16.433 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:16.431+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:16.764 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:16.762+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:16.763025+0000) 2022-04-03T23:50:17.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:17.401+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:17.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:17.442+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:17.764 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:17.762+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:17.763235+0000) 2022-04-03T23:50:18.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:18.415+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:18.434 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:18.432+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:18.765 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:18.762+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:18.763414+0000) 2022-04-03T23:50:19.397 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:19.395+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:19.423 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:19.421+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:19.765 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:19.762+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:19.763620+0000) 2022-04-03T23:50:20.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:20.414+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:20.432 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:20.429+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:20.765 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:20.763+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:20.763855+0000) 2022-04-03T23:50:21.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:21.440+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:21.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:21.452+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:21.765 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:21.763+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:21.764093+0000) 2022-04-03T23:50:22.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:22.427+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:22.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:22.456+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:22.688 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:50:22.692 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:50:22.765 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:22.763+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:22.764350+0000) 2022-04-03T23:50:23.375 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:50:23.373+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:50:23.478 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:23.475+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:23.493 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:23.491+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:23.766 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:23.763+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:23.764569+0000) 2022-04-03T23:50:24.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:24.435+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:24.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:24.466+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:24.766 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:24.764+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:24.764788+0000) 2022-04-03T23:50:25.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:25.481+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-03T23:50:25.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:25.494+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:25.766 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:25.764+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:25.765013+0000) 2022-04-03T23:50:26.441 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:26.439+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:26.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:26.519+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:26.766 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:26.764+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:26.765226+0000) 2022-04-03T23:50:27.397 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:27.395+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:27.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:27.546+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:27.767 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:27.764+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:27.765442+0000) 2022-04-03T23:50:28.366 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:28.364+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:28.533 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:28.531+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:28.767 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:28.765+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:28.765665+0000) 2022-04-03T23:50:29.329 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:29.327+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:29.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:29.513+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:29.767 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:29.765+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:29.765891+0000) 2022-04-03T23:50:30.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:30.282+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:30.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:30.532+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:30.767 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:30.765+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:30.766099+0000) 2022-04-03T23:50:31.306 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:31.304+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:31.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:31.522+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:31.767 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:31.765+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:31.766291+0000) 2022-04-03T23:50:32.344 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:32.342+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:32.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:32.566+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:32.767 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:32.766+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:32.766495+0000) 2022-04-03T23:50:33.337 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:33.335+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:33.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:33.525+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:33.768 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:33.765+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:33.766688+0000) 2022-04-03T23:50:34.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:34.366+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:34.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:34.512+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:34.768 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:34.766+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:34.766876+0000) 2022-04-03T23:50:35.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:35.339+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:35.558 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:35.556+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:35.768 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:35.766+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:35.767076+0000) 2022-04-03T23:50:36.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:36.339+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:36.570 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:36.569+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:36.768 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:36.766+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:36.767279+0000) 2022-04-03T23:50:37.360 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:37.358+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:37.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:37.533+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:37.769 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:37.767+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:37.767496+0000) 2022-04-03T23:50:38.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:38.355+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:38.390 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:50:38.388+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:50:38.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:38.527+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:38.769 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:38.767+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:38.767700+0000) 2022-04-03T23:50:39.396 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:39.394+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:39.506 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:39.504+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:39.769 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:39.767+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:39.767891+0000) 2022-04-03T23:50:40.403 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:40.401+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:40.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:40.475+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:40.769 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:40.767+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:40.768090+0000) 2022-04-03T23:50:41.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:41.391+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:41.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:41.438+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:41.769 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:41.768+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:41.768305+0000) 2022-04-03T23:50:42.372 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:42.369+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:42.462 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:42.459+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:42.769 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:42.768+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:42.768498+0000) 2022-04-03T23:50:43.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:43.346+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:43.431 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:43.428+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:43.770 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:43.768+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:43.768684+0000) 2022-04-03T23:50:44.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:44.308+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:44.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:44.422+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:44.770 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:44.768+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:44.768876+0000) 2022-04-03T23:50:45.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:45.348+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:45.421 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:45.418+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:45.770 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:45.768+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:45.769072+0000) 2022-04-03T23:50:46.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:46.377+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:46.396 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:46.393+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:46.770 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:46.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:46.769280+0000) 2022-04-03T23:50:47.356 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:47.353+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:47.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:47.408+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:47.771 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:47.768+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:47.769493+0000) 2022-04-03T23:50:48.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:48.370+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:48.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:48.386+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:48.771 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:48.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:48.769681+0000) 2022-04-03T23:50:49.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:49.337+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:49.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:49.350+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:49.771 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:49.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:49.769889+0000) 2022-04-03T23:50:50.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:50.302+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:50.382 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:50.380+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:50.771 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:50.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:50.770113+0000) 2022-04-03T23:50:51.283 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:51.280+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:51.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:51.349+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:51.771 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:51.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:51.770317+0000) 2022-04-03T23:50:52.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:52.320+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:52.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:52.392+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:52.772 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:52.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:52.770555+0000) 2022-04-03T23:50:52.850 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:50:52.854 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:50:53.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:53.349+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:53.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:53.357+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:53.406 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:50:53.403+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:50:53.772 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:53.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:53.770793+0000) 2022-04-03T23:50:54.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:54.324+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:54.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:54.333+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:54.772 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:54.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:54.770978+0000) 2022-04-03T23:50:55.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:55.337+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:55.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:55.338+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:55.772 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:55.770+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:55.771139+0000) 2022-04-03T23:50:56.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:56.293+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:56.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:56.316+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:56.772 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:56.770+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:56.771330+0000) 2022-04-03T23:50:57.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:57.294+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:57.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:57.308+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:57.773 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:57.770+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:57.771518+0000) 2022-04-03T23:50:58.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:58.252+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:58.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:58.316+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:58.773 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:58.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:58.771739+0000) 2022-04-03T23:50:59.267 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:50:59.264+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:50:59.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:50:59.335+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:50:59.773 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:50:59.771+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:50:59.771949+0000) 2022-04-03T23:51:00.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:00.279+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:00.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:00.289+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:00.773 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:00.771+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:00.772140+0000) 2022-04-03T23:51:01.298 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:01.296+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:01.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:01.318+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:01.773 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:01.771+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:01.772362+0000) 2022-04-03T23:51:02.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:02.274+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:02.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:02.337+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:02.774 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:02.771+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:02.772584+0000) 2022-04-03T23:51:03.267 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:03.265+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:03.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:03.357+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:03.774 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:03.772+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:03.772761+0000) 2022-04-03T23:51:04.312 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:04.309+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:04.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:04.381+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:04.774 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:04.772+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:04.772988+0000) 2022-04-03T23:51:05.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:05.325+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:05.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:05.376+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:05.775 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:05.772+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:05.773155+0000) 2022-04-03T23:51:06.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:06.335+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:06.374 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:06.372+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:06.774 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:06.772+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:06.773306+0000) 2022-04-03T23:51:07.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:07.293+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:07.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:07.381+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:07.774 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:07.772+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:07.773480+0000) 2022-04-03T23:51:08.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:08.308+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:08.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:08.365+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:08.422 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:51:08.420+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:51:08.775 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:08.772+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:08.773680+0000) 2022-04-03T23:51:09.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:09.280+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:09.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:09.328+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:09.775 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:09.773+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:09.773885+0000) 2022-04-03T23:51:10.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:10.308+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:10.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:10.322+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:10.775 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:10.773+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:10.774096+0000) 2022-04-03T23:51:11.303 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:11.301+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:11.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:11.345+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:11.775 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:11.773+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:11.774324+0000) 2022-04-03T23:51:12.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:12.317+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:12.334 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:12.331+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:12.776 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:12.773+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:12.774563+0000) 2022-04-03T23:51:13.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:13.300+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:13.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:13.313+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:13.776 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:13.774+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:13.774751+0000) 2022-04-03T23:51:14.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:14.268+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:14.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:14.290+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:14.776 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:14.774+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:14.774919+0000) 2022-04-03T23:51:15.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:15.296+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:15.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:15.309+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:15.776 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:15.774+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:15.775093+0000) 2022-04-03T23:51:16.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:16.313+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:16.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:16.359+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:16.776 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:16.774+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:16.775315+0000) 2022-04-03T23:51:17.300 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:17.297+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:17.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:17.321+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:17.777 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:17.775+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:17.775535+0000) 2022-04-03T23:51:18.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:18.292+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:18.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:18.327+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:18.777 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:18.775+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:18.775736+0000) 2022-04-03T23:51:19.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:19.296+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:19.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:19.360+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:19.777 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:19.775+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:19.775956+0000) 2022-04-03T23:51:20.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:20.330+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:20.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:20.336+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:20.777 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:20.775+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:20.776174+0000) 2022-04-03T23:51:21.332 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:21.330+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:21.374 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:21.371+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:21.777 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:21.776+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:21.776398+0000) 2022-04-03T23:51:22.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:22.291+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:22.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:22.326+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:22.778 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:22.776+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:22.776631+0000) 2022-04-03T23:51:22.997 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:51:23.000 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:51:23.300 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:23.299+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:23.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:23.326+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:23.438 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:51:23.436+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:51:23.778 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:23.776+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:23.776856+0000) 2022-04-03T23:51:24.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:24.301+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:24.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:24.324+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:24.778 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:24.776+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:24.777062+0000) 2022-04-03T23:51:25.329 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:25.327+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:25.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:25.332+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:25.778 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:25.776+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:25.777282+0000) 2022-04-03T23:51:26.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:26.367+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:26.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:26.378+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:26.779 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:26.777+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:26.777492+0000) 2022-04-03T23:51:27.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:27.406+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:27.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:27.423+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:27.779 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:27.777+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:27.777685+0000) 2022-04-03T23:51:28.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:28.425+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:28.432 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:28.430+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:28.779 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:28.777+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:28.777894+0000) 2022-04-03T23:51:29.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:29.397+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:29.407 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:29.406+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:29.779 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:29.777+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:29.778097+0000) 2022-04-03T23:51:30.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:30.416+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:30.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:30.439+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:30.779 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:30.778+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:30.778321+0000) 2022-04-03T23:51:31.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:31.395+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:31.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:31.454+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:31.780 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:31.778+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:31.778556+0000) 2022-04-03T23:51:32.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:32.396+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:32.480 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:32.478+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:32.780 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:32.778+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:32.778789+0000) 2022-04-03T23:51:33.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:33.353+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:33.524 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:33.523+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:33.780 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:33.778+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:33.778914+0000) 2022-04-03T23:51:34.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:34.392+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:34.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:34.511+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:34.780 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:34.779+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:34.779122+0000) 2022-04-03T23:51:35.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:35.350+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:35.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:35.550+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:35.780 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:35.778+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:35.779326+0000) 2022-04-03T23:51:36.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:36.378+0000 7f88d0ac1700 -1 osd.4 133 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:36.598 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:36.596+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:36.780 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:36.778+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:36.779510+0000) 2022-04-03T23:51:37.227 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-03T23:51:37.224+0000 7f3f3b6d1700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T23:51:37.242 INFO:tasks.ceph.mds.f.smithi186.stderr: -1> 2022-04-03T23:51:37.224+0000 7f3f3b6d1700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-03T23:51:37.265 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-03T23:51:37.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:37.340+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:37.591 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:37.589+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:37.781 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:37.778+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:37.779718+0000) 2022-04-03T23:51:38.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:38.344+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:38.454 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:51:38.451+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:51:38.608 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:38.605+0000 7fd3b190b700 -1 osd.6 133 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:38.781 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:38.778+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:38.779900+0000) 2022-04-03T23:51:39.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:39.296+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:39.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:39.629+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:39.781 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:39.779+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:39.780100+0000) 2022-04-03T23:51:40.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:40.273+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:40.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:40.609+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:40.781 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:40.779+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:40.780298+0000) 2022-04-03T23:51:41.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:41.241+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:41.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:41.620+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:41.782 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:41.779+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:41.780527+0000) 2022-04-03T23:51:42.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:42.242+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 48 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:42.672 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:42.669+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:42.782 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:42.779+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:42.780729+0000) 2022-04-03T23:51:43.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:43.279+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:43.707 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:43.705+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:43.782 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:43.779+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:43.780913+0000) 2022-04-03T23:51:44.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:44.273+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:44.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:44.660+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:44.782 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:44.780+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:44.781132+0000) 2022-04-03T23:51:45.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:45.243+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:45.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:45.666+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:45.782 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:45.780+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:45.781334+0000) 2022-04-03T23:51:46.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:46.250+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:46.681 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:46.678+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:46.782 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:46.780+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:46.781521+0000) 2022-04-03T23:51:47.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:47.292+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:47.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:47.667+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:47.783 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:47.780+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:47.781731+0000) 2022-04-03T23:51:48.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:48.301+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:48.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:48.712+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:48.783 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:48.781+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:48.781970+0000) 2022-04-03T23:51:49.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:49.331+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:49.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:49.684+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:49.783 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:49.781+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:49.782142+0000) 2022-04-03T23:51:50.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:50.287+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:50.703 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:50.701+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:50.783 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:50.781+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:50.782346+0000) 2022-04-03T23:51:51.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:51.331+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:51.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:51.692+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:51.784 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:51.781+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:51.782569+0000) 2022-04-03T23:51:52.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:52.342+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:52.692 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:52.689+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:52.784 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:52.782+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:52.782817+0000) 2022-04-03T23:51:53.135 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:51:53.139 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:51:53.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:53.326+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:53.470 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:51:53.467+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:51:53.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:53.675+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:53.784 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:53.782+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:53.783027+0000) 2022-04-03T23:51:54.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:54.346+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:54.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:54.715+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:54.784 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:54.782+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:54.783167+0000) 2022-04-03T23:51:55.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:55.366+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:55.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:55.666+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:55.784 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:55.782+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:55.783382+0000) 2022-04-03T23:51:56.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:56.403+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:56.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:56.710+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:56.785 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:56.782+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:56.783538+0000) 2022-04-03T23:51:57.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:57.366+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:57.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:57.699+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:57.785 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:57.783+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:57.783725+0000) 2022-04-03T23:51:58.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:58.341+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:58.685 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:58.683+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:58.785 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:58.783+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:58.783934+0000) 2022-04-03T23:51:59.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:51:59.335+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:51:59.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:51:59.638+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:51:59.785 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:51:59.783+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:51:59.784145+0000) 2022-04-03T23:52:00.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:00.304+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:00.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:00.684+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:00.786 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:00.783+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:00.784362+0000) 2022-04-03T23:52:01.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:01.326+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:01.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:01.652+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:01.786 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:01.784+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:01.784591+0000) 2022-04-03T23:52:02.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:02.337+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:02.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:02.662+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:02.786 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:02.784+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:02.784832+0000) 2022-04-03T23:52:03.332 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:03.330+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:03.660 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:03.658+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:03.786 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:03.784+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:03.784989+0000) 2022-04-03T23:52:04.356 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:04.353+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:04.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:04.635+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:04.786 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:04.784+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:04.785200+0000) 2022-04-03T23:52:05.376 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:05.374+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:05.627 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:05.625+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:05.787 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:05.784+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:05.785401+0000) 2022-04-03T23:52:06.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:06.418+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:06.619 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:06.616+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:06.787 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:06.784+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:06.785644+0000) 2022-04-03T23:52:07.438 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:07.435+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:07.651 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:07.649+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:07.787 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:07.785+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:07.785854+0000) 2022-04-03T23:52:08.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:08.430+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:08.486 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:52:08.483+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:52:08.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:08.636+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:08.787 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:08.785+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:08.786061+0000) 2022-04-03T23:52:09.423 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:09.420+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:09.659 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:09.656+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:09.787 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:09.785+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:09.786236+0000) 2022-04-03T23:52:10.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:10.425+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:10.643 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:10.640+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:10.788 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:10.786+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:10.786446+0000) 2022-04-03T23:52:11.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:11.384+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:11.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:11.613+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:11.788 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:11.785+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:11.786675+0000) 2022-04-03T23:52:12.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:12.387+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:12.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:12.591+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:12.788 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:12.786+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:12.786852+0000) 2022-04-03T23:52:13.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:13.405+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:13.578 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:13.576+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:13.788 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:13.786+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:13.787051+0000) 2022-04-03T23:52:14.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:14.369+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:14.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:14.574+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:14.788 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:14.787+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:14.787262+0000) 2022-04-03T23:52:15.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:15.389+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:15.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:15.576+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:15.789 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:15.787+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:15.787489+0000) 2022-04-03T23:52:16.396 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:16.394+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:16.585 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:16.583+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:16.789 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:16.786+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:16.787720+0000) 2022-04-03T23:52:17.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:17.372+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:17.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:17.592+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:17.789 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:17.787+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:17.787930+0000) 2022-04-03T23:52:18.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:18.361+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:18.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:18.586+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:18.789 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:18.788+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:18.788115+0000) 2022-04-03T23:52:19.412 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:19.410+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:19.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:19.570+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:19.789 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:19.787+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:19.788284+0000) 2022-04-03T23:52:20.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:20.434+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:20.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:20.541+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:20.790 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:20.787+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:20.788485+0000) 2022-04-03T23:52:21.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:21.438+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:21.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:21.550+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:21.790 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:21.787+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:21.788685+0000) 2022-04-03T23:52:22.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:22.486+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:22.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:22.508+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:22.790 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:22.787+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:22.788905+0000) 2022-04-03T23:52:23.296 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:52:23.300 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:52:23.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:23.465+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:23.502 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:52:23.498+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:52:23.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:23.536+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:23.790 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:23.788+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:23.789115+0000) 2022-04-03T23:52:24.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:24.457+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:24.498 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:24.496+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:24.790 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:24.788+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:24.789316+0000) 2022-04-03T23:52:25.469 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:25.468+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:25.501 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:25.499+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:25.791 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:25.788+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:25.789537+0000) 2022-04-03T23:52:26.453 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:26.451+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:26.484 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:26.482+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:26.791 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:26.788+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:26.789691+0000) 2022-04-03T23:52:27.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:27.493+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:27.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:27.497+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:27.791 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:27.788+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:27.789900+0000) 2022-04-03T23:52:28.474 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:28.472+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:28.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:28.540+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:28.791 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:28.789+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:28.790114+0000) 2022-04-03T23:52:29.509 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:29.507+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:29.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:29.514+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:29.791 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:29.789+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:29.790307+0000) 2022-04-03T23:52:30.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:30.527+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:30.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:30.529+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:30.792 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:30.789+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:30.790539+0000) 2022-04-03T23:52:31.501 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:31.499+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:31.568 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:31.566+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:31.792 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:31.790+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:31.790756+0000) 2022-04-03T23:52:32.522 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:32.519+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:32.528 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:32.526+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:32.792 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:32.790+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:32.790983+0000) 2022-04-03T23:52:33.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:33.522+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:33.530 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:33.527+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:33.792 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:33.790+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:33.791155+0000) 2022-04-03T23:52:34.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:34.511+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:34.550 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:34.547+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:34.792 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:34.790+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:34.791309+0000) 2022-04-03T23:52:35.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:35.521+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:35.556 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:35.554+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:35.793 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:35.790+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:35.791517+0000) 2022-04-03T23:52:36.500 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:36.497+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:36.558 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:36.555+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:36.793 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:36.791+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:36.791721+0000) 2022-04-03T23:52:37.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:37.518+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:37.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:37.604+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:37.793 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:37.791+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:37.791937+0000) 2022-04-03T23:52:38.518 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:52:38.515+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:52:38.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:38.566+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:38.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:38.573+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:38.793 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:38.791+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:38.792116+0000) 2022-04-03T23:52:39.569 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:39.566+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:39.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:39.577+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:39.793 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:39.791+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:39.792275+0000) 2022-04-03T23:52:40.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:40.573+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:40.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:40.610+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:40.794 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:40.791+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:40.792481+0000) 2022-04-03T23:52:41.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:41.564+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:41.590 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:41.588+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:41.794 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:41.792+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:41.792707+0000) 2022-04-03T23:52:42.612 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:42.609+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:42.617 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:42.614+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:42.794 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:42.792+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:42.792924+0000) 2022-04-03T23:52:43.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:43.602+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:43.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:43.655+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:43.794 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:43.792+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:43.793140+0000) 2022-04-03T23:52:44.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:44.617+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:44.634 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:44.632+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:44.795 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:44.792+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:44.793377+0000) 2022-04-03T23:52:45.621 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:45.619+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:45.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:45.680+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:45.795 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:45.793+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:45.793581+0000) 2022-04-03T23:52:46.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:46.586+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:46.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:46.660+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:46.795 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:46.793+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:46.793822+0000) 2022-04-03T23:52:47.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:47.577+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:47.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:47.676+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:47.795 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:47.793+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:47.794036+0000) 2022-04-03T23:52:48.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:48.538+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:48.650 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:48.647+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:48.795 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:48.793+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:48.794247+0000) 2022-04-03T23:52:49.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:49.579+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:49.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:49.674+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:49.796 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:49.794+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:49.794461+0000) 2022-04-03T23:52:50.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:50.553+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:50.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:50.689+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:50.796 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:50.794+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:50.794702+0000) 2022-04-03T23:52:51.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:51.578+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:51.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:51.727+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:51.796 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:51.794+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:51.794950+0000) 2022-04-03T23:52:52.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:52.534+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:52.773 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:52.771+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:52.796 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:52.794+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:52.795156+0000) 2022-04-03T23:52:53.452 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:52:53.456 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:52:53.521 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:52:53.519+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:52:53.586 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:53.584+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:53.740 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:53.737+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:53.796 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:53.795+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:53.795372+0000) 2022-04-03T23:52:54.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:54.595+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:54.769 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:54.766+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:54.796 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:54.794+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:54.795533+0000) 2022-04-03T23:52:55.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:55.557+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:55.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:55.730+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:55.797 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:55.795+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:55.795747+0000) 2022-04-03T23:52:56.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:56.544+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:56.716 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:56.713+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:56.797 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:56.795+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:56.795942+0000) 2022-04-03T23:52:57.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:57.551+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:57.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:57.700+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:57.797 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:57.795+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:57.796145+0000) 2022-04-03T23:52:58.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:58.567+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:58.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:58.666+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:58.797 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:58.796+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:58.796305+0000) 2022-04-03T23:52:59.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:52:59.535+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:52:59.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:52:59.696+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:52:59.797 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:52:59.796+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:52:59.796510+0000) 2022-04-03T23:53:00.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:00.541+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:00.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:00.687+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:00.798 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:00.796+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:00.796719+0000) 2022-04-03T23:53:01.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:01.518+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:01.707 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:01.704+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:01.798 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:01.796+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:01.796936+0000) 2022-04-03T23:53:02.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:02.483+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:02.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:02.675+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:02.798 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:02.796+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:02.797118+0000) 2022-04-03T23:53:03.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:03.492+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:03.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:03.626+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:03.798 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:03.797+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:03.797285+0000) 2022-04-03T23:53:04.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:04.539+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:04.636 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:04.633+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:04.799 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:04.797+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:04.797490+0000) 2022-04-03T23:53:05.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:05.578+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:05.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:05.619+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:05.799 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:05.796+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:05.797732+0000) 2022-04-03T23:53:06.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:06.594+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:06.622 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:06.620+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:06.799 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:06.796+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:06.797888+0000) 2022-04-03T23:53:07.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:07.613+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:07.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:07.659+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:07.799 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:07.797+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:07.798042+0000) 2022-04-03T23:53:08.537 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:53:08.534+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:53:08.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:08.599+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:08.653 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:08.651+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:08.799 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:08.797+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:08.798247+0000) 2022-04-03T23:53:09.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:09.563+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:09.619 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:09.616+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:09.799 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:09.797+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:09.798452+0000) 2022-04-03T23:53:10.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:10.590+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:10.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:10.642+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:10.800 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:10.797+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:10.798674+0000) 2022-04-03T23:53:11.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:11.595+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:11.646 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:11.644+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:11.800 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:11.797+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:11.798874+0000) 2022-04-03T23:53:12.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:12.599+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:12.684 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:12.682+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:12.800 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:12.798+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:12.799052+0000) 2022-04-03T23:53:13.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:13.590+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:13.701 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:13.699+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:13.800 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:13.798+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:13.799257+0000) 2022-04-03T23:53:14.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:14.625+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:14.697 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:14.695+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:14.801 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:14.798+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:14.799469+0000) 2022-04-03T23:53:15.666 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:15.664+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:15.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:15.677+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:15.801 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:15.798+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:15.799709+0000) 2022-04-03T23:53:16.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:16.641+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:16.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:16.708+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:16.801 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:16.799+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:16.799867+0000) 2022-04-03T23:53:17.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:17.636+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:17.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:17.724+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:17.801 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:17.799+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:17.800052+0000) 2022-04-03T23:53:18.629 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:18.627+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:18.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:18.756+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:18.801 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:18.799+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:18.800220+0000) 2022-04-03T23:53:19.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:19.670+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:19.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:19.742+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:19.801 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:19.799+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:19.800337+0000) 2022-04-03T23:53:20.697 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:20.695+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:20.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:20.708+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:20.802 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:20.799+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:20.800558+0000) 2022-04-03T23:53:21.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:21.661+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:21.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:21.698+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:21.802 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:21.800+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:21.800804+0000) 2022-04-03T23:53:22.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:22.693+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:22.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:22.748+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:22.802 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:22.800+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:22.801030+0000) 2022-04-03T23:53:23.553 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:53:23.550+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:53:23.599 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:53:23.602 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:53:23.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:23.695+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:23.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:23.759+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:23.802 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:23.800+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:23.801259+0000) 2022-04-03T23:53:24.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:24.709+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:24.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:24.713+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:24.802 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:24.800+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:24.801469+0000) 2022-04-03T23:53:25.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:25.663+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:25.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:25.698+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:25.803 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:25.801+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:25.801685+0000) 2022-04-03T23:53:26.708 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:26.706+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:26.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:26.741+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:26.803 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:26.801+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:26.801890+0000) 2022-04-03T23:53:27.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:27.719+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:27.740 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:27.737+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:27.803 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:27.801+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:27.802105+0000) 2022-04-03T23:53:28.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:28.687+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:28.703 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:28.701+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:28.804 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:28.801+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:28.802327+0000) 2022-04-03T23:53:29.720 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:29.717+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:29.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:29.723+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:29.804 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:29.801+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:29.802541+0000) 2022-04-03T23:53:30.689 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:30.686+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:30.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:30.753+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:30.804 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:30.802+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:30.802705+0000) 2022-04-03T23:53:31.656 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:31.653+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:31.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:31.799+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:31.804 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:31.802+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:31.802865+0000) 2022-04-03T23:53:32.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:32.638+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:32.804 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:32.802+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:32.803059+0000) 2022-04-03T23:53:32.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:32.843+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:33.661 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:33.658+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:33.804 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:33.802+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:33.803244+0000) 2022-04-03T23:53:33.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:33.845+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:34.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:34.677+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:34.804 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:34.802+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:34.803436+0000) 2022-04-03T23:53:34.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:34.860+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:35.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:35.694+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:35.805 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:35.803+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:35.803647+0000) 2022-04-03T23:53:35.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:35.857+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:36.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:36.699+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:36.805 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:36.803+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:36.803865+0000) 2022-04-03T23:53:36.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:36.890+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:37.753 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:37.750+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:37.805 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:37.803+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:37.803997+0000) 2022-04-03T23:53:37.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:37.870+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:38.569 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:53:38.566+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:53:38.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:38.779+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:38.805 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:38.803+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:38.804203+0000) 2022-04-03T23:53:38.884 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:38.881+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:39.806 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:39.804+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:39.804428+0000) 2022-04-03T23:53:39.818 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:39.815+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:39.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:39.853+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:40.806 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:40.804+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:40.804638+0000) 2022-04-03T23:53:40.817 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:40.814+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:40.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:40.862+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:41.806 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:41.804+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:41.804880+0000) 2022-04-03T23:53:41.828 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:41.825+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:41.880 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:41.878+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:42.806 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:42.804+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:42.805096+0000) 2022-04-03T23:53:42.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:42.830+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:42.857 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:42.854+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:43.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:43.792+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:43.806 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:43.805+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:43.805315+0000) 2022-04-03T23:53:43.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:43.829+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:44.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:44.745+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:44.807 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:44.805+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:44.805550+0000) 2022-04-03T23:53:44.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:44.805+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:45.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:45.747+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:45.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:45.778+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:45.807 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:45.805+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:45.805749+0000) 2022-04-03T23:53:46.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:46.754+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:46.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:46.791+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:46.807 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:46.805+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:46.805967+0000) 2022-04-03T23:53:47.759 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:47.756+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:47.765 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:47.762+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:47.807 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:47.806+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:47.806188+0000) 2022-04-03T23:53:48.804 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:48.801+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:48.807 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:48.805+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:48.808 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:48.806+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:48.806392+0000) 2022-04-03T23:53:49.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:49.773+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:49.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:49.784+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:49.808 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:49.806+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:49.806608+0000) 2022-04-03T23:53:50.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:50.747+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:50.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:50.765+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:50.808 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:50.805+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:50.806817+0000) 2022-04-03T23:53:51.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:51.709+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:51.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:51.792+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:51.808 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:51.805+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:51.806975+0000) 2022-04-03T23:53:52.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:52.661+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:52.808 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:52.806+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:52.807193+0000) 2022-04-03T23:53:52.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:52.841+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:53.585 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:53:53.581+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:53:53.672 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:53.670+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:53.738 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:53:53.742 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:53:53.808 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:53.806+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:53.807438+0000) 2022-04-03T23:53:53.867 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:53.865+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:54.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:54.629+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:54.809 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:54.805+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:54.807661+0000) 2022-04-03T23:53:54.821 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:54.819+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:55.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:55.592+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:55.809 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:55.806+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:55.807891+0000) 2022-04-03T23:53:55.851 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:55.848+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:56.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:56.613+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:56.807 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:56.805+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:56.809 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:56.807+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:56.808094+0000) 2022-04-03T23:53:57.637 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:57.635+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:57.809 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:57.807+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:57.808298+0000) 2022-04-03T23:53:57.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:57.831+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:58.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:58.654+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:58.801 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:58.799+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:58.809 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:58.807+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:58.808528+0000) 2022-04-03T23:53:59.626 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:53:59.623+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:53:59.792 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:53:59.790+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:53:59.810 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:53:59.807+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:53:59.808715+0000) 2022-04-03T23:54:00.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:00.585+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:00.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:00.748+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:00.810 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:00.808+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:00.808921+0000) 2022-04-03T23:54:01.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:01.610+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:01.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:01.705+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:01.810 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:01.808+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:01.809119+0000) 2022-04-03T23:54:02.615 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:02.613+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:02.687 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:02.684+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:02.810 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:02.808+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:02.809361+0000) 2022-04-03T23:54:03.601 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:03.598+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:03.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:03.686+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:03.811 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:03.808+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:03.809613+0000) 2022-04-03T23:54:04.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:04.600+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:04.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:04.644+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:04.811 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:04.809+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:04.809839+0000) 2022-04-03T23:54:05.596 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:05.594+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:05.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:05.630+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:05.811 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:05.809+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:05.810005+0000) 2022-04-03T23:54:06.563 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:06.561+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:06.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:06.652+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:06.811 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:06.809+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:06.810168+0000) 2022-04-03T23:54:07.597 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:07.595+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:07.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:07.669+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:07.812 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:07.809+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:07.810386+0000) 2022-04-03T23:54:08.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:08.546+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:08.600 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:54:08.598+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:54:08.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:08.686+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:08.812 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:08.809+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:08.810601+0000) 2022-04-03T23:54:09.568 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:09.566+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:09.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:09.658+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:09.812 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:09.810+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:09.810877+0000) 2022-04-03T23:54:10.593 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:10.591+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:10.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:10.644+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:10.812 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:10.810+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:10.811078+0000) 2022-04-03T23:54:11.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:11.593+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:11.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:11.667+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:11.812 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:11.810+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:11.811324+0000) 2022-04-03T23:54:12.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:12.579+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:12.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:12.701+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:12.813 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:12.810+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:12.811527+0000) 2022-04-03T23:54:13.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:13.536+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:13.670 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:13.668+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:13.813 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:13.811+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:13.811759+0000) 2022-04-03T23:54:14.516 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:14.514+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:14.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:14.678+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:14.813 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:14.811+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:14.812014+0000) 2022-04-03T23:54:15.558 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:15.556+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:15.693 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:15.691+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:15.813 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:15.811+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:15.812228+0000) 2022-04-03T23:54:16.559 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:16.557+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:16.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:16.715+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:16.814 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:16.811+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:16.812463+0000) 2022-04-03T23:54:17.586 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:17.584+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:17.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:17.740+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:17.814 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:17.812+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:17.812678+0000) 2022-04-03T23:54:18.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:18.578+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:18.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:18.701+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:18.814 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:18.812+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:18.812883+0000) 2022-04-03T23:54:19.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:19.603+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:19.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:19.715+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:19.814 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:19.812+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:19.813069+0000) 2022-04-03T23:54:20.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:20.642+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:20.710 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:20.708+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:20.814 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:20.812+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:20.813298+0000) 2022-04-03T23:54:21.600 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:21.598+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:21.720 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:21.718+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:21.815 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:21.812+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:21.813546+0000) 2022-04-03T23:54:22.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:22.618+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:22.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:22.678+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:22.815 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:22.813+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:22.813765+0000) 2022-04-03T23:54:23.616 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:54:23.614+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:54:23.626 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:23.623+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:23.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:23.648+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:23.815 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:23.813+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:23.813997+0000) 2022-04-03T23:54:23.894 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:54:23.898 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:54:24.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:24.626+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:24.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:24.676+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:24.815 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:24.813+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:24.814186+0000) 2022-04-03T23:54:25.645 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:25.642+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:25.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:25.666+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:25.815 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:25.813+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:25.814391+0000) 2022-04-03T23:54:26.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:26.659+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:26.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:26.676+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:26.816 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:26.813+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:26.814593+0000) 2022-04-03T23:54:27.619 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:27.616+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:27.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:27.682+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:27.816 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:27.814+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:27.814801+0000) 2022-04-03T23:54:28.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:28.604+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:28.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:28.711+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:28.816 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:28.814+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:28.814990+0000) 2022-04-03T23:54:29.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:29.585+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:29.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:29.733+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:29.816 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:29.814+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:29.815208+0000) 2022-04-03T23:54:30.565 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:30.562+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:30.717 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:30.714+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:30.816 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:30.815+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:30.815394+0000) 2022-04-03T23:54:31.540 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:31.537+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:31.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:31.713+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:31.817 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:31.814+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:31.815630+0000) 2022-04-03T23:54:32.556 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:32.553+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:32.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:32.714+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:32.817 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:32.815+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:32.815867+0000) 2022-04-03T23:54:33.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:33.580+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:33.764 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:33.761+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:33.817 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:33.815+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:33.816058+0000) 2022-04-03T23:54:34.573 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:34.570+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:34.783 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:34.780+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:34.817 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:34.816+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:34.816212+0000) 2022-04-03T23:54:35.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:35.525+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:35.818 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:35.815+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:35.816438+0000) 2022-04-03T23:54:35.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:35.822+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:36.516 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:36.513+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:36.818 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:36.815+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:36.816680+0000) 2022-04-03T23:54:36.867 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:36.865+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:37.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:37.526+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:37.818 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:37.815+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:37.819 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:37.815+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:37.816871+0000) 2022-04-03T23:54:38.558 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:38.555+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:38.632 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:54:38.629+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:54:38.804 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:38.801+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:38.818 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:38.816+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:38.817086+0000) 2022-04-03T23:54:39.598 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:39.595+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:39.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:39.799+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:39.818 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:39.816+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:39.817319+0000) 2022-04-03T23:54:40.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:40.632+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:40.819 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:40.816+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:40.817535+0000) 2022-04-03T23:54:40.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:40.826+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:41.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:41.661+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:41.786 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:41.783+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 43 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:41.819 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:41.815+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:41.817750+0000) 2022-04-03T23:54:42.661 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:42.658+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:42.808 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:42.806+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:42.819 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:42.817+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:42.817996+0000) 2022-04-03T23:54:43.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:43.659+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:43.819 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:43.817+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:43.818212+0000) 2022-04-03T23:54:43.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:43.840+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:44.673 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:44.670+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:44.819 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:44.817+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:44.818423+0000) 2022-04-03T23:54:44.876 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:44.874+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:45.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:45.698+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:45.820 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:45.817+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:45.818635+0000) 2022-04-03T23:54:45.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:45.852+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:46.673 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:46.670+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:46.820 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:46.818+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:46.818851+0000) 2022-04-03T23:54:46.823 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:46.820+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:47.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:47.673+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:47.820 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:47.818+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:47.819071+0000) 2022-04-03T23:54:47.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:47.865+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:48.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:48.688+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:48.820 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:48.818+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:48.819300+0000) 2022-04-03T23:54:48.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:48.879+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:49.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:49.699+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:49.820 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:49.818+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:49.819487+0000) 2022-04-03T23:54:49.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:49.867+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:50.661 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:50.658+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:50.821 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:50.818+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:50.819688+0000) 2022-04-03T23:54:50.841 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:50.838+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:51.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:51.697+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:51.821 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:51.819+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:51.819914+0000) 2022-04-03T23:54:51.885 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:51.882+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:52.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:52.685+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:52.821 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:52.819+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:52.820132+0000) 2022-04-03T23:54:52.934 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:52.931+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:53.643 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:53.641+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:53.648 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:54:53.646+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:54:53.821 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:53.819+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:53.820347+0000) 2022-04-03T23:54:53.947 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:53.944+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:54.010 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:54:54.013 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:54:54.634 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:54.632+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:54.822 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:54.819+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:54.820597+0000) 2022-04-03T23:54:54.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:54.940+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:55.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:55.620+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:55.822 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:55.820+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:55.820847+0000) 2022-04-03T23:54:55.972 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:55.969+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:56.624 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:56.621+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:56.822 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:56.820+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:56.821001+0000) 2022-04-03T23:54:56.970 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:56.968+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:57.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:57.575+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:57.822 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:57.820+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:57.821213+0000) 2022-04-03T23:54:57.988 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:57.986+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:58.606 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:58.604+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:58.822 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:58.820+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:58.821440+0000) 2022-04-03T23:54:59.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:54:59.001+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:54:59.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:54:59.568+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:54:59.823 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:54:59.821+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:54:59.821641+0000) 2022-04-03T23:55:00.043 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:00.041+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:00.601 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:00.598+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:00.823 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:00.821+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:00.821829+0000) 2022-04-03T23:55:01.015 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:01.013+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:01.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:01.621+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:01.823 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:01.821+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:01.822012+0000) 2022-04-03T23:55:02.032 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:02.030+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:02.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:02.660+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:02.823 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:02.821+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:02.822218+0000) 2022-04-03T23:55:03.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:03.053+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:03.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:03.708+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:03.823 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:03.821+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:03.822444+0000) 2022-04-03T23:55:04.061 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:04.059+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:04.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:04.739+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:04.824 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:04.822+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:04.822651+0000) 2022-04-03T23:55:05.047 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:05.045+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:05.756 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:05.754+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:05.824 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:05.822+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:05.822875+0000) 2022-04-03T23:55:06.061 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:06.059+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:06.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:06.799+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:06.824 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:06.822+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:06.823063+0000) 2022-04-03T23:55:07.019 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:07.017+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:07.792 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:07.790+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:07.824 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:07.822+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:07.823273+0000) 2022-04-03T23:55:08.025 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:08.023+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:08.664 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:55:08.661+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:55:08.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:08.799+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:08.824 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:08.822+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:08.823464+0000) 2022-04-03T23:55:09.060 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:09.058+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:09.825 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:09.823+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:09.823696+0000) 2022-04-03T23:55:09.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:09.841+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:10.050 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:10.048+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:10.825 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:10.823+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:10.823938+0000) 2022-04-03T23:55:10.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:10.875+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:11.059 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:11.057+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:11.825 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:11.823+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:11.824104+0000) 2022-04-03T23:55:11.841 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:11.840+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:12.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:12.014+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:12.825 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:12.824+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:12.824312+0000) 2022-04-03T23:55:12.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:12.868+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:12.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:12.994+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:13.826 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:13.824+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:13.824528+0000) 2022-04-03T23:55:13.884 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:13.882+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:14.029 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:14.027+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:14.826 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:14.824+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:14.824745+0000) 2022-04-03T23:55:14.834 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:14.833+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:15.015 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:15.013+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:15.814 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:15.813+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:15.826 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:15.824+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:15.824975+0000) 2022-04-03T23:55:16.046 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:16.044+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:16.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:16.811+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:16.826 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:16.825+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:16.825142+0000) 2022-04-03T23:55:17.062 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:17.060+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:17.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:17.804+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:17.826 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:17.824+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:17.825295+0000) 2022-04-03T23:55:18.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:18.064+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:18.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:18.821+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:18.826 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:18.824+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:18.825479+0000) 2022-04-03T23:55:19.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:19.078+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:19.827 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:19.825+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:19.825649+0000) 2022-04-03T23:55:19.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:19.843+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:20.035 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:20.032+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:20.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:20.803+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:20.827 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:20.825+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:20.825856+0000) 2022-04-03T23:55:20.999 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:20.995+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:21.827 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:21.824+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:21.826017+0000) 2022-04-03T23:55:21.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:21.839+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:21.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:21.993+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:22.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:22.817+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:22.827 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:22.825+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:22.826174+0000) 2022-04-03T23:55:22.989 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:22.986+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:23.679 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:55:23.676+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:55:23.827 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:23.825+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:23.826378+0000) 2022-04-03T23:55:23.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:23.865+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:23.983 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:23.980+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:24.148 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:55:24.152 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:55:24.828 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:24.825+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:24.826620+0000) 2022-04-03T23:55:24.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:24.845+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:25.023 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:25.019+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:25.828 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:25.825+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:25.826833+0000) 2022-04-03T23:55:25.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:25.879+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:26.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:26.053+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:26.828 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:26.826+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:26.826968+0000) 2022-04-03T23:55:26.925 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:26.923+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:27.045 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:27.042+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:27.828 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:27.826+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:27.827120+0000) 2022-04-03T23:55:27.975 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:27.972+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:28.074 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:28.072+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:28.828 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:28.825+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:28.827348+0000) 2022-04-03T23:55:29.000 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:28.997+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:29.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:29.066+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:29.829 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:29.826+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:29.827550+0000) 2022-04-03T23:55:30.045 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:30.042+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:30.078 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:30.075+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:30.829 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:30.826+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:30.827714+0000) 2022-04-03T23:55:31.058 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:31.055+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:31.078 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:31.076+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:31.829 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:31.827+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:31.827924+0000) 2022-04-03T23:55:32.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:32.047+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:32.086 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:32.084+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:32.829 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:32.827+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:32.828149+0000) 2022-04-03T23:55:33.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:33.060+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:33.104 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:33.101+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:33.830 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:33.827+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:33.828318+0000) 2022-04-03T23:55:34.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:34.064+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:34.097 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:34.095+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:34.830 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:34.827+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:34.828479+0000) 2022-04-03T23:55:35.052 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:35.049+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:35.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:35.092+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:35.830 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:35.827+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:35.828641+0000) 2022-04-03T23:55:36.045 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:36.042+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:36.130 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:36.128+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:36.830 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:36.828+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:36.828804+0000) 2022-04-03T23:55:37.054 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:37.051+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:37.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:37.133+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:37.830 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:37.828+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:37.829021+0000) 2022-04-03T23:55:38.091 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:38.088+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:38.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:38.168+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:38.695 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:55:38.693+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:55:38.830 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:38.828+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:38.829247+0000) 2022-04-03T23:55:39.130 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:39.128+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:39.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:39.185+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:39.831 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:39.828+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:39.829469+0000) 2022-04-03T23:55:40.086 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:40.083+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:40.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:40.186+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:40.831 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:40.829+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:40.829696+0000) 2022-04-03T23:55:41.126 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:41.124+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:41.157 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:41.155+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:41.831 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:41.829+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:41.829924+0000) 2022-04-03T23:55:42.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:42.079+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:42.145 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:42.142+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:42.831 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:42.829+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:42.830152+0000) 2022-04-03T23:55:43.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:43.052+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:43.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:43.159+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:43.831 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:43.829+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:43.830311+0000) 2022-04-03T23:55:44.086 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:44.084+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:44.148 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:44.145+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:44.832 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:44.829+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:44.830551+0000) 2022-04-03T23:55:45.040 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:45.038+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:45.142 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:45.140+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:45.832 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:45.830+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:45.830700+0000) 2022-04-03T23:55:46.029 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:46.026+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:46.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:46.124+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:46.832 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:46.830+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:46.830919+0000) 2022-04-03T23:55:46.983 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:46.981+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:47.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:47.154+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:47.832 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:47.830+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:47.831160+0000) 2022-04-03T23:55:47.938 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:47.936+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:48.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:48.119+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:48.833 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:48.830+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:48.831357+0000) 2022-04-03T23:55:48.931 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:48.928+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:49.115 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:49.113+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:49.833 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:49.831+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:49.831552+0000) 2022-04-03T23:55:49.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:49.907+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:50.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:50.131+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:50.833 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:50.831+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:50.831715+0000) 2022-04-03T23:55:50.884 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:50.882+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:51.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:51.124+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:51.833 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:51.831+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:51.831908+0000) 2022-04-03T23:55:51.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:51.907+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:52.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:52.121+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:52.833 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:52.831+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:52.832109+0000) 2022-04-03T23:55:52.919 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:52.916+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:53.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:53.154+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:53.711 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:55:53.709+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:55:53.833 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:53.831+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:53.832322+0000) 2022-04-03T23:55:53.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:53.912+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:54.185 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:54.183+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:54.304 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:55:54.308 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:55:54.834 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:54.832+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:54.832494+0000) 2022-04-03T23:55:54.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:54.880+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:55.213 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:55.210+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:55.834 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:55.831+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:55.832712+0000) 2022-04-03T23:55:55.919 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:55.916+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:56.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:56.173+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:56.834 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:56.832+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:56.832892+0000) 2022-04-03T23:55:56.895 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:56.893+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:57.222 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:57.220+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:57.834 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:57.832+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:57.833091+0000) 2022-04-03T23:55:57.916 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:57.914+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:58.247 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:58.245+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:58.835 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:58.832+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:58.833265+0000) 2022-04-03T23:55:58.886 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:58.884+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:55:59.294 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:55:59.292+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:55:59.835 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:55:59.833+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:55:59.833476+0000) 2022-04-03T23:55:59.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:55:59.870+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:00.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:00.315+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:00.835 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:00.833+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:00.833689+0000) 2022-04-03T23:56:00.838 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:00.836+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:01.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:01.360+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:01.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:01.831+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:01.835 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:01.833+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:01.833910+0000) 2022-04-03T23:56:02.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:02.388+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:02.835 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:02.834+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:02.834159+0000) 2022-04-03T23:56:02.854 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:02.852+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:03.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:03.374+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:03.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:03.831+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:03.835 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:03.834+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:03.834358+0000) 2022-04-03T23:56:04.402 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:04.400+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:04.836 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:04.834+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:04.834557+0000) 2022-04-03T23:56:04.852 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:04.850+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:05.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:05.444+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:05.836 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:05.834+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:05.834716+0000) 2022-04-03T23:56:05.859 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:05.857+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:06.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:06.481+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:06.821 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:06.819+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:06.836 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:06.833+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:06.834904+0000) 2022-04-03T23:56:07.438 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:07.436+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:07.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:07.775+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:07.836 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:07.834+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:07.835099+0000) 2022-04-03T23:56:08.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:08.405+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:08.727 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:56:08.724+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:56:08.765 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:08.763+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:08.836 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:08.834+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:08.835292+0000) 2022-04-03T23:56:09.360 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:09.358+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:09.796 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:09.794+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:09.836 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:09.834+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:09.835506+0000) 2022-04-03T23:56:10.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:10.405+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:10.814 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:10.812+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:10.837 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:10.834+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:10.835714+0000) 2022-04-03T23:56:11.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:11.371+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:11.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:11.830+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:11.837 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:11.835+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:11.835924+0000) 2022-04-03T23:56:12.366 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:12.364+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:12.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:12.790+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:12.837 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:12.835+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:12.836132+0000) 2022-04-03T23:56:13.350 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:13.347+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:13.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:13.755+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:13.837 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:13.835+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:13.836355+0000) 2022-04-03T23:56:14.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:14.365+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:14.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:14.722+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:14.838 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:14.835+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:14.836543+0000) 2022-04-03T23:56:15.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:15.348+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:15.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:15.768+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:15.838 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:15.835+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:15.836763+0000) 2022-04-03T23:56:16.307 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:16.304+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:16.775 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:16.772+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:16.838 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:16.836+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:16.836943+0000) 2022-04-03T23:56:17.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:17.270+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:17.794 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:17.791+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:17.838 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:17.836+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:17.837148+0000) 2022-04-03T23:56:18.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:18.280+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:18.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:18.757+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:18.839 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:18.836+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:18.837299+0000) 2022-04-03T23:56:19.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:19.285+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:19.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:19.722+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:19.839 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:19.836+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:19.837506+0000) 2022-04-03T23:56:20.258 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:20.255+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:20.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:20.730+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:20.839 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:20.836+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:20.837709+0000) 2022-04-03T23:56:21.267 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:21.264+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:21.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:21.729+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:21.839 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:21.837+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:21.837926+0000) 2022-04-03T23:56:22.253 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:22.250+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:22.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:22.710+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:22.839 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:22.837+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:22.838150+0000) 2022-04-03T23:56:23.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:23.294+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:23.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:23.676+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:23.743 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:56:23.741+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:56:23.839 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:23.837+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:23.838357+0000) 2022-04-03T23:56:24.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:24.333+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:24.468 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:56:24.472 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:56:24.676 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:24.673+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:24.840 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:24.837+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:24.838608+0000) 2022-04-03T23:56:25.360 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:25.357+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:25.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:25.655+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:25.840 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:25.838+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:25.838842+0000) 2022-04-03T23:56:26.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:26.336+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:26.623 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:26.620+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:26.840 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:26.838+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:26.839023+0000) 2022-04-03T23:56:27.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:27.294+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:27.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:27.650+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:27.840 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:27.838+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:27.839195+0000) 2022-04-03T23:56:28.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:28.314+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:28.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:28.694+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:28.841 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:28.838+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:28.839402+0000) 2022-04-03T23:56:29.300 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:29.298+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:29.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:29.687+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:29.841 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:29.839+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:29.839617+0000) 2022-04-03T23:56:30.255 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:30.253+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:30.710 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:30.707+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:30.841 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:30.839+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:30.839868+0000) 2022-04-03T23:56:31.251 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:31.248+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:31.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:31.707+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:31.841 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:31.839+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:31.840060+0000) 2022-04-03T23:56:32.214 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:32.211+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:32.673 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:32.670+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:32.841 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:32.839+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:32.840306+0000) 2022-04-03T23:56:33.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:33.181+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:33.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:33.637+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:33.842 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:33.840+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:33.840523+0000) 2022-04-03T23:56:34.157 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:34.155+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:34.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:34.677+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:34.842 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:34.840+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:34.840756+0000) 2022-04-03T23:56:35.185 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:35.183+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:35.662 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:35.659+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:35.842 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:35.840+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:35.841003+0000) 2022-04-03T23:56:36.202 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:36.199+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:36.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:36.696+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:36.843 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:36.840+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:36.841223+0000) 2022-04-03T23:56:37.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:37.247+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:37.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:37.650+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:37.843 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:37.841+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:37.841448+0000) 2022-04-03T23:56:38.214 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:38.211+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:38.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:38.699+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:38.759 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:56:38.756+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:56:38.843 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:38.841+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:38.841668+0000) 2022-04-03T23:56:39.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:39.194+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:39.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:39.720+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:39.843 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:39.841+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:39.841888+0000) 2022-04-03T23:56:40.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:40.161+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:40.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:40.711+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:40.843 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:40.841+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:40.842109+0000) 2022-04-03T23:56:41.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:41.180+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:41.706 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:41.703+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:41.844 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:41.842+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:41.842338+0000) 2022-04-03T23:56:42.142 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:42.139+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:42.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:42.731+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:42.843 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:42.841+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:42.842456+0000) 2022-04-03T23:56:43.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:43.172+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:43.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:43.704+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:43.844 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:43.842+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:43.842655+0000) 2022-04-03T23:56:44.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:44.153+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:44.674 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:44.671+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:44.844 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:44.842+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:44.842872+0000) 2022-04-03T23:56:45.119 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:45.116+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:45.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:45.699+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:45.844 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:45.842+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:45.843071+0000) 2022-04-03T23:56:46.098 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:46.095+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:46.739 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:46.736+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:46.844 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:46.843+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:46.843241+0000) 2022-04-03T23:56:47.089 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:47.087+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:47.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:47.740+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:47.845 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:47.843+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:47.843440+0000) 2022-04-03T23:56:48.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:48.073+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:48.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:48.720+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:48.845 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:48.843+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:48.843654+0000) 2022-04-03T23:56:49.108 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:49.106+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:49.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:49.734+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:49.845 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:49.843+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:49.843883+0000) 2022-04-03T23:56:50.073 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:50.071+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:50.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:50.746+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:50.845 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:50.843+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:50.844087+0000) 2022-04-03T23:56:51.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:51.032+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:51.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:51.713+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:51.845 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:51.843+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:51.844254+0000) 2022-04-03T23:56:51.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:51.986+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:52.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:52.719+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:52.845 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:52.843+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:52.844458+0000) 2022-04-03T23:56:52.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:52.988+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:53.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:53.724+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:53.774 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:56:53.771+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:56:53.846 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:53.843+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:53.844682+0000) 2022-04-03T23:56:54.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:54.037+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:54.757 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:56:54.760 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:56:54.774 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:54.772+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:54.846 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:54.843+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:54.844923+0000) 2022-04-03T23:56:55.001 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:54.999+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:55.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:55.768+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:55.846 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:55.844+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:55.845123+0000) 2022-04-03T23:56:56.018 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:56.016+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:56.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:56.789+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:56.846 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:56.844+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:56.845287+0000) 2022-04-03T23:56:57.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:57.010+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:57.792 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:57.790+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:57.846 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:57.844+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:57.845488+0000) 2022-04-03T23:56:58.000 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:57.998+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:58.759 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:58.757+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:58.847 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:58.844+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:58.845680+0000) 2022-04-03T23:56:58.978 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:56:58.976+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:56:59.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:56:59.804+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:56:59.847 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:56:59.844+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:56:59.845895+0000) 2022-04-03T23:57:00.011 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:00.009+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:00.772 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:00.770+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:00.847 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:00.845+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:00.846091+0000) 2022-04-03T23:57:00.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:00.982+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:01.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:01.771+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:01.847 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:01.845+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:01.846255+0000) 2022-04-03T23:57:01.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:01.935+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:02.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:02.791+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:02.848 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:02.845+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:02.846487+0000) 2022-04-03T23:57:02.932 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:02.930+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:03.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:03.747+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:03.848 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:03.845+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:03.846657+0000) 2022-04-03T23:57:03.967 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:03.965+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:04.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:04.748+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:04.848 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:04.846+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:04.846825+0000) 2022-04-03T23:57:04.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:04.925+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:05.797 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:05.795+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:05.848 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:05.846+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:05.847015+0000) 2022-04-03T23:57:05.909 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:05.907+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:06.797 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:06.795+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:06.849 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:06.846+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:06.847210+0000) 2022-04-03T23:57:06.920 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:06.918+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:07.840 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:07.837+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:07.848 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:07.846+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:07.847431+0000) 2022-04-03T23:57:07.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:07.928+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:08.791 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:57:08.788+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:57:08.839 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:08.836+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:08.849 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:08.846+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:08.847635+0000) 2022-04-03T23:57:08.930 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:08.927+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:09.849 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:09.847+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:09.847835+0000) 2022-04-03T23:57:09.887 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:09.884+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:09.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:09.936+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:10.849 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:10.847+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:10.848013+0000) 2022-04-03T23:57:10.913 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:10.910+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:10.921 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:10.918+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:11.849 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:11.847+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:11.848184+0000) 2022-04-03T23:57:11.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:11.945+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:11.960 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:11.957+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:12.849 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:12.847+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:12.848355+0000) 2022-04-03T23:57:12.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:12.930+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:13.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:13.005+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:13.850 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:13.847+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:13.848560+0000) 2022-04-03T23:57:13.913 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:13.910+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:14.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:14.014+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:14.850 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:14.848+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:14.848716+0000) 2022-04-03T23:57:14.917 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:14.914+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:15.009 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:15.006+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:15.850 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:15.848+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:15.848955+0000) 2022-04-03T23:57:15.936 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:15.933+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:15.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:15.971+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:16.850 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:16.848+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:16.849168+0000) 2022-04-03T23:57:16.949 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:16.946+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:16.972 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:16.969+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:17.850 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:17.848+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:17.849318+0000) 2022-04-03T23:57:17.921 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:17.918+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:18.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:18.009+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:18.851 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:18.849+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:18.849524+0000) 2022-04-03T23:57:18.968 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:18.965+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:19.060 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:19.057+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:19.851 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:19.848+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:19.849710+0000) 2022-04-03T23:57:20.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:20.001+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:20.053 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:20.051+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:20.851 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:20.849+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:20.849915+0000) 2022-04-03T23:57:20.965 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:20.962+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:21.099 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:21.096+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:21.851 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:21.849+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:21.850094+0000) 2022-04-03T23:57:21.965 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:21.963+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:22.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:22.072+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:22.851 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:22.849+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:22.850251+0000) 2022-04-03T23:57:22.919 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:22.916+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:23.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:23.108+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:23.806 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:57:23.804+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:57:23.851 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:23.850+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:23.850512+0000) 2022-04-03T23:57:23.932 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:23.930+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:24.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:24.128+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:24.852 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:24.850+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:24.850673+0000) 2022-04-03T23:57:24.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:24.940+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:24.991 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:57:24.995 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:57:25.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:25.158+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:25.852 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:25.850+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:25.850884+0000) 2022-04-03T23:57:25.993 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:25.990+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:26.116 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:26.113+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:26.852 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:26.850+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:26.851020+0000) 2022-04-03T23:57:27.035 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:27.032+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:27.096 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:27.094+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:27.852 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:27.850+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:27.851215+0000) 2022-04-03T23:57:28.023 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:28.020+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:28.090 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:28.087+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:28.852 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:28.851+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:28.851407+0000) 2022-04-03T23:57:28.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:28.992+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:29.118 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:29.116+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:29.853 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:29.851+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:29.851601+0000) 2022-04-03T23:57:30.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:30.000+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:30.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:30.092+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:30.853 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:30.851+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:30.851835+0000) 2022-04-03T23:57:31.036 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:31.033+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:31.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:31.128+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:31.853 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:31.851+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:31.852039+0000) 2022-04-03T23:57:32.064 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:32.061+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:32.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:32.125+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:32.853 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:32.852+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:32.852261+0000) 2022-04-03T23:57:33.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:33.064+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:33.151 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:33.149+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:33.854 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:33.852+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:33.852480+0000) 2022-04-03T23:57:34.030 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:34.027+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:34.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:34.171+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:34.854 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:34.852+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:34.852715+0000) 2022-04-03T23:57:35.042 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:35.039+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:35.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:35.134+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:35.854 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:35.852+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:35.852954+0000) 2022-04-03T23:57:35.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:35.994+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:36.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:36.085+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:36.854 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:36.852+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:36.853173+0000) 2022-04-03T23:57:37.032 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:37.029+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:37.110 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:37.108+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:37.855 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:37.852+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:37.853402+0000) 2022-04-03T23:57:38.036 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:38.034+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:38.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:38.123+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:38.823 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:57:38.819+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:57:38.855 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:38.852+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:38.853663+0000) 2022-04-03T23:57:39.047 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:39.044+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:39.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:39.129+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:39.855 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:39.852+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:39.853876+0000) 2022-04-03T23:57:40.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:40.070+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:40.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:40.121+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:40.855 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:40.853+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:40.854102+0000) 2022-04-03T23:57:41.107 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:41.104+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:41.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:41.167+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:41.856 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:41.853+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:41.854349+0000) 2022-04-03T23:57:42.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:42.112+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:42.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:42.201+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:42.856 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:42.853+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:42.854569+0000) 2022-04-03T23:57:43.142 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:43.140+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:43.160 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:43.158+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:43.856 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:43.853+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:43.854791+0000) 2022-04-03T23:57:44.104 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:44.101+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:44.160 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:44.157+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:44.856 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:44.853+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:44.854993+0000) 2022-04-03T23:57:45.078 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:45.076+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:45.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:45.145+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:45.856 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:45.854+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:45.855183+0000) 2022-04-03T23:57:46.048 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:46.045+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:46.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:46.164+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:46.857 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:46.854+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:46.855366+0000) 2022-04-03T23:57:47.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:47.083+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:47.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:47.173+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:47.857 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:47.854+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:47.855555+0000) 2022-04-03T23:57:48.123 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:48.121+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:48.186 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:48.185+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:48.857 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:48.855+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:48.855790+0000) 2022-04-03T23:57:49.143 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:49.141+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:49.210 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:49.208+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:49.857 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:49.855+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:49.856023+0000) 2022-04-03T23:57:50.114 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:50.112+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:50.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:50.187+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:50.857 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:50.855+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:50.856246+0000) 2022-04-03T23:57:51.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:51.083+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:51.196 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:51.194+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:51.857 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:51.855+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:51.856460+0000) 2022-04-03T23:57:52.096 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:52.094+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:52.217 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:52.215+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:52.858 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:52.855+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:52.856634+0000) 2022-04-03T23:57:53.084 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:53.082+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:53.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:53.180+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:53.838 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:57:53.836+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:57:53.858 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:53.856+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:53.856865+0000) 2022-04-03T23:57:54.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:54.071+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:54.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:54.217+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:54.858 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:54.856+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:54.857063+0000) 2022-04-03T23:57:55.082 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:55.081+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:55.135 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:57:55.139 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:57:55.263 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:55.261+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:55.858 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:55.856+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:55.857317+0000) 2022-04-03T23:57:56.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:56.071+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:56.222 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:56.220+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:56.859 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:56.856+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:56.857538+0000) 2022-04-03T23:57:57.034 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:57.032+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:57.228 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:57.226+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:57.859 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:57.857+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:57.857700+0000) 2022-04-03T23:57:57.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:57.993+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:58.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:58.270+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:58.859 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:58.857+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:58.857891+0000) 2022-04-03T23:57:59.017 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:57:59.015+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:57:59.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:57:59.279+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:57:59.859 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:57:59.857+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:57:59.858117+0000) 2022-04-03T23:58:00.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:00.001+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:00.256 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:00.254+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:00.859 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:00.857+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:00.858284+0000) 2022-04-03T23:58:01.033 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:01.031+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:01.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:01.244+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:01.859 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:01.857+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:01.858471+0000) 2022-04-03T23:58:01.991 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:01.989+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:02.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:02.275+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:02.860 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:02.858+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:02.858669+0000) 2022-04-03T23:58:03.001 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:02.998+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:03.294 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:03.291+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:03.860 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:03.858+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:03.858890+0000) 2022-04-03T23:58:03.980 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:03.977+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:04.333 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:04.330+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:04.860 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:04.858+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:04.859102+0000) 2022-04-03T23:58:04.974 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:04.972+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:05.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:05.378+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:05.860 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:05.858+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:05.859325+0000) 2022-04-03T23:58:05.983 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:05.980+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:06.331 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:06.328+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:06.860 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:06.859+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:06.859520+0000) 2022-04-03T23:58:07.027 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:07.024+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:07.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:07.312+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:07.861 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:07.859+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:07.859715+0000) 2022-04-03T23:58:08.068 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:08.065+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:08.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:08.349+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:08.854 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:58:08.852+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:58:08.861 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:08.859+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:08.859903+0000) 2022-04-03T23:58:09.069 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:09.066+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:09.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:09.386+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:09.861 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:09.859+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:09.860084+0000) 2022-04-03T23:58:10.108 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:10.105+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:10.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:10.413+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:10.862 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:10.859+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:10.860299+0000) 2022-04-03T23:58:11.135 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:11.131+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:11.446 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:11.443+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:11.862 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:11.860+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:11.860486+0000) 2022-04-03T23:58:12.183 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:12.180+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:12.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:12.486+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:12.862 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:12.860+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:12.860685+0000) 2022-04-03T23:58:13.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:13.227+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:13.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:13.512+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:13.862 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:13.860+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:13.860884+0000) 2022-04-03T23:58:14.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:14.258+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:14.547 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:14.545+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:14.862 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:14.860+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:14.861107+0000) 2022-04-03T23:58:15.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:15.235+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:15.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:15.570+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:15.863 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:15.861+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:15.861326+0000) 2022-04-03T23:58:16.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:16.282+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:16.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:16.536+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:16.863 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:16.861+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:16.861545+0000) 2022-04-03T23:58:17.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:17.310+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:17.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:17.536+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:17.863 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:17.861+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:17.861791+0000) 2022-04-03T23:58:18.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:18.274+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:18.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:18.581+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:18.863 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:18.861+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:18.862027+0000) 2022-04-03T23:58:19.233 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:19.231+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:19.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:19.541+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:19.863 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:19.862+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:19.862251+0000) 2022-04-03T23:58:20.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:20.248+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:20.522 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:20.519+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:20.864 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:20.862+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:20.862477+0000) 2022-04-03T23:58:21.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:21.259+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:21.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:21.542+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:21.864 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:21.861+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:21.862721+0000) 2022-04-03T23:58:22.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:22.241+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:22.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:22.581+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:22.864 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:22.861+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:22.862955+0000) 2022-04-03T23:58:23.233 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:23.230+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:23.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:23.589+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:23.864 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:23.861+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:23.863084+0000) 2022-04-03T23:58:23.870 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:58:23.867+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:58:24.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:24.189+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:24.636 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:24.633+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:24.864 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:24.862+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:24.863281+0000) 2022-04-03T23:58:25.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:25.212+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:25.273 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:58:25.277 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:58:25.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:25.603+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:25.865 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:25.862+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:25.863531+0000) 2022-04-03T23:58:26.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:26.251+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:26.601 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:26.598+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:26.865 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:26.861+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:26.863747+0000) 2022-04-03T23:58:27.212 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:27.209+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:27.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:27.574+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:27.865 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:27.863+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:27.863998+0000) 2022-04-03T23:58:28.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:28.181+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:28.575 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:28.572+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:28.865 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:28.862+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:28.864138+0000) 2022-04-03T23:58:29.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:29.220+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:29.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:29.549+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:29.865 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:29.863+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:29.864302+0000) 2022-04-03T23:58:30.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:30.253+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:30.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:30.515+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:30.866 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:30.863+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:30.864503+0000) 2022-04-03T23:58:31.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:31.288+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:31.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:31.565+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:31.866 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:31.863+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:31.864740+0000) 2022-04-03T23:58:32.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:32.288+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:32.530 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:32.528+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:32.866 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:32.864+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:32.864981+0000) 2022-04-03T23:58:33.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:33.262+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:33.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:33.529+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:33.866 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:33.864+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:33.865160+0000) 2022-04-03T23:58:34.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:34.252+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:34.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:34.481+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:34.866 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:34.864+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:34.865397+0000) 2022-04-03T23:58:35.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:35.224+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:35.457 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:35.454+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:35.867 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:35.864+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:35.865577+0000) 2022-04-03T23:58:36.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:36.217+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:36.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:36.410+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:36.867 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:36.865+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:36.865790+0000) 2022-04-03T23:58:37.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:37.214+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:37.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:37.442+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:37.867 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:37.865+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:37.866002+0000) 2022-04-03T23:58:38.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:38.207+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:38.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:38.401+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:38.867 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:38.865+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:38.866168+0000) 2022-04-03T23:58:38.877 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:58:38.875+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:58:39.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:39.191+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:39.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:39.414+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:39.868 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:39.865+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:39.866359+0000) 2022-04-03T23:58:40.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:40.194+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:40.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:40.433+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:40.868 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:40.865+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:40.866578+0000) 2022-04-03T23:58:41.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:41.214+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:41.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:41.390+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:41.868 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:41.866+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:41.866843+0000) 2022-04-03T23:58:42.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:42.252+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:42.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:42.399+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:42.868 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:42.866+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:42.867028+0000) 2022-04-03T23:58:43.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:43.277+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:43.374 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:43.372+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:43.868 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:43.866+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:43.867197+0000) 2022-04-03T23:58:44.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:44.328+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:44.384 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:44.382+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:44.869 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:44.866+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:44.867413+0000) 2022-04-03T23:58:45.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:45.345+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:45.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:45.368+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:45.873 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:45.867+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:45.867635+0000) 2022-04-03T23:58:46.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:46.327+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:46.379 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:46.378+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:46.869 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:46.867+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:46.867858+0000) 2022-04-03T23:58:47.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:47.355+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:47.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:47.364+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:47.869 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:47.867+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:47.868082+0000) 2022-04-03T23:58:48.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:48.314+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:48.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:48.397+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:48.869 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:48.867+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:48.868270+0000) 2022-04-03T23:58:49.355 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:49.353+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:49.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:49.442+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:49.869 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:49.867+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:49.868404+0000) 2022-04-03T23:58:50.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:50.376+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:50.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:50.393+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:50.870 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:50.868+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:50.868544+0000) 2022-04-03T23:58:51.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:51.412+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:51.432 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:51.430+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:51.870 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:51.867+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:51.868741+0000) 2022-04-03T23:58:52.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:52.416+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:52.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:52.434+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:52.870 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:52.868+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:52.868909+0000) 2022-04-03T23:58:53.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:53.446+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:53.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:53.453+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:53.870 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:53.868+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:53.869048+0000) 2022-04-03T23:58:53.893 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:58:53.891+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:58:54.407 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:54.405+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:54.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:54.462+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:54.870 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:54.868+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:54.869217+0000) 2022-04-03T23:58:55.422 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:58:55.426 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:55.424+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:55.426 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:58:55.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:55.459+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:55.871 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:55.869+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:55.869417+0000) 2022-04-03T23:58:56.424 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:56.422+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:56.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:56.438+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:56.871 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:56.869+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:56.869618+0000) 2022-04-03T23:58:57.405 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:57.403+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:57.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:57.428+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:57.871 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:57.869+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:57.869819+0000) 2022-04-03T23:58:58.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:58.424+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:58.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:58.432+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:58.871 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:58.869+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:58.869969+0000) 2022-04-03T23:58:59.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:58:59.452+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:58:59.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:58:59.456+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:58:59.871 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:58:59.869+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:58:59.870158+0000) 2022-04-03T23:59:00.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:00.403+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:00.432 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:00.429+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:00.872 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:00.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:00.870357+0000) 2022-04-03T23:59:01.424 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:01.421+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:01.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:01.479+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:01.872 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:01.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:01.870602+0000) 2022-04-03T23:59:02.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:02.406+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:02.464 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:02.461+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:02.872 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:02.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:02.870842+0000) 2022-04-03T23:59:03.441 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:03.438+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:03.446 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:03.444+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:03.872 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:03.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:03.871028+0000) 2022-04-03T23:59:04.400 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:04.397+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:04.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:04.417+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:04.872 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:04.871+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:04.871227+0000) 2022-04-03T23:59:05.356 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:05.353+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:05.378 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:05.376+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:05.873 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:05.871+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:05.871426+0000) 2022-04-03T23:59:06.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:06.328+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:06.386 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:06.384+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:06.873 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:06.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:06.871660+0000) 2022-04-03T23:59:07.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:07.372+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:07.428 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:07.425+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:07.873 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:07.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:07.871880+0000) 2022-04-03T23:59:08.387 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:08.384+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:08.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:08.401+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:08.873 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:08.871+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:08.872099+0000) 2022-04-03T23:59:08.909 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:59:08.906+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:59:09.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:09.388+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:09.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:09.441+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:09.873 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:09.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:09.872279+0000) 2022-04-03T23:59:10.365 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:10.362+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:10.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:10.492+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:10.874 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:10.871+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:10.872447+0000) 2022-04-03T23:59:11.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:11.385+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:11.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:11.531+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:11.874 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:11.871+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:11.872659+0000) 2022-04-03T23:59:12.433 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:12.430+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:12.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:12.509+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:12.874 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:12.871+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:12.872879+0000) 2022-04-03T23:59:13.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:13.453+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:13.523 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:13.521+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:13.874 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:13.872+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:13.873073+0000) 2022-04-03T23:59:14.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:14.408+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:14.550 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:14.548+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:14.874 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:14.872+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:14.873267+0000) 2022-04-03T23:59:15.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:15.373+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:15.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:15.568+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:15.875 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:15.872+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:15.873495+0000) 2022-04-03T23:59:16.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:16.388+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:16.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:16.522+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:16.875 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:16.871+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:16.873645+0000) 2022-04-03T23:59:17.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:17.420+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:17.520 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:17.517+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:17.875 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:17.873+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:17.873848+0000) 2022-04-03T23:59:18.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:18.403+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:18.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:18.549+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:18.875 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:18.873+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:18.874068+0000) 2022-04-03T23:59:19.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:19.416+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:19.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:19.533+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:19.875 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:19.873+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:19.874236+0000) 2022-04-03T23:59:20.410 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:20.408+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:20.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:20.522+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:20.876 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:20.873+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:20.874449+0000) 2022-04-03T23:59:21.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:21.406+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:21.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:21.546+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:21.876 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:21.873+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:21.874652+0000) 2022-04-03T23:59:22.438 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:22.436+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:22.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:22.594+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:22.876 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:22.874+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:22.874863+0000) 2022-04-03T23:59:23.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:23.449+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:23.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:23.616+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:23.876 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:23.874+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:23.874995+0000) 2022-04-03T23:59:23.913 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:59:23.910+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:59:24.452 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:24.449+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:24.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:24.641+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:24.876 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:24.874+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:24.875189+0000) 2022-04-03T23:59:25.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:25.460+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:25.550 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:59:25.554 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:59:25.610 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:25.607+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:25.876 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:25.874+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:25.875392+0000) 2022-04-03T23:59:26.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:26.418+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:26.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:26.579+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:26.877 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:26.874+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:26.875593+0000) 2022-04-03T23:59:27.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:27.398+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:27.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:27.576+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:27.877 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:27.875+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:27.875822+0000) 2022-04-03T23:59:28.403 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:28.401+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:28.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:28.576+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:28.877 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:28.875+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:28.876006+0000) 2022-04-03T23:59:29.420 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:29.418+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:29.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:29.621+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:29.877 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:29.875+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:29.876227+0000) 2022-04-03T23:59:30.422 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:30.420+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:30.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:30.591+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:30.877 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:30.875+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:30.876446+0000) 2022-04-03T23:59:31.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:31.453+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:31.603 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:31.600+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:31.878 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:31.876+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:31.876667+0000) 2022-04-03T23:59:32.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:32.415+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:32.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:32.633+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:32.878 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:32.876+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:32.876895+0000) 2022-04-03T23:59:33.397 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:33.395+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:33.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:33.633+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:33.878 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:33.876+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:33.877035+0000) 2022-04-03T23:59:34.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:34.445+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:34.678 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:34.675+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:34.878 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:34.876+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:34.877228+0000) 2022-04-03T23:59:35.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:35.479+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:35.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:35.651+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:35.879 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:35.876+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:35.877429+0000) 2022-04-03T23:59:36.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:36.508+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:36.693 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:36.691+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:36.879 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:36.877+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:36.877630+0000) 2022-04-03T23:59:37.526 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:37.524+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:37.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:37.676+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:37.879 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:37.877+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:37.877878+0000) 2022-04-03T23:59:38.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:38.543+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:38.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:38.641+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:38.879 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:38.877+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:38.878021+0000) 2022-04-03T23:59:38.929 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:59:38.926+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:59:39.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:39.511+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:39.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:39.597+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:39.879 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:39.877+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:39.878225+0000) 2022-04-03T23:59:40.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:40.516+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:40.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:40.589+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:40.880 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:40.878+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:40.878400+0000) 2022-04-03T23:59:41.496 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:41.494+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:41.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:41.557+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:41.880 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:41.878+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:41.878617+0000) 2022-04-03T23:59:42.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:42.457+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:42.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:42.511+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:42.880 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:42.878+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:42.878839+0000) 2022-04-03T23:59:43.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:43.425+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:43.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:43.530+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:43.880 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:43.878+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:43.878988+0000) 2022-04-03T23:59:44.428 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:44.426+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:44.493 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:44.491+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:44.881 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:44.878+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:44.879225+0000) 2022-04-03T23:59:45.465 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:45.462+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:45.500 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:45.498+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:45.880 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:45.879+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:45.879459+0000) 2022-04-03T23:59:46.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:46.494+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:46.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:46.508+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:46.881 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:46.879+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:46.879667+0000) 2022-04-03T23:59:47.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:47.503+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:47.526 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:47.524+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:47.881 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:47.879+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:47.879881+0000) 2022-04-03T23:59:48.474 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:48.472+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:48.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:48.487+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:48.881 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:48.879+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:48.880095+0000) 2022-04-03T23:59:49.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:49.437+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:49.446 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:49.444+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:49.881 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:49.880+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:49.880261+0000) 2022-04-03T23:59:50.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:50.441+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:50.466 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:50.464+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:50.882 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:50.880+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:50.880456+0000) 2022-04-03T23:59:51.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:51.433+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:51.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:51.443+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:51.882 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:51.878+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:51.880715+0000) 2022-04-03T23:59:52.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:52.385+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:52.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:52.479+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:52.882 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:52.879+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:52.880938+0000) 2022-04-03T23:59:53.434 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:53.431+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:53.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:53.508+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:53.882 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:53.880+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:53.881126+0000) 2022-04-03T23:59:53.944 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-03T23:59:53.941+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-03T23:59:54.418 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:54.410+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:54.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:54.526+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:54.883 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:54.880+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:54.881319+0000) 2022-04-03T23:59:55.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:55.374+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:55.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:55.498+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:55.684 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:59:55.689 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-03T23:59:55.883 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:55.880+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:55.881579+0000) 2022-04-03T23:59:56.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:56.370+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:56.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:56.510+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:56.883 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:56.880+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:56.881837+0000) 2022-04-03T23:59:57.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:57.352+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:57.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:57.535+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:57.883 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:57.881+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:57.882064+0000) 2022-04-03T23:59:58.376 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:58.373+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:58.575 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:58.572+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:58.883 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:58.881+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:58.882253+0000) 2022-04-03T23:59:59.396 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-03T23:59:59.393+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-03T23:59:59.540 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-03T23:59:59.538+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-03T23:59:59.884 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-03T23:59:59.881+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T22:59:59.882478+0000) 2022-04-04T00:00:00.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:00.433+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:00.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:00.580+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:00.884 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:00.881+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:00.882663+0000) 2022-04-04T00:00:01.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:01.456+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:01.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:01.610+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:01.884 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:01.882+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:01.882837+0000) 2022-04-04T00:00:02.503 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:02.500+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:02.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:02.581+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:02.884 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:02.882+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:02.883013+0000) 2022-04-04T00:00:03.522 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:03.519+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:03.608 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:03.605+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:03.884 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:03.882+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:03.883204+0000) 2022-04-04T00:00:04.507 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:04.504+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:04.650 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:04.647+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:04.884 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:04.882+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:04.883361+0000) 2022-04-04T00:00:05.498 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:05.495+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:05.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:05.655+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:05.885 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:05.882+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:05.883495+0000) 2022-04-04T00:00:06.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:06.488+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:06.701 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:06.698+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:06.885 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:06.882+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:06.883680+0000) 2022-04-04T00:00:07.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:07.466+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:07.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:07.707+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:07.885 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:07.883+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:07.883879+0000) 2022-04-04T00:00:08.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:08.427+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:08.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:08.714+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:08.885 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:08.883+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:08.884045+0000) 2022-04-04T00:00:08.960 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:00:08.958+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:00:09.421 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:09.417+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:09.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:09.697+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:09.885 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:09.883+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:09.884239+0000) 2022-04-04T00:00:10.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:10.466+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:10.730 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:10.727+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:10.886 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:10.883+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:10.884432+0000) 2022-04-04T00:00:11.453 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:11.450+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:11.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:11.690+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:11.886 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:11.883+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:11.884680+0000) 2022-04-04T00:00:12.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:12.493+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:12.650 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:12.647+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:12.886 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:12.884+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:12.884905+0000) 2022-04-04T00:00:13.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:13.527+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:13.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:13.695+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:13.887 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:13.884+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:13.885454+0000) 2022-04-04T00:00:14.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:14.571+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:14.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:14.666+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:14.887 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:14.885+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:14.885623+0000) 2022-04-04T00:00:15.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:15.571+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:15.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:15.669+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:15.887 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:15.885+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:15.885833+0000) 2022-04-04T00:00:16.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:16.578+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:16.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:16.685+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:16.887 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:16.885+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:16.886043+0000) 2022-04-04T00:00:17.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:17.566+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:17.716 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:17.713+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:17.887 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:17.885+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:17.886231+0000) 2022-04-04T00:00:18.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:18.530+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:18.709 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:18.707+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:18.887 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:18.885+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:18.886425+0000) 2022-04-04T00:00:19.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:19.552+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:19.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:19.676+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:19.888 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:19.886+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:19.886651+0000) 2022-04-04T00:00:20.550 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:20.548+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:20.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:20.710+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:20.888 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:20.886+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:20.886869+0000) 2022-04-04T00:00:21.528 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:21.525+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:21.685 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:21.683+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:21.888 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:21.886+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:21.887071+0000) 2022-04-04T00:00:22.490 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:22.488+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:22.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:22.677+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:22.888 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:22.886+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:22.887272+0000) 2022-04-04T00:00:23.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:23.444+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:23.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:23.691+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:23.889 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:23.887+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:23.887471+0000) 2022-04-04T00:00:23.976 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:00:23.973+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:00:24.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:24.411+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:24.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:24.721+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:24.889 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:24.887+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:24.887701+0000) 2022-04-04T00:00:25.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:25.442+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:25.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:25.755+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:25.828 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:00:25.833 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:00:25.889 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:25.887+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:25.887940+0000) 2022-04-04T00:00:26.405 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:26.402+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:26.730 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:26.727+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:26.889 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:26.887+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:26.888167+0000) 2022-04-04T00:00:27.426 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:27.423+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:27.701 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:27.698+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:27.890 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:27.888+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:27.888398+0000) 2022-04-04T00:00:28.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:28.414+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:28.711 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:28.708+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:28.890 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:28.888+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:28.888620+0000) 2022-04-04T00:00:29.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:29.371+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:29.692 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:29.690+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:29.890 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:29.888+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:29.888846+0000) 2022-04-04T00:00:30.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:30.342+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:30.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:30.735+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:30.890 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:30.888+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:30.889100+0000) 2022-04-04T00:00:31.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:31.384+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:31.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:31.730+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:31.890 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:31.888+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:31.889298+0000) 2022-04-04T00:00:32.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:32.396+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:32.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:32.725+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:32.891 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:32.889+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:32.889461+0000) 2022-04-04T00:00:33.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:33.369+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:33.701 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:33.699+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:33.891 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:33.889+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:33.889650+0000) 2022-04-04T00:00:34.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:34.375+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:34.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:34.730+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:34.891 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:34.889+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:34.889861+0000) 2022-04-04T00:00:35.402 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:35.400+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:35.686 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:35.684+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:35.891 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:35.889+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:35.890079+0000) 2022-04-04T00:00:36.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:36.438+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:36.689 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:36.687+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:36.891 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:36.890+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:36.890293+0000) 2022-04-04T00:00:37.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:37.435+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:37.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:37.693+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:37.892 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:37.889+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:37.890461+0000) 2022-04-04T00:00:38.412 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:38.410+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:38.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:38.714+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:38.892 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:38.889+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:38.890642+0000) 2022-04-04T00:00:38.987 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:00:38.983+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:00:39.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:39.363+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:39.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:39.713+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:39.892 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:39.889+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:39.890851+0000) 2022-04-04T00:00:40.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:40.327+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:40.701 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:40.699+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:40.892 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:40.890+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:40.891069+0000) 2022-04-04T00:00:41.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:41.309+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:41.706 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:41.704+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:41.892 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:41.890+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:41.891307+0000) 2022-04-04T00:00:42.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:42.333+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:42.692 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:42.690+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:42.893 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:42.890+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:42.891529+0000) 2022-04-04T00:00:43.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:43.351+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:43.645 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:43.643+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:43.893 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:43.890+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:43.891749+0000) 2022-04-04T00:00:44.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:44.349+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:44.650 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:44.648+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:44.893 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:44.891+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:44.891998+0000) 2022-04-04T00:00:45.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:45.344+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:45.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:45.669+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:45.893 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:45.891+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:45.892198+0000) 2022-04-04T00:00:46.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:46.317+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:46.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:46.623+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:46.894 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:46.891+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:46.892419+0000) 2022-04-04T00:00:47.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:47.310+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:47.575 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:47.574+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:47.894 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:47.891+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:47.892645+0000) 2022-04-04T00:00:48.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:48.298+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:48.565 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:48.563+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:48.894 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:48.892+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:48.892868+0000) 2022-04-04T00:00:49.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:49.267+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:49.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:49.525+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:49.894 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:49.892+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:49.893109+0000) 2022-04-04T00:00:50.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:50.290+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:50.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:50.518+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:50.895 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:50.892+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:50.893326+0000) 2022-04-04T00:00:51.332 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:51.329+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:51.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:51.478+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:51.895 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:51.892+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:51.893570+0000) 2022-04-04T00:00:52.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:52.330+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:52.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:52.520+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:52.895 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:52.893+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:52.893823+0000) 2022-04-04T00:00:53.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:53.293+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:53.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:53.533+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:53.895 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:53.893+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:53.894043+0000) 2022-04-04T00:00:54.003 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:00:54.000+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:00:54.337 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:54.334+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:54.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:54.557+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:54.899 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:54.893+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:54.894282+0000) 2022-04-04T00:00:55.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:55.333+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:55.527 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:55.524+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:55.896 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:55.893+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:55.894508+0000) 2022-04-04T00:00:55.984 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:00:55.988 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:00:56.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:56.306+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:56.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:56.511+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:56.896 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:56.894+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:56.894751+0000) 2022-04-04T00:00:57.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:57.262+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:57.480 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:57.477+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:57.896 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:57.894+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:57.894956+0000) 2022-04-04T00:00:58.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:58.290+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:58.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:58.464+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:58.896 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:58.894+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:58.895098+0000) 2022-04-04T00:00:59.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:00:59.304+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:00:59.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:00:59.424+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:00:59.896 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:00:59.894+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:00:59.895291+0000) 2022-04-04T00:01:00.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:00.291+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:00.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:00.451+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:00.897 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:00.894+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:00.895511+0000) 2022-04-04T00:01:01.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:01.267+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:01.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:01.410+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:01.897 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:01.895+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:01.895796+0000) 2022-04-04T00:01:02.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:02.250+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:02.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:02.452+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:02.897 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:02.895+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:02.896022+0000) 2022-04-04T00:01:03.252 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:03.249+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:03.422 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:03.420+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:03.897 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:03.895+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:03.896272+0000) 2022-04-04T00:01:04.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:04.252+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:04.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:04.464+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:04.898 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:04.896+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:04.896516+0000) 2022-04-04T00:01:05.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:05.257+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:05.424 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:05.422+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:05.898 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:05.896+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:05.896740+0000) 2022-04-04T00:01:06.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:06.289+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:06.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:06.442+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:06.898 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:06.896+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:06.896906+0000) 2022-04-04T00:01:07.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:07.311+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:07.478 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:07.475+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:07.898 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:07.896+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:07.897107+0000) 2022-04-04T00:01:08.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:08.360+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:08.524 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:08.522+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:08.899 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:08.896+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:08.897326+0000) 2022-04-04T00:01:09.018 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:01:09.016+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:01:09.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:09.338+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:09.564 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:09.561+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:09.899 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:09.897+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:09.897514+0000) 2022-04-04T00:01:10.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:10.312+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:10.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:10.600+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:10.899 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:10.896+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:10.897728+0000) 2022-04-04T00:01:11.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:11.328+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:11.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:11.577+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:11.899 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:11.897+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:11.897961+0000) 2022-04-04T00:01:12.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:12.312+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:12.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:12.532+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:12.899 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:12.897+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:12.898129+0000) 2022-04-04T00:01:13.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:13.323+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:13.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:13.549+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:13.900 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:13.897+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:13.898278+0000) 2022-04-04T00:01:14.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:14.292+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:14.508 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:14.506+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:14.900 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:14.897+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:14.898475+0000) 2022-04-04T00:01:15.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:15.302+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:15.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:15.471+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:15.900 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:15.898+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:15.898692+0000) 2022-04-04T00:01:16.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:16.275+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:16.470 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:16.468+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:16.900 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:16.898+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:16.898903+0000) 2022-04-04T00:01:17.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:17.321+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:17.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:17.456+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:17.900 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:17.898+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:17.899095+0000) 2022-04-04T00:01:18.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:18.298+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:18.422 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:18.419+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:18.901 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:18.899+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:18.899267+0000) 2022-04-04T00:01:19.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:19.302+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:19.392 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:19.389+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:19.901 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:19.898+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:19.899520+0000) 2022-04-04T00:01:20.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:20.315+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:20.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:20.372+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:20.901 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:20.899+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:20.899739+0000) 2022-04-04T00:01:21.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:21.299+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:21.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:21.419+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:21.901 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:21.899+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:21.899933+0000) 2022-04-04T00:01:22.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:22.277+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:22.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:22.377+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:22.901 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:22.899+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:22.900136+0000) 2022-04-04T00:01:23.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:23.269+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:23.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:23.389+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:23.901 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:23.899+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:23.900309+0000) 2022-04-04T00:01:24.034 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:01:24.031+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:01:24.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:24.243+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:24.355 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:24.353+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:24.902 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:24.899+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:24.900529+0000) 2022-04-04T00:01:25.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:25.209+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:25.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:25.365+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:25.902 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:25.899+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:25.900727+0000) 2022-04-04T00:01:26.124 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:01:26.128 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:01:26.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:26.186+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:26.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:26.382+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:26.902 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:26.899+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:26.900953+0000) 2022-04-04T00:01:27.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:27.206+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:27.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:27.412+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:27.902 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:27.900+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:27.901131+0000) 2022-04-04T00:01:28.212 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:28.210+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:28.436 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:28.433+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:28.902 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:28.900+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:28.901343+0000) 2022-04-04T00:01:29.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:29.214+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:29.398 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:29.396+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:29.903 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:29.900+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:29.901565+0000) 2022-04-04T00:01:30.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:30.251+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:30.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:30.351+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:30.903 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:30.900+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:30.901734+0000) 2022-04-04T00:01:31.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:31.262+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:31.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:31.387+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:31.903 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:31.901+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:31.901916+0000) 2022-04-04T00:01:32.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:32.249+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:32.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:32.407+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:32.903 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:32.901+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:32.902121+0000) 2022-04-04T00:01:33.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:33.228+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:33.384 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:33.382+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:33.903 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:33.901+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:33.902322+0000) 2022-04-04T00:01:34.182 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:34.180+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:34.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:34.428+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:34.903 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:34.901+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:34.902495+0000) 2022-04-04T00:01:35.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:35.185+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:35.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:35.477+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:35.904 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:35.901+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:35.902636+0000) 2022-04-04T00:01:36.152 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:36.150+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:36.434 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:36.432+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:36.904 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:36.902+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:36.902835+0000) 2022-04-04T00:01:37.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:37.164+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:37.422 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:37.420+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:37.904 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:37.902+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:37.903040+0000) 2022-04-04T00:01:38.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:38.193+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:38.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:38.391+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:38.904 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:38.902+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:38.903255+0000) 2022-04-04T00:01:39.050 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:01:39.048+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:01:39.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:39.205+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:39.392 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:39.390+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:39.904 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:39.902+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:39.903492+0000) 2022-04-04T00:01:40.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:40.243+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:40.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:40.425+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:40.905 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:40.902+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:40.903664+0000) 2022-04-04T00:01:41.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:41.248+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:41.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:41.404+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:41.905 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:41.903+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:41.903865+0000) 2022-04-04T00:01:42.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:42.202+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:42.392 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:42.390+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:42.905 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:42.903+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:42.904074+0000) 2022-04-04T00:01:43.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:43.199+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:43.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:43.392+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:43.905 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:43.903+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:43.904278+0000) 2022-04-04T00:01:44.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:44.161+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:44.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:44.367+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:44.905 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:44.903+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:44.904473+0000) 2022-04-04T00:01:45.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:45.194+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:45.342 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:45.339+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:45.906 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:45.904+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:45.904655+0000) 2022-04-04T00:01:46.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:46.212+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:46.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:46.350+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:46.906 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:46.904+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:46.904848+0000) 2022-04-04T00:01:47.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:47.216+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:47.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:47.365+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:47.906 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:47.904+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:47.905032+0000) 2022-04-04T00:01:48.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:48.222+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:48.348 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:48.346+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:48.906 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:48.904+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:48.905216+0000) 2022-04-04T00:01:49.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:49.222+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:49.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:49.378+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:49.906 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:49.904+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:49.905393+0000) 2022-04-04T00:01:50.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:50.263+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:50.384 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:50.382+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:50.907 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:50.905+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:50.905608+0000) 2022-04-04T00:01:51.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:51.265+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:51.410 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:51.407+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:51.907 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:51.905+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:51.905831+0000) 2022-04-04T00:01:52.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:52.287+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:52.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:52.402+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:52.907 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:52.905+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:52.905985+0000) 2022-04-04T00:01:53.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:53.289+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:53.364 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:53.361+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:53.907 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:53.905+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:53.906165+0000) 2022-04-04T00:01:54.066 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:01:54.064+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:01:54.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:54.337+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:54.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:54.380+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:54.907 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:54.906+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:54.906373+0000) 2022-04-04T00:01:54.908 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:54.906+0000 7fd623a1a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T00:01:54.909 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:54.906+0000 7fd623a1a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T00:01:54.912 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:54.910+0000 7fd62421b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:54.911202+0000) 2022-04-04T00:01:55.345 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:55.342+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:55.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:55.344+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:55.908 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:55.906+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:55.906524+0000) 2022-04-04T00:01:56.259 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:01:56.262 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:01:56.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:56.311+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:56.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:56.379+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:56.908 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:56.906+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:56.906692+0000) 2022-04-04T00:01:57.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:57.284+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:57.374 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:57.371+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:57.908 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:57.906+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:57.906849+0000) 2022-04-04T00:01:58.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:58.274+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:58.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:58.383+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:58.908 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:58.906+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:58.907022+0000) 2022-04-04T00:01:59.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:01:59.246+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:01:59.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:01:59.382+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:01:59.908 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:01:59.906+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:01:59.907184+0000) 2022-04-04T00:02:00.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:00.241+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:00.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:00.338+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:00.908 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:00.907+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:00.907328+0000) 2022-04-04T00:02:01.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:01.283+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:01.313 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:01.310+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:01.909 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:01.907+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:01.907514+0000) 2022-04-04T00:02:02.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:02.270+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:02.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:02.301+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:02.909 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:02.907+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:02.907660+0000) 2022-04-04T00:02:03.293 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:03.290+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:03.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:03.330+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:03.909 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:03.907+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:03.907804+0000) 2022-04-04T00:02:04.271 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:04.268+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:04.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:04.317+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:04.909 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:04.907+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:04.907969+0000) 2022-04-04T00:02:05.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:05.279+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:05.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:05.311+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:05.909 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:05.908+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:05.908127+0000) 2022-04-04T00:02:06.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:06.285+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:06.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:06.305+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:06.909 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:06.908+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:06.908259+0000) 2022-04-04T00:02:07.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:07.273+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:07.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:07.305+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:07.909 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:07.907+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:07.908402+0000) 2022-04-04T00:02:08.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:08.267+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:08.291 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:08.288+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:08.910 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:08.907+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:08.908586+0000) 2022-04-04T00:02:09.082 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:02:09.079+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:02:09.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:09.292+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:09.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:09.321+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:09.910 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:09.907+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:09.908727+0000) 2022-04-04T00:02:10.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:10.340+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:10.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:10.349+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:10.910 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:10.907+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:10.908929+0000) 2022-04-04T00:02:11.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:11.301+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:11.332 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:11.329+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:11.910 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:11.908+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:11.909090+0000) 2022-04-04T00:02:12.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:12.294+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:12.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:12.337+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:12.910 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:12.907+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:12.909243+0000) 2022-04-04T00:02:13.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:13.293+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:13.298 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:13.296+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:13.911 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:13.907+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:13.909413+0000) 2022-04-04T00:02:14.283 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:14.280+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:14.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:14.320+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:14.911 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:14.908+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:14.909584+0000) 2022-04-04T00:02:15.256 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:15.253+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:15.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:15.299+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:15.911 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:15.908+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:15.909759+0000) 2022-04-04T00:02:16.208 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:16.206+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:16.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:16.313+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:16.911 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:16.909+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:16.909956+0000) 2022-04-04T00:02:17.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:17.195+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:17.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:17.331+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:17.912 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:17.909+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:17.910126+0000) 2022-04-04T00:02:18.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:18.181+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:18.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:18.321+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:18.911 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:18.909+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:18.910269+0000) 2022-04-04T00:02:19.157 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:19.155+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:19.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:19.339+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:19.912 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:19.909+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:19.910414+0000) 2022-04-04T00:02:20.133 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:20.130+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:20.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:20.378+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:20.912 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:20.909+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:20.910611+0000) 2022-04-04T00:02:21.166 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:21.164+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:21.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:21.390+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:21.912 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:21.910+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:21.910833+0000) 2022-04-04T00:02:22.177 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:22.174+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:22.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:22.352+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:22.912 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:22.910+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:22.911001+0000) 2022-04-04T00:02:23.208 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:23.206+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:23.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:23.325+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:23.912 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:23.910+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:23.911176+0000) 2022-04-04T00:02:24.087 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:02:24.084+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:02:24.190 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:24.187+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:24.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:24.310+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:24.912 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:24.910+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:24.911346+0000) 2022-04-04T00:02:25.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:25.196+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:02:25.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:25.314+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:25.913 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:25.910+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:25.911492+0000) 2022-04-04T00:02:26.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:26.242+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:26.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:26.356+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:26.373 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:02:26.377 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:02:26.913 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:26.911+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:26.911676+0000) 2022-04-04T00:02:27.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:27.244+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:27.375 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:27.373+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:27.913 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:27.911+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:27.911861+0000) 2022-04-04T00:02:28.275 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:28.273+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:28.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:28.367+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:28.913 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:28.911+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:28.912039+0000) 2022-04-04T00:02:29.256 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:29.254+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:29.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:29.357+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:29.913 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:29.911+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:29.912211+0000) 2022-04-04T00:02:30.222 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:30.220+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:30.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:30.385+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:30.914 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:30.911+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:30.912377+0000) 2022-04-04T00:02:31.241 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:31.239+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:31.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:31.379+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:31.914 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:31.912+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:31.912546+0000) 2022-04-04T00:02:32.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:32.280+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:32.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:32.404+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:32.914 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:32.912+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:32.912720+0000) 2022-04-04T00:02:33.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:33.321+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:33.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:33.377+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:33.914 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:33.912+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:33.912907+0000) 2022-04-04T00:02:34.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:34.339+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:34.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:34.357+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:34.914 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:34.912+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:34.913074+0000) 2022-04-04T00:02:35.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:35.295+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:35.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:35.391+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:35.914 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:35.912+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:35.913248+0000) 2022-04-04T00:02:36.348 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:36.346+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:36.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:36.439+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:36.914 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:36.913+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:36.913395+0000) 2022-04-04T00:02:37.348 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:37.346+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:37.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:37.412+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:37.915 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:37.913+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:37.913570+0000) 2022-04-04T00:02:38.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:38.371+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:38.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:38.391+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:38.915 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:38.913+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:38.913742+0000) 2022-04-04T00:02:39.102 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:02:39.100+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:02:39.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:39.352+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:39.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:39.361+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:39.915 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:39.913+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:39.913889+0000) 2022-04-04T00:02:40.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:40.323+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:40.400 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:40.397+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:40.915 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:40.913+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:40.914032+0000) 2022-04-04T00:02:41.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:41.330+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:41.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:41.441+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:41.915 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:41.913+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:41.914236+0000) 2022-04-04T00:02:42.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:42.309+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:42.449 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:42.446+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:42.916 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:42.914+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:42.914382+0000) 2022-04-04T00:02:43.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:43.274+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:43.434 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:43.431+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:43.916 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:43.914+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:43.914578+0000) 2022-04-04T00:02:44.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:44.244+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:44.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:44.408+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:44.916 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:44.914+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:44.914757+0000) 2022-04-04T00:02:45.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:45.271+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:45.446 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:45.443+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:45.916 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:45.914+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:45.914905+0000) 2022-04-04T00:02:46.271 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:46.268+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:46.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:46.480+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:46.916 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:46.914+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:46.915058+0000) 2022-04-04T00:02:47.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:47.235+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:47.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:47.452+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:47.916 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:47.915+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:47.915244+0000) 2022-04-04T00:02:48.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:48.217+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:48.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:48.494+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:48.916 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:48.915+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:48.915413+0000) 2022-04-04T00:02:49.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:49.222+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:49.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:49.538+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:49.917 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:49.915+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:49.915565+0000) 2022-04-04T00:02:50.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:50.258+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:50.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:50.530+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:50.917 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:50.915+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:50.915744+0000) 2022-04-04T00:02:51.301 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:51.298+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:51.533 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:51.530+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:51.917 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:51.915+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:51.915962+0000) 2022-04-04T00:02:52.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:52.305+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:52.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:52.491+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:52.917 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:52.915+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:52.916151+0000) 2022-04-04T00:02:53.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:53.303+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:53.517 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:53.515+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:53.917 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:53.915+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:53.916275+0000) 2022-04-04T00:02:54.118 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:02:54.115+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:02:54.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:54.292+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:54.516 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:54.513+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:54.917 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:54.915+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:54.916435+0000) 2022-04-04T00:02:55.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:55.317+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:55.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:55.543+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:55.918 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:55.915+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:55.916560+0000) 2022-04-04T00:02:56.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:56.337+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:56.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:56.507+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:56.511 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:02:56.515 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:02:56.918 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:56.915+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:56.916721+0000) 2022-04-04T00:02:57.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:57.299+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:57.466 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:57.463+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:57.918 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:57.915+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:57.916879+0000) 2022-04-04T00:02:58.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:58.258+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:58.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:58.457+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:58.918 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:58.915+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:58.917017+0000) 2022-04-04T00:02:59.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:02:59.256+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:02:59.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:02:59.482+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:02:59.918 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:02:59.916+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:02:59.917190+0000) 2022-04-04T00:03:00.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:00.250+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:00.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:00.492+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:00.919 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:00.916+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:00.917371+0000) 2022-04-04T00:03:01.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:01.242+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:01.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:01.483+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:01.919 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:01.916+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:01.917567+0000) 2022-04-04T00:03:02.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:02.270+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:02.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:02.518+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:02.919 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:02.917+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:02.917793+0000) 2022-04-04T00:03:03.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:03.316+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:03.516 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:03.514+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:03.920 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:03.917+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:03.918006+0000) 2022-04-04T00:03:04.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:04.331+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:04.522 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:04.519+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:04.919 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:04.917+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:04.918178+0000) 2022-04-04T00:03:05.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:05.305+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:05.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:05.550+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:05.920 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:05.917+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:05.918372+0000) 2022-04-04T00:03:06.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:06.302+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:06.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:06.579+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:06.920 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:06.917+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:06.918524+0000) 2022-04-04T00:03:07.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:07.296+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:07.618 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:07.615+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:07.920 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:07.918+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:07.918710+0000) 2022-04-04T00:03:08.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:08.318+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:08.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:08.622+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:08.920 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:08.918+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:08.918892+0000) 2022-04-04T00:03:09.134 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:03:09.132+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:03:09.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:09.293+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:09.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:09.664+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:09.920 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:09.918+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:09.919069+0000) 2022-04-04T00:03:10.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:10.264+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:10.630 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:10.628+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:10.920 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:10.918+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:10.919221+0000) 2022-04-04T00:03:11.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:11.262+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:11.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:11.579+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:11.921 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:11.918+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:11.919403+0000) 2022-04-04T00:03:12.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:12.225+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:12.619 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:12.616+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:12.921 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:12.918+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:12.919560+0000) 2022-04-04T00:03:13.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:13.193+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:13.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:13.601+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:13.921 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:13.919+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:13.919748+0000) 2022-04-04T00:03:14.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:14.238+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:14.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:14.632+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:14.921 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:14.919+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:14.919955+0000) 2022-04-04T00:03:15.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:15.220+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:15.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:15.629+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:15.921 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:15.919+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:15.920110+0000) 2022-04-04T00:03:16.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:16.259+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:16.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:16.626+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:16.921 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:16.919+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:16.920244+0000) 2022-04-04T00:03:17.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:17.219+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:17.622 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:17.619+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:17.921 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:17.919+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:17.920403+0000) 2022-04-04T00:03:18.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:18.221+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:18.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:18.646+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:18.922 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:18.920+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:18.920530+0000) 2022-04-04T00:03:19.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:19.244+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:19.622 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:19.619+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:19.922 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:19.920+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:19.920710+0000) 2022-04-04T00:03:20.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:20.205+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:20.650 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:20.648+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:20.922 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:20.920+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:20.920887+0000) 2022-04-04T00:03:21.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:21.239+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:21.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:21.605+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:21.922 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:21.920+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:21.921043+0000) 2022-04-04T00:03:22.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:22.213+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:22.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:22.631+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:22.922 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:22.920+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:22.921189+0000) 2022-04-04T00:03:23.235 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:23.233+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:23.645 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:23.643+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:23.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:23.920+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:23.921337+0000) 2022-04-04T00:03:24.150 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:03:24.148+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:03:24.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:24.208+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:24.645 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:24.643+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:24.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:24.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:24.921514+0000) 2022-04-04T00:03:25.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:25.240+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:25.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:25.663+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:25.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:25.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:25.921695+0000) 2022-04-04T00:03:26.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:26.240+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:26.643 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:26.641+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:26.651 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:03:26.655 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:03:26.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:26.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:26.921899+0000) 2022-04-04T00:03:27.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:27.201+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:27.661 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:27.659+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:27.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:27.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:27.922096+0000) 2022-04-04T00:03:28.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:28.197+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:28.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:28.610+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:28.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:28.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:28.922230+0000) 2022-04-04T00:03:29.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:29.173+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:29.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:29.623+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:29.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:29.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:29.922366+0000) 2022-04-04T00:03:30.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:30.167+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:30.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:30.621+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:30.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:30.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:30.922508+0000) 2022-04-04T00:03:31.174 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:31.172+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:31.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:31.576+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:31.924 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:31.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:31.922691+0000) 2022-04-04T00:03:32.158 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:32.156+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:32.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:32.542+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:32.924 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:32.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:32.922859+0000) 2022-04-04T00:03:33.130 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:33.129+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:33.525 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:33.523+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:33.924 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:33.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:33.922993+0000) 2022-04-04T00:03:34.138 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:34.136+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:34.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:34.483+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:34.924 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:34.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:34.923137+0000) 2022-04-04T00:03:35.121 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:35.119+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:35.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:35.499+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:35.924 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:35.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:35.923295+0000) 2022-04-04T00:03:36.141 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:36.138+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:36.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:36.548+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:36.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:36.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:36.923480+0000) 2022-04-04T00:03:37.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:37.146+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:37.519 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:37.516+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:37.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:37.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:37.923645+0000) 2022-04-04T00:03:38.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:38.190+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:38.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:38.478+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:38.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:38.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:38.923810+0000) 2022-04-04T00:03:39.166 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:03:39.162+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:03:39.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:39.189+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:39.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:39.527+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:39.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:39.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:39.923963+0000) 2022-04-04T00:03:40.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:40.189+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:40.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:40.570+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:40.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:40.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:40.924131+0000) 2022-04-04T00:03:41.160 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:41.157+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:41.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:41.533+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:41.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:41.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:41.924304+0000) 2022-04-04T00:03:42.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:42.156+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:42.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:42.533+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:42.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:42.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:42.924433+0000) 2022-04-04T00:03:43.126 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:43.123+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:43.487 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:43.484+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:43.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:43.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:43.924551+0000) 2022-04-04T00:03:44.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:44.102+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:44.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:44.509+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:44.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:44.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:44.924692+0000) 2022-04-04T00:03:45.125 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:45.122+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:45.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:45.518+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:45.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:45.924+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:45.924893+0000) 2022-04-04T00:03:46.103 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:46.100+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:46.505 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:46.502+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:46.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:46.924+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:46.925069+0000) 2022-04-04T00:03:47.088 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:47.086+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:47.473 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:47.470+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:47.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:47.924+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:47.925244+0000) 2022-04-04T00:03:48.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:48.109+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:48.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:48.507+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:48.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:48.924+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:48.925413+0000) 2022-04-04T00:03:49.092 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:49.090+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:49.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:49.464+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:49.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:49.924+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:49.925575+0000) 2022-04-04T00:03:50.095 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:50.092+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:50.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:50.509+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:50.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:50.924+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:50.925736+0000) 2022-04-04T00:03:51.056 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:51.053+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:51.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:51.533+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:51.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:51.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:51.925902+0000) 2022-04-04T00:03:52.008 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:52.005+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:52.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:52.485+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:52.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:52.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:52.926029+0000) 2022-04-04T00:03:52.971 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:52.968+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:53.499 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:53.496+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:53.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:53.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:53.926231+0000) 2022-04-04T00:03:53.928 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:53.926+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:54.181 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:03:54.179+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:03:54.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:54.545+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:54.917 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:54.914+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:54.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:54.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:54.926402+0000) 2022-04-04T00:03:55.507 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:55.504+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:55.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:55.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:55.926546+0000) 2022-04-04T00:03:55.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:55.942+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:56.465 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:56.462+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:56.808 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:03:56.811 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:03:56.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:56.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:56.926689+0000) 2022-04-04T00:03:56.969 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:56.966+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:57.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:57.416+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:57.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:57.926+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:57.926860+0000) 2022-04-04T00:03:57.983 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:57.981+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:58.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:58.462+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:58.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:58.926+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:58.927004+0000) 2022-04-04T00:03:58.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:58.940+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:03:59.473 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:03:59.471+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:03:59.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:03:59.926+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:03:59.927180+0000) 2022-04-04T00:03:59.929 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:03:59.926+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:00.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:00.427+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:00.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:00.926+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:00.927358+0000) 2022-04-04T00:04:00.962 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:00.930+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:01.446 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:01.444+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:01.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:01.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:01.927520+0000) 2022-04-04T00:04:01.958 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:01.956+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:02.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:02.452+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:02.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:02.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:02.927663+0000) 2022-04-04T00:04:02.950 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:02.947+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:03.448 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:03.446+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:03.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:03.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:03.927843+0000) 2022-04-04T00:04:03.931 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:03.928+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:04.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:04.455+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:04.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:04.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:04.928019+0000) 2022-04-04T00:04:04.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:04.976+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:05.453 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:05.451+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:05.930 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:05.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:05.928194+0000) 2022-04-04T00:04:06.023 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:06.020+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:06.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:06.488+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:06.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:06.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:06.928367+0000) 2022-04-04T00:04:07.005 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:07.002+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:07.473 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:07.470+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:07.930 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:07.928+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:07.928543+0000) 2022-04-04T00:04:08.011 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:08.009+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:08.507 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:08.504+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:08.930 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:08.928+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:08.928715+0000) 2022-04-04T00:04:09.014 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:09.011+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:09.197 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:04:09.195+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:04:09.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:09.533+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:09.930 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:09.928+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:09.928836+0000) 2022-04-04T00:04:10.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:10.053+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:10.530 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:10.527+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:10.930 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:10.928+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:10.928977+0000) 2022-04-04T00:04:11.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:11.013+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:11.578 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:11.575+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:11.930 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:11.928+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:11.929119+0000) 2022-04-04T00:04:11.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:11.993+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:12.600 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:12.598+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:12.930 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:12.928+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:12.929294+0000) 2022-04-04T00:04:13.033 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:13.030+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:13.557 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:13.555+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:13.931 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:13.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:13.929472+0000) 2022-04-04T00:04:13.988 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:13.986+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:14.526 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:14.524+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:14.931 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:14.928+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:14.929680+0000) 2022-04-04T00:04:14.941 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:14.939+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:15.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:15.491+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:15.919 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:15.917+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:15.931 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:15.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:15.929859+0000) 2022-04-04T00:04:16.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:16.532+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:16.888 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:16.886+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:16.931 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:16.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:16.930009+0000) 2022-04-04T00:04:17.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:17.492+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:17.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:17.913+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:17.931 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:17.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:17.930190+0000) 2022-04-04T00:04:18.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:18.462+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:18.912 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:18.910+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:18.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:18.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:18.930367+0000) 2022-04-04T00:04:19.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:19.483+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:19.879 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:19.877+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:19.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:19.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:19.930546+0000) 2022-04-04T00:04:20.452 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:20.450+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:20.907 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:20.904+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:20.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:20.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:20.930696+0000) 2022-04-04T00:04:21.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:21.402+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:21.906 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:21.904+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:21.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:21.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:21.930889+0000) 2022-04-04T00:04:22.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:22.419+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:22.860 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:22.859+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:22.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:22.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:22.931070+0000) 2022-04-04T00:04:23.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:23.412+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:23.910 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:23.908+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:23.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:23.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:23.931254+0000) 2022-04-04T00:04:24.213 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:04:24.210+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:04:24.412 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:24.410+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:24.887 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:24.885+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:24.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:24.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:24.931408+0000) 2022-04-04T00:04:25.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:25.442+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:25.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:25.896+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:25.933 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:25.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:25.931559+0000) 2022-04-04T00:04:26.396 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:26.394+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:26.928 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:26.926+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:26.930 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:04:26.934 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:04:26.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:26.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:26.931679+0000) 2022-04-04T00:04:27.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:27.383+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:27.933 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:27.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:27.931858+0000) 2022-04-04T00:04:27.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:27.941+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:28.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:28.373+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:28.933 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:28.931+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:28.932063+0000) 2022-04-04T00:04:28.977 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:28.975+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:29.364 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:29.362+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:29.934 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:29.931+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:29.932265+0000) 2022-04-04T00:04:29.986 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:29.984+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:30.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:30.378+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:30.933 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:30.931+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:30.932449+0000) 2022-04-04T00:04:30.940 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:30.938+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:31.366 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:31.365+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:31.923 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:31.920+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:31.934 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:31.931+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:31.932607+0000) 2022-04-04T00:04:32.359 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:32.356+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:32.934 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:32.932+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:32.932820+0000) 2022-04-04T00:04:32.935 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:32.932+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:33.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:33.366+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:33.934 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:33.932+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:33.933008+0000) 2022-04-04T00:04:33.939 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:33.937+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:34.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:34.354+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:34.934 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:34.932+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:34.933164+0000) 2022-04-04T00:04:34.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:34.977+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:35.396 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:35.393+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:35.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:35.932+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:35.933345+0000) 2022-04-04T00:04:36.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:36.009+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:36.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:36.388+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:36.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:36.932+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:36.933523+0000) 2022-04-04T00:04:36.998 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:36.995+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:37.403 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:37.400+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:37.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:37.932+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:37.933674+0000) 2022-04-04T00:04:38.008 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:38.005+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:38.433 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:38.430+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:38.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:38.933+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:38.933856+0000) 2022-04-04T00:04:39.034 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:39.031+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:39.229 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:04:39.227+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:04:39.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:39.427+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:39.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:39.933+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:39.934046+0000) 2022-04-04T00:04:40.026 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:40.023+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:40.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:40.378+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:40.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:40.933+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:40.934231+0000) 2022-04-04T00:04:41.005 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:41.002+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:41.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:41.373+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:41.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:41.933+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:41.934431+0000) 2022-04-04T00:04:41.966 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:41.964+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:42.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:42.366+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:42.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:42.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:42.934615+0000) 2022-04-04T00:04:42.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:42.992+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:43.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:43.404+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:43.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:43.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:43.934785+0000) 2022-04-04T00:04:43.974 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:43.971+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:44.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:44.359+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:44.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:44.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:44.934980+0000) 2022-04-04T00:04:44.958 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:44.955+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:45.348 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:45.345+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:45.929 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:45.926+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:45.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:45.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:45.935171+0000) 2022-04-04T00:04:46.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:46.372+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:46.918 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:46.915+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:46.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:46.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:46.935330+0000) 2022-04-04T00:04:47.350 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:47.348+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:47.918 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:47.915+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:47.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:47.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:47.935516+0000) 2022-04-04T00:04:48.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:48.396+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:48.931 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:48.928+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:48.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:48.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:48.935698+0000) 2022-04-04T00:04:49.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:49.432+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:49.890 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:49.887+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:49.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:49.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:49.935873+0000) 2022-04-04T00:04:50.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:50.402+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:50.866 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:50.863+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:50.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:50.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:50.936015+0000) 2022-04-04T00:04:51.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:51.413+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:51.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:51.866+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:51.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:51.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:51.936178+0000) 2022-04-04T00:04:52.395 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:52.392+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:52.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:52.859+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:52.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:52.936+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:52.936359+0000) 2022-04-04T00:04:53.346 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:53.343+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:53.871 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:53.868+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:53.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:53.936+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:53.936512+0000) 2022-04-04T00:04:54.245 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:04:54.243+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:04:54.338 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:54.336+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:54.841 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:54.839+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:54.938 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:54.936+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:54.936694+0000) 2022-04-04T00:04:55.372 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:55.370+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:55.819 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:55.817+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:55.938 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:55.936+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:55.936897+0000) 2022-04-04T00:04:56.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:56.378+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:56.846 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:56.844+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:56.938 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:56.936+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:56.937096+0000) 2022-04-04T00:04:57.077 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:04:57.081 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:04:57.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:57.360+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:57.830 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:57.827+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:57.938 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:57.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:57.937327+0000) 2022-04-04T00:04:58.331 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:58.328+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:58.845 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:58.842+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:58.938 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:58.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:58.937477+0000) 2022-04-04T00:04:59.283 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:04:59.281+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:04:59.884 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:04:59.882+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:04:59.938 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:04:59.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:04:59.937624+0000) 2022-04-04T00:05:00.303 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:00.301+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:00.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:00.875+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:00.939 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:00.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:00.937824+0000) 2022-04-04T00:05:01.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:01.338+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:01.836 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:01.833+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:01.939 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:01.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:01.937975+0000) 2022-04-04T00:05:02.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:02.312+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:02.884 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:02.881+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:02.939 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:02.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:02.938174+0000) 2022-04-04T00:05:03.359 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:03.357+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:03.884 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:03.881+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:03.939 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:03.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:03.938380+0000) 2022-04-04T00:05:04.319 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:04.316+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:04.880 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:04.878+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:04.939 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:04.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:04.938578+0000) 2022-04-04T00:05:05.298 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:05.295+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:05.857 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:05.855+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:05.940 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:05.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:05.938808+0000) 2022-04-04T00:05:06.338 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:06.335+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:06.899 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:06.897+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:06.940 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:06.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:06.939023+0000) 2022-04-04T00:05:07.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:07.326+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:07.867 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:07.864+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:07.940 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:07.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:07.939215+0000) 2022-04-04T00:05:08.332 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:08.330+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:08.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:08.820+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:08.940 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:08.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:08.939436+0000) 2022-04-04T00:05:09.256 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:05:09.252+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:05:09.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:09.347+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:09.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:09.820+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:09.941 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:09.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:09.939636+0000) 2022-04-04T00:05:10.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:10.388+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:10.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:10.822+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:10.941 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:10.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:10.939855+0000) 2022-04-04T00:05:11.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:11.378+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:11.786 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:11.784+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:11.941 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:11.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:11.940064+0000) 2022-04-04T00:05:12.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:12.380+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:12.835 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:12.832+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:12.942 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:12.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:12.940262+0000) 2022-04-04T00:05:13.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:13.376+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:13.831 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:13.829+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:13.941 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:13.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:13.940491+0000) 2022-04-04T00:05:14.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:14.351+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:14.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:14.880+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:14.942 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:14.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:14.940677+0000) 2022-04-04T00:05:15.333 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:15.331+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:15.903 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:15.901+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:15.942 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:15.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:15.940887+0000) 2022-04-04T00:05:16.304 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:16.302+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:16.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:16.853+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:16.942 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:16.940+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:16.941065+0000) 2022-04-04T00:05:17.270 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:17.268+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:17.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:17.817+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:17.942 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:17.940+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:17.941243+0000) 2022-04-04T00:05:18.253 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:18.251+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:18.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:18.809+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:18.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:18.940+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:18.941448+0000) 2022-04-04T00:05:19.206 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:19.205+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:19.830 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:19.828+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:19.942 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:19.940+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:19.941665+0000) 2022-04-04T00:05:20.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:20.217+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:20.849 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:20.847+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:20.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:20.941+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:20.941903+0000) 2022-04-04T00:05:21.216 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:21.214+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:21.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:21.841+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:21.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:21.941+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:21.942116+0000) 2022-04-04T00:05:22.262 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:22.260+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:22.801 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:22.799+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:22.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:22.941+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:22.942318+0000) 2022-04-04T00:05:23.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:23.252+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:23.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:23.778+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:23.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:23.941+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:23.942541+0000) 2022-04-04T00:05:24.204 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:24.203+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:24.271 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:05:24.269+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:05:24.827 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:24.825+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:24.944 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:24.942+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:24.942689+0000) 2022-04-04T00:05:25.212 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:25.210+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:05:25.875 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:25.874+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:25.944 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:25.942+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:25.942839+0000) 2022-04-04T00:05:26.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:26.225+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:26.838 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:26.837+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:26.944 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:26.942+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:26.943042+0000) 2022-04-04T00:05:27.175 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:05:27.179 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:05:27.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:27.246+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:27.867 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:27.864+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:27.944 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:27.942+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:27.943270+0000) 2022-04-04T00:05:28.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:28.292+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:28.895 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:28.892+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:28.944 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:28.942+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:28.943444+0000) 2022-04-04T00:05:29.343 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:29.340+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:29.890 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:29.887+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:29.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:29.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:29.943650+0000) 2022-04-04T00:05:30.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:30.313+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:30.908 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:30.905+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:30.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:30.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:30.943865+0000) 2022-04-04T00:05:31.290 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:31.287+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:31.902 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:31.899+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:31.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:31.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:31.944086+0000) 2022-04-04T00:05:32.318 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:32.315+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:32.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:32.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:32.944291+0000) 2022-04-04T00:05:32.947 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:32.944+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:33.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:33.306+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:33.917 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:33.914+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:33.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:33.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:33.944493+0000) 2022-04-04T00:05:34.322 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:34.319+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:34.896 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:34.893+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:34.946 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:34.944+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:34.944700+0000) 2022-04-04T00:05:35.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:35.289+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:35.866 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:35.863+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:35.946 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:35.944+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:35.944879+0000) 2022-04-04T00:05:36.280 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:36.277+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:36.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:36.831+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:36.946 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:36.944+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:36.945116+0000) 2022-04-04T00:05:37.241 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:37.238+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:37.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:37.874+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:37.946 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:37.944+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:37.945323+0000) 2022-04-04T00:05:38.208 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:38.206+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:38.890 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:38.887+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:38.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:38.945+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:38.945493+0000) 2022-04-04T00:05:39.221 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:39.218+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:39.287 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:05:39.285+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:05:39.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:39.859+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:39.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:39.945+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:39.945704+0000) 2022-04-04T00:05:40.172 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:40.170+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:40.901 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:40.898+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:40.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:40.945+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:40.945876+0000) 2022-04-04T00:05:41.206 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:41.203+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:41.860 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:41.857+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:41.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:41.945+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:41.946057+0000) 2022-04-04T00:05:42.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:42.167+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:42.845 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:42.842+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:42.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:42.945+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:42.946271+0000) 2022-04-04T00:05:43.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:43.196+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:43.818 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:43.815+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:43.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:43.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:43.946497+0000) 2022-04-04T00:05:44.209 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:44.207+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:44.857 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:44.855+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:44.948 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:44.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:44.946749+0000) 2022-04-04T00:05:45.216 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:45.213+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:45.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:45.808+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:45.948 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:45.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:45.946987+0000) 2022-04-04T00:05:46.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:46.184+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:46.807 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:46.804+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:46.948 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:46.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:46.947194+0000) 2022-04-04T00:05:47.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:47.154+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:47.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:47.776+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:47.948 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:47.947+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:47.947404+0000) 2022-04-04T00:05:48.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:48.176+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:48.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:48.752+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:48.949 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:48.947+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:48.947616+0000) 2022-04-04T00:05:49.204 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:49.201+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:49.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:49.755+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:49.949 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:49.947+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:49.947834+0000) 2022-04-04T00:05:50.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:50.153+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:50.776 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:50.773+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:50.949 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:50.947+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:50.948032+0000) 2022-04-04T00:05:51.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:51.108+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:51.809 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:51.806+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:51.949 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:51.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:51.948263+0000) 2022-04-04T00:05:52.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:52.081+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:52.828 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:52.826+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:52.949 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:52.947+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:52.948436+0000) 2022-04-04T00:05:53.096 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:53.094+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:53.837 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:53.834+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:53.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:53.947+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:53.948614+0000) 2022-04-04T00:05:54.145 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:54.143+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:54.303 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:05:54.300+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:05:54.825 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:54.823+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:54.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:54.947+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:54.948835+0000) 2022-04-04T00:05:55.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:55.095+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:55.825 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:55.822+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:55.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:55.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:55.949028+0000) 2022-04-04T00:05:56.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:56.130+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:56.849 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:56.846+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:56.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:56.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:56.949251+0000) 2022-04-04T00:05:57.116 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:57.113+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:57.474 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:05:57.478 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:05:57.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:57.861+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:57.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:57.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:57.949490+0000) 2022-04-04T00:05:58.109 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:58.106+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:58.849 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:58.846+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:58.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:58.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:58.949695+0000) 2022-04-04T00:05:59.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:05:59.108+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:05:59.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:05:59.799+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:05:59.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:05:59.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:05:59.949910+0000) 2022-04-04T00:06:00.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:00.120+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:00.814 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:00.811+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:00.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:00.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:00.950111+0000) 2022-04-04T00:06:01.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:01.159+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:01.860 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:01.858+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:01.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:01.949+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:01.950335+0000) 2022-04-04T00:06:02.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:02.179+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:02.898 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:02.896+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:02.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:02.949+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:02.950500+0000) 2022-04-04T00:06:03.199 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:03.196+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:03.879 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:03.876+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:03.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:03.949+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:03.950676+0000) 2022-04-04T00:06:04.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:04.225+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:04.840 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:04.838+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:04.952 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:04.950+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:04.950900+0000) 2022-04-04T00:06:05.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:05.185+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:05.881 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:05.878+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:05.952 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:05.950+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:05.951096+0000) 2022-04-04T00:06:06.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:06.165+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:06.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:06.860+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:06.952 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:06.950+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:06.951243+0000) 2022-04-04T00:06:07.180 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:07.178+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:07.830 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:07.828+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:07.952 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:07.950+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:07.951437+0000) 2022-04-04T00:06:08.151 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:08.149+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:08.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:08.830+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:08.953 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:08.950+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:08.951634+0000) 2022-04-04T00:06:09.102 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:09.100+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:09.319 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:06:09.317+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:06:09.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:09.840+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:09.953 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:09.951+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:09.951843+0000) 2022-04-04T00:06:10.099 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:10.097+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:10.831 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:10.830+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:10.953 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:10.951+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:10.952032+0000) 2022-04-04T00:06:11.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:11.078+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:11.838 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:11.836+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:11.953 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:11.951+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:11.952236+0000) 2022-04-04T00:06:12.042 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:12.040+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:12.847 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:12.845+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:12.953 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:12.951+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:12.952432+0000) 2022-04-04T00:06:12.994 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:12.993+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:13.873 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:13.871+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:13.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:13.952+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:13.952643+0000) 2022-04-04T00:06:14.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:14.012+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:14.923 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:14.921+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:14.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:14.952+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:14.952857+0000) 2022-04-04T00:06:14.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:14.983+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:15.919 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:15.917+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:15.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:15.952+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:15.953014+0000) 2022-04-04T00:06:16.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:16.003+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:16.891 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:16.889+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:16.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:16.952+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:16.953170+0000) 2022-04-04T00:06:17.002 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:17.001+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:17.860 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:17.858+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:17.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:17.952+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:17.953371+0000) 2022-04-04T00:06:18.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:18.002+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:18.892 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:18.890+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:18.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:18.952+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:18.953606+0000) 2022-04-04T00:06:18.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:18.971+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:19.880 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:19.878+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:19.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:19.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:19.953828+0000) 2022-04-04T00:06:19.961 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:19.959+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:20.838 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:20.836+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:20.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:20.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:20.954023+0000) 2022-04-04T00:06:20.969 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:20.967+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:21.874 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:21.872+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:21.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:21.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:21.954170+0000) 2022-04-04T00:06:21.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:21.971+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:22.881 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:22.878+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:22.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:22.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:22.954367+0000) 2022-04-04T00:06:22.976 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:22.974+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:23.906 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:23.903+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:23.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:23.948+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:23.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:23.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:23.954562+0000) 2022-04-04T00:06:24.335 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:06:24.332+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:06:24.866 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:24.863+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:24.935 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:24.932+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:24.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:24.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:24.954757+0000) 2022-04-04T00:06:25.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:25.859+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:25.892 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:25.889+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:25.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:25.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:25.954944+0000) 2022-04-04T00:06:26.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:26.812+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:26.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:26.918+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:26.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:26.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:26.955113+0000) 2022-04-04T00:06:27.628 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:06:27.632 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:06:27.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:27.765+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:27.944 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:27.941+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:27.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:27.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:27.955348+0000) 2022-04-04T00:06:28.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:28.754+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:28.910 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:28.907+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:28.957 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:28.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:28.955546+0000) 2022-04-04T00:06:29.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:29.748+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:29.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:29.944+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:29.957 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:29.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:29.955757+0000) 2022-04-04T00:06:30.744 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:30.742+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:30.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:30.921+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:30.957 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:30.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:30.955982+0000) 2022-04-04T00:06:31.710 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:31.707+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:31.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:31.883+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:31.957 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:31.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:31.956197+0000) 2022-04-04T00:06:32.759 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:32.756+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:32.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:32.879+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:32.957 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:32.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:32.956397+0000) 2022-04-04T00:06:33.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:33.747+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:33.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:33.835+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:33.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:33.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:33.956592+0000) 2022-04-04T00:06:34.706 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:34.704+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:34.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:34.819+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:34.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:34.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:34.956833+0000) 2022-04-04T00:06:35.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:35.738+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:35.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:35.810+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:35.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:35.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:35.957019+0000) 2022-04-04T00:06:36.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:36.755+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:36.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:36.776+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:36.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:36.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:36.957225+0000) 2022-04-04T00:06:37.761 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:37.758+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:37.766 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:37.763+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:37.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:37.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:37.957430+0000) 2022-04-04T00:06:38.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:38.726+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:38.786 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:38.784+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:38.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:38.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:38.957637+0000) 2022-04-04T00:06:39.350 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:06:39.347+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:06:39.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:39.754+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:39.766 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:39.764+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:39.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:39.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:39.957855+0000) 2022-04-04T00:06:40.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:40.726+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:40.773 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:40.770+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:40.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:40.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:40.958063+0000) 2022-04-04T00:06:41.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:41.697+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:41.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:41.753+0000 7fd3b190b700 -1 osd.6 134 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:41.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:41.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:41.958276+0000) 2022-04-04T00:06:42.444 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-04T00:06:42.442+0000 7f470e231700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-04T00:06:42.459 INFO:tasks.ceph.mds.f.smithi186.stderr: -1> 2022-04-04T00:06:42.442+0000 7f470e231700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-04T00:06:42.481 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-04T00:06:42.708 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:42.705+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:42.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:42.728+0000 7f88d0ac1700 -1 osd.4 134 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:42.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:42.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:42.958480+0000) 2022-04-04T00:06:43.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:43.680+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:43.736 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:43.733+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:43.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:43.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:43.958692+0000) 2022-04-04T00:06:44.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:44.716+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:44.749 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:44.746+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:44.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:44.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:44.958886+0000) 2022-04-04T00:06:45.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:45.734+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:45.769 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:45.767+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:45.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:45.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:45.959088+0000) 2022-04-04T00:06:46.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:46.746+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 50 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:06:46.787 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:46.784+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:46.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:46.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:46.959286+0000) 2022-04-04T00:06:47.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:47.774+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:06:47.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:47.780+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:47.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:47.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:47.959453+0000) 2022-04-04T00:06:48.763 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:48.760+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:06:48.826 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:48.823+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:48.961 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:48.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:48.959654+0000) 2022-04-04T00:06:49.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:49.783+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:06:49.819 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:49.816+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:49.961 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:49.959+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:49.959856+0000) 2022-04-04T00:06:50.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:50.771+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:06:50.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:50.848+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:50.961 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:50.959+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:50.960034+0000) 2022-04-04T00:06:51.796 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:51.794+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:06:51.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:51.837+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:51.961 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:51.959+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:51.960247+0000) 2022-04-04T00:06:52.788 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:52.786+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:06:52.859 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:52.856+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:52.962 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:52.959+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:52.960450+0000) 2022-04-04T00:06:53.823 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:53.821+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:06:53.895 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:53.892+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:53.962 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:53.959+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:53.960658+0000) 2022-04-04T00:06:54.366 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:06:54.364+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:06:54.871 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:54.869+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:06:54.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:54.885+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:54.962 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:54.960+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:54.960898+0000) 2022-04-04T00:06:55.888 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:55.886+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:06:55.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:55.891+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:55.962 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:55.960+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:55.961124+0000) 2022-04-04T00:06:56.923 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:56.920+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:06:56.932 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:56.930+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:56.962 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:56.960+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:56.961324+0000) 2022-04-04T00:06:57.759 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:06:57.763 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:06:57.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:57.902+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:57.959 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:57.957+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:06:57.962 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:57.960+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:57.961564+0000) 2022-04-04T00:06:58.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:58.860+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:58.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:58.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:58.961727+0000) 2022-04-04T00:06:58.976 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:06:58.974+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:06:59.884 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:06:59.881+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:06:59.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:06:59.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:06:59.961961+0000) 2022-04-04T00:07:00.023 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:00.021+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:00.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:00.883+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:00.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:00.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:00.962107+0000) 2022-04-04T00:07:00.987 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:00.985+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:01.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:01.840+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:01.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:01.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:01.962316+0000) 2022-04-04T00:07:01.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:01.994+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:02.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:02.865+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:02.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:02.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:02.962464+0000) 2022-04-04T00:07:03.039 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:03.037+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:03.844 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:03.842+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:03.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:03.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:03.962681+0000) 2022-04-04T00:07:03.990 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:03.988+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:04.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:04.794+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:04.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:04.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:04.962855+0000) 2022-04-04T00:07:04.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:04.995+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:05.771 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:05.769+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:05.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:05.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:05.963078+0000) 2022-04-04T00:07:06.028 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:06.026+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:06.787 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:06.785+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:06.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:06.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:06.963270+0000) 2022-04-04T00:07:07.052 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:07.050+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:07.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:07.771+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:07.965 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:07.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:07.963506+0000) 2022-04-04T00:07:08.006 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:08.004+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:08.795 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:08.793+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:08.964 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:08.962+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:08.965 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:08.963+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:08.963712+0000) 2022-04-04T00:07:09.382 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:07:09.380+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:07:09.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:09.822+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:09.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:09.941+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:09.965 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:09.963+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:09.963861+0000) 2022-04-04T00:07:10.833 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:10.831+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:10.939 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:10.937+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:10.965 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:10.963+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:10.964075+0000) 2022-04-04T00:07:11.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:11.879+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:11.932 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:11.930+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:11.965 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:11.963+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:11.964258+0000) 2022-04-04T00:07:12.844 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:12.842+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:12.928 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:12.926+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:12.965 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:12.964+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:12.964432+0000) 2022-04-04T00:07:13.884 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:13.882+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:13.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:13.883+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:13.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:13.964+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:13.964642+0000) 2022-04-04T00:07:14.839 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:14.837+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:14.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:14.921+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:14.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:14.964+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:14.964857+0000) 2022-04-04T00:07:15.839 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:15.837+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:15.906 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:15.904+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:15.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:15.964+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:15.965068+0000) 2022-04-04T00:07:16.816 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:16.814+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:16.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:16.872+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:16.967 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:16.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:16.965285+0000) 2022-04-04T00:07:17.783 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:17.781+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:17.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:17.862+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:17.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:17.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:17.965492+0000) 2022-04-04T00:07:18.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:18.736+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:18.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:18.847+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:18.967 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:18.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:18.965688+0000) 2022-04-04T00:07:19.693 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:19.690+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:19.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:19.883+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:19.967 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:19.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:19.965903+0000) 2022-04-04T00:07:20.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:20.669+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:20.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:20.834+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:20.967 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:20.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:20.966077+0000) 2022-04-04T00:07:21.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:21.694+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:21.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:21.876+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:21.974 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:21.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:21.966254+0000) 2022-04-04T00:07:22.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:22.645+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:22.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:22.835+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:22.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:22.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:22.966436+0000) 2022-04-04T00:07:23.664 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:23.661+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:23.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:23.799+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:23.967 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:23.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:23.966593+0000) 2022-04-04T00:07:24.398 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:07:24.395+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:07:24.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:24.628+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:24.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:24.758+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:24.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:24.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:24.966834+0000) 2022-04-04T00:07:25.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:25.590+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:25.730 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:25.728+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:25.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:25.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:25.966976+0000) 2022-04-04T00:07:26.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:26.550+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:26.754 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:26.751+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:26.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:26.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:26.967193+0000) 2022-04-04T00:07:27.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:27.589+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:27.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:27.787+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:27.940 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:07:27.944 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:07:27.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:27.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:27.967384+0000) 2022-04-04T00:07:28.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:28.566+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:28.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:28.791+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:28.969 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:28.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:28.967619+0000) 2022-04-04T00:07:29.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:29.521+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:29.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:29.796+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:29.969 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:29.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:29.967857+0000) 2022-04-04T00:07:30.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:30.484+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:30.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:30.812+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:30.969 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:30.967+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:30.968051+0000) 2022-04-04T00:07:31.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:31.514+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:31.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:31.802+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:31.969 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:31.967+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:31.968292+0000) 2022-04-04T00:07:32.533 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:32.530+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:32.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:32.833+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:32.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:32.967+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:32.968521+0000) 2022-04-04T00:07:33.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:33.559+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:33.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:33.832+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:33.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:33.967+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:33.968727+0000) 2022-04-04T00:07:34.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:34.514+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:34.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:34.804+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:34.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:34.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:34.968935+0000) 2022-04-04T00:07:35.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:35.552+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:35.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:35.836+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:35.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:35.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:35.969071+0000) 2022-04-04T00:07:36.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:36.555+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:36.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:36.808+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:36.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:36.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:36.969267+0000) 2022-04-04T00:07:37.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:37.519+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:37.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:37.812+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:37.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:37.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:37.969453+0000) 2022-04-04T00:07:38.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:38.491+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:38.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:38.797+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:38.971 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:38.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:38.969640+0000) 2022-04-04T00:07:39.414 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:07:39.411+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:07:39.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:39.446+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:39.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:39.786+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:39.971 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:39.969+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:39.969854+0000) 2022-04-04T00:07:40.489 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:40.486+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:40.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:40.748+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:40.971 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:40.969+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:40.970006+0000) 2022-04-04T00:07:41.490 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:41.487+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:41.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:41.781+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:41.971 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:41.969+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:41.970226+0000) 2022-04-04T00:07:42.451 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:42.449+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:42.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:42.772+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:42.971 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:42.969+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:42.970392+0000) 2022-04-04T00:07:43.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:43.416+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:43.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:43.777+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:43.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:43.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:43.970554+0000) 2022-04-04T00:07:44.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:44.414+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:44.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:44.758+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:44.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:44.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:44.970739+0000) 2022-04-04T00:07:45.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:45.369+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:45.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:45.742+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:45.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:45.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:45.970940+0000) 2022-04-04T00:07:46.382 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:46.380+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:46.711 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:46.708+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:46.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:46.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:46.971073+0000) 2022-04-04T00:07:47.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:47.423+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:47.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:47.750+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:47.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:47.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:47.971272+0000) 2022-04-04T00:07:48.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:48.402+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:48.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:48.768+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:48.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:48.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:48.971430+0000) 2022-04-04T00:07:49.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:49.360+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:49.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:49.731+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:49.973 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:49.971+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:49.971645+0000) 2022-04-04T00:07:50.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:50.396+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:50.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:50.733+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:50.973 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:50.971+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:50.971852+0000) 2022-04-04T00:07:51.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:51.347+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:51.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:51.690+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:51.973 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:51.971+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:51.972065+0000) 2022-04-04T00:07:52.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:52.377+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:52.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:52.733+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:52.973 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:52.971+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:52.972266+0000) 2022-04-04T00:07:53.413 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:53.411+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:53.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:53.723+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:53.974 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:53.972+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:53.972534+0000) 2022-04-04T00:07:54.378 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:54.376+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:54.428 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:07:54.426+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:07:54.752 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:54.750+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:54.974 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:54.972+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:54.972813+0000) 2022-04-04T00:07:55.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:55.387+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:55.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:55.755+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:55.974 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:55.972+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:55.973004+0000) 2022-04-04T00:07:56.424 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:56.422+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:56.721 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:56.718+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:56.974 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:56.972+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:56.973223+0000) 2022-04-04T00:07:57.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:57.407+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:57.767 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:57.765+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:57.974 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:57.973+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:57.973470+0000) 2022-04-04T00:07:58.088 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:07:58.092 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:07:58.364 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:58.362+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:58.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:58.747+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:58.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:58.973+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:58.973714+0000) 2022-04-04T00:07:59.337 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:07:59.336+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:07:59.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:07:59.737+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:07:59.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:07:59.973+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:07:59.973896+0000) 2022-04-04T00:08:00.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:00.317+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:00.786 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:00.784+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:00.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:00.973+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:00.974110+0000) 2022-04-04T00:08:01.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:01.348+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:01.738 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:01.736+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:01.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:01.974+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:01.974367+0000) 2022-04-04T00:08:02.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:02.389+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:02.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:02.730+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:02.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:02.974+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:02.974561+0000) 2022-04-04T00:08:03.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:03.409+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:03.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:03.768+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:03.976 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:03.974+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:03.974801+0000) 2022-04-04T00:08:04.378 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:04.376+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:04.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:04.730+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:04.976 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:04.974+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:04.975047+0000) 2022-04-04T00:08:05.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:05.359+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:05.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:05.730+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:05.976 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:05.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:05.975251+0000) 2022-04-04T00:08:06.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:06.321+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:06.683 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:06.681+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:06.977 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:06.974+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:06.975487+0000) 2022-04-04T00:08:07.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:07.321+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:07.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:07.696+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:07.977 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:07.974+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:07.975655+0000) 2022-04-04T00:08:08.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:08.343+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:08.681 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:08.679+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:08.977 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:08.974+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:08.975890+0000) 2022-04-04T00:08:09.396 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:09.394+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:09.444 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:08:09.441+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:08:09.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:09.675+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:09.977 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:09.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:09.976145+0000) 2022-04-04T00:08:10.412 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:10.410+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:10.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:10.627+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:10.977 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:10.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:10.976348+0000) 2022-04-04T00:08:11.432 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:11.430+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:11.612 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:11.610+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:11.978 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:11.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:11.976574+0000) 2022-04-04T00:08:12.401 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:12.399+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:12.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:12.577+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:12.978 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:12.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:12.976852+0000) 2022-04-04T00:08:13.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:13.365+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:13.606 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:13.604+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:13.978 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:13.976+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:13.977075+0000) 2022-04-04T00:08:14.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:14.358+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:14.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:14.589+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:14.978 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:14.976+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:14.977295+0000) 2022-04-04T00:08:15.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:15.390+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:15.617 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:15.614+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:15.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:15.976+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:15.977549+0000) 2022-04-04T00:08:16.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:16.360+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:16.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:16.584+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:16.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:16.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:16.977766+0000) 2022-04-04T00:08:17.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:17.387+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:17.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:17.622+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:17.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:17.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:17.978003+0000) 2022-04-04T00:08:18.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:18.394+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:18.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:18.600+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:18.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:18.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:18.978175+0000) 2022-04-04T00:08:19.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:19.346+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:19.574 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:19.571+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:19.980 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:19.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:19.978413+0000) 2022-04-04T00:08:20.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:20.356+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:20.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:20.539+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:20.980 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:20.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:20.978663+0000) 2022-04-04T00:08:21.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:21.382+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:21.527 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:21.524+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:21.980 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:21.978+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:21.978889+0000) 2022-04-04T00:08:22.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:22.371+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:22.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:22.564+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:22.980 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:22.978+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:22.979038+0000) 2022-04-04T00:08:23.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:23.381+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:23.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:23.539+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:23.980 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:23.978+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:23.979226+0000) 2022-04-04T00:08:24.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:24.378+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:24.460 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:08:24.458+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:08:24.500 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:24.497+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:24.980 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:24.978+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:24.979412+0000) 2022-04-04T00:08:25.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:25.344+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:25.480 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:25.477+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:25.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:25.978+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:25.979662+0000) 2022-04-04T00:08:26.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:26.370+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:26.496 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:26.493+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:26.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:26.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:26.979902+0000) 2022-04-04T00:08:27.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:27.405+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:27.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:27.479+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:27.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:27.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:27.980124+0000) 2022-04-04T00:08:28.170 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:08:28.174 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:08:28.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:28.441+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:28.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:28.485+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:28.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:28.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:28.980393+0000) 2022-04-04T00:08:29.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:29.413+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:29.526 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:29.523+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:29.982 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:29.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:29.980533+0000) 2022-04-04T00:08:30.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:30.450+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:30.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:30.533+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:30.982 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:30.980+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:30.980667+0000) 2022-04-04T00:08:31.437 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:31.434+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:31.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:31.543+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:31.982 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:31.980+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:31.980890+0000) 2022-04-04T00:08:32.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:32.455+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:32.526 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:32.522+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:32.982 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:32.980+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:32.981104+0000) 2022-04-04T00:08:33.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:33.428+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:33.530 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:33.527+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:33.982 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:33.980+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:33.981317+0000) 2022-04-04T00:08:34.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:34.436+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:34.499 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:34.497+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:34.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:34.981+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:34.981521+0000) 2022-04-04T00:08:35.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:35.449+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:35.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:35.461+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:35.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:35.981+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:35.981707+0000) 2022-04-04T00:08:36.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:36.408+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:36.462 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:36.459+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:36.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:36.981+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:36.981922+0000) 2022-04-04T00:08:37.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:37.380+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:37.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:37.457+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:37.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:37.981+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:37.982137+0000) 2022-04-04T00:08:38.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:38.428+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:38.505 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:38.503+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:38.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:38.981+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:38.982374+0000) 2022-04-04T00:08:39.471 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:39.468+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:39.476 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:08:39.474+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:08:39.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:39.512+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:39.984 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:39.982+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:39.982561+0000) 2022-04-04T00:08:40.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:40.465+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:40.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:40.496+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:40.984 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:40.982+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:40.982794+0000) 2022-04-04T00:08:41.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:41.438+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:41.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:41.481+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:41.984 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:41.982+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:41.983003+0000) 2022-04-04T00:08:42.477 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:42.474+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:42.484 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:42.481+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:42.984 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:42.982+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:42.983218+0000) 2022-04-04T00:08:43.434 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:43.431+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:43.533 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:43.531+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:43.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:43.983+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:43.983440+0000) 2022-04-04T00:08:44.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:44.391+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:44.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:44.571+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:44.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:44.982+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:44.983632+0000) 2022-04-04T00:08:45.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:45.379+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:45.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:45.572+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:45.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:45.983+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:45.983861+0000) 2022-04-04T00:08:46.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:46.404+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:46.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:46.588+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:46.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:46.983+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:46.984059+0000) 2022-04-04T00:08:47.379 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:47.376+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:47.600 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:47.598+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:47.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:47.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:47.984188+0000) 2022-04-04T00:08:48.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:48.349+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:48.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:48.561+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:48.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:48.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:48.984387+0000) 2022-04-04T00:08:49.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:49.313+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:49.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:49.555+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:49.986 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:49.983+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:49.984573+0000) 2022-04-04T00:08:50.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:50.334+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:50.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:50.535+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:50.986 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:50.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:50.984750+0000) 2022-04-04T00:08:51.294 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:51.292+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:51.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:51.525+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:51.986 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:51.983+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:51.984941+0000) 2022-04-04T00:08:52.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:52.248+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:52.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:52.495+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:52.986 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:52.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:52.985100+0000) 2022-04-04T00:08:53.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:53.243+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:53.498 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:53.496+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:53.986 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:53.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:53.985306+0000) 2022-04-04T00:08:54.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:54.282+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:54.479 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:08:54.476+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:08:54.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:54.523+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:54.987 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:54.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:54.985499+0000) 2022-04-04T00:08:55.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:55.286+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:55.486 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:55.484+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:55.987 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:55.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:55.985674+0000) 2022-04-04T00:08:56.251 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:56.249+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:56.502 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:56.500+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:56.987 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:56.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:56.985880+0000) 2022-04-04T00:08:57.280 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:57.278+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:57.539 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:57.537+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:57.987 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:57.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:57.986028+0000) 2022-04-04T00:08:58.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:58.318+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:58.324 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:08:58.328 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:08:58.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:58.557+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:58.988 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:58.985+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:58.986242+0000) 2022-04-04T00:08:59.344 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:08:59.342+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:08:59.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:08:59.573+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:08:59.988 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:08:59.985+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:08:59.986415+0000) 2022-04-04T00:09:00.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:00.347+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:00.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:00.590+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:00.988 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:00.985+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:00.986634+0000) 2022-04-04T00:09:01.372 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:01.370+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:01.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:01.557+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:01.988 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:01.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:01.986881+0000) 2022-04-04T00:09:02.398 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:02.396+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:02.510 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:02.508+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:02.988 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:02.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:02.987050+0000) 2022-04-04T00:09:03.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:03.421+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:03.539 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:03.537+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:03.988 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:03.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:03.987257+0000) 2022-04-04T00:09:04.452 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:04.450+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:04.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:04.491+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:04.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:04.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:04.987463+0000) 2022-04-04T00:09:05.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:05.458+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:05.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:05.495+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:05.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:05.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:05.987684+0000) 2022-04-04T00:09:06.466 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:06.464+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:06.511 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:06.509+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:06.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:06.987+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:06.987926+0000) 2022-04-04T00:09:07.468 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:07.466+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:07.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:07.508+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:07.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:07.987+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:07.988109+0000) 2022-04-04T00:09:08.446 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:08.445+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:08.550 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:08.548+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:08.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:08.987+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:08.988315+0000) 2022-04-04T00:09:09.446 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:09.443+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:09.495 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:09:09.493+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:09:09.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:09.512+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:09.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:09.987+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:09.988470+0000) 2022-04-04T00:09:10.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:10.438+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:10.534 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:10.531+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:10.990 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:10.987+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:10.988683+0000) 2022-04-04T00:09:11.468 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:11.465+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:11.569 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:11.566+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:11.990 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:11.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:11.988934+0000) 2022-04-04T00:09:12.460 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:12.457+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:12.562 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:12.559+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:12.990 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:12.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:12.989099+0000) 2022-04-04T00:09:13.451 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:13.448+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:13.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:13.584+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:13.990 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:13.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:13.989257+0000) 2022-04-04T00:09:14.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:14.453+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:14.636 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:14.633+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:14.990 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:14.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:14.989418+0000) 2022-04-04T00:09:15.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:15.446+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:15.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:15.639+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:15.991 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:15.989+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:15.989560+0000) 2022-04-04T00:09:16.438 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:16.435+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:16.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:16.653+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:16.991 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:16.989+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:16.989804+0000) 2022-04-04T00:09:17.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:17.438+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:17.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:17.662+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:17.991 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:17.989+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:17.990017+0000) 2022-04-04T00:09:18.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:18.436+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:18.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:18.655+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:18.991 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:18.989+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:18.990246+0000) 2022-04-04T00:09:19.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:19.391+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:19.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:19.664+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:19.992 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:19.989+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:19.990460+0000) 2022-04-04T00:09:20.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:20.386+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:20.649 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:20.646+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:20.992 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:20.990+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:20.990674+0000) 2022-04-04T00:09:21.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:21.369+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:21.617 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:21.614+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:21.992 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:21.990+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:21.990919+0000) 2022-04-04T00:09:22.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:22.338+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:22.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:22.655+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:22.992 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:22.990+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:22.991122+0000) 2022-04-04T00:09:23.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:23.338+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:23.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:23.669+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:23.992 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:23.990+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:23.991374+0000) 2022-04-04T00:09:24.375 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:24.373+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:24.497 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:09:24.494+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:09:24.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:24.697+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:24.993 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:24.990+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:24.991600+0000) 2022-04-04T00:09:25.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:25.396+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:25.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:25.739+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:25.993 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:25.991+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:25.991858+0000) 2022-04-04T00:09:26.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:26.436+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:26.711 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:26.709+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:26.993 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:26.991+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:26.992074+0000) 2022-04-04T00:09:27.429 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:27.427+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:27.701 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:27.699+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:27.993 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:27.992+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:27.992306+0000) 2022-04-04T00:09:28.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:28.453+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:28.471 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:09:28.475 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:09:28.720 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:28.717+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:28.994 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:28.992+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:28.992538+0000) 2022-04-04T00:09:29.415 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:29.412+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:29.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:29.760+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:29.994 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:29.992+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:29.992731+0000) 2022-04-04T00:09:30.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:30.416+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:30.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:30.728+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:30.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:30.992+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:30.992944+0000) 2022-04-04T00:09:31.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:31.425+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:31.756 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:31.753+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:31.994 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:31.992+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:31.993203+0000) 2022-04-04T00:09:32.021 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-04T00:09:32.019+0000 7f2e43b1f700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T00:09:32.022 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-04T00:09:32.019+0000 7f2e44320700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T00:09:32.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:32.422+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:32.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:32.725+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:32.994 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:32.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:32.993418+0000) 2022-04-04T00:09:33.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:33.430+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:33.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:33.732+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:33.995 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:33.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:33.993560+0000) 2022-04-04T00:09:34.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:34.411+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:34.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:34.696+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:34.995 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:34.992+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:34.993742+0000) 2022-04-04T00:09:35.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:35.413+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:35.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:35.736+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:35.995 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:35.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:35.993954+0000) 2022-04-04T00:09:36.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:36.365+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:36.720 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:36.718+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:36.995 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:36.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:36.994194+0000) 2022-04-04T00:09:37.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:37.326+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:37.769 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:37.767+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:37.995 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:37.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:37.994447+0000) 2022-04-04T00:09:38.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:38.289+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:38.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:38.741+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:38.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:38.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:38.994615+0000) 2022-04-04T00:09:39.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:39.291+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:39.513 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:09:39.509+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:09:39.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:39.714+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:39.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:39.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:39.994795+0000) 2022-04-04T00:09:40.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:40.333+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:40.746 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:40.743+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:40.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:40.994+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:40.995004+0000) 2022-04-04T00:09:41.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:41.304+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:41.771 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:41.768+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:41.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:41.994+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:41.995223+0000) 2022-04-04T00:09:42.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:42.349+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:42.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:42.736+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:42.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:42.994+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:42.995469+0000) 2022-04-04T00:09:43.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:43.323+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:43.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:43.698+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:43.997 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:43.994+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:43.995663+0000) 2022-04-04T00:09:44.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:44.314+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:44.672 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:44.670+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:44.997 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:44.994+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:44.995848+0000) 2022-04-04T00:09:45.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:45.311+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:45.654 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:45.652+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:45.997 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:45.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:45.996067+0000) 2022-04-04T00:09:46.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:46.264+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:46.646 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:46.643+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:46.997 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:46.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:46.996260+0000) 2022-04-04T00:09:47.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:47.242+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 2 slow ops, oldest is osd_op(mds.0.925:2 3.b 3.3270c60b (undecoded) ondisk+retry+read+known_if_redirected+full_force+supports_pool_eio e125) 2022-04-04T00:09:47.645 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:47.642+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:47.997 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:47.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:47.996410+0000) 2022-04-04T00:09:48.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:48.199+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:09:48.649 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:48.647+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:48.998 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:48.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:48.996628+0000) 2022-04-04T00:09:49.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:49.161+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:09:49.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:49.697+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:49.998 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:49.996+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:49.996897+0000) 2022-04-04T00:09:50.174 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:50.172+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:09:50.689 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:50.687+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:50.998 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:50.996+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:50.997163+0000) 2022-04-04T00:09:51.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:51.198+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:09:51.736 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:51.734+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:51.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:51.996+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:51.997437+0000) 2022-04-04T00:09:52.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:52.195+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:09:52.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:52.770+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:52.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:52.996+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:52.997628+0000) 2022-04-04T00:09:53.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:53.235+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:09:53.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:53.727+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:53.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:53.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:53.997878+0000) 2022-04-04T00:09:54.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:54.202+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:09:54.529 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:09:54.526+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:09:54.771 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:54.769+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:54.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:54.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:54.998124+0000) 2022-04-04T00:09:55.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:55.242+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:09:55.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:55.809+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:55.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:55.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:55.998322+0000) 2022-04-04T00:09:56.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:56.264+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:09:56.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:56.827+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:56.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:56.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:56.998539+0000) 2022-04-04T00:09:57.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:57.245+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:09:57.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:57.810+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:58.000 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:57.998+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:57.998818+0000) 2022-04-04T00:09:58.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:58.280+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:09:58.610 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:09:58.613 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:09:58.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:58.818+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:09:59.000 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:58.998+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:58.999093+0000) 2022-04-04T00:09:59.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:09:59.305+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:09:59.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:09:59.866+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:00.000 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:09:59.998+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:09:59.999341+0000) 2022-04-04T00:10:00.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:00.278+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:00.869 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:00.867+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:01.001 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:00.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:00.999593+0000) 2022-04-04T00:10:01.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:01.238+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:01.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:01.863+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:02.001 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:01.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:01.999874+0000) 2022-04-04T00:10:02.284 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:02.282+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:02.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:02.873+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:03.001 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:02.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:03.000108+0000) 2022-04-04T00:10:03.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:03.258+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:03.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:03.884+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:04.001 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:03.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:04.000331+0000) 2022-04-04T00:10:04.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:04.278+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:04.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:04.901+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:05.002 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:05.000+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:05.000583+0000) 2022-04-04T00:10:05.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:05.275+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:05.892 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:05.889+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:06.002 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:06.000+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:06.000855+0000) 2022-04-04T00:10:06.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:06.262+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:06.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:06.878+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:07.002 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:07.000+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:07.001107+0000) 2022-04-04T00:10:07.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:07.288+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:07.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:07.920+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:08.002 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:08.000+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:08.001352+0000) 2022-04-04T00:10:08.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:08.285+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:08.972 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:08.970+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:09.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:09.001+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:09.001597+0000) 2022-04-04T00:10:09.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:09.269+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:09.545 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:10:09.542+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:10:09.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:09.949+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:10.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:10.001+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:10.001861+0000) 2022-04-04T00:10:10.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:10.246+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:10.920 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:10.917+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:11.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:11.001+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:11.002059+0000) 2022-04-04T00:10:11.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:11.292+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:11.895 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:11.892+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:12.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:12.001+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:12.002280+0000) 2022-04-04T00:10:12.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:12.266+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:12.880 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:12.877+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:13.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:13.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:13.002534+0000) 2022-04-04T00:10:13.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:13.288+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:13.859 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:13.856+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:14.004 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:14.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:14.002742+0000) 2022-04-04T00:10:14.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:14.305+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:14.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:14.877+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:15.004 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:15.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:15.002952+0000) 2022-04-04T00:10:15.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:15.328+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:15.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:15.900+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:16.004 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:16.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:16.003155+0000) 2022-04-04T00:10:16.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:16.290+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:16.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:16.870+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:17.004 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:17.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:17.003303+0000) 2022-04-04T00:10:17.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:17.241+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:17.826 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:17.823+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:18.004 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:18.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:18.003488+0000) 2022-04-04T00:10:18.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:18.251+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:18.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:18.788+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:19.005 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:19.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:19.003706+0000) 2022-04-04T00:10:19.212 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:19.209+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:19.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:19.829+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:20.005 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:20.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:20.003953+0000) 2022-04-04T00:10:20.220 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:20.217+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:20.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:20.824+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:21.005 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:21.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:21.004197+0000) 2022-04-04T00:10:21.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:21.217+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:21.859 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:21.857+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:22.005 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:22.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:22.004455+0000) 2022-04-04T00:10:22.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:22.245+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:22.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:22.844+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:23.006 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:23.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:23.004727+0000) 2022-04-04T00:10:23.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:23.275+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:23.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:23.861+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:24.006 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:24.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:24.004997+0000) 2022-04-04T00:10:24.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:24.269+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:24.561 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:10:24.557+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:10:24.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:24.843+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:25.006 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:25.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:25.005246+0000) 2022-04-04T00:10:25.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:25.311+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:25.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:25.832+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:26.006 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:26.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:26.005444+0000) 2022-04-04T00:10:26.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:26.313+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:26.817 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:26.815+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:27.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:27.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:27.005665+0000) 2022-04-04T00:10:27.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:27.286+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:27.862 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:27.859+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:28.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:28.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:28.005845+0000) 2022-04-04T00:10:28.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:28.288+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:28.687 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:10:28.692 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:10:28.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:28.875+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:29.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:29.005+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:29.005987+0000) 2022-04-04T00:10:29.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:29.302+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:29.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:29.915+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:30.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:30.005+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:30.006210+0000) 2022-04-04T00:10:30.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:30.279+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:30.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:30.901+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:31.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:31.005+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:31.006455+0000) 2022-04-04T00:10:31.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:31.318+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:31.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:31.920+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:32.008 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:32.005+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:32.006705+0000) 2022-04-04T00:10:32.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:32.279+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:32.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:32.908+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:33.008 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:33.006+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:33.006895+0000) 2022-04-04T00:10:33.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:33.294+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:33.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:33.926+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:34.008 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:34.006+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:34.007120+0000) 2022-04-04T00:10:34.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:34.318+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:34.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:34.885+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:35.008 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:35.006+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:35.007383+0000) 2022-04-04T00:10:35.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:35.348+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:35.913 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:35.911+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:36.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:36.006+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:36.007636+0000) 2022-04-04T00:10:36.337 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:36.334+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:36.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:36.938+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:37.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:37.007+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:37.007877+0000) 2022-04-04T00:10:37.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:37.302+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:37.909 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:37.907+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:38.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:38.007+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:38.008119+0000) 2022-04-04T00:10:38.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:38.269+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:38.916 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:38.914+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:39.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:39.007+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:39.008341+0000) 2022-04-04T00:10:39.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:39.285+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:39.570 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:10:39.567+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:10:39.901 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:39.898+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:40.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:40.007+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:40.008556+0000) 2022-04-04T00:10:40.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:40.270+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:40.925 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:40.923+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:41.010 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:41.008+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:41.008730+0000) 2022-04-04T00:10:41.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:41.262+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:41.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:41.946+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:42.010 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:42.008+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:42.008925+0000) 2022-04-04T00:10:42.220 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:42.218+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:42.962 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:42.960+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:43.010 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:43.008+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:43.009141+0000) 2022-04-04T00:10:43.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:43.254+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:43.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:43.945+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:44.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:44.008+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:44.009376+0000) 2022-04-04T00:10:44.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:44.298+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:44.996 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:44.994+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:45.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:45.008+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:45.009604+0000) 2022-04-04T00:10:45.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:45.338+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:45.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:45.953+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:46.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:46.009+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:46.009892+0000) 2022-04-04T00:10:46.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:46.337+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:46.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:46.922+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:47.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:47.009+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:47.010080+0000) 2022-04-04T00:10:47.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:47.371+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:47.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:47.971+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:48.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:48.009+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:48.010296+0000) 2022-04-04T00:10:48.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:48.348+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:48.961 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:48.959+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:49.012 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:49.010+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:49.010529+0000) 2022-04-04T00:10:49.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:49.316+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:49.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:49.991+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:50.012 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:50.010+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:50.010730+0000) 2022-04-04T00:10:50.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:50.359+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:50.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:50.979+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:51.012 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:51.010+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:51.010944+0000) 2022-04-04T00:10:51.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:51.332+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:51.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:51.988+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:52.012 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:52.010+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:52.011209+0000) 2022-04-04T00:10:52.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:52.368+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:52.959 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:52.957+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:53.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:53.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:53.011424+0000) 2022-04-04T00:10:53.413 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:53.411+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:53.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:53.921+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:54.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:54.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:54.011672+0000) 2022-04-04T00:10:54.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:54.459+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:54.585 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:10:54.583+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:10:54.894 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:54.892+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:55.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:55.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:55.011920+0000) 2022-04-04T00:10:55.511 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:55.509+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:55.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:55.922+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:56.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:56.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:56.012166+0000) 2022-04-04T00:10:56.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:56.559+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:56.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:56.951+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:57.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:57.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:57.012401+0000) 2022-04-04T00:10:57.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:57.527+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:57.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:57.937+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:58.014 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:58.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:58.012621+0000) 2022-04-04T00:10:58.506 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:58.504+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:58.822 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:10:58.826 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:10:58.936 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:58.934+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:10:59.014 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:10:59.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:10:59.012841+0000) 2022-04-04T00:10:59.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:10:59.515+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:10:59.962 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:10:59.959+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:00.014 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:00.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:00.013036+0000) 2022-04-04T00:11:00.493 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:00.490+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:01.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:01.007+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:01.014 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:01.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:01.013155+0000) 2022-04-04T00:11:01.450 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:01.447+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:02.014 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:02.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:02.013336+0000) 2022-04-04T00:11:02.043 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:02.040+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:02.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:02.466+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:03.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:03.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:03.013447+0000) 2022-04-04T00:11:03.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:03.046+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:03.490 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:03.487+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:04.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:04.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:04.013613+0000) 2022-04-04T00:11:04.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:04.021+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:04.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:04.482+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:04.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:04.995+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:05.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:05.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:05.013738+0000) 2022-04-04T00:11:05.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:05.496+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:06.009 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:06.006+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:06.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:06.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:06.013913+0000) 2022-04-04T00:11:06.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:06.455+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:07.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:07.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:07.014073+0000) 2022-04-04T00:11:07.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:07.046+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:07.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:07.442+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:08.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:08.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:08.014233+0000) 2022-04-04T00:11:08.084 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:08.082+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:08.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:08.455+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:09.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:09.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:09.014436+0000) 2022-04-04T00:11:09.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:09.095+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:09.472 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:09.469+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:09.592 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:11:09.588+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:11:10.016 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:10.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:10.014670+0000) 2022-04-04T00:11:10.052 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:10.049+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:10.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:10.433+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:11.016 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:11.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:11.014861+0000) 2022-04-04T00:11:11.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:11.094+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:11.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:11.478+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:12.016 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:12.014+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:12.015086+0000) 2022-04-04T00:11:12.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:12.059+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:12.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:12.458+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:13.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:13.010+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:13.016 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:13.014+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:13.015300+0000) 2022-04-04T00:11:13.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:13.442+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:13.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:13.964+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:14.016 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:14.014+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:14.015541+0000) 2022-04-04T00:11:14.484 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:14.482+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:14.934 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:14.932+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:15.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:15.014+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:15.015753+0000) 2022-04-04T00:11:15.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:15.527+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:15.915 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:15.912+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:16.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:16.015+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:16.015974+0000) 2022-04-04T00:11:16.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:16.480+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:16.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:16.897+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:17.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:17.015+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:17.016190+0000) 2022-04-04T00:11:17.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:17.442+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:17.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:17.900+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:18.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:18.015+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:18.016416+0000) 2022-04-04T00:11:18.431 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:18.428+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:18.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:18.946+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:19.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:19.015+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:19.016573+0000) 2022-04-04T00:11:19.446 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:19.443+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:19.926 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:19.923+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:20.018 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:20.015+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:20.016734+0000) 2022-04-04T00:11:20.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:20.433+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:20.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:20.918+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:21.018 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:21.016+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:21.016920+0000) 2022-04-04T00:11:21.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:21.408+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:21.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:21.936+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:22.018 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:22.016+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:22.017159+0000) 2022-04-04T00:11:22.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:22.400+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:22.926 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:22.923+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:23.018 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:23.016+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:23.017367+0000) 2022-04-04T00:11:23.415 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:23.412+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:23.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:23.953+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:24.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:24.016+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:24.017519+0000) 2022-04-04T00:11:24.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:24.415+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:24.608 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:11:24.605+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:11:24.913 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:24.911+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:25.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:25.017+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:25.017656+0000) 2022-04-04T00:11:25.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:25.395+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:25.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:25.937+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:26.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:26.017+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:26.017855+0000) 2022-04-04T00:11:26.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:26.368+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:26.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:26.982+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:27.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:27.017+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:27.018050+0000) 2022-04-04T00:11:27.375 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:27.373+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:28.000 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:27.997+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:28.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:28.017+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:28.018271+0000) 2022-04-04T00:11:28.413 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:28.411+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:28.953 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:11:28.957 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:11:28.978 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:28.976+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:29.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:29.017+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:29.018506+0000) 2022-04-04T00:11:29.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:29.456+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:29.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:29.996+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:30.020 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:30.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:30.018674+0000) 2022-04-04T00:11:30.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:30.442+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:30.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:30.987+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:31.020 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:31.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:31.018828+0000) 2022-04-04T00:11:31.434 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:31.432+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:31.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:31.990+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:32.020 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:32.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:32.019041+0000) 2022-04-04T00:11:32.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:32.428+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:33.020 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:33.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:33.019228+0000) 2022-04-04T00:11:33.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:33.024+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:33.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:33.432+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:34.021 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:34.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:34.019449+0000) 2022-04-04T00:11:34.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:34.039+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:34.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:34.480+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:35.021 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:35.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:35.019630+0000) 2022-04-04T00:11:35.070 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:35.068+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:35.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:35.493+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:36.021 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:36.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:36.019840+0000) 2022-04-04T00:11:36.053 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:36.050+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:36.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:36.528+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:37.021 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:37.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:37.020059+0000) 2022-04-04T00:11:37.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:37.033+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:37.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:37.573+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:38.021 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:38.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:38.020268+0000) 2022-04-04T00:11:38.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:38.078+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:38.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:38.613+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:39.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:39.020+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:39.020467+0000) 2022-04-04T00:11:39.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:39.065+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:39.624 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:11:39.621+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:11:39.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:39.639+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:40.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:40.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:40.020569+0000) 2022-04-04T00:11:40.051 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:40.049+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:40.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:40.603+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:41.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:41.020+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:41.020752+0000) 2022-04-04T00:11:41.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:41.077+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:41.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:41.594+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:42.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:42.020+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:42.020948+0000) 2022-04-04T00:11:42.032 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:42.031+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:42.622 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:42.620+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:43.006 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:43.004+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:43.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:43.020+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:43.021136+0000) 2022-04-04T00:11:43.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:43.643+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:44.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:44.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:44.021312+0000) 2022-04-04T00:11:44.042 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:44.040+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:44.648 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:44.646+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:45.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:45.017+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:45.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:45.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:45.021515+0000) 2022-04-04T00:11:45.608 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:45.606+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:46.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:46.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:46.021723+0000) 2022-04-04T00:11:46.050 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:46.049+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:46.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:46.651+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:47.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:47.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:47.021911+0000) 2022-04-04T00:11:47.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:47.065+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:47.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:47.626+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:48.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:48.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:48.022116+0000) 2022-04-04T00:11:48.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:48.083+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:48.662 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:48.660+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:49.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:49.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:49.022273+0000) 2022-04-04T00:11:49.118 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:49.116+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:49.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:49.624+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:50.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:50.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:50.022394+0000) 2022-04-04T00:11:50.168 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:50.166+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:50.609 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:50.607+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:51.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:51.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:51.022537+0000) 2022-04-04T00:11:51.168 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:51.166+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:51.600 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:51.598+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:52.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:52.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:52.022680+0000) 2022-04-04T00:11:52.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:52.164+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:52.621 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:52.620+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:53.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:53.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:53.022858+0000) 2022-04-04T00:11:53.210 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:53.208+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:53.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:53.624+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:54.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:54.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:54.023028+0000) 2022-04-04T00:11:54.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:54.230+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:54.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:54.617+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:54.639 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:11:54.636+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:11:55.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:55.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:55.023209+0000) 2022-04-04T00:11:55.239 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:55.236+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:55.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:55.596+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:56.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:56.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:56.023419+0000) 2022-04-04T00:11:56.283 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:56.280+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:56.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:56.563+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:57.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:57.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:57.023602+0000) 2022-04-04T00:11:57.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:57.258+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:57.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:57.524+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:58.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:58.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:58.023834+0000) 2022-04-04T00:11:58.271 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:58.268+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:58.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:58.564+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:11:59.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:11:59.023+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:11:59.024042+0000) 2022-04-04T00:11:59.096 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:11:59.100 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:11:59.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:11:59.294+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:11:59.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:11:59.529+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:00.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:00.023+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:00.024272+0000) 2022-04-04T00:12:00.312 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:00.310+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:00.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:00.539+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:01.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:01.023+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:01.024449+0000) 2022-04-04T00:12:01.271 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:01.268+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:01.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:01.512+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:02.026 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:02.023+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:02.024584+0000) 2022-04-04T00:12:02.240 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:02.237+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:02.550 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:02.547+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:03.026 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:03.024+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:03.024788+0000) 2022-04-04T00:12:03.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:03.279+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:03.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:03.540+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:04.026 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:04.024+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:04.024997+0000) 2022-04-04T00:12:04.252 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:04.249+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:04.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:04.551+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:05.026 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:05.024+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:05.025202+0000) 2022-04-04T00:12:05.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:05.208+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:05.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:05.577+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:06.026 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:06.024+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:06.025379+0000) 2022-04-04T00:12:06.209 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:06.207+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:06.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:06.529+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:07.027 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:07.024+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:07.025575+0000) 2022-04-04T00:12:07.220 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:07.217+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:07.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:07.576+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:08.027 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:08.025+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:08.025799+0000) 2022-04-04T00:12:08.234 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:08.232+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:08.586 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:08.583+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:09.027 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:09.025+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:09.026009+0000) 2022-04-04T00:12:09.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:09.223+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:09.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:09.558+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:09.655 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:12:09.653+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:12:10.027 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:10.025+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:10.026224+0000) 2022-04-04T00:12:10.196 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:10.193+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:10.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:10.527+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:11.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:11.025+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:11.026440+0000) 2022-04-04T00:12:11.186 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:11.184+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:11.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:11.523+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:12.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:12.025+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:12.026679+0000) 2022-04-04T00:12:12.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:12.223+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:12.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:12.532+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:13.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:13.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:13.026902+0000) 2022-04-04T00:12:13.255 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:13.252+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:13.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:13.485+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:14.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:14.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:14.027109+0000) 2022-04-04T00:12:14.220 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:14.218+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:14.471 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:14.468+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:15.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:15.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:15.027322+0000) 2022-04-04T00:12:15.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:15.227+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:15.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:15.479+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:16.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:16.027+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:16.027538+0000) 2022-04-04T00:12:16.228 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:16.226+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:16.508 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:16.506+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:17.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:17.027+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:17.027761+0000) 2022-04-04T00:12:17.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:17.184+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:17.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:17.511+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:18.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:18.027+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:18.028007+0000) 2022-04-04T00:12:18.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:18.180+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:18.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:18.492+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:19.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:19.027+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:19.028203+0000) 2022-04-04T00:12:19.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:19.176+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:19.510 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:19.508+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:20.030 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:20.027+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:20.028393+0000) 2022-04-04T00:12:20.180 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:20.177+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:20.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:20.557+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:21.030 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:21.028+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:21.028568+0000) 2022-04-04T00:12:21.191 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:21.189+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:21.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:21.602+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:22.030 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:22.028+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:22.028758+0000) 2022-04-04T00:12:22.157 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:22.155+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:22.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:22.639+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:23.030 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:23.028+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:23.028995+0000) 2022-04-04T00:12:23.118 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:23.116+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:23.666 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:23.663+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:24.030 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:24.028+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:24.029171+0000) 2022-04-04T00:12:24.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:24.125+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:24.671 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:12:24.669+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:12:24.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:24.686+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:25.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:25.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:25.029352+0000) 2022-04-04T00:12:25.098 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:25.096+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:25.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:25.693+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:26.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:26.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:26.029511+0000) 2022-04-04T00:12:26.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:26.130+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:26.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:26.683+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:27.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:27.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:27.029648+0000) 2022-04-04T00:12:27.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:27.138+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:27.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:27.695+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:28.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:28.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:28.029846+0000) 2022-04-04T00:12:28.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:28.186+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:28.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:28.727+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:29.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:29.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:29.030064+0000) 2022-04-04T00:12:29.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:29.186+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:29.230 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:12:29.233 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:12:29.689 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:29.686+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:30.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:30.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:30.030301+0000) 2022-04-04T00:12:30.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:30.158+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:30.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:30.670+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:31.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:31.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:31.030503+0000) 2022-04-04T00:12:31.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:31.180+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:31.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:31.621+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:32.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:32.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:32.030684+0000) 2022-04-04T00:12:32.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:32.153+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:32.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:32.666+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:33.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:33.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:33.030878+0000) 2022-04-04T00:12:33.113 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:33.110+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:33.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:33.665+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:34.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:34.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:34.031008+0000) 2022-04-04T00:12:34.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:34.114+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:34.676 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:34.674+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:35.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:35.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:35.031188+0000) 2022-04-04T00:12:35.068 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:35.066+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:35.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:35.682+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:36.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:36.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:36.031372+0000) 2022-04-04T00:12:36.090 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:36.088+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:36.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:36.664+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:37.033 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:37.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:37.031571+0000) 2022-04-04T00:12:37.104 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:37.102+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:37.625 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:37.623+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:38.033 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:38.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:38.031721+0000) 2022-04-04T00:12:38.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:38.080+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:38.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:38.611+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:39.033 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:39.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:39.031906+0000) 2022-04-04T00:12:39.077 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:39.075+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:39.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:39.599+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:39.687 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:12:39.683+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:12:40.033 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:40.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:40.032093+0000) 2022-04-04T00:12:40.087 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:40.086+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:40.629 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:40.627+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:41.033 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:41.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:41.032294+0000) 2022-04-04T00:12:41.106 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:41.104+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:41.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:41.652+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:42.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:42.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:42.032481+0000) 2022-04-04T00:12:42.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:42.069+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:42.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:42.695+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:43.027 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:43.025+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:43.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:43.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:43.032673+0000) 2022-04-04T00:12:43.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:43.712+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:44.011 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:44.009+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:44.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:44.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:44.032857+0000) 2022-04-04T00:12:44.708 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:44.706+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:44.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:44.963+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:45.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:45.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:45.033050+0000) 2022-04-04T00:12:45.744 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:45.742+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:45.987 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:45.985+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:46.035 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:46.032+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:46.033264+0000) 2022-04-04T00:12:46.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:46.789+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:46.994 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:46.992+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:47.035 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:47.032+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:47.033503+0000) 2022-04-04T00:12:47.745 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:47.743+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:48.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:48.011+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:48.035 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:48.033+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:48.033758+0000) 2022-04-04T00:12:48.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:48.700+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:48.986 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:48.984+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:49.035 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:49.033+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:49.033979+0000) 2022-04-04T00:12:49.710 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:49.708+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:50.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:50.012+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:50.035 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:50.033+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:50.034203+0000) 2022-04-04T00:12:50.717 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:50.714+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:50.978 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:50.975+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:51.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:51.033+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:51.034427+0000) 2022-04-04T00:12:51.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:51.679+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:52.011 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:52.009+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:52.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:52.033+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:52.034668+0000) 2022-04-04T00:12:52.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:52.636+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:52.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:52.979+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:53.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:53.034+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:53.034872+0000) 2022-04-04T00:12:53.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:53.615+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:53.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:53.935+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:54.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:54.034+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:54.035017+0000) 2022-04-04T00:12:54.607 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:54.604+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:54.689 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:12:54.687+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:12:54.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:54.944+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:55.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:55.034+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:55.035200+0000) 2022-04-04T00:12:55.634 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:55.631+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:55.910 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:55.907+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:56.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:56.034+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:56.035339+0000) 2022-04-04T00:12:56.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:56.637+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:56.941 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:56.938+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:57.037 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:57.034+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:57.035526+0000) 2022-04-04T00:12:57.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:57.612+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:57.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:57.947+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:58.037 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:58.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:58.035726+0000) 2022-04-04T00:12:58.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:58.571+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:58.906 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:58.903+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:12:59.037 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:12:59.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:12:59.035935+0000) 2022-04-04T00:12:59.370 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:12:59.374 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:12:59.621 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:12:59.618+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:12:59.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:12:59.891+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:00.037 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:00.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:00.036175+0000) 2022-04-04T00:13:00.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:00.632+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:00.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:00.860+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:01.037 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:01.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:01.036379+0000) 2022-04-04T00:13:01.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:01.602+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:01.859 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:01.857+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:02.038 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:02.036+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:02.036618+0000) 2022-04-04T00:13:02.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:02.553+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:02.856 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:02.853+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:03.038 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:03.036+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:03.036872+0000) 2022-04-04T00:13:03.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:03.524+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:03.871 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:03.868+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:04.038 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:04.036+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:04.037077+0000) 2022-04-04T00:13:04.479 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:04.476+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:04.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:04.855+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:05.039 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:05.036+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:05.037282+0000) 2022-04-04T00:13:05.501 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:05.498+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:05.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:05.894+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:06.039 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:06.037+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:06.037505+0000) 2022-04-04T00:13:06.471 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:06.469+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:06.915 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:06.912+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:07.039 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:07.037+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:07.037657+0000) 2022-04-04T00:13:07.460 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:07.457+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:07.867 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:07.864+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:08.039 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:08.037+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:08.037876+0000) 2022-04-04T00:13:08.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:08.459+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:08.866 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:08.864+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:09.039 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:09.037+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:09.038093+0000) 2022-04-04T00:13:09.501 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:09.499+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:09.705 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:13:09.703+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:13:09.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:09.857+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:10.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:10.037+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:10.038315+0000) 2022-04-04T00:13:10.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:10.489+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:10.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:10.856+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:11.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:11.038+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:11.038533+0000) 2022-04-04T00:13:11.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:11.445+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:11.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:11.862+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:12.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:12.038+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:12.038759+0000) 2022-04-04T00:13:12.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:12.427+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:12.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:12.856+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:13.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:13.038+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:13.039001+0000) 2022-04-04T00:13:13.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:13.413+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:13.902 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:13.899+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:14.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:14.038+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:14.039182+0000) 2022-04-04T00:13:14.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:14.391+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:14.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:14.948+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:15.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:15.038+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:15.039350+0000) 2022-04-04T00:13:15.426 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:15.423+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:15.996 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:15.993+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:16.041 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:16.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:16.039605+0000) 2022-04-04T00:13:16.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:16.411+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:17.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:17.023+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:17.041 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:17.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:17.039846+0000) 2022-04-04T00:13:17.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:17.439+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:18.041 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:18.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:18.040067+0000) 2022-04-04T00:13:18.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:18.065+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:18.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:18.478+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:19.041 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:19.040+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:19.040285+0000) 2022-04-04T00:13:19.078 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:19.076+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:19.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:19.478+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:20.042 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:20.040+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:20.040509+0000) 2022-04-04T00:13:20.087 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:20.085+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:20.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:20.502+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:21.042 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:21.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:21.040749+0000) 2022-04-04T00:13:21.089 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:21.087+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:21.517 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:21.514+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:22.042 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:22.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:22.041045+0000) 2022-04-04T00:13:22.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:22.115+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:22.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:22.540+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:23.042 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:23.040+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:23.041267+0000) 2022-04-04T00:13:23.113 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:23.111+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:23.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:23.564+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:24.043 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:24.040+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:24.041465+0000) 2022-04-04T00:13:24.115 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:24.113+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:24.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:24.584+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:24.721 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:13:24.718+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:13:25.043 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:25.040+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:25.041683+0000) 2022-04-04T00:13:25.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:25.151+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:25.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:25.633+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:26.043 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:26.040+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:26.041908+0000) 2022-04-04T00:13:26.186 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:26.183+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:26.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:26.647+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:27.043 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:27.041+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:27.042129+0000) 2022-04-04T00:13:27.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:27.185+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:27.606 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:27.603+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:28.043 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:28.040+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:28.042364+0000) 2022-04-04T00:13:28.177 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:28.174+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:28.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:28.558+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:29.044 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:29.041+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:29.042602+0000) 2022-04-04T00:13:29.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:29.130+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:29.516 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:13:29.519 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:13:29.570 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:29.567+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:30.044 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:30.041+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:30.042857+0000) 2022-04-04T00:13:30.104 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:30.102+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:30.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:30.534+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:31.044 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:31.042+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:31.043074+0000) 2022-04-04T00:13:31.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:31.093+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:31.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:31.489+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:32.044 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:32.042+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:32.043266+0000) 2022-04-04T00:13:32.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:32.053+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:32.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:32.445+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:33.020 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:33.018+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:33.044 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:33.042+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:33.043409+0000) 2022-04-04T00:13:33.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:33.495+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:33.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:33.990+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:34.045 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:34.042+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:34.043615+0000) 2022-04-04T00:13:34.486 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:34.484+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:35.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:35.031+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:35.045 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:35.043+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:35.043849+0000) 2022-04-04T00:13:35.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:35.530+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:36.030 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:36.028+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:36.045 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:36.043+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:36.044051+0000) 2022-04-04T00:13:36.490 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:36.488+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:37.031 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:37.029+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:37.045 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:37.043+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:37.044276+0000) 2022-04-04T00:13:37.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:37.441+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:38.046 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:38.043+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:38.044491+0000) 2022-04-04T00:13:38.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:38.065+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:38.480 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:38.478+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:39.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:39.035+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:39.046 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:39.044+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:39.044696+0000) 2022-04-04T00:13:39.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:39.527+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:39.737 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:13:39.734+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:13:40.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:40.018+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:40.046 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:40.044+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:40.044866+0000) 2022-04-04T00:13:40.502 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:40.500+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:41.046 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:41.044+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:41.045074+0000) 2022-04-04T00:13:41.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:41.058+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:41.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:41.540+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:42.046 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:42.044+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:42.045275+0000) 2022-04-04T00:13:42.102 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:42.100+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:42.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:42.551+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:43.047 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:43.044+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:43.045477+0000) 2022-04-04T00:13:43.072 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:43.070+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:43.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:43.539+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:44.047 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:44.045+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:44.045637+0000) 2022-04-04T00:13:44.060 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:44.058+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:44.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:44.552+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:45.047 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:45.045+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:45.045880+0000) 2022-04-04T00:13:45.070 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:45.068+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:45.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:45.554+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:46.047 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:46.045+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:46.046043+0000) 2022-04-04T00:13:46.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:46.082+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:46.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:46.577+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:47.047 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:47.045+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:47.046232+0000) 2022-04-04T00:13:47.104 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:47.101+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:47.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:47.609+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:48.048 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:48.045+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:48.046405+0000) 2022-04-04T00:13:48.149 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:48.146+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:48.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:48.657+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:49.048 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:49.046+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:49.046587+0000) 2022-04-04T00:13:49.166 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:49.164+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:49.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:49.635+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:50.048 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:50.046+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:50.046803+0000) 2022-04-04T00:13:50.172 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:50.169+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:50.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:50.640+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:51.048 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:51.046+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:51.046978+0000) 2022-04-04T00:13:51.158 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:51.155+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:51.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:51.685+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:52.048 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:52.046+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:52.047207+0000) 2022-04-04T00:13:52.168 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:52.165+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:52.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:52.646+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:53.048 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:53.047+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:53.047428+0000) 2022-04-04T00:13:53.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:53.179+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:53.609 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:53.606+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:54.049 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:54.047+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:54.047644+0000) 2022-04-04T00:13:54.199 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:54.196+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:54.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:54.599+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:54.738 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:13:54.736+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:13:55.049 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:55.047+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:55.047852+0000) 2022-04-04T00:13:55.196 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:55.193+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:55.623 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:55.620+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:56.049 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:56.047+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:56.048052+0000) 2022-04-04T00:13:56.155 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:56.152+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:56.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:56.663+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:57.049 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:57.047+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:57.048249+0000) 2022-04-04T00:13:57.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:57.200+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:57.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:57.651+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:58.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:58.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:58.048442+0000) 2022-04-04T00:13:58.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:58.154+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:58.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:58.672+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:59.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:13:59.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:13:59.048598+0000) 2022-04-04T00:13:59.145 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:13:59.142+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:13:59.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:13:59.641+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:13:59.655 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:13:59.659 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:14:00.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:00.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:00.048767+0000) 2022-04-04T00:14:00.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:00.186+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:00.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:00.635+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:01.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:01.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:01.048950+0000) 2022-04-04T00:14:01.149 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:01.146+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:01.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:01.625+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:02.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:02.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:02.049153+0000) 2022-04-04T00:14:02.160 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:02.158+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:02.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:02.591+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:03.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:03.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:03.049340+0000) 2022-04-04T00:14:03.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:03.176+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:03.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:03.587+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:04.051 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:04.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:04.049547+0000) 2022-04-04T00:14:04.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:04.163+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:04.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:04.587+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:05.051 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:05.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:05.049701+0000) 2022-04-04T00:14:05.214 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:05.211+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:05.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:05.538+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:06.051 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:06.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:06.049905+0000) 2022-04-04T00:14:06.221 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:06.218+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:06.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:06.577+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:07.051 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:07.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:07.050042+0000) 2022-04-04T00:14:07.210 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:07.208+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:07.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:07.612+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:08.051 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:08.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:08.050229+0000) 2022-04-04T00:14:08.186 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:08.183+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:08.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:08.594+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:09.051 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:09.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:09.050379+0000) 2022-04-04T00:14:09.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:09.170+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:09.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:09.560+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:09.754 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:14:09.751+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:14:10.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:10.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:10.050565+0000) 2022-04-04T00:14:10.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:10.134+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:10.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:10.594+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:11.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:11.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:11.050767+0000) 2022-04-04T00:14:11.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:11.101+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:11.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:11.615+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:12.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:12.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:12.050930+0000) 2022-04-04T00:14:12.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:12.085+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:12.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:12.622+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:13.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:13.045+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:13.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:13.050+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:13.051070+0000) 2022-04-04T00:14:13.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:13.658+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:14.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:14.050+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:14.051274+0000) 2022-04-04T00:14:14.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:14.053+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:14.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:14.682+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:15.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:15.050+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:15.051444+0000) 2022-04-04T00:14:15.053 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:15.051+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:15.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:15.716+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:16.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:16.046+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:16.053 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:16.050+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:16.051629+0000) 2022-04-04T00:14:16.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:16.695+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:17.053 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:17.051+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:17.051803+0000) 2022-04-04T00:14:17.060 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:17.058+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:17.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:17.649+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:18.050 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:18.048+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:18.053 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:18.051+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:18.051949+0000) 2022-04-04T00:14:18.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:18.625+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:19.053 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:19.051+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:19.052153+0000) 2022-04-04T00:14:19.069 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:19.067+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:19.607 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:19.605+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:20.020 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:20.018+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:20.053 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:20.051+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:20.052322+0000) 2022-04-04T00:14:20.585 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:20.583+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:20.994 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:20.991+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:21.054 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:21.051+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:21.052508+0000) 2022-04-04T00:14:21.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:21.573+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:22.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:22.022+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:22.054 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:22.051+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:22.052701+0000) 2022-04-04T00:14:22.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:22.536+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:22.996 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:22.994+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:23.054 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:23.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:23.052935+0000) 2022-04-04T00:14:23.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:23.566+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:23.963 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:23.961+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:24.054 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:24.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:24.053120+0000) 2022-04-04T00:14:24.573 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:24.571+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:24.771 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:14:24.768+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:14:24.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:24.980+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:25.054 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:25.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:25.053342+0000) 2022-04-04T00:14:25.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:25.559+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:25.971 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:25.969+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:26.055 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:26.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:26.053547+0000) 2022-04-04T00:14:26.552 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:26.550+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:26.976 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:26.974+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:27.055 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:27.053+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:27.053714+0000) 2022-04-04T00:14:27.523 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:27.521+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:27.954 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:27.952+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:28.055 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:28.053+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:28.053903+0000) 2022-04-04T00:14:28.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:28.529+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:28.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:28.945+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:29.055 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:29.053+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:29.054087+0000) 2022-04-04T00:14:29.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:29.502+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:29.794 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:14:29.798 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:14:29.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:29.920+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:30.055 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:30.053+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:30.054266+0000) 2022-04-04T00:14:30.508 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:30.506+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:30.932 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:30.930+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:31.055 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:31.053+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:31.054460+0000) 2022-04-04T00:14:31.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:31.536+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:31.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:31.929+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:32.056 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:32.054+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:32.054666+0000) 2022-04-04T00:14:32.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:32.577+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:32.902 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:32.900+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:33.056 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:33.054+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:33.054870+0000) 2022-04-04T00:14:33.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:33.565+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:33.902 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:33.900+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:34.056 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:34.054+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:34.055091+0000) 2022-04-04T00:14:34.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:34.561+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:34.916 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:34.914+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:35.057 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:35.054+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:35.055312+0000) 2022-04-04T00:14:35.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:35.565+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:35.880 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:35.878+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:36.057 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:36.055+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:36.055552+0000) 2022-04-04T00:14:36.526 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:36.524+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:36.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:36.833+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:37.057 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:37.055+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:37.055741+0000) 2022-04-04T00:14:37.537 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:37.535+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:37.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:37.827+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:38.057 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:38.055+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:38.055916+0000) 2022-04-04T00:14:38.583 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:38.581+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:38.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:38.786+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:39.057 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:39.055+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:39.056075+0000) 2022-04-04T00:14:39.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:39.576+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:39.786 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:14:39.784+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:14:39.817 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:39.815+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:40.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:40.055+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:40.056278+0000) 2022-04-04T00:14:40.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:40.568+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:40.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:40.775+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:41.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:41.056+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:41.056506+0000) 2022-04-04T00:14:41.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:41.585+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:41.786 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:41.783+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:42.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:42.056+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:42.056720+0000) 2022-04-04T00:14:42.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:42.623+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:42.804 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:42.801+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:43.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:43.056+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:43.056895+0000) 2022-04-04T00:14:43.583 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:43.580+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:43.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:43.772+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:44.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:44.056+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:44.057057+0000) 2022-04-04T00:14:44.600 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:44.597+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:44.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:44.728+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:45.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:45.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:45.057281+0000) 2022-04-04T00:14:45.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:45.629+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:45.716 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:45.714+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:46.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:46.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:46.057474+0000) 2022-04-04T00:14:46.662 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:46.659+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:46.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:46.740+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:47.059 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:47.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:47.057646+0000) 2022-04-04T00:14:47.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:47.646+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:47.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:47.786+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:48.059 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:48.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:48.057856+0000) 2022-04-04T00:14:48.623 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:48.620+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:48.751 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:48.748+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:49.059 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:49.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:49.058058+0000) 2022-04-04T00:14:49.607 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:49.604+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:49.751 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:49.748+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:50.059 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:50.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:50.058250+0000) 2022-04-04T00:14:50.582 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:50.579+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:50.724 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:50.720+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:51.059 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:51.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:51.058463+0000) 2022-04-04T00:14:51.595 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:51.592+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:51.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:51.723+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:52.060 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:52.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:52.058677+0000) 2022-04-04T00:14:52.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:52.559+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:52.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:52.758+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:53.060 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:53.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:53.058856+0000) 2022-04-04T00:14:53.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:53.588+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:53.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:53.732+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:54.060 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:54.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:54.059035+0000) 2022-04-04T00:14:54.546 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:54.544+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:54.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:54.699+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:54.802 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:14:54.799+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:14:55.060 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:55.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:55.059191+0000) 2022-04-04T00:14:55.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:55.565+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:55.705 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:55.702+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:56.060 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:56.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:56.059365+0000) 2022-04-04T00:14:56.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:56.598+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:56.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:56.686+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:57.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:57.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:57.059558+0000) 2022-04-04T00:14:57.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:57.615+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:57.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:57.688+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:58.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:58.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:58.059790+0000) 2022-04-04T00:14:58.664 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:58.661+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:58.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:58.666+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:59.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:14:59.059+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:14:59.059930+0000) 2022-04-04T00:14:59.633 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:14:59.630+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:14:59.662 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:14:59.659+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:14:59.943 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:14:59.947 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:15:00.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:00.059+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:00.060105+0000) 2022-04-04T00:15:00.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:00.592+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:00.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:00.647+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:01.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:01.059+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:01.060315+0000) 2022-04-04T00:15:01.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:01.570+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:01.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:01.647+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:02.062 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:02.059+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:02.060518+0000) 2022-04-04T00:15:02.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:02.529+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:02.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:02.657+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:03.062 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:03.059+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:03.060730+0000) 2022-04-04T00:15:03.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:03.494+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:03.656 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:03.653+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:04.062 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:04.060+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:04.060957+0000) 2022-04-04T00:15:04.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:04.509+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:04.689 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:04.687+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:05.062 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:05.060+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:05.061165+0000) 2022-04-04T00:15:05.520 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:05.517+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:05.687 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:05.685+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:06.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:06.060+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:06.061360+0000) 2022-04-04T00:15:06.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:06.544+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:06.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:06.661+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:07.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:07.060+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:07.061572+0000) 2022-04-04T00:15:07.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:07.591+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:07.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:07.613+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:08.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:08.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:08.061805+0000) 2022-04-04T00:15:08.582 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:08.579+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:08.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:08.647+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:09.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:09.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:09.062005+0000) 2022-04-04T00:15:09.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:09.616+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:09.631 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:09.629+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:09.818 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:15:09.815+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:15:10.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:10.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:10.062221+0000) 2022-04-04T00:15:10.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:10.574+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:10.637 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:10.633+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:11.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:11.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:11.062444+0000) 2022-04-04T00:15:11.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:11.581+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:11.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:11.600+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:12.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:12.062+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:12.062621+0000) 2022-04-04T00:15:12.582 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:12.579+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:12.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:12.582+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:13.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:13.062+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:13.062848+0000) 2022-04-04T00:15:13.602 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:13.599+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:13.609 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:13.607+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:14.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:14.062+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:14.063005+0000) 2022-04-04T00:15:14.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:14.552+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:14.589 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:14.586+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:15.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:15.062+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:15.063207+0000) 2022-04-04T00:15:15.537 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:15.534+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:15.547 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:15.544+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:16.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:16.062+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:16.063373+0000) 2022-04-04T00:15:16.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:16.516+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:16.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:16.536+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:17.065 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:17.063+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:17.063572+0000) 2022-04-04T00:15:17.510 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:17.507+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:17.555 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:17.552+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:18.065 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:18.063+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:18.063729+0000) 2022-04-04T00:15:18.486 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:18.483+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:18.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:18.542+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:19.065 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:19.063+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:19.063926+0000) 2022-04-04T00:15:19.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:19.480+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:19.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:19.508+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:20.065 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:20.063+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:20.064130+0000) 2022-04-04T00:15:20.438 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:20.436+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:20.533 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:20.531+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:21.065 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:21.063+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:21.064326+0000) 2022-04-04T00:15:21.436 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:21.434+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:21.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:21.570+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:22.066 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:22.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:22.064541+0000) 2022-04-04T00:15:22.451 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:22.449+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:22.578 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:22.576+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:23.066 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:23.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:23.064752+0000) 2022-04-04T00:15:23.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:23.465+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:23.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:23.601+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:24.066 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:24.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:24.064982+0000) 2022-04-04T00:15:24.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:24.442+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:24.575 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:24.573+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:24.833 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:15:24.831+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:15:25.066 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:25.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:25.065180+0000) 2022-04-04T00:15:25.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:25.446+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:25.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:25.621+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:26.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:26.065+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:26.065388+0000) 2022-04-04T00:15:26.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:26.433+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:26.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:26.638+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:27.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:27.065+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:27.065633+0000) 2022-04-04T00:15:27.424 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:27.422+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:27.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:27.668+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:28.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:28.065+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:28.065869+0000) 2022-04-04T00:15:28.450 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:28.448+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:28.627 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:28.625+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:29.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:29.065+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:29.066091+0000) 2022-04-04T00:15:29.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:29.417+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:29.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:29.585+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:30.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:30.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:30.066233+0000) 2022-04-04T00:15:30.074 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:15:30.078 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:15:30.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:30.440+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:30.591 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:30.590+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:31.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:31.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:31.066415+0000) 2022-04-04T00:15:31.396 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:31.395+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:31.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:31.623+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:32.068 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:32.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:32.066592+0000) 2022-04-04T00:15:32.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:32.433+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:32.654 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:32.652+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:33.068 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:33.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:33.066790+0000) 2022-04-04T00:15:33.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:33.456+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:33.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:33.698+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:34.068 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:34.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:34.067008+0000) 2022-04-04T00:15:34.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:34.407+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:34.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:34.727+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:35.068 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:35.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:35.067268+0000) 2022-04-04T00:15:35.434 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:35.432+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:35.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:35.726+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:36.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:36.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:36.067454+0000) 2022-04-04T00:15:36.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:36.437+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:36.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:36.734+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:37.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:37.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:37.067650+0000) 2022-04-04T00:15:37.468 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:37.465+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:37.756 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:37.753+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:38.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:38.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:38.067888+0000) 2022-04-04T00:15:38.498 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:38.496+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:38.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:38.778+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:39.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:39.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:39.068092+0000) 2022-04-04T00:15:39.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:39.485+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:39.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:39.788+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:39.849 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:15:39.846+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:15:40.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:40.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:40.068258+0000) 2022-04-04T00:15:40.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:40.512+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:40.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:40.835+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:41.070 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:41.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:41.068479+0000) 2022-04-04T00:15:41.552 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:41.549+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:41.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:41.805+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:42.070 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:42.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:42.068732+0000) 2022-04-04T00:15:42.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:42.584+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:42.854 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:42.851+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:43.070 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:43.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:43.068995+0000) 2022-04-04T00:15:43.603 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:43.600+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:43.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:43.837+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:44.070 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:44.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:44.069237+0000) 2022-04-04T00:15:44.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:44.646+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:44.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:44.848+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:45.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:45.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:45.069495+0000) 2022-04-04T00:15:45.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:45.634+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:45.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:45.828+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:46.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:46.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:46.069732+0000) 2022-04-04T00:15:46.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:46.588+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:46.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:46.857+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:47.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:47.069+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:47.069929+0000) 2022-04-04T00:15:47.550 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:47.547+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:47.891 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:47.888+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:48.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:48.069+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:48.070173+0000) 2022-04-04T00:15:48.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:48.576+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:48.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:48.914+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:49.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:49.069+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:49.070350+0000) 2022-04-04T00:15:49.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:49.545+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:49.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:49.915+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:50.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:50.069+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:50.070587+0000) 2022-04-04T00:15:50.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:50.558+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:50.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:50.920+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:51.072 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:51.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:51.070857+0000) 2022-04-04T00:15:51.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:51.512+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:51.919 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:51.917+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:52.072 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:52.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:52.071117+0000) 2022-04-04T00:15:52.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:52.496+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:52.915 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:52.912+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:53.072 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:53.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:53.071366+0000) 2022-04-04T00:15:53.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:53.527+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:53.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:53.894+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:54.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:54.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:54.071553+0000) 2022-04-04T00:15:54.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:54.518+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:54.865 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:15:54.863+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:15:54.939 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:54.936+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:55.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:55.071+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:55.071790+0000) 2022-04-04T00:15:55.570 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:55.568+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:55.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:55.900+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:56.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:56.071+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:56.072045+0000) 2022-04-04T00:15:56.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:56.531+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:56.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:56.926+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:57.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:57.071+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:57.072216+0000) 2022-04-04T00:15:57.564 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:57.561+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:57.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:57.954+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:58.074 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:58.071+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:58.072432+0000) 2022-04-04T00:15:58.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:58.542+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:58.954 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:58.952+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:15:59.074 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:15:59.072+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:15:59.072676+0000) 2022-04-04T00:15:59.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:15:59.584+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:15:59.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:15:59.947+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:00.074 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:00.072+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:00.072892+0000) 2022-04-04T00:16:00.226 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:16:00.230 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:16:00.570 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:00.567+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:00.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:00.935+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:01.074 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:01.072+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:01.073161+0000) 2022-04-04T00:16:01.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:01.539+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:01.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:01.981+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:02.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:02.072+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:02.073423+0000) 2022-04-04T00:16:02.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:02.573+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:03.001 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:02.998+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:03.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:03.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:03.073669+0000) 2022-04-04T00:16:03.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:03.588+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:03.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:03.979+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:04.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:04.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:04.073914+0000) 2022-04-04T00:16:04.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:04.635+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:04.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:04.979+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:05.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:05.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:05.074163+0000) 2022-04-04T00:16:05.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:05.617+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:06.002 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:05.999+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:06.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:06.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:06.074361+0000) 2022-04-04T00:16:06.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:06.642+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:07.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:07.016+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:07.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:07.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:07.074578+0000) 2022-04-04T00:16:07.634 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:07.631+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:08.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:08.057+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:08.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:08.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:08.074799+0000) 2022-04-04T00:16:08.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:08.678+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:09.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:09.046+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:09.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:09.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:09.075024+0000) 2022-04-04T00:16:09.712 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:09.709+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:09.881 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:16:09.879+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:16:10.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:10.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:10.075263+0000) 2022-04-04T00:16:10.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:10.081+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:10.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:10.665+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:11.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:11.075+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:11.075482+0000) 2022-04-04T00:16:11.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:11.108+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:11.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:11.698+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:12.077 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:12.075+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:12.075693+0000) 2022-04-04T00:16:12.158 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:12.155+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:12.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:12.693+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:13.077 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:13.075+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:13.075866+0000) 2022-04-04T00:16:13.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:13.124+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:13.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:13.740+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:14.077 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:14.075+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:14.076040+0000) 2022-04-04T00:16:14.090 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:14.087+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:14.752 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:14.749+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:15.077 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:15.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:15.076230+0000) 2022-04-04T00:16:15.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:15.132+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:15.727 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:15.724+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:16.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:16.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:16.076473+0000) 2022-04-04T00:16:16.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:16.093+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:16.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:16.753+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:17.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:17.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:17.076663+0000) 2022-04-04T00:16:17.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:17.126+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:17.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:17.709+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:18.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:18.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:18.076912+0000) 2022-04-04T00:16:18.149 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:18.147+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:18.691 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:18.689+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:19.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:19.077+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:19.077098+0000) 2022-04-04T00:16:19.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:19.168+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:19.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:19.719+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:20.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:20.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:20.077238+0000) 2022-04-04T00:16:20.142 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:20.140+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:20.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:20.686+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:21.079 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:21.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:21.077443+0000) 2022-04-04T00:16:21.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:21.141+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:21.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:21.707+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:22.079 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:22.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:22.077665+0000) 2022-04-04T00:16:22.184 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:22.182+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:22.674 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:22.672+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:23.079 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:23.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:23.077887+0000) 2022-04-04T00:16:23.168 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:23.166+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:23.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:23.696+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:24.079 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:24.077+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:24.078114+0000) 2022-04-04T00:16:24.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:24.120+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:24.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:24.735+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:24.897 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:16:24.894+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:16:25.079 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:25.077+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:25.078261+0000) 2022-04-04T00:16:25.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:25.121+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:25.710 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:25.708+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:26.077 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:26.075+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:26.079 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:26.077+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:26.078452+0000) 2022-04-04T00:16:26.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:26.752+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:27.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:27.026+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:27.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:27.077+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:27.078678+0000) 2022-04-04T00:16:27.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:27.735+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:28.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:28.057+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:28.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:28.077+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:28.078891+0000) 2022-04-04T00:16:28.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:28.739+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:29.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:29.078+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:29.079037+0000) 2022-04-04T00:16:29.090 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:29.088+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:29.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:29.694+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:30.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:30.078+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:30.079177+0000) 2022-04-04T00:16:30.130 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:30.128+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:30.353 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:16:30.357 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:16:30.674 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:30.672+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:31.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:31.078+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:31.079411+0000) 2022-04-04T00:16:31.169 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:31.166+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:31.634 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:31.631+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:32.081 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:32.078+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:32.079607+0000) 2022-04-04T00:16:32.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:32.175+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:32.655 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:32.652+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:33.081 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:33.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:33.079832+0000) 2022-04-04T00:16:33.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:33.185+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:33.633 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:33.630+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:34.081 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:34.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:34.080033+0000) 2022-04-04T00:16:34.151 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:34.148+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:34.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:34.666+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:35.081 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:35.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:35.080250+0000) 2022-04-04T00:16:35.119 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:35.117+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:35.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:35.713+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:36.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:36.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:36.080468+0000) 2022-04-04T00:16:36.152 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:36.150+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:36.752 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:36.749+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:37.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:37.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:37.080676+0000) 2022-04-04T00:16:37.138 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:37.135+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:37.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:37.788+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:38.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:38.080+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:38.080886+0000) 2022-04-04T00:16:38.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:38.170+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:38.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:38.747+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:39.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:39.080+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:39.081078+0000) 2022-04-04T00:16:39.130 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:39.127+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:39.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:39.755+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:39.901 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:16:39.898+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:16:40.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:40.080+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:40.081315+0000) 2022-04-04T00:16:40.084 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:40.081+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:40.712 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:40.710+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:41.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:41.072+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:41.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:41.080+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:41.081543+0000) 2022-04-04T00:16:41.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:41.668+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:42.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:42.056+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:42.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:42.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:42.081705+0000) 2022-04-04T00:16:42.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:42.713+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:43.073 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:43.070+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:43.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:43.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:43.081913+0000) 2022-04-04T00:16:43.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:43.754+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:44.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:44.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:44.082141+0000) 2022-04-04T00:16:44.114 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:44.112+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:44.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:44.710+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:45.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:45.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:45.082331+0000) 2022-04-04T00:16:45.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:45.125+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:45.666 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:45.663+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:46.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:46.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:46.082500+0000) 2022-04-04T00:16:46.157 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:46.155+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:46.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:46.697+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:47.084 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:47.082+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:47.082657+0000) 2022-04-04T00:16:47.104 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-04T00:16:47.102+0000 7f2e4331e700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T00:16:47.105 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-04T00:16:47.102+0000 7f2e43b1f700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T00:16:47.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:47.176+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:47.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:47.694+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:48.084 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:48.082+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:48.082870+0000) 2022-04-04T00:16:48.144 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:48.141+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:48.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:48.660+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:49.084 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:49.082+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:49.083109+0000) 2022-04-04T00:16:49.192 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:49.189+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:49.659 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:49.656+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:50.084 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:50.082+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:50.083325+0000) 2022-04-04T00:16:50.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:50.186+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:50.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:50.679+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:51.085 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:51.083+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:51.083549+0000) 2022-04-04T00:16:51.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:51.228+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:51.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:51.712+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:52.085 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:52.083+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:52.083759+0000) 2022-04-04T00:16:52.223 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:52.221+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:52.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:52.729+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:53.085 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:53.083+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:53.083963+0000) 2022-04-04T00:16:53.247 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:53.244+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:53.754 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:53.751+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:54.085 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:54.083+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:54.084227+0000) 2022-04-04T00:16:54.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:54.274+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:54.765 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:54.762+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:54.916 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:16:54.914+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:16:55.085 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:55.084+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:55.084465+0000) 2022-04-04T00:16:55.287 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:55.285+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:55.792 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:55.789+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:56.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:56.084+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:56.084688+0000) 2022-04-04T00:16:56.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:56.241+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:56.837 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:56.834+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:57.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:57.084+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:57.084933+0000) 2022-04-04T00:16:57.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:57.229+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:57.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:57.812+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:58.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:58.084+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:58.085153+0000) 2022-04-04T00:16:58.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:58.226+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:58.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:58.840+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:16:59.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:16:59.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:16:59.085346+0000) 2022-04-04T00:16:59.196 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:16:59.193+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:16:59.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:16:59.810+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:00.087 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:00.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:00.085566+0000) 2022-04-04T00:17:00.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:00.193+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:00.429 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:17:00.433 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:17:00.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:00.799+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:01.087 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:01.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:01.085857+0000) 2022-04-04T00:17:01.213 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:01.210+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:01.763 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:01.761+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:02.087 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:02.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:02.086123+0000) 2022-04-04T00:17:02.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:02.246+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:02.717 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:02.714+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:03.087 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:03.086+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:03.086362+0000) 2022-04-04T00:17:03.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:03.201+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:03.724 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:03.721+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:04.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:04.086+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:04.086600+0000) 2022-04-04T00:17:04.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:04.229+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:04.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:04.710+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:05.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:05.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:05.086804+0000) 2022-04-04T00:17:05.205 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:05.203+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:05.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:05.754+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:06.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:06.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:06.086952+0000) 2022-04-04T00:17:06.237 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:06.235+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:06.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:06.729+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:07.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:07.086+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:07.087098+0000) 2022-04-04T00:17:07.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:07.186+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:07.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:07.694+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:08.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:08.086+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:08.087317+0000) 2022-04-04T00:17:08.234 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:08.232+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:08.739 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:08.736+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:09.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:09.086+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:09.087522+0000) 2022-04-04T00:17:09.234 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:09.231+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:09.720 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:09.718+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:09.932 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:17:09.929+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:17:10.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:10.086+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:10.087736+0000) 2022-04-04T00:17:10.185 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:10.183+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:10.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:10.764+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:11.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:11.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:11.088001+0000) 2022-04-04T00:17:11.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:11.195+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:11.772 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:11.770+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:12.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:12.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:12.088201+0000) 2022-04-04T00:17:12.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:12.224+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:12.783 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:12.781+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:13.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:13.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:13.088413+0000) 2022-04-04T00:17:13.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:13.201+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:13.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:13.830+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:14.090 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:14.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:14.088557+0000) 2022-04-04T00:17:14.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:14.193+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:14.844 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:14.843+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:15.090 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:15.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:15.088697+0000) 2022-04-04T00:17:15.221 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:15.219+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:15.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:15.860+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:16.090 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:16.088+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:16.088883+0000) 2022-04-04T00:17:16.261 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:16.259+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:16.895 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:16.893+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:17.090 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:17.088+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:17.089098+0000) 2022-04-04T00:17:17.306 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:17.304+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:17.874 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:17.872+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:18.090 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:18.088+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:18.089324+0000) 2022-04-04T00:17:18.291 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:18.289+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:18.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:18.907+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:19.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:19.088+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:19.089507+0000) 2022-04-04T00:17:19.245 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:19.243+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:19.946 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:19.944+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:20.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:20.088+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:20.089723+0000) 2022-04-04T00:17:20.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:20.196+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:20.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:20.994+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:21.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:21.089+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:21.089916+0000) 2022-04-04T00:17:21.166 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:21.164+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:22.024 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:22.022+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:22.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:22.089+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:22.090163+0000) 2022-04-04T00:17:22.149 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:22.147+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:23.031 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:23.029+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:23.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:23.089+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:23.090394+0000) 2022-04-04T00:17:23.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:23.145+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:24.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:24.010+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:24.092 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:24.089+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:24.090609+0000) 2022-04-04T00:17:24.151 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:24.149+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:24.934 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:17:24.931+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:17:25.002 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:25.000+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:25.092 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:25.090+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:25.090815+0000) 2022-04-04T00:17:25.194 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:25.192+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:17:25.959 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:25.957+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:26.092 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:26.090+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:26.090985+0000) 2022-04-04T00:17:26.240 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:26.237+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:26.985 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:26.982+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:27.092 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:27.090+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:27.091154+0000) 2022-04-04T00:17:27.238 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:27.235+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:28.018 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:28.015+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:28.092 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:28.090+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:28.091372+0000) 2022-04-04T00:17:28.239 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:28.236+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:29.060 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:29.057+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:29.093 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:29.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:29.091642+0000) 2022-04-04T00:17:29.205 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:29.202+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:30.059 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:30.056+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:30.093 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:30.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:30.091909+0000) 2022-04-04T00:17:30.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:30.228+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:30.567 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:17:30.571 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:17:31.086 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:31.083+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:31.093 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:31.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:31.092183+0000) 2022-04-04T00:17:31.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:31.208+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:32.090 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:32.087+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:32.093 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:32.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:32.092437+0000) 2022-04-04T00:17:32.168 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:32.165+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:33.058 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:33.055+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:33.094 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:33.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:33.092649+0000) 2022-04-04T00:17:33.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:33.175+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:34.094 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:34.092+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:34.092901+0000) 2022-04-04T00:17:34.104 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:34.102+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:34.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:34.223+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:35.094 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:35.092+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:35.093146+0000) 2022-04-04T00:17:35.122 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:35.119+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:35.230 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:35.227+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:36.095 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:36.092+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:36.093391+0000) 2022-04-04T00:17:36.119 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:36.116+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:36.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:36.247+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:37.095 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:37.093+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:37.093593+0000) 2022-04-04T00:17:37.135 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:37.132+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:37.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:37.295+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:38.095 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:38.093+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:38.093853+0000) 2022-04-04T00:17:38.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:38.112+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:38.337 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:38.334+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:39.095 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:39.093+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:39.094067+0000) 2022-04-04T00:17:39.147 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:39.145+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:39.338 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:39.335+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:39.945 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:17:39.943+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:17:40.095 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:40.093+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:40.094211+0000) 2022-04-04T00:17:40.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:40.181+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:40.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:40.375+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:41.096 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:41.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:41.094429+0000) 2022-04-04T00:17:41.176 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:41.173+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:41.350 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:41.348+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:42.096 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:42.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:42.094650+0000) 2022-04-04T00:17:42.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:42.175+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:42.364 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:42.361+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:43.096 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:43.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:43.094831+0000) 2022-04-04T00:17:43.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:43.182+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:43.350 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:43.347+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:44.096 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:44.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:44.094956+0000) 2022-04-04T00:17:44.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:44.191+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:44.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:44.359+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:45.096 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:45.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:45.095162+0000) 2022-04-04T00:17:45.162 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:45.159+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:45.346 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:45.343+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:46.097 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:46.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:46.095363+0000) 2022-04-04T00:17:46.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:46.184+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:46.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:46.368+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:47.097 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:47.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:47.095576+0000) 2022-04-04T00:17:47.206 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:47.204+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:47.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:47.364+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:48.097 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:48.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:48.095830+0000) 2022-04-04T00:17:48.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:48.176+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:48.410 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:48.407+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:49.097 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:49.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:49.096024+0000) 2022-04-04T00:17:49.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:49.216+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:49.434 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:49.431+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:50.097 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:50.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:50.096212+0000) 2022-04-04T00:17:50.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:50.266+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:50.441 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:50.438+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:51.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:51.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:51.096405+0000) 2022-04-04T00:17:51.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:51.312+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:51.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:51.455+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:52.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:52.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:52.096614+0000) 2022-04-04T00:17:52.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:52.314+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:52.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:52.406+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:53.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:53.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:53.096848+0000) 2022-04-04T00:17:53.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:53.302+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:53.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:53.427+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:54.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:54.096+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:54.097035+0000) 2022-04-04T00:17:54.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:54.314+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:54.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:54.415+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:54.961 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:17:54.957+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:17:55.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:55.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:55.097198+0000) 2022-04-04T00:17:55.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:55.363+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:55.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:55.412+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:56.099 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:56.096+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:56.097399+0000) 2022-04-04T00:17:56.370 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:56.368+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:56.400 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:56.397+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:57.099 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:57.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:57.097635+0000) 2022-04-04T00:17:57.338 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:57.336+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:57.423 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:57.420+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:58.099 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:58.096+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:58.097853+0000) 2022-04-04T00:17:58.374 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:58.371+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:58.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:58.441+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:17:59.099 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:17:59.097+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:17:59.098071+0000) 2022-04-04T00:17:59.400 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:17:59.397+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:17:59.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:17:59.463+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:00.099 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:00.097+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:00.098235+0000) 2022-04-04T00:18:00.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:00.372+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:00.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:00.458+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:00.731 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:18:00.735 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:18:01.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:01.097+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:01.098418+0000) 2022-04-04T00:18:01.334 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:01.332+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:01.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:01.492+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:02.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:02.097+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:02.098652+0000) 2022-04-04T00:18:02.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:02.282+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:02.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:02.464+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:03.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:03.098+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:03.098867+0000) 2022-04-04T00:18:03.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:03.333+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:03.478 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:03.475+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:04.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:04.098+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:04.099112+0000) 2022-04-04T00:18:04.365 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:04.363+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:04.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:04.489+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:05.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:05.098+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:05.099316+0000) 2022-04-04T00:18:05.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:05.407+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:05.490 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:05.487+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:06.101 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:06.098+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:06.099567+0000) 2022-04-04T00:18:06.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:06.365+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:06.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:06.496+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:07.101 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:07.099+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:07.099761+0000) 2022-04-04T00:18:07.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:07.365+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:07.506 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:07.504+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:08.101 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:08.099+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:08.099947+0000) 2022-04-04T00:18:08.348 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:08.346+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:08.526 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:08.524+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:09.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:09.099+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:09.100151+0000) 2022-04-04T00:18:09.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:09.303+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:09.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:09.530+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:09.977 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:18:09.974+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:18:10.101 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:10.099+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:10.100343+0000) 2022-04-04T00:18:10.333 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:10.331+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:10.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:10.512+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:11.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:11.099+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:11.100540+0000) 2022-04-04T00:18:11.337 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:11.335+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:11.544 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:11.542+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:12.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:12.100+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:12.100793+0000) 2022-04-04T00:18:12.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:12.324+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:12.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:12.558+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:13.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:13.100+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:13.101018+0000) 2022-04-04T00:18:13.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:13.309+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:13.573 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:13.571+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:14.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:14.100+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:14.101251+0000) 2022-04-04T00:18:14.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:14.328+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:14.573 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:14.571+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:15.103 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:15.100+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:15.101468+0000) 2022-04-04T00:18:15.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:15.293+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:15.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:15.565+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:16.103 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:16.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:16.101671+0000) 2022-04-04T00:18:16.313 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:16.311+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:16.582 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:16.580+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:17.103 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:17.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:17.101831+0000) 2022-04-04T00:18:17.338 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:17.336+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:17.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:17.563+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:18.103 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:18.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:18.102028+0000) 2022-04-04T00:18:18.360 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:18.358+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:18.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:18.547+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:19.103 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:19.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:19.102208+0000) 2022-04-04T00:18:19.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:19.391+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:19.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:19.585+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:20.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:20.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:20.102401+0000) 2022-04-04T00:18:20.345 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:20.343+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:20.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:20.582+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:21.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:21.102+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:21.102567+0000) 2022-04-04T00:18:21.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:21.324+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:21.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:21.588+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:22.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:22.102+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:22.102742+0000) 2022-04-04T00:18:22.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:22.294+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:22.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:22.566+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:23.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:23.102+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:23.102936+0000) 2022-04-04T00:18:23.347 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:23.344+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:23.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:23.526+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:24.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:24.102+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:24.103159+0000) 2022-04-04T00:18:24.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:24.354+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:24.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:24.573+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:24.992 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:18:24.990+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:18:25.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:25.103+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:25.103321+0000) 2022-04-04T00:18:25.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:25.317+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:25.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:25.616+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:26.105 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:26.103+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:26.103485+0000) 2022-04-04T00:18:26.356 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:26.353+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:26.617 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:26.614+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:27.105 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:27.103+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:27.103671+0000) 2022-04-04T00:18:27.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:27.389+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:27.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:27.573+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:28.105 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:28.103+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:28.103849+0000) 2022-04-04T00:18:28.387 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:28.384+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:28.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:28.542+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:29.105 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:29.103+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:29.104060+0000) 2022-04-04T00:18:29.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:29.411+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:29.526 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:29.523+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:30.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:30.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:30.104277+0000) 2022-04-04T00:18:30.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:30.401+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:30.520 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:30.517+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:30.869 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:18:30.873 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:18:31.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:31.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:31.104500+0000) 2022-04-04T00:18:31.403 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:31.401+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:31.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:31.554+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:32.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:32.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:32.104721+0000) 2022-04-04T00:18:32.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:32.418+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:32.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:32.552+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:33.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:33.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:33.104868+0000) 2022-04-04T00:18:33.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:33.391+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:33.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:33.545+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:34.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:34.103+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:34.105056+0000) 2022-04-04T00:18:34.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:34.403+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:34.502 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:34.499+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:35.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:35.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:35.105282+0000) 2022-04-04T00:18:35.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:35.387+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:35.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:35.532+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:36.107 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:36.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:36.105480+0000) 2022-04-04T00:18:36.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:36.377+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:36.519 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:36.503+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:37.107 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:37.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:37.105673+0000) 2022-04-04T00:18:37.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:37.365+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:37.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:37.512+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:38.107 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:38.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:38.105913+0000) 2022-04-04T00:18:38.408 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:38.405+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:38.470 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:38.467+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:39.107 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:39.105+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:39.106172+0000) 2022-04-04T00:18:39.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:39.359+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:39.471 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:39.468+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:40.008 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:18:40.005+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:18:40.107 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:40.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:40.106324+0000) 2022-04-04T00:18:40.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:40.361+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:40.422 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:40.420+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:41.108 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:41.105+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:41.106544+0000) 2022-04-04T00:18:41.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:41.364+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:41.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:41.462+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:42.108 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:42.105+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:42.106806+0000) 2022-04-04T00:18:42.397 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:42.394+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:42.464 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:42.461+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:43.108 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:43.106+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:43.107002+0000) 2022-04-04T00:18:43.348 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:43.345+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:43.437 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:43.434+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:44.108 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:44.106+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:44.107176+0000) 2022-04-04T00:18:44.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:44.381+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:44.451 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:44.448+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:45.109 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:45.106+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:45.107387+0000) 2022-04-04T00:18:45.365 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:45.362+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:45.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:45.446+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:46.109 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:46.106+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:46.107614+0000) 2022-04-04T00:18:46.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:46.365+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:46.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:46.493+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:47.109 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:47.107+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:47.107856+0000) 2022-04-04T00:18:47.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:47.354+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:47.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:47.539+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:48.109 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:48.107+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:48.108079+0000) 2022-04-04T00:18:48.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:48.333+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:48.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:48.496+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:49.109 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:49.107+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:49.108251+0000) 2022-04-04T00:18:49.359 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:49.356+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:49.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:49.482+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:50.110 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:50.107+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:50.108445+0000) 2022-04-04T00:18:50.359 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:50.356+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:50.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:50.503+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:51.110 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:51.107+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:51.108691+0000) 2022-04-04T00:18:51.358 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:51.355+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:51.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:51.491+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:52.110 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:52.108+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:52.108934+0000) 2022-04-04T00:18:52.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:52.320+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:52.475 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:52.472+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:53.110 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:53.108+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:53.109163+0000) 2022-04-04T00:18:53.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:53.351+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:53.464 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:53.462+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:54.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:54.108+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:54.109408+0000) 2022-04-04T00:18:54.304 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:54.301+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:54.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:54.483+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:55.024 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:18:55.022+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:18:55.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:55.109+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:55.109658+0000) 2022-04-04T00:18:55.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:55.277+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:55.442 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:55.439+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:56.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:56.109+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:56.109851+0000) 2022-04-04T00:18:56.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:56.297+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:56.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:56.479+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:57.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:57.109+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:57.110018+0000) 2022-04-04T00:18:57.258 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:57.256+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:57.479 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:57.477+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:58.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:58.109+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:58.110123+0000) 2022-04-04T00:18:58.307 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:58.305+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:58.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:58.495+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:18:59.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:18:59.109+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:18:59.110257+0000) 2022-04-04T00:18:59.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:18:59.299+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:18:59.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:18:59.484+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:00.112 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:00.109+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:00.110468+0000) 2022-04-04T00:19:00.325 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:00.323+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:00.493 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:00.490+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:00.943 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:19:00.947 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:19:01.112 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:01.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:01.110740+0000) 2022-04-04T00:19:01.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:01.286+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:01.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:01.503+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:02.112 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:02.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:02.111009+0000) 2022-04-04T00:19:02.319 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:02.317+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:02.462 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:02.460+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:03.112 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:03.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:03.111225+0000) 2022-04-04T00:19:03.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:03.359+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:03.462 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:03.460+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:04.113 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:04.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:04.111409+0000) 2022-04-04T00:19:04.356 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:04.354+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:04.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:04.415+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:05.113 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:05.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:05.111607+0000) 2022-04-04T00:19:05.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:05.367+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:05.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:05.396+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:06.113 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:06.111+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:06.111755+0000) 2022-04-04T00:19:06.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:06.391+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:06.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:06.428+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:07.113 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:07.111+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:07.111964+0000) 2022-04-04T00:19:07.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:07.380+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:07.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:07.415+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:08.113 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:08.111+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:08.112148+0000) 2022-04-04T00:19:08.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:08.361+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:08.464 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:08.462+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:09.113 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:09.111+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:09.112349+0000) 2022-04-04T00:19:09.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:09.380+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:09.480 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:09.479+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:10.032 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:19:10.029+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:19:10.114 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:10.112+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:10.112557+0000) 2022-04-04T00:19:10.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:10.386+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:10.454 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:10.452+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:11.114 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:11.112+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:11.112733+0000) 2022-04-04T00:19:11.396 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:11.394+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:11.500 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:11.498+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:12.114 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:12.112+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:12.112992+0000) 2022-04-04T00:19:12.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:12.379+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:12.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:12.480+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:13.114 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:13.112+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:13.113168+0000) 2022-04-04T00:19:13.358 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:13.356+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:13.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:13.516+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:14.114 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:14.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:14.113388+0000) 2022-04-04T00:19:14.365 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:14.363+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:14.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:14.525+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:15.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:15.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:15.113591+0000) 2022-04-04T00:19:15.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:15.375+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:15.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:15.516+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:16.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:16.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:16.113804+0000) 2022-04-04T00:19:16.424 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:16.422+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:16.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:16.492+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:17.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:17.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:17.113985+0000) 2022-04-04T00:19:17.387 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:17.384+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:17.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:17.456+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:18.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:18.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:18.114174+0000) 2022-04-04T00:19:18.434 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:18.431+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:18.498 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:18.495+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:19.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:19.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:19.114325+0000) 2022-04-04T00:19:19.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:19.382+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:19.477 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:19.474+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:20.116 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:20.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:20.114477+0000) 2022-04-04T00:19:20.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:20.409+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:20.478 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:20.475+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:21.116 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:21.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:21.114654+0000) 2022-04-04T00:19:21.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:21.422+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:21.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:21.501+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:22.116 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:22.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:22.114862+0000) 2022-04-04T00:19:22.424 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:22.421+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:22.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:22.496+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:23.116 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:23.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:23.115043+0000) 2022-04-04T00:19:23.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:23.422+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:23.464 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:23.461+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:24.116 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:24.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:24.115246+0000) 2022-04-04T00:19:24.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:24.403+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:24.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:24.459+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:25.034 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:19:25.031+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:19:25.116 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:25.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:25.115394+0000) 2022-04-04T00:19:25.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:25.427+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:25.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:25.480+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:26.117 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:26.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:26.115583+0000) 2022-04-04T00:19:26.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:26.451+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:26.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:26.513+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:27.117 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:27.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:27.115855+0000) 2022-04-04T00:19:27.410 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:27.408+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:27.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:27.502+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:28.117 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:28.115+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:28.115981+0000) 2022-04-04T00:19:28.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:28.365+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:28.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:28.492+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:29.117 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:29.115+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:29.116160+0000) 2022-04-04T00:19:29.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:29.333+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:29.510 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:29.507+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:30.117 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:30.115+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:30.116309+0000) 2022-04-04T00:19:30.360 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:30.357+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:30.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:30.543+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:31.089 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:19:31.093 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:19:31.117 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:31.115+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:31.116498+0000) 2022-04-04T00:19:31.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:31.321+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:31.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:31.496+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:32.118 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:32.115+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:32.116664+0000) 2022-04-04T00:19:32.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:32.279+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:32.463 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:32.461+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:33.118 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:33.116+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:33.116869+0000) 2022-04-04T00:19:33.262 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:33.259+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:33.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:33.491+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:34.118 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:34.116+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:34.117089+0000) 2022-04-04T00:19:34.258 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:34.255+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:34.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:34.488+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:35.118 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:35.116+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:35.117289+0000) 2022-04-04T00:19:35.247 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:35.244+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:35.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:35.458+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:36.119 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:36.116+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:36.117430+0000) 2022-04-04T00:19:36.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:36.201+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:36.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:36.464+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:37.119 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:37.116+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:37.117651+0000) 2022-04-04T00:19:37.216 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:37.213+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:37.502 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:37.500+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:38.119 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:38.117+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:38.117826+0000) 2022-04-04T00:19:38.204 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:38.201+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:38.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:38.510+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:39.119 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:39.117+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:39.118049+0000) 2022-04-04T00:19:39.193 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:39.190+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:39.471 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:39.469+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:40.049 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:19:40.047+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:19:40.119 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:40.117+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:40.118267+0000) 2022-04-04T00:19:40.190 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:40.187+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:40.457 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:40.454+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:41.120 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:41.117+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:41.118514+0000) 2022-04-04T00:19:41.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:41.170+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:41.409 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:41.406+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:42.120 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:42.118+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:42.118740+0000) 2022-04-04T00:19:42.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:42.129+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:42.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:42.441+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:43.120 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:43.118+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:43.119005+0000) 2022-04-04T00:19:43.180 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:43.177+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:43.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:43.449+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:44.120 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:44.118+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:44.119129+0000) 2022-04-04T00:19:44.166 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:44.163+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:44.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:44.405+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:45.120 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:45.118+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:45.119334+0000) 2022-04-04T00:19:45.129 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:45.126+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:45.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:45.383+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:46.121 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:46.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:46.119499+0000) 2022-04-04T00:19:46.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:46.124+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:46.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:46.384+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:47.121 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:47.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:47.119689+0000) 2022-04-04T00:19:47.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:47.118+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:47.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:47.380+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:48.121 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:48.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:48.119907+0000) 2022-04-04T00:19:48.169 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:48.167+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:48.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:48.349+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:49.121 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:49.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:49.120120+0000) 2022-04-04T00:19:49.162 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:49.160+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:49.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:49.313+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:50.121 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:50.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:50.120325+0000) 2022-04-04T00:19:50.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:50.126+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:50.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:50.307+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:51.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:51.101+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:51.121 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:51.120+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:51.120515+0000) 2022-04-04T00:19:51.301 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:51.299+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:52.122 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:52.120+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:52.120730+0000) 2022-04-04T00:19:52.144 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:52.142+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:52.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:52.283+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:53.122 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:53.120+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:53.120941+0000) 2022-04-04T00:19:53.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:53.171+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:53.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:53.322+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:54.122 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:54.120+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:54.121133+0000) 2022-04-04T00:19:54.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:54.171+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:54.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:54.320+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:55.065 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:19:55.063+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:19:55.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:55.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:55.121327+0000) 2022-04-04T00:19:55.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:55.151+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:55.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:55.364+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:56.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:56.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:56.121487+0000) 2022-04-04T00:19:56.180 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:56.178+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:56.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:56.319+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:57.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:57.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:57.121688+0000) 2022-04-04T00:19:57.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:57.225+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:57.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:57.334+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:58.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:58.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:58.121910+0000) 2022-04-04T00:19:58.233 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:58.232+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:58.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:58.313+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:19:59.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:19:59.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:19:59.122131+0000) 2022-04-04T00:19:59.216 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:19:59.214+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:19:59.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:19:59.359+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:00.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:00.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:00.122351+0000) 2022-04-04T00:20:00.220 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:00.218+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:00.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:00.346+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:01.124 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:01.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:01.122577+0000) 2022-04-04T00:20:01.161 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:20:01.165 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:20:01.181 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:01.179+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:01.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:01.320+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:02.124 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:02.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:02.122829+0000) 2022-04-04T00:20:02.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:02.131+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:02.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:02.289+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:03.124 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:03.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:03.123046+0000) 2022-04-04T00:20:03.138 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:03.137+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:03.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:03.292+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:04.120 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:04.118+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:04.124 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:04.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:04.123257+0000) 2022-04-04T00:20:04.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:04.338+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:05.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:05.079+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:05.125 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:05.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:05.123475+0000) 2022-04-04T00:20:05.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:05.297+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:06.116 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:06.114+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:06.125 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:06.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:06.123615+0000) 2022-04-04T00:20:06.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:06.316+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:07.125 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:07.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:07.123821+0000) 2022-04-04T00:20:07.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:07.133+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:07.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:07.276+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:08.106 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:08.104+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:08.125 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:08.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:08.124022+0000) 2022-04-04T00:20:08.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:08.235+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:09.114 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:09.112+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:09.125 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:09.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:09.124247+0000) 2022-04-04T00:20:09.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:09.214+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:10.077 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:20:10.073+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:20:10.104 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:10.102+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:10.125 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:10.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:10.124463+0000) 2022-04-04T00:20:10.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:10.196+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:11.092 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:11.090+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:11.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:11.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:11.124656+0000) 2022-04-04T00:20:11.195 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:11.193+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:12.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:12.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:12.124876+0000) 2022-04-04T00:20:12.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:12.124+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:12.195 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:12.192+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:13.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:13.124+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:13.125043+0000) 2022-04-04T00:20:13.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:13.150+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:13.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:13.154+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:14.116 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:14.113+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:14.119 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:14.116+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:14.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:14.124+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:14.125264+0000) 2022-04-04T00:20:15.093 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:15.090+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:15.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:15.124+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:15.125483+0000) 2022-04-04T00:20:15.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:15.132+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:16.093 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:16.090+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:16.127 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:16.124+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:16.125642+0000) 2022-04-04T00:20:16.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:16.132+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:17.054 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:17.051+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:17.104 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:17.102+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:17.127 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:17.125+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:17.125850+0000) 2022-04-04T00:20:18.020 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:18.017+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:18.087 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:18.085+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:18.127 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:18.125+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:18.126043+0000) 2022-04-04T00:20:19.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:19.036+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:19.048 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:19.045+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:19.127 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:19.125+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:19.126240+0000) 2022-04-04T00:20:20.005 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:20.002+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:20.095 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:20.092+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:20.127 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:20.125+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:20.126463+0000) 2022-04-04T00:20:20.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:20.979+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:21.128 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:21.125+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:21.126603+0000) 2022-04-04T00:20:21.144 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:21.141+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:21.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:21.988+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:22.128 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:22.126+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:22.126805+0000) 2022-04-04T00:20:22.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:22.163+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:22.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:22.939+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:23.126 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:23.123+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:23.128 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:23.126+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:23.127008+0000) 2022-04-04T00:20:23.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:23.937+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:24.128 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:24.126+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:24.127175+0000) 2022-04-04T00:20:24.131 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:24.129+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:24.987 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:24.984+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:20:25.092 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:20:25.090+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:20:25.128 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:25.126+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:25.127368+0000) 2022-04-04T00:20:25.169 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:25.166+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:25.954 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:25.952+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:26.129 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:26.126+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:26.127559+0000) 2022-04-04T00:20:26.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:26.174+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:26.930 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:26.927+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:27.129 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:27.127+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:27.127740+0000) 2022-04-04T00:20:27.165 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:27.162+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:27.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:27.894+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:28.129 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:28.127+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:28.127948+0000) 2022-04-04T00:20:28.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:28.151+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:28.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:28.848+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:29.129 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:29.127+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:29.128185+0000) 2022-04-04T00:20:29.158 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:29.156+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:29.862 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:29.860+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:30.129 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:30.127+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:30.128387+0000) 2022-04-04T00:20:30.202 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:30.199+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:30.844 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:30.841+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:31.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:31.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:31.128605+0000) 2022-04-04T00:20:31.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:31.170+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:31.309 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:20:31.313 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:20:31.867 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:31.865+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:32.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:32.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:32.128854+0000) 2022-04-04T00:20:32.220 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:32.218+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:32.869 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:32.866+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:33.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:33.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:33.129055+0000) 2022-04-04T00:20:33.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:33.259+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:33.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:33.915+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:34.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:34.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:34.129277+0000) 2022-04-04T00:20:34.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:34.220+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:34.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:34.962+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:35.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:35.129+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:35.129474+0000) 2022-04-04T00:20:35.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:35.248+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:35.963 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:35.961+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:36.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:36.129+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:36.129643+0000) 2022-04-04T00:20:36.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:36.250+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:36.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:36.915+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:37.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:37.129+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:37.129838+0000) 2022-04-04T00:20:37.213 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:37.211+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:37.967 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:37.965+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:38.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:38.129+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:38.130065+0000) 2022-04-04T00:20:38.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:38.165+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:39.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:39.005+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:39.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:39.129+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:39.130232+0000) 2022-04-04T00:20:39.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:39.196+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:40.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:40.039+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:40.108 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:20:40.106+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:20:40.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:40.130+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:40.130435+0000) 2022-04-04T00:20:40.235 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:40.232+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:41.069 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:41.067+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:41.132 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:41.130+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:41.130627+0000) 2022-04-04T00:20:41.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:41.279+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:42.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:42.109+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:42.132 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:42.130+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:42.130858+0000) 2022-04-04T00:20:42.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:42.326+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:43.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:43.118+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:43.132 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:43.130+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:43.131075+0000) 2022-04-04T00:20:43.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:43.307+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:44.118 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:44.115+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:44.132 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:44.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:44.131330+0000) 2022-04-04T00:20:44.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:44.321+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:45.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:45.085+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:45.132 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:45.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:45.131548+0000) 2022-04-04T00:20:45.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:45.284+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:46.077 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:46.075+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:46.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:46.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:46.131746+0000) 2022-04-04T00:20:46.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:46.315+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:47.086 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:47.083+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:47.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:47.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:47.132011+0000) 2022-04-04T00:20:47.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:47.342+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:48.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:48.109+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:48.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:48.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:48.132246+0000) 2022-04-04T00:20:48.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:48.299+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:49.102 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:49.100+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:49.134 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:49.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:49.132403+0000) 2022-04-04T00:20:49.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:49.289+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:50.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:50.057+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:50.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:50.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:50.132568+0000) 2022-04-04T00:20:50.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:50.258+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:51.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:51.074+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:51.134 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:51.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:51.132764+0000) 2022-04-04T00:20:51.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:51.255+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:52.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:52.026+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:52.134 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:52.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:52.132983+0000) 2022-04-04T00:20:52.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:52.291+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:53.001 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:52.999+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:53.134 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:53.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:53.133176+0000) 2022-04-04T00:20:53.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:53.258+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:53.975 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:53.973+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:54.134 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:54.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:54.133331+0000) 2022-04-04T00:20:54.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:54.259+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:54.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:54.951+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:55.111 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:20:55.107+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:20:55.134 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:55.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:55.133513+0000) 2022-04-04T00:20:55.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:55.306+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:55.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:55.951+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:56.135 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:56.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:56.133656+0000) 2022-04-04T00:20:56.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:56.297+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:56.915 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:56.913+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:57.135 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:57.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:57.133858+0000) 2022-04-04T00:20:57.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:57.342+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:57.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:57.868+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:58.135 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:58.133+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:58.134014+0000) 2022-04-04T00:20:58.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:58.342+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:58.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:58.868+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:20:59.135 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:20:59.133+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:20:59.134192+0000) 2022-04-04T00:20:59.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:20:59.339+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:20:59.902 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:20:59.900+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:00.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:00.133+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:00.134414+0000) 2022-04-04T00:21:00.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:00.352+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:00.895 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:00.893+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:01.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:01.133+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:01.134621+0000) 2022-04-04T00:21:01.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:01.360+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:01.447 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:21:01.451 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:21:01.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:01.885+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:02.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:02.134+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:02.134888+0000) 2022-04-04T00:21:02.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:02.397+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:02.866 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:02.864+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:03.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:03.134+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:03.135120+0000) 2022-04-04T00:21:03.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:03.408+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:03.850 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:03.848+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:04.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:04.134+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:04.135330+0000) 2022-04-04T00:21:04.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:04.389+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:04.817 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:04.815+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:05.137 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:05.134+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:05.135509+0000) 2022-04-04T00:21:05.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:05.402+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:05.787 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:05.785+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:06.137 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:06.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:06.135718+0000) 2022-04-04T00:21:06.442 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:06.440+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:06.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:06.806+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:07.137 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:07.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:07.135964+0000) 2022-04-04T00:21:07.454 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:07.451+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:07.774 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:07.771+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:08.137 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:08.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:08.136180+0000) 2022-04-04T00:21:08.503 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:08.500+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:08.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:08.785+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:09.137 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:09.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:09.136333+0000) 2022-04-04T00:21:09.473 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:09.470+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:09.738 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:09.735+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:10.127 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:21:10.124+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:21:10.138 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:10.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:10.136546+0000) 2022-04-04T00:21:10.202 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-04T00:21:10.200+0000 7f2fd661a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T00:21:10.203 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-04T00:21:10.200+0000 7f2fd6e1b700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T00:21:10.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:10.510+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:10.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:10.747+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:11.138 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:11.136+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:11.136756+0000) 2022-04-04T00:21:11.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:11.488+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:11.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:11.755+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:12.138 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:12.136+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:12.136976+0000) 2022-04-04T00:21:12.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:12.510+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:12.722 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:12.719+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:13.138 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:13.136+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:13.137179+0000) 2022-04-04T00:21:13.475 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:13.473+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:13.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:13.690+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:14.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:14.136+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:14.137400+0000) 2022-04-04T00:21:14.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:14.512+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:14.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:14.677+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:15.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:15.136+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:15.137609+0000) 2022-04-04T00:21:15.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:15.558+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:15.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:15.723+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:16.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:16.137+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:16.137836+0000) 2022-04-04T00:21:16.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:16.566+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:16.738 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:16.735+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:17.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:17.137+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:17.138064+0000) 2022-04-04T00:21:17.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:17.564+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:17.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:17.785+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:18.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:18.137+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:18.138234+0000) 2022-04-04T00:21:18.528 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:18.525+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:18.828 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:18.825+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:19.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:19.138+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:19.138428+0000) 2022-04-04T00:21:19.484 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:19.481+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:19.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:19.832+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:20.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:20.138+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:20.138618+0000) 2022-04-04T00:21:20.519 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:20.516+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:20.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:20.827+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:21.140 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:21.138+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:21.138833+0000) 2022-04-04T00:21:21.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:21.509+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:21.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:21.820+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:22.140 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:22.138+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:22.138982+0000) 2022-04-04T00:21:22.464 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:22.462+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:22.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:22.799+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:23.140 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:23.138+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:23.139140+0000) 2022-04-04T00:21:23.438 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:23.435+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:23.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:23.818+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:24.140 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:24.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:24.139337+0000) 2022-04-04T00:21:24.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:24.436+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:24.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:24.796+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:25.141 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:25.138+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:25.139625+0000) 2022-04-04T00:21:25.142 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:21:25.139+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:21:25.473 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:25.470+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:25.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:25.782+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:26.141 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:26.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:26.139859+0000) 2022-04-04T00:21:26.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:26.437+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:26.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:26.779+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:27.141 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:27.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:27.140065+0000) 2022-04-04T00:21:27.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:27.432+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:27.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:27.808+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:28.141 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:28.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:28.140284+0000) 2022-04-04T00:21:28.450 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:28.447+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:28.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:28.818+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:29.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:29.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:29.140507+0000) 2022-04-04T00:21:29.463 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:29.460+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:29.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:29.827+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:30.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:30.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:30.140716+0000) 2022-04-04T00:21:30.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:30.488+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:30.874 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:30.871+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:31.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:31.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:31.140960+0000) 2022-04-04T00:21:31.490 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:31.487+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:31.577 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:21:31.581 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:21:31.907 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:31.904+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:32.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:32.141+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:32.141206+0000) 2022-04-04T00:21:32.533 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:32.530+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:32.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:32.872+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:33.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:33.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:33.141361+0000) 2022-04-04T00:21:33.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:33.545+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:33.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:33.836+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:34.143 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:34.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:34.141567+0000) 2022-04-04T00:21:34.507 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:34.504+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:34.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:34.825+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:35.143 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:35.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:35.141812+0000) 2022-04-04T00:21:35.507 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:35.504+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:35.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:35.813+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:36.143 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:36.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:36.142021+0000) 2022-04-04T00:21:36.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:36.516+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:36.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:36.852+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:37.143 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:37.141+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:37.142196+0000) 2022-04-04T00:21:37.489 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:37.486+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:37.895 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:37.893+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:38.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:38.141+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:38.142421+0000) 2022-04-04T00:21:38.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:38.467+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:38.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:38.894+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:39.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:39.141+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:39.142609+0000) 2022-04-04T00:21:39.462 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:39.460+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:39.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:39.863+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:40.144 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:21:40.140+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:21:40.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:40.141+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:40.142788+0000) 2022-04-04T00:21:40.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:40.452+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:40.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:40.885+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:41.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:41.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:41.142964+0000) 2022-04-04T00:21:41.418 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:41.416+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:41.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:41.935+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:42.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:42.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:42.143147+0000) 2022-04-04T00:21:42.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:42.400+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:42.901 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:42.898+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:43.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:43.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:43.143302+0000) 2022-04-04T00:21:43.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:43.375+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:43.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:43.862+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:44.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:44.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:44.143537+0000) 2022-04-04T00:21:44.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:44.363+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:44.862 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:44.860+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:45.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:45.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:45.143800+0000) 2022-04-04T00:21:45.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:45.314+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:45.869 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:45.867+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:46.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:46.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:46.143979+0000) 2022-04-04T00:21:46.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:46.304+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:46.908 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:46.905+0000 7fd3b190b700 -1 osd.6 135 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:47.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:47.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:47.144147+0000) 2022-04-04T00:21:47.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:47.348+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:47.663 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-04T00:21:47.661+0000 7f116bc99700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-04T00:21:47.678 INFO:tasks.ceph.mds.f.smithi186.stderr: -1> 2022-04-04T00:21:47.661+0000 7f116bc99700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-04T00:21:47.700 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-04T00:21:47.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:47.931+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:48.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:48.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:48.144262+0000) 2022-04-04T00:21:48.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:48.380+0000 7f88d0ac1700 -1 osd.4 135 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:48.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:48.931+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:49.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:49.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:49.144450+0000) 2022-04-04T00:21:49.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:49.357+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:49.901 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:49.899+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:50.146 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:50.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:50.144625+0000) 2022-04-04T00:21:50.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:50.389+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:50.852 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:50.850+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:51.146 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:51.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:51.144801+0000) 2022-04-04T00:21:51.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:51.390+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:51.844 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:51.842+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:52.146 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:52.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:52.144982+0000) 2022-04-04T00:21:52.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:52.385+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 52 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:52.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:52.807+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:53.146 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:53.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:53.145164+0000) 2022-04-04T00:21:53.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:53.348+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:53.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:53.845+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:54.146 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:54.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:54.145312+0000) 2022-04-04T00:21:54.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:54.390+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:54.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:54.853+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:55.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:55.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:55.145559+0000) 2022-04-04T00:21:55.148 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:21:55.145+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:21:55.400 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:55.398+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:55.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:55.811+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:56.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:56.145+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:56.145789+0000) 2022-04-04T00:21:56.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:56.381+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:56.765 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:56.763+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:57.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:57.145+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:57.145990+0000) 2022-04-04T00:21:57.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:57.425+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:57.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:57.787+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:58.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:58.145+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:58.146208+0000) 2022-04-04T00:21:58.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:58.454+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:58.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:58.797+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:21:59.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:21:59.145+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:21:59.146418+0000) 2022-04-04T00:21:59.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:21:59.493+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:21:59.767 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:21:59.765+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:00.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:00.146+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:00.146630+0000) 2022-04-04T00:22:00.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:00.479+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:00.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:00.753+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:01.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:01.146+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:01.146848+0000) 2022-04-04T00:22:01.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:01.481+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:01.724 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:22:01.728 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:22:01.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:01.751+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:02.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:02.146+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:02.147061+0000) 2022-04-04T00:22:02.523 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:02.520+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:02.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:02.800+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:03.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:03.146+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:03.147264+0000) 2022-04-04T00:22:03.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:03.485+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:03.852 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:03.849+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:04.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:04.147+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:04.147447+0000) 2022-04-04T00:22:04.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:04.518+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:04.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:04.878+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:05.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:05.147+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:05.147646+0000) 2022-04-04T00:22:05.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:05.488+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:05.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:05.928+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:06.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:06.147+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:06.147856+0000) 2022-04-04T00:22:06.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:06.480+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:06.907 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:06.904+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:07.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:07.147+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:07.148042+0000) 2022-04-04T00:22:07.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:07.452+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:07.862 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:07.859+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:08.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:08.147+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:08.148242+0000) 2022-04-04T00:22:08.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:08.453+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:08.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:08.827+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:09.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:09.148+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:09.148445+0000) 2022-04-04T00:22:09.477 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:09.474+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:09.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:09.827+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:10.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:10.148+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:10.148591+0000) 2022-04-04T00:22:10.158 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:22:10.156+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:22:10.466 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:10.463+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:10.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:10.805+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:11.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:11.148+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:11.148763+0000) 2022-04-04T00:22:11.442 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:11.440+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:11.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:11.808+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:12.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:12.148+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:12.148985+0000) 2022-04-04T00:22:12.463 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:12.460+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:12.857 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:12.854+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:13.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:13.148+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:13.149179+0000) 2022-04-04T00:22:13.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:13.441+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:13.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:13.819+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:14.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:14.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:14.149411+0000) 2022-04-04T00:22:14.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:14.456+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:14.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:14.778+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:15.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:15.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:15.149643+0000) 2022-04-04T00:22:15.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:15.462+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:15.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:15.777+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:16.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:16.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:16.149880+0000) 2022-04-04T00:22:16.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:16.511+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:16.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:16.825+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:17.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:17.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:17.150050+0000) 2022-04-04T00:22:17.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:17.515+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:17.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:17.808+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:18.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:18.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:18.150252+0000) 2022-04-04T00:22:18.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:18.503+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:18.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:18.847+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:19.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:19.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:19.150474+0000) 2022-04-04T00:22:19.537 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:19.534+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:19.819 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:19.816+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:20.152 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:20.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:20.150690+0000) 2022-04-04T00:22:20.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:20.571+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:20.804 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:20.802+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:21.152 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:21.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:21.150890+0000) 2022-04-04T00:22:21.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:21.551+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:21.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:21.822+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:22.152 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:22.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:22.151088+0000) 2022-04-04T00:22:22.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:22.558+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:22.862 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:22.860+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:23.152 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:23.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:23.151311+0000) 2022-04-04T00:22:23.595 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:23.593+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:23.862 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:23.860+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:24.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:24.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:24.151531+0000) 2022-04-04T00:22:24.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:24.546+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:24.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:24.901+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:25.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:25.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:25.151747+0000) 2022-04-04T00:22:25.174 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:22:25.171+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:22:25.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:25.558+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:25.926 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:25.923+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:26.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:26.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:26.151972+0000) 2022-04-04T00:22:26.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:26.527+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:26.901 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:26.898+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:27.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:27.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:27.152212+0000) 2022-04-04T00:22:27.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:27.493+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:27.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:27.882+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:28.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:28.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:28.152457+0000) 2022-04-04T00:22:28.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:28.494+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:28.883 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:28.881+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:29.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:29.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:29.152626+0000) 2022-04-04T00:22:29.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:29.510+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:29.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:29.840+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:30.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:30.152+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:30.152827+0000) 2022-04-04T00:22:30.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:30.510+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:30.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:30.884+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:31.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:31.152+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:31.153026+0000) 2022-04-04T00:22:31.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:31.554+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:31.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:31.848+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:31.864 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:22:31.868 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:22:32.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:32.152+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:32.153246+0000) 2022-04-04T00:22:32.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:32.511+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:32.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:32.880+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:33.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:33.152+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:33.153453+0000) 2022-04-04T00:22:33.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:33.480+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:33.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:33.858+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:34.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:34.152+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:34.153674+0000) 2022-04-04T00:22:34.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:34.459+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:34.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:34.845+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:35.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:35.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:35.153897+0000) 2022-04-04T00:22:35.423 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:35.421+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:35.867 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:35.864+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:36.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:36.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:36.154120+0000) 2022-04-04T00:22:36.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:36.438+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:36.826 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:36.823+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:37.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:37.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:37.154322+0000) 2022-04-04T00:22:37.486 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:37.483+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:37.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:37.870+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:38.156 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:38.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:38.154540+0000) 2022-04-04T00:22:38.474 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:38.471+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:38.862 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:38.859+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:39.156 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:39.154+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:39.154744+0000) 2022-04-04T00:22:39.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:39.491+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:39.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:39.875+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:40.156 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:40.154+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:40.154952+0000) 2022-04-04T00:22:40.190 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:22:40.188+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:22:40.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:40.457+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:40.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:40.914+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:41.156 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:41.154+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:41.155178+0000) 2022-04-04T00:22:41.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:41.467+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:41.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:41.953+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:42.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:42.154+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:42.155396+0000) 2022-04-04T00:22:42.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:42.480+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:42.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:42.982+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:43.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:43.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:43.155567+0000) 2022-04-04T00:22:43.526 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:43.524+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:44.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:44.022+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:44.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:44.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:44.155748+0000) 2022-04-04T00:22:44.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:44.536+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:45.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:45.001+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:45.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:45.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:45.155960+0000) 2022-04-04T00:22:45.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:45.514+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:45.986 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:45.985+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:46.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:46.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:46.156179+0000) 2022-04-04T00:22:46.507 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:46.505+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:47.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:47.006+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:47.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:47.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:47.156401+0000) 2022-04-04T00:22:47.520 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:47.519+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:47.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:47.993+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:48.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:48.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:48.156633+0000) 2022-04-04T00:22:48.486 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:48.484+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:49.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:49.035+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:49.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:49.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:49.156832+0000) 2022-04-04T00:22:49.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:49.460+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:50.068 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:50.067+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:50.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:50.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:50.156998+0000) 2022-04-04T00:22:50.480 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:50.478+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:51.115 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:51.113+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:51.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:51.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:51.157197+0000) 2022-04-04T00:22:51.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:51.492+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:52.113 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:52.111+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:52.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:52.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:52.157422+0000) 2022-04-04T00:22:52.517 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:52.515+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:53.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:53.154+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:53.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:53.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:53.157628+0000) 2022-04-04T00:22:53.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:53.502+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:54.120 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:54.118+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:54.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:54.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:54.157846+0000) 2022-04-04T00:22:54.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:54.507+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:55.120 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:55.118+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:55.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:55.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:55.158104+0000) 2022-04-04T00:22:55.192 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:22:55.189+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:22:55.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:55.489+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:56.093 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:56.091+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:56.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:56.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:56.158261+0000) 2022-04-04T00:22:56.526 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:56.524+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:57.099 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:57.098+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:57.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:57.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:57.158454+0000) 2022-04-04T00:22:57.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:57.543+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:58.050 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:58.048+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:58.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:58.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:58.158628+0000) 2022-04-04T00:22:58.533 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:58.530+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:22:59.058 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:22:59.056+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:22:59.160 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:22:59.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:22:59.158861+0000) 2022-04-04T00:22:59.566 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:22:59.563+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:00.099 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:00.096+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:00.160 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:00.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:00.159002+0000) 2022-04-04T00:23:00.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:00.578+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:01.096 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:01.093+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:01.160 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:01.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:01.159182+0000) 2022-04-04T00:23:01.573 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:01.570+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:02.060 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:23:02.064 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:23:02.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:02.131+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:02.160 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:02.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:02.159424+0000) 2022-04-04T00:23:02.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:02.598+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:03.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:03.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:03.159633+0000) 2022-04-04T00:23:03.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:03.168+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:03.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:03.629+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:04.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:04.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:04.159857+0000) 2022-04-04T00:23:04.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:04.195+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:04.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:04.642+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:05.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:05.150+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:05.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:05.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:05.160078+0000) 2022-04-04T00:23:05.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:05.680+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:06.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:06.109+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:06.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:06.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:06.160283+0000) 2022-04-04T00:23:06.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:06.638+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:07.087 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:07.084+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:07.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:07.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:07.160503+0000) 2022-04-04T00:23:07.606 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:07.604+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:08.064 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:08.061+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:08.162 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:08.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:08.160729+0000) 2022-04-04T00:23:08.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:08.627+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:09.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:09.082+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:09.162 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:09.160+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:09.160958+0000) 2022-04-04T00:23:09.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:09.584+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:10.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:10.125+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:10.162 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:10.160+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:10.161132+0000) 2022-04-04T00:23:10.208 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:23:10.205+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:23:10.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:10.556+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:11.152 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:11.149+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:11.162 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:11.160+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:11.161341+0000) 2022-04-04T00:23:11.583 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:11.580+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:12.140 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:12.138+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:12.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:12.160+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:12.161544+0000) 2022-04-04T00:23:12.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:12.602+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:13.114 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:13.111+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:13.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:13.160+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:13.161751+0000) 2022-04-04T00:23:13.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:13.587+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:14.106 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:14.104+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:14.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:14.161+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:14.161934+0000) 2022-04-04T00:23:14.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:14.624+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:15.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:15.082+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:15.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:15.161+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:15.162136+0000) 2022-04-04T00:23:15.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:15.585+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:16.072 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:16.070+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:16.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:16.161+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:16.162346+0000) 2022-04-04T00:23:16.572 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:16.569+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:17.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:17.095+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:17.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:17.161+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:17.162549+0000) 2022-04-04T00:23:17.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:17.574+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:18.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:18.125+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:18.164 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:18.161+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:18.162736+0000) 2022-04-04T00:23:18.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:18.540+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:19.110 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:19.107+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:19.164 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:19.162+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:19.162946+0000) 2022-04-04T00:23:19.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:19.528+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:20.092 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:20.090+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:20.164 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:20.162+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:20.163113+0000) 2022-04-04T00:23:20.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:20.546+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:21.069 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:21.066+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:21.164 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:21.162+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:21.163335+0000) 2022-04-04T00:23:21.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:21.586+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:22.094 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:22.092+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:22.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:22.162+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:22.163538+0000) 2022-04-04T00:23:22.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:22.624+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:23.089 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:23.087+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:23.287 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:23.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:23.163749+0000) 2022-04-04T00:23:23.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:23.637+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:24.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:24.059+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:24.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:24.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:24.163997+0000) 2022-04-04T00:23:24.673 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:24.671+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:25.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:25.051+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:25.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:25.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:25.164207+0000) 2022-04-04T00:23:25.224 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:23:25.222+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:23:25.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:25.716+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:26.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:26.074+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:26.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:26.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:26.164419+0000) 2022-04-04T00:23:26.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:26.669+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:27.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:27.079+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:27.166 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:27.164+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:27.164642+0000) 2022-04-04T00:23:27.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:27.713+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:28.064 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:28.061+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:28.166 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:28.164+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:28.164867+0000) 2022-04-04T00:23:28.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:28.699+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:29.034 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:29.031+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:29.166 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:29.164+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:29.165070+0000) 2022-04-04T00:23:29.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:29.667+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:30.050 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:30.047+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:30.166 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:30.164+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:30.165285+0000) 2022-04-04T00:23:30.689 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:30.687+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:31.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:31.056+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:31.166 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:31.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:31.165509+0000) 2022-04-04T00:23:31.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:31.652+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:32.089 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:32.087+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:32.167 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:32.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:32.165693+0000) 2022-04-04T00:23:32.193 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:23:32.197 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:23:32.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:32.697+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:33.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:33.109+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:33.167 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:33.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:33.165882+0000) 2022-04-04T00:23:33.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:33.726+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:34.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:34.069+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:34.167 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:34.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:34.166105+0000) 2022-04-04T00:23:34.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:34.738+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:35.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:35.026+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:35.167 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:35.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:35.166298+0000) 2022-04-04T00:23:35.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:35.749+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:36.040 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:36.038+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:36.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:36.166+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:36.166503+0000) 2022-04-04T00:23:36.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:36.738+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:37.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:37.025+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:37.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:37.166+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:37.166716+0000) 2022-04-04T00:23:37.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:37.719+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:37.994 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:37.992+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:38.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:38.166+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:38.166915+0000) 2022-04-04T00:23:38.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:38.720+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:38.972 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:38.970+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:39.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:39.166+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:39.167118+0000) 2022-04-04T00:23:39.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:39.700+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:39.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:39.956+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:40.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:40.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:40.167292+0000) 2022-04-04T00:23:40.239 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:23:40.237+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:23:40.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:40.727+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:40.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:40.953+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:41.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:41.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:41.167492+0000) 2022-04-04T00:23:41.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:41.753+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:41.970 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:41.968+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:42.169 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:42.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:42.167703+0000) 2022-04-04T00:23:42.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:42.710+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:43.011 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:43.010+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:43.169 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:43.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:43.167920+0000) 2022-04-04T00:23:43.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:43.680+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:44.060 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:44.058+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:44.169 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:44.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:44.168119+0000) 2022-04-04T00:23:44.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:44.707+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:45.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:45.077+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:45.169 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:45.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:45.168393+0000) 2022-04-04T00:23:45.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:45.684+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:46.100 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:46.098+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:46.169 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:46.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:46.168594+0000) 2022-04-04T00:23:46.720 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:46.718+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:47.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:47.120+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:47.170 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:47.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:47.168743+0000) 2022-04-04T00:23:47.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:47.715+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:48.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:48.080+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:48.170 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:48.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:48.168964+0000) 2022-04-04T00:23:48.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:48.679+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:49.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:49.064+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:49.170 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:49.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:49.169152+0000) 2022-04-04T00:23:49.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:49.705+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:50.057 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:50.055+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:50.170 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:50.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:50.169350+0000) 2022-04-04T00:23:50.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:50.745+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:51.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:51.059+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:51.170 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:51.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:51.169532+0000) 2022-04-04T00:23:51.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:51.731+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:52.029 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:52.027+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:52.171 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:52.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:52.169716+0000) 2022-04-04T00:23:52.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:52.757+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:52.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:52.997+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:53.171 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:53.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:53.169924+0000) 2022-04-04T00:23:53.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:53.725+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:54.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:54.045+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:54.171 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:54.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:54.170117+0000) 2022-04-04T00:23:54.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:54.767+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:55.051 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:55.048+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:55.171 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:55.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:55.170325+0000) 2022-04-04T00:23:55.255 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:23:55.252+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:23:55.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:55.792+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:56.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:56.036+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:56.171 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:56.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:56.170564+0000) 2022-04-04T00:23:56.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:56.782+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:57.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:57.032+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:57.172 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:57.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:57.170796+0000) 2022-04-04T00:23:57.819 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:57.817+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:58.036 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:58.034+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:58.172 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:58.170+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:58.171029+0000) 2022-04-04T00:23:58.846 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:58.843+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:23:59.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:23:59.001+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:23:59.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:23:59.170+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:23:59.171233+0000) 2022-04-04T00:23:59.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:23:59.874+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:00.045 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:00.043+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:00.172 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:00.170+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:00.171414+0000) 2022-04-04T00:24:00.875 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:00.873+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:01.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:01.052+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:01.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:01.170+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:01.171616+0000) 2022-04-04T00:24:01.858 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:01.855+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:02.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:02.079+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:02.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:02.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:02.171855+0000) 2022-04-04T00:24:02.449 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:24:02.453 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:24:02.892 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:02.890+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:03.086 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:03.083+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:03.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:03.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:03.172082+0000) 2022-04-04T00:24:03.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:03.874+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:04.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:04.125+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:04.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:04.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:04.172287+0000) 2022-04-04T00:24:04.881 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:04.879+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:05.096 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:05.093+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:05.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:05.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:05.172492+0000) 2022-04-04T00:24:05.893 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:05.890+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:06.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:06.122+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:06.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:06.172+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:06.172725+0000) 2022-04-04T00:24:06.922 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:06.920+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:07.090 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:07.084+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:07.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:07.172+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:07.172957+0000) 2022-04-04T00:24:07.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:07.879+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:08.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:08.119+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:08.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:08.172+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:08.173177+0000) 2022-04-04T00:24:08.907 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:08.905+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:09.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:09.105+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:09.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:09.172+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:09.173402+0000) 2022-04-04T00:24:09.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:09.887+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:10.148 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:10.145+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:10.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:10.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:10.173604+0000) 2022-04-04T00:24:10.271 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:24:10.269+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:24:10.874 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:10.872+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:11.108 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:11.106+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:11.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:11.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:11.173829+0000) 2022-04-04T00:24:11.848 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:11.845+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:12.155 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:12.152+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:12.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:12.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:12.174044+0000) 2022-04-04T00:24:12.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:12.875+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:13.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:13.169+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:13.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:13.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:13.174258+0000) 2022-04-04T00:24:13.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:13.867+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:14.145 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:14.142+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:14.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:14.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:14.174426+0000) 2022-04-04T00:24:14.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:14.822+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:15.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:15.100+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:15.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:15.174+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:15.174631+0000) 2022-04-04T00:24:15.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:15.792+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:16.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:16.130+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:16.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:16.174+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:16.174865+0000) 2022-04-04T00:24:16.803 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:16.801+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:17.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:17.115+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:17.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:17.174+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:17.175086+0000) 2022-04-04T00:24:17.831 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:17.828+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:18.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:18.161+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:18.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:18.174+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:18.175321+0000) 2022-04-04T00:24:18.804 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:18.801+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:19.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:19.161+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:19.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:19.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:19.175548+0000) 2022-04-04T00:24:19.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:19.798+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:20.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:20.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:20.175786+0000) 2022-04-04T00:24:20.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:20.195+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:20.752 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:20.749+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:21.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:21.167+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:21.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:21.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:21.176008+0000) 2022-04-04T00:24:21.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:21.754+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:22.172 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:22.170+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:22.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:22.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:22.176239+0000) 2022-04-04T00:24:22.235 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-04T00:24:22.233+0000 7f2fd761c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T00:24:22.236 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-04T00:24:22.233+0000 7f2fd661a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T00:24:22.790 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:22.788+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:23.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:23.159+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:23.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:23.176+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:23.176449+0000) 2022-04-04T00:24:23.766 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:23.764+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:24.145 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:24.142+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:24.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:24.176+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:24.176643+0000) 2022-04-04T00:24:24.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:24.740+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:25.113 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:25.111+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:25.178 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:25.176+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:25.176822+0000) 2022-04-04T00:24:25.287 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:24:25.285+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:24:25.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:25.725+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:26.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:26.086+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:26.178 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:26.176+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:26.177028+0000) 2022-04-04T00:24:26.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:26.713+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:27.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:27.052+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:27.178 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:27.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:27.177246+0000) 2022-04-04T00:24:27.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:27.708+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:28.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:28.082+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:28.178 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:28.176+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:28.177483+0000) 2022-04-04T00:24:28.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:28.720+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:29.084 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:29.082+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:29.178 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:29.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:29.177693+0000) 2022-04-04T00:24:29.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:29.713+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:30.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:30.060+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:30.179 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:30.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:30.177863+0000) 2022-04-04T00:24:30.710 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:30.708+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:31.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:31.069+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:31.179 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:31.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:31.178054+0000) 2022-04-04T00:24:31.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:31.681+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:32.086 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:32.084+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:32.179 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:32.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:32.178262+0000) 2022-04-04T00:24:32.514 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:24:32.521 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:24:32.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:32.699+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:33.093 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:33.091+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:33.179 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:33.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:33.178480+0000) 2022-04-04T00:24:33.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:33.682+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:34.064 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:34.062+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:34.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:34.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:34.178761+0000) 2022-04-04T00:24:34.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:34.721+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:35.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:35.110+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:35.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:35.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:35.178895+0000) 2022-04-04T00:24:35.694 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:35.692+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:36.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:36.093+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:36.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:36.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:36.179077+0000) 2022-04-04T00:24:36.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:36.726+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:37.114 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:37.112+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:37.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:37.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:37.179236+0000) 2022-04-04T00:24:37.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:37.719+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:38.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:38.069+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:38.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:38.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:38.179446+0000) 2022-04-04T00:24:38.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:38.741+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:39.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:39.072+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:39.181 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:39.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:39.179653+0000) 2022-04-04T00:24:39.768 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:39.766+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:40.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:40.101+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:40.181 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:40.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:40.179831+0000) 2022-04-04T00:24:40.303 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:24:40.300+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:24:40.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:40.813+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:41.072 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:41.070+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:41.181 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:41.179+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:41.180040+0000) 2022-04-04T00:24:41.776 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:41.774+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:42.114 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:42.109+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:42.181 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:42.179+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:42.180257+0000) 2022-04-04T00:24:42.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:42.734+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:43.155 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:43.153+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:43.181 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:43.179+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:43.180481+0000) 2022-04-04T00:24:43.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:43.720+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:44.138 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:44.136+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:44.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:44.179+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:44.180696+0000) 2022-04-04T00:24:44.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:44.676+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:45.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:45.165+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:45.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:45.180+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:45.180903+0000) 2022-04-04T00:24:45.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:45.680+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:46.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:46.124+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:46.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:46.180+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:46.181119+0000) 2022-04-04T00:24:46.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:46.644+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:47.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:47.129+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:47.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:47.180+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:47.181352+0000) 2022-04-04T00:24:47.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:47.611+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:48.084 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:48.081+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:48.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:48.180+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:48.181558+0000) 2022-04-04T00:24:48.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:48.643+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:49.089 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:49.087+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:49.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:49.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:49.181753+0000) 2022-04-04T00:24:49.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:49.684+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:50.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:50.085+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:50.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:50.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:50.181967+0000) 2022-04-04T00:24:50.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:50.644+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:51.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:51.078+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:51.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:51.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:51.182166+0000) 2022-04-04T00:24:51.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:51.614+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:52.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:52.033+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:52.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:52.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:52.182377+0000) 2022-04-04T00:24:52.577 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:52.574+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 17 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:53.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:53.034+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:53.184 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:53.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:53.182536+0000) 2022-04-04T00:24:53.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:53.611+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:54.020 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:54.017+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:54.184 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:54.182+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:54.182752+0000) 2022-04-04T00:24:54.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:54.599+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:54.986 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:54.984+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:55.184 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:55.182+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:55.182981+0000) 2022-04-04T00:24:55.319 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:24:55.317+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:24:55.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:55.648+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:55.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:55.945+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:56.184 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:56.182+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:56.183159+0000) 2022-04-04T00:24:56.609 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:56.607+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:56.902 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:56.899+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:57.184 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:57.182+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:57.183372+0000) 2022-04-04T00:24:57.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:57.568+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:57.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:57.894+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:58.185 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:58.183+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:58.183593+0000) 2022-04-04T00:24:58.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:58.608+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:58.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:58.883+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:24:59.185 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:24:59.183+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:24:59.183836+0000) 2022-04-04T00:24:59.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:24:59.585+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:24:59.920 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:24:59.918+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:00.185 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:00.183+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:00.183949+0000) 2022-04-04T00:25:00.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:00.616+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:00.920 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:00.917+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:01.185 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:01.183+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:01.184167+0000) 2022-04-04T00:25:01.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:01.581+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:01.910 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:01.907+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:02.185 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:02.183+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:02.184357+0000) 2022-04-04T00:25:02.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:02.562+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:02.669 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:25:02.674 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:25:02.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:02.954+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:03.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:03.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:03.184592+0000) 2022-04-04T00:25:03.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:03.590+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:03.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:03.995+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:04.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:04.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:04.184821+0000) 2022-04-04T00:25:04.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:04.577+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:05.014 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:05.012+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:05.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:05.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:05.184954+0000) 2022-04-04T00:25:05.533 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:05.531+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:06.018 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:06.016+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:06.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:06.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:06.185172+0000) 2022-04-04T00:25:06.540 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:06.538+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:07.043 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:07.040+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:07.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:07.185+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:07.185380+0000) 2022-04-04T00:25:07.500 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:07.498+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:07.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:07.995+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:08.187 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:08.185+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:08.185587+0000) 2022-04-04T00:25:08.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:08.519+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:08.968 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:08.965+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:09.187 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:09.185+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:09.185831+0000) 2022-04-04T00:25:09.510 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:09.507+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:09.954 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:09.952+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:10.187 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:10.185+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:10.186000+0000) 2022-04-04T00:25:10.322 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:25:10.320+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:25:10.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:10.518+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:10.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:10.928+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:11.187 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:11.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:11.186173+0000) 2022-04-04T00:25:11.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:11.511+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:11.892 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:11.890+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:12.187 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:12.185+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:12.186340+0000) 2022-04-04T00:25:12.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:12.463+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:12.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:12.852+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:13.188 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:13.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:13.186542+0000) 2022-04-04T00:25:13.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:13.495+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:13.866 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:13.864+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:14.188 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:14.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:14.186758+0000) 2022-04-04T00:25:14.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:14.540+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:14.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:14.887+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:15.188 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:15.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:15.186990+0000) 2022-04-04T00:25:15.526 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:15.523+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:15.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:15.918+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:16.188 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:16.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:16.187198+0000) 2022-04-04T00:25:16.478 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:16.475+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:16.920 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:16.918+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:17.188 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:17.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:17.187414+0000) 2022-04-04T00:25:17.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:17.450+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:17.874 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:17.872+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:18.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:18.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:18.187643+0000) 2022-04-04T00:25:18.474 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:18.471+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:18.884 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:18.882+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:19.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:19.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:19.187862+0000) 2022-04-04T00:25:19.498 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:19.496+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:19.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:19.858+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:20.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:20.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:20.188075+0000) 2022-04-04T00:25:20.478 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:20.475+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:20.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:20.853+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:21.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:21.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:21.188235+0000) 2022-04-04T00:25:21.489 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:21.486+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:21.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:21.822+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:22.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:22.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:22.188448+0000) 2022-04-04T00:25:22.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:22.490+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:22.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:22.791+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:23.190 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:23.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:23.188665+0000) 2022-04-04T00:25:23.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:23.526+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:23.767 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:23.765+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:24.190 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:24.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:24.188821+0000) 2022-04-04T00:25:24.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:24.478+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:24.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:24.782+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:25.190 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:25.188+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:25.189005+0000) 2022-04-04T00:25:25.338 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:25:25.336+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:25:25.463 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:25.460+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:25.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:25.732+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:26.190 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:26.188+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:26.189152+0000) 2022-04-04T00:25:26.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:26.454+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:26.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:26.758+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:27.190 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:27.188+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:27.189336+0000) 2022-04-04T00:25:27.463 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:27.461+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:27.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:27.737+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:28.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:28.188+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:28.189468+0000) 2022-04-04T00:25:28.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:28.443+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:28.716 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:28.714+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:29.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:29.188+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:29.189688+0000) 2022-04-04T00:25:29.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:29.446+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:29.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:29.698+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:30.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:30.189+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:30.189881+0000) 2022-04-04T00:25:30.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:30.412+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:30.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:30.741+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:31.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:31.189+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:31.190065+0000) 2022-04-04T00:25:31.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:31.369+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:31.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:31.725+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:32.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:32.189+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:32.190258+0000) 2022-04-04T00:25:32.402 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:32.400+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:32.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:32.740+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:32.801 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:25:32.805 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:25:33.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:33.189+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:33.190501+0000) 2022-04-04T00:25:33.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:33.375+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:33.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:33.715+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:34.192 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:34.190+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:34.190704+0000) 2022-04-04T00:25:34.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:34.347+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:34.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:34.667+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:35.192 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:35.190+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:35.190899+0000) 2022-04-04T00:25:35.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:35.365+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:35.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:35.692+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:36.192 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:36.190+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:36.191038+0000) 2022-04-04T00:25:36.405 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:36.404+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:36.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:36.700+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:37.192 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:37.190+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:37.191226+0000) 2022-04-04T00:25:37.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:37.358+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:37.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:37.685+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:38.192 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:38.190+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:38.191425+0000) 2022-04-04T00:25:38.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:38.351+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:38.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:38.653+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:39.193 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:39.190+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:39.191629+0000) 2022-04-04T00:25:39.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:39.301+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:39.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:39.686+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:40.193 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:40.191+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:40.191865+0000) 2022-04-04T00:25:40.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:40.320+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:40.354 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:25:40.352+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:25:40.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:40.645+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:41.193 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:41.191+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:41.192051+0000) 2022-04-04T00:25:41.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:41.304+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:41.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:41.638+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:42.193 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:42.191+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:42.192249+0000) 2022-04-04T00:25:42.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:42.278+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:42.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:42.664+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:43.194 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:43.191+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:43.192450+0000) 2022-04-04T00:25:43.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:43.301+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:43.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:43.665+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:44.194 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:44.192+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:44.192659+0000) 2022-04-04T00:25:44.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:44.319+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:44.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:44.618+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:45.194 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:45.192+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:45.192867+0000) 2022-04-04T00:25:45.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:45.289+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:45.591 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:45.588+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:46.194 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:46.192+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:46.193090+0000) 2022-04-04T00:25:46.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:46.310+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:46.557 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:46.554+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:47.194 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:47.192+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:47.193292+0000) 2022-04-04T00:25:47.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:47.274+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:47.526 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:47.523+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:48.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:48.193+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:48.193495+0000) 2022-04-04T00:25:48.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:48.282+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:48.520 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:48.517+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:49.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:49.193+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:49.193699+0000) 2022-04-04T00:25:49.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:49.261+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:49.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:49.498+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:50.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:50.193+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:50.193890+0000) 2022-04-04T00:25:50.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:50.255+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:50.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:50.526+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:51.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:51.193+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:51.194089+0000) 2022-04-04T00:25:51.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:51.271+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:51.500 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:51.497+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:52.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:52.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:52.194264+0000) 2022-04-04T00:25:52.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:52.259+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:52.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:52.546+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:53.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:53.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:53.194476+0000) 2022-04-04T00:25:53.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:53.213+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:53.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:53.545+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:54.174 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:54.172+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:54.196 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:54.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:54.194670+0000) 2022-04-04T00:25:54.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:54.582+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:55.196 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:55.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:55.194865+0000) 2022-04-04T00:25:55.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:55.208+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:55.370 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:25:55.368+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:25:55.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:55.546+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:56.196 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:56.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:56.195053+0000) 2022-04-04T00:25:56.206 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:56.203+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:56.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:56.576+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:57.196 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:57.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:57.195211+0000) 2022-04-04T00:25:57.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:57.198+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:57.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:57.613+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:58.196 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:58.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:58.195389+0000) 2022-04-04T00:25:58.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:58.206+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:58.596 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:58.594+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:25:59.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:25:59.193+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:25:59.197 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:25:59.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:25:59.195590+0000) 2022-04-04T00:25:59.578 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:25:59.576+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:00.197 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:00.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:00.195840+0000) 2022-04-04T00:26:00.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:00.240+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:00.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:00.617+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:01.197 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:01.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:01.196055+0000) 2022-04-04T00:26:01.214 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:01.212+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:01.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:01.577+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:02.197 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:02.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:02.196265+0000) 2022-04-04T00:26:02.235 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:02.232+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:02.615 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:02.613+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:02.941 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:26:02.945 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:26:03.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:03.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:03.196508+0000) 2022-04-04T00:26:03.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:03.251+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:03.617 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:03.614+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:04.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:04.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:04.196709+0000) 2022-04-04T00:26:04.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:04.258+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:04.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:04.661+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:05.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:05.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:05.196889+0000) 2022-04-04T00:26:05.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:05.297+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:05.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:05.698+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:06.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:06.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:06.197088+0000) 2022-04-04T00:26:06.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:06.326+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:06.747 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:06.745+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:07.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:07.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:07.197240+0000) 2022-04-04T00:26:07.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:07.344+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:07.707 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:07.704+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:08.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:08.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:08.197406+0000) 2022-04-04T00:26:08.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:08.312+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:08.678 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:08.676+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:09.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:09.196+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:09.197610+0000) 2022-04-04T00:26:09.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:09.349+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:09.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:09.638+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:10.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:10.196+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:10.197740+0000) 2022-04-04T00:26:10.364 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:10.361+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:10.386 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:26:10.383+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:26:10.609 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:10.607+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:11.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:11.197+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:11.197924+0000) 2022-04-04T00:26:11.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:11.332+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:11.649 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:11.646+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:12.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:12.197+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:12.198098+0000) 2022-04-04T00:26:12.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:12.310+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:12.627 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:12.625+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:13.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:13.197+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:13.198294+0000) 2022-04-04T00:26:13.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:13.308+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:13.672 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:13.669+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:14.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:14.197+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:14.198416+0000) 2022-04-04T00:26:14.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:14.271+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:14.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:14.692+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:15.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:15.197+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:15.198573+0000) 2022-04-04T00:26:15.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:15.258+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:15.651 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:15.649+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:16.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:16.197+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:16.198713+0000) 2022-04-04T00:26:16.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:16.241+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:16.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:16.677+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:17.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:17.198+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:17.198895+0000) 2022-04-04T00:26:17.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:17.262+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:17.706 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:17.704+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:18.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:18.198+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:18.199032+0000) 2022-04-04T00:26:18.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:18.232+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:18.711 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:18.709+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:19.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:19.198+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:19.199229+0000) 2022-04-04T00:26:19.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:19.275+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:19.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:19.733+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:20.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:20.198+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:20.199400+0000) 2022-04-04T00:26:20.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:20.243+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:20.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:20.740+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:21.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:21.198+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:21.199546+0000) 2022-04-04T00:26:21.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:21.264+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:21.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:21.727+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:22.201 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:22.199+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:22.199690+0000) 2022-04-04T00:26:22.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:22.242+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:22.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:22.773+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:23.201 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:23.199+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:23.199911+0000) 2022-04-04T00:26:23.212 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:23.210+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:23.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:23.731+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:24.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:24.162+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:24.201 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:24.199+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:24.200148+0000) 2022-04-04T00:26:24.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:24.711+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:25.131 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:25.129+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:25.201 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:25.199+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:25.200323+0000) 2022-04-04T00:26:25.402 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:26:25.400+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:26:25.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:25.730+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:26.176 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:26.174+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:26.201 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:26.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:26.200543+0000) 2022-04-04T00:26:26.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:26.760+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:27.202 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:27.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:27.200676+0000) 2022-04-04T00:26:27.206 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:27.204+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:27.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:27.739+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:28.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:28.162+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:28.202 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:28.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:28.200858+0000) 2022-04-04T00:26:28.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:28.778+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:29.117 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:29.115+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:29.202 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:29.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:29.200980+0000) 2022-04-04T00:26:29.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:29.827+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:30.125 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:30.123+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:30.202 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:30.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:30.201154+0000) 2022-04-04T00:26:30.857 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:30.855+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:31.142 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:31.141+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:31.202 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:31.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:31.201342+0000) 2022-04-04T00:26:31.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:31.874+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:32.183 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:32.181+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:32.202 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:32.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:32.201510+0000) 2022-04-04T00:26:32.916 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:32.914+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:33.089 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:26:33.094 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:26:33.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:33.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:33.201704+0000) 2022-04-04T00:26:33.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:33.220+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:33.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:33.922+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:34.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:34.187+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:34.202 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:34.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:34.201828+0000) 2022-04-04T00:26:34.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:34.920+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:35.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:35.200+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:35.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:35.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:35.201986+0000) 2022-04-04T00:26:35.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:35.886+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:36.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:36.165+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:36.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:36.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:36.202124+0000) 2022-04-04T00:26:36.919 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:36.917+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:37.162 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:37.161+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:37.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:37.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:37.202295+0000) 2022-04-04T00:26:37.898 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:37.895+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:38.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:38.182+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:38.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:38.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:38.202421+0000) 2022-04-04T00:26:38.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:38.914+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:39.202 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:39.199+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:39.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:39.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:39.202632+0000) 2022-04-04T00:26:39.920 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:39.917+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:40.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:40.190+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:40.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:40.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:40.202797+0000) 2022-04-04T00:26:40.418 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:26:40.415+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:26:40.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:40.876+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:41.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:41.156+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:41.204 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:41.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:41.202942+0000) 2022-04-04T00:26:41.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:41.925+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:42.129 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:42.127+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:42.204 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:42.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:42.203036+0000) 2022-04-04T00:26:42.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:42.950+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:43.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:43.162+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:43.204 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:43.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:43.203224+0000) 2022-04-04T00:26:43.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:43.950+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:44.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:44.187+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:44.204 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:44.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:44.203349+0000) 2022-04-04T00:26:44.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:44.992+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:45.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:45.164+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:45.205 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:45.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:45.203567+0000) 2022-04-04T00:26:45.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:45.982+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:46.171 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:46.168+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:46.205 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:46.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:46.203743+0000) 2022-04-04T00:26:46.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:46.948+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:47.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:47.156+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:47.207 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:47.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:47.203964+0000) 2022-04-04T00:26:47.986 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:47.983+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:48.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:48.172+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:48.205 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:48.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:48.204138+0000) 2022-04-04T00:26:49.011 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:49.008+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:49.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:49.184+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:49.205 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:49.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:49.204325+0000) 2022-04-04T00:26:49.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:49.963+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:50.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:50.147+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:50.205 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:50.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:50.204481+0000) 2022-04-04T00:26:50.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:50.950+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:51.120 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:51.118+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:51.205 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:51.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:51.204627+0000) 2022-04-04T00:26:51.925 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:51.923+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:52.146 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:52.143+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:52.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:52.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:52.204842+0000) 2022-04-04T00:26:52.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:52.936+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:53.116 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:53.114+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:53.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:53.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:53.205057+0000) 2022-04-04T00:26:53.932 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:53.930+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:54.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:54.153+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:54.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:54.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:54.205205+0000) 2022-04-04T00:26:54.956 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:54.954+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:55.135 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:55.133+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:55.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:55.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:55.205415+0000) 2022-04-04T00:26:55.433 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:26:55.431+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:26:55.976 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:55.973+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:56.106 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:56.104+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:56.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:56.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:56.205575+0000) 2022-04-04T00:26:56.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:56.995+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:57.062 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:57.060+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:57.207 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:57.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:57.205740+0000) 2022-04-04T00:26:58.036 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:58.033+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:58.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:58.103+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:58.207 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:58.205+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:58.205890+0000) 2022-04-04T00:26:59.073 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:26:59.071+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:26:59.084 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:26:59.081+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:26:59.207 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:26:59.205+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:26:59.206086+0000) 2022-04-04T00:27:00.039 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:00.037+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:00.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:00.104+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:00.207 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:00.205+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:00.206298+0000) 2022-04-04T00:27:01.010 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:01.007+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:01.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:01.109+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:01.207 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:01.205+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:01.206480+0000) 2022-04-04T00:27:01.984 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:01.982+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:02.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:02.115+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:02.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:02.206+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:02.206698+0000) 2022-04-04T00:27:03.015 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:03.012+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:03.150 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:03.148+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:03.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:03.206+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:03.206886+0000) 2022-04-04T00:27:03.229 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:27:03.233 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:27:04.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:04.053+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:04.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:04.154+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:04.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:04.206+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:04.207112+0000) 2022-04-04T00:27:05.041 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:05.038+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:05.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:05.132+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:05.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:05.206+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:05.207256+0000) 2022-04-04T00:27:06.059 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:06.056+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:06.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:06.130+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:06.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:06.206+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:06.207457+0000) 2022-04-04T00:27:07.088 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:07.085+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:07.101 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:07.099+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:07.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:07.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:07.207662+0000) 2022-04-04T00:27:08.132 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:08.129+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:08.142 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:08.140+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:08.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:08.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:08.207879+0000) 2022-04-04T00:27:09.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:09.119+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:09.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:09.161+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:09.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:09.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:09.208009+0000) 2022-04-04T00:27:10.105 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:10.103+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:10.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:10.198+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:10.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:10.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:10.208152+0000) 2022-04-04T00:27:10.449 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:27:10.447+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:27:11.133 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:11.130+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:11.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:11.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:11.208298+0000) 2022-04-04T00:27:11.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:11.234+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:12.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:12.124+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:12.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:12.198+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:12.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:12.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:12.208450+0000) 2022-04-04T00:27:13.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:13.154+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:13.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:13.205+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:13.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:13.208+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:13.208586+0000) 2022-04-04T00:27:14.110 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:14.107+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:14.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:14.201+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:14.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:14.208+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:14.208814+0000) 2022-04-04T00:27:15.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:15.063+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:15.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:15.208+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:15.208975+0000) 2022-04-04T00:27:15.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:15.236+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:16.036 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:16.034+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:16.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:16.208+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:16.209176+0000) 2022-04-04T00:27:16.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:16.214+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:17.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:17.044+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:17.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:17.196+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:17.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:17.208+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:17.209311+0000) 2022-04-04T00:27:18.092 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:18.091+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:18.152 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:18.151+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:18.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:18.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:18.209484+0000) 2022-04-04T00:27:19.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:19.110+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:19.118 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:19.117+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:19.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:19.208+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:19.209710+0000) 2022-04-04T00:27:20.106 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:20.105+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:20.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:20.135+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:20.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:20.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:20.209881+0000) 2022-04-04T00:27:21.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:21.073+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:21.165 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:21.164+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:21.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:21.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:21.209992+0000) 2022-04-04T00:27:22.040 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:22.038+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:22.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:22.199+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:22.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:22.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:22.210132+0000) 2022-04-04T00:27:23.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:23.078+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:23.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:23.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:23.210314+0000) 2022-04-04T00:27:23.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:23.230+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:24.070 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:24.069+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:24.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:24.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:24.210518+0000) 2022-04-04T00:27:24.214 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:24.213+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:25.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:25.059+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:25.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:25.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:25.210730+0000) 2022-04-04T00:27:25.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:25.242+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:25.465 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:27:25.463+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:27:26.022 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:26.020+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:26.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:26.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:26.210962+0000) 2022-04-04T00:27:26.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:26.266+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:26.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:26.989+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:27.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:27.211+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:27.211132+0000) 2022-04-04T00:27:27.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:27.237+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:28.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:28.013+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:28.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:28.209+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:28.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:28.211+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:28.211306+0000) 2022-04-04T00:27:28.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:28.979+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:29.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:29.162+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:29.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:29.211+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:29.211487+0000) 2022-04-04T00:27:30.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:30.023+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:30.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:30.202+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:30.213 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:30.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:30.211682+0000) 2022-04-04T00:27:31.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:31.031+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:31.213 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:31.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:31.211882+0000) 2022-04-04T00:27:31.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:31.246+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:32.006 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:32.004+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:32.213 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:32.211+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:32.212113+0000) 2022-04-04T00:27:32.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:32.255+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:33.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:33.045+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:33.213 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:33.211+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:33.212328+0000) 2022-04-04T00:27:33.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:33.267+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:33.366 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:27:33.370 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:27:34.092 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:34.090+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:34.213 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:34.211+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:34.212573+0000) 2022-04-04T00:27:34.284 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:34.281+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:35.094 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:35.091+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:35.214 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:35.211+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:35.212786+0000) 2022-04-04T00:27:35.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:35.273+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:36.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:36.110+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:36.214 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:36.212+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:36.213008+0000) 2022-04-04T00:27:36.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:36.296+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:37.148 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:37.145+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:37.214 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:37.212+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:37.213227+0000) 2022-04-04T00:27:37.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:37.323+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:38.110 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:38.107+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:38.214 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:38.212+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:38.213439+0000) 2022-04-04T00:27:38.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:38.278+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:39.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:39.125+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:39.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:39.212+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:39.213655+0000) 2022-04-04T00:27:39.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:39.229+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:40.152 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:40.149+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:40.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:40.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:40.213868+0000) 2022-04-04T00:27:40.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:40.276+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:40.476 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:27:40.474+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:27:41.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:41.179+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:41.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:41.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:41.214070+0000) 2022-04-04T00:27:41.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:41.285+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:42.210 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:42.208+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:42.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:42.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:42.214291+0000) 2022-04-04T00:27:42.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:42.296+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:43.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:43.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:43.214515+0000) 2022-04-04T00:27:43.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:43.226+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:43.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:43.272+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:44.199 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:44.197+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:44.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:44.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:44.214796+0000) 2022-04-04T00:27:44.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:44.250+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:45.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:45.208+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:45.216 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:45.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:45.215011+0000) 2022-04-04T00:27:45.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:45.229+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:46.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:46.177+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:46.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:46.198+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:46.216 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:46.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:46.215220+0000) 2022-04-04T00:27:47.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:47.141+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:47.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:47.153+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:47.216 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:47.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:47.215404+0000) 2022-04-04T00:27:48.101 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:48.098+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:48.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:48.190+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:48.216 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:48.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:48.215552+0000) 2022-04-04T00:27:49.101 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:49.099+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:49.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:49.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:49.215703+0000) 2022-04-04T00:27:49.220 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:49.217+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:50.114 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:50.112+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:50.172 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:50.170+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:50.216 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:50.215+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:50.215813+0000) 2022-04-04T00:27:51.101 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:51.099+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:51.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:51.154+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:51.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:51.215+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:51.216015+0000) 2022-04-04T00:27:52.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:52.105+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:52.138 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:52.136+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:52.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:52.215+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:52.216166+0000) 2022-04-04T00:27:53.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:53.124+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:53.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:53.184+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:53.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:53.215+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:53.216350+0000) 2022-04-04T00:27:54.094 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:54.092+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:54.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:54.197+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:54.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:54.215+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:54.216530+0000) 2022-04-04T00:27:55.064 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:55.062+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:55.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:55.216+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:55.216670+0000) 2022-04-04T00:27:55.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:55.227+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:55.492 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:27:55.490+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:27:56.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:56.039+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:56.218 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:56.216+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:56.216844+0000) 2022-04-04T00:27:56.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:56.259+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:57.086 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:57.083+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:57.218 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:57.216+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:57.217053+0000) 2022-04-04T00:27:57.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:57.298+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:58.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:58.109+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:58.218 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:58.216+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:58.217264+0000) 2022-04-04T00:27:58.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:58.284+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:27:59.078 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:27:59.076+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:27:59.218 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:27:59.217+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:27:59.217478+0000) 2022-04-04T00:27:59.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:27:59.306+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:00.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:00.030+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:00.219 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:00.217+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:00.217701+0000) 2022-04-04T00:28:00.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:00.355+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:00.987 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:00.985+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:01.219 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:01.217+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:01.217880+0000) 2022-04-04T00:28:01.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:01.350+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:02.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:02.030+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:02.219 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:02.217+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:02.218061+0000) 2022-04-04T00:28:02.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:02.383+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:03.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:03.037+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:03.219 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:03.217+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:03.218273+0000) 2022-04-04T00:28:03.402 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:03.400+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:03.525 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:28:03.528 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:28:04.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:04.030+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:04.219 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:04.217+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:04.218517+0000) 2022-04-04T00:28:04.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:04.433+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:05.032 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:05.030+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:05.219 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:05.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:05.218707+0000) 2022-04-04T00:28:05.480 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:05.478+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:06.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:06.023+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:06.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:06.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:06.218914+0000) 2022-04-04T00:28:06.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:06.479+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:07.070 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:07.068+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:07.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:07.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:07.219073+0000) 2022-04-04T00:28:07.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:07.454+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:08.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:08.061+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:08.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:08.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:08.219195+0000) 2022-04-04T00:28:08.489 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:08.487+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:09.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:09.077+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:09.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:09.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:09.219331+0000) 2022-04-04T00:28:09.502 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:09.500+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:10.099 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:10.097+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:10.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:10.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:10.219472+0000) 2022-04-04T00:28:10.502 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:10.500+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:10.508 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:28:10.506+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:28:11.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:11.131+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:11.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:11.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:11.219641+0000) 2022-04-04T00:28:11.552 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:11.549+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:12.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:12.137+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:12.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:12.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:12.219816+0000) 2022-04-04T00:28:12.566 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:12.565+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:13.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:13.165+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:13.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:13.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:13.220019+0000) 2022-04-04T00:28:13.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:13.582+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:14.196 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:14.194+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:14.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:14.220+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:14.220173+0000) 2022-04-04T00:28:14.621 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:14.619+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:15.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:15.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:15.220357+0000) 2022-04-04T00:28:15.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:15.244+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:15.656 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:15.654+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:16.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:16.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:16.220483+0000) 2022-04-04T00:28:16.252 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:16.251+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:16.693 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:16.692+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:17.216 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:17.215+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:17.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:17.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:17.220620+0000) 2022-04-04T00:28:17.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:17.699+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:18.222 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:18.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:18.220839+0000) 2022-04-04T00:28:18.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:18.259+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:18.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:18.727+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:19.222 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:19.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:19.220977+0000) 2022-04-04T00:28:19.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:19.276+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:19.727 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:19.725+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:20.222 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:20.220+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:20.221106+0000) 2022-04-04T00:28:20.294 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:20.292+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:20.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:20.758+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:21.222 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:21.220+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:21.221243+0000) 2022-04-04T00:28:21.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:21.277+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:21.746 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:21.744+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:22.222 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:22.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:22.221423+0000) 2022-04-04T00:28:22.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:22.315+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:22.786 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:22.784+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:23.222 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:23.220+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:23.221632+0000) 2022-04-04T00:28:23.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:23.276+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:23.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:23.754+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:24.223 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:24.220+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:24.221853+0000) 2022-04-04T00:28:24.256 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:24.254+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:24.717 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:24.716+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:25.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:25.213+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:25.223 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:25.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:25.222070+0000) 2022-04-04T00:28:25.524 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:28:25.522+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:28:25.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:25.745+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:26.223 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:26.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:26.222282+0000) 2022-04-04T00:28:26.240 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:26.238+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:26.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:26.745+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:27.196 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:27.194+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:27.223 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:27.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:27.222496+0000) 2022-04-04T00:28:27.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:27.774+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:28.217 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:28.214+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:28.223 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:28.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:28.222717+0000) 2022-04-04T00:28:28.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:28.745+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:29.200 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:29.197+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:29.224 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:29.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:29.222864+0000) 2022-04-04T00:28:29.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:29.727+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:30.224 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:30.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:30.223066+0000) 2022-04-04T00:28:30.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:30.224+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:30.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:30.694+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:31.224 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:31.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:31.223234+0000) 2022-04-04T00:28:31.236 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:31.233+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:31.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:31.666+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:32.222 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:32.220+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:32.224 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:32.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:32.223442+0000) 2022-04-04T00:28:32.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:32.711+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:33.212 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:33.209+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:33.225 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:33.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:33.223661+0000) 2022-04-04T00:28:33.681 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:28:33.685 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:28:33.759 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:33.756+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:34.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:34.200+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:34.225 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:34.223+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:34.223908+0000) 2022-04-04T00:28:34.766 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:34.762+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:35.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:35.212+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:35.225 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:35.223+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:35.224116+0000) 2022-04-04T00:28:35.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:35.723+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:36.225 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:36.223+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:36.224342+0000) 2022-04-04T00:28:36.243 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:36.240+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:36.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:36.735+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:37.225 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:37.223+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:37.224490+0000) 2022-04-04T00:28:37.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:37.286+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:37.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:37.698+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:38.225 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:38.224+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:38.224645+0000) 2022-04-04T00:28:38.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:38.313+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:38.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:38.714+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:39.226 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:39.224+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:39.224851+0000) 2022-04-04T00:28:39.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:39.275+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:39.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:39.754+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:40.226 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:40.224+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:40.225060+0000) 2022-04-04T00:28:40.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:40.321+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:40.540 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:28:40.538+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:28:40.772 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:40.770+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:41.226 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:41.224+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:41.225212+0000) 2022-04-04T00:28:41.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:41.311+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:41.818 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:41.815+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:42.226 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:42.224+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:42.225392+0000) 2022-04-04T00:28:42.291 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:42.289+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:42.858 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:42.855+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:43.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:43.225+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:43.225569+0000) 2022-04-04T00:28:43.332 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:43.329+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:43.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:43.870+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:44.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:44.225+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:44.225760+0000) 2022-04-04T00:28:44.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:44.378+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:44.901 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:44.899+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:45.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:45.225+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:45.225902+0000) 2022-04-04T00:28:45.354 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:45.352+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:45.895 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:45.893+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:46.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:46.225+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:46.226080+0000) 2022-04-04T00:28:46.337 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:46.334+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:46.908 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:46.905+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:47.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:47.225+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:47.226221+0000) 2022-04-04T00:28:47.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:47.308+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:47.879 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:47.876+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:48.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:48.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:48.226361+0000) 2022-04-04T00:28:48.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:48.262+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:48.893 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:48.890+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:49.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:49.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:49.226588+0000) 2022-04-04T00:28:49.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:49.241+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:49.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:49.861+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:50.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:50.212+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:50.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:50.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:50.226766+0000) 2022-04-04T00:28:50.845 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:50.843+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:51.214 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:51.212+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:51.228 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:51.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:51.226980+0000) 2022-04-04T00:28:51.858 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:51.855+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:52.213 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:52.209+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:52.228 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:52.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:52.227188+0000) 2022-04-04T00:28:52.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:52.830+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:53.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:53.187+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:53.228 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:53.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:53.227394+0000) 2022-04-04T00:28:53.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:53.830+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:54.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:54.213+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:54.228 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:54.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:54.227632+0000) 2022-04-04T00:28:54.790 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:54.787+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:55.224 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:55.222+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:55.228 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:55.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:55.227846+0000) 2022-04-04T00:28:55.556 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:28:55.554+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:28:55.810 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:55.807+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:56.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:56.225+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:56.229 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:56.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:56.228013+0000) 2022-04-04T00:28:56.846 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:56.843+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:57.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:57.204+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:57.229 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:57.228+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:57.228200+0000) 2022-04-04T00:28:57.849 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:57.846+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:58.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:58.165+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:58.229 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:58.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:58.228396+0000) 2022-04-04T00:28:58.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:58.879+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:28:59.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:28:59.213+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:28:59.229 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:28:59.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:28:59.228625+0000) 2022-04-04T00:28:59.917 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:28:59.915+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:00.230 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:00.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:00.228850+0000) 2022-04-04T00:29:00.253 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:00.251+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:00.951 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:00.949+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:01.230 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:01.228+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:01.229105+0000) 2022-04-04T00:29:01.247 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:01.244+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:01.906 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:01.904+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:02.230 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:02.228+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:02.229342+0000) 2022-04-04T00:29:02.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:02.287+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:02.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:02.913+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:03.230 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:03.228+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:03.229558+0000) 2022-04-04T00:29:03.262 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:03.261+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:03.812 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:29:03.815 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:29:03.895 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:03.893+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:04.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:04.228+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:04.229817+0000) 2022-04-04T00:29:04.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:04.242+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:04.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:04.907+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:05.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:05.229+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:05.230015+0000) 2022-04-04T00:29:05.251 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:05.250+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:05.883 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:05.881+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:06.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:06.229+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:06.230202+0000) 2022-04-04T00:29:06.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:06.244+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:06.891 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:06.889+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:07.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:07.229+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:07.230407+0000) 2022-04-04T00:29:07.237 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:07.235+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:07.911 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:07.909+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:08.232 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:08.229+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:08.230625+0000) 2022-04-04T00:29:08.241 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:08.239+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:08.880 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:08.879+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:09.208 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:09.207+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:09.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:09.229+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:09.230827+0000) 2022-04-04T00:29:09.917 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:09.915+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:10.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:10.168+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:10.232 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:10.230+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:10.231017+0000) 2022-04-04T00:29:10.572 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:29:10.570+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:29:10.959 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:10.957+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:11.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:11.194+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:11.232 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:11.230+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:11.231173+0000) 2022-04-04T00:29:11.974 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:11.972+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:12.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:12.199+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:12.232 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:12.230+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:12.231385+0000) 2022-04-04T00:29:12.970 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:12.969+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:13.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:13.230+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:13.231615+0000) 2022-04-04T00:29:13.236 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:13.235+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:13.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:13.994+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:14.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:14.187+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:14.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:14.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:14.231835+0000) 2022-04-04T00:29:14.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:14.991+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:15.217 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:15.215+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:15.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:15.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:15.232014+0000) 2022-04-04T00:29:16.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:16.010+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:16.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:16.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:16.232218+0000) 2022-04-04T00:29:16.256 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:16.255+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:17.040 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:17.038+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:17.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:17.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:17.232413+0000) 2022-04-04T00:29:17.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:17.277+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:17.994 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:17.992+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:18.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:18.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:18.232615+0000) 2022-04-04T00:29:18.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:18.319+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:18.961 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:18.959+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:19.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:19.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:19.232852+0000) 2022-04-04T00:29:19.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:19.284+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:20.000 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:19.999+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:20.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:20.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:20.233058+0000) 2022-04-04T00:29:20.261 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:20.260+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:20.997 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:20.996+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:21.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:21.227+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:21.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:21.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:21.233279+0000) 2022-04-04T00:29:22.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:22.001+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:22.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:22.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:22.233507+0000) 2022-04-04T00:29:22.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:22.257+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:22.971 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:22.969+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:23.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:23.233+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:23.233722+0000) 2022-04-04T00:29:23.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:23.292+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:23.927 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:23.924+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:24.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:24.233+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:24.233893+0000) 2022-04-04T00:29:24.289 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:24.287+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:24.904 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:24.901+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:25.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:25.233+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:25.234097+0000) 2022-04-04T00:29:25.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:25.274+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:25.588 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:29:25.586+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:29:25.874 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:25.872+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:26.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:26.233+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:26.234298+0000) 2022-04-04T00:29:26.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:26.277+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:26.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:26.867+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:27.234 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:27.231+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:27.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:27.234+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:27.234451+0000) 2022-04-04T00:29:27.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:27.894+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:28.236 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:28.234+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:28.234651+0000) 2022-04-04T00:29:28.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:28.276+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:28.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:28.861+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:29.230 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:29.228+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:29.236 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:29.234+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:29.234815+0000) 2022-04-04T00:29:29.831 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:29.829+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:30.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:30.193+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:30.236 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:30.234+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:30.235026+0000) 2022-04-04T00:29:30.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:30.789+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:31.236 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:31.234+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:31.235238+0000) 2022-04-04T00:29:31.238 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:31.235+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:31.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:31.747+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:32.236 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:32.235+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:32.235470+0000) 2022-04-04T00:29:32.283 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:32.281+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:32.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:32.698+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:33.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:33.235+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:33.235675+0000) 2022-04-04T00:29:33.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:33.286+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:33.712 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:33.709+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:33.979 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:29:33.983 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:29:34.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:34.235+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:34.235903+0000) 2022-04-04T00:29:34.271 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:34.269+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:34.752 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:34.749+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:35.236 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:35.234+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:35.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:35.235+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:35.236076+0000) 2022-04-04T00:29:35.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:35.799+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:36.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:36.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:36.236299+0000) 2022-04-04T00:29:36.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:36.284+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:36.819 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:36.817+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:37.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:37.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:37.236445+0000) 2022-04-04T00:29:37.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:37.321+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:37.858 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:37.856+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:38.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:38.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:38.236621+0000) 2022-04-04T00:29:38.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:38.307+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:38.860 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:38.858+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:39.238 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:39.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:39.236760+0000) 2022-04-04T00:29:39.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:39.314+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:39.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:39.887+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:40.238 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:40.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:40.236997+0000) 2022-04-04T00:29:40.354 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:40.352+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:40.604 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:29:40.601+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:29:40.908 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:40.906+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:41.238 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:41.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:41.237217+0000) 2022-04-04T00:29:41.322 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:41.320+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:41.870 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:41.868+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:42.238 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:42.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:42.237443+0000) 2022-04-04T00:29:42.274 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:42.271+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:42.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:42.870+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:43.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:43.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:43.237671+0000) 2022-04-04T00:29:43.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:43.307+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:43.886 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:43.883+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:44.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:44.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:44.237894+0000) 2022-04-04T00:29:44.343 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:44.341+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:44.925 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:44.922+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:45.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:45.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:45.238024+0000) 2022-04-04T00:29:45.301 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:45.299+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:45.890 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:45.887+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:46.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:46.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:46.238185+0000) 2022-04-04T00:29:46.294 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:46.291+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:46.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:46.875+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:47.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:47.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:47.238353+0000) 2022-04-04T00:29:47.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:47.257+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:47.841 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:47.839+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:48.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:48.216+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:48.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:48.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:48.238558+0000) 2022-04-04T00:29:48.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:48.840+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:49.240 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:49.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:49.238724+0000) 2022-04-04T00:29:49.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:49.248+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:49.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:49.861+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:50.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:50.199+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:50.240 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:50.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:50.238900+0000) 2022-04-04T00:29:50.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:50.827+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:51.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:51.185+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:51.240 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:51.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:51.239107+0000) 2022-04-04T00:29:51.825 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:51.822+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:52.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:52.161+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:52.240 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:52.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:52.239330+0000) 2022-04-04T00:29:52.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:52.819+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:53.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:53.129+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:53.240 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:53.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:53.239544+0000) 2022-04-04T00:29:53.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:53.779+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:54.102 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:54.100+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:54.240 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:54.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:54.239713+0000) 2022-04-04T00:29:54.766 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:54.764+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:55.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:55.052+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:55.241 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:55.239+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:55.239918+0000) 2022-04-04T00:29:55.619 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:29:55.617+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:29:55.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:55.727+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:56.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:56.036+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:56.241 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:56.239+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:56.240121+0000) 2022-04-04T00:29:56.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:56.723+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:57.042 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:57.039+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:57.241 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:57.239+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:57.240349+0000) 2022-04-04T00:29:57.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:57.745+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:58.021 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:58.018+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:58.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:58.239+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:58.240567+0000) 2022-04-04T00:29:58.798 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:58.795+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:58.972 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:58.970+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:29:59.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:29:59.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:29:59.240827+0000) 2022-04-04T00:29:59.798 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:29:59.796+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:29:59.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:29:59.993+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:00.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:00.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:00.241008+0000) 2022-04-04T00:30:00.759 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:00.757+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:00.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:00.951+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:01.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:01.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:01.241162+0000) 2022-04-04T00:30:01.724 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:01.722+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:01.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:01.948+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:02.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:02.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:02.241333+0000) 2022-04-04T00:30:02.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:02.722+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:02.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:02.950+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:03.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:03.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:03.241457+0000) 2022-04-04T00:30:03.727 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:03.726+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:03.996 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:03.994+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:04.124 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:30:04.127 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:30:04.243 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:04.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:04.241692+0000) 2022-04-04T00:30:04.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:04.716+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:05.029 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:05.027+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:05.243 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:05.241+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:05.241831+0000) 2022-04-04T00:30:05.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:05.677+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:06.050 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:06.048+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:06.243 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:06.241+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:06.242062+0000) 2022-04-04T00:30:06.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:06.661+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:07.044 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:07.043+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:07.243 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:07.241+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:07.242271+0000) 2022-04-04T00:30:07.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:07.649+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:08.005 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:08.004+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:08.243 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:08.241+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:08.242513+0000) 2022-04-04T00:30:08.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:08.627+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:08.960 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:08.959+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:09.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:09.242+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:09.242695+0000) 2022-04-04T00:30:09.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:09.595+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:09.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:09.910+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:10.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:10.242+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:10.242921+0000) 2022-04-04T00:30:10.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:10.585+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:10.621 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:30:10.619+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:30:10.907 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:10.905+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:11.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:11.242+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:11.243159+0000) 2022-04-04T00:30:11.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:11.561+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:11.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:11.880+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:12.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:12.242+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:12.243440+0000) 2022-04-04T00:30:12.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:12.597+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:12.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:12.916+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:13.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:13.243+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:13.243676+0000) 2022-04-04T00:30:13.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:13.611+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:13.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:13.920+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:14.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:14.243+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:14.243930+0000) 2022-04-04T00:30:14.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:14.641+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:14.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:14.946+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:15.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:15.243+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:15.244130+0000) 2022-04-04T00:30:15.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:15.678+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:15.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:15.922+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:16.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:16.243+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:16.244327+0000) 2022-04-04T00:30:16.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:16.712+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:16.880 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:16.876+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:17.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:17.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:17.244571+0000) 2022-04-04T00:30:17.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:17.662+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:17.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:17.877+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:18.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:18.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:18.244831+0000) 2022-04-04T00:30:18.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:18.693+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:18.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:18.860+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:19.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:19.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:19.245028+0000) 2022-04-04T00:30:19.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:19.740+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:19.859 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:19.857+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:20.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:20.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:20.245263+0000) 2022-04-04T00:30:20.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:20.726+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:20.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:20.827+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:21.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:21.245+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:21.245508+0000) 2022-04-04T00:30:21.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:21.768+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:21.814 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:21.812+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:22.247 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:22.245+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:22.245784+0000) 2022-04-04T00:30:22.776 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:22.774+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:22.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:22.781+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:23.247 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:23.245+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:23.245981+0000) 2022-04-04T00:30:23.746 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:23.743+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:23.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:23.782+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:24.247 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:24.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:24.246197+0000) 2022-04-04T00:30:24.752 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:24.749+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:24.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:24.804+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:25.247 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:25.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:25.246442+0000) 2022-04-04T00:30:25.632 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:30:25.630+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:30:25.796 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:25.794+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:25.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:25.842+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:26.247 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:26.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:26.246686+0000) 2022-04-04T00:30:26.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:26.835+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:26.845 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:26.842+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:27.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:27.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:27.246931+0000) 2022-04-04T00:30:27.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:27.861+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:27.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:27.883+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:28.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:28.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:28.247173+0000) 2022-04-04T00:30:28.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:28.831+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:28.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:28.902+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:29.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:29.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:29.247406+0000) 2022-04-04T00:30:29.867 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:29.865+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:29.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:29.895+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:30.249 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:30.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:30.247615+0000) 2022-04-04T00:30:30.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:30.826+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:30.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:30.929+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:31.249 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:31.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:31.247874+0000) 2022-04-04T00:30:31.836 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:31.833+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:31.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:31.948+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:32.249 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:32.247+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:32.248141+0000) 2022-04-04T00:30:32.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:32.861+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:32.959 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:32.957+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:33.249 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:33.247+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:33.248335+0000) 2022-04-04T00:30:33.856 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:33.853+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:33.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:33.920+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:34.249 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:34.247+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:34.248501+0000) 2022-04-04T00:30:34.273 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:30:34.276 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:30:34.819 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:34.816+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:34.936 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:34.933+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:35.250 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:35.247+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:35.248719+0000) 2022-04-04T00:30:35.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:35.777+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:35.954 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:35.951+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:36.250 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:36.248+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:36.248867+0000) 2022-04-04T00:30:36.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:36.792+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:36.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:36.977+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:37.250 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:37.248+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:37.249071+0000) 2022-04-04T00:30:37.762 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:37.759+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:37.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:37.929+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:38.250 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:38.248+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:38.249269+0000) 2022-04-04T00:30:38.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:38.756+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:38.932 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:38.930+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:39.251 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:39.248+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:39.249513+0000) 2022-04-04T00:30:39.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:39.768+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:39.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:39.938+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:40.251 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:40.248+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:40.249718+0000) 2022-04-04T00:30:40.647 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:30:40.645+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:30:40.809 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:40.806+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:40.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:40.949+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:41.251 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:41.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:41.249884+0000) 2022-04-04T00:30:41.854 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:41.852+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:41.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:41.956+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:42.251 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:42.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:42.250106+0000) 2022-04-04T00:30:42.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:42.813+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:42.943 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:42.941+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:43.251 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:43.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:43.250348+0000) 2022-04-04T00:30:43.850 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:43.846+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:43.968 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:43.966+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:44.252 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:44.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:44.250612+0000) 2022-04-04T00:30:44.895 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:44.893+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:44.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:44.938+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:45.252 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:45.250+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:45.250902+0000) 2022-04-04T00:30:45.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:45.912+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:45.975 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:45.972+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:46.252 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:46.250+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:46.251109+0000) 2022-04-04T00:30:46.932 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:46.929+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:46.963 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:46.961+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:47.252 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:47.250+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:47.251339+0000) 2022-04-04T00:30:47.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:47.946+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:47.994 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:47.992+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:48.252 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:48.250+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:48.251577+0000) 2022-04-04T00:30:48.990 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:48.987+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:48.994 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:48.992+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:49.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:49.251+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:49.251793+0000) 2022-04-04T00:30:50.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:50.023+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:50.026 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:50.023+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:50.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:50.251+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:50.251982+0000) 2022-04-04T00:30:50.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:50.990+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:51.036 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:51.034+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:51.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:51.251+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:51.252194+0000) 2022-04-04T00:30:51.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:51.977+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:52.051 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:52.048+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:52.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:52.251+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:52.252386+0000) 2022-04-04T00:30:53.029 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:53.027+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:53.068 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:53.066+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:53.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:53.251+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:53.252620+0000) 2022-04-04T00:30:53.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:53.981+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:54.054 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:54.052+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:54.254 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:54.252+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:54.252809+0000) 2022-04-04T00:30:54.944 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:54.943+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:55.048 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:55.046+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:55.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:55.252+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:55.253021+0000) 2022-04-04T00:30:55.663 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:30:55.661+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:30:55.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:55.975+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:56.046 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:56.044+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:56.254 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:56.252+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:56.253244+0000) 2022-04-04T00:30:57.005 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:57.003+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:57.049 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:57.047+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:57.254 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:57.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:57.253436+0000) 2022-04-04T00:30:57.956 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:57.954+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:58.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:58.084+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:58.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:58.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:58.253686+0000) 2022-04-04T00:30:58.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:58.980+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:30:59.043 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:30:59.041+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:30:59.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:30:59.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:30:59.253906+0000) 2022-04-04T00:30:59.944 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:30:59.942+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:00.074 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:00.072+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:00.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:00.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:00.254074+0000) 2022-04-04T00:31:00.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:00.980+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:01.122 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:01.120+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:01.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:01.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:01.254284+0000) 2022-04-04T00:31:01.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:01.964+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:02.101 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:02.099+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:02.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:02.254+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:02.254533+0000) 2022-04-04T00:31:02.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:02.983+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:03.124 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:03.122+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:03.256 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:03.254+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:03.254733+0000) 2022-04-04T00:31:03.975 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:03.973+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:04.104 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:04.103+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:04.256 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:04.254+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:04.254898+0000) 2022-04-04T00:31:04.403 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:31:04.407 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:31:04.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:04.945+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:05.059 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:05.057+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:05.256 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:05.254+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:05.255071+0000) 2022-04-04T00:31:05.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:05.972+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:06.054 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:06.053+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:06.256 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:06.254+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:06.255277+0000) 2022-04-04T00:31:06.994 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:06.992+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:07.075 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:07.073+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:07.256 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:07.255+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:07.255553+0000) 2022-04-04T00:31:08.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:08.014+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:08.028 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:08.027+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:08.257 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:08.255+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:08.255844+0000) 2022-04-04T00:31:09.022 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:09.020+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:09.056 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:09.055+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:09.257 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:09.255+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:09.256119+0000) 2022-04-04T00:31:10.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:10.014+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:10.089 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:10.088+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:10.257 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:10.256+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:10.256384+0000) 2022-04-04T00:31:10.679 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:31:10.677+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:31:10.986 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:10.985+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:11.078 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:11.076+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:11.258 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:11.256+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:11.256631+0000) 2022-04-04T00:31:11.993 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:11.990+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:12.031 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:12.029+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:12.258 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:12.255+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:12.256883+0000) 2022-04-04T00:31:13.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:13.035+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:13.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:13.060+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:13.258 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:13.256+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:13.257126+0000) 2022-04-04T00:31:14.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:14.070+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:14.087 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:14.085+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:14.258 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:14.256+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:14.257371+0000) 2022-04-04T00:31:15.046 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:15.043+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:15.114 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:15.112+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:15.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:15.256+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:15.257616+0000) 2022-04-04T00:31:16.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:16.064+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:16.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:16.141+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:16.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:16.256+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:16.257884+0000) 2022-04-04T00:31:17.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:17.035+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:17.190 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:17.188+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:17.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:17.257+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:17.258142+0000) 2022-04-04T00:31:17.989 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:17.986+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:18.205 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:18.203+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:18.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:18.257+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:18.258386+0000) 2022-04-04T00:31:19.022 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:19.019+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:19.240 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:19.237+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:19.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:19.257+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:19.258638+0000) 2022-04-04T00:31:20.054 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:20.051+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:20.224 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:20.221+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:20.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:20.257+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:20.258812+0000) 2022-04-04T00:31:21.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:21.053+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:21.214 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:21.212+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:21.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:21.258+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:21.258982+0000) 2022-04-04T00:31:22.020 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:22.017+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:22.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:22.165+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:22.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:22.258+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:22.259124+0000) 2022-04-04T00:31:22.981 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:22.978+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:23.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:23.151+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:23.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:23.258+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:23.259328+0000) 2022-04-04T00:31:23.959 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:23.956+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:24.130 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:24.128+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:24.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:24.258+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:24.259566+0000) 2022-04-04T00:31:24.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:24.992+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:25.180 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:25.177+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:25.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:25.259+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:25.259825+0000) 2022-04-04T00:31:25.695 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:31:25.693+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:31:25.954 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:25.951+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:26.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:26.208+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:26.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:26.259+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:26.259996+0000) 2022-04-04T00:31:26.973 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:26.970+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:27.199 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:27.196+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:27.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:27.259+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:27.260203+0000) 2022-04-04T00:31:28.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:28.010+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:28.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:28.227+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:28.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:28.259+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:28.260451+0000) 2022-04-04T00:31:29.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:29.036+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:29.262 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:29.259+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:29.260674+0000) 2022-04-04T00:31:29.269 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:29.267+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:30.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:30.002+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:30.238 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:30.236+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:30.262 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:30.260+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:30.260878+0000) 2022-04-04T00:31:31.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:31.000+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:31.196 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:31.193+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:31.262 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:31.260+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:31.261078+0000) 2022-04-04T00:31:31.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:31.977+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:32.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:32.223+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:32.262 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:32.260+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:32.261239+0000) 2022-04-04T00:31:33.027 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:33.024+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:33.253 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:33.251+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:33.262 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:33.260+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:33.261460+0000) 2022-04-04T00:31:34.045 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:34.042+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:34.263 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:34.261+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:34.261690+0000) 2022-04-04T00:31:34.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:34.282+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:34.543 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:31:34.547 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:31:35.063 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:35.060+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:35.263 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:35.261+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:35.261890+0000) 2022-04-04T00:31:35.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:35.300+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:36.097 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:36.094+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:36.263 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:36.261+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:36.262127+0000) 2022-04-04T00:31:36.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:36.293+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:37.064 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:37.062+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:37.263 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:37.261+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:37.262335+0000) 2022-04-04T00:31:37.344 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:37.341+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:38.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:38.109+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:38.263 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:38.262+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:38.262542+0000) 2022-04-04T00:31:38.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:38.318+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:39.087 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:39.085+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:39.264 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:39.262+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:39.262729+0000) 2022-04-04T00:31:39.275 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:39.273+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:40.078 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:40.075+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:40.264 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:40.262+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:40.262935+0000) 2022-04-04T00:31:40.271 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:40.269+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:40.711 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:31:40.709+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:31:41.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:41.083+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:41.264 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:41.262+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:41.263177+0000) 2022-04-04T00:31:41.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:41.275+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:42.044 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:42.041+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:42.264 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:42.262+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:42.263439+0000) 2022-04-04T00:31:42.283 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:42.280+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:43.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:43.065+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:43.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:43.263+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:43.263674+0000) 2022-04-04T00:31:43.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:43.270+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:44.099 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:44.097+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:44.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:44.257+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:44.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:44.263+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:44.263858+0000) 2022-04-04T00:31:45.120 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:45.117+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:45.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:45.263+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:45.264059+0000) 2022-04-04T00:31:45.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:45.284+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:46.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:46.156+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:46.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:46.263+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:46.264269+0000) 2022-04-04T00:31:46.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:46.274+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:47.158 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:47.156+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:47.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:47.264+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:47.264511+0000) 2022-04-04T00:31:47.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:47.295+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:48.119 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:48.117+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:48.267 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:48.263+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:48.264747+0000) 2022-04-04T00:31:48.322 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:48.319+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:49.131 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:49.129+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:49.266 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:49.264+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:49.265009+0000) 2022-04-04T00:31:49.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:49.319+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:50.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:50.153+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:50.266 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:50.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:50.265221+0000) 2022-04-04T00:31:50.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:50.307+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:51.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:51.177+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:51.267 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:51.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:51.265464+0000) 2022-04-04T00:31:51.289 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:51.287+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:52.134 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:52.132+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:52.267 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:52.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:52.265672+0000) 2022-04-04T00:31:52.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:52.271+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:53.174 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:53.172+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:53.267 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:53.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:53.265884+0000) 2022-04-04T00:31:53.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:53.275+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:54.214 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:54.212+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:54.255 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:54.253+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:54.267 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:54.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:54.266076+0000) 2022-04-04T00:31:55.171 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:55.169+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:55.267 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:55.266+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:55.266289+0000) 2022-04-04T00:31:55.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:55.271+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:55.727 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:31:55.725+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:31:56.134 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:56.132+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:56.224 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:56.223+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:56.267 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:56.266+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:56.266526+0000) 2022-04-04T00:31:57.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:57.146+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:57.267 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:57.265+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:57.268 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:57.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:57.266763+0000) 2022-04-04T00:31:58.139 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:58.137+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:58.268 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:58.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:58.266981+0000) 2022-04-04T00:31:58.271 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:58.270+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:31:59.094 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:31:59.093+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:31:59.268 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:31:59.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:31:59.267097+0000) 2022-04-04T00:31:59.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:31:59.294+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:00.078 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:00.076+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:00.268 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:00.266+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:00.267307+0000) 2022-04-04T00:32:00.274 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:00.273+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:01.121 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:01.119+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:01.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:01.231+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:01.268 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:01.266+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:01.267543+0000) 2022-04-04T00:32:02.110 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:02.108+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:02.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:02.249+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:02.269 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:02.266+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:02.267826+0000) 2022-04-04T00:32:03.119 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:03.117+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:03.269 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:03.267+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:03.270 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:03.267+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:03.268039+0000) 2022-04-04T00:32:04.103 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:04.101+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:04.269 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:04.267+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:04.268254+0000) 2022-04-04T00:32:04.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:04.308+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:04.615 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:32:04.618 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:32:05.084 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:05.083+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:05.269 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:05.267+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:05.268520+0000) 2022-04-04T00:32:05.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:05.349+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:06.051 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:06.049+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:06.270 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:06.267+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:06.268762+0000) 2022-04-04T00:32:06.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:06.387+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:07.009 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:07.006+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:07.270 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:07.268+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:07.269033+0000) 2022-04-04T00:32:07.355 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:07.352+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:08.014 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:08.011+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:08.270 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:08.268+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:08.269273+0000) 2022-04-04T00:32:08.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:08.396+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:09.040 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:09.037+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:09.270 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:09.268+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:09.269504+0000) 2022-04-04T00:32:09.448 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:09.446+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:10.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:10.052+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:10.271 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:10.268+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:10.269720+0000) 2022-04-04T00:32:10.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:10.483+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:10.742 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:32:10.740+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:32:11.082 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:11.079+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:11.271 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:11.269+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:11.269961+0000) 2022-04-04T00:32:11.480 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:11.477+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:12.037 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:12.035+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:12.271 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:12.269+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:12.270218+0000) 2022-04-04T00:32:12.468 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:12.465+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:13.074 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:13.072+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:13.271 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:13.269+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:13.270409+0000) 2022-04-04T00:32:13.466 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:13.464+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:14.072 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:14.069+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:14.271 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:14.269+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:14.270575+0000) 2022-04-04T00:32:14.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:14.464+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:15.022 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:15.019+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:15.272 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:15.270+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:15.270816+0000) 2022-04-04T00:32:15.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:15.426+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:16.049 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:16.046+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:16.272 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:16.270+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:16.271058+0000) 2022-04-04T00:32:16.408 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:16.405+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:17.058 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:17.055+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:17.272 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:17.270+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:17.271299+0000) 2022-04-04T00:32:17.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:17.364+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:18.040 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:18.037+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:18.272 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:18.270+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:18.271493+0000) 2022-04-04T00:32:18.329 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:18.326+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:19.027 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:19.024+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:19.272 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:19.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:19.271665+0000) 2022-04-04T00:32:19.354 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:19.352+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:19.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:19.992+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:20.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:20.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:20.271866+0000) 2022-04-04T00:32:20.364 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:20.362+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:21.031 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:21.028+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:21.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:21.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:21.272079+0000) 2022-04-04T00:32:21.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:21.367+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:22.068 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:22.065+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:22.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:22.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:22.272334+0000) 2022-04-04T00:32:22.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:22.359+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:23.031 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:23.028+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:23.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:23.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:23.272597+0000) 2022-04-04T00:32:23.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:23.383+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:24.060 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:24.057+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:24.274 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:24.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:24.272882+0000) 2022-04-04T00:32:24.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:24.389+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:25.033 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:25.031+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:25.274 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:25.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:25.273127+0000) 2022-04-04T00:32:25.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:25.371+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:32:25.758 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:32:25.756+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:32:25.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:25.993+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:26.274 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:26.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:26.273372+0000) 2022-04-04T00:32:26.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:26.405+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:27.031 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:27.029+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:27.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:27.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:27.273634+0000) 2022-04-04T00:32:27.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:27.391+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:28.060 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:28.057+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:28.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:28.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:28.273873+0000) 2022-04-04T00:32:28.347 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:28.344+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:29.068 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:29.066+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:29.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:29.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:29.274049+0000) 2022-04-04T00:32:29.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:29.299+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:30.099 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:30.096+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:30.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:30.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:30.274264+0000) 2022-04-04T00:32:30.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:30.277+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:31.101 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:31.099+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:31.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:31.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:31.274529+0000) 2022-04-04T00:32:31.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:31.277+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:32.057 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:32.054+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:32.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:32.274+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:32.274804+0000) 2022-04-04T00:32:32.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:32.281+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:33.020 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:33.017+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:33.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:33.274+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:33.275053+0000) 2022-04-04T00:32:33.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:33.274+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:34.064 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:34.062+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:34.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:34.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:34.275299+0000) 2022-04-04T00:32:34.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:34.313+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:34.764 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:32:34.768 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:32:35.032 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:35.030+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:35.277 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:35.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:35.275510+0000) 2022-04-04T00:32:35.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:35.291+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:36.069 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:36.066+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:36.252 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:36.250+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:36.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:36.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:36.275694+0000) 2022-04-04T00:32:37.045 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:37.042+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:37.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:37.252+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:37.277 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:37.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:37.275866+0000) 2022-04-04T00:32:38.037 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:38.034+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:38.277 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:38.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:38.276085+0000) 2022-04-04T00:32:38.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:38.283+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:38.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:38.989+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:39.242 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:39.239+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:39.277 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:39.276+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:39.276335+0000) 2022-04-04T00:32:40.018 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:40.013+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:40.278 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:40.276+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:40.276584+0000) 2022-04-04T00:32:40.287 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:40.285+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:40.774 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:32:40.772+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:32:41.023 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:41.021+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:41.278 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:41.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:41.276858+0000) 2022-04-04T00:32:41.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:41.333+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:41.994 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:41.991+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:42.278 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:42.276+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:42.277065+0000) 2022-04-04T00:32:42.334 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:42.332+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:42.954 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:42.952+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:43.278 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:43.276+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:43.277286+0000) 2022-04-04T00:32:43.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:43.333+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:43.991 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:43.989+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:44.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:44.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:44.277538+0000) 2022-04-04T00:32:44.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:44.307+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:44.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:44.990+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:45.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:45.276+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:45.277759+0000) 2022-04-04T00:32:45.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:45.355+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:46.024 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:46.022+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:46.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:46.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:46.277923+0000) 2022-04-04T00:32:46.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:46.352+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:47.014 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:47.013+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:47.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:47.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:47.278147+0000) 2022-04-04T00:32:47.343 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:47.341+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:48.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:48.002+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:48.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:48.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:48.278368+0000) 2022-04-04T00:32:48.306 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:48.304+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:49.039 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:49.037+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:49.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:49.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:49.278578+0000) 2022-04-04T00:32:49.307 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:49.305+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:49.999 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:49.997+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:50.275 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:50.274+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:50.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:50.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:50.278804+0000) 2022-04-04T00:32:51.015 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:51.014+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:51.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:51.258+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:51.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:51.278+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:51.278998+0000) 2022-04-04T00:32:52.000 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:51.998+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:52.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:52.242+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:52.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:52.278+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:52.279237+0000) 2022-04-04T00:32:53.023 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:53.021+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:53.238 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:53.236+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:53.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:53.278+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:53.279437+0000) 2022-04-04T00:32:54.051 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:54.050+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:54.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:54.244+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:54.281 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:54.278+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:54.279680+0000) 2022-04-04T00:32:55.040 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:55.038+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:55.224 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:55.223+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:55.281 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:55.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:55.279909+0000) 2022-04-04T00:32:55.790 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:32:55.787+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:32:56.083 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:56.081+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:56.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:56.231+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:56.281 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:56.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:56.280144+0000) 2022-04-04T00:32:57.054 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:57.052+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:57.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:57.233+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:57.281 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:57.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:57.280353+0000) 2022-04-04T00:32:58.030 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:58.028+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:58.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:58.271+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:58.281 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:58.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:58.280557+0000) 2022-04-04T00:32:58.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:58.990+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:32:59.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:32:59.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:32:59.280765+0000) 2022-04-04T00:32:59.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:32:59.298+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:32:59.956 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:32:59.954+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:00.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:00.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:00.281024+0000) 2022-04-04T00:33:00.283 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:00.280+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:00.923 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:00.920+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:01.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:01.244+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:01.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:01.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:01.281243+0000) 2022-04-04T00:33:01.967 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:01.964+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:02.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:02.225+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:02.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:02.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:02.281471+0000) 2022-04-04T00:33:02.999 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:02.997+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:03.234 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:03.232+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:03.283 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:03.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:03.281674+0000) 2022-04-04T00:33:03.976 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:03.973+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:04.266 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:04.263+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:04.283 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:04.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:04.281893+0000) 2022-04-04T00:33:04.912 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:33:04.916 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:33:04.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:04.989+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:05.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:05.276+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:05.283 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:05.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:05.282157+0000) 2022-04-04T00:33:05.957 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:05.954+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:06.242 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:06.239+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:06.283 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:06.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:06.282338+0000) 2022-04-04T00:33:06.924 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:06.921+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:07.263 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:07.260+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:07.283 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:07.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:07.282478+0000) 2022-04-04T00:33:07.908 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:07.905+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:08.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:08.282+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:08.282604+0000) 2022-04-04T00:33:08.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:08.299+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:08.895 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:08.893+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:09.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:09.282+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:09.282764+0000) 2022-04-04T00:33:09.325 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:09.322+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:09.861 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:09.858+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:10.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:10.282+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:10.282945+0000) 2022-04-04T00:33:10.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:10.283+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:10.805 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:33:10.803+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:33:10.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:10.827+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:11.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:11.282+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:11.283104+0000) 2022-04-04T00:33:11.318 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:11.316+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:11.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:11.830+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:12.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:12.282+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:12.283288+0000) 2022-04-04T00:33:12.334 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:12.331+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:12.788 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:12.785+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:13.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:13.283+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:13.283472+0000) 2022-04-04T00:33:13.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:13.339+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:13.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:13.740+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:14.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:14.283+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:14.283695+0000) 2022-04-04T00:33:14.354 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:14.351+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:14.792 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:14.790+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:15.285 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:15.283+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:15.283871+0000) 2022-04-04T00:33:15.397 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:15.395+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:15.754 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:15.751+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:16.285 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:16.283+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:16.284071+0000) 2022-04-04T00:33:16.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:16.367+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:16.710 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:16.707+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:17.285 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:17.283+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:17.284263+0000) 2022-04-04T00:33:17.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:17.379+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:17.745 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:17.743+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:18.285 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:18.284+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:18.284469+0000) 2022-04-04T00:33:18.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:18.416+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:18.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:18.769+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:19.286 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:19.284+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:19.284687+0000) 2022-04-04T00:33:19.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:19.402+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:19.765 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:19.763+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:20.286 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:20.284+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:20.284924+0000) 2022-04-04T00:33:20.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:20.377+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:20.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:20.725+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:21.286 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:21.284+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:21.285097+0000) 2022-04-04T00:33:21.400 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:21.397+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:21.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:21.681+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:22.286 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:22.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:22.285307+0000) 2022-04-04T00:33:22.372 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:22.369+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:22.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:22.678+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:23.286 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:23.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:23.285495+0000) 2022-04-04T00:33:23.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:23.365+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:23.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:23.629+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:24.287 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:24.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:24.285702+0000) 2022-04-04T00:33:24.343 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:24.340+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:24.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:24.586+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:25.287 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:25.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:25.285910+0000) 2022-04-04T00:33:25.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:25.383+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:25.607 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:25.605+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:25.816 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:33:25.813+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:33:26.287 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:26.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:26.286130+0000) 2022-04-04T00:33:26.398 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:26.396+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:26.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:26.568+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:27.287 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:27.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:27.286312+0000) 2022-04-04T00:33:27.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:27.388+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:27.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:27.566+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:28.287 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:28.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:28.286493+0000) 2022-04-04T00:33:28.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:28.396+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:28.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:28.534+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:29.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:29.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:29.286670+0000) 2022-04-04T00:33:29.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:29.348+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:29.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:29.514+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:30.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:30.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:30.286907+0000) 2022-04-04T00:33:30.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:30.367+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:30.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:30.557+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:31.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:31.286+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:31.287139+0000) 2022-04-04T00:33:31.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:31.350+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:31.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:31.557+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:32.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:32.286+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:32.287355+0000) 2022-04-04T00:33:32.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:32.304+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:32.586 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:32.584+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:33.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:33.281+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:33.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:33.286+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:33.287528+0000) 2022-04-04T00:33:33.558 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:33.555+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:34.289 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:34.286+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:34.287731+0000) 2022-04-04T00:33:34.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:34.293+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:34.570 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:34.568+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:35.031 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:33:35.035 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:33:35.289 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:35.287+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:35.287971+0000) 2022-04-04T00:33:35.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:35.335+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:35.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:35.548+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:36.289 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:36.287+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:36.288191+0000) 2022-04-04T00:33:36.307 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:36.305+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:36.526 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:36.524+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:37.289 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:37.287+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:37.288392+0000) 2022-04-04T00:33:37.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:37.314+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:37.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:37.559+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:38.289 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:38.287+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:38.288550+0000) 2022-04-04T00:33:38.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:38.349+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:38.608 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:38.606+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:39.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:39.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:39.288758+0000) 2022-04-04T00:33:39.331 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:39.329+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:39.577 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:39.574+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:40.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:40.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:40.288959+0000) 2022-04-04T00:33:40.325 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:40.323+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:40.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:40.567+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:40.832 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:33:40.830+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:33:41.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:41.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:41.289146+0000) 2022-04-04T00:33:41.334 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:41.332+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:41.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:41.529+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:42.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:42.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:42.289339+0000) 2022-04-04T00:33:42.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:42.319+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:42.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:42.552+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:43.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:43.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:43.289524+0000) 2022-04-04T00:33:43.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:43.308+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:43.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:43.567+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:44.291 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:44.289+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:44.289726+0000) 2022-04-04T00:33:44.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:44.349+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:44.608 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:44.606+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:45.291 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:45.289+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:45.289919+0000) 2022-04-04T00:33:45.319 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:45.318+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:45.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:45.616+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:46.291 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:46.289+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:46.290103+0000) 2022-04-04T00:33:46.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:46.319+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:46.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:46.597+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:47.291 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:47.289+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:47.290286+0000) 2022-04-04T00:33:47.329 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:47.328+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:47.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:47.594+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:48.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:48.289+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:48.290497+0000) 2022-04-04T00:33:48.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:48.340+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:48.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:48.554+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:49.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:49.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:49.290702+0000) 2022-04-04T00:33:49.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:49.383+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:49.546 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:49.544+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:50.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:50.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:50.290849+0000) 2022-04-04T00:33:50.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:50.350+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:50.544 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:50.543+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:51.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:51.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:51.291033+0000) 2022-04-04T00:33:51.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:51.367+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:51.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:51.525+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:52.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:52.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:52.291266+0000) 2022-04-04T00:33:52.355 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:52.353+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:52.517 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:52.515+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:53.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:53.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:53.291410+0000) 2022-04-04T00:33:53.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:53.387+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:53.477 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:53.475+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:54.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:54.291+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:54.291610+0000) 2022-04-04T00:33:54.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:54.421+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:54.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:54.514+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:55.293 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:55.291+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:55.291853+0000) 2022-04-04T00:33:55.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:55.383+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:55.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:55.534+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:55.848 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:33:55.846+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:33:56.293 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:56.291+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:56.292016+0000) 2022-04-04T00:33:56.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:56.424+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:56.582 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:56.579+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:57.293 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:57.291+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:57.292103+0000) 2022-04-04T00:33:57.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:57.415+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:57.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:57.602+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:58.293 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:58.291+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:58.292279+0000) 2022-04-04T00:33:58.436 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:58.433+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:58.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:58.612+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:33:59.293 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:33:59.292+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:33:59.292453+0000) 2022-04-04T00:33:59.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:33:59.414+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:33:59.586 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:33:59.584+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:00.294 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:00.292+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:00.292661+0000) 2022-04-04T00:34:00.468 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:00.465+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:00.625 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:00.622+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:01.294 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:01.292+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:01.292875+0000) 2022-04-04T00:34:01.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:01.442+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:01.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:01.658+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:02.294 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:02.292+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:02.293004+0000) 2022-04-04T00:34:02.470 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:02.468+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:02.687 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:02.684+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:03.294 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:03.292+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:03.293213+0000) 2022-04-04T00:34:03.505 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:03.502+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:03.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:03.719+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:04.294 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:04.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:04.293414+0000) 2022-04-04T00:34:04.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:04.535+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:04.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:04.731+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:05.169 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:34:05.173 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:34:05.294 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:05.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:05.293593+0000) 2022-04-04T00:34:05.502 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:05.499+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:05.694 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:05.692+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:06.295 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:06.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:06.293735+0000) 2022-04-04T00:34:06.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:06.478+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:06.744 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:06.741+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:07.295 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:07.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:07.293823+0000) 2022-04-04T00:34:07.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:07.511+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:07.706 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:07.703+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:08.296 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:08.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:08.294013+0000) 2022-04-04T00:34:08.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:08.478+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:08.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:08.695+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:09.295 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:09.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:09.294259+0000) 2022-04-04T00:34:09.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:09.507+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:09.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:09.668+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:10.295 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:10.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:10.294458+0000) 2022-04-04T00:34:10.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:10.461+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:10.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:10.681+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:10.864 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:34:10.861+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:34:11.296 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:11.292+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:11.294675+0000) 2022-04-04T00:34:11.490 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:11.488+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:11.662 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:11.659+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:12.296 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:12.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:12.294929+0000) 2022-04-04T00:34:12.487 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:12.484+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:12.674 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:12.671+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:13.296 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:13.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:13.295146+0000) 2022-04-04T00:34:13.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:13.512+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:13.691 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:13.688+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:14.296 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:14.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:14.295334+0000) 2022-04-04T00:34:14.508 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:14.506+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:14.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:14.709+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:15.296 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:15.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:15.295530+0000) 2022-04-04T00:34:15.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:15.491+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:15.674 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:15.671+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:16.297 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:16.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:16.295748+0000) 2022-04-04T00:34:16.465 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:16.463+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:16.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:16.697+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:17.297 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:17.295+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:17.295992+0000) 2022-04-04T00:34:17.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:17.490+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:17.735 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:17.732+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:18.297 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:18.295+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:18.296206+0000) 2022-04-04T00:34:18.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:18.477+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:18.783 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:18.781+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:19.297 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:19.295+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:19.296428+0000) 2022-04-04T00:34:19.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:19.491+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:19.817 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:19.815+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:20.298 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:20.295+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:20.296667+0000) 2022-04-04T00:34:20.468 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:20.466+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:20.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:20.792+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:21.298 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:21.296+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:21.296862+0000) 2022-04-04T00:34:21.432 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:21.430+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:21.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:21.784+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:22.298 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:22.296+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:22.297097+0000) 2022-04-04T00:34:22.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:22.459+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:22.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:22.737+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:23.298 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:23.296+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:23.297303+0000) 2022-04-04T00:34:23.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:23.461+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:23.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:23.700+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:24.298 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:24.296+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:24.297505+0000) 2022-04-04T00:34:24.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:24.423+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:24.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:24.711+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:25.299 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:25.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:25.297699+0000) 2022-04-04T00:34:25.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:25.464+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:25.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:25.722+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:25.880 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:34:25.877+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:34:26.299 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:26.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:26.297873+0000) 2022-04-04T00:34:26.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:26.423+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:26.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:26.693+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:27.299 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:27.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:27.298096+0000) 2022-04-04T00:34:27.438 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:27.435+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:27.659 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:27.657+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:28.299 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:28.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:28.298317+0000) 2022-04-04T00:34:28.436 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:28.434+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:28.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:28.611+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:29.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:29.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:29.298538+0000) 2022-04-04T00:34:29.472 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:29.470+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:29.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:29.636+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:30.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:30.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:30.298731+0000) 2022-04-04T00:34:30.442 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:30.439+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:30.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:30.609+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:31.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:31.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:31.298929+0000) 2022-04-04T00:34:31.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:31.484+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:31.564 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:31.561+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:32.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:32.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:32.299169+0000) 2022-04-04T00:34:32.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:32.493+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:32.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:32.519+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:33.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:33.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:33.299384+0000) 2022-04-04T00:34:33.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:33.489+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:33.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:33.552+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:34.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:34.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:34.299589+0000) 2022-04-04T00:34:34.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:34.452+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:34.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:34.549+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:35.282 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:34:35.286 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:34:35.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:35.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:35.299721+0000) 2022-04-04T00:34:35.475 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:35.473+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:35.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:35.541+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:36.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:36.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:36.299882+0000) 2022-04-04T00:34:36.446 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:36.444+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:36.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:36.528+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:37.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:37.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:37.300043+0000) 2022-04-04T00:34:37.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:37.456+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:37.550 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:37.549+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:38.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:38.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:38.300228+0000) 2022-04-04T00:34:38.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:38.407+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:38.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:38.546+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:39.302 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:39.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:39.300477+0000) 2022-04-04T00:34:39.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:39.416+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:39.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:39.529+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:40.302 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:40.300+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:40.300686+0000) 2022-04-04T00:34:40.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:40.400+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:40.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:40.483+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:40.896 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:34:40.893+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:34:41.302 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:41.300+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:41.300872+0000) 2022-04-04T00:34:41.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:41.421+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:41.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:41.497+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:42.302 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:42.300+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:42.301060+0000) 2022-04-04T00:34:42.441 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:42.439+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:42.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:42.495+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:43.302 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:43.300+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:43.301286+0000) 2022-04-04T00:34:43.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:43.421+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:43.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:43.493+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:44.302 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:44.301+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:44.301486+0000) 2022-04-04T00:34:44.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:44.391+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:44.511 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:44.509+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:45.303 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:45.301+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:45.301713+0000) 2022-04-04T00:34:45.365 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:45.363+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:45.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:45.539+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:46.303 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:46.301+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:46.301936+0000) 2022-04-04T00:34:46.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:46.318+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:46.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:46.528+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:47.303 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:47.301+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:47.302099+0000) 2022-04-04T00:34:47.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:47.338+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:47.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:47.529+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:48.303 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:48.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:48.302283+0000) 2022-04-04T00:34:48.384 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:48.382+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:48.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:48.536+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:49.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:49.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:49.302493+0000) 2022-04-04T00:34:49.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:49.361+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:49.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:49.547+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:50.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:50.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:50.302721+0000) 2022-04-04T00:34:50.398 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:50.396+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:50.500 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:50.498+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:51.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:51.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:51.302859+0000) 2022-04-04T00:34:51.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:51.346+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:51.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:51.533+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:52.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:52.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:52.303063+0000) 2022-04-04T00:34:52.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:52.320+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:52.522 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:52.519+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:53.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:53.279+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:53.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:53.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:53.303255+0000) 2022-04-04T00:34:53.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:53.501+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:54.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:54.297+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:54.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:54.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:54.303404+0000) 2022-04-04T00:34:54.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:54.473+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:55.252 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:55.250+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:55.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:55.301+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:55.303636+0000) 2022-04-04T00:34:55.451 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:55.448+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:55.910 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:34:55.907+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:34:56.270 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:56.268+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:56.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:56.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:56.303891+0000) 2022-04-04T00:34:56.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:56.455+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:57.274 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:57.271+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:57.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:57.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:57.304030+0000) 2022-04-04T00:34:57.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:57.449+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:58.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:58.295+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:58.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:58.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:58.304197+0000) 2022-04-04T00:34:58.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:58.491+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:34:59.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:34:59.275+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:34:59.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:34:59.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:34:59.304363+0000) 2022-04-04T00:34:59.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:34:59.535+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:00.306 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:00.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:00.304564+0000) 2022-04-04T00:35:00.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:00.318+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:00.502 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:00.499+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:01.283 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:01.280+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:01.306 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:01.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:01.304807+0000) 2022-04-04T00:35:01.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:01.509+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:02.264 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:02.261+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:02.306 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:02.304+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:02.305066+0000) 2022-04-04T00:35:02.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:02.556+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:03.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:03.273+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:03.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:03.304+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:03.305292+0000) 2022-04-04T00:35:03.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:03.544+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:04.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:04.299+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:04.306 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:04.304+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:04.305505+0000) 2022-04-04T00:35:04.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:04.585+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:05.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:05.304+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:05.305699+0000) 2022-04-04T00:35:05.347 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:05.344+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:05.396 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:35:05.400 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:35:05.617 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:05.614+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:06.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:06.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:06.305889+0000) 2022-04-04T00:35:06.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:06.308+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:06.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:06.628+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:07.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:07.283+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:07.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:07.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:07.306056+0000) 2022-04-04T00:35:07.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:07.651+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:08.291 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:08.288+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:08.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:08.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:08.306252+0000) 2022-04-04T00:35:08.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:08.677+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:09.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:09.270+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:09.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:09.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:09.306414+0000) 2022-04-04T00:35:09.662 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:09.660+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:10.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:10.224+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:10.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:10.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:10.306599+0000) 2022-04-04T00:35:10.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:10.645+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:10.926 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:35:10.924+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:35:11.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:11.247+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:11.308 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:11.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:11.306820+0000) 2022-04-04T00:35:11.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:11.643+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:12.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:12.294+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:12.308 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:12.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:12.306996+0000) 2022-04-04T00:35:12.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:12.644+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:13.308 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:13.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:13.307151+0000) 2022-04-04T00:35:13.342 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:13.340+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:13.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:13.633+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:14.308 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:14.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:14.307342+0000) 2022-04-04T00:35:14.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:14.374+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:14.629 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:14.627+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:15.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:15.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:15.307572+0000) 2022-04-04T00:35:15.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:15.396+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:15.600 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:15.598+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:16.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:16.307+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:16.307789+0000) 2022-04-04T00:35:16.446 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:16.444+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:16.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:16.646+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:17.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:17.307+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:17.307990+0000) 2022-04-04T00:35:17.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:17.411+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:17.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:17.647+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:18.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:18.307+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:18.308150+0000) 2022-04-04T00:35:18.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:18.423+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:18.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:18.675+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:19.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:19.307+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:19.308347+0000) 2022-04-04T00:35:19.379 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:19.377+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:19.633 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:19.631+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:20.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:20.308+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:20.308562+0000) 2022-04-04T00:35:20.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:20.374+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:20.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:20.677+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:21.310 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:21.308+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:21.308834+0000) 2022-04-04T00:35:21.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:21.392+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:21.678 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:21.676+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:22.310 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:22.308+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:22.309094+0000) 2022-04-04T00:35:22.396 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:22.394+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:22.712 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:22.710+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:23.310 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:23.308+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:23.309296+0000) 2022-04-04T00:35:23.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:23.414+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:23.693 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:23.691+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:24.311 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:24.309+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:24.309494+0000) 2022-04-04T00:35:24.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:24.386+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:24.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:24.646+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:25.311 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:25.309+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:25.309716+0000) 2022-04-04T00:35:25.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:25.371+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:35:25.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:25.654+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:25.941 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:35:25.939+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:35:26.311 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:26.309+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:26.309882+0000) 2022-04-04T00:35:26.372 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:26.369+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:26.704 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:26.702+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:27.311 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:27.309+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:27.310051+0000) 2022-04-04T00:35:27.402 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:27.400+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:27.704 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:27.702+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:28.311 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:28.309+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:28.310245+0000) 2022-04-04T00:35:28.408 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:28.406+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:28.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:28.735+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:29.311 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:29.310+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:29.310466+0000) 2022-04-04T00:35:29.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:29.383+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:29.704 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:29.702+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:30.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:30.310+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:30.310686+0000) 2022-04-04T00:35:30.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:30.415+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:30.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:30.729+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:31.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:31.310+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:31.310911+0000) 2022-04-04T00:35:31.384 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:31.382+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:31.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:31.701+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:32.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:32.310+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:32.311164+0000) 2022-04-04T00:35:32.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:32.337+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:32.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:32.709+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:33.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:33.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:33.311390+0000) 2022-04-04T00:35:33.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:33.322+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:33.717 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:33.715+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:34.300 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:34.298+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:34.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:34.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:34.311567+0000) 2022-04-04T00:35:34.687 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:34.685+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:35.313 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:35.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:35.311838+0000) 2022-04-04T00:35:35.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:35.334+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:35.510 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:35:35.514 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:35:35.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:35.725+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:36.294 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:36.292+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:36.313 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:36.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:36.312079+0000) 2022-04-04T00:35:36.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:36.695+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:37.306 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:37.304+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:37.313 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:37.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:37.312274+0000) 2022-04-04T00:35:37.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:37.714+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:38.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:38.275+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:38.313 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:38.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:38.312421+0000) 2022-04-04T00:35:38.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:38.714+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:39.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:39.229+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:39.314 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:39.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:39.312582+0000) 2022-04-04T00:35:39.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:39.712+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:40.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:40.258+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:40.314 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:40.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:40.312761+0000) 2022-04-04T00:35:40.724 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:40.722+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:40.957 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:35:40.954+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:35:41.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:41.257+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:41.314 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:41.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:41.312972+0000) 2022-04-04T00:35:41.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:41.709+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:42.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:42.247+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:42.314 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:42.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:42.313220+0000) 2022-04-04T00:35:42.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:42.669+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:43.230 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:43.228+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:43.315 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:43.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:43.313449+0000) 2022-04-04T00:35:43.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:43.648+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:44.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:44.234+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:44.314 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:44.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:44.313680+0000) 2022-04-04T00:35:44.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:44.642+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:45.283 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:45.281+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:45.315 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:45.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:45.313905+0000) 2022-04-04T00:35:45.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:45.653+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:46.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:46.246+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:46.315 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:46.313+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:46.314100+0000) 2022-04-04T00:35:46.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:46.628+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:47.258 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:47.256+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:47.315 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:47.313+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:47.314296+0000) 2022-04-04T00:35:47.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:47.612+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:48.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:48.257+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:48.315 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:48.313+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:48.314491+0000) 2022-04-04T00:35:48.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:48.588+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:49.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:49.223+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:49.316 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:49.313+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:49.314707+0000) 2022-04-04T00:35:49.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:49.546+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:50.238 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:50.235+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:50.316 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:50.314+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:50.314884+0000) 2022-04-04T00:35:50.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:50.587+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:51.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:51.216+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:51.316 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:51.314+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:51.315074+0000) 2022-04-04T00:35:51.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:51.637+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:52.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:52.265+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:52.316 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:52.314+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:52.315316+0000) 2022-04-04T00:35:52.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:52.610+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:53.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:53.286+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:53.316 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:53.314+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:53.315481+0000) 2022-04-04T00:35:53.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:53.646+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:54.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:54.313+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:54.317 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:54.314+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:54.315677+0000) 2022-04-04T00:35:54.617 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:54.614+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:55.317 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:55.315+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:55.315889+0000) 2022-04-04T00:35:55.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:55.360+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:55.633 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:55.630+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:55.973 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:35:55.971+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:35:56.317 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:56.315+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:56.316113+0000) 2022-04-04T00:35:56.398 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:56.395+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:56.642 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:56.639+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:57.317 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:57.315+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:57.316378+0000) 2022-04-04T00:35:57.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:57.443+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:57.607 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:57.604+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:58.318 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:58.315+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:58.316603+0000) 2022-04-04T00:35:58.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:58.426+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:58.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:58.557+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:35:59.318 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:35:59.316+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:35:59.316838+0000) 2022-04-04T00:35:59.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:35:59.459+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:35:59.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:35:59.552+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:00.318 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:00.316+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:00.316945+0000) 2022-04-04T00:36:00.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:00.426+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:00.517 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:00.514+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:01.318 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:01.316+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:01.317137+0000) 2022-04-04T00:36:01.471 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:01.469+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:01.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:01.496+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:02.318 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:02.316+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:02.317395+0000) 2022-04-04T00:36:02.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:02.452+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:02.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:02.488+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:03.318 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:03.316+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:03.317606+0000) 2022-04-04T00:36:03.457 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:03.455+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:03.474 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:03.472+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:04.319 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:04.317+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:04.317839+0000) 2022-04-04T00:36:04.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:04.415+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:04.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:04.510+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:05.319 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:05.317+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:05.318044+0000) 2022-04-04T00:36:05.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:05.456+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:05.479 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:05.477+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:05.649 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:36:05.653 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:36:06.319 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:06.317+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:06.318225+0000) 2022-04-04T00:36:06.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:06.448+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:06.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:06.496+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:07.319 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:07.318+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:07.318453+0000) 2022-04-04T00:36:07.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:07.420+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:07.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:07.486+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:08.320 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:08.318+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:08.318654+0000) 2022-04-04T00:36:08.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:08.449+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:08.460 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:08.458+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:09.320 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:09.318+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:09.318859+0000) 2022-04-04T00:36:09.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:09.461+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:09.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:09.466+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:10.320 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:10.318+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:10.319054+0000) 2022-04-04T00:36:10.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:10.412+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:10.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:10.511+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:10.989 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:36:10.987+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:36:11.320 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:11.318+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:11.319272+0000) 2022-04-04T00:36:11.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:11.455+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:11.484 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:11.482+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:12.321 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:12.319+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:12.319482+0000) 2022-04-04T00:36:12.453 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:12.450+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:12.479 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:12.477+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:13.321 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:13.319+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:13.319624+0000) 2022-04-04T00:36:13.438 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:13.436+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:13.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:13.452+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:14.321 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:14.319+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:14.319839+0000) 2022-04-04T00:36:14.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:14.408+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:14.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:14.490+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:15.321 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:15.319+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:15.320059+0000) 2022-04-04T00:36:15.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:15.440+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:15.475 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:15.473+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:16.321 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:16.320+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:16.320279+0000) 2022-04-04T00:36:16.429 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:16.426+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:16.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:16.477+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:17.321 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:17.320+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:17.320481+0000) 2022-04-04T00:36:17.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:17.457+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:17.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:17.489+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:18.322 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:18.320+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:18.320739+0000) 2022-04-04T00:36:18.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:18.432+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:18.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:18.490+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:19.322 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:19.320+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:19.320915+0000) 2022-04-04T00:36:19.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:19.438+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:19.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:19.456+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:20.322 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:20.320+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:20.321113+0000) 2022-04-04T00:36:20.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:20.415+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:20.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:20.481+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:21.322 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:21.321+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:21.321304+0000) 2022-04-04T00:36:21.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:21.369+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:21.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:21.480+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:22.323 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:22.321+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:22.321505+0000) 2022-04-04T00:36:22.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:22.375+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:22.464 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:22.462+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:23.323 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:23.320+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:23.321729+0000) 2022-04-04T00:36:23.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:23.328+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:23.510 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:23.507+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:24.283 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:24.281+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:24.323 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:24.320+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:24.321981+0000) 2022-04-04T00:36:24.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:24.539+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:25.303 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:25.301+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:25.323 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:25.321+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:25.322197+0000) 2022-04-04T00:36:25.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:25.490+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:26.005 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:36:26.002+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:36:26.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:26.295+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:26.323 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:26.321+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:26.322357+0000) 2022-04-04T00:36:26.493 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:26.491+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:27.324 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:27.321+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:27.322495+0000) 2022-04-04T00:36:27.344 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:27.342+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:27.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:27.446+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:28.324 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:28.321+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:28.322630+0000) 2022-04-04T00:36:28.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:28.369+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:28.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:28.415+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:29.324 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:29.321+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:29.322848+0000) 2022-04-04T00:36:29.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:29.377+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:29.418 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:29.416+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:30.324 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:30.322+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:30.323061+0000) 2022-04-04T00:36:30.347 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:30.346+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:30.413 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:30.411+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:31.325 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:31.322+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:31.323318+0000) 2022-04-04T00:36:31.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:31.367+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:31.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:31.443+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:32.325 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:32.322+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:32.323594+0000) 2022-04-04T00:36:32.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:32.381+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:32.412 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:32.410+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:33.325 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:33.323+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:33.323874+0000) 2022-04-04T00:36:33.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:33.389+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:33.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:33.404+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:34.325 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:34.323+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:34.324135+0000) 2022-04-04T00:36:34.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:34.388+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:34.418 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:34.416+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:35.325 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:35.323+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:35.324355+0000) 2022-04-04T00:36:35.342 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:35.341+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:35.463 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:35.462+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:35.713 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:36:35.717 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:36:36.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:36.321+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:36.325 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:36.323+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:36.324551+0000) 2022-04-04T00:36:36.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:36.486+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:37.326 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:37.323+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:37.324727+0000) 2022-04-04T00:36:37.329 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:37.327+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:37.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:37.528+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:38.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:38.279+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:38.326 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:38.324+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:38.324915+0000) 2022-04-04T00:36:38.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:38.497+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:39.326 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:39.324+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:39.325133+0000) 2022-04-04T00:36:39.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:39.328+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:39.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:39.527+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:40.326 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:40.324+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:40.325371+0000) 2022-04-04T00:36:40.337 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:40.335+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:40.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:40.496+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:41.021 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:36:41.019+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:36:41.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:41.307+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:41.326 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:41.324+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:41.325580+0000) 2022-04-04T00:36:41.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:41.533+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:42.327 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:42.325+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:42.325842+0000) 2022-04-04T00:36:42.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:42.354+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:42.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:42.518+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:43.327 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:43.325+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:43.326069+0000) 2022-04-04T00:36:43.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:43.380+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:43.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:43.483+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:44.327 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:44.325+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:44.326217+0000) 2022-04-04T00:36:44.384 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:44.381+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:44.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:44.506+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:45.327 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:45.325+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:45.326439+0000) 2022-04-04T00:36:45.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:45.427+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:45.471 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:45.469+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:46.328 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:46.326+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:46.326662+0000) 2022-04-04T00:36:46.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:46.380+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:46.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:46.473+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:47.328 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:47.326+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:47.326833+0000) 2022-04-04T00:36:47.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:47.333+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:47.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:47.496+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:48.289 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:48.286+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:48.328 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:48.326+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:48.326962+0000) 2022-04-04T00:36:48.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:48.522+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:49.328 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:49.326+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:49.327129+0000) 2022-04-04T00:36:49.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:49.327+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:49.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:49.553+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:50.328 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:50.326+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:50.327282+0000) 2022-04-04T00:36:50.360 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:50.357+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:50.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:50.588+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:51.328 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:51.327+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:51.327498+0000) 2022-04-04T00:36:51.333 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:51.330+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:51.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:51.581+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:52.329 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:52.327+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:52.327747+0000) 2022-04-04T00:36:52.364 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:52.362+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:52.539 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:52.536+0000 7f88d0ac1700 -1 osd.4 136 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:52.885 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-04T00:36:52.883+0000 7ff9c44c8700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-04T00:36:52.900 INFO:tasks.ceph.mds.f.smithi186.stderr: -1> 2022-04-04T00:36:52.883+0000 7ff9c44c8700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-04T00:36:52.922 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-04T00:36:53.329 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:53.327+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:53.328008+0000) 2022-04-04T00:36:53.332 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:53.330+0000 7fd3b190b700 -1 osd.6 136 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:53.508 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:53.506+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:54.329 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:54.327+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:54.328235+0000) 2022-04-04T00:36:54.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:54.359+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:54.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:54.518+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:55.329 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:55.328+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:55.328455+0000) 2022-04-04T00:36:55.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:55.409+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:55.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:55.513+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:56.036 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:36:56.034+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:36:56.330 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:56.328+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:56.328697+0000) 2022-04-04T00:36:56.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:56.373+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:56.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:56.561+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:57.330 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:57.328+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:57.328900+0000) 2022-04-04T00:36:57.396 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:57.393+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:57.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:57.540+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:58.330 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:58.328+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:58.329120+0000) 2022-04-04T00:36:58.392 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:58.389+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:58.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:58.566+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:36:59.330 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:36:59.329+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:36:59.329319+0000) 2022-04-04T00:36:59.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:36:59.406+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:36:59.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:36:59.566+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:00.330 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:00.329+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:00.329462+0000) 2022-04-04T00:37:00.449 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:00.447+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:00.528 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:00.525+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:01.331 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:01.329+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:01.329644+0000) 2022-04-04T00:37:01.453 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:01.450+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:01.523 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:01.521+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:02.331 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:02.329+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:02.329883+0000) 2022-04-04T00:37:02.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:02.438+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:02.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:02.532+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:03.331 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:03.329+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:03.330135+0000) 2022-04-04T00:37:03.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:03.482+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:03.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:03.539+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:04.331 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:04.329+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:04.330354+0000) 2022-04-04T00:37:04.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:04.464+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:04.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:04.546+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:05.332 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:05.330+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:05.330610+0000) 2022-04-04T00:37:05.482 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:05.480+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:05.534 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:05.531+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:05.827 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:37:05.831 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:37:06.332 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:06.330+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:06.330904+0000) 2022-04-04T00:37:06.519 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:06.517+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:06.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:06.528+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:07.332 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:07.330+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:07.331155+0000) 2022-04-04T00:37:07.491 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:07.489+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:07.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:07.542+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:08.332 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:08.330+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:08.331381+0000) 2022-04-04T00:37:08.474 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:08.471+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:08.564 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:08.562+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:09.332 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:09.330+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:09.331552+0000) 2022-04-04T00:37:09.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:09.503+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:09.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:09.604+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:10.333 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:10.330+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:10.331763+0000) 2022-04-04T00:37:10.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:10.511+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:10.597 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:10.595+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:11.052 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:37:11.049+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:37:11.333 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:11.331+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:11.332022+0000) 2022-04-04T00:37:11.493 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:11.491+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:11.624 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:11.621+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:12.333 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:12.331+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:12.332249+0000) 2022-04-04T00:37:12.511 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:12.509+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:12.585 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:12.582+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:13.333 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:13.331+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:13.332479+0000) 2022-04-04T00:37:13.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:13.474+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:13.569 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:13.567+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:14.334 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:14.331+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:14.332645+0000) 2022-04-04T00:37:14.520 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:14.518+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:14.612 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:14.610+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:15.334 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:15.331+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:15.332876+0000) 2022-04-04T00:37:15.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:15.506+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:15.573 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:15.570+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:16.334 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:16.332+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:16.333118+0000) 2022-04-04T00:37:16.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:16.519+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:16.574 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:16.571+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:17.334 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:17.332+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:17.333369+0000) 2022-04-04T00:37:17.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:17.483+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:17.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:17.577+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:18.335 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:18.332+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:18.333607+0000) 2022-04-04T00:37:18.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:18.481+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:18.570 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:18.568+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:19.335 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:19.333+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:19.333819+0000) 2022-04-04T00:37:19.508 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:19.507+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:19.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:19.593+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:20.335 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:20.333+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:20.334042+0000) 2022-04-04T00:37:20.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:20.484+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:20.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:20.570+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:21.335 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:21.333+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:21.334252+0000) 2022-04-04T00:37:21.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:21.507+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:21.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:21.535+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:22.336 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:22.333+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:22.334515+0000) 2022-04-04T00:37:22.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:22.541+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:22.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:22.558+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:23.336 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:23.333+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:23.334735+0000) 2022-04-04T00:37:23.520 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:23.518+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:23.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:23.596+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:24.336 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:24.334+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:24.335000+0000) 2022-04-04T00:37:24.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:24.475+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:24.604 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:24.603+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:25.336 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:25.334+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:25.335243+0000) 2022-04-04T00:37:25.506 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:25.504+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:25.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:25.618+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:26.058 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:37:26.056+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:37:26.336 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:26.334+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:26.335432+0000) 2022-04-04T00:37:26.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:26.486+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:26.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:26.652+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:27.337 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:27.335+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:27.335656+0000) 2022-04-04T00:37:27.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:27.453+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:27.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:27.655+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:28.337 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:28.335+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:28.335887+0000) 2022-04-04T00:37:28.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:28.501+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:28.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:28.658+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:29.337 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:29.335+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:29.336048+0000) 2022-04-04T00:37:29.504 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:29.503+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:29.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:29.648+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:30.337 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:30.335+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:30.336299+0000) 2022-04-04T00:37:30.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:30.527+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:30.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:30.636+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:31.337 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:31.335+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:31.336522+0000) 2022-04-04T00:37:31.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:31.527+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:31.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:31.622+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:32.339 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:32.336+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:32.336766+0000) 2022-04-04T00:37:32.540 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:32.538+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:32.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:32.586+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:33.338 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:33.336+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:33.337042+0000) 2022-04-04T00:37:33.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:33.510+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:33.583 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:33.581+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:34.338 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:34.336+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:34.337282+0000) 2022-04-04T00:37:34.470 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:34.468+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:34.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:34.625+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:35.338 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:35.337+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:35.337486+0000) 2022-04-04T00:37:35.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:35.501+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:35.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:35.595+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:35.923 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:37:35.927 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:37:36.339 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:36.337+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:36.337749+0000) 2022-04-04T00:37:36.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:36.509+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:36.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:36.584+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:37.339 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:37.337+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:37.337998+0000) 2022-04-04T00:37:37.519 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:37.516+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:37.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:37.557+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:38.339 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:38.337+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:38.338182+0000) 2022-04-04T00:37:38.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:38.508+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:38.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:38.577+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:39.340 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:39.338+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:39.338415+0000) 2022-04-04T00:37:39.480 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:39.477+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:39.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:39.611+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:40.340 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:40.337+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:40.338683+0000) 2022-04-04T00:37:40.498 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:40.495+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:40.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:40.650+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:41.074 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:37:41.072+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:37:41.340 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:41.338+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:41.338873+0000) 2022-04-04T00:37:41.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:41.486+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:41.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:41.610+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:42.340 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:42.338+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:42.339098+0000) 2022-04-04T00:37:42.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:42.526+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:42.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:42.655+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:43.340 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:43.339+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:43.339297+0000) 2022-04-04T00:37:43.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:43.501+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:43.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:43.672+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:44.341 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:44.339+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:44.339515+0000) 2022-04-04T00:37:44.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:44.534+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:44.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:44.636+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:45.341 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:45.339+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:45.339801+0000) 2022-04-04T00:37:45.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:45.494+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:45.687 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:45.684+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:46.341 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:46.339+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:46.340011+0000) 2022-04-04T00:37:46.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:46.448+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:46.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:46.735+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:47.341 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:47.340+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:47.340256+0000) 2022-04-04T00:37:47.473 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:47.470+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:47.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:47.735+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:48.341 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:48.340+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:48.340466+0000) 2022-04-04T00:37:48.505 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:48.502+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:48.712 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:48.710+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:49.341 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:49.340+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:49.340666+0000) 2022-04-04T00:37:49.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:49.518+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:49.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:49.694+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:50.342 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:50.340+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:50.340884+0000) 2022-04-04T00:37:50.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:50.557+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:50.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:50.663+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:51.342 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:51.341+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:51.341124+0000) 2022-04-04T00:37:51.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:51.543+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:51.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:51.628+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:52.342 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:52.340+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:52.341370+0000) 2022-04-04T00:37:52.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:52.574+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:52.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:52.595+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:53.343 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:53.340+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:53.341566+0000) 2022-04-04T00:37:53.617 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:53.614+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:53.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:53.617+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:54.343 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:54.340+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:54.341673+0000) 2022-04-04T00:37:54.606 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:54.603+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:54.633 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:54.631+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:55.343 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:55.340+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:55.341884+0000) 2022-04-04T00:37:55.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:55.592+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:55.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:55.652+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:56.090 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:37:56.087+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:37:56.343 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:56.341+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:56.342095+0000) 2022-04-04T00:37:56.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:56.596+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:56.637 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:56.634+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:57.343 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:57.341+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:57.342330+0000) 2022-04-04T00:37:57.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:57.548+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:57.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:57.677+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:58.343 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:58.341+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:58.342502+0000) 2022-04-04T00:37:58.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:58.587+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:58.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:58.714+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:37:59.344 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:37:59.341+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:37:59.342747+0000) 2022-04-04T00:37:59.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:37:59.611+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:37:59.678 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:37:59.676+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:00.344 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:00.341+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:00.342927+0000) 2022-04-04T00:38:00.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:00.566+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:00.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:00.664+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:01.344 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:01.342+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:01.343093+0000) 2022-04-04T00:38:01.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:01.602+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:01.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:01.669+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:02.344 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:02.342+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:02.343320+0000) 2022-04-04T00:38:02.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:02.639+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:02.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:02.648+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:03.344 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:03.342+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:03.343467+0000) 2022-04-04T00:38:03.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:03.613+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:03.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:03.627+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:04.344 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:04.342+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:04.343542+0000) 2022-04-04T00:38:04.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:04.643+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:04.673 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:04.671+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:05.345 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:05.342+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:05.343753+0000) 2022-04-04T00:38:05.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:05.670+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:05.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:05.688+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:06.055 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:38:06.059 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:38:06.345 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:06.343+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:06.344020+0000) 2022-04-04T00:38:06.701 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:06.699+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:06.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:06.705+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:07.345 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:07.343+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:07.344216+0000) 2022-04-04T00:38:07.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:07.713+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:07.720 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:07.718+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:08.345 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:08.343+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:08.344417+0000) 2022-04-04T00:38:08.672 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:08.670+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:08.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:08.709+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:09.346 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:09.343+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:09.344632+0000) 2022-04-04T00:38:09.643 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:09.641+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:09.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:09.754+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:10.346 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:10.344+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:10.344872+0000) 2022-04-04T00:38:10.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:10.618+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:10.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:10.728+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:11.092 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:38:11.089+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:38:11.346 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:11.344+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:11.345037+0000) 2022-04-04T00:38:11.596 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:11.594+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:11.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:11.724+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:12.346 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:12.344+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:12.345206+0000) 2022-04-04T00:38:12.645 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:12.644+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:12.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:12.740+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:13.346 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:13.344+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:13.345372+0000) 2022-04-04T00:38:13.666 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:13.664+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:13.784 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:13.782+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:14.346 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:14.345+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:14.345543+0000) 2022-04-04T00:38:14.619 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:14.616+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:14.739 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:14.736+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:15.347 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:15.345+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:15.345766+0000) 2022-04-04T00:38:15.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:15.586+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:15.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:15.740+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:16.347 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:16.345+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:16.345991+0000) 2022-04-04T00:38:16.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:16.575+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:16.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:16.724+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:17.347 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:17.345+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:17.346190+0000) 2022-04-04T00:38:17.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:17.550+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:17.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:17.727+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:18.347 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:18.345+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:18.346372+0000) 2022-04-04T00:38:18.520 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:18.518+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:18.710 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:18.708+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:19.348 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:19.346+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:19.346613+0000) 2022-04-04T00:38:19.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:19.490+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:19.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:19.695+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:20.348 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:20.346+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:20.346759+0000) 2022-04-04T00:38:20.519 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:20.518+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:20.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:20.659+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:21.348 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:21.346+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:21.346958+0000) 2022-04-04T00:38:21.507 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:21.505+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:21.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:21.682+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:22.348 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:22.346+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:22.347238+0000) 2022-04-04T00:38:22.496 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:22.495+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:22.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:22.633+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:23.348 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:23.347+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:23.347430+0000) 2022-04-04T00:38:23.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:23.542+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:23.666 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:23.664+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:24.349 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:24.347+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:24.347612+0000) 2022-04-04T00:38:24.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:24.566+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:24.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:24.639+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:25.349 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:25.347+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:25.347791+0000) 2022-04-04T00:38:25.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:25.527+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:25.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:25.641+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:26.107 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:38:26.105+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:38:26.349 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:26.347+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:26.347964+0000) 2022-04-04T00:38:26.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:26.487+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:26.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:26.647+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:27.349 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:27.347+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:27.348192+0000) 2022-04-04T00:38:27.507 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:27.505+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:27.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:27.696+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:28.349 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:28.348+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:28.348300+0000) 2022-04-04T00:38:28.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:28.536+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:28.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:28.682+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:29.350 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:29.348+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:29.348448+0000) 2022-04-04T00:38:29.517 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:29.516+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:29.724 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:29.722+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:30.350 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:30.348+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:30.348614+0000) 2022-04-04T00:38:30.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:30.530+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:30.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:30.728+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:31.350 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:31.348+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:31.348791+0000) 2022-04-04T00:38:31.516 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:31.513+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:31.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:31.697+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:32.350 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:32.348+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:32.349020+0000) 2022-04-04T00:38:32.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:32.488+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:32.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:32.699+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:33.350 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:33.349+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:33.349177+0000) 2022-04-04T00:38:33.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:33.442+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:33.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:33.745+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:34.350 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:34.349+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:34.349423+0000) 2022-04-04T00:38:34.471 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:34.468+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:34.745 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:34.742+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:35.351 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:35.349+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:35.349664+0000) 2022-04-04T00:38:35.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:35.481+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:35.766 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:35.763+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:36.200 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:38:36.204 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:38:36.351 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:36.348+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:36.349932+0000) 2022-04-04T00:38:36.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:36.486+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:36.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:36.782+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:37.351 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:37.349+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:37.350170+0000) 2022-04-04T00:38:37.506 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:37.503+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:37.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:37.757+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:38.351 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:38.349+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:38.350292+0000) 2022-04-04T00:38:38.547 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:38.544+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:38.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:38.739+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:39.351 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:39.349+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:39.350471+0000) 2022-04-04T00:38:39.573 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:39.571+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:39.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:39.753+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:40.351 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:40.349+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:40.350541+0000) 2022-04-04T00:38:40.615 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:40.612+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:40.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:40.792+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:41.114 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:38:41.111+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:38:41.352 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:41.349+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:41.350747+0000) 2022-04-04T00:38:41.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:41.629+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:41.827 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:41.824+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:42.352 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:42.349+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:42.350969+0000) 2022-04-04T00:38:42.618 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:42.615+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:42.826 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:42.824+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:43.352 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:43.350+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:43.351133+0000) 2022-04-04T00:38:43.610 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:43.607+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:43.804 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:43.801+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:44.352 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:44.350+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:44.351317+0000) 2022-04-04T00:38:44.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:44.565+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:44.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:44.832+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:45.353 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:45.350+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:45.351596+0000) 2022-04-04T00:38:45.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:45.525+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:45.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:45.831+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:46.353 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:46.351+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:46.351835+0000) 2022-04-04T00:38:46.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:46.510+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:46.821 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:46.819+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:47.353 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:47.351+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:47.352059+0000) 2022-04-04T00:38:47.480 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:47.478+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:47.775 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:47.773+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:48.353 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:48.351+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:48.352246+0000) 2022-04-04T00:38:48.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:48.442+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:48.769 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:48.767+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:49.354 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:49.351+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:49.352480+0000) 2022-04-04T00:38:49.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:49.413+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:49.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:49.746+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:50.354 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:50.351+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:50.352733+0000) 2022-04-04T00:38:50.466 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:50.464+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:50.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:50.707+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:51.354 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:51.352+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:51.352952+0000) 2022-04-04T00:38:51.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:51.501+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:51.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:51.738+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:52.354 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:52.352+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:52.353119+0000) 2022-04-04T00:38:52.533 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:52.530+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:52.735 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:52.732+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:53.354 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:53.352+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:53.353371+0000) 2022-04-04T00:38:53.502 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:53.500+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:53.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:53.764+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:54.354 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:54.352+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:54.353477+0000) 2022-04-04T00:38:54.478 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:54.476+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:54.781 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:54.779+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:55.355 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:55.352+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:55.353707+0000) 2022-04-04T00:38:55.450 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:55.447+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:55.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:55.797+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:56.130 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:38:56.128+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:38:56.355 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:56.353+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:56.353924+0000) 2022-04-04T00:38:56.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:56.491+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:56.774 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:56.772+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:57.355 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:57.353+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:57.354130+0000) 2022-04-04T00:38:57.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:57.476+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:57.799 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:57.797+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:58.355 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:58.353+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:58.354338+0000) 2022-04-04T00:38:58.522 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:58.520+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:58.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:58.775+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:38:59.355 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:38:59.354+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:38:59.354546+0000) 2022-04-04T00:38:59.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:38:59.483+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:38:59.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:38:59.769+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:00.356 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:00.353+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:00.354708+0000) 2022-04-04T00:39:00.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:00.495+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:00.807 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:00.805+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:01.356 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:01.354+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:01.354917+0000) 2022-04-04T00:39:01.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:01.543+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:01.804 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:01.802+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:02.356 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:02.354+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:02.355112+0000) 2022-04-04T00:39:02.508 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:02.506+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:02.790 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:02.788+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:03.356 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:03.354+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:03.355346+0000) 2022-04-04T00:39:03.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:03.461+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:03.836 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:03.834+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:04.356 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:04.355+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:04.355484+0000) 2022-04-04T00:39:04.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:04.424+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:04.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:04.809+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:05.356 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:05.354+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:05.355680+0000) 2022-04-04T00:39:05.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:05.437+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:05.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:05.771+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:06.331 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:39:06.335 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:39:06.357 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:06.355+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:06.355871+0000) 2022-04-04T00:39:06.441 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:06.439+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:06.759 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:06.756+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:07.357 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:07.355+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:07.356053+0000) 2022-04-04T00:39:07.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:07.433+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:07.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:07.717+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:08.357 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:08.355+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:08.356239+0000) 2022-04-04T00:39:08.386 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:08.384+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:08.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:08.724+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:09.357 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:09.356+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:09.356407+0000) 2022-04-04T00:39:09.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:09.359+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:09.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:09.699+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:10.358 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:10.356+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:10.356590+0000) 2022-04-04T00:39:10.374 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:10.373+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:10.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:10.697+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:11.146 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:39:11.144+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:39:11.358 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:11.356+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:11.356815+0000) 2022-04-04T00:39:11.359 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:11.357+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:11.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:11.693+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:12.358 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:12.356+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:12.356935+0000) 2022-04-04T00:39:12.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:12.397+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:12.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:12.682+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:13.358 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:13.356+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:13.357053+0000) 2022-04-04T00:39:13.420 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:13.419+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:13.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:13.695+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:14.358 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:14.356+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:14.357190+0000) 2022-04-04T00:39:14.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:14.380+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:14.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:14.686+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:15.358 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:15.357+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:15.357405+0000) 2022-04-04T00:39:15.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:15.370+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:15.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:15.700+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:16.358 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:16.357+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:16.357588+0000) 2022-04-04T00:39:16.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:16.388+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:16.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:16.659+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:17.359 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:17.357+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:17.357779+0000) 2022-04-04T00:39:17.364 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:17.363+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:17.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:17.622+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:18.359 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:18.357+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:18.357931+0000) 2022-04-04T00:39:18.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:18.375+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:18.625 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:18.623+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:19.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:19.337+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:19.359 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:19.358+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:19.358091+0000) 2022-04-04T00:39:19.623 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:19.621+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:20.330 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:20.328+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:20.359 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:20.357+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:20.358272+0000) 2022-04-04T00:39:20.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:20.650+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:21.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:21.303+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:21.359 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:21.357+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:21.358470+0000) 2022-04-04T00:39:21.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:21.612+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:22.309 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:22.307+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:22.359 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:22.357+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:22.358626+0000) 2022-04-04T00:39:22.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:22.604+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:23.322 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:23.321+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:23.360 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:23.357+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:23.358864+0000) 2022-04-04T00:39:23.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:23.555+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:24.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:24.322+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:24.360 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:24.357+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:24.359076+0000) 2022-04-04T00:39:24.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:24.594+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:25.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:25.276+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:25.360 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:25.358+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:25.359251+0000) 2022-04-04T00:39:25.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:25.590+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:26.161 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:39:26.159+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:39:26.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:26.320+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:26.360 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:26.358+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:26.359460+0000) 2022-04-04T00:39:26.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:26.637+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:27.274 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:27.271+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:27.361 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:27.358+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:27.359721+0000) 2022-04-04T00:39:27.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:27.629+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:28.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:28.299+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:28.361 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:28.358+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:28.359928+0000) 2022-04-04T00:39:28.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:28.642+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:29.333 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:29.330+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:29.361 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:29.359+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:29.360123+0000) 2022-04-04T00:39:29.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:29.651+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:30.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:30.311+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:30.361 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:30.359+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:30.360318+0000) 2022-04-04T00:39:30.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:30.664+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:31.334 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:31.331+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:31.361 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:31.359+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:31.360532+0000) 2022-04-04T00:39:31.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:31.685+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:32.303 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:32.300+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:32.361 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:32.360+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:32.360779+0000) 2022-04-04T00:39:32.678 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:32.676+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:33.325 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:33.323+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:33.362 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:33.360+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:33.360936+0000) 2022-04-04T00:39:33.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:33.657+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:34.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:34.275+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:34.362 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:34.360+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:34.361009+0000) 2022-04-04T00:39:34.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:34.656+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:35.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:35.289+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:35.362 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:35.360+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:35.361223+0000) 2022-04-04T00:39:35.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:35.609+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:36.274 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:36.272+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:36.362 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:36.360+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:36.361371+0000) 2022-04-04T00:39:36.470 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:39:36.474 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:39:36.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:36.655+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:37.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:37.281+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:37.363 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:37.360+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:37.361595+0000) 2022-04-04T00:39:37.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:37.613+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:38.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:38.270+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:38.363 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:38.361+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:38.361805+0000) 2022-04-04T00:39:38.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:38.609+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:39.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:39.273+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:39.363 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:39.361+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:39.362033+0000) 2022-04-04T00:39:39.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:39.616+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:40.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:40.270+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:40.363 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:40.361+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:40.362176+0000) 2022-04-04T00:39:40.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:40.591+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:41.177 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:39:41.175+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:39:41.274 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:41.272+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:41.363 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:41.361+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:41.362334+0000) 2022-04-04T00:39:41.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:41.552+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:42.294 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:42.292+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:42.363 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:42.361+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:42.362540+0000) 2022-04-04T00:39:42.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:42.556+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:43.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:43.251+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:43.364 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:43.362+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:43.362754+0000) 2022-04-04T00:39:43.528 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:43.525+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:44.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:44.255+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:44.364 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:44.362+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:44.362945+0000) 2022-04-04T00:39:44.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:44.546+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:45.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:45.276+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:45.364 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:45.362+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:45.363074+0000) 2022-04-04T00:39:45.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:45.569+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:46.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:46.246+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:46.364 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:46.362+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:46.363258+0000) 2022-04-04T00:39:46.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:46.564+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:47.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:47.294+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:47.364 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:47.362+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:47.363520+0000) 2022-04-04T00:39:47.537 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:47.535+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:48.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:48.293+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:48.364 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:48.363+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:48.363700+0000) 2022-04-04T00:39:48.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:48.502+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:49.262 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:49.260+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:49.365 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:49.363+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:49.363883+0000) 2022-04-04T00:39:49.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:49.502+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:50.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:50.302+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:50.365 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:50.363+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:50.364131+0000) 2022-04-04T00:39:50.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:50.525+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:51.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:51.321+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:51.365 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:51.363+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:51.364350+0000) 2022-04-04T00:39:51.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:51.503+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:52.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:52.276+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:52.365 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:52.364+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:52.364527+0000) 2022-04-04T00:39:52.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:52.553+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:53.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:53.308+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:53.366 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:53.364+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:53.364758+0000) 2022-04-04T00:39:53.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:53.581+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:54.354 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:54.351+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:54.366 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:54.364+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:54.364979+0000) 2022-04-04T00:39:54.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:54.544+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:55.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:55.313+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:55.366 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:55.364+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:55.365186+0000) 2022-04-04T00:39:55.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:55.548+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:56.193 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:39:56.191+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:39:56.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:56.266+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:56.366 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:56.365+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:56.365449+0000) 2022-04-04T00:39:56.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:56.565+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:57.221 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:57.219+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:57.366 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:57.365+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:57.365632+0000) 2022-04-04T00:39:57.572 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:57.569+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 25 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:58.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:58.181+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:58.367 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:58.365+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:58.365824+0000) 2022-04-04T00:39:58.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:58.617+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:39:59.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:39:59.186+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:39:59.367 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:39:59.365+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:39:59.366053+0000) 2022-04-04T00:39:59.664 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:39:59.661+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:00.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:00.171+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:00.367 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:00.366+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:00.366299+0000) 2022-04-04T00:40:00.673 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:00.671+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:01.145 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:01.143+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:01.368 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:01.366+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:01.366513+0000) 2022-04-04T00:40:01.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:01.625+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:02.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:02.180+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:02.368 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:02.366+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:02.366636+0000) 2022-04-04T00:40:02.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:02.582+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:03.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:03.215+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:03.368 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:03.366+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:03.366883+0000) 2022-04-04T00:40:03.622 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:03.620+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:04.185 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:04.183+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:04.368 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:04.366+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:04.367030+0000) 2022-04-04T00:40:04.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:04.614+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:05.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:05.152+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:05.368 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:05.366+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:05.367190+0000) 2022-04-04T00:40:05.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:05.639+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:06.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:06.165+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:06.368 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:06.366+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:06.367334+0000) 2022-04-04T00:40:06.535 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:40:06.539 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:40:06.603 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:06.602+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:07.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:07.137+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:07.368 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:07.366+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:07.367550+0000) 2022-04-04T00:40:07.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:07.650+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:08.145 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:08.143+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:08.369 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:08.366+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:08.367735+0000) 2022-04-04T00:40:08.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:08.612+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:09.104 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:09.102+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:09.369 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:09.366+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:09.367922+0000) 2022-04-04T00:40:09.606 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:09.604+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:10.116 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:10.114+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:10.369 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:10.367+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:10.368006+0000) 2022-04-04T00:40:10.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:10.643+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:11.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:11.084+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:11.199 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:40:11.196+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:40:11.369 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:11.367+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:11.368223+0000) 2022-04-04T00:40:11.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:11.626+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:12.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:12.054+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:12.369 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:12.367+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:12.368411+0000) 2022-04-04T00:40:12.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:12.655+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:13.060 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:13.058+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:13.370 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:13.367+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:13.368627+0000) 2022-04-04T00:40:13.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:13.643+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:14.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:14.080+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:14.370 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:14.367+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:14.368866+0000) 2022-04-04T00:40:14.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:14.693+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:15.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:15.044+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:15.370 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:15.368+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:15.369070+0000) 2022-04-04T00:40:15.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:15.664+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:16.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:16.047+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:16.370 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:16.368+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:16.369247+0000) 2022-04-04T00:40:16.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:16.652+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:17.056 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:17.054+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:17.371 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:17.368+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:17.369485+0000) 2022-04-04T00:40:17.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:17.667+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:18.051 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:18.049+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:18.371 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:18.368+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:18.369703+0000) 2022-04-04T00:40:18.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:18.643+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:19.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:19.014+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:19.371 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:19.369+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:19.369899+0000) 2022-04-04T00:40:19.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:19.596+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:19.971 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:19.968+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:20.371 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:20.369+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:20.370051+0000) 2022-04-04T00:40:20.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:20.569+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:20.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:20.970+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:21.371 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:21.369+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:21.370188+0000) 2022-04-04T00:40:21.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:21.571+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:21.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:21.997+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:22.371 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:22.369+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:22.370372+0000) 2022-04-04T00:40:22.600 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:22.598+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:22.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:22.987+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:23.371 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:23.369+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:23.370597+0000) 2022-04-04T00:40:23.623 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:23.621+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:23.994 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:23.991+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:24.372 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:24.370+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:24.370802+0000) 2022-04-04T00:40:24.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:24.651+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:24.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:24.949+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:25.372 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:25.370+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:25.370998+0000) 2022-04-04T00:40:25.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:25.641+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:25.936 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:25.933+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:26.215 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:40:26.212+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:40:26.372 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:26.370+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:26.371222+0000) 2022-04-04T00:40:26.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:26.610+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:26.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:26.902+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:27.372 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:27.370+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:27.371463+0000) 2022-04-04T00:40:27.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:27.650+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:27.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:27.894+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:28.373 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:28.370+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:28.371607+0000) 2022-04-04T00:40:28.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:28.694+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:28.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:28.851+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:29.373 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:29.371+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:29.371831+0000) 2022-04-04T00:40:29.662 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:29.659+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:29.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:29.820+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:30.373 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:30.371+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:30.371946+0000) 2022-04-04T00:40:30.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:30.613+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:30.834 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:30.831+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:31.373 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:31.371+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:31.372103+0000) 2022-04-04T00:40:31.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:31.592+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:31.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:31.849+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:32.373 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:32.371+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:32.372276+0000) 2022-04-04T00:40:32.570 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:32.567+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:32.857 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:32.854+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:33.373 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:33.371+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:33.372458+0000) 2022-04-04T00:40:33.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:33.534+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:33.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:33.863+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:34.374 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:34.372+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:34.372695+0000) 2022-04-04T00:40:34.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:34.546+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:34.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:34.879+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:35.374 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:35.372+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:35.372880+0000) 2022-04-04T00:40:35.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:35.538+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:35.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:35.873+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:36.374 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:36.372+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:36.373065+0000) 2022-04-04T00:40:36.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:36.545+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:36.617 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:40:36.620 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:40:36.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:36.921+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:37.374 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:37.372+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:37.373306+0000) 2022-04-04T00:40:37.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:37.573+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:37.930 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:37.928+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:38.375 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:38.373+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:38.373491+0000) 2022-04-04T00:40:38.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:38.595+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:38.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:38.925+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:39.375 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:39.373+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:39.373629+0000) 2022-04-04T00:40:39.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:39.563+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:39.901 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:39.898+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:40.375 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:40.373+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:40.373793+0000) 2022-04-04T00:40:40.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:40.548+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:40.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:40.869+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:41.222 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:40:41.219+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:40:41.375 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:41.373+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:41.373960+0000) 2022-04-04T00:40:41.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:41.510+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:41.867 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:41.864+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:42.375 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:42.373+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:42.374135+0000) 2022-04-04T00:40:42.500 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:42.498+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:42.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:42.865+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:43.375 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:43.374+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:43.374376+0000) 2022-04-04T00:40:43.522 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:43.519+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:43.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:43.838+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:44.375 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:44.374+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:44.374494+0000) 2022-04-04T00:40:44.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:44.490+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:44.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:44.877+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:45.376 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:45.374+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:45.374677+0000) 2022-04-04T00:40:45.464 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:45.461+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:45.925 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:45.923+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:46.376 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:46.374+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:46.374876+0000) 2022-04-04T00:40:46.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:46.465+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:46.967 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:46.965+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:47.376 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:47.374+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:47.375111+0000) 2022-04-04T00:40:47.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:47.441+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:47.935 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:47.933+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:48.376 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:48.375+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:48.375323+0000) 2022-04-04T00:40:48.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:48.397+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:48.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:48.977+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:49.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:49.352+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:49.376 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:49.374+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:49.375456+0000) 2022-04-04T00:40:49.941 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:49.939+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:50.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:50.358+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:50.376 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:50.374+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:50.375608+0000) 2022-04-04T00:40:50.986 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:50.984+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:51.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:51.350+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:51.377 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:51.374+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:51.375803+0000) 2022-04-04T00:40:51.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:51.980+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:52.337 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:52.335+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:52.377 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:52.374+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:52.376010+0000) 2022-04-04T00:40:52.936 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:52.933+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:53.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:53.370+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:53.377 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:53.375+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:53.376171+0000) 2022-04-04T00:40:53.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:53.902+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:54.377 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:54.375+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:54.376329+0000) 2022-04-04T00:40:54.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:54.409+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:54.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:54.936+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:55.378 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:55.375+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:55.376569+0000) 2022-04-04T00:40:55.426 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:55.424+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:55.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:55.909+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:56.232 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:40:56.229+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:40:56.378 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:56.375+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:56.376747+0000) 2022-04-04T00:40:56.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:56.390+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:56.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:56.870+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:57.378 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:57.375+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:57.376907+0000) 2022-04-04T00:40:57.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:57.396+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:57.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:57.894+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:58.378 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:58.376+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:58.377032+0000) 2022-04-04T00:40:58.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:58.388+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:58.857 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:58.855+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:40:59.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:40:59.344+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:40:59.378 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:40:59.376+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:40:59.377238+0000) 2022-04-04T00:40:59.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:40:59.879+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:00.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:00.342+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:00.378 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:00.376+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:00.377451+0000) 2022-04-04T00:41:00.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:00.894+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:01.379 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:01.376+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:01.377627+0000) 2022-04-04T00:41:01.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:01.387+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:01.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:01.891+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:02.379 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:02.377+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:02.377884+0000) 2022-04-04T00:41:02.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:02.396+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:02.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:02.874+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:03.379 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:03.377+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:03.378118+0000) 2022-04-04T00:41:03.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:03.403+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:03.916 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:03.914+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:04.379 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:04.377+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:04.378264+0000) 2022-04-04T00:41:04.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:04.392+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:04.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:04.883+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:05.379 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:05.377+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:05.378375+0000) 2022-04-04T00:41:05.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:05.388+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:05.901 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:05.899+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:06.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:06.376+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:06.379 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:06.377+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:06.378514+0000) 2022-04-04T00:41:06.672 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:41:06.676 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:41:06.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:06.868+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:07.380 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:07.378+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:07.378755+0000) 2022-04-04T00:41:07.388 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:07.386+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:07.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:07.902+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:08.380 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:08.378+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:08.378975+0000) 2022-04-04T00:41:08.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:08.384+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:08.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:08.874+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:09.380 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:09.378+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:09.379136+0000) 2022-04-04T00:41:09.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:09.409+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:09.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:09.862+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:10.380 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:10.378+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:10.379342+0000) 2022-04-04T00:41:10.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:10.428+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:10.891 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:10.889+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:11.248 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:41:11.245+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:41:11.380 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:11.378+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:11.379498+0000) 2022-04-04T00:41:11.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:11.394+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:11.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:11.842+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:12.381 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:12.379+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:12.379693+0000) 2022-04-04T00:41:12.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:12.403+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:12.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:12.828+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:13.381 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:13.379+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:13.379874+0000) 2022-04-04T00:41:13.388 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:13.387+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:13.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:13.806+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:14.356 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:14.354+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:14.381 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:14.379+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:14.380110+0000) 2022-04-04T00:41:14.768 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:14.765+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:15.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:15.322+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:15.381 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:15.379+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:15.380336+0000) 2022-04-04T00:41:15.738 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:15.735+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:16.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:16.326+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:16.382 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:16.380+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:16.380573+0000) 2022-04-04T00:41:16.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:16.747+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:17.356 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:17.354+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:17.382 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:17.380+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:17.380837+0000) 2022-04-04T00:41:17.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:17.712+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:18.382 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:18.380+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:18.381101+0000) 2022-04-04T00:41:18.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:18.392+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:18.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:18.666+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:19.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:19.367+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:19.382 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:19.380+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:19.381370+0000) 2022-04-04T00:41:19.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:19.688+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:20.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:20.352+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:20.383 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:20.381+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:20.381608+0000) 2022-04-04T00:41:20.685 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:20.682+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:21.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:21.323+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:21.383 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:21.381+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:21.381783+0000) 2022-04-04T00:41:21.651 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:21.648+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:22.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:22.313+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:22.383 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:22.381+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:22.381968+0000) 2022-04-04T00:41:22.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:22.620+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:23.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:23.362+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:23.383 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:23.381+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:23.382157+0000) 2022-04-04T00:41:23.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:23.655+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:24.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:24.326+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:24.383 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:24.381+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:24.382344+0000) 2022-04-04T00:41:24.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:24.660+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:25.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:25.345+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:25.383 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:25.382+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:25.382529+0000) 2022-04-04T00:41:25.618 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:25.616+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:26.250 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:41:26.247+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:41:26.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:26.379+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:26.383 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:26.382+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:26.382720+0000) 2022-04-04T00:41:26.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:26.664+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:27.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:27.332+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:27.384 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:27.382+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:27.382856+0000) 2022-04-04T00:41:27.673 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:27.670+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:28.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:28.295+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:28.384 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:28.382+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:28.383056+0000) 2022-04-04T00:41:28.681 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:28.679+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:29.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:29.283+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:29.384 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:29.382+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:29.383239+0000) 2022-04-04T00:41:29.646 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:29.643+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:30.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:30.288+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:30.384 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:30.383+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:30.383388+0000) 2022-04-04T00:41:30.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:30.614+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:31.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:31.283+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:31.384 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:31.383+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:31.383592+0000) 2022-04-04T00:41:31.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:31.585+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:32.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:32.290+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:32.385 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:32.383+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:32.383845+0000) 2022-04-04T00:41:32.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:32.603+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:33.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:33.256+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:33.385 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:33.383+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:33.384027+0000) 2022-04-04T00:41:33.609 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:33.607+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:34.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:34.302+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:34.385 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:34.383+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:34.384211+0000) 2022-04-04T00:41:34.582 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:34.579+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:35.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:35.308+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:35.385 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:35.382+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:35.384385+0000) 2022-04-04T00:41:35.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:35.602+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:36.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:36.337+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:36.386 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:36.383+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:36.384597+0000) 2022-04-04T00:41:36.575 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:36.573+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:36.783 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:41:36.787 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:41:37.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:37.371+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:37.386 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:37.383+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:37.384851+0000) 2022-04-04T00:41:37.599 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:37.596+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:38.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:38.361+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:38.386 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:38.384+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:38.385012+0000) 2022-04-04T00:41:38.559 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:38.557+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:39.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:39.341+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:39.386 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:39.384+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:39.385181+0000) 2022-04-04T00:41:39.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:39.512+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:40.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:40.317+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:40.386 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:40.384+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:40.385400+0000) 2022-04-04T00:41:40.547 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:40.545+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:41.265 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:41:41.262+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:41:41.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:41.327+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:41.387 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:41.384+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:41.385640+0000) 2022-04-04T00:41:41.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:41.568+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:42.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:42.339+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:42.387 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:42.384+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:42.385888+0000) 2022-04-04T00:41:42.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:42.590+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:43.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:43.361+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:43.387 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:43.385+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:43.386049+0000) 2022-04-04T00:41:43.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:43.623+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:44.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:44.313+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:44.387 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:44.385+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:44.386266+0000) 2022-04-04T00:41:44.672 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:44.670+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:45.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:45.324+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:45.387 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:45.385+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:45.386457+0000) 2022-04-04T00:41:45.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:45.630+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:46.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:46.340+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:46.388 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:46.385+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:46.386683+0000) 2022-04-04T00:41:46.661 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:46.659+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:47.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:47.292+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:47.388 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:47.386+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:47.386880+0000) 2022-04-04T00:41:47.692 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:47.689+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:48.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:48.311+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:48.388 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:48.386+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:48.386998+0000) 2022-04-04T00:41:48.721 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:48.720+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:49.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:49.300+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:49.388 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:49.386+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:49.387195+0000) 2022-04-04T00:41:49.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:49.768+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:50.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:50.308+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:50.388 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:50.386+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:50.387398+0000) 2022-04-04T00:41:50.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:50.781+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:51.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:51.356+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:51.388 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:51.386+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:51.387596+0000) 2022-04-04T00:41:51.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:51.792+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:52.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:52.347+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:52.389 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:52.387+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:52.387810+0000) 2022-04-04T00:41:52.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:52.783+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:53.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:53.328+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:53.389 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:53.387+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:53.387975+0000) 2022-04-04T00:41:53.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:53.832+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:54.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:54.323+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:54.389 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:54.387+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:54.388167+0000) 2022-04-04T00:41:54.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:54.805+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:55.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:55.284+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:55.389 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:55.387+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:55.388383+0000) 2022-04-04T00:41:55.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:55.846+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:56.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:56.279+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:56.281 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:41:56.279+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:41:56.390 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:56.388+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:56.388643+0000) 2022-04-04T00:41:56.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:56.822+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:57.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:57.313+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:57.390 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:57.388+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:57.388895+0000) 2022-04-04T00:41:57.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:57.805+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:58.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:58.265+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:58.390 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:58.388+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:58.389140+0000) 2022-04-04T00:41:58.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:58.847+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:41:59.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:41:59.301+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:41:59.390 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:41:59.388+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:41:59.389326+0000) 2022-04-04T00:41:59.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:41:59.841+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:00.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:00.311+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:00.391 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:00.389+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:00.389550+0000) 2022-04-04T00:42:00.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:00.800+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:01.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:01.271+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:01.391 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:01.389+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:01.389817+0000) 2022-04-04T00:42:01.833 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:01.831+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:02.283 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:02.281+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:02.391 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:02.389+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:02.390072+0000) 2022-04-04T00:42:02.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:02.791+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:03.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:03.241+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:03.391 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:03.389+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:03.390315+0000) 2022-04-04T00:42:03.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:03.752+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:04.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:04.256+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:04.392 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:04.390+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:04.390563+0000) 2022-04-04T00:42:04.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:04.740+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:05.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:05.265+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:05.392 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:05.390+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:05.390831+0000) 2022-04-04T00:42:05.734 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:05.732+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:06.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:06.285+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:06.392 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:06.390+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:06.391074+0000) 2022-04-04T00:42:06.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:06.757+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:06.905 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:42:06.909 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:42:07.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:07.326+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:07.392 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:07.390+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:07.391347+0000) 2022-04-04T00:42:07.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:07.733+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:08.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:08.339+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:08.392 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:08.391+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:08.391599+0000) 2022-04-04T00:42:08.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:08.737+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:09.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:09.339+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:09.393 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:09.391+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:09.391779+0000) 2022-04-04T00:42:09.720 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:09.717+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:10.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:10.335+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:10.393 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:10.391+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:10.392022+0000) 2022-04-04T00:42:10.683 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:10.680+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:11.297 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:42:11.295+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:42:11.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:11.312+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:11.393 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:11.391+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:11.392192+0000) 2022-04-04T00:42:11.660 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:11.658+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:12.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:12.264+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:12.393 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:12.392+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:12.392343+0000) 2022-04-04T00:42:12.678 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:12.675+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:13.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:13.287+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:13.393 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:13.392+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:13.392517+0000) 2022-04-04T00:42:13.692 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:13.690+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:14.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:14.263+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:14.394 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:14.392+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:14.392727+0000) 2022-04-04T00:42:14.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:14.693+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:15.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:15.244+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:15.394 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:15.392+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:15.392989+0000) 2022-04-04T00:42:15.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:15.653+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:16.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:16.239+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:16.394 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:16.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:16.393235+0000) 2022-04-04T00:42:16.633 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:16.631+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:17.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:17.270+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:17.394 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:17.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:17.393460+0000) 2022-04-04T00:42:17.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:17.632+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:18.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:18.286+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:18.395 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:18.392+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:18.393638+0000) 2022-04-04T00:42:18.590 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:18.587+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:19.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:19.246+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:19.395 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:19.392+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:19.393863+0000) 2022-04-04T00:42:19.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:19.548+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:20.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:20.271+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:20.395 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:20.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:20.394106+0000) 2022-04-04T00:42:20.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:20.530+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:21.271 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:21.268+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:21.395 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:21.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:21.394301+0000) 2022-04-04T00:42:21.557 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:21.554+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:22.252 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:22.249+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:22.395 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:22.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:22.394523+0000) 2022-04-04T00:42:22.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:22.529+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:23.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:23.270+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:23.395 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:23.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:23.394718+0000) 2022-04-04T00:42:23.562 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:23.560+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:24.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:24.252+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:24.396 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:24.393+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:24.394922+0000) 2022-04-04T00:42:24.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:24.535+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:25.233 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:25.230+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:25.396 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:25.394+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:25.395158+0000) 2022-04-04T00:42:25.502 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:25.500+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:26.231 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:26.229+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:26.313 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:42:26.310+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:42:26.396 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:26.394+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:26.395403+0000) 2022-04-04T00:42:26.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:26.526+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:27.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:27.236+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:27.397 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:27.394+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:27.395628+0000) 2022-04-04T00:42:27.557 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:27.555+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:28.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:28.259+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:28.397 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:28.394+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:28.395881+0000) 2022-04-04T00:42:28.555 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:28.553+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:29.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:29.306+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:29.397 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:29.395+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:29.396084+0000) 2022-04-04T00:42:29.569 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:29.566+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:30.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:30.261+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:30.397 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:30.395+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:30.396297+0000) 2022-04-04T00:42:30.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:30.591+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:31.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:31.265+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:31.397 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:31.395+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:31.396542+0000) 2022-04-04T00:42:31.582 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:31.579+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:32.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:32.271+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:32.398 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:32.396+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:32.396777+0000) 2022-04-04T00:42:32.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:32.539+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:33.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:33.233+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:33.398 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:33.396+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:33.396968+0000) 2022-04-04T00:42:33.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:33.534+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:34.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:34.193+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:34.398 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:34.396+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:34.397188+0000) 2022-04-04T00:42:34.550 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:34.547+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:35.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:35.190+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:35.398 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:35.396+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:35.397360+0000) 2022-04-04T00:42:35.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:35.498+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:36.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:36.220+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:36.398 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:36.396+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:36.397570+0000) 2022-04-04T00:42:36.502 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:36.500+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:37.027 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:42:37.031 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:42:37.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:37.262+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:37.399 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:37.397+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:37.397872+0000) 2022-04-04T00:42:37.484 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:37.482+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:38.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:38.236+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:38.399 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:38.397+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:38.398114+0000) 2022-04-04T00:42:38.496 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:38.494+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:39.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:39.227+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:39.399 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:39.397+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:39.398360+0000) 2022-04-04T00:42:39.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:39.477+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:40.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:40.238+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:40.400 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:40.397+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:40.398599+0000) 2022-04-04T00:42:40.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:40.441+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:41.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:41.241+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:41.328 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:42:41.326+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:42:41.400 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:41.398+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:41.398870+0000) 2022-04-04T00:42:41.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:41.471+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:42.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:42.243+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:42.400 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:42.398+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:42.399072+0000) 2022-04-04T00:42:42.498 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:42.496+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:43.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:43.238+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:43.400 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:43.398+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:43.399302+0000) 2022-04-04T00:42:43.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:43.477+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:44.213 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:44.210+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:44.400 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:44.398+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:44.399432+0000) 2022-04-04T00:42:44.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:44.516+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:45.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:45.252+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:45.401 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:45.399+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:45.399649+0000) 2022-04-04T00:42:45.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:45.513+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:46.271 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:46.268+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:46.401 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:46.399+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:46.399894+0000) 2022-04-04T00:42:46.530 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:46.528+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:47.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:47.310+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:47.401 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:47.399+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:47.400145+0000) 2022-04-04T00:42:47.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:47.533+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:48.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:48.310+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:48.401 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:48.399+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:48.400342+0000) 2022-04-04T00:42:48.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:48.493+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:49.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:49.266+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:49.401 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:49.400+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:49.400566+0000) 2022-04-04T00:42:49.477 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:49.476+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:50.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:50.243+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:50.401 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:50.400+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:50.400831+0000) 2022-04-04T00:42:50.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:50.499+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:51.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:51.240+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:51.402 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:51.400+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:51.401052+0000) 2022-04-04T00:42:51.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:51.454+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:52.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:52.265+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:52.402 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:52.400+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:52.401276+0000) 2022-04-04T00:42:52.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:52.490+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:53.271 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:53.269+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:53.402 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:53.401+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:53.401514+0000) 2022-04-04T00:42:53.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:53.487+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:54.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:54.260+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:54.403 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:54.401+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:54.401797+0000) 2022-04-04T00:42:54.480 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:54.479+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:55.213 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:55.212+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:55.403 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:55.401+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:55.402045+0000) 2022-04-04T00:42:55.453 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:55.451+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:56.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:56.179+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:56.344 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:42:56.342+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:42:56.403 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:56.402+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:56.402288+0000) 2022-04-04T00:42:56.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:56.483+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:57.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:57.156+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:57.404 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:57.402+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:57.402505+0000) 2022-04-04T00:42:57.504 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:57.502+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:58.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:58.201+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:58.404 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:58.402+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:58.402743+0000) 2022-04-04T00:42:58.466 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:58.465+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:42:59.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:42:59.245+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:42:59.404 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:42:59.402+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:42:59.402983+0000) 2022-04-04T00:42:59.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:42:59.486+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:00.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:00.268+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:00.404 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:00.403+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:00.403173+0000) 2022-04-04T00:43:00.484 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:00.483+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:01.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:01.235+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:01.404 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:01.403+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:01.403319+0000) 2022-04-04T00:43:01.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:01.458+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:02.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:02.221+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:02.405 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:02.403+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:02.403559+0000) 2022-04-04T00:43:02.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:02.495+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:03.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:03.213+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:03.405 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:03.402+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:03.403787+0000) 2022-04-04T00:43:03.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:03.528+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:04.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:04.223+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:04.405 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:04.402+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:04.404024+0000) 2022-04-04T00:43:04.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:04.539+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:05.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:05.184+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:05.405 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:05.403+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:05.404273+0000) 2022-04-04T00:43:05.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:05.492+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:06.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:06.145+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:06.405 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:06.403+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:06.404521+0000) 2022-04-04T00:43:06.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:06.456+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:07.158 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:43:07.161 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:43:07.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:07.175+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:07.406 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:07.403+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:07.404811+0000) 2022-04-04T00:43:07.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:07.413+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:08.142 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:08.140+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:08.406 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:08.404+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:08.404984+0000) 2022-04-04T00:43:08.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:08.412+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:09.149 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:09.147+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:09.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:09.387+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:09.406 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:09.404+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:09.405159+0000) 2022-04-04T00:43:10.117 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:10.114+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:10.406 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:10.404+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:10.405403+0000) 2022-04-04T00:43:10.426 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:10.423+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:11.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:11.070+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:11.360 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:43:11.358+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:43:11.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:11.388+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:11.406 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:11.404+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:11.405643+0000) 2022-04-04T00:43:12.024 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:12.021+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:12.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:12.369+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:12.407 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:12.405+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:12.405874+0000) 2022-04-04T00:43:13.017 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:13.014+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:13.379 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:13.376+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:13.407 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:13.405+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:13.406090+0000) 2022-04-04T00:43:13.977 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:13.974+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:14.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:14.365+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:14.407 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:14.405+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:14.406319+0000) 2022-04-04T00:43:14.959 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:14.957+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:15.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:15.325+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:15.407 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:15.405+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:15.406495+0000) 2022-04-04T00:43:15.976 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:15.974+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:16.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:16.313+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:16.408 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:16.405+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:16.406661+0000) 2022-04-04T00:43:17.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:17.000+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:17.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:17.265+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:17.408 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:17.406+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:17.406860+0000) 2022-04-04T00:43:18.021 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:18.018+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:18.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:18.244+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:18.408 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:18.406+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:18.407080+0000) 2022-04-04T00:43:19.042 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:19.040+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:19.221 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:19.218+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:19.408 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:19.406+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:19.407324+0000) 2022-04-04T00:43:20.027 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:20.024+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:20.236 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:20.233+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:20.409 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:20.406+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:20.407571+0000) 2022-04-04T00:43:21.036 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:21.033+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:21.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:21.246+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:21.409 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:21.407+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:21.407826+0000) 2022-04-04T00:43:22.026 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:22.023+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:22.266 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:22.263+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:22.409 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:22.407+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:22.408079+0000) 2022-04-04T00:43:22.984 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:22.982+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:23.304 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:23.302+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:23.409 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:23.407+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:23.408308+0000) 2022-04-04T00:43:23.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:23.941+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:24.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:24.286+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:24.494 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:24.407+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:24.408453+0000) 2022-04-04T00:43:24.900 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:24.898+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:25.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:25.247+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:25.409 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:25.408+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:25.408599+0000) 2022-04-04T00:43:25.918 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:25.916+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:26.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:26.289+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:26.376 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:43:26.374+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:43:26.410 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:26.408+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:26.408833+0000) 2022-04-04T00:43:26.951 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:26.948+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:27.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:27.246+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:27.410 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:27.408+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:27.409081+0000) 2022-04-04T00:43:27.957 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:27.954+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:28.222 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:28.220+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:28.410 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:28.408+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:28.409299+0000) 2022-04-04T00:43:28.922 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:28.919+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:29.242 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:29.240+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:29.411 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:29.409+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:29.409537+0000) 2022-04-04T00:43:29.920 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:29.917+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:30.234 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:30.232+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:30.411 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:30.409+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:30.409735+0000) 2022-04-04T00:43:30.876 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:30.874+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:31.243 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:31.241+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:31.411 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:31.409+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:31.409925+0000) 2022-04-04T00:43:31.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:31.866+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:32.236 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:32.233+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:32.411 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:32.409+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:32.410140+0000) 2022-04-04T00:43:32.914 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:32.911+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:33.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:33.257+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:33.411 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:33.409+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:33.410307+0000) 2022-04-04T00:43:33.905 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:33.903+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:34.290 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:34.287+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:34.411 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:34.410+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:34.410465+0000) 2022-04-04T00:43:34.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:34.867+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:35.242 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:35.240+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:35.412 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:35.410+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:35.410705+0000) 2022-04-04T00:43:35.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:35.865+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:36.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:36.205+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:36.412 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:36.410+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:36.410929+0000) 2022-04-04T00:43:36.851 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:36.848+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:37.239 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:43:37.243 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:43:37.255 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:37.253+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:37.412 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:37.410+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:37.411146+0000) 2022-04-04T00:43:37.817 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:37.814+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:38.270 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:38.267+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:38.412 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:38.411+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:38.411340+0000) 2022-04-04T00:43:38.809 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:38.807+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:39.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:39.300+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:39.412 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:39.410+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:39.411490+0000) 2022-04-04T00:43:39.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:39.827+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:40.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:40.283+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:40.412 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:40.411+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:40.411623+0000) 2022-04-04T00:43:40.817 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:40.814+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:41.322 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:41.320+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:41.392 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:43:41.390+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:43:41.413 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:41.411+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:41.411846+0000) 2022-04-04T00:43:41.790 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:41.787+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:42.300 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:42.298+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:42.413 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:42.411+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:42.412062+0000) 2022-04-04T00:43:42.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:42.754+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:43.263 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:43.261+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:43.413 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:43.411+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:43.412241+0000) 2022-04-04T00:43:43.789 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:43.787+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:44.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:44.244+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:44.413 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:44.412+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:44.412449+0000) 2022-04-04T00:43:44.753 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:44.751+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:45.242 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:45.240+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:45.414 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:45.412+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:45.412635+0000) 2022-04-04T00:43:45.784 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:45.782+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:46.209 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:46.207+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:46.414 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:46.412+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:46.412845+0000) 2022-04-04T00:43:46.818 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:46.816+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:47.233 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:47.231+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:47.414 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:47.411+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:47.413075+0000) 2022-04-04T00:43:47.835 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:47.834+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:48.258 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:48.256+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:48.414 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:48.412+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:48.413306+0000) 2022-04-04T00:43:48.821 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:48.819+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:49.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:49.230+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:49.414 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:49.411+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:49.413494+0000) 2022-04-04T00:43:49.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:49.822+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:50.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:50.225+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:50.415 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:50.412+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:50.413673+0000) 2022-04-04T00:43:50.823 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:50.821+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:51.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:51.257+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:51.415 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:51.412+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:51.413861+0000) 2022-04-04T00:43:51.835 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:51.833+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:52.301 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:52.299+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:52.415 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:52.412+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:52.414049+0000) 2022-04-04T00:43:52.844 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:52.842+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:53.312 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:53.310+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:53.415 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:53.413+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:53.414228+0000) 2022-04-04T00:43:53.840 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:53.838+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:54.287 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:54.286+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:54.415 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:54.413+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:54.414410+0000) 2022-04-04T00:43:54.823 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:54.821+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:55.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:55.283+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:55.416 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:55.413+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:55.414637+0000) 2022-04-04T00:43:55.854 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:55.852+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:56.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:56.279+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:56.407 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:43:56.405+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:43:56.415 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:56.413+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:56.414793+0000) 2022-04-04T00:43:56.890 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:56.889+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:57.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:57.293+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:57.416 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:57.414+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:57.415000+0000) 2022-04-04T00:43:57.934 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:57.932+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:58.318 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:58.316+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:58.416 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:58.414+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:58.415164+0000) 2022-04-04T00:43:58.955 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:58.954+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:43:59.355 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:43:59.353+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:43:59.416 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:43:59.414+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:43:59.415381+0000) 2022-04-04T00:43:59.944 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:43:59.941+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:00.319 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:00.317+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:00.416 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:00.414+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:00.415520+0000) 2022-04-04T00:44:00.906 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:00.903+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:01.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:01.312+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:01.417 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:01.414+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:01.415718+0000) 2022-04-04T00:44:01.898 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:01.895+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:02.306 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:02.303+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:02.417 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:02.415+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:02.415916+0000) 2022-04-04T00:44:02.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:02.906+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:03.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:03.271+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:03.417 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:03.415+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:03.416176+0000) 2022-04-04T00:44:03.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:03.861+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:04.289 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:04.286+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:04.417 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:04.415+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:04.416423+0000) 2022-04-04T00:44:04.876 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:04.873+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:05.327 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:05.325+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:05.417 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:05.415+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:05.416595+0000) 2022-04-04T00:44:05.888 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:05.885+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:06.322 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:06.319+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:06.418 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:06.416+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:06.416831+0000) 2022-04-04T00:44:06.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:06.866+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:07.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:07.317+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:07.360 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:44:07.364 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:44:07.418 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:07.416+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:07.417012+0000) 2022-04-04T00:44:07.846 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:07.844+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:08.329 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:08.327+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:08.418 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:08.416+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:08.417198+0000) 2022-04-04T00:44:08.846 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:08.844+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:09.337 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:09.334+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:09.418 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:09.416+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:09.417433+0000) 2022-04-04T00:44:09.812 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:09.809+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:10.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:10.299+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:10.419 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:10.417+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:10.417625+0000) 2022-04-04T00:44:10.860 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:10.857+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:11.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:11.283+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:11.412 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:44:11.410+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:44:11.419 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:11.417+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:11.417868+0000) 2022-04-04T00:44:11.892 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:11.889+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:12.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:12.248+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:12.419 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:12.417+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:12.418090+0000) 2022-04-04T00:44:12.931 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:12.929+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:13.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:13.224+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:13.419 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:13.417+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:13.418306+0000) 2022-04-04T00:44:13.960 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:13.958+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:14.242 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:14.239+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:14.419 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:14.417+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:14.418451+0000) 2022-04-04T00:44:14.968 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:14.965+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:15.206 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:15.203+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:15.420 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:15.418+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:15.418648+0000) 2022-04-04T00:44:15.991 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:15.988+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:16.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:16.176+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:16.420 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:16.418+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:16.418820+0000) 2022-04-04T00:44:16.976 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:16.973+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:17.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:17.173+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:17.420 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:17.418+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:17.419033+0000) 2022-04-04T00:44:17.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:17.942+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:18.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:18.153+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:18.420 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:18.418+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:18.419264+0000) 2022-04-04T00:44:18.940 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:18.938+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:19.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:19.121+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:19.420 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:19.419+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:19.419493+0000) 2022-04-04T00:44:19.925 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:19.923+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:20.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:20.122+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:20.421 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:20.419+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:20.419591+0000) 2022-04-04T00:44:20.942 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:20.940+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:21.099 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:21.097+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:21.421 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:21.419+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:21.419810+0000) 2022-04-04T00:44:21.935 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:21.933+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:22.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:22.095+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:22.421 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:22.419+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:22.419970+0000) 2022-04-04T00:44:22.950 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:22.948+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:23.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:23.122+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:23.421 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:23.419+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:23.420184+0000) 2022-04-04T00:44:23.993 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:23.991+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:24.098 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:24.096+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:24.421 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:24.420+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:24.420447+0000) 2022-04-04T00:44:25.022 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:25.020+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:25.104 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:25.102+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:25.421 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:25.419+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:25.420672+0000) 2022-04-04T00:44:26.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:26.002+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:26.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:26.069+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:26.422 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:26.420+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:26.420906+0000) 2022-04-04T00:44:26.428 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:44:26.426+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:44:26.959 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:26.957+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:27.023 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:27.020+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:27.422 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:27.420+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:27.421183+0000) 2022-04-04T00:44:27.934 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:27.932+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:27.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:27.990+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:28.422 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:28.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:28.421443+0000) 2022-04-04T00:44:28.910 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:28.908+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:29.022 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:29.020+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:29.423 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:29.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:29.421645+0000) 2022-04-04T00:44:29.887 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:29.884+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:29.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:29.988+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:30.423 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:30.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:30.421879+0000) 2022-04-04T00:44:30.895 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:30.893+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:31.009 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:31.007+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:31.423 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:31.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:31.422120+0000) 2022-04-04T00:44:31.924 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:31.922+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:32.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:32.011+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:32.423 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:32.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:32.422375+0000) 2022-04-04T00:44:32.876 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:32.874+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:33.044 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:33.042+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:33.423 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:33.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:33.422624+0000) 2022-04-04T00:44:33.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:33.832+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:34.005 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:34.004+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:34.424 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:34.421+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:34.422865+0000) 2022-04-04T00:44:34.789 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:34.786+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:34.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:34.994+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:35.424 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:35.422+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:35.423112+0000) 2022-04-04T00:44:35.798 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:35.795+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:36.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:36.011+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:36.424 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:36.422+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:36.423321+0000) 2022-04-04T00:44:36.789 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:36.787+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:37.044 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:37.042+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:37.424 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:37.422+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:37.423536+0000) 2022-04-04T00:44:37.432 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:44:37.436 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:44:37.818 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:37.815+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:38.002 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:38.000+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:38.425 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:38.422+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:38.423766+0000) 2022-04-04T00:44:38.828 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:38.825+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:38.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:38.980+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:39.425 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:39.423+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:39.424047+0000) 2022-04-04T00:44:39.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:39.778+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:39.971 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:39.969+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:40.425 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:40.423+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:40.424229+0000) 2022-04-04T00:44:40.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:40.754+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:40.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:40.956+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:41.425 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:41.423+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:41.424448+0000) 2022-04-04T00:44:41.444 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:44:41.441+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:44:41.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:41.771+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:41.941 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:41.940+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:42.426 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:42.423+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:42.424743+0000) 2022-04-04T00:44:42.790 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:42.788+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:42.908 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:42.907+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:43.426 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:43.424+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:43.424907+0000) 2022-04-04T00:44:43.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:43.749+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:43.929 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:43.928+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:44.426 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:44.424+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:44.425109+0000) 2022-04-04T00:44:44.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:44.793+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:44.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:44.976+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:45.426 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:45.424+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:45.425267+0000) 2022-04-04T00:44:45.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:45.745+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:45.986 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:45.984+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:46.426 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:46.424+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:46.425427+0000) 2022-04-04T00:44:46.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:46.784+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:46.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:46.950+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:47.427 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:47.424+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:47.425653+0000) 2022-04-04T00:44:47.794 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:47.792+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:47.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:47.979+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:48.427 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:48.425+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:48.425838+0000) 2022-04-04T00:44:48.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:48.754+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:49.006 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:49.004+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:49.427 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:49.425+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:49.426051+0000) 2022-04-04T00:44:49.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:49.756+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:49.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:49.962+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:50.427 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:50.425+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:50.426224+0000) 2022-04-04T00:44:50.759 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:50.757+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:50.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:50.948+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:51.427 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:51.425+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:51.426424+0000) 2022-04-04T00:44:51.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:51.717+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:51.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:51.983+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:52.428 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:52.425+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:52.426645+0000) 2022-04-04T00:44:52.689 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:52.687+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:52.941 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:52.940+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:53.428 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:53.426+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:53.426895+0000) 2022-04-04T00:44:53.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:53.730+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:53.926 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:53.925+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:54.428 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:54.426+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:54.427143+0000) 2022-04-04T00:44:54.705 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:54.703+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:54.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:54.919+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:55.428 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:55.426+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:55.427324+0000) 2022-04-04T00:44:55.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:55.695+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:55.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:55.916+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:56.428 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:56.426+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:56.427485+0000) 2022-04-04T00:44:56.460 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:44:56.458+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:44:56.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:56.682+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:56.880 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:56.877+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:57.429 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:57.427+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:57.427707+0000) 2022-04-04T00:44:57.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:57.676+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:57.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:57.845+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:58.429 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:58.427+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:58.427952+0000) 2022-04-04T00:44:58.717 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:58.714+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:58.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:58.803+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:44:59.429 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:44:59.427+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:44:59.428193+0000) 2022-04-04T00:44:59.693 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:44:59.691+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:44:59.828 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:44:59.825+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:00.429 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:00.427+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:00.428426+0000) 2022-04-04T00:45:00.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:00.739+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:00.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:00.806+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:01.430 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:01.428+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:01.428643+0000) 2022-04-04T00:45:01.739 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:01.737+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:01.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:01.782+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:02.430 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:02.428+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:02.428851+0000) 2022-04-04T00:45:02.762 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:02.760+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:02.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:02.808+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:03.430 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:03.428+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:03.428998+0000) 2022-04-04T00:45:03.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:03.755+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:03.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:03.844+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:04.430 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:04.428+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:04.429207+0000) 2022-04-04T00:45:04.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:04.737+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:04.817 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:04.815+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:05.430 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:05.429+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:05.429401+0000) 2022-04-04T00:45:05.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:05.731+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:05.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:05.862+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:06.430 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:06.429+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:06.429531+0000) 2022-04-04T00:45:06.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:06.707+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:06.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:06.820+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:07.431 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:07.429+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:07.429748+0000) 2022-04-04T00:45:07.546 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:45:07.550 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:45:07.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:07.727+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:07.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:07.859+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:08.431 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:08.429+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:08.429984+0000) 2022-04-04T00:45:08.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:08.700+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:08.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:08.851+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:09.431 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:09.429+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:09.430164+0000) 2022-04-04T00:45:09.687 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:09.684+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:09.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:09.875+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:10.431 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:10.430+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:10.430346+0000) 2022-04-04T00:45:10.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:10.733+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:10.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:10.925+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:11.432 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:11.429+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:11.430585+0000) 2022-04-04T00:45:11.476 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:45:11.473+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:45:11.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:11.706+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:11.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:11.916+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:12.432 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:12.429+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:12.430733+0000) 2022-04-04T00:45:12.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:12.681+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:12.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:12.956+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:13.432 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:13.430+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:13.430950+0000) 2022-04-04T00:45:13.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:13.731+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:13.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:13.925+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:14.432 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:14.431+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:14.431191+0000) 2022-04-04T00:45:14.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:14.728+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:14.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:14.909+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:15.432 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:15.431+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:15.431436+0000) 2022-04-04T00:45:15.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:15.699+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:15.898 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:15.895+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:16.433 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:16.431+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:16.431684+0000) 2022-04-04T00:45:16.727 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:16.724+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:16.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:16.898+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:17.433 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:17.430+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:17.431942+0000) 2022-04-04T00:45:17.753 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:17.750+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:17.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:17.930+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:18.433 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:18.431+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:18.432195+0000) 2022-04-04T00:45:18.759 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:18.756+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:18.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:18.916+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:19.434 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:19.431+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:19.432413+0000) 2022-04-04T00:45:19.804 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:19.802+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:19.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:19.946+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:20.434 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:20.431+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:20.432640+0000) 2022-04-04T00:45:20.797 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:20.795+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:20.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:20.980+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:21.434 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:21.431+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:21.432878+0000) 2022-04-04T00:45:21.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:21.747+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:21.987 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:21.985+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:22.434 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:22.432+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:22.433078+0000) 2022-04-04T00:45:22.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:22.729+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:22.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:22.979+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:23.434 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:23.432+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:23.433292+0000) 2022-04-04T00:45:23.764 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:23.761+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:23.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:23.950+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:24.435 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:24.432+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:24.433539+0000) 2022-04-04T00:45:24.796 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:24.794+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:24.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:24.915+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:25.435 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:25.432+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:25.433796+0000) 2022-04-04T00:45:25.810 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:25.808+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:25.936 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:25.934+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:26.435 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:26.433+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:26.434012+0000) 2022-04-04T00:45:26.486 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:45:26.484+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:45:26.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:26.797+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:26.935 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:26.933+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:27.435 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:27.433+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:27.434273+0000) 2022-04-04T00:45:27.850 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:27.848+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:27.967 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:27.965+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:28.436 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:28.433+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:28.434514+0000) 2022-04-04T00:45:28.827 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:28.825+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:28.986 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:28.984+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:29.435 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:29.434+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:29.434804+0000) 2022-04-04T00:45:29.849 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:29.848+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:29.959 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:29.957+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:30.436 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:30.434+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:30.435056+0000) 2022-04-04T00:45:30.876 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:30.874+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:30.914 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:30.911+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:31.436 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:31.434+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:31.435303+0000) 2022-04-04T00:45:31.879 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:31.877+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:31.946 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:31.944+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:32.437 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:32.434+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:32.435592+0000) 2022-04-04T00:45:32.854 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:32.852+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:32.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:32.920+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:33.437 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:33.435+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:33.435825+0000) 2022-04-04T00:45:33.856 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:33.854+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:33.968 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:33.967+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:34.437 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:34.435+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:34.436044+0000) 2022-04-04T00:45:34.871 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:34.870+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:34.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:34.923+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:35.437 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:35.435+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:35.436192+0000) 2022-04-04T00:45:35.840 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:35.838+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:35.910 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:35.908+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:36.437 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:36.435+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:36.436326+0000) 2022-04-04T00:45:36.809 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:36.807+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:36.884 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:36.882+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:37.438 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:37.435+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:37.436549+0000) 2022-04-04T00:45:37.626 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:45:37.630 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:45:37.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:37.827+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:37.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:37.845+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:38.438 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:38.436+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:38.436852+0000) 2022-04-04T00:45:38.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:38.832+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:38.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:38.888+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:39.438 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:39.436+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:39.437054+0000) 2022-04-04T00:45:39.831 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:39.829+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:39.914 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:39.912+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:40.438 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:40.436+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:40.437266+0000) 2022-04-04T00:45:40.798 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:40.796+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:40.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:40.898+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:41.438 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:41.436+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:41.437509+0000) 2022-04-04T00:45:41.502 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:45:41.500+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:45:41.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:41.822+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:41.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:41.945+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:42.439 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:42.437+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:42.437790+0000) 2022-04-04T00:45:42.781 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:42.779+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:42.941 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:42.940+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:43.439 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:43.437+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:43.437984+0000) 2022-04-04T00:45:43.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:43.827+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:43.960 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:43.958+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:44.439 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:44.437+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:44.438171+0000) 2022-04-04T00:45:44.848 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:44.846+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:44.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:44.929+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:45.439 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:45.437+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:45.438435+0000) 2022-04-04T00:45:45.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:45.841+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:45.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:45.947+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:46.440 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:46.438+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:46.438651+0000) 2022-04-04T00:45:46.850 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:46.848+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:46.946 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:46.944+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:47.440 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:47.438+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:47.438908+0000) 2022-04-04T00:45:47.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:47.820+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:47.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:47.896+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:48.440 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:48.438+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:48.439174+0000) 2022-04-04T00:45:48.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:48.809+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:48.926 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:48.923+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:49.440 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:49.439+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:49.439449+0000) 2022-04-04T00:45:49.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:49.779+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:49.899 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:49.897+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:50.441 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:50.439+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:50.439697+0000) 2022-04-04T00:45:50.786 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:50.783+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:50.925 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:50.923+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:51.441 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:51.439+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:51.439942+0000) 2022-04-04T00:45:51.796 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:51.794+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:51.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:51.962+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:52.441 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:52.439+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:52.440208+0000) 2022-04-04T00:45:52.797 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:52.794+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:52.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:52.993+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:53.441 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:53.440+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:53.440419+0000) 2022-04-04T00:45:53.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:53.770+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:53.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:53.946+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:54.442 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:54.440+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:54.440662+0000) 2022-04-04T00:45:54.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:54.733+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:54.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:54.990+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:55.442 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:55.440+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:55.440913+0000) 2022-04-04T00:45:55.778 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:55.775+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:55.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:55.990+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:56.442 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:56.440+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:56.441156+0000) 2022-04-04T00:45:56.518 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:45:56.516+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:45:56.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:56.821+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:57.032 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:57.029+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:57.442 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:57.441+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:57.441406+0000) 2022-04-04T00:45:57.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:57.799+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:58.005 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:58.002+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:58.443 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:58.441+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:58.441649+0000) 2022-04-04T00:45:58.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:58.818+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:45:58.994 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:45:58.992+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:45:59.443 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:45:59.441+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:45:59.441820+0000) 2022-04-04T00:45:59.846 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:45:59.843+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:00.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:00.022+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:00.443 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:00.441+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:00.442057+0000) 2022-04-04T00:46:00.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:00.803+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:01.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:01.001+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:01.443 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:01.441+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:01.442268+0000) 2022-04-04T00:46:01.778 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:01.776+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:02.031 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:02.029+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:02.443 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:02.441+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:02.442464+0000) 2022-04-04T00:46:02.825 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:02.823+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:02.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:02.997+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:03.444 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:03.441+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:03.442657+0000) 2022-04-04T00:46:03.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:03.866+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:04.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:04.044+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:04.444 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:04.441+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:04.442895+0000) 2022-04-04T00:46:04.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:04.831+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:05.068 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:05.065+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:05.444 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:05.442+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:05.443065+0000) 2022-04-04T00:46:05.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:05.865+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:06.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:06.017+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:06.444 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:06.442+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:06.443210+0000) 2022-04-04T00:46:06.847 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:06.845+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:07.051 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:07.048+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:07.444 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:07.442+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:07.443414+0000) 2022-04-04T00:46:07.765 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:46:07.769 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:46:07.844 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:07.842+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:08.045 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:08.043+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:08.445 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:08.442+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:08.443595+0000) 2022-04-04T00:46:08.836 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:08.833+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:09.017 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:09.014+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:09.445 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:09.442+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:09.443765+0000) 2022-04-04T00:46:09.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:09.789+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:09.994 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:09.992+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:10.445 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:10.443+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:10.443951+0000) 2022-04-04T00:46:10.799 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:10.796+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:10.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:10.995+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:11.445 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:11.443+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:11.444123+0000) 2022-04-04T00:46:11.521 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:46:11.519+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:46:11.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:11.785+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:11.961 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:11.959+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:12.445 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:12.443+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:12.444259+0000) 2022-04-04T00:46:12.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:12.753+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:12.961 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:12.959+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:13.445 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:13.443+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:13.444476+0000) 2022-04-04T00:46:13.744 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:13.741+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:14.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:14.005+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:14.446 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:14.443+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:14.444647+0000) 2022-04-04T00:46:14.761 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:14.759+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:15.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:15.023+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:15.446 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:15.444+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:15.444901+0000) 2022-04-04T00:46:15.724 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:15.722+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:16.056 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:16.053+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:16.446 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:16.444+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:16.445056+0000) 2022-04-04T00:46:16.678 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:16.676+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:17.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:17.072+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:17.446 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:17.444+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:17.445266+0000) 2022-04-04T00:46:17.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:17.647+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:18.108 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:18.105+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:18.446 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:18.444+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:18.445424+0000) 2022-04-04T00:46:18.609 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:18.607+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:19.119 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:19.117+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:19.446 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:19.444+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:19.445548+0000) 2022-04-04T00:46:19.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:19.648+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:20.101 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:20.099+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:20.447 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:20.444+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:20.445740+0000) 2022-04-04T00:46:20.662 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:20.660+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:21.149 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:21.147+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:21.447 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:21.445+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:21.445979+0000) 2022-04-04T00:46:21.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:21.688+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:22.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:22.181+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:22.447 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:22.445+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:22.446207+0000) 2022-04-04T00:46:22.689 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:22.687+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:23.149 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:23.147+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:23.447 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:23.445+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:23.446402+0000) 2022-04-04T00:46:23.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:23.642+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:24.155 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:24.153+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:24.448 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:24.445+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:24.446569+0000) 2022-04-04T00:46:24.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:24.595+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:25.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:25.168+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:25.448 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:25.446+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:25.446830+0000) 2022-04-04T00:46:25.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:25.634+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:26.216 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:26.214+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:26.448 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:26.446+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:26.447029+0000) 2022-04-04T00:46:26.537 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:46:26.535+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:46:26.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:26.589+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:27.192 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:27.190+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:27.448 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:27.446+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:27.447251+0000) 2022-04-04T00:46:27.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:27.588+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:28.210 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:28.208+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:28.449 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:28.446+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:28.447430+0000) 2022-04-04T00:46:28.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:28.543+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:29.213 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:29.212+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:29.449 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:29.447+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:29.447651+0000) 2022-04-04T00:46:29.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:29.498+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:30.220 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:30.219+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:30.449 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:30.447+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:30.447836+0000) 2022-04-04T00:46:30.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:30.492+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:31.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:31.205+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:31.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:31.443+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:31.449 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:31.447+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:31.448027+0000) 2022-04-04T00:46:32.190 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:32.188+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:32.423 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:32.421+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:32.449 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:32.447+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:32.448198+0000) 2022-04-04T00:46:33.177 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:33.175+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:33.449 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:33.447+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:33.448381+0000) 2022-04-04T00:46:33.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:33.464+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:34.202 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:34.200+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:34.450 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:34.448+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:34.448628+0000) 2022-04-04T00:46:34.503 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:34.502+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:35.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:35.172+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:35.450 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:35.448+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:35.448874+0000) 2022-04-04T00:46:35.533 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:35.532+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:36.223 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:36.221+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:36.450 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:36.448+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:36.449060+0000) 2022-04-04T00:46:36.498 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:36.497+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:37.242 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:37.240+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:37.450 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:37.449+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:37.449303+0000) 2022-04-04T00:46:37.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:37.482+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:37.895 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:46:37.899 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:46:38.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:38.276+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:38.450 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:38.449+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:38.449481+0000) 2022-04-04T00:46:38.454 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:38.453+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:39.306 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:39.304+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:39.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:39.423+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:39.450 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:39.448+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:39.449695+0000) 2022-04-04T00:46:40.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:40.338+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:40.437 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:40.435+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:40.451 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:40.449+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:40.449870+0000) 2022-04-04T00:46:41.331 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:41.329+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:41.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:41.389+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:41.451 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:41.449+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:41.450086+0000) 2022-04-04T00:46:41.553 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:46:41.551+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:46:42.313 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:42.312+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:42.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:42.402+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:42.451 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:42.450+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:42.450337+0000) 2022-04-04T00:46:43.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:43.275+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:43.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:43.391+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:43.451 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:43.450+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:43.450584+0000) 2022-04-04T00:46:44.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:44.295+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:44.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:44.441+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:44.452 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:44.450+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:44.450861+0000) 2022-04-04T00:46:45.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:45.273+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:45.452 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:45.451+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:45.451085+0000) 2022-04-04T00:46:45.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:45.489+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:46.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:46.315+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:46.452 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:46.449+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:46.451272+0000) 2022-04-04T00:46:46.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:46.522+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:47.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:47.274+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:47.452 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:47.450+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:47.451503+0000) 2022-04-04T00:46:47.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:47.534+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:48.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:48.299+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:48.452 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:48.450+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:48.451697+0000) 2022-04-04T00:46:48.501 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:48.498+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:49.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:49.294+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:49.453 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:49.450+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:49.451931+0000) 2022-04-04T00:46:49.519 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:49.517+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:50.261 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:50.259+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:50.453 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:50.451+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:50.452143+0000) 2022-04-04T00:46:50.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:50.566+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:51.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:51.251+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:51.453 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:51.451+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:51.452318+0000) 2022-04-04T00:46:51.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:51.532+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:52.304 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:52.301+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:52.454 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:52.451+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:52.452534+0000) 2022-04-04T00:46:52.519 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:52.517+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:53.335 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:53.333+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:53.454 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:53.451+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:53.452791+0000) 2022-04-04T00:46:53.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:53.560+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:54.290 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:54.288+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:54.454 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:54.452+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:54.453016+0000) 2022-04-04T00:46:54.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:54.558+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:55.308 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:55.306+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:55.454 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:55.452+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:55.453227+0000) 2022-04-04T00:46:55.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:55.546+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:56.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:56.270+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:56.455 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:56.452+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:56.453468+0000) 2022-04-04T00:46:56.554 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:46:56.552+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:46:56.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:56.560+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:57.270 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:57.268+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:57.455 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:57.452+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:57.453722+0000) 2022-04-04T00:46:57.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:57.534+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:58.255 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:58.253+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:58.455 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:58.453+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:58.453939+0000) 2022-04-04T00:46:58.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:58.490+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:59.212 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:46:59.210+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:46:59.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:46:59.452+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:46:59.642 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:46:59.453+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:46:59.454175+0000) 2022-04-04T00:47:00.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:00.229+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:00.455 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:00.453+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:00.454416+0000) 2022-04-04T00:47:00.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:00.455+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:01.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:01.213+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:01.429 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:01.427+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:01.455 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:01.453+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:01.454663+0000) 2022-04-04T00:47:02.239 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:02.236+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:02.456 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:02.454+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:02.454844+0000) 2022-04-04T00:47:02.480 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:02.477+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:03.239 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:03.237+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:03.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:03.446+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:03.456 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:03.454+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:03.455032+0000) 2022-04-04T00:47:04.289 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:04.287+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:04.456 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:04.454+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:04.455218+0000) 2022-04-04T00:47:04.490 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:04.488+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:05.255 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:05.252+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:05.457 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:05.454+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:05.455446+0000) 2022-04-04T00:47:05.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:05.494+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:06.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:06.284+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:06.457 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:06.455+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:06.455686+0000) 2022-04-04T00:47:06.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:06.512+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:07.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:07.282+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:07.457 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:07.455+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:07.455930+0000) 2022-04-04T00:47:07.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:07.511+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:08.018 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:47:08.022 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:47:08.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:08.322+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:08.457 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:08.455+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:08.456175+0000) 2022-04-04T00:47:08.468 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:08.466+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:09.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:09.279+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:09.457 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:09.455+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:09.456422+0000) 2022-04-04T00:47:09.507 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:09.504+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:10.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:10.276+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:10.458 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:10.456+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:10.456660+0000) 2022-04-04T00:47:10.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:10.456+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:11.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:11.276+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:11.458 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:11.456+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:11.456909+0000) 2022-04-04T00:47:11.460 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:11.458+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:11.570 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:47:11.568+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:47:12.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:12.323+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:12.458 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:12.456+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:12.457186+0000) 2022-04-04T00:47:12.487 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:12.484+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:13.283 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:13.281+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:13.458 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:13.456+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:13.457423+0000) 2022-04-04T00:47:13.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:13.526+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:14.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:14.312+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:14.459 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:14.457+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:14.457657+0000) 2022-04-04T00:47:14.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:14.566+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:15.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:15.270+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:15.459 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:15.457+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:15.457876+0000) 2022-04-04T00:47:15.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:15.563+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:16.236 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:16.234+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:16.459 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:16.457+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:16.458107+0000) 2022-04-04T00:47:16.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:16.522+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:17.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:17.242+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:17.459 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:17.457+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:17.458276+0000) 2022-04-04T00:47:17.476 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:17.474+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:18.241 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:18.238+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:18.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:18.432+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:18.459 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:18.458+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:18.458492+0000) 2022-04-04T00:47:19.282 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:19.280+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:19.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:19.397+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:19.460 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:19.458+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:19.458736+0000) 2022-04-04T00:47:20.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:20.324+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:20.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:20.432+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:20.460 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:20.458+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:20.458974+0000) 2022-04-04T00:47:21.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:21.318+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:21.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:21.415+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:21.460 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:21.458+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:21.459118+0000) 2022-04-04T00:47:22.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:22.309+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:22.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:22.379+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:22.460 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:22.459+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:22.459336+0000) 2022-04-04T00:47:23.303 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:23.301+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:23.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:23.336+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:23.460 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:23.459+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:23.459556+0000) 2022-04-04T00:47:24.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:24.312+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:24.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:24.377+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:24.461 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:24.459+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:24.459791+0000) 2022-04-04T00:47:25.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:25.300+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:47:25.378 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:25.376+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:25.461 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:25.459+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:25.459992+0000) 2022-04-04T00:47:26.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:26.270+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:26.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:26.338+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:26.461 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:26.460+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:26.460155+0000) 2022-04-04T00:47:26.586 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:47:26.584+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:47:27.261 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:27.259+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:27.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:27.355+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:27.461 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:27.460+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:27.460388+0000) 2022-04-04T00:47:28.234 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:28.232+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:28.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:28.375+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:28.462 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:28.459+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:28.460602+0000) 2022-04-04T00:47:29.264 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:29.262+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:29.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:29.396+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:29.462 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:29.460+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:29.460866+0000) 2022-04-04T00:47:30.266 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:30.264+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:30.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:30.349+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:30.462 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:30.460+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:30.461086+0000) 2022-04-04T00:47:31.267 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:31.265+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:31.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:31.353+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:31.462 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:31.460+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:31.461320+0000) 2022-04-04T00:47:32.308 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:32.306+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:32.400 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:32.398+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:32.462 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:32.460+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:32.461523+0000) 2022-04-04T00:47:33.354 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:33.352+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:33.437 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:33.435+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:33.463 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:33.460+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:33.461679+0000) 2022-04-04T00:47:34.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:34.387+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:34.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:34.446+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:34.463 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:34.460+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:34.461895+0000) 2022-04-04T00:47:35.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:35.347+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:35.463 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:35.461+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:35.462137+0000) 2022-04-04T00:47:35.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:35.495+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:36.368 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:36.366+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:36.463 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:36.460+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:36.462341+0000) 2022-04-04T00:47:36.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:36.507+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:37.403 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:37.402+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:37.464 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:37.461+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:37.462583+0000) 2022-04-04T00:47:37.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:37.519+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:38.148 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:47:38.152 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:47:38.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:38.367+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:38.464 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:38.461+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:38.462886+0000) 2022-04-04T00:47:38.552 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:38.551+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:39.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:39.376+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:39.464 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:39.462+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:39.463131+0000) 2022-04-04T00:47:39.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:39.522+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:40.400 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:40.397+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:40.464 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:40.462+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:40.463376+0000) 2022-04-04T00:47:40.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:40.551+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:41.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:41.358+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:41.464 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:41.462+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:41.463565+0000) 2022-04-04T00:47:41.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:41.576+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:41.602 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:47:41.600+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:47:42.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:42.403+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:42.465 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:42.463+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:42.463825+0000) 2022-04-04T00:47:42.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:42.625+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:43.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:43.387+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:43.465 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:43.463+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:43.464060+0000) 2022-04-04T00:47:43.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:43.657+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:44.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:44.374+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:44.465 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:44.463+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:44.464271+0000) 2022-04-04T00:47:44.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:44.705+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:45.399 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:45.397+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:45.465 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:45.463+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:45.464454+0000) 2022-04-04T00:47:45.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:45.723+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:46.426 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:46.423+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:46.466 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:46.463+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:46.464601+0000) 2022-04-04T00:47:46.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:46.685+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:47.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:47.434+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:47.466 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:47.464+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:47.464750+0000) 2022-04-04T00:47:47.666 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:47.663+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:48.424 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:48.421+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:48.466 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:48.464+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:48.464899+0000) 2022-04-04T00:47:48.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:48.642+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:49.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:49.387+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:49.466 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:49.464+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:49.465061+0000) 2022-04-04T00:47:49.633 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:49.630+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:50.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:50.434+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:50.466 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:50.464+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:50.465234+0000) 2022-04-04T00:47:50.664 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:50.661+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:51.432 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:51.429+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:51.466 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:51.464+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:51.465408+0000) 2022-04-04T00:47:51.676 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:51.674+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:52.422 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:52.419+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:52.466 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:52.464+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:52.465597+0000) 2022-04-04T00:47:52.661 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:52.659+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:53.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:53.377+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:53.467 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:53.465+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:53.465857+0000) 2022-04-04T00:47:53.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:53.643+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:54.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:54.411+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:54.467 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:54.465+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:54.466033+0000) 2022-04-04T00:47:54.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:54.625+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:55.395 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:55.393+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:55.467 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:55.465+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:55.466232+0000) 2022-04-04T00:47:55.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:55.614+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:56.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:56.371+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:56.467 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:56.465+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:56.466476+0000) 2022-04-04T00:47:56.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:56.566+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:56.606 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:47:56.604+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:47:57.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:57.419+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:57.467 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:57.466+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:57.466707+0000) 2022-04-04T00:47:57.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:57.608+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:58.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:58.409+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:58.468 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:58.466+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:58.466893+0000) 2022-04-04T00:47:58.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:58.613+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:47:59.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:47:59.374+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:47:59.468 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:47:59.466+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:47:59.467076+0000) 2022-04-04T00:47:59.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:47:59.623+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:00.372 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:00.370+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:00.468 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:00.466+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:00.467293+0000) 2022-04-04T00:48:00.606 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:00.603+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:01.382 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:01.379+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:01.469 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:01.466+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:01.467561+0000) 2022-04-04T00:48:01.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:01.577+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:02.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:02.377+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:02.469 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:02.467+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:02.467835+0000) 2022-04-04T00:48:02.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:02.594+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:03.354 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:03.351+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:03.469 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:03.467+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:03.468061+0000) 2022-04-04T00:48:03.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:03.548+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:04.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:04.373+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:04.469 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:04.467+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:04.468224+0000) 2022-04-04T00:48:04.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:04.597+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:05.329 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:05.326+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:05.469 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:05.468+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:05.468354+0000) 2022-04-04T00:48:05.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:05.551+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:06.346 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:06.343+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:06.469 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:06.468+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:06.468491+0000) 2022-04-04T00:48:06.523 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:06.521+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:07.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:07.378+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:07.470 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:07.468+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:07.468733+0000) 2022-04-04T00:48:07.479 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:07.476+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:08.270 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:48:08.275 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:48:08.338 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:08.336+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:08.462 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:08.459+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:08.470 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:08.468+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:08.469039+0000) 2022-04-04T00:48:09.348 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:09.346+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:09.470 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:09.468+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:09.469191+0000) 2022-04-04T00:48:09.498 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:09.496+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:10.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:10.391+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:10.470 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:10.469+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:10.469375+0000) 2022-04-04T00:48:10.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:10.486+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:11.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:11.427+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:11.470 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:11.469+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:11.469566+0000) 2022-04-04T00:48:11.490 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:11.488+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:11.622 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:48:11.620+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:48:12.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:12.389+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:12.471 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:12.469+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:12.469812+0000) 2022-04-04T00:48:12.477 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:12.474+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:13.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:13.386+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:13.471 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:13.468+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:13.471 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:13.469+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:13.470024+0000) 2022-04-04T00:48:14.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:14.391+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:14.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:14.445+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:14.471 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:14.470+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:14.470142+0000) 2022-04-04T00:48:15.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:15.413+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:15.421 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:15.419+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:15.471 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:15.469+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:15.470295+0000) 2022-04-04T00:48:16.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:16.378+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:16.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:16.430+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:16.471 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:16.468+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:16.470441+0000) 2022-04-04T00:48:17.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:17.359+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:17.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:17.397+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:17.472 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:17.469+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:17.470651+0000) 2022-04-04T00:48:18.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:18.339+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:18.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:18.369+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:18.472 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:18.469+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:18.470783+0000) 2022-04-04T00:48:19.387 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:19.385+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:19.409 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:19.408+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:19.472 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:19.470+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:19.471014+0000) 2022-04-04T00:48:20.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:20.372+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:20.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:20.430+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:20.472 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:20.470+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:20.471163+0000) 2022-04-04T00:48:21.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:21.417+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:21.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:21.419+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:21.472 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:21.470+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:21.471336+0000) 2022-04-04T00:48:22.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:22.386+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:22.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:22.391+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:22.472 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:22.470+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:22.471524+0000) 2022-04-04T00:48:23.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:23.348+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:23.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:23.437+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:23.472 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:23.470+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:23.471720+0000) 2022-04-04T00:48:24.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:24.306+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:24.446 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:24.445+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:24.473 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:24.470+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:24.471891+0000) 2022-04-04T00:48:25.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:25.316+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:25.422 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:25.420+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:25.473 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:25.471+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:25.472073+0000) 2022-04-04T00:48:26.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:26.336+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:26.436 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:26.434+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:26.473 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:26.471+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:26.472321+0000) 2022-04-04T00:48:26.638 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:48:26.636+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:48:27.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:27.379+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:27.426 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:27.424+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:27.473 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:27.471+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:27.472539+0000) 2022-04-04T00:48:28.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:28.384+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:28.457 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:28.455+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:28.474 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:28.472+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:28.472797+0000) 2022-04-04T00:48:29.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:29.367+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:29.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:29.416+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:29.474 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:29.472+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:29.472995+0000) 2022-04-04T00:48:30.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:30.324+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:30.422 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:30.421+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:30.474 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:30.472+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:30.473170+0000) 2022-04-04T00:48:31.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:31.338+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:31.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:31.442+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:31.508 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:31.472+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:31.473371+0000) 2022-04-04T00:48:32.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:32.309+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:32.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:32.460+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:32.474 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:32.472+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:32.473469+0000) 2022-04-04T00:48:33.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:33.263+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:33.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:33.412+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:33.475 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:33.472+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:33.473676+0000) 2022-04-04T00:48:34.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:34.298+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:34.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:34.387+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:34.475 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:34.473+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:34.473887+0000) 2022-04-04T00:48:35.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:35.337+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:35.345 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:35.342+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:35.475 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:35.473+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:35.474097+0000) 2022-04-04T00:48:36.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:36.336+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:36.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:36.375+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:36.475 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:36.473+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:36.474242+0000) 2022-04-04T00:48:37.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:37.340+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:37.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:37.364+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:37.475 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:37.473+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:37.474353+0000) 2022-04-04T00:48:38.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:38.330+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:38.384 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:38.382+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:38.475 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:38.473+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:38.474474+0000) 2022-04-04T00:48:38.500 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:48:38.504 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:48:39.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:39.347+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:39.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:39.402+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:39.476 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:39.474+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:39.474638+0000) 2022-04-04T00:48:40.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:40.333+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:40.424 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:40.422+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:40.476 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:40.474+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:40.474855+0000) 2022-04-04T00:48:41.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:41.303+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:41.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:41.382+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:41.476 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:41.474+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:41.475026+0000) 2022-04-04T00:48:41.654 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:48:41.651+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:48:42.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:42.287+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:42.355 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:42.352+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:42.476 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:42.474+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:42.475246+0000) 2022-04-04T00:48:43.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:43.299+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:43.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:43.359+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:43.476 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:43.474+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:43.475424+0000) 2022-04-04T00:48:44.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:44.289+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:44.366 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:44.363+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:44.476 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:44.475+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:44.475560+0000) 2022-04-04T00:48:45.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:45.327+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:45.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:45.387+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:45.477 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:45.475+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:45.475814+0000) 2022-04-04T00:48:46.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:46.322+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:46.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:46.427+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:46.477 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:46.475+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:46.476051+0000) 2022-04-04T00:48:47.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:47.357+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:47.468 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:47.466+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:47.477 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:47.475+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:47.476195+0000) 2022-04-04T00:48:48.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:48.317+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:48.422 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:48.419+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:48.477 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:48.476+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:48.476342+0000) 2022-04-04T00:48:49.284 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:49.281+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:49.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:49.388+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:49.478 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:49.476+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:49.476505+0000) 2022-04-04T00:48:50.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:50.247+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:50.355 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:50.353+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:50.478 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:50.476+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:50.476673+0000) 2022-04-04T00:48:51.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:51.209+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:51.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:51.376+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:51.478 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:51.476+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:51.476887+0000) 2022-04-04T00:48:52.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:52.235+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:52.424 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:52.422+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:52.478 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:52.476+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:52.477094+0000) 2022-04-04T00:48:53.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:53.259+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:53.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:53.411+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:53.478 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:53.476+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:53.477213+0000) 2022-04-04T00:48:54.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:54.271+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:54.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:54.404+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:54.478 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:54.477+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:54.477331+0000) 2022-04-04T00:48:55.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:55.302+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:55.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:55.412+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:55.478 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:55.477+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:55.477518+0000) 2022-04-04T00:48:56.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:56.339+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:56.396 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:56.394+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:56.479 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:56.477+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:56.477731+0000) 2022-04-04T00:48:56.669 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:48:56.667+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:48:57.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:57.328+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:57.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:57.388+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:57.479 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:57.477+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:57.477941+0000) 2022-04-04T00:48:58.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:58.293+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:58.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:58.370+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:58.479 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:58.477+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:58.478041+0000) 2022-04-04T00:48:59.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:48:59.312+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:48:59.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:48:59.415+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:48:59.479 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:48:59.477+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:48:59.478227+0000) 2022-04-04T00:49:00.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:00.348+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:00.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:00.432+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:00.479 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:00.477+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:00.478341+0000) 2022-04-04T00:49:01.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:01.348+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:01.402 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:01.399+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:01.479 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:01.477+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:01.478524+0000) 2022-04-04T00:49:02.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:02.370+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:02.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:02.387+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:02.480 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:02.477+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:02.478742+0000) 2022-04-04T00:49:03.350 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:03.347+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:03.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:03.397+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:03.480 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:03.477+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:03.479013+0000) 2022-04-04T00:49:04.362 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:04.359+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:04.434 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:04.432+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:04.480 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:04.478+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:04.479241+0000) 2022-04-04T00:49:05.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:05.350+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:05.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:05.430+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:05.481 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:05.478+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:05.479486+0000) 2022-04-04T00:49:06.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:06.380+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:06.401 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:06.398+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:06.480 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:06.478+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:06.479631+0000) 2022-04-04T00:49:07.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:07.411+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:07.434 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:07.431+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:07.481 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:07.478+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:07.479819+0000) 2022-04-04T00:49:08.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:08.388+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:08.473 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:08.470+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:08.481 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:08.479+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:08.479982+0000) 2022-04-04T00:49:08.580 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:49:08.584 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:49:09.346 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:09.344+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:09.481 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:09.479+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:09.480120+0000) 2022-04-04T00:49:09.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:09.491+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:10.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:10.374+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:10.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:10.459+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:10.481 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:10.479+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:10.480238+0000) 2022-04-04T00:49:11.348 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:11.345+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:11.432 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:11.430+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:11.481 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:11.479+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:11.480442+0000) 2022-04-04T00:49:11.684 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:49:11.682+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:49:12.322 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:12.320+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:12.388 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:12.386+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:12.482 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:12.479+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:12.480680+0000) 2022-04-04T00:49:13.312 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:13.311+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:13.407 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:13.405+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:13.482 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:13.480+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:13.480892+0000) 2022-04-04T00:49:14.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:14.275+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:14.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:14.359+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:14.482 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:14.480+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:14.481055+0000) 2022-04-04T00:49:15.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:15.311+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:15.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:15.322+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:15.482 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:15.480+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:15.481246+0000) 2022-04-04T00:49:16.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:16.284+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:16.312 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:16.311+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:16.482 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:16.480+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:16.481392+0000) 2022-04-04T00:49:17.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:17.298+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:17.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:17.316+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:17.483 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:17.480+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:17.481607+0000) 2022-04-04T00:49:18.298 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:18.296+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:18.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:18.296+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:18.483 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:18.481+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:18.481818+0000) 2022-04-04T00:49:19.266 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:19.265+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:19.271 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:19.269+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:19.483 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:19.481+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:19.482053+0000) 2022-04-04T00:49:20.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:20.300+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:20.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:20.305+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:20.483 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:20.481+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:20.482261+0000) 2022-04-04T00:49:21.332 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:21.331+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:21.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:21.348+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:21.483 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:21.481+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:21.482409+0000) 2022-04-04T00:49:22.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:22.360+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:22.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:22.378+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:22.483 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:22.482+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:22.482626+0000) 2022-04-04T00:49:23.334 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:23.332+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:23.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:23.369+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:23.484 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:23.482+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:23.482891+0000) 2022-04-04T00:49:24.338 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:24.336+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:24.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:24.343+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:24.484 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:24.482+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:24.483159+0000) 2022-04-04T00:49:25.366 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:25.364+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:25.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:25.371+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:25.484 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:25.482+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:25.483335+0000) 2022-04-04T00:49:26.382 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:26.381+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:26.400 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:26.398+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:26.484 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:26.483+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:26.483506+0000) 2022-04-04T00:49:26.700 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:49:26.698+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:49:27.378 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:27.376+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:27.426 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:27.424+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:27.485 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:27.483+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:27.483692+0000) 2022-04-04T00:49:28.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:28.396+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:28.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:28.406+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:28.485 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:28.483+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:28.483834+0000) 2022-04-04T00:49:29.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:29.352+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:29.390 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:29.387+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:29.485 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:29.483+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:29.484033+0000) 2022-04-04T00:49:30.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:30.325+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:30.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:30.380+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:30.485 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:30.483+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:30.484198+0000) 2022-04-04T00:49:31.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:31.348+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:31.387 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:31.384+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:31.485 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:31.483+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:31.484397+0000) 2022-04-04T00:49:32.360 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:32.358+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:32.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:32.379+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:32.485 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:32.484+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:32.484567+0000) 2022-04-04T00:49:33.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:33.333+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:33.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:33.359+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:33.486 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:33.484+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:33.484715+0000) 2022-04-04T00:49:34.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:34.367+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:34.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:34.374+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:34.486 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:34.484+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:34.484897+0000) 2022-04-04T00:49:35.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:35.328+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:35.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:35.415+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:35.486 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:35.484+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:35.485068+0000) 2022-04-04T00:49:36.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:36.327+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:36.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:36.416+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:36.486 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:36.484+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:36.485149+0000) 2022-04-04T00:49:37.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:37.311+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:37.421 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:37.418+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:37.486 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:37.485+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:37.485320+0000) 2022-04-04T00:49:38.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:38.292+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:38.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:38.416+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:38.487 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:38.485+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:38.485548+0000) 2022-04-04T00:49:38.746 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:49:38.750 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:49:39.252 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:39.249+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:39.374 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:39.372+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:39.487 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:39.485+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:39.485810+0000) 2022-04-04T00:49:40.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:40.272+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:40.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:40.392+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:40.487 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:40.485+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:40.485994+0000) 2022-04-04T00:49:41.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:41.247+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:41.346 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:41.343+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:41.487 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:41.486+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:41.486194+0000) 2022-04-04T00:49:41.716 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:49:41.714+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:49:42.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:42.248+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:42.326 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:42.324+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:42.487 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:42.486+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:42.486413+0000) 2022-04-04T00:49:43.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:43.245+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:43.298 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:43.295+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:43.488 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:43.486+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:43.486626+0000) 2022-04-04T00:49:44.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:44.258+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:44.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:44.286+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:44.488 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:44.485+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:44.486779+0000) 2022-04-04T00:49:45.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:45.216+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:45.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:45.326+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:45.488 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:45.485+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:45.486995+0000) 2022-04-04T00:49:46.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:46.216+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:46.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:46.297+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:46.488 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:46.486+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:46.487206+0000) 2022-04-04T00:49:47.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:47.180+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:47.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:47.272+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:47.488 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:47.486+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:47.487421+0000) 2022-04-04T00:49:48.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:48.151+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:48.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:48.287+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:48.489 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:48.486+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:48.487622+0000) 2022-04-04T00:49:49.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:49.123+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:49.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:49.286+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:49.489 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:49.486+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:49.487857+0000) 2022-04-04T00:49:50.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:50.110+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:50.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:50.299+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:50.489 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:50.487+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:50.488025+0000) 2022-04-04T00:49:51.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:51.129+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:51.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:51.344+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:51.489 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:51.487+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:51.488165+0000) 2022-04-04T00:49:52.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:52.123+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:52.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:52.372+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:52.489 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:52.487+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:52.488261+0000) 2022-04-04T00:49:53.115 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:53.113+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:53.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:53.325+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:53.490 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:53.487+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:53.488452+0000) 2022-04-04T00:49:54.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:54.125+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:54.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:54.316+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:54.490 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:54.487+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:54.488595+0000) 2022-04-04T00:49:55.120 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:55.118+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:55.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:55.351+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:55.490 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:55.488+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:55.488825+0000) 2022-04-04T00:49:56.114 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:56.112+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:56.376 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:56.373+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:56.490 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:56.488+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:56.489037+0000) 2022-04-04T00:49:56.732 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:49:56.730+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:49:57.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:57.074+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:57.396 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:57.394+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:57.490 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:57.488+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:57.489232+0000) 2022-04-04T00:49:58.040 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:58.038+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:58.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:58.388+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:58.490 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:58.488+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:58.489380+0000) 2022-04-04T00:49:59.036 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:49:59.034+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:49:59.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:49:59.340+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:49:59.490 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:59.488+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:59.489568+0000) 2022-04-04T00:49:59.491 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:49:59.489+0000 7fd62421b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:49:59.490629+0000) 2022-04-04T00:50:00.057 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:00.054+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:00.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:00.372+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:00.491 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:00.488+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:00.489699+0000) 2022-04-04T00:50:01.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:01.069+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:01.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:01.340+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:01.491 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:01.489+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:01.489832+0000) 2022-04-04T00:50:02.060 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:02.058+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:02.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:02.338+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:02.491 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:02.489+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:02.490037+0000) 2022-04-04T00:50:03.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:03.064+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:03.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:03.328+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:03.491 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:03.489+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:03.490185+0000) 2022-04-04T00:50:04.069 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:04.067+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:04.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:04.325+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:04.491 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:04.489+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:04.490334+0000) 2022-04-04T00:50:05.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:05.064+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:05.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:05.280+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:05.492 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:05.489+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:05.490488+0000) 2022-04-04T00:50:06.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:06.036+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:06.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:06.270+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:06.492 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:06.490+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:06.490639+0000) 2022-04-04T00:50:06.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:06.995+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:07.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:07.258+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:07.492 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:07.490+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:07.490808+0000) 2022-04-04T00:50:08.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:08.036+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:08.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:08.251+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:08.492 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:08.490+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:08.490982+0000) 2022-04-04T00:50:08.818 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:50:08.822 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:50:09.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:09.003+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:09.206 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:09.205+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:09.492 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:09.490+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:09.491158+0000) 2022-04-04T00:50:09.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:09.983+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:10.174 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:10.172+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:10.492 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:10.490+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:10.491298+0000) 2022-04-04T00:50:10.969 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:10.967+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:11.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:11.135+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:11.492 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:11.491+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:11.491442+0000) 2022-04-04T00:50:11.748 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:50:11.746+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:50:11.925 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:11.923+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:12.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:12.151+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:12.493 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:12.491+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:12.491584+0000) 2022-04-04T00:50:12.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:12.939+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:13.119 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:13.117+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:13.493 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:13.491+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:13.491753+0000) 2022-04-04T00:50:13.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:13.980+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:14.133 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:14.131+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:14.493 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:14.491+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:14.491928+0000) 2022-04-04T00:50:14.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:14.975+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:15.169 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:15.167+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:15.493 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:15.491+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:15.492073+0000) 2022-04-04T00:50:16.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:16.011+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:16.169 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:16.168+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:16.493 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:16.491+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:16.492213+0000) 2022-04-04T00:50:16.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:16.976+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:17.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:17.202+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:17.493 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:17.492+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:17.492337+0000) 2022-04-04T00:50:17.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:17.982+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:18.217 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:18.216+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:18.494 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:18.492+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:18.492486+0000) 2022-04-04T00:50:19.031 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:19.029+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:19.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:19.217+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:19.494 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:19.492+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:19.492667+0000) 2022-04-04T00:50:19.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:19.996+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:20.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:20.233+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:20.494 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:20.492+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:20.492846+0000) 2022-04-04T00:50:21.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:21.007+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:21.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:21.214+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:21.494 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:21.492+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:21.492994+0000) 2022-04-04T00:50:22.058 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:22.057+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:22.171 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:22.170+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:22.494 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:22.493+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:22.493182+0000) 2022-04-04T00:50:23.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:23.105+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:23.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:23.165+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:23.494 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:23.492+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:23.493318+0000) 2022-04-04T00:50:24.123 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:24.120+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:24.140 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:24.138+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:24.495 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:24.493+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:24.493501+0000) 2022-04-04T00:50:25.128 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:25.125+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:25.177 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:25.175+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T00:50:25.495 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:25.493+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:25.493680+0000) 2022-04-04T00:50:26.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:26.172+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:26.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:26.175+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:26.495 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:26.493+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:26.493850+0000) 2022-04-04T00:50:26.758 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:50:26.756+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:50:27.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:27.170+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:27.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:27.220+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:27.495 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:27.493+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:27.494002+0000) 2022-04-04T00:50:28.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:28.122+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:28.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:28.212+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:28.495 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:28.494+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:28.494102+0000) 2022-04-04T00:50:29.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:29.104+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:29.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:29.206+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:29.495 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:29.492+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:29.494244+0000) 2022-04-04T00:50:30.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:30.109+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:30.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:30.166+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:30.495 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:30.493+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:30.494368+0000) 2022-04-04T00:50:31.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:31.082+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:31.138 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:31.136+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:31.495 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:31.492+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:31.494568+0000) 2022-04-04T00:50:32.109 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:32.106+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:32.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:32.179+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:32.496 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:32.493+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:32.494749+0000) 2022-04-04T00:50:33.148 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:33.145+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:33.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:33.216+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:33.496 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:33.493+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:33.494930+0000) 2022-04-04T00:50:34.116 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:34.113+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:34.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:34.194+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:34.496 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:34.494+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:34.495105+0000) 2022-04-04T00:50:35.149 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:35.147+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:35.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:35.154+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:35.496 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:35.494+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:35.495283+0000) 2022-04-04T00:50:36.114 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:36.112+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:36.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:36.165+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:36.496 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:36.494+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:36.495446+0000) 2022-04-04T00:50:37.138 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:37.135+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:37.195 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:37.193+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:37.496 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:37.494+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:37.495569+0000) 2022-04-04T00:50:38.090 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:38.088+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:38.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:38.176+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:38.497 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:38.494+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:38.495660+0000) 2022-04-04T00:50:38.981 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:50:38.985 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:50:39.133 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:39.130+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:39.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:39.150+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:39.497 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:39.495+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:39.495818+0000) 2022-04-04T00:50:40.106 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:40.103+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:40.140 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:40.138+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:40.497 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:40.495+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:40.495982+0000) 2022-04-04T00:50:41.076 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:41.074+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:41.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:41.133+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:41.497 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:41.495+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:41.496090+0000) 2022-04-04T00:50:41.774 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:50:41.772+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:50:42.061 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:42.059+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:42.096 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:42.093+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:42.497 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:42.495+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:42.496311+0000) 2022-04-04T00:50:43.027 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:43.025+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:43.119 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:43.117+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:43.497 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:43.495+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:43.496464+0000) 2022-04-04T00:50:44.020 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:44.017+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:44.070 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:44.068+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:44.498 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:44.495+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:44.496613+0000) 2022-04-04T00:50:45.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:45.036+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:45.044 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:45.041+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:45.498 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:45.496+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:45.496803+0000) 2022-04-04T00:50:46.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:46.006+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:46.061 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:46.059+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:46.498 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:46.496+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:46.497012+0000) 2022-04-04T00:50:46.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:46.996+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:47.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:47.070+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:47.498 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:47.496+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:47.497180+0000) 2022-04-04T00:50:47.986 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:47.984+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:48.118 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:48.115+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:48.498 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:48.496+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:48.497332+0000) 2022-04-04T00:50:49.020 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:49.018+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:49.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:49.135+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:49.498 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:49.496+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:49.497480+0000) 2022-04-04T00:50:50.042 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:50.039+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:50.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:50.174+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:50.499 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:50.497+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:50.497630+0000) 2022-04-04T00:50:51.091 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:51.088+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:51.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:51.195+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:51.499 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:51.497+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:51.497814+0000) 2022-04-04T00:50:52.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:52.132+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:52.154 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:52.152+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:52.499 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:52.497+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:52.498000+0000) 2022-04-04T00:50:53.106 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:53.103+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:53.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:53.173+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:53.499 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:53.497+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:53.498181+0000) 2022-04-04T00:50:54.107 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:54.105+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:54.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:54.208+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:54.499 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:54.497+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:54.498350+0000) 2022-04-04T00:50:55.104 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:55.101+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:55.204 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:55.202+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:55.499 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:55.498+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:55.498534+0000) 2022-04-04T00:50:56.109 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:56.106+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:56.247 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:56.245+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:56.500 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:56.498+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:56.498714+0000) 2022-04-04T00:50:56.790 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:50:56.788+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:50:57.133 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:57.131+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:57.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:57.275+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:57.500 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:57.498+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:57.498917+0000) 2022-04-04T00:50:58.100 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:58.098+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:58.271 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:58.269+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:58.500 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:58.498+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:58.499100+0000) 2022-04-04T00:50:59.120 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:50:59.118+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:50:59.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:50:59.277+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:50:59.500 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:50:59.498+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:50:59.499218+0000) 2022-04-04T00:51:00.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:00.162+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:00.233 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:00.231+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:00.500 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:00.499+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:00.499372+0000) 2022-04-04T00:51:01.204 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:01.202+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:01.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:01.208+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:01.501 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:01.499+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:01.499555+0000) 2022-04-04T00:51:02.163 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:02.161+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:02.217 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:02.215+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:02.501 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:02.499+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:02.499740+0000) 2022-04-04T00:51:03.194 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:03.192+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:03.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:03.260+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:03.501 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:03.499+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:03.499949+0000) 2022-04-04T00:51:04.204 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:04.202+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:04.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:04.256+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:04.501 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:04.499+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:04.500136+0000) 2022-04-04T00:51:05.192 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:05.191+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:05.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:05.213+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:05.501 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:05.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:05.500315+0000) 2022-04-04T00:51:06.216 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:06.214+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:06.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:06.243+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:06.501 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:06.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:06.500431+0000) 2022-04-04T00:51:07.202 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:07.200+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:07.220 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:07.219+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:07.502 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:07.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:07.500598+0000) 2022-04-04T00:51:08.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:08.242+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:08.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:08.242+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:08.502 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:08.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:08.500748+0000) 2022-04-04T00:51:09.061 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:51:09.065 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:51:09.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:09.241+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:09.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:09.283+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:09.502 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:09.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:09.500959+0000) 2022-04-04T00:51:10.253 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:10.251+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:10.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:10.254+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:10.502 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:10.501+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:10.501122+0000) 2022-04-04T00:51:11.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:11.261+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:11.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:11.275+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:11.502 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:11.501+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:11.501298+0000) 2022-04-04T00:51:11.806 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:51:11.804+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:51:12.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:12.246+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:12.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:12.280+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:12.502 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:12.501+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:12.501396+0000) 2022-04-04T00:51:13.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:13.276+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:13.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:13.290+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:13.502 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:13.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:13.501496+0000) 2022-04-04T00:51:14.242 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:14.240+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:14.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:14.328+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:14.503 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:14.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:14.501625+0000) 2022-04-04T00:51:15.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:15.278+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:15.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:15.307+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:15.503 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:15.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:15.501794+0000) 2022-04-04T00:51:16.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:16.266+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:16.289 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:16.288+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:16.503 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:16.500+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:16.501967+0000) 2022-04-04T00:51:17.233 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:17.231+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:17.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:17.252+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:17.503 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:17.501+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:17.502145+0000) 2022-04-04T00:51:18.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:18.235+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:18.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:18.246+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:18.503 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:18.501+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:18.502301+0000) 2022-04-04T00:51:19.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:19.270+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:19.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:19.284+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:19.504 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:19.501+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:19.502418+0000) 2022-04-04T00:51:20.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:20.272+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:20.294 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:20.292+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:20.503 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:20.501+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:20.502530+0000) 2022-04-04T00:51:21.315 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:21.312+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:21.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:21.321+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:21.504 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:21.501+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:21.502690+0000) 2022-04-04T00:51:22.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:22.294+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:22.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:22.320+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:22.504 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:22.502+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:22.502846+0000) 2022-04-04T00:51:23.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:23.313+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:23.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:23.359+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:23.504 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:23.502+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:23.502993+0000) 2022-04-04T00:51:24.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:24.348+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:24.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:24.349+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:24.504 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:24.502+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:24.503175+0000) 2022-04-04T00:51:25.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:25.308+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:25.346 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:25.343+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:25.504 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:25.502+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:25.503293+0000) 2022-04-04T00:51:26.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:26.302+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:26.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:26.328+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:26.504 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:26.502+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:26.503446+0000) 2022-04-04T00:51:26.821 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:51:26.819+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:51:27.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:27.314+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:27.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:27.376+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:27.505 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:27.502+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:27.503567+0000) 2022-04-04T00:51:28.312 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:28.309+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:28.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:28.389+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:28.505 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:28.502+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:28.503671+0000) 2022-04-04T00:51:29.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:29.281+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:29.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:29.390+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:29.505 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:29.503+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:29.503799+0000) 2022-04-04T00:51:30.287 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:30.284+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:30.356 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:30.353+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:30.505 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:30.503+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:30.503922+0000) 2022-04-04T00:51:31.303 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:31.300+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:31.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:31.390+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:31.505 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:31.503+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:31.504091+0000) 2022-04-04T00:51:32.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:32.274+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:32.418 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:32.416+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:32.505 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:32.503+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:32.504260+0000) 2022-04-04T00:51:33.263 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:33.261+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:33.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:33.423+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:33.505 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:33.503+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:33.504355+0000) 2022-04-04T00:51:34.270 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:34.267+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:34.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:34.406+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:34.505 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:34.503+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:34.504468+0000) 2022-04-04T00:51:35.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:35.314+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:35.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:35.377+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:35.506 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:35.504+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:35.504627+0000) 2022-04-04T00:51:36.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:36.269+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:36.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:36.378+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:36.506 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:36.504+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:36.504798+0000) 2022-04-04T00:51:37.275 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:37.272+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:37.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:37.331+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:37.506 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:37.504+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:37.504984+0000) 2022-04-04T00:51:38.255 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:38.253+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:38.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:38.344+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:38.506 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:38.504+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:38.505158+0000) 2022-04-04T00:51:39.201 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:51:39.205 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:51:39.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:39.243+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:39.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:39.346+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:39.506 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:39.504+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:39.505324+0000) 2022-04-04T00:51:40.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:40.252+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:40.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:40.374+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:40.507 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:40.505+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:40.505502+0000) 2022-04-04T00:51:41.283 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:41.281+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:41.421 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:41.418+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:41.507 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:41.505+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:41.505650+0000) 2022-04-04T00:51:41.831 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:51:41.829+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:51:42.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:42.320+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:42.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:42.453+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:42.507 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:42.505+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:42.505807+0000) 2022-04-04T00:51:43.331 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:43.328+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:43.484 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:43.482+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:43.507 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:43.505+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:43.505987+0000) 2022-04-04T00:51:44.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:44.360+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:44.507 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:44.505+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:44.506164+0000) 2022-04-04T00:51:44.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:44.516+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:45.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:45.374+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:45.507 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:45.505+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:45.506314+0000) 2022-04-04T00:51:45.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:45.522+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:46.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:46.386+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:46.507 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:46.506+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:46.506442+0000) 2022-04-04T00:51:46.533 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:46.531+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:47.410 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:47.407+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:47.508 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:47.506+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:47.506589+0000) 2022-04-04T00:51:47.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:47.565+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:48.448 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:48.445+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:48.508 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:48.506+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:48.506686+0000) 2022-04-04T00:51:48.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:48.566+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:49.477 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:49.475+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:49.508 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:49.506+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:49.506831+0000) 2022-04-04T00:51:49.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:49.550+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:50.506 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:50.504+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:50.508 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:50.506+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:50.506999+0000) 2022-04-04T00:51:50.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:50.529+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:51.508 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:51.506+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:51.507163+0000) 2022-04-04T00:51:51.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:51.511+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:51.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:51.528+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:52.470 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:52.468+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:52.508 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:52.507+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:52.507275+0000) 2022-04-04T00:51:52.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:52.539+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:53.446 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:53.444+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:53.508 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:53.507+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:53.507471+0000) 2022-04-04T00:51:53.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:53.582+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:54.408 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:54.406+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:54.508 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:54.507+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:54.507609+0000) 2022-04-04T00:51:54.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:54.559+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:55.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:55.428+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:55.509 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:55.507+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:55.507786+0000) 2022-04-04T00:51:55.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:55.512+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:56.403 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:56.401+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:56.509 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:56.507+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:56.507868+0000) 2022-04-04T00:51:56.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:56.532+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:56.847 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:51:56.845+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:51:57.359 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:57.358+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:57.509 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:57.507+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:57.508023+0000) 2022-04-04T00:51:57.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:57.545+0000 7f88d0ac1700 -1 osd.4 137 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:58.101 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-04T00:51:58.099+0000 7f4d73597700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-04T00:51:58.116 INFO:tasks.ceph.mds.f.smithi186.stderr: -1> 2022-04-04T00:51:58.099+0000 7f4d73597700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-04T00:51:58.138 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-04T00:51:58.364 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:58.362+0000 7fd3b190b700 -1 osd.6 137 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:58.509 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:58.507+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:58.508208+0000) 2022-04-04T00:51:58.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:58.512+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:51:59.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:51:59.315+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:51:59.509 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:51:59.507+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:51:59.508413+0000) 2022-04-04T00:51:59.537 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:51:59.535+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:00.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:00.321+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:00.510 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:00.507+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:00.508537+0000) 2022-04-04T00:52:00.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:00.574+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:01.331 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:01.329+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:01.510 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:01.507+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:01.508674+0000) 2022-04-04T00:52:01.600 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:01.599+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:02.283 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:02.281+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:02.510 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:02.507+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:02.508829+0000) 2022-04-04T00:52:02.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:02.612+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 56 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:03.298 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:03.296+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:03.510 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:03.508+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:03.508951+0000) 2022-04-04T00:52:03.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:03.631+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:04.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:04.309+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:04.510 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:04.508+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:04.509128+0000) 2022-04-04T00:52:04.622 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:04.620+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:05.337 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:05.335+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:05.510 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:05.508+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:05.509329+0000) 2022-04-04T00:52:05.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:05.641+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:06.307 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:06.305+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:06.510 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:06.508+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:06.509505+0000) 2022-04-04T00:52:06.656 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:06.655+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:07.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:07.259+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:07.511 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:07.508+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:07.509681+0000) 2022-04-04T00:52:07.661 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:07.657+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:08.252 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:08.250+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:08.511 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:08.508+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:08.509751+0000) 2022-04-04T00:52:08.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:08.651+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:09.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:09.244+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:09.340 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:52:09.343 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:52:09.511 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:09.509+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:09.509921+0000) 2022-04-04T00:52:09.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:09.609+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:10.213 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:10.212+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:10.511 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:10.509+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:10.510126+0000) 2022-04-04T00:52:10.585 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:10.584+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:11.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:11.176+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:11.511 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:11.509+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:11.510285+0000) 2022-04-04T00:52:11.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:11.628+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:11.863 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:52:11.861+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:52:12.209 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:12.207+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:12.511 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:12.509+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:12.510416+0000) 2022-04-04T00:52:12.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:12.650+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:13.225 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:13.222+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:13.512 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:13.509+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:13.510536+0000) 2022-04-04T00:52:13.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:13.639+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:14.191 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:14.188+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:14.512 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:14.509+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:14.510682+0000) 2022-04-04T00:52:14.655 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:14.652+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:15.184 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:15.182+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:15.512 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:15.510+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:15.510849+0000) 2022-04-04T00:52:15.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:15.675+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:16.168 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:16.165+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:16.512 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:16.510+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:16.511030+0000) 2022-04-04T00:52:16.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:16.720+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:17.212 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:17.209+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:17.512 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:17.510+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:17.511216+0000) 2022-04-04T00:52:17.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:17.681+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:18.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:18.223+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:18.512 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:18.510+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:18.511309+0000) 2022-04-04T00:52:18.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:18.676+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:19.176 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:19.174+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:19.512 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:19.510+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:19.511483+0000) 2022-04-04T00:52:19.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:19.716+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:20.190 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:20.187+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:20.513 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:20.511+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:20.511550+0000) 2022-04-04T00:52:20.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:20.739+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:21.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:21.226+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:21.513 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:21.511+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:21.511678+0000) 2022-04-04T00:52:21.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:21.711+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:22.222 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:22.220+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:22.513 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:22.511+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:22.511826+0000) 2022-04-04T00:52:22.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:22.737+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:23.230 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:23.227+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:23.513 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:23.511+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:23.511941+0000) 2022-04-04T00:52:23.693 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:23.690+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:24.220 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:24.217+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:24.513 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:24.511+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:24.512071+0000) 2022-04-04T00:52:24.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:24.724+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:25.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:25.232+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:25.513 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:25.511+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:25.512174+0000) 2022-04-04T00:52:25.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:25.731+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:26.271 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:26.268+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:26.513 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:26.511+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:26.512296+0000) 2022-04-04T00:52:26.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:26.755+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:26.879 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:52:26.877+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:52:27.301 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:27.299+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:27.513 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:27.512+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:27.512415+0000) 2022-04-04T00:52:27.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:27.711+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:28.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:28.265+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:28.514 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:28.512+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:28.512597+0000) 2022-04-04T00:52:28.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:28.754+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:29.286 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:29.284+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:29.514 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:29.512+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:29.512778+0000) 2022-04-04T00:52:29.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:29.710+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:30.280 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:30.278+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:30.514 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:30.512+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:30.512957+0000) 2022-04-04T00:52:30.735 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:30.732+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:31.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:31.293+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:31.514 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:31.512+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:31.513108+0000) 2022-04-04T00:52:31.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:31.728+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:32.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:32.337+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:32.514 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:32.512+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:32.513270+0000) 2022-04-04T00:52:32.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:32.735+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:33.365 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:33.363+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:33.514 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:33.513+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:33.513347+0000) 2022-04-04T00:52:33.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:33.730+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:34.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:34.360+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:34.515 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:34.512+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:34.513519+0000) 2022-04-04T00:52:34.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:34.729+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:35.392 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:35.389+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:35.515 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:35.513+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:35.513667+0000) 2022-04-04T00:52:35.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:35.755+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:36.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:36.402+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:36.515 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:36.513+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:36.513826+0000) 2022-04-04T00:52:36.735 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:36.732+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:37.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:37.421+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:37.519 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:37.513+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:37.513944+0000) 2022-04-04T00:52:37.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:37.690+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:38.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:38.455+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:38.517 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:38.513+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:38.514094+0000) 2022-04-04T00:52:38.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:38.649+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:39.420 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:52:39.424 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:52:39.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:39.484+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:39.515 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:39.513+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:39.514280+0000) 2022-04-04T00:52:39.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:39.622+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:40.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:40.437+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:40.515 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:40.514+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:40.514373+0000) 2022-04-04T00:52:40.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:40.573+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:41.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:41.457+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:41.515 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:41.514+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:41.514500+0000) 2022-04-04T00:52:41.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:41.538+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:41.895 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:52:41.892+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:52:42.492 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:42.490+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:42.515 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:42.513+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:42.514640+0000) 2022-04-04T00:52:42.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:42.586+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:43.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:43.481+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:43.516 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:43.513+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:43.514755+0000) 2022-04-04T00:52:43.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:43.574+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:44.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:44.455+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:44.516 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:44.513+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:44.514878+0000) 2022-04-04T00:52:44.540 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:44.537+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:45.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:45.484+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:45.516 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:45.514+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:45.515041+0000) 2022-04-04T00:52:45.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:45.559+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:46.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:46.457+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:46.516 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:46.514+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:46.515242+0000) 2022-04-04T00:52:46.583 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:46.580+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:47.446 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:47.443+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:47.516 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:47.514+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:47.515409+0000) 2022-04-04T00:52:47.558 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:47.555+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:48.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:48.478+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:48.516 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:48.514+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:48.515558+0000) 2022-04-04T00:52:48.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:48.540+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:49.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:49.435+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:49.517 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:49.514+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:49.515719+0000) 2022-04-04T00:52:49.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:49.561+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:50.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:50.417+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:50.517 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:50.514+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:50.515814+0000) 2022-04-04T00:52:50.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:50.582+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:51.420 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:51.418+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:51.517 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:51.515+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:51.515934+0000) 2022-04-04T00:52:51.550 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:51.548+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:52.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:52.454+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:52.508 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:52.506+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:52.517 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:52.515+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:52.516088+0000) 2022-04-04T00:52:53.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:53.467+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:53.484 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:53.482+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:53.517 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:53.515+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:53.516183+0000) 2022-04-04T00:52:54.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:54.437+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:54.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:54.510+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:54.517 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:54.515+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:54.516336+0000) 2022-04-04T00:52:55.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:55.425+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:55.517 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:55.515+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:55.516520+0000) 2022-04-04T00:52:55.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:55.526+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:56.472 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:56.470+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:56.518 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:56.515+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:56.516699+0000) 2022-04-04T00:52:56.562 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:56.560+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:56.903 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:52:56.901+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:52:57.468 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:57.466+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:57.518 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:57.516+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:57.516845+0000) 2022-04-04T00:52:57.568 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:57.567+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:58.475 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:58.473+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:58.518 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:58.516+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:58.517024+0000) 2022-04-04T00:52:58.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:58.613+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:52:59.518 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:52:59.516+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:52:59.517296+0000) 2022-04-04T00:52:59.523 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:52:59.522+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:52:59.660 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:52:59.658+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:00.518 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:00.516+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:00.517444+0000) 2022-04-04T00:53:00.528 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:00.526+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:00.615 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:00.614+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:01.519 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:01.516+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:01.517645+0000) 2022-04-04T00:53:01.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:01.559+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:01.609 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:01.608+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:02.519 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:02.517+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:02.517863+0000) 2022-04-04T00:53:02.570 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:02.568+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:02.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:02.621+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:03.519 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:03.517+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:03.518052+0000) 2022-04-04T00:53:03.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:03.579+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:03.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:03.604+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:04.520 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:04.517+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:04.518280+0000) 2022-04-04T00:53:04.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:04.589+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:04.630 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:04.627+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:05.519 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:05.517+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:05.518463+0000) 2022-04-04T00:53:05.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:05.581+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:05.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:05.634+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:06.520 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:06.518+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:06.518671+0000) 2022-04-04T00:53:06.562 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:06.560+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:06.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:06.586+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:07.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:07.516+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:07.520 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:07.518+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:07.518912+0000) 2022-04-04T00:53:07.558 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:07.556+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:08.520 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:08.518+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:08.519045+0000) 2022-04-04T00:53:08.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:08.518+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:08.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:08.546+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:09.487 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:53:09.491 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:53:09.506 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:09.504+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:09.520 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:09.518+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:09.519238+0000) 2022-04-04T00:53:09.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:09.541+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:10.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:10.456+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:10.520 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:10.518+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:10.519482+0000) 2022-04-04T00:53:10.533 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:10.530+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:11.413 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:11.411+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:11.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:11.485+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:11.520 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:11.519+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:11.519658+0000) 2022-04-04T00:53:11.919 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:53:11.917+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:53:12.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:12.368+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:12.521 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:12.519+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:12.519911+0000) 2022-04-04T00:53:12.526 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:12.524+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:13.401 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:13.399+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:13.521 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:13.519+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:13.520113+0000) 2022-04-04T00:53:13.565 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:13.562+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:14.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:14.366+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:14.521 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:14.519+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:14.520340+0000) 2022-04-04T00:53:14.599 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:14.596+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:15.375 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:15.372+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:15.522 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:15.520+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:15.520515+0000) 2022-04-04T00:53:15.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:15.640+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:16.332 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:16.329+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:16.522 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:16.520+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:16.520743+0000) 2022-04-04T00:53:16.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:16.677+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:17.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:17.285+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:17.522 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:17.520+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:17.520990+0000) 2022-04-04T00:53:17.674 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:17.671+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:18.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:18.320+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:18.522 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:18.520+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:18.521198+0000) 2022-04-04T00:53:18.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:18.667+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:19.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:19.295+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:19.522 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:19.521+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:19.521365+0000) 2022-04-04T00:53:19.639 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:19.637+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:20.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:20.338+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:20.523 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:20.520+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:20.521611+0000) 2022-04-04T00:53:20.622 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:20.619+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:21.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:21.317+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:21.523 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:21.521+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:21.521802+0000) 2022-04-04T00:53:21.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:21.638+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:22.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:22.271+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:22.523 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:22.521+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:22.521974+0000) 2022-04-04T00:53:22.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:22.644+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:23.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:23.258+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:23.523 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:23.521+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:23.522146+0000) 2022-04-04T00:53:23.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:23.638+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:24.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:24.285+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:24.523 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:24.521+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:24.522300+0000) 2022-04-04T00:53:24.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:24.662+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:25.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:25.289+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:25.523 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:25.522+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:25.522511+0000) 2022-04-04T00:53:25.624 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:25.622+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:26.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:26.261+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:26.524 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:26.522+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:26.522725+0000) 2022-04-04T00:53:26.660 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:26.657+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:26.935 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:53:26.932+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:53:27.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:27.264+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:27.524 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:27.521+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:27.522870+0000) 2022-04-04T00:53:27.692 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:27.689+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:28.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:28.224+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:28.524 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:28.521+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:28.522972+0000) 2022-04-04T00:53:28.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:28.675+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:29.182 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:29.180+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:29.524 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:29.522+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:29.523146+0000) 2022-04-04T00:53:29.653 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:29.651+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:30.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:30.171+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:30.524 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:30.522+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:30.523314+0000) 2022-04-04T00:53:30.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:30.603+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:31.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:31.197+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:31.524 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:31.522+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:31.523500+0000) 2022-04-04T00:53:31.651 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:31.649+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:32.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:32.177+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:32.525 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:32.522+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:32.523681+0000) 2022-04-04T00:53:32.624 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:32.621+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:33.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:33.186+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:33.525 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:33.522+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:33.523892+0000) 2022-04-04T00:53:33.612 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:33.610+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:34.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:34.178+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:34.525 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:34.523+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:34.524098+0000) 2022-04-04T00:53:34.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:34.621+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:35.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:35.135+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:35.525 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:35.523+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:35.524304+0000) 2022-04-04T00:53:35.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:35.663+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:36.123 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:36.121+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:36.526 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:36.523+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:36.524523+0000) 2022-04-04T00:53:36.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:36.619+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:37.140 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:37.138+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:37.526 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:37.523+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:37.524733+0000) 2022-04-04T00:53:37.622 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:37.620+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:38.134 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:38.131+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:38.526 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:38.524+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:38.524946+0000) 2022-04-04T00:53:38.650 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:38.647+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:39.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:39.083+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:39.526 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:39.524+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:39.525171+0000) 2022-04-04T00:53:39.615 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:53:39.619 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:53:39.630 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:39.627+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:40.045 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:40.043+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:40.526 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:40.524+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:40.525321+0000) 2022-04-04T00:53:40.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:40.605+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:41.060 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:41.057+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:41.527 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:41.524+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:41.525474+0000) 2022-04-04T00:53:41.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:41.579+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:41.936 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:53:41.934+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:53:42.026 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:42.023+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:42.527 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:42.524+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:42.525691+0000) 2022-04-04T00:53:42.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:42.605+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:43.066 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:43.063+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:43.527 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:43.525+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:43.525932+0000) 2022-04-04T00:53:43.599 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:43.597+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:44.026 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:44.024+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:44.527 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:44.525+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:44.526136+0000) 2022-04-04T00:53:44.612 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:44.610+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:44.984 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:44.981+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:45.527 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:45.525+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:45.526367+0000) 2022-04-04T00:53:45.651 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:45.649+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:45.981 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:45.979+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:46.527 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:46.525+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:46.526497+0000) 2022-04-04T00:53:46.678 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:46.676+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:46.955 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:46.953+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:47.528 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:47.526+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:47.526699+0000) 2022-04-04T00:53:47.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:47.708+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:47.972 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:47.970+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:48.528 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:48.526+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:48.526858+0000) 2022-04-04T00:53:48.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:48.753+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:48.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:48.995+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:49.528 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:49.526+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:49.527030+0000) 2022-04-04T00:53:49.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:49.759+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:50.024 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:50.023+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:50.528 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:50.526+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:50.527216+0000) 2022-04-04T00:53:50.730 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:50.728+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:51.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:51.014+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:51.528 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:51.526+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:51.527427+0000) 2022-04-04T00:53:51.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:51.753+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:52.030 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:52.029+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:52.529 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:52.527+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:52.527637+0000) 2022-04-04T00:53:52.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:52.803+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:53.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:53.015+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:53.529 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:53.527+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:53.527815+0000) 2022-04-04T00:53:53.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:53.787+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:53.998 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:53.997+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:54.529 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:54.527+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:54.528024+0000) 2022-04-04T00:53:54.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:54.743+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:55.026 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:55.024+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:55.529 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:55.527+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:55.528230+0000) 2022-04-04T00:53:55.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:55.699+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:56.006 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:56.005+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:56.529 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:56.527+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:56.528372+0000) 2022-04-04T00:53:56.660 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:56.659+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:56.952 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:53:56.950+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:53:56.986 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:56.984+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:57.530 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:57.528+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:57.528546+0000) 2022-04-04T00:53:57.654 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:57.652+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:57.947 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:57.946+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:58.530 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:58.528+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:58.528675+0000) 2022-04-04T00:53:58.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:58.681+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:58.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:58.943+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:53:59.530 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:53:59.528+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:53:59.528885+0000) 2022-04-04T00:53:59.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:53:59.724+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:53:59.954 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:53:59.952+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:00.530 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:00.528+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:00.529087+0000) 2022-04-04T00:54:00.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:00.688+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:00.971 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:00.969+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:01.530 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:01.528+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:01.529294+0000) 2022-04-04T00:54:01.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:01.715+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:02.010 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:02.008+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:02.530 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:02.529+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:02.529504+0000) 2022-04-04T00:54:02.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:02.689+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:02.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:02.976+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:03.531 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:03.528+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:03.529740+0000) 2022-04-04T00:54:03.660 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:03.658+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:03.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:03.993+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:04.531 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:04.529+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:04.529899+0000) 2022-04-04T00:54:04.653 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:04.650+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:04.954 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:04.951+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:05.531 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:05.529+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:05.530105+0000) 2022-04-04T00:54:05.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:05.680+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:05.931 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:05.928+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:06.532 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:06.530+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:06.530288+0000) 2022-04-04T00:54:06.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:06.691+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:06.946 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:06.944+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:07.532 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:07.530+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:07.530492+0000) 2022-04-04T00:54:07.654 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:07.652+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:07.993 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:07.991+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:08.532 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:08.530+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:08.530668+0000) 2022-04-04T00:54:08.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:08.617+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:09.007 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:09.004+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:09.532 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:09.530+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:09.530822+0000) 2022-04-04T00:54:09.599 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:09.597+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:09.719 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:54:09.723 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:54:10.051 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:10.048+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:10.532 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:10.530+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:10.531024+0000) 2022-04-04T00:54:10.575 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:10.572+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:11.058 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:11.056+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:11.532 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:11.530+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:11.531224+0000) 2022-04-04T00:54:11.566 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:11.563+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:11.968 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:54:11.965+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:54:12.051 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:12.048+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:12.532 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:12.529+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:12.531357+0000) 2022-04-04T00:54:12.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:12.549+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:13.027 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:13.024+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:13.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:13.511+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:13.532 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:13.530+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:13.531481+0000) 2022-04-04T00:54:14.013 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:14.011+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:14.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:14.492+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:14.532 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:14.530+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:14.531602+0000) 2022-04-04T00:54:14.988 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:14.986+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:15.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:15.488+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:15.533 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:15.530+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:15.531790+0000) 2022-04-04T00:54:16.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:16.002+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:16.533 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:16.530+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:16.531963+0000) 2022-04-04T00:54:16.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:16.536+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:17.019 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:17.016+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:17.533 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:17.531+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:17.532068+0000) 2022-04-04T00:54:17.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:17.549+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:17.985 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:17.983+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:18.533 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:18.531+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:18.532294+0000) 2022-04-04T00:54:18.574 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:18.572+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:19.024 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:19.022+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:19.533 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:19.531+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:19.532546+0000) 2022-04-04T00:54:19.601 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:19.598+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:19.988 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:19.986+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:20.534 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:20.531+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:20.532723+0000) 2022-04-04T00:54:20.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:20.570+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:20.952 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:20.949+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:21.534 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:21.532+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:21.532852+0000) 2022-04-04T00:54:21.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:21.600+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:21.951 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:21.948+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:22.534 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:22.532+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:22.533048+0000) 2022-04-04T00:54:22.617 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:22.614+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:22.988 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:22.985+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:23.534 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:23.532+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:23.533254+0000) 2022-04-04T00:54:23.598 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:23.595+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:23.977 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:23.974+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:24.534 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:24.532+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:24.533428+0000) 2022-04-04T00:54:24.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:24.578+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:24.994 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:24.991+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:25.535 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:25.532+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:25.533633+0000) 2022-04-04T00:54:25.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:25.626+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:25.976 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:25.973+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:26.535 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:26.533+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:26.533842+0000) 2022-04-04T00:54:26.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:26.660+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:26.955 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:26.952+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:26.970 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:54:26.968+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:54:27.535 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:27.533+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:27.533967+0000) 2022-04-04T00:54:27.651 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:27.648+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:27.974 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:27.972+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:28.535 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:28.533+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:28.534193+0000) 2022-04-04T00:54:28.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:28.600+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:28.927 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:28.925+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:29.535 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:29.533+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:29.534400+0000) 2022-04-04T00:54:29.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:29.646+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:29.893 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:29.891+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:30.536 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:30.533+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:30.534607+0000) 2022-04-04T00:54:30.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:30.688+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:30.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:30.852+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:31.536 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:31.534+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:31.534810+0000) 2022-04-04T00:54:31.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:31.675+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:31.848 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:31.846+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:32.536 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:32.534+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:32.535021+0000) 2022-04-04T00:54:32.720 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:32.718+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:32.844 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:32.842+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:33.536 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:33.534+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:33.535140+0000) 2022-04-04T00:54:33.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:33.758+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:33.846 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:33.843+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:34.536 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:34.534+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:34.535291+0000) 2022-04-04T00:54:34.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:34.804+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:34.836 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:34.833+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:35.536 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:35.534+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:35.535458+0000) 2022-04-04T00:54:35.810 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:35.807+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:35.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:35.820+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:36.537 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:36.535+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:36.535643+0000) 2022-04-04T00:54:36.786 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:36.783+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:36.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:36.822+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:37.537 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:37.535+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:37.535845+0000) 2022-04-04T00:54:37.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:37.798+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:37.867 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:37.864+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:38.537 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:38.535+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:38.535951+0000) 2022-04-04T00:54:38.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:38.797+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:38.821 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:38.819+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:39.537 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:39.535+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:39.536091+0000) 2022-04-04T00:54:39.774 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:39.772+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:39.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:39.780+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:39.874 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:54:39.878 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:54:40.537 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:40.535+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:40.536268+0000) 2022-04-04T00:54:40.736 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:40.734+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:40.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:40.765+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:41.537 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:41.535+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:41.536456+0000) 2022-04-04T00:54:41.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:41.708+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:41.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:41.734+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:41.987 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:54:41.984+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:54:42.537 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:42.536+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:42.536546+0000) 2022-04-04T00:54:42.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:42.742+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:42.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:42.771+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:43.538 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:43.536+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:43.536734+0000) 2022-04-04T00:54:43.719 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:43.717+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:43.768 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:43.766+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:44.538 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:44.536+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:44.536930+0000) 2022-04-04T00:54:44.722 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:44.720+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:44.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:44.748+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:45.538 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:45.536+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:45.537082+0000) 2022-04-04T00:54:45.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:45.692+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:45.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:45.785+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:46.538 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:46.536+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:46.537265+0000) 2022-04-04T00:54:46.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:46.740+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:46.818 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:46.816+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:47.538 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:47.537+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:47.537438+0000) 2022-04-04T00:54:47.716 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:47.714+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:47.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:47.842+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:48.539 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:48.537+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:48.537620+0000) 2022-04-04T00:54:48.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:48.727+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:48.845 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:48.844+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:49.539 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:49.537+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:49.537817+0000) 2022-04-04T00:54:49.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:49.715+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:49.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:49.880+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:50.539 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:50.537+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:50.537938+0000) 2022-04-04T00:54:50.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:50.725+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:50.893 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:50.892+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:51.539 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:51.537+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:51.538088+0000) 2022-04-04T00:54:51.696 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:51.694+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:51.895 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:51.893+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:52.539 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:52.537+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:52.538278+0000) 2022-04-04T00:54:52.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:52.744+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:52.884 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:52.882+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:53.539 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:53.538+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:53.538398+0000) 2022-04-04T00:54:53.774 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:53.772+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:53.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:53.887+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:54.539 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:54.538+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:54.538543+0000) 2022-04-04T00:54:54.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:54.798+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:54.926 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:54.924+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:55.540 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:55.538+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:55.538725+0000) 2022-04-04T00:54:55.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:55.783+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:55.952 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:55.951+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:56.540 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:56.537+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:56.538897+0000) 2022-04-04T00:54:56.746 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:56.745+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:56.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:56.994+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:57.002 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:54:56.999+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:54:57.540 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:57.537+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:57.539026+0000) 2022-04-04T00:54:57.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:57.740+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:57.955 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:57.952+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:58.540 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:58.538+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:58.539214+0000) 2022-04-04T00:54:58.711 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:58.708+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:54:58.997 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:54:58.994+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:54:59.540 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:54:59.538+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:54:59.539385+0000) 2022-04-04T00:54:59.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:54:59.742+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:00.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:00.013+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:00.540 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:00.538+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:00.539498+0000) 2022-04-04T00:55:00.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:00.757+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:01.057 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:01.054+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:01.541 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:01.538+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:01.539675+0000) 2022-04-04T00:55:01.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:01.783+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:02.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:02.064+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:02.541 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:02.538+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:02.539893+0000) 2022-04-04T00:55:02.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:02.809+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:03.066 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:03.063+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:03.541 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:03.539+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:03.540027+0000) 2022-04-04T00:55:03.828 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:03.825+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:04.021 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:04.018+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:04.541 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:04.539+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:04.540155+0000) 2022-04-04T00:55:04.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:04.803+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:04.999 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:04.996+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:05.541 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:05.539+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:05.540281+0000) 2022-04-04T00:55:05.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:05.819+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:06.007 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:06.004+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:06.541 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:06.539+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:06.540457+0000) 2022-04-04T00:55:06.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:06.818+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:07.032 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:07.030+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:07.541 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:07.539+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:07.540595+0000) 2022-04-04T00:55:07.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:07.786+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:08.065 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:08.062+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:08.542 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:08.540+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:08.540816+0000) 2022-04-04T00:55:08.773 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:08.770+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:09.071 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:09.068+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:09.542 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:09.540+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:09.540986+0000) 2022-04-04T00:55:09.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:09.817+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:09.997 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:55:10.001 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:55:10.101 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:10.098+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:10.542 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:10.540+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:10.541197+0000) 2022-04-04T00:55:10.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:10.808+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:11.065 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:11.062+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:11.542 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:11.540+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:11.541368+0000) 2022-04-04T00:55:11.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:11.827+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:12.018 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:55:12.016+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:55:12.033 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:12.030+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:12.543 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:12.540+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:12.541553+0000) 2022-04-04T00:55:12.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:12.787+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:13.052 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:13.050+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:13.543 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:13.541+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:13.541757+0000) 2022-04-04T00:55:13.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:13.817+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:14.063 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:14.060+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:14.543 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:14.541+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:14.541925+0000) 2022-04-04T00:55:14.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:14.800+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:15.070 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:15.067+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:15.543 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:15.541+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:15.542134+0000) 2022-04-04T00:55:15.844 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:15.842+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:16.098 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:16.095+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:16.543 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:16.541+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:16.542346+0000) 2022-04-04T00:55:16.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:16.829+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:17.051 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:17.049+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:17.544 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:17.541+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:17.542525+0000) 2022-04-04T00:55:17.844 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:17.841+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:18.084 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:18.080+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:18.544 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:18.542+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:18.542738+0000) 2022-04-04T00:55:18.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:18.810+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:19.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:19.113+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:19.544 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:19.542+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:19.542969+0000) 2022-04-04T00:55:19.816 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:19.814+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:20.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:20.083+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:20.544 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:20.542+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:20.543179+0000) 2022-04-04T00:55:20.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:20.844+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:21.091 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:21.089+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:21.544 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:21.542+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:21.543382+0000) 2022-04-04T00:55:21.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:21.807+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:22.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:22.113+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:22.544 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:22.543+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:22.543512+0000) 2022-04-04T00:55:22.859 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:22.856+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:23.066 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:23.064+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:23.545 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:23.543+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:23.543686+0000) 2022-04-04T00:55:23.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:23.811+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:24.057 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:24.055+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:24.545 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:24.543+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:24.543874+0000) 2022-04-04T00:55:24.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:24.843+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:25.063 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:25.060+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:25.545 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:25.543+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:25.544047+0000) 2022-04-04T00:55:25.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:25.822+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:26.025 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:26.023+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:26.545 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:26.543+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:26.544235+0000) 2022-04-04T00:55:26.814 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:26.811+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:27.034 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:55:27.032+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:55:27.044 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:27.042+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:27.545 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:27.544+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:27.544398+0000) 2022-04-04T00:55:27.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:27.822+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:28.011 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:28.009+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:28.545 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:28.543+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:28.544590+0000) 2022-04-04T00:55:28.795 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:28.793+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:28.986 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:28.983+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:29.546 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:29.543+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:29.544746+0000) 2022-04-04T00:55:29.844 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:29.842+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:29.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:29.950+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:30.546 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:30.544+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:30.544954+0000) 2022-04-04T00:55:30.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:30.843+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:30.968 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:30.965+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:31.546 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:31.544+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:31.545125+0000) 2022-04-04T00:55:31.892 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:31.889+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:31.983 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:31.981+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:32.546 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:32.545+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:32.545355+0000) 2022-04-04T00:55:32.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:32.921+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:32.955 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:32.953+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:33.546 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:33.545+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:33.545537+0000) 2022-04-04T00:55:33.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:33.919+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:33.923 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:33.921+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:34.547 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:34.545+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:34.545762+0000) 2022-04-04T00:55:34.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:34.963+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:34.970 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:34.968+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:35.547 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:35.545+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:35.545967+0000) 2022-04-04T00:55:35.956 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:35.953+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:35.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:35.971+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:36.547 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:36.546+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:36.546181+0000) 2022-04-04T00:55:36.970 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:36.968+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:37.000 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:36.998+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:37.547 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:37.546+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:37.546385+0000) 2022-04-04T00:55:37.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:37.946+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:37.990 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:37.988+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:38.547 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:38.546+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:38.546591+0000) 2022-04-04T00:55:38.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:38.898+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:38.999 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:38.997+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:39.548 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:39.546+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:39.546836+0000) 2022-04-04T00:55:39.943 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:39.941+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:40.041 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:40.039+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:40.102 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:55:40.106 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:55:40.548 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:40.545+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:40.547085+0000) 2022-04-04T00:55:40.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:40.892+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:41.006 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:41.004+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:41.548 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:41.546+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:41.547324+0000) 2022-04-04T00:55:41.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:41.848+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:42.040 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:42.038+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:42.050 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:55:42.047+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:55:42.548 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:42.546+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:42.547540+0000) 2022-04-04T00:55:42.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:42.820+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:43.013 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:43.011+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 58 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:43.549 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:43.546+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:43.547733+0000) 2022-04-04T00:55:43.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:43.862+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:43.968 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:43.966+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:44.549 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:44.546+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:44.547938+0000) 2022-04-04T00:55:44.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:44.886+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:44.984 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:44.982+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:45.549 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:45.547+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:45.548135+0000) 2022-04-04T00:55:45.934 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:45.932+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:46.028 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:46.027+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:46.549 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:46.547+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:46.548313+0000) 2022-04-04T00:55:46.899 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:46.897+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:46.989 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:46.987+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:47.549 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:47.547+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:47.548463+0000) 2022-04-04T00:55:47.871 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:47.869+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:47.983 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:47.981+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:48.550 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:48.547+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:48.548659+0000) 2022-04-04T00:55:48.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:48.894+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:48.956 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:48.954+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:49.550 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:49.547+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:49.548865+0000) 2022-04-04T00:55:49.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:49.886+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:49.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:49.946+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:50.550 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:50.548+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:50.549029+0000) 2022-04-04T00:55:50.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:50.886+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:50.935 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:50.933+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:51.550 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:51.548+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:51.549240+0000) 2022-04-04T00:55:51.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:51.916+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:51.965 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:51.963+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:52.550 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:52.548+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:52.549519+0000) 2022-04-04T00:55:52.909 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:52.906+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:53.008 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:53.005+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:53.551 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:53.548+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:53.549699+0000) 2022-04-04T00:55:53.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:53.914+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:53.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:53.993+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:54.551 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:54.549+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:54.549911+0000) 2022-04-04T00:55:54.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:54.874+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:54.998 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:54.995+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:55.551 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:55.549+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:55.550130+0000) 2022-04-04T00:55:55.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:55.909+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:56.024 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:56.021+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:56.551 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:56.549+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:56.550286+0000) 2022-04-04T00:55:56.935 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:56.933+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:56.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:56.993+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:57.066 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:55:57.064+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:55:57.551 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:57.549+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:57.550444+0000) 2022-04-04T00:55:57.915 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:57.912+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:58.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:58.000+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:58.551 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:58.549+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:58.550570+0000) 2022-04-04T00:55:58.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:58.925+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:59.027 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:59.024+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:55:59.552 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:55:59.550+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:55:59.550791+0000) 2022-04-04T00:55:59.894 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:55:59.891+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:55:59.999 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:55:59.996+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:00.552 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:00.550+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:00.551006+0000) 2022-04-04T00:56:00.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:00.924+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:01.007 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:01.004+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:01.552 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:01.550+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:01.551208+0000) 2022-04-04T00:56:01.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:01.963+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:01.985 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:01.983+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:02.552 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:02.550+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:02.551370+0000) 2022-04-04T00:56:03.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:03.001+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:03.031 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:03.029+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:03.553 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:03.550+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:03.551598+0000) 2022-04-04T00:56:03.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:03.981+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:04.070 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:04.067+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:04.553 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:04.551+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:04.551794+0000) 2022-04-04T00:56:05.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:05.001+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:05.026 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:05.024+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:05.553 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:05.551+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:05.551982+0000) 2022-04-04T00:56:06.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:06.004+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:06.040 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:06.038+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:06.553 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:06.551+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:06.552190+0000) 2022-04-04T00:56:06.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:06.989+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:07.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:07.022+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:07.553 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:07.551+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:07.552397+0000) 2022-04-04T00:56:07.981 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:07.979+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:08.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:08.034+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:08.554 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:08.552+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:08.552563+0000) 2022-04-04T00:56:09.022 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:09.019+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:09.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:09.035+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:09.554 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:09.552+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:09.552839+0000) 2022-04-04T00:56:10.011 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:10.009+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:10.020 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:10.017+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:10.257 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:56:10.261 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:56:10.554 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:10.552+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:10.553060+0000) 2022-04-04T00:56:10.975 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:10.972+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:11.020 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:11.017+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:11.554 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:11.552+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:11.553270+0000) 2022-04-04T00:56:11.973 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:11.971+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:11.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:11.985+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:12.081 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:56:12.079+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:56:12.554 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:12.553+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:12.553484+0000) 2022-04-04T00:56:12.931 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:12.929+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:12.969 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:12.967+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:13.555 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:13.553+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:13.553668+0000) 2022-04-04T00:56:13.963 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:13.960+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:13.974 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:13.971+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:14.555 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:14.553+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:14.553865+0000) 2022-04-04T00:56:14.934 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:14.932+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:14.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:14.985+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:15.555 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:15.553+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:15.554044+0000) 2022-04-04T00:56:15.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:15.931+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:15.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:15.996+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:16.555 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:16.553+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:16.554218+0000) 2022-04-04T00:56:16.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:16.946+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:16.967 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:16.964+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:17.555 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:17.553+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:17.554391+0000) 2022-04-04T00:56:17.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:17.975+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:18.010 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:18.007+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:18.556 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:18.554+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:18.554601+0000) 2022-04-04T00:56:18.981 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:18.979+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:19.000 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:18.997+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:19.556 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:19.554+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:19.554863+0000) 2022-04-04T00:56:20.002 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:20.000+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:20.018 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:20.016+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:20.556 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:20.554+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:20.555015+0000) 2022-04-04T00:56:20.976 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:20.973+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:20.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:20.996+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:21.556 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:21.555+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:21.555168+0000) 2022-04-04T00:56:21.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:21.941+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:22.022 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:22.020+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:22.556 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:22.555+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:22.555383+0000) 2022-04-04T00:56:22.958 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:22.955+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:23.014 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:23.012+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:23.556 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:23.555+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:23.555586+0000) 2022-04-04T00:56:23.939 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:23.937+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:23.967 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:23.965+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:24.557 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:24.555+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:24.555812+0000) 2022-04-04T00:56:24.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:24.949+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:24.980 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:24.978+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:25.557 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:25.554+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:25.555897+0000) 2022-04-04T00:56:25.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:25.916+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:26.019 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:26.017+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:26.557 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:26.554+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:26.556100+0000) 2022-04-04T00:56:26.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:26.926+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:27.046 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:27.044+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:27.097 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:56:27.094+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:56:27.557 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:27.555+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:27.556303+0000) 2022-04-04T00:56:27.932 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:27.930+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:28.077 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:28.075+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:28.557 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:28.555+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:28.556515+0000) 2022-04-04T00:56:28.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:28.885+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:29.066 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:29.064+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:29.558 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:29.555+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:29.556730+0000) 2022-04-04T00:56:29.852 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:29.850+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:30.108 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:30.106+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:30.558 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:30.556+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:30.556948+0000) 2022-04-04T00:56:30.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:30.840+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:31.078 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:31.076+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:31.558 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:31.556+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:31.557077+0000) 2022-04-04T00:56:31.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:31.821+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:32.119 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:32.117+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:32.558 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:32.556+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:32.557237+0000) 2022-04-04T00:56:32.786 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:32.785+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:33.098 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:33.096+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:33.558 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:33.556+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:33.557371+0000) 2022-04-04T00:56:33.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:33.792+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:34.145 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:34.144+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:34.558 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:34.556+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:34.557517+0000) 2022-04-04T00:56:34.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:34.823+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:35.158 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:35.157+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:35.559 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:35.556+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:35.557696+0000) 2022-04-04T00:56:35.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:35.777+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:36.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:36.179+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:36.559 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:36.557+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:36.557902+0000) 2022-04-04T00:56:36.774 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:36.773+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:37.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:37.221+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:37.559 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:37.557+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:37.558114+0000) 2022-04-04T00:56:37.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:37.804+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:38.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:38.237+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:38.559 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:38.557+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:38.558297+0000) 2022-04-04T00:56:38.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:38.840+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:39.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:39.216+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:39.559 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:39.557+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:39.558429+0000) 2022-04-04T00:56:39.880 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:39.878+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:40.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:40.260+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:40.320 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:56:40.324 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:56:40.559 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:40.557+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:40.558665+0000) 2022-04-04T00:56:40.898 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:40.896+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:41.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:41.264+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:41.560 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:41.558+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:41.558824+0000) 2022-04-04T00:56:41.920 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:41.919+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:42.113 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:56:42.111+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:56:42.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:42.284+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:42.560 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:42.558+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:42.558967+0000) 2022-04-04T00:56:42.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:42.901+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:43.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:43.287+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:43.560 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:43.558+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:43.559077+0000) 2022-04-04T00:56:43.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:43.935+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:44.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:44.241+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:44.560 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:44.558+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:44.559266+0000) 2022-04-04T00:56:44.944 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:44.942+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:45.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:45.258+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:45.560 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:45.558+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:45.559534+0000) 2022-04-04T00:56:45.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:45.982+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:46.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:46.273+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:46.561 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:46.559+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:46.559685+0000) 2022-04-04T00:56:46.975 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:46.973+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:47.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:47.239+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:47.561 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:47.559+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:47.559910+0000) 2022-04-04T00:56:47.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:47.962+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:48.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:48.190+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:48.561 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:48.559+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:48.560152+0000) 2022-04-04T00:56:48.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:48.996+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:49.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:49.225+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:49.561 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:49.559+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:49.560399+0000) 2022-04-04T00:56:50.027 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:50.024+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:50.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:50.255+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:50.562 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:50.560+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:50.560643+0000) 2022-04-04T00:56:51.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:51.045+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:51.252 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:51.250+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:51.562 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:51.560+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:51.560788+0000) 2022-04-04T00:56:52.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:52.058+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:52.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:52.235+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:52.562 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:52.560+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:52.561033+0000) 2022-04-04T00:56:53.090 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:53.087+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:53.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:53.267+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:53.562 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:53.560+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:53.561212+0000) 2022-04-04T00:56:54.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:54.122+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:54.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:54.276+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:54.563 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:54.561+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:54.561464+0000) 2022-04-04T00:56:55.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:55.151+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:55.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:55.275+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:55.566 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:55.561+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:55.561713+0000) 2022-04-04T00:56:56.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:56.119+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:56.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:56.305+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:56.563 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:56.561+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:56.561966+0000) 2022-04-04T00:56:57.129 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:56:57.127+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:56:57.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:57.140+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:57.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:57.343+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:57.563 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:57.561+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:57.562186+0000) 2022-04-04T00:56:58.100 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:58.097+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:58.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:58.344+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:58.563 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:58.562+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:58.562393+0000) 2022-04-04T00:56:59.065 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:56:59.062+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:56:59.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:56:59.304+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:56:59.563 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:56:59.562+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:56:59.562615+0000) 2022-04-04T00:57:00.036 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:00.034+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:00.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:00.293+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:00.564 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:00.562+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:00.562878+0000) 2022-04-04T00:57:01.017 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:01.014+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:01.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:01.341+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:01.564 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:01.562+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:01.563122+0000) 2022-04-04T00:57:01.968 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:01.965+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:02.382 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:02.379+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:02.564 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:02.563+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:02.563294+0000) 2022-04-04T00:57:02.962 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:02.959+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:03.428 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:03.425+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:03.565 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:03.563+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:03.563517+0000) 2022-04-04T00:57:03.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:03.963+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:04.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:04.441+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:04.565 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:04.563+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:04.563714+0000) 2022-04-04T00:57:04.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:04.970+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:05.489 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:05.486+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:05.565 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:05.563+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:05.563940+0000) 2022-04-04T00:57:05.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:05.954+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:06.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:06.529+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:06.565 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:06.563+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:06.564181+0000) 2022-04-04T00:57:06.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:06.908+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:07.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:07.511+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:07.565 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:07.564+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:07.564434+0000) 2022-04-04T00:57:07.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:07.871+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:08.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:08.478+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:08.565 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:08.564+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:08.564541+0000) 2022-04-04T00:57:08.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:08.916+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:09.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:09.453+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:09.566 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:09.563+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:09.564728+0000) 2022-04-04T00:57:09.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:09.937+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:10.501 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:10.499+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:10.566 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:10.563+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:10.564866+0000) 2022-04-04T00:57:10.577 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:57:10.581 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:57:10.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:10.890+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:11.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:11.491+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:11.566 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:11.564+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:11.565086+0000) 2022-04-04T00:57:11.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:11.928+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:12.144 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:57:12.142+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:57:12.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:12.529+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:12.566 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:12.564+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:12.565320+0000) 2022-04-04T00:57:12.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:12.949+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:13.546 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:13.543+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:13.566 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:13.564+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:13.565518+0000) 2022-04-04T00:57:13.909 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:13.907+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:14.566 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:14.564+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:14.565594+0000) 2022-04-04T00:57:14.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:14.565+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:14.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:14.936+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:15.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:15.560+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:15.566 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:15.564+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:15.565715+0000) 2022-04-04T00:57:15.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:15.894+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:16.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:16.543+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:16.567 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:16.564+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:16.565903+0000) 2022-04-04T00:57:16.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:16.870+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:17.567 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:17.564+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:17.566091+0000) 2022-04-04T00:57:17.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:17.590+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:17.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:17.901+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:18.567 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:18.565+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:18.566228+0000) 2022-04-04T00:57:18.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:18.638+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:18.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:18.877+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:19.567 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:19.565+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:19.566452+0000) 2022-04-04T00:57:19.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:19.642+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:19.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:19.829+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:20.568 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:20.565+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:20.566648+0000) 2022-04-04T00:57:20.610 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:20.608+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:20.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:20.829+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:21.568 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:21.566+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:21.566884+0000) 2022-04-04T00:57:21.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:21.611+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:21.798 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:21.795+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:22.568 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:22.566+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:22.567161+0000) 2022-04-04T00:57:22.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:22.641+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:22.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:22.776+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:23.568 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:23.566+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:23.567419+0000) 2022-04-04T00:57:23.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:23.597+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:23.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:23.786+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:24.568 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:24.566+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:24.567611+0000) 2022-04-04T00:57:24.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:24.635+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:24.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:24.746+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:25.569 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:25.567+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:25.567786+0000) 2022-04-04T00:57:25.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:25.642+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:25.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:25.782+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:26.569 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:26.567+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:26.568038+0000) 2022-04-04T00:57:26.674 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:26.672+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:26.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:26.782+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:27.160 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:57:27.158+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:57:27.569 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:27.567+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:27.568224+0000) 2022-04-04T00:57:27.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:27.697+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:27.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:27.780+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:28.569 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:28.567+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:28.568465+0000) 2022-04-04T00:57:28.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:28.670+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:28.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:28.740+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:29.570 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:29.568+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:29.568731+0000) 2022-04-04T00:57:29.662 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:29.660+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:29.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:29.773+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:30.570 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:30.568+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:30.569012+0000) 2022-04-04T00:57:30.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:30.628+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:30.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:30.758+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:31.570 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:31.568+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:31.569260+0000) 2022-04-04T00:57:31.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:31.611+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:31.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:31.774+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:32.570 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:32.568+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:32.569511+0000) 2022-04-04T00:57:32.633 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:32.631+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:32.747 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:32.746+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:33.571 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:33.569+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:33.569660+0000) 2022-04-04T00:57:33.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:33.615+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:33.768 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:33.766+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:34.571 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:34.569+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:34.569895+0000) 2022-04-04T00:57:34.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:34.662+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:34.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:34.797+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:35.571 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:35.569+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:35.570114+0000) 2022-04-04T00:57:35.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:35.626+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:35.768 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:35.766+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:36.571 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:36.569+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:36.570363+0000) 2022-04-04T00:57:36.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:36.625+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:36.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:36.782+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:37.572 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:37.570+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:37.570606+0000) 2022-04-04T00:57:37.666 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:37.665+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:37.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:37.802+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:38.572 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:38.570+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:38.570875+0000) 2022-04-04T00:57:38.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:38.645+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:38.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:38.840+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:39.572 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:39.570+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:39.571128+0000) 2022-04-04T00:57:39.664 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:39.663+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:39.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:39.884+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:40.572 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:40.570+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:40.571373+0000) 2022-04-04T00:57:40.617 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:40.615+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:40.663 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:57:40.666 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:57:40.850 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:40.849+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:41.572 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:41.570+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:41.573 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:41.570+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:41.571650+0000) 2022-04-04T00:57:41.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:41.837+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:42.176 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:57:42.174+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:57:42.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:42.528+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:42.573 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:42.571+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:42.571876+0000) 2022-04-04T00:57:42.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:42.875+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:43.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:43.534+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:43.573 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:43.571+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:43.572079+0000) 2022-04-04T00:57:43.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:43.876+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:44.573 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:44.571+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:44.572272+0000) 2022-04-04T00:57:44.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:44.571+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:44.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:44.898+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:45.573 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:45.572+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:45.572505+0000) 2022-04-04T00:57:45.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:45.573+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:45.907 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:45.904+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:46.574 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:46.572+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:46.572665+0000) 2022-04-04T00:57:46.604 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:46.601+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:46.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:46.945+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:47.574 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:47.572+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:47.572882+0000) 2022-04-04T00:57:47.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:47.627+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:47.961 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:47.958+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:48.574 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:48.572+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:48.573106+0000) 2022-04-04T00:57:48.595 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:48.592+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:48.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:48.918+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:49.572 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:49.570+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:49.574 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:49.573+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:49.573326+0000) 2022-04-04T00:57:49.936 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:49.934+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:50.566 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:50.563+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:50.574 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:50.573+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:50.573574+0000) 2022-04-04T00:57:50.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:50.963+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:51.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:51.540+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:51.575 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:51.573+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:51.573849+0000) 2022-04-04T00:57:51.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:51.955+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:52.575 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:52.573+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:52.574115+0000) 2022-04-04T00:57:52.586 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:52.583+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:52.978 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:52.975+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:53.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:53.561+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:53.575 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:53.574+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:53.574362+0000) 2022-04-04T00:57:53.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:53.953+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:54.575 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:54.573+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:54.574544+0000) 2022-04-04T00:57:54.586 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:54.583+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:54.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:54.963+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:55.576 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:55.573+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:55.574759+0000) 2022-04-04T00:57:55.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:55.596+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:55.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:55.980+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:56.576 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:56.573+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:56.575027+0000) 2022-04-04T00:57:56.577 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:56.574+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:56.986 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:56.983+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:57.192 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:57:57.189+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:57:57.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:57.552+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:57.576 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:57.573+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:57.575272+0000) 2022-04-04T00:57:57.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:57.979+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:58.540 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:58.534+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:58.576 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:58.574+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:58.575518+0000) 2022-04-04T00:57:58.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:58.988+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:57:59.552 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:57:59.549+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:57:59.577 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:57:59.574+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:57:59.575826+0000) 2022-04-04T00:57:59.976 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:57:59.973+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:00.578 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:00.575+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:00.576088+0000) 2022-04-04T00:58:00.595 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:00.590+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:00.936 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:00.933+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:01.577 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:01.575+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:01.576337+0000) 2022-04-04T00:58:01.608 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:01.605+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:01.968 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:01.966+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:02.577 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:02.575+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:02.576617+0000) 2022-04-04T00:58:02.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:02.637+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:02.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:02.920+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:03.578 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:03.575+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:03.576792+0000) 2022-04-04T00:58:03.642 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:03.640+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:03.894 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:03.891+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:04.578 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:04.576+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:04.577032+0000) 2022-04-04T00:58:04.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:04.676+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:04.939 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:04.936+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:05.578 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:05.576+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:05.577237+0000) 2022-04-04T00:58:05.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:05.712+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:05.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:05.919+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:06.579 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:06.576+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:06.577502+0000) 2022-04-04T00:58:06.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:06.680+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:06.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:06.920+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:07.579 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:07.576+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:07.577749+0000) 2022-04-04T00:58:07.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:07.643+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:07.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:07.921+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:08.579 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:08.577+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:08.577978+0000) 2022-04-04T00:58:08.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:08.670+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:08.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:08.883+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:09.579 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:09.577+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:09.578151+0000) 2022-04-04T00:58:09.720 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:09.718+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:09.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:09.874+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:10.579 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:10.577+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:10.578326+0000) 2022-04-04T00:58:10.728 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:58:10.732 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:58:10.745 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:10.743+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:10.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:10.902+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:11.580 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:11.577+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:11.578556+0000) 2022-04-04T00:58:11.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:11.735+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:11.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:11.862+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:12.208 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:58:12.206+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:58:12.580 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:12.578+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:12.578840+0000) 2022-04-04T00:58:12.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:12.692+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:12.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:12.834+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:13.580 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:13.578+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:13.579082+0000) 2022-04-04T00:58:13.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:13.665+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:13.871 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:13.869+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:14.580 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:14.578+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:14.579283+0000) 2022-04-04T00:58:14.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:14.652+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:14.857 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:14.854+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:15.580 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:15.578+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:15.579503+0000) 2022-04-04T00:58:15.604 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:15.602+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:15.837 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:15.836+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:16.580 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:16.579+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:16.579675+0000) 2022-04-04T00:58:16.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:16.644+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:16.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:16.803+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:17.581 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:17.579+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:17.579893+0000) 2022-04-04T00:58:17.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:17.603+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:17.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:17.800+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:18.581 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:18.579+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:18.580131+0000) 2022-04-04T00:58:18.634 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:18.632+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:18.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:18.787+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:19.581 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:19.579+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:19.580403+0000) 2022-04-04T00:58:19.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:19.583+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:19.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:19.833+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:20.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:20.543+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:20.581 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:20.580+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:20.580621+0000) 2022-04-04T00:58:20.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:20.828+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:21.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:21.561+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:21.582 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:21.580+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:21.580858+0000) 2022-04-04T00:58:21.869 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:21.867+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:22.573 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:22.571+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:22.582 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:22.580+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:22.581066+0000) 2022-04-04T00:58:22.869 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:22.867+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:23.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:23.576+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:23.582 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:23.580+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:23.581199+0000) 2022-04-04T00:58:23.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:23.863+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:24.582 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:24.580+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:24.581377+0000) 2022-04-04T00:58:24.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:24.611+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:24.856 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:24.854+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:25.583 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:25.581+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:25.581560+0000) 2022-04-04T00:58:25.608 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:25.606+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:25.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:25.874+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:26.583 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:26.581+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:26.581752+0000) 2022-04-04T00:58:26.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:26.625+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:26.919 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:26.918+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:27.224 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:58:27.222+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:58:27.583 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:27.581+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:27.581983+0000) 2022-04-04T00:58:27.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:27.634+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:27.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:27.937+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:28.583 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:28.581+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:28.582162+0000) 2022-04-04T00:58:28.617 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:28.615+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:28.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:28.888+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:29.583 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:29.582+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:29.582368+0000) 2022-04-04T00:58:29.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:29.650+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:29.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:29.847+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:30.583 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:30.582+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:30.582591+0000) 2022-04-04T00:58:30.648 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:30.646+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:30.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:30.806+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:31.584 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:31.582+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:31.582823+0000) 2022-04-04T00:58:31.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:31.626+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:31.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:31.791+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:32.584 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:32.582+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:32.583071+0000) 2022-04-04T00:58:32.623 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:32.621+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:32.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:32.813+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:33.584 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:33.583+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:33.583297+0000) 2022-04-04T00:58:33.623 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:33.621+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:33.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:33.848+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:34.584 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:34.583+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:34.583503+0000) 2022-04-04T00:58:34.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:34.612+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:34.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:34.811+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:35.585 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:35.583+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:35.583719+0000) 2022-04-04T00:58:35.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:35.586+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:35.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:35.803+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:36.585 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:36.583+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:36.583935+0000) 2022-04-04T00:58:36.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:36.626+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:36.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:36.771+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:37.585 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:37.584+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:37.584083+0000) 2022-04-04T00:58:37.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:37.641+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:37.730 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:37.727+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:38.585 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:38.583+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:38.584207+0000) 2022-04-04T00:58:38.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:38.616+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:38.701 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:38.698+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:39.585 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:39.583+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:39.584399+0000) 2022-04-04T00:58:39.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:39.594+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:39.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:39.735+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:40.585 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:40.583+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:40.584600+0000) 2022-04-04T00:58:40.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:40.593+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:40.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:40.767+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:40.859 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:58:40.863 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:58:41.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:41.558+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:41.586 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:41.583+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:41.584861+0000) 2022-04-04T00:58:41.730 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:41.727+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:42.239 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:58:42.236+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:58:42.544 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:42.542+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 47 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:42.586 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:42.584+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:42.585027+0000) 2022-04-04T00:58:42.769 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:42.767+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:43.586 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:43.584+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:43.585248+0000) 2022-04-04T00:58:43.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:43.586+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:43.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:43.797+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:44.586 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:44.584+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:44.585437+0000) 2022-04-04T00:58:44.629 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:44.627+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:44.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:44.788+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:45.586 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:45.584+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:45.585617+0000) 2022-04-04T00:58:45.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:45.658+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:45.816 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:45.814+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:46.587 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:46.584+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:46.585840+0000) 2022-04-04T00:58:46.693 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:46.690+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:46.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:46.785+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:47.587 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:47.585+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:47.586057+0000) 2022-04-04T00:58:47.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:47.667+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:47.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:47.815+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:48.587 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:48.585+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:48.586265+0000) 2022-04-04T00:58:48.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:48.652+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:48.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:48.791+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:49.587 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:49.585+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:49.586467+0000) 2022-04-04T00:58:49.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:49.633+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:49.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:49.780+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:50.587 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:50.585+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:50.586650+0000) 2022-04-04T00:58:50.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:50.629+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:50.786 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:50.784+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:51.588 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:51.586+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:51.586864+0000) 2022-04-04T00:58:51.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:51.649+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:51.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:51.816+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:52.588 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:52.586+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:52.587090+0000) 2022-04-04T00:58:52.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:52.686+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:52.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:52.829+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:53.591 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:53.586+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:53.587242+0000) 2022-04-04T00:58:53.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:53.644+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:53.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:53.845+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:54.588 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:54.586+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:54.587443+0000) 2022-04-04T00:58:54.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:54.634+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:54.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:54.823+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:55.588 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:55.586+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:55.587646+0000) 2022-04-04T00:58:55.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:55.667+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:55.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:55.810+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:56.589 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:56.587+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:56.587868+0000) 2022-04-04T00:58:56.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:56.642+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:56.837 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:56.835+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:57.242 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:58:57.240+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:58:57.589 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:57.587+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:57.588016+0000) 2022-04-04T00:58:57.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:57.609+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:57.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:57.821+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:58.589 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:58.587+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:58.588240+0000) 2022-04-04T00:58:58.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:58.642+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:58.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:58.870+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:58:59.589 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:58:59.587+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:58:59.588459+0000) 2022-04-04T00:58:59.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:58:59.665+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:58:59.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:58:59.862+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:00.590 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:00.588+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:00.588665+0000) 2022-04-04T00:59:00.634 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:00.632+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:00.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:00.841+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:01.590 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:01.588+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:01.588867+0000) 2022-04-04T00:59:01.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:01.609+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:01.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:01.799+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:02.591 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:02.588+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:02.589118+0000) 2022-04-04T00:59:02.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:02.588+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:02.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:02.816+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:03.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:03.552+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:03.595 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:03.588+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:03.589336+0000) 2022-04-04T00:59:03.769 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:03.767+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:04.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:04.549+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:04.591 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:04.589+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:04.589526+0000) 2022-04-04T00:59:04.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:04.803+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:05.573 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:05.571+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:05.591 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:05.589+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:05.589736+0000) 2022-04-04T00:59:05.814 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:05.812+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:06.591 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:06.589+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:06.589963+0000) 2022-04-04T00:59:06.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:06.596+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:06.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:06.845+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:07.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:07.551+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:07.591 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:07.589+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:07.590198+0000) 2022-04-04T00:59:07.856 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:07.854+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:08.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:08.557+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:08.591 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:08.589+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:08.590446+0000) 2022-04-04T00:59:08.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:08.859+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:09.550 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:09.547+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:09.591 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:09.590+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:09.590633+0000) 2022-04-04T00:59:09.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:09.833+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:10.523 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:10.521+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:10.592 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:10.590+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:10.590856+0000) 2022-04-04T00:59:10.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:10.862+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:11.023 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:59:11.027 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:59:11.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:11.546+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:11.592 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:11.590+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:11.591039+0000) 2022-04-04T00:59:11.898 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:11.896+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:12.258 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:59:12.256+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:59:12.528 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:12.525+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:12.592 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:12.590+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:12.591235+0000) 2022-04-04T00:59:12.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:12.887+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:13.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:13.552+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:13.592 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:13.590+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:13.591426+0000) 2022-04-04T00:59:13.892 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:13.890+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:14.510 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:14.509+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:14.592 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:14.591+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:14.591646+0000) 2022-04-04T00:59:14.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:14.915+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:15.517 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:15.515+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:15.593 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:15.591+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:15.591859+0000) 2022-04-04T00:59:15.910 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:15.908+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:16.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:16.557+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:16.593 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:16.591+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:16.592011+0000) 2022-04-04T00:59:16.945 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:16.943+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:17.593 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:17.592+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:17.592217+0000) 2022-04-04T00:59:17.602 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:17.601+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:17.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:17.989+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:18.593 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:18.592+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:18.592428+0000) 2022-04-04T00:59:18.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:18.611+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:18.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:18.994+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:19.593 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:19.592+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:19.592610+0000) 2022-04-04T00:59:19.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:19.611+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:19.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:19.991+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:20.594 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:20.592+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:20.592760+0000) 2022-04-04T00:59:20.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:20.629+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:20.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:20.957+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:21.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:21.586+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:21.594 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:21.592+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:21.592921+0000) 2022-04-04T00:59:21.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:21.938+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:22.594 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:22.593+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:22.593137+0000) 2022-04-04T00:59:22.629 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:22.628+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:22.909 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:22.907+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:23.594 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:23.592+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:23.593358+0000) 2022-04-04T00:59:23.666 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:23.664+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:23.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:23.874+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:24.594 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:24.592+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:24.593544+0000) 2022-04-04T00:59:24.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:24.637+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:24.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:24.894+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:25.595 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:25.592+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:25.593751+0000) 2022-04-04T00:59:25.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:25.649+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:25.891 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:25.889+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:26.595 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:26.592+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:26.593928+0000) 2022-04-04T00:59:26.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:26.636+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:26.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:26.871+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:27.274 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:59:27.271+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:59:27.595 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:27.593+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:27.594134+0000) 2022-04-04T00:59:27.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:27.673+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:27.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:27.920+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:28.595 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:28.593+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:28.594354+0000) 2022-04-04T00:59:28.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:28.682+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:28.883 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:28.881+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:29.595 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:29.593+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:29.594548+0000) 2022-04-04T00:59:29.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:29.683+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:29.920 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:29.918+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:30.596 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:30.593+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:30.594748+0000) 2022-04-04T00:59:30.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:30.721+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:30.967 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:30.965+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:31.596 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:31.594+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:31.594955+0000) 2022-04-04T00:59:31.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:31.735+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:31.939 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:31.938+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:32.596 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:32.594+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:32.595173+0000) 2022-04-04T00:59:32.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:32.753+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:32.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:32.970+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:33.596 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:33.594+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:33.595373+0000) 2022-04-04T00:59:33.708 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:33.706+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:33.925 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:33.922+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:34.596 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:34.594+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:34.595548+0000) 2022-04-04T00:59:34.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:34.715+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:34.939 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:34.936+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:35.597 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:35.595+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:35.595763+0000) 2022-04-04T00:59:35.674 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:35.672+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:35.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:35.894+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:36.597 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:36.595+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:36.596013+0000) 2022-04-04T00:59:36.706 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:36.703+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:36.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:36.919+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:37.597 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:37.595+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:37.596215+0000) 2022-04-04T00:59:37.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:37.700+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:37.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:37.934+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:38.597 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:38.595+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:38.596431+0000) 2022-04-04T00:59:38.947 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:38.738+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:38.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:38.888+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:39.597 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:39.595+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:39.596668+0000) 2022-04-04T00:59:39.774 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:39.772+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:39.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:39.855+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:40.598 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:40.596+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:40.596896+0000) 2022-04-04T00:59:40.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:40.769+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:40.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:40.894+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:41.161 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:59:41.165 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T00:59:41.598 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:41.596+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:41.597093+0000) 2022-04-04T00:59:41.778 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:41.775+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:41.902 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:41.900+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:42.276 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:59:42.274+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:59:42.598 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:42.596+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:42.597256+0000) 2022-04-04T00:59:42.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:42.736+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:42.906 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:42.904+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:43.598 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:43.596+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:43.597427+0000) 2022-04-04T00:59:43.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:43.765+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:43.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:43.926+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:44.599 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:44.597+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:44.597644+0000) 2022-04-04T00:59:44.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:44.716+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:44.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:44.974+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:45.599 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:45.597+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:45.597861+0000) 2022-04-04T00:59:45.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:45.711+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:45.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:45.940+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:46.599 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:46.597+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:46.598030+0000) 2022-04-04T00:59:46.735 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:46.732+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:46.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:46.909+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:47.599 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:47.597+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:47.598170+0000) 2022-04-04T00:59:47.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:47.711+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:47.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:47.919+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:48.599 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:48.597+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:48.598339+0000) 2022-04-04T00:59:48.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:48.687+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:48.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:48.924+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:49.599 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:49.597+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:49.598500+0000) 2022-04-04T00:59:49.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:49.725+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:49.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:49.902+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:50.600 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:50.598+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:50.598687+0000) 2022-04-04T00:59:50.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:50.720+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:50.902 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:50.900+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:51.600 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:51.598+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:51.598870+0000) 2022-04-04T00:59:51.753 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:51.751+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:51.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:51.931+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:52.600 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:52.598+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:52.599058+0000) 2022-04-04T00:59:52.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:52.734+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:52.916 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:52.914+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:53.600 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:53.598+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:53.599285+0000) 2022-04-04T00:59:53.739 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:53.737+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:53.919 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:53.917+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:54.600 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:54.599+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:54.599489+0000) 2022-04-04T00:59:54.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:54.694+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:54.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:54.879+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:55.601 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:55.599+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:55.599711+0000) 2022-04-04T00:59:55.670 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:55.667+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:55.852 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:55.849+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:56.601 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:56.599+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:56.599883+0000) 2022-04-04T00:59:56.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:56.655+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:56.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:56.873+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:57.292 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T00:59:57.290+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T00:59:57.601 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:57.599+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:57.600057+0000) 2022-04-04T00:59:57.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:57.694+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:57.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:57.857+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:58.601 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:58.599+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:58.600211+0000) 2022-04-04T00:59:58.683 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:58.681+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:58.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:58.858+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T00:59:59.601 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T00:59:59.599+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-03T23:59:59.600413+0000) 2022-04-04T00:59:59.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T00:59:59.647+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T00:59:59.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T00:59:59.902+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:00.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:00.599+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:00.602 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:00.600+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:00.600617+0000) 2022-04-04T01:00:00.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:00.886+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:01.602 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:01.600+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:01.600817+0000) 2022-04-04T01:00:01.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:01.645+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:01.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:01.843+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:02.602 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:02.600+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:02.600995+0000) 2022-04-04T01:00:02.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:02.610+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:02.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:02.870+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:03.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:03.585+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:03.602 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:03.600+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:03.601166+0000) 2022-04-04T01:00:03.905 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:03.903+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:04.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:04.559+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:04.603 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:04.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:04.601376+0000) 2022-04-04T01:00:04.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:04.948+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:05.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:05.560+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:05.602 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:05.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:05.601469+0000) 2022-04-04T01:00:05.907 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:05.905+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:06.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:06.589+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:06.603 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:06.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:06.601604+0000) 2022-04-04T01:00:06.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:06.861+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:07.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:07.597+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:07.602 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:07.600+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:07.601852+0000) 2022-04-04T01:00:07.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:07.843+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:08.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:08.581+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:08.603 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:08.600+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:08.601989+0000) 2022-04-04T01:00:08.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:08.824+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:09.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:09.565+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:09.603 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:09.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:09.602105+0000) 2022-04-04T01:00:09.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:09.870+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:10.550 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:10.548+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:10.603 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:10.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:10.602262+0000) 2022-04-04T01:00:10.869 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:10.867+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:11.300 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:00:11.304 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:00:11.533 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:11.531+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:11.603 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:11.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:11.602503+0000) 2022-04-04T01:00:11.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:11.825+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:12.308 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:00:12.305+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:00:12.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:12.486+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:12.604 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:12.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:12.602725+0000) 2022-04-04T01:00:12.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:12.873+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:13.525 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:13.519+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:13.604 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:13.601+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:13.602928+0000) 2022-04-04T01:00:13.894 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:13.893+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:14.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:14.479+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:14.604 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:14.602+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:14.603145+0000) 2022-04-04T01:00:14.929 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:14.928+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:15.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:15.490+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:15.604 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:15.602+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:15.603346+0000) 2022-04-04T01:00:15.963 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:15.961+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:16.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:16.480+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:16.604 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:16.602+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:16.603555+0000) 2022-04-04T01:00:16.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:16.940+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:17.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:17.450+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:17.605 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:17.602+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:17.603731+0000) 2022-04-04T01:00:17.971 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:17.969+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:18.444 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:18.443+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:18.605 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:18.603+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:18.603854+0000) 2022-04-04T01:00:18.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:18.948+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:19.453 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:19.451+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:19.605 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:19.603+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:19.604064+0000) 2022-04-04T01:00:19.934 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:19.932+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:20.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:20.438+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:20.605 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:20.603+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:20.604223+0000) 2022-04-04T01:00:20.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:20.935+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:21.475 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:21.473+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:21.605 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:21.603+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:21.604417+0000) 2022-04-04T01:00:21.978 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:21.976+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:22.522 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:22.521+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:22.605 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:22.603+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:22.604607+0000) 2022-04-04T01:00:23.021 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:23.019+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:23.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:23.510+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:23.606 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:23.604+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:23.604860+0000) 2022-04-04T01:00:23.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:23.996+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:24.540 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:24.539+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:24.606 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:24.604+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:24.605023+0000) 2022-04-04T01:00:25.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:25.002+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:25.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:25.552+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:25.606 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:25.604+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:25.605211+0000) 2022-04-04T01:00:25.994 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:25.992+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:26.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:26.555+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:26.606 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:26.604+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:26.605416+0000) 2022-04-04T01:00:27.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:27.033+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:27.309 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:00:27.307+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:00:27.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:27.553+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:27.606 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:27.604+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:27.605628+0000) 2022-04-04T01:00:28.073 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:28.070+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:28.544 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:28.542+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:28.606 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:28.605+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:28.605835+0000) 2022-04-04T01:00:29.058 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:29.055+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:29.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:29.579+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:29.607 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:29.605+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:29.606051+0000) 2022-04-04T01:00:30.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:30.094+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:30.607 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:30.605+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:30.606203+0000) 2022-04-04T01:00:30.608 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:30.605+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:31.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:31.051+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:31.601 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:31.598+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:31.607 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:31.605+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:31.606418+0000) 2022-04-04T01:00:32.098 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:32.095+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:32.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:32.560+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:32.607 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:32.606+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:32.606597+0000) 2022-04-04T01:00:33.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:33.101+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:33.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:33.546+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:33.607 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:33.606+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:33.606808+0000) 2022-04-04T01:00:34.140 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:34.138+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:34.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:34.589+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:34.608 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:34.606+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:34.606954+0000) 2022-04-04T01:00:35.146 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:35.144+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:35.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:35.557+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:35.608 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:35.606+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:35.607131+0000) 2022-04-04T01:00:36.140 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:36.138+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:36.541 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:36.538+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:36.608 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:36.606+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:36.607252+0000) 2022-04-04T01:00:37.138 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:37.136+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:37.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:37.518+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:37.608 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:37.607+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:37.607450+0000) 2022-04-04T01:00:38.094 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:38.092+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:38.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:38.529+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:38.608 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:38.607+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:38.607665+0000) 2022-04-04T01:00:39.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:39.080+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:39.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:39.572+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:39.609 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:39.607+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:39.607839+0000) 2022-04-04T01:00:40.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:40.059+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:40.539 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:40.536+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:40.609 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:40.607+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:40.608015+0000) 2022-04-04T01:00:41.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:41.044+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:41.430 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:00:41.434 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:00:41.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:41.560+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:41.609 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:41.607+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:41.608260+0000) 2022-04-04T01:00:42.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:42.017+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:42.325 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:00:42.323+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:00:42.528 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:42.526+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:42.609 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:42.607+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:42.608496+0000) 2022-04-04T01:00:43.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:43.009+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:43.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:43.566+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:43.609 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:43.608+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:43.608643+0000) 2022-04-04T01:00:43.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:43.995+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:44.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:44.591+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:44.609 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:44.608+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:44.608864+0000) 2022-04-04T01:00:44.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:44.948+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:45.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:45.547+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:45.610 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:45.608+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:45.609093+0000) 2022-04-04T01:00:45.935 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:45.933+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:46.507 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:46.505+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:46.610 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:46.609+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:46.609304+0000) 2022-04-04T01:00:46.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:46.890+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:47.460 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:47.457+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:47.610 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:47.609+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:47.609512+0000) 2022-04-04T01:00:47.891 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:47.889+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:48.442 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:48.439+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:48.611 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:48.609+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:48.609721+0000) 2022-04-04T01:00:48.902 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:48.900+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:49.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:49.416+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:49.611 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:49.609+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:49.609918+0000) 2022-04-04T01:00:49.908 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:49.906+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:50.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:50.406+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:50.611 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:50.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:50.610137+0000) 2022-04-04T01:00:50.894 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:50.891+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:51.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:51.400+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:51.611 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:51.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:51.610345+0000) 2022-04-04T01:00:51.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:51.920+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:52.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:52.389+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:52.611 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:52.609+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:52.610560+0000) 2022-04-04T01:00:52.932 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:52.930+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:53.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:53.358+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:53.612 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:53.609+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:53.610784+0000) 2022-04-04T01:00:53.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:53.948+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:54.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:54.408+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:54.612 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:54.609+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:54.611009+0000) 2022-04-04T01:00:54.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:54.909+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:55.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:55.418+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:55.612 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:55.609+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:55.611172+0000) 2022-04-04T01:00:55.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:55.901+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:56.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:56.446+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:56.612 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:56.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:56.611348+0000) 2022-04-04T01:00:56.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:56.897+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:57.341 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:00:57.338+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:00:57.477 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:57.475+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:57.612 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:57.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:57.611567+0000) 2022-04-04T01:00:57.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:57.876+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:58.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:58.492+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:58.613 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:58.610+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:58.611830+0000) 2022-04-04T01:00:58.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:58.827+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:00:59.537 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:00:59.535+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:00:59.613 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:00:59.611+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:00:59.612063+0000) 2022-04-04T01:00:59.797 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:00:59.795+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:00.526 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:00.523+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:00.613 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:00.611+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:00.612271+0000) 2022-04-04T01:01:00.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:00.835+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:01.570 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:01.568+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:01.613 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:01.611+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:01.612540+0000) 2022-04-04T01:01:01.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:01.790+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:02.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:02.578+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:02.613 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:02.611+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:02.612735+0000) 2022-04-04T01:01:02.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:02.776+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:03.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:03.578+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:03.614 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:03.612+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:03.612952+0000) 2022-04-04T01:01:03.826 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:03.824+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:04.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:04.577+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:04.614 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:04.612+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:04.613175+0000) 2022-04-04T01:01:04.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:04.853+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:05.586 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:05.584+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:05.614 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:05.612+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:05.613364+0000) 2022-04-04T01:01:05.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:05.823+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:06.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:06.550+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:06.614 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:06.612+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:06.613424+0000) 2022-04-04T01:01:06.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:06.778+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:07.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:07.564+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:07.615 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:07.612+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:07.613610+0000) 2022-04-04T01:01:07.756 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:07.754+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:08.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:08.553+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:08.615 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:08.613+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:08.613726+0000) 2022-04-04T01:01:08.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:08.779+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:09.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:09.547+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:09.615 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:09.613+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:09.613882+0000) 2022-04-04T01:01:09.795 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:09.794+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:10.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:10.507+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:10.615 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:10.613+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:10.614025+0000) 2022-04-04T01:01:10.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:10.827+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:11.486 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:11.485+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:11.536 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:01:11.540 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:01:11.615 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:11.613+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:11.614233+0000) 2022-04-04T01:01:11.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:11.783+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:12.342 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:01:12.340+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:01:12.503 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:12.501+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:12.615 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:12.613+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:12.614475+0000) 2022-04-04T01:01:12.786 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:12.784+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:13.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:13.484+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:13.615 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:13.614+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:13.614691+0000) 2022-04-04T01:01:13.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:13.787+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:14.484 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:14.483+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:14.616 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:14.614+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:14.614872+0000) 2022-04-04T01:01:14.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:14.782+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:15.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:15.479+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:15.616 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:15.614+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:15.615047+0000) 2022-04-04T01:01:15.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:15.739+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:16.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:16.438+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:16.616 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:16.614+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:16.615266+0000) 2022-04-04T01:01:16.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:16.726+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:17.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:17.464+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:17.616 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:17.614+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:17.615492+0000) 2022-04-04T01:01:17.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:17.686+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:18.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:18.431+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:18.617 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:18.615+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:18.615744+0000) 2022-04-04T01:01:18.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:18.656+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:19.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:19.410+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:19.617 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:19.615+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:19.615969+0000) 2022-04-04T01:01:19.639 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:19.638+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:20.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:20.367+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:20.617 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:20.615+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:20.616166+0000) 2022-04-04T01:01:20.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:20.646+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:21.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:21.380+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:21.617 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:21.615+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:21.616330+0000) 2022-04-04T01:01:21.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:21.673+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:22.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:22.331+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:22.617 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:22.616+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:22.616525+0000) 2022-04-04T01:01:22.709 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:22.706+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:23.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:23.356+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:23.617 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:23.616+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:23.616665+0000) 2022-04-04T01:01:23.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:23.662+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:24.375 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:24.372+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:24.618 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:24.616+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:24.616832+0000) 2022-04-04T01:01:24.637 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:24.634+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:25.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:25.332+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:25.618 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:25.616+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:25.617057+0000) 2022-04-04T01:01:25.627 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:25.624+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:26.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:26.317+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:26.599 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:26.597+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:26.618 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:26.616+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:26.617256+0000) 2022-04-04T01:01:27.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:27.348+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:27.358 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:01:27.356+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:01:27.618 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:27.617+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:27.617430+0000) 2022-04-04T01:01:27.646 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:27.643+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:28.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:28.381+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:28.618 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:28.617+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:28.617611+0000) 2022-04-04T01:01:28.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:28.636+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:29.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:29.359+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:29.619 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:29.617+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:29.617849+0000) 2022-04-04T01:01:29.683 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:29.681+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:30.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:30.366+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:30.619 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:30.617+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:30.618014+0000) 2022-04-04T01:01:30.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:30.711+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:31.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:31.337+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:31.619 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:31.617+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:31.618202+0000) 2022-04-04T01:01:31.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:31.685+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:32.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:32.358+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:32.619 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:32.618+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:32.618392+0000) 2022-04-04T01:01:32.692 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:32.689+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:33.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:33.369+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:33.619 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:33.618+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:33.618584+0000) 2022-04-04T01:01:33.719 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:33.717+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:34.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:34.324+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:34.620 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:34.618+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:34.618801+0000) 2022-04-04T01:01:34.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:34.678+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:35.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:35.362+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:35.621 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:35.618+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:35.619040+0000) 2022-04-04T01:01:35.637 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:35.635+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:36.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:36.313+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:36.620 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:36.618+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:36.619273+0000) 2022-04-04T01:01:36.637 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:36.635+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:37.301 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:37.299+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:37.620 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:37.617+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:37.619510+0000) 2022-04-04T01:01:37.681 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:37.679+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:38.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:38.305+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:38.620 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:38.618+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:38.619696+0000) 2022-04-04T01:01:38.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:38.687+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:39.353 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:39.351+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:39.621 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:39.618+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:39.619943+0000) 2022-04-04T01:01:39.736 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:39.733+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:40.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:40.363+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:40.621 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:40.619+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:40.620206+0000) 2022-04-04T01:01:40.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:40.731+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:41.405 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:41.402+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:41.621 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:41.619+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:41.620449+0000) 2022-04-04T01:01:41.683 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:01:41.687 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:01:41.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:41.735+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:42.374 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:01:42.371+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:01:42.423 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:42.421+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:42.622 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:42.619+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:42.620705+0000) 2022-04-04T01:01:42.706 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:42.704+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:43.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:43.467+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:43.622 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:43.619+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:43.620941+0000) 2022-04-04T01:01:43.704 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:43.702+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:44.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:44.494+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:44.622 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:44.620+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:44.621182+0000) 2022-04-04T01:01:44.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:44.741+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:45.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:45.495+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:45.623 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:45.620+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:45.621443+0000) 2022-04-04T01:01:45.707 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:45.705+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:46.451 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:46.449+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:46.622 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:46.620+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:46.621690+0000) 2022-04-04T01:01:46.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:46.753+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:47.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:47.402+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:47.623 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:47.621+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:47.621933+0000) 2022-04-04T01:01:47.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:47.727+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:48.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:48.367+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:48.623 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:48.621+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:48.622108+0000) 2022-04-04T01:01:48.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:48.712+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:49.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:49.378+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:49.623 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:49.621+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:49.622339+0000) 2022-04-04T01:01:49.692 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:49.690+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:50.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:50.358+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:50.623 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:50.621+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:50.622577+0000) 2022-04-04T01:01:50.734 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:50.732+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:51.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:51.402+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:51.624 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:51.622+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:51.622836+0000) 2022-04-04T01:01:51.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:51.756+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:52.402 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:52.400+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:52.624 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:52.622+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:52.623115+0000) 2022-04-04T01:01:52.751 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:52.749+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:53.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:53.441+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:53.624 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:53.622+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:53.623357+0000) 2022-04-04T01:01:53.771 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:53.769+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:54.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:54.483+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:54.624 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:54.622+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:54.623581+0000) 2022-04-04T01:01:54.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:54.774+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:55.442 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:55.439+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:55.625 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:55.623+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:55.623782+0000) 2022-04-04T01:01:55.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:55.748+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:56.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:56.438+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:56.625 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:56.623+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:56.623998+0000) 2022-04-04T01:01:56.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:56.709+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:57.375 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:01:57.373+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:01:57.402 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:57.400+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:57.625 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:57.623+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:57.624177+0000) 2022-04-04T01:01:57.678 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:57.676+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:58.405 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:58.403+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:58.627 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:58.623+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:58.624394+0000) 2022-04-04T01:01:58.631 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:58.629+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:01:59.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:01:59.355+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:01:59.625 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:01:59.623+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:01:59.624679+0000) 2022-04-04T01:01:59.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:01:59.662+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:00.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:00.336+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:00.626 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:00.624+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:00.624925+0000) 2022-04-04T01:02:00.675 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:00.673+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:01.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:01.331+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:01.626 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:01.624+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:01.625042+0000) 2022-04-04T01:02:01.711 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:01.709+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:02.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:02.325+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:02.626 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:02.624+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:02.625294+0000) 2022-04-04T01:02:02.696 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:02.694+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:03.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:03.328+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:03.626 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:03.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:03.625524+0000) 2022-04-04T01:02:03.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:03.692+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:04.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:04.315+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:04.626 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:04.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:04.625692+0000) 2022-04-04T01:02:04.707 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:04.706+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:05.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:05.319+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:05.627 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:05.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:05.625859+0000) 2022-04-04T01:02:05.722 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:05.721+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:06.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:06.307+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:06.627 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:06.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:06.626082+0000) 2022-04-04T01:02:06.722 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:06.720+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:07.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:07.260+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:07.627 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:07.625+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:07.626324+0000) 2022-04-04T01:02:07.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:07.712+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:08.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:08.268+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:08.627 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:08.626+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:08.626587+0000) 2022-04-04T01:02:08.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:08.732+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:09.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:09.228+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:09.628 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:09.626+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:09.626814+0000) 2022-04-04T01:02:09.719 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:09.718+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:10.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:10.179+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:10.628 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:10.626+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:10.627053+0000) 2022-04-04T01:02:10.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:10.713+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:11.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:11.192+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:11.628 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:11.626+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:11.627299+0000) 2022-04-04T01:02:11.758 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:02:11.762 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:02:11.764 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:11.762+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:12.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:12.179+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:12.389 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:02:12.387+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:02:12.628 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:12.626+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:12.627573+0000) 2022-04-04T01:02:12.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:12.742+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:13.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:13.224+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:13.629 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:13.627+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:13.627815+0000) 2022-04-04T01:02:13.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:13.788+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:14.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:14.221+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:14.629 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:14.627+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:14.628084+0000) 2022-04-04T01:02:14.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:14.739+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:15.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:15.236+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:15.629 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:15.628+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:15.628322+0000) 2022-04-04T01:02:15.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:15.784+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:16.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:16.246+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:16.629 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:16.628+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:16.628501+0000) 2022-04-04T01:02:16.740 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:16.738+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:17.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:17.241+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:17.630 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:17.628+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:17.628714+0000) 2022-04-04T01:02:17.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:17.732+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:18.221 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:18.218+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 54 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:18.630 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:18.628+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:18.628962+0000) 2022-04-04T01:02:18.716 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:18.713+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:19.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:19.233+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:19.630 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:19.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:19.629208+0000) 2022-04-04T01:02:19.749 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:19.746+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:20.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:20.272+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:20.630 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:20.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:20.629447+0000) 2022-04-04T01:02:20.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:20.791+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:21.271 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:21.268+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:21.630 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:21.628+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:21.629643+0000) 2022-04-04T01:02:21.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:21.742+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:22.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:22.229+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:22.631 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:22.628+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:22.629883+0000) 2022-04-04T01:02:22.764 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:22.762+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:23.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:23.184+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:23.631 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:23.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:23.630097+0000) 2022-04-04T01:02:23.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:23.730+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:24.147 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:24.144+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:24.631 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:24.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:24.630310+0000) 2022-04-04T01:02:24.684 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:24.682+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:25.108 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:25.106+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:25.631 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:25.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:25.630520+0000) 2022-04-04T01:02:25.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:25.731+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:02:26.097 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:26.095+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:26.631 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:26.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:26.630709+0000) 2022-04-04T01:02:26.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:26.748+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:27.106 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:27.104+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:27.405 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:02:27.402+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:02:27.632 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:27.629+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:27.630928+0000) 2022-04-04T01:02:27.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:27.731+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:28.060 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:28.057+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:28.632 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:28.630+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:28.631123+0000) 2022-04-04T01:02:28.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:28.739+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:29.057 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:29.055+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:29.632 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:29.630+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:29.631312+0000) 2022-04-04T01:02:29.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:29.748+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:30.090 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:30.088+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:30.632 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:30.630+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:30.631534+0000) 2022-04-04T01:02:30.734 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:30.732+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:31.125 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:31.123+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:31.632 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:31.630+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:31.631704+0000) 2022-04-04T01:02:31.754 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:31.752+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:32.134 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:32.132+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:32.633 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:32.631+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:32.631896+0000) 2022-04-04T01:02:32.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:32.758+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:33.089 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:33.086+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:33.633 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:33.631+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:33.632086+0000) 2022-04-04T01:02:33.786 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:33.784+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:34.064 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:34.062+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:34.633 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:34.631+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:34.632313+0000) 2022-04-04T01:02:34.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:34.801+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:35.109 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:35.107+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:35.633 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:35.631+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:35.632576+0000) 2022-04-04T01:02:35.769 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:35.766+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:36.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:36.071+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:36.634 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:36.632+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:36.632785+0000) 2022-04-04T01:02:36.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:36.803+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:37.117 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:37.115+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:37.634 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:37.632+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:37.633005+0000) 2022-04-04T01:02:37.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:37.853+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:38.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:38.109+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:38.634 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:38.632+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:38.633264+0000) 2022-04-04T01:02:38.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:38.834+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:39.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:39.159+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:39.634 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:39.632+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:39.633513+0000) 2022-04-04T01:02:39.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:39.859+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:40.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:40.179+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:40.635 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:40.633+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:40.633763+0000) 2022-04-04T01:02:40.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:40.818+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:41.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:41.222+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:41.635 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:41.633+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:41.634014+0000) 2022-04-04T01:02:41.809 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:41.807+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:41.822 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:02:41.826 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:02:42.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:42.244+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:42.420 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:02:42.419+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:02:42.635 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:42.633+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:42.634281+0000) 2022-04-04T01:02:42.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:42.845+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:43.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:43.198+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:43.635 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:43.633+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:43.634503+0000) 2022-04-04T01:02:43.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:43.859+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:44.213 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:44.211+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:44.636 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:44.634+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:44.634791+0000) 2022-04-04T01:02:44.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:44.845+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:45.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:45.171+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:45.636 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:45.634+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:45.635062+0000) 2022-04-04T01:02:45.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:45.820+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:46.220 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:46.218+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:46.636 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:46.634+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:46.635228+0000) 2022-04-04T01:02:46.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:46.811+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:47.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:47.173+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:47.636 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:47.634+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:47.635444+0000) 2022-04-04T01:02:47.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:47.804+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:48.130 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:48.128+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:48.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:48.635+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:48.635653+0000) 2022-04-04T01:02:48.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:48.830+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:49.083 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:49.081+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:49.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:49.635+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:49.635858+0000) 2022-04-04T01:02:49.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:49.851+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:50.072 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:50.070+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:50.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:50.635+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:50.636085+0000) 2022-04-04T01:02:50.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:50.847+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:51.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:51.109+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:51.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:51.635+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:51.636275+0000) 2022-04-04T01:02:51.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:51.817+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:52.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:52.110+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:52.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:52.636+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:52.636471+0000) 2022-04-04T01:02:52.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:52.829+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:53.089 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:53.086+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:53.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:53.636+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:53.636647+0000) 2022-04-04T01:02:53.816 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:53.814+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:54.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:54.079+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:54.637 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:54.636+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:54.636864+0000) 2022-04-04T01:02:54.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:54.804+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:55.111 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:55.109+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:55.638 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:55.636+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:55.637028+0000) 2022-04-04T01:02:55.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:55.777+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:56.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:56.110+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:56.638 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:56.636+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:56.637185+0000) 2022-04-04T01:02:56.765 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:56.763+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:57.082 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:57.081+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:57.436 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:02:57.434+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:02:57.638 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:57.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:57.637378+0000) 2022-04-04T01:02:57.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:57.747+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:58.041 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:58.040+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:58.638 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:58.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:58.637605+0000) 2022-04-04T01:02:58.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:58.744+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:59.005 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:59.004+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:02:59.639 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:02:59.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:02:59.637842+0000) 2022-04-04T01:02:59.705 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:02:59.703+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:02:59.984 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:02:59.982+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:00.639 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:00.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:00.638016+0000) 2022-04-04T01:03:00.692 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:00.691+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:00.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:00.993+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:01.639 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:01.638+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:01.638160+0000) 2022-04-04T01:03:01.705 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:01.704+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:01.963 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:01.961+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:02.639 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:02.638+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:02.638303+0000) 2022-04-04T01:03:02.659 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:02.657+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:02.967 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:02.966+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:03.639 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:03.638+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:03.638437+0000) 2022-04-04T01:03:03.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:03.687+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:03.966 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:03.964+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:04.639 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:04.638+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:04.638523+0000) 2022-04-04T01:03:04.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:04.638+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:04.987 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:04.986+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:05.639 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:05.638+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:05.640 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:05.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:05.638729+0000) 2022-04-04T01:03:06.027 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:06.025+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:06.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:06.627+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:06.640 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:06.637+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:06.638933+0000) 2022-04-04T01:03:06.986 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:06.984+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:07.640 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:07.638+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:07.639084+0000) 2022-04-04T01:03:07.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:07.639+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:07.983 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:07.981+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:08.640 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:08.638+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:08.639284+0000) 2022-04-04T01:03:08.653 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:08.652+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:08.963 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:08.961+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:09.640 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:09.638+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:09.639487+0000) 2022-04-04T01:03:09.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:09.643+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:09.923 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:09.922+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:10.640 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:10.638+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:10.639640+0000) 2022-04-04T01:03:10.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:10.662+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:10.949 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:10.948+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:11.640 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:11.638+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:11.639793+0000) 2022-04-04T01:03:11.708 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:11.707+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:11.951 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:03:11.955 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:03:11.988 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:11.986+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:12.452 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:03:12.450+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:03:12.641 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:12.639+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:12.639966+0000) 2022-04-04T01:03:12.718 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:12.715+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:12.950 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:12.948+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:13.641 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:13.639+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:13.640114+0000) 2022-04-04T01:03:13.668 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-04T01:03:13.665+0000 7f2e43b1f700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T01:03:13.668 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-04T01:03:13.665+0000 7f2e44320700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T01:03:13.706 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:13.704+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:13.999 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:13.996+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:14.641 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:14.639+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:14.640328+0000) 2022-04-04T01:03:14.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:14.730+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:14.986 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:14.983+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:15.641 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:15.639+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:15.640493+0000) 2022-04-04T01:03:15.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:15.757+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:15.962 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:15.960+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:16.641 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:16.639+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:16.640666+0000) 2022-04-04T01:03:16.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:16.791+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:16.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:16.913+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:17.642 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:17.640+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:17.640895+0000) 2022-04-04T01:03:17.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:17.747+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:17.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:17.930+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:18.642 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:18.640+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:18.641111+0000) 2022-04-04T01:03:18.751 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:18.748+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:18.965 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:18.962+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:19.642 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:19.640+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:19.641241+0000) 2022-04-04T01:03:19.751 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:19.748+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:19.939 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:19.936+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:20.642 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:20.640+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:20.641379+0000) 2022-04-04T01:03:20.708 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:20.706+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:20.913 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:20.910+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:21.642 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:21.640+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:21.641572+0000) 2022-04-04T01:03:21.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:21.692+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:21.866 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:21.864+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:22.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:22.641+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:22.641709+0000) 2022-04-04T01:03:22.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:22.697+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:22.901 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:22.898+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:23.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:23.641+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:23.641864+0000) 2022-04-04T01:03:23.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:23.746+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:23.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:23.860+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:24.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:24.641+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:24.642027+0000) 2022-04-04T01:03:24.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:24.774+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:24.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:24.875+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:25.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:25.641+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:25.642212+0000) 2022-04-04T01:03:25.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:25.725+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:25.837 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:25.834+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:26.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:26.641+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:26.642366+0000) 2022-04-04T01:03:26.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:26.712+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:26.854 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:26.851+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:27.468 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:03:27.466+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:03:27.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:27.641+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:27.642504+0000) 2022-04-04T01:03:27.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:27.723+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:27.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:27.813+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:28.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:28.642+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:28.642660+0000) 2022-04-04T01:03:28.752 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:28.749+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:28.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:28.860+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:29.643 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:29.642+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:29.642810+0000) 2022-04-04T01:03:29.751 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:29.749+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:29.865 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:29.863+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:30.644 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:30.642+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:30.642984+0000) 2022-04-04T01:03:30.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:30.710+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:30.904 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:30.902+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:31.644 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:31.642+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:31.643143+0000) 2022-04-04T01:03:31.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:31.729+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:31.924 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:31.922+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:32.644 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:32.642+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:32.643347+0000) 2022-04-04T01:03:32.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:32.680+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:32.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:32.951+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:33.644 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:33.643+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:33.643460+0000) 2022-04-04T01:03:33.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:33.667+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:33.976 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:33.974+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:34.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:34.619+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:34.644 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:34.643+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:34.643615+0000) 2022-04-04T01:03:35.021 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:35.019+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:35.586 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:35.583+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:35.645 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:35.643+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:35.643852+0000) 2022-04-04T01:03:36.061 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:36.059+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:36.599 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:36.597+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:36.645 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:36.643+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:36.644045+0000) 2022-04-04T01:03:37.084 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:37.082+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:37.591 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:37.588+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:37.645 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:37.643+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:37.644195+0000) 2022-04-04T01:03:38.106 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:38.104+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:38.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:38.544+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:38.645 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:38.644+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:38.644391+0000) 2022-04-04T01:03:39.109 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:39.106+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:39.540 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:39.538+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:39.645 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:39.644+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:39.644552+0000) 2022-04-04T01:03:40.132 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:40.130+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:40.494 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:40.492+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:40.646 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:40.644+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:40.644806+0000) 2022-04-04T01:03:41.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:41.176+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:41.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:41.521+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:41.646 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:41.644+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:41.644987+0000) 2022-04-04T01:03:42.065 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:03:42.068 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:03:42.220 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:42.217+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:42.478 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:42.476+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:42.484 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:03:42.482+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:03:42.646 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:42.644+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:42.645254+0000) 2022-04-04T01:03:43.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:43.174+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:43.471 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:43.468+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:43.646 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:43.645+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:43.645413+0000) 2022-04-04T01:03:44.213 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:44.210+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:44.496 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:44.494+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:44.646 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:44.645+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:44.645620+0000) 2022-04-04T01:03:45.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:45.253+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:45.465 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:45.462+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:45.647 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:45.645+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:45.645866+0000) 2022-04-04T01:03:46.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:46.206+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:46.442 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:46.439+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:46.647 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:46.645+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:46.646074+0000) 2022-04-04T01:03:47.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:47.227+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:47.409 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:47.406+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:47.647 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:47.646+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:47.646191+0000) 2022-04-04T01:03:48.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:48.204+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:48.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:48.402+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:48.647 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:48.646+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:48.646327+0000) 2022-04-04T01:03:49.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:49.205+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:49.449 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:49.447+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:49.647 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:49.646+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:49.646520+0000) 2022-04-04T01:03:50.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:50.242+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:50.412 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:50.410+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:50.648 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:50.645+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:50.646718+0000) 2022-04-04T01:03:51.221 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:51.219+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:51.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:51.445+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:51.648 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:51.645+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:51.646937+0000) 2022-04-04T01:03:52.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:52.260+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:52.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:52.412+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:52.648 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:52.646+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:52.647120+0000) 2022-04-04T01:03:53.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:53.286+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:53.453 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:53.451+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:53.648 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:53.646+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:53.647325+0000) 2022-04-04T01:03:54.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:54.265+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:54.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:54.442+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:54.648 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:54.646+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:54.647509+0000) 2022-04-04T01:03:55.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:55.250+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:55.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:55.444+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:55.648 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:55.646+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:55.647627+0000) 2022-04-04T01:03:56.220 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:56.218+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:56.473 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:56.472+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:56.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:56.646+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:56.647822+0000) 2022-04-04T01:03:57.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:57.228+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:57.500 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:03:57.497+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:03:57.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:57.520+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:57.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:57.647+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:57.648044+0000) 2022-04-04T01:03:58.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:58.215+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:58.498 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:58.497+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:58.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:58.647+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:58.648142+0000) 2022-04-04T01:03:59.170 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:03:59.168+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:03:59.490 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:03:59.489+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:03:59.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:03:59.647+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:03:59.648297+0000) 2022-04-04T01:04:00.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:00.198+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:00.509 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:00.508+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:00.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:00.647+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:00.648458+0000) 2022-04-04T01:04:01.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:01.242+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:01.500 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:01.499+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:01.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:01.647+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:01.648633+0000) 2022-04-04T01:04:02.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:02.232+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:02.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:02.542+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:02.649 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:02.648+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:02.648790+0000) 2022-04-04T01:04:03.284 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:03.282+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:03.559 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:03.557+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:03.650 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:03.648+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:03.648959+0000) 2022-04-04T01:04:04.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:04.313+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:04.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:04.542+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:04.650 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:04.648+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:04.649131+0000) 2022-04-04T01:04:05.284 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:05.283+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:05.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:05.534+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:05.650 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:05.648+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:05.649253+0000) 2022-04-04T01:04:06.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:06.319+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:06.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:06.583+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:06.650 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:06.648+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:06.649396+0000) 2022-04-04T01:04:07.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:07.286+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:07.556 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:07.554+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:07.650 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:07.648+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:07.649525+0000) 2022-04-04T01:04:08.283 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:08.280+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:08.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:08.546+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:08.650 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:08.648+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:08.649625+0000) 2022-04-04T01:04:09.252 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:09.250+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:09.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:09.542+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:09.651 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:09.649+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:09.649812+0000) 2022-04-04T01:04:10.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:10.269+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:10.593 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:10.590+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:10.651 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:10.649+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:10.650035+0000) 2022-04-04T01:04:11.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:11.295+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:11.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:11.542+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:11.651 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:11.649+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:11.650193+0000) 2022-04-04T01:04:12.221 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:04:12.224 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:04:12.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:12.303+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:12.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:12.509+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:12.515 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:04:12.514+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:04:12.651 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:12.649+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:12.650414+0000) 2022-04-04T01:04:13.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:13.325+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:13.499 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:13.497+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:13.651 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:13.650+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:13.650582+0000) 2022-04-04T01:04:14.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:14.369+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:14.468 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:14.466+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:14.651 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:14.650+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:14.650838+0000) 2022-04-04T01:04:15.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:15.395+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:15.432 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:15.430+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:15.652 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:15.650+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:15.651021+0000) 2022-04-04T01:04:16.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:16.415+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:16.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:16.417+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:16.652 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:16.650+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:16.651157+0000) 2022-04-04T01:04:17.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:17.367+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:17.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:17.453+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:17.652 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:17.650+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:17.651270+0000) 2022-04-04T01:04:18.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:18.359+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:18.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:18.405+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:18.652 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:18.650+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:18.651443+0000) 2022-04-04T01:04:19.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:19.320+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:19.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:19.373+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:19.652 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:19.651+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:19.651670+0000) 2022-04-04T01:04:20.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:20.333+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:20.424 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:20.422+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:20.653 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:20.651+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:20.651875+0000) 2022-04-04T01:04:21.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:21.377+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:21.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:21.415+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:21.653 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:21.651+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:21.652057+0000) 2022-04-04T01:04:22.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:22.391+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:22.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:22.415+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:22.653 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:22.651+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:22.652279+0000) 2022-04-04T01:04:23.371 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:23.369+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:23.457 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:23.455+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:23.653 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:23.652+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:23.652405+0000) 2022-04-04T01:04:24.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:24.325+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:24.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:24.459+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:24.653 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:24.652+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:24.652588+0000) 2022-04-04T01:04:25.322 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:25.320+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:25.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:25.428+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:25.653 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:25.652+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:25.652758+0000) 2022-04-04T01:04:26.337 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:26.335+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:26.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:26.443+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:26.654 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:26.652+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:26.652975+0000) 2022-04-04T01:04:27.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:27.318+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:27.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:27.427+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:27.531 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:04:27.530+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:04:27.654 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:27.652+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:27.653096+0000) 2022-04-04T01:04:28.370 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:28.368+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:28.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:28.444+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:28.654 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:28.653+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:28.653289+0000) 2022-04-04T01:04:29.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:29.383+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:29.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:29.428+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:29.654 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:29.653+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:29.653497+0000) 2022-04-04T01:04:30.412 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:30.410+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:30.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:30.416+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:30.654 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:30.653+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:30.653662+0000) 2022-04-04T01:04:31.428 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:31.426+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:31.458 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:31.456+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:31.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:31.653+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:31.653860+0000) 2022-04-04T01:04:32.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:32.408+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:32.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:32.414+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:32.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:32.653+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:32.654061+0000) 2022-04-04T01:04:33.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:33.432+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:33.452 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:33.450+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:33.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:33.654+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:33.654247+0000) 2022-04-04T01:04:34.428 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:34.425+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:34.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:34.454+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:34.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:34.654+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:34.654365+0000) 2022-04-04T01:04:35.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:35.384+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:35.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:35.484+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:35.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:35.653+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:35.654538+0000) 2022-04-04T01:04:36.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:36.379+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:36.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:36.519+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:36.655 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:36.653+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:36.654736+0000) 2022-04-04T01:04:37.412 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:37.410+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:37.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:37.534+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:37.656 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:37.653+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:37.654997+0000) 2022-04-04T01:04:38.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:38.412+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:38.499 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:38.497+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:38.656 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:38.654+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:38.655194+0000) 2022-04-04T01:04:39.422 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:39.420+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:39.478 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:39.476+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:39.656 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:39.654+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:39.655410+0000) 2022-04-04T01:04:40.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:40.395+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:40.472 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:40.470+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:40.656 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:40.654+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:40.655627+0000) 2022-04-04T01:04:41.428 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:41.425+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:41.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:41.442+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:41.657 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:41.654+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:41.655834+0000) 2022-04-04T01:04:42.282 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:04:42.286 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:04:42.453 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:42.452+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:42.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:42.478+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:42.547 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:04:42.545+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:04:42.657 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:42.655+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:42.656082+0000) 2022-04-04T01:04:43.450 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:43.448+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:43.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:43.458+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:43.657 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:43.655+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:43.656269+0000) 2022-04-04T01:04:44.471 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:44.469+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:44.480 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:44.478+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:44.657 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:44.655+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:44.656520+0000) 2022-04-04T01:04:45.482 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:45.481+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:45.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:45.490+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:45.657 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:45.656+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:45.656801+0000) 2022-04-04T01:04:46.468 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:46.466+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:46.504 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:46.502+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:46.658 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:46.656+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:46.657028+0000) 2022-04-04T01:04:47.426 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:47.425+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:47.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:47.455+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:47.658 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:47.656+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:47.657202+0000) 2022-04-04T01:04:48.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:48.408+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:48.487 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:48.485+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:48.658 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:48.656+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:48.657416+0000) 2022-04-04T01:04:49.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:49.448+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:49.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:49.509+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:49.658 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:49.656+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:49.657602+0000) 2022-04-04T01:04:50.480 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:50.478+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:50.509 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:50.508+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:50.658 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:50.657+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:50.657750+0000) 2022-04-04T01:04:51.475 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:51.473+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:51.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:51.543+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:51.659 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:51.657+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:51.657940+0000) 2022-04-04T01:04:52.480 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:52.478+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:52.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:52.500+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:52.659 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:52.657+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:52.658108+0000) 2022-04-04T01:04:53.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:53.451+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:53.520 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:53.518+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:53.659 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:53.657+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:53.658300+0000) 2022-04-04T01:04:54.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:54.423+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:54.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:54.547+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:54.659 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:54.657+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:54.658523+0000) 2022-04-04T01:04:55.432 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:55.430+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:55.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:55.558+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:55.659 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:55.658+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:55.658741+0000) 2022-04-04T01:04:56.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:56.457+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:56.586 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:56.585+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:56.659 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:56.658+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:56.658970+0000) 2022-04-04T01:04:57.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:57.415+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:57.550 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:57.548+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:57.563 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:04:57.561+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:04:57.660 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:57.658+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:57.659159+0000) 2022-04-04T01:04:58.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:58.390+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:58.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:58.529+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:58.660 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:58.658+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:58.659347+0000) 2022-04-04T01:04:59.364 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:04:59.362+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:04:59.522 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:04:59.521+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:04:59.660 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:04:59.659+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:04:59.659579+0000) 2022-04-04T01:05:00.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:00.384+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:00.557 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:00.555+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:00.660 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:00.659+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:00.659762+0000) 2022-04-04T01:05:01.412 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:01.410+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:01.555 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:01.552+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:01.661 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:01.659+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:01.659980+0000) 2022-04-04T01:05:02.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:02.402+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:02.568 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:02.565+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:02.661 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:02.659+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:02.660221+0000) 2022-04-04T01:05:03.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:03.369+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:03.609 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:03.606+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:03.661 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:03.659+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:03.660448+0000) 2022-04-04T01:05:04.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:04.360+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:04.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:04.578+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:04.661 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:04.660+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:04.660647+0000) 2022-04-04T01:05:05.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:05.328+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:05.557 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:05.554+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:05.662 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:05.660+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:05.660844+0000) 2022-04-04T01:05:06.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:06.326+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:06.550 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:06.548+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:06.662 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:06.660+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:06.661055+0000) 2022-04-04T01:05:07.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:07.279+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:07.569 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:07.566+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:07.662 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:07.660+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:07.661210+0000) 2022-04-04T01:05:08.332 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:08.329+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:08.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:08.546+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:08.662 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:08.661+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:08.661348+0000) 2022-04-04T01:05:09.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:09.282+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:09.506 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:09.504+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:09.662 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:09.661+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:09.661494+0000) 2022-04-04T01:05:10.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:10.241+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:10.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:10.458+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:10.662 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:10.661+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:10.661633+0000) 2022-04-04T01:05:11.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:11.291+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:11.411 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:11.409+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:11.662 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:11.661+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:11.661757+0000) 2022-04-04T01:05:12.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:12.301+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:12.414 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:05:12.419 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:05:12.422 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:12.418+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:12.579 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:05:12.577+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:05:12.663 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:12.661+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:12.662004+0000) 2022-04-04T01:05:13.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:13.293+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:13.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:13.458+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:13.663 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:13.661+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:13.662167+0000) 2022-04-04T01:05:14.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:14.296+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:14.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:14.440+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:14.663 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:14.662+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:14.662362+0000) 2022-04-04T01:05:15.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:15.267+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:15.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:15.399+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:15.663 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:15.662+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:15.662536+0000) 2022-04-04T01:05:16.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:16.314+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:16.394 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:16.391+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:16.663 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:16.662+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:16.662681+0000) 2022-04-04T01:05:17.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:17.339+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:17.422 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:17.420+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:17.664 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:17.662+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:17.662884+0000) 2022-04-04T01:05:18.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:18.308+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:18.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:18.413+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:18.664 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:18.662+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:18.663086+0000) 2022-04-04T01:05:19.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:19.314+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:19.432 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:19.430+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:19.664 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:19.662+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:19.663299+0000) 2022-04-04T01:05:20.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:20.293+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:20.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:20.462+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:20.664 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:20.661+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:20.663470+0000) 2022-04-04T01:05:21.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:21.274+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:21.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:21.437+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:21.664 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:21.661+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:21.663622+0000) 2022-04-04T01:05:21.665 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:21.663+0000 7fd623a1a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T01:05:21.666 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:21.663+0000 7fd62421b700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T01:05:21.668 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:21.666+0000 7fd623a1a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:21.667945+0000) 2022-04-04T01:05:22.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:22.289+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:22.447 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:22.445+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:22.664 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:22.662+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:22.663794+0000) 2022-04-04T01:05:23.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:23.291+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:23.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:23.493+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:23.665 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:23.662+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:23.663929+0000) 2022-04-04T01:05:24.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:24.305+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:24.533 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:24.530+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:24.665 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:24.663+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:24.664074+0000) 2022-04-04T01:05:25.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:25.274+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:25.582 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:25.579+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:05:25.665 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:25.663+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:25.664182+0000) 2022-04-04T01:05:26.235 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:26.233+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:26.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:26.627+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:26.665 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:26.662+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:26.664354+0000) 2022-04-04T01:05:27.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:27.241+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:27.595 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:05:27.592+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:05:27.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:27.609+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:27.665 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:27.663+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:27.664526+0000) 2022-04-04T01:05:28.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:28.248+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:28.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:28.654+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:28.665 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:28.663+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:28.664635+0000) 2022-04-04T01:05:29.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:29.234+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:29.666 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:29.664+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:29.664816+0000) 2022-04-04T01:05:29.689 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:29.686+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:30.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:30.243+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:30.666 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:30.664+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:30.664959+0000) 2022-04-04T01:05:30.685 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:30.683+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:31.213 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:31.211+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:31.666 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:31.664+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:31.665126+0000) 2022-04-04T01:05:31.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:31.733+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:32.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:32.256+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:32.666 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:32.664+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:32.665287+0000) 2022-04-04T01:05:32.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:32.754+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:33.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:33.253+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:33.666 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:33.664+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:33.665462+0000) 2022-04-04T01:05:33.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:33.726+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:34.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:34.248+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:34.666 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:34.664+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:34.665582+0000) 2022-04-04T01:05:34.703 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:34.702+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:35.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:35.276+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:35.666 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:35.665+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:35.665716+0000) 2022-04-04T01:05:35.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:35.674+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:36.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:36.275+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:36.667 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:36.665+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:36.665841+0000) 2022-04-04T01:05:36.697 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:36.695+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:37.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:37.239+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:37.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:37.662+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:37.666 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:37.665+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:37.665957+0000) 2022-04-04T01:05:38.220 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:38.218+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:38.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:38.618+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:38.667 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:38.665+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:38.666122+0000) 2022-04-04T01:05:39.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:39.255+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:39.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:39.657+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:39.667 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:39.665+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:39.666252+0000) 2022-04-04T01:05:40.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:40.283+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:40.636 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:40.634+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:40.667 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:40.665+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:40.666444+0000) 2022-04-04T01:05:41.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:41.272+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:41.597 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:41.595+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:41.667 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:41.666+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:41.666590+0000) 2022-04-04T01:05:42.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:42.275+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:42.610 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:05:42.609+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:05:42.611 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:05:42.615 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:05:42.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:42.643+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:42.667 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:42.666+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:42.666824+0000) 2022-04-04T01:05:43.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:43.278+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:43.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:43.619+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:43.667 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:43.666+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:43.666988+0000) 2022-04-04T01:05:44.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:44.275+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:44.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:44.571+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:44.668 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:44.666+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:44.667110+0000) 2022-04-04T01:05:45.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:45.259+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:45.599 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:45.598+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:45.668 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:45.666+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:45.667249+0000) 2022-04-04T01:05:46.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:46.248+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:46.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:46.612+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:46.668 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:46.667+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:46.667442+0000) 2022-04-04T01:05:47.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:47.292+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:47.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:47.627+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:47.668 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:47.666+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:47.667608+0000) 2022-04-04T01:05:48.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:48.306+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:48.669 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:48.667+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:48.667823+0000) 2022-04-04T01:05:48.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:48.675+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:49.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:49.341+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:49.669 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:49.667+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:49.667996+0000) 2022-04-04T01:05:49.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:49.723+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:50.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:50.324+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:50.669 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:50.667+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:50.668153+0000) 2022-04-04T01:05:50.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:50.698+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:51.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:51.354+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:51.669 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:51.667+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:51.668263+0000) 2022-04-04T01:05:51.720 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:51.719+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:52.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:52.306+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:52.669 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:52.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:52.668491+0000) 2022-04-04T01:05:52.719 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:52.718+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:53.271 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:53.269+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:53.669 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:53.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:53.668629+0000) 2022-04-04T01:05:53.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:53.727+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:54.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:54.222+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:54.669 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:54.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:54.668752+0000) 2022-04-04T01:05:54.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:54.751+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:55.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:55.177+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:55.670 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:55.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:55.668924+0000) 2022-04-04T01:05:55.765 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:55.764+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:56.136 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:56.134+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:56.670 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:56.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:56.669047+0000) 2022-04-04T01:05:56.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:56.748+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:57.143 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:57.141+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:57.627 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:05:57.625+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:05:57.670 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:57.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:57.669231+0000) 2022-04-04T01:05:57.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:57.729+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:58.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:58.186+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:58.670 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:58.669+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:58.669416+0000) 2022-04-04T01:05:58.738 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:58.736+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:05:59.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:05:59.190+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:05:59.670 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:05:59.668+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:05:59.669612+0000) 2022-04-04T01:05:59.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:05:59.770+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:00.191 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:00.189+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:00.671 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:00.669+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:00.669814+0000) 2022-04-04T01:06:00.767 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:00.764+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:01.176 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:01.174+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:01.671 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:01.669+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:01.669961+0000) 2022-04-04T01:06:01.790 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:01.788+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:02.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:02.152+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:02.671 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:02.670+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:02.670153+0000) 2022-04-04T01:06:02.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:02.777+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:03.138 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:03.136+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:03.671 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:03.670+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:03.670306+0000) 2022-04-04T01:06:03.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:03.760+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:04.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:04.161+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:04.671 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:04.669+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:04.670446+0000) 2022-04-04T01:06:04.773 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:04.771+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:05.127 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:05.124+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:05.671 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:05.669+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:05.670598+0000) 2022-04-04T01:06:05.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:05.724+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:06.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:06.154+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:06.671 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:06.669+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:06.670815+0000) 2022-04-04T01:06:06.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:06.768+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:07.139 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:07.137+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:07.672 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:07.670+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:07.670987+0000) 2022-04-04T01:06:07.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:07.818+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:08.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:08.165+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:08.672 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:08.670+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:08.671137+0000) 2022-04-04T01:06:08.856 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:08.854+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:09.146 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:09.144+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:09.672 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:09.670+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:09.671269+0000) 2022-04-04T01:06:09.898 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:09.895+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:10.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:10.184+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:10.672 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:10.670+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:10.671437+0000) 2022-04-04T01:06:10.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:10.876+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:11.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:11.225+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:11.672 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:11.670+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:11.671618+0000) 2022-04-04T01:06:11.852 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:11.850+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:12.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:12.205+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:12.642 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:06:12.640+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:06:12.672 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:12.670+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:12.671820+0000) 2022-04-04T01:06:12.682 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:06:12.686 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:06:12.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:12.879+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:13.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:13.190+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:13.673 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:13.671+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:13.671985+0000) 2022-04-04T01:06:13.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:13.886+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:14.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:14.207+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:14.673 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:14.671+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:14.672155+0000) 2022-04-04T01:06:14.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:14.909+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:15.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:15.234+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:15.673 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:15.671+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:15.672350+0000) 2022-04-04T01:06:15.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:15.945+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:16.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:16.258+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:16.673 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:16.671+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:16.672530+0000) 2022-04-04T01:06:16.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:16.963+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:17.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:17.234+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:17.674 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:17.671+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:17.672695+0000) 2022-04-04T01:06:17.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:17.919+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:18.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:18.277+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:18.674 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:18.672+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:18.672820+0000) 2022-04-04T01:06:18.889 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:18.887+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:19.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:19.308+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:19.674 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:19.672+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:19.672911+0000) 2022-04-04T01:06:19.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:19.856+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:20.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:20.302+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:20.674 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:20.672+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:20.673080+0000) 2022-04-04T01:06:20.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:20.834+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:21.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:21.274+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:21.674 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:21.672+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:21.673257+0000) 2022-04-04T01:06:21.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:21.817+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:22.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:22.258+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:22.674 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:22.672+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:22.673473+0000) 2022-04-04T01:06:22.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:22.820+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:23.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:23.259+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:23.674 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:23.673+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:23.673641+0000) 2022-04-04T01:06:23.859 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:23.857+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:24.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:24.268+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:24.674 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:24.673+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:24.673802+0000) 2022-04-04T01:06:24.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:24.836+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:25.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:25.309+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:25.675 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:25.673+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:25.673982+0000) 2022-04-04T01:06:25.705 INFO:tasks.ceph.mds.e.smithi077.stderr:2022-04-04T01:06:25.704+0000 7f2e44320700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T01:06:25.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:25.868+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:26.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:26.359+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:26.675 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:26.673+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:26.674105+0000) 2022-04-04T01:06:26.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:26.826+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:27.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:27.365+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:27.658 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:06:27.656+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:06:27.675 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:27.673+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:27.674283+0000) 2022-04-04T01:06:27.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:27.804+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:28.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:28.356+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:28.675 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:28.673+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:28.674414+0000) 2022-04-04T01:06:28.777 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:28.775+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:29.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:29.385+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:29.675 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:29.674+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:29.674565+0000) 2022-04-04T01:06:29.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:29.780+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:30.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:30.401+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:30.675 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:30.674+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:30.674757+0000) 2022-04-04T01:06:30.826 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:30.824+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:31.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:31.362+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:31.676 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:31.674+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:31.674934+0000) 2022-04-04T01:06:31.834 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:31.832+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:32.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:32.375+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:32.676 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:32.674+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:32.675113+0000) 2022-04-04T01:06:32.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:32.879+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:33.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:33.356+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:33.676 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:33.674+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:33.675321+0000) 2022-04-04T01:06:33.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:33.844+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:34.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:34.322+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:34.676 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:34.675+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:34.675462+0000) 2022-04-04T01:06:34.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:34.859+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:35.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:35.340+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:35.676 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:35.675+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:35.675625+0000) 2022-04-04T01:06:35.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:35.841+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:36.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:36.330+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:36.677 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:36.675+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:36.675750+0000) 2022-04-04T01:06:36.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:36.861+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:37.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:37.326+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:37.677 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:37.675+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:37.675921+0000) 2022-04-04T01:06:37.856 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:37.854+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:38.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:38.331+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:38.677 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:38.675+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:38.676066+0000) 2022-04-04T01:06:38.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:38.829+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:39.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:39.291+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:39.677 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:39.675+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:39.676194+0000) 2022-04-04T01:06:39.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:39.823+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:40.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:40.289+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:40.677 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:40.676+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:40.676336+0000) 2022-04-04T01:06:40.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:40.828+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:41.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:41.291+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:41.677 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:41.676+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:41.676540+0000) 2022-04-04T01:06:41.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:41.829+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:42.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:42.325+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:42.674 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:06:42.672+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:06:42.677 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:42.675+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:42.676691+0000) 2022-04-04T01:06:42.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:42.798+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:42.806 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:06:42.810 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:06:43.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:43.347+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:43.678 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:43.676+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:43.676867+0000) 2022-04-04T01:06:43.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:43.818+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:44.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:44.378+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:44.678 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:44.676+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:44.677053+0000) 2022-04-04T01:06:44.786 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:44.785+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:45.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:45.361+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:45.678 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:45.677+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:45.677245+0000) 2022-04-04T01:06:45.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:45.834+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:46.375 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:46.373+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:46.678 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:46.677+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:46.677410+0000) 2022-04-04T01:06:46.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:46.837+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:47.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:47.366+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:47.678 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:47.677+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:47.677494+0000) 2022-04-04T01:06:47.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:47.862+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:48.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:48.384+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:48.678 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:48.676+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:48.677614+0000) 2022-04-04T01:06:48.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:48.875+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:49.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:49.404+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:49.678 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:49.676+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:49.677791+0000) 2022-04-04T01:06:49.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:49.875+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:50.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:50.450+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:50.679 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:50.676+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:50.677956+0000) 2022-04-04T01:06:50.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:50.898+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:51.496 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:51.493+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:51.679 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:51.676+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:51.678065+0000) 2022-04-04T01:06:51.929 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:51.926+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:52.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:52.449+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:52.679 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:52.677+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:52.678227+0000) 2022-04-04T01:06:52.906 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:52.903+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:53.473 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:53.470+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:53.679 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:53.677+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:53.678395+0000) 2022-04-04T01:06:53.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:53.949+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:54.484 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:54.482+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:54.679 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:54.677+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:54.678538+0000) 2022-04-04T01:06:54.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:54.955+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:55.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:55.440+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:55.679 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:55.677+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:55.678726+0000) 2022-04-04T01:06:55.980 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:55.978+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:56.422 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:56.419+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:56.680 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:56.678+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:56.678931+0000) 2022-04-04T01:06:56.980 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:56.978+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:57.432 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:57.429+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:57.680 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:57.678+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:57.679072+0000) 2022-04-04T01:06:57.690 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:06:57.687+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:06:57.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:57.944+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:58.478 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:58.475+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:58.680 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:58.678+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:58.679216+0000) 2022-04-04T01:06:58.908 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:58.905+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:06:59.480 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:06:59.477+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:06:59.680 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:06:59.678+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:06:59.679357+0000) 2022-04-04T01:06:59.939 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:06:59.937+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:00.523 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:00.521+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:00.680 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:00.678+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:00.679500+0000) 2022-04-04T01:07:00.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:00.972+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:01.498 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:01.496+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:01.680 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:01.678+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:01.679660+0000) 2022-04-04T01:07:02.005 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:02.003+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:02.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:02.536+0000 7f88d0ac1700 -1 osd.4 138 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:02.680 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:02.679+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:02.679855+0000) 2022-04-04T01:07:02.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:02.997+0000 7fd3b190b700 -1 osd.6 138 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:03.309 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-04T01:07:03.307+0000 7f70e23d9700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-04T01:07:03.325 INFO:tasks.ceph.mds.f.smithi186.stderr: -1> 2022-04-04T01:07:03.307+0000 7f70e23d9700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-04T01:07:03.346 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-04T01:07:03.552 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:03.549+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:03.681 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:03.679+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:03.679992+0000) 2022-04-04T01:07:03.972 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:03.967+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:04.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:04.506+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:04.681 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:04.679+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:04.680163+0000) 2022-04-04T01:07:05.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:05.004+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:05.544 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:05.542+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:05.681 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:05.679+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:05.680349+0000) 2022-04-04T01:07:06.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:06.005+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:06.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:06.533+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:06.681 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:06.679+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:06.680529+0000) 2022-04-04T01:07:07.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:07.022+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:07.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:07.545+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 62 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:07.681 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:07.680+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:07.680667+0000) 2022-04-04T01:07:08.050 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:08.048+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:08.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:08.578+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:08.681 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:08.680+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:08.680785+0000) 2022-04-04T01:07:09.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:09.053+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:09.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:09.534+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:09.682 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:09.680+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:09.680880+0000) 2022-04-04T01:07:10.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:10.059+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:10.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:10.548+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:10.682 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:10.680+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:10.680976+0000) 2022-04-04T01:07:11.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:11.035+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:11.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:11.587+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:11.682 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:11.680+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:11.681101+0000) 2022-04-04T01:07:12.029 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:12.027+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:12.603 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:12.601+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:12.682 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:12.680+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:12.681250+0000) 2022-04-04T01:07:12.705 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:07:12.704+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:07:12.865 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:07:12.869 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:07:13.011 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:13.009+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:13.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:13.576+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:13.682 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:13.680+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:13.681391+0000) 2022-04-04T01:07:14.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:14.046+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:14.617 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:14.615+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:14.682 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:14.681+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:14.681540+0000) 2022-04-04T01:07:15.098 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:15.095+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:15.648 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:15.646+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:15.682 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:15.681+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:15.681683+0000) 2022-04-04T01:07:16.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:16.065+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:16.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:16.643+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:16.682 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:16.681+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:16.681862+0000) 2022-04-04T01:07:17.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:17.054+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:17.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:17.641+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:17.683 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:17.681+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:17.682036+0000) 2022-04-04T01:07:18.099 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:18.097+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:18.673 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:18.671+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:18.683 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:18.681+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:18.682169+0000) 2022-04-04T01:07:19.087 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:19.085+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:19.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:19.626+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:19.683 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:19.681+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:19.682286+0000) 2022-04-04T01:07:20.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:20.036+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:20.656 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:20.654+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:20.683 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:20.682+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:20.682434+0000) 2022-04-04T01:07:21.006 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:21.004+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:21.683 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:21.682+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:21.682586+0000) 2022-04-04T01:07:21.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:21.696+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:21.972 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:21.970+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:22.684 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:22.682+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:22.682760+0000) 2022-04-04T01:07:22.739 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:22.737+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:22.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:22.947+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:23.684 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:23.682+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:23.682951+0000) 2022-04-04T01:07:23.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:23.785+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:23.926 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:23.924+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:24.684 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:24.682+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:24.683091+0000) 2022-04-04T01:07:24.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:24.822+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:24.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:24.964+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:25.684 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:25.683+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:25.683260+0000) 2022-04-04T01:07:25.808 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:25.805+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:25.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:25.971+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:26.684 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:26.683+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:26.683455+0000) 2022-04-04T01:07:26.763 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:26.761+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:26.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:26.941+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:27.685 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:27.683+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:27.683654+0000) 2022-04-04T01:07:27.721 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:07:27.720+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:07:27.804 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:27.803+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:27.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:27.909+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:28.685 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:28.683+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:28.683870+0000) 2022-04-04T01:07:28.838 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:28.836+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:28.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:28.931+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:29.685 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:29.683+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:29.684019+0000) 2022-04-04T01:07:29.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:29.811+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:29.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:29.891+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:30.685 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:30.684+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:30.684204+0000) 2022-04-04T01:07:30.765 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:30.763+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:30.908 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:30.906+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:31.685 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:31.684+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:31.684375+0000) 2022-04-04T01:07:31.812 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:31.811+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:31.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:31.930+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:32.685 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:32.684+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:32.684536+0000) 2022-04-04T01:07:32.846 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:32.845+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:32.916 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:32.915+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:33.685 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:33.683+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:33.684714+0000) 2022-04-04T01:07:33.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:33.862+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:33.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:33.948+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:34.686 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:34.683+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:34.684875+0000) 2022-04-04T01:07:34.900 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:34.899+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:34.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:34.980+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:35.686 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:35.684+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:35.685059+0000) 2022-04-04T01:07:35.928 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:35.926+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:35.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:35.949+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:36.686 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:36.684+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:36.685203+0000) 2022-04-04T01:07:36.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:36.944+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:36.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:36.956+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:37.686 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:37.684+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:37.685378+0000) 2022-04-04T01:07:37.910 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:37.909+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:38.002 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:38.001+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:38.686 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:38.684+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:38.685572+0000) 2022-04-04T01:07:38.918 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:38.917+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:39.050 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:39.049+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:39.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:39.684+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:39.685748+0000) 2022-04-04T01:07:39.899 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:39.898+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:40.045 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:40.044+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:40.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:40.685+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:40.685916+0000) 2022-04-04T01:07:40.927 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:40.925+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:41.029 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:41.028+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:41.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:41.685+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:41.686074+0000) 2022-04-04T01:07:41.931 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:41.929+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:42.034 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:42.033+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:42.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:42.685+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:42.686238+0000) 2022-04-04T01:07:42.737 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:07:42.735+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:07:42.923 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:42.922+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:43.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:43.002+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:43.014 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:07:43.017 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:07:43.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:43.685+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:43.686408+0000) 2022-04-04T01:07:43.966 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:43.964+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:44.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:44.012+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:44.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:44.685+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:44.686616+0000) 2022-04-04T01:07:44.983 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:44.982+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:44.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:44.997+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:45.687 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:45.686+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:45.686820+0000) 2022-04-04T01:07:46.009 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:46.008+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:46.028 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:46.026+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:46.688 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:46.686+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:46.686948+0000) 2022-04-04T01:07:46.984 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:46.982+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:47.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:47.005+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:47.688 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:47.686+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:47.687065+0000) 2022-04-04T01:07:47.971 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:47.969+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:47.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:47.996+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:48.688 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:48.686+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:48.687214+0000) 2022-04-04T01:07:49.017 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:49.014+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:49.018 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:49.014+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:49.688 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:49.686+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:49.687412+0000) 2022-04-04T01:07:49.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:49.981+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:50.056 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:50.053+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:50.688 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:50.686+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:50.687625+0000) 2022-04-04T01:07:50.936 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:50.934+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:51.064 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:51.062+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:51.689 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:51.687+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:51.687799+0000) 2022-04-04T01:07:51.905 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:51.902+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:52.087 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:52.085+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:52.689 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:52.687+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:52.687937+0000) 2022-04-04T01:07:52.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:52.893+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:53.131 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:53.129+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:53.689 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:53.687+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:53.688114+0000) 2022-04-04T01:07:53.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:53.865+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:54.091 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:54.088+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:54.689 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:54.687+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:54.688291+0000) 2022-04-04T01:07:54.856 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:54.854+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:55.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:55.103+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:55.689 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:55.687+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:55.688425+0000) 2022-04-04T01:07:55.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:55.828+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:56.084 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:56.082+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:56.689 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:56.688+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:56.688591+0000) 2022-04-04T01:07:56.817 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:56.814+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:57.131 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:57.129+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:57.689 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:57.688+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:57.688717+0000) 2022-04-04T01:07:57.753 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:07:57.751+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:07:57.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:57.821+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:58.172 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:58.169+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:58.690 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:58.688+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:58.688864+0000) 2022-04-04T01:07:58.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:58.822+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:07:59.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:07:59.220+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:07:59.690 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:07:59.688+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:07:59.689052+0000) 2022-04-04T01:07:59.866 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:07:59.864+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:00.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:00.247+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:00.690 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:00.688+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:00.689162+0000) 2022-04-04T01:08:00.856 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:00.854+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:01.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:01.202+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:01.690 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:01.688+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:01.689269+0000) 2022-04-04T01:08:01.895 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:01.893+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:02.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:02.216+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:02.690 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:02.689+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:02.689441+0000) 2022-04-04T01:08:02.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:02.920+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:03.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:03.177+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:03.690 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:03.689+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:03.689597+0000) 2022-04-04T01:08:03.874 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:03.872+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:04.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:04.194+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:04.690 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:04.689+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:04.689796+0000) 2022-04-04T01:08:04.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:04.846+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:05.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:05.222+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:05.691 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:05.689+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:05.689930+0000) 2022-04-04T01:08:05.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:05.807+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:06.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:06.219+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:06.691 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:06.689+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:06.690059+0000) 2022-04-04T01:08:06.766 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:06.764+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:07.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:07.193+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:07.691 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:07.689+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:07.690199+0000) 2022-04-04T01:08:07.797 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:07.795+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:08.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:08.226+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:08.691 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:08.690+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:08.690365+0000) 2022-04-04T01:08:08.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:08.818+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:09.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:09.223+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:09.691 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:09.690+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:09.690524+0000) 2022-04-04T01:08:09.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:09.774+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:10.271 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:10.268+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:10.691 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:10.689+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:10.690726+0000) 2022-04-04T01:08:10.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:10.774+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:11.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:11.310+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:11.692 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:11.690+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:11.690959+0000) 2022-04-04T01:08:11.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:11.753+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:12.283 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:12.281+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:12.692 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:12.690+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:12.691122+0000) 2022-04-04T01:08:12.758 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:08:12.757+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:08:12.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:12.760+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:13.169 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:08:13.173 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:08:13.235 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:13.233+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:13.692 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:13.691+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:13.691280+0000) 2022-04-04T01:08:13.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:13.730+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:14.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:14.279+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:14.692 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:14.691+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:14.691446+0000) 2022-04-04T01:08:14.774 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:14.771+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:15.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:15.274+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:15.692 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:15.691+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:15.691573+0000) 2022-04-04T01:08:15.795 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:15.793+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:16.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:16.282+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:16.692 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:16.691+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:16.691710+0000) 2022-04-04T01:08:16.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:16.780+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:17.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:17.321+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:17.692 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:17.690+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:17.691845+0000) 2022-04-04T01:08:17.757 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:17.755+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:18.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:18.273+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:18.693 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:18.690+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:18.691999+0000) 2022-04-04T01:08:18.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:18.714+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:19.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:19.303+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:19.693 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:19.691+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:19.692169+0000) 2022-04-04T01:08:19.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:19.742+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:20.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:20.326+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:20.693 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:20.691+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:20.692315+0000) 2022-04-04T01:08:20.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:20.698+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:21.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:21.343+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:21.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:21.686+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:21.693 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:21.691+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:21.692520+0000) 2022-04-04T01:08:22.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:22.391+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:22.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:22.647+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:22.693 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:22.691+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:22.692711+0000) 2022-04-04T01:08:23.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:23.371+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:23.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:23.661+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:23.694 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:23.691+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:23.692917+0000) 2022-04-04T01:08:24.337 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:24.335+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:24.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:24.674+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:24.694 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:24.692+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:24.693125+0000) 2022-04-04T01:08:25.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:25.348+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:25.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:25.674+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:25.694 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:25.692+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:25.693331+0000) 2022-04-04T01:08:26.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:26.329+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:26.694 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:26.692+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:26.693546+0000) 2022-04-04T01:08:26.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:26.716+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:27.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:27.378+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:27.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:27.669+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:27.694 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:27.693+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:27.693805+0000) 2022-04-04T01:08:27.774 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:08:27.772+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:08:28.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:28.352+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:28.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:28.639+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:28.695 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:28.693+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:28.694014+0000) 2022-04-04T01:08:29.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:29.367+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:29.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:29.613+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:29.695 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:29.693+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:29.694219+0000) 2022-04-04T01:08:30.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:30.379+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:30.608 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:30.607+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:30.695 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:30.693+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:30.694433+0000) 2022-04-04T01:08:31.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:31.371+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:31.574 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:31.573+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:31.695 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:31.693+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:31.694641+0000) 2022-04-04T01:08:32.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:32.391+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:32.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:32.619+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:32.695 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:32.694+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:32.694822+0000) 2022-04-04T01:08:33.428 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:33.426+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:33.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:33.587+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:33.696 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:33.694+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:33.695002+0000) 2022-04-04T01:08:34.430 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:34.428+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:34.591 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:34.590+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:34.696 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:34.694+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:34.695210+0000) 2022-04-04T01:08:35.388 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:35.387+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:35.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:35.587+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:35.696 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:35.694+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:35.695416+0000) 2022-04-04T01:08:36.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:36.341+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:36.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:36.600+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:36.696 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:36.694+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:36.695626+0000) 2022-04-04T01:08:37.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:37.321+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:37.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:37.593+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:37.696 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:37.695+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:37.695862+0000) 2022-04-04T01:08:38.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:38.370+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:38.565 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:38.564+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:38.696 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:38.695+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:38.696015+0000) 2022-04-04T01:08:39.401 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:39.400+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:39.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:39.577+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:39.697 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:39.695+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:39.696196+0000) 2022-04-04T01:08:40.424 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:40.423+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:40.600 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:40.599+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:40.697 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:40.695+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:40.696410+0000) 2022-04-04T01:08:41.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:41.418+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:41.609 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:41.606+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:41.699 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:41.696+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:41.696621+0000) 2022-04-04T01:08:42.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:42.378+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:42.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:42.611+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:42.698 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:42.696+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:42.696877+0000) 2022-04-04T01:08:42.790 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:08:42.788+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:08:43.294 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:08:43.298 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:08:43.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:43.339+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:43.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:43.654+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:43.698 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:43.696+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:43.697084+0000) 2022-04-04T01:08:44.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:44.386+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:44.652 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:44.650+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:44.698 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:44.696+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:44.697292+0000) 2022-04-04T01:08:45.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:45.382+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:45.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:45.601+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:45.698 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:45.697+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:45.697472+0000) 2022-04-04T01:08:46.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:46.344+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:46.585 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:46.583+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:46.698 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:46.697+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:46.697660+0000) 2022-04-04T01:08:47.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:47.381+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:47.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:47.538+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:47.698 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:47.697+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:47.697874+0000) 2022-04-04T01:08:48.428 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:48.426+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:48.546 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:48.543+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:48.699 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:48.697+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:48.698046+0000) 2022-04-04T01:08:49.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:49.450+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:49.501 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:49.499+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:49.699 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:49.697+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:49.698245+0000) 2022-04-04T01:08:50.477 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:50.474+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:50.480 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:50.478+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:50.699 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:50.698+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:50.698442+0000) 2022-04-04T01:08:51.468 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:51.466+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:51.523 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:51.520+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:51.699 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:51.698+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:51.698612+0000) 2022-04-04T01:08:52.469 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:52.467+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:52.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:52.559+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:52.699 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:52.698+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:52.698833+0000) 2022-04-04T01:08:53.466 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:53.463+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:53.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:53.587+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:53.700 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:53.698+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:53.698996+0000) 2022-04-04T01:08:54.497 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:54.495+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:54.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:54.629+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:54.700 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:54.698+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:54.699217+0000) 2022-04-04T01:08:55.467 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:55.464+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:55.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:55.597+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:55.700 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:55.698+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:55.699382+0000) 2022-04-04T01:08:56.438 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:56.436+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:56.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:56.634+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:56.700 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:56.699+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:56.699517+0000) 2022-04-04T01:08:57.461 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:57.458+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:57.610 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:57.607+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:57.700 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:57.698+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:57.699724+0000) 2022-04-04T01:08:57.806 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:08:57.804+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:08:58.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:58.491+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:58.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:58.595+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:58.701 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:58.699+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:58.699915+0000) 2022-04-04T01:08:59.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:08:59.461+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:08:59.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:08:59.617+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:08:59.701 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:08:59.699+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:08:59.700085+0000) 2022-04-04T01:09:00.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:00.434+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:00.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:00.652+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:00.701 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:00.700+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:00.700284+0000) 2022-04-04T01:09:01.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:01.479+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:01.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:01.613+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:01.701 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:01.699+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:01.700498+0000) 2022-04-04T01:09:02.500 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:02.499+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:02.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:02.569+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:02.701 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:02.699+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:02.700676+0000) 2022-04-04T01:09:03.473 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:03.470+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:03.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:03.569+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:03.701 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:03.699+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:03.700864+0000) 2022-04-04T01:09:04.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:04.519+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:04.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:04.618+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:04.702 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:04.700+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:04.701076+0000) 2022-04-04T01:09:05.562 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:05.559+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:05.664 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:05.662+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:05.702 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:05.700+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:05.701286+0000) 2022-04-04T01:09:06.582 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:06.580+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:06.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:06.656+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:06.702 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:06.699+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:06.701428+0000) 2022-04-04T01:09:07.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:07.549+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:07.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:07.696+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:07.702 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:07.700+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:07.701607+0000) 2022-04-04T01:09:08.507 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:08.505+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:08.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:08.687+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:08.702 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:08.700+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:08.701734+0000) 2022-04-04T01:09:09.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:09.533+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:09.687 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:09.685+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:09.702 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:09.701+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:09.701924+0000) 2022-04-04T01:09:10.517 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:10.515+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:10.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:10.698+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:10.703 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:10.701+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:10.702129+0000) 2022-04-04T01:09:11.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:11.537+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:11.703 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:11.701+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:11.702299+0000) 2022-04-04T01:09:11.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:11.719+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:12.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:12.511+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:12.703 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:12.701+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:12.702483+0000) 2022-04-04T01:09:12.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:12.739+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:12.822 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:09:12.820+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:09:13.424 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:09:13.428 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:09:13.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:13.483+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:13.703 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:13.701+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:13.702732+0000) 2022-04-04T01:09:13.784 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:13.782+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:14.506 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:14.503+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:14.704 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:14.702+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:14.702892+0000) 2022-04-04T01:09:14.745 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:14.743+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:15.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:15.501+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:15.704 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:15.702+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:15.703102+0000) 2022-04-04T01:09:15.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:15.755+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:16.547 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:16.544+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:16.704 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:16.702+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:16.703284+0000) 2022-04-04T01:09:16.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:16.714+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:17.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:17.550+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:17.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:17.691+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:17.704 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:17.702+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:17.703479+0000) 2022-04-04T01:09:18.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:18.585+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:18.691 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:18.689+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:18.704 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:18.702+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:18.703700+0000) 2022-04-04T01:09:19.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:19.618+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:19.705 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:19.703+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:19.703869+0000) 2022-04-04T01:09:19.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:19.738+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:20.652 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:20.650+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:20.705 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:20.703+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:20.704019+0000) 2022-04-04T01:09:20.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:20.749+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:21.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:21.685+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:21.705 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:21.703+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:21.704217+0000) 2022-04-04T01:09:21.710 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:21.708+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:22.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:22.668+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:22.705 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:22.703+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:22.704435+0000) 2022-04-04T01:09:22.706 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:22.705+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:23.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:23.619+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:23.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:23.684+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:23.706 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:23.703+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:23.704590+0000) 2022-04-04T01:09:24.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:24.586+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:24.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:24.679+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:24.705 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:24.704+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:24.704801+0000) 2022-04-04T01:09:25.618 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:25.616+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:25.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:25.679+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:25.706 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:25.704+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:25.705010+0000) 2022-04-04T01:09:26.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:26.606+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:26.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:26.694+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:26.706 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:26.704+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:26.705160+0000) 2022-04-04T01:09:27.610 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:27.608+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:27.706 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:27.704+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:27.705373+0000) 2022-04-04T01:09:27.712 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:27.711+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:27.838 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:09:27.836+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:09:28.585 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:28.583+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:28.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:28.673+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:28.706 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:28.705+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:28.705563+0000) 2022-04-04T01:09:29.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:29.570+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:29.656 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:29.654+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:29.706 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:29.705+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:29.705762+0000) 2022-04-04T01:09:30.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:30.593+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:30.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:30.642+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:30.707 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:30.705+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:30.705971+0000) 2022-04-04T01:09:31.586 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:31.585+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:31.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:31.611+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:31.707 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:31.705+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:31.706144+0000) 2022-04-04T01:09:32.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:32.576+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:32.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:32.630+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:32.707 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:32.705+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:32.706301+0000) 2022-04-04T01:09:33.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:33.540+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:33.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:33.585+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:33.707 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:33.706+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:33.706486+0000) 2022-04-04T01:09:34.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:34.551+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:34.634 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:34.632+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:34.707 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:34.706+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:34.706675+0000) 2022-04-04T01:09:35.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:35.570+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:35.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:35.649+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:35.708 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:35.706+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:35.706859+0000) 2022-04-04T01:09:36.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:36.557+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:36.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:36.639+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:36.708 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:36.706+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:36.707036+0000) 2022-04-04T01:09:37.586 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:37.584+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:37.622 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:37.619+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:37.708 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:37.706+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:37.707172+0000) 2022-04-04T01:09:38.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:38.612+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:38.621 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:38.619+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:38.708 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:38.707+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:38.707328+0000) 2022-04-04T01:09:39.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:39.580+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:39.667 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:39.665+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:39.708 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:39.707+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:39.707532+0000) 2022-04-04T01:09:40.627 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:40.624+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:40.708 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:40.707+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:40.707759+0000) 2022-04-04T01:09:40.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:40.707+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:41.593 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:41.590+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:41.662 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:41.659+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:41.709 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:41.707+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:41.707984+0000) 2022-04-04T01:09:42.599 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:42.597+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:42.676 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:42.674+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:42.709 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:42.707+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:42.708153+0000) 2022-04-04T01:09:42.854 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:09:42.852+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:09:43.538 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:09:43.542 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:09:43.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:43.639+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:43.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:43.670+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:43.709 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:43.708+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:43.708390+0000) 2022-04-04T01:09:44.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:44.662+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:44.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:44.678+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:44.709 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:44.708+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:44.708526+0000) 2022-04-04T01:09:45.618 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:45.616+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:45.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:45.695+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:45.709 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:45.708+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:45.708739+0000) 2022-04-04T01:09:46.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:46.581+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:46.710 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:46.707+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:46.708958+0000) 2022-04-04T01:09:46.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:46.734+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:47.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:47.539+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:47.710 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:47.708+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:47.709167+0000) 2022-04-04T01:09:47.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:47.758+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:48.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:48.547+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:48.710 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:48.708+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:48.709321+0000) 2022-04-04T01:09:48.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:48.746+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:49.557 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:49.554+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:49.713 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:49.708+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:49.709496+0000) 2022-04-04T01:09:49.736 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:49.734+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:50.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:50.578+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:50.711 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:50.708+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:50.709687+0000) 2022-04-04T01:09:50.769 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:50.767+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:51.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:51.533+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:51.711 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:51.708+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:51.709879+0000) 2022-04-04T01:09:51.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:51.775+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:52.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:52.539+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:52.711 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:52.709+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:52.710076+0000) 2022-04-04T01:09:52.789 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:52.787+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:53.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:53.536+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:53.711 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:53.709+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:53.710271+0000) 2022-04-04T01:09:53.816 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:53.814+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:54.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:54.520+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:54.711 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:54.709+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:54.710461+0000) 2022-04-04T01:09:54.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:54.811+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:55.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:55.471+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:55.711 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:55.709+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:55.710651+0000) 2022-04-04T01:09:55.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:55.836+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:56.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:56.509+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:56.712 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:56.710+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:56.710876+0000) 2022-04-04T01:09:56.849 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:56.847+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:57.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:57.482+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:57.712 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:57.710+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:57.711086+0000) 2022-04-04T01:09:57.870 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:09:57.868+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:09:57.875 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:57.873+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:58.498 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:58.496+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:58.712 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:58.710+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:58.711290+0000) 2022-04-04T01:09:58.836 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:58.834+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:09:59.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:09:59.501+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:09:59.712 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:09:59.710+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:09:59.711487+0000) 2022-04-04T01:09:59.848 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:09:59.845+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:00.493 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:00.491+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:00.712 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:00.710+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:00.711701+0000) 2022-04-04T01:10:00.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:00.804+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:01.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:01.539+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:01.713 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:01.711+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:01.711892+0000) 2022-04-04T01:10:01.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:01.780+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:02.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:02.565+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:02.713 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:02.711+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:02.712094+0000) 2022-04-04T01:10:02.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:02.731+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:03.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:03.586+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:03.713 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:03.711+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:03.712216+0000) 2022-04-04T01:10:03.774 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:03.772+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:04.593 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:04.591+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:04.713 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:04.711+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:04.712493+0000) 2022-04-04T01:10:04.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:04.769+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:05.596 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:05.594+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:05.714 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:05.712+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:05.712718+0000) 2022-04-04T01:10:05.774 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:05.771+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:06.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:06.580+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:06.714 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:06.712+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:06.712939+0000) 2022-04-04T01:10:06.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:06.789+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:07.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:07.626+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:07.714 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:07.712+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:07.713135+0000) 2022-04-04T01:10:07.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:07.800+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 60 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:08.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:08.645+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:08.714 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:08.712+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:08.713333+0000) 2022-04-04T01:10:08.840 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:08.839+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:09.654 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:09.652+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:09.714 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:09.712+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:09.713550+0000) 2022-04-04T01:10:09.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:09.811+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:10.684 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:10.681+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:10.714 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:10.713+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:10.713731+0000) 2022-04-04T01:10:10.840 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:10.838+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:11.715 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:11.713+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:11.713933+0000) 2022-04-04T01:10:11.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:11.727+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:11.879 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:11.877+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:12.715 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:12.713+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:12.714139+0000) 2022-04-04T01:10:12.722 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:12.721+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:12.885 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:10:12.884+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:10:12.928 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:12.926+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:13.618 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:10:13.622 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:10:13.715 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:13.713+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:13.714403+0000) 2022-04-04T01:10:13.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:13.731+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:13.939 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:13.937+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:14.696 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:14.693+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:14.715 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:14.714+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:14.714639+0000) 2022-04-04T01:10:14.977 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:14.975+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:15.715 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:15.714+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:15.714830+0000) 2022-04-04T01:10:15.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:15.733+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:15.978 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:15.976+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:16.716 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:16.714+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:16.715025+0000) 2022-04-04T01:10:16.756 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:16.754+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:16.986 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:16.984+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:17.716 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:17.714+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:17.715221+0000) 2022-04-04T01:10:17.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:17.768+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:17.969 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:17.968+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:18.716 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:18.714+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:18.715437+0000) 2022-04-04T01:10:18.809 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:18.808+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:18.990 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:18.988+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:19.716 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:19.715+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:19.715611+0000) 2022-04-04T01:10:19.790 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:19.788+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:20.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:20.003+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:20.716 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:20.715+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:20.715838+0000) 2022-04-04T01:10:20.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:20.808+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:20.957 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:20.956+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:21.717 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:21.715+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:21.716009+0000) 2022-04-04T01:10:21.773 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:21.772+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:21.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:21.991+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:22.717 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:22.715+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:22.716176+0000) 2022-04-04T01:10:22.747 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:22.745+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:22.998 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:22.997+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:23.717 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:23.716+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:23.716408+0000) 2022-04-04T01:10:23.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:23.795+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:24.027 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:24.025+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:24.717 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:24.716+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:24.716637+0000) 2022-04-04T01:10:24.790 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:24.788+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:25.029 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:25.028+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:25.718 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:25.716+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:25.716851+0000) 2022-04-04T01:10:25.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:25.834+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:26.013 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:26.012+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:26.718 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:26.716+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:26.717051+0000) 2022-04-04T01:10:26.837 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:26.836+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:27.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:27.037+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:27.718 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:27.717+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:27.717224+0000) 2022-04-04T01:10:27.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:27.885+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:27.896 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:10:27.894+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:10:28.066 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:28.065+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:28.718 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:28.717+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:28.717395+0000) 2022-04-04T01:10:28.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:28.869+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:29.044 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:29.043+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:29.718 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:29.717+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:29.717592+0000) 2022-04-04T01:10:29.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:29.844+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:30.051 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:30.050+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:30.718 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:30.716+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:30.717756+0000) 2022-04-04T01:10:30.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:30.833+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:31.101 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:31.099+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:31.719 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:31.716+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:31.717981+0000) 2022-04-04T01:10:31.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:31.847+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:32.091 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:32.089+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:32.719 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:32.717+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:32.718187+0000) 2022-04-04T01:10:32.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:32.812+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:33.089 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:33.087+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:33.719 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:33.717+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:33.718340+0000) 2022-04-04T01:10:33.834 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:33.832+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:34.048 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:34.045+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:34.719 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:34.717+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:34.718522+0000) 2022-04-04T01:10:34.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:34.836+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:35.086 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:35.083+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:35.719 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:35.717+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:35.718725+0000) 2022-04-04T01:10:35.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:35.870+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:36.132 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:36.130+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:36.719 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:36.717+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:36.718861+0000) 2022-04-04T01:10:36.894 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:36.892+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:37.093 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:37.091+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:37.720 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:37.718+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:37.719061+0000) 2022-04-04T01:10:37.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:37.852+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:38.048 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:38.046+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:38.720 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:38.718+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:38.719258+0000) 2022-04-04T01:10:38.895 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:38.893+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:39.025 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:39.023+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:39.720 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:39.718+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:39.719445+0000) 2022-04-04T01:10:39.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:39.939+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:40.052 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:40.050+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:40.720 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:40.718+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:40.719661+0000) 2022-04-04T01:10:40.970 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:40.968+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:41.065 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:41.063+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:41.721 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:41.719+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:41.719884+0000) 2022-04-04T01:10:41.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:41.929+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:42.086 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:42.084+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:42.721 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:42.719+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:42.720094+0000) 2022-04-04T01:10:42.905 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:42.903+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:42.912 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:10:42.910+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:10:43.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:43.071+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:43.721 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:43.719+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:43.720318+0000) 2022-04-04T01:10:43.774 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:10:43.778 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:10:43.930 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:43.928+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:44.046 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:44.044+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:44.721 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:44.719+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:44.720547+0000) 2022-04-04T01:10:44.883 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:44.881+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:45.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:45.014+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:45.721 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:45.720+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:45.720823+0000) 2022-04-04T01:10:45.856 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:45.854+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:46.056 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:46.054+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:46.722 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:46.720+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:46.721048+0000) 2022-04-04T01:10:46.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:46.840+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:47.095 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:47.092+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:47.722 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:47.720+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:47.721239+0000) 2022-04-04T01:10:47.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:47.836+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:48.097 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:48.095+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:48.722 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:48.720+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:48.721446+0000) 2022-04-04T01:10:48.790 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:48.788+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:49.103 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:49.101+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:49.722 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:49.721+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:49.721616+0000) 2022-04-04T01:10:49.795 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:49.793+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:50.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:50.083+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:50.723 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:50.721+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:50.721830+0000) 2022-04-04T01:10:50.764 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:50.763+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:51.131 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:51.129+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:51.723 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:51.721+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:51.722049+0000) 2022-04-04T01:10:51.795 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:51.792+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:52.090 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:52.088+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:52.723 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:52.721+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:52.722311+0000) 2022-04-04T01:10:52.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:52.820+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:53.101 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:53.099+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:53.723 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:53.721+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:53.722535+0000) 2022-04-04T01:10:53.809 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:53.807+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:54.062 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:54.059+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:54.723 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:54.721+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:54.722736+0000) 2022-04-04T01:10:54.844 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:54.842+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:55.102 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:55.100+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:55.724 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:55.722+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:55.722950+0000) 2022-04-04T01:10:55.892 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:55.890+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:56.061 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:56.058+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:56.724 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:56.722+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:56.723071+0000) 2022-04-04T01:10:56.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:56.875+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:57.092 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:57.090+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:57.724 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:57.722+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:57.723280+0000) 2022-04-04T01:10:57.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:57.844+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:57.921 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:10:57.919+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:10:58.091 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:58.088+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:58.724 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:58.723+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:58.723490+0000) 2022-04-04T01:10:58.850 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:58.848+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:10:59.089 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:10:59.087+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:10:59.724 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:10:59.723+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:10:59.723703+0000) 2022-04-04T01:10:59.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:10:59.852+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:00.096 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:00.094+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:00.725 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:00.723+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:00.723861+0000) 2022-04-04T01:11:00.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:00.847+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:01.132 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:01.130+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:01.725 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:01.723+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:01.724090+0000) 2022-04-04T01:11:01.819 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:01.817+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:02.183 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:02.181+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:02.725 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:02.723+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:02.724275+0000) 2022-04-04T01:11:02.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:02.810+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:03.149 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:03.147+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:03.725 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:03.724+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:03.724470+0000) 2022-04-04T01:11:03.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:03.821+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:04.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:04.103+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:04.725 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:04.724+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:04.724696+0000) 2022-04-04T01:11:04.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:04.851+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:05.136 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:05.134+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:05.726 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:05.724+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:05.724854+0000) 2022-04-04T01:11:05.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:05.867+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:06.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:06.153+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:06.726 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:06.724+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:06.725020+0000) 2022-04-04T01:11:06.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:06.828+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:07.154 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:07.152+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:07.726 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:07.724+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:07.725188+0000) 2022-04-04T01:11:07.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:07.836+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:08.145 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:08.144+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:08.726 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:08.725+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:08.725354+0000) 2022-04-04T01:11:08.817 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:08.815+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:09.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:09.136+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:09.726 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:09.725+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:09.725532+0000) 2022-04-04T01:11:09.867 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:09.865+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:10.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:10.162+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:10.726 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:10.725+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:10.725716+0000) 2022-04-04T01:11:10.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:10.857+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:11.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:11.176+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:11.727 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:11.725+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:11.725873+0000) 2022-04-04T01:11:11.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:11.859+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:12.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:12.185+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:12.727 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:12.725+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:12.726051+0000) 2022-04-04T01:11:12.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:12.884+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:12.937 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:11:12.935+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:11:13.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:13.207+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:13.727 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:13.726+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:13.726268+0000) 2022-04-04T01:11:13.879 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:11:13.882 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:11:13.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:13.884+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:14.195 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:14.193+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:14.727 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:14.726+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:14.726446+0000) 2022-04-04T01:11:14.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:14.854+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:15.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:15.185+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:15.727 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:15.725+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:15.726643+0000) 2022-04-04T01:11:15.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:15.883+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:16.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:16.203+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:16.727 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:16.725+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:16.726822+0000) 2022-04-04T01:11:16.914 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:16.913+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:17.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:17.243+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:17.728 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:17.725+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:17.727009+0000) 2022-04-04T01:11:17.874 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:17.873+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:18.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:18.223+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:18.728 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:18.726+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:18.727208+0000) 2022-04-04T01:11:18.869 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:18.867+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:19.226 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:19.224+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:19.728 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:19.726+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:19.727411+0000) 2022-04-04T01:11:19.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:19.834+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:20.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:20.176+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:20.728 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:20.726+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:20.727579+0000) 2022-04-04T01:11:20.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:20.807+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:21.142 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:21.141+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:21.728 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:21.726+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:21.727807+0000) 2022-04-04T01:11:21.816 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:21.815+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:22.127 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:22.125+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:22.729 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:22.727+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:22.728019+0000) 2022-04-04T01:11:22.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:22.797+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:23.095 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:23.094+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:23.729 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:23.727+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:23.728215+0000) 2022-04-04T01:11:23.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:23.757+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:24.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:24.053+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:24.729 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:24.727+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:24.728394+0000) 2022-04-04T01:11:24.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:24.730+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:25.020 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:25.018+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:25.729 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:25.727+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:25.728562+0000) 2022-04-04T01:11:25.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:25.779+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:26.046 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:26.043+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:26.730 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:26.728+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:26.728808+0000) 2022-04-04T01:11:26.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:26.798+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:27.086 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:27.084+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:27.730 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:27.728+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:27.729032+0000) 2022-04-04T01:11:27.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:27.803+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:27.952 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:11:27.951+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:11:28.041 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:28.038+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:28.730 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:28.728+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:28.729270+0000) 2022-04-04T01:11:28.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:28.845+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:29.063 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:29.061+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:29.730 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:29.728+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:29.729498+0000) 2022-04-04T01:11:29.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:29.810+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:30.059 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:30.057+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:30.730 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:30.728+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:30.729732+0000) 2022-04-04T01:11:30.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:30.816+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:31.037 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:31.035+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:31.731 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:31.729+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:31.729973+0000) 2022-04-04T01:11:31.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:31.852+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:32.009 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:32.007+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:32.731 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:32.729+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:32.730181+0000) 2022-04-04T01:11:32.833 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:32.831+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:33.026 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:33.024+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:33.731 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:33.729+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:33.730398+0000) 2022-04-04T01:11:33.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:33.804+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:33.980 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:33.978+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:34.731 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:34.729+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:34.730623+0000) 2022-04-04T01:11:34.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:34.787+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:35.025 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:35.022+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:35.731 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:35.730+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:35.730820+0000) 2022-04-04T01:11:35.765 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:35.762+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:36.050 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:36.048+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:36.732 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:36.730+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:36.730976+0000) 2022-04-04T01:11:36.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:36.778+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:37.066 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:37.064+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:37.732 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:37.730+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:37.731166+0000) 2022-04-04T01:11:37.804 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:37.802+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:38.051 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:38.048+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:38.732 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:38.730+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:38.731369+0000) 2022-04-04T01:11:38.819 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:38.817+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:39.095 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:39.092+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:39.732 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:39.730+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:39.731589+0000) 2022-04-04T01:11:39.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:39.789+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:40.120 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:40.117+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:40.732 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:40.731+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:40.731853+0000) 2022-04-04T01:11:40.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:40.780+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:41.123 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:41.121+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:41.733 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:41.731+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:41.732013+0000) 2022-04-04T01:11:41.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:41.790+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:42.099 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:42.097+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:42.733 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:42.731+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:42.732233+0000) 2022-04-04T01:11:42.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:42.827+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:42.968 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:11:42.966+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:11:43.121 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:43.119+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:43.733 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:43.732+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:43.732448+0000) 2022-04-04T01:11:43.816 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:43.814+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:43.996 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:11:44.000 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:11:44.131 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:44.129+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:44.733 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:44.732+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:44.732662+0000) 2022-04-04T01:11:44.804 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:44.802+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:45.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:45.165+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:45.734 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:45.732+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:45.732856+0000) 2022-04-04T01:11:45.756 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:45.754+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:46.217 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:46.215+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:46.734 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:46.732+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:46.733059+0000) 2022-04-04T01:11:46.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:46.770+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:47.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:47.166+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:47.734 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:47.732+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:47.733250+0000) 2022-04-04T01:11:47.740 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:47.738+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:48.124 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:48.122+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:48.706 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:48.704+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:48.734 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:48.733+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:48.733448+0000) 2022-04-04T01:11:49.132 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:49.130+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:49.734 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:49.733+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:49.733651+0000) 2022-04-04T01:11:49.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:49.743+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:50.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:50.162+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:50.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:50.700+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:50.735 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:50.733+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:50.733854+0000) 2022-04-04T01:11:51.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:51.157+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:51.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:51.697+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:51.735 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:51.733+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:51.734091+0000) 2022-04-04T01:11:52.124 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:52.122+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:52.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:52.725+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:52.735 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:52.734+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:52.734331+0000) 2022-04-04T01:11:53.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:53.153+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:53.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:53.693+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:53.735 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:53.733+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:53.734485+0000) 2022-04-04T01:11:54.126 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:54.124+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:54.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:54.698+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:54.735 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:54.734+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:54.734703+0000) 2022-04-04T01:11:55.136 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:55.134+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:55.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:55.665+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:55.735 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:55.734+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:55.734907+0000) 2022-04-04T01:11:56.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:56.177+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:56.708 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:56.706+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:56.736 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:56.734+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:56.735132+0000) 2022-04-04T01:11:57.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:57.187+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:57.736 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:57.735+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:57.735339+0000) 2022-04-04T01:11:57.740 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:57.738+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:57.984 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:11:57.982+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:11:58.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:58.217+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:58.736 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:58.735+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:58.735549+0000) 2022-04-04T01:11:58.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:58.788+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:11:59.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:11:59.184+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:11:59.736 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:11:59.734+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:11:59.735696+0000) 2022-04-04T01:11:59.764 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:11:59.762+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:00.195 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:00.192+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:00.737 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:00.734+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:00.735824+0000) 2022-04-04T01:12:00.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:00.748+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:01.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:01.178+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:01.736 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:01.734+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:01.735961+0000) 2022-04-04T01:12:01.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:01.783+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:02.221 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:02.219+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:02.737 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:02.735+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:02.736133+0000) 2022-04-04T01:12:02.833 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:02.831+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:03.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:03.227+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:03.737 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:03.735+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:03.736318+0000) 2022-04-04T01:12:03.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:03.799+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:04.212 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:04.209+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:04.737 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:04.735+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:04.736478+0000) 2022-04-04T01:12:04.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:04.778+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:05.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:05.252+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:05.737 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:05.734+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:05.736679+0000) 2022-04-04T01:12:05.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:05.740+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:06.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:06.247+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:06.738 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:06.735+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:06.736883+0000) 2022-04-04T01:12:06.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:06.784+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:07.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:07.253+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:07.738 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:07.736+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:07.737087+0000) 2022-04-04T01:12:07.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:07.824+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:08.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:08.288+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:08.738 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:08.736+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:08.737289+0000) 2022-04-04T01:12:08.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:08.824+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:09.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:09.277+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:09.738 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:09.736+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:09.737482+0000) 2022-04-04T01:12:09.841 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:09.839+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:10.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:10.313+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:10.738 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:10.736+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:10.737626+0000) 2022-04-04T01:12:10.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:10.872+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:11.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:11.319+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:11.738 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:11.737+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:11.737823+0000) 2022-04-04T01:12:11.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:11.823+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:12.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:12.347+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:12.739 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:12.737+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:12.738023+0000) 2022-04-04T01:12:12.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:12.795+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:13.000 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:12:12.998+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:12:13.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:13.347+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:13.739 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:13.737+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:13.738240+0000) 2022-04-04T01:12:13.809 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:13.808+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:14.127 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:12:14.131 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:12:14.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:14.342+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:14.739 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:14.737+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:14.738439+0000) 2022-04-04T01:12:14.768 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:14.767+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:15.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:15.369+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:15.739 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:15.737+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:15.738588+0000) 2022-04-04T01:12:15.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:15.799+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:16.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:16.375+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:16.739 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:16.738+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:16.738762+0000) 2022-04-04T01:12:16.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:16.800+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:17.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:17.398+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:17.740 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:17.738+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:17.738968+0000) 2022-04-04T01:12:17.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:17.844+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:18.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:18.441+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:18.740 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:18.738+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:18.739159+0000) 2022-04-04T01:12:18.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:18.835+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:19.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:19.447+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:19.740 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:19.738+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:19.739342+0000) 2022-04-04T01:12:19.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:19.869+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:20.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:20.414+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:20.740 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:20.738+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:20.739529+0000) 2022-04-04T01:12:20.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:20.820+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:21.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:21.379+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:21.740 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:21.739+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:21.739692+0000) 2022-04-04T01:12:21.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:21.797+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:22.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:22.342+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:22.740 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:22.739+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:22.739844+0000) 2022-04-04T01:12:22.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:22.820+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:23.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:23.308+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:23.741 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:23.739+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:23.740045+0000) 2022-04-04T01:12:23.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:23.822+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:24.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:24.330+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:24.741 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:24.739+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:24.740270+0000) 2022-04-04T01:12:24.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:24.853+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:25.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:25.311+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:25.741 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:25.739+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:25.740489+0000) 2022-04-04T01:12:25.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:25.853+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:26.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:26.274+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:26.741 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:26.740+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:26.740694+0000) 2022-04-04T01:12:26.862 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:26.860+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:27.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:27.239+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:27.742 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:27.740+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:27.740857+0000) 2022-04-04T01:12:27.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:27.849+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:28.015 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:12:28.014+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:12:28.271 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:28.269+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:28.742 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:28.740+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:28.741047+0000) 2022-04-04T01:12:28.819 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:28.816+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:29.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:29.255+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:29.742 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:29.740+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:29.741245+0000) 2022-04-04T01:12:29.862 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:29.860+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:30.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:30.232+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:30.742 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:30.740+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:30.741393+0000) 2022-04-04T01:12:30.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:30.858+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:31.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:31.238+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:31.742 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:31.741+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:31.741550+0000) 2022-04-04T01:12:31.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:31.880+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:32.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:32.206+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:32.742 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:32.741+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:32.741696+0000) 2022-04-04T01:12:32.837 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:32.835+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:33.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:33.182+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:33.742 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:33.741+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:33.741878+0000) 2022-04-04T01:12:33.859 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:33.857+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:34.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:34.208+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:34.743 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:34.741+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:34.742079+0000) 2022-04-04T01:12:34.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:34.884+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:35.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:35.244+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:35.743 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:35.742+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:35.742300+0000) 2022-04-04T01:12:35.910 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:35.908+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:36.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:36.277+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:36.743 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:36.742+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:36.742559+0000) 2022-04-04T01:12:36.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:36.949+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:37.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:37.253+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:37.743 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:37.742+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:37.742840+0000) 2022-04-04T01:12:37.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:37.971+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:38.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:38.279+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:38.744 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:38.742+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:38.743011+0000) 2022-04-04T01:12:38.961 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:38.959+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:39.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:39.239+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:39.744 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:39.743+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:39.743196+0000) 2022-04-04T01:12:40.009 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:40.006+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:40.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:40.241+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:40.744 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:40.743+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:40.743399+0000) 2022-04-04T01:12:40.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:40.979+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:41.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:41.216+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:41.744 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:41.743+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:41.743623+0000) 2022-04-04T01:12:42.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:42.023+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:42.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:42.236+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:42.745 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:42.743+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:42.743872+0000) 2022-04-04T01:12:43.031 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:12:43.029+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:12:43.057 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:43.055+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:43.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:43.195+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:43.745 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:43.743+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:43.744103+0000) 2022-04-04T01:12:44.052 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:44.049+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:44.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:44.224+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:44.417 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:12:44.421 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:12:44.745 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:44.742+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:44.744298+0000) 2022-04-04T01:12:45.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:45.072+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:45.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:45.225+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:45.745 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:45.743+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:45.744503+0000) 2022-04-04T01:12:46.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:46.060+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:46.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:46.199+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:46.745 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:46.743+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:46.744705+0000) 2022-04-04T01:12:47.027 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:47.025+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:47.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:47.163+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:47.746 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:47.743+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:47.744950+0000) 2022-04-04T01:12:47.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:47.981+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:48.142 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:48.140+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:48.746 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:48.744+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:48.745163+0000) 2022-04-04T01:12:48.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:48.949+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:49.176 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:49.174+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:49.746 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:49.743+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:49.745416+0000) 2022-04-04T01:12:49.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:49.949+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:50.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:50.155+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:50.746 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:50.744+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:50.745593+0000) 2022-04-04T01:12:50.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:50.931+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:51.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:51.159+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:51.747 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:51.744+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:51.745853+0000) 2022-04-04T01:12:51.906 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:51.904+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:52.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:52.171+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:52.747 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:52.745+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:52.745984+0000) 2022-04-04T01:12:52.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:52.922+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:53.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:53.155+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:53.747 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:53.745+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:53.746051+0000) 2022-04-04T01:12:53.895 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:53.894+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:54.191 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:54.189+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:54.747 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:54.745+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:54.746145+0000) 2022-04-04T01:12:54.851 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:54.849+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:55.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:55.190+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:55.747 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:55.745+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:55.746300+0000) 2022-04-04T01:12:55.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:55.896+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:56.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:56.217+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:56.747 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:56.745+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:56.746521+0000) 2022-04-04T01:12:56.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:56.938+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:57.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:57.259+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:57.748 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:57.745+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:57.746748+0000) 2022-04-04T01:12:57.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:57.955+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:58.036 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:12:58.034+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:12:58.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:58.267+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:58.748 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:58.746+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:58.746988+0000) 2022-04-04T01:12:58.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:58.980+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:12:59.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:12:59.311+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:12:59.748 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:12:59.746+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:12:59.747216+0000) 2022-04-04T01:12:59.971 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:12:59.969+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:00.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:00.327+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:00.748 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:00.746+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:00.747386+0000) 2022-04-04T01:13:00.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:00.949+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:01.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:01.316+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:01.748 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:01.746+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:01.747579+0000) 2022-04-04T01:13:01.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:01.986+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:02.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:02.328+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:02.748 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:02.747+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:02.747831+0000) 2022-04-04T01:13:02.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:02.952+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:03.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:03.378+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:03.749 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:03.747+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:03.748029+0000) 2022-04-04T01:13:04.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:04.002+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:04.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:04.375+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:04.749 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:04.747+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:04.748256+0000) 2022-04-04T01:13:05.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:05.026+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:05.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:05.413+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:05.749 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:05.747+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:05.748437+0000) 2022-04-04T01:13:06.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:06.046+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:06.401 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:06.400+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:06.749 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:06.748+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:06.748640+0000) 2022-04-04T01:13:07.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:07.083+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:07.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:07.388+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:07.750 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:07.748+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:07.748843+0000) 2022-04-04T01:13:08.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:08.045+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:08.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:08.350+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:08.750 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:08.748+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:08.748973+0000) 2022-04-04T01:13:09.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:09.027+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:09.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:09.333+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:09.750 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:09.748+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:09.749122+0000) 2022-04-04T01:13:10.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:10.025+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:10.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:10.372+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:10.750 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:10.748+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:10.749281+0000) 2022-04-04T01:13:11.001 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:11.000+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:11.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:11.367+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:11.750 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:11.749+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:11.749480+0000) 2022-04-04T01:13:12.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:12.003+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:12.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:12.321+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:12.751 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:12.749+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:12.749722+0000) 2022-04-04T01:13:13.027 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:13.025+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:13.044 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:13:13.042+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:13:13.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:13.341+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 64 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:13.751 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:13.749+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:13.749900+0000) 2022-04-04T01:13:13.986 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:13.985+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:14.356 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:14.354+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:14.488 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:13:14.492 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:13:14.750 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:14.749+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:14.750113+0000) 2022-04-04T01:13:14.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:14.952+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:15.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:15.349+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:15.751 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:15.749+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:15.750268+0000) 2022-04-04T01:13:15.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:15.938+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:16.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:16.305+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:16.751 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:16.750+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:16.750447+0000) 2022-04-04T01:13:16.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:16.926+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:17.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:17.259+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:17.751 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:17.750+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:17.750659+0000) 2022-04-04T01:13:17.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:17.876+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:18.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:18.261+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:18.752 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:18.750+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:18.750868+0000) 2022-04-04T01:13:18.908 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:18.906+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:19.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:19.279+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:19.752 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:19.750+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:19.751097+0000) 2022-04-04T01:13:19.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:19.859+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:20.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:20.266+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:20.752 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:20.751+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:20.751340+0000) 2022-04-04T01:13:20.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:20.825+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:21.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:21.249+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:21.752 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:21.751+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:21.751560+0000) 2022-04-04T01:13:21.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:21.841+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:22.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:22.226+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:22.753 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:22.751+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:22.751721+0000) 2022-04-04T01:13:22.819 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:22.816+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:23.218 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:23.216+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:23.753 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:23.751+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:23.751956+0000) 2022-04-04T01:13:23.862 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:23.860+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:24.252 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:24.250+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:24.753 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:24.752+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:24.752154+0000) 2022-04-04T01:13:24.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:24.877+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:25.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:25.222+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:25.753 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:25.752+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:25.752353+0000) 2022-04-04T01:13:25.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:25.837+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:26.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:26.257+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:26.753 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:26.752+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:26.752574+0000) 2022-04-04T01:13:26.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:26.843+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:27.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:27.234+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:27.753 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:27.751+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:27.752763+0000) 2022-04-04T01:13:27.826 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:27.823+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:28.060 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:13:28.057+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:13:28.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:28.216+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:28.754 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:28.751+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:28.752968+0000) 2022-04-04T01:13:28.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:28.804+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:29.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:29.259+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:29.754 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:29.752+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:29.753175+0000) 2022-04-04T01:13:29.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:29.810+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:30.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:30.275+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:30.754 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:30.752+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:30.753368+0000) 2022-04-04T01:13:30.767 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:30.764+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:31.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:31.227+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:31.754 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:31.751+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:31.753583+0000) 2022-04-04T01:13:31.766 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:31.764+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:32.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:32.262+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:32.718 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:32.716+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:32.754 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:32.752+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:32.753838+0000) 2022-04-04T01:13:33.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:33.278+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:33.681 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:33.678+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:33.755 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:33.752+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:33.754076+0000) 2022-04-04T01:13:34.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:34.306+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:34.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:34.691+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:34.755 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:34.753+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:34.754274+0000) 2022-04-04T01:13:35.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:35.297+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:35.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:35.733+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:35.755 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:35.753+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:35.754513+0000) 2022-04-04T01:13:36.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:36.312+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:36.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:36.741+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:36.755 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:36.753+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:36.754737+0000) 2022-04-04T01:13:37.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:37.275+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:37.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:37.713+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:37.756 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:37.754+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:37.754975+0000) 2022-04-04T01:13:38.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:38.265+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:38.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:38.739+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:38.756 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:38.754+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:38.755199+0000) 2022-04-04T01:13:39.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:39.261+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:39.719 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:39.717+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:39.756 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:39.754+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:39.755419+0000) 2022-04-04T01:13:40.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:40.255+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:40.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:40.739+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:40.756 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:40.754+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:40.755642+0000) 2022-04-04T01:13:41.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:41.278+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:41.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:41.724+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:41.756 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:41.755+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:41.755859+0000) 2022-04-04T01:13:42.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:42.248+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:42.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:42.733+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:42.757 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:42.755+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:42.756066+0000) 2022-04-04T01:13:43.075 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:13:43.074+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:13:43.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:43.207+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:43.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:43.730+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:43.757 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:43.755+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:43.756226+0000) 2022-04-04T01:13:44.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:44.224+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:44.640 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:13:44.644 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:13:44.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:44.737+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:44.757 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:44.755+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:44.756473+0000) 2022-04-04T01:13:45.271 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:45.269+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:45.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:45.733+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:45.757 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:45.755+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:45.756701+0000) 2022-04-04T01:13:46.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:46.230+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:46.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:46.712+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:46.758 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:46.756+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:46.756874+0000) 2022-04-04T01:13:47.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:47.249+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:47.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:47.663+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:47.758 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:47.756+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:47.757077+0000) 2022-04-04T01:13:48.235 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:48.233+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:48.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:48.668+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:48.758 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:48.756+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:48.757332+0000) 2022-04-04T01:13:49.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:49.199+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:49.716 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:49.714+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:49.758 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:49.756+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:49.757574+0000) 2022-04-04T01:13:50.213 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:50.210+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:50.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:50.726+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:50.758 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:50.757+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:50.757757+0000) 2022-04-04T01:13:51.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:51.176+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:51.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:51.751+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:51.758 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:51.757+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:51.757957+0000) 2022-04-04T01:13:52.138 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:52.137+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:52.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:52.725+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:52.759 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:52.757+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:52.758158+0000) 2022-04-04T01:13:53.157 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:53.155+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:53.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:53.737+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:53.759 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:53.757+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:53.758399+0000) 2022-04-04T01:13:54.119 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:54.117+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:54.720 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:54.718+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:54.759 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:54.758+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:54.758638+0000) 2022-04-04T01:13:55.146 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:55.144+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:55.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:55.721+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:55.759 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:55.758+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:55.758860+0000) 2022-04-04T01:13:56.135 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:56.134+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:56.685 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:56.684+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:56.760 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:56.758+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:56.759082+0000) 2022-04-04T01:13:57.086 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:57.085+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:57.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:57.726+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:57.760 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:57.758+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:57.759265+0000) 2022-04-04T01:13:58.091 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:13:58.089+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:13:58.107 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:58.106+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:58.760 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:58.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:58.759445+0000) 2022-04-04T01:13:58.771 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:58.770+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:59.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:13:59.149+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:13:59.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:13:59.729+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:13:59.760 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:13:59.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:13:59.759645+0000) 2022-04-04T01:14:00.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:00.198+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:00.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:00.753+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:00.760 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:00.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:00.759882+0000) 2022-04-04T01:14:01.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:01.184+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:01.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:01.725+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:01.760 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:01.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:01.760049+0000) 2022-04-04T01:14:02.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:02.187+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:02.689 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:02.688+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:02.761 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:02.759+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:02.760189+0000) 2022-04-04T01:14:03.154 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:03.153+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:03.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:03.654+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:03.761 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:03.760+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:03.760329+0000) 2022-04-04T01:14:04.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:04.177+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:04.697 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:04.696+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:04.761 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:04.760+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:04.760493+0000) 2022-04-04T01:14:05.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:05.196+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:05.674 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:05.673+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:05.761 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:05.760+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:05.760670+0000) 2022-04-04T01:14:06.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:06.179+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:06.673 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:06.671+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:06.761 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:06.760+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:06.760862+0000) 2022-04-04T01:14:07.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:07.184+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:07.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:07.714+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:07.761 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:07.760+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:07.760984+0000) 2022-04-04T01:14:08.213 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:08.212+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:08.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:08.694+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:08.762 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:08.761+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:08.761173+0000) 2022-04-04T01:14:09.172 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:09.171+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:09.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:09.656+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:09.762 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:09.761+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:09.761399+0000) 2022-04-04T01:14:10.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:10.201+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:10.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:10.668+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:10.762 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:10.761+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:10.761612+0000) 2022-04-04T01:14:11.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:11.201+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:11.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:11.680+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:11.763 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:11.761+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:11.761854+0000) 2022-04-04T01:14:12.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:12.198+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:12.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:12.639+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:12.763 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:12.761+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:12.762094+0000) 2022-04-04T01:14:13.107 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:14:13.104+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:14:13.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:13.235+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:13.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:13.665+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:13.763 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:13.761+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:13.762317+0000) 2022-04-04T01:14:14.241 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:14.239+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:14.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:14.632+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:14.763 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:14.761+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:14.762484+0000) 2022-04-04T01:14:14.784 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:14:14.788 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:14:15.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:15.246+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:15.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:15.621+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:15.763 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:15.761+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:15.762678+0000) 2022-04-04T01:14:16.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:16.289+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:16.659 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:16.657+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:16.764 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:16.761+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:16.762919+0000) 2022-04-04T01:14:17.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:17.259+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:17.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:17.639+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:17.764 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:17.762+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:17.763131+0000) 2022-04-04T01:14:18.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:18.237+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:18.617 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:18.615+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:18.764 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:18.762+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:18.763312+0000) 2022-04-04T01:14:19.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:19.254+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:19.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:19.635+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:19.764 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:19.762+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:19.763431+0000) 2022-04-04T01:14:20.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:20.252+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:20.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:20.680+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:20.764 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:20.762+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:20.763622+0000) 2022-04-04T01:14:21.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:21.257+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:21.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:21.730+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:21.764 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:21.762+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:21.763812+0000) 2022-04-04T01:14:22.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:22.219+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:22.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:22.729+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:22.764 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:22.763+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:22.763971+0000) 2022-04-04T01:14:23.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:23.261+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:23.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:23.685+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:23.765 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:23.763+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:23.764138+0000) 2022-04-04T01:14:24.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:24.304+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:24.654 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:24.652+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:24.765 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:24.763+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:24.764361+0000) 2022-04-04T01:14:25.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:25.309+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:25.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:25.645+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:25.765 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:25.763+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:25.764509+0000) 2022-04-04T01:14:26.337 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:26.335+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:26.661 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:26.659+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:26.765 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:26.763+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:26.764722+0000) 2022-04-04T01:14:27.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:27.318+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:27.697 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:27.694+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:27.766 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:27.764+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:27.764965+0000) 2022-04-04T01:14:28.123 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:14:28.121+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:14:28.364 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:28.362+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:28.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:28.692+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:28.766 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:28.764+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:28.765186+0000) 2022-04-04T01:14:29.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:29.346+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:29.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:29.665+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:29.766 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:29.764+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:29.765376+0000) 2022-04-04T01:14:30.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:30.319+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:30.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:30.659+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:30.766 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:30.764+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:30.765598+0000) 2022-04-04T01:14:31.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:31.365+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:31.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:31.638+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:31.766 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:31.765+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:31.765834+0000) 2022-04-04T01:14:32.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:32.408+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:32.673 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:32.671+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:32.767 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:32.765+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:32.765995+0000) 2022-04-04T01:14:33.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:33.457+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:33.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:33.639+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:33.767 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:33.765+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:33.766136+0000) 2022-04-04T01:14:34.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:34.495+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:34.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:34.680+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:34.767 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:34.765+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:34.766304+0000) 2022-04-04T01:14:35.500 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:35.497+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:35.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:35.642+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:35.767 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:35.765+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:35.766495+0000) 2022-04-04T01:14:36.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:36.497+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:36.603 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:36.601+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:36.767 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:36.766+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:36.766657+0000) 2022-04-04T01:14:37.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:37.547+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:37.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:37.611+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:37.767 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:37.766+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:37.766832+0000) 2022-04-04T01:14:38.546 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:38.544+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:38.652 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:38.650+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:38.768 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:38.766+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:38.767013+0000) 2022-04-04T01:14:39.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:39.592+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:39.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:39.654+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:39.768 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:39.766+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:39.767245+0000) 2022-04-04T01:14:40.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:40.596+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:40.626 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:40.624+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:40.768 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:40.767+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:40.767441+0000) 2022-04-04T01:14:41.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:41.577+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:41.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:41.667+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:41.769 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:41.766+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:41.767644+0000) 2022-04-04T01:14:42.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:42.622+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:42.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:42.691+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:42.769 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:42.767+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:42.767878+0000) 2022-04-04T01:14:43.139 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:14:43.137+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:14:43.621 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:43.619+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:43.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:43.738+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:43.769 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:43.767+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:43.768106+0000) 2022-04-04T01:14:44.623 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:44.621+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:44.707 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:44.704+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:44.769 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:44.767+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:44.768328+0000) 2022-04-04T01:14:44.934 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:14:44.938 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:14:45.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:45.577+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:45.692 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:45.690+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:45.769 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:45.768+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:45.768571+0000) 2022-04-04T01:14:46.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:46.588+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:46.701 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:46.699+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:46.769 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:46.768+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:46.768749+0000) 2022-04-04T01:14:47.569 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:47.568+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:47.718 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:47.716+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:47.770 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:47.768+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:47.768954+0000) 2022-04-04T01:14:48.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:48.542+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:48.764 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:48.762+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:48.770 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:48.768+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:48.769153+0000) 2022-04-04T01:14:49.510 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:49.508+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:49.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:49.737+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:49.770 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:49.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:49.769301+0000) 2022-04-04T01:14:50.500 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:50.498+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:50.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:50.727+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:50.770 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:50.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:50.769475+0000) 2022-04-04T01:14:51.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:51.454+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:51.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:51.712+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:51.770 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:51.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:51.769681+0000) 2022-04-04T01:14:52.470 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:52.468+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:52.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:52.680+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:52.771 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:52.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:52.769932+0000) 2022-04-04T01:14:53.519 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:53.517+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:53.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:53.663+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:53.771 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:53.770+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:53.770130+0000) 2022-04-04T01:14:54.502 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:54.500+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:54.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:54.661+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:54.771 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:54.770+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:54.770345+0000) 2022-04-04T01:14:55.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:55.507+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:55.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:55.612+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:55.771 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:55.770+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:55.770549+0000) 2022-04-04T01:14:56.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:56.534+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:56.615 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:56.614+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:56.771 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:56.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:56.770750+0000) 2022-04-04T01:14:57.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:57.555+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:57.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:57.656+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:57.772 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:57.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:57.770933+0000) 2022-04-04T01:14:58.155 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:14:58.152+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:14:58.583 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:58.582+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:58.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:58.628+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:58.771 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:58.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:58.771031+0000) 2022-04-04T01:14:59.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:14:59.595+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:14:59.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:14:59.663+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:14:59.772 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:14:59.770+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:14:59.771183+0000) 2022-04-04T01:15:00.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:00.577+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:00.626 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:00.624+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:00.772 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:00.770+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:00.771357+0000) 2022-04-04T01:15:01.600 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:01.598+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:01.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:01.609+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:01.772 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:01.769+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:01.771588+0000) 2022-04-04T01:15:02.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:02.578+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:02.582 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:02.581+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:02.772 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:02.770+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:02.771832+0000) 2022-04-04T01:15:03.566 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:03.565+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:03.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:03.589+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:03.773 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:03.771+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:03.772038+0000) 2022-04-04T01:15:04.590 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:04.588+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:04.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:04.630+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:04.773 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:04.771+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:04.772247+0000) 2022-04-04T01:15:05.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:05.583+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:05.610 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:05.609+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:05.773 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:05.771+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:05.772430+0000) 2022-04-04T01:15:06.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:06.539+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:06.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:06.645+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:06.773 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:06.771+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:06.772630+0000) 2022-04-04T01:15:07.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:07.568+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:07.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:07.674+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:07.773 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:07.772+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:07.772858+0000) 2022-04-04T01:15:08.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:08.575+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:08.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:08.723+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:08.774 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:08.772+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:08.773062+0000) 2022-04-04T01:15:09.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:09.617+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:09.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:09.738+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:09.774 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:09.772+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:09.773237+0000) 2022-04-04T01:15:10.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:10.658+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:10.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:10.756+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:10.774 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:10.772+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:10.773415+0000) 2022-04-04T01:15:11.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:11.644+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:11.774 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:11.772+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:11.773626+0000) 2022-04-04T01:15:11.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:11.779+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:12.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:12.595+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:12.775 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:12.773+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:12.773846+0000) 2022-04-04T01:15:12.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:12.829+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:13.171 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:15:13.169+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:15:13.552 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:13.549+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:13.775 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:13.773+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:13.774045+0000) 2022-04-04T01:15:13.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:13.870+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:14.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:14.595+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:14.775 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:14.773+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:14.774249+0000) 2022-04-04T01:15:14.892 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:14.889+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:15.004 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:15:15.008 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:15:15.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:15.612+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:15.775 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:15.773+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:15.774424+0000) 2022-04-04T01:15:15.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:15.935+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:16.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:16.611+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:16.775 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:16.773+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:16.774580+0000) 2022-04-04T01:15:16.926 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:16.924+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:17.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:17.574+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:17.775 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:17.774+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:17.774765+0000) 2022-04-04T01:15:17.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:17.884+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:18.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:18.553+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:18.776 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:18.774+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:18.774964+0000) 2022-04-04T01:15:18.883 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:18.880+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:19.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:19.549+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:19.776 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:19.774+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:19.775149+0000) 2022-04-04T01:15:19.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:19.873+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:20.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:20.565+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:20.776 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:20.774+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:20.775308+0000) 2022-04-04T01:15:20.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:20.846+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:21.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:21.591+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:21.776 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:21.774+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:21.775490+0000) 2022-04-04T01:15:21.859 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:21.857+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:22.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:22.565+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:22.776 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:22.775+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:22.775662+0000) 2022-04-04T01:15:22.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:22.809+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:23.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:23.543+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:23.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:23.772+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:23.776 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:23.775+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:23.775821+0000) 2022-04-04T01:15:24.570 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:24.568+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:24.777 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:24.775+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:24.776055+0000) 2022-04-04T01:15:24.790 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:24.787+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:25.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:25.546+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:25.777 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:25.775+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:25.776246+0000) 2022-04-04T01:15:25.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:25.782+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:26.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:26.578+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:26.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:26.757+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:26.777 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:26.776+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:26.776457+0000) 2022-04-04T01:15:27.609 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:27.607+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:27.711 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:27.709+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:27.777 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:27.776+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:27.776669+0000) 2022-04-04T01:15:28.187 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:15:28.185+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:15:28.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:28.652+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:28.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:28.723+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:28.778 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:28.776+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:28.776880+0000) 2022-04-04T01:15:29.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:29.700+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:29.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:29.746+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:29.778 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:29.776+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:29.777106+0000) 2022-04-04T01:15:30.727 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:30.725+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:30.734 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:30.732+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:30.778 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:30.777+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:30.777351+0000) 2022-04-04T01:15:31.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:31.685+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:31.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:31.771+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:31.778 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:31.777+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:31.777592+0000) 2022-04-04T01:15:32.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:32.700+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:32.752 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:32.750+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:32.779 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:32.777+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:32.777821+0000) 2022-04-04T01:15:33.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:33.739+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:33.825 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:33.777+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:33.778020+0000) 2022-04-04T01:15:33.826 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:33.755+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:34.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:34.707+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:34.734 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:34.732+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:34.779 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:34.778+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:34.778248+0000) 2022-04-04T01:15:35.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:35.718+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:35.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:35.726+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:35.779 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:35.778+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:35.778462+0000) 2022-04-04T01:15:36.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:36.759+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:36.768 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:36.766+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:36.780 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:36.778+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:36.778683+0000) 2022-04-04T01:15:37.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:37.726+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:37.780 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:37.778+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:37.778906+0000) 2022-04-04T01:15:37.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:37.791+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:38.694 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:38.691+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:38.780 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:38.779+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:38.779094+0000) 2022-04-04T01:15:38.795 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:38.792+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:39.676 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:39.674+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:39.780 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:39.779+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:39.779288+0000) 2022-04-04T01:15:39.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:39.828+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:40.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:40.682+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:40.780 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:40.778+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:40.779516+0000) 2022-04-04T01:15:40.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:40.823+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:41.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:41.648+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:41.780 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:41.777+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:41.779730+0000) 2022-04-04T01:15:41.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:41.834+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:42.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:42.647+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:42.781 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:42.778+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:42.780001+0000) 2022-04-04T01:15:42.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:42.840+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:43.202 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:15:43.200+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:15:43.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:43.663+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:43.781 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:43.779+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:43.780221+0000) 2022-04-04T01:15:43.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:43.859+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:44.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:44.635+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:44.781 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:44.779+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:44.780419+0000) 2022-04-04T01:15:44.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:44.847+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:45.160 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:15:45.164 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:15:45.609 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:45.608+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:45.781 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:45.779+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:45.780645+0000) 2022-04-04T01:15:45.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:45.819+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:46.623 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:46.622+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:46.782 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:46.779+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:46.780871+0000) 2022-04-04T01:15:46.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:46.842+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:47.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:47.574+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:47.782 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:47.780+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:47.781101+0000) 2022-04-04T01:15:47.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:47.862+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:48.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:48.590+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:48.782 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:48.780+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:48.781280+0000) 2022-04-04T01:15:48.864 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:48.862+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:49.604 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:49.602+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:49.782 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:49.780+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:49.781496+0000) 2022-04-04T01:15:49.891 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:49.890+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:50.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:50.648+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:50.782 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:50.780+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:50.781721+0000) 2022-04-04T01:15:50.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:50.885+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:51.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:51.619+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:51.783 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:51.781+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:51.781938+0000) 2022-04-04T01:15:51.894 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:51.892+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:52.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:52.639+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:52.783 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:52.781+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:52.782118+0000) 2022-04-04T01:15:52.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:52.852+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:53.608 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:53.607+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:53.783 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:53.781+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:53.782315+0000) 2022-04-04T01:15:53.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:53.831+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:54.642 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:54.641+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:54.783 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:54.781+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:54.782453+0000) 2022-04-04T01:15:54.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:54.798+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:55.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:55.609+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:55.783 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:55.781+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:55.782641+0000) 2022-04-04T01:15:55.795 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:55.794+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:56.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:56.592+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:56.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:56.752+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:56.783 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:56.782+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:56.782855+0000) 2022-04-04T01:15:57.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:57.556+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:57.784 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:57.782+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:57.783071+0000) 2022-04-04T01:15:57.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:57.793+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:58.218 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:15:58.217+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:15:58.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:58.566+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:58.784 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:58.782+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:58.783264+0000) 2022-04-04T01:15:58.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:58.808+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:15:59.528 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:15:59.527+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:15:59.784 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:15:59.782+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:15:59.783456+0000) 2022-04-04T01:15:59.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:15:59.788+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:00.566 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:00.564+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:00.784 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:00.783+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:00.783645+0000) 2022-04-04T01:16:00.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:00.798+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:01.531 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:01.529+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:01.784 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:01.783+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:01.783875+0000) 2022-04-04T01:16:01.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:01.789+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:02.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:02.574+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:02.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:02.759+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:02.785 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:02.783+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:02.784080+0000) 2022-04-04T01:16:03.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:03.559+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:03.785 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:03.783+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:03.784251+0000) 2022-04-04T01:16:03.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:03.791+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:04.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:04.540+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:04.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:04.758+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:04.785 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:04.783+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:04.784466+0000) 2022-04-04T01:16:05.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:05.541+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:05.720 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:05.717+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:05.785 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:05.784+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:05.784687+0000) 2022-04-04T01:16:06.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:06.503+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:06.754 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:06.752+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:06.785 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:06.784+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:06.784900+0000) 2022-04-04T01:16:07.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:07.465+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:07.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:07.770+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:07.786 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:07.784+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:07.785118+0000) 2022-04-04T01:16:08.472 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:08.470+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:08.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:08.780+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:08.786 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:08.784+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:08.785346+0000) 2022-04-04T01:16:09.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:09.490+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:09.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:09.775+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:09.786 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:09.785+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:09.785562+0000) 2022-04-04T01:16:10.474 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:10.472+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:10.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:10.768+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:10.786 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:10.785+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:10.785807+0000) 2022-04-04T01:16:11.489 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:11.486+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:11.787 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:11.785+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:11.786024+0000) 2022-04-04T01:16:11.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:11.801+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:12.503 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:12.501+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:12.787 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:12.785+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:12.786269+0000) 2022-04-04T01:16:12.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:12.785+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:13.234 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:16:13.232+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:16:13.514 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:13.511+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 8 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:13.787 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:13.786+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:13.786491+0000) 2022-04-04T01:16:13.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:13.801+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:14.492 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:14.489+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:14.787 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:14.786+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:14.786708+0000) 2022-04-04T01:16:14.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:14.816+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:15.325 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:16:15.329 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:16:15.510 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:15.508+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:15.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:15.768+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:15.788 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:15.786+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:15.786950+0000) 2022-04-04T01:16:16.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:16.516+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:16.746 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:16.744+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:16.788 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:16.786+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:16.787097+0000) 2022-04-04T01:16:17.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:17.526+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:17.788 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:17.787+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:17.787300+0000) 2022-04-04T01:16:17.790 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:17.788+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:18.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:18.533+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:18.786 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:18.784+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:18.788 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:18.787+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:18.787500+0000) 2022-04-04T01:16:19.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:19.526+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:19.788 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:19.787+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:19.787725+0000) 2022-04-04T01:16:19.901 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:19.787+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:20.485 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:20.483+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:20.788 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:20.787+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:20.787946+0000) 2022-04-04T01:16:20.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:20.825+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:21.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:21.441+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:21.789 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:21.787+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:21.788056+0000) 2022-04-04T01:16:21.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:21.798+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:22.451 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:22.449+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:22.789 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:22.788+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:22.788225+0000) 2022-04-04T01:16:22.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:22.800+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:23.473 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:23.471+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:23.789 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:23.788+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:23.788420+0000) 2022-04-04T01:16:23.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:23.808+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:24.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:24.466+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:24.766 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:24.763+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:24.789 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:24.787+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:24.788641+0000) 2022-04-04T01:16:25.442 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:25.439+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:25.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:25.778+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:25.789 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:25.787+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:25.788861+0000) 2022-04-04T01:16:26.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:26.395+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:26.790 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:26.788+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:26.789081+0000) 2022-04-04T01:16:26.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:26.792+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:27.413 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:27.410+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:27.790 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:27.788+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:27.789281+0000) 2022-04-04T01:16:27.844 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:27.842+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:28.249 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:16:28.246+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:16:28.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:28.371+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:28.790 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:28.787+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:28.789408+0000) 2022-04-04T01:16:28.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:28.891+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:29.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:29.325+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:29.790 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:29.788+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:29.789607+0000) 2022-04-04T01:16:29.926 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:29.924+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:30.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:30.308+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:30.790 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:30.788+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:30.789793+0000) 2022-04-04T01:16:30.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:30.952+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:31.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:31.288+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:31.791 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:31.789+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:31.789994+0000) 2022-04-04T01:16:31.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:31.946+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:32.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:32.295+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:32.791 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:32.789+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:32.790180+0000) 2022-04-04T01:16:32.987 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:32.985+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:33.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:33.322+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:33.791 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:33.789+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:33.790378+0000) 2022-04-04T01:16:33.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:33.964+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:34.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:34.360+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:34.791 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:34.789+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:34.790611+0000) 2022-04-04T01:16:34.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:34.983+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:35.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:35.321+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:35.792 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:35.790+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:35.790824+0000) 2022-04-04T01:16:36.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:36.013+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:36.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:36.312+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:36.792 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:36.790+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:36.790997+0000) 2022-04-04T01:16:37.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:37.032+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:37.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:37.268+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:37.792 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:37.790+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:37.791212+0000) 2022-04-04T01:16:37.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:37.990+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:38.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:38.317+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:38.792 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:38.790+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:38.791401+0000) 2022-04-04T01:16:39.009 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:39.007+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:39.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:39.348+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:39.792 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:39.790+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:39.791634+0000) 2022-04-04T01:16:39.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:39.982+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:40.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:40.345+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:40.792 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:40.791+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:40.791824+0000) 2022-04-04T01:16:40.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:40.938+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:41.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:41.392+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:41.793 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:41.791+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:41.792039+0000) 2022-04-04T01:16:41.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:41.911+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:42.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:42.347+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:42.793 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:42.791+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:42.792237+0000) 2022-04-04T01:16:42.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:42.922+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:43.265 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:16:43.263+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:16:43.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:43.311+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:43.793 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:43.791+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:43.792431+0000) 2022-04-04T01:16:43.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:43.889+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:44.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:44.290+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:44.794 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:44.792+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:44.792678+0000) 2022-04-04T01:16:44.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:44.841+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:45.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:45.285+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:45.464 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:16:45.468 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:16:45.794 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:45.792+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:45.792957+0000) 2022-04-04T01:16:45.833 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:45.831+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:46.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:46.273+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:46.794 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:46.792+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:46.793198+0000) 2022-04-04T01:16:46.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:46.831+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:47.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:47.230+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:47.794 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:47.792+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:47.793403+0000) 2022-04-04T01:16:47.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:47.810+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:48.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:48.190+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:48.794 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:48.793+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:48.793546+0000) 2022-04-04T01:16:48.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:48.844+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:49.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:49.155+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:49.794 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:49.793+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:49.793711+0000) 2022-04-04T01:16:49.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:49.835+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:50.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:50.190+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:50.795 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:50.793+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:50.793900+0000) 2022-04-04T01:16:50.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:50.823+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:51.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:51.207+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:51.795 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:51.793+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:51.794085+0000) 2022-04-04T01:16:51.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:51.807+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:52.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:52.220+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:52.771 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:52.769+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:52.795 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:52.793+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:52.794265+0000) 2022-04-04T01:16:53.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:53.183+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:53.795 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:53.794+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:53.794469+0000) 2022-04-04T01:16:53.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:53.811+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:54.196 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:54.195+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:54.795 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:54.794+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:54.794616+0000) 2022-04-04T01:16:54.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:54.839+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:55.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:55.166+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:55.795 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:55.794+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:55.794827+0000) 2022-04-04T01:16:55.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:55.840+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:56.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:56.182+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:56.796 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:56.794+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:56.795047+0000) 2022-04-04T01:16:56.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:56.870+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:57.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:57.164+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:57.796 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:57.794+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:57.795271+0000) 2022-04-04T01:16:57.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:57.842+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:58.158 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:58.156+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:58.281 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:16:58.279+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:16:58.796 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:58.795+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:58.795432+0000) 2022-04-04T01:16:58.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:58.874+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:16:59.160 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:16:59.158+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:16:59.796 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:16:59.794+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:16:59.795619+0000) 2022-04-04T01:16:59.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:16:59.919+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:00.152 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:00.150+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:00.797 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:00.795+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:00.795848+0000) 2022-04-04T01:17:00.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:00.891+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:01.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:01.200+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:01.797 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:01.795+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:01.796065+0000) 2022-04-04T01:17:01.909 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:01.906+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:02.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:02.243+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:02.797 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:02.796+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:02.796325+0000) 2022-04-04T01:17:02.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:02.953+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:03.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:03.279+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:03.797 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:03.796+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:03.796517+0000) 2022-04-04T01:17:03.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:03.955+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:04.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:04.241+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:04.797 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:04.795+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:04.796738+0000) 2022-04-04T01:17:04.967 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:04.964+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:05.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:05.268+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:05.798 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:05.796+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:05.796963+0000) 2022-04-04T01:17:05.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:05.931+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:06.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:06.227+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:06.798 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:06.797+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:06.797155+0000) 2022-04-04T01:17:06.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:06.962+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:07.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:07.178+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:07.798 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:07.797+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:07.797376+0000) 2022-04-04T01:17:07.980 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:07.978+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:08.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:08.196+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:08.798 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:08.797+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:08.797579+0000) 2022-04-04T01:17:09.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:09.017+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:09.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:09.162+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:09.798 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:09.796+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:09.797794+0000) 2022-04-04T01:17:09.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:09.979+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:10.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:10.135+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:10.799 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:10.796+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:10.797988+0000) 2022-04-04T01:17:10.960 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:10.958+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:11.165 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:11.163+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:11.799 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:11.797+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:11.798201+0000) 2022-04-04T01:17:11.962 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:11.959+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:12.190 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:12.188+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:12.799 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:12.797+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:12.798395+0000) 2022-04-04T01:17:12.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:12.955+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:13.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:13.177+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:13.296 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:17:13.294+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:17:13.799 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:13.797+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:13.798534+0000) 2022-04-04T01:17:13.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:13.909+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:14.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:14.171+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:14.799 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:14.796+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:14.798712+0000) 2022-04-04T01:17:14.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:14.956+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:15.146 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:15.144+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:15.613 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:17:15.617 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:17:15.799 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:15.797+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:15.798922+0000) 2022-04-04T01:17:15.919 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:15.917+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:16.138 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:16.135+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:16.800 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:16.798+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:16.799113+0000) 2022-04-04T01:17:16.962 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:16.960+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:17.120 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:17.118+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:17.800 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:17.798+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:17.799226+0000) 2022-04-04T01:17:17.986 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:17.984+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:18.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:18.112+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:18.800 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:18.798+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:18.799367+0000) 2022-04-04T01:17:19.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:19.023+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:19.127 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:19.126+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:19.800 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:19.798+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:19.799557+0000) 2022-04-04T01:17:20.021 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:20.019+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:20.149 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:20.147+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:20.800 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:20.799+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:20.799757+0000) 2022-04-04T01:17:21.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:21.011+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:21.102 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:21.100+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:21.801 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:21.799+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:21.799946+0000) 2022-04-04T01:17:22.030 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:22.028+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:22.094 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:22.092+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:22.801 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:22.799+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:22.800097+0000) 2022-04-04T01:17:23.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:23.013+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:23.088 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:23.086+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:23.801 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:23.799+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:23.800292+0000) 2022-04-04T01:17:24.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:24.053+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:24.121 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:24.119+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:24.801 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:24.799+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:24.800512+0000) 2022-04-04T01:17:25.032 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:25.030+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:17:25.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:25.103+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:25.801 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:25.800+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:25.800747+0000) 2022-04-04T01:17:26.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:26.036+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:26.101 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:26.098+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:26.802 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:26.800+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:26.800990+0000) 2022-04-04T01:17:27.029 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:27.026+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:27.111 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:27.109+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:27.802 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:27.800+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:27.801187+0000) 2022-04-04T01:17:28.001 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:27.999+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:28.140 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:28.137+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:28.312 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:17:28.310+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:17:28.802 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:28.800+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:28.801354+0000) 2022-04-04T01:17:29.001 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:28.999+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:29.099 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:29.097+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:29.802 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:29.800+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:29.801527+0000) 2022-04-04T01:17:30.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:30.008+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:30.054 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:30.052+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:30.803 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:30.801+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:30.801803+0000) 2022-04-04T01:17:31.051 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:31.050+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:31.069 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:31.067+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:31.803 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:31.801+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:31.801999+0000) 2022-04-04T01:17:32.027 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:32.025+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:32.096 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:32.094+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:32.803 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:32.801+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:32.802234+0000) 2022-04-04T01:17:33.006 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:33.004+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:33.064 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:33.062+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:33.804 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:33.801+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:33.802462+0000) 2022-04-04T01:17:34.040 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:34.038+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:34.099 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:34.097+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:34.803 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:34.802+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:34.802663+0000) 2022-04-04T01:17:35.017 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:35.015+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:35.103 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:35.101+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:35.804 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:35.802+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:35.802878+0000) 2022-04-04T01:17:35.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:35.984+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:36.134 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:36.132+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:36.804 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:36.802+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:36.803052+0000) 2022-04-04T01:17:37.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:37.032+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:37.119 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:37.118+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:37.804 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:37.802+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:37.803244+0000) 2022-04-04T01:17:38.057 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:38.056+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:38.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:38.084+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:38.804 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:38.803+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:38.803462+0000) 2022-04-04T01:17:39.079 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:39.078+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:39.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:39.082+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:39.804 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:39.803+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:39.803681+0000) 2022-04-04T01:17:40.078 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:40.077+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:40.086 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:40.085+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:40.804 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:40.803+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:40.803898+0000) 2022-04-04T01:17:41.076 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:41.075+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:41.091 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:41.090+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:41.805 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:41.803+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:41.804108+0000) 2022-04-04T01:17:42.030 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:42.029+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:42.130 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:42.129+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:42.805 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:42.804+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:42.804348+0000) 2022-04-04T01:17:42.982 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:42.981+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:43.090 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:43.089+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:43.328 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:17:43.326+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:17:43.805 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:43.804+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:43.804537+0000) 2022-04-04T01:17:43.935 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:43.933+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:44.114 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:44.113+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:44.808 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:44.804+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:44.804728+0000) 2022-04-04T01:17:44.939 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:44.938+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:45.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:45.158+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:45.759 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:17:45.763 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:17:45.805 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:45.804+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:45.804981+0000) 2022-04-04T01:17:45.980 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:45.979+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:46.202 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:46.201+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:46.806 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:46.804+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:46.805159+0000) 2022-04-04T01:17:47.023 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:47.022+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:47.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:47.200+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:47.806 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:47.805+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:47.805351+0000) 2022-04-04T01:17:48.061 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:48.059+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:48.168 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:48.166+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:48.806 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:48.805+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:48.805549+0000) 2022-04-04T01:17:49.082 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:49.081+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:49.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:49.125+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:49.806 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:49.805+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:49.805765+0000) 2022-04-04T01:17:50.048 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:50.047+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:50.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:50.163+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:50.807 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:50.805+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:50.805990+0000) 2022-04-04T01:17:51.073 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:51.071+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:51.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:51.176+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:51.807 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:51.806+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:51.806211+0000) 2022-04-04T01:17:52.087 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:52.084+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:52.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:52.195+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:52.807 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:52.806+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:52.806454+0000) 2022-04-04T01:17:53.128 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:53.126+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:53.177 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:53.175+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:53.807 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:53.805+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:53.806638+0000) 2022-04-04T01:17:54.082 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:54.080+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:54.130 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:54.128+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:54.807 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:54.805+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:54.806833+0000) 2022-04-04T01:17:55.044 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:55.041+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:55.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:55.086+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:55.808 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:55.805+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:55.807028+0000) 2022-04-04T01:17:56.045 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:56.042+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:56.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:56.122+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:56.808 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:56.806+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:56.807252+0000) 2022-04-04T01:17:57.009 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:57.006+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:57.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:57.131+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:57.808 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:57.806+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:57.807469+0000) 2022-04-04T01:17:57.988 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:57.986+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:58.096 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:58.094+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:58.344 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:17:58.341+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:17:58.808 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:58.806+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:58.807689+0000) 2022-04-04T01:17:58.990 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:17:58.988+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:17:59.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:17:59.129+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:17:59.809 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:17:59.806+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:17:59.807897+0000) 2022-04-04T01:18:00.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:00.002+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:00.181 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:00.179+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:00.809 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:00.807+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:00.808107+0000) 2022-04-04T01:18:00.974 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:00.972+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:01.204 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:01.202+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:01.809 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:01.807+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:01.808315+0000) 2022-04-04T01:18:01.993 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:01.991+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:02.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:02.195+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:02.809 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:02.807+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:02.808516+0000) 2022-04-04T01:18:02.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:02.993+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:03.240 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:03.238+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:03.809 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:03.807+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:03.808711+0000) 2022-04-04T01:18:03.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:03.994+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:04.220 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:04.218+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:04.810 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:04.808+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:04.808865+0000) 2022-04-04T01:18:04.984 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:04.981+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:05.177 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:05.175+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:05.810 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:05.808+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:05.808996+0000) 2022-04-04T01:18:06.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:06.001+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:06.150 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:06.148+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:06.810 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:06.808+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:06.809148+0000) 2022-04-04T01:18:06.991 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:06.989+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:07.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:07.124+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:07.810 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:07.808+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:07.809342+0000) 2022-04-04T01:18:07.968 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:07.966+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:08.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:08.134+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:08.810 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:08.808+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:08.809552+0000) 2022-04-04T01:18:08.947 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:08.944+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:09.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:09.152+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:09.811 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:09.809+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:09.809764+0000) 2022-04-04T01:18:09.931 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:09.929+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:10.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:10.125+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:10.811 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:10.809+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:10.810005+0000) 2022-04-04T01:18:10.950 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:10.948+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:11.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:11.139+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:11.811 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:11.809+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:11.810173+0000) 2022-04-04T01:18:11.941 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:11.939+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:12.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:12.171+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:12.811 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:12.809+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:12.810374+0000) 2022-04-04T01:18:12.898 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:12.896+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:13.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:13.199+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:13.360 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:18:13.358+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:18:13.811 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:13.809+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:13.810609+0000) 2022-04-04T01:18:13.859 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:13.857+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:14.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:14.151+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:14.811 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:14.810+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:14.810854+0000) 2022-04-04T01:18:14.831 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:14.829+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:15.118 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:15.116+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:15.812 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:15.810+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:15.811074+0000) 2022-04-04T01:18:15.859 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:15.857+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:15.917 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:18:15.921 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:18:16.102 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:16.101+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:16.812 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:16.810+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:16.811319+0000) 2022-04-04T01:18:16.837 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:16.834+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:17.145 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:17.143+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:17.812 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:17.810+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:17.811487+0000) 2022-04-04T01:18:17.854 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:17.852+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:18.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:18.181+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:18.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:18.804+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:18.812 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:18.811+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:18.811689+0000) 2022-04-04T01:18:19.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:19.134+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:19.763 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:19.761+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:19.812 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:19.811+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:19.811914+0000) 2022-04-04T01:18:20.180 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:20.178+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:20.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:20.739+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:20.813 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:20.811+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:20.812131+0000) 2022-04-04T01:18:21.138 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:21.136+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:21.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:21.731+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:21.813 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:21.811+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:21.812372+0000) 2022-04-04T01:18:22.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:22.122+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:22.754 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:22.752+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:22.813 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:22.812+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:22.812536+0000) 2022-04-04T01:18:23.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:23.086+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:23.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:23.735+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:23.813 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:23.812+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:23.812709+0000) 2022-04-04T01:18:24.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:24.061+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:24.762 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:24.760+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:24.814 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:24.812+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:24.812924+0000) 2022-04-04T01:18:25.040 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:25.038+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:25.783 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:25.781+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:25.814 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:25.812+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:25.813143+0000) 2022-04-04T01:18:25.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:25.995+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:26.739 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:26.737+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:26.814 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:26.813+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:26.813394+0000) 2022-04-04T01:18:26.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:26.952+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:27.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:27.749+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:27.814 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:27.813+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:27.813621+0000) 2022-04-04T01:18:27.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:27.953+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:28.375 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:18:28.373+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:18:28.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:28.712+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:28.815 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:28.813+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:28.813862+0000) 2022-04-04T01:18:28.994 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:28.992+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:29.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:29.685+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:29.815 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:29.813+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:29.814040+0000) 2022-04-04T01:18:29.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:29.987+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:30.664 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:30.662+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:30.815 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:30.814+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:30.814253+0000) 2022-04-04T01:18:30.969 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:30.968+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:31.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:31.680+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:31.815 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:31.813+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:31.814414+0000) 2022-04-04T01:18:31.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:31.950+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:32.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:32.697+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:32.815 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:32.814+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:32.814688+0000) 2022-04-04T01:18:33.001 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:33.000+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:33.735 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:33.733+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:33.815 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:33.814+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:33.814943+0000) 2022-04-04T01:18:34.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:34.037+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:34.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:34.745+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:34.816 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:34.814+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:34.815102+0000) 2022-04-04T01:18:35.040 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:35.039+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:35.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:35.727+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:35.816 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:35.815+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:35.815245+0000) 2022-04-04T01:18:36.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:36.057+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:36.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:36.748+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:36.816 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:36.815+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:36.815468+0000) 2022-04-04T01:18:37.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:37.015+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:37.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:37.766+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:37.816 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:37.814+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:37.815695+0000) 2022-04-04T01:18:38.057 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:38.056+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:38.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:38.731+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:38.816 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:38.814+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:38.815917+0000) 2022-04-04T01:18:39.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:39.069+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:39.762 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:39.760+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:39.817 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:39.815+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:39.816140+0000) 2022-04-04T01:18:40.086 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:40.084+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:40.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:40.775+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:40.817 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:40.815+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:40.816367+0000) 2022-04-04T01:18:41.044 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:41.042+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:41.788 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:41.786+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:41.817 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:41.815+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:41.816559+0000) 2022-04-04T01:18:42.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:42.036+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:42.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:42.759+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:42.817 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:42.815+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:42.816758+0000) 2022-04-04T01:18:43.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:43.065+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:43.391 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:18:43.388+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:18:43.759 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:43.758+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:43.818 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:43.816+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:43.817006+0000) 2022-04-04T01:18:44.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:44.095+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:44.803 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:44.801+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:44.818 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:44.816+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:44.817223+0000) 2022-04-04T01:18:45.059 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:45.058+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:45.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:45.791+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:45.818 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:45.816+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:45.817441+0000) 2022-04-04T01:18:45.980 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:18:45.984 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:18:46.023 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:46.021+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:46.783 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:46.781+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:46.818 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:46.816+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:46.817595+0000) 2022-04-04T01:18:47.053 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:47.051+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:47.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:47.741+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:47.818 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:47.817+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:47.817799+0000) 2022-04-04T01:18:48.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:48.010+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:48.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:48.699+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:48.819 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:48.817+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:48.817987+0000) 2022-04-04T01:18:49.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:49.002+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:49.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:49.690+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:49.819 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:49.817+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:49.818184+0000) 2022-04-04T01:18:49.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:49.971+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:50.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:50.720+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:50.819 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:50.817+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:50.818320+0000) 2022-04-04T01:18:50.959 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:50.957+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:51.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:51.705+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:51.819 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:51.817+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:51.818463+0000) 2022-04-04T01:18:51.914 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:51.912+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:52.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:52.711+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:52.819 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:52.817+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:52.818693+0000) 2022-04-04T01:18:52.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:52.920+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:53.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:53.736+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:53.819 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:53.818+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:53.818860+0000) 2022-04-04T01:18:53.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:53.948+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:54.694 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:54.691+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:54.820 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:54.818+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:54.819021+0000) 2022-04-04T01:18:54.986 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:54.984+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:55.720 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:55.718+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:55.820 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:55.818+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:55.819222+0000) 2022-04-04T01:18:55.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:55.977+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:56.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:56.723+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:56.820 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:56.818+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:56.819401+0000) 2022-04-04T01:18:56.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:56.950+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:57.754 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:57.752+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:57.820 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:57.819+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:57.819630+0000) 2022-04-04T01:18:57.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:57.971+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:58.407 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:18:58.405+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:18:58.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:58.774+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:58.820 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:58.819+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:58.819849+0000) 2022-04-04T01:18:59.001 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:18:58.998+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:18:59.764 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:18:59.761+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:18:59.821 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:18:59.819+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:18:59.820057+0000) 2022-04-04T01:19:00.040 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:00.038+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:00.762 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:00.759+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:00.821 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:00.819+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:00.820247+0000) 2022-04-04T01:19:01.020 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:01.018+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:01.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:01.783+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:01.821 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:01.819+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:01.820468+0000) 2022-04-04T01:19:01.976 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:01.974+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:02.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:02.769+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:02.821 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:02.820+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:02.820700+0000) 2022-04-04T01:19:02.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:02.988+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:03.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:03.727+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:03.822 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:03.820+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:03.820896+0000) 2022-04-04T01:19:03.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:03.987+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:04.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:04.765+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:04.821 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:04.820+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:04.821067+0000) 2022-04-04T01:19:04.978 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:04.976+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:05.794 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:05.791+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:05.822 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:05.820+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:05.821261+0000) 2022-04-04T01:19:06.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:06.010+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:06.822 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:06.820+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:06.821411+0000) 2022-04-04T01:19:06.823 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:06.821+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:06.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:06.979+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:07.822 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:07.821+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:07.821572+0000) 2022-04-04T01:19:07.856 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:07.854+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:07.962 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:07.960+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:08.822 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:08.820+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:08.821754+0000) 2022-04-04T01:19:08.902 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:08.899+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:08.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:08.989+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:09.823 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:09.821+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:09.821982+0000) 2022-04-04T01:19:09.896 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:09.893+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:09.978 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:09.976+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:10.823 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:10.821+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:10.822170+0000) 2022-04-04T01:19:10.911 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:10.908+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:10.978 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:10.976+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:11.823 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:11.822+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:11.822355+0000) 2022-04-04T01:19:11.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:11.866+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:11.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:11.938+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:12.823 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:12.822+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:12.822524+0000) 2022-04-04T01:19:12.917 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:12.914+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:12.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:12.981+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:13.423 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:19:13.421+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:19:13.823 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:13.822+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:13.822717+0000) 2022-04-04T01:19:13.944 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:13.942+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:13.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:13.983+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:14.824 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:14.822+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:14.822901+0000) 2022-04-04T01:19:14.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:14.894+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:15.034 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:15.031+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:15.824 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:15.822+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:15.823050+0000) 2022-04-04T01:19:15.923 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:15.921+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:16.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:16.014+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:16.060 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:19:16.064 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:19:16.824 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:16.822+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:16.823221+0000) 2022-04-04T01:19:16.932 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:16.930+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:17.001 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:16.999+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:17.824 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:17.823+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:17.823454+0000) 2022-04-04T01:19:17.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:17.931+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:17.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:17.975+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:18.824 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:18.822+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:18.823618+0000) 2022-04-04T01:19:18.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:18.946+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:18.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:18.949+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:19.824 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:19.823+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:19.823810+0000) 2022-04-04T01:19:19.913 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:19.911+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:19.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:19.951+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:20.825 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:20.823+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:20.823929+0000) 2022-04-04T01:19:20.926 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:20.924+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:20.962 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:20.960+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:21.825 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:21.822+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:21.824085+0000) 2022-04-04T01:19:21.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:21.876+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:22.002 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:22.001+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:22.825 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:22.823+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:22.824236+0000) 2022-04-04T01:19:22.880 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:22.878+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:22.969 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:22.967+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:23.825 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:23.822+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:23.824386+0000) 2022-04-04T01:19:23.913 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:23.911+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:23.972 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:23.970+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:24.825 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:24.823+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:24.824580+0000) 2022-04-04T01:19:24.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:24.909+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:24.941 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:24.939+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:25.825 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:25.823+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:25.824822+0000) 2022-04-04T01:19:25.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:25.903+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:25.978 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:25.977+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:26.826 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:26.823+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:26.824992+0000) 2022-04-04T01:19:26.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:26.922+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:26.958 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:26.957+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:27.826 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:27.824+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:27.825185+0000) 2022-04-04T01:19:27.902 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:27.901+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:27.940 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:27.939+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:28.438 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:19:28.436+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:19:28.826 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:28.823+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:28.825397+0000) 2022-04-04T01:19:28.883 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:28.882+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:28.921 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:28.919+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:29.826 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:29.824+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:29.825602+0000) 2022-04-04T01:19:29.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:29.876+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:29.904 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:29.903+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:30.827 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:30.824+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:30.825812+0000) 2022-04-04T01:19:30.889 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:30.888+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:30.918 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:30.916+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:31.827 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:31.825+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:31.826048+0000) 2022-04-04T01:19:31.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:31.862+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:31.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:31.941+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:32.827 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:32.825+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:32.826283+0000) 2022-04-04T01:19:32.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:32.889+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:32.935 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:32.934+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:33.827 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:33.825+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:33.826474+0000) 2022-04-04T01:19:33.874 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:33.872+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:33.938 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:33.937+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:34.827 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:34.825+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:34.826672+0000) 2022-04-04T01:19:34.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:34.892+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:34.954 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:34.953+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:35.827 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:35.826+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:35.826899+0000) 2022-04-04T01:19:35.919 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:35.918+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:35.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:35.941+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:36.828 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:36.826+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:36.827121+0000) 2022-04-04T01:19:36.876 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:36.875+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:36.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:36.941+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:37.828 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:37.826+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:37.827344+0000) 2022-04-04T01:19:37.904 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:37.903+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:37.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:37.936+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:38.828 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:38.826+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:38.827570+0000) 2022-04-04T01:19:38.874 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:38.872+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:38.959 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:38.957+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:39.828 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:39.827+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:39.827820+0000) 2022-04-04T01:19:39.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:39.863+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:39.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:39.910+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:40.829 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:40.827+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:40.828057+0000) 2022-04-04T01:19:40.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:40.910+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:40.912 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:40.910+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:41.829 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:41.827+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:41.828300+0000) 2022-04-04T01:19:41.881 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:41.879+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:41.898 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:41.896+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:42.829 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:42.827+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:42.828505+0000) 2022-04-04T01:19:42.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:42.862+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:42.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:42.883+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:43.454 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:19:43.452+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:19:43.829 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:43.828+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:43.828708+0000) 2022-04-04T01:19:43.845 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:43.843+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:43.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:43.909+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:44.830 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:44.828+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:44.828903+0000) 2022-04-04T01:19:44.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:44.830+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:44.884 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:44.882+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:45.830 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:45.828+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:45.829156+0000) 2022-04-04T01:19:45.854 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:45.852+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:45.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:45.869+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:46.192 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:19:46.196 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:19:46.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:46.811+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:46.830 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:46.828+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:46.829402+0000) 2022-04-04T01:19:46.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:46.879+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:47.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:47.799+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:47.830 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:47.829+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:47.829589+0000) 2022-04-04T01:19:47.837 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:47.834+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:48.831 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:48.829+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:48.829798+0000) 2022-04-04T01:19:48.837 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:48.835+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:48.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:48.843+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:49.831 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:49.829+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:49.829986+0000) 2022-04-04T01:19:49.854 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:49.852+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:49.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:49.875+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:50.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:50.817+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:50.831 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:50.829+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:50.830181+0000) 2022-04-04T01:19:50.895 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:50.893+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:51.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:51.805+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:51.831 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:51.829+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:51.830398+0000) 2022-04-04T01:19:51.919 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:51.917+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:52.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:52.794+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:52.831 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:52.830+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:52.830669+0000) 2022-04-04T01:19:52.928 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:52.925+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:53.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:53.813+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:53.831 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:53.830+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:53.830895+0000) 2022-04-04T01:19:53.974 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:53.972+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:54.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:54.787+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:54.832 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:54.830+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:54.831089+0000) 2022-04-04T01:19:54.964 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:54.962+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:55.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:55.781+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:55.832 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:55.830+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:55.831260+0000) 2022-04-04T01:19:55.990 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:55.988+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:56.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:56.756+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:56.832 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:56.831+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:56.831433+0000) 2022-04-04T01:19:57.019 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:57.017+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:57.792 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:57.790+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:57.832 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:57.831+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:57.831651+0000) 2022-04-04T01:19:57.998 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:57.995+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:58.470 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:19:58.468+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:19:58.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:58.805+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:58.832 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:58.831+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:58.831825+0000) 2022-04-04T01:19:58.951 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:58.949+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:19:59.817 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:19:59.815+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:19:59.832 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:19:59.831+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:19:59.831958+0000) 2022-04-04T01:19:59.965 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:19:59.963+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:00.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:00.801+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:00.833 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:00.831+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:00.832115+0000) 2022-04-04T01:20:00.937 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:00.935+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:01.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:01.776+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:01.833 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:01.832+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:01.832281+0000) 2022-04-04T01:20:01.927 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:01.925+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:02.809 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:02.806+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:02.833 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:02.832+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:02.832425+0000) 2022-04-04T01:20:02.941 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:02.939+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:03.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:03.758+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:03.833 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:03.832+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:03.832617+0000) 2022-04-04T01:20:03.928 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:03.926+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:04.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:04.781+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:04.834 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:04.832+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:04.832857+0000) 2022-04-04T01:20:04.944 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:04.942+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:05.792 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:05.790+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:05.833 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:05.832+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:05.833043+0000) 2022-04-04T01:20:05.925 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:05.923+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:06.786 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:06.784+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:06.834 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:06.832+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:06.833237+0000) 2022-04-04T01:20:06.896 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:06.894+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:07.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:07.818+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:07.834 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:07.832+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:07.833457+0000) 2022-04-04T01:20:07.849 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:07.846+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:08.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:08.791+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:08.809 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:08.807+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:08.834 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:08.832+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:08.833686+0000) 2022-04-04T01:20:09.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:09.748+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:09.768 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:09.766+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:09.835 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:09.832+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:09.833896+0000) 2022-04-04T01:20:10.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:10.708+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:10.754 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:10.752+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:10.835 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:10.833+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:10.834113+0000) 2022-04-04T01:20:11.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:11.705+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:11.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:11.710+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:11.835 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:11.833+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:11.834256+0000) 2022-04-04T01:20:12.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:12.685+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:12.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:12.699+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:12.835 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:12.833+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:12.834474+0000) 2022-04-04T01:20:13.486 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:20:13.484+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:20:13.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:13.729+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:13.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:13.733+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:13.836 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:13.833+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:13.834697+0000) 2022-04-04T01:20:14.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:14.682+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:14.718 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:14.717+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:14.836 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:14.834+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:14.834916+0000) 2022-04-04T01:20:15.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:15.720+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:15.767 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:15.765+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:15.836 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:15.834+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:15.835087+0000) 2022-04-04T01:20:16.330 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:20:16.333 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:20:16.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:16.732+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:16.763 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:16.761+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:16.836 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:16.834+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:16.835346+0000) 2022-04-04T01:20:17.718 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:17.716+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:17.778 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:17.776+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:17.836 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:17.834+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:17.835568+0000) 2022-04-04T01:20:18.673 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:18.672+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:18.816 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:18.814+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:18.836 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:18.834+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:18.835738+0000) 2022-04-04T01:20:19.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:19.653+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:19.837 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:19.835+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:19.835873+0000) 2022-04-04T01:20:19.866 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:19.864+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:20.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:20.612+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:20.837 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:20.835+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:20.836028+0000) 2022-04-04T01:20:20.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:20.877+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:21.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:21.639+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:21.837 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:21.835+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:21.836189+0000) 2022-04-04T01:20:21.910 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:21.909+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:22.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:22.604+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:22.837 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:22.835+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:22.836311+0000) 2022-04-04T01:20:22.923 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:22.922+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:23.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:23.578+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:23.837 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:23.835+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:23.836473+0000) 2022-04-04T01:20:23.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:23.881+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:24.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:24.531+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:24.837 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:24.835+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:24.836596+0000) 2022-04-04T01:20:24.885 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:24.884+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:25.570 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:25.569+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:20:25.837 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:25.836+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:25.836824+0000) 2022-04-04T01:20:25.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:25.888+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:26.558 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:26.557+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:26.838 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:26.836+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:26.836935+0000) 2022-04-04T01:20:26.841 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:26.840+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:27.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:27.530+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:27.794 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:27.792+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:27.838 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:27.836+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:27.837088+0000) 2022-04-04T01:20:28.502 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:20:28.500+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:20:28.522 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:28.521+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:28.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:28.833+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:28.838 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:28.836+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:28.837248+0000) 2022-04-04T01:20:29.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:29.547+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:29.786 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:29.784+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:29.838 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:29.836+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:29.837473+0000) 2022-04-04T01:20:30.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:30.543+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:30.826 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:30.825+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:30.838 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:30.837+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:30.837679+0000) 2022-04-04T01:20:31.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:31.527+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:31.778 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:31.777+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:31.838 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:31.837+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:31.837847+0000) 2022-04-04T01:20:32.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:32.543+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:32.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:32.755+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:32.839 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:32.837+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:32.837990+0000) 2022-04-04T01:20:33.505 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:33.504+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:33.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:33.750+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:33.839 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:33.837+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:33.838182+0000) 2022-04-04T01:20:34.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:34.514+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:34.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:34.791+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:34.839 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:34.837+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:34.838368+0000) 2022-04-04T01:20:35.507 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:35.506+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:35.819 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:35.817+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:35.839 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:35.838+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:35.838465+0000) 2022-04-04T01:20:36.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:36.534+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:36.788 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:36.786+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:36.840 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:36.838+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:36.838629+0000) 2022-04-04T01:20:37.540 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:37.537+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:37.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:37.809+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:37.839 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:37.838+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:37.838865+0000) 2022-04-04T01:20:38.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:38.550+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:38.797 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:38.795+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:38.840 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:38.838+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:38.839035+0000) 2022-04-04T01:20:39.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:39.599+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:39.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:39.809+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:39.840 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:39.838+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:39.839196+0000) 2022-04-04T01:20:40.604 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:40.602+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:40.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:40.832+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:40.840 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:40.838+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:40.839380+0000) 2022-04-04T01:20:41.646 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:41.643+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:41.840 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:41.839+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:41.839597+0000) 2022-04-04T01:20:41.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:41.876+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:42.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:42.605+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:42.841 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:42.839+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:42.839844+0000) 2022-04-04T01:20:42.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:42.839+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:43.517 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:20:43.516+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:20:43.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:43.612+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:43.841 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:43.839+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:43.840008+0000) 2022-04-04T01:20:43.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:43.853+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:44.650 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:44.648+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:44.810 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:44.807+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:44.841 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:44.839+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:44.840147+0000) 2022-04-04T01:20:45.609 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:45.607+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:45.797 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:45.795+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:45.841 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:45.840+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:45.840349+0000) 2022-04-04T01:20:46.453 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:20:46.457 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:20:46.627 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:46.625+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:46.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:46.746+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:46.841 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:46.840+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:46.840497+0000) 2022-04-04T01:20:47.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:47.618+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:47.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:47.709+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:47.841 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:47.840+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:47.840659+0000) 2022-04-04T01:20:48.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:48.592+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:48.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:48.689+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:48.841 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:48.840+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:48.840879+0000) 2022-04-04T01:20:49.631 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:49.628+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:49.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:49.650+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:49.842 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:49.840+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:49.841074+0000) 2022-04-04T01:20:50.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:50.663+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:50.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:50.694+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:50.842 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:50.840+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:50.841212+0000) 2022-04-04T01:20:51.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:51.672+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:51.678 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:51.676+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:51.842 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:51.840+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:51.841382+0000) 2022-04-04T01:20:52.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:52.684+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:52.722 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:52.720+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:52.842 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:52.840+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:52.841538+0000) 2022-04-04T01:20:53.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:53.652+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:53.763 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:53.761+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:53.842 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:53.840+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:53.841737+0000) 2022-04-04T01:20:54.687 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:54.684+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:54.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:54.808+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:54.843 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:54.840+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:54.841879+0000) 2022-04-04T01:20:55.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:55.711+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:55.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:55.776+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:55.843 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:55.841+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:55.842039+0000) 2022-04-04T01:20:56.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:56.694+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:56.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:56.820+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:56.843 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:56.841+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:56.842185+0000) 2022-04-04T01:20:57.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:57.686+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:57.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:57.787+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:57.843 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:57.841+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:57.842392+0000) 2022-04-04T01:20:58.533 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:20:58.531+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:20:58.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:58.651+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:58.740 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:58.738+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:58.843 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:58.841+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:58.842569+0000) 2022-04-04T01:20:59.676 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:20:59.673+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:20:59.752 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:20:59.750+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:20:59.843 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:20:59.841+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:20:59.842738+0000) 2022-04-04T01:21:00.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:00.670+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:00.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:00.753+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:00.844 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:00.842+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:00.842949+0000) 2022-04-04T01:21:01.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:01.696+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:01.756 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:01.754+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:01.844 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:01.842+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:01.843168+0000) 2022-04-04T01:21:02.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:02.735+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:02.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:02.786+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:02.844 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:02.842+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:02.843371+0000) 2022-04-04T01:21:03.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:03.729+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:03.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:03.781+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:03.844 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:03.842+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:03.843516+0000) 2022-04-04T01:21:04.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:04.739+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:04.769 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:04.768+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:04.844 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:04.842+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:04.843671+0000) 2022-04-04T01:21:05.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:05.770+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:05.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:05.781+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:05.845 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:05.843+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:05.843860+0000) 2022-04-04T01:21:06.739 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:06.737+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:06.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:06.742+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:06.845 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:06.843+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:06.844046+0000) 2022-04-04T01:21:07.708 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:07.706+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:07.763 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:07.761+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:07.845 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:07.843+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:07.844237+0000) 2022-04-04T01:21:08.740 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:08.738+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:08.809 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:08.806+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:08.845 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:08.843+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:08.844421+0000) 2022-04-04T01:21:09.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:09.728+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:09.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:09.798+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:09.845 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:09.843+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:09.844588+0000) 2022-04-04T01:21:10.711 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:10.709+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:10.810 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:10.808+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:10.845 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:10.844+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:10.844761+0000) 2022-04-04T01:21:11.716 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:11.714+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:11.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:11.818+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:11.846 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:11.844+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:11.844982+0000) 2022-04-04T01:21:12.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:12.711+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:12.812 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:12.810+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:12.846 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:12.844+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:12.845187+0000) 2022-04-04T01:21:13.549 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:21:13.547+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:21:13.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:13.664+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:13.835 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:13.833+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:13.846 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:13.844+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:13.845372+0000) 2022-04-04T01:21:14.645 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:14.643+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:14.809 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:14.807+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:14.846 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:14.844+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:14.845506+0000) 2022-04-04T01:21:15.619 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:15.617+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:15.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:15.823+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:15.846 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:15.845+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:15.845662+0000) 2022-04-04T01:21:16.583 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:21:16.587 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:21:16.608 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:16.607+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:16.835 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:16.833+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:16.847 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:16.845+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:16.845885+0000) 2022-04-04T01:21:17.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:17.579+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:17.804 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:17.803+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:17.847 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:17.845+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:17.846083+0000) 2022-04-04T01:21:18.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:18.550+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:18.812 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:18.811+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:18.847 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:18.845+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:18.846301+0000) 2022-04-04T01:21:19.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:19.592+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:19.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:19.781+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:19.847 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:19.846+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:19.846467+0000) 2022-04-04T01:21:20.627 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:20.625+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:20.784 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:20.782+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:20.847 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:20.846+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:20.846614+0000) 2022-04-04T01:21:21.606 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:21.604+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:21.818 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:21.816+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:21.847 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:21.846+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:21.846820+0000) 2022-04-04T01:21:22.564 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:22.562+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:22.848 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:22.846+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:22.846978+0000) 2022-04-04T01:21:22.867 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:22.866+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:23.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:23.533+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:23.848 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:23.846+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:23.847196+0000) 2022-04-04T01:21:23.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:23.881+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:24.564 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:24.563+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:24.848 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:24.847+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:24.847395+0000) 2022-04-04T01:21:24.874 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:24.873+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:25.520 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:25.518+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:25.848 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:25.847+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:25.847622+0000) 2022-04-04T01:21:25.888 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:25.887+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:26.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:26.512+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:26.849 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:26.847+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:26.847843+0000) 2022-04-04T01:21:26.883 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:26.882+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:27.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:27.558+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:27.849 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:27.847+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:27.848089+0000) 2022-04-04T01:21:27.891 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:27.890+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:28.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:28.547+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:28.565 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:21:28.563+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:21:28.849 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:28.848+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:28.848306+0000) 2022-04-04T01:21:28.908 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:28.906+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:29.563 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:29.561+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:29.849 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:29.848+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:29.848522+0000) 2022-04-04T01:21:29.879 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:29.877+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:30.573 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:30.572+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:30.849 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:30.848+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:30.848672+0000) 2022-04-04T01:21:30.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:30.880+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:31.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:31.574+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:31.845 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:31.843+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:31.849 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:31.848+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:31.848822+0000) 2022-04-04T01:21:32.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:32.535+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:32.850 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:32.848+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:32.849026+0000) 2022-04-04T01:21:32.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:32.870+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:33.527 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:33.525+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:33.850 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:33.849+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:33.849243+0000) 2022-04-04T01:21:33.904 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:33.902+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:34.517 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:34.514+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:34.850 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:34.848+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:34.849460+0000) 2022-04-04T01:21:34.894 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:34.892+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:35.564 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:35.561+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:35.850 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:35.848+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:35.849608+0000) 2022-04-04T01:21:35.916 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:35.914+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:36.524 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:36.522+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:36.850 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:36.848+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:36.849777+0000) 2022-04-04T01:21:36.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:36.907+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:37.490 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:37.488+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:37.851 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:37.848+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:37.849948+0000) 2022-04-04T01:21:37.871 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:37.869+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:38.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:38.509+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:38.844 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:38.841+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:38.851 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:38.849+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:38.850148+0000) 2022-04-04T01:21:39.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:39.516+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:39.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:39.804+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:39.851 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:39.849+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:39.850340+0000) 2022-04-04T01:21:40.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:40.535+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:40.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:40.757+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:40.851 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:40.849+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:40.850499+0000) 2022-04-04T01:21:41.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:41.546+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:41.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:41.726+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:41.851 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:41.849+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:41.850677+0000) 2022-04-04T01:21:42.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:42.541+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:42.778 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:42.775+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:42.851 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:42.849+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:42.850831+0000) 2022-04-04T01:21:43.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:43.510+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:43.580 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:21:43.578+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:21:43.786 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:43.783+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:43.851 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:43.850+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:43.851005+0000) 2022-04-04T01:21:44.500 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:44.498+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:44.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:44.778+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:44.852 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:44.850+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:44.851203+0000) 2022-04-04T01:21:45.462 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:45.460+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:45.753 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:45.751+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:45.852 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:45.850+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:45.851447+0000) 2022-04-04T01:21:46.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:46.426+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:46.655 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:21:46.659 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:21:46.761 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:46.759+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:46.852 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:46.850+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:46.851689+0000) 2022-04-04T01:21:47.446 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:47.443+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:47.763 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:47.761+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:47.852 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:47.851+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:47.851811+0000) 2022-04-04T01:21:48.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:48.417+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:48.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:48.753+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:48.853 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:48.851+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:48.851996+0000) 2022-04-04T01:21:49.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:49.376+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:49.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:49.745+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:49.853 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:49.851+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:49.852157+0000) 2022-04-04T01:21:50.380 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:50.378+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:50.764 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:50.762+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:50.853 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:50.851+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:50.852331+0000) 2022-04-04T01:21:51.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:51.347+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:51.783 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:51.780+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:51.853 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:51.851+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:51.852535+0000) 2022-04-04T01:21:52.356 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:52.354+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:52.799 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:52.796+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:52.853 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:52.852+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:52.852754+0000) 2022-04-04T01:21:53.366 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:53.365+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:53.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:53.791+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:53.853 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:53.852+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:53.852961+0000) 2022-04-04T01:21:54.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:54.402+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:54.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:54.798+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:54.854 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:54.852+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:54.853134+0000) 2022-04-04T01:21:55.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:55.365+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:55.807 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:55.804+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:55.854 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:55.852+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:55.853222+0000) 2022-04-04T01:21:56.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:56.414+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:56.845 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:56.842+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:56.854 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:56.852+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:56.853412+0000) 2022-04-04T01:21:57.402 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:57.400+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:57.854 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:57.853+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:57.853631+0000) 2022-04-04T01:21:57.858 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:57.856+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:58.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:58.355+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:58.596 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:21:58.594+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:21:58.855 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:58.853+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:58.853851+0000) 2022-04-04T01:21:58.907 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:58.905+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:21:59.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:21:59.313+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:21:59.855 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:21:59.853+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:21:59.854042+0000) 2022-04-04T01:21:59.955 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:21:59.953+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:00.356 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:00.354+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:00.855 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:00.853+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:00.854267+0000) 2022-04-04T01:22:00.985 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:00.983+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:01.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:01.381+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:01.855 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:01.854+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:01.854482+0000) 2022-04-04T01:22:01.940 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:01.938+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:02.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:02.411+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:02.855 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:02.854+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:02.854682+0000) 2022-04-04T01:22:02.985 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:02.984+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:03.425 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:03.423+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:03.855 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:03.854+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:03.854853+0000) 2022-04-04T01:22:03.961 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:03.959+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:04.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:04.399+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:04.856 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:04.854+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:04.855046+0000) 2022-04-04T01:22:04.993 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:04.990+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:05.367 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:05.365+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:05.856 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:05.854+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:05.855282+0000) 2022-04-04T01:22:05.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:05.943+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:06.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:06.374+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:06.856 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:06.855+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:06.855498+0000) 2022-04-04T01:22:06.965 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:06.963+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:07.392 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:07.390+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:07.856 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:07.855+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:07.855724+0000) 2022-04-04T01:22:07.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:07.943+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:08.413 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:08.410+0000 7fd3b190b700 -1 osd.6 139 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:08.520 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-04T01:22:08.518+0000 7f8dd4f7b700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-04T01:22:08.536 INFO:tasks.ceph.mds.f.smithi186.stderr: -1> 2022-04-04T01:22:08.518+0000 7f8dd4f7b700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-04T01:22:08.559 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-04T01:22:08.857 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:08.855+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:08.855983+0000) 2022-04-04T01:22:08.936 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:08.935+0000 7f88d0ac1700 -1 osd.4 139 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:09.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:09.421+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:09.857 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:09.855+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:09.856176+0000) 2022-04-04T01:22:09.972 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:09.970+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:10.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:10.387+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:10.857 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:10.855+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:10.856399+0000) 2022-04-04T01:22:10.955 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:10.954+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:11.402 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:11.401+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:11.857 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:11.855+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:11.856534+0000) 2022-04-04T01:22:11.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:11.943+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:12.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:12.391+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:12.858 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:12.856+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:12.856780+0000) 2022-04-04T01:22:12.908 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:12.907+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 66 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:13.369 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:13.367+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:13.614 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:22:13.610+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:22:13.858 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:13.856+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:13.857028+0000) 2022-04-04T01:22:13.951 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:13.949+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:14.370 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:14.368+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:14.858 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:14.856+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:14.857187+0000) 2022-04-04T01:22:14.960 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:14.959+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:15.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:15.379+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:15.858 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:15.857+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:15.857355+0000) 2022-04-04T01:22:15.969 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:15.968+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:16.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:16.425+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:16.793 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:22:16.797 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:22:16.858 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:16.856+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:16.857527+0000) 2022-04-04T01:22:16.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:16.978+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:17.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:17.428+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:17.858 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:17.856+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:17.857745+0000) 2022-04-04T01:22:17.934 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:17.933+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:18.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:18.405+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:18.859 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:18.857+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:18.857965+0000) 2022-04-04T01:22:18.919 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:18.918+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:19.388 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:19.387+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:19.859 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:19.857+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:19.858152+0000) 2022-04-04T01:22:19.873 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:19.869+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:20.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:20.414+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:20.851 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:20.849+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:20.859 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:20.857+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:20.858339+0000) 2022-04-04T01:22:21.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:21.426+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:21.810 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:21.809+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:21.859 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:21.857+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:21.858536+0000) 2022-04-04T01:22:22.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:22.472+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:22.769 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:22.768+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:22.859 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:22.857+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:22.858676+0000) 2022-04-04T01:22:23.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:23.450+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:23.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:23.768+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:23.859 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:23.857+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:23.858822+0000) 2022-04-04T01:22:24.500 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:24.498+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:24.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:24.720+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:24.859 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:24.858+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:24.858996+0000) 2022-04-04T01:22:25.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:25.529+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:25.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:25.768+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:25.860 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:25.858+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:25.859192+0000) 2022-04-04T01:22:26.574 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:26.571+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:26.790 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:26.787+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:26.860 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:26.857+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:26.859366+0000) 2022-04-04T01:22:27.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:27.549+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:27.753 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:27.751+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:27.860 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:27.858+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:27.859528+0000) 2022-04-04T01:22:28.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:28.579+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:28.628 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:22:28.625+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:22:28.794 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:28.791+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:28.860 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:28.858+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:28.859716+0000) 2022-04-04T01:22:29.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:29.626+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:29.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:29.754+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:29.860 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:29.859+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:29.859898+0000) 2022-04-04T01:22:30.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:30.622+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:30.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:30.800+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:30.861 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:30.859+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:30.860028+0000) 2022-04-04T01:22:31.606 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:31.604+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:31.848 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:31.846+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:31.861 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:31.859+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:31.860192+0000) 2022-04-04T01:22:32.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:32.603+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:32.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:32.808+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:32.861 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:32.859+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:32.860405+0000) 2022-04-04T01:22:33.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:33.557+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:33.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:33.791+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:33.861 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:33.859+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:33.860630+0000) 2022-04-04T01:22:34.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:34.533+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:34.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:34.775+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:34.861 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:34.860+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:34.860798+0000) 2022-04-04T01:22:35.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:35.581+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:35.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:35.777+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:35.861 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:35.860+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:35.860941+0000) 2022-04-04T01:22:36.610 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:36.608+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:36.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:36.809+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:36.862 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:36.860+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:36.861103+0000) 2022-04-04T01:22:37.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:37.574+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:37.818 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:37.816+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:37.862 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:37.860+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:37.861319+0000) 2022-04-04T01:22:38.622 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:38.620+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:38.860 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:38.858+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:38.862 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:38.860+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:38.861510+0000) 2022-04-04T01:22:39.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:39.640+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:39.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:39.832+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:39.862 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:39.861+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:39.861722+0000) 2022-04-04T01:22:40.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:40.661+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:40.859 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:40.857+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:40.862 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:40.861+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:40.861907+0000) 2022-04-04T01:22:41.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:41.633+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:41.827 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:41.825+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:41.863 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:41.861+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:41.862104+0000) 2022-04-04T01:22:42.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:42.623+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:42.789 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:42.787+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:42.863 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:42.861+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:42.862274+0000) 2022-04-04T01:22:43.644 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:22:43.642+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:22:43.646 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:43.643+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:43.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:43.820+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:43.863 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:43.861+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:43.862432+0000) 2022-04-04T01:22:44.692 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:44.690+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:44.863 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:44.861+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:44.862615+0000) 2022-04-04T01:22:44.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:44.861+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:45.718 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:45.716+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:45.863 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:45.862+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:45.862840+0000) 2022-04-04T01:22:45.870 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:45.868+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:46.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:46.686+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:46.864 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:46.862+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:46.862984+0000) 2022-04-04T01:22:46.905 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:46.904+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:47.041 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:22:47.044 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:22:47.721 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:47.719+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:47.864 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:47.862+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:47.863145+0000) 2022-04-04T01:22:47.896 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:47.894+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:48.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:48.679+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:48.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:48.853+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:48.864 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:48.862+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:48.863307+0000) 2022-04-04T01:22:49.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:49.660+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:49.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:49.812+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:49.864 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:49.863+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:49.863519+0000) 2022-04-04T01:22:50.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:50.698+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:50.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:50.830+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:50.864 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:50.863+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:50.863705+0000) 2022-04-04T01:22:51.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:51.700+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:51.831 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:51.829+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:51.864 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:51.863+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:51.863904+0000) 2022-04-04T01:22:52.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:52.692+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:52.858 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:52.856+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:52.865 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:52.863+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:52.864135+0000) 2022-04-04T01:22:53.660 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:53.658+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:53.865 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:53.864+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:53.864322+0000) 2022-04-04T01:22:53.900 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:53.898+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:54.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:54.690+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:54.865 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:54.864+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:54.864478+0000) 2022-04-04T01:22:54.917 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:54.915+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:55.674 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:55.671+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:55.865 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:55.864+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:55.864548+0000) 2022-04-04T01:22:55.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:55.870+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:56.626 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:56.624+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:56.824 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:56.822+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:56.865 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:56.864+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:56.864718+0000) 2022-04-04T01:22:57.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:57.639+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:57.858 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:57.856+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:57.865 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:57.864+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:57.864940+0000) 2022-04-04T01:22:58.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:58.630+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:58.659 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:22:58.658+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:22:58.818 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:58.816+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:58.866 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:58.864+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:58.865121+0000) 2022-04-04T01:22:59.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:22:59.585+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:22:59.814 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:22:59.811+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:22:59.866 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:22:59.865+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:22:59.865269+0000) 2022-04-04T01:23:00.574 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:00.572+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:00.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:00.785+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:00.866 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:00.865+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:00.865428+0000) 2022-04-04T01:23:01.599 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:01.597+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:01.825 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:01.822+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:01.866 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:01.865+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:01.865640+0000) 2022-04-04T01:23:02.619 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:02.617+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:02.856 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:02.855+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:02.866 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:02.865+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:02.865852+0000) 2022-04-04T01:23:03.606 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:03.604+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:03.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:03.831+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:03.867 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:03.864+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:03.866041+0000) 2022-04-04T01:23:04.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:04.619+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:04.853 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:04.851+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:04.867 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:04.865+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:04.866174+0000) 2022-04-04T01:23:05.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:05.653+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:05.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:05.827+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:05.867 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:05.865+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:05.866309+0000) 2022-04-04T01:23:06.630 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:06.628+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:06.866 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:06.864+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:06.867 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:06.865+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:06.866468+0000) 2022-04-04T01:23:07.631 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:07.629+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:07.835 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:07.834+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:07.867 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:07.865+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:07.866604+0000) 2022-04-04T01:23:08.620 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:08.619+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:08.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:08.860+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:08.867 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:08.865+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:08.866803+0000) 2022-04-04T01:23:09.628 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:09.627+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:09.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:09.841+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:09.867 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:09.866+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:09.866993+0000) 2022-04-04T01:23:10.600 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:10.598+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:10.868 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:10.866+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:10.867225+0000) 2022-04-04T01:23:10.892 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:10.891+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:11.617 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:11.616+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:11.859 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:11.858+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:11.868 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:11.866+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:11.867381+0000) 2022-04-04T01:23:12.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:12.645+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:12.868 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:12.866+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:12.867553+0000) 2022-04-04T01:23:12.903 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:12.901+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:13.604 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:13.603+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:13.675 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:23:13.673+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:23:13.868 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:13.866+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:13.867742+0000) 2022-04-04T01:23:13.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:13.947+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:14.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:14.580+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:14.869 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:14.867+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:14.867969+0000) 2022-04-04T01:23:14.980 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:14.979+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:15.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:15.539+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:15.869 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:15.867+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:15.868151+0000) 2022-04-04T01:23:15.956 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:15.954+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:16.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:16.586+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:16.869 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:16.867+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:16.868306+0000) 2022-04-04T01:23:16.926 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:16.925+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:17.103 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:23:17.107 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:23:17.545 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:17.543+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:17.869 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:17.867+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:17.868517+0000) 2022-04-04T01:23:17.942 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:17.941+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:18.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:18.570+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:18.869 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:18.867+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:18.868704+0000) 2022-04-04T01:23:18.921 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:18.920+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:19.540 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:19.539+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:19.870 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:19.868+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:19.868834+0000) 2022-04-04T01:23:19.899 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:19.897+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:20.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:20.547+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:20.870 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:20.868+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:20.869006+0000) 2022-04-04T01:23:20.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:20.945+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:21.508 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:21.506+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:21.870 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:21.868+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:21.869220+0000) 2022-04-04T01:23:21.944 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:21.941+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:22.476 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:22.473+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:22.870 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:22.868+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:22.869461+0000) 2022-04-04T01:23:22.947 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:22.944+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:23.512 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:23.510+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:23.870 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:23.869+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:23.869676+0000) 2022-04-04T01:23:23.919 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:23.916+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:24.478 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:24.476+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:24.870 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:24.869+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:24.869859+0000) 2022-04-04T01:23:24.875 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:24.873+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:25.507 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:25.504+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:25.871 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:25.869+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:25.870077+0000) 2022-04-04T01:23:25.901 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:25.899+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:26.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:26.472+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:26.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:26.853+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:26.871 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:26.869+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:26.870310+0000) 2022-04-04T01:23:27.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:27.461+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:27.871 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:27.869+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:27.870384+0000) 2022-04-04T01:23:27.905 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:27.903+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:28.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:28.456+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:28.691 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:23:28.689+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:23:28.858 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:28.856+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:28.871 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:28.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:28.870528+0000) 2022-04-04T01:23:29.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:29.457+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:29.850 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:29.848+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:29.871 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:29.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:29.870697+0000) 2022-04-04T01:23:30.505 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:30.503+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:30.865 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:30.863+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:30.871 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:30.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:30.870871+0000) 2022-04-04T01:23:31.472 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:31.470+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:31.872 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:31.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:31.871002+0000) 2022-04-04T01:23:31.891 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:31.889+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:32.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:32.452+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:32.872 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:32.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:32.871148+0000) 2022-04-04T01:23:32.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:32.875+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:33.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:33.472+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:33.872 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:33.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:33.871323+0000) 2022-04-04T01:23:33.883 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:33.881+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:34.429 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:34.426+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:34.872 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:34.871+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:34.871400+0000) 2022-04-04T01:23:34.905 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:34.903+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:35.468 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:35.466+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:35.860 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:35.857+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:35.872 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:35.870+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:35.871572+0000) 2022-04-04T01:23:36.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:36.428+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:36.872 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:36.871+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:36.871718+0000) 2022-04-04T01:23:36.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:36.907+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:37.454 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:37.451+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:37.872 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:37.871+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:37.871899+0000) 2022-04-04T01:23:37.941 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:37.939+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:38.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:38.479+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:38.873 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:38.871+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:38.872075+0000) 2022-04-04T01:23:38.971 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:38.969+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:39.478 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:39.476+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:39.873 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:39.871+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:39.872251+0000) 2022-04-04T01:23:39.944 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:39.941+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:40.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:40.508+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:40.873 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:40.872+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:40.872384+0000) 2022-04-04T01:23:40.937 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:40.935+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:41.528 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:41.526+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:41.873 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:41.871+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:41.872601+0000) 2022-04-04T01:23:41.958 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:41.955+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:42.563 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:42.561+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:42.873 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:42.872+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:42.872816+0000) 2022-04-04T01:23:42.981 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:42.979+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:43.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:43.542+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:43.707 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:23:43.705+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:23:43.874 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:43.872+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:43.872925+0000) 2022-04-04T01:23:44.015 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:44.012+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:44.507 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:44.505+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:44.874 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:44.872+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:44.873050+0000) 2022-04-04T01:23:45.060 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:45.058+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:45.506 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:45.504+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:45.874 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:45.873+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:45.873246+0000) 2022-04-04T01:23:46.036 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:46.034+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:46.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:46.549+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:46.874 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:46.873+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:46.873420+0000) 2022-04-04T01:23:47.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:47.036+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:47.192 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:23:47.195 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:23:47.558 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:47.556+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:47.874 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:47.873+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:47.873618+0000) 2022-04-04T01:23:48.009 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:48.007+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:48.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:48.579+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:48.874 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:48.872+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:48.873815+0000) 2022-04-04T01:23:49.047 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:49.046+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:49.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:49.575+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:49.875 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:49.872+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:49.873942+0000) 2022-04-04T01:23:50.018 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:50.016+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:50.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:50.550+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:50.875 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:50.873+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:50.874112+0000) 2022-04-04T01:23:51.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:51.011+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:51.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:51.552+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:51.875 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:51.873+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:51.874261+0000) 2022-04-04T01:23:52.058 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:52.057+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:52.573 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:52.571+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:52.875 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:52.873+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:52.874461+0000) 2022-04-04T01:23:53.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:53.102+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:53.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:53.540+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:53.875 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:53.873+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:53.874627+0000) 2022-04-04T01:23:54.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:54.134+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:54.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:54.579+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:54.876 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:54.873+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:54.874797+0000) 2022-04-04T01:23:55.104 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:55.102+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:55.606 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:55.603+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:55.876 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:55.874+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:55.874953+0000) 2022-04-04T01:23:56.082 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:56.081+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:56.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:56.574+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:56.876 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:56.874+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:56.875150+0000) 2022-04-04T01:23:57.033 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:57.032+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:57.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:57.611+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:57.876 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:57.874+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:57.875359+0000) 2022-04-04T01:23:58.035 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:58.033+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:58.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:58.622+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:58.723 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:23:58.721+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:23:58.876 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:58.874+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:58.875574+0000) 2022-04-04T01:23:59.031 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:23:59.029+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:23:59.624 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:23:59.622+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:23:59.877 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:23:59.875+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:23:59.875804+0000) 2022-04-04T01:24:00.049 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:00.047+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:00.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:00.657+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:00.877 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:00.875+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:00.876006+0000) 2022-04-04T01:24:01.078 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:01.076+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:01.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:01.634+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:01.877 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:01.875+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:01.876216+0000) 2022-04-04T01:24:02.102 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:02.101+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:02.630 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:02.629+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:02.877 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:02.875+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:02.876417+0000) 2022-04-04T01:24:03.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:03.114+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:03.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:03.670+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:03.877 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:03.875+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:03.876636+0000) 2022-04-04T01:24:04.109 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:04.108+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:04.631 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:04.629+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:04.878 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:04.876+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:04.876845+0000) 2022-04-04T01:24:05.107 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:05.105+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:05.649 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:05.648+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:05.878 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:05.876+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:05.877058+0000) 2022-04-04T01:24:06.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:06.104+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:06.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:06.630+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:06.878 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:06.876+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:06.877253+0000) 2022-04-04T01:24:07.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:07.135+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:07.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:07.640+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:07.878 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:07.876+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:07.877468+0000) 2022-04-04T01:24:08.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:08.113+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:08.604 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:08.603+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:08.878 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:08.877+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:08.877695+0000) 2022-04-04T01:24:09.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:09.136+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:09.636 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:09.634+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:09.878 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:09.877+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:09.877837+0000) 2022-04-04T01:24:10.107 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:10.106+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:10.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:10.593+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:10.879 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:10.877+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:10.878076+0000) 2022-04-04T01:24:11.124 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:11.123+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:11.586 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:11.584+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:11.879 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:11.877+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:11.878276+0000) 2022-04-04T01:24:12.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:12.163+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:12.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:12.600+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:12.879 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:12.877+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:12.878501+0000) 2022-04-04T01:24:13.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:13.205+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:13.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:13.634+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:13.739 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:24:13.737+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:24:13.879 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:13.878+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:13.878685+0000) 2022-04-04T01:24:14.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:14.157+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:14.675 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:14.673+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:14.880 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:14.878+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:14.878884+0000) 2022-04-04T01:24:15.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:15.147+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:15.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:15.690+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:15.880 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:15.878+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:15.879075+0000) 2022-04-04T01:24:16.143 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:16.141+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:16.706 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:16.704+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:16.880 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:16.878+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:16.879273+0000) 2022-04-04T01:24:17.118 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:17.115+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:17.323 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:24:17.326 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:24:17.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:17.712+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:17.880 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:17.879+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:17.879527+0000) 2022-04-04T01:24:18.100 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:18.097+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:18.721 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:18.719+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:18.880 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:18.879+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:18.879701+0000) 2022-04-04T01:24:19.088 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:19.086+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:19.767 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:19.765+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:19.880 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:19.879+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:19.879856+0000) 2022-04-04T01:24:20.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:20.079+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:20.809 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:20.806+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:20.881 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:20.879+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:20.880059+0000) 2022-04-04T01:24:21.056 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:21.054+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:21.771 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:21.769+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:21.881 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:21.879+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:21.880285+0000) 2022-04-04T01:24:22.074 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:22.072+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:22.817 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:22.815+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:22.881 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:22.880+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:22.880449+0000) 2022-04-04T01:24:23.057 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:23.054+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:23.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:23.846+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:23.881 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:23.880+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:23.880654+0000) 2022-04-04T01:24:24.072 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:24.070+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:24.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:24.841+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:24.881 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:24.880+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:24.880877+0000) 2022-04-04T01:24:25.063 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:25.061+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:25.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:25.818+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:25.882 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:25.880+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:25.881086+0000) 2022-04-04T01:24:26.028 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:26.026+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:26.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:26.843+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:26.882 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:26.881+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:26.881299+0000) 2022-04-04T01:24:27.042 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:27.040+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:27.833 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:27.831+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:27.882 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:27.881+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:27.881509+0000) 2022-04-04T01:24:28.001 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:27.999+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:28.754 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:24:28.752+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:24:28.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:28.834+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:28.882 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:28.881+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:28.881700+0000) 2022-04-04T01:24:28.963 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:28.961+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:29.797 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:29.795+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:29.883 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:29.881+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:29.881910+0000) 2022-04-04T01:24:29.924 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:29.922+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:30.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:30.806+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:30.883 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:30.881+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:30.882063+0000) 2022-04-04T01:24:30.911 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:30.909+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:31.768 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:31.766+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:31.883 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:31.882+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:31.882250+0000) 2022-04-04T01:24:31.922 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:31.920+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:32.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:32.760+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:32.883 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:32.881+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:32.882426+0000) 2022-04-04T01:24:32.902 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:32.900+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:33.747 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:33.744+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:33.878 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:33.876+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:33.883 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:33.881+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:33.882597+0000) 2022-04-04T01:24:34.790 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:34.788+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:34.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:34.830+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:34.883 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:34.881+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:34.882841+0000) 2022-04-04T01:24:35.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:35.768+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:35.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:35.840+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:35.883 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:35.881+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:35.883024+0000) 2022-04-04T01:24:36.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:36.811+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:36.865 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:36.862+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:36.884 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:36.881+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:36.883177+0000) 2022-04-04T01:24:37.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:37.805+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:37.841 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:37.839+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:37.884 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:37.882+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:37.883381+0000) 2022-04-04T01:24:38.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:38.787+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:38.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:38.880+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:38.884 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:38.882+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:38.883603+0000) 2022-04-04T01:24:39.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:39.828+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:39.884 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:39.882+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:39.883804+0000) 2022-04-04T01:24:39.893 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:39.892+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:40.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:40.853+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:40.884 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:40.883+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:40.884009+0000) 2022-04-04T01:24:40.895 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:40.893+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:41.885 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:41.883+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:41.884227+0000) 2022-04-04T01:24:41.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:41.902+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:41.905 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:41.902+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:42.885 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:42.883+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:42.884455+0000) 2022-04-04T01:24:42.894 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:42.892+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:42.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:42.941+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:43.770 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:24:43.768+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:24:43.885 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:43.883+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:43.884665+0000) 2022-04-04T01:24:43.926 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:43.924+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:43.928 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:43.925+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:44.883 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:44.881+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:44.885 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:44.884+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:44.884882+0000) 2022-04-04T01:24:44.908 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:44.907+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:45.839 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:45.837+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:45.886 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:45.884+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:45.885136+0000) 2022-04-04T01:24:45.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:45.922+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:46.825 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:46.823+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:46.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:46.880+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:46.886 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:46.884+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:46.885363+0000) 2022-04-04T01:24:47.478 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:24:47.482 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:24:47.836 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:47.834+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:47.886 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:47.884+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:47.885613+0000) 2022-04-04T01:24:47.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:47.926+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:48.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:48.841+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:48.886 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:48.885+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:48.885841+0000) 2022-04-04T01:24:48.895 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:48.893+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:49.836 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:49.834+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:49.871 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:49.870+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:49.887 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:49.885+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:49.886033+0000) 2022-04-04T01:24:50.835 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:50.833+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:50.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:50.863+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:50.887 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:50.885+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:50.886254+0000) 2022-04-04T01:24:51.790 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:51.788+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:51.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:51.877+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:51.887 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:51.885+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:51.886421+0000) 2022-04-04T01:24:52.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:52.800+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:52.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:52.834+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:52.887 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:52.886+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:52.886635+0000) 2022-04-04T01:24:53.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:53.832+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:53.872 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:53.870+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:53.887 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:53.886+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:53.886872+0000) 2022-04-04T01:24:54.859 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:54.858+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:54.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:54.863+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:54.888 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:54.886+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:54.887082+0000) 2022-04-04T01:24:55.888 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:55.886+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:55.889 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:55.886+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:55.887326+0000) 2022-04-04T01:24:55.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:55.887+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:56.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:56.873+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:56.888 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:56.886+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:56.887543+0000) 2022-04-04T01:24:56.936 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:56.935+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:57.888 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:57.887+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:57.887739+0000) 2022-04-04T01:24:57.893 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:57.892+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:57.920 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:57.919+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:58.786 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:24:58.784+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:24:58.856 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:58.855+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:58.888 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:58.887+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:58.887869+0000) 2022-04-04T01:24:58.919 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:58.918+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:59.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:24:59.870+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:24:59.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:24:59.876+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:24:59.888 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:24:59.887+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:24:59.888077+0000) 2022-04-04T01:25:00.880 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:00.879+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:00.889 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:00.887+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:00.888281+0000) 2022-04-04T01:25:00.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:00.895+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:01.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:01.853+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:01.869 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:01.868+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:01.889 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:01.888+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:01.888427+0000) 2022-04-04T01:25:02.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:02.876+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:02.889 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:02.888+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:02.888637+0000) 2022-04-04T01:25:02.913 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:02.912+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:03.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:03.880+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:03.889 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:03.888+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:03.888828+0000) 2022-04-04T01:25:03.920 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:03.919+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:04.890 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:04.888+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:04.889012+0000) 2022-04-04T01:25:04.906 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:04.904+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:04.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:04.909+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:05.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:05.881+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:05.890 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:05.888+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:05.889214+0000) 2022-04-04T01:25:05.901 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:05.900+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:06.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:06.868+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:06.890 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:06.889+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:06.889422+0000) 2022-04-04T01:25:06.905 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:06.903+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:07.890 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:07.888+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:07.889659+0000) 2022-04-04T01:25:07.906 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:07.905+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:07.919 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:07.917+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:08.890 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:08.889+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:08.889837+0000) 2022-04-04T01:25:08.899 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:08.898+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:08.941 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:08.940+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:09.891 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:09.889+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:09.890050+0000) 2022-04-04T01:25:09.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:09.902+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:09.951 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:09.949+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:10.891 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:10.890+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:10.890203+0000) 2022-04-04T01:25:10.939 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:10.936+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:10.993 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:10.990+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:11.891 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:11.889+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:11.890425+0000) 2022-04-04T01:25:11.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:11.943+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:11.969 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:11.966+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:12.891 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:12.890+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:12.890657+0000) 2022-04-04T01:25:12.910 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:12.908+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 16 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:12.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:12.963+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:13.802 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:25:13.800+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:25:13.870 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:13.867+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:13.891 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:13.890+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:13.890873+0000) 2022-04-04T01:25:13.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:13.951+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:14.892 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:14.890+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:14.891058+0000) 2022-04-04T01:25:14.916 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:14.913+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:14.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:14.954+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:15.892 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:15.891+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:15.891234+0000) 2022-04-04T01:25:15.936 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:15.934+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:15.952 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:15.950+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:16.892 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:16.890+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:16.891444+0000) 2022-04-04T01:25:16.946 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:16.944+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:16.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:16.949+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:17.608 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:25:17.612 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:25:17.892 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:17.890+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:17.891702+0000) 2022-04-04T01:25:17.927 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:17.925+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:17.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:17.991+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:18.893 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:18.890+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:18.891880+0000) 2022-04-04T01:25:18.916 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:18.913+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:19.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:19.037+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:19.880 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:19.878+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:19.893 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:19.891+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:19.892057+0000) 2022-04-04T01:25:20.069 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:20.066+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:20.893 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:20.891+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:20.892261+0000) 2022-04-04T01:25:20.924 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:20.922+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:21.021 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:21.018+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:21.893 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:21.891+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:21.892518+0000) 2022-04-04T01:25:21.921 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:21.918+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:22.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:22.017+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:22.893 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:22.891+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:22.892766+0000) 2022-04-04T01:25:22.913 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:22.910+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:23.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:23.060+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:23.894 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:23.891+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:23.893010+0000) 2022-04-04T01:25:23.949 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:23.947+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:24.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:24.093+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:24.894 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:24.892+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:24.893181+0000) 2022-04-04T01:25:24.912 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:24.910+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:25.120 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:25.117+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:25.894 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:25.892+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:25.893395+0000) 2022-04-04T01:25:25.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:25.951+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:26.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:26.157+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:26.894 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:26.892+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:26.893605+0000) 2022-04-04T01:25:26.994 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:26.992+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:27.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:27.123+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:27.895 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:27.893+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:27.893818+0000) 2022-04-04T01:25:27.982 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:27.980+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:28.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:28.145+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:28.817 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:25:28.815+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:25:28.895 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:28.893+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:28.894009+0000) 2022-04-04T01:25:28.935 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:28.933+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:29.173 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:29.170+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:29.895 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:29.893+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:29.894192+0000) 2022-04-04T01:25:29.917 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:29.915+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:30.162 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:30.159+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:30.895 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:30.893+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:30.894352+0000) 2022-04-04T01:25:30.959 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:30.957+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:31.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:31.154+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:31.895 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:31.893+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:31.894561+0000) 2022-04-04T01:25:31.917 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:31.915+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:32.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:32.119+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:32.895 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:32.894+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:32.894766+0000) 2022-04-04T01:25:32.958 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:32.956+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:33.168 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:33.165+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:33.896 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:33.894+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:33.894957+0000) 2022-04-04T01:25:33.938 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:33.936+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:34.208 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:34.206+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:34.896 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:34.894+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:34.895153+0000) 2022-04-04T01:25:34.959 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:34.956+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:35.171 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:35.169+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:35.896 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:35.894+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:35.895355+0000) 2022-04-04T01:25:35.998 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:35.996+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:36.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:36.184+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:36.896 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:36.894+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:36.895557+0000) 2022-04-04T01:25:37.002 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:37.000+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:37.150 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:37.148+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:37.896 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:37.895+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:37.895763+0000) 2022-04-04T01:25:38.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:38.035+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:38.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:38.190+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:38.897 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:38.895+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:38.895992+0000) 2022-04-04T01:25:38.999 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:38.997+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:39.180 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:39.178+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:39.897 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:39.895+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:39.896192+0000) 2022-04-04T01:25:39.955 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:39.953+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:40.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:40.224+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:40.897 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:40.895+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:40.896359+0000) 2022-04-04T01:25:40.986 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:40.983+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:41.204 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:41.202+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:41.897 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:41.895+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:41.896589+0000) 2022-04-04T01:25:41.946 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:41.944+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:42.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:42.224+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:42.897 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:42.896+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:42.896757+0000) 2022-04-04T01:25:42.920 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:42.918+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:43.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:43.187+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:43.833 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:25:43.831+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:25:43.898 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:43.896+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:43.896976+0000) 2022-04-04T01:25:43.938 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:43.936+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:44.142 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:44.140+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:44.890 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:44.888+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:44.898 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:44.896+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:44.897131+0000) 2022-04-04T01:25:45.148 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:45.146+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:45.898 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:45.896+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:45.897317+0000) 2022-04-04T01:25:45.919 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:45.917+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:46.114 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:46.113+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:46.898 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:46.897+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:46.897516+0000) 2022-04-04T01:25:46.914 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:46.912+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:47.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:47.065+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:47.697 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:25:47.701 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:25:47.899 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:47.897+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:47.897801+0000) 2022-04-04T01:25:47.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:47.908+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:48.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:48.080+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:48.899 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:48.897+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:48.897960+0000) 2022-04-04T01:25:48.923 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:48.921+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:49.070 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:49.068+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:49.899 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:49.897+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:49.898153+0000) 2022-04-04T01:25:49.910 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:49.908+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:50.104 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:50.102+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:50.891 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:50.889+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:50.899 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:50.898+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:50.898352+0000) 2022-04-04T01:25:51.069 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:51.067+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:51.844 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:51.843+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:51.899 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:51.898+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:51.898534+0000) 2022-04-04T01:25:52.032 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:52.031+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:52.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:52.832+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:52.899 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:52.898+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:52.898764+0000) 2022-04-04T01:25:53.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:53.054+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:53.831 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:53.829+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:53.900 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:53.898+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:53.899010+0000) 2022-04-04T01:25:54.022 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:54.021+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:54.870 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:54.868+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:54.900 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:54.898+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:54.899205+0000) 2022-04-04T01:25:54.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:54.980+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:55.900 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:55.899+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:55.899394+0000) 2022-04-04T01:25:55.904 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:55.903+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:56.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:56.002+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:56.900 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:56.899+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:56.899592+0000) 2022-04-04T01:25:56.952 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:56.951+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:56.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:56.953+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:57.901 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:57.899+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:57.899854+0000) 2022-04-04T01:25:57.967 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:57.966+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:57.971 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:57.970+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:58.849 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:25:58.847+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:25:58.901 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:58.899+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:58.900076+0000) 2022-04-04T01:25:58.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:58.949+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:58.986 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:58.984+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:25:59.901 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:25:59.899+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:25:59.900249+0000) 2022-04-04T01:25:59.972 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:25:59.971+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:25:59.973 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:25:59.971+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:00.901 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:00.900+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:00.900449+0000) 2022-04-04T01:26:00.925 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:00.923+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:00.938 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:00.937+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:01.901 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:01.899+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:01.900637+0000) 2022-04-04T01:26:01.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:01.920+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:01.964 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:01.963+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:02.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:02.873+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:02.902 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:02.899+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:02.900856+0000) 2022-04-04T01:26:02.959 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:02.958+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:03.856 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:03.855+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:03.902 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:03.900+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:03.901038+0000) 2022-04-04T01:26:03.941 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:03.939+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:04.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:04.845+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:04.902 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:04.900+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:04.901259+0000) 2022-04-04T01:26:04.985 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:04.983+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:05.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:05.885+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:05.902 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:05.900+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:05.901462+0000) 2022-04-04T01:26:05.964 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:05.962+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:06.902 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:06.900+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:06.901628+0000) 2022-04-04T01:26:06.916 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:06.913+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:06.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:06.993+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:07.902 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:07.900+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:07.901809+0000) 2022-04-04T01:26:07.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:07.949+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:08.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:08.001+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:08.903 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:08.901+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:08.902006+0000) 2022-04-04T01:26:08.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:08.988+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:09.041 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:09.038+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:09.903 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:09.901+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:09.902154+0000) 2022-04-04T01:26:10.013 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:10.010+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:10.031 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:10.029+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:10.903 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:10.901+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:10.902342+0000) 2022-04-04T01:26:10.970 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:10.968+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:11.034 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:11.032+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:11.903 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:11.901+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:11.902527+0000) 2022-04-04T01:26:11.942 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:11.940+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:11.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:11.985+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:12.903 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:12.902+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:12.902791+0000) 2022-04-04T01:26:12.961 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:12.959+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:12.971 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:12.968+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:13.865 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:26:13.863+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:26:13.904 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:13.902+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:13.902970+0000) 2022-04-04T01:26:13.962 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:13.959+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:13.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:13.990+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:14.904 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:14.902+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:14.903158+0000) 2022-04-04T01:26:14.993 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:14.991+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:15.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:15.030+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:15.904 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:15.902+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:15.903305+0000) 2022-04-04T01:26:16.039 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:16.037+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:16.043 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:16.041+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:16.904 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:16.902+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:16.903462+0000) 2022-04-04T01:26:17.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:17.036+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:17.048 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:17.046+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:17.820 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:26:17.823 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:26:17.904 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:17.902+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:17.903677+0000) 2022-04-04T01:26:18.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:18.033+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:18.077 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:18.074+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:18.905 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:18.903+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:18.903885+0000) 2022-04-04T01:26:18.994 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:18.992+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:19.068 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:19.066+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:19.905 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:19.903+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:19.904086+0000) 2022-04-04T01:26:19.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:19.996+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:20.060 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:20.057+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:20.905 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:20.903+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:20.904251+0000) 2022-04-04T01:26:21.023 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:21.021+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:21.049 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:21.047+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:21.905 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:21.903+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:21.904440+0000) 2022-04-04T01:26:22.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:22.031+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:22.035 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:22.033+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:22.905 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:22.904+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:22.904619+0000) 2022-04-04T01:26:23.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:23.010+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:23.021 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:23.019+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:23.905 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:23.904+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:23.904841+0000) 2022-04-04T01:26:24.013 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:24.011+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:24.023 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:24.021+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:24.906 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:24.904+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:24.905012+0000) 2022-04-04T01:26:25.000 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:24.998+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:25.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:25.039+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:25.906 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:25.904+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:25.905203+0000) 2022-04-04T01:26:25.961 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:25.958+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:26.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:26.074+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:26.906 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:26.904+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:26.905400+0000) 2022-04-04T01:26:26.947 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:26.945+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:27.053 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:27.051+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:27.906 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:27.905+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:27.905605+0000) 2022-04-04T01:26:27.976 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:27.974+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:28.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:28.014+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:28.867 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:26:28.865+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:26:28.906 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:28.905+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:28.905836+0000) 2022-04-04T01:26:28.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:28.981+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:28.989 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:28.987+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:29.907 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:29.905+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:29.906025+0000) 2022-04-04T01:26:29.956 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:29.954+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:30.017 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:30.015+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:30.907 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:30.905+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:30.906222+0000) 2022-04-04T01:26:30.936 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:30.933+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:31.042 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:31.040+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:31.907 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:31.906+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:31.906394+0000) 2022-04-04T01:26:31.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:31.931+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:32.001 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:31.999+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:32.907 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:32.906+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:32.906574+0000) 2022-04-04T01:26:32.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:32.913+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:32.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:32.986+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:33.907 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:33.906+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:33.906797+0000) 2022-04-04T01:26:33.920 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:33.918+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:33.996 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:33.994+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:34.908 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:34.906+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:34.907015+0000) 2022-04-04T01:26:34.931 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:34.929+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:34.971 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:34.969+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:35.908 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:35.906+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:35.907149+0000) 2022-04-04T01:26:35.923 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:35.920+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:36.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:36.014+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:36.876 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:36.874+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:36.908 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:36.907+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:36.907337+0000) 2022-04-04T01:26:37.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:37.017+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:37.900 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:37.897+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:37.908 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:37.906+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:37.907514+0000) 2022-04-04T01:26:38.030 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:38.028+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:38.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:38.895+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:38.908 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:38.907+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:38.907705+0000) 2022-04-04T01:26:38.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:38.987+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:39.909 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:39.907+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:39.907863+0000) 2022-04-04T01:26:39.930 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:39.928+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:39.978 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:39.976+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:40.909 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:40.907+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:40.908007+0000) 2022-04-04T01:26:40.976 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:40.974+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:40.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:40.979+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:41.909 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:41.908+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:41.908204+0000) 2022-04-04T01:26:41.944 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:41.942+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:42.002 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:41.999+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:42.909 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:42.908+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:42.908419+0000) 2022-04-04T01:26:42.946 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:42.944+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:42.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:42.990+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:43.882 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:26:43.880+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:26:43.909 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:43.907+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:43.908642+0000) 2022-04-04T01:26:43.934 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:43.932+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:43.987 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:43.986+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:44.909 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:44.908+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:44.908856+0000) 2022-04-04T01:26:44.928 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:44.926+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:44.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:44.941+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:45.897 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:45.895+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:45.910 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:45.907+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:45.909087+0000) 2022-04-04T01:26:45.913 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:45.912+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:46.883 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:46.882+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:46.905 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:46.903+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:46.910 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:46.908+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:46.909283+0000) 2022-04-04T01:26:47.841 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:47.840+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:47.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:47.898+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:47.907 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:26:47.911 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:26:47.912 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:47.908+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:47.909458+0000) 2022-04-04T01:26:48.844 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:48.842+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:48.902 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:48.900+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:48.910 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:48.908+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:48.909602+0000) 2022-04-04T01:26:49.805 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:49.804+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:49.910 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:49.908+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:49.909798+0000) 2022-04-04T01:26:49.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:49.930+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:50.794 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:50.792+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:50.911 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:50.909+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:50.909989+0000) 2022-04-04T01:26:50.932 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:50.931+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:51.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:51.765+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:51.911 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:51.909+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:51.910188+0000) 2022-04-04T01:26:51.944 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:51.943+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:52.816 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:52.814+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:52.911 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:52.909+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:52.910390+0000) 2022-04-04T01:26:52.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:52.990+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:53.840 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:53.837+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:53.911 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:53.909+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:53.910575+0000) 2022-04-04T01:26:53.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:53.981+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:54.807 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:54.805+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:54.911 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:54.909+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:54.910838+0000) 2022-04-04T01:26:54.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:54.939+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:55.762 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:55.760+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:55.912 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:55.910+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:55.911086+0000) 2022-04-04T01:26:55.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:55.916+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:56.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:56.720+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:56.912 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:56.910+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:56.911302+0000) 2022-04-04T01:26:56.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:56.916+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:57.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:57.722+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:57.912 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:57.910+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:57.911512+0000) 2022-04-04T01:26:57.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:57.922+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:58.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:58.714+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:58.890 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:26:58.888+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:26:58.912 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:58.910+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:58.911701+0000) 2022-04-04T01:26:58.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:58.932+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:26:59.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:26:59.730+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:26:59.913 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:26:59.911+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:26:59.911900+0000) 2022-04-04T01:26:59.925 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:26:59.924+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:00.687 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:00.685+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:00.889 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:00.887+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:00.913 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:00.911+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:00.912094+0000) 2022-04-04T01:27:01.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:01.706+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:01.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:01.837+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:01.913 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:01.911+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:01.912310+0000) 2022-04-04T01:27:02.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:02.680+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:02.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:02.810+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:02.913 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:02.911+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:02.912526+0000) 2022-04-04T01:27:03.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:03.705+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:03.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:03.853+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:03.913 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:03.912+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:03.912701+0000) 2022-04-04T01:27:04.673 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:04.671+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:04.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:04.876+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:04.914 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:04.912+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:04.912887+0000) 2022-04-04T01:27:05.648 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:05.646+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:05.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:05.852+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:05.914 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:05.912+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:05.913073+0000) 2022-04-04T01:27:06.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:06.655+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:06.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:06.824+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:06.914 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:06.912+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:06.913261+0000) 2022-04-04T01:27:07.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:07.616+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:07.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:07.818+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:07.914 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:07.912+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:07.913490+0000) 2022-04-04T01:27:08.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:08.594+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:08.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:08.822+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:08.914 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:08.913+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:08.913675+0000) 2022-04-04T01:27:09.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:09.564+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:09.841 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:09.838+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:09.915 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:09.913+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:09.913899+0000) 2022-04-04T01:27:10.520 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:10.518+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:10.819 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:10.816+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:10.915 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:10.913+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:10.914134+0000) 2022-04-04T01:27:11.518 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:11.516+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:11.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:11.856+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:11.915 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:11.913+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:11.914317+0000) 2022-04-04T01:27:12.546 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:12.543+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:12.856 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:12.853+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:12.915 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:12.914+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:12.914481+0000) 2022-04-04T01:27:13.550 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:13.548+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:13.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:13.840+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:13.906 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:27:13.904+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:27:13.915 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:13.914+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:13.914651+0000) 2022-04-04T01:27:14.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:14.509+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:14.889 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:14.886+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:14.915 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:14.914+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:14.914822+0000) 2022-04-04T01:27:15.549 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:15.547+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:15.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:15.838+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:15.916 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:15.914+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:15.914988+0000) 2022-04-04T01:27:16.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:16.534+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:16.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:16.815+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:16.916 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:16.914+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:16.915146+0000) 2022-04-04T01:27:17.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:17.551+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:17.786 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:17.783+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:17.916 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:17.914+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:17.915336+0000) 2022-04-04T01:27:18.057 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:27:18.061 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:27:18.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:18.533+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:18.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:18.774+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:18.916 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:18.915+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:18.915578+0000) 2022-04-04T01:27:19.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:19.507+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:19.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:19.747+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:19.917 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:19.915+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:19.915829+0000) 2022-04-04T01:27:20.464 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:20.462+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:20.790 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:20.788+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:20.917 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:20.915+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:20.916017+0000) 2022-04-04T01:27:21.478 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:21.475+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:21.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:21.822+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:21.917 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:21.915+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:21.916158+0000) 2022-04-04T01:27:22.484 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:22.481+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:22.817 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:22.815+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:22.917 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:22.916+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:22.916366+0000) 2022-04-04T01:27:23.470 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:23.467+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:23.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:23.799+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:23.917 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:23.916+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:23.916589+0000) 2022-04-04T01:27:24.428 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:24.426+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:24.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:24.797+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:24.917 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:24.916+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:24.916759+0000) 2022-04-04T01:27:25.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:25.408+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:25.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:25.834+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:25.918 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:25.916+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:25.916985+0000) 2022-04-04T01:27:26.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:26.370+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:26.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:26.818+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:26.918 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:26.917+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:26.917208+0000) 2022-04-04T01:27:27.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:27.383+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:27.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:27.777+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:27.918 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:27.917+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:27.917427+0000) 2022-04-04T01:27:28.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:28.364+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:28.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:28.773+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:28.918 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:28.917+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:28.917601+0000) 2022-04-04T01:27:28.922 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:27:28.920+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:27:29.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:29.388+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:29.763 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:29.761+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:29.919 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:29.916+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:29.917851+0000) 2022-04-04T01:27:30.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:30.438+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:30.777 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:30.774+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:30.919 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:30.916+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:30.917985+0000) 2022-04-04T01:27:31.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:31.406+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:31.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:31.777+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:31.919 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:31.917+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:31.918186+0000) 2022-04-04T01:27:32.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:32.391+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:32.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:32.750+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:32.919 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:32.917+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:32.918370+0000) 2022-04-04T01:27:33.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:33.365+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:33.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:33.776+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:33.919 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:33.917+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:33.918532+0000) 2022-04-04T01:27:34.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:34.330+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:34.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:34.746+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:34.919 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:34.917+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:34.918724+0000) 2022-04-04T01:27:35.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:35.355+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:35.730 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:35.728+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:35.920 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:35.917+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:35.918934+0000) 2022-04-04T01:27:36.384 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:36.382+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:36.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:36.742+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:36.920 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:36.918+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:36.919138+0000) 2022-04-04T01:27:37.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:37.395+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:37.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:37.740+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:37.920 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:37.918+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:37.919332+0000) 2022-04-04T01:27:38.369 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:38.367+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:38.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:38.778+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:38.920 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:38.918+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:38.919555+0000) 2022-04-04T01:27:39.375 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:39.373+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:39.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:39.811+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:39.920 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:39.918+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:39.919751+0000) 2022-04-04T01:27:40.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:40.366+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:40.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:40.842+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:40.920 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:40.919+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:40.919952+0000) 2022-04-04T01:27:41.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:41.381+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:41.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:41.880+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:41.921 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:41.919+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:41.920092+0000) 2022-04-04T01:27:42.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:42.367+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:42.921 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:42.919+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:42.920315+0000) 2022-04-04T01:27:42.926 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:42.925+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:43.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:43.375+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:43.921 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:43.919+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:43.920544+0000) 2022-04-04T01:27:43.924 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:27:43.922+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:27:43.941 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:43.939+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:44.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:44.371+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:44.922 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:44.920+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:44.920805+0000) 2022-04-04T01:27:44.946 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:44.944+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:45.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:45.412+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:45.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:45.911+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:45.922 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:45.920+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:45.921027+0000) 2022-04-04T01:27:46.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:46.370+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:46.922 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:46.920+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:46.921250+0000) 2022-04-04T01:27:46.939 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:46.938+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:47.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:47.345+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:47.922 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:47.920+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:47.921474+0000) 2022-04-04T01:27:47.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:47.929+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:48.194 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:27:48.198 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:27:48.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:48.361+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:48.891 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:48.890+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:48.922 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:48.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:48.921764+0000) 2022-04-04T01:27:49.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:49.410+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:49.919 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:49.918+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:49.922 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:49.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:49.921944+0000) 2022-04-04T01:27:50.382 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:50.381+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:50.889 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:50.888+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:50.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:50.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:50.922147+0000) 2022-04-04T01:27:51.356 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:51.354+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:51.892 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:51.891+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:51.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:51.921+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:51.922367+0000) 2022-04-04T01:27:52.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:52.321+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:52.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:52.873+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:52.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:52.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:52.922542+0000) 2022-04-04T01:27:53.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:53.318+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:53.874 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:53.873+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:53.923 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:53.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:53.922752+0000) 2022-04-04T01:27:54.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:54.280+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:54.909 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:54.907+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:54.924 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:54.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:54.923006+0000) 2022-04-04T01:27:55.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:55.326+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:55.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:55.874+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:55.924 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:55.922+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:55.923235+0000) 2022-04-04T01:27:56.358 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:56.356+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:56.828 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:56.826+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:56.924 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:56.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:56.923433+0000) 2022-04-04T01:27:57.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:57.381+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:57.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:57.830+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:57.924 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:57.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:57.923584+0000) 2022-04-04T01:27:58.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:58.381+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:58.845 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:58.843+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:58.924 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:58.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:58.923797+0000) 2022-04-04T01:27:58.940 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:27:58.938+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:27:59.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:27:59.408+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:27:59.798 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:27:59.795+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:27:59.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:27:59.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:27:59.924027+0000) 2022-04-04T01:28:00.405 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:00.403+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:00.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:00.755+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:00.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:00.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:00.924215+0000) 2022-04-04T01:28:01.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:01.361+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:01.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:01.777+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:01.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:01.924+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:01.924394+0000) 2022-04-04T01:28:02.375 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:02.372+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:02.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:02.813+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:02.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:02.923+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:02.924614+0000) 2022-04-04T01:28:03.423 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:03.420+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:03.850 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:03.847+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:03.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:03.924+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:03.924860+0000) 2022-04-04T01:28:04.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:04.403+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:04.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:04.851+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:04.925 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:04.924+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:04.925053+0000) 2022-04-04T01:28:05.451 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:05.448+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:05.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:05.807+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:05.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:05.924+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:05.925234+0000) 2022-04-04T01:28:06.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:06.425+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:06.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:06.758+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:06.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:06.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:06.925430+0000) 2022-04-04T01:28:07.454 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:07.452+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:07.752 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:07.750+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:07.926 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:07.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:07.925598+0000) 2022-04-04T01:28:08.478 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:08.476+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:08.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:08.757+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:08.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:08.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:08.925813+0000) 2022-04-04T01:28:09.431 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:09.429+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:09.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:09.792+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:09.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:09.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:09.926017+0000) 2022-04-04T01:28:10.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:10.444+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:10.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:10.820+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:10.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:10.926+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:10.926240+0000) 2022-04-04T01:28:11.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:11.492+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:11.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:11.805+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:11.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:11.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:11.926479+0000) 2022-04-04T01:28:12.472 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:12.469+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:12.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:12.809+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:12.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:12.926+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:12.926655+0000) 2022-04-04T01:28:13.440 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:13.437+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:13.859 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:13.857+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:13.927 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:13.926+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:13.926842+0000) 2022-04-04T01:28:13.955 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:28:13.954+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:28:14.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:14.443+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:14.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:14.868+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:14.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:14.925+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:14.927008+0000) 2022-04-04T01:28:15.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:15.396+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:15.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:15.898+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:15.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:15.926+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:15.927204+0000) 2022-04-04T01:28:16.376 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:16.374+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:16.883 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:16.880+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:16.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:16.926+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:16.927370+0000) 2022-04-04T01:28:17.378 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:17.376+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:17.867 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:17.864+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:17.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:17.926+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:17.927526+0000) 2022-04-04T01:28:18.332 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:28:18.336 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:28:18.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:18.368+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:18.826 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:18.823+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:18.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:18.926+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:18.927723+0000) 2022-04-04T01:28:19.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:19.384+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:19.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:19.786+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:19.928 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:19.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:19.927982+0000) 2022-04-04T01:28:20.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:20.388+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:20.765 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:20.764+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:20.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:20.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:20.928198+0000) 2022-04-04T01:28:21.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:21.377+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:21.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:21.776+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:21.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:21.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:21.928410+0000) 2022-04-04T01:28:22.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:22.352+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:22.768 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:22.766+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:22.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:22.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:22.928610+0000) 2022-04-04T01:28:23.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:23.348+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:23.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:23.798+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:23.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:23.927+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:23.928804+0000) 2022-04-04T01:28:24.335 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:24.333+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:24.763 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:24.761+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:24.929 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:24.928+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:24.928968+0000) 2022-04-04T01:28:25.316 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:25.314+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:25.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:25.724+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:25.930 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:25.928+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:25.929144+0000) 2022-04-04T01:28:26.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:26.317+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:26.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:26.746+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:26.930 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:26.928+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:26.929363+0000) 2022-04-04T01:28:27.337 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:27.335+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:27.741 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:27.739+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:27.930 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:27.928+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:27.929604+0000) 2022-04-04T01:28:28.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:28.349+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:28.786 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:28.784+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:28.930 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:28.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:28.929829+0000) 2022-04-04T01:28:28.971 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:28:28.969+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:28:29.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:29.302+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:29.769 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:29.767+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:29.931 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:29.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:29.929999+0000) 2022-04-04T01:28:30.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:30.307+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:30.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:30.798+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:30.931 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:30.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:30.930136+0000) 2022-04-04T01:28:31.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:31.341+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:31.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:31.844+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:31.931 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:31.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:31.930333+0000) 2022-04-04T01:28:32.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:32.347+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:32.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:32.811+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:32.931 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:32.929+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:32.930551+0000) 2022-04-04T01:28:33.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:33.300+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:33.786 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:33.785+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:33.931 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:33.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:33.930751+0000) 2022-04-04T01:28:34.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:34.260+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:34.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:34.773+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:34.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:34.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:34.931007+0000) 2022-04-04T01:28:35.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:35.249+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:35.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:35.774+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:35.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:35.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:35.931226+0000) 2022-04-04T01:28:36.284 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:36.283+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:36.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:36.743+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:36.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:36.930+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:36.931446+0000) 2022-04-04T01:28:37.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:37.274+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:37.747 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:37.746+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:37.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:37.931+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:37.931623+0000) 2022-04-04T01:28:38.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:38.301+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:38.769 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:38.768+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:38.932 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:38.931+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:38.931862+0000) 2022-04-04T01:28:39.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:39.320+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:39.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:39.759+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:39.933 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:39.931+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:39.932083+0000) 2022-04-04T01:28:40.364 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:40.363+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:40.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:40.778+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:40.933 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:40.931+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:40.932306+0000) 2022-04-04T01:28:41.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:41.397+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:41.768 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:41.767+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:41.933 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:41.932+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:41.932507+0000) 2022-04-04T01:28:42.375 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:42.373+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:42.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:42.758+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:42.933 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:42.932+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:42.932724+0000) 2022-04-04T01:28:43.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:43.365+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:43.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:43.747+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:43.934 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:43.932+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:43.932883+0000) 2022-04-04T01:28:43.987 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:28:43.985+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:28:44.400 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:44.399+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:44.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:44.794+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:44.934 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:44.932+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:44.933072+0000) 2022-04-04T01:28:45.418 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:45.417+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:45.804 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:45.803+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:45.934 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:45.932+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:45.933320+0000) 2022-04-04T01:28:46.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:46.455+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:46.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:46.793+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:46.934 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:46.933+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:46.933493+0000) 2022-04-04T01:28:47.464 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:47.463+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:47.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:47.787+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:47.934 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:47.933+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:47.933685+0000) 2022-04-04T01:28:48.479 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:28:48.483 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:28:48.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:48.511+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:48.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:48.807+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:48.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:48.933+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:48.933939+0000) 2022-04-04T01:28:49.522 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:49.521+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:49.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:49.823+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:49.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:49.933+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:49.934111+0000) 2022-04-04T01:28:50.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:50.495+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:50.798 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:50.795+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:50.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:50.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:50.934282+0000) 2022-04-04T01:28:51.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:51.512+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:51.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:51.757+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:51.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:51.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:51.934475+0000) 2022-04-04T01:28:52.489 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:52.487+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:52.716 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:52.714+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:52.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:52.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:52.934661+0000) 2022-04-04T01:28:53.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:53.518+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:53.754 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:53.751+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:53.935 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:53.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:53.934865+0000) 2022-04-04T01:28:54.484 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:54.482+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:54.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:54.779+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:54.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:54.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:54.934988+0000) 2022-04-04T01:28:55.442 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:55.440+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:55.801 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:55.798+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:55.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:55.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:55.935145+0000) 2022-04-04T01:28:56.463 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:56.461+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:56.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:56.845+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:56.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:56.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:56.935352+0000) 2022-04-04T01:28:57.505 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:57.503+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:57.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:57.886+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:57.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:57.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:57.935520+0000) 2022-04-04T01:28:58.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:58.544+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:58.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:58.874+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:58.936 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:58.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:58.935693+0000) 2022-04-04T01:28:59.003 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:28:59.000+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:28:59.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:28:59.575+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:28:59.899 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:28:59.897+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:28:59.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:28:59.934+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:28:59.935929+0000) 2022-04-04T01:29:00.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:00.610+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:00.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:00.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:00.936153+0000) 2022-04-04T01:29:00.939 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:00.937+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:01.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:01.640+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:01.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:01.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:01.936359+0000) 2022-04-04T01:29:01.968 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:01.966+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:02.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:02.654+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:02.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:02.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:02.936578+0000) 2022-04-04T01:29:02.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:02.983+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:03.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:03.651+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:03.937 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:03.935+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:03.936816+0000) 2022-04-04T01:29:03.972 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:03.971+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:04.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:04.641+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:04.938 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:04.936+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:04.937033+0000) 2022-04-04T01:29:04.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:04.964+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:05.648 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:05.646+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:05.938 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:05.936+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:05.937211+0000) 2022-04-04T01:29:05.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:05.986+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:06.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:06.695+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:06.938 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:06.936+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:06.937421+0000) 2022-04-04T01:29:07.002 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:07.000+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:07.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:07.719+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:07.938 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:07.936+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:07.937660+0000) 2022-04-04T01:29:07.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:07.996+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:08.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:08.686+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:08.939 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:08.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:08.937863+0000) 2022-04-04T01:29:08.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:08.963+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:09.708 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:09.706+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:09.939 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:09.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:09.938070+0000) 2022-04-04T01:29:09.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:09.953+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:10.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:10.741+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:10.939 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:10.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:10.938243+0000) 2022-04-04T01:29:10.962 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:10.960+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:11.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:11.697+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:11.939 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:11.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:11.938456+0000) 2022-04-04T01:29:11.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:11.962+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:12.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:12.721+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:12.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:12.937+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:12.938697+0000) 2022-04-04T01:29:12.976 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:12.974+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:13.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:13.741+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:13.940 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:13.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:13.938900+0000) 2022-04-04T01:29:13.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:13.951+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:14.019 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:29:14.017+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:29:14.735 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:14.733+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:14.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:14.931+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:14.940 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:14.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:14.939075+0000) 2022-04-04T01:29:15.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:15.717+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:15.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:15.884+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:15.940 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:15.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:15.939236+0000) 2022-04-04T01:29:16.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:16.684+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:16.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:16.885+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:16.940 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:16.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:16.939383+0000) 2022-04-04T01:29:17.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:17.688+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:17.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:17.845+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:17.940 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:17.938+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:17.939538+0000) 2022-04-04T01:29:18.605 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:29:18.609 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:29:18.674 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:18.672+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:18.837 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:18.834+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:18.940 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:18.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:18.939763+0000) 2022-04-04T01:29:19.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:19.650+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:19.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:19.821+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:19.941 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:19.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:19.939994+0000) 2022-04-04T01:29:20.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:20.675+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:20.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:20.828+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:20.941 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:20.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:20.940205+0000) 2022-04-04T01:29:21.687 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:21.685+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:21.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:21.820+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:21.941 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:21.939+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:21.940361+0000) 2022-04-04T01:29:22.678 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:22.676+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:22.787 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:22.786+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:22.941 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:22.940+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:22.940504+0000) 2022-04-04T01:29:23.724 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:23.722+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:23.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:23.787+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:23.941 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:23.940+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:23.940697+0000) 2022-04-04T01:29:24.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:24.737+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:24.753 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:24.750+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:24.942 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:24.940+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:24.940905+0000) 2022-04-04T01:29:25.710 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:25.708+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:25.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:25.798+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:25.942 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:25.940+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:25.941114+0000) 2022-04-04T01:29:26.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:26.690+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:26.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:26.811+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:26.942 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:26.940+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:26.941338+0000) 2022-04-04T01:29:27.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:27.645+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:27.768 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:27.766+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:27.942 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:27.941+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:27.941503+0000) 2022-04-04T01:29:28.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:28.653+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:28.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:28.724+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:28.942 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:28.941+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:28.941695+0000) 2022-04-04T01:29:29.035 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:29:29.033+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:29:29.643 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:29.642+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:29.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:29.676+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:29.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:29.941+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:29.941901+0000) 2022-04-04T01:29:30.654 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:30.653+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:30.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:30.709+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:30.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:30.941+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:30.942053+0000) 2022-04-04T01:29:31.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:31.655+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:31.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:31.695+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:31.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:31.941+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:31.942271+0000) 2022-04-04T01:29:32.672 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:32.670+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:32.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:32.687+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:32.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:32.942+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:32.942516+0000) 2022-04-04T01:29:33.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:33.645+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:33.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:33.661+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:33.943 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:33.942+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:33.942669+0000) 2022-04-04T01:29:34.625 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:34.624+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:34.652 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:34.650+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:34.944 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:34.942+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:34.942856+0000) 2022-04-04T01:29:35.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:35.633+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:35.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:35.694+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:35.944 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:35.942+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:35.943093+0000) 2022-04-04T01:29:36.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:36.647+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:36.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:36.690+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:36.944 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:36.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:36.943262+0000) 2022-04-04T01:29:37.623 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:37.621+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:37.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:37.734+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:37.944 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:37.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:37.943463+0000) 2022-04-04T01:29:38.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:38.586+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:38.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:38.736+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:38.944 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:38.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:38.943675+0000) 2022-04-04T01:29:39.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:39.622+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:39.709 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:39.707+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:39.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:39.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:39.943899+0000) 2022-04-04T01:29:40.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:40.644+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:40.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:40.686+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:40.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:40.944+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:40.944133+0000) 2022-04-04T01:29:41.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:41.648+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:41.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:41.693+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:41.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:41.944+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:41.944354+0000) 2022-04-04T01:29:42.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:42.635+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:42.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:42.713+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:42.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:42.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:42.944580+0000) 2022-04-04T01:29:43.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:43.676+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:43.696 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:43.694+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:43.945 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:43.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:43.944826+0000) 2022-04-04T01:29:44.050 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:29:44.048+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:29:44.653 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:44.650+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:44.720 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:44.717+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:44.946 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:44.943+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:44.944985+0000) 2022-04-04T01:29:45.660 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:45.657+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:45.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:45.727+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:45.946 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:45.944+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:45.945240+0000) 2022-04-04T01:29:46.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:46.642+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:46.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:46.715+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:46.946 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:46.944+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:46.945459+0000) 2022-04-04T01:29:47.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:47.679+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:47.683 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:47.681+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:47.946 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:47.944+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:47.945637+0000) 2022-04-04T01:29:48.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:48.680+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:48.720 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:48.718+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:48.753 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:29:48.757 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:29:48.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:48.944+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:48.945850+0000) 2022-04-04T01:29:49.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:49.691+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:49.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:49.740+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:49.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:49.945+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:49.946060+0000) 2022-04-04T01:29:50.734 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:50.732+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:50.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:50.746+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:50.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:50.945+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:50.946237+0000) 2022-04-04T01:29:51.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:51.773+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:51.788 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:51.786+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:51.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:51.945+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:51.946437+0000) 2022-04-04T01:29:52.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:52.757+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:52.819 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:52.817+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:52.947 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:52.945+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:52.946665+0000) 2022-04-04T01:29:53.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:53.718+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:53.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:53.851+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:53.948 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:53.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:53.946848+0000) 2022-04-04T01:29:54.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:54.695+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:54.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:54.851+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:54.948 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:54.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:54.947055+0000) 2022-04-04T01:29:55.655 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:55.653+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:55.841 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:55.839+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:55.948 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:55.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:55.947277+0000) 2022-04-04T01:29:56.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:56.628+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:56.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:56.794+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:56.948 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:56.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:56.947505+0000) 2022-04-04T01:29:57.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:57.670+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:57.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:57.805+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:57.948 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:57.946+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:57.947742+0000) 2022-04-04T01:29:58.625 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:58.622+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:58.785 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:58.782+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:58.949 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:58.947+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:58.948007+0000) 2022-04-04T01:29:59.066 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:29:59.065+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:29:59.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:29:59.589+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:29:59.740 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:29:59.738+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:29:59.949 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:29:59.947+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:29:59.948222+0000) 2022-04-04T01:30:00.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:00.579+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:00.763 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:00.761+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:00.949 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:00.947+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:00.948414+0000) 2022-04-04T01:30:01.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:01.626+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:01.797 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:01.795+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:01.949 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:01.947+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:01.948602+0000) 2022-04-04T01:30:02.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:02.581+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:02.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:02.791+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:02.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:02.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:02.948857+0000) 2022-04-04T01:30:03.600 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:03.598+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:03.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:03.770+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:03.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:03.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:03.949046+0000) 2022-04-04T01:30:04.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:04.634+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:04.749 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:04.746+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:04.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:04.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:04.949243+0000) 2022-04-04T01:30:05.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:05.612+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:05.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:05.725+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:05.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:05.948+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:05.949434+0000) 2022-04-04T01:30:06.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:06.652+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:06.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:06.759+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:06.950 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:06.949+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:06.949640+0000) 2022-04-04T01:30:07.700 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:07.698+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:07.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:07.777+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:07.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:07.949+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:07.949857+0000) 2022-04-04T01:30:08.737 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:08.735+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:08.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:08.812+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:08.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:08.949+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:08.950072+0000) 2022-04-04T01:30:09.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:09.700+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:09.797 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:09.795+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:09.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:09.949+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:09.950300+0000) 2022-04-04T01:30:10.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:10.690+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:10.769 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:10.767+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:10.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:10.950+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:10.950536+0000) 2022-04-04T01:30:11.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:11.716+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:11.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:11.773+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:11.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:11.949+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:11.950689+0000) 2022-04-04T01:30:12.735 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:12.733+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:12.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:12.753+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:12.951 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:12.950+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:12.950892+0000) 2022-04-04T01:30:13.710 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:13.707+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:13.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:13.768+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:13.952 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:13.950+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:13.951119+0000) 2022-04-04T01:30:14.069 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:30:14.067+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:30:14.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:14.695+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:14.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:14.780+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:14.952 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:14.950+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:14.951344+0000) 2022-04-04T01:30:15.661 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:15.659+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:15.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:15.819+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:15.952 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:15.951+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:15.951539+0000) 2022-04-04T01:30:16.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:16.644+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:16.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:16.794+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:16.952 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:16.951+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:16.951758+0000) 2022-04-04T01:30:17.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:17.625+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:17.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:17.753+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:17.953 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:17.951+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:17.952004+0000) 2022-04-04T01:30:18.619 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:18.617+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:18.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:18.723+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:18.877 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:30:18.882 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:30:18.953 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:18.951+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:18.952217+0000) 2022-04-04T01:30:19.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:19.651+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:19.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:19.750+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:19.953 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:19.952+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:19.952382+0000) 2022-04-04T01:30:20.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:20.680+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:20.791 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:20.789+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:20.953 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:20.952+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:20.952592+0000) 2022-04-04T01:30:21.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:21.726+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:21.834 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:21.833+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:21.953 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:21.952+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:21.952853+0000) 2022-04-04T01:30:22.769 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:22.768+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:22.856 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:22.854+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:22.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:22.952+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:22.953055+0000) 2022-04-04T01:30:23.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:23.765+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:23.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:23.813+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:23.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:23.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:23.953250+0000) 2022-04-04T01:30:24.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:24.793+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:24.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:24.809+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:24.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:24.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:24.953472+0000) 2022-04-04T01:30:25.794 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:25.792+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:25.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:25.821+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:25.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:25.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:25.953686+0000) 2022-04-04T01:30:26.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:26.756+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:26.795 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:26.794+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:26.954 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:26.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:26.953880+0000) 2022-04-04T01:30:27.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:27.713+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:27.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:27.837+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:27.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:27.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:27.954130+0000) 2022-04-04T01:30:28.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:28.719+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:28.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:28.828+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:28.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:28.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:28.954311+0000) 2022-04-04T01:30:29.085 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:30:29.082+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:30:29.704 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:29.703+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:29.828 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:29.827+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:29.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:29.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:29.954452+0000) 2022-04-04T01:30:30.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:30.739+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:30.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:30.830+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:30.955 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:30.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:30.954653+0000) 2022-04-04T01:30:31.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:31.701+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:31.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:31.877+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:31.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:31.953+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:31.954894+0000) 2022-04-04T01:30:32.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:32.685+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:32.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:32.921+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:32.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:32.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:32.955142+0000) 2022-04-04T01:30:33.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:33.712+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:33.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:33.916+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:33.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:33.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:33.955330+0000) 2022-04-04T01:30:34.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:34.680+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:34.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:34.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:34.955473+0000) 2022-04-04T01:30:34.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:34.965+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:35.705 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:35.704+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:35.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:35.926+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:35.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:35.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:35.955643+0000) 2022-04-04T01:30:36.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:36.739+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:36.944 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:36.943+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:36.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:36.954+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:36.955847+0000) 2022-04-04T01:30:37.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:37.756+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:37.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:37.902+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:37.956 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:37.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:37.955984+0000) 2022-04-04T01:30:38.738 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:38.737+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:38.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:38.874+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:38.957 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:38.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:38.956174+0000) 2022-04-04T01:30:39.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:39.720+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:39.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:39.825+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:39.957 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:39.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:39.956395+0000) 2022-04-04T01:30:40.717 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:40.715+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:40.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:40.874+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:40.957 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:40.955+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:40.956604+0000) 2022-04-04T01:30:41.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:41.732+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:41.839 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:41.837+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:41.957 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:41.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:41.956766+0000) 2022-04-04T01:30:42.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:42.775+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:42.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:42.797+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:42.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:42.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:42.956974+0000) 2022-04-04T01:30:43.781 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:43.778+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:43.820 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:43.818+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:43.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:43.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:43.957120+0000) 2022-04-04T01:30:44.101 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:30:44.099+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:30:44.765 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:44.762+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:44.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:44.807+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:44.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:44.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:44.957295+0000) 2022-04-04T01:30:45.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:45.717+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:45.818 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:45.816+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:45.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:45.956+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:45.957551+0000) 2022-04-04T01:30:46.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:46.672+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:46.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:46.787+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:46.958 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:46.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:46.957809+0000) 2022-04-04T01:30:47.670 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:47.667+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:47.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:47.782+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:47.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:47.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:47.958063+0000) 2022-04-04T01:30:48.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:48.649+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:48.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:48.792+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:48.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:48.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:48.958192+0000) 2022-04-04T01:30:49.015 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:30:49.019 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:30:49.609 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:49.607+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:49.836 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:49.834+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:49.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:49.957+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:49.958419+0000) 2022-04-04T01:30:50.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:50.628+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:50.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:50.845+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:50.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:50.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:50.958612+0000) 2022-04-04T01:30:51.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:51.678+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:51.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:51.858+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:51.959 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:51.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:51.958834+0000) 2022-04-04T01:30:52.659 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:52.657+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:52.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:52.866+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:52.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:52.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:52.959024+0000) 2022-04-04T01:30:53.689 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:53.686+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:53.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:53.884+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:53.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:53.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:53.959185+0000) 2022-04-04T01:30:54.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:54.642+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:54.861 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:54.859+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:54.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:54.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:54.959355+0000) 2022-04-04T01:30:55.613 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:55.611+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:55.899 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:55.897+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:55.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:55.958+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:55.959593+0000) 2022-04-04T01:30:56.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:56.636+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:56.871 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:56.869+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:56.960 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:56.959+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:56.959805+0000) 2022-04-04T01:30:57.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:57.673+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:57.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:57.822+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:57.961 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:57.959+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:57.959986+0000) 2022-04-04T01:30:58.673 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:58.671+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:58.795 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:58.793+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:58.961 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:58.959+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:58.960180+0000) 2022-04-04T01:30:59.102 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:30:59.100+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:30:59.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:30:59.720+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:30:59.814 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:30:59.812+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:30:59.961 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:30:59.960+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:30:59.960386+0000) 2022-04-04T01:31:00.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:00.747+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:00.787 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:00.785+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:00.961 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:00.960+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:00.960567+0000) 2022-04-04T01:31:01.751 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:01.749+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:01.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:01.756+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:01.961 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:01.960+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:01.960746+0000) 2022-04-04T01:31:02.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:02.719+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:02.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:02.746+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:02.962 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:02.960+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:02.960984+0000) 2022-04-04T01:31:03.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:03.710+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:03.766 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:03.765+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:03.962 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:03.960+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:03.961202+0000) 2022-04-04T01:31:04.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:04.720+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:04.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:04.730+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:04.962 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:04.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:04.961393+0000) 2022-04-04T01:31:05.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:05.712+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:05.718 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:05.716+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:05.962 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:05.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:05.961556+0000) 2022-04-04T01:31:06.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:06.701+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:06.747 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:06.745+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:06.962 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:06.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:06.961720+0000) 2022-04-04T01:31:07.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:07.670+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:07.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:07.737+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:07.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:07.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:07.961974+0000) 2022-04-04T01:31:08.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:08.675+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:08.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:08.713+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:08.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:08.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:08.962130+0000) 2022-04-04T01:31:09.674 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:09.672+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:09.711 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:09.709+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:09.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:09.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:09.962275+0000) 2022-04-04T01:31:10.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:10.666+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:10.676 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:10.674+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:10.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:10.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:10.962456+0000) 2022-04-04T01:31:11.683 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:11.681+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:11.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:11.695+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:11.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:11.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:11.962683+0000) 2022-04-04T01:31:12.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:12.655+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:12.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:12.697+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:12.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:12.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:12.962880+0000) 2022-04-04T01:31:13.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:13.634+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:13.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:13.693+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:13.963 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:13.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:13.962989+0000) 2022-04-04T01:31:14.118 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:31:14.115+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:31:14.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:14.592+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:14.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:14.724+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:14.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:14.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:14.963118+0000) 2022-04-04T01:31:15.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:15.637+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:15.721 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:15.719+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:15.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:15.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:15.963317+0000) 2022-04-04T01:31:16.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:16.656+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:16.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:16.688+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:16.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:16.961+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:16.963562+0000) 2022-04-04T01:31:17.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:17.651+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:17.707 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:17.706+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:17.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:17.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:17.963801+0000) 2022-04-04T01:31:18.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:18.651+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:18.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:18.691+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:18.965 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:18.963+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:18.963996+0000) 2022-04-04T01:31:19.171 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:31:19.174 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:31:19.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:19.644+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:19.708 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:19.707+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:19.965 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:19.962+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:19.964173+0000) 2022-04-04T01:31:20.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:20.635+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:20.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:20.738+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:20.965 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:20.963+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:20.964368+0000) 2022-04-04T01:31:21.625 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:21.624+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:21.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:21.744+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:21.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:21.963+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:21.964566+0000) 2022-04-04T01:31:22.622 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:22.621+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:22.787 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:22.785+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:22.965 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:22.964+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:22.964822+0000) 2022-04-04T01:31:23.642 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:23.641+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:23.825 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:23.824+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:23.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:23.964+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:23.965004+0000) 2022-04-04T01:31:24.657 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:24.655+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:24.787 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:24.786+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:24.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:24.964+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:24.965198+0000) 2022-04-04T01:31:25.705 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:25.704+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:25.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:25.757+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:25.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:25.964+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:25.965422+0000) 2022-04-04T01:31:26.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:26.698+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:26.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:26.736+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:26.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:26.964+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:26.965672+0000) 2022-04-04T01:31:27.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:27.694+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:27.765 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:27.764+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:27.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:27.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:27.965836+0000) 2022-04-04T01:31:28.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:28.727+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:28.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:28.771+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:28.967 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:28.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:28.966029+0000) 2022-04-04T01:31:29.134 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:31:29.132+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:31:29.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:29.746+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:29.775 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:29.774+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:29.967 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:29.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:29.966250+0000) 2022-04-04T01:31:30.775 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:30.774+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:30.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:30.778+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:30.967 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:30.965+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:30.966491+0000) 2022-04-04T01:31:31.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:31.740+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:31.774 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:31.773+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:31.967 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:31.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:31.966660+0000) 2022-04-04T01:31:32.704 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:32.703+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:32.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:32.756+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:32.967 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:32.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:32.966872+0000) 2022-04-04T01:31:33.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:33.665+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:33.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:33.805+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:33.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:33.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:33.967092+0000) 2022-04-04T01:31:34.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:34.654+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:34.826 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:34.824+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:34.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:34.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:34.967273+0000) 2022-04-04T01:31:35.636 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:35.634+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:35.786 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:35.783+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:35.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:35.966+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:35.967468+0000) 2022-04-04T01:31:36.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:36.642+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:36.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:36.753+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:36.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:36.967+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:36.967657+0000) 2022-04-04T01:31:37.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:37.592+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:37.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:37.778+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:37.968 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:37.967+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:37.967856+0000) 2022-04-04T01:31:38.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:38.578+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:38.825 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:38.822+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:38.969 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:38.967+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:38.968027+0000) 2022-04-04T01:31:39.615 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:39.612+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:39.852 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:39.850+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:39.969 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:39.967+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:39.968217+0000) 2022-04-04T01:31:40.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:40.610+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:40.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:40.826+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:40.969 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:40.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:40.968442+0000) 2022-04-04T01:31:41.637 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:41.635+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:41.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:41.804+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:41.969 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:41.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:41.968662+0000) 2022-04-04T01:31:42.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:42.654+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:42.841 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:42.838+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:42.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:42.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:42.968889+0000) 2022-04-04T01:31:43.615 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:43.613+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:43.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:43.813+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:43.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:43.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:43.969108+0000) 2022-04-04T01:31:44.137 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:31:44.135+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:31:44.610 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:44.607+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:44.849 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:44.847+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:44.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:44.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:44.969320+0000) 2022-04-04T01:31:45.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:45.653+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:45.809 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:45.807+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:45.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:45.968+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:45.969563+0000) 2022-04-04T01:31:46.697 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:46.695+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:46.817 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:46.814+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:46.970 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:46.969+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:46.969805+0000) 2022-04-04T01:31:47.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:47.743+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:47.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:47.829+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:47.971 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:47.969+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:47.970061+0000) 2022-04-04T01:31:48.782 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:48.780+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:48.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:48.811+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:48.971 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:48.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:48.970279+0000) 2022-04-04T01:31:49.304 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:31:49.308 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:31:49.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:49.743+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:49.801 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:49.799+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:49.971 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:49.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:49.970530+0000) 2022-04-04T01:31:50.762 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:50.759+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:50.784 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:50.782+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:50.971 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:50.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:50.970808+0000) 2022-04-04T01:31:51.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:51.756+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:51.768 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:51.766+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:51.971 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:51.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:51.971010+0000) 2022-04-04T01:31:52.746 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:52.744+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:52.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:52.789+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:52.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:52.971+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:52.971183+0000) 2022-04-04T01:31:53.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:53.748+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:53.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:53.800+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:53.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:53.971+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:53.971374+0000) 2022-04-04T01:31:54.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:54.769+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:54.838 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:54.836+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:54.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:54.971+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:54.971569+0000) 2022-04-04T01:31:55.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:55.726+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:55.871 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:55.869+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:55.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:55.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:55.971764+0000) 2022-04-04T01:31:56.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:56.715+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:56.871 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:56.869+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:56.972 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:56.970+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:56.972001+0000) 2022-04-04T01:31:57.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:57.741+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:57.910 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:57.907+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:57.973 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:57.971+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:57.972230+0000) 2022-04-04T01:31:58.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:58.700+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:58.880 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:58.878+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:58.973 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:58.971+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:58.972442+0000) 2022-04-04T01:31:59.153 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:31:59.150+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:31:59.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:31:59.658+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:31:59.866 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:31:59.865+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:31:59.973 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:31:59.971+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:31:59.972677+0000) 2022-04-04T01:32:00.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:00.686+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:00.836 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:00.834+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:00.973 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:00.971+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:00.972891+0000) 2022-04-04T01:32:01.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:01.662+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:01.821 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:01.819+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:01.974 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:01.972+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:01.973081+0000) 2022-04-04T01:32:02.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:02.627+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:02.844 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:02.842+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:02.974 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:02.972+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:02.973274+0000) 2022-04-04T01:32:03.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:03.646+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:03.847 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:03.845+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:03.974 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:03.972+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:03.973492+0000) 2022-04-04T01:32:04.602 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:04.600+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:04.808 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:04.806+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:04.974 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:04.972+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:04.973689+0000) 2022-04-04T01:32:05.598 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:05.596+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:05.853 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:05.851+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:05.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:05.973+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:05.973851+0000) 2022-04-04T01:32:06.553 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:06.550+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:06.885 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:06.883+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:06.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:06.973+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:06.974052+0000) 2022-04-04T01:32:07.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:07.592+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:07.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:07.870+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:07.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:07.973+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:07.974257+0000) 2022-04-04T01:32:08.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:08.641+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:08.873 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:08.871+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:08.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:08.973+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:08.974489+0000) 2022-04-04T01:32:09.686 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:09.684+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:09.922 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:09.920+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:09.975 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:09.973+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:09.974707+0000) 2022-04-04T01:32:10.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:10.665+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:10.875 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:10.873+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:10.976 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:10.974+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:10.974938+0000) 2022-04-04T01:32:11.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:11.662+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:11.845 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:11.843+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:11.976 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:11.974+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:11.975139+0000) 2022-04-04T01:32:12.666 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:12.664+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:12.807 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:12.805+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:12.976 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:12.974+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:12.975322+0000) 2022-04-04T01:32:13.631 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:13.629+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:13.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:13.756+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:13.976 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:13.974+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:13.975521+0000) 2022-04-04T01:32:14.169 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:32:14.167+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:32:14.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:14.605+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:14.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:14.708+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:14.976 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:14.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:14.975741+0000) 2022-04-04T01:32:15.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:15.646+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:15.704 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:15.703+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:15.976 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:15.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:15.975928+0000) 2022-04-04T01:32:16.681 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:16.680+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:16.744 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:16.743+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:16.977 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:16.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:16.976116+0000) 2022-04-04T01:32:17.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:17.664+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:17.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:17.738+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:17.977 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:17.975+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:17.976340+0000) 2022-04-04T01:32:18.709 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:18.707+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:18.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:18.729+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:18.977 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:18.976+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:18.976554+0000) 2022-04-04T01:32:19.458 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:32:19.462 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:32:19.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:19.678+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:19.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:19.757+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:19.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:19.976+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:19.976731+0000) 2022-04-04T01:32:20.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:20.656+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:20.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:20.709+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:20.978 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:20.976+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:20.976905+0000) 2022-04-04T01:32:21.706 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:21.704+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:21.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:21.727+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:21.978 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:21.976+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:21.977119+0000) 2022-04-04T01:32:22.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:22.691+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:22.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:22.737+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:22.978 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:22.976+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:22.977312+0000) 2022-04-04T01:32:23.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:23.659+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:23.740 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:23.739+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:23.978 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:23.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:23.977503+0000) 2022-04-04T01:32:24.666 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:24.665+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:24.719 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:24.717+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:24.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:24.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:24.977716+0000) 2022-04-04T01:32:25.710 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:25.709+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:25.763 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:25.762+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:32:25.978 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:25.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:25.977872+0000) 2022-04-04T01:32:26.694 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:26.693+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:26.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:26.808+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:26.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:26.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:26.978059+0000) 2022-04-04T01:32:27.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:27.667+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:27.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:27.788+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:27.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:27.977+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:27.978276+0000) 2022-04-04T01:32:28.691 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:28.690+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:28.834 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:28.832+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:28.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:28.978+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:28.978491+0000) 2022-04-04T01:32:29.175 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:32:29.174+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:32:29.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:29.685+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:29.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:29.845+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:29.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:29.978+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:29.978678+0000) 2022-04-04T01:32:30.658 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:30.656+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:30.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:30.880+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:30.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:30.978+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:30.978815+0000) 2022-04-04T01:32:31.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:31.652+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:31.910 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:31.907+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:31.979 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:31.978+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:31.979007+0000) 2022-04-04T01:32:32.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:32.683+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:32.938 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:32.936+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:32.980 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:32.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:32.979205+0000) 2022-04-04T01:32:33.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:33.687+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:33.980 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:33.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:33.979381+0000) 2022-04-04T01:32:33.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:33.979+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:34.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:34.673+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:34.946 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:34.944+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:34.980 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:34.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:34.979541+0000) 2022-04-04T01:32:35.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:35.690+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:35.955 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:35.952+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:35.980 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:35.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:35.979705+0000) 2022-04-04T01:32:36.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:36.686+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:36.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:36.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:36.979873+0000) 2022-04-04T01:32:36.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:36.980+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:37.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:37.699+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:37.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:37.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:37.980075+0000) 2022-04-04T01:32:37.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:37.982+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:38.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:38.679+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:38.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:38.980+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:38.980292+0000) 2022-04-04T01:32:39.000 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:38.998+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:39.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:39.638+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:39.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:39.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:39.980477+0000) 2022-04-04T01:32:40.032 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:40.030+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:40.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:40.590+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:40.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:40.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:40.980677+0000) 2022-04-04T01:32:40.987 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:40.985+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:41.610 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:41.607+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:41.981 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:41.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:41.980839+0000) 2022-04-04T01:32:42.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:42.005+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:42.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:42.573+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:42.982 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:42.980+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:42.981043+0000) 2022-04-04T01:32:42.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:42.986+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:43.544 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:43.541+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:43.982 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:43.980+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:43.981267+0000) 2022-04-04T01:32:43.991 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:43.989+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:44.192 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:32:44.189+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:32:44.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:44.566+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:44.982 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:44.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:44.981473+0000) 2022-04-04T01:32:45.029 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:45.027+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:45.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:45.572+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:45.982 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:45.979+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:45.981690+0000) 2022-04-04T01:32:46.000 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:45.998+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:46.602 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:46.600+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:46.982 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:46.980+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:46.981831+0000) 2022-04-04T01:32:47.009 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:47.007+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:47.633 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:47.631+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:47.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:47.981+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:47.982016+0000) 2022-04-04T01:32:47.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:47.995+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:48.659 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:48.657+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:48.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:48.981+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:48.982212+0000) 2022-04-04T01:32:49.014 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:49.012+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:49.639 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:32:49.643 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:32:49.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:49.695+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:49.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:49.981+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:49.982408+0000) 2022-04-04T01:32:50.021 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:50.020+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:50.687 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:50.684+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:50.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:50.981+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:50.982599+0000) 2022-04-04T01:32:51.018 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:51.016+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:51.684 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:51.682+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:51.983 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:51.982+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:51.982840+0000) 2022-04-04T01:32:52.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:52.059+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:52.651 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:52.649+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:52.984 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:52.982+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:52.983060+0000) 2022-04-04T01:32:53.053 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:53.051+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:53.630 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:53.628+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:53.984 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:53.982+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:53.983247+0000) 2022-04-04T01:32:54.045 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:54.044+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:54.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:54.597+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:54.984 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:54.982+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:54.983468+0000) 2022-04-04T01:32:55.020 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:55.017+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:55.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:55.612+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:55.984 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:55.982+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:55.983674+0000) 2022-04-04T01:32:56.023 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:56.022+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:56.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:56.610+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:56.984 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:56.983+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:56.983865+0000) 2022-04-04T01:32:57.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:57.054+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:57.656 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:57.654+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:57.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:57.983+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:57.984060+0000) 2022-04-04T01:32:58.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:58.006+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:58.629 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:58.627+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:58.965 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:32:58.963+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:32:58.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:58.983+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:58.984281+0000) 2022-04-04T01:32:59.207 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:32:59.206+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:32:59.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:32:59.591+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:32:59.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:32:59.983+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:32:59.984503+0000) 2022-04-04T01:33:00.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:00.005+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:00.558 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:00.556+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:00.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:00.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:00.984691+0000) 2022-04-04T01:33:00.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:00.988+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:01.595 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:01.593+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:01.986 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:01.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:01.984883+0000) 2022-04-04T01:33:02.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:02.025+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:02.582 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:02.579+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:02.985 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:02.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:02.985095+0000) 2022-04-04T01:33:02.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:02.996+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:03.583 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:03.581+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:03.986 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:03.984+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:03.985287+0000) 2022-04-04T01:33:04.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:04.023+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:04.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:04.588+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:04.986 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:04.985+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:04.985478+0000) 2022-04-04T01:33:05.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:05.046+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:05.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:05.623+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:05.986 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:05.985+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:05.985672+0000) 2022-04-04T01:33:06.027 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:06.025+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:06.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:06.596+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:06.986 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:06.985+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:06.985894+0000) 2022-04-04T01:33:07.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:07.036+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:07.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:07.595+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:07.987 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:07.985+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:07.986082+0000) 2022-04-04T01:33:08.087 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:08.085+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:08.558 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:08.557+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:08.987 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:08.985+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:08.986282+0000) 2022-04-04T01:33:09.100 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:09.099+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:09.537 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:09.536+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:09.987 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:09.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:09.986498+0000) 2022-04-04T01:33:10.101 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:10.099+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:10.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:10.494+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:10.987 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:10.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:10.986715+0000) 2022-04-04T01:33:11.138 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:11.137+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:11.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:11.448+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:11.987 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:11.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:11.986904+0000) 2022-04-04T01:33:12.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:12.120+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:12.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:12.496+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:12.988 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:12.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:12.987022+0000) 2022-04-04T01:33:13.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:13.152+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:13.501 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:13.500+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:13.988 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:13.986+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:13.987215+0000) 2022-04-04T01:33:14.138 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:14.136+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:14.223 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:33:14.221+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:33:14.473 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:14.471+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:14.988 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:14.987+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:14.987441+0000) 2022-04-04T01:33:15.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:15.134+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:15.480 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:15.479+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:15.988 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:15.987+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:15.987630+0000) 2022-04-04T01:33:16.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:16.178+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:16.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:16.502+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:16.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:16.987+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:16.987839+0000) 2022-04-04T01:33:17.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:17.228+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:17.544 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:17.543+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:17.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:17.987+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:17.988047+0000) 2022-04-04T01:33:18.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:18.264+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:18.590 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:18.589+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:18.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:18.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:18.988223+0000) 2022-04-04T01:33:19.302 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:19.301+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:19.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:19.566+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:19.777 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:33:19.782 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:33:19.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:19.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:19.988431+0000) 2022-04-04T01:33:20.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:20.264+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:20.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:20.613+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:20.989 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:20.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:20.988629+0000) 2022-04-04T01:33:21.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:21.314+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:21.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:21.595+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:21.990 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:21.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:21.988848+0000) 2022-04-04T01:33:22.318 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:22.316+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:22.611 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:22.609+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:22.990 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:22.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:22.989059+0000) 2022-04-04T01:33:23.343 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:23.342+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:23.573 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:23.571+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:23.990 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:23.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:23.989288+0000) 2022-04-04T01:33:24.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:24.355+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:24.615 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:24.612+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:24.990 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:24.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:24.989454+0000) 2022-04-04T01:33:25.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:25.375+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:25.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:25.577+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:25.990 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:25.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:25.989617+0000) 2022-04-04T01:33:26.343 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:26.341+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:26.568 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:26.566+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:26.990 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:26.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:26.989832+0000) 2022-04-04T01:33:27.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:27.309+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:27.603 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:27.601+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:27.991 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:27.989+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:27.990009+0000) 2022-04-04T01:33:28.267 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:28.265+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:28.609 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:28.607+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:28.991 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:28.988+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:28.990155+0000) 2022-04-04T01:33:29.239 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:29.236+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:29.240 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:33:29.236+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:33:29.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:29.641+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:29.991 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:29.989+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:29.990388+0000) 2022-04-04T01:33:30.238 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:30.235+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:30.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:30.612+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:30.991 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:30.989+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:30.990554+0000) 2022-04-04T01:33:31.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:31.227+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:31.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:31.590+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:31.991 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:31.989+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:31.990747+0000) 2022-04-04T01:33:32.256 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:32.254+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:32.574 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:32.571+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:32.992 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:32.990+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:32.990965+0000) 2022-04-04T01:33:33.263 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:33.260+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:33.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:33.539+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:33.992 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:33.990+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:33.991181+0000) 2022-04-04T01:33:34.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:34.216+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:34.535 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:34.533+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:34.992 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:34.990+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:34.991401+0000) 2022-04-04T01:33:35.213 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:35.211+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:35.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:35.578+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:35.992 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:35.990+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:35.991589+0000) 2022-04-04T01:33:36.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:36.225+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:36.609 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:36.607+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:36.992 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:36.991+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:36.991813+0000) 2022-04-04T01:33:37.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:37.233+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:37.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:37.612+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:37.993 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:37.991+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:37.991993+0000) 2022-04-04T01:33:38.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:38.185+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:38.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:38.596+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:38.993 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:38.991+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:38.992182+0000) 2022-04-04T01:33:39.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:39.225+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:39.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:39.594+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:39.993 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:39.991+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:39.992413+0000) 2022-04-04T01:33:40.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:40.195+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:40.644 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:40.642+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:40.993 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:40.991+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:40.992629+0000) 2022-04-04T01:33:41.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:41.181+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:41.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:41.672+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:41.993 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:41.992+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:41.992871+0000) 2022-04-04T01:33:42.225 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:42.223+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:42.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:42.643+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:42.994 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:42.992+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:42.993106+0000) 2022-04-04T01:33:43.180 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:43.178+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:43.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:43.667+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:43.994 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:43.992+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:43.993308+0000) 2022-04-04T01:33:44.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:44.173+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:44.255 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:33:44.253+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:33:44.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:44.692+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:44.994 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:44.992+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:44.993487+0000) 2022-04-04T01:33:45.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:45.201+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:45.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:45.693+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:45.995 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:45.992+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:45.993695+0000) 2022-04-04T01:33:46.209 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:46.207+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:46.740 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:46.738+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:46.994 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:46.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:46.993884+0000) 2022-04-04T01:33:47.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:47.163+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:47.726 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:47.724+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:47.995 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:47.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:47.994099+0000) 2022-04-04T01:33:48.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:48.163+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:48.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:48.771+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:48.995 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:48.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:48.994235+0000) 2022-04-04T01:33:49.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:49.176+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:49.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:49.740+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:49.933 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:33:49.962 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:33:49.995 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:49.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:49.994396+0000) 2022-04-04T01:33:50.149 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:50.147+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:50.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:50.771+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:50.995 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:50.993+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:50.994559+0000) 2022-04-04T01:33:51.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:51.119+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:51.809 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:51.807+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:51.995 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:51.994+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:51.994760+0000) 2022-04-04T01:33:52.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:52.073+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:52.764 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:52.762+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:52.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:52.994+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:52.994982+0000) 2022-04-04T01:33:53.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:53.110+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:53.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:53.789+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:53.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:53.994+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:53.995150+0000) 2022-04-04T01:33:54.094 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:54.092+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:54.814 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:54.812+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:54.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:54.994+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:54.995359+0000) 2022-04-04T01:33:55.091 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:55.090+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:55.812 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:55.810+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:55.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:55.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:55.995591+0000) 2022-04-04T01:33:56.099 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:56.096+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:56.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:56.785+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:56.996 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:56.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:56.995798+0000) 2022-04-04T01:33:57.137 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:57.135+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:57.769 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:57.768+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:57.997 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:57.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:57.995998+0000) 2022-04-04T01:33:58.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:58.172+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:58.762 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:58.760+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:58.997 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:58.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:58.996184+0000) 2022-04-04T01:33:59.176 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:33:59.174+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:33:59.271 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:33:59.269+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:33:59.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:33:59.748+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:33:59.997 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:59.996+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:33:59.996373+0000) 2022-04-04T01:33:59.998 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:59.996+0000 7fd623a1a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T01:33:59.998 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:33:59.996+0000 7fd62421b700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T01:34:00.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:00.001+0000 7fd623a1a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:00.002627+0000) 2022-04-04T01:34:00.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:00.158+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:00.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:00.708+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:00.997 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:00.995+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:00.996513+0000) 2022-04-04T01:34:01.209 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:01.207+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:01.705 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:01.703+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:01.997 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:01.996+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:01.996651+0000) 2022-04-04T01:34:02.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:02.187+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:02.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:02.687+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:02.997 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:02.996+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:02.996843+0000) 2022-04-04T01:34:03.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:03.205+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:03.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:03.698+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:03.998 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:03.996+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:03.996994+0000) 2022-04-04T01:34:04.255 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:04.253+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:04.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:04.653+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:04.998 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:04.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:04.997160+0000) 2022-04-04T01:34:05.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:05.244+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:05.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:05.604+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:05.998 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:05.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:05.997313+0000) 2022-04-04T01:34:06.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:06.230+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:06.591 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:06.589+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:06.998 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:06.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:06.997490+0000) 2022-04-04T01:34:07.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:07.280+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:07.588 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:07.586+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:07.998 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:07.996+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:07.997677+0000) 2022-04-04T01:34:08.262 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:08.261+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:08.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:08.560+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:08.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:08.996+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:08.997836+0000) 2022-04-04T01:34:09.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:09.278+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:09.556 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:09.555+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:09.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:09.996+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:09.997986+0000) 2022-04-04T01:34:10.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:10.297+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:10.528 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:10.527+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:10.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:10.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:10.998183+0000) 2022-04-04T01:34:11.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:11.276+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:11.500 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:11.498+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:11.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:11.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:11.998361+0000) 2022-04-04T01:34:12.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:12.282+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:12.537 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:12.536+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:12.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:12.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:12.998535+0000) 2022-04-04T01:34:13.243 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:13.241+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:13.488 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:13.487+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:13.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:13.997+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:13.998700+0000) 2022-04-04T01:34:14.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:14.271+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:14.286 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:34:14.284+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:34:14.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:14.460+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:14.999 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:14.998+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:14.998884+0000) 2022-04-04T01:34:15.234 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:15.233+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:15.481 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:15.480+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:16.000 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:15.998+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:15.999068+0000) 2022-04-04T01:34:16.225 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:16.223+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:16.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:16.515+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:17.000 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:16.998+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:16.999243+0000) 2022-04-04T01:34:17.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:17.264+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:17.484 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:17.483+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:18.000 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:17.998+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:17.999445+0000) 2022-04-04T01:34:18.313 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:18.311+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:18.445 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:18.443+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:19.000 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:18.998+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:18.999627+0000) 2022-04-04T01:34:19.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:19.278+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:19.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:19.396+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:20.000 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:19.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:19.999791+0000) 2022-04-04T01:34:20.114 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:34:20.118 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:34:20.256 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:20.253+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:20.415 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:20.413+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:21.001 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:20.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:20.999910+0000) 2022-04-04T01:34:21.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:21.252+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:21.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:21.367+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:22.001 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:21.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:22.000023+0000) 2022-04-04T01:34:22.283 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:22.281+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:22.400 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:22.398+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:23.001 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:22.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:23.000178+0000) 2022-04-04T01:34:23.318 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:23.316+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:23.408 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:23.405+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:24.001 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:23.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:24.000321+0000) 2022-04-04T01:34:24.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:24.322+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:24.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:24.447+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:25.001 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:24.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:25.000446+0000) 2022-04-04T01:34:25.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:25.321+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:25.400 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:25.397+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:26.001 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:25.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:26.000543+0000) 2022-04-04T01:34:26.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:26.322+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:26.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:26.372+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:27.001 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:26.999+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:27.000660+0000) 2022-04-04T01:34:27.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:27.349+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:27.365 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:27.363+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:28.001 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:28.000+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:28.000864+0000) 2022-04-04T01:34:28.363 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:28.361+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:28.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:28.363+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:29.002 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:29.000+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:29.001076+0000) 2022-04-04T01:34:29.302 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:34:29.300+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:34:29.364 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:29.362+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:29.403 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:29.401+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:30.002 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:30.000+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:30.001257+0000) 2022-04-04T01:34:30.341 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:30.338+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:30.386 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:30.385+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:31.002 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:31.000+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:31.001441+0000) 2022-04-04T01:34:31.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:31.293+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:31.428 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:31.426+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:32.002 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:32.001+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:32.001600+0000) 2022-04-04T01:34:32.278 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:32.275+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:32.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:32.404+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:33.002 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:33.001+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:33.001763+0000) 2022-04-04T01:34:33.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:33.251+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:33.404 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:33.402+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:34.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:34.001+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:34.001946+0000) 2022-04-04T01:34:34.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:34.270+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:34.405 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:34.403+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:35.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:35.001+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:35.002124+0000) 2022-04-04T01:34:35.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:35.296+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:35.432 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:35.430+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:36.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:36.001+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:36.002287+0000) 2022-04-04T01:34:36.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:36.248+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:36.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:36.433+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:37.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:37.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:37.002413+0000) 2022-04-04T01:34:37.221 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:37.219+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:37.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:37.389+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:38.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:38.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:38.002604+0000) 2022-04-04T01:34:38.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:38.258+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:38.342 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:38.340+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:39.003 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:39.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:39.002800+0000) 2022-04-04T01:34:39.261 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:39.259+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:39.298 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:39.296+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:40.004 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:40.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:40.002978+0000) 2022-04-04T01:34:40.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:40.288+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:40.312 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:40.310+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:41.004 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:41.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:41.003150+0000) 2022-04-04T01:34:41.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:41.243+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:41.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:41.321+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:42.004 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:42.002+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:42.003274+0000) 2022-04-04T01:34:42.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:42.228+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:42.332 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:42.330+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:43.004 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:43.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:43.003450+0000) 2022-04-04T01:34:43.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:43.221+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:43.298 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:43.296+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:44.005 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:44.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:44.003660+0000) 2022-04-04T01:34:44.202 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:44.199+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:44.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:44.283+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:44.317 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:34:44.316+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:34:45.010 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:45.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:45.003811+0000) 2022-04-04T01:34:45.249 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:45.247+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:45.290 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:45.288+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:46.005 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:46.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:46.003919+0000) 2022-04-04T01:34:46.293 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:46.291+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:46.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:46.297+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:47.004 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:47.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:47.004018+0000) 2022-04-04T01:34:47.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:47.271+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:47.343 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:47.341+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:48.005 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:48.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:48.004205+0000) 2022-04-04T01:34:48.307 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:48.306+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:48.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:48.316+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:49.005 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:49.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:49.004351+0000) 2022-04-04T01:34:49.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:49.289+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:49.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:49.349+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:50.005 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:50.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:50.004547+0000) 2022-04-04T01:34:50.279 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:34:50.283 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:34:50.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:50.302+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:50.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:50.320+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:51.006 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:51.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:51.004721+0000) 2022-04-04T01:34:51.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:51.308+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:51.348 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:51.345+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:52.005 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:52.003+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:52.004931+0000) 2022-04-04T01:34:52.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:52.277+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:52.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:52.315+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:53.006 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:53.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:53.005119+0000) 2022-04-04T01:34:53.276 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:53.274+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:53.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:53.322+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:54.006 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:54.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:54.005302+0000) 2022-04-04T01:34:54.228 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:54.225+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:54.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:54.274+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:55.006 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:55.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:55.005498+0000) 2022-04-04T01:34:55.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:55.248+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:55.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:55.266+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:56.006 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:56.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:56.005664+0000) 2022-04-04T01:34:56.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:56.222+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:56.271 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:56.269+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:57.006 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:57.004+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:57.005802+0000) 2022-04-04T01:34:57.213 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:57.211+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:57.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:57.249+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:58.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:58.005+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:58.005981+0000) 2022-04-04T01:34:58.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:58.246+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:58.274 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:58.273+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:59.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:34:59.005+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:34:59.006153+0000) 2022-04-04T01:34:59.291 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:34:59.290+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:34:59.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:34:59.293+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:34:59.333 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:34:59.331+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:35:00.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:00.005+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:00.006299+0000) 2022-04-04T01:35:00.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:00.323+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:00.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:00.326+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:01.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:01.005+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:01.006458+0000) 2022-04-04T01:35:01.290 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:01.289+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:01.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:01.319+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:02.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:02.005+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:02.006615+0000) 2022-04-04T01:35:02.313 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:02.311+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:02.340 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:02.339+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:03.007 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:03.006+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:03.006823+0000) 2022-04-04T01:35:03.358 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:03.356+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:03.382 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:03.381+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:04.008 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:04.006+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:04.006996+0000) 2022-04-04T01:35:04.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:04.370+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:04.374 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:04.373+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:05.008 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:05.006+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:05.007138+0000) 2022-04-04T01:35:05.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:05.359+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:05.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:05.413+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:06.008 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:06.006+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:06.007323+0000) 2022-04-04T01:35:06.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:06.386+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:06.438 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:06.437+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:07.008 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:07.006+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:07.007507+0000) 2022-04-04T01:35:07.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:07.361+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:07.477 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:07.475+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:08.008 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:08.007+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:08.007691+0000) 2022-04-04T01:35:08.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:08.350+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:08.431 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:08.429+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:09.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:09.007+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:09.007865+0000) 2022-04-04T01:35:09.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:09.311+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:09.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:09.478+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:10.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:10.007+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:10.008047+0000) 2022-04-04T01:35:10.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:10.297+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:10.510 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:10.508+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:11.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:11.007+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:11.008254+0000) 2022-04-04T01:35:11.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:11.259+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:11.477 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:11.476+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:12.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:12.007+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:12.008398+0000) 2022-04-04T01:35:12.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:12.269+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:12.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:12.516+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:13.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:13.008+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:13.008576+0000) 2022-04-04T01:35:13.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:13.285+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:13.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:13.512+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:14.009 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:14.008+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:14.008750+0000) 2022-04-04T01:35:14.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:14.267+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:14.349 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:35:14.347+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:35:14.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:14.520+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:15.010 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:15.008+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:15.008920+0000) 2022-04-04T01:35:15.220 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:15.217+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:15.475 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:15.473+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:16.010 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:16.008+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:16.009066+0000) 2022-04-04T01:35:16.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:16.230+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:16.458 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:16.456+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:17.010 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:17.008+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:17.009262+0000) 2022-04-04T01:35:17.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:17.243+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:17.500 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:17.498+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:18.010 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:18.008+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:18.009432+0000) 2022-04-04T01:35:18.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:18.290+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:18.499 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:18.497+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:19.010 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:19.009+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:19.009605+0000) 2022-04-04T01:35:19.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:19.325+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:19.520 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:19.518+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:20.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:20.009+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:20.009798+0000) 2022-04-04T01:35:20.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:20.337+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:20.427 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:35:20.431 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:35:20.527 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:20.525+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:21.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:21.009+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:21.009984+0000) 2022-04-04T01:35:21.375 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:21.372+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:21.507 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:21.505+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:22.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:22.009+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:22.010177+0000) 2022-04-04T01:35:22.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:22.324+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:22.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:22.457+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:23.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:23.010+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:23.010359+0000) 2022-04-04T01:35:23.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:23.370+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:23.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:23.417+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:24.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:24.010+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:24.010512+0000) 2022-04-04T01:35:24.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:24.360+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:24.430 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:24.428+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:25.012 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:25.009+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:25.010689+0000) 2022-04-04T01:35:25.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:25.390+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:25.401 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:25.399+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:35:26.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:26.010+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:26.010825+0000) 2022-04-04T01:35:26.356 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:26.353+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:26.379 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:26.377+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:27.011 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:27.010+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:27.010937+0000) 2022-04-04T01:35:27.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:27.311+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:27.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:27.416+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:28.012 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:28.010+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:28.011103+0000) 2022-04-04T01:35:28.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:28.329+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:28.443 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:28.441+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:29.012 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:29.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:29.011286+0000) 2022-04-04T01:35:29.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:29.316+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:29.365 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:35:29.363+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:35:29.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:29.472+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:30.012 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:30.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:30.011466+0000) 2022-04-04T01:35:30.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:30.304+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:30.441 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:30.439+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:31.012 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:31.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:31.011645+0000) 2022-04-04T01:35:31.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:31.277+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:31.432 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:31.429+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:32.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:32.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:32.011819+0000) 2022-04-04T01:35:32.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:32.257+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:32.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:32.458+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:33.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:33.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:33.012006+0000) 2022-04-04T01:35:33.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:33.249+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:33.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:33.421+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:34.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:34.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:34.012177+0000) 2022-04-04T01:35:34.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:34.276+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:34.444 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:34.442+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:35.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:35.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:35.012327+0000) 2022-04-04T01:35:35.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:35.294+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:35.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:35.483+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:36.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:36.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:36.012484+0000) 2022-04-04T01:35:36.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:36.267+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:36.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:36.533+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:37.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:37.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:37.012627+0000) 2022-04-04T01:35:37.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:37.228+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:37.533 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:37.532+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:38.013 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:38.011+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:38.012795+0000) 2022-04-04T01:35:38.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:38.186+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:38.506 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:38.505+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:39.014 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:39.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:39.012974+0000) 2022-04-04T01:35:39.165 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:39.163+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:39.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:39.550+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:40.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:40.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:40.013147+0000) 2022-04-04T01:35:40.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:40.189+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:40.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:40.574+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:41.014 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:41.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:41.013306+0000) 2022-04-04T01:35:41.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:41.226+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:41.598 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:41.596+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:42.014 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:42.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:42.013505+0000) 2022-04-04T01:35:42.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:42.195+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:42.619 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:42.617+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:43.014 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:43.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:43.013700+0000) 2022-04-04T01:35:43.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:43.206+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:43.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:43.614+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:44.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:44.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:44.013889+0000) 2022-04-04T01:35:44.169 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:44.167+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:44.381 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:35:44.378+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:35:44.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:44.612+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:45.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:45.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:45.014035+0000) 2022-04-04T01:35:45.134 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:45.132+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:45.566 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:45.564+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:46.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:46.012+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:46.014186+0000) 2022-04-04T01:35:46.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:46.114+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:46.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:46.541+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:47.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:47.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:47.014356+0000) 2022-04-04T01:35:47.069 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:47.067+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:47.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:47.493+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:48.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:48.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:48.014535+0000) 2022-04-04T01:35:48.096 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:48.094+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:48.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:48.457+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:49.015 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:49.013+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:49.014693+0000) 2022-04-04T01:35:49.133 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:49.131+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:49.473 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:49.472+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:50.016 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:50.014+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:50.014870+0000) 2022-04-04T01:35:50.165 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:50.163+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:50.432 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:50.430+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:50.587 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:35:50.591 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:35:51.016 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:51.014+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:51.015079+0000) 2022-04-04T01:35:51.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:51.206+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:51.417 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:51.415+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:52.016 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:52.014+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:52.015283+0000) 2022-04-04T01:35:52.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:52.182+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:52.465 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:52.463+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:53.016 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:53.014+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:53.015451+0000) 2022-04-04T01:35:53.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:53.187+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:53.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:53.512+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:54.016 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:54.014+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:54.015609+0000) 2022-04-04T01:35:54.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:54.154+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:54.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:54.477+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:55.016 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:55.015+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:55.015799+0000) 2022-04-04T01:35:55.117 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:55.116+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:55.491 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:55.490+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:56.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:56.015+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:56.015945+0000) 2022-04-04T01:35:56.165 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:56.163+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:56.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:56.534+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:57.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:57.015+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:57.016107+0000) 2022-04-04T01:35:57.126 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:57.125+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:57.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:57.550+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:58.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:58.015+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:58.016257+0000) 2022-04-04T01:35:58.129 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:58.127+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:58.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:58.569+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:35:59.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:35:59.015+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:35:59.016447+0000) 2022-04-04T01:35:59.082 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:35:59.081+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:35:59.396 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:35:59.395+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:35:59.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:35:59.606+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:00.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:00.016+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:00.016646+0000) 2022-04-04T01:36:00.093 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:00.092+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:00.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:00.582+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:01.017 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:01.016+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:01.016827+0000) 2022-04-04T01:36:01.093 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:01.092+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:01.114 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-04T01:36:01.112+0000 7f2fd661a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T01:36:01.114 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-04T01:36:01.112+0000 7f2fd761c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T01:36:01.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:01.549+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:02.018 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:02.016+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:02.017021+0000) 2022-04-04T01:36:02.102 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:02.101+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:02.542 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:02.540+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:03.018 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:03.016+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:03.017237+0000) 2022-04-04T01:36:03.152 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:03.151+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:03.558 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:03.557+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:04.018 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:04.016+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:04.017389+0000) 2022-04-04T01:36:04.117 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:04.115+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:04.598 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:04.597+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:05.018 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:05.017+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:05.017606+0000) 2022-04-04T01:36:05.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:05.161+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:05.570 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:05.569+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:06.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:06.017+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:06.017808+0000) 2022-04-04T01:36:06.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:06.192+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:06.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:06.530+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:07.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:07.017+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:07.017998+0000) 2022-04-04T01:36:07.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:07.162+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:07.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:07.559+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:08.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:08.017+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:08.018140+0000) 2022-04-04T01:36:08.162 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:08.159+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:08.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:08.550+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:09.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:09.017+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:09.018258+0000) 2022-04-04T01:36:09.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:09.192+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:09.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:09.512+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:10.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:10.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:10.018410+0000) 2022-04-04T01:36:10.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:10.170+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:10.506 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:10.503+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:11.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:11.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:11.018549+0000) 2022-04-04T01:36:11.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:11.172+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:11.539 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:11.536+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:12.019 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:12.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:12.018722+0000) 2022-04-04T01:36:12.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:12.178+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:12.516 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:12.514+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:13.020 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:13.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:13.018944+0000) 2022-04-04T01:36:13.141 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:13.139+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:13.554 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:13.551+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:14.020 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:14.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:14.019127+0000) 2022-04-04T01:36:14.133 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:14.131+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:14.412 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:36:14.410+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:36:14.574 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:14.572+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:15.020 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:15.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:15.019303+0000) 2022-04-04T01:36:15.097 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:15.095+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:15.622 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:15.619+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:16.020 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:16.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:16.019405+0000) 2022-04-04T01:36:16.083 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:16.081+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:16.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:16.633+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:17.020 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:17.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:17.019548+0000) 2022-04-04T01:36:17.099 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:17.097+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:17.601 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:17.598+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:18.020 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:18.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:18.019727+0000) 2022-04-04T01:36:18.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:18.113+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:18.582 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:18.580+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:19.020 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:19.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:19.019932+0000) 2022-04-04T01:36:19.069 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:19.067+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:19.590 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:19.588+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:20.021 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:20.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:20.020116+0000) 2022-04-04T01:36:20.078 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:20.076+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:20.630 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:20.627+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:20.739 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:36:20.743 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:36:21.021 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:21.018+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:21.020338+0000) 2022-04-04T01:36:21.042 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:21.040+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:21.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:21.636+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:22.005 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:22.003+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:22.021 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:22.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:22.020509+0000) 2022-04-04T01:36:22.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:22.630+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:22.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:22.993+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:23.021 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:23.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:23.020699+0000) 2022-04-04T01:36:23.606 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:23.603+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:23.994 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:23.992+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:24.021 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:24.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:24.020878+0000) 2022-04-04T01:36:24.609 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:24.607+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:24.980 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:24.978+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:25.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:25.019+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:25.021029+0000) 2022-04-04T01:36:25.637 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:25.634+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:25.989 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:25.987+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:26.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:26.020+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:26.021178+0000) 2022-04-04T01:36:26.661 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:26.659+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:26.991 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:26.989+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:27.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:27.020+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:27.021360+0000) 2022-04-04T01:36:27.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:27.696+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:28.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:28.020+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:28.021540+0000) 2022-04-04T01:36:28.035 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:28.033+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:28.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:28.691+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:29.022 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:29.020+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:29.021720+0000) 2022-04-04T01:36:29.046 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:29.044+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:29.428 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:36:29.425+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:36:29.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:29.696+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:30.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:30.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:30.021927+0000) 2022-04-04T01:36:30.082 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:30.079+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:30.660 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:30.658+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:31.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:31.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:31.022134+0000) 2022-04-04T01:36:31.058 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:31.055+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:31.678 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:31.676+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:32.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:32.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:32.022272+0000) 2022-04-04T01:36:32.064 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:32.062+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:32.660 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:32.658+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:33.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:33.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:33.022392+0000) 2022-04-04T01:36:33.093 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:33.091+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:33.633 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:33.630+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:34.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:34.021+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:34.022580+0000) 2022-04-04T01:36:34.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:34.134+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:34.606 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:34.604+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:35.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:35.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:35.022715+0000) 2022-04-04T01:36:35.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:35.146+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:35.625 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:35.623+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:36.023 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:36.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:36.022839+0000) 2022-04-04T01:36:36.118 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:36.116+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:36.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:36.666+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:37.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:37.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:37.022957+0000) 2022-04-04T01:36:37.142 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:37.140+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:37.624 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:37.622+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:38.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:38.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:38.023163+0000) 2022-04-04T01:36:38.126 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:38.124+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:38.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:38.654+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:39.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:39.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:39.023256+0000) 2022-04-04T01:36:39.100 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:39.098+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:39.701 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:39.699+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:40.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:40.022+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:40.023416+0000) 2022-04-04T01:36:40.139 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:40.137+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:40.735 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:40.732+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:41.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:41.023+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:41.023570+0000) 2022-04-04T01:36:41.154 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:41.152+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:41.742 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:41.740+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:42.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:42.023+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:42.023696+0000) 2022-04-04T01:36:42.143 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:42.141+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:42.708 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:42.706+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:43.024 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:43.023+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:43.023799+0000) 2022-04-04T01:36:43.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:43.153+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:43.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:43.737+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:44.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:44.023+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:44.023970+0000) 2022-04-04T01:36:44.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:44.154+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:44.444 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:36:44.442+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:36:44.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:44.768+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:45.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:45.023+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:45.024096+0000) 2022-04-04T01:36:45.156 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:45.154+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:45.774 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:45.772+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:46.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:46.023+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:46.024157+0000) 2022-04-04T01:36:46.116 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:46.115+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:46.803 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:46.801+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:47.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:47.023+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:47.024324+0000) 2022-04-04T01:36:47.089 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:47.087+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:47.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:47.805+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:48.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:48.024+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:48.024517+0000) 2022-04-04T01:36:48.059 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:48.058+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:48.850 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:48.849+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:49.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:49.024+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:49.024687+0000) 2022-04-04T01:36:49.050 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:49.049+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:49.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:49.852+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:50.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:50.024+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:50.024844+0000) 2022-04-04T01:36:50.048 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:50.047+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:50.870 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:36:50.874 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:36:50.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:50.881+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:51.025 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:51.024+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:51.025005+0000) 2022-04-04T01:36:51.051 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:51.050+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:51.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:51.903+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:52.026 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:52.024+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:52.025225+0000) 2022-04-04T01:36:52.094 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:52.092+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:52.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:52.878+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:53.026 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:53.025+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:53.025425+0000) 2022-04-04T01:36:53.127 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:53.125+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:53.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:53.869+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:54.026 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:54.025+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:54.025583+0000) 2022-04-04T01:36:54.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:54.154+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:54.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:54.874+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:55.026 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:55.025+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:55.025763+0000) 2022-04-04T01:36:55.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:55.162+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:55.831 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:55.829+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:56.026 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:56.025+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:56.025923+0000) 2022-04-04T01:36:56.204 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:56.203+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:56.804 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:56.802+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:57.027 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:57.025+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:57.026041+0000) 2022-04-04T01:36:57.192 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:57.191+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:57.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:57.774+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:58.027 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:58.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:58.026194+0000) 2022-04-04T01:36:58.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:58.239+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:58.763 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:58.762+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:36:59.027 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:36:59.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:36:59.026344+0000) 2022-04-04T01:36:59.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:36:59.288+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:36:59.460 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:36:59.458+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:36:59.719 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:36:59.718+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:00.027 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:00.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:00.026547+0000) 2022-04-04T01:37:00.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:00.328+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:00.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:00.732+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:01.027 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:01.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:01.026796+0000) 2022-04-04T01:37:01.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:01.367+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:01.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:01.725+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:02.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:02.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:02.027020+0000) 2022-04-04T01:37:02.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:02.366+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:02.716 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:02.714+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:03.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:03.027+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:03.027262+0000) 2022-04-04T01:37:03.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:03.401+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:03.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:03.692+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:04.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:04.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:04.027483+0000) 2022-04-04T01:37:04.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:04.381+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:04.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:04.676+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:05.028 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:05.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:05.027712+0000) 2022-04-04T01:37:05.396 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:05.394+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:05.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:05.726+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:06.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:06.026+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:06.027941+0000) 2022-04-04T01:37:06.421 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:06.418+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:06.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:06.685+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:07.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:07.027+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:07.028163+0000) 2022-04-04T01:37:07.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:07.369+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:07.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:07.663+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:08.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:08.027+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:08.028386+0000) 2022-04-04T01:37:08.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:08.352+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:08.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:08.637+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:09.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:09.027+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:09.028583+0000) 2022-04-04T01:37:09.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:09.346+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:09.646 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:09.644+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:10.029 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:10.027+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:10.028796+0000) 2022-04-04T01:37:10.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:10.319+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:10.685 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:10.683+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:11.030 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:11.028+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:11.029051+0000) 2022-04-04T01:37:11.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:11.305+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:11.678 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:11.675+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:12.030 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:12.028+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:12.029308+0000) 2022-04-04T01:37:12.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:12.258+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:12.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:12.699+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:13.030 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:13.028+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:13.029576+0000) 2022-04-04T01:37:13.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:13.297+0000 7f88d0ac1700 -1 osd.4 140 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:13.734 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:13.731+0000 7fd3b190b700 -1 osd.6 140 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:13.737 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-04T01:37:13.735+0000 7f6a1f20e700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-04T01:37:13.752 INFO:tasks.ceph.mds.f.smithi186.stderr: -5> 2022-04-04T01:37:13.735+0000 7f6a1f20e700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-04T01:37:13.775 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-04T01:37:14.030 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:14.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:14.029830+0000) 2022-04-04T01:37:14.301 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:14.299+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:14.475 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:37:14.474+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:37:14.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:14.724+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:15.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:15.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:15.030026+0000) 2022-04-04T01:37:15.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:15.284+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:15.696 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:15.694+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:16.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:16.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:16.030238+0000) 2022-04-04T01:37:16.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:16.263+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:16.678 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:16.676+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:17.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:17.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:17.030397+0000) 2022-04-04T01:37:17.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:17.271+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:17.659 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:17.657+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:18.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:18.029+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:18.030573+0000) 2022-04-04T01:37:18.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:18.257+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 68 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:18.659 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:18.657+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:19.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:19.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:19.030720+0000) 2022-04-04T01:37:19.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:19.221+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:19.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:19.669+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:20.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:20.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:20.030915+0000) 2022-04-04T01:37:20.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:20.268+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:20.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:20.712+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:21.017 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:37:21.022 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:37:21.031 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:21.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:21.031092+0000) 2022-04-04T01:37:21.255 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:21.254+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:21.686 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:21.684+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:22.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:22.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:22.031277+0000) 2022-04-04T01:37:22.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:22.257+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:22.684 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:22.683+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:23.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:23.030+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:23.031451+0000) 2022-04-04T01:37:23.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:23.271+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:23.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:23.729+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:24.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:24.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:24.031624+0000) 2022-04-04T01:37:24.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:24.273+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:24.757 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:24.755+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:25.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:25.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:25.031835+0000) 2022-04-04T01:37:25.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:25.311+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:25.804 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:25.802+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:26.032 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:26.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:26.032065+0000) 2022-04-04T01:37:26.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:26.290+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:26.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:26.779+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:27.033 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:27.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:27.032224+0000) 2022-04-04T01:37:27.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:27.275+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:27.767 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:27.765+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:28.033 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:28.031+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:28.032430+0000) 2022-04-04T01:37:28.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:28.266+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:28.757 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:28.755+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:29.033 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:29.032+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:29.032623+0000) 2022-04-04T01:37:29.233 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:29.231+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:29.491 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:37:29.489+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:37:29.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:29.724+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:30.033 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:30.032+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:30.032807+0000) 2022-04-04T01:37:30.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:30.243+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:30.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:30.680+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:31.033 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:31.032+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:31.032923+0000) 2022-04-04T01:37:31.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:31.278+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:31.722 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:31.719+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:32.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:32.032+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:32.033131+0000) 2022-04-04T01:37:32.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:32.255+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:32.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:32.767+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:33.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:33.032+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:33.033287+0000) 2022-04-04T01:37:33.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:33.302+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:33.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:33.809+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:34.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:34.033+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:34.033476+0000) 2022-04-04T01:37:34.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:34.343+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:34.833 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:34.831+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:35.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:35.033+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:35.033665+0000) 2022-04-04T01:37:35.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:35.392+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:35.850 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:35.849+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:36.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:36.033+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:36.033875+0000) 2022-04-04T01:37:36.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:36.414+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:36.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:36.879+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:37.034 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:37.033+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:37.034033+0000) 2022-04-04T01:37:37.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:37.433+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:37.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:37.915+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:38.035 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:38.033+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:38.034168+0000) 2022-04-04T01:37:38.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:38.397+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:38.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:38.875+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:39.035 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:39.034+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:39.034333+0000) 2022-04-04T01:37:39.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:39.350+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:39.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:39.916+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:40.035 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:40.034+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:40.034537+0000) 2022-04-04T01:37:40.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:40.316+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:40.870 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:40.867+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:41.035 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:41.034+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:41.034727+0000) 2022-04-04T01:37:41.357 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:41.355+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:41.913 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:41.911+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:42.035 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:42.034+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:42.034944+0000) 2022-04-04T01:37:42.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:42.327+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:42.903 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:42.901+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:43.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:43.034+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:43.035117+0000) 2022-04-04T01:37:43.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:43.332+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:43.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:43.866+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:44.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:44.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:44.035331+0000) 2022-04-04T01:37:44.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:44.301+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:44.507 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:37:44.505+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:37:44.841 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:44.840+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:45.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:45.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:45.035523+0000) 2022-04-04T01:37:45.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:45.344+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:45.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:45.876+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:46.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:46.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:46.035712+0000) 2022-04-04T01:37:46.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:46.332+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:46.915 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:46.913+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:47.036 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:47.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:47.035857+0000) 2022-04-04T01:37:47.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:47.308+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:47.891 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:47.890+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:48.037 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:48.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:48.036041+0000) 2022-04-04T01:37:48.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:48.323+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:48.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:48.889+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:49.038 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:49.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:49.036233+0000) 2022-04-04T01:37:49.364 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:49.363+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:49.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:49.939+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:50.037 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:50.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:50.036445+0000) 2022-04-04T01:37:50.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:50.394+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:50.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:50.956+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:51.037 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:51.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:51.036679+0000) 2022-04-04T01:37:51.106 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:37:51.110 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:37:51.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:51.344+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:51.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:51.976+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:52.037 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:52.035+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:52.036893+0000) 2022-04-04T01:37:52.334 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:52.333+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:52.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:52.957+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:53.038 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:53.036+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:53.037089+0000) 2022-04-04T01:37:53.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:53.348+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:53.987 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:53.986+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:54.038 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:54.036+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:54.037246+0000) 2022-04-04T01:37:54.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:54.318+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:54.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:54.947+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:55.038 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:55.036+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:55.037374+0000) 2022-04-04T01:37:55.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:55.306+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:55.966 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:55.964+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:56.038 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:56.036+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:56.037561+0000) 2022-04-04T01:37:56.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:56.269+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:56.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:56.985+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:57.038 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:57.036+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:57.037655+0000) 2022-04-04T01:37:57.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:57.309+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:57.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:57.979+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:58.039 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:58.037+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:58.037859+0000) 2022-04-04T01:37:58.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:58.298+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:59.005 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:59.002+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:37:59.038 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:37:59.037+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:37:59.038031+0000) 2022-04-04T01:37:59.301 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:37:59.299+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:37:59.523 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:37:59.521+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:37:59.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:37:59.974+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:00.039 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:00.037+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:00.038252+0000) 2022-04-04T01:38:00.294 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:00.292+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:00.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:00.962+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:01.039 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:01.037+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:01.038477+0000) 2022-04-04T01:38:01.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:01.300+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:01.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:01.946+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:02.039 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:02.037+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:02.038659+0000) 2022-04-04T01:38:02.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:02.323+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:02.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:02.980+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:03.039 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:03.038+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:03.038847+0000) 2022-04-04T01:38:03.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:03.316+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:03.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:03.939+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:04.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:04.038+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:04.039062+0000) 2022-04-04T01:38:04.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:04.319+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:04.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:04.925+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:05.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:05.038+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:05.039207+0000) 2022-04-04T01:38:05.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:05.301+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:05.895 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:05.892+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:06.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:06.038+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:06.039377+0000) 2022-04-04T01:38:06.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:06.255+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:06.915 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:06.913+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:07.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:07.038+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:07.039582+0000) 2022-04-04T01:38:07.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:07.220+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:07.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:07.945+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:08.040 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:08.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:08.039806+0000) 2022-04-04T01:38:08.242 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:08.240+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:08.905 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:08.902+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:09.041 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:09.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:09.039997+0000) 2022-04-04T01:38:09.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:09.274+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:09.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:09.922+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:10.041 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:10.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:10.040189+0000) 2022-04-04T01:38:10.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:10.321+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:10.925 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:10.923+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:11.041 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:11.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:11.040330+0000) 2022-04-04T01:38:11.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:11.296+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:11.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:11.951+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:12.041 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:12.039+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:12.040529+0000) 2022-04-04T01:38:12.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:12.251+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:12.960 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:12.958+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:13.041 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:13.040+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:13.040725+0000) 2022-04-04T01:38:13.257 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:13.255+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:13.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:13.921+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:14.041 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:14.040+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:14.040887+0000) 2022-04-04T01:38:14.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:14.293+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:14.539 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:38:14.537+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:38:14.959 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:14.957+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:15.041 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:15.040+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:15.041080+0000) 2022-04-04T01:38:15.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:15.320+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:15.976 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:15.974+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:16.042 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:16.040+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:16.041331+0000) 2022-04-04T01:38:16.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:16.370+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:17.001 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:16.998+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:17.042 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:17.041+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:17.041553+0000) 2022-04-04T01:38:17.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:17.409+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:17.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:17.956+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:18.042 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:18.041+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:18.041728+0000) 2022-04-04T01:38:18.453 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:18.451+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:18.943 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:18.942+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:19.042 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:19.041+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:19.041937+0000) 2022-04-04T01:38:19.493 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:19.491+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:19.934 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:19.932+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:20.043 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:20.041+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:20.042165+0000) 2022-04-04T01:38:20.468 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:20.465+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:20.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:20.903+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:21.043 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:21.042+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:21.042328+0000) 2022-04-04T01:38:21.271 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:38:21.274 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:38:21.453 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:21.452+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:21.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:21.937+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:22.043 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:22.041+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:22.042562+0000) 2022-04-04T01:38:22.438 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:22.436+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:22.934 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:22.932+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:23.043 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:23.042+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:23.042829+0000) 2022-04-04T01:38:23.442 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:23.440+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:23.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:23.981+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:24.043 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:24.042+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:24.043055+0000) 2022-04-04T01:38:24.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:24.397+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:25.021 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:25.019+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:25.044 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:25.042+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:25.043245+0000) 2022-04-04T01:38:25.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:25.369+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:26.044 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:26.043+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:26.043474+0000) 2022-04-04T01:38:26.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:26.069+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:26.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:26.372+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:27.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:27.026+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:27.044 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:27.043+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:27.043718+0000) 2022-04-04T01:38:27.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:27.415+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:28.045 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:28.043+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:28.043954+0000) 2022-04-04T01:38:28.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:28.074+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:28.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:28.366+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:29.045 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:29.043+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:29.044092+0000) 2022-04-04T01:38:29.093 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:29.092+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:29.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:29.350+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:29.555 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:38:29.553+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:38:30.045 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:30.043+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:30.044201+0000) 2022-04-04T01:38:30.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:30.053+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:30.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:30.364+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:31.045 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:31.044+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:31.044365+0000) 2022-04-04T01:38:31.065 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:31.063+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:31.375 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:31.373+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:32.045 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:32.044+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:32.044519+0000) 2022-04-04T01:38:32.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:32.077+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:32.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:32.396+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:33.045 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:33.043+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:33.044736+0000) 2022-04-04T01:38:33.115 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:33.113+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:33.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:33.361+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:34.046 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:34.043+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:34.044978+0000) 2022-04-04T01:38:34.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:34.153+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:34.404 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:34.402+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:35.046 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:35.043+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:35.045207+0000) 2022-04-04T01:38:35.119 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:35.116+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:35.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:35.358+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:36.046 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:36.044+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:36.045437+0000) 2022-04-04T01:38:36.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:36.140+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:36.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:36.383+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:37.046 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:37.044+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:37.045605+0000) 2022-04-04T01:38:37.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:37.096+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:37.407 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:37.405+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:38.046 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:38.044+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:38.045766+0000) 2022-04-04T01:38:38.141 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:38.140+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:38.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:38.369+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:39.047 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:39.045+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:39.045960+0000) 2022-04-04T01:38:39.152 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:39.151+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:39.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:39.393+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:40.047 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:40.045+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:40.046184+0000) 2022-04-04T01:38:40.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:40.122+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:40.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:40.389+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:41.047 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:41.045+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:41.046374+0000) 2022-04-04T01:38:41.104 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:41.103+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:41.378 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:41.376+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:42.047 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:42.045+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:42.046508+0000) 2022-04-04T01:38:42.069 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:42.067+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:42.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:42.330+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:43.047 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:43.045+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:43.046647+0000) 2022-04-04T01:38:43.115 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:43.114+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:43.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:43.341+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:44.048 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:44.046+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:44.046865+0000) 2022-04-04T01:38:44.115 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:44.114+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:44.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:44.347+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:44.570 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:38:44.569+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:38:45.048 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:45.046+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:45.047084+0000) 2022-04-04T01:38:45.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:45.132+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:45.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:45.369+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:46.048 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:46.046+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:46.047263+0000) 2022-04-04T01:38:46.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:46.158+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:46.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:46.325+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:47.048 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:47.046+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:47.047444+0000) 2022-04-04T01:38:47.194 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:47.193+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:47.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:47.326+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:48.048 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:48.046+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:48.047620+0000) 2022-04-04T01:38:48.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:48.206+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:48.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:48.327+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:49.049 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:49.047+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:49.047853+0000) 2022-04-04T01:38:49.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:49.230+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:49.312 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:49.311+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:50.049 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:50.047+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:50.048053+0000) 2022-04-04T01:38:50.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:50.231+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:50.275 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:50.274+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:51.049 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:51.047+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:51.048219+0000) 2022-04-04T01:38:51.227 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:51.226+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:51.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:51.311+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:51.401 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:38:51.405 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:38:52.049 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:52.047+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:52.048434+0000) 2022-04-04T01:38:52.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:52.229+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:52.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:52.344+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:53.049 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:53.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:53.048676+0000) 2022-04-04T01:38:53.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:53.204+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:53.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:53.360+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:54.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:54.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:54.048829+0000) 2022-04-04T01:38:54.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:54.233+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:54.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:54.316+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:55.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:55.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:55.049002+0000) 2022-04-04T01:38:55.261 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:55.259+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:55.317 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:55.316+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:56.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:56.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:56.049231+0000) 2022-04-04T01:38:56.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:56.275+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:56.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:56.326+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:57.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:57.048+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:57.049427+0000) 2022-04-04T01:38:57.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:57.307+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:57.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:57.318+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:58.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:58.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:58.049634+0000) 2022-04-04T01:38:58.284 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:58.282+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:58.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:58.294+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:59.050 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:38:59.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:38:59.049830+0000) 2022-04-04T01:38:59.254 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:38:59.251+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:38:59.293 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:38:59.291+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:38:59.586 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:38:59.584+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:39:00.051 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:00.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:00.050037+0000) 2022-04-04T01:39:00.220 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:00.217+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:00.244 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:00.242+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:01.051 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:01.049+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:01.050224+0000) 2022-04-04T01:39:01.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:01.222+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:01.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:01.248+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:02.051 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:02.050+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:02.050425+0000) 2022-04-04T01:39:02.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:02.184+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:02.281 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:02.279+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:03.051 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:03.050+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:03.050667+0000) 2022-04-04T01:39:03.205 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:03.203+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:03.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:03.309+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:04.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:04.050+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:04.050915+0000) 2022-04-04T01:39:04.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:04.214+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:04.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:04.312+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:05.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:05.050+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:05.051155+0000) 2022-04-04T01:39:05.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:05.175+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:05.285 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:05.283+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:06.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:06.051+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:06.051378+0000) 2022-04-04T01:39:06.215 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:06.212+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:06.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:06.322+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:07.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:07.051+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:07.051610+0000) 2022-04-04T01:39:07.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:07.225+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:07.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:07.351+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:08.052 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:08.051+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:08.051873+0000) 2022-04-04T01:39:08.182 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:08.180+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:08.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:08.359+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:09.053 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:09.051+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:09.052071+0000) 2022-04-04T01:39:09.134 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:09.132+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:09.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:09.318+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:10.053 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:10.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:10.052289+0000) 2022-04-04T01:39:10.100 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:10.098+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:10.360 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:10.358+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:11.053 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:11.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:11.052517+0000) 2022-04-04T01:39:11.089 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:11.087+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:11.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:11.355+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:12.053 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:12.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:12.052714+0000) 2022-04-04T01:39:12.075 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:12.073+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:12.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:12.350+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:13.031 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:13.028+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:13.053 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:13.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:13.052924+0000) 2022-04-04T01:39:13.321 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:13.318+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:14.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:14.014+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:14.054 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:14.053+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:14.053160+0000) 2022-04-04T01:39:14.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:14.297+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:14.602 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:39:14.600+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:39:15.040 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:15.038+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:15.054 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:15.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:15.053269+0000) 2022-04-04T01:39:15.290 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:15.288+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:16.037 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:16.035+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:16.054 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:16.053+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:16.053472+0000) 2022-04-04T01:39:16.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:16.248+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:17.054 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:17.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:17.053648+0000) 2022-04-04T01:39:17.068 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:17.066+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:17.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:17.263+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:18.055 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:18.052+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:18.053860+0000) 2022-04-04T01:39:18.065 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:18.063+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:18.223 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:18.220+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:19.055 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:19.053+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:19.054085+0000) 2022-04-04T01:39:19.058 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:19.056+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:19.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:19.217+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:20.052 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:20.050+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:20.055 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:20.053+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:20.054303+0000) 2022-04-04T01:39:20.185 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:20.183+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:21.055 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:21.053+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:21.054528+0000) 2022-04-04T01:39:21.062 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:21.060+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:21.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:21.217+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:21.524 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:39:21.528 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:39:22.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:22.036+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:22.055 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:22.053+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:22.054726+0000) 2022-04-04T01:39:22.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:22.193+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:23.045 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:23.043+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:23.055 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:23.054+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:23.054971+0000) 2022-04-04T01:39:23.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:23.177+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:24.044 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:24.042+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:24.056 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:24.054+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:24.055191+0000) 2022-04-04T01:39:24.202 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:24.200+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:25.032 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:25.030+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:25.056 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:25.054+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:25.055387+0000) 2022-04-04T01:39:25.169 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:25.167+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:26.056 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:26.054+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:26.055576+0000) 2022-04-04T01:39:26.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:26.065+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:26.184 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:26.182+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:27.056 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:27.054+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:27.055736+0000) 2022-04-04T01:39:27.098 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:27.096+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:27.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:27.180+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:28.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:28.053+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:28.056 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:28.055+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:28.055970+0000) 2022-04-04T01:39:28.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:28.153+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:29.026 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:29.024+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:29.057 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:29.055+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:29.056184+0000) 2022-04-04T01:39:29.200 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:29.198+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:29.618 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:39:29.616+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:39:30.032 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:30.030+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:30.057 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:30.055+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:30.056424+0000) 2022-04-04T01:39:30.237 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:30.236+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:30.988 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:30.986+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:31.057 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:31.055+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:31.056608+0000) 2022-04-04T01:39:31.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:31.277+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:31.940 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:31.939+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:32.057 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:32.056+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:32.056765+0000) 2022-04-04T01:39:32.242 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:32.241+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:32.961 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:32.960+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:33.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:33.056+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:33.057003+0000) 2022-04-04T01:39:33.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:33.217+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:33.952 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:33.951+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:34.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:34.056+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:34.057229+0000) 2022-04-04T01:39:34.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:34.173+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:34.940 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:34.939+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:35.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:35.056+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:35.057420+0000) 2022-04-04T01:39:35.160 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:35.159+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:35.934 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:35.932+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:36.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:36.056+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:36.057627+0000) 2022-04-04T01:39:36.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:36.155+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:36.978 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:36.977+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:37.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:37.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:37.057766+0000) 2022-04-04T01:39:37.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:37.106+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:38.018 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:38.017+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:38.058 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:38.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:38.057993+0000) 2022-04-04T01:39:38.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:38.073+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:38.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:38.991+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:39.029 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:39.028+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:39.059 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:39.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:39.058152+0000) 2022-04-04T01:39:39.959 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:39.957+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:40.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:40.003+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:40.059 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:40.057+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:40.058351+0000) 2022-04-04T01:39:40.982 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:40.981+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:40.983 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:40.982+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:41.059 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:41.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:41.058534+0000) 2022-04-04T01:39:41.946 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:41.944+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:41.969 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:41.968+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:42.059 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:42.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:42.058697+0000) 2022-04-04T01:39:42.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:42.919+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:42.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:42.947+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:43.060 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:43.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:43.058924+0000) 2022-04-04T01:39:43.881 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:43.879+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:43.938 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:43.937+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:44.060 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:44.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:44.059143+0000) 2022-04-04T01:39:44.634 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:39:44.632+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:39:44.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:44.921+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:44.975 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:44.974+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:45.060 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:45.058+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:45.059368+0000) 2022-04-04T01:39:45.930 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:45.928+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:46.014 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:46.011+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:46.060 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:46.059+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:46.059552+0000) 2022-04-04T01:39:46.957 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:46.955+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:46.985 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:46.983+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:47.060 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:47.059+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:47.059794+0000) 2022-04-04T01:39:47.930 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:47.928+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:47.998 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:47.996+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:48.060 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:48.059+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:48.060057+0000) 2022-04-04T01:39:48.954 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:48.951+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:49.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:49.001+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:49.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:49.059+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:49.060286+0000) 2022-04-04T01:39:49.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:49.915+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:49.967 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:49.965+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:50.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:50.060+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:50.060517+0000) 2022-04-04T01:39:50.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:50.884+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:50.959 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:50.957+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:51.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:51.060+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:51.060764+0000) 2022-04-04T01:39:51.654 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:39:51.658 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:39:51.889 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:51.887+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:51.966 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:51.964+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:52.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:52.060+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:52.060992+0000) 2022-04-04T01:39:52.901 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:52.899+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:53.009 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:53.007+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:53.061 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:53.060+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:53.061222+0000) 2022-04-04T01:39:53.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:53.925+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:54.010 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:54.008+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:54.062 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:54.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:54.061407+0000) 2022-04-04T01:39:54.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:54.971+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:55.043 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:55.041+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:55.062 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:55.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:55.061519+0000) 2022-04-04T01:39:55.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:55.977+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:56.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:56.014+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:56.062 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:56.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:56.061665+0000) 2022-04-04T01:39:56.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:56.988+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:57.018 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:57.016+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:57.062 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:57.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:57.061861+0000) 2022-04-04T01:39:58.014 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:58.012+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:58.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:58.014+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:58.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:58.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:58.062063+0000) 2022-04-04T01:39:59.011 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:59.008+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:39:59.031 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:39:59.029+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:39:59.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:39:59.062+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:39:59.062214+0000) 2022-04-04T01:39:59.649 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:39:59.648+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:39:59.968 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:39:59.966+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:00.014 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:00.012+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:00.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:00.062+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:00.062429+0000) 2022-04-04T01:40:00.991 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:00.989+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:01.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:01.039+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:01.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:01.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:01.062567+0000) 2022-04-04T01:40:02.003 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:02.001+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:02.015 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:02.013+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:02.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:02.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:02.062741+0000) 2022-04-04T01:40:02.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:02.981+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:03.032 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:03.030+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:03.063 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:03.061+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:03.062977+0000) 2022-04-04T01:40:03.935 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:03.932+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:04.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:04.053+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:04.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:04.062+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:04.063181+0000) 2022-04-04T01:40:04.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:04.938+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:05.059 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:05.057+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:05.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:05.062+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:05.063371+0000) 2022-04-04T01:40:05.902 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:05.900+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:06.040 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:06.038+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:06.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:06.062+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:06.063553+0000) 2022-04-04T01:40:06.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:06.931+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:07.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:07.062+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:07.063739+0000) 2022-04-04T01:40:07.065 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:07.063+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:07.918 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:07.916+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:08.038 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:08.036+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:08.064 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:08.063+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:08.063934+0000) 2022-04-04T01:40:08.934 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:08.931+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:09.062 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:09.060+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:09.065 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:09.063+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:09.064156+0000) 2022-04-04T01:40:09.914 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:09.912+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:10.065 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:10.063+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:10.064377+0000) 2022-04-04T01:40:10.105 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:10.103+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:10.898 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:10.896+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:11.065 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:11.063+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:11.064608+0000) 2022-04-04T01:40:11.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:11.148+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:11.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:11.883+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:12.065 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:12.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:12.064764+0000) 2022-04-04T01:40:12.103 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:12.102+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:12.850 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:12.848+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:13.065 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:13.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:13.064958+0000) 2022-04-04T01:40:13.142 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:13.140+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:13.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:13.841+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:14.066 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:14.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:14.065196+0000) 2022-04-04T01:40:14.176 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:14.174+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:14.665 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:40:14.663+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:40:14.857 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:14.854+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:15.066 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:15.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:15.065450+0000) 2022-04-04T01:40:15.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:15.157+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:15.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:15.833+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:16.066 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:16.064+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:16.065687+0000) 2022-04-04T01:40:16.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:16.192+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:16.808 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:16.807+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:17.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:17.065+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:17.065823+0000) 2022-04-04T01:40:17.221 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:17.220+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:17.857 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:17.855+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:18.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:18.065+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:18.065979+0000) 2022-04-04T01:40:18.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:18.236+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 26 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:18.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:18.853+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:19.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:19.065+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:19.066183+0000) 2022-04-04T01:40:19.217 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:19.215+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:19.812 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:19.810+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:20.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:20.065+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:20.066386+0000) 2022-04-04T01:40:20.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:20.263+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:20.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:20.769+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:21.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:21.065+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:21.066581+0000) 2022-04-04T01:40:21.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:21.290+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:21.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:21.724+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:21.785 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:40:21.789 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:40:22.067 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:22.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:22.066849+0000) 2022-04-04T01:40:22.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:22.261+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:22.760 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:22.757+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:23.068 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:23.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:23.067067+0000) 2022-04-04T01:40:23.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:23.214+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:23.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:23.778+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:24.068 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:24.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:24.067264+0000) 2022-04-04T01:40:24.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:24.218+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:24.780 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:24.779+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:25.068 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:25.066+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:25.067485+0000) 2022-04-04T01:40:25.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:25.209+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:25.809 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:25.808+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:26.068 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:26.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:26.067714+0000) 2022-04-04T01:40:26.167 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:26.165+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:26.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:26.787+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:27.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:27.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:27.067944+0000) 2022-04-04T01:40:27.217 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:27.215+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:27.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:27.819+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:28.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:28.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:28.068152+0000) 2022-04-04T01:40:28.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:28.174+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:28.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:28.831+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:29.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:29.067+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:29.068328+0000) 2022-04-04T01:40:29.154 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:29.153+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:29.681 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:40:29.679+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:40:29.874 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:29.872+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:30.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:30.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:30.068524+0000) 2022-04-04T01:40:30.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:30.188+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:30.894 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:30.893+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:31.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:31.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:31.068699+0000) 2022-04-04T01:40:31.181 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:31.180+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:31.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:31.859+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:32.069 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:32.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:32.068887+0000) 2022-04-04T01:40:32.206 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:32.205+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:32.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:32.864+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:33.070 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:33.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:33.069096+0000) 2022-04-04T01:40:33.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:33.231+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:33.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:33.839+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:34.070 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:34.068+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:34.069293+0000) 2022-04-04T01:40:34.206 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:34.205+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:34.860 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:34.859+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:35.070 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:35.069+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:35.069480+0000) 2022-04-04T01:40:35.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:35.249+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:35.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:35.810+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:36.070 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:36.069+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:36.069693+0000) 2022-04-04T01:40:36.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:36.266+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:36.795 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:36.794+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:37.070 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:37.069+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:37.069884+0000) 2022-04-04T01:40:37.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:37.269+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:37.773 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:37.772+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:38.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:38.069+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:38.070064+0000) 2022-04-04T01:40:38.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:38.297+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:38.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:38.793+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:39.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:39.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:39.070224+0000) 2022-04-04T01:40:39.297 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:39.296+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:39.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:39.770+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:40.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:40.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:40.070426+0000) 2022-04-04T01:40:40.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:40.305+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:40.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:40.768+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:41.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:41.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:41.070677+0000) 2022-04-04T01:40:41.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:41.284+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:41.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:41.727+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:42.071 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:42.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:42.070906+0000) 2022-04-04T01:40:42.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:42.316+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:42.752 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:42.750+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:43.072 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:43.071+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:43.071091+0000) 2022-04-04T01:40:43.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:43.297+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:43.778 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:43.775+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:44.072 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:44.071+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:44.071287+0000) 2022-04-04T01:40:44.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:44.261+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:44.697 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:40:44.695+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:40:44.828 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:44.825+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:45.072 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:45.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:45.071515+0000) 2022-04-04T01:40:45.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:45.262+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:45.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:45.824+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:46.072 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:46.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:46.071736+0000) 2022-04-04T01:40:46.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:46.217+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:46.792 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:46.790+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:47.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:47.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:47.072002+0000) 2022-04-04T01:40:47.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:47.222+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:47.834 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:47.831+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:48.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:48.071+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:48.072227+0000) 2022-04-04T01:40:48.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:48.258+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:48.834 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:48.832+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:49.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:49.071+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:49.072448+0000) 2022-04-04T01:40:49.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:49.217+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:49.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:49.865+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:50.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:50.070+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:50.072649+0000) 2022-04-04T01:40:50.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:50.234+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:50.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:50.835+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:51.073 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:51.071+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:51.072896+0000) 2022-04-04T01:40:51.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:51.271+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:51.799 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:51.797+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:51.959 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:40:51.963 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:40:52.074 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:52.072+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:52.073155+0000) 2022-04-04T01:40:52.300 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:52.298+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:52.829 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:52.827+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:53.074 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:53.072+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:53.073354+0000) 2022-04-04T01:40:53.330 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:53.328+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:53.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:53.851+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:54.074 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:54.072+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:54.073535+0000) 2022-04-04T01:40:54.337 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:54.335+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:54.863 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:54.861+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:55.074 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:55.072+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:55.073703+0000) 2022-04-04T01:40:55.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:55.322+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:55.830 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:55.828+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:56.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:56.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:56.073884+0000) 2022-04-04T01:40:56.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:56.308+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:56.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:56.792+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:57.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:57.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:57.074072+0000) 2022-04-04T01:40:57.323 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:57.321+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:57.746 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:57.744+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:58.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:58.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:58.074248+0000) 2022-04-04T01:40:58.362 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:58.360+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:58.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:58.769+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:40:59.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:40:59.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:40:59.074425+0000) 2022-04-04T01:40:59.387 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:40:59.384+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:40:59.713 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:40:59.711+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:40:59.805 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:40:59.803+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:00.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:00.073+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:00.074635+0000) 2022-04-04T01:41:00.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:00.408+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:00.773 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:00.771+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:01.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:01.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:01.074849+0000) 2022-04-04T01:41:01.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:01.377+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:01.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:01.781+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:02.075 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:02.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:02.075028+0000) 2022-04-04T01:41:02.428 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:02.426+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:02.832 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:02.830+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:03.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:03.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:03.075254+0000) 2022-04-04T01:41:03.403 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:03.400+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:03.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:03.856+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:04.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:04.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:04.075429+0000) 2022-04-04T01:41:04.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:04.414+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:04.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:04.888+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:05.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:05.074+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:05.075525+0000) 2022-04-04T01:41:05.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:05.383+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:05.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:05.935+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:06.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:06.075+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:06.075679+0000) 2022-04-04T01:41:06.424 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:06.422+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:06.916 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:06.914+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:07.076 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:07.075+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:07.075874+0000) 2022-04-04T01:41:07.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:07.393+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:07.946 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:07.944+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:08.077 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:08.075+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:08.076094+0000) 2022-04-04T01:41:08.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:08.352+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:08.978 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:08.976+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:09.077 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:09.075+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:09.076295+0000) 2022-04-04T01:41:09.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:09.393+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:10.007 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:10.005+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:10.077 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:10.075+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:10.076536+0000) 2022-04-04T01:41:10.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:10.431+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:11.056 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:11.054+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:11.077 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:11.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:11.076732+0000) 2022-04-04T01:41:11.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:11.433+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:12.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:12.005+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:12.077 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:12.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:12.076946+0000) 2022-04-04T01:41:12.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:12.425+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:12.970 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:12.968+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:13.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:13.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:13.077154+0000) 2022-04-04T01:41:13.470 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:13.467+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:13.975 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:13.973+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:14.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:14.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:14.077318+0000) 2022-04-04T01:41:14.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:14.446+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:14.728 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:41:14.726+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:41:14.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:14.931+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:15.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:15.077+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:15.077466+0000) 2022-04-04T01:41:15.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:15.492+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:15.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:15.974+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:16.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:16.076+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:16.077605+0000) 2022-04-04T01:41:16.480 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:16.478+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:17.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:17.010+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:17.078 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:17.077+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:17.077767+0000) 2022-04-04T01:41:17.475 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:17.473+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:17.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:17.986+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:18.079 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:18.077+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:18.078015+0000) 2022-04-04T01:41:18.431 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:18.428+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:18.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:18.972+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:19.079 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:19.077+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:19.078170+0000) 2022-04-04T01:41:19.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:19.391+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:19.930 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:19.929+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:20.079 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:20.078+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:20.078347+0000) 2022-04-04T01:41:20.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:20.350+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:20.926 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:20.924+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:21.079 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:21.078+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:21.078533+0000) 2022-04-04T01:41:21.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:21.379+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:21.929 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:21.927+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:22.079 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:22.078+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:22.078697+0000) 2022-04-04T01:41:22.173 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:41:22.177 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:41:22.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:22.340+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:22.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:22.926+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:23.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:23.078+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:23.078904+0000) 2022-04-04T01:41:23.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:23.305+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:23.919 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:23.918+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:24.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:24.078+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:24.079076+0000) 2022-04-04T01:41:24.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:24.280+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:24.947 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:24.945+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:25.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:25.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:25.079253+0000) 2022-04-04T01:41:25.284 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:25.282+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:25.943 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:25.942+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:26.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:26.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:26.079408+0000) 2022-04-04T01:41:26.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:26.301+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:26.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:26.903+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:27.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:27.078+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:27.079573+0000) 2022-04-04T01:41:27.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:27.297+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:27.868 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:27.866+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:28.080 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:28.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:28.079798+0000) 2022-04-04T01:41:28.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:28.324+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:28.904 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:28.903+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:29.081 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:29.078+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:29.079972+0000) 2022-04-04T01:41:29.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:29.309+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:29.737 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:41:29.734+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:41:29.946 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:29.945+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:30.081 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:30.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:30.080154+0000) 2022-04-04T01:41:30.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:30.277+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:30.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:30.978+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:31.081 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:31.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:31.080365+0000) 2022-04-04T01:41:31.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:31.258+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:31.967 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:31.965+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:32.081 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:32.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:32.080570+0000) 2022-04-04T01:41:32.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:32.265+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:33.006 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:33.004+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:33.081 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:33.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:33.080738+0000) 2022-04-04T01:41:33.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:33.236+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:34.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:34.023+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:34.081 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:34.079+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:34.080921+0000) 2022-04-04T01:41:34.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:34.214+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:35.052 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:35.050+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:35.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:35.080+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:35.081118+0000) 2022-04-04T01:41:35.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:35.217+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:36.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:36.060+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:36.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:36.080+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:36.081286+0000) 2022-04-04T01:41:36.184 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:36.181+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:37.030 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:37.028+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:37.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:37.080+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:37.081413+0000) 2022-04-04T01:41:37.189 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:37.186+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:38.014 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:38.012+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:38.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:38.080+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:38.081553+0000) 2022-04-04T01:41:38.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:38.220+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:38.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:38.997+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:39.082 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:39.080+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:39.081739+0000) 2022-04-04T01:41:39.183 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:39.181+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:39.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:39.996+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:40.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:40.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:40.081946+0000) 2022-04-04T01:41:40.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:40.135+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:40.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:40.989+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:41.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:41.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:41.082098+0000) 2022-04-04T01:41:41.124 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:41.122+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:41.961 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:41.959+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:42.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:42.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:42.082254+0000) 2022-04-04T01:41:42.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:42.166+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:42.987 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:42.985+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:43.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:43.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:43.082410+0000) 2022-04-04T01:41:43.125 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:43.123+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:44.021 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:44.019+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:44.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:44.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:44.082595+0000) 2022-04-04T01:41:44.085 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:44.083+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:44.753 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:41:44.751+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:41:44.989 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:44.986+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:45.083 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:45.081+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:45.082708+0000) 2022-04-04T01:41:45.111 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:45.109+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:46.031 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:46.029+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:46.084 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:46.082+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:46.082858+0000) 2022-04-04T01:41:46.102 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:46.100+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:47.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:47.052+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:47.084 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:47.082+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:47.083058+0000) 2022-04-04T01:41:47.133 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:47.130+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:48.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:48.024+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:48.084 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:48.082+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:48.083241+0000) 2022-04-04T01:41:48.129 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:48.127+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:48.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:48.983+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:49.084 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:49.082+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:49.083382+0000) 2022-04-04T01:41:49.169 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:49.167+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:49.971 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:49.969+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:50.084 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:50.082+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:50.083507+0000) 2022-04-04T01:41:50.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:50.177+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:50.941 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:50.939+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:51.084 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:51.082+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:51.083684+0000) 2022-04-04T01:41:51.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:51.172+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:51.939 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:51.936+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:52.084 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:52.083+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:52.083888+0000) 2022-04-04T01:41:52.197 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:52.195+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:52.243 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:41:52.247 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:41:52.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:52.951+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:53.085 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:53.083+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:53.084072+0000) 2022-04-04T01:41:53.234 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:53.231+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:53.961 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:53.959+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:54.085 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:54.083+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:54.084261+0000) 2022-04-04T01:41:54.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:54.198+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:54.941 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:54.939+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:55.085 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:55.083+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:55.084371+0000) 2022-04-04T01:41:55.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:55.225+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:55.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:55.945+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:56.085 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:56.084+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:56.084559+0000) 2022-04-04T01:41:56.270 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:56.268+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:56.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:56.922+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:57.085 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:57.084+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:57.084801+0000) 2022-04-04T01:41:57.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:57.249+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:57.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:57.886+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:58.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:58.084+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:58.084940+0000) 2022-04-04T01:41:58.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:58.264+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:58.857 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:58.855+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:41:59.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:41:59.084+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:41:59.085073+0000) 2022-04-04T01:41:59.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:41:59.266+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:41:59.769 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:41:59.767+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:41:59.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:41:59.874+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:00.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:00.084+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:00.085265+0000) 2022-04-04T01:42:00.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:00.247+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:00.912 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:00.910+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:01.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:01.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:01.085451+0000) 2022-04-04T01:42:01.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:01.252+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:01.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:01.946+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:02.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:02.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:02.085602+0000) 2022-04-04T01:42:02.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:02.221+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:02.935 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:02.933+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:03.086 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:03.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:03.085823+0000) 2022-04-04T01:42:03.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:03.238+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:03.950 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:03.948+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:04.087 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:04.085+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:04.086047+0000) 2022-04-04T01:42:04.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:04.263+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:04.923 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:04.921+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:05.087 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:05.086+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:05.086236+0000) 2022-04-04T01:42:05.292 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:05.290+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:05.905 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:05.903+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:06.087 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:06.086+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:06.086376+0000) 2022-04-04T01:42:06.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:06.294+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:06.891 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:06.889+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:07.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:07.086+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:07.086576+0000) 2022-04-04T01:42:07.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:07.286+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:07.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:07.848+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:08.087 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:08.086+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:08.086754+0000) 2022-04-04T01:42:08.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:08.321+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:08.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:08.886+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:09.087 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:09.086+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:09.087002+0000) 2022-04-04T01:42:09.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:09.277+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:09.919 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:09.917+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:10.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:10.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:10.087186+0000) 2022-04-04T01:42:10.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:10.313+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:10.891 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:10.889+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:11.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:11.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:11.087381+0000) 2022-04-04T01:42:11.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:11.345+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:11.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:11.887+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:12.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:12.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:12.087505+0000) 2022-04-04T01:42:12.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:12.317+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:12.921 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:12.920+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:13.088 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:13.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:13.087738+0000) 2022-04-04T01:42:13.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:13.353+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:13.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:13.948+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:14.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:14.086+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:14.087983+0000) 2022-04-04T01:42:14.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:14.371+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:14.784 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:42:14.782+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:42:14.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:14.947+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:15.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:15.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:15.088154+0000) 2022-04-04T01:42:15.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:15.419+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:15.934 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:15.933+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:16.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:16.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:16.088291+0000) 2022-04-04T01:42:16.371 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:16.370+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:16.976 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:16.974+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:17.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:17.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:17.088403+0000) 2022-04-04T01:42:17.332 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:17.330+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:17.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:17.947+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:18.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:18.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:18.088563+0000) 2022-04-04T01:42:18.342 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:18.341+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:18.944 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:18.942+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:19.089 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:19.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:19.088702+0000) 2022-04-04T01:42:19.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:19.361+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:19.925 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:19.924+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:20.090 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:20.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:20.088884+0000) 2022-04-04T01:42:20.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:20.385+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:20.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:20.963+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:21.090 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:21.087+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:21.089084+0000) 2022-04-04T01:42:21.339 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:21.338+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:21.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:21.940+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:22.093 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:22.088+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:22.089238+0000) 2022-04-04T01:42:22.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:22.354+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:22.376 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:42:22.381 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:42:22.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:22.972+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:23.090 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:23.088+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:23.089411+0000) 2022-04-04T01:42:23.338 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:23.336+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:23.945 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:23.944+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:24.090 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:24.088+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:24.089635+0000) 2022-04-04T01:42:24.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:24.379+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:24.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:24.951+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:25.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:25.089+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:25.089854+0000) 2022-04-04T01:42:25.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:25.378+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:25.945 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:25.943+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:26.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:26.089+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:26.090028+0000) 2022-04-04T01:42:26.426 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:26.425+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:26.980 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:26.979+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:27.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:27.089+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:27.090223+0000) 2022-04-04T01:42:27.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:27.397+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:28.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:28.025+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:28.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:28.089+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:28.090395+0000) 2022-04-04T01:42:28.449 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:28.448+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:29.054 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:29.052+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:29.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:29.089+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:29.090632+0000) 2022-04-04T01:42:29.415 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:29.414+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:29.800 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:42:29.798+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:42:30.068 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:30.066+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:30.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:30.090+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:30.090834+0000) 2022-04-04T01:42:30.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:30.376+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:31.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:31.086+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:31.091 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:31.090+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:31.091003+0000) 2022-04-04T01:42:31.391 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:31.388+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:32.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:32.071+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:32.092 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:32.090+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:32.091194+0000) 2022-04-04T01:42:32.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:32.409+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:33.052 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:33.050+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:33.092 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:33.090+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:33.091447+0000) 2022-04-04T01:42:33.422 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:33.420+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:34.029 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:34.027+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:34.092 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:34.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:34.091619+0000) 2022-04-04T01:42:34.437 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:34.435+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:35.053 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:35.050+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:35.092 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:35.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:35.091833+0000) 2022-04-04T01:42:35.482 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:35.480+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:36.086 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:36.084+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:36.092 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:36.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:36.092029+0000) 2022-04-04T01:42:36.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:36.507+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:37.073 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:37.071+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:37.093 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:37.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:37.092219+0000) 2022-04-04T01:42:37.474 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:37.472+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:38.055 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:38.052+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:38.093 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:38.091+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:38.092461+0000) 2022-04-04T01:42:38.486 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:38.483+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:39.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:39.061+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:39.093 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:39.092+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:39.092675+0000) 2022-04-04T01:42:39.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:39.534+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:40.069 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:40.066+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:40.094 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:40.092+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:40.092899+0000) 2022-04-04T01:42:40.583 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:40.581+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:41.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:41.022+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:41.094 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:41.092+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:41.093119+0000) 2022-04-04T01:42:41.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:41.569+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:42.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:42.072+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:42.094 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:42.092+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:42.093336+0000) 2022-04-04T01:42:42.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:42.594+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:43.094 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:43.093+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:43.093545+0000) 2022-04-04T01:42:43.122 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:43.119+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:43.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:43.559+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:44.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:44.076+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:44.094 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:44.093+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:44.093734+0000) 2022-04-04T01:42:44.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:44.511+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:44.814 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:42:44.812+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:42:45.065 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:45.063+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:45.095 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:45.093+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:45.093976+0000) 2022-04-04T01:42:45.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:45.535+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:46.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:46.060+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:46.095 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:46.093+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:46.094119+0000) 2022-04-04T01:42:46.572 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:46.569+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:47.095 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:47.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:47.094349+0000) 2022-04-04T01:42:47.096 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:47.094+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:47.546 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:47.544+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:48.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:48.077+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:48.095 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:48.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:48.094568+0000) 2022-04-04T01:42:48.511 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:48.508+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:49.095 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:49.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:49.094829+0000) 2022-04-04T01:42:49.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:49.122+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:49.546 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:49.544+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:50.096 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:50.094+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:50.095020+0000) 2022-04-04T01:42:50.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:50.134+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:50.581 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:50.579+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:51.093 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:51.090+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:51.096 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:51.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:51.095243+0000) 2022-04-04T01:42:51.596 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:51.593+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:52.096 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:52.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:52.095458+0000) 2022-04-04T01:42:52.123 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:52.121+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:52.524 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:42:52.528 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:42:52.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:52.596+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:53.097 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:53.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:53.095721+0000) 2022-04-04T01:42:53.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:53.129+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:53.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:53.644+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:54.097 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:54.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:54.095929+0000) 2022-04-04T01:42:54.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:54.145+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:54.673 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:54.670+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:55.097 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:55.096+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:55.096163+0000) 2022-04-04T01:42:55.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:55.145+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:55.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:55.711+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:56.097 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:56.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:56.096353+0000) 2022-04-04T01:42:56.168 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:56.166+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:56.688 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:56.686+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:57.097 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:57.096+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:57.096494+0000) 2022-04-04T01:42:57.119 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:57.117+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:57.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:57.729+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:58.097 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:58.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:58.096622+0000) 2022-04-04T01:42:58.110 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:58.109+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:58.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:58.769+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:59.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:42:59.077+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:42:59.097 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:42:59.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:42:59.096824+0000) 2022-04-04T01:42:59.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:42:59.748+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:42:59.830 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:42:59.827+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:43:00.086 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:00.084+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:00.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:00.096+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:00.097044+0000) 2022-04-04T01:43:00.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:00.714+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:01.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:01.096+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:01.097267+0000) 2022-04-04T01:43:01.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:01.125+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:01.708 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:01.706+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:02.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:02.086+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:02.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:02.095+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:02.097484+0000) 2022-04-04T01:43:02.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:02.667+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:03.093 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:03.092+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:03.098 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:03.096+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:03.097721+0000) 2022-04-04T01:43:03.674 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:03.672+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:04.099 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:04.097+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:04.097916+0000) 2022-04-04T01:43:04.117 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:04.115+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:04.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:04.705+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:05.099 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:05.097+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:05.098108+0000) 2022-04-04T01:43:05.156 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:05.154+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:05.721 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:05.719+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:06.099 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:06.097+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:06.098330+0000) 2022-04-04T01:43:06.145 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:06.143+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:06.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:06.680+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:07.099 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:07.097+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:07.098544+0000) 2022-04-04T01:43:07.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:07.152+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:07.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:07.678+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:08.099 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:08.098+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:08.098763+0000) 2022-04-04T01:43:08.108 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:08.106+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:08.638 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:08.636+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:09.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:09.098+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:09.099014+0000) 2022-04-04T01:43:09.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:09.142+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:09.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:09.662+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:10.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:10.098+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:10.099203+0000) 2022-04-04T01:43:10.125 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:10.124+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:10.672 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:10.671+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:11.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:11.098+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:11.099421+0000) 2022-04-04T01:43:11.135 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:11.134+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:11.643 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:11.641+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:12.093 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:12.092+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:12.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:12.098+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:12.099633+0000) 2022-04-04T01:43:12.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:12.604+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:13.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:13.045+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:13.100 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:13.099+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:13.099878+0000) 2022-04-04T01:43:13.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:13.561+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:14.079 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:14.077+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:14.101 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:14.099+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:14.100065+0000) 2022-04-04T01:43:14.559 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:14.558+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:14.845 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:43:14.844+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:43:15.064 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:15.063+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:15.101 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:15.099+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:15.100194+0000) 2022-04-04T01:43:15.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:15.561+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:16.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:16.047+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:16.101 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:16.099+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:16.100355+0000) 2022-04-04T01:43:16.544 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:16.543+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:17.018 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:17.016+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:17.101 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:17.099+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:17.100569+0000) 2022-04-04T01:43:17.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:17.537+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:18.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:18.012+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:18.101 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:18.100+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:18.100801+0000) 2022-04-04T01:43:18.585 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:18.583+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:19.009 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:19.008+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:19.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:19.100+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:19.101027+0000) 2022-04-04T01:43:19.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:19.615+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:20.042 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:20.041+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:20.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:20.100+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:20.101226+0000) 2022-04-04T01:43:20.572 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:20.571+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:21.006 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:21.004+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:21.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:21.100+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:21.101443+0000) 2022-04-04T01:43:21.566 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:21.565+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:22.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:22.014+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:22.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:22.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:22.101658+0000) 2022-04-04T01:43:22.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:22.553+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:22.668 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:43:22.672 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:43:23.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:23.015+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:23.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:23.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:23.101923+0000) 2022-04-04T01:43:23.538 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:23.537+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:24.042 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:24.039+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:24.102 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:24.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:24.102085+0000) 2022-04-04T01:43:24.584 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:24.582+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:25.089 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:25.087+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:25.103 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:25.101+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:25.102319+0000) 2022-04-04T01:43:25.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:25.618+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:26.069 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:26.067+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:26.103 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:26.102+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:26.102482+0000) 2022-04-04T01:43:26.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:26.652+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:27.058 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:27.055+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:27.103 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:27.102+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:27.102678+0000) 2022-04-04T01:43:27.633 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:27.631+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:28.087 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:28.084+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:28.103 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:28.102+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:28.102834+0000) 2022-04-04T01:43:28.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:28.660+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:29.075 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:29.073+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:29.103 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:29.102+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:29.103030+0000) 2022-04-04T01:43:29.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:29.618+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:29.862 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:43:29.860+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:43:30.034 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:30.032+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:30.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:30.102+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:30.103246+0000) 2022-04-04T01:43:30.580 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:30.577+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:31.025 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:31.023+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:31.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:31.103+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:31.103474+0000) 2022-04-04T01:43:31.616 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:31.614+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:32.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:32.005+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:32.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:32.103+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:32.103687+0000) 2022-04-04T01:43:32.602 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:32.600+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:33.051 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:33.048+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:33.104 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:33.103+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:33.103905+0000) 2022-04-04T01:43:33.577 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:33.575+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:34.080 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:34.078+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:34.105 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:34.103+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:34.104120+0000) 2022-04-04T01:43:34.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:34.545+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:35.070 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:35.068+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:35.105 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:35.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:35.104363+0000) 2022-04-04T01:43:35.586 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:35.584+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:36.053 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:36.051+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:36.105 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:36.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:36.104583+0000) 2022-04-04T01:43:36.636 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:36.634+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:37.091 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:37.089+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:37.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:37.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:37.104832+0000) 2022-04-04T01:43:37.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:37.610+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:38.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:38.071+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:38.105 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:38.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:38.104991+0000) 2022-04-04T01:43:38.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:38.610+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:39.098 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:39.096+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:39.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:39.105+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:39.105181+0000) 2022-04-04T01:43:39.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:39.624+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:40.091 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:40.089+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:40.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:40.105+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:40.105396+0000) 2022-04-04T01:43:40.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:40.629+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:41.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:41.064+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:41.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:41.105+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:41.105561+0000) 2022-04-04T01:43:41.661 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:41.659+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:42.077 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:42.074+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:42.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:42.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:42.105699+0000) 2022-04-04T01:43:42.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:42.706+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:43.106 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:43.104+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:43.105872+0000) 2022-04-04T01:43:43.119 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:43.117+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:43.706 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:43.703+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:44.107 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:44.105+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:44.106066+0000) 2022-04-04T01:43:44.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:44.119+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:44.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:44.720+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:44.878 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:43:44.875+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:43:45.093 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:45.091+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:45.107 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:45.105+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:45.106292+0000) 2022-04-04T01:43:45.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:45.714+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:46.107 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:46.105+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:46.106493+0000) 2022-04-04T01:43:46.112 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:46.110+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:46.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:46.745+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:47.107 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:47.105+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:47.106727+0000) 2022-04-04T01:43:47.144 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:47.142+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:47.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:47.778+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:48.108 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:48.106+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:48.106995+0000) 2022-04-04T01:43:48.172 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:48.170+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:48.772 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:48.770+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:49.108 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:49.106+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:49.107200+0000) 2022-04-04T01:43:49.181 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:49.179+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:49.733 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:49.731+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:50.108 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:50.106+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:50.107400+0000) 2022-04-04T01:43:50.223 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:50.221+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:50.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:50.711+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:51.108 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:51.106+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:51.107601+0000) 2022-04-04T01:43:51.247 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:51.245+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:51.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:51.673+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:52.108 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:52.107+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:52.107838+0000) 2022-04-04T01:43:52.215 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:52.214+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:52.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:52.684+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:52.810 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:43:52.814 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:43:53.109 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:53.107+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:53.108113+0000) 2022-04-04T01:43:53.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:53.176+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:53.668 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:53.667+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:54.109 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:54.107+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:54.108316+0000) 2022-04-04T01:43:54.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:54.180+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:54.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:54.684+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:55.109 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:55.107+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:55.108547+0000) 2022-04-04T01:43:55.144 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:55.142+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:55.655 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:55.653+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:56.109 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:56.108+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:56.108812+0000) 2022-04-04T01:43:56.166 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:56.163+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:56.623 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:56.621+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:57.110 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:57.108+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:57.109058+0000) 2022-04-04T01:43:57.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:57.205+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:57.597 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:57.595+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:58.110 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:58.108+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:58.109248+0000) 2022-04-04T01:43:58.199 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:58.197+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:58.553 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:58.550+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:59.110 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:43:59.108+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:43:59.109436+0000) 2022-04-04T01:43:59.208 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:43:59.206+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:43:59.504 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:43:59.502+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:43:59.892 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:43:59.890+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:44:00.110 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:00.108+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:00.109660+0000) 2022-04-04T01:44:00.188 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:00.187+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:00.475 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:00.473+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:01.110 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:01.109+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:01.109859+0000) 2022-04-04T01:44:01.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:01.196+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:01.511 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:01.509+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:02.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:02.109+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:02.110085+0000) 2022-04-04T01:44:02.247 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:02.246+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:02.470 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:02.468+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:03.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:03.109+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:03.110272+0000) 2022-04-04T01:44:03.223 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:03.222+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:03.454 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:03.453+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:04.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:04.109+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:04.110469+0000) 2022-04-04T01:44:04.192 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:04.191+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:04.483 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:04.482+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:05.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:05.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:05.110622+0000) 2022-04-04T01:44:05.172 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:05.171+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:05.528 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:05.527+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:06.111 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:06.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:06.110835+0000) 2022-04-04T01:44:06.201 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:06.200+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:06.493 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:06.491+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:07.112 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:07.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:07.111072+0000) 2022-04-04T01:44:07.217 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:07.216+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:07.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:07.494+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:08.112 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:08.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:08.111298+0000) 2022-04-04T01:44:08.192 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:08.190+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:08.450 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:08.448+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:09.112 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:09.110+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:09.111535+0000) 2022-04-04T01:44:09.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:09.188+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:09.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:09.451+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:10.112 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:10.111+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:10.111704+0000) 2022-04-04T01:44:10.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:10.168+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:10.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:10.447+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:11.112 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:11.111+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:11.111901+0000) 2022-04-04T01:44:11.136 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:11.134+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:11.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:11.492+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:12.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:12.102+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:12.113 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:12.111+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:12.112122+0000) 2022-04-04T01:44:12.472 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:12.470+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:13.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:13.075+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:13.113 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:13.111+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:13.112312+0000) 2022-04-04T01:44:13.473 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:13.472+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:14.053 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:14.052+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:14.113 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:14.112+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:14.112547+0000) 2022-04-04T01:44:14.523 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:14.522+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:14.908 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:44:14.906+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:44:15.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:15.014+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:15.113 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:15.112+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:15.112735+0000) 2022-04-04T01:44:15.512 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:15.511+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:15.972 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:15.971+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:16.113 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:16.112+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:16.112951+0000) 2022-04-04T01:44:16.557 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:16.556+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:16.962 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:16.961+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:17.114 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:17.112+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:17.113105+0000) 2022-04-04T01:44:17.560 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:17.558+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:17.996 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:17.993+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:18.114 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:18.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:18.113266+0000) 2022-04-04T01:44:18.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:18.542+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:18.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:18.951+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:19.114 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:19.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:19.113430+0000) 2022-04-04T01:44:19.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:19.513+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:19.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:19.989+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:20.114 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:20.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:20.113591+0000) 2022-04-04T01:44:20.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:20.528+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:20.985 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:20.982+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:21.114 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:21.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:21.113802+0000) 2022-04-04T01:44:21.509 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:21.506+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:21.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:21.972+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:22.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:22.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:22.114030+0000) 2022-04-04T01:44:22.550 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:22.547+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:22.949 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:44:22.953 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:44:22.975 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:22.973+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:23.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:23.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:23.114257+0000) 2022-04-04T01:44:23.513 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:23.511+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:24.012 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:24.009+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:24.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:24.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:24.114462+0000) 2022-04-04T01:44:24.497 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:24.495+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:24.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:24.986+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:25.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:25.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:25.114650+0000) 2022-04-04T01:44:25.532 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:25.530+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:26.022 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:26.019+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:26.115 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:26.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:26.114847+0000) 2022-04-04T01:44:26.545 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:26.542+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:27.031 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:27.029+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:27.116 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:27.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:27.115038+0000) 2022-04-04T01:44:27.582 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:27.579+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:27.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:27.981+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:28.116 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:28.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:28.115236+0000) 2022-04-04T01:44:28.628 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:28.626+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:28.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:28.949+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:29.116 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:29.113+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:29.115386+0000) 2022-04-04T01:44:29.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:29.674+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:29.924 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:44:29.921+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:44:29.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:29.971+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:30.116 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:30.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:30.115549+0000) 2022-04-04T01:44:30.637 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:30.634+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:30.998 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:30.996+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:31.116 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:31.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:31.115691+0000) 2022-04-04T01:44:31.647 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:31.645+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:31.969 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:31.966+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:32.116 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:32.114+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:32.115893+0000) 2022-04-04T01:44:32.622 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:32.620+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:32.956 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:32.954+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:33.117 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:33.115+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:33.116122+0000) 2022-04-04T01:44:33.598 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:33.596+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:34.001 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:33.999+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:34.117 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:34.115+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:34.116339+0000) 2022-04-04T01:44:34.624 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:34.622+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:34.958 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:34.956+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:35.117 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:35.115+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:35.116537+0000) 2022-04-04T01:44:35.622 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:35.619+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:35.984 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:35.981+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:36.117 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:36.115+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:36.116677+0000) 2022-04-04T01:44:36.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:36.667+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:36.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:36.962+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:37.117 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:37.116+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:37.116862+0000) 2022-04-04T01:44:37.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:37.667+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:37.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:37.992+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:38.117 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:38.116+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:38.117050+0000) 2022-04-04T01:44:38.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:38.651+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:38.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:38.949+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:39.118 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:39.116+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:39.117237+0000) 2022-04-04T01:44:39.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:39.684+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:39.996 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:39.994+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:40.118 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:40.116+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:40.117368+0000) 2022-04-04T01:44:40.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:40.688+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:40.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:40.975+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:41.118 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:41.116+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:41.117564+0000) 2022-04-04T01:44:41.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:41.716+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:41.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:41.930+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:42.118 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:42.117+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:42.117764+0000) 2022-04-04T01:44:42.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:42.749+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:42.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:42.975+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:43.118 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:43.117+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:43.118012+0000) 2022-04-04T01:44:43.788 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:43.786+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:43.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:43.988+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:44.119 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:44.117+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:44.118225+0000) 2022-04-04T01:44:44.827 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:44.824+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:44.939 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:44:44.938+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:44:45.036 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:45.034+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:45.119 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:45.117+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:45.118457+0000) 2022-04-04T01:44:45.849 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:45.847+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:46.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:46.008+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:46.119 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:46.118+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:46.118673+0000) 2022-04-04T01:44:46.888 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:46.886+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:47.035 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:47.033+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:47.119 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:47.118+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:47.118864+0000) 2022-04-04T01:44:47.859 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:47.857+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:48.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:48.006+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:48.119 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:48.118+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:48.119010+0000) 2022-04-04T01:44:48.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:48.831+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:49.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:49.044+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:49.120 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:49.118+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:49.119191+0000) 2022-04-04T01:44:49.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:49.830+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:50.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:50.080+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:50.120 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:50.118+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:50.119422+0000) 2022-04-04T01:44:50.845 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:50.842+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:51.089 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:51.088+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:51.120 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:51.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:51.119640+0000) 2022-04-04T01:44:51.830 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:51.828+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:52.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:52.064+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:52.120 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:52.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:52.119827+0000) 2022-04-04T01:44:52.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:52.841+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:53.063 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:53.062+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:53.105 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:44:53.109 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:44:53.120 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:53.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:53.119963+0000) 2022-04-04T01:44:53.851 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:53.849+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:54.021 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:54.019+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:54.121 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:54.119+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:54.120178+0000) 2022-04-04T01:44:54.881 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:54.879+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:55.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:55.037+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:55.121 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:55.120+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:55.120390+0000) 2022-04-04T01:44:55.888 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:55.886+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:55.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:55.995+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:56.121 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:56.120+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:56.120520+0000) 2022-04-04T01:44:56.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:56.931+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:56.981 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:56.979+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:57.121 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:57.120+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:57.120715+0000) 2022-04-04T01:44:57.884 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:57.883+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:57.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:57.938+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:58.121 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:58.120+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:58.120919+0000) 2022-04-04T01:44:58.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:58.867+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:58.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:58.941+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:59.122 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:44:59.120+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:44:59.121029+0000) 2022-04-04T01:44:59.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:44:59.862+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:44:59.910 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:44:59.909+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:44:59.955 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:44:59.954+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:45:00.122 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:00.120+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:00.121186+0000) 2022-04-04T01:45:00.907 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:00.906+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:00.922 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:00.920+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:01.122 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:01.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:01.121409+0000) 2022-04-04T01:45:01.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:01.867+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:01.882 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:01.881+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:02.122 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:02.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:02.121603+0000) 2022-04-04T01:45:02.826 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:02.825+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:02.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:02.923+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:03.122 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:03.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:03.121833+0000) 2022-04-04T01:45:03.875 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:03.874+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:03.917 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:03.915+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:04.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:04.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:04.122047+0000) 2022-04-04T01:45:04.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:04.830+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:04.959 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:04.958+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:05.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:05.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:05.122274+0000) 2022-04-04T01:45:05.803 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:05.801+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:05.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:05.952+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:06.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:06.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:06.122459+0000) 2022-04-04T01:45:06.852 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:06.850+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:06.995 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:06.994+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:07.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:07.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:07.122653+0000) 2022-04-04T01:45:07.803 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:07.801+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:07.978 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:07.977+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:08.123 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:08.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:08.122878+0000) 2022-04-04T01:45:08.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:08.833+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:09.017 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:09.015+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:09.124 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:09.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:09.123114+0000) 2022-04-04T01:45:09.814 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:09.813+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:10.011 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:10.010+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:10.124 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:10.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:10.123359+0000) 2022-04-04T01:45:10.852 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:10.850+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:10.972 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:10.971+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:11.124 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:11.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:11.123540+0000) 2022-04-04T01:45:11.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:11.828+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:11.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:11.951+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:12.124 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:12.121+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:12.123690+0000) 2022-04-04T01:45:12.830 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:12.827+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:12.987 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:12.985+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:13.124 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:13.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:13.123927+0000) 2022-04-04T01:45:13.838 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:13.835+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:13.940 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:13.937+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:14.125 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:14.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:14.124127+0000) 2022-04-04T01:45:14.835 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:14.833+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:14.971 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:45:14.968+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:45:14.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:14.986+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:15.125 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:15.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:15.124273+0000) 2022-04-04T01:45:15.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:15.827+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:15.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:15.946+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:16.125 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:16.122+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:16.124458+0000) 2022-04-04T01:45:16.799 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:16.797+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:16.909 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:16.907+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:17.125 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:17.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:17.124615+0000) 2022-04-04T01:45:17.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:17.803+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:17.949 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:17.946+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:18.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:18.123+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:18.124849+0000) 2022-04-04T01:45:18.804 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:18.801+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:18.913 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:18.910+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:19.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:19.124+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:19.125005+0000) 2022-04-04T01:45:19.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:19.830+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:19.905 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:19.903+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:20.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:20.124+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:20.125198+0000) 2022-04-04T01:45:20.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:20.829+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:20.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:20.928+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:21.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:21.124+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:21.125385+0000) 2022-04-04T01:45:21.792 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:21.790+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:21.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:21.962+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:22.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:22.124+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:22.125571+0000) 2022-04-04T01:45:22.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:22.745+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:22.980 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:22.978+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:23.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:23.125+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:23.125851+0000) 2022-04-04T01:45:23.229 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:45:23.232 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:45:23.727 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:23.725+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:23.968 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:23.966+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:24.126 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:24.125+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:24.126062+0000) 2022-04-04T01:45:24.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:24.684+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:24.951 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:24.949+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:25.127 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:25.125+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:25.126268+0000) 2022-04-04T01:45:25.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:25.692+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:25.959 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:25.957+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:26.127 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:26.125+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:26.126409+0000) 2022-04-04T01:45:26.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:26.660+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:26.956 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:26.954+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:27.127 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:27.125+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:27.126561+0000) 2022-04-04T01:45:27.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:27.616+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:27.974 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:27.972+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:28.127 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:28.126+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:28.126740+0000) 2022-04-04T01:45:28.649 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:28.646+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:28.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:28.940+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:29.127 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:29.126+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:29.126922+0000) 2022-04-04T01:45:29.610 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:29.608+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:29.979 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:29.977+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:29.987 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:45:29.985+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:45:30.128 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:30.126+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:30.127179+0000) 2022-04-04T01:45:30.642 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:30.640+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:30.959 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:30.957+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:31.128 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:31.126+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:31.127362+0000) 2022-04-04T01:45:31.686 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:31.683+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:31.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:31.988+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:32.128 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:32.126+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:32.127572+0000) 2022-04-04T01:45:32.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:32.721+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:33.021 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:33.019+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:33.128 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:33.127+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:33.127819+0000) 2022-04-04T01:45:33.765 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:33.763+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:33.988 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:33.986+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:34.129 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:34.127+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:34.127990+0000) 2022-04-04T01:45:34.759 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:34.757+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:35.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:35.022+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:35.129 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:35.127+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:35.128179+0000) 2022-04-04T01:45:35.756 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:35.754+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:36.024 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:36.021+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:36.129 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:36.127+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:36.128333+0000) 2022-04-04T01:45:36.720 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:36.717+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:37.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:37.013+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:37.129 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:37.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:37.128519+0000) 2022-04-04T01:45:37.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:37.694+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:38.002 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:38.000+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:38.129 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:38.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:38.128724+0000) 2022-04-04T01:45:38.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:38.689+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:39.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:39.017+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:39.129 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:39.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:39.128960+0000) 2022-04-04T01:45:39.701 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:39.699+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:40.051 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:40.049+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:40.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:40.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:40.129140+0000) 2022-04-04T01:45:40.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:40.717+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:41.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:41.014+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:41.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:41.128+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:41.129351+0000) 2022-04-04T01:45:41.697 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:41.695+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:41.992 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:41.990+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:42.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:42.129+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:42.129535+0000) 2022-04-04T01:45:42.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:42.674+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:43.028 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:43.026+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:43.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:43.129+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:43.129741+0000) 2022-04-04T01:45:43.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:43.716+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:44.010 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:44.008+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:44.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:44.129+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:44.129839+0000) 2022-04-04T01:45:44.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:44.677+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:44.996 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:45:44.994+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:45:45.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:45.036+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:45.130 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:45.129+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:45.130016+0000) 2022-04-04T01:45:45.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:45.693+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:45.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:45.988+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:46.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:46.129+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:46.130264+0000) 2022-04-04T01:45:46.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:46.680+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:46.972 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:46.971+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:47.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:47.130+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:47.130493+0000) 2022-04-04T01:45:47.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:47.713+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:47.999 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:47.997+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:48.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:48.130+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:48.130680+0000) 2022-04-04T01:45:48.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:48.673+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:49.038 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:49.036+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:49.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:49.130+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:49.130866+0000) 2022-04-04T01:45:49.716 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:49.714+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:50.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:50.047+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:50.131 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:50.130+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:50.131066+0000) 2022-04-04T01:45:50.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:50.724+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:51.096 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:51.094+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:51.132 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:51.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:51.131295+0000) 2022-04-04T01:45:51.725 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:51.724+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:52.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:52.046+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:52.132 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:52.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:52.131492+0000) 2022-04-04T01:45:52.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:52.727+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:53.097 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:53.095+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:53.132 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:53.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:53.131704+0000) 2022-04-04T01:45:53.577 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:45:53.580 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:45:53.776 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:53.775+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:54.067 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:54.066+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:54.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:54.130+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:54.131909+0000) 2022-04-04T01:45:54.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:54.749+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:55.046 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:55.045+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:55.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:55.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:55.132098+0000) 2022-04-04T01:45:55.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:55.706+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:56.014 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:56.012+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:56.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:56.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:56.132300+0000) 2022-04-04T01:45:56.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:56.745+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:56.997 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:56.996+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:57.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:57.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:57.132514+0000) 2022-04-04T01:45:57.784 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:57.783+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:58.015 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:58.014+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:58.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:58.131+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:58.132734+0000) 2022-04-04T01:45:58.794 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:58.793+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:45:58.977 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:45:58.976+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:45:59.133 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:45:59.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:45:59.132968+0000) 2022-04-04T01:45:59.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:45:59.786+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:00.012 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:46:00.009+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:46:00.014 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:00.013+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:00.134 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:00.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:00.133123+0000) 2022-04-04T01:46:00.795 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:00.793+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:00.976 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:00.975+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:01.134 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:01.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:01.133320+0000) 2022-04-04T01:46:01.794 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:01.793+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:01.964 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:01.963+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:02.134 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:02.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:02.133524+0000) 2022-04-04T01:46:02.805 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:02.804+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:02.990 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:02.989+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:03.134 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:03.132+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:03.133752+0000) 2022-04-04T01:46:03.836 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:03.835+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:03.993 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:03.992+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:04.135 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:04.133+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:04.133991+0000) 2022-04-04T01:46:04.815 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:04.814+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:04.976 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:04.970+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:05.135 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:05.133+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:05.134208+0000) 2022-04-04T01:46:05.817 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:05.815+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:05.945 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:05.943+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:06.135 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:06.133+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:06.134374+0000) 2022-04-04T01:46:06.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:06.779+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:06.952 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:06.950+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:07.135 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:07.133+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:07.134562+0000) 2022-04-04T01:46:07.798 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:07.796+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:07.975 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:07.973+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:08.135 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:08.134+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:08.134743+0000) 2022-04-04T01:46:08.775 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:08.773+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:08.939 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:08.937+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:09.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:09.134+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:09.134927+0000) 2022-04-04T01:46:09.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:09.758+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:09.899 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:09.897+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:10.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:10.134+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:10.135116+0000) 2022-04-04T01:46:10.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:10.768+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:10.858 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:10.856+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:11.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:11.134+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:11.135324+0000) 2022-04-04T01:46:11.780 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:11.778+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:11.862 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:11.859+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:12.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:12.134+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:12.135544+0000) 2022-04-04T01:46:12.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:12.740+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:12.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:12.875+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:13.136 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:13.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:13.135790+0000) 2022-04-04T01:46:13.732 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:13.730+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:13.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:13.840+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:14.137 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:14.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:14.135998+0000) 2022-04-04T01:46:14.723 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:14.721+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:14.852 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:14.850+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:15.028 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:46:15.026+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:46:15.137 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:15.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:15.136225+0000) 2022-04-04T01:46:15.708 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:15.705+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:15.902 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:15.900+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:16.137 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:16.135+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:16.136401+0000) 2022-04-04T01:46:16.706 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:16.704+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:16.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:16.931+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:17.137 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:17.136+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:17.136586+0000) 2022-04-04T01:46:17.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:17.679+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:17.948 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:17.946+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:18.137 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:18.136+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:18.136821+0000) 2022-04-04T01:46:18.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:18.719+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:18.906 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:18.903+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:19.138 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:19.136+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:19.137023+0000) 2022-04-04T01:46:19.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:19.679+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:19.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:19.931+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:20.138 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:20.136+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:20.137242+0000) 2022-04-04T01:46:20.717 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:20.715+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:20.973 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:20.971+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:21.138 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:21.137+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:21.137477+0000) 2022-04-04T01:46:21.676 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:21.674+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:21.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:21.925+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:22.138 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:22.137+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:22.137667+0000) 2022-04-04T01:46:22.689 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:22.687+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:22.911 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:22.909+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:23.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:23.137+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:23.137878+0000) 2022-04-04T01:46:23.657 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:46:23.661 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:46:23.693 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:23.691+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:23.876 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:23.874+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:24.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:24.137+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:24.138059+0000) 2022-04-04T01:46:24.699 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:24.696+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:24.899 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:24.897+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:25.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:25.137+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:25.138242+0000) 2022-04-04T01:46:25.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:25.652+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:25.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:25.851+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:26.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:26.138+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:26.138457+0000) 2022-04-04T01:46:26.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:26.638+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:26.813 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:26.811+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:27.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:27.138+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:27.138698+0000) 2022-04-04T01:46:27.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:27.638+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:27.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:27.777+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:28.139 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:28.138+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:28.138866+0000) 2022-04-04T01:46:28.641 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:28.639+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:28.763 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:28.761+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:29.140 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:29.138+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:29.139058+0000) 2022-04-04T01:46:29.648 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:29.646+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:29.734 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:29.732+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:30.044 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:46:30.042+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:46:30.140 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:30.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:30.139288+0000) 2022-04-04T01:46:30.633 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:30.631+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:30.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:30.689+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:31.140 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:31.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:31.139502+0000) 2022-04-04T01:46:31.660 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:31.658+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:31.716 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:31.714+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:32.140 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:32.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:32.139723+0000) 2022-04-04T01:46:32.620 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:32.618+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:32.703 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:32.701+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:33.141 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:33.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:33.139973+0000) 2022-04-04T01:46:33.612 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:33.610+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:33.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:33.748+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:34.141 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:34.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:34.140191+0000) 2022-04-04T01:46:34.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:34.617+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:34.759 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:34.757+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:35.141 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:35.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:35.140411+0000) 2022-04-04T01:46:35.659 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:35.657+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:35.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:35.736+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:36.141 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:36.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:36.140629+0000) 2022-04-04T01:46:36.709 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:36.707+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:36.720 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:36.718+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:37.141 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:37.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:37.140858+0000) 2022-04-04T01:46:37.693 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:37.691+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:37.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:37.724+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:38.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:38.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:38.141074+0000) 2022-04-04T01:46:38.645 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:38.643+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:38.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:38.686+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:39.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:39.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:39.141275+0000) 2022-04-04T01:46:39.605 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:39.603+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:39.652 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:39.651+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:40.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:40.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:40.141462+0000) 2022-04-04T01:46:40.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:40.560+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:40.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:40.696+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:41.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:41.139+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:41.141696+0000) 2022-04-04T01:46:41.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:41.597+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:41.746 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:41.744+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:42.142 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:42.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:42.141864+0000) 2022-04-04T01:46:42.579 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:42.577+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:42.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:42.794+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:43.143 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:43.140+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:43.142042+0000) 2022-04-04T01:46:43.558 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:43.556+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:43.754 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:43.753+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:44.143 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:44.141+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:44.142276+0000) 2022-04-04T01:46:44.562 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:44.560+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:44.724 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:44.722+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:45.059 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:46:45.057+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:46:45.143 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:45.141+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:45.142464+0000) 2022-04-04T01:46:45.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:45.534+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:45.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:45.690+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:46.143 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:46.141+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:46.142676+0000) 2022-04-04T01:46:46.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:46.546+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:46.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:46.668+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:47.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:47.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:47.142916+0000) 2022-04-04T01:46:47.520 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:47.519+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:47.643 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:47.641+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:48.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:48.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:48.143121+0000) 2022-04-04T01:46:48.555 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:48.554+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:48.683 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:48.682+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:49.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:49.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:49.143337+0000) 2022-04-04T01:46:49.602 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:49.601+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:49.660 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:49.659+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:50.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:50.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:50.143532+0000) 2022-04-04T01:46:50.631 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:50.630+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:50.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:50.676+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:51.144 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:51.142+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:51.143728+0000) 2022-04-04T01:46:51.600 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:51.599+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:51.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:51.668+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:52.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:52.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:52.143970+0000) 2022-04-04T01:46:52.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:52.550+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:52.673 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:52.671+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:53.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:53.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:53.144178+0000) 2022-04-04T01:46:53.521 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:53.519+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:53.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:53.699+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:53.746 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:46:53.749 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:46:54.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:54.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:54.144359+0000) 2022-04-04T01:46:54.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:54.541+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:54.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:54.653+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:55.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:55.143+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:55.144536+0000) 2022-04-04T01:46:55.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:55.527+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:55.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:55.666+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:56.145 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:56.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:56.144758+0000) 2022-04-04T01:46:56.527 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:56.525+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:56.643 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:56.642+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:57.146 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:57.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:57.144942+0000) 2022-04-04T01:46:57.530 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:57.529+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:57.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:57.660+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:58.146 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:58.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:58.145124+0000) 2022-04-04T01:46:58.566 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:58.565+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:46:58.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:58.612+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:59.146 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:46:59.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:46:59.145312+0000) 2022-04-04T01:46:59.570 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:46:59.569+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:46:59.594 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:46:59.593+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:00.075 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:47:00.073+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:47:00.146 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:00.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:00.145485+0000) 2022-04-04T01:47:00.576 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:00.574+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:00.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:00.579+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:01.146 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:01.144+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:01.145717+0000) 2022-04-04T01:47:01.543 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:01.541+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:01.607 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:01.605+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:02.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:02.145+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:02.145907+0000) 2022-04-04T01:47:02.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:02.585+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:02.640 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:02.638+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:03.146 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:03.145+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:03.146045+0000) 2022-04-04T01:47:03.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:03.580+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:03.627 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:03.625+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:04.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:04.145+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:04.146225+0000) 2022-04-04T01:47:04.610 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:04.608+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:04.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:04.621+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:05.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:05.145+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:05.146452+0000) 2022-04-04T01:47:05.599 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:05.597+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:05.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:05.653+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:06.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:06.146+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:06.146651+0000) 2022-04-04T01:47:06.617 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:06.615+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:06.626 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:06.624+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:07.147 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:07.146+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:07.146851+0000) 2022-04-04T01:47:07.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:07.662+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:07.673 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:07.671+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:08.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:08.146+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:08.147039+0000) 2022-04-04T01:47:08.663 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:08.661+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:08.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:08.711+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:09.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:09.146+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:09.147253+0000) 2022-04-04T01:47:09.651 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:09.649+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:09.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:09.701+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:10.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:10.147+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:10.147460+0000) 2022-04-04T01:47:10.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:10.653+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:10.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:10.711+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:11.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:11.147+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:11.147640+0000) 2022-04-04T01:47:11.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:11.686+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:11.692 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:11.690+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:12.148 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:12.147+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:12.147861+0000) 2022-04-04T01:47:12.658 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:12.655+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:12.690 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:12.687+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:13.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:13.147+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:13.148036+0000) 2022-04-04T01:47:13.674 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:13.672+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:13.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:13.680+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:14.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:14.147+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:14.148250+0000) 2022-04-04T01:47:14.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:14.685+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:14.705 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:14.703+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:15.091 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:47:15.089+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:47:15.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:15.147+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:15.148486+0000) 2022-04-04T01:47:15.703 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:15.701+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:15.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:15.753+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:16.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:16.148+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:16.148658+0000) 2022-04-04T01:47:16.660 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:16.658+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:16.794 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:16.791+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:17.149 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:17.148+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:17.148898+0000) 2022-04-04T01:47:17.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:17.660+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:17.790 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:17.788+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:18.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:18.148+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:18.149120+0000) 2022-04-04T01:47:18.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:18.680+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:18.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:18.832+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:19.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:19.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:19.149340+0000) 2022-04-04T01:47:19.720 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:19.718+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:19.837 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:19.835+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:19.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:19.962+0000 7fd623a1a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T01:47:19.964 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:19.962+0000 7fd62421b700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T01:47:19.966 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:19.964+0000 7fd623a1a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:19.965223+0000) 2022-04-04T01:47:20.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:20.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:20.149513+0000) 2022-04-04T01:47:20.770 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:20.768+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:20.797 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:20.795+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:21.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:21.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:21.149660+0000) 2022-04-04T01:47:21.753 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:21.750+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:21.763 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:21.761+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:22.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:22.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:22.149821+0000) 2022-04-04T01:47:22.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:22.793+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:22.799 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:22.797+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:23.150 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:23.148+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:23.149976+0000) 2022-04-04T01:47:23.777 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:23.775+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:23.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:23.821+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:23.878 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:47:23.882 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:47:24.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:24.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:24.150157+0000) 2022-04-04T01:47:24.805 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:24.803+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:24.834 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:24.833+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:25.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:25.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:25.150343+0000) 2022-04-04T01:47:25.761 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:25.759+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:25.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:25.810+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:47:26.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:26.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:26.150543+0000) 2022-04-04T01:47:26.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:26.713+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:26.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:26.844+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:27.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:27.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:27.150712+0000) 2022-04-04T01:47:27.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:27.701+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:27.855 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:27.853+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:28.151 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:28.149+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:28.150862+0000) 2022-04-04T01:47:28.706 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:28.704+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:28.896 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:28.894+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:29.152 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:29.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:29.151038+0000) 2022-04-04T01:47:29.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:29.748+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:29.877 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:29.874+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:30.107 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:47:30.104+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:47:30.152 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:30.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:30.151177+0000) 2022-04-04T01:47:30.788 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:30.786+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:30.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:30.922+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:31.152 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:31.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:31.151329+0000) 2022-04-04T01:47:31.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:31.785+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:31.931 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:31.930+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:32.152 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:32.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:32.151484+0000) 2022-04-04T01:47:32.744 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:32.743+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:32.894 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:32.892+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:33.152 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:33.150+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:33.151676+0000) 2022-04-04T01:47:33.775 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:33.773+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:33.927 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:33.925+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:34.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:34.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:34.151888+0000) 2022-04-04T01:47:34.819 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:34.817+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:34.936 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:34.935+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:35.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:35.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:35.152025+0000) 2022-04-04T01:47:35.836 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:35.833+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:35.962 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:35.959+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:36.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:36.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:36.152139+0000) 2022-04-04T01:47:36.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:36.803+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:37.008 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:37.006+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:37.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:37.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:37.152322+0000) 2022-04-04T01:47:37.764 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:37.761+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:37.960 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:37.958+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:38.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:38.151+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:38.152508+0000) 2022-04-04T01:47:38.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:38.800+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:38.983 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:38.981+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:39.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:39.152+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:39.152686+0000) 2022-04-04T01:47:39.781 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:39.780+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:40.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:40.015+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:40.153 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:40.152+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:40.152867+0000) 2022-04-04T01:47:40.823 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:40.821+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:41.034 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:41.032+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:41.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:41.152+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:41.153037+0000) 2022-04-04T01:47:41.844 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:41.843+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:42.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:42.031+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:42.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:42.152+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:42.153210+0000) 2022-04-04T01:47:42.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:42.871+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:43.056 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:43.054+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:43.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:43.152+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:43.153385+0000) 2022-04-04T01:47:43.879 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:43.877+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:44.023 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:44.022+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:44.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:44.152+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:44.153511+0000) 2022-04-04T01:47:44.852 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:44.850+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:45.041 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:45.040+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:45.122 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:47:45.121+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:47:45.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:45.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:45.153631+0000) 2022-04-04T01:47:45.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:45.876+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:46.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:46.060+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:46.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:46.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:46.153749+0000) 2022-04-04T01:47:46.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:46.831+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:47.019 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:47.018+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:47.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:47.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:47.153910+0000) 2022-04-04T01:47:47.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:47.862+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:48.016 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:48.014+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:48.154 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:48.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:48.154058+0000) 2022-04-04T01:47:48.901 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:48.899+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:49.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:49.059+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:49.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:49.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:49.154182+0000) 2022-04-04T01:47:49.880 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:49.879+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:50.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:50.048+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:50.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:50.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:50.154310+0000) 2022-04-04T01:47:50.871 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:50.869+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:51.095 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:51.093+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:51.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:51.153+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:51.154409+0000) 2022-04-04T01:47:51.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:51.827+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:52.061 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:52.059+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:52.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:52.154+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:52.154551+0000) 2022-04-04T01:47:52.790 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:52.788+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:53.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:53.106+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:53.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:53.154+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:53.154713+0000) 2022-04-04T01:47:53.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:53.740+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:54.033 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:47:54.038 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:47:54.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:54.075+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:54.155 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:54.154+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:54.154867+0000) 2022-04-04T01:47:54.774 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:54.772+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:55.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:55.121+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:55.156 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:55.154+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:55.155038+0000) 2022-04-04T01:47:55.816 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:55.814+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:56.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:56.125+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:56.156 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:56.154+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:56.155233+0000) 2022-04-04T01:47:56.842 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:56.840+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:57.156 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:57.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:57.155392+0000) 2022-04-04T01:47:57.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:57.159+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:57.866 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:57.864+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:58.156 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:58.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:58.155573+0000) 2022-04-04T01:47:58.157 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:58.154+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:58.894 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:58.892+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:47:59.130 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:47:59.128+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:47:59.156 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:47:59.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:47:59.155789+0000) 2022-04-04T01:47:59.857 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:47:59.855+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:00.119 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:00.117+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:00.138 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:48:00.136+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:48:00.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:00.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:00.155964+0000) 2022-04-04T01:48:00.832 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:00.830+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:01.107 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:01.104+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:01.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:01.155+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:01.156093+0000) 2022-04-04T01:48:01.836 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:01.834+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:02.110 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:02.107+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:02.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:02.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:02.156244+0000) 2022-04-04T01:48:02.817 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:02.814+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:03.146 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:03.143+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:03.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:03.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:03.156433+0000) 2022-04-04T01:48:03.788 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:03.785+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:04.113 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:04.110+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:04.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:04.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:04.156621+0000) 2022-04-04T01:48:04.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:04.747+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:05.072 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:05.070+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:05.157 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:05.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:05.156805+0000) 2022-04-04T01:48:05.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:05.780+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:06.098 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:06.096+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:06.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:06.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:06.156981+0000) 2022-04-04T01:48:06.771 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:06.768+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:07.070 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:07.068+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:07.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:07.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:07.157162+0000) 2022-04-04T01:48:07.744 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:07.742+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:08.065 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:08.062+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:08.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:08.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:08.157344+0000) 2022-04-04T01:48:08.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:08.739+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:09.027 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:09.025+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:09.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:09.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:09.157519+0000) 2022-04-04T01:48:09.790 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:09.787+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:10.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:10.069+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:10.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:10.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:10.157702+0000) 2022-04-04T01:48:10.754 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:10.751+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:11.073 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:11.071+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:11.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:11.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:11.157842+0000) 2022-04-04T01:48:11.785 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:11.783+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:12.113 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:12.110+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:12.158 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:12.156+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:12.157967+0000) 2022-04-04T01:48:12.835 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:12.832+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:13.102 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:13.100+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:13.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:13.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:13.158128+0000) 2022-04-04T01:48:13.839 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:13.837+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:14.100 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:14.097+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:14.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:14.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:14.158237+0000) 2022-04-04T01:48:14.866 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:14.864+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:15.084 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:15.081+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:15.154 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:48:15.151+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:48:15.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:15.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:15.158333+0000) 2022-04-04T01:48:15.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:15.895+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:16.090 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:16.088+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:16.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:16.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:16.158516+0000) 2022-04-04T01:48:16.881 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:16.879+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:17.066 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:17.064+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:17.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:17.157+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:17.158656+0000) 2022-04-04T01:48:17.925 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:17.923+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:18.026 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:18.024+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:18.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:18.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:18.158802+0000) 2022-04-04T01:48:18.883 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:18.881+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:19.053 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:19.051+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:19.159 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:19.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:19.158970+0000) 2022-04-04T01:48:19.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:19.853+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:20.096 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:20.094+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:20.160 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:20.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:20.159137+0000) 2022-04-04T01:48:20.892 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:20.890+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:21.101 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:21.098+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:21.160 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:21.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:21.159316+0000) 2022-04-04T01:48:21.854 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:21.851+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:22.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:22.122+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:22.160 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:22.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:22.159434+0000) 2022-04-04T01:48:22.883 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:22.881+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:23.143 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:23.141+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:23.160 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:23.158+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:23.159599+0000) 2022-04-04T01:48:23.930 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:23.928+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:24.132 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:24.130+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:24.160 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:24.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:24.159738+0000) 2022-04-04T01:48:24.164 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:48:24.169 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:48:24.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:24.913+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:25.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:25.151+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:25.160 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:25.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:25.159901+0000) 2022-04-04T01:48:25.884 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:25.882+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:26.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:26.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:26.160098+0000) 2022-04-04T01:48:26.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:26.195+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:26.837 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:26.835+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:27.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:27.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:27.160253+0000) 2022-04-04T01:48:27.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:27.177+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:27.848 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:27.846+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:28.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:28.159+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:28.160405+0000) 2022-04-04T01:48:28.192 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:28.190+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:28.850 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:28.848+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:29.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:29.160+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:29.160588+0000) 2022-04-04T01:48:29.169 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:29.167+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:29.868 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:29.865+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:30.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:30.160+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:30.160724+0000) 2022-04-04T01:48:30.169 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:30.166+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:30.170 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:48:30.168+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:48:30.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:30.913+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:31.161 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:31.160+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:31.160926+0000) 2022-04-04T01:48:31.164 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:31.162+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:31.935 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:31.933+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:32.158 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:32.156+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:32.162 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:32.160+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:32.161094+0000) 2022-04-04T01:48:32.912 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:32.910+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:33.149 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:33.148+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:33.162 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:33.160+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:33.161290+0000) 2022-04-04T01:48:33.921 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:33.919+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:34.113 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:34.112+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:34.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:34.161+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:34.161434+0000) 2022-04-04T01:48:34.951 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:34.950+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:35.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:35.070+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:35.162 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:35.161+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:35.161564+0000) 2022-04-04T01:48:35.987 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:35.985+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:36.098 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:36.096+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:36.162 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:36.161+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:36.161695+0000) 2022-04-04T01:48:37.019 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:37.017+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:37.085 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:37.083+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:37.162 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:37.161+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:37.161834+0000) 2022-04-04T01:48:37.974 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:37.973+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:38.047 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:38.045+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:38.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:38.161+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:38.162001+0000) 2022-04-04T01:48:38.974 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:38.973+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:39.040 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:39.038+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:39.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:39.161+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:39.162176+0000) 2022-04-04T01:48:39.937 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:39.935+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:40.070 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:40.068+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:40.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:40.162+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:40.162345+0000) 2022-04-04T01:48:40.923 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:40.921+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:41.102 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:41.100+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:41.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:41.162+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:41.162496+0000) 2022-04-04T01:48:41.901 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:41.900+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:42.076 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:42.074+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:42.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:42.162+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:42.162652+0000) 2022-04-04T01:48:42.891 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:42.890+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:43.027 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:43.026+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:43.163 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:43.162+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:43.162823+0000) 2022-04-04T01:48:43.876 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:43.874+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:44.071 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:44.069+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:44.164 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:44.162+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:44.163002+0000) 2022-04-04T01:48:44.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:44.881+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:45.074 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:45.072+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:45.164 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:45.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:45.163179+0000) 2022-04-04T01:48:45.185 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:48:45.184+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:48:45.910 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:45.908+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:46.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:46.122+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:46.164 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:46.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:46.163358+0000) 2022-04-04T01:48:46.931 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:46.929+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:47.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:47.129+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:47.164 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:47.162+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:47.163528+0000) 2022-04-04T01:48:47.980 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:47.979+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:48.096 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:48.094+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:48.164 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:48.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:48.163698+0000) 2022-04-04T01:48:48.968 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:48.965+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:49.070 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:49.068+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:49.164 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:49.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:49.163850+0000) 2022-04-04T01:48:49.922 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:49.920+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:50.092 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:50.090+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:50.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:50.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:50.164030+0000) 2022-04-04T01:48:50.896 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:50.894+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:51.103 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:51.100+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:51.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:51.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:51.164211+0000) 2022-04-04T01:48:51.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:51.913+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:52.130 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:52.128+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:52.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:52.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:52.164369+0000) 2022-04-04T01:48:52.913 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:52.911+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:53.083 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:53.080+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:53.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:53.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:53.164545+0000) 2022-04-04T01:48:53.902 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:53.900+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:54.093 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:54.091+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:54.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:54.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:54.164726+0000) 2022-04-04T01:48:54.261 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:48:54.265 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:48:54.886 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:54.884+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:55.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:55.137+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:55.165 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:55.163+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:55.164933+0000) 2022-04-04T01:48:55.895 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:55.892+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:56.166 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:56.164+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:56.165109+0000) 2022-04-04T01:48:56.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:56.181+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:56.859 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:56.856+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:57.162 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:57.160+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:57.166 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:57.164+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:57.165264+0000) 2022-04-04T01:48:57.830 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:57.827+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:58.166 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:58.164+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:58.165444+0000) 2022-04-04T01:48:58.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:58.185+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:58.781 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:58.779+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:48:59.166 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:48:59.164+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:48:59.165625+0000) 2022-04-04T01:48:59.172 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:48:59.170+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:48:59.781 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:48:59.778+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:00.167 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:00.164+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:00.165806+0000) 2022-04-04T01:49:00.169 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:00.167+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:00.201 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:49:00.199+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:49:00.811 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:00.809+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:01.167 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:01.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:01.165993+0000) 2022-04-04T01:49:01.219 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:01.217+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:01.767 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:01.765+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:02.167 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:02.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:02.166132+0000) 2022-04-04T01:49:02.206 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:02.203+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:02.783 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:02.780+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:03.167 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:03.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:03.166321+0000) 2022-04-04T01:49:03.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:03.195+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:03.826 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:03.824+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:04.167 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:04.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:04.166493+0000) 2022-04-04T01:49:04.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:04.226+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:04.870 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:04.868+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:05.167 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:05.165+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:05.166671+0000) 2022-04-04T01:49:05.194 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:05.192+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:05.877 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:05.874+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:06.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:06.166+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:06.166854+0000) 2022-04-04T01:49:06.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:06.205+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:06.886 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:06.884+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:07.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:07.166+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:07.167030+0000) 2022-04-04T01:49:07.181 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:07.179+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:07.859 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:07.857+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:08.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:08.166+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:08.167189+0000) 2022-04-04T01:49:08.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:08.193+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:08.838 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:08.836+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:09.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:09.166+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:09.167338+0000) 2022-04-04T01:49:09.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:09.196+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:09.807 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:09.804+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:10.157 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:10.155+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:10.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:10.166+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:10.167527+0000) 2022-04-04T01:49:10.844 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:10.842+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:11.148 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:11.146+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:11.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:11.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:11.167683+0000) 2022-04-04T01:49:11.813 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:11.811+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:12.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:12.130+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:12.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:12.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:12.167887+0000) 2022-04-04T01:49:12.830 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:12.828+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:13.166 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:13.164+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:13.168 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:13.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:13.168095+0000) 2022-04-04T01:49:13.809 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:13.807+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:14.150 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:14.147+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:14.169 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:14.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:14.168280+0000) 2022-04-04T01:49:14.770 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:14.767+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:15.169 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:15.167+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:15.168463+0000) 2022-04-04T01:49:15.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:15.187+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:15.217 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:49:15.215+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:49:15.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:15.817+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:16.169 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:16.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:16.168640+0000) 2022-04-04T01:49:16.217 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:16.214+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:16.804 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:16.802+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:17.169 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:17.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:17.168798+0000) 2022-04-04T01:49:17.246 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:17.244+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:17.839 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:17.837+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:18.170 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:18.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:18.168945+0000) 2022-04-04T01:49:18.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:18.258+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:18.834 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:18.832+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:19.170 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:19.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:19.169123+0000) 2022-04-04T01:49:19.307 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:19.305+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:19.833 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:19.830+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:20.170 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:20.168+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:20.169294+0000) 2022-04-04T01:49:20.293 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:20.291+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:20.857 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:20.854+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:21.170 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:21.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:21.169470+0000) 2022-04-04T01:49:21.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:21.333+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:21.845 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:21.843+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:22.170 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:22.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:22.169652+0000) 2022-04-04T01:49:22.301 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:22.299+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:22.875 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:22.873+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:23.171 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:23.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:23.169856+0000) 2022-04-04T01:49:23.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:23.323+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:23.901 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:23.899+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:24.171 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:24.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:24.170049+0000) 2022-04-04T01:49:24.341 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:24.339+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:24.423 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:49:24.427 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:49:24.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:24.862+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:25.171 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:25.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:25.170222+0000) 2022-04-04T01:49:25.374 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:25.372+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:25.862 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:25.860+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:26.171 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:26.169+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:26.170378+0000) 2022-04-04T01:49:26.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:26.372+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:26.848 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:26.846+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:27.171 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:27.170+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:27.170558+0000) 2022-04-04T01:49:27.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:27.337+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:27.805 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:27.803+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:28.171 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:28.170+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:28.170737+0000) 2022-04-04T01:49:28.342 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:28.340+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:28.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:28.758+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:29.172 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:29.170+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:29.170959+0000) 2022-04-04T01:49:29.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:29.327+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:29.778 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:29.777+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:30.172 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:30.170+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:30.171130+0000) 2022-04-04T01:49:30.233 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:49:30.231+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:49:30.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:30.316+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:30.786 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:30.784+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:31.172 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:31.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:31.171310+0000) 2022-04-04T01:49:31.306 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:31.304+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:31.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:31.785+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:32.172 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:32.170+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:32.171505+0000) 2022-04-04T01:49:32.303 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:32.302+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:32.779 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:32.777+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:33.172 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:33.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:33.171668+0000) 2022-04-04T01:49:33.293 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:33.292+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:33.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:33.742+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:34.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:34.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:34.171819+0000) 2022-04-04T01:49:34.269 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:34.268+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:34.751 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:34.750+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:35.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:35.170+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:35.172000+0000) 2022-04-04T01:49:35.318 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:35.316+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:35.790 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:35.789+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:36.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:36.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:36.172124+0000) 2022-04-04T01:49:36.290 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:36.289+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:36.763 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:36.761+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:37.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:37.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:37.172253+0000) 2022-04-04T01:49:37.311 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:37.310+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:37.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:37.781+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:38.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:38.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:38.172397+0000) 2022-04-04T01:49:38.304 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:38.303+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:38.797 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:38.796+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:39.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:39.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:39.172551+0000) 2022-04-04T01:49:39.299 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:39.297+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:39.784 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:39.783+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:40.173 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:40.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:40.172725+0000) 2022-04-04T01:49:40.258 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:40.257+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:40.784 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:40.782+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:41.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:41.171+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:41.172874+0000) 2022-04-04T01:49:41.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:41.277+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:41.757 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:41.755+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:42.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:42.172+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:42.173081+0000) 2022-04-04T01:49:42.245 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:42.243+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:42.722 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:42.720+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:43.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:43.172+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:43.173302+0000) 2022-04-04T01:49:43.247 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:43.244+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:43.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:43.672+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:44.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:44.172+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:44.173475+0000) 2022-04-04T01:49:44.284 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:44.282+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:44.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:44.677+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:45.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:45.172+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:45.173663+0000) 2022-04-04T01:49:45.249 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:49:45.247+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:49:45.315 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:45.312+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:45.714 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:45.711+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:46.174 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:46.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:46.173846+0000) 2022-04-04T01:49:46.305 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:46.302+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:46.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:46.676+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:47.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:47.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:47.174010+0000) 2022-04-04T01:49:47.329 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:47.326+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:47.713 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:47.711+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:48.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:48.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:48.174173+0000) 2022-04-04T01:49:48.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:48.293+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:48.755 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:48.753+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:49.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:49.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:49.174329+0000) 2022-04-04T01:49:49.289 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:49.286+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:49.791 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:49.789+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:50.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:50.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:50.174472+0000) 2022-04-04T01:49:50.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:50.275+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:50.802 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:50.799+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:51.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:51.173+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:51.174632+0000) 2022-04-04T01:49:51.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:51.266+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:51.778 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:51.776+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:52.175 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:52.174+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:52.174808+0000) 2022-04-04T01:49:52.260 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:52.258+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:52.822 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:52.820+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:53.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:53.174+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:53.174981+0000) 2022-04-04T01:49:53.280 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:53.278+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:53.793 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:53.791+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:54.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:54.174+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:54.175106+0000) 2022-04-04T01:49:54.247 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:54.245+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:54.573 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:49:54.577 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:49:54.758 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:54.756+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:55.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:55.174+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:55.175265+0000) 2022-04-04T01:49:55.228 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:55.226+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:55.749 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:55.747+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:56.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:56.174+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:56.175434+0000) 2022-04-04T01:49:56.275 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:56.272+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:56.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:56.705+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:57.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:57.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:57.175560+0000) 2022-04-04T01:49:57.322 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:57.320+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:57.747 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:57.744+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:58.176 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:58.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:58.175693+0000) 2022-04-04T01:49:58.289 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:58.287+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:58.774 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:58.772+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:49:59.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:49:59.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:49:59.175889+0000) 2022-04-04T01:49:59.275 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:49:59.273+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:49:59.729 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:49:59.727+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:00.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:00.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:00.176024+0000) 2022-04-04T01:50:00.264 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:50:00.263+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:50:00.295 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:00.293+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:00.681 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:00.679+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:01.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:01.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:01.176201+0000) 2022-04-04T01:50:01.307 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:01.305+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:01.635 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:01.633+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:02.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:02.175+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:02.176327+0000) 2022-04-04T01:50:02.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:02.350+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:02.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:02.678+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:03.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:03.176+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:03.176493+0000) 2022-04-04T01:50:03.323 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:03.321+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:03.704 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:03.702+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:04.177 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:04.176+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:04.176625+0000) 2022-04-04T01:50:04.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:04.312+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:04.678 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:04.676+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:05.178 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:05.176+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:05.176810+0000) 2022-04-04T01:50:05.344 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:05.342+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:05.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:05.716+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:06.178 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:06.176+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:06.177022+0000) 2022-04-04T01:50:06.329 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:06.327+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:06.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:06.712+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:07.178 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:07.176+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:07.177189+0000) 2022-04-04T01:50:07.343 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:07.342+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:07.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:07.731+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:08.178 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:08.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:08.177371+0000) 2022-04-04T01:50:08.298 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:08.296+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:08.735 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:08.733+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:09.178 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:09.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:09.177541+0000) 2022-04-04T01:50:09.342 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:09.339+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:09.704 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:09.702+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:10.179 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:10.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:10.177709+0000) 2022-04-04T01:50:10.307 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:10.305+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:10.679 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:10.677+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:11.179 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:11.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:11.177891+0000) 2022-04-04T01:50:11.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:11.338+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:11.633 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:11.631+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:12.179 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:12.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:12.178052+0000) 2022-04-04T01:50:12.308 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:12.306+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:12.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:12.648+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:13.179 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:13.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:13.178230+0000) 2022-04-04T01:50:13.268 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:13.266+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:13.634 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:13.632+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:14.179 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:14.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:14.178429+0000) 2022-04-04T01:50:14.266 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:14.264+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:14.614 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:14.612+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:15.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:15.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:15.178622+0000) 2022-04-04T01:50:15.221 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:15.218+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:15.280 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:50:15.279+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:50:15.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:15.563+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:16.176 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:16.175+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:16.179 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:16.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:16.178808+0000) 2022-04-04T01:50:16.575 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:16.573+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:17.137 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:17.134+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:17.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:17.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:17.178967+0000) 2022-04-04T01:50:17.563 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:17.561+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:18.162 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:18.160+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:18.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:18.179+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:18.179111+0000) 2022-04-04T01:50:18.536 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:18.534+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:19.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:19.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:19.179256+0000) 2022-04-04T01:50:19.192 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:19.191+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:19.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:19.560+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:20.169 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:20.167+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:20.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:20.177+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:20.179480+0000) 2022-04-04T01:50:20.529 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:20.526+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:21.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:21.137+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:21.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:21.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:21.179692+0000) 2022-04-04T01:50:21.501 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:21.499+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:22.146 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:22.143+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:22.180 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:22.178+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:22.179902+0000) 2022-04-04T01:50:22.543 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:22.541+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:23.155 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:23.154+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:23.181 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:23.179+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:23.180128+0000) 2022-04-04T01:50:23.528 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:23.526+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:24.146 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:24.144+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:24.181 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:24.179+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:24.180313+0000) 2022-04-04T01:50:24.490 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:24.489+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:24.896 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:50:24.900 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:50:25.138 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:25.136+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e120) 2022-04-04T01:50:25.181 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:25.179+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:25.180493+0000) 2022-04-04T01:50:25.498 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:25.497+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:26.181 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:26.179+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:26.180698+0000) 2022-04-04T01:50:26.184 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:26.183+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:26.519 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:26.517+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:27.178 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:27.176+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:27.181 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:27.180+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:27.180908+0000) 2022-04-04T01:50:27.551 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:27.550+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:28.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:28.137+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:28.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:28.180+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:28.181101+0000) 2022-04-04T01:50:28.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:28.586+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:29.142 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:29.140+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:29.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:29.180+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:29.181301+0000) 2022-04-04T01:50:29.587 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:29.585+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:30.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:30.168+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:30.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:30.180+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:30.181497+0000) 2022-04-04T01:50:30.296 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:50:30.294+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:50:30.592 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:30.591+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:31.167 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:31.166+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:31.182 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:31.180+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:31.181690+0000) 2022-04-04T01:50:31.593 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:31.591+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:32.134 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:32.132+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:32.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:32.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:32.181890+0000) 2022-04-04T01:50:32.567 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:32.565+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:33.170 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:33.169+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:33.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:33.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:33.182037+0000) 2022-04-04T01:50:33.610 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:33.608+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:34.131 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:34.129+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:34.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:34.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:34.182235+0000) 2022-04-04T01:50:34.618 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:34.617+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:35.175 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:35.173+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:35.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:35.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:35.182428+0000) 2022-04-04T01:50:35.661 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:35.659+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:36.183 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:36.181+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:36.182579+0000) 2022-04-04T01:50:36.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:36.197+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:36.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:36.676+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:37.184 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:37.182+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:37.182828+0000) 2022-04-04T01:50:37.202 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:37.200+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:37.720 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:37.718+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:38.184 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:38.182+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:38.183039+0000) 2022-04-04T01:50:38.244 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:38.242+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:38.742 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:38.740+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:39.184 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:39.182+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:39.183192+0000) 2022-04-04T01:50:39.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:39.195+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:39.763 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:39.760+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:40.184 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:40.182+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:40.183390+0000) 2022-04-04T01:50:40.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:40.232+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:40.787 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:40.784+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:41.184 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:41.182+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:41.183576+0000) 2022-04-04T01:50:41.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:41.233+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:41.790 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:41.788+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:42.185 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:42.183+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:42.183793+0000) 2022-04-04T01:50:42.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:42.229+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:42.782 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:42.779+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:43.185 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:43.183+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:43.184002+0000) 2022-04-04T01:50:43.252 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:43.249+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:43.830 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:43.828+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:44.185 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:44.183+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:44.184164+0000) 2022-04-04T01:50:44.243 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:44.241+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:44.858 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:44.855+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:45.185 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:45.183+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:45.184376+0000) 2022-04-04T01:50:45.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:45.286+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:45.309 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:50:45.307+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:50:45.841 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:45.839+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:46.185 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:46.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:46.184576+0000) 2022-04-04T01:50:46.312 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:46.310+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:46.823 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:46.820+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:47.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:47.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:47.184823+0000) 2022-04-04T01:50:47.280 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:47.278+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:47.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:47.818+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:48.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:48.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:48.185039+0000) 2022-04-04T01:50:48.317 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:48.315+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:48.858 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:48.856+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:49.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:49.184+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:49.185206+0000) 2022-04-04T01:50:49.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:49.345+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:49.872 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:49.870+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:50.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:50.185+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:50.185390+0000) 2022-04-04T01:50:50.365 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:50.363+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:50.861 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:50.859+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:51.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:51.185+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:51.185590+0000) 2022-04-04T01:50:51.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:51.403+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:51.864 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:51.862+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:52.186 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:52.185+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:52.185755+0000) 2022-04-04T01:50:52.407 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:52.405+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:52.885 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:52.883+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:53.187 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:53.185+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:53.185982+0000) 2022-04-04T01:50:53.375 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:53.373+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:53.883 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:53.880+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:54.187 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:54.185+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:54.186193+0000) 2022-04-04T01:50:54.332 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:54.330+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:54.857 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:54.855+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:55.051 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:50:55.055 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:50:55.187 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:55.185+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:55.186421+0000) 2022-04-04T01:50:55.296 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:55.294+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:55.898 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:55.895+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:56.187 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:56.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:56.186632+0000) 2022-04-04T01:50:56.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:56.295+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:56.915 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:56.912+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:57.187 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:57.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:57.186831+0000) 2022-04-04T01:50:57.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:57.322+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:57.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:57.930+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:58.188 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:58.186+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:58.186984+0000) 2022-04-04T01:50:58.333 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:58.331+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:50:58.974 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:50:58.971+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:50:59.188 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:50:59.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:50:59.187189+0000) 2022-04-04T01:50:59.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:50:59.314+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:00.005 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:00.003+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:00.188 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:00.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:00.187399+0000) 2022-04-04T01:51:00.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:00.286+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:00.325 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:51:00.323+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:51:01.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:01.010+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:01.188 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:01.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:01.187625+0000) 2022-04-04T01:51:01.327 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:01.324+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:02.003 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:02.001+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:02.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:02.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:02.187869+0000) 2022-04-04T01:51:02.353 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:02.350+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:02.987 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:02.985+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:03.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:03.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:03.188089+0000) 2022-04-04T01:51:03.351 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:03.349+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:03.960 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:03.958+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:04.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:04.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:04.188276+0000) 2022-04-04T01:51:04.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:04.333+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:04.940 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:04.938+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:05.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:05.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:05.188456+0000) 2022-04-04T01:51:05.291 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:05.289+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:05.906 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:05.904+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:06.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:06.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:06.188666+0000) 2022-04-04T01:51:06.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:06.263+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:06.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:06.907+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:07.189 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:07.187+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:07.188919+0000) 2022-04-04T01:51:07.259 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:07.257+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:07.904 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:07.902+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:08.190 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:08.188+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:08.189133+0000) 2022-04-04T01:51:08.258 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:08.256+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:08.885 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:08.883+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:09.190 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:09.188+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:09.189310+0000) 2022-04-04T01:51:09.275 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:09.272+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:09.879 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:09.877+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:10.190 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:10.188+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:10.189463+0000) 2022-04-04T01:51:10.226 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:10.224+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:10.912 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:10.910+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:11.190 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:11.188+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:11.189674+0000) 2022-04-04T01:51:11.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:11.230+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:11.893 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:11.891+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:12.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:12.189+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:12.189897+0000) 2022-04-04T01:51:12.241 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:12.239+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:12.846 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:12.844+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:13.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:13.189+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:13.190106+0000) 2022-04-04T01:51:13.206 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:13.205+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:13.846 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:13.844+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:14.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:14.189+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:14.190319+0000) 2022-04-04T01:51:14.203 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:14.201+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:14.869 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:14.867+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:15.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:15.189+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:15.190489+0000) 2022-04-04T01:51:15.218 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:15.215+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:15.341 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:51:15.339+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:51:15.888 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:15.886+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:16.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:16.189+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:16.190685+0000) 2022-04-04T01:51:16.207 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:16.205+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:16.889 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:16.888+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:17.189 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:17.187+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:17.191 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:17.190+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:17.190895+0000) 2022-04-04T01:51:17.925 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:17.923+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:18.192 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:18.190+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:18.191101+0000) 2022-04-04T01:51:18.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:18.196+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:18.913 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:18.911+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:19.192 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:19.190+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:19.191296+0000) 2022-04-04T01:51:19.242 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:19.241+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:19.912 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:19.910+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:20.192 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:20.190+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:20.191507+0000) 2022-04-04T01:51:20.271 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:20.270+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:20.962 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:20.961+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:21.192 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:21.191+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:21.191678+0000) 2022-04-04T01:51:21.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:21.312+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:21.960 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:21.959+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:22.193 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:22.191+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:22.191886+0000) 2022-04-04T01:51:22.269 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:22.268+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:22.911 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:22.910+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:23.193 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:23.191+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:23.192100+0000) 2022-04-04T01:51:23.265 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:23.264+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:23.911 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:23.909+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:24.193 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:24.191+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:24.192306+0000) 2022-04-04T01:51:24.293 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:24.292+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:24.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:24.952+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:25.128 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:51:25.132 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:51:25.193 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:25.191+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:25.192577+0000) 2022-04-04T01:51:25.316 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:25.315+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:25.931 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:25.929+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:26.193 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:26.192+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:26.192826+0000) 2022-04-04T01:51:26.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:26.322+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:26.969 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:26.968+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:27.194 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:27.192+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:27.193035+0000) 2022-04-04T01:51:27.374 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:27.372+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:28.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:28.011+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:28.194 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:28.192+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:28.193243+0000) 2022-04-04T01:51:28.355 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:28.353+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:29.058 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:29.057+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:29.194 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:29.192+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:29.193471+0000) 2022-04-04T01:51:29.319 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:29.317+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:30.043 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:30.041+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:30.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:30.193+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:30.193697+0000) 2022-04-04T01:51:30.304 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:30.303+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:30.357 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:51:30.355+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:51:31.052 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:31.051+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:31.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:31.193+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:31.193906+0000) 2022-04-04T01:51:31.280 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:31.277+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:32.007 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:32.005+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:32.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:32.193+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:32.194114+0000) 2022-04-04T01:51:32.297 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:32.295+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:33.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:33.052+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:33.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:33.193+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:33.194331+0000) 2022-04-04T01:51:33.324 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:33.321+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:34.022 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:34.019+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:34.195 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:34.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:34.194540+0000) 2022-04-04T01:51:34.336 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:34.333+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:35.011 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:35.009+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:35.196 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:35.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:35.194794+0000) 2022-04-04T01:51:35.350 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:35.348+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:36.002 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:35.999+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:36.196 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:36.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:36.195023+0000) 2022-04-04T01:51:36.381 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:36.379+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:36.966 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:36.963+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:37.196 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:37.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:37.195211+0000) 2022-04-04T01:51:37.344 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:37.341+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:37.987 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:37.984+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:38.196 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:38.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:38.195424+0000) 2022-04-04T01:51:38.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:38.387+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:38.938 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:38.936+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:39.196 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:39.194+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:39.195571+0000) 2022-04-04T01:51:39.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:39.347+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:39.960 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:39.957+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:40.196 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:40.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:40.195818+0000) 2022-04-04T01:51:40.355 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:40.352+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:40.969 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:40.967+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:41.197 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:41.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:41.195924+0000) 2022-04-04T01:51:41.340 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:41.338+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:42.014 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:42.012+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:42.197 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:42.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:42.196167+0000) 2022-04-04T01:51:42.292 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:42.289+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:43.007 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:43.005+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:43.197 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:43.196+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:43.196373+0000) 2022-04-04T01:51:43.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:43.247+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:44.018 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:44.016+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:44.197 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:44.195+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:44.196606+0000) 2022-04-04T01:51:44.221 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:44.219+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:44.999 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:44.997+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:45.176 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:45.174+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:45.197 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:45.196+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:45.196799+0000) 2022-04-04T01:51:45.361 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:51:45.359+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:51:46.032 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:46.029+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:46.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:46.195+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:46.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:46.196+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:46.196972+0000) 2022-04-04T01:51:47.042 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:47.040+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:47.185 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:47.183+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:47.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:47.196+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:47.197198+0000) 2022-04-04T01:51:48.025 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:48.023+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:48.169 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:48.167+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:48.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:48.196+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:48.197435+0000) 2022-04-04T01:51:49.071 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:49.068+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:49.179 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:49.177+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:49.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:49.196+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:49.197626+0000) 2022-04-04T01:51:50.068 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:50.065+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:50.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:50.145+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:50.198 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:50.196+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:50.197840+0000) 2022-04-04T01:51:51.106 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:51.104+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:51.127 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:51.124+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:51.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:51.197+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:51.198010+0000) 2022-04-04T01:51:52.128 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:52.126+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:52.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:52.135+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:52.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:52.197+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:52.198185+0000) 2022-04-04T01:51:53.111 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:53.109+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:53.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:53.165+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:53.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:53.197+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:53.198374+0000) 2022-04-04T01:51:54.153 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:54.151+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:54.154 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:54.152+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:54.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:54.197+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:54.198585+0000) 2022-04-04T01:51:55.153 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:55.150+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:55.193 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:55.191+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:55.199 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:55.197+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:55.198799+0000) 2022-04-04T01:51:55.288 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:51:55.292 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:51:56.111 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:56.108+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:56.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:56.198+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:56.198997+0000) 2022-04-04T01:51:56.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:56.234+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:57.072 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:57.070+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:57.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:57.198+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:57.199175+0000) 2022-04-04T01:51:57.252 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:57.250+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:58.077 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:58.075+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:58.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:58.198+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:58.199404+0000) 2022-04-04T01:51:58.256 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:58.254+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:51:59.032 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:51:59.030+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:51:59.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:51:59.198+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:51:59.199601+0000) 2022-04-04T01:51:59.274 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:51:59.272+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:00.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:00.053+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:00.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:00.199+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:00.199758+0000) 2022-04-04T01:52:00.313 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:00.311+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:00.377 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:52:00.375+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:52:01.065 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:01.062+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:01.200 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:01.199+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:01.199941+0000) 2022-04-04T01:52:01.332 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:01.330+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:02.044 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:02.042+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:02.201 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:02.199+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:02.200082+0000) 2022-04-04T01:52:02.338 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:02.336+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:03.035 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:03.033+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:03.201 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:03.199+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:03.200276+0000) 2022-04-04T01:52:03.342 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:03.340+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:04.060 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:04.058+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:04.201 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:04.199+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:04.200498+0000) 2022-04-04T01:52:04.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:04.390+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:05.100 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:05.098+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:05.201 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:05.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:05.200707+0000) 2022-04-04T01:52:05.433 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:05.430+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:06.060 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:06.058+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:06.202 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:06.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:06.200919+0000) 2022-04-04T01:52:06.412 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:06.409+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:07.089 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:07.087+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:07.202 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:07.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:07.201033+0000) 2022-04-04T01:52:07.400 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:07.398+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:08.116 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:08.114+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:08.202 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:08.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:08.201215+0000) 2022-04-04T01:52:08.416 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:08.415+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:09.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:09.161+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:09.202 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:09.200+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:09.201441+0000) 2022-04-04T01:52:09.386 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:09.384+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:10.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:10.178+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:10.202 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:10.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:10.201610+0000) 2022-04-04T01:52:10.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:10.434+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:11.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:11.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:11.201857+0000) 2022-04-04T01:52:11.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:11.227+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:11.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:11.417+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:12.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:12.186+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:12.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:12.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:12.202067+0000) 2022-04-04T01:52:12.437 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:12.435+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:13.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:13.176+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:13.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:13.201+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:13.202281+0000) 2022-04-04T01:52:13.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:13.434+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:14.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:14.198+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:14.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:14.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:14.202479+0000) 2022-04-04T01:52:14.460 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:14.459+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:15.203 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:15.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:15.202682+0000) 2022-04-04T01:52:15.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:15.237+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:15.393 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:52:15.391+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:52:15.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:15.421+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:16.204 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:16.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:16.202878+0000) 2022-04-04T01:52:16.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:16.221+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:16.402 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:16.400+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:17.204 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:17.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:17.203119+0000) 2022-04-04T01:52:17.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:17.266+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:17.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:17.372+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:18.204 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:18.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:18.203317+0000) 2022-04-04T01:52:18.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:18.305+0000 7f88d0ac1700 -1 osd.4 141 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:18.376 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:18.375+0000 7fd3b190b700 -1 osd.6 141 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:18.949 INFO:tasks.ceph.mds.f.smithi186.stderr:2022-04-04T01:52:18.948+0000 7fc77ebd8700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-04T01:52:18.964 INFO:tasks.ceph.mds.f.smithi186.stderr: -1> 2022-04-04T01:52:18.948+0000 7fc77ebd8700 -1 MDSIOContextBase: failed with -110, restarting... 2022-04-04T01:52:18.987 INFO:tasks.ceph.mds.f.smithi186.stdout:starting mds.f at 2022-04-04T01:52:19.204 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:19.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:19.203517+0000) 2022-04-04T01:52:19.266 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:19.264+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:19.426 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:19.425+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:20.204 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:20.202+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:20.203694+0000) 2022-04-04T01:52:20.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:20.306+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:20.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:20.453+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:21.204 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:21.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:21.203884+0000) 2022-04-04T01:52:21.262 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:21.260+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:21.442 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:21.441+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:22.205 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:22.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:22.204038+0000) 2022-04-04T01:52:22.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:22.251+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:22.490 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:22.488+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:23.205 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:23.203+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:23.204259+0000) 2022-04-04T01:52:23.219 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:23.218+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 70 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:23.495 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:23.493+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:24.205 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:24.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:24.204419+0000) 2022-04-04T01:52:24.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:24.245+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:24.531 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:24.530+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:25.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:25.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:25.204619+0000) 2022-04-04T01:52:25.264 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:25.262+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:25.355 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:52:25.359 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:52:25.551 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:25.549+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:26.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:26.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:26.204873+0000) 2022-04-04T01:52:26.217 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:26.215+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:26.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:26.545+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:27.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:27.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:27.205101+0000) 2022-04-04T01:52:27.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:27.247+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:27.560 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:27.558+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:28.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:28.205+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:28.205310+0000) 2022-04-04T01:52:28.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:28.253+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:28.565 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:28.562+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:29.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:29.205+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:29.205453+0000) 2022-04-04T01:52:29.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:29.296+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:29.601 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:29.599+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:30.206 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:30.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:30.205687+0000) 2022-04-04T01:52:30.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:30.296+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:30.394 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:52:30.392+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:52:30.561 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:30.558+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:31.207 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:31.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:31.205893+0000) 2022-04-04T01:52:31.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:31.248+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:31.536 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:31.534+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:32.207 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:32.204+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:32.206059+0000) 2022-04-04T01:52:32.291 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:32.288+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:32.573 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:32.571+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:33.207 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:33.205+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:33.206272+0000) 2022-04-04T01:52:33.285 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:33.283+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:33.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:33.539+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:34.207 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:34.205+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:34.206444+0000) 2022-04-04T01:52:34.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:34.323+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:34.562 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:34.560+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:35.207 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:35.205+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:35.206601+0000) 2022-04-04T01:52:35.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:35.372+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:35.552 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:35.550+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:36.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:36.205+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:36.206810+0000) 2022-04-04T01:52:36.356 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:36.354+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:36.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:36.527+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:37.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:37.206+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:37.206925+0000) 2022-04-04T01:52:37.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:37.324+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:37.524 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:37.522+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:38.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:38.206+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:38.207112+0000) 2022-04-04T01:52:38.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:38.300+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:38.549 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:38.547+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:39.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:39.206+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:39.207258+0000) 2022-04-04T01:52:39.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:39.334+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:39.533 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:39.530+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:40.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:40.206+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:40.207434+0000) 2022-04-04T01:52:40.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:40.359+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:40.556 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:40.554+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:41.208 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:41.206+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:41.207649+0000) 2022-04-04T01:52:41.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:41.321+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:41.548 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:41.546+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:42.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:42.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:42.207810+0000) 2022-04-04T01:52:42.354 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:42.352+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:42.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:42.585+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:43.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:43.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:43.208026+0000) 2022-04-04T01:52:43.386 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:43.384+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:43.563 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:43.561+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:44.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:44.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:44.208233+0000) 2022-04-04T01:52:44.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:44.349+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:44.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:44.593+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:45.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:45.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:45.208370+0000) 2022-04-04T01:52:45.361 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:45.359+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:45.410 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:52:45.408+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:52:45.567 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:45.565+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:46.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:46.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:46.208526+0000) 2022-04-04T01:52:46.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:46.361+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:46.608 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:46.606+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:47.209 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:47.207+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:47.208715+0000) 2022-04-04T01:52:47.399 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:47.397+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:47.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:47.637+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:48.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:48.208+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:48.208889+0000) 2022-04-04T01:52:48.396 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:48.393+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:48.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:48.678+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:49.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:49.208+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:49.209099+0000) 2022-04-04T01:52:49.431 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:49.429+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:49.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:49.711+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:50.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:50.208+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:50.209253+0000) 2022-04-04T01:52:50.428 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:50.426+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:50.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:50.729+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:51.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:51.208+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:51.209363+0000) 2022-04-04T01:52:51.402 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:51.400+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:51.724 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:51.722+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:52.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:52.208+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:52.209499+0000) 2022-04-04T01:52:52.446 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:52.444+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:52.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:52.693+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:53.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:53.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:53.209626+0000) 2022-04-04T01:52:53.401 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:53.399+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:53.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:53.700+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:54.210 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:54.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:54.209748+0000) 2022-04-04T01:52:54.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:54.365+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:54.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:54.711+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:55.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:55.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:55.209909+0000) 2022-04-04T01:52:55.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:55.411+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:55.502 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:52:55.507 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:52:55.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:55.677+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:56.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:56.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:56.210112+0000) 2022-04-04T01:52:56.422 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:56.420+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:56.661 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:56.659+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:57.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:57.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:57.210237+0000) 2022-04-04T01:52:57.469 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:57.467+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:57.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:57.639+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:58.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:58.209+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:58.210434+0000) 2022-04-04T01:52:58.503 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:58.501+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:58.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:58.646+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:52:59.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:52:59.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:52:59.210636+0000) 2022-04-04T01:52:59.494 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:52:59.492+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:52:59.673 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:52:59.671+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:00.211 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:00.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:00.210835+0000) 2022-04-04T01:53:00.420 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:53:00.418+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:53:00.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:00.498+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:00.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:00.714+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:01.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:01.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:01.211036+0000) 2022-04-04T01:53:01.542 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:01.540+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:01.713 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:01.711+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:02.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:02.210+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:02.211239+0000) 2022-04-04T01:53:02.589 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:02.587+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:02.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:02.747+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:03.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:03.211+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:03.211458+0000) 2022-04-04T01:53:03.565 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:03.563+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:03.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:03.775+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:04.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:04.211+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:04.211662+0000) 2022-04-04T01:53:04.606 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:04.604+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:04.736 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:04.734+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:05.212 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:05.211+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:05.211855+0000) 2022-04-04T01:53:05.571 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:05.569+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:05.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:05.697+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:06.213 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:06.211+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:06.212055+0000) 2022-04-04T01:53:06.558 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:06.556+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:06.747 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:06.746+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:07.214 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:07.211+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:07.212263+0000) 2022-04-04T01:53:07.570 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:07.568+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:07.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:07.780+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:08.213 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:08.212+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:08.212467+0000) 2022-04-04T01:53:08.578 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:08.576+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:08.754 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:08.753+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:09.213 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:09.212+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:09.212691+0000) 2022-04-04T01:53:09.621 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:09.619+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:09.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:09.711+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:10.214 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:10.212+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:10.212857+0000) 2022-04-04T01:53:10.653 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:10.652+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:10.673 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:10.672+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:11.214 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:11.212+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:11.213065+0000) 2022-04-04T01:53:11.652 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:11.650+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:11.674 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:11.672+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:12.214 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:12.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:12.213269+0000) 2022-04-04T01:53:12.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:12.700+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:12.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:12.716+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:13.214 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:13.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:13.213485+0000) 2022-04-04T01:53:13.715 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:13.713+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:13.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:13.729+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:14.214 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:14.212+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:14.213698+0000) 2022-04-04T01:53:14.703 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:14.701+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:14.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:14.751+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:15.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:15.212+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:15.213904+0000) 2022-04-04T01:53:15.428 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:53:15.425+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:53:15.689 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:15.688+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:15.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:15.775+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:16.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:16.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:16.214108+0000) 2022-04-04T01:53:16.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:16.679+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:16.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:16.777+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:17.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:17.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:17.214326+0000) 2022-04-04T01:53:17.730 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:17.729+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:17.754 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:17.753+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:18.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:18.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:18.214529+0000) 2022-04-04T01:53:18.689 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:18.688+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:18.706 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:18.705+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:19.215 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:19.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:19.214736+0000) 2022-04-04T01:53:19.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:19.667+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:19.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:19.676+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:20.216 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:20.213+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:20.214936+0000) 2022-04-04T01:53:20.677 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:20.675+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:20.695 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:20.693+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:21.216 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:21.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:21.215088+0000) 2022-04-04T01:53:21.671 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:21.668+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:21.722 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:21.720+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:22.216 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:22.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:22.215283+0000) 2022-04-04T01:53:22.663 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:22.660+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:22.756 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:22.754+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:23.216 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:23.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:23.215493+0000) 2022-04-04T01:53:23.639 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:23.637+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:23.749 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:23.747+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:24.216 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:24.214+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:24.215645+0000) 2022-04-04T01:53:24.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:24.683+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:24.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:24.713+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:25.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:25.215+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:25.215847+0000) 2022-04-04T01:53:25.646 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:25.643+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:25.654 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:53:25.664 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:53:25.718 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:25.716+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:26.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:26.215+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:26.216081+0000) 2022-04-04T01:53:26.669 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:26.667+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:26.678 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:26.675+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:27.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:27.215+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:27.216318+0000) 2022-04-04T01:53:27.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:27.635+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:27.693 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:27.691+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:28.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:28.215+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:28.216544+0000) 2022-04-04T01:53:28.627 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:28.624+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:28.743 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:28.741+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:29.217 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:29.215+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:29.216700+0000) 2022-04-04T01:53:29.676 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:29.674+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:29.696 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:29.694+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:30.218 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:30.216+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:30.216910+0000) 2022-04-04T01:53:30.444 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:53:30.442+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:53:30.673 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:30.670+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:30.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:30.713+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:31.218 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:31.216+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:31.217119+0000) 2022-04-04T01:53:31.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:31.682+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:31.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:31.727+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:32.218 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:32.216+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:32.217298+0000) 2022-04-04T01:53:32.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:32.688+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:32.712 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:32.710+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:33.218 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:33.216+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:33.217479+0000) 2022-04-04T01:53:33.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:33.660+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:33.711 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:33.709+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:34.218 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:34.217+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:34.217695+0000) 2022-04-04T01:53:34.681 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:34.679+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:34.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:34.683+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:35.219 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:35.217+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:35.217913+0000) 2022-04-04T01:53:35.653 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:35.651+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:35.707 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:35.705+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:36.219 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:36.217+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:36.218082+0000) 2022-04-04T01:53:36.665 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:36.663+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:36.702 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:36.700+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:37.219 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:37.217+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:37.218294+0000) 2022-04-04T01:53:37.619 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:37.616+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:37.728 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:37.726+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:38.219 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:38.217+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:38.218497+0000) 2022-04-04T01:53:38.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:38.612+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:38.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:38.678+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:39.219 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:39.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:39.218650+0000) 2022-04-04T01:53:39.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:39.635+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:39.650 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:39.647+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:40.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:40.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:40.218823+0000) 2022-04-04T01:53:40.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:40.677+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:40.698 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:40.696+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:41.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:41.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:41.219036+0000) 2022-04-04T01:53:41.678 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:41.676+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:41.744 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:41.742+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:42.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:42.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:42.219244+0000) 2022-04-04T01:53:42.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:42.630+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:42.746 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:42.744+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:43.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:43.218+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:43.219439+0000) 2022-04-04T01:53:43.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:43.636+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:43.789 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:43.786+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:44.220 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:44.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:44.219651+0000) 2022-04-04T01:53:44.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:44.655+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:44.820 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:44.818+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:45.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:45.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:45.219879+0000) 2022-04-04T01:53:45.460 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:53:45.458+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:53:45.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:45.630+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:45.773 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:45.771+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:46.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:46.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:46.220081+0000) 2022-04-04T01:53:46.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:46.667+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:46.794 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:46.792+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:47.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:47.219+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:47.220316+0000) 2022-04-04T01:53:47.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:47.627+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:47.843 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:47.842+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:48.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:48.220+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:48.220524+0000) 2022-04-04T01:53:48.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:48.662+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:48.805 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:48.802+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:49.221 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:49.220+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:49.220730+0000) 2022-04-04T01:53:49.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:49.697+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:49.788 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:49.786+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:50.222 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:50.220+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:50.220920+0000) 2022-04-04T01:53:50.734 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:50.732+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:50.829 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:50.827+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:51.222 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:51.220+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:51.221096+0000) 2022-04-04T01:53:51.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:51.685+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:51.800 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:51.798+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:52.222 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:52.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:52.221308+0000) 2022-04-04T01:53:52.728 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:52.726+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:52.806 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:52.804+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:53.222 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:53.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:53.221495+0000) 2022-04-04T01:53:53.685 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:53.683+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:53.854 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:53.852+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:54.222 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:54.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:54.221701+0000) 2022-04-04T01:53:54.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:54.686+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:54.851 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:54.849+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:55.223 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:55.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:55.221912+0000) 2022-04-04T01:53:55.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:55.736+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:55.784 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:53:55.787 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:53:55.900 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:55.898+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:56.223 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:56.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:56.222098+0000) 2022-04-04T01:53:56.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:56.691+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:56.859 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:56.857+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:57.223 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:57.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:57.222322+0000) 2022-04-04T01:53:57.675 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:57.674+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:57.888 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:57.886+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:58.223 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:58.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:58.222556+0000) 2022-04-04T01:53:58.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:58.689+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:58.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:58.931+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:53:59.223 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:53:59.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:53:59.222765+0000) 2022-04-04T01:53:59.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:53:59.691+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:53:59.919 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:53:59.918+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:00.224 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:00.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:00.222951+0000) 2022-04-04T01:54:00.476 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:54:00.473+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:54:00.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:00.656+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:00.900 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:00.899+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:01.224 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:01.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:01.223139+0000) 2022-04-04T01:54:01.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:01.619+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:01.854 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:01.853+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:02.224 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:02.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:02.223350+0000) 2022-04-04T01:54:02.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:02.656+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:02.845 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:02.843+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:03.224 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:03.221+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:03.223582+0000) 2022-04-04T01:54:03.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:03.645+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:03.892 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:03.890+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:04.225 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:04.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:04.223830+0000) 2022-04-04T01:54:04.684 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:04.682+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:04.926 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:04.925+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:05.225 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:05.223+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:05.224062+0000) 2022-04-04T01:54:05.704 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:05.702+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:05.966 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:05.964+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:06.225 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:06.222+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:06.224223+0000) 2022-04-04T01:54:06.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:06.656+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:06.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:06.995+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:07.225 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:07.223+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:07.224393+0000) 2022-04-04T01:54:07.690 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:07.688+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:08.011 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:08.009+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:08.225 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:08.223+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:08.224604+0000) 2022-04-04T01:54:08.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:08.716+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:09.046 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:09.044+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:09.226 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:09.223+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:09.224840+0000) 2022-04-04T01:54:09.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:09.698+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:10.055 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:10.054+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:10.226 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:10.224+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:10.225051+0000) 2022-04-04T01:54:10.668 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:10.667+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:11.033 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:11.032+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:11.226 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:11.224+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:11.225258+0000) 2022-04-04T01:54:11.680 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:11.678+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:12.037 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:12.035+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:12.226 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:12.224+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:12.225414+0000) 2022-04-04T01:54:12.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:12.642+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:13.005 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:13.004+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:13.226 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:13.224+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:13.225635+0000) 2022-04-04T01:54:13.671 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:13.669+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:14.001 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:14.000+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:14.226 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:14.225+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:14.225813+0000) 2022-04-04T01:54:14.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:14.710+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:15.023 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:15.021+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:15.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:15.225+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:15.225969+0000) 2022-04-04T01:54:15.491 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:54:15.490+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:54:15.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:15.721+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:15.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:15.989+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:16.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:16.225+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:16.226180+0000) 2022-04-04T01:54:16.749 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:16.746+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:16.988 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:16.986+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:17.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:17.225+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:17.226382+0000) 2022-04-04T01:54:17.798 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:17.795+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:18.019 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:18.016+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:18.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:18.225+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:18.226602+0000) 2022-04-04T01:54:18.798 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:18.795+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:18.973 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:18.970+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:19.227 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:19.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:19.226827+0000) 2022-04-04T01:54:19.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:19.799+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:19.960 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:19.957+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:20.228 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:20.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:20.227021+0000) 2022-04-04T01:54:20.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:20.838+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:20.914 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:20.912+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:21.228 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:21.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:21.227237+0000) 2022-04-04T01:54:21.871 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:21.868+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:21.964 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:21.961+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:22.228 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:22.226+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:22.227463+0000) 2022-04-04T01:54:22.834 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:22.831+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:22.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:22.994+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:23.228 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:23.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:23.227675+0000) 2022-04-04T01:54:23.849 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:23.846+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:24.024 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:24.021+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:24.228 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:24.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:24.227889+0000) 2022-04-04T01:54:24.866 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:24.863+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:24.978 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:24.973+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:25.229 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:25.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:25.228117+0000) 2022-04-04T01:54:25.885 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:25.883+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:25.931 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:54:25.935 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:54:25.960 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:25.957+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:26.229 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:26.227+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:26.228374+0000) 2022-04-04T01:54:26.884 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:26.882+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:26.953 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:26.951+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:27.229 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:27.228+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:27.228584+0000) 2022-04-04T01:54:27.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:27.833+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:27.935 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:27.933+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:28.230 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:28.228+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:28.228821+0000) 2022-04-04T01:54:28.821 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:28.819+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:28.918 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:28.915+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:29.230 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:29.228+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:29.229024+0000) 2022-04-04T01:54:29.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:29.836+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:29.895 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:29.893+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:30.230 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:30.228+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:30.229227+0000) 2022-04-04T01:54:30.507 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:54:30.505+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:54:30.827 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:30.825+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:30.867 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:30.864+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:31.230 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:31.229+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:31.229439+0000) 2022-04-04T01:54:31.866 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:31.863+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:31.894 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:31.892+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:32.230 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:32.229+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:32.229639+0000) 2022-04-04T01:54:32.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:32.819+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:32.938 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:32.936+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:33.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:33.229+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:33.229851+0000) 2022-04-04T01:54:33.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:33.798+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:33.949 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:33.947+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:34.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:34.229+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:34.230011+0000) 2022-04-04T01:54:34.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:34.798+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:34.929 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:34.927+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:35.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:35.229+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:35.230187+0000) 2022-04-04T01:54:35.774 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:35.772+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:35.933 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:35.931+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:36.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:36.230+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:36.230385+0000) 2022-04-04T01:54:36.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:36.751+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:36.901 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:36.899+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:37.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:37.230+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:37.230537+0000) 2022-04-04T01:54:37.751 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:37.748+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:37.881 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:37.879+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:38.231 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:38.230+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:38.230754+0000) 2022-04-04T01:54:38.748 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:38.746+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:38.841 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:38.839+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:39.232 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:39.230+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:39.231006+0000) 2022-04-04T01:54:39.744 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:39.742+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:39.830 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:39.828+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:40.232 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:40.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:40.231237+0000) 2022-04-04T01:54:40.763 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:40.760+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:40.855 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:40.853+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:41.232 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:41.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:41.231427+0000) 2022-04-04T01:54:41.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:41.737+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:41.879 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:41.877+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:42.232 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:42.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:42.231632+0000) 2022-04-04T01:54:42.781 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:42.779+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:42.883 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:42.881+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:43.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:43.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:43.231862+0000) 2022-04-04T01:54:43.800 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:43.798+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:43.925 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:43.922+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:44.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:44.230+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:44.232020+0000) 2022-04-04T01:54:44.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:44.760+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:44.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:44.946+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:45.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:45.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:45.232179+0000) 2022-04-04T01:54:45.523 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:54:45.520+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:54:45.762 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:45.760+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:45.902 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:45.900+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:46.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:46.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:46.232391+0000) 2022-04-04T01:54:46.734 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:46.732+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:46.909 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:46.907+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:47.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:47.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:47.232595+0000) 2022-04-04T01:54:47.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:47.697+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:47.865 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:47.863+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:48.233 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:48.231+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:48.232799+0000) 2022-04-04T01:54:48.722 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:48.720+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:48.901 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:48.899+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:49.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:49.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:49.233004+0000) 2022-04-04T01:54:49.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:49.752+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:49.882 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:49.880+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:50.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:50.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:50.233137+0000) 2022-04-04T01:54:50.740 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:50.737+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:50.863 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:50.860+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:51.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:51.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:51.233262+0000) 2022-04-04T01:54:51.692 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:51.690+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:51.901 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:51.899+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:52.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:52.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:52.233433+0000) 2022-04-04T01:54:52.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:52.723+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:52.905 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:52.904+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:53.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:53.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:53.233567+0000) 2022-04-04T01:54:53.705 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:53.703+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:53.911 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:53.909+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:54.234 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:54.232+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:54.233730+0000) 2022-04-04T01:54:54.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:54.689+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:54.948 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:54.947+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:55.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:55.233+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:55.233884+0000) 2022-04-04T01:54:55.689 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:55.688+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:55.989 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:55.987+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:56.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:56.233+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:56.234085+0000) 2022-04-04T01:54:56.281 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:54:56.285 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:54:56.655 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:56.654+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:57.017 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:57.015+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:57.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:57.233+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:57.234285+0000) 2022-04-04T01:54:57.647 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:57.646+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:58.021 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:58.019+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:58.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:58.233+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:58.234493+0000) 2022-04-04T01:54:58.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:58.615+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:58.975 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:58.974+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:54:59.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:54:59.233+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:54:59.234664+0000) 2022-04-04T01:54:59.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:54:59.622+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:54:59.943 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:54:59.941+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:00.235 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:00.234+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:00.234838+0000) 2022-04-04T01:55:00.539 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:55:00.537+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:55:00.643 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:00.641+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:00.987 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:00.985+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:01.236 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:01.234+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:01.235048+0000) 2022-04-04T01:55:01.626 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:01.625+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:02.028 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:02.026+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:02.236 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:02.234+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:02.235233+0000) 2022-04-04T01:55:02.662 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:02.660+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:03.033 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:03.031+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:03.236 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:03.234+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:03.235359+0000) 2022-04-04T01:55:03.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:03.620+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:04.071 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:04.070+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:04.236 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:04.234+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:04.235536+0000) 2022-04-04T01:55:04.585 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:04.583+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:05.101 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:05.099+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:05.236 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:05.235+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:05.235738+0000) 2022-04-04T01:55:05.537 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:05.536+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:06.149 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:06.148+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:06.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:06.235+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:06.235960+0000) 2022-04-04T01:55:06.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:06.527+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:07.123 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:07.121+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:07.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:07.235+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:07.236113+0000) 2022-04-04T01:55:07.538 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:07.536+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:08.130 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:08.129+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:08.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:08.235+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:08.236265+0000) 2022-04-04T01:55:08.572 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:08.571+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:09.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:09.177+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:09.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:09.235+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:09.236482+0000) 2022-04-04T01:55:09.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:09.619+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:10.228 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:10.226+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:10.237 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:10.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:10.236683+0000) 2022-04-04T01:55:10.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:10.590+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:11.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:11.177+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:11.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:11.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:11.236809+0000) 2022-04-04T01:55:11.615 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:11.612+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:12.169 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:12.166+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:12.238 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:12.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:12.236991+0000) 2022-04-04T01:55:12.568 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:12.566+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:13.149 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:13.147+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:13.238 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:13.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:13.237129+0000) 2022-04-04T01:55:13.574 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:13.571+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:14.198 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:14.195+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:14.238 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:14.236+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:14.237317+0000) 2022-04-04T01:55:14.559 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:14.557+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:15.173 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:15.171+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:15.238 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:15.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:15.237469+0000) 2022-04-04T01:55:15.540 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:55:15.538+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:55:15.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:15.542+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:16.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:16.205+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:16.238 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:16.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:16.237663+0000) 2022-04-04T01:55:16.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:16.539+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:17.229 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:17.227+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:17.238 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:17.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:17.237868+0000) 2022-04-04T01:55:17.529 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:17.527+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:18.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:18.190+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:18.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:18.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:18.238010+0000) 2022-04-04T01:55:18.515 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:18.513+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:19.158 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:19.156+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:19.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:19.237+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:19.238173+0000) 2022-04-04T01:55:19.555 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:19.553+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:20.143 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:20.141+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:20.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:20.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:20.238363+0000) 2022-04-04T01:55:20.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:20.602+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:21.158 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:21.156+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:21.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:21.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:21.238576+0000) 2022-04-04T01:55:21.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:21.585+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:22.191 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:22.189+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:22.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:22.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:22.238729+0000) 2022-04-04T01:55:22.636 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:22.633+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:23.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:23.166+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 36 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:23.239 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:23.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:23.238954+0000) 2022-04-04T01:55:23.609 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:23.607+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:24.207 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:24.205+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:24.240 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:24.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:24.239119+0000) 2022-04-04T01:55:24.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:24.629+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:25.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:25.228+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:25.240 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:25.239+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:25.239300+0000) 2022-04-04T01:55:25.670 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:25.668+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:26.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:26.196+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:26.240 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:26.239+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:26.239462+0000) 2022-04-04T01:55:26.354 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:55:26.357 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:55:26.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:26.690+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:27.169 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:27.167+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:27.240 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:27.239+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:27.239650+0000) 2022-04-04T01:55:27.673 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:27.670+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:28.195 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:28.193+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:28.241 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:28.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:28.239859+0000) 2022-04-04T01:55:28.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:28.695+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:29.171 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:29.169+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:29.241 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:29.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:29.240052+0000) 2022-04-04T01:55:29.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:29.696+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:30.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:30.209+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:30.241 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:30.239+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:30.240259+0000) 2022-04-04T01:55:30.548 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:55:30.545+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:55:30.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:30.713+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:31.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:31.209+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:31.241 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:31.239+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:31.240452+0000) 2022-04-04T01:55:31.721 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:31.718+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:32.208 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:32.206+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:32.241 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:32.238+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:32.240655+0000) 2022-04-04T01:55:32.746 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:32.744+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:33.174 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:33.171+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:33.241 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:33.239+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:33.240811+0000) 2022-04-04T01:55:33.757 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:33.755+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:34.163 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:34.161+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:34.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:34.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:34.240987+0000) 2022-04-04T01:55:34.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:34.741+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:35.195 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:35.193+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:35.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:35.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:35.241136+0000) 2022-04-04T01:55:35.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:35.742+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:36.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:36.159+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:36.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:36.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:36.241331+0000) 2022-04-04T01:55:36.751 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:36.748+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:37.171 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:37.169+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:37.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:37.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:37.241445+0000) 2022-04-04T01:55:37.738 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:37.736+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:38.148 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:38.146+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:38.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:38.240+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:38.241573+0000) 2022-04-04T01:55:38.750 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:38.748+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:39.144 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:39.141+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:39.242 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:39.241+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:39.241798+0000) 2022-04-04T01:55:39.789 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:39.788+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:40.176 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:40.174+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:40.243 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:40.241+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:40.241981+0000) 2022-04-04T01:55:40.749 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:40.748+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:41.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:41.209+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:41.391 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:41.241+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:41.242191+0000) 2022-04-04T01:55:41.794 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:41.792+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:42.243 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:42.241+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:42.242403+0000) 2022-04-04T01:55:42.245 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:42.242+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:42.767 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:42.765+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:43.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:43.240+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:43.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:43.241+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:43.242566+0000) 2022-04-04T01:55:43.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:43.724+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:44.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:44.242+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:44.242766+0000) 2022-04-04T01:55:44.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:44.275+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:44.773 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:44.771+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:45.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:45.242+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:45.243008+0000) 2022-04-04T01:55:45.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:45.271+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:45.564 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:55:45.562+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:55:45.815 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:45.813+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:46.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:46.236+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:46.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:46.242+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:46.243206+0000) 2022-04-04T01:55:46.811 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:46.809+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:47.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:47.208+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:47.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:47.242+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:47.243429+0000) 2022-04-04T01:55:47.816 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:47.814+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:48.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:48.242+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:48.243654+0000) 2022-04-04T01:55:48.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:48.245+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:48.840 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:48.838+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:49.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:49.214+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:49.244 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:49.243+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:49.243869+0000) 2022-04-04T01:55:49.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:49.886+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:50.213 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:50.211+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:50.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:50.243+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:50.244035+0000) 2022-04-04T01:55:50.838 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:50.837+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:51.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:51.238+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:51.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:51.243+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:51.244190+0000) 2022-04-04T01:55:51.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:51.846+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:52.222 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:52.220+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:52.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:52.243+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:52.244366+0000) 2022-04-04T01:55:52.826 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:52.824+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:53.174 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:53.173+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:53.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:53.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:53.244528+0000) 2022-04-04T01:55:53.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:53.842+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:54.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:54.223+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:54.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:54.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:54.244736+0000) 2022-04-04T01:55:54.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:54.863+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:55.179 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:55.178+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:55.245 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:55.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:55.244961+0000) 2022-04-04T01:55:55.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:55.841+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:56.149 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:56.148+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:56.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:56.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:56.245156+0000) 2022-04-04T01:55:56.610 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:55:56.614 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:55:56.879 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:56.878+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:57.123 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:57.121+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:57.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:57.244+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:57.245321+0000) 2022-04-04T01:55:57.895 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:57.894+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:58.094 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:58.092+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:58.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:58.245+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:58.245531+0000) 2022-04-04T01:55:58.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:58.889+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:55:59.066 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:55:59.064+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:55:59.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:55:59.245+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:55:59.245660+0000) 2022-04-04T01:55:59.906 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:55:59.905+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:00.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:00.110+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:00.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:00.245+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:00.245860+0000) 2022-04-04T01:56:00.580 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:56:00.578+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:56:00.865 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:00.863+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:01.145 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:01.143+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:01.246 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:01.245+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:01.245972+0000) 2022-04-04T01:56:01.884 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:01.882+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:02.168 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:02.166+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:02.247 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:02.245+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:02.246171+0000) 2022-04-04T01:56:02.843 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:02.842+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:03.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:03.210+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:03.247 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:03.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:03.246358+0000) 2022-04-04T01:56:03.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:03.873+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:04.182 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:04.179+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:04.247 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:04.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:04.246502+0000) 2022-04-04T01:56:04.880 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:04.878+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:05.211 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:05.209+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:05.247 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:05.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:05.246625+0000) 2022-04-04T01:56:05.848 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:05.846+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:06.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:06.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:06.246799+0000) 2022-04-04T01:56:06.251 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:06.247+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:06.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:06.885+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:07.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:07.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:07.246907+0000) 2022-04-04T01:56:07.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:07.269+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:07.886 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:07.883+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:08.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:08.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:08.247074+0000) 2022-04-04T01:56:08.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:08.304+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:08.887 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:08.884+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:09.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:09.247+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:09.247254+0000) 2022-04-04T01:56:09.295 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:09.293+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:09.878 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:09.876+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:10.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:10.247+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:10.247480+0000) 2022-04-04T01:56:10.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:10.265+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:10.901 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:10.897+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:11.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:11.235+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:11.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:11.247+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:11.247701+0000) 2022-04-04T01:56:11.933 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:11.930+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:12.248 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:12.246+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:12.247906+0000) 2022-04-04T01:56:12.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:12.256+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:12.919 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:12.916+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:13.249 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:13.247+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:13.248110+0000) 2022-04-04T01:56:13.269 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:13.267+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:13.875 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:13.873+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:14.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:14.245+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:14.249 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:14.247+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:14.248271+0000) 2022-04-04T01:56:14.873 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:14.870+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:15.201 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:15.199+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:15.249 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:15.247+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:15.248452+0000) 2022-04-04T01:56:15.596 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:56:15.593+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:56:15.828 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:15.826+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:16.177 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:16.174+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:16.249 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:16.247+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:16.248657+0000) 2022-04-04T01:56:16.823 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:16.821+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:17.165 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:17.162+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:17.250 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:17.247+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:17.248856+0000) 2022-04-04T01:56:17.862 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:17.860+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:18.206 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:18.204+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:18.249 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:18.248+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:18.249032+0000) 2022-04-04T01:56:18.822 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:18.820+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:19.174 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:19.172+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:19.250 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:19.248+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:19.249230+0000) 2022-04-04T01:56:19.853 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:19.851+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:20.136 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:20.134+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:20.250 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:20.248+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:20.249343+0000) 2022-04-04T01:56:20.846 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:20.844+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:21.107 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:21.105+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:21.250 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:21.248+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:21.249546+0000) 2022-04-04T01:56:21.893 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:21.891+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:22.099 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:22.096+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:22.250 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:22.248+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:22.249681+0000) 2022-04-04T01:56:22.937 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:22.935+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:23.101 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:23.099+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:23.251 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:23.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:23.249933+0000) 2022-04-04T01:56:23.942 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:23.940+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:24.053 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:24.051+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:24.251 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:24.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:24.250147+0000) 2022-04-04T01:56:24.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:24.921+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:25.045 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:25.042+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:25.251 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:25.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:25.250363+0000) 2022-04-04T01:56:25.890 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:25.888+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:26.065 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:26.063+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:26.251 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:26.249+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:26.250544+0000) 2022-04-04T01:56:26.782 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:56:26.786 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:56:26.850 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:26.848+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:27.020 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:27.018+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:27.251 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:27.250+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:27.250806+0000) 2022-04-04T01:56:27.880 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:27.878+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:28.056 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:28.054+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:28.252 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:28.250+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:28.251010+0000) 2022-04-04T01:56:28.888 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:28.886+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:29.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:29.014+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:29.252 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:29.250+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:29.251234+0000) 2022-04-04T01:56:29.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:29.840+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:30.021 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:30.019+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:30.252 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:30.250+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:30.251435+0000) 2022-04-04T01:56:30.612 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:56:30.610+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:56:30.847 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:30.845+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:30.979 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:30.976+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:31.252 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:31.250+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:31.251587+0000) 2022-04-04T01:56:31.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:31.799+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:31.970 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:31.968+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:32.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:32.251+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:32.251820+0000) 2022-04-04T01:56:32.788 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:32.786+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:32.944 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:32.942+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:33.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:33.251+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:33.252006+0000) 2022-04-04T01:56:33.807 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:33.804+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:33.988 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:33.986+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:34.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:34.251+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:34.252211+0000) 2022-04-04T01:56:34.842 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:34.840+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:34.992 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:34.990+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:35.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:35.251+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:35.252378+0000) 2022-04-04T01:56:35.806 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:35.804+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:36.029 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:36.027+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:36.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:36.252+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:36.252603+0000) 2022-04-04T01:56:36.765 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:36.763+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:37.040 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:37.038+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:37.253 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:37.252+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:37.252830+0000) 2022-04-04T01:56:37.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:37.730+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:38.045 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:38.043+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:38.254 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:38.252+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:38.253019+0000) 2022-04-04T01:56:38.702 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:38.700+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:39.074 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:39.072+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:39.254 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:39.252+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:39.253236+0000) 2022-04-04T01:56:39.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:39.710+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:40.044 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:40.041+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:40.254 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:40.252+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:40.253405+0000) 2022-04-04T01:56:40.722 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:40.720+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:41.053 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:41.051+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:41.254 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:41.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:41.253576+0000) 2022-04-04T01:56:41.687 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:41.685+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:42.076 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:42.074+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:42.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:42.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:42.253802+0000) 2022-04-04T01:56:42.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:42.677+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:43.079 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:43.077+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:43.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:43.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:43.254011+0000) 2022-04-04T01:56:43.700 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:43.698+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:44.046 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:44.045+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:44.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:44.253+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:44.254194+0000) 2022-04-04T01:56:44.699 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:44.697+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:45.095 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:45.094+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:45.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:45.254+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:45.254370+0000) 2022-04-04T01:56:45.616 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:56:45.614+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:56:45.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:45.730+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:46.132 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:46.130+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:46.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:46.254+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:46.254575+0000) 2022-04-04T01:56:46.747 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:46.746+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:47.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:47.149+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:47.255 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:47.254+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:47.254841+0000) 2022-04-04T01:56:47.711 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:47.709+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:48.186 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:48.184+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:48.256 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:48.254+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:48.254955+0000) 2022-04-04T01:56:48.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:48.715+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:49.139 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:49.137+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:49.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:49.254+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:49.255158+0000) 2022-04-04T01:56:49.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:49.667+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:50.155 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:50.153+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:50.256 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:50.255+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:50.255370+0000) 2022-04-04T01:56:50.635 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:50.633+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:51.149 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:51.148+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:51.256 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:51.255+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:51.255608+0000) 2022-04-04T01:56:51.621 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:51.620+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:52.183 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:52.181+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:52.256 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:52.255+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:52.255844+0000) 2022-04-04T01:56:52.633 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:52.631+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:53.139 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:53.137+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:53.257 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:53.255+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:53.256030+0000) 2022-04-04T01:56:53.595 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:53.594+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:54.137 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:54.136+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:54.257 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:54.256+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:54.256236+0000) 2022-04-04T01:56:54.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:54.604+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:55.094 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:55.093+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:55.257 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:55.255+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:55.256432+0000) 2022-04-04T01:56:55.588 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:55.586+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:56.081 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:56.080+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:56.257 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:56.255+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:56.256597+0000) 2022-04-04T01:56:56.570 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:56.569+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:56.930 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:56:56.934 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:56:57.033 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:57.032+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:57.257 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:57.255+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:57.256859+0000) 2022-04-04T01:56:57.526 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:57.524+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:57.987 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:57.986+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:58.258 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:58.256+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:58.257042+0000) 2022-04-04T01:56:58.532 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:58.530+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:58.981 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:58.978+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:56:59.258 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:56:59.256+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:56:59.257216+0000) 2022-04-04T01:56:59.511 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:56:59.509+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:56:59.994 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:56:59.991+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:00.258 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:00.256+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:00.257406+0000) 2022-04-04T01:57:00.518 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:00.516+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:00.619 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:57:00.616+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:57:00.987 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:00.984+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:01.258 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:01.256+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:01.257610+0000) 2022-04-04T01:57:01.502 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:01.500+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:01.971 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:01.968+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:02.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:02.256+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:02.257811+0000) 2022-04-04T01:57:02.475 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:02.473+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:02.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:02.992+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:03.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:03.257+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:03.257970+0000) 2022-04-04T01:57:03.502 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:03.499+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:04.041 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:04.039+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:04.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:04.257+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:04.258129+0000) 2022-04-04T01:57:04.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:04.452+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:05.040 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:05.037+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:05.259 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:05.257+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:05.258254+0000) 2022-04-04T01:57:05.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:05.413+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:05.996 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:05.993+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:06.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:06.257+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:06.258448+0000) 2022-04-04T01:57:06.462 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:06.460+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:07.016 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:07.014+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:07.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:07.257+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:07.258619+0000) 2022-04-04T01:57:07.432 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:07.429+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:08.018 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:08.016+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:08.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:08.258+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:08.258836+0000) 2022-04-04T01:57:08.450 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:08.448+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:09.028 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:09.025+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:09.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:09.258+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:09.258971+0000) 2022-04-04T01:57:09.456 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:09.454+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:10.071 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:10.068+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:10.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:10.258+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:10.259137+0000) 2022-04-04T01:57:10.457 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:10.455+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:11.112 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:11.109+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:11.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:11.258+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:11.259308+0000) 2022-04-04T01:57:11.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:11.421+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:12.154 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:12.151+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:12.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:12.258+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:12.259484+0000) 2022-04-04T01:57:12.435 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:12.432+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:13.194 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:13.191+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:13.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:13.258+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:13.259666+0000) 2022-04-04T01:57:13.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:13.412+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:14.152 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:14.150+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:14.260 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:14.259+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:14.259858+0000) 2022-04-04T01:57:14.364 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-04T01:57:14.362+0000 7f2fd6e1b700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T01:57:14.365 INFO:tasks.ceph.mds.c.smithi077.stderr:2022-04-04T01:57:14.362+0000 7f2fd661a700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2022-04-04T01:57:14.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:14.438+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:15.118 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:15.115+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:15.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:15.259+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:15.260053+0000) 2022-04-04T01:57:15.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:15.415+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:15.635 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:57:15.633+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:57:16.084 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:16.080+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:16.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:16.259+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:16.260258+0000) 2022-04-04T01:57:16.385 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:16.382+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:17.095 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:17.092+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:17.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:17.259+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:17.260386+0000) 2022-04-04T01:57:17.377 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:17.375+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:18.069 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:18.067+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:18.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:18.260+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:18.260613+0000) 2022-04-04T01:57:18.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:18.336+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:19.072 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:19.070+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:19.261 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:19.260+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:19.260812+0000) 2022-04-04T01:57:19.373 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:19.371+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:20.063 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:20.061+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:20.262 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:20.260+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:20.260975+0000) 2022-04-04T01:57:20.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:20.413+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:21.109 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:21.107+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:21.262 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:21.260+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:21.261166+0000) 2022-04-04T01:57:21.391 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:21.388+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:22.146 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:22.144+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:22.262 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:22.260+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:22.261372+0000) 2022-04-04T01:57:22.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:22.424+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:23.121 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:23.118+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:23.262 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:23.261+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:23.261533+0000) 2022-04-04T01:57:23.412 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:23.410+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:24.139 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:24.137+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:24.262 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:24.260+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:24.261724+0000) 2022-04-04T01:57:24.445 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:24.443+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:25.115 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:25.112+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:25.263 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:25.261+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:25.261902+0000) 2022-04-04T01:57:25.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:25.461+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:26.067 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:26.065+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:26.263 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:26.261+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:26.262112+0000) 2022-04-04T01:57:26.482 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:26.480+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:27.025 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:27.023+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:27.052 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:57:27.056 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:57:27.263 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:27.261+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:27.262353+0000) 2022-04-04T01:57:27.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:27.482+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:28.004 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:28.002+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:28.263 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:28.262+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:28.262561+0000) 2022-04-04T01:57:28.503 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:28.501+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:29.039 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:29.037+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:29.264 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:29.262+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:29.262762+0000) 2022-04-04T01:57:29.459 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:29.457+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:30.023 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:30.021+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:30.264 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:30.262+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:30.263012+0000) 2022-04-04T01:57:30.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:30.461+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:30.637 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:57:30.635+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:57:30.995 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:30.993+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:31.264 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:31.262+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:31.263217+0000) 2022-04-04T01:57:31.502 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:31.500+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:31.956 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:31.954+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:32.264 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:32.263+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:32.263434+0000) 2022-04-04T01:57:32.513 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:32.511+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:32.958 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:32.956+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:33.264 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:33.263+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:33.263673+0000) 2022-04-04T01:57:33.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:33.542+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:33.970 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:33.968+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:34.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:34.263+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:34.263873+0000) 2022-04-04T01:57:34.573 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:34.571+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:34.961 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:34.959+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:35.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:35.263+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:35.263994+0000) 2022-04-04T01:57:35.593 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:35.591+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:36.012 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:36.010+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:36.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:36.264+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:36.264178+0000) 2022-04-04T01:57:36.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:36.636+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:37.045 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:37.043+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:37.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:37.263+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:37.264362+0000) 2022-04-04T01:57:37.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:37.636+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:38.043 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:38.042+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:38.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:38.264+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:38.264563+0000) 2022-04-04T01:57:38.652 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:38.650+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:39.044 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:39.043+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:39.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:39.264+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:39.264671+0000) 2022-04-04T01:57:39.609 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:39.607+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:40.011 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:40.010+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:40.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:40.263+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:40.264835+0000) 2022-04-04T01:57:40.630 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:40.629+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:41.053 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:41.051+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:41.265 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:41.263+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:41.264986+0000) 2022-04-04T01:57:41.653 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:41.652+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:42.102 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:42.100+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:42.266 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:42.264+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:42.265165+0000) 2022-04-04T01:57:42.694 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:42.692+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:43.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:43.148+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:43.266 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:43.264+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:43.265369+0000) 2022-04-04T01:57:43.688 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:43.687+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:44.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:44.148+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:44.266 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:44.264+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:44.265576+0000) 2022-04-04T01:57:44.730 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:44.728+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:45.176 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:45.174+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:45.266 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:45.264+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:45.265791+0000) 2022-04-04T01:57:45.653 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:57:45.650+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:57:45.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:45.689+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:46.191 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:46.189+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:46.266 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:46.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:46.265947+0000) 2022-04-04T01:57:46.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:46.678+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:47.180 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:47.178+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:47.267 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:47.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:47.266165+0000) 2022-04-04T01:57:47.716 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:47.714+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:48.188 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:48.186+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:48.267 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:48.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:48.266369+0000) 2022-04-04T01:57:48.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:48.753+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:49.213 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:49.212+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:49.267 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:49.265+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:49.266602+0000) 2022-04-04T01:57:49.727 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:49.725+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:50.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:50.208+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:50.267 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:50.266+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:50.266853+0000) 2022-04-04T01:57:50.698 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:50.696+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:51.227 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:51.225+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:51.268 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:51.266+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:51.267074+0000) 2022-04-04T01:57:51.691 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:51.689+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:52.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:52.258+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:52.268 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:52.266+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:52.267287+0000) 2022-04-04T01:57:52.678 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:52.676+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:53.268 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:53.266+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:53.267496+0000) 2022-04-04T01:57:53.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:53.304+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:53.725 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:53.723+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:54.268 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:54.266+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:54.267667+0000) 2022-04-04T01:57:54.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:54.319+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:54.731 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:54.729+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:55.269 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:55.267+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:55.267903+0000) 2022-04-04T01:57:55.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:55.323+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:55.745 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:55.742+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:56.269 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:56.267+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:56.268113+0000) 2022-04-04T01:57:56.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:56.322+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:56.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:56.729+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:57.184 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:57:57.188 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:57:57.269 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:57.267+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:57.268353+0000) 2022-04-04T01:57:57.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:57.278+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:57.737 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:57.734+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:58.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:58.245+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:58.269 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:58.267+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:58.268542+0000) 2022-04-04T01:57:58.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:58.690+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:57:59.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:57:59.245+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:57:59.269 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:57:59.268+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:57:59.268763+0000) 2022-04-04T01:57:59.733 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:57:59.731+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:00.243 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:00.240+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:00.270 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:00.268+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:00.268935+0000) 2022-04-04T01:58:00.669 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:58:00.667+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:58:00.758 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:00.755+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:01.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:01.222+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:01.270 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:01.268+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:01.269140+0000) 2022-04-04T01:58:01.779 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:01.776+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:02.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:02.246+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:02.270 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:02.268+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:02.269351+0000) 2022-04-04T01:58:02.787 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:02.784+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:03.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:03.265+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:03.270 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:03.269+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:03.269612+0000) 2022-04-04T01:58:03.793 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:03.790+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:04.236 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:04.233+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:04.270 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:04.269+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:04.269814+0000) 2022-04-04T01:58:04.743 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:04.741+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:05.271 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:05.269+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:05.270002+0000) 2022-04-04T01:58:05.279 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:05.277+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:05.739 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:05.737+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:06.271 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:06.269+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:06.270227+0000) 2022-04-04T01:58:06.303 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:06.301+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:06.776 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:06.774+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:07.271 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:07.269+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:07.270440+0000) 2022-04-04T01:58:07.287 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:07.285+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:07.824 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:07.822+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:08.271 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:08.270+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:08.270641+0000) 2022-04-04T01:58:08.304 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:08.302+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:08.819 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:08.817+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:09.272 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:09.270+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:09.270847+0000) 2022-04-04T01:58:09.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:09.349+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:09.810 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:09.807+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:10.272 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:10.270+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:10.271044+0000) 2022-04-04T01:58:10.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:10.325+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:10.796 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:10.794+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:11.272 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:11.270+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:11.271250+0000) 2022-04-04T01:58:11.355 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:11.352+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:11.753 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:11.751+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:12.272 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:12.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:12.271382+0000) 2022-04-04T01:58:12.336 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:12.334+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:12.722 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:12.720+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:13.272 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:13.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:13.271558+0000) 2022-04-04T01:58:13.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:13.382+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:13.732 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:13.730+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:14.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:14.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:14.271713+0000) 2022-04-04T01:58:14.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:14.393+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:14.726 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:14.724+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:15.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:15.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:15.271869+0000) 2022-04-04T01:58:15.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:15.371+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:15.670 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:58:15.668+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:58:15.681 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:15.679+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:16.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:16.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:16.272040+0000) 2022-04-04T01:58:16.385 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:16.383+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:16.669 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:16.667+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:17.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:17.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:17.272212+0000) 2022-04-04T01:58:17.347 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:17.344+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:17.648 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:17.645+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:18.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:18.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:18.272353+0000) 2022-04-04T01:58:18.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:18.361+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:18.656 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:18.653+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:19.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:19.271+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:19.272475+0000) 2022-04-04T01:58:19.359 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:19.357+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:19.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:19.612+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:20.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:20.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:20.272605+0000) 2022-04-04T01:58:20.310 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:20.308+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:20.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:20.621+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:21.273 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:21.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:21.272801+0000) 2022-04-04T01:58:21.414 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:21.326+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:21.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:21.576+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:22.274 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:22.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:22.272928+0000) 2022-04-04T01:58:22.373 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:22.371+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:22.564 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:22.562+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:23.274 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:23.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:23.273062+0000) 2022-04-04T01:58:23.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:23.331+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:23.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:23.582+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:24.274 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:24.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:24.273222+0000) 2022-04-04T01:58:24.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:24.368+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:24.604 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:24.602+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:25.274 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:25.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:25.273389+0000) 2022-04-04T01:58:25.324 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:25.322+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:25.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:25.605+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:26.274 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:26.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:26.273592+0000) 2022-04-04T01:58:26.367 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:26.365+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:26.623 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:26.621+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:27.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:27.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:27.273797+0000) 2022-04-04T01:58:27.324 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:58:27.332 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:58:27.390 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:27.388+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:27.644 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:27.642+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:28.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:28.272+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:28.274004+0000) 2022-04-04T01:58:28.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:28.379+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:28.684 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:28.682+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:29.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:29.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:29.274153+0000) 2022-04-04T01:58:29.372 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:29.369+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:29.641 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:29.640+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:30.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:30.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:30.274326+0000) 2022-04-04T01:58:30.350 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:30.348+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:30.651 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:30.649+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:30.686 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:58:30.683+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:58:31.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:31.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:31.274471+0000) 2022-04-04T01:58:31.345 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:31.343+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:31.636 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:31.634+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:32.275 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:32.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:32.274643+0000) 2022-04-04T01:58:32.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:32.303+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:32.612 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:32.610+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:33.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:33.273+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:33.274811+0000) 2022-04-04T01:58:33.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:33.276+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:33.587 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:33.585+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:34.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:34.274+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:34.274923+0000) 2022-04-04T01:58:34.299 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:34.297+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:34.611 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:34.609+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:35.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:35.258+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:35.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:35.274+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:35.275086+0000) 2022-04-04T01:58:35.577 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:35.576+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:36.220 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:36.218+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:36.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:36.274+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:36.275314+0000) 2022-04-04T01:58:36.594 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:36.593+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:37.258 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:37.257+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:37.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:37.274+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:37.275548+0000) 2022-04-04T01:58:37.643 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:37.642+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:38.221 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:38.220+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:38.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:38.274+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:38.275717+0000) 2022-04-04T01:58:38.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:38.639+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:39.253 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:39.251+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:39.276 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:39.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:39.275935+0000) 2022-04-04T01:58:39.679 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:39.678+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:40.271 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:40.270+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:40.277 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:40.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:40.276097+0000) 2022-04-04T01:58:40.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:40.712+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:41.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:41.259+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:41.277 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:41.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:41.276269+0000) 2022-04-04T01:58:41.715 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:41.714+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:42.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:42.247+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:42.277 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:42.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:42.276416+0000) 2022-04-04T01:58:42.723 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:42.722+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:43.232 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:43.230+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:43.277 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:43.275+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:43.276610+0000) 2022-04-04T01:58:43.772 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:43.771+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:44.250 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:44.248+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:44.277 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:44.276+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:44.276833+0000) 2022-04-04T01:58:44.802 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:44.801+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:45.233 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:45.232+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:45.278 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:45.276+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:45.277052+0000) 2022-04-04T01:58:45.702 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:58:45.700+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:58:45.754 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:45.752+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:46.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:46.198+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:46.278 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:46.276+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:46.277184+0000) 2022-04-04T01:58:46.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:46.754+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:47.178 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:47.176+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:47.278 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:47.276+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:47.277381+0000) 2022-04-04T01:58:47.746 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:47.744+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:48.202 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:48.200+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:48.278 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:48.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:48.277557+0000) 2022-04-04T01:58:48.712 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:48.710+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:49.195 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:49.193+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:49.278 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:49.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:49.277704+0000) 2022-04-04T01:58:49.693 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:49.691+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:50.161 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:50.159+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:50.278 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:50.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:50.277859+0000) 2022-04-04T01:58:50.650 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:50.647+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:51.124 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:51.121+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:51.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:51.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:51.278009+0000) 2022-04-04T01:58:51.649 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:51.647+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:52.159 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:52.156+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:52.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:52.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:52.278172+0000) 2022-04-04T01:58:52.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:52.611+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:53.162 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:53.160+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:53.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:53.277+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:53.278371+0000) 2022-04-04T01:58:53.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:53.605+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:54.141 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:54.138+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:54.279 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:54.278+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:54.278508+0000) 2022-04-04T01:58:54.565 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:54.563+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:55.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:55.148+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:55.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:55.278+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:55.278710+0000) 2022-04-04T01:58:55.604 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:55.602+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:56.126 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:56.124+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:56.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:56.278+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:56.278949+0000) 2022-04-04T01:58:56.579 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:56.577+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:57.169 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:57.167+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:57.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:57.278+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:57.279160+0000) 2022-04-04T01:58:57.443 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:58:57.447 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:58:57.596 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:57.593+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:58.193 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:58.190+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:58.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:58.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:58.279384+0000) 2022-04-04T01:58:58.557 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:58.555+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:58:59.164 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:58:59.162+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:58:59.280 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:58:59.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:58:59.279609+0000) 2022-04-04T01:58:59.554 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:58:59.552+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:00.127 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:00.125+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:00.281 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:00.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:00.279856+0000) 2022-04-04T01:59:00.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:00.578+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:00.704 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:59:00.702+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:59:01.114 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:01.112+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:01.281 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:01.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:01.280075+0000) 2022-04-04T01:59:01.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:01.590+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:02.083 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:02.080+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:02.281 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:02.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:02.280289+0000) 2022-04-04T01:59:02.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:02.578+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:03.125 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:03.122+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:03.281 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:03.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:03.280526+0000) 2022-04-04T01:59:03.610 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:03.608+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:04.150 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:04.148+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:04.281 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:04.279+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:04.280750+0000) 2022-04-04T01:59:04.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:04.581+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:05.187 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:05.185+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:05.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:05.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:05.281013+0000) 2022-04-04T01:59:05.573 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:05.571+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:06.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:06.172+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:06.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:06.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:06.281172+0000) 2022-04-04T01:59:06.581 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:06.579+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:07.217 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:07.214+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:07.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:07.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:07.281376+0000) 2022-04-04T01:59:07.550 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:07.548+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:08.203 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:08.201+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:08.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:08.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:08.281568+0000) 2022-04-04T01:59:08.586 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:08.584+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:09.239 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:09.237+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:09.282 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:09.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:09.281824+0000) 2022-04-04T01:59:09.614 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:09.612+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:10.246 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:10.244+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:10.283 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:10.280+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:10.282041+0000) 2022-04-04T01:59:10.642 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:10.640+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:11.259 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:11.256+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:11.283 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:11.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:11.282267+0000) 2022-04-04T01:59:11.667 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:11.665+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:12.223 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:12.221+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:12.283 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:12.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:12.282469+0000) 2022-04-04T01:59:12.640 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:12.638+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:13.199 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:13.197+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:13.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:13.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:13.282643+0000) 2022-04-04T01:59:13.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:13.611+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:14.230 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:14.227+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:14.283 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:14.281+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:14.282826+0000) 2022-04-04T01:59:14.638 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:14.636+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:15.248 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:15.246+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:15.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:15.282+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:15.283023+0000) 2022-04-04T01:59:15.632 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:15.630+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:15.720 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:59:15.717+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:59:16.274 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:16.272+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:16.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:16.282+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:16.283263+0000) 2022-04-04T01:59:16.649 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:16.647+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:17.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:17.282+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:17.283499+0000) 2022-04-04T01:59:17.322 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:17.320+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:17.622 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:17.620+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:18.284 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:18.282+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:18.283699+0000) 2022-04-04T01:59:18.366 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:18.364+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:18.583 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:18.581+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:19.285 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:19.283+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:19.283910+0000) 2022-04-04T01:59:19.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:19.318+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:19.629 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:19.627+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:20.285 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:20.283+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:20.284084+0000) 2022-04-04T01:59:20.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:20.326+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:20.677 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:20.675+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:21.285 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:21.283+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:21.284280+0000) 2022-04-04T01:59:21.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:21.330+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:21.664 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:21.662+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:22.285 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:22.283+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:22.284496+0000) 2022-04-04T01:59:22.289 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:22.288+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:22.703 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:22.701+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:23.286 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:23.283+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:23.284699+0000) 2022-04-04T01:59:23.319 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:23.317+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:23.682 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:23.679+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:24.286 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:24.284+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:24.284914+0000) 2022-04-04T01:59:24.302 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:24.300+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:24.657 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:24.655+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:25.265 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:25.263+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:25.381 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:25.284+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:25.285111+0000) 2022-04-04T01:59:25.622 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:25.621+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:26.286 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:26.284+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:26.285278+0000) 2022-04-04T01:59:26.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:26.312+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:26.605 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:26.603+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:27.272 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:27.270+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:27.286 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:27.284+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:27.285434+0000) 2022-04-04T01:59:27.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:27.605+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:27.636 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:59:27.640 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:59:28.286 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:28.284+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:28.285647+0000) 2022-04-04T01:59:28.318 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:28.317+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:28.636 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:28.634+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:29.287 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:29.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:29.285869+0000) 2022-04-04T01:59:29.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:29.304+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:29.609 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:29.607+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:30.287 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:30.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:30.286083+0000) 2022-04-04T01:59:30.305 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:30.304+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:30.597 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:30.595+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:30.736 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:59:30.734+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:59:31.287 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:31.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:31.286285+0000) 2022-04-04T01:59:31.328 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:31.327+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:31.586 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:31.584+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:32.287 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:32.285+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:32.286491+0000) 2022-04-04T01:59:32.326 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:32.325+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:32.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:32.583+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:33.287 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:33.286+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:33.286691+0000) 2022-04-04T01:59:33.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:33.330+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:33.580 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:33.578+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:34.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:34.286+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:34.286906+0000) 2022-04-04T01:59:34.374 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:34.372+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:34.569 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:34.567+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:35.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:35.286+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:35.287062+0000) 2022-04-04T01:59:35.332 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:35.330+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:35.571 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:35.570+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:36.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:36.286+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:36.287250+0000) 2022-04-04T01:59:36.344 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:36.342+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:36.576 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:36.575+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:37.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:37.287+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:37.287433+0000) 2022-04-04T01:59:37.351 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:37.350+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:37.544 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:37.542+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:38.288 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:38.287+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:38.287608+0000) 2022-04-04T01:59:38.352 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:38.350+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:38.584 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:38.582+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:39.289 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:39.287+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:39.287828+0000) 2022-04-04T01:59:39.348 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:39.346+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:39.541 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:39.539+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:40.289 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:40.287+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:40.288014+0000) 2022-04-04T01:59:40.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:40.297+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:40.591 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:40.590+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:41.289 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:41.287+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:41.288249+0000) 2022-04-04T01:59:41.333 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:41.332+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:41.607 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:41.606+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:42.289 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:42.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:42.288426+0000) 2022-04-04T01:59:42.298 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:42.296+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:42.574 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:42.572+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:43.256 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:43.253+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:43.289 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:43.287+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:43.288648+0000) 2022-04-04T01:59:43.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:43.533+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:44.216 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:44.214+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:44.289 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:44.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:44.288875+0000) 2022-04-04T01:59:44.500 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:44.497+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:45.200 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:45.198+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:45.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:45.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:45.289094+0000) 2022-04-04T01:59:45.479 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:45.477+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:45.737 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T01:59:45.735+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T01:59:46.210 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:46.207+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:46.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:46.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:46.289236+0000) 2022-04-04T01:59:46.485 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:46.482+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:47.240 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:47.238+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:47.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:47.289+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:47.289460+0000) 2022-04-04T01:59:47.490 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:47.488+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:48.273 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:48.271+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:48.290 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:48.288+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:48.289685+0000) 2022-04-04T01:59:48.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:48.486+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:49.263 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:49.261+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:49.291 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:49.289+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:49.289893+0000) 2022-04-04T01:59:49.440 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:49.438+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:50.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:50.278+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:50.291 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:50.289+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:50.290077+0000) 2022-04-04T01:59:50.433 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:50.430+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:51.291 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:51.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:51.290280+0000) 2022-04-04T01:59:51.314 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:51.312+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:51.386 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:51.384+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:52.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:52.285+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:52.291 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:52.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:52.290500+0000) 2022-04-04T01:59:52.344 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:52.342+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:53.268 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:53.266+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:53.291 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:53.289+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:53.290665+0000) 2022-04-04T01:59:53.383 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:53.381+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:54.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:54.289+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:54.290869+0000) 2022-04-04T01:59:54.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:54.309+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:54.361 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:54.359+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:55.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:55.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:55.291070+0000) 2022-04-04T01:59:55.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:55.323+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:55.360 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:55.358+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:56.276 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:56.274+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:56.417 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:56.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:56.291285+0000) 2022-04-04T01:59:56.418 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:56.389+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:57.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:57.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:57.291496+0000) 2022-04-04T01:59:57.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:57.317+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:57.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:57.417+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:57.833 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:59:57.837 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T01:59:58.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:58.274+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:58.292 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:58.290+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:58.291746+0000) 2022-04-04T01:59:58.414 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:58.412+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T01:59:59.293 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T01:59:59.291+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T00:59:59.291974+0000) 2022-04-04T01:59:59.309 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T01:59:59.306+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T01:59:59.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T01:59:59.387+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:00.293 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:00.291+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:00.292185+0000) 2022-04-04T02:00:00.320 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:00.318+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:00.372 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:00.370+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:00.753 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T02:00:00.750+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T02:00:01.293 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:01.291+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:01.292417+0000) 2022-04-04T02:00:01.296 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:01.293+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:01.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:01.326+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:02.293 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:02.291+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:02.292636+0000) 2022-04-04T02:00:02.314 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:02.312+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:02.346 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:02.344+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:03.278 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:03.276+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:03.293 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:03.292+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:03.292832+0000) 2022-04-04T02:00:03.306 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:03.304+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:04.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:04.264+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:04.272 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:04.270+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:04.294 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:04.292+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:04.293076+0000) 2022-04-04T02:00:05.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:05.227+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:05.294 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:05.292+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:05.293247+0000) 2022-04-04T02:00:05.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:05.309+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:06.234 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:06.232+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:06.294 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:06.292+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:06.293485+0000) 2022-04-04T02:00:06.313 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:06.310+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:07.185 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:07.182+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:07.282 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:07.279+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:07.294 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:07.292+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:07.293691+0000) 2022-04-04T02:00:08.185 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:08.183+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:08.252 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:08.250+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:08.295 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:08.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:08.293890+0000) 2022-04-04T02:00:09.206 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:09.204+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:09.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:09.245+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:09.295 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:09.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:09.294099+0000) 2022-04-04T02:00:10.182 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:10.180+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:10.281 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:10.278+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:10.295 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:10.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:10.294268+0000) 2022-04-04T02:00:11.148 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:11.146+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:11.237 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:11.235+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:11.295 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:11.293+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:11.294519+0000) 2022-04-04T02:00:12.165 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:12.163+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:12.209 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:12.207+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:12.295 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:12.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:12.294699+0000) 2022-04-04T02:00:13.180 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:13.178+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:13.206 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:13.204+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:13.296 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:13.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:13.294913+0000) 2022-04-04T02:00:14.174 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:14.172+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:14.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:14.210+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:14.296 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:14.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:14.295122+0000) 2022-04-04T02:00:15.195 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:15.193+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:15.202 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:15.200+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:15.296 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:15.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:15.295322+0000) 2022-04-04T02:00:15.769 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T02:00:15.767+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T02:00:16.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:16.173+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:16.206 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:16.204+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:16.296 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:16.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:16.295533+0000) 2022-04-04T02:00:17.135 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:17.133+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:17.159 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:17.157+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:17.297 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:17.294+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:17.295740+0000) 2022-04-04T02:00:18.127 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:18.125+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:18.154 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:18.152+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:18.297 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:18.295+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:18.296000+0000) 2022-04-04T02:00:19.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:19.137+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:19.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:19.172+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:19.297 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:19.295+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:19.296229+0000) 2022-04-04T02:00:20.105 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:20.102+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:20.217 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:20.215+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:20.297 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:20.295+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:20.296436+0000) 2022-04-04T02:00:21.139 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:21.137+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:21.224 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:21.222+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:21.297 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:21.296+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:21.296655+0000) 2022-04-04T02:00:22.137 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:22.135+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:22.238 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:22.237+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:22.298 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:22.296+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:22.296851+0000) 2022-04-04T02:00:23.129 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:23.128+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:23.277 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:23.275+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:23.298 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:23.296+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:23.297065+0000) 2022-04-04T02:00:24.147 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:24.146+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:24.298 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:24.296+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:24.297241+0000) 2022-04-04T02:00:24.321 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:24.320+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:25.140 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:25.138+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:25.298 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:25.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:25.297458+0000) 2022-04-04T02:00:25.325 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:25.323+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:26.174 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:26.172+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:26.286 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:26.285+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:26.298 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:26.296+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:26.297676+0000) 2022-04-04T02:00:27.129 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:27.127+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:27.288 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:27.279+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:27.299 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:27.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:27.297927+0000) 2022-04-04T02:00:27.898 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T02:00:27.904 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T02:00:28.121 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:28.119+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:28.299 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:28.297+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:28.298109+0000) 2022-04-04T02:00:28.327 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:28.326+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:29.077 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:29.076+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:29.299 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:29.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:29.298312+0000) 2022-04-04T02:00:29.331 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:29.329+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:30.037 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:30.036+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:30.299 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:30.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:30.298534+0000) 2022-04-04T02:00:30.379 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:30.377+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:30.770 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T02:00:30.768+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T02:00:31.042 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:31.040+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:31.299 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:31.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:31.298737+0000) 2022-04-04T02:00:31.415 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:31.414+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:32.082 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:32.080+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:32.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:32.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:32.298904+0000) 2022-04-04T02:00:32.438 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:32.437+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:33.118 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:33.117+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:33.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:33.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:33.299119+0000) 2022-04-04T02:00:33.447 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:33.445+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:34.088 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:34.087+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:34.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:34.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:34.299320+0000) 2022-04-04T02:00:34.467 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:34.465+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:35.126 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:35.125+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:35.300 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:35.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:35.299513+0000) 2022-04-04T02:00:35.431 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:35.430+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:36.124 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:36.122+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:36.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:36.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:36.299702+0000) 2022-04-04T02:00:36.452 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:36.451+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:37.142 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:37.140+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:37.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:37.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:37.299847+0000) 2022-04-04T02:00:37.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:37.459+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:38.150 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:38.148+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:38.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:38.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:38.299991+0000) 2022-04-04T02:00:38.459 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:38.457+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:39.186 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:39.183+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:39.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:39.298+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:39.300118+0000) 2022-04-04T02:00:39.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:39.453+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:40.229 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:40.226+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:40.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:40.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:40.300311+0000) 2022-04-04T02:00:40.499 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:40.497+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:41.209 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:41.207+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:41.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:41.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:41.300551+0000) 2022-04-04T02:00:41.477 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:41.475+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:42.252 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:42.249+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:42.301 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:42.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:42.300711+0000) 2022-04-04T02:00:42.495 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:42.493+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:43.262 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:43.259+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:43.302 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:43.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:43.300903+0000) 2022-04-04T02:00:43.472 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:43.469+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:44.248 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:44.245+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:44.302 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:44.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:44.301041+0000) 2022-04-04T02:00:44.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:44.452+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:45.235 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:45.233+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:45.302 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:45.300+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:45.301251+0000) 2022-04-04T02:00:45.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:45.417+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:45.786 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T02:00:45.784+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T02:00:46.274 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:46.271+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:46.302 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:46.300+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:46.301470+0000) 2022-04-04T02:00:46.415 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:46.412+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:47.273 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:47.270+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:47.303 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:47.299+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:47.301738+0000) 2022-04-04T02:00:47.407 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:47.405+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:48.232 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:48.229+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:48.303 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:48.301+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:48.301994+0000) 2022-04-04T02:00:48.392 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:48.390+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:49.198 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:49.196+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:49.303 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:49.301+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:49.302220+0000) 2022-04-04T02:00:49.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:49.366+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:50.197 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:50.194+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:50.303 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:50.301+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:50.302439+0000) 2022-04-04T02:00:50.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:50.381+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:51.221 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:51.219+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:51.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:51.301+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:51.302644+0000) 2022-04-04T02:00:51.418 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:51.416+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:52.225 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:52.223+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:52.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:52.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:52.302859+0000) 2022-04-04T02:00:52.383 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:52.380+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:53.192 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:53.190+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:53.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:53.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:53.303087+0000) 2022-04-04T02:00:53.409 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:53.407+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:54.160 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:54.157+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:54.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:54.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:54.303281+0000) 2022-04-04T02:00:54.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:54.368+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:55.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:55.180+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:55.304 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:55.302+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:55.303504+0000) 2022-04-04T02:00:55.337 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:55.335+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:56.161 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:56.158+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:56.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:56.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:56.303721+0000) 2022-04-04T02:00:56.368 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:56.365+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:57.183 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:57.180+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:57.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:57.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:57.303936+0000) 2022-04-04T02:00:57.329 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:57.327+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:58.088 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T02:00:58.194 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T02:00:58.204 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:58.202+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:58.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:58.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:58.304100+0000) 2022-04-04T02:00:58.370 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:58.368+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:00:59.247 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:00:59.244+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:00:59.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:00:59.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:00:59.304311+0000) 2022-04-04T02:00:59.419 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:00:59.417+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:00.257 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:00.254+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:00.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:00.303+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:00.304533+0000) 2022-04-04T02:01:00.434 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:00.432+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:00.802 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T02:01:00.800+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T02:01:01.279 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:01.277+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:01.305 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:01.304+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:01.304730+0000) 2022-04-04T02:01:01.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:01.431+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:02.270 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:02.268+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:02.306 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:02.304+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:02.304955+0000) 2022-04-04T02:01:02.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:02.441+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:03.288 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:03.286+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:03.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:03.304+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:03.305203+0000) 2022-04-04T02:01:03.455 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:03.453+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:04.249 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:04.247+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:04.306 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:04.304+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:04.305372+0000) 2022-04-04T02:01:04.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:04.404+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:05.211 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:05.209+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:05.306 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:05.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:05.305561+0000) 2022-04-04T02:01:05.395 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:05.393+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:06.250 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:06.248+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:06.306 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:06.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:06.305804+0000) 2022-04-04T02:01:06.428 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:06.425+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:07.253 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:07.251+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:07.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:07.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:07.305950+0000) 2022-04-04T02:01:07.435 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:07.433+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:08.231 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:08.229+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:08.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:08.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:08.306087+0000) 2022-04-04T02:01:08.400 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:08.398+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:09.277 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:09.275+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:09.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:09.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:09.306245+0000) 2022-04-04T02:01:09.424 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:09.422+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:10.303 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:10.301+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:10.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:10.305+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:10.306388+0000) 2022-04-04T02:01:10.441 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:10.439+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:11.291 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:11.289+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:11.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:11.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:11.306530+0000) 2022-04-04T02:01:11.431 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:11.428+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:12.307 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:12.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:12.306671+0000) 2022-04-04T02:01:12.310 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:12.308+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:12.397 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:12.395+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:13.308 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:13.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:13.306832+0000) 2022-04-04T02:01:13.339 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:13.336+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:13.406 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:13.404+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:14.308 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:14.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:14.307025+0000) 2022-04-04T02:01:14.357 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:14.355+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:14.412 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:14.409+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:15.308 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:15.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:15.307229+0000) 2022-04-04T02:01:15.328 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:15.327+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:15.456 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:15.454+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:15.803 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T02:01:15.801+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T02:01:16.308 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:16.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:16.307399+0000) 2022-04-04T02:01:16.378 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:16.377+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:16.453 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:16.451+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:17.308 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:17.307+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:17.307569+0000) 2022-04-04T02:01:17.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:17.418+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:17.424 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:17.423+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:18.308 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:18.307+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:18.307750+0000) 2022-04-04T02:01:18.431 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:18.429+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:18.450 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:18.448+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:19.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:19.307+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:19.307908+0000) 2022-04-04T02:01:19.412 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:19.410+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:19.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:19.418+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:20.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:20.307+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:20.308097+0000) 2022-04-04T02:01:20.393 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:20.392+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:20.428 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:20.427+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:21.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:21.307+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:21.308285+0000) 2022-04-04T02:01:21.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:21.378+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:21.392 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:21.390+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:22.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:22.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:22.308478+0000) 2022-04-04T02:01:22.403 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:22.401+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:22.426 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:22.425+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:23.309 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:23.306+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:23.308738+0000) 2022-04-04T02:01:23.411 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:23.409+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:23.433 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:23.432+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:24.310 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:24.307+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:24.308978+0000) 2022-04-04T02:01:24.381 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:24.380+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:24.464 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:24.463+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:25.310 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:25.308+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:25.309197+0000) 2022-04-04T02:01:25.409 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:25.407+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:25.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:25.484+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:26.310 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:26.308+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:26.309410+0000) 2022-04-04T02:01:26.360 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:26.359+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:26.475 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:26.473+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:27.310 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:27.307+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:27.309633+0000) 2022-04-04T02:01:27.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:27.378+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:27.463 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:27.462+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:28.311 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:28.308+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:28.309851+0000) 2022-04-04T02:01:28.343 DEBUG:teuthology.orchestra.run.smithi077:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T02:01:28.347 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-04-04T02:01:28.398 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:28.396+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:28.474 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:28.473+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:29.311 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:29.309+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:29.310118+0000) 2022-04-04T02:01:29.442 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:29.440+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:29.516 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:29.515+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:30.311 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:30.309+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:30.310346+0000) 2022-04-04T02:01:30.428 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:30.426+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:30.535 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:30.534+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:30.819 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T02:01:30.816+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T02:01:31.311 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:31.309+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:31.310584+0000) 2022-04-04T02:01:31.427 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:31.425+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:31.514 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:31.513+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:32.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:32.309+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:32.310736+0000) 2022-04-04T02:01:32.442 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:32.439+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:32.523 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:32.520+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:33.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:33.310+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:33.310917+0000) 2022-04-04T02:01:33.410 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:33.407+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:33.481 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:33.479+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:34.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:34.310+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:34.311115+0000) 2022-04-04T02:01:34.407 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:34.404+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:34.487 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:34.484+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:35.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:35.310+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:35.311335+0000) 2022-04-04T02:01:35.402 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:35.400+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:35.488 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:35.485+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:36.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:36.310+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:36.311503+0000) 2022-04-04T02:01:36.448 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:36.445+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:36.451 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:36.449+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:37.312 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:37.310+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:37.311673+0000) 2022-04-04T02:01:37.415 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:37.412+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:37.442 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:37.439+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:38.313 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:38.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:38.311870+0000) 2022-04-04T02:01:38.455 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:38.452+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:38.486 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:38.484+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:39.313 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:39.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:39.312095+0000) 2022-04-04T02:01:39.406 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:39.404+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:39.465 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:39.463+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:40.313 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:40.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:40.312248+0000) 2022-04-04T02:01:40.389 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:40.387+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:40.493 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:40.491+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:41.313 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:41.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:41.312412+0000) 2022-04-04T02:01:41.439 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:41.437+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:41.528 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:41.526+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:42.313 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:42.311+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:42.312591+0000) 2022-04-04T02:01:42.396 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:42.393+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:42.524 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:42.522+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:43.314 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:43.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:43.312820+0000) 2022-04-04T02:01:43.384 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:43.382+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:43.522 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:43.519+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:44.314 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:44.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:44.313013+0000) 2022-04-04T02:01:44.426 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:44.424+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:44.548 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:44.546+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:45.314 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:45.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:45.313208+0000) 2022-04-04T02:01:45.442 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:45.440+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:45.561 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:45.559+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:45.835 INFO:tasks.ceph.mon.a.smithi077.stderr:2022-04-04T02:01:45.833+0000 7f66829bc700 -1 mon.a@0(leader) e1 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-04-04T02:01:46.314 INFO:tasks.ceph.mds.a.smithi077.stderr:2022-04-04T02:01:46.312+0000 7fd61fa12700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-04-04T01:01:46.313430+0000) 2022-04-04T02:01:46.419 INFO:tasks.ceph.osd.6.smithi186.stderr:2022-04-04T02:01:46.417+0000 7fd3b190b700 -1 osd.6 142 get_health_metrics reporting 24 slow ops, oldest is osd_op(client.4213.0:34 2.1 2.3e6a8311 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e73) 2022-04-04T02:01:46.547 INFO:tasks.ceph.osd.4.smithi186.stderr:2022-04-04T02:01:46.545+0000 7f88d0ac1700 -1 osd.4 142 get_health_metrics reporting 72 slow ops, oldest is osd_op(mds.0.52:2 3.b 3.3270c60b (undecoded) ondisk+read+known_if_redirected+full_force+supports_pool_eio e65) 2022-04-04T02:01:46.688 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-04-04T02:01:46.692 DEBUG:teuthology.task.console_log:Killing console logger for smithi077 2022-04-04T02:01:46.693 DEBUG:teuthology.task.console_log:Killing console logger for smithi186 2022-04-04T02:01:46.693 DEBUG:teuthology.exit:Finished running handlers